From text to threats: A language model approach to software vulnerability detection

Figures & Tables

Model	Score	SARD	SeVC	Devign	D2A

VulBERTa	88.7	84.2	80.5	81.8	79.9
SySeVR	81.5	82.6	78.3	80.2	72.7
DistilVulBERT	94.0	91.4	82.2	87.5	85.9

Hyperparameter	GPT-2	CodeBERT	LSTM

Learning rate	0.001	0.0005	0.01
Batch size	32	64	128
Epochs	5	10	3
Optimizer	Adam	AdamW	RMSprop
Dropout rate	0.1	0.05	0.2
Hidden units	768	312	256
Attention heads	12	8	–
Layers	12	12	1

Require: Set of labeled training data D = {(x_i,y_i)}

Require: Set of K teacher models T = T_k

Require: Student model S

Ensure: Trained student model

1: Initialize student model parameters θ_S randomly.

2: for each teacher model T_k ∈ T do

3: Compute predictions p_k (x) for each x_i ∈ D.

4: Initialize student model weights to match T_k.

5: Train student model on D using: KDLoss

(θS,θT(k);D)=1n∑i=1nDKL(pk(xi)∥qs(xi;θS,θT(k)))

(\theta_S,\theta^{(k)}_{T};D)=\frac{1}{n}\sum^\nolimits{n}_{i=1}D_{KL}(p_k(x_i)\parallel q_s (x_i;\theta_S,\theta^{(k)}_T))

where D_KL denotes Kullback-Leibler divergence and qs(xi;θS,θT(k))

q_s (x_i;\theta_S,\theta^{(k)}_T)

is the softmax output of student model.

6: end for

7: return Trained student model S