Fig. 1
An overview of our defense framework.
Table 1
The average accuracy and the standard deviation for BERT base on SARD, D2A, REVEAL and DEVIGN over 5 training set sizes.
| Line | Examples | Method | SARD | D2A | REVEAL | Devign |
|---|---|---|---|---|---|---|
| 1. | | T |= 0 | unsupervised (avg) | 38.8±9.6 | 69.5±7.2 | 44.0±9.1 | 39.1±4.3 |
| 2. | | T |= 0 | unsupervised (max) | 42.8±0.0 | 79.4±0.0 | 56.4±0.0 | 43.8±0.0 |
| 3. | | T |= 0 | iPet | 66.7±0.2 | 89.5±0.1 | 73.7±0.1 | 63.6±0.1 |
| 4. | | T |= 15 | supervised | 32.1±1.6 | 25.0±0.1 | 10.1±0.1 | 34.2±2.1 |
| 5. | | T |= 15 | Pet | 52.9±0.1 | 87.5±0.0 | 63.8±0.2 | 41.8±0.1 |
| 6. | | T |= 15 | iPet | 57.6±0.0 | 89.3±0.1 | 70.7±0.1 | 43.2±0.0 |
| 7. | | T |= 60 | supervised | 44.8±2.7 | 82.1±2.5 | 52.5±3.1 | 45.6±1.8 |
| 8. | | T |= 60 | Pet | 60.0±0.1 | 86.3±0.0 | 66.2±0.1 | 63.9±0.0 |
| 9. | | T |= 60 | iPet | 64.7±0.1 | 88.4±0.1 | 69.7±0.0 | 67.4±0.3 |
| 10. | | T |= 200 | supervised | 53.0±3.1 | 86.0±0.7 | 62.9±0.9 | 47.9±2.8 |
| 11. | | T |= 200 | Pet | 61.9±0.0 | 88.3±0.1 | 69.2±0.0 | 74.7±0.3 |
| 12. | | T |= 200 | iPet | 62.9±0.0 | 89.6±0.1 | 71.2±0.1 | 78.4±0.7 |
| 13. | | τ |= 1000 | supervised | 63.0±0.5 | 86.9±0.4 | 70.5±0.3 | 73.1±0.2 |
| 14. | | τ |= 1000 | Pet | 68.8±0.1 | 89.9±0.2 | 72.7±0.0 | 85.3±0.2 |
Fig. 2
The inclusion of Additional Language Modeling during training resulted in improvements in accuracy for PET.
Table 2
A comparison of PET with VulBERTa and VulDeBERT methods using BERT (base).
| Ex. | Method | SARD | D2A | REVEAL | Devign |
|---|---|---|---|---|---|
| | T |= 15 | VulDeBERT | 40.45 | 72.6 | 36.7 | 34.7 |
| | T |= 15 | VulBERTa | 43.23 | 81.1 | 320.6 | 32.9 |
| | T |= 15 | Pet | 49.60 | 84.1 | 59.0 | 39.5 |
| | T |= 15 | iPet | 54.60 | 87.5 | 67.0 | 42.1 |
| | T |= 60 | VulDeBERT | 46.6 | 83.0 | 60.2 | 40.8 |
| | T |= 60 | VulBERTa | 39.5 | 84.8 | 61.5 | 34.8 |
| | T |= 60 | Pet | 55.3 | 86.4 | 63.3 | 55.1 |
| | T |= 60 | iPet | 57.7 | 87.3 | 69.6 | 56.3 |