KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #codex #typescript #ai-agents #claude-code #dsh #dsh-plugin #open-source #cli #deepseek-harness #macos #ai-agent

smartbugs-results

★ 86 Open GitHub ↗

This repository contains the RAW results of the vulnerability analysis of 9 tools on 47,587 smart contracts. These results are presented and discussed in Durieux et al. (ICSE 2020)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Vulnerability Analysis of Smart Contracts using SmartBugs

This repository contains the RAW results of the vulnerability analysis of 9 tools on 47,587 smart contracts.
We used two datasets of vulnerabilities: 1) 69 annotated vulnerable contracts 2) 47,518 contracts taken from the Ethereum network.

The raw results of the analysis on the first dataset are stored in `results/<name_tool>/curated/<contract_name>`.
The raw results of the analysis on the second dataset are stored in `results/<name_tool>/icse20/<contract_address>`.

These results are presented and discussed in [the following ICSE 2020 paper](https://arxiv.org/abs/1910.10601):

```
Thomas Durieux, João F. Ferreira, Rui Abreu, and Pedro Cruz.
Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts.
In ICSE 2020.
```

## Structure of the repository

```
├─ metadata
│  ├─ balances.json
│  ├─ duplicates.json
│  ├─ eth_price.json
│  ├─ nb_lines.csv
│  ├─ results_curated.json
│  ├─ results_wild.json
│  ├─ unique_contracts.csv
│  ├─ vulnerabilities.json
│  └─ vulnerabilities_mapping.csv
├─ plots
│  └─ <plot_name>.png
├─ results
│  └─ <tool_name>
│     └─ <dataset_name>
│        └─ <contract_address>
│           ├─ <result.log>  # stdout of the analysis
│           └─ <result.json> # parsable output analysis
├─ script
│  ├─ combine_appraoches.py
│  ├─ generate_plot.py
│  ├─ generate_results_curated.py
│  └─ generate_results_wild.py

```

## SB Curated Results

# Execution Time Stats

|  #  | Tool       | Avg. Execution Time | Total Execution Time |
| --- | ---------- | ------------------- | -------------------- |
|   1 | Honeybadger | 0:00:46    | 0:53:11    |
|   2 | Maian      | 0:02:57    | 3:23:50    |
|   3 | Manticore  | 0:08:11    | 5:03:04    |
|   4 | Mythril    | 0:01:13    | 1:23:42    |
|   5 | Osiris     | 0:00:44    | 0:50:03    |
|   6 | Oyente     | 0:00:36    | 0:41:29    |
|   7 | Securify   | 0:01:00    | 1:09:08    |
|   8 | Slither    | 0:00:03    | 0:03:35    |
|   9 | Smartcheck | 0:00:06    | 0:06:34    |

Total: 13:34:37

# Accuracy

|  Category           | Honeybadger |    Maian    |  Manticore  |   Mythril   |   Osiris    |   Oyente    |  Securify   |   Slither   | Smartcheck  |    Total    |
| ------------------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| Access Control      |   0/19   0% |   0/19   0% |   4/19  21% |   4/19  21% |   0/19   0% |   0/19   0% |   0/19   0% |   4/19  21% |   2/19  11% |   5/19  26% |
| Arithmetic          |   0/22   0% |   0/22   0% |   4/22  18% |  15/22  68% |  11/22  50% |  12/22  55% |   0/22   0% |   0/22   0% |   1/22   5% |  19/22  86% |
| Denial Service      |    0/7   0% |    0/7   0% |    0/7   0% |    0/7   0% |    0/7   0% |    0/7   0% |    0/7   0% |    0/7   0% |    0/7   0% |   0/ 7   0% |
| Front Running       |    0/7   0% |    0/7   0% |    0/7   0% |    2/7  29% |    0/7   0% |    0/7   0% |    2/7  29% |    0/7   0% |    0/7   0% |   2/ 7  29% |
| Reentrancy          |    0/8   0% |    0/8   0% |    2/8  25% |    5/8  62% |    5/8  62% |    5/8  62% |    5/8  62% |    7/8  88% |    5/8  62% |   7/ 8  88% |
| Time Manipulation   |    0/5   0% |    0/5   0% |    1/5  20% |    0/5   0% |    0/5   0% |    0/5   0% |    0/5   0% |    2/5  40% |    1/5  20% |   3/ 5  60% |
| Unchecked Low Calls |   0/12   0% |   0/12   0% |   2/12  17% |   5/12  42% |   0/12   0% |   0/12   0% |   3/12  25% |   4/12  33% |   4/12  33% |   9/12  75% |
| Other               |    2/3  67% |    0/3   0% |    0/3   0% |    0/3   0% |    0/3   0% |    0/3   0% |    0/3   0% |    3/3 100% |    0/3   0% |   3/ 3 100% |
| Total               |  2/115   2% |  0/115   0% | 13/115  11% | 31/115  27% | 16/115  14% | 17/115  15% | 10/115   9% | 20/115  17% | 13/115  11% | 48/115  42% |

# Nb Detected Vulnerabilities

| Category            | Honeybadger |    Maian    |  Manticore  |   Mythril   |   Osiris    |   Oyente    |  Securify   |   Slither   | Smartcheck  |    Total    |
| ------------------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| Access Control      |           0 |          10 |          28 |          24 |           0 |           0 |           6 |          20 |           3 |          91 |
| Arithmetic          |           0 |           0 |          11 |          92 |          62 |          69 |           0 |           0 |          23 |         257 |
| Denial Service      |           0 |           0 |           0 |           0 |          27 |          11 |           0 |           2 |          19 |          59 |
| Front Running       |           0 |           0 |           0 |          21 |           0 |           0 |          55 |           0 |           0 |          76 |
| Reentrancy          |           0 |           0 |           4 |          16 |           5 |           5 |          32 |          15 |           7 |          84 |
| Time Manipulation   |           0 |           0 |           4 |           0 |           4 |           5 |           0 |           5 |           2 |          20 |
| Unchecked Low Calls |           0 |           0 |           4 |          30 |           0 |           0 |          21 |          13 |          14 |          82 |
| Other               |           5 |           2 |          25 |          32 |           0 |           0 |           0 |          28 |           8 |         100 |
| Total               |           5 |          12 |          76 |         215 |          98 |          90 |         114 |          83 |          76 |         769 |

# Combine tools 
|             | Honeybadger |    Maian    |  Manticore  |   Mythril   |   Osiris    |   Oyente    |  Securify   |   Slither   | Smartcheck  |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| Honeybadger |             | 2/115    2% | 15/115  13% | 33/115  29% | 18/115  16% | 19/115  17% | 12/115  10% | 20/115  17% | 15/115  13% |
| Maian       |             |             | 13/115  11% | 31/115  27% | 16/115  14% | 17/115  15% | 10/115   9% | 20/115  17% | 13/115  11% |
| Manticore   |             |             |             | 32/115  28% | 26/115  23% | 26/115  23% | 19/115  17% | 27/115  23% | 20/115  17% |
| Mythril     |             |             |             |             | 33/115  29% | 33/115  29% | 31/115  27% | 42/115  37% | 33/115  29% |
| Osiris      |             |             |             |             |             | 22/115  19% | 21/115  18% | 31/115  27% | 23/115  20% |
| Oyente      |             |             |             |             |             |             | 22/115  19% | 32/115  28% | 25/115  22% |
| Securify    |             |             |             |             |             |             |             | 25/115  22% | 16/115  14% |
| Slither     |             |             |             |             |             |             |             |             | 25/115  22% |
| Smartcheck  |             |             |             |             |             |             |             |             |             |

## SB Wild

# Execution Time Stat

|  #  | Tool       | Avg. Execution Time | Total Execution Time |
| --- | ---------- | ------------------- | -------------------- |
|   1 | Honeybadger | 0:01:38    | 23 days, 13:40:00 |
|   2 | Maian      | 0:05:16    | 49 days, 10:06:15 |
|   3 | Manticore  | 0:24:28    | 184 days, 1:59:02 |
|   4 | Mythril    | 0:01:24    | 46 days, 7:46:55 |
|   5 | Osiris     | 0:00:34    | 18 days, 10:19:01 |
|   6 | Oyente     | 0:00:30    | 16 days, 4:50:11 |
|   7 | Securify   | 0:06:37    | 217 days, 22:46:26 |
|   8 | Slither    | 0:00:05    | 2 days, 15:09:36 |
|   9 | Smartcheck | 0:00:10    | 5 days, 12:33:14 |

Total: 564 days, 3:10:39

# Nb Detected Vulnerabilities

| Category            | Honeybadger |    Maian    |  Manticore  |   Mythril   |   Osiris    |   Oyente    |  Securify   |   Slither   | Smartcheck  |    Total    |
| ------------------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
| Access Control      | 0 0.00%     | 44 0.09%    | 47 0.10%    | 1076 2.27%  | 0 0.00%     | 2 0.00%     | 614 1.29%   | 2356 4.97%  | 384 0.81%   | 3801 8.01%  |
| Arithmetic          | 1 0.00%     | 0 0.00%     | 102 0.21%   | 18515 39.02% | 13922 29.34% | 34306 72.30% | 0 0.00%     | 0 0.00%     | 7430 15.66% | 37597 79.23% |
| Denial Service      | 0 0.00%     | 0 0.00%     | 0 0.00%     | 0 0.00%     | 485 1.02%   | 880 1.85%   | 0 0.00%     | 2555 5.38%  | 11621 24.49% | 12419 26.17% |
| Front Running       | 0 0.00%     | 0 0.00%     | 0 0.00%     | 2015 4.25%  | 0 0.00%     | 0 0.00%     | 7217 15.21% | 0 0.00%     | 0 0.00%     | 8161 17.20% |
| Reentrancy          | 19 0.04%    | 0 0.00%     | 2 0.00%     | 8454 17.82% | 496 1.05%   | 308 0.65%   | 2033 4.28%  | 8764 18.47% | 847 1.78%   | 14747 31.08% |
| Time Manipulation   | 0 0.00%     | 0 0.00%     | 90 0.19%    | 0 0.00%     | 1470 3.10%  | 1452 3.06%  | 0 0.00%     | 1988 4.19%  | 68 0.14%    | 4069 8.58%  |
| Unchecked Low Calls | 0 0.00%     | 0 0.00%     | 4 0.01%     | 443 0.93%   | 0 0.00%     | 0 0.00%     | 592 1.25%   | 12199 25.71% | 2867 6.04%  | 14656 30.89% |
| Other               | 26 0.05%    | 135 0.28%   | 1032 2.17%  | 11126 23.45% | 0 0.00%     | 0 0.00%     | 561 1.18%   | 9133 19.25% | 14113 29.74% | 28355 59.76% |
| Total               | 46 0.10%    | 179 0.38%   | 1203 2.54%  | 22994 48.46% | 14665 30.91% | 34764 73.26% | 8781 18.51% | 22269 46.93% | 24906 52.49% | 44589 93.97% |

smartbugs-results

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README