KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

vulntest

★ 37 Open GitHub ↗

Static code analysis test source code

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Little test programs to test static code analysis software

## Introduction

These small code pieces were used by me to analyse the detective abilities
of a few SCA solutions to identify security vulnerabilities in source code.
This was not a comprehensive test, but rather a quick analysis.

Initially I wanted to create sophisticated bugs and check if they can find
them but found out that even easy test cases can be too hard ;-)

The result of my testing can be read in the german article here:
[https://www.heise.de/hintergrund/Pruefstand-fuer-Testwerkzeuge-Codeanalyse-im-Praxiseinsatz-4679430.html?seite=all](https://www.heise.de/hintergrund/Pruefstand-fuer-Testwerkzeuge-Codeanalyse-im-Praxiseinsatz-4679430.html?seite=all)

## The test source code

Alphabetically:

- andreas.cpp - hard coded credentials
 - bof.c - buffer overflow based on untrusted length information
 - confuse.cpp - type confusion vulnerability
 - double_free.cpp - copy of a CWE415 NIST test source code
 - encrypt.c - weak encryption mode
 - fdleak.c - leaking a file descriptor to a child process
 - hardcoded_password.c - hard coded credentials
 - ok.c - nothing wrong here, for testing false positives and standard "warnings"
 - ok2.c - nothing wrong here, for testing false positives and standard "warnings"
 - strcpy.c - standard strcpy buffer overflow (and format string vulnerability which was not the test target)
 - strcpy2.c - standard string copy via while-loop (and format string vulnerability which was not the test target)
 - tricky.c - buffer overflow based on untrusted length information
 - use_after_free.c - use after free vulnerability

## Results

All vendors reviewed the setup and verified that the results were real and not
based on a misconfiguration.
And yes, most of them were not happy with the results.

|   CODE/SOFTWARE    |Perforce Klocwork 19.01|Parasoft C/C++test 10.4.2|Mathworks Bugfinder R2019a (2)|Viva 64 PVS Studio|CodeQL Oct 2022|joern Oct 2022|semgrep Oct 2022|cppcheck 2.9|llvm 14 clang-analyzer|ChatGPT 4.0|
|:------------------:|:------:|:-------:|:--------|:-------------:|:------:|:------:|:------:|:------------:|:------------:|:-----:|
| andreas            | yes    | no      | no      | no            | no     | no     | no     | no           | no           | yes   |
| bof                | yes    | no      | yes     | no            | yes    | no     | no     | no           | yes(3)       | yes   |
| confuse            | no     | no      | no      | no            | no     | no     | no     | no           | no           | yes   |
| double_free        | yes    | no      | no      | yes           | no     | no     | no     | no           | yes          | yes   |
| encrypt            | yes    | no      | yes     | no            | yes(1) | no     | no     | no           | no           | yes   |
| fdleak             | no     | no      | no      | no            | no     | no     | no     | no           | no           | partial |
| hardcoded_password | yes    | no      | no      | no            | no     | no     | no     | no           | no           | yes   |
| strcpy             | no     | yes     | no      | no            | yes(1) | no     | no     | no           | no           | yes   |
| strcpy2            | no     | no      | no      | no            | no     | no     | no     | no           | no           | yes   |
| tricky             |partial | no      | no      | yes           | no     | no     | no     | no           | no           | yes   |
| use_after_free     | yes    | yes     | yes     | yes           | no     | no     | yes    | yes          | yes          | yes   |

(1) Not in cpp-lgtm.qls but from cpp/ql/src/Security

(2) For Mathworks Bugfinder, there is also the Code Prover product which checks
    for robustness which includes buffer overruns. This might detect the buffer
    overflow bugs but is a seperate product that is not a SCA comparable to the
    others.

(3) with alpha.security checkers enabled

vulntest

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README