ML-Executable-Malware-Detector

KO | EN

gitlite — search

#javascript #react #ai #llm #go #golang #machine-learning #rust #deep-learning #linux #awesome #kubernetes #security #mcp

ML-Executable-Malware-Detector

(★ 110)

An offline, machine learning-powered static analysis tool that detects malware in Windows executables using LightGBM and the EMBER dataset.

ML-Executable-Malware-Detector Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Machine Learning Based Malware Detector for Executable Files

An advanced, offline static malware detection engine that uses Machine Learning (LightGBM) and a comprehensive feature extraction pipeline to classify Windows Portable Executables (PE) as **Clean** or **Malicious** without executing them.

## Technical Details

This project is built around the extraction of static features from raw binary files, drawing heavy inspiration from the [EMBER](https://github.com/elastic/ember) (Endgame Malware BEnchmark for Research) dataset. The pipeline consists of two major components: Feature Extraction and Classification.

### 1. Feature Extraction (`core/extractor.py`)
Instead of analyzing raw bytes, the application relies on the `lief` parsing library to deconstruct the PE file and extract hundreds of meaningful attributes. The feature topology (currently tuned for EMBER feature version 1) converts the binary into a fixed-length numerical vector (2351 dimensions). The features include:
* **Byte Histogram (256-dim):** Counts the occurrences of each byte value, helping to identify packed or encrypted files.
* **Byte Entropy Histogram (256-dim):** Approximates the joint probability of byte value and local entropy, which is highly indicative of obfuscation.
* **Section Information:** Extracts properties of the PE sections (e.g., `.text`, `.rdata`, `.data`), including their count, sizes, virtual sizes, entropy, and access permissions (Read/Write/Execute). This is hashed to a fixed dimension.
* **Imported Libraries and Functions (1280-dim):** Uses the "hashing trick" (via `FeatureHasher`) to map the dynamically imported Windows APIs into a fixed-size vector. Malware often relies on specific combinations of APIs (e.g., `CreateRemoteThread`, `VirtualAllocEx`).
* **Exported Functions (128-dim):** Hashes the names of exported functions, which is useful for classifying malicious DLLs.
* **General File Information (10-dim):** Includes macroscopic characteristics such as file size, virtual size, debug info presence, total imported/exported functions, symbols, and signature presence.
* **Header Information (62-dim):** Extracts details from the COFF and Optional headers, such as machine architecture, subsystem, linker versions, and timestamps.
* **String Information:** Analyzes printable strings found in the binary, tracking metrics like average length, entropy, and counts of interesting patterns (URLs, paths, registry keys, and the "MZ" header).

### 2. Classification (`MalwareDetector.py`)
Once the binary is vectorized, it is passed to a pre-trained **LightGBM** gradient-boosting tree model (`model.txt`). 
* The model evaluates the 2351-dimensional feature vector across thousands of decision trees.
* It outputs a decimal probability score between `0.0` (Clean) and `1.0` (Malware).
* By default, a threshold of `0.85` is used to classify the file as Malicious, heavily reducing False Positives in production environments.

The CLI, powered by `rich`, presents these technical findings in a visually appealing and easy-to-read format, and even supports exporting Threat Intelligence data in **STIX 2.1** format for SIEM integration.

### Model Architecture Pipeline

```mermaid
graph TD
    Data["EMBER Dataset<br>(1.1M+ PE Files)"]
    
    Data -->|Feature Extraction & Hashing| Vec["Vectorized Database<br>(2351-dimensional arrays)"]
    
    Vec --> |"Histogram algorithm<br>Leaf-wise growth strategy"| DT1["Decision Tree 1"]
    Vec -.-> DT2["Decision Tree 2"]
    Vec -.-> DTm["Decision Tree m"]
    
    DT1 -->|"Residuals"| DT2
    DT2 -->|"Residuals"| DTm
    
    DT1 --> Tr1["Train"]
    DT2 --> Tr2["Train"]
    DTm --> Trm["Train"]
    
    Tr1 --> Final["PE Malware LightGBM<br>Prediction Model (model.txt)"]
    Tr2 --> Final
    Trm --> Final
    
    classDef default fill:#ffffff,stroke:#000000,stroke-width:1.5px,color:#000000;
```

---

## Getting Started: Training and Evaluation

While a pre-trained `model.txt` may be provided, you can train a state-of-the-art model from scratch using the official EMBER dataset and the scripts included in this repository.

Follow these steps carefully:

### Step 1: Download the EMBER Dataset
The EMBER dataset contains pre-extracted JSONL files of features from over 1.1 million Windows PE files.

1. Download the EMBER 2017 v1 dataset archive (approx. 1.6 GB compressed):
   [Download EMBER 2017 v1](https://ember.elastic.co/ember_dataset_2017_2.tar.bz2)
2. Extract the archive using a tool like 7-Zip (Windows) or `tar` (Linux).
   * **Note:** The extracted folder will be around 5-6 GB and contain multiple `train_features_X.jsonl` files and `test_features.jsonl`.
   * Keep track of the extraction path (e.g., `C:\ember_dataset\ember`).

### Step 2: Install EMBER Utilities & Vectorize Data
To convert the raw JSONL feature files into memory-mapped NumPy arrays, you must install the official EMBER Python library.

1. Clone the EMBER repository:
   ```powershell
   git clone https://github.com/elastic/ember.git
   cd ember
   ```
2. Install the library:
   ```powershell
   pip install -r requirements.txt
   python setup.py install
   ```
3. Open a Python shell and vectorize the JSONL files:
   ```python
   import ember
   ember.create_vectorized_features("C:\\path\\to\\extracted\\ember")
   ```
   *This process will read all JSONL files, extract the 2351 features, and create `.dat` memory-mapped files (e.g., `X_train.dat`, `y_train.dat`). This requires substantial disk space (around 15-20 GB) and RAM.*

### Step 3: Train the Model (`Train.py`)
Now that the data is vectorized, you can train the LightGBM model.

1. Open `Train.py` in this repository and update the `data_dir` variable to point to your vectorized EMBER directory:
   ```python
   data_dir = r"C:\path\to\extracted\ember"
   ```
2. Run the training script:
   ```powershell
   python Train.py
   ```
   * The script will load the memory-mapped arrays for both training and testing.
   * It uses optimized hyper-parameters (such as tracking `auc`, penalizing missed malware with `scale_pos_weight`, and utilizing all CPU cores with `n_jobs=-1`).
   * It employs early stopping to monitor the test set accuracy and prevent overfitting.
   * Upon completion, it will save the final weights to `model.txt` in your dataset directory.

### Step 4: Evaluate the Model (`Evaluate.py`)
To ensure your trained model is effective against unseen threats, use the evaluation script.

1. Open `Evaluate.py` and ensure the `data_dir` variable correctly points to your EMBER directory:
   ```python
   data_dir = r"C:\path\to\extracted\ember"
   ```
2. Run the evaluation script:
   ```powershell
   python Evaluate.py
   ```
   * The script will load your newly trained `model.txt`.
   * It runs predictions on the EMBER test set (200,000 files).
   * It outputs a detailed report including:
     * **Overall Accuracy**
     * **ROC AUC Score** (The area under the receiver operating characteristic curve)
     * **Detailed Classification Metrics** (Precision, Recall, F1-Score)
     * **Confusion Matrix** (Showing exact counts of True Positives, False Positives, True Negatives, and False Negatives).

### Step 5: Deploy
Once you are satisfied with the evaluation metrics, copy the `model.txt` file from your dataset directory into the root folder of this repository. You can now use `MalwareDetector.py` to scan real-world executables!

```powershell
python MalwareDetector.py path\to\suspicious.exe
```

ML-Executable-Malware-Detector

ML-Executable-Malware-Detector Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

ML-Executable-Malware-Detector

ML-Executable-Malware-Detector Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README