KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

Malware-Detection-using-N-Gram-Frequency

★ 33 Open GitHub ↗

An attempt to detect malware using Opcodes and Hexadecimal Instructions.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Malware-Detection-using-N-Gram-Frequency

## Introduction:
Malware(Malicious Software) refers to any software intentionally designed to cause damage to a computer, server, client, or computer networks. The common Malware types, include(but not restricted to) Virus, Trojans, Spyware, Worms and Ransomware. A Malware can compromise computers/smart devices, steal confidential information, penetrate networks, and cripple critical infrastructure etc.
Realising the potential damage Malwares can cause, people resort to "Anti-Virus" software for protection.
A typical Anti-Virus(could be called as "Anti-Malware" also, although there are certain differences) software detects Malwares through the following ways:

**Signature-Based Detection**:

Signatures are bytes of data or hashes of an object, that would uniquely identify the object. In computing, all objects have attributes that can be used to create a unique signature. When an anti-malware solution provider identifies an object as malicious, its signature is added to a database of known malware. These repositories may contain hundreds of millions of signatures that identify malicious objects. The IDS(Intrusion detection system) analyzes the packets travelling across the network and checks if it matches the signature of known malware. If the match(es) is found, necessary actions are taken.

**Behaviour-Based Detection**:

Behaviour-based malware detection evaluates an object based on its intended actions before it can execute that behaviour. Attempts to perform actions that are clearly abnormal or unauthorized would indicate the object is malicious, or at least suspicious. Some examples of suspicious behaviour/actions include an attempt to discover a sandbox environment, disabling security controls, installing rootkits, and registering for autostart.

Signature-Based Detection has its own set of demerits. For it to identify malware, it's signature should have already been present in the database. Attackers are using Novel Malwares, which Signature-Based Detection systems fail to recognize. Thus this method does not protect us against Zero-Day Attacks. Although various organizations are setting up "Honeypots" across the globe to identify and study new Malwares, the volume of new Malwares emerging every day is just huge.

## The idea:

* It has been mentioned earlier that Signature Based Detections is not effective against Zero-Day attacks. In this project, we'll analyse Opcodes and Instructions of Malware and Benign(Non-Malware/ Unharmful) files and use Machine Learning to predict potential Malware.
* All Computer Programs/Applications are converted into Assembly Language before execution and that includes Malware too.
* Upon manually analysing a few Malware and Benign files, some distinctive features are observed, which incude:
  1. It could be observed that on an average malware files tends to have more number of instructions/opcodes that involves packed values/data(example: andps,divps,pcmpeqd etc.).
  2. Many Malware files tend to contain more number of cryptographic hash functions like SHA256.
* Therefore, analysing the opcodes of malware and benign files could help in identifying potential malware.
* Thus an attempt has been made to identify potential malware by using a Machine Learning model that feeds on Opcode and Hexadecimal Instructions.

## Dataset Creation:

### Data Collection:
* The intent is to train a Machine Learning Model to classify Benign and Malware files.
* All the Malware and Benign files used in this project are Windows Executable Files/Applications.
* The Malware samples/files were downloaded from the following websites:
  1. https://virusshare.com/
  2. https://dasmalwerk.eu/
  3. http://www.tekdefense.com/downloads/malware-samples/
 * A total of 600 Malware files were downloaded from these sites. The Malware samples include a variety of Viruses, Trojans, Worms, Ransomware etc., predominantly from 2014-2016 timespan. 
 * An Ubuntu Virtual Machine was set up and the Malwares were downloaded in the Virtual Machine. 
 * A total of 504 Windows Applications/Software were taken from my computer to be used as Benign files for the project.
 
 ### Creation of Datasets:
* The downloaded Malware files and the Benign files were disassembled in the Ubuntu Virtual Machine using the shell instruction “objdump –d {file name}” (This command will extract the Opcode Sequence of the executable section of the file). The output is then stored in a text file.
* These text files are converted into .csv files to facilitate easier data manipulation and processing through python pandas library.
* Samples of the Malware and Benign .txt and .csv files are uploaded in the Github page as well.
 
 ### Dataset Size Reduction:
  The Malware samples were downloaded in bulk from various sites. The Malware downloaded from one site were downloaded once again from another site. This caused a huge number of duplicate files in the dataset. To solve the problem, the hashes of the Malware and Benign files were generated, and unique Hashes were inserted into the Dataset(Explained in greater detail in the code).
  In the end, there were 490 Benign Files and 459 Malware Files present in the Dataset. 
  
## Methods of Detection:
The following ways were used to feed the data into the Machine Learning Algorithms:
 1. Detection using Opcode Count with N-Grams=1.
 2. Detection using Opcode Count with N-Grams=2.
 3. Detection using Opcode Frequency with N-Grams=1.
 4. Detection using Opcode Frequency with N-Grams=2.
 5. Detection using Hexadecimal Instruction Count with N-Grams=1.
 6. Detection using Hexadecimal Instruction Count with N-Grams=2.
 7. Detection using Hexadecimal Instruction Frequency with N-Grams=1.
 8. Detection using Hexadecimal Instruction Frequency with N-Grams=2.
 
 **Detection using Opcode Count:**
 
  In this method, the Opcodes(eg: JMP, ADD, etc.) are used as the Feature Vector. Each Unique Opcode would occupy a column in the dataset and it would contain the number of times that particular Opcode occurs in a particular file(Opcode Count) as its value.
 
 **Detection using Opocde Frequency:**
 
 Similar to "Opcode Count Method", but, instead of the Opcode Count, Opcode Frequency is used.
 
 Opcode Frequency of a particular Opcode in a file = 
 ( ( Number of time a particular Opcode occurs in the file) / (Total number of Opcode Occurrence in the file ) ) * 100
  
  **Detection using Hexadecimal Instructions:**
  
  In this method, instead of the Opcodes, Hexadecimal Instructions are used. By using the Hexadecimal Instructions, we would be able to incorporate the information about Opcode as well as the Operands into the Machine Learning Algorithms. For example, The Hexadecimal equivalent of "INC %eax" is 40 and the Hexadecimal equivalent of "INC %ebx" is 43.
 The Hexadecimal Instruction changes even though the Opcode is same for both the Instructions(Operand changes).
  Also, Some opcodes are represented by more instructions with the same meaning, using different syntaxes. For example, JE/JNZ have the same hexadecimal opcode of 74. Thus using Hexadecimal Instructions would reduce redundancy.

**N-Grams:**
 
  An N-Gram is a contiguous sequence of n items from a given sample of text.
  For example: For the sentence "I love petting dogs", the 1-Gram Sequence is {'I','love','petting','dogs'} and the 2-Gram Sequence is {'I love','love petting','petting dogs'}.
  The Opcode Sequence of each file is split into tokens based on the N-Grams mentioned.
  
 # Code:
 The Code is uploaded and a detailed explanation is given as comments.
 
# Prediction:
The data is fed to different Machine Learning Classifier Algorithms. The Accuracy obtained through different methods using different algorithms is mentioned below.

Accuracy= ((TP+TN)/(TP+FP+FN+TN))*100

TP: True Positive i.e The number of elements the model correctly classifies as malware which are originally a malware.

FP: False Positive i.e The number of elements the model falsely classifies as malware which are originally benign.

FN: False Negative i.e The number of elements the model falsely classifies as benign which are originally a malware.

TN: True Negative i.e The number of elements the model correctly classifies as benign which are originally benign.

Detection using Opcode Count with N-Grams=1:
 1. Naive Bayes: 64.70%
 2. SVC('Linear' Kernel): 82.77%
 3. Random Forest: 87.81%
 4. SVC('RBF' Kernel): 64.28%
 5. Decision Trees: 89.91%
 6. KNN: 78.15%

Detection using Opcode Count with N-Grams=2:
 1. Naive Bayes: 72.68%
 2. SVC('Linear' Kernel): 86.13%
 3. Random Forest: 90.33%
 4. SVC('RBF' Kernel): 81.93%
 5. Decision Trees: 83.61%
 6. KNN: 83.19%

Detection using Opcode Frequency with N-Grams=1:
 1. Naive Bayes: 66.38%
 2. SVC('Linear' Kernel): 86.13%
 3. Random Forest: 89.49%
 4. SVC('RBF' Kernel): 68.48%
 5. Decision Trees: 86.13%
 6. KNN: 80.25%
 
Detection using Opcode Frequency with N-Grams=2:
 1. Naive Bayes: 81.93%
 2. SVC('Linear' Kernel): 89.49%
 3. Random Forest: 92.01%
 4. SVC('RBF' Kernel): 84.87%
 5. Decision Trees: 83.19%
 6. KNN: 83.19%
 
Detection using Hexadecimal Instruction Count with N-Grams=1:
 1. Naive Bayes: 58.15%
 2. SVC('Linear' Kernel): 79.91%
 3. Random Forest: 85.77%
 4. SVC('RBF' Kernel): 71.54%
 5. Decision Trees: 77.82%
 6. KNN: 79.91%
 
Detection using Hexadecimal Instruction Count with N-Grams=2:
 1. Naive Bayes: 71.54%
 2. SVC('Linear' Kernel): 84.93%
 3. Random Forest: 88.70%
 4. SVC('RBF' Kernel): 80.75%
 5. Decision Trees: 81.17%
 6. KNN: 83.26%

Detection using Hexadecimal Instruction Frequency with N-Grams=1:
 1. Naive Bayes: 66.94%
 2. SVC('Linear' Kernel): 88.28%
 3. Random Forest: 89.95%
 4. SVC('RBF' Kernel): 58.99%
 5. Decision Trees: 85.77%
 6. KNN: 79.91%

Detection using Hexadecimal Instruction Frequency with N-Grams=2:
 1. Naive Bayes: 70.29%
 2. SVC('Linear' Kernel): 90.37%
 3. Random Forest: 89.95%
 4. SVC('RBF' Kernel): 83.68%
 5. Decision Trees: 76.65%
 6. KNN: 83.68%

## Result:

The maximum accuracy obtained was 92.01%. Detection using Opcode Frequency with N-Grams=2 has higher accuracy on an average. Thus potential Malwares could be identified with reasonable accuracy analysing the Opcodes and feeding theme to Machine Learning Algorithms.

## Future Work:
 1. The size of the Dataset can be increased.
 2. The N-Gram range can be increased.

Malware-Detection-using-N-Gram-Frequency

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README