Microsoft Malware Classification

Domain : Cyber-Security
Type of problem: Multi-class Classification
Objective: Predict the probability of a data-point belonging to each of the nine classes of malware.
Performance Metric: Reduce the Multi-class log loss.
Solution:

For every malware we have two files: .asm file and .bytes file
Total dataset consists of 200 GB of data, one of the largest publicly available datasets.
Extracted features from files (Unigrams, Bigrams, Opcode, Pixel Intensity, Size). Used batch processing for text extraction and preprocessing to avoid memory overshoot.
Performed Hypothesis testing method (Chi-Squared) for feature selection to reduce dimensionality.
Designed and Calibrated (wherever needed) KNN, Logistic Regression, Random Forest and XGBoost on different subsets of features.
Achieved a Multi-Class log loss of 0.018.

Utilized Google Cloud Platform (GCP) Compute Engine for memory intensive tasks and Colab for GPU requisite tasks.

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
0.Microsoft_Malware_EDA_and_Feature_Extraction.ipynb		0.Microsoft_Malware_EDA_and_Feature_Extraction.ipynb
1.Microsoft_Malware_Bigrams_Feature_Extraction.ipynb		1.Microsoft_Malware_Bigrams_Feature_Extraction.ipynb
2.Microsoft_Malware_Bigrams_Feature_ Selection.ipynb		2.Microsoft_Malware_Bigrams_Feature_ Selection.ipynb
3.Microsoft_Malware_Pixel_Features_Extraction.ipynb		3.Microsoft_Malware_Pixel_Features_Extraction.ipynb
4.Microsoft_Malware_Classification_Model.ipynb		4.Microsoft_Malware_Classification_Model.ipynb
README.md		README.md

Provide feedback