Sangamesh.Dandin
Résumé
All projects

Case 03 / 09Malware MLCompleted

BYTEHUNTER.

Malware triage for Windows executables, with one model deciding the verdict.

A FastAPI and React malware-triage app. Uploaded files are hashed and cached by SHA-256, parsed with LIEF into a 2,381-value feature vector and scored by the pre-trained EMBER 2018 LightGBM model. One threshold scale turns that score into SAFE, SUSPICIOUS or MALICIOUS, with a heuristic scorer as the fallback when the model isn't available. Random Forest and Logistic Regression add an agreement check, an XGBoost model trained on Microsoft BIG-2015 suggests a malware family for high scores, and SHAP-based explanations are summarised as text. Results are stored in SQLite and exported as PDF reports.

What I built

  • Built the scan API: batch upload, SHA-256 caching, LIEF feature extraction into a 2,381-value vector, and verdict thresholds on the LightGBM score.
  • Trained an XGBoost family classifier on Microsoft BIG-2015 and Random Forest and Logistic Regression comparison models, and added SHAP-based explanations.
  • Added a heuristic fallback scorer, SQLite scan history and ReportLab PDF reports.
Problem
Malware triage
Built with
LightGBM, LIEF, FastAPI
Result
Safe / suspicious / malicious
  • Python
  • LightGBM
  • LIEF
  • XGBoost
  • SHAP
  • FastAPI
  • React
  • Docker

▶Demo

See it work.

An animated walkthrough of the interface, not a screen recording. The file, score and findings shown are example values, not a real scan.

01Architecture

How it's built.

Drawn from the project's source code, not its README: what each part actually does and how a request moves through it.

ByteHunter verdict anatomy An uploaded file is checked against a SHA-256 cache, parsed with LIEF into a 2,381-value vector and scored by LightGBM, or by a heuristic scorer when no LightGBM model loads; that single score maps to SAFE, SUSPICIOUS or MALICIOUS, while SHAP, Random Forest and Logistic Regression comparison, an XGBoost family classifier and an import-name lookup only annotate the result, which is stored in SQLite for history and PDF reports. ANNOTATES THE RESULT · NEVER CHANGES THE VERDICT DECIDES THE VERDICT UPLOAD MISS IMPORT NAMES SAME VECTOR VECTOR EXPLAINS SCORE IF SCORE > 50 NO MODEL HEURISTIC SAVE RESULT SHA-256 HIT POST /api/analyze ≤ 5 files, ≤ 50 MB any file type Seen SHA-256? LIEF features L1 lief.PE.parse(bytes) 2,005 values + zero pad = 2,381-dim vector non-PE: entropy + histogram PRIMARY SCORE L2 LightGBM ember_model_2018 (pre-trained) p(malicious) × 100 hot-swap: models/pending, 24 h loaded at startup if present Verdict + risk level same thresholds for model or heuristic score SAFE < 30 · LOW SUSPICIOUS 30–65 · MEDIUM MALICIOUS > 65 · HIGH API lookup L4 10 named imports → tags CreateRemoteThread → injection IsDebuggerPresent → anti-debug static references, not runtime RF + Logistic Reg. predict_proba × 100 agreement % across scores warning if σ of scores > 20 not used in the verdict SHAP L3 TreeExplainer on LightGBM top 8 |SHAP| features → grouped explanation text no explainer: entropy proxy XGBoost family BIG-2015 · 9 families 258 raw-byte features, not the 2,381 vector → Worm, Adware, Backdoor… HEURISTIC · NOT ML Heuristic scorer if LightGBM missing or fails ext/magic mismatch +20 entropy > 7.2 +25 · strings embedded MZ +20 · cap 100 STORE SQLite scans UNIQUE sha256 · result_json GET /api/history[/{id}] GET /api/report/{id} → PDF ReportLab, from stored JSON Only one number sets the verdict: LightGBM’s score, or the heuristic when LightGBM can’t run. The repository ships no LightGBM, XGBoost or SHAP files, so a fresh checkout scores every file with the heuristic scorer. VERIFIED LIMITS · FOUND IN CODE, NOT STATED IN THE README L1 The vector is a custom layout padded to 2,381, not EMBER’s published feature order, and imports are hashed with Python’s per-process hash(). Not validated. L2 Without ember_model_2018 the heuristic score is used, but the response still reports model_used = “LightGBM — EMBER 2018”. L3 The API response drops shap_features and warning_signs, so the dashboard’s SHAP chart does not render; SHAP reaches users only as grouped text. L4 Exact name matching: RegSetValueEx, URLDownloadToFile and InternetOpenUrl lack the A/W suffix real imports carry, so those three never match. LEGEND Decision Pipeline step Primary score Annotator Heuristic, not ML Store Verdict score Data flow Conditional HTTP L1 Verified limit

Read from the code at commit 3e86d53 · Open full size ↗

02Under the hood

How it works.

  1. 01

    Cache

    A SHA-256 lookup returns earlier results instantly

  2. 02

    Features

    LIEF parses the file into a 2,381-value vector

  3. 03

    Score

    Pre-trained LightGBM sets the verdict, with a heuristic fallback

  4. 04

    Annotate

    Agreement models, the XGBoost family and SHAP add context

  5. 05

    Store

    SQLite history and PDF reports

→Up next · 04 / 09

Next project.

SpearSim — Privacy-minded phishing simulations with follow-up training.

Back to all projects