> For the complete documentation index, see [llms.txt](https://aashraymt.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://aashraymt.gitbook.io/docs/sms-spam-detector/network-anomaly-detector.md). # network-anomaly-detector ### WHY I think understanding defensive mechanisms is a core component of mastering offensive security. As modern network perimeters increasingly rely on Machine Learning and Agentic systems to flag intrusions, developing sophisticated exploits or discovering novel vulnerabilities requires knowing exactly how these defensive models "think." By building a Random Forest anomaly detector from scratch, we gain a white-box perspective on how normal traffic baselines are established and, more importantly, where the blind spots in confidence scoring lie. This project isn't just about learning to catch anomalies; it is about understanding the mathematical threshold of detection. This is the Link to the GitHub for this project, in case you want to check it out. {% embed url="" %} ### Introduction & Theoretical Concepts #### **Network Anomaly Detection** Anomaly detection is the process of identifying data points, events, or observations that deviate significantly from a dataset's normal behavior. In the realm of cybersecurity, these anomalies often serve as the primary indicators of malicious activities, network intrusions, or zero-day security breaches. Because network environments generate complex, high-dimensional data, we leverage **Random Forests** an ensemble of decision trees to efficiently process and detect these anomalous patterns. #### **Deep Dive: Random Forests** A Random Forest is a powerful ensemble machine-learning algorithm. Rather than relying on a single decision tree, it builds a "forest" of multiple trees and aggregates their predictions to produce a final, highly accurate result. By combining multiple outputs, the model generalizes better, resists overfitting, and maintains robust performance across high-dimensional feature spaces. Three core concepts govern the construction of a Random Forest: * **Bootstrapping:** The algorithm creates multiple subsets of the training data through random sampling *with replacement*. Each of these subsets is used to train an individual decision tree. * **Tree Construction:** To ensure diversity and reduce correlation among the individual trees, the algorithm evaluates only a random subset of features at every node split. * **Aggregation & Voting:** Once all trees are trained, the forest aggregates their outputs. * For **Classification**, it uses majority voting: $$ \hat{y} = \text{mode}{h\_1(x), h\_2(x), \dots, h\_B(x)} $$ * For **Regression**, it averages the predictions: $$ \hat{y} = \frac{1}{B} \sum\_{b=1}^{B} h\_b(x) $$ *(Where $B$ is the total number of trees, and $h\_b(x)$ is the prediction of the $b$-th tree).* #### **The Rationale: Anomaly Detection via Normalcy** When deployed for anomaly detection, a Random Forest is trained **exclusively on data representing normal network conditions**. Why train only on normal data? 1. **Zero-Day Detection:** By establishing a baseline of "normal," the model can flag unseen, malicious patterns without ever having encountered that specific attack signature in its training data. 2. **Confidence Scoring:** New data points are evaluated against the learned normal behavior. Traffic that does not fit the established pathways of the decision trees, or that yields high prediction variance/low confidence scores, is immediately flagged as a potential intrusion. ### Model Architecture & Implementation Details #### **The NSL-KDD Dataset** To train and validate our model, we utilize a modified version of the **NSL-KDD dataset**. This dataset refines the original KDD Cup 1999 dataset by eliminating redundant records and balancing class distributions. It is a standard benchmark in cybersecurity research because it provides labeled instances of normal traffic and various specific attack types, enabling both binary classification and multi-class detection tasks. #### **Implementation Pipeline: Data Ingestion** **1. Downloading the Dataset** Before loading the data into our pipeline, we retrieve the `.zip` archive from its remote source and extract it locally. ``` import requests, zipfile, io # URL for the NSL-KDD dataset url = "https://academy.hackthebox.com/storage/modules/292/KDD_dataset.zip" # Download the zip file and extract its contents response = requests.get(url) z = zipfile.ZipFile(io.BytesIO(response.content)) z.extractall('.') # Extracts to the current directory ```
**2. Environment Setup & Library Imports** Properly loading and structuring the data requires specific Python libraries for data manipulation, modeling, and visualization. ``` import numpy as np import pandas as pd from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, classification_report import seaborn as sns import matplotlib.pyplot as plt ``` > **Note on Dependencies:** > \* `numpy` / `pandas`: Data loading and DataFrame manipulation. > > * `sklearn`: Model training (`RandomForestClassifier`) and evaluation metrics. > * `seaborn` / `matplotlib`: Visualizing network feature distributions and model results. **3. Feature Mapping & DataFrame Initialization** The raw NSL-KDD dataset lacks headers. We must explicitly map the features to meaningful column names to ensure the data is properly structured for the training pipeline. Python ``` # Set the file path to the dataset file_path = r'KDD+.txt' # Define the 43 column names corresponding to the NSL-KDD dataset columns = [ 'duration', 'protocol_type', 'service', 'flag', 'src_bytes', 'dst_bytes', 'land', 'wrong_fragment', 'urgent', 'hot', 'num_failed_logins', 'logged_in', 'num_compromised', 'root_shell', 'su_attempted', 'num_root', 'num_file_creations', 'num_shells', 'num_access_files', 'num_outbound_cmds', 'is_host_login', 'is_guest_login', 'count', 'srv_count', 'serror_rate', 'srv_serror_rate', 'rerror_rate', 'srv_rerror_rate', 'same_srv_rate', 'diff_srv_rate', 'srv_diff_host_rate', 'dst_host_count', 'dst_host_srv_count', 'dst_host_same_srv_rate', 'dst_host_diff_srv_rate', 'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate', 'dst_host_serror_rate', 'dst_host_srv_serror_rate', 'dst_host_rerror_rate', 'dst_host_srv_rerror_rate', 'attack', 'level' ] # Read the combined NSL-KDD dataset into a DataFrame df = pd.read_csv(file_path, names=columns) print(df.head()) ```
#### **Network Feature Overview** Below is a categorized snapshot of the network features extracted for the model: | **Feature Category** | **Examples** | **Description** | | -------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | Basic Features | `duration`, `protocol_type`, `src_bytes`, `dst_bytes` | Fundamental attributes of the individual TCP connections. | | Content Features | `hot`, `num_failed_logins`, `logged_in`, `num_compromised` | Attributes extracted from the data payload indicating potential suspicious behavior (e.g., multiple failed root logins). | | Traffic Features | `count`, `srv_count`, `serror_rate`, `same_srv_rate` | Computed statistics over a time window regarding connections to the same host or port. | | Labels | `attack`, `level` | The target variables classifying the traffic as normal or specifying the attack type. | ### Model Training & Validation #### **Training the Model** With the NSL-KDD data preprocessed into multi-class sets, the next step is to train our Random Forest model. We configure the model specifically for multi-class classification, aiming to categorize network traffic into distinct attack types or normal behavior. Python ``` # Train RandomForest model for multi-class classification rf_model_multi = RandomForestClassifier(random_state=1337) rf_model_multi.fit(multi_train_X, multi_train_y) ``` > **Note on Reproducibility:** > We initialize the `RandomForestClassifier` with the `random_state` parameter set to **1337**. This ensures that the randomized processes (like bootstrapping and feature selection) yield the exact same results across different runs. The `.fit()` method then builds the forest by learning patterns from the training features (`multi_train_X`) and target variables (`multi_train_y`). #### **Evaluating on the Validation Set** To ensure the model generalizes well and isn't simply memorizing the training data, we assess its performance using a dedicated validation set. Python ``` # Predict and evaluate the model on the validation set multi_predictions = rf_model_multi.predict(multi_val_X) accuracy = accuracy_score(multi_val_y, multi_predictions) precision = precision_score(multi_val_y, multi_predictions, average='weighted') recall = recall_score(multi_val_y, multi_predictions, average='weighted') f1 = f1_score(multi_val_y, multi_predictions, average='weighted') print(f"Validation Set Evaluation:") print(f"Accuracy: {accuracy:.4f}") print(f"Precision: {precision:.4f}") print(f"Recall: {recall:.4f}") print(f"F1-Score: {f1:.4f}") # Confusion Matrix for Validation Set conf_matrix = confusion_matrix(multi_val_y, multi_predictions) class_labels = ['Normal', 'DoS', 'Probe', 'Privilege', 'Access'] sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues', xticklabels=class_labels, yticklabels=class_labels) plt.title('Network Anomaly Detection - Validation Set') plt.xlabel('Predicted') plt.ylabel('Actual') plt.show() # Classification Report for Validation Set print("Classification Report for Validation Set:") print(classification_report(multi_val_y, multi_predictions, target_names=class_labels)) ```
**Understanding the Metrics** We rely on four primary statistical metrics from `sklearn.metrics` to gauge success: * **Accuracy:** The overall proportion of correctly classified network connections. * **Precision:** The ratio of true positive predictions to the total predicted positives (minimizing false alarms). * **Recall:** The ratio of true positive predictions to the total actual positives (ensuring we don't miss real attacks). * **F1-Score:** The harmonic mean of Precision and Recall, providing a balanced metric for uneven class distributions. The **Confusion Matrix** provides a granular visual breakdown via Seaborn, mapping out exactly where the model succeeds and where it confuses specific classes (e.g., mistaking a *Probe* for *Normal* traffic). #### **Final Testing on Unseen Data** Once validated, we run a final evaluation on the test set. This represents entirely unseen data and serves as the ultimate benchmark for how the model will perform in a real-world network environment. ``` # Final evaluation on the test set test_multi_predictions = rf_model_multi.predict(test_X) test_accuracy = accuracy_score(test_y, test_multi_predictions) test_precision = precision_score(test_y, test_multi_predictions, average='weighted') test_recall = recall_score(test_y, test_multi_predictions, average='weighted') test_f1 = f1_score(test_y, test_multi_predictions, average='weighted') print("\nTest Set Evaluation:") print(f"Accuracy: {test_accuracy:.4f}") print(f"Precision: {test_precision:.4f}") print(f"Recall: {test_recall:.4f}") print(f"F1-Score: {test_f1:.4f}") # Confusion Matrix for Test Set test_conf_matrix = confusion_matrix(test_y, test_multi_predictions) sns.heatmap(test_conf_matrix, annot=True, fmt='d', cmap='Blues', xticklabels=class_labels, yticklabels=class_labels) plt.title('Network Anomaly Detection') plt.xlabel('Predicted') plt.ylabel('Actual') plt.show() # Classification Report for Test Set print("Classification Report for Test Set:") print(classification_report(test_y, test_multi_predictions, target_names=class_labels)) ``` ### Model Export & External Evaluation #### **Saving the Model** To deploy the trained model or submit it for grading, it must be serialized and saved to disk. We use the `joblib` library to export the model weights and architecture into a `.joblib` file. Python ``` import joblib # Save the trained model to a file model_filename = 'network_anomaly_detection_model.joblib' joblib.dump(rf_model_multi, model_filename) print(f"Model saved to {model_filename}") ``` #### **This is the output ----->** ``` duration protocol_type service flag src_bytes dst_bytes land \ 0 0 tcp ftp_data SF 491 0 0 1 0 udp other SF 146 0 0 2 0 tcp private S0 0 0 0 3 0 tcp http SF 232 8153 0 4 0 tcp http SF 199 420 0 wrong_fragment urgent hot ... dst_host_same_srv_rate \ 0 0 0 0 ... 0.17 1 0 0 0 ... 0.00 2 0 0 0 ... 0.10 3 0 0 0 ... 1.00 4 0 0 0 ... 1.00 dst_host_diff_srv_rate dst_host_same_src_port_rate \ 0 0.03 0.17 1 0.60 0.88 2 0.05 0.00 3 0.00 0.03 4 0.00 0.00 dst_host_srv_diff_host_rate dst_host_serror_rate \ 0 0.00 0.00 1 0.00 0.00 2 0.00 1.00 3 0.04 0.03 4 0.00 0.00 dst_host_srv_serror_rate dst_host_rerror_rate dst_host_srv_rerror_rate \ 0 0.00 0.05 0.00 1 0.00 0.00 0.00 2 1.00 0.00 0.00 3 0.01 0.00 0.01 4 0.00 0.00 0.00 attack level 0 normal 20 1 normal 15 2 neptune 19 3 normal 21 4 normal 21 [5 rows x 43 columns] Validation Set Evaluation: Accuracy: 0.9950 Precision: 0.9949 Recall: 0.9950 F1-Score: 0.9949 Classification Report for Validation Set: precision recall f1-score support Normal 0.99 1.00 1.00 18519 DoS 1.00 1.00 1.00 12784 Probe 0.99 0.99 0.99 3409 Privilege 0.82 0.38 0.51 24 Access 0.97 0.92 0.95 908 accuracy 1.00 35644 macro avg 0.96 0.86 0.89 35644 weighted avg 0.99 1.00 0.99 35644 Test Set Evaluation: Accuracy: 0.9949 Precision: 0.9947 Recall: 0.9949 F1-Score: 0.9947 Classification Report for Test Set: precision recall f1-score support Normal 0.99 1.00 1.00 15402 DoS 1.00 1.00 1.00 10721 Probe 0.99 1.00 1.00 2796 Privilege 0.62 0.24 0.34 21 Access 0.96 0.92 0.94 764 accuracy 0.99 29704 macro avg 0.91 0.83 0.85 29704 weighted avg 0.99 0.99 0.99 29704 Model saved to network_anomaly_detection_model.joblib ```
#### **Automated Model Evaluation** To verify your model's real-world efficacy, it must be uploaded to the evaluation portal running on the Playground VM. **Method 1: API Upload via Jupyter (Playground VM)** If you are actively running the Playground VM, you can automate the submission directly from your Jupyter Notebook using the `requests` library. Python ``` import requests import json # Define the URL of the API endpoint url = "http://localhost:8001/api/upload" # Path to the model file you want to upload model_file_path = "network_anomaly_detection_model.joblib" # Open the file in binary mode and send the POST request with open(model_file_path, "rb") as model_file: files = {"model": model_file} response = requests.post(url, files=files) # Pretty print the response from the server print(json.dumps(response.json(), indent=4)) ``` ### Post-Evaluation: Proof of Mastery (The Flag) In a practical, competitive environment, validating a machine learning model goes beyond local Jupyter notebook testing. The ultimate verification of your architecture's success is governed by the remote evaluation portal. When you submit your serialized `.joblib` model to the endpoint, the server evaluates its predictive capabilities against a hidden, rigorous holdout dataset. It tests whether your Random Forest can genuinely distinguish between benign traffic and sophisticated network intrusions without overfitting. If your model achieves the required performance threshold—such as a perfect **1.0** accuracy rating on the server side—the portal will authenticate your success by returning a cryptographic **Capture The Flag (CTF) token**.