Log Analysis

This project is a Django web application for uploading web server log files, parsing them, and presenting a simple analysis dashboard in the browser.

The current implementation focuses on Apache-style access logs. Users can upload a log file, review upload metadata, open an analysis page for that file, and inspect field validation counts, top request frequencies, and suspicious activity alerts generated from rule-based checks.

What The Application Currently Does

• Upload log files through a Django form
• Store file metadata such as filename, uploader name, file size, upload time, and processing status
• Save uploaded files either to local media storage or to AWS S3 when S3 is enabled
• Parse Apache access log lines into structured fields
• Count valid values and frequency distributions for key fields
• Detect suspicious patterns such as request bursts, repeated auth failures, path scanning, sensitive path probes, suspicious HTTP methods, payload probes, and slow request abuse
• Show results in server-rendered HTML pages
• Delete individual logs or clear all uploaded logs

How It Works

1. A user uploads a log file from the browser.
2. The app stores the file and creates a Log record in the database.
3. Opening the analysis page parses the file synchronously.
4. The parser validates each line and extracts fields such as IP, timestamp, method, path, protocol, status code, referer, user agent, and response time.
5. The analyzer computes field counts and frequency tables.
6. Rule-based checks generate suspicious activity alerts.
7. Summary rows and alerts are stored in the database and rendered in the UI.

Suspicious Activity Rules

The current analyzer flags these patterns:

• High request volume from one IP within a single minute
• Repeated authentication failures from one IP
• Broad path scanning with many 404 responses
• Probing for sensitive paths such as /admin, /.env, /phpmyadmin, or /backup
• Suspicious HTTP methods such as TRACE and CONNECT
• Obvious attack strings in the request path
• Many very slow requests from the same IP

Tech Stack

• Python
• Django
• pandas
• django-environ
• django-storages
• boto3
• SQLite for local development
• AWS S3 for optional uploaded file storage

Project Structure

log-analysis/
|-- backend/
|   |-- manage.py
|   |-- server/              Django settings and URL configuration
|   |-- logs/                Upload, parsing, analysis, models, and views
|   |-- pages/               Basic site pages
|   `-- Templates/           Server-rendered HTML templates
|-- requirements.txt
`-- README.md

Key Django Routes

• / - home page
• /logs/ - uploaded log list
• /logs/upload/ - upload form
• /logs/<log_id>/analysis/ - analysis page for one uploaded log
• /logs/<log_id>/delete/ - delete one uploaded log
• /logs/delete-all/ - delete all uploaded logs

Data Stored By The App

The application stores:

• Uploaded log metadata in the Log model
• Saved analysis summary counts in the LogAnalysis model
• Top IP, status code, method, and path rows in separate models
• Suspicious activity alerts in the Alerts model

Log Format Assumption

The parser is written for Apache-style access logs that include these fields:

• IP
• logname
• user
• timestamp
• request
• status code
• response size
• referer
• user agent
• response time

If a line does not match the expected format, it is still processed, but invalid fields are recorded as empty or null values and the line is marked invalid.

Local Setup

1. Create and activate a virtual environment.
2. Install dependencies:

pip install -r requirements.txt

3. Move into the Django project directory:

cd backend

4. Apply migrations:

python manage.py migrate

5. Start the development server:

python manage.py runserver

6. Open http://127.0.0.1:8000/

Environment Configuration

The settings use django-environ and read environment variables from backend/.env.

Local development can run without S3 by setting:

USE_S3=False
DEBUG=True
ALLOWED_HOSTS=127.0.0.1,localhost

When USE_S3=True, uploaded files are stored in S3 and the AWS variables from backend/env.example are required.