Log File Analyzer with Visual Dashboard

Every web server writes logs, yet most small teams read them only during outages — grepping millions of lines under pressure to answer when did checkout start failing. Enterprise log platforms are expensive and need infrastructure; the alternative is manual scanning. This project builds a focused, visual analyzer that closes the gap: format-aware parsing turns raw lines into structured records (timestamp, IP, method, path, status, latency), and a single-pass aggregation engine computes the dashboards — per-hour error series, status-code distribution, endpoint latency percentiles, IP behavior profiles — with heuristics that flag anomalies like error-rate jumps, scan-like IP patterns and the single most frequent exception collapsed into one actionable card.

1. The user uploads a log file or pastes log text; the analyzer auto-detects the format.
2. A streaming parser extracts fields line by line; malformed lines are quarantined and counted, not silently dropped.
3. The aggregation engine computes time-bucketed counts, status histograms, endpoint percentiles and IP profiles in one pass.
4. Heuristics compare error rates against a rolling baseline and flag jumps, scan patterns and slow endpoints.
5. Application exceptions are fingerprinted by normalized message plus top frame, collapsing repeats into cards.
6. Dashboards render charts and tables; clicking a spike or card drills down to the raw lines behind it.

Tech stack:

• Python (Flask/FastAPI) parser and aggregation engine
• HTML5, CSS3, JavaScript with canvas charts
• SQLite/PostgreSQL for parsed records
• Streaming parse for large files
• Sample log generators for demo and testing

• [Format-aware parsing] Regex grammars for Nginx/Apache combined logs plus common app formats, with auto-detection.
• [Structured record store] Every line becomes timestamp, IP, method, path, status and latency — indexed for queries.
• [Error-trend dashboards] Per-hour request/error series with baseline comparison and spike highlighting.
• [Latency analysis] Per-endpoint average and p95 tables that surface the slowest routes instantly.
• [IP behavior flags] Request-rate and 404-ratio profiling to spot scanners and aggressive crawlers.
• [Exception grouping] Recurring stack traces fingerprinted into single cards with first/last-seen counts.
• [Drill-down] Click any aggregate to see the underlying raw log lines.

• Complete analyzer web application (upload, parse, dashboards)
• Streaming parser with format grammars
• Single-pass aggregation engine
• Sample log generator for demonstrations
• Project report PDF (parsing design, aggregation algorithms)
• PPT presentation for final review
• Viva Q&A preparation document (regex parsing, percentiles, anomaly heuristics)

• The demo analyzes uploaded files; live tail/streaming mode is future scope.
• Custom log formats need a user-supplied regex; only common formats are built in.
• Anomaly thresholds are heuristic defaults, tunable per deployment, not ML-trained detectors.
• Very large files (multi-GB) need chunked upload handling beyond the demo's scope.
• Alerting (error-rate threshold to webhook) is future scope; the build is analysis-only.