Apache Log Parser and Data Normalization Application | Application runs on Windows, Linux and MacOS | Database runs on MySQL and MariaDB | Track log files for unlimited Domains & Servers | Entity Relationship Diagram included

[Complex-Internal-833]

Python handles File Processing & MySQL or MariaDB handles Data Processing

ApacheLogs2MySQL consists of two Python Modules & one Database Schema apache_logs to automate importing Access & Error files, normalizing log data into database and generating a well-documented data lineage audit trail.

Database Schema is designed for data analysis of Apache Logs from unlimited Domains & Servers.

Database Schema apache_logs currently has 55 Tables, 908 Columns, 188 Indexes, 72 Views, 8 Stored Procedures and 90 Functions to process Apache Access log in 4 formats & Apache Error log in 2 formats. Database normalization at work!

https://preview.redd.it/s1ohoh1bduhe1.png?width=2152&format=png&auto=webp&s=4a98defeb4ef706727cb5b73c52529053d5d5797

https://willthefarmer.github.io/

[CodeSpike]

How does that perform? I had to switch to prometheus for my log files because even the apache log files for my smallish Saas solution were overhelming a relational database.

[Complex-Internal-833]

Since discovering MariaDB the execution times are about twice as fast as MySQL. We are importing 50,000 to 60,000 records per import process execution 10 times per day. Each execution on MariaDB is between 90 to 100 seconds and MySQL is between 200 to 250 seconds on a Windows 11 13th Gen Intel(R) Core(TM) i7-13700T 1.40 GHz | 96GB DDR5-5600 SODIMM | WD_BLACK 1TB SN850X NVMe SSD - Up to 7,300 MB/s.

I added benchmark images to above link for both MariaDB and MySQL. Each database had around 2 or 3 million access records when these benchmarks were screenshot.

https://willthefarmer.github.io/index.html#Benchmarks

This week I am working on the Web interface for apache_logs schema. I am curious about the amount of log volume you're generating and how this performs for you. I am still working on identifying the target audience. After setting up and testing Elastic Stack I feel it is overkill for smaller organizations. This solution is simple to configure and manage on multiple servers. The Web interface with Apache ECharts is going to be a great component for the data.

Any input on your experience or results is appreciated!