Project Details

The Log Server is a bit like an SQL database. It runs as a daemon, and you connect to it via a TCP socket to give it data and then run queries.

However, it's optimised for a very special task: keeping logs of things.

Log entries are structured records with fields, and are indexed on configured fields. They're not organised into fixed tables; any log entry with an 'IP' field, if 'IP' is listed as an indexed field, will be indexed under that IP, regardless of the type of log record.

Rather than reinventing SQL, we use embedded Python as a query language, either directly or (for efficiency and security) by having configured 'stored procedures' and login accounts restricted to only be able to run them rather than submitting arbitrary queries.

Also, the on-disk format is based on simple delimited text, meaning the raw data can be easily processed by existing tools, if needs be.

See log-server-notes.txt in the tarball for technical details. When implementation is complete (we're up to Stage 3 so far), a proper manual will be written.