[Patch] Re-work log file layout

[hodgestar]

Currently build log files are written into a single folder with one log file for each step of each build. This can cause the number of files in the folder to quickly exceed sensible limits and makes locating the log files for a particular build awkward.

I propose that the log files be split out into folders of the form log/bitten/32/32100 where the '3100' is the build id associated with the log and the '32' is the build id with the last three characters removed (and '0' for build ids less than four characters). This should allow for log files for about a million builds without exceeding a thousand files per directory.

At the same time it would make sense to combine the log messages and log levels files (currently each log message file has an associated levels file contain the log level; messages and levels are matched up by line number in the respective files).

This feature was originally discussed as part of #462.

[mpotter@…]

I'm currently fighting some performance issues associated with backups of my Trac host. While examining what could be causing this issue I found an issue associated with the trac/log/bitten directory; couldn't even get a directory listing and many other tools would lockup. To resolve I have since ZIP'ed up many of the log files in this directory, leaving a little more than a month's worth. I now have 72,312 files in this directory. Doing some math to compute how many files I had, I get 769,775 files. Log file 737902.log was just written this morning, therefore if I had not been pruning my log file I would currently have about 1,475,804 files in this one directory.

[mpotter@…]

Replying to hodgestar:

I propose that the log files be split out into folders of the form log/bitten/32/32100 where the '3100' is the build id associated with the log and the '32' is the build id with the last three characters removed (and '0' for build ids less than four characters).

I assume you meant '32100'.

[Mark Potter <mpotter@…>]

Possible Patch

[Mark Potter <mpotter@…>]

Looking at the code and database layout I see two possible approaches to this issue.

• Change the meaning of bitten_log filename to instead of being the file name in the directory given by bitten logs_dir, be a relative path from there.
• Change get_log_file method to check for the named file directly off of bitten logs_dir and as a sub-directory directly from there, favoring the new sub-directory style.

The patch above takes the second approach. Sorry at tab slipped through.

[Mark Potter <mpotter@…>]

Replying to Mark Potter <mpotter@…>:

Sorry at tab slipped through.

Just looked at this patch again; sorry a few tabs slipped through.