[PATCH] Store log messages in files rather than database rows

[dfraser]

Storing log messages in the database is very expensive. We should switch to storing them in files

[dfraser]

IRC chat from today:

(04:11:56 PM) ***davidfraser finally gets fed up with log files being stored in the database
(04:12:10 PM) davidfraser: How would it go down to put log files in text files instead?
(04:12:56 PM) cmlenz: +1
(04:13:33 PM) cmlenz: and remove the stderr/stdout differentiation, is unreliable anyway, and a lot of code
(04:14:18 PM) davidfraser: Yay
(04:14:22 PM) ***davidfraser will have a look
(04:18:44 PM) thm: davidfraser: full ack
(04:19:27 PM) thm: especially as old build logs are not easily accesible after changing a build conf

[dfraser]

See also #302, #246

Note that this could enable things like gzipping the log messages...

[dfraser]

I've now actually run the upgrade on two large systems (3.5GB and 1.1GB sqlite database due to all the log messages). With the current code:

• If anything goes wrong in the log file migration, the bitten_log table ends up empty. I could fix this with a db.commit() at the end of the filename column addition...
• The old bitten_log_message table is not dropped. After manually dropping this you need to run vacuum to shrink the database.

The levels are currently stored in a supplementary log file, although nothing is done with them - I thought it safer to preserve them at the moment.

This might need more testing before it gets committed to trunk - not sure whether that would be more effectively done in a branch or by persuading someone, or I should just go ahead and be brave, but I'd hate to lose people's log messages :-)

[thomas.moschny@…]

Does this patch work with the (experimental) 0.11 version of Bitten?

[dfraser]

Replying to thomas.moschny@…:

Does this patch work with the (experimental) 0.11 version of Bitten?

It does now after that update; this has of course become the trunk version now... sorry for slow reply

[dfraser]

Updated patch that applies to new trunk that was taken from 0.11 branch (and fixes silly error)

[ebray]

Thanks for the patch. I haven't really run into this limitation myself yet, but I can imagine that on some projects it will become a problem. I'm also considering adding an extension to compress older log files.

One small problem I ran into is that in upgrades.py there should be something like:

if not os.path.isabs(logs_dir):
    logs_dir = os.path.join(env.path, logs_dir)

Otherwise it breaks unless I'm actually running trac-admin from my Trac environment directory, which I usually don't.

[dfraser]

Replying to ebray:

Thanks for the patch. I haven't really run into this limitation myself yet, but I can imagine that on some projects it will become a problem. I'm also considering adding an extension to compress older log files.

Yes compressing log files should be easy with this.

One small problem I ran into is that in upgrades.py there should be something like:

if not os.path.isabs(logs_dir):
    logs_dir = os.path.join(env.path, logs_dir)

Otherwise it breaks unless I'm actually running trac-admin from my Trac environment directory, which I usually don't.

Great thanks I'll add that to the patch. So I'm assuming otherwise it works for you...

[dfraser]

Enables storing log messages in files rather than database (adjusted to check abs path on upgrade)

[ebray]

Replying to dfraser:

Great thanks I'll add that to the patch. So I'm assuming otherwise it works for you...

Yep, seems to work fine. The only other problem is that I had to manually give my web server user ownership of the log/bitten directory. Not sure how to better deal with that though.

[anonymous]

Just applied this patch to my copy of Bitten. So far so good -- excellent work!

[dfraser]

Replying to anonymous:

Just applied this patch to my copy of Bitten. So far so good -- excellent work!

OK, on the basis of the feedback so far I'm going to commit this - I've just added a message to say that people need to check permissions and look at dropping the table.

This is still svn so anyone using latest svn should be prepared to do a little bit of testing and all the feedback so far indicates this is working fine.

[dfraser]

Applied in r590. Waiting for feedback from mailing list before marking this as fixed...

[dfraser]

Fixes to tests in r591 to make them handle creating logs dirs

[felix.schwarz@…]

Actually there is an error in trunk/bitten/upgrades.py?rev=591#L333​ env.log_warning does not exist, this should be env.log.warning

[dfraser]

Replying to felix.schwarz@…:

Actually there is an error in trunk/bitten/upgrades.py?rev=591#L333​ env.log_warning does not exist, this should be env.log.warning

Woops, thanks! Fixed in r592

[dfraser]

This looks like it's working to everyone's satisfaction now.

The remaining step is to make the upgrade automatically drop the table

[dfraser]

Added the final step of dropping the bitten_log_message table in r693. Note that this will not drop tables of people who have already upgraded - but hey, they're the cutting-edge few who're running on trunk, so they should have taken care of it themselves.