[patch] Poor performance when there are a lot of configurations

[J. Berger <jeberger@…>]

When there are more than a few configurations and/or platforms (we have a total of 12 builds for each revision), the server becomes very slow. This is at least in part due to the fact that the same db queries are repeated several times for each http access.

The attached patch caches the db queries in memory for the duration of each http request, which provides significant speedups.

[r.wilczek@…]

We have 24 slaves here, serving 411 configs with 880 platforms.

I just applied your patch and will report the effects.

[r.wilczek@…]

I am sorry, but I had to revert your patch.

When the slaves started their requests, the server's load exploded and did not recover. The server was not able to handle the requests any more (in fact, the system was no longer responsive at all).

And now I have to write some serious sounding sentences to get this post pass the Spamfilter. Maybe using some Wiki Formatting does the job?

• strike
• bold
• italic

  rm -rf spamfilter-plugin
  

[J. Berger <jeberger@…>]

Replying to r.wilczek@…:

When the slaves started their requests, the server's load exploded and did not recover. The server was not able to handle the requests any more (in fact, the system was no longer responsive at all).

That sounds a bit extreme for a result. The patch can incur a small performance hit in some configurations, but not to the point you are describing. In order to better understand what's happening, I have some questions about your setup:

• Are all your configs on a single trac environment or are they on multiple environments? If the latter, how many configs do you have on a single environment (max and min)?
• Are all your configs on the same repos path or different repos paths? Given the number of configs you report, I assume the latter so how many configs do you have on a single repos path?
• How long is the repos history for the paths you are building?
• Are you building all revisions or only the latest?

[anonymous]

• All our configs are on a single trac environment (virtualized 2-core machine, 8GB RAM, connected to a local PostgreSQL)
• The 411 configs point to 306 different repos-paths, so there are 411/306 == 1.34 configs/path.
  • The configs define 880 platforms, which results in
    • an average of 880/411 == 2.14 platforms per config. The maximum is 19 platforms/config.
    • an average of 880/306 == 2.87 platforms per path.
  • As an example, last week the system completed 918 builds from monday to friday; active build time is approximately from 09:45am to 19:00pm.
    • so there where 183.6 builds/day,
    • resulting in an average of ~20 builds/active hour (if a 19-platform-config is triggered, there are much more builds/hour of course).
• The history-length differs. Long living branches have a larger history than feature-branches with a short life-cycle. When a branch is reintegrated and recreated, I usually reset the build-config, deleting historic builds, steps, reports and report_items. Currently there are 46151 builds recorded in the table bitten_build; so 46151/306 == 150.82 is the average build-history per config (which is not built revision per config).
• We are building the latest revisions only.
• 24 slaves request the server every 2 minutes, so there is an average of 12 requests/minute as long as there is no pending build.

To tell the truth, there are no performance-issues without your patch.

[Roland Wilczek <r.wilczek@…>]

Error ...

Replying to anonymous:

• [...] Currently there are 46151 builds recorded in the table bitten_build; so 46151/306 == 150.82 is the average build-history per config (which is not built revision per config).

• [...] Currently there are 46151 builds recorded in the table bitten_build; so 46151/411 == 112.28 is the average build-history per config (which is not built revision per config).

[J. Berger <jeberger@…>]

Thanks for your detailed reply. Given your configuration, my patch should not improve performance significantly. The main performance gains will occur when there are lots of builds for the same repo path (with different configs and/or platforms), but there should be no gain for builds that concern different paths.

I've attached a new version of the patch, that should have a lot less negative impact on setups like yours. The second patch is standalone (use only one or the patches).

On my system (admittedly not as large as yours, but running on a very old server), I have a single repo path resulting in 12 builds for every revision. The timings for a single slave request with no pending builds are:

• With the current bitten version: 27s.
• With my patch (either version): 8.5s (that is 3.5 times shorter).

[Roland Wilczek <r.wilczek@…>]

27 secs for a single slave-request? How comes? Even 8.5 secs are awesome!

[J. Berger <jeberger@…>]

Replying to Roland Wilczek <r.wilczek@…>:

27 secs for a single slave-request? How comes? Even 8.5 secs are awesome!

Well, the server is an old P4 2.5GHz with 1GB of memory.

According to the profiler, most of the time is spent querying the svn backend (mostly in the get_entries and get_history functions in svn_fs.py). Hence the patch's efforts to minimize these calls by caching the results.

[J. Berger <jeberger@…>]

Added v3 of the patch, which fixes an issue that caused only the latest build to be visible for each platform. Older revisions would link to the latest instead of their respective builds.

[fbrettschneider@…]

I tried to apply patch v3 but it doesn't fit to the current SVN version of queue.py. Could you please explain which patch replaces its predecessor and which patch is to apply over another patch?

[fbrettschneider@…]

Replying to fbrettschneider@…:

I tried to apply patch v3 but it doesn't fit to the current SVN version of queue.py.

Sorry, my local changes led to the mismatch.

So I assume your v3 patch replaces the other two ones, right?

[fbrettschneider@…]

Replying to fbrettschneider@…:

Replying to fbrettschneider@…:

I tried to apply patch v3 but it doesn't fit to the current SVN version of queue.py.

Sorry, my local changes led to the mismatch.

Now I've found out that my local changes are because of my applied t765_multi_repos_r1010.patch. (see #765)

repos.get_node(config.path, config.max_rev) turned to repos.get_node(repos_path) in that patch. Could you please provide your patch version based on an applied multi-repos patch?