[upgrade] Can't remove duplicate builds with one-liner

[Olaf Meeuwissen <olaf.meeuwissen@…>]

Tried upgrading our Bitten plugin from trunk@641​ to trunk@880​. Got the following when trying to upgrade the Trac (0.11.7) environment:

(virtualenv)$ trac-admin /path/to/trac/env upgrade
Adds a unique index on (config, platform, rev) to the bitten_build
table. Also drops the old index on bitten_build that serves no real
purpose anymore.

Config Name, Revision, Platform :: [<list of build ids>]
--------------------------------------------------------
source, 316, 1 :: [229, 230]
source, 390, 1 :: [270, 271]
--------------------------------------------------------

Duplicate builds found. You can remove the builds you don't want to
keep by using this one-line command:

(virtualenv)$ python -c "from bitten.model import Build; from trac.env import Environment; Build.fetch(Environment('/path/to/trac/env'), <buildid>).delete()"

...where <buildid> is the id of the build to remove.

Upgrades cannot be performed until conflicts are resolved.
The upgrade script will now exit with an error:

Command failed: 
(virtualenv)$ python -c "from bitten.model import Build; from trac.env import Environment; Build.fetch(Environment('/path/to/trac/env'), 316).delete()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named bitten.model

For what it's worth, the Bitten plugin is installed in the Trac environment:

(virtualenv)$ ls /path/to/trac/env/plugins
Bitten-0.6dev_r641-py2.5.egg
Bitten-0.7dev_r880-py2.5.egg

and the whole project in question is running from a virtualenv setup.

I've rolled back my changes for now and am using trunk@641​ again for the time being.

Seeing that there really is no difference between the 0.6.x branch and trunk, this would affect 0.6b2 as well.

[hodgestar]

The problem is that eggs in /path/to/trac/env/plugins are not available inside the virtualenv Python because they're not installed in the virtualenv (they're only available when running Trac itself).

For the moment you can do:

import sys; sys.path.append('/path/to/trac/env/plugins/Bitten-0.7dev_r880-py2.5.egg')

and then hopefully import bitten will succeed.

Ideally it would be nice to provide this sort of clean-up functionality via trac-admin but that Trac functionality isn't available until Bitten moves to Trac 0.12.

[osimons]

BTW, there is no order or logic in how Trac loads plugins/eggs from plugins folders. You should not keep more than one copy around here (or anywhere else loadable by the Trac/Python? process), as Trac may easily decide to load the "wrong" version.

Also, notice that Bitten installed as egg in Trac plugins directory means that none of its slave-functionality will be working. The bitten-slave console script won't be registered, nor will any of the Bitten build tools be properly registered as setuptools entry_points. This is then essentially a 'master-only' installation of Bitten. It should work fine for Trac-only usage, even if it isn't the documented way of installing Bitten.

[anonymous]

Replying to osimons:

BTW, there is no order or logic in how Trac loads plugins/eggs from plugins folders. You should not keep more than one copy around here (or anywhere else loadable by the Trac/Python? process), as Trac may easily decide to load the "wrong" version.

Ouch, the Admin->Plugins tab no longer lets a TRAC_ADMIN uninstall plugins ...

Also, notice that Bitten installed as egg in Trac plugins directory means that none of its slave-functionality will be working. The bitten-slave console script won't be registered, nor will any of the Bitten build tools be properly registered as setuptools entry_points. This is then essentially a 'master-only' installation of Bitten. It should work fine for Trac-only usage, even if it isn't the documented way of installing Bitten.

That's okay. The (production) server hosting our Trac environments has no development tools anyway. Our builds are done on separate machines, typically using a very bare bones virtual machine or two (in snapshot mode), and are instructed on how to pull in all the pieces required for a build. In principle that could be put in the recipe but it turned out to be easier for us to put that logic in a shell script that is part of the checkout and run from the recipe with sh:exec.

[anonymous]

Replying to hodgestar:

The problem is that eggs in /path/to/trac/env/plugins are not available inside the virtualenv Python because they're not installed in the virtualenv (they're only available when running Trac itself).

Thought so much.

For the moment you can do:

import sys; sys.path.append('/path/to/trac/env/plugins/Bitten-0.7dev_r880-py2.5.egg')

and then hopefully import bitten will succeed.

The import succeeds alright now, but deleting generates a traceback.

(virtualenv)$ python
Python 2.5.2 (r252:60911, Jan 24 2010, 17:44:40) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.path.append('/path/to/trac/env/plugins/Bitten-0.7dev_r880-py2.5.egg')
>>> from bitten.model import Build; from trac.env import Environment 
>>> Build.fetch(Environment('/path/to/trac/env'),229).delete()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/bitten/model.py", line 524, in fetch
  File "/virtualenv/lib/python2.5/site-packages/Trac-0.11.7-py2.5.egg/trac/db/util.py", line 64, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/virtualenv/lib/python2.5/site-packages/Trac-0.11.7-py2.5.egg/trac/db/sqlite_backend.py", line 80, in execute
    result = PyFormatCursor.execute(self, *args)
  File "/virtualenv/lib/python2.5/site-packages/Trac-0.11.7-py2.5.egg/trac/db/sqlite_backend.py", line 59, in execute
    args or [])
  File "/virtualenv/lib/python2.5/site-packages/Trac-0.11.7-py2.5.egg/trac/db/sqlite_backend.py", line 51, in _rollback_on_error
    return function(self, *args, **kwargs)
pysqlite2.dbapi2.OperationalError: no such column: last_activity

I finally ended up using sqlitebrowser to remove the records associated with the duplicate bitten_build ids. After that

(virtualenv)$ trac-admin /path/to/trac/env upgrade
Adds a unique index on (config, platform, rev) to the bitten_build
table. Also drops the old index on bitten_build that serves no real
purpose anymore.
Fixes any auto increment sequences that might have been left in an
inconsistent state.  Upgrade scripts for schema versions > 10 should
handle sequence updates correctly themselves.
Bitten upgrade to version 10 done.
Renames or removes \*.log.level files created by older versions of
migrate_logs_to_files.
Remove \*.log.levels files without a matching \*.log file (old Bitten
versions did not delete .log.levels files when builds were deleted)
Bitten upgrade to version 11 done.
Add a column for storing the last activity to the build table.
Bitten upgrade to version 12 done.
Upgrade done.

seems to have gone alright, though I haven't tried a build yet.

Ideally it would be nice to provide this sort of clean-up functionality via trac-admin but that Trac functionality isn't available until Bitten moves to Trac 0.12.

Before that happens, a Bitten 0.6 release'd be nice ;-)

[Olaf Meeuwissen <olaf.meeuwissen@…>]

Replying to anonymous: Sorry forgot to set my user name and email address :-(

[...] though I haven't tried a build yet.

First build just succeeded without a hitch.

BTW, looking back over the traceback and the upgrade log I was wondering where the last_activity column got added and what the initial one-liner assumes in terms of schema version. It appears my upgrade started from version 9.

[hodgestar]

The one-liner is now failing because the database isn't up to date enough to use the schemas in bitten.model. The last_activity column got added fairly recently. Since the one-liner is part of the upgrade it really shouldn't rely on bitten.model at all.

[osimons]

Rescheduling. Upgrades should be a priority for 0.6.

[hodgestar]

I've written a script for deleting builds without relying on Bitten itself. It was committed to trunk in r940, r941 and r942 and was backported to 0.6.x in r947. See source:trunk/contrib/deletebuild.py​.