DBXML modifications very slow using XmlModify

Petra Chong · *nix forums beginner Joined: 11 Oct 2005 Posts: 8

Hello all,

I have a DBXML database running on Windows XP that I am reading from
and writing to using Python. The pattern of usage of this application
is as follows:

1. Get a message from an external source. Messages describe changes to
objects, which have unique IDs.
2. Transform that message into XML, find out which object it describes
by getting the object ID
3. Insert that entire XML document into the database if no document
with that object ID exists. If a document with that object ID does
exist, insert only the changes, and store the previous ones (so nodes
are versioned).
4. Changes are done using XmlModify.addUpdateStep() and
XmlModify.addAppendStep() with XmlModify.execute() at the end.
5. The database is indexed on the object ID as well as the node
versions.
6. All modifications are done within a transaction; the changes
described in a message are therefore either applied completely, or not
at all.

I notice that modifications are very slow and can take up to a minute.
Logging and profiling confirm that the time is being taken in the call
to XmlModify.execute() and not anywhere else.

I am quite new when it comes to dealing with BDB / DBXML; is there
anything really obvious that I should be looking at? I have not looked
at tuning the database at all, beyond adding indexes on obvious things.

Thanks,

PC

Petra Chong · *nix forums beginner Joined: 11 Oct 2005 Posts: 8

I have experimented with adding transaction checkpoints after every
modification and got some success- in most cases, the time taken to
commit goes down to < 1 second.

However, after a certain period of running (I'm trying to determine
this at present), the commits start to take 20 seconds. Memory usage
spikes up to 300MB from 50MB.

I now have the following questions..

1. Why is this the case?

2. Can anyone point me to a good description of what checkpoints are
and what the difference is between these and transaction commits? I
have read the on-disk documentation that comes with the install, and I
don't have a clear picture of what checkpoints are.

3. What could be causing the sudden spike in transaction commit time?

Many thanks,

PC

Google