Database on Linux/ext3 unrecoverable after power down

yaakov · *nix forums beginner Joined: 16 May 2006 Posts: 8

Thank you very much for warning me.

Florian Weimer · *nix forums Guru Joined: 19 Feb 2005 Posts: 418

* yaakov:

yaakov · *nix forums beginner Joined: 16 May 2006 Posts: 8

It turned out that switching off the disk's write cache with "hdparm
-w0 /dev/hda1" resolved the problem.

(In the previous tests I did this once, but did not repeat it after the
first reboot. Of course, after the reboot the write cache was switched
on by default and the following power-down corrupted the data base).

Switching off the write cache results in a noticable performance loss.
I will have to look into optimizations like storing log files on
different drives (without write cache) and data files on large drives
with caches enabled...

Thank you very much for your help.

yaakov · *nix forums beginner Joined: 16 May 2006 Posts: 8

Florian Weimer · *nix forums Guru Joined: 19 Feb 2005 Posts: 418

* yaakov:

yaakov · *nix forums beginner Joined: 16 May 2006 Posts: 8

reading the documentation, I found the following pieces of information:

* use hdarm to switch off the write-cache of the disk.

As predicted, this reduces performance, but the problem is still
present.

yaakov · *nix forums beginner Joined: 16 May 2006 Posts: 8

I am quite new to BerkeleyDB --- working my way through the
documentation in order to be able to use BerkeleyDB in our project.

In one of our tests, we lost the whole data base after a power
shutdown --- this is exactly what we want to avoid in our application.
Please help me by pointing out how I have to use BerkeleyDB correctly
to avoid data loss.

Here are the details:

Debian Linux; ext3 filesys block size: 4096.
BerkeleyDB.pm 0.26; BerkeleyDB 4.3, block size: 4096.

A simple perl script (see below) opens a transactional data base and
adds entries and checks random entries for correctness.

The script and the data base perfectly survive killing the script (I
tried this more than 1000 times). But when I switch down the power and
restart the computer, I get the following symptoms:

Startup:
kjournald starting. Commit interval 5 seconds
EXT3-fs: hda1: orphan cleanup on readonly fs
EXT3-fs: hda1: 3 orphan inodes deleted
EXT3-fs: recovery complete.

This indicates that some write operation had not been completed
properly and that the ext3 journal recovered the file system.

I started the script again (hoping that the database survived the
power-down), but it did not open the environment.

Now, running db4.3_recover -c in the data base directory, I get the
following messages:

db_recover: Log sequence error: page LSN 20 4378733; previous LSN 20
4379147
db_recover: Recovery function for LSN 20 4378733 failed on forward pass
db_recover: PANIC: Invalid argument
db_recover: PANIC: fatal region error detected; run recovery
db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database
recovery

The error log file contains the same messages.

The documentation states about errors
Log sequence error: page LSN # ######; previous LSN ## ######.:

A database update was made outside of a transaction. Check that your
application passes a transaction handle to all opens and updates of
transactionally protected databases. This error leaves the environment
unrecoverable, and the databases must be dumped and reloaded.

Please point me to the right place how I can ensure that our data bases
survive power down.

Here is the script:

use BerkeleyDB ;

# open data base environment
my $Env = BerkeleyDB::Env->new (
-Home => '/home/www/111/F',
-Flags => DB_INIT_MPOOL | DB_INIT_LOCK |
DB_INIT_LOG | DB_INIT_TXN | DB_CREATE | DB_RECOVER,
-ErrFile => '/home/www/111/F/db_error.log',
) or die "Cannot open enviroment: $!\n";

# open a data base within an transaction
my $txn0 = $Env->txn_begin(); my %db;
my $db=tie %db, 'BerkeleyDB::Btree',
-Env => $Env,
-Filename => "db.db",
-Flags => DB_CREATE,
-Txn => $txn0 ,
or die "Cannot open 'authoritys.db': $!\n";

# Commit the opening of the database
$txn0->txn_commit();

# Print information about the data base
my $stat=$db->db_stat; print "$_=$stat->{$_}\n" for sort keys %$stat;

# read and write the data base within transactions
my $r;
while(1) { # inifinite loop -- stop by signal or power down

# Create a transaction object, associate it with the database
my $txn1 = $Env->txn_begin();
$db->Txn($txn1);

# keep an index where to write in the data base
my $w=1+($db{"w"}||0); $db{"w"}=$w;
# write some data
$db{(reverse $w)x20}=$w x 100;

# get random, previously written index
$r=1+int(rand($w-1));
# check the data for integrity
unless ( $db{(reverse $r)x20} eq ($r x 100)) {
print "read error: $r\n";
}

# commit the transaction and start all over again.
$txn1->txn_commit () ;
}

Google