niXforums Forum Index
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   PreferencesPreferences   Log in to check your private messagesLog in to check your private messages   Log inLog in 
·  nixdoc.net ·  man pages ·  Linux HOWTOs ·  FreeBSD Tips ·  Forums
navigation Forum index » Databases » Berkeley DB
size of DatabaseEntry and size of .jdb file
Post new topic   Reply to topic Page 1 of 1 [6 Posts] View previous topic :: View next topic
Author Message
Mark Hayes
*nix forums beginner


Joined: 27 May 2005
Posts: 35

PostPosted: Wed Apr 13, 2005 11:10 pm    Post subject: Re: size of DatabaseEntry and size of .jdb file Reply with quote
Xin wrote:
Quote:
Hi Mark,

Thanks again for your quick reply. Actually in my application we need
to build a large database over gigabytes. The size I provided is only
for a testing program. I tried to track in the source code in order to
find what is being written to the .jdb file, but unfortunately I got
lost. Could you kindly pointed out from which method of which class can
I see what is actually written to the .jdb file for each entry (the
part that I can see both the header and the databaseentry itself are
being written)?

All log entries are written by the classes in the
com.sleepycat.je.log.entry package. These classes writer header
information, and they use classes in the com.sleepycat.je.tree package
to write record contents. For example, the log.entry.LNLogEntry class
writes the header for a leaf node, and it uses the tree.LN class to
write the content.

Just in case this wasn't clear, to perform measurements on disk space
utilization you will need to create a database with many log files, and
then ensure that the cleaner has run.

Mark
Back to top
Xin
*nix forums beginner


Joined: 01 Aug 2005
Posts: 6

PostPosted: Wed Apr 13, 2005 10:59 pm    Post subject: Re: size of DatabaseEntry and size of .jdb file Reply with quote
Hi Mark,

Thanks again for your quick reply. Actually in my application we need
to build a large database over gigabytes. The size I provided is only
for a testing program. I tried to track in the source code in order to
find what is being written to the .jdb file, but unfortunately I got
lost. Could you kindly pointed out from which method of which class can
I see what is actually written to the .jdb file for each entry (the
part that I can see both the header and the databaseentry itself are
being written)?

Thanks so much.

Best,
Xin
Back to top
Mark Hayes
*nix forums beginner


Joined: 27 May 2005
Posts: 35

PostPosted: Wed Apr 13, 2005 1:47 pm    Post subject: Re: size of DatabaseEntry and size of .jdb file Reply with quote
Xin wrote:
Quote:
Hi Mark,

Thanks for your quick reply.

I understand that there is overhead of each entry, would you please
tell me that's in the overhead?

In general, there is a header per entry that includes the size,
transaction information, type, etc. Btree information is logged as
separate internal nodes.

Quote:
If I only insert tuples, will using the cleaner help to reduce the .jdb
file size? I tested, but did not see much change. Before using the
cleaner, it's 9.55MB; after using it, it's 9.10MB. By the way, shall
the env.cleanLog( ) be used after env.sync( ) or before env.sync( ) ? I
configured my environment and database to be nontransactional.

If you are using sync(), then calling cleanLog() after sync is best.
Note that the cleaner thread calls cleanLog() automatically, so you only
need to call cleanLog() to perform explicit cleaning in exceptional cases.

The default size of a log file is 10MB. Several log files must be
created before cleaning occurs. This is described here:

http://www.sleepycat.com/blogs/bdb-je/archives/2004/12/02/11.38.03/

If you want to create a minimum sized log, you can dump and load the log
using the DbDump and DbLoad utilities.

What is your goal in looking at the size of the JDB files?

Is your application highly constrained in terms of disk space?

Is it realistic that a small database (less than 10 MB) would be created?

Mark
Back to top
Xin
*nix forums beginner


Joined: 01 Aug 2005
Posts: 6

PostPosted: Tue Apr 12, 2005 5:02 pm    Post subject: Re: size of DatabaseEntry and size of .jdb file Reply with quote
Hi Mark,

Thanks for your quick reply.

I understand that there is overhead of each entry, would you please
tell me that's in the overhead?

If I only insert tuples, will using the cleaner help to reduce the .jdb
file size? I tested, but did not see much change. Before using the
cleaner, it's 9.55MB; after using it, it's 9.10MB. By the way, shall
the env.cleanLog( ) be used after env.sync( ) or before env.sync( ) ? I
configured my environment and database to be nontransactional.

Thanks,
Xin
Back to top
Mark Hayes
*nix forums beginner


Joined: 27 May 2005
Posts: 35

PostPosted: Sat Apr 09, 2005 3:10 pm    Post subject: Re: size of DatabaseEntry and size of .jdb file Reply with quote
Xin wrote:

Quote:
what else is stored in .jdb file besides the key/data pair and
something to maintain the Btree? Is there anyway to reduce the .jdb
file size to make it approaching the sum of every key/data pair size.

For small keys or data, the size of the .jdb files will always be
larger than the actual key/data size because of the overhead per entry.
Additionally, the cleaner may not have run yet on the .jdb file.

Please see the following FAQ entry for more information:

http://www.sleepycat.com/blogs/bdb-je/archives/2004/12/02/11.38.03/

Mark
Back to top
Xin
*nix forums beginner


Joined: 01 Aug 2005
Posts: 6

PostPosted: Sat Apr 09, 2005 2:41 pm    Post subject: size of DatabaseEntry and size of .jdb file Reply with quote
Hi,

I configure the Bdb to be nontransactional and allow duplicate. Then I
use the simple code below to store a thousand key/data pairs.

The byteArray size for DatabaseEntry theKey is 4, 5 and 6 when i is
smaller than 10, 100 and 1000 separately. The byte Array size for
DatabaseEntry theData is 7, 11 and 15 when i is smaller than 10, 100
and 1000 separately(i is the number of the for loop).

However, the .jdb file size is finally 76KB, which is far more larger
than the real size of the key/data pair.

When I use
Quote:
java com.sleepycat.je.util.DbDump -h -p -s to print out the database
records, the size of all of the records is 25.8KB.


what else is stored in .jdb file besides the key/data pair and
something to maintain the Btree? Is there anyway to reduce the .jdb
file size to make it approaching the sum of every key/data pair size.

for(int i=0; i<1000; i++){

String stheKey = new String ("doc" + i);

DatabaseEntry theKey = new DatabaseEntry(stheKey.getBytes("UTF-8"));

String stheData = new String("doc" + i + i + i + i);

DatabaseEntry theData = new DatabaseEntry(stheData.getBytes("UTF- 8"));

try {

myDbEnv.getElemTupleDB().put(null, theKey, theData);

System.out.println(stheData);
} catch (DatabaseException dbe) {
System.out.println("Error putting entry " + i);
throw dbe;
}

Thanks a lot for help.

Cheer,
Xin
Back to top
Google
Back to top
Display posts from previous:   
Post new topic   Reply to topic Page 1 of 1 [6 Posts] View previous topic :: View next topic
The time now is Thu Jan 08, 2009 3:10 am | All times are GMT
navigation Forum index » Databases » Berkeley DB
Jump to:  

Similar Topics
Topic Author Forum Replies Last Post
No new posts Running php file everyday on scheduled time sachin PHP 1 Fri Jul 21, 2006 12:49 pm
No new posts Regarding thesaurus iso file Srikanth modules 0 Fri Jul 21, 2006 10:42 am
No new posts how can i get a file descriptor not used? mars system 0 Fri Jul 21, 2006 7:41 am
No new posts small GTK "Open file" dialog David Siroky Debian 0 Fri Jul 21, 2006 7:30 am
No new posts Trouble Declaring 3D Array in Header File free2klim C++ 1 Fri Jul 21, 2006 4:07 am

Credit Cards | Credit Counseling | Bankruptcy | Debt Consolidation | Guitar Lessons
Copyright © 2004-2005 DeniX Solutions SRL
 
Other DeniX Solutions sites: Unix/Linux blog |  electronics forum |  medicine forum |  science forum | 
Privacy Policy


Powered by phpBB © 2001, 2005 phpBB Group

[ Time: 0.1646s ][ Queries: 20 (0.0778s) ][ GZIP on - Debug on ]