various embedded databases
For a project I wanted a simple key-value embedded database to include in own programs. After searching the web I found different libraries and benchmarked them. These tests are only for my personal usecase(inserting strings into the database based on strings as keys), so please don't start a flamewar because of these results!size of testdata (keys and inserted values): 3234kB(3.1583MB); 173528 keys
(I've used the great highcharts javascript library for the chart, you can click onto the legend to hide data!)
I've tested inserting 173528 strings into the database (the testdata was taken from patl) and measured how many memory was allocated by the routines. Reading back the data was equally fast in every tested library.
I've tested kyoto cabinet, patl, hamsterdb, redis-embedded (original redis), biopython trie implementation and radixtree(take care, in the implementation to remove elements the delete is commented out and a memleak is created, dunno why!). I've tried jdktrie, but it crashed and I've abadonned testing it.
Radixtree is the fastest and most memory efficient library, there are no temporary memory allocations. Redis is very memory hungry, but keep in mind that the original is meant to be an extra server process which gets data over tcpip and thus every api-function works with dynamic allocated strings and stuff like that. And redis does cover much more datatypes than for example radixtree. hamsterdb is also a complete database system and can't be compared with radixtree or the trie from biopython.
Here is the used benchmark-code from kyoto as an example:
_CrtMemState memStateStart, memStateEnd, memStateDiff;
_CrtMemCheckpoint(&memStateStart);
u64 t0;
kyotocabinet::PolyDB polyDBs[TEST_SIZE];
for(int i=0; i<TEST_SIZE; ++i)
polyDBs[i].open(":", kyotocabinet::PolyDB::OWRITER | kyotocabinet::PolyDB::OCREATE);
for(t0=GetMicroTime(),i=0; i<TEST_SIZE;++i) {
for(auto iter=testdata.begin(), end=testdata.end(); iter!=end; ++iter) {
polyDBs[i].set(*iter, toInsert);
}
}
printf("kyotocab insert %8d %8dusec\n",TEST_SIZE*testdata.size(),int(GetMicroTime()-t0));
printf("kyotocab mem stats:\n");
_CrtMemCheckpoint(&memStateEnd);
_CrtMemDifference(&memStateDiff, &memStateStart, &memStateEnd);
_CrtMemDumpStatistics(&memStateDiff);
printf("----\n");