Lightning Memory-Mapped Database

A discussion on Twitter broke out about LMDB (Lightning Memory-Mapped Database) and I realized I’ve been using it for quite some time now and haven’t written about it as of yet. So here goes!

LMDB is a database storage engine similar to LevelDB or BDB which database authors often use as a base for building databases on top of. LMDB was designed as a replacement for BDB within the OpenLDAP project but it has been pretty useful to use with other databases as well. It’s API design is highly influenced by BDB so that replacing BDB is straight forward.

LMDB is a transactional ACID key/value storage engine (I don’t throw ACID around very easily because ACID is a strict set of guarantees and requires significant work, you can’t just put lipstick on an LSM-tree or ignore index inconsistencies and claim ACID) that is implemented using an Append-only Copy-on-write B+Tree (different than the currently popular LSM-tree based storage engines). LMDB provides:

• Nested transactions.
• Serializable Isolation (highest guarantee of I in ACID possible).
• MVCC (Multiversion concurrency control) to provide concurrency.
• Since it’s not an LSM-tree implementation, it doesn’t require compaction or clean-up processes.
• Resiliency to data corruption without requiring a transaction log. No recovery process after a restart.
• Removes the need for managing caches and cache tuning by using memory mapping.
• A single writer that never blocks readers which removes the need for complex locking and removes the possibility of deadlocks.
• Multiple readers run lockless and never block each other or the single writer.
• Support for batched writes.

Howard Chu has a great session video available that discusses the history and design of LMDB along with benchmark results. You can find some other benchmark results available here.

With any storage engine you should sufficiently test long term workloads that are similar to the characteristics of your production workload to test if LMDB is a good fit. Each storage engine behaves a little differently. Don’t judge based on benchmarks that run for minute(s). As I wrote about before, even short term workloads running for 3 hours can unlock some details that will never be seen in a 5 minute benchmark. Please do the due diligence and pick the right choice for your workload.

There are several language wrappers for LMDB.

multilizer pdf translator 8.3.2
Java
Erlang
C++
Node.js
Ruby
convert secured pdf to word file
Lua
Go
Objective-C