Lear how to develop increasingly sophisticated databases from scratch
I’m excited to share DB From Zero (dbfromzero.com) with you today! This is a new project that aims to explore different aspects and components of databases by developing increasingly sophisticated prototypes. Additionally, benchmarking is performed to quantify the impact of different design parameters and workloads, and thereby improve our intuition about databases. Today I’d like to share with you one of the recent projects.
Log-Structured Merge-Tree for Persistent Reads and Writes
The most recent project explores Log-Structured Merge-Tree for Persistent Reads and Writes. The LSMTree is an interesting data structure that is commonly used in developing high-performance key/value stores. Projects including Google’s LevelDB and Facebook’s fork of LevelDB, RocksDB, use this data structure to support for high-volumes of random access reads and writes.
As discussed further in the project background section, an LSMTree leverages multiple types of data containers with different properties, chiefly in-memory vs. on-disk, to achieve this high performance.
Creating an LSMTree involves developing several interesting components with appropriate coordination between them to support concurrent operations. (details) In addition to development and testing, benchmarking is also performed to quantify the performance of our protoype LSMTree and we find it can achieve up to 328,400 operations/second using a high performance NVM drive and appropriate configuration and workloads.
Additional details on benchmarking and more results can be found here.
More to come…
As of current, four projects have been created to explore different components of databases though hands on development. And this is just the start! More projects are in the pipeline and I’m quite excited from some of the future topics concerning transactions and serialization-level isolation.
Check back soon for new projects. Also, I’d appreciate your feedback on the current projects as well as ideas for new projects. Contact me at mattthew.hagy@gmail.com.