PropaneDB: a database for ProtocolBuffers
Published:
PropaneDB is an open source NoSQL database to store ProtocolBuffer objects in serialized form. It is intended for use with GRPC microservices. PropaneDB enables storing GRPC messages directly without first converting them to specific format just for persistency . The proto file already defines the messages exchanged between microservices, so we might as well use this same definition for storage.
I got the idea for this database some time ago when I was working on a web application backend in Golang. I was using GRPC for the communication between microservices and Postgres as my database. The disadvantage is that you actually need quite a lot of boilerplate code in order to store GRPC (Protobuf) messages in an SQL database. In Go you often need extra annotations(field tags) in structs when you want to use ORM fields of a struct onto tables in an SQL database (especially when you want to use associations). But when using GRPC, the structs are generated by the protobuf compiler and thus you would need to define a new struct that contains the required information and copy all the data from the GRPC message struct to this new struct.
I also experimented with using NoSQL document databases in combination with Protobuf structs, but this often also requires field tags for the conversion to JSON. I prefer to have a single source of truth, which is in my case the protobuf file. Can we store the data from a Protobuf message directly in a database? Well, with PropaneDB we can!
PropaneDB is based on the RocksDB storage engine and thus implemented in C++. The database interface is based on GRPC , so it should be relatively simple to build a PropaneDB driver for any language that is supported by GRPC. I started with a PropaneDB driver in Go, but drivers in other languages follow in due time. When you store an Protobuf object in PropaneDB, it first has to be converted to the GRPC Any type. When you retrieve an object from PropaneDB, you have to unpack the Any type to the original datatype before it can be used.
Today the first version of PropaneDB was released. It contains a very basic implementation to store and retrieve Protobuf objects from the database. I invite you to play around with it and tell me what you think. https://github.com/elan8/propanedb