Last year I worked on building a highly available decentralised service with eventual consistency. The goal was to have a service that could be deployed as a sidecar within the POD of any other service on Kubernetes and allow the main service to manage specific platform flags locally. The requirements of the project were in a way that were sympathetic to eventual consistency. This meant that although the value of the flags could be out of date in each replica at any moment in time, as long as the values in all the replicas (sidecars) were convergent, the requirements were satisfied.
Other requirements were that we could have tens or hundreds of millions of these types of flags at any point in time and we potentially needed to be able to access multiple of them with low latencies within API calls. These requirements combined with the fact that eventual consistency was a good fit, led me to think about implementing the project with CRDTs. CRDT stands for Conflict-free Replicated Data Type. In short, it’s a set of techniques to replicate data within multiple replicas with eventual consistency. This means that there is no one central database that all requests must go to and the replicas don’t have to coordinate with each other in order to answer questions about the state of the system.
Recently, I gave a talk at GopherCon 2021 about what CRDTs are, various types of CRDTs, how they can be leveraged and implemented a few of them in Go. It was fun to think about how to present relatively complex ideas in a way that are useful to attendees. Here’s a link to the talk. I hope you enjoy it.