Book Reports
I’m nearing the end of Designing Data Intensive Applications and realized that instead of waiting until the end of the book to do a report on my learnings, I should have been blogging what I was learning as I read each section. That was a mistake that I won’t make again. Giving check ins on my learnings will reduce the overhead associated with one large post and reduce the size of the task. It’ll also help me keep track of what I learn throughout the book instead of grouping all my learnings at once. I’m bound to forget something I read.
I just finished a section in the book on streaming applications.
Things I learned:
- Event Sourcing
- Change Data Capture
Event Sourcing
I read a section about Event Sourcing that didn’t make a ton of sense to me. Event Sourcing was a different way about thinking about a database from the applications perspective. Instead of reading the database as a conventional KV store or a structured datastore, the application would use some of the underlying implementation details. Namely, it would use the CDC log. The idea was that if an application needed to read a recorded, immutable history of events that happened instead of looking up a KV store, this would be a better solution. Although, it forces some compute complexity on the application side. The application has to read the log, then reconstruct any valuable information that it needs. The book gave the example of a shopping cart. DynamoDB is a KV store, but sometimes, it’s valuable to keep the history of items that a user adds/deletes from a shopping cart than the actual shopping cart contents. Semantically, they can be made to represent the same information. The cart can be “reconstructed” from the history of events, or the cart can be “read” from the DynamoDB datastore. I guess one way of doing this would be to keep a history of the changed events at a key, instead of the actual value.
So instead of keeping
1. eggs
2. ham
3. kindle
The value at a given key would have
1. add eggs
2. add ham
3. add kindle
Then the application client would reconstruct the history into a cart. Shoppers could then add and remove the most recently added items. Storing the data like this could also be used to resolve write conflicts. IE, we can compare two “histories” as vector clocks and determine which is the “latest.” I don’t remember reading this in the DynamoDB paper though.
Change Data Capture
CDC events often get used to replicate data from one database type to another. Often, an ElasticSearch or a BigQuery isn’t enough to answer all the data needs required by an org and you eventually find yourself using multiple databases for different types of querying. To replicate data between these types of datastores, you need a way to replicate it without duplicating data or losing data.
Enter CDC events. Instead of trying to write some large ass state machine to track which rows/documents are in each database, you “subscribe” to all the write/update/delete events on a database. Then write some ETLs job that listens on that queue and replicates the data in the event to another data source. If you have all the write/update/delete events that brought the dataset to where it is now, you have all the information you need to rebuild the database from a certain index. Obviously, you probably can’t keep the entire log, but some point of it to where you at least have the latest for every key/row.
Don’t try to have an application write to two datastores. If one write fails, you’ll find yourself in a world of pain.