Data mesh is quickly becoming a way for companies to roll out their data strategy. If you haven’t already learned about data mesh, I suggest doing so. It comes with organizational and technical changes. I think a crucial part of your data mesh revolves around the choice of publish/subscribe technologies.
At the crux of data mesh is a desire for flexibility. This flexibility extends from the creation of data products (publish) all the way to the consumption of data products (subscribe). Greater flexibility in data mesh means that teams creating and consuming data products have even less coordination between themselves.
To help us look at the pub/sub needs for data teams, we’ll compare Apache Pulsar and Apache Kafka.
There isn’t a vast difference between publishing data in Pulsar or Kafka from a cursory look. They both have similar publish APIs. However, Kafka lacks a built-in schema registry, whereas Pulsar has a built-in schema registry that prevents incorrect schema usage. Data mesh calls for discovery and schema services. However, I still believe these should be built into the technology itself and not be added on.
Another significant difference between Kafka and Pulsar that isn’t as well known is that once data is published into a Kafka topic, it can’t be consumed in any other way than through reading that partition. This manifests when teams have to choose between wanting an ordered topic or random ordering, such as a round-robin layout. Before a team can answer this question, the team needs to know how the data will be consumed. In a data mesh scenario, the team producing the data product may be able to talk to the team or predict all the ways their data will be consumed in current or future use cases. In these situations, the team would be forced to publish their data product twice, once into an ordered topic and again into a random topic. The Kafka solution adds complexity in both the operations, by doubling the data and load, and programmatically by having to know which topic is ordered or random.
With Pulsar, this choice is dramatically different. Pulsar allows for different methods of consumption that can be chosen by the consumer. In this case, the producer can produce in an ordered manner. The data product consumer can choose from several ways of maintaining order or randomized data. Pulsar is flexible enough to let the downstream consuming team choose for themselves.
A common use case for pub/sub is messaging. With messaging, consumption requires the ability to individually acknowledge that a message was consumed and processed. Kafka’s messaging support entails workarounds whereas Pulsar supports this natively. Once again, Kafka requires more coordination and operational considerations. Pulsar allows for the consumer of the data product to make the choice.
I always recommend using managed services whenever you are running on the cloud. There are two main managed services for Pulsar on the major cloud providers. StreamNative has StreamNative Cloud and DataStax has Astra Streaming.
I think a critical feature of the technologies for data mesh is flexibility. We don’t want to have to complicate our technical landscape with the various limitations of our core technologies. One of those core technologies is our choice of pub/sub. Data teams won’t know ahead of time all of the ways that the data products will be consumed. We can clearly see that Pulsar will make it far easier for data mesh while Kafka will complicate it. When doing data mesh, I highly recommend Pulsar as the pub/sub of choice.
