iMac computer on office desk overlooking the city buildings

The Reasons for Data Mesh on Pulsar

Blog Summary: (AI Summaries by Summarizes)
  • Data mesh is becoming a popular way for companies to implement their data strategy.
  • Flexibility is a crucial aspect of data mesh, from creating data products to consuming them.
  • Apache Pulsar and Apache Kafka are two popular choices for publish/subscribe technologies in data mesh.
  • Pulsar has a built-in schema registry, while Kafka lacks one.
  • Kafka requires teams to publish data products twice if they want both ordered and random ordering, adding complexity to operations and programming.

Data mesh is quickly becoming a way for companies to roll out their data strategy. If you haven’t already learned about data mesh, I suggest doing so. It comes with organizational and technical changes. I think a crucial part of your data mesh revolves around the choice of publish/subscribe technologies.

At the crux of data mesh is a desire for flexibility. This flexibility extends from the creation of data products (publish) all the way to the consumption of data products (subscribe). Greater flexibility in data mesh means that teams creating and consuming data products have even less coordination between themselves.

To help us look at the pub/sub needs for data teams, we’ll compare Apache Pulsar and Apache Kafka.

There isn’t a vast difference between publishing data in Pulsar or Kafka from a cursory look. They both have similar publish APIs. However, Kafka lacks a built-in schema registry, whereas Pulsar has a built-in schema registry that prevents incorrect schema usage. Data mesh calls for discovery and schema services. However, I still believe these should be built into the technology itself and not be added on.

Another significant difference between Kafka and Pulsar that isn’t as well known is that once data is published into a Kafka topic, it can’t be consumed in any other way than through reading that partition. This manifests when teams have to choose between wanting an ordered topic or random ordering, such as a round-robin layout. Before a team can answer this question, the team needs to know how the data will be consumed. In a data mesh scenario, the team producing the data product may be able to talk to the team or predict all the ways their data will be consumed in current or future use cases. In these situations, the team would be forced to publish their data product twice, once into an ordered topic and again into a random topic. The Kafka solution adds complexity in both the operations, by doubling the data and load, and programmatically by having to know which topic is ordered or random.

With Pulsar, this choice is dramatically different. Pulsar allows for different methods of consumption that can be chosen by the consumer. In this case, the producer can produce in an ordered manner. The data product consumer can choose from several ways of maintaining order or randomized data. Pulsar is flexible enough to let the downstream consuming team choose for themselves.

A common use case for pub/sub is messaging. With messaging, consumption requires the ability to individually acknowledge that a message was consumed and processed. Kafka’s messaging support entails workarounds whereas Pulsar supports this natively. Once again, Kafka requires more coordination and operational considerations. Pulsar allows for the consumer of the data product to make the choice.

I always recommend using managed services whenever you are running on the cloud. There are two main managed services for Pulsar on the major cloud providers. StreamNative has StreamNative Cloud and DataStax has Astra Streaming.

I think a critical feature of the technologies for data mesh is flexibility. We don’t want to have to complicate our technical landscape with the various limitations of our core technologies. One of those core technologies is our choice of pub/sub. Data teams won’t know ahead of time all of the ways that the data products will be consumed. We can clearly see that Pulsar will make it far easier for data mesh while Kafka will complicate it. When doing data mesh, I highly recommend Pulsar as the pub/sub of choice.


Related Posts

zoomed in line graph photo

Data Teams Survey 2023 Follow-Up

Blog Summary: (AI Summaries by Summarizes) Many companies, regardless of size, are using data mesh as a methodology. Smaller companies may not necessarily need a

Laptop on a table showing a graph of data

Data Teams Survey 2023 Results

Blog Summary: (AI Summaries by Summarizes) A survey was conducted between January 24, 2023, and February 28, 2023, to gather data for the book “Data

Black and white photo of three corporate people discussing with a view of the city's buildings

Analysis of Confluent Buying Immerok

Blog Summary: (AI Summaries by Summarizes) Confluent has announced the acquisition of Immerok, which represents a significant shift in strategy for Confluent. The future of

Tall modern buildings with the view of the ocean's horizon

Brief History of Data Engineering

Blog Summary: (AI Summaries by Summarizes) Google created MapReduce and GFS in 2004 for scalable systems. Apache Hadoop was created in 2005 by Doug Cutting

Big Data Institute horizontal logo

Independent Anniversary

Blog Summary: (AI Summaries by Summarizes) The author founded Big Data Institute eight years ago as an independent, big data consulting company. Independence allows for