Kafka REST and JQuery Helper

Blog Summary: (AI Summaries by Summarizes)
  • The author is open sourcing a module they wrote for their Real-time Data Engineering class.
  • The module uses Apache Spark and Apache Kafka to process data and display it in real-time on a webpage.
  • The module is called KafkaRESTHelper and makes it easier to interface with the Kafka REST server and JQuery.
  • The module assumes that the data is in Kafka and accessible by the Kafka REST server, and that the value of the payload is a valid JSON string.
  • To use the module, initialize the KafkaRESTHelper helper with the URL of where to access the Kafka REST server.

I’m open sourcing one of the modules I wrote for my Real-time Data Engineering class. We use Apache Spark and Apache Kafka to process data. Then, we show the data in real-time on a webpage using this JavaScript module to pull in data from Kafka via the Kafka REST interface.

The KafkaRESTHelper makes it easier to interface with the Kafka REST server and JQuery.

You can view the module at work in this video:

Configuration

This helper function makes a few assumptions about data and URLs. It assumes that the data is in Kafka and accessible by the Kafka REST server. It assumes the value of the payload is a valid JSON string.

Usage

Initialize the KafkaRESTHelper helper. The parameter is the URL of where to access the Kafka REST server. This example shows how to use Kafka REST behind a proxy:

var kafkaRESTHelper = new KafkaRESTHelper("/kafkarest/consumers/");

Next, you’ll create a createConsumerInstance by passing in the consumer group name, the topic name to consume, the function to call when data is retrieved, and the interval in millisecond to poll for data on the topic. This example shows an example call:

kafkaRESTHelper.createConsumerInstance("totalsbygameid", "totalsbygameid",
    byGameId, 10000)

The callback function will be called with a JavaScript object containing all of the data for the topic merged together. Here is an example function:

function byGameId(data) {
  console.log(data)
}

This function would power the rest of page. It might be used for charting or another function.

The module supports multiple calls to the createConsumerInstance function with different topics or the same topic.

TODOs

This code doesn’t delete the consumer instances automatically. I’d need to maintain a list of consumer nstances created and then register an onclose event to delete them.

Related Posts

The Difference Between Learning and Doing

Blog Summary: (AI Summaries by Summarizes)There are several types of learning videos: hype, low effort, novice, and professional.It is important to avoid hype, low-effort, and

The Data Discovery Team

Blog Summary: (AI Summaries by Summarizes)The concept of a “data discovery team” is introduced, which focuses on searching for data in an enterprise data reality.Data

Black and white photo of three corporate people discussing with a view of the city's buildings

Current 2023 Announcements

Blog Summary: (AI Summaries by Summarizes)Confluent’s Current Conference featured several announcements that are important for both technologists and investors.Confluent has two existing moats (replication and

zoomed in line graph photo

Data Teams Survey 2023 Follow-Up

Blog Summary: (AI Summaries by Summarizes)Many companies, regardless of size, are using data mesh as a methodology.Smaller companies may not necessarily need a data mesh

Laptop on a table showing a graph of data

Data Teams Survey 2023 Results

Blog Summary: (AI Summaries by Summarizes)A survey was conducted between January 24, 2023, and February 28, 2023, to gather data for the book “Data Teams”

Black and white photo of three corporate people discussing with a view of the city's buildings

Analysis of Confluent Buying Immerok

Blog Summary: (AI Summaries by Summarizes)Confluent has announced the acquisition of Immerok, which represents a significant shift in strategy for Confluent.The future of primarily ksqlDB

Tall modern buildings with the view of the ocean's horizon

Brief History of Data Engineering

Blog Summary: (AI Summaries by Summarizes)Google created MapReduce and GFS in 2004 for scalable systems.Apache Hadoop was created in 2005 by Doug Cutting based on