Creating Work Queues with Apache Kafka and Apache Pulsar

Jesse+ by | Aug 29, 2018 | Blog, Business, Data Engineering, Data Engineering is hard | 2 comments

A common use case for using Kafka and Pulsar is to create work queues. The two technologies offer different implementations for accomplishing this use case. I’ll discuss the ways of implementing work queues in Kafka and Pulsar as well as the relative strengths of...

Why Real-time is the Future

Jesse+ by | May 1, 2018 | Blog, Business, Data Engineering, Data Engineering is hard | 0 comments

One of the benefits of teaching and consulting is the sheer number of organizations, teams, and people I get to work with. Since I deal with so many different groups, I can see patterns emerge much faster than others. One pattern I saw early on was real-time Big Data....

The Four Types of Technologies You Need for Real-time Big Data Systems

Jesse+ by | Apr 26, 2018 | Blog, Business, Data Engineering, Data Engineering is hard | 0 comments

Creating real-time data pipelines bring new challenges. There are new concepts and technologies that you’ll need to learn and understand. To help you understand the basic technologies you need in a real-time data pipeline, I break it down into 4 general types. These...

What Are Batch and Real-time Big Data?

Jesse+ by | Apr 24, 2018 | Blog, Business, Data Engineering, Data Engineering is hard | 2 comments

The move from batch to real-time Big Data represents change. It will entail using brand new technologies and concepts that you haven’t dealt with before. Batch Big Data Let’s start off by defining batch Big Data. For batch, all data must be there when the processing...

Data engineers vs. data scientists

Jesse+ by | Apr 11, 2018 | Blog, Business, Data Engineering, Data Engineering is hard | 1 comment

I wrote a post for the O’Reilly data blog going into my latest thoughts and views on data engineers versus data scientists. I continue on to talk about machine learning engineers.

Should You Even Do Big Data?

Jesse+ by | Mar 28, 2018 | Blog, Business, Data Engineering, Data Engineering is hard | 0 comments

There’s an elephant in the room with Big Data. If an organization tries to half-ass their way through a Big Data project, they’re going to fail (usually a 5-10% odds of success). Given this really low success rate, should you even do Big Data? When I worked at a Big...