Should You Even Do Big Data?

Blog Summary: (AI Summaries by Summarizes)
  • Big Data projects have a low success rate (usually 5-10%).
  • Half-assed Big Data projects will fail without specific changes.
  • Companies often reach out for help when their Big Data projects are failing.
  • A cheap, quick, and easy fix is not possible for failing Big Data projects.
  • The book "Data Engineering Teams" provides advice on how to run successful Big Data projects.

There’s an elephant in the room with Big Data. If an organization tries to half-ass their way through a Big Data project, they’re going to fail (usually a 5-10% odds of success). Given this really low success rate, should you even do Big Data?

When I worked at a Big Data vendor, I couldn’t tell people they’re going to fail. As I interacted with their team, I could see they were going to fail. I couldn’t bite the hand that fed me. As a result, the companies wasted millions in projects that went absolutely nowhere.

Without specific changes, a half-assed project will go nowhere. I see it all the time because companies reach out to me for help in mentoring their data engineering teams.

The message is usually something like:

Hi we’re a large enterprise and I’m managing the data team. We have a company-wide mandate to improve our data infrastructure to handle the increase in customers we’re expecting.

We’re having trouble making progress. We thought our project would take 6 months, but we’re already 9 months into the project and have nothing to show for it.

Our upper management is starting to ansy. Our business side is getting more vocal asking where their promised upgrades are and when they’re going to get something.

How can you help us?

There are a few meanings to this email:

  • What is the cheap, quick, and easy fix to get out of this bind?
  • Can you give us some pointers to get back on track?
  • We need help and we need your help.

A cheap, quick, and easy fix isn’t possible. A company seeking these solutions isn’t going to succeed because they’ve cheaped out and cut corners at all levels.

I know this because I follow up with these companies months after they send me an email. The answer is always the same. “We’re still figuring things out.” Put another way, 6 months have gone by, they’ve wasted another million dollars, and they don’t have anything to show for it.

When a company looks for some pointers, I send them my Data Engineering Teams book. The book shares my advice on how to staff and create data engineering teams. It shows how to run successful projects.

The teams that really want to guarantee their success, get mentored directly by me.

If you aren’t committed, should you even do Big Data? No. Save your time and money. Do something else that you can half-ass your way through. The risk/reward just isn’t there and your money is better spent redoing the graphics on your website.

Related Posts

The Difference Between Learning and Doing

Blog Summary: (AI Summaries by Summarizes)There are several types of learning videos: hype, low effort, novice, and professional.It is important to avoid hype, low-effort, and

The Data Discovery Team

Blog Summary: (AI Summaries by Summarizes)The concept of a “data discovery team” is introduced, which focuses on searching for data in an enterprise data reality.Data

Black and white photo of three corporate people discussing with a view of the city's buildings

Current 2023 Announcements

Blog Summary: (AI Summaries by Summarizes)Confluent’s Current Conference featured several announcements that are important for both technologists and investors.Confluent has two existing moats (replication and

zoomed in line graph photo

Data Teams Survey 2023 Follow-Up

Blog Summary: (AI Summaries by Summarizes)Many companies, regardless of size, are using data mesh as a methodology.Smaller companies may not necessarily need a data mesh

Laptop on a table showing a graph of data

Data Teams Survey 2023 Results

Blog Summary: (AI Summaries by Summarizes)A survey was conducted between January 24, 2023, and February 28, 2023, to gather data for the book “Data Teams”

Black and white photo of three corporate people discussing with a view of the city's buildings

Analysis of Confluent Buying Immerok

Blog Summary: (AI Summaries by Summarizes)Confluent has announced the acquisition of Immerok, which represents a significant shift in strategy for Confluent.The future of primarily ksqlDB

Tall modern buildings with the view of the ocean's horizon

Brief History of Data Engineering

Blog Summary: (AI Summaries by Summarizes)Google created MapReduce and GFS in 2004 for scalable systems.Apache Hadoop was created in 2005 by Doug Cutting based on