Should You Even Do Big Data?

There’s an elephant in the room with Big Data. If an organization tries to half-ass their way through a Big Data project, they’re going to fail (usually a 5-10% odds of success). Given this really low success rate, should you even do Big Data?

When I worked at a Big Data vendor, I couldn’t tell people they’re going to fail. As I interacted with their team, I could see they were going to fail. I couldn’t bite the hand that fed me. As a result, the companies wasted millions in projects that went absolutely nowhere.

Without specific changes, a half-assed project will go nowhere. I see it all the time because companies reach out to me for help in mentoring their data engineering teams.

The message is usually something like:

Hi we’re a large enterprise and I’m managing the data team. We have a company-wide mandate to improve our data infrastructure to handle the increase in customers we’re expecting.

We’re having trouble making progress. We thought our project would take 6 months, but we’re already 9 months into the project and have nothing to show for it.

Our upper management is starting to ansy. Our business side is getting more vocal asking where their promised upgrades are and when they’re going to get something.

How can you help us?

There are a few meanings to this email:

• What is the cheap, quick, and easy fix to get out of this bind?
• Can you give us some pointers to get back on track?
• We need help and we need your help.

A cheap, quick, and easy fix isn’t possible. A company seeking these solutions isn’t going to succeed because they’ve cheaped out and cut corners at all levels.

I know this because I follow up with these companies months after they send me an email. The answer is always the same. “We’re still figuring things out.” Put another way, 6 months have gone by, they’ve wasted another million dollars, and they don’t have anything to show for it.

When a company looks for some pointers, I send them my Data Engineering Teams book. The book shares my advice on how to staff and create data engineering teams. It shows how to run successful projects.

The teams that really want to guarantee their success, get mentored directly by me.

If you aren’t committed, should you even do Big Data? No. Save your time and money. Do something else that you can half-ass your way through. The risk/reward just isn’t there and your money is better spent redoing the graphics on your website.