Ability Gap – Why We Need Data Engineers

Blog Summary: (AI Summaries by Summarizes)
  • Big Data is a complicated field with many new technologies and changes within technologies that make it time prohibitive to keep up with.
  • There is an ability gap when it comes to Big Data concepts, where some people simply won't understand them on their best day.
  • The level of complexity in Big Data is vastly greater than in previous industry shifts, which makes it difficult for individuals and students to learn.
  • The ability gap isn't relegated to just non-technical or technical roles, as even salespeople can understand Big Data concepts with ease while some technical roles struggle.
  • Companies and managers should know that Data Engineers or Data Scientists are worth their higher salary because they don't have an ability gap.

I had a conversation with another person in the Big Data field. We were discussing whether the Data Engineers would become a more common job title and migrate out of Silicon Valley. I told him yes. Big Data is downright complicated on many levels. There are too many new technologies and changes within technologies where it is time prohibitive to keep up with. Engineers will need to specialize in Big Data and keeping up with it.

Ability Gap

As a trainer, I’ve lost count of the number of students and companies I’ve taught. One thing is common throughout my teaching, there is an ability gap. Some people simply won’t understand Big Data concepts on their best day.

Analysts usually talk about skills gaps when referring to a new technology. They believe it’s a matter of an individual simply learning and eventually mastering that technology. I believe that too, except when it comes to Big Data.

Higher Complexity

Big Data isn’t your average industry shift. Just like all of the shifts before it, it’s revolutionary. Unlike the previous shifts, the level of complexity is vastly greater.

This manifests itself in the individuals and students trying to learn Big Data. I’ll talk to people where I’ve figured out they have no chance of understanding the Big Data concept I’m talking about. They’ve simply hit their ability gap. They’ll keep asking the same question over and over in a vain attempt to understand.

Big Data for Everyone?

This issue isn’t relegated to just non-technical or technical roles. I’ve had people with the technical role of creating a Big Data solution hitting their ability gap in a sad way. I’ve had sales people able to converse and understand Big Data concepts with ease.

I’ve spent copious time talking to other thought leaders and instructors about this issue. Is the ability gap merely that an individual needs more time to process the information? In my experience, the answer is no. Is the issue that materials are presented too complicated? I explain Big Data concepts with Legos and playing cards in a way that even most laypeople can understand. Understanding anything past that is outside their ability.

The Need for Data Engineers

I write this piece for several reasons. One, I want companies and managers to know there is a valid reason why a Data Engineer or Data Scientist is worth their higher salary. They’ve shown that they don’t have an ability gap. Two, I want individuals to know adding Big Data to your resume will not be easy and may be outside of your reach. And that’s ok. Third, I want individuals to have realistic expectations in their abilities. Banging your head against wall to learn something outside your ability is a futile endeavor. Your time is better spent improving or learning something else.

Conversely, you shouldn’t take this piece as discouraging you from learning Big Data skills. My experience shows that a qualified instructor is the best way to learn these concepts. The instructor can realize a misunderstanding or a roadblock and help you through it. This will always be the best way to learn.

Do we need specialists in Big Data? I say a hearty yes! I see the manifestations of this need while teaching. The success or failure of Big Data projects hinges on qualified Data Engineers architecting and writing these solutions.

Related Posts

The Difference Between Learning and Doing

Blog Summary: (AI Summaries by Summarizes)There are several types of learning videos: hype, low effort, novice, and professional.It is important to avoid hype, low-effort, and

The Data Discovery Team

Blog Summary: (AI Summaries by Summarizes)The concept of a “data discovery team” is introduced, which focuses on searching for data in an enterprise data reality.Data

Black and white photo of three corporate people discussing with a view of the city's buildings

Current 2023 Announcements

Blog Summary: (AI Summaries by Summarizes)Confluent’s Current Conference featured several announcements that are important for both technologists and investors.Confluent has two existing moats (replication and

zoomed in line graph photo

Data Teams Survey 2023 Follow-Up

Blog Summary: (AI Summaries by Summarizes)Many companies, regardless of size, are using data mesh as a methodology.Smaller companies may not necessarily need a data mesh

Laptop on a table showing a graph of data

Data Teams Survey 2023 Results

Blog Summary: (AI Summaries by Summarizes)A survey was conducted between January 24, 2023, and February 28, 2023, to gather data for the book “Data Teams”

Black and white photo of three corporate people discussing with a view of the city's buildings

Analysis of Confluent Buying Immerok

Blog Summary: (AI Summaries by Summarizes)Confluent has announced the acquisition of Immerok, which represents a significant shift in strategy for Confluent.The future of primarily ksqlDB

Tall modern buildings with the view of the ocean's horizon

Brief History of Data Engineering

Blog Summary: (AI Summaries by Summarizes)Google created MapReduce and GFS in 2004 for scalable systems.Apache Hadoop was created in 2005 by Doug Cutting based on