The Veteran Skill on a Data Engineering Team

Blog Summary: (AI Summaries by Summarizes)
  • A veteran skill is often overlooked and unknown to data engineering teams.
  • A project veteran is someone who has worked with Big Data and has had their solution in production for a while.
  • A veteran brings a great deal of experience to the team and can prevent difficult code, overly complex systems, and production problems and outages.
  • A junior engineer knows what to do, but a senior engineer knows why you should do it.
  • A data engineering team without a veteran is in a precarious position and it really isn't an option to continue a project without experienced feedback.

In my book Data Engineering Teams, I talk about a skill that’s often overlooked and unknown to data engineering teams. Teams often don’t know they need a veteran, think they can’t afford a veteran, or don’t understand why you need a veteran on the team.

In Chapter 3 “Data Engineering Teams”, I give my definition of the veteran skill on a data engineering team:

A project veteran is someone who has worked with Big Data and has had their solution in production for a while. This person is ideally someone who has extensive experience in distributed systems or, at the very least, extensive multi-threading experience. This person brings a great deal of experience to the team.

Sometimes when I mentor a data engineering team, they’ll give various reasons why they don’t have a veteran. I’ll go through these reasons and this will help you understand why a veteran is so important.

Teams That Don’t Understand Why Or Think They Need a Veteran

Sometimes, teams will have a misunderstanding of what a veteran does on the team. They’ll think that the veteran is a the curmudgeon that stops all of their reindeer games. The veteran is there to stop them from doing any new or experimental. This stick-in-the-mud just says no to everything.

There are times when the veteran needs to tell you that you’re being stupid. This is an aspect that separates the good veterans from the bad/mediocre ones. The good veterans will tell you how to get the same or similar thing done a different and better way. A good veteran has the experience to draw on to point to the better option that still gets the job done. A bad or mediocre veteran will just it can’t be done.

To give you an example in my work with teams, I’ve had junior- to mid- level engineers spend 2 months on a project. They implemented the pipeline the way a person without experience would. They had spent 2 months creating an overly complex system that would be difficult for another person to maintain the code, error prone in production, and add unnecessary complexity to a system. I spent an hour with the engineer and reduced the complexity and code by 90%. The system would work vastly better in production. Remember that the lack of a veteran doesn’t just result in difficult code and overly complex systems. It results in production problems and outages. Unless you’re a developer, you won’t know how much of your code is held together with duct tape and hope. Part of a veteran’s value is to sniff out and prevent these abuses.

My definition of the difference between a junior and senior is that a junior engineers knows what to do, but a senior engineer knows why you should do it. When a person only knows what to do, they’ll use and abuse technologies that aren’t meant for the job. A person who knows why will choose the best tool for the job based on experience. A veteran knows why a tool or architecture is the best route and why.

When You Can’t Afford or Find a Veteran

A data engineering team without a veteran is in a precarious position. All of the team’s pipelines, architectures, and systems are suspect. It really isn’t an option to continue a project without experienced feedback.

Some teams will try to cultivate experience in their existing teams. This can be difficult and time consuming. For these longer periods of cultivation, retention becomes an issue as the team member becomes more attractive to other companies looking for their veteran. Other teams have difficulty hiring a veteran. This often stems from low ball offers and unattractive teams/companies/projects/etc. Take an honest look at why you’re not getting anyone and try to fix that. Sometimes HR (Human Resource) people don’t understand that hiring a veteran isn’t the same as hiring a lead or senior Software Engineer. These skills are in demand and qualified veteran Data Engineers don’t need to settle.

I’ve seen this to be a common enough issue with data engineering teams that I have different levels of engagement to mentor teams. These range from a short-term architecture review to a long-term mentoring relationship. I step in and act as the team’s veteran and more.

Related Posts

zoomed in line graph photo

Data Teams Survey 2023 Follow-Up

Blog Summary: (AI Summaries by Summarizes)Many companies, regardless of size, are using data mesh as a methodology.Smaller companies may not necessarily need a data mesh

Laptop on a table showing a graph of data

Data Teams Survey 2023 Results

Blog Summary: (AI Summaries by Summarizes)A survey was conducted between January 24, 2023, and February 28, 2023, to gather data for the book “Data Teams”

Black and white photo of three corporate people discussing with a view of the city's buildings

Analysis of Confluent Buying Immerok

Blog Summary: (AI Summaries by Summarizes)Confluent has announced the acquisition of Immerok, which represents a significant shift in strategy for Confluent.The future of primarily ksqlDB

Tall modern buildings with the view of the ocean's horizon

Brief History of Data Engineering

Blog Summary: (AI Summaries by Summarizes)Google created MapReduce and GFS in 2004 for scalable systems.Apache Hadoop was created in 2005 by Doug Cutting based on

Big Data Institute horizontal logo

Independent Anniversary

Blog Summary: (AI Summaries by Summarizes)The author founded Big Data Institute eight years ago as an independent, big data consulting company.Independence allows for an unbiased