In my book Data Engineering Teams, I talk about a skill that’s often overlooked and unknown to data engineering teams. Teams often don’t know they need a veteran, think they can’t afford a veteran, or don’t understand why you need a veteran on the team.
In Chapter 3 “Data Engineering Teams”, I give my definition of the veteran skill on a data engineering team:
A project veteran is someone who has worked with Big Data and has had their solution in production for a while. This person is ideally someone who has extensive experience in distributed systems or, at the very least, extensive multi-threading experience. This person brings a great deal of experience to the team.
Sometimes when I mentor a data engineering team, they’ll give various reasons why they don’t have a veteran. I’ll go through these reasons and this will help you understand why a veteran is so important.
Teams That Don’t Understand Why Or Think They Need a Veteran
Sometimes, teams will have a misunderstanding of what a veteran does on the team. They’ll think that the veteran is a the curmudgeon that stops all of their reindeer games. The veteran is there to stop them from doing any new or experimental. This stick-in-the-mud just says no to everything.
There are times when the veteran needs to tell you that you’re being stupid. This is an aspect that separates the good veterans from the bad/mediocre ones. The good veterans will tell you how to get the same or similar thing done a different and better way. A good veteran has the experience to draw on to point to the better option that still gets the job done. A bad or mediocre veteran will just it can’t be done.
To give you an example in my work with teams, I’ve had junior- to mid- level engineers spend 2 months on a project. They implemented the pipeline the way a person without experience would. They had spent 2 months creating an overly complex system that would be difficult for another person to maintain the code, error prone in production, and add unnecessary complexity to a system. I spent an hour with the engineer and reduced the complexity and code by 90%. The system would work vastly better in production. Remember that the lack of a veteran doesn’t just result in difficult code and overly complex systems. It results in production problems and outages. Unless you’re a developer, you won’t know how much of your code is held together with duct tape and hope. Part of a veteran’s value is to sniff out and prevent these abuses.
My definition of the difference between a junior and senior is that a junior engineers knows what to do, but a senior engineer knows why you should do it. When a person only knows what to do, they’ll use and abuse technologies that aren’t meant for the job. A person who knows why will choose the best tool for the job based on experience. A veteran knows why a tool or architecture is the best route and why.
When You Can’t Afford or Find a Veteran
A data engineering team without a veteran is in a precarious position. All of the team’s pipelines, architectures, and systems are suspect. It really isn’t an option to continue a project without experienced feedback.
Some teams will try to cultivate experience in their existing teams. This can be difficult and time consuming. For these longer periods of cultivation, retention becomes an issue as the team member becomes more attractive to other companies looking for their veteran. Other teams have difficulty hiring a veteran. This often stems from low ball offers and unattractive teams/companies/projects/etc. Take an honest look at why you’re not getting anyone and try to fix that. Sometimes HR (Human Resource) people don’t understand that hiring a veteran isn’t the same as hiring a lead or senior Software Engineer. These skills are in demand and qualified veteran Data Engineers don’t need to settle.
I’ve seen this to be a common enough issue with data engineering teams that I have different levels of engagement to mentor teams. These range from a short-term architecture review to a long-term mentoring relationship. I step in and act as the team’s veteran and more.
