It’s Time to Change How We Manage Data Teams

Blog Summary: (AI Summaries by Summarizes)
  • Centralizing innovation and strategy onto a small management team is a common practice in business and technology decision-making.
  • This approach ignores the potential insights and perspectives of individual contributors who are closest to the data and business problems.
  • Emergence is a management philosophy that involves getting out of the way and listening to the people closest to the data.
  • Companies like Stitch Fix have successfully implemented emergence by creating circumstances where people can do good work and getting out of the way.
  • Turning individual contributors into leaders is the exact process that data teams need to make in order for emergence to happen.

As a distributed systems person, I’m used to figuring out how to spread a problem out to the most number of computers possible. Spreading out a problem lets me leverage my resources far better and faster. However, we’re failing to apply this optimization to our own real-world management and innovation.

In business and technology decision-making, we’re centralizing the innovation and strategy onto just a relatively small part – the management team. We aren’t using the individual contributors to spread out the problem to more people to get more ideas, perspectives, and solutions to problems. We’re leaving some of the best and brightest – more importantly, people closest to the data and business problems – out of the conversation.

Much of the thinking and management philosophy comes from industrial age methods. With Agile, we’ve come to realize that we need to involve individual contributors more in planning. We’ll ask them how long something will take (scoping or story points). They can try to add technical tasks (technical debt) to their task list but don’t allow for new features or ideation. This process of ideation is left to a product manager or some other part of management.

Next come the data teams who try to replicate this industrial age top-down innovation. We’re thinking this is the same knowledge worker that needs to be told what to do and ignoring the crucial data aspects that make data teams unique. We’re still missing out on the insights that our individual contributors might have. When are we going to start listening to the people closest to the data? What insights could the data teams closest to the business problem have? If we aren’t listening or even asking these questions, we’re missing out on a considerable amount of brainpower.

While writing Data Teams, I started to find companies who did this. Their management got out of the way, and they started listening to the people closest to the data. I’ve started calling this emergence.

What does emergence look like in the real world? I set out to find that answer during my interview with Stitch Fix’s Brad Klingenberg (Chief Analytics Office) and Eric Colson (Chief Analytics Office Emeritus). Brad said, “The approach we’ve taken is much more like being a gardener, you just want to create circumstances where people can do good work and, occasionally you need to trim a branch back or make room for a new sapling, but generally you’re just trying to get the conditions right to then get out of the way.” You can read the rest of the interview in my Data Teams book. Stitch Fix created a site talking about how they cultivate algorithms.

I recently read David Marquet’s Turn the Ship Around: A True Story of Turning Followers Into Leaders. David’s thesis was that he turned a low-performing submarine by taking their followers (enlisted) and making them leaders. I realize the book about the U.S. Navy and submarines. However, I argue that the mindset shift of turning individual contributors (followers) into leaders is the exact process that data teams need to make in order for emergence to happen. He also shows that this process can work even when the stakes are as high as life and death on a submarine. If you can work with stakes that high, we know it can work for other last risky situations too.

This idea comes out in Aaron Dignan’s Brave New Work. A big theme of the book is that teams need a clear purpose. This purpose comes after better integration of the data teams and communication with a business. The data teams will directly see the value and results of what they’re doing. A direct view into results creates a virtuous cycle that will create more effective teams.

As we come out of the COVID era, we have to leverage our team’s talents to their maximum potential. By using a select few, we aren’t utilizing the entire team to our advantage. We have to start using the whole team for our innovation and strategy. It’s only then that we can use analytics and insights to their highest potential.

Related Posts

zoomed in line graph photo

Data Teams Survey 2023 Follow-Up

Blog Summary: (AI Summaries by Summarizes)Many companies, regardless of size, are using data mesh as a methodology.Smaller companies may not necessarily need a data mesh

Laptop on a table showing a graph of data

Data Teams Survey 2023 Results

Blog Summary: (AI Summaries by Summarizes)A survey was conducted between January 24, 2023, and February 28, 2023, to gather data for the book “Data Teams”

Black and white photo of three corporate people discussing with a view of the city's buildings

Analysis of Confluent Buying Immerok

Blog Summary: (AI Summaries by Summarizes)Confluent has announced the acquisition of Immerok, which represents a significant shift in strategy for Confluent.The future of primarily ksqlDB

Tall modern buildings with the view of the ocean's horizon

Brief History of Data Engineering

Blog Summary: (AI Summaries by Summarizes)Google created MapReduce and GFS in 2004 for scalable systems.Apache Hadoop was created in 2005 by Doug Cutting based on

Big Data Institute horizontal logo

Independent Anniversary

Blog Summary: (AI Summaries by Summarizes)The author founded Big Data Institute eight years ago as an independent, big data consulting company.Independence allows for an unbiased