Brief History of Data Engineering

Brief History of Data Engineering

by Jesse Anderson | Dec 13, 2022 | Blog, Data Engineering | 0 comments

In the beginning, there was Google. Google looked over the expanse of the growing internet and realized they’d need scalable systems. They created MapReduce and GFS in 2004. They published the papers for them in the same year. Doug Cutting took those papers and...
Ten Years On – The Million Monkeys Project

Ten Years On – The Million Monkeys Project

by Jesse Anderson | Oct 13, 2021 | Blog, Data Engineering, Data Engineering is hard, Million Monkeys | 1 comment

I want to tell you a story about how my life changed. It wasn’t a cult, new religion, or programming language. A million monkeys changed my life. Ten years ago, I randomly recreated every work of Shakespeare. It was quite a project on the technical, news, and...
It’s Time to Change How We Manage Data Teams

It’s Time to Change How We Manage Data Teams

by Jesse Anderson | Feb 24, 2021 | Blog, Business, Data Engineering, Data Engineering is hard | 0 comments

As a distributed systems person, I’m used to figuring out how to spread a problem out to the most number of computers possible. Spreading out a problem lets me leverage my resources far better and faster. However, we’re failing to apply this optimization...
What It Looks Like When a Team Is Missing

What It Looks Like When a Team Is Missing

by Jesse Anderson | Oct 13, 2020 | Blog, Business, Data Engineering, Data Engineering is hard | 0 comments

Data teams require all of their parts to be complete and succeed. When one of the teams of a data team is missing, the other teams will suffer. Often, organizations or team members don’t understand what’s happening when a team is missing. They blame...
Announcement: Data Teams Is Out!

Announcement: Data Teams Is Out!

by Jesse Anderson | Sep 23, 2020 | Blog, Business, Data Engineering, Data Engineering is hard, Magnum Opus | 0 comments

I’m thrilled to announce that Data Teams: A unified management model for successful data-focused teams is available for purchase! My goal is to drive a real increase in the percentage of successful big data projects. Data Teams represents years of work and...
Kafka’s Got a Brand-New Poll

Kafka’s Got a Brand-New Poll

by Jesse Anderson | Sep 11, 2020 | Blog, Data Engineering, Data Engineering is hard | 0 comments

Kafka 2.0 added a new poll() method that takes a Duration as an argument. The previous poll() took a long as an argument. The differences between the two polls don’t stop there. You should know about the differences before porting your poll from a long to a...
« Older Entries
Twitter Linkedin Rss

© Jesse Anderson 2022

Join the Newsletter