One of the benefits of teaching and consulting is the sheer number of organizations, teams, and people I get to work with. Since I deal with so many different groups, I can see patterns emerge much faster than others.
One pattern I saw early on was real-time Big Data. Organizations wanted to do things in real-time. Teams had projects that required real-time. People had ideas the required real-time systems.
And we couldnâ€™t do it
We lacked the systems that could scale to the sizes and amounts of data needed. As a direct result, we had to do terrible workarounds.
As I work with my clients around the world, theyâ€™re moving from batch processing to real-time processing. They tell me the stories about how they wanted to do real-time, but could only approximate the system in batch.
Let me share one of their stories.
One large financial company was feeling the need for real-time processing. The use case required real-time, but the project was started at a time when real-time Big Data wasnâ€™t feasible. As a result, they had to go with batch processing over 24 hour windows. This didnâ€™t meet the needs of the business, but that was that was possible.
They would try to lower their batch sizes to be smaller and smaller time windows. What started as a 24 hour batch window gradually decreased down to 30-60 minutes. The business was all over the team to turn around the data faster and faster. It wasnâ€™t acceptable to be 24 hours behind.
But the team couldnâ€™t go any lower than 30-60 minutes. Going to that small of a batch window caused all sorts of operational headaches. The systems just couldnâ€™t keep up with the demand and the operations team crumbled.
The business wanted to be no more than a minute behind whatâ€™s currently happening. There was nothing more that the team could do. They had to move to a real-time system.
I mentored the team on their transition to real-time. They could finally accomplish their original use case and its requirements.
Now we have the systems that can scale and do real-time Big Data.
As I work with these teams on their moves to real-time, theyâ€™re able to circle back with the business and actually deliver. This is the part that I love. I love being able to remove the pain that a web of terrible workarounds causes and producing a resilient real-time system.
This is why real-time is the future. The business and use case wanted real-time processing. We as data engineers can deliver it now.