Forget About Batch Processing – Centerfield Streams Data In Real-time

Our Data Warehouse (DW) team has been hard at work the last few months to further improve the speed of our operational reporting. The quicker our business teams have actionable data at their fingertips, the more we can optimize our operations, from marketing through to the sales center. 

Prior to this initiative we utilized batch processing across multiple systems to transfer data to our data warehouse, which typically took several minutes to compile and present call, sales, and agent data. Now, thanks to the incredible efforts of our awesome DW team, Centerfield can stream this data in real-time – in just over 200 milliseconds on average. Can you say fastball?!

We can now make near real-time decisions in optimizing marketing spend, call center operations, workforce management, and sales operations. Call reporting is now available in just 298 milliseconds on average. Order reports are available in as fast as 200 milliseconds, and agent session reports are available in just 140 milliseconds. We’ve also been able to integrate this capability with Tableau.

Our DW team was able to do this with the help of various technologies and software products, including Rabbit MQ, Talend, Amazon S3, MySQL, MongoDB, and SQS. Teams have automated alerts in place to notify them of any discrepancies and to ensure data is consistently being processed as fast as possible. A reconciliation process was also implemented to make sure no data gets left behind.

“Data Warehouse is typically used for historical purposes, but no one is really using it for real-time reporting. We want to change that,” said Santosh Gumidelli, Director of Data Warehouse and Architecture. “Our goal is to make Data Warehouse an active, real-time component of our reporting and optimization processes.”

Thanks to the ongoing success of this initiative, our DW team is working on applying these methods to other data transfers that still utilize batch processing. Eventually, the goal is to fully automate the agent selection process to prioritize top-performing agents that are more likely to convert customers and close sales of specific products and services.