Kafka Meets Scale

This is the story of the streaming real-time inventory management system that we developed in Walmart. With a presence of more than 11000 stores throughout the world, there are not many companies that have the challenges of scale and volume of data as Walmart today. The heart of our retail business is “Inventory”, key to stock replenishment and other decision making in the company. Adding to this we had a requirement to enable all core systems and users to do real-time tracking of items moving within the supply chain. To fulfill these requirements, we built an event-driven system that tracks inventory in near real-time, with events at the heart of the system. We chose Kafka as our pipeline, Cassandra to persist hot data, and several other data stores for cold storage and visualizations. In this approach we have a stream of events flowing through Kafka from multiple sources being aggregated and re-aggregated at various levels of item groupings, feeding into Cassandra and Kafka for downstream consumers.

To efficiently deal with the volume while ensuring no data loss and accurate calculations, we optimized every component and application in the workflow. A lot of lessons were learnt on:

- how to leverage Kafka and Cassandra at scale,

- how to produce and consume from Kafka in an optimum manner,

- how to efficiently aggregate data in chronological item sequence using Cassandra while ensuring no dirty reads or data loss,

- telemetry and real-time audits provide the necessary health and quality checks for the system,

- building various analytics and ML models to detect and fix anomalies in the system.

We look forward to meeting you and discussing in detail our journey and connecting you for solutions to some of your problems.