Modulai joined forces with Abios Gaming to build an object detection and OCR solution for extracting real-time information from gaming tournament video streams. The system tracks weapons, tools, and player interactions to provide a structured event stream, enabling Abios to generalize their solution to new game titles faster and more reliably.
Challenge
Abios Gaming provides an API for live information on games, teams, and players in esports tournaments. To strengthen their offer, Modulai joined their tech team to build an object detection and OCR solution for extracting information on the fly from real-time video streams of gaming tournaments.
Solution
The system detects and tracks timestamped events corresponding to the objects players possess or use, providing a structured stream of events that captures essential information about the tournament's evolution. In the absence of large annotated image datasets, the core of the project was to create a robust pipeline for generating synthetic datasets with images and annotations, and to train a neural network for improved detection, localization, and semantic segmentation.
Tools
A pipeline for synthetic data generation was developed to solve the object detection task. Several object detection implementations in PyTorch and TensorFlow were tested and tuned. The final solution was based on a Mask R-CNN that substantially improved the existing classical computer vision approach, enabling the client to generalize to new game titles more simply and quickly.
Value created
Modulai built and handed over the detection pipeline for Abios to run and extend themselves. Built on a Mask R-CNN trained on synthetically generated data, it substantially outperformed the classical computer vision approach Abios had used before, and because the synthetic data pipeline removed the need for large hand-annotated datasets, Abios could bring the system to new game titles far faster than before. The approach went on to support computer vision data across multiple esports titles in Abios' live API.
Learn more
