Regional police department · Critical infrastructure · Physical AI · MLOps
Testing a control-room communication platform with a robot that can read the radio's screen.
A regional police department evaluated a control-room communication platform that can carry radio over VoIP. It had to test the platform against traditional radio transmission, in every area and including its new features. The radio devices could not leave the department, so we put a robot arm in front of one. The robot presses the keys and a camera reads the radio's display. Getting that reading right was the hard part. We built a test set for the screen tracking and one for the OCR, iterated until the tracking held at an SSIM of 85% and the OCR read every menu text in the test set correctly, and did it in real time, inside a small container on the customer's own server.
Snapshot
- Client
- Regional police department (name withheld)
- Users
- Control-room staff and testers at the police department
- Product
- Robot arm and operation console for automated testing of a control-room communication platform that carries radio over VoIP
- Engagement
- January 2024 to April 2025
- Scope
- Robot control and operation console, screen tracking and OCR, test sets and model comparison, deployment in a small container on the customer's server
- Stack
- Robotics, G-code, OpenCV, Tesseract, EasyOCR, PaddleOCR, open-weight multimodal LLMs, MLflow, scikit-learn, Docker
The challenge
A regional police department had to test a control-room communication platform against traditional radio. Our part was the physical AI system behind the test: a robot, a vision pipeline and the MLOps that made it reliable.
Six problems followed, each from the one before:
The test rig
1Control-room platform under test
2Police server: our console and vision pipeline run here, in a container
3Robot arm: moves in three dimensions and presses the radio's keys
4Radio device: stays at the police department
5Camera: films the radio's screen
Commands to the arm, camera images back
Calls between platform and radio, in both directions
Most tests start in the platform, which calls the radio. In one case a radio device triggers a call through the software. The arm and the camera cover both.
Robot and console
A test needs a real radio to receive the transmission from the control room.
Robot and console
The radio cannot leave the police department, so a robot arm operates it on site.
Screen tracking and OCR
Most actions on the radio take several key presses through menus, not one click. The robot has to navigate, so a camera has to read the pixelated screen to know where it is.
Screen tracking and OCR
Radio-side actions, such as calling other devices through the platform, must be confirmed. The screen is where the result shows.
Screen tracking and OCR
The reading must be reliable: isolate the screen, read only on menu changes, get words and numbers right.
MLOps and deployment
All of it must run in under a second, on limited hardware.
Our approach: the robot and console
One test step
- 1
Command
The test asks for a key press or a number.
- 2
G-code
A library turns it into CNC movements in 3D.
- 3
Press
The arm moves and presses the key.
- 4
Look
The camera films the radio's screen.
- 5
Read
When the menu changed, OCR reads the text.
- 6
Check
The screen confirms what the radio did, such as a call placed.
One action on the radio takes many such steps through its menus. Typing a number or moving within a menu repeats steps 1 to 4 without a reading. Step 5 only runs on a menu change.
Robot and console
- 01
Put a robot arm in front of the radio
The arm presses the radio's keys and types numbers, so tests run without the device leaving the police department.
- 02
Turn test commands into G-code
A library translates test commands into G-code, which drives the robot's CNC movements in three dimensions.
- 03
Give operators a console
A web console, built with React and a Python backend, lets operators issue commands to the robot. It runs from a development device over SSH, or as a container on the customer's server.
The vision pipeline, layer by layer
The robot has to see the radio's screen for two reasons. Most actions take several key presses through menus, so it needs to know where it is. And actions started on the radio, such as a call to another device through the platform, are only confirmed on the screen. Everything ran in real time: the screen had to be read and answered in under a second, measured on the customer's own server, on limited hardware, inside a small container. A test step cannot wait for a slow reading. Each layer of the pipeline got its own test set and a score, and we iterated against that score.
From camera image to text
1 · Camera image
The whole radio. The number block must stay out of the cut-out.
Group 3
Call
2 · Screen cut-out
The cut-out is graded against the wanted one. SSIM of 85% counts as good enough.
3 · Text
Left and right words on one line are read separately.
Only a menu change triggers a reading
Typing a number
skip
Moving in a menu
skip
New menu
read
Moving in a menu
skip
Typing a number
skip
The menu texts shown are examples.
- 01
Isolate the screen
The camera sees the whole radio, and the radio's number block must stay out of the cut-out. We tried different OpenCV methods and ended up separating the screen from the rest of the image by colour, so the cut-out always holds the screen. We graded every method against the cut-out we wanted: first its image size, then its similarity, measured as mean squared error at pixel level and as SSIM, which also compares contrast and luminance.
- 02
Detect menu changes
The screen changes constantly. Typing a number or moving within one menu must not trigger a reading. Only a change of menu should. We tried different ways to track what changes on the screen and graded them on the same test set, until the calibration matched the bounds we wanted.
- 03
Find a threshold that holds
We needed a threshold that counts as good enough. Tuning it on our golden image dataset was easy. Making it work on new data was the hard part, because a threshold that fits the golden images can fail on a new camera frame. After several iterations, an SSIM of 85% turned out to be the sweet spot.
- 04
Read the menu texts
The radio has its own vocabulary. Some menu items contain numbers, which confused word recognition, and some lines hold a left and a right word that must be read separately. We fine-tuned Tesseract on this vocabulary and read each part of a line on its own. A CSV with the expected text of every screen scored each run for accuracy, and we iterated until Tesseract read every menu text in the test set correctly.
How the engines compared
Before fine-tuning, we compared the options on the same images, with different tilt, pixelation and unusual vocabulary, and tracked speed against accuracy. That showed how much work each one needs per image quality.
EasyOCR and PaddleOCR are deep-learning pipelines with a separate text-detection stage, which makes them heavier and slower on a CPU. Tesseract also uses a neural network, but it is much smaller and faster. A multimodal LLM works differently: it reads the screen directly, without a tracking and OCR pipeline.
Engine
Strengths
Limits for this project
Tesseract
Small, fast on a CPU, mature. Strong on clean, upright text.
Struggles with skewed, noisy or stylized text. Needs effort to fine-tune.
EasyOCR
Easy to use from Python. Good on real-world images.
Heavier because of PyTorch and slower than Tesseract on a CPU. Can struggle with dense digits.
PaddleOCR
Full pipeline with detection and recognition. Good on noisy, low-resolution images.
More moving parts and a heavier dependency tree.
Open-weight multimodal LLMs
Most accurate out of the box. Take the whole image, so screen isolation and OCR could be skipped. Menu-change tracking is still needed.
Too slow for the sub-second response we needed, on the customer's server.
Why a fine-tuned Tesseract
This was an MLOps problem more than a modeling one. The question was not which option reads text best in general. It was which one fits the hardware and the time limit.
The open-weight multimodal LLMs were the most accurate out of the box. We could have skipped the screen isolation and the OCR, because they take the whole image. They would still have needed the menu-change tracking. But measured on the customer's server they could not meet the sub-second limit. The other OCR engines were too slow as well. We discussed buying faster hardware to host a more capable LLM, and decided to stay with a fine-tuned Tesseract.
Business outcomes
Robot and console
Testing continued with the radios on site
No radio device left the police department. The robot operated it from commands.
Screen tracking and OCR
The screen is tracked reliably
The system finds the screen without the number block and reads it only when the menu changes.
Menu texts are read correctly, in real time
The OCR reads every menu text in the test set correctly and keeps up with the live test, including items with numbers and lines with two words.
MLOps and deployment
Decisions rest on recorded runs
Thresholds and model choices come from test sets and tracked runs, and the container stays small.
Test coverage
The full experience could be tested
The platform works in both directions. In one case a radio device triggers a call through the software. In most cases the software starts the interaction itself. The robot arm covered both, so the police department could test the platform as its users experience it.
What we learned
A threshold that fits your test data may not fit new data
Tuning against the golden images was easy. Making the threshold hold on new camera frames took the iterations, and it settled at an SSIM of 85%.
Give every problem a measurable target
Image similarity for the screen tracking and expected text for the OCR turned "looks right" into a score that each change could be checked against.
Let the constraints choose the model
The open-weight LLMs were the most accurate out of the box, but too slow for a sub-second response. With limited hardware and a small container, a fine-tuned Tesseract was the option that fit.
Design for where the system runs
The radios could not leave the police department, and the robot could only be reached locally. Deployment shaped the design from the start.
Working on something similar?
Tell us about the workflow. The first meeting is free.
Next case study →
FGS Global: One quality standard for an enterprise AI agent network.