AI meets Live TV testing for a leading telecoms provider
Challenge
Thousands of daily Live TV tests created heavy manual work, with even a small failure rate causing delays and relying on specialist knowledge.
Approach
We applied AI to automatically analyse recordings, logs and screenshots, providing clear, actionable insights for every test.
Results
The system now processes 350–400 tests per day with 95% accuracy, quickly identifying root causes and cutting manual effort.
Challenge: Test failures slowing QA
Leading telecommunications and media provider
A leading telecommunications and media provider delivers live TV and on-demand services across mobile and connected devices. To protect the quality of this experience, the organisation runs large-scale automated testing across its platforms.
A critical challenge emerged when testing Live TV playback on Android devices. Occasionally, instead of video content, an error message appeared on the device. The main difficulty was not any single type of failure but the sheer volume of results to validate. Thousands of automated executions run daily, and even a low failure rate translates into significant manual effort.
Failures could arise from a variety of causes, including unexpected content, new application versions, DRM (Digital Rights Management) system updates, account changes, or visual discrepancies. Investigating these failures required engineers to manually review logs, switch between USB recordings and camera footage and interpret low-quality images. This process was time-consuming and relied heavily on specialist knowledge, making the management of large-scale validation the central challenge.
Our approach: AI-powered test analysis
Resillion enhanced its Test-as-a-Service (TaaS) platform with AI-driven test result analysis, building on earlier innovation work with Google.
The solution combines:
- Automated interaction with real devices via USB
- External camera capture to observe DRM-protected playback
- AI analysis of recordings, screenshots, logs and timelines
An AI-enabled analysis layer was introduced to automatically interpret failed test executions and present clear, actionable insights.
Key capabilities include:
AI-generated step descriptions
Each test step is summarised to confirm expected versus actual behaviour.
Dynamic error detection
In this case, the AI detected and extracted the on-screen error message from the camera recording:
Clear rationale and evidence
Relevant timestamps, error messages and reasoning are surfaced immediately, guiding users directly to the right artefacts.
Side-by-side result comparison
The original automation outcome is shown alongside the AI’s analysis, removing ambiguity and speeding up validation.
Examples
Real failures
DRM licence errors, playback errors, page loads, blocked flows
Environmental issues
Network issue, device noise, OS issues
Automation errors
Element-not-found, flaky tests, script errors, validation issues, missing assertions
Data issues
Wrong user state, invalid setup, invalid/not recognised live data
False positives
Motion detection false alarms, navigation issues, incorrect KPI measurements, failed assertions
False negatives
Missing assertions, misinterpreted playback
What was previously an unclear failure is now turned into a clear diagnosis within seconds.
Results: Faster root cause detection
The test result analysis now runs at 95% accuracy, with Resillion building monitoring, training and AI capabilities directly into the client’s workflows and supporting the team.
The AI-enabled solution delivers clear operational and business benefits:
Faster failure investigation
Engineers move directly to root cause instead of manually checking recordings and logs.
Reduced reliance on specialist knowledge
Clear AI explanations allow the wider team to support with the test analysis without extensive training or handovers.
Quicker issue resolution
Any errors are spotted immediately, helping teams focus on the correct systems and owners.
Highly scalable and cost-efficient
- Daily analysis handles 350–400 test results on average, scaling up to 600 on busier days.
- Analysis is completed in minutes through horizontal scaling.
- Operating costs are approximately £20 per day, with potential for further cost reduction as model optimisations continue.
Human validation is still part of the workflow today, with a view to automate in the future.
Why Resillion
Resillion has strong experience in media quality testing, DRM-protected environments, and large-scale automation. This expertise lets us solve problems that traditional test automation can’t.
By combining our Test-as-a-Service platform, AI-powered analysis, and cloud-based scalability, we help clients deliver high-quality, reliable services quickly — even as platforms and DRM protections change.
