Winning with Test Automation
Any successful TV service is reliant on the delivery of a consistent and high quality end-user experience. For a leading subscription VOD service, such as Top Up TV in the UK, the key to achieving this is to have a robust, yet efficient testing process.
Testing must ensure that the service will stand up to real-world usage on any supported set-top box, but the process must also be able to adapt to new, more complex service offerings.
Key Benefits Realised
- Consistent, reliable and repeatable test results
- Test team able to focus on more valuable test analysis
- Automation makes new test scenarios viable
- Automatic results correlation and reporting
For Top Up TV, this process originally relied on the manual execution of a large set of tests on every set-top box model that supported the service. These test spanned everything from channel changes to using menus to setting recordings on the PVR. While time consuming, full test coverage is essential to ensure the quality of the end-user experience.
Introducing Automation
MiriATE is a proven automated test system for set-top boxes and the services consumed through them. MiriATE is able to automatically interpret video and audio output, while controlling the set-top box inputs and environment. This ‘black box’ testing approach works on commercial versions of the set-top box, without any reliance on a de-bug interface.
So MiriATE performs and observes tests in the same way as a human – but faster, more reliably, consistently and without getting bored or tired. Importantly, this ensures that MiriATE can automate existing manual tests.
Mirifice worked with Top Up TV to review a set of their existing manual tests and identified that 60% were quick and easy to automate. Mirifice also helped Top Up TV with test scripting.
“We chose the MiriATE product to improve the reliability of rote testing and also to reduce the manual testing effort. We wanted to free up the team to focus on more valuable test analysis and problem investigation.”
Simon Dore, Chief Technical Officer at Top Up TV
Top Up TV is the only Sports, Entertainment and Movie subscription TV service in the UK on the digital terrestrial platform, Freeview. Launched in 2004, Top Up TV offers premium sport, films and entertainment content nationally. It currently offers a wide range of international and domestic TV and movies from global brands, including Warner Brothers, Disney, Universal and BBC Worldwide. Plus sport from Sky Sports 1, Sky Sports 2 and ESPN, with no annual contract and no requirement to buy bundled packages of channels or services. Sports packs will give subscribers live access to every televised Barclays Premier League and most Champions League matches, plus a vast array of other sports. For more information or to subscribe to Top Up TV, go to www.topuptv.com or call 08444 158158.
THE RESULTS?
For the test set considered in the first phase of automation at Top Up TV:
- 60% of tests were easily automated
- Removing 66.5 manual hours from each test cycle
- Reducing the cycle time to run this group of tests by 50%
- Improving the consistency and reliability of testing
By automating a proportion of manual tests, the team at Top Up TV has been able to focus on analysing test results and investigating problems, allowing them to improve productivity. Each MiriATE test run provides a ‘pass’ or ‘fail’ summary and a detailed results log. An engineer can analyse a problem using the results log and can also reliably re-create the problem by re-running the MiriATE test.
“By running manual tests in parallel with MiriATE we verified the accuracy and reliability of the system. MiriATE scripting proved flexible enough to handle diverse test scenarios and the system also delivered test results and data efficiently.”
Kieron Edwards, Director of Technology and Platform at Top Up TV
The reduction in test cycle time has allowed the team to identify problems earlier and to test devices thoroughly without any adverse impact on time-to-market. Automation has also made repetitive stress and stability testing a cost effective option.
Finally, by using the MiriATE system, the test team is able to concentrate on devising tests for more complex service usage scenarios and new features, ensuring a continued high quality of experience when using the Top Up TV service.
Moving forwards
Top Up TV has upgraded to the latest MiriATE version, to take advantage of new features, which enables even more complex tests to be automated. This is helping Top Up TV to retain its position as a leading VOD player in the UK market.