Managed automated set top box (STB) testing
Testing and validating video platform changes through the STB
If you’re like most operators, the set top box is probably the most important way end-users access your TV or video service. For these end-users, the complexity of the technology underpinning your video delivery platform is largely invisible and unimportant. Their perception of the quality of your service is decided by the experience of interacting with that service through one device: the STB. This makes the set top box the most relevant place to test the effects of all new releases and changes taking place throughout the entire video delivery chain, beyond the obvious firmware upgrades or middleware updates. Through STB test automation, you will detect DNS or connection problems, rebuffering, unavailable VOD assets, authentication issues and more – before your end-user calls in.
Set top box test automation introduces speed and consistency
Vendors like Witbe, Accenture and Eurofins offer automated STB test technology that can consistently and continuously execute test sequences on your set top box or streaming devices. More so than manual testing – which can be labor-intensive as well as error-prone – automated STB tests ensure that your end-user quality of service isn’t compromised when change takes place in the underpinning video delivery solution. STB test automation allows you to run progression tests to ensure your release or change actually fulfils its intended purpose. At the same time, automated STB testing technology will continuously run through scenarios that mimic end-user behaviors in all its many variations, as regression tests detect, identify and highlight unintended consequences before end-user impact takes place. As a result, your developers or vendor-partners are able to fix the underlying issue before your end-user quality of experience is affected.
Understanding QoE through automated DVB, OTT and IPTV testing
Beyond determining if your TV service works as intended, automated testing can help you determine and quantify your end-user’s quality of experience. KPI’s captured by STB testing robots include:
- Channel availability and channel change time;
- Audio and video quality, quantified through objective scoring;
- Video start-up time for VOD, PVR/nPVR and time-shift;
- Asset availability for VOD, PVR/nPVR and time-shift;
- UI user response time;
- Video player buffering, and more.
This allows DVB, OTT or IPTV testing to provide operators with a comprehensive understanding of their end-user’s quality of experience (QoE).
Scripting solved, through managed automated STB testing
Even though advances in technology have allowed STB test automation to introduce massive efficiencies in video platform testing operations, there are still some areas where domain expertise and manual processes are required. Specifically, the STB test scripting process can be laborious and seemingly endless: Changes to the UI and underlying features/functions continuously necessitate changes in most – if not all – scripts.
Divitel’s Managed Automated Set Top Box Testing service doesn’t just provide the necessary technology, we also take care of scripting and planning, leveraging our skills, scale and technology to unburden you from unnecessary and non-strategic operational processes. This includes unmetered access to test scripters that will employ Divitel’s scripting knowledge base to rapidly create, update and improve highly effective STB testing scripts for your devices. Tests that will run on a locally deployed STB test automation robot – or robots – provided by Divitel, placed within your network. In addition to access to scripting resources, the managed automated STB testing service provides you with your own dedicated test engineer, who will plan and execute tests for you remotely.
STB test dashboarding, reporting and benchmarking
Divitel’s Managed Automated Set Top Box Testing service is a fully managed service delivered remotely from our Video Application Lifecycle Testlab facility. From this high-tech environment, scripting and planning engineers ensure that test scenarios run on your locally deployed test automation robot. Output from these efforts is presented to you in clear and actionable real-time dashboard that relay test-performance, as well as quality of service (QoS) KPI’s, going back 12 months. Output that can also be connected to your operational intelligence platform or logging tools like Splunk. Additionally, on a monthly bases your customer success manager will engage you in strategic sessions around your QoS in the previous month and the planning for the coming month. During these sessions, you will receive additional reporting in the form of your performance versus industry benchmarks on several KPI’s, allowing you to assess your service’s performance versus the competition’s average.
The economics of managed automated testing for STB
Automated testing allows you to address video service quality and performance issues before they negatively impact your end-user experience. It does this much faster and more consistently than similar manual efforts, reducing first-line support issues and improving Quality of Service, which in turn will positively impact your service’s customer retention.
Our managed automated STB testing service doesn’t require you to invest in acquiring test automation technology. For the duration of the service, we will provide you with your own dedicated test automation robot on premise. As the orchestration and reporting platform used to manage your tests remotely is multi-tenant, the resulting benefits of scale allow us to greatly improve the economics of this proposition for TV operators of any size. In – almost – all cases, the investment associated with this fully managed service is significantly lower than the cost associated with acquiring the necessary technology, let alone finding and hiring the people needed to operate it.
More information on managed automated testing for TV operators
Want to discuss how managed automated testing of your STB can improve your end-user quality of service? Our consultants look forward to exploring how managed testing can address your video business challenges. Send an email to [email protected] or click the ‘request more info’ button below and fill out the form on this page. We’ll get back to you within 24 hours to provide you with additional information, schedule a call or schedule a demo of our STB test automation technology.
Request more info
AI set top box (STB) test scripting and interpretation
Set top box testing technology is able to handle a growing share of possible end-user use cases, without relying on human assistance to execute the necessary test scenarios. For many operators, a new bottleneck has appeared, though:
The set top box test scripting and interpretation bottleneck
Writing the test scripts used for automated testing of video services requires skill, experience and near-real time understanding of end-user behaviors as they develop. While technology from vendors like Witbe, Accenture or Eurofins handles rapid execution of varied and complicated test scenarios, the technology still needs to be told what to do and what to look for. Scripts are dynamic and undergoing change constantly, as the underlying platform changes and user behaviors develop. The production of the necessary scripts is specialized, often time-consuming work that requires trained experts. Additionally, as the underlying video delivery platforms become increasingly complex, interpreting the output of tests requires a lot of time and expertise.
Automated test automation scripting
As clunky as the description may sound, automated test automation scripting addresses a pressing issue for TV operators that are using set top box test automation. By employing machine vision and other forms of artificial intelligence (AI), Divitel is aggressively developing the technology needed to automate the scripting process. Technology that is able to understand real-life usage of your television service and automatically turns it into highly effective test scripts, continuously. Additionally, AI assists in keeping existing scripts up to date as small changes to the UI or other service characteristics render scripts unusable.
Interpreting test automation results
As set top box test automation technology develops, the data these tools generates becomes ever-more voluminous. In addition to simple pass-fail information, KPI’s like channel-change time, VOD asset availability, OTT player buffering and object audio/video quality tests produce an – often noisy – sea of information. Today, Divitel’s managed automated set top box testing service employs skilled engineers to interpret all this data. Increasingly, these same engineers are able to lean on machine learning and other AI technology to assist in anomaly- and root cause detection. Alternatively or additionally, the data generated can be fed into an operational intelligence platform for further dissemination and interpretation. The result is a great increase in speed and a reduction of needed resources.
The economics of AI set top box (STB) test scripting an interpretation
AI set top box test scripting allows you to rapidly develop and iterate on the scripts necessary for automated testing of TV and video services. The speed at which you can execute tests on new scenarios increases, as downtime related to rescripting based on UI and feature changes decreases dramatically. On the road to near-continuous deployment of updates and changes without unintended end-user impact, automated scripting is an absolute necessity.
Divitel supports video service operators with scripting and interpretation of data related to the automated testing of set top boxes. By leveraging skilled engineers, a growing knowledge base and increasingly assisted by artificial intelligence, this service is a highly effective and economical alternative to inhouse solutions.
More information on AI set top box test scripting and interpretation
Want to discuss how AI set top box test scripting and interpretation can speed up your development cycle, while improving the quality and economics of your own automated testing efforts? Send an email to [email protected] or click the ‘request more info’ button below and fill out the form on this page. We’ll get back to you within 24 hours to provide you with additional information or will happily schedule a call to discuss your specific challenges.
Request more info
Managed automated OTT video service and application testing
The problem with mobile video service applications
The quality of mobile video service applications is oftentimes below par. To see just how bad most applications are faring, one only needs to look at the iOS or Google Play app stores:
The average customer rating for entertainment applications is (much) lower than the average for applications overall. Within entertainment, video applications fare much worse then – for example – gaming, with many renown operators and broadcasters seeing app-ratings that average out below 3 or even 2 stars. The amount of 1-star ratings are a symptom of a problem plaguing most video applications out there today. Just check out a small sample of customer reviews for a large British operator. Keep in mind, these reviews haven’t been filtered: You’re getting the last 6 reviews out of the total, not just the ‘bad’ ones. For this particular app, 15 out of the 20 most recent ratings were 1-star.
OTT video service and application development and operations
Partly, misunderstandings surrounding the mobile application operations process are to blame. Many TV Everywhere services, OTT SVOD applications and other video content apps are developed externally or bought ‘out of the box’ from vendors. After their initial release, operators, broadcasters and content owners will usually have an SLA with the developer for some minor patches, but there is no process in place to continuous develop and evolve the application as time goes on. The result is slow time to market with much-have features (for fun, count how many 1-star ratings for video applications are related to a lack of Chromecast support), but also an inability to rapidly respond to user-experience issues as they are reported by the end-users.
The challenges of Android device and OS fragmentation
End-to-end OTT solutions address many of the challenges involved in developing and operating these OTT video or TV everywhere services. There’s another important driver of customer-dissatisfaction with these applications and services, though: Releases go live without (adequate) testing. The reason for this lack in proper testing is twofold. On the one hand, use-cases are increasingly complex as the capabilities of the applications involved, grows and expands. Even more pressing though, is the issue of fragmentation:
In 2015, over 20.000 different devices were running on Android. Devices that run on wildly different hardware and often sport different aspect ratios and different resolutions. Not only do video application operators need to test against (a good portion) of these different devices; There’s also massive fragmentation in the versions of Android end-users are actually running. A few months back, Dan Luu wrote this excellent article on the state of Android update uptake. Over 1 billion (!) devices are currently running Android versions that are 2 or more years old. Testing all these combinations of device and OS is impossible, but if you want to safeguard your quality of experience for even half of your user base, you’ll probably have to test – at least – dozens.
TV everywhere and streaming video app testing and the cloud
The answer to the challenges involved in testing TV everywhere and streaming video services and apps, is automation. Increasingly intelligent test automation technology allows Divitel to rapidly run through countless test scenarios, closely mimicking real life behaviors on the physical devices being used by your end-user. To ensure that this automated testing actually detects issues that would occur in the real world, we’re connecting to device clouds around the globe. These device clouds offer cloud-based access to a wide range of different devices featuring different configurations. As such, we’re able to pick the configurations needed to test for an acceptable share of your user base (usually 75% or over) and employ these on a ‘pay per use’ bases.
The result? A dramatic reduction in complaints and negative user reviews related to non – or barely – functioning OTT video service applications.
The benefits on managed automated OTT video service and application testing
To power our managed automated OTT video service and application testing efforts, Divitel continuously evaluates, selects and integrates the most effective test automation technology. This ensures that we can optimally assist broadcasters, content owners and other video service operators, without the need for clients to invest in expensive equipment and specialists. Our customers can count on the support of a dedicated customer success manager, on-demand automated testing resources and ready-to-use insights related to the test-performance of their applications and services.
In addition to simple pass-fail test results, this managed service provides clients with invaluable insights into the customer’s quality of experience:
- Video streaming quality;
- OTT live streaming QA;
- OTT player buffering;
- VOD asset testing (availability, load-time);
- Video streaming performance testing;
- IPTV stream testing;
- Jitteriness, stuttering, lag and more.
The economics of automated OTT video service and application testing
Better application reviews translate directly into better conversion on your user acquisition efforts. The difference in average app store conversion between a 2-star and a 4-star application is a whopping 540%.
Divitel’s managed automated OTT video service and application testing ensures your releases meet the necessary quality standards and work on the devices your users have. This leads to better reviews, fewer complaints, fewer customer care interactions and higher retention for your service. At the same time, by offering a fully managed service, the necessity to acquire testing technology and hire specialists disappears, introducing efficiencies that translate into lower cost of ownership for your service.