Achieve IoT Success by Testing the Consumer Experience

by   |   April 20, 2016 5:30 am   |   0 Comments

Paul Bruce, API Product Marketing Manager, SmartBear

Paul Bruce, API Product Marketing Manager, SmartBear

Consumer devices connected to the Internet of Things (IoT) present exciting new opportunities for both consumers and businesses. But the IoT also presents completely new challenges in both technical design and system verification. IoT device infrastructure must be reliable, scalable, and safe. The best way to prove that your devices are prepared to meet this challenge is to perform comprehensive testing on both the hardware and software involved in the end-user experience.

Consider what makes a great IoT experience. The best kinds of experiences are those that are well thought out. Apple’s iPhone and Watch are examples of end-to-end design thinking, asking questions such as:

    • How will people be initially attracted to my IoT product?


    • Why will they want to stay on my platform across product replacements?


  • How will my new IoT device integrate with third-party services?


Great IoT products follow intentional user experience design. Things that are simple, relatable, and that consistently work as expected will always win the consumer’s heart. Beyond a reliable experience, users also need to feel safe, a concern expressed in reports indicating that security is a primary concern that consumer technologies will be facing in 2016.

Successful IoT experiences also come from strong brand associations. The Phillips Hue, Amazon Nest, and Nike+ all know that consumers carry expectations from one product associated with an established brand to new products. These IoT devices benefit from reputations that new IoT vendors just don’t have.

The Biggest Challenges in IoT

Security. Security is by far the biggest consumer issue for the IoT industry. It is not only difficult to ensure across a diverse technology stack such as with current IoT hardware and low-level data transmission protocols, but security also often takes a back seat to problems with compatibility, performance, and reliability simply due to a deferred effect on revenue.

Data privacy. Privacy is a primary concern for all industries now that “every company is a software company.” Breaches have devastating impacts not only on revenue, but also on consumers’ trust in brands. Forbes estimates that Home Depot will suffer a 6 percent drop in EBITDA going into 2017 on account of data breach remediation for attacks in 2014 and 2015. The lesson to IoT vendors is to update your security testing strategy regularly, exercise it regularly, and progressively update it to cover untested surface area.

Related Stories

The Quiet Revolution of the Internet of Things.
Read the story »

The Internet of Things and the Necessity of Fog Computing.
Read the story »

The Internet of Things: What’s Possible vs. What’s Practical.
Read the story »

The IoT: A Mess in the Making.
Read the story »

Universal integration. Universal integration with other devices and services between IoT vendors also will continue to affect consumer experiences, as we have seen in the largely proprietary home automation consumer market. It’s not enough for IoT vendors to innovate; they must provide a way to integrate into a shared environment of devices and services. One recent example of success in IoT integration this is the Amazon Echo, using open standards and web APIs to ignite third-party vendor support by Samsung, Spotify, Uber, Logitech, and other big consumer device and service providers.

Performance. Performance of IoT services has a considerable impact on how consumers feel about their experience. IoT consumer devices are often the focal point of frustration and failure due to back-end latency and poorly thought through infrastructure design. As performance expert Mark Tomlinson said, “If you think about gadgets particularly making rich use of a set of back-end APIs, people are opening up the gadget for the first time and boom, it’s going to start doing software updates, account creation, just to get set up.” In this case, spikes in traffic against back-end services leave consumers wondering why they paid for a spinning progress indicator.

Versioning. Versioning of services, particularly in APIs, is also a concern in the IoT products you deliver to consumers. Once devices are deployed, you must provide backward compatibility to the original device with every service update. The bigger your company brand, the more you need to worry about harming consumer perceptions by providing inconsistent or poorly maintained services. In most cases, this means that you must thoroughly regression test your APIs upon every change to either software or hosting configuration.

Side note: Why are the IoT and API Economies So Closely Related?

In a recent survey, over 44 percent software professionals say that IoT will fuel API growth in the next few years, and one out of every five say that they already support IoT experiences. Just as public API growth was largely fueled by mobile device growth rates between 2010 and 2015, we will also see a correlation to IoT device growth rates between now and 2020.

You need only look at the changes in user interface design between IoT devices and traditional software (screens) to understand why getting APIs right is so important to successful IoT experiences. Minimalism dominates hardware because the less you ship, the less can go wrong; this is as true for IoT devices as it is for the APIs supporting them: The smaller the surface area for design, development, testing, and monitoring, the easier it is to ship a faster and error-free experience.

Fortunately for consumers, changes in the IoT hardware market will affect API testing procedures less than they will affect device-specific development. There still will be the primary challenges of device testing, namely:

    • Battery life, power loss recovery, and power consumption


    • Unreliable network connectivity


    • Data integrity after power failure


    • Data security, both on device and over the wire


    • Functionality and performance


    • Failure handling with invalid data from server


  • Human unpredictability and device composability


Wherever there’s a need to integrate, APIs are involved. Testing your APIs thoroughly helps you isolate the cause of failures with IoT devices because you know where the problem isn’t.

How Great IoT Experiences Are Tested

If you want end-to-end quality in your IoT consumer experience, incorporating solid design principles in your hardware and software is a good start, but it’s not enough. You also need a testing strategy that proves success before you ship code or devices. A proper IoT testing strategy includes:

    • Real-user or consumer success criteria


    • A back-end services (i.e., API) testing strategy with functional and regression testing, performance verifications, and security checks


    • A well-known set of manual tests to cover the majority cases of real use


    • Explorative testing to uncover unanticipated conditions


  • Regular review of data privacy considerations


To accommodate these testing activities, the culture of your hardware and software engineering teams may have to change. IoT often shifts testing to the left, calling for developers to perform their own smoke testing with a real device while they are writing code or changing hardware configuration. This accounts for “happy path” conditions, but fails to produce resilient, reliable IoT experiences because it rarely represents end-to-end usage (like from unboxing to dashboard reporting).

However, involving developers and engineers in both design and usage thinking also has a positive effect on your IoT experience: Superfluous and risky features will be identified quickly, before they reach consumers. For experienced testing professionals, this is a godsend in that when a product arrives to them, it will less be likely to contain obvious flaws that waste their time and will require fewer development cycles to resolve.

The Internet of Things presents software and hardware businesses with the opportunity to create real value for consumers, but also introduces the risk of failure to revenue and brand. To trick is to realize the former and avoid the latter by testing what you provide. IoT products and the services they rely on need to be simple and reliable while also allowing for integration.

If you are a device vendor, unit sales is not enough to indicate consumer success. You must look into how your devices are being used by real people to understand how to achieve the next level of demand for new hardware. Similarly, if you are a middleware provider, API throughput and volume of data are only operational metrics and do not represent consumer satisfaction. To satisfy the modern IoT consumer, you must focus your budget more on end-to-end quality through product development and testing than on your marketing campaigns.

When someone uses your device, thoughts should be positive. The value your IoT product brings should be apparent simply by normal use. The easiest way to achieve these goals is to design and test with the consumer in mind.

Paul Bruce is API Product Marketing Manager at SmartBear. A programmer at 13 years old, Paul loves to code, but enjoys real conversations with real people about stuff that really matters. He is currently API Product Marketing Manager at SmartBear. Previously, Paul was a systems developer for a Massachusetts-area college for more than seven years, specializing in enterprise software and informational system architecture, programming, business analysis, product management, and vendor integration. With over a decade in software solutions development and a performance engineering background, he now focuses on communicating the value of truly excellent digital experiences in the connected world. Follow Paul on Twitter @paulsbruce. Read more at:

Subscribe to Data Informed for the latest information and news on big data and analytics for the enterprise.

Tags: , , , , , ,

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>