IoT Testing Challenges and Legacy

We live in a world where your car will soon be a far better driver than you, and your fridge will be more responsible as well, even ordering groceries when supplies start to run low. This is feasible through the advancements of embedded systems into the Internet of Things (IoT). The quality and performance of such systems still require the improvement of existing testing methods to ensure quality while taking into consideration speed and scalability.

Embedded systems have been around for decades now, while the IoT is still in its infancy. The difference between the two is that embedded systems are self-contained and isolated, while IoT gadgets are in permanent communication with the server and, soon, with each other. Such a degree of integration  and interoperability poses significant problems regarding testing and robustness due to differences in protocols and security.

Old Tools, New Challenges

Traditional testing methods take time, which is no longer a viable option. The entire development process needs to be agile, detecting problems as early as possible. The new mantra is “test early, test often” and the result is continuous integration, which is even more important in an IoT environment. The sensors, communication channels, protocols, software-hardware interactions, and configurations determine an unprecedented magnitude.

The good news is that testers have all the necessary tools in place, they just need to re-think processes. Before, companies usually chose between manual and automatic testing. Nowadays, the choice is only between the degree of automation and the percentage of manual testing.

Existing testing methodologies for mobile testing that are suitable for apps are also useful for testing IoT devices. Most IoT can be regarded as apps and tested in such a way due to APIs. The only difference is the shift of focus away from the application towards an encompassing user experience and adding hardware testing.

Fragmented ecosystem

A few years ago, mobile was a term applied to laptops, smartphones, and tablets. It is slowly becoming a word to characterize any item that can host a processor and a network connection device. Wearables, TVs, even home security systems, all are now in this growing category thanks to incorporated sensors. The extreme diversity and fragmentation both regarding devices, operating systems, and protocols lead to increasing requirements for QA departments.

Sensors, the gateways to the servers, the cloud, and the application are all subjects to elaborate tests. All these layers function according to different rules and call for various tests. The problem is that some tests require the actual device or testing between devices to be reliable. This is the most important difference from software development.

Testing & QA’s skills

While the tools won’t change dramatically, the approach to testing will. It is no longer sufficient to have a tester behind a monitor. Real life setting should be a stage of testing. Methods from design, ergonomics and user experience will be remodeled to suit IoT needs.

Testers and QA will move further towards development, and we can expect to see more tightly knit teams. Testers will learn more about coding and developers will get a test-first mindset. The real change is that in the IoT world, testers become as important as developers. Black box testing is not an option in this fast paced world.

Testing

The environment

The biggest challenge posed by IoT testing is the environment where it operates. Companies have not made the transition to “on device” testing yet. The problem is the CFOs don’t happily approve the purchasing of a device like a car or even smartTV for testing purposes.

Simulators and emulators are not enough to test even mobile apps, let alone IoT. The lines between digital and physical are fading away through the integration of sensors. However, a flawless performance requires running numerous scenarios on different settings to test not only application logic, but interaction.

Test automation

Automation is a growing trend and a necessity. Due to the scale of the matrix of possibilities when taking into consideration hardware/software/communication combinations, manual testing is just not feasible anymore. Add pressure to market, and suddenly, automation emerges as the only viable option.

Automation is an expensive method, requiring highly skilled specialists to write a new script for every modification to the product. For now, it is safe to assume that automated testing is a suitable candidate for mature processes. User experience should not be automated, but replaced by user generated data on a different range of devices.

On the other hand, ad-hoc testing, usually triggered by a bug detection will disappear. This widely-used practice is no longer a viable alternative for IoT, especially from a business perspective. Users demand flawless experiences and just trying to fire-fight problems is not scalable.

Standards

There are no generally accepted standards when it comes to IoT, and this impacts security and large-scale adoption. Both established organizations such as the IEEE and other alliances or even independent companies have designed attempts to define a single standard.

Security and Privacy

The greatest concerns related to IoT are security and privacy. Without properly defined standards, hackers can hijack the devices linked to a network “ many of which owner have paid for “ and transform them into listening or monitoring tools. Moreover, since an IoT device consists of several layers and operates at a low network level, general Internet security rules designed for apps and browsers are not enough.  Security must be re-tested after every update.

Conclusions

The IoT brings new challenges related to testing. Some organizations will complain about the scale of the challenges, including data amounts, channels, and protocols. Others will try to overcome resource limitations and integration problems. Automation will play an increasingly important role, without retiring the manual tester, who will need to update his UX skills. The most important item will be defining a commonly accepted security framework for privacy testing. Safe IoT will require a collective effort that surpasses any current test problems.