Apple has recently launched ARKit to help developers make augmented reality applications for the iOS platform. Google has several projects like Daydream and Google Cardboard to provide low cost virtual reality experiences. The industry standards, the Oculus Rift and the HTC Vive have also grown leaps and bounds in the last few years. Virtual and Augmented reality are finally beginning to be viable, after decades of experimentation. But there is still a long way to go. Here are the top 4 technical challenges AR and VR need to solve in 2018 to become a household staple.
Latency
Latency in drawing new content is the biggest technical challenge AR and VR face today, and has been since the inception of these ideas. Any system is bound to have a threshold latency which comes from the frame rate of the content being drawn, the refresh rate of the display, and the input lag from the interaction that started the new content draw. The question of what the human eye perceives as fluid motion is a very complex one, but VR headsets typically target a latency less than 30-40 ms. Problems originating from latency are not isolated to the AR / VR field. All systems that have the same interface of interaction between a human and the corresponding content draw on a screen face the same issue. VR and AR are no longer limited to the entertainment industry only. There is an immense potential in delivering services remotely using AR / VR.Doctors around the world are experimenting with these technologies to deliver their services remotely. In order for such novel applications to be successful, latency issues must be solved.
Spherical sound design
AR and VR have the unique problem of requiring spherical sound design by default. One challenge to solve in 2018 is of making the appropriate tools that can handle such sound design. While spatial sound design is nothing new, VR requires at least 6 degrees of freedom to work as expected – that means that perceived audio has to come from the conventional 3 directions (x, y, z), as well as the diagonals between them to give the impression that the sound is attached to the object that generates it. Another challenge is to make sure that the user does not get overwhelmed by sensory overload. The world of VR can be chaotic, and the human brain is not built to handle such chaos effectively. The sound designers are required to make sure that the sounds are culled appropriately. We need better tools for that in 2018.
Cost and Streaming
VR has a cost problem. Both the Oculus Rift and the HTC Vive cost upwards of $500 for the combo of headset and controllers. Projects like Google Cardboard try to provide an affordable experience but fall short considerably in terms of quality. To solve one component of the latency problem mentioned above, the driver for the content has to be powerful enough to maintain a recommended frame rate of more than 90 fps. This is the reason that VR content only works well on powerful machines. Hardware manufacturers are marking their products as VR Ready if it can run interactive VR content smoothly.
Another cost factor associated with AR and VR is of streaming the content. Because the content needs to be in high resolution, and needs to rendered at a much higher refresh rate, the bandwidth required for streaming grows a lot. Currently the HTC Vive has a display resolution of 2160×1200, which is just slightly above 1080p. VR is moving towards 4K now (4K resolution for each eye, effectively being 4096×4096 in total). Netflix estimates that the recommended bandwidth for streaming 4K content is 25 Mbps. Good 4G connection can pull that off but at double the bandwidth required for 4K VR, 4G falls short. The telecom industry is gearing up for the next generation upgrade to 5G networks in the coming years, and that will surely become the turning point of VR streaming. There are some new projects like Conduit that aim to reduce the required bandwidth for streaming VR content in 4K.
Compatibility With Other Hardware
Different VR headsets have different content formats today. The content that runs of Oculus Rift cannot run on HTC Vive without modifications. It is the console wars all over again. AR and VR need a consolidation body that can enable creators to make content for multiple platforms with ease. It might be too early for such initiatives, given the nascent stage VR is in today, but it is certainly the way forward to ensure that the user has adequate choices. AR is also heading in the same direction.
Virtual and Augmented Reality are a different beast than conventional 2D systems. Some of the challenges VR / AR face, like maintaining low latency and compatibility, are the same in 2D systems as well – just that the throughput required is much higher. Other challenges like spatial audio design and interaction design are altogether very new. 2018 is going to be a very exciting year for AR and VR, for sure.
