Enhancing how you see the world

How new sectors are embracing innovative ways to use augmented reality

By Antoinette Price

Have you experienced augmented reality (AR)? If you have ever had keyhole surgery, reached your destination by using a satellite navigation system or used an AR catalogue to see how furniture looks in your home before buying it, the answer is yes.

Ikea furniture app IKEA furniture placement app (Photo: YouTube)

Like many cutting-edge technologies, the scope for AR applications is broad, encompassing sectors as diverse as education, entertainment and gaming, manufacturing, marketing and advertising, medical, military, navigation and tourism. It continues to expand rapidly, but the industry faces important challenges it will have to overcome if AR technology is to become mainstream.

Mobile communication boosts AR

Figures from eMarketer show that two billion consumers worldwide are expected to be using smart phones by 2016 and over half of mobile phone users globally will have smartphones in 2018. Riding this trend, consumer adoption of AR is expanding thanks to wider brand and retailer engagement through mobile channels, now considered by a number of companies to be a primary communications medium for customers.

A report by Juniper Research estimates that annual revenues from mobile AR services and applications will reach USD 1,2 billion in 2015. While games accounted for 40% of AR downloads in 2013, the report predicts that mainstream lifestyle, enterprise and general entertainment applications using AR should achieve annual mobile revenues in excess of USD 1 billion within five to six years. The report also highlights the potential for AR app usage with smart wearables such as smart glasses, and forecasts that AR app user numbers will be approaching 200 million by 2018.

Layering technology

So how does AR technology work? It enhances reality by overlaying the user’s field of vision with supplementary digital information in real time, usually by means of computer-generated graphics. For example, when a tennis player queries a serve, TV viewers see a line which follows the ball to show whether it was in or out.

The key to this technology lies in the software. Special 3D augmented reality programmes allow the developer to link animation or contextual digital information in the computer programme with an AR "marker" in the real world.

Standardization ensures that manufacturers, companies and users can produce, sell and use reliable, safe products and services. Several IEC Technical Committees (TCs) and Subcommittees (SCs) prepare International Standards for AR applications and for the technology that enables them.

ISO/IEC JTC1/SC 29: Coding of audio, picture, multi-media and hypermedia information, has published ISO/IEC 23000-13, Information technology - Multimedia application format (MPEG-A) – Part 13: Augmented reality application format. It focuses on the data formats used to provide an AR presentation and is designed to enable the use of 2D/3D multimedia content.

This Standard specifies:

  • scene description elements for representing AR content
  • mechanisms to connect to local and remote sensors and actuators
  • mechanisms to integrate compressed media (image, audio, video, graphics)
  • mechanisms to connect to remote resources such as maps and compressed media

It’s all done with sensors

Microelectromechanical systems (MEMS) sensors can be found in everything from automobiles, PCs, medical devices and industrial applications to – more recently – portable consumer electronics. This is due to a reduction in their cost, size and power consumption. MEMS, which detect the orientation of a device, the direction in which it is moving and its absolute location in three dimensional space, are a key part of the technology required for location-based services using AR applications. Some examples of MEMS are health and fitness monitors, gaming or tracking systems.

IEC SC 47E: Discrete semiconductor devices, and IEC SC 47F: Microelectromechanical systems, prepare a number of International Standards that enable manufacturers to build better, more efficient and reliable sensors and MEMS. Together they facilitate the design, manufacture, use and reuse of MEMS.

Innovative AR apps for medical and healthcare

Augmented reality is transforming the medical and healthcare sectors significantly for patients, doctors and pharmacy management alike.

AR apps can be very useful for teaching anatomy and surgical procedures, by overlaying digital information in the form of video, audio or 3D models onto human skeletons using head mounted displays. For example, projecting a CT scan onto a patient gives doctors an “X-ray” vision of patients can provide key contextual cues for diagnosing patients and learning.

Seeing is understanding with the EyeDecide app. Designed as a patient engagement package, the AR app educates patients as well as improving their care management. It enables sharing customized care plans and treatments and uses interactive 3D animation, text description, and radiographic imagery to explain the most common eye conditions and procedures to patients. They are then more informed when it comes to deciding about their care thanks to images and anatomical layers used to demonstrate disease progression. There personal information is easily uploaded and shared, which improves care quality and saves consultation time.

Some patients who take medication regularly sometimes forget to do so. Users of wearables such as Google Glass can install an app which allows them to receive reminders of when to take their medication.

According to the National Parkinson Foundation, Parkinson’s is the second most common neurodegenerative disease after Alzheimer’s, affecting an estimated four million worldwide. An AR app has been developed to help sufferers of this movement disorder, which affects the nerve cells and makes it impossible to move. By combining dance with a Google Glass AR app, patients can improve body balance, do seated routines in the mornings and evenings and standing routines for when they suddenly freeze in mid walk, to get moving again. By tapping the glasses or speaking, users receive a walking speed which is set by visual stimulation and music. The app provides a portable, intuitive interface offering real-time, on demand assistance.

Other apps have been developed for people with mental health issues. Empathear helps families and others understand schizophrenia better. Smart phone microphones pick up the user’s surroundings and, in conjunction with mood settings, sounds and whispers, enable users to gain an improved understanding of how sufferers feel, depending on their mood and external environment.

PTSD Coach, an app for veterans and military service members with post-traumatic stress disorder, provides education about the condition and information about professional care, self-assessment, support and tools for managing daily stresses.

Who else is augmenting reality?

The gaming and military sectors have long used AR applications, however others are finding innovative and creative ways to apply this technology including:

Retail: a number of high-street and high-end brands have created an array of innovative AR apps for their products, such as an AR dressing room, which allows shoppers to try on their clothes virtually or an AR makeup mirror (photo) that lets customers see which make-up shades suit their skin tones before purchasing them. A well-known global furniture company has launched an AR catalogue that enables customers to 'place' the piece of furniture in their homes with an app that then measures the size of products relative to the surrounding room.

Location-based services: several similar apps have been developed in different countries. They allow users to point their phones at buildings as they walk along a street and receive information about restaurants and other sites in the area, such as write-ups of various businesses that have registered with the app. This type of service has boomed with the increased use of smart phones, since it only requires a camera, a global positioning system (GPS) and an Internet connection in order to receive information in real time.

Automotive: as a means of keeping drivers looking ahead rather than down if they need to access additional information, car manufacturers are considering integrating AR features into the next generation of head-up displays (HUDs). HUDs present virtual images that are focused beyond the instrument panel and appear to be superimposed on the driver's view of the road ahead. However, there is increasing concern that this could be too distracting and compromise safety when drivers shift their focus from the road to reading information displayed on the windscreen. More testing is needed before this technology becomes widely available.

Manufacturing: AR applications are improving product design by allowing manufacturers to experiment with possible scenarios before they make parts. Virtual objects may be shown attached to products so as to convey how each part is to be assembled or disassembled. Text instructions or video clips can show a procedure by referencing specific points or objects along the way. Such explanations, delivered in real time, reduce worker fatigue and errors and improve quality and delivery time. AR applications are also used for machining and robot operational planning and for assembly, shop floor layout, maintenance and for teaching and implementing manufacturing procedures.

Mining: an Australian company has introduced one of the first AR applications for mining, allowing operators to use handheld devices for conformance in the field. A laser scanner collects surface data as mining happens. The app then automatically registers this data against the 3D design of mine plans which have been imported previously and are used for comparison. This takes under 10 minutes on a smartphone, tablet or laptop.

Obstacles to mainstream use

Many industries have already embraced AR and are seeing great improvements in their products, services and operational processes as a result. Innovative ways of using the technology will continue to be developed in new areas. However, there are a number of important issues that need solutions to be found before AR is widely accepted.

Technical glitches: for applications which need to track location, GPS is only accurate to within about eight metres and doesn’t work well indoors. There are also the usual issues for the devices running the apps, such as Wi-Fi connectivity and battery life.

Users of wearables such as smart glasses may experience information overload and become distracted by what is appearing before their eyes. This may also be the case with head-up displays on car windscreens. In the latter, users are required to shift their focus from the data to the road and may also misjudge the speed of oncoming traffic. While improved built-in sensors might be a solution, more testing is required. Additionally, aesthetics play a role in wide-scale acceptance in certain industries and some hardware devices are still not very attractive.

Privacy: knowing what products are on sale in a shop or discovering the history of a city by pointing your phone at a building is one thing, but imagine being able to see another user’s Facebook, Twitter, Amazon, LinkedIn or other online profiles simply by pointing the phone at that person. This will soon be possible using image recognition software and AR applications. While users have chosen to create online profiles, they may not wish total strangers to be given direct access to their personal information. Online data security and protection are not new topics, but this kind of instant accessibility needs to be considered carefully from legal and ethical viewpoints before building this technology into our devices.

A man wearing Smart glasses Seeing through smart glasses (Photo: Wikipedia)
Location-based service app Location-based service app (Photo: thedigitalmarketingbureau.com)
Ikea furniture app IKEA furniture placement app (Photo: YouTube)