Imagine being able to point your smartphone’s video camera at a store shelf and instantly get personalized, detailed information about the products on your device’s display screen.
In a few months, some shoppers no longer will have to imagine this marriage of the Internet, data analytics, mobile technology and the physical spaces of a shopping aisle.
IBM researchers at the company’s lab in Haifa, Israel, are developing an “augmented reality” mobile shopping application that should be offered by some supermarket chains to customers before the end of the year.
Customers will download an app after entering a store and use their smartphones or tablets to create a profile that details their product preferences. In a supermarket, that could mean dairy- and gluten-free foods, low-sodium products, low-calorie beverages or items on sale. The app is configured to work with the store’s inventory system fed by other sources, such as nutritional data from consumer packaged goods companies.
When a shopper pans store shelves with the camera of their mobile smart device, the app recognizes the products through image-processing technology and displays relevant information based on preferences input by the customer, as well as other data such as purchasing history. The information then is superimposed over the products on the device’s screen using augmented reality technology. The app can offer personalized coupons or inform the customer of applicable discounts. The system relies on optical character recognition, image shapes and colors and the shelf positioning of goods, rather than bar codes or RFID tags.
“If you program it to say, ‘I am allergic to peanuts,’ the app will tell you which products are safe and which to avoid,” says Jill Puleri, head of IBM’s Global Retail and Business Services unit. “It’s designed to help consumers ease their way through the shopping process,” Puleri says. “It’s fast, and I think it’s going to be really important for some consumers.”
More important, potentially, for some retailers. The mobile revolution has compounded the threat to bricks-and-mortar retailers posed by online shopping. Not only has the Internet empowered consumers to shop for the lowest prices – often reducing stores to showrooms in which consumers check out the merchandise in person before buying it for less online — it has enabled retail websites to amass valuable information about their customers, something a bricks-and-mortar operation can accomplish much less effectively. One public example of the tension arose in May, when Target said it would stop selling Amazon’s Kindle e-book reader in its stores. Amazon offers a Price Check app to compare prices at the store and online and ran a promotion to give users a bigger price break during the last Christmas holiday shopping season.
The IBM app suggests there are additional ways to compete. In addition to screening products, shoppers also have the option of using the IBM app to tap into their social networks for product reviews, adding another personalized data stream to the process that could prompt a purchase on the spot or help build long-term loyalty.
Beyond what IBM’s augmented reality app may offer retailers on a customer-by-customer basis, it has the potential to create a treasure trove of metadata regarding product sales trends, hot in-store selling spots, traffic patterns and inventory issues, all of which could be used to maximize revenue per square foot, a critical metric in retail. It could also create new kinds of data products for retailers who are able to analyze their findings on behalf of the manufacturers who are their customers.
Technology Taken from Traffic Systems and Medical Imaging
The technology that soon will enable all this – the advanced image-processing technology – was developed at IBM’s Haifa lab and already has been utilized for purposes as disparate as recognizing license plates for a road toll system in Stockholm, Sweden, and spotting malignant growths in medical images.
IBM researchers applied the techniques used in those systems and then wrote algorithms that would capture and present the appropriate data to shoppers as a digital overlay on top of the product images.
As Amnon Ribak, project leader for the augmented shopping app, suggests in an IBM Research blog post, it was not easy to accommodate for a range of in-store settings. “We’ve already submitted a number of patent applications based on the new techniques we discovered to overcome the challenges in recognizing products with less than ideal lighting, shadows, and reflections,” Ribak says.
Ribak says in an email that there are a number of options for data feeds and that while nutritional information is a straightforward example, there is are other options. “Almost every real application today, and mobile applications in particular, [are] built with connectivity in mind, and mashing [up] information from various data sources,” he adds. IBM says it is working with supermarket clients on the augmented reality shopping app. “I can’t tell you who they are,” Puleri says, “but they are looking to roll it out by the end of the year.” While the app will make its debut in supermarkets, “it can be applied to just about anything,” she says.
Shoppers’ Device Use Drives Retailers
The kind of augmented reality application that IBM is discussing may be new to retailers but it addresses an top agenda item for the industry, says Richard Mader, the National Retail Federation’s director emeritus for the Association of Retail Technology Standards (ARTS). At a recent ARTS conference, 70 percent of the attending retailers said developing mobile applications was their top tech priority.
“You’ve got to be reaching out to your customers, staying in touch with them, but not really spamming them,” Mader says. “We’re still in a very new world. A lot of minds are using these new devices in new ways.”
Smartphones have changed the customer experience in a store. Mader says shopping has always been a social experience, but the ability mobile devices have to instantly share pictures—of a particularly fetching outfit, for example—is making shopping social even if the customer’s friends are many miles away. Mobile applications that allow users to pay for merchandise with a phone also represent an industry advance.
“It’s the customer that’s leading the mobile revolution,” Mader says. “They’re demanding it.”