Shoplifting Amazon Go

Some companies want to replace retail with automation. Amazon’s Go stores are an attempt to replace the convenience store with an even more convenient shopping experience. With no staff members in sight, the stores instead contain a dizzying array of technology that tracks you as you shop. You can walk in, grab your groceries, and walk out. The goods are automagically charged to your Amazon account before you’ve gotten a block down the road. On a recent visit to Seattle, the home of Amazon HQ, I spent a bit of time in these stores to figure out how they work.

The first location I visited was conveniently located in a central part of Seattle: nondescript except for the Amazon Go sign out front. As I approached, I quickly downloaded the app onto my phone and it prompted me to sign in with my Amazon account—something I had already set up with my credit card details. I was ready to shop, or shoplift, which is more akin to how the whole experience felt.

Entering involved navigating a confusing array of entrance and exit turnstiles—I, of course, tried to enter via the exit. Trying to maintain my cool as I turned around, I scanned the barcode on my phone against the top of the green entry turnstile and was permitted to enter. Technology: 1, Vaughan: 0.

Inside was devoid of people, apart from one couple who were shopping for dinner. They were in the packaged dinner section looking at a chicken salad. A lone employee in a bright orange shirt pushed a trolley around as he restocked the shelves, and I was surprised to see him: I had half expected his job to be done by robots.

The store itself was laid out less like a supermarket and more like a corridor with some shelves in it. The total footprint was fairly small, and the products on display leaned towards convenient snack foods rather than the more substantial items that tended to be sold in regular grocery stores. Other than pre-cut fruit salads in tubs, there was no fresh produce to be found. One side of the space was dedicated to a continuous refrigerator, filled with a variety of sandwiches that included everything from breakfast bagels to filled rolls. The labelled sushi section contained no actual sushi but instead more breakfast rolls. I was visiting early, shortly after 9am, so maybe sushi just wasn’t breakfast food.

I imagined the fridge space was repurposed throughout the day to optimise it for what customers were most likely to be shopping for. This would give the lone Amazonian plenty to do as the day progressed—constantly rearranging shelf space and bringing out new products as the hours ticked by. Right now he seemed to be restocking after the morning commuter rush, as some of the choices on the shelves were already exhausted. There was no mango juice left, which was particularly disappointing for me as that was exactly what I wanted.

So what kind of technology allows you to fill a bag with goods and walk straight out again? I was half expecting vending machines or secret buttons and levers that detected when a product was taken off the shelf. As I instinctively picked a sandwich up to check its ingredients, I thought, “ah crap, I guess I just bought a sandwich.” But there was no discernible sensor on the shelf to detect that the sandwich had been removed from its place. I looked above me to try and locate a person hiding in the rafters with a tablet, watching my every move. This wasn’t quite the case but there were dozens—if not hundreds—of small black boxes in the ceiling with round lenses on them that I could only assume were cameras.

Although I liked to imagine someone keeping an eye on me from a windowless room at Amazon HQ, there are in fact only computers involved in the process, programmed to track my browsing and recognise when I’ve put a sandwich in my bag and when I’ve just picked it up to check its contents. I put the sandwich back and wondered whether it would appear on my receipt when I left.

The multitude of cameras appear to track the few feet in front of each shelf to detect when a person reaches in and grabs a product. Each item is wrapped or packaged in a box to make the products both uniform and distinguishable to the robot vision. Unfortunately, the amount of single-use packaging is enough to make an environmentalist’s eyes water. Once a product is removed and disappears out of range of the camera, it is assumed to be taken and added to the virtual cart of whoever it saw taking it off the shelf.

I wondered if I could politely ask a fellow shopper to pick up something for me: “Excuse me, sir, can you reach that cheese on the top shelf, please?” Boom, you’ve just tricked someone into buying you cheese. Video analytics can tell what is taken off and put back onto the shelves, but it can’t see what you do with the products once they are off the shelf. If you handed the product to someone else, it’s still in your cart. Something to watch out for.

The Amazon Go aisles are clean and bright and there is no need to talk to anyone. Young 20-somethings breeze in, grab things off shelves, and breeze out again. They’re quick. I spent 17 minutes selecting my two sandwiches and a couple of drinks, worrying the whole time that I was doing something wrong. 

Despite the hype about being the grocery store of the future, Go stores are not set to replace your local grocer just yet—unless all you want is a sandwich and a stick of gum. But Amazon are notorious for their pioneering technology, and it is clear that these small-footprint stores are an experiment to test it in the market. They’ll figure out how to track carrots, bulk-bin quinoa, and artisan bread in no time.

I decided that I had picked up enough to test if I was a shoplifter or a futuristic shopper. I exited through the red exit turnstiles with trepidation, expecting the scream of shoplifter alarms or huge gates to come down around me. There was none of that. I walked out onto the sidewalk, still somewhat anxious. I wasn’t sure what to make of the whole experience. Did it even work? Well I still had my sandwiches, at least. I unlocked my phone—it had been in my pocket since I scanned into the store—and opened the Amazon Go app. My blood ran cold. There was nothing in my virtual cart at all. No receipt, no indication that I had bought anything at all.

Should I go back into the store and ask someone to check? Who would I ask? The Amazonian restocker didn’t look like he could hack into the app for me. There was a coffee shop next door, so I decided to buy a coffee and wait to see if anyone came running. One latte later, my receipt appeared on my phone, each item correct and accounted for. Phew, I mean, damn!

I wanted to learn more about the future of retail, at least how Amazon saw it, so I decided to enlist some insider help. I happened to know someone who worked on top-secret Amazon products, and I thought he might be able to tell me about the company’s attempted domination over retail. He lived in Seattle, of course, so I hit him up for dinner. 

Angus is someone you might describe as a serial product builder. No, not breakfast cereals, but a serial builder of products. He’s worked at Microsoft, Xero, and now Amazon, creating cutting-edge new products that change our world. If you’ve ever used Office365 or the revolutionary online accounting software, Xero, you would have encountered some of Angus’ handiwork. He is now working on what he describes as top-secret Amazon projects, and despite knowing him for some time he still would not tell me exactly what he was working on. Loyal to Amazon to the core. We arrived at a compromise over dinner and a couple of cocktails (five). He agreed to take me on a tour of Amazon HQ and their on-campus Go store in the heart of downtown Seattle. 

Like most technology empires these days, Amazon is in the real estate business. As well as their empire-building online, they also tend to become owner-occupiers of whole cities or neighbourhoods. Microsoft has basically taken over the city of Redmond with its cookie-cutter campus buildings. Google is the same, owning most of San Francisco’s Mountain View. Apple has Cupertino, and Amazon has Seattle—or at least a dozen blocks of it. Amazon asserts its dominance from skyscrapers that look like Amazon products themselves.

Angus agreed to meet at Amazon Spheres, a giant spherical glasshouse that grows not only rare plant species but also new ideas in its open-air, multi-level structure of meeting spaces. It was definitely something to marvel at, and we ordered coffee and sat admiring the swarms of Amazon employees who were congregating among the ferns. After giving my brain a thorough reoxygenation after its evening of cocktails, we decided to give the Amazon Go store another go, so to speak. This time, I’d have the help of an Amazon local.

Today’s experience was a little bit different, as there were a few of us shopping together this time, as a group. I wondered how the Go store would handle a regular family shopping with a bonus level of chaos introduced in the form of small children. Angus explained how it worked: multiple people could be let into the store on the same phone by tagging any dependents onto your account as they passed through the turnstiles. Then as they rampage through the store, Amazon tracks each person separately but collects all of the items they take off the shelves into one basket on one app. 

Now, kids are generally not known for their predictability or adherence to rules.

“What if one of my kids took a drink off the shelf, drank it, and then put the empty bottle back on the shelf? Would Go know they drank it?”

“Probably not,” Angus replied, “But then does your kid normally drink items in the supermarket and put them back on the shelf?” He had a point. The etiquette of shopping here was not unlike that of a traditional supermarket, and as far as I was aware, my kids didn’t normally do that. 

“What if one of my kids took a sandwich, handed it to another one of my kids, and they put it back on the shelf?”

Yes, Go can track that. Go treats you all as the same shopper.

We tried to imagine new ways to confuse Go and hack the system. You could cause havoc in the store if you tried, throwing products around, eating food, and just generally being irksome. But like Angus said, that involved acting in a socially irresponsible manner, which you could do in any other regular supermarket too. Shoplifting was nothing new here, but most people don’t act out and so 99.9% of the time it all works. Technology makes the whole shopping experience faster and easier, and over time the accuracy of Amazon Go—tracking all of the weird experiments I would try with my kids—will only get better.

On the way out of the store, we remembered that we needed to grab some antihistamines. Three of us huddled together around the drugs shelf, each inspecting different brands and their active ingredients. My partner was looking for a particular type of antihistamine, and when we found it we left with it in our basket. 

Outside on the sidewalk we thanked Angus for his amazing hospitality and left him to get back to his real work. While rounding the corner I checked my receipt to see if our shenanigans resulted in any of my items being put in Angus’ basket and vice versa. It all appeared correct—except the antihistamines. They weren’t on my receipt. I quickly messaged Angus.

“We hacked the store! We didn’t get charged for the antihistamines. Check your receipt!”

Angus replied, “Just checked. I didn’t get charged either. You must’ve hacked it you crazy kiwi.”

The most expensive item from our visit was missing. We must have confused the cameras by all reaching into the antihistamine shelf at the same time, taking multiple brands out and putting back the ones we didn’t want. This must have been a little too hard for Go to track, so it assumed we didn’t take any. After all, it’s probably better for the technology to undercharge than overcharge and create a bad experience. A small price to pay when all of the other cost savings presumably make it worthwhile. In fact, when one reporter accidentally shoplifted at the first Amazon Go store in 2018, she realised that the occurrence was expected to be so uncommon that there was not even a feature on the app where customers could report it. Looks like Amazon is happy to shout us our antihistamines every once in a while. 

Besides, Amazon is blazing a new trail here. They’re playing a long game, fine-tuning the technology so that they can scale it up in the future. Most disruptive technology takes ten years to change an industry—often even longer. Amazon Go stores have only been with us for a couple of years, so we need to give them time. But for now, Amazon might want to keep a tab on the antihistamine aisle.