A brief history of VR and AR

By the time Howard Rheingold's "Virtual Reality" was published in 1991, the Sensorama was already a "slowly deteriorating" relic stashed away in a cabana next the pool at its inventor's West Los Angeles home. Rheingold describes awe — even surprise — that the system was still operable almost 30 years after its introduction.

"I was transported to the driver's seat of a motorcycle in Brooklyn in the 1950s," the author writes. "I heard the engine start. I felt a growing vibration through the handlebar, and the 3D photo that filled much of my field of view came alive, animating into a yellow, scratchy, but still effective 3D motion picture."

The experience is immediately identifiable to anyone who has spent time in a modern VR headset. In the early 90s, it no doubt felt "a bit like looking up the Wright Brothers and taking their original prototype out for a spin," as the book describes. At the dawn of the decade that gave us both "The Real World" and "The End of History," virtual reality seemed to hold the keys to the next great paradigm shift.

The year the book was published, Sega announced a VR peripheral for the Genesis. That October also saw the release of Virtuality's 1000 Series, a headset that would make its way into arcades with titles like Dactyl Nightmare, a first-person platform shooter that finds the player pursued by an angry pterodactyl. Four years later, Nintendo followed up the wild and enduring success of the Game Boy with Virtual Boy, a headset/console built around (very red) stereoscopic vision.

VR also dominated the pop culture, featuring prominently in films like "Johnny Mnemonic," "Lawnmower Man" and "Virtuosity." The technology served as a visually rich shorthand for dystopian cyberpunk fears in an age of rapidly accelerating technology.

By the end of the decade, however, the veneer wore off. Sega VR's 1994 launch date came and went. The product was initially delayed before it was ultimately canceled. Virtual Boy, meanwhile, did make it to store shelves and has since been regarded as Nintendo's single greatest misstep.

For decades, the technology has felt ahead of its time, beginning with Sensorama in 1962. Considered by many to the starting point for what we now know as VR, the system looks more arcade cabinet than VR headset. The user is seated in a stool in front of the machine, their head obscured by a hood.

Morton Helig's 1962 patent describes a system that delivers a realistic simulation for potential work and military training purposes:

There are increasing demands today for ways and means to teach and train individuals without actually subjecting the individuals to possible hazards of particular situations. For example, the armed services must instruct men in the operation and maintenance of extremely complicated and potentially dangerous equipment, and it is desirable to educate the men with the least possible danger to their lives and to possible damage to costly equipment.

Ultimately, however, his work would focus on "Experience Theater" — machines designed to provoke all the senses. It's easy to see how, in the early 1960s, such a system could be viewed as a logical next step beyond the film and television of the day. The demo described in the book finds the user riding a motorcycle down a Brooklyn street. The borough appears in stereoscopic vision, as the wind hits the riders face, the handlebars vibrate and smells overwhelm. For all of its exciting innovation, however, cost was the major sticking point. It's a ubiquitous theme throughout the history of VR.

Sketch from Morton Helig's 1962 patent. Image Credits: Morton Helig

The form factor was different, but Sensorama otherwise hewed closely to modern conceptions of VR. Three years prior to the creation of Sensorama, Helig was granted a patent for a device that appears even more remarkably prescient. The Telesphere Mask was, effectively, a headset with two lenses that provided stereoscopic 3D images for viewing TV shows.

Per the patent:

My invention generally speaking comprises the following elements: a hollow casing, a pair of optical units, a pair of television tube units, a pair of ear phones and a pair of air discharge nozzles, all co-acting to cause the user to comfortably see the images, hear the sound effects and to be sensitive to the air discharge of said nozzles. One object of my invention is to provide easily adjustable and comfortable means for causing the apparatus containing the optical units, to be held in proper position, on the head of the user so that the apparatus does not sag, and so that its weight is evenly distributed over the bone structure of the front and back of the head, without the necessity of holding the apparatus up by hand.

It was, effectively, a version of the Sensorama designed to be worn on one's head. Both devices suffered the same fate of all things invented decades before they could conceivably be brought to life by existing technologies. Helig died in 1997 at age 70. He survived long enough to see virtual reality become a true pop cultural phenomenon, but never managed to truly capitalize on its success. His inventions, meanwhile, languished in boxes in his Southern Californian home.

An even more direct ancestor to today's modern XR (extended reality) dates back to the mid-60s, when a team at MIT's Lincoln Laboratory began their own headset experiments. In 1968, the team produced a system they jokingly deemed "The Sword of Damocles," owing to a large structural beam that hung over the wearer's head. One of the system's key innovations was a magnetic tracking system, designed to monitor head movements to shift the display orientation accordingly.

"Our objective in this project has been to surround the user with displayed three-dimensional information," project lead Ivan Sutherland wrote in a paper describing the system. "Because we use a homogeneous coordinate representation, we can display objects which appear to be close to the user or which appear to be infinitely far away. We can display objects beside the user or behind him which will become visible to him if he turns around."

Sutherland, who is still around today at 85, contributed important technological innovations for decades. His best-known innovation is likely Sketchpad, a CAD predecessor and computer graphics program that was — much like his headset — decades ahead of its time.

In a 2013 Time interview, he briefly — and modestly — spoke of The Sword of Damocles, noting, "The name virtual reality might be applied, but it didn't come along until ten years later."

NASA promotion shot of VIEW (Virtual Interactive Environment Workstation). Image Credits: NASA

NASA got in on the action in the mid-70s. Here the applications began to once again focus on simulation for the workplace — the "workplace" in this instance being, of course, space. At the Jet Propulsion Laboratory, David Em made strides in the creation of explorable virtual landscapes, while the Ames Research Center iterated on VIEW (Virtual Interactive Environment Workstation).

The system featured a head-mounted display with head tracking that could be used to explore virtual environments or real remote images from a camera — foreshadowing future breakthroughs in teleoperation. There were gloves designed to track hand movements to interact with virtual objects and a full body "DataSuit" for further movement tracking.

Reflecting on the breakthroughs of the era, Stephen Ellis, the head of Ames’ Advanced Displays and Spatial Perception Laboratory noted simply, "The technology of the 1980s was not mature enough." It's yet another pervasive theme that crops up again and again throughout VR's long history.

The term "augmented reality" wasn't officially coined until 1990, but the 80s also saw important breakthroughs in the space. Much of that work was built atop decades of military research into heads-up display (HUD) units for aircraft. Steve Mann is commonly referred to as the "father of wearable computing" for innovations like the EyeTap, which combined computer processing with graphical design and textual overlays. Reflecting on his early work, Mann writes:

I started exploring various ways to do this during my youth in the 1970s, when most computers were the size of large rooms and wireless data networks were unheard of. The first versions I built sported separate transmitting and receiving antennas, including rabbit ears, which I’m sure looked positively ridiculous atop my head. But building a wearable general-purpose computer with wireless digital-communications capabilities was itself a feat. I was proud to have pulled that off and didn't really care what I looked like.

The 80s ushered the term "virtual reality" into the popular lexicon and set the stage for the aforementioned attempts at consumer and arcade VR. Dot-com wasn't the only bubble to burst at the turn of the millennium, however. In the 90s, virtual reality was the future of entertainment, gaming, socialization and work. In the 00s, however, that future seemed to simply dissipate with a few notable exceptions, including the 2003 launch of early metaverse, Second Life, which remains in operation nearly 20 years later.

Palmer Luckey, co-founder of Oculus VR Inc., left, plays the new video game "Eagle Flight VR" during an Ubisoft news conference before the start of the E3 Gaming Conference on June 13, 2016 in Los Angeles, California. Photo by Kevork Djansezian/Getty Images

The first Oculus Rift prototype reignited the conversation when it arrived in 2012. For the first time in VR's long history, it felt as though technology had finally caught up to concept. Breakthroughs in display and computing technology driven by the smartphone industry laid the groundwork for the viable consumer headset. Facebook certainly believed so. In 2014, the social media giant purchased Oculus for nearly $3 billion.

In the intervening decade, virtual reality has begun to take on outsized influence at the company, culminating with its 2021 rebrand to "Meta." The company believed it had purchased a front row seat to the future. "The first metaverse that gains real traction is likely to be the last," Oculus exec Jason Rubin wrote in a 50-page internal document laying out the strategy. "We must act first, and go big, or we risk being one of those wannabes."

Google revealed Glass the same year the first Oculus prototype hit the scene. That year's I/O continues to be the apex in terms of excitement, owing far more to the crew of Glass-wearing skydivers than the Nexus Q. "You’ve seen some really compelling demos here," Sergey Brin told the crowd. "They were slick, they were robust. This is going to be nothing like that." Glass was released for developers in February 2013 for $1,500. Sales opened for consumers later the same year.

The following year, Google introduced Cardboard, a super cheap and extremely clever smartphone accessory. It shipped flat, with a pair of lenses built in. Once folded, a phone was placed inside for a budget-rate VR experience. The platform's affordability drove strong consumer interest, with around 15 million units shipped over the system's life. In fact, the program was only officially discontinued in 2021. It managed to outlive the Daydream platform the company introduced at its I/O 2016.

Image Credits: TechCrunch

When HTC began imagining life after the smartphone, it looked a lot like VR. A collaboration with Valve, the product was introduced as a demo under the "SteamVR hardware system" banner at 2015's GDC show. Later that year at Mobile World Congress, HTC announced the Vive name during its official unveil. As the company's smartphone fortunes began to wane, it increasingly shifted eggs into the Vive basket.

In 2016, Microsoft began shipping a developer edition of its mixed reality HoloLens system, priced at $3,000. It was an innovative and impressive product, largely aimed at enterprise applications. The company has yet to introduce a consumer version of the product. 2018, meanwhile, saw the release of the Spielberg-directed Ready Player One, which is at once a nostalgia fest and a love letter to VR's potential.

Magic Leap officially announced its One headset after years of speculation and rumors. That was fueled, in part, by early funding in excess of $1 billion. The startup would ultimately follow Microsoft's lead with a pivot into enterprise. "We really saw that there was a value to be derived from AR much sooner from enterprise," CTO Daniel Diez told me during at interview during this year's CES. "The feedback we were getting from them was that. It also gave us insight into how the product needed to evolve to be truly purpose-built for enterprise, and that's what you see in the Magic Leap 2."

Image Credits: Apple

Magic Leap is far from alone in those struggles. The history of virtual reality is littered with stories of smart people and innovative companies running headfirst into actual reality. Next Monday, Apple is expected to become the latest to stare down that wall. The company has been in the AR business for a minute, launching the development ARKit in 2017 as part of iOS 11 (Google's ARCore arrived in 2018). Anticipation has thus far been a cocktail of skepticism and faith in the company's track record. It certainly does have a long, fruitful history of breathing new life into existing categories like the mobile phone, mp3 player, smartwatch and headphones.

All of the conversations I’ve had with competitors this year point to an excitement at that possibility. The hope is that if Apple finds success, it could reverse the fortunes of many in the industry as interest in and excitement around VR increases. Years of rumors paint a far less rosy picture, however. The headset — which might be named the "Reality Pro" — has apparently been in development for eight years. Apple has a long history of waiting until products are just right, but shareholders apparently got sick of waiting.

Tim Cook is said to be less than thrilled with the form factor. His dream of a headset that looks like a standard pair of glasses was just too difficult for engineers to execute. Instead, we’ll likely see something akin to a traditional VR rig. The executive is no doubt aware that the product — for good or ill — will be tied to his legacy as CEO.

Interest in the category has undoubtedly risen ahead of the announcement, and the underlying technologies have advanced by leaps and bounds. But the question remains whether mixed reality is finally ready to stop being ahead of its time.