When did you first hear the term augmented reality (AR)? If you are new to the tech space, your first encounter with this visual interactive technology was likely the Apple Vision Pro or the Meta Quest headsets. However, gamers would likely have first interacted with the technology in the mobile game Pokemon Go, which was released in 2016. Movie buffs might also reference the 2008 Iron Man movie, where J.A.R.V.I.S. gave a glimpse into what the future of AR could be.
Interestingly, the origins of this technology predate all these modern milestones. The first AR device was built in 1968 when US computer scientist Ivan Sutherland invented a head-mounted display that could place two-dimensional images within a 3D environment. It was a massively heavy device that was suspended from the ceiling and only showed rudimentary wireframe rooms. However, that invention became the foundational stone for AR in the decades to come.
What is augmented reality (AR)
One of the most interesting technologies in the modern world, AR is a conceptually straightforward mechanism. It creates a 2D or 3D overlay of digital information on top of a real-world environment. It essentially shows you information, objects, animation, or an entire visual scene in the middle of your environment. For example, the Pokemon Go game could place moving Pokemon on top of your table or on the floor as long as you pointed your camera towards it.
The Apple Vision Pro and Meta Quest have taken AR’s capabilities further. While watching objects or a movie with this effect is a cool novelty experience, true immersion comes when a user is not only able to view digital information in the real world but is also able to interact with it. And the mixed reality headsets allow exactly that. People using these headsets can watch a movie, play a game, or use an app that is essentially a floating screen in the middle of your room. It can be expanded or minimised based on your preference.
This growing understanding of augmented reality and the development of the hardware infrastructure to support such technology has today ensured that these experiences are not just limited to particular apps or high-end devices, but are present all around us to experience in a platformless environment. Today, multiple Indian companies are creating AR content that can be experienced right on the smartphone with one click of a button on a website or by scanning a QR code — without the need to purchase or download something.
Some major names in the space include Imaginate, Parallax Labs, PlayShifu, Xtendr, and more. Here’s an example to put it into the context of what exactly they are building. If you’re a fan of Maggi, you might remember that the company recently released its Korean BBQ noodles, where the package had a visible QR code. Scanning it would open an AR session, allowing you to cook the noodles in a 3D interactive space. The experience acted as a ‘how-to’ for those unfamiliar with the product and allowed people to play a mini-game without having to download or install anything.
This particular platformless AR experience was built by Xtendr, and there is a lot of nitty-gritty behind creating such experiences. To understand the technology behind such AR technology and its potential, we, at Gadgets 360, had a conversation with Anurag Sachdeva, the founder and CEO of the company.
Xtendr’s foray into platformless AR
The vision to build an AR-focused company is not a common one. But for Sachdeva, this path was paved by his vast 23 years of experience with tech-focused companies. Before trying his hands at entrepreneurship, he was heading Rovio Entertainment’s Angry Birds business for India, Southeast Asia, and the Middle East. The company did quite a bit of AR engagement; however, all of it was driven through the apps.
Being a marketing professional, Sachdeva wanted to tap into the engagement that comes with the immersive experience of AR, but without the limitations that an app brings. That idea led to the founding of Xtendr in May 2020. However, the path was not devoid of challenges. Starting a company in the middle of a global pandemic meant adjustments needed to be made to remain operational.
“We were the first in the region who were creating AR experiences in an appless manner. In the initial days, one concern was to understand our target demographic at a time when people were also dealing with the COVID-19 pandemic,” said Sachdeva. Despite the initial hiccup, the startup was able to pull in two brands, and the founder claims they saw great results in terms of engagement. This development gave Xtendr the momentum it needed.
Today, Xtendr has worked with more than 50 brands across India, the Middle East, and Southeast Asia. Some of the brands it has worked with include Redbull, Coca-Cola, ISRO, Unilever, Nestle Maggi, ITC Classmate, and more. But how exactly do these ‘platformless AR experiences’ work? We delved deeper.
Breaking down Xtendr’s AR experiences
Before jumping in, let us take a moment to understand what Xtendr does. The startup works with consumer brands and other organisations and creates branded content in the AR space. These can be anything from a small animation playing on your bed to a fully interactive cricket game that you can play in your garden. Once it has been activated via a URL or a QR code, it loads up on your device without the need to download an app. These experiences, also known as web AR, run entirely on the cloud.
However, building such experiences requires multiple components to work just right. The latency must be low so these experiences do not take forever to load on devices with poor internet connection. The experiences should be entirely AR-supported, create a sense of immersion, and be interactive.
Sachdeva told us that Xtendr uses spatial computing and artificial intelligence (AI) to create augmented reality. Immersion has been a prime focus for the company, which aims to “augment the human vision.”
The actual creation process involves first finding an idea and creating a narrative-led storytelling through characters, objects, and elements. Once the storyboard is done, the technical heavy lifting begins at the backend. The content is then converted into a program that can be executed seamlessly on the end user’s device.
The company primarily uses JavaScript to write the programs using the Three.js library. Web AR and VR frameworks such as A-Frame, SLAM (Simultaneous Localisation and Mapping), and React VR are also used in development. All this goes to the cloud, where the fronted comes together, and the AR experience is ready to be distributed.
The role of AI
Interestingly, the startup also incorporates generative AI into some AI experiences. The company uses large language models (LLMs) for characters in cases when they want to give users a conversational experience. AI is also used to generate art and let users experience AI-generated visual media.
These tools are used to design a gamified experience where the user can talk to a chatbot about the brand or do quirky things like painting the sky. Some AI models Xtendr has worked with include Inworld, Midjourney, ChatGPT, Gemini, Dall-E, and more.
Solving the latency problem
A key part of creating a platformless AR experience is also its distribution. Users would prefer their content to load in seconds, and ensuring that across budget to premium devices and across poor to excellent internet speeds is a big challenge. This becomes significant, especially as the experiences include 3D characters, complex artwork, and interactive elements.
Xtendr has created gamified engagements in the past where users could play a game and challenge another player online or on the leaderboard, where many others are also putting down their scores. With a single AR experience reaching up to two million impressions, it becomes important to ensure consistency.
The startup uses Amazon Web Services (AWS) for its cloud-based needs, which solves a part of the problem as the cloud computing giant handles some of the most traffic-driven platforms, such as Netflix, Twitch, LinkedIn, Facebook, and more. The other part is solved by optimising the 3D renders to ensure none is too heavy for any device. Sachdeva emphasised the need for balance between art and interaction, which contribute equally to engagement.
A look towards the future of augmented reality
For a technology that first arrived 56 years ago, AR remains in its nascent stage largely due to a lack of technology that could support the vision. But in the last few years, it has really come into the mainstream and has become viable and cost-effective. This is why companies are experimenting with it more and more. Today, AR has grown from a party trick to a utility tool. The digital overlay on a human vision is opening newer avenues for being productive online while still being connected to the real world instead of stuck to a screen.
Sachdeva holds firm that the future of AR is bright as all the key pieces needed to exponentially grow the technology are finally coming together. “5G is going to become mainstream in the next couple of years, which will solve many of the latency issues further. AR-supported hardware, high processing power, the emergence of generative AI and the entry of mixed reality headsets [by Apple and Meta], everything is coming together to create the next highway for immersive media,” he added.
