INSUBCONTINENT EXCLUSIVE:
Matt Miesnieks
Contributor
Matt Miesnieks is a partner at Super Ventures.
More posts by this
contributor
Why the YouTube of AR won&t beYouTube
The product design challenges of AR on smartphones
The martial arts actor
Jet Li turned down a role in the Matrix and has been invisible on our screens because he does not want his fighting moves 3D-captured and
Soon everyone will be wearing 3D-capable cameras to support augmented reality (often referred to as mixed reality) applications
Everyone will have to deal with the sorts of digital-capture issues across every part of our life that Jet Li avoided in key roles and
musicians have struggled to deal with since Napster
AR means anyone can rip, mix and burn reality itself.
Tim Cook has warned the industry about &the data industrial complex& and advocated for
It doesn&t take too much thinking about where some parts of the tech industry are headed to see AR ushering in a dystopian future where we
are bombarded with unwelcome visual distractions, and our every eye movement and emotional reaction is tracked for ad targeting
But as Tim Cook also said, &it doesn&t have to be creepy.& The industry has made data-capture mistakes while building today tech platforms,
and it shouldn&t repeat them.
Dystopia is easy for us to imagine, as humans are hard-wired for loss aversion
This hard-wiring refers to people tendency to prefer avoiding a loss versus an equal win
It better to avoid losing $5 than to find $5
It an evolutionary survival mechanism that made us hyper-alert for threats
The loss of being eaten by a tiger was more impactful than the gain of finding some food to eat
When it comes to thinking about the future, we instinctively overreact to the downside risk and underappreciate the upside benefits.
How can
we get a sense of what AR will mean in our everyday lives, that is (ironically) based in reality
When we look at the tech stack enabling AR,
it important to note there is now a new type of data being captured, unique to AR
It the computer vision-generated, machine-readable 3D map of the world
AR systems use it to synchronize or localize themselves in 3D space (and with each other)
The operating system services based on this data are referred to as the &AR Cloud.& This data has never been captured at scale before, and
the AR Cloud is 100 percent necessary for AR experiences to work at all, at scale.
Fundamental capabilities such as persistence, multi-user
and occlusions outdoor all need it
Imagine a super version of Google Earth, but machines instead of people use it
This data set is entirely separate to the content and user data used by AR apps (e.g
login account details, user analytics, 3D assets, etc.).
The AR Cloud services are often thought of as just being a &point cloud,& which
leads people to imagine simplistic solutions to manage this data
This data actually has potentially many layers, all of them providing varying degrees of usefulness to different use cases
The term &point& is just a shorthand way of referring to a concept, a 3D point in space
The data format for how that point is selected and described is unique to every state-of-the-art AR system.
The critical thing to note is
that for an AR system to work best, the computer vision algorithms are tied so tightly to the data that they effectively become the same
Apple ARKit algorithms wouldn&t work with Google ARCore data even if Google gave them access
Same for HoloLens, Magic Leap and all the startups in the space
The performance of open-source mapping solutions are generations behind leading commercial systems.
So we&ve established that these &AR
Clouds& will remain proprietary for some time, but exactly what data is in there, and should I be worried that it is being
collected
AR makes it possible to capture everything…
The list of data that could be saved is long
At a minimum, it the computer vision (SLAM) map data, but it could also include a wireframe 3D model, a photo-realistic 3D model and even
real-time updates of your &pose& (exactly where you are and what you are looking at), plus much more
Just with pose alone, think about the implications on retail given the ability to track foot traffic to provide data on the best merchandise
placement or best locations for ads in store (and at home).
The lower layers of this stack are only useful to machines, but as you add more
layers on top, it quickly starts to become very private
Take, for example, a photo-realistic 3D model of my kid bedroom captured just by a visitor walking down the hall and glancing in while
wearing AR glasses.
There no single silver bullet to solving these problems
Not only are there many challenges, but there are also many types of challenges to be solved.
Tech problems that are solved and need to be
applied
Much of the AR Cloud data is just regular data
It should be managed the way all cloud data should be managed
Good passwords, good security, backups, etc
GDPR should be implemented
In fact, regulation might be the only way to force good behavior, as major platforms have shown little willingness to regulate themselves
Europe is leading the way here; China is a whole different story.
A couple of interesting aspects to AR data are:
Similar to Maps or
Streetview, how &fresh& should the data be, and how much historical data should be saved
Do we need to save a map with where your couch was positioned last week What scale or resolution should be saved
There little value in a cm-scale model of the world, except for a map of the area right around you.
The biggest aspect that is difficult but
doable is no personally identifying information leaves the phone
This is equivalent to the image data that your phone processes before you press the shutter and upload it
Users should know what is being uploaded and why it is OK to capture it
Anything that is personally identifying (e.g
the color texture of a 3D scan) should always be opt-in and carefully explained how it is being used
Homomorphic transformations should be applied to all data that leaves the device, to remove anything human readable or identifiable, and yet
still leave the data in a state that algorithms can interpret for very specific relocalization functionality (when run on the device).
There
also the problem of &private clouds& in that a corporate campus might want a private and accurate AR cloud for its employees
This can easily be hosted on a private server
The tricky part is if a member of the public walks around the site wearing AR glasses, a new model (possibly saved on another vendor
platform) will be captured.
Tech challenges the AR industry still needs to solve
There are some problems we know about, but we don&t know
Examples are:
Segmenting rooms: You could capture a model of your house, but one side of an inner apartment wall is your apartment while the
other side is someone else apartment
Most privacy methods to date have relied on something like a private radius around your GPS location, but AR will need more precise ways to
detect what is &your space.&
Identifying rights to a space is a massive challenge
Fortunately, social contracts and existing laws are in place for most of these problems, as AR Cloud data is pretty much the same as
There are public spaces, semi-public (a building lobby), semi-private (my living room) and private (my bedroom)
The trick is getting the AR devices to know who you are and what it should capture (e.g
my glasses can capture my house, but yours can&t capture my house).
Managing the capture of a place from multiple people, and stitching that
into a single model and discarding overlapping and redundant data makes ownership of the final model tricky.
The Web has the concept of a
robots.txt file, which a website owner can host on their site, and the web data collection engines (e.g
Google, etc.) agree to only collect the data that the robots.txt file asks them to
Unsurprisingly this can be hard to enforce on the web, where each site has a pretty clear owner
Some agreed type of &robots.txt& for real-world places would be a great (but maybe unrealistic) solution
Like web crawlers, it will be hard to force this on devices, but like with cookies and many ad-tracking technologies, people should at least
be able to tell devices what they want and hopefully market forces or future innovations can require platforms to respect it
The really hard aspect of this attractive idea is &whose robots.txt is authoritative for a place.& I shouldn&t be able to create a
robots.txt for Central Park in NYC, but I should for my house
How is this to be verified and enforced
Social contracts need to emerge and be adopted
A big part of solving AR privacy problems will come
from developing a social contract that identifies when and where it appropriate to use a device
When camera phones were introduced in the early 2000s, there was a mild panic about how they could be misused; for example, cameras used
secretly in bathrooms or taking your photos in public without a person permission
The OEMs tried to head off that public fear by having the cameras make a &click& sound
Adding that feature helped society adopt the new technology and become accustomed to it pretty quickly
As a result of having the technology in consumers hands, society adopted a social contract — learning when and where it is OK to hold up
your phone for a picture and when it is not.
… [but ]the platform doesn&t need to capture everything in order to deliver a great AR
UX.
Companies added to this social contract, as well
Sites like Flickr developed policies to manage images of private places and things and how to present them (if at all)
Similar social learning took place with Google Glass versus Snap Spectacles
Snap took the learnings from Glass and solved many of those social problems (e.g
they are sunglasses, so we naturally take them off indoors, and they show a clear indicator when recording)
This is where the product designers need to be involved to solve the problems for broad adoption.
Challenges the industry cannot predict
AR
New mediums come along only every 15 years or so, and no one can predict how they will be used
SMS experts never predicted Twitter and Mobile Mapping experts never predicted Uber
Platform companies, even the best-intentioned *will* make mistakes.
These are not tomorrow challenges for future generations or science
The product development decisions the AR industry is making over the next 12-24 months will play out in the next five years.
This is where
AR platform companies are going to have to rely on doing a great job of:
Ensuring their business model incentives are aligned with doing the
right thing by the people whose data they capture; and
Communicating their values and earning the trust of the people whose data they
Values need to become an even more explicit dimension of product design
Apple has always done a great job of this
Everyone needs to take it more seriously as tech products become more and more personal.
What should the AR players be doing today to not be
creepy
Here what needs to be done at a high level, which pioneers in AR believe is the minimum:
Personal Data Never Leaves Device, Opt In
Only: No personally identifying data required for the service to work leaves the device
Give users the option to opt in to sharing additional personal data if they choose for better apps feedback
Personal data does NOT have to leave the device in order for the tech to work; anyone arguing otherwise doesn&t have the technical skills
and shouldn&t be building AR platforms.
Encrypted IDs: Coarse Location IDs (e.g
Wi-Fi network name) are encrypted on the device, and it not possible to tell a location from the GPS coordinates of a specific SLAM map
file, beyond generalities.
Data Describing Locations Only Accessible When Physically at Location: An app can&t access the data describing
a physical location unless you are physically in that location
That helps by relying on the social contract of having physical permission to be there, and if you can physically see the scene with your
eyes, then the platform can be confident that it OK to let you access the computer vision data describing what a scene looks
like.
Machine-Readable Data Only: The data that does leave the phone is only able to be interpreted by proprietary homomorphic algorithms
No known science should be able to reverse engineer this data into anything human readable.
App Developers Host User Data On Their
Servers, Not The Platforms: App developers, not the AR platform company, host the application and end user-specific data re: usernames,
logins, application state, etc
The AR Cloud platform should only manage a digital replica of reality
The AR Cloud platform can&t abuse an app user data because they never touch or see it.
Business Models Pay for Use Versus Selling Data: A
business model based on developers or end users paying for what they use ensures the platform won&t be tempted to collect more than
Don&t create financial incentives to collect extra data to sell to third parties.
Privacy Values on Day One: Publish your values around
privacy, not just your policies, and ask to be held accountable to them
There are many unknowns, and people need to trust the platform to do the right thing when mistakes are made
Values-driven companies like Mozilla or Apple will have a trust advantage over other platforms whose values we don&t know.
User and
Developer Ownership and Control: Figure out how to give end users and app developers appropriate levels of ownership and control over data
that originates from their device
The goal (we&re not there yet) should be to support GDPR standards globally.
Constant Transparency and Education: Work to educate the
market and be as transparent as possible about policies and what is known and unknown, and seek feedback on where people feel &the line&
should be in all the new gray areas
Be clear on all aspects of the bargain that users enter into when trading some data for a benefit.
Informed Consent, Always: Make a
sincere attempt at informed consent with regard to data capture (triply so if the company has an ad-based business model)
This goes beyond an EULA, and IMO should be in plain English and include diagrams
Even then, it impossible for end users to understand the full potential.
Even apart from the creep factor, remember there always the chance
that a hack or a government agency legally accesses the data captured by the platform
You can&t expose what you don&t collect, and it doesn&t need to be collected
That way people accessing any exposed data can&t tell precisely where an individual map file refers to (the end user encrypts it, the
platform doesn&t need the keys), and even if they did, the data describing the location in detail can&t be interpreted.
There no
single silver bullet to solving these problems.
Blockchain is not a panacea for these problems — specifically as applied to the
foundational AR Cloud SLAM data sets
The data is proprietary and centralized, and if managed professionally, the data is secure and the right people have the access they need
There no value to the end user from blockchain that we can find
However, I believe there is value to AR content creators, in the same way that blockchain brings value to any content created for mobile
There nothing inherently special about AR content (apart from a more precise location ID) that makes it different.
For anyone interested,
the Immersive Web working group at W3C and Mozilla are starting to dig further into the various risks and mitigations.
Where should we put
our hope
This is a tough question
AR startups need to make money to survive, and as Facebook has shown, it was a good business model to persuade consumers to click OK and let
the platform collect everything
Advertising as a business model creates inherently misaligned incentives with regard to data capture
On the other hand, there are plenty of examples where capturing data makes the product better (e.g
Waze or Google search).
Education and market pressure will help, as will (possibly necessary) privacy regulation
Beyond that we will act in accordance with the social contracts we adopt with each other re: appropriate use.
The two key takeaways are that
AR makes it possible to capture everything, and that the platform doesn&t need to capture everything in order to deliver a great AR UX.
If
you draw a parallel with Google, in that web crawling was trying to figure out what computers should be allowed to read, AR is widely
distributing computer vision, and we need to figure out what computers should be allowed to see.
The good news is that the AR industry can
avoid the creepy aspects of today data collection methods without hindering innovation
The public is aware of the impact of these decisions and they are choosing which applications they will use based on these issues
Companies like Apple are taking a stand on privacy
And most encouragingly, every AR industry leader I know is enthusiastically engaged in public and private discussions to try to understand
and address the realities of meeting the challenge.
