Achieving level-ups in life through software processes and gamification
Saturday, October 8, 2016
Progressive Enhancement for the VR Web
Modern VR developers
are wizards of the platform. Andrew Mo commented that they are the pioneers,
the ones who survived dysentery on the Oregon Trail. It was meant as a joke,
but every well timed joke carries more weight when it reflects a bit of the reality
and gravity of the situation. Modern VR developers really are thriving in the
ecosystem against all odds in tinier markets than their Mobile app and gaming
counterparts while meeting a performance curve that requires deep knowledge of
every limitation in the computing platform.
John Carmack, in his
keynote, said that the challenge present in Mobile VR development is like
dropping everyone a level. The AAA developers become indie devs, the indie devs
are hobbyists and the hobbyists have just failed.
If VR is dominated
by these early pioneers then where does the web fit in? These VR pioneering
familiar due to years of working with OpenGL, the language, performance and
build/packaging/deployment characteristics are all quite different from that of
a VR application in the store. Many new skills have to be employed to be
successful in the VR Web.
There is a shining
light in the dark here. Most people when they hear about WebVR immediately
think about games or experiences coded entirely in web technologies, in
technology will take or even just draw parallels with what VR is today. Since
today, VR is dominated by completely 3D immersive experiences, always inside of
a headset, it can be hard to imagine another, smaller step that we could take.
imagining. What does a smaller step look like? How do we progressively evolve
and enhance the web rather than assuming that we have to take the same major
leaps that the VR pioneers have made to date? How do we reduce our risk and
increase the chance of reward? How do we increase our target market size so
that it greatly exceeds the constraint of people with consumer grade VR already
Augmentations and Metadata
Our first goal has
to be that existing websites continue to serve billions of users. We need to
progressively update sites to have some VR functionality, but not make that a
requirement for interaction. Just like a favicon can be used by a site to
slightly customize the browsing experience and make their site stand out in
history, favorites or the bookmark bar, a VR ready site could supply a model,
photosphere, 360 video or even an entire scene. This extra content would be
hidden from the vast majority of users, but would be sniffed out by the next
generation of VR ready browsers and then used to improve the user experience.
One of the most
compelling options is to have your website provide an environment that can be
rendered around your page. In this way you can set up a simple scene, decide
where your content would get projected to and the browser would handle the rest
through open web standards such
as glTF. This isn't even a stretch of the imagination as a recent partnership between
OTOY and Samsung is working on the right standards. I was able to sync up
with Jules at OC3 and I have to say, there is a pretty big future in this
technology and I'm happy to add it to the list of simple things existing
website developers can do without having to spend years learning and working in
VR and 3D graphics. Stick a meta or link tag in your head, or push it down as
an http header (this is why meta+http-equiv is probably the best approach here)
and you'll get more mileage out of users with a VR headset.
It doesn't stop here
though. While this changes the environment your page runs in, it doesn't allow
you to really customize the iconography of your site the way a simple, 3D model
would be able to. Another glTF opportunity is in delivering favicon models that
a browser can use to represent sites in collections like the tab list, most
recently visited sites and favorites. A beautifully rendered and potentially
animated 3D model could go a long way to putting your website on the mantle of
everyone's future VR home space.
I think there is
more value to be had in the Web
Application Manifest specification too. For instance, why can't we specify
all of our screenshots, videos and icons for an installable store page? A VR
Browser would now be able to offer an install capability for your website that
looks beautiful and rivals any existing app-store. Or if you like the app-store
then you can specify the correct linkage and "prefer" your native
experience. The browser in this case would redirect to the platform store,
install your native application and off you go. I see this as exceptionally
valuable for an existing VR centric developer who wants to augment their
discovery through the web.
Our next goal is to
start incrementally nudging our users into the VR space. While augmentations
work for existing VR users and are ways to provide more VR like experiences on
the existing web, we can do even better. Instead of a VR only experience we can
build hybrid, WebGL+WebVR applications that are available to anyone with a
How does this work?
Well, we start with the commonality. Everyone has a browser today capable of
some basic WebGL. This means any experiences you build can be presented to that
user in 3D through the canvas tag in a kind of magic window. You can even go full
screen and deliver something pretty immersive.
To improve this
further, we can abstract input libraries that work across a large set of
devices. Touchpad, touch, mouse, keyboard, device orientation, gamepad and the
list continues to grow each day. By having a unified model for handling each
type of input you can have a great experience that the user can navigate with
mouse and keyboard or spin around in their chair and see through the window of
their mobile phone. Many of these experiences start bordering on the realism
and power of VR without yet delivering VR.
The last thing we do
to nail this hybrid experience is detect the presence of WebVR. We've landed a
property in the spec on the navigator object,
vrEnabled. This will return true if we think there is a chance the user could
use the device in VR. While I think there are some usability issues with such a
simple property that will result in maybe some browser UX for turning VR on and
off, this is a great start.
This is the next
level of write once, run anywhere, but its built on the concept of progressive
enhancement. Don't limit your user reach, instead scale your experience up as
you detect the capabilities required. I've recently stated that
this is a fundamental belief to our design of WebVR and I truly believe in
maintaining this approach as long as there are users who can benefit from these
I wanted to give an
example of one of these experiences and so here
you can see a 360 tour we've linked from our Oculus WebVR Developer portal.
There will be many more examples to come, but this will run in any browser and
progressively enable features as it detects them. My favorite is simply going
there in your phone and looking around using device motion.
I can't speak highly
enough of the value in building experiences like this. For this reason we at
Oculus and Facebook will be releasing numerous options for building these
experiences in minutes to hours rather than in days or more. Both content
generation and viewing needs to be made easier. We need to make great
photosphere export from pretty much any application (any developer should be
able to emit and share a photosphere from anywhere in their
game/application/experience), optimize how we transmit and display photospheres
with potential improvements to streaming them in, with simple to use libraries.
It has to be just as easy to extend this to use 360 video. Even a simple
looping 360 video could provide your user with AMAZING bits of animation and
value. Finally, extending this to make it interactive with text and buttons.
You can go further if you have the skills to do so, but the basic libraries
such as React VR will allow anyone to create great experiences with the above
out of the Way
Once we've extended
existing websites, gotten some simple libraries in place and people start to
build VR content, then final step is get the hell out of the way. This is
easier said than done. There is a large contingent of web technology naysayers
that have pointed out certain flaws in the web platform that make it non ideal
for VR. I'll give some examples so it is clear that I do listen and try to
resolve those issues.
tuned language. It is tuned by browser vendors based on the content out
there on the web today. Tomorrow, they could tune it differently. There
are so many knobs in the run-time. C++ is no different. There are knobs
you can tune when you pipe your code into cl.exe and it WILL result in
differences in the run-time behavior and speed of your application. The
browser is running in a chosen configuration to maximize the potential for
a set of sites determined by usage.
GC sucks. To which I reply,
yes, the general model of GC that is present in your average web runtime
does indeed suck. However, this is again, only one such configuration of a
GC. Most GC's are heuristic in nature and so we can tune those heuristics
to work amazingly well for VR or hybrid scenarios. I won't go into
details, but let's just say I have a list of work ready for when I hire my
GC expert into the Carmel team ;-)
Binding overhead is too high.
To which I respond, "Why?" There is no requirement that there be
any overhead. An enlightened JIT with appropriate binding meta-data can do
quite a bit here. While there are some restrictions in WebGL binding that slow
things down by-design I have high hopes that we can arrive at some
solutions to fix even that.
That list is not
exhaustive. When you have full control over the environment and aren't part of
a multi-tenant, collaborative runtime like a web browser, then you can be
insanely clever. But insanely clever is only useful for the top 1% of
applications once we enable anyone with a browser to build for VR. We need to
get out of the way and make the common cases for the remaining 99% of the
content not suck. That’s our space, that's our opportunity, that’s our goal.
Beyond the standard
arguments against VR and the web I think there are more practical issues
staring us in the face. The biggest one is the prevalence of full VR browsers.
The first fully featured VR browser is going to be a deep integration of the
web platform, OS, device and shell. There is simply too much going on today for
a web application to be able to run with zero hiccups and an overall lack of
measurement and feedback between the various sub-systems. Tuning everything
from the async event loop, thread priorities and which sub-systems are running
when in and out of VR is a very important set of next steps. Combining these
enhancements with UX and Input models that enable people to have a level of
trust in their to the extent that they could interact with a merchant in
virtual space, whatever form that might take.
Right now we are
experiencing the power of VR shells that plug into basic browser navigation
models and do simple content analysis/extraction. This is an early days approach that
only achieves a small fraction of the final vision.
For the graphics
stack we need some basics like HTML to Texture support in order to bring in the
full power of the 2D web into our 3D environments. I've been referring to 2D UX
as having a "density of information" that can't be easily achieved in
3D environments. We need this in order for VR experiences to enable deep
productivity and learning. Think about how often, in our "real" 3D
world you break out your phone, laptop, book and other 2D format materials to
quickly digest information. I don't think VR is any different in this regard.
John Carmack, during his keynote, also noted that there was huge value in 2D UX
because we've had so many years of experience developing. I believe this to be
true and think that an HTML to Texture will broaden the use cases for WebVR
We also need to enable full access to the pipeline. More and more I'm hearing about
the necessity for compute shaders and access to gl extensions like multi-view.
Even native is insufficient for delivering super high quality VR leading to advances
in hardware, OS and software. The web needs to provide access to these
capabilities quickly and safely. This may mean restrictions on how we compose
and host content, but we need to experiment and find those restrictions now
rather than holding the web back as a core VR graphics platform.
To close out, note
how this section details yet more progressive enhancement. Each of the new
capabilities and improvements we make will still take time to filter out to the
ecosystem. This is no different in the native world where extensions like
multi-view which have been defined for a couple of years are still not
uniformly distributed. So producing experiences that reach the largest possible
audience and gradually increase capability based on the device and browser
features you detect is going to be key.
Over the next few
months I'll be providing some tutorials talking about strategies and libraries
you can use to enable VR progressive enhancement for your existing sites. You
can also check out more of our demos and sign up for more information about our
libraries like React VR at the
Oculus Web VR Developer Portal.