Reflecting on My Internship @ Spatial
For the past 5 months, I have been working as an AR/VR Software Engineering Intern at Spatial. Despite having had interest in the field for a while, I had a deep dive in AR/VR through this internship, and I am excited to share what I achieved and what I have learned through this opportunity.
Before I start, I would like to thank Mo for always guiding throughout the internship and being the coolest mentor ever, and Anand and Jinha for allowing me to have this truly amazing opportunity that have taught me priceless lessons and skills.
Articulated Hand Tracking: Interaction Formed by Your Instinct
One of the features that I worked on at Spatial was transitioning the interaction medium from the physical controllers to articulated hand tracking on Oculus Quest.
Oculus Quest, which has recently been one of our most popular platform, offers the ability to use your hands as controllers in the headset. Unlike the traditional methods of interaction, we can finally utilize what most of us have had since we were born, without the barrier of physical buttons and joysticks — removing a layer of boundary to bring the digital world closer to us. I was absolutely hooked as soon as I experienced this new feature.
Working closely with the design team, we went through tones of design iterations on how to make the new hand interactions as natural as possible, i.e.“Instinctual Interaction”. Designing a new system of interaction had many challenges, though.
For example, with the physical controllers, pushing the thumb stick left or right makes the user turn. How do we enable the same interaction with hand tracking? Sci-fi movies like Minority Report seems to tell us that using complicated gestures and swipe motions is the solution. But think about it: Gestures normally do not have anything inherently indicative about its corresponding action. Put it simply, your instinct doesn’t associate a V-sign or a swiping motion with the action of turning, unless explicitly trained. Hence, we cannot simply map the function of each controller button to a hand gesture.
With articulated hand tracking being such a new interaction medium, there aren’t any “right answers” of solving this problem either. How did we come up with a solution? Answer: Prototypes, prototypes, prototypes. We tried all sorts of stuff, including snap-turning by dragging & rotating the floor, teleporting with a Spider Man-like gesture, and so on.
Luckily, Mixed Reality Toolkit (MRTK) by Microsoft made prototyping a lot easier, with many of the non-trivial core features (e.g. teleportation) already implemented by MRTK, along with its automatic handling input events to the far- and near-interaction pointers. MRTK-Quest was also of great importance in implementing hand tracking, and Eric, the creator of the repo, was very helpful for resolving any issues we had with hand tracking.
The design iteration was a long process, but feeling that I am perhaps taking a glimpse into what the bearing of generational change in computer interaction may have looked like, such as the iPhone in 2007 or the Apple Machintosh in 1984, brought myself an overwhelming excitement. Applying the principle of instinctive interaction with the touchable turn buttons was an instance that brought an emotion like this.
I also worked closely with Jinha, the co-founders, who have dedicated his whole career to constantly redefining what it means to use computers, and exploring designs and features that no one in this world has figured out the right answers to yet (check out his TED talks for more details). Working with and learning from someone of this level of experience was truly an invaluable and inspiring experience.
A Few Things I learned
Communication is everything; document the work progress
Software engineering is a team effort; there are many engineers and designers involved in the process of finding solutions to issues, and issues themselves are also raised from diverse sources.Without documenting what I’ve been working on and where I am stuck, it becomes impossible to explain my thought process for any mildly non-trivial issue. Consequently, it becomes difficult to ask for help and to make use of the resources that I have available through the highly-skilled engineers of the team. Thus, writing down each step of the way as a concrete, coherent sentence is one of the most important skills for software engineers, arguably as much as the technical ones.
Utilize the issue tracker as much as possible
Using our Agile issue tracker software to organize and document my progress, add new issues that were brought up to me on Slack chats, and move issues to appropriate pipelines are quite simple things to do, but they were massively helpful for giving my brain a break from having to do a mental organization of my tasks, and for communicating what I am working and what I need help with.
Don’t be afraid to ask for help
This is probably one of the most brought up points with any internship reflection, and yet never fails to prove itself as one of the most valuable lessons. There are so many of extremely smart engineers and designers in the team, and they’ve always been open to help. The first point about documenting the work progress helps to make this easier.
Key Takeaway
This opportunity further cemented my interest in the XR field. I never thought that my first co-op job would be one that aligns so much with my passion, and I am extremely grateful that I got to work on something that makes my heart pound out of excitement.
With my next co-op term coming up in Winter 2021, I would love to build my expertise in the XR area with a deeper dive. Another part of myself, though, is compelling me to broaden my skill set and expand my horizon beyond just AR/VR. Regardless of what happens, I am truly excited to see what the future holds for me, and to find out what new things I will be learning over that journey.
