I have broad interests and knowledge within computer technology and product design, with greatest depth in computer vision, machine learning, analysis of user behavior, and intuitive gesture user experience design. My breadth has been an asset as it has allowed me to synthesize knowledge from a variety of areas to design new products. I was the technical cofounder of Reactrix in 2002, and spent several years there leading a team that created some incredible vision-based interactive experiences.
I am currently working full-time on a new startup called MatterPort — our aim is to make 3D capture as easy, as cheap, and as pervasive as digital photography.
Technology trends I care about:
There are many medium-term technology trends that I believe will reshape our world. Here are a few that I particularly care about:
- A merging of the physical and virtual worlds will begin to occur, giving us the advantages of both. Detailed knowledge about the physical world is becoming far easier to search, access, and manipulate online as sensor technologies and databases enable pervasive information capture. On the flip side, digital information about things in the physical world is becoming much easier to gather, share, and access while out and about. Over time, these two realms will coexist so thoroughly that they will be inseparable. Notions of presence and appearance will become fluid; people will be able to use telepresence robots to virtually appear in other parts of the physical world, and the experience of doing so will not differ significantly from controlling an avatar in a virtual world. Similarly, doing a search or inventory of objects in the physical world will be as fast and painless as a web search is today.
- More endeavors in society will come to resemble well-designed games. A well-designed game provides a smooth challenge ramp such that it never becomes so easy that it is boring and never becomes so hard that it is frustrating. The middle difficulty path provides the optimal amount of engagement, pleasure, and learning. As our lives become more quantified due to new data gathering and analysis tools, it will be easier for game mechanics to be built around specific tasks in our lives and for these game mechanics to be adjusted to create an engaging challenge ramp.
- Pervasive physical and mental health monitoring will prevent disease and optimize our performance. Once there are sensors on and inside our bodies that can detect the levels of various hormones, nutrients, and other chemicals in the body as well as our physical movements and actions, it will become easy to prevent a variety of diseases by catching them in their very early stages. By linking this information to information about the person’s physical whereabouts, potential disease threats in the environment can be detected and eliminated. In addition, by connecting the monitoring system to a suitable system for altering these levels, it will become possible to optimize the function of human body for any given situation. A similar set of tools will likely develop for monitoring and optimizing mental performance on given tasks, whether it’s giving a speech, learning a sport, studying for a test, or simply relaxing. In the short term, these mental tools can be as simple as ones that monitor our behavior at a computer; there’s plenty of behavioral information that can be gleaned from it.
- Manufactured products will become aware of their environment. With sensor and processor prices dropping, it will become cheap to imbue basic environmental awareness into products, allowing them to see, hear, feel, and understand their surroundings at a basic level. Objects will be able to react and respond to the presence of people nearby, allowing them to engage in a dialogue instead of passively waiting to be activated. Cheap networking technologies will allow an object’s observations and internal information to be shared, allowing people or other devices to be alerted to a potential issue if necessary.
- Improvements in computer vision will drive massive growth in robotics. Once robots can easily get a 3D map of their surroundings, many difficult object recognition, manipulation, and navigation tasks become a lot easier. This will eventually allow the automation of numerous tasks such as restocking of inventory, construction, maintenance, inspection, and mobile surveillance. People in those jobs will move to more supervisory roles, allowing the same number of people to accomplish far more.
- Mass customization aided by data mining will give everyone the products they want. Mass customization is the ability of an advanced manufacturing system to turn out products customized to the individual requirements of each individual customer. While we’re just beginning to see the era of mass customization, I think it will truly take off once people’s tastes can be automatically recognized by analysis of their existing likes and dislikes. While many detail-oriented and creative people love the customization process itself, the rest of the world would appreciate having a product tailored for them after a quick and minimal decision process.
- Social networks will become much better at helping us navigate landscapes of people and accompanying information in a focused manner. Current social networks tend not to filter by content very well, and have a lot of junk amidst the useful information. In addition, current interfaces are too linear, preventing users from being able to easily surf and absorb targeted information. However, better analysis of how users in a network respond to pieces of information and how these pieces of information relate to one another will allow information to be sorted as to usefulness and clustered into areas of interest.
My experience at Reactrix:
While in college I realized that grad school and academia were not the place for me. I wanted to be closer to the economic feedback cycle between the creators of technology and the end users.
After graduating, I spent some time working as a research engineer at Google before leaving to become the technical founder of my own company.
The company, Reactrix, created interactive video projections that bridged the gap between the physical and virtual worlds. Our central product was a 6x8ft projected image on the floor that reacted to the exact shapes of the users walking over the image, allowing them to, for example, kick a virtual soccer ball and have it react in a manner similar to a real soccer ball. Our first installation was in 2002, and by 2005 we had over 100 installations in malls and movie theaters throughout the US. By 2007, we had over 300 installations. Our revenue model was based on selling interactive advertisements on the displays; through numerous studies we were able to show that our interactive ads were far better than billboards and videos ads at getting people to pay attention, actively engage with the content, and, most importantly, remember the content.
In making the product work, we had to solve numerous challenges that had not been faced before, including:
- Making real time body tracking reliable in a wide range of unpredictable environments, including varying lighting conditions and floor conditions. At the time vision systems capable of doing this did not exist.
- Making real time body tracking reliable for people wearing any clothing and without the requirement for special sensors or markers.
- Doing the sensing, vision processing, and graphics quickly enough that people experienced no appreciable delay in interacting with virtual objects.
- Developing a vocabulary of gestures and visual feedback that would be simple and easy to learn in a few seconds by nontechnical people who had never seen the system before
- Developing a content creation engine that allowed artists with some scripting ability to create content that took the shape and position of bodies as input. At the time consumer multitouch systems and content authoring tools for them did not exist. In addition, the content creation system had to be fast and flexible enough to allow the creation of a wide variety of content while meeting the extremely tight timelines of the advertising world.
- Developing a system for automatically monitoring camera, computer, projector, and floor performance, diagnosing common issues, and delivering remote software and content upgrades. While such systems are standard now, at the time most digital displays in the out-of-home advertising market did not even have an internet connection.
- Creating interactive advertising content that was interesting enough that people would not only pay attention to it, but also play with it and remember it.
In early 2008 we also released an interactive wall display that allowed multiple users to interact using gestures (using no special equipment) at a distance of up to 15ft from the display. This display contained numerous innovations including highly reliable 3D vision software and hardware as well as intuitive easy-to-learn interfaces for buttons and other interface elements.
The company looked at many potential business models, including retail installations, special permanent one-off installations, tradeshows, and events, before deciding on building out an advertising network. The company took the aggressive risk of taking on the full expense of not only developing the technology but also deploying hundreds of displays and selling all the advertising.
Ultimately, this turned out to be a mistake for several reasons. History has shown that it takes around five years for new out-of-home media networks, especially ones utilizing new technologies, to break even. Compared to online advertising, the out-of-home advertising industry has been very slow to adapt to change. As a result, the company needed to take on an enormous amount of funding in order to run the network while advertising revenues were ramping up. This led to the company’s downfall when the market crashed in 2008 and funding dried up. Unfortunately, the large amount of funding we took on essentially locked us into this business model; we had too much invested in it to switch gears and focus on a different market.
Sections of the company also had a cost-is-no-object mentality that led to expenses being substantially higher than they should have been. For example, in our rush to secure exclusive rights to interactive ads in the malls, the company agreed to very high minimum revenue share payments to the malls, and this formed a significant chunk of our expenses in our final years when advertising revenue did not grow at the predicted rate.
There was a philosophical clash between the leaner, more careful hiring and spending approach on the engineering side and the money-is-no-object aggressive hiring and spending that existed in some other parts of the company. Aesthetically and intuitively, I’m more drawn to a bootstrapping approach that forces companies to have solid demonstrations of market success, preferably in the form of profitability, as soon as possible. While this approach is not always the correct one, it allows for early and frequent course adjustments on the path to success. The fact that several of our competitors, who had inferior technology at similar prices but stayed small and focused on selling interactive displays, are still around provides some validation of this.
In the end I got to develop some very exciting and groundbreaking products, and had the chance to work with some very talented individuals. For that I am grateful. The company also provided me with a long list of lessons, both in what to do and what not to do.