PreviewLabs

You may remember some of the buzz around the Microsoft Kinect a decade ago – a camera with depth perception that allows you to become a part of some video games and lets you control them through gestures. Nowadays, motion recognition cameras such as the Kinect, also referred to as RGB-D cameras, offer more value in non-gaming settings.

To give you a sense of the possibilities of these 3D sensors beyond video games, we’ll run you through a brief history, from the Xbox 360 in 2010 to the StereoLabs ZED 2 in 2019.

Coming up in a future post, we’ll provide an overview of some of the projects for which we used this type of technology at PreviewLabs.

2010: Xbox 360 Kinect, More Than a Gimmick

The Kinect might be one of the most gimmicky products in gaming this past decade. Originally released for Microsoft’s Xbox 360, it aimed to hop on the motion controls wave set in motion by Nintendo’s Wii console in 2006. Through full-body gestures and voice commands, Xbox 360 owners could control their beloved console without ever needing a traditional controller. The Kinect SDK boasted appealing features like facial recognition and voice recognition.

2010: OpenNI

The company that worked with Microsoft on the tech of the original Kinect, PrimeSense, was also the driving force behind OpenNI. This open source tool gave you access to the Kinect without the official SDK, and was released on the exact same day as the Kinect. This allowed people to start tinkering with the Kinect from day one on PC, allowing to get some value out of the device without ever connecting it to an Xbox. Apart from a variety of interesting hobbyist projects, this tool enabled some of the first business-to-business uses of the motion tracking technology.

Fun fact: While PrimeSense worked with Microsoft on the Kinect, it was acquired by Apple in 2014.

2011: Kinect for Windows

After one year, Microsoft made it a bit easier for PC owners to use and work with the Kinect. In 2011, they released a dedicated Windows-version of the Kinect, along with an SDK that enabled third party developers to unleash the full potential of the technology.

2012: Leap Motion

In 2012, Leap Motion Inc. introduced us to the Leap Motion Controller. This piece of PC hardware lets you control your computer with hand gestures and finger motions. Through two infrared (IR) cameras and several IR LEDs, the Leap Motion offers users the possibility to control applications without having to touch anything. Later the technology also expanded its territory to Virtual Reality (VR) where it was used as a hand tracking device. Even though the Leap Motion seems a lot more limited in its possibilities as a depth sensing camera, it also has the ability to determine the depth of the pixels it observes.

Leap Motion Inc. was acquired by rival company Ultrahaptics in 2019, after which the company changed its name to Ultraleap.

2013: Xbox One Kinect, Kinect’s Final Chapter on Xbox

In 2013, Microsoft announced it would bundle the new Xbox One Kinect with its upcoming console, so it would be possible for everyone to control it using the Kinect. However, this would have made the Xbox One a full $100 higher than the upcoming PlayStation 4. Shortly after, they ended up announcing that the Xbox One would be shipped without the Kinect as well, while making it available as a separate purchase.

The choice for the Kinect as required part of the Xbox One would have enabled developers to use it for their games without worrying about some players not getting to use these features, but this didn’t end up happening.

This update to the Kinect, also called the Kinect v2, featured cameras with significantly higher resolution and allowed for a higher frame rate, which in turn facilitated reading facial expressions, tracking up to six players (compared to two for the original Kinect), as well tracking the users’ heart rate.

Unfortunately, Kinect’s role in console gaming seems like a distant memory for as Microsoft cut their losses.

2014: Kinect v2, a Short Life Lived

After releasing a more advanced Kinect for Xbox One, Microsoft figured they also needed to make it available for Windows. However, it was discontinued in 2015 because Microsoft had released an adapter that made the Xbox One Kinect accessible with PC as well.

While the use of the Kinect for games declined, its technology would later show up in Microsoft’s AR headsets HoloLens and HoloLens 2, and eventually in the Azure Kinect (covered further on).

2015: Stereolabs ZED Stereo Camera

The ZED Stereo Camera stepped in after Microsoft stopped production of the Kinect 2 in 2015. Quite literally, as the first cameras shipped in the same month Microsoft announced the end of the Kinect for Windows v2. The focus of developer Stereolabs seems more broad, since it markets the ZED for use in applications like VR, AR, drones, and robotics.

One of the key features that distinguished it from its competitors is the range at which it allows for depth perception – 20 meters, compared to 4.5 meters for the Kinect v2 according to its developer documentation.

In order for the ZED SDK to calculate the depth, it requires use of NVIDA graphics acceleration – which limits the amount of ZED cameras you can connect to a single computer.

2015: Intel Also Sees Sense in Motion Tracking

Intel also enters the motion tracking camera sphere with the Intel RealSense product family. Since 2015, it has launched a variety of cameras, focusing on the tracking of movements and gestures, as well as depth. Currently, the Intel RealSense D400 series offers developers a variety of depth options. The Intel RealSense technology has been implemented in a variety of laptops and tablet computers from various manufacturers since its launch.

One of the advantages of the Intel RealSense over the ZED is that it can be connected to pretty much any computer, without requiring specific other hardware or without severely limiting the amount of cameras that can be connected.

2017: iPhone X, the First Depth Sensing Smartphone

In 2017, Apple broke new ground for camera technology in smartphones. It introduced the very first depth sensing camera to its iPhone lineup with the iPhone X. Their TrueDepth sensing technology made features like Face ID and Animoji (later rebranded to Memoji) possible. The software developed to enable these features in the flagship phone came from the same people behind the first Kinect: PrimeSense.

2018: Stereolabs ZED Mini

Stereolabs followed the ZED up by a Stereolabs ZED Mini in 2018, as a smaller camera, designed to be attached to VR headsets.

2018: Azure Kinect, the Future Is Intelligent

For a while, the discontinuation of the Xbox One Kinect seemed like the end of Microsoft’s foray into motion tracking cameras. Current Xbox One consoles (Xbox One S and Xbox One X) don’t even have a dedicated port for the Kinect anymore as the hardware was discontinued in 2017. At Microsoft there were still believers, though, resulting in the announcement of the Azure Kinect in 2018. With this new Kinect, the company officially doubles down on providing business value and steps away from gaming applications. Built to be integrated with the Azure Cloud technology, the Azure Kinect’s marketing focusses on Artificial Intelligence (AI) usage. It features the same time-of-flight camera as the HoloLens 2, seven microphones and a camera, all bundled in a compact design that clearly steps away from the design form of previous iterations.

2019: Stereolabs ZED 2

The Stereolabs ZED 2 continued along the same vein as it predecessor, but including a number of improvements among which a larger field of view. Perhaps the largest improvements are within the software as they included features such as skeletal tracking and multi camera synchronization, features which the Kinect and Azure Kinect had for a while.

2020: iPhone 12 with LiDAR

The iPhone 12’s LiDAR enabled camera give the phone a sense of depth perception which was previously only possible with the front facing camera on iPhone X and later models, enabling a range of new possibilities, including 3D scanning, enabled by Apple’s Depth API. This puts the power of depth perception right in your hands. Given the recent release of the device, we’re looking forward to see the extent at which this will be embraced by the development community.

How we Use this Technology

At PreviewLabs, we’ve been prototyping and experimenting with depth sensing motion cameras ever since the first Kinect hit the shelves.

Are you looking to leverage the power of motion cameras for your own project? Reach out to us, we will gladly help you build the prototypes to test your ideas.