Motion Tracking Cameras: a Brief History

You may remember some of the buzz around the Microsoft Kinect a decade ago – a camera with depth perception that allows you to become a part of some video games and lets you control them through gestures. Nowadays, motion recognition cameras such as the Kinect, also referred to as RGB-D cameras, offer more value in non-gaming settings.

To give you a sense of the possibilities of these 3D sensors beyond video games, we’ll run you through a brief history, from the Xbox 360 in 2010 to the StereoLabs ZED 2 in 2019.

Coming up in a future post, we’ll provide an overview of some of the projects for which we used this type of technology at PreviewLabs.

2010: Xbox 360 Kinect, More Than a Gimmick

The Kinect might be one of the most gimmicky products in gaming this past decade. Originally released for Microsoft’s Xbox 360, it aimed to hop on the motion controls wave set in motion by Nintendo’s Wii console in 2006. Through full-body gestures and voice commands, Xbox 360 owners could control their beloved console without ever needing a traditional controller. The Kinect SDK boasted appealing features like facial recognition and voice recognition.

The Kinect aimed to bring motion sensing camera technology to the masses as an Xbox 360 peripheral. While a lot of the gameplay enabled through the Kinect was truly exciting, in this marketing video, the capabilities of the device were somewhat exaggerated.

2010: OpenNI

The company that worked with Microsoft on the tech of the original Kinect, PrimeSense, was also the driving force behind OpenNI. This open source tool gave you access to the Kinect without the official SDK, and was released on the exact same day as the Kinect. This allowed people to start tinkering with the Kinect from day one on PC, allowing to get some value out of the device without ever connecting it to an Xbox. Apart from a variety of interesting hobbyist projects, this tool enabled some of the first business-to-business uses of the motion tracking technology.

Fun fact: While PrimeSense worked with Microsoft on the Kinect, it was acquired by Apple in 2014.

2011: Kinect for Windows

After one year, Microsoft made it a bit easier for PC owners to use and work with the Kinect. In 2011, they released a dedicated Windows-version of the Kinect, along with an SDK that enabled third party developers to unleash the full potential of the technology.

Kinect for Windows SDK

2012: Leap Motion

In 2012, Leap Motion Inc. introduced us to the Leap Motion Controller. This piece of PC hardware lets you control your computer with hand gestures and finger motions. Through two infrared (IR) cameras and several IR LEDs, the Leap Motion offers users the possibility to control applications without having to touch anything. Later the technology also expanded its territory to Virtual Reality (VR) where it was used as a hand tracking device. Even though the Leap Motion seems a lot more limited in its possibilities as a depth sensing camera, it also has the ability to determine the depth of the pixels it observes.

Leap Motion Inc. was acquired by rival company Ultrahaptics in 2019, after which the company changed its name to Ultraleap.

The Leap Motion Controller focuses on hand gestures - as seen for its use in VR (for non VR use, check out the original introduction video of the Leap Motion).

2013: Xbox One Kinect, Kinect’s Final Chapter on Xbox

In 2013, Microsoft announced it would bundle the new Xbox One Kinect with its upcoming console, so it would be possible for everyone to control it using the Kinect. However, this would have made the Xbox One a full $100 higher than the upcoming PlayStation 4. Shortly after, they ended up announcing that the Xbox One would be shipped without the Kinect as well, while making it available as a separate purchase.

The choice for the Kinect as required part of the Xbox One would have enabled developers to use it for their games without worrying about some players not getting to use these features, but this didn’t end up happening.

This update to the Kinect, also called the Kinect v2, featured cameras with significantly higher resolution and allowed for a higher frame rate, which in turn facilitated reading facial expressions, tracking up to six players (compared to two for the original Kinect), as well tracking the users’ heart rate.

Unfortunately, Kinect’s role in console gaming seems like a distant memory for as Microsoft cut their losses.

The depth sensing capabilities of the Xbox One Kinect were far more advanced than to its predecessor.

2014: Kinect v2, a Short Life Lived

After releasing a more advanced Kinect for Xbox One, Microsoft figured they also needed to make it available for Windows. However, it was discontinued in 2015 because Microsoft had released an adapter that made the Xbox One Kinect accessible with PC as well.

While the use of the Kinect for games declined, its technology would later show up in Microsoft’s AR headsets HoloLens and HoloLens 2, and eventually in the Azure Kinect (covered further on).

Kinect V2

2015: Stereolabs ZED Stereo Camera

The ZED Stereo Camera stepped in after Microsoft stopped production of the Kinect 2 in 2015. Quite literally, as the first cameras shipped in the same month Microsoft announced the end of the Kinect for Windows v2. The focus of developer Stereolabs seems more broad, since it markets the ZED for use in applications like VR, AR, drones, and robotics.

One of the key features that distinguished it from its competitors is the range at which it allows for depth perception – 20 meters, compared to 4.5 meters for the Kinect v2 according to its developer documentation.

In order for the ZED SDK to calculate the depth, it requires use of NVIDA graphics acceleration – which limits the amount of ZED cameras you can connect to a single computer.

The StereoLabs ZED camera's introduction video, showing video footage along with depth camera footage.

2015: Intel Also Sees Sense in Motion Tracking

Intel also enters the motion tracking camera sphere with the Intel RealSense product family. Since 2015, it has launched a variety of cameras, focusing on the tracking of movements and gestures, as well as depth. Currently, the Intel RealSense D400 series offers developers a variety of depth options. The Intel RealSense technology has been implemented in a variety of laptops and tablet computers from various manufacturers since its launch.

One of the advantages of the Intel RealSense over the ZED is that it can be connected to pretty much any computer, without requiring specific other hardware or without severely limiting the amount of cameras that can be connected.

Intel RealSense

2017: iPhone X, the First Depth Sensing Smartphone

In 2017, Apple broke new ground for camera technology in smartphones. It introduced the very first depth sensing camera to its iPhone lineup with the iPhone X. Their TrueDepth sensing technology made features like Face ID and Animoji (later rebranded to Memoji) possible. The software developed to enable these features in the flagship phone came from the same people behind the first Kinect: PrimeSense.

The iPhone X enabled depth sensing technology for the first time to the iconic smartphone brand.

2018: Stereolabs ZED Mini

Stereolabs followed the ZED up by a Stereolabs ZED Mini in 2018, as a smaller camera, designed to be attached to VR headsets.

Some would disagree, but in the case of the Sterolabs ZED, smaller doesn't necessarily mean less impressive…

2018: Azure Kinect, the Future Is Intelligent

For a while, the discontinuation of the Xbox One Kinect seemed like the end of Microsoft’s foray into motion tracking cameras. Current Xbox One consoles (Xbox One S and Xbox One X) don’t even have a dedicated port for the Kinect anymore as the hardware was discontinued in 2017. At Microsoft there were still believers, though, resulting in the announcement of the Azure Kinect in 2018. With this new Kinect, the company officially doubles down on providing business value and steps away from gaming applications. Built to be integrated with the Azure Cloud technology, the Azure Kinect’s marketing focusses on Artificial Intelligence (AI) usage. It features the same time-of-flight camera as the HoloLens 2, seven microphones and a camera, all bundled in a compact design that clearly steps away from the design form of previous iterations.

Azure Kinect leaves the gaming past of the brand far behind it, focusing on business applications.

2019: Stereolabs ZED 2

The Stereolabs ZED 2 continued along the same vein as it predecessor, but including a number of improvements among which a larger field of view. Perhaps the largest improvements are within the software as they included features such as skeletal tracking and multi camera synchronization, features which the Kinect and Azure Kinect had for a while.

Skeleton tracking at scale with Stereolabs ZED 2.

2020: iPhone 12 with LiDAR

The iPhone 12’s LiDAR enabled camera give the phone a sense of depth perception which was previously only possible with the front facing camera on iPhone X and later models, enabling a range of new possibilities, including 3D scanning, enabled by Apple’s Depth API. This puts the power of depth perception right in your hands. Given the recent release of the device, we’re looking forward to see the extent at which this will be embraced by the development community.

iPhone 12 Pro being used to quickly capture 3D objects

How we Use this Technology

At PreviewLabs, we’ve been prototyping and experimenting with depth sensing motion cameras ever since the first Kinect hit the shelves.

Are you looking to leverage the power of motion cameras for your own project? Reach out to us, we will gladly help you build the prototypes to test your ideas.

Leave a comment

Your email address will not be published. Required fields are marked *