Experiments

Hey you, get inside my computer!

Hey you, get inside my computer!

People like to see themselves in photos and videos, and I’m no exception to that. I’m sure part of it has to do with vanity, but I like to think a larger part of it is being able to see yourself as others see you in the world. Modern technology makes this pretty easy and we take it largely for granted, but imagine living at a time when affordable good quality mirrors first became affordable for the masses and the impact that had on the psyche of human beings. Not to mention fashion and make-up :P

Mirrors, cameras, audio recording devices, and video recorders have all allowed people get a glimpse of how other people see and interact with them. It’s both fascinating and disturbing to see yourself from this vantage point. It can be hard to reconcile your everyday experience in the first person with what you see and hear of yourself from an occasional third party perspective.

I’m sure everyone alive today can remember times from their childhood, and even more recently, that they’ve used technology for the purpose of seeing how they fit into the world. If you say otherwise, I’ll remind you that it’s a safe bet one of the first things you did when you woke up this morning was look into a mirror.

In the last few years technology has reached a point where it is affordable for people to create full colour 3D scans of themselves. Now for the first time you can see a version of yourself from any angle, any lens length, and any lighting condition by tweaking a couple of variables on a computer. It’s technology that’s not quite in every home or phone just yet, but it will be soon. The rise in the popularity of 3D printing will make sure of that.

This is the first in a series of blog posts I’ll be doing that talks about the investigations we’ve done into 3D scanning people over the last year and a half. I’ll try my best to balance technical details of how it’s done with descriptions of practical applications and potential uses for technology. I’ll start this first post with a description of the primary hardware and software we use to get people into computers.

Yes dear, I really do need to buy a XBox for work.

The Xbox/Microsoft Kinect is an amazing piece of technology, not only for what it does but for the fact that it is a consumer product (80 million XBox 360s sold worldwide according to Wikipedia - not all have Kinects though). From a technical perspective what makes it cool is that it captures z-depth data along with image data. The image data shows you what colour something is at a particular pixel of a image, but z-depth data tells you how far away from the camera the thing that is displayed at that pixel is.

We purchased a Kinect back in 2011 and our first experiments with it were using the excellent open source library RGBDToolkit. I found the point clouds that the Kinect generated particularly beautiful. That a device that only costs $109.99 could generate a point cloud like the one in the Radiohead House of Cards video, and improve upon that by adding colour, was very exciting. It was one of those rare moments where it felt like the world just took a giant step forward.

It wasn’t until mid 2012 that I started to look around for software that could be used to turn a Kinect into a proper 3D scanner. Of the solutions I found at that time the only one that could capture geometry and colour data at the same time was KScan3D ($299US). There are other options now that can do this like ReconstructMe, Skanect, and Scenect - I haven’t tried them myself, but having quickly reviewed their workflow and features before writing this post, KScan3D appears to still be the best choice for my purposes. Mainly for the high degree of control and flexibility over how the model is assembled.

KScan3D works by capturing multiple snapshots of image and z-depth data from the Kinect that you can then arrange and align into a complete 360º 3D shell of an object/person. The individual snapshots start as point clouds that are then turned into meshes which you can smooth, and trim points from, in advance of combining them together. These adjustment features can be very useful. For example, when colour data has been attached to a surface of the scan which is close to perpendicular to the scanning image plane, the image data can appear distorted when it is projected across the surface. Being able to cut out the distorted bad bits improves the final results a great deal.

The software can also help you align the snapshots using either “mesh alignment” or “fine alignment”. “Mesh alignment” can help you roughly line up two similar snapshots, even if they are oriented in different directions. However, depending on how far you have moved the Kinect between snap shots and how much commonality there is between the two resulting meshes, “mesh alignment” can get it wrong. In which case you will have to manually orient and align the snapshots. This can be labour intensive, but it’s always good to have the ability to manually override something if you don’t like how it was automatically oriented. Once two scans get fairly close though, “fine alignment” does a excellent job of doing the detailed alignment (as you would expect) of the meshes.

Once the meshes are aligned you can “combine” them into a single object. A combined object can then be “finalized”, which calculates an averaged 3D surface and colour for the model using all the vertices (vertex, vertices plural, is a fancy work for point) in the combined object. It also applies a Poisson filter, which fills in gaps and holes in the model, and provides an option to decimate the resulting model, which reduces the number of points and polygons. All of these functions provide a number of options and controls to allow for good control over how the final model turns out.

Considering it was captured with such an inexpensive piece of equipment, the geometry the Kinect produces is pretty good, albeit the quality of the colour data is a bit lackluster. It is, after all, captured with what is basically just a 640x480 VGA webcam in the Kinect, so there will be video noise and grain in the image data. The Kinect also automatically adjusts its exposure and white balance, so unless you have good, even diffused light, it may capture the subject with different degrees of brightness and exposure. KScan3D does a reasonably good job of mitigating these issues when it finalizes the model and averages the image data captured from multiple captures. I’ve read that ASUS’s Kinect clone has better colour image quality than the XBox Kinect, but I haven’t had and opportunity to try it yet. It’s on my todo list though.

In subsequent posts in this series I’ll talk about how we avoid the “uncanny valley” when scanning people. I'll also detail the workflow we use to take a model from KScan3D and convert it so that it can be used with Adobe Flash with Away3D, WebGL, Unity 3D, as well as for 3D printing.

Clayton Partridge's picture
BY: Clayton Partridge
FOUNDING PARTNER

Clayton is a developer, and likes to make things out of other things.

comments powered by Disqus