Kinect SDK 1.0 - 1 - Introduction to the API

16. April 2012 16:21 by Renaud in   //  Tags:   //   Comments (0)

 

 1. Introduction to the API
 2. Use the ColorImageStream
 3. Track the users with the SkeletonStream
 4. Kinect in depth!
 5. Speech recognition

It's been 2 months now since Kinect for Windows SDK 1.0 is out. We are of course waiting for the next release (SDK 1.5 announced for late May), but for now I'd like to introduce to you the current version, so that you'll be ready to welcome what's coming in the near future !

For this blogpost, I assume you satisfy the following requirements:

  • You should know the basics of C# (VB.NET is working also, but all my project samples are written in C# ! )
  • You need Visual Studio 2010 (if you don't have it yet, install the free Visual Studio 2010 Express edition now)
  • And the last but not least, you need a Kinect device!

 

Hardware overview

I won't go too deep in the details, because that's not really what is important for us, developers. The Kinect is built with a standard camera, and an IR emitter coupled with a receiver. Together they will create a depth map of the environment. At the bottom of the Kinect, you can find 4 microphones. The Kinect sensor stands on a tilt motor, and you can change its elevation angle with a few lines of code.

Download and install the SDK

To go further, download the SDK 1.0 by clicking on the image below.

Télécharger le Kinect SDK 1.0

Before installing it, you should first make sure that you have uninstalled any previous version. If everything went fine during the installation, you should now see 4 new programs in you Control Panel.

Installation completed

Microsoft shipped some good project samples with the SDK. They will be very helpful in a few minutes. If you didn't installed them yet, you can launch the Kinect SDK Sample Browser from the start menu :

You can have a look at all the projects available, but the one we really need for the moment is the KinectExplorer package.

Theoretical basics

The Kinect SDK 1.0 API doesn't contain a lot of classes, and is quite easy to use it. You'll find all the documentation you need in the installation folder : ~ProgramsMicrosoft SDKsKinectv1.0DocumentationKinectSDK.chm or on the MSDN website : Kinect for Windows SDK et Microsoft.Kinect namespace.

The main object in a Kinect project is of course the KinectSensor. One instance of that class gives you control over one Kinect device plugged in to your computer. Then you'll focus on the 3 streams (color, depth, skeleton), that will give you access to all the information that the Kinect collects for you!

First, let's have a look at how to detect if at least one device is plugged in (since the SDK 1.0, it's possible to use up to 4 devices simultaneously) !

Open Visual Studio and create a first WPF project. Add a new reference to the Microsoft.Kinect.dll assembly, and edit the MainWindow class:

MainWindow.xaml.cs
        public KinectSensor Kinect {get; set;}
        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
            // Walk through KinectSensors to find the first one with a Connected status
            var firstKinect = (from k in KinectSensor.KinectSensors
                               where k.Status == KinectStatus.Connected
                               select k).FirstOrDefault();

            if (firstKinect != null)
                SetNewKinect(firstKinect);

            KinectSensor.KinectSensors.StatusChanged += KinectSensors_StatusChanged;
        }

The above code will use the KinectSensors static property of the KinectSensor class to retrieve all the available devices. That property is just a KinectSensorCollection, which is in fact a ReadOnlyCollection<KinectSensor> with an extra event to tell you if the status of one of the KinectSensor changed. The goal is to iterate through the KinectSensor's and find the first one with Status equals to KinectStatus.Connected, which means that it's ready to be used!

If a Kinect is found we will call the SetNewKinect method, described below. In any case, we also add an event handler to the StatusChanged event, so that we will know about any changes. For example if the current Kinect is unplugged, or if a new one is plugged but is still intializing, ...

        private void SetNewKinect(KinectSensor newKinect)
        {
            if (kinect != newKinect)
            {
                if (kinect != null)
                    StopKinect(Kinect);

                if (newKinect != null)
                    OpenKinect(newKinect);
            }

            Kinect = newKinect;
        }

The SetNewKinect method will compare the newKinect with the current one, and if they are different, it will call the StopKinect method with the old sensor, and the OpenKinect with the new one.

        private void OpenKinect(KinectSensor newKinect)
        {
            // TODO: Enable needed streams with desired formats and parameters
            newKinect.Start();
        }

        private void StopKinect(KinectSensor oldKinect)
        {
            oldKinect.Stop();
        }

And finally here is the event handler for the StatusChanged. If we have a newly connected Kinect, we will use to replace the old one. If the current Kinect has a new status (which is not connected), we will replace it by null. So, with this, we are now sure that the Kinect property always contains either null or a fully connected device.

        void KinectSensors_StatusChanged(object sender, StatusChangedEventArgs e)
        {
            if (e.Sensor != null && e.Status == KinectStatus.Connected)
            {
                SetNewKinect(e.Sensor);
            }
            else if (e.Sensor == Kinect)
            {
                // The current Kinect isn't connected anymore
                SetNewKinect(null);
            }
        }

That's all ! If you launch the application, it will start/stop your Kinect while you plug it in and out! But yes, that's true, for the moment you just don't see nothing! We will make something better in a few minutes.

Understand and handle the KinectStatus

The following table contains all the possible values for the KinectSensor status, and the meaning of each of them. In a Kinect application, it's really important to give some feedback to the final user about his Kinect device, or even to tell him that your application requires one!

So, now you know how to retrieve a Kinect! But don't worry, you won't have to write the above code every time. You can use the KinectSensorChooser control from the WpfViewers project (that you installed with the Kinect SDK Sample browser). It will do all the logic for you, as I will show you in the next project.

If you are a developer, and that you installed the SDK, be aware that you can use a Kinect for Xbox sensor! The Kinect for PC is advised because it allows you to use all the features of the SDK, such as the "near" mode, but you can already get started with your standard Kinect (read this article to understand the differences between Kinect for Windows and Kinect for Xbox).

Two ways to retrieve data: event-driven vs. polling

Now that you can access your Kinect by code, you probably want to make something cool with it. Then you'll have to use the different streams provided by the kinect, such as the color frame, depth information for each pixels, and of course the skeletons. The Kinect API gives the developer two ways to get data from those streams, and you'll choose one or the other depending of the kind of application you plan to create.

If you develop a desktop event-driven application (the basic windows/textblock/button application), you'll let the Kinect warn you whenever new data is available. At the opposition, if you create a loop-based application (for example an XNA video game), you'll ask yourself for new information when you do need for it! That's what they call "polling".

In the following examples, I'll mainly use the first approach.

First project: Tilt Motor Controller

For easiness, you can already download the source code here: WpfKinect1

To make sure that the Kinect works properly, we will create a first small project right now. The aim of this project is to display the color stream (basically just like a webcam), and also the ElevationAngle property that we are going to update by code.

Let's start with a new WPF project called WpfKinect. Once again, you have to add a reference to the Microsoft.Kinect.dll assembly. This time, add also the existing WpfViewers project to the solution, so that you can reuse some useful controls.

We now have two projects within our solution. In the WpfKinect project, add a reference to the WpfViewers project. After this is done, and if you open the XAML editor, you'll see that the Toolbox contains a few more controls:

Microsoft.Samples.Kinect.WpfViewers

I slightly changed the Xaml code of the MainWindow.xaml page to display the 3 things:

  • A KinectColorViewer that displays what the Kinect sees.
  • A TiltMotorController, to display and change the tilt angle, and that we are going to create in a few minutes.
  • A KinectSensorChooser, which will help us to find a KinectSensor and give feedback about its status to the user.
    <Grid>
        <Grid.ColumnDefinitions >
            <ColumnDefinition Width="Auto"/>
            <ColumnDefinition Width="Auto" />
        </Grid.ColumnDefinitions>
        <viewers:KinectColorViewer Grid.Column="0" Width="640" Height="480"
                                   Kinect="{Binding ElementName=SensorChooser, Path=Kinect}"/>
        <local:TiltMotorController Grid.Column="1" 
                                   HorizontalAlignment="Center" VerticalAlignment="Center"
                                   Kinect="{Binding ElementName=SensorChooser, Path=Kinect}"/>
        <viewers:KinectSensorChooser Grid.Column="1" x:Name="SensorChooser" 
                                     KinectSensorChanged="SensorChooser_KinectSensorChanged"
                                     VerticalAlignment="Stretch" Background="White" Margin="10" />
    </Grid>

As you can see, both the KinectColorViewer and the TiltMotorController have a Kinect property which is bound to the the Kinect property of the KinectSensorChooser.

In the code-behind, we will just implement the eventhandler SensorChooser_KinectSensorChanged, to start the sensor and activate the needed streams, which means the ColorStream in our case !

        private void SensorChooser_KinectSensorChanged(object sender, DependencyPropertyChangedEventArgs e)
        {
            if (e.OldValue != null)
            {
                ((KinectSensor)e.OldValue).Stop();
            }

            if (e.NewValue != null)
            {
                var newSensor = (KinectSensor)e.NewValue;
                newSensor.ColorStream.Enable();
                newSensor.Start();
            }
        }

That's basically all you need to do to start playing the Kinect video! Now we are going to build the TiltMotorController.

To be able to set the Kinect property directly in the Xaml of the MainWindow page, we have to add a Dependency Property to the TiltMotorController. For that, we will use the DependyProperty.Register method with the following parameters:

  • The property name (as it will be used in the XAML).
  • The property type.
  • The type of the property's owner.
  • And finally an UIPropertyMetadata that takes two parameters: the default value, and a static callback method used whenever the value changes.
public static readonly DependencyProperty KinectProperty =
            DependencyProperty.Register("Kinect",
            typeof(KinectSensor),
            typeof(TiltMotorController),
            new UIPropertyMetadata(null, new PropertyChangedCallback(KinectChanged)));

A simple Kinect property will help us to access to DependencyProperty value:

        public KinectSensor Kinect
        {
            get { return (KinectSensor)GetValue(KinectProperty); }
            set { SetValue(KinectProperty, value); }
        }
And then, the static callback method, and another private method which will add/remove event handlers to the KinectSensor.
        private static void KinectChanged(DependencyObject d, DependencyPropertyChangedEventArgs args)
        {
            TiltMotorController tiltMotor = (TiltMotorController)d;
            tiltMotor.OnKinectChanged((KinectSensor)args.OldValue, (KinectSensor)args.NewValue);
        }

        private void OnKinectChanged(KinectSensor oldKinectSensor, KinectSensor newKinectSensor)
        {
            if (oldKinectSensor != null)
            {
                oldKinectSensor.AllFramesReady -= newKinectSensor_AllFramesReady;
            }

            if (newKinectSensor != null && newKinectSensor.Status == KinectStatus.Connected)
            {
                newKinectSensor.AllFramesReady += newKinectSensor_AllFramesReady;
            }
        }

Download the sources if you want to see the XAML, it's just a few TextBlock, a Slider and some binding. The slider value is bound to the ElevationAngle property of the KinectSensor, and two buttons, Up and Down, will respectively increase or decrease the value by 5°.  If an exception is thrown during the operation, the error message is displayed in a TextBlock.

                if (Kinect != null)
                {
                    try
                    {
                        Kinect.ElevationAngle = value;
                    }
                    catch (Exception e)
                    {
                        AngleException = e.Message;
                        OnNotifyProperty("AngleException");
                    }
                }

Why do we catch the exception? The reason is that it's not really advised to play with the tilt motor! It is not designed for repetitive or long moves. So, as they know we like to use things in a way they are are not build for, they restricted the use of the property setter:

  • You can't edit the angle more than once per second
  • After 15 consecutive calls, you have to wait for 20 seconds at least.
If you don't respect that, an bad exception will be thrown !

Now, if you execute the project and that you take your Kinect sensor in your hand, you'll see that the tilt angle value changes depending of the orientation of the sensor! For the Kinect, the 0° is the horizontal plan.

That's the end of the first part!

We are not that far yet, but at least you know that your Kinect is working! :) In the next posts, we will go further by using the different formats of the ColorStream, we will track the skeleton and the user's position, we will use the depth sensor, and finally we will see how to add some speech recognition to provide an even better user experience!

Add comment

  Country flag

biuquote
Loading

TextBox

About the author

I'm a developer, blog writer, and author, mainly focused on Microsoft technologies (but not only Smile). I'm Microsoft MVP Client Development since July 2013.

Microsoft Certified Professional

I'm currently working as an IT Evangelist with an awesome team at the Microsoft Innovation Center Belgique, where I spend time and energy helping people to develop their projects. I also give training to enthusiastic developers and organize afterworks with the help of the Belgian community.

MIC Belgique

Take a look at my first book (french only): Développez en HTML 5 pour Windows 8

Développez en HTML5 pour Windows 8

Membre de l'association Fier d'être développeur

TextBox

Month List