Building a SLAM bot with a Kinect

As our project for the semester, Kapil, Tanmay and I will be building a bot that performs Simultaneous Localization and Mapping (or SLAM, in short) under Dr. J. L. Raheja at CEERI (Central Electronics Engineering Research Institute) , Pilani. Here, I’ll be writing about the difficulties we faced, what we did, a few good resources which helped us out, etc. All the code that we’ve written can be found here.

The entire SLAM project would be done using MATLAB. The first thing we decided to do was to build an obstacle avoider bot using the kinect, as a warm-up task of sorts. This would be a first step in several things: getting ourselves a working bot, controlling this bot using MATLAB, getting data from the kinect and analyzing it in real time with MATLAB, and finally, combinning all these steps (namely, analyzing images from the kinect depth sensor in real time, and using the information obtained from them to make our obstacle avoider).  Here’s how each of these steps panned out in detail:

  • Assembling the bot

    CEERI had purchased a bot from (Streak), which could carry a laptop and kinect on it comfortably. However, the software which was to burn the code onto the microcontroller (and to be used for serial communication with it) failed to run on any of our laptops (although it worked fine on an old Windows XP 32-bit system). Thus, we decided to use an Arduino instead, and purchased separate motor drivers (the original motor drivers was integrated with the microcontroller board). We also purchased new LiPo batteries, since the original Li-Ion batteries we had received was, well, non-functional. Oh well. We now have a fully assembled, working bot 😀

    A list of the parts we used is as follows:

    Component Specification Number
    Microcontroller Freeduino Mega 2560 (link) 1
    Chassis High Strength PVC alloy Unbreakable body (of Streak) 1
    Wheels Tracked wheel Big 10cm (of Streak) 4
    Motor 300 RPM, 30kgcm DC geared motor (of Streak) 4
    Motor driver 20A Dual DC Motor Driver (link) 1
    Battery Lithium Polymer 3 Cell, 11.1V, 5000mAh (link) 1
    Battery protection circuit Protection Circuit for 3 Cell Li-Po Battery (link) 1
    Battery charger Lithium Polymer Balance Charger 110-240V AC (link) 1
    RGBD sensor Microsoft Kinect for Xbox 1
  • Controlling the bot via MATLAB using an Arduino

    The obstacle avoider would be controlled using the Arduino board, via serial communication with MATLAB, which would be processing images taken from the kinect’s depth sensor to do the obstacle avoiding.Thus,  we needed to setup MATLAB to Arduino communication (which I had already worked on before, though). The code can be found here. The code requires MATLAB, the Arduino IDE and Arduino I/O for MATLAB. Note that the Serial port takes a very long time to show in the Arduino IDE, and this can be solved by following this set of instructions.

  • Obtaining data from the kinect, and processing the kinect’s images in real time

    This was fairly straightforward, thanks to MATLAB’s Image Acquisition Toolbox. All that was needed in addition to this was installing the Kinect for Windows SDK (I have v1.8 installed). The code can be found here. Of course, this is a continuous video input, and we’ll be using individual images later on to draw onto them easily.

  • Making the bot

    To make the bot itself, we used the depth image to locate where obstacles were located. We divided the screen into 3 parts, and took into consideration how many obstacles lying within a certain distance from the kinect were present in each of the 3 parts. The bot would then take the path with the least number of obstacles. The number of obstacles were counted by counting the number of centroids of each connected component. The pseudocode is as follows:

    initialize serial connection with Arduino
    setup the required pins on the Arduino
    start the kinect’s video input
    while (time elapsed < total run time required)

    acquire the depth and RGB image from the kinect
    threshold the depth image, so that only objects nearby are considered
    remove noise (by removing components whose connected area is less than a certain
    obtain the individual connected components, each representing an object
    obtain the centroid of each connected component
    divide the image into 3 parts, and make the bot take the direction which has fewest
    end while

    Here’s an image of what the laptop placed on top of the bot shows when the bot is moving:

    Screenshot of obstacle avoider

    Screenshot of obstacle avoider

  • Problems

    The main problem that we faced was that the kinect has can’t detect objects that are closer than 80cm. Thus, if an obstacle appears in front of the bot when the bot turns, and the obstacle is closer than 80cm, the bot can’t detect it. The kinect just returns a zero value for nearby objects, and for both objects that are too close to the kinect (< 80cm away) and too far away from the kinect (> 4m away), the kinect’s depth sensor returns a zero value. There were a few possible solutions we could think of to this (we’d read 2 of these up somewhere online, each from a different site, but I don’t remember the resources, I’m afraid):

    1. Incline the kinect at an angle from a height, so that an obstacle at the foot of the bot is slightly more than 80cm away.
    2. Take the closest pixel’s depth value that is non-zero as the undefined value.
    3. Take the closest pixel’s depth value that is non-zero, and use some sort of threshold to determine whether the zero point represents an object closer than 80cm, or one further than 4m. For example, if the neighboring pixels to a connetced component are, say, on an average, 1m away, the entire connected component is likely to represent an object closer than 80cm, while they are, on an average, 3m away, the connected component likely represents a distant object (like a wall) farther than 4m away. Note that in the examples just mentioned, I’ve used the nearest neighbouring pixels’ values’ averages, since a single undefined point is unlikely.
    4. Use 2 ultrasonic sensors to get information about when an obstacle closer than 80cm to the bot exists.
  • Possible Improvements

    A few possible improvements to the obstacle avoider are:

    1. Account for the size of the object, and use are instead of centroids, so that the bot takes the path with the smallest clutter.
    2. Account for distance of objects: at present, the bot merely considers all objects within a threshhold, and ignores objects further away. However, a possible improvement would be that if there are several objects in one direction, and very few in another, the bot would take the direction of fewer obstacles (provided, of course, that there are no obstacles it has to worry about in its immediate vicinity). However, at present, the bot would just continue going in a straight line, even if that would mean more obstacles in the future.
    3. Divide the screen into more parts, so that the turning of the bot is more accurate

Now, onto SLAM!

Here are a few sub-tasks that are involved:

  • Visual Odometry We decided to use the kinect itself for odometry. For this, we would need to estimate movement about the x, y and z directions, and the rotation about the 3 axes. For this, we used the paper “Fast 6D Odometry Based on Visual Features and Depth” as reference. The first step to this would be to use a feature detector.

    Initially, we had planned to use the SIFT algorithm. For this, we tried using VLFeat. Here‘s some quick test code that we wrote to test it out (it assumes that VLFeat has been setup, as described here). Here’s what it looks like:

    Applying SIFT on 2 RGB images using VLFeat

    Applying SIFT on 2 RGB images using VLFeat

    However, SIFT is a little too slow, and it a little over a second for it to be run on an image.

    So, we tried implementing it with Matlab’s SURF functions. It seems to be much, well, cleaner, and way more efficient (taking only a little under a tenth of a second). Here‘s a MATLAB function to take 2 input images, and here’s the result:

    Applying SURF on 2 RGB images using MATLAB

    Applying SURF on 2 RGB images using MATLAB

One response to “Building a SLAM bot with a Kinect”

  1. anti stripping agent for bitumen and asphalt says :

    It’s remarkable in favor of me to have a site,
    which is helpful designed for my know-how. thanks admin

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: