Building a SLAM bot with a Kinect
As our project for the semester, Kapil, Tanmay and I will be building a bot that performs Simultaneous Localization and Mapping (or SLAM, in short) under Dr. J. L. Raheja at CEERI (Central Electronics Engineering Research Institute) , Pilani. Here, I’ll be writing about the difficulties we faced, what we did, a few good resources which helped us out, etc. All the code that we’ve written can be found here.
The entire SLAM project would be done using MATLAB. The first thing we decided to do was to build an obstacle avoider bot using the kinect, as a warm-up task of sorts. This would be a first step in several things: getting ourselves a working bot, controlling this bot using MATLAB, getting data from the kinect and analyzing it in real time with MATLAB, and finally, combinning all these steps (namely, analyzing images from the kinect depth sensor in real time, and using the information obtained from them to make our obstacle avoider). Here’s how each of these steps panned out in detail:
- Assembling the bot
CEERI had purchased a bot from robokits.in (Streak), which could carry a laptop and kinect on it comfortably. However, the software which was to burn the code onto the microcontroller (and to be used for serial communication with it) failed to run on any of our laptops (although it worked fine on an old Windows XP 32-bit system). Thus, we decided to use an Arduino instead, and purchased separate motor drivers (the original motor drivers was integrated with the microcontroller board). We also purchased new LiPo batteries, since the original Li-Ion batteries we had received was, well, non-functional. Oh well. We now have a fully assembled, working bot 😀
A list of the parts we used is as follows:
Component Specification Number Microcontroller Freeduino Mega 2560 (link) 1 Chassis High Strength PVC alloy Unbreakable body (of Streak) 1 Wheels Tracked wheel Big 10cm (of Streak) 4 Motor 300 RPM, 30kgcm DC geared motor (of Streak) 4 Motor driver 20A Dual DC Motor Driver (link) 1 Battery Lithium Polymer 3 Cell, 11.1V, 5000mAh (link) 1 Battery protection circuit Protection Circuit for 3 Cell Li-Po Battery (link) 1 Battery charger Lithium Polymer Balance Charger 110-240V AC (link) 1 RGBD sensor Microsoft Kinect for Xbox 1
- Controlling the bot via MATLAB using an Arduino
The obstacle avoider would be controlled using the Arduino board, via serial communication with MATLAB, which would be processing images taken from the kinect’s depth sensor to do the obstacle avoiding.Thus, we needed to setup MATLAB to Arduino communication (which I had already worked on before, though). The code can be found here. The code requires MATLAB, the Arduino IDE and Arduino I/O for MATLAB. Note that the Serial port takes a very long time to show in the Arduino IDE, and this can be solved by following this set of instructions.
- Obtaining data from the kinect, and processing the kinect’s images in real time
This was fairly straightforward, thanks to MATLAB’s Image Acquisition Toolbox. All that was needed in addition to this was installing the Kinect for Windows SDK (I have v1.8 installed). The code can be found here. Of course, this is a continuous video input, and we’ll be using individual images later on to draw onto them easily.
- Making the bot
To make the bot itself, we used the depth image to locate where obstacles were located. We divided the screen into 3 parts, and took into consideration how many obstacles lying within a certain distance from the kinect were present in each of the 3 parts. The bot would then take the path with the least number of obstacles. The number of obstacles were counted by counting the number of centroids of each connected component. The pseudocode is as follows:
Here’s an image of what the laptop placed on top of the bot shows when the bot is moving:
The main problem that we faced was that the kinect has can’t detect objects that are closer than 80cm. Thus, if an obstacle appears in front of the bot when the bot turns, and the obstacle is closer than 80cm, the bot can’t detect it. The kinect just returns a zero value for nearby objects, and for both objects that are too close to the kinect (< 80cm away) and too far away from the kinect (> 4m away), the kinect’s depth sensor returns a zero value. There were a few possible solutions we could think of to this (we’d read 2 of these up somewhere online, each from a different site, but I don’t remember the resources, I’m afraid):
- Incline the kinect at an angle from a height, so that an obstacle at the foot of the bot is slightly more than 80cm away.
- Take the closest pixel’s depth value that is non-zero as the undefined value.
- Take the closest pixel’s depth value that is non-zero, and use some sort of threshold to determine whether the zero point represents an object closer than 80cm, or one further than 4m. For example, if the neighboring pixels to a connetced component are, say, on an average, 1m away, the entire connected component is likely to represent an object closer than 80cm, while they are, on an average, 3m away, the connected component likely represents a distant object (like a wall) farther than 4m away. Note that in the examples just mentioned, I’ve used the nearest neighbouring pixels’ values’ averages, since a single undefined point is unlikely.
- Use 2 ultrasonic sensors to get information about when an obstacle closer than 80cm to the bot exists.
- Possible Improvements
A few possible improvements to the obstacle avoider are:
- Account for the size of the object, and use are instead of centroids, so that the bot takes the path with the smallest clutter.
- Account for distance of objects: at present, the bot merely considers all objects within a threshhold, and ignores objects further away. However, a possible improvement would be that if there are several objects in one direction, and very few in another, the bot would take the direction of fewer obstacles (provided, of course, that there are no obstacles it has to worry about in its immediate vicinity). However, at present, the bot would just continue going in a straight line, even if that would mean more obstacles in the future.
- Divide the screen into more parts, so that the turning of the bot is more accurate
Now, onto SLAM!
Here are a few sub-tasks that are involved:
- Visual Odometry We decided to use the kinect itself for odometry. For this, we would need to estimate movement about the x, y and z directions, and the rotation about the 3 axes. For this, we used the paper “Fast 6D Odometry Based on Visual Features and Depth” as reference. The first step to this would be to use a feature detector.
Initially, we had planned to use the SIFT algorithm. For this, we tried using VLFeat. Here‘s some quick test code that we wrote to test it out (it assumes that VLFeat has been setup, as described here). Here’s what it looks like:
However, SIFT is a little too slow, and it a little over a second for it to be run on an image.
So, we tried implementing it with Matlab’s SURF functions. It seems to be much, well, cleaner, and way more efficient (taking only a little under a tenth of a second). Here‘s a MATLAB function to take 2 input images, and here’s the result: