How did I start building the LEGO sorter

Seven or eight years ago I got LEGO Mindstorms NXT set. Wonderful set. I built a few basic models and then decided to build my own construction – parts sorter. With a color sensor it looked like a piece of cake.

I worked on the sorter for a few weeks. The most difficult task was to calibrate the sensor for the individual colors. Finally it worked, somehow. But anyway I always found out the “dark measure box” wasn’t dark enough – the sunshine was different again and again and I could calibrate again…

Our son liked that a LOT, so one day he came to “take a look”. When I got back from work I just swept the debris to a box and started to build something else (some LEGO train, probably).

Few weeks ago in 2017, during the house cleanup, I found those debris. Disassembled, sorted out, sided. And exactly that afternoon my friend sent me a link to an article about LEGO sorter based on a neural network. So I started to build again. And to program a little bit, too.

Gaining images…

At first, I wanted to check I could take the images and detect there was something present (or that the image was blank). So I built the LEGO conveyor belt, put an old USB camera over it and wrote a simple program which took images from the camera and saved those to the disc. It worked although the image quality was really, really bad. As I’ve discovered later, the camera took just 5 frames per second. The camera even doesn’t have any autofocus… what is good. When I tried a newer camera with autofocus it was unable to focus the running belt. The result was even worse.

Žlutá kostka Zelená kostka Červená kostka Modrá kostka

…processing images…

I cropped the images to get just the centre (see above). The thought was that when I could see a part of a brick, the original image should contain the whole brick.  Using the Neuroph library, I started to train a “yes/no” neural network.

And finally I realized this was enough to sort the colors, not only “brick/no brick”. So I threw the neural network away and started to train a new one, which was able to recognize red, green, yellow and white (based on the pile of the bricks I used). I didn’t try the light/dark gray bricks – blured gray bricks on the gray belt were not visible at all.

And – it worked. It started to sort the images.

… and sorting the bricks

So lets build something to sort the bricks, not only images. I downloaded the LeJOS library and tried to flash the firmware in LEGO NXT brick. Well, hell… The program erased everything in NXT brick and reported there is no NXT brick connected. The NXT brick stopped to do anything, only clicked silently. So called “clicking of the death”… Finally, I’ve managed to repair it. If you ever try that, remember to install the LEGO drivers in Windows DISCONNECTED from the Internet. And the final point – there was no need to flash the brick.

When I verified I could control the motors from PC, I built a simple pusher to throw the bricks off the belt. You can see the result on youtube.

And you can see the applied Murphy’s law, too. The first brick filmed is sorted wrong :).

To be continued…