Last time I wrote about separating round parts. Today I’ll get back to the original sorter. And to the neural networks, too.



I’ve just one NXT set. That means 3 motors. I was thinking how to modify the pushers to be able to push parts to both sides of the conveyor belt. That would allow me to sort to six categories. The previous version was not too good for that – it would be necessary to pre-set the pusher to the right position. And it could push another parts in the wrong moment. Moreover it would require much longer axes to build. Finally I’ve used a construction with LEGO treads (so those are useable anyway 🙂 ). I’ve added 1:3 gear to the NXT motor to get faster push. It shots the parts off the belt quite nicely.



Possibilities of the simple webcam were quite limited. And the images were lousy, too. So I looked for a replacement. And because some time ago I constructed a system for controlling water level in water tabks using RaspberryPI (and because RaspberryPI has a nice camera), the decision was very simple. So I’ve used a RaspberryPI system camera. It allows manually set most of the parameters and is able to provide 10 images per second (and there is probably still a space for an improvement).

I’ve used a Blinkt module for illumination. But during finetuning the camera I’ve found out it was insufficient. So I’ve added one piece of a LED stripe. (I tried to use about 20cm of it, but got a completely overexposed image).


Image classification

The biggest changes are at the software part. I threw away that Java library and started to study Python and deep neural networks (Jeremy Howard at has an excellent set of lectures online). At a start they say – “Get an Nvidia graphics card”. I didn’t have any, so I tried to use a free capacity of an 8 processor server. The calculations took overnight with (mostly) poor results in the morning.

So finally I paid for a virtual machine with Nvidia. One neural network calculation took now about 2 minutes, which was great (Today I have more images and more complex neural networks, but it still fits under 90 minutes).

The classification run on a classical computer without Nvidia, which alloved processig speed of one image per second. But one image per second was simply too slow, many parts passed through unnoticed. Tried to increase the speed, but unsuccessfully. 

But workmate of me had an old Nvidia. Too slow for gaming, but still useful for the classification. The time of the classification is insignificant by now. So I can process those 10 images per second.

So now the RaspberryPi continuously takes images and sends them to the server with Nvidia. The server performs the classification using the precalculated neural network (and calculates the position on the image with another neural net). Then it calculates the delay and sends the results to the Rasberry. The result is queued for the specified time. After the timeout, Raspberry sends the command to NXT brick to fire the pusher. And when all this works, the LEGO parts ends in a correct box.


And the result?

After a lot of experiments I got a working neural net for recognizing 10 colors – Black, Blue, DBG, DkGreen, Green, L(B)G, MdBlue, Red, White and Yellow. I didn’t have enough parts of other colors to learn on.

The sorting is not perfect, the machine makes a few types of errors:

  • Bricks are too close. Pusher hits two or more at once.
  • The neural net is not sure enough. In such a case I just let the brick pass.
  • Neural net does not know the color and didn’t learn that. So the Tan parts get to gray and yellow. Dark red and orange get to red etc.
  • The pusher just strikes.
  • And sometimes the neural net makes a wrong decision. But this is not often, color classification is simple task.

You can see the result over here.


What next?

This is still a prototype and I know I need to improve some parts od the solution:

  • I need to calibrate the camera – now I use the automatic setting. Especially the white parts are overexposed.
  • I need to increase the field of view of the camera and rebuild the pushers. This will allow me to sort the bigger parts (now I can sort very small parts only).
  • Camera takes the pictures directly from the top. It’s sufficient for color sorting, but for shape sorting it would be a problem.
  • And a lot of small improvements in both software and construction.

And that all means I can probably throw all the collected images away and start over again. 🙂