Winters are long and cold here. And there’s no reason to hurry up, “skirt time” comes in May, usually.
We (me and my great neighbor) visited Prague’s most famous concert hall, Rudolfinum, in April.
So it was an evening of colourful calves:). Full hall of opaque tights. I would like to paint it.
The first version of LEGO sorter worked quite well, so I started with improvements.
At first I wanted to improve images quality. The images were blurred with movement and long exposition. So in every color was added some gray. These looked really bad (well, 5 frames per second):
I checked the camera library API for some way to set the camera shutter speed. (Well I know there is no real shutter in these cameras, in real the frame rate is in effect). But there is no way to set the frame rate directly. These small cheap webcams don’t allow that, these functionalities are embedded in the camera. But it can be done indirectly – by improving the lighting. The camera considers then it has enough light to increase the frame rate. So I threw away the lamp from the first version and used 2 Watts bicycle lamp. And it worked, camera started to take the images at rate 30 frames per second which is the camera maximum. The result was significantly better:
And the results of neural network were significantly better, too. But there was still a problem to distinguish yellow and white. Seeing the images I haven’t been surprised. I just thought I needed a better camera with a higher color scale, which wouldn’t give me overexposed images. But then I realized the brightness control is embedded in the camera, too. And again with no way to direct control but with a possibility to control that indirectly. In the camera box I had black background to avoid disturbances. It wasn’t on the cropped images, but the camera shot that, of course. So I replaced the black background tiles with white tiles. The overall brightness got quite down and the results were better again:
With such images I tried to enhance the neural network and add more colors – blue, black and light gray (especially the blurred light gray on dark gray background had been unrecognizable even with eyes). The results were better again. The only problem remained was to distinguish white and light gray – the network made a lot of mistakes here. I’m afraid this needs more complicated neural network and that two layers are not enough.
With a better neural network results I started to improve the sorter itself. I extended the conveyor belt and added two more pushers (I only had one NXT set, so was limited to three motors). So finally I was able to sort out three colors. Red, green and blue in a current configuration.
The original program was a single thread application. When it found a red brick, just slept for a short moment and pushed it off the conveyor belt. It was impractical for the three motors – during the sleep the program stopped to do anything. So I modified the program to use two threads. One thread captured the images and when found anything what should be pushed off the belt, created a push request with adequate delay and put it in the request list. The second thread checked the list and when found anything to process at the moment, issued a push action on the responsible motor.
Not all modifications were successful, of course. I tried to modify the pushers to push in both directions to be able to sort 6 colors, but I failed with the current construction. It would be necessary to exceed the leading axles. At least 16 studs long axles would be necessary. I didn’t have these and moreover at this length it would probably bent too much. I was thinking about a construction with 12 studs long axles, but finally used a complete different solution. Would be mentioned in a next article.
I tried to figure out a better conveyor belt, something smoother then the original LEGO belt. The structure of the current belt would probably cause complications to more sophisticated neural network. I tried to use bicycle tube, but with no clear result.
And finally the video…
Seven or eight years ago I got LEGO Mindstorms NXT set. Wonderful set. I built a few basic models and then decided to build my own construction – parts sorter. With a color sensor it looked like a piece of cake.
I worked on the sorter for a few weeks. The most difficult task was to calibrate the sensor for the individual colors. Finally it worked, somehow. But anyway I always found out the “dark measure box” wasn’t dark enough – the sunshine was different again and again and I could calibrate again…
Our son liked that a LOT, so one day he came to “take a look”. When I got back from work I just swept the debris to a box and started to build something else (some LEGO train, probably).
Few weeks ago in 2017, during the house cleanup, I found those debris. Disassembled, sorted out, sided. And exactly that afternoon my friend sent me a link to an article about LEGO sorter based on a neural network. So I started to build again. And to program a little bit, too.
At first, I wanted to check I could take the images and detect there was something present (or that the image was blank). So I built the LEGO conveyor belt, put an old USB camera over it and wrote a simple program which took images from the camera and saved those to the disc. It worked although the image quality was really, really bad. As I’ve discovered later, the camera took just 5 frames per second. The camera even doesn’t have any autofocus… what is good. When I tried a newer camera with autofocus it was unable to focus the running belt. The result was even worse.
I cropped the images to get just the centre (see above). The thought was that when I could see a part of a brick, the original image should contain the whole brick. Using the Neuroph library, I started to train a “yes/no” neural network.
And finally I realized this was enough to sort the colors, not only “brick/no brick”. So I threw the neural network away and started to train a new one, which was able to recognize red, green, yellow and white (based on the pile of the bricks I used). I didn’t try the light/dark gray bricks – blured gray bricks on the gray belt were not visible at all.
And – it worked. It started to sort the images.
So lets build something to sort the bricks, not only images. I downloaded the LeJOS library and tried to flash the firmware in LEGO NXT brick. Well, hell… The program erased everything in NXT brick and reported there is no NXT brick connected. The NXT brick stopped to do anything, only clicked silently. So called “clicking of the death”… Finally, I’ve managed to repair it. If you ever try that, remember to install the LEGO drivers in Windows DISCONNECTED from the Internet. And the final point – there was no need to flash the brick.
When I verified I could control the motors from PC, I built a simple pusher to throw the bricks off the belt. You can see the result on youtube.
And you can see the applied Murphy’s law, too. The first brick filmed is sorted wrong :).
To be continued…
