Anyone who grew up with Lego will remember buying and building individual sets, only to see their collection morph into an unruly, unsorted mass over time. Sorting through a vast array of bricks to find a particular piece has long been the bane of many a Lego enthusiast but now one canny constructor has developed a system that uses artificial intelligence to sort through large quantities automatically.

Jacques Mattheij observed that there was plenty of money to be made selling Lego on the second-hand market and plenty of eBay listings for bricks in bulk. With that in mind, he began prototyping a rather amazing machine that could classify Lego by shape and color.

Committing himself to the project, he bought enough Lego to fill his garage and set about putting together the hardware and software that would work together to sort through it.

The problem with bulk Lego lots is that they tend to contain lots of bricks that need to be weeded out before they can be sold, as noted in Mattheij’s blog post detailing his project. Any fake parts obviously need to go, as do any discolored, damaged, or otherwise dirty bricks.

In its current incarnation, the system loads bricks from a hopper onto a conveyor belt that runs them past a camera that is hooked up to a PC. Setting up the camera to recognize particular pieces presented all kinds of challenges. According to Mattheij’s count, there are 38,000 different shapes of Lego brick, which can be one of more than 100 stringently defined colors.

Mattheij tried various different methods, but eventually settled upon training a neural network to differentiate between different pieces. The finished system is apparently able to classify a brick in just 30ms, running on an Nvidia GeForce GTX 1080 Ti GPU.

While Mattheij concedes that his project is far from the finished product and could benefit from various revisions, its current iteration is good enough to accept kilos of Lego at a time and sort it with some accuracy. It would take a serious Lego collection to warrant an investment in this kind of hardware but there are certainly devotees out there would love to have access to Mattheij’s creation.