Automated Beverages

How it started

A cold refreshing beer on a Friday afternoon is something that no employee should have to forego. In our office, we have a very important piece of kit – a beer fridge. It’s the home of our employees’ favourite tipples, both of the alcoholic and non-alcoholic variety of course.

The problem

No one remembers to re-stocks the beer fridge. We often find employees hiding under the stairs crying about these inhumane living conditions. 

The solution

One solution would be to allocate someone to regularly monitor the fridge and manually count the stock. 

Boring.

…or we could turn this into a fun AI project that could;

• automatically detect and count stock, and; 
• send emails to the correct people to restock. 

It was a no-brainer – we chose the latter.

Laying around the office we found a Raspberry Pi and an old webcam, which we decided to put to good use. Placed on the ceiling, the webcam would plug into the Pi and take pictures at predetermined intervals. These pictures would be passed through the AI which would then detect and count the amount of stock in the fridge. Once the stock reached a predetermined count, an email would be sent to the appropriate person to restock the fridge.

The YOLO Algorithm

In order to build AI for this task I would require something that could detect and count our stock. For this endeavour I chose the YOLO (You Only Look Once) algorithm. 

The YOLO algorithm uses a convolutional neural network in order to detect (count and label) objects in real time. At a base level, this algorithm breaks down the picture into lots of tiny squares. It then detects what is or is not inside each square and then draws a box around the detected object (called a bounding box). The algorithm uses weights to detect each object and label them and these weights are obtained by training it on correctly labelled images. 

There are many versions of this algorithm with trade offs including accuracy and compute power needed, but I ultimately decided to use the YOLOv4. The code for this was shared here by the creators. There are many tutorials of how to use this code and one of the most useful for me was theAIGuysCode which helped form the basis of my script. 

fswebcam

Alongside using the YOLO algorithm, a script was written to take a picture every few hours using the usb webcam plugged into the Pi. For this I used fswebcam and installed it onto the Pi.  While I could have run fswebcam from the terminal, I wanted to customise the file that was saved. I used (one of my favourite python features) an f string (see an example below) to incorporate the date and time into the filename when running fswebcam. I then played around with the resolution and frames until I got a picture I liked. For me this was a simple 1080p shot, with no banner.  I was also forced to skip the first couple of frames that the camera took because it could not adjust its exposure quick enough, and I was getting a lot of black images.

It would then proceed to run the picture it just took through the YOLO algorithm and output the picture from the webcam with the bounding boxes drawn and a count of each object.

Reminder system 

As is the standard with image recognition scripts, the program labels each object it finds in the images based on what objects it has been trained on. I chose to break the objects down into three simple categories: beer, wine and soft/non-alcoholic drinks. Should the AI detect less than 5 objects from any category, the script would send an email (using the email and MIME python packages) to restock the fridge. The email was largely generic, inserting the count of objects and their labels via an f string.

I also decided to give it a list of intro lines to choose from to keep things more interesting.

Training the model

The next stage was to start collecting the appropriate images and labelling them in order to train YOLO on. For this I collected hundreds of images from google images and collated them with pre-existing photos of the various beverages that we had on hand.

Once all the images were collected, the next step was to label them. I had to manually draw the bounding boxes that the YOLO algorithm could identify, then label them with the correct category (beer, wine, soft/non-alcoholic drink). This stage was labour intensive, but simple. I used LabelImg, which is an image annotation tool. It requires a classes.txt file with one line and one name for each of the labels/classes that you will use to classify each bounding box you draw.

LabelImg also allowed me to save the annotated bounding boxes in the format needed for the YOLO algorithm. It turns out this format is quite simple – a single .txt file per image labelled and a single line per bounding box drawn on that image The format has the number of the label as specified in the order it appears in our classes.txt file, followed by the 4 points of the bounding box we have drawn (x min, y min, x max and y max). In the example below I labelled a coke can so this is my second class (soft_drink) which appears as 2 followed by the 4 points of the bounding box.

Once it was done, I could augment (such as rotate, flip, greyscale) the labelled images to turn hundreds of images into thousands of images in order to train the algorithm. The AI could then be trained on these correctly labelled images in order to correctly identify the objects I had labelled.

After the training phase, I could test the algorithm on a brand new, unlabelled image to ascertain how well it was performing. Based on its performance I would iterate through retraining (altering parameters, using new input images and conditions) and retesting until I arrived at a state with the performance I desired.

Once the AI was written, trained and tested, I uploaded the code onto the Pi via an SSH connection. I used this github guide to set up the SSH on the Pi, connected to it in my chosen IDE and then copied over the python scripts. The Pi and webcam were then set up to observe the fridge and it was all systems go.

A few hours later I received the following email;

A great success! Within one week of building the insert cool beer AI name, employees’ tears were at an all-time low, and they could continue as normal knowing they would never again come in to work with an empty beer fridge.