Testing Results: Our First Cafeteria Trial
At Tossit, we are committed to revolutionizing food waste management in cafeterias through our cutting-edge computer vision technology. Our advanced system accurately identifies and quantifies hundreds of food items, even in complex cafeteria settings, empowering you to make data-driven decisions and minimize waste.
Introduction
We recently conducted our first real-world test with Tossit in a dining hall cafeteria. This test was a crucial step in evaluating the performance of our computer vision system for monitoring food waste on plates and validating our approach in a real-world setting.
In this blog post, we'll share the results and insights gathered from this initial test, including detailed visualizations, performance metrics, and our plans for further improvement based on the findings.
Methodology
The test was carried out during a relatively slow morning on a finals week at the dining hall. Despite the lower attendance, we were able to collect usable data from 6 plates (wow, right) out of the 12 that came down the dish drop in roughly 45 minutes. The other plates either contained cereal or were empty, and therefore excluded from the analysis.
During this testing phase, we manually labeled the foods on the plates, as the YOLO (You Only Look Once) instance segmentation model, responsible for identifying and segmenting different food items, was not utilized. This allowed us to primarily focus on evaluating the performance of our weight estimation algorithm, which calculates the weight of the food items based on the depth map and polygon data obtained from the computer vision system.
Results and Performance Metrics
To assess the performance of the weight estimation algorithm, we calculated several performance metrics by comparing the model's predicted weights to the actual measured weights of the food items:
- Mean Absolute Error (MAE): 6.65 grams
- Mean Squared Error (MSE): 85.30
- Root Mean Squared Error (RMSE): 9.24 grams
- R-squared (R²) Score: 0.95
These metrics suggest that the weight estimation algorithm performed remarkably well, with the predicted weights deviating from the actual weights by an average of 6-9 grams. The high R² value of 0.95 indicates that approximately 95% of the variance in the measured weights can be explained by the model's predictions, demonstrating a strong fit.
Visualizations
To further illustrate the performance of the algorithm, we have included three visualizations:
Figure 1: Scatter plot of measured weights vs. model-returned weights, with most values closely following the ideal fit line.
Figure 2: Line plot comparing measured and model-returned weights for each plate, highlighting the discrepancy for Plate 4 (x axis/sample index values 0 and 1) due to the issue with mixed sausage and eggs.
Figure 3: Bar chart displaying the errors in model predictions for each plate, with Plate 4 exhibiting a large positive error (overestimation).
Discussion
While the overall performance of the weight estimation algorithm was impressive, the test results highlighted the importance of accurate YOLO labels for obtaining correct weight estimations. The discrepancy observed for Plate 4 was likely due to the issue of mixed sausage and eggs, where the manual weighing process may have grouped the food items differently than how the algorithm segmented them.
This finding has shifted our trajectory for the upcoming summer. Initially, we had planned to focus on training the weight estimation algorithm using the data collected from this test. However, the successful performance of the weight estimation algorithm and the identified significance of accurate YOLO labels have prompted us to prioritize the improvement of the YOLO model.
Future Plans
Moving forward, we aim to enhance the YOLO model's accuracy by refining the labeling process and training the model on a more diverse dataset. This will involve creating a more extensive and detailed dataset that includes various food items and their combinations to improve the model's ability to accurately segment and identify different foods.
Additionally, we plan to integrate the YOLO model with the weight estimation algorithm to create a more robust and accurate system that can provide real-time food waste monitoring and analytics in cafeteria settings. By combining these two components, we aim to deliver a comprehensive solution that empowers cafeteria managers to make informed decisions and reduce food waste effectively.