Bespoke Update: Samsung Enhances AI for Your Fridge

The concept of applying a software update to your refrigerator might sound unusual, yet that’s precisely what Samsung is doing with its Bespoke refrigerators this week. Surprisingly, this patch is achieving significant improvements in providing genuinely useful machine learning capabilities in a contemporary icebox. Samsung has previously incorporated AI-powered features such as automatic food recognition and meal planning for its Bespoke line over the past couple of years. However, during my review of its flagship model last year, I discovered that the AI capabilities were still in a developmental phase. Initially, the refrigerator could identify about 60 different fresh foods, including various fruits and vegetables, and another 50 packaged items like yogurt and popcorn. While this was a decent starting point, it fell short given the vast array of items available in a typical grocery store.

Moreover, users often needed to input additional information, such as the quantity of items or the date they were added. This requirement made the concept of AI-assisted grocery tracking more cumbersome than it should be. Personally, I prefer to avoid typing on my fridge, and it seems the engineers at Samsung share this sentiment, likely motivating this significant update to their Bespoke refrigerator software.

A significant enhancement in this update is the integration of support for Google Gemini, which brings several important benefits. By merging Samsung’s existing on-device object recognition technology with Google’s cloud-based models, the number of foods that can be identified has skyrocketed from just over 100 to more than 2,000. While this enhancement necessitates connecting the refrigerator to Wi-Fi, considering the multitude of other smart features it supports, like calendar integration and video playback, this is a minor requirement.

Another notable improvement is the expanded voice control capabilities, enabling users to ask the fridge to adjust settings, check details like the last replacement date of the water filter, or even assist with troubleshooting issues. Depending on the specific problem, the fridge can also provide a tutorial on how to resolve the issue.

For more intricate or challenging problems, Samsung has introduced a feature called Reliability AI, which monitors the fridge’s components and helps identify potential faults before they escalate. In situations where the device requires servicing, the AI can supply detailed information to agents, possibly allowing them to address certain issues remotely.

For instance, a Samsung representative explained that if a customer reports that the ice cubes are clumping together, the Reliability AI could enable agents to adjust the amount of water being added to the ice tray, all without needing to send a technician to the home. Importantly, Samsung states that while the fridge will track and monitor device health metrics, owners must grant explicit consent for repair personnel to access this data. Furthermore, if a problem necessitates in-home servicing, sharing this data with repair technicians helps accelerate problem identification and resolution rather than starting from scratch during an on-site visit.

During the eight months I’ve been using my test unit, it has not experienced any mechanical issues, so I haven’t had the chance to evaluate Samsung’s Repairability AI. However, after exploring it first at Samsung’s headquarters, I have been using an early version of the Bespoke line’s new software for the past two weeks, which includes enhanced support for cloud-based object recognition. The improvement is remarkable. Even after a period of usage, I remain astonished at how many different foods it can recognize. For instance, I had a can of Bull Head Shallot Sauce, a niche ingredient from Taiwan typically used in Asian dishes, stored deep in my fridge. The AI recognized it effortlessly, automatically tagging it and noting when it was first added to the AI Food Manager.

Additionally, the system has become significantly better at recognizing brands and tracking the quantity of specific ingredients, leading to more detailed listings. It can differentiate between Diet Coke and Coke Zero, accurately noting how many of each item is present. Despite the fridge often needing to connect to a cloud server for item recognition, responses typically appear swiftly, often within a few seconds. Moreover, for certain items like avocados, the fridge keeps track of their age and sends notifications when they approach expiration. While it may not always be accurate, a simple reminder to check on items is all I need, and the fridge delivers just that.

Sam Rutherford for Engadget

Moreover, I’ve noticed that the fridge now remembers when you frequently remove a specific food item and prompts you to add that item to your shopping list. This gentle reminder to restock staples you regularly use is a thoughtful touch and occurs in a way that remains unobtrusive. Consequently, you can simply check your phone while at the store instead of manually curating a list every week. Additionally, because the fridge excels at recognizing and tracking its contents, it can offer better recipe suggestions based on the ingredients you already have at hand.

However, like many current models, the AI doesn’t consistently get every detail right. For example, I was initially impressed when it labeled a tub of plant-based cream cheese as “Philadelphia Plant-based,” only to realize that the description was incomplete, as the AI merely read the text on the lid without the intelligence to accurately finish the label. To be fair, it provides ample information to help me discern what’s inside the fridge while glancing at the Food Manager, though it is not always perfectly accurate.

One issue with this update is that, similar to many current AI services, Samsung’s new software can exhibit overconfidence or be prone to errors. For instance, during one interaction, as my wife was replacing an item in the fridge, the algorithm mistakenly identified a brightly colored bandage on her finger as a vegetable, which it certainly is not. At times, it seems to simply guess. Nevertheless, I would argue that transitioning from approximately 100 identifiable items to over 2,000 is a significant and welcome advancement, even with those limitations.

Another curious aspect is that although Samsung is leveraging Google’s AI models for many new features within the fridge, there are no visible references to Gemini inside the device itself. This is somewhat disappointing since Bixby remains the only digital assistant available for direct interaction.

As I have previously mentioned, Samsung’s AI food recognition continues to evolve, and I believe this assertion still holds true. With this latest update, the company has significantly progressed toward fulfilling the promise of a refrigerator equipped with genuinely useful AI-powered features. What once felt more like an impressive technology demonstration has rapidly transformed into a valuable tool for managing your groceries, despite some occasional hiccups.