Ubicoustics is an AI system that recognizes activities from the sound

Researchers at the Carnegie Mellon University (CMU) developed a new technology that enables smart devices to sense their surroundings. In simple words, it simply allows devices to detect what’s happening around them and be more helpful.

“A smart speaker sitting on a kitchen countertop cannot figure out if it is in a kitchen, let alone know what a person is doing in a kitchen,” said Chris Harrison, a researcher at CMU’s Human-Computer Interaction Institute. “But if these devices understood what was happening around them, they could be much more helpful.”

Project Ubicoustics is all about software that uses devices’ always-on built-in microphones to sense what exactly is going on in the background. The team named their project as Ubicoustics because of the ubiquity of devices in the modern computing devices.

The sophisticated sound-labeling model included in the system recognizes the sound received through the microphone of a device to detect its location, for example, a kitchen, bedroom, workshop, or offices. It does so with the help of high-quality sound effects libraries built into it.

The AI system which shows an accuracy of 80% -- competitive with human accuracy – according to the team, will aim for a 99% in order to produce a system that will not mistake a sound for something else and create havoc by unnecessarily triggering other smart devices to take action. “Better microphones, higher sampling rates, and different model architectures all might increase accuracy with further research,” said the team.