While digital artificial intelligence (A.I.) assistants are still (mostly) dumb, new research shows that Siri has improved quite a bit. Comparatively speaking, Apple’s digital assistant is now “smarter” than Amazon’s Alexa.
Loup Ventures has been actively testing A.I. assistants for a while, comparing them to one another in the process; in December, the firm published findings for voice-only home hubs. (It joins a growing amount of research into “home A.I.,” including one recent study that concluded all assistants are pretty stupid, for what it’s worth.)
Loup Ventures’s latest test, aimed at Siri, involves 800 questions graded harshly (whenever Siri pointed users to the web for an answer, the firm counted that as “incorrect’). The software understood 99 percent of questions, and answered 75.4 percent correctly. That’s enough for a solid ‘C’ grade, which is an improvement over the ‘D’ that Siri earned in April 2017 (when it also understood 94.4 percent of questions).
While a nine-percent uptick in correctness is great, Loup noted that it’s “more or less in line with the high rate of improvement we are noting with smart speaker-based assistants like Amazon Alexa and Google Home.” These results, it added, “[cannot] be compared to one another directly as a smartphone-based digital assistant responds differently to queries, geared more toward calling up information on the device’s screen or controlling the device itself.”
In other words, Siri has improved at the same rate as home hubs, but Loup resists comparing the results because of form-factor differences. If the platforms were directly correlated, Siri would rank ahead of Alexa (63.81 percent) and Cortana (56.38 percent), but it’s still a far cry from Google’s 81.1 percent score. (The average score of Google Assistant, Amazon Alexa and Microsoft Cortana is around 67 percent, so Siri is at least better than average.)
It’s an interesting time for voice-first assistants. Siri’s HomePod, Apple’s first foray into a screen-less future for Siri, remains unreleased. When the company announced its choice to delay HomePod, I speculated it was because Siri needed more work. Since then, Apple has released a whitepaper on how its machine-learning efforts scale while the underlying data framework continues to respect user privacy.
At CES this year, Google hosted a huge booth laced with Google Assistant gadgets, and has released a directory for making the most of its technology. Recently, it released a selection of voice-only home hubs for consumers; at CES, various partners (including Lenovo) showed off their own takes on the technology.
Amazon was also a presence at CES, mostly via third-party devices integrated with Alexa. “Works with Alexa” is now the de facto ‘us, too!’ battlecry for the smart-home gadget market (the famed ‘Internet of Things’ is alive, well, and talking back with Amazon’s help).
We’re eagerly waiting to see just how good Siri-for-HomePod is versus Alexa and Google Assistant. If Apple’s right, Siri’s HomePod functionality will give you everything you need to listen to music while controlling your home; data shows most people use their home speakers to play music – right where HomePod is hanging its hat.
There’s also HomeKit, Apple’s connected home platform, which supports a broad ecosystem of gadgets. At least if HomePod doesn’t turn out to be the best hub available, the alternatives are much cheaper.