Amazon uses children’s dead grandmother in morbid demo of Alexa audio deepfake

amazon echo dot gen 4
Enlarge / 4th generation Amazon Echo Dot smart speaker.


Amazon is figuring out how to get its Alexa voice assistant to make a deepfake voice for anyone, dead or alive, with just a short recording. The company demonstrated the feature at its re: Mars conference in Las Vegas on Wednesday, using the emotional trauma of the ongoing pandemic and grief to sell interest.

Amazon’s re: Mars focuses on artificial intelligence, machine learning, robotics and other new technologies, with technical experts and industry leaders on stage. During the second day’s keynote, Rohit Prasad, senior vice president and chief researcher for Alexa AI at Amazon, showed a feature that is being developed for Alexa.

In the demo, a child asks Alexa, “Can Grandma finish reading me The Wizard of OzAlexa answers, “Okay,” with her typical feminine robot voice, but then the voice of the child’s grandmother comes out of the speaker to read L. Frank Baum’s tale.

You can watch the demo below:

Amazon re: MARCH 2022 – Day 2 – Keynote.

Prasad only said that Amazon is “working on” the Alexa capacity and did not specify what work is left and when / if it will be available.

However, he provided few technical details.

“This required invention where we had to learn to produce a high quality voice with less than a minute of recording versus hours of recording in a studio,” he said. “The way we made it happen is by framing the problem as a voice conversion task and not a speech generation task.”

Prasad briefly discussed how the feature works.
Enlarge / Prasad briefly discussed how the feature works.

Of course, deepfaking has gained a controversial reputation. Yet there has been some effort to use technology as a tool rather than a means of horror.

Deepfakes specifically, as noted by The Verge, have been exploited in the media to help compensate for when e.g. a podcaster messes up a line, or when the star of a project suddenly dies, as happened with the Anthony Bourdain documentary Roadrunner.

There are even cases of people using artificial intelligence to create chatbots that work to communicate as if they were a lost loved one, the publication noted.

Alexa would not even be the first consumer product to use deepfake audio to fill in a family member who may not be there in person. The Takara Tomy smart speaker, as pointed out by Gizmodo, uses AI to read children’s bedtime stories with a parent’s voice. Parents reportedly upload their voices, so to speak, by reading a manuscript for about 15 minutes. Although this differs markedly from the Amazon demo, in that the owner of the product decides to give their vocals rather than the product using the voice of someone who is probably unable to give their permission.

Aside from concerns about deepfakes being used for scams, rip-offs and other malicious activity, there are already some worrying things about how Amazon is designing the feature that does not even have a release date yet.

Before showing the demo, Prasad talked about Alexa giving users a “friendship relationship.”

“In this camaraderie role, human qualities of empathy and affect are the key to building trust,” the boss said. “These traits have become even more important in these times of the ongoing pandemic, where so many of us have lost someone we love. While AI may not remove the pain of loss, it can certainly make their memories last.”

Prasad added that the feature “enables lasting personal relationships.”

It is true that countless people are in serious search of human “empathy and affect” in response to emotional distress initiated by the COVID-19 pandemic. However, Amazon’s AI voice assistant is not the place to satisfy these human needs. Alexa can also not enable “lasting personal relationships” with people who are no longer with us.

It is not hard to believe that there are good intentions behind this developmental function and that it can be a great comfort to hear the voice of someone you miss. We could even see ourselves having fun with a feature like this, theoretically. Making Alexa make a friend sound like they said something silly is harmless. And as we’ve discussed above, there are other companies that leverage deepfake technology in ways similar to what Amazon demonstrated.

But it’s a huge, unrealistic, problematic leap to create an evolving Alexa capability as a way to revive a connection to deceased family members. Meanwhile, it feels unmotivated to pull at the heartstrings by bringing in pandemic-related grief and loneliness. There are some places Amazon does not belong, and grief counseling is one of them.

Leave a Comment

Your email address will not be published.