To make sure you never miss out on your favourite NEW stories, we're happy to send you some reminders

Click 'OK' then 'Allow' to enable notifications

Not now
OK
Advert
Advert
Advert

Microsoft shares tool that can mimic voice and speech with 3 seconds of sample audio

Rhiannon Ingle

Published 
| Last updated 

Microsoft shares tool that can mimic voice and speech with 3 seconds of sample audio

Featured Image Credit: Shutterstock

The software giant has shared a tool that can replicate voice and speech with just a few seconds of an audio sample.

The new technology is being referred to as "text to speech synthesis".

Loading…

Advert

Microsoft explain that they have trained a neural codec language model, called VALL-E, to synthesise speech.

In a post on GitHub, Microsoft revealed that the text to speech synthesis training data stretched to "60K hours of English speech which is hundreds of times larger than existing systems."

Just using three seconds of an audio sample, VALL-E is able to accurately mimic voice and speech.

"VALL-E emerges in-context learning capabilities and can be used to synthesise high-quality personalised speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt," Microsoft outlines.

Advert

Not only is this new technology able to synthesise speech, but it can also take into consideration different emotions and moods that can influence the tone or pitch of speech.

The software company used over 60,000 minutes of speech data. Credit: Microsoft
The software company used over 60,000 minutes of speech data. Credit: Microsoft

The post details: "In addition, we find VALL-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis."

Such emotions and feelings include anger, sleepy, neutral, amused and disgusted with very distinctive variations differing between each one.

Advert

In short, the technology, created by Meta, analyses the ways in which a person sounds and then breaks down that specific information into various components.

Such components are called tokens and these are used as training data to establish matches between what the technology "understands" about how one particular voice would sound if it were to speak in words or phrases other than what was provided in three-second audio sample.

Or, as Microsoft puts it: "To synthesise personalised speech (e.g., zero-shot TTS), VALL-E generates the corresponding acoustic tokens conditioned on the acoustic tokens of the 3-second enrolled recording and the phoneme prompt, which constrain the speaker and content information respectively.

The technology only need a three-second audio sample to synthesise speech. Credit: Pixabay
The technology only need a three-second audio sample to synthesise speech. Credit: Pixabay
Advert

"Finally, the generated acoustic tokens are used to synthesise the final waveform with the corresponding neural codec decoder."

While the technology is revolutionary in many ways, the software corporation has acknowledged the "potential risks" that surround VALL-E in an ethics statement also posted on GitHub.

It reads: "Since VALL-E could synthesise speech that maintains speaker identity, it may carry potential risks in misuse of the model, such as spoofing voice identification or impersonating a specific speaker.

"We conducted the experiments under the assumption that the user agree to be the target speaker in speech synthesis.

Advert

"When the model is generalised to unseen speakers in the real world, it should include a protocol to ensure that the speaker approves the use of their voice and a synthesised speech detection model," it concludes.

Topics: Technology, Microsoft

Rhiannon Ingle
More like this
Advert
Advert
Advert

Chosen for YouChosen for You

Technology

Urgent warning issued to iPhone users to turn on new thief-blocking security feature

8 hours ago

Most Read StoriesMost Read

'Most disturbing website on Internet' can find every single picture that exists of you

16 hours ago