Abstract

Electromyography (EMG)-to-speech (ETS) synthesis generates speech from articulatory muscle activity without acoustic input. We propose TAP-ETS, a time-aligned phoneme guiding framework that conditions mel-spectrogram generation on frame-wise phoneme sequences through cross-attention. Unlike prior work that treats phonemes as auxiliary supervision, our model injects aligned phoneme embeddings directly into the decoder. We also introduce refinement strategies that redistribute merged semantic guidance over frame-wise EMG signals, enabling the seamless integration of arbitrary phoneme- or text-level correction methods without modifying the synthesis model. On the Gaddy silent EMG benchmark, TAP-ETS reduces WER from 25.12% to 19.77%, achieving state-of-the-art performance and demonstrating the importance of accurate frame-level alignment.

Figures

Figure 1

Figure 1. Schematic diagram of ETS training with time-aligned phoneme conditioning.

Figure 2

Figure 2. Schematic diagram of semantic guiding with time-aligned phoneme refinement during inference.

Figure 3

Figure 3. Schematic diagram of semantic guiding with TAP. (a) illustrates the Levenshtein Distance based alignment. (b) shows the training procedure of the masking-based refinement model.

Speech Synthesis Samples from Silent EMG

The following samples are all speech synthesized from silent EMG. They are generated using EMG signals from the test set of Gaddy et al. We present the top 10 samples that show the largest WER improvement over the baseline.

# GT Text Baseline Ours
01 besides that there was quite a heap of bicycles
Transcript: but since then there was quite a game of mysegals
WER: 55.56%
Transcript: besides that there was quite a team of bicycles
WER: 11.11%
02 henderson he called you saw that shooting star last night
Transcript: enders in tikon do you follow that shooting star last night
WER: 50.00%
Transcript: henderson he called he saw that shooting star last night
WER: 10.00%
03 such things i told myself could not be
Transcript: some of the things i told myself could not be me
WER: 50.00%
Transcript: so things i told myself could not be .
WER: 12.50%
04 i found a little crowd of perhaps 20 people surrounding the huge hole in which the cylinder lay
Transcript: i felt a little crowd of perhaps twinning people surround the huge whole airmen to the cylinder lay
WER: 35.29%
Transcript: i found a little crowd of perhaps 20 people surrounding the huge hole in which the cylinder lay
WER: 0.00%
05 fearful massacres in the thames valley
Transcript: fearful masters enter the thames of aeneas
WER: 66.67%
Transcript: fearful maskers into the thames valley
WER: 33.33%
06 that was it
Transcript: that was data
WER: 33.33%
Transcript: that was it
WER: 0.00%
07 in spite of ogilvy i still believed that there were men in mars
Transcript: it might have killed me i still believe and there were pen and bars
WER: 76.92%
Transcript: and might will be i still believe and there were men in mars
WER: 46.15%
08 the horse took the bit between his teeth and bolted
Transcript: the horse looked the mid between the steve and bolted
WER: 40.00%
Transcript: the horse took the bit between his teeth and bolt
WER: 10.00%
09 all night long the martians were hammering and stirring sleepless indefatigable at work upon the machines they were making ready and ever and again a puff of greenish white smoke whirled up to the starlit sky
Transcript: all dying long the martians were hammering and stirring sleeved this and defitting a bowl and worked upon the machine they were making ready and ever and again upon a greenish white smoke whirled up to the starlit sky
WER: 36.11%
Transcript: all night long the martians were hammering and stirring sleepless and unthinkable at work about the machines they were making ready and ever and again a puff of greenish white smoke whirled up to the starlit sky
WER: 8.33%
10 it hardly seemed a fair fight to me at that time
Transcript: it hardly seemed to verify it to me at that time
WER: 27.27%
Transcript: it hardly seemed a fair fight to me at that time
WER: 0.00%