Figure 1:
The proposed methodology for LSTM-based transformer. LSTM, long short-term memory.
Figure 2:
LSTM block structure. LSTM, long short-term memory.
Figure 3:
Overview of the data augmentation strategy and training pipeline for the proposed model.
Figure 4:
Diagram of proposed framework in combination of multilingual supervised training and semi-supervised training for Indian languages using untranscribed data. The proposed enhanced LSTM-Transformer architecture is used to train the supervised mode (left); amalgamation of synthetic data from TTS system (middle); semi-supervised training with both transcribed and untranscribed data (right). LSTM, long short-term memory; TTS, text-to-speech.
Figure 5:
Comparative results.
Table 1:
Dataset details for each language
| Hindi | Marathi | Odia | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Train | Test | Val. | Train | Test | Val. | Train | Test | Val. | |
| Size in hours | 95.05 | 5.55 | 5.49 | 93.89 | 5.0 | 0.67 | 94.54 | 5.49 | 4.66 |
| Channel compression | 3GP | 3GP | 3GP | 3GP | 3GP | M4A | M4A | M4A | M4A |
| Unique sentences | 4506 | 386 | 316 | 2543 | 200 | 120 | 820 | 65 | 124 |
| # Speakers | 59 | 19 | 18 | 31 | 31 | – | – | – | – |
| Words in vocabulary | 6092 | 1681 | 1359 | 3245 | 547 | 350 | 1584 | 224 | 334 |
Table 2:
WER (%) for Indian languages
| Languages | Kaldi-based GMM-HMM TDNN | End-to-End transformer proposed LSTM | ||
|---|---|---|---|---|
| Hindi | 31.39 | 20.45 | 12.2 | 11.4 |
| Marathi | 18.61 | 16.6 | 11.2 | 10.6 |
| Odia | 35.28 | 28.10 | 23.2 | 21.3 |
Table 3:
WER (%) for Indian languages
| Model | Hindi | Marathi | Odia | |||
|---|---|---|---|---|---|---|
| w/o LM | with LM | w/o LM | with LM | w/o LM | with LM | |
| Proposed LSTM transformer (Baseline) | 14.1 | 11.4 | 12.7 | 10.6 | 24.6 | 21.3 |
| + Synthetic data augmentation | 13.5 | 10.9 | 12.3 | 10.2 | 24.2 | 21.0 |
| + Semi-supervised training | 13.2 | 10.5 | 12.1 | 9.8 | 23.9 | 20.6 |
Figure 6:
WER on Odia, Hindi, and Marathi using (A) TDNN (B) Transformer (C) Proposed LSTM Transformer.