reading is essentially a process of translation of textual sequences into their phonetic representations

spoken word thus play a fundamental role in reading acquisition

highly accurate automatic speech recognition (ASR ) systems exist for many languages but they are still strongly biased towards accurate processing of adult voices

HOWEVER: in reading acquisition or reading fostering scenarios one deals with speakers whoseutterances of sequences-to-be-read exhibit peculiar characteristics

majority of those who learn how to read are children

children are physiologically (differences in size  and anatomy of vocal tract; teeth change) and cognitively different from adults

children voices are different from adult voices (e.g. Fundamental frequency F of male voice =~ 112.0 Hz; F(female voice) =~ 195.8 Hz; F(boy voice) =~ 250.0 Hz; F(girl voice) =~ 244.0 Hz

datasets for children’s speech that are publicly available are quite scarce

kidsTALC (Rumberg et al, 2022) authors report 26.2 % word-error-rate (WER) of typically developing monolingual German children

in majority of reading exercises which are included in the Primer we already know the text in advance

we already know what utterances could be considered as correct lectures and which not

to every specific exercise - like vowel or syllable recognition – Primer associates a specific language model (scorer) which constraints the connectionist temporal classification (CTC) beam search to restricted amount of exercise-relevant answers

significantly constraining the search space of plausible solutions

INNOVATION 1 : implementation of exercise-specific scorers transforms a generic ASR problem (difficult) into multi-class classification problem (easier)

In our previous HMPL Curriculum 1 (HMPL-C1) study which focused on extending foreign language vocabulary for human learners and increase of speech-recognition accuracy of artificial learners, we have „observed increase in amount of matches between expected and predicted labels which was caused both by increase of human learner’s vocabulary, as well as by increase of recognition accuracy of machine’s speech-to-text model“ (Hromada & Kim, "Proof-of-concept of feasibility of human–machine peer learning for German noun vocabulary learning", Frontiers in Education, 2023)

Learner 1 (L1) - is a 5-year old – pre-school bilingual (90% German, 10% Slovak) daughter of the main author of this article

three HMPL-C2 exercise 1 (E1) sessions were executed on days 1, 3 and 5 of the study

each HMPL-C2-E1 session consisted of human-testing phase followed by a mutual human-machine learning phase

in each phase, sequences consisted of 5 repetitions of syllables started with occlusive labial consonant M or B and followed by the vowel A, E, I, O or U, thus yielding sequences from “MA MA MA MA MA” to “BU BU BU BU BU"

speech recordings collected during the learning phase subsequently provided input for the acoustic-model fine-tuning process

sequences of five vowels resp. CV syllables which were displayed by DP were considered to provide the “reference”; output of the model yielded the hypotheses

data provided by L1 during three testing phases on days 1, 3 and 5 were evaluated by means of 5 different models

DeepSpeech_de = baseline model; KIds-0 : Deepspeech_de fine-tuned with kidsTALC; KIds-L1-1 : KIds-0 fine-tuned with data provided by L1 during day 1 learning phase;KIds-L1-3 : KIds-1 fine-tuned with data provided by L1 during day 3 learning phase;KIds-L1-5 : KIds-5 fine-tuned with data provided by L1 during day 5 learning phase

NOTE: WER-decrease between rows corresponds to increase of accuracy of the ASR model; WER-decrease between columns points to increase in L1's reading competence

As of 2023, there exists no publicly available ASR model which could accurately and reliably process child speech.

In our IHIET 2023 article, we introduce two innovations with which the problem can be partially bypasssed in context of digitally supported reading acquisition app:

1. Transformation of a generic ASR problem into a sort of extended multi-class classification problem by means of extending a generic acoustic model with a domain-specific, minimalist language model (“scorer”).
2. Human-machine peer learning (HMPL) whereby the artificial utterence-processing tutor U incrementally and gradually adapts its parameters to a particular learner, a human individual I.

In concrete terms, we have shown that after three sessions focusing on acquisition of grapheme-vowel and CV-bigrapheme correspondences had lead, in case of one particular learner, to decrease of WER from 96% to 48%.