Projects

PATENTS

4/18/2018                    Patent Serial No. 9,947,322          NAU Case 2013-015

Title: “Systems and Methods for Automated Evaluation of Human Speech”

This invention is a method of detecting prosody in human speech in terms of prominent syllables, tone units, and tonic syllable tone choices as defined by David Brazil’s framework for representing prosodic features in discourse. It automatically determines these communicative features from a raw audio wave file in steps: 1) determine the phones that make up the utterance, 2) group the phones into syllables, 3) identify the prominent syllables, 4) divide the utterance into tone units, 5) determine the tone (falling, rising, rising-falling, falling-rising, or neutral) of the tonic syllables (last prominent syllable in a tone unit), and 6) determine the pitch (low, mid, or high) of the tonic syllables. The invention develops a program, which can be used for language learners, teachers, linguists, electronic engineers, computer scientists, or anyone in speech communication, who wish to better operationalize the property of human speech into a machine. Employing a text-independent system, the current invention provides a unique model for salient features of prosody in spoken parameters and integrates the stress and pitch tone features relevant to improving the performance of automated speech systems.

CURRENT PROJECTS

Title: Developing an Affective Interactive Oral Communication Tutor, Research Bridge or Seeds Awards (NAU RBS Funded, 2019-2020)

The US workforce increasingly draws upon a linguistically and culturally diverse pool of talent. However, limited English speaking ability or the presence of a strong non-native accent can reduce employment opportunities for those entering the workplace. Improving foundational communication skills and intelligibility is therefore integral to the professional success of nonnative English (or English as a second language) speakers. The research intends to develop and evaluate an affective interactive oral communication tutor (AIOCT) system in which language learners can practice and improve oral communication skills at their own convenience while receiving instantaneous, personalized guidance from the computer. The system will employ a device to engage in real-life interaction with learners and use advanced technologies based on automatic speech recognition (ASR) and virtual environments to provide practice and personalized feedback on English oral communication skills. The current pilot study will use MySpeechTainer as an initial version of AIOCT. The AIOCT will also be equipped with logging capabilities to record all user-system interactions. This learning system will be available on a range of devices including smartphones, so that learning can be pervasive and independent of time and space. System evaluation will address both individual components as well as the complete system through a one-month study investigating the learning gains. Logs of user-system interactions will be analyzed to gain insight into the processes underlying non-native language learning. Currently, a mobile-assisted pronunciation training program has been developed and we are piloting the program with international teaching assistants at various institutions.

Title: The relationship between learners’ backgrounds and proficiency of young English language learners in Mexico (funded by Alianza Inter-Universitaria Sonora-Arizon, 2019-2021).

The current study examined to what extent YLE backgrounds affect their English proficiency scores and measured fluency features. It further identified the relationships between YLE proficiency and oral performance in storytelling contexts. Fluency was operationalized as speech rates and pauses. Sixth elementary school students (i.e., largely 6th graders) participated from varying English learning backgrounds in Mexico. The YLE information included specific backgrounds (e.g., age, English exposure, parents’ income and education), individual difference (e.g., self-efficacy, test anxiety, cognitive strategy use), and general motivation to learn English. Oral communication skills were measured through story retells and examined in relation to their proficiency scores. The participants’ speech samples were analyzed for temporal features (Kang & Johnson, 2018). The research design revolved around correlation and regression approaches showing that learners’ background (44 to 62%) and individual difference (23-37%) factors explained the outcome variables. In addition, parent education along with motivation was the most significant and consistent background factors that predicted proficiency, narrative, and fluency . These results may inform policy and planning for syllabus design, teacher training, materials development, and awareness on language use outside the school context.

Title: Investigation of relationship among learner background, linguistic progression, and score gain on IELTS (funded by IELTS, 2019-2020)

This project investigated to what extent IELTS test performances (i.e., overall test scores, speaking section scores, and linguistic constructs of speaking) changed over the period of 3 months.  It further examined how learner background variables affected their linguistic progress and band score gains on the IELTS. Fifty-two Korean students, enrolled in IELTS preparation classes, participated. Participants’ proficiency levels were determined by their in-house placement test scores (i.e., roughly 16 beginners, 17 intermediate, and 19 advanced). Once participants completed the pre-test survey, they took the pre-arranged official IELTS test. Participants’ hours of study and target language use information was collected weekly. The post-survey and online interviews were conducted at the end of the 3-month period right after the official IELTS post-test. The individual long-run speaking responses from the pre- and post- tests were used for speech analysis (i.e., pronunciation and lexico-grammatical features) to examine their linguistic gains over time.  The results showed that students made various progress in English over the 3-month period with an average gain of slightly less than half a band (.3) and with the most score gain in the writing skill and the least score gain in the speaking skill. Approximately 60% of the participants gained .5 or 1 band scores.  In particular, hours of study and level of proficiency predicted the band score gains most potently. Together with the amount of target language, the background variables explained 34% of variance in the score gains. Fluency features revealed the most significant improvement over time, but complex relationships were found between learner background characteristics and speech construct changes.  Findings offer useful implications to the development of language testing and assessment as well as curriculum planning.