Speech Analysis of Patients with Cleft Palate Using Artificial Intelligence Techniques: A Systematic Review

This review was conducted following the Preferred Reporting Items for Systematic review and Aeta-analysis (PRISMA) guidelines and the 2020 checklist.³⁶

This systematic review focuses on the applications of artificial intelligence techniques used in automatic speech analysis for patients with cleft lip and/or palate (CL/P). Inclusion criteria required that studies: (1) describe machine learning algorithms for automatic detection of cleft speech, (2) report quantitative data describing accuracy of the tool’s algorithm against some gold standard metric such as ratings provided by speech language pathologists, (3) focus on software that was trained using real person speech samples. Studies describing algorithms that detect hypernasality in both pediatric and adult populations were included. Only published journal articles were considered for review. All publication years were considered

Exclusion criteria were as follows: (1) reviews, technique articles, book chapters, conference proceedings, and papers, incomplete articles (eg, only abstracts available), (2) articles not available in English, (3) studies in which AI algorithms was not directly involved in the analysis of a patient’s speech, (4) articles that did not focus on speech analysis of patients with cleft lip and/or palate (eg, surgical techniques).

Systematic search was conducted on June 28, 2021, with a medical school reference librarian in 8 databases (PubMed, Scopus, ACM digital library, Linguistics, and language behavior abstracts, Institute of Electrical, and Electronics Engineers, CINAHL, PsychInfo, Cochrane). Medical Subject Heading terms include ones describing cleft speech “Velopharyngeal Insufficiency,” “Hypernasal,” “Cleft lip,” and “Cleft palate,” and terms describing AI/ML algorithms “Machine Learning,” “Neural Networks,” “Artificial Intelligence,” and “Machine Language.” Wildcard asterisk was applied to all search terms and the Boolean operator “OR” was used between similar terms such as ones describing cleft speech or AI/ML algorithms, while the Boolean operator “AND” was used to link the 2 sets of terms, ones describing cleft speech, and ones describing AI/ML algorithms. The full syntax of the search was the following: (“Artificial Intelligence”[MeSH Terms] OR “artificial intelligence”[tw] OR “Machine Learning”[MeSH Terms] OR “machine language”[tw] OR “machine learning”[tw] OR “neural network”[tw] OR “neural networks”[tw] AND (“Cleft Lip”[Mesh] OR “cleft lip”[tw] OR “Cleft Palate”[Mesh] OR “cleft palate”[tw] OR “cleft speech”[tw] OR “Velopharyngeal Insufficiency”[mesh] OR “velopharyngeal insufficiency”[tw] OR hypernasal*[tw]). Note that studies describing machine learning algorithms trained in foreign languages were also included because the goals of the studies were similar and focused on detecting a common feature of velopharyngeal insufficiency, hypernasality.

Search results from all databases were combined and duplicates were removed. Title and abstract screenings were conducted by 2 reviewers independently for concordance of article relevance. Cohen’s Kappa was calculated for inter-rater reliability analysis yielding k = 0.919 and k = 1, respectively. The full-text review of post-screening studies was done for eligibility, and disagreements were discussed to reach consensus.

Three categories of data were collected from the included articles: (1) the type of machine learning algorithm used, (2) speech sample database variables, and (3) whether feature extraction was required prior to training of the AI algorithms. Type of ML algorithm refers to the AI algorithm’s structure (eg, deep learning or support vector machine) utilized to train the hypernasality detection algorithm. Speech sample database variables included the size of the database (eg, number of speech samples), the language used, and demographic data, such as age ranges (eg, pediatric vs adult), and sex distribution. Feature extraction, described in the introduction of this paper, describes a pre-processing procedure that computer scientists use to extract the most useful phonetic features from a speech recording to feed the AI algorithm to achieve the highest accuracy in detection of hypernasality. Traditionally, feature extraction is a characteristic component of non-deep learning algorithms, while more advanced deep learning algorithms typically do not require this preprocessing step in its development, though some deep learning algorithms still utilize feature extraction. For a non-deep learning algorithm to detect hypernasality, distinct phonetic features need to be extracted from whole speech recordings using the many models of speech production, such as Mel Frequency Cepstral Coefficients (MFCCs), Glottal Activity Detection (GADs), Shimmer, and Jitter just to name a few. On the other hand, deep learning algorithms did not require a preprocessing feature extraction step and can be trained using entire speech recordings.¹⁰ Lastly, quantitative data on each algorithm’s concordance were collected. The concordance score is the main method of analyzing the success of an algorithm at detecting hypernasality and defined as the agreement between algorithmic detection and perceptual detection of hypernasality by speech clinicians, reported as a numerical value between 0 and 1.0.

The majority of algorithms fell in this category (n = 5).^10-13,33 This category represents the traditional approach of pairing feature extraction techniques with non-deep learning algorithms (Table 1). The concordance of these automatic hypernasality detection algorithms (AHDA) against professional speech clinician assessment ranged from 0.81 to 0.92, with an average concordance of 0.85, the lowest of the 3 groups. The average number of cleft speech database sizes used to train each algorithm was 3587. The types of machine learning algorithms utilized include Support Vector Machine, K-Nearest Neighbor, Hidden Markov Model, and Gaussian Mixture Modeling.

Two algorithms fell in this category (n = 2).^32,33 This category represents the non-traditional approach of combining traditional feature engineering with more advanced deep learning algorithms. One of the studies³² showed that LSTM-DRNN had more robust feature extraction ability and better classification ability compared to non-deep learning algorithms combined with the preprocessing step. The concordance of these AHDAs were extremely similar, ranging from 0.9334 to 0.9335, the highest of the 3 groups, even though the algorithms were trained across 2 completely different patient populations in India and China respectively (Table 1). The average number of cleft speech database sizes used to train each algorithm was 3921. The types of ML algorithms used include Deep Neural Network and LSTM-DRNN respectively.

Three algorithms fell in this category (n = 3).^15,37 This category represents the most advanced approach to AHDAs using feature independent deep learning algorithms. The systems are end-to-end meaning there is no feature extraction involved and entire sentences can be fed in the algorithm without preprocessing. The concordance of these AHDAs range from 0.797 to 0.975, with an average of 0.91, the middle of the 3 groups. The average number of cleft speech database sizes used to train each algorithm was 6306 (Table 1). Due to the lack of a comparable quantifying speech sample metric in one of the studies,³⁷ the size of its training database was not factored into the average database size. The types of ML algorithms used include Deep Neural Networks and Convoluted Neural Networks.

One of the 3 algorithms³³ should be noted to have trained their algorithm using healthy speech corpus and not hypernasal speech, which has never been done previously. All other studies described in this review utilized speech from cleft palate patients. While their concordance was lower in the healthy speech corpus group (0.797), the flexibility of deep neural networks to be able to learn from healthy speech should not be overlooked, especially given that the main bottleneck in AHDA development is the shortage of quality cleft speech samples. Due to the lack of a comparable quantifying speech sample metric, its 100 h of healthy speech corpus was not factored into the average number of cleft speech database size for the last group of algorithms.

Another observation from the results presented in Table 1 is the correlation between the number of speech samples used to train the algorithms and the concordance of the algorithms to professional speech clinician ratings. The trendline in Figure 2 is relatively flat and the R² value is low (.396) indicating a weak relationship. While a positive relationship between the 2 measures exists (number of speech samples available and the level of concordance achieved) the relationship is not strong. Additionally, the number of speech samples varied greatly, ranging from 300 to 10 080. However, despite the large differences between the number of cleft speech sample sizes, the difference between concordances were not nearly as large, ranging from 0.81 to 0.98. It should be noted that in Figure 2 one algorithm³³ was excluded due its use of non-cleft speech corpus and lack of a comparable quantifying speech sample metric, which was present in all other algorithms. Thus, the assumption that training with a larger amount of data will always lead to better models is not correct, other factors such as the quality of the input data is equally as important and applies to both non-deep learning and deep learning algorithms.⁸

In the literature, artificial intelligence has been established as an appropriate screening methodology for a number of health-related outcomes such as breast cancer, glaucoma, and cardiovascular health to just name a few.^38-40 Despite the advantages of artificial intelligence and machine learning, the use of AI and ML to assess hypernasality is relatively new. Of the 8 studies discussed in this review, 5 papers were published within the past 3 years and all but1 study was published within the past 6 years. The results from these studies are promising and invite further research into this new frontier of hypernasal speech assessment.

All studies found that ML algorithms were able to detect hypernasal speech with high concordance compared to professional speech clinician ratings. Furthermore, this remained true despite a variety of factors which differed from study to study such as different native languages, types of ML algorithms used, and patient populations tested (pediatric and adult). These findings support the adaptability of ML algorithms and enhance the feasibility of using ML as a screening tool in future clinical practice. These initial findings are promising and suggest that ML algorithms have unharnessed potential for large-scale, low-cost, and objective speech assessment screening for cleft palate patients.

Having more data is almost always better in training machine learning algorithms; more data translates to more features, more dimensions, and measures to explore, and more detailed results. Thus, building a quality cleft speech database is critical in the advancement of AHDA research, but remains one of its main obstacles. The quantity of cleft speech data may also be a practical concern for institutions wanting to build their own AHDA to provide low-cost and rapid screenings. Since both non-deep learning and deep learning algorithms can achieve similar levels of concordance, an institution with a limited cleft speech database may utilize non-deep learning algorithms for their accuracy requiring fewer speech samples, while an institution with a large cleft speech database may prefer deep learning algorithms which can achieve higher accuracies but require large amounts of data.

Our systematic review is limited by the available studies. A number of the studies were published by the same author, which may have unintended publication bias from researchers who have previously been successful in designing and executing experiments. Some authors also trained their algorithms using the same cleft speech corpus; it is possible that some speech samples may have been used in multiple studies. This may have reduced the variability of speech samples analyzed by each algorithm. Sex is a significant factor which can influence speech acoustics but was only reported by 1 paper out of the 8.¹⁵ Not knowing the distribution or whether a cleft speech database used to train an algorithm is biased toward male or female speech limits its generalizability. Additionally, these ML algorithms were developed in different countries and trained with different native languages, including English (United States), Kannada (India), Mandarin (China), Persian (Iran), Columbian (Columbia), Spanish (Spain, Chile, Ecuador), German (Germany), and Peruvian (Peru). While the mechanical production of hypernasality due to velopharyngeal insufficiency remains constant across different languages, innate characteristics of each language may have unintentional effects on the ability of ML algorithms to detect hypernasality. For example, an algorithm trained with Mandarin may not function as accurately when used on English speech samples. This phenomenon has not been studied in literature and thus reduces the utility of direct comparison between concordances achieved by these algorithms.

Finally, as with any study, this study is limited to the population and techniques studied and may or may not be generalizable to automated analyses of hypernasality in all languages. However, the languages addressed in this paper are already quite diverse, and the concordances of the ML algorithms show a similar degree of accuracy. Therefore, we believe that ML algorithms may be applicable to detect hypernasality in a much wider range of languages and populations.

Continued assessment of speech on a routine basis is a crucial component of postoperative care for children with cleft palate. This provides a useful post-surgical application where ML algorithms can help address health care gaps and obstacles in cleft speech care. The goal of regular post-surgical speech screenings is to monitor and screen for worsening hypernasal speech, which can affect up to 20% to 40% of primary cleft palate repair patients.⁴ However, some children are not able to see SLPs regularly, especially if the limited visits are due to financial or geographic barriers (eg, patients who live in rural areas or in places where SLPs are not readily accessible). This approach both complements the trend toward telemedicine and home-based interventions in speech-language pathology^41-43 and provides a potential solution to the mismatch in provision of speech services that disproportionately affects vulnerable populations.⁴⁴ Speech intelligibility involves both the listener and speaker.⁴⁵ ML algorithms can eliminate the listener bias of close contacts, including family and teachers, who may have adapted to better understand the child’s speech, without relying on the expert ears of SLPs who may not be available to all communities. The rapid results and high accessibility make ML algorithms an appropriate tool for SLPs to measure progress and serve to augment and complement their specialized speech services. Automated hypernasality detection systems cannot and should not replace professional speech therapy, but it has the potential to extend the reach of SLPs into underserved areas and remove health care barriers experienced by cleft speech patients.

Overall, this review of current literature suggests that ML is an appropriate platform for the assessment of hypernasality, with the potential to expand into new speech frontiers, such as postoperative remote monitoring of cleft palate surgery. Methods to improve reliability and adoption into clinical practice include training ML algorithms with diverse patient populations, including different languages, and testing in clinical settings and using speech samples beyond its training database. Future avenues of research can explore the inclusion of cleft and non-cleft speech samples in a training database, since 1 study demonstrated the ability to train ML algorithms on non-cleft speech. This can increase the size of the training sample size and overcome the limitations of collecting cleft speech samples.