From sounds to words: The relation between phonological and lexical processing of tone in L2 Mandarin

While at the phonological level Mandarin has four tone categories, described by discrete labels as illustrated above, the acoustic realization of tone in actual speech varies substantially with different talkers, contexts and speech rate (Jongman et al., 2006). Thus, a critical part of processing words in tonal languages involves identifying discrete tone categories from highly variable acoustic realizations. To study the categorical perception of tone, experimental research uses synthetic stimuli from tone-pair continua created by manipulating pitch (and potentially other suprasegmental parameters, such as duration). Research going back to Wang (1976), who conducted identification and discrimination tasks with synthetic tokens varying along the T1–T2 continuum, has found that L1 Mandarin listeners perceive tone more categorically than naive L1 English listeners with no experience of tonal languages. Subsequent studies including continua of all six possible tone pairs in Mandarin have confirmed these findings (Hallé et al., 2004; Ling and Schafer, 2016; Peng et al., 2010; Shen and Froud, 2016; Xi et al., 2010; Xu et al., 2006).

At the level of lexical processing in tonal languages, most previous work has focused on the independent contributions of tones versus segments in the process of lexical access. Early studies reached a general agreement that tones may be a weaker cue than segments. By using a homophone judgment task of written characters in Mandarin, Taft and Chen (1992) found that native listeners took longer and were less accurate when a pair of words differed only by tone than only by vowel quality. Cutler and Chen (1997) examined how native speakers of Cantonese (a tonal language similar to Mandarin, but with six tones) process lexical tones in spoken words. Results from a lexical decision task showed that listeners were most likely to accept a non-word as a word when the only difference between the two was in tones. A speeded same-different judgment task with the same participants showed slower and less accurate responses when the only difference between two words was in tones. The researchers concluded that lexical tone might be a weaker cue due to the comparatively late availability of the tonal cue compared to segmental cue. Similar results were obtained by Ye and Connine (1999) in a simple tone-vowel detection task. However, they also found that tone might play a more important role when presented in context. As pointed out by Liu and Samuel (2007), the tasks employed in many early studies are mostly sensitive to sub-lexical processing, where listeners could respond without accessing lexical meaning.

In tasks requiring listeners to access lexical representations, findings have indicated more similar roles for tonal and segmental cues. Malins and Joanisse (2010) conducted a real-time spoken word recognition experiment using the visual word paradigm, in which native Mandarin speakers listened to words and were asked to select the corresponding picture from a set of four, including the target (e.g. chuang2 ‘bed’), a competitor overlapping with the target in different phonological domains (segmental: chuang1 ‘window’; cohort: chuan2 ‘ship’; rhyme: huang2 ‘yellow’; tonal: niu2 ‘cow’), and two phonologically unrelated distractors. Of critical interest were the segmental and cohort conditions, where the divergence of competitors from targets was argued to occur at comparable time points (see appendix in Malins and Joanisse, 2010). Eye gaze patterns did not differ between the two conditions, leading the authors to conclude that ‘tonal and segmental information are accessed concurrently and play comparable roles’ in word recognition by Mandarin native speakers (Malins and Joanisse, 2010: 407). A follow-up study (Malins and Joanisse, 2012) using an event-related potential (ERP) paradigm supported the conclusion that tonal and segmental cues were both accessed as soon as they were available, while also indicating potentially different underlying processing mechanisms for tonal versus segmental information.

So far, we have considered two lines of research on the role of tone relative to segments, each leading to somewhat different conclusions. In experiments encouraging sub-lexical processing and not requiring access to lexical meanings, tone appears to be a weaker cue than segments (Cutler and Chen, 1997; Ye and Connine, 1999). In spoken word recognition tasks, which require full lexical access, tones and segments appear to be on a more equal footing (Malins and Joanisse, 2010, 2012). A third line of research, focusing specifically on the interaction of tonal and segmental cues, provides evidence for the dynamic relationship between different cues, and draws attention to the limitations of isolating individual cues in experimental contexts, emphasizing the importance of processing tonal cues along with segmental cues as one entity in successful word recognition. In a form priming task, Sereno and Lee (2015) found weak priming effects when the prime and target matched only in segmental content, but much larger effects when they matched in both segmental and tonal content (see also Lee, 2007), indicating that the syllable as a unit, rather than its individual component parts, may play a critical role in word processing. Using the Garner speeded classification paradigm, Tong et al. (2008) investigated the interaction between tones and segments from a different angle. In this task, participants have to classify stimuli according to a specific dimension (e.g. consonant, tone, or vowel) while ignoring variation on other dimensions. Results from native Mandarin speakers showed that variation in vowel quality interfered more in the classification of stimuli by tone than the reverse, which led the authors to conclude that the processing of tone was not independent, but integrated into the processing of vowels. Further evidence comes from another speeded classification experiment conducted by Lin and Francis (2014). Using stimuli constituting legitimate words in both English and Mandarin, they found that L1 Mandarin listeners showed symmetric interference between consonants and tone, regardless of whether stimuli were presented in an English or a Mandarin context, while L1 English listeners did not show any interference from the non-target dimension in either direction. They suggested that native Mandarin listeners process tonal and segmental cues in a more integrated manner regardless of the ambient language, while English listeners with no experience of tonal languages processed them more separately.

These findings are consistent with ‘selective perception routines (SPRs)’ as proposed in Strange’s (2011) Automatic Selective Perception (ASP) model: L1 Mandarin listeners have developed the routine of perceiving and processing tonal cues along with segmental cues for lexical access through life-long experience with their tonal L1. This L1 SPR constitutes a highly over-learned pattern so that even when instruction and stimuli are in English, native Mandarin listeners are unable to inhibit the automatic processing of tone. On the other hand, English listeners unaccustomed to processing pitch at a lexical level might treat pitch as intonation and process it separately from segmental content, since in English, suprasegmental features are rarely distinctive at a lexical level. Even in some rare cases, when English words only differ from each other in stress, English listeners do not appear to use such suprasegmental information during lexical access (Cutler, 1986). English listeners are thus more likely to process suprasegmental features such as pitch at a post-lexical level, without immediate impact on lexical access (Lin & Francis, 2014). Framed within the ASP model, the task for L2 learners then is to inhibit entrenched L1 SPRs, and learn new SPRs to optimize use of the most reliable linguistic cues in the L2. A key goal of our study is to gain a better understanding of the extent to which English learners of Mandarin are able to do so in the context of using tonal and segmental cues during lexical processing.

While lexical tone presents a novel phonological dimension for L2 learners with non-tonal L1s, previous work provides ample evidence that listeners with non-tonal language backgrounds are sensitive to pitch contrasts (Hallé et al., 2004), and that accuracy of tone identification can be improved significantly even after short-term training (Wang, 2013; Wang et al., 1999; Wong and Perrachione, 2007). At the same time, L2 learners, even those with intermediate to advanced proficiency, show persistent difficulty with the processing of tone at a lexical level (Pelzl et al., 2019; Qin, 2017). This gap between perception and lexical processing of tones has begun to be addressed in a recent study by Pelzl et al. (2019). They found that although English-speaking learners of Mandarin were as successful as native Mandarin speakers at identifying tones in isolated syllables in a tone identification task, the same learners were less likely to correctly reject non-words in a lexical decision task than native speakers when the non-words and words differed only by tone, indicating a ‘disconnect between L2 abilities to categorize tones as phonetic objects and abilities to utilize those categories as lexical cues’ (Pelzl et al., 2019: 69). A third task using an ERP paradigm provided further neurophysiological evidence of learners’ difficulty in using tone as a cue in lexical processing. These findings present important evidence for a dissociation between phonological and lexical processing ability related to tone in L2, based on group-level analyses comparing L2 with L1 speakers on different tasks. Remaining unexplored is the relation between these abilities at the level of the individual learner, a question that we aim to address in the present study.

Many factors can contribute to this disconnect between phonological and lexical processing of tones by L2 learners. At a phonological level, although L2 learners were able to achieve native-like accuracy in identifying the four standard Mandarin tones in isolated syllables (Pelzl et al., 2019), L2 listeners may have more difficulty handling the lack of invariance of tone realization in actual speech, where they have to assign a specific token quickly into a phonological tone category. In a study by Ling et al. (2016), L2 learners of Mandarin (L1 English) with various proficiency, along with native Mandarin listeners and native English listeners with no experience of Mandarin, participated in an identification task designed to assess categorical perception of tone. In each trial, listeners heard a token from one of the six synthesized continua of all possible Mandarin tone pairs (e.g. T1–T2, T1–T3) with manipulated pitch, and they were instructed to decide which tone they heard in each pair. Results showed that L2 learners patterned between the other two groups on the steepness of identification slopes, suggesting L2 learners perceive tone less categorically than native speakers, yet more categorically than listeners with no experience of a tonal language. Moreover, the steepness of identification slopes, i.e. the degree of categorical perception, correlated significantly with learners’ proficiency in Mandarin. These findings are consistent with those of Shen and Froud (2016), who reported results from advanced learners of Mandarin on a similar identification task with T1–T4 and T2–T3 pairs showing that advanced learners’ performance was similar to that of native speakers. However, in a follow-up ERP study (Shen and Froud, 2019), native Mandarin speakers, but not L2 learners, showed electrophysiological evidence of categorical perception of tones. L2 learners also showed larger P300 responses, suggesting more attentional focus in the processing of tones. As Shen and Froud (2019: 263) pointed out, ‘[i]f phonetic categorization by adult learners is less efficient and requires more attention compared to native Chinese speakers, higher-level processes such as lexical access may also be more challenging for learners.’ More specifically, we might expect L2 learners to rely on tonal cues less than native listeners at a lexical level. This is consistent with Pelzl et al.’s (2019) finding that L2 learners were more likely to reject segmentally mismatched than tonally mismatched non-words, while native listeners rejected them at equal rates, indicating that, compared to native listeners, L2 learners allocated less weight to tones relative to segments.

In addition to the bottom-up difficulty of drawing phonological information from acoustic input, the top-down influence of highly over-learned L1 automatic selective perception routines (SPRs; Strange, 2011) may also make L2 learners less likely than native speakers to use tonal cues in word recognition. In English, suprasegmental features are rarely lexically distinctive. Although some English words differ from each other only in stress, English listeners do not appear to use such suprasegmental information during lexical access, presumably because such cases are exceedingly rare (Cutler, 1986). In Mandarin, on the other hand, tone is a key component of lexical representations and is indispensable in word processing.

The question that arises is whether long-term exposure and learning experience lead to increased use of tonal cues in L2. Zou et al. (2017) addressed this question using an ABX classification task with beginner and advanced Dutch learners of Mandarin. Results showed that, like the naive Dutch control group, beginner learners with 8–20 months learning experience were less accurate at classifying stimuli based on tones alone than Mandarin native speakers. The advanced learners with 3–14 years learning experience showed significantly higher accuracy than both the Dutch control group and the beginner learners, and did not differ significantly from the native Mandarin speakers. These findings indicate that the development of tone acquisition in L2 involves increasing cue strength on tonal cues along with improving proficiency. However, this study only investigated processing at a phonological level, in a task that did not involve lexical access. We thus still do not know whether advanced L2 learners will use tonal and segmental cues to similar extents as native listeners in word recognition. It remains possible that during a more resource-demanding, higher-level process such as lexical access, attention to the comparatively less familiar tonal cue will remain reduced even in more advanced learners. We explore this possibility here in a visual-world eye-tracking experiment, inspired by Malins and Joanisse (2010), in which we investigate how L1 and L2 speakers of Mandarin weigh tonal cues along with segmental cues in real-time word recognition.

As mentioned earlier, despite the wealth of research on tone in speech perception and lexical processing, very little is known about the relation between the two, especially in L2. As far as we are aware, only one study has directly examined this ‘phonetic–phonological–lexical continuity’ (Wong and Perrachione, 2007: 565) in L2 tone learning. In their study, English speakers with no experience of tone languages participated in multi-session training during which they were exposed to English pseudo-words with pitch patterns resembling Mandarin tones. A tone identification task and a spoken word recognition task were conducted to measure learners’ ability of perceiving pitch patterns and word learning success. Results showed a significant correlation between participants’ performance in the two tasks, supporting a continuity of phonetic–phonological and lexical abilities in adult L2 word learning. The generalizability of these findings to the acquisition of tone in L2 Mandarin, however, is somewhat limited given that stimuli consisted of English pseudo-words with Mandarin tones, and participants were naive English listeners with no learning experience of a tonal language. Inspired by Wong and Perrachione’s (2007) work, we aim to shed further light on the phonetic–phonological–lexical continuity in L2 tone learning by exploring the relation between performance on a categorical perception and a real-time word recognition task among English-speaking L2 learners of Mandarin.

A total of 30 native and 34 L2 speakers of Mandarin participated in this study. Data from 5 L2 participants was excluded due to exposure to Chinese in childhood (4) or professional music experience² (1), leaving data from 29 L2 speakers for analysis. Native speakers (21 female, mean age: 25.6 years, range: 20–36) were recruited from among the international student community at the University of Hawai‘i at Mānoa (n = 20), as well as at Peking University (n = 10). All native speakers reported being born in Mainland China and self-identified as native speakers of Mandarin.

L2 learners (11 female, mean age = 24.4 years, range: 19–41) were recruited at the University of Hawai‘i at Mānoa (n = 3), as well as at Peking University, the University of Hong Kong and Chinese University of Hong Kong (n = 26). All L2 learners self-identified as native speakers of English and started to learn Mandarin after age 12 (mean age of onset: 20.0 years, SD = 4.5). To ensure basic familiarity with the Mandarin vocabulary used in the experimental materials, only participants who were taking or had taken 3rd-year Chinese classes (or above) in the USA, or intermediate/advanced classes in China, were admitted to the study. L2 proficiency was assessed through self-ratings of speaking (M = 2.8, SD = 1.1), listening (M = 2.9, SD =1.1) and reading (M = 3.0, SD = 1.0) skills on a 5-point scale, as well as through a listening proficiency test adapted from the Hanyu Shuiping Kaoshi (HSK or Chinese Standard Exam) level 4 (Confucius Institute in Atlanta, 2017). The HSK level 4 test is a boundary test to differentiate intermediate to advanced proficiency. One L2 learner did not complete the listening proficiency test. The remaining 28 scored from 40% to 100% (M = 76.9%, SD = 17.5%). Information on music experience and language experience was collected by questionnaire. The study protocol was approved by the Institutional Review Board at University of Hawai‘i at Mānoa, and participants were compensated with extra course credit or a small amount of money.

Before coming to the lab, all participants completed a web-based questionnaire to collect information on basic demographics, language background, music experience, and self-ratings of speaking, listening and reading ability in both Mandarin and English. In the lab session, all participants completed the visual-world eye-tracking experiment followed by the identification task. L2 learners additionally completed the listening proficiency test at the end.

Data from a total of 2,124 trials (59 participants, 36 experimental items) on the eye-tracking experiment was first inspected for valid mouse-click responses. Trials with no mouse-click (L1: 1, L2: 1) and trials in which the participant clicked on an image before noun onset (L1: 4) were excluded, as were trials in which the timing of the click exceeded 3 SDs of the group’s average reaction time (RT) (L1: 16, L2: 17). The remaining data were inspected for track loss. Trials with more than 16% (= M+3SD) missing sample points were excluded (L1: 25, L2: 4). In all, a total of 3.2% of the data (68/2124 trials; L1: 4.3%, L2: 2.1%) was discarded.

Following standard procedures in categorical perception studies, we calculated the proportion of participants’ Sound-A vs. Sound-B responses (e.g. Sound A = T1, Sound B = T2, in T1–T2 pair) for each tone pair and step. The results, collapsed over the six different tone pairs, are illustrated in Figure 4. At both endpoints, participants in all groups were highly accurate at identifying tone on tokens with natural pitch and intensity, indicating no general difficulty in perceiving standard tones on isolated syllables. The slope of identification curves serves as a measure of the degree of categorical perception, with steeper slopes indicating more categorical perception (Xu et al., 2006). Visual inspection of Figure 4 shows that the L2-at-chance group had the shallowest curve and the L1 group had the steepest curve, while the L2-above-chance group patterned between the two. Slope values for each participant and tone pair were submitted to a linear mixed effect model with Group as a fixed effect and participants and tone pairs as random effects. Result showed that the slope for the L2-above-chance group (M = −1.82, SD = 0.57) was significantly steeper than for the L2-at-chance group (M = −1.49, SD = 0.60; b = 0.34, p = .002), but also significantly shallower than for the L1 group (M = −2.57, SD = 0.61; b = −0.75, p < .001).

Finally, in order to investigate whether listeners’ ability to perceive tone categorically was related to their use of tonal cues in spoken word recognition, we conducted correlation tests between participants’ identification slopes and their proportion of looks to the competitor in the SC condition in the visual world task. Recall that the latter was calculated over trials in which participants correctly selected the target. Since the number of such trials was low and variable in the L2-at-chance group, which was likely guessing in the SC condition, we confine this analysis to the L2-above-chance and the L1 groups. As illustrated in Figure 5, whereas no significant relation was found in the L1 group (τ = 0.17, p = .19), a strong positive correlation emerged in the L2-above-chance group (τ = 0.63, p = .001), showing that learners with steeper identification slopes were less likely to look at competitors. These findings suggest that difficulty with using tonal cues in L2 spoken word recognition is related to the ability to perceive tone categorically at a phonological level.