What are naturalistic comprehension paradigms teaching us about language?

Uri Hasson¹ and Giovanna Egidi¹

1 Center for Mind & Brain Sciences (CIMeC), University of Trento, Mattarello (TN), Italy, 38123

Chapter in press in R. Willems (Ed.), Cognitive Neuroscience of Natural Language Use. Cambridge University Press

Corresponding author:

Uri Hasson, PhD

Center for Mind & Brain Sciences (CIMeC)

University of Trento, Mattarello (TN), Italy, 38123

Email: uri.hasson@unitn.it

Phone: +39-3203055679

Abstract

Naturalistic paradigms of language comprehension offer a potential wealth of information for understanding how language processing occurs in everyday use. This information, however, is not immediately apparent and can only be interpreted when considering 1) basic processes that underlie language comprehension (e.g., memory encoding, memory retrieval, integration, prediction of incoming content) 2) processes that modulate or accompany comprehension (e.g., mood effects, attentional biases, emotional responses) and 3) the relation between language-induced activity and pre-existing, semantically rich baseline processes in the brain. Considering these issues conjointly, we outline a general interpretive framework for naturalistic studies of language. We argue that ignoring such issues can lead to serious misinterpretations of neurobiological data

1.Introduction

The study of natural language has been a topic of increasing interest in recent years. As outlined in other chapters in this collection, two central aspects of this line of research have been the departure from studying processes strictly limited to the scope of single words or sentences and a strong interest in characterizing the neurobiology of language processing as it occurs in natural circumstances. Our focus in this chapter will mostly be on the contribution of functional magnetic resonance imaging (fMRI) to this enterprise.

Indeed, a principal strength of studying natural language with fMRI is that the method offers a window into the comprehension process without asking participants to engage in either strategic analysis of the input (a typical demand in psychology studies) or any sort of overt behavior. The naturalistic approach to language comprehension with fMRI has been successful on two fronts. The first is testing hypotheses developed from research using non-naturalistic paradigms (e.g., identifying regions identified in action verb processing, e.g., (Wallentin et al., 2011b)). The second, more important one is obtaining new insights on issues that can only be studied within extended discourse, such as processing of narrative-scale event boundaries (Speer, Zacks, & Reynolds, 2007), identification of temporal integration scales in language processing (Lerner, Honey, Silbert, & Hasson, 2011) or distinguishing between networks involved in monotonic accumulation of information vs. processing of inconsistencies (Egidi & Caramazza, 2013).

Other strengths and features of the naturalistic approach to the study of language are reviewed in detail in this volume. Our goal in this chapter is to foreground interpretive difficulties that may emerge within naturalistic paradigms. We point to two specific ones: First, what we call the keyhole error, which is the tendency to interpret the function of areas tracking a language manipulation as being involved in language processing and second what we call the semantic baseline problem, which refers to the fact that quantifying brain activity during natural language comprehension in reference to an implicit and not-well understood baseline may result in theoretical misinterpretations. While these concerns apply, to some extent, to more traditional language studies as well, the lower level of control that characterizes natural language studies means that results obtained within these paradigms are particularly susceptible to these potential confounds.

2.Defining the keyhole error: Are we looking at linguistic processes?

The keyhole error is the tendency to interpret findings identified in natural language paradigms as indicators of language processing, which is akin to looking at the world through a keyhole and concluding that the world is structured very similarly to the keyhole (see Figure 1). As such, areas sensitive to inconsistencies may be interpreted as mediating integration of propositions (Hasson, Nusbaum, & Small, 2007), networks showing synchronized activity during processing of a narrative may be interpreted as sensitive to narrative content (Hasson, Furman, Clark, Dudai, & Davachi, 2008; Wilson, Molnar-Szakacs, & Iacoboni, 2008), and regions discriminating scrambled from coherent narratives may be taken to be linked in a meaningful way to linguistic computations (Xu, Kemeny, Park, Frattali, & Braun, 2005).

[Figure 1: The keyhole error: the world appears shaped like a keyhole when viewed through one. A view of Rome through a keyhole on the Aventine hill]

These interpretations may miss the mark because they do not distinguish processes that are considered core for language comprehension from two other types of processes that take place during comprehension. The first type are processes that accompany semantic and syntactic computations but do not reflect language processing. These can be thought of as affiliated processes, as they occur simultaneously with the computations that are considered core aspects of language comprehension. The second type are basic processes that serve information processing most generally in both linguistic and non-linguistic domains. This means that the same functions identified for language processing may be performed in other domains as well.

Affiliated processes

The areas mediating affiliated process are those where activity fluctuations track discourse features not because of these areas’ involvement in linguistic computations, but because of their involvement in processes that usually accompany language processing. These processes can manifest in the recruitment of additional regions (or networks) in support of linguistic functions or in the modulation of activity in regions that process language. The additional areas may be parts of systems associated with attention, emotion, or memory encoding or retrieval.

These processes likely interact with language comprehension in subtle ways; they may be a consequence of linguistic processing or accompany it from early stages. For example, emotional responses accompany the linguistic processing of a narrative and can influence comprehension of subsequent text (e.g., (Allbritton & Gerrig, 1991; Egidi & Gerrig, 2009). Similarly, increased relevance of a portion of a linguistic stream due to a semantic factor can determine the amount of influence of prior context on integration (Egidi & Gerrig, 2006). While the distinction between affiliated and core language processes is not clear-cut, it is yet an important distinction between two response profiles that may be found during language comprehension. Progress in neurobiology of language will allow researchers to make these distinctions more clearly.

Basic processes

A second manifestation of the keyhole error is that activation patterns in regions typically associated with language comprehension can reflect non-linguistic processes as well. The same brain regions involved in language processing may perform similar computations in other domains. Potential basic processes include those mediating the monotonic accumulation of information (Egidi & Caramazza, 2013; Humphries, Willard, Buchsbaum, & Hickok, 2001), which occurs in discourse comprehension but also in other domains that require integration of input streams. These processes also include the construction of a hierarchical narrative structure, which is a general process that underlies the comprehension of narratives communicated via language, vignettes, non-verbal cartoons, or movies as well as interpretation of short non-verbal communications that could rely on similar networks to those used in language (Xu, Gannon, Emmorey, Smith, & Braun, 2009). In addition, the construction, representation, and use of statistical information, thought to be a fundamental linguistic competence underlying grammar use and statistical learning, is a process that takes place with inputs that are less complex and familiar to people than language (e.g., tones and bird calls, (Tremblay, Baroni, & Hasson, 2012). The construction and evaluation of predictions at lexical and sub-lexical levels may be mediated by systems that play a general role in predictive processes.

Basic processes differ from affiliated processes in the type of computations they support: affiliated processes modulate comprehension but are not necessary for it. Basic processes, however, are indispensable to the computations that underlie language-related competence. Hypothetically, suspension of affiliated processes would only modify or partially impair linguistic processing, whereas suspension of basic processes would essentially disable linguistic processing.

Nuisance factors

A final cause of the keyhole error originates in the unique nature of the Blood Oxygen Level Dependent (BOLD) signal used in fMRI. Variations in the BOLD signal may, in some cases, be caused solely by physiological processes. For instance, brain regions where BOLD varies with fluctuations in autonomic nervous system activity (Birn, 2012) or eye movements (Ramot et al., 2011) overlap with regions involved in semantic processing. However, from the perspective of language theories of brain function these BOLD fluctuations may be considered epiphenomenal as they arise as a result of the same circumstances in which language is comprehended, but it is unclear whether they can affect the comprehension process itself.

In the following sections we overview the situations in which the keyhole error can more likely occur. We discuss how this error can be avoided by adopting a broad perspective of the neuroscientific and behavioral literature. Our discussion of the keyhole error is structured into the following sections. We first substantiate the proposition that language understanding can be thought of as engaging many of the systems involved in encoding real-world percepts. We then discuss the role of the following processes during language comprehension: the role of internally generated emotional responses, attention, memory encoding and memory reinstatement, general processes of semantic compositionality, the representation of statistical relations and prediction. Finally, we outline the mediating role of autonomic effects.

2.1. The experience of narrative: From language to the social world

A host of processes extensively studied in the context of social cognition are likely engaged during the comprehension of discourse: these involve empathy, perspective taking or more generally theory of mind and constituted affiliated processes to language comprehension. According to some theoretical positions, mostly based on behavioral evidence, the process of understanding discourse, in particular in the form of narratives, is similar in many respects to the experience of everyday life (Gerrig, 1993). On this view, narratives can achieve a sort of transportation. Comprehenders are shifted into a narrative reality, which to some extent substitutes real life as the frame of reference (Gerrig, 1993; Green & Brock, 2000). To illustrate, propositions that only make sense in a narrative world (e.g., an elephant flying) are processed easily even though they are clearly false in the real world. Indeed some neuroimaging work supports this proposition (Menenti, Petersson, Scheeringa, & Hagoort, 2009). The use of narratives in neurobiological studies of language can therefore engage a host of social-cognitive functions beyond elaboration of the linguistic stimuli. According to Mar and Oatley (2008), narratives simulate the interpersonal relationships in the story and induce affective empathy for the characters. For this reason, narratives are intuitively ascribed a crucial value as an outlet for empathic development and growth in literary and philosophical studies (Keen, 2007; Mar & Oatley, 2008).

We further note that for this reason, narratives are often used as media to convey a scenario, even when language processing is not the focus of the research. For example, the literature on theory of mind often utilizes narratives to study people’s ability to ascribe to others beliefs, emotions, and motivations, and there is a certain overlap between the core network involved in theory of mind and the regions associated with narrative comprehension (which also include areas of the default-mode network; Mar, 2011), thus demonstrating a certain overlap between basic social functions and language comprehension. Consequently, regions theoretically linked to language operations may be involved not in the semantic/syntactic operations that are at the core interest of linguistic theories, but in processes that are more limited in scope and deal with the contents used in extended narratives, which oftentimes refer to social interactions. We note that the effects of specific contents have been investigated in several prior works, such as those investigating comprehension of language describing biological motion (Deen & McCarthy, 2010), inconsistencies in emotional or temporal content (Ferstl, Rinck, & von Cramon, 2005), comprehension of action-related information (Desai, Binder, Conant, & Seidenberg, 2010), or processing of linguistically-conveyed descriptions of visually vivid, emotional, or action-related content (Chow et al., 2013).

2.2. Emotions and language processing

One difficulty in interpreting brain activity patterns evoked by naturalistic stimuli is that they have the power to invoke comprehenders’ emotional responses perhaps more than controlled stimuli. As such, neurobiological systems implicated in the generation of emotions or the interface between emotion and information processing may be strongly linked to language processing, with emotion being a major affiliated process. There is work showing that narrative-induced fluctuations in emotion may be related to activity in language processing networks (Wallentin et al., 2011a), and that generally, processing emotional content is associated with different activation profiles in lateral temporal cortex, and with particular connectivity modes of left IFG and MTG (Chow et al., 2013). For example, in the study by Wallentin and colleagues (Wallentin et al., 2011a), an independent group of participants rated each sentence of a children’s story for arousal (on a scale from extreme boredom to extreme arousal) and valence (on a scale from strong negative emotions to strong positive emotions). This resulted in time series of fluctuations on these two dimensions. These time series were then used as explanatory variables in regression models that accounted for BOLD activity while a different group of participants listened to the story. The results showed that increased arousal was associated with stronger BOLD activity in a network of bilateral frontal and temporal regions that are usually associated with language processing, thus demonstrating an overlap between regions involved in linguistic and emotional processing (see Figure 2). In addition, the study also found that positive valence was associated with increased activation in inferior parietal and medial regions, which are part of the default-mode network and whose activity is associated with mind wandering, which is a state clearly linked to manipulation of contents (see Mason et al., 2007).

These findings are consistent with cognitive theories of comprehension, which have highlighted the sensitivity of comprehenders (in particular readers) to emotional aspects of a narrative. Because of people’s ability to be transported into a narrative (Gerrig, 1993; Green & Brock, 2000), they experience emotional responses to the content of a narrative, develop preferences for certain outcomes, or empathize with characters. Behavioral work has shown that these forms of engagement can impact the comprehension process itself so that information that does not accord with comprehenders’ preferences can be more difficult to integrate with prior context. For example, when a story protagonist is depicted as honest and deserving, readers have a preference for a narrative outcome with a positive implication for the protagonist, but when a protagonist is depicted as dishonest, readers’ show the opposite preference. This preference is seen in that endings that mismatch these preferences are read slower than matching endings (Allbritton & Gerrig, 1991; Rapp & Gerrig, 2006).

Narratives are also considered powerful devices for inducing people to experience moods. Indeed for this reason narratives are often used in experimental psychological studies to place participants in different moods, and have been shown to be one of the most effective methods of affect induction (e.g., (Gerrards-Hesse, Spies, & Hesse, 1994; Westermann, Spies, Stahl, & Hesse, 1996). The narratives used in these studies are in the form of humorous or gloomy written texts (Egidi & Gerrig, 2009; Forgas, 1998), movies (Egidi & Nusbaum, 2012) or are self-generated (e.g., (Bless et al., 1996)). Being in a happy or a sad mood after such mood induction has been shown to affect several aspects of language processing. For example, it can affect how consistency is built in discourse processing. When comprehenders placed in a happy or sad mood read stories that can end with a positive or a negative ending (both of which are logically consistent with prior context and equally likely to occur) they judge as more surprising those endings that do not match their mood’s valence (i.e., happy endings are judged as more surprising by sad participants and sad endings by happy participants; (Egidi & Gerrig, 2009). Moods also elicit a rapid neural response that signals difficulty of integration: a greater Evoked-Related Potential (ERP) N400 component is found for positive or negative story endings that mismatch listeners’ moods, even though these endings are linguistically consistent with prior context and equally likely to occur (Egidi & Nusbaum, 2012).

Emotional processes that emerge during discourse processing, particularly within narratives, can therefore affect subsequent linguistic processing and should be considered within neurobiological theories of language. These findings also suggest that a strict dichotomy between “cold” and “hot” linguistic processing should be reconsidered. The networks engaged in the generation of emotions triggered by content, and the networks in which these emotions subsequently moderate operations related to language comprehension may overlap with those linked to lower level language processing, or may interface with them sporadically in a non-stationary manner. Understanding brain mechanisms implicated in the continuous fluctuation of arousal levels and internal emotional states would be a natural way of partitioning between the two processes. In the meanwhile, the interpretation of activity patterns that track linguistic content should be done while keeping in mind that such actiity may reflect the waxing and waning of internal emotional states.

2.3. Attention and language processing

An interesting property of naturalistic stimuli is that their richness allows comprehenders relative freedom in selecting what discourse topics are of interest or deserve greater attention. Thus, activity observed during comprehension can reflect an interaction between the semantic, syntactic or discourse-level features of the stimulus on the one hand, and comprehenders’ own motivations on the other. Such covert shifts in attention are an affiliated process that is worth of examination in the context of language comprehension. Interestingly, differences in attention orientation can induce changes in activation in regions strongly linked to language comprehension. For example, Cooper et al. (2011) presented participants with narratives and asked them to focus either on the temporal aspects of the events (i.e., when events occur), on spatial aspects (i.e., where events occur), or on action aspects (which events occur). The stories presented were identical, yet the results documented significant differences in lateral regions involved in language comprehension: the superior temporal gyrus (STG), superior temporal sulcus (STS) and supratemporal plane (STP) between the three conditions, with stronger activity for the action-tracking and space-tracking conditions than for the time-tracking condition. The attentional manipulation affected not only the magnitude of activity but the distribution of regional activity pattern as documented by a multivariate analysis. Specifically, when quantifying the activation pattern across all voxels in left inferior frontal gyrus’ (IFG)-Pars Triangularis (observing all voxels in region as unit of analysis) the results showed a relatively strong similarity in activity patterns for the space- and action-monitoring conditions only, whereas the time-monitoring condition was associated with different pattern of distributed activity. Thus, even core language regions such as IFG are sensitive to strategic modulations of attention that comprehenders adopt. This suggests a strong top-down influence of other neural systems over basic integrative linguistic functions.

General attention systems in parietal cortex have been linked to semantic and pragmatic aspects of language processing. Kristensen and colleagues (Kristensen, Wang, Petersson, & Hagoort, 2012) manipulated speech prosody for a set of sentences so as to place specific words in focus. When the activation associated with listening to these sentences was compared to that associated with listening to prosodically unmarked texts, a set of regions including the inferior and posterior parietal cortex was found. Independently, these same areas were identified for this participant group in an auditory spatial attention task. The author took these findings to indicate that the processing of pitch accent utilized, at least in part, a general-attention network, concluding that, “linguistic attention is not separate from non-linguistic attention.” (Kristensen et al., 2012, p.8).

Other evidence of the role of attention in language processing is provided by work investigating the neural substrates of local and global integration during narrative comprehension (Egidi and Caramazza, 2013). The study found that integration of a sentence with distal context involves systems implicated in working memory load (Wager & Smith, 2003) or top-down attention (Cabeza, Ciaramelli, Olson, & Moscovitch, 2008; Corbetta & Shulman, 2002). The former system accesses the distal information that is relevant at the time of integration, and therefore shows more activity when more contextual information needs to be available. This process is similar to what occurs in task-cueing paradigms (Phillips, Velanova, Wolk, & Wheeler, 2009). The latter system uses context in a top–down preparatory manner for the most sensible information, similarly to what happens in visual top-down attention (Cabeza et al., 2008; Corbetta & Shulman, 2002). These results show that even basic linguistic functions such as accessing (or holding in working memory) relevant contextual linguistic information and integrating it with incoming input are fundamentally attentional functions.

The link between attention and language is not limited to the discourse or sentence level, but is also found at lower levels of language processing. At a sub-lexical level, attention to auditory stimuli increases activity in the superior temporal gyrus (Petkov et al., 2004). One concern therefore is that factors that affect the attention to the auditory stimulus itself could manifest themselves in STG activity changes.

2.4. Memory encoding is part of language processing

Memory processes form a core element of language comprehension. As people understand what is said, they encode the content to memory, and maintain it so that it is accessible when later content is introduced. Thus, continuous encoding to memory constitutes a basic process that underlies language comprehension, narrative comprehension and a multitude of additional cognitive domains. Traditionally, however, neurobiological studies have treated comprehension and memory as separate functions, consistent with the view that they are subserved by at least partially non-overlapping networks. However, the few language studies that have investigated the relationship between comprehension and memory encoding suggest a different picture.

Hasson et al. (2007) examined the relation between the network implicated in a semantic contrast between inconsistent and consistent story endings and the network implicated in a memory-related contrast between remembered and forgotten story endings. The study found a striking overlap between the two functional networks thus defined, indicating a shared cortical topology between inconsistency detection and encoding to memory. These data appear to indicate that it is a categorical error to conceive of semantic integration and memory encoding as two distinct cognitive processes. Instead, it appears that the memory encoding is achieved, at least in some networks, as a direct outcome of information processing in the region. Work by Hasson et al (2008) also showed a similar pattern using movies and a different analysis method. In this study, inter-subject synchronization of BOLD time series during movie viewing was associated with better memory for movie content, in regions including the right anterior temporal cortex and STG. In addition, greater activity in bilateral IFG was related to memory in this manner as well (though in absence of changes in synchrony).

The work by Egidi and Caramazza (2013) mentioned in section 2.3 suggests that not one, but several non-overlapping networks subserve the encoding of language content to memory, and these depend on the nature of language computations performed. For texts where the meaning of a final sentence was determined only by the most recent context, there was a relatively weak relation between brain activity during ending comprehension and subsequent memory for this ending. In contrast, when the meaning of the final sentence depended on integration with a broader context, activity in a bilateral fronto-temporal semantic network was related to subsequent memory. Thus, the amount of information considered during integration and the ease with which a sentence is integrated with recent or more distal context is related to the networks that encode content to memory.

2.5. Accessing context: More on the role of memory processes

An account of mechanisms underlying contextual integration is lacking in naturalistic studies of language. The dominant paradigm for studying contextual effects consists of placing target sentences against the backdrop of different sorts of contexts. The actual mechanisms implicated in the integration, however, have not been specified to date. In general, it is accepted that incoming content is integrated (or evaluated) in relation to the cognitive representations of the prior context. The questions that remain, however, are how much context is made available at any given point, and how it is accessed online. Naturalistic studies seem to hold the implicit assumption that integration takes place against a representation of the context where contextual elements are equally accessible, and indeed context is modeled as ‘holistic gestalt’. This is perhaps most evident in theories arguing that discourse is represented via simulation-like mental models, where the resulting representation is dependent not on the order in which information arrives, but the overall situation it describes. This assumption props the view that from a neurobiological and cognitive perspective, shorter and longer narratives are essentially processed via the same systems.

Recent work (Egidi & Caramazza, 2013), however, suggests that the online construction of meaning during language comprehension does not follow this simple process model. Instead, separate systems appear to be invoked depending on whether incoming information is integrated against recent or distal context, and whether the incoming information is coherent with more or less recently introduced content. Specifically, the study identified a set of regions that showed greater activity when a final sentence in a story was consistent with local context (mostly parietal, insular, and medial regions). It also identified regions sensitive to attentional load, including the supramarginal gyrus (SMG), superior parietal lobule (SPL) and the anterior intra-parietal sulcus (aIPS), which showed greater activity when global context was relevant for the comprehension of this final sentence. Finally, the study identified regions associated with top–down attention processes (SPL, IPS) that reflected readiness to the most sensible incoming information once the global context had been taken into account. Thus, the length of a text may in and of itself induce activity in different systems. Minimal texts that can be understood by relying on local textual relations may utilize one system, whereas those that require integration over a larger context may utilize other systems. The implication for naturalistic paradigms is that one cannot assume that comprehending different types of texts fundamentally involves the same brain networks. Thus, just as word and sentence processing are usually considered as regulated by different mechanisms, we argue that there are different levels of processing within discourse comprehension itself, as the integration process is highly sensitive to the amount of prior context that is potentially related to incoming information.

An important determinant of discourse processing are mechanisms by which prior context (linguistic and extralinguistic) is brought to bear on the integration of incoming information. Psychological theories of discourse comprehension have proposed memory-based mechanisms, mostly automatic, that allow re-accessing prior context via memory cues present in working memory. These cues trigger a resonance process that selectively increases the activation of relevant prior information (Gerrig & O'Brien, 2005). This work has shown that context is not maintained active so that all information is equally accessible at all times. Instead, the relative accessibility of prior information waxes and wanes as function of its relation to incoming information (Gerrig & McKoon, 2001). This resonance process appears to be automatic and non-strategic in that it can bring to mind even prior information that is irrelevant to the sentence being understood. From this perspective, continuous and automatic memory retrieval is a core aspect of online comprehension (Gerrig & O'Brien, 2005). Other information tracking mechanisms, mostly strategic, have also been proposed. According to this latter view, comprehenders keep track of relevant information contained in a narrative (e.g., time, place, characters’ goals) and continuously update their situation model of the text each time they encounter a modification of these parameters (van den Broek, Rapp, & Kendeou, 2005). However, despite the demonstrated importance of both memory-access mechanisms for discourse comprehension, the neurobiological bases of reinstatement of discourse contexts have not been examined to date.

Interestingly, the study of the neural correlates of re-instatement has been gaining significant prominence in studies of memory, though with yet little impact on neurobiological theories of language comprehension. Examining this work may shed light on how information is continuously re-accessed and re-elaborated in relation to incoming information. Particularly relevant work, (Johnson, McDuff, Rugg, & Norman, 2009) evaluated whether the subjective experiences of having previously encountered an item or being familiar with it, rely on similar neural networks. They presented participants with words to memorize under three different encoding contexts. Using multi-voxel pattern analysis, they showed that activation patterns during a recognition task could discriminate which of the three contexts the word was encoded. Classification was accurate for both recollection (when participants explicitly remembered having seen the word in the list) and familiarity (participants were sure it was old). The areas that could discriminate the encoding context based on recall activity were regions highly involved in language processing: left IFG, left superior frontal gyrus (SFG) and posterior STS and middle temporal gyrus (MTG). Thus, this study shows that regions associated with core language functions are involved in fundamental memory-based processes.

It is important to note that contextual reinstatement is detailed and can therefore be a good basis for accessing prior context during language comprehension. Activation patterns during recognition of a target word can signal whether the word was encoded in the context of negative or positive emotional information, in an auditory or visual modality, or even whether it was encoded in relation to visual spatial information or visual object information (Danker & Anderson, 2010). This may explain how incoming contents encountered during language comprehension prompt retrieval of prior context.

Thus, both behavioral and neurobiological work on contextual reintegration suggest that an important aspect of activity evident in natural language paradigms may coincide with the reactivation of prior information and the context in which it was introduced. Separating the networks mediating this core memory process in linguistic contexts is an open question for future research.

2.6. General compositional processes shared by linguistic and non-linguistic domains

Whether semantic integration processes identified with natural language paradigms are language-specific or are shared by other domains is a fundamental question that has not been extensively examined in the literature. The emerging picture is that there are neurobiological systems mediating general compositional processes for both linguistic and non-linguistic domains. For example, Humphries et al. (Humphries et al., 2001) presented two types of stimuli to participants in an fMRI study: either sentence pairs or sequences of environmental sounds that described the same events (e.g., either the sentences there was a gunshot and then someone ran away or the sound of a gun and the sound of footsteps fading into the distance). Both types of stimuli evoked neural activity in bilateral temporal regions associated with semantic processing, including posterior superior temporal gyrus and sulcus, thus showing a common neural substrate for the semantic processing necessary for sentence and sound-sequence comprehension.

Visual and language-conveyed narratives may also rely on the same cognitive systems. It has been shown that visually presented narratives containing an unexpected element evoke an ERP response similar to the N400 found in language comprehension (Sitnikova, Holcomb, Kiyonaga, & Kuperberg, 2008). This study by Sitnikova et al. (2008) showed that actions inconsistent with a narrative goal additionally evoked a P600 effect often associated with syntactic repair. Such findings confirm the reasonable expectation that high-level integration processes underlying linguistically conveyed narratives do subserve comprehension of narratives in general. Analogously, semantic processing of verbal stimuli and environmental sounds has been found to elicit similar neural responses (e.g., (Cummings et al., 2006; Saygin, Dick, Wilson, Dronkers, & Bates, 2003)).

The comprehension of narratives, whether read or presented in forms of movies, may rely on systems that code for the hierarchical organization of events. This hypothesis has been extensively investigated by Zacks and colleagues (e.g., (Zacks et al., 2001; Zacks & Swallow, 2007); see Chapter X in this volume). Beyond identifying networks whose activity tracks event boundaries for narrative text (Speer et al., 2007) and action sequences (Zacks et al., 2001), this group’s research has also examined movie perception. In one study (Zacks et al., 2001), participants were scanned while watching a movie. They then watched it again while segmenting it to coarse or fine-grained events. Regions that tracked event boundaries included mainly posterior regions—both medial and lateral (cuneus, precuneus, SPL, posterior cingulate, right parieto-temporal regions), with smaller involvement of frontal cortex. Whitney and colleagues (2009) coded narrative shifts in auditory stories by identifying changes in basic narrative building blocks, such as character, time, location and action. By using random sentence boundaries as a control condition, the authors identified brain regions where activity tracked such changes. They documented sensitivity to narrative shifts in the right precuneus and posterior cingulate cortex, as well as the middle aspect of cingulate gyrus on the left. The role of this network in responding to event boundaries was also identified in several other studies based on verbal materials (e.g., (Speer et al., 2007)) thus showing a common neural architecture underlying comprehension of events in narratives conveyed by several media and real-life experiences as well.

Do these findings indicate language processing or a more general semantic integration process? Work by Hasson and colleagues, inspired in part by methods developed by Zacks and his group, suggests that the brain systems implicated in event segmentation may actually be mediating an even lower-level function that is not related to semantic schema-based knowledge but to more general changes in environmental structure. In one study (Tobia, Iacovella, Davis, & Hasson, 2012) participants listened to long series of rapidly presented tones (rate = 3.3Hz) in which 4 tones were presented repeatedly. Some sections of the series were highly ordered as determined by their transition constraints (e.g., 123441233412334) whereas others were random; over time-scales of about 10 seconds, shifts of regularity occurred so that participants had several opportunities for noticing changes in regularity. Participants indicated when they noticed a change in regularity in the auditory sequence. On the basis of these ratings, it was possible to quantify points in the auditory stream where there was consensus regarding regularity changes, thus allowing the identification of brain systems where activity prior to the key press was associated with the perception of a change. This analysis identified a network of regions whose topography strongly resembled that found in the studies by Zacks and colleagues.

Another approach (Tobia, Iacovella, & Hasson, 2012) examined if there are neural systems sensitive to changes in the degree of input-regularity over time, in particular whether these regions overlap with language processing areas. Participants passively listened to the same type of stimuli used in Tobia et al. (Tobia, Iacovella, Davis, et al., 2012). The relative degree of disorder in 10-second sliding windows (consisting of 32 transitions between 4 tones) was manipulated to change slowly over time, so that it was possible to formally calculate the rate in which regularity changed over time and identify regions sensitive to the degree of change in regularity. The analysis revealed a set of midline regions including the lingual gyrus and cuneus bilaterally, as well as the central aspect of the cingulate gyrus, the pre-supplementary motor area (SMA), the left insula and left ventral premotor cortex (vPMC).

In summary, examining the neurobiological basis of the segmentation of inputs into units is a promising framework for future work. It seems that this system, consisting mainly of posterior midline regions in occipital cortex and central midline regions of the anterior cingulate mediates both schema-based segmentation of narratives as well as purely statistical-based parsing of non-narrative inputs. It is possible be that the ability to segment narratives into units developed from the simpler coding of statistics in the context of perceptual learning.

2.7. Core systems for the representation of statistical relationships

The domain of prediction and statistical learning has been repeatedly linked to that of language processing (on both the lexical and sub-lexical level), but the relation between the two is still a new domain of research. Predictions depend on a statistical knowledge base, but research identifying the systems that support such predictions has not arrived at clear-cut conclusions. Some have proposed that there is a basic (hippocampally-centered) system with the capacity to represent statistical relationships between elements. On the micro-scale of individual tokens, this system supports associative learning, for example, that item B tends to appear after item A (Turk-Browne, Scholl, Johnson, & Chun, 2010). Furthermore, it has also been suggested that the same system encodes macro-scale statistical features of the input so that it tracks the overall regularity (entropy) of input streams (Strange, Duggins, Penny, Dolan, & Friston, 2005). Such a system may play an important role in language processing, as there are multiple levels of regularity at different stages of linguistic processing: phonemic, syllabic, morphological and sentential. Theoretical argumentation and behavioral research have argued for the existence of such a basic system. The auditory scaffolding hypothesis (Conway, Pisoni, & Kronenberger, 2009) exemplifies this approach in arguing that people’s intense experience with sound patterns provides a supporting structure, or scaffold, for the capacity to track sequential structure in multiple domains, including non-linguistic ones.

There is evidence that brain regions that are considered to perform core language functions also mediate statistical processes. For example, Peterson and colleagues (Petersson, Folia, & Hagoort, 2012) have shown that the left IFG is sensitive to violations of a previously learned artificial grammar. This result supports the notion that the involvement of this region in syntactic processing may be due to its more basic involvement in representing abstract grammatical structure. The authors suggest that the left IFG is not involved in specific language-related computations such as the evaluation of long-range dependencies (syntactic movement) or representation of nested structure but ‘is a generic on-line structured sequence processor’ (Peterson et al., 2012; p. 89). It has also been suggested that left IFG is sensitive to the complexity of structured inputs. In one study (Bahlmann, Schubotz, Mueller, Koester, & Friederici, 2009), artificial grammars based on a simple associative transition structure (modeled via 1st order Markov process) were contrasted with more complicated grammars that consisted of a hierarchical dependency rule of the form (e.g. An Bn) where A and B were visual stimuli with different feature types. A region-of-interest analysis of posterior left IFG (BA44) showed greater activity for the hierarchical condition. The authors interpret the findings as supporting the hypothesis that the region is ‘engaged in domain-general processing of hierarchical sequences’ (p. 166). In addition, the hierarchical grammar evoked greater activity in right pre-sma and the left precentral gyrus (PCG) and sulcus, which are areas extensively involved in both general auditory sequence learning (Brown et al., 2013) and explicit prediction of future stimuli (Schubotz, 2007). The involvement of these regions in predictive processes may indicate that differences in activity are not due to differences in complexity, but to the fact that naturally occurring predictive mechanisms were more heavily utilized in one condition. We note however, that this study, conducted in a non-naturalistic paradigm, required grammaticality judgments after each stimulus, and these could have been more difficult for the hierarchical stimuli, thus biasing some of the mentioned BOLD difference patterns.

The study by Tobia et al. (2012b presented in section 2.6) further found that structure-tracking may take quite nuanced forms, with immediate applications to language research, as this tracking is mediated by lateral temporal and ventral premotor regions. One design feature of that study was that when computed over 10 sec sliding windows, two distinct features of the stimuli would change continuously: the relative diversity of tokens (the extent to which the proportion of different tokens were similar), and the strength of contingencies between them (the degree to which each token predicted the subsequent one). The results indicated sensitivity to both relative diversity and strength of contingencies in areas typically linked to language comprehension. Sensitivity to diversity was found bilaterally in the posterior aspect of supratemporal plane, SMG and insula. Sensitivity to the strength of transition constraints was found in several regions including the ventral premotor cortex and lateral temporal cortex. Thus, when a context (even an overly simplified one) licenses predictions via frequency- or transition-based constraints, different networks are invoked. They may be implicated in tracking the regularity of the context or even in making predictions. Given the involvement of both premotor and temporal cortex and sub-lexical and lexical processing, it is important to consider whether these regions may be more generally implicated in prediction per se.

Neurobiological studies of responses to musical structure reveal similar systems (Seger et al., 2013). Seger et al. presented participants with classical music pieces that could end with an expected resolution, or 3 types of resolutions that varied in their degree of expectancy violation. They found sensitivity to the degree of musical surprise in left anterior STG, posterior aspect of left IFG and SMA. All these regions showed increased activity with greater surprise.

Despite such findings, overlaps in activation patterns as seen for music, language, and artificial grammars do not necessarily constitute evidence of similar computations in these domains. Rogalsky et al. (Rogalsky, Rong, Saberi, & Hickok, 2011) targeted this issue in a study comparing activation patterns to jabberwocky sentences, scrambled jabberwocky sentences and musical progressions. In regions where activity overlapped for music and sentences, a multi-voxel pattern analysis (MVPA) classifier still successfully discriminated speech from music. In addition, regions associated with syntactic well-formedness (sentences as compared to scrambled sentences) were not associated with musical processing. Thus, although these regions track structure and regularity in several domains, they are still sensitive to the type of input being processed.

To conclude, while there is some evidence that temporal, inferior frontal and premotor regions associated with language comprehension may mediate lower-level prediction based processes, more work needs to be done on whether similar computations are carried out when processing environmental inputs, music or language itself.

2.8. Autonomic effects

We have already introduced the interpretive difficulties that may arise as a consequence of emotional responses to discourse content in naturalistic language paradigms (see section 2.2). It is reasonable to expect that such responses will be manifested in particular activation patterns, e.g., in the mesolimbic system as these regions may invoke the emotional response itself. A similar, but fundamentally different effect of cognition/emotion/attention on BOLD activity is introduced when the Autonomic Nervous System (ANS) affects BOLD fluctuation patterns via a purely physiological (non-functional) pathway (see (Iacovella & Hasson, 2011) for a recent discussion). In this pathway, cardiac and respiratory fluctuations affect BOLD patterns due to variations in cerebral blood flow, vasodilation, concentration of carbon dioxide, and even respiration-induced motion. Somewhat alarmingly, regions where the BOLD signal is affected in this way overlap with the distribution of regions identified as the default-mode network (DMN), a network involved in semantic processing (Binder, Desai, Graves, & Conant, 2009).

The aforementioned effects are considered purely physiological. However, there are brain systems that monitor ANS activity, perhaps as a source of additional (visceral) information. For instance, the insula has been implicated in monitoring of arousal, and particularly cardiac rate (Pollatos, Schandry, Auer, & Kaufmann, 2007). Relatedly, the somatic maker hypothesis (Bechara, Damasio, & Damasio, 2000) suggests that regions involved in decision making may monitor visceral (though more emotional) responses and integrate those in decision contexts.

The study of Wallentin and colleagues (Wallentin et al., 2011a) described in Section 2.2, in which brain activity during listening to a children’s story was correlated with arousal and valence ratings for each sentence, also calculated the correlation between the arousal ratings and an ANS measure collected during story listening. This measure reflected Heart-Rate Variability (HRV) and was quantified as the ratio of low- to high-frequency power in the cardiac response profile. The study found a correlation between the arousal ratings and this ANS measure. As discussed earlier, the results also showed that increased arousal was associated with stronger BOLD activity in frontal and temporal regions usually associated with language processing. The heart rate variability measure may therefore have also contributed to the BOLD pattern of results, thus showing a common neural substrate not only to language and emotion, but to autonomic responses as well.

Metz-Lutz and colleagues (2010) also documented an impact of narrative events on ANS indices. They presented participants with a dramatic play in an MR scanner, and later asked those participants to provide a report of their thoughts and feelings for every unit of the play. They could thus identify events associated with participants’ adhesion to the narrative, that is, moments in which they experienced the narrative as real. They also identified a more objectively defined set of events based on the director’s instructions. The findings brought out an interesting relation between event structure and the ANS index. Both objectively defined and subjectively defined narrative events were linked to changes in heart rate patterns (as quantified via a heart-rate variance measure). Specifically, both events were associated with a reduction in the HRV autocorrelation function, i.e., reduced power in the lower frequency of the HRV. This pattern indicates vagal influence on heart rate activity, i.e., involvement of the parasympathetic system that mediates more temporally-extended physiological processes. In addition, subjectively-defined events were also associated with invoking a more rapid heart rate. Here too then the literature points to systematic relations between narrative structure and changes in autonomic function, which can easily lead to BOLD changes in and of themselves via the pathways discussed above.

To summarize, the existing literature suggests that contents that evoke emotional responses cause concomitant changes in BOLD fluctuations via different pathways, some of which are related to functional activation, and some which are due purely to nuisance factors and can be treated as epiphenomenal. Simply partialling out the effect of simultaneously recorded physiological responses from the BOLD signal is a poor solution to the problem, since, as mentioned, these changes can be meaningfully related to the cognitive and emotional states introduced by the content.

3. The semantic baseline problem: Are we looking at qualitatively meaningful evoked changes or induced changes to ongoing activity patterns?

The final point we address is that quantifying activity relative to baseline may result in misinterpretation of the role of brain regions or networks. This is because the baseline state itself may constitute a cognitive state similar to the one in place during semantic processing of language content. What we call the semantic baseline problem thus refers to the difficulty in determining whether the observed activity fluctuations reflect a qualitatively important perturbation to an otherwise impoverished baseline state, or alternatively, a relatively subtle modulation of baseline activity that is inherently rich in semantic processing.

On the interpretive level, the above-mentioned concerns are similar in nature to the distinction between evoked and induced activity as raised in the fields of Electroencephalography and Magnetoencephalography research. In those, evoked activity refers to activation patterns that are phase locked to a stimulus; i.e., these reflect activity in a system that was inactive prior to stimulus presentation. In contrast, induced activity refers to a change in ongoing brain activity profiles that are in place prior to stimulus presentation, e.g., changes in the power or phase of a particular frequency band that exhibited oscillation prior to stimulus presentation. As such, evoked changes are identified by averaging activity time-locked to appearance of particular stimuli, whereas induced activity can only be identified when averaging time-frequency snapshots of power profiles after stimulus appearance.

If what we observe during studies conducted with natural language paradigms is only a minor shaping of the baseline state, then the background can and should be treated as a dominant cognitive element, with language serving as a subtle but systematic modulator of that state. For this reason, understanding the relation between baseline processes and activity patterns in natural language paradigms is a core issue for a correct interpretation of experimental findings. While it may appear more intuitive to interpret findings under the assumption that language processing induces fundamental changes, there is a body of work suggesting that brain networks implicated in particular cognitive processes may maintain structured connectivity outside the context of those cognitive functions.

The principle has been demonstrated in a number of studies. For instance, Simmons and Martin (2012) showed that regions associated with tool use or social cognition (i.e., regions activated by tool- or social-related contents) maintain their differential connectivity patterns during the resting state. Networks sensitive to action observation also show coherent fluctuations during rest (Molinari et al., 2013).

With respect to language, several studies suggest that regions known to mediate language comprehension form functional networks during the resting state; that is, they do not organize ad-hoc when language input is experienced, but maintain functional connectivity outside a language processing context. One study (Hampson, Peterson, Skudlarski, Gatenby, & Gore, 2002) examined whole-brain resting-state correlations of IFG regions (BA44, 45) and documented strong connectivity with bilateral STG and MTG. A more specific analysis examined correlation between left IFG and left BA39 (including also the angular gyrus). The correlation between these two regions was significant, and the strength of correlation varied with scores on a reading comprehension subtest.

Expanding on this work, Koyama et al. (2010) examined the resting-state functional connectivity of 6 different left hemisphere seed regions that have been specifically linked to reading and spoken language comprehension (i.e., fusiform gyrus, IFG, PCG). Some of the networks demonstrated topography typically found during word reading, for example, connectivity between FFG and IFG. Most interesting, when examining the overlap of these 6 networks, common overlap for 5 of the 6 were found in left posterior MTG and the posterior aspect of left IFG. The authors suggested that these resting state connectivity patterns may be the consequence of people’s intensive experience with reading, which results in continued maintenance of the networks offline. Whether these networks maintain some type of endogenously driven semantic processing is an interesting question for future work.

The aforementioned findings lead to the question of whether synchronized activity that constitutes resting state networks is maintained during comprehension, and whether language comprehension at all instantiates ad-hoc connectivity networks. Keeping in mind that BOLD synchronized activity is typically dominated by power in lower frequencies (< 0.1 Hz), it is important to know if this synchronized activity is actually perturbed during processing of exogenous stimuli. Lohmann et al. (2010) examined this particular issue. They analyzed data from 4 studies of language processing and 2 studies not involving language processing in a way that enabled tracking low frequency BOLD fluctuations independent of the experimental paradigm per se. They partialled out variance that could be attributed to the experimental design, low-pass filtered the data to focus on the low-frequency range (< 0.1 Hz), and examined connectivity patterns in the signal. This approach could determine whether language processing specifically induces patterns of functional-connectivity that are independent of the experimental timing per se (given the low frequency filter). Seed regions were defined in BA44 and frontal operculum, and the authors documented strong connectivity to contralateral right hemisphere areas as well as left STS. Importantly, this connectivity with left STS was not found in the 2 non-language studies. Taken together, these studies show that some semantic-processing networks are maintained during rest, but that language processing may be further associated with instantiation of particular functional-connectivity patterns in the low-frequency range. Thus, extended language inputs may either make use of pre-existing connectivity networks or instantiate specific networks during comprehension.

Speaking in favor of the idea that language processing may reflect modulations of activity in pre-existing networks are several studies that have specifically targeted the nature of resting state activity and its susceptibility to the specific thoughts experienced during rest. Doucet et al. (2012) examined how resting state activity within 5 networks depended on the nature of endogenous thoughts reported by participants. They derived a representative time series from each network, and determined whether connectivity between these networks varied with participants’ (self-reported) engagement in imagery or inner language. They found that increased engagement in internal language was associated with reduced activity in the default-mode network and a frontoparietal network. A similar study (Preminger, Harmelech, & Malach, 2011) showed that specific types of guided internal deliberations (near future planning vs. episodic retrieval) can systematically alter activity in areas implicated in mind wandering. They documented a number of regions that showed higher activity during near-future planning, and interestingly, these included the lateral temporal cortex bilaterally, IPL bilaterally and the left IFG/IFS. Thus, internal deliberation in absence of language processing can result in systematic activity changes in areas implicated in semantic access (the so-called DMN; (Binder et al., 2009)) as well as regions associated with sentential semantic and syntactic processing (IFG, temporal cortex).

Finally, we consider the possibility that natural language paradigms can induce changes in what are naturally occurring fluctuation dynamics in a given region. To illustrate: Skipper et al. (2009) showed that meaningful gestures are associated with an interesting activation pattern in the ventral and dorsal premotor cortex: when a movie contained meaningful gestures, activity peaks in these regions were associated with gestures. In contrast, when a movie contained non-meaningful gestures, these were equally associated with peaks and pits in the time series. Interestingly, peaks in the BOLD signal occurred quite frequently in both conditions, with a mean interval of ~ 7 sec. The findings suggest that the premotor cortex, which is involved in action production, is sensitive to the meaningfulness of gestures in the context of language comprehension. It is possible however that gesture-based content does not directly evoke these often-occurring peak/pit patterns, but serves to simply adjust an existing intrinsic (baseline) activity mode of the region. Recent work on intrinsic profiles of rapid fluctuations during rest (Davis, Jovicich, Iacovella, & Hasson, 2013) demonstrates that the premotor cortex shows a spontaneous, non-random pattern of activity fluctuations during rest, with a very similar peak-to-peak interval to that documented by Skipper et al (6sec). Given this endogenous pattern, one possibility is that the presence of meaningful stimuli serves to align the internal activity generators in the region. On this explanation, the premotor cortex is pre-disposed to react quickly to frequently changing stimuli (during rest) and this capacity is utilized also in the presence of meaningful stimuli to allow rapid reactions to those.

4. Conclusion

Our goal in this chapter was to outline an interpretive framework for addressing data obtained with fMRI naturalistic paradigms. We began the chapter by asking what naturalistic paradigms tell us about language, and we hope to have convinced the reader that while these paradigms hold a potential wealth of information for understanding how language processing occurs ‘in the wild’, this information is not readily apparent but can only be deciphered by considering a set of processes that either underlie language comprehension or accompany it.

We have attempted to show that interpretations of data collected within naturalistic paradigms will benefit from being informed by at least two separate domains of study. On the one hand, there is much theoretical research (supported by neurobiological findings) pushing the boundaries of language comprehension to consider more farther-afield issues such as (i) access to particular contents (e.g., emotional, temporal, action-based), (ii) the role of attention in language, (iii) and social-cognition elements such as empathy and theory of mind. On the other hand, various processes take place during language comprehension and may co-vary with it, thus making it more difficult to separate brain areas or activity patterns mediating linguistic information processing from areas mediating other co-occurring processes.

To alleviate interpretive difficulties and to ground neurobiological theories in a broader context of cognitive processing, future research could collect a large body of covariate information, including physiological measures, as well as resting state information from individual participants. This covariate information would allow modeling the changes to endogenous activity that are introduced by language processing per se. Information about systems mediating affiliated processes such as auditory attention, emotional responses, or recollection processes for the same participants could also help dissociate core linguistic processes from affiliated ones.

Bibliography

Allbritton, David W., & Gerrig, Richard J. (1991). Participatory responses in text understanding. Journal of Memory and Language, 30(5), 603-626. doi: http://dx.doi.org/10.1016/0749-596X(91)90028-I

Bahlmann, J., Schubotz, R. I., Mueller, J. L., Koester, D., & Friederici, A. D. (2009). Neural circuits of hierarchical visuo-spatial sequence processing. Brain Res, 1298, 161-170. doi: 10.1016/j.brainres.2009.08.017

Bechara, A., Damasio, H., & Damasio, A. R. (2000). Emotion, decision making and the orbitofrontal cortex. Cereb Cortex, 10(3), 295-307.

Binder, J. R., Desai, R. H., Graves, W. W., & Conant, L. L. (2009). Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cereb Cortex, 19(12), 2767-2796. doi: 10.1093/cercor/bhp055

Birn, R. M. (2012). The role of physiological noise in resting-state functional connectivity. Neuroimage, 62(2), 864-870. doi: 10.1016/j.neuroimage.2012.01.016

Bless, H., Clore, G. L., Schwarz, N., Golisano, V., Rabe, C., & Woelk, M. . (1996). Mood and the use of scripts: Does a happy mood really lead to mindlessness. J Pers Soc Psychol, 71, 665-679.

Brown, R. M., Chen, J. L., Hollinger, A., Penhune, V. B., Palmer, C., & Zatorre, R. J. (2013). Repetition suppression in auditory-motor regions to pitch and temporal structure in music. J Cogn Neurosci, 25(2), 313-328. doi: 10.1162/jocn_a_00322

Cabeza, R., Ciaramelli, E., Olson, I. R., & Moscovitch, M. (2008). The parietal cortex and episodic memory: an attentional account. Nat Rev Neurosci, 9(8), 613-625. doi: 10.1038/nrn2459

Chow, H. M., Mar, R. A., Xu, Y., Liu, S., Wagage, S., & Braun, A. R. (2013). Embodied Comprehension of Stories: Interactions between Language Regions and Modality-specific Neural Systems. J Cogn Neurosci. doi: 10.1162/jocn_a_00487

Conway, C. M., Pisoni, D. B., & Kronenberger, W. G. (2009). The Importance of Sound for Cognitive Sequencing Abilities: The Auditory Scaffolding Hypothesis. Curr Dir Psychol Sci, 18(5), 275-279. doi: 10.1111/j.1467-8721.2009.01651.x

Cooper, E. A., Hasson, U., & Small, S. L. (2011). Interpretation-mediated changes in neural activity during language comprehension. Neuroimage, 55(3), 1314-1323. doi: 10.1016/j.neuroimage.2011.01.003

Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nat Rev Neurosci, 3(3), 201-215. doi: 10.1038/nrn755

Cummings, A., Ceponiene, R., Koyama, A., Saygin, A. P., Townsend, J., & Dick, F. (2006). Auditory semantic networks for words and natural sounds. Brain Res, 1115(1), 92-107. doi: 10.1016/j.brainres.2006.07.050

Danker, J. F., & Anderson, J. R. (2010). The ghosts of brain states past: remembering reactivates the brain regions engaged during encoding. Psychol Bull, 136(1), 87-102. doi: 10.1037/a0017937

Davis, B., Jovicich, J., Iacovella, V., & Hasson, U. (2013). Functional and Developmental Significance of Amplitude Variance Asymmetry in the BOLD Resting-State Signal. Cereb Cortex. doi: 10.1093/cercor/bhs416

Deen, B., & McCarthy, G. (2010). Reading about the actions of others: biological motion imagery and action congruency influence brain activity. Neuropsychologia, 48(6), 1607-1615. doi: 10.1016/j.neuropsychologia.2010.01.028

Desai, R. H., Binder, J. R., Conant, L. L., & Seidenberg, M. S. (2010). Activation of sensory-motor areas in sentence comprehension. Cereb Cortex, 20(2), 468-478. doi: 10.1093/cercor/bhp115

Doucet, G., Naveau, M., Petit, L., Zago, L., Crivello, F., Jobard, G., . . . Joliot, M. (2012). Patterns of hemodynamic low-frequency oscillations in the brain are modulated by the nature of free thought during rest. Neuroimage, 59(4), 3194-3200. doi: 10.1016/j.neuroimage.2011.11.059

Egidi, G., & Caramazza, A. (2013). Cortical systems for local and global integration in discourse comprehension. Neuroimage, 71, 59-74. doi: 10.1016/j.neuroimage.2013.01.003

Egidi, G., & Gerrig, R. J. (2009). How valence affects language processing: Negativity bias and mood congruence in narrative comprehension. Mem Cognit, 37(5), 547-555. doi: 10.3758/MC.37.5.547

Egidi, G., & Gerrig, R. J. . (2006). Readers’ experiences of characters’ goals and actions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 1322-1329.

Egidi, G., & Nusbaum, H. C. (2012). Emotional language processing: how mood affects integration processes during discourse comprehension. Brain Lang, 122(3), 199-210. doi: 10.1016/j.bandl.2011.12.008

Ferstl, E. C., Rinck, M., & von Cramon, D. Y. (2005). Emotional and temporal aspects of situation model processing during text comprehension: an event-related fMRI study. J Cogn Neurosci, 17(5), 724-739. doi: 10.1162/0898929053747658

Forgas, J. P. (1998). Asking nicely? The effects of mood on responding to more or less polite requests. Personality & Social Psychology Bulletin, 24, 173-185.

Gerrards-Hesse, A., Spies, K., & Hesse, F. W. . (1994). Experimental inductions of emotional states and their effectiveness: A review. British Journal of Psychology, 85, 55-78.

Gerrig, Richard J. (1993). Experiencing narrative worlds : on the psychological activities of reading. New Haven: Yale University Press.

Gerrig, Richard J., & McKoon, Gail. (2001). Memory processes and experiential continuity. Psychol Sci, 12(1), 81-85. doi: 10.1111/1467-9280.00314

Gerrig, Richard J., & O'Brien, Edward J. . (2005). The Scope of Memory-Based Processing. Discourse Processes, 39(2-3), 225-242.

Green, M. C., & Brock, T. C. (2000). The role of transportation in the persuasiveness of public narratives. J Pers Soc Psychol, 79(5), 701-721.

Hampson, M., Peterson, B. S., Skudlarski, P., Gatenby, J. C., & Gore, J. C. (2002). Detection of functional connectivity using temporal correlations in MR images. Hum Brain Mapp, 15(4), 247-262.

Hasson, U., Furman, O., Clark, D., Dudai, Y., & Davachi, L. (2008). Enhanced intersubject correlations during movie viewing correlate with successful episodic encoding. Neuron, 57(3), 452-462. doi: 10.1016/j.neuron.2007.12.009

Hasson, U., Nusbaum, H. C., & Small, S. L. (2007). Brain networks subserving the extraction of sentence information and its encoding to memory. Cereb Cortex, 17(12), 2899-2913. doi: 10.1093/cercor/bhm016

Humphries, C., Willard, K., Buchsbaum, B., & Hickok, G. (2001). Role of anterior temporal cortex in auditory sentence comprehension: an fMRI study. Neuroreport, 12(8), 1749-1752.

Iacovella, V., & Hasson, U. (2011). The relationship between BOLD signal and autonomic nervous system functions: implications for processing of "physiological noise". Magn Reson Imaging, 29(10), 1338-1345. doi: 10.1016/j.mri.2011.03.006

Johnson, J. D., McDuff, S. G., Rugg, M. D., & Norman, K. A. (2009). Recollection, familiarity, and cortical reinstatement: a multivoxel pattern analysis. Neuron, 63(5), 697-708. doi: 10.1016/j.neuron.2009.08.011

Keen, S. (2007). Empathy and the novel. New York. NY: Oxford University Press.

Koyama, M. S., Kelly, C., Shehzad, Z., Penesetti, D., Castellanos, F. X., & Milham, M. P. (2010). Reading networks at rest. Cereb Cortex, 20(11), 2549-2559. doi: 10.1093/cercor/bhq005

Kristensen, L. B., Wang, L., Petersson, K. M., & Hagoort, P. (2012). The Interface Between Language and Attention: Prosodic Focus Marking Recruits a General Attention Network in Spoken Language Comprehension. Cereb Cortex. doi: 10.1093/cercor/bhs164

Lerner, Y., Honey, C. J., Silbert, L. J., & Hasson, U. (2011). Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J Neurosci, 31(8), 2906-2915. doi: 10.1523/JNEUROSCI.3684-10.2011

Lohmann, G., Hoehl, S., Brauer, J., Danielmeier, C., Bornkessel-Schlesewsky, I., Bahlmann, J., . . . Friederici, A. (2010). Setting the frame: the human brain activates a basic low-frequency network for language processing. Cereb Cortex, 20(6), 1286-1292. doi: 10.1093/cercor/bhp190

Mar, R. A. (2011). The neural bases of social cognition and story comprehension. Annu Rev Psychol, 62, 103-134. doi: 10.1146/annurev-psych-120709-145406

Mar, R. A., & Oatley, K. (2008). The function of fiction is the abstraction and simulation of social experience. Perspectives on Psychological Science, 3, 173-192.

Mason, M. F., Norton, M. I., Van Horn, J. D., Wegner, D. M., Grafton, S. T., & Macrae, C. N. (2007). Wandering minds: the default network and stimulus-independent thought. Science, 315(5810), 393-395. doi: 10.1126/science.1131295

Menenti, L., Petersson, K. M., Scheeringa, R., & Hagoort, P. (2009). When elephants fly: differential sensitivity of right and left inferior frontal gyri to discourse and world knowledge. J Cogn Neurosci, 21(12), 2358-2368. doi: 10.1162/jocn.2008.21163

Metz-Lutz, M. N., Bressan, Y., Heider, N., & Otzenberger, H. (2010). What Physiological Changes and Cerebral Traces Tell Us about Adhesion to Fiction During Theater-Watching? Front Hum Neurosci, 4. doi: 10.3389/fnhum.2010.00059

Molinari, E., Baraldi, P., Campanella, M., Duzzi, D., Nocetti, L., Pagnoni, G., & Porro, C. A. (2013). Human parietofrontal networks related to action observation detected at rest. Cereb Cortex, 23(1), 178-186. doi: 10.1093/cercor/bhr393

Petersson, K. M., Folia, V., & Hagoort, P. (2012). What artificial grammar learning reveals about the neurobiology of syntax. Brain Lang, 120(2), 83-95. doi: 10.1016/j.bandl.2010.08.003

Petkov, C. I., Kang, X., Alho, K., Bertrand, O., Yund, E. W., & Woods, D. L. (2004). Attentional modulation of human auditory cortex. Nat Neurosci, 7(6), 658-663. doi: 10.1038/nn1256

Phillips, J. S., Velanova, K., Wolk, D. A., & Wheeler, M. E. (2009). Left posterior parietal cortex participates in both task preparation and episodic retrieval. Neuroimage, 46, 1209-1221.

Pollatos, O., Schandry, R., Auer, D. P., & Kaufmann, C. (2007). Brain structures mediating cardiovascular arousal and interoceptive awareness. Brain Res, 1141, 178-187. doi: 10.1016/j.brainres.2007.01.026

Preminger, S., Harmelech, T., & Malach, R. (2011). Stimulus-free thoughts induce differential activation in the human default network. Neuroimage, 54(2), 1692-1702. doi: 10.1016/j.neuroimage.2010.08.036

Ramot, M., Wilf, M., Goldberg, H., Weiss, T., Deouell, L. Y., & Malach, R. (2011). Coupling between spontaneous (resting state) fMRI fluctuations and human oculo-motor activity. Neuroimage, 58(1), 213-225. doi: 10.1016/j.neuroimage.2011.06.015

Rapp, D. N., & Gerrig, R. J. (2006). Predilections for narrative outcomes: The impact of story contexts and reader preferences. Journal of Memory and Language, 54(54-67).

Rogalsky, C., Rong, F., Saberi, K., & Hickok, G. (2011). Functional anatomy of language and music perception: temporal and structural factors investigated using functional magnetic resonance imaging. J Neurosci, 31(10), 3843-3852. doi: 10.1523/JNEUROSCI.4515-10.2011

Saygin, A. P., Dick, F., Wilson, S. M., Dronkers, N. F., & Bates, E. (2003). Neural resources for processing language and environmental sounds: evidence from aphasia. Brain, 126(Pt 4), 928-945.

Schubotz, R. I. (2007). Prediction of external events with our motor system: towards a new framework. Trend in Cognitive Sciences, 11, 211-218.

Seger, C. A., Spiering, B. J., Sares, A. G., Quraini, S. I., Alpeter, C., David, J., & Thaut, M. H. (2013). Corticostriatal Contributions to Musical Expectancy Perception. J Cogn Neurosci. doi: 10.1162/jocn_a_00371

Simmons, W. K., & Martin, A. (2012). Spontaneous resting-state BOLD fluctuations reveal persistent domain-specific neural networks. Soc Cogn Affect Neurosci, 7(4), 467-475. doi: 10.1093/scan/nsr018

Sitnikova, T., Holcomb, P. J., Kiyonaga, K. A., & Kuperberg, G. R. (2008). Two neurocognitive mechanisms of semantic integration during the comprehension of visual real-world events. J Cogn Neurosci, 20(11), 2037-2057. doi: 10.1162/jocn.2008.20143

Skipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., & Small, S. L. (2009). Gestures orchestrate brain networks for language understanding. Curr Biol, 19(8), 661-667. doi: 10.1016/j.cub.2009.02.051

Speer, N. K., Zacks, J. M., & Reynolds, J. R. (2007). Human brain activity time-locked to narrative event boundaries. Psychol Sci, 18(5), 449-455. doi: 10.1111/j.1467-9280.2007.01920.x

Strange, B. A., Duggins, A., Penny, W., Dolan, R. J., & Friston, K. J. (2005). Information theory, novelty and hippocampal responses: unpredicted or unpredictable? Neural Networks, 18(3), 225-230.

Tobia, M. J., Iacovella, V., Davis, B., & Hasson, U. (2012). Neural systems mediating recognition of changes in statistical regularities. Neuroimage, 63(3), 1730-1742. doi: 10.1016/j.neuroimage.2012.08.017

Tobia, M. J., Iacovella, V., & Hasson, U. (2012). Multiple sensitivity profiles to diversity and transition structure in non-stationary input. Neuroimage, 60(2), 991-1005. doi: 10.1016/j.neuroimage.2012.01.041

Tremblay, P., Baroni, M., & Hasson, U. (2012). Processing of speech and non-speech sounds in the supratemporal plane: Auditory input preference does not predict sensitivity to statistical structure. Neuroimage, 66C, 318-332. doi: 10.1016/j.neuroimage.2012.10.055

Turk-Browne, N. B., Scholl, B. J., Johnson, M. K., & Chun, M. M. (2010). Implicit perceptual anticipation triggered by statistical learning. J Neurosci, 30(33), 11177-11187. doi: 10.1523/JNEUROSCI.0858-10.2010

van den Broek, P., Rapp, D. N., & Kendeou, P. (2005). Integrating Memory-Based and Constructionist Processes in Accounts of Reading Comprehension. Discourse Processes, 39(299-316).

Wager, T. D., & Smith, E. E. (2003). Neuroimaging studies of working memory: a meta-analysis. Cogn Affect Behav Neurosci, 3(4), 255-274.

Wallentin, M., Nielsen, A. H., Vuust, P., Dohn, A., Roepstorff, A., & Lund, T. E. (2011a). Amygdala and heart rate variability responses from listening to emotionally intense parts of a story. Neuroimage, 58(3), 963-973. doi: 10.1016/j.neuroimage.2011.06.077

Wallentin, M., Nielsen, A. H., Vuust, P., Dohn, A., Roepstorff, A., & Lund, T. E. (2011b). BOLD response to motion verbs in left posterior middle temporal gyrus during story comprehension. Brain Lang, 119(3), 221-225. doi: 10.1016/j.bandl.2011.04.006

Westermann, R., Spies, K., Stahl, G., & Hesse, F. W. . (1996). Relative effectiveness and validity of mood induction procedures: A meta-analysis. European Journal of Social Psychology(26), 557-580.

Whitney, C., Huber, W., Klann, J., Weis, S., Krach, S., & Kircher, T. (2009). Neural correlates of narrative shifts during auditory story comprehension. Neuroimage, 47(1), 360-366. doi: 10.1016/j.neuroimage.2009.04.037

Wilson, S. M., Molnar-Szakacs, I., & Iacoboni, M. (2008). Beyond superior temporal cortex: intersubject correlations in narrative speech comprehension. Cereb Cortex, 18(1), 230-242. doi: 10.1093/cercor/bhm049

Xu, J., Gannon, P. J., Emmorey, K., Smith, J. F., & Braun, A. R. (2009). Symbolic gestures and spoken language are processed by a common neural system. Proc Natl Acad Sci U S A, 106(49), 20664-20669. doi: 10.1073/pnas.0909197106

Xu, J., Kemeny, S., Park, G., Frattali, C., & Braun, A. (2005). Language in context: emergent features of word, sentence, and narrative comprehension. Neuroimage, 25(3), 1002-1015. doi: 10.1016/j.neuroimage.2004.12.013

Zacks, J. M., Braver, T. S., Sheridan, M. A., Donaldson, D. I., Snyder, A. Z., Ollinger, J. M., . . . Raichle, M. E. (2001). Human brain activity time-locked to perceptual event boundaries. Nat Neurosci, 4(6), 651-655. doi: 10.1038/88486

Zacks, J. M., & Swallow, K. M. (2007). Event Segmentation. Curr Dir Psychol Sci, 16(2), 80-84. doi: 10.1111/j.1467-8721.2007.00480.x