My Literature Review (rough draft)

Greetings all! So I have part of my literature review written as a rough draft, but before I submit it here I would like to review some challenges that I faced in writing it and describe how I spent my time. Last week I spent a few hours combing over the videos of our test subjects being taught by our prototype virtual tutors, and took notes on behaviors that were commonly appearing. Part of this method can be though of as grounded theory. Personally, I’m using an ethnographic approach to reviewing this data, called interpretive description. This method is a relatively new method in qualitative data analysis, that allows one to extract (interpret) the big concepts from various descriptions (in my case videos), to find the most important questions to ask about the data.

My only challenge so far has been to try to structure my literature review in a way that will generate questions. It is far easier to read many articles and write a summary than to think of a gap in the field, in my opinion. So without further ado, here is my first pass at the literature review, using my outline as a guide.

Comfort with virtual agents can depend on factors like eye gaze of the virtual agent and ability of the virtual agent to respond to social cues. One notable social cue that is important when one is trying to form a bond, whether with human or virtual agent, is the idea of self disclosure. Self disclosure is the sharing of personal information about oneself. It could be something as simple as “I’m not good at math” (cite Madaio annotation manual here) or as profound as “ I never liked math in childhood and felt pressured by my parents to do it.” When people are comfortable, they disclose information about themselves and while they do that, there’s always a question appearing in their minds of whether it is worth self disclosing to the partner (Farber 2006). Some people tend to self disclose less, and this is may be due to shyness. Although historically, shyness is a vague term, lately the literature has proposed it be conceptualized in three components: affective, cognitive, and social. Shyness is believed to be a trait of personality that causes behaviors somewhere between neuroticism and extrovertedness. Those may sound like very opposite terms, but somewhere in between is shyness (Crozier 2005). In fact, the cognitive aspect of shyness can be considered similar to anxiety, in that one part of shyness is the fear of socializing and meeting new people (this symptoms is present in cases that are not clinically severe, just everyday shy people) (Crozier 2005). Shyness is important to discuss in terms of self disclosure because it can inhibit self disclosure because anxiety reduces self disclosure (Madaio 2016), and it can be due to a feeling of inferiority or lack of comfort. In order to make virtual agents that humans are comfortable dealing with, we must consider shyness and self disclosure as important factors. It remains a question in the field as to whether human shyness (measured by the shyness scale of Cheek and Buss) (Crozier 2005) can be minimized by virtual agents and what strategies are useful for this.

Self disclosure is increased in patients who are feeling anxious by virtual agents disclosing human back stories (“I was born in Italy, my parents never loved me”), as opposed to machine back stories (“My parts are made from Dell and my mother is a CPU”) (Kang 2012). Kang et al studied the effects of human-story telling virtual agents on self disclosure on psychotherapy sessions but they did not consider how human story-telling can improve self disclosure in regular, non-patient people, like kids studying math in middle school. Also, Yu (2013) demonstrated that depressed patients (similar to severe shyness that extends into neuroticism ,where people are not just unmotivated to socialize, but afraid to socialize) have a tendency to begin talking very early on after a rapport building question is posed, particualrly a self disclosure question. In other words, patients who are clinically shy tend to want self disclosure. Now we have to extend this into the dimension of regular shy people, who are not patients in psychotherapy. As stated before, regular shy people feel the same elements of shyness in clinically depressed shy people, but to a lesser degree, so efforts should be made to see how the theories used to help clinically shy people can carry over to regular shy people.


Yu, Z., Scherer, S., Devault, D., Gratch, J., Stratou, G., Morency, L-P., Cassell, J. (2013). Multimodal Prediction of Psychological Disorders: Learning Verbal and Nonverbal Commonalities in Adjacency Pairs. Presented at SenDial.

Subjects. N = 130 dyads in semi-structured interviews. Each subject was paired with 1 virtual human interviewer.

Results. Depressed people speak for less time responding to questions to the rapport building phase of the session (e.g. Do you travel a lot?) (last page, 2 paragraphs above conclusion). Speech onset time and speech rate correlate negatively with depression.

Methods. The idea is to use the HCRF (hidden conditional random fields), which is a mathematical/computational method that learns commonalities (called hidden states) between variables, makes these “hidden states” into categories by which variables are grouped, and considers the time dimension — important to this study because it parses behavior with respect to each question posed over time. This study uses the HCRF together with multimodal data (verbal and nonverbal, i.e. acoustics, visual data) to categorize verbal and nonverbal behaviors into categories of psychological disorders. Previous studies did not consider as many modes of data, but have shown correlations between certain modes, e.g. speech rate, and psychological disorders, e.g. depression. Importantly, this study makes categories at the adjacency pair (question-response pair) level, whereas previous pairs used a whole-session level that was too general.

Ozyesil, Z. (2012). The Prediction Level of Self-Esteem on Humor Style and Positive-Negative Affect. Psychology, Vol. 3 (8), 638-641.

Subjects. N = 440 undergraduate students (77% female), chosen by simple random sampling. Mean age = 20.33, with std dev = 1.73.

Results. Most interesting is that self esteem is negatively correlated (p < 0.001) with self defeating humor and negative affect. Moreover, self-esteem explained 4.2% of the total variance in negative affect. Self esteem is a personal judgement of worthiness, and is a personality trait. Humor is also a personality trait, and affiliative humor is believed to increase interpersonal bonds (rapport), and thus is used by people with high self esteem; the presence of one predicts the other reliably (high positive correlation). Low self esteem is believed to lead to anxiety and depression, so these people tend to have negative affect. Higher levels of self defeating humor thus predict lower levels of self esteem and higher levels of depression. If these people develop higher self esteem, it improves their positive affect, and thus improves their interpersonal relationships.

Methods. The study used the Rosenberg self esteem scale (questionnaire that rates participants on a 10 item scale, like a Likert scale), the Humor Styles Questionnaire, and the Positive and Negative Affects Schedule. The study tried to demonstrate humor style can be predicted by self esteem. Self esteem is a personality trait, not just a simple positive evaluation of the self; personality traits are dynamic characteristic patterns of thoughts, emotion, and actions that make a person similar or different from others. Humor comes in four types for the purposes of this study: self enhancing, affiliative, aggressive, and self defeating. The first type helps reduce conflict while protecting the self, and is a coping strategy. The second promotes interpersonal bonding, the third is used as a way to criticize others, and the last occurs when people allows themselves to be the butt of jokes for the sake of forming interpersonal bonds, but they are in fact hindering positive feelings about themselves.

Crozier, W. Ray. “Measuring Shyness: Analysis of the Revised Cheek and Buss Shyness Scale.” Personality and Individual Differences, vol. 38, no. 8, 2005, pp. 1947–1956., doi:10.1016/j.paid.2004.12.002.

Farber, Barry A. Self-Disclosure in Psychotherapy. Guilford Press, 2006.

Kang, Sin-Hwa, et al. “Understanding the Nonverbal Behavior of Socially Anxious People during Intimate Self-Disclosure.” Intelligent Virtual Agents Lecture Notes in Computer Science, 2012, pp. 212–217., doi:10.1007/978-3-642-33197-8_22.

Kang et al. (2008). “Agreeable People Like Agreeable Virtual Tutors”.

Outlining My Research

I had one main accomplishment last week. This past week I produced a rough draft of an outline, which will serve to guide me as I write my literature review.

One big challenge I faced in compiling this outline was in finding enough information on anxiety or vulnerability, my latest direction of research. This topic is mainly covered by articles dealing with clinically severe (and innate trait) anxiety, not the state-based anxiety that occurs naturally during human-human interaction. Thus, I did a lot of digging around. The interaction of anxiety and vulnerability with rapport, namely that more anxiety contributes to forming lower rapport, led me to believe that self disclosure was important, because anxiety reduces the amount of self disclosure being reciprocated by a tutor-tutee dyad.

Self disclosure is an important part of the rapport building model. Specifically, the model for rapport building that I find most useful is the model presented by Zhao et al (2014), reproduced below (Figure 2).



Most models of rapport building strategies include self disclosure, coordination, and mutual attentiveness, whether the model was created for tutor-tutee relationships, business-customer relationships, or boss-employee relationships. I have looked previously at economics research in rapport building and found many components of the Zhao et al model. The reason certain elements of the model are ubiquitous is because, as it seems, all human rapport building boils down to being able to sharing personal information about oneself in order to create a salient boundary between the outside world and your group of friends, which leads to better support and understanding for each other (note Fig. 2 mentions supporting the “true-self”). Going forward, any time I refer to a rapport model, or strategies for building rapport, I am always referring to the Zhao et al model above.

Last week I spent 5 hours watching an re-watching all the video data available to me. This data was collected for a prototype virtual algebra tutor (which mine and my colleagues’ research will improve), which spoke to a middle school student to offer suggestions on how to solve an algebra problem. I had to watch these videos because I had to make sure that the topics I’m looking for in my review of the literature (namely, anxiety and rapport and self disclosure), are going to be found in the data. If this is not the case, then I’m just wasting time learning about topics that I will never be able to find evidence for in our data corpus. Interestingly, some students indeed seemed more anxious than others, and levels of self disclosure varied greatly, as did apparent extroversion and face threatening behavior (perhaps due to personality differences). Of course, a formal statistical test is required to see if these are significant interactions, but at least now I know my research is not completely detached from the reality of my data.

The other 7 hours were spent meeting with my CREU team-mate and research mentor (2 hours) and doing research (5 hours). Note, research was not a major component of the time I worked last week because that step is almost complete.

I have attached my outline to this post, but note it is rough, and certain sections need to be taken out to make the scope of the outline less broad and more focused.



  1. Persona Effect
    1. People universally agree computers do not have human understanding of emotion, yet some people treat them nicely…”This rejection of anthropomorphism stands in stark contrast to people’s actual behavior in our labs and in more naturalistic settings. In this article, we will argue that there is clear evidence that individuals mindlessly (Langer, 1989) apply social rules and expectations to computers.”
    2. Mindlessness is defined by Nass and Moon (2000) to be the conscious attention giving to a subset of contexts, so that you are responding to overly simplistic scripts that ignore details, like when a human ignores that a virtual agent that is smiling is not really there, but is just pixels., and human smiles back. Many computer-human social interactions are due to a degree of mindlessness, and many source have suggested there is som subconscious understanding of virtual agent as social actor, such as when humans reported feeling more connected with agents that smiled more but they were not aware than agents were programmed to smile more or less. Here is the theory of mindless behavior:: “. We can conclude that individuals are responding mindlessly to computers to the extent that they apply social scripts—scripts for human-human interaction—that are inappropriate for human-computer interaction, essentially ignoring the cues that reveal the essential asocial nature of a computer” (Nass and Moon, 2000).
  2. Self efficacy
    1. Self efficacy effects: Self-efficacy beliefs influence academic choices as students are prone to engage in tasks in which they feel confident and avoid those in which they do not.
    2. Self efficacy is the belief in one’s ability to do something successfully, and it is a better predictor for academic success than prior academic success and academic interest.
    3. But there are some limits to the use of self efficacy as a predictor for academic success: “The effects of efficacy beliefs on achievement are usually stronger for high school and college students than for elementary students”
  3. Self disclosure
    1. Al this ties into self disclosure. Self disclosure causes a risk beause one could reveal a secret that causes rejection by peers (Kelly 1996). Kelly also says that the worth of revealing secrets is a complex calculation that is successful based on whether the secret keeper will act friendly after having benn told the secret (Kelly 1996). So, when a child reveals something to a virtual agent, he is calculating how worthy a vvirtual agent is of receiving the secret.
    2. Kang and Gratch (2012) said that computers with human back stories lead people to reveal more about themselves than using computer back stories, which makes sense. Anxious people seem to self disclose more.
    3. Zhao et al (2014) claim self dislosure is used to build coordination and other researchers claim slef disclosure helps the friends differentiate themselves from the outside universe (eg. This is you and me, we are different from them), which is a way to establish rapport.
    4. Is self disclosure part of the larger idea of reciprocation?
    5. How to define “comfortablness” with a virtual agent

Works Cited

Nass and Moon (2000). Machines and Mindlessness. Journal of Social Issues, Vol. 56, No. 1, 2000, pp. 81–103

Kelly, A. E., & McKillop, K. J. (1996). Consequences of revealing personal secrets. Psychological Bulletin, 120(3), 450-465.

Kang, Gratch (2012). Socially anxious people reveal more personal information with virtual counselors that talk about themselves using intimate human back stories.

Kang et al. (2008). “Agreeable People Like Agreeable Virtual Tutors”.

This source determines specific components of rapport (either coordination, attentiveness, or positivity) that are benefit disproportionately from specific personality types (Big Five personality types).


Erik E. Noftle and Phillip R. Shaver (2006). “Attachment Dimensions & Personality Traits: Associations and Comparative Ability to Predict Relationship Quality.” Journal of Research in Personality, (40)  179-208.

This source is useful because it shows that there IS a correlation between certain personality types and certain relationship types. For example, attachment avoidance is highly positively correlated with neuroticism (the tendency to be cold or depressed or anxious). This shows that in human-human relationships, personality matters.


Ogan et al (2011). “Rudeness and Rapport: Insults and Learning Gains in Peer Tutoring.” page 1.

This study describes how positivity decreases as strangers become friends. Ogan shows that insults are correlated with lower learning gains in stranger dyads but higher learning gains in friendly dyads. Also, some examples of insults are provided. It is shown that in general, students are more playful with language, whereas tutors are more focused on the task at hand

Cutrone (2011) “Face theory, japanese culture”.

This source claims the concept of face DOES exist in japanese culture, and probably across all cultures, even though previous sociologists said the concept of face is alien to japanese culture.

Johnson, L., Paola, R. “Politeness in Dialog: ‘Run the factory, that’s what I’d do.'” Intelligent Tutoring Systems, 206 – 243. 2004.

This source describes very concisely and neatly the “persona effect”. This effect somewhat answers the question “how believable should we make the tutoring system?” because the answer is that the system need not be believable, but it MUST be able to account for the persona effect, and to do that it must respond to social cues in a human-like way. This means humans are comforted by the system, even if they know it’s not a real sentient being.

Furthermore, this  source has a good definition of the politeness theory of Brown and Levinson, which includes the definition of “Face”. Their theory said that every culture has a conecpt of face, and positive face is the desire to be approved by others, whereas negative face is the desire to do what one wants regardless of others’ opinions. When tutors threaten negative face (by being forcefull or giving orders) or positive face (by saying the tutee is wrong too harshly), the learning gains suffer.

Park, A., Ickes, W., & Robinson, R. L. (2014). More f#!%ing rudeness: Reliable personality predictors of verbal rudeness and other ugly confrontational behaviors. Journal of Aggression, Conflict and Peace Research, 6(1), 26-43. Retrieved from

The source above is surprisingly NOT about the role of laughter specifically, but rather how in a story telling dyadic paradigm, the listener’s prosody, nonverbal behavior, and affective stance affect the telling of the story by the teller.

Park, A., Ickes, W., & Robinson, R. L. (2014). More f#!%ing rudeness: Reliable personality predictors of verbal rudeness and other ugly confrontational behaviors. Journal of Aggression, Conflict and Peace Research, 6(1), 26-43. Retrieved from

The paper above tries to demonstrate certain personalities are predisposed to rudeness and ugly confrontational behavior in general. This supports my hypothesis that certain personalities can be predisposed to confront a virtual peer tutor.

Mendoza-Denton, Norma (1999). Turn initial “no”: collaborative opposition among Latina adolescents. Source is a chapter in a larger book, “Reinventing Identities: The Gendered Self In discourse” (1999). Eds, Mary Bucholtz, A. C. Liang, Laurel A. Sutton. Oxford University Press. ISBN 3 1735 040 492 310. (From Hillman Library, location P-120, S48, R47, 1999).

This source states that stance is composed of both verbal and nonverbal cues. For example, gesture, intonation, and physical action are all deployed in concert to embody stance in girls’ conflictive evaluation of one another’s turns in hopscotch. Subjects were all Latinas from Mexico in an American high school. This paper is a response to other theories that polarize (and generalize) all women’s behavior as cooperative and all men’s behavior as conflictive. This paper aims to demonstrate conflict AND cooperation working in women’s interactions. This debunks myths like Latina submissiveness.  To the end of shedding light on women’s interactions, this paper describes turn initial “no” as a “no” that is a discourse marker. In linguistics, discourse markers signal cataphoric or anaphoric relations between units of talk, “bracketing” them. The point is turn initial “no” is not just the negation word “no”, but rather has polysemy (many meanings).

Che et al (2012). Problem Solving Strategies of Girls and Boys in Problem Solving Strategies.

More than double as many girls as boys wrote down an answer using a procedure that was memorized or poorly explained. However, both boys and girls were mature in their problem solving thinking.

Zhao et al (2014). Towards a Dyadic Copmutational Model of Rapport Management.

This source outlines a model of rapport building. It is useful because this source is from the Articulab, where all the rest of my research is taking place, so I will be using this model going forward. The original purpose of the paper was to propose a computer architecture that can support a computer program that is able to check the rapport state, and adjust its outputs to this rapport state (using the FIgure 2 model aforementioned). I believe this model was implemented in the prototype tutor experiment that I watched videos of.

How does anxiety interact with rapport?

This question will be the subject of my next week of research. The past week involved reading some more articles, but with a different purpose than when this project started. Briefly stated, my mentor and I have decided to choose one single avenue of research from the past week: the interaction of anxiety/vulnerability and rapport. We discussed how to structure this to write a literature review.

There are four challenges to extracting information from the literature on anxiety and rapport:

  • There is a tendency for me to read too much from a single author; in my case, I tend to read too many articles from Jonathan Gratch, from the University of Southern California’s Creative Technology Institute, who focuses on the interaction of virtual agents with real humans. To solve this, I am trying to read more from authors like Eric Noftle and Phillips Shaver, who have done work in the psychology of personality.
  • Some papers (e.g. “Agreeable People Like Agreeable Virtual Humans” by Jonathan Gratch) conflate the definition of anxiety and vulnerability but these two have different meanings. For example, in a team of children who are tasked with developing a law of physics for seesaws after observing one in real life, the child who has been taught more about physics may be more anxious to speak (so as to not seem overbearing) but also feel more confident in what he is saying. Whereas, the child who is less confident is less anxious to speak. Such behavior was observed in “Processes and Consequences of Peer Collaboration: A Vygotskian Analysis” (Tudge, 1992).
  • Anxiety is often referred to by Gratch as neuroticism, as defined in the Big Five Personality Traits. These traits however, are innate characteristics, and my mentor and I agreed that we would like to refrain from assuming that an innate characteristic will remain stable throughout a given context or period of time. For example, if Susy is anxious and there’s a snake in the room, she might have trouble building rapport, but my mentor and I would argue that the context (snake) is hindering Susy’s naturally friendly disposition. Also, Susy may have had a bad day. The point is that we would like to create an agent that is adaptive to different emotional states a tutee is in, rather than classify people as always anxious or always extroverted.
  • Many papers focus on observable behavior, e.g. a subject is smiling more frequently when a virtual agent smiles more frequently. While this is interesting, I am realizing it is more productive for me to take a more abstract view of the behavior and by increasing the level of abstraction, find implications in the psychology of human–virtual agent interaction.

The last challenge listed above brings me to my next point in this blogpost, which is this week’s meeting with my mentor. He revealed to me a hierarchy in research topics, reproduced here from the highest (1) to lowest level (3).

  1. Underlying psychological/social state
  2. Latent (not directly observable) social functions
  3. Observable behaviors

This is a useful way to organize my research, which I have briefly done below.

  1. This is rapport, e.g. whether it is being destroyed or built up. Also, Soo Youn Oh (2016) describes surveys he uses to measure positive affect and psychological presence, both of which are psychosocial states.
  2. The best examples of this are Gratch (2008) and Zhao (2014). Both articles break down rapport into three common components: mutual attentiveness, coordination, and positivity. Coordination occurs when a dyad is able to feel in-sync, and understand each other’s expectations from the interaction. Positivity, defined slightly differently in both articles, is roughly the act of partners being encouraging towards each other and warm. All these social functions are impossible to directly measure, because no one really knows whether two people really feel in sync, for example, so we use observable behaviors as landmarks.
  3. Observable behaviors are easy to quantify, such as Youn Oh (2016), who measured the difference between a slight smile and a very pronounced smile. Another example is measuring the number of utterances of each partner in a dyad to find who is dominating the interaction.

The hierarchy helps me recognize what to focus on when reading articles, namely the wider implications of their findings rather than specific effects like increased number of smiles. This also helps me find topics that will carry the greatest impact in my upcoming literature review article. When I write it, I will find questions about broad implications that will be more interesting and beneficial for the field than simply comparing observable behavior results from other experiments. Without further ado, here is a summary of my research:

  • The effect of personality on rapport first came into my mind when reading Noftle and Shaver (2005). They describe in a review article format how adults with a more reclusive personality tend to be afraid of attachment, a condition called attachment anxiety. Then, a connection between virtual agents and human anxiety appeared in Johnson (2004), which describes the Persona Effect. The Effect occurs when humans treat socially engaging virtual agents as social actors, meaning we feel the need to be polite or reciprocate kindness to virtual agents. Importantly, we interact with virtual agents in a way similar towards other humans, so perhaps anxiety with humans carries on to virtual agent interactions. Then it gets complicated. We can’t label a subject as anxious and then expect them to be anxious every moment they interact with our agent.; human behavior fluctuates based on context and time. So what we’re really looking for, instead of linking innate anxiety with dyad coordination or mutual attentiveness, is how anxiety is produced by a specific event in the interaction and how to manage it at that specific time. Gratch (2008) states that he expected the more anxious subjects to benefit more from the interaction with his virtual agent, but he found no correlation. Thus, he suggests modifying his definitions of positivity, and I would suggest separating the definitions of anxiety and vulnerability. This brings me to my next article, Gratch (2012) who demonstrated that people who were feeling socially anxious, as self-reported, at the beginning of a session with a virtual counselor also had higher self-reported positive affect and feelings of rapport (than did people who were no as anxious to being with). This supports the idea that there is some kind of highly significant interaction between state-based anxiety and rapport, but Gratch (2012) and Gratch (2008) contradict each other, in that Gratch (2008) did not find such a correlation. Therefore, my idea is that Gratch needs a more nuanced definition of anxiety, breaking it down into true anxiety and separating that from vulnerability.

Including the meeting with my mentor, I have spent 14 hours researching two different path for research, and upon meeting with my mentor I decided the better path was the anxiety/rapport research.

Incidentally, another avenue of research that will be beneficial for someone else in the Articulab to pursue would be subconscious mimicry. Research shows that humans mimic observable behaviors when building rapport, but are not entirely aware of it. One article demonstrated how humans felt like they were bonding closer to a virtual agent that smiled slightly wider than another. The humans did not notice the difference in smiles, but the self-reported measures of positive affect and presence demonstrated that higher rapport correlated with wider smiles from the virtual agent. Thus, humans were subconsciously processing behavioral observations and using that to inform their internal opinion of the rapport level. Furthermore, similar research showed a similar effect with head nods by a virtual agent that were designed to occur a little while after the human (telling a story) nodded their own head. Once again, humans did not report anything about head nods, but did report higher feelings of rapport. It would be interesting to find out more about subconscious social processing.

Making connections

This past week I spent most of my time reading again about information that is required to answer the questions 1 through 3 of the previous week’s blog entry.  I dropped question 2 because it seemed irrelevant, after meeting with my research mentor. THe most important highlight of this past week was the meeting with my mentor, where the research I had done culminated into a lively exchange of ideas and an interesting discussion on the difference between human-virtual agent interaction and human-human interaction.

The meeting with my research mentor took two hours. The rest of my 13 hours this past week went to reading various articles. I had focused on finding more information on two broad areas, that summarize questions 1 through 3 very well.

  1. What is engagement in tutoring? How can we measure engagement? This is asked because engagement is a very vague topic, and how one defines it, whether by the number of utterances a subject makes or the number of problem -focused statements he makes, can be tricky.
  2. What is different about human-human and human-agent rapport? The Articulab has done a lot of research on human-human rapport in order to build a socially sensitive virtual agent, but more research needs to be done on how humans actually perceive agents.
    1. Some sub-questions to this topic include: If a human knows the agent is not really human, will they be less motivated to finish the task? Is it proper for a virtual agent to seek eye contact with a  person or is this too imposing? Surprisingly, research has shown that the more photorealistic an agent is, the more eye contact is expected by the human it interacts with. On the other hand, an agent that is not at al photorealistic comes across as threatening or artificial when it seeks eye contact.

My extensive research found many articles looking at various kinds of behavior differences in humans when they are faced with other humans versus virtual agents. These articles asked questions like, “Do humans desire eye contact with the virtual agent? Does a head nod at the wrong time decrease the rapport a human builds with an agent? If the agent mimics head movements and posture from the human, will the agent establish higher rapport?” The answers, for those who are curious, are it depends, yes, and yes.

In order to find a proper operationalization for “engagement”, I looked at the theory of working alliances for teaching. The theory states that patients that form a bond with their therapist (it was originally made for therapist-patient relations and later adapted to education) , are very reliably able to achieve better clinical outcomes than patients who don’t bond. In education, this means the student must feel that himself and the teacher are working towards a common goal the student has, as opposed to a unidirectional boss-servant kind of relationship. Anyway, the importance of this was mainly to measure engagement, but it seemed too abstract. Other researchers, like Jonathan Gratch at the University of Southern California, believe that engagement can be measured by amount of speech. In his paper “Agreeable people like agreeable virtual tutors” (2008), John Gratch uses a storytelling paradigm where a human tells an agent a story (but thinks the agent is actually an avatar controlled by a human).  For Gratch, more engagement is observable via longer stories, which indicate the human wants to continue speaking to the virtual agent. However, this still seems too simple a measure, because conceivably a child could utter several sentences to his teacher but not be engaged by the material he is learning. Such is the difficulty of operationalizing engagement.

My greatest challenge this week was to tie the ideas I found together. Though in this blog I briefly mention the ideas from articles about body posture mimicry and head nods, there are many articles I read on such topics and each had its own goal. Some articles pointed out contradictions in others, which my mentor said is common in the scientific world, because articles re not meant to be flawless statements of fact, but rather educated guesses at the phenomenon that is occurring. Anyway, this challenge was overcome by meeting with my mentor and mapping out all the ideas and discarding ideas that deviate from the central mission of the CREU project One topic that was discarded was whether quality of communication is increased by agents who avoid gazing directly at humans. This was discarded because “quality of communication” was not clearly operationalized in the article. It was not thoroughly explained enough. Another discarded topic came from an article that concluded that middle school students speak in patterns that alternate between playful social talk and focused on-task talk, indicating their tendency to be building rapport even when trying to focus on something that’s socially irrelevant. While such a topic is interesting, the question arises as to what is important from this conclusion. Interesting conclusions might be not lead to new avenues of research.

Standing on the Shoulders of Giants

Sir Godfrey Kneller, Sir Isaac Newton, 1689justine-cassell

To understand this blog, it’s best to begin with an explanation of the work of Dr. Justine Cassell, principal investigator of the ArticuLab at Carnegie Mellon University. She has made significant progress in the field of embodied conversational agents, which are computer programs that are displayed with a body image and which try to imitate human behavior in some way (verbally or nonverbally or both) in order to help humans; for example, by teaching them math or being their personal assistant (like Apple’s Siri). The ArticuLab, where I do all research posted on this blog, studies human interaction in sociocultural contexts and designs computational systems that are sensitive to these contexts. The RAPT project (Rapport Aligning Peer Tutor) in the ArticuLab is headed by Michael Madaio. RAPT is researching optimal strategies for teaching math to middle school aged children. Recent research in learning sciences has shown that students learn better when they form a friendly bond with a tutor, and also that virtual tutors can improve student performance by up to two standard deviations. RAPT capitalizes on these two important discoveries, by building a virtual tutor that can respond in real-time to human friendship-building behavior (also called rapport). Hopefully some day, such a virtual tutor will make human learning not only more efficient, but fun and engaging.

This is where my personal work at the ArticuLab begins (summary can be found at the end of this post). By standing on the shoulders of giants, we can make important discoveries without reinventing the wheel. To that effect, my first week began with a literature review. My goal for this first week is to use the literature review to find research questions for future investigation in one of three categories (as described in the project proposal submitted to CREU):

A. How does a student’s learner profile (e.g. degree of extroversion or conscientiousness)      and the design of a virtual tutor (e.g. its ability to manage face) affect the building of          rapport between virtual tutor and tutee?

B. How does rapport impact learning (behaviors and outcomes) differently between                human-human interactions and human-virtual agent interactions?

C. Does rapport that is built during the tutoring process cause greater student                          engagement in the tutoring process and how can we measure that engagement (e.g. by      counting explanations or utterances)?

In approximately 9 hours, I explored 11 articles. I began by first reading articles authored by ArticuLab members to find out what the ArticuLab’s response to categories A through C are. I specifically sought articles loosely covering the subject of categories A through C and discarded the other articles. Next, I read the articles cited in the aforementioned articles. Finally, I picked the following broad questions for research, out of the countless fascinating avenues of research in this burgeoning field:

  1. Are certain personalities more likely to insult or disengage from virtual agents?
  2. Are there gender differences in problem solving behavior and to what extent are these important?
  3. How does anxiety predetermine what rapport building strategies a student uses (which a virtual tutor must be sensitive to)?

Below is a review of sources that led me to these questions (1-3), with numbers corresponding to the questions (1-3) above. The complete citations for the works mentioned below can be found in the bibliography at the end of this post.

  1. Question 1 fits into Category A because it seeks a connection between innate characteristics of the student (e.g. personality traits) and rapport-destructive behaviors (e.g. insults, abuse). Noftle and Shaver (2006) described how a human’s personality affects the kinds of relationships which that human seeks with other humans; for example, highly anxious people tend to fear attachment (high rapport) with others. Furthermore, Park et al. (2014) demonstrated that anger, frustration, and narcissism are all significant (p < 0.01) positive predictors of rude, rapport-destructive behavior in humans. Due to the persona effect (Johnson et al., 2004), whereby humans treat intelligent virtual agents as social actors, one can assume the observations on human-human relationships carry over to human-virtual agent relationships. Therefore, the personalities of the students being tutored can destroy rapport with virtual agents, namely traits like neuroticism or narcissism. These traits are part of the Big Five Personality Traits, and we measured these traits through surveys in prior experiments with human-virtual agent dyads. Putting the articles together, Question 1 arises. Due to the fact we have the data to answer this question, it seems like a promising direction for further research.
  2. Question 2 is derived from the research of leaders in linguistics research using feminist ethnography to rebut old theories of learning based on heterosexist ideologies, which polarize boys’ behavior as solely conflictive and girls’ behavior as solely cooperative. The question of gender differences appears in this blog due to Category B; I am interested in the effects that differences in gender have on communication styles and how this could cause a virtual agent that is insensitive to these differences to have more difficulty building rapport because it appears so inhuman. In a book chapter written by Norma Mendoza-Denton (1999), she describes the polysemy (roughly, multiple meanings) of the word “no” in Spanish spoken by Latina high school students. Sometimes it’s used as an affirmative word and sometimes as a prelude to a summary of what was said by previous speaker; both uses are not the conventional use of “no” for confrontation or negation. Additionally, Ardington (2006) describes how, in a tutoring paradigm, the speech pattern of two high school girls oscillates between teasing wordplay and directions to focus. For example, when completing homework, one girl makes a face threatening statement via text in a joking manner (“dingbat!”, which is a cute marsupial) and the other girl demonstrates she is not offended by the joke by replying “dingbat!” which is followed by “stop it! we’re supposed to be doing homework!”. Ardington (2006) claims this kind of play-talk often goes on in the background of serious activity. Thus, Question 2 arises because it would be beneficial to rapport-building efforts to have a virtual tutor that engages in play-talk, not necessarily only with girls but also with boys and any gender; this makes the agent appear less artificial or stoic.
  3. Question 3 fits into Category A. Question 3 asks whether anxiety, as measured by our survey, can affect rapport building behavior. Kang et al. (2008) compared personalities of different subjects to their feelings of rapport with a virtual agent that used nonverbal cues (e.g. posture mirroring, head nods). Kang et al. found that each personality corresponded to the pursuit of one of three components or rapport: coordination, mutual attentiveness, or positivity (giving each other happy emotions). For example, highly agreeable people correlated highly with self-rated reports of coordination with the virtual agent, whom they were giving instructions to but were told it was human. Overall, agreeableness correlated more significantly with rapport than any other trait in the Big Five Personality group. This leads to the question of which personality traits do and don’t influence rapport. More interestingly, Question 3 asks what the effect of anxiety is, because Kang et al. found conflicting results that higher anxiety people do not benefit any more than low anxiety people from building rapport with an agent (whom they think is human). These questions both fit into Category A because the learner profile in Category A is derived from our surveys that measured the Big Five Personality traits in all our subjects.

Once a week, I will update this blog with challenges I faced in my research. surprising challenges posed by Michael’s rejection of a gender difference question. This week was not too tough but I faced one unexpected challenge (mentioned later). Luckily, I’ve already done research on learning sciences, so my research for this literature review did not have to be so extensive. I knew what topics are best suited for our lab, and what kinds of rabbit holes to avoid (such as topics in learning sciences that are too broad or narrow). One big challenge was not being able to access my previous research for this lab, because the administration needed time to renew my access to its server. This precluded me from accessing my previous work, and building off of it. However, due to my experience at this lab this was not a big setback.

The surprising challenge I faced was the controversy behind my Question 2. After an hour meeting with my research mentor, where we discussed our future plans for this semester, we came to my Question 2. It became immediately apparent that in asking Question 2, we do not attempt to make a hypothesis that stereotypes a gender as being limited to one learning behavior over another. Clearly, Question 2 is controversial due to the involvement of gender. We decided that we would carefully explore the subject if it ever arises, and personally, I would like to modify the question to address speech patterns in middle school students in general, not just girls, and avoid heading in the direction of separating one gender’s behavior from another.

SUMMARY: I read 11 articles in 9 hours, created 3 research questions, and met with my research mentor for another 2 hours to get tips on research, make concept maps, and discuss progress. We agreed that we would meet this Thursday to narrow down the research questions even more, and then hopefully move on to the next stage, which is developing a hypothesis to answer the question.


This website is dedicated to publishing my research in learning sciences, particularly on the topic of how virtual agents can most optimally interact (socially and pedagogically) with children to teach them math. This blog was created in agreement with the CREU program, and I’m grateful for the opportunity to research such a fascinating field, and to be guided by the expertise of the researchers at the renowned ArticuLab at Carnegie Mellon University, particularly my research mentor, Michael Madaio. Thanks also to Dr. Justine Cassell for her guidance and providing the necessary equipment and lab space, as well as for hiring great people! The next post will describe the purpose of this blog in detail, for newcomers to this blog.