Examining accuracy for Systematic observation approach of Personality Assessment
Sara Faili
School of Psychology, Social Work, and Social Policy
University of South Australia
RESEARCH REPORT2
Abstract
Context: People make personality attribution using images and stranger’s behavior. Here, the
basis of these personality attributions on strangers are investigated to explore their impacts and
influence to decision making processes. The study also examines the benefits of studying the
accuracy of personality impressions and the methodological approaches involved inaccuracy of
judgments. A group of 4 researchers investigated the impact of making personality attributions
in decision-making processes. The study hypothesis is “perceived personality traits will be
reflected in observed behavior. Behavioral data findings will confirm that Captain Kirk is more
arrogant (less agreeable) and Captain Picard is less proud.” Three sets of behavior were used to
assess Captain Kirk and Captain Picard personality (arrogance). From the data analysis, the
hypothesis is not supported. The study findings conclude that people do not always make
accurate judgments about strangers because they just either guess or assess their behavior based
on their social norms, expectations or environmental situations at that moment.
RESEARCH REPORT3
Introduction
On a daily basis, we knowingly or unknowingly tend to judge other people. The judgment
we make on other people’s personality reflects a significant part of their social world. Therefore,
the accurate personality assessment is crucial because it influences their opportunities,
expectancies, and their reputation in general (Dumont, 2010). According to Funder, “In the end,
we become what other people perceive or misperceive us to be”(Funder, 2013, p. 176). This
indicates that people’s judgment on the individuals can negatively or positively affect other
people’s opportunities. This illustrates the need to make accurate personality assessment.
There are several more approaches for personality assessments that have been
developed. These include the use of self-report questionnaires, structured interviews, projective
techniques, objective tests and systematic observation of people. According to Dumont, the
easiest strategy to assess people’s personality is a systematic observation in naturalistic
situations, and to keep a record of the individual behaviors of the person on interest (Dumont,
2010, p.345). This paper explores the effectiveness of using behavioral observations in assessing
personality.
Personality refers to a group of psychological traits and mechanisms inherent in an
individual that are organized in a manner that they relatively influence their adaptations to the
social, intrapsychic and physical environment. In the society we live, our personalities are
assessed every hour by our peers, ourselves, strangers and the relatives. Therefore, it can be
argued that there is some percent of accuracy in examining our personality traits and those of our
peers. According to Funder, these personality judgments do matters probably even more than
those rendered by professionals. This is because they influence others people’s perceptions about
the individual and thus their accuracy matters (Funders, 2013, p. 193). This study’s objective is to
RESEARCH REPORT4
examine the relevance of defining personality trait and the methodological approaches to
accurately assessing these traits in other people.
The Realistic Accuracy Model (RAM) highlights the four strategies that enhance the
accuracy of personality assessment. The first point states that the individual being assessed
should do something that is relevant; that is an act that informs the trait being judged. The second
strategy is that information about the act being judged should be available. This brings us to the
third strategy that states that the person judging should be well informed so as to detect the trait
to be judged. The last point is that the judge should utilize the information appropriately (Funder,
2013, p. 191). Moreover, the accuracy of personality assessment can be examined using two
significant assessment queries between the observer (interjudge agreement) and predictability of
person’s behavior.
Four factors can improve or reduce the accuracy of personality assessment. These
include; a) good judge, b) good target, c) good trait and d) good information (Funder 2013). A
good judge is highly knowledgeable about the personality trait being assessed, high cognitive
ability and overall intelligence. A good target is a person whose behavior is predictable. This is a
person who is organized, stable and consistent. A good trait is one that is visible and easily
recognizable. Good information is achieved by adequacy and quality of the information used to
make accurate personality assessment (Funder, 2013).
The Five-Factor Model of personality (also known as the Big Five) is a model that
categorizes personality in five distinct factors based on individual variations in emotional and
social status. The five factors include; extraversion, neuroticism, openness, agreeableness, and
conscientiousness (McAdams and Pals, 2006). This report explores the trait of arrogance
(agreeableness) trait in among two individuals so as to evaluate the relevance of accuracy of
RESEARCH REPORT5
personality assessment and to examine the methodological approaches that define and accurately
assess individual’s traits. In this model, agreeableness personality trait is manifested by being
kind, warm, considerate and cooperative (Tobin and Gadke, 2015, 463). This implies that a
friendly person is one who is nor arrogant and gets quite well with their peers.
This study’s objective is to examine the relevance of defining personality trait and the
methodological approaches to accurately assessing these traits in other people. The study
hypothesis is “perceived personality traits will be reflected in observed behavior. Behavioral data
findings will confirm that Captain Kirk is more arrogant (less agreeable) and Captain Picard is
less proud.”
Method
Participants: A group of four researchers Member 1, Member 2, Member 3, and Member 4
watched scenes from Star Trek Movies to determine the validity of personality attributions for
Captain James, T. Kirk, and Captain Jean – Luc Picard (Herringer, 2000).
Materials: The group watched scenes from Star Trek Movies. A hypothesis was constructed
regarding the level of arrogance (agreeableness trait) of Captain Kirk and Captain Picard.
Conceptual and operational definitions were developed as follows; the degree to which one
tolerates the other captain. That is the number a captain belittles the other captain. The
operational definition includes; a) number of times a captain interrupts another person’s
conversations, b) some occasions a captain belittles someone else experiences or opinion and c)
number of times a captain talk about or praise themselves.
Procedure: The researchers watched two scenes of ‘Two Captains’ video lasted for 51.58
minutes. While watching the two scenes, each researcher independently rated the behavioral
indicators of the three sets of traits developed for the purpose of this study; that is the 3 set of
RESEARCH REPORT6
behavioral indicator traits were used to assess Captain Kirk and Captain Picard personality
(arrogance). Data analysis was conducted by calculating mean, standard deviations, and inter-
observer reliability (cut off 100%) of the data collected by independently. The independent
ratings as observed are as illustrated by Table 1 & 2.
Results
This Observer- Report Data (O- data) was performed to assess the accuracy of
personality assessment and to explore the methodological approaches related to the accurate
assessment of personality traits. Table 1 and 2 indicate the independent scores of the four group
members and include the group mean standard deviations, and inter-observer reliability scores
for each behavior of the two captains.
To test the study hypothesis, we identify the Captains score with more group means. The
group means of the two captains are significant. As indicated in Table 1, Captain Kirk group
means for behavior 1,2, and 3 are 10.75, 12.5, and21.5 respectively. In Table 2, Captain Picard
the groups of means are 12.5, 12, and 15.75. From this analysis, it is evident that Captain Kirk
Scored higher in behavior 2 and 3, but they both had a close score in behavior 2. However, our
group had poor IRR in the behavior of both captains as IRR scores in both was zero in at least
two behaviors. The IRR was assessed by comparing one group members scores (mine) with the
others. It was calculated by finding the percentage of the total scores that were below +/- 1 of the
participant’s scores.
Table 1 Captains Kirk Rating
RESEARCH REPORT7
Table 3 Captain Picard rating
Captain Picard
Frequency of behaviours
Behaviour 1 Behaviour 2 Behaviour 3
Group member 1 12 13 26
Group member 2 7 5 17
Group member 3 15 17 22
Group member 4 9 15 22
Group means (SD) 10.75 (3.5) 12.5 (5.3) 21.25 (3.8)
Inter-observer Reliability 0 0 0.5
Captain Kirk
Frequency of behaviours
Behaviour 1 Behaviour 2 Behaviour 3
Group member 1 13 14 24
Group member 2 14 8 9
Group member 3 10 11 13
Group member 4 12 15 17
Group means (SD) 12.25 (1.70) 12 (3.2) 15.75 (6.4)
Inter-observer Reliability 0.2 0 0
Discussion
The means of the three behaviors indicate some significant difference between the two
captains (see Table 2& 3). Also, the inter-observer reliability scores of the two captains indicate
very poor inter-observer reliability scores. Therefore, from the data analysis, the study findings
fail to supports the hypothesis that “perceived personality traits will be reflected in observed
behavior. Behavior data will confirm that Captain Kirk is more arrogant and Captain Picard is
less arrogant.”
RESEARCH REPORT8
Various factors influence accurate personality assessment through observations. For
instance, the social expectancy effect where an incorrect belief or assumptions held by the rater
or observer makes them act (in this case score) in a manner that elicits biased analysis (Jamieson
et al., 2016). In this case, some of the observer’s results could have been influenced partly by
their social expectations, which could have made them rate either of the two captains in a biased
manner. Another factor is the observer drift, a cognitive phenomenon that involves a gradual
shift from the original judgment by the observer, which makes the observer make inconsistent
judgment. This raises the issue of observer accuracy vs. observer agreement (Hall, Goh, Mast, &
Hagedorn, 2015). In this case, probably the source of poor IRR was due to the observers drift.
Also, Funder argues that most people personality assessment is based on their
constructions of reality. Therefore, in personality psychology, there are no accurate or inaccurate
interpretations of personality assessment because all interpretations are just “social constructions’
(Funder, 2013, p. 177). Following this argument, it is probable that the poor IRR reported by our
group was due to poor conceptual and operational definitions.
As aforementioned, various factors are likely to affect the accuracy of personality
assessment include; a) the good judge- possibility that some people judgments are more accurate,
b) the good target – possibility that some people can be easily judged than others, c) good trait –
possibility that some behaviors can be quickly judged accurately and d) good information- ability
that one is well informed when making personality assessment. Therefore, personality
assessment accuracy is determined by the quality and quantity of the information and its
relevance to the traits being studied (Funder, 2013). In this context, the group observers were a
good judge as indicated by the group of means. The trait assessed (captains arrogance) is easily
identifiable and judged, and the study target was appropriate. The main issue as indicated by the
RESEARCH REPORT9
poor IRR was information quality and adequacy to make a final personal assessment. This
implies that while some behaviors were identified as arrogant by some group members, others
perceived them as not arrogant.
The study findings indicate that making an accurate judgment is tough. However, it can
be improved through four different ways. Previously, the improvements have focused on making
the judges or observers to think more and to use the good logic to prevent inferential errors.
Although necessary, the efforts focus only on utilization phase only. Therefore, to improve
personality assessment accuracy, then one should develop an interpersonal environment where
the judged persons can be themselves. Importantly, one must minimize tensions and other
distractions that could make one miss relevant information (Funder, 2013).
Limitations
The study limitation is in this research includes a poor operational definition of the term
“arrogance.” Appropriate conceptual analysis and operational definitions are important as it
influences the study’s validity. In this case, it can be argued the poor inter-observer reliability
scores was caused by disagreements in operational definitions. Also, the researchers observed the
video separately. Therefore, their ability to rate the captain’s behaviors could have been
influenced by different experiences or circumstances. These findings indicate that observing
individual actions alone is not adequate to judge someone’s personality.
In summary, it is evident that most people make personality attribution using observed
information and as guided by their social constructs. The study findings conclude that people do
not always make accurate judgments about strangers because they just either guess or assess their
behavior based on their social norms, expectations or environmental situations at that moment.
This paper has indicated ways personality attributions influence people’s decision making
RESEARCH REPORT10
processes. Therefore, to increase personality assessment accuracy, it is important to put into
considerations RAMs strategies and the four factors above that influence personality assessment
accuracy.
RESEARCH REPORT11
References
Borkenau, P., Mosch, A., Tandler, N., &Wolf, A. (2014). The accuracy of judgments of
personality based on textual information on the main life domains. Journal of
Personality, 84, 214–224.
Dumont, F. (2010). A history of personality psychology: Theory, Science, and Research from
Hellenism to the twenty-first century. Cambridge: Cambridge University Press.
Funder, D. C. (2013). The personality puzzle (6 th ed). New York: W.W. Norton & Company, 171-
195.
Hall, J. A., Goh, J. X., Mast, M. S., & Hagedorn, C. (2015). Individual differences in accurately
judging personality from text. Journal of Personality,84, 433–435.
doi:10.1111/jopy.12170
Herringer, L. G. (2000). The two Captains: A research exercise using Star Trek.The teaching of
Psychology, 27, 50-51. doi: 10.1207/s15328023top2701_12
Jamieson, J. P., Peters, B. J., Greenwood, E. J., &Altose, A. J. (2016). Reappraising Stress
Arousal Improves Performance and Reduces Evaluation Anxiety in Classroom Exam
Situations. Social Psychological and Personality Science, 7(6), 579-587.
Larsen, R. J., & Buss, D. M. (2014). Personality psychology: Domains of knowledge about
human nature. New York: McGraw-Hill Education.
McAdams, D. P & Pals, J. L. (2006). A new Big Five: Fundamental principles for an integrative
science of personality. American Psychologist, 61(3), 204-217.
