Psychometrics is a field that explains survey development in conducting studies. Psychometric
study checks if a survey accomplishes what it intends to measure. It strives to assertain whether a
certain variable accurately measures a given construct (like opinions regarding a certain area of
study). Psychometrics is known to assist researchers and evaluators to choose appropriate
surveys and identify whether their survey is performing as intended (Kumar, 2009). Techniques
for measuring variables in a study involve levels of measurement, reliability and validity.
There are 4 levels of measurement which are associated with any outcome, and they include;
Nominal level data: This is data produced when a measure categorizes people but the outcome of
the measure which is numbers does not represent any hierarchy.
Ordinal level data: The scores in this scale are arranged in order, ranging from the lowest to the
highest. This is the main difference between ordinal and nominal scales. Ordinal scales provide a
mechanism to compare the extent to which 2 subjects influence the dependant variable but do not
have equal intervals.
Interval level data: It usually involves order between scores and equal intervals between units of
measurement, and it does not have a true zero.
Ratio level data: This category involves order between scores and equal intervals between the
measurement units, and it has a true zero (Kumar, 2009).
Reliability is a concept applied in measurement where one gains confidence that the outcome of
a measure does not change and produces the same result each and every time when it is used.
Although measures are not reliable in the entire period, a measure is assumed to be reliable for
use only with a particular population under study. The reliability coefficient is the mostly used
statistic when investigating the reliability of a test. The several types of reliability include;
Test-Retest Reliability: It refers to the test’s consistency in test results, especially self-report
tests, when administered on more than one occasion. To determine the coefficient for this type of
reliability, the same test is given to a group of subjects on at least two separate occasions
Internal Consistency: is a type of reliability used to test measures of outcome which usually test a
single concept, and assesses the magnitude to which all items in the outcome measure address
the same underlying concept
Intra-rater Reliability: It shows the consistency with which a rater performs and scores an
outcome measure.
Inter-rater Reliability: It shows the consistency with which 2 raters come to a consensus when
they administer and score a measure of outcome (Kumar, 2009).
Validity is the degree to which an instrument of measure achieves its intended purpose. The
different aspects of validity include;
Face validity: It measures the magnitude by which a test seems to measure what is intended.
Content validity: It measures the degree to which a test brings on board all the items which
should be included to represent the concept under study.
Criterion Validity: It can also be referred to as predictive validity because it measures the extent
to which one variable or several variables can predict the outcome of other variables in study.
Construct validity: It strives to assess whether a test can measure the variables of interest in a
study as required by a clinician or researcher (Kumar, 2009).
Robert (2013) conducts a study to investigate the “Psychological and Neurological Bases of
Leader Self-Complexity and Effects on Adaptive Decision-Making”. Robert looks into the
leader’s complex working environments and the need to change their decision making and
behaviors to produce effective responses. The researcher uses techniques used in clinical and
social psychology to make an index for psychological LSC (leader self-complexity). A multi
method approach is used to form and test hypotheses of the study which states that psychological
and neurological bases are both measures of LSC.
In conducting the study, the researcher tests the following hypothesis;
H0: Leaders who have higher levels of neurological LSC usually demonstrate higher levels of
adaptive decision-making.
103 participants who volunteered were military leaders with their levels of experience ranging
from low to high. The use of a self-reporting exercise(test retest reliability), and externally coded
responses provides the multi method procedure which limits common method biases which may
occur and hence increase the reliability and validity of the study. The study measurements on
leadership attributes were conducted using a three point scale with the following units of
measurement: not important = 0, important = 1 or very important = 2. The used scale provided
for ordinal level data measurement.
Apart from measuring the leadership attributes of the military personnel, the researcher also
conducted an adaptive decision making measure. The researcher studied the importance of LSC
by taking adaptive decision making as a form of measuring adaptive leadership performance
which was measured by placing the military personnel through a 4 part tactical military scenario.
The soldiers’ responses were audio-recorded, transcribed and content-analyze. The researcher
followed the process to ensure that the outcome of the adaptive decision-making tests
administered on the military personnel.
A three point scale with the following units; “low = 1, moderate = 2, or high = 3” was created
and then used to rate each of the three adaptable decision making aspects (situational analysis,
decisiveness, and positivity) in the 4 part scenario exercise.
“Coding of the content was done through the assistance of 2 independent military
personnel outside the research team, each former military officer with significant experience and
expertise with the type of tactical situation used in the scenario (content validity). A training
session was conducted to familiarize each rater with the indicators and to practice on sample
participant responses. Practice coding protocols were discussed and more refined coding rules
were developed to eliminate ambiguity in the coding rules (content validity). Each rater then
independently coded each of the participant transcripts in randomized order. After initial coding
produced a mean Cohen’s kappa = .73, reflecting substantial inter rater reliability, discrepant
scores between raters were identified, discussed and resolved using a consensus approach in a
second iteration, ultimately achieving 100% reliability”.
After conducting a correlation test, both psychological and neurological markers of LSC were
positively associated with demonstrated levels of adaptive decision-making on a complex
leadership task, as rated by expert coders. More complex leaders portray higher adaptive
thinking, decisiveness, and positivity in decision making as they sorted daily task needs and
made leadership decisions in response to an evolving, four-part leadership scenario that escalated
in complexity in each part according to the results of the regression analysis conducted.
From the first correlation test conducted, the ordinal measurement scale used was not the best
method for providing scores to be analyzed since a Regression analysis and Pearson Correlation
tests use interval or ratio data so as to obtain more powerful results (Gay, et al, 2009).
Triangulation employed during data collection enabled presentation of valid and reliable data.
References
Gay, L, R., Mills, E. G., & Airasian, P., (2009). Educational Research: Competencies for
Analysis and Applications (10 th ed.)
Kumar, R. (2009). Research Methodology: A step-by-step Guide for Beginners. Greater Kalash:
Sage Publications.
