Using bad measuring instruments can have disastrous consequences. Therefore newly developed tests, questionaires, etc. have to meet the highest standards. Along with reliability and objectivity validity is one of the most important quality criteria. An instrument is regarded as valid, if it measures, what it is supposed to measure. For example: A (supposed) intelligence test which measures creativity or sustained attention may be reliable, but it is not valid. In this episode we will discuss the different methods to validate a measuring instrument: - face validtiy - criterion validity - predictive - postdictive - construct validity - discriminant construct validity - concurrent construct validity
hello and welcome to the world of
Diagnostics validity the most important
quality aspect of a measurement
instrument or does my test really
measure what it's supposed to measure
[Music]
in order to offer good and appropriate
interventions we need Diagnostics we
need valid measurement instruments
because we need to figure out what is
the real problem for example think of a
pupil that is suffering from getting
really bad grades in order to help this
pupil we need to figure out what is the
reason for these bad grades maybe there
are family problems the parents are in
danger of getting divorced or maybe uh a
grandpa has recently died or maybe the
kid is just um fighting zombies every
night and
killing enemies soldiers in well not in
real life but maybe in a ual reality and
as it's getting too little sleep it's
not able to concentrate at school but
there may be also other reasons maybe
the general mental abilities are not
high enough for the school it's just
attending and this could be measured by
an intelligence test but there could
also be motivational problems or even
some depressive
symptoms um that could be assessed with
a questionnaire on depressive symptoms
but all these gathered informations are
just as good as the method they are
assessed with and the most
important quality aspect for measurement
instruments like questionnaires or tests
like intelligence tests creativity tests
attention test
Etc the most important quality aspect is
the so-called
validity let me give you a short example
of an completely invalid intelligence
test for this test um you just have to
think of a number between 1 and
10 just choose a number maybe not a too
obvious one you got got one okay um I
suppose let's have a little mind reading
here I suppose it's been
7 or maybe
three if I got it right don't worry it's
not mind reading it's just probability
most people go for seven or three and my
test says well if you go for seven or
three you are very intelligent if you
chose another number you are not so
this is a completely pseudo
psychological test as you find thousands
on the internet it is completely invalid
because it's not really measuring what
it's supposed to measure it does not
measure your intelligence all it
measures is your preference for a number
between 1 and 10 and it's not saying
anything about you or your intelligence
and it's obious obvious everybody can
tell hey come on this is this
is not an intelligence test and in fact
this is one way to validate the test
it's the so-called face
validity you show your newly developed
test to an expert and the expert tells
you by looking at the test well come on
you don't measure intelligence with this
test the face validity is reg Ed as a
quite weak way of validation as also
experts sometimes can fail so there are
more important ways of validating a test
by the way every test or every
measurement instrument has just one
validity the different kinds of validity
we are going to talk about like face
validity Criterion validity construct
validity these are just methods to
validate the test these are not
different kinds of validity these are
methods to validate the tests or the
questionnaire
[Music]
Etc
[Music]
a stronger way of validating a test is
the so-called Criterion
validity a Criterion is a
concrete
measurable event end or
result that you try to predict with your
measurement instrument it's a
theoretically expected result or event
for example in case of an intelligence
test we expect it to be a good predictor
for a Criterion like school
grades because from Theory at school
cognitive abilities are very important
so we expect an intelligence test a good
intelligence test should be able to
predict school grades and in fact good
intelligence tests are really able to
predict the Criterion um school grades
or other kinds of educational success
and if you think of my mind reader
intelligence test there probably would
be no correlation at all between the
results of my test and and educational
outcomes so my test has no Criterion
validity but intelligence tests are not
only good predictors of educational
outcomes but also of Occupational
outcomes on average people that score
higher in intelligence tests are rated
higher by their
employers their bosses are just more
satisfied by their work and this is
especially the case in jobs that require
a lot of logical abstract thinking and
the correlation of course is smaller for
jobs that don't regard the solving of
very complex
problems but intelligence and this is
especially important for companies is
not the only one really good predictor
of how good an employee is going going
to make
it in a very famous and often cited meta
analysis Schmid and Hunter analyzed the
predictive validity of a lot of methods
to figure out is this employee really
the right one for us and besides General
mental ability tests especially work
tests you have the job applicant work in
your company for a short time in order
to find out is he doing well or not so
well um work tests as well as structured
interviews in which you follow a clear
structure of when to ask which
question these were the top three
methods in order to find out is this job
applicant really the right one for
us
interestingly as investment centers
which are quite uneconomical because
they not only cost a lot of money but
also a lot of time assessment centers
were weaker in predicting the quality of
the
employee and what had no validity at all
were tests of
graphology so tests in which you analyze
the
handwriting the predictions made based
on these tests were extremely
bad according to the time when the data
for the
Criterion like for example school grades
and when the data of the measurement
instrument for example the intelligence
test according to the time when the data
are collected we differentiate between
different kinds of Criterion
validity um if the data for the
Criterion and the for in our example
intelligence test are collected at the
same time we speak of concurrent
validity if we want to predict a
Criterion if for example we want to test
whether our intelligence test is able to
predict the school grades 3 years later
we speak of the so-called predictive
validity predictive validity is quite
common whereas the so-called postdictive
validity in which you already have the
data of the Criterion for example you
are interested in Nobel Prize winners
and then you look in the past for
informations in their
biography that could be a good predictor
for winning the Nobel
Prize maybe every one of them took a nap
every
afternoon probably the most important
way of validation is the so-called
construct validity constructs in
comparison to Criterion are not concrete
they are abstract concepts for example
intelligence
creativity
neuroticism
conscientiousness openness and so on
these are abstract
Concepts that we have more or less
defined but they are still changing over
time um for example the definition for
intelligence we have today is not the
same we had 100 or 50 years ago so if we
have an newly developed intelligence
test and we want to know its construct
validity this means we have to make sure
that this test is really meas measuring
the construct it's supposed to measure
so our newly developed intelligence test
should measure the same as other
established intelligence tests this is
meant by the so-called convergent
construct
validity but on the other hand it's not
only supposed to measure the same as um
established intelligence tests like the
Raven metrices or the tell culture Fair
intelligence test it's also supposed not
to measure other constructs so an
intelligence test should not measure
creativity and it should not measure
test
anxiety so to make sure that it's really
measuring what it's supposed to measure
we need a sample of people that has
presented our newly developed
intelligence test um we need to give
them another intelligence test in order
to test for convergent construct
validity but we also should give them
for example an creativity test or an
test anxiety test or an test this is
often done or in test measuring
attention um and then we have to look at
the results is there a correlation
between the results of the test anxiety
test or the attention test and my newly
developed intelligence
test if there's only a small
correlation um this would be okay
because from Theory we suppose okay
there is a little correlation between
test anxiety and intelligence tests and
also there is a little correlation
between attention tests and intelligence
test this is okay as well but if there
is a really high correlation for example
between an established intention test
and my newly developed intelligence test
this would be highly problematic because
my test claims to measure intelligence
and not attention and so as a result our
job would be to improve the intelligence
test by looking for items that are
highly correlated with the attention
test and we should eliminate those items
and create better items that are really
measuring
intelligence so that's it for today on
validity if you should be a little bit
more interested in a method for testing
construct
validity um the so-called multitrade
multi method analysis check out the
video on that topic you can find it on
my YouTube channel or on the website
learning psychology.com
[Music]
[Applause]
[Music]
net
[Music]