Brain Can: horsemen of Apocalyptic statistics

BRAIN CAN
184 followers
583 Video Views·Aug 3, 2025
Inspired by the Nature report about the irreproducibility crisis, I met with the data scientist Alex Kirko to discuss how common mistakes in statistics violate scientific concepts.
To familiarize yourself with the topic, read in NATURE PORTFOLIO "Six factors affecting reproducibility in life science research and how to handle them".
Thank you for watching!
Comment, like, share, subscribe.
hello everyone this is Brian Ken Channel
and this episode will be in new format
and we will discuss statistics because
statistics this is what any researcher
has to deal with and my guest for today
episode is a great expert in data
Science and Mathematics Alexander Kiko
so you can tell a bit about yourself
oh uh
glad to be here so I worked a number of
years
with medical models and statistics
primarily in finance and banking but
lately in advertisements my education is
in risk management and probability
Theory so I have worked a fair bit with
statistical hypothesis testing with
different methodologies you can use
and so on and so forth and my latest
work has dealt quite explicitly with
probability Theory and will getting
significant results that a company can
base some decisions on because in
business
as in science it is important to be sure
that the statistical findings that you
base your decisions on are actually real
because people pay
um
so
um I actually have couple of friends
that were
successful researchers actually but they
moved to data science because in data
science salaries are much higher than in
research tell me why are you paid more
because we make money directly for a
company and not through the grant system
as most fundamental research I suspect
that is the primary reason actually it's
not because we are smarter because our
methods are better or something like
that it's simply that the amount of
money available in the industry is
larger so people get paid more
and so I can conclude
conclu that the real data
bring money
yes just like a direct direct
consequence
so and now I will explain you why I
decided to dedicate this episode
um statistics
seven years ago
nature
published a paper about reproducibility
reproducibility problem
accordingly to the survey that nature
has done but here about 70 percent of
researchers in biology cannot reproduce
findings of their colleagues
and approximately 60 percent of them
cannot reproduce their previous findings
this is
a crucial crisis because scientific
thinking is based on the idea that if
you have a finding and that other people
can repeat or replicate this finding
this indicates that this finding is true
and that Concepts behind the spine are
also true and if we cannot reproduce
them
we put at risk
research that
stands behind it
and another paper estimated
costs of this Europe reducible tea
of clinical research and this cost is
approximately
28 billion dollars per year
so and this is here all together it has
a great impact on
um
clinical and medical research because it
means that many of studies that people
make every year uh
they do not lead to real Improvement in
patients
and many things can hide hide behind
this picture I'm not going to talk about
different methodologies that are
difficult to translate and not go into
talk about variability in biology what
we will talk about is statistics because
any researcher deals with it and this is
quite a subtle method because many
things in statistics based on
assumptions right
and if you want you can manipulate these
assumptions to our video data and to fit
your data to the hypothesis that you
initially had and in this situation this
is already could be absolutely a
different picture
so
based on what I've just told what is a
statistical significance
okay so statistical significance very
basically is our confidence that the
results we get in an experiment or the
results we see in the data set are an
actual finding and not just a
consequence of randomness
because
you can understand that whatever result
you're looking for you can just get it
randomly with some small probability you
can you can throw a coin 10 times and
this can come up heads every time just
very rarely so in science and in
business we need to be sure that what we
see is with some very high level of
confidence not random and statistical
significance tell us as that
how
so yeah how does it tell yeah this gets
a bit more difficult because the way
people think about statistics uh
normally is that you have a hypothesis
and you try to prove it and you prove it
with some probability that is actually
not how things work in science or in
statistics so you cannot prove a
hypothesis instead with Statistics you
can only reject one so the way
scientists work and the way data
scientists work is we have a hypothesis
let's say that men are taller on average
than women
and we try to disprove the opposite of
that
so the opposite hypothesis would be that
men are the same height as women or
actually the same all over height
so we take our data we assume a certain
distribution that let's say the height
is normally distributed
and we try to see how improbable is the
data
we get given this opposite hypothesis so
in this case we would get much larger
average heights for women than women and
the probability that the sample of men
we got was just randomly taller than the
sample of women regard would be very low
for a reasonably sized sample like in
the hundreds and based on that we would
be able to see that we have
quite probably have found something and
it's not just random noise so this is
how all hypothesis disproving functions
in science you assume the opposite and
try to disprove it and if you cannot and
if you can disprove the opposite then
you have probably found something
how can
I distinguish noise in my data from not
nice
uh so
this is a difficult topic yes it sounds
like very easy but this is actually not
you understand that
you need some sort of empirical
threshold to separate the noise from
your findings
so in economics we usually say that if
we are if we have statistical
significance of 0.95 so 95 percent then
that is good enough to publish a
scientific paper probably if everything
was done correctly in in in stem fields
in physics and everything the threshold
is normally much higher something like
99.99 in order to be sure that whatever
you have isn't nonsense this depends uh
partially on how expensive it is to get
samples
when when getting samples is very
expensive scientists and in business
also we normally say okay we can go with
a little lower
confidence yeah
very dangerous path but
um
speaking about this threshold
if we are looking for a very small tiny
but still very important difference for
example if we talk about improvements a
new treatment generally this is subtle
very little differences but still they
have a great impact on patient life
in this case how can we choose the
better threshold
does it really is it related somehow
with the sample size
yes and the problem that we are studying
yes of course I mean that is actually
the question the three questions
so there is no one answer because it
depends on how large the difference is
that you suspect is there it depends on
how expensive it is to run the trial or
on the experiment how how much each
sample costs because you may Simply Be
constrained by Financial Resources
okay let's take an ideal situation
everything is free yes everything is
free and we are looking for a very small
difference uh yes and we want to be as
much correct as we could yeah so
there are various empirical solutions to
this normally if you're looking for
small differences you need thousands of
samples in order to be reasonably sure
that you have found something so people
who are working on a shortage of uh
animals in experiments think about it
we need solvents
yeah you can you can suspect that begin
to suspect that you got something useful
with a small sample but in order to have
a strong result that you are confident
in
you normally need large sample sizes all
very large differences you're looking
for so yes
it's a
fire about this
business
in the in some parameter that we are
looking at and uh
this sample size so sure it's very
simple so if you are so yeah
so if you are looking uh if you are
looking for something very small like
for some reason you suspect that people
with red hair make more money in
academic jobs
slightly more
then you would give because whatever
good luck you will have you will need
very large sample size to detect this
but if you're but if you suddenly woke
up one day and decided to test if a 30
year old adult man
can
weigh more than a two-year-old child
then you would need a very small sample
size because the difference you are
suspecting is like
10 times the weight between them so for
various so for very large differences
normally you can get by with dozens of
samples that is not that is not a
problem but very small differences you
normally either need to be very creative
to somehow increase them like sub sample
where you suspect they are the strongest
and that is my next question yes uh
because again this is quite often a
problem in reality that we have to deal
with small samples
what can we do in this case
so
I mean the best answer that people would
give you in the university is just get
more samples so I can find somewhere a
lot of money and people find funding yes
unfortunately that is often not
available as you know
so what you can do is you can make your
sampling not just random as it should be
purely theoretically you can have a
hypothesis where the differences are the
largest and not have for example a
representative population that
represents the larger population in your
country or something but pick the
specific people where you expect the
effect to be the strongest
and then you can use basically
statistical tricks in order to
extrapolate your findings onto the wider
population
so that is also possible this is called
the broadly speaking the class imbalance
problems in statistics we try to get as
much data as possible for a rare
and significant things and we don't need
as much data for common
Place results and findings so you can
when you run your experiment you can
change the balance between classes
relative to what is in the real world
and then you can adjust it
mathematically back to what you find in
the real world of course you introduce
more uncertainty
if everything is free it's better to
just get more samples things aren't free
about uncertainty uh I had about a big
high be value hunting so could you
please tell me what is it oh BYU hunting
has little to do with uncertainty it's
more just a very bad academic practice
so and still very common because I made
it in papers yes it's it's very easy to
explain with an example
uh let's say I'm trying to prove that I
have an unfair coin so it comes up heads
let's say modern trails so a fair thing
to do this would be to just throw a coin
10 times see if the number is near five
or not and based on that make the
conclusion but what I can do if I really
really want to get the result I can
throw the coin 10 times and I can run
this experiment 10 times
and one of those 10 will probably be
quite extreme and then I just put that
into my paper and forget about the other
nine experiments I read
I just so so this is an exaggerated
example in science people normally do a
different thing which is equivalent like
you suspect that something causes an
illness and you have like 20 factors
and you check the statistical
significance of each of these factors
independently
and then you just pick one you found and
don't mention the other 19.
and
the fair way to do this is that you need
a much higher statistical significance
if you run multiple independent checks
so you need a lot more samples to to
have the same confidence in your results
but what you can do is an unscrupulous
scientists and scrupulous you can just
do this we found a core of lie
scientific lie I think some of people do
it unconsciously
because they think if if they if they
got a bad result or result that they
didn't expect that probably something
wrong with their matters always I don't
know their practical skills probably but
it happened
it happened in science that great
discoveries
came from really unexpected sides
so I I ask anybody working in science
before you throw something that you
think that it is not that you expect it
or that it doesn't fit your initial
hypothesis
don't do it because probably
very
basic and still unknown things
could be hidden in this data yes you can
always you can you can include it and
then in discussion you can explain or at
least you can try to explain uh causes
of this points
but they are still very important and
remember negative result is also result
yes when throwing away throwing away
data should be done explicitly with
explanation why it is done because when
you throw away valid data non-randomly
this is how you end up with the
irreproducible yes exactly crazy results
yes
and it's very common like I come from
economics and it is
a lot of people simply don't understand
how statistics work like they they
really do not understand that you cannot
run 30 independent experiments to prove
your one hypothesis then pick the one
experiments which looks nice and just
forget about the rest this is
the wrong way to do statistics yes so
and to the size in a science in general
yeah because remember again if we are
looking at very subtle at very simple
things and mechanisms that are not
known yet
we can find something important in
things that we don't understand and that
we don't consider as important and just
throw it so do not throw your data
explain it
yes I think this is a perfect point to
finish thank you Alex for this great
talk always a pleasure
