How does a medical team know where to get their information? How do they separate trustworthy medical research from misleading studies?
In this deep-dive conversation, neonatologists Dr. Tala and Dr. Veeral explain how healthcare professionals use medical literature, PubMed, OpenEvidence, UpToDate, Cochrane Reviews, peer-reviewed research, systematic reviews, meta-analyses, and AI-powered tools to make evidence-based medical decisions.
You'll learn:
How doctors search PubMed effectively What peer review actually means How UpToDate and OpenEvidence work Why some studies are more trustworthy than others Understanding systematic reviews and meta-analyses How to interpret forest plots and confidence intervals The difference between good journals and predatory journals How AI is changing evidence-based medicine How clinicians evaluate medical research in real-world patient care
Whether you're a doctor, nurse, medical student, researcher, healthcare professional, or simply interested in evidence-based medicine, this episode provides a practical guide to navigating the overwhelming world of medical research.
--------------------------------------------------------------------- Do you want extra Tala Talks education? A neonatal tip a week? Access to LIVE videos? Behind-the-scenes members-only videos? Worksheets of the videos? Do you want to decide which videos we should be publishing next? If so- join our new (low-priced!) membership community! Join here: https://www.youtube.com/channel/UCxv2qGB7HR5lR2D7IdoUAJA/join Help us guide the future of this channel and get involved in our community!
For PDF summaries of the videos, links to relevant articles (+ 50 free Multiple Choice Questions/ Answers and explanations!!), AND a bunch of other stuff, subscribe to our Neonatal Nuggets Newsletter here: https://talatalksnicu.substack.com/subscribe ---------------------------------------------------------------------
---------------------------------------------------------------------------- ***TIMESTAMPS*** 0:00 Introduction: How to Navigate Medical Literature 2:10 The Medical Literature Problem 2:37 Tool #1: UpToDate – Quick Clinical Summaries 3:38 What Is Peer Review? (Explained Simply) 4:31 How to Actually Use UpToDate 7:52 The CAP Trial & How to Find Seminal Papers 8:52 Tool #2: Open Evidence – AI-Trained Only on Medical Literature 14:00 Tool #3: PubMed – Searching Primary Medical Literature 24:10 Journal Quality – Flagship, Tier 2, and Predatory Journals 32:07 Tool #4: Cochrane Library – Systematic Reviews & Meta-Analyses 35:16 How to Read a Forest Plot 46:55 Tool #5: AAP, ACOG & Society Guidelines 47:23 Tool #6: Google – Tips for Providers & Families 50:42 Academic Institution Websites (UCSF, CHOP, Stanford, Hopkins) 55:41 Summary – Which Tool to Use When
---------------------------------------------------------------------------- Dr. Tala is a board-certified neonatologist and has worked in busy level III and IV units for the past 18 years. She has won multiple teaching awards throughout her time as a neonatologist. ----------------------------------------------------------------------------
*Disclaimer*: This video is intended for educational purposes only, and while we strive to give the most accurate information, errors may occur. Subsequently, this video should not be a replacement for medical advice.
How do you use the online world to decide what you want to actually do at the bedside?
The hard part is finding the papers, finding which of those papers are applicable to your baby.
The number of papers in the medical literature now doubles every seven to ten months.
Peer-reviewed is one of the features of the medical literature that is designed to
improve and be self-correcting.
Hi, I'm Dr. Tala and today I'm absolutely thrilled to be talking to my good friend,
Dr. Viral Toilia, who is a clinical neonatologist as well as really a leader in neonatal research.
Apart from being an excellent neonatologist, he also currently serves as the Director of
the Clinical Data Warehouse of Research of Pediatrics.
He is extremely knowledgeable about how research is done as well as how to actually read the
literature.
So thank you so much for being here today.
What we are actually going to talk about today is how to actually navigate medical literature.
So more specifically, say you're at the bedside or you're a parent or you're a provider and
you're in the NICU and you have a question and you don't really know what direction to
go in, then how do you use the online world to decide what you want to actually do at
the bedside?
So thank you so much for being here today.
Tala, I'm thrilled to be here.
Thank you so much for having me.
Thank you.
Okay, so let's start with, let's say we have a baby who was just born and it's a 30-week
baby who is on CPAP and you can't decide whether to start caffeine or not.
You go through your hospital guidelines, you realize that there isn't a hospital guideline
for that, that it's just kind of left up to the providers.
And so you're trying to decide from a kind of evidence-based standpoint, should you be
starting caffeine?
What would you do?
So I think this is a great example for a couple of reasons.
The first is that there are clearly papers that might address this specific question.
The hard part is finding the papers, finding which of those papers are applicable to your
baby specifically, and kind of in general what the consensus is about that medical
literature.
So the medical literature is this large, vast, there's, you know, the number of papers in
the medical literature now doubles every 7 to 10 months.
That's ridiculous.
It's an insane amount of information.
So sometimes when you have a query, the best way to kind of break it down is not to look
for the primary paper to try to find a summary of what the medical literature says.
So there are a couple of different ways and a couple of different tools that you can use
to do that.
The first that I'm going to show you is something that's called UpToDate.
UpToDate is a subscription service.
Most hospitals have a subscription to UpToDate.
You can also get a personal subscription UpToDate, but I think it's pretty expensive.
But as every hospital that I've ever worked in has a subscription, an institutional subscription
UpToDate, and I think that that's true as well.
And that's like for everybody in the hospital.
If you have logged into a computer, then you can get to UpToDate.
It's not just for providers or whatever.
That's right.
Yeah, everyone.
Anyway, so let's start with the UpToDate.
So, you know, there's a search box here.
There's now an Ask Expert AI feature, which I've never used.
But in this case, what we're looking for is caffeine in premature infants.
OK.
Let's see what it comes up with.
So this is not AI, UpToDate?
This is not AI, right.
So this is written by a group of authors.
Is it peer-reviewed?
It is not peer-reviewed in a traditional sense that a journal is peer-reviewed.
And what is the traditional sense of peer-reviewed?
I feel people throw that around a lot, but not everybody completely understands what that means.
Yeah.
So peer-reviewed is one of the features of the medical literature that is designed to
improve and be self-correcting.
So if I have a paper or a paper that we've written, so we've written a couple of papers
together, what will happen is that I'll send that paper to a journal for the journal to
decide if they want to publish it.
That journal will in turn find a couple of people that they consider experts in the field
to review the study, to review the manuscript that we've set in, and then give the editor
of the journal some feedback on that study or that manuscript.
So basically just a objective look at the information that's being put down by different
experts that are unattached to the original manuscript, basically.
That's right.
In this case, with UpToDate, it's not externally peer-reviewed.
UpToDate, though, is not peer-reviewed in that traditional sense.
So when you do a query in UpToDate, in this case, we'll pull up this article on apnea
prematurity, you'll find that there is an author listed here at the top.
You'll find when the literature that they've included in the article is reviewed through
and when the topic was last updated.
My understanding of UpToDate is that in addition to the author who traditionally is chosen
as an expert in the field—
Dr. Richard Martin.
—the section editor and the deputy editor also review these, but they're not peer-reviewed
in a traditional sense.
Okay.
And in some cases, that's really good because they can provide you some additional context
for the decisions you're making.
In some cases, though, that's challenging because they're giving you their opinion.
Yes.
And that's not necessarily what you want.
Yeah.
I think about that when I'm peer-reviewing.
I'm like, this is what we do, but it's not necessarily the correct thing.
Okay, so we looked this up.
So how do you actually find the information?
Where do you look to see, should I be starting this baby on caffeine on this?
So this is kind of your fastest, quickest way of getting summary information.
And it gives you kind of a one-page review of how apnea prematurity is managed.
Some of this stuff is more helpful, but in this case, it talks about the difference between
prevention of apnea prematurity.
In this case, we have a 30-week-old.
Maybe they haven't started having apnea yet, and so you'd be talking about prophylactic
use of caffeine.
There's also treatment use of caffeine, where you're actually treating the symptoms as
they're happening.
And it kind of goes through and kind of outlines some of those challenges.
That's interesting they say, in our practice as well.
So that is, did it go up a little bit?
I've definitely read that more and up to date.
Yes, here we go.
Apnea expels a frequent prolonged or associated bradycardia or frequent oxygen desats.
In our practice, we use an oxygen saturation threshold of less than 85%.
And I think that that's a window into, these are opinions.
They're not, you know, and they're, you know, I mean, they're all opinions, but this is,
this author's opinion and their clinical judgment of how this gets practiced.
That's great.
And then you were going all the way down to go point out the references.
Yes.
So the nice thing about all of these up-to-date articles is there's two things.
One is that there are links to other things.
So in this case, this article is just about management of apnea prematurity.
If you want to understand kind of the pathogenesis or kind of why it happens or how it might
show up clinically, there's a link to another article in up-to-date.
But then the other really nice thing is when you get, you know, this, this kind of talks
about how you would treat it.
So caffeine is not the only way you can treat apnea prematurity.
If you look over here at the sub tabs, you can also look at different kind of respiratory
support measures in addition to the medications.
I love that it gives doses.
Doses are never given in articles.
It's very nice.
This is really, I think, geared towards the practicing clinician.
Yeah.
And then you get to kind of, you know, all the way through to discharge.
When do you send babies home that have apnea prematurity?
Yeah.
They oftentimes will show some society guidelines, which you can talk about a little bit later.
And then they provide references for everything.
Okay.
So those, that's great.
Those are basically all those individual papers that we would go and look for that might provide
some of the information, but you'd have to read a lot more to get there.
That's right.
So for instance, I know, and you know, because we've, we've known the literature that one
of the seminal papers on caffeine was this study that was published in 2006 in the New
England Journal of Medicine by Barbara Schmidt.
It was called the CAP trial.
It was a very important trial.
Sometimes in this case, I'm on the Baylor network and it directs me directly to this,
to the PubMed index page for this study.
So this is not the study site.
This is kind of the large reference that, that manages the medical literature.
But how you would know that without kind of expert knowledge in the field is a little
bit more challenging.
You wouldn't know that that's the specific article that you need to go to.
But by looking at up to date, it's then going to take you to at least the selection of articles
that are very important.
The most important ones.
Okay.
So that's a very good starting place for general knowledge.
Right.
What about, this is obviously not AI.
It's a very real, very expert human being.
You also talked about using AI.
Yes.
Like a general.
So there's, there's a new tool that's, that's been developed.
That's called Open Evidence.
I'm going to, and Open Evidence is a large language model, kind of like ChatGPT.
And so it's a chat bot.
The distinguishing feature about Open Evidence versus asking medical questions to ChatGPT
or Claude is that Open Evidence is trained only on the medical literature.
And so the likelihood of it hallucinating or coming up with something that is totally
untrue is really mitigated.
Okay.
When you say only medical literature, literally you have to be published in a high impact journal.
Is it also including like, you know, my YouTube videos or, or is it literally only from journals?
My understanding...
Or textbooks or whatever as well.
My understanding of Open Evidence is that it is not connected to the internet.
Okay.
It is only pulling from specific medical literature sources.
Okay.
That's great.
Yeah.
The nice thing about Open Evidence, so Open Evidence is a little bit harder to access.
There are a few free questions I guess you get a month, but generally you'll need an NPI number
to access, to access.
And that is for everybody outside the US, that's a national provider ID or something.
That's right.
It's like a 12 letter number that literally the only time I use it is when I'm asked by
something like this.
I don't even know when it's used, but in no way telling anybody to do this, but like all
our NPI numbers are all over the internet.
Like, I mean, I...
That's how I find mine.
I know, me too.
If somebody ever asked it for like discharge planning, I'll literally Google it and write
it out from that.
So it's not like going through a high level of security to get in, but I guess it's just
trying to emphasize the fact that this really is medical data.
And if you are a man on the street, then this is probably not the information that
you're trying to get.
It's not like they're trying to prevent people from getting it, but it's just not that practical
for people if you have no medical background at all.
That's correct.
This is really designed for people in the medical.
It's designed and intentionally designed just for people in the medical.
Okay.
So Open Evidence, and you literally just typed openevidence.com.
Yes.
Okay.
So now let's ask a medical question.
Should I start caffeine on a 30 week premature infant?
Okay.
Was it not happy with the relative clinical context?
Did it want to know more?
Oh, yes, it should.
Oh, it just gives you black or white.
Oh, yeah, yeah, yeah.
Yes, it should.
So this is a little bit more, remember, this is written by an AI.
Yeah.
No, I love it.
I totally start caffeine on this one.
Caffeine for all.
So would I.
Yeah.
But then it justifies it.
Yeah.
And it tells you why international and international guidelines for caffeine.
And then the nice thing about this, again,
standard dosing for the FDA label and the landmark CAP trial.
That's the trial that you and I knew was the landmark trial.
Exactly.
But you kind of have to dig for that information.
And that's great that it actually points out exactly where all the information is coming from.
Yes.
In each one.
And those are real links.
You know how in chat GPT sometimes it gives you fake.
Yes.
So these are obviously.
These are all real links.
And then it provides the references down below.
Okay.
And then, of course, just like any good chat bot, it says, hey, what else?
Yeah.
What else do you want to know?
Yeah, that's fantastic.
So that is.
Do you get a few free questions a month?
I think you get like several free questions a month.
And then is it like I know chat GPT is only like $8 a month to get more questions than that.
Do you know that?
So open evidence, I believe right now at the current time, once you log in is free for
physicians.
Oh, wow.
I don't know about anything else.
I know that now, obviously.
Yeah.
We're in a we're in a fun time where there's a lot of free opportunities.
I don't know if that's going to stay that way forever or if it's going to switch to
a subscription service or how that works.
But right now, at least it's free.
And I think it's it's it's actually in use by a large percentage of physicians already
have access to those.
All right.
I feel like the caffeine question was kind of more I mean, more kind of black or white
or just what if you have a situation where you're bedside and you have let's stay on
the apneas.
So you have a baby who is three days old and starts having apneas and you're it's three
o'clock in the morning and you're like delirious and you're trying to think, OK, am I missing
something?
Where would you go for that information to kind of like look through differential diagnoses?
Yeah.
So I think that either of these tools could potentially be used for that for for some
of that, at least as well.
For instance, in the apnea prematurity, this is this the first article that today pulled
up was the management.
But if you're a little bit more worried about the pathogenesis, sometimes going to the clinical
manifestations will give you kind of a differential diagnosis, clinical presentation, early or
episodic, late or recurrent apneas.
That's right.
So I guess at that point it is kind of going more through stuff.
And then a lot of times there's a differential diagnosis section.
So these are the other things that it could be.
OK.
What if you are in a hospital that you don't normally work in and they're like, we are
following the blank guidelines?
Yes.
So how do you get access to those guidelines?
That's a that's a great question as well.
So there are several different sets of guidelines that we can talk about.
So the kind of the highest level are national guidelines.
So national guidelines are guidelines that are put out by national like actual governing
bodies of, you know, so for instance, the United States, the FDA has very strict guidelines
on, you know, kind of you should use this medicine for this indication.
Now, one of the problems in pediatrics is almost 70 percent of the medicines we use,
at least in the NICU, are off label.
Right.
So that doesn't really help us a lot of times, at least from a drug perspective.
Right under those kind of nationalized kind of governing body guidelines, so like CDC,
FDA are guidelines by organizations that are kind of physician organizations that have
tried to collect collect the medical literature and try to define those.
So in apnea, there are great guidelines from the American Academy of Pediatrics.
Right.
So does everybody have access to those?
So can you just look up?
Let's let's see if we can.
The best way to find those is a lot of times there's a link to those in up to date.
These are society guidelines.
And in this case, if we click on this, it'll give us the Canadian ones.
And the United States, European ones, the United Kingdom ones, and even in India.
That's great.
And so to answer your specific question, oftentimes, let's see where the AAP
apnea one is.
It depends on your hospital system, whether you have access to this.
I do feel like a lot of the policy statements released by the AAP
in pediatrics are they are open access.
And I love the AAP does that.
So like a lot of, you know, obviously for this one as well, like I think when it has
a checkmark next to it, it means that it is free.
Right.
Like that anybody can look at it.
Yes.
So so you actually bring up a really important point about access, which is that there are
some journal articles and some journals themselves that are what they called open access.
Yeah.
So that means anyone can download the article.
Then there are other journals that are subscription only.
Yeah.
And, you know, there's some combination of two.
So, for instance, everyone knows about like Journal of the American Medical Association
or JAMA.
That's like one of the kind of premier journals.
Yeah.
JAMA Now is an offshoot that's called JAMA Network Open Access, where every single article
is accessible.
The PDF is accessible to everyone.
You don't need a subscription.
Are authors paying for that?
Who pays for that?
Because I've been asked in the past, do you want to pay like three thousand dollars to
make this open access?
And I'm like, I've already done all this research.
Like put it out there for free, dude.
This is an area that that is frustrating for me.
Yeah.
Because when we talked about peer review, what we didn't talk about is who got paid
and how that happens.
Well, peer reviewers don't get paid.
Peer reviewers do not get paid.
It is a pretty thankless job.
Thankless volunteer position.
Not only do peer reviewers not get paid, but in most high quality medical journals, the
authors do not get paid either.
So when you or I submitted an article to a journal, recently we submitted a journal article
to American Journal of Perinatology that was accepted.
That article, we did all the work for.
Right.
We submitted it.
We did not get paid for that.
Nope.
And then the journal sent it off to someone to review it who did not get paid for it.
Right.
And then once it was accepted for publication, they asked us if we wanted to pay additional
money to make it available to everyone.
Right.
Yeah.
So that doesn't feel great.
No, it's a very strange system.
Yeah.
And there is really only a few publishing houses that kind of house all this medical
literature.
But it is nice, though, that now that we have this open evidence that even though you might
not have direct access to that journal article, you have access to that being reviewed within
all the other articles when they're going into something.
Absolutely.
So it's great because there are so many articles that we're not having access to.
Absolutely.
Okay.
All right.
So we've been through open evidence and up to date.
What are other websites that you commonly use to get answers, whether it's from a
bedside standpoint or even like from a more kind of like academic understanding standpoint
or even research?
Yeah.
So the most important website in the medical literature is called PubMed.
And PubMed is maintained by the National Library of Medicine, United States.
It's just pubmed.ncbi.nml.
But you can also just Google PubMed.
Right.
And this is completely free.
Everybody can use this.
It is 100% free.
It is an index of most of the leading medical journals in the world.
Yeah.
And what it does is it index each of those medical journals, every single article.
In many cases, it gives you the abstract of the article.
The abstract is the short summary of the article that the authors write to kind of tell you what
is in the article.
And almost all of those abstracts are freely accessible.
Yeah.
So one of the best ways I find to access the medical literature is you go to PubMed and
you almost use this like a Google search.
Yeah.
You can just type in, in this case, caffeine premature.
Okay.
Press that.
Press search right there because I'm interested in something.
Okay.
This is the interesting thing about this, right?
Okay.
So the first one that comes up is in current neuropharmacology, right?
The second one is the one that we were talking about, the huge article in New England Journal
by Dr. Schmidt that came out in 2006.
I feel like sometimes the articles that you really want are just tucked in there.
And I have difficulties with my keyword search with PubMed.
When I know the article that I'm looking for, I think it's fine.
But sometimes when you want more of a kind of like, there could be something huge that
came out in New England Journal that you don't know specifically to look for.
And it might be like on the second page of research results.
Do you have any advice about that?
This is, I think, one of the most challenging parts of PubMed.
It's a little bit like Google search where sometimes the best link that you're looking
for is five pages in your Google search after the ad support links that start off with.
There are a couple of ways I think that I use to navigate that.
The first is that if you start with the review literature, so those are review articles,
things like up-to-date open evidence, they generally will be referencing the same papers.
So then you know that those are the important papers.
There are also some tools that are part of searching in PubMed that are helpful for that.
So let's take you through just a few of those right now.
So the first is, of course, an advanced search where you can add different things.
Obviously, you can add, and it'll tell you how many additional, you know, kind of
results you're going to get.
And that's one way to iterate.
So caffeine, premature, you could put in apnea.
It would probably limit the number of articles even more.
The question you're asking, though, is like, what are the most important articles?
Yes.
It's hard to really find that information.
So you use like the other, the up-to-date or whatever to like get that.
That's right.
Is there a, I think there is, but I haven't actually done this.
Is there a way to set up PubMed that every single time there is an important article coming out
on syphilis or whatever, it sends you the information?
Absolutely.
So those are these create alerts.
Okay.
So I can put that in and I'm like, I want to know everything on apnea prematurity forever.
Yes.
And it will send that to my email.
It will send you a link every time there's a new journal article that's indexed in PubMed.
That's fantastic because I also worry about that.
Like, especially if you're doing something for research, it's six months later,
up-to-date maybe was updated a year ago, which is still pretty up-to-date,
but then there's like a year of possible something huge came out in some sort of,
okay, that's great.
The other thing you can do, one of the other tools is the sorting tools.
Okay.
So you've got best match, most recent.
Okay.
And then publication date.
So sometimes when I'm trying to make sure that I'm up-to-date with the literature.
Yeah.
I'll type in what I'm interested in and I'll do a sort by the most recent papers and then just go
back.
Okay.
All right.
There are another couple of things that I think are pretty helpful with this.
One of the things that I find when I was starting out that was really helpful is using some of
these filters on the left-hand side.
So, you know, obviously if you can limit this to things that have an abstract,
that gives you more information.
So just as an example, when you go to this abstract, it tells you more than just the
title.
It actually gives you the background, how they did the study, what the full results
are and their conclusion.
Yeah.
So a lot of times if you're looking for a specific medical journal, you start with the
abstract.
Yeah.
In fact, and we've had this conversation a lot.
I suspect that 90% of the time when you publish a journal in an article or rather 90% of the
people are only reading the abstract.
They're not necessarily getting into that.
Yeah.
Maybe not even reading the entire abstract.
And I know we've had that conversation.
Even some when people are watching YouTube videos, they're not always watching the entire
YouTube video.
Definitely not.
I can tell you that flat out.
I can see the retention curves fall off.
Yeah.
So in addition to the abstract, there are all these.
So we've now sorted this by only articles that have abstracts, but then you can also
do things like a review article.
Yes.
And what this will pull up is articles that have, that are, that have tried to collect
all the medical literature and are primarily editorial in nature.
So a lot of times these are not necessarily peer reviewed.
So for instance, and then you can actually even do a further narrowing, which is a free
full text.
Okay.
So in this case, we have this article, which was published in 2019, which is a full text
that's free to access about caffeine and clinical outcomes in premature neonates.
Okay.
So let me just ask you, are you like snobbish to use another word about which journals the
review articles are published in?
The answer to that is yes and no.
Okay.
I think that there are clearly some journals that are very high quality.
Yeah.
I don't think, however, there is a clear relation between the highest quality, like the hardest
journals to get into and how good the peer review is.
Like a New England Journal.
Right.
You know, JAMA, like even pediatrics is probably in that scale for neonatology.
Right.
Yeah.
So, so those, so those are kind of landmark flagship journals.
So those are like New England Journal of Medicine, JAMA, Lancet, things like that.
If we're talking about not neonatology, things like science, those are like the really, really
difficult journals to get into.
What makes them difficult is not just the peer review, it's that they have an audience
that really demands really innovative, really important, critically important results.
Yeah.
So sometimes you have something that's fascinating, or at least fascinating to me because I'm
a nerd about kind of this stuff, but it won't be in one of those high power journals just
because it doesn't have quite the same draw.
Right.
That doesn't mean it's a bad study.
Yeah.
It doesn't mean that it was a poorly done study.
It just means the audience for that study may not be the New England Journal of Medicine
audience.
There is a, so you have like kind of the flagship journals and then there's a second tier,
which are things that I think was where most of the neonatology literature is honestly.
So those are journals like Pediatrics, Journal of Pediatrics, Journal of Perinatology,
American Journal of Perinatology, and that kind of group, Society for Pediatric Research
puts one out that's called Pediatric Research.
There's JAMA Pediatrics.
Those are all excellent, excellent journals, generally very high quality peer review.
Those are studies that I would trust.
Okay.
Then there's kind of a middle ground of journals where sometimes the information is very good
and sometimes it's a little bit harder to assess.
Those are things, those are journals where most things that get submitted, you know,
kind of are published.
All published.
Children is one of those journals.
Sometimes it's, it's very high quality information I've published in that journal.
So I don't, I don't think that, you know, I don't think that it's bad, but you have to
have a better sense of knowing what the good stuff is and knowing what stuff is not as
high quality.
And then there are the journals that like on LinkedIn, the editors of the journals are
sending us emails like, you have to send me a paper right now that you've done studies
on and you have to pay me $500 and I will get it published.
So when Tal and I did a paper together, I didn't warn her that her inbox was going to
be flooded by these requests.
So sorry.
They call those predatory journals.
And what they are really designed to do is if you are a starting out researcher and you
want to get lots of publications, you can pay these journals and they will publish almost
anything.
Yeah.
And those journals, a lot of them are not indexed on PubMed actually.
Okay.
But some of them now are.
There are certain indexing requirements to be indexed in PubMed.
And it's a little bit of an art to figure out which of those journals exist.
And sometimes they have names that are very similar to the really good journals.
Like they're totally like they're the never England journal of pediatrics.
But can you also talk about the impact factor of journals?
Because that does give you an idea of like, I know that it doesn't correlate directly,
but what does that mean?
Yeah.
So I'm going to use this as an example.
Okay.
So the impact factor is calculated by the number of times a specific article after it's
been published gets cited in the medical literature.
Okay.
And it is generally a sign of how often other investigators are using your article when
they're writing their paper.
Right.
So the impact factor is dependent on the single article.
That's right.
So there is an impact factor for caffeine for apnea prematurity.
Okay.
There's also an impact ratio for actual journals.
That's right.
That's correct.
So that's based on the individual impact articles.
That's right.
And that's a little bit more challenging because some journals have lower impact factors compared
to other journals if you were to compare them.
So New England has a very high impact factor for the journal.
Because it's also not esoteric.
I mean, New England Journal is applicable to literally every form of medicine.
That is part of it.
But the other part of it is that New England Journal, everyone reads that.
And so having the same quality of article published in New England Journal versus maybe
not New England Journal is going to likely increase the impact factor of that article
in itself.
So for instance, I was very lucky a few years ago to publish an article in the New England
Journal of Medicine.
That is by far my most widely cited paper.
I think part of that is I thought it was a good paper.
But part of that legitimately is because it was in the New England Journal of Medicine.
Yeah.
Hey, how many review going back and forth with New England Journal did that take before
it was finally published?
That was a very long process.
Yeah, like several rounds and different experts.
I think maybe six or seven rounds.
Wow.
That's a lot.
That is a lot.
It was, I mean, I think that that was a very high degree of scrutiny.
Yeah.
And actually, like I said, I've only been through that process with New England once,
but it unquestionably improved the quality of the study that we did.
Well, what's so interesting is you said this to me when we were writing, and he's taught
me just so much about the whole research field, is that no matter how good you think you've
made it, no matter the level of reviews, you're still going to read your article and be like,
oh, I wish I didn't put that in.
Yeah.
Or I could have phrased this differently or whatever.
Like there really is no such thing as perfection.
There is not.
And I know this, and I still will look back at articles that I'm very proud of.
And when I read them again a year, I'm like, oh, man, I wish I hadn't written that way.
But then it's kind of memorialized, and you can't change it.
Forever.
Yeah, forever.
OK, going back to PubMed.
So one of the other ways that you can search for important articles is PubMed gives you
an abstract, but it also gives you some things that you might be interested in reading.
So a lot of times, it'll tell you that it'll show you other articles that have referenced
that paper or other articles that are similar.
So this is a little bit like a rabbit hole, that if you start with one paper that you
find is important, you can keep clicking on other papers that kind of were in the same
vein.
And actually, I found this very, very helpful, this kind of similar articles.
And then, of course, here you have these are the articles that cited that article, the
original article that you've got.
Which doesn't mean as much.
I mean, every single time somebody is going to write an article on caffeine, they're going
to cite that 2006 article.
So it doesn't tell you as much.
I'll tell you the other big way I use PubMed is actually using the way that it cites.
So it writes down the citation.
Can you pull that up?
So I use this actually a lot on YouTube when I'm telling people where I got the article.
So if you just click that, if you're writing a paper or something, then that is exactly
how you want to put the citation.
And everybody accepts that type of citation.
Do you remember years ago, every single journal wanted a different method of citation?
Like, that was just the end of me.
I think that's where I was like, I don't know if I can do this.
There are still journals that want specific ways of citing.
But the nice thing about PubMed is that this number is unique.
It's also in the URL here.
It is unique to this specific article.
OK.
So if you just remember that one number, you're always able to find that article.
I will commit that number to memory.
OK.
So is there anything else about PubMed?
There's not anything about PubMed.
But I did want to talk about other ways to find aggregate information.
So PubMed, like we talked about, you find summary information.
You can pull up full text.
You can pull up full text that are reviews.
There's also another category down here that are called systematic reviews.
Oh, I have not used that.
And this is a really important thing.
Systematic reviews are something very specific.
A systematic review, the primary people that do this is the Cochran Library.
Yeah.
So the Cochran Library, what they've decided is that the medical literature is so broad.
And you oftentimes have more than one study published about one topic.
So one of the ways that you try to understand the medical literature is what they do
is called a systematic review, where they take all of the published data.
That is, they evaluate it for the quality of the study.
And when you say they, again, it's a group of people.
That's correct.
OK.
Just like UpToDate, basically.
That's right.
Yeah.
And Cochran Library, just to say, it kind of is the same thing as UpToDate.
You do have to pay for access to it.
But again, a lot of hospitals generally have access to the Cochran Library as well.
That's correct.
And I think some of the articles are available for free.
A lot of them are available for free.
Yeah.
The Cochran Library is one of the higher quality ways to summarize data.
Indications for timing and initiation of caffeine in preterm infants.
OK.
That's kind of going back exactly to the original conversation.
That's right.
Like, when do we start that?
So this is on Cochran.
But Cochran also, it might have that specific topic or it might not.
That's right.
It's very hit or miss.
Yes, exactly.
You don't know exactly if it's going to answer the specific question.
Something esoteric or something, not just esoteric, something where there's not a lot
of data for.
Yeah.
And when I say not a lot of data, there are not a lot of published studies for.
Right.
Cochran isn't going to be able to review that.
Right.
Well, I feel like they do review it.
Well, sometimes they will tell you.
We found six articles and we have no idea.
Right.
We don't know.
That's right.
It actually is very helpful.
So Cochran is a study of studies.
So in this case, rather than saying we looked at patients, it said, hey, we searched these
different databases.
OK.
From up to this date, we only included randomized controlled trials
that had these things.
And these are the outcomes that we looked at.
And then they synthesized those.
They ended up with 11 randomized controlled trials and enrolling 774 infants.
And this is what it shows in terms of results.
So I think this is another place like, you know, we went to open evidence that should
this baby be started on caffeine?
This is another place that if Cochran has an answer, it's usually very, very high quality.
Yeah.
So is this what you would call a meta-analysis?
Yes.
So there are meta-analyses that Cochran does, and then there are also meta-analyses that
other people do that are published outside of Cochran.
This is where you start getting a little bit slippery because sometimes meta-analyses are
really high quality and sometimes they're very poor quality.
We will talk about that later.
But for now, generally, the meta-analyses done by Cochran are very high quality because
they're literally only choosing.
Do they only choose randomized controlled trials in their meta-analysis?
They always tell you.
OK.
They almost always use randomized controlled trials.
And then can you go down?
I love the way that Cochran presents the data as well.
So one of these things is something called a PICO statement.
Do you want to talk about this now?
Yeah, sure.
So a PICO statement or a PICO, P-I-C-O-T or P-I-C-O-S, is a framework for reading any
sort of paper.
OK.
And it really gets to the heart of evidence-based medicine.
So evidence-based medicine isn't just randomized trials.
That's the science of it.
But the art of it is applying that body of medical literature to the patient that you
are seeing.
OK.
And in order to do that, you have to understand, is this study applicable to my patient?
Yes.
Or is the study generalizable to my patient population?
And one of the ways, one framework for understanding that is this PICO framework.
And that stands, the P-I-C-O stands for the population of babies you're looking at,
the intervention, the comparison group.
Sometimes people call that the control group if you're not doing an intervention.
And then the outcome.
And it shows you all.
The nice thing about the PICO framework is that it's a nice way of reading any paper
that's a, even randomized trials, not randomized trials, it kind of tells you how close it
is to your baby that you're dealing with.
The last part of PICO is the T, which I think is really important, and that's timing.
OK.
Timing is important for a number of reasons.
Because outcomes like death, for instance, in the NICU are different than death at two
years.
Sometimes when we look at timing, like if you're looking for, in general medicine, the
risk of a heart attack after taking something like a statin, if you're looking at a study
that's five years, that's going to be different than a study that's 10 or 15 years, right?
I think what you're getting at was kind of some of the graphs and how they show you
how they've synthesized the information.
And usually there's a table that summarizes what they found.
But the really nice thing are these graphs.
Hopefully this one has it.
See if I can find it.
And look at that.
It goes through all the, I love that.
OK.
Yes, there we go.
So this is kind of what a meta-analysis will oftentimes look like.
And can you talk us through that?
What's it actually called that?
That type of graph?
Is that a forest plot?
Yes, it's a forest plot.
So a forest plot, what it does is it lists the studies that it's including.
It shows you the events in each study and the total number of patients.
And then it gives you an estimate for whether that treatment favors the intervention
or favors the alternative to intervention.
And in this case, this is really interesting.
There's only two trials that met its criteria.
There are very few patients in these trials.
And this was caffeine initiated within two hours versus caffeine initiated within 24 hours.
So this graph is very specifically for that question.
And they only found two trials.
That's right.
So on the left of the vertical line, basically,
if the entire shape and line is to the left of it, it means it's significant.
That's right.
Because it's not crossing over the confidence interval.
It can't cross this one.
For it to be 100% significant.
For it to be statistically significant.
Statistically, 100%, yeah.
So it looks like it's all leaning.
I know you're not going to like that.
It's all leaning towards favoring initiating caffeine within two hours
based on this very small study.
That's right.
And then the other thing that I love about this is that it's kind of weighted
for the number of patients involved, right?
That's right.
Obviously, if a study has 30 patients or a study has whatever it is, you know, 10 patients or.
So this study only had 10.
And so the box, the little blue box is very small.
This study had 30.
And so the box is bigger.
Then when you get to this diamond plot,
then that is, it is represented as a proportion of the two, the aggregate of the trials.
So it's like a weighted aggregate of the trials.
And I feel like for me, that's just such a great summary of what it's really showing you.
So even in that, even though it looks like it's favoring because it crosses that perpendicular
line, if you actually did the math behind this, it would not be statistically significant.
Yeah.
They show you the math right here.
So this is a risk ratio of 0.64.
So when, when the risk is less than one, that favors, whatever it is that you're doing.
Yeah.
If the risk is more than one, it favors the alternative.
Yeah.
In this case, a risk ratio that's less than one favors initiating caffeine as soon as
possible after birth versus waiting.
But you can see that there is this, these other numbers in the brackets,
that's the 95% confidence interval.
So people have heard of p-values, p-value.
We'll talk about that later.
We can talk about that later.
But essentially this confidence interval is how confident that the actual truth lies
somewhere.
And you're generally highly confident that the truth lies somewhere between these two
numbers.
But because the high end of this number crosses one, you can't say that it's statistically
significant.
Okay.
I love that.
So you would go, in what scenarios are you going to Cochrane?
So it's a little bit atypical to have a very specific question that is asked.
Yes, I agree.
You know, that's answered by Cochrane.
I find it helpful with things where you're trying to get a general sense of the entire
high quality medical literature.
Yeah.
So one of those things that's been in the news a lot is probiotics.
Okay.
And trying to understand whether probiotics reduce the risk of neck and sepsis and death
and stuff like that.
So I might look at probiotics in Cochrane.
Okay, let's do that.
Because while you're doing that, I will also say that I also really like Cochrane to figure
out if there really is any information.
Because it will flat out tell you, we've literally got four studies on this.
And you know how things happen in medicine.
People start talking about it and there are like waves of excitement about different subjects.
And you start to get this feeling that there's all this data published.
And then you go to Cochrane and you're like, there's a handful of trials with like a handful
of babies in this.
Absolutely.
So this is a great way of separating the hype from the reality.
Is there a robust medical literature to support something or not?
And you're right.
Cochrane is very good about saying, hey, we didn't find anything that was high quality.
So this is probiotics.
Probiotics is one of those ones that, you know, so in here we have 60 trials, including
11,000 infants.
So this is very specifically probiotics for preventing neck?
That's correct.
What was the title of this?
Yes.
Or just to prevent neck?
Probiotics prevent neck, yes.
Okay.
Very good.
And preterm or very low birth weight babies.
Okay.
So we've got our, you know, we've got our PICO statement.
So PICO, so let's go over that.
Infant, low birth weight or preterm.
Intervention is probiotics.
Comparison is placebo.
So whether that was, so they actually did give like normal saline or something.
They weren't just a control of nothing.
That's it.
So it's a great question.
Depends on the trial.
Yeah.
Right.
So, and they will go into that.
A true placebo is, is, is slightly different than.
Blinded.
Right.
When it's blinded.
So, you know, and we'll talk about that in a second, but obviously if it's blinded,
it means that the person doing the intervention, whether you're giving probiotics or giving the
placebo, you don't know what it is.
And if you're not doing anything, then it's not going to be a big stretch of the imagination.
If you're giving the baby something, you've got to assume it's the probiotics.
So, and then the outcomes and they just pick these for this.
So mortality, neck, sepsis and death.
Right.
Okay.
So let's go through this.
Cause I, I love Cochran.
I feel like it's so objective.
So, and this is, this is I love these diagrams.
It shows you what they started with.
This is probably too small to see, but it kind of shows you what they started.
You know, when they searched all the citations and why they excluded some.
And then we get to, this is a, a graphical representation of all the trials.
And it tells you whether they used a random sequence generation for randomization,
how they controlled.
These are very, very specific types of ways in which the trial was carried out.
So this is a good look at how detailed they get when they're evaluating.
This is the stuff that I will call Dr. Tolyar about and be like,
what do you think of this trial?
And he'll be like, well, the way that they randomized it wasn't adequate,
or it wasn't completely blinded, or maybe it was blinded when they were doing it,
but it wasn't blinded when they were looking at the results,
kind of the little details that I don't necessarily always think about.
So, and then, you know, we, we kind of get to that, that same.
Look at that.
Look how lovely that is.
And so, so there've been a, you know, a bunch of trials.
This is specifically.
To prevent neck.
To prevent.
Yeah, this one's for neck.
Oh, with a different bacteria though.
Yes, these are different.
Yeah.
So the one, this is one of the challenges.
Yes.
Different probiotics.
Okay.
So this one's bifidobacterium.
Yeah.
This is the, the one is bifidobacterium.
Look at the diamond.
It's clearly on the favor side, and I can't tell if it's actually crossing the line.
So if you look here.
It's 0.96.
So that is statistically significant that bifidobacterium in very low birth weight babies
prevents neck.
Right.
And so let's go down.
And so the next one is for lactobacillus.
Again, again, the diamond is to the left of the vertical line.
So favors giving the lactobacillus to prevent it.
Saccharomyces, which is a yeast.
Doesn't.
I love that.
No, I'm joking.
Are you anti-fungi?
I'm so terrified about fungi.
I know it's great.
Okay.
And then bacillus.
And then the combination of the bifidobacterium and lactobacillus.
These are when we start getting combination probiotics.
I mean.
So you can tell that still.
It's so many studies of each one.
So detailed.
Yes.
And then if you go all the way down, does it include absolutely any study?
So oftentimes that's what this one is right here.
Yes.
No, but that's plus.
Does that mean that they've given all three of the bacteria or does it mean.
I believe this is every baby in all of these studies.
Okay.
Collectively together.
Yeah.
5,000, 5,569.
Yeah.
And it shows that favors probiotics and preventing neck.
And one of the things we can talk about this now, or we can talk about it later.
Why this bar is so long in this bar is a lot.
Not quite as long.
Okay.
If you can say it in three sentences.
Yes.
So in the reason this bar is so big is because we have less.
We have more uncertainty about the data and we're about the findings and certainty in
data is a consequence of how big your sample size is.
Right.
People call this power analysis.
Okay.
So when you have more patients, you have greater ability to be confident in your results.
And so the reason that this is so much longer than this one really comes down to the difference in
the size of the trials.
Okay.
And the number of events in relation to the size of the trials.
Which by the time that you're actually doing the little diamond and accumulating all the data,
it's why it's not going to be as long because you've accumulated.
So generally the diamond itself, unless all the trials are like absolutely massive,
it's going to be a lot bigger.
That's right.
But like the, the thinner, the diamond, then the less variation.
The greater your certainty is that the truth lies somewhere in between.
Okay.
So for instance, this diamond, we don't know where the truth is,
but we know that it is statistically significant.
Yeah.
But this diamond is really narrow.
So we have a pretty good sense that the truth is, you know, it's between the edges here.
So it's like right there.
Yeah.
We have greater certainty.
FDA.
Okay.
Right.
What other, what other, before we finish, what other websites do you use?
Whether it's research or...
So there are, so these are ways to get summary data, but sometimes you need,
you know, so you can also go to organization, individual organizational pages.
So the AAP when they put out clinical reports is very helpful.
ACOG, which is the American College of Obstetrics and Gynecology,
also puts out, you know, kind of position papers and stuff like that.
Then there's Google.
Yeah.
Which I use.
I use.
And I think that Google is one of those things that if you know how to use it well.
Right.
It can be very valuable.
Yeah.
And I think what I found is that actually when I'm talking to family,
sometimes Google is really helpful because it allows me to really
find information that will be the most helpful to families.
And not just that, you also know exactly what you're going into when you Google something.
So if the parents are asking you about how worried they should be because
the baby just swabbed MRSA positive, like non-infected, but some hospitals swab for MRSA,
then you Google MRSA positivity.
You are going to know exactly what those parents are going to see as soon as they Google it,
because all the parents are going to be Googling it.
Yes.
So I think that's also important for us to know what they're seeing so that we can also
speak to that as well.
I think it's an under, it's an underappreciated part of communicating with families that
most families have done some research before they come in to talk to you.
I can't even remember the last family that hasn't, you know.
And, you know, now we're in the stage where some of that research is Googling some of its,
you know, reels or social media or TikTok or something like that.
Or AI.
Or AI, yeah.
I mean, certainly we have some family that just had a baby and, you know, I was like,
hey, if you have questions about your baby, let us know that this is kind of what we do.
And they're like, yeah, but we've been asking Chad GPT a lot of questions.
Yes, you have.
I'm like, I'm sure you have.
But please let us know if you, you know, let us know anyway, because we're not Chad GPT.
Sometimes Chad GPT doesn't tell you how certain it is about the answer that it gives you.
It definitely doesn't.
And Chad GPT still makes mistakes.
It does.
And it hallucinates.
It hallucinates.
And by the way, I don't know if it's learning that well.
Like, I still feel like it kind of makes the same mistakes that it's made three months ago.
So it's definitely better.
But like for pure medical knowledge, there's stuff that comes out that is still wrong.
It is a hundred percent that that's true.
Okay.
So we decided that we're going to Google.
What, uh, do you have any, um, causes of apnea in a newborn baby?
Because that's the type of, no, a parent wouldn't Google that though.
Let's try that.
Let's try to say my baby has apnea.
Yes.
Okay.
And this is not helpful.
Yeah.
And so it's going to sleep apnea.
Exactly.
Right.
So that isn't so.
But I will say that if I'm going to a parent of a newborn that they think they're having
apnea in the newborn nursery.
And now, I mean, I kind of didn't think of this before, but now when I'm admitting that
baby and I'm saying, these are the things that I'm worried about.
I'm going to bring up like sleep apnea because this is the first thing that they're going to
see.
That's right.
And I'm sure that's going to be a part of them.
That's like, well, what type of doctor do we have?
He didn't even talk about it.
Talha didn't even mention the sleep apnea, which is like the most important thing we
need to worry about.
Okay.
So what do you do when you're Googling to try to get better medical information?
If you are.
So just like everything else, a Google search, the more you put into it, the better the outcomes
you get.
So filtering is very helpful.
I found, particularly when I'm talking to families, that Googling from sites that are
either national organizations or academic institutions are generally very helpful.
So when you say academic institutions, like you'll go to the Stanford webpage or like
CHOP or whatever.
They've got great websites and Dell Children's has a great sleep apnea.
So, and then, you know, in this case, it's true.
We probably want to do premature.
Or even like a time baby, but whatever.
Some of this is very, very high, you know, what's new in apnea, but this is actually,
this is an ad for a course.
I love that.
Here, the UCSF one consensus guidelines for management of apnea prematurity.
They are on it.
So this is really helpful.
This is another place.
So UCSF, Stanford put a lot of their guidelines online.
Johns Hopkins has done it.
So a lot of these academic institutions, all these guidelines, again, a lot of times they're
not peer-reviewed in a traditional sense.
Externally.
They are sometimes internally peer-reviewed, but in general, the higher quality academic
centers, they're doing stuff that is based on the medical literature.
Right.
So one of the ways that I use Google that I find really helpful, just to use your previous
example, is that if I'm talking to a family about micronathia, so obviously.
Small chin.
It's a mandibular hypoplasia, a condition where the baby's long.
And to be fair, there are lots of medical terms.
When I see them, I still have to Google them to figure out what they are.
What I found very helpful for a lot of families is I'll do a Google search for whatever the
condition is.
And then if you go to images, it actually, a lot of times will bring up images that might
be helpful when you're talking to families.
So this could be a helpful image at explaining why if a baby has a small jaw, it might make
it harder for them to breathe.
And then you can go to that image and you can print it out.
And I know a lot of people are visual learners, and this is really helpful, I feel like, for
families.
Yeah.
And also just something very objective that they can compare to their baby, when this
is all so new and so unheard of in the NICU world.
So you use the images.
Is there anything that you try to stay away from on Google?
Like, I try to stay away from any, you know, adverts or private, like, then you're just,
you don't know.
That's not information if they're trying to sell their, you know, tongue-tie surgery or
not probably the best information.
No, I think it's fascinating and instructive that when we were looking for CHOP stuff,
one of the first, you know, one of the first things was a course, a medical course.
Right.
So I tend to really try to focus on high-quality source information.
So for me, that's academic institutions primarily or societies.
Okay.
There are now a lot of really great advocacy organizations, parent advocacy organizations.
So a couple of those are like Hope for HIE or NEC Society.
March of Dimes is another great site.
Those are great.
Those are great resources.
The problem is it's hard to distinguish a good, unbiased, high-quality disease interest
organization or patient advocacy organization from one that has like an agenda.
Yeah.
Right.
So there are all these other people that are, you know, in some cases sponsored by wellness
people that are trying to sell you a supplement that put up a really nice website that's like,
oh, your baby is perfect, except they have this, you know.
They desperately need red light therapy.
Right. Exactly.
Exactly.
They really need this, and everything that every doctor tells you is wrong, except for
they need this red light therapy.
And oh, we have a partner institution that will sell you this set up for $20,000.
That is a problem.
Okay.
Anything else?
Or can we kind of sum up the big website that you use?
I think that those are most of the websites that I use.
I do just want to say on Google, do you ever use like the professional tabs?
So if you go to Google and you go down, like put micrognathia under the all mode,
like not the AI, and then go all the way down to the bottom.
Like they, oh, no, they sometimes under one of those tabs,
it says micrognathia and it says prefer professionals.
And I feel like sometimes that tab will actually take you to,
oh, how to pronounce micrognathia.
That's also good sometimes.
It will take you to like actual articles that are involved in that.
Oh, there, they're all on the right.
Well, those are kind of like.
These are the ads.
Oh, those are.
Yeah.
Those are ads from Cleveland Clinic and CHOP.
Yeah.
Oh, they pay to be put there?
They pay to be put there.
How funny is that?
Well, because if you're a parent and your child has micrognathia and they're trying to recruit
families to come to their centers, that's why they put those there.
Yeah.
Well, I feel like normally there is a professional part on, oh, I love this thing.
It's so stressful.
And I'm like, is any of it a motorbike?
Like I completely panic with those.
I know everything about studies, but I cannot tell you if that's a motorbike or a picture.
I don't even know if I'm human, honestly.
Okay.
Let's summarize this quickly.
So the big websites that you use and give me just like a couple of words on each one of
when you use them.
So up to date and open evidence, I use when I want quick summaries of diseases or treatment
plans.
Okay.
When I want to do a more detailed dive, if I want to look in the primary literature,
I'll go to PubMed.
Sometimes if up to date or open evidence doesn't have a lot of information, PubMed for rare
diseases, for instance, PubMed is the right way to go.
Okay.
If I want to get a broader view of the literature, I'll look at the meta-analyses.
And then finally, Google sometimes is an adjuvant to all these things.
And when you say the meta-analyses, you mean you go to specifically Cochran?
Cochran primarily.
Okay.
But other people will do meta-analyses and those will be published in high quality journals
as well.
Okay.
Cochran are not the only people doing meta-analyses.
Okay.
Thank you so much.
That was so helpful.
I hope that you all learned something.
And honestly, he probably just scratched the surface of this.
But thank you again so much for being here.
If you have any questions, then please write, comment below and I'll get back to Dr. Toglia.
Thank you.