The Surprising Truth About AI Investments! 95% of GenAI Pilots Failed

Silicon Valley Front Row
168 followers
MIT found that most generative AI projects didn't succeed as intended. This is due to zero return on investment, and resistance to 'friction.'
Chapters
Intro and haircuts
AI hairstylist and barber?
MIT study on GenAI pilot failure
Verification tax
Psychiatrist testimony on LLM's
Friction, time, and effort
David's wooden board and 'barriers to entry'
Comments from viewers
Questions for viewers
Link:
https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf
=========
David Lam and Steve Ispas on Silicon Valley Front Row (SVFR) podcast
#MIT #artificialintelligence #genai
I looked at the article and I was very surprised with all the hype, with all this excitement,
with all the investment. We talk about NVIDIA giving a hundred billion dollars to open AI.
With all of this, you would think that, you know, the success rate would be the opposite,
would be 95% success rate, not 5%. I would assume that some are getting negative results.
Welcome back to another episode of Silicon Valley Front Row. I'm David Lam,
and this is Steve Ispas. Hey, so for people who have been watching us,
they might notice that something is a little bit different about this episode.
Let's ask them if they figure it out, but maybe tell them anyway.
Okay. It is because we swapped sides. So this is the first episode where you're on the left side.
I'm on the right. Yeah. And also we both got haircuts recently. I got mine last night
and it's a home haircut. So Johnny, please don't zoom in too much on this haircut.
Yeah. And what about you? I just got a haircut like a couple of hours ago.
And there's a bit of a story behind it. I went on over the weekend to get a haircut and it was
taking forever. This lady that I always go to, you know, there's a guy, they're getting his haircut.
And then just when I thought he's done, he's going to, she's going to wash his hair. Right. So
I just left. I didn't have patience to wait anymore. Was it a walk-in or?
It was a walk-in. It was a walk-in. But then today in our office, we put the paper together,
the newspaper together on Wednesdays. There's one of the staff who's doing haircuts for 20 years.
He's an expert. So nice. So he did it, you know, 20 minutes done. Very convenient.
Yeah. It's nice. That's always good. You know, can you imagine AI barber?
You know, I was thinking about that. Then it's really funny because I saw a video on X about
someone who walks on this little, like a phone booth or, you know, like a little booth on the
street and they have really big hair. In this case, a girl, she has like long hair. She sticks
her head in and then a few seconds later she comes out and he has all braided, you know, and
the funniest thing is on the bottom. It says this video is AI generated, you know, so,
so it's not real, but yeah, that's, I guess it could happen. Yeah. I guess I could.
Would you use it? I don't think so. I wouldn't trust those scissors.
So, so what are we talking about today? Yeah. So today we're going to talk about
this article from, uh, based on an MIT study, it found that 95% of groups that use generative AI
failed, you know, like, what do you think about that? Yeah. I looked at the article and I was
very surprised. I mean, with all the hype, with all this excitement, although the investment,
we talk about NVIDIA giving a hundred billion dollars to open AI with all of this, you would
think that, you know, the success rate would be the opposite. It would be 95% success rate,
not 5%. Yep. And they have the graph right here out of the report that shows what's implemented.
And just to clarify on this one, uh, general purpose LLMs, they had 40% success. And by that
it's really just chat, GPT and PI co-pilot. That's what they mean that 40% of the people
end up using those tools. Um, even that I thought it would be higher than that, but it's, it's,
it's 40%. But when it comes to this more tasks specific, uh, gen AI, it's 5%. So
very surprising. I'll read a short quote here from the report. It says to be specific,
despite 30 to $40 billion in enterprise investment into gen AI, the report uncovered a surprising
result that 95% of the organizations are getting zero return. And I actually think they're more
polite than, than it is because I would assume that some are getting negative results in the
sense that they waste time because every time you have to implement something, you have to figure
it out, right? So they waste time. So they probably have negative returns, meaning they,
they wasted more time than, than the results they got. Interesting. So more on this, um,
report. I mean, what's the reason is it mainly the reason of wasting time that these businesses
or these groups failed? It sounds like they talk about friction and by friction, they really mean
resistance to a new thing. And I think it's more than that. I think they don't actually see the
value. They don't see the value. They don't think that it probably saves more time. It's probably
not worth learning how much you have to learn or implement to get, to get a better result or to,
to make up for the time you wasted in learning. And actually another reason that they mentioned
is what they call the verification tax. So what they mean by verification tax is that
what's taxing time, you know, verifying. So a lot of these tools are wrong. They give the wrong
result or they give the wrong explanation. So that's why a lot of the people use them. They
actually have to verify the time they verify actually takes longer than if they would have
done it by themselves. It's a double verification. Yeah. So, you know about that. So I was listening
to a U S house hearing where Congress members were hearing testimony from researchers from a
researcher and doctors. So this psychiatrist is saying that the large language models, sometimes
they're being used in research reports. But he's cautioning against that and making sure saying
that people should be the final reviewer to check over these things. I think that ties into what
you mentioned now more about this report by the MIT students. It's a 2025 report. And the study
was conducted from January through June this year, and they reviewed more than 300 AI initiatives,
conducted interviews with 52 organizations where they, and they surveyed senior leaders among
different sectors. So this is a pretty wide survey or study. One more thing I would like to add about
the students doing this report. I think that's a really good thing because students are usually
the ones who come out with new ideas. They're not as much concerned with being politically correct
or following a narrative. They feel that this is the right information. They're just going to do
it. They want to establish themselves. They're learning, they're young, right? So they want to
go ahead and run with it. I think that's fantastic because again, if this study was done by a think
tank that's being funded by an AI company or an AI consortium or someone related to AI,
I think the results would have been quite different, would have made it look a lot better.
And even this is looking, making, looking pretty good, but, but still the obvious result is that
they're not working 95% of the time and they have zero results. So students doing it, I think is
props to the students. Yeah. And another thing is from the report, I saw that part of the,
the consequences of avoiding friction is because after there's an error or so, some of the people,
the users don't try to fix it. Don't try to train or modify these, the AI and that added onto
this, the failure rate. Ah, good point. Good point. Let me read also another quote here from
the report. It is again, 60% of the organization evaluated tools, 20% reach pilot stage and just
5% reach production. Most failed due to brittle workflows, lack of contextual learning and
misalignment with day-to-day operations. Now this misalignment with day-to-day operations,
I think it's a polite way to say that they are either useless or they create more work than
it's actually being and ended up as a result. So basically the solution is worse than the problem.
So, or it takes more work to do the, you know, to do it through them than doing it yourself.
Good paradox. Yeah. So what have these initiatives or companies managed to train the AI models
and didn't avoid friction? What do you think the results would have been different? And if so,
like by how much? That's a good question. Probably probably would have been different,
but I think in the long run by doing that, they would have less productivity overall. And for so
many companies, it's not the hype or the, or the political consensus or the narrative that matters
is the bottom line, right? The P and L. And then you try hard to implement something and maybe
actually make it work in some ways. But if that all the process took longer or actually dropped
your P and L, you, you, you lost money or productivity because of that, then any CEO,
they have to be crazy to go with it. Now, if in the future that improves, maybe that's a different
story, but I tend to think that these companies are pretty smart and they know they realize that
it's not worth doing it. And it's not just for not teaching the tool the right reasons. It's
probably more of a productivity issue. It, it, it takes more effort to use it than not to use it.
That's what I would guess is the bottom line. It's almost like, like if a weird example,
so you have to do a calculation, right? On, you know, we want to do three digit number calculation.
So it takes a bit of time to do it on paper. So you say one person starts to calculate on paper,
the other ones use a calculator, but he doesn't have a calculator. So you have to go out to look
for one who has to go buy one. It takes him half an hour to come back with a calculator.
And this guy finished like in five minutes. So that's the kind of example.
Oh, gotcha. Yeah. And if they have their phone on them,
and stuff, right. But no, I didn't have a phone. Yeah. I see. So they're relying on a tool
and it's gotta be on. Right. So what if the app crashes the calculator app? Exactly. Have it.
But yeah, there is a learning process or if it's repetitive, if the function is repetitive,
like chat GPT, or these things, it's repetitive, because all you got to do is you type it in and
you get answers. Right. But if you get to do much more complicated steps, and then you did one,
but the next time you have to do something else, so then it's going to again, be a productivity.
This reminds me of a term that my brother used before called the barrier to entry.
Quick story about it's funny because he gifted me a wooden, a wooden charcuterie board. Okay. And
I use it to cut meat. Yeah. But it wasn't until like a year after he gifted it that I used it.
And the reason why was, I didn't have the right knife, the right oil, because a brand new wooden
board, apparently you have to oil it for a bit. So then it's to protect the board. Yes. So he told
me that when he gifted it to me, but I didn't get it until a year later. And I told him, Hey,
I finally got it. And he's like, Oh, wow. So I guess that's the barriers to entry. Right?
That's a very good example. Well, hopefully we can ask our audience what they think. Maybe we
can go to the questions. Why don't you go ahead? Yeah. So in every episode, we ask questions.
And so we want to read about two questions this episode from our viewers. All right. And then
we'll ask new questions for this episode. That's right. So actually, the first one is,
one of our popular episodes was the one about the environmental impact and water use with
Jim Beach, our very great guest. So someone responded in Spanish, actually, a whole paragraph,
but I translated it to English. Let me show it here. And we appreciate all the comments you have,
please keep them coming. And even if they're in different languages, we appreciate it even more.
Now, David, I do speak Spanish a little bit, but I think it's better to read it in English so
everyone can easily understand. Do you want to read the first sentence started off in Spanish?
Let me try.
Yeah, I think English is going to be a lot easier.
So that's from a person named, well, the username is numbers. It's 10-500-4099. So
mucho gracias. Thank you. So he said, or this person said, I always try to recycle.
I live near a recycling center and that helps a lot. I've tried to do the same with organic waste,
but the options where I live are very expensive. The immediate impact I see is on my family.
My younger children are more aware of what they consume and the waste that generates.
They opt for cardboard or glass packaging if they need to take things there. Having a recycling
center is also a great help. And many local people are joining in and bringing their recyclables.
The best impact I've seen is the increased awareness it's generated among people and
businesses nearby. I know it's limited to a few families and it's just what I observe,
but it's a big change. So thank you again for this comment. What do you think, Steve?
They have a great point. I think it always feels good to recycle as anyone, you know,
helps the environment, helps everything, but sometimes it's not easy and you have to have the
process, the right environment to do it. And to recycle water through a plant, it's not,
you know, it's not as easy as it sounds because in order to recycle it, you need a whole system
that brings it back in, which may be using a lot of power that you're trying to generate
in the first place. So the difficulty of recycling is it comes into place.
You know, I feel good when I try to recycle because I have a separate box in the kitchen.
There's a trash exactly. And then the recycling box. Very easy, right? Yeah. So, but apparently
you're so for food waste, like to go boxes or like soda cups, you have to remove the
residue. So if there's like oil or food in it, apparently it affects how the recycling process
goes. So, um, I try to rinse my stuff, but I found out like kind of just takes a longer process.
The second question from last time we're going to go from is from E hoops 31. She says, it looks
like a student. She says the water heats up, which is part of the reason it's bad to put it back into
the environment. Maybe, maybe not. We don't know. I think she's probably right. Why can't we use
that energy instead of throwing it out with the wastewater? I think it's an excellent question.
Again, it sounds like a student. Uh, and yes, I think in physics and probably in, in, in school,
you learn that if you have energy in, in heat, potentially there's a, there's a system, there's
a device that you can put it through and then create energy, take that energy from the heat.
And then you get cold water coming on the other side. And then you have both gains, right? You
is that to have that device, that system that creates transform this energy into useful energy,
heat into energy, it's, it's one very expensive and hard to implement on a large scale.
Cause imagine implementing this for the data centers for the AI or how, how much of a system,
how big do you need it to be right now? Can it be done? Maybe this will actually be something
interesting to, to find out if, if this would be implemented by Nvidia or open AI or whoever
runs the data centers, my guess would be that it costs them a lot more to do this than to get
fresh water or new water from the river or whatever they get it from.
But it'd be interesting to see, I think great questions. Please keep your questions coming.
We appreciate that. Uh, we try to get through as many as we can and to make it fun for you.
Now let's talk about the questions for this week. Yeah. All right. So, uh, first question is,
do you use AI or chat GPT in your daily life? Yes. We'd like to see how many of you use it
in daily life. And the second question is, would you use a more advanced AI technology
system other than chat GPT or open or co-pilot or whatever, meaning something to do your taxes,
something to do your haircut, you know, whatever was available,
would you venture into something like that? Please put your comments below.
