3. Project Update- Newsletter Agent Progress - SCAI017
Sloup
1 follower
9 Video Views·Sep 15, 2025
Nate Herk - AI Automation Society Plus\10. Client Builds (Real Projects)\4. Newsletter & Research Agent\3. Project Update- Newsletter Agent Progress - SCAI017
Okay, hey guys, just wanted to give a quick update
on this project.
This is sort of like the newsletter agent project.
And so far, all I've shown within here has been
the research part, the getting the data part.
So, I've shown those different flows
where we had to scrape websites
and then scrape those websites for article links
and then scrape those article links for content,
title, content, date, feed that into Airtable.
And then what I did right here was basically
just feeding the scrape results into Airtable.
There was like a couple hundred records,
so it was looping through, uploading them one by one
with the relevant metadata, which in this case right now,
we're just doing the URL and the date
so that every single article that we're feeding in
to Pinecone, obviously that's gonna get split up
into different vectors,
so four or five vectors per article.
But all of those vectors will have that same URL
and date associated with it
so that when the article is being generated,
it's able to associate every single piece of information
it's being sourced.
So the goal is, this newsletter is kind of a snippet
of a ton of information over the past month or so.
And then within each snippet, we're saying,
hey, here's the original source and here's the date
if you wanna go take a look at it.
So I wanted to show this
because I was having a lot of trouble initially
with the prompting and with being able to get the agent
to actually pull back the metadata.
There were a lot of issues with the way that the agent,
like when you look in the logs,
you can see what happens chronologically.
There was issues with the agent
talking to this article database,
where it was just saying,
hey, can you give me relevant information?
And it would just pull that back
and I couldn't get it to get the URL.
So I ended up just basically saying,
when you talk to the articles database,
you need to ask for all metadata and all the information
and then we'll figure out what to do with it.
What's going on here is there's an air table base
of different keywords that the client would end up
putting in different topics or keywords
or features or whatever.
And we can say, okay, let's make some articles
on these keywords and whatever.
Right now, I'm basically just saying,
create an article using information from the past month,
from here, and then tailor the article,
make it about a topic that you choose
from this keyword database basically.
So, oops, let's hit this off, we'll see what happens.
So like I said, it's gonna go here
and it's gonna grab a keyword
and then it's gonna go search through the articles database
to make that, the summary, the newsletter.
And the idea is that when it's asking the article database,
it's gonna get the URLs and the dates
because we said the past month,
so it has to filter through and make sure
it's only getting content from the past month.
Right now, it's basically summarizing all of the information
after it searched through the vector store
and now the agent has all this information
from the vector database, it has article summaries,
it has the dates of the articles and the URLs,
the original URLs.
So, okay, so here's what we get.
So it decided to do sustainability,
that's the topic that it ended up grabbing from the keywords.
So it did sustainability
and we've got different sections here,
so it only did three sections this time.
But as you can see,
it's linking to the original URLs right here
and let's see, 10-15.
So this one didn't work for the past month,
as you can see, that's October.
So gotta go in and fix some of that stuff,
but these ones filtered back right.
We need, like we were just getting really small snippets
and then you can actually, you know,
that probably didn't show anything
because Loom is only recording the current tab,
but it opens up the original links
if you wanna look at them.
And now let's see over here.
So it hit the scouting keywords
and it filters them all out, right?
And then from there, what we got is,
I'll choose sustainability as our keyword for the newsletter,
it's highly relevant, timely topic.
Now let's query the article database
for recent information on sustainability.
It hits the article database and it says,
find articles about sustainability from the past month,
return all metadata, including URLs,
publication dates, titles and authors.
So originally it was just saying this part,
like find articles about sustainability.
But now we have it basically saying,
okay, when you activate this tool,
you need to ask for all metadata every time.
So then it pulls back the articles that are relevant
and then also is pulling back those URLs as you can see,
which is good because now the agent itself
is going to have all this context to make some articles.
So here's the input being fed into the agent.
As you can see, this is like what's in the vector database.
So we have URL, article title, article date,
URL, article title, article date,
got all this information.
So it's looking at all of these vectors obviously
in order to make its summary.
And then finally we get the output right here.
So more prompting has to be done
as far as how many sections do we want?
How many paragraphs per section do we want?
Because these are a little short obviously.
But this is just kind of showing the initial hurdle
that I was having with getting the sources pulled in here.
Originally it was showing no sources
and then it was showing literally just like closed brackets
was saying source URL input here.
And then it was pulling like example.com
and it was just taking me to random websites.
And then I think Anthropic started pulling up
websites that weren't in our vector database
and was pulling in websites
that just were relevant to sustainability.
And I was like, oh my goodness, this is a headache.
But now I'm more on the right path, I think.
And now it just comes down to more of the iterating
and the refining and the re-prompting and testing
because as you can see,
we're getting some better stuff back here now.
Yeah, so I think I'll just pause there.
Don't want, didn't want this one to go too long.
I just wanted to show you guys
how I was working through a little issue
and now from here, trying to work on making it better.
So any questions, feel free to leave a comment down below
and we can chat about it or if you guys have any ideas,
please let me know.
So thanks guys.
