step 4: the filtering system (this is the game-changer)
the biggest problem with AI research tools is they give you the same insights over and over
"people want clear skin" - yeah, we know that already
you need a system that only shows you NEW insights
this is where embeddings come in
embeddings turn text into numbers that capture meaning
"I love food" and "I enjoy eating" use different words
but mean the same thing embeddings see they're similar because the numbers are nearly identical
step 5: duplicate detection process
when an AI finds a new insight, embeddings help narrow down your database to the top 5-10most similar existing insights
then an LLM compares the new insight to just those 5 and determines if it's truly new or a duplicate
this is way more accurate than asking AI to compare against your entire database
step 6: clean daily reporting
instead of drowning in repetitive information, you get reports with only genuinely new insights
each insight comes with actual quotes from Reddit users as supporting evidence
your research stays valuable instead of becoming noise
what this creates:
a clean, growing database of customer insights that doesn't get cluttered with duplicates
daily reports that actually teach you something new about your customers
real customer language you can use in ads and copy
to recap how to automate customer research with AI:
> target relevant subreddits where customers hang out
> scrape top daily threads automatically
> extract insights in specific categories with focused prompts
> use embeddings and LLMs to filter out duplicate insights
> get clean daily reports with only new findings
• • •
Missing some Tweet in this thread? You can try to
force a refresh
how to give an AI agent memory to make it smarter over time:
most AI system treats every conversation like it's meeting you for the first time
for example a customer first complains about a broken chair, then the AI responds like it's a new issue
but the same customer complains about another broken chair 2 months later
but in most cases an AI system still treats it like a new convo and acts like it's never happened before
what an AI with memory would do:
"Hello David, following up on your previous case from August where you experienced issues with an ergonomic chair, I sincerely apologize this is happening again. We appreciate your continued trust since 2023..."
it references past interactions and treats customers like actual humans