A system is a structured approach to solving problems or accomplishing tasks. It's a set of organized principles, methods, or procedures that guide how we think about and tackle challenges. In the context of AI and RAG applications, my system includes:
* A framework for evaluating different technologies and tools
* A decision-making process for prioritizing development efforts
* A methodology for diagnosing and improving application performance
* A set of standardized metrics and benchmarks for measuring success
The beauty of a system is that it provides consistency and repeatability. Instead of reinventing the wheel each time you face a similar problem, you have a trusted process to fall back on. This is especially valuable in the fast-paced, often uncertain world of AI development.
A good system doesn't constrain creativity or flexibility. Rather, it provides a foundation that allows you to be more efficient with routine tasks, freeing up mental energy for innovation and tackling unique challenges.
This is what I plan on teaching you in our course
Rather than implementing whatever is the hot blog post of the day, I'm going to share what i've learned from building search systems at large companies like Meta and Stitchfix.
Share anecdotes from my consulting and draw parallels to classic search problems.maven.com/applied-llms/r…
The benefit of having a system is that the system itself can be productized. We can examine it over time and make improvements. By making the steps explicit, we can allocate the right amount of time and people to each step. We can decide to focus on specific steps and make targeted improvements to the application and the process we implement to improve the application. This can determine how we plan our roadmap, pitch a client, or even run standups and product reviews.
A system also helps people stay on the same page, standardize communications, and spend more time doing what must be done.
If you think your team should benefit from a system to improve your ai applications, apply now!
We already have participants from Microsoft, Anthropic, Accenture, PwC, Adobe, Amazon, and more!
You're working on a new AI-powered RAG application, but the process is hectic. There are many competing priorities, and not enough development time. Even if you had the time for everything, you're unsure how to improve the system. You know that somewhere in this chaotic mix is "the right path" - a sequence of actions that results in the most growth in the least amount of time. However, you're lucky if you're even going in the right direction, as each day of work feels like another roll of the dice.
To build with new AI systems, you obviously need technical skills - that's the baseline. But what separates the successful from the unsuccessful is not technical, but rather the frameworks for decision-making and resource allocation. Knowing what's worth working on, how to prioritize, what tradeoffs are worth making, what metrics to look at, and what to ignore, etc.tome.app/fivesixseven/a…
If you don't have these skills, your success entirely depends on someone above you having them and telling you exactly what to work on. You know you have to improve and make it better, but that doesn't give you a plan you can execute day-to-day. Avoid wasting engineering cycles, losing customers, or worse, never shipping.
Fortunately, these skills are not a magical trait that you either have or don't. They are a separate skill distinct from the technical skills needed for building with AI systems, and many never have the chance to learn them. But you can learn them, just like anything else. As someone who's been building recommendation systems and working with machine learning models for the past seven years, my goal is to give you the skills you need to succeed.
After taking this course, you'll be that founder/engineer/product manager who:
- Knows how and exactly what to prioritize
- Can immediately boil every problem down into a few key metrics
- Can coldstart an evaluation pipeline for your RAG Application
- Understands the table stakes for lexical, semantic search and how to use rerankers
- Keeps bringing new product suggestions to meetings
- Knows exactly which tradeoffs are worth making, and by how much
- Can catch major problems before they even become problems (and be able to prove and measure the pre-emptive impact)
- Quickly gets buy-in from everyone on your team by always having the right data to back up your strategy
- Can manage up in larger company settings, giving perfect briefs to managers so they know exactly what resources to give you, and how to make the case for it to other decision-makers
- And much more... learning the full playbook that companies pay outside consultants (like me) tens of thousands to implement
once a week i tell a founder "stop trying to finetune models, and just go sell, use opus, use 4-turbo, and just raise prices, find value, go sell, and sell to rich people,
stop selling to developers, sell to capital allocators, and not wage workers. make your roadster, get the money, and make the model 3 afterwards.
how am i signing a 6 figure contract in the month as as a solo bootstrapped twitter influencer with a suspended business account and a with an open source library and you are not!?
just promise that the thing they buy will give them status.
your note taking app is about "being a better executor"
your meetings app is about "being a better sales person"
your rag app is about "being a better decision maker"
your diligence ai agent is about "avoiding profit erosion"
being a hard worker and a 'cracked engineer' gets you in the arena
it is just the price of admission.
but it is now what makes you win.
unless maybe you're top 1% but even the transformer authors are just raising money with nothing to sell and leaving their startup for a paycheck.
Introducing Instructor Hub in 150 lines of python code
1. uses raw.githubcontent as the backend to get version control and a cdn (serverless lol) 2. uses pytest-examples to lint and test every example (never merge bag code to hub) 3. you can view cookbooks from the cli and pull code directly to disk.
why?
This means that all the code you pull is linted and tested, and match up 1:1 to the documentation, everything in the hub.
Which means you own all of the code, no magic code, just python and pydantic, and openai.
Theres lots to do, but none of it is needed yet. once we get > 30 items we'll implement search.
trust , i've training models and deployed production applications that serve at >350M requests a day.
just need `pip install` and some good naming conventions
1. jinja - prompting frameworks 2. numpy - vector search 3. sqlite - evals, one row per exp 4. boto3 - data management, s3 and some folder structure
??? 5. google sheets ;) - experiment tracking w/ a link to the artifacts saved in S3/GCS.
Disagree?
I've been training models in @PyTorch , and deploying them via @FastAPI since the library came out!
we did large image classification tasks where the folder structure reflected class labels and had a config.json in each directory.
our early a/b tests exported to google sheets and we served similar item recommendations via numpy brute force 3M skus with 40 dimension per vector (umap over resnet and matrix factorization machines)
@PyTorch @FastAPI I have nothing to sell you and sometimes i tweet about applied machine learning with @arnaudai (ex google brain) so give us a follow!