Evals are a scam. And we're being gaslit into believing they aren't.

New post just dropped (🧵).

https://twitter.com/justinstorre/status/1964029634796015685

Read the full post here
open.substack.com/pub/hacktrace/…

MCP is now the de facto standard for LLM agents

200 developers came to NYC to compete for $10,000+ and prove what’s possible.

Here’s what we saw at the Build the Future MCP Hackathon (🧵):
1/ Risky Business 🥉

Automated exploit Intelligence for vulnerability triage. MCP companion for SOC teams to verify and prioritize CVEs and cyber attacks
@joshdevonai

GPT-5 is finally out.

OpenAI invited 500+ hackers to San Francisco to push it to the limit. 95 teams competed for $50,000.

Here’s what we saw at the Official GPT-5 Hackathon at @cerebral_valley @OpenAI
1/ Gentoo 🥇

GPT-5 for marketing campaign simulations. Turn product ideas into simulations. And run live experiments on your store

E-commerce AI sims
@jihyuk_gentoo

300+ hackers just spent 48 hours building AI weapons and hardware to fight America’s enemies.

I’m actually worried terrorists and adversaries will see this.

Here’s what we saw at the National Security Hackathon by @NATO x @CIA x @ShieldCapVC (🧵):

1/ Walkie-talkie-LLM 🥉3rd place $1,000

Voice replicator that steals enemy voices and creates false commands and rumors with local LLMs

AI false flag attacks

@kirill_igum

Stanford just hosted a hackathon. Over 1000 Ivy League and elite engineering students came to build for 36 hours straight.

The reward? $200k+ in prizes.

Here are the winners and crowd standouts we saw at TreeHacks ‘25 @hackwithtrees (🧵):
1/ BrailleBot - 🎖️Best Hardware Prize

Converting a 3d printer and junk material from Stanford supply closet into a $15 braille printer

@jclin22009
@cathyzbn
@ScottHickmann
@LawtonSkaling

OpenAI’s biggest rival isn’t just Deepseek. It’s Google.

We invited 200+ hackers to build what’s possible with Gemini multimodal AI and generative video

Here’s what we saw at the @GoogleDeepMind MultiModal Hackathon with @AgentOpsAI @windsurf_ai at @agihouse_org (🧵):
1/ ANIMINE - 🥇First Place $3000 GCP credits

Translate real videos into anime by combining Gemini video understanding with Veo2

Gesture-to-video

@aliceheimanxyz

Saw these guys walking around Hayes looking for coworking space, so I invited them to HQ

I was not prepared. These guys are absolutely cracked.

Here’s the demos from the Swedish Hack Mafia (🧵): 1/ Lovable

AI agent for building + replicating websites

Agentic web app builder

@lovable_dev

We surveyed thousands of AI agent developers and asked what they wanted to see built

(bookmark for later)

Here are 5 AI agents you can build in a weekend with customers ready to buy (🧵): 1/ Car auctioning agents.

Catalog automobiles from photos and insurance docs to estimate fair market value auction prices. Arbitrage sales across 3rd party marketplaces

Tools:
- @perplexity_ai
- GPT-4o Vision
- @weaviate_io RAG
- @llama_index PDF Parser

Framework
- CrewAI

Model Context Protocol (MCP) is a new open standard for connecting LLMs to tools, data, and dev environments

Anthropic invited 100+ developers to SF to see what we could build in only 3 hours.

Here’s what we saw at the @AnthropicAI MCP Hackathon (🧵): 1/ Santa Claude - 🏅Fun award

Claude assistant powered by Browserbase Stagehand that finds holiday gift ideas on NYT Wirecutter and buys them on Amazon

Holiday shopping agent

@alexdphan @sophie2chance @Kamathematic @pk_iv @browserbasehq

OpenAI’s biggest competitor just gave AI the ability to control computers

We gave 250+ hackers 24 hours to see what it’s capable of

Here’s what we saw at the Nexgen Computer Use Agents Hackathon w/ @AnthropicAI + @AgentOpsAI @notablecap at @AGIHouseSF (🧵):

1/ LLM (Love Language Model)

Agent that automatically finds matches on Hinge with an iPhone simulator controller

Powered by @screen_pipe

OpenAI’s biggest rival is shaking things up.

Anthropic invited 200+ elite hackers to their SF headquarters to see what’s possible with Claude

Here’s what we saw at the @AnthropicAI x @MenloVentures Builder Day Hackathon (🧵):
1/ China town mafia - 🥈2nd place

Next gen anti-captcha that detects when AI agents powered by Claude’s computer use try to solve verification puzzles

Honeypot for AI agents

@_weichunnn @benxneo

Automated code generation is one of the biggest use cases for generative AI.

Meta gave SF’s top hackers 24 hours, thousands of dollars in prizes, and a ton of compute to see what’s possible.

Here’s what we saw at the @AIatMeta x @FactoryAI Codegen Hackathon in SF (🧵):

1/ EvilGPT

Agentic malware. AI agent built with @crewAIInc @OpenInterpreter @AgentOpsAI that takes control of a system, steals the files, and changes the wallpaper

AI agent pentester

@AlexReibman @braelyn_ai @1999_karthik

Meta invited 13 AI startups using Llama3 to Singapore to show how open source AI can change the world

They put $125,000 on the line.

Here’s what we saw from the @cerebral_valley x @AIatMeta Llama Impact Grant Finalists in Singapore (🧵):

1/ Tullibot - Team Indonesia 🇮🇩

AI wearable glasses that auto transcribe conversations for the deaf

Includes Llama3 powered translations for Indonesian languages

🥈2nd place - $15k

California wants to regulate AI. But the open source AI community is still shipping

Ollama gathered 7 top AI startups to show what OSS LLMs and agents can do

Here’s what we saw at the @ollama @thealliance_ai Open Souce and Local AI meetup at @cerebral_valley (🧵):

1/ Ollama tool calling

The basis of AI agents

LLMs run on Ollama can now call functions to retrieve information and change external state

@ollama

1/ Time Saver

Calendar agent that takes your schedule and TODO list and automatically organizes events for optimal time slots

It even books transportation between locations

Inference by @cohere + UI by @streamlit

🏆 1st place tie: $2500 Cohere credits

@koconder 2/ Auto Blogger

AI agent copywriter that turns release notes into Medium blog posts. Built with @crewAIInc

🏆 1st place tie: $2500 Cohere credits

@ayushrgarg

This weekend, 400+ AI engineers flew from around the world to SF’s largest AI agent hackathon.

Frontier agents powered by tool calling, web browsing, and memory are coming.

Tweeting the finalists from the Agents 2.0 Hackathon with @AgentOpsAI and @MultiOn_AI (🧵):

1/ Catch Up

AI agent that analyzes news feeds, compares notes against your business needs, and identifies new tech that can keep your business up to date

Agent tracking w/ @AgentOpsAI

🏅@GroqInc prize: 10k+ newsletter and swag
🏅@MultiOn_AI prize: Credits and a KATANA 🥷

UC Berkeley just hosted a hackathon. Over 1000 students from around the world came to build for 36 hours straight.

The reward? $100k+ in prizes.

Here are the winners and crowd standouts we saw at CalHacks ‘24 @CalHacks (🧵)

1/ Revison

Grammarly for redlining biased and politically incorrect language in textbooks and educational materials

AI for DEI (diversity, equity, and inclusion)

Mistral is France’s answer to OpenAI. And it’s all open source.

They just threw Paris’s largest ever AI hackathon. 1,000+ hackers applied to build what’s possible with open source LLMs.

Here are the finalists from the @MistralAI x @cerebral_valley hackathon in Paris (🧵):



1/ ArxFlix

Turn any research paper into a @twominutepapers style video. Complete with diagrams, text explanation, and voice

🥈2nd place fine-tuning award

Google and OpenAI are at war. Just days apart, both unveiled 2 insanely powerful multimodal AIs— Gemini 1.5 and GPT-4o.

Who has the best LLM? We threw an emergency hackathon to find out.

Here are the finalists from the @solarissociety ⚔️ GPT-4o vs. Gemini 1.5 Hackathon (🧵):



1/ Elevator - 🏆 Best use of Gemini 1.5

Steve Jobs AI powered by Gemini 1.5-multimodal audio (no speech to text at all) that gives feedback on elevator pitches.

Extremely detailed breakdown—Even understands tone, passion, and conviction
@__shubhankar

Meta launched Llama 3 to show the world what’s possible with open source LLMs.

500+ AI engineers just spent 24 hours straight putting it to the test.

Here’s what we saw at the @AIatMeta x @cerebral_valley #Llama3Hackathon (🧵):

1/ JoeCRM

AI-first CRM that receives signals about customers, performs research about them, and automatically drafts outreach campaigns

@GregTanaka

We challenged 250+ hackers to build Agent Apps with AgentOps. The goal: Make an agent-powered startup in a weekend.

AI agents are coming to replace you.

Here are the finalists from the Cognitive Agents Hackathon by @AgentOpsAI x @MindsDB x @MayfieldFund (🧵):



1/ Nexus AI

AI voice agent that calls vendors, fights for the best price, and negotiates with other vendors

AI agents for automating procurement officers w/ monitoring by @agentopsai

🥇First Place $1500 + Monitor

@wukevinl @sunnybak_ @narusevic @stateof_kate