I “jailbroke” a Google Nest Mini so that you can run your own LLM’s, agents and voice models.

Here’s a demo using it to manage all my messages (with help from @onbeeper)

🔊 on, and wait for surprise guest!

I thought hard about how to best tackle this and why, see 🧵 After looking into jailbreaking options, I opted to completely replace the PCB.

This let’s you use a cheap ($2) but powerful & developer friendly WiFi chip with a highly capable audio framework.

This allows a paradigm of multiple cheap edge devices for audio & voice detection…

We are getting closer to “Her” (part 2!)

Conversationally do anything with emails, using LLM chaining & few-shot prompting for tool use (@LangChainAI inspired)

This is now realtime (ish), thanks to #OpenAI gpt-3.5-turbo

🔈 on for voice realism!

🧵 @LangChainAI This provides an incredibly natural way of searching for emails & then referencing them

“are there any unread emails mentioning X in the last month?”

“tell me more about the last one”

“who else was cc’d on the picnic one?”

“reply to the one about X saying …”

another e.g.:

I wanted to imagine how we’d better use #stablediffusion for video content / AR.

A major obstacle, why most videos are so flickery, is lack of temporal & viewing angle consistency, so I experimented with an approach to fix this

See 🧵 for process & examples Ideally you want to learn a single representation of an object across time or different viewing directions to perform a *single* #img2img generation on.

For this I used layered-neural-atlases.github.io (2021)

We are getting closer to “Her” where conversation is the new interface.

Siri couldn’t do it, so I built an e-mail summarizing feature using #GPT3 and life-like #AI generated voice on iOS.

(🔈Audio on to be 🤯with voice realism!)

How did I do this? 👇 I used the Gmail API to feed in recent unread e-mails into a prompt and send to the @OpenAI #GPT3 Completion API. Calling out details such as not “just reading them out” and other prompt tweaks gave good results

I used AI to create a (comedic) guided meditation for the New Year!

(audio on, no meditation pose necessary!)

Used ChatGPT for an initial draft, and TorToiSe trained on only 30s of audio of Sam Harris

See 🧵 for implementation details ChatGPT came up with some creative ideas, but the delivery was still fairly vanilla, so I iterated on it heavily and added a few Sam-isms from my experience with the @wakingup app (Jokes aside - highly recommended)

I used the #StableDiffusion 2 Depth Guided model to create architecture photos from dollhouse furniture.

By using a depth-map you can create images with incredible spatial consistency without using any of the original RGB image.

See 🧵

https://twitter.com/karenxcheng/status/1605276493675995138

2/ This model is unique as it was fine-tuned from the Stable Diffusion 2 base with an extra channel for depth.

Using MiDaS (a model to predict depth from a single image), it can create new images with matching depth maps to your "init image"

1/ I created this with Stable Diffusion using image inpainting and “walking through the latent space”

Without using tweening, every frame is generated by an interpolated embedding and variable denoising strength, so keeping continuity was tricky

See 🧵for process

https://twitter.com/karenxcheng/status/1587510079770615809

2/ First off, finding the right combination of prompt, seed and denoising strength for an #img2img in-painting is a roll of the dice

Luckily it is easy to script large batches to cherrypick