thread on my process of getting GPT-3 to write a script to integrate #stablediffusion into google sheets, unlocking a function "stable()" that lets you gen images in sheets
GPT3 had already written an integration for itself for sheets, so I figured this should be doable, too
GPT-3 won't just mek app, it needs context
wanted to use @replicatehq's API for this, so as a first step I pasted their API documentation into GPT-3, along with: "The following is the documentation for calling the Replicate API:"
<FULL Documentation>
"Your task is to..."
"..write a Google Apps Script in JS code that allows a Google Sheets user to call this API using a function "stable(prompt)", where prompt passes on a string.
The following is needed to call the right model:
<curl to call right model>
Here is the App Script code:"
And the code GPT-3 came up with worked--ish...it sent the prompt to Replicate and Replicate created the right image, but the script would then try to use a function called "sheet.insertImage()" which didn't seem to work.
What I needed instead was the URL of the image..
So I wrote:
"Modify the code of our function in the following ways:
Get rid of all attempts to render an image. Delete all code that tries to use "insertImage". Then, modify the function to simply return the URL from "output" as a string into the currently active cell:"
Having learned *some* JS code from watching GPT-3 do it, I had the script write the output of what it was returning into my sheet and found it looked like this "JSON" (?) from the documentation (see image) - NOT the prediction, but some sort of status?
So this wasn't the actual prediction! I had to tell GPT-3 to get the actual prediction first, and then retrieve the URL from that.
I wrote: "modify the function so that it takes the returned "id" from this object to call the API with GET <info from documentation>..."
..."Rewrite the function so that it gets the prediction, calling it in line with the above information. Once the prediction has succeeded, simply return the 'output' string:"
and that worked!
gpt3 used a while loop to wait for the prediction to succeed & then returned the URL
and this is how i got a function stable(prompt) that would return a #stablediffusion image URL which I could then display within sheets with =IMAGE(URL)
funnily, by that point I had gained some understanding of what the code was actually doing and was able to modify it myself
• • •
Missing some Tweet in this thread? You can try to
force a refresh
if both are governed by a world building prompt, (establishing their lore & physics), you could easily bridge any asset...
"lore inpainting" > narrative APIs
so instead of establishing hard rules or standards around how assets & characters between different worlds would interact with one another, you just let a language model figure it out on the fly
build deterministic censoring rules on top of that
it all started with this MJ image, which instantly pulled me into a 70s sci-fi world - wtf is this place??
I had to go in 🫡
I "played" Midjourney to go a littler deeper into these mines - (many ppl use it like that, like a game!)
and damn, did I find some strange strange down there
a resource curse, of sorts?
I fired up DaVinci Resolve (free video editor) to create a trailer of "what I had seen"...
The experience of being able to "prompt a trailer into existence" on the fly while discovering this world myself was incredibly thrilling, a glimpse of the future
I began to wonder what others might find in this universe & set up Twitter community votes for the first "episode":