Thread by @yleflour

I already love #GPT4

I love it so much that I'm getting it to turn itself into a fully functional pair programming assistant inside of a #vscode extension

4 hours in, it’s already writing 90% of its own code

To do this, I'm teaching it a new language

1/? 👇

#GPT4 is surely amazing, but its top current limitation is interactivity. Out of the box, #chatgpt4 has no way of directly with outside systems, which is a drag

The thing is, you can actually teach it to

But first, let’s analyze the way we interact with the chat website

2/?

When asking for code or shell commands, #GPT4 always responds with a code block

There are a couple of ways I interact with those

1. For commands, select, copy, paste. Line by line

2. For files, I click “copy” button

3. For partial files: select, copy, paste

3/?

Learning #1: The code blocks are a good container

#GPT4 is still very talkative, so we can’t waste time analyzing the whole prompt. But code blocks are self-contained and usually focus only on code or commands

Should be great to get precise actions

4/?

Learning #2: The copy-paste tool is the quickest way to interact

It is just a single click, so almost like a “run” button. But I didn’t want the extra step of pasting.

So my first goal was to turn this “copy” into “run” as fast as possible

5/?

Let’s ask it to set up a NodeJS script that fetches the clipboard data every 500ms. If it changes, trigger a callback with its content

Copy, paste, edit

I now get the clipboard’s content, but we use the clipboard all the time, so it’s gibberish. How do we fix this? 🤔

6/?

We don’t have to get very far in order to achieve this. As devs, we solved this issue a long time ago.

As we can’t always rely on file extensions, most file types start with a header. This is why you can determine most file types just with the binary content.

7/?

So I need my code blocks to start with a header which will help me identify #GPT4 commands

Let’s use “// PAIRPROG <COMMAND> <PATH>”

It should be specific enough to avoid mistakes

8/?

So I'll define a set of commands and teach it through a carefully crafted prompt.

We will need WRITE, APPEND, MERGE_JSON, MERGE_YAML and EXECUTE Commands

Iterate a bit. Functional enough.

9/?

Now my #GPT4 chat renders actionable code blocks. It can:

- Tell me to execute commands
- Tell me to write in files

All with a code block

I can copy those blocks and read them easily

But still no magic… 😞

10/?

Fortunately, I now have all the tools to get #gpt4 to self program it’s own extension

👶🏻 “Write the parse function tests”

🤖 “Fine with me”

👶🏻 “First write a PARSE_COMMAND_REGEX for your Commands. I need to extract type, content and path”

🤖 “Sure thing”

Tests pass ✅

11/?

👶🏻 “Create executeShellCommand”

🤖 “Of course”

👶🏻 “Create executeWriteFile”

🤖 “You bet”

Let’s wire it (myself) in shell.ts which launches the clipboard watcher, parses the copied items, and calls the right execution method

pnpm start

Magic happens… 🪄

12/?

From now on it will code all its missing executions one by one, all I have to do is click copy, refactor a bit (thanks @GitHubCopilot and @p42ai) and test

4 hours in, nice 🔥

Cool cool cool, but not enough…

13/?

But wait up !!!

I’m not done, but I really want to talk about what I’m working on in a Twitter thread.

But a thread isn’t a good thread if it doesn’t end with a call to action 😮

I need to spin up a site fast to collect emails

14/?

I know how to buy my 1000th domain name in 10s but I need that sweet code

👶🏻 “I need a nextjs static site using tailwind to collect emails, it’ll use send in blue for form collection”

🤖 ”Let’s do a few things”

click, refactor, delete

pnpm dev

Good enough…

15/?

👶🏻 “This site needs a logo, get me the image prompt to generate it”

Wow, truly ugly results

👶🏻 “Just give me your version as a SVG file, I need a vector based logo anyway”

🤖 “Sure thing”

👶🏻 “Add it to the website”

Aaaah, perfect… 👌

16/?

Fill out a few forms, click a few buttons (including “copy”)…

Fill out more forms for DNS and whatnot

git push

🪄 pairprog.io 🪄

17/?

Wait. End of thread but no extension?

It's on it's way, so make sure to follow my #gpt4 adventures and subscribe to that mighty mailing list 😘

pairprog.io

I already have tons bonker ideas in mind for this and it w̶o̶n̶'̶t̶ will write itself 🚀

18/18

🏁 Let's reset the 🧵 counter and launch with part 2 of my #gpt4 adventures

Time to go back to work on the extension

👶🏻 Write a "pairProg: open chat" command to open a side webview

🤖 Sure

▶️ copy, run

👶🏻 Now use the #gpt4 chat url

Crap...

Yeah figures 🤦‍♂️

1/19

💡Learning #3: It's really nice to know that #gpt4 has a good understanding of software limitations. Sure, you can force it to bypass them but at least it "knows" its 💩

2/19

👶🏻 So just asking to be sure, cause I have my doubts, can we inject JS in an iframe ?

🤖 Well... Sorry to be the bearer some bad news

Well I know what my next prompt will be...

3/19

👶🏻 Okay let's spin up pairprog-chrome-extension from root

🤖 LOL

I'm definitely not waiting 2 hours...

4/19

Let's go back to the extension with GPT 3.5 then

✅ Get it up to date on the rules with the initial prompt

✅ Provide project's structure

✅ Go with the prompt

🤖 ...

Well, that's a shame...

5/19

💡Learning #4: I'm already feeling helpless if I can't keep on using #gpt4. And it's a weird feeling I hadn't had for a while

It's like going back to pure JS after working with @typescript for a long time

But worse

I'm still not waiting 1.8 hours, so, what should I do 🤔

6/19

☝️Actually GPT3.5 is not that bad but it needs way more context

I've been using a single #gpt4 chat for a couple of days, of course a brand new GPT3.5 chat needs to play catch up.

And it actually comes with the bonus of increased prompt parsing and response speed ⚡️

7/19

My guess is that the prompt and response speed should be taken into account based on the problem to solve

#gpt4: Incredibly good at complex tasks, requires basic context. But slow as hell

#gpt3: Very fast. But needs more context and to be trained with updated docs

8/19

Back to developing...

I'm adding commands left and right for faster interactions in the VSCode extension. GPT3 handles it quite well actually. Thankfully the "code" API hasn't changed much since #gpt3's training.

My initial prompt is now at it's 150th iteration

9/19

But how do you write a #gpt3 #gpt4 initial prompt you ask 🤨

Well, let me tell you about "standards" in #LeanEngineering...

For that, I have to thank @Michael_Balle, @regis_medina, @marekkalnik, and a lot of other people (can't tag them all)👇

10/19

A standard is about being able to self check and validate something you or someone else delivers

It consist of 3 core parts and an extra 1
🔸Why
🔸Checks
🔸Do not
🔸Example

11/19

🔸The "Why" is about purpose.

It should tell the person about what you're trying to achieve and why it's important

Prompt: "You can use Commands impact our project by writing code blocks in your response"

12/19

🔸The "Checks" are tools to verify the validity of your delivery.

Usually you leave out the precise "how", but with #gpt4 and #gp3 you have to be specific

Prompt: "The first line of any code block must ALWAYS be "// PAIRPROG <ACTION>" no matter the coding language."

13/19

🔸The "Do not" are mistakes to be avoided

In the case of #chatgpt, it's usually mistakes in it's interpretation of the rules. Instead of writing more "Check" a "Do not" is easier to verify

Prompt: "Do not a write a code block without a Command"

14/19

🔸The "example" speaks for itself

It's usually a bad practice as it gives out too much "how" and humans tend to take it as a strict instruction

But in the case of #gpt4 it's often required to improve it's understanding

I won't write the example here, you've seen plenty

15/19

A standard is a way to verify "the best level of quality we know of today"

Most important part is "today"

Which is why I update my initial prompt (mostly with "Do not"s) every time #chatgpt makes a mistake in rule application

Which is why I'm at my 158th iteration 😉

16/19

And let me tell you, both #gpt3 and #gpt4 understand standards really really well

And that's why this system will be a key part of my extension 💪

17/19

BTW if you want a good introduction to #LeanEngineering I advise you to dig into @regis_medina's "Learning to Scale" book

IMO it's the easiest way to get into Lean Engineering philosophy and tools

I use those everyday and it's likely to pop up again in later threads 🧵

18/19

Ok, I think it's enough for Part 2

Don't forget to follow me for part 3 and subscribe to pairprog.io to get news on my extension

See you later 😉

19/19