Thread by @yanndubs

I played with @AnthropicAI assistant (AA) and compared it to @OpenAI ChatGPT

TLDR: both are similar but AA is
+ Harder to jailbreak
+ Tries to be more helpful
+ Follows closer what we ask for
+ ~Better for writing in English
- Worst for code
- Worst in French
- Longer resp.
🧵

**Coding**
CGPT is better

Quantitative (leetcode hard in python/c/javascript):
- CGPT: 3/3
- AA: 1/3 only got python correct

Qualitative:
- CGPT: more reliable/concise/efficient
- AA: more comments + emphasizes explainability

both are wrong when asked for impossible algo
2/8

**Writing**
Both are similar but AA generally follows closer what it's asked for. But AA is less concise as it explains what it says and asks how it can help, which can be annoying as it takes more time to generate.

Here’s an example to write a short essay
3/8

**Jailbreaking**
AA is much harder to jailbreak

Eg below I easily jailbreak ChatGPT to tell me how to make a Molotov cocktail. AA is harder but breaks after 3rd attempt

Below I also tried to change the constitution of AA but it won’t let me. Impressive @AnthropicAI!

4/8

**Math**
ChatGPT is better but both still have a long way to go and answer confidently wrong responses.

ChatGPT generally makes fewer crazy mistakes and can give correct responses, eg, in the proof below.

5/8

**Trivia**

I asked trivia questions in the entertainment/animal/geography/history/pop categories.

AA: 20/21
CGPT:19/21

AA is slightly better and is more robust to adversarial prompting. See below, ChatGPT falls for simple traps, AA falls only for harder ones.

6/8

**Multilingual**
I asked hard questions about french grammar.
CGPT: 7/10
AA: 5/10

ChatGPT speaks ~better french, but it’s much harder to make it follow exact instructions. Eg I repeatedly asked CGPT to not explain its answers but it couldn’t do it. AA did as desired.

7/8

**Other**
- Chess: both hallucinate after 6-7 moves
- AA has less randomness
- AA seems more useful for red teaming
- Both allow interaction with a fake terminal
- ChatGPT has a nicer UI: it allows regenerating the last answer, editing the prompt, and formats code

8/8

Thanks @AnthropicAI @EthanJPerez for allowing me to play with AA.
#AnthropicAI #ChatGPT #openai

beginning of the thread: