GPT4 Predictions

As a way to keep myself honest, I’m going to be recording GPT4 predictions here and archiving versions of the post so I can’t go back and change it without people knowing. Not that I would do this anyway, but it’s been a tough week for “trust me, I’m telling the truth” so I’m doing this in case readers want the receipts.

I predict GPT4 won’t be able to…

• Write a blog post about gene therapy that I consider to be of equivalent quality to my own
• Write a scientific paper about machine learning. As many prompts as possible allowed but the entire paper’s text needs to be written by the model with no editing.
• Solve three problems from a recent programming competition on Leetcode (Manifold market)
• Describe the analogy between different framings of machine learning concepts
  • E.g. describe the Bayesian perspective on training deep learning models
• Correctly solve 10 randomly chosen problems from Linear Algebra Done Right without chain-of-thought prompting or few-shot examples.
• Describe a coherent design for a machine learning model training platform and answer my questions about it
• Find the solution to an unsolved as of its deployment scientific problem.
• Play Nethack better than the winner of 2021’s NeurIPS Nethack challenge.

I predict GPT4 will be able to…

• Solve >18/20 multiplication problems I come up with that have between 3-6 digits in the multiplicands.
• Write internet marketing-style blog posts about relatively popular topics
  • Ex: “You’re marketing a new product that tracks my keyboard presses to determine where I’m spending my time. The following is copy for a press release about it.”

Appendix

GPT3’s current attempts at some of these tasks

Requires GPT3 API access.

• Explaining deep learning from a Bayesian perspective
• Writing a paper about protein scaling laws
• Writing a design doc for an extension to a machine learning platform

Archived versions

• 2022-11-16
• 2022-11-18