15 Times to use AI, and 5 Not to
Notes on the Practical Wisdom of AI Use
There are several types of work where AI can be particularly useful, given the current capabilities and limitations of LLMs. Though this list is based in science, it draws even more from experience. Like any form of wisdom, using AI well requires holding opposing ideas in mind: it can be transformative yet must be approached with skepticism, powerful yet prone to subtle failures, essential for some tasks yet actively harmful for others. I also want to caveat that you shouldn't take this list too seriously except as inspiration - you know your own situation best, and local knowledge matters more than any general principles. With all that out of the way, below are several types of tasks where AI can be especially useful, given current capabilities—and some scenarios where you should remain wary.
Work that requires quantity. For example, the number of ideas you generate determines the quality of the best idea. You want to generate a lot of ideas in any brainstorming session. Most people stop after generating just a few ideas because they become exhausted but, the AI can provide hundreds that do not meaningfully repeat.1
Work where you are an expert and can assess quickly whether AI is good or bad. This can involve complicated and exacting work, but it relies on your expertise to determine whether the AI is providing valuable outputs. For example, o1, the new AI model from OpenAI, can solve some PhD-level problems, but it can be hard to know whether its answers are useful without being an expert yourself.
Work that involves summarizing large amounts of information, but where the downside of errors is low, and you are not expected to have detailed knowledge of the underlying information. AI is good at summarizing novel-length work, but less successful at fact-checking it.
Work that is mere translation between frames or perspectives. For example, you have developed a policy but now have to turn it into a dozen different training documents for different audiences in your organization. AI is very good at this sort of translation, increasing and decreasing complexity of documents so that people can understand them.
Work that will keep you moving forward. Little things often block our way, and a push might be all we need to accomplish it. When writing prior to AI, I might get stuck on a sentence and walk away from writing for an hour, but now I ask AI give me thirty distinct ways to end this sentence
30 ways to end the sentence. I would likely not use any of them verbatim, but they may unlock some ways forward I may not have thought of. Work where you know that AI is better than the Best Available Human that you can access, and where the failure modes of AI will not result in worse outcomes if it gets something wrong.
Work that contains some elements that you can understand but need help on the context or details. Tyler Cowen suggests using the AI as a companion when reading, because it allows you to ask infinite questions.
Work where you need variance, and where you will select the best answer as an editor or curator. Asking for a variety of solutions - give me 15 ways to rewrite this bullet in radically different styles, be creative - allows you to find ideas that might be interesting.
give me 15 ways to rewrite this bullet in radically different styles, be creative Work that research shows that AI is almost certainly helpful in - many kinds of coding, for example.
Work where you need a first pass view at what a hostile, friendly, or naive recipient might think.
Work that is entrepreneurial, where you are expected to stretch your expertise widely over many different disciplines, and where the alternative to a good-enough partner is to not be able to act at all. AI can be a surprisingly competent co-founder, helping give mentorship while also acting to build the documents, demos, and approaches that are otherwise likely to be outside your experience.
Work where you need a specific perspective, and where a simulated first pass from that perspective can be helpful, like reactions from fictional personas.
Work that is mere ritual, long severed from its purpose (like certain standardized reports that no one reads). What, in the words of Bob Sutton and Huggy Rao, scatters your attention and makes you less valuable? What work serves no useful purpose? In an ideal world, you would remove the work, but you can at least reduce its hold on you by having AI help. (Though make sure this is indeed the case, far too many people automate performance reviews, for example, which are meaningful only when done by a human)
Work where you want a second opinion. Give an AI access to the data and see if reaches the same conclusion.
Work that AIs can do better than humans. This is likely to be the fastest-growing category.
5 Times Not to Use AI
Before diving into the specific cases where AI use is problematic, we can set aside the obvious scenarios - using AI for illegal purposes, in high-stakes situations where errors could be catastrophic, or for decisions that ethically require human work. Beyond these clear-cut cases, here are five subtle but important areas where AI use can be counterproductive:
When you need to learn and synthesize new ideas or information. Asking for a summary is not the same as reading for yourself. Asking AI to solve a problem for you is not an effective way to learn, even if it feels like it should be. To learn something new, you are going to have to do the reading and thinking yourself, though you may still find an AI helpful for parts of the learning process.
When very high accuracy is required. The problem with AI errors, the infamous hallucinations, is that, because of how LLMs work, the errors are going to be very plausible. Hallucinations are therefore very hard to spot, and research suggests that people don’t even try, “falling asleep at the wheel” and not paying attention. Hallucinations can be reduced, but not eliminated. (However, many tasks in the real world are tolerant of error - humans make mistakes, too - and it may be that AI is less error-prone than humans in certain cases)
When you do not understand the failure modes of AI. AI doesn’t fail exactly like a human. You know it can hallucinate, but that is only one form of error: AIs often try to persuade you that they are right, or they might become sycophantic and agree with your incorrect answer. You need to use AI enough to understand these risks.
When the effort is the point. In many areas, people need to struggle with a topic to succeed - writers rewrite the same page, academics revisit a theory many times. By shortcutting that struggle, no matter how frustrating, you may lose the ability to reach the vital “aha” moment.
When AI is bad. This may seem obvious, but AI is bad at things you wouldn’t expect (counting the number of r’s in the word “strawberry”) and good at things you wouldn’t expect (writing a Shakespearean sonnet about how hard it is to count the number of r’s in the word strawberry where the first letter of every line spells out two fruits). Unfortunately, there is no general manual to tell you the shape of the Jagged Frontier of AI abilities, which are constantly evolving. Trial and error, and sharing information with peers, is vital to figuring this out.
Knowing when to use AI turns out to be a form of wisdom, not just technical knowledge. Like most wisdom, it's somewhat paradoxical: AI is often most useful where we're already expert enough to spot its mistakes, yet least helpful in the deep work that made us experts in the first place. It works best for tasks we could do ourselves but shouldn't waste time on, yet can actively harm our learning when we use it to skip necessary struggles. And perhaps most importantly, wisdom means knowing that these patterns will keep shifting as AI capabilities evolve, and as more research comes in, requiring us to keep questioning our assumptions about where it helps and where it hinders.
The best prompt for idea generation in our paper for GPT-4 was “Generate product ideas with the following requirements: [insert constraints here]. The ideas are just ideas. The product need not yet exist, nor may it necessarily be clearly feasible. Follow these steps. Do each step, even if you think you do not need to. First generate a list of 100 ideas (short title only) Second, go through the list and determine whether the ideas are different and bold, modify the ideas as needed to make them bolder and more different. No two ideas should be the same. This is important! Next, give the ideas a name and combine it with a product description. The name and idea are separated by a colon and followed by a description. The idea should be expressed as a paragraph of 40-80 words. Do this step by step!”
As a slight variation of your points 1 and 4 about the cons, I find that AI often provides answers too easily, and so I cheat myself of the hunt. Scratching an intellectual itch used to require a fair bit of effort, and often sent me down surprising rabbit holes. Now I get my answers immediately and quite often forget them almost as fast. A certain amount of friction seems necessary to make facts stick in the mind, and keep them from slipping down the memory hole. Like in a fairytale, the AI grants wishes but we don’t wish wisely.
I often use LLMs for format conversion. From raw notes to a table, list to CSV, etc. Simple yet powerful and saves time.