We’re seeing a lot of students, teachers, and scientists produce more, and worse, outputs because of improper AI use. It’s possible to use it better: Have it imitate a critic. Learning and creativity come from getting criticism and experiencing friction.
We found that when knowledge workers intentionally use AI to challenge their ideas – to generate friction – they can significantly improve their performance. Previous research on human-AI collaboration for loan evaluations had similar results.
A study I was not involved with found that creative writers produce better copy when they use AI as a sounding board rather than to ghostwrite.
Could you flesh this out for me, I’m not sure I understand? I think you’re saying:
I don’t see why (2) implies (1), but I agree with (2). As I intended it, (1) is my main claim. I follow it up with
I don’t see how (2) helps with (3) either.
GenAI is an averager (as all empirical models are), so it tends towards predicting the mean of whatever it’s trained on (in a given context). But it also has noise added, so some variance comes back, but there’s no guaranteed that that mean+noise produces something meaningful/true/valuable.
But, GenAI is very good at producing syntactically correct language. This is a problem because it lulls the reader into a sense that the author knows what it’s talking about, when it doesn’t. When a junior scientist produces text, the awkward language alerts the reviewer to the poor thinking (same with a junior coder producing weak code). With an LLM, you don’t get that - it produces the impression of knowledge without any actual understanding.
This, combined with the lure of efficiency and the feeling of effectiveness, make it a honey pot for quick but sloppy thinking. I think this is what connects 2 to 1, especially in domains where it’s hard to get external validation from other people who can understand what you’re trying to mean, and not just say.
As for your point 3: I agree. People don’t spot flaws already, and that’s part of why we saw the replication crisis in behavioural science. Adding LLMs to the mix will just make things like that more likely.
(There is a separate claim about genAI producing the average/typical behavior from its inputs; this isn’t true for RLHF and boosting reasons, have a gander at boosting and wisdom of the crowds for how you can take a cheap source of mediocrity and create something much better. It is true that genAI outputs feel unoriginal and omnipresent, as so very many folks are throwing around genAI outputs everywhere. But I think that’s not a property of the statistics, rather of the economics/behavioral science.)
So I think you’ve landed on the problem that I have with student use, and that the article has with scientist default use. It is very bad for folks to use AI to try to do the knowledge work directly, yet people are making this mistake constantly. YSK: it’s much better when you make AI increase the friction of knowledge work. (note neither are claiming that this use is particularly good; we’re doing damage control.)
I definitely notice sloppy thinking when I see it in someone else’s review of my papers. You certainly pick it up when you read climate-change-denialists. Experts are great at noticing sloppy thinking when it disagrees with them. That is the relationship you want with genAI if you are using it for knowledge work. Make it disagree with you, so you notice where it is very sloppy (and sometimes where you’ve been sloppy because the bad thinker is kinda right).