GenAI is an averager (as all empirical models are), so it tends towards predicting the mean of whatever it’s trained on (in a given context).

Perhaps this sentence is not what you wanted then? The mean of whatever it’s trained on (that is, the training data) need not be the mean of the trained LLM function. This is because we use complicated loss functions to update our LLMs, stochastic processes in training, and that we change those functions after training on input data by using RLHF and other models (eg, via using a mixture of itself to get better outputs with the boosting algorithm).

You are right that once we fix an LLM’s parameters, at temp 0 they are deterministic, though they are recursive functions so it’s weird to talk about a long string of outputs at temp 0 as the ‘mean’ of the LLM; high temp outputs won’t ‘regress’ to this temp 0 output over time, for example. There also may be hallucinations or mistakes that are common at temp 0, but extremely rare at all other temperatures (because these mistakes only get made when it looks at its own temp 0 context). So I’m not sure the deterministic output string(s) are particularly useful.

(An analogy that I suspect isn’t very good: the butterfly effect, adding a single small air perturbation can lead to drastically different deterministic weather simulation results. This is very much the norm for LLM outputs)

(They are somewhat more than a neural network; the ‘attention’ layers add something pretty strange. And output temperature isn’t implemented as a noise term, but this is a fine way to think of it.)

(There is a separate claim about genAI producing the average/typical behavior from its inputs; this isn’t true for RLHF and boosting reasons, have a gander at boosting and wisdom of the crowds for how you can take a cheap source of mediocrity and create something much better. It is true that genAI outputs feel unoriginal and omnipresent, as so very many folks are throwing around genAI outputs everywhere. But I think that’s not a property of the statistics, rather of the economics/behavioral science.)

So I think you’ve landed on the problem that I have with student use, and that the article has with scientist default use. It is very bad for folks to use AI to try to do the knowledge work directly, yet people are making this mistake constantly. YSK: it’s much better when you make AI increase the friction of knowledge work. (note neither are claiming that this use is particularly good; we’re doing damage control.)

I definitely notice sloppy thinking when I see it in someone else’s review of my papers. You certainly pick it up when you read climate-change-denialists. Experts are great at noticing sloppy thinking when it disagrees with them. That is the relationship you want with genAI if you are using it for knowledge work. Make it disagree with you, so you notice where it is very sloppy (and sometimes where you’ve been sloppy because the bad thinker is kinda right).

As I’ve been messing around with it, I am becoming a fan of ‘hallucination -> new context’.

(Why is consciousness required to tell true/false statements? I think I’d even say ‘that sign is dishonest/wrong/a liar’, regardless of the fact that the sign is very much dead.)

Knowledge work is generally pretty expensive; it’s done in dense places, uses a lot of air travel, has pretty big long-term consequences. Even the high-end estimates on AI costs are probably worth modest gains for knowledge workers.

Idk about you, but my mentors and peers are all busy stressing over the polycrisis.

Could you flesh this out for me, I’m not sure I understand? I think you’re saying:

1. experts are not well posed to catch biases and untruths generated by genAI in their (research work)
2. because as an academic climate scientist, your day-to-day work is spent on new climate science (and not established climate science).

I don’t see why (2) implies (1), but I agree with (2). As I intended it, (1) is my main claim. I follow it up with

3. if an expert cannot spot a bias or flaw in genAI output, then they wouldn’t catch it from a peer either.

I don’t see how (2) helps with (3) either.

If you’re doing knowledge work, you are an expert. You’re very likely to catch biases and untruths (and those you don’t catch, humanity couldn’t catch. These concerns are just as applicable to a human by themselves.)

Thoughts on the study from the article? link

Sometimes knowledge work is about learning. That’s also a nice time to skip AI. But I can see from the data that student’s aren’t doing that, so we harm-reduce.