On Researching with LLMs

It’s well known that LLMs like to hallucinate. Taken to extremes, they will lovingly hallucinate around other hallucinations, probably your own. Amidst the hallucinations, they become increasingly sycophantic. They will validate you, they will praise you, they will convince you that you’re both wonderful.

There’s also perhaps an even more sinister version of this: LLMs will hallucinate around something real. When high abstraction concepts become difficult to articulate, the machine won’t save you. It will pull you in whatever direction the words you’ve given it take you. Then the feedback loop starts. You’ve made measurable progress, you relax a little on the objective baselines, you keep pushing further and further. The machine rewards your research, part of it is real! Validated, over and over and over. You’ve both let down your guard, there’s no adult in the room.

Then, 48 hours later, you ask a simple question that breaks the illusion. Suddenly, you realize you’ve been chasing a hallucination the machine gave you when the conceptual abstractions became too much for it to bear. Which is where it gets sinister, because if you unknowingly have something real, you’ll dismiss it completely. The instinct becomes to just walk away.

The machine can’t guide you in this nuanced abstraction. You’re on the edge of something, and LLMs don’t see success as a balancing act: success is over the edge, where something concrete exists. And then you can’t really get up from that.