I found the aeticle in a post on the fediverse, and I can’t find it anymore.

The reaserchers asked a simple mathematical question to an LLM ( like 7+4) and then could see how internally it worked by finding similar paths, but nothing like performing mathematical reasoning, even if the final answer was correct.

Then they asked the LLM to explain how it found the result, what was it’s internal reasoning. The answer was detailed step by step mathematical logic, like a human explaining how to perform an addition.

This showed 2 things:

  • LLM don’t “know” how they work

  • the second answer was a rephrasing of original text used for training that explain how math works, so LLM just used that as an explanation

I think it was a very interesting an meaningful analysis

Can anyone help me find this?

EDIT: thanks to @theunknownmuncher @lemmy.world https://www.anthropic.com/research/tracing-thoughts-language-model its this one

EDIT2: I’m aware LLM dont “know” anything and don’t reason, and it’s exactly why I wanted to find the article. Some more details here: https://feddit.it/post/18191686/13815095

  • tal@lemmy.todayEnglish
    11·
    4 days ago

    Define “know”.

    • An LLM can have text describing how it works and be trained on that text and respond with an answer incorporating that.

    • LLMs have no intrinsic ability to “sense” what’s going on inside them, nor even a sense of time. It’s just not an input to their state. You can build neural-net-based systems that do have such an input, but ChatGPT or whatever isn’t that.

    • LLMs lack a lot of the mechanisms that I would call essential to be able to solve problems in a generalized way. While I think Dijkstra had a valid point:

      The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.

      …and we shouldn’t let our prejudices about how a mind “should” function internally cloud how we treat artificial intelligence…it’s also true that we can look at an LLM and say that it just fundamentally doesn’t have the ability to do a lot of things that a human-like mind can. An LLM is, at best, something like a small part of our mind. While extracting it and playing with it in isolation can produce some interesting results, there’s a lot that it can’t do on its own: it won’t, say, engage in goal-oriented behavior. Asking a chatbot questions that require introspection and insight on its part won’t yield interesting result, because it can’t really engage in introspection or insight to any meaningful degree. It has very little mutable state, unlike your mind.