Joscha Bach
speaker
510 appearances
2 recordings
2 series
first heard Aug 2023
last heard 23 Jan
Joscha Bach’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Jan 2026 with 1.
Appearances
That's clearly something that our nervous systems can do to some degree. And there is a problem with these models being coherent. And I suspect that all these problems are solvable without a technological revolution. We don't need fundamentally new algorithms to change that.
For instance, you can enlarge the context window and thereby basically create working memory in which you train everything that happens during the day. And if that is not sufficient, you add a database and you write some clever mechanisms that the system learns to use to swap out in and out stuff from its prompt context.
And if that is not sufficient, if your database is full in the evening, overnight you just train. The system is going to sleep and dream and is going to train the stuff from its database into the Lauter model by fine-tuning it, building additional layers and so on. And then the next day it starts with a fresh database in the morning with fresh ice and has integrated all this stuff.
You know, when you talk to people and you have strong disagreements about something, which means that in their mind they have a faulty belief or you have a faulty belief with a lot of dependencies on it, very often you will not achieve agreement in one session. But you need to sleep about this once or multiple times before you have integrated all these necessary changes in your mind.
So maybe it's already somewhat similar.
Yeah. And of course, we can combine the language model with models that get coupled to reality in real time and can build multimodal model and bridge between vision models and language models and so on. So there is no reason to believe that the language models will necessarily run into some problem that will prevent them from becoming generally intelligent. But I don't know that.
It's just I don't see proof that they wouldn't. My issue is I don't like them. I think that they're inefficient. I think that they use way too much compute. I think that given the amazing hardware that we have, we could build something that is much more beautiful than our own mind. And this thing is not as beautiful as our own mind, despite being so much larger.
It's the only thing that works right now. So it's not the only game in town, but it's the only thing that has this utility with so much simplicity. There's a bunch of relatively simple algorithms that you can understand in relatively few weeks that can be scaled up massively.
Yeah, Claude Shannon, when you described chess, suggested that there are two main strategies in which you could play chess. One is that you are making a very complicated plan that reaches far into the future and you try not to make a mistake while enacting it. And this is basically the human strategy.
And the other strategy is that you are brute forcing your way to success, which means you make a tree of possible moves where you look at, in principle, every move that is open to you or the possible answers. And you try to make this as deeply as possible.
Of course, you optimize, you cut off trees that don't look very promising, and you use libraries of endgame and earlygame and so on to optimize this entire process. But this brute force strategy is how most of the chess programs were built. And this is how computers get better than humans at playing chess. And I look at the large language models, I feel that I'm observing the same thing.
It's basically the brute force strategy to thought by training the thing on pretty much the entire internet and then in the limit it gets coherent to a degree that approaches human coherence. And on a side effect, it's able to do things that no human could do. It's able to sift through massive amounts of text relatively quickly and summarize them quickly and never lapses in attention and
I still have the illusion that when I play with ChatGPT that it's in principle not doing anything that I could not do if I had Google at my disposal and I get all the resources from the internet and spend enough time on it. But the thing that I have an extremely...
autistic, stupid intern in a way that is extremely good at drudgery, and I can offload the drudgery to the degree that I'm able to automate the management of the intern, is something that is difficult for me to overhype at this point, because we have not yet started to scratch the surface of what's possible with this.
Yeah. What gets really interesting, I find, is when you turn it into a multi-agent system. So, for instance, you can get the system to generate a dialogue between a patient and a doctor very easily. But what's more interesting is you have one instance of CHAT-GPT that is the patient and you tell it in the prompt what kind of complicated syndrome it has.
And the other one is a therapist who doesn't know anything about this patient. And you just have these two instances battling it out and observe the psychiatrist or a psychologist trying to analyze the patient and trying to figure out what's wrong with the patient.
And if you try to take a very large problem, for instance, how to build a company, and you turn this into lots and lots of sub-problems, then often you can get to a level where the language model is able to solve this. What I also found interesting is, based on the observation that ChatGPT is pretty good at translating between programming languages,
But sometimes there's difficulty to write very long coherent algorithms that you need to co-write them with a human author. Why not design a language that is suitable for this? So some kind of pseudocode that is more relaxed than Python and that allows you to sometimes specify a problem vaguely in human terms and let ChatGPT take care of the rest.
And you can use ChatGPT to develop that syntax for it and develop new kinds of programming paradigms in this way. So we very soon get to the point where this question, age-old question for us computer scientists, what is the best programming language and can we write a better programming language now?
I think that almost every serious computer scientist goes through a phase like this in their life. This question is almost no longer relevant because what is different between the programming languages is not what they let the computer do, but what they let you think about what the computer should be doing.
Showing 281–300 of 510 · page 15 of 26
← Previous
Next →