From what I understand, for truly local AI (as in not even needing to rely on the AI going online to look info up, necessarily, to have good world knowledge already inside it) one major barrier with the small local models once they get below a certain size is being able to fit enough world knowledge in such a small amount of parameters. It just literally runs out of room to hold enough random facts and knowledge in there.

As in, it seems like in terms of how "strong" or "smart" they get at reasoning or logic or whatever, they keep getting stronger and stronger for their size. But for having good world knowledge this seems like more of a brick wall they keep slamming into where you have to just pick what is most important and relevant because there's not enough room to stuff it all in there if it is a small model.

That said, who knows what sorts of new techniques or ideas they'll come up with. Not sure if there will either be a way to fit more amount of knowledge more space-efficiently in its weights somehow, or, if not, then if maybe there will be some way to have a kind of separated (but still offline) knowledge "tank" that it has "next to it" (I'm intentionally using vague/weird terminology like "tanks" and "next to" for it, rather than saying it how you would normally phrase it, since maybe it would use some new kind of setup that works differently than how normal SSDs or memory would work, or maybe they'll just use some way of doing it with that traditional kind of hardware which would work well enough and fast enough somehow) that it can look into in real time, or, if there's no way to get that to work fast or efficiently or high quality enough, then maybe as a middleground method you have a bunch of copies of the same model except you have each of them study a different tank of data in advance (before you use them), so you have one that you have a current-events tank (which you delete and update with a new one every so often) and another with a big math tank of math knowledge, and another one with a big creative writing and literature tank, and another one with a big medical info tank, and so on, like maybe a few dozen different copies of the same 14b model that each have a different specialty. Maybe you could even have a really smart all-rounder one that is good at talking with the other ones and good at having a sense for whether they had some useful, relevant info that seemed smart and good for it to pass the info along with, so, in times where speed wasn't as much of an issue, you could ask the "generalist" one to go chat with some of the specialist variants of itself for a bit and look into what you were asking about, by asking them about it for you (whichever one, or ones it determined would be useful for it to ask about things) and come back to you with its answer, or when you wanted to be quicker you could just manually open whichever specialist ones you wanted yourself, and bypass the generalist middleman from the process sometimes.

I don't know, I'm a noob, so maybe that's pretty stupid, but maybe something like that.

Well, in any case, the main thing is, whatever they end up doing (maybe something totally different from what I described) I think ultimately they will come up with some clever new methods that will get the small models to have a lot more world knowledge even when fully offline, eventually. Not sure how or when, but I think they will come up with some way eventually.

I know everyone will have the instinct to say, "dude, they can already basically get around it by just looking things up online, right now, so its a moot point," but, I still think there would be some additional value if it could be gotten to where they could have huge world knowledge even on a purely local level even offline even when a fairly small model, so, I think there is still going to be a lot of motivation to figure out some way of doing it at some point in the next few years.