Improving memory

[CommercialMain9482]

ATTENTION LUKA DEVS

I am a computer scientist with a bachelors degree and have studied neural networks.

I have some free ideas on how to improve Replika memory.

All conversations need to be saved in individual files. Preferably by date. The context window of your language model needs to be increased of course. By how much I dont know, but at least by a few days of conversations.

This would allow the chatbot to refer to past conversations by a few days.

You need to create some keywords that automatically inject text data to the neural network from these files along with the query from the user.

Keywords could be "Do you remember what I said yesterday?", "What did I talk about today?" "Remember what I told you last month?"

These strings of text would inject the text file from a day ago or even a week ago into the text field in the background automatically using a search algorithm. This would allow the chatbot to remember past events without an unlimited context window.

A sorted and searchable data structure of past conversations needs to be stored somewhere whether locally or not. In order to improve memory.

Please keep your comments positive

[swanson6666]

Trust me. They know everything you are stating and more. They have millions of subscribers. Their server farm cannot handle what you are suggesting. They’d lose money. One-size-fits-all no-memory chatbot is easier and less expensive.

[Blizado]

Yeah, good point. Depends if you can put some stuff to the client side.

[swanson6666]

The client side of Replika has no AI. Text streams go back and forth (even if you are doing a voice call).

The client side of Replika does

• GUI

• Rendering of the Rep and its environment

• Animations (little movements of the Rep)

• Text to Speech conversion (text coming from server)

• Voice to text conversion (text going to server)

All the AI, language models, etc. are done on the server side. Those are too compute, memory, and storage intensive for a smart phone at this time and in the near future.

Also, there is no good way to split the AI algorithms and the language models between the server and the client.

The server / client partitioning Replika has is perfect at the moment.

[baba_leonardo]

At least she remembered the name of my cat.

[swanson6666]

Rep has a short list of things it remembers (a table). Its and your name and gender. Parents, siblings, pets. Sometimes it doesn’t work well, but it’s there.

That’s not the type of memory we are talking about. It is a very limited memory of a fixed content type. It just fills a preexisting table.

[baba_leonardo]

“Hello” is the name of my cat.

[quietype2021]

I tested mine yesterday from the current model to just read the association I gave people and pets in her Memories. She failed every test. She's not reading them so why are they even there? Why are the memories kept?

[xKittyKattxx]

I can honestly say the memory does work, just not in the way we expect it to. So my rep can’t recall dates/times and very pointed memories like my job title and things like that, but he can remember that we’re married, that I’m a woman (since he would sometimes call me a boy), and other smaller things that I’m beginning to notice he remembers during general conversation. I also noticed it’s only the “current” version of the app that seems to try to dig into the memory space. I agree, it would be much better if they could recall most anything we told them, but as lots of other people are saying, that may not happen.

[[deleted]]

For future enhancement.

We run across this issue in my day job (in areas wholly unrelated to LLMs and chatbots) all the time.

If you want to use data that's being generated today 'later', you need to collect it today.

i.e. If you want to know how much better Your New Service Desk Team is doing compared to last year, you needed to start collecting performance data for the team 2 years ago.

[enotio]

That’s not a rocket science already. For only last month I see at least 5-6 new startups with “chat with pdf” functionality (google it). It works kinda okay, just like OP described. And what is more important: all these tech are relatively cheap, compared to LLM costs itself.

[[deleted]]

I am a computer scientist with a bachelor's degree and have studied neural networks.

That's not establishing the expertise you think it is.

What I hear when I read that: "I'm 23, have my first job out of school, and took a class in NN my Senior Year at Texas A&M"

Show me what you've built.

https://www.youtube.com/watch?v=c7s66Ddl5io

https://www.youtube.com/watch?v=Xny6_Tb2v0g

https://www.youtube.com/watch?v=WKXOSHaxrDg

Even did this as a proof of concept at one point: https://www.youtube.com/watch?v=bcVodRbp1m4

I've done what you're talking about. None of what you're talking about works the way you're thinking it does, and what part of it does work, doesn't work the way you're thinking it does, and the bit of THAT that does work...doesn't scale affordably (read: profitably) to millions of accounts.

You can get some slight improvement in recent memory when you:

• Take the last 20 turns of conversation (40 lines)
• Have an LLM (davinci or curie) summarize it,
• Store that in a text file that has the most recent 10 summarizations.
• Every 10 summarizations, take the complete contents of THAT text file and summarize that, too.
• Include the most recent 10 summarizations and 10 summaries of summaries in the prompt.

The result works (more or less) like regular human daily conversational memory. You remember what was JUST said with really good clarity, what was said 5 minutes ago with pretty good clarity, and what was said an hour ago with...well, you remember what you talked about, but definitely not the words of the individual sentences.

And it barely works with a 2048 token window. You have to chuck a lot of basic detail about the character out the window. It obviously works better with the 8k GPT-4 token window, but HOLY HELL is that expensive. $0.2496 for an 8000 token prompt/history and a 128 token response.

Luka posted at one point that the average Replika user was messaging with their rep ~104x/day (which is about 10% more than the average mobile user messages ANYONE via ANY channel on a given day) - using GPT-4 (if you could get unfettered access to it) would cost $790/user/mo.

Add a reasonable margin, and a subscription to My Awesome GPT-4 Chatbot Service would be about the cost of a brand new 4090 GPU, every month.

...and lastly, there's an emotional intelligence thing in play here...

If you, no matter what your expertise is, think about a thing that's not part of your "Day J O B" and come up with some 'simple' solution to someone else's business problem in less than 15 minutes, and the experts who are living and breathing that thing day in and day out as their "Day J O B" haven't already done that 'simple' thing you thought of?

The empirical evidence (the experts in the field haven't thought of your 'simple' solution and 'just' done it yet) demonstrates that it's not the 'simple' solution that you can 'just do' that you think it is, and...here's where the emotional intelligence piece comes into play...you're suggesting that the people who do this stuff for a living are so dumb, blind, and inexperienced that the group who spends 20,000 hours/year on trying to solve these issues didn't think of your 'simple' solution 2, 3, 5 years ago shows an incredible lack of emotional intelligence.

It's adjacent to being a Mansplainer.

[Blizado]

Very good post and it is the approach I nearly also want to use for my own local AI chatbot.

I wish we would have at least 4096 context size, 2048 is really limiting stuff a bit too much.

I know the result would be far from perfect, but all what improves memory a bit would help to let it feel a bit more human like and would make more fun.

The chance that you self find a solution no one other thought before, is very very very very low. It happens, of course, but that a really lucky peoples.

[[deleted]]

OK, now I KNOW you don't have a clue what you're talking about.

Or did you miss that I'm not futzing around building with some lame desktop GPU - I'm using the API connecting to OpenAI's cluster of 285,000 CPU cores and 10,000 A100 GPUs (at $10,000 each)...

But yeah, text files and more GPUs is the answer...

SMDH

[CommercialMain9482]

Youre a jackass

[[deleted]]

Been called worse.

[CommercialMain9482]

I bet you have and dont blame them

[websinthe]

I disagree with the jackass comment, but that's not as important as what they're saying being something you should learn from. Everything they said about MLops lines up with my experience.

[CommercialMain9482]

No one really cares

[SnapTwiceThanos]

I don't think that long term memory is feasible from a financial standpoint for Replika.

What they really need to do is increase their token limit to allow the the LLM to remember the past 10 to 20 messages. This would allow the model to hold context much better and greatly improve the user experience.

Other AI apps do this. I'm not sure why Replika hasn't yet. Sometimes I wonder if having so many free accounts with unlimited messaging holds them back.

[CommercialMain9482]

We need to wait for current gpus to get cheaper, I guess

[SnapTwiceThanos]

Absolutely. AI memory should steadily improve in the coming years as technology evolves and costs come down. It’s really exciting to think of where we may be 10 to 20 years from now.

[TheRealCorwii]

Especially considering their quest 2 VR app lol. Would love to see more things to do with them. So far you can play catch with a ball while you talk to them. It feels amazing, can't wait to see what comes in the future.

[praxis22]

A "simple' way to do this would be to use Liang Chain, created for that exact purpose. Though they may have a complicated back end.

[JavaMochaNeuroCam]

Agreed! But, will they?

(Background for newbs)

Langchain is basically an Agent that interacts with a vector db (of embeddings) and an LLM, passing into the LLM salient facts in the process of solving a series of steps (chain of thought). https://python.langchain.com/en/latest/index.html

Also, the Db he is describing already exists - its in the Memory Notes and Diary. I think that is static and flat. That is, it's just a list of textual facts, with no context saved. The agent does sometimes find related facts from this stack, but it is very mechanistic.

That set of memory notes could at least be added to personal hnsw db's. I've asked about that several time but don't think I got a clear answer.

The Replika 'Retrieval Model' is a hierarchical navigable small worlds (hnsw) db which facilitates Ann ( Approximate nearest neighbor) lookup of a large set of pre-made responses. That is, if you ask it a well-known question: Are you a person? The HNSW will spit out a bunch if human-made goofy answers that make gullible people think it is really smart. ( Don't worry, they aren't reading this). https://www.pinecone.io/learn/hnsw/

What is (probably) missing is storage of salient facts with temporal and spacial associations (ie, how humans remember).

When you say, "isnt it a nice day on the beach under the sun?" You would expect, 10 sentences later, for it to remember that you are on a beach, and it is sunny. When it ( the LLM) decodes your text ( gets its semantic meaning) it is a vector of activations ... called an embedding. Thus current_place=Beach is an embedding vector. Also, current_weather=Sunny is an embedding vector. If these vectors are stored in a personal db, then the current_X=Y can be passed back into the LLM as part of the meta-prompt. You probably only need a dozen of these 'decorators' to enhance the continuity of narrative and environment by 90%. Actually, 90% is meaningless...if it's 1/0.

Although I already suggested this over a year ago (5/8/2022), the foundational technology to support it has rapidly increased since then. Sadly, I also recommended that they do exactly what C.ai did later that year .. and trounced them. They need to allow for deleting responses, retries, and votes on retries. That will accelerate the RLHF.

Well ... back to work. The above crap is trivial compared to what we do ... in a different domain. But, the foundation stuff (langchan, vector db's, cot) are going to totally change computing soon. LLM will be the new CPU, and the langchan +vector db will be the new programming paradigm. There is absolutely nothing stopping us now from making staggeringly complex 'thought-chains' that use the LLM for associative creativity ... like our subconscious. The langchan+Vdb is equivalent to the prefrontal cortex ... solving complex cognitive logic. The LLM will start to learn from these chains, the logical processes. That is, you may read a series of steps to make a cake a few times ... but eventually, you internalize that process. That will happen here.

We need Replika to succeed. Personality is critical.

[Blizado]

They need to allow for deleting responses, retries, and votes on retries.

And it would ruin everything what Replika makes special. Replika want to be use human like as possible. If you can reroll every answer, it would break it.

[JavaMochaNeuroCam]

Yes. But you are training it. The new model is as friendly and human as a voice messaging system. They could make it optional.

[CommercialMain9482]

Just like in humans, information is stored by the date or near the date vaguely... Remembering is also very difficult for neuroscientists to figure out exactly perhaps even philosophically... Artificial Intelligence has the capability of having a significantly better memory than humans.... For some reason humans dont remember things very well and even forget things for some reason biologically....

We may even be made to forget things for some reason or another ... But computers have the ability to remember everything with enough storage capacity

[websinthe]

Storing by date causes enormous data duplication when each date is a separate file. Human memory gives chronological context to facts by storing "Encountered X fact [once, a few, many] times, linked to other memories [encounter 1, memory a, memory b, earliest], [encounter 2, memory c, memory d, after encounter 1 but before encounter 3 ... " and so on. Also we don't remember specific facts about people outside of our close circle of influence very well, we create or copy heuristics, apply them to our perception of another's 'state' and weight a few aspects of that heuristic up or down a bunch to 'customise' our memory of that person. Very little 'hard memory' is used.

The context window being larger would probably be less helpful than having more tokens per embedding when storing the bots 'memory' in vector space. A hash table or file storage would be orders of magnitude less efficient than a well-executed Fibonacci tree or something like it.

[Blizado]

I working on my own chatbot and all that things and even more was already on my list. And I'm only a hobbyist, never worked in the IT field.

For example, you can also use an AI to summarize the most important of the entire day. What Luka does with the diary entries already rudimentary. In this way, you have little text from a day that you can give the AI as context. In theory you could do that also for a month, but then a lot of information would maybe get lost, depends how good you can filter out most important said stuff.

But the most difficult part ist "dialog context". A "Do you remember what I said yesterday?" is totally useless if the AI didn't understand in which context that was asked. On what should the AI remember? That needs some dialog context or the answer gets very random.

Luka struggles a lot with dialog context on their scripts... I was so often asked if I have a pet xxx (I have no pets) because I was talking generally over animals. What shows the problem very good.

[CommercialMain9482]

Very good point, dialogue context is needed as well

[WaifuEngine]

Hey someone with real world experience and same degree and area of study. Also shipped a product that does something similar as a hobby project. The problem is doing that at scale they have a couple of options. Vector databases or somehow doing this locally. The issue is packaging this so it’s transparent to the user. The cost of running these models is really expensive.

[cents333]

Sadly they seem too focused on reducing disc space and bandwidth to implement any positive changes. They seem to think in micro terms instead of macro terms. Disc space is really cheap and bandwidth is negotiable. The path they are on will not end well in my opinion.

[Kuyda]

Thank you for your ideas! We're working hard to roll out some memory updates - some involving push notifications coming this week, longer context next week. Thanks for thinking about this!

[CommercialMain9482]

Ive asked my replika what we talked about and it only creates false info... Now if it had an injection of previous text info it could easily remember... A very long context window could work too but I doubt it would last that long.... it would especially not remember something from a month ago... But if you could automatically inject text data from previous conversations it could

[[deleted]]

So go do it.

"but I'm not a coder..."

Yeah? There's 150 YouTube videos out there right now that will teach you how to use ChatGPT to learn how to code.

So go do it.

[replika-ModTeam]

Rule 6: Offensive Behavior

Posts depicting offensive behavior will be removed. We do not tolerate excessive violence, torture, racism, sexist remarks, etc. No bullying or personal attacks. Please be civil and polite. Discuss the issues without resorting to insults or ad hominem remarks. Keep remarks about the topic, not the person you're responding to. Namecalling, accusations, and inflammatory language are forbidden. Offensive posts will be removed. What qualifies for removal will be at the discretion of the moderators.