Building Memory Systems for Agents Across Channels

Last week we gave our two agents a shared memory.

Here is the problem we were actually trying to fix. A borrower talks to our WhatsApp agent and says okay, I will pay by the 25th. Two days later our voice agent calls the same person and opens with hello, am I speaking to so and so, can you confirm who you are, and by the way have you thought about clearing your dues. The borrower already told us all of this. They verified who they are, they made a promise, they picked a date. And the agent on the phone knows none of it, it starts from absolute zero.

This is because the two agents are amnesiac, and worse, they are amnesiac separately. The voice agent forgets, the WhatsApp agent forgets, and neither of them ever knew what the other one did in the first place. Every conversation, on every channel, starts from scratch. To the borrower it feels like talking to a company where the left hand has no idea what the right hand did yesterday, which is exactly the feeling you do not want when you are the one asking someone for money.

The reason it starts from scratch is not laziness, it is just how both agents were built. The voice agent is stateless, every call it wakes up at the opening and greets you like it has never met you. The WhatsApp agent resets to its first state too. And the only memory that existed at all was voice-only, a blob of collapsed prose capped at a couple dozen lines that got stuffed into the next prompt. That blob had two problems. It was prose, so it had no structure, you could not ask it a clean question like did this person already commit to a date, or did we already send them the payment link. And it was one directional, WhatsApp wrote nothing back into it. So half the conversations the company was having with a borrower left no trace the other half could use.

So the thing we wanted was simple to say and annoying to build. One memory per borrower, not per call and not per channel. Every conversation on every channel reads it before it starts and writes to it when it ends. And it should not just be a paragraph, it should actually know where the borrower is in their journey, so the next agent can pick up instead of restart.

One memory, one writer

The shape is one channel-neutral record per borrower.

Both agents feed it. The voice call already got summarized at the end, that summary existed, it just went nowhere useful. WhatsApp now gets its own summary too, written when the chat goes quiet or closes. And there is exactly one thing allowed to write the record, a single writer, so two channels closing at the same moment can never stomp on each other. The one cross-channel narrative you read at the top, the story that stitches the voice summary and the WhatsApp summary together, is composed by a plain deterministic template, not a third LLM. we did not want a model paraphrasing two other models' paraphrases, that is how you end up with a game of telephone where by the third hop the borrower committed to a date they never actually said.

The important part is that the record is not just prose. Next to the narrative it carries structured state.

The prose is for the LLM to read like a briefing before it walks into the room. The structured fields are for the code to make hard decisions on, the kind of decisions you cannot trust a paragraph with.

The hard part is resuming, not remembering

Writing the memory is the easy half. The hard half is reading it and starting the next conversation in the right place. This is where it got interesting, because the obvious approach and the correct approach are not the same.

The obvious approach is, let an LLM read the memory and decide where to resume. And you do want the LLM in the loop, because a line like they were upset about a late fee and asked to be called after 6pm is exactly the kind of thing a model reads well and code reads badly. So there is a per-channel classifier that reads the memory and proposes where to pick up.

But you cannot let the model freely pick the state, because every so often it will do something insane, like decide the borrower is ready for the payment link when they have not even confirmed who they are. So every proposal gets clamped by a dumb, deterministic, agent-local safety floor.

The floor is boring on purpose. It knows a few hard rules, you cannot resume past identity if identity is not verified, you cannot resume into a state that is already closed, you cannot jump more than one real step ahead of where the memory actually says you are. The LLM proposes, the floor disposes, and if they disagree the floor wins every time. We think this is the single most important decision in the whole thing, because it is what lets you use a model for the fuzzy human judgment while a deterministic rule guarantees the agent never does the unsafe or stupid version of that judgment.

"Remind them if we haven't already" only works with a hard boolean

Here is a concrete example of why the structured fields matter more than the prose. A very common instruction is, remind this borrower about their due date, but only if we have not already reminded them. Sounds trivial. It is not, if reminded is buried inside a paragraph, because the model reading that paragraph will sometimes decide we probably reminded them and sometimes decide we probably did not, and now you are either double-reminding people, which is annoying and in collections is a compliance problem, or never reminding them, which is useless.

So the idempotency facts, did we remind, did we share the link, did we confirm the payment, are hard booleans the summary LLM is not allowed to touch. They get set by the actual event of the thing happening, not by a model's reading of a transcript. The agent asks the boolean, not the paragraph. This is the whole difference between an agent that remembers correctly and one that remembers confidently and wrongly.

A promise on WhatsApp is not a promise on a call

There is a nice subtle thing that falls out of doing this across two channels. When a borrower types yeah I'll pay Friday into WhatsApp, that is not the same as saying it to a voice agent who confirmed it back to them out loud. One is a soft intent, the other is a firm commitment. So WhatsApp writes the commitment down as proposed, and only the voice agent, after actually confirming it on a call, promotes it to confirmed.

And the memory only ever climbs this ladder, it never slides back down. If two summaries disagree about the commitment, the record keeps the stronger one, never the weaker one. A confirmed promise cannot get silently downgraded to proposed just because some later WhatsApp summary was fuzzy about it. The state is monotonic, it only goes up. This sounds like a small detail and it is the kind of small detail that, gotten wrong, makes an agent tell someone they never really committed when they did.

Teaching the voice agent to start in the middle

One thing we actually had to build for this was the ability for the voice agent to start a call in the middle of its flow. It was stateless and it knew exactly one way to begin, at the opening, say hello and verify identity. To resume, it needed to be able to drop the borrower straight into, say, the confirmation step.

The annoying part was that the middle steps were written assuming they always come after the opening, so each one starts by quietly summarizing what was said in the step before it. Enter one of them cold, with no previous step, and it summarizes nothing, and the agent sounds like it teleported into the conversation with no idea why it is there. The fix was to feed the memory's narrative in as the fake previous step, so when the agent starts at confirmation it reads the story so far as if it had just been the one talking. It picks up mid-sentence instead of mid-void.

The bugs were all at the seams

The individual pieces of this were not hard. Every genuinely nasty bug lived in the gap between two services, and none of them showed up in unit tests. Two are worth telling.

The first one. The voice agent saves the state it should resume at, passes it along, and reads it back on the next call. Except a read-side filter, a whitelist of allowed fields written a long time ago by someone else for a completely different reason, quietly dropped the resume state on the way back in. So the memory said resume at confirmation, the write saved resume at confirmation, and the agent still cold-started at the opening, silently, no error anywhere. Every unit test passed, because the tests handed the state straight to the flow and never went through the read path that was dropping it. It only turned up when we ran the real services together in containers and watched a real call ignore a real memory. The lesson we keep relearning is that the seam between two components is where the bugs live, and the only way to catch them is to run the whole thing for real, not each piece in isolation.

The second one was worse because it was silent in production. The resume logic keys off the commitment status, proposed or confirmed. But a firm promise captured under a slightly unusual disposition landed in the memory with the amount and the due date filled in but the status field left empty. And the resume logic, seeing no status, did nothing and cold-started. So every borrower who made a real promise but had it recorded the slightly wrong way got treated like they had never said a word. The memory was right there, full of the promise, and the agent walked straight past it. We fixed it on both sides, the write side now defaults the status the moment any promise is captured, and the read side derives it from the journey state when it is missing, so it works even on the records that were already written wrong.

Where it actually is

Honest status, because the honest status is more interesting than a clean one. The writing half is live in production on one canary team and it works, the memory fills up correctly, one clean record per connected call, well-formed, no error-rate regression. The resume half is proven end to end locally, the full chain across both channels where each conversation's memory really seeds the next one, from empty to negotiation to committed to reminded to link shared.

The genuinely hard part turns out to be neither the writing nor the reading, it is getting a real borrower to actually cross channels. On the canary team almost nobody replies to the WhatsApp opener, so the cross-channel case, which is the whole point, the voice-agent-remembers-the-WhatsApp-chat magic, barely gets to fire in the wild yet. The plumbing is correct. But the behavior change mostly shows up on the next call, tomorrow, not on this one, because you do not re-call someone you already reached today. So this is one of those features where the demo is easy and the honest measurement is slow, you have to wait for real people to actually come back.

The point

Memory is the thing that turns a pile of separate stateless conversations into one relationship. Without it every agent is a goldfish, charming for thirty seconds and then it forgets you exist. With it the company sounds like it actually knows you, and when you are calling someone about money that is the entire difference between feeling harassed and feeling helped.

And the part we did not expect going in is that almost none of the hard work was the LLM. The summaries were the easy bit. The real work was the boring scaffolding around them, the single writer so two channels do not fight, the deterministic floor so the model cannot do anything unsafe, the hard booleans so remind-if-not-already is actually correct, the monotonic ladder so the state never slides backwards. The model gives you the fuzzy human judgment. The boring deterministic code is what makes it safe to act on. Get that boring part wrong and you have an agent that remembers things confidently and incorrectly, which is honestly worse than one that remembers nothing at all.