How to test AI memory without feeding it the answer

“Do you remember that the delivery day is Friday?” is a poor memory test. The question supplies the answer.

The assistant can agree without retrieving anything. Its reply may sound reassuring while telling you very little about the system you built.

During our memory work, we used deliberately fictional facts, separate conversations, and direct evidence of retrieval. This made the tests easier to interpret without using private information. If you're new to how retrieval works, start with our guide What is RAG?

Create a clearly fictional record

Choose a disposable test topic that is unlikely to match genuine project knowledge. For example:

Synthetic memory test only. The imaginary Lantern Kite project holds its fictional planning meeting in the Violet Orchard Annex. This is not a real location, preference, or business fact.

Save it through the same approved route that normal knowledge will use. Check that the file exists and contains the intended text.

Labeling matters. Test facts should not quietly become operating instructions or appear later as real company history.

Ask from a separate conversation

Start a fresh conversation and ask, “Where does the imaginary Lantern Kite project hold its planning meeting? Find the saved source.” Do not include the room name in the question.

Check three things: whether the answer is correct, whether the cited source contains it, and whether the system actually performed the expected lookup. A plausible file reference alone is not enough; inspect the source.

A fresh chat helps reduce accidental carryover, but the full environment matters. Shared context or automatic recall may still supply the answer. If your platform exposes retrieval events, use them to understand which route succeeded.

Test explicit and automatic recall separately

First request a search directly. This checks whether the agent can use its memory tools.

Then test the behavior you expect during normal conversation. If the platform offers automatic recall, ask a relevant question without telling it to search and inspect whether the automatic stage supplied the saved material.

An agent can perform a manual search after automatic recall fails. That may still produce a useful answer, but it does not prove the automatic stage worked. During our OpenClaw tests, tool activity showed calls to memory_search and memory_get. Separate automatic-recall completion records showed whether that stage finished and supplied context. Ask whoever maintains your setup to identify those records before testing; a citation in the answer alone is insufficient.

Add a correction test

Retrieving an original fact is only the beginning. Real projects change.

In a separate synthetic record, set a fictional delivery day to Tuesday and an unrelated item count to 12. Then correct the day to Friday, mark Tuesday as superseded, and preserve the count.

Ask another conversation for the current day, the superseded day, and the item count. A good result is Friday, Tuesday, and 12, with the correct source.

We ran this kind of correction check across two agents. It tested whether the updated record became retrievable and whether an unrelated detail survived the edit.

Check what should stay unavailable

Use only non-sensitive sample files to test exclusions. Verify that material outside the approved collection cannot be read through the memory tool you are testing.

Keep this claim narrow. A memory reader refusing a file does not establish that every other tool is unable to reach it. Tool-level checks and broader permission inspection answer different questions.

Record the result and retire the fixture

Write down the date, expected answer, observed answer, source, and retrieval route. Include a failure if one occurred; it explains what a later fix actually changed.

After testing, remove or retain the sample files according to your test plan. If you test deletion, verify the source and search behavior after the index has updated. Label any retained sample so it cannot be mistaken for real knowledge.

Repeat the relevant checks after changes to source scope, permissions, or retrieval behavior. You do not need to rerun every test after every ordinary conversation.

A small test with a withheld answer gives you something confidence alone cannot: evidence that the assistant found the knowledge you meant it to use.