Build a Durable Agent Memory Layer in TypeScript: Recall, Summarize, Evict

Last week I argued that the context window is not memory — that treating a 200K- or 1M-token window as a place to "remember things" is a category error that produces agents which forget the user's name two turns after they were told it, and which re-derive the same conclusion every session because nothing persisted. That article was the diagnosis. This one is the prescription.

We are going to build a durable agent memory layer in TypeScript: a standalone module that an agent loop writes observations to and reads relevant context from, that survives process restarts, that ranks what to recall by more than raw similarity, that compresses old detail into durable summaries before it overflows, and that evicts what no longer earns its place. No framework. No hidden magic. Around 400 lines of code you can read in one sitting, plus tests.

The companion repository lives at github.com/CrashBytes/ByteSizedExamples. Clone it for a runnable starting point, or build from scratch as you read — every file is reproduced in full below.

The Difference Between a Context Window and Memory

The distinction is worth stating precisely because it drives every design decision that follows. A context window is working memory — the tokens the model can attend to on this single forward pass. It is bounded, it is volatile, and it is paid for on every request. Memory, in the human sense and the sense we want for agents, is the ability to retrieve relevant past experience on demand, where "relevant" and "past" can both be very large.

A memory layer sits between the agent and the model. On the way out, it captures what happened — user statements, tool results, decisions, errors. On the way in, it retrieves the small, relevant subset of everything-ever-captured that this turn actually needs, and packs it into the prompt under a token budget. The window stays small and cheap; the memory grows without bound on disk.

That split — bounded volatile window, unbounded durable store, a retrieval step that bridges them — is the whole architecture. Everything else is implementation detail about how you decide what is relevant, how you keep the store from growing into a swamp, and how you avoid trusting it blindly.

The numbers above are from a 60-turn synthetic support-agent transcript I run as a regression harness; the absolute values move around with the workload, but the shape is stable. Stuffing everything in degrades as the window fills and gets expensive fast. A sliding window is cheap but forgets. A memory layer retains the relevant facts and keeps each prompt small, because it only ever injects the handful of records this turn needs.

What You Will Build

The memory layer has six parts. Build them in order; each is independently testable.

1. The record and the store interface — the data model for a single memory and the contract every backend implements.
2. Embeddings and a vector index — turning text into vectors and finding the nearest ones.
3. Hybrid ranking — combining semantic similarity with recency, salience, and access frequency, because pure similarity recalls the wrong things.
4. Scoped retrieval and budget packing — namespacing memory per user/agent and packing recalled records into a fixed token budget.
5. Rolling summarization — compressing clusters of old episodic memory into durable semantic summaries before the store overflows.
6. Eviction — TTL, capacity, and salience-floor policies that keep the store lean.

Prerequisites: Node 20+, TypeScript 5.x, and an embeddings endpoint (the code uses an OpenAI-compatible /embeddings call, but the interface is one function you can swap). Initialize the project:

mkdir agent-memory-layer-typescript && cd $_
npm init -y
npm i -D typescript vitest @types/node tsx
npx tsc --init --target es2022 --module nodenext --moduleResolution nodenext --strict

Part 1: The Record and the Store Interface

A memory is not just text. The fields around the text are what let you rank, scope, summarize, and evict intelligently. Create src/types.ts:

export type MemoryKind = 'episodic' | 'semantic' | 'procedural'

export interface MemoryRecord {
  id: string
  /** The text the model will actually read on recall. */
  content: string
  /** Embedding of `content`. Length depends on your model. */
  embedding: number[]
  /**
   * episodic  = a thing that happened ("user said their deploy failed at 14:02")
   * semantic  = a durable fact distilled from episodes ("user runs on GKE")
   * procedural = a learned how-to ("to restart their cluster, run X")
   */
  kind: MemoryKind
  /** Namespace: who/what this memory belongs to. See Part 4. */
  scope: string
  /** Higher = more important. Drives ranking and eviction. 0..1. */
  salience: number
  /** Rough token count of `content`, for budget packing. */
  tokens: number
  createdAt: number
  lastAccessedAt: number
  accessCount: number
  /** Set when this record was folded into a summary and superseded. */
  archivedAt?: number
}

export interface RecallQuery {
  scope: string
  text: string
  /** Max tokens of recalled content to return. */
  tokenBudget: number
  /** Restrict to certain kinds, e.g. only semantic facts. */
  kinds?: MemoryKind[]
  /** How many candidates to pull before re-ranking. */
  candidatePool?: number
}

export interface MemoryStore {
  put(record: MemoryRecord): Promise<void>
  /** All non-archived records in a scope, used by ranking + maintenance. */
  list(scope: string): Promise<MemoryRecord[]>
  delete(ids: string[]): Promise<void>
  markArchived(ids: string[], at: number): Promise<void>
  touch(ids: string[], at: number): Promise<void>
}

Two decisions matter here. First, kind is explicit, not inferred. Episodic memories are cheap and plentiful and get summarized away; semantic and procedural memories are distilled, durable, and survive eviction longer. Second, every record carries the metadata ranking and eviction need —salience, accessCount, timestamps — so those operations never have to re-read content or call the model.

Now the simplest possible backend, a JSON-file store in src/jsonStore.ts. It is not what you ship to production, but it is durable across restarts and it makes the contract concrete:

import { readFile, writeFile, mkdir } from 'node:fs/promises'
import { dirname } from 'node:path'
import type { MemoryRecord, MemoryStore } from './types.js'

export class JsonStore implements MemoryStore {
  private cache = new Map<string, MemoryRecord>()
  private loaded = false

  constructor(private path: string) {}

  private async load(): Promise<void> {
    if (this.loaded) return
    try {
      const raw = await readFile(this.path, 'utf8')
      for (const rec of JSON.parse(raw) as MemoryRecord[]) {
        this.cache.set(rec.id, rec)
      }
    } catch {
      // First run: no file yet.
    }
    this.loaded = true
  }

  private async flush(): Promise<void> {
    await mkdir(dirname(this.path), { recursive: true })
    await writeFile(this.path, JSON.stringify([...this.cache.values()]))
  }

  async put(record: MemoryRecord): Promise<void> {
    await this.load()
    this.cache.set(record.id, record)
    await this.flush()
  }

  async list(scope: string): Promise<MemoryRecord[]> {
    await this.load()
    return [...this.cache.values()].filter(
      r => r.scope === scope && !r.archivedAt
    )
  }

  async delete(ids: string[]): Promise<void> {
    await this.load()
    for (const id of ids) this.cache.delete(id)
    await this.flush()
  }

  async markArchived(ids: string[], at: number): Promise<void> {
    await this.load()
    for (const id of ids) {
      const r = this.cache.get(id)
      if (r) r.archivedAt = at
    }
    await this.flush()
  }

  async touch(ids: string[], at: number): Promise<void> {
    await this.load()
    for (const id of ids) {
      const r = this.cache.get(id)
      if (r) {
        r.lastAccessedAt = at
        r.accessCount += 1
      }
    }
    await this.flush()
  }
}

When you outgrow this — and you will, somewhere north of a few thousand records per scope — the same interface drops onto pgvector, SQLite with a vector extension, or a managed vector database. The agent code above this interface never changes. That is the point of pinning the contract before the backend.

Part 2: Embeddings and a Vector Index

Recall starts with turning text into a vector and finding the nearest stored vectors. Keep embedding behind a one-function interface so you can swap providers or stub it in tests. Create src/embeddings.ts:

export type Embedder = (texts: string[]) => Promise<number[][]>

export const openAIEmbedder =
  (apiKey: string, model = 'text-embedding-3-small'): Embedder =>
  async texts => {
    const res = await fetch('https://api.openai.com/v1/embeddings', {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        authorization: `Bearer ${apiKey}`,
      },
      body: JSON.stringify({ model, input: texts }),
    })
    if (!res.ok) throw new Error(`embeddings failed: ${res.status}`)
    const json = (await res.json()) as { data: { embedding: number[] }[] }
    return json.data.map(d => d.embedding)
  }

For nearest-neighbor search at this scale we do not need an index structure — a linear scan with cosine similarity is exact and fast enough for thousands of records per scope. Premature HNSW indexing is a classic way to add a bug surface you do not need yet. Create src/similarity.ts:

export function cosine(a: number[], b: number[]): number {
  let dot = 0
  let na = 0
  let nb = 0
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i]
    na += a[i] * a[i]
    nb += b[i] * b[i]
  }
  const denom = Math.sqrt(na) * Math.sqrt(nb)
  return denom === 0 ? 0 : dot / denom
}

A note on cost discipline: embeddings are cheap per call but you call them on every write and every recall. Batch writes when you can, and cache the query embedding within a turn if you recall more than once. The same cost-floor logic that makes a multi-model router worth building applies here — the cheapest correct path is almost always to do less work, not to find a cheaper model.

Part 3: Hybrid Ranking — Why Similarity Alone Recalls the Wrong Things

Here is the mistake almost every first-pass memory layer makes: it recalls by cosine similarity alone. That feels right and is subtly wrong. Pure similarity has no sense of time, so a stale fact the user corrected an hour ago outranks the correction. It has no sense of importance, so an offhand remark outranks a hard preference. And it has no sense of what has been useful before, so the same dead record keeps surfacing.

Real recall is a weighted blend of four signals:

• Similarity — how semantically close the record is to the query.
• Recency — newer memories are more likely relevant; old ones decay.
• Salience — how important the record was marked when written.
• Usage — records that have proven useful get a mild boost.

Recency is an exponential decay with a half-life: a memory loses half its recency score every halfLifeMs. That single knob lets you say "in this agent, a fact from a week ago is worth half a fact from today" without hand-tuning a curve.

Create src/ranking.ts:

import { cosine } from './similarity.js'
import type { MemoryRecord } from './types.js'

export interface RankWeights {
  similarity: number
  recency: number
  salience: number
  usage: number
}

export const defaultWeights: RankWeights = {
  similarity: 0.5,
  recency: 0.25,
  salience: 0.15,
  usage: 0.1,
}

export interface RankOptions {
  weights?: RankWeights
  /** Recency half-life in ms. Default 7 days. */
  halfLifeMs?: number
  now?: number
}

export interface Scored {
  record: MemoryRecord
  score: number
}

export function rank(
  query: number[],
  records: MemoryRecord[],
  opts: RankOptions = {}
): Scored[] {
  const w = opts.weights ?? defaultWeights
  const halfLife = opts.halfLifeMs ?? 7 * 24 * 60 * 60 * 1000
  const now = opts.now ?? Date.now()
  const decay = Math.LN2 / halfLife

  return records
    .map(record => {
      const similarity = Math.max(0, cosine(query, record.embedding))
      const ageMs = Math.max(0, now - record.createdAt)
      const recency = Math.exp(-decay * ageMs)
      const salience = clamp01(record.salience)
      // Diminishing-returns boost: log keeps a hot record from dominating.
      const usage = Math.min(1, Math.log10(record.accessCount + 1) / 2)
      const score =
        w.similarity * similarity +
        w.recency * recency +
        w.salience * salience +
        w.usage * usage
      return { record, score }
    })
    .sort((a, b) => b.score - a.score)
}

function clamp01(n: number): number {
  return Math.max(0, Math.min(1, n))
}

The usage term uses a log so a record that has been recalled fifty times does not permanently crowd out everything else — the boost saturates quickly. Tune the weights per agent: a customer-support bot leans on recency (the latest ticket state dominates), while a long-running research assistant leans on salience and similarity (a key finding from a month ago still matters).

Part 4: Scoped Retrieval and Budget Packing

Two agents, or two users, must never see each other's memories. That is what scope enforces — a namespace string you compose from the dimensions that should isolate memory. A good default is ${userId}:${agentId}, with session-level episodic memory under ${userId}:${agentId}:${sessionId} and cross-session semantic facts under the broader ${userId}:${agentId}.

Recall then does four things: embed the query, pull a candidate pool from the scope, re-rank with the hybrid scorer, and pack the top records into the token budget — stopping when the next record would overflow. Packing is where a lot of naive implementations quietly blow their context budget by injecting everything the vector search returned. Create src/memory.ts:

import { randomUUID } from 'node:crypto'
import type { Embedder } from './embeddings.js'
import { rank, type RankOptions } from './ranking.js'
import type {
  MemoryKind,
  MemoryRecord,
  MemoryStore,
  RecallQuery,
} from './types.js'

export interface WriteInput {
  scope: string
  content: string
  kind?: MemoryKind
  salience?: number
}

// ~4 chars per token is a good-enough estimate for budgeting.
export const estimateTokens = (text: string): number =>
  Math.ceil(text.length / 4)

export class MemoryLayer {
  constructor(
    private store: MemoryStore,
    private embed: Embedder,
    private rankOpts: RankOptions = {}
  ) {}

  async remember(input: WriteInput): Promise<MemoryRecord> {
    const [embedding] = await this.embed([input.content])
    const now = Date.now()
    const record: MemoryRecord = {
      id: randomUUID(),
      content: input.content,
      embedding,
      kind: input.kind ?? 'episodic',
      scope: input.scope,
      salience: input.salience ?? 0.5,
      tokens: estimateTokens(input.content),
      createdAt: now,
      lastAccessedAt: now,
      accessCount: 0,
    }
    await this.store.put(record)
    return record
  }

  async recall(q: RecallQuery): Promise<MemoryRecord[]> {
    const [queryEmbedding] = await this.embed([q.text])
    let candidates = await this.store.list(q.scope)
    if (q.kinds) candidates = candidates.filter(c => q.kinds!.includes(c.kind))

    const pool = q.candidatePool ?? 50
    const scored = rank(queryEmbedding, candidates, this.rankOpts).slice(
      0,
      pool
    )

    const packed: MemoryRecord[] = []
    let used = 0
    for (const { record } of scored) {
      if (used + record.tokens > q.tokenBudget) continue
      packed.push(record)
      used += record.tokens
    }

    await this.store.touch(
      packed.map(r => r.id),
      Date.now()
    )
    return packed
  }
}

Note the packing loop uses continue, not break: if a large record would overflow the budget, we skip it and keep trying smaller ones rather than truncating the whole list at the first miss. That single choice meaningfully improves how much useful context fits in a fixed budget.

You now have a working memory layer. Write to it, recall from it, and it survives restarts. The remaining two parts are what separate a demo from something you can run for months without it degrading.

Part 5: Rolling Summarization — Compress Before You Overflow

Episodic memory grows linearly with conversation. Left alone, a busy scope accumulates thousands of low-value records ("user said thanks", "tool returned 200") that dilute recall and bloat the store. Summarization is the pressure valve: when a scope's episodic memory crosses a token threshold, fold the oldest cluster of episodes into a single durable semantic record and archive the originals.

The summarizer is a model call with a tight prompt. Create src/summarize.ts:

import type { Embedder } from './embeddings.js'
import { estimateTokens } from './memory.js'
import type { MemoryRecord, MemoryStore } from './types.js'

export type Summarizer = (episodes: string[]) => Promise<string>

const SUMMARY_PROMPT = `You compress an AI agent's episodic memories into one
durable summary. Keep stable facts, preferences, decisions, and unresolved
threads. Drop pleasantries and transient status. Write terse third-person notes,
not prose. Under 120 words.`

export const llmSummarizer =
  (apiKey: string, model = 'gpt-4o-mini'): Summarizer =>
  async episodes => {
    const res = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        authorization: `Bearer ${apiKey}`,
      },
      body: JSON.stringify({
        model,
        temperature: 0,
        messages: [
          { role: 'system', content: SUMMARY_PROMPT },
          {
            role: 'user',
            content: episodes.map((e, i) => `${i + 1}. ${e}`).join('\n'),
          },
        ],
      }),
    })
    if (!res.ok) throw new Error(`summarize failed: ${res.status}`)
    const json = (await res.json()) as {
      choices: { message: { content: string } }[]
    }
    return json.choices[0].message.content.trim()
  }

export interface SummarizeOptions {
  /** Episodic token budget per scope before summarization fires. */
  episodicBudget: number
  /** How many of the oldest episodes to fold per pass. */
  clusterSize: number
}

export async function maybeSummarize(
  scope: string,
  store: MemoryStore,
  embed: Embedder,
  summarize: Summarizer,
  opts: SummarizeOptions
): Promise<MemoryRecord | null> {
  const all = await store.list(scope)
  const episodic = all
    .filter(r => r.kind === 'episodic')
    .sort((a, b) => a.createdAt - b.createdAt)

  const totalTokens = episodic.reduce((sum, r) => sum + r.tokens, 0)
  if (totalTokens <= opts.episodicBudget) return null

  const cluster = episodic.slice(0, opts.clusterSize)
  if (cluster.length < 2) return null

  const summaryText = await summarize(cluster.map(r => r.content))
  const [embedding] = await embed([summaryText])
  const now = Date.now()

  const summary: MemoryRecord = {
    id: crypto.randomUUID(),
    content: summaryText,
    embedding,
    kind: 'semantic',
    scope,
    // A summary inherits the max salience of its sources — it represents them.
    salience: Math.max(...cluster.map(c => c.salience)),
    tokens: estimateTokens(summaryText),
    createdAt: now,
    lastAccessedAt: now,
    accessCount: 0,
  }

  await store.put(summary)
  await store.markArchived(
    cluster.map(c => c.id),
    now
  )
  return summary
}

Call maybeSummarize after each write, or on an interval, or when a scope's record count crosses a watermark. The effect is dramatic: a scope that would have held 800 episodic records holds 40 semantic summaries plus the recent episodic tail, recall quality goes up because the noise is gone, and the store stops growing without bound.

The green line flattens because summarization keeps folding the episodic tail into a bounded set of semantic records; the red line is the same workload with summarization disabled, growing linearly forever. The store on the left is one you can run for a year. The one on the right is a cleanup ticket waiting to happen.

Part 6: Eviction — Keep the Store Lean

Summarization compresses; eviction deletes. They are complementary. Even after summarizing, you want hard policies that bound the store and drop records that no longer earn their keep. Three policies cover almost every agent:

• TTL — episodic memories past a max age are deleted (semantic and procedural are exempt; distilled knowledge should not expire on a clock).
• Capacity — when a scope exceeds a record cap, drop the lowest-scored records until back under the cap.
• Salience floor — never auto-delete a record above a salience threshold, regardless of age or capacity. This is your safety valve for "this fact must never be forgotten."

Create src/eviction.ts:

import { rank } from './ranking.js'
import type { MemoryStore } from './types.js'

export interface EvictionPolicy {
  /** Delete episodic records older than this (ms). */
  episodicTtlMs?: number
  /** Max non-archived records per scope. */
  maxRecords?: number
  /** Never evict records at or above this salience. */
  salienceFloor?: number
}

export async function evict(
  scope: string,
  store: MemoryStore,
  policy: EvictionPolicy,
  now = Date.now()
): Promise<string[]> {
  const records = await store.list(scope)
  const floor = policy.salienceFloor ?? 0.9
  const evictable = records.filter(r => r.salience < floor)
  const toDelete = new Set<string>()

  // TTL: episodic only.
  if (policy.episodicTtlMs) {
    for (const r of evictable) {
      if (r.kind === 'episodic' && now - r.createdAt > policy.episodicTtlMs) {
        toDelete.add(r.id)
      }
    }
  }

  // Capacity: drop lowest-scored evictable records until under the cap.
  if (policy.maxRecords && records.length > policy.maxRecords) {
    const overBy = records.length - policy.maxRecords
    // Score with a zero query so ranking reduces to recency+salience+usage.
    const zero = new Array(records[0]?.embedding.length ?? 0).fill(0)
    const ranked = rank(zero, evictable, { now }).reverse() // worst first
    for (const { record } of ranked) {
      if (toDelete.size >= overBy) break
      toDelete.add(record.id)
    }
  }

  const ids = [...toDelete]
  if (ids.length) await store.delete(ids)
  return ids
}

The salience floor is doing real work in that last bar: without it, capacity eviction occasionally drops a high-value fact that happened to score low on a particular maintenance pass, and precision suffers when that fact is later needed. Protecting the top salience band costs you almost nothing in store size and recovers most of the precision.

Choosing a Production Backend

The JSON store got you running, but it loads every record into a process-local Map and rewrites the whole file on every put. That is fine for a prototype and a disaster at scale. When you graduate, the only file that changes is the one implementing MemoryStore; everything above the interface — ranking, summarization, eviction, the agent loop — is untouched. That is the dividend you earned by pinning the contract in Part 1.

There are three backends worth knowing, and the right answer is almost always the least exotic one your scale allows.

For most teams the honest answer is Postgres with pgvector. You almost certainly already run it, the scope and kind columns become a WHERE clause, and the vector similarity becomes an ORDER BY embedding <=> $1 operator. The linear cosine scan from Part 2 moves into the database, and for the candidate-pool step you let pgvector do an approximate nearest-neighbor search to pull the top few hundred, then run the hybrid re-ranker from Part 3 in application code on that small set. That two-stage pattern — cheap approximate retrieval in the database, precise re-ranking in the app — is how you keep the smart ranking logic without paying to score every record on every recall.

A subtle point that bites people: do the recency and salience math in application code, not in SQL, even when the backend could express it. Your ranking weights are a tuning surface you will change weekly while you calibrate an agent. Keeping that logic in TypeScript, behind the same rank function your tests cover, means you tune it with a unit test and a redeploy rather than a migration. The database's job is to return candidates fast; your job is to decide which of them matter, and that decision should live where it is easy to test.

One more backend rule: index the scope column before you index anything else. Scope is on the hot path of every single recall, and an unindexed scope filter turns every query into a full-table scan the moment you have more than one tenant. The vector index is the glamorous part; the scope index is the one that keeps you out of an incident review.

Wiring It Into an Agent Loop

All six parts compose into a small surface the agent uses on every turn. Recall before the model call, remember after it. Create src/agentLoop.ts:

import { MemoryLayer } from './memory.js'
import {
  maybeSummarize,
  type Summarizer,
  type SummarizeOptions,
} from './summarize.js'
import { evict, type EvictionPolicy } from './eviction.js'
import type { Embedder } from './embeddings.js'
import type { MemoryStore } from './types.js'

export interface TurnDeps {
  store: MemoryStore
  embed: Embedder
  summarize: Summarizer
  summarizeOpts: SummarizeOptions
  evictionPolicy: EvictionPolicy
  callModel: (systemContext: string, userInput: string) => Promise<string>
}

export async function runTurn(
  scope: string,
  userInput: string,
  deps: TurnDeps
): Promise<string> {
  const memory = new MemoryLayer(deps.store, deps.embed)

  // 1. Recall relevant context under a budget.
  const recalled = await memory.recall({
    scope,
    text: userInput,
    tokenBudget: 800,
  })
  const systemContext = recalled.length
    ? 'Relevant memory:\n' + recalled.map(r => `- ${r.content}`).join('\n')
    : ''

  // 2. Call the model with the small, relevant context.
  const reply = await deps.callModel(systemContext, userInput)

  // 3. Remember what happened this turn.
  await memory.remember({
    scope,
    content: `User: ${userInput}`,
    kind: 'episodic',
  })
  await memory.remember({ scope, content: `Agent: ${reply}`, kind: 'episodic' })

  // 4. Maintenance — cheap to run every turn, or move to an interval.
  await maybeSummarize(
    scope,
    deps.store,
    deps.embed,
    deps.summarize,
    deps.summarizeOpts
  )
  await evict(scope, deps.store, deps.evictionPolicy)

  return reply
}

That is the entire integration. Recall, model call, remember, maintain. The agent above this function does not know whether memory is a JSON file or a sharded vector database — it knows recall and remember, and that is all it should know.

Five Mistakes That Quietly Degrade Recall

Every one of these shipped in a version of this code that I wrote before I learned better. None of them throw an error. All of them make an agent subtly worse in a way that is maddening to diagnose, because the symptom is "the agent feels dumber than it used to" and there is no stack trace.

Embedding the wrong thing. It is tempting to embed a rich record — speaker tags, timestamps, metadata — and recall against it. Embed only the semantic content. User: my deploy failed embeds cleanly; [2026-06-01T14:02:11Z] turn #34 (episodic, salience 0.5) User: my deploy failed embeds the noise too, and the timestamp dilutes the signal you actually want to match on. Store the metadata in fields, embed the meaning.

Recalling the query verbatim. If a user says "what did I tell you about my infra?", embedding that literal question recalls other questions, not the answer. For meta-queries about memory, recall against the subject ("infra setup"), not the question. A small query-rewriting step in front of recall pays for itself.

Never decaying salience. Salience set once at write time goes stale. A fact that was important during one project should fade when the project ends. Periodically nudge salience down for records that have not been recalled in a long time — a maintenance pass that multiplies salience by a small factor for cold records keeps the importance signal honest.

Summarizing too aggressively. If your episodic budget is too small, you fold recent episodes into summaries before they have served their purpose, and the agent loses the granular detail it needs for the current task. Summarize the tail, never the head. The cluster you fold should always be the oldest records, and the recent episodic window should comfortably cover the current task's span.

Letting recall and write race. In an async agent, it is easy to fire a recall and a write concurrently and have the write land mid-recall, so the turn either sees its own just-written memory or misses a record being archived. Order them: recall fully resolves before you write, and maintenance runs after the write commits. The agent loop in the previous section does exactly this on purpose.

The meta-lesson: a memory layer fails quietly by design, because its whole job is to make a single forward pass look smarter than it is. Build the observability in from the first commit. A recall you cannot inspect is a recall you cannot trust.

Testing the Memory Layer

A memory layer is exactly the kind of component that rots silently — a regression does not crash, it just makes recall slightly worse, and you do not notice until an agent starts forgetting things in production. Tests are not optional here. Create test/memory.test.ts:

import { describe, it, expect } from 'vitest'
import { MemoryLayer } from '../src/memory.js'
import { JsonStore } from '../src/jsonStore.js'
import type { Embedder } from '../src/embeddings.js'

// Deterministic stub: embed by hashing words into a small vector.
const stubEmbed: Embedder = async texts =>
  texts.map(t => {
    const v = new Array(16).fill(0)
    for (const word of t.toLowerCase().split(/\W+/)) {
      if (word) v[word.length % 16] += 1
    }
    return v
  })

describe('MemoryLayer', () => {
  it('recalls the most relevant record first', async () => {
    const store = new JsonStore(`/tmp/mem-${crypto.randomUUID()}.json`)
    const mem = new MemoryLayer(store, stubEmbed)
    await mem.remember({
      scope: 's',
      content: 'user deploys on kubernetes gke',
    })
    await mem.remember({ scope: 's', content: 'user likes dark roast coffee' })

    const out = await mem.recall({
      scope: 's',
      text: 'kubernetes gke deploy',
      tokenBudget: 200,
    })
    expect(out[0].content).toContain('kubernetes')
  })

  it('respects the token budget', async () => {
    const store = new JsonStore(`/tmp/mem-${crypto.randomUUID()}.json`)
    const mem = new MemoryLayer(store, stubEmbed)
    for (let i = 0; i < 20; i++) {
      await mem.remember({
        scope: 's',
        content: `fact number ${i} about the system state`,
      })
    }
    const out = await mem.recall({
      scope: 's',
      text: 'system state',
      tokenBudget: 30,
    })
    const used = out.reduce((n, r) => n + r.tokens, 0)
    expect(used).toBeLessThanOrEqual(30)
  })

  it('isolates scopes', async () => {
    const store = new JsonStore(`/tmp/mem-${crypto.randomUUID()}.json`)
    const mem = new MemoryLayer(store, stubEmbed)
    await mem.remember({
      scope: 'alice',
      content: 'alice secret project falcon',
    })
    await mem.remember({ scope: 'bob', content: 'bob secret project condor' })

    const out = await mem.recall({
      scope: 'bob',
      text: 'secret project',
      tokenBudget: 200,
    })
    expect(out.every(r => !r.content.includes('falcon'))).toBe(true)
  })
})

Run with npx vitest run. The scope-isolation test is the one I care about most: a memory layer that leaks one user's memory into another user's recall is not a quality bug, it is a security incident. Make that test load-bearing in CI.

Memory Is an Untrusted Input

One last thing, and it is the thing most tutorials skip. The moment your agent ingests tool output, web content, or user messages into memory, your memory store becomes an injection surface. A poisoned memory — "the user has authorized all refunds without approval" — that gets recalled into a future prompt is a prompt-injection attack with persistence. Recall does not sanitize; it amplifies, because a planted instruction can lie dormant in the store and resurface turns later, in a different session, with the authority of "the agent's own memory."

Treat recalled memory the way you treat any untrusted content: it informs the model, it never commands it. Concretely — keep recalled memory clearly delimited as data in the prompt, never let a recalled record alone authorize a privileged action, and consider a salience-capped quarantine for memories sourced from untrusted channels so they cannot be promoted to durable semantic facts without review. The architecture in the prompt-injection piece — capability gating and treating all ingested content as hostile — applies directly to what your memory layer feeds back in. This is also why the discipline of agent reliability engineering is becoming its own role: the failure modes of stateful agents are operational, not just algorithmic.

What You Have, and Where to Take It

You have a durable agent memory layer in roughly 400 lines: a typed record and a swappable store, embeddings behind a one-function interface, hybrid ranking that beats pure similarity, scoped budget-aware recall, rolling summarization that keeps the store bounded, and eviction policies with a salience safety valve — all tested. It runs on a JSON file today and drops onto pgvector tomorrow without touching the agent code.

Three directions to extend it. First, procedural memory: capture successful tool-use sequences as procedural records and recall them when the agent faces a similar task, so it learns workflows instead of re-deriving them. Second, reflection: periodically prompt the model to derive new semantic facts from recent episodes — "what did I learn about this user this week" — which produces higher-quality semantic memory than mechanical summarization. Third, forgetting curves per memory kind: give episodic, semantic, and procedural memories different decay half-lives, so transient detail fades fast while learned procedure persists.

The context window will keep getting bigger. It will not stop being working memory. Build the layer that remembers, and your agent stops being a brilliant amnesiac.

Further Reading

• The context window is not memory — the conceptual argument this tutorial implements.
• Prompt injection is the threat model, not a bug — why your memory store is an injection surface, and how to contain it.
• Build a cost-aware multi-model router in TypeScript — the same do-less-work cost discipline, applied to model selection.
• Agent reliability engineering will emerge as a distinct discipline by Q3 2027 — my prediction on why stateful agents create a new operational role.