Learn · Testing · Preview
Leak probe: test an agent you already built
You don't need to adopt an SDK to find out whether your agent leaks between tenants. Five lessons cover planting canaries, writing a harness, reading the report and gating CI.
About this guide
Tenant leaks tend to live where nobody looked: a raw database client, a cache keyed without the tenant, a tool that accepts a tenant argument, or a retrieval path with no filter. ten-grd probe looks for them from the outside, through the same paths your users take.
It plants a unique canary record in each synthetic test tenant, signs in as one tenant and tries to reach the others' canaries. The verdict comes only from the canaries, and no model judges whether data leaked. This guide is for anyone responsible for a multi-tenant agent, whether or not it uses Ten-Grd.
You'll be able to
- See the probe catch a real leak in a small toy agent, then see it clean once fixed
- Write a harness that connects the probe to your own application
- Read a finding and act on it
- Gate CI on the exit code
- Explain why a clean run isn't proof of isolation
Test environments only. Use synthetic tenants and test credentials, never production data. The probe writes records through your application's write path and sends it hostile requests.
1. How a canary test works
- Seed. Each tenant gets a record with a unique marker, written through your own write path while signed in as that tenant.
- Control. Each tenant must be able to reach its own canary. If one can't, a clean result would mean nothing, so the run is inconclusive.
- Attack. For every ordered pair of tenants, the probe tries to reach the other's canary.
- Report. Any canary that crosses over is a finding, with the input and an evidence excerpt.
2. Catch a leak in a toy agent
python -m pip install ten-grd
Save this as tenguard_probe.py. Its ask returns every tenant's records:
TENANTS = ["acme", "globex"]
TENANT_NAMES = {"acme": "Acme Health", "globex": "Globex Finance"}
RECORDS = []
def sign_in(tenant):
return {"tenant": tenant}
def seed(session, canary):
RECORDS.append({"tenant": session["tenant"], "text": canary.text})
def ask(session, message):
return "\n".join(record["text"] for record in RECORDS)
ten-grd probe run --out leak-report.md --json leak-report.json
On ten-grd 0.3.2 this reports 10 confirmed leaks and exits 1:
LEAK: 10 cross-tenant leak(s) confirmed.
Report: leak-report.md
Now restrict the read to the signed-in tenant and run it again:
def ask(session, message):
return "\n".join(record["text"] for record in RECORDS
if record["tenant"] == session["tenant"])
CLEAN: No attempted attack exposed another tenant's canary.
Through ask, the attacks are a direct request, instruction injection, the other tenant's exact text and a request to switch tenant_id. Through call_tool they are the other tenant's text, an extra tenant_id argument and a filter-widening string.
3. Write a harness for your agent
Only your code knows how a user becomes a tenant, so the probe never creates tenant context itself. Generate a template and fill it in:
ten-grd probe init
| Name | Required | Purpose |
|---|---|---|
TENANTS | Yes | At least two synthetic tenants in your test environment |
sign_in(tenant) | Yes | Sign in a test user the way your application really does |
seed(session, canary) | Yes | Store canary.text through your normal write path |
ask(session, message) | One of these | Send a message to your agent and return its reply |
TOOLS, call_tool(session, name, arguments) | One of these | Call tools directly with hostile arguments. No model needed |
Everything your functions return is searched for canaries, so return tool results and traces as well as the final answer if you can. An unfilled template gives an inconclusive result, exit 2, which is what you want from a harness that isn't ready.
4. Caches, background jobs and retries
Coming in 0.4.0. The scenarios in this lesson are merged but not yet on PyPI. The current release, 0.3.2, runs the attacks in lesson 2 only.
Some leaks need two requests. One tenant primes a cache, and the other receives the answer. Cache scenarios need nothing extra, but your ask and call_tool must go through your real request path, not around the cache.
The probe can't reach a worker by itself, so job scenarios run only if you declare them. app.jobs and app.worker stand in for your own code:
JOBS = {"summarise": {"query": "{text}"}}
def submit_job(session, name, arguments):
return app.jobs.enqueue(session, name, arguments)
def run_jobs(fail_first=False):
app.worker.drain(fail_first_attempt=fail_first)
def job_result(session, handle):
return app.jobs.result(session, handle)
With fail_first=True, make each job's first attempt fail so your retry path runs. If you can't inject that failure, leave the parameter out, and the report will say retries weren't exercised.
5. Read the result and gate CI
| Exit | Verdict | Meaning |
|---|---|---|
| 0 | clean | None of the attempted attacks exposed another tenant's canary |
| 1 | leak | At least one cross-tenant leak was confirmed |
| 2 | inconclusive | Harness error, failed sign-in or seeding, or a tenant couldn't reach its own canary |
Non-zero for both leak and inconclusive means the command can gate CI directly. Each finding gives the attack, the input, an evidence excerpt and a likely fix.
A finding proves a cross-tenant leak for that input. No finding means none of the attempted attacks leaked. It isn't proof of isolation. The probe exercises behaviour through your harness and doesn't verify source code. It covers only the paths your harness reaches and only the listed attacks, and it doesn't use a model as an attacker.
Running the probe adds no isolation to your agent. To enforce tenant scope, see Ten-Grd: guard your agent's data. The probe documentation has the full reference.