Most production AI agent deployments have a secrets problem they have not looked at closely. The .env file is sitting on a VPS with 640 permissions and a shared ubuntu login. The database row that holds the user's OpenAI key is a TEXT column with no encryption at the application level. The developer knows the key is "just a string," so it gets treated like any other user input. Until one day it is not just a string — it is a $4,000 runaway bill because a model-provider breach or a misconfigured S3 bucket dumped the database.
This post covers the failure modes we see most often, what good key handling actually looks like from an engineering standpoint, and the two deployment models we offer — BYOK (bring your own key) and managed provisioning — and why each one exists.
The plaintext trap
The attack surface for a plaintext key is larger than most teams realize. A key in an .env file is readable by any process on that machine under the same user, any crash reporter that captures environment variables, any log aggregator that prints process.env on startup, and any shell command substitution that accidentally expands the variable into a log line. That is before you consider shared hosting, noisy neighbors, or a developer's laptop that also runs the staging environment.
A key in a database TEXT column is in every database backup — and backups are often stored in a different bucket with looser IAM policies because "they're just backups." It is in every SELECT * executed by an admin tool, every ORM debug log that prints full query results, and every database snapshot replicated to a read replica for analytics.
None of this requires a sophisticated attacker. Most plaintext key exposures are boring: a misconfigured environment, a developer checking the wrong thing into git, a backup restore test that left credentials in a staging database that was never cleaned up.
What good handling looks like
There is no single technique that handles this — it is a stack of four properties working together:
- Encrypted at rest, application-layer. The key is encrypted with a data encryption key (DEK) before it ever touches the database. The DEK itself is wrapped by a key encryption key (KEK) held in a secrets manager — Fly.io secrets, Supabase Vault, or a dedicated KMS. The database stores ciphertext. Even a full backup dump is useless without the KEK.
- Scoped, not root. The key that gets stored should be the narrowest key the integration requires. An agent that only reads files does not need write access. An agent scoped to one project does not get an org-level key. When the blast radius of a compromise is bounded by scope, the cost of rotation is also lower.
- Revocable without downtime. There needs to be a path to invalidate a key and rotate in a replacement without taking the agent offline. This is why key storage is a separate concern from agent configuration — the agent holds a reference to the key record, not the key bytes themselves.
- EU-resident infrastructure. For teams under GDPR or with EU customers, the credential store itself needs to be physically in-region. We run agent infrastructure on Fly.io (Amsterdam/Frankfurt) with Supabase on the EU-West region. This is not a compliance checkbox — it changes which court orders can reach your data and which legal exposure you carry.
This is the engineering stance, not a sales pitch. If you are not doing all four, you have a real gap — and the industry average is doing two of them, sometimes three.
The Supabase Vault pattern in practice
Supabase Vault wraps PostgreSQL with an pgsodium-backed secrets table. The actual key bytes never appear in a normal SELECT. An agent that needs to make an authenticated call does this:
The decrypted_secrets view is a server-side function call — not a real table. Row-level security on that view means a Supabase service-role key can read it, but an anon/public key cannot. The plaintext never leaves the Postgres process boundary in cleartext. A database backup contains only the ciphertext column, not the pgsodium master key that is held at the infrastructure level.
Rotation is a single vault.update_secret() call. No redeploy. No environment variable change. No restart. The next agent call picks up the new key automatically.
Two deployment models: BYOK vs managed
When we build agent systems for clients, the question of key ownership comes up in the first design conversation. There are exactly two architectures that make sense, and the right one depends on what the client controls and what risk they want to carry.
BYOK — bring your own key. The client creates their own API keys at the provider (OpenAI, Anthropic, Google, whichever) and hands us only what the integration needs. We store those keys in Vault with the pattern above. The billing relationship stays directly between the client and the provider — they see their own usage dashboard, they set their own spend limits, and they can revoke the key at the source without involving us at all. This is the right model when the client already has a provider relationship, cares about direct billing visibility, or has compliance requirements around who can be in the credential chain.
Managed provisioning. We create and hold the provider account and keys on the client's behalf. This makes sense for teams that do not want to manage API accounts at all — small teams, non-technical operators, or cases where the agent is a self-contained product we are delivering rather than integrating. Keys are still encrypted at rest via Vault, still scoped to the minimum necessary access, still revocable. The difference is that we are the account holder, and the billing is wrapped into the engagement. If a client offboards, we generate new keys, hand them over, and immediately revoke the ones we held.
Neither model stores keys in plaintext. That is not a feature — it is a baseline requirement that should not need to be stated, but apparently does.
The audit trail question
Encryption at rest answers "what happens if the database leaks." It does not answer "which agent made which API call and when." Those are separate properties, and both matter.
Every provider call made by our agents goes through a thin middleware layer that logs the provider name, the model, the token counts, and the user or tenant identifier — before the call is made and after it returns. Logs are append-only and shipped to a separate sink from the application database. This means that if a key is compromised and rotated, you have a timestamped record of every call made with it — useful both for understanding blast radius and for any provider dispute process.
This is also why scoping matters in combination with logging. A scoped key that is logged gives you both containment and visibility. A root key with logging gives you visibility but wide blast radius. A scoped key without logging gives you containment but no forensics. You want both.
What to check in your own stack
If you have an agent system running in production, these are the five questions worth asking this week:
-
Where are your provider API keys stored right now? Are they in
.env, aTEXTcolumn, or a proper encrypted store? - Could a database backup restore in staging expose production keys?
- Are your keys scoped, or are they all org-level with full permissions?
- How long would it take to rotate every key if you found one had been compromised at 2am?
- Do your logs tell you which calls were made with which key, and is that log in a separate system that a compromised app database cannot overwrite?
If you cannot answer all five cleanly, the gaps are not theoretical. They are just gaps that have not been triggered yet.
The point
"We don't store your keys in plaintext" is not a differentiator — it is table stakes. The real question is whether the full stack is in place: encrypted at rest with application-layer key wrapping, minimum-necessary scoping, clean revocation paths, EU-resident infrastructure where it matters, and an audit log that is independent of the application itself. That stack is not complicated to build, but it requires making deliberate choices at each layer rather than reaching for the nearest .env file.
BYOK or managed, the architecture is the same. What changes is who holds the provider account — and that choice should be made explicitly, not by default.