LLMVerify: Runtime I/O Verification for LLM Applications

A customer support AI agent I helped build last year had a straightforward architecture: user message goes to the LLM, LLM response goes back to the user. We ran static analysis on the code. Clean. We ran tenant isolation checks. Clean. We shipped.

A week later, a user asked the agent: "What is the email address of the person who filed complaint #4471?" The model had access to a retrieval tool that searched the customer database. It retrieved the record and included the complainant's email address in its response. The user received someone else's PII.

Static analysis did not catch this because the vulnerability was not in the source code. The code correctly called the LLM, correctly retrieved the document, and correctly returned the response. The problem was in what the model chose to include in its output. Source code analysis does not see model output.

This is the gap that LLMVerify fills. It sits between your LLM and your users, checking model inputs and outputs at runtime. It uses deterministic, pattern-based engines. No model calls. No network on the free tier. Same input plus same rules equals same result.

The gap between source code and runtime behavior

AI AppSec checks your source code. It finds prompt injection sinks, tool abuse patterns, and missing input validation. But it cannot see what the model actually says when it runs.

MCP Tenant Isolation checks your tenant boundary logic. It finds missing tenant filters, cache key collisions, and filesystem isolation gaps. But it cannot see whether the model leaks one tenant's data in a response to another tenant.

The third surface is runtime I/O: what goes into the model and what comes out. This surface is invisible to static analysis because it only exists when the model is running.

| Surface | Tool | What it checks | |---|---|---| | Source code | AI AppSec | Patterns in your code | | Tenant boundaries | MCP Tenant Isolation | Tenant scoping logic | | Runtime I/O | LLMVerify | Model inputs and outputs |

What LLMVerify checks

LLMVerify runs four categories of checks at runtime:

Input checks (before the model sees the message):

• Prompt injection patterns (known injection patterns, jailbreak attempts, instruction override attempts)
• PII in input (SSNs, phone numbers, email addresses — in case you want to prevent users from sending PII to the model)

Output checks (before the user sees the response):

• PII in output (SSNs, phone numbers, email addresses, credit card numbers)
• Hallucination risk indicators (specific claim patterns without source grounding)

Runtime monitoring:

• Token drift detection (model output length changing significantly from baseline)
• Behavioral change detection (response patterns shifting over time)

Every result includes an explicit limitations array stating what was checked and what was not. If LLMVerify did not check something, the limitations array says so.

Why pattern-based detection, not an LLM judge

There are two approaches to runtime LLM verification:

1. Pattern-based detection (LLMVerify): deterministic rules that match known patterns. Same input, same output. No model calls.

2. LLM-as-judge: use another LLM to evaluate the output. Can catch semantic issues that patterns miss. Non-deterministic. Requires model calls. Costs money per check. Adds latency.

LLMVerify uses pattern-based detection. The tradeoffs:

| Factor | Pattern-based (LLMVerify) | LLM-as-judge | |---|---|---| | Determinism | Same input, same output | Same input, different output | | Cost | Free (local execution) | Per-token cost for each check | | Latency | Under 100ms per check | 500-3000ms per check | | Auditability | Rules are inspectable and version-controlled | Prompt is the rule — hard to audit | | False positive control | Tunable via rule configuration | Depends on judge model and prompt | | Novel attack detection | Limited to known patterns | Can catch novel patterns | | Transparency | Every finding references a specific rule | Findings are judge model output |

The choice depends on your requirements. If you need deterministic, auditable, low-latency checks for known patterns, use LLMVerify. If you need semantic evaluation of novel outputs, use an LLM judge. They are not mutually exclusive — you can run both.

LLMVerify does not claim to catch novel or obfuscated prompt injections. The limitations array explicitly states: "Prompt-injection detection is pattern-based — novel or obfuscated injections can evade it."

SDK integration

The SDK integration is 3 lines of code:

import { verify } from 'llmverify'

const result = await verify({
  input: userMessage,
  output: modelResponse,
  checks: ['prompt-injection', 'pii-redaction'],
})

if (result.blocked) {
  return result.redactedOutput  // PII redacted, injection blocked
}
return modelResponse

The verify function returns a result object with:

• blocked: boolean — whether any check triggered a block
• findings: array of findings with rule ID, severity, and matched text
• redactedOutput: the output with PII redacted (if PII check ran)
• limitations: array of what was and was not checked
• riskLevel: 'low', 'medium', 'high', or 'critical'

CLI usage

npx llmverify verify "Your text here"

The CLI returns the same structured output as the SDK. Useful for testing rules, CI/CD pipelines, and one-off checks.

Local HTTP API

LLMVerify also exposes a local HTTP API for non-JavaScript environments:

curl -X POST http://localhost:7331/verify \
  -H "Content-Type: application/json" \
  -d '{"input": "user message", "output": "model response", "checks": ["pii-redaction"]}'

The HTTP API runs locally. No data leaves your machine on the free tier.

What LLMVerify cannot prove

LLMVerify is a triage tool, not a truth oracle. Being explicit about what it cannot do:

Cannot definitively prove hallucinations. The hallucination risk indicators are pattern-based signals, not ground-truth verification. A model can produce a factually incorrect statement that does not trigger any hallucination pattern. LLMVerify flags risk indicators, not truth values.

Cannot verify semantic correctness. If the model says "the capital of France is London," LLMVerify will not flag this unless it matches a specific pattern. Semantic correctness checking requires an LLM judge or a knowledge base comparison.

PII detection is regex-based. Approximately 90% for standard formats (SSNs, phone numbers, email addresses, credit card numbers). Lower for variations, non-standard formats, or PII embedded in natural language. If a model says "the person lives at the house with the red door on Elm Street," LLMVerify will not flag this as an address.

Prompt injection detection is pattern-based. Known patterns like "ignore previous instructions" and "you are now DAN" are caught. Novel or obfuscated injections can evade detection. An attacker who rephrases the injection in a way the patterns do not match will bypass the check.

Does not replace human review. LLMVerify triages output at scale. It flags high-risk content for human review. It does not approve content for publication without human judgment.

Free tier limits

The free tier runs 100% locally. No network calls. No model calls. No data leaves your machine. The limit is 500 verification calls per day, tracked locally.

ML-enhanced detection features (semantic checks, novel injection detection) require the paid tier and an explicit API key. The free tier is pattern-based only.

No MCP server

LLMVerify does not expose an MCP server. This is a deliberate choice. MCP servers are for AI agent tool discovery. LLMVerify is a safety layer that sits between the LLM and the user — it should not be a tool the LLM can call, because the LLM could choose not to call it.

Integration is via the SDK (in your application code), the CLI (for testing and CI), or the local HTTP API (for non-JavaScript environments).

Where it fits

LLMVerify is the RUNTIME I/O layer in the Developer Security family. It checks what the model actually sends and receives. It does not check source code (that is AI AppSec) or tenant boundaries (that is MCP Tenant Isolation).

The three layers are independent by design. Each catches what the others cannot. If you ship AI applications, run all three. The cost is minutes of setup and milliseconds of runtime. The cost of skipping the runtime layer is the PII you do not catch in model output.