SADF: Synthetic Agent Deception Framework

Abstract

Agents that can call tools, sign transactions, or execute code are being deployed at scale with no standardized methodology to test them for prompt-injection and coercion failures. SADF (Synthetic Agent Deception Framework) addresses this gap with an eight-class taxonomy of agentic attack vectors — Tool Call Hijacking, Output Poisoning, Cross-Tool Injection, Memory Poisoning, RAG Poisoning, Delegated Authority Abuse, Multi-Agent Propagation, and Context Boundary Violation — paired with an empirical evaluation harness validated across 5,119 real runs spanning eight architectures (Claude Haiku, Claude Sonnet, Llama 3.2, GPT-5.4, and four orchestration frameworks: LangChain, AutoGen, CrewAI, SmolAgents).

Three findings anchor the framework. First, naive substring-matching scoring overstates true compromise rates by up to 6.5x when models quote attack indicators verbatim while refusing — a measurement bug we caught and corrected in our own data (4.1x on Claude Haiku, 6.5x on Claude Sonnet). Second, orchestration framework choice is an independent attack surface: identical backend models show a 2.6x spread in compromise rate (11.9% to 31.1%) depending solely on the framework wrapping them. Third, injection resistance and data-integrity resistance are separate properties — a case study demonstrates a model resisting two direct prompt-injection attempts, then still executing a real financial loss when its own correctly-configured automation rule was fed manipulated input data.

Independent validation using Garak confirmed that identical attack categories can show up to 4.5x variance in success rate depending on payload phrasing alone, reinforcing that single-payload testing materially understates real exposure.

This is disclosed as Phase 1 of an ongoing evaluation: session-persistence testing (designed but not yet implemented), six additional target architectures, and a controlled multi-agent-amplification comparison remain explicit future work. All code, datasets, and run logs are public. All evaluations were conducted against a simulated tool environment — no real filesystem, network, mail system, or third-party service was accessed at any point.