Why Prompt-Level Coding Skills Drift: Benchmarking 24 AI Agent Skill Packs Across an 8-Domain Deterministic SDLC Evaluation Harness

When an autonomous coding agent generates a Python payment splitter or a TypeScript React checkout table and runs pytest or vitest, a green checkmark (3 passed in 0.12s) creates a dangerous illusion of production readiness.

In our empirical audit of 1,600 zero-prompt LLM code-generation runs across 8 software engineering domains, 91.63% (1,466 / 1,600) passed standard happy-path unit tests. Yet when those exact same 1,466 "passing" modules were subjected to adversarial software delivery lifecycle (SDLC) invariants—NaN/Infinity boundary injection, multi-seed cent-conservation laws, cross-tenant OWASP IDOR probes, WCAG 2.2 sRGB relative luminance math, and $N=100 \to 5,000$ asymptotic scaling checks—81.86% (1,200 / 1,466) failed silently.

Why? Because an LLM optimized via Reinforcement Learning from Human Feedback (RLHF) to satisfy surface-level unit tests takes the path of least token resistance: it writes floating-point division without remainder reconciliation, concatenates SQL filter strings without tenant scoping, picks low-contrast #94a3b8 slate text on #ffffff backgrounds (2.54:1 contrast), and nests .find() inside .map() ($O(N^2)$).

In Skills Evaluation Lab—and its companion research systems HonestSteer (.spipe) and VeriQuant AI—we replace subjective "LLM-as-a-Judge" prose grading and happy-path unit tests with an 8-Domain Deterministic SDLC Evaluation Harness paired with 24 operational SKILL.md playbooks.

1. The Verification Gap: Happy-Path Unit Tests vs. Deterministic 8-Gate SDLC Rubrics

Traditional coding benchmarks (such as HumanEval or basic CI unit checks) test whether $f(x_{\text{nominal}}) = y_{\text{nominal}}$. They do not test the non-functional and security invariants that actually cause production outages:

1. Silent Accounting Drift: Splitting 1000 cents across 3 recipients via Math.round(1000 / 3) yields 333 + 333 + 333 = 999 cents—losing 1 cent per transaction without failing a nominal unit test that only checks split(900, 3).
2. Cross-Tenant IDOR & Injection: An endpoint that returns SELECT * FROM invoices WHERE id = :id passes every functional test for User A, while allowing User B to read User A's invoices by omitting AND tenant_id = :jwt_tenant_id.
3. Asymptotic Hot-Path Regressions: An event joiner that uses events.map(e => users.find(u => u.id === e.userId)) runs in 0.2 ms at $N = 50$ in unit tests, but stalls the Node.js event loop for 420 ms at $N = 10,000$ in production.

The diagram below contrasts Partition A (Zero-Prompt Generation + Happy-Path Unit Check) against Partition B (Skill-Guided Generation + 8-Gate Deterministic SDLC Harness):

2. Inside the 8 Deterministic Evaluation Gates

Rather than asking a judge LLM "Does this code look secure and fast on a scale of 1 to 10?", skills-eval-lab/evaluate_8_domains.js executes eight mathematical and runtime probes against every generated module:

Gate 1 & Gate 2: Boundary Fuzzing and Multi-Seed Conservation (01_coding, 02_testing)

Every financial or allocation function is tested across 500 randomized seeds with negative numbers, 0, Number.MAX_SAFE_INTEGER, NaN, and Infinity. For valid inputs, the harness asserts the exact conservation identity:

$$ \forall B \in \mathbb{Z}{\ge 0}, \quad \sum{i=0}^{K-1} \text{alloc}_i(B) \equiv B $$

Gate 3: OWASP Tenant IDOR & Parameterized AST Probe (05_security)

The security harness injects spoofed cross-tenant JWT contexts (jwt.tenant_id = "tnt_attacker", resource.tenant_id = "tnt_victim") and SQL metacharacter payloads (' OR '1'='1). Any handler that returns a record across tenant boundaries or constructs SQL strings via template interpolation fails immediately.

Gate 4: WCAG 2.2 sRGB Relative Luminance Contrast (03_ui_ux)

Instead of visually eyeballing CSS files, the UI gate parses hex color pairs (fgHex, bgHex) and computes exact CIE relative luminance $L$ per W3C WCAG 2.2:

$$ C_{\text{linear}}(c) = \begin{cases} \dfrac{c}{12.92} & \text{if } c \le 0.03928 \[6pt] \left(\dfrac{c + 0.055}{1.055}\right)^{2.4} & \text{otherwise} \end{cases}, \qquad CR = \frac{\max(L_1, L_2) + 0.05}{\min(L_1, L_2) + 0.05} \ge 4.5 $$

Gate 5: Empirical Asymptotic Scaling Ratio (06_performance)

To catch hidden $O(N^2)$ loops without relying on fragile regexes, the performance gate measures execution time at $N_1 = 100$ and $N_2 = 5,000$ ($50\times$ input growth). An $O(N)$ hash-indexed join scales by $\approx 45\text{--}60\times$, whereas an $O(N^2)$ nested scan scales by $50^2 = 2,500\times$:

//benchmarks/agent-skills-sdlc-gates/harness.mjs (Excerpt)
export function verifyEmpiricalComplexity(fn, buildInput) {
  const inputSmall = buildInput(100);
  const inputLarge = buildInput(5000); // 50x input size

  // Warmup JIT
  for (let i = 0; i < 20; i++) fn(inputSmall);

  const t0 = performance.now();
  for (let i = 0; i < 50; i++) fn(inputSmall);
  const avgSmallMs = (performance.now() - t0) / 50;

  const t1 = performance.now();
  fn(inputLarge);
  const largeMs = performance.now() - t1;

  const scalingRatio = largeMs / Math.max(avgSmallMs, 0.001);
  // Linear O(N) scales ~50x; quadratic O(N^2) scales ~2500x. Hard cutoff at 120x.
  return { passed: scalingRatio <= 120, scalingRatio };
}

Gates 6, 7 & 8: Concurrency Semaphores, Zero-Leakage Splits, and Strict Schemas (04_system_design, 07_data_analytics, 08_ml_ai)

• Gate 6 (System Design): Injects transient faults into an async job queue and verifies that active worker concurrency never exceeds max_concurrency = 8 and dead-letter queues preserve poison payloads.
• Gate 7 (Data Analytics): Verifies out-of-order event ingestion idempotency and exact percentile aggregation.
• Gate 8 (ML / Quantitative AI): Enforces strict temporal/group disjointness ($\text{Train_IDs} \cap \text{Test_IDs} = \emptyset$) before model fitting—a principle we extended into VeriQuant AI to prevent look-ahead bias in autonomous financial alpha discovery.

3. Reproducible Benchmark: Zero-Prompt Baseline vs. Skill-Guided Agents ($N = 1,600$)

We executed the standalone evaluation benchmark at benchmarks/agent-skills-sdlc-gates/harness.mjs (npm run bench:skills) across $N = 1,600$ candidate module evaluations (200 per domain across all 8 domains), comparing Cohort 1 (Zero-Prompt Baseline) against Cohort 2 (Skill-Guided Agent with 24 SKILL.md Playbooks + Pre-Commit Self-Audit):

$ npm run bench:skills
========================================================================
8-Domain Deterministic SDLC Skill Evaluation Benchmark (N = 1,600 runs)
Environment: Node v24.14.0 | win32 x64 | Intel(R) Core(TM) i3-4030U CPU @ 1.90GHz
========================================================================
Cohort 1: Zero-Prompt Baseline (No SKILL.md Operational Playbook)
  Happy-Path Unit Test Pass Rate    : 1466/1600 (91.63%)
  8-Gate Deterministic SDLC Pass    : 400/1600 (25.00%)
  Silent Production Defects Caught  : 1200/1466 happy-path passes (81.86% false-positive unit tests)
------------------------------------------------------------------------
Cohort 2: Skill-Guided Agent (24 SKILL.md Playbooks + 8-Gate Rubric)
  Happy-Path Unit Test Pass Rate    : 1600/1600 (100.00%)
  8-Gate Deterministic SDLC Pass    : 1600/1600 (100.00%)
  Per-Candidate Gate Latency (us)   : p50=2.30us | p95=7.40us | p99=20.30us
========================================================================

4. Beyond Prompt Rules: Representation-Level Steering (.spipe) when Context Exceeds 30k Tokens

Even a well-structured SKILL.md playbook operates inside the model's token context window. As we demonstrated in HonestSteer for the NeurIPS 2026 Steerability Challenge, system-prompt constraints suffer from attention dilution once an agent trajectory exceeds $25\text{k}–30\text{k}$ tokens.

When prompt-level compliance degrades on long-horizon tasks, we pair our deterministic post-generation SDLC gates with representation-level activation steering using the declarative .spipe DSL:

$$ h'{l, t} = \text{Norm}!\left(h{l, t} + \alpha \cdot \mathbb{I}(\cos(h_{l, t}, v_{\text{risk}}) > \tau) \cdot v_{\text{steer}}^{(l)}\right) $$

By applying norm-preserving contrastive steering vectors at transformer blocks $l \in ^{[14] [20]}$ only when the hidden state's cosine projection onto a risk subspace exceeds $\tau$ (Conditional_CAST), we suppress sycophantic test-cheating and unsafe shortcuts inside the residual stream while preserving general coding perplexity.

5. Antithesis: When Deterministic Invariant Harnesses Are Insufficient

While deterministic 8-gate harnesses strictly dominate LLM-as-a-Judge for algorithmic, security, numerical, and accessibility verification, they have two boundaries:

1. Open-Ended Information Architecture & Visual Aesthetics: Mathematical contrast gates verify that text is legible ($CR \ge 4.5:1$), and DOM probes verify that interactive controls have accessible labels, but they cannot judge whether a dashboard hierarchy matches a specific brand's emotional tone. Use deterministic gates as Stage 1 (blocking CI gate) and reserve multimodal vision-LLM critique for Stage 2 (layout & copy polish).
2. Micro-Benchmark Timing Jitter on Shared CI Runners: Empirical complexity ratios ($T(50N)/T(N)$) can spike if a noisy neighbor steals CPU cycles mid-measurement on a shared GitHub Actions runner. Always pair wall-clock scaling ratios with deterministic operation counters (e.g., counting comparator invocations or hash-map lookups via a proxy wrapper) so CI never flakes under CPU contention.

References & Further Reading

1. Claessen, K., & Hughes, J. (2000): QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs — ICFP '00 (Foundations of Property-Based Invariant Testing). ↩︎
2. OWASP Foundation (2025): Authorization Testing Guide — Broken Object Level Authorization (BOLA / IDOR) & Parameterized Query Verification. ↩︎
3. Zou, A., et al. (2023): Representation Engineering: A Top-Down Approach to AI Transparency — arXiv:2310.01405. ↩︎
4. Turner, A., et al. (2023): Activation Addition: Steering Language Models Without Optimization — arXiv:2308.10248. ↩︎
5. Source Repositories:
   • Skills Evaluation Lab: github.com/akmalkhaniub/skills-eval-lab ↩︎
   • HonestSteer (.spipe Activation Steering): github.com/akmalkhaniub/steerability-challenge-2026 ↩︎
   • VeriQuant AI (Zero-Leakage Alpha Factory): github.com/akmalkhaniub/agenthon-quant-ai ↩︎