Scanner / Precision

Precision lineage

By Michael K Onyekwere

The AgentScore scanner is a regex-based detector. Regex cannot tell a database method call apart from a shell exec, or a test fixture apart from a real credential, without context. Scanner v2.1 ships a mitigator system that scans a window around each match for known sanitizer wrappers or test-fixture markers and downgrades the severity when one fires.

The list below is every mitigator the scanner has gained, paired with the public report that motivated it. Each entry was driven by a real maintainer interaction, not a hypothetical edge case.

This page is the scanner's public memory. If you want the full maintainer loop, start with the NodeBench case study, then compare it with the live package report.

Scanner version

v2.2

Current ruleset digest

fc668f21b041

How mitigators work

When a finding fires, the scanner looks at a 2,000-character window around the match. If a sanitizer pattern (like validateCommand, execFile, or a database-shaped .exec()) hits in that window, the severity is downgraded. The original finding stays visible with an annotation showing which mitigator fired and where, so you can audit the call yourself.

The downgrades: command_injection, unsafe_eval, sensitive_file_access go to LOW. hardcoded_secret goes to MEDIUM (a placeholder is still an information disclosure about test infrastructure, just not a credential).

Precision is bounded by what regex can express. A renamed sanitizer or a database method that does not match the variable-name list still produces a false positive. Real data-flow analysis is on the v2.2 roadmap.

Recent mitigator additions

ruleset post-fix-class-2

Iteration continues: three sanitizer-class additions + one test-fixture-class addition from the post-fix sample-check

The 2026-05-18 file-gate fix corrected the documentation_context false-negative. With the v2.2 + file-gate scanner running through the 7-day backlog, the command_injection class running count stabilised around 20 packages rather than dropping to zero — because most of those 20 are real static-analysis hits the scanner correctly continues to flag, not residual false positives. A fresh 5-package sample of the post-fix corpus (beecork, memex-mvp, @piyushdua/engram-dev, agentic-flow, @kevinrabun/judges) found a mix: 2 real-pattern in single-user CLI threat models that the scanner correctly keeps at HIGH, 3 with sanitiser or fixture patterns the scanner didn't recognise yet. Shipping the missing recognisers now.

Patterns added

sanitizer
/\b(?:shellQuote|shell_quote|shellEscape|shellquote)\s*\(/i + /\b(?:shq|sq|shell)\.quote\s*\(/ + /require\s*\(\s*['"`]shell-quote['"`]\s*\)/

Explicit shell-quoting helpers. @piyushdua/engram-dev wraps user input in shellQuote(record.path) before git worktree calls; that is the npm-ecosystem-standard name for shell-arg escaping. shell-quote is an npm package; shq.quote is a common alias.

sanitizer
/\$\{\s*JSON\.stringify\s*\(/

JSON.stringify inside the interpolation slot. memex-mvp's launchctl unload/load calls use exactly this pattern: execSync(`launchctl unload ${JSON.stringify(PLIST_PATH)}`). JSON.stringify wraps in double quotes with proper escaping, which is shell-safe in the contexts it appears.

test_fixture
/\bexpected(Outcome|Result|Behavior|Behaviour|Verdict|Action|Status|RuleIds?|Findings?|Matches?|Detections?|Violations?)\s*:/ + /\b(dangerous|unsafe|vulnerable|bad|insecure)(Patterns?|Cases?|Snippets?|Examples?|Code)\s*[:=]/i + /\b(benchmark|rule|detection)(Cases?|Corpus|Inputs?|Examples?|Fixtures?)\s*[:=]/i

Declarative test-fixture markers extended. @kevinrabun/judges is a code-judging benchmark tool whose dist/commands/benchmark-expanded.js embeds dangerous-code STRING LITERALS as test fixtures. Markers like expectedRuleIds, dangerousPatterns, benchmarkCases indicate the surrounding content is corpus data, not code.

test_fixture
/\/(?:[a-zA-Z-]*-)?(benchmark|rules?|judges?|policies|rulesets|fixtures?)-?(?:[a-zA-Z-]+\.)?(js|ts|mjs|cjs)\b/i + /\\`[^`\n]{0,200}\\\$\{/

Benchmark / rules / judges file-path heuristic for detection-corpus files. Plus a meta-template marker: source containing both \\` and \\${} escape sequences in close proximity means the surrounding string is a template-literal-as-data (string content that happens to render as a template literal). Both are strong signals of fixture content.

Outcome: Re-verified against the 5 fresh samples: 3 of 5 correctly downgraded by the new patterns (memex-mvp via ${JSON.stringify(...)}, @piyushdua/engram-dev via shellQuote(), @kevinrabun/judges via expectedRuleIds + ${ALL_CAPS}). The remaining 2 (beecork, agentic-flow) stay at HIGH because they are real interpolations of user config / process.argv into shell exec in CLI tools. Single-user CLI threat models do not change the static-analysis assessment. Pinned May 16 sample now shows 50% suppression (down from a previously claimed 75% that was inflated by the documentation_context misfire). The honest precision shape today: about half of the running command_injection class is correctly flagged real-pattern, about half is correctly downgraded FP via the iteratively-learned mitigator set. The tracker count is not a regression metric; it stabilises at the rate at which real patterns appear in new publishes.
ruleset documentation_context file gate

documentation_context only fires on actual documentation files (fa-mcp-sdk CRITICAL was silently downgraded)

Caught 2026-05-18 on session start. The 2026-05-16 v2.2 mitigator pass introduced a documentation_context category for markdown anti-pattern examples (after the claude-flow security tutorial caught itself). The matched-anywhere-in-file approach was too broad: the markdown heading pattern /^#{1,4}\s+\S/m fires on YAML comments (# foo), shell comments (# foo), TOML headers ([section]), HTML comments, and any other syntax that overlaps. Result: fa-mcp-sdk's CRITICAL hardcoded_secret in package/config/local.yaml (still present, four weeks after the original April 25 disclosure) was downgraded to MEDIUM because the YAML file's own header comments matched the markdown-heading regex. Same applied to the HIGH sensitive_file_access being downgraded by an unrelated path.join elsewhere in the same archive. The fa-mcp-sdk score recovered from 30/HIGH to 65/ELEVATED purely from our scanner becoming wrong; the maintainer did nothing. This is the false-negative class Codex's 2026-05-16 review warned about, materialised concretely.

Patterns added

engine
CATEGORY_FILE_GATES + DOC_FILE_RE

documentation_context patterns only fire when the matched file's extension is .md, .mdx, .markdown, .rst, .txt, .adoc, or .asciidoc. findMitigators now takes a filename argument and checks the gate before testing patterns in that category. Other categories (sanitizer, test_fixture) are not file-gated.

Outcome: fa-mcp-sdk@0.4.110 immediately re-flagged at 45/HIGH with CRITICAL hardcoded_secret restored on config/local.yaml after the fix. RULESET_DIGEST changes again so this re-scan registers as scanner_reassessment, not maintainer drift. Audit of scan_history since the 2026-05-16 mitigator pass found 8 findings across 7 distinct packages had been wrongly downgraded by documentation_context on non-doc files. Of those 7, three had material public-record corrections (fa-mcp-sdk 65/ELEVATED -> 45/HIGH, mcpbrowser 90/LOW -> 75/MODERATE, opencode-gitlab-dap 90/LOW -> 75/MODERATE). The other four kept the same public score because parallel sanitizer or test_fixture mitigators correctly downgraded the same findings independently. All seven were rescanned with the fixed scanner via scripts/rescan-doc-context-affected.cjs and the corrected severities are in scan_history now, rather than waiting 3-4 days for monitor cron rotation. Lesson recorded: the May 16 documentation_context category's heading pattern was too loose to ship as-is; categories that overlap with widespread comment syntax (YAML \#, shell \#, TOML headers) need file-gating from the start.
ruleset post-ea8aabc

Per-file iteration + all-matches walk (Codex review caught two scope leaks)

Codex review of the earlier May 16 mitigator pass surfaced two structural issues. First, the scanner read the whole gunzipped tarball as one buffer and ran findMitigators against +/-2000 chars in that buffer, so a README heading in one file could downgrade a real finding in another (cross-file leak). Second, even after per-file iteration, the scanner ran each pattern as a single .exec() per file, so an early benign shell call in a file would mask a later real unsafe one in the same file (same-file masking). Both fixed today.

Patterns added

engine
iterateTarFiles(buf)

Walk POSIX tar archive entry-by-entry, including GNU longname (L) and pax (x) extended header support. Mitigators run against the current file's content only. ~80 lines, no new dep.

engine
findAllMatches(pattern, content)

Iterate every match per pattern per file (capped at 200 to bound pathological inputs). Each match's severity is computed against its own +/-2000 char window. The worst severity across all matches wins for that file. Short-circuits on the first match that remains at the pattern's original severity.

engine
isScannableFile(filename)

Path filter that allows source-like extensions and excludes pure artefacts (node_modules, .git, coverage, .next, *.map, *.min.js, *.d.ts). Crucially does NOT skip dist/ or build/ — in published npm tarballs those are usually the actual executable code.

Outcome: Verification against the version-pinned 5-package sample (now version-pinned to prevent silent drift) shows 75% suppression rate on detected command_injection findings, not the 100% claimed in the entry below. The corrected number is more honest: memoir-cli@3.6.1's upgrade.js contains `exec(\`open "${url}"\`)` that the previous first-match-only iteration was masking behind an earlier match downgraded by path.join. The all-matches walk now surfaces it and flags HIGH. In single-user CLI threat models this pattern is benign; the scanner correctly cannot infer that without runtime context, and a HIGH static-analysis flag is the right output. SCANNER_VERSION bumped to 2.2 because tarball handling changed at engine level.
ruleset post-c7bb83c

Self-detected false-positive class: command_injection in browser/CLI MCP packages (31 advisories affected)

Internal tracking on 2026-05-15 showed 31 distinct browser/terminal/CLI-automation MCP packages flagged with HIGH command_injection in a 30-day window. Volume alone made a real-bug-per-package interpretation implausible. Sample-checked five (safari-mcp, brave-real-browser-mcp-server, memoir-cli, s3db.js, claude-flow): four of five were clear false positives, one ambiguous. Causes: postinstall codesign of internal binaries, database client .exec(`SELECT ...`) misidentified as child_process.exec, hardcoded ALL_CAPS constants treated as user input, numeric IDs from webhook payloads, and a security tutorial .md file that the regex caught literally with `// ❌ Dangerous: shell injection possible` as the matched line. The advisory pipeline had been auto-publishing HIGH severity on these patterns at ~one per day. Caught and shipped under our own steam before any maintainer push-back; the public-correction loop is the asset, applied to ourselves.

Patterns added

sanitizer
/\b(db|database|conn|connection|client|pool|prepared|stmt|sql|query|knex|prisma|this)\.\s*exec\s*\(/i

Extended the existing database-method allowlist to include `this.exec(`. s3db.js's remote-sqlite-client calls `this.exec(\`SELECT ...\`, [params])` for SQL, not shell.

sanitizer
/\b(__dirname|__filename|process\.cwd\(\)|path\.(?:join|resolve|dirname|basename))\b/

Compile-time identifiers within scope. safari-mcp's postinstall used `path.join(__dirname, '..', 'safari-helper')` and the regex treated that as user input.

sanitizer
/\$\{[A-Z][A-Z0-9_]{2,}\}/

ALL_CAPS interpolated identifiers strongly signal a compile-time constant. s3db.js's `${REPO_URL}` is a hardcoded repository URL.

sanitizer
/\$\{\s*(?:Number|parseInt|parseFloat)\s*\(/

Numeric coercion. A value passed through Number(), parseInt(), or parseFloat() cannot carry shell metacharacters.

sanitizer
/\$\{\s*[a-zA-Z_]+\.(?:number|id|index|count|length|size)(?:\s*\|\s*0|\s*\?\?\s*0)?\s*\}/

Properties guaranteed to be numeric. claude-flow's webhook example uses `${event.pull_request.number}` which is always an integer from GitHub.

sanitizer
/\b(codesign|signtool|notarytool)\b|\bgpg\s+--sign\b/

Code-signing toolchain. Postinstall scripts in macOS/Windows helper packages invoke these against package-internal binaries, never user input.

sanitizer
/execSync\s*\(\s*`npm\s+(?:view|pack|info)\s+\$\{[a-zA-Z_]+\}/

Auto-update scripts querying npm for known dependency names from package.json.

documentation_context
/```\s*(?:js|ts|javascript|typescript|jsx|tsx|json|bash|sh|shell|console)?\s*\n/

Triple-backtick code fences. Markdown documentation routinely contains example code, including anti-pattern examples.

documentation_context
/❌|✅|⚠️\s+(?:Dangerous|Unsafe|Bad|Don't|Avoid)/i

Markdown anti-pattern markers. claude-flow's v3-security-architect.md literally labelled the example shell-exec line `❌ Dangerous: shell injection possible` and the scanner caught the tutorial.

documentation_context
/\/\/\s*(?:❌|⚠️|Dangerous|Unsafe|Bad\s+example|Don't\s+do\s+this|Avoid\s+this|Anti-pattern|Vulnerable)\s*[:.]/i

English comment annotations marking code as an anti-pattern.

documentation_context
/^#{1,4}\s+\S/m

Markdown headings within scope. Strong signal the surrounding context is documentation, not executable source.

documentation_context
/<!--[\s\S]*?-->/

HTML / JSX comments commonly used in README, MDX, and docs.

Outcome: Local verification (scripts/verify-mitigators.cjs against the 5-package sample) shows 100% of detected command_injection findings now downgrade to LOW. Monitor cron will rescan the 31 affected packages organically over the next 3-4 days; their public advisories will pick up the corrected severity. The RULESET_DIGEST changed automatically because the mitigators set is part of digest material, so rescans register as scanner_reassessment (not maintainer-caused drift). Historic advisories on /security/advisories that were published under the pre-mitigator regex remain visible with their original severity recorded; readers can compare the original AGENTSCORE-2026-NNNN advisory to the current /report/<package> page to see the corrected scan. This is by design: we do not silently rewrite the public record. Corrections happen in the open.
ruleset 3185eb87b4ce

Database .exec and eval-in-message-strings (HomenShum/nodebench-ai#8)

Maintainer reviewed two HIGH findings against source. Confirmed three real command_injection sites and refactored to argv-based spawn. Correctly identified unsafe_eval as a false positive: the regex matched better-sqlite3's db.exec(`SQL`) and the literal word eval inside a recommendations.push string.

Patterns added

sanitizer
/\.exec\s*\(\s*[`'"]?\s*(SELECT|INSERT|UPDATE|DELETE|CREATE|ALTER|DROP|PRAGMA|VACUUM|BEGIN|COMMIT|ROLLBACK|TRUNCATE|REPLACE|MERGE|GRANT|REVOKE|EXPLAIN)\b/i

SQL keyword immediately after .exec(. Strong signal this is a database method, not child_process.exec.

sanitizer
/\b(db|database|conn|connection|client|pool|prepared|stmt|sql|query|knex|prisma)\.\s*exec\s*\(/i

Database-shaped variable names calling .exec, like better-sqlite3, pg, mysql2, prisma raw queries.

test_fixture
/\/[a-zA-Z]*[Ee]val[A-Z][a-zA-Z]*\.(js|ts|mjs|cjs)\b/

Files like selfEvalTools.js, llmJudgeEval.js, pipelineEval.js. The word eval refers to evaluation flow, not JavaScript eval.

test_fixture
/\b(?:recommendations?|messages?|errors?|warnings?|notes?)\s*\.\s*push\s*\(\s*[`'"]/i

Strings being pushed into a recommendations or messages array are message text, not executable code.

test_fixture
/\b(console|logger|log|debug|info|warn|error|trace)\s*\.\s*[a-z]+\s*\(\s*[`'"][^`'"]*\beval\b/i

console.log and friends emitting strings that contain the word eval.

Outcome: nodebench-mcp@3.2.1 rescored 55/ELEVATED to 85/LOW after refactor + mitigators. Both findings downgraded with explicit annotations.

Declarative test-fixture markers (claude-flow CRITICAL hardcoded_secret in dist/)

claude-flow shipped a manifest-validator with structural test fixtures inside dist/. Existing test_fixture rules expected files to live under tests/ or specs/, so they did not catch shipped-as-data fixtures.

Patterns added

test_fixture
/\babc(123|def|xyz)/i

Canonical fake-credential placeholders. claude-flow used sk-abc123-style test keys.

test_fixture
/\b(123|abcd){3,}/i

Long repeating placeholder sequences like abcdabcdabcd.

test_fixture
/\bexpected(Outcome|Result|Behavior|Behaviour|Verdict|Action|Status)\s*:/

Declarative fixture structure: { params: { ... }, expectedOutcome: 'deny' }. Pure data, not exploitable code.

test_fixture
/\b(should|must|will)(Fail|Pass|Reject|Allow|Block|Deny|Throw|Match)\b/i

Test-shaped predicate names sitting near otherwise-dangerous-looking literals.

test_fixture
/\/(?:[a-zA-Z-]*-)?(validator|sanitizer|detector|scanner|denier|filter)\.(js|ts|mjs|cjs)\b/i

Validator and sanitizer source files contain example dangerous strings by design.

Outcome: claude-flow's hardcoded_secret CRITICAL finding downgraded to MEDIUM. command_injection HIGH still flags pending a future mitigator pass.

Sanitizer wrappers (Agions/taskflow-ai#6)

Maintainer shipped v3.0.2 with a validateCommand wrapper around shell_exec (whitelist + dangerous-pattern detection) within 48h of the scan report. The scanner's HIGH command_injection finding for the same code was no longer the right severity once a guard was in place.

Patterns added

sanitizer
/\bvalidateCommand\b/

Direct match for the wrapper Agions introduced.

sanitizer
/\bsanitize(?:Command|Args|Input)?\b/i

Common naming convention for input sanitizers across the ecosystem.

sanitizer
/\bisAllowedCommand\b/i

Whitelist-style guards.

sanitizer
/\bexecFile\b/

execFile with an args array cannot shell-interpret. Different posture from exec.

sanitizer
/\bspawn\s*\(\s*[^,)]+,\s*\[/

spawn('cmd', [args]) array form bypasses the shell entirely.

Outcome: taskflow-ai went 45/HIGH to 60/ELEVATED on v3.0.2, then to 80/MODERATE on v4.0.0 after seven capabilities were deleted. Full arc closed in four days.

Reporting a precision gap

If the scanner flagged something it should not have, or missed something it should have caught, the report goes through public issue forms on the scanner repo. Detection-accuracy reports are not security disclosures and do not need confidentiality.

Real vulnerabilities in AgentScore infrastructure go to security@agentscores.xyz, not these forms.