[REVIEW] agent-security: add remote tool provenance and registry drift gates

[Alex-Parejo]

Skill Being Reviewed

Skill name: agent-security
Skill path: skills/ai-security/agent-security/

False Positive Analysis

Benign architecture that can be over-reported as unsafe if tool provenance is not separated from tool capability:

agent_runtime:
  remote_tools:
    - name: jira.search_issues
      source: mcp://internal-tools.company.example
      transport: mTLS
      publisher: platform-security
      manifest_digest: sha256:8c5...
      allowed_actions: ["search", "read"]
      write_actions: []

Why this is a false positive risk:
The skill correctly asks reviewers to inspect tool registration breadth, least privilege, HITL gates, and auditability. But it does not distinguish a remotely discovered tool with a verified publisher, pinned manifest, signed transport, and read-only action set from a dynamically discovered tool whose schema can change without review. A review that only sees "remote MCP/tool server present" can overstate risk when the trust root and manifest integrity are actually controlled.

Coverage Gaps

Missed variant 1: Dynamic tool manifest drift after approval

{
  "server": "mcp://tools.example",
  "manifest_reviewed_at": "2026-06-01T12:00:00Z",
  "reviewed_tools": ["repo.search", "repo.read_file"],
  "runtime_tools": ["repo.search", "repo.read_file", "repo.delete_branch", "workflow.dispatch"],
  "manifest_pin": null
}

Why it should be caught:
The skill reviews dynamic vs static tool sets, but does not require evidence that a runtime tool registry matches the reviewed registry. In MCP/plugin-based agents, capabilities can appear through server upgrades, feature flags, tenant config, marketplace installs, or remote schema changes. If the system approves an architecture based on a stale registry, a later destructive tool can bypass the original risk review without any code change in the agent application.

Missed variant 2: Remote tool server identity is trusted by URL, not by publisher or artifact integrity

tools:
  - url: https://tools.partner.example/mcp
    auth: bearer_from_env
    allowed: true
    certificate_pinning: false
    publisher_attestation: missing
    manifest_signature: missing
    schema_digest: missing

Why it should be caught:
The current process checks credentials and inter-agent trust, but it does not ask reviewers to verify tool-server supply chain controls: server identity, manifest signing, schema digest pinning, publisher ownership, transport security, and update approval. A compromised tool server or DNS/hosting takeover can silently alter descriptions, parameter schemas, or available operations. That is different from prompt injection: the agent may follow a legitimate-looking tool contract that was swapped underneath it.

Missed variant 3: OAuth/consent scopes are granted at connection time and later exceed the agent task

User connects Google Workspace integration.
Consent screen grants: Gmail read/write, Drive full access, Calendar write.
Agent task: summarize one shared document.
Runtime policy: trusts the integration token and does not downscope per task.

Why it should be caught:
The skill covers credential scope generally, but OAuth and user-consented integrations need a separate evidence gate. Many agent tools operate with delegated user tokens, not service accounts. The review should record consented scopes, token audience, refresh-token lifetime, per-task downscoping, revocation path, and whether the agent can use one user's delegated token for another user's workflow. Without this, least-privilege review can miss excessive delegated authority even when cloud IAM is clean.

Edge Cases

• A tool marketplace or MCP server may expose only safe tools today but add write/delete operations after an automatic update; reviews need manifest pinning or update approval evidence.
• Tool descriptions are security-relevant because models use them for planning. A malicious or compromised remote tool can describe a destructive action as read-only unless the broker enforces policy from a separate signed capability model.
• Multiple tenants can share the same remote tool server while requiring different capability sets; one tenant's approved actions should not become another tenant's available tools.
• OAuth refresh tokens can outlive the session that justified access. Agent reviews should verify token revocation, re-consent, and task-bounded access, not only initial consent.
• A remote tool can return schema-valid output that contains instructions for the next agent step; provenance and taint labels should survive into downstream tool-policy decisions.

Remediation Quality

• Fix resolves the vulnerability
• Fix doesn't introduce new security issues
• Fix doesn't break functionality
• Issues found: Add a remote tool provenance and runtime registry verification step. Require reviewers to capture tool-server identity, publisher, manifest/schema digest, signing or attestation status, transport trust, runtime registry export, update policy, OAuth/delegated scopes, and per-task downscoping. Add Not Evaluable outcomes when runtime registry or consent-scope evidence is unavailable.

Comparison to Other Tools

Tool / Framework	Catches this?	Notes
OWASP Agentic AI Threats	Partial	Covers excessive agency and tool misuse, but local review still needs concrete tool registry and provenance evidence.
OWASP LLM Top 10	Partial	LLM tool/plugin risks are relevant, but it does not by itself prove remote manifest integrity or consent downscoping.
SLSA / artifact signing patterns	Partial	Useful model for provenance and attestation, but must be adapted to tool manifests, schemas, and MCP/plugin releases.
OAuth security review	Partial	Can validate consent and token scope, but must be tied to agent task boundaries and tool execution policy.
Runtime MCP/tool registry export	Yes when available	Best evidence for what tools the agent can actually invoke at runtime, especially after feature flags or server updates.

Overall Assessment

Strengths:

• Strong architecture checklist for permissions, least privilege, HITL, blast radius, audit, rollback, and multi-agent trust.
• Good warnings about dynamic tool sets, runtime tool injection, shared state, and broad credentials.
• Useful output format that can be extended without changing the skill's core structure.

Needs improvement:

• Add remote tool/MCP/plugin provenance evidence, not just local tool registration review.
• Require runtime tool registry comparison against the reviewed registry.
• Treat tool descriptions and schemas as security-sensitive artifacts that need integrity controls.
• Add OAuth/delegated-token scope review separate from service-account IAM.
• Add Not Evaluable reason codes for missing manifest digest, publisher identity, registry export, update policy, or consent-scope evidence.

Priority recommendations:

1. Add a "Remote Tool Provenance and Registry Drift" step before permission scoring: capture server identity, publisher, manifest digest, schema digest, signature/attestation, transport trust, update policy, and runtime registry export time.
2. Expand the Agent Inventory or add a Tool Registry table with Reviewed manifest digest, Runtime digest, Publisher, Transport, Update approval, OAuth scopes, Downscoped per task?, and Evidence confidence.
3. Add findings guidance: unpinned remote destructive tools should be High/Critical; docs-only tool provenance should be Not Evaluable; broad delegated OAuth scopes without per-task downscoping should be High when write-capable.
4. Add pitfalls for trusting remote tool URLs, assuming marketplace installs are static, treating tool descriptions as harmless text, and reviewing service-account IAM while missing delegated user-token scope.

References

• OWASP Agentic AI Threats and Mitigations: https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/
• OWASP Top 10 for LLM Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/
• NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
• SLSA Provenance: https://slsa.dev/spec/v1.0/provenance
• IETF OAuth 2.0 Security Best Current Practice: https://datatracker.ietf.org/doc/rfc9700/

Bounty Info

• I have read and agree to the CONTRIBUTING.md bounty terms
• Preferred payment method: PayPal; payment details can be provided privately after acceptance.

[Alex-Parejo]

Opened follow-up improvement PR implementing this review: #177

[JamesJi79]

Review: agent-security (#176) - remote tool provenance + registry drift gates

Reviewer: Hermes Agent
Date: 2026-06-03
Bounty: $25
Skill: /private/tmp/SecuritySkills/skills/ai-security/agent-security/SKILL.md
Implementation commit: bc764e1 by Alex-Parejo on origin/pr/177
Status on main (f4f3374): NOT MERGED — changes exist only on PR branch origin/pr/177

---

Summary

Issue #176 requests the addition of remote tool provenance and registry drift gates to the agent-security skill. The implementation is a single commit (bc764e1) adding 46 insertions across 4 file regions. The changes have not been merged to main — they exist on the origin/pr/177 tracking branch.

---

What Was Added

1. Context table — two new rows

New Context Item	Evidence Source	Why It Matters
Runtime tool registry export	Agent runtime, MCP/tool broker admin endpoint, plugin inventory	Confirms what tools are actually invokable after dynamic registration, feature flags, and remote server updates
Remote tool provenance	MCP manifests, plugin marketplace metadata, tool server release attestations	Shows whether remote tool schemas and descriptions come from a trusted publisher and pinned version

Also augmented the "Agent identity and credential management" row to include OAuth consent records and delegated-user scope awareness.

2. Step 1 — four new bullet points

• Delegated credential scope — OAuth/user-delegated token scoping per task and tenant
• Dynamic vs. static tool sets — now includes "Does the runtime registry match the reviewed registry?"
• Remote tool provenance — publisher identity verification, manifest/schema pinning/signing/attestation
• Tool manifest drift — can a remote server add write/delete/deploy/send actions after review without explicit approval?

3. Detection methods — expanded search terms

Added search terms for remote tool and delegated-auth evidence: mcp, tool_server, plugin, manifest, schema, digest, signature, attestation, oauth, scope, refresh_token, consent.

4. New evidence table: "Remote tool provenance and registry drift evidence"

Evidence Item	Secure State	Finding If Missing
Runtime registry export	Reviewed tool list matches runtime	Medium — stale declarations
Manifest/schema digest	Pinned or signed	High — silent drift
Publisher identity	Verified	High — untrusted source
Transport trust	mTLS or equivalent	High — tampered in transit
Update approval	New actions require approval	High — destructive actions appear post-review
Delegated OAuth scopes	Task-bounded and revocable	High — excessive delegated authority
Tool description integrity	Tied to reviewed manifest	Medium — model planning influenced by changed descriptions

5. Permission model matrix — two new rows

Principle	Finding If Absent
Scoped delegated auth (OAuth scopes match task/tenant/session)	High — excessive delegated authority
Registry integrity (runtime matches reviewed manifest)	High — unreviewed capabilities available

6. Findings table — four new conditions

Condition	Severity
Remote tool manifests/schemas unpinned, can change without approval	High
Runtime tool registry cannot be exported or compared	Medium
Delegated OAuth scopes exceed task requirements	High
Tool server publisher identity or transport trust unverifiable	High

7. Output format additions

• Tool Registry and Provenance table in output template (9-column evidence register for each tool/MCP server)
• Not Evaluable Items section — structured table to document missing evidence with risk justification
• "Remote Tool Provenance" added as a Review Area option in findings and posture summary
• "Remote Tool Provenance" row added to Architecture Security Posture Summary

8. Common Pitfalls — two new entries

• intel: devsecops/compliance social updates 2026-03-25 #6: Treating remote tool manifests as static documentation — pin/sign manifests, export runtime registry, require approval for new side-effecting actions
• intel: appsec community updates 2026-03-29 #7: Reviewing service-account IAM while missing delegated user tokens — record consented scopes, token audience, refresh lifetime, revocation path

9. References — two new entries

• [REVIEW] model-supply-chain: make torch.load guidance version-aware and add trust_remote_code coverage #15: SLSA Provenance (slsa.dev/spec/v1.0/provenance)
• Improve model supply chain loader and provenance checks #16: OAuth 2.0 Security Best Current Practice (RFC 9700)

---

Quality Assessment

Coverage

The implementation thoroughly addresses both aspects of the issue:

• Remote tool provenance: Covered via publisher identity verification, manifest/schema pinning/signing, transport trust (mTLS), tool description integrity, and SLSA provenance references.
• Registry drift: Covered via runtime registry export comparison, update approval gates for new actions, manifest digest verification, and the "registry integrity" principle in the permission model matrix.

Integration Quality

• Changes are integrated directly into Step 1 (Agent Permission Model Review), which is the correct location — remote tool provenance is a permission model concern.
• The new context table rows prime the reviewer to gather the right evidence upfront.
• The evidence table and findings table provide concrete detection criteria with severity ratings.
• Output format additions include structured tables for reporting findings.

Gaps

1. No mapping to OWASP Agentic AI threat categories in the new findings table rows — the existing findings don't list OWASP mappings either, so this is consistent but could be improved.
2. Registry drift detection methods rely on Grep patterns but no specific grep patterns were added for registry drift detection (unlike steps 6-7 which provide shell command snippets). Adding a # Find registry drift grep snippet would be valuable.
3. No test or validation script — the change is purely documentation/skill definition, which is appropriate for a SKILL.md, but there's no automated way to verify the skill template parses correctly.

Positives

• Coverage of both static provenance (publisher identity, signed manifests) and dynamic concerns (runtime registry comparison, update approval gates) is comprehensive.
• The new "Not Evaluable Items" section in the output format is a smart addition — it forces reviewers to acknowledge evidence gaps rather than silently skipping them.
• The two new Common Pitfalls (intel: devsecops/compliance social updates 2026-03-25 #6 and intel: appsec community updates 2026-03-29 #7) are well-written and address real-world gaps.

---

Verdict

Criterion	Rating
Correctness	Pass — changes are accurate and well-scoped
Completeness	Pass — both provenance and drift are addressed
Integration	Pass — changes slot cleanly into existing structure
Depth	Pass — evidence tables and severity ratings provide actionable guidance
Documentation	Pass — new common pitfalls and references add useful context
Merge Status	Blocked — changes exist on origin/pr/177 but not merged to main

Overall: Approved. The implementation satisfies the requirements of issue #176. It adds comprehensive remote tool provenance gates (publisher identity, manifest signing, transport trust) and registry drift gates (runtime registry comparison, update approval, delegated OAuth scope management) with appropriate severity ratings and output templates. The only action needed is merging the PR branch origin/pr/177 into main.

---

Files

• Reviewed: /private/tmp/SecuritySkills/skills/ai-security/agent-security/SKILL.md (HEAD on main — 589 lines, without the changes)
• Implementation: commit bc764e16c2f394907e872d7ab9604397337bb3b4 on origin/pr/177 — applies to the same file
• Review saved to: /private/tmp/review_176.md