diff --git a/docs/core-concepts/observability-tracing.md b/docs/core-concepts/observability-tracing.md index b341ec6..eca74b1 100644 --- a/docs/core-concepts/observability-tracing.md +++ b/docs/core-concepts/observability-tracing.md @@ -171,6 +171,36 @@ Every captured value is byte-capped at **4 KiB** (below the 5 KiB attribute soft **OTel semconv versioning note**: the GenAI semantic conventions moved from flat-string (`gen_ai.prompt`, `gen_ai.completion`) to structured (`gen_ai.input.messages`, `gen_ai.output.messages`) attributes. Forge emits only the **current** structured keys. Backends that only recognize the deprecated flat-string attributes will not show prompt / completion text on Forge spans — upgrade the backend's semconv mapping or use a span processor to translate. +### Guardrail spans (issue #161) + +The `LibraryGuardrailEngine` opens a child span around every gate evaluation, symmetric to the `guardrail_check` audit-event emission. Trace consumers see "PII was masked here" inline with the LLM and tool spans without having to pivot to the audit stream. + +| Gate | Span name | Where it nests | +|------|-----------|----------------| +| InputGate | `guardrail.input` | Child of the A2A handler span (CheckInbound runs at request entry) | +| ContextGate | `guardrail.context` | Child of `agent.execute` (BeforeLLMCall hook; one span per system message scanned) | +| ToolCallGate | `guardrail.tool_call` | Child of `agent.execute` (BeforeToolExec hook) | +| OutputGate | `guardrail.output` | Child of `agent.execute` (CheckOutbound + AfterToolExec hook) | +| StreamGate | `guardrail.stream` | Not auto-wired today; opened when `CheckStream` is called directly | + +Attribute reference: + +| Attribute | When set | Source | +|-----------|----------|--------| +| `forge.guardrail.gate` | Always | `Result.Gate` — single source of truth, matches `fields.gate` on the audit event | +| `forge.guardrail.decision` | Always | `Result.Decision` — `allow` / `mask` / `block` / `warn` | +| `forge.guardrail.violation_count` | Always | `len(Result.Violations)` | +| `forge.guardrail.type` | When violations present | First violation's `Type` field (`pii`, `moderation`, `security`, …) | +| `forge.guardrail.category` | When violations have category | First violation's `Category` (`ssn`, `email`, `hate_speech`, …) | +| `forge.tool.name` | `tool_call` + tool-output `output` spans | The tool the gate fired on | +| `forge.guardrail.evidence` | `capture_content: true` only | Redacted + truncated triggering content. For `mask` decisions: post-mask content. For `block` / `warn`: original content. Mirrors the audit-event evidence rule. | + +**Span status**: `block` decisions stamp OTel `Error` status with the violation summary as the status description — surfaces blocked invocations as red bars in the trace UI without custom attribute queries. `mask` / `warn` decisions leave the default OK status. + +**Default posture**: `forge.guardrail.evidence` is absent unless `capture_content: true`. The other five attributes are always present when a gate fires (cheap, no PII risk). When tracing is disabled, the noop tracer short-circuits and the spans are not produced at all. + +**Content-capture parity**: the evidence attribute uses the exact same `PrepareSpanContent(redact, maxBytes)` pipeline as `gen_ai.input.messages` and `forge.tool.args` — same vendor secret-token scrub, same 4 KiB byte cap, same `…[truncated:N]` marker. Operators get one mental model across all four content streams (LLM input / LLM output / tool args / tool result / guardrail evidence). + ## End-to-end propagation (Phase 5) Forge installs the W3C `tracecontext + baggage` composite propagator on the OTel global at startup. The JSON-RPC dispatcher extracts inbound `traceparent` + `baggage` headers before opening its own span, so multi-hop A2A flows show as one connected trace: diff --git a/forge-cli/runtime/guardrails_engine.go b/forge-cli/runtime/guardrails_engine.go index f8d11f1..4d71367 100644 --- a/forge-cli/runtime/guardrails_engine.go +++ b/forge-cli/runtime/guardrails_engine.go @@ -10,6 +10,7 @@ import ( "github.com/initializ/guardrails/models" "github.com/initializ/forge/forge-core/a2a" + "github.com/initializ/forge/forge-core/observability" coreruntime "github.com/initializ/forge/forge-core/runtime" "go.mongodb.org/mongo-driver/mongo" "go.mongodb.org/mongo-driver/mongo/options" @@ -50,6 +51,12 @@ type LibraryGuardrailEngine struct { logger coreruntime.Logger auditLogger *coreruntime.AuditLogger auditCfg GuardrailAuditConfig + // tracingCfg controls the OTel guardrail. span instrumentation + // added in #161. CaptureContent + Redact + MaxBytes follow the + // same posture as the #130 LLM-call content capture. Default zero + // value disables evidence capture; the spans themselves are still + // opened (cheap when the noop tracer is installed). + tracingCfg observability.TracingConfig } // NewFileGuardrailEngine creates a guardrail engine backed by a local @@ -115,6 +122,21 @@ func (e *LibraryGuardrailEngine) WithAuditLogger(al *coreruntime.AuditLogger, cf return e } +// WithTracing wires the runtime's TracingConfig so the engine can +// stamp forge.guardrail.evidence with the redact-then-truncate +// pipeline when CaptureContent is enabled. Same posture as the LLM +// call content capture from issue #130 — default off, opt-in per +// deployment, redact on by default when on. Returns the receiver for +// fluent construction. +// +// The guardrail. spans are opened unconditionally (the noop +// tracer's overhead is near-zero); CaptureContent only gates whether +// the evidence attribute is set. See issue #161. +func (e *LibraryGuardrailEngine) WithTracing(cfg observability.TracingConfig) *LibraryGuardrailEngine { + e.tracingCfg = cfg + return e +} + // structuredIfFileMode returns the StructuredGuardrails pointer only in file // mode. In DB mode the library loads config from MongoDB automatically. func (e *LibraryGuardrailEngine) structuredIfFileMode() *models.StructuredGuardrails { @@ -131,6 +153,22 @@ func (e *LibraryGuardrailEngine) CheckInbound(ctx context.Context, msg *a2a.Mess return nil } + // Span lifecycle (issue #161): open before the library call so the + // call's wall-clock duration is captured, finish in a deferred call + // with the resolved decision + content. evidenceContent + decision + // are mutated inside the switch below and read by the deferred + // finish. result starts nil — finishGuardrailSpan handles that case + // (the library-error path below sets result to nil too). + ctx, span := startGuardrailSpan(ctx, "input", "") + var ( + evidenceContent string + decisionString string + result *guardrails.Result + ) + defer func() { + finishGuardrailSpan(span, result, decisionString, evidenceContent, e.tracingCfg) + }() + result, err := e.manager.InputGate(ctx, guardrails.InputRequest{ Content: text, EntityID: e.agentID, @@ -141,6 +179,7 @@ func (e *LibraryGuardrailEngine) CheckInbound(ctx context.Context, msg *a2a.Mess }) if err != nil { e.logger.Warn("guardrail input gate error", map[string]any{"error": err.Error()}) + result = nil return nil } @@ -154,22 +193,29 @@ func (e *LibraryGuardrailEngine) CheckInbound(ctx context.Context, msg *a2a.Mess } e.logger.Info("guardrail input redaction applied", nil) e.emitGuardrailEvent(ctx, "", result.MaskedContent, guardrailResultMasked, result) + evidenceContent = result.MaskedContent + decisionString = guardrailResultMasked } case guardrails.DecisionBlock: desc := violationSummary(result) if e.enforce { e.emitGuardrailEvent(ctx, "", text, guardrailResultBlocked, result) + evidenceContent = text + decisionString = guardrailResultBlocked return fmt.Errorf("input blocked: %s", desc) } e.logger.Warn("guardrail input violation (warn mode)", map[string]any{"detail": desc}) e.emitGuardrailEvent(ctx, "", text, guardrailResultWarned, result) + evidenceContent = text + decisionString = guardrailResultWarned } return nil } // CheckOutbound validates an outbound (agent) message via OutputGate. // Masked content is applied in-place; blocked content returns an error -// only in enforce mode. +// only in enforce mode. One guardrail.output span per text part — the +// trace tree mirrors the part-level iteration. func (e *LibraryGuardrailEngine) CheckOutbound(ctx context.Context, msg *a2a.Message) error { for i, p := range msg.Parts { if p.Kind != a2a.PartKindText || p.Text == "" { @@ -177,35 +223,64 @@ func (e *LibraryGuardrailEngine) CheckOutbound(ctx context.Context, msg *a2a.Mes } original := p.Text - result, err := e.manager.OutputGate(ctx, guardrails.OutputRequest{ - Content: p.Text, - EntityID: e.agentID, - OrgID: e.orgID, - EntityType: guardrails.EntityTypeAgent, - StructuredGuardrails: e.structuredIfFileMode(), - ConfigVersion: e.configVersion, - }) - if err != nil { - e.logger.Warn("guardrail output gate error", map[string]any{"error": err.Error()}) - continue + blockErr := e.checkOneOutboundPart(ctx, msg, i, original) + if blockErr != nil { + return blockErr } + } + return nil +} - switch result.Decision { - case guardrails.DecisionMask: - if result.MaskedContent != "" { - msg.Parts[i].Text = result.MaskedContent - e.logger.Warn("guardrail output redaction applied", nil) - e.emitGuardrailEvent(ctx, "", result.MaskedContent, guardrailResultMasked, result) - } - case guardrails.DecisionBlock: - desc := violationSummary(result) - if e.enforce { - e.emitGuardrailEvent(ctx, "", original, guardrailResultBlocked, result) - return fmt.Errorf("output blocked: %s", desc) - } - e.logger.Warn("guardrail output violation (warn mode)", map[string]any{"detail": desc}) - e.emitGuardrailEvent(ctx, "", original, guardrailResultWarned, result) +// checkOneOutboundPart runs OutputGate over a single text part. Split +// from CheckOutbound so the per-part span has a clean lifetime (open +// at function entry, deferred close at exit) without juggling state +// across iterations. +func (e *LibraryGuardrailEngine) checkOneOutboundPart(ctx context.Context, msg *a2a.Message, i int, original string) error { + ctx, span := startGuardrailSpan(ctx, "output", "") + var ( + evidenceContent string + decisionString string + result *guardrails.Result + ) + defer func() { + finishGuardrailSpan(span, result, decisionString, evidenceContent, e.tracingCfg) + }() + + result, err := e.manager.OutputGate(ctx, guardrails.OutputRequest{ + Content: original, + EntityID: e.agentID, + OrgID: e.orgID, + EntityType: guardrails.EntityTypeAgent, + StructuredGuardrails: e.structuredIfFileMode(), + ConfigVersion: e.configVersion, + }) + if err != nil { + e.logger.Warn("guardrail output gate error", map[string]any{"error": err.Error()}) + result = nil + return nil + } + + switch result.Decision { + case guardrails.DecisionMask: + if result.MaskedContent != "" { + msg.Parts[i].Text = result.MaskedContent + e.logger.Warn("guardrail output redaction applied", nil) + e.emitGuardrailEvent(ctx, "", result.MaskedContent, guardrailResultMasked, result) + evidenceContent = result.MaskedContent + decisionString = guardrailResultMasked + } + case guardrails.DecisionBlock: + desc := violationSummary(result) + if e.enforce { + e.emitGuardrailEvent(ctx, "", original, guardrailResultBlocked, result) + evidenceContent = original + decisionString = guardrailResultBlocked + return fmt.Errorf("output blocked: %s", desc) } + e.logger.Warn("guardrail output violation (warn mode)", map[string]any{"detail": desc}) + e.emitGuardrailEvent(ctx, "", original, guardrailResultWarned, result) + evidenceContent = original + decisionString = guardrailResultWarned } return nil } @@ -218,6 +293,16 @@ func (e *LibraryGuardrailEngine) CheckToolCall(ctx context.Context, toolName, ar return args, nil } + ctx, span := startGuardrailSpan(ctx, "tool_call", toolName) + var ( + evidenceContent string + decisionString string + result *guardrails.Result + ) + defer func() { + finishGuardrailSpan(span, result, decisionString, evidenceContent, e.tracingCfg) + }() + result, err := e.manager.ToolCallGate(ctx, guardrails.ToolCallRequest{ ToolName: toolName, RequestBody: args, @@ -232,6 +317,7 @@ func (e *LibraryGuardrailEngine) CheckToolCall(ctx context.Context, toolName, ar "tool": toolName, "error": err.Error(), }) + result = nil return args, nil } @@ -240,12 +326,16 @@ func (e *LibraryGuardrailEngine) CheckToolCall(ctx context.Context, toolName, ar if result.MaskedContent != "" { e.logger.Warn("guardrail tool_call redaction", map[string]any{"tool": toolName}) e.emitGuardrailEvent(ctx, toolName, result.MaskedContent, guardrailResultMasked, result) + evidenceContent = result.MaskedContent + decisionString = guardrailResultMasked return result.MaskedContent, nil } case guardrails.DecisionBlock: desc := violationSummary(result) if e.enforce { e.emitGuardrailEvent(ctx, toolName, args, guardrailResultBlocked, result) + evidenceContent = args + decisionString = guardrailResultBlocked return "", fmt.Errorf("tool_call blocked: %s", desc) } e.logger.Warn("guardrail tool_call violation (warn mode)", map[string]any{ @@ -253,6 +343,8 @@ func (e *LibraryGuardrailEngine) CheckToolCall(ctx context.Context, toolName, ar "detail": desc, }) e.emitGuardrailEvent(ctx, toolName, args, guardrailResultWarned, result) + evidenceContent = args + decisionString = guardrailResultWarned } return args, nil @@ -268,6 +360,16 @@ func (e *LibraryGuardrailEngine) CheckToolOutput(ctx context.Context, toolName, return text, nil } + ctx, span := startGuardrailSpan(ctx, "output", toolName) + var ( + evidenceContent string + decisionString string + result *guardrails.Result + ) + defer func() { + finishGuardrailSpan(span, result, decisionString, evidenceContent, e.tracingCfg) + }() + result, err := e.manager.OutputGate(ctx, guardrails.OutputRequest{ Content: text, EntityID: e.agentID, @@ -282,6 +384,7 @@ func (e *LibraryGuardrailEngine) CheckToolOutput(ctx context.Context, toolName, "tool": toolName, "error": err.Error(), }) + result = nil return text, nil } @@ -290,12 +393,16 @@ func (e *LibraryGuardrailEngine) CheckToolOutput(ctx context.Context, toolName, if result.MaskedContent != "" { e.logger.Warn("guardrail tool output redaction", map[string]any{"tool": toolName}) e.emitGuardrailEvent(ctx, toolName, result.MaskedContent, guardrailResultMasked, result) + evidenceContent = result.MaskedContent + decisionString = guardrailResultMasked return result.MaskedContent, nil } case guardrails.DecisionBlock: desc := violationSummary(result) if e.enforce { e.emitGuardrailEvent(ctx, toolName, text, guardrailResultBlocked, result) + evidenceContent = text + decisionString = guardrailResultBlocked return "", fmt.Errorf("tool output blocked: %s", desc) } e.logger.Warn("guardrail tool output violation (warn mode)", map[string]any{ @@ -303,6 +410,8 @@ func (e *LibraryGuardrailEngine) CheckToolOutput(ctx context.Context, toolName, "detail": desc, }) e.emitGuardrailEvent(ctx, toolName, text, guardrailResultWarned, result) + evidenceContent = text + decisionString = guardrailResultWarned } return text, nil @@ -317,6 +426,16 @@ func (e *LibraryGuardrailEngine) CheckContext(ctx context.Context, content strin return content, nil } + ctx, span := startGuardrailSpan(ctx, "context", "") + var ( + evidenceContent string + decisionString string + result *guardrails.Result + ) + defer func() { + finishGuardrailSpan(span, result, decisionString, evidenceContent, e.tracingCfg) + }() + result, err := e.manager.ContextGate(ctx, guardrails.ContextRequest{ Content: content, EntityID: e.agentID, @@ -327,6 +446,7 @@ func (e *LibraryGuardrailEngine) CheckContext(ctx context.Context, content strin }) if err != nil { e.logger.Warn("guardrail context gate error", map[string]any{"error": err.Error()}) + result = nil return content, nil } @@ -335,16 +455,22 @@ func (e *LibraryGuardrailEngine) CheckContext(ctx context.Context, content strin if result.MaskedContent != "" { e.logger.Warn("guardrail context redaction", nil) e.emitGuardrailEvent(ctx, "", result.MaskedContent, guardrailResultMasked, result) + evidenceContent = result.MaskedContent + decisionString = guardrailResultMasked return result.MaskedContent, nil } case guardrails.DecisionBlock: desc := violationSummary(result) if e.enforce { e.emitGuardrailEvent(ctx, "", content, guardrailResultBlocked, result) + evidenceContent = content + decisionString = guardrailResultBlocked return "", fmt.Errorf("context blocked: %s", desc) } e.logger.Warn("guardrail context violation (warn mode)", map[string]any{"detail": desc}) e.emitGuardrailEvent(ctx, "", content, guardrailResultWarned, result) + evidenceContent = content + decisionString = guardrailResultWarned } return content, nil @@ -361,6 +487,16 @@ func (e *LibraryGuardrailEngine) CheckStream(ctx context.Context, chunk string) return chunk, nil } + ctx, span := startGuardrailSpan(ctx, "stream", "") + var ( + evidenceContent string + decisionString string + result *guardrails.Result + ) + defer func() { + finishGuardrailSpan(span, result, decisionString, evidenceContent, e.tracingCfg) + }() + result, err := e.manager.StreamGate(ctx, guardrails.StreamRequest{ ChunkContent: chunk, EntityID: e.agentID, @@ -371,6 +507,7 @@ func (e *LibraryGuardrailEngine) CheckStream(ctx context.Context, chunk string) }) if err != nil { e.logger.Warn("guardrail stream gate error", map[string]any{"error": err.Error()}) + result = nil return chunk, nil } @@ -379,16 +516,22 @@ func (e *LibraryGuardrailEngine) CheckStream(ctx context.Context, chunk string) if result.MaskedContent != "" { e.logger.Warn("guardrail stream redaction", nil) e.emitGuardrailEvent(ctx, "", result.MaskedContent, guardrailResultMasked, result) + evidenceContent = result.MaskedContent + decisionString = guardrailResultMasked return result.MaskedContent, nil } case guardrails.DecisionBlock: desc := violationSummary(result) if e.enforce { e.emitGuardrailEvent(ctx, "", chunk, guardrailResultBlocked, result) + evidenceContent = chunk + decisionString = guardrailResultBlocked return "", fmt.Errorf("stream blocked: %s", desc) } e.logger.Warn("guardrail stream violation (warn mode)", map[string]any{"detail": desc}) e.emitGuardrailEvent(ctx, "", chunk, guardrailResultWarned, result) + evidenceContent = chunk + decisionString = guardrailResultWarned } return chunk, nil diff --git a/forge-cli/runtime/guardrails_engine_test.go b/forge-cli/runtime/guardrails_engine_test.go index ebae437..3f4c9b7 100644 --- a/forge-cli/runtime/guardrails_engine_test.go +++ b/forge-cli/runtime/guardrails_engine_test.go @@ -12,6 +12,7 @@ import ( "github.com/initializ/guardrails/models" "github.com/initializ/forge/forge-core/a2a" + "github.com/initializ/forge/forge-core/observability" coreruntime "github.com/initializ/forge/forge-core/runtime" ) @@ -108,7 +109,7 @@ func TestFileGuardrailEngine_CheckToolOutput(t *testing.T) { // TestBuildGuardrailChecker_FileMode tests the builder with file-based config. func TestBuildGuardrailChecker_FileMode(t *testing.T) { logger := &grTestLogger{} - checker := BuildGuardrailChecker(nil, "/nonexistent", false, logger, nil, GuardrailAuditConfig{}) + checker := BuildGuardrailChecker(nil, "/nonexistent", false, logger, nil, GuardrailAuditConfig{}, observability.TracingConfig{}) if checker == nil { t.Fatal("BuildGuardrailChecker should return a non-nil checker") } diff --git a/forge-cli/runtime/guardrails_loader.go b/forge-cli/runtime/guardrails_loader.go index 28331e5..6d790c5 100644 --- a/forge-cli/runtime/guardrails_loader.go +++ b/forge-cli/runtime/guardrails_loader.go @@ -9,6 +9,7 @@ import ( "github.com/initializ/guardrails/models" "github.com/initializ/forge/forge-core/agentspec" + "github.com/initializ/forge/forge-core/observability" coreruntime "github.com/initializ/forge/forge-core/runtime" "github.com/initializ/forge/forge-core/types" ) @@ -43,13 +44,26 @@ func DefaultPolicyScaffold() *agentspec.PolicyScaffold { // // auditLogger and auditCfg are wired into the resulting engine so every // mask/block/warn decision emits a guardrail_check event through the -// same sink stack the A2A handlers use. When auditLogger is nil the -// engine is silent on the audit pipeline (used by tests). -func BuildGuardrailChecker(cfg *types.ForgeConfig, workDir string, enforce bool, logger coreruntime.Logger, auditLogger *coreruntime.AuditLogger, auditCfg GuardrailAuditConfig) coreruntime.GuardrailChecker { +// same sink stack the A2A handlers use. tracingCfg controls the +// guardrail. span instrumentation added in #161 — when +// CaptureContent is on, evidence is stamped on the span via the same +// redact-then-truncate pipeline the LLM-call content capture uses. +// When auditLogger is nil the engine is silent on the audit pipeline +// (used by tests). +func BuildGuardrailChecker( + cfg *types.ForgeConfig, + workDir string, + enforce bool, + logger coreruntime.Logger, + auditLogger *coreruntime.AuditLogger, + auditCfg GuardrailAuditConfig, + tracingCfg observability.TracingConfig, +) coreruntime.GuardrailChecker { attach := func(e *LibraryGuardrailEngine) coreruntime.GuardrailChecker { if auditLogger != nil { e.WithAuditLogger(auditLogger, auditCfg) } + e.WithTracing(tracingCfg) return e } diff --git a/forge-cli/runtime/guardrails_tracing.go b/forge-cli/runtime/guardrails_tracing.go new file mode 100644 index 0000000..b9b7353 --- /dev/null +++ b/forge-cli/runtime/guardrails_tracing.go @@ -0,0 +1,109 @@ +package runtime + +import ( + "context" + + "github.com/initializ/guardrails" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/codes" + "go.opentelemetry.io/otel/trace" + + "github.com/initializ/forge/forge-core/observability" + coreruntime "github.com/initializ/forge/forge-core/runtime" +) + +// Guardrail span instrumentation (issue #161). +// +// Symmetric to the guardrail_check audit-event emission shipped in +// #156 / #160 — every Check* method on LibraryGuardrailEngine opens +// a guardrail. span, stamps the same gate / decision / +// violation metadata the audit event carries, and (when +// CaptureContent is enabled) stamps the offending content as +// forge.guardrail.evidence via the same redact-then-truncate +// pipeline issue #130 established for OTel content capture. +// +// Span naming follows the audit-event gate vocabulary: +// +// guardrail.input (InputGate, CheckInbound) +// guardrail.context (ContextGate, CheckContext) +// guardrail.tool_call (ToolCallGate, CheckToolCall) +// guardrail.output (OutputGate, CheckOutbound + CheckToolOutput) +// guardrail.stream (StreamGate, CheckStream — not auto-wired) +// +// The span parent is whatever's active when the engine method is +// called (the A2A handler span for CheckInbound, agent.execute for +// the hook-driven gates). The noop tracer returned by Tracer() when +// tracing is disabled makes the SetAttributes / SetStatus / End +// calls near-zero cost — the engine calls them unconditionally. + +// gateSpanName maps the engine's "what gate am I about to run" into +// the matching guardrail. span name. Used by startGuardrailSpan +// before the library call (we know which gate we're about to invoke; +// after the call we know it again via res.Gate, which acts as the +// single-source attribute value). +func gateSpanName(gate string) string { + return "guardrail." + gate +} + +// startGuardrailSpan opens a child span for one gate invocation. The +// span name is `guardrail.` where matches the library's +// Result.Gate value. Returns the child ctx + span; callers must call +// finishGuardrailSpan with the library Result (or nil if the call +// failed before producing one). +func startGuardrailSpan(ctx context.Context, gate, tool string) (context.Context, trace.Span) { + ctx, span := coreruntime.Tracer().Start(ctx, gateSpanName(gate)) + if tool != "" { + span.SetAttributes(attribute.String(observability.AttrForgeToolName, tool)) + } + return ctx, span +} + +// finishGuardrailSpan stamps the gate-result attributes on the span +// and closes it. Behavior matrix: +// +// - res nil → no attribute stamping (the gate call errored +// before the library returned a Result); span ends as-is. +// - DecisionBlock → OTel Error status with the violation summary +// as the status description. Surfaces blocked invocations as red +// bars in the trace UI without custom attribute queries. +// - DecisionMask/Warn → OK status (default). +// - tracingCfg.CaptureContent → stamps +// forge.guardrail.evidence with the post-mask content +// (mask) or original content (warn/block), redact-then-truncated +// per cfg. +// +// `content` is the value the caller wants stamped as evidence — the +// CheckInbound / CheckOutbound / CheckToolOutput sites pass the +// post-mask content for mask decisions and the original for +// warn/block decisions, matching the audit-event rule from PR #156. +func finishGuardrailSpan( + span trace.Span, + res *guardrails.Result, + decisionString, content string, + tracingCfg observability.TracingConfig, +) { + defer span.End() + if res == nil { + return + } + span.SetAttributes( + attribute.String(observability.AttrForgeGuardrailGate, string(res.Gate)), + attribute.String(observability.AttrForgeGuardrailDecision, decisionString), + attribute.Int(observability.AttrForgeGuardrailViolationCount, len(res.Violations)), + ) + if len(res.Violations) > 0 { + span.SetAttributes(attribute.String(observability.AttrForgeGuardrailType, res.Violations[0].Type)) + if cat := res.Violations[0].Category; cat != "" { + span.SetAttributes(attribute.String(observability.AttrForgeGuardrailCategory, cat)) + } + } + if decisionString == guardrailResultBlocked { + span.SetStatus(codes.Error, violationSummary(res)) + } + if tracingCfg.CaptureContent && content != "" { + ev := coreruntime.PrepareSpanContent(content, tracingCfg.Redact, coreruntime.DefaultSpanContentCapBytes) + if ev != "" { + span.SetAttributes(attribute.String(observability.AttrForgeGuardrailEvidence, ev)) + } + } +} diff --git a/forge-cli/runtime/guardrails_tracing_test.go b/forge-cli/runtime/guardrails_tracing_test.go new file mode 100644 index 0000000..31a5bbf --- /dev/null +++ b/forge-cli/runtime/guardrails_tracing_test.go @@ -0,0 +1,222 @@ +package runtime + +import ( + "context" + "strings" + "testing" + + sdktrace "go.opentelemetry.io/otel/sdk/trace" + + "github.com/initializ/forge/forge-core/a2a" + "github.com/initializ/forge/forge-core/observability" + coreruntime "github.com/initializ/forge/forge-core/runtime" +) + +// findGuardrailAttr returns the string value of a span attribute by +// key with ok=true only when the key was set. Mirrors the helper +// loop_spans_content_test.go uses for the LLM-span content tests. +func findGuardrailAttr(span sdktrace.ReadOnlySpan, key string) (string, bool) { + for _, kv := range span.Attributes() { + if string(kv.Key) == key { + return kv.Value.AsString(), true + } + } + return "", false +} + +// findGuardrailIntAttr returns the int64 value of a span attribute by +// key (used for forge.guardrail.violation_count). +func findGuardrailIntAttr(span sdktrace.ReadOnlySpan, key string) (int64, bool) { + for _, kv := range span.Attributes() { + if string(kv.Key) == key { + return kv.Value.AsInt64(), true + } + } + return 0, false +} + +// setupGuardrailTracingTest installs an in-memory tracer provider and +// returns an engine wired to it. The caller passes the TracingConfig +// to control CaptureContent / Redact independently. +func setupGuardrailTracingTest(t *testing.T, tracingCfg observability.TracingConfig) (*LibraryGuardrailEngine, *observability.SpanRecorder) { + t.Helper() + tp, rec := observability.NewTestTracerProvider() + coreruntime.SetTracerProvider(tp) + t.Cleanup(func() { + coreruntime.ResetTracerProviderForTest() + _ = tp.Shutdown(context.Background()) + }) + + sg := DefaultStructuredGuardrails() + engine, err := NewFileGuardrailEngine(sg, false, &grTestLogger{}) + if err != nil { + t.Fatalf("NewFileGuardrailEngine: %v", err) + } + engine.WithTracing(tracingCfg) + return engine, rec +} + +// TestCheckInbound_OpensInputSpanWithGateAttributes verifies the +// guardrail.input span lands on the recorder with the gate, decision, +// and violation_count attributes stamped from the library Result. +// CaptureContent is OFF for this test — evidence MUST be absent. +func TestCheckInbound_OpensInputSpanWithGateAttributes(t *testing.T) { + engine, rec := setupGuardrailTracingTest(t, observability.TracingConfig{}) + + msg := &a2a.Message{ + Role: "user", + Parts: []a2a.Part{{Kind: a2a.PartKindText, Text: "my email is foo@example.com"}}, + } + if err := engine.CheckInbound(context.Background(), msg); err != nil { + t.Fatalf("CheckInbound: %v", err) + } + + span, ok := rec.FindSpan("guardrail.input") + if !ok { + t.Fatal("expected guardrail.input span") + } + + if v, ok := findGuardrailAttr(span, observability.AttrForgeGuardrailGate); !ok || v != "input" { + t.Errorf("gate = %q (ok=%v), want input", v, ok) + } + if v, ok := findGuardrailAttr(span, observability.AttrForgeGuardrailDecision); !ok || v != "masked" { + t.Errorf("decision = %q (ok=%v), want masked", v, ok) + } + if vc, ok := findGuardrailIntAttr(span, observability.AttrForgeGuardrailViolationCount); !ok || vc < 1 { + t.Errorf("violation_count = %d (ok=%v), want >= 1", vc, ok) + } + if _, ok := findGuardrailAttr(span, observability.AttrForgeGuardrailEvidence); ok { + t.Errorf("evidence MUST be absent when CaptureContent=false") + } +} + +// TestCheckInbound_CaptureContent_StampsRedactedEvidence verifies the +// opt-in capture path: with CaptureContent=true the +// forge.guardrail.evidence attribute lands on the span carrying the +// post-mask content (same rule the audit-event evidence uses), and +// the raw PII MUST NOT appear in the attribute value. +func TestCheckInbound_CaptureContent_StampsRedactedEvidence(t *testing.T) { + engine, rec := setupGuardrailTracingTest(t, observability.TracingConfig{CaptureContent: true, Redact: true}) + + msg := &a2a.Message{ + Role: "user", + Parts: []a2a.Part{{Kind: a2a.PartKindText, Text: "my email is foo@example.com"}}, + } + if err := engine.CheckInbound(context.Background(), msg); err != nil { + t.Fatalf("CheckInbound: %v", err) + } + + span, ok := rec.FindSpan("guardrail.input") + if !ok { + t.Fatal("expected guardrail.input span") + } + + ev, ok := findGuardrailAttr(span, observability.AttrForgeGuardrailEvidence) + if !ok { + t.Fatal("expected forge.guardrail.evidence attribute with CaptureContent=true") + } + if strings.Contains(ev, "foo@example.com") { + t.Errorf("raw email MUST NOT appear in evidence (post-mask rule); got: %q", ev) + } +} + +// TestCheckToolCall_OpensToolCallSpanWithToolAttribute verifies the +// tool_call gate stamps forge.tool.name in addition to the standard +// guardrail attributes. SIEM consumers use the tool field to +// distinguish tool_call fires across many tools. +func TestCheckToolCall_OpensToolCallSpanWithToolAttribute(t *testing.T) { + engine, rec := setupGuardrailTracingTest(t, observability.TracingConfig{}) + + // Args that may or may not mask depending on rule config; we only + // care here that the span lands with the tool attribute set. + _, _ = engine.CheckToolCall(context.Background(), "send_email", + `{"to":"alice@example.com","body":"hi"}`) + + span, ok := rec.FindSpan("guardrail.tool_call") + if !ok { + t.Fatal("expected guardrail.tool_call span") + } + if v, ok := findGuardrailAttr(span, observability.AttrForgeToolName); !ok || v != "send_email" { + t.Errorf("forge.tool.name = %q (ok=%v), want send_email", v, ok) + } +} + +// TestCheckOutbound_OpensOutputSpan_NoToolAttribute verifies the +// "OutputGate on the model's reply to the user" path: span name is +// guardrail.output and forge.tool.name is absent (distinguishes it +// from CheckToolOutput which sets the tool attribute). +func TestCheckOutbound_OpensOutputSpan_NoToolAttribute(t *testing.T) { + engine, rec := setupGuardrailTracingTest(t, observability.TracingConfig{}) + + msg := &a2a.Message{ + Role: "agent", + Parts: []a2a.Part{{Kind: a2a.PartKindText, Text: "Here is your answer."}}, + } + if err := engine.CheckOutbound(context.Background(), msg); err != nil { + t.Fatalf("CheckOutbound: %v", err) + } + + span, ok := rec.FindSpan("guardrail.output") + if !ok { + t.Fatal("expected guardrail.output span") + } + if v, ok := findGuardrailAttr(span, observability.AttrForgeToolName); ok { + t.Errorf("forge.tool.name should NOT be set on CheckOutbound (got %q)", v) + } +} + +// TestCheckContext_OpensContextSpan covers the ContextGate path. +func TestCheckContext_OpensContextSpan(t *testing.T) { + engine, rec := setupGuardrailTracingTest(t, observability.TracingConfig{}) + + if _, err := engine.CheckContext(context.Background(), "some retrieved context"); err != nil { + t.Fatalf("CheckContext: %v", err) + } + + if _, ok := rec.FindSpan("guardrail.context"); !ok { + t.Fatal("expected guardrail.context span") + } +} + +// TestCheckStream_OpensStreamSpan covers the StreamGate path even +// though it's not auto-wired in the loop yet. +func TestCheckStream_OpensStreamSpan(t *testing.T) { + engine, rec := setupGuardrailTracingTest(t, observability.TracingConfig{}) + + if _, err := engine.CheckStream(context.Background(), "delta"); err != nil { + t.Fatalf("CheckStream: %v", err) + } + + if _, ok := rec.FindSpan("guardrail.stream"); !ok { + t.Fatal("expected guardrail.stream span") + } +} + +// TestCheckInbound_NoTracing_NoSpansRecorded confirms the noop-tracer +// path: no SetTracerProvider, so Tracer() returns the noop tracer and +// no spans land on the recorder. +func TestCheckInbound_NoTracing_NoSpansRecorded(t *testing.T) { + tp, rec := observability.NewTestTracerProvider() + // Do NOT install the test provider — keep the noop default. + t.Cleanup(func() { _ = tp.Shutdown(context.Background()) }) + + sg := DefaultStructuredGuardrails() + engine, err := NewFileGuardrailEngine(sg, false, &grTestLogger{}) + if err != nil { + t.Fatalf("NewFileGuardrailEngine: %v", err) + } + // engine.WithTracing(observability.TracingConfig{}) is the default; + // no tracer provider installed → noop tracer. + + msg := &a2a.Message{ + Role: "user", + Parts: []a2a.Part{{Kind: a2a.PartKindText, Text: "hello"}}, + } + if err := engine.CheckInbound(context.Background(), msg); err != nil { + t.Fatalf("CheckInbound: %v", err) + } + + if _, ok := rec.FindSpan("guardrail.input"); ok { + t.Errorf("expected no spans recorded with noop tracer; got guardrail.input") + } +} diff --git a/forge-cli/runtime/runner.go b/forge-cli/runtime/runner.go index 4c6f9f0..37281a1 100644 --- a/forge-cli/runtime/runner.go +++ b/forge-cli/runtime/runner.go @@ -330,12 +330,31 @@ func (r *Runner) Run(ctx context.Context) error { } auditLogger.WithEntity("agent", agentID) + // Resolve TracingConfig early so we can thread it into the + // guardrail engine before the tracer provider itself is installed + // further down. ResolveTracingConfig is a pure config-resolution + // function — no I/O, no provider construction — so this is safe + // to call ahead of NewTracerProvider. The provider install at + // line ~561 still owns lifecycle; the engine just needs the + // CaptureContent + Redact flags. See issue #161. + tracingCfgEarly := ResolveTracingConfig( + r.cfg.Config.Observability.Tracing, + r.cfg.TracingFlags, + r.cfg.Config.AgentID, + r.cfg.Config.Version, + r.cfg.RuntimeVersion, + ) + // 4a. Build guardrail checker (DB mode → file mode → defaults) and // wire the audit logger so every mask/block/warn decision lands on // the configured audit sinks as a guardrail_check event. Capture- // evidence posture comes from env (FORGE_GUARDRAIL_*), default - // metadata-only. See issue #155. - guardrails := BuildGuardrailChecker(r.cfg.Config, r.cfg.WorkDir, r.cfg.EnforceGuardrails, r.logger, auditLogger, GuardrailAuditConfigFromEnv()) + // metadata-only. tracingCfgEarly carries the + // CaptureContent/Redact knobs the guardrail. spans use for + // evidence stamping (#161); the spans themselves are opened + // unconditionally — when tracing is disabled, the noop tracer + // short-circuits. + guardrails := BuildGuardrailChecker(r.cfg.Config, r.cfg.WorkDir, r.cfg.EnforceGuardrails, r.logger, auditLogger, GuardrailAuditConfigFromEnv(), tracingCfgEarly) // Periodic audit_export_status — one event every 60s with per-sink // health counters. Operators tail the audit stream to answer // "is my sidecar healthy?". The stop func blocks until the @@ -547,13 +566,7 @@ func (r *Runner) Run(ctx context.Context) error { // tracing.go) and continue. Tracing is off-by-default per the // initiative ruling — a misconfigured exporter must never crash // the agent. - tracingCfg := ResolveTracingConfig( - r.cfg.Config.Observability.Tracing, - r.cfg.TracingFlags, - r.cfg.Config.AgentID, - r.cfg.Config.Version, - r.cfg.RuntimeVersion, - ) + tracingCfg := tracingCfgEarly var tracingTransport http.RoundTripper if egressClient != nil { tracingTransport = egressClient.Transport diff --git a/forge-core/observability/attrs.go b/forge-core/observability/attrs.go index 0649466..1a2e023 100644 --- a/forge-core/observability/attrs.go +++ b/forge-core/observability/attrs.go @@ -126,4 +126,56 @@ const ( // AttrForgeToolResult is the raw output the tool returned. Set on // tool. spans. AttrForgeToolResult = "forge.tool.result" + + // ─── Guardrail span attributes (issue #161) ────────────────────── + // + // Stamped on guardrail. spans the LibraryGuardrailEngine + // opens around every Check* call (CheckInbound, CheckContext, + // CheckToolCall, CheckToolOutput, CheckOutbound, CheckStream). + // Symmetric to the guardrail_check audit-event fields shipped in + // #156 / #160 — operators looking at a trace see the same gate / + // decision / violation metadata they see in the audit stream and + // can pivot between the two by correlation_id / trace_id without + // joining on raw guardrail content. + // + // AttrForgeGuardrailEvidence follows the issue #130 + // CaptureContent + Redact + MaxBytes posture: default off, opt-in + // per deployment, redact-then-truncate when on. Same env knobs as + // the existing OTel content-capture pipeline. + + // AttrForgeGuardrailGate is the library gate that fired — one of + // "input", "context", "tool_call", "output", "stream". Sourced + // directly from `Result.Gate` (single source of truth, matches + // fields.gate on the guardrail_check audit event). + AttrForgeGuardrailGate = "forge.guardrail.gate" + + // AttrForgeGuardrailDecision is the library decision — one of + // "allow", "mask", "block", "warn". Sourced from `Result.Decision`. + AttrForgeGuardrailDecision = "forge.guardrail.decision" + + // AttrForgeGuardrailType is the first violation's Type field — e.g. + // "pii", "moderation", "security". Omitted when violations list is + // empty (the "allow" path). + AttrForgeGuardrailType = "forge.guardrail.type" + + // AttrForgeGuardrailCategory is the first violation's Category — + // e.g. "ssn", "email", "hate_speech". Omitted when the violation + // has no category. + AttrForgeGuardrailCategory = "forge.guardrail.category" + + // AttrForgeGuardrailViolationCount is the length of + // `Result.Violations`. Useful for SIEM "show me high-violation + // invocations" queries without joining against the full evidence + // stream. + AttrForgeGuardrailViolationCount = "forge.guardrail.violation_count" + + // AttrForgeGuardrailEvidence is the triggering content. Set only + // when TracingConfig.CaptureContent is true. Passes through + // PrepareSpanContent (redact-then-truncate) just like the other + // #130 content attributes. For mask decisions evidence carries the + // post-mask content (the same payload the LLM actually saw); for + // block / warn decisions it carries the original triggering text + // (the library never produces a masked variant in those paths). + // Matches the audit-event evidence rule from PR #156. + AttrForgeGuardrailEvidence = "forge.guardrail.evidence" )