Skip to content

docs(tutorial): fix step 15 candidate-vs-baseline coherence#221

Open
placerda wants to merge 1 commit into
developfrom
docs/tutorial-step15-coherence
Open

docs(tutorial): fix step 15 candidate-vs-baseline coherence#221
placerda wants to merge 1 commit into
developfrom
docs/tutorial-step15-coherence

Conversation

@placerda
Copy link
Copy Markdown
Contributor

Why

Step 15 of the prompt-agent quickstart instructed the reader to paste
back a byte-identical copy of the baseline prompt as "the fix". That
made prompt_deploy stage report action: "reused" and keep
candidate_agent: travel-agent:2. When the reader then looked at the
Release Evidence card and saw Target: travel-agent:2, the natural
reaction was "I just shipped a fix — why is nothing new here?".

The card was correct (the Release Evidence Target mirrors what
agentops.yaml pins — the declarative baseline), but the tutorial
never made the candidate-vs-pin distinction explicit, so the story
read as "the fix didn't take".

What this PR changes

Docs-only. No code, no schema, no behavior.

  • Mental model section. Adds a 4th bullet that names the split:
    PRs create and evaluate a new candidate Foundry version; the
    agentops.yaml agent: pin (and therefore the Release Evidence
    Target field) only moves when you explicitly promote a baseline.
  • Journey table row "Fix and redeploy". Updates the description
    to match the new step 15: push an improved prompt that restores the
    guardrails and adds a small refinement, so a brand-new candidate
    version is created (instead of a reused no-op).
  • Step 15 itself.
    • Replaces the byte-identical restore prompt with an improved
      prompt: keeps all four original structural rules, adds one narrow
      rule ("include currency code when known"). The change is small
      enough that similarity scores against the existing dataset should
      stay above thresholds, but real enough that
      prompt_deploy stage reports action: "created" and a new
      candidate_agent: travel-agent:N.
    • Rewrites the explanation block to walk the reader through the
      new foundry-agent.json, the candidate-vs-pin split, and where
      durable cross-environment identity lives (prompt_sha256,
      git_sha).
    • Adds a recording-note callout: if the reader already pushed a
      byte-identical restore in an earlier take, push one more commit
      with the improved prompt and the stage step will report
      action: "created" instead of reused.
  • Success criteria checklist. Updates the "you restored the
    prompt" bullet to mention the new candidate version and the
    unchanged baseline pin, so the checklist matches step 15.

Why a narrow additive change for the new prompt

A purely-restorative "fix" can't show the
action: "created" + new-candidate-version part of the story — by
definition, byte-identical content reuses the existing version. We
need the fix to be a real content change. We also can't change the
prompt too much, or we risk dropping similarity scores below the
PR-gate thresholds and turning step 15 back into a failing run.

The chosen new rule (require currency code when monetary amounts are
mentioned) is narrow and additive, keeps the four structural
guardrails intact, and stays in scope for the dataset used by the
tutorial.

Release impact

None — this is a docs change. Will land on develop and ship with the
next release cut.

Test plan

  • Inspected the rendered Markdown locally; numbered headings and
    table formatting still align.
  • Grepped for stale "Restore"/"restored prompt" wording — only
    the success-criteria checklist line was affected and is
    updated.
  • No code paths changed; no test suite run required.

Related context

Step 15 told readers to paste a byte-identical baseline as 	he fix, which made prompt_deploy stage report action: reused. The Release Evidence card then showed Target: travel-agent:2 unchanged, which felt contradictory because the tutorial never named the candidate-vs-baseline split.

Updates: (1) mental-model gains a 4th bullet naming the split; (2) journey-table Fix and redeploy row updated; (3) step 15 prompt is now a small additive improvement (require currency codes) so stage reports action: created with a new candidate_agent version; (4) explanation block walks through foundry-agent.json, the Target vs candidate_agent fields, and where durable identity lives; (5) recording-note tells readers who already pushed a byte-identical restore to push one more commit; (6) success-criteria checklist matches the new step 15.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant