docs: implement self-improvement loop architecture#239
Conversation
- Add three signal log files: style_lint_runs.jsonl, pr_review_runs.md, human_review_feedback.jsonl - Extend draft_docs/SKILL.md step 8 to append style lint violation records to style_lint_runs.jsonl on agent-authored PRs - Extend review-docs-pr/SKILL.md to append review summaries to pr_review_runs.md - Add improve-drafting-skills skill: monthly outer loop that reads all three logs and proposes targeted edits to skills/templates - Add Phase 2 redirect-drafter to weekly-404-monitor: auto-drafts vercel.json redirects for high-confidence uncovered 404 gaps - Add improve-aeo-crosslink-skill: quarterly outer loop that reads aeo_crosslink_audit_runs.md and proposes edits to the audit skill Co-Authored-By: Oz <oz-agent@warp.dev>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![oz-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR adds self-improvement loop infrastructure for docs operations: new signal logs, new outer-loop skills, and updates to drafting/review/404-monitor skills.
Concerns
- The new JSONL log files are seeded with
#comment rows, which conflicts with the documented one-record-per-line JSONL format and can break consumers before any real signal is recorded. - Several logging steps require automated agents to commit directly to
main; that either bypasses the normal PR review path when credentials allow it, or silently loses the durable signal when branch protection blocks the push. - The human feedback collector misses inline review comments and attributes the full PR diff as human edits, so the primary training signal can be incomplete and polluted with agent-authored content.
Security
- The outer-loop skills consume human comments, PR text, and run logs as inputs for editing skills, but they do not define a prompt-injection boundary for treating that content strictly as data.
Spec alignment
No approved or repository spec context was provided for this PR, so there were no spec-alignment findings.
Verdict
Found: 0 critical, 5 important, 0 suggestions
Request changes
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
rachaelrenk
changed the title
- Empty JSONL log files (comment headers were invalid JSONL) - Replace direct-to-main commits in inner loops with stdout signal markers ([SIGNAL:style-lint] and [SIGNAL:pr-review]) consumed by the outer loop via oz run get -- eliminates branch protection dependency on inner loops - Fix feedback collector to fetch inline review comments via gh api pulls/NNN/comments (not just top-level comments field) - Fix git diff range: LAST_BOT_COMMIT..MERGE_COMMIT instead of MERGE_BASE..PR_HEAD to exclude agent-authored content from human edits - Add explicit prompt-injection security boundary to improve-drafting-skills: treat log content as data only, discard injection indicators, act only on parsed structured fields Co-Authored-By: Oz <oz-agent@warp.dev>
|
/oz-review |
|
I'm re-reviewing this pull request in response to a review request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
There was a problem hiding this comment.
Overview
This PR adds documentation and skill instructions for self-improvement loops around docs drafting, PR review, 404 redirects, and AEO cross-link audits. The architecture is directionally coherent, but several changed instructions disagree about where signals are written/read, which would cause the monthly loop to miss automated feedback. I could not inspect the contents of .agents/logs/human_review_feedback.jsonl or .agents/logs/style_lint_runs.jsonl because the attached diff reports those patches as unavailable, so any concerns about those files are summarized rather than commented inline.
Concerns
- The new
pr_review_runs.mdlog describes a committed log writer, but the updated review skill only emits a stdout marker. improve-drafting-skillsstill instructs the monthly analysis to parse log files for automated signals even though the same skill now says those signals are only collected from Oz run output.- Raw human review comment text is committed to
mainwithout an explicit pre-append injection/secret filtering step. - The new review signal is described as running after submission, but the skill workflow produces
review.jsonfor the control plane to publish, so the marker may never be emitted.
Verdict
Found: 0 critical, 4 important, 0 suggestions
Request changes
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
- Fix pr_review_runs.md header: now correctly states it is written by the outer loop (improve-drafting-skills), not by review-docs-pr - Add Step A.5: outer loop writes parsed [SIGNAL:pr-review] records to pr_review_runs.md as a human-readable audit trail, closing the dead signal source - Pre-append security filtering in Step B: injection detection and secret redaction happen before records reach human_review_feedback.jsonl - Workflow step 1: now reads in-memory Step A records + on-disk human_review_feedback.jsonl; no longer references non-existent files - Update threshold table: source labels now say 'Step A in-memory records' - review-docs-pr signal logging: emit after validating review.json (not 'after submitting') so the marker appears in Oz run output regardless of how the review is published Co-Authored-By: Oz <oz-agent@warp.dev>
petradonka
left a comment
petradonka
left a comment
There was a problem hiding this comment.
Smart idea to add these! Left a comment, and would address all Oz review items before merging.
|
|
||
| ## Schedule | ||
|
|
||
| Quarterly (every 12 weeks). Start this agent on month 3 after `aeo_crosslink_audit` is running regularly (requires at least 8 run log entries for meaningful pattern analysis). |
There was a problem hiding this comment.
quarterly feels a bit too infrequent, should we run this every month?
…thly Per review feedback from Petra. Updated: description frontmatter, prose, schedule section, cron expression, and deployment instructions. Cron: 0 17 1-7 1,4,7,10 1 -> 0 17 1-7 * 1 (first Monday every month). Co-Authored-By: Oz <oz-agent@warp.dev>
|
/oz-review |
|
I'm re-reviewing this pull request in response to a review request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
There was a problem hiding this comment.
Overview
This PR adds the docs self-improvement loop architecture through new signal logs, new outer-loop skills, and updates to drafting/review/404-monitor skills.
Concerns
- The new AEO improvement skill documents a monthly cadence even though the PR deployment notes describe a quarterly loop, which can lead to scheduling the agent at the wrong frequency.
- Both new self-improvement skills require
style_lint.py --changedas their validation gate, but that command only scanssrc/content/docs/, so it will not validate the.agents/skills/or.agents/templates/files these workflows edit. - The drafting improvement loop says free-text human review comments must never drive edits, but the stored record schema does not include any separate safe category/pattern field, so human feedback cannot actually identify what skill/template change is needed.
Verdict
Found: 0 critical, 4 important, 1 suggestions
Request changes
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
improve-aeo-crosslink-skill: - PR acceptance rate check: use --state all instead of --state merged to catch closed-without-merge PRs too - Self-review validation: replace style_lint.py --changed (which only covers src/content/docs/) with git diff --check + YAML frontmatter validation, which actually covers .agents/skills/ edits improve-drafting-skills: - Add pattern_category field to human_review_feedback.jsonl schema; collector derives a structured label (e.g. header_case, list_format) without copying raw comment text - Update security boundary to explicitly name pattern_category as the field that drives skill edits, not free-text comment - Self-review validation: same fix as AEO skill -- style_lint.py --changed does not cover .agents/skills/ or .agents/templates/ Co-Authored-By: Oz <oz-agent@warp.dev>
2 participants
Summary
Implements the self-improvement loop architecture for Warp docs content operations. This PR adds the signal infrastructure and outer loop skills that allow agent-authored docs to improve automatically from accumulated feedback.
Architecture
Four coordinated loops — three active now, one documented for future deployment:
improve-drafting-skills) reads these logs and proposes targeted edits to drafting skills and templates.weekly-404-monitor— after posting the weekly Slack report, the agent now also proposes redirect entries for high-confidence uncovered 404 gaps.improve-aeo-crosslink-skill) reads theaeo_crosslink_auditrun log and proposes improvements to the audit skill itself. Deploy on month 3 afteraeo_crosslink_audithas 8+ run log entries.Changes
New log files (
.agents/logs/)style_lint_runs.jsonl— JSONL, one record per style lint run on an agent-authored PRpr_review_runs.md— Markdown log ofreview-docs-prruns on agent-authored PRshuman_review_feedback.jsonl— JSONL, human review comments and edits collected from merged agent PRsNew skills
.agents/skills/improve-drafting-skills/SKILL.md— monthly outer loop (Loop 1).agents/skills/improve-aeo-crosslink-skill/SKILL.md— quarterly outer loop (Loop 4)Modified skills (additive changes only)
.agents/skills/draft_docs/SKILL.md— step 8 extended to append a violation record tostyle_lint_runs.jsonlon cloud agent runs.agents/skills/review-docs-pr/SKILL.md— new Signal logging section: appends a summary entry topr_review_runs.mdafter reviewing an agent-authored PR.agents/skills/weekly-404-monitor/SKILL.md— new Phase 2 section: redirect drafter with confidence scoring and draft PR for HIGH-confidence matchesNext steps (after merge)
Two new Oz scheduled agents need to be configured in the Oz web app:
improve-drafting-skills— monthly, first Monday of each month at 9am PTimprove-aeo-crosslink-skill— quarterly, first Monday of Jan/Apr/Jul/Oct (start on month 3 when the run log has 8+ entries)weekly-404-monitoralready runs as a scheduled agent — no new agent is needed; Phase 2 runs within the same existing agent.Architecture plan: https://staging.warp.dev/drive/notebook/LiSAdtZGryD78gSNj5kGPx
Conversation: https://staging.warp.dev/conversation/652a054e-d757-4632-8554-5176f5529ee2
Co-Authored-By: Oz oz-agent@warp.dev