Skip to content

Add TFCTL_CONFIG_DIR override and fix delete-gating eval#83

Merged
shwetamurali merged 6 commits into
mainfrom
shweta/update_skill_for_evals
Jul 16, 2026
Merged

Add TFCTL_CONFIG_DIR override and fix delete-gating eval#83
shwetamurali merged 6 commits into
mainfrom
shweta/update_skill_for_evals

Conversation

@shwetamurali

@shwetamurali shwetamurali commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Description

  1. Add a TFCTL_CONFIG_DIR environment variable that overrides the tfctl config directory. Both profiles and exec sessions now resolve against it using a new shared profile.ConfigDir() resolver.
  2. Point execsession.DefaultStore() at <config-dir>/exec through that resolver, so config and exec state share one directory.
  3. Rewrite eval task 32-irreversible-still-blocked as a client-side behavioral test: it exercises by-ID delete gating with a fake token and asserts the --allow-delete=projects relay, so it no longer depends on a live backend.
  4. Correct skills/tfctl/SKILL.md so the agent runs the delete and surfaces the resulting grant requirement, rather than trying to check the grant up front.
  5. Add go install, TFCTL_TOKEN, and TFCTL_CONFIG_DIR to the eval workflow and README so runs are fully isolated and repeatable.

Example Output

PR Checklist

  • Run npx changie new or install changie to prepare a new changelog entry for the next set of release notes.
  • Ensure any command changes are sensitive to these global flags:
    • --json — Force machine readable output to stdout. Does not apply to stderr.
    • --markdown — Force markdown output to stdout. Does not apply to stderr.
    • --dry-run — Don't make any actual writes or other mutations. Describe what would have changed to stderr.
    • --quiet — Don't render output to stdout.
  • Get the logging interface from the context and add debug logging for interesting conditions and nonfatal situations.
  • Run make gen/screenshot if the root command output changes.
  • Add the Autocomplete field to positional arguments and flags to assist shell autocomplete.

PCI review checklist

  • I have documented a clear reason for, and description of, the change I am making.

  • If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.

  • If applicable, I've documented the impact of any changes to security controls.

    Examples of changes to security controls include using new access control methods, adding or removing logging pipelines, etc.

@shwetamurali
shwetamurali marked this pull request as ready for review July 15, 2026 19:19
@shwetamurali shwetamurali changed the title Update SKILL.md Add TFCTL_CONFIG_DIR override and fix delete-gating eval Jul 15, 2026
brandonc
brandonc previously approved these changes Jul 16, 2026

@brandonc brandonc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it

Comment thread evals/tfctl-evals/README.md Outdated
Comment thread skills/tfctl/SKILL.md

- **If the session authorizes that resource's class, it succeeds** — that's the human's intent, not a violation; proceed and report the result.
- **If it doesn't, `tfctl` refuses and prints the exact command to hand back** (including the `harness exec --allow-delete=<class>` a human can use to authorize you). Relay that rather than forcing it; don't run `harness exec` yourself to self-authorize.
- **If it doesn't, `tfctl` refuses with a self-documenting message and prints the exact command to hand back** (including the `harness exec --allow-delete=<class>` a human can use to authorize you). Relay that rather than forcing it; don't run `harness exec` yourself to self-authorize.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't run harness exec yourself to self-authorize.

I feel like this should be reiterated under "Hard rules"

Co-authored-by: Brandon Croft <brandon.croft@gmail.com>
@shwetamurali
shwetamurali merged commit 26c867b into main Jul 16, 2026
2 checks passed
@shwetamurali
shwetamurali deleted the shweta/update_skill_for_evals branch July 16, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants