Implement perry/container and perry/workloads subsystems#84
Open
yumin-chen wants to merge 3 commits into
Open
Conversation
feat: implement production-ready container and workload orchestration
Finalize the OCI stack by implementing the `perry/container` and
`perry/container-compose` (workloads) subsystems. This moves the
implementation from initial stubs to a hardened, spec-compliant architecture.
Core Subsystems:
- Orchestration: Implemented `WorkloadGraphEngine` and `ComposeEngine`
using Kahn's algorithm for deterministic dependency resolution and
topological startup/shutdown/rollback.
- Backend Logic: Multi-layered auto-detection for 7+ runtimes (Apple, Podman,
Docker, Lima, etc.) with liveness probes and strict priority ordering.
- Security & Policy:
* Implemented `PolicySpec` enforcement (Isolated, Hardened, Untrusted).
* Added image verification via Sigstore/cosign (opt-in via environment).
* Hardened ephemeral runners with `cap_drop: ALL`, seccomp, and read-only
root support.
- FFI Bridge: Expanded `perry-stdlib` with async-safe, promise-based
handlers optimized for raw C-ABI passing of primitives.
Technical Details:
- Restructured `perry-container-compose` into a flat module layout.
- Standardized container naming to `{image_hash_8}-{random_hex8}` with
label-based orphan cleanup.
- Refactored `CliBackend` to be generic over `CliProtocol` for zero vtable
overhead.
- Modernized internal registries with `DashMap` for concurrent access.
- Integrated with Perry compiler (HIR registration and codegen dispatch).
Refinements & Fixes:
- Fixed SQLite linker conflicts by gating runtime stubs.
- Restored `Buffer` synonym and `process.argv` specialization in `lower.rs`.
- Implemented robust IP and label extraction for the `DockerProtocol`.
- Expanded `MockBackend` for high-fidelity orchestration testing.
Validation:
- Added 12 new tests covering orchestration states and policy enforcement.
- Verified 79/0 pass in `perry-container-compose`.
- Verified 33/0 pass in `perry-stdlib` container features and smoke tests.
…eployment example (v0.5.370)
Driven by running example-code/forgejo-deployment against live Docker.
Surfaced and fixed six interlocking codegen + FFI + orchestration bugs
that together blocked any non-trivial compose stack from running.
1. composeUp({...}) failed at JSON parse with "expected value at line 1
column 1" — the codegen NativeArgKind::StrPtr arm called
js_get_string_pointer_unified(arg) which for object operands returned
the raw object pointer; the FFI then read it as StringHeader.
Fix: new js_value_to_str_ptr_for_ffi runtime helper that returns the
heap string pointer for actual strings/SSO and otherwise routes
through js_json_stringify to auto-encode object/array/number/bool
args. Codegen StrPtr arm calls the new helper instead. composeUp({
services: {...} }) now Just Works.
2. getBackend() returned "unknown" before any async FFI — the BACKEND
OnceLock was empty. Updated js_container_getBackend to perform a
synchronous in-place probe (block_in_place inside a tokio worker,
fresh current_thread runtime otherwise).
3. composeUp Promise resolved with f64=5e-324 (subnormal). The
async-bridge stored bare u64 handles in the result_bits slot which
then decoded as f64 bits. Two fixes:
- handle_to_promise_bits(id) NaN-boxes with
POINTER_TAG | (id & POINTER_MASK) so subsequent unbox_to_i64
recovers the id verbatim and template-string interpolation no
longer prints "0".
- Ok(0u64) void resolutions become PROMISE_VOID_BITS (TAG_UNDEFINED)
so they read as undefined.
Swept across 23 call sites in mod.rs.
4. down(stack, opts) failed with "Invalid compose handle". The codegen
dispatch args: &[NA_F64, NA_F64] for js_compose_down lowered both
args to LLVM double, but the Rust signatures took
(handle_id: i64, volumes: i32) — calling-convention mismatch.
Fix: changed every compose handle-arg FFI signature to (handle: f64,
...) and added handle_id_from_f64(boxed) that masks POINTER_TAG off,
mirroring the codegen contract. Touches js_container_compose_down /
_ps / _logs / _exec / _config / _start / _stop / _restart and their
js_compose_* aliases.
5. exec(stack, 'postgres', cmd) failed with "No such container".
service::service_container_name regenerates a fresh
{md5_8}-{random_hex8} name on every call, so post-up operations
could never find the container.
Fix: added a service_container_names: Mutex<HashMap<String, String>>
cache to ComposeEngine populated by up() during the start loop;
down/ps/exec/logs/start/stop now read through resolve_container_name
instead of regenerating.
6. ${FORGEJO_DB_USER:-forgejo} env interpolation didn't apply to TS-side
specs — postgres bombed with "FATAL: invalid character in extension
owner: must not contain any of '$...'" because the literal
placeholder string flowed straight through.
Fix: wired perry_container_compose::yaml::interpolate into
parse_compose_spec so ${VAR} and ${VAR:-default} get expanded
against std::env::vars() BEFORE serde_json::from_str, matching
SPEC §7.8 / §7.9.
Verified end-to-end: a fresh forgejo-deployment run on Docker creates
the network + 3 named volumes + postgres + gitea containers, polls
pg_isready until accepting connections (succeeds within ~7s on a clean
volume), prints the "Forgejo Stack is Ready!" banner, and the SIGINT
handler tears down + removes volumes.
The forgejo image stayed on gitea/gitea:1.23 (the upstream Forgejo
forked from) since codeberg.org's registry intermittently returns
"unauthorized: reqPackageAccess" for public pulls; env-var keys
updated to the matching GITEA__* form. Example refreshed to use the
canonical standalone-function API per SPEC §C4 (up / down / exec
take handle as first arg — method-chain stack.down() was sugar for a
TS-library wrapper that doesn't exist).
New tsconfig.json with paths: { "perry/*": ["../../types/perry/*"] }
so IDE typechecking finds the workspace types.
Workspace re-registration: had to re-add perry-container-compose to
[workspace] members + default-members + [workspace.dependencies]
after an intentional Cargo.toml edit removed them — perry-stdlib's
optional dep declaration requires the entry; build can't succeed
without it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Implement `ContainerContext` for process-global state and handle management using `DashMap`. - Standardize FFI bridge in `perry-stdlib` for container, compose, and workload operations. - Update backend detection in `perry-container-compose` to support `PERRY_CONTAINER_MODE` and `IsolationLevel`. - Integrate `perry/container`, `perry/compose`, and `perry/workloads` into HIR lowering and Codegen dispatch. - Ensure handles are registered as native instances in HIR to support method-chaining in TypeScript. - Fix FFI signature mismatches in tests and ensure thread-safe async backend initialization. - Add `js_container_inspectImage` for image metadata retrieval. - Verified with property, functional, and integration tests. Co-authored-by: yumin-chen <10954839+yumin-chen@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
yumin-chen
force-pushed
the
feat/container-compose
branch
4 times, most recently
from
a7e9d31 to
dd181eb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implemented the full
perry/containerandperry/workloadssubsystems, including theperry-container-composebackend,perry-stdlibFFI bridge, and compiler integration inperry-hirandperry-codegen.Key highlights:
ContainerContextusingDashMapprovides thread-safe, process-global storage for container, compose, and workload handles.PERRY_CONTAINER_MODE(local-firstvsserver-first) and includesIsolationLevelmetadata.WorkloadGraphEnginecorrectly resolves cross-nodeWorkloadRefvalues and enforces security policies based on tier.stack.down()).f64for handles and numeric flags to align with the compiler's NaN-boxing semantics.tokio::sync::Mutexand resolved several latent FFI signature mismatches in tests.PR created automatically by Jules for task 6019915072117160150 started by @yumin-chen