diff --git a/.agent/specs/overlay-filesystem-api-spec.md b/.agent/specs/overlay-filesystem-api-spec.md new file mode 100644 index 000000000..86411ccaf --- /dev/null +++ b/.agent/specs/overlay-filesystem-api-spec.md @@ -0,0 +1,657 @@ +# Overlay Filesystem API + +## Status + +Reviewed after adversarial pass. This spec assumes breaking API changes are acceptable. + +## Summary + +The default VM filesystem should behave like a Docker/container root filesystem: + +- the VM root is an overlay filesystem +- there is exactly one writable upper layer for the live VM +- there are zero or more immutable lower snapshot layers beneath it +- the built-in base filesystem snapshot from bundled `base-filesystem.json` is included by default unless explicitly disabled +- extra mounts behave like Docker volumes or bind mounts: they are separate mount boundaries, not additional layers inside the root overlay + +This spec defines the public API and user model for that behavior. + +It is intentionally separate from [overlay-metadata-proposal.md](/home/nathan/a3/.agent/specs/overlay-metadata-proposal.md), which covers how overlay semantics should be implemented in the metadata engine itself. + +## Goals + +1. Make the default root filesystem an overlay filesystem by default. +2. Match Linux OverlayFS and Docker mental models closely enough that behavior is predictable. +3. Distinguish clearly between: + - a mounted filesystem + - a layer + - storage plumbing used inside a layer +4. Keep the high-level API simple for ordinary users. +5. Keep the low-level API explicit for advanced users and filesystem driver authors. + +## Non-Goals + +1. Exposing raw metadata-store or block-store objects in the main `AgentOs.create()` API. +2. Making arbitrary mounted `VirtualFileSystem` implementations automatically usable as overlay layers. +3. Supporting multiple writable layers in one overlay view. +4. OCI import/export in phase 1. + +## Normative Model + +### Root Filesystem + +The VM root filesystem is an overlay filesystem by default. + +Conceptually: + +```text +/ + = overlay( + upper = writable live layer, + lowers = [zero or more frozen snapshots, including the bundled base snapshot by default] + ) +``` + +That is the same shape used by Linux OverlayFS and Docker `overlay2`: + +- one writable upper +- zero or more read-only lowers +- one merged mounted view + +### Extra Mounts + +Extra mounts are separate mount boundaries. + +Examples: + +- `/workspace` mounted from a host directory +- `/data` mounted from another overlay filesystem +- `/proc` and `/dev` mounted as special kernel filesystems + +These behave like Linux bind mounts or Docker volumes: + +- they hide the underlying path while mounted +- they are not absorbed into the parent overlay's layer stack +- cross-mount rename/link and similar inode-moving operations keep normal mount semantics like `EXDEV` + +### Middle Layers + +"Middle layers" in a Docker-like stack are just lower snapshot layers with precedence. + +Example: + +```text +upper = writable live diff +lower0 = most recent frozen snapshot +lower1 = older frozen snapshot +lower2 = base filesystem snapshot +``` + +They are not additional writable uppers. + +## API Principles + +### Principle 1: Root Should Be Config, Not Manual Composition + +Most users should not call `createOverlayFilesystem()` just to get a normal VM root. + +The root should be overlay-backed by default via `AgentOs.create(...)` config. + +### Principle 2: Mounts and Layers Must Be Different Concepts + +Mounts are kernel-visible mounted filesystems. + +Layers are overlay-building blocks. + +Do not collapse these into one generic interface. + +### Principle 3: Layers Share One Base Interface + +A writable upper layer and a frozen snapshot layer should be the same underlying concept with different invariants. + +Users should not need to implement "the layer twice." + +### Principle 4: Storage Plumbing Is Lower-Level Than Layer APIs + +Metadata stores and block stores are implementation details of layers. + +High-level users should work with layer handles and filesystem configs, not raw metadata/block objects. + +## Public API Proposal + +### 1. Root Filesystem Configuration + +Add an explicit root filesystem config to `AgentOs.create()`. + +Proposed shape: + +```ts +type OverlayFilesystemMode = "ephemeral" | "read-only"; + +type RootSnapshotExport = { kind: "snapshot-export"; source: unknown }; + +type RootLowerInput = + | { kind: "bundled-base-filesystem" } + | RootSnapshotExport; + +interface RootFilesystemConfig { + type?: "overlay"; // default + mode?: OverlayFilesystemMode; // default "ephemeral" + disableDefaultBaseLayer?: boolean; // default false + lowers?: RootLowerInput[]; // highest-precedence lower first +} + +interface AgentOsOptions { + rootFilesystem?: RootFilesystemConfig; + mounts?: MountConfig[]; + // existing options omitted +} +``` + +Default behavior: + +- if `rootFilesystem` is omitted, AgentOs creates: + - an ephemeral writable upper layer backed by a filesystem layer rooted in a tmp dir in the default internal temp-dir-backed layer store + - one lower layer from the bundled built base filesystem artifact imported from `base-filesystem.json` +- if `rootFilesystem.mode` is `"read-only"`, AgentOs mounts the lowers without a writable upper +- if `disableDefaultBaseLayer` is not set, the bundled base snapshot is appended as the deepest lower layer +- if `disableDefaultBaseLayer` is `true`, the lower stack comes only from `rootFilesystem.lowers` +- if `disableDefaultBaseLayer` is `true` and no lowers are provided, the root overlay has no lower layers +- if `rootFilesystem.lowers` is provided, `lowers[0]` is the highest-precedence lower and the last entry is the deepest/base lower +- AgentOs imports each `RootLowerInput` into the internal root store before composing `/` + +Documentation requirement: + +- `disableDefaultBaseLayer` must be documented in the `AgentOs.create()` API docs because it changes the default root shape substantially + +Bundled base-layer requirement: + +- core should statically import the built base filesystem JSON artifact into the bundle +- callers should not need to load `base-filesystem.json` manually +- the default root path should not depend on a user-visible async import step for the base layer +- `RootFilesystemConfig` intentionally accepts root-lower inputs rather than arbitrary `SnapshotLayerHandle`s because the root store remains core-owned in phase 1 + +Equivalent conceptual expansion: + +```ts +rootFilesystem: { + type: "overlay", + mode: "ephemeral", + disableDefaultBaseLayer: false, +} +``` + +### 2. Mount API + +Mounts should support both plain mounted filesystems and declarative overlay mounts. + +```ts +interface PlainMountConfig { + path: string; + driver: VirtualFileSystem; + readOnly?: boolean; +} + +interface OverlayMountConfig { + path: string; + filesystem: { + type: "overlay"; + store: LayerStore; + mode?: OverlayFilesystemMode; // default "ephemeral" + lowers: SnapshotLayerHandle[]; // highest-precedence lower first + }; +} + +type MountConfig = PlainMountConfig | OverlayMountConfig; +``` + +This keeps one consistent user model: + +- plain mounts for host-dir, proc/dev, and custom VFS drivers +- declarative overlay mounts for paths like `/data` +- no separate overlay-mount helper is required in phase 1 + +Implementation rule: + +- plain mounts take an existing `VirtualFileSystem` +- overlay mounts are resolved through the provided `LayerStore` +- `OverlayMountConfig` is the preferred public AgentOs mount surface for overlay filesystems +- prebuilding an overlay `VirtualFileSystem` and passing it through `PlainMountConfig` is a lower-level/internal pattern rather than the primary public API +- if `filesystem.mode` is omitted or `"ephemeral"`, AgentOs creates one fresh writable upper by calling `filesystem.store.createWritableLayer()` and owns that upper for the lifetime of the mount +- if `filesystem.mode` is `"read-only"`, AgentOs creates no upper layer for the mount +- the auto-created writable upper for a declarative overlay mount is not exposed directly in phase 1 + +### 3. Layer Handle API + +Introduce explicit layer handles for overlay construction. + +```ts +interface LayerHandle { + kind: "writable" | "snapshot"; + storeId: string; + layerId: string; +} + +interface WritableLayerHandle extends LayerHandle { + kind: "writable"; + leaseId: string; +} + +interface SnapshotLayerHandle extends LayerHandle { + kind: "snapshot"; +} +``` + +Important point: + +- `WritableLayerHandle` and `SnapshotLayerHandle` are not separate storage models +- they are constrained forms of the same underlying layer abstraction +- `LayerHandle` is an opaque, store-bound handle returned by a `LayerStore`; callers should not manually construct one from plain data +- `storeId` defines the compatibility domain for overlay composition +- all layers in one overlay view must come from the same `storeId` +- writable handles are leased capabilities; raw `layerId` alone is not enough to reopen an active writer safely +- snapshot handles are reopenable descriptors within the same compatible store +- snapshot handles may be serialized as descriptors; writable handles should be treated as live capabilities rather than durable IDs + +### 4. Layer Store API + +Advanced users and backend authors need a way to create, open, import, and seal layers. + +Proposed shape: + +```ts +interface LayerStore { + readonly storeId: string; + createWritableLayer(): Promise; + importSnapshot(source: SnapshotImportSource): Promise; + openSnapshotLayer(layerId: string): Promise; + sealLayer(layer: WritableLayerHandle): Promise; + createOverlayFilesystem( + options: + | { + mode?: "ephemeral"; + upper: WritableLayerHandle; + lowers: SnapshotLayerHandle[]; + } + | { + mode: "read-only"; + lowers: SnapshotLayerHandle[]; + }, + ): VirtualFileSystem; +} +``` + +Possible snapshot import sources in phase 1: + +- base filesystem JSON artifact +- explicit snapshot export/import format + +This is the right level for: + +- SQLite-backed local layers +- SQLite metadata + S3 block-store layers +- future cloud/persistent stores + +Core ownership rule: + +- `LayerStore` should be a core API in `packages/core` +- backend packages implement or return `LayerStore` +- AgentOs uses a core-owned default internal temp-dir-backed `LayerStore` for the root overlay in phase 1 +- public custom root-store injection for `/` is deferred in phase 1 + +Lifecycle rules: + +- `sealLayer()` creates a new immutable snapshot layer ID from the current visible tree +- `sealLayer()` invalidates the writable handle it sealed +- a writable layer may be attached to at most one active overlay view +- writable layers are not reopened by raw `layerId` while active +- writable layers are single-writer only in phase 1 +- the default root writable layer is a tmp-dir-backed filesystem layer managed by core's internal root `LayerStore` + +### 5. Snapshot Import Source + +`SnapshotImportSource` is intentionally backend-agnostic but phase 1 only needs to guarantee our internal snapshot/base-artifact inputs. + +```ts +type SnapshotImportSource = + | { kind: "base-filesystem-artifact"; source: unknown } + | { kind: "snapshot-export"; source: unknown }; +``` + +The exact payload shape is backend-specific. The important requirement is that importing produces a `SnapshotLayerHandle`. + +Phase-1 note: + +- the default root base snapshot should be bundled into core from `base-filesystem.json` +- callers should not need to load that JSON manually +- OCI import/export is deferred and tracked in `TODO.md` + +### 6. Root Snapshot Lifecycle + +If the root is an overlay filesystem by default, AgentOs needs a high-level lifecycle API for it. + +Proposed shape: + +```ts +interface AgentOs { + snapshotRootFilesystem(): Promise; +} +``` + +Semantics: + +- creates a new frozen snapshot-export descriptor from the current visible root tree +- does not require callers to manipulate the live root upper handle directly +- the returned value is reusable in `rootFilesystem.lowers` or via `LayerStore.importSnapshot(...)` + +This remains async because sealing the live root into a durable snapshot may involve metadata and block-store work even though the default base snapshot itself is bundled into core synchronously at build time. + +## What Users Implement + +There should be two extension surfaces. + +### A. Mounted Filesystem Drivers + +If a user wants to mount a filesystem directly, they implement: + +```ts +interface VirtualFileSystem { ... } +``` + +Examples: + +- host directory projection +- special kernel filesystems +- a custom remote filesystem + +These are mountable. + +They are not automatically valid overlay layers. + +### B. Overlay Storage Backends + +If a user wants a filesystem to participate in layered overlay semantics, they should implement the layer/storage side: + +- a `LayerStore` +- and, internally, whatever metadata/block abstractions the layer engine needs + +Examples: + +- SQLite metadata + local block store +- SQLite metadata + S3 block store +- future Postgres metadata + object storage block store + +These produce `LayerHandle`s that can be used to build an overlay filesystem. + +### Backend Author API + +Backend packages should expose a factory that returns a `LayerStore`. + +Example: + +```ts +function createLayerStore(config: BackendSpecificConfig): LayerStore; +``` + +Core should depend on: + +- `LayerStore` +- `WritableLayerHandle` +- `SnapshotLayerHandle` + +Core should not depend on backend-specific metadata or block-store types. + +## What Users Should Not Implement Twice + +Users should not implement: + +- one interface for writable layers +- another interface for snapshot layers + +Instead: + +- implement the storage engine once +- represent writable vs frozen as layer state/invariants + +Example lifecycle: + +```ts +const upper = await store.createWritableLayer(); +const snapshot = await store.sealLayer(upper); +``` + +Same layer model, different state. + +## Metadata Ownership + +### Each Layer Owns Metadata + +Each layer should own the metadata needed to describe the filesystem tree represented by that layer. + +That means: + +- a layer has its own inode/dentry/chunk mapping state +- layer identity is explicit +- layers are not anonymous slices of some parent mount table + +Important clarification: + +- this is logical ownership, not necessarily one physical database per layer +- a `LayerStore` may keep multiple layers in one shared metadata engine or one shared block store +- the requirement is that layer identity, isolation, and durability are explicit in the storage model +- inode allocation, dentries, and chunk references must all be scoped by `layer_id` +- cross-layer references are forbidden except through explicit provenance/import rules + +### Writable Upper Layer + +The active writable upper layer is where live overlay state lives: + +- whiteouts +- opaque directories +- copy-up provenance +- writable dentries/inodes/chunk updates + +### Frozen Lower Layers + +Frozen lower layers are snapshot layers suitable for use as overlay lowers. + +They should not carry live overlay runtime state such as: + +- active whiteouts +- active opaque markers +- live copy-up bookkeeping + +If a writable layer is sealed into a reusable lower snapshot, the resulting snapshot must behave like an ordinary immutable lower layer. The internal storage encoding is an implementation detail. + +### Middle Layers + +Middle layers follow the same rule as any other lower: + +- they use the same `LayerHandle` abstraction +- they are just frozen snapshots with higher precedence than older lowers + +So yes: + +- they can use the same interface +- they just are not writable + +## Why a Plain Mounted VFS Is Not a Layer + +Overlay construction needs semantics that a generic mounted `VirtualFileSystem` does not provide: + +- layer identity +- sealing/freezing +- whiteout rules +- copy-up destination control +- provenance +- fsck/validation +- snapshot import/export + +So this should be invalid: + +```ts +someStore.createOverlayFilesystem({ + upper: someArbitraryVirtualFileSystem, + lowers: [someOtherVirtualFileSystem], +}); +``` + +The overlay builder should consume explicit layer handles, not arbitrary mounted filesystems. + +## S3, Host Mounts, and Other Examples + +### S3-Backed Overlay Filesystem + +S3 can support overlay filesystems if it is used as block storage under a layer store. + +Good model: + +- durable metadata backend is present +- S3 stores blocks/chunks +- the layer store produces layer handles +- `LayerStore.createOverlayFilesystem(...)` builds the merged VFS from those handles + +Bad model: + +- "mount raw S3 VFS and expect the parent/root overlay to absorb it" +- "use S3 alone as the layer backend without durable metadata" + +### Host Directory Mount + +A host directory mount is a direct mount boundary, not a layer. + +It should be treated like a Docker bind mount: + +- mountable at a path +- separate from root overlay internals +- not implicitly usable as a lower or upper layer + +### Another Overlay Mount + +This is valid: + +```ts +const dataStore = createLayerStore({ + // backend-specific config +}); + +const dataBase = await dataStore.importSnapshot(seedSnapshot); + +await AgentOs.create({ + mounts: [ + { + path: "/data", + filesystem: { + type: "overlay", + store: dataStore, + lowers: [dataBase], + }, + }, + ], +}); +``` + +Here: + +- `/` is one overlay filesystem +- `/data` is another overlay filesystem +- they are still separate mount boundaries + +## Recommended High-Level Examples + +### Example 1: Default Root + +```ts +const vm = await AgentOs.create(); +``` + +Behavior: + +- root is an overlay filesystem automatically +- lower base layer comes from the bundled built base filesystem artifact from `base-filesystem.json` +- upper layer is a fresh writable layer in the internal temp-dir-backed layer store + +### Example 2: Root With Extra Middle Layers + +```ts +const vm = await AgentOs.create({ + rootFilesystem: { + disableDefaultBaseLayer: true, + lowers: [ + snapshotExportA, + snapshotExportB, + { kind: "bundled-base-filesystem" }, + ], + }, +}); +``` + +Interpretation: + +- `snapshotExportA` has higher precedence than `snapshotExportB` +- `snapshotExportB` has higher precedence than the bundled base filesystem lower +- because `disableDefaultBaseLayer` is `true`, AgentOs does not append another bundled base lower underneath these inputs +- AgentOs still creates one writable upper layer automatically for the live VM + +### Example 3: Root Plus Host Bind Mount + +```ts +const vm = await AgentOs.create({ + mounts: [ + { path: "/workspace", driver: hostDirFs, readOnly: false }, + ], +}); +``` + +Interpretation: + +- `/workspace` is a separate mount boundary +- root overlay does not see through it + +### Example 4: Root Plus S3-Backed Layered Mount + +```ts +const dataStore = createLayerStore({ + // durable metadata backend + S3 block store config +}); + +const seedBase = await dataStore.importSnapshot(seedSnapshot); + +const vm = await AgentOs.create({ + mounts: [ + { + path: "/data", + filesystem: { + type: "overlay", + store: dataStore, + lowers: [seedBase], + }, + }, + ], +}); +``` + +Interpretation: + +- `/data` has overlay semantics internally +- `/data` is still a separate mount from `/` + +## Invariants + +1. One overlay view has exactly one writable upper layer, or zero uppers in explicit read-only mode. +2. Lower layers are frozen snapshots. +3. Lower ordering is explicit: `lowers[0]` is the highest-precedence lower. +4. Middle layers are just higher-precedence frozen lower layers. +5. Mounts and layers are distinct concepts. +6. Arbitrary mounted `VirtualFileSystem` drivers are not valid overlay layers unless they explicitly expose layer handles through a layer store. +7. Root is an overlay filesystem by default. +8. Cross-mount rename/link and similar inode-moving operations retain normal mount semantics rather than overlay-specific magic. +9. A writable layer may be attached to at most one active overlay view. +10. The bundled base filesystem snapshot is included by default unless `disableDefaultBaseLayer` is set. +11. The default root writable upper lives in the core-owned temp-dir-backed layer store unless a later design explicitly adds custom root-store injection. +12. Public overlay-mount configuration is declarative in phase 1; no separate overlay-mount helper API is required. + +## Deferred + +1. OCI import/export support for overlay layers and snapshots is deferred beyond phase 1 and tracked in `TODO.md`. +2. Public custom root-store injection for `/` is deferred beyond phase 1; root store ownership remains in core for now. diff --git a/.agent/specs/overlay-metadata-proposal.md b/.agent/specs/overlay-metadata-proposal.md new file mode 100644 index 000000000..1cc4abc3e --- /dev/null +++ b/.agent/specs/overlay-metadata-proposal.md @@ -0,0 +1,936 @@ +# Durable Overlay Filesystem in Core Metadata + +## Status + +Reviewed after adversarial pass. This proposal assumes breaking changes are acceptable. + +## Summary + +We should replace the current wrapper-based overlay in `packages/core/src/backends/overlay-backend.ts` with a metadata-native overlay implementation in core. + +Today the overlay implementation is: + +- one lower `VirtualFileSystem` +- one upper `VirtualFileSystem` +- an in-memory `Set` of whiteouts + +That is not durable. Deletes of lower-layer files exist only in process memory, not in persistent metadata. The replacement should store all overlay semantics in the metadata layer itself: + +- immutable, materialized lower layers +- one writable upper layer per view +- durable whiteouts +- durable opaque directories +- durable copy-up provenance +- crash-safe interaction with block storage + +In our runtime model, whiteouts and opaque markers are live only in the active upper layer. Frozen lower snapshots are materialized trees and must not contain live whiteout or opaque markers. + +The goal is to match Linux OverlayFS behavior as closely as our API allows, not to build a generic copy-on-write filesystem that only resembles it. + +## Sources + +Primary references that define the target semantics: + +- Linux OverlayFS docs: https://docs.kernel.org/filesystems/overlayfs.html +- Linux kernel doc mirror: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html +- Docker `overlay2` docs: https://docs.docker.com/engine/storage/drivers/overlayfs-driver/ +- OCI layer format: https://raw.githubusercontent.com/opencontainers/image-spec/main/layer.md + +Relevant subsections: + +- Upper/lower/workdir: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#upper-and-lower +- Whiteouts and opaque directories: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#whiteouts-and-opaque-directories +- Non-directories and copy-up: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#non-directories +- Renaming directories / `EXDEV`: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#renaming-directories +- Inode properties / `xino`: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#inode-properties +- `xino`: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#xino +- `metacopy`: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#metadata-only-copy-up + +## What OverlayFS Does + +Linux OverlayFS presents: + +- one writable upper layer +- one or more read-only lower layers +- one merged view + +The behaviors we should treat as normative: + +1. Upper beats lower for any non-directory name collision. +2. Directory names merge unless the upper directory is opaque. +3. Deleting a lower entry records a whiteout in upper. +4. Replacing lower directory contents while keeping the directory uses an opaque upper directory. +5. Writing or mutating metadata on a lower non-directory triggers copy-up first. +6. By default, renaming a lower or merged directory returns `EXDEV`. +7. OverlayFS can optionally preserve more hardlink behavior with `index=on`; without that feature, lower hardlinks may diverge after copy-up. +8. OCI layers serialize deletes as `.wh.` and opaque directories as `.wh..wh..opq`. + +Important caller-visible details from the kernel docs: + +- Only name lists merge. Directory metadata comes from upper if upper exists. +- Non-directory `st_dev` / `st_ino` may come from the visible source object and may change after copy-up unless `xino` is used. +- `readdir(3)` has cursor caching behavior that depends on open directory handles. + +## Explicit Compatibility Decisions + +These choices make phase 1 concrete and keep it aligned with real OverlayFS behavior. + +1. Phase 1 should match OverlayFS with `index=off` for lower hardlinks. +2. Phase 1 should add `dev` to `VirtualStat`. Without `st_dev`, we cannot represent OverlayFS-like object identity accurately. +3. Phase 1 does not need a kernel-style `workdir`. +4. Phase 1 does not need to expose full POSIX `seekdir`/offset behavior at the public API boundary, but the metadata layer should still use view-bound directory handles so merged `readdir` snapshots are explicit and stable per open handle. +5. Frozen lower layers in our runtime are materialized trees, not live OCI diff layers. OCI whiteout and opaque markers are consumed during import and are not re-interpreted from lower layers at lookup time. + +Decision 3 needs explanation: + +- Linux OverlayFS needs `workdir` because it implements copy-up and rename over a real upper filesystem. +- Our equivalent should be SQLite transactions for metadata plus staged block writes and deferred block GC for data. +- That gives us the same correctness property: no visible partial mutation after crash. + +## Why the Current Model Is Wrong + +The current model is structurally incorrect for durable overlay semantics: + +1. Lower-layer deletes are transient. +2. Overlay state is split across lower fs, upper fs, and wrapper-local memory. +3. There is no durable representation of whiteouts or opaque directories. +4. Copy-up provenance is implicit. +5. There is no clean path to persistent overlays, snapshots, or OCI diff import/export. + +## Breaking Changes We Should Make + +### Replace `FsMetadataStore` With a Layer-Aware API + +The current metadata model assumes one canonical tree: + +- one inode namespace +- one dentry table +- one symlink table +- one chunk map + +That is the wrong abstraction for overlay semantics. The new API should be explicitly view-aware. + +Minimum shape: + +```ts +interface OverlayView { + viewId: number; + upperLayerId: number; + lowerLayerIds: number[]; // highest-precedence lower first +} + +interface NodeRef { + viewId: number; + layerId: number; + ino: number; +} +``` + +Recommended direction: + +- add a new `LayeredMetadataStore` interface +- replace plain inode arguments/returns with `NodeRef` +- keep `transaction(fn)` as a first-class API contract for multi-table overlay mutations +- update `ChunkedVFS` to operate in the context of a view-bound metadata session, not a raw global store +- make metadata operations explicitly view-bound or layer-bound so per-layer inode numbers are never ambiguous at call sites +- make merged-directory iteration a first-class operation so `readdir` caching semantics are explicit instead of accidental +- keep an adapter only if we need temporary compatibility with old single-layer callers + +### Extend `VirtualStat` + +Current `VirtualStat` only exposes `ino`. OverlayFS semantics depend on the `st_dev` / `st_ino` pair. + +Required breaking change: + +```ts +interface VirtualStat { + dev: number; + ino: number; + mode: number; + size: number; + isDirectory: boolean; + isSymbolicLink: boolean; + atimeMs: number; + mtimeMs: number; + ctimeMs: number; + birthtimeMs: number; + nlink: number; + uid: number; + gid: number; +} +``` + +Phase 1 identity policy: + +- directories report overlay-view `dev` +- non-directories report identity from the currently visible source object +- non-directory `dev` / `ino` may change after copy-up, matching OverlayFS without `xino` + +If we later want `xino`-like stability, that is a separate feature. + +### Versioning Must Also Become Layer-Aware + +If versioning stays, the current `FsMetadataStoreVersioning` shape is no longer sufficient because it keys everything by bare `ino`. + +Required breaking changes: + +- add `layerId` to `createVersion` +- add `layerId` to `getVersion` +- add `layerId` to `listVersions` +- add `layerId` to `getVersionChunkMap` +- add `layerId` to `deleteVersions` +- add `layerId` to `restoreVersion` + +Until `metacopy` exists, the public API should still behave as if `storageMode` is effectively `inline | chunked`. + +## Proposed Schema + +The schema below is the durable phase-1 model. Optional phase-2 tables are called out separately. + +### 0. Schema Metadata + +```sql +CREATE TABLE schema_meta ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +); +``` + +Required keys: + +- `schema_version` +- `feature_flags` + +Version mismatches must fail closed. Migration is one-way. + +### 1. Layers + +```sql +CREATE TABLE layers ( + layer_id INTEGER PRIMARY KEY AUTOINCREMENT, + kind TEXT NOT NULL CHECK(kind IN ('base', 'snapshot', 'upper')), + writable INTEGER NOT NULL CHECK(writable IN (0, 1)), + frozen INTEGER NOT NULL DEFAULT 0 CHECK(frozen IN (0, 1)), + state TEXT NOT NULL CHECK(state IN ('draft', 'active', 'sealed', 'deleted')), + created_at_ms INTEGER NOT NULL, + sealed_at_ms INTEGER, + description TEXT +); +``` + +Notes: + +- lower layers must be `frozen=1` +- upper layers are writable only while their view is active +- every layer root is always inode `1` in that layer +- enforce with triggers: `kind='upper'` implies `writable=1 AND frozen=0`, and `kind IN ('base','snapshot')` implies `writable=0 AND frozen=1` + +Using `ino=1` per layer keeps root resolution simple because the primary key is `(layer_id, ino)`. + +### 1.1 Layer Inode Counters + +```sql +CREATE TABLE layer_counters ( + layer_id INTEGER PRIMARY KEY, + next_ino INTEGER NOT NULL, + FOREIGN KEY (layer_id) REFERENCES layers(layer_id) ON DELETE CASCADE +); +``` + +This replaces the old single global allocator. New inode numbers are allocated per layer inside the same transaction that creates the inode. + +### 2. Overlay Views + +```sql +CREATE TABLE overlay_views ( + view_id INTEGER PRIMARY KEY AUTOINCREMENT, + upper_layer_id INTEGER NOT NULL, + state TEXT NOT NULL DEFAULT 'active' + CHECK(state IN ('active', 'sealed', 'deleted')), + created_at_ms INTEGER NOT NULL, + sealed_at_ms INTEGER, + description TEXT, + UNIQUE (upper_layer_id), + FOREIGN KEY (upper_layer_id) REFERENCES layers(layer_id) +); + +CREATE TABLE overlay_view_lowers ( + view_id INTEGER NOT NULL, + lower_order INTEGER NOT NULL, + lower_layer_id INTEGER NOT NULL, + PRIMARY KEY (view_id, lower_order), + UNIQUE (view_id, lower_layer_id), + FOREIGN KEY (view_id) REFERENCES overlay_views(view_id) ON DELETE CASCADE, + FOREIGN KEY (lower_layer_id) REFERENCES layers(layer_id) +); +``` + +Rules: + +- one upper layer belongs to at most one active view +- lower layers may be shared across many views +- `lower_order=0` is the highest-precedence lower layer +- once a layer is referenced as a lower layer, it must never become writable again +- enforce with triggers: `lower_layer_id` must never equal `upper_layer_id`, lowers must be frozen/non-writable, and uppers must be writable/non-frozen + +### 3. Inodes + +```sql +CREATE TABLE inodes ( + layer_id INTEGER NOT NULL, + ino INTEGER NOT NULL, + type TEXT NOT NULL CHECK(type IN ('file', 'directory', 'symlink')), + mode INTEGER NOT NULL, + uid INTEGER NOT NULL DEFAULT 0, + gid INTEGER NOT NULL DEFAULT 0, + size INTEGER NOT NULL DEFAULT 0, + nlink INTEGER NOT NULL DEFAULT 0, + atime_ms INTEGER NOT NULL, + mtime_ms INTEGER NOT NULL, + ctime_ms INTEGER NOT NULL, + birthtime_ms INTEGER NOT NULL, + storage_mode TEXT NOT NULL + CHECK(storage_mode IN ('inline', 'chunked', 'metacopy')), + inline_content BLOB, + + overlay_opaque INTEGER NOT NULL DEFAULT 0 CHECK(overlay_opaque IN (0, 1)), + + origin_layer_id INTEGER, + origin_ino INTEGER, + + data_origin_layer_id INTEGER, + data_origin_ino INTEGER, + + redirect_path TEXT, + + PRIMARY KEY (layer_id, ino), + FOREIGN KEY (layer_id) REFERENCES layers(layer_id), + FOREIGN KEY (origin_layer_id, origin_ino) REFERENCES inodes(layer_id, ino), + FOREIGN KEY (data_origin_layer_id, data_origin_ino) REFERENCES inodes(layer_id, ino), + + CHECK (overlay_opaque = 0 OR type = 'directory'), + CHECK ((origin_layer_id IS NULL) = (origin_ino IS NULL)), + CHECK ((data_origin_layer_id IS NULL) = (data_origin_ino IS NULL)), + CHECK (redirect_path IS NULL OR type = 'directory'), + CHECK ( + (storage_mode = 'inline' AND type = 'file' AND data_origin_layer_id IS NULL AND data_origin_ino IS NULL) OR + (storage_mode = 'chunked' AND type = 'file' AND inline_content IS NULL AND data_origin_layer_id IS NULL AND data_origin_ino IS NULL) OR + (storage_mode = 'metacopy' AND type = 'file' AND inline_content IS NULL AND data_origin_layer_id IS NOT NULL AND data_origin_ino IS NOT NULL) OR + (type IN ('directory', 'symlink') AND storage_mode = 'inline' AND inline_content IS NULL AND data_origin_layer_id IS NULL AND data_origin_ino IS NULL) + ) +); +``` + +Meaning of overlay-specific columns: + +- `overlay_opaque`: only meaningful on upper-layer directories +- `origin_layer_id` / `origin_ino`: durable copy-up provenance +- `data_origin_layer_id` / `data_origin_ino`: reserved for future `metacopy` +- `redirect_path`: reserved for future `redirect_dir`; if enabled later it stores the original overlay-root-absolute path after directory-only copy-up + +Phase-1 rules: + +- `storage_mode='metacopy'` is reserved but must not be emitted +- whiteout rows may exist only in active upper layers +- only upper-layer directories may carry `overlay_opaque=1` +- `redirect_path`, if present in a later phase, must be a normalized absolute overlay path +- `storage_mode` should default to `inline` on inode creation and be promoted to `chunked` by write paths, so phase-1 callers do not need to pass it explicitly +- `chunks` may exist only for file inodes and `symlinks` may exist only for symlink inodes; enforce with write-path checks plus `fsck`, and add triggers if we want DB-level enforcement + +### 4. Directory Entries + +```sql +CREATE TABLE dentries ( + layer_id INTEGER NOT NULL, + parent_ino INTEGER NOT NULL, + name TEXT NOT NULL, + entry_kind TEXT NOT NULL CHECK(entry_kind IN ('normal', 'whiteout')), + child_ino INTEGER, + + PRIMARY KEY (layer_id, parent_ino, name), + + FOREIGN KEY (layer_id, parent_ino) REFERENCES inodes(layer_id, ino), + FOREIGN KEY (layer_id, child_ino) REFERENCES inodes(layer_id, ino), + + CHECK ( + (entry_kind = 'normal' AND child_ino IS NOT NULL) OR + (entry_kind = 'whiteout' AND child_ino IS NULL) + ), + CHECK (name <> '' AND name <> '.' AND name <> '..' AND instr(name, '/') = 0) +); + +CREATE INDEX idx_dentries_child + ON dentries(layer_id, child_ino); +``` + +This is the core durability change. + +A lower-layer delete is represented as: + +- a row in the upper layer +- same logical parent path and `name` +- attached to the corresponding upper-layer parent directory inode +- `entry_kind='whiteout'` + +That is the durable equivalent of an OverlayFS whiteout. + +Important write-path rule: + +- if the parent directory exists only in lower, the implementation must first create or copy-up the ancestor directory chain in upper so the whiteout row has a real upper-layer parent inode + +### 5. Symlinks + +```sql +CREATE TABLE symlinks ( + layer_id INTEGER NOT NULL, + ino INTEGER NOT NULL, + target TEXT NOT NULL, + PRIMARY KEY (layer_id, ino), + FOREIGN KEY (layer_id, ino) REFERENCES inodes(layer_id, ino) +); +``` + +### 6. Chunk Mapping + +```sql +CREATE TABLE chunks ( + layer_id INTEGER NOT NULL, + ino INTEGER NOT NULL, + chunk_index INTEGER NOT NULL, + block_key TEXT NOT NULL, + PRIMARY KEY (layer_id, ino, chunk_index), + FOREIGN KEY (layer_id, ino) REFERENCES inodes(layer_id, ino) +); +``` + +This stays close to the current design. The main change is layer scoping. + +### 7. Pending Block Operations + +```sql +CREATE TABLE pending_block_ops ( + txn_id TEXT NOT NULL, + block_key TEXT NOT NULL, + op TEXT NOT NULL + CHECK(op IN ('publish', 'retire')), + created_at_ms INTEGER NOT NULL, + PRIMARY KEY (txn_id, block_key, op) +); +``` + +Purpose: + +- SQLite savepoints do not roll back block-store side effects +- recovery must know which block publishes and retirements were in flight +- crash before cleanup may leak blocks temporarily, but must never create visible metadata pointing at missing data + +### 8. Optional `index=on` Support (Phase 2) + +Phase 1 should match OverlayFS with `index=off`. That means lower hardlinks may diverge after one name is copied up. + +If we later want `index=on`-style behavior, add: + +```sql +CREATE TABLE copy_up_index ( + upper_layer_id INTEGER NOT NULL, + lower_layer_id INTEGER NOT NULL, + lower_ino INTEGER NOT NULL, + upper_ino INTEGER NOT NULL, + PRIMARY KEY (upper_layer_id, lower_layer_id, lower_ino), + UNIQUE (upper_layer_id, upper_ino), + FOREIGN KEY (upper_layer_id, upper_ino) REFERENCES inodes(layer_id, ino), + FOREIGN KEY (lower_layer_id, lower_ino) REFERENCES inodes(layer_id, ino) +); +``` + +This is not needed for phase-1 correctness. + +### 9. Optional Versions Table + +If versioning remains, it must also be layer-scoped: + +```sql +CREATE TABLE versions ( + layer_id INTEGER NOT NULL, + ino INTEGER NOT NULL, + version INTEGER NOT NULL, + size INTEGER NOT NULL, + created_at_ms INTEGER NOT NULL, + storage_mode TEXT NOT NULL, + inline_content BLOB, + chunk_map TEXT, + PRIMARY KEY (layer_id, ino, version), + FOREIGN KEY (layer_id, ino) REFERENCES inodes(layer_id, ino) +); +``` + +## How Resolution Works + +Path resolution in a view should be: + +1. Start from `(upper_layer_id, 1)` and from `(lower_layer_id, 1)` for each lower layer. +2. For each path component: + - check upper first + - if upper has a whiteout for the name: stop with `ENOENT` + - if upper has a normal non-directory: choose upper and stop lower lookup for that name + - if upper has a normal directory: + - if the upper directory is opaque: choose upper only + - otherwise merge with matching lower directories + - if upper has no entry: + - if the highest-precedence matching lower entry is a non-directory, choose that visible lower entry + - if one or more matching lower entries are directories, merge all matching lower directories in precedence order until a non-directory in a higher-precedence lower blocks the name +3. For merged directories: + - upper names come first + - lower names are appended only if not shadowed by upper or by a whiteout +4. Directory metadata comes from upper if an upper directory exists; lower directory metadata is hidden +5. Merged `readdir` results should be cached per opened directory handle and rebuilt after reopen or rewind, matching the OverlayFS model rather than recalculating mid-stream + +This matches the kernel rule that only name lists merge while directory metadata comes from upper. + +Important invariant: + +- lower layers are materialized trees +- live whiteouts and opaque markers in frozen lower layers are invalid and must be rejected by validation/fsck + +## How Mutations Work + +### Create New File + +- ensure ancestor directories exist in upper first +- create upper inode +- create upper dentry +- no lower mutation + +### Write Existing Upper File + +- modify upper inode and upper chunk map only + +### Write Existing Lower File + +- resolve visible lower object +- copy it up first +- write only to upper copy +- lower remains unchanged + +### `chmod` / `chown` / `utimes` / `truncate` on Lower File + +Phase 1: + +- full copy-up + +Phase 2: + +- optional metadata-only copy-up using `metacopy` + +### Delete Upper-Only File + +- remove upper dentry +- remove upper inode if link count reaches zero +- record unreachable blocks in `pending_block_ops` +- no whiteout required + +### Delete Upper File That Shadows Lower Content + +- remove the upper dentry +- remove the upper inode if link count reaches zero +- insert a whiteout for the same `parent/name` in the same transaction +- the lower object must remain hidden; merged lookup returns `ENOENT` + +### Delete Lower-Only File + +- ensure ancestor directories exist in upper first +- insert upper whiteout row +- do not mutate lower + +### Delete Lower or Merged Directory + +Default semantics: + +- `removeDir` on a non-empty merged view returns `ENOTEMPTY` +- if the visible merged directory is empty, ensure ancestor directories exist in upper first and insert a whiteout at the parent/name +- if the directory has upper participation and lower content beneath the same name, remove the upper state and publish the whiteout atomically + +### Delete Directory Contents But Keep Directory + +- create upper directory if needed +- remove any upper children being deleted +- mark the upper directory `overlay_opaque=1` + +This matches the semantic difference between: + +- whiteout: hide the named lower entry itself +- opaque directory: keep the directory, hide lower children + +### Rename + +- rename upper-only file or upper-only directory within upper normally +- rename lower-only or merged directory returns `EXDEV` by default +- future phase 2 may add `redirect_dir` + +### Hardlinks + +Phase 1 rule: + +- creating a hardlink to a lower non-directory triggers copy-up first +- once one lower hardlinked name is copied up, alias preservation is not guaranteed across the remaining lower names + +That is consistent with OverlayFS `index=off`. + +## Durability Protocol + +This is the part the original draft was too vague about. SQLite metadata is transactional, but block storage is not. + +Phase-1 write ordering should be: + +### Create / overwrite / copy-up + +1. Write new blocks to fresh block-store keys first. +2. Record pending `publish` / `retire` entries in `pending_block_ops`. +3. Commit metadata in one SQLite transaction so the inode and chunk map point at the new keys. +4. Finalize the `pending_block_ops` journal and retire old keys after commit on a best-effort basis. + +Crash outcomes: + +- crash before metadata commit: leaked blocks only, no visible partial file +- crash after metadata commit but before GC: visible file is correct, old blocks leak temporarily + +### Delete / truncate + +1. Commit metadata first, removing or replacing the chunk references. +2. Record pending `retire` entries in `pending_block_ops`. +3. Delete them asynchronously. + +The invariant is: + +- visible metadata must never reference a block that is missing +- leaks are acceptable temporarily +- partial visible state is not + +Required SQLite settings for this contract: + +- `journal_mode=WAL` +- `synchronous=FULL` +- `foreign_keys=ON` + +## OCI Mapping + +Internal SQL encoding and OCI tar encoding are not the same thing. The important requirement is semantic equivalence. + +Canonical import rules: + +- `.wh.` becomes one whiteout row for `` +- `.wh..wh..opq` becomes `overlay_opaque=1` on the containing upper directory + +Canonicalization rules: + +- if a layer contains both `.wh.` and a normal ``, the whiteout is redundant because whiteouts only apply to lower layers; keep the normal entry +- if a layer contains `.wh..wh..opq` and explicit child whiteouts for lower-only children, accept both but canonicalize to the opaque directory plus the remaining actual upper entries +- import order must not matter; OCI explicitly requires opaque handling to be order-independent +- reject ordinary OCI export/import paths whose basename begins with `.wh.` unless they are actual whiteout markers; OCI reserves that prefix +- OCI markers are consumed during import; frozen lower layers in our runtime are materialized trees, not live OCI diff layers + +Raw-layer note: + +- if a frozen layer is inspected through a raw layer API, it is a materialized tree without overlay whiteout/opaque markers +- `entry_kind='whiteout'` and `overlay_opaque=1` are only valid in active upper layers + +## Why This Matches OverlayFS Closely + +### Whiteouts + +OverlayFS uses upper whiteouts to hide lower names. Our equivalent is a durable upper-layer whiteout dentry row. + +### Opaque Directories + +OverlayFS uses an opaque marker on upper directories. Our equivalent is `overlay_opaque=1` on upper directories. + +### Copy-Up + +OverlayFS copies up on the first operation that requires write access or metadata mutation. Our equivalent is full copy-up into upper before that operation. + +### `EXDEV` on Lower or Merged Directory Rename + +OverlayFS returns `EXDEV` by default. We should do the same. + +### `index=off` Hardlink Behavior + +OverlayFS without `index=on` can break lower hardlink aliasing on copy-up. Phase 1 should document and match that instead of accidentally promising stronger behavior than the kernel default. + +### No Separate Workdir + +Linux needs `workdir` because its implementation runs over a real upper filesystem. We can replace that with: + +- SQLite transactions +- staged block writes +- deferred block GC + +That preserves the same correctness property without a separate visible filesystem layer. + +## Recommended Phases + +### Phase 1: Required Parity + +Implement: + +- upper + multiple lowers +- durable whiteouts +- durable opaque directories +- full copy-up on the first operation requiring write access or metadata mutation +- full copy-up on metadata mutation +- merged directory listing +- `EXDEV` for lower and merged directory rename +- `VirtualStat.dev` +- durable overlay views in SQLite +- staged block writes plus durable `pending_block_ops` + +Do not implement yet: + +- `redirect_dir` +- `metacopy` +- `index=on` +- export-grade origin verification + +### Phase 2: Closer Parity + +Implement: + +- `redirect_dir` +- `metacopy` +- `index=on` equivalent using `copy_up_index` +- OCI import/export helpers +- optional `xino`-like stable identity if we decide we need it + +## Test Matrix + +This is the minimum set. Missing items are gaps. + +### A. View Construction + +1. Create a view with one upper and one lower. +2. Create a view with one upper and multiple lowers. +3. Reject a view with a writable lower. +4. Reject duplicate lower order. +5. Reject a frozen upper. +6. Re-open a persisted view after restart. +7. Verify every layer root is inode `1`. +8. Reject reuse of the same upper by two active views. +9. Reject duplicate `(view_id, lower_layer_id)` rows. + +### B. Basic Visibility + +1. Read file that exists only in lower. +2. Read file that exists only in upper. +3. Upper file hides lower file of same name. +4. Upper symlink hides lower file of same name. +5. Upper file hides lower directory of same name. +6. Upper directory merges with lower directory of same name. +7. Directory metadata comes from upper when upper directory exists. + +### C. Whiteouts + +1. Delete lower-only file creates one upper whiteout row. +2. Deleted lower-only file does not resolve in merged view. +3. Deleted lower-only file is still accessible in the raw lower layer. +4. Deleting lower-only file does not create a copied-up inode. +5. Whiteout hides matching lower file in directory listing. +6. Whiteout hides matching lower directory in directory listing. +7. Whiteout survives restart. +8. Recreating the same path removes the whiteout and exposes the new upper entry. +9. Delete lower-only file whose parent exists only in lower first creates the ancestor upper directory chain. +10. Lower layers never contain `entry_kind='whiteout'` rows. + +### D. Opaque Directories + +1. Opaque upper directory hides lower children. +2. Opaque upper directory still exposes upper children. +3. Non-opaque upper directory merges lower children. +4. Opaque marker survives restart. +5. Listing an opaque directory does not leak lower names. +6. OCI import of redundant explicit whiteouts beneath an opaque directory canonicalizes correctly. +7. Lower layers never contain `overlay_opaque=1`. + +### E. Copy-Up + +1. First operation requiring write access or metadata mutation on a lower file performs copy-up. +2. Second write to the same file does not copy-up again. +3. Copy-up preserves mode, uid, gid, timestamps, symlink target, and file size. +4. Copy-up ensures ancestor directories exist in upper. +5. Metadata mutation on lower file also triggers copy-up. +6. Hardlink creation against lower file triggers copy-up first. +7. Symlink creation does not trigger copy-up. +8. Phase 1 explicitly allows one lower hardlink to diverge from its lower siblings after copy-up. + +### F. Directory Behavior + +1. `mkdir -p` on existing lower directory is a no-op. +2. `mkdir -p` on lower symlink-to-directory does not replace the symlink. +3. New upper directory is visible immediately. +4. Empty merged directory lists correctly. +5. Merged directory with both upper and lower names returns a deduplicated list. +6. Merged directory listings remain stable for the lifetime of an opened directory handle and are rebuilt after reopen or rewind. + +### G. Rename Semantics + +1. Rename upper-only file works. +2. Rename lower-only file copies up then removes the old visible name. +3. Rename upper-only directory works. +4. Rename lower-only directory returns `EXDEV`. +5. Rename merged directory returns `EXDEV`. +6. Future `redirect_dir` tests record redirect metadata and resolve correctly. +7. Rename of a lower hardlinked file does not promise alias preservation in phase 1. +8. Crash during lower-directory rename fallback does not leave half-copied directories or stray whiteouts. + +### H. Delete Semantics + +1. Delete upper-only file removes upper dentry and upper inode if link count reaches zero. +2. Delete lower-only file creates whiteout only. +3. Deleting an upper file that shadowed lower removes the upper state and leaves a whiteout, so merged lookup returns `ENOENT`. +4. Delete lower-only empty directory creates whiteout at parent/name. +5. Delete contents of merged directory while keeping directory sets opaque marker. +6. Recursive delete on mixed upper/lower subtree behaves deterministically. +7. `removeDir` on a merged directory with any visible child returns `ENOTEMPTY`. +8. Crash during opaque-directory replacement does not leave both lower children visible and upper opaque metadata committed. + +### I. Symlinks and Hardlinks + +1. Lower symlink resolves through overlay. +2. Copy-up of a symlink preserves target. +3. Hardlink to lower file copies up source first. +4. Hardlink counts remain correct within upper. +5. Phase 1 documents divergence of lower hardlinks after copy-up. +6. Phase 2 `index=on` tests preserve linked aliases across copy-up. + +### J. Multiple Lower Layers + +1. Highest-precedence lower wins over deeper lower for non-directories. +2. Upper whiteout hides all matching names from all lowers. +3. Opaque upper directory hides all lower children beneath it. +4. Same-named directories across multiple lowers merge in precedence order. +5. Copy-up provenance points to the first visible lower object. +6. Frozen lower snapshots are materialized trees and do not contain overlay whiteouts or opaque markers. + +### K. Persistence and Recovery + +1. Restart after whiteout preserves delete. +2. Restart after copy-up preserves upper data. +3. Restart after opaque marker preserves hidden lower children. +4. Crash during whiteout insertion rolls back cleanly. +5. Crash during copy-up never leaves partial visible state. +6. Crash during copy-up with chunked data does not leave a visible inode pointing at missing blocks. +7. Crash after block writes but before metadata commit leaks blocks only. +8. Crash after metadata commit but before GC leaves leaked old blocks only. +9. Kill-9 or simulated power-fail during whiteout, copy-up, rename, snapshot seal, and view attach reopens as either all-old or all-new state. + +### L. Transactions and Concurrency + +1. Whiteout insertion is atomic. +2. Copy-up and dentry replacement are atomic. +3. Rename sequence is atomic. +4. Directory listing never returns both a whiteout and the hidden lower name. +5. Constraint violations reject impossible states. +6. Concurrent copy-up of the same visible lower file does not produce two visible upper files. +7. Concurrent delete and write of the same visible lower path resolves to one committed winner. +8. Concurrent whiteout and recreate of the same path never exposes the lower entry in between. +9. Concurrent readers see either the old visible object or the new one, never a half-published mixed state. + +### M. Impossible-State Tests + +These must be rejected by schema, write-path checks, or `fsck`. + +1. Whiteout row with child inode set. +2. Normal dentry row without child inode. +3. `overlay_opaque=1` on non-directory inode. +4. Chunk rows for symlink or directory inodes. +5. Dentry referencing child inode in a different layer. +6. Upper layer modified after it becomes sealed or frozen. +7. Lower layer referenced by a view while still writable. +8. Invalid dentry names: empty string, `.`, `..`, or names containing `/`. +9. `origin_layer_id` without `origin_ino`, or vice versa. +10. Opaque directory flag on a non-upper layer directory. +11. Whiteout row on a non-upper layer. + +### N. `fsck` and Invariant Checks + +1. Every layer has exactly one root inode at `(layer_id, 1)`. +2. Every active view upper is writable and not frozen. +3. Every referenced lower is frozen. +4. Every `origin_layer_id` / `origin_ino` pair resolves. +5. Every `pending_block_ops` entry is replayable or safely discardable on recovery. +6. No chunk rows exist for non-file inodes. +7. No symlink row exists for a non-symlink inode. +8. No lower layer contains a whiteout row or an opaque marker. + +### O. OCI Interoperability + +1. Export whiteout row as `.wh.`. +2. Import `.wh.` as upper whiteout row. +3. Export explicit whiteouts by default for opaque directory replacement. +4. Import opaque whiteout as `overlay_opaque=1`. +5. Accept explicit whiteouts and opaque whiteout forms. +6. Import order of `.wh..wh..opq` and sibling entries does not matter. +7. Importing both `.wh.` and `` in the same layer canonicalizes to the normal entry because whiteouts only apply to lower layers. +8. OCI export/import rejects ordinary paths whose basename begins with `.wh.` because OCI reserves that prefix for whiteout markers. + +### P. Linux Differential Oracle + +1. Replay the same operation corpus against kernel OverlayFS and the metadata implementation. +2. Compare visible tree shape, `stat` results, rename/unlink/rmdir outcomes, and directory listing results. +3. Document every intentional deviation instead of allowing silent drift. + +## Comparison to the Current Core Metadata + +Current SQLite metadata model: + +- one inode namespace +- one dentry table +- one symlink table +- one chunk map +- optional versions + +Missing today: + +- no layer ordering +- no overlay view object +- no durable whiteouts +- no durable opaque directories +- no copy-up provenance +- no crash-safe overlay delete semantics + +Proposed model adds: + +- explicit layers +- explicit overlay views +- layer-scoped inodes and dentries +- durable whiteout rows +- durable opaque directories +- explicit lower ordering +- explicit pending block journal for crash safety + +## Migration Safety + +1. Ship an offline one-way migrator from the legacy single-tree metadata format to the layered schema. +2. Make schema-version mismatches fail closed. +3. Before cutover, export a legacy snapshot/base-filesystem artifact so there is a downgrade escape hatch outside the new DB format. + +## Recommendation + +We should implement this as a new layered metadata subsystem, not as an incremental patch on top of the in-memory whiteout wrapper. + +Recommended path: + +1. Keep the wrapper only as temporary compatibility for existing tests. +2. Add layer-aware SQLite metadata with the phase-1 schema above. +3. Extend `VirtualStat` with `dev`. +4. Update `ChunkedVFS` to resolve against a view-bound metadata session and use `NodeRef` instead of plain inode numbers internally. +5. Move base root filesystems and future snapshots onto immutable materialized lower layers plus durable upper metadata. +6. Gate rollout on `schema_version` checks, mount-time `fsck`, and the Linux differential test corpus. + +That is the cleanest route to durable overlay behavior that still maps directly onto Linux OverlayFS semantics. + +## Open Questions + +1. Do we want to ship code for only one lower layer first while keeping the schema multi-lower from day one? +2. Do we want full copy-up only in phase 1, or do we want `metacopy` immediately? +3. Is OverlayFS `index=off` behavior sufficient permanently, or do we want a later `index=on` equivalent? +4. Do we need OCI import/export in the first implementation, or only the internal durable model? +5. Do we want an adapter for the old `FsMetadataStore` API, or do we replace it outright? diff --git a/CLAUDE.md b/CLAUDE.md index 31da23829..559491dac 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -62,6 +62,9 @@ The registry software packages depend on `@rivet-dev/agent-os-registry-types` (i ## Architecture +- **The VM base filesystem artifact is derived from Alpine Linux, but runtime source should stay generic.** `packages/core/src/` must not hardcode Alpine-specific defaults or import Alpine-named helpers. The runtime consumes `packages/core/fixtures/base-filesystem.json` as the default root layer. +- **Base filesystem rebuild flow:** first capture a fresh Alpine snapshot with `pnpm --dir packages/core snapshot:alpine-defaults`, which writes `packages/core/fixtures/alpine-defaults.json`. Then run `pnpm --dir packages/core build:base-filesystem`, which rewrites the required AgentOs-specific values (for example `HOSTNAME=agent-os` and `/etc/hostname`) and emits `packages/core/fixtures/base-filesystem.json`. AgentOs uses that built artifact as the lower layer of an overlay-backed root filesystem. +- **The default VM filesystem model should be Docker-like.** The root filesystem should be a layered overlay view with one writable upper layer on top of one or more immutable lower snapshot layers. The base filesystem artifact is the initial lower layer; additional frozen lower layers may be stacked beneath the writable upper if needed. Do not design the default VM root as a pile of ad hoc post-boot mutations. - **Everything runs inside the VM.** Agent processes, servers, network requests -- all spawned inside the secure-exec kernel, never on the host. This is a hard rule with no exceptions. - The `AgentOs` class wraps a secure-exec `Kernel` and proxies its API directly - **All public methods on AgentOs must accept and return JSON-serializable data.** No object references (Session, ManagedProcess, ShellHandle) in the public API. Reference resources by ID (session ID, PID, shell ID). This keeps the API flat and portable across serialization boundaries (HTTP, RPC, IPC). @@ -80,11 +83,15 @@ The registry software packages depend on `@rivet-dev/agent-os-registry-types` (i - The old `fs-sqlite` and `fs-postgres` packages were deleted. They are replaced by the secure-exec `SqliteMetadataStore` and the `ChunkedVFS` composition layer. - File system drivers live in `registry/file-system/` (see Registry section above). They implement the `FsBlockStore` interface and are passed via `type: "custom"` mount. -- The Rivet actor integration (in the Rivet repo at `rivetkit-typescript/packages/rivetkit/src/agent-os/`) uses `ChunkedVFS(InMemoryMetadataStore + InMemoryBlockStore)` as a temporary in-memory solution. A persistent backend (actor KV-backed metadata + actor storage-backed blocks) is planned. +- The Rivet actor integration (in the Rivet repo at `rivetkit-typescript/packages/rivetkit/src/agent-os/`) currently uses `ChunkedVFS(InMemoryMetadataStore + InMemoryBlockStore)` as legacy temporary infrastructure. This is not an acceptable long-term model for filesystem correctness. Filesystem semantics must move to durable metadata and block storage rather than transient in-memory state. ## Filesystem Conventions - **OS-level content uses mounts, not post-boot writes.** If agentOS needs custom directories in the VM (e.g., `/etc/agentos/`), mount a pre-populated filesystem at boot — don't create the kernel and then write files into it afterward. This keeps the root filesystem clean and makes OS-provided paths read-only so agents can't tamper with them. +- **Filesystem semantics must be durable.** Any state that changes filesystem behavior — including overlay deletes, whiteouts, tombstones, copy-up state, directory entries, inode metadata, or file contents — must be represented in durable filesystem or metadata storage. Do not implement correctness-critical filesystem behavior with in-memory side tables, in-memory whiteout sets, or other transient hacks. +- **Overlay filesystem behavior must match Linux OverlayFS as closely as possible, including mount-boundary semantics.** Treat the kernel OverlayFS docs as normative. OverlayFS overlays directory trees, not the mount table: the merged hierarchy is its own standalone mount, not a bind mount over underlying mounts. Do not design root overlay logic that "sees through" or absorbs unrelated mounted filesystems. Mounted filesystems remain separate mount boundaries, and cross-mount operations must keep normal mount semantics (`EXDEV`, separate identity, separate read-only rules). If we want overlay behavior inside a mounted filesystem such as an S3-backed or host-backed mount, that mounted filesystem must implement the layered metadata semantics itself rather than relying on the parent/root overlay to compose across the mount boundary. +- **User-facing filesystem APIs should distinguish mounts from layers.** Mounts are separate mounted filesystems presented to the kernel VFS. Layers are overlay-building blocks used to construct a layered filesystem. Do not collapse those into one generic concept. A plain mounted `VirtualFileSystem` is not automatically a valid overlay layer. Overlay construction should consume explicit layer handles: one writable upper layer plus zero or more immutable lower snapshot layers. +- **Middle layers in a Docker-like stack should be frozen layers, not extra writable uppers.** Linux OverlayFS supports one writable upper per overlay mount. Additional stacked layers should be represented as immutable snapshot/materialized lower layers. They may share the same layer-handle interface as the upper layer, but their state must mark them frozen/read-only. Any live whiteouts, opaque markers, or copy-up bookkeeping belong only to the active writable upper; once a layer is sealed into a reusable lower snapshot, it must be materialized into an ordinary read-only tree. - **Never interfere with the user's filesystem or code.** Don't write config files, instruction files, or metadata into the user's working directory or project tree. Use dedicated OS paths (`/etc/`, `/var/`, etc.) or CLI flags instead. If an agent framework requires a file in the project directory (e.g., OpenCode's context paths), prefer absolute paths to OS-managed locations over creating files in cwd. - **Agent prompt injection must be non-destructive.** Each agent has its own mechanism for loading instructions (CLI flags, env vars, config files). When injecting OS instructions: preserve the agent's existing user-provided instructions (CLAUDE.md, AGENTS.md, etc.), append rather than replace, and always provide `skipOsInstructions` opt-out. User configuration is never clobbered — user env vars override ours via spread order. diff --git a/TODO.md b/TODO.md new file mode 100644 index 000000000..1ef573749 --- /dev/null +++ b/TODO.md @@ -0,0 +1,3 @@ +# TODO + +- Add OCI import/export support for overlay filesystem layers and snapshots after phase 1. The phase-1 API should only guarantee the bundled base filesystem artifact and the internal snapshot export/import format. diff --git a/docs/filesystem.mdx b/docs/filesystem.mdx new file mode 100644 index 000000000..7f1f17373 --- /dev/null +++ b/docs/filesystem.mdx @@ -0,0 +1,93 @@ +--- +title: Filesystem +description: Layered filesystem model for AgentOs virtual machines. +icon: "folder" +--- + +`AgentOs` gives each VM a Linux-like filesystem with a layered root, predictable defaults, and optional mounted filesystems for custom behavior. + +## Layering model + +Each VM starts with three filesystem concepts working together: + +- **Base filesystem** — the default root layout and OS files that every VM starts with. +- **Writable overlay** — the per-VM writable layer that records changes without mutating the base filesystem. +- **Mounted filesystems** — additional filesystems mounted at specific paths such as `/etc/agentos` or user-provided mount points. + +This means a fresh VM always starts from the same known root layout, while each VM still gets an isolated writable view of that root. + +## How the overlay works + +The default root filesystem uses copy-on-write overlay behavior. + +- **Reads** fall through to the base filesystem when a path has not been changed in the VM. +- **Writes** go to the writable overlay. If a file already exists in the base filesystem, it is copied into the writable layer before the change is applied. +- **Deletes** hide lower-layer entries from the VM's view, so a deleted base file stays deleted for that VM even though the shared base layer is unchanged. +- **New files and directories** are created only in the writable layer. + +This keeps startup fast, makes the default root deterministic, and lets each VM change its filesystem freely without affecting other VMs. + +## Default filesystem + +The default base filesystem is intended to mimic Alpine Linux as closely as practical for agent workloads. + +- It uses an Alpine-like directory layout such as `/bin`, `/etc`, `/home`, `/tmp`, `/usr`, and `/var`. +- It includes a default user home at `/home/user`. +- It preserves Alpine-like environment and shell defaults such as `PATH`, `PAGER`, prompt shape, and common `/etc` files. +- It keeps familiar symlinks and permissions where they matter for normal shell and tool behavior. + +AgentOs normalizes the small pieces that should be stable across VMs, such as the hostname being `agent-os`, while keeping the rest of the base layout as close as possible to Alpine. + +## Root filesystem config + +`AgentOs.create()` now accepts a `rootFilesystem` option that configures the root overlay directly. + +- `mode: "ephemeral"` keeps the default writable root behavior. +- `mode: "read-only"` keeps the merged root visible and mounts it read-only from the start of VM boot. +- `disableDefaultBaseLayer: true` removes the bundled base snapshot from the lower stack. +- `lowers` lets you add one or more snapshot exports as lower layers, ordered highest-precedence first. + +If you omit `rootFilesystem`, AgentOs still creates the default overlay-backed root with the bundled base snapshot as its deepest lower. + +If you set `disableDefaultBaseLayer: true` and do not provide any `lowers`, the VM starts from a minimal synthetic root that contains only the boot-critical POSIX directories and runtime command stubs. + +## Mounts and precedence + +Mounted filesystems replace a subtree at the mount point. + +- A mount at `/data` makes `/data` resolve to that mounted filesystem instead of the default root layer. +- Mounts are useful for read-only instruction files, host-backed directories, persistent stores, or custom virtual filesystems. +- The overlay-backed root still applies everywhere that is not covered by a mount. + +In practice, the path lookup order is: + +1. Mounted filesystem at the path, if one exists. +2. Writable overlay for the default root. +3. Base filesystem for the default root. + +For `mode: "read-only"`, step 2 is omitted entirely: the VM reads directly from the merged lower stack. + +Mount entries can also be declarative overlay mounts. Those use a `LayerStore` plus one or more snapshot handles to build an isolated overlay at the mount path, instead of requiring you to prebuild a `VirtualFileSystem` yourself. + +## Root snapshots + +`AgentOs.snapshotRootFilesystem()` exports the current visible root tree as a reusable snapshot descriptor. + +You can feed that snapshot back into a later VM through `rootFilesystem.lowers`, or import it into a `LayerStore` for declarative overlay mounts. + +## Why this design exists + +- It gives agents a familiar Linux-style root filesystem. +- It keeps the default VM state deterministic across runs. +- It avoids mutating the shared base filesystem at boot time. +- It makes it easy to add read-only or custom filesystems at specific paths. +- It keeps per-VM writes isolated and disposable. + +## Rebuilding the base filesystem + +The default root is generated in two steps: + +1. Capture a fresh Alpine snapshot. +2. Build the AgentOs base filesystem artifact from that snapshot, applying the small AgentOs-specific normalizations. + +The runtime itself consumes the built base filesystem artifact, not the Alpine snapshot directly. diff --git a/package.json b/package.json index 0a5946148..ffab9e33e 100644 --- a/package.json +++ b/package.json @@ -12,7 +12,8 @@ "test:watch": "npx turbo watch test", "check-types": "npx turbo check-types", "lint": "pnpm biome check .", - "fmt": "pnpm biome check --write --diagnostic-level=error ." + "fmt": "pnpm biome check --write --diagnostic-level=error .", + "shell": "pnpm --filter @rivet-dev/agent-os-shell shell" }, "devDependencies": { "@biomejs/biome": "^2.3", @@ -22,6 +23,7 @@ "@rivet-dev/agent-os-codex-agent": "workspace:*", "@rivet-dev/agent-os-core": "workspace:*", "@rivet-dev/agent-os-pi": "workspace:*", + "@rivet-dev/agent-os-pi-rust": "workspace:*", "@types/node": "^22.19.15", "turbo": "^2.5.6", "typescript": "^5.9.2" diff --git a/packages/core/fixtures/alpine-defaults.json b/packages/core/fixtures/alpine-defaults.json new file mode 100644 index 000000000..9111411e3 --- /dev/null +++ b/packages/core/fixtures/alpine-defaults.json @@ -0,0 +1,520 @@ +{ + "image": "alpine:3.22", + "createdAt": "2026-04-01T18:44:12.482Z", + "environment": { + "env": { + "CHARSET": "UTF-8", + "HOME": "/home/user", + "HOSTNAME": "1e3e3d3175cb", + "LANG": "C.UTF-8", + "LC_COLLATE": "C", + "LOGNAME": "user", + "PAGER": "less", + "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "SHELL": "/bin/sh", + "USER": "user" + }, + "prompt": "\\h:\\w\\$ " + }, + "filesystem": { + "entries": [ + { + "path": "/", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/bin", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/alpine-release", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "3.22.3\n" + }, + { + "path": "/etc/apk", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/busybox-paths.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/crontabs", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/group", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "root:x:0:root\nbin:x:1:root,bin,daemon\ndaemon:x:2:root,bin,daemon\nsys:x:3:root,bin\nadm:x:4:root,daemon\ntty:x:5:\ndisk:x:6:root\nlp:x:7:lp\nkmem:x:9:\nwheel:x:10:root\nfloppy:x:11:root\nmail:x:12:mail\nnews:x:13:news\nuucp:x:14:uucp\ncron:x:16:cron\naudio:x:18:\ncdrom:x:19:\ndialout:x:20:root\nftp:x:21:\nsshd:x:22:\ninput:x:23:\ntape:x:26:root\nvideo:x:27:root\nnetdev:x:28:\nkvm:x:34:kvm\ngames:x:35:\nshadow:x:42:\nwww-data:x:82:\nusers:x:100:games\nntp:x:123:\nabuild:x:300:\nutmp:x:406:\nping:x:999:\nnogroup:x:65533:\nnobody:x:65534:\nuser:x:1000:\n" + }, + { + "path": "/etc/hostname", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "1e3e3d3175cb\n" + }, + { + "path": "/etc/logrotate.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/modprobe.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/modules-load.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/network", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/nsswitch.conf", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "# musl itself does not support NSS, however some third-party DNS\n# implementations use the nsswitch.conf file to determine what\n# policy to follow.\n# Editing this file is not recommended.\nhosts: files dns\n" + }, + { + "path": "/etc/opt", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/os-release", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "../usr/lib/os-release" + }, + { + "path": "/etc/passwd", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "root:x:0:0:root:/root:/bin/sh\nbin:x:1:1:bin:/bin:/sbin/nologin\ndaemon:x:2:2:daemon:/sbin:/sbin/nologin\nlp:x:4:7:lp:/var/spool/lpd:/sbin/nologin\nsync:x:5:0:sync:/sbin:/bin/sync\nshutdown:x:6:0:shutdown:/sbin:/sbin/shutdown\nhalt:x:7:0:halt:/sbin:/sbin/halt\nmail:x:8:12:mail:/var/mail:/sbin/nologin\nnews:x:9:13:news:/usr/lib/news:/sbin/nologin\nuucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin\ncron:x:16:16:cron:/var/spool/cron:/sbin/nologin\nftp:x:21:21::/var/lib/ftp:/sbin/nologin\nsshd:x:22:22:sshd:/dev/null:/sbin/nologin\ngames:x:35:35:games:/usr/games:/sbin/nologin\nntp:x:123:123:NTP:/var/empty:/sbin/nologin\nguest:x:405:100:guest:/dev/null:/sbin/nologin\nnobody:x:65534:65534:nobody:/:/sbin/nologin\nuser:x:1000:1000::/home/user:/bin/sh\n" + }, + { + "path": "/etc/periodic", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/profile", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "export PATH=\"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"\n\nexport PAGER=less\numask 022\n\n# use nicer PS1 for bash and busybox ash\nif [ -n \"$BASH_VERSION\" -o \"$BB_ASH_VERSION\" ]; then\n\tPS1='\\h:\\w\\$ '\n# use nicer PS1 for zsh\nelif [ -n \"$ZSH_VERSION\" ]; then\n\tPS1='%m:%~%# '\n# set up fallback default PS1\nelse\n\t: \"${HOSTNAME:=$(hostname)}\"\n\tPS1='${HOSTNAME%%.*}:$PWD'\n\t[ \"$(id -u)\" -eq 0 ] && PS1=\"${PS1}# \" || PS1=\"${PS1}\\$ \"\nfi\n\nfor script in /etc/profile.d/*.sh ; do\n\tif [ -r \"$script\" ] ; then\n\t\t. \"$script\"\n\tfi\ndone\nunset script\n" + }, + { + "path": "/etc/profile.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/secfixes.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/shadow", + "type": "file", + "mode": "640", + "uid": 0, + "gid": 42, + "content": "root:*::0:::::\nbin:!::0:::::\ndaemon:!::0:::::\nlp:!::0:::::\nsync:!::0:::::\nshutdown:!::0:::::\nhalt:!::0:::::\nmail:!::0:::::\nnews:!::0:::::\nuucp:!::0:::::\ncron:!::0:::::\nftp:!::0:::::\nsshd:!::0:::::\ngames:!::0:::::\nntp:!::0:::::\nguest:!::0:::::\nnobody:!::0:::::\nuser:!:20544:0:99999:7:::\n" + }, + { + "path": "/etc/shells", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "# valid login shells\n/bin/sh\n/bin/ash\n" + }, + { + "path": "/etc/ssl", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/ssl1.1", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/sysctl.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/udhcpc", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/home", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/home/user", + "type": "directory", + "mode": "2755", + "uid": 1000, + "gid": 1000 + }, + { + "path": "/lib", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/lib/apk", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/lib/firmware", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/lib/modules-load.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/lib/sysctl.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/media", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/media/cdrom", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/media/floppy", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/media/usb", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/mnt", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/opt", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/root", + "type": "directory", + "mode": "700", + "uid": 0, + "gid": 0 + }, + { + "path": "/run", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/run/lock", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/sbin", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/srv", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/sys", + "type": "directory", + "mode": "555", + "uid": 0, + "gid": 0 + }, + { + "path": "/tmp", + "type": "directory", + "mode": "1777", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/bin", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/bin/env", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "/bin/busybox" + }, + { + "path": "/usr/lib", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/lib/os-release", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "NAME=\"Alpine Linux\"\nID=alpine\nVERSION_ID=3.22.3\nPRETTY_NAME=\"Alpine Linux v3.22\"\nHOME_URL=\"https://alpinelinux.org/\"\nBUG_REPORT_URL=\"https://gitlab.alpinelinux.org/alpine/aports/-/issues\"\n" + }, + { + "path": "/usr/local", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/sbin", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/share", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/cache", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/empty", + "type": "directory", + "mode": "555", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/lib", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/local", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/lock", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "../run/lock" + }, + { + "path": "/var/log", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/mail", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/opt", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/run", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "../run" + }, + { + "path": "/var/spool", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/spool/cron", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/spool/cron/crontabs", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "../../../etc/crontabs" + }, + { + "path": "/var/tmp", + "type": "directory", + "mode": "1777", + "uid": 0, + "gid": 0 + } + ] + } +} diff --git a/packages/core/fixtures/base-filesystem.json b/packages/core/fixtures/base-filesystem.json new file mode 100644 index 000000000..3e24bdbb8 --- /dev/null +++ b/packages/core/fixtures/base-filesystem.json @@ -0,0 +1,528 @@ +{ + "source": { + "snapshotPath": "alpine-defaults.json", + "image": "alpine:3.22", + "snapshotCreatedAt": "2026-04-01T18:44:12.482Z", + "builtAt": "2026-04-01T19:03:51.260Z", + "transforms": [ + "Normalize HOSTNAME to agent-os", + "Preserve the captured user-level environment and filesystem layout as the AgentOs base layer" + ] + }, + "environment": { + "env": { + "CHARSET": "UTF-8", + "HOME": "/home/user", + "HOSTNAME": "agent-os", + "LANG": "C.UTF-8", + "LC_COLLATE": "C", + "LOGNAME": "user", + "PAGER": "less", + "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "SHELL": "/bin/sh", + "USER": "user" + }, + "prompt": "\\h:\\w\\$ " + }, + "filesystem": { + "entries": [ + { + "path": "/", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/bin", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/alpine-release", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "3.22.3\n" + }, + { + "path": "/etc/apk", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/busybox-paths.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/crontabs", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/group", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "root:x:0:root\nbin:x:1:root,bin,daemon\ndaemon:x:2:root,bin,daemon\nsys:x:3:root,bin\nadm:x:4:root,daemon\ntty:x:5:\ndisk:x:6:root\nlp:x:7:lp\nkmem:x:9:\nwheel:x:10:root\nfloppy:x:11:root\nmail:x:12:mail\nnews:x:13:news\nuucp:x:14:uucp\ncron:x:16:cron\naudio:x:18:\ncdrom:x:19:\ndialout:x:20:root\nftp:x:21:\nsshd:x:22:\ninput:x:23:\ntape:x:26:root\nvideo:x:27:root\nnetdev:x:28:\nkvm:x:34:kvm\ngames:x:35:\nshadow:x:42:\nwww-data:x:82:\nusers:x:100:games\nntp:x:123:\nabuild:x:300:\nutmp:x:406:\nping:x:999:\nnogroup:x:65533:\nnobody:x:65534:\nuser:x:1000:\n" + }, + { + "path": "/etc/hostname", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "agent-os\n" + }, + { + "path": "/etc/logrotate.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/modprobe.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/modules-load.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/network", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/nsswitch.conf", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "# musl itself does not support NSS, however some third-party DNS\n# implementations use the nsswitch.conf file to determine what\n# policy to follow.\n# Editing this file is not recommended.\nhosts: files dns\n" + }, + { + "path": "/etc/opt", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/os-release", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "../usr/lib/os-release" + }, + { + "path": "/etc/passwd", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "root:x:0:0:root:/root:/bin/sh\nbin:x:1:1:bin:/bin:/sbin/nologin\ndaemon:x:2:2:daemon:/sbin:/sbin/nologin\nlp:x:4:7:lp:/var/spool/lpd:/sbin/nologin\nsync:x:5:0:sync:/sbin:/bin/sync\nshutdown:x:6:0:shutdown:/sbin:/sbin/shutdown\nhalt:x:7:0:halt:/sbin:/sbin/halt\nmail:x:8:12:mail:/var/mail:/sbin/nologin\nnews:x:9:13:news:/usr/lib/news:/sbin/nologin\nuucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin\ncron:x:16:16:cron:/var/spool/cron:/sbin/nologin\nftp:x:21:21::/var/lib/ftp:/sbin/nologin\nsshd:x:22:22:sshd:/dev/null:/sbin/nologin\ngames:x:35:35:games:/usr/games:/sbin/nologin\nntp:x:123:123:NTP:/var/empty:/sbin/nologin\nguest:x:405:100:guest:/dev/null:/sbin/nologin\nnobody:x:65534:65534:nobody:/:/sbin/nologin\nuser:x:1000:1000::/home/user:/bin/sh\n" + }, + { + "path": "/etc/periodic", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/profile", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "export PATH=\"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"\n\nexport PAGER=less\numask 022\n\n# use nicer PS1 for bash and busybox ash\nif [ -n \"$BASH_VERSION\" -o \"$BB_ASH_VERSION\" ]; then\n\tPS1='\\h:\\w\\$ '\n# use nicer PS1 for zsh\nelif [ -n \"$ZSH_VERSION\" ]; then\n\tPS1='%m:%~%# '\n# set up fallback default PS1\nelse\n\t: \"${HOSTNAME:=$(hostname)}\"\n\tPS1='${HOSTNAME%%.*}:$PWD'\n\t[ \"$(id -u)\" -eq 0 ] && PS1=\"${PS1}# \" || PS1=\"${PS1}\\$ \"\nfi\n\nfor script in /etc/profile.d/*.sh ; do\n\tif [ -r \"$script\" ] ; then\n\t\t. \"$script\"\n\tfi\ndone\nunset script\n" + }, + { + "path": "/etc/profile.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/secfixes.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/shadow", + "type": "file", + "mode": "640", + "uid": 0, + "gid": 42, + "content": "root:*::0:::::\nbin:!::0:::::\ndaemon:!::0:::::\nlp:!::0:::::\nsync:!::0:::::\nshutdown:!::0:::::\nhalt:!::0:::::\nmail:!::0:::::\nnews:!::0:::::\nuucp:!::0:::::\ncron:!::0:::::\nftp:!::0:::::\nsshd:!::0:::::\ngames:!::0:::::\nntp:!::0:::::\nguest:!::0:::::\nnobody:!::0:::::\nuser:!:20544:0:99999:7:::\n" + }, + { + "path": "/etc/shells", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "# valid login shells\n/bin/sh\n/bin/ash\n" + }, + { + "path": "/etc/ssl", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/ssl1.1", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/sysctl.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/etc/udhcpc", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/home", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/home/user", + "type": "directory", + "mode": "2755", + "uid": 1000, + "gid": 1000 + }, + { + "path": "/lib", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/lib/apk", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/lib/firmware", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/lib/modules-load.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/lib/sysctl.d", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/media", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/media/cdrom", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/media/floppy", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/media/usb", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/mnt", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/opt", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/root", + "type": "directory", + "mode": "700", + "uid": 0, + "gid": 0 + }, + { + "path": "/run", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/run/lock", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/sbin", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/srv", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/sys", + "type": "directory", + "mode": "555", + "uid": 0, + "gid": 0 + }, + { + "path": "/tmp", + "type": "directory", + "mode": "1777", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/bin", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/bin/env", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "/bin/busybox" + }, + { + "path": "/usr/lib", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/lib/os-release", + "type": "file", + "mode": "644", + "uid": 0, + "gid": 0, + "content": "NAME=\"Alpine Linux\"\nID=alpine\nVERSION_ID=3.22.3\nPRETTY_NAME=\"Alpine Linux v3.22\"\nHOME_URL=\"https://alpinelinux.org/\"\nBUG_REPORT_URL=\"https://gitlab.alpinelinux.org/alpine/aports/-/issues\"\n" + }, + { + "path": "/usr/local", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/sbin", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/usr/share", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/cache", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/empty", + "type": "directory", + "mode": "555", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/lib", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/local", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/lock", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "../run/lock" + }, + { + "path": "/var/log", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/mail", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/opt", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/run", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "../run" + }, + { + "path": "/var/spool", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/spool/cron", + "type": "directory", + "mode": "755", + "uid": 0, + "gid": 0 + }, + { + "path": "/var/spool/cron/crontabs", + "type": "symlink", + "mode": "777", + "uid": 0, + "gid": 0, + "target": "../../../etc/crontabs" + }, + { + "path": "/var/tmp", + "type": "directory", + "mode": "1777", + "uid": 0, + "gid": 0 + } + ] + } +} diff --git a/packages/core/package.json b/packages/core/package.json index 561de8e6a..28f661538 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -29,6 +29,8 @@ "scripts": { "check-types": "tsc --noEmit", "build": "tsc", + "build:base-filesystem": "node ./scripts/build-base-filesystem.mjs", + "snapshot:alpine-defaults": "node ./scripts/snapshot-alpine-defaults.mjs", "test": "vitest run" }, "dependencies": { diff --git a/packages/core/scripts/build-base-filesystem.mjs b/packages/core/scripts/build-base-filesystem.mjs new file mode 100644 index 000000000..af02e1b88 --- /dev/null +++ b/packages/core/scripts/build-base-filesystem.mjs @@ -0,0 +1,71 @@ +#!/usr/bin/env node + +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const DEFAULT_INPUT = fileURLToPath( + new URL("../fixtures/alpine-defaults.json", import.meta.url), +); +const DEFAULT_OUTPUT = fileURLToPath( + new URL("../fixtures/base-filesystem.json", import.meta.url), +); + +const BASE_HOSTNAME = "agent-os"; +const BASE_USER = "user"; +const BASE_HOME = `/home/${BASE_USER}`; + +function readJson(pathname) { + return JSON.parse(readFileSync(pathname, "utf-8")); +} + +function normalizeEntry(entry) { + if (entry.path === "/etc/hostname" && entry.type === "file") { + return { + ...entry, + content: `${BASE_HOSTNAME}\n`, + }; + } + + return entry; +} + +function buildBaseFilesystem(snapshot, inputPath) { + return { + source: { + snapshotPath: path.basename(inputPath), + image: snapshot.image, + snapshotCreatedAt: snapshot.createdAt, + builtAt: new Date().toISOString(), + transforms: [ + "Normalize HOSTNAME to agent-os", + "Preserve the captured user-level environment and filesystem layout as the AgentOs base layer", + ], + }, + environment: { + env: { + ...snapshot.environment.env, + HOME: BASE_HOME, + HOSTNAME: BASE_HOSTNAME, + LOGNAME: BASE_USER, + USER: BASE_USER, + }, + prompt: snapshot.environment.prompt, + }, + filesystem: { + entries: snapshot.filesystem.entries.map(normalizeEntry), + }, + }; +} + +function main() { + const [inputPath = DEFAULT_INPUT, outputPath = DEFAULT_OUTPUT] = process.argv.slice(2); + const snapshot = readJson(inputPath); + const baseFilesystem = buildBaseFilesystem(snapshot, inputPath); + + mkdirSync(path.dirname(outputPath), { recursive: true }); + writeFileSync(outputPath, `${JSON.stringify(baseFilesystem, null, 2)}\n`); + process.stdout.write(`Wrote ${outputPath}\n`); +} + +main(); diff --git a/packages/core/scripts/snapshot-alpine-defaults.mjs b/packages/core/scripts/snapshot-alpine-defaults.mjs new file mode 100644 index 000000000..4d5e85db0 --- /dev/null +++ b/packages/core/scripts/snapshot-alpine-defaults.mjs @@ -0,0 +1,236 @@ +#!/usr/bin/env node + +import { execFileSync } from "node:child_process"; +import { mkdirSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const DEFAULT_IMAGE = process.env.ALPINE_IMAGE ?? "alpine:3.22"; +const DEFAULT_OUTPUT = fileURLToPath( + new URL("../fixtures/alpine-defaults.json", import.meta.url), +); + +const DEFAULT_ENV_KEYS = [ + "CHARSET", + "HOME", + "HOSTNAME", + "LANG", + "LC_COLLATE", + "LOGNAME", + "PAGER", + "PATH", + "SHELL", + "USER", +]; + +const TEXT_FILE_PATHS = [ + "/etc/alpine-release", + "/etc/group", + "/etc/hostname", + "/etc/nsswitch.conf", + "/etc/passwd", + "/etc/profile", + "/etc/shadow", + "/etc/shells", + "/usr/lib/os-release", +]; + +const METADATA_ONLY_FILE_PATHS = [ + "/usr/bin/env", +]; + +const SYMLINK_PATHS = [ + "/etc/os-release", + "/var/lock", + "/var/run", + "/var/spool/cron/crontabs", +]; + +function addParentDirectories(paths) { + for (const entry of [...paths]) { + let current = path.posix.dirname(entry); + while (current !== "." && current !== "/") { + paths.add(current); + current = path.posix.dirname(current); + } + } +} + +function runDocker(args, options = {}) { + return execFileSync("docker", args, { + encoding: "utf-8", + stdio: ["ignore", "pipe", "pipe"], + ...options, + }); +} + +function dockerExec(containerId, args) { + return runDocker(["exec", containerId, ...args]); +} + +function parseEnv(stdout) { + const result = {}; + + for (const line of stdout.split("\n")) { + if (!line.trim()) { + continue; + } + + const separator = line.indexOf("="); + if (separator === -1) { + continue; + } + + const key = line.slice(0, separator); + const value = line.slice(separator + 1); + result[key] = value; + } + + return result; +} + +function mapFileType(type) { + switch (type) { + case "directory": + return "directory"; + case "regular file": + return "file"; + case "symbolic link": + return "symlink"; + default: + throw new Error(`Unsupported file type: ${type}`); + } +} + +function shouldIncludePath(path) { + if (path === "/dev" || path.startsWith("/dev/")) { + return false; + } + if (path === "/proc" || path.startsWith("/proc/")) { + return false; + } + if (path.startsWith("/sys/")) { + return false; + } + if (path === "/etc/mtab") { + return false; + } + return true; +} + +function collectPaths(containerId) { + const discovered = dockerExec(containerId, [ + "sh", + "-lc", + "find / -maxdepth 2 -type d | sort", + ]); + + const paths = new Set( + discovered + .split("\n") + .map((path) => path.trim()) + .filter(Boolean) + .filter(shouldIncludePath), + ); + + for (const path of TEXT_FILE_PATHS) { + paths.add(path); + } + for (const path of METADATA_ONLY_FILE_PATHS) { + paths.add(path); + } + for (const path of SYMLINK_PATHS) { + paths.add(path); + } + paths.add("/"); + addParentDirectories(paths); + + return [...paths].sort((a, b) => a.localeCompare(b)); +} + +function readEntry(containerId, path) { + const statOutput = dockerExec(containerId, [ + "sh", + "-lc", + `stat -c '%F\t%a\t%u\t%g' '${path}'`, + ]).trim(); + const [rawType, mode, uid, gid] = statOutput.split("\t"); + const entry = { + path, + type: mapFileType(rawType), + mode, + uid: Number(uid), + gid: Number(gid), + }; + + if (entry.type === "symlink") { + entry.target = dockerExec(containerId, ["readlink", path]).trim(); + } + + if (entry.type === "file" && TEXT_FILE_PATHS.includes(path)) { + entry.content = dockerExec(containerId, ["cat", path]); + } + + return entry; +} + +function extractPrompt(profileContent) { + const match = profileContent.match(/PS1='([^']+)'/); + if (!match) { + throw new Error("Unable to extract PS1 from /etc/profile"); + } + return match[1]; +} + +function main() { + const outputPath = process.argv[2] ?? DEFAULT_OUTPUT; + const containerId = runDocker([ + "run", + "--detach", + "--rm", + DEFAULT_IMAGE, + "sh", + "-lc", + "adduser -D user >/dev/null 2>&1 && sleep infinity", + ]).trim(); + + try { + const rawEnv = parseEnv( + dockerExec(containerId, ["sh", "-lc", "su user -c env"]), + ); + const env = Object.fromEntries( + DEFAULT_ENV_KEYS + .filter((key) => rawEnv[key] !== undefined) + .map((key) => [key, rawEnv[key]]), + ); + + const profileContent = dockerExec(containerId, ["cat", "/etc/profile"]); + const entries = collectPaths(containerId).map((path) => + readEntry(containerId, path), + ); + + const snapshot = { + image: DEFAULT_IMAGE, + createdAt: new Date().toISOString(), + environment: { + env, + prompt: extractPrompt(profileContent), + }, + filesystem: { + entries, + }, + }; + + mkdirSync(path.dirname(outputPath), { recursive: true }); + writeFileSync(outputPath, `${JSON.stringify(snapshot, null, 2)}\n`); + process.stdout.write(`Wrote ${outputPath}\n`); + } finally { + try { + runDocker(["rm", "--force", containerId]); + } catch { + // Container may already be gone. + } + } +} + +main(); diff --git a/packages/core/src/agent-os.ts b/packages/core/src/agent-os.ts index 259213625..be070b321 100644 --- a/packages/core/src/agent-os.ts +++ b/packages/core/src/agent-os.ts @@ -1,14 +1,14 @@ import { spawn as spawnChildProcess } from "node:child_process"; import { mkdtempSync, + readdirSync, readFileSync, rmSync, + statSync, writeFileSync, } from "node:fs"; import { tmpdir } from "node:os"; import { - basename, - dirname, join, posix as posixPath, relative as relativeHostPath, @@ -19,7 +19,6 @@ import { allowAll, createInMemoryFileSystem, createKernel, - type FsMount, type Kernel, type KernelExecOptions, type KernelExecResult, @@ -43,6 +42,7 @@ import { generateMasterShim, generateToolkitShim, } from "./host-tools-shims.js"; +import { createHostBinRuntime } from "./host-bin-runtime.js"; /** Process tree node: extends kernel ProcessInfo with child references. */ export interface ProcessTreeNode extends KernelProcessInfo { @@ -100,6 +100,24 @@ import { import { createPythonRuntime } from "@rivet-dev/agent-os-python"; import { createWasmVmRuntime } from "@rivet-dev/agent-os-posix"; import { AcpClient } from "./acp-client.js"; +import { + createBootstrapAwareFilesystem, + getBaseEnvironment, + getBaseFilesystemEntries, +} from "./base-filesystem.js"; +import { + snapshotVirtualFilesystem, + type FilesystemEntry, +} from "./filesystem-snapshot.js"; +import { + createDefaultRootLowerInput, + createInMemoryLayerStore, + createSnapshotExport, + type LayerStore, + type OverlayFilesystemMode, + type RootSnapshotExport, + type SnapshotLayerHandle, +} from "./layers.js"; import { AGENT_CONFIGS, type AgentConfig, type AgentType } from "./agents.js"; import { getHostDirBackendMeta } from "./backends/host-dir-backend.js"; import { @@ -149,8 +167,19 @@ interface AcpTerminalState { outputByteLimit: number; } +export type RootLowerInput = + | { kind: "bundled-base-filesystem" } + | RootSnapshotExport; + +export interface RootFilesystemConfig { + type?: "overlay"; + mode?: OverlayFilesystemMode; + disableDefaultBaseLayer?: boolean; + lowers?: RootLowerInput[]; +} + /** Configuration for mounting a filesystem driver at a path. */ -export interface MountConfig { +export interface PlainMountConfig { /** Path inside the VM to mount at. */ path: string; /** The filesystem driver to mount. */ @@ -159,6 +188,18 @@ export interface MountConfig { readOnly?: boolean; } +export interface OverlayMountConfig { + path: string; + filesystem: { + type: "overlay"; + store: LayerStore; + mode?: OverlayFilesystemMode; + lowers: SnapshotLayerHandle[]; + }; +} + +export type MountConfig = PlainMountConfig | OverlayMountConfig; + export interface AgentOsOptions { /** * Software to install in the VM. Each entry provides agents, tools, @@ -176,6 +217,8 @@ export interface AgentOsOptions { * Defaults to process.cwd(). */ moduleAccessCwd?: string; + /** Root filesystem configuration. Defaults to an overlay with the bundled base snapshot as its deepest lower. */ + rootFilesystem?: RootFilesystemConfig; /** Filesystems to mount at boot time. */ mounts?: MountConfig[]; /** Additional instructions appended to the base OS instructions written to /etc/agentos/instructions.md. */ @@ -240,6 +283,284 @@ export interface SpawnedProcessInfo { exitCode: number | null; } +function isOverlayMountConfig(config: MountConfig): config is OverlayMountConfig { + return "filesystem" in config; +} + +const KERNEL_POSIX_BOOTSTRAP_DIRS = [ + "/dev", + "/proc", + "/tmp", + "/bin", + "/lib", + "/sbin", + "/boot", + "/etc", + "/root", + "/run", + "/srv", + "/sys", + "/opt", + "/mnt", + "/media", + "/home", + "/usr", + "/usr/bin", + "/usr/games", + "/usr/include", + "/usr/lib", + "/usr/libexec", + "/usr/man", + "/usr/local", + "/usr/local/bin", + "/usr/sbin", + "/usr/share", + "/usr/share/man", + "/var", + "/var/cache", + "/var/empty", + "/var/lib", + "/var/lock", + "/var/log", + "/var/run", + "/var/spool", + "/var/tmp", + "/etc/agentos", +] as const; + +const NODE_RUNTIME_BOOTSTRAP_COMMANDS = ["node", "npm", "npx"] as const; +const PYTHON_RUNTIME_BOOTSTRAP_COMMANDS = ["python", "python3", "pip"] as const; +const KERNEL_COMMAND_STUB = "#!/bin/sh\n# kernel command stub\n"; + +function isWasmBinaryFile(path: string): boolean { + try { + const header = readFileSync(path); + return ( + header.length >= 4 + && header[0] === 0x00 + && header[1] === 0x61 + && header[2] === 0x73 + && header[3] === 0x6d + ); + } catch { + return false; + } +} + +function collectBootstrapWasmCommands(commandDirs: string[]): string[] { + const commands: string[] = []; + const seen = new Set(); + + for (const dir of commandDirs) { + let entries: string[]; + try { + entries = readdirSync(dir).sort((a, b) => a.localeCompare(b)); + } catch { + continue; + } + + for (const entry of entries) { + if (entry.startsWith(".")) { + continue; + } + + const fullPath = join(dir, entry); + try { + if (statSync(fullPath).isDirectory()) { + continue; + } + } catch { + continue; + } + + if (!isWasmBinaryFile(fullPath) || seen.has(entry)) { + continue; + } + + seen.add(entry); + commands.push(entry); + } + } + + return commands; +} + +function collectConfiguredLowerPaths(config?: RootFilesystemConfig): Set { + const paths = new Set(); + + for (const lower of config?.lowers ?? []) { + if (lower.kind !== "snapshot-export") { + continue; + } + for (const entry of lower.source.filesystem.entries) { + paths.add(entry.path); + } + } + + if (!config?.disableDefaultBaseLayer) { + for (const entry of getBaseFilesystemEntries()) { + paths.add(entry.path); + } + } + + return paths; +} + +function createKernelBootstrapLower( + config: RootFilesystemConfig | undefined, + commandNames: string[], +): RootSnapshotExport | null { + const existingPaths = collectConfiguredLowerPaths(config); + const entries: FilesystemEntry[] = [ + { + path: "/", + type: "directory", + mode: "755", + uid: 0, + gid: 0, + }, + ]; + + for (const dir of KERNEL_POSIX_BOOTSTRAP_DIRS) { + if (existingPaths.has(dir)) { + continue; + } + entries.push({ + path: dir, + type: "directory", + mode: "755", + uid: 0, + gid: 0, + }); + } + + if (!existingPaths.has("/usr/bin/env")) { + entries.push({ + path: "/usr/bin/env", + type: "file", + mode: "644", + uid: 0, + gid: 0, + content: "AA==", + encoding: "base64", + }); + } + + const uniqueCommands = [...new Set(commandNames)].sort((a, b) => a.localeCompare(b)); + for (const command of uniqueCommands) { + const stubPath = `/bin/${command}`; + if (existingPaths.has(stubPath)) { + continue; + } + entries.push({ + path: stubPath, + type: "file", + mode: "755", + uid: 0, + gid: 0, + content: KERNEL_COMMAND_STUB, + encoding: "utf8", + }); + } + + return entries.length > 1 ? createSnapshotExport(entries) : null; +} + +async function createRootFilesystem( + config?: RootFilesystemConfig, + bootstrapLower?: RootSnapshotExport | null, +): Promise<{ + filesystem: VirtualFileSystem; + finishKernelBootstrap: () => void; + rootView: VirtualFileSystem; +}> { + const rootStore = createInMemoryLayerStore(); + const normalizedConfig = config ?? {}; + const lowerInputs = normalizedConfig.lowers + ? [...normalizedConfig.lowers] + : []; + + if (bootstrapLower) { + lowerInputs.push(bootstrapLower); + } + + if (!normalizedConfig.disableDefaultBaseLayer) { + lowerInputs.push({ kind: "bundled-base-filesystem" }); + } + + const lowers = await Promise.all( + lowerInputs.map((lower) => rootStore.importSnapshot( + lower.kind === "bundled-base-filesystem" + ? createDefaultRootLowerInput() + : lower, + )), + ); + + const rootView = normalizedConfig.mode === "read-only" + ? rootStore.createOverlayFilesystem({ + mode: "read-only", + lowers, + }) + : rootStore.createOverlayFilesystem({ + upper: await rootStore.createWritableLayer(), + lowers, + }); + + if (normalizedConfig.mode === "read-only") { + return { + filesystem: rootView, + finishKernelBootstrap: () => {}, + rootView, + }; + } + + const { filesystem, finishKernelBootstrap } = createBootstrapAwareFilesystem( + rootView, + rootView, + ); + + return { + filesystem, + finishKernelBootstrap, + rootView, + }; +} + +async function resolveMounts( + mounts?: MountConfig[], +): Promise> { + if (!mounts) { + return []; + } + + return Promise.all(mounts.map(async (mount) => { + if (!isOverlayMountConfig(mount)) { + return { + path: mount.path, + fs: mount.driver, + readOnly: mount.readOnly, + }; + } + + const mode = mount.filesystem.mode ?? "ephemeral"; + const fs = mode === "read-only" + ? mount.filesystem.store.createOverlayFilesystem({ + mode: "read-only", + lowers: mount.filesystem.lowers, + }) + : mount.filesystem.store.createOverlayFilesystem({ + upper: await mount.filesystem.store.createWritableLayer(), + lowers: mount.filesystem.lowers, + }); + + return { + path: mount.path, + fs, + readOnly: mode === "read-only", + }; + })); +} + export class AgentOs { readonly kernel: Kernel; private _sessions = new Map(); @@ -273,6 +594,7 @@ export class AgentOs { private _acpTerminals = new Map(); private _acpTerminalCounter = 0; private _env: Record; + private _rootFilesystem: VirtualFileSystem; private constructor( kernel: Kernel, @@ -281,6 +603,7 @@ export class AgentOs { softwareAgentConfigs: Map, hostMounts: HostMountInfo[], env: Record, + rootFilesystem: VirtualFileSystem, ) { this.kernel = kernel; this._moduleAccessCwd = moduleAccessCwd; @@ -288,23 +611,35 @@ export class AgentOs { this._softwareAgentConfigs = softwareAgentConfigs; this._hostMounts = hostMounts; this._env = env; + this._rootFilesystem = rootFilesystem; } static async create(options?: AgentOsOptions): Promise { - const filesystem = createInMemoryFileSystem(); + // Process software descriptors first so the root lower can include the + // exact command stubs Secure Exec will register during boot. + const processed = processSoftware(options?.software ?? []); + const bootstrapLower = createKernelBootstrapLower( + options?.rootFilesystem, + [ + ...collectBootstrapWasmCommands(processed.commandDirs), + ...NODE_RUNTIME_BOOTSTRAP_COMMANDS, + ...PYTHON_RUNTIME_BOOTSTRAP_COMMANDS, + ], + ); + const { + filesystem, + finishKernelBootstrap, + rootView, + } = await createRootFilesystem(options?.rootFilesystem, bootstrapLower); const hostNetworkAdapter = createNodeHostNetworkAdapter(); const moduleAccessCwd = options?.moduleAccessCwd ?? process.cwd(); - // Process software descriptors to collect WASM dirs, module roots, and agent configs. - const processed = processSoftware(options?.software ?? []); - - const mounts = options?.mounts?.map((m) => ({ - path: m.path, - fs: m.driver, - readOnly: m.readOnly, - })); + const mounts = await resolveMounts(options?.mounts); const hostMounts = (options?.mounts ?? []) .flatMap((mount) => { + if (isOverlayMountConfig(mount)) { + return []; + } const meta = getHostDirBackendMeta(mount.driver); if (!meta) { return []; @@ -333,11 +668,7 @@ export class AgentOs { ...(toolsServer ? [toolsServer.port] : []), ]; - const env: Record = { - HOME: "/home/user", - USER: "user", - PATH: "/usr/local/bin:/usr/bin:/bin", - }; + const env: Record = getBaseEnvironment(); if (toolsServer) { env.AGENTOS_TOOLS_PORT = String(toolsServer.port); } @@ -383,7 +714,11 @@ export class AgentOs { : undefined, }), ); + if (processed.toolBinaries.size > 0) { + await kernel.mount(createHostBinRuntime(processed.toolBinaries)); + } await kernel.mount(createPythonRuntime()); + finishKernelBootstrap(); const vm = new AgentOs( kernel, @@ -392,6 +727,7 @@ export class AgentOs { processed.agentConfigs, hostMounts, env, + rootView, ); vm._toolsServer = toolsServer; vm._toolKits = toolKits ?? []; @@ -534,11 +870,55 @@ export class AgentOs { return entry.proc.wait(); } + private _assertSafeAbsolutePath(path: string): void { + if (!path.startsWith("/")) { + throw new Error(`Path must be absolute: ${path}`); + } + if (posixPath.normalize(path) !== path) { + throw new Error(`Path must be normalized: ${path}`); + } + } + + private _vfs(): VirtualFileSystem { + return (this.kernel as unknown as { vfs: VirtualFileSystem }).vfs; + } + + private async _copyPath(from: string, to: string): Promise { + const stat = await this._vfs().lstat(from); + if (stat.isSymbolicLink) { + const target = await this._vfs().readlink(from); + await this._vfs().symlink(target, to); + return; + } + if (stat.isDirectory) { + await this._mkdirp(posixPath.dirname(to)); + if (!(await this.kernel.exists(to))) { + await this.kernel.mkdir(to); + } + await this._vfs().chmod(to, stat.mode); + await this._vfs().chown(to, stat.uid, stat.gid); + const entries = await this.kernel.readdir(from); + for (const entry of entries) { + if (entry === "." || entry === "..") continue; + const fromPath = from === "/" ? `/${entry}` : `${from}/${entry}`; + const toPath = to === "/" ? `/${entry}` : `${to}/${entry}`; + await this._copyPath(fromPath, toPath); + } + return; + } + const content = await this.kernel.readFile(from); + await this.writeFile(to, content); + await this._vfs().chmod(to, stat.mode); + await this._vfs().chown(to, stat.uid, stat.gid); + } + async readFile(path: string): Promise { + this._assertSafeAbsolutePath(path); return this.kernel.readFile(path); } async writeFile(path: string, content: string | Uint8Array): Promise { + this._assertSafeAbsolutePath(path); return this.kernel.writeFile(path, content); } @@ -546,6 +926,7 @@ export class AgentOs { const results: BatchWriteResult[] = []; for (const entry of entries) { try { + this._assertSafeAbsolutePath(entry.path); // Create parent directories as needed const parentDir = entry.path.substring( 0, @@ -571,6 +952,7 @@ export class AgentOs { const results: BatchReadResult[] = []; for (const path of paths) { try { + this._assertSafeAbsolutePath(path); const content = await this.kernel.readFile(path); results.push({ path, content }); } catch (err: unknown) { @@ -586,6 +968,7 @@ export class AgentOs { /** Recursively create directories (mkdir -p). */ private async _mkdirp(path: string): Promise { + this._assertSafeAbsolutePath(path); const parts = path.split("/").filter(Boolean); let current = ""; for (const part of parts) { @@ -597,10 +980,12 @@ export class AgentOs { } async mkdir(path: string): Promise { + this._assertSafeAbsolutePath(path); return this.kernel.mkdir(path); } async readdir(path: string): Promise { + this._assertSafeAbsolutePath(path); return this.kernel.readdir(path); } @@ -608,6 +993,7 @@ export class AgentOs { path: string, options?: ReaddirRecursiveOptions, ): Promise { + this._assertSafeAbsolutePath(path); const maxDepth = options?.maxDepth; const exclude = options?.exclude ? new Set(options.exclude) : undefined; const results: DirEntry[] = []; @@ -658,29 +1044,47 @@ export class AgentOs { } async stat(path: string): Promise { + this._assertSafeAbsolutePath(path); return this.kernel.stat(path); } async exists(path: string): Promise { + this._assertSafeAbsolutePath(path); return this.kernel.exists(path); } + async snapshotRootFilesystem(): Promise { + return createSnapshotExport( + await snapshotVirtualFilesystem(this._rootFilesystem), + ); + } + mountFs(path: string, driver: VirtualFileSystem, options?: { readOnly?: boolean }): void { + this._assertSafeAbsolutePath(path); this.kernel.mountFs(path, driver, { readOnly: options?.readOnly }); } unmountFs(path: string): void { + this._assertSafeAbsolutePath(path); this.kernel.unmountFs(path); } async move(from: string, to: string): Promise { - return this.kernel.rename(from, to); + this._assertSafeAbsolutePath(from); + this._assertSafeAbsolutePath(to); + const sourceStat = await this._vfs().lstat(from); + if (!sourceStat.isDirectory || sourceStat.isSymbolicLink) { + return this.kernel.rename(from, to); + } + await this._copyPath(from, to); + await this.delete(from, { recursive: true }); } async delete( path: string, options?: { recursive?: boolean }, ): Promise { + this._assertSafeAbsolutePath(path); const s = await this.kernel.stat(path); if (s.isDirectory) { if (options?.recursive) { diff --git a/packages/core/src/backends/overlay-backend.ts b/packages/core/src/backends/overlay-backend.ts index 81c1d4736..0ea5a8c63 100644 --- a/packages/core/src/backends/overlay-backend.ts +++ b/packages/core/src/backends/overlay-backend.ts @@ -1,10 +1,9 @@ /** * Overlay (copy-on-write) filesystem backend. * - * Layers a writable upper filesystem over a read-only lower filesystem. - * Reads check the upper first, then fall through to the lower. - * Writes always go to the upper. Deletes record a "whiteout" in the upper - * so that the file appears deleted even if it exists in the lower. + * Layers an optional writable upper filesystem over zero or more lower + * filesystems. Reads resolve from highest precedence to lowest. Writes + * go to the writable upper only, with copy-up and whiteout behavior. */ import * as posixPath from "node:path/posix"; @@ -17,366 +16,587 @@ import { } from "@secure-exec/core"; export interface OverlayBackendOptions { - /** Read-only base layer. Never written to. */ - lower: VirtualFileSystem; - /** Writable upper layer. Defaults to a fresh InMemoryFileSystem. */ + /** Legacy single lower layer. */ + lower?: VirtualFileSystem; + /** Lower layers ordered highest-precedence first. */ + lowers?: VirtualFileSystem[]; + /** Writable upper layer. Defaults to a fresh in-memory filesystem in ephemeral mode. */ upper?: VirtualFileSystem; + /** Overlay mode. Defaults to ephemeral. */ + mode?: "ephemeral" | "read-only"; } -/** - * Create a copy-on-write overlay filesystem. - * Reads fall through from upper to lower. Writes go to upper only. - * Deletes record whiteout markers so files in lower appear removed. - */ export function createOverlayBackend( options: OverlayBackendOptions, ): VirtualFileSystem { - const lower = options.lower; - const upper = options.upper ?? createInMemoryFileSystem(); + const mode = options.mode ?? "ephemeral"; + if (mode === "read-only" && options.upper) { + throw new Error("Read-only overlays cannot accept a writable upper layer"); + } - // Whiteout set: paths that have been "deleted" in the overlay. - // If a path is in this set, it should not be visible even if it exists in lower. + const configuredLowers = options.lowers + ?? (options.lower ? [options.lower] : []); + const lowers = configuredLowers.length > 0 + ? configuredLowers + : [createInMemoryFileSystem()]; + const upper = mode === "read-only" + ? null + : options.upper ?? createInMemoryFileSystem(); const whiteouts = new Set(); - function normPath(p: string): string { - return posixPath.normalize(p); + function normPath(path: string): string { + return posixPath.normalize(path); + } + + function isWhitedOut(path: string): boolean { + return whiteouts.has(normPath(path)); + } + + function addWhiteout(path: string): void { + whiteouts.add(normPath(path)); } - function isWhitedOut(p: string): boolean { - return whiteouts.has(normPath(p)); + function removeWhiteout(path: string): void { + whiteouts.delete(normPath(path)); } - function addWhiteout(p: string): void { - whiteouts.add(normPath(p)); + function throwReadOnly(): never { + throw new KernelError("EROFS", "read-only file system"); } - function removeWhiteout(p: string): void { - whiteouts.delete(normPath(p)); + async function existsInFilesystem( + filesystem: VirtualFileSystem, + path: string, + ): Promise { + return hasEntryInFilesystem(filesystem, path); } - /** Check if path exists in upper layer. */ - async function existsInUpper(p: string): Promise { + async function hasEntryInFilesystem( + filesystem: VirtualFileSystem, + path: string, + ): Promise { try { - return await upper.exists(p); + if (path === "/") { + await filesystem.stat(path); + } else { + await filesystem.lstat(path); + } + return true; } catch { return false; } } + async function existsInUpper(path: string): Promise { + if (!upper) { + return false; + } + return existsInFilesystem(upper, path); + } + + async function hasEntryInUpper(path: string): Promise { + if (!upper) { + return false; + } + return hasEntryInFilesystem(upper, path); + } + + async function findLowerByExists( + path: string, + ): Promise { + for (const lower of lowers) { + if (await existsInFilesystem(lower, path)) { + return lower; + } + } + return null; + } + + async function findLowerByEntry( + path: string, + ): Promise<{ filesystem: VirtualFileSystem; stat: VirtualStat } | null> { + for (const lower of lowers) { + try { + return { + filesystem: lower, + stat: path === "/" ? await lower.stat(path) : await lower.lstat(path), + }; + } catch { + // Try the next lower layer. + } + } + return null; + } + + async function mergedLstat(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + if (await hasEntryInUpper(path)) { + return upper!.lstat(path); + } + const lower = await findLowerByEntry(path); + if (!lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + return lower.stat; + } + + async function ensureAncestorDirectoriesInUpper(path: string): Promise { + if (!upper) { + throwReadOnly(); + } + + const normalized = normPath(path); + const parts = normalized.split("/").filter(Boolean); + let current = ""; + + for (let index = 0; index < parts.length - 1; index++) { + current += `/${parts[index]}`; + if (await existsInUpper(current)) { + continue; + } + + const lower = await findLowerByExists(current); + if (lower) { + const stat = await lower.stat(current); + if (!stat.isDirectory) { + throw new KernelError("ENOTDIR", `not a directory: ${current}`); + } + await upper.mkdir(current); + await upper.chmod(current, stat.mode); + await upper.chown(current, stat.uid, stat.gid); + continue; + } + + await upper.mkdir(current); + } + } + + async function copyUpPath(path: string): Promise { + if (!upper) { + throwReadOnly(); + } + if (await hasEntryInUpper(path)) { + return; + } + + await ensureAncestorDirectoriesInUpper(path); + + const lower = await findLowerByEntry(path); + if (!lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + + if (lower.stat.isSymbolicLink) { + const target = await lower.filesystem.readlink(path); + await upper.symlink(target, path); + return; + } + + if (lower.stat.isDirectory) { + await upper.mkdir(path); + await upper.chmod(path, lower.stat.mode); + await upper.chown(path, lower.stat.uid, lower.stat.gid); + return; + } + + const data = await lower.filesystem.readFile(path); + await upper.writeFile(path, data); + await upper.chmod(path, lower.stat.mode); + await upper.chown(path, lower.stat.uid, lower.stat.gid); + } + + async function pathExistsInMergedView(path: string): Promise { + if (isWhitedOut(path)) { + return false; + } + if (await hasEntryInUpper(path)) { + return true; + } + return (await findLowerByEntry(path)) !== null; + } + const backend: VirtualFileSystem = { - async readFile(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async readFile(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - if (await existsInUpper(p)) { - return upper.readFile(p); + if (await existsInUpper(path)) { + return upper!.readFile(path); } - return lower.readFile(p); + const lower = await findLowerByExists(path); + if (!lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + return lower.readFile(path); }, - async readTextFile(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async readTextFile(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + if (await existsInUpper(path)) { + return upper!.readTextFile(path); } - if (await existsInUpper(p)) { - return upper.readTextFile(p); + const lower = await findLowerByExists(path); + if (!lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - return lower.readTextFile(p); + return lower.readTextFile(path); }, - async readDir(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such directory: ${p}`); + async readDir(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such directory: ${path}`); } + let directoryExists = false; const entries = new Set(); - // Collect from lower first (if directory exists) - try { - const lowerEntries = await lower.readDir(p); - for (const e of lowerEntries) { - if (e === "." || e === "..") continue; - const childPath = posixPath.join(normPath(p), e); - if (!isWhitedOut(childPath)) { - entries.add(e); + for (let index = lowers.length - 1; index >= 0; index--) { + try { + const lowerEntries = await lowers[index].readDir(path); + directoryExists = true; + for (const entry of lowerEntries) { + if (entry === "." || entry === "..") continue; + const childPath = posixPath.join(normPath(path), entry); + if (!isWhitedOut(childPath)) { + entries.add(entry); + } } + } catch { + // This lower does not contribute a directory here. } - } catch { - // Lower may not have this directory — that's fine } - // Overlay upper entries - try { - const upperEntries = await upper.readDir(p); - for (const e of upperEntries) { - if (e === "." || e === "..") continue; - entries.add(e); + if (upper) { + try { + const upperEntries = await upper.readDir(path); + directoryExists = true; + for (const entry of upperEntries) { + if (entry === "." || entry === "..") continue; + entries.add(entry); + } + } catch { + // No upper directory at this path. } - } catch { - // Upper may not have this directory either } - // If neither layer had the directory, throw - if (entries.size === 0) { - // Verify at least one layer has it - const lowerExists = await lower.exists(p).catch(() => false); - const upperExists = await upper.exists(p).catch(() => false); - if (!lowerExists && !upperExists) { - throw new KernelError("ENOENT", `no such directory: ${p}`); - } + if (!directoryExists) { + throw new KernelError("ENOENT", `no such directory: ${path}`); } return [...entries]; }, - async readDirWithTypes(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such directory: ${p}`); + async readDirWithTypes(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such directory: ${path}`); } + let directoryExists = false; const entriesByName = new Map(); - // Lower first - try { - const lowerEntries = await lower.readDirWithTypes(p); - for (const e of lowerEntries) { - if (e.name === "." || e.name === "..") continue; - const childPath = posixPath.join(normPath(p), e.name); - if (!isWhitedOut(childPath)) { - entriesByName.set(e.name, e); + for (let index = lowers.length - 1; index >= 0; index--) { + try { + const lowerEntries = await lowers[index].readDirWithTypes(path); + directoryExists = true; + for (const entry of lowerEntries) { + if (entry.name === "." || entry.name === "..") continue; + const childPath = posixPath.join(normPath(path), entry.name); + if (!isWhitedOut(childPath)) { + entriesByName.set(entry.name, entry); + } } + } catch { + // This lower does not contribute a directory here. } - } catch { - // Lower may not have this directory } - // Upper overwrites - try { - const upperEntries = await upper.readDirWithTypes(p); - for (const e of upperEntries) { - if (e.name === "." || e.name === "..") continue; - entriesByName.set(e.name, e); + if (upper) { + try { + const upperEntries = await upper.readDirWithTypes(path); + directoryExists = true; + for (const entry of upperEntries) { + if (entry.name === "." || entry.name === "..") continue; + entriesByName.set(entry.name, entry); + } + } catch { + // No upper directory at this path. } - } catch { - // Upper may not have this directory } - if (entriesByName.size === 0) { - const lowerExists = await lower.exists(p).catch(() => false); - const upperExists = await upper.exists(p).catch(() => false); - if (!lowerExists && !upperExists) { - throw new KernelError("ENOENT", `no such directory: ${p}`); - } + if (!directoryExists) { + throw new KernelError("ENOENT", `no such directory: ${path}`); } return [...entriesByName.values()]; }, async writeFile( - p: string, + path: string, content: string | Uint8Array, ): Promise { - // Writing removes any whiteout for this path - removeWhiteout(p); - // Ensure parent directory exists in upper - const parent = posixPath.dirname(p); - if (parent !== p) { - try { - await upper.mkdir(parent, { recursive: true }); - } catch { - // May already exist - } + if (!upper) { + throwReadOnly(); + } + removeWhiteout(path); + if (await findLowerByEntry(path)) { + await copyUpPath(path); + } else { + await ensureAncestorDirectoriesInUpper(path); } - return upper.writeFile(p, content); + return upper.writeFile(path, content); }, - async createDir(p: string): Promise { - removeWhiteout(p); - return upper.createDir(p); + async createDir(path: string): Promise { + if (!upper) { + throwReadOnly(); + } + removeWhiteout(path); + if (await pathExistsInMergedView(path)) { + throw new KernelError("EEXIST", `file exists: ${path}`); + } + await ensureAncestorDirectoriesInUpper(path); + return upper.createDir(path); }, async mkdir( - p: string, + path: string, options?: { recursive?: boolean }, ): Promise { - removeWhiteout(p); - return upper.mkdir(p, options); - }, - - async exists(p: string): Promise { - if (isWhitedOut(p)) { - return false; + removeWhiteout(path); + if (await pathExistsInMergedView(path)) { + const stat = await mergedLstat(path); + if (options?.recursive && stat.isDirectory && !stat.isSymbolicLink) { + return; + } + throw new KernelError("EEXIST", `file exists: ${path}`); } - if (await existsInUpper(p)) { - return true; + if (!upper) { + throwReadOnly(); } - return lower.exists(p); + await ensureAncestorDirectoriesInUpper(path); + return upper.mkdir(path, options); }, - async stat(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async exists(path: string): Promise { + return pathExistsInMergedView(path); + }, + + async stat(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - if (await existsInUpper(p)) { - return upper.stat(p); + if (await existsInUpper(path)) { + return upper!.stat(path); } - return lower.stat(p); + const lower = await findLowerByExists(path); + if (!lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + return lower.stat(path); }, - async removeFile(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async removeFile(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + const lower = await findLowerByExists(path); + const upperExists = await existsInUpper(path); + if (!upperExists && !lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - // If in upper, remove from upper - if (await existsInUpper(p)) { - await upper.removeFile(p); + if (!upper) { + throwReadOnly(); } - // Record whiteout so lower version is hidden - addWhiteout(p); + if (upperExists) { + await upper.removeFile(path); + } + addWhiteout(path); }, - async removeDir(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such directory: ${p}`); + async removeDir(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such directory: ${path}`); + } + const lower = await findLowerByExists(path); + const upperExists = await existsInUpper(path); + if (!upperExists && !lower) { + throw new KernelError("ENOENT", `no such directory: ${path}`); + } + if (!upper) { + throwReadOnly(); } - if (await existsInUpper(p)) { - await upper.removeDir(p); + if (upperExists) { + await upper.removeDir(path); } - addWhiteout(p); + addWhiteout(path); }, async rename(oldPath: string, newPath: string): Promise { - // Copy-up: read from wherever it exists, write to upper, whiteout old + if (!upper) { + throwReadOnly(); + } const data = await backend.readFile(oldPath); await backend.writeFile(newPath, data); await backend.removeFile(oldPath); }, - async realpath(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async realpath(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + if (await existsInUpper(path)) { + return upper!.realpath(path); } - if (await existsInUpper(p)) { - return upper.realpath(p); + const lower = await findLowerByExists(path); + if (!lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - return lower.realpath(p); + return lower.realpath(path); }, async symlink(target: string, linkPath: string): Promise { + if (!upper) { + throwReadOnly(); + } removeWhiteout(linkPath); + await ensureAncestorDirectoriesInUpper(linkPath); return upper.symlink(target, linkPath); }, - async readlink(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async readlink(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + if (await hasEntryInUpper(path)) { + return upper!.readlink(path); } - if (await existsInUpper(p)) { - return upper.readlink(p); + const lower = await findLowerByEntry(path); + if (!lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - return lower.readlink(p); + return lower.filesystem.readlink(path); }, - async lstat(p: string): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async lstat(path: string): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - if (await existsInUpper(p)) { - return upper.lstat(p); + if (await hasEntryInUpper(path)) { + return path === "/" ? upper!.stat(path) : upper!.lstat(path); } - return lower.lstat(p); + const lower = await findLowerByEntry(path); + if (!lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + return lower.stat; }, async link(oldPath: string, newPath: string): Promise { - removeWhiteout(newPath); - // Copy-up to upper for link - if (!(await existsInUpper(oldPath))) { - const data = await lower.readFile(oldPath); - await upper.writeFile(oldPath, data); + if (!upper) { + throwReadOnly(); } + removeWhiteout(newPath); + await copyUpPath(oldPath); + await ensureAncestorDirectoriesInUpper(newPath); return upper.link(oldPath, newPath); }, - async chmod(p: string, mode: number): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async chmod(path: string, modeValue: number): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - // Copy-up if only in lower - if (!(await existsInUpper(p))) { - const data = await lower.readFile(p); - await upper.writeFile(p, data); + if (!upper) { + throwReadOnly(); } - return upper.chmod(p, mode); + if (!(await existsInUpper(path))) { + await copyUpPath(path); + } + return upper.chmod(path, modeValue); }, - async chown(p: string, uid: number, gid: number): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async chown(path: string, uid: number, gid: number): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + if (!upper) { + throwReadOnly(); } - if (!(await existsInUpper(p))) { - const data = await lower.readFile(p); - await upper.writeFile(p, data); + if (!(await existsInUpper(path))) { + await copyUpPath(path); } - return upper.chown(p, uid, gid); + return upper.chown(path, uid, gid); }, - async utimes(p: string, atime: number, mtime: number): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async utimes(path: string, atime: number, mtime: number): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + if (!upper) { + throwReadOnly(); } - if (!(await existsInUpper(p))) { - const data = await lower.readFile(p); - await upper.writeFile(p, data); + if (!(await existsInUpper(path))) { + await copyUpPath(path); } - await upper.utimes(p, atime, mtime); - const updated = await upper.stat(p); - // Some backends (notably the in-memory core VFS) interpret utimes - // inputs as seconds rather than milliseconds. Normalize them here so - // the overlay presents a consistent millisecond-based contract. + await upper.utimes(path, atime, mtime); + const updated = await upper.stat(path); + // Some backends interpret utimes inputs as seconds rather than + // milliseconds. Normalize them here so the overlay presents a + // consistent millisecond-based contract. if (updated.atimeMs === atime * 1000 && updated.mtimeMs === mtime * 1000) { - await upper.utimes(p, atime / 1000, mtime / 1000); + await upper.utimes(path, atime / 1000, mtime / 1000); } }, - async truncate(p: string, length: number): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + async truncate(path: string, length: number): Promise { + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - if (!(await existsInUpper(p))) { - const data = await lower.readFile(p); - await upper.writeFile(p, data); + if (!upper) { + throwReadOnly(); } - return upper.truncate(p, length); + if (!(await existsInUpper(path))) { + await copyUpPath(path); + } + return upper.truncate(path, length); }, async pread( - p: string, + path: string, offset: number, length: number, ): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + if (await existsInUpper(path)) { + return upper!.pread(path, offset, length); } - if (await existsInUpper(p)) { - return upper.pread(p, offset, length); + const lower = await findLowerByExists(path); + if (!lower) { + throw new KernelError("ENOENT", `no such file: ${path}`); } - return lower.pread(p, offset, length); + return lower.pread(path, offset, length); }, async pwrite( - p: string, + path: string, offset: number, data: Uint8Array, ): Promise { - if (isWhitedOut(p)) { - throw new KernelError("ENOENT", `no such file: ${p}`); + if (isWhitedOut(path)) { + throw new KernelError("ENOENT", `no such file: ${path}`); + } + if (!upper) { + throwReadOnly(); } - // Copy-up if only in lower. - if (!(await existsInUpper(p))) { - const content = await lower.readFile(p); - await upper.writeFile(p, content); + if (!(await existsInUpper(path))) { + await copyUpPath(path); } - return upper.pwrite(p, offset, data); + return upper.pwrite(path, offset, data); }, }; diff --git a/packages/core/src/base-filesystem.ts b/packages/core/src/base-filesystem.ts new file mode 100644 index 000000000..6488996a1 --- /dev/null +++ b/packages/core/src/base-filesystem.ts @@ -0,0 +1,253 @@ +import { readFileSync } from "node:fs"; +import * as posixPath from "node:path/posix"; +import { KernelError, type VirtualFileSystem } from "@secure-exec/core"; +import { createOverlayBackend } from "./backends/overlay-backend.js"; +import { + createFilesystemFromEntries, + type FilesystemEntry, +} from "./filesystem-snapshot.js"; + +export interface BaseFilesystemEnvironment { + env: Record; + prompt: string; +} + +export type BaseFilesystemEntry = FilesystemEntry; + +export interface BaseFilesystemSnapshot { + source?: { + snapshotPath?: string; + image?: string; + snapshotCreatedAt?: string; + builtAt?: string; + transforms?: string[]; + }; + environment: BaseFilesystemEnvironment; + filesystem: { + entries: BaseFilesystemEntry[]; + }; +} + +const SNAPSHOT_URL = new URL("../fixtures/base-filesystem.json", import.meta.url); +const SUPPRESSED_KERNEL_BOOTSTRAP_DIRS = new Set([ + "/boot", + "/usr/games", + "/usr/include", + "/usr/libexec", + "/usr/man", +]); +const SUPPRESSED_KERNEL_BOOTSTRAP_FILES = new Set([ + "/usr/bin/env", +]); + +let snapshotCache: BaseFilesystemSnapshot | null = null; + +function loadSnapshot(): BaseFilesystemSnapshot { + if (snapshotCache) { + return snapshotCache; + } + + snapshotCache = JSON.parse( + readFileSync(SNAPSHOT_URL, "utf-8"), + ) as BaseFilesystemSnapshot; + + return snapshotCache; +} + +function normalizePath(path: string): string { + const normalized = posixPath.normalize(path); + return normalized === "." ? "/" : normalized; +} + +export async function createBaseLowerFilesystem(): Promise { + return createFilesystemFromEntries(getBaseFilesystemEntries()); +} + +export function createBootstrapAwareFilesystem( + filesystem: VirtualFileSystem, + existingRoot: VirtualFileSystem, + options?: { readOnlyAfterBootstrap?: boolean }, +): { + filesystem: VirtualFileSystem; + finishKernelBootstrap: () => void; +} { + let bootstrapActive = true; + let writesLocked = false; + + function throwReadOnly(): never { + throw new KernelError("EROFS", "read-only file system"); + } + + async function rootHasPath(path: string): Promise { + try { + return await existingRoot.exists(path); + } catch { + return false; + } + } + + const wrapped: VirtualFileSystem = { + ...filesystem, + async createDir(path: string): Promise { + if (writesLocked) { + throwReadOnly(); + } + const normalized = normalizePath(path); + if ( + bootstrapActive + && SUPPRESSED_KERNEL_BOOTSTRAP_DIRS.has(normalized) + && !(await rootHasPath(normalized)) + ) { + return; + } + return filesystem.createDir(path); + }, + + async mkdir( + path: string, + options?: { recursive?: boolean }, + ): Promise { + if (writesLocked) { + if (options?.recursive && await rootHasPath(path)) { + return; + } + throwReadOnly(); + } + const normalized = normalizePath(path); + if ( + bootstrapActive + && options?.recursive + && SUPPRESSED_KERNEL_BOOTSTRAP_DIRS.has(normalized) + && !(await rootHasPath(normalized)) + ) { + return; + } + return filesystem.mkdir(path, options); + }, + + async writeFile( + path: string, + content: string | Uint8Array, + ): Promise { + if (writesLocked) { + throwReadOnly(); + } + const normalized = normalizePath(path); + if ( + bootstrapActive + && SUPPRESSED_KERNEL_BOOTSTRAP_FILES.has(normalized) + ) { + return; + } + return filesystem.writeFile(path, content); + }, + + async removeFile(path: string): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.removeFile(path); + }, + + async removeDir(path: string): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.removeDir(path); + }, + + async rename(oldPath: string, newPath: string): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.rename(oldPath, newPath); + }, + + async symlink(target: string, linkPath: string): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.symlink(target, linkPath); + }, + + async link(oldPath: string, newPath: string): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.link(oldPath, newPath); + }, + + async chmod(path: string, mode: number): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.chmod(path, mode); + }, + + async chown(path: string, uid: number, gid: number): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.chown(path, uid, gid); + }, + + async utimes(path: string, atime: number, mtime: number): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.utimes(path, atime, mtime); + }, + + async truncate(path: string, length: number): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.truncate(path, length); + }, + + async pwrite( + path: string, + offset: number, + data: Uint8Array, + ): Promise { + if (writesLocked) { + throwReadOnly(); + } + return filesystem.pwrite(path, offset, data); + }, + }; + + return { + filesystem: wrapped, + finishKernelBootstrap(): void { + bootstrapActive = false; + writesLocked = options?.readOnlyAfterBootstrap ?? false; + }, + }; +} + +export function getBaseFilesystemSnapshot(): BaseFilesystemSnapshot { + return loadSnapshot(); +} + +export function getBaseEnvironment(): Record { + const snapshot = loadSnapshot(); + + return { + ...snapshot.environment.env, + PS1: snapshot.environment.prompt, + }; +} + +export function getBaseFilesystemEntries(): BaseFilesystemEntry[] { + return loadSnapshot().filesystem.entries; +} + +export async function createBaseRootFilesystem(): Promise<{ + filesystem: VirtualFileSystem; + finishKernelBootstrap: () => void; +}> { + const lower = await createBaseLowerFilesystem(); + const overlay = createOverlayBackend({ lower }); + return createBootstrapAwareFilesystem(overlay, lower); +} diff --git a/packages/core/src/filesystem-snapshot.ts b/packages/core/src/filesystem-snapshot.ts new file mode 100644 index 000000000..544cb8ac0 --- /dev/null +++ b/packages/core/src/filesystem-snapshot.ts @@ -0,0 +1,164 @@ +import * as posixPath from "node:path/posix"; +import { + createInMemoryFileSystem, + type VirtualFileSystem, +} from "@secure-exec/core"; + +export interface FilesystemEntry { + path: string; + type: "directory" | "file" | "symlink"; + mode: string; + uid: number; + gid: number; + content?: string; + encoding?: "utf8" | "base64"; + target?: string; +} + +function parseMode(mode: string): number { + return Number.parseInt(mode, 8); +} + +export function sortFilesystemEntries( + entries: FilesystemEntry[], +): FilesystemEntry[] { + return [...entries].sort((a, b) => { + const depthA = a.path === "/" ? 0 : a.path.split("/").filter(Boolean).length; + const depthB = b.path === "/" ? 0 : b.path.split("/").filter(Boolean).length; + + if (depthA !== depthB) { + return depthA - depthB; + } + + return a.path.localeCompare(b.path); + }); +} + +async function applyDirectory( + filesystem: VirtualFileSystem, + entry: FilesystemEntry, +): Promise { + if (entry.path !== "/") { + await filesystem.mkdir(entry.path, { recursive: true }); + } + await filesystem.chmod(entry.path, parseMode(entry.mode)); + await filesystem.chown(entry.path, entry.uid, entry.gid); +} + +async function applyFile( + filesystem: VirtualFileSystem, + entry: FilesystemEntry, +): Promise { + const content = entry.content ?? ""; + await filesystem.writeFile( + entry.path, + entry.encoding === "base64" ? Buffer.from(content, "base64") : content, + ); + await filesystem.chmod(entry.path, parseMode(entry.mode)); + await filesystem.chown(entry.path, entry.uid, entry.gid); +} + +async function applySymlink( + filesystem: VirtualFileSystem, + entry: FilesystemEntry, +): Promise { + if (!entry.target) { + throw new Error(`Missing symlink target for ${entry.path}`); + } + await filesystem.symlink(entry.target, entry.path); +} + +export async function createFilesystemFromEntries( + entries: FilesystemEntry[], +): Promise { + const filesystem = createInMemoryFileSystem(); + const sortedEntries = sortFilesystemEntries(entries); + + for (const entry of sortedEntries) { + if (entry.type === "directory") { + await applyDirectory(filesystem, entry); + } + } + + for (const entry of sortedEntries) { + if (entry.type === "file") { + await applyFile(filesystem, entry); + } + } + + for (const entry of sortedEntries) { + if (entry.type === "symlink") { + await applySymlink(filesystem, entry); + } + } + + return filesystem; +} + +function toModeString(mode: number): string { + return `0${(mode & 0o7777).toString(8)}`; +} + +async function snapshotPath( + filesystem: VirtualFileSystem, + path: string, + entries: FilesystemEntry[], +): Promise { + const stat = path === "/" ? await filesystem.stat(path) : await filesystem.lstat(path); + + if (stat.isSymbolicLink) { + entries.push({ + path, + type: "symlink", + mode: toModeString(stat.mode), + uid: stat.uid, + gid: stat.gid, + target: await filesystem.readlink(path), + }); + return; + } + + if (stat.isDirectory) { + entries.push({ + path, + type: "directory", + mode: toModeString(stat.mode), + uid: stat.uid, + gid: stat.gid, + }); + + const dirEntries = await filesystem.readDirWithTypes(path); + const children = dirEntries + .map((entry) => entry.name) + .filter((name) => name !== "." && name !== "..") + .sort((a, b) => a.localeCompare(b)); + + for (const child of children) { + const childPath = path === "/" + ? posixPath.join("/", child) + : posixPath.join(path, child); + await snapshotPath(filesystem, childPath, entries); + } + return; + } + + const content = Buffer.from(await filesystem.readFile(path)).toString("base64"); + entries.push({ + path, + type: "file", + mode: toModeString(stat.mode), + uid: stat.uid, + gid: stat.gid, + content, + encoding: "base64", + }); +} + +export async function snapshotVirtualFilesystem( + filesystem: VirtualFileSystem, + rootPath = "/", +): Promise { + const entries: FilesystemEntry[] = []; + await snapshotPath(filesystem, rootPath, entries); + return entries; +} diff --git a/packages/core/src/host-bin-runtime.ts b/packages/core/src/host-bin-runtime.ts new file mode 100644 index 000000000..c3d26c717 --- /dev/null +++ b/packages/core/src/host-bin-runtime.ts @@ -0,0 +1,117 @@ +import type { + DriverProcess, + KernelInterface, + KernelRuntimeDriver, + ProcessContext, +} from "@secure-exec/core"; +import { spawn as hostSpawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { dirname } from "node:path"; + +function resolveHostCwd(cwd: string): string { + return existsSync(cwd) ? cwd : process.cwd(); +} + +class HostBinRuntime implements KernelRuntimeDriver { + readonly name = "host-bin"; + readonly commands: string[]; + private readonly binaries: Map; + + constructor(commands: Map) { + this.binaries = new Map(commands); + this.commands = [...this.binaries.keys()]; + } + + async init(_kernel: KernelInterface): Promise { + } + + spawn( + command: string, + args: string[], + ctx: ProcessContext, + ): DriverProcess { + const commandName = command.includes("/") + ? (command.split("/").pop() ?? command) + : command; + const hostPath = this.binaries.get(commandName); + if (!hostPath) { + throw new Error(`Unknown host binary command: ${command}`); + } + + const env = { ...ctx.env }; + env.LD_LIBRARY_PATH = env.LD_LIBRARY_PATH + ? `${dirname(hostPath)}:${env.LD_LIBRARY_PATH}` + : dirname(hostPath); + + const child = hostSpawn(hostPath, args, { + cwd: resolveHostCwd(ctx.cwd), + env, + stdio: ["pipe", "pipe", "pipe"], + }); + + let resolveExit!: (code: number) => void; + const exitPromise = new Promise((resolve) => { + resolveExit = resolve; + }); + + const proc: DriverProcess = { + onStdout: null, + onStderr: null, + onExit: null, + writeStdin(data) { + child.stdin?.write(data); + }, + closeStdin() { + child.stdin?.end(); + }, + kill(signal) { + try { + child.kill(signal > 0 ? signal : undefined); + } catch { + } + }, + wait() { + return exitPromise; + }, + }; + + child.stdout?.on("data", (chunk) => { + const data = new Uint8Array(chunk); + ctx.onStdout?.(data); + proc.onStdout?.(data); + }); + child.stderr?.on("data", (chunk) => { + const data = new Uint8Array(chunk); + ctx.onStderr?.(data); + proc.onStderr?.(data); + }); + child.on("error", (error) => { + const data = new TextEncoder().encode(`${error.message}\n`); + ctx.onStderr?.(data); + proc.onStderr?.(data); + resolveExit(1); + proc.onExit?.(1); + }); + child.on("close", (code, signal) => { + const exitCode = + typeof code === "number" + ? code + : signal + ? 128 + 15 + : 1; + resolveExit(exitCode); + proc.onExit?.(exitCode); + }); + + return proc; + } + + async dispose(): Promise { + } +} + +export function createHostBinRuntime( + commands: Map, +): KernelRuntimeDriver { + return new HostBinRuntime(commands); +} diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 4b215d4d5..d2b67126d 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -25,12 +25,16 @@ export type { BatchWriteResult, CreateSessionOptions, DirEntry, + OverlayMountConfig, McpServerConfig, McpServerConfigLocal, McpServerConfigRemote, MountConfig, + PlainMountConfig, ProcessTreeNode, ReaddirRecursiveOptions, + RootFilesystemConfig, + RootLowerInput, SessionInfo, SpawnedProcessInfo, } from "./agent-os.js"; @@ -57,6 +61,20 @@ export type { HostDirBackendOptions } from "./backends/host-dir-backend.js"; export { createHostDirBackend } from "./backends/host-dir-backend.js"; export type { OverlayBackendOptions } from "./backends/overlay-backend.js"; export { createOverlayBackend } from "./backends/overlay-backend.js"; +export type { + FilesystemSnapshotExport, + LayerHandle, + LayerStore, + OverlayFilesystemMode, + RootSnapshotExport, + SnapshotImportSource, + SnapshotLayerHandle, + WritableLayerHandle, +} from "./layers.js"; +export { + createInMemoryLayerStore, + createSnapshotExport, +} from "./layers.js"; export type { CronAction, CronEvent, diff --git a/packages/core/src/layers.ts b/packages/core/src/layers.ts new file mode 100644 index 000000000..d274d79ec --- /dev/null +++ b/packages/core/src/layers.ts @@ -0,0 +1,314 @@ +import { randomUUID } from "node:crypto"; +import { type VirtualFileSystem } from "@secure-exec/core"; +import { getBaseFilesystemSnapshot, type BaseFilesystemSnapshot } from "./base-filesystem.js"; +import { createOverlayBackend } from "./backends/overlay-backend.js"; +import { + createFilesystemFromEntries, + snapshotVirtualFilesystem, + type FilesystemEntry, +} from "./filesystem-snapshot.js"; + +export type OverlayFilesystemMode = "ephemeral" | "read-only"; + +export interface FilesystemSnapshotExport { + format: "agent-os-filesystem-snapshot-v1"; + filesystem: { + entries: FilesystemEntry[]; + }; +} + +export type RootSnapshotExport = { + kind: "snapshot-export"; + source: FilesystemSnapshotExport; +}; + +export interface LayerHandle { + kind: "writable" | "snapshot"; + storeId: string; + layerId: string; +} + +export interface WritableLayerHandle extends LayerHandle { + kind: "writable"; + leaseId: string; +} + +export interface SnapshotLayerHandle extends LayerHandle { + kind: "snapshot"; +} + +export type SnapshotImportSource = + | { kind: "base-filesystem-artifact"; source: BaseFilesystemSnapshot | unknown } + | { kind: "snapshot-export"; source: FilesystemSnapshotExport | unknown }; + +export interface LayerStore { + readonly storeId: string; + createWritableLayer(): Promise; + importSnapshot(source: SnapshotImportSource): Promise; + openSnapshotLayer(layerId: string): Promise; + sealLayer(layer: WritableLayerHandle): Promise; + createOverlayFilesystem( + options: + | { + mode?: "ephemeral"; + upper: WritableLayerHandle; + lowers: SnapshotLayerHandle[]; + } + | { + mode: "read-only"; + lowers: SnapshotLayerHandle[]; + }, + ): VirtualFileSystem; +} + +interface WritableLayerState { + kind: "writable"; + fs: VirtualFileSystem; + leaseId: string; + valid: boolean; + activeOverlay: VirtualFileSystem | null; +} + +interface SnapshotLayerState { + kind: "snapshot"; + snapshot: FilesystemSnapshotExport; + fs: VirtualFileSystem; +} + +type LayerState = WritableLayerState | SnapshotLayerState; + +function cloneSnapshotHandle( + storeId: string, + layerId: string, +): SnapshotLayerHandle { + return { kind: "snapshot", storeId, layerId }; +} + +function cloneWritableHandle( + storeId: string, + layerId: string, + leaseId: string, +): WritableLayerHandle { + return { kind: "writable", storeId, layerId, leaseId }; +} + +function isBaseFilesystemSnapshot( + value: unknown, +): value is BaseFilesystemSnapshot { + if (!value || typeof value !== "object") { + return false; + } + + const filesystem = (value as { filesystem?: unknown }).filesystem; + if (!filesystem || typeof filesystem !== "object") { + return false; + } + + return Array.isArray( + (filesystem as { entries?: unknown }).entries, + ); +} + +function isFilesystemSnapshotExport( + value: unknown, +): value is FilesystemSnapshotExport { + if (!value || typeof value !== "object") { + return false; + } + + if ((value as { format?: unknown }).format !== "agent-os-filesystem-snapshot-v1") { + return false; + } + + const filesystem = (value as { filesystem?: unknown }).filesystem; + if (!filesystem || typeof filesystem !== "object") { + return false; + } + + return Array.isArray( + (filesystem as { entries?: unknown }).entries, + ); +} + +function normalizeSnapshotExport(source: SnapshotImportSource): FilesystemSnapshotExport { + if (source.kind === "base-filesystem-artifact") { + if (!isBaseFilesystemSnapshot(source.source)) { + throw new Error("Invalid base filesystem artifact"); + } + return { + format: "agent-os-filesystem-snapshot-v1", + filesystem: { + entries: source.source.filesystem.entries, + }, + }; + } + + if (!isFilesystemSnapshotExport(source.source)) { + throw new Error("Invalid snapshot export"); + } + + return source.source; +} + +export function createSnapshotExport( + entries: FilesystemEntry[], +): RootSnapshotExport { + return { + kind: "snapshot-export", + source: { + format: "agent-os-filesystem-snapshot-v1", + filesystem: { entries }, + }, + }; +} + +export function createInMemoryLayerStore(): LayerStore { + const storeId = `memory-layer-store:${randomUUID()}`; + const layers = new Map(); + + function getLayerState(handle: LayerHandle): LayerState { + if (handle.storeId !== storeId) { + throw new Error( + `Layer ${handle.layerId} belongs to store ${handle.storeId}, not ${storeId}`, + ); + } + + const state = layers.get(handle.layerId); + if (!state) { + throw new Error(`Unknown layer: ${handle.layerId}`); + } + if (state.kind !== handle.kind) { + throw new Error(`Layer kind mismatch for ${handle.layerId}`); + } + return state; + } + + return { + storeId, + + async createWritableLayer(): Promise { + const layerId = randomUUID(); + const leaseId = randomUUID(); + layers.set(layerId, { + kind: "writable", + fs: await createFilesystemFromEntries([ + { + path: "/", + type: "directory", + mode: "0755", + uid: 0, + gid: 0, + }, + ]), + leaseId, + valid: true, + activeOverlay: null, + }); + return cloneWritableHandle(storeId, layerId, leaseId); + }, + + async importSnapshot( + source: SnapshotImportSource, + ): Promise { + const snapshot = normalizeSnapshotExport(source); + const layerId = randomUUID(); + layers.set(layerId, { + kind: "snapshot", + snapshot, + fs: await createFilesystemFromEntries(snapshot.filesystem.entries), + }); + return cloneSnapshotHandle(storeId, layerId); + }, + + async openSnapshotLayer(layerId: string): Promise { + const state = layers.get(layerId); + if (!state || state.kind !== "snapshot") { + throw new Error(`Unknown snapshot layer: ${layerId}`); + } + return cloneSnapshotHandle(storeId, layerId); + }, + + async sealLayer(layer: WritableLayerHandle): Promise { + const state = getLayerState(layer); + if (state.kind !== "writable") { + throw new Error(`Layer ${layer.layerId} is not writable`); + } + if (!state.valid || state.leaseId !== layer.leaseId) { + throw new Error(`Writable layer ${layer.layerId} is no longer valid`); + } + + const entries = await snapshotVirtualFilesystem( + state.activeOverlay ?? state.fs, + ); + const snapshot = createSnapshotExport(entries).source; + const layerId = randomUUID(); + + layers.set(layerId, { + kind: "snapshot", + snapshot, + fs: await createFilesystemFromEntries(snapshot.filesystem.entries), + }); + + state.valid = false; + state.activeOverlay = null; + + return cloneSnapshotHandle(storeId, layerId); + }, + + createOverlayFilesystem(options): VirtualFileSystem { + const lowers = options.lowers.map((lower) => { + const state = getLayerState(lower); + if (state.kind !== "snapshot") { + throw new Error(`Layer ${lower.layerId} is not a snapshot`); + } + return lower; + }); + + if (options.mode === "read-only") { + return createOverlayBackend({ + mode: "read-only", + lowers: lowers.map((lower) => { + const state = getLayerState(lower); + if (state.kind !== "snapshot") { + throw new Error(`Layer ${lower.layerId} is not a snapshot`); + } + return state.fs; + }), + }); + } + + const upperState = getLayerState(options.upper); + if (upperState.kind !== "writable") { + throw new Error(`Layer ${options.upper.layerId} is not writable`); + } + if (!upperState.valid || upperState.leaseId !== options.upper.leaseId) { + throw new Error(`Writable layer ${options.upper.layerId} is no longer valid`); + } + if (upperState.activeOverlay) { + throw new Error( + `Writable layer ${options.upper.layerId} is already attached to an overlay`, + ); + } + + const overlay = createOverlayBackend({ + upper: upperState.fs, + lowers: lowers.map((lower) => { + const state = getLayerState(lower); + if (state.kind !== "snapshot") { + throw new Error(`Layer ${lower.layerId} is not a snapshot`); + } + return state.fs; + }), + }); + upperState.activeOverlay = overlay; + return overlay; + }, + }; +} + +export function createDefaultRootLowerInput(): SnapshotImportSource { + return { + kind: "base-filesystem-artifact", + source: getBaseFilesystemSnapshot(), + }; +} diff --git a/packages/core/src/packages.ts b/packages/core/src/packages.ts index e20338b78..750d2973c 100644 --- a/packages/core/src/packages.ts +++ b/packages/core/src/packages.ts @@ -230,10 +230,37 @@ export interface ProcessedSoftware { commandDirs: string[]; /** Host-to-VM path mappings for ModuleAccessFileSystem. */ softwareRoots: SoftwareRoot[]; + /** Registered host binary commands exposed by tool software. */ + toolBinaries: Map; /** Agent configs registered by agent software. */ agentConfigs: Map; } +function resolvePackageBinPath( + hostDir: string, + packageName: string, + binName: string, +): string { + const pkg = JSON.parse( + readFileSync(join(hostDir, "package.json"), "utf-8"), + ) as { bin?: string | Record }; + + let binEntry: string | undefined; + if (typeof pkg.bin === "string") { + binEntry = pkg.bin; + } else if (typeof pkg.bin === "object" && pkg.bin !== null) { + binEntry = pkg.bin[binName] ?? Object.values(pkg.bin)[0]; + } + + if (!binEntry) { + throw new Error( + `No bin entry "${binName}" found in ${packageName}/package.json`, + ); + } + + return join(hostDir, binEntry); +} + /** Check if a descriptor is a typed software descriptor (has a `type` field). */ function isTypedDescriptor(desc: AnySoftwareDescriptor): desc is AgentSoftwareDescriptor | ToolSoftwareDescriptor | WasmCommandSoftwareDescriptor { return "type" in desc && typeof (desc as SoftwareDescriptor).type === "string"; @@ -252,6 +279,7 @@ export function processSoftware( ): ProcessedSoftware { const commandDirs: string[] = []; const softwareRoots: SoftwareRoot[] = []; + const toolBinaries = new Map(); const agentConfigs = new Map(); // Flatten nested arrays (meta-packages export arrays of sub-packages). @@ -306,12 +334,17 @@ export function processSoftware( const vmDir = `/root/node_modules/${reqPkg}`; softwareRoots.push({ hostPath: hostDir, vmPath: vmDir }); } - // Tool bin registration is handled by the caller (AgentOs.create) - // since it requires kernel access. + for (const [commandName, packageName] of Object.entries(pkg.bins)) { + const hostDir = resolvePackageDir(pkg.packageDir, packageName); + toolBinaries.set( + commandName, + resolvePackageBinPath(hostDir, packageName, commandName), + ); + } break; } } } - return { commandDirs, softwareRoots, agentConfigs }; + return { commandDirs, softwareRoots, toolBinaries, agentConfigs }; } diff --git a/packages/core/src/sqlite-bindings.ts b/packages/core/src/sqlite-bindings.ts index c2abd3135..72260e408 100644 --- a/packages/core/src/sqlite-bindings.ts +++ b/packages/core/src/sqlite-bindings.ts @@ -315,11 +315,12 @@ export function createSqliteBindings(kernel: Kernel): BindingTree { : null; try { if (hostPath) { - if (await kernel.exists(vmPath)) { + const vmPathString = vmPath ?? path ?? ":memory:"; + if (await kernel.exists(vmPathString)) { mkdirSync(hostDirname(hostPath), { recursive: true }); writeFileSync( hostPath, - Buffer.from(await kernel.readFile(vmPath)), + Buffer.from(await kernel.readFile(vmPathString)), ); } } diff --git a/packages/core/tests/agent-os-base-filesystem.test.ts b/packages/core/tests/agent-os-base-filesystem.test.ts new file mode 100644 index 000000000..dfd8db024 --- /dev/null +++ b/packages/core/tests/agent-os-base-filesystem.test.ts @@ -0,0 +1,180 @@ +import { afterEach, beforeEach, describe, expect, test } from "vitest"; +import coreutils from "@rivet-dev/agent-os-coreutils"; +import { AgentOs } from "../src/agent-os.js"; +import { + getBaseEnvironment, + getBaseFilesystemEntries, +} from "../src/base-filesystem.js"; +import { hasRegistryCommands } from "./helpers/registry-commands.js"; + +describe("AgentOs base filesystem", () => { + let vm: AgentOs; + const textDecoder = new TextDecoder(); + + beforeEach(async () => { + vm = await AgentOs.create(); + }); + + afterEach(async () => { + await vm.dispose(); + }); + + test("default environment matches the base environment", () => { + expect(vm.kernel.env).toEqual(getBaseEnvironment()); + expect((vm.kernel as unknown as { cwd: string }).cwd).toBe("/home/user"); + }); + + test("default filesystem matches the base layer", async () => { + const vfs = (vm.kernel as unknown as { + vfs: { + lstat: (path: string) => Promise<{ + mode: number; + uid: number; + gid: number; + isDirectory: boolean; + isSymbolicLink: boolean; + }>; + readlink: (path: string) => Promise; + }; + }).vfs; + + for (const entry of getBaseFilesystemEntries()) { + if (entry.type === "symlink") { + const stat = await vfs.lstat(entry.path); + expect(stat.isSymbolicLink).toBe(true); + expect(stat.isDirectory).toBe(false); + expect(stat.mode & 0o7777).toBe(Number.parseInt(entry.mode, 8)); + expect(stat.uid).toBe(entry.uid); + expect(stat.gid).toBe(entry.gid); + expect(await vfs.readlink(entry.path)).toBe(entry.target); + continue; + } + + const stat = await vm.stat(entry.path); + expect(stat.isDirectory).toBe(entry.type === "directory"); + expect(stat.isSymbolicLink).toBe(false); + expect(stat.mode & 0o7777).toBe(Number.parseInt(entry.mode, 8)); + expect(stat.uid).toBe(entry.uid); + expect(stat.gid).toBe(entry.gid); + + if (entry.type === "file" && entry.content !== undefined) { + const data = await vm.readFile(entry.path); + expect(textDecoder.decode(data)).toBe(entry.content); + } + } + }); + + test("overlay writes and deletes do not mutate the shared base layer", async () => { + const baselineProfile = textDecoder.decode( + await vm.readFile("/etc/profile"), + ); + + await vm.writeFile("/tmp/overlay-only.txt", "overlay data"); + await vm.delete("/etc/profile"); + + expect(textDecoder.decode(await vm.readFile("/tmp/overlay-only.txt"))).toBe( + "overlay data", + ); + await expect(vm.readFile("/etc/profile")).rejects.toThrow("ENOENT"); + + const secondVm = await AgentOs.create(); + try { + expect(await secondVm.exists("/tmp/overlay-only.txt")).toBe(false); + expect( + textDecoder.decode(await secondVm.readFile("/etc/profile")), + ).toBe(baselineProfile); + } finally { + await secondVm.dispose(); + } + }); + + test("rootFilesystem can disable the bundled base layer", async () => { + await vm.dispose(); + vm = await AgentOs.create({ + rootFilesystem: { + disableDefaultBaseLayer: true, + }, + }); + + await expect(vm.readFile("/etc/profile")).rejects.toThrow("ENOENT"); + await vm.mkdir("/work"); + await vm.writeFile("/work/hello.txt", "from empty root"); + expect(textDecoder.decode(await vm.readFile("/work/hello.txt"))).toBe( + "from empty root", + ); + }); + + test("read-only roots expose lowers but reject writes", async () => { + await vm.dispose(); + vm = await AgentOs.create({ + rootFilesystem: { + mode: "read-only", + }, + }); + + expect(textDecoder.decode(await vm.readFile("/etc/profile"))).toContain( + "PATH", + ); + await expect( + vm.writeFile("/home/user/blocked.txt", "blocked"), + ).rejects.toThrow("EROFS"); + }); + + test("read-only roots can boot from a preseeded lower without a writable upper", async () => { + await vm.dispose(); + vm = await AgentOs.create({ + rootFilesystem: { + mode: "read-only", + disableDefaultBaseLayer: true, + }, + }); + + expect(await vm.exists("/boot")).toBe(true); + expect(await vm.exists("/usr/bin/env")).toBe(true); + expect(await vm.exists("/bin/node")).toBe(true); + expect(await vm.exists("/bin/python")).toBe(true); + await expect( + vm.writeFile("/tmp/blocked.txt", "blocked"), + ).rejects.toThrow("EROFS"); + }); + + test.skipIf(!hasRegistryCommands)( + "read-only roots preseed WASM command stubs before runtime mount", + async () => { + await vm.dispose(); + vm = await AgentOs.create({ + software: [coreutils], + rootFilesystem: { + mode: "read-only", + disableDefaultBaseLayer: true, + }, + }); + + expect(await vm.exists("/bin/sh")).toBe(true); + expect(await vm.exists("/bin/ls")).toBe(true); + expect(await vm.exists("/bin/env")).toBe(true); + }, + ); + + test("snapshotRootFilesystem exports a reusable lower snapshot", async () => { + await vm.writeFile("/home/user/snap.txt", "snapshotted"); + const snapshot = await vm.snapshotRootFilesystem(); + + const secondVm = await AgentOs.create({ + rootFilesystem: { + disableDefaultBaseLayer: true, + lowers: [snapshot], + }, + }); + try { + expect( + textDecoder.decode(await secondVm.readFile("/home/user/snap.txt")), + ).toBe("snapshotted"); + expect( + textDecoder.decode(await secondVm.readFile("/etc/profile")), + ).toBe(textDecoder.decode(await vm.readFile("/etc/profile"))); + } finally { + await secondVm.dispose(); + } + }); +}); diff --git a/packages/core/tests/layers.test.ts b/packages/core/tests/layers.test.ts new file mode 100644 index 000000000..ccaaf6cce --- /dev/null +++ b/packages/core/tests/layers.test.ts @@ -0,0 +1,33 @@ +import { describe, expect, test } from "vitest"; +import { createInMemoryLayerStore } from "../src/index.js"; + +describe("layer store", () => { + test("sealed writable layers become reusable read-only snapshots", async () => { + const store = createInMemoryLayerStore(); + const upper = await store.createWritableLayer(); + const overlay = store.createOverlayFilesystem({ + upper, + lowers: [], + }); + + await overlay.mkdir("/data", { recursive: true }); + await overlay.writeFile("/data/note.txt", "hello from layer"); + + const snapshot = await store.sealLayer(upper); + const reopened = await store.openSnapshotLayer(snapshot.layerId); + const readOnlyOverlay = store.createOverlayFilesystem({ + mode: "read-only", + lowers: [reopened], + }); + + expect(await readOnlyOverlay.readTextFile("/data/note.txt")).toBe( + "hello from layer", + ); + expect(() => + store.createOverlayFilesystem({ + upper, + lowers: [], + }) + ).toThrow("no longer valid"); + }); +}); diff --git a/packages/core/tests/mount.test.ts b/packages/core/tests/mount.test.ts index 67f75feb2..f400da417 100644 --- a/packages/core/tests/mount.test.ts +++ b/packages/core/tests/mount.test.ts @@ -1,5 +1,10 @@ import { afterEach, describe, expect, test } from "vitest"; -import { AgentOs, createInMemoryFileSystem } from "../src/index.js"; +import { + AgentOs, + createInMemoryFileSystem, + createInMemoryLayerStore, + createSnapshotExport, +} from "../src/index.js"; describe("mount integration", () => { let vm: AgentOs; @@ -78,4 +83,84 @@ describe("mount integration", () => { vm.writeFile("/ro/blocked.txt", "should fail"), ).rejects.toThrow("EROFS"); }); + + test("declarative overlay mounts create an isolated writable upper", async () => { + const store = createInMemoryLayerStore(); + const lower = await store.importSnapshot({ + kind: "snapshot-export", + source: createSnapshotExport([ + { + path: "/", + type: "directory", + mode: "0755", + uid: 0, + gid: 0, + }, + { + path: "/seed.txt", + type: "file", + mode: "0644", + uid: 0, + gid: 0, + content: Buffer.from("seeded").toString("base64"), + encoding: "base64", + }, + ]).source, + }); + + vm = await AgentOs.create({ + mounts: [ + { + path: "/data", + filesystem: { + type: "overlay", + store, + lowers: [lower], + }, + }, + ], + }); + + expect(new TextDecoder().decode(await vm.readFile("/data/seed.txt"))).toBe( + "seeded", + ); + await vm.writeFile("/data/new.txt", "overlay mount"); + expect(new TextDecoder().decode(await vm.readFile("/data/new.txt"))).toBe( + "overlay mount", + ); + }); + + test("read-only overlay mounts reject writes", async () => { + const store = createInMemoryLayerStore(); + const lower = await store.importSnapshot({ + kind: "snapshot-export", + source: createSnapshotExport([ + { + path: "/", + type: "directory", + mode: "0755", + uid: 0, + gid: 0, + }, + ]).source, + }); + + vm = await AgentOs.create({ + mounts: [ + { + path: "/data", + filesystem: { + type: "overlay", + store, + mode: "read-only", + lowers: [lower], + }, + }, + ], + }); + + await expect( + vm.writeFile("/data/blocked.txt", "should fail"), + ).rejects.toThrow("EROFS"); + }); }); diff --git a/packages/core/tests/overlay-backend.test.ts b/packages/core/tests/overlay-backend.test.ts index 57ce7de76..2c56a4e78 100644 --- a/packages/core/tests/overlay-backend.test.ts +++ b/packages/core/tests/overlay-backend.test.ts @@ -93,6 +93,17 @@ describe("OverlayBackend (layer behavior)", () => { await expect(overlay.stat("/data/base.txt")).rejects.toThrow("ENOENT"); }); + test("deleting a lower-layer file preserves lower data and does not copy it into upper", async () => { + await overlay.removeFile("/data/base.txt"); + + expect(await overlay.exists("/data/base.txt")).toBe(false); + expect(await lower.readTextFile("/data/base.txt")).toBe("base content"); + expect(await upper.exists("/data/base.txt")).toBe(false); + + const entries = await overlay.readDir("/data"); + expect(entries).not.toContain("base.txt"); + }); + test("readdir merges both layers and excludes whiteouts", async () => { await overlay.mkdir("/data", { recursive: true }); await overlay.writeFile("/data/upper-only.txt", "upper only"); @@ -144,4 +155,87 @@ describe("OverlayBackend (layer behavior)", () => { expect(await lower.exists("/data/new.txt")).toBe(false); }); + + test("mkdir -p on an existing lower directory is a no-op", async () => { + await lower.chmod("/data", 0o2755); + + await overlay.mkdir("/data", { recursive: true }); + + expect(await upper.exists("/data")).toBe(false); + const stat = await overlay.stat("/data"); + expect(stat.mode & 0o7777).toBe(0o2755); + }); + + test("writeFile copies lower parent directory metadata into upper", async () => { + await lower.chmod("/data", 0o2755); + await lower.chown("/data", 1000, 1000); + + await overlay.writeFile("/data/new.txt", "new content"); + + const parentStat = await upper.stat("/data"); + expect(parentStat.mode & 0o7777).toBe(0o2755); + expect(parentStat.uid).toBe(1000); + expect(parentStat.gid).toBe(1000); + }); + + test("mkdir -p on a lower symlink-to-directory preserves the symlink", async () => { + await lower.mkdir("/run/lock", { recursive: true }); + await lower.symlink("../run/lock", "/var-lock"); + + await expect( + overlay.mkdir("/var-lock", { recursive: true }), + ).rejects.toThrow("EEXIST"); + + const stat = await overlay.lstat("/var-lock"); + expect(stat.isSymbolicLink).toBe(true); + expect(await overlay.readlink("/var-lock")).toBe("../run/lock"); + expect(await upper.exists("/var-lock")).toBe(false); + }); + + test("multiple lowers resolve highest-precedence layer first", async () => { + const higher = createInMemoryFileSystem(); + const deeper = createInMemoryFileSystem(); + + await higher.mkdir("/etc", { recursive: true }); + await deeper.mkdir("/etc", { recursive: true }); + await higher.writeFile("/etc/config.txt", "from higher"); + await deeper.writeFile("/etc/config.txt", "from deeper"); + await deeper.writeFile("/etc/deep-only.txt", "deep only"); + + const multiLowerOverlay = createOverlayBackend({ + lowers: [higher, deeper], + }); + + expect(await multiLowerOverlay.readTextFile("/etc/config.txt")).toBe( + "from higher", + ); + expect(await multiLowerOverlay.readTextFile("/etc/deep-only.txt")).toBe( + "deep only", + ); + }); + + test("read-only overlays allow reads and reject writes", async () => { + const readOnlyOverlay = createOverlayBackend({ + mode: "read-only", + lowers: [lower], + }); + + expect(await readOnlyOverlay.readTextFile("/data/base.txt")).toBe( + "base content", + ); + await expect( + readOnlyOverlay.writeFile("/data/new.txt", "blocked"), + ).rejects.toThrow("EROFS"); + }); + + test("read-only mkdir -p on an existing lower directory is a no-op", async () => { + const readOnlyOverlay = createOverlayBackend({ + mode: "read-only", + lowers: [lower], + }); + + await expect( + readOnlyOverlay.mkdir("/data", { recursive: true }), + ).resolves.toBeUndefined(); + }); }); diff --git a/packages/shell/.npmrc b/packages/shell/.npmrc new file mode 100644 index 000000000..1774dfb09 --- /dev/null +++ b/packages/shell/.npmrc @@ -0,0 +1,4 @@ +# Use npm-published software packages (not workspace links) because they +# include pre-built WASM binaries. The workspace copies have empty wasm/ +# dirs since the native build (Rust nightly + wasi-sdk) is not run locally. +link-workspace-packages=false diff --git a/packages/shell/package.json b/packages/shell/package.json new file mode 100644 index 000000000..70cdf9ef8 --- /dev/null +++ b/packages/shell/package.json @@ -0,0 +1,32 @@ +{ + "name": "@rivet-dev/agent-os-shell", + "private": true, + "type": "module", + "bin": { + "agent-os-shell": "./dist/main.js" + }, + "scripts": { + "build": "tsc", + "check-types": "tsc --noEmit", + "shell": "tsx src/main.ts" + }, + "dependencies": { + "@rivet-dev/agent-os-core": "workspace:*", + "@rivet-dev/agent-os-common": "0.0.260331072558", + "@rivet-dev/agent-os-jq": "0.0.260331072558", + "@rivet-dev/agent-os-ripgrep": "0.0.260331072558", + "@rivet-dev/agent-os-fd": "0.0.260331072558", + "@rivet-dev/agent-os-tree": "0.0.260331072558", + "@rivet-dev/agent-os-file": "0.0.260331072558", + "@rivet-dev/agent-os-zip": "0.0.260331072558", + "@rivet-dev/agent-os-unzip": "0.0.260331072558", + "@rivet-dev/agent-os-yq": "0.0.260331072558", + "@rivet-dev/agent-os-codex": "0.0.260331072558", + "pyodide": "^0.28.3" + }, + "devDependencies": { + "@types/node": "^22.19.3", + "tsx": "^4.19.2", + "typescript": "^5.7.2" + } +} diff --git a/packages/shell/src/main.ts b/packages/shell/src/main.ts new file mode 100644 index 000000000..eed332433 --- /dev/null +++ b/packages/shell/src/main.ts @@ -0,0 +1,117 @@ +#!/usr/bin/env node + +import { AgentOs } from "@rivet-dev/agent-os-core"; + +// Software packages — uses npm-published versions which include pre-built +// WASM binaries. Workspace copies have empty wasm/ dirs since the native +// build (Rust nightly + wasi-sdk) is not run locally. +// curl, wget, sqlite3 are excluded (not yet published, need patched wasi-libc). +import common from "@rivet-dev/agent-os-common"; +import jq from "@rivet-dev/agent-os-jq"; +import ripgrep from "@rivet-dev/agent-os-ripgrep"; +import fd from "@rivet-dev/agent-os-fd"; +import tree from "@rivet-dev/agent-os-tree"; +import file from "@rivet-dev/agent-os-file"; +import zip from "@rivet-dev/agent-os-zip"; +import unzip from "@rivet-dev/agent-os-unzip"; +import yq from "@rivet-dev/agent-os-yq"; +import codex from "@rivet-dev/agent-os-codex"; + +const software = [ + common, + jq, + ripgrep, + fd, + tree, + file, + zip, + unzip, + yq, + codex, +]; + +function printUsage(): void { + console.error( + [ + "Usage:", + " agent-os-shell [--work-dir ] [--] [command] [args...]", + "", + "Options:", + " --work-dir Set the working directory inside the VM (default: /home/user)", + " --help, -h Show this help", + "", + "Examples:", + " pnpm shell", + " pnpm shell --work-dir /tmp/demo", + " pnpm shell -- node -e 'console.log(42)'", + ].join("\n"), + ); +} + +interface CliOptions { + workDir?: string; + command: string; + args: string[]; +} + +function parseArgs(argv: string[]): CliOptions { + const options: CliOptions = { + command: "bash", + args: [], + }; + + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + if (arg === "--") { + const trailing = argv.slice(i + 1); + if (trailing.length > 0) { + options.command = trailing[0]; + options.args = trailing.slice(1); + } + break; + } + + if (!arg.startsWith("-")) { + options.command = arg; + options.args = argv.slice(i + 1); + break; + } + + switch (arg) { + case "--work-dir": + if (!argv[i + 1]) { + throw new Error("--work-dir requires a path"); + } + options.workDir = argv[++i]; + break; + case "--help": + case "-h": + printUsage(); + process.exit(0); + default: + throw new Error(`Unknown argument: ${arg}`); + } + } + + return options; +} + +const cli = parseArgs(process.argv.slice(2)); + +const vm = await AgentOs.create({ + software, +}); + +const cwd = cli.workDir ?? "/home/user"; + +console.error("agent-os shell"); +console.error(`cwd: ${cwd}`); + +const exitCode = await vm.kernel.connectTerminal({ + command: cli.command, + args: cli.args, + cwd, +}); + +await vm.dispose(); +process.exit(exitCode); diff --git a/packages/shell/tsconfig.json b/packages/shell/tsconfig.json new file mode 100644 index 000000000..e44a71030 --- /dev/null +++ b/packages/shell/tsconfig.json @@ -0,0 +1,15 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "NodeNext", + "moduleResolution": "NodeNext", + "declaration": true, + "outDir": "./dist", + "rootDir": "./src", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist"] +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 2d775e1ed..68388086b 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9,7 +9,7 @@ overrides: patchedDependencies: '@secure-exec/nodejs@0.2.1': - hash: 1b55bfdc13a5329270980574e382c56c27c145390cb03e3eb7bce6c620be4c69 + hash: dc7b85660389e571beac82d61aa14936e29803ba733156fb8d3df12504c847dc path: patches/@secure-exec__nodejs@0.2.1.patch '@secure-exec/v8@0.2.1': hash: e542d0ba16b29b7dd6e777f4f16d26ddf6da49ac7f8286912419de975a3a5ead @@ -156,7 +156,7 @@ importers: version: 0.2.1 '@secure-exec/nodejs': specifier: ^0.2.1 - version: 0.2.1(patch_hash=1b55bfdc13a5329270980574e382c56c27c145390cb03e3eb7bce6c620be4c69) + version: 0.2.1(patch_hash=dc7b85660389e571beac82d61aa14936e29803ba733156fb8d3df12504c847dc) '@secure-exec/v8': specifier: ^0.2.1 version: 0.2.1(patch_hash=e542d0ba16b29b7dd6e777f4f16d26ddf6da49ac7f8286912419de975a3a5ead) @@ -277,7 +277,7 @@ importers: version: 0.2.1 '@secure-exec/nodejs': specifier: ^0.2.1 - version: 0.2.1(patch_hash=1b55bfdc13a5329270980574e382c56c27c145390cb03e3eb7bce6c620be4c69) + version: 0.2.1(patch_hash=dc7b85660389e571beac82d61aa14936e29803ba733156fb8d3df12504c847dc) pino: specifier: ^10.3.1 version: 10.3.1 @@ -362,7 +362,7 @@ importers: devDependencies: '@secure-exec/nodejs': specifier: ^0.2.1 - version: 0.2.1(patch_hash=1b55bfdc13a5329270980574e382c56c27c145390cb03e3eb7bce6c620be4c69) + version: 0.2.1(patch_hash=dc7b85660389e571beac82d61aa14936e29803ba733156fb8d3df12504c847dc) '@types/node': specifier: ^22.10.2 version: 22.19.15 @@ -382,6 +382,55 @@ importers: specifier: ^5.7.2 version: 5.9.3 + packages/shell: + dependencies: + '@rivet-dev/agent-os-codex': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + '@rivet-dev/agent-os-common': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + '@rivet-dev/agent-os-core': + specifier: workspace:* + version: link:../core + '@rivet-dev/agent-os-fd': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + '@rivet-dev/agent-os-file': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + '@rivet-dev/agent-os-jq': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + '@rivet-dev/agent-os-ripgrep': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + '@rivet-dev/agent-os-tree': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + '@rivet-dev/agent-os-unzip': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + '@rivet-dev/agent-os-yq': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + '@rivet-dev/agent-os-zip': + specifier: 0.0.260331072558 + version: 0.0.260331072558 + pyodide: + specifier: ^0.28.3 + version: 0.28.3 + devDependencies: + '@types/node': + specifier: ^22.19.3 + version: 22.19.15 + tsx: + specifier: ^4.19.2 + version: 4.21.0 + typescript: + specifier: ^5.7.2 + version: 5.9.3 + registry/agent/claude: dependencies: '@agentclientprotocol/sdk': @@ -428,9 +477,6 @@ importers: '@rivet-dev/agent-os-core': specifier: workspace:* version: link:../../../packages/core - jsonc-parser: - specifier: ^3.3.1 - version: 3.3.1 devDependencies: '@types/node': specifier: ^22.10.2 @@ -1824,6 +1870,60 @@ packages: '@protobufjs/utf8@1.1.0': resolution: {integrity: sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==} + '@rivet-dev/agent-os-codex@0.0.260331072558': + resolution: {integrity: sha512-Fs+w0vkr/z1tjIWg52OYytttXXiQFkS1ahVm28SjlusTFLy156ofai7me2ZDYnfCsA4DvKi/T5XFBl/2z5oBGg==} + + '@rivet-dev/agent-os-common@0.0.260331072558': + resolution: {integrity: sha512-bkMU6yqLxNLoXYA2/f2qRf0JrlqbIiBRKRpP3d+tbmF+VV2wrAY+iLJDHN6Jw+Q8q3CkT6jB5KTnCZkcYEF3RQ==} + + '@rivet-dev/agent-os-coreutils@0.0.260331072558': + resolution: {integrity: sha512-vI/J2MJnpsJQZ7F5DU/udGqGMtliqhTg28lE6XZ/qEGyjUvmEQ9AiV8CaAcX5HrImAUWOofBbTv6/XGKjOhT1w==} + + '@rivet-dev/agent-os-diffutils@0.0.260331072558': + resolution: {integrity: sha512-OiTesXIVqGjEHm/LBvqa20kBu8b+W741EyjRptDKkBju+nQY1hKs0+qFE3vSuBe8N8I7F1GdoXUdwzj8easong==} + + '@rivet-dev/agent-os-fd@0.0.260331072558': + resolution: {integrity: sha512-mTy1EGKm8dqV7gFcOv4JyDoYynRgEKNIxPcdEW9tK2PkQO/OPfnIDPbBVExowIn90R+fpAQl0RbNip9U6lotng==} + + '@rivet-dev/agent-os-file@0.0.260331072558': + resolution: {integrity: sha512-o/MElu1v1M7Z4hod3XO1VAU5doVEeHJSqX9cLVH/Jaw8B7M/xvCMOuQr8cU1nLF6q34ZNCfExOycRWCeFTCxUA==} + + '@rivet-dev/agent-os-findutils@0.0.260331072558': + resolution: {integrity: sha512-Bqc85t0jDJlDj/3afLg4iuzk1qGVHMroRx6HDEWQ2kdishfVMTMUSfac+q5IbqUOaQquhf3ixZzTt+utyc7Qew==} + + '@rivet-dev/agent-os-gawk@0.0.260331072558': + resolution: {integrity: sha512-Y2vW1ox1C0ky0Wij2RLLt3NKRY8/pghoy5CM0fDU91bAnWMyB2Jn7aD/GsluK7h8X8ptwqL0q8E1szASIlJK4A==} + + '@rivet-dev/agent-os-grep@0.0.260331072558': + resolution: {integrity: sha512-VlBOteHBFlAtlKfh1abk0FouJwgtqAXEqQGFU8Gnk6FLNgRss5S0yXBnj9p0qwrmZsKlopljrKJIpJp1GCwJrw==} + + '@rivet-dev/agent-os-gzip@0.0.260331072558': + resolution: {integrity: sha512-id9iOrZFAuaXaFWIIiQEo70Ze/poI7hodSqWfo9ro0sajV0xI7kSp0BbXgS4PvGc9O2+VWcOE2qzqTUnKNlBzQ==} + + '@rivet-dev/agent-os-jq@0.0.260331072558': + resolution: {integrity: sha512-bTDtT8tOMx6M8oQ9GoNo3s/nOxbbiDWLWqUjhJweNNVBp8AMPixJ2kq3jbgiUPepHZtScYRWjQkUggGjb1BgQw==} + + '@rivet-dev/agent-os-ripgrep@0.0.260331072558': + resolution: {integrity: sha512-RX1S6M2T2ghTqkkBHyesm5dFWaQsIrr6EEIlk6UdFYS0G7pcZ11UvTgeirVr2yOvyPRlRrA7FcmupHadQ1wrbw==} + + '@rivet-dev/agent-os-sed@0.0.260331072558': + resolution: {integrity: sha512-i/6ifWCcGE2TEfPWR5ig5tMVKUc0qL0ng59htgS+sqt6zdMaPIllwVztOgXNxMEdFM1TF646e/OkMfMzmYZDCA==} + + '@rivet-dev/agent-os-tar@0.0.260331072558': + resolution: {integrity: sha512-LAa8fm4jombNBQRR42O/57WxY6VA13Uea8JFChh6yPN4VtFP84KI5Kg4/Zzne3YDronGZlgJivznNUH3dtTMzA==} + + '@rivet-dev/agent-os-tree@0.0.260331072558': + resolution: {integrity: sha512-6/+agxo9rKWT7I6i0Fk8oGxKWYyiM0mHsTr2QLB8d0Lz/Bax/pv7ca0E+IwfFaEG1J19LgTnrxf/9YwUSThF7Q==} + + '@rivet-dev/agent-os-unzip@0.0.260331072558': + resolution: {integrity: sha512-W+TkYvcgzaC3+Sc07NckEG45/dVM+k/Ez4KVjIxmwNdrgiMsqPKZvnBIlMX0C6nXjnE6PUlVcdU/Qq+Uvp0mbA==} + + '@rivet-dev/agent-os-yq@0.0.260331072558': + resolution: {integrity: sha512-vUx12neBEWp3v9Lhbkx6eT9pMTdbcmQQEK8kugepfqRiYjus2VpN7KCotaTDSnMPZlqoUhQC7Nu8PMwUgwN5IQ==} + + '@rivet-dev/agent-os-zip@0.0.260331072558': + resolution: {integrity: sha512-v4Y6FvgyLSENCmUUZMUtC2pDIC1dpUq9NGg5ODAWliOOj/8ahFhMa1hTRWb4gKZQjQOevrlJTfBcmJ4FaObgYw==} + '@rollup/rollup-android-arm-eabi@4.60.1': resolution: {integrity: sha512-d6FinEBLdIiK+1uACUttJKfgZREXrF0Qc2SmLII7W2AD8FfiZ9Wjd+rD/iRuf5s5dWrr1GgwXCvPqOuDquOowA==} cpu: [arm] @@ -3102,9 +3202,6 @@ packages: json-schema@0.4.0: resolution: {integrity: sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA==} - jsonc-parser@3.3.1: - resolution: {integrity: sha512-HUgH65KyejrUFPvHFPbqOY0rsFip3Bo5wb4ngvdi1EpCYWUQDC5V+Y7mZws+DLkr4M//zQJoanu1SP+87Dv1oQ==} - jwa@2.0.1: resolution: {integrity: sha512-hRF04fqJIP8Abbkq5NKGN0Bbr3JxlQ+qhZufXVr0DvujKy93ZCbXZMHDL4EOtodSbCWxOqR8MS1tXA5hwqCXDg==} @@ -5162,6 +5259,51 @@ snapshots: '@protobufjs/utf8@1.1.0': {} + '@rivet-dev/agent-os-codex@0.0.260331072558': {} + + '@rivet-dev/agent-os-common@0.0.260331072558': + dependencies: + '@rivet-dev/agent-os-coreutils': 0.0.260331072558 + '@rivet-dev/agent-os-diffutils': 0.0.260331072558 + '@rivet-dev/agent-os-findutils': 0.0.260331072558 + '@rivet-dev/agent-os-gawk': 0.0.260331072558 + '@rivet-dev/agent-os-grep': 0.0.260331072558 + '@rivet-dev/agent-os-gzip': 0.0.260331072558 + '@rivet-dev/agent-os-sed': 0.0.260331072558 + '@rivet-dev/agent-os-tar': 0.0.260331072558 + + '@rivet-dev/agent-os-coreutils@0.0.260331072558': {} + + '@rivet-dev/agent-os-diffutils@0.0.260331072558': {} + + '@rivet-dev/agent-os-fd@0.0.260331072558': {} + + '@rivet-dev/agent-os-file@0.0.260331072558': {} + + '@rivet-dev/agent-os-findutils@0.0.260331072558': {} + + '@rivet-dev/agent-os-gawk@0.0.260331072558': {} + + '@rivet-dev/agent-os-grep@0.0.260331072558': {} + + '@rivet-dev/agent-os-gzip@0.0.260331072558': {} + + '@rivet-dev/agent-os-jq@0.0.260331072558': {} + + '@rivet-dev/agent-os-ripgrep@0.0.260331072558': {} + + '@rivet-dev/agent-os-sed@0.0.260331072558': {} + + '@rivet-dev/agent-os-tar@0.0.260331072558': {} + + '@rivet-dev/agent-os-tree@0.0.260331072558': {} + + '@rivet-dev/agent-os-unzip@0.0.260331072558': {} + + '@rivet-dev/agent-os-yq@0.0.260331072558': {} + + '@rivet-dev/agent-os-zip@0.0.260331072558': {} + '@rollup/rollup-android-arm-eabi@4.60.1': optional: true @@ -5269,7 +5411,7 @@ snapshots: dependencies: better-sqlite3: 12.8.0 - '@secure-exec/nodejs@0.2.1(patch_hash=1b55bfdc13a5329270980574e382c56c27c145390cb03e3eb7bce6c620be4c69)': + '@secure-exec/nodejs@0.2.1(patch_hash=dc7b85660389e571beac82d61aa14936e29803ba733156fb8d3df12504c847dc)': dependencies: '@secure-exec/core': 0.2.1 '@secure-exec/v8': 0.2.1(patch_hash=e542d0ba16b29b7dd6e777f4f16d26ddf6da49ac7f8286912419de975a3a5ead) @@ -6687,8 +6829,6 @@ snapshots: json-schema@0.4.0: {} - jsonc-parser@3.3.1: {} - jwa@2.0.1: dependencies: buffer-equal-constant-time: 1.0.1 @@ -7224,7 +7364,7 @@ snapshots: secure-exec@0.2.1: dependencies: '@secure-exec/core': 0.2.1 - '@secure-exec/nodejs': 0.2.1(patch_hash=1b55bfdc13a5329270980574e382c56c27c145390cb03e3eb7bce6c620be4c69) + '@secure-exec/nodejs': 0.2.1(patch_hash=dc7b85660389e571beac82d61aa14936e29803ba733156fb8d3df12504c847dc) semver@7.7.4: {} diff --git a/registry/agent/pi-rust/bin/libsqlite3.so b/registry/agent/pi-rust/bin/libsqlite3.so new file mode 100755 index 000000000..8113a79c7 Binary files /dev/null and b/registry/agent/pi-rust/bin/libsqlite3.so differ diff --git a/registry/agent/pi-rust/bin/pi-rust b/registry/agent/pi-rust/bin/pi-rust new file mode 100755 index 000000000..39109b60b Binary files /dev/null and b/registry/agent/pi-rust/bin/pi-rust differ diff --git a/registry/agent/pi-rust/bin/pi-rust.manifest.json b/registry/agent/pi-rust/bin/pi-rust.manifest.json new file mode 100644 index 000000000..ad8f17c94 --- /dev/null +++ b/registry/agent/pi-rust/bin/pi-rust.manifest.json @@ -0,0 +1,23 @@ +{ + "sourceDir": "/home/nathan/misc/pi_agent_rust", + "sourceCommit": "6e48d3dc907ceb663cbcf011c1986f8a1c0a036f", + "build": { + "command": "cargo build --release --locked --bin pi --no-default-features", + "profileOverrides": { + "optLevel": "z", + "lto": "fat", + "codegenUnits": "1", + "panic": "abort", + "strip": "symbols", + "debug": "0" + } + }, + "output": { + "path": "/home/nathan/a3/registry/agent/pi-rust/bin/pi-rust", + "sizeBytes": 20884392 + }, + "sqliteLibrary": { + "path": "/home/nathan/a3/registry/agent/pi-rust/bin/libsqlite3.so", + "sizeBytes": 1118856 + } +} diff --git a/registry/agent/pi-rust/package.json b/registry/agent/pi-rust/package.json new file mode 100644 index 000000000..8eb230da0 --- /dev/null +++ b/registry/agent/pi-rust/package.json @@ -0,0 +1,35 @@ +{ + "name": "@rivet-dev/agent-os-pi-rust", + "version": "0.1.0", + "type": "module", + "license": "Apache-2.0", + "main": "./dist/index.js", + "types": "./dist/index.d.ts", + "files": [ + "dist", + "bin" + ], + "bin": { + "pi-rust-acp": "./dist/adapter.js", + "pi-rust": "./bin/pi-rust" + }, + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js", + "default": "./dist/index.js" + } + }, + "scripts": { + "build": "node ./scripts/build-pi-rust.mjs && tsc", + "check-types": "tsc --noEmit" + }, + "dependencies": { + "@agentclientprotocol/sdk": "^0.16.1", + "@rivet-dev/agent-os-core": "workspace:*" + }, + "devDependencies": { + "@types/node": "^22.10.2", + "typescript": "^5.7.2" + } +} diff --git a/registry/agent/pi-rust/pi-rust b/registry/agent/pi-rust/pi-rust new file mode 120000 index 000000000..ad710bc2e --- /dev/null +++ b/registry/agent/pi-rust/pi-rust @@ -0,0 +1 @@ +../../registry/agent/pi-rust \ No newline at end of file diff --git a/registry/agent/pi-rust/scripts/build-pi-rust.mjs b/registry/agent/pi-rust/scripts/build-pi-rust.mjs new file mode 100644 index 000000000..2a6ce5189 --- /dev/null +++ b/registry/agent/pi-rust/scripts/build-pi-rust.mjs @@ -0,0 +1,173 @@ +#!/usr/bin/env node + +import { spawnSync } from "node:child_process"; +import { + mkdirSync, + statSync, + writeFileSync, + copyFileSync, + chmodSync, +} from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { homedir } from "node:os"; +import { fileURLToPath } from "node:url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const packageDir = resolve(__dirname, ".."); +const sourceDir = resolve( + process.env.PI_RUST_SOURCE_DIR ?? join(homedir(), "misc", "pi_agent_rust"), +); +const outputDir = resolve(packageDir, "bin"); +const outputBin = resolve(outputDir, "pi-rust"); +const manifestPath = resolve(outputDir, "pi-rust.manifest.json"); +const sqliteBuildDir = resolve(sourceDir, "target/pi-rust-sqlite"); +const builtSqliteLib = resolve(sqliteBuildDir, "libsqlite3.so"); +const outputSqliteLib = resolve(outputDir, "libsqlite3.so"); +const stubPath = resolve( + sourceDir, + "legacy_pi_mono_code/pi-mono/packages/ai/src/models.generated.ts", +); +const builtBin = resolve(sourceDir, "target/release/pi"); + +function run(command, args, options = {}) { + const result = spawnSync(command, args, { + stdio: "inherit", + ...options, + }); + if (result.status !== 0) { + throw new Error( + `Command failed (${result.status ?? "unknown"}): ${command} ${args.join(" ")}`, + ); + } + return result; +} + +function findBundledSqliteSource() { + const result = spawnSync( + "bash", + [ + "-lc", + [ + "find", + resolve(homedir(), ".cargo/registry/src"), + "-path", + "'*libsqlite3-sys-*/sqlite3/sqlite3.c'", + "|", + "head", + "-n", + "1", + ].join(" "), + ], + { + encoding: "utf8", + }, + ); + if (result.status !== 0 || !result.stdout.trim()) { + throw new Error("Unable to locate bundled sqlite3.c from libsqlite3-sys"); + } + return result.stdout.trim(); +} + +mkdirSync(dirname(stubPath), { recursive: true }); +mkdirSync(sqliteBuildDir, { recursive: true }); +try { + statSync(stubPath); +} catch { + writeFileSync( + stubPath, + [ + "// Stub generated for local Agent OS pi-rust builds when the pi-mono submodule is absent.", + "// Upstream uses the same approach in its release workflow.", + "export const MODELS = {} as const;", + "", + ].join("\n"), + "utf8", + ); +} + +run( + "cc", + [ + "-O2", + "-fPIC", + "-shared", + findBundledSqliteSource(), + "-o", + builtSqliteLib, + "-ldl", + "-lpthread", + ], + { + cwd: sourceDir, + }, +); + +run( + "cargo", + ["build", "--release", "--locked", "--bin", "pi", "--no-default-features"], + { + cwd: sourceDir, + env: { + ...process.env, + LIBRARY_PATH: [sqliteBuildDir, process.env.LIBRARY_PATH] + .filter(Boolean) + .join(":"), + CARGO_PROFILE_RELEASE_OPT_LEVEL: + process.env.CARGO_PROFILE_RELEASE_OPT_LEVEL ?? "z", + CARGO_PROFILE_RELEASE_LTO: process.env.CARGO_PROFILE_RELEASE_LTO ?? "fat", + CARGO_PROFILE_RELEASE_CODEGEN_UNITS: + process.env.CARGO_PROFILE_RELEASE_CODEGEN_UNITS ?? "1", + CARGO_PROFILE_RELEASE_PANIC: + process.env.CARGO_PROFILE_RELEASE_PANIC ?? "abort", + CARGO_PROFILE_RELEASE_STRIP: + process.env.CARGO_PROFILE_RELEASE_STRIP ?? "symbols", + CARGO_PROFILE_RELEASE_DEBUG: + process.env.CARGO_PROFILE_RELEASE_DEBUG ?? "0", + }, + }, +); + +mkdirSync(outputDir, { recursive: true }); +copyFileSync(builtBin, outputBin); +copyFileSync(builtSqliteLib, outputSqliteLib); +chmodSync(outputBin, 0o755); +chmodSync(outputSqliteLib, 0o755); + +const gitHead = spawnSync("git", ["rev-parse", "HEAD"], { + cwd: sourceDir, + encoding: "utf8", +}); + +writeFileSync( + manifestPath, + `${JSON.stringify( + { + sourceDir, + sourceCommit: + gitHead.status === 0 ? gitHead.stdout.trim() : undefined, + build: { + command: "cargo build --release --locked --bin pi --no-default-features", + profileOverrides: { + optLevel: process.env.CARGO_PROFILE_RELEASE_OPT_LEVEL ?? "z", + lto: process.env.CARGO_PROFILE_RELEASE_LTO ?? "fat", + codegenUnits: + process.env.CARGO_PROFILE_RELEASE_CODEGEN_UNITS ?? "1", + panic: process.env.CARGO_PROFILE_RELEASE_PANIC ?? "abort", + strip: process.env.CARGO_PROFILE_RELEASE_STRIP ?? "symbols", + debug: process.env.CARGO_PROFILE_RELEASE_DEBUG ?? "0", + }, + }, + output: { + path: outputBin, + sizeBytes: statSync(outputBin).size, + }, + sqliteLibrary: { + path: outputSqliteLib, + sizeBytes: statSync(outputSqliteLib).size, + }, + }, + null, + 2, + )}\n`, + "utf8", +); diff --git a/registry/agent/pi-rust/src/adapter.ts b/registry/agent/pi-rust/src/adapter.ts new file mode 100644 index 000000000..b5cf7c8a2 --- /dev/null +++ b/registry/agent/pi-rust/src/adapter.ts @@ -0,0 +1,1140 @@ +#!/usr/bin/env node + +import { + type Agent, + AgentSideConnection, + RequestError, + type AuthenticateRequest, + type AuthenticateResponse, + type CancelNotification, + type InitializeRequest, + type InitializeResponse, + type NewSessionRequest, + type NewSessionResponse, + ndJsonStream, + type PromptRequest, + type PromptResponse, + type SessionNotification, + type SetSessionConfigOptionRequest, + type SetSessionConfigOptionResponse, + type SetSessionModeRequest, + type SetSessionModeResponse, +} from "@agentclientprotocol/sdk"; +import { spawn, type ChildProcess } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { + chmodSync, + closeSync, + createReadStream, + existsSync, + mkdirSync, + openSync, + readFileSync, + statSync, + writeSync, +} from "node:fs"; +import { + basename, + isAbsolute, + join, + resolve as resolvePath, +} from "node:path"; + +type JsonRecord = Record; +type ThinkingLevel = "off" | "minimal" | "low" | "medium" | "high" | "xhigh"; + +type RpcModel = { + id: string; + name?: string; + provider: string; + reasoning: boolean; +}; + +type RpcResponse = { + type: "response"; + id?: string; + command: string; + success: boolean; + data?: JsonRecord | null; + error?: string; + errorHints?: unknown; +}; + +type PendingRequest = { + command: string; + resolve: (value: JsonRecord | undefined) => void; + reject: (reason?: unknown) => void; +}; + +type PromptExecution = { + cancelRequested: boolean; + resolve: (value: PromptResponse) => void; + reject: (reason?: unknown) => void; + promise: Promise; +}; + +type PiRustSessionState = { + sessionId: string; + cwd: string; + rpc: RpcProcess; + currentModel: RpcModel | null; + availableModels: RpcModel[]; + thinkingLevel: ThinkingLevel; + activePrompt: PromptExecution | null; + currentToolCalls: Map; + editSnapshots: Map; + lastEmit: Promise; +}; + +let appendSystemPrompt: string | undefined; +const argv = process.argv.slice(2); +for (let i = 0; i < argv.length; i++) { + if (argv[i] === "--append-system-prompt" && i + 1 < argv.length) { + appendSystemPrompt = argv[i + 1]; + i++; + } +} + +function thinkingModesFor(model: RpcModel | null): ThinkingLevel[] { + if (!model?.reasoning) return ["off"]; + const levels: ThinkingLevel[] = [ + "off", + "minimal", + "low", + "medium", + "high", + ]; + if ( + new Set([ + "gpt-5.1-codex-max", + "gpt-5.2", + "gpt-5.4", + "gpt-5.2-codex", + "gpt-5.3-codex", + "gpt-5.3-codex-spark", + ]).has(model.id) + ) { + levels.push("xhigh"); + } + return levels; +} + +function formatThinkingLabel(level: ThinkingLevel): string { + switch (level) { + case "off": + return "Off"; + case "minimal": + return "Minimal"; + case "low": + return "Low"; + case "medium": + return "Medium"; + case "high": + return "High"; + case "xhigh": + return "XHigh"; + } +} + +function modelOptionValue(model: RpcModel): string { + return `${model.provider}/${model.id}`; +} + +function parseModelOptionValue(value: string): { provider: string; modelId: string } { + const slash = value.indexOf("/"); + if (slash <= 0 || slash === value.length - 1) { + throw RequestError.invalidParams({ value }, "model value must be provider/model"); + } + return { + provider: value.slice(0, slash), + modelId: value.slice(slash + 1), + }; +} + +function parseThinkingLevel(value: string): ThinkingLevel { + const normalized = value.trim().toLowerCase(); + if ( + normalized === "off" || + normalized === "minimal" || + normalized === "low" || + normalized === "medium" || + normalized === "high" || + normalized === "xhigh" + ) { + return normalized; + } + throw RequestError.invalidParams({ value }, "unsupported thinking level"); +} + +function parseRpcModel(value: unknown): RpcModel | null { + if (!value || typeof value !== "object") return null; + const record = value as JsonRecord; + const id = typeof record.id === "string" ? record.id : null; + const provider = typeof record.provider === "string" ? record.provider : null; + if (!id || !provider) return null; + return { + id, + name: typeof record.name === "string" ? record.name : undefined, + provider, + reasoning: record.reasoning === true, + }; +} + +function createModes(session: PiRustSessionState) { + return { + currentModeId: session.thinkingLevel, + availableModes: thinkingModesFor(session.currentModel).map((id) => ({ + id, + name: `Thinking: ${formatThinkingLabel(id)}`, + label: formatThinkingLabel(id), + })), + }; +} + +function createConfigOptions(session: PiRustSessionState) { + const options: Array> = [ + { + type: "select", + id: "thought_level", + name: "Thought Level", + category: "thought_level", + currentValue: session.thinkingLevel, + options: thinkingModesFor(session.currentModel).map((value) => ({ + value, + name: formatThinkingLabel(value), + })), + }, + ]; + + if (session.availableModels.length > 0) { + options.unshift({ + type: "select", + id: "model", + name: "Model", + category: "model", + currentValue: session.currentModel + ? modelOptionValue(session.currentModel) + : undefined, + options: session.availableModels.map((model) => ({ + value: modelOptionValue(model), + name: model.name + ? `${model.provider}/${model.id} (${model.name})` + : `${model.provider}/${model.id}`, + })), + }); + } + + return options; +} + +function sendLine(stream: NodeJS.WritableStream, value: JsonRecord): void { + stream.write(`${JSON.stringify(value)}\n`); +} + +class RpcProcess { + private child: ChildProcess; + private stdoutBuffer = ""; + private stderrBuffer = ""; + private closed = false; + private readonly pending = new Map(); + private eventHandler: (event: JsonRecord) => void = () => {}; + private closeHandler: (error: RequestError) => void = () => {}; + + constructor( + command: string, + args: string[], + cwd: string, + libraryPath?: string, + ) { + const childEnv = { ...process.env }; + const effectiveLibraryPath = + libraryPath ?? process.env.PI_RUST_LIBRARY_PATH; + if (effectiveLibraryPath) { + childEnv.LD_LIBRARY_PATH = [ + effectiveLibraryPath, + process.env.LD_LIBRARY_PATH, + ] + .filter(Boolean) + .join(":"); + } + this.child = spawn(command, args, { + cwd, + env: childEnv, + stdio: ["pipe", "pipe", "pipe"], + }); + + this.child.stdout?.on("data", (chunk) => { + this.stdoutBuffer += Buffer.from(chunk).toString("utf8"); + this.processStdoutBuffer(); + }); + this.child.stderr?.on("data", (chunk) => { + this.stderrBuffer += Buffer.from(chunk).toString("utf8"); + }); + this.child.on("error", (error) => { + const requestError = RequestError.internalError( + { cause: error.message, stderr: this.stderr() }, + "failed to spawn pi-rust", + ); + this.failPending(requestError); + this.closed = true; + this.closeHandler(requestError); + }); + this.child.on("exit", (code, signal) => { + if (this.closed) return; + this.closed = true; + const requestError = RequestError.internalError( + { code, signal, stderr: this.stderr() }, + "pi-rust exited unexpectedly", + ); + this.failPending(requestError); + this.closeHandler(requestError); + }); + } + + setEventHandler(handler: (event: JsonRecord) => void): void { + this.eventHandler = handler; + } + + setCloseHandler(handler: (error: RequestError) => void): void { + this.closeHandler = handler; + } + + request(command: string, payload: JsonRecord = {}): Promise { + if (this.closed || !this.child.stdin) { + return Promise.reject( + RequestError.internalError( + { stderr: this.stderr() }, + "pi-rust process is not available", + ), + ); + } + + const id = randomUUID(); + return new Promise((resolve, reject) => { + this.pending.set(id, { command, resolve, reject }); + sendLine(this.child.stdin!, { + id, + type: command, + ...payload, + }); + }); + } + + close(): void { + if (this.closed) return; + this.closed = true; + this.child.kill("SIGTERM"); + this.failPending( + RequestError.internalError( + { stderr: this.stderr() }, + "pi-rust process closed", + ), + ); + } + + stderr(): string { + return this.stderrBuffer.trim(); + } + + private processStdoutBuffer(): void { + while (true) { + const newline = this.stdoutBuffer.indexOf("\n"); + if (newline === -1) break; + const line = this.stdoutBuffer.slice(0, newline).trim(); + this.stdoutBuffer = this.stdoutBuffer.slice(newline + 1); + if (!line) continue; + + let value: JsonRecord; + try { + value = JSON.parse(line) as JsonRecord; + } catch { + continue; + } + + if (value.type === "response") { + this.handleResponse(value as RpcResponse); + continue; + } + + this.eventHandler(value); + } + } + + private handleResponse(response: RpcResponse): void { + if (!response.id) return; + const pending = this.pending.get(response.id); + if (!pending) return; + this.pending.delete(response.id); + + if (response.success) { + pending.resolve( + response.data && typeof response.data === "object" + ? (response.data as JsonRecord) + : undefined, + ); + return; + } + + pending.reject( + RequestError.internalError( + { + command: pending.command, + error: response.error, + errorHints: response.errorHints, + stderr: this.stderr(), + }, + response.error ?? `pi-rust ${pending.command} failed`, + ), + ); + } + + private failPending(error: RequestError): void { + for (const pending of this.pending.values()) { + pending.reject(error); + } + this.pending.clear(); + } +} + +function isModuleOverlayPath(path: string): boolean { + return path.startsWith("/root/node_modules/"); +} + +async function copyFileChunked( + sourcePath: string, + destinationPath: string, +): Promise { + const sourceStat = statSync(sourcePath); + const destinationExists = + existsSync(destinationPath) && + statSync(destinationPath).size === sourceStat.size; + if (destinationExists) return; + + const fd = openSync(destinationPath, "w", 0o755); + try { + await new Promise((resolve, reject) => { + const stream = createReadStream(sourcePath, { + highWaterMark: 1024 * 1024, + }); + stream.on("data", (chunk: string | Buffer) => { + const buffer = Buffer.isBuffer(chunk) + ? chunk + : Buffer.from(chunk); + writeSync(fd, buffer, 0, buffer.length); + }); + stream.on("end", resolve); + stream.on("error", reject); + }); + } finally { + closeSync(fd); + } + + chmodSync(destinationPath, sourceStat.mode & 0o777); +} + +async function materializePiRustLaunch(): Promise<{ + command: string; + libraryPath?: string; +}> { + const command = process.env.PI_RUST_COMMAND ?? "pi-rust"; + const libraryPath = + process.env.PI_RUST_LIBRARY_PATH_HOST ?? + process.env.PI_RUST_LIBRARY_PATH; + + if (!isModuleOverlayPath(command) && !libraryPath) { + return { command, libraryPath }; + } + + const tempDir = `/tmp/agent-os-pi-rust-${process.pid}`; + mkdirSync(tempDir, { recursive: true }); + + let effectiveCommand = command; + if (isModuleOverlayPath(command)) { + effectiveCommand = join(tempDir, basename(command)); + await copyFileChunked(command, effectiveCommand); + } + + let effectiveLibraryPath = libraryPath; + if (libraryPath && isModuleOverlayPath(libraryPath)) { + const sqliteSource = join(libraryPath, "libsqlite3.so"); + const sqliteDestination = join(tempDir, "libsqlite3.so"); + await copyFileChunked(sqliteSource, sqliteDestination); + effectiveLibraryPath = tempDir; + } + + return { + command: effectiveCommand, + libraryPath: effectiveLibraryPath, + }; +} + +class PiRustAgent implements Agent { + private readonly sessions = new Map(); + + constructor(private readonly conn: AgentSideConnection) { + setTimeout(() => { + void this.conn.closed.then(() => { + for (const session of this.sessions.values()) { + session.rpc.close(); + } + this.sessions.clear(); + }); + }, 0); + } + + async initialize( + _params: InitializeRequest, + ): Promise { + return { + protocolVersion: 1, + agentInfo: { + name: "pi-rust-acp", + title: "Pi Rust ACP adapter", + version: "0.1.0", + }, + agentCapabilities: { + tool_calls: true, + text_messages: true, + reasoning: true, + streaming_deltas: true, + session_lifecycle: true, + promptCapabilities: { + audio: false, + embeddedContext: false, + image: false, + }, + sessionCapabilities: { + close: {}, + }, + } as any, + }; + } + + async newSession( + params: NewSessionRequest, + ): Promise { + const launch = await materializePiRustLaunch(); + const args = [ + "--mode", + "rpc", + "--no-session", + "--no-extensions", + "--no-skills", + "--no-prompt-templates", + "--no-themes", + "--hide-cwd-in-prompt", + "--no-migrations", + ]; + if (appendSystemPrompt) { + args.push("--append-system-prompt", appendSystemPrompt); + } + + const rpc = new RpcProcess( + launch.command, + args, + params.cwd, + launch.libraryPath, + ); + const session: PiRustSessionState = { + sessionId: randomUUID(), + cwd: params.cwd, + rpc, + currentModel: null, + availableModels: [], + thinkingLevel: "off", + activePrompt: null, + currentToolCalls: new Map(), + editSnapshots: new Map(), + lastEmit: Promise.resolve(), + }; + rpc.setEventHandler((event) => this.handleRpcEvent(session, event)); + rpc.setCloseHandler((error) => { + session.activePrompt?.reject(error); + }); + + try { + await this.refreshState(session); + } catch (error) { + rpc.close(); + throw error; + } + + this.sessions.set(session.sessionId, session); + + return { + sessionId: session.sessionId, + modes: createModes(session) as any, + configOptions: createConfigOptions(session) as any, + }; + } + + async prompt(params: PromptRequest): Promise { + const session = this.requireSession(params.sessionId); + if (session.activePrompt) { + throw RequestError.invalidRequest( + { sessionId: params.sessionId }, + "session already has an active prompt", + ); + } + + const text = (params.prompt ?? []) + .map((part: { type?: string; text?: string }) => + part.type === "text" ? (part.text ?? "") : "", + ) + .join(""); + + const execution = this.createPromptExecution(); + session.activePrompt = execution; + session.currentToolCalls.clear(); + session.editSnapshots.clear(); + + try { + await session.rpc.request("prompt", { message: text }); + const response = await execution.promise; + await session.lastEmit; + return response; + } catch (error) { + execution.reject(error); + throw error; + } finally { + session.activePrompt = null; + } + } + + async cancel(params: CancelNotification): Promise { + const session = this.requireSession(params.sessionId); + if (session.activePrompt) { + session.activePrompt.cancelRequested = true; + } + await session.rpc.request("abort").catch(() => undefined); + } + + async setSessionMode( + params: SetSessionModeRequest, + ): Promise { + const session = this.requireSession(params.sessionId); + const level = parseThinkingLevel(params.modeId); + await session.rpc.request("set_thinking_level", { level }); + await this.refreshState(session); + await this.conn.sessionUpdate({ + sessionId: session.sessionId, + update: { + sessionUpdate: "current_mode_update", + currentModeId: session.thinkingLevel, + }, + }); + await this.conn.sessionUpdate({ + sessionId: session.sessionId, + update: { + sessionUpdate: "config_option_update", + configOptions: createConfigOptions(session) as any, + }, + }); + return {}; + } + + async setSessionConfigOption( + params: SetSessionConfigOptionRequest, + ): Promise { + const session = this.requireSession(params.sessionId); + if (typeof params.value !== "string") { + throw RequestError.invalidParams( + { value: params.value }, + "pi-rust config options must be strings", + ); + } + + if (params.configId === "model") { + const { provider, modelId } = parseModelOptionValue(params.value); + await session.rpc.request("set_model", { + provider, + modelId, + }); + } else if (params.configId === "thought_level") { + await session.rpc.request("set_thinking_level", { + level: parseThinkingLevel(params.value), + }); + } else { + throw RequestError.invalidParams( + { configId: params.configId }, + "unsupported pi-rust config option", + ); + } + + await this.refreshState(session); + const configOptions = createConfigOptions(session); + await this.conn.sessionUpdate({ + sessionId: session.sessionId, + update: { + sessionUpdate: "current_mode_update", + currentModeId: session.thinkingLevel, + }, + }); + await this.conn.sessionUpdate({ + sessionId: session.sessionId, + update: { + sessionUpdate: "config_option_update", + configOptions: configOptions as any, + }, + }); + return { + configOptions: configOptions as any, + }; + } + + async authenticate( + _params: AuthenticateRequest, + ): Promise { + } + + private requireSession(sessionId: string): PiRustSessionState { + const session = this.sessions.get(sessionId); + if (!session) { + throw RequestError.invalidParams({ sessionId }, "unknown session"); + } + return session; + } + + private createPromptExecution(): PromptExecution { + let resolve!: (value: PromptResponse) => void; + let reject!: (reason?: unknown) => void; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + return { + cancelRequested: false, + resolve, + reject, + promise, + }; + } + + private async refreshState(session: PiRustSessionState): Promise { + const [state, modelsPayload] = await Promise.all([ + session.rpc.request("get_state"), + session.rpc.request("get_available_models"), + ]); + const models = Array.isArray(modelsPayload?.models) + ? modelsPayload.models.map(parseRpcModel).filter(Boolean) + : []; + session.availableModels = models as RpcModel[]; + session.currentModel = parseRpcModel(state?.model); + session.thinkingLevel = + typeof state?.thinkingLevel === "string" + ? parseThinkingLevel(state.thinkingLevel) + : "off"; + if (typeof state?.sessionId === "string" && state.sessionId.trim()) { + if (session.sessionId !== state.sessionId) { + this.sessions.delete(session.sessionId); + session.sessionId = state.sessionId; + this.sessions.set(session.sessionId, session); + } + } + } + + private emit( + session: PiRustSessionState, + update: SessionNotification["update"], + ): Promise { + session.lastEmit = session.lastEmit + .then(() => + this.conn.sessionUpdate({ + sessionId: session.sessionId, + update, + }), + ) + .catch(() => {}); + return session.lastEmit; + } + + private handleRpcEvent(session: PiRustSessionState, event: JsonRecord): void { + switch (event.type) { + case "message_update": { + const assistantMessageEvent = + event.assistantMessageEvent && + typeof event.assistantMessageEvent === "object" + ? (event.assistantMessageEvent as JsonRecord) + : null; + if (!assistantMessageEvent) break; + + if (assistantMessageEvent.type === "text_delta") { + this.emit(session, { + sessionUpdate: "agent_message_chunk", + content: { + type: "text", + text: String(assistantMessageEvent.delta ?? ""), + }, + }); + } else if (assistantMessageEvent.type === "thinking_delta") { + this.emit(session, { + sessionUpdate: "agent_thought_chunk", + content: { + type: "text", + text: String(assistantMessageEvent.delta ?? ""), + }, + }); + } else if ( + assistantMessageEvent.type === "toolcall_start" || + assistantMessageEvent.type === "toolcall_delta" || + assistantMessageEvent.type === "toolcall_end" + ) { + this.handleToolCallMessage(session, assistantMessageEvent); + } + break; + } + + case "tool_execution_start": + this.handleToolExecutionStart( + session, + event as unknown as { + toolCallId: string; + toolName: string; + args: unknown; + }, + ); + break; + + case "tool_execution_update": + this.handleToolExecutionUpdate( + session, + event as unknown as { + toolCallId: string; + partialResult: unknown; + }, + ); + break; + + case "tool_execution_end": + this.handleToolExecutionEnd( + session, + event as unknown as { + toolCallId: string; + result: unknown; + isError: boolean; + }, + ); + break; + + case "agent_end": { + const execution = session.activePrompt; + if (!execution) break; + const error = + typeof event.error === "string" && event.error.trim() + ? event.error + : null; + if (error) { + execution.reject( + RequestError.internalError( + { error, stderr: session.rpc.stderr() }, + error, + ), + ); + } else { + execution.resolve({ + stopReason: execution.cancelRequested ? "cancelled" : "end_turn", + }); + } + break; + } + } + } + + private handleToolCallMessage( + session: PiRustSessionState, + assistantMessageEvent: JsonRecord, + ): void { + const toolCall = + (assistantMessageEvent.toolCall as JsonRecord | undefined) ?? + ( + ((assistantMessageEvent.partial as JsonRecord | undefined)?.content as + | Array + | undefined) + )?.[(assistantMessageEvent.contentIndex as number) ?? 0]; + if (!toolCall) return; + + const toolCallId = String(toolCall.id ?? ""); + const toolName = String(toolCall.name ?? "tool"); + if (!toolCallId) return; + + const rawInput = this.parseToolArgs(toolCall); + const locations = this.toToolCallLocations(session, rawInput); + const existingStatus = session.currentToolCalls.get(toolCallId); + const status = existingStatus ?? "pending"; + + if (!existingStatus) { + session.currentToolCalls.set(toolCallId, "pending"); + this.emit(session, { + sessionUpdate: "tool_call", + toolCallId, + title: toolName, + kind: toToolKind(toolName), + status: "pending", + locations, + rawInput, + }); + } else { + this.emit(session, { + sessionUpdate: "tool_call_update", + toolCallId, + status: status as "pending", + locations, + rawInput, + }); + } + } + + private handleToolExecutionStart( + session: PiRustSessionState, + event: { toolCallId: string; toolName: string; args: unknown }, + ): void { + const { toolCallId, toolName, args } = event; + const rawInput = + args && typeof args === "object" ? (args as JsonRecord) : undefined; + + if (toolName === "edit" && rawInput) { + const path = typeof rawInput.path === "string" ? rawInput.path : undefined; + if (path) { + try { + const absolutePath = isAbsolute(path) + ? path + : resolvePath(session.cwd, path); + const oldText = readFileSync(absolutePath, "utf8"); + session.editSnapshots.set(toolCallId, { path, oldText }); + } catch { + } + } + } + + const locations = this.toToolCallLocations(session, rawInput); + if (!session.currentToolCalls.has(toolCallId)) { + session.currentToolCalls.set(toolCallId, "in_progress"); + this.emit(session, { + sessionUpdate: "tool_call", + toolCallId, + title: toolName, + kind: toToolKind(toolName), + status: "in_progress", + locations, + rawInput, + }); + } else { + session.currentToolCalls.set(toolCallId, "in_progress"); + this.emit(session, { + sessionUpdate: "tool_call_update", + toolCallId, + status: "in_progress", + locations, + rawInput, + }); + } + } + + private handleToolExecutionUpdate( + session: PiRustSessionState, + event: { toolCallId: string; partialResult: unknown }, + ): void { + const text = toolResultToText(event.partialResult); + this.emit(session, { + sessionUpdate: "tool_call_update", + toolCallId: event.toolCallId, + status: "in_progress", + content: text + ? [{ type: "content", content: { type: "text", text } }] + : undefined, + rawOutput: + event.partialResult && typeof event.partialResult === "object" + ? (event.partialResult as JsonRecord) + : undefined, + }); + } + + private handleToolExecutionEnd( + session: PiRustSessionState, + event: { toolCallId: string; result: unknown; isError: boolean }, + ): void { + const text = toolResultToText(event.result); + const snapshot = session.editSnapshots.get(event.toolCallId); + + let content: + | Array< + | { type: "diff"; path: string; oldText: string; newText: string } + | { type: "content"; content: { type: "text"; text: string } } + > + | undefined; + + if (!event.isError && snapshot) { + try { + const absolutePath = isAbsolute(snapshot.path) + ? snapshot.path + : resolvePath(session.cwd, snapshot.path); + const newText = readFileSync(absolutePath, "utf8"); + if (newText !== snapshot.oldText) { + content = [ + { + type: "diff", + path: snapshot.path, + oldText: snapshot.oldText, + newText, + }, + ...(text + ? [ + { + type: "content" as const, + content: { + type: "text" as const, + text, + }, + }, + ] + : []), + ]; + } + } catch { + } + } + + if (!content && text) { + content = [ + { + type: "content", + content: { + type: "text", + text, + }, + }, + ]; + } + + this.emit(session, { + sessionUpdate: "tool_call_update", + toolCallId: event.toolCallId, + status: event.isError ? "failed" : "completed", + content, + rawOutput: + event.result && typeof event.result === "object" + ? (event.result as JsonRecord) + : undefined, + }); + + session.currentToolCalls.delete(event.toolCallId); + session.editSnapshots.delete(event.toolCallId); + } + + private parseToolArgs(toolCall: JsonRecord): JsonRecord | undefined { + if (toolCall.arguments && typeof toolCall.arguments === "object") { + return toolCall.arguments as JsonRecord; + } + const partialArgs = String(toolCall.partialArgs ?? ""); + if (!partialArgs) return undefined; + try { + return JSON.parse(partialArgs) as JsonRecord; + } catch { + return { partialArgs }; + } + } + + private toToolCallLocations( + session: PiRustSessionState, + args: JsonRecord | undefined, + ): Array<{ path: string; line?: number }> | undefined { + const path = typeof args?.path === "string" ? args.path : undefined; + if (!path) return undefined; + return [ + { + path: isAbsolute(path) ? path : resolvePath(session.cwd, path), + }, + ]; + } +} + +function toToolKind(toolName: string): "read" | "edit" | "other" { + if (toolName === "read") return "read"; + if (toolName === "write" || toolName === "edit") return "edit"; + return "other"; +} + +function toolResultToText(result: unknown): string { + if (!result || typeof result !== "object") return ""; + const record = result as JsonRecord; + const content = record.content; + if (Array.isArray(content)) { + const texts = content + .map((item) => + item && + typeof item === "object" && + (item as JsonRecord).type === "text" && + typeof (item as JsonRecord).text === "string" + ? String((item as JsonRecord).text) + : "", + ) + .filter(Boolean); + if (texts.length > 0) return texts.join(""); + } + + const details = + record.details && typeof record.details === "object" + ? (record.details as JsonRecord) + : undefined; + const stdout = + (typeof details?.stdout === "string" ? details.stdout : undefined) ?? + (typeof record.stdout === "string" ? record.stdout : undefined) ?? + (typeof details?.output === "string" ? details.output : undefined) ?? + (typeof record.output === "string" ? record.output : undefined); + const stderr = + (typeof details?.stderr === "string" ? details.stderr : undefined) ?? + (typeof record.stderr === "string" ? record.stderr : undefined); + const exitCode = + (typeof details?.exitCode === "number" ? details.exitCode : undefined) ?? + (typeof record.exitCode === "number" ? record.exitCode : undefined) ?? + (typeof details?.code === "number" ? details.code : undefined) ?? + (typeof record.code === "number" ? record.code : undefined); + + if ( + (typeof stdout === "string" && stdout.trim()) || + (typeof stderr === "string" && stderr.trim()) + ) { + const parts: string[] = []; + if (typeof stdout === "string" && stdout.trim()) parts.push(stdout); + if (typeof stderr === "string" && stderr.trim()) { + parts.push(`stderr:\n${stderr}`); + } + if (typeof exitCode === "number") { + parts.push(`exit code: ${exitCode}`); + } + return parts.join("\n\n").trimEnd(); + } + + return typeof record.error === "string" ? record.error : ""; +} + +const input = new WritableStream({ + write(chunk) { + return new Promise((resolve) => { + process.stdout.write(chunk, () => resolve()); + }); + }, +}); + +const output = new ReadableStream({ + start(controller) { + process.stdin.on("data", (chunk: Buffer) => { + controller.enqueue(new Uint8Array(chunk)); + }); + process.stdin.on("end", () => controller.close()); + process.stdin.on("error", (error: Error) => controller.error(error)); + }, +}); + +const stream = ndJsonStream(input, output); +const connection = new AgentSideConnection( + (conn: AgentSideConnection) => new PiRustAgent(conn), + stream, +); + +process.stdin.resume(); +process.stdin.on("end", () => { + process.exit(0); +}); + +void connection.closed; diff --git a/registry/agent/pi-rust/src/index.ts b/registry/agent/pi-rust/src/index.ts new file mode 100644 index 000000000..1bd2925f8 --- /dev/null +++ b/registry/agent/pi-rust/src/index.ts @@ -0,0 +1,56 @@ +import { defineSoftware } from "@rivet-dev/agent-os-core"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const packageDir = resolve(__dirname, ".."); + +const piRustCommand = defineSoftware({ + name: "pi-rust-command", + type: "tool" as const, + packageDir, + requires: ["@rivet-dev/agent-os-pi-rust"], + bins: { + "pi-rust": "@rivet-dev/agent-os-pi-rust", + }, +}); + +const piRustAgent = defineSoftware({ + name: "pi-rust", + type: "agent" as const, + packageDir, + requires: ["@rivet-dev/agent-os-pi-rust"], + agent: { + id: "pi-rust", + acpAdapter: "@rivet-dev/agent-os-pi-rust", + agentPackage: "@rivet-dev/agent-os-pi-rust", + env: (ctx) => ({ + MALLOC_ARENA_MAX: "1", + PI_RUST_COMMAND: "pi-rust", + PI_RUST_LIBRARY_PATH_HOST: resolve(packageDir, "bin"), + PI_RUST_LIBRARY_PATH: `${ctx.resolvePackage("@rivet-dev/agent-os-pi-rust")}/bin`, + }), + prepareInstructions: async ( + kernel, + _cwd, + additionalInstructions, + opts, + ) => { + const parts: string[] = []; + if (!opts?.skipBase) { + const data = await kernel.readFile("/etc/agentos/instructions.md"); + parts.push(new TextDecoder().decode(data)); + } + if (additionalInstructions) parts.push(additionalInstructions); + if (opts?.toolReference) parts.push(opts.toolReference); + parts.push("---"); + const instructions = parts.join("\n\n"); + if (!instructions) return {}; + return { args: ["--append-system-prompt", instructions] }; + }, + }, +}); + +const piRust = [piRustCommand, piRustAgent] as const; + +export default piRust; diff --git a/registry/agent/pi-rust/tsconfig.json b/registry/agent/pi-rust/tsconfig.json new file mode 100644 index 000000000..bff731325 --- /dev/null +++ b/registry/agent/pi-rust/tsconfig.json @@ -0,0 +1,10 @@ +{ + "extends": "../../../tsconfig.base.json", + "compilerOptions": { + "declaration": true, + "outDir": "./dist", + "rootDir": "./src" + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist"] +} diff --git a/scripts/benchmarks/bench-utils.ts b/scripts/benchmarks/bench-utils.ts index c13be8cc0..068f6581e 100644 --- a/scripts/benchmarks/bench-utils.ts +++ b/scripts/benchmarks/bench-utils.ts @@ -1,8 +1,5 @@ import { AgentOs, type SoftwareInput } from "@rivet-dev/agent-os-core"; import { coreutils } from "@rivet-dev/agent-os-common"; -import claude from "@rivet-dev/agent-os-claude"; -import codex from "@rivet-dev/agent-os-codex-agent"; -import pi from "@rivet-dev/agent-os-pi"; import { LLMock } from "@copilotkit/llmock"; import os from "node:os"; @@ -61,10 +58,36 @@ export interface Workload { settleMs: number; } +async function loadPiSoftware(): Promise { + const { default: pi } = await import("@rivet-dev/agent-os-pi"); + return [pi]; +} + +async function loadPiRustSoftware(): Promise { + const { default: piRust } = await import("@rivet-dev/agent-os-pi-rust"); + return [piRust]; +} + +async function loadClaudeSoftware(): Promise { + const { default: claude } = await import("@rivet-dev/agent-os-claude"); + return [claude]; +} + +async function loadCodexSoftware(): Promise { + const { default: codex } = await import("@rivet-dev/agent-os-codex-agent"); + return [...codex]; +} + +async function resolveSoftware( + software: SoftwareInput[] | (() => Promise), +): Promise { + return typeof software === "function" ? software() : software; +} + function makeAgentSessionWorkload(opts: { agentId: string; description: string; - software: SoftwareInput[]; + loadSoftware: () => Promise; processMarker: string; }): Workload { return { @@ -73,7 +96,7 @@ function makeAgentSessionWorkload(opts: { createVm: async () => { const { port } = await ensureLlmock(); return AgentOs.create({ - software: opts.software, + software: await opts.loadSoftware(), loopbackExemptPorts: [port], }); }, @@ -129,19 +152,25 @@ export const WORKLOADS: Record = { "pi-session": makeAgentSessionWorkload({ agentId: "pi", description: "VM with PI agent session via createSession", - software: [pi], + loadSoftware: loadPiSoftware, processMarker: "agent-os-pi", }), + "pi-rust-session": makeAgentSessionWorkload({ + agentId: "pi-rust", + description: "VM with PI Rust agent session via createSession", + loadSoftware: loadPiRustSoftware, + processMarker: "agent-os-pi-rust", + }), "claude-session": makeAgentSessionWorkload({ agentId: "claude", description: "VM with Claude agent session via createSession", - software: [claude], + loadSoftware: loadClaudeSoftware, processMarker: "agent-os-claude", }), "codex-session": makeAgentSessionWorkload({ agentId: "codex", description: "VM with Codex agent session via createSession", - software: [...codex], + loadSoftware: loadCodexSoftware, processMarker: "agent-os-codex-agent", }), }; @@ -165,7 +194,7 @@ export async function createBenchVm(): Promise { */ export async function createAgentSessionVm( agentId: string, - software: SoftwareInput[], + software: SoftwareInput[] | (() => Promise), ): Promise<{ vm: AgentOs; coldStartMs: number; @@ -173,7 +202,7 @@ export async function createAgentSessionVm( const { url, port } = await ensureLlmock(); const t0 = performance.now(); const vm = await AgentOs.create({ - software, + software: await resolveSoftware(software), loopbackExemptPorts: [port], }); await vm.createSession(agentId, { @@ -188,7 +217,7 @@ export async function createAgentSessionVm( /** Convenience alias for PI agent session. */ export function createPiSessionVm() { - return createAgentSessionVm("pi", [pi]); + return createAgentSessionVm("pi", loadPiSoftware); } // ── Stats and formatting ──────────────────────────────────────────── diff --git a/scripts/benchmarks/coldstart.bench.ts b/scripts/benchmarks/coldstart.bench.ts index f70ff7369..4721834dd 100644 --- a/scripts/benchmarks/coldstart.bench.ts +++ b/scripts/benchmarks/coldstart.bench.ts @@ -4,6 +4,7 @@ * Measures time from AgentOs.create() through workload ready: * --workload=echo Minimal VM + first exec("echo hello") completing * --workload=pi-session VM + createSession("pi") completing (ACP handshake done) + * --workload=pi-rust-session VM + createSession("pi-rust") completing (ACP handshake done) * --workload=claude-session VM + createSession("claude") completing (ACP handshake done) * --workload=codex-session VM + createSession("codex") completing (ACP handshake done) * @@ -12,6 +13,7 @@ * Usage: * npx tsx scripts/benchmarks/coldstart.bench.ts --workload=echo * npx tsx scripts/benchmarks/coldstart.bench.ts --workload=pi-session --iterations=3 + * npx tsx scripts/benchmarks/coldstart.bench.ts --workload=pi-rust-session --iterations=3 * npx tsx scripts/benchmarks/coldstart.bench.ts --workload=claude-session --iterations=3 */ diff --git a/scripts/benchmarks/memory.bench.ts b/scripts/benchmarks/memory.bench.ts index 83a5dc6e0..758a23a4d 100644 --- a/scripts/benchmarks/memory.bench.ts +++ b/scripts/benchmarks/memory.bench.ts @@ -8,6 +8,7 @@ * Workloads: * --workload=sleep (default) Minimal VM with idle Node.js process * --workload=pi-session VM with PI agent session via createSession + * --workload=pi-rust-session VM with PI Rust agent session via createSession * --workload=claude-session VM with Claude agent session via createSession * --workload=codex-session VM with Codex agent session via createSession * @@ -16,6 +17,7 @@ * Usage: * npx tsx --expose-gc benchmarks/memory.bench.ts * npx tsx --expose-gc benchmarks/memory.bench.ts --workload=pi-session --count=1 + * npx tsx --expose-gc benchmarks/memory.bench.ts --workload=pi-rust-session --count=1 * npx tsx --expose-gc benchmarks/memory.bench.ts --workload=claude-session --count=1 */ diff --git a/scripts/benchmarks/run-benchmarks.sh b/scripts/benchmarks/run-benchmarks.sh index 0f731be18..c7fa33740 100755 --- a/scripts/benchmarks/run-benchmarks.sh +++ b/scripts/benchmarks/run-benchmarks.sh @@ -32,6 +32,9 @@ run "coldstart-echo" \ run "memory-pi-session" \ --expose-gc scripts/benchmarks/memory.bench.ts --workload=pi-session --count=3 +run "memory-pi-rust-session" \ + --expose-gc scripts/benchmarks/memory.bench.ts --workload=pi-rust-session --count=3 + run "memory-claude-session" \ --expose-gc scripts/benchmarks/memory.bench.ts --workload=claude-session --count=3