perf: parallelize the federation pull command fan-out (#449)#463
Merged
Conversation
Completes #449 — the `perseus memory federation pull` command resolved each subscription's narrative serially (remote subs do a live urlopen). Resolve them in a ThreadPoolExecutor via a `_pull_one` helper that returns (record, progress_line), then emit records + prints serially in subscription order. Output is unchanged; only the I/O is parallelized. (digest + push fan-out shipped in #457/#461.) Tests: test_federation_pull_parallel_preserves_order (3 stubbed remotes report in subscription order); federation suites green (28). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tcconnally
added a commit
that referenced
this pull request
Jun 26, 2026
…k fetch (#472) _resolve_remote_narrative fetched the remote unconditionally first and only read the cache on failure, so a render that includes remote subscriptions paid a synchronous HTTP request every time even when a fresh cached copy existed. cache_ttl_s was effectively dead for the hot path despite the docstring's "using cache when fresh". Read the cache first; _read_remote_cache already returns None once past cache_ttl_s, so a non-None result is within TTL and is served directly. Fetch (and refresh the cache) only on a miss or expiry. The fan-out at the digest, conflicts, and pull call sites is already parallel (#457/#463), so combined with this most renders become zero-network. Tests: a fresh cache serves without calling _fetch_remote_narrative; an expired cache falls through to a fetch. Closes #467
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes #449. The
perseus memory federation pullcommand resolved each subscription's narrative serially (remote subs do a liveurlopen). Resolve them in aThreadPoolExecutorvia a_pull_onehelper returning(record, progress_line), then emit records + prints serially in subscription order. Output unchanged; only the I/O is parallelized. (digest + push fan-out shipped in #457/#461.)Tests:
test_federation_pull_parallel_preserves_order; federation suites green (28).🤖 Generated with Claude Code