From b7229fc056df438537ed78c7382289ade4eee4ae Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Mon, 22 Jun 2026 16:04:01 +0200 Subject: [PATCH 01/17] test(db): failing repro for nested toArray dropped children (#1501) Adds a failing test reproducing #1501: with three collection levels (products -> priceRanges -> region), when two priceRanges in different parent groups share the same deepest correlation key (regionId === 1), one of the two nested `region` arrays comes back empty. The nested pipeline buffer is shared by reference across per-parent-group states (createPerEntryIncludesStates) and drainNestedBuffers deletes a buffer entry after routing it to the first matching parent group, so the sibling that drains second finds nothing. Note: the minimal repro in the issue does not trigger the bug as written (its dummy `eq(p.id, _.id)` correlation against a single-row anchor with findOne collapses to one product, so the two overlapping siblings never coexist in the output). This test puts both sibling groups in the result so the collision actually occurs. Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/db/tests/query/includes.test.ts | 116 +++++++++++++++++++++++ 1 file changed, 116 insertions(+) diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index 0124ebbeac..635def175c 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -4902,6 +4902,122 @@ describe(`includes subqueries`, () => { expect(data().runs[0].texts).toBe(run1TextsBefore) }) + + // Reproduction for https://github.com/TanStack/db/issues/1501 + // + // 3 collection levels: products -> priceRanges -> region. + // Two priceRanges in DIFFERENT parent groups share the same deepest + // correlation key (regionId === 1): + // - priceRange 1 belongs to product 1 (T-Shirt), regionId 1 + // - priceRange 3 belongs to product 2 (Hoodie), regionId 1 + // Both should resolve their nested `region` to [{ id: 1, name: 'Europe' }]. + // + // Observed bug: the nested region pipeline buffer is shared by reference + // across per-parent-group states (createPerEntryIncludesStates) and + // drainNestedBuffers deletes a buffer entry after routing it to the first + // matching parent group. The sibling that drains second finds nothing, so + // one of the two `region` arrays comes back empty. + it(`resolves nested grandchildren for sibling groups sharing a correlation key`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `repro-1501-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `repro-1501-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 2 }, + { id: 3, productId: 2, regionId: 1 }, // same regionId as priceRange 1 + ], + }), + ) + + const regions = createCollection( + localOnlyCollectionOptions({ + id: `repro-1501-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `repro-1501-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + + await collection.preload() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + regionId: 1, + region: [{ id: 1, name: `Europe` }], + }, + { + id: 2, + regionId: 2, + region: [{ id: 2, name: `North America` }], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { + id: 3, + regionId: 1, + region: [{ id: 1, name: `Europe` }], + }, + ], + }, + ]) + }) }) describe(`many sibling toArray includes with chained derived collections`, () => { From 28a1aa2c4c9d727e0a98b9d7f3ca1e349c40bb36 Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Mon, 22 Jun 2026 16:57:12 +0200 Subject: [PATCH 02/17] fix(db): fan nested toArray includes out to siblings sharing a correlation key (#1501) With 3+ levels of nested toArray includes, when two children in different parent groups shared the same deepest correlation key, only one received the nested rows and the other came back empty. Two compounding causes: - nestedRoutingIndex mapped each nested correlation key to a single parent group (last-writer-wins), and the shared buffer entry was deleted after routing to the first match, so sibling groups sharing the key were dropped. - the nested pipeline does not re-emit already-materialized rows, so a parent group that starts referencing an existing correlation key after the rows were drained (e.g. a sibling inserted after the initial load) saw nothing. Fixes: - nestedRoutingIndex now maps a nested correlation key to a Set of parent groups; drainNestedBuffers fans buffered grandchild changes out to every ready parent group before dropping the buffer entry. - a per-level cumulative snapshot of net-present grandchild rows seeds late-arriving parent groups from what their siblings already received. Routing-index inserts/deletes and parent-delete cleanup are updated to maintain the per-key parent sets. Adds tests covering initial load, a sibling inserted after load, and deleting one of two siblings that share a correlation key. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../nested-toarray-shared-buffer-overlap.md | 9 + .../query/live/collection-config-builder.ts | 211 +++++++++++++++--- packages/db/tests/query/includes.test.ts | 185 +++++++++++++-- 3 files changed, 355 insertions(+), 50 deletions(-) create mode 100644 .changeset/nested-toarray-shared-buffer-overlap.md diff --git a/.changeset/nested-toarray-shared-buffer-overlap.md b/.changeset/nested-toarray-shared-buffer-overlap.md new file mode 100644 index 0000000000..17571d8b27 --- /dev/null +++ b/.changeset/nested-toarray-shared-buffer-overlap.md @@ -0,0 +1,9 @@ +--- +'@tanstack/db': patch +--- + +fix(db): nested `toArray` includes dropping children when sibling parent groups share a correlation key + +With three (or more) levels of nested `toArray` includes, when two children in different parent groups shared the same deepest correlation key, only one of them received the nested rows and the other came back as an empty array. The nested-pipeline routing index mapped each nested correlation key to a single parent group and the shared buffer entry was deleted after routing to the first match, so sibling groups sharing the key were dropped. + +The routing index now maps a nested correlation key to all parent groups that reference it and fans buffered grandchild changes out to each. A per-level snapshot of already-materialized rows also seeds parent groups that start referencing an existing correlation key after the rows were drained (e.g. inserted after the initial load), since the pipeline does not re-emit them. diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index b40e6e431a..8edcb8f688 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -1150,6 +1150,13 @@ function createOrderByComparator( } } +type SnapshotRow = { + value: any + orderByIndex: string | undefined + /** Net multiplicity (inserts − deletes) currently materialized for this row */ + count: number +} + /** * Shared buffer setup for a single nested includes level. * Pipeline output writes into the buffer; during flush the buffer is drained @@ -1159,6 +1166,15 @@ type NestedIncludesSetup = { compilationResult: IncludesCompilationResult /** Shared buffer: nestedCorrelationKey → Map */ buffer: Map>> + /** + * Cumulative net-present grandchild rows per nested correlation key. The + * buffer holds only deltas since the last drain and is cleared once drained, + * so a parent group that starts referencing an existing correlation key + * *after* the rows were already drained (the pipeline does not re-emit them) + * would otherwise see nothing. The snapshot lets such late-arriving parent + * groups be seeded with the rows their siblings already received. + */ + snapshot: Map> /** For 3+ levels of nesting */ nestedSetups?: Array } @@ -1184,8 +1200,14 @@ type IncludesOutputState = { correlationToParentKeys: Map> /** Shared nested pipeline setups (one per nested includes level) */ nestedSetups?: Array - /** nestedCorrelationKey → parentCorrelationKey */ - nestedRoutingIndex?: Map + /** + * nestedCorrelationKey → Set. + * One nested correlation key can map to multiple parent groups when sibling + * parents share the same correlation value (e.g. two price ranges that + * reference the same region), so buffered grandchild changes must fan out to + * every parent group rather than a single one. + */ + nestedRoutingIndex?: Map> /** parentCorrelationKey → Set */ nestedRoutingReverseIndex?: Map> } @@ -1298,6 +1320,7 @@ function setupNestedPipelines( const setup: NestedIncludesSetup = { compilationResult: entry, buffer, + snapshot: new Map(), } // Recursively set up deeper levels @@ -1342,6 +1365,81 @@ function createPerEntryIncludesStates( }) } +/** + * Folds a drained delta into a nested setup's cumulative snapshot, tracking the + * net multiplicity per child row and dropping rows (and empty keys) once their + * net count reaches zero. + */ +function accumulateSnapshot( + setup: NestedIncludesSetup, + nestedCorrelationKey: unknown, + childChanges: Map>, +): void { + let snap = setup.snapshot.get(nestedCorrelationKey) + if (!snap) { + snap = new Map() + setup.snapshot.set(nestedCorrelationKey, snap) + } + + for (const [childKey, changes] of childChanges) { + let row = snap.get(childKey) + if (!row) { + row = { value: changes.value, orderByIndex: changes.orderByIndex, count: 0 } + snap.set(childKey, row) + } + row.count += changes.inserts - changes.deletes + if (changes.inserts > 0) { + row.value = changes.value + if (changes.orderByIndex !== undefined) { + row.orderByIndex = changes.orderByIndex + } + } + if (row.count <= 0) { + snap.delete(childKey) + } + } + + if (snap.size === 0) { + setup.snapshot.delete(nestedCorrelationKey) + } +} + +/** + * Seeds a parent group's per-entry state with the rows already materialized for + * a nested correlation key. Used when a parent group starts referencing a key + * whose rows were drained (and cleared from the buffer) in an earlier flush, so + * the pipeline will not re-emit them. + */ +function seedParentFromSnapshot( + state: IncludesOutputState, + setupIndex: number, + parentCorrelationKey: unknown, + nestedCorrelationKey: unknown, +): void { + const setup = state.nestedSetups![setupIndex]! + const snap = setup.snapshot.get(nestedCorrelationKey) + if (!snap || snap.size === 0) return + + const entry = state.childRegistry.get(parentCorrelationKey) + if (!entry || !entry.includesStates) return + + const entryState = entry.includesStates[setupIndex]! + let byChild = entryState.pendingChildChanges.get(nestedCorrelationKey) + if (!byChild) { + byChild = new Map() + entryState.pendingChildChanges.set(nestedCorrelationKey, byChild) + } + for (const [childKey, row] of snap) { + if (byChild.has(childKey)) continue + byChild.set(childKey, { + deletes: 0, + inserts: row.count, + value: row.value, + orderByIndex: row.orderByIndex, + }) + } +} + /** * Drains shared buffers into per-entry states using the routing index. * Returns the set of parent correlation keys that had changes routed to them. @@ -1356,43 +1454,57 @@ function drainNestedBuffers(state: IncludesOutputState): Set { const toDelete: Array = [] for (const [nestedCorrelationKey, childChanges] of setup.buffer) { - const parentCorrelationKey = + const parentCorrelationKeys = state.nestedRoutingIndex!.get(nestedCorrelationKey) - if (parentCorrelationKey === undefined) { + if (parentCorrelationKeys === undefined || parentCorrelationKeys.size === 0) { // Unroutable — parent not yet seen; keep in buffer continue } - const entry = state.childRegistry.get(parentCorrelationKey) - if (!entry || !entry.includesStates) { - continue - } - - // Route changes into this entry's per-entry state at position i - const entryState = entry.includesStates[i]! - for (const [childKey, changes] of childChanges) { - let byChild = entryState.pendingChildChanges.get(nestedCorrelationKey) - if (!byChild) { - byChild = new Map() - entryState.pendingChildChanges.set(nestedCorrelationKey, byChild) + // A single nested correlation key can map to multiple parent groups when + // sibling parents share the same correlation value. Fan the buffered + // changes out to each ready parent group; only drop the buffer entry once + // it has been routed to at least one parent. + let routedToAny = false + for (const parentCorrelationKey of parentCorrelationKeys) { + const entry = state.childRegistry.get(parentCorrelationKey) + if (!entry || !entry.includesStates) { + continue } - const existing = byChild.get(childKey) - if (existing) { - existing.inserts += changes.inserts - existing.deletes += changes.deletes - if (changes.inserts > 0) { - existing.value = changes.value - if (changes.orderByIndex !== undefined) { - existing.orderByIndex = changes.orderByIndex + + // Route changes into this entry's per-entry state at position i + const entryState = entry.includesStates[i]! + for (const [childKey, changes] of childChanges) { + let byChild = entryState.pendingChildChanges.get(nestedCorrelationKey) + if (!byChild) { + byChild = new Map() + entryState.pendingChildChanges.set(nestedCorrelationKey, byChild) + } + const existing = byChild.get(childKey) + if (existing) { + existing.inserts += changes.inserts + existing.deletes += changes.deletes + if (changes.inserts > 0) { + existing.value = changes.value + if (changes.orderByIndex !== undefined) { + existing.orderByIndex = changes.orderByIndex + } } + } else { + byChild.set(childKey, { ...changes }) } - } else { - byChild.set(childKey, { ...changes }) } + + dirtyCorrelationKeys.add(parentCorrelationKey) + routedToAny = true } - dirtyCorrelationKeys.add(parentCorrelationKey) - toDelete.push(nestedCorrelationKey) + if (routedToAny) { + // Fold the drained delta into the cumulative snapshot so a parent group + // that starts referencing this nested key later can be seeded with it. + accumulateSnapshot(setup, nestedCorrelationKey, childChanges) + toDelete.push(nestedCorrelationKey) + } } for (const key of toDelete) { @@ -1415,7 +1527,8 @@ function updateRoutingIndex( ): void { if (!state.nestedSetups) return - for (const setup of state.nestedSetups) { + for (let i = 0; i < state.nestedSetups.length; i++) { + const setup = state.nestedSetups[i]! for (const [, change] of childChanges) { if (change.inserts > 0) { // Read the nested routing key from the INCLUDES_ROUTING stamp. @@ -1431,13 +1544,33 @@ function updateRoutingIndex( ) if (nestedCorrelationKey != null) { - state.nestedRoutingIndex!.set(nestedRoutingKey, correlationKey) + let parents = state.nestedRoutingIndex!.get(nestedRoutingKey) + if (!parents) { + parents = new Set() + state.nestedRoutingIndex!.set(nestedRoutingKey, parents) + } + const isNewParent = !parents.has(correlationKey) + parents.add(correlationKey) let reverseSet = state.nestedRoutingReverseIndex!.get(correlationKey) if (!reverseSet) { reverseSet = new Set() state.nestedRoutingReverseIndex!.set(correlationKey, reverseSet) } reverseSet.add(nestedRoutingKey) + + // If this parent group is newly associated with a nested key whose + // rows were already drained (and cleared from the buffer) in an + // earlier flush, the pipeline will not re-emit them. Seed this parent + // from the cumulative snapshot so it receives the same rows its + // siblings already have. + if (isNewParent) { + seedParentFromSnapshot( + state, + i, + correlationKey, + nestedRoutingKey, + ) + } } } else if (change.deletes > 0 && change.inserts === 0) { // Remove from routing index @@ -1451,7 +1584,13 @@ function updateRoutingIndex( ) if (nestedCorrelationKey != null) { - state.nestedRoutingIndex!.delete(nestedRoutingKey) + const parents = state.nestedRoutingIndex!.get(nestedRoutingKey) + if (parents) { + parents.delete(correlationKey) + if (parents.size === 0) { + state.nestedRoutingIndex!.delete(nestedRoutingKey) + } + } const reverseSet = state.nestedRoutingReverseIndex!.get(correlationKey) if (reverseSet) { @@ -1479,7 +1618,15 @@ function cleanRoutingIndexOnDelete( const nestedKeys = state.nestedRoutingReverseIndex.get(correlationKey) if (nestedKeys) { for (const nestedKey of nestedKeys) { - state.nestedRoutingIndex!.delete(nestedKey) + // Remove only this parent from the nested key's parent set; other + // sibling parents may still reference the same nested correlation key. + const parents = state.nestedRoutingIndex!.get(nestedKey) + if (parents) { + parents.delete(correlationKey) + if (parents.size === 0) { + state.nestedRoutingIndex!.delete(nestedKey) + } + } } state.nestedRoutingReverseIndex.delete(correlationKey) } diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index 635def175c..ccc0a2c9af 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -4903,20 +4903,10 @@ describe(`includes subqueries`, () => { expect(data().runs[0].texts).toBe(run1TextsBefore) }) - // Reproduction for https://github.com/TanStack/db/issues/1501 - // - // 3 collection levels: products -> priceRanges -> region. - // Two priceRanges in DIFFERENT parent groups share the same deepest - // correlation key (regionId === 1): - // - priceRange 1 belongs to product 1 (T-Shirt), regionId 1 - // - priceRange 3 belongs to product 2 (Hoodie), regionId 1 - // Both should resolve their nested `region` to [{ id: 1, name: 'Europe' }]. - // - // Observed bug: the nested region pipeline buffer is shared by reference - // across per-parent-group states (createPerEntryIncludesStates) and - // drainNestedBuffers deletes a buffer entry after routing it to the first - // matching parent group. The sibling that drains second finds nothing, so - // one of the two `region` arrays comes back empty. + // Three collection levels (products -> priceRanges -> region). When two + // price ranges in different parent groups point at the same deepest + // correlation key (regionId 1, one under each product), each must still + // resolve its own copy of the nested `region` array. it(`resolves nested grandchildren for sibling groups sharing a correlation key`, async () => { type Product = { id: number; title: string } type PriceRange = { id: number; productId: number; regionId: number } @@ -4924,7 +4914,7 @@ describe(`includes subqueries`, () => { const products = createCollection( localOnlyCollectionOptions({ - id: `repro-1501-products`, + id: `shared-corr-products`, getKey: (p) => p.id, initialData: [ { id: 1, title: `T-Shirt` }, @@ -4935,7 +4925,7 @@ describe(`includes subqueries`, () => { const priceRanges = createCollection( localOnlyCollectionOptions({ - id: `repro-1501-price-ranges`, + id: `shared-corr-price-ranges`, getKey: (r) => r.id, initialData: [ { id: 1, productId: 1, regionId: 1 }, @@ -4947,7 +4937,7 @@ describe(`includes subqueries`, () => { const regions = createCollection( localOnlyCollectionOptions({ - id: `repro-1501-regions`, + id: `shared-corr-regions`, getKey: (r) => r.id, initialData: [ { id: 1, name: `Europe` }, @@ -4963,7 +4953,7 @@ describe(`includes subqueries`, () => { ]) const collection = createLiveQueryCollection({ - id: `repro-1501-live`, + id: `shared-corr-live`, query: (q) => q.from({ p: products }).select(({ p }) => ({ id: p.id, @@ -5018,6 +5008,165 @@ describe(`includes subqueries`, () => { }, ]) }) + + // When a second parent group starts referencing a deepest correlation key + // that another group already resolved (the sibling price range is inserted + // after the initial load), the newly inserted group must also receive the + // nested grandchildren. + it(`fans nested grandchildren out to a sibling group inserted after load`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-incremental-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-incremental-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-incremental-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-incremental-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + // Insert a second price range under a different product, sharing regionId 1. + priceRanges.insert({ id: 3, productId: 2, regionId: 1 }) + await new Promise((r) => setTimeout(r, 50)) + + const tree = toTree(collection) + const tshirt = tree.find((p: any) => p.title === `T-Shirt`) + const hoodie = tree.find((p: any) => p.title === `Hoodie`) + expect( + tshirt.priceRanges.find((pr: any) => pr.id === 1).region, + ).toEqual([{ id: 1, name: `Europe` }]) + expect( + hoodie.priceRanges.find((pr: any) => pr.id === 3).region, + ).toEqual([{ id: 1, name: `Europe` }]) + }) + + // When two parent groups share a deepest correlation key and one of them is + // deleted, the surviving group must keep its nested grandchildren. + it(`keeps grandchildren on the surviving sibling after the other is deleted`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-delete-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-delete-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 3, productId: 2, regionId: 1 }, + ], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-delete-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-delete-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + // Delete the Hoodie's price range (the sibling sharing regionId 1). + priceRanges.delete(3) + await new Promise((r) => setTimeout(r, 50)) + + const tree = toTree(collection) + const tshirt = tree.find((p: any) => p.title === `T-Shirt`) + const hoodie = tree.find((p: any) => p.title === `Hoodie`) + expect( + tshirt.priceRanges.find((pr: any) => pr.id === 1).region, + ).toEqual([{ id: 1, name: `Europe` }]) + expect(hoodie.priceRanges).toEqual([]) + }) }) describe(`many sibling toArray includes with chained derived collections`, () => { From 5d5259ec5c50c6bf7c19155aac0d01347a5e9650 Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Wed, 24 Jun 2026 07:56:25 +0000 Subject: [PATCH 03/17] ci: apply automated fixes --- .../query/live/collection-config-builder.ts | 18 ++++++++++-------- packages/db/tests/query/includes.test.ts | 18 +++++++++--------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 8edcb8f688..a8dbe8d113 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -1384,7 +1384,11 @@ function accumulateSnapshot( for (const [childKey, changes] of childChanges) { let row = snap.get(childKey) if (!row) { - row = { value: changes.value, orderByIndex: changes.orderByIndex, count: 0 } + row = { + value: changes.value, + orderByIndex: changes.orderByIndex, + count: 0, + } snap.set(childKey, row) } row.count += changes.inserts - changes.deletes @@ -1456,7 +1460,10 @@ function drainNestedBuffers(state: IncludesOutputState): Set { for (const [nestedCorrelationKey, childChanges] of setup.buffer) { const parentCorrelationKeys = state.nestedRoutingIndex!.get(nestedCorrelationKey) - if (parentCorrelationKeys === undefined || parentCorrelationKeys.size === 0) { + if ( + parentCorrelationKeys === undefined || + parentCorrelationKeys.size === 0 + ) { // Unroutable — parent not yet seen; keep in buffer continue } @@ -1564,12 +1571,7 @@ function updateRoutingIndex( // from the cumulative snapshot so it receives the same rows its // siblings already have. if (isNewParent) { - seedParentFromSnapshot( - state, - i, - correlationKey, - nestedRoutingKey, - ) + seedParentFromSnapshot(state, i, correlationKey, nestedRoutingKey) } } } else if (change.deletes > 0 && change.inserts === 0) { diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index ccc0a2c9af..1d0ce57f08 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -5081,12 +5081,12 @@ describe(`includes subqueries`, () => { const tree = toTree(collection) const tshirt = tree.find((p: any) => p.title === `T-Shirt`) const hoodie = tree.find((p: any) => p.title === `Hoodie`) - expect( - tshirt.priceRanges.find((pr: any) => pr.id === 1).region, - ).toEqual([{ id: 1, name: `Europe` }]) - expect( - hoodie.priceRanges.find((pr: any) => pr.id === 3).region, - ).toEqual([{ id: 1, name: `Europe` }]) + expect(tshirt.priceRanges.find((pr: any) => pr.id === 1).region).toEqual([ + { id: 1, name: `Europe` }, + ]) + expect(hoodie.priceRanges.find((pr: any) => pr.id === 3).region).toEqual([ + { id: 1, name: `Europe` }, + ]) }) // When two parent groups share a deepest correlation key and one of them is @@ -5162,9 +5162,9 @@ describe(`includes subqueries`, () => { const tree = toTree(collection) const tshirt = tree.find((p: any) => p.title === `T-Shirt`) const hoodie = tree.find((p: any) => p.title === `Hoodie`) - expect( - tshirt.priceRanges.find((pr: any) => pr.id === 1).region, - ).toEqual([{ id: 1, name: `Europe` }]) + expect(tshirt.priceRanges.find((pr: any) => pr.id === 1).region).toEqual([ + { id: 1, name: `Europe` }, + ]) expect(hoodie.priceRanges).toEqual([]) }) }) From 7bd39a89deeb451304b5df371a16fba3798a5ec5 Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Wed, 24 Jun 2026 10:30:10 +0200 Subject: [PATCH 04/17] test(db): cover shared-correlation-key fix for collection and materialize includes The nested-includes routing fix is independent of how each level is materialized. Add regression tests proving sibling parent groups that share a deepest correlation key resolve their grandchildren when the nested levels are left as live Collections (no wrapper) and when wrapped with materialize(), mirroring the existing toArray coverage. Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/db/tests/query/includes.test.ts | 184 +++++++++++++++++++++++ 1 file changed, 184 insertions(+) diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index 1d0ce57f08..8e73fa48c2 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -5167,6 +5167,190 @@ describe(`includes subqueries`, () => { ]) expect(hoodie.priceRanges).toEqual([]) }) + + // The shared-correlation-key routing is independent of how each level is + // materialized, so the same guarantee must hold when the nested levels are + // left as live Collections (no toArray/materialize wrapper). + it(`resolves nested grandchildren for sibling groups when levels stay Collections`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-collection-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-collection-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 2 }, + { id: 3, productId: 2, regionId: 1 }, + ], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-collection-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-collection-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + })), + })), + }) + await collection.preload() + + // toTree recursively unwraps the nested live Collections into arrays. + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { id: 1, regionId: 1, region: [{ id: 1, name: `Europe` }] }, + { + id: 2, + regionId: 2, + region: [{ id: 2, name: `North America` }], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { id: 3, regionId: 1, region: [{ id: 1, name: `Europe` }] }, + ], + }, + ]) + }) + + // Same guarantee for materialize(), which produces array/singleton + // snapshots through the same nested-includes routing. + it(`resolves nested grandchildren for sibling groups with materialize()`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-materialize-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-materialize-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 2 }, + { id: 3, productId: 2, regionId: 1 }, + ], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-materialize-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-materialize-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: materialize( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: materialize( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { id: 1, regionId: 1, region: [{ id: 1, name: `Europe` }] }, + { + id: 2, + regionId: 2, + region: [{ id: 2, name: `North America` }], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { id: 3, regionId: 1, region: [{ id: 1, name: `Europe` }] }, + ], + }, + ]) + }) }) describe(`many sibling toArray includes with chained derived collections`, () => { From d91456575a974a0d0fd0e5fa19e0fdcd66f4c104 Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Fri, 26 Jun 2026 14:09:56 +0200 Subject: [PATCH 05/17] test(db): cover late-arrival snapshot re-emit in materialize() variant Add a post-load insert assertion to the shared-correlation-key materialize() regression test: inserting a sibling group that references an already-materialized correlation key must be seeded via the cumulative snapshot without disturbing the existing group's nested rows. Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/db/tests/query/includes.test.ts | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index 8e73fa48c2..91a43e8463 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -5350,6 +5350,24 @@ describe(`includes subqueries`, () => { ], }, ]) + + // Post-load: insert a price range under Hoodie that references regionId 2, + // a correlation key already materialized for T-Shirt at load. This drives + // the late-arrival snapshot re-emit path through materialize() — the new + // sibling group must be seeded with the already-drained North America row + // without disturbing T-Shirt's existing nested rows. + priceRanges.insert({ id: 4, productId: 2, regionId: 2 }) + await new Promise((r) => setTimeout(r, 50)) + + const tree = toTree(collection) + const tshirt = tree.find((p: any) => p.title === `T-Shirt`) + const hoodie = tree.find((p: any) => p.title === `Hoodie`) + expect(tshirt.priceRanges.find((pr: any) => pr.id === 2).region).toEqual([ + { id: 2, name: `North America` }, + ]) + expect(hoodie.priceRanges.find((pr: any) => pr.id === 4).region).toEqual([ + { id: 2, name: `North America` }, + ]) }) }) From 33bfcaa52913c41ce4ed94a3b988aae495bd9c05 Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Fri, 26 Jun 2026 16:45:13 +0200 Subject: [PATCH 06/17] fix(db): refcount nested toArray routes by child key (#1501) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Set routing index introduced by the previous commit collapses multiple child rows in the same parent group that share a nested correlation key into a single route entry. Deleting one such sibling emptied the entry and dropped the whole route, so the surviving sibling stopped receiving grandchild changes (reported by @samwillis). Track the referencing child keys per (nestedKey, parentGroup) so the parent route is only dropped once its last referencing child row is gone. Also fix a routing hole this exposed: an update that changes a child row's nested correlation key (e.g. a price range's regionId) only carries the new key, so the row's stale reference under the old key was never released — a later sibling delete then mis-routed grandchild changes. A per-setup childKey -> nestedKey map records each row's current nested key so updates can release the old reference, scoped per nested setup so a change to one nested include never disturbs another on the same child. Tests cover: same-parent siblings sharing a key with one deleted, an update that changes the nested key followed by a sibling delete, and isolation between two nested includes on the same child row. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../query/live/collection-config-builder.ts | 156 ++++++++-- packages/db/tests/query/includes.test.ts | 282 ++++++++++++++++++ 2 files changed, 408 insertions(+), 30 deletions(-) diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index a8dbe8d113..b204587965 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -1201,15 +1201,30 @@ type IncludesOutputState = { /** Shared nested pipeline setups (one per nested includes level) */ nestedSetups?: Array /** - * nestedCorrelationKey → Set. + * nestedCorrelationKey → (parentCorrelationKey → Set). * One nested correlation key can map to multiple parent groups when sibling * parents share the same correlation value (e.g. two price ranges that * reference the same region), so buffered grandchild changes must fan out to * every parent group rather than a single one. + * + * Within a single parent group, multiple child rows can share the same nested + * correlation key (e.g. two price ranges in the same product both pointing at + * region 1). We track the referencing child keys so the parent group is only + * dropped from the route once its *last* referencing child row is removed — + * deleting one sibling must not strand the survivor. */ - nestedRoutingIndex?: Map> + nestedRoutingIndex?: Map>> /** parentCorrelationKey → Set */ nestedRoutingReverseIndex?: Map> + /** + * Per nested setup: parentCorrelationKey → (childKey → current nestedKey). + * Records which nested key each child row currently routes to, so an update + * that changes a child row's nested correlation key can drop its *previous* + * reference (the update change only carries the new key). Keyed per-setup so a + * change to one nested include never disturbs a different nested include on the + * same child row. + */ + nestedRoutingChildToNested?: Array>> } type ChildCollectionEntry = { @@ -1458,12 +1473,8 @@ function drainNestedBuffers(state: IncludesOutputState): Set { const toDelete: Array = [] for (const [nestedCorrelationKey, childChanges] of setup.buffer) { - const parentCorrelationKeys = - state.nestedRoutingIndex!.get(nestedCorrelationKey) - if ( - parentCorrelationKeys === undefined || - parentCorrelationKeys.size === 0 - ) { + const parentRoutes = state.nestedRoutingIndex!.get(nestedCorrelationKey) + if (parentRoutes === undefined || parentRoutes.size === 0) { // Unroutable — parent not yet seen; keep in buffer continue } @@ -1473,7 +1484,7 @@ function drainNestedBuffers(state: IncludesOutputState): Set { // changes out to each ready parent group; only drop the buffer entry once // it has been routed to at least one parent. let routedToAny = false - for (const parentCorrelationKey of parentCorrelationKeys) { + for (const parentCorrelationKey of parentRoutes.keys()) { const entry = state.childRegistry.get(parentCorrelationKey) if (!entry || !entry.includesStates) { continue @@ -1527,6 +1538,43 @@ function drainNestedBuffers(state: IncludesOutputState): Set { * Maps nested correlation keys to parent correlation keys so that * grandchild changes can be routed to the correct per-entry state. */ +/** + * Removes a single child row's reference to a nested routing key from a parent + * group's route, dropping the parent (and the nested key, and the reverse-index + * entry) once no child row in the group references the key anymore. + */ +function removeChildKeyFromRoute( + state: IncludesOutputState, + correlationKey: unknown, + nestedRoutingKey: unknown, + childKey: unknown, +): void { + const parents = state.nestedRoutingIndex!.get(nestedRoutingKey) + const childKeys = parents?.get(correlationKey) + if (!parents || !childKeys) return + + childKeys.delete(childKey) + // Only drop the parent group from the route once its last child row + // referencing this nested key is gone — a surviving sibling in the same + // parent group must keep receiving grandchild changes. + if (childKeys.size === 0) { + parents.delete(correlationKey) + if (parents.size === 0) { + state.nestedRoutingIndex!.delete(nestedRoutingKey) + } + // The reverse index tracks parent → nested keys at group granularity, so + // only drop the entry when no child row in this parent group references the + // nested key anymore. + const reverseSet = state.nestedRoutingReverseIndex!.get(correlationKey) + if (reverseSet) { + reverseSet.delete(nestedRoutingKey) + if (reverseSet.size === 0) { + state.nestedRoutingReverseIndex!.delete(correlationKey) + } + } + } +} + function updateRoutingIndex( state: IncludesOutputState, correlationKey: unknown, @@ -1534,9 +1582,15 @@ function updateRoutingIndex( ): void { if (!state.nestedSetups) return + // Lazily allocate the per-setup childKey → nestedKey tracking maps. + if (!state.nestedRoutingChildToNested) { + state.nestedRoutingChildToNested = state.nestedSetups.map(() => new Map()) + } + for (let i = 0; i < state.nestedSetups.length; i++) { const setup = state.nestedSetups[i]! - for (const [, change] of childChanges) { + const childToNested = state.nestedRoutingChildToNested[i]! + for (const [childKey, change] of childChanges) { if (change.inserts > 0) { // Read the nested routing key from the INCLUDES_ROUTING stamp. // Must use the composite routing key (not raw correlationKey) to match @@ -1550,14 +1604,38 @@ function updateRoutingIndex( nestedParentContext, ) + // An update (inserts > 0 && deletes > 0) can change a child row's nested + // correlation key (e.g. a price range's regionId changes). The change + // only carries the NEW key, so drop the row's previous reference for + // THIS setup using the recorded mapping before re-routing it. + // + // This relies on the compiler stamping the FULL INCLUDES_ROUTING map on + // every emitted row (one entry per nested include field), so for an + // unrelated nested include the recomputed nestedRoutingKey equals the + // recorded one and the guard below is a no-op — a change to one nested + // include never disturbs the recorded key of another on the same row. + const perParent = childToNested.get(correlationKey) + const prevNestedKey = perParent?.get(childKey) + if (prevNestedKey !== undefined && prevNestedKey !== nestedRoutingKey) { + removeChildKeyFromRoute(state, correlationKey, prevNestedKey, childKey) + perParent!.delete(childKey) + } + if (nestedCorrelationKey != null) { let parents = state.nestedRoutingIndex!.get(nestedRoutingKey) if (!parents) { - parents = new Set() + parents = new Map() state.nestedRoutingIndex!.set(nestedRoutingKey, parents) } - const isNewParent = !parents.has(correlationKey) - parents.add(correlationKey) + let childKeys = parents.get(correlationKey) + // The parent group is "new" for this nested key only when no child row + // in it referenced the key before; that's the case that needs seeding. + const isNewParent = !childKeys || childKeys.size === 0 + if (!childKeys) { + childKeys = new Set() + parents.set(correlationKey, childKeys) + } + childKeys.add(childKey) let reverseSet = state.nestedRoutingReverseIndex!.get(correlationKey) if (!reverseSet) { reverseSet = new Set() @@ -1565,6 +1643,16 @@ function updateRoutingIndex( } reverseSet.add(nestedRoutingKey) + // Record the row's current nested key for this setup so a later update + // that changes it can release the old reference. Reuse perParent when + // it already exists to avoid a second lookup. + let recorded = perParent + if (!recorded) { + recorded = new Map() + childToNested.set(correlationKey, recorded) + } + recorded.set(childKey, nestedRoutingKey) + // If this parent group is newly associated with a nested key whose // rows were already drained (and cleared from the buffer) in an // earlier flush, the pipeline will not re-emit them. Seed this parent @@ -1573,6 +1661,10 @@ function updateRoutingIndex( if (isNewParent) { seedParentFromSnapshot(state, i, correlationKey, nestedRoutingKey) } + } else if (perParent && perParent.size === 0) { + // The row no longer has a nested key (cleared via update) and held no + // others — drop the now-empty per-parent record. + childToNested.delete(correlationKey) } } else if (change.deletes > 0 && change.inserts === 0) { // Remove from routing index @@ -1586,21 +1678,17 @@ function updateRoutingIndex( ) if (nestedCorrelationKey != null) { - const parents = state.nestedRoutingIndex!.get(nestedRoutingKey) - if (parents) { - parents.delete(correlationKey) - if (parents.size === 0) { - state.nestedRoutingIndex!.delete(nestedRoutingKey) - } - } - const reverseSet = - state.nestedRoutingReverseIndex!.get(correlationKey) - if (reverseSet) { - reverseSet.delete(nestedRoutingKey) - if (reverseSet.size === 0) { - state.nestedRoutingReverseIndex!.delete(correlationKey) - } - } + removeChildKeyFromRoute( + state, + correlationKey, + nestedRoutingKey, + childKey, + ) + } + const perParent = childToNested.get(correlationKey) + if (perParent) { + perParent.delete(childKey) + if (perParent.size === 0) childToNested.delete(correlationKey) } } } @@ -1620,8 +1708,9 @@ function cleanRoutingIndexOnDelete( const nestedKeys = state.nestedRoutingReverseIndex.get(correlationKey) if (nestedKeys) { for (const nestedKey of nestedKeys) { - // Remove only this parent from the nested key's parent set; other - // sibling parents may still reference the same nested correlation key. + // The whole parent group is gone, so drop it from the nested key's route + // (along with all the child keys it tracked); other sibling parent groups + // may still reference the same nested correlation key. const parents = state.nestedRoutingIndex!.get(nestedKey) if (parents) { parents.delete(correlationKey) @@ -1632,6 +1721,13 @@ function cleanRoutingIndexOnDelete( } state.nestedRoutingReverseIndex.delete(correlationKey) } + + // Drop the per-setup childKey → nestedKey records for this parent group. + if (state.nestedRoutingChildToNested) { + for (const childToNested of state.nestedRoutingChildToNested) { + childToNested.delete(correlationKey) + } + } } /** diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index 91a43e8463..d03681164d 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -5091,6 +5091,204 @@ describe(`includes subqueries`, () => { // When two parent groups share a deepest correlation key and one of them is // deleted, the surviving group must keep its nested grandchildren. + it(`isolates a nested correlation-key update from a second nested include on the same child`, async () => { + type Product = { id: number; title: string } + type PriceRange = { + id: number + productId: number + regionId: number + currencyId: number + } + type Region = { id: number; name: string } + type Currency = { id: number; code: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `temp2-products`, + getKey: (p) => p.id, + initialData: [{ id: 1, title: `T-Shirt` }], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `temp2-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1, currencyId: 9 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `temp2-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + const currencies = createCollection( + localOnlyCollectionOptions({ + id: `temp2-currencies`, + getKey: (c) => c.id, + initialData: [{ id: 9, code: `EUR` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + currencies.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `temp2-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + currency: toArray( + q + .from({ c: currencies }) + .where(({ c }) => eq(c.id, pr.currencyId)) + .select(({ c }) => ({ id: c.id, code: c.code })), + ), + })), + ), + })), + }) + await collection.preload() + + // Change ONLY regionId; currency must still resolve, and a later currency + // rename must still reach this price range. + priceRanges.update(1, (draft) => { + draft.regionId = 2 + }) + await new Promise((r) => setTimeout(r, 50)) + + currencies.update(9, (draft) => { + draft.code = `USD` + }) + await new Promise((r) => setTimeout(r, 50)) + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + region: [{ id: 2, name: `North America` }], + currency: [{ id: 9, code: `USD` }], + }, + ], + }, + ]) + }) + + it(`keeps the survivor's data when a child changes its nested key then a sibling sharing the old key is deleted`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `temp-upd-products`, + getKey: (p) => p.id, + initialData: [{ id: 1, title: `T-Shirt` }], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `temp-upd-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 1 }, + ], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `temp-upd-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `temp-upd-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + // pr_1 moves from region 1 to region 2 (both pr_1, pr_2 started at region 1) + priceRanges.update(1, (draft) => { + draft.regionId = 2 + }) + await new Promise((r) => setTimeout(r, 50)) + + // delete pr_2 (the remaining referencer of region 1) + priceRanges.delete(2) + await new Promise((r) => setTimeout(r, 50)) + + // rename region 1 — nothing references it anymore, must NOT affect pr_1 + regions.update(1, (draft) => { + draft.name = `Renamed Europe` + }) + await new Promise((r) => setTimeout(r, 50)) + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { id: 1, regionId: 2, region: [{ id: 2, name: `North America` }] }, + ], + }, + ]) + }) + it(`keeps grandchildren on the surviving sibling after the other is deleted`, async () => { type Product = { id: number; title: string } type PriceRange = { id: number; productId: number; regionId: number } @@ -5168,6 +5366,90 @@ describe(`includes subqueries`, () => { expect(hoodie.priceRanges).toEqual([]) }) + it(`keeps routing when one of multiple same-parent siblings sharing a nested key is deleted`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-same-parent-products`, + getKey: (p) => p.id, + initialData: [{ id: 1, title: `T-Shirt` }], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-same-parent-price-ranges`, + getKey: (r) => r.id, + initialData: [ + { id: 1, productId: 1, regionId: 1 }, + { id: 2, productId: 1, regionId: 1 }, + ], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-same-parent-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-same-parent-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.delete(1) + await new Promise((r) => setTimeout(r, 50)) + + regions.update(1, (draft) => { + draft.name = `Renamed Europe` + }) + await new Promise((r) => setTimeout(r, 50)) + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 2, + regionId: 1, + region: [{ id: 1, name: `Renamed Europe` }], + }, + ], + }, + ]) + }) + // The shared-correlation-key routing is independent of how each level is // materialized, so the same guarantee must hold when the nested levels are // left as live Collections (no toArray/materialize wrapper). From ea83de90929d80c32d321ddf41821e3293a04ef7 Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Fri, 26 Jun 2026 14:46:33 +0000 Subject: [PATCH 07/17] ci: apply automated fixes --- packages/db/src/query/live/collection-config-builder.ts | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index b204587965..14d18033c2 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -1617,7 +1617,12 @@ function updateRoutingIndex( const perParent = childToNested.get(correlationKey) const prevNestedKey = perParent?.get(childKey) if (prevNestedKey !== undefined && prevNestedKey !== nestedRoutingKey) { - removeChildKeyFromRoute(state, correlationKey, prevNestedKey, childKey) + removeChildKeyFromRoute( + state, + correlationKey, + prevNestedKey, + childKey, + ) perParent!.delete(childKey) } From 1553c4ef7e70728814e0bd837418f7711cbcc94b Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Mon, 29 Jun 2026 10:06:03 +0200 Subject: [PATCH 08/17] test(db): cover two nested includes on the same child sharing a correlation value Two sibling nested toArray includes on the same child row (region and currency on a price range) that correlate on the same value must resolve independently: re-pointing one include's correlation key must not stop the other include from receiving later changes. Co-Authored-By: Claude Opus 4.8 (1M context) --- packages/db/tests/query/includes.test.ts | 109 +++++++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index d03681164d..8fbc42f8be 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -5091,6 +5091,115 @@ describe(`includes subqueries`, () => { // When two parent groups share a deepest correlation key and one of them is // deleted, the surviving group must keep its nested grandchildren. + it(`resolves two nested includes on the same child independently when they share a correlation value`, async () => { + type Product = { id: number; title: string } + type PriceRange = { + id: number + productId: number + regionId: number + currencyId: number + } + type Region = { id: number; name: string } + type Currency = { id: number; code: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-value-products`, + getKey: (p) => p.id, + initialData: [{ id: 1, title: `T-Shirt` }], + }), + ) + // The price range points at region 1 and currency 1: both nested includes + // correlate on the same value. + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-value-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1, currencyId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-value-regions`, + getKey: (r) => r.id, + initialData: [ + { id: 1, name: `Europe` }, + { id: 2, name: `North America` }, + ], + }), + ) + const currencies = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-value-currencies`, + getKey: (c) => c.id, + initialData: [{ id: 1, code: `EUR` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + currencies.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-value-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + currency: toArray( + q + .from({ c: currencies }) + .where(({ c }) => eq(c.id, pr.currencyId)) + .select(({ c }) => ({ id: c.id, code: c.code })), + ), + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + // Re-point only the region include; the currency include still resolves 1. + priceRanges.update(1, (draft) => { + draft.regionId = 2 + }) + await new Promise((r) => setTimeout(r, 50)) + + // A later currency change must still reach the currency include. + currencies.update(1, (draft) => { + draft.code = `USD` + }) + await new Promise((r) => setTimeout(r, 50)) + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + currency: [{ id: 1, code: `USD` }], + region: [{ id: 2, name: `North America` }], + }, + ], + }, + ]) + }) + it(`isolates a nested correlation-key update from a second nested include on the same child`, async () => { type Product = { id: number; title: string } type PriceRange = { From 492f0852669b81b62cef9cacba58f5261bdf4683 Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Mon, 29 Jun 2026 11:25:30 +0200 Subject: [PATCH 09/17] fix(db): scope nested toArray route store per nested include (#1501) The child-key route refcount tracked the removal decision per nested setup, but the route store itself (nestedRoutingIndex / nestedRoutingReverseIndex) was shared across all sibling nested includes at a level. Because the routing key does not encode the include field, two sibling includes that resolve the same correlation value for the same child row (e.g. a price range with regionId === currencyId) collapse into a single child-key set. Removing one include's reference then emptied the shared route and stranded the other include, so a later change no longer reached it. Make nestedRoutingIndex and nestedRoutingReverseIndex arrays indexed by nested setup, mirroring nestedRoutingChildToNested. Each include owns its own route store, so equal correlation values from different includes can no longer remove each other's routes. removeChildKeyFromRoute now takes the per-setup maps, and parent-delete cleanup iterates every setup. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../query/live/collection-config-builder.ts | 69 +++++++++++-------- 1 file changed, 41 insertions(+), 28 deletions(-) diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 14d18033c2..03d74f00e6 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -907,8 +907,8 @@ export class CollectionConfigBuilder< entry.childCompilationResult.includes, syncState, ) - state.nestedRoutingIndex = new Map() - state.nestedRoutingReverseIndex = new Map() + state.nestedRoutingIndex = state.nestedSetups.map(() => new Map()) + state.nestedRoutingReverseIndex = state.nestedSetups.map(() => new Map()) } return state @@ -1201,7 +1201,7 @@ type IncludesOutputState = { /** Shared nested pipeline setups (one per nested includes level) */ nestedSetups?: Array /** - * nestedCorrelationKey → (parentCorrelationKey → Set). + * Per nested setup: nestedCorrelationKey → (parentCorrelationKey → Set). * One nested correlation key can map to multiple parent groups when sibling * parents share the same correlation value (e.g. two price ranges that * reference the same region), so buffered grandchild changes must fan out to @@ -1212,10 +1212,14 @@ type IncludesOutputState = { * region 1). We track the referencing child keys so the parent group is only * dropped from the route once its *last* referencing child row is removed — * deleting one sibling must not strand the survivor. + * + * Keyed per-setup (one map per nested include field): two sibling includes can + * resolve the same correlation value (regionId === currencyId), so a shared + * store would let one include's cleanup drop the route the other still needs. */ - nestedRoutingIndex?: Map>> - /** parentCorrelationKey → Set */ - nestedRoutingReverseIndex?: Map> + nestedRoutingIndex?: Array>>> + /** Per nested setup: parentCorrelationKey → Set. */ + nestedRoutingReverseIndex?: Array>> /** * Per nested setup: parentCorrelationKey → (childKey → current nestedKey). * Records which nested key each child row currently routes to, so an update @@ -1372,8 +1376,8 @@ function createPerEntryIncludesStates( if (setup.nestedSetups) { state.nestedSetups = setup.nestedSetups - state.nestedRoutingIndex = new Map() - state.nestedRoutingReverseIndex = new Map() + state.nestedRoutingIndex = setup.nestedSetups.map(() => new Map()) + state.nestedRoutingReverseIndex = setup.nestedSetups.map(() => new Map()) } return state @@ -1470,10 +1474,11 @@ function drainNestedBuffers(state: IncludesOutputState): Set { for (let i = 0; i < state.nestedSetups.length; i++) { const setup = state.nestedSetups[i]! + const routeIndex = state.nestedRoutingIndex![i]! const toDelete: Array = [] for (const [nestedCorrelationKey, childChanges] of setup.buffer) { - const parentRoutes = state.nestedRoutingIndex!.get(nestedCorrelationKey) + const parentRoutes = routeIndex.get(nestedCorrelationKey) if (parentRoutes === undefined || parentRoutes.size === 0) { // Unroutable — parent not yet seen; keep in buffer continue @@ -1544,12 +1549,13 @@ function drainNestedBuffers(state: IncludesOutputState): Set { * entry) once no child row in the group references the key anymore. */ function removeChildKeyFromRoute( - state: IncludesOutputState, + routeIndex: Map>>, + reverseIndex: Map>, correlationKey: unknown, nestedRoutingKey: unknown, childKey: unknown, ): void { - const parents = state.nestedRoutingIndex!.get(nestedRoutingKey) + const parents = routeIndex.get(nestedRoutingKey) const childKeys = parents?.get(correlationKey) if (!parents || !childKeys) return @@ -1560,16 +1566,16 @@ function removeChildKeyFromRoute( if (childKeys.size === 0) { parents.delete(correlationKey) if (parents.size === 0) { - state.nestedRoutingIndex!.delete(nestedRoutingKey) + routeIndex.delete(nestedRoutingKey) } // The reverse index tracks parent → nested keys at group granularity, so // only drop the entry when no child row in this parent group references the // nested key anymore. - const reverseSet = state.nestedRoutingReverseIndex!.get(correlationKey) + const reverseSet = reverseIndex.get(correlationKey) if (reverseSet) { reverseSet.delete(nestedRoutingKey) if (reverseSet.size === 0) { - state.nestedRoutingReverseIndex!.delete(correlationKey) + reverseIndex.delete(correlationKey) } } } @@ -1590,6 +1596,8 @@ function updateRoutingIndex( for (let i = 0; i < state.nestedSetups.length; i++) { const setup = state.nestedSetups[i]! const childToNested = state.nestedRoutingChildToNested[i]! + const routeIndex = state.nestedRoutingIndex![i]! + const reverseIndex = state.nestedRoutingReverseIndex![i]! for (const [childKey, change] of childChanges) { if (change.inserts > 0) { // Read the nested routing key from the INCLUDES_ROUTING stamp. @@ -1618,7 +1626,8 @@ function updateRoutingIndex( const prevNestedKey = perParent?.get(childKey) if (prevNestedKey !== undefined && prevNestedKey !== nestedRoutingKey) { removeChildKeyFromRoute( - state, + routeIndex, + reverseIndex, correlationKey, prevNestedKey, childKey, @@ -1627,10 +1636,10 @@ function updateRoutingIndex( } if (nestedCorrelationKey != null) { - let parents = state.nestedRoutingIndex!.get(nestedRoutingKey) + let parents = routeIndex.get(nestedRoutingKey) if (!parents) { parents = new Map() - state.nestedRoutingIndex!.set(nestedRoutingKey, parents) + routeIndex.set(nestedRoutingKey, parents) } let childKeys = parents.get(correlationKey) // The parent group is "new" for this nested key only when no child row @@ -1641,10 +1650,10 @@ function updateRoutingIndex( parents.set(correlationKey, childKeys) } childKeys.add(childKey) - let reverseSet = state.nestedRoutingReverseIndex!.get(correlationKey) + let reverseSet = reverseIndex.get(correlationKey) if (!reverseSet) { reverseSet = new Set() - state.nestedRoutingReverseIndex!.set(correlationKey, reverseSet) + reverseIndex.set(correlationKey, reverseSet) } reverseSet.add(nestedRoutingKey) @@ -1684,7 +1693,8 @@ function updateRoutingIndex( if (nestedCorrelationKey != null) { removeChildKeyFromRoute( - state, + routeIndex, + reverseIndex, correlationKey, nestedRoutingKey, childKey, @@ -1710,21 +1720,24 @@ function cleanRoutingIndexOnDelete( ): void { if (!state.nestedRoutingReverseIndex) return - const nestedKeys = state.nestedRoutingReverseIndex.get(correlationKey) - if (nestedKeys) { + // The whole parent group is gone, so drop it from every nested setup's route + // (along with all the child keys it tracked); other sibling parent groups may + // still reference the same nested correlation key. + for (let i = 0; i < state.nestedRoutingReverseIndex.length; i++) { + const reverseIndex = state.nestedRoutingReverseIndex[i]! + const routeIndex = state.nestedRoutingIndex![i]! + const nestedKeys = reverseIndex.get(correlationKey) + if (!nestedKeys) continue for (const nestedKey of nestedKeys) { - // The whole parent group is gone, so drop it from the nested key's route - // (along with all the child keys it tracked); other sibling parent groups - // may still reference the same nested correlation key. - const parents = state.nestedRoutingIndex!.get(nestedKey) + const parents = routeIndex.get(nestedKey) if (parents) { parents.delete(correlationKey) if (parents.size === 0) { - state.nestedRoutingIndex!.delete(nestedKey) + routeIndex.delete(nestedKey) } } } - state.nestedRoutingReverseIndex.delete(correlationKey) + reverseIndex.delete(correlationKey) } // Drop the per-setup childKey → nestedKey records for this parent group. From 071da99cc6d108f12db73aa494f3be8fb37d73e2 Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Mon, 29 Jun 2026 09:26:57 +0000 Subject: [PATCH 10/17] ci: apply automated fixes --- packages/db/src/query/live/collection-config-builder.ts | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 03d74f00e6..8da38fb1bf 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -908,7 +908,9 @@ export class CollectionConfigBuilder< syncState, ) state.nestedRoutingIndex = state.nestedSetups.map(() => new Map()) - state.nestedRoutingReverseIndex = state.nestedSetups.map(() => new Map()) + state.nestedRoutingReverseIndex = state.nestedSetups.map( + () => new Map(), + ) } return state From 7f12eb0534a6d5c7bc166223bc0949ede9d99d96 Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Mon, 29 Jun 2026 11:45:24 +0200 Subject: [PATCH 11/17] test(db): cover late-arriving sibling nested includes stay reactive --- packages/db/tests/query/includes.test.ts | 122 ++++++++++++++++++++++- 1 file changed, 121 insertions(+), 1 deletion(-) diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index 8fbc42f8be..58684bc53b 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -12,7 +12,11 @@ import { import { createCollection } from '../../src/collection/index.js' import { CleanupQueue } from '../../src/collection/cleanup-queue.js' import { localOnlyCollectionOptions } from '../../src/local-only.js' -import { mockSyncCollectionOptions, stripVirtualProps } from '../utils.js' +import { + flushPromises, + mockSyncCollectionOptions, + stripVirtualProps, +} from '../utils.js' import type { SyncConfig } from '../../src/types.js' type Project = { @@ -5089,6 +5093,122 @@ describe(`includes subqueries`, () => { ]) }) + it(`keeps deeper nested includes reactive for a sibling group added after load`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string; countryId: number } + type Country = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-late-sibling-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-late-sibling-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-late-sibling-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe`, countryId: 1 }], + }), + ) + const countries = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-late-sibling-countries`, + getKey: (c) => c.id, + initialData: [{ id: 1, name: `France` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + countries.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-late-sibling-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + regionId: pr.regionId, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ + id: r.id, + name: r.name, + country: toArray( + q + .from({ c: countries }) + .where(({ c }) => eq(c.id, r.countryId)) + .select(({ c }) => ({ id: c.id, name: c.name })), + ), + })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.insert({ id: 2, productId: 2, regionId: 1 }) + await flushPromises() + + priceRanges.delete(1) + await flushPromises() + + countries.update(1, (draft) => { + draft.name = `Renamed France` + }) + await flushPromises() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { + id: 2, + regionId: 1, + region: [ + { + id: 1, + name: `Europe`, + country: [{ id: 1, name: `Renamed France` }], + }, + ], + }, + ], + }, + ]) + }) + // When two parent groups share a deepest correlation key and one of them is // deleted, the surviving group must keep its nested grandchildren. it(`resolves two nested includes on the same child independently when they share a correlation value`, async () => { From d6144556d728cae658b1ac34a588e68c2c5bb045 Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Mon, 29 Jun 2026 12:09:00 +0200 Subject: [PATCH 12/17] fix(db): clone nested snapshot rows before replay --- .../db/src/query/live/collection-config-builder.ts | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 8da38fb1bf..15a0a98025 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -1386,6 +1386,14 @@ function createPerEntryIncludesStates( }) } +function cloneSnapshotValue(value: T): T { + if (value == null || typeof value !== `object`) { + return value + } + + return (Array.isArray(value) ? [...value] : { ...value }) as T +} + /** * Folds a drained delta into a nested setup's cumulative snapshot, tracking the * net multiplicity per child row and dropping rows (and empty keys) once their @@ -1406,7 +1414,7 @@ function accumulateSnapshot( let row = snap.get(childKey) if (!row) { row = { - value: changes.value, + value: cloneSnapshotValue(changes.value), orderByIndex: changes.orderByIndex, count: 0, } @@ -1414,7 +1422,7 @@ function accumulateSnapshot( } row.count += changes.inserts - changes.deletes if (changes.inserts > 0) { - row.value = changes.value + row.value = cloneSnapshotValue(changes.value) if (changes.orderByIndex !== undefined) { row.orderByIndex = changes.orderByIndex } @@ -1459,7 +1467,7 @@ function seedParentFromSnapshot( byChild.set(childKey, { deletes: 0, inserts: row.count, - value: row.value, + value: cloneSnapshotValue(row.value), orderByIndex: row.orderByIndex, }) } From dd420dae1fe547b55d0a2af67ad8734ef3318544 Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Mon, 29 Jun 2026 12:30:56 +0200 Subject: [PATCH 13/17] test(db): cover recursive nested include sibling fanout --- packages/db/tests/query/includes.test.ts | 363 +++++++++++++++++++++++ 1 file changed, 363 insertions(+) diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index 58684bc53b..a8b7c820ce 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -5209,6 +5209,369 @@ describe(`includes subqueries`, () => { ]) }) + it(`keeps shared nested includes reactive for sibling groups added after load at depth 3`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-3-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-3-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-3-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-depth-3-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ id: r.id, name: r.name })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.insert({ id: 2, productId: 2, regionId: 1 }) + await flushPromises() + + regions.update(1, (draft) => { + draft.name = `Renamed Europe` + }) + await flushPromises() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { id: 1, region: [{ id: 1, name: `Renamed Europe` }] }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { id: 2, region: [{ id: 1, name: `Renamed Europe` }] }, + ], + }, + ]) + }) + + it(`keeps shared nested includes reactive for sibling groups added after load at depth 4`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string; countryId: number } + type Country = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-4-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-4-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-4-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe`, countryId: 1 }], + }), + ) + const countries = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-4-countries`, + getKey: (c) => c.id, + initialData: [{ id: 1, name: `France` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + countries.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-depth-4-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ + id: r.id, + name: r.name, + country: toArray( + q + .from({ c: countries }) + .where(({ c }) => eq(c.id, r.countryId)) + .select(({ c }) => ({ id: c.id, name: c.name })), + ), + })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.insert({ id: 2, productId: 2, regionId: 1 }) + await flushPromises() + + countries.update(1, (draft) => { + draft.name = `Renamed France` + }) + await flushPromises() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + region: [ + { + id: 1, + name: `Europe`, + country: [{ id: 1, name: `Renamed France` }], + }, + ], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { + id: 2, + region: [ + { + id: 1, + name: `Europe`, + country: [{ id: 1, name: `Renamed France` }], + }, + ], + }, + ], + }, + ]) + }) + + it(`keeps shared nested includes reactive for sibling groups added after load at depth 5`, async () => { + type Product = { id: number; title: string } + type PriceRange = { id: number; productId: number; regionId: number } + type Region = { id: number; name: string; countryId: number } + type Country = { id: number; name: string; zoneId: number } + type Zone = { id: number; name: string } + + const products = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-products`, + getKey: (p) => p.id, + initialData: [ + { id: 1, title: `T-Shirt` }, + { id: 2, title: `Hoodie` }, + ], + }), + ) + const priceRanges = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-price-ranges`, + getKey: (r) => r.id, + initialData: [{ id: 1, productId: 1, regionId: 1 }], + }), + ) + const regions = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-regions`, + getKey: (r) => r.id, + initialData: [{ id: 1, name: `Europe`, countryId: 1 }], + }), + ) + const countries = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-countries`, + getKey: (c) => c.id, + initialData: [{ id: 1, name: `France`, zoneId: 1 }], + }), + ) + const zones = createCollection( + localOnlyCollectionOptions({ + id: `shared-corr-depth-5-zones`, + getKey: (z) => z.id, + initialData: [{ id: 1, name: `Eurozone` }], + }), + ) + + await Promise.all([ + products.preload(), + priceRanges.preload(), + regions.preload(), + countries.preload(), + zones.preload(), + ]) + + const collection = createLiveQueryCollection({ + id: `shared-corr-depth-5-live`, + query: (q) => + q.from({ p: products }).select(({ p }) => ({ + id: p.id, + title: p.title, + priceRanges: toArray( + q + .from({ pr: priceRanges }) + .where(({ pr }) => eq(pr.productId, p.id)) + .select(({ pr }) => ({ + id: pr.id, + region: toArray( + q + .from({ r: regions }) + .where(({ r }) => eq(r.id, pr.regionId)) + .select(({ r }) => ({ + id: r.id, + name: r.name, + country: toArray( + q + .from({ c: countries }) + .where(({ c }) => eq(c.id, r.countryId)) + .select(({ c }) => ({ + id: c.id, + name: c.name, + zone: toArray( + q + .from({ z: zones }) + .where(({ z }) => eq(z.id, c.zoneId)) + .select(({ z }) => ({ + id: z.id, + name: z.name, + })), + ), + })), + ), + })), + ), + })), + ), + })), + }) + await collection.preload() + + priceRanges.insert({ id: 2, productId: 2, regionId: 1 }) + await flushPromises() + + zones.update(1, (draft) => { + draft.name = `Renamed Eurozone` + }) + await flushPromises() + + expect(toTree(collection)).toEqual([ + { + id: 1, + title: `T-Shirt`, + priceRanges: [ + { + id: 1, + region: [ + { + id: 1, + name: `Europe`, + country: [ + { + id: 1, + name: `France`, + zone: [{ id: 1, name: `Renamed Eurozone` }], + }, + ], + }, + ], + }, + ], + }, + { + id: 2, + title: `Hoodie`, + priceRanges: [ + { + id: 2, + region: [ + { + id: 1, + name: `Europe`, + country: [ + { + id: 1, + name: `France`, + zone: [{ id: 1, name: `Renamed Eurozone` }], + }, + ], + }, + ], + }, + ], + }, + ]) + }) + // When two parent groups share a deepest correlation key and one of them is // deleted, the surviving group must keep its nested grandchildren. it(`resolves two nested includes on the same child independently when they share a correlation value`, async () => { From 62a6dd11ab56a10112b22d11dfbbf2214f95a46c Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Mon, 29 Jun 2026 10:32:39 +0000 Subject: [PATCH 14/17] ci: apply automated fixes --- packages/db/tests/query/includes.test.ts | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index a8b7c820ce..0d9c05e733 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -5281,16 +5281,12 @@ describe(`includes subqueries`, () => { { id: 1, title: `T-Shirt`, - priceRanges: [ - { id: 1, region: [{ id: 1, name: `Renamed Europe` }] }, - ], + priceRanges: [{ id: 1, region: [{ id: 1, name: `Renamed Europe` }] }], }, { id: 2, title: `Hoodie`, - priceRanges: [ - { id: 2, region: [{ id: 1, name: `Renamed Europe` }] }, - ], + priceRanges: [{ id: 2, region: [{ id: 1, name: `Renamed Europe` }] }], }, ]) }) From 5b4fa2ebc17b109c64e395fa49065d0ab1a1aa2b Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Mon, 29 Jun 2026 12:45:38 +0200 Subject: [PATCH 15/17] fix(db): share nested include routes with shared buffers --- .../query/live/collection-config-builder.ts | 229 ++++++++++-------- 1 file changed, 126 insertions(+), 103 deletions(-) diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 15a0a98025..3f6789b058 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -907,10 +907,6 @@ export class CollectionConfigBuilder< entry.childCompilationResult.includes, syncState, ) - state.nestedRoutingIndex = state.nestedSetups.map(() => new Map()) - state.nestedRoutingReverseIndex = state.nestedSetups.map( - () => new Map(), - ) } return state @@ -1159,6 +1155,21 @@ type SnapshotRow = { count: number } +type NestedRouteIndex = Map< + unknown, + Map>> +> + +type NestedRouteReverseIndex = Map< + IncludesOutputState, + Map> +> + +type NestedRouteChildToNested = Map< + IncludesOutputState, + Map> +> + /** * Shared buffer setup for a single nested includes level. * Pipeline output writes into the buffer; during flush the buffer is drained @@ -1177,6 +1188,14 @@ type NestedIncludesSetup = { * groups be seeded with the rows their siblings already received. */ snapshot: Map> + /** + * Shared route store for this shared buffer. Routes target concrete + * IncludesOutputState instances so one emitted child row can fan out across + * every per-entry state that references the same nested correlation key. + */ + routingIndex: NestedRouteIndex + routingReverseIndex: NestedRouteReverseIndex + routingChildToNested: NestedRouteChildToNested /** For 3+ levels of nesting */ nestedSetups?: Array } @@ -1202,35 +1221,6 @@ type IncludesOutputState = { correlationToParentKeys: Map> /** Shared nested pipeline setups (one per nested includes level) */ nestedSetups?: Array - /** - * Per nested setup: nestedCorrelationKey → (parentCorrelationKey → Set). - * One nested correlation key can map to multiple parent groups when sibling - * parents share the same correlation value (e.g. two price ranges that - * reference the same region), so buffered grandchild changes must fan out to - * every parent group rather than a single one. - * - * Within a single parent group, multiple child rows can share the same nested - * correlation key (e.g. two price ranges in the same product both pointing at - * region 1). We track the referencing child keys so the parent group is only - * dropped from the route once its *last* referencing child row is removed — - * deleting one sibling must not strand the survivor. - * - * Keyed per-setup (one map per nested include field): two sibling includes can - * resolve the same correlation value (regionId === currencyId), so a shared - * store would let one include's cleanup drop the route the other still needs. - */ - nestedRoutingIndex?: Array>>> - /** Per nested setup: parentCorrelationKey → Set. */ - nestedRoutingReverseIndex?: Array>> - /** - * Per nested setup: parentCorrelationKey → (childKey → current nestedKey). - * Records which nested key each child row currently routes to, so an update - * that changes a child row's nested correlation key can drop its *previous* - * reference (the update change only carries the new key). Keyed per-setup so a - * change to one nested include never disturbs a different nested include on the - * same child row. - */ - nestedRoutingChildToNested?: Array>> } type ChildCollectionEntry = { @@ -1342,6 +1332,9 @@ function setupNestedPipelines( compilationResult: entry, buffer, snapshot: new Map(), + routingIndex: new Map(), + routingReverseIndex: new Map(), + routingChildToNested: new Map(), } // Recursively set up deeper levels @@ -1378,8 +1371,6 @@ function createPerEntryIncludesStates( if (setup.nestedSetups) { state.nestedSetups = setup.nestedSetups - state.nestedRoutingIndex = setup.nestedSetups.map(() => new Map()) - state.nestedRoutingReverseIndex = setup.nestedSetups.map(() => new Map()) } return state @@ -1482,54 +1473,60 @@ function drainNestedBuffers(state: IncludesOutputState): Set { if (!state.nestedSetups) return dirtyCorrelationKeys - for (let i = 0; i < state.nestedSetups.length; i++) { - const setup = state.nestedSetups[i]! - const routeIndex = state.nestedRoutingIndex![i]! + for (const setup of state.nestedSetups) { const toDelete: Array = [] for (const [nestedCorrelationKey, childChanges] of setup.buffer) { - const parentRoutes = routeIndex.get(nestedCorrelationKey) - if (parentRoutes === undefined || parentRoutes.size === 0) { + const stateRoutes = setup.routingIndex.get(nestedCorrelationKey) + if (stateRoutes === undefined || stateRoutes.size === 0) { // Unroutable — parent not yet seen; keep in buffer continue } // A single nested correlation key can map to multiple parent groups when - // sibling parents share the same correlation value. Fan the buffered - // changes out to each ready parent group; only drop the buffer entry once - // it has been routed to at least one parent. + // sibling parents share the same correlation value, and at depth 4+ those + // parents may live in different per-entry states. Fan the buffered changes + // out to every ready target before clearing the shared buffer entry. let routedToAny = false - for (const parentCorrelationKey of parentRoutes.keys()) { - const entry = state.childRegistry.get(parentCorrelationKey) - if (!entry || !entry.includesStates) { - continue - } - - // Route changes into this entry's per-entry state at position i - const entryState = entry.includesStates[i]! - for (const [childKey, changes] of childChanges) { - let byChild = entryState.pendingChildChanges.get(nestedCorrelationKey) - if (!byChild) { - byChild = new Map() - entryState.pendingChildChanges.set(nestedCorrelationKey, byChild) + for (const [targetState, parentRoutes] of stateRoutes) { + const targetSetupIndex = targetState.nestedSetups?.indexOf(setup) ?? -1 + if (targetSetupIndex < 0) continue + + for (const parentCorrelationKey of parentRoutes.keys()) { + const entry = targetState.childRegistry.get(parentCorrelationKey) + if (!entry || !entry.includesStates) { + continue } - const existing = byChild.get(childKey) - if (existing) { - existing.inserts += changes.inserts - existing.deletes += changes.deletes - if (changes.inserts > 0) { - existing.value = changes.value - if (changes.orderByIndex !== undefined) { - existing.orderByIndex = changes.orderByIndex + + // Route changes into this entry's per-entry state at the same setup. + const entryState = entry.includesStates[targetSetupIndex]! + for (const [childKey, changes] of childChanges) { + let byChild = + entryState.pendingChildChanges.get(nestedCorrelationKey) + if (!byChild) { + byChild = new Map() + entryState.pendingChildChanges.set(nestedCorrelationKey, byChild) + } + const existing = byChild.get(childKey) + if (existing) { + existing.inserts += changes.inserts + existing.deletes += changes.deletes + if (changes.inserts > 0) { + existing.value = changes.value + if (changes.orderByIndex !== undefined) { + existing.orderByIndex = changes.orderByIndex + } } + } else { + byChild.set(childKey, { ...changes }) } - } else { - byChild.set(childKey, { ...changes }) } - } - dirtyCorrelationKeys.add(parentCorrelationKey) - routedToAny = true + if (targetState === state) { + dirtyCorrelationKeys.add(parentCorrelationKey) + } + routedToAny = true + } } if (routedToAny) { @@ -1559,13 +1556,14 @@ function drainNestedBuffers(state: IncludesOutputState): Set { * entry) once no child row in the group references the key anymore. */ function removeChildKeyFromRoute( - routeIndex: Map>>, - reverseIndex: Map>, + setup: NestedIncludesSetup, + state: IncludesOutputState, correlationKey: unknown, nestedRoutingKey: unknown, childKey: unknown, ): void { - const parents = routeIndex.get(nestedRoutingKey) + const stateRoutes = setup.routingIndex.get(nestedRoutingKey) + const parents = stateRoutes?.get(state) const childKeys = parents?.get(correlationKey) if (!parents || !childKeys) return @@ -1576,16 +1574,23 @@ function removeChildKeyFromRoute( if (childKeys.size === 0) { parents.delete(correlationKey) if (parents.size === 0) { - routeIndex.delete(nestedRoutingKey) + stateRoutes!.delete(state) + if (stateRoutes!.size === 0) { + setup.routingIndex.delete(nestedRoutingKey) + } } // The reverse index tracks parent → nested keys at group granularity, so // only drop the entry when no child row in this parent group references the // nested key anymore. - const reverseSet = reverseIndex.get(correlationKey) + const reverse = setup.routingReverseIndex.get(state) + const reverseSet = reverse?.get(correlationKey) if (reverseSet) { reverseSet.delete(nestedRoutingKey) if (reverseSet.size === 0) { - reverseIndex.delete(correlationKey) + reverse!.delete(correlationKey) + if (reverse!.size === 0) { + setup.routingReverseIndex.delete(state) + } } } } @@ -1598,16 +1603,13 @@ function updateRoutingIndex( ): void { if (!state.nestedSetups) return - // Lazily allocate the per-setup childKey → nestedKey tracking maps. - if (!state.nestedRoutingChildToNested) { - state.nestedRoutingChildToNested = state.nestedSetups.map(() => new Map()) - } - for (let i = 0; i < state.nestedSetups.length; i++) { const setup = state.nestedSetups[i]! - const childToNested = state.nestedRoutingChildToNested[i]! - const routeIndex = state.nestedRoutingIndex![i]! - const reverseIndex = state.nestedRoutingReverseIndex![i]! + let childToNested = setup.routingChildToNested.get(state) + if (!childToNested) { + childToNested = new Map() + setup.routingChildToNested.set(state, childToNested) + } for (const [childKey, change] of childChanges) { if (change.inserts > 0) { // Read the nested routing key from the INCLUDES_ROUTING stamp. @@ -1636,8 +1638,8 @@ function updateRoutingIndex( const prevNestedKey = perParent?.get(childKey) if (prevNestedKey !== undefined && prevNestedKey !== nestedRoutingKey) { removeChildKeyFromRoute( - routeIndex, - reverseIndex, + setup, + state, correlationKey, prevNestedKey, childKey, @@ -1646,10 +1648,15 @@ function updateRoutingIndex( } if (nestedCorrelationKey != null) { - let parents = routeIndex.get(nestedRoutingKey) + let stateRoutes = setup.routingIndex.get(nestedRoutingKey) + if (!stateRoutes) { + stateRoutes = new Map() + setup.routingIndex.set(nestedRoutingKey, stateRoutes) + } + let parents = stateRoutes.get(state) if (!parents) { parents = new Map() - routeIndex.set(nestedRoutingKey, parents) + stateRoutes.set(state, parents) } let childKeys = parents.get(correlationKey) // The parent group is "new" for this nested key only when no child row @@ -1660,10 +1667,15 @@ function updateRoutingIndex( parents.set(correlationKey, childKeys) } childKeys.add(childKey) - let reverseSet = reverseIndex.get(correlationKey) + let reverse = setup.routingReverseIndex.get(state) + if (!reverse) { + reverse = new Map() + setup.routingReverseIndex.set(state, reverse) + } + let reverseSet = reverse.get(correlationKey) if (!reverseSet) { reverseSet = new Set() - reverseIndex.set(correlationKey, reverseSet) + reverse.set(correlationKey, reverseSet) } reverseSet.add(nestedRoutingKey) @@ -1703,8 +1715,8 @@ function updateRoutingIndex( if (nestedCorrelationKey != null) { removeChildKeyFromRoute( - routeIndex, - reverseIndex, + setup, + state, correlationKey, nestedRoutingKey, childKey, @@ -1728,32 +1740,39 @@ function cleanRoutingIndexOnDelete( state: IncludesOutputState, correlationKey: unknown, ): void { - if (!state.nestedRoutingReverseIndex) return + if (!state.nestedSetups) return // The whole parent group is gone, so drop it from every nested setup's route // (along with all the child keys it tracked); other sibling parent groups may // still reference the same nested correlation key. - for (let i = 0; i < state.nestedRoutingReverseIndex.length; i++) { - const reverseIndex = state.nestedRoutingReverseIndex[i]! - const routeIndex = state.nestedRoutingIndex![i]! - const nestedKeys = reverseIndex.get(correlationKey) + for (const setup of state.nestedSetups) { + const reverseIndex = setup.routingReverseIndex.get(state) + const nestedKeys = reverseIndex?.get(correlationKey) if (!nestedKeys) continue for (const nestedKey of nestedKeys) { - const parents = routeIndex.get(nestedKey) + const stateRoutes = setup.routingIndex.get(nestedKey) + const parents = stateRoutes?.get(state) if (parents) { parents.delete(correlationKey) if (parents.size === 0) { - routeIndex.delete(nestedKey) + stateRoutes!.delete(state) + if (stateRoutes!.size === 0) { + setup.routingIndex.delete(nestedKey) + } } } } - reverseIndex.delete(correlationKey) - } + reverseIndex!.delete(correlationKey) + if (reverseIndex!.size === 0) { + setup.routingReverseIndex.delete(state) + } - // Drop the per-setup childKey → nestedKey records for this parent group. - if (state.nestedRoutingChildToNested) { - for (const childToNested of state.nestedRoutingChildToNested) { + const childToNested = setup.routingChildToNested.get(state) + if (childToNested) { childToNested.delete(correlationKey) + if (childToNested.size === 0) { + setup.routingChildToNested.delete(state) + } } } } @@ -2121,6 +2140,10 @@ function hasPendingIncludesChanges( if (state.pendingChildChanges.size > 0) return true if (state.nestedSetups && hasNestedBufferChanges(state.nestedSetups)) return true + for (const entry of state.childRegistry.values()) { + if (entry.includesStates && hasPendingIncludesChanges(entry.includesStates)) + return true + } } return false } From 29b8fb16589d48a87342001a6779f14357b2d751 Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Mon, 29 Jun 2026 10:48:44 +0000 Subject: [PATCH 16/17] ci: apply automated fixes --- packages/db/src/query/live/collection-config-builder.ts | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index 3f6789b058..3120b220c7 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -2141,7 +2141,10 @@ function hasPendingIncludesChanges( if (state.nestedSetups && hasNestedBufferChanges(state.nestedSetups)) return true for (const entry of state.childRegistry.values()) { - if (entry.includesStates && hasPendingIncludesChanges(entry.includesStates)) + if ( + entry.includesStates && + hasPendingIncludesChanges(entry.includesStates) + ) return true } } From e6bb8f3dd7b86bcda516cfa062b03590b94ea287 Mon Sep 17 00:00:00 2001 From: Kevin De Porre Date: Mon, 29 Jun 2026 14:42:43 +0200 Subject: [PATCH 17/17] chore: update nested includes changeset --- .changeset/nested-toarray-shared-buffer-overlap.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.changeset/nested-toarray-shared-buffer-overlap.md b/.changeset/nested-toarray-shared-buffer-overlap.md index 17571d8b27..7e0cc5399b 100644 --- a/.changeset/nested-toarray-shared-buffer-overlap.md +++ b/.changeset/nested-toarray-shared-buffer-overlap.md @@ -2,8 +2,8 @@ '@tanstack/db': patch --- -fix(db): nested `toArray` includes dropping children when sibling parent groups share a correlation key +fix(db): keep deeply nested includes in sync when sibling groups share nested correlation keys -With three (or more) levels of nested `toArray` includes, when two children in different parent groups shared the same deepest correlation key, only one of them received the nested rows and the other came back as an empty array. The nested-pipeline routing index mapped each nested correlation key to a single parent group and the shared buffer entry was deleted after routing to the first match, so sibling groups sharing the key were dropped. +Deeply nested includes could drop or stop updating nested rows when sibling parent groups shared the same nested correlation key, especially when one sibling group was inserted after the initial load. Shared nested pipeline buffers were being drained through route state that was scoped too narrowly, so one branch could consume a buffered update before other branches that referenced the same nested row received it. -The routing index now maps a nested correlation key to all parent groups that reference it and fans buffered grandchild changes out to each. A per-level snapshot of already-materialized rows also seeds parent groups that start referencing an existing correlation key after the rows were drained (e.g. inserted after the initial load), since the pipeline does not re-emit them. +Nested route state is now shared at the same scope as the nested buffer and routes updates to every concrete destination branch before clearing the buffer. Snapshot replay still seeds late-arriving sibling groups with already-materialized rows, and recursive pending-change detection ensures deeper routed updates are flushed back up through the result tree.