fix(gitlab): split large group sync into bounded subgroup fetches

[RitwijParmar]

Fixes #1139

Summary

• replace GitLab group sync's single includeSubgroups: true project query with recursive subgroup traversal
• fetch direct project and subgroup pages separately with bounded perPage: 100 requests
• de-duplicate projects by GitLab project id while walking the group tree

This avoids the expensive GitLab server-side query that can time out for large namespaces such as redhat/centos-stream/*, while preserving recursive subgroup coverage.

Verification

• yarn workspace @sourcebot/backend test gitlab.test.ts
• yarn workspace @sourcebot/backend build
• yarn workspace @sourcebot/backend test

Summary by CodeRabbit

Release Notes

• Bug Fixes

  • Improved GitLab group synchronization for large namespaces by using bounded, offset-based pagination and explicitly traversing group subtrees.
  • Projects are collected in a consistent order and deduplicated, reducing the risk of timeouts during sync.

• Tests

  • Added coverage for recursive group/subgroup traversal and pagination termination, including non-advancing next-page scenarios.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4c58d61d-9ae9-4600-a926-bfb605664a1d

📥 Commits

Reviewing files that changed from the base of the PR and between cbdf353 and f944f96.

📒 Files selected for processing (2)

• packages/backend/src/gitlab.test.ts
• packages/backend/src/gitlab.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• packages/backend/src/gitlab.ts

---

Walkthrough

Replaces the single Groups.allProjects call (with includeSubgroups: true) in getGitLabReposFromConfig with a new exported getGitLabProjectsForGroupTree function that performs explicit offset pagination via a fetchAllGitLabPages helper and iteratively traverses subgroups breadth-first, deduplicating results by project ID.

Changes

GitLab group tree traversal with bounded pagination

Layer / File(s)	Summary
Pagination helper, getGitLabProjectsForGroupTree, and call-site wiring packages/backend/src/gitlab.ts, CHANGELOG.md	Adds GITLAB_PAGE_SIZE, a fetchAllGitLabPages offset-pagination helper, and the exported getGitLabProjectsForGroupTree function that traverses subgroups breadth-first and deduplicates projects by ID. The getGitLabReposFromConfig groups branch replaces Groups.allProjects(..., includeSubgroups: true) with the new function. GroupSchema is added to the GitLab REST import. Changelog records the fix.
Test coverage for subgroup tree traversal and pagination packages/backend/src/gitlab.test.ts	Adds a test that stubs paginated Groups.allProjects and Groups.allSubgroups across nested subgroup levels, validates returned project ordering, asserts all allProjects calls use includeSubgroups: false, and confirms includeSubgroups: true is never invoked. Adds a second test verifying pagination termination when GitLab returns non-advancing next values.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• sourcebot-dev/sourcebot#585: Modifies packages/backend/src/gitlab.ts within getGitLabReposFromConfig, specifically the GitLab client initialization and group-project fetching logic that this PR also changes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly describes the main change: replacing large group sync with bounded subgroup fetches to address timeouts.
Linked Issues check	✅ Passed	The PR implements pagination and subgroup traversal with bounded requests to replace the timeout-causing includeSubgroups query, directly addressing issue #1139's requirement.
Out of Scope Changes check	✅ Passed	All changes align with the stated objective: pagination helper, subgroup traversal function, and integration into config groups handling directly address the timeout issue.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

packages/backend/src/gitlab.ts (1)

56-60: ⚡ Quick win

Avoid Array.shift() in the traversal queue for large group trees.

Line 60 makes dequeue O(n), so walking large subgroup trees becomes O(n²). This path is exactly the large-namespace hot path this PR is optimizing.

♻️ Proposed refactor

 export const getGitLabProjectsForGroupTree = async (
     api: GitLabApi,
     rootGroup: string,
 ): Promise<ProjectSchema[]> => {
     const projectsById = new Map<number, ProjectSchema>();
     const groupsToVisit: Array<string | number> = [rootGroup];
+    let queueIndex = 0;
     const visitedGroups = new Set<string>();

-    while (groupsToVisit.length > 0) {
-        const group = groupsToVisit.shift()!;
+    while (queueIndex < groupsToVisit.length) {
+        const group = groupsToVisit[queueIndex++]!;
         const groupKey = String(group);
         if (visitedGroups.has(groupKey)) {
             continue;
         }
         visitedGroups.add(groupKey);

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/backend/src/gitlab.ts` around lines 56 - 60, The traversal queue is
using Array.shift() which performs in O(n) time, causing the overall group tree
traversal to be O(n²) for large subgroup trees. Replace the shift-based
dequeuing with an index-based approach by adding an index variable to track the
current position in the groupsToVisit array, then access elements via that index
instead of calling shift(). This will make dequeue operations O(1) and improve
the overall performance of the traversal logic.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/backend/src/gitlab.ts`:
- Around line 31-46: The pagination loop in the while block trusts that
response.paginationInfo?.next always contains a new page value that advances
forward, but if the upstream response returns the same page or a regressing page
number, the loop will never terminate and hang the worker. Add a guard condition
after checking if nextPage exists to verify that nextPage is actually greater
than the current page value, and if it is not advancing (i.e., nextPage is less
than or equal to page), break out of the loop to prevent an infinite sync hang.

---

Nitpick comments:
In `@packages/backend/src/gitlab.ts`:
- Around line 56-60: The traversal queue is using Array.shift() which performs
in O(n) time, causing the overall group tree traversal to be O(n²) for large
subgroup trees. Replace the shift-based dequeuing with an index-based approach
by adding an index variable to track the current position in the groupsToVisit
array, then access elements via that index instead of calling shift(). This will
make dequeue operations O(1) and improve the overall performance of the
traversal logic.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: cd5ed975-df6a-4aad-bfdb-fcd6e0b5eda4

📥 Commits

Reviewing files that changed from the base of the PR and between 9320065 and cbdf353.

📒 Files selected for processing (3)

• CHANGELOG.md
• packages/backend/src/gitlab.test.ts
• packages/backend/src/gitlab.ts