Add Backpack support to Stack

[philippedev101]

Addresses #2540.

This PR adds support for GHC's Backpack module system, the feature request that has been open since August 2016. After this change, Stack can build projects that use signatures, mixins, and cross-package mixin linking, which previously only worked under cabal-install.

Background

Backpack lets you write a library against an abstract interface (a signature) and have the consumer decide which concrete implementation to plug in. The compiler recompiles the library for each implementation, so there's no runtime overhead. When the signature and the implementation live in the same package (using sub-libraries), this is "private Backpack" and has always worked in Stack. The hard part is cross-package Backpack, where the signature lives in one package and the implementation in another. This requires the build tool to create extra instantiation build steps, which Stack didn't do before.

What changed

The work breaks down into three layers:

Per-component build plan. The build plan is now keyed by ComponentKey (package name + component) instead of just package name. This was a prerequisite: Backpack instantiation tasks need their own entries in the plan alongside the regular library task for the same package. This also lets Stack detect intra-package sub-library dependencies (the Backpack pattern) and avoid splitting those packages into independent component tasks, which would break the build order that Cabal expects.

Cross-package instantiation. When a consumer package uses mixins to depend on an indefinite package, Stack now scans the consumer's dependencies, resolves which modules fill which signatures, and creates CInst (component instantiation) tasks. Each CInst task runs Setup configure --instantiate-with=Sig=impl-pkg:Module followed by Setup build in its own inst-<hash> build directory. The hash is derived from the signature-to-implementation mapping, so different instantiations of the same package get different build artifacts.

This handles the full range of Backpack patterns: default renaming (name identity), explicit ModuleRenaming, HidingRenaming (propagating unfilled holes), multiple instantiations of the same indefinite package with different implementations, transitive chains where an indefinite package depends on another indefinite package, sub-library signatures and implementations, and indefinite packages from Hackage or snapshots (not just local packages). CInst tasks also get their own config cache entries, precompiled cache support, and haddock generation.

Documentation and changelog. A new topic page at doc/topics/backpack.md introduces what Backpack is, walks through its features (signatures, mixin linking, renaming, multiple instantiations, sub-libraries, reexported modules, Template Haskell restrictions), and then explains how Stack supports it, including what the build output looks like and what the limitations are. The page is linked from mkdocs.yml under Topics, and cross-referenced from the package description tutorial (mentioning the signatures field) and the multi-package projects tutorial (mentioning Backpack as a use case for multi-package setups). A major changes entry has been added to ChangeLog.md under the unreleased section.

Testing

103 unit tests in ConstructPlanSpec cover the instantiation logic: signature resolution, renaming, hiding, deduplication, transitive chains, multiple instantiations, sub-library mixins, ADRFound (installed) packages, config cache round-trips, and various warning/error paths.

8 integration tests build real multi-package Backpack projects end-to-end: private Backpack, sub-library dependencies, cross-package with default renaming, explicit renaming, sub-library mixins, transitive chains, and multiple instantiations.

The full existing test suite (814 tests) continues to pass. The two pre-existing Stack.Config/loadConfig failures are environmental (TLS certificates) and unrelated.

What's not covered

requires hiding that actually hides signatures does not create a partial instantiation. Cabal requires a closing substitution, so the hidden signatures propagate as holes to the consumer. Template Haskell in indefinite packages doesn't work, but that's a GHC limitation that affects cabal-install equally.

[mpilgrem]

@philippedev101, thanks! There is a lot to read but, in advance of that, in implementing component-based builds, did you encounter, and overcome, the performance issue that was blocking @theobat making progress at:

• Component-based builds #6356
• Component-based builds #6356 (comment)

[philippedev101]

@mpilgrem Thanks for the pointer to #6356 and @theobat's work. After going through the discussion there and the branches on their fork in some detail: short answer, this PR takes a different architectural approach that sidesteps the performance problem, at the cost of not solving the broader set of issues that full per-component builds would address.

How the two approaches differ

@theobat's approach was to make Stack build each component of a package as a separate Setup configure / Setup build cycle. So for a package with a library and an executable, Stack would call the Setup process twice instead of once. This is what cabal-install does and it's the "correct" long-term direction, but as @theobat found, it introduces a 30-40% overhead on integration tests because each subprocess invocation has a fixed cost that adds up.

The approach in this PR is narrower. The build plan is keyed per-component using a ComponentKey type (package name + component), but the actual Setup invocations for regular packages are unchanged: one configure, one build, same as before. The only additional subprocess calls happen for Backpack instantiation tasks (CInst), and those only exist when a project actually uses cross-package Backpack. Projects that don't use Backpack see zero additional subprocess invocations and no performance difference.

What this does not solve

Full per-component execution would address several long-standing issues that this approach leaves untouched:

Unnecessary recompilation when switching between stack build and stack test (#2800). Right now, if you build just the library and then run tests, Stack reconfigures the entire package with test dependencies included. This causes the library to be unregistered and recompiled from scratch even though nothing in it changed. Per-component execution would configure and build the test suite separately, leaving the library alone.

Rebuilding components that aren't dependencies of the target (#6569). If you ask for stack build foo:exe:my-exe but a sub-library that the exe doesn't depend on has changed, Stack still rebuilds because it configures and builds the whole package. Per-component execution would only touch the components in the actual dependency chain.

Parallel builds within a package (#4391). A package with five independent executables currently builds them sequentially (Cabal handles the ordering internally). Per-component execution would let Stack schedule them in parallel, the same way it parallelizes across packages.

Component-level cycle resolution (#2583). When package A's library depends on package B's library, and B's test suite depends on A's library, that looks like a cycle at the package level. At the component level it's not: build A:lib, then B:lib, then both test suites. Per-component execution could handle this.

Where this PR sits

There are roughly three points on the spectrum:

1. No Backpack support at all (current Stack).
2. Backpack support that works correctly with no performance penalty for non-Backpack projects, but doesn't tackle the broader per-component issues (this PR).
3. Full per-component builds that solve all of the above plus Backpack, but with the subprocess overhead problem that needs to be addressed first.

This PR lands at point 2. It gets cross-package Backpack working today without regressing anything for existing users.

Transitioning to full per-component builds later

The ComponentKey plan infrastructure that this PR introduces is a step toward point 3. The plan is already split per-component, dependency edges between components within the same package are already tracked (intraPackageDeps), and the action scheduler already generates one action per component key. The remaining work to reach full per-component execution would be:

• Per-component Setup configure: thread the component name from ComponentKey into the Setup configure call so it targets a single component. Cabal has supported this since 2.0 and the types are already in place.
• Per-component config cache: key the config cache by ComponentKey instead of PackageIdentifier, so that configuring the test suite doesn't invalidate the library's cache. This PR already does this for CInst tasks (ConfigCacheTypeInstantiation), the same pattern extends to regular components.
• Per-component dependency flags: scope the --dependency flags passed to Setup configure to just the dependencies of the component being built, rather than the union of all component dependencies.
• Per-component Task creation: currently multiple ComponentKey entries in the plan share the same Task object. For true per-component execution, each component would have its own Task with its own dependency sets.

None of these changes require rethinking the plan-level architecture. They're about threading component information through the execution layer. And they're independent of the subprocess performance question, which could be tackled separately (the in-process Distribution.Simple.defaultMainArgs approach discussed in #6356 looks promising for build-type: Simple packages).

[mpilgrem]

@philippedev101, the failing STAN CI can wait, but the integration tests are failing on all operating systems for a reason that I can reproduce locally (on Windows 11), namely:

stack test does not work with this Stack executable in Stack's own project directory. It does not build the test suite (only the library) and so fails with:

TestSuiteExeMissing False "stack-unit-test.exe" "stack" "stack-unit-test"
Completed 2 action(s).

Error: [S-7282]
       Stack failed to execute the build plan.

       While executing the build plan, Stack encountered the error:

       Error: [S-1995]
       Test suite failure for package stack-3.10.0
           stack-unit-test:  executable not found

(Note to self: the poor output of the TestSuiteExeMissing in the log is an existing bug that needs to be fixed.)

However, if stack test is then commanded a second time, the library and the test suite are built.

(If stack test in then commanded a third time, the library (only) is built again - when it should be a no-op.)

I've changed the PR status to 'draft', in the interim.

[mpilgrem]

A minimal reproducer of the problem is a simple one-package project that has a custom-setup:. For example:

> stack new test6865
> cd test6865
> # Edit package.yaml to introduce a custom-setup (see below)
> stack test
...
TestSuiteExeMissing False "test6865-test.exe" "test6865" "test6865-test"
Completed 2 action(s).

Error: [S-7282]
       Stack failed to execute the build plan.

       While executing the build plan, Stack encountered the error:

       Error: [S-1995]
       Test suite failure for package test6865-0.1.0.0
           test6865-test:  executable not found

The package.yaml has:

custom-setup:
  dependencies:
  - base >= 4.7 && < 5
  - Cabal

[philippedev101]

@mpilgrem Thanks for the clean repro and the draft-status pause. Root cause identified, fix pushed as a follow-up commit on this branch. Short answer: for build-type: Custom the primary and final tasks end up keyed by the same ComponentKey CLib, and toActions was dropping the final's build action on the assumption the primary would cover it. It does not: the primary runs with isFinalBuild=False which skips --enable-tests/--enable-benchmarks at configure time and narrows the Setup build targets to lib and exe via primaryComponentOptions.

Root cause

After Phase 2, the build plan is split into plan.tasks (primary) and plan.finals (tests/benches), both keyed by ComponentKey. For Simple build-type packages shouldSplitComponents returns True, addFinal keys each final under a distinct CTest/CBench, and everything works.

For Custom / Configure / Hooks / Make, shouldSplitComponents returns False and addFinal falls back to ComponentKey name CLib. Both maps now key the same package under the same ComponentKey. toActions had:

finalBuild = case mbuild of
  Just _  -> []
  Nothing -> [ ... singleBuild ... isFinalBuild=True ... ]

When both mbuild and mfinal existed on the shared key, finalBuild became [] and the primary ran alone with isFinalBuild=False. Cabal configured without --enable-tests, built only lib:foo exe:bar, then ATRunTests fired and reported TestSuiteExeMissing because the test executable was never produced.

Stack's own project reproduces this because stack.cabal has build-type: Custom plus a custom-setup block, same shape as your test6865 minimal repro.

Fix

singleBuild gains an isMergedBuild :: Bool parameter threaded through realConfigAndBuild, realBuild, and buildAndCopyOpts. Three non-split modes, three target lists:

isFinalBuild	isMergedBuild	build targets	copy targets
False	_	primary	primary
True	True	primary ++ finals	primary
True	False	finals	[]

In toActions, the primary action now calls singleBuild ac ee task installedMap hasFinal hasFinal ck where hasFinal = isJust mfinal. So when a primary fires and a final exists on the same ComponentKey, the primary runs in merged mode: configure gets --enable-tests/--enable-benchmarks, setup build gets lib:foo exe:bar test:baz bench:qux in a single invocation, setup copy installs only the primary components. The pre-existing finalBuild action (which fires when no primary task runs because the library was clean) keeps its finals-only semantics via isMergedBuild=False, so it does not redundantly rebuild the library.

Two downstream guards match the new split: doHaddock and shouldCopy both key off not (isFinalBuild && not isMergedBuild). Haddock runs on merged builds, copy/register fires for plain primary, merged, and Backpack CLib/CInst builds, and both skip only for pure finals-only builds. Simple split packages are unaffected: mfinal is Nothing on their primary CLib keys because finals live on distinct CTest/CBench keys.

The decision table lives in a small pure helper dispatchBuildOpts that buildAndCopyOpts delegates to, which keeps it unit-testable without constructing a LocalPackage.

Tests

Seven new integration tests under tests/integration/tests/:

• custom-setup-test: the minimal repro. Before the fix, fails with TestSuiteExeMissing. After, the suite runs and passes.
• custom-setup-test-exe: mirrors the stack new default layout (lib + exe + test) plus custom-setup.
• custom-setup-bench: before the fix, fails with [Cabal-1453] No benchmarks enabled.
• custom-setup-test-and-bench: both final components together.
• custom-setup-test-idempotent: first run passes, second stack test is a no-op (no [N of M] Compiling, no Installing library, no Registering library).
• custom-setup-build-then-test: the stack build then stack test workflow.
• custom-setup-lib-only: Custom-setup with only a library. Guards the no-finals path against future regressions.

Thirteen new unit tests in tests/unit/Stack/Build/ExecuteSpec.hs cover all five arms of dispatchBuildOpts plus edge cases and componentTarget round-trips.

Stack's own stack test on the branch now passes (863 examples, 0 failures).

One incidental fix

While running the Backpack integration suite as a regression check I noticed backpack-cross-package-rename had been committed failing. Test-data bug: Consumer.hs imported Sig, but the mixin renames the signature requirement via sig-pkg requires (Sig as Impl), so the consumer has to import Impl. Fixed the import. The test now exercises the rename end-to-end.

What this does not change

The classic stack build then stack test flip-flop (#2800) is still present: switching between the two triggers a library reconfigure because --enable-tests changes, and Cabal unregisters and rebuilds the library. Same constraint I noted earlier about non-split packages sharing legacy single-invocation semantics. Full per-component execution would fix it; out of scope for this PR by design.

Happy to move the PR out of draft.

[mpilgrem]

@philippedev101, unfortunately, other existing integration tests fail. I had a look at one of them: 3229-exe-targets (reproducing its steps locally, on Windows).

With Stack 3.9.3, the first stack build :alpha has:

foo> configure (lib + exe)
Configuring foo-0...
foo> build (lib + exe) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
[1 of 1] Compiling Foo
Preprocessing executable 'alpha' for foo-0...
Building executable 'alpha' for foo-0...
[1 of 1] Compiling Main
[2 of 2] Linking .stack-work\dist\1a191874\build\alpha\alpha.exe
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable alpha in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...

and the second stack build :alpha (when the alpha source code is dirty) has:

foo-0: unregistering (local file changes: app\Alpha.hs)
foo> build (lib + exe) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'alpha' for foo-0...
Building executable 'alpha' for foo-0...
[1 of 1] Compiling Main [Source file changed]
[2 of 2] Linking .stack-work\dist\1a191874\build\alpha\alpha.exe [Objects changed]
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable alpha in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...

Unlike when the test was originally written, in neither case is the beta executable built.

The 'backpack' version of Stack has, for the first stack build :alpha:

foo> configure
Configuring foo-0...
foo> build with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
[1 of 1] Compiling Foo
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Registering library for foo-0...
foo> build (exe:alpha) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'alpha' for foo-0...
Building executable 'alpha' for foo-0...
[1 of 1] Compiling Main
[2 of 2] Linking .stack-work\dist\1a191874\build\alpha\alpha.exe
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable alpha in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...
foo> build (exe:beta) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'beta' for foo-0...
Building executable 'beta' for foo-0...
[1 of 1] Compiling Main
[2 of 2] Linking .stack-work\dist\1a191874\build\beta\beta.exe
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable beta in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...
Completed 3 action(s).

In this case, the beta executable is built. That could be viewed as a regression, relative to Stack 3.9.3.

For the second stack build :alpha (when the alpha source code is dirty):

foo-0: unregistering (local file changes: app\Alpha.hs)
foo> build with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Registering library for foo-0...
foo> build (exe:alpha) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'alpha' for foo-0...
Building executable 'alpha' for foo-0...
[1 of 1] Compiling Main [Source file changed]
[2 of 2] Linking .stack-work\dist\1a191874\build\alpha\alpha.exe [Objects changed]
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable alpha in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...
foo> build (exe:beta) with ghc-9.10.3
Preprocessing library for foo-0...
Building library for foo-0...
Preprocessing executable 'beta' for foo-0...
Building executable 'beta' for foo-0...
foo> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\lib\x86_64-windows-ghc-9.10.3-b42a\foo-0-2rEae6A22yFBRUicjP8Lud
Installing executable beta in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3229-exe-targets\files\.stack-work\install\23d8aeba\bin
Registering library for foo-0...
Completed 3 action(s).

The Preprocessing executable 'beta' for foo-0... may well be a no-op, but the integration test does not view the presence of the message that way (from its Main.hs):

                 stackCheckStderr
                     ["build", ":alpha"]
                     (rejectMessage
                          (unlines
                               ["Preprocessing executable 'beta' for foo-0..."]))))

Perhaps the test should check for the absence of Installing executable beta in?

[mpilgrem]

@philippedev101, I rebased on the master branch after updating (after a false start) the integration test 3229-exe-targets - now named 6451-exe-targets. The regression sees 6451-exe-targets fail with:

Main.hs: Did not expect message here: 
"Installing executable beta in"

[mpilgrem]

I then looked at integration test 3959-order-of-flags. That is a test that involves a package with a test suite but no library.

With Stack 3.9.3, all is fine:

> stack --test --no-run-tests build
All test running disabled by --no-run-tests flag. To mute this message in future, set notify-if-no-run-tests: false in
Stack's configuration.
issue3959> configure (test)
Configuring issue3959-0.1.0.0...
issue3959> build (test) with ghc-9.10.3
Preprocessing test suite 'test' for issue3959-0.1.0.0...
Building test suite 'test' for issue3959-0.1.0.0...
[1 of 2] Compiling Main
[2 of 2] Compiling Paths_issue3959
[3 of 3] Linking .stack-work\dist\1a191874\build\test\test.exe

The 'backpack' version of Stack fails with (extract):

> stack --test --no-run-tests build
All test running disabled by --no-run-tests flag. To mute this message in future, set notify-if-no-run-tests: false in
Stack's configuration.
issue3959> configure
Configuring issue3959-0.1.0.0...
issue3959> build with ghc-9.10.3
Error: [Cabal-4444]
Unknown build target 'lib:issue3959'.
There is no library component 'issue3959' or component 'lib'.

Stack is trying to build a library component, and one that does not exist.

[mpilgrem]

@philippedev101, I have rebased on master again, as I have updated integration test 3996-sublib-not-depended-upon, which fails. This is a package with a main library and an internal library. The main library, however, does not depend on the internal library.

With Stack 3.9.3, all is fine:

myPackage> configure (lib + sub-lib)
Configuring myPackage-0.1.0.0...
myPackage> build (lib + sub-lib) with ghc-9.10.3
Preprocessing library for myPackage-0.1.0.0...
Building library for myPackage-0.1.0.0...
[1 of 1] Compiling MyPackage
Preprocessing library 'internal' for myPackage-0.1.0.0...
Building library 'internal' for myPackage-0.1.0.0...
[1 of 1] Compiling Internal
myPackage> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3996-sublib-not-depended-upon\files\.stack-work\install\6e44f233\lib\x86_64-windows-ghc-9.10.3-b42a\myPackage-0.1.0.0-HXh8re4RjQk4Ufw3IIAClw
Installing internal library internal in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3996-sublib-not-depended-upon\files\.stack-work\install\6e44f233\lib\x86_64-windows-ghc-9.10.3-b42a\myPackage-0.1.0.0-GPLsRg6s6iQKSRZ63H28HF-internal
Registering library for myPackage-0.1.0.0...
Registering library 'internal' for myPackage-0.1.0.0...

The 'backpack' version of Stack fails with (extract):

myPackage> configure
Configuring myPackage-0.1.0.0...
myPackage> build with ghc-9.10.3
Preprocessing library for myPackage-0.1.0.0...
Building library for myPackage-0.1.0.0...
[1 of 1] Compiling MyPackage
myPackage> copy/register
Installing library in D:\Users\mike\Code\GitHub\commercialhaskell\stack\tests\integration\tests\3996-sublib-not-depended-upon\files\.stack-work\install\6e44f233\lib\x86_64-windows-ghc-9.10.3-b42a\myPackage-0.1.0.0-HXh8re4RjQk4Ufw3IIAClw
Error: Cabal-simple_O_vy6YIf_3.12.1.0_ghc-9.10.3.exe:
'D:\sr\programs\x86_64-windows\ghc-9.10.3\bin\ghc-9.10.3.exe' exited with an
error:
<command line>: Could not find module ‘Internal’.
Use -v to see a list of the files searched for.

[mpilgrem]

Given these three regressions, and the other failing integration tests, I have changed the status of the pull request to draft.