perf: defer timezonefinder & pandas imports to cut startup time#482
Open
mrosseel wants to merge 284 commits into
Open
perf: defer timezonefinder & pandas imports to cut startup time#482mrosseel wants to merge 284 commits into
mrosseel wants to merge 284 commits into
Conversation
The withOpenconnect parameter does not exist. VPN plugins are already excluded via plugins = lib.mkForce [] in the module config. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
NetworkManager fix-paths.patch embeds openconnect binary path. Override with a dummy script to avoid pulling openconnect -> stoken -> GTK 427MB. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
MM is a build-time dependency of NM, not just runtime. Reverts to working 376MB state. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
iwd is ~10x lighter than NetworkManager for WiFi-only use. - Removes modemmanager, VPN plugins, glib/polkit bloat - Uses systemd-networkd for DHCP - iwctl CLI available for managing networks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
profiles/base.nix (imported by sd-image) adds w3m which pulls perl (150MB). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
nixos-container pulls perl for container management. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add pinctrl to PWM overlay to route PWM0_1 to GPIO 13 - Change boot-splash to static mode (no animation) - Fix boot-splash color from blue to red (RGB565) - Fix NFS deploy script /etc symlinks for PAM Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Primary build on self-hosted aarch64 (Pi5) - fast native - Fallback to ubuntu-latest with QEMU if Pi5 unavailable - Also trigger on nixos branch pushes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
pifinder-dev doesn't exist, use pifinder-netboot Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The closure's /etc is a symlink, so direct paths don't work. Use /etc/static which resolves through the NixOS etc derivation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Type=idle waits for ALL other services to finish before starting, causing ~2 minute delay. Type=simple starts immediately. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ln -sfT can't overwrite a directory, need rm -rf first Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update bootstrap.nix to fetch from correct repo/branch - Update NIXOS_STATUS.md: WiFi warning is cosmetic, not a real issue Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add pifinder.repoUrl module option to both services.nix and bootstrap.nix so switching between mrosseel (testing) and brickbots (production) repos requires only changing the default value in one place. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Builds bootstrap SD image on Pi5 self-hosted runner, extracts boot/rootfs partitions, and creates tarball for RPi OS -> NixOS migration. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds push trigger for nixos branch when bootstrap.nix, flake.nix, or the workflow itself changes. Uses VERSION env var for flexibility. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add -type f to find command to match actual image files inside the sd-image subdirectory. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The rootfs contains root-owned files and lost+found directories that require elevated permissions to access. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove any existing tarball, staging dir, and mount points before starting to avoid permission issues from previous failed runs. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Port manifest/channel infrastructure from migration branch: - _parse_version with pre-release support - _fetch_manifest / _fetch_version_txt with fallback - _filter_upgrades, _available_channels - Multi-phase menu: channel → version → action - UIReleaseNotes with _strip_markdown - Full test suite for version parsing, filtering, markdown Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shell.nix provides Python 3.9 + native build deps (linuxHeaders, gcc, SDL2, etc.) with LD_LIBRARY_PATH for scipy/numpy. .envrc auto-creates venv and installs requirements via direnv. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace runtime camera selection (config.txt dtoverlay) with build-time device tree overlays and NixOS specialisations. All three camera variants (imx462, imx296, imx477) are pre-built; switching is instant via `switch-to-configuration boot` + reboot. - Remove runtimeCameraSelection option, always bake overlay into DT - Change default camera from imx296 to imx462 - Add specialisations to SD image (imx296, imx477) and netboot (imx296, imx462) - Add pifinder-switch-camera wrapper script and sudoers entry - Preserve camera choice across upgrades via /var/lib/pifinder/camera-type - Delete Debian sys_utils.py, rename sys_utils_nixos.py to sys_utils.py - Simplify utils.py get_sys_utils() (no more /etc/NIXOS check) - Delegate callbacks.py get_camera_type() to sys_utils - Deploy script: use FDTDIR in extlinux.conf, remove dtoverlay from config.txt Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The pifinder systemd service has a restricted PATH that doesn't include sudo. Use /run/current-system/sw/bin/sudo for all calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These commands are called outside of sudo (which has secure_path), so they need full paths in the restricted pifinder service PATH. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bump nixpkgs input to nixos-25.11 - Update Python 3.12 → 3.13 across all references - Switch simplejpeg wheel to cp313 - Add explicit pyproject/build-system to 13 packages (required by 25.11) - Add dontCheckRuntimeDeps for packages with env-level deps - Rename hardware.pulseaudio → services.pulseaudio Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- u-boot SD image: use custom ubootSD with CONFIG_CMD_SYSBOOT, hardcoded load address, correct partition (0:2) - RPi.GPIO: patch cpuinfo.c for aarch64 where /proc/cpuinfo lacks Hardware line and u-boot replaces firmware DTB - I2C1: add overlay to fdtoverlay chain (nixos-hardware was bypassed) - sudo: substituteInPlace pifinder-src to use /run/wrappers/bin/sudo - polkit: add reboot/shutdown D-Bus rules for pifinder user - /boot: mount firmware at /boot/firmware so extlinux.conf works Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- u-boot: disable PCI/USB/NET probing (CONFIG_PREBOOT="", CONFIG_PCI=n, CONFIG_USB=n, etc.) — eliminates ~3-4s of unnecessary hardware scanning - boot-splash: move earlier with DefaultDependencies=false, poll for SPI device instead of relying on systemd restart - Disable NetworkManager-wait-online (7.1s savings, NM still works async) - pwm-permissions: no longer blocks pifinder startup (runs in parallel) - pifinder-src: pre-compile .pyc bytecode at build time Net result: splash ~7s (was 21s), menu ~45s (was 65s). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…not version string
…anifest terminology (ADRs 0004-0006)
…ions protect themselves
…DR 0004), Type=notify readiness - release.yml builds images.pifinder-migration and cuts the tarball from it - SD image seeds /var/lib/pifinder/current-build.json with its own store path; version labels resolve via the update manifest; pifinder-build.json deleted - pifinder.service Type=notify: app sends READY=1 once the UI is constructed and drawing (dev-safe no-op without NOTIFY_SOCKET) — the watchdog health check now means 'the screen is live'
…ery-hold hints on failure screens (ADR 0005/0006)
…ration code lives on main, one implementation (brickbots#521)
…on only and lives on main (brickbots#521)
…V, catalog refactor)
…eroff latch on GPIO14 (ADR 0007)
…6 dev boxes python-libinput/rpi-gpio/rpi-hardware-pwm/python-prctl/dbus-python/picamera2/ spidev/pygobject/pyhotkey/pynput are aarch64-only: on dev machines they either can't build (evdev kernel headers, imp removal, cairo/dbus C deps) or are meaningless off-device (dev uses sys_utils_fake and the pygame keyboard). The device env (uv2nix, aarch64) is unchanged. Dev flow: uv sync -> pytest (CI marker set passes 735/735 under 3.13).
…oadcom wifi/BT blobs
…rop the baked-target existence condition (ADR 0003)
…y the update UI keys on
…ng shifted, gpsd pointed at a nonexistent device
The repeated block in UBX-NAV-SVINFO is chn, svid, flags, quality, cno, elev, azim — but the parser started reading at chn, so every field was shifted one byte: satellite IDs were channel numbers, the used flag was svid bit 0, and the C/N0 was the 0-7 quality indicator. Because "cno" was really the quality indicator (>=1 even for idle SBAS/QZSS channels still searching), the sats-seen count was inflated to near the full channel count, and the per-satellite used flags were random — which is why uSat was previously treated as stale and ignored. With the offsets fixed: - nSat only counts satellites with an actual signal (cno > 0) - uSat from SVINFO is valid, so report it as the used count instead of relying solely on NAV-SOL - elevation/azimuth are decoded as signed values per the spec Also update the used count from NAV-PVT numSV, which was parsed but never surfaced — on protVer >= 15 receivers gpsd enables NAV-PVT instead of NAV-SOL, so the used count stayed 0 forever despite a valid fix. And actually set got_sat_update when NAV-SAT arrives so the SVINFO fallback defers to it as intended. Verified against a live MAX-M8 (PROTVER 14) stream: SVINFO uSat now matches NAV-SOL numSV, and IDs/C-N0 match gpsd's own SKY decode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
Newer u-blox receivers (later M8 firmware, M9, M10) get NAV-PVT + NAV-SAT from gpsd instead of NAV-SOL + NAV-SVINFO, so this path is what recent PiFinder GPS units actually exercise. - take "used" from the dedicated svUsed flag (bit 3) instead of inferring it from the quality indicator - only count satellites with an actual signal: NAV-SAT lists every known satellite, so nSat was inflated by idle entries - decode elevation/azimuth as signed values Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
During cold start the receiver reports an estimated C/N0 for almanac-predicted satellites it is still trying to confirm, so a cno > 0 filter makes the seen count start around 20+ and sink to the real value as candidates fail to confirm. Gate on the quality indicator instead: only code-locked signals (quality >= 4) count as seen, in both NAV-SVINFO and NAV-SAT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
mrosseel
force-pushed
the
lazy-imports
branch
4 times, most recently
from
51b0bde to
9dfd21e
Compare
Gating the seen count on quality >= 4 (code locked) made the display flap to zero during marginal re-acquisition — walking out to sky view showed 0-3 satellites bouncing instead of the steady climb to a lock. Satellites at quality 2-3 (signal acquired) have a real signal and belong in the count; only quality-1 search candidates, whose C/N0 is an estimate, still don't. Same threshold for NAV-SVINFO and NAV-SAT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
state.py imported timezonefinder and built a TimezoneFinder() in SharedState.__init__, and plot.py imported pandas at module level. Both load during boot — SharedState is constructed at startup, and plot.py is pulled in via menu_structure -> UIChart. Defer the TimezoneFinder construction to the first set_location(), and pandas to the plot functions that use it, so neither blocks startup. comets.py is intentionally left unchanged: its module-level 'from skyfield.data import mpc' imports pandas regardless, so deferring pandas there has no effect without also deferring skyfield. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-implements the salvageable part of #378 (boot-speed), cleanly on top of the
nixosbranch.#378's perf work was lost during that branch's own internal merges — its HEAD no longer contains the lazy-import changes, and they were never merged upstream either. This brings back the part that still matters for startup time.
What this defers
state.py):SharedState.__init__eagerly importedtimezonefinderand constructedTimezoneFinder()(which loads its dataset) at startup. It's now constructed lazily on the firstset_location(), i.e. after boot.plot.py): imported at module level.plot.pysits on the startup path (menu_structure→UIChart→plot), so this loaded pandas during boot. Deferred into the four functions that actually use it.comets.pyis intentionally left unchanged: its module-levelfrom skyfield.data import mpcimports pandas regardless, so deferring pandas there is a no-op without also deferring skyfield (out of scope here).Verification
state.pyno longer loadstimezonefinder(confirmed at runtime).plot.py's only module-level pandas reference is removed;skyfield.data.hipparcosalready imports pandas lazily, and nothing else inplot.py's import chain pulls pandas.py_compileclean on all touched files.🤖 Generated with Claude Code