CI: lint, secret scan, and end-to-end pipeline smoke test#3
Open
Alvin-Nahabwe wants to merge 7 commits into
Open
CI: lint, secret scan, and end-to-end pipeline smoke test#3Alvin-Nahabwe wants to merge 7 commits into
Alvin-Nahabwe wants to merge 7 commits into
Conversation
Semantic segmentation was broken end-to-end. Three independent defects: - image_segmentation_train.py used Dataset.from_dict() without importing Dataset (NameError on the default is_presplit path), and referenced all_mask_paths before assignment on the non-presplit path. - image_segmentation_inference.py used pred_seg in the overlay step but never computed it; the post-processing block was empty. The SMP branch also read model.config, which raw U-Net models do not have. Logits are now upsampled to the source resolution before argmax, and the palette is seeded so overlays are stable across runs. - /inference/image-segmentation shelled out to semantic_segmentation_ inference.py, which does not exist. Also in this change: - Segmentation augmentation called CoarseDropout with the removed max_holes/max_height API, crashing whenever augmentation was enabled. Aligned to the current API already used by the classification path. - Removed a duplicated --early_stopping_threshold argument and a duplicated, unreachable except block. - Removed /inference/asr: it shells out to a nonexistent asr_inference.py and ASR has no training endpoint, registry entry, or script. - CORS allowed "*" together with credentials, which browsers reject. Origins are now set via ALLOWED_ORIGINS and credentials are only enabled for an explicit allow-list. - is_valid_checkpoint_path used startswith, so a sibling directory named model_outputs_evil passed the model_outputs containment check. - The pid migration issued a raw-string SELECT, which raises ObjectNotExecutableError on SQLAlchemy 2.x, so it silently never ran. - API_KEY now warns loudly when falling back to the development default. Verified with python -m py_compile on every changed file. Not exercised against a GPU: no training or inference run was performed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Running the pipeline against a small synthetic dataset (the fixes in the
previous commit only got it to compile) surfaced six runtime defects that
static analysis could not, all triggered by current library versions
(transformers 5.x, albumentations 2.x, segmentation-models-pytorch 0.5).
Shared path (SegFormer and SMP):
- apply_transforms received masks as datasets' {path,bytes} dicts, not
PIL images, and crashed on the first batch. Added mask_to_array(),
which also reads palette ("P") masks as raw class ids rather than
remapping them through .convert("L").
- Mean IoU silently computed 0: masks were flattened to 1D (mean_iou
needs per-image 2D arrays) and SegFormer's quarter-resolution logits
were never upsampled to the label size. Fixed both; this also restores
early stopping and best-model selection, which key off eval_mean_iou.
SMP U-Net / U-Net++ path (previously crashed four different ways):
- The Trainer sets model.config.use_cache, but SMP exposes config as a
read-only property. Shadow it (and load_state_dict) on a per-instance
subclass carrying a PretrainedConfig; SMP's class is left untouched.
- SmpTrainer only overrode compute_loss, so evaluation called the model
with pixel_values=/labels= kwargs it does not accept. Added a matching
prediction_step.
- load_best_model_at_end calls load_state_dict(state_dict, strict) posi-
tionally; SMP takes strict only as a keyword. Added a positional->kw
adapter that preserves SMP's timm-encoder key remapping.
- The Trainer saved weights as model.safetensors with no config.json,
but inference read pytorch_model.bin and needed config.json. Write the
SMP config on save; load safetensors-or-bin on inference. Also guarded
image_processor.save_pretrained (None for SMP).
Verified: both model families complete train -> evaluate -> save ->
inference on a synthetic fixture (CPU), producing real per-class IoU and
valid mask overlays. py_compile passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
object_detection_train.py passed tokenizer=image_processor to the HF Trainer, but the deprecated `tokenizer` argument was removed in transformers 5.x (renamed processing_class). DETR/YOLOS training crashed with "Trainer.__init__() got an unexpected keyword argument 'tokenizer'". The classification and segmentation scripts already use processing_class. Verified: DETR completes train -> evaluate -> save -> inference on a synthetic COCO fixture, producing real mAP metrics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every dependency in the DL service was unpinned, so a fresh install resolved whatever was latest on PyPI. That is the direct cause of the runtime breakage fixed in the pipeline PR: the code had drifted behind transformers 5.x, albumentations 2.x and segmentation-models-pytorch 0.5. A clean `pip install` produced a service that crashed on two of its three segmentation model families and on object detection. This pins requirements.txt to the exact versions verified end-to-end (full train -> evaluate -> save -> inference) for all three tasks and both model families per task, on Python 3.12 / CPU torch: image classification (ViT), object detection (DETR + YOLO), semantic segmentation (SegFormer + SMP U-Net). - torch is pinned by version only; the CUDA base image pulls the matching CUDA build from PyPI. - gradio and wandb are removed: neither is imported anywhere (training sets WANDB_DISABLED=true), and gradio adds a large unused tree. - The pinned stack needs Python >= 3.11 (numpy 2.4 / pandas 3.0), so the Dockerfile base moves from ubuntu22.04/py3.10 to ubuntu24.04/py3.12 and installs into a venv (24.04's system Python is externally managed). Verified: pins run all tasks on CPU/py3.12. NOT yet verified: the CUDA container build itself (needs a GPU host) — torch==2.13.0 and the ubuntu24.04 CUDA base tag were confirmed to exist, but the image has not been built here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduces .github/workflows/ci.yml with three jobs: - lint: ruff (advisory for now) + compileall on every module. - secrets: gitleaks history scan (binary, not the Action, to avoid the org-license requirement). - smoke: installs the pinned deps on CPU and runs scripts/smoke_test.py, which generates tiny synthetic datasets and drives the real training and inference scripts for every task and model family (classification, object detection via DETR and YOLO, segmentation via SegFormer and SMP U-Net), asserting each completes and produces its artifacts. The smoke test is the regression guard for the class of bug this whole effort fixed: library-version drift that only shows up at runtime. It was validated locally end-to-end (all five paths pass) before commit. This branch merges the pipeline fixes and the dependency pins so the smoke job is green here. See PRs #1 (code) and #2 (pins) for the reviewed substance; this can either be merged as one unit or rebased to CI-only after those land. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
.github/workflows/ci.yml(3 jobs) andscripts/smoke_test.py.ruff(advisory) +compileallon every modulegitleakshistory scan (binary, not the Action — the Action needs a paid license for org repos)The smoke test is the regression guard for the exact class of bug this whole effort fixed: dependency drift that only surfaces at runtime. A fresh
pip installhad been producing a service that crashed on object detection and on two of three segmentation model families — CI now fails the build if that ever recurs. Validated locally end-to-end before commit:Merge order (important)
This branch bundles the commits from #1 (code fixes) and #2 (dependency pins) so the smoke job is green here — the smoke test can only pass on fixed code with a known stack.
Recommended: review and merge #1 and #2 first (they hold the actual substance, in reviewable stages). Once they land, this PR's diff automatically collapses to just
ci.yml+smoke_test.py(the shared commits become no-ops), and it can be merged as a clean CI-only change.Alternatively, merge this one PR as the whole unit.
Notes / follow-ups
ruffis advisory (|| true) for now — the existing tree has lint noise; flip it to blocking once cleaned.no-code-app(Rparse()+ lint + gitleaks) is coming in a separate PR on that repo.🤖 Generated with Claude Code