Skip to content

Feat/persistency import export#166

Open
viktorbeck98 wants to merge 7 commits into
developmentfrom
feat/persistency-import-export
Open

Feat/persistency import export#166
viktorbeck98 wants to merge 7 commits into
developmentfrom
feat/persistency-import-export

Conversation

@viktorbeck98

Copy link
Copy Markdown
Collaborator

feat: export and import EventPersistency state on demand

Problem

There was no way to transfer a detector's trained state between environments (e.g. training → production) or take a manual snapshot outside the auto-save timer cycle, without constructing a full PersistencySaver with its threading machinery.

What changed

Layer 1 — standalone functions (detectmatelibrary.utils.persistency)

save and load are extracted from PersistencySaver's internals and exposed as module-level functions. PersistencySaver now delegates to them, so the serialisation format lives in exactly one place.

Layer 2 — detector-level wrappers (CoreDetector)

export_state and import_state are thin wrappers around the above. import_state is thread-safe: it acquires the saver lock before loading when a PersistencySaver is running.

Bytes overload

Both layers accept an optional path. When omitted, save/export_state return the state as an in-memory zip archive (bytes); load/import_state accept either a path string or those bytes. This lets an API send or receive state without touching the filesystem.

Metadata

event_data_class is now written to the metadata JSON and restored on load, so a detector loaded into a fresh EventPersistency picks the correct backend for new events automatically. Existing snapshots without the field continue to load fine.

Usage

# File-based (unchanged behaviour)
detector.export_state("s3://bucket/snapshots/my-detector")
detector.import_state("s3://bucket/snapshots/my-detector")

# Bytes-based (new — no disk I/O)
data: bytes = detector.export_state()
detector.import_state(data)

# Standalone (advanced / no detector)
from detectmatelibrary.utils import persistency
data = persistency.save(ep)
persistency.load(ep, data)

What did not change

  • PersistencySaver public API (save, load, start, stop) is unchanged.
  • On-disk format is unchanged; one new optional metadata field added, old snapshots remain loadable.
  • EventPersistency gains no new public methods.

Tests

  • TestStandaloneSaveLoad — 7 new tests covering the bytes overload roundtrip
  • TestDetectorExportImportState — 3 new tests covering bytes export/import at the detector level, including a running-saver safety check

viktorbeck98 and others added 6 commits June 3, 2026 14:34
…metadata

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…, fix load() thread safety

- Add module-level _load helper that mirrors _save
- Restore event_data_class from metadata on load
- PersistencySaver.load() now acquires self._lock before mutating persistency
- Use fs.pipe() for atomic file writes in _save to eliminate read/write race
- Add test_load_restores_event_data_class and TestPersistencySaverThreadSafety

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ctor

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
save(ep) returns a zip archive as bytes; load(ep, bytes) restores from
it. export_state() and import_state() on CoreDetector gain the same
overload, so callers can transfer state over a network API without
touching the filesystem.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@viktorbeck98 viktorbeck98 requested a review from ipmach June 3, 2026 14:37
@viktorbeck98 viktorbeck98 self-assigned this Jun 3, 2026
@viktorbeck98 viktorbeck98 added the enhancement New feature or request label Jun 3, 2026
@ipmach

ipmach commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

There are merging conflicts and I added some comments in the code.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • This logic is not part of the detector but the persistency itself. The detector should only have to call. persistency.save/load.
  • I will just return None and not raise an exception if there is not persistency configure. Because the detector can still work without persistency and we should not crash the program if we dont need to.
  • We do not use multithreats detectors so the thread-safe is just added complexity.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Soon we will have more parsers and alert aggregation methods. export_state/import_state methods should be for every class as we will need to save and import from those classes as well. I think it is better if they are in the core component

@ernstleierzopf

Copy link
Copy Markdown
Contributor

There are merging conflicts and I added some comments in the code.

I fixed those merge conflicts as they happened because of the renaming of tests into detectmatelibrary_tests. No changes required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants