Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/config/fault-manager.rst
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,12 @@ Basic Snapshot Settings
- Maximum message size to capture (bytes). Larger messages are truncated.
* - ``snapshots.default_topics``
- ``[]``
- List of topics to capture for all faults.
- List of topics to capture for all faults. Empty entries are ignored.
* - ``snapshots.entity_default``
- ``true``
- Zero-config fallback: when no explicit config matches a fault code,
capture the reporting source node's own published topics. Set to
``false`` to opt out.
* - ``snapshots.config_file``
- ``""``
- Path to YAML file with fault-specific snapshot configurations.
Expand Down
8 changes: 8 additions & 0 deletions docs/config/server.rst
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,14 @@ Configure how the gateway connects to the fault manager services and event topic
- float
- ``5.0``
- Timeout for fault manager service calls such as ``list_faults`` and ``get_snapshots``.
* - ``entity_freeze_frame.enabled``
- bool
- ``true``
- Zero-config freeze-frames for plugin-backed entities: when a fault from a
plugin-owned entity confirms, snapshot the entity's current data values
(via its plugin's DataProvider) and merge them into the fault detail's
``environment_data.snapshots``. Explicit snapshot config in the fault
manager always wins when present. Only active when plugins are loaded.

When ``fault_manager.namespace`` is set, the gateway also subscribes to the matching
fault event topic (for example ``/robot1/fault_manager/events`` instead of the default
Expand Down
72 changes: 68 additions & 4 deletions docs/tutorials/snapshots.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ preserve topic data when faults are confirmed, enabling post-mortem debugging.
Overview
--------

When a fault transitions to CONFIRMED status, the system can automatically
capture data from configured ROS 2 topics. This snapshot preserves the
system state at the moment of fault occurrence, similar to:
When a fault transitions to CONFIRMED status, the system automatically
captures data from ROS 2 topics. This snapshot preserves the system state
at the moment of fault occurrence, similar to:

- **AUTOSAR DEM freeze frames** - diagnostic data captured at fault detection
- **SOVD environment data** - system context for fault analysis
Expand All @@ -24,6 +24,11 @@ Snapshots are useful for:
- Understanding system state when a fault occurred
- Post-mortem analysis without real-time access to the robot

Capture works out of the box with **zero configuration**: when no explicit
snapshot config matches a fault code, the faulting entity's own data is
captured by default (see :ref:`entity-default-freeze-frames`). Explicit
configuration always overrides the zero-config fallback when present.

.. note::

Snapshots are automatically deleted when a fault is cleared via the
Expand Down Expand Up @@ -69,7 +74,12 @@ Configure snapshot capture via fault manager parameters:
- Enable/disable snapshot capture
* - ``snapshots.default_topics``
- ``[]``
- Topics to capture for all faults
- Topics to capture for all faults (empty entries are ignored)
* - ``snapshots.entity_default``
- ``true``
- Zero-config fallback: when no explicit config matches a fault code,
capture the reporting source node's own published topics. Set to
``false`` to opt out (unconfigured faults then get no freeze-frame).
* - ``snapshots.config_file``
- ``""``
- Path to YAML config for fault-specific topics
Expand Down Expand Up @@ -130,6 +140,8 @@ For fault-specific topic capture, create a YAML configuration file:
1. ``fault_specific`` - Exact match for fault code
2. ``patterns`` - Regex pattern match (first matching pattern wins)
3. ``default_topics`` - Fallback for all faults
4. Entity-default - zero-config fallback when nothing above matches
(``snapshots.entity_default``, on by default)

**Launch with config file:**

Expand All @@ -140,6 +152,58 @@ For fault-specific topic capture, create a YAML configuration file:
-p snapshots.config_file:=/path/to/snapshots.yaml \
-p snapshots.default_topics:="['/diagnostics']"

.. _entity-default-freeze-frames:

Zero-Config Entity Freeze-Frames
--------------------------------

With no snapshot configuration at all, every confirmed fault still carries
at-fault-time context: the faulting entity's own current data. Two paths
cover the two kinds of entities, both on by default with an opt-out flag,
and explicit config (``fault_specific`` / ``patterns`` / ``default_topics``)
always wins when it matches.

**ROS-backed entities** (``snapshots.entity_default``, fault manager): when
no explicit config matches the fault code, the fault manager captures the
topics *published by the fault's reporting source node(s)* (resolved from
``source_id``), excluding per-node noise (``/rosout``,
``/parameter_events``) and capped at 16 topics. These captures always sample
on demand, even when ``snapshots.background_capture`` is enabled, because
the topics are not known until the fault confirms. If the source is not a
live node (e.g. a plugin entity id) or publishes nothing, no freeze-frame
row is written.

**Plugin-backed entities** (``entity_freeze_frame.enabled``, gateway): PLC
apps bridged by protocol plugins report faults under their bare SOVD entity
id and their live values are not ROS topics, so the fault manager cannot
capture them. Instead, the gateway snapshots the entity's current data
values (from the owning plugin's ``DataProvider``, i.e. the latest polled
values) when the fault confirms, and merges them into the fault detail's
``environment_data.snapshots`` as a standard ``freeze_frame`` entry named
after the entity - unless the fault manager already captured a freeze-frame
for that fault (explicit config wins). Disable with:

.. code-block:: bash

ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p entity_freeze_frame.enabled:=false

Example plugin-entity freeze-frame in the fault response:

.. code-block:: json

{
"type": "freeze_frame",
"name": "beckhoff_plc_app",
"data": {"tank_level": 87.5, "pump_running": true},
"x-medkit": {
"topic": "",
"message_type": "",
"full_data": {"tank_level": 87.5, "pump_running": true},
"captured_at": "2026-07-14T12:00:00.000Z"
}
}

Querying Snapshots
------------------

Expand Down
9 changes: 8 additions & 1 deletion src/ros2_medkit_fault_manager/config/snapshots.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,14 @@
# Topic resolution priority:
# 1. fault_specific - exact fault code match (highest priority)
# 2. patterns - regex pattern match on fault code
# 3. default_topics - fallback for all other faults (lowest priority)
# 3. default_topics - fallback for all other faults
# 4. entity-default - zero-config fallback when nothing above matches:
# the reporting source node's own published topics are captured
# (parameter snapshots.entity_default, on by default)
#
# A fault code listed in fault_specific (or matched by a pattern) with an
# EMPTY topic list is an explicit per-fault opt-out: nothing is captured and
# the entity-default fallback is skipped for that code.
#
# Usage:
# ros2 run ros2_medkit_fault_manager fault_manager_node \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,12 @@ struct SnapshotConfig {
/// Default topics to capture if no specific or pattern match
std::vector<std::string> default_topics;

/// Zero-config fallback when NO explicit config matches a fault code: capture
/// the topics published by the fault's reporting source node(s) instead of
/// skipping the capture entirely. Explicit config (fault_specific / patterns /
/// default_topics) always wins when present.
bool entity_default{true};

/// Rosbag recording configuration (optional)
RosbagConfig rosbag;
};
Expand Down Expand Up @@ -151,7 +157,11 @@ class SnapshotCapture {
/// Capture snapshots for a fault that was just confirmed
///
/// If the fault code resolves to no capture set (not in fault_specific, no pattern
/// match, no default_topics), capture returns early: no freeze_frames row is written
/// match, no default_topics), the entity-default fallback (when enabled) captures
/// the reporting source node's own published topics instead. A code explicitly
/// listed in fault_specific or matched by a pattern never falls through - a
/// present-but-empty topic list is a per-fault opt-out. Only when nothing
/// resolves does capture return early: no freeze_frames row is written
/// (no empty {} row) and FaultStorage::get_freeze_frame() returns nullopt for it.
/// @param fault_code The fault code that was confirmed
void capture(const std::string & fault_code);
Expand All @@ -169,7 +179,18 @@ class SnapshotCapture {
private:
/// Resolve which topics to capture for a given fault code
/// Priority: fault_specific > patterns > default_topics
std::vector<std::string> resolve_topics(const std::string & fault_code) const;
/// @param[out] explicit_match true when the code hit fault_specific or a
/// pattern (even with an empty list - an explicit opt-out) or
/// default_topics applied; such codes never fall through to the
/// entity-default capture.
std::vector<std::string> resolve_topics(const std::string & fault_code, bool & explicit_match) const;

/// Entity-default fallback: topics published by the fault's reporting source
/// node(s), excluding per-node noise (/rosout, /parameter_events). Non-FQN
/// sources (bare plugin entity ids - the gateway covers those) are skipped,
/// and empty is returned when no source resolves to a live node. Never
/// throws; any failure degrades to empty.
std::vector<std::string> resolve_entity_topics(const std::string & fault_code) const;

/// Capture a single topic on-demand (creates temporary subscription)
/// On success also records the captured value into @p freeze_frame under the topic key.
Expand Down
11 changes: 10 additions & 1 deletion src/ros2_medkit_fault_manager/src/fault_manager_node.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

#include <yaml-cpp/yaml.h>

#include <algorithm>
#include <cctype>
#include <cinttypes>
#include <filesystem>
Expand Down Expand Up @@ -1061,9 +1062,17 @@ SnapshotConfig FaultManagerNode::create_snapshot_config() {
}
config.max_message_size = static_cast<size_t>(max_message_size_param);

// Default topics (catch-all)
// Default topics (catch-all). Empty entries are dropped: launch files cannot
// express an empty string array, so [''] is the conventional "no defaults"
// and must not shadow the entity-default fallback below.
config.default_topics =
declare_parameter<std::vector<std::string>>("snapshots.default_topics", std::vector<std::string>{});
config.default_topics.erase(std::remove(config.default_topics.begin(), config.default_topics.end(), std::string{}),
config.default_topics.end());

// Zero-config fallback: capture the reporting node's own published topics
// when no explicit config matches the fault code (opt-out).
config.entity_default = declare_parameter<bool>("snapshots.entity_default", true);

// Load fault_specific and patterns from YAML config file if provided
auto config_file = declare_parameter<std::string>("snapshots.config_file", "");
Expand Down
111 changes: 105 additions & 6 deletions src/ros2_medkit_fault_manager/src/snapshot_capture.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
#include <rclcpp/serialized_message.hpp>
#include <set>
#include <thread>
#include <utility>

#include "ros2_medkit_fault_manager/time_utils.hpp"
#include "ros2_medkit_serialization/json_serializer.hpp"
Expand Down Expand Up @@ -122,7 +123,16 @@ void SnapshotCapture::capture(const std::string & fault_code) {
return;
}

auto topics = resolve_topics(fault_code);
bool explicit_match = false;
auto topics = resolve_topics(fault_code, explicit_match);
// Zero-config fallback: nothing in the explicit config even mentioned this
// code. A present-but-empty fault_specific/pattern entry is an explicit
// per-fault opt-out (explicit_match == true) and must NOT fall through here.
bool entity_scoped = false;
if (topics.empty() && !explicit_match && config_.entity_default) {
topics = resolve_entity_topics(fault_code);
entity_scoped = !topics.empty();
}
if (topics.empty()) {
// Unconfigured fault code: resolve_topics() yielded nothing, so capture returns early.
// No freeze_frames row is written (we never persist an empty {} row) and no
Expand All @@ -132,15 +142,17 @@ void SnapshotCapture::capture(const std::string & fault_code) {
return;
}

RCLCPP_INFO(node_->get_logger(), "Capturing snapshots for fault '%s' (%zu topics)", fault_code.c_str(),
topics.size());
RCLCPP_INFO(node_->get_logger(), "Capturing snapshots for fault '%s' (%zu topics%s)", fault_code.c_str(),
topics.size(), entity_scoped ? ", entity-default" : "");

// Accumulate a compact dict of captured topic values alongside the per-topic snapshots.
nlohmann::json freeze_frame = nlohmann::json::object();
size_t captured_count = 0;
for (const auto & topic : topics) {
bool success = false;
if (config_.background_capture) {
// Entity-default topics are not known at construction, so the background
// cache never holds them - always sample those on demand.
if (config_.background_capture && !entity_scoped) {
success = capture_topic_from_cache(fault_code, topic, freeze_frame);
} else {
success = capture_topic_on_demand(fault_code, topic, freeze_frame);
Expand All @@ -161,6 +173,14 @@ void SnapshotCapture::capture(const std::string & fault_code) {
// An empty capture must never clobber an existing non-empty frame: a re-confirm while
// the source publishers are down would otherwise replace the retained record with {}.
if (captured_count == 0) {
// Entity-default runs are best-effort: an empty one must not create a row
// (absence still means "nothing configured captured anything") nor clobber
// a retained frame from an earlier explicit-config capture.
if (entity_scoped) {
RCLCPP_DEBUG(node_->get_logger(), "Entity-default capture for fault '%s' sampled nothing; no row written",
fault_code.c_str());
return;
}
auto existing = storage_->get_freeze_frame(fault_code);
if (existing.has_value() && existing->data != "{}") {
RCLCPP_WARN(node_->get_logger(), "Nothing captured for fault '%s'; keeping previously retained freeze-frame",
Expand All @@ -176,7 +196,9 @@ void SnapshotCapture::capture(const std::string & fault_code) {
storage_->store_freeze_frame(frame);
}

std::vector<std::string> SnapshotCapture::resolve_topics(const std::string & fault_code) const {
std::vector<std::string> SnapshotCapture::resolve_topics(const std::string & fault_code, bool & explicit_match) const {
explicit_match = true;

// Priority 1: Exact match in fault_specific
auto it = config_.fault_specific.find(fault_code);
if (it != config_.fault_specific.end()) {
Expand All @@ -192,15 +214,92 @@ std::vector<std::string> SnapshotCapture::resolve_topics(const std::string & fau
}
}

// Priority 3: Default topics
// Priority 3: Default topics (empty means "not configured", not an opt-out)
if (!config_.default_topics.empty()) {
RCLCPP_DEBUG(node_->get_logger(), "Using default_topics for '%s'", fault_code.c_str());
return config_.default_topics;
}

explicit_match = false;
return {};
}

std::vector<std::string> SnapshotCapture::resolve_entity_topics(const std::string & fault_code) const {
// Bound the on-demand sweep: each silent-but-published topic costs up to
// timeout_sec, so a node with many declared topics must not stall a capture
// pool worker indefinitely.
static constexpr size_t kMaxEntityTopics = 16;

std::set<std::string> topics;
// Guarded like RosbagCapture::resolve_entity_topics: this runs on a capture
// pool worker with no outer catch, and storage_->get_fault() can throw on
// the sqlite backend. Any failure degrades to "no entity-default capture".
try {
auto fault = storage_->get_fault(fault_code);
if (!fault || fault->reporting_sources.empty()) {
return {};
}

// Only ROS node FQNs (e.g. "/ns/planner_server") participate. Plugin
// entities report under bare SOVD ids (e.g. "tank_process"); treating one
// as (name, ns="/") could match an unrelated root-namespace node of the
// same name and freeze-frame its topics. Their zero-config frames come
// from the gateway's EntityFreezeFrameCapture instead.
std::set<std::pair<std::string, std::string>> wanted;
for (const auto & source : fault->reporting_sources) {
if (source.size() < 2 || source.front() != '/') {
continue;
}
const auto slash = source.rfind('/');
const std::string name = source.substr(slash + 1);
const std::string ns = (slash == 0) ? "/" : source.substr(0, slash);
if (!name.empty()) {
wanted.emplace(name, ns);
}
}
if (wanted.empty()) {
return {};
}

// The entity's "own data" = topics it publishes. Every node publishes
// /rosout and /parameter_events; those say nothing about the entity.
static const std::set<std::string> kExcluded = {"/rosout", "/parameter_events"};

auto published_by_wanted = [&wanted](const std::vector<rclcpp::TopicEndpointInfo> & eps) {
for (const auto & ep : eps) {
if (wanted.count({ep.node_name(), ep.node_namespace()})) {
return true;
}
}
return false;
};
for (const auto & [topic, types] : node_->get_topic_names_and_types()) {
if (kExcluded.count(topic)) {
continue;
}
if (published_by_wanted(node_->get_publishers_info_by_topic(topic))) {
topics.insert(topic);
if (topics.size() >= kMaxEntityTopics) {
RCLCPP_WARN(node_->get_logger(),
"Entity-default capture for fault '%s' capped at %zu topics; remaining topics skipped",
fault_code.c_str(), kMaxEntityTopics);
break;
}
}
}
} catch (const std::exception & e) {
RCLCPP_WARN(node_->get_logger(), "Entity-default topic resolution failed for fault '%s': %s", fault_code.c_str(),
e.what());
return {};
}

if (!topics.empty()) {
RCLCPP_INFO(node_->get_logger(), "Entity-default capture for fault '%s': %zu topic(s) from the reporting node(s)",
fault_code.c_str(), topics.size());
}
return {topics.begin(), topics.end()};
}

bool SnapshotCapture::capture_topic_on_demand(const std::string & fault_code, const std::string & topic,
nlohmann::json & freeze_frame) {
// Get topic type
Expand Down
Loading
Loading