VictoriaMetrics stack in CTF

book/src/SUMMARY.md

@@ -29,12 +29,13 @@
   - [Copying Files](framework/copying_files.md)
   - [Running in Kubernetes](./framework/kubernetes.md)
 - [Observability Stack](framework/observability/observability_stack.md)
-  - [Overview](framework/observability/observability_stack.md)
   - [Metrics](framework/observability/metrics.md)
   - [Logs](framework/observability/logs.md)
   - [Profiling](framework/observability/profiling.md)
   - [PostgreSQL](framework/observability/postgresql.md)
   - [BlockScout](framework/observability/blockscout.md)
+- [Observability Stack (VictoriaMetrics)](framework/observability-victoria/observability_stack_victoria.md)
+  - [Logs](framework/observability-victoria/logs.md)
 - [Components](framework/components/overview.md)
   - [Overview](framework/components/overview.md)
   - [Blockchains](framework/components/blockchains/overview.md)

book/src/framework/observability-victoria/logs.md

@@ -0,0 +1,43 @@
+# Logs
+
+WASP load data is stored in `VictoriaLogs` and queried with [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/). The bundled **WASP (VictoriaLogs)** dashboard is loaded automatically, but you can also explore the raw data in [localhost:3000](http://localhost:3000/explore) (`VictoriaLogs` datasource) or directly via the VictoriaLogs UI at [localhost:9428](http://localhost:9428/select/vmui).
+
+Every WASP record carries these fields: `go_test_name`, `gen_name`, `call_group`, `branch`, `commit` and `test_data_type` (`stats` or `responses`). Numeric values (`current_rps`, `current_instances`, `duration`, ...) live in the JSON `_msg`, so add `| unpack_json` before aggregating them.
+
+[Explore](http://localhost:3000/explore?schemaVersion=1&panes=%7B%22il3%22:%7B%22datasource%22:%22victorialogs%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22victoriametrics-logs-datasource%22,%22uid%22:%22victorialogs%22%7D,%22editorMode%22:%22code%22,%22expr%22:%22%2A%22,%22queryType%22:%22instant%22%7D%5D,%22range%22:%7B%22from%22:%22now-5m%22,%22to%22:%22now%22%7D%7D%7D&orgId=1)
+
+## Example WASP Log Queries
+
+Queries:
+- All data for a single test
+```sql
+go_test_name:="TestMyLoad"
+```
+- Periodic generator stats (RPS, VUs, sampling)
+```sql
+go_test_name:="TestMyLoad" AND test_data_type:stats
+```
+- Individual responses (one log line per call)
+```sql
+go_test_name:="TestMyLoad" AND test_data_type:responses
+```
+- Current RPS per generator
+```sql
+test_data_type:stats | unpack_json | stats by (go_test_name, gen_name) max(current_rps) value
+```
+- Current VUs (virtual users) per generator
+```sql
+test_data_type:stats | unpack_json | stats by (go_test_name, gen_name) max(current_instances) value
+```
+- Failed responses
+```sql
+test_data_type:responses AND _msg:"\"failed\":true"
+```
+- Timed out responses
+```sql
+test_data_type:responses AND _msg:"\"timeout\":true"
+```
+- Latency quantiles (p99/p95/p50, in nanoseconds) per call group
+```sql
+test_data_type:responses | unpack_json | stats by (gen_name, call_group) quantile(0.99, duration) value
+```

book/src/framework/observability-victoria/metrics.md

@@ -0,0 +1 @@
+# Metrics

book/src/framework/observability-victoria/observability_stack_victoria.md

@@ -0,0 +1,36 @@
+# Local Observability Stack (VictoriaMetrics)
+
+Minimal setup for experimenting with OTEL metrics & logs in Grafana.
+
+## Components
+
+| Service | Port | Purpose |
+|---|---|---|
+| Grafana | 3000 | UI, anonymous admin enabled |
+| OTEL Collector | 4317 (gRPC), 4318 (HTTP) | Receives OTLP from your app |
+| VictoriaMetrics | 8428 | Metrics TSDB (Prom remote_write in, MetricsQL out) |
+| VictoriaLogs | 9428 | Logs DB (OTLP in, LogsQL out) |
+
+## Run
+
+```bash
+# start the observability stack
+ctf obs up -vm
+# remove the stack with all the data (volumes)
+ctf obs d -vm
+# restart the stack removing all the data (volumes)
+ctf obs r -vm
+```
+
+## Developing
+
+Change compose files under `framework/cmd/observability` and restart the stack (removing volumes too)
+```
+just reload-cli && ctf obs r
+```
+
+## Local Dashboards (Docker)
+
+You can create a dashboard using [UI](http://localhost:3000) and put them under `$pwd/dashboards` folder then commit, they'll be loaded automatically on start and you can find them [here](http://localhost:3000/dashboards) under `local` directory.
+
+`$pwd` is you current working directory from which you call `ctf obs u`

framework/.changeset/v0.16.4.md

@@ -0,0 +1 @@
+- VictoriaMetrics stack in CTF, docs

framework/cmd/main.go

@@ -19,6 +19,7 @@ import (
 
 func main() {
 	app := &cli.App{
+		Version:   "v0.16.4",
 		Name:      "ctf",
 		Usage:     "Chainlink Testing Framework CLI",
 		UsageText: "'ctf' is a useful utility that can:\n- clean up test docker containers\n- modify test files\n- create a local observability stack with Grafana/Loki/Pyroscope",
@@ -535,12 +536,21 @@ Be aware that any TODO requires your attention before your run the final test!
 								Usage:   "Spin up all the observability services",
 								Value:   false,
 							},
+							&cli.BoolFlag{
+								Name:    "victoria",
+								Aliases: []string{"vm"},
+								Usage:   "Spin up all the observability services (VictoriaMetrics)",
+								Value:   false,
+							},
 						},
 						Description: "Spins up a local observability stack. Has two modes, standard (Loki, Prometheus, Grafana and OTEL) and full including also Tempo, Cadvisor and PostgreSQL metrics",
 						Action: func(c *cli.Context) error {
 							if c.Bool("full") {
 								return framework.ObservabilityUpFull()
 							}
+							if c.Bool("victoria") {
+								return framework.ObservabilityVictoriaMetricsUp()
+							}
 							return framework.ObservabilityUp()
 						},
 					},
@@ -555,9 +565,20 @@ Be aware that any TODO requires your attention before your run the final test!
 								Usage:   "Removes all the observability services (this flag exists for compatibility, all the services are always removed with 'down')",
 								Value:   false,
 							},
+							&cli.BoolFlag{
+								Name:    "victoria",
+								Aliases: []string{"vm"},
+								Usage:   "Spin up all the observability services (VictoriaMetrics)",
+								Value:   false,
+							},
 						},
 						Description: "Removes local observability stack",
-						Action:      func(c *cli.Context) error { return framework.ObservabilityDown() },
+						Action: func(c *cli.Context) error {
+							if c.Bool("victoria") {
+								return framework.ObservabilityVictoriaDown()
+							}
+							return framework.ObservabilityDown()
+						},
 					},
 					{
 						Name:    "restart",
@@ -570,16 +591,30 @@ Be aware that any TODO requires your attention before your run the final test!
 								Usage:   "Restart all observability services (this flag exists for compatibility, all the services are always removed with 'down')",
 								Value:   false,
 							},
+							&cli.BoolFlag{
+								Name:    "victoria",
+								Aliases: []string{"vm"},
+								Usage:   "Spin up all the observability services (VictoriaMetrics)",
+								Value:   false,
+							},
 						},
 						Description: "Restart a local observability stack",
 						Action: func(c *cli.Context) error {
-							// always remove all the containers and volumes to clean up the data
-							if err := framework.ObservabilityDown(); err != nil {
-								return err
+							if c.Bool("victoria") {
+								if err := framework.ObservabilityVictoriaDown(); err != nil {
+									return err
+								}
+							} else {
+								if err := framework.ObservabilityDown(); err != nil {
+									return err
+								}
 							}
 							if c.Bool("full") {
 								return framework.ObservabilityUpFull()
 							}
+							if c.Bool("victoria") {
+								return framework.ObservabilityVictoriaMetricsUp()
+							}
 							return framework.ObservabilityUp()
 						},
 					},

framework/observability.go

@@ -14,6 +14,13 @@ import (
 var EmbeddedObservabilityFiles embed.FS
 
 const (
+	LGTMDockerComposePath            = "compose"
+	VictoriaMetricsDockerComposePath = "compose-victoria-metrics"
+
+	LocalVictoriaMetrics   = "http://localhost:8428"
+	LocalVictoriaLogs      = "http://localhost:9428"
+	LocalOTELCollectorHTTP = "http://localhost:4318"
+	LocalOTELCollectorgRPC = "http://localhost:4317"
 	LocalGrafanaBaseURL    = "http://localhost:3000"
 	LocalLokiBaseURL       = "http://localhost:3030"
 	LocalPrometheusBaseURL = "http://localhost:9099"
@@ -181,7 +188,7 @@ func ObservabilityUpOnlyLoki() error {
 	if err != nil {
 		return err
 	}
-	composeDir := filepath.Join(obsDir, "compose")
+	composeDir := filepath.Join(obsDir, LGTMDockerComposePath)
 	_ = DefaultNetwork(nil)
 	if err := NewPromtail(); err != nil {
 		return err
@@ -208,7 +215,7 @@ func ObservabilityUp() error {
 	if err != nil {
 		return err
 	}
-	composeDir := filepath.Join(obsDir, "compose")
+	composeDir := filepath.Join(obsDir, LGTMDockerComposePath)
 	_ = DefaultNetwork(nil)
 	if err := NewPromtail(); err != nil {
 		return err
@@ -228,6 +235,34 @@ func ObservabilityUp() error {
 	return nil
 }
 
+// ObservabilityVictoriaMetricsUp VictoriaMetrics stack for load testing and performance investigations
+func ObservabilityVictoriaMetricsUp() error {
+	L.Info().Msg("Creating local observability stack (VictoriaMetrics)")
+	if err := extractAllFiles("observability"); err != nil {
+		return err
+	}
+	obsDir, err := getObservabilityDir()
+	if err != nil {
+		return err
+	}
+	composeDir := filepath.Join(obsDir, VictoriaMetricsDockerComposePath)
+	_ = DefaultNetwork(nil)
+	err = RunCommand("bash", "-c", fmt.Sprintf(`
+		cd %s && \
+		docker compose up -d
+	`, composeDir))
+	if err != nil {
+		return err
+	}
+	fmt.Println()
+	L.Info().Msgf("Grafana: %s", LocalGrafanaBaseURL)
+	L.Info().Msgf("OTEL Collector HTTP: %s", LocalOTELCollectorHTTP)
+	L.Info().Msgf("OTEL Collector gRPC: %s", LocalOTELCollectorgRPC)
+	L.Info().Msgf("VictoriaMetrics: %s", LocalVictoriaMetrics)
+	L.Info().Msgf("VictoriaLogs: %s", LocalVictoriaLogs)
+	return nil
+}
+
 // ObservabilityUpFull full stack for load testing and performance investigations
 func ObservabilityUpFull() error {
 	L.Info().Msg("Creating full local observability stack")
@@ -238,7 +273,7 @@ func ObservabilityUpFull() error {
 	if err != nil {
 		return err
 	}
-	composeDir := filepath.Join(obsDir, "compose")
+	composeDir := filepath.Join(obsDir, LGTMDockerComposePath)
 	_ = DefaultNetwork(nil)
 	if err := NewPromtail(); err != nil {
 		return err
@@ -260,13 +295,27 @@ func ObservabilityUpFull() error {
 	return nil
 }
 
+func ObservabilityVictoriaDown() error {
+	L.Info().Msg("Removing local observability stack (Victoria Metrics)")
+	obsDir, err := getObservabilityDir()
+	if err != nil {
+		return err
+	}
+	composeDir := filepath.Join(obsDir, VictoriaMetricsDockerComposePath)
+	_ = RunCommand("bash", "-c", fmt.Sprintf(`
+		cd %s && \
+		docker compose down -v
+	`, composeDir))
+	return nil
+}
+
 func ObservabilityDown() error {
 	L.Info().Msg("Removing local observability stack")
 	obsDir, err := getObservabilityDir()
 	if err != nil {
 		return err
 	}
-	composeDir := filepath.Join(obsDir, "compose")
+	composeDir := filepath.Join(obsDir, LGTMDockerComposePath)
 	_ = RunCommand("bash", "-c", fmt.Sprintf(`
 		cd %s && \
 		docker compose down -v && docker rm -f promtail

framework/observability/compose-victoria-metrics/README.md

@@ -0,0 +1,43 @@
+# Local VictoriaMetrics + VictoriaLogs + OTEL stack
+
+Minimal setup for experimenting with OTEL metrics & logs in Grafana.
+
+## Components
+
+| Service | Port | Purpose |
+|---|---|---|
+| Grafana | 3000 | UI, anonymous admin enabled |
+| OTEL Collector | 4317 (gRPC), 4318 (HTTP) | Receives OTLP from your app |
+| VictoriaMetrics | 8428 | Metrics TSDB (Prom remote_write in, MetricsQL out) |
+| VictoriaLogs | 9428 | Logs DB (OTLP in, LogsQL out) |
+
+## Run
+
+```bash
+docker compose up -d
+```
+
+Grafana: <http://localhost:3000> (no login required, anonymous admin).
+
+Both datasources (`VictoriaMetrics`, `VictoriaLogs`) are auto-provisioned.
+
+## Point your Go app at it
+
+In the OTEL exporter, use endpoint `localhost:4317` (gRPC) or `localhost:4318` (HTTP). Both metrics and logs go to the same collector — it fans them out to VM/VL.
+
+## Tear down
+
+```bash
+docker compose down          # keep data
+docker compose down -v       # wipe volumes
+```
+
+## Quick sanity checks
+
+```bash
+# Metrics ingest
+curl http://localhost:8428/api/v1/query?query=up
+
+# Logs ingest
+curl 'http://localhost:9428/select/logsql/query?query=*&limit=10'
+```