[DOC-14397]: DOC: Application Telemetry needs additional info.

modules/metrics-reference/pages/application-telemetry-metrics.adoc

@@ -0,0 +1,119 @@
+= Application Telemetry Metrics
+:description: Couchbase can be configured to collect metrics related to Couchbase SDK calls made from applications. 
+:stem: asciimath
+:xrefstyle: short
+
+[abstract]
+{description}
+
+== Overview
+
+Couchbase Server can be configured to collect metrics related to Couchbase SDK calls made from applications. 
+The client SDKs send metrics to the Server that are then handled like regular server-side metrics, and you can access them through the Prometheus endpoint and the Stats API.
+
+== Prerequisites
+
+You must enable application telemetry collection on your Couchbase Server installation. 
+You can do this by executing the following curl command:
+
+[source, bash, subs="+quotes"]
+----
+curl -u Administrator:password -X POST \
+http[s]://##{host}##:##{port}##/settings/appTelemetry \
+-d enabled=true 
+----
+
+For more information about enabling application telemetry, see xref:rest-api:application-telemetry.adoc[]
+
+== Telemetry Counters
+
+Couchbase can record the number of SDK calls made for specific services. 
+The metrics follow the same basic patterns:
+
+[%header]
+|===
+| Metric| Description
+
+| `sdk_\{service}_r_total`
+| The total number of operations
+
+|`sdk_\{service}_r_canceled`
+| The total number of operations that were canceled.
+
+| `sdk_\{service}_r_timedout`
+| The total number of operations that timed out.
+
+|===
+
+To access the counter for a particular service, you replace `\{service}` with one of the following:
+
+* `kv`
+* `query`
+* `search`
+* `analytics`
+* `management`
+* `eventing`
+
+To access the number of calls that timed out for the Query Service, 
+you would use the command: `sdk_query_r_timedout`.
+
+== Histograms
+<<available-histograms>> below shows the available application telemetry metrics. 
+The `_bucket` at the end of the metric names is a special suffix that signifies that the line should be interpreted as a bucket of that histogram. 
+Also, there are 2 special metrics associated with each histogram: `_sum` and `_count` that represent the overall sum of all histogram buckets, 
+and the total number of the points in the histogram. 
+The `_sum` value should be a total sum of the points in milliseconds represented as an integer.
+
+[#available-histograms]
+=== Available Histograms
+
+* `sdk_kv_retrieval_duration_milliseconds_bucket`
+* `sdk_kv_retrieval_duration_milliseconds_count`
+* `sdk_kv_retrieval_duration_milliseconds_sum`
+* `sdk_kv_mutation_nondurable_duration_milliseconds_bucket`
+* `sdk_kv_mutation_nondurable_duration_milliseconds_count`
+* `sdk_kv_mutation_nondurable_duration_milliseconds_sum`
+* `sdk_kv_mutation_durable_duration_milliseconds_bucket`
+* `sdk_kv_mutation_durable_duration_milliseconds_count`
+* `sdk_kv_mutation_durable_duration_milliseconds_sum`
+* `sdk_query_duration_milliseconds_bucket`
+* `sdk_query_duration_milliseconds_count`
+* `sdk_query_duration_milliseconds_sum`
+* `sdk_search_duration_milliseconds_bucket`
+* `sdk_search_duration_milliseconds_count`
+* `sdk_search_duration_milliseconds_sum`
+* `sdk_analytics_duration_milliseconds_bucket`
+* `sdk_analytics_duration_milliseconds_count`
+* `sdk_analytics_duration_milliseconds_sum`
+* `sdk_eventing_duration_milliseconds_bucket`
+* `sdk_eventing_duration_milliseconds_count`
+* `sdk_eventing_duration_milliseconds_sum`
+* `sdk_management_duration_milliseconds_bucket`
+* `sdk_management_duration_milliseconds_count`
+* `sdk_management_duration_milliseconds_sum`
+
+
+
+[#histogram-output]
+.Histogram output
+====
+[source]
+----
+sdk_kv_retrieval_duration_milliseconds_bucket{le="1"} 24054
+sdk_kv_retrieval_duration_milliseconds_bucket{le="10"} 33444
+sdk_kv_retrieval_duration_milliseconds_bucket{le="100"} 100392
+sdk_kv_retrieval_duration_milliseconds_bucket{le="500"} 129389
+sdk_kv_retrieval_duration_milliseconds_bucket{le="1000"} 133988
+sdk_kv_retrieval_duration_milliseconds_bucket{le="2500"} 139823
+sdk_kv_retrieval_duration_milliseconds_bucket{le="+Inf"} 144320
+sdk_kv_retrieval_duration_milliseconds_sum{} 53423000
+sdk_kv_retrieval_duration_milliseconds_count{} 144320 
+----
+====
+xref:histogram-output[xrefstyle=short] demonstrates a series of `kv` retrieval times for the buckets in a node.
+Each line gives the retrieval count for a particular range. 
+For example, the `le="1"` label is the upper bound of the latency bucket, so that range is stem:[le 1 ms].
+The value would fall into all buckets where the upper bound is higher. 
+As shown for the retrieval duration values, the requests completed in stem:[le 1] ms total 24,054, those in stem:[le 10] ms total 33,444 and so on.
+
+There are 2 special metrics associated with each histogram: `_sum` and `_count` that represent the overall sum of all buckets, and total number of the points in the histogram. 

modules/metrics-reference/pages/metrics-reference.adoc

@@ -26,3 +26,5 @@ In this section, metrics are listed according to service, as follows:
 * xref:metrics-reference:ns-server-metrics.adoc[Cluster Manager Metrics]
 
 * xref:metrics-reference:xdcr-metrics.adoc[XDCR Metrics]
+
+* xref:rest-api:application-telemetry.adoc[Application Telemetry Metrics]

modules/rest-api/pages/application-telemetry.adoc

@@ -9,10 +9,40 @@
 == Description
 
 Having Couchbase Server collect telemetry from your applications can help you troubleshoot client issues.
-This telemetry data is useful to diagnose issues such as poor performance or timeouts.
+This visibility into metrics data from the application's point of view is useful to diagnose issues such as poor performance or timeouts.
 
-When you enable application telemetry, Couchbase Server advertises to SDK clients that it can collect telemetry data.
-When an SDK client connects to a cluster with application telemetry enabled, it opens a WebSocket connection to a node in the cluster.
+[mermaid]
+.Telemetry Architecture
+----
+flowchart LR
+Client@{shape:rectange, label: "SDK Client"}
+
+subgraph Node1 [Couchbase Node]
+    direction LR
+    Server1@{shape:rectangle, label: "Couchbase Server Cluster Manager"}
+    Prometheus1@{shape:rectangle, label: "Prometheus"}
+    Server1 -----> Prometheus1
+end
+
+subgraph Node2 [Couchbase Node]
+    direction LR
+    Server2@{shape:rectangle, label: "Couchbase Server Cluster Manager"}
+    Prometheus2@{shape:rectangle, label: "Prometheus"}
+    Server2 --> Prometheus2
+
+end
+
+Client --> Node1
+Client --> Node2
+Client -. websocket .-> Server2
+Server2 -. metrics forwarding .-> Server1
+
+----
+
+
+When you enable application telemetry, Couchbase Server advertises to SDK clients that it can collect telemetry data. 
+When an SDK client connects to a cluster with application telemetry enabled, it opens a WebSocket connection to a node in the cluster. +
+The clients send metrics to any node in the cluster — the cluster forwards the metrics on to other nodes. +
 Couchbase Server uses this connection to periodically gather telemetry data from the client in Prometheus format.
 
 NOTE: Application telemetry is off by default in Couchbase Server 8.0.
@@ -33,7 +63,6 @@ You cannot enable application telemetry if your cluster is running in mixed mode
 
 * Your applications must use a recent SDK version that supports application telemetry.
 The following table lists the SDKs that support application telemetry along with the version where they added support.
-
 +
 |===
 | SDK | Minimum Version with Application Telemetry Support
@@ -73,6 +102,41 @@ The following table lists the SDKs that support application telemetry along with
 The default management port is 8091 for unencrypted connections and 18091 for encrypted connections.
 Make sure any firewall rules between your clients and the nodes allow traffic on the management port.
 
+[IMPORTANT]
+.Enabling telemetry
+====
+
+Telemetry is turned off by default on Couchbase Server.
+
+Use the following `curl` command to view the telemetry state on your server:
+
+[source, bash]
+----
+curl -u Administrator:password -X GET http://localhost:8091/settings/appTelemetry | jq
+----
+
+This returns a JSON object that contains the telemetry state as `true` or `false`.
+
+[source, json]
+----
+{
+  "enabled": false,
+  "maxScrapeClientsPerNode": 1024,
+  "scrapeIntervalSeconds": 60
+}
+----
+
+Execute the following `curl` command to enable telemetry on your cluster.
+
+[source, bash, subs="+quotes"]
+----
+curl -u Administrator:password -X POST \
+http://localhoast:8091/settings/appTelemetry \
+-d enabled=true 
+----
+
+====
+
 == HTTP Methods 
 
 This API endpoint supports the following methods:
@@ -228,7 +292,7 @@ The default value is `60`.
 
 +
 You can increase this value to reduce the overhead of collecting telemetry data on your nodes.
-However, increasing this value means Couchbase Server will miss collecting more telemetry data from clients before they disconnect.
+However, increasing this value means Couchbase Server misses collecting more telemetry data from clients before they disconnect.
 For example, suppose you set this value to `300`.
 Then Couchbase Server could lose up to 5 minutes of telemetry data from a client that disconnects just before the next scheduled telemetry collection.
 
@@ -251,15 +315,15 @@ This object has the same format as the <<#get-status-responses,response from the
 
 `400 Bad Request`::
 Returned if you attempt to enable application telemetry on a cluster that's running in mixed mode where some nodes are running a version earlier than 8.0. 
-All of the nodes in the cluster must be running version 8.0 or later to enable application telemetry.
+All the nodes in the cluster must be running version 8.0 or later to enable application telemetry.
 See <<prerequisites>> for more requirements.
 
 `403 Forbidden`::
 Returned if you do not have the proper roles to call this API.
 See <<#config-privs>> for a list of the required roles.
 
 `404 Not Found`::
-Returned if you attempt to call the endpoint on a running a version of Couchbase Server earlier than 8.0. 
+Returned if you attempt to call the endpoint on a version of Couchbase Server earlier than 8.0. 
 
 [#config-examples]
 === Examples