OpenTelemetry documentation (#739)

Documentation for opentelemetry feature https://codeberg.org/forgejo/forgejo/pulls/3972

Reviewed-on: https://codeberg.org/forgejo/docs/pulls/739
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: TheFox0x7 <thefox0x7@gmail.com>
Co-committed-by: TheFox0x7 <thefox0x7@gmail.com>

docs/admin/config-cheat-sheet.md

@@ -1095,6 +1095,37 @@ This section only does "set" config, a removed config key from this section won'
 - `ENABLED_ISSUE_BY_REPOSITORY`: **false**: Enable issue by repository metrics with format `gitea_issues_by_repository{repository="org/repo"} 5`.
 - `TOKEN`: **\<empty\>**: You need to specify the token, if you want to include in the authorization the metrics . The same token need to be used in prometheus parameters `bearer_token` or `bearer_token_file`.
 
+## OpenTelemetry (`opentelemetry`)
+
+- `ENABLED`: **false**: Feature flag toggle.
+- `SERVICE_NAME`: **forgejo**: Service name for the application
+- `RESOURCE_ATTRIBUTES`: **\<empty\>**: Comma separated custom attributes for the application
+- `RESOURCE_DETECTORS`: **\<empty\>**: Active decoders for attributes, available values:
+  - `sdk` - adds information about opentelemetry sdk used by Forgejo
+  - `process` - adds information about the process
+  - `os` - adds information about the OS Forgejo is running on
+  - `host` - adds information about the host Forgejo is running on
+- `TRACES_SAMPLER` **parentbased_always_on**: Set sampler used by trace exporter, accepted values are:
+  - `always_off` - never samples spans
+  - `always_on` - always samples spans
+  - `traceidratio` - samples based on given ratio given in SAMPLER_ARG
+  - `parentbased_always_off` - samples based on parent span, never samples spans without parent spans
+  - `parentbased_always_on` - samples based on parent span, always samples spans without parent spans
+  - `parentbased_traceidratio` - samples based on parent span, samples spans without parent spans on given ratio given in SAMPLER_ARG
+- `TRACER_SAMPLER_ARG`: **\<empty\>**: Argument for the sampler, only applies to traceidratio based samplers. `traceidratio` expects a value between 0-1, based on which it samples (`0` it acts like `always_off`, `1` like `always_on`).
+- `TRACES_EXPORTER`: **otlp**: Controls which exporter should traces use, setting it to `none` disablses traces.
+
+## OpenTelemetry Traces (`opentelemetry.exporter.otlp`)
+
+- `ENDPOINT`: **http://localhost:4318**: Endpoint for OTLP trace exporter, accepts URLs in format `http://localhost:4318`. Setting `unix://` or `http://` disables secure transport.
+- `PROTOCOL`: **http/protobuf**: Controls which protocol is used for exports. Supports `http/protobuf` and `grpc`.
+- `COMPRESSION`: **\<empty\>**: Enable compression, accepted values are `gzip` and ``, defaults to no compression
+- `TIMEOUT`: **10s**: Sets timeout on the trace exporter
+- `HEADERS`: **\<empty\>**: Comma separated additional headers for the trace exporter, accepts headers in `key=value` format. Overwrites general `opentelemetry.HEADERS` on conflicts.
+- `CERTIFICATE`: **\<empty\>**: Path to certificate for TLS
+- `CLIENT_CERTIFICATE`: **\<empty\>**: Path to client certificate for TLS
+- `CLIENT_KEY`: **\<empty\>**: Path to client key for TLS
+
 ## API (`api`)
 
 - `ENABLE_SWAGGER`: **true**: Enables the API documentation endpoints (`/api/swagger`, `/api/v1/swagger`, …). True or false.

docs/admin/index.md

@@ -23,6 +23,7 @@ These documents are targeted to people who run Forgejo on their machines.
 - [Recommended Settings and Tips](./recommendations/)
 - [GPG Commit Signatures](./signing/)
 - [Moderation tools](./moderation/)
+- [OpenTelemetry](./otel/)
 - [Adopt existing git directories](./adopt/)
 - [Interface customization](./customization/)
 - [OAuth2 provider](./oauth2-provider/)

docs/admin/otel.md

@@ -0,0 +1,42 @@
+---
+title: 'Open Telemetry integration'
+license: 'CC-BY-SA-4.0'
+---
+
+[OpenTelemetry](https://opentelemetry.io/) is a open source standard which allows for instrumentation of an application.
+It enabled administrator to measure Forgejo performance to find potential issues
+and act on them before they become unmanagable or just for debugging performance.
+
+Forgejo currently implements only tracing part of the specification, with metrics handled by Prometheus.
+
+## Quick Setup
+
+Administrator can enable the feature by setting the feature flag `[opentelemetry].ENABLED` to `true`. The predefined exporter configuration will try to send data to `localhost:4318` via `http/protobuf` protocol.
+
+## Resources
+
+Administrator can set the name and custom attributes to enrich the exported items with more metadata. By default Forgejo sets `service.name` to `forgejo`, but this can be changed with `[opentelemetry].SERVICE_NAME` setting.
+Custom resource attributes can be set with `[opentelemetry].RESOURCE_ATTRIBUTES` in format `key=value,key2=value2`. It's best to follow the [Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/resource/) for resource while adding custom attributes as collection stores may visualize them better or handle in a more specific way.
+
+Additionally more metadata can be automatically added using resource detectors controllable with `[opentelemetry].RESOURCE_DETECTORS`. Available options can be found in [config cheat sheet](../config-cheat-sheet/#opentelemetry-opentelemetry). The detectors aren't a stable Open Telemetry specification and can be removed or added later releases.
+
+## Tracing
+
+Traces are one of signals in Open Telemetry specification. Gathering of them can be turned off and on individually by setting `[opentelemetry].TRACES_EXPORTER=none` to disable exporter and therefore traces. Alternatively it can be set to `otlp`, which is the default and only other supported value, to enable the OTLP exporter.
+
+Which spans will be sent can be infuenced with `[opentelemetry].TRACES_SAMPLER` and `[opentelemetry].TRACES_SAMPLER_ARG` options. Not all samplers support the argument. More details can be found in the [config cheat sheet](../config-cheat-sheet/#opentelemetry-opentelemetry).
+
+## OTLP Exporter
+
+The default exporter is configurable in `[opentelemetry.exporter.otlp]` section and it's subsections if needed. It follows naming of the [opentelemetry exporter configuration options](https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options) all of which are listed below:
+
+- `ENDPOINT=http://localhost:4318` - urlf for the collector
+- `CERTIFICATE=` - path to certificate for TLS
+- `CLIENT_CERTIFICATE=` - path to client certificate for TLS
+- `CLIENT_KEY=` - path to client key for TLS
+- `COMPRESSION=` - compression switch, can be set to gzip
+- `HEADERS=` - key value list of headers to use
+- `PROTOCOL=http/protobuf` - which protocol to use for exports, supports `grpc` and `http/protobuf`
+- `TIMEOUT=10s` - controls the timeout of the export
+
+If `ENDPOINT` contains `unix` or `http` the insecure flag is transparently set.