L51.0 Get Metrics In with OpenTelemetry10%3.0 Monitor Using Built-in Content10%

Getting data in, and Kubernetes

Metrics reach Splunk Observability Cloud through the Splunk Distribution of the OpenTelemetry Collector, configured as receivers, processors and exporters wired together in pipelines. This lesson covers installing it on Linux, reading and editing its config, diagnosing common errors, and then using the Kubernetes pages that its Helm-deployed agents feed.

Before you can chart anything, something has to carry your data to Splunk. That carrier is the Collector, a small program that works like a mail room: packages come in, get sorted, and go out on trucks. You will learn how to install it, read its route sheet, spot the three classic ways it breaks, and then look at how Kubernetes data shows up as a building directory with an inspector's trouble list.

INTERACTIVE

Wire a metrics pipelinePlan the mail carrier's route

Every component below is defined in the config. Only the ones you switch on are listed in service: pipelines: metrics:, and only those run. Then break it on purpose.

The Collector is a mail carrier with three jobs: pick up mail (receivers), sort and bundle it (processors), and deliver it (exporters). The config file lists everyone it knows, but the carrier only stops where the route sheet says: that's the pipeline. Know someone who isn't on the route? No error. Their mail just never gets picked up.


        
01 · BLUEPRINT 1.5

What the Collector is and its building blocks

The Splunk Distribution of the OpenTelemetry Collector receives, processes and exports telemetry. The docs list it as supported on Kubernetes, Linux and Windows. Its configuration is built from a few component types.

  • Receivers bring data in.
  • Processors change data before export, for example by filtering.
  • Exporters send data to one or more backends.
  • Extensions add abilities such as health checks and diagnostics.
  • Connectors link two pipelines, acting as an exporter at the end of one and a receiver at the start of the next.

The service section lists the extensions and the pipelines. A pipeline ties receivers, processors and exporters together. You can define several pipelines, and one receiver or exporter can appear in more than one. A receiver or exporter does nothing until a pipeline lists it.

Think of the Collector as a mail room. Receivers are the loading dock where packages arrive. Processors are the sorting table where staff stamp, filter or bundle packages. Exporters are the outgoing trucks to the destination. The pipeline is the route sheet that says which dock feeds which table and which truck.

Key exam point: a dock or a truck that is not written on any route sheet never gets used, even if it is built and ready.

The mail room. Docks bring packages in, sorting tables change them, trucks send them out. The route sheet (pipeline) decides who is actually used.

Remember for the exam
The Splunk Distribution of the OpenTelemetry Collector is documented as supported on Kubernetes, Linux and Windows.doc
Receivers determine how data gets into the Collector, and exporters define where data goes.doc
Receivers and exporters are not enabled until they are listed in a pipeline.doc
A connector acts as an exporter at the end of one pipeline and a receiver at the start of another.doc
Extensions add capabilities such as diagnostics and health checks, and they are listed under the service section, not inside pipelines.doc
02 · BLUEPRINT 1.1

Installing the Collector on Linux

The installer script is the standard way to deploy the Collector on Linux. The host needs systemd, curl and sudo, and a supported distribution on amd64 or arm64 (for example Ubuntu, Debian, Red Hat, Amazon Linux). You download the script and run it with a realm, a memory value and your access token.

  • --realm sets the realm used to build the endpoints. The default is us0.
  • --memory is total memory in MiB. The default is 512.
  • The token comes after --, which is needed if the token starts with a dash.
  • --mode is agent (host monitoring) or gateway (data forwarding). The default is agent.

Other install paths exist: Docker image (Linux, multiarch), a standalone binary, and a tar file. For Docker and binary installs you set SPLUNK_REALM and SPLUNK_ACCESS_TOKEN as environment variables.

Installing the Collector is like hiring a mail-room clerk. The installer script is the hiring agency. It needs a few things on the building site first: a service manager (systemd), a way to download (curl) and admin rights (sudo).

You tell the agency three things: which realm (the region of Splunk's cloud your account lives in, such as us1), how much memory the clerk may use, and your access token (the badge that proves you may deliver to your org). The default clerk works on the host itself, which is called agent mode. A central depot that collects from many clerks is gateway mode.

Hiring agency. The script is the agency: tell it the region, the budget and show your badge, and a clerk starts work on the host.

Installer optionPurposeDefault
--realmRealm used for generated endpointsus0
--memoryTotal memory in MiB512
--mode agent|gatewayHost monitoring or data forwardingagent
--collector-config <path>Use a custom config fileDepends on mode
--listen-interface <ip>Address receivers listen on127.0.0.1 in agent mode
--uninstallRemove the CollectorNone
Remember for the exam
The Linux installer script requires systemd, curl and sudo.doc
The installer script supports amd64 and arm64 on listed Linux distributions such as Ubuntu, Debian, Red Hat and Amazon Linux.doc
The installer's --realm option defaults to us0 and sets the realm used for the generated endpoints.doc
The access token is passed after a double dash, which is required if the token starts with a dash.doc
The installer's --mode option accepts agent or gateway and defaults to agent.doc
The Collector Docker image is a multiarch manifest and needs SPLUNK_REALM and SPLUNK_ACCESS_TOKEN set, with no defaults.doc
03 · BLUEPRINT 1.2

Config files, environment variables and default components

The installer places files in /etc/otel/collector. agent_config.yaml is the agent-mode config, gateway_config.yaml is the gateway one, and splunk-otel-collector.conf holds the systemd environment variables, including the token and realm. After editing the environment file, restart the service.

The default agent config enables the hostmetrics receiver (10 second interval, with scrapers such as cpu, memory, disk, filesystem, network, load, paging and processes) and the signalfx exporter for metrics. Its metrics pipeline runs memory_limiter, batch and resourcedetection processors before exporting. Variables such as SPLUNK_ACCESS_TOKEN, SPLUNK_REALM, SPLUNK_INGEST_URL and SPLUNK_LISTEN_INTERFACE fill in the config at runtime. The health check extension listens on port 13133.

Back to the mail room. The route sheet is agent_config.yaml. The sticky note on the door with your badge number and region is splunk-otel-collector.conf. The route sheet says things like ${SPLUNK_ACCESS_TOKEN}, which means "copy the badge number from the sticky note".

Out of the box the clerk already reads host vitals (hostmetrics: CPU, memory, disk and so on) every 10 seconds and ships them out through the signalfx truck, which is the one that talks to Splunk Observability Cloud for metrics. Change the sticky note and you must restart the clerk.

Sticky note on the door. Secrets and region sit in one environment file. The config file borrows them by name, so you change them in one place.

Component typeWhat it doesExample
ReceiverBrings data into the Collectorhostmetrics (CPU, memory, disk), otlp (ports 4317 and 4318)
ProcessorChanges data before exportbatch, memory_limiter, resourcedetection
ExporterSends data to a backendsignalfx (metrics and events to Observability Cloud)
ExtensionAdds a capability, not in pipelineshealth_check (port 13133), zpages
ConnectorLinks two pipelinesExporter in one pipeline, receiver in the next
ServiceLists extensions and wires pipelinesservice: pipelines: metrics:
Remember for the exam
The Linux config files live in /etc/otel/collector: agent_config.yaml, gateway_config.yaml and splunk-otel-collector.conf.doc
splunk-otel-collector.conf holds the systemd environment variables, including the token and realm, and must have correct values when the Collector runs as a service.doc
The default hostmetrics receiver collects every 10 seconds, and the process scraper is disabled by default.doc
The default metrics pipeline sends hostmetrics and otlp data through memory_limiter, batch and resourcedetection to the signalfx exporter.doc
The health_check extension listens on port 13133 in the default config.doc
After editing the environment file, restart with sudo systemctl restart splunk-otel-collector.doc
04 · BLUEPRINT 1.2 · 1.3

Editing the config, plus agent and gateway modes

To add a component: define it under its section, add settings if needed (an empty entry often works because components have defaults), then list it in the matching pipeline under service.pipelines. Save and run sudo systemctl restart splunk-otel-collector. Processors run in the order the pipeline lists them. You can check a file before running with otelcol validate --config=sample.yaml.

The table shows a short annotated metrics-only config in the style of the defaults. It was written from the default config page and has not been run.

Agent mode runs on the host with the application. Gateway mode runs as a standalone service and receives data from agents, then forwards it to Observability Cloud. To route an agent through a gateway, point the agent's pipeline exporters at the gateway and set SPLUNK_GATEWAY_URL.

Adding a part to the mail room takes three moves: build it (define it), give it settings (often none), and write it on the route sheet (the pipeline). Skip the last move and nothing happens, and the Collector will not warn you. Then restart the clerk so he reads the new sheet.

An agent is a clerk on each floor. A gateway is a central depot. If you want floor clerks to hand packages to the depot instead of the post office, change their outgoing truck to the depot and tell them the depot's address (SPLUNK_GATEWAY_URL).

Route sheet. Build the part, then write it on the route sheet. The Collector never warns about a part that is built but unused.

Config lineWhat it means
extensions: health_check: endpoint: 0.0.0.0:13133Extension that answers 'is the Collector up' on port 13133
receivers: hostmetrics: collection_interval: 10sDefines the receiver and reads host stats every 10 seconds
scrapers: cpu: memory: load:Which host stats to gather (empty entries use defaults)
processors: batch: and resourcedetection: detectors: [system]Groups data into batches and adds host details
exporters: signalfx: access_token: ${SPLUNK_ACCESS_TOKEN}Metrics exporter; reads the token from an environment variable
api_url / ingest_url: https://ingest.${SPLUNK_REALM}.signalfx.comEndpoints built from the realm
service: extensions: [health_check]Turns the extension on (extensions are not in pipelines)
pipelines: metrics: receivers/processors/exportersThe route sheet: only components named here are used
Remember for the exam
A component must be listed in a pipeline under service.pipelines to take effect, and processors run in the order listed.doc
Adding an empty component entry is sometimes enough, because components often use default values.doc
The command otelcol validate --config=<file> checks a config file for format and reference errors before you run the Collector.doc
In host monitoring (agent) mode config, SPLUNK_GATEWAY_URL is set to the gateway address, and the agent's pipeline exporters are changed to point at it.doc
The default config includes an otlp_grpc/gateway exporter that sends to SPLUNK_GATEWAY_URL on port 4317, used only when routing through a gateway.doc
The hostmetrics scrapers can be turned on or off in splunk_metrics_config_linux.yaml, followed by a service restart.doc
05 · BLUEPRINT 1.4

Troubleshooting common errors

Most config failures are one of three kinds. Bad indentation stops the Collector: a receiver not indented under receivers: gives has invalid keys. A pipeline naming an undefined component stops it too, with references receiver "syslog" which is not configured. A defined component missing from every pipeline gives no error at all and the data simply never arrives.

Check a file with otelcol validate --config=&lt;file&gt;. Check the running service with the health check at http://localhost:13133 (look for "Server available"), and read logs with sudo journalctl -u splunk-otel-collector. HTTP errors: 401 means the access token or realm is wrong, 404 means a wrong endpoint or a network or firewall problem, 429 means the org is not provisioned for that traffic. If port 8888 is taken, change the internal metrics port. curl http://localhost:8888/metrics shows the Collector's own metrics, and splunk-support-bundle.sh in /etc/otel/collector gathers diagnostics for a support case.

Three ways a mail room breaks. (1) The route sheet is smudged (bad indentation): the clerk refuses to start. (2) The route sheet names a dock that was never built (undefined receiver): the clerk refuses and tells you which dock. (3) A dock is built but not on the route sheet: the clerk starts happily and nothing arrives. That last one is the sneaky one.

If packages are rejected at the post office: 401 means your badge (token) or region (realm) is wrong. 404 means the address or network path is wrong. 429 means you are sending more than your account allows. To peek inside, ask the clerk http://localhost:13133 if he is alive, and read his diary with journalctl.

Quiet failure. Loud failures are easy: the clerk will not start and names the problem. The dangerous one is the dock nobody wrote on the route sheet.

SymptomLikely causeFix
has invalid keysA component is not indented under its sectionIndent it two spaces under receivers:, processors: and so on
references receiver "x" which is not configuredPipeline names a receiver that is not definedDefine it, or remove it from the pipeline
No error, no dataReceiver defined but in no pipelineAdd it to service: pipelines:
HTTP 401Wrong access token or realmFix SPLUNK_ACCESS_TOKEN or SPLUNK_REALM, restart
HTTP 404Wrong endpoint, or network or firewall blockCheck URLs and outbound access
bind: address already in usePort clash on internal metrics port 8888Change the port, for example to 8889
Remember for the exam
A receiver that is not indented under receivers: produces an error that says the configuration has invalid keys.doc
A pipeline that names an undefined receiver fails with: references receiver "syslog" which is not configured.doc
A receiver defined but not listed in any pipeline produces no error, and data does not reach the back end.doc
The health check at http://localhost:13133 returns a JSON status of Server available when the Collector is running.doc
A 401 error means the configured access token or realm is incorrect.doc
sudo journalctl -u splunk-otel-collector shows the service logs on Linux.doc
splunk-support-bundle.sh in /etc/otel/collector collects config, status, logs, metrics and host information into a tar.gz in /tmp.doc
06 · BLUEPRINT 3.4 · 3.5 · 3.6

Kubernetes: Helm install, navigator and analyzer

The Collector reaches Kubernetes through a Helm chart. You add the chart repo, then helm install with splunkObservability.realm, splunkObservability.accessToken (an org token with ingest scope) and clusterName. The chart has three components: an agent DaemonSet (one per node, collects node-level data), a cluster receiver (a single-pod Deployment for cluster-wide data) and an optional gateway. The Kubernetes cluster receiver supports only the metrics pipeline, and one instance can monitor a whole cluster.

In Observability Cloud, open Infrastructure &gt; Kubernetes entities (the page earlier docs call the Kubernetes navigator). Entity types include clusters, nodes, pods, containers and workloads. Aggregate views are Table, Dashboard and Heatmap. The detail view has a Cluster map tab showing child resources. The K8s analyzer works for nodes, pods and containers and flags pending or failed pods, containers with high restart counts, nodes not ready, and nodes with high CPU or memory. It suggests shared properties, such as one workload or region, as filters.

Picture an apartment building. Each node is a floor, each pod is an apartment, each container is a room. Helm is the moving company that installs a clerk on every floor (the agent), one building-wide clerk (the cluster receiver) and, if the building is huge, a central depot (the gateway). You must give the movers your region, your badge and the building's name (clusterName).

The navigator is the building directory with a color view. The analyzer is the inspector's list of trouble spots: apartments that will not open, rooms that keep restarting, floors that are overheating. It also tells you what the broken ones have in common, such as the same landlord (workload). The Dashboard view shows which units use the most resources.

Apartment building. Floors are nodes, apartments are pods, rooms are containers. The analyzer lists the trouble spots.

Helm chart componentRuns asCollects
AgentDaemonSet, one per nodeNode-level data sources
Cluster receiverDeployment, single podCluster-wide data from the Kubernetes API
Gateway (optional)DeploymentReceives from agents and cluster receiver, forwards onward
Remember for the exam
Helm install needs splunkObservability.realm, splunkObservability.accessToken (ingest scope) and clusterName for Observability Cloud.doc
The Helm chart has an agent (DaemonSet), a cluster receiver (single-pod Deployment) and an optional gateway.doc
One instance of the Kubernetes cluster receiver can monitor an entire cluster, and it supports only the metrics pipeline.doc
The Kubernetes entities page lives under Infrastructure > Kubernetes entities and offers Table, Dashboard and Heatmap aggregate views.doc
The K8s analyzer is available for nodes, pods and containers and flags pending or failed pods, high restart counts, not-ready nodes and nodes with high CPU or memory.doc
The Cluster map tab in the detail view shows an interactive hierarchy of child resources for clusters, nodes, pods and containers.doc
Viewing navigator data needs the READ_NAVIGATOR permission and viewing dashboards needs READ_DASHBOARD.doc
HANDS ON

Install the Splunk OTel Collector and see host metrics arrive (realm us1)

No send.sh needed: the Collector's hostmetrics receiver sends real host data. You need an access token with ingest scope saved in ~/.config/splunk-o11y/ingest-token, plus Docker Desktop on a Mac or a Linux machine or VM. The installer script is Linux only (it needs systemd).

Steps
  1. Check the token file exists and is private, without printing it: run `test -s ~/.config/splunk-o11y/ingest-token && ls -l ~/.config/splunk-o11y/ingest-token`. If the mode is not -rw-------, run `chmod 600 ~/.config/splunk-o11y/ingest-token`. The token must be an org access token with ingest scope.
    You should see: One line showing the file with size above 0 and mode -rw-------. No token text on screen.
  2. Decide where to run it. The installer script needs Linux with systemd, so it will not run on a Mac. The easy path on macOS is Docker Desktop, which runs the official Linux image (multiarch, includes ARM64). Start Docker Desktop and run `docker info`.
    You should see: Server details print, no 'cannot connect to the Docker daemon' error. (If you prefer the real installer script, use a Linux VM and see the note at the end of the lab.)
  3. Make a private env file from the token file so the token never appears in your typed command: `umask 077; printf 'SPLUNK_ACCESS_TOKEN=%s\nSPLUNK_REALM=us1\n' "$(cat ~/.config/splunk-o11y/ingest-token)" > ~/.config/splunk-o11y/otel.env`
    You should see: No output. `ls -l ~/.config/splunk-o11y/otel.env` shows mode -rw-------.
  4. Start the Collector with the image defaults: `docker run --rm -d --env-file ~/.config/splunk-o11y/otel.env -p 13133:13133 --name otelcol quay.io/signalfx/splunk-otel-collector:latest`
    You should see: A container ID prints. `docker ps` lists otelcol as Up.
  5. Check health and logs: `curl http://localhost:13133` then `docker logs otelcol 2>&1 | tail -30`.
    You should see: JSON containing "status": "Server available". The logs show the Collector starting. A line mentioning 401 or unauthorized means the token or realm is wrong (the realm here is us1).
  6. In Observability Cloud (us1), go to Infrastructure and open the Hosts view (it was called Infrastructure > My Data Center > Hosts in the docs read for this lesson; the menu name may have changed). Find the new host. Its name will likely be the container ID. Then open the Metric Finder and search for `cpu`.
    You should see: A host appears within a minute or two, with CPU and memory charts. The metrics describe the Docker Desktop Linux VM or container, not your Mac itself.
  7. Write your own config. Run: `mkdir -p ~/otel-lab && cat > ~/otel-lab/good.yaml <<'YAML'` then paste: extensions: {health_check: {endpoint: 0.0.0.0:13133}} / receivers: {hostmetrics: {collection_interval: 10s, scrapers: {cpu: {}, memory: {}, load: {}}}} / processors: {batch: {}, resourcedetection: {detectors: [system], override: true}} / exporters: {signalfx: {access_token: '${SPLUNK_ACCESS_TOKEN}', api_url: 'https://api.${SPLUNK_REALM}.signalfx.com', ingest_url: 'https://ingest.${SPLUNK_REALM}.signalfx.com', sync_host_metadata: true}} / service: {extensions: [health_check], pipelines: {metrics: {receivers: [hostmetrics], processors: [batch, resourcedetection], exporters: [signalfx]}}} (write it as normal indented block YAML, one key per line, the same content as the annotated table in section 4), end with `YAML` on its own line. Then: `docker stop otelcol; docker run --rm -d --env-file ~/.config/splunk-o11y/otel.env -p 13133:13133 -v ~/otel-lab/good.yaml:/etc/collector.yaml:ro -e SPLUNK_CONFIG=/etc/collector.yaml --name otelcol quay.io/signalfx/splunk-otel-collector:latest`
    You should see: Container stays Up, `curl http://localhost:13133` still answers, and host metrics keep arriving. If it exits, `docker logs otelcol` names the problem (a typo in your YAML is the usual cause).
  8. BREAK IT ON PURPOSE (loud failure): copy good.yaml to bad.yaml and change the metrics pipeline line to `receivers: [hostmetrics, syslog]` without defining a syslog receiver. Stop the container and start it again with bad.yaml mounted instead, but without `--rm` and `-d`, so the error prints in your terminal.
    You should see: The Collector exits. The log has a line like: `service::pipelines::metrics: references receiver "syslog" which is not configured`, followed by `application run finished with error`.
  9. BREAK IT QUIETLY: in good.yaml, remove hostmetrics from the pipeline's receivers list but leave the receiver defined, so the pipeline reads `receivers: []`. Stop and start the container again with the file.
    You should see: No startup error, health check answers, but no new data points arrive for that host. This is the silent failure the exam asks about. Put hostmetrics back and restart to see data return.
  10. Clean up: `docker stop otelcol`. Delete ~/.config/splunk-o11y/otel.env if you do not plan to reuse it.
    You should see: Container gone. The host stops reporting and later shows as inactive.
Practice · lesson 5

Check yourself

Questions are worded like the real exam. Press 1 to 4 to answer and Enter to continue.

Questions are worded like the real exam. Read the explanation after each one, even when you get it right.