Wire a metrics pipelinePlan the mail carrier's route
Every component below is defined in the config. Only the ones you switch on are listed in service: pipelines: metrics:, and only those run. Then break it on purpose.
Metrics reach Splunk Observability Cloud through the Splunk Distribution of the OpenTelemetry Collector, configured as receivers, processors and exporters wired together in pipelines. This lesson covers installing it on Linux, reading and editing its config, diagnosing common errors, and then using the Kubernetes pages that its Helm-deployed agents feed.
Before you can chart anything, something has to carry your data to Splunk. That carrier is the Collector, a small program that works like a mail room: packages come in, get sorted, and go out on trucks. You will learn how to install it, read its route sheet, spot the three classic ways it breaks, and then look at how Kubernetes data shows up as a building directory with an inspector's trouble list.
Every component below is defined in the config. Only the ones you switch on are listed in service: pipelines: metrics:, and only those run. Then break it on purpose.
The Splunk Distribution of the OpenTelemetry Collector receives, processes and exports telemetry. The docs list it as supported on Kubernetes, Linux and Windows. Its configuration is built from a few component types.
The service section lists the extensions and the pipelines. A pipeline ties receivers, processors and exporters together. You can define several pipelines, and one receiver or exporter can appear in more than one. A receiver or exporter does nothing until a pipeline lists it.
Think of the Collector as a mail room. Receivers are the loading dock where packages arrive. Processors are the sorting table where staff stamp, filter or bundle packages. Exporters are the outgoing trucks to the destination. The pipeline is the route sheet that says which dock feeds which table and which truck.
Key exam point: a dock or a truck that is not written on any route sheet never gets used, even if it is built and ready.
The installer script is the standard way to deploy the Collector on Linux. The host needs systemd, curl and sudo, and a supported distribution on amd64 or arm64 (for example Ubuntu, Debian, Red Hat, Amazon Linux). You download the script and run it with a realm, a memory value and your access token.
--realm sets the realm used to build the endpoints. The default is us0.--memory is total memory in MiB. The default is 512.--, which is needed if the token starts with a dash.--mode is agent (host monitoring) or gateway (data forwarding). The default is agent.Other install paths exist: Docker image (Linux, multiarch), a standalone binary, and a tar file. For Docker and binary installs you set SPLUNK_REALM and SPLUNK_ACCESS_TOKEN as environment variables.
Installing the Collector is like hiring a mail-room clerk. The installer script is the hiring agency. It needs a few things on the building site first: a service manager (systemd), a way to download (curl) and admin rights (sudo).
You tell the agency three things: which realm (the region of Splunk's cloud your account lives in, such as us1), how much memory the clerk may use, and your access token (the badge that proves you may deliver to your org). The default clerk works on the host itself, which is called agent mode. A central depot that collects from many clerks is gateway mode.
| Installer option | Purpose | Default |
|---|---|---|
| --realm | Realm used for generated endpoints | us0 |
| --memory | Total memory in MiB | 512 |
| --mode agent|gateway | Host monitoring or data forwarding | agent |
| --collector-config <path> | Use a custom config file | Depends on mode |
| --listen-interface <ip> | Address receivers listen on | 127.0.0.1 in agent mode |
| --uninstall | Remove the Collector | None |
The installer places files in /etc/otel/collector. agent_config.yaml is the agent-mode config, gateway_config.yaml is the gateway one, and splunk-otel-collector.conf holds the systemd environment variables, including the token and realm. After editing the environment file, restart the service.
The default agent config enables the hostmetrics receiver (10 second interval, with scrapers such as cpu, memory, disk, filesystem, network, load, paging and processes) and the signalfx exporter for metrics. Its metrics pipeline runs memory_limiter, batch and resourcedetection processors before exporting. Variables such as SPLUNK_ACCESS_TOKEN, SPLUNK_REALM, SPLUNK_INGEST_URL and SPLUNK_LISTEN_INTERFACE fill in the config at runtime. The health check extension listens on port 13133.
Back to the mail room. The route sheet is agent_config.yaml. The sticky note on the door with your badge number and region is splunk-otel-collector.conf. The route sheet says things like ${SPLUNK_ACCESS_TOKEN}, which means "copy the badge number from the sticky note".
Out of the box the clerk already reads host vitals (hostmetrics: CPU, memory, disk and so on) every 10 seconds and ships them out through the signalfx truck, which is the one that talks to Splunk Observability Cloud for metrics. Change the sticky note and you must restart the clerk.
| Component type | What it does | Example |
|---|---|---|
| Receiver | Brings data into the Collector | hostmetrics (CPU, memory, disk), otlp (ports 4317 and 4318) |
| Processor | Changes data before export | batch, memory_limiter, resourcedetection |
| Exporter | Sends data to a backend | signalfx (metrics and events to Observability Cloud) |
| Extension | Adds a capability, not in pipelines | health_check (port 13133), zpages |
| Connector | Links two pipelines | Exporter in one pipeline, receiver in the next |
| Service | Lists extensions and wires pipelines | service: pipelines: metrics: |
To add a component: define it under its section, add settings if needed (an empty entry often works because components have defaults), then list it in the matching pipeline under service.pipelines. Save and run sudo systemctl restart splunk-otel-collector. Processors run in the order the pipeline lists them. You can check a file before running with otelcol validate --config=sample.yaml.
The table shows a short annotated metrics-only config in the style of the defaults. It was written from the default config page and has not been run.
Agent mode runs on the host with the application. Gateway mode runs as a standalone service and receives data from agents, then forwards it to Observability Cloud. To route an agent through a gateway, point the agent's pipeline exporters at the gateway and set SPLUNK_GATEWAY_URL.
Adding a part to the mail room takes three moves: build it (define it), give it settings (often none), and write it on the route sheet (the pipeline). Skip the last move and nothing happens, and the Collector will not warn you. Then restart the clerk so he reads the new sheet.
An agent is a clerk on each floor. A gateway is a central depot. If you want floor clerks to hand packages to the depot instead of the post office, change their outgoing truck to the depot and tell them the depot's address (SPLUNK_GATEWAY_URL).
| Config line | What it means |
|---|---|
| extensions: health_check: endpoint: 0.0.0.0:13133 | Extension that answers 'is the Collector up' on port 13133 |
| receivers: hostmetrics: collection_interval: 10s | Defines the receiver and reads host stats every 10 seconds |
| scrapers: cpu: memory: load: | Which host stats to gather (empty entries use defaults) |
| processors: batch: and resourcedetection: detectors: [system] | Groups data into batches and adds host details |
| exporters: signalfx: access_token: ${SPLUNK_ACCESS_TOKEN} | Metrics exporter; reads the token from an environment variable |
| api_url / ingest_url: https://ingest.${SPLUNK_REALM}.signalfx.com | Endpoints built from the realm |
| service: extensions: [health_check] | Turns the extension on (extensions are not in pipelines) |
| pipelines: metrics: receivers/processors/exporters | The route sheet: only components named here are used |
Most config failures are one of three kinds. Bad indentation stops the Collector: a receiver not indented under receivers: gives has invalid keys. A pipeline naming an undefined component stops it too, with references receiver "syslog" which is not configured. A defined component missing from every pipeline gives no error at all and the data simply never arrives.
Check a file with otelcol validate --config=<file>. Check the running service with the health check at http://localhost:13133 (look for "Server available"), and read logs with sudo journalctl -u splunk-otel-collector. HTTP errors: 401 means the access token or realm is wrong, 404 means a wrong endpoint or a network or firewall problem, 429 means the org is not provisioned for that traffic. If port 8888 is taken, change the internal metrics port. curl http://localhost:8888/metrics shows the Collector's own metrics, and splunk-support-bundle.sh in /etc/otel/collector gathers diagnostics for a support case.
Three ways a mail room breaks. (1) The route sheet is smudged (bad indentation): the clerk refuses to start. (2) The route sheet names a dock that was never built (undefined receiver): the clerk refuses and tells you which dock. (3) A dock is built but not on the route sheet: the clerk starts happily and nothing arrives. That last one is the sneaky one.
If packages are rejected at the post office: 401 means your badge (token) or region (realm) is wrong. 404 means the address or network path is wrong. 429 means you are sending more than your account allows. To peek inside, ask the clerk http://localhost:13133 if he is alive, and read his diary with journalctl.
| Symptom | Likely cause | Fix |
|---|---|---|
| has invalid keys | A component is not indented under its section | Indent it two spaces under receivers:, processors: and so on |
| references receiver "x" which is not configured | Pipeline names a receiver that is not defined | Define it, or remove it from the pipeline |
| No error, no data | Receiver defined but in no pipeline | Add it to service: pipelines: |
| HTTP 401 | Wrong access token or realm | Fix SPLUNK_ACCESS_TOKEN or SPLUNK_REALM, restart |
| HTTP 404 | Wrong endpoint, or network or firewall block | Check URLs and outbound access |
| bind: address already in use | Port clash on internal metrics port 8888 | Change the port, for example to 8889 |
The Collector reaches Kubernetes through a Helm chart. You add the chart repo, then helm install with splunkObservability.realm, splunkObservability.accessToken (an org token with ingest scope) and clusterName. The chart has three components: an agent DaemonSet (one per node, collects node-level data), a cluster receiver (a single-pod Deployment for cluster-wide data) and an optional gateway. The Kubernetes cluster receiver supports only the metrics pipeline, and one instance can monitor a whole cluster.
In Observability Cloud, open Infrastructure > Kubernetes entities (the page earlier docs call the Kubernetes navigator). Entity types include clusters, nodes, pods, containers and workloads. Aggregate views are Table, Dashboard and Heatmap. The detail view has a Cluster map tab showing child resources. The K8s analyzer works for nodes, pods and containers and flags pending or failed pods, containers with high restart counts, nodes not ready, and nodes with high CPU or memory. It suggests shared properties, such as one workload or region, as filters.
Picture an apartment building. Each node is a floor, each pod is an apartment, each container is a room. Helm is the moving company that installs a clerk on every floor (the agent), one building-wide clerk (the cluster receiver) and, if the building is huge, a central depot (the gateway). You must give the movers your region, your badge and the building's name (clusterName).
The navigator is the building directory with a color view. The analyzer is the inspector's list of trouble spots: apartments that will not open, rooms that keep restarting, floors that are overheating. It also tells you what the broken ones have in common, such as the same landlord (workload). The Dashboard view shows which units use the most resources.
| Helm chart component | Runs as | Collects |
|---|---|---|
| Agent | DaemonSet, one per node | Node-level data sources |
| Cluster receiver | Deployment, single pod | Cluster-wide data from the Kubernetes API |
| Gateway (optional) | Deployment | Receives from agents and cluster receiver, forwards onward |
No send.sh needed: the Collector's hostmetrics receiver sends real host data. You need an access token with ingest scope saved in ~/.config/splunk-o11y/ingest-token, plus Docker Desktop on a Mac or a Linux machine or VM. The installer script is Linux only (it needs systemd).
Questions are worded like the real exam. Press 1 to 4 to answer and Enter to continue.
Questions are worded like the real exam. Read the explanation after each one, even when you get it right.