> For the complete documentation index, see [llms.txt](https://docs.groundcover.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.groundcover.com/collect-data/data-sources/profiling-endpoints.md). # Profiling Endpoints Send or scrape application profiles and uncover performance bottlenecks Profiles show which functions in your application spend CPU time, allocate memory, or wait on locks. groundcover can receive profiles in two ways: | Method | How it works | Use it for | Default | Minimum sensor version | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | ------------------------------------- | ---------------------- | | [Push profiles](#push-profiles) | Your application or collector sends raw pprof or OTLP Profiles to the sensor or to your BYOC endpoint. | Applications or collectors that already produce pprof or OTLP Profiles, inside or outside Kubernetes | Enabled, no Helm configuration needed | 1.13.1 | | [Scrape Go pprof endpoints](#scrape-go-pprof-endpoints) | The sensor pulls profiles from annotated pods that expose Go's `net/http/pprof` handlers. | Go applications running in Kubernetes, without changing how they send data | Disabled, enable in Helm | 1.13.114 | Both methods use the same profiling pipeline and attribute profiles to the source workload. ## Push profiles Send profiles generated inside your Kubernetes cluster to the sensor. Send profiles generated outside the cluster to your BYOC endpoint. ### Where to send profiles | Profiles generated | Endpoint | Authentication | | ------------------------------ | -------------------------------------------------- | -------------- | | Inside the Kubernetes cluster | `groundcover-sensor.groundcover.svc.cluster.local` | Not required | | Outside the Kubernetes cluster | `{BYOC_ENDPOINT}` | Ingestion key | Replace the service name or `groundcover` namespace if you use a different Helm release name or namespace. See [Ingestion Endpoints](/collect-data/byoc/ingestion-endpoints.md) for authentication methods and instructions for finding your BYOC endpoint. ### Accepted formats * **Raw pprof:** A Google pprof protobuf body, uncompressed or gzip-compressed. Send it with `POST /profiling/upload` over HTTP port `8028` in-cluster or HTTPS port `443` externally. Include either the `workload` or `serviceName` query parameter. * **OTLP Profiles over HTTP:** A protobuf-encoded OTLP `ExportProfilesServiceRequest` with `Content-Type: application/x-protobuf`. Send it with `POST /v1development/profiles` over HTTP port `4318` in-cluster or HTTPS port `443` externally. * **OTLP Profiles over gRPC:** An OTLP Profiles export using `opentelemetry.proto.collector.profiles.v1development.ProfilesService/Export` on port `4317` in-cluster or port `443` externally. {% hint style="info" %} Raw pprof uploads are limited to 32 MiB before and after decoding. Each pprof can contain up to 64 sample types and 10,000 decoded sample rows. {% endhint %} ## Scrape Go pprof endpoints The sensor can periodically pull pprof profiles from Go applications that expose the standard `net/http/pprof` handlers. Each sensor scrapes only the annotated pods on its own node, at the pod IP, so a pod is never scraped by more than one sensor. The scraped profiles go through the same pipeline as raw pprof uploads and keep the source pod's namespace, pod, container and workload. Scraping is disabled by default. To use it, expose pprof in your application, enable scraping in Helm, and annotate the pods to scrape. ### Expose pprof in your application Import `net/http/pprof` and serve it on a dedicated port. Listen on all interfaces: the sensor connects to the pod IP, so a server bound to `localhost` can't be scraped. ```go package main import ( "log" "net/http" _ "net/http/pprof" // registers the /debug/pprof/ handlers ) func main() { go func() { log.Println(http.ListenAndServe(":6060", nil)) }() // ... the rest of your application } ``` {% hint style="info" %} Go records block and mutex profiles only after your application enables them, for example with `runtime.SetBlockProfileRate(10000)` and `runtime.SetMutexProfileFraction(10)`. Until then those profiles have no samples and nothing is stored for them. {% endhint %} ### Enable scraping ```yaml agent: sensor: receivers: profiles: scrape: enabled: true interval: 1m ``` All settings are under `agent.sensor.receivers.profiles.scrape`: | Setting | Default | Description | | ------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | `false` | Scrape annotated pods. | | `interval` | `1m` | How often each target is scraped. The minimum is `10s`. A profile type whose timeout is longer than the interval isn't scraped: `cpu` needs at least `20s`, `block` and `mutex` need at least `40s`. | | `defaultPort` | `0` | Port to scrape when a pod sets neither `.port` nor `.port_name`. See [Choose the port](#choose-the-port). | | `extraAnnotationPrefixes` | `[]` | Additional annotation prefixes to read. See [Use your own annotation prefix](#use-your-own-annotation-prefix). | Only the sensor DaemonSet scrapes. Ingestor replicas never do. ### Annotate your pods Opt pods in per profile type by adding annotations to the pod template: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: checkout namespace: shop spec: selector: matchLabels: app: checkout template: metadata: labels: app: checkout annotations: profiles.groundcover.com/cpu.scrape: "true" profiles.groundcover.com/cpu.port: "6060" profiles.groundcover.com/memory.scrape: "true" profiles.groundcover.com/memory.port: "6060" profiles.groundcover.com/goroutine.scrape: "true" profiles.groundcover.com/goroutine.port: "6060" spec: containers: - name: checkout image: ports: - name: pprof containerPort: 6060 ``` Each annotation is `profiles.groundcover.com/.`. Settings don't carry over between profile types, so repeat `.port` and any other setting for each type you scrape. | Annotation | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `.scrape` | Set to `"true"` to scrape this profile type. Any other value is ignored. | | `.port` | Port number the pprof server listens on. | | `.port_name` | Name of a TCP container port to scrape instead of a number. | | `.path` | Request path and query. Defaults to the type's path in the table below. | | `.scheme` | `http` (default) or `https`. HTTPS targets are scraped without certificate verification, because pod IPs never match a serving certificate. | | `.sample_types` | Comma-separated pprof sample types to keep, for example `inuse_space,inuse_objects`. Unset keeps every sample type. A name the profile doesn't contain fails the scrape. | The profile types and the requests the sensor sends by default: | Type | Default request | Profile | Timeout | | ----------- | --------------------------------- | ---------- | ------- | | `cpu` | `/debug/pprof/profile?seconds=10` | 10s window | 20s | | `memory` | `/debug/pprof/heap` | Snapshot | 10s | | `goroutine` | `/debug/pprof/goroutine` | Snapshot | 10s | | `block` | `/debug/pprof/block?seconds=30` | 30s window | 40s | | `mutex` | `/debug/pprof/mutex?seconds=30` | 30s window | 40s | A windowed profile covers only its window once per interval. With the defaults, a CPU profile covers 10 seconds of every minute. The sensor picks up new pods and annotation changes automatically. {% hint style="warning" %} Go runs only one CPU profile at a time. If another profiler already profiles the same process, such as a continuous profiling SDK or Grafana Alloy, CPU scrapes fail with `target_busy`. groundcover doesn't read Grafana's `profiles.grafana.com` annotations for this reason. {% endhint %} ### Choose the port The sensor picks the port for each profile type in this order: 1. The `.port` annotation. 2. The `.port_name` annotation. 3. The `defaultPort` Helm setting, if it isn't `0`. 4. A container port whose name ends with `-profiles`, then one whose name ends with `-metrics`, then one named `http2`. If none applies, the target is skipped. To scrape without port annotations, give the pprof port a name ending with `-profiles`: ```yaml spec: template: metadata: annotations: profiles.groundcover.com/cpu.scrape: "true" profiles.groundcover.com/memory.scrape: "true" spec: containers: - name: checkout image: ports: - name: go-profiles containerPort: 6060 ``` Profiles are attributed to the running container that declares the scraped port, including native sidecars. If the port isn't declared, the profile is attributed to a container only when the pod has a single container. Declare the pprof port as a `containerPort` so profiles are attributed to the right container. ### Customize requests Override the path, scheme or sample types for a profile type: ```yaml annotations: profiles.groundcover.com/cpu.scrape: "true" profiles.groundcover.com/cpu.port_name: "pprof" profiles.groundcover.com/cpu.path: "/internal/pprof/profile?seconds=15" profiles.groundcover.com/cpu.scheme: "https" profiles.groundcover.com/memory.scrape: "true" profiles.groundcover.com/memory.port_name: "pprof" profiles.groundcover.com/memory.sample_types: "inuse_space,inuse_objects" ``` A path must start with `/`. For `cpu`, `block` and `mutex`, a path without `seconds` gets the type's default window. A `seconds` value must be at least `1` and leave 5 seconds of the type's timeout: up to `15` for `cpu` and `35` for `block` and `mutex`. Otherwise the target is skipped. The sensor masks every query value except `seconds` in its logs. Redirects aren't followed. Request headers and authentication aren't supported. ### Use your own annotation prefix To annotate pods under your own domain, add it to `extraAnnotationPrefixes`: ```yaml agent: sensor: receivers: profiles: scrape: enabled: true extraAnnotationPrefixes: - profiles.acme.com ``` Pods can then use `profiles.acme.com/cpu.scrape`, `profiles.acme.com/cpu.port`, and so on. `profiles.groundcover.com` is always read. A pod annotated under several prefixes is scraped once per profile type, with all settings for that type taken from the first prefix it uses: `profiles.groundcover.com`, then the list in order. {% hint style="warning" %} Don't add `profiles.grafana.com` if Grafana Alloy also profiles your pods. Both would request CPU profiles from the same process and one of them would fail. {% endhint %} ### Limits * Each response is limited to 8 MiB compressed and 32 MiB decompressed. * Each profile is limited to 10,000 decoded sample rows, counted after `.sample_types` is applied. A Go heap profile has four sample types, so a large heap profile can exceed the limit. Keep fewer sample types with `memory.sample_types`, or raise `agent.sensor.receivers.profiles.maxDecodedSampleRows`. Higher limits use more sensor memory. * Each sensor runs up to 64 scrapes at a time. Scrapes beyond that are skipped until the next interval. * A failing or slow target doesn't delay other targets. It's retried on the next interval. ### Troubleshoot scraping The sensor logs scraping problems. To find them, run the following command, replacing `groundcover` if you installed groundcover in a different namespace: ```bash kubectl logs -n groundcover -l app=sensor -c sensor --tail=-1 \ | grep -E "profile scrap|profile types with invalid" ``` | Log message | Meaning | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `profile scrape targets skipped` | Targets the sensor won't scrape, with an example pod and the cause. The `reason` field is `no_port` (no port found), `invalid_port` (`.port` isn't a port number) or `invalid_target` (invalid path, scheme, `seconds` or sample types). Logged when the problem first appears. | | `profile scrape failed` | A scrape failed. The `result` field gives the reason, and `namespace` and `pod` name the target. Logged on the first failure and every 10th consecutive failure. | | `profile scrape target recovered` | A failing target succeeded again. | | `profile types with invalid scrape settings are not scraped` | `interval` is too short for some profile types. Those types aren't scraped and the others are. | | `profile scraping is disabled` | The scrape settings are invalid, for example an `interval` below `10s` or an invalid `extraAnnotationPrefixes` entry. | Common `result` values: | Result | What to check | | --------------------- | ----------------------------------------------------------------------------------------------------------------- | | `connection_error` | The pprof server isn't reachable at the pod IP and port. Make sure it listens on all interfaces, not `localhost`. | | `timeout` | The target didn't respond within the type's timeout. | | `http_4xx` | The path is wrong or the endpoint requires authentication. | | `http_3xx` | The endpoint redirects. Set `.path` to the final path. | | `target_busy` | Another profiler holds Go's CPU profiler. | | `validation_error` | `.sample_types` names a type the profile doesn't have, or the profile exceeds the sample row limit. | | `too_large` | The response exceeds the size limits. | | `skipped_concurrency` | More than 64 scrapes were running on the node. | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.groundcover.com/collect-data/data-sources/profiling-endpoints.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.