Metrics
For a basic monitoring setup, enable the chart’s ServiceMonitor and use its alerts and Grafana dashboards.
monitoring: serviceMonitor: enabled: trueThis requires the Prometheus Operator. For a failed or stuck review, the dashboard Queue is usually the quickest place to start. The tables below are the metric reference for custom queries and alerts.
The management port serves /metrics alongside /healthz and /readyz.
Apart from the Go runtime and process collectors and
promhttp_metric_handler_errors_total, every series is prefixed kritika_.
None is labelled by a reviewed pull request or commit. A scrape that fails
is logged at error level and counted in
promhttp_metric_handler_errors_total by cause: gathering or encoding. The
chart’s monitoring.serviceMonitor.enabled creates a Prometheus Operator
ServiceMonitor for it.
| Series | Labels | What it counts |
|---|---|---|
kritika_webhooks_total |
connection, outcome | deliveries: enqueued, recorded, skipped, ignored, ping, unsigned (the App has no webhook secret), unauthorized, unparsable, too_large, unreadable, unknown_connection (with no connection), undeclared_account, error |
kritika_polls_total |
connection, outcome | backstop polls: ok, error |
kritika_polled_pull_requests_total |
connection | open pull requests the backstop polls handed to ingest |
kritika_reviews_total |
account, status | reviews by terminal status: completed, skipped, capped, superseded, canceled, failed |
kritika_review_duration_seconds |
account | a review’s wall time, from its first job’s pickup to its terminal status, across its stages |
kritika_review_stage_duration_seconds |
account, stage | durations of preparation and publication operations, failed attempts included. Nested stages overlap |
kritika_review_slot_wait_seconds |
account, model | actual elapsed slot-snooze intervals observed at the next pickup, queue delay included |
kritika_findings_total |
account, severity, category | findings posted |
kritika_confidence_scores_total |
account, score, risk | reviews the confidence model scored, by score (0 to 5) and risk: low, medium, high, critical |
kritika_confidence_held_total |
account, what | confidence answers kritika held to its own rule: risk given as no level and taken as critical, step raised to a moved version’s step floor, floor raised to a path’s floor (confidence) |
kritika_followups_total |
account, outcome | mentions, and replies in finding threads that do not mention the bot, handled: answered, limited, ignored, failed |
kritika_threads_total |
account, outcome | finding threads a person resolved or unresolved: dismissed, addressed, restored, ignored, failed |
kritika_index_runs_total |
account, mode, status | index runs |
kritika_index_chunks_total |
account | chunks embedded into the index |
kritika_context_chunks_total |
account, stage | context chunks a review’s prompt was given, by stage: overlay, definition, caller, similar (the index) |
kritika_runner_runs_total |
account, kind, outcome | runner Jobs: success, failed, deadline |
kritika_runner_duration_seconds |
kind | a runner Job’s time from start to finish |
kritika_lease_wait_seconds |
account, model | time waiting for a model concurrency slot |
kritika_review_snoozes_total |
account, model | reviews put back on the queue because every model slot was held |
kritika_jobs_rescued_total |
kind, state | jobs a replica died working, handed back to the queue by the leader: retryable, discarded (no attempts left), cancelled (a cancel was asked) |
kritika_model_calls_total |
account, model, role, outcome | model calls: ok, error. model is the one that answered, so OpenRouter’s fallback shows under its own. role on this and the series below is review (a split review’s summary merge included), followup, confidence or embedding |
kritika_model_tokens_total |
account, model, role, direction | tokens: input, cached, output |
kritika_model_cost_usd_total |
account, model, role | provider-reported cost |
kritika_model_unpriced_calls_total |
account, model, role | calls with no reported cost and no configured price, which kritika_model_cost_usd_total leaves out |
kritika_forge_rate_limits_total |
connection, outcome | responses GitHub refused for a rate limit. waited: the request waited the limit out and was sent again. refused: the wait was over a minute, or the request cannot be resent |
kritika_egress_requests_total |
kind, outcome | requests runner pods made through the gateway. kind: connect or http. outcome: allowed, refused, error |
kritika_transcript_writes_total |
kind, outcome | model calls recorded for the transcript view. kind: agent_step (reviews and follow-ups), confidence, or merge (a split review’s summary). outcome: ok, error |
kritika_db_pool_connections |
pool, state | connections the app and owner pools hold, by state: idle, acquired, constructing |
kritika_db_pool_max_connections |
pool | each pool’s ceiling. A pool at its ceiling queues jobs and requests |
kritika_db_pool_acquires_total, kritika_db_pool_empty_acquires_total, kritika_db_pool_acquire_seconds_total |
pool | connections taken from a pool, how many had to wait for one to free, and the time spent waiting |
kritika_config_drift |
1 while this replica’s file differs from the applied one | |
kritika_config_error |
1 while the leader’s latest attempt to apply the configuration to the store was refused. The last applied configuration stays live meanwhile | |
kritika_leader |
1 while this replica holds the leader lock | |
kritika_build_info |
version, revision | always 1, labelled with the running build’s version and the commit it was built from |
Queue and vector-search measurements
Section titled “Queue and vector-search measurements”| Series | Labels | Meaning |
|---|---|---|
kritika_queue_runnable_jobs |
queue | Due available, retryable and scheduled jobs in the shared database |
kritika_queue_oldest_runnable_seconds |
queue | Time since the oldest runnable job’s scheduled_at. Zero when empty |
kritika_queue_stats_up |
Whether this replica read the queue snapshot successfully | |
kritika_queue_wait_seconds |
queue | Histogram from scheduled_at to worker start, recorded for every pickup |
kritika_vector_search_seconds |
account, outcome | Histogram of database search transaction time, outcome ok, error or canceled |
Each replica reads the same queue state on scrape with a two-second timeout.
Use max by (namespace, job, queue) for backlog and oldest age, never a sum
across replicas. Keep the deployment’s cluster label as well when querying
multiple clusters. A failed read emits kritika_queue_stats_up = 0 and omits
the queue gauges. It does not report an empty queue or retain stale values.
Known queues emit zero when empty. Queue labels are limited to review,
review_prepare, review_publish, followup, thread and index.
Preparation and publication each have their own worker capacity and retry
attempts. Thread jobs already queued on followup remain attributed to that
queue until they finish.
Runnable work includes overdue retries and scheduled jobs that wait for
River’s scheduler to promote them. It excludes future work, pending jobs,
running jobs and terminal jobs. A paused review, followup or thread queue
still counts its overdue work, so it can raise KritikaQueueDelayed.
Oldest age measures time since eligibility, not time since creation. Queue wait also excludes intentional delays: scheduling, snooze and retry backoff. It records another sample on each pickup. kritika clamps negative waits from clock skew to zero. Recording at worker entry makes delays visible before long jobs finish.
Histograms record different work on each replica, so sum their bucket rates before computing quantiles:
histogram_quantile(0.95, sum by (namespace, job, queue, le) (rate(kritika_queue_wait_seconds_bucket[5m])))Vector timing begins after embedding, lease release and usage recording. It
includes pool acquisition, account transaction setup, all vectors in the
request, result decoding and commit. It excludes embedding-provider latency,
model-slot waits and result filtering. Requests without a compatible index or
with an embedding failure produce no vector-search sample. The histogram’s
_count provides search volume and database error counts. Use the existing
model-call counters for embedding failures. No repository, query text, job ID
or generation becomes a metric label.
Client cancellations and expired request contexts during the database phase
use outcome="canceled". That outcome is excluded from both the numerator and
the denominator of the error ratio. All outcome series are initialized at zero
for configured accounts before workers start. Prometheus can then count a
first error after it scrapes a zero sample. increase() still cannot
reconstruct events before the first scrape.
The delay alert uses the latest successful snapshot within five minutes to bridge short read failures. A fresh empty snapshot clears it. The snapshot-success ratio also detects repeated failures. Queue snapshots use the application pool and its least-privilege role, and acquisition time counts against the two-second budget. Pool saturation is a legitimate snapshot failure. Inspect the existing pool-wait metrics alongside queue health when diagnosing it.
Download cache
Section titled “Download cache”The gateway exports cache telemetry per replica, and Prometheus aggregates the replicas. The admin UI’s Download cache card only reports the responding replica’s configuration. It does not probe storage or claim that storage is healthy.
| Metric | Labels | Meaning |
|---|---|---|
kritika_cache_requests_total |
outcome | One decision per admitted HTTP request while caching is enabled: local_hit, s3_hit, miss, or bypass |
kritika_cache_bypasses_total |
reason | Why a request was forwarded without the cache |
kritika_cache_operation_seconds |
operation, outcome | Duration histogram for origin_fetch, s3_lookup, s3_read, and s3_upload |
kritika_cache_transfer_bytes_total |
operation | Actual body bytes consumed by cache fills, S3 reads/uploads, or accepted by response writers (response_local, response_s3) |
kritika_cache_responses_total |
source, outcome | Responses after serving, by local/s3 source and ok/error/canceled result |
kritika_cache_active_fills |
Current origin fills. Excludes background S3 uploads | |
kritika_cache_local_bytes |
Retained files at the latest directory scan | |
kritika_cache_local_limit_bytes |
Retained-entry budget, not a hard disk limit | |
kritika_cache_evictions_total |
Local entries evicted to meet the budget | |
kritika_cache_upload_skips_total |
reason | Publications skipped for queue_full, circuit_open, entry_unavailable, or present, the digest’s entry another replica published |
Bypass reasons are bounded: ineligible, fill_error,
entry_unavailable, canceled, fill_busy, fill_capacity, too_large,
private, encoded, unknown_length, and digest_mismatch for an
origin’s body without the digest the request named. URLs, object keys, run IDs and
arbitrary errors are not metric labels. CONNECT tunnels are excluded.
A miss is a request that the gateway serves after a wait for a fill. This
includes requests that share one in-process fill. Failed fills report a
bypass instead. The hit ratio is hits divided by hits plus misses. Inspect
bypasses alongside it. Operation outcomes are ok, error, and canceled,
plus these:
missfor an absent or invalid S3 entry, or for one replaced or removed while it was being read.skippedfor an origin response that cannot be stored.
Cache availability does not affect readiness. A failed or circuit-open S3 lookup is a miss that fills locally, not a bypass. Circuit-open lookups and skipped uploads do not contribute to S3 operation counts, so they cannot dilute the storage-error rate. Read and upload circuits recover independently. Uploads run in the background, and their duration is separate from origin fill time.
Transfer counters include partial operations. Origin bytes cover cache fill attempts, not all gateway forwarding or estimated bandwidth savings. Upload bytes count the HTTP request body, but not the SDK’s checksum pass. Response bytes count writes that the HTTP writer accepts, with multipart framing included. HEAD and 304 responses contribute zero.
A hit does not prove delivery. To find stream failures and client cancellations, read the response results. S3 read duration includes the wait for the downstream reader.
Local usage excludes fills in progress and evicted files that are still held open. It is updated at startup and at eviction scans. Compare it with the entry budget to understand eviction pressure. Use pod filesystem metrics for actual disk usage. Bucket usage and lifecycle retention belong in the storage provider’s monitoring.
At debug level, egress cache download records a completed serving attempt with
cache_key, source, result, HTTP status, method, bytes and elapsed time.
Storage failures produce egress cache storage error at warning level, limited
to one warning per minute per replica. Every failed operation still appears in
metrics. The runner’s gateway_cache install field indicates the route used.
It is not proof of a cache hit. A retry around the cache has its own warning.
Most used tools
Section titled “Most used tools”The admin UI’s Most used tools graph ranks the top ten command names over 7, 30 or 90 days, across currently served accounts. Each started process counts, including nonzero exits, timeouts and retry runs. Rejected commands, failed installations and processes that fail to start do not count. Versions and providers share a command name. The counts are executions, not download counts.
Counts are stored in agent_runs.command_counts alongside the saved result,
so they survive restarts. They also aggregate across replicas without
Prometheus labels for arbitrary command names. Split review parts share one
counter. The period uses the result’s save time. Active runs, and runs lost
before saving their result, are not included.
Alerts
Section titled “Alerts”The chart’s monitoring.prometheusRule.enabled creates a Prometheus Operator
PrometheusRule with the alerts below, grouped by the metrics Service’s
namespace and job. Each alert’s wait is set by a value
(monitoring.prometheusRule.*For). additionalRuleLabels adds a routing
label to every rule, and additionalRuleAnnotations adds an annotation.
| Alert | Fires when | Severity | Wait |
|---|---|---|---|
KritikaNoLeader |
sum(kritika_leader) == 0: no replica holds the leader lock |
critical | 5m |
KritikaMultipleLeaders |
sum(last_over_time(kritika_leader[1m])) > 1: more than one replica reported holding it within multipleLeadersWindow |
critical | 2m |
KritikaConfigError |
kritika_config_error == 1: the leader failed to apply the file |
warning | 5m |
KritikaConfigDrift |
kritika_config_drift == 1: a replica’s file differs from the applied one |
warning | 15m |
KritikaQueueDelayed |
Oldest runnable review/followup/thread job exceeds 600 seconds. Uses the last successful sample within 5m and max across replicas | warning | 15m |
KritikaQueueStatsUnavailable |
Every replica succeeds on fewer than 75% of snapshot reads over 5m | warning | 5m |
KritikaCacheStorageErrors |
More than 10% S3 operation errors, with at least 10 non-canceled operations in 15m. Cold misses are not errors | warning | 10m |
KritikaVectorSearchErrors |
More than 10% database search errors, with at least 10 non-canceled requests in 15m | warning | 10m |
Without a leader, nothing migrates, applies configuration, polls or rescues the jobs a dead replica left. A standby takes the lock within one leader retry interval once its owner connection reaches the database. Postgres drops a dead leader’s session, and with it the lock, within about a minute. Two leaders can show for one retry interval during a database failover, before the old leader’s lock connection fails. If they show for longer than that, the replicas are not on one database. Drift is expected for the minute or two a rollout takes, until the new pod leads and applies its file.
Queue age and alert waits are configurable under monitoring.prometheusRule.
The defaults are starting thresholds, not measured production SLOs. Tune
queue age for normal review traffic. Index work remains visible on the
dashboard but is excluded from the delay alert, because onboarding
deliberately waits behind higher-priority updates. Vector latency is graphed
without an alert threshold until a representative production baseline is
available.
Kubernetes Events
Section titled “Kubernetes Events”kritika records Events that kubectl describe and kubectl events show and
that an event collector gathers. On a runner Job, an Event says why kritika
ended its run. On a kritika serve pod, it says what its leader did.
| On | Reason | Type | When |
|---|---|---|---|
| a runner Job | Superseded |
Normal | a newer head of the pull request superseded the run. The note names it |
| a runner Job | Replaced |
Normal | a newer review of the pull request replaced the run |
| a runner Job | Cancelled |
Normal | the run was cancelled from the dashboard |
| a runner Job | HeartbeatLost |
Warning | the runner’s heartbeat went stale, so kritika ended the run |
| a runner Job | Disrupted |
Warning | Kubernetes stopped its pod, by a node drain, a preemption or an eviction. The note says how |
| a runner Job | NeverStarted |
Warning | its pod failed to start, as for an image that cannot be pulled or a pod no node takes. Or its pod was never created, as when the namespace refuses it on admission. The note says why |
| a runner Job | Interrupted |
Normal | kritika serve stopped, and the job runs again |
| a runner Job | Interrupted |
Warning | the job’s heartbeat stopped landing, so kritika handed it back to its queue |
| a runner Job | TimedOut |
Warning | the job ran out of time |
| a runner Job | StatusUnreadable |
Warning | kritika lost sight of the Job’s status, as when someone else deleted it, so it deleted the Job |
| a kritika serve pod | LeaderElected |
Normal | the replica took the lead |
| a kritika serve pod | ConfigApplied |
Normal | the leader applied the configuration, named by its hash |
| a kritika serve pod | ConfigRefused |
Warning | the store refused the configuration, and the last applied one stays live, as kritika_config_error says too |
A run that ended on its own, finished or failed, gets none of these. For such
a run, the Job controller’s own Events on its Job tell what happened.
kritika deletes the Job of a run it ends, so
kubectl events --for job/<name> finds those Events once kubectl describe
no longer can. A pod’s Events need its name and UID, which the chart passes
as KRITIKA_POD_NAME and KRITIKA_POD_UID. The chart’s Role lets kritika
create Events in its namespace. An Event it cannot record is logged.
Kubernetes keeps Events for an hour unless the API server’s --event-ttl
says otherwise.
Grafana dashboard
Section titled “Grafana dashboard”The chart’s monitoring.dashboards.enabled renders a Grafana dashboard of
most of the series above (charts/kritika/dashboards/kritika.json). It is a
ConfigMap labelled grafana_dashboard: "1" for the kube-prometheus-stack
sidecar. With monitoring.dashboards.grafanaOperator.enabled, it is a
GrafanaDashboard for grafana-operator, which imports the ConfigMap into the
Grafana instance that grafanaOperator.matchLabels selects.
The dashboard is filtered by the metrics Service’s namespace and job,
then by account and model. Its rows follow a review’s path:
- health (leaders, configuration, drift, rescued jobs)
- intake (webhooks, backstop polls, rate limits)
- reviews (status, duration, findings, follow-ups, the wait for a model slot)
- models (calls, tokens, cost per hour and over the time range)
- runner Jobs, the gateway’s egress and the index
- database pools
- queue delays and vector searches
- the gateway’s download cache
Queue panels span all accounts. Vector panels follow the account filter.
Reproducing the vector-search baseline
Section titled “Reproducing the vector-search baseline”Preparation also emits INFO logs for diagnosing time outside the vector-search histogram:
parts context builtrecords the split review’s shared scan counts and elapsed time.similar-code request endedrecords the runner’s whole retrieval request.- The gateway separates
similar embedding slotwait time fromsimilar queries embeddedprovider time. review model slots grantedrecords the worker’s estimated part count and the slots granted, both by model. Compare it withagent started in partsfor the actual part count and initial slots.- During a split review,
review model slots adjustedrecords changes to the worker’s grant, withslotsandremaining_partsby model. Remaining parts include active work and exclude completed checkpoints.
The worker and runner read capacity every two seconds, so a review can gain slots after it starts. Recorded reviews that wait for their first slot take priority over extra slots: the grant drops by what they are short of, and the worker lets a slot go once the runner’s parts cannot be in it. Elapsed fields are Go durations, and the JSON logger encodes them as nanoseconds. Split parts retrieve similar code concurrently, bounded by their granted parallelism. These request durations can overlap, so do not sum them to estimate preparation wall time.
Run on a machine with Docker through the integration task, which creates and removes a disposable database. No provider credentials or model calls are needed:
mise run test-integration -- -run '^$' -bench BenchmarkSimilarChunks -benchtime=20x -vBenchmarkSimilarChunks exercises the SimilarChunks SQL with
vchordrq.prefilter = on, 1,024 and 1,536 dimensions, and 1,000 or 10,000
chunks in the target generation. A second repository in the same account and
a second account each contribute the same number of vectors: 3,000 or 30,000
total rows.
Each case excludes either no paths or 10% of target paths. Data comes from a
fixed-seed, 16-cluster synthetic distribution with one chunk per path.
All retrieval runs as kritika_app with RLS (row-level security) enabled, the
transaction’s account setting, the generation filter and the actual path
exclusions. A foreign-account lookup must return no results. Recall@4 compares
the application’s results with an exact table scan for 20 deterministic queries.
The harness disables index and bitmap scans only for that reference. It makes sure that the reference uses a sequential-scan plan. It restores those scans before further application queries. It compares chunk paths directly because fixture paths have a distinct prefix for each generation. This also keeps the ground truth on the same full-result SQL as the application, rather than on a separate row-identifier query. VectorChord’s recall evaluator is another option for production query samples.
The verbose output captures EXPLAIN (ANALYZE, BUFFERS) for both plans, plus the
PostgreSQL and VectorChord versions. Setup, recall evaluation and plan collection
are excluded from the reported ns/op. Timing includes the account transaction,
vector serialization, search and result decoding for one query vector. It is a
warm microbenchmark with the integration task’s race detector enabled, not a
cold-start or concurrent-traffic test.
Local baseline, 2026-10-12
Section titled “Local baseline, 2026-10-12”The measurements ran on Linux amd64, AMD EPYC 7502P, with the task’s
ghcr.io/tensorchord/vchord-postgres:pg18-v1.1.1 container (PostgreSQL 18.3,
VectorChord 1.1.1). Each mean below covers 20 timed requests. EXPLAIN times are
individual diagnostic queries, not averages or percentiles.
| Dimensions | Target chunks | Excluded paths | Mean request (ms) | Recall@4 | VectorChord EXPLAIN (ms) | Exact EXPLAIN (ms) |
|---|---|---|---|---|---|---|
| 1,024 | 1,000 | 0 | 2.27 | 1.00 | 0.44 | 8.83 |
| 1,024 | 1,000 | 100 | 2.48 | 1.00 | 0.41 | 8.71 |
| 1,024 | 10,000 | 0 | 6.75 | 1.00 | 5.05 | 98.42 |
| 1,024 | 10,000 | 1,000 | 17.72 | 1.00 | 4.82 | 92.23 |
| 1,536 | 1,000 | 0 | 2.88 | 1.00 | 0.52 | 10.43 |
| 1,536 | 1,000 | 100 | 3.03 | 1.00 | 0.54 | 9.40 |
| 1,536 | 10,000 | 0 | 9.43 | 1.00 | 6.13 | 144.10 |
| 1,536 | 10,000 | 1,000 | 19.30 | 1.00 | 5.89 | 97.31 |
Every application plan chose index_chunks_embedding_idx. Every exact plan
used a sequential scan and top-N sort. The filters included both the generation
and the RLS account predicate. Plans reported shared-buffer hits for these warm
queries. The command above emits the full buffer counts.
The current index performed well on this fixture. The longer requests with large exclusion lists include their client-side encoding and transfer, so do not attribute all of that difference to the index. These small, synthetic samples are not evidence of production recall or an SLO. They do not justify a change to partitioning, search parameters, prewarming or worker concurrency. Before you tune that configuration, use the runtime histograms and repeat with representative embeddings and traffic. No production database was measured for this baseline.
Review completion timings
Section titled “Review completion timings”kritika_review_stage_duration_seconds separates the work after the runner
answers. Stage labels are fixed operation names:
| Stage | Interval |
|---|---|
agent_to_prepare |
Agent row creation to the start of a preparation attempt |
prepare |
Finding processing, summary merging and confidence scoring |
merge, confidence |
Each operation, with its model-slot wait and retries |
checkpoint |
Saving preparation or publication state and any queue handoff |
deliver |
Publication lock acquisition, delivery and persistence |
write_back |
Rendering and forge operations |
forge_sticky |
Finding, creating or updating the sticky comment |
forge_inline, forge_status |
Posting the inline review or setting commit status |
forge_approval, forge_resolve_threads |
Approval processing or resolving the batch of old threads |
forge_thread_states |
Reading back the state of every thread, when resolving left an old one open |
forge_summary_update |
Rewriting the summary with thread links and approval |
persist |
Committing the completed publication |
agent_to_published |
Agent row creation to successfully committed publication, across queue handoffs and retries |
These intervals overlap: do not sum stages or their percentiles. Forge stages are operations and can make multiple HTTP requests.
agent_to_* uses the database’s agent-record timestamp, which approximates
agent completion and requires aligned database and worker clocks. kritika
clamps negative durations to zero. A checkpoint without an agent row produces
no such sample. Other stages include failed attempts.
All review stage ended logs carry review, run, job, stage and
elapsed. JSON duration fields are nanoseconds.
For example, compare publication-tail p95 by account:
histogram_quantile(0.95, sum by (account, le) ( rate(kritika_review_stage_duration_seconds_bucket{stage="agent_to_published"}[1h])))Queue delay already uses kritika_queue_wait_seconds with
queue="review_prepare" or queue="review_publish". The
resuming review publication log adds review/run/job identity, attempt number,
queue_wait (since scheduled eligibility) and since_enqueued (including
retry backoff and earlier attempts). Both end at worker pickup.
Executor logs split completion handling into runner pods read,
runner log tail read and the enclosing runner result collected, joined by
job_name. runner job completion observed marks when the watch sees
completion. agent completion observed links that job to the run and reports
agent_to_executor_return and elapsed.
agent_to_executor_return ends when the supervised run returns to the worker,
so it includes log collection and any supervisor check still in progress.
elapsed also includes token revocation and loading the agent row. Their
individual durations appear in runner completion handled as revoke_elapsed
and row_read_elapsed. These logs distinguish lifecycle delay from log-read
latency.
kritika_review_slot_wait_seconds measures each recorded slot snooze from its
start to the next review pickup. Early wakes shorten it. Scheduling delays
lengthen it. For aggregate observed waiting time, sum
increase(kritika_review_slot_wait_seconds_sum[1h]) by account and model. To
inspect repeated waits, group review model slot wait ended logs by job.
This metric is separate from kritika_lease_wait_seconds, which measures waits
inside an active worker. The slot wait overlaps queue delay and must not be
added to it. Canceled jobs never picked up again have no sample.
A process crash before River saves cleared metadata can cause a repeated observation. A pickup that fails before the wait is recorded leaves it to the next attempt, retry backoff included. These are operational metrics, not an exactly-once accounting ledger.