Woodpecker v0.1.47
Release notes Home Operations GitHub

Operations

Two complementary guides for operating Woodpecker: a command-by-command reference and incident response workflows.

These pages include the local CLI additions beyond v0.1.46. Build wp from the current checkout for metadata repair, raw etcd access, truncation and the improved diagnostics.

Choose a guide

DocumentUse it when
CLI HandbookYou need a command’s syntax, parameters, examples, output interpretation or limitations. Covers environment, runtime progress, data diagnosis and metadata.
TroubleshootingYou are diagnosing read/write stalls or data corruption and need step-by-step recovery guidance.

Start with evidence

Confirm the selected context and address mappings, then distinguish runtime health, metadata consistency and actual data readability. The guides explain what each result can establish before an intervention.

CLI Handbook · Overview

Find an operation by its command group, understand its parameters, and interpret its output.

24 top-level commands/groups · 81 executable operations

New to wp? Start with installation, connection setup and your first commands →

The established operations are available in v0.1.47; Interactive shell and native diagnosis are source-build additions and are not included in v0.1.47. This reference is generated from source. Check wp version and wp --help against your binary before following an example.

Each topic has command syntax, parameter explanations, a demo, illustrative output and result interpretation. The overview lists executable operations: node is a command group, while node list is an operation within it.

All operations at a glance

Search by command, purpose or parameter, then click an operation to open its topic at the matching detail. Connection parameters are explained in CLI setup.

CLI setup · quick lookup

Operation Purpose Required / important parameters
Command group: wp completion
wp completion bash Generate the autocompletion script for the bash shell. No required command-specific arguments; uses selected context/defaults.
wp completion fish Generate the autocompletion script for the fish shell. No required command-specific arguments; uses selected context/defaults.
wp completion powershell Generate the autocompletion script for powershell. No required command-specific arguments; uses selected context/defaults.
wp completion zsh Generate the autocompletion script for the zsh shell. No required command-specific arguments; uses selected context/defaults.
Command group: wp ctx
wp ctx list List all contexts defined in cli.yaml No required command-specific arguments; uses selected context/defaults.
wp ctx use Switch current-context to the given name and persist to cli.yaml <name>: Local CLI context name
wp ctx view Print the resolved fields of the active context No required command-specific arguments; uses selected context/defaults.
Command group: wp report
wp report analyze Explicitly ask the configured Agent to explain supplied findings No required command-specific arguments; uses selected context/defaults.
wp report export Copy original JSON or derive a compact findings packet No required command-specific arguments; uses selected context/defaults.
wp report inspect Show recorded evidence detail without querying sources No required command-specific arguments; uses selected context/defaults.
Command: wp version
wp version Show wp CLI version and build info No required command-specific arguments; uses selected context/defaults.
Command: wp webui
wp webui Open a local operations and diagnosis workspace using cli.yaml No required command-specific arguments; uses selected context/defaults.

Interactive shell · quick lookup

Operation Purpose Required / important parameters
Command: wp shell
wp shell Open an interactive operational shell --context: service context, or use current-context / selection; connect attaches tenant etcd and optional storage inside the session

Process environment · quick lookup

Operation Purpose Required / important parameters
Command group: wp env
wp env diff Compare env / runtime / host / build across all nodes No required command-specific arguments; uses selected context/defaults.
wp env show Show env vars + runtime + host + build for a single node <node>: Discovered node ID

Cluster & node status · quick lookup

Operation Purpose Required / important parameters
Command group: wp cluster
wp cluster gossip-diff Compare memberlist views across nodes to detect partitions No required command-specific arguments; uses selected context/defaults.
wp cluster health Red/yellow/green health check for the cluster No required command-specific arguments; uses selected context/defaults.
wp cluster info Show cluster overview (nodes, state, topology) No required command-specific arguments; uses selected context/defaults.
Command group: wp node
wp node cancel-decommission Cancel an in-progress decommission (decommissioning -> active) <node>: Discovered node ID
wp node decommission Trigger graceful node decommission (blocks until safe by default) <node>: Discovered node ID
wp node drain-status Show (or watch) decommission progress for a node <node>: Discovered node ID
wp node healthz Query a node's health probe (the one readiness uses) <node>: Discovered node ID
wp node list List all server nodes in the cluster No required command-specific arguments; uses selected context/defaults.
wp node log-health Show per-log read/write health observed by a node <node>: Discovered node ID
wp node restart (Not implemented) Restart a node — use your orchestrator instead <node>: Discovered node ID
wp node show Show detailed status for a single node <node>: Discovered node ID

Kubernetes helpers · quick lookup

Operation Purpose Required / important parameters
Command group: wp k8s
wp k8s doctor Diagnose K8s cluster issues (not yet implemented) No required command-specific arguments; uses selected context/defaults.
wp k8s logs Tail logs from a Woodpecker pod <node-or-pod>: Actual pod name, or a node ordinal resolved with --wp-cluster
wp k8s scale Scale the cluster via kubectl patch --replicas int: Target replica count
wp k8s status Show cluster status via kubectl No required command-specific arguments; uses selected context/defaults.

Server configuration · quick lookup

Operation Purpose Required / important parameters
Command group: wp config
wp config diff Compare configurations across nodes No required command-specific arguments; uses selected context/defaults.
wp config show Show the resolved configuration of a single node <node>: Discovered node ID

Runtime log levels · quick lookup

Operation Purpose Required / important parameters
Command group: wp logging
wp logging get-level Show current log level of a node (or all nodes) No required command-specific arguments; uses selected context/defaults.
wp logging set-level Change log level of a node (or all nodes) No required command-specific arguments; uses selected context/defaults.

Profiles & trace · quick lookup

Operation Purpose Required / important parameters
Command: wp profile
wp profile Download a Go profile or execution trace from a node <node>: Discovered node ID

In-flight operations · quick lookup

Operation Purpose Required / important parameters
Command group: wp ops
wp ops list List in-flight operations <node>: Discovered node ID
wp ops show Show detail of a single in-flight operation <node>: Discovered node ID
--op-id string: Operation ID
wp ops stats Show op registry utilization and eviction statistics <node>: Discovered node ID

Metrics & scenarios · quick lookup

Operation Purpose Required / important parameters
Command group: wp metrics
wp metrics list List all metric series exposed by a node <node>: Discovered node ID
wp metrics report Run scenario-based metrics analysis --scenario: analysis scenario, or --list to enumerate scenarios
wp metrics snapshot Point-in-time metric snapshot from one or all nodes No required command-specific arguments; uses selected context/defaults.
wp metrics top Cross-node top-N comparison of a specific metric --by string: Metric name to rank by
wp metrics watch Real-time stream of a metric with trend arrows <metric-name>: Prometheus metric family to watch
<node>: Discovered node ID

Native diagnosis · quick lookup

Operation Purpose Required / important parameters
Command: wp diagnose
wp diagnose Diagnose current WP service state or sample a live window using existing No required command-specific arguments; uses selected context/defaults.

Writer runtime · quick lookup

Operation Purpose Required / important parameters
Command group: wp logstore
wp logstore buffer Show buffer bytes summary, sorted by size <node>: Discovered node ID
wp logstore compact Force compaction on a segment at an exact confirmed entry ID <node>: Discovered node ID
--expected-last-entry-id int: Exact LastEntryId from coordinator metadata; an incorrect value can publish an invalid footer (-1 only for an empty segment, required)
--log int: Log ID
--seg int: Segment ID
wp logstore fence Fence a segment on a single node. <node>: Discovered node ID
--log int: Log ID
--reason string: Reason for fencing
--seg int: Segment ID
wp logstore fence-quorum Fence a segment across enough of its quorum to interrupt writes (high-risk, requires --reason) <logName>: Logical log name in metadata
<segmentId>: Numeric segment ID within that log
--reason string: Reason for fencing
wp logstore flush-queue Show flush queue depth summary, sorted by depth <node>: Discovered node ID
wp logstore force-flush Force sync on a specific segment or all segments <node>: Discovered node ID
wp logstore lac Show how far a segment is confirmed readable across its quorum <logName>: Logical log name in metadata
<segmentId>: Numeric segment ID within that log
wp logstore segment-show Show a segment's active writer on a node (not historical segment metadata) <node>: Discovered node ID
--log int: Log ID
--seg int: Segment ID
wp logstore segments List active segments on a node <node>: Discovered node ID

Local data inventory · quick lookup

Operation Purpose Required / important parameters
Command group: wp instance
wp instance data List the instances holding node-local data. No required command-specific arguments; uses selected context/defaults.

Log progress & retention · quick lookup

Operation Purpose Required / important parameters
Command group: wp log
wp log readers Show where each of a log's readers has read to <logName>: Logical log name in metadata
wp log scan Compare metadata with replica local coverage; raw verifies local block checksums <logName>: Logical log name in metadata
For explicit storage: paired --bucket / --root-path
wp log skip-range add Declare an entry range unreadable so readers move past it. <logName>: Logical log name in metadata
<segmentId>: Numeric segment ID within that log
--reason string: Why these entries are being given up
--from-entry / --to-entry: inclusive entry range
--reason: accepted-loss reason; -y commits the reviewed declaration
wp log skip-range list Show the declared skip ranges. No required command-specific arguments; uses selected context/defaults.
wp log skip-range remove Withdraw a declared range, so readers stop skipping it. <logName>: Logical log name in metadata
<segmentId>: Numeric segment ID within that log
--from-entry / --to-entry: inclusive entry range
wp log truncate Advance the truncation frontier with the SDK's metadata-update semantics and additional operator guards <logName>: Logical log name in metadata
--entry int: Target entry ID (default -1)
--segment int: Target segment ID (default -1)
Apply only: --apply --expected-revision --reason --backup

Segment diagnosis · quick lookup

Operation Purpose Required / important parameters
Command group: wp segment
wp segment inspect Walk a segment's blocks on every replica and report what is wrong where <logName>: Logical log name in metadata
<segmentId>: Numeric segment ID within that log
For explicit storage: paired --bucket / --root-path
wp segment probe Ask every replica how far it can read a segment <logName>: Logical log name in metadata
<segmentId>: Numeric segment ID within that log
For explicit storage: paired --bucket / --root-path

Metadata inspection & repair · quick lookup

Operation Purpose Required / important parameters
Command group: wp meta
wp meta check Validate log/segment metadata constraints (does not verify physical data) <logName>: Logical log name in metadata
wp meta export Export one metadata namespace at a consistent etcd revision --output-file string: New snapshot file (required; never overwritten)
wp meta log list list log metadata and etcd revision No required command-specific arguments; uses selected context/defaults.
wp meta log patch Patch policy/statistics/timestamps, segment lastEntryId and state <logName>: Logical log name in metadata
--patch-file string: JSON object containing supported scalar fields
Apply only: --apply --offline --expected-revision --reason --backup
wp meta log show show log metadata and etcd revision <logName>: Logical log name in metadata
wp meta quorum list list quorum metadata and etcd revision No required command-specific arguments; uses selected context/defaults.
wp meta quorum show show quorum metadata and etcd revision <ID>: Numeric standalone quorum ID
wp meta segment list list segment metadata and etcd revision <logName>: Logical log name in metadata
wp meta segment patch Patch policy/statistics/timestamps, segment lastEntryId and state <logName>: Logical log name in metadata
<segmentId>: Numeric segment ID within that log
--patch-file string: JSON object containing supported scalar fields
Apply only: --apply --offline --expected-revision --reason --backup
wp meta segment show show segment metadata and etcd revision <logName>: Logical log name in metadata
<segmentId>: Numeric segment ID within that log

Compacted-mark distribution · quick lookup

Operation Purpose Required / important parameters
Command group: wp marking
wp marking confirm Confirm one compacted-mark distribution record. <logId>: Numeric log identity
<segmentId>: Numeric segment ID within that log
wp marking list List compacted-mark distribution records (default: only NOTIFY_PENDING_MANUAL) No required command-specific arguments; uses selected context/defaults.

Raw etcd exploration · quick lookup

Operation Purpose Required / important parameters
Command group: wp etcd
wp etcd cat Print only one key's value; use --encoding raw for exact bytes <key>: Exact raw etcd key
wp etcd get Show one exact key, revisions, lease and encoded value <key>: Exact raw etcd key
wp etcd list List raw keys by literal prefix; omit prefix to explore all keys No required command-specific arguments; uses selected context/defaults.

Historical WAL diagnosis · quick lookup

Operation Purpose Required / important parameters
Command group: wp diag
wp diag catalog List implemented checks, formulas, references and scenario manifests; no source/model queries No required command-specific arguments; uses selected context/defaults.
wp diag metrics check Run one check; no implicit execution of other checks <name>: Local CLI context name
wp diag scenario Run related Metrics/Trace checks and summarize their unchanged results by WAL step No required command-specific arguments; uses selected context/defaults.
wp diag trace check Run one check; no implicit execution of other checks <name>: Local CLI context name

Connection and naming

Follow CLI setup for a Helm-installed Milvus/Woodpecker deployment, including access from outside Kubernetes.

To retain connection settings while running several operations, use Interactive shell. Select the service context first, then explicitly attach the tenant's etcd metadata and optional storage location. The shell page explains which commands are available at each stage.

Name Meaning
logName / logId Logical metadata name / numeric log identity. Node-local logstore operations use the numeric ID.
meta-prefix Complete etcd namespace configured by the client, for example milvus-prod/woodpecker.
bucket + root-path The physical storage instance. Use both on scan, inspect and probe when automatic discovery cannot locate an empty or unloaded target. The instance data command uses --root.

Admin discovery supplies quorum identities; the workstation fans out to each node. Map every target to an accessible admin origin. A working seed connection alone does not establish peer reachability.

If the metadata namespace is unknown, start with Raw etcd exploration: list literal keys, narrow the prefix, then inspect a value. Those read-only operations need an etcd connection but no admin endpoint or metadata prefix.

For decoded records, metadata consistency and controlled correction, use Metadata inspection & repair. Normal retention advancement uses log truncate; it is separate from metadata patching and skip-range declarations.

Choose the intended change

Operation Intent / required preparation
meta log/segment patch Correct proven metadata errors using a JSON file containing only supported fields. Apply needs stopped writers/auditors, target revision, reason and a new backup.
log skip-range add Accept a precisely verified unreadable range when all usable alternatives fail. Requires explicit loss acceptance; it does not repair files or compaction.
log truncate Advance verified application retention. Apply needs the log revision, reason and a new backup; later auditor rounds retire earlier finalized segments.

Preview success does not commit a change. Metadata patch/truncate previews normally exit 0; skip-range and fencing previews can exit 7 because acceptance was not given. Verify application recovery independently after any intervention.

Shared parameters

Parameter / type Use and default
--admin-port int Admin port for fan-out peer discovery (default 9091)
--concurrency int Fan-out concurrency (default 8)
--context string CLI context name (overrides current-context in cli.yaml)
--endpoint string Admin HTTP seed endpoint (e.g. http://node:9091)
-h, --help help for wp
--no-color Disable color in output
--node-admin-url stringArray Map a node ID, advertised address, or host to an admin URL (KEY=URL; repeatable; overrides context mappings)
-o, --output string Output format: table|wide|json|yaml (default "table")
--strict Treat partial fan-out failures as errors
--timeout duration Per-request timeout (default 30s)
-v, --verbose count Increase verbosity (-v, -vv, -vvv)

Output formats vary by operation. Profile downloads binary data, and some diagnostic operations return findings with a nonzero exit code. Read the specific command's help and interpretation rather than treating every successful request as a health check.

The source-build wp diagnose entry point provides versioned native WP findings with explicit snapshot/live-window coverage, counter reset checks, registry limits and conflicting same-log progress. It has two implemented stall rules; exit 0 means evaluation completed, including inconclusive outcomes. Use its JSON evidence and original context/node mappings for follow-up. Use wp diag for retained Prometheus/Jaeger checks and wp webui for the local operations workspace. These inspect Woodpecker itself; Milvus-owned and Loki diagnosis are outside the tool.

Metrics scenario limits

stuck-read and stuck-write require persistent matching operations as well as zero successful completion progress during the window. An idle cluster alone is not a stall. Latency rules currently use window-average histogram latency rather than the advertised p99/baseline model. Several scenario names have only partial evaluators; use counters, operation IDs, reader positions and replica evidence together. A scenario reporting no findings is not proof that the whole cluster is healthy.

Exit codes and expected findings

Code Meaning
0 Completed within the command's stated scope.
1 / 2 / 3 / 4 Network / usage / target absent / state conflict.
5 / 6 / 7 Wait timeout / strict partial failure / acceptance not given or aborted.
8 / 9 Yellow / red diagnostic finding.
10 / 11 / 12 / 13 Not implemented / resource absent / configuration / prerequisite error.
100 + N Executed kubectl exit code N.

Read the exit status immediately with echo $?. Piping through another command can hide the original status unless pipefail is enabled. Example output throughout the handbook is illustrative and may omit fields; replace names and IDs with values from your own cluster.

CLI Handbook overview / CLI setup

CLI setup

Configure wp and run your first commands from outside Kubernetes.

0. Install wp

Assume Milvus with Woodpecker is already installed by Helm. Run wp on your workstation using a released binary or a source build. The shell examples below use bash/zsh on macOS or Linux.

Download from GitHub

Select the release and OS/CPU architecture on the GitHub Releases page. These are standalone binaries, not archives. For example, macOS on Apple Silicon:

Example / output
export WP_VERSION=v0.1.47
mkdir -p "$HOME/.local/bin"
curl -fL "https://github.com/zilliztech/woodpecker/releases/download/${WP_VERSION}/wp-darwin-arm64" \
  -o "$HOME/.local/bin/wp"
chmod +x "$HOME/.local/bin/wp"
export PATH="$HOME/.local/bin:$PATH"
wp version

For Linux, use wp-linux-amd64 or wp-linux-arm64; Windows uses wp-windows-amd64.exe. Use an available asset matching your machine.

Alternative: build from source
Example / output
git clone https://github.com/zilliztech/woodpecker.git
cd woodpecker
# Optionally check out the release/branch you want to build.
# git checkout v0.1.47
make wpcli
mkdir -p "$HOME/.local/bin"
cp bin/wp "$HOME/.local/bin/wp"
export PATH="$HOME/.local/bin:$PATH"
wp version

Install Git, make and the Go version required by go.mod first. You may also run ./bin/wp directly. A source build exposes the commands in that checkout; a released binary may not contain later development additions documented in this handbook. Check wp --help.

1. Connect to the Helm deployment

Use your real kubeconfig and namespace. Find the Woodpecker and etcd pods:

Example / output
export KUBECONFIG=/path/to/your/kubeconfig
kubectl -n milvus get pods -o wide

The examples below assume namespace milvus, Woodpecker pods milvus-woodpecker-0/1/2 and etcd pod milvus-etcd-0. Replace these names with the Helm deployment's actual names. Start each needed forward in its own terminal:

Example / output
kubectl -n milvus port-forward pod/milvus-woodpecker-0 19091:9091
kubectl -n milvus port-forward pod/milvus-woodpecker-1 29091:9091
kubectl -n milvus port-forward pod/milvus-woodpecker-2 39091:9091
# Needed when querying coordinator metadata or raw etcd keys:
kubectl -n milvus port-forward pod/milvus-etcd-0 32379:2379

The seed discovers membership; it does not proxy other nodes. Inspect the advertised node IDs/addresses, then map every node you need to reach:

Example / output
wp cluster info --endpoint http://127.0.0.1:19091

2. Save the connection in cli.yaml

The usual personal location is ~/.woodpecker/cli.yaml. Create the directory and save this example YAML there, using the node IDs reported by your cluster:

Example / output
mkdir -p "$HOME/.woodpecker"
Example / output
current-context: milvus-prod
contexts:
  milvus-prod:
    endpoint: http://127.0.0.1:19091
    node_admin_urls:
      milvus-woodpecker-0: http://127.0.0.1:19091
      milvus-woodpecker-1: http://127.0.0.1:29091
      milvus-woodpecker-2: http://127.0.0.1:39091
    k8s:
      namespace: milvus
      kubeconfig: /path/to/your/kubeconfig
    # v0.1.47 and later; v0.1.46 uses explicit etcd flags instead.

endpoint is the admin seed. node_admin_urls maps discovered node IDs or advertised service addresses to reachable admin origins. current-context selects milvus-prod when --context is omitted:

wp node list
wp --context milvus-prod node list

To keep cli.yaml in another directory, select it explicitly:

Example / output
export WOODPECKER_CLI_CONFIG=/path/to/your/cli.yaml
wp ctx list
wp --context milvus-prod ctx view
Config locations and precedence

Search order: WOODPECKER_CLI_CONFIG → cli.yaml beside the resolved wp binary → $XDG_CONFIG_HOME/woodpecker/cli.yaml → ~/.woodpecker/cli.yaml. The working directory and /etc/woodpecker are not default locations. Explicit command flags override matching context fields. WOODPECKER_ENDPOINT overrides the context's endpoint unless --endpoint is given.

The k8s block supplies defaults to wp k8s helpers, not to your shell's kubectl. Operator-specific helpers target a WoodpeckerCluster CR; for a Helm deployment without that CR, use kubectl directly. Do not use the Helm release name as k8s.cluster unless it is also the actual CR name.

3. Run without cli.yaml, or add shortcuts

For a built-in interactive entry point, see Interactive shell: it requires a service context and retains an explicitly connected tenant for subsequent commands. The examples below describe ordinary bash/zsh shortcuts and one-shot CLI invocations.

A config file is optional. Supply the admin endpoint and node mappings directly:

Example / output
wp node list --endpoint http://127.0.0.1:19091 \
  --node-admin-url milvus-woodpecker-0=http://127.0.0.1:19091 \
  --node-admin-url milvus-woodpecker-1=http://127.0.0.1:29091 \
  --node-admin-url milvus-woodpecker-2=http://127.0.0.1:39091

Replace node IDs with the discovered IDs, full advertised addresses or hostnames. Repeat --node-admin-url for each peer; --endpoint alone only makes the seed reachable. CLI flags can also override an existing cli.yaml for one invocation.

Optional: source shell functions for shorter commands

After creating the milvus-prod context, save this as a file of your choice, for example ~/wp-env.sh:

Example / output
export KUBECONFIG=/path/to/your/kubeconfig
# Set this only when cli.yaml is outside the usual personal location:
# export WOODPECKER_CLI_CONFIG=/path/to/your/cli.yaml

k() { kubectl --kubeconfig "$KUBECONFIG" -n milvus "$@"; }
w() { wp --context milvus-prod "$@"; }
Example / output
source ~/wp-env.sh
k get pods
w node list
w cluster health

source defines the functions in the current shell. w forwards arguments to wp and selects milvus-prod for each invocation; it does not change current-context in the file.

4. Configure etcd for metadata queries

Current source builds keep client metadata authority outside cli.yaml. Supply reachable etcd endpoints and the full client metadata prefix explicitly, or use a private --client-parameters-file. Older context-etcd/client-profile fields are not selected; there is no Admin-config discovery fallback.

Parameters Use
--etcd Reachable etcd endpoints; use commas for multiple addresses.
--meta-prefix Complete Woodpecker metadata prefix configured by the Milvus client. The example below uses milvus-prod/woodpecker; use the actual prefix, not an assumed Helm release name.
--etcd-username / --etcd-password Etcd authentication credentials when required.
--etcd-cacert CA certificate file on the workstation.
--etcd-cert / --etcd-key Workstation client certificate and private-key files.
--etcd-tls-min-version Minimum TLS version: 1.0, 1.1, 1.2 or 1.3.

For example, inspect registrations for a real Milvus log name using explicit connection settings:

Example / output
wp log readers my-channel \
  --etcd 127.0.0.1:32379 \
  --meta-prefix milvus-prod/woodpecker
Commands Need etcd settings?
meta; log readers/scan/skip-range/truncate; segment probe/inspect; logstore lac/fence-quorum; marking list/confirm Yes: they look up log/segment/quorum or operational metadata. Supply endpoint and complete prefix through explicit flags or a private client parameter file. meta and truncate need no admin connection when both etcd settings are supplied.
etcd list/get/cat Etcd endpoints and any required TLS/auth settings only. No admin discovery or metadata prefix: keys/prefixes are literal. See Raw etcd exploration.
cluster, node, config, env, logging, profile, ops, metrics, instance; other node-local logstore commands No etcd parameters: these use node admin endpoints. Do not append --etcd to commands that do not accept it.
ctx, completion, version; k8s helpers No etcd connection. These use local config/build information or kubectl.
Authentication/TLS example
Example / output
wp log readers my-channel \
  --etcd etcd.example.com:2379 --meta-prefix milvus-prod/woodpecker \
  --etcd-username operator --etcd-password 'YOUR_PASSWORD' \
  --etcd-cacert /path/to/ca.crt \
  --etcd-cert /path/to/client.crt --etcd-key /path/to/client.key \
  --etcd-tls-min-version 1.2

Certificate paths must exist on the workstation. An explicitly selected private client parameter file can contain these fields, independently of server contexts:

Example / output
endpoints: [http://127.0.0.1:32379]
meta_prefix: tenant/woodpecker
username: operator
password: YOUR_PASSWORD
tls_enabled: true
cacert: /path/to/ca.crt
# cert/key are a pair, required only for mutual TLS:
cert: /path/to/client.crt
key: /path/to/client.key
tls_min_version: "1.2"

Use a CA alone for server-authenticated TLS, or tls_enabled: true with system trust roots. Supplying cert/key enables mutual TLS; they must be supplied together. The current source default minimum TLS version is 1.2. Explicit flags override the private parameter file. ctx view shows server connection settings; client metadata parameters are supplied separately.

5. Check the connection and interpret the result

Example / output
wp ctx list
wp ctx view
wp node list
wp cluster health

Illustrative excerpts:

Example / output
NAME         ENDPOINT                ACTIVE
milvus-prod  http://127.0.0.1:19091   *

wp: config /home/alice/.woodpecker/cli.yaml · context milvus-prod · endpoint http://127.0.0.1:19091

NAME                  STATE   HEALTH
milvus-woodpecker-0    active  OK
milvus-woodpecker-1    active  OK
milvus-woodpecker-2    active  OK

Cluster Health: YELLOW
  active: 3  decommissioning: 0  unreachable: 0
  - zero redundancy (active == N)

The star marks the saved default context; ctx view shows resolved server settings; client metadata is a separate request scope. Three OK node responses confirm all three admin origins are reachable. YELLOW with active=3 and expected-replicas=3 means no spare capacity, not a connection failure.

If the seed refuses the connection, check its port-forward. If only one peer fails, check that peer's forward and mapping. If a metadata target is absent, check the actual prefix and log name. Use node IDs from node list and log/segment IDs reported by the application or log scan.

Interactive shell

For context selection, tenant connection parameters and command availability, open the dedicated Interactive shell handbook.

wp completion bash

Generate the autocompletion script for the bash shell.

Example / output
wp completion bash

Parameters to know

Parameter / type Use and default
--no-descriptions disable completion descriptions

Demo

Example / output
wp completion bash > wp-completion.bash

Output and interpretation

Example / output
Generated shell completion script

Generation only writes stdout; it does not edit shell startup files. Load it using your shell’s completion mechanism. For zsh: initialize compinit, then source the generated file.

Finding or error example
Example / output
Shell setup does not load the script
Complete command help
Example / output
Generate the autocompletion script for the bash shell.

This script depends on the 'bash-completion' package.
If it is not installed already, you can install it via your OS's package manager.

To load completions in your current shell session:

	source <(wp completion bash)

To load completions for every new session, execute once:

#### Linux:

	wp completion bash > /etc/bash_completion.d/wp

#### macOS:

	wp completion bash > $(brew --prefix)/etc/bash_completion.d/wp

You will need to start a new shell for this setup to take effect.

Usage:
  wp completion bash

Flags:
  -h, --help              help for bash
      --no-descriptions   disable completion descriptions

wp completion fish

Generate the autocompletion script for the fish shell.

Example / output
wp completion fish [flags]

Parameters to know

Parameter / type Use and default
--no-descriptions disable completion descriptions

Demo

Example / output
wp completion fish > wp-completion.fish

Output and interpretation

Example / output
Generated shell completion script

Generation only writes stdout; it does not edit shell startup files. Load it using your shell’s completion mechanism. For zsh: initialize compinit, then source the generated file.

Finding or error example
Example / output
Shell setup does not load the script
Complete command help
Example / output
Generate the autocompletion script for the fish shell.

To load completions in your current shell session:

	wp completion fish | source

To load completions for every new session, execute once:

	wp completion fish > ~/.config/fish/completions/wp.fish

You will need to start a new shell for this setup to take effect.

Usage:
  wp completion fish [flags]

Flags:
  -h, --help              help for fish
      --no-descriptions   disable completion descriptions

wp completion powershell

Generate the autocompletion script for powershell.

Example / output
wp completion powershell [flags]

Parameters to know

Parameter / type Use and default
--no-descriptions disable completion descriptions

Demo

Example / output
wp completion powershell > wp-completion.powershell

Output and interpretation

Example / output
Generated shell completion script

Generation only writes stdout; it does not edit shell startup files. Load it using your shell’s completion mechanism. For zsh: initialize compinit, then source the generated file.

Finding or error example
Example / output
Shell setup does not load the script
Complete command help
Example / output
Generate the autocompletion script for powershell.

To load completions in your current shell session:

	wp completion powershell | Out-String | Invoke-Expression

To load completions for every new session, add the output of the above command
to your powershell profile.

Usage:
  wp completion powershell [flags]

Flags:
  -h, --help              help for powershell
      --no-descriptions   disable completion descriptions

wp completion zsh

Generate the autocompletion script for the zsh shell.

Example / output
wp completion zsh [flags]

Parameters to know

Parameter / type Use and default
--no-descriptions disable completion descriptions

Demo

Example / output
wp completion zsh > wp-completion.zsh

Output and interpretation

Example / output
Generated shell completion script

Generation only writes stdout; it does not edit shell startup files. Load it using your shell’s completion mechanism. For zsh: initialize compinit, then source the generated file.

Finding or error example
Example / output
Shell setup does not load the script
Complete command help
Example / output
Generate the autocompletion script for the zsh shell.

If shell completion is not already enabled in your environment you will need
to enable it.  You can execute the following once:

	echo "autoload -U compinit; compinit" >> ~/.zshrc

To load completions in your current shell session:

	source <(wp completion zsh)

To load completions for every new session, execute once:

#### Linux:

	wp completion zsh > "${fpath[1]}/_wp"

#### macOS:

	wp completion zsh > $(brew --prefix)/share/zsh/site-functions/_wp

You will need to start a new shell for this setup to take effect.

Usage:
  wp completion zsh [flags]

Flags:
  -h, --help              help for zsh
      --no-descriptions   disable completion descriptions

wp ctx list

List all contexts defined in cli.yaml

Example / output
wp ctx list [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp ctx list

Output and interpretation

Example / output
NAME         ENDPOINT                ACTIVE
milvus-prod  http://127.0.0.1:19091   *

list and view inspect local CLI configuration. use persists current-context; --context selects one invocation without changing it. No server configuration is changed.

Finding or error example
Example / output
Context does not exist / config parse or write error
Complete command help
Example / output
List all contexts defined in cli.yaml

Usage:
  wp ctx list [flags]

Flags:
  -h, --help   help for list

wp ctx use

Switch current-context to the given name and persist to cli.yaml

Example / output
wp ctx use <name> [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp ctx use milvus-prod

Output and interpretation

Example / output
Switched to context "milvus-prod"

list and view inspect local CLI configuration. use persists current-context; --context selects one invocation without changing it. No server configuration is changed.

Finding or error example
Example / output
Context does not exist / config parse or write error
Complete command help
Example / output
Switch current-context to the given name and persist to cli.yaml

Usage:
  wp ctx use <name> [flags]

Flags:
  -h, --help   help for use

wp ctx view

Print the resolved fields of the active context

Example / output
wp ctx view [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp --context milvus-prod ctx view

Output and interpretation

Example / output
Active context:
  endpoint:     http://127.0.0.1:19091
  admin_port:   9091
  timeout:      30s
  concurrency:  8
  strict:       false

list and view inspect local CLI configuration. use persists current-context; --context selects one invocation without changing it. No server configuration is changed.

Finding or error example
Example / output
Context does not exist / config parse or write error
Complete command help
Example / output
Print the resolved fields of the active context

Usage:
  wp ctx view [flags]

Flags:
  -h, --help   help for view

wp report analyze

Explicitly ask the configured Agent to explain supplied findings

Example / output
wp report analyze <report.json|-> [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp --context local report analyze diagnosis.json --config /path/to/cli.yaml

Output and interpretation

Example / output
The explicitly configured backend-local Agent explains abnormal findings.

This explicitly invokes a model with supplied evidence. It cannot automatically query sources, run commands or repair the cluster.

Finding or error example
Example / output
No Agent configuration, no findings or a model failure leaves the original report unchanged.
Complete command help
Example / output
Explicitly ask the configured Agent to explain supplied findings

Usage:
  wp report analyze <report.json|-> [flags]

Flags:
  -h, --help   help for analyze

wp report export

Copy original JSON or derive a compact findings packet

Example / output
wp report export <report.json|-> [flags]

Parameters to know

Parameter / type Use and default
--packet Export findings, supporting context, gaps and provenance; no model call

Demo

Example / output
wp report export diagnosis.json --packet > findings.json

Output and interpretation

Example / output
Exports a bounded findings packet with original report provenance.

Without --packet, copies the original report. It does not rerun checks.

Finding or error example
Example / output
Malformed or oversized reports are rejected.
Complete command help
Example / output
Copy original JSON or derive a compact findings packet

Usage:
  wp report export <report.json|-> [flags]

Flags:
  -h, --help     help for export
      --packet   Export findings, supporting context, gaps and provenance; no model call

wp report inspect

Show recorded evidence detail without querying sources

Example / output
wp report inspect <report.json|-> <check> [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp report inspect diagnosis.json server-disk-watermark

Output and interpretation

Example / output
Shows the recorded calculation, measurements and available curves.

Consumes saved JSON without a source query or model call.

Finding or error example
Example / output
Detail omitted during collection cannot be reconstructed without another check.
Complete command help
Example / output
Show recorded evidence detail without querying sources

Usage:
  wp report inspect <report.json|-> <check> [flags]

Flags:
  -h, --help   help for inspect

wp version

Show wp CLI version and build info

Example / output
wp version [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp version

Output and interpretation

Example / output
wp version ...
# Version, commit and build details reflect this CLI binary.

This reports the CLI build, not all node versions. Use env show or cluster info for remote builds.

Finding or error example
Example / output
Unavailable build fields in an unversioned build
Complete command help
Example / output
Show wp CLI version and build info

Usage:
  wp version [flags]

Flags:
  -h, --help   help for version

wp webui

Open a local operations and diagnosis workspace using cli.yaml

Example / output
wp webui [flags]

Parameters to know

Parameter / type Use and default
--config string Shared cli.yaml (default: normal wp configuration discovery)
--listen string Loopback host:port (default "127.0.0.1:8794")
--metrics-rules string Metrics rules file beside cli.yaml by default
--trace-rules string Trace rules file beside cli.yaml by default

Demo

Example / output
wp webui --config /path/to/cli.yaml --listen 127.0.0.1:8795

Output and interpretation

Example / output
Open the printed local URL.

Diagnostics finds issues; Operations confirms and handles them; Command Explorer exposes native commands. No diagnostic action automatically repairs the cluster.

Finding or error example
Example / output
A missing or invalid cli.yaml prevents startup.
Complete command help
Example / output
Open a local operations and diagnosis workspace using cli.yaml

Usage:
  wp webui [flags]

Flags:
      --config string          Shared cli.yaml (default: normal wp configuration discovery)
  -h, --help                   help for webui
      --listen string          Loopback host:port (default "127.0.0.1:8794")
      --metrics-rules string   Metrics rules file beside cli.yaml by default
      --trace-rules string     Trace rules file beside cli.yaml by default

CLI Handbook overview / Interactive shell

Interactive shell

Select a service context, explicitly connect tenant metadata and storage, and run operations in one interactive session.

Source-build feature: wp shell is not included in v0.1.47. Build a checkout containing the command with make wpcli and run ./bin/wp shell --help to check availability. Examples below use wp from that build. Output is illustrative; replace names, prefixes and storage locations with those of your deployment.

Keep one terminal open while operating a multi-tenant Woodpecker service. Select the service once, then explicitly connect the tenant whose metadata you want to inspect. Inside the session, omit the leading wp; existing operation arguments, previews and confirmations still apply.

1. Configure and select the service context

A service context is required. For a Helm installation accessed from outside Kubernetes, first establish the Admin port forwards described in CLI setup. Save this example as ~/.woodpecker/cli.yaml:

Example / output
current-context: milvus-prod
contexts:
  milvus-prod:
    endpoint: http://127.0.0.1:19091
    timeout: 10s
    node_admin_urls:
      milvus-woodpecker-0: http://127.0.0.1:19091
      milvus-woodpecker-1: http://127.0.0.1:29091
      milvus-woodpecker-2: http://127.0.0.1:39091
  milvus-staging:
    endpoint: https://wp-staging.example.com:9091

endpoint is the Admin seed. node_admin_urls maps the node IDs or advertised addresses returned by your cluster to workstation-accessible Admin origins. A reachable seed alone does not make the other replicas reachable. The staging origin above is a placeholder; configure its actual mappings when required.

Example / output
wp shell --context milvus-prod

For a configuration file in another location, set the environment variable in the same terminal before starting:

Example / output
export WOODPECKER_CLI_CONFIG=/path/to/cli.yaml
wp shell --context milvus-prod

Without --context, an interactive wp shell always displays a context selector, even when a current context exists. Enter a displayed number or exact name; Enter accepts a valid current-context. Piped input requires an explicit --context. Missing or invalid configuration reports the relevant configuration path and setup instructions. See CLI setup for the full configuration search order.

Example / output
Service context: milvus-prod
Config file:     /Users/example/.woodpecker/cli.yaml
Admin endpoint:  http://127.0.0.1:19091
Node mappings:   3
Tenant metadata: not configured

wp[context=milvus-prod | meta=disconnected]>

The prompt identifies two independent targets: context is the Woodpecker service cluster; meta is the selected tenant connection, or disconnected. The latter means no tenant is configured, not a network health verdict. Entering the shell does not contact the services or automatically attach the context's etcd settings. The shell pins its chosen service endpoint rather than inheriting WOODPECKER_ENDPOINT.

2. Connect the tenant: etcd, metadata prefix and storage

From the interactive prompt, connect using the tenant's workstation-accessible etcd endpoint and complete Woodpecker metadata prefix. Keep this example on one line when entering it:

Example / output
connect tenant-a --etcd 127.0.0.1:32379 --meta-prefix tenant-a/woodpecker --bucket tenant-a-bucket --root-path milvus/wp
Parameter Meaning and requirement
tenant-a Required operator-selected connection name, displayed as meta=tenant-a. It is an alias; it does not derive or change the actual metadata prefix.
--etcd Required reachable etcd endpoint; comma-separate multiple endpoints. Outside Kubernetes, use the forwarded or otherwise reachable address.
--meta-prefix Required complete metadata namespace configured by the client, for example tenant-a/woodpecker. Do not supply a particular log key or confuse it with the physical storage root.
--bucket Physical storage bucket for this tenant. Optional for metadata-only operations; provide together with --root-path.
--root-path Physical storage instance root, for example milvus/wp. Provide together with --bucket; it need not equal the metadata prefix.
--etcd-username / --etcd-password Credentials when etcd authentication is enabled. Each new connection starts with fresh settings; omitted credentials are not inherited from the previous tenant.
TLS endpoint Use an https:// etcd endpoint or supply certificate flags to enable TLS. There is no separate --etcd-tls switch in this interface.
--etcd-cacert Local CA certificate file when needed. Certificate flags enable TLS.
--etcd-cert / --etcd-key Local client certificate and private key for mutual TLS; provide both.
--etcd-tls-min-version Minimum TLS version when TLS is enabled: 1.0, 1.1, 1.2 or 1.3; default 1.2.
Example / output
Service context: milvus-prod
Tenant:          tenant-a
Etcd endpoints:  127.0.0.1:32379
Meta prefix:     tenant-a/woodpecker
Bucket:          tenant-a-bucket
Root path:       milvus/wp

wp[context=milvus-prod | meta=tenant-a]>

Success verifies an etcd read under the supplied prefix, even if it is empty. It does not establish that you selected the intended tenant, that its quorum belongs to this service context, or that its data is healthy. Check status and the returned metadata before operating. connect --help lists the complete connection flags.

Metadata-only and TLS connection examples
Example / output
# Metadata-only connection; no bucket/root-path is needed.
connect tenant-a --etcd 127.0.0.1:32379 --meta-prefix tenant-a/woodpecker

# Custom CA and client certificates; use files on this workstation.
connect tenant-a --etcd etcd.example.com:2379 --meta-prefix tenant-a/woodpecker --etcd-cacert /path/to/ca.pem --etcd-cert /path/to/client.pem --etcd-key /path/to/client-key.pem

Each successful connect replaces the whole tenant connection. A metadata-only reconnect also clears a previous bucket/root-path. A failed connection retains the previous selected tenant and reports that the selection is unchanged.

3. Which commands can run now?

Session state Available operations Examples inside the shell
Service context only
meta=disconnected
Service runtime and node diagnostics: cluster/node, environment, configuration, logging, profiles, in-flight operations, metrics, local instance inventory and node-local logstore operations. Kubernetes helpers use the service context's Kubernetes settings. Required services/tools must still be reachable. node list
cluster health
config show node-0
logstore segments node-0
instance data node-0
Service context + tenant etcd and metadata prefix All meta, etcd, and marking operations; log readers, log skip-range, log truncate; quorum-aware logstore lac and logstore fence-quorum. Mutations retain their existing validation, acceptance and backup requirements. meta log list
meta check application-log
log readers application-log
log skip-range list application-log
etcd list tenant-a/woodpecker/
Tenant connection + bucket and root-path log scan, segment inspect, and segment probe inherit the selected data location. In interactive mode these three require both location values, either from connect or explicit command flags. log scan application-log --mode raw
segment inspect application-log 0
segment probe application-log 0
Any session state Local session inspection, switching, help and exit. Operation help is available before a tenant is attached. status
ctx list
help meta log
connect --help
exit

Connecting makes the required parameters available; it does not make every operation succeed. Physical reads also require reachable quorum node Admin endpoints, and mutation commands still require their operation-specific evidence and safeguards. Raw etcd commands use the literal key/prefix you supply; --meta-prefix does not automatically scope or prepend raw etcd paths.

Example / output
meta log list
log readers application-log
log scan application-log --mode raw
segment inspect application-log 0

# Override data location for this one operation only.
segment probe application-log 0 --bucket another-bucket --root-path another/root

Use actual log names from meta log list and segment IDs from meta segment list application-log. A per-command override does not modify the next command's defaults. When a tenant storage location is attached, instance data and node log-health also inherit it as a filter (their root flag is --root). They remain usable without a tenant connection. This shell's explicit storage-location rule does not change the existing non-interactive CLI's discovery behavior.

4. Inspect the session with status

Example / output
status
Example / output
Service context: milvus-prod
Config file:     /Users/example/.woodpecker/cli.yaml
Admin endpoint:  http://127.0.0.1:19091
Node mappings:   3
Tenant:          tenant-a
Etcd endpoints:  127.0.0.1:32379
Meta prefix:     tenant-a/woodpecker
Bucket:          tenant-a-bucket
Root path:       milvus/wp

status reads local session settings without making network requests, so it works while services are down or no tenant is attached. When credentials are configured, it reports that authentication is configured without printing them. An empty bucket/root-path means physical inspection needs explicit values.

Example / output
status --check
Example / output
Admin check: OK
Etcd check: OK

The check form also prints the session configuration, then performs bounded Admin and etcd checks. Read the two results separately. When disconnected it prints Etcd check: skipped (no tenant configured); a failed Admin check does not hide the separate etcd result. Neither check verifies application progress or physical data integrity.

5. Disconnect or switch a service/tenant

Example / output
disconnect
status
use milvus-staging
connect tenant-b --etcd 127.0.0.1:42379 --meta-prefix tenant-b/woodpecker
status

disconnect clears the entire tenant connection, including credentials, TLS, metadata prefix and storage location. It retains the service context. A successful use <context> switches the service and clears any tenant attachment; connect the intended tenant again. A failed context selection leaves the current service and tenant unchanged.

Example / output
wp[context=milvus-prod | meta=tenant-a]> disconnect
Tenant metadata: not configured
wp[context=milvus-prod | meta=disconnected]> use milvus-staging
Service context: milvus-staging
Config file:     /Users/example/.woodpecker/cli.yaml
Admin endpoint:  https://wp-staging.example.com:9091
Node mappings:   0
Tenant metadata: not configured
wp[context=milvus-staging | meta=disconnected]>

Session-local switching does not write current-context or affect another terminal. Inside the session, ctx list and ctx view inspect saved CLI configuration; use status to inspect the active tenant attachment. Use use for session-local switching. The existing ctx use command is a persistent change to the original configuration file and does not retarget this running session; invoke it only when you intend to change the default on disk. Exit and start a new shell to reload an edited configuration file.

6. Interpret missing configuration and connection failures

If a metadata-dependent command runs while disconnected, it fails before contacting Admin or etcd, even if the service context contains etcd settings or you pass connection flags on that operation:

Example / output
wp[context=milvus-prod | meta=disconnected]> meta log list
wp shell: Prerequisite: no tenant metadata connection configured.
Connect with: connect <tenant> --etcd <host:port> --meta-prefix <prefix>.
Physical inspection also needs --bucket <bucket> --root-path <root-path>.
Run status to view the session
Result Meaning / next step
no tenant metadata connection configured Run connect, then retry. Service-only commands and help remain available.
physical inspection requires --bucket ... and --root-path ... Supply the paired location flags on scan/inspect/probe, or reconnect the tenant with both. No physical inspection has started.
tenant connection failed ...; active tenant unchanged: ... Check endpoint forwarding, authentication and local TLS files. The previous tenant remains selected, or the session stays disconnected. Run status before continuing.
Admin check: failed ... / Etcd check: failed ... Configuration exists but that check could not succeed. A timeout or authentication failure is different from a missing connection configuration; the selected context/tenant remains intact.

7. Help, completion and exit

Example / output
help
connect --help
help segment inspect
ctx list
exit

Tab completes command/flag names and context names after use. Arrow keys recall in-memory history; connect and credential-bearing commands are excluded from history. Ctrl+C cancels the current operation and returns to the prompt. Use exit, quit or Ctrl+D to leave.

This is a command session, not bash/zsh: quoted arguments work, but environment-variable expansion, pipes and redirection are not evaluated. Run exports and port forwards in your ordinary terminal. Enter each example operation as a separate command. Existing command findings can return errors while the shell remains open for further diagnosis.

wp shell

Open an interactive operational shell. Without --context, choose a context from cli.yaml.

Example / output
wp shell [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp shell --context milvus-prod
# Inside the interactive session:
status
connect tenant-a --etcd 127.0.0.1:32379 --meta-prefix tenant-a/woodpecker --bucket tenant-a-bucket --root-path milvus/wp
meta log list

Output and interpretation

Example / output
wp[context=milvus-prod | meta=disconnected]>
# After connect succeeds:
wp[context=milvus-prod | meta=tenant-a]>

The service context is required; attaching a tenant is explicit and optional. Service commands work while disconnected. Metadata operations require connect, and interactive scan/inspect/probe also require bucket/root-path. status shows local selection; status --check checks reachability. Switching service contexts clears the tenant connection. This source-build feature is not included in v0.1.47.

Finding or error example
Example / output
wp shell: Prerequisite: no tenant metadata connection configured. Connect with: connect <tenant> --etcd <host:port> --meta-prefix <prefix>. Run status to view the session
Complete command help
Example / output
Open an interactive operational shell. Without --context, choose a context from cli.yaml.
Start with no tenant metadata attached; use connect to attach a tenant.
For piped input, specify --context explicitly. Type help for commands or exit to leave.

Usage:
  wp shell [flags]

Flags:
  -h, --help   help for shell

CLI Handbook overview / Process environment

Process environment

Inspect and compare process, Go runtime, host and build information.

wp env diff

Compare env / runtime / host / build across all nodes

Example / output
wp env diff [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp env diff --strict

Output and interpretation

Example / output
node-0: (reference)
node-1: identical

Compare build, Go runtime and deployment environment. Differences are evidence to investigate, not automatic data corruption.

Finding or error example
Example / output
node-1: DRIFT (...)
node-2: UNREACHABLE (not compared)
Complete command help
Example / output
Compare env / runtime / host / build across all nodes

Usage:
  wp env diff [flags]

Flags:
  -h, --help   help for diff

wp env show

Show env vars + runtime + host + build for a single node

Example / output
wp env show <node> [flags]

Parameters to know

Parameter / type Use and default
--section string section filter: env|runtime|host|build|all (default "all")

Demo

Example / output
wp env show node-0 -o json

Output and interpretation

Example / output
{"env":{...},"runtime":{...},"host":{...},"build":{...}}

Compare build, Go runtime and deployment environment. Differences are evidence to investigate, not automatic data corruption.

Finding or error example
Example / output
Error: TargetNotFound: node-missing
Complete command help
Example / output
Show env vars + runtime + host + build for a single node

Usage:
  wp env show <node> [flags]

Flags:
  -h, --help             help for show
      --section string   section filter: env|runtime|host|build|all (default "all")

CLI Handbook overview / Cluster & node status

Cluster & node status

Inspect membership, reachability, health, gossip and node drain lifecycle.

wp cluster gossip-diff

Compare memberlist views across nodes to detect partitions

Example / output
wp cluster gossip-diff [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp cluster gossip-diff

Output and interpretation

Example / output
Peer membership views agree

Agreement is among responding peers. Use --strict for incomplete fan-out; inspect whether a new/removed node has converged.

Finding or error example
Example / output
Membership views differ / unreachable peer
Complete command help
Example / output
Compare memberlist views across nodes to detect partitions

Usage:
  wp cluster gossip-diff [flags]

Flags:
  -h, --help   help for gossip-diff

wp cluster health

Red/yellow/green health check for the cluster

Example / output
wp cluster health [flags]

Parameters to know

Parameter / type Use and default
--expected-replicas int expected replica count baseline (default 3) (default 3)

Demo

Example / output
wp cluster health --expected-replicas 3

Output and interpretation

Example / output
Cluster Health: GREEN

GREEN requires active > N, no unreachable/decommissioning nodes and no version skew. RED: active < N or any node unreachable. YELLOW: active == N, decommissioning or mixed versions. The default N=3 is a baseline, not inferred per-log quorum health.

Finding or error example
Example / output
Cluster Health: YELLOW
active: 3
zero redundancy (active == N)
Complete command help
Example / output
Red/yellow/green health check for the cluster

Usage:
  wp cluster health [flags]

Flags:
      --expected-replicas int   expected replica count baseline (default 3) (default 3)
  -h, --help                    help for health

wp cluster info

Show cluster overview (nodes, state, topology)

Example / output
wp cluster info [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp cluster info

Output and interpretation

Example / output
members: [node-0, node-1, node-2]

Membership is the seed’s current view, not proof of data availability. Compare peer views with gossip-diff.

Finding or error example
Example / output
Network: seed unavailable
Complete command help
Example / output
Show cluster overview (nodes, state, topology)

Usage:
  wp cluster info [flags]

Flags:
  -h, --help   help for info

wp node cancel-decommission

Cancel an in-progress decommission (decommissioning -> active)

Example / output
wp node cancel-decommission <node> [flags]

Parameters to know

Parameter / type Use and default
-y, --yes skip interactive confirmation

Demo

Example / output
wp node cancel-decommission node-0

Output and interpretation

Example / output
decommission cancelled on node-0; returning to active

Cancels the lifecycle operation; re-check node state and application progress.

Finding or error example
Example / output
Error: StateConflict: ...
Complete command help
Example / output
Cancel an in-progress decommission (decommissioning -> active)

Usage:
  wp node cancel-decommission <node> [flags]

Flags:
  -h, --help   help for cancel-decommission
  -y, --yes    skip interactive confirmation

wp node decommission

Trigger graceful node decommission (blocks until safe by default)

Example / output
wp node decommission <node> [flags]

Parameters to know

Parameter / type Use and default
--async return immediately instead of waiting for safe_to_terminate
--heartbeat-interval duration heartbeat line interval when no progress change (default 15s)
--interval duration progress poll interval (default 3s)
--timeout duration overall timeout for the wait (default 30m0s)
-y, --yes skip interactive confirmation

Demo

Example / output
wp node decommission node-0 --async

Output and interpretation

Example / output
decommission started on node-0
# --async returns before drain completion; follow with drain-status.

Changes node lifecycle state. Check capacity first; --async returns after request rather than completion. Timeout does not authorize termination.

Finding or error example
Example / output
state=decommissioning remaining_processors=1 has_local_data=true safe=false
Complete command help
Example / output
Trigger graceful node decommission (blocks until safe by default)

Usage:
  wp node decommission <node> [flags]

Flags:
      --async                         return immediately instead of waiting for safe_to_terminate
      --heartbeat-interval duration   heartbeat line interval when no progress change (default 15s)
  -h, --help                          help for decommission
      --interval duration             progress poll interval (default 3s)
      --timeout duration              overall timeout for the wait (default 30m0s)
  -y, --yes                           skip interactive confirmation

wp node drain-status

Show (or watch) decommission progress for a node

Example / output
wp node drain-status <node> [flags]

Parameters to know

Parameter / type Use and default
--interval duration watch poll interval (default 3s)
--timeout duration watch total timeout (default 30m0s)
-w, --watch continuously watch until safe_to_terminate

Demo

Example / output
wp node drain-status node-0

Output and interpretation

Example / output
safe_to_terminate: true

Terminate only after safe_to_terminate=true. active means decommission has not completed; processors and local data independently prevent a safe drain. -w polls.

Finding or error example
Example / output
state: active
remaining_processors: 1
has_local_data: true
safe_to_terminate: false
Complete command help
Example / output
Show (or watch) decommission progress for a node

Usage:
  wp node drain-status <node> [flags]

Flags:
  -h, --help                help for drain-status
      --interval duration   watch poll interval (default 3s)
      --timeout duration    watch total timeout (default 30m0s)
  -w, --watch               continuously watch until safe_to_terminate

wp node healthz

Query a node's health probe (the one readiness uses)

Example / output
wp node healthz <node> [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp node healthz node-0

Output and interpretation

Example / output
Health probe on node-0: ok

Process health/readiness does not establish log read/write progress or replica data integrity.

Finding or error example
Example / output
Health probe on node-0: failed
# Read individual dependency/check codes in the report.
Complete command help
Example / output
Query a node's health probe (the one readiness uses)

Usage:
  wp node healthz <node> [flags]

Flags:
  -h, --help   help for healthz

wp node list

List all server nodes in the cluster

Example / output
wp node list [flags]

Parameters to know

Parameter / type Use and default
--az string filter by availability zone
--region string filter by region
--rg string filter by resource group
--state string filter by state: active|decommissioning|decommissioned|unreachable|all (default "all")

Demo

Example / output
wp node list --strict

Output and interpretation

Example / output
NAME                  STATE   HEALTH
milvus-woodpecker-0    active  OK
milvus-woodpecker-1    active  OK

Use node IDs from this list in node-scoped commands; distinguish membership from HTTP reachability.

Finding or error example
Example / output
milvus-woodpecker-2    unreachable
# With --strict, incomplete fan-out returns a nonzero exit code.
Complete command help
Example / output
List all server nodes in the cluster

Usage:
  wp node list [flags]

Flags:
      --az string       filter by availability zone
  -h, --help            help for list
      --region string   filter by region
      --rg string       filter by resource group
      --state string    filter by state: active|decommissioning|decommissioned|unreachable|all (default "all")

wp node log-health

Show per-log read/write health observed by a node

Example / output
wp node log-health <node> [flags]

Parameters to know

Parameter / type Use and default
--bucket string Filter to one instance's bucket (requires --root)
--root string Filter to one instance's root path (requires --bucket)

Demo

Example / output
wp node log-health node-0

Output and interpretation

Example / output
tracked: 1  healthy: 1  stalled: 0  failed: 0  idle: 0

Write/read health describe client/log activity tracked on this node. Idle is inactivity, not a checksum failure. Stalled and Failed require looking at operations and last failure; this registry is not a complete metadata inventory.

Finding or error example
Example / output
reason: idle_logs_present
tracked: 1  healthy: 0  stalled: 0  failed: 0  idle: 1
Complete command help
Example / output
Show per-log read/write health observed by a node

Usage:
  wp node log-health <node> [flags]

Flags:
      --bucket string   Filter to one instance's bucket (requires --root)
  -h, --help            help for log-health
      --root string     Filter to one instance's root path (requires --bucket)

wp node restart

(Not implemented) Restart a node — use your orchestrator instead

Example / output
wp node restart <node> [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp node restart node-0

Output and interpretation

Example / output
No supported success path

This command is currently a stub. Restart through the orchestrator; first verify drain safety when doing planned replacement.

Finding or error example
Example / output
NotImplemented
Complete command help
Example / output
(Not implemented) Restart a node — use your orchestrator instead

Usage:
  wp node restart <node> [flags]

Flags:
  -h, --help   help for restart

wp node show

Show detailed status for a single node

Example / output
wp node show <node> [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp node show node-0 -o json

Output and interpretation

Example / output
{"node_id":"milvus-woodpecker-0","state":"active","is_decommissioning":false,...}

Inspect one discovered node. A missing target is different from an HTTP failure.

Finding or error example
Example / output
Error: Network: /admin/node returned status 503
Complete command help
Example / output
Show detailed status for a single node

Usage:
  wp node show <node> [flags]

Flags:
  -h, --help   help for show

CLI Handbook overview / Kubernetes helpers

Kubernetes helpers

Check kubectl prerequisites and plan or execute Kubernetes status, logs and scaling actions.

wp k8s doctor

Diagnose K8s cluster issues (not yet implemented)

Example / output
wp k8s doctor [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp k8s doctor

Output and interpretation

Example / output
No supported success path: this operation is not implemented.

Use kubectl and the status/logs helpers for diagnosis; this placeholder does not check prerequisites or cluster health.

Finding or error example
Example / output
Error: NotImplemented: k8s doctor
# Exit code 10 is expected.
Complete command help
Example / output
Diagnose K8s cluster issues (not yet implemented)

Usage:
  wp k8s doctor [flags]

Flags:
  -h, --help   help for doctor

wp k8s logs

Tail logs from a Woodpecker pod

Example / output
wp k8s logs <node-or-pod> [flags]

Parameters to know

Parameter / type Use and default
-x, --execute Execute kubectl commands (default: print only)
-f, --follow Stream logs
--kube-context string kubectl context name
--kubeconfig string Path to kubeconfig file
--kubectl string Path to kubectl binary
-n, --namespace string Kubernetes namespace
--since string Only show logs newer than duration (e.g. 1h)
--tail int Number of recent lines to show (default 100)
--wp-cluster string WoodpeckerCluster CR name

Demo

Example / output
wp k8s logs milvus-woodpecker-0 --namespace milvus

Output and interpretation

Example / output
kubectl ... -n milvus logs milvus-woodpecker-0 --tail=100

status, logs and scale print plans by default; -x executes. scale targets a WoodpeckerCluster CR, not an arbitrary StatefulSet. Set the correct kube context and namespace. doctor checks local prerequisites.

Finding or error example
Example / output
Missing kubectl, kubeconfig or resource / kubectl exit status
Complete command help
Example / output
Tail logs from a Woodpecker pod

Usage:
  wp k8s logs <node-or-pod> [flags]

Flags:
  -x, --execute               Execute kubectl commands (default: print only)
  -f, --follow                Stream logs
  -h, --help                  help for logs
      --kube-context string   kubectl context name
      --kubeconfig string     Path to kubeconfig file
      --kubectl string        Path to kubectl binary
  -n, --namespace string      Kubernetes namespace
      --since string          Only show logs newer than duration (e.g. 1h)
      --tail int              Number of recent lines to show (default 100)
      --wp-cluster string     WoodpeckerCluster CR name

wp k8s scale

Scale the cluster via kubectl patch

Example / output
wp k8s scale [flags]

Parameters to know

Parameter / type Use and default
-x, --execute Execute kubectl commands (default: print only)
--kube-context string kubectl context name
--kubeconfig string Path to kubeconfig file
--kubectl string Path to kubectl binary
-n, --namespace string Kubernetes namespace
--replicas int Target replica count (required)
--wp-cluster string WoodpeckerCluster CR name

Demo

Example / output
wp k8s scale --namespace milvus --wp-cluster wp --replicas 4

Output and interpretation

Example / output
Scaling wp to 4 replicas...
kubectl ... patch woodpeckercluster wp --type merge -p ...
# Printed plan only; --execute runs it.

status, logs and scale print plans by default; -x executes. scale targets a WoodpeckerCluster CR, not an arbitrary StatefulSet. Set the correct kube context and namespace. doctor checks local prerequisites. This targets a WoodpeckerCluster CR, not a Helm release.

Finding or error example
Example / output
Missing kubectl, kubeconfig or resource / kubectl exit status
Complete command help
Example / output
Scale the cluster via kubectl patch

Usage:
  wp k8s scale [flags]

Flags:
  -x, --execute               Execute kubectl commands (default: print only)
  -h, --help                  help for scale
      --kube-context string   kubectl context name
      --kubeconfig string     Path to kubeconfig file
      --kubectl string        Path to kubectl binary
  -n, --namespace string      Kubernetes namespace
      --replicas int          Target replica count (required)
      --wp-cluster string     WoodpeckerCluster CR name

wp k8s status

Show cluster status via kubectl

Example / output
wp k8s status [flags]

Parameters to know

Parameter / type Use and default
-x, --execute Execute kubectl commands (default: print only)
--kube-context string kubectl context name
--kubeconfig string Path to kubeconfig file
--kubectl string Path to kubectl binary
-n, --namespace string Kubernetes namespace
--wp-cluster string WoodpeckerCluster CR name

Demo

Example / output
wp k8s status --namespace milvus --wp-cluster woodpecker-prod

Output and interpretation

Example / output
kubectl ... get woodpeckercluster woodpecker-prod -o wide
kubectl ... describe woodpeckercluster woodpecker-prod
# Printed plan only; --execute runs it.

status, logs and scale print plans by default; -x executes. scale targets a WoodpeckerCluster CR, not an arbitrary StatefulSet. Set the correct kube context and namespace. doctor checks local prerequisites. status/scale require an operator-managed WoodpeckerCluster CR. For a Helm deployment without that CR, use kubectl get pods directly.

Finding or error example
Example / output
Missing kubectl, kubeconfig or resource / kubectl exit status
Complete command help
Example / output
Show cluster status via kubectl

Usage:
  wp k8s status [flags]

Flags:
  -x, --execute               Execute kubectl commands (default: print only)
  -h, --help                  help for status
      --kube-context string   kubectl context name
      --kubeconfig string     Path to kubeconfig file
      --kubectl string        Path to kubectl binary
  -n, --namespace string      Kubernetes namespace
      --wp-cluster string     WoodpeckerCluster CR name

CLI Handbook overview / Server configuration

Server configuration

Inspect server configuration and compare configuration drift across nodes.

wp config diff

Compare configurations across nodes

Example / output
wp config diff [node-a] [node-b] [flags]

Parameters to know

Parameter / type Use and default
--all diff all nodes in the cluster against the first reachable node
--reference string explicit reference node name

Demo

Example / output
wp config diff --all --strict

Output and interpretation

Example / output
node-0: (reference)
node-1: identical
node-2: identical

This is server configuration, not client SDK configuration. Compare actual values and failed respondents separately.

Finding or error example
Example / output
node-1: DRIFT (1 fields differ)
node-2: UNREACHABLE (not compared)
Error: YellowFinding: config drift detected
Complete command help
Example / output
Compare configurations across nodes

Usage:
  wp config diff [node-a] [node-b] [flags]

Flags:
      --all                diff all nodes in the cluster against the first reachable node
  -h, --help               help for diff
      --reference string   explicit reference node name

wp config show

Show the resolved configuration of a single node

Example / output
wp config show <node> [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp config show node-0 -o json

Output and interpretation

Example / output
{"woodpecker":{"storage":{"type":"service"}}}

This is server configuration, not client SDK configuration. Compare actual values and failed respondents separately.

Finding or error example
Example / output
Error: Network: /admin/config returned status 503
Complete command help
Example / output
Show the resolved configuration of a single node

Usage:
  wp config show <node> [flags]

Flags:
  -h, --help   help for show

CLI Handbook overview / Runtime log levels

Runtime log levels

Read or temporarily change server logging levels.

wp logging get-level

Show current log level of a node (or all nodes)

Example / output
wp logging get-level [node] [flags]

Parameters to know

Parameter / type Use and default
--all Show log level on all nodes

Demo

Example / output
wp logging get-level node-0

Output and interpretation

Example / output
NODE    LEVEL
node-0  info

Runtime logging changes affect this node. Record the old level and restore it after a short reproduction; this is separate from the log metadata command group.

Finding or error example
Example / output
node-0  (unreachable)
# In an --all report, inspect each row.
Complete command help
Example / output
Show current log level of a node (or all nodes)

Usage:
  wp logging get-level [node] [flags]

Flags:
      --all    Show log level on all nodes
  -h, --help   help for get-level

wp logging set-level

Change log level of a node (or all nodes)

Example / output
wp logging set-level [node] [flags]

Parameters to know

Parameter / type Use and default
--all Set log level on all nodes
--level string Target log level (debug, info, warn, error)

Demo

Example / output
wp logging set-level node-0 --level debug

Output and interpretation

Example / output
Log level set to "debug" on node-0

Runtime logging changes affect this node. Record the old level and restore it after a short reproduction; this is separate from the log metadata command group.

Finding or error example
Example / output
Error: Usage: invalid log level ...
Complete command help
Example / output
Change log level of a node (or all nodes)

Usage:
  wp logging set-level [node] [flags]

Flags:
      --all            Set log level on all nodes
  -h, --help           help for set-level
      --level string   Target log level (debug, info, warn, error)

CLI Handbook overview / Profiles & trace

Profiles & trace

Collect CPU, memory, goroutine, mutex and execution-trace diagnostics.

wp profile

Download a Go profile or execution trace from a node

Example / output
wp profile <node> [flags]

Parameters to know

Parameter / type Use and default
--output-file string output file path (default: ./<node>-<type>-<timestamp>.pb.gz; trace uses .trace)
--seconds int duration for cpu/trace profiles (default 30)
--type string profile type: cpu|heap|goroutine|allocs|mutex|block|threadcreate|trace (default "cpu")

Demo

Example / output
wp profile node-0 --type cpu --seconds 10 --output-file node0-cpu.pb.gz

Output and interpretation

Example / output
Saved: node0-cpu.pb.gz
To analyze: go tool pprof -http=127.0.0.1:8081 node0-cpu.pb.gz

CPU uses /debug/pprof/profile. Trace uses go tool trace, not pprof, and saves .trace. allocs is cumulative allocation, heap is live memory; a sparse mutex profile does not prove absence of deadlocks. Goroutine pprof aggregates stacks; it is not a raw stack dump.

Finding or error example
Example / output
pprof returned status 404 / timeout
Complete command help
Example / output
Download a Go profile or execution trace from a node

Usage:
  wp profile <node> [flags]

Flags:
  -h, --help                 help for profile
      --output-file string   output file path (default: ./<node>-<type>-<timestamp>.pb.gz; trace uses .trace)
      --seconds int          duration for cpu/trace profiles (default 30)
      --type string          profile type: cpu|heap|goroutine|allocs|mutex|block|threadcreate|trace (default "cpu")

CLI Handbook overview / In-flight operations

In-flight operations

Inspect running operations, their age and operation-registry pressure.

wp ops list

List in-flight operations

Example / output
wp ops list <node> [flags]

Parameters to know

Parameter / type Use and default
--limit int Max results
--log int Filter by log ID
--longer-than int Only ops running longer than N ms
--seg int Filter by segment ID
--type string Filter by op type (e.g. file.flush)

Demo

Example / output
wp ops list node-0 --longer-than 30000

Output and interpretation

Example / output
OP_TYPE  OP_ID  TRACE_ID  LOG:SEG
# No rows means no matching tracked operation at this instant.

An empty table means no tracked in-flight operations at this instant, not no historical workload. Compare age and operation IDs over time.

Finding or error example
Example / output
logstore.add_entry  append-op-123  ...  42:7
# The same old operation across samples needs investigation.
Complete command help
Example / output
List in-flight operations

Usage:
  wp ops list <node> [flags]

Flags:
  -h, --help              help for list
      --limit int         Max results
      --log int           Filter by log ID
      --longer-than int   Only ops running longer than N ms
      --seg int           Filter by segment ID
      --type string       Filter by op type (e.g. file.flush)

wp ops show

Show detail of a single in-flight operation

Example / output
wp ops show <node> [flags]

Parameters to know

Parameter / type Use and default
--op-id string Operation ID (required)

Demo

Example / output
wp ops show node-0 --op-id append-op-123

Output and interpretation

Example / output
Identity
  op_id: append-op-123
  op_type: logstore.add_entry
Context
  log_id: 42
  segment_id: 7

Copy a real OP_ID from ops list. A completed operation disappearing is expected; registry capacity is bounded. Replace append-op-123 with an ID returned by ops list.

Finding or error example
Example / output
Error: ResourceNotFound: ...
# It may have completed or been evicted since listing.
Complete command help
Example / output
Show detail of a single in-flight operation

Usage:
  wp ops show <node> [flags]

Flags:
  -h, --help           help for show
      --op-id string   Operation ID (required)

wp ops stats

Show op registry utilization and eviction statistics

Example / output
wp ops stats <node> [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp ops stats node-0

Output and interpretation

Example / output
Capacity
  capacity: 1024
  in_use: 0 (0.0%)
Eviction Totals
  old: 0

Counters can be cumulative. Compare snapshots and queue state; eviction can hide old operations, so no listed op is not always conclusive.

Finding or error example
Example / output
old: 12 (STALL SIGNAL)
# This counter is cumulative; sample again to see whether it is growing.
Complete command help
Example / output
Show op registry utilization and eviction statistics

Usage:
  wp ops stats <node> [flags]

Flags:
  -h, --help   help for stats

CLI Handbook overview / Metrics & scenarios

Metrics & scenarios

Inspect Prometheus samples, trends and supported scenario findings.

wp metrics list

List all metric series exposed by a node

Example / output
wp metrics list <node> [flags]

Parameters to know

Parameter / type Use and default
--filter string Substring filter on metric name

Demo

Example / output
wp metrics list node-0

Output and interpretation

Example / output
NAME                                          TYPE     SERIES  HELP
woodpecker_server_logstore_operations_total    COUNTER  6       ...

Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.

Finding or error example
Example / output
# No matching metric: check --filter, node build and whether the series has been created.
Complete command help
Example / output
List all metric series exposed by a node

Usage:
  wp metrics list <node> [flags]

Flags:
      --filter string   Substring filter on metric name
  -h, --help            help for list

wp metrics report

Run scenario-based metrics analysis

Example / output
wp metrics report [node] [flags]

Parameters to know

Parameter / type Use and default
--interval duration Scrape interval within window (default 1s)
--list List available scenarios
--scenario string Scenario name to run
--window duration Collection window duration (default 30s)

Demo

Example / output
wp metrics report node-0 --scenario stuck-read --window 30s --interval 1s

Output and interpretation

Example / output
Result: OK — no findings for scenario "stuck-read"

Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.

Finding or error example
Example / output
Findings (1):
  [RED] read_stall: ...
    -> wp ops list ...
Error: RedFinding: ...
Complete command help
Example / output
Run scenario-based metrics analysis

Usage:
  wp metrics report [node] [flags]

Flags:
  -h, --help                help for report
      --interval duration   Scrape interval within window (default 1s)
      --list                List available scenarios
      --scenario string     Scenario name to run
      --window duration     Collection window duration (default 30s)

wp metrics snapshot

Point-in-time metric snapshot from one or all nodes

Example / output
wp metrics snapshot [node] [flags]

Parameters to know

Parameter / type Use and default
--all Snapshot from all nodes
--metric string Filter to specific metric name

Demo

Example / output
wp metrics snapshot node-0

Output and interpretation

Example / output
NODE    METRIC                                      LABELS  VALUE
node-0  woodpecker_server_logstore_operations_total  ...     5

Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.

Finding or error example
Example / output
# Missing rows can mean absent series or a failed scrape. Check peer reachability.
Complete command help
Example / output
Point-in-time metric snapshot from one or all nodes

Usage:
  wp metrics snapshot [node] [flags]

Flags:
      --all             Snapshot from all nodes
  -h, --help            help for snapshot
      --metric string   Filter to specific metric name

wp metrics top

Cross-node top-N comparison of a specific metric

Example / output
wp metrics top [flags]

Parameters to know

Parameter / type Use and default
--by string Metric name to rank by (required)
--top int Show top N nodes (default 10)

Demo

Example / output
wp metrics top --by woodpecker_server_logstore_operations_total

Output and interpretation

Example / output
RANK  NODE    VALUE
1     node-0  120
2     node-1  100

Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.

Finding or error example
Example / output
# Ranking is by the requested metric value. A cumulative counter is not an interval rate.
Complete command help
Example / output
Cross-node top-N comparison of a specific metric

Usage:
  wp metrics top [flags]

Flags:
      --by string   Metric name to rank by (required)
  -h, --help        help for top
      --top int     Show top N nodes (default 10)

wp metrics watch

Real-time stream of a metric with trend arrows

Example / output
wp metrics watch <metric-name> <node> [flags]

Parameters to know

Parameter / type Use and default
--interval duration Scrape interval (default 1s)

Demo

Example / output
wp metrics watch woodpecker_server_logstore_operations_total node-0 --interval 1s

Output and interpretation

Example / output
TIMESTAMP     VALUE  DELTA  TREND
12:00:01.000  5      -      ...
12:00:02.000  6      +1     ...

Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.

Finding or error example
Example / output
12:00:02.000  (error)
# Or (not found): inspect the scrape/metric selection.
Complete command help
Example / output
Real-time stream of a metric with trend arrows

Usage:
  wp metrics watch <metric-name> <node> [flags]

Flags:
  -h, --help                help for watch
      --interval duration   Scrape interval (default 1s)

CLI Handbook overview / Native diagnosis

Native diagnosis

Inspect current WP Metrics/Admin evidence through bounded snapshot or live sampling. Use wp diag for retained historical evidence.

wp diagnose

Diagnose current WP service state or sample a live window using existing

Example / output
wp diagnose [node] [flags]

Parameters to know

Parameter / type Use and default
--budget duration Whole-command wall-clock budget including discovery (max 10m) (default 1m0s)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--fail-on-findings Return exit 8 for warning/critical findings after rendering the report
--interval duration Interval between live sampling rounds (default 1s)
--list List the implemented typed rule contracts without connecting
--log-name string Explicit WP-client metadata log; requires --segment, --etcd and --meta-prefix
--max-nodes int Maximum discovered targets; select a node if exceeded (default 8)
--max-ops int Maximum ops per node per sample (1..1000) (default 100)
--max-readers int Maximum reader records and lease checks per metadata sample (1..64) (default 16)
--max-response-bytes int Maximum bytes per discovery/source response (default 2097152)
--max-samples int Maximum sampling rounds per node (2..121) (default 31)
--max-series int Maximum retained completion-counter series per sample (default 500)
--max-writers int Maximum writer detail requests per node per sample (1..64; write/all) (default 16)
--mode string Observation mode: live|snapshot; no historical retrieval (default "live")
--read-mode string Expected reader use: unknown|tail|catch-up; not historical metrics retrieval (default "unknown")
--reader string Select one reader by exact name; otherwise inspect up to --max-readers
--scenario string Native scenario: all|write|read|stuck-write|stuck-read (default "all")
--segment int Explicit metadata segment ID for replica/reader checks
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
--window duration Live observation window starting now (max 5m) (default 30s)

Etcd connection flags: see CLI setup.

Demo

Example / output
wp --context milvus-prod diagnose --scenario write --window 30s --max-writers 16 -o json > write-report.json
# Current disk admission; progress rules remain inconclusive:
wp --context milvus-prod diagnose node-0 --scenario write --mode snapshot -o json
# Optional replica check: select the Milvus WP-client metadata namespace explicitly.
wp --context milvus-prod diagnose --scenario write --log-name YOUR_LOG --segment 0 --etcd http://127.0.0.1:2379 --meta-prefix YOUR_WP_CLIENT_PREFIX -o json
# Native read scenario:
wp --context milvus-prod diagnose --scenario read --window 30s -o json > read-report.json
# Optional reader check with a suitable reporting window and explicit WP-client namespace:
wp --context milvus-prod diagnose --scenario read --read-mode catch-up --log-name YOUR_LOG --segment 0 --etcd http://127.0.0.1:2379 --meta-prefix YOUR_WP_CLIENT_PREFIX --window 65s --interval 5s --max-samples 14 --budget 90s -o json
# Existing focused append check:
wp --context milvus-prod diagnose node-0 --scenario stuck-write --window 30s -o json

Output and interpretation

Example / output
{"schema_version":"wp.diagnosis/v1","collection_status":"completed","observation_mode":"live_window","findings":[{"rule_id":"write_stall","outcome":"suspected_stall","evidence_strength":"suspected","unknown":["tenant_association_unknown"]}]}
# Abbreviated illustrative output; inspect actual observations and coverage_gaps.

Source-build capability. Reuses the WP service context and node Admin mappings. The write scenario checks append, writer-local buffer/flush trends, actual disk watermarks and optional explicitly selected log/segment quorum. Missing metadata does not fall back to another namespace. All includes these checks; focused stuck-write/read contracts remain compatible. --window samples from now; snapshot cannot prove a sustained stall. Queue pressure does not prove a disk/fsync cause, and missing replica observations do not prove quorum loss. Maintenance/object-store attribution remains an explicit evidence gap. Exit 0 means completed evaluation, not healthy; --fail-on-findings opts into exit 8 for warning/critical findings. Same-log completions cannot prove an individual op finished; resets, registry limits and missing evidence remain explicit. No external Prometheus, trace or Loki client is included. Source-build capability. Reuses the WP service context and node Admin mappings. The read scenario separates success/EOF and pending server activity from leased reader progress and configured quorum evidence. Reader checks require explicit log/segment and WP-client etcd/prefix; there is no fallback namespace. The source client profile reports about every 30 seconds, so short unchanged-position windows are inconclusive. Fresh renewed attempts plus available data and catch-up expectation are needed for a no-progress suspicion. A missing replica is not quorum loss or corruption; a metadata end is not payload verification. Server ops lack reader identity, and duplicate numeric log IDs do not establish a tenant join. Existing focused stuck-write/read contracts remain compatible. --window samples now, not history. Exit 0 means completed evaluation, not healthy; --fail-on-findings opts into exit 8 for warning/critical findings. No external Prometheus, trace or Loki client is included.

Finding or error example
Example / output
PartialCollection (exit 6): required metrics/ops/writers were unavailable or writer details exceeded the bound; report retained.
Budget exhausted (exit 5) / cancellation (exit 7): partial evidence retained.
Complete command help
Example / output
Diagnose current WP service state or sample a live window using existing
metrics/Admin APIs. With no node argument, inspect all discovered nodes within
the target budget. Reuses cli.yaml, --context, --endpoint and node mappings.

Use --scenario write for append, flush, disk admission and explicit replica
checks, or --scenario read for activity, reader progress and quorum visibility.
Focused stuck-write/stuck-read retain their original v1 contracts. Snapshot mode
captures current evidence but cannot establish a sustained stall or buildup.
Reader progress needs the 30s client-profile reporting cadence and an explicit
read expectation. --window starts sampling now;
historical Prometheus, trace and Loki queries are available under wp diag.

Exit 0 means evaluation completed, including inconclusive/conflicting findings;
it is not a health assertion. --fail-on-findings returns 8 for warning/critical findings.
Required collection failure returns 1 (all failed) or 6 (partial), cancellation 7,
and exhausted wall-clock budget 5. JSON/YAML evidence is retained on these errors.

Usage:
  wp diagnose [node] [flags]

Flags:
      --budget duration                 Whole-command wall-clock budget including discovery (max 10m) (default 1m0s)
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --fail-on-findings                Return exit 8 for warning/critical findings after rendering the report
  -h, --help                            help for diagnose
      --interval duration               Interval between live sampling rounds (default 1s)
      --list                            List the implemented typed rule contracts without connecting
      --log-name string                 Explicit WP-client metadata log; requires --segment, --etcd and --meta-prefix
      --max-nodes int                   Maximum discovered targets; select a node if exceeded (default 8)
      --max-ops int                     Maximum ops per node per sample (1..1000) (default 100)
      --max-readers int                 Maximum reader records and lease checks per metadata sample (1..64) (default 16)
      --max-response-bytes int          Maximum bytes per discovery/source response (default 2097152)
      --max-samples int                 Maximum sampling rounds per node (2..121) (default 31)
      --max-series int                  Maximum retained completion-counter series per sample (default 500)
      --max-writers int                 Maximum writer detail requests per node per sample (1..64; write/all) (default 16)
      --meta-prefix string              Required full client metadata prefix
      --mode string                     Observation mode: live|snapshot; no historical retrieval (default "live")
      --read-mode string                Expected reader use: unknown|tail|catch-up; not historical metrics retrieval (default "unknown")
      --reader string                   Select one reader by exact name; otherwise inspect up to --max-readers
      --scenario string                 Native scenario: all|write|read|stuck-write|stuck-read (default "all")
      --segment int                     Explicit metadata segment ID for replica/reader checks
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
      --window duration                 Live observation window starting now (max 5m) (default 30s)

CLI Handbook overview / Writer runtime

Writer runtime

Inspect loaded writers, buffers, flush queues and LAC; perform targeted flush, compact or fence actions.

wp logstore buffer

Show buffer bytes summary, sorted by size

Example / output
wp logstore buffer <node> [flags]

Parameters to know

Parameter / type Use and default
--top int Show top N segments (default 20)

Demo

Example / output
wp logstore buffer node-0

Output and interpretation

Example / output
LOG:SEG  BUFFER_BYTES  BUFFER_ENTRIES
42:7     4096          4

This inspects node-local runtime state. Take repeated samples and correlate long-running operations; one nonzero queue is not automatically a stall.

Finding or error example
Example / output
# A sustained/growing value with outstanding appends requires flush-path investigation.
Complete command help
Example / output
Show buffer bytes summary, sorted by size

Usage:
  wp logstore buffer <node> [flags]

Flags:
  -h, --help      help for buffer
      --top int   Show top N segments (default 20)

wp logstore compact

Force compaction on a segment at an exact confirmed entry ID

Example / output
wp logstore compact <node> [flags]

Parameters to know

Parameter / type Use and default
--expected-last-entry-id int Exact LastEntryId from coordinator metadata; an incorrect value can publish an invalid footer (-1 only for an empty segment, required)
--log int Log ID (required)
--seg int Segment ID (required)

Demo

Example / output
wp logstore compact node-0 --log 2 --seg 0 --expected-last-entry-id 99

Output and interpretation

Example / output
compact completed on node-0

Use the exact LastEntryId from verified Completed metadata, never guess from a replica’s longer tail. This acts on one replica; client auditor owns lifecycle advancement and mark distribution.

Finding or error example
Example / output
Error: StateConflict: ...
# Verify the exact coordinator last entry; inspect node logs for the rejected compaction.
Complete command help
Example / output
Force compaction on a segment at an exact confirmed entry ID

Usage:
  wp logstore compact <node> [flags]

Flags:
      --expected-last-entry-id int   Exact LastEntryId from coordinator metadata; an incorrect value can publish an invalid footer (-1 only for an empty segment, required)
  -h, --help                         help for compact
      --log int                      Log ID (required)
      --seg int                      Segment ID (required)

wp logstore fence

Fence a segment on a single node.

Example / output
wp logstore fence <node> [flags]

Parameters to know

Parameter / type Use and default
--log int Log ID (required)
--reason string Reason for fencing (required)
--seg int Segment ID (required)
-y, --yes Skip confirmation

Demo

Example / output
wp logstore fence node-0 --log 2 --seg 0 --reason "Reviewed recovery"

Output and interpretation

Example / output
About to fence log 2 segment 0 on node-0. Reason: Reviewed recovery
# Preview only; no -y means no fencing and exit code 7.

Changes write acceptance on one node. One-node fencing does not necessarily block an entire quorum; use fence-quorum for deliberate quorum intervention. Re-run with -y only after reviewing the node and quorum impact.

Finding or error example
Example / output
Missing writer / fence failure
Complete command help
Example / output
Fence a segment on a single node.

On a multi-node quorum this removes one acknowledgement and the write carries on through the rest; interrupting a write takes wq-aq+1 nodes, which is what `fence-quorum` does.

Usage:
  wp logstore fence <node> [flags]

Flags:
  -h, --help            help for fence
      --log int         Log ID (required)
      --reason string   Reason for fencing (required)
      --seg int         Segment ID (required)
  -y, --yes             Skip confirmation

wp logstore fence-quorum

Fence a segment across enough of its quorum to interrupt writes (high-risk, requires --reason)

Example / output
wp logstore fence-quorum <logName> <segmentId> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--nodes strings Fence only these quorum members (default: every node in the quorum)
--reason string Reason for fencing (required)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
-y, --yes Skip confirmation

Etcd connection flags: see CLI setup.

Demo

Example / output
wp logstore fence-quorum my-channel 0 --reason "Verified writer recovery" --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
Selected ...
# Preview only; without -y no RPC fencing is committed (exit 7).

The command previews quorum impact and requires acceptance (-y bypasses the prompt). Fencing can interrupt writes and trigger rolling/recovery; it does not repair data. Re-run the reviewed command with -y to commit.

Finding or error example
Example / output
Acceptance not given / too few selected reachable members
Complete command help
Example / output
Fence a segment across enough of its quorum to interrupt writes (high-risk, requires --reason)

Usage:
  wp logstore fence-quorum <logName> <segmentId> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for fence-quorum
      --meta-prefix string              Required full client metadata prefix
      --nodes strings                   Fence only these quorum members (default: every node in the quorum)
      --reason string                   Reason for fencing (required)
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
  -y, --yes                             Skip confirmation

wp logstore flush-queue

Show flush queue depth summary, sorted by depth

Example / output
wp logstore flush-queue <node> [flags]

Parameters to know

Parameter / type Use and default
--top int Show top N segments (default 20)

Demo

Example / output
wp logstore flush-queue node-0

Output and interpretation

Example / output
LOG:SEG  QUEUE_DEPTH  QUEUE_CAP
42:7     0            16

This inspects node-local runtime state. Take repeated samples and correlate long-running operations; one nonzero queue is not automatically a stall.

Finding or error example
Example / output
42:7     16           16
# Correlate a persistently full queue with file.flush operations and backend latency.
Complete command help
Example / output
Show flush queue depth summary, sorted by depth

Usage:
  wp logstore flush-queue <node> [flags]

Flags:
  -h, --help      help for flush-queue
      --top int   Show top N segments (default 20)

wp logstore force-flush

Force sync on a specific segment or all segments

Example / output
wp logstore force-flush <node> [flags]

Parameters to know

Parameter / type Use and default
--log int Log ID (0 = all)
--seg int Segment ID (0 = all)

Demo

Example / output
wp logstore force-flush node-0 --log 2 --seg 0

Output and interpretation

Example / output
force-flush completed on node-0

Mutates a loaded writer. It cannot repair corrupted payloads; inspect queues and backend errors first.

Finding or error example
Example / output
Error: StateConflict: ...
Complete command help
Example / output
Force sync on a specific segment or all segments

Usage:
  wp logstore force-flush <node> [flags]

Flags:
  -h, --help      help for force-flush
      --log int   Log ID (0 = all)
      --seg int   Segment ID (0 = all)

wp logstore lac

Show how far a segment is confirmed readable across its quorum

Example / output
wp logstore lac <logName> <segmentId> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp logstore lac my-channel 0 --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
quorum LAC: 99 (aq=2 of 3 nodes at or past it)

LAC is confirmed progress reported per replica, not a full checksum verification. Active metadata is not a final boundary. Map every quorum member’s admin address.

Finding or error example
Example / output
quorum LAC: unknown — 1 of 3 nodes answered, fewer than aq=2
Complete command help
Example / output
Show how far a segment is confirmed readable across its quorum

Usage:
  wp logstore lac <logName> <segmentId> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for lac
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp logstore segment-show

Show a segment's active writer on a node (not historical segment metadata)

Example / output
wp logstore segment-show <node> [flags]

Parameters to know

Parameter / type Use and default
--log int Log ID (required)
--seg int Segment ID (required)

Demo

Example / output
wp logstore segment-show node-0 --log 2 --seg 0

Output and interpretation

Example / output
writer: ...
buffer: ...
flush_state: ...
# Illustrative excerpt; fields depend on the storage backend.

--log and --seg are numeric IDs. Historical/unloaded segments need segment inspect and log scan, not this active-writer endpoint.

Finding or error example
Example / output
Error: TargetNotFound: ...
# No loaded writer; historical data may still exist.
Complete command help
Example / output
Show a segment's active writer on a node (not historical segment metadata)

Usage:
  wp logstore segment-show <node> [flags]

Flags:
  -h, --help      help for segment-show
      --log int   Log ID (required)
      --seg int   Segment ID (required)

wp logstore segments

List active segments on a node

Example / output
wp logstore segments <node> [flags]

Parameters to know

Parameter / type Use and default
--log int Filter by log ID
--writable string Filter by writable state (true/false)

Demo

Example / output
wp logstore segments node-0

Output and interpretation

Example / output
LOG_ID SEG_ID BACKEND WRITABLE FENCED FINALIZED ENTRIES BLOCKS

Lists loaded writer objects, not every segment on disk or in etcd. An empty table does not establish no stored data; use log scan and instance data.

Finding or error example
Example / output
Header only / node unavailable
Complete command help
Example / output
List active segments on a node

Usage:
  wp logstore segments <node> [flags]

Flags:
  -h, --help              help for segments
      --log int           Filter by log ID
      --writable string   Filter by writable state (true/false)

CLI Handbook overview / Local data inventory

Local data inventory

List bucket/root instances holding node-local data before drain or reclamation.

wp instance data

List the instances holding node-local data.

Example / output
wp instance data [node] [flags]

Parameters to know

Parameter / type Use and default
--all Query every node in the cluster
--bucket string Filter to one instance's bucket (requires --root)
--root string Filter to one instance's root path (requires --bucket)

Demo

Example / output
wp instance data --all --strict

Output and interpretation

Example / output
NODE    BUCKET         ROOT_PATH  LOGS  SEGMENTS  LIVE  SIZE_BYTES  PROCS  DELETE_STATE
node-0  milvus-bucket  milvus/wp  1     2         1     1048576     1      active

Inventory local instances before deleting or draining. --bucket requires --root (this command uses --root, unlike segment --root-path). No CLI instance-delete command is currently registered; the scenario guide documents the admin API.

Finding or error example
Example / output
A node has scan_errors or is unreachable. A partial inventory does not establish that no data remains.
Complete command help
Example / output
List the instances holding node-local data.

Pair this with 'POST /admin/instance/delete' to reclaim an orphaned instance: query
first, decide, then delete only the nodes that reported the instance.

An unreachable node still holds its data, so a node shown as UNREACHABLE — or any
node reporting scan errors — means the picture is incomplete and nothing should be
deleted on the strength of it. Use --strict to turn that into a non-zero exit.

Usage:
  wp instance data [node] [flags]

Flags:
      --all             Query every node in the cluster
      --bucket string   Filter to one instance's bucket (requires --root)
  -h, --help            help for data
      --root string     Filter to one instance's root path (requires --bucket)

CLI Handbook overview / Log progress & retention

Log progress & retention

Inspect reader progress and coverage, manage verified skip ranges, and advance checkpoint-backed retention.

wp log readers

Show where each of a log's readers has read to

Example / output
wp log readers <logName> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp log readers my-channel --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
READER      OPEN_POS  CURRENT  LAST_REPORT  OPEN_FOR  STATE
reader-123  7:0       7:50     3s           5m        live

Reads leased registration under readers/<logId>/. No live readers means no usable registration here, not no Milvus queries. Reports normally refresh every 30 seconds; compare two snapshots to establish progress. Lease expiry or wrong namespace can hide readers.

Finding or error example
Example / output
Log my-channel (id 42) — no live readers
# Or a stale report: check activity and namespace before declaring a stall.
Complete command help
Example / output
Show where each of a log's readers has read to

Usage:
  wp log readers <logName> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for readers
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp log scan

Compare metadata with replica local coverage; raw verifies local block checksums

Example / output
wp log scan <logName> [flags]

Parameters to know

Parameter / type Use and default
--bucket string Storage bucket; pair with --root-path to locate unloaded or empty segments
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--from-segment int First segment to scan
--mode string quick reads only structure and metadata; raw reads the data through the real checksums (default "quick")
--root-path string Storage instance root path; pair with --bucket
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
--to-segment int Last segment to scan (default: the newest) (default -1)

Etcd connection flags: see CLI setup.

Demo

Example / output
wp log scan my-channel --mode raw --bucket milvus-bucket --root-path milvus/wp --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
SEGMENT STATE META LAST REACHES REPLICAS VERDICT
1 Active -1 (open) - 3/3 open
Per replica: local_empty / none / no_local_blocks

REACHES is the union of observed local ranges; - means none observed. 3/3 counts usable inspection responses, not three healthy data copies. quick checks structure; raw verifies local blocks. Sealed/compacted data in object storage is not verified. Explicit bucket/root-path resolves empty or unloaded targets; pending writes are not inspected.

Finding or error example
Example / output
unknown: HTTP 404 ... / short: missing entries 50..99
Complete command help
Example / output
Compare metadata with replica local coverage; raw verifies local block checksums

Usage:
  wp log scan <logName> [flags]

Flags:
      --bucket string                   Storage bucket; pair with --root-path to locate unloaded or empty segments
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --from-segment int                First segment to scan
  -h, --help                            help for scan
      --meta-prefix string              Required full client metadata prefix
      --mode string                     quick reads only structure and metadata; raw reads the data through the real checksums (default "quick")
      --root-path string                Storage instance root path; pair with --bucket
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
      --to-segment int                  Last segment to scan (default: the newest) (default -1)

wp log skip-range add

Declare an entry range unreadable so readers move past it.

Example / output
wp log skip-range add <logName> <segmentId> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--force Declare the range even though a replica can still read part of it
--from-entry int First entry to skip (inclusive) (default -1)
--reason string Why these entries are being given up (required)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
--to-entry int Last entry to skip (inclusive) (default -1)
-y, --yes Proceed without the confirmation prompt

Etcd connection flags: see CLI setup.

Demo

Example / output
wp log skip-range add my-channel 0 --from-entry 50 --to-entry 50 --reason "Confirmed unrecoverable entry" --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
Ranges and reasons / preview or committed declaration

A declaration makes readers skip accepted data loss; it does not fix replicas or advance compaction. Entry bounds are inclusive. add previews before confirmation (-y accepts); remove does not restore skipped/reclaimed data. Inspect all replicas and the healthy tail first.

Finding or error example
Example / output
Gate refuses because a replica can serve / acceptance not given
Complete command help
Example / output
Declare an entry range unreadable so readers move past it.

This gives up data. Entries in the range are never delivered to any reader of this log
again, and nothing restores them: withdrawing the range only exposes whatever is still on
disk. So the quorum is asked first, and a replica that can still read part of the range
refuses the declaration.

Usage:
  wp log skip-range add <logName> <segmentId> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --force                           Declare the range even though a replica can still read part of it
      --from-entry int                  First entry to skip (inclusive) (default -1)
  -h, --help                            help for add
      --meta-prefix string              Required full client metadata prefix
      --reason string                   Why these entries are being given up (required)
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
      --to-entry int                    Last entry to skip (inclusive) (default -1)
  -y, --yes                             Proceed without the confirmation prompt

wp log skip-range list

Show the declared skip ranges.

Example / output
wp log skip-range list [logName] [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp log skip-range list my-channel --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
log 42 segment 7: 50-50
# Illustrative excerpt; inspect the recorded reason and age.

A declaration makes readers skip accepted data loss; it does not fix replicas or advance compaction. Entry bounds are inclusive. add previews before confirmation (-y accepts); remove does not restore skipped/reclaimed data. Inspect all replicas and the healthy tail first.

Finding or error example
Example / output
# No declarations is normal when no loss has been accepted.
Complete command help
Example / output
Show the declared skip ranges.

With no log name every log is listed, including ranges whose log no longer exists: log ids
are never reused, so such a range can never apply to anything again, but it stays in the
record until it is removed.

Usage:
  wp log skip-range list [logName] [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for list
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp log skip-range remove

Withdraw a declared range, so readers stop skipping it.

Example / output
wp log skip-range remove <logName> <segmentId> | --log-id N <segmentId> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--from-entry int First entry of the range to withdraw (inclusive) (default -1)
--log-id int Withdraw by log id instead of name, for a range whose log has been deleted (default -1)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
--to-entry int Last entry of the range to withdraw (inclusive) (default -1)

Etcd connection flags: see CLI setup.

Demo

Example / output
wp log skip-range remove my-channel 0 --from-entry 50 --to-entry 50 --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
Removed: ...
# Inspect list afterward to verify the remaining intervals.

A declaration makes readers skip accepted data loss; it does not fix replicas or advance compaction. Entry bounds are inclusive. add previews before confirmation (-y accepts); remove does not restore skipped/reclaimed data. Inspect all replicas and the healthy tail first.

Finding or error example
Example / output
Error: StateConflict: ...
# Refresh and retry after reviewing concurrent changes.
Complete command help
Example / output
Withdraw a declared range, so readers stop skipping it.

Withdrawing does not restore anything: it only lets readers try the entries again, which
is what to do once a replica has been rebuilt. A reader that already moved past the range
does not come back for it.

Usage:
  wp log skip-range remove <logName> <segmentId> | --log-id N <segmentId> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --from-entry int                  First entry of the range to withdraw (inclusive) (default -1)
  -h, --help                            help for remove
      --log-id int                      Withdraw by log id instead of name, for a range whose log has been deleted (default -1)
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
      --to-entry int                    Last entry of the range to withdraw (inclusive) (default -1)

wp log truncate

Advance the truncation frontier with the SDK's metadata-update semantics and additional operator guards. Defaults to preview. The target must be a non-empty Completed or Sealed segment. Subsequent auditor rounds mark earlier finalized segments Truncated and reclaim data according to retention and reader protection. The frontier segment is retained. Apply can cause permanent data loss; verify application checkpoints first.

Example / output
wp log truncate <logName> [flags]

Parameters to know

Parameter / type Use and default
--apply Advance the frontier (default: preview only)
--backup string New pre-change backup file; never overwritten
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--entry int Target entry ID (required) (default -1)
--expected-revision int Log record ModRevision from preview
--reason string Operator reason recorded in audit
--segment int Target segment ID (required) (default -1)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Verify the application checkpoint first. For the preview/apply sequence and asynchronous cleanup, follow the retention workflow.

Example / output
wp log truncate my-channel --segment 8 --entry 0 --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "applied": false,
  "unchanged": false,
  "before": {
    "revision": 17099,
    "metadata": {
      "truncated_segment_id": "6",
      "truncated_entry_id": "99"
    }
  },
  "after": {
    "revision": 17099,
    "metadata": {
      "truncated_segment_id": "8",
      "truncated_entry_id": "0"
    }
  },
  "target_segment_revision": 175,
  "snapshot_revision": 18000
}

Preview requires a non-empty Completed/Sealed target and entry within 0..lastEntryId. Apply requires --apply --expected-revision (the log before.revision), --reason and a new --backup; no offline flag is required. The log and target revisions are compared and an audit record is written atomically. The frontier segment stays retained; earlier finalized segments become eligible for auditor/retention/reader-protected cleanup. Unchanged frontier means applied=false, unchanged=true with no artifacts. Verify application checkpoints first: metadata backup cannot restore reclaimed data.

Finding or error example
Example / output
Error: StateConflict: truncation cannot move backward or restore deleted data
# Active/empty targets, identity mismatches, stale revisions and concurrent updates are also rejected.
Complete command help
Example / output
Advance the truncation frontier with the SDK's metadata-update semantics and additional operator guards. Defaults to preview. The target must be a non-empty Completed or Sealed segment. Subsequent auditor rounds mark earlier finalized segments Truncated and reclaim data according to retention and reader protection. The frontier segment is retained. Apply can cause permanent data loss; verify application checkpoints first.

Usage:
  wp log truncate <logName> [flags]

Flags:
      --apply                           Advance the frontier (default: preview only)
      --backup string                   New pre-change backup file; never overwritten
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --entry int                       Target entry ID (required) (default -1)
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --expected-revision int           Log record ModRevision from preview
  -h, --help                            help for truncate
      --meta-prefix string              Required full client metadata prefix
      --reason string                   Operator reason recorded in audit
      --segment int                     Target segment ID (required) (default -1)
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

CLI Handbook overview / Segment diagnosis

Segment diagnosis

Probe actual entry reads and inspect local blocks to locate damage on each quorum member.

wp segment inspect

Walk a segment's blocks on every replica and report what is wrong where

Example / output
wp segment inspect <logName> <segmentId> [flags]

Parameters to know

Parameter / type Use and default
--bucket string Storage bucket; pair with --root-path (otherwise discovered on each node)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--from-block int First block to walk (default: the start of the segment)
--max-blocks int Stop after this many blocks per node (default: the node's own bound)
--root-path string Storage instance root path; pair with --bucket
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp segment inspect my-channel 1 --bucket milvus-bucket --root-path milvus/wp --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
BLOCK ENTRIES READABLE RECOVERABLE STATUS
- - - - no_local_blocks

Walks local blocks and records to locate damage boundaries; it can report later recoverable entries despite a bad block. no_local_blocks says nothing about shared object integrity. Use actual timestamps/metadata separately; --max-blocks bounds the survey.

Finding or error example
Example / output
Block 0: checksum mismatch; entry 50 corrupt; later good entries
Complete command help
Example / output
Walk a segment's blocks on every replica and report what is wrong where

Usage:
  wp segment inspect <logName> <segmentId> [flags]

Flags:
      --bucket string                   Storage bucket; pair with --root-path (otherwise discovered on each node)
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --from-block int                  First block to walk (default: the start of the segment)
  -h, --help                            help for inspect
      --max-blocks int                  Stop after this many blocks per node (default: the node's own bound)
      --meta-prefix string              Required full client metadata prefix
      --root-path string                Storage instance root path; pair with --bucket
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp segment probe

Ask every replica how far it can read a segment

Example / output
wp segment probe <logName> <segmentId> [flags]

Parameters to know

Parameter / type Use and default
--bucket string Storage bucket; pair with --root-path (otherwise discovered on each node)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--from-entry int Entry to read from (default: the start of the segment)
--max-entries int Stop after this many entries per node (default: the node's own bound)
--root-path string Storage instance root path; pair with --bucket
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp segment probe my-channel 1 --bucket milvus-bucket --root-path milvus/wp --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
SOURCE object_storage  SERVED 0-0 (1)  OUTCOME end_of_file  VERDICT served

Actually probes sequential entry reads and stops at a failure or bound. 0-0 (1) means one entry, ID 0. Object storage responses share one copy. Active tail/no local data has unknown final boundary and must not be called a permanent stall; Completed shortfall is compared to metadata. cap_reached is a bound, not EOF.

Finding or error example
Example / output
Completed expected 99, replica stops at 49 → short / RedFinding if all fail
Complete command help
Example / output
Ask every replica how far it can read a segment

Usage:
  wp segment probe <logName> <segmentId> [flags]

Flags:
      --bucket string                   Storage bucket; pair with --root-path (otherwise discovered on each node)
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --from-entry int                  Entry to read from (default: the start of the segment)
  -h, --help                            help for probe
      --max-entries int                 Stop after this many entries per node (default: the node's own bound)
      --meta-prefix string              Required full client metadata prefix
      --root-path string                Storage instance root path; pair with --bucket
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

CLI Handbook overview / Metadata inspection & repair

Metadata inspection & repair

Read decoded coordinator records, check metadata constraints, export snapshots and apply reviewed corrections.

1. Read the record before choosing a repair

Use a current source build containing wp meta and wp log truncate; these additions are not in v0.1.46. Complete CLI setup first. Supply explicit reachable etcd endpoints and the complete client metadata prefix for these commands, or select a private client parameter file. Server contexts do not select metadata authority. Pure metadata views need no node Admin connection or bucket/root path.

Example / output
wp --context milvus-prod meta log list
wp --context milvus-prod meta log show my-channel
wp --context milvus-prod meta segment list my-channel
wp --context milvus-prod meta segment show my-channel 7
wp --context milvus-prod meta check my-channel

Use the real logical log name. list includes persisted history even when node-local writer objects are unloaded. show wraps decoded metadata with its exact key, modification revision and lease; int64 fields appear as decimal strings. For modern segments, inspect the embedded quorum in segment show; separately stored quorum records may be absent.

2. Create a small JSON patch file

Pass only fields to change, not the show envelope or a full exported record. For example, create the files used by the command demos below:

Example / output
cat > ./log-policy.json <<'JSON'
{"max_segment_roll_time_seconds": 120}
JSON

# Only after independently proving that 99 is the correct finalized tail:
cat > ./segment-repair.json <<'JSON'
{"lastEntryId": "99"}
JSON
Patch target Writable fields / constraints
log max_segment_roll_time_seconds, max_segment_roll_size_bytes, compaction_buffer_size_bytes, max_compaction_file_count. Nonnegative; zero is allowed. These are reserved metadata fields, not live client runtime configuration.
segment size, createTime, completionTime, sealedTime, lastEntryId, state. Times are Unix milliseconds; size is bytes. Lifecycle times must agree with state. Integers may be numbers or decimal strings.
Protected Identity (logId/segNo), quorum and log truncation fields. Quorums are read-only; advance normal retention with log truncate.

Active requires lastEntryId=-1 and zero completion/sealed times. A verified rollback to Active must reset all three together. Completed/Sealed lastEntryId=-1 means finalized empty; Sealed needs a valid sealedTime. A key/segNo mismatch is rejected. Changing state does not generate compacted objects, repair data or execute recovery; Truncated can enable cleanup.

State-correction JSON examples

Use only the state independently verified for this segment. These are alternative patch-file contents, not automatic recovery steps. For a verified rollback to Active:

Example / output
{"state":"Active","lastEntryId":"-1","completionTime":"0","sealedTime":"0"}

For a verified Completed-to-Sealed correction with existing compacted files and a known sealing time:

Example / output
{"state":"Sealed","lastEntryId":"99","sealedTime":"1791806000000"}

The timestamp is illustrative: replace it with the verified Unix-millisecond value, after completionTime. State names are Active, Completed, Sealed and Truncated (numbers 0, 2, 3, 4); reserved state 1 is rejected. A legacy createTime=0 must also be corrected with a verified positive creation time.

3. Preview, stop metadata writers, then apply

Example / output
wp --context milvus-prod meta segment patch my-channel 7 \
  --patch-file ./segment-repair.json

A preview returns applied=false, before/after records, changes and snapshot_revision. Review all changes. Use the target's before.revision for apply. If it is 174, stop the relevant writers/auditors and keep them stopped throughout the repair:

Example / output
wp --context milvus-prod meta segment patch my-channel 7 \
  --patch-file ./segment-repair.json --apply --offline \
  --expected-revision 174 --reason 'Correct verified tail after software fix' \
  --backup ./segment-before-repair.json

wp --context milvus-prod meta segment show my-channel 7
wp --context milvus-prod meta check my-channel

--offline acknowledges your action; it does not stop clients. Apply checks revision and writer-lock absence, persists a new backup, and writes the correction and audit record atomically. Restart affected clients after repair so cached metadata is refreshed. applied=true/new_revision confirms the write, not data readability. Reused backup paths fail; stale revisions or concurrent changes require a fresh preview and a new backup filename.

Example / output
# Example warning on an otherwise valid scalar repair:
"warnings": [
  "embedded quorum (unchanged): invalid quorum sizes or node count"
]

An existing embedded-quorum problem is preserved and reported in warnings; it does not block an independent scalar repair. meta check still reports that unresolved problem. Lifecycle and truncation-bound violations continue to block the patch. Do not mark a segment Sealed or lower a tail just to silence a physical corruption finding.

4. Interpret checks, exports and failures

Result Meaning / next step
check: state=ok, issues=[]; exit 0 The inspected metadata constraints passed. Probe/inspect separately for physical data; verify application progress separately.
check: state=invalid; exit 9 Read the exact key/field in issues. A bad frontier, lifecycle or quorum is metadata evidence, not automatic permission to rewrite it.
TargetNotFound; exit 3 The exact record is absent. Verify context, prefix and logical IDs before diagnosing missing payloads.
Patch StateConflict; exit 4 Read the constraint, revision or writer-lock error. Re-read records and correct the verified cause; do not blindly retry apply.
export / backup New mode-0600 file with readable known metadata and original value_base64/revisions/leases. File and parent directory must sync before patch commits. Unknown/malformed records retain their bytes. This is not a payload backup or automatic rollback command.

If commit returns a network error, re-read the target and audit records before retrying: the write outcome may be unknown. For a complete operational example, follow verified metadata repair in Troubleshooting.

wp meta check

Validate log/segment metadata constraints (does not verify physical data)

Example / output
wp meta check <logName> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp meta check my-channel --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "issues": [],
  "log_name": "my-channel",
  "scope": "metadata only; use segment inspect/log scan for physical data",
  "segments_checked": 2,
  "snapshot_revision": 18000,
  "state": "ok"
}

Checks log policy/frontier fields, segment identity/state/timestamps, embedded quorum structure and frontier bounds at one snapshot. Exit 9 plus issues means inconsistent metadata. state=ok establishes metadata consistency only; it does not prove payload readability or application progress.

Finding or error example
Example / output
{
  "state": "invalid",
  "issues": [
    "milvus-prod/woodpecker/logs/my-channel/segments/7: key must use canonical segment ID 8 matching segNo"
  ]
}
Error: RedFinding: 1 metadata constraint violations
Complete command help
Example / output
Validate log/segment metadata constraints (does not verify physical data)

Usage:
  wp meta check <logName> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for check
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp meta export

Export one metadata namespace at a consistent etcd revision

Example / output
wp meta export [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--output-file string New snapshot file (required; never overwritten)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp meta export --output-file ./metadata-snapshot.json --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "file": "./metadata-snapshot.json",
  "keys": 5,
  "snapshot_revision": 18000
}

Creates a new mode-0600 namespace snapshot. The file contains readable metadata for known protobuf records plus byte-exact value_base64, revisions and leases. Unknown/corrupt values retain bytes and encoding/decode errors. It is a metadata backup, not a payload backup or automatic restore command; an existing file is never overwritten.

Finding or error example
Example / output
Error: Config: create new backup file: open ./metadata-snapshot.json: file exists
Complete command help
Example / output
Export one metadata namespace at a consistent etcd revision

Usage:
  wp meta export [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for export
      --meta-prefix string              Required full client metadata prefix
      --output-file string              New snapshot file (required; never overwritten)
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp meta log list

list log metadata and etcd revision

Example / output
wp meta log list [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp meta log list --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "prefix": "milvus-prod/woodpecker",
  "snapshot_revision": 18000,
  "records": [
    {
      "kind": "log",
      "key": "milvus-prod/woodpecker/logs/my-channel",
      "revision": 17099,
      "lease": 0,
      "metadata": {
        "logId": "42",
        "truncated_segment_id": "6",
        "truncated_entry_id": "99"
      }
    }
  ]
}

Lists top-level log records, including unloaded history. records is a snapshot, not a list of live node writers. Narrow with meta log show; use the record revision for a patch.

Finding or error example
Example / output
# records: [] means this listing has no matching records.
# An undecodable record instead returns a configuration error.
Complete command help
Example / output
list log metadata and etcd revision

Usage:
  wp meta log list [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for list
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp meta log patch

Patch policy/statistics/timestamps, segment lastEntryId and state. Identity, quorum and log truncation are protected. Segment state correction does not create compacted files or repair replica contents; Truncated can enable cleanup. Applying requires stopped writers/auditors, a revision, reason, and a new backup file.

Example / output
wp meta log patch <logName> [flags]

Parameters to know

Parameter / type Use and default
--apply Commit the previewed correction (default: no writes)
--backup string New pre-change backup file (required for apply; never overwritten)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--expected-revision int Required target ModRevision for apply
--offline Confirm relevant writers/auditors are stopped and will remain stopped during repair
--patch-file string JSON object containing supported scalar fields (required)
--reason string Operator's reason, recorded with the change
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Create the patch file using the JSON examples in the metadata walkthrough before running this preview. Applying requires separately reviewed evidence and the guards described there.

Example / output
wp meta log patch my-channel --patch-file ./log-policy.json --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "applied": false,
  "before": {
    "revision": 17099
  },
  "changes": {
    "max_segment_roll_time_seconds": {
      "before": "60",
      "after": "120"
    }
  }
}

Create log-policy.json using the walkthrough above. Only max_segment_roll_time_seconds, max_segment_roll_size_bytes, compaction_buffer_size_bytes and max_compaction_file_count are patchable (nonnegative; zero allowed). They are reserved metadata fields, not live runtime configuration. Preview writes nothing. Apply uses the target before.revision plus offline acknowledgement, reason and a new durable backup; unknown fields and leases are preserved.

Finding or error example
Example / output
Error: Usage: --apply requires --offline, --expected-revision, --reason, and --backup
# A stale revision or live writer lock returns StateConflict; no correction is applied.
Complete command help
Example / output
Patch policy/statistics/timestamps, segment lastEntryId and state. Identity, quorum and log truncation are protected. Segment state correction does not create compacted files or repair replica contents; Truncated can enable cleanup. Applying requires stopped writers/auditors, a revision, reason, and a new backup file.

Usage:
  wp meta log patch <logName> [flags]

Flags:
      --apply                           Commit the previewed correction (default: no writes)
      --backup string                   New pre-change backup file (required for apply; never overwritten)
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --expected-revision int           Required target ModRevision for apply
  -h, --help                            help for patch
      --meta-prefix string              Required full client metadata prefix
      --offline                         Confirm relevant writers/auditors are stopped and will remain stopped during repair
      --patch-file string               JSON object containing supported scalar fields (required)
      --reason string                   Operator's reason, recorded with the change
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp meta log show

show log metadata and etcd revision

Example / output
wp meta log show <logName> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp meta log show my-channel --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "kind": "log",
  "key": "milvus-prod/woodpecker/logs/my-channel",
  "revision": 17099,
  "lease": 0,
  "metadata": {
    "logId": "42",
    "truncated_segment_id": "6",
    "truncated_entry_id": "99"
  }
}

Shows the exact log key, ModRevision, lease and decoded fields. int64 values are decimal strings. A missing target means the key is absent in this namespace, not a damaged replica. Log truncation fields are changed with log truncate, not generic patch.

Finding or error example
Example / output
Error: TargetNotFound: target not found: milvus-prod/woodpecker/logs/my-channel
Complete command help
Example / output
show log metadata and etcd revision

Usage:
  wp meta log show <logName> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for show
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp meta quorum list

list quorum metadata and etcd revision

Example / output
wp meta quorum list [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp meta quorum list --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "prefix": "milvus-prod/woodpecker",
  "snapshot_revision": 18000,
  "records": [
    {
      "kind": "quorum",
      "key": "milvus-prod/woodpecker/quorums/1",
      "revision": 100,
      "lease": 0,
      "metadata": {
        "id": "1",
        "es": 3,
        "wq": 3,
        "aq": 2,
        "nodes": [
          "node0:18080",
          "node1:18080",
          "node2:18080"
        ]
      }
    }
  ]
}

Lists separately stored quorum records. Modern segments embed their quorum, so an empty list can be normal; inspect meta segment show to see that topology. This group cannot patch quorum records.

Finding or error example
Example / output
# records: [] means this listing has no matching records.
# An undecodable record instead returns a configuration error.
Complete command help
Example / output
list quorum metadata and etcd revision

Usage:
  wp meta quorum list [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for list
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp meta quorum show

show quorum metadata and etcd revision

Example / output
wp meta quorum show <ID> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp meta quorum show 1 --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "kind": "quorum",
  "key": "milvus-prod/woodpecker/quorums/1",
  "revision": 100,
  "lease": 0,
  "metadata": {
    "id": "1",
    "es": 3,
    "wq": 3,
    "aq": 2,
    "nodes": [
      "node0:18080",
      "node1:18080",
      "node2:18080"
    ]
  }
}

Shows one separately stored quorum by numeric ID; it does not fan out or test node health. es is ensemble size, wq write quorum, aq acknowledgement quorum. Modern embedded quorum is read through meta segment show.

Finding or error example
Example / output
Error: TargetNotFound: target not found: milvus-prod/woodpecker/quorums/1
Complete command help
Example / output
show quorum metadata and etcd revision

Usage:
  wp meta quorum show <ID> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for show
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp meta segment list

list segment metadata and etcd revision

Example / output
wp meta segment list <logName> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp meta segment list my-channel --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "prefix": "milvus-prod/woodpecker",
  "snapshot_revision": 18000,
  "records": [
    {
      "kind": "segment",
      "key": "milvus-prod/woodpecker/logs/my-channel/segments/7",
      "revision": 174,
      "lease": 0,
      "metadata": {
        "segNo": "7",
        "lastEntryId": "99",
        "state": "Completed"
      }
    }
  ]
}

Lists persisted segments under this exact log, including unloaded history. The parent log must exist. State and lastEntryId are metadata claims; probe/inspect physical data separately.

Finding or error example
Example / output
# records: [] means this listing has no matching records.
# An undecodable record instead returns a configuration error.
Complete command help
Example / output
list segment metadata and etcd revision

Usage:
  wp meta segment list <logName> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for list
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp meta segment patch

Patch policy/statistics/timestamps, segment lastEntryId and state. Identity, quorum and log truncation are protected. Segment state correction does not create compacted files or repair replica contents; Truncated can enable cleanup. Applying requires stopped writers/auditors, a revision, reason, and a new backup file.

Example / output
wp meta segment patch <logName> <segmentId> [flags]

Parameters to know

Parameter / type Use and default
--apply Commit the previewed correction (default: no writes)
--backup string New pre-change backup file (required for apply; never overwritten)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--expected-revision int Required target ModRevision for apply
--offline Confirm relevant writers/auditors are stopped and will remain stopped during repair
--patch-file string JSON object containing supported scalar fields (required)
--reason string Operator's reason, recorded with the change
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Create the patch file using the JSON examples in the metadata walkthrough before running this preview. Applying requires separately reviewed evidence and the guards described there.

Example / output
wp meta segment patch my-channel 7 --patch-file ./segment-repair.json --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "applied": false,
  "before": {
    "revision": 174
  },
  "changes": {
    "lastEntryId": {
      "before": "999",
      "after": "99"
    }
  }
}

Create segment-repair.json using the walkthrough above and independent data evidence. Writable fields are size, createTime, completionTime, sealedTime, lastEntryId and state. Lifecycle constraints remain strict; key identity, quorum and log truncation fields are protected. An existing invalid embedded quorum is preserved and reported in warnings rather than blocking a scalar repair. meta check will still report that unresolved quorum. A state edit does not create compacted files or repair replica contents.

Finding or error example
Example / output
Error: StateConflict: lastEntryId is below the log truncation frontier
# Other examples: key/segNo mismatch; Active segment requires lastEntryId -1; Sealed segment requires sealedTime.
Complete command help
Example / output
Patch policy/statistics/timestamps, segment lastEntryId and state. Identity, quorum and log truncation are protected. Segment state correction does not create compacted files or repair replica contents; Truncated can enable cleanup. Applying requires stopped writers/auditors, a revision, reason, and a new backup file.

Usage:
  wp meta segment patch <logName> <segmentId> [flags]

Flags:
      --apply                           Commit the previewed correction (default: no writes)
      --backup string                   New pre-change backup file (required for apply; never overwritten)
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --expected-revision int           Required target ModRevision for apply
  -h, --help                            help for patch
      --meta-prefix string              Required full client metadata prefix
      --offline                         Confirm relevant writers/auditors are stopped and will remain stopped during repair
      --patch-file string               JSON object containing supported scalar fields (required)
      --reason string                   Operator's reason, recorded with the change
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp meta segment show

show segment metadata and etcd revision

Example / output
wp meta segment show <logName> <segmentId> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp meta segment show my-channel 7 --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
{
  "kind": "segment",
  "key": "milvus-prod/woodpecker/logs/my-channel/segments/7",
  "revision": 174,
  "lease": 0,
  "metadata": {
    "segNo": "7",
    "lastEntryId": "99",
    "state": "Completed"
  }
}

Shows authoritative persisted metadata and its embedded quorum when present. Active lastEntryId=-1 is an unknown final boundary; Completed/Sealed lastEntryId=-1 means finalized empty. A key/segNo mismatch blocks patching. This read does not verify local or object-storage payloads.

Finding or error example
Example / output
Error: TargetNotFound: target not found: milvus-prod/woodpecker/logs/my-channel/segments/7
Complete command help
Example / output
show segment metadata and etcd revision

Usage:
  wp meta segment show <logName> <segmentId> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for show
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

CLI Handbook overview / Compacted-mark distribution

Compacted-mark distribution

Inspect notification records and settle verified manual mark-distribution cases.

wp marking confirm

Confirm one compacted-mark distribution record.

Example / output
wp marking confirm <logId> <segmentId> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--force confirm (transition to OPERATOR_CONFIRMED) even if the record is not PENDING_MANUAL; IN_PROGRESS/COMPLETED are normally managed automatically
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp marking confirm 2 0 --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
Record set to OPERATOR_CONFIRMED

After verifying the permanently unavailable/unacked replica, settle notification retries. This preserves the record until truncation reaps it; it does not repair a segment or delete data immediately.

Finding or error example
Example / output
StateConflict for IN_PROGRESS / COMPLETED without --force
Complete command help
Example / output
Confirm one compacted-mark distribution record.

Meant for records parked as NOTIFY_PENDING_MANUAL: after verifying the unacked node
(typically permanently dead or removed from the cluster), confirm transitions the record
to NOTIFY_OPERATOR_CONFIRMED, which settles it (the auditor stops retrying and stops
counting it against the log). The etcd record itself is intentionally NOT deleted here;
it is reaped when the segment is truncated, like every other marking record.
Refuses records still IN_PROGRESS or COMPLETED unless --force is given (those are
managed automatically and normally need no operator action).

Usage:
  wp marking confirm <logId> <segmentId> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
      --force                           confirm (transition to OPERATOR_CONFIRMED) even if the record is not PENDING_MANUAL; IN_PROGRESS/COMPLETED are normally managed automatically
  -h, --help                            help for confirm
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp marking list

List compacted-mark distribution records (default: only NOTIFY_PENDING_MANUAL)

Example / output
wp marking list [flags]

Parameters to know

Parameter / type Use and default
--all-states include IN_PROGRESS and COMPLETED records, not just PENDING_MANUAL
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--log int restrict to one log id (default: all logs)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp marking list --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker

Output and interpretation

Example / output
COMPLETED / operator-confirmed records

Records describe compacted-mark distribution, not the data payload. Investigate unacked nodes and physical cleanup before manual settlement.

Finding or error example
Example / output
NOTIFY_PENDING_MANUAL with unacked nodes
Complete command help
Example / output
List compacted-mark distribution records (default: only NOTIFY_PENDING_MANUAL)

Usage:
  wp marking list [flags]

Flags:
      --all-states                      include IN_PROGRESS and COMPLETED records, not just PENDING_MANUAL
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for list
      --log int                         restrict to one log id (default: all logs)
      --meta-prefix string              Required full client metadata prefix
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

CLI Handbook overview / Raw etcd exploration

Raw etcd exploration

Discover namespaces and inspect exact raw keys, revisions and values without writing etcd.

1. Discover keys before choosing a metadata prefix

This command group is included in v0.1.47 and later; it is not included in v0.1.46. Install a matching release or build from source, and confirm wp etcd --help before using these examples.

Use this read-only group when you can reach etcd but do not yet know the Woodpecker namespace. It connects directly to etcd: no Woodpecker admin endpoint, node mapping or metadata prefix is required. Supply a reachable endpoint explicitly, or select a private client parameter file.

Example / output
# Omit the prefix to explore all keys (first page only).
wp etcd list --etcd 127.0.0.1:32379 --limit 20
# Narrow the listing using an actual prefix found in that output.
wp etcd list milvus-prod/ --etcd 127.0.0.1:32379 --limit 20

Illustrative excerpt:

Example / output
KEY                              MOD_REV  VERSION  LEASE
milvus-prod/woodpecker/version    90       1        0

1 keys shown; snapshot revision 105; more=false

A prefix is literal: list / lists only keys starting with a slash; it does not mean all keys. Empty results mean no matching keys in this snapshot, not an unhealthy cluster. A connection or permission error means the listing could not be established. This group never prepends a metadata prefix.

2. Inspect a key and understand its value

Example / output
wp etcd get milvus-prod/woodpecker/version --etcd 127.0.0.1:32379 -o json
wp etcd cat milvus-prod/woodpecker/version --etcd 127.0.0.1:32379 --encoding base64
Operation / option What you get
get; -o json / -o yaml Exact key, key revision, lease, snapshot revision, encoding and value. Default get output is JSON.
cat The value alone. It is useful for piping or saving a single value.
--encoding auto Readable UTF-8 text is printed as text; binary/control-byte content becomes base64. get reports the selected value_encoding; cat prints a base64 notice on stderr.
--encoding text / base64 / hex Choose the representation explicitly. text rejects invalid UTF-8. Protobuf bytes do not become decoded metadata simply by selecting text.
cat --encoding raw Exact original bytes, without an appended newline. Redirect to a file rather than displaying binary content in a terminal.
Example / output
wp etcd cat milvus-prod/woodpecker/version \
  --etcd 127.0.0.1:32379 --encoding raw > /tmp/wp-version.pb

A TargetNotFound result means that exact key is absent at the requested snapshot. Base64 is a reversible byte representation, not evidence of damaged data. This raw group does not decode, patch or restore metadata; the file above preserves one value, not a complete etcd backup.

Once you know the Woodpecker prefix, use Metadata inspection & repair for decoded log/segment records and readable namespace exports. Supply that complete prefix through context or --meta-prefix.

3. Continue a listing at the same snapshot

Use JSON output for pagination. When more is true, copy next_key_base64 and snapshot_revision from that response into the next request, keeping the same prefix:

Example / output
wp etcd list milvus-prod/ --etcd 127.0.0.1:32379 --limit 20 -o json

# Replace these placeholders with the returned cursor and snapshot revision.
wp etcd list milvus-prod/ --etcd 127.0.0.1:32379 --limit 20 -o json \
  --after-key-base64 'CURSOR_FROM_PREVIOUS_RESPONSE' --revision 105

Repeat until more=false. The base64 cursor preserves binary keys. The keys array contains this page; count describes matching keys remaining in the queried range, so it can exceed the page length. --after-key is an alternative for ordinary text keys; never combine it with --after-key-base64.

Use --revision on get/cat too when comparing values with a listing. If etcd has compacted that revision, restart the listing at a new snapshot; do not silently mix pages from different revisions.

4. Reuse explicit client parameters

Keep etcd authority separate from server contexts. For repeated operations, explicitly select a private client parameter file:

Example / output
# /path/to/client-parameters.yaml
endpoints: [http://127.0.0.1:32379]
# Add host-local TLS/authentication settings when required.
Example / output
wp etcd list --client-parameters-file /path/to/client-parameters.yaml --limit 20
wp etcd get tenant/woodpecker/version --etcd http://127.0.0.1:32379

Explicit flags override the private parameter file. Raw operations do not use context metadata authority or Admin discovery. See etcd connection settings for authentication/TLS fields. Recent input suggestions are independent values, not a saved tenant profile.

wp etcd cat

Print only one key's value; use --encoding raw for exact bytes

Example / output
wp etcd cat <key> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--encoding string Value encoding: auto|text|base64|hex; cat also supports raw (default "auto")
--revision int Read at this etcd revision (0: latest)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp etcd cat milvus-prod/woodpecker/version --etcd 127.0.0.1:32379 --encoding base64

Output and interpretation

Example / output
CAEQACAu
# Encoded value only; binary auto output also explains its encoding on stderr.

cat writes the value without an envelope. auto/text/base64/hex output ends with a newline; --encoding raw preserves exact bytes and adds none. Redirect raw output to a file for byte-exact inspection. Neither cat nor get changes the key.

Finding or error example
Example / output
Error: Usage: value is not UTF-8; use --encoding base64, hex or raw
# Only cat supports --encoding raw.
Complete command help
Example / output
Print only one key's value; use --encoding raw for exact bytes

Usage:
  wp etcd cat <key> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --encoding string                 Value encoding: auto|text|base64|hex; cat also supports raw (default "auto")
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for cat
      --revision int                    Read at this etcd revision (0: latest)
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp etcd get

Show one exact key, revisions, lease and encoded value

Example / output
wp etcd get <key> [flags]

Parameters to know

Parameter / type Use and default
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--encoding string Value encoding: auto|text|base64|hex; cat also supports raw (default "auto")
--revision int Read at this etcd revision (0: latest)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp etcd get milvus-prod/woodpecker/version --etcd 127.0.0.1:32379

Output and interpretation

Example / output
{"key":{"key":"milvus-prod/woodpecker/version","mod_revision":90,"lease":0,...},"snapshot_revision":105,"value_encoding":"base64","value":"CAEQACAu"}

get reads one exact key and returns key metadata plus an encoded value. auto keeps readable UTF-8 text and base64-encodes binary protobuf data; this command does not decode Woodpecker protobuf fields. Use --revision to read a retained historical snapshot and -o yaml for YAML.

Finding or error example
Example / output
Error: TargetNotFound: milvus-prod/woodpecker/missing-key
Complete command help
Example / output
Show one exact key, revisions, lease and encoded value

Usage:
  wp etcd get <key> [flags]

Flags:
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --encoding string                 Value encoding: auto|text|base64|hex; cat also supports raw (default "auto")
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for get
      --revision int                    Read at this etcd revision (0: latest)
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

wp etcd list

List raw keys by literal prefix; omit prefix to explore all keys

Example / output
wp etcd list [prefix] [flags]

Parameters to know

Parameter / type Use and default
--after-key string Exclusive page start; use the previous response's next_key
--after-key-base64 string Exclusive page start from next_key_base64; preserves binary keys
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--limit int Maximum keys per page (default 100)
--revision int Read at this etcd snapshot revision (0: latest)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

Etcd connection flags: see CLI setup.

Demo

Example / output
wp etcd list milvus-prod/ --etcd 127.0.0.1:32379 --limit 20 -o json

Output and interpretation

Example / output
{"prefix":"milvus-prod/","snapshot_revision":105,"count":1,"more":false,"keys":[{"key":"milvus-prod/woodpecker/version","mod_revision":90,...}]}

Omit the prefix to explore all keys; list / searches only keys beginning with /. The limit controls page size, not the total count. When more=true, use the returned next_key_base64 and the same snapshot_revision for the next page. The context meta_prefix is never prepended.

Finding or error example
Example / output
Error: Usage: --after-key and --after-key-base64 are mutually exclusive
# A compacted --revision instead produces a Network error.
Complete command help
Example / output
List raw keys by literal prefix; omit prefix to explore all keys

Usage:
  wp etcd list [prefix] [flags]

Flags:
      --after-key string                Exclusive page start; use the previous response's next_key
      --after-key-base64 string         Exclusive page start from next_key_base64; preserves binary keys
      --client-parameters-file string   Private temporary etcd parameters file; not cli.yaml or a candidate profile
      --etcd string                     Required client etcd endpoints (comma-separated); independent of server context
      --etcd-cacert string              Local etcd TLS CA cert file; explicit per-run setting
      --etcd-cert string                Local etcd TLS client cert file; explicit per-run setting
      --etcd-key string                 Local etcd TLS client key file; explicit per-run setting
      --etcd-password string            Etcd auth password; explicit per-run setting
      --etcd-tls-min-version string     Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
      --etcd-username string            Etcd auth username; explicit per-run setting
  -h, --help                            help for list
      --limit int                       Maximum keys per page (default 100)
      --revision int                    Read at this etcd snapshot revision (0: latest)
      --standalone                      Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery

CLI Handbook overview / Historical WAL diagnosis

Historical WAL diagnosis

Run Woodpecker-only atomic Metrics/Trace checks or summarize a fixed scenario over an incident window.

wp diag catalog

List implemented checks, formulas, references and scenario manifests; no source/model queries

Example / output
wp diag catalog [flags]

Parameters to know

Parameter / type Use and default
--check string Check name or stable ID
--source string metrics|trace

Demo

Example / output
wp diag catalog --source metrics --check writer-latency

Output and interpretation

Example / output
Lists the metric, formula, rule and required inputs without querying a source.

The CLI, rules and WebUI share the same canonical check keys.

Finding or error example
Example / output
An unknown key is rejected.
Complete command help
Example / output
List implemented checks, formulas, references and scenario manifests; no source/model queries

Usage:
  wp diag catalog [flags]

Flags:
      --check string    Check name or stable ID
  -h, --help            help for catalog
      --source string   metrics|trace

wp diag metrics check

Run one check; no implicit execution of other checks

Example / output
wp diag metrics check <name> [flags]

Parameters to know

Parameter / type Use and default
--threshold float Override this check threshold in native units (Trace durations use ms; error counts allow zero)

Demo

Example / output
wp --context local diag metrics check writer-latency --window 15m --details all -o json > writes.json

Output and interpretation

Example / output
A wp.diag/v1 report for the selected Writer timer.

This checks recorded WP client Writer operations. It does not infer application end-to-end latency.

Finding or error example
Example / output
Missing source configuration, failed queries and insufficient coverage remain explicit.
Complete command help
Example / output
Run one check; no implicit execution of other checks

Usage:
  wp diag metrics check <name> [flags]

Flags:
  -h, --help              help for check
      --threshold float   Override this check threshold in native units (Trace durations use ms; error counts allow zero)

wp diag scenario

Run related Metrics/Trace checks and summarize their unchanged results by WAL step

Example / output
wp diag scenario <name|all> [flags]

Parameters to know

No additional operation-specific flags. Connection settings come from CLI setup.

Demo

Example / output
wp --context local diag scenario slow-writes --window 15m --node-id woodpecker-0 --details all -o json > diagnosis.json

Output and interpretation

Example / output
Independent Metrics/Trace findings summarized by WAL step.

A scenario aggregates check results without a second root-cause engine or automatic cross-source association.

Finding or error example
Example / output
Configure at least one required source; checks without a source are skipped.
Complete command help
Example / output
Run related Metrics/Trace checks and summarize their unchanged results by WAL step

Usage:
  wp diag scenario <name|all> [flags]

Flags:
  -h, --help   help for scenario

wp diag trace check

Run one check; no implicit execution of other checks

Example / output
wp diag trace check <name> [flags]

Parameters to know

Parameter / type Use and default
--threshold float Override this check threshold in native units (Trace durations use ms; error counts allow zero)

Demo

Example / output
wp --context local diag trace check server-append-duration --window 15m --details all -o json > spans.json

Output and interpretation

Example / output
A source-local report for selected retained append spans.

Sampling, retention and emitted tags bound the evidence. No spans does not imply zero operation latency.

Finding or error example
Example / output
Missing spans or a query budget prevent a full assessment.
Complete command help
Example / output
Run one check; no implicit execution of other checks

Usage:
  wp diag trace check <name> [flags]

Flags:
  -h, --help              help for check
      --threshold float   Override this check threshold in native units (Trace durations use ms; error counts allow zero)

Troubleshooting

Choose one of three troubleshooting paths. Each starts with a short command plan, then shows detailed steps, result interpretation and recovery verification.

Choose the symptom

ScenarioStart whenMain path
Read StallsA reader/checkpoint is not advancing or the application waits for a WAL entry.Reader position → confirmed tail → replica probe → locate blocker → verify same-reader progress.
Write StallsAppends remain outstanding or durable confirmation stops.Submission → in-flight ops → writer queues → LAC → targeted recovery → append verification.
Data Corruption Diagnosis & RecoveryCRC/read errors, short replica coverage or replica coverage disagrees with the expected boundary.Metadata/data evidence → scan → probe/inspect → replica recovery, verified repair or accepted loss → application verification → checkpoint-backed retention.

Before running the steps

Complete CLI setup, including reachable admin origins for every quorum member. Each scenario includes a copyable connection block; replace its illustrative names and IDs with discovered values. Etcd parameters are used only by metadata-consuming commands.

Use the CLI built from the documented source checkout. Examples are illustrative excerpts; actual names, IDs and outcomes come from your deployment.

How to read a diagnostic result

ResultWhat it means
Idle / caught upNo new work may be available. Confirm workload and tail before declaring a stall.
Unknown / unreachable / cannot answerEvidence is incomplete. Restore transport, mappings or target location before treating it as missing data.
Short / checksum failure / all replicas failCorrelate with the finalized metadata boundary and identify the exact affected entry range.
Compacted / object_storageScan does not verify shared objects; probe can read them. Replies from three nodes may represent one shared copy.
Preview / acceptance not givenNo intervention was committed. meta patch/truncate previews normally exit 0; skip-range/fencing previews can exit 7 because acceptance was not given.
meta check: invalid / exit 9Metadata constraints failed; inspect issues. An independently verified metadata repair uses the guarded patch workflow, while physical damage needs its own recovery.
Command succeededVerify the application checkpoint/append result. An admin RPC succeeding is not the incident's closure.

Use the exit-code reference for scripts. For draining, configuration drift and disk inventory, use their CLI topic pages or the maintenance cookbook; these are separate from the three troubleshooting paths.

Read Stalls

Decide whether the reader is caught up, not reporting, unable to reach a replica, or blocked by unreadable data.

Plan: position → tail → actual reads → recovery

StepCommandIntent
1log readers, twiceIdentify the reader and determine whether its reported position moves.
2meta show/check; logstore lac; log scanCompare the reader with metadata constraints, confirmed progress and physical coverage.
3segment probeAsk all quorum members to actually read at the suspected position.
4segment inspect; ops list/showLocate damaged blocks/records or a long-running request; distinguish data failures from transport failures.
5Repeat the reader sample and application readVerify the same reader resumes and reaches the expected tail after the appropriate recovery.
Set the example connection and log before running the commands

Complete CLI setup first. Replace the example context, log name/ID, segment, node and storage space with values from your cluster. Check your CLI version and operation help before running these examples.

Example / output
export WP_CONTEXT=milvus-prod
export WP_LOG=my-channel
export WP_LOG_ID=42
export WP_SEG=7
export WP_NODE=milvus-woodpecker-0
export WP_ETCD=127.0.0.1:32379
export WP_META_PREFIX=milvus-prod/woodpecker
export WP_BUCKET=milvus-bucket
export WP_DATA_ROOT=milvus/wp

w() { wp --context "$WP_CONTEXT" "$@"; }
wm() { w "$@" --etcd "$WP_ETCD" --meta-prefix "$WP_META_PREFIX"; }

Use w for admin-only operations and wm for commands that consume coordinator metadata. Add the authentication/TLS parameters from CLI setup to wm when required. Node URLs/mappings come from the selected context. Output below is illustrative and abbreviated; adapt positions to your actual evidence.

Current source builds can also save etcd settings in the selected context. The explicit flags in wm override that context for each invocation, which is useful when switching between different metadata namespaces.

1. Observe the reader twice

Example / output
wm log readers "$WP_LOG"
sleep 35
wm log readers "$WP_LOG"

Readers normally report on a 30-second cadence. A single unchanged checkpoint is not a stall. For example, the application last returned 7:49 and is waiting for 7:50; the registry can report that pending position:

Example / output
READER       OPEN_POS  CURRENT  LAST_REPORT  OPEN_FOR  STATE
reader-123   7:0       7:50     3s           5m        live
# A later sample still reports CURRENT 7:50.
ResultInterpretation / next step
Position advancesReading is progressing. Compare application consumption/processing latency instead of treating this as a Woodpecker read stall.
Fresh report, unchanged positionThe reader is reporting, but could simply be caught up. Compare with the confirmed tail next.
stale reportReporting is not keeping up. Inspect client logs, blocked calls and connectivity; this alone does not prove corruption.
no read since openNo new position has been reported since opening; this is not proof of an unreadable first entry.
no live readersNo usable leased reader registration exists under this metadata namespace. Check context/prefix, client activity and leases. Milvus query traffic does not imply an open WAL reader.

CURRENT is the reader's reported position at a report tick, not a count of delivered records or an exact last-returned entry. Use application logs/checkpoints to identify the next entry actually required.

2. Compare the confirmed tail and metadata

Example / output
wm meta log show "$WP_LOG"
wm meta segment show "$WP_LOG" "$WP_SEG"
wm meta check "$WP_LOG"
wm logstore lac "$WP_LOG" "$WP_SEG"
wm log scan "$WP_LOG" --from-segment "$WP_SEG" --to-segment "$WP_SEG" \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
Example / output
# Illustrative scan summary:
SEGMENT  STATE      META LAST  REACHES  REPLICAS  VERDICT
7        Completed  99         0-99     3/3       ok

NODE_ID                 DURABLE_ENTRY  STATE
milvus-woodpecker-0      99             ok
milvus-woodpecker-1      99             ok
milvus-woodpecker-2      99             ok
quorum LAC: 99 (aq=2 of 3 nodes at or past it)

If the application needs 50 and the finalized boundary is 99, there should be data to read. If an Active reader is already at the current tail, waiting can be normal. Active metadata is not a final boundary. LAC establishes confirmed progress, not checksum integrity; for Completed/Sealed segments the CLI can use metadata when no live writer remains.

check returns state=ok/issues=[] for metadata constraints only; state=invalid/exit 9 names inconsistent fields, identity or quorum. If a software bug is independently proven, use the controlled metadata repair workflow. An Active lastEntryId=-1 is normal and must not be replaced by an observed LAC to make it look finalized.

3. Read the suspected position on every replica

Example / output
wm segment probe "$WP_LOG" "$WP_SEG" \
  --from-entry 50 --max-entries 100 \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"

A readable Completed segment returns the requested range through its final entry. These excerpts keep only the result columns:

Example / output
# Normal: all three nodes reach the expected final entry 99.
NODE_ID                 SOURCE      SERVED      OUTCOME      VERDICT
milvus-woodpecker-0      local_disk  50-99 (50)  end_of_file  served
milvus-woodpecker-1      local_disk  50-99 (50)  end_of_file  served
milvus-woodpecker-2      local_disk  50-99 (50)  end_of_file  served

# Abnormal: entry 50 cannot be read on any copy.
NODE_ID                 SOURCE      SERVED   OUTCOME  VERDICT
milvus-woodpecker-0      local_disk  nothing  error    cannot serve
milvus-woodpecker-1      local_disk  nothing  error    cannot serve
milvus-woodpecker-2      local_disk  nothing  error    cannot serve

SERVED=50-99 (50) means 50 returned entries, IDs 50 through 99 inclusive. nothing means no entry was returned from this requested start, not that the entire segment is empty. Read DETAIL for the actual error before deciding on recovery.

Probe resultMeaning / action
served, end_of_file at 99The requested finalized tail was readable. Investigate client handling, retries or processing above Woodpecker.
cap_reachedThe requested window ended; this is not EOF. Increase max-entries or move from-entry.
Active: entry_not_found / no local dataThe final boundary is unknown. It may be an unwritten/caught-up tail; pending writes are not established by a no-local-data response.
One or two replicas fail; another reaches 99Failover can cover the failed copies. Preserve the healthy copy and diagnose the unhealthy nodes.
All reachable replicas fail at required entry 50There is no usable local alternative at that position. Inspect the exact damage and healthy tail before any data-loss decision.
unreachable / not in memberlist / HTTP 404 / cannot answerCoverage is not established. Fix connectivity, mapping or storage-instance selection; do not interpret a failed request as confirmed data loss.

For Sealed segments, SOURCE=object_storage means the probe reads the shared compacted object. Three such replies are three reads of one shared copy, not three independent copies. See probe parameters and output fields.

4. Locate the blocker

Example / output
wm segment inspect "$WP_LOG" "$WP_SEG" --max-blocks 4096 \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
w ops list "$WP_NODE" --type logstore.get_batch_entries \
  --log "$WP_LOG_ID" --seg "$WP_SEG" --longer-than 30000
# Copy a real OP_ID from the list:
w ops show "$WP_NODE" --op-id REAL_OP_ID

Inspect walks beyond the first damaged block and can expose recoverable records after the bad entry. checksum_failed/record_checksum_failed differs from a bound or missing local data. Ops list is only the instantaneous in-flight registry; empty output on one node does not prove the client is idle. Check the other quorum nodes too.

5. Choose recovery and prove progress

EvidenceRecoveryVerify
Endpoint/mapping failureRestore the relevant connection or node availability.Probe now returns usable results; application reader progresses.
Healthy tail; no physical faultInspect/fix the client retry, reporting or application processing path.Same reader resumes; application checkpoint advances.
Confirmed unrecoverable range on every copyFollow the corruption workflow, including explicit acceptance of any skipped entries.Same reader passes the range and reaches the verified healthy tail; record the accepted loss.
Example / output
sleep 35
wm log readers "$WP_LOG"
wm segment probe "$WP_LOG" "$WP_SEG" --from-entry 51 --max-entries 100 \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"

The CLI probe confirms data readability; it does not automatically advance the application's reader. The closure is the same reader progressing plus application-level read success, not just a green command.

Write Stalls

Find where append progress stops: application submission, node operations, buffering/flush, or quorum confirmation.

Plan: workload → operations → writer queues → LAC → recovery

StepCommandIntent
1node list/healthz; application append evidenceConfirm the app is submitting and the relevant replicas are reachable.
2ops list/show/statsLocate long-running append or flush operations without mistaking idle for a stall.
3logstore segments/segment-show/buffer/flush-queueIdentify the loaded writer and its buffer or flush bottleneck.
4logstore lac, twiceDetermine whether the ack quorum's durable frontier advances.
5Targeted fix, or reviewed fence-quorumRestore the failing stage, then verify application append success and LAC progress.
Set the example connection and log before running the commands

Complete CLI setup first. Replace the example context, log name/ID, segment, node and storage space with values from your cluster. Check your CLI version and operation help before running these examples.

Example / output
export WP_CONTEXT=milvus-prod
export WP_LOG=my-channel
export WP_LOG_ID=42
export WP_SEG=7
export WP_NODE=milvus-woodpecker-0
export WP_ETCD=127.0.0.1:32379
export WP_META_PREFIX=milvus-prod/woodpecker
export WP_BUCKET=milvus-bucket
export WP_DATA_ROOT=milvus/wp

w() { wp --context "$WP_CONTEXT" "$@"; }
wm() { w "$@" --etcd "$WP_ETCD" --meta-prefix "$WP_META_PREFIX"; }

Use w for admin-only operations and wm for commands that consume coordinator metadata. Add the authentication/TLS parameters from CLI setup to wm when required. Node URLs/mappings come from the selected context. Output below is illustrative and abbreviated; adapt positions to your actual evidence.

Current source builds can also save etcd settings in the selected context. The explicit flags in wm override that context for each invocation, which is useful when switching between different metadata namespaces.

1. Confirm activity and reachability

Example / output
w node list --strict
w node healthz "$WP_NODE"

First inspect the application's append attempts, returned errors and deadlines. Healthz answers process health, not write progress. One slow replica can be absorbed by the ack quorum; a cluster health color does not identify which segment stopped.

ResultInterpretation
All nodes reachable; no application append attemptsStart with workload/submission. There is no server-side write stall to prove yet.
App submits but no relevant operation reaches any replicaCheck the client's queue, RPC path and discovery/configuration before the node.
Replica unreachableCheck the address mapping/port-forward, pod state and network; then measure whether the remaining quorum still confirms writes.

2. Find the long-running operation

Example / output
w ops list "$WP_NODE" --log "$WP_LOG_ID" --seg "$WP_SEG" \
  --longer-than 30000 -o json
w ops stats "$WP_NODE"
# Use an ID actually returned above:
w ops show "$WP_NODE" --op-id REAL_OP_ID
Example / output
# Illustrative operation excerpt:
{"op_type":"logstore.add_entry", "op_id":"append-op-123",
 "log_id":42, "segment_id":7, "started_at":"..."}

Repeat on all quorum members. Compare IDs and ages across samples: the same append persisting is different from many short-lived appends. file.flush points at the flush path. Empty output is an instant snapshot; registry evictions and app work that never reached a node can be invisible.

OutputInterpretation
ops list has no matching items; stats in_use=0No tracked operation is in flight at this instant. With no workload this is normal; with pending client appends, check submission and other nodes.
The same OP_ID remains older than 30000ms in consecutive samplesA tracked operation is long-running. Use show to inspect its type, context and stage.
ops show returns TargetNotFoundThe operation finished or was evicted before the query. Refresh list rather than treating it as a missing segment.
stats old eviction total growsOld operations are being dropped from this bounded registry. Investigate alongside queues and errors; the cumulative total alone does not prove a current stall.

3. Inspect the loaded writer

Example / output
w logstore segments "$WP_NODE" --log "$WP_LOG_ID"
w logstore segment-show "$WP_NODE" --log "$WP_LOG_ID" --seg "$WP_SEG"
w logstore buffer "$WP_NODE"
w logstore flush-queue "$WP_NODE"
Example / output
# Buffer / flush-queue excerpts under load:
LOG:SEG  BUFFER_BYTES  BUFFER_ENTRIES
42:7     1048576       100

LOG:SEG  QUEUE_DEPTH  QUEUE_CAP
42:7     8            16

These are current occupancies, not latency measurements. Re-run while appends are outstanding: buffers draining and queue depth returning toward zero indicate progress; growing or persistently full queues need correlation with flush operations and backend errors. A single nonzero sample can be normal batching.

PatternMeaning / next check
Buffer/queue increases; flush operation does not finishInspect disk/object-storage errors, I/O latency and flush scheduling; correlate operation details.
No arriving appends and little queued workLikely upstream inactivity or a client/RPC issue; not evidence of a blocked flush.
segments shows only headers / segment-show is absentThese list loaded writer objects, not all persisted segments. Confirm the current segment and owner; do not infer no data.
One node behind; others advanceMeasure quorum confirmation. A lagging replica need not stall the application.

Read persisted metadata when the writer registry is insufficient:

Example / output
wm meta segment list "$WP_LOG"
wm meta segment show "$WP_LOG" "$WP_SEG"
wm meta check "$WP_LOG"
wm log scan "$WP_LOG" --from-segment "$WP_SEG" --to-segment "$WP_SEG" \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"

meta list/show include unloaded history. check: state=ok validates metadata only; state=invalid/exit 9 identifies a constraint failure. For a proven metadata bug, use the offline repair workflow and restart affected clients afterward. Editing Active lastEntryId/state is not a substitute for writer recovery, and truncate is a retention operation rather than a fix for pending appends.

4. Compare LAC twice

Example / output
wm logstore lac "$WP_LOG" "$WP_SEG"
sleep 10
wm logstore lac "$WP_LOG" "$WP_SEG"
Example / output
NODE_ID                DURABLE_ENTRY  STATE
milvus-woodpecker-0     99             ok
milvus-woodpecker-1     99             ok
milvus-woodpecker-2     70             ok
quorum LAC: 99 (aq=2 of 3 nodes at or past it)
ResultInterpretation
LAC advancesDurability confirmation progresses. Investigate client completion/latency rather than assuming every append is blocked.
LAC unchanged while appends remain outstandingConfirmation is not advancing. Combine queues, errors and quorum-member evidence to locate the stage.
Fewer than AQ usable repliesLAC is unknown; restore reachability or replica availability. Unknown is not a confirmed smaller frontier.
no writerNode answered but has no loaded writer for this segment. Completed/Sealed metadata may still establish a boundary; check whether the application has rolled to a newer segment.

The CLI's --timeout bounds its own admin/metadata requests, not the application's append deadline. A 2s send budget does not guarantee an end-to-end append within 2s: confirmation, retries and rolling add time.

5. Restore the stage, then re-test append

Fix verified connectivity, storage or deployment failures first. A manual flush applies to a loaded writer and does not repair corrupted data:

Example / output
w logstore force-flush "$WP_NODE" --log "$WP_LOG_ID" --seg "$WP_SEG"

If the intended recovery is to fence the old writer and let the application roll/reopen through its normal recovery path, review quorum impact first:

Example / output
# Preview: no -y means no commit and exit code 7.
wm logstore fence-quorum "$WP_LOG" "$WP_SEG" \
  --reason "Reviewed recovery of stalled writer"
# After reviewing the selected nodes and impact:
wm logstore fence-quorum "$WP_LOG" "$WP_SEG" \
  --reason "Reviewed recovery of stalled writer" -y

Fencing can interrupt writes. The default WQ=3/AQ=2 needs two selected members to cross the interruption gate; fencing only one does not guarantee an interrupted quorum. StateConflict/insufficient reachable members means no verified quorum intervention, not success. Check fence-quorum for selection details.

6. Verify at the application and quorum

Example / output
w ops list "$WP_NODE" --log "$WP_LOG_ID" --longer-than 30000
wm log scan "$WP_LOG" --mode quick \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
# Select the actual current segment if rolling changed it:
wm logstore lac "$WP_LOG" "$WP_SEG"

Require new application appends to succeed, the durable frontier to advance under workload, and old operations/queues to clear. A fence RPC or healthy pod alone is not the recovery result. If physical reads fail or CRC errors appear, switch to the corruption workflow.

Data Corruption Diagnosis & Recovery

Establish the metadata boundary, identify readable copies and exact bad entries, then choose recovery without discarding healthy data.

Plan: preserve → compare → locate → choose → verify

StepCommandIntent
1meta export/show/check; deployment data backupPreserve decoded/raw metadata and original payloads; distinguish inconsistent metadata from physical damage.
2log scan quick/rawIdentify suspicious segments and distinguish failed inspection from confirmed missing coverage.
3segment probe/inspectFind which copies read, where sequential reading stops and which later records remain recoverable.
4Replica recovery, verified skip-range, or meta segment patchChoose based on healthy copies, accepted loss or an independently proven metadata bug.
5Application verification; log truncate; marking if retiring dataVerify recovery/checkpoints, then separately observe auditor/retention closure.
Set the example connection and log before running the commands

Complete CLI setup first. Replace the example context, log name/ID, segment, node and storage space with values from your cluster. Check your CLI version and operation help before running these examples.

Example / output
export WP_CONTEXT=milvus-prod
export WP_LOG=my-channel
export WP_LOG_ID=42
export WP_SEG=7
export WP_NODE=milvus-woodpecker-0
export WP_ETCD=127.0.0.1:32379
export WP_META_PREFIX=milvus-prod/woodpecker
export WP_BUCKET=milvus-bucket
export WP_DATA_ROOT=milvus/wp

w() { wp --context "$WP_CONTEXT" "$@"; }
wm() { w "$@" --etcd "$WP_ETCD" --meta-prefix "$WP_META_PREFIX"; }

Use w for admin-only operations and wm for commands that consume coordinator metadata. Add the authentication/TLS parameters from CLI setup to wm when required. Node URLs/mappings come from the selected context. Output below is illustrative and abbreviated; adapt positions to your actual evidence.

Current source builds can also save etcd settings in the selected context. The explicit flags in wm override that context for each invocation, which is useful when switching between different metadata namespaces.

1. Preserve evidence

Record the application checkpoint, log name/ID, segment ID, deployment version and replica locations. Preserve original replica files and relevant objects through your deployment's backup procedure before changing them. Capture the scan, probe and inspect results below; they are diagnostics, not a data backup.

Example / output
wm meta export --output-file ./metadata-before-corruption.json
wm meta log show "$WP_LOG"
wm meta segment show "$WP_LOG" "$WP_SEG"
wm meta check "$WP_LOG"

Export creates a new file containing readable known metadata and byte-exact original values, revisions and leases. It does not back up replica files/objects. check exits 0 with state=ok or 9 with state=invalid and issues; either result is metadata-only. A valid Completed boundary of 99 expects entries 0..99, but check cannot establish that those payloads exist. If independently verified data/history proves the boundary is wrong, follow metadata repair.

2. Find the affected segment

Example / output
wm log scan "$WP_LOG" --mode quick \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
wm log scan "$WP_LOG" --mode raw --from-segment "$WP_SEG" --to-segment "$WP_SEG" \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
Example / output
# Example normal summary (DETAIL column omitted):
SEGMENT  STATE      META LAST  REACHES  REPLICAS  VERDICT
7        Completed  99         0-99     3/3       ok
8        Active     -1 (open)  -        3/3       open

# Example missing finalized tail, after a complete local survey:
SEGMENT  STATE      META LAST  REACHES  REPLICAS  VERDICT
7        Completed  99         0-49     3/3       short

# Example failed inspection requests:
SEGMENT  STATE      META LAST  REACHES  REPLICAS  VERDICT
7        Completed  99         -        0/3       unknown

The short example establishes a local coverage gap at 50..99; it does not explain the cause. Inspect/probe next. The unknown example establishes no coverage at all; inspect the per-replica DETAIL (for example HTTP 404, timeout or missing mapping) before deciding there is missing data.

VerdictWhat was established
okObserved local coverage matches the metadata expectation within this survey.
shortEvery relevant replica survey completed, and some expected range is absent from their combined local coverage. Any incomplete/unreachable alternative makes that missing range unknown.
unknownRequest, location, response or survey bound prevented a conclusion. Read the per-replica HTTP/error details.
open / local_emptyActive boundary remains unknown; replicas found no local durable entries. This is not automatically corruption.
compactedLocal copies are not authoritative. Scan has not verified shared object-storage data; use probe for actual reads.
reclaimedTruncated data is intentionally eligible for cleanup; absence is not automatically loss.

REACHES is the union of observed local ranges; '-' is no observed range. REPLICAS counts usable replies, not independent healthy copies. quick checks structure; raw verifies local block checksums. Neither establishes pending writes or end-to-end object readability.

3. Separate two damaged copies from all three

Example / output
wm segment probe "$WP_LOG" "$WP_SEG" --from-entry 0 --max-entries 200 \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
wm segment inspect "$WP_LOG" "$WP_SEG" --max-blocks 4096 \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
Example / output
# Two failed copies and one healthy copy, abbreviated:
NODE                    SERVED       OUTCOME       VERDICT
milvus-woodpecker-0      0-49 (50)    error         cannot serve
milvus-woodpecker-1      0-49 (50)    error         cannot serve
milvus-woodpecker-2      0-99 (100)   end_of_file   served

The healthy copy can support reader failover. Preserve it; do not declare readable entries lost. The CLI does not offer a generic copy/repair command. Recover damaged replicas through the supported deployment/recovery path, then verify their coverage. Compaction can still fail if it requires damaged copies.

Example / output
# If all three instead stop before entry 50:
No replica can serve entry 50 ...
# Inspect may separately identify:
STATUS: checksum_failed
record stop: record_checksum_failed at entry 50
recoverable entries after the bad record: 51..99

Probe stops where sequential reading fails; inspect surveys beyond a bad block and can expose the tail. Confirm exact entry boundaries and each copy's result. A timeout, ambiguous path, 404 or cap_reached is not proof that an entry is unrecoverable. Object_storage replies inspect one shared copy, not three independent ones.

Example / output
# Inspect excerpt for one node, assuming entry 50 occupies its own block:
BLOCK  ENTRIES  READABLE  RECOVERABLE  STATUS
0      0-49     all       -            ok
1      50-50    none      -            checksum_failed
2      51-99    all       -            ok

READABLE=none means normal reads reject that block. RECOVERABLE describes individually verified records inside a rejected block; it is evidence for repair, not a promise that the SDK can return them. Here the later block is independently valid. If entry 50 instead shares a damaged block with other entries, verify that whole block's consequences rather than assuming only 50 is affected.

4A. Accept a confirmed unreadable range

Use this branch only when every usable copy fails and the business accepts loss of the specific entries. The example assumes only entry 50 is confirmed unrecoverable, with 51..99 verified readable:

Example / output
# Preview and check the evidence/gate; no -y means no commit (exit 7).
wm log skip-range add "$WP_LOG" "$WP_SEG" \
  --from-entry 50 --to-entry 50 --reason "All copies fail at 50; verified healthy tail"
# After explicit acceptance of losing entry 50:
wm log skip-range add "$WP_LOG" "$WP_SEG" \
  --from-entry 50 --to-entry 50 --reason "Accepted loss of entry 50" -y
wm log skip-range list "$WP_LOG"
Example / output
# Successful add for log ID 42, segment 7:
Declared: log 42 segment 7 now skips 50-50
ResultMeaning
Gate refuses because a copy can servePreserve/read that copy; do not force past healthy data just to make a command pass.
Acceptance not givenExpected preview behavior, no declaration committed.
Declaration committedReaders can skip the inclusive range. Files remain damaged; compaction is not repaired by the declaration.

Keep the same stalled reader running. Wait for refresh/report timing and verify it crosses 50 and reaches 99. For a reader returning original entries 0..99, skipping just 50 means 99 returned records; record IDs and application checkpoints establish what was lost. Removing the declaration does not restore data.

4B. Correct a proven metadata bug

Use this branch only when independent file validation, acknowledged-write history and the responsible software fix establish the correct metadata. A replica losing acknowledged data does not justify lowering the expected boundary. Example: metadata incorrectly says lastEntryId=999, but verified history proves this finalized segment ended at 99.

Example / output
cat > ./segment-repair.json <<'JSON'
{"lastEntryId":"99"}
JSON
wm meta segment patch "$WP_LOG" "$WP_SEG" --patch-file ./segment-repair.json
Example / output
# Illustrative preview excerpt:
{"applied":false,"before":{"revision":174},
 "changes":{"lastEntryId":{"before":"999","after":"99"}}}

Review before/after/changes, then stop relevant writers/auditors and keep them stopped. --offline only acknowledges this prerequisite. Use the target segment's before.revision (174 in this example), not the log or snapshot revision:

Example / output
wm meta segment patch "$WP_LOG" "$WP_SEG" --patch-file ./segment-repair.json \
  --apply --offline --expected-revision 174 \
  --reason 'Correct independently verified tail after software fix' \
  --backup ./segment-before-repair.json
wm meta segment show "$WP_LOG" "$WP_SEG"
wm meta check "$WP_LOG"
ResultInterpretation / action
applied=true, new_revision, audit_key, backupThe metadata write committed. Re-read the record, restart affected clients to refresh caches, and verify physical reads/application progress.
Key/segNo mismatch; tail below truncation frontier; invalid lifecycleThe correction is rejected. Resolve the identity or invariant with verified evidence; do not rewrite state simply to make check pass.
warnings: embedded quorum (unchanged)An independent scalar repair may proceed, preserving that bad quorum. check still reports the unresolved quorum; this is not a clean recovery result.
Writer lock / stale revision / concurrent updateNo correction was committed by this attempt. Stop the metadata writer or refresh the preview; use a new backup filename for retry.
Commit network errorThe outcome may be unknown. Inspect current metadata and audit records before retrying.

State and timestamp correction uses the same flow; follow the metadata field rules and examples. A state edit cannot create compacted files or repair replicas. Log truncation fields are protected from generic patching.

5. Close the compaction/retention loop

Reader recovery and compaction are separate. Skip-range does not fix a damaged Completed copy. Later eligible segments can be compacted independently; the damaged segment may continue failing until repaired or deliberately retired.

After verifying resumed application progress and a durable checkpoint beyond the damaged segment, advance normal retention with log truncate. The example assumes segment 8 is non-empty Completed/Sealed and its entry 0 is a verified safe boundary. Advancing into 8 permits retiring segment 7; truncating to 7:99 retains segment 7 itself.

Example / output
wm meta segment show "$WP_LOG" 8
wm log truncate "$WP_LOG" --segment 8 --entry 0

The preview has applied=false, unchanged=false, before/after and target_segment_revision. Use the log before.revision for apply (17099 in this illustrative example), not the target segment revision. No --offline flag is needed for this normal retention operation; coordinate it with the application's checkpoint policy.

Example / output
wm log truncate "$WP_LOG" --segment 8 --entry 0 \
  --apply --expected-revision 17099 \
  --reason 'Verified durable application checkpoint beyond damaged segment 7' \
  --backup ./log-before-truncate.json
wm meta log show "$WP_LOG"

applied=true/new_revision confirms the frontier update and audit record, not immediate deletion. An unchanged frontier returns unchanged=true/applied=false with no new artifacts. Active/empty targets, backward movement, bad identity, missing apply guards and stale/concurrent revisions are rejected; inspect the message and preview again rather than forcing a change.

A live application log handle/auditor marks earlier finalized segments Truncated in subsequent rounds, then applies retention and reader protection. No active auditor means the background transition will not occur. Physical deletion is asynchronous; a metadata backup cannot restore deleted files. Use scan and marking to observe the result:

Example / output
wm log scan "$WP_LOG" --mode quick \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
wm marking list
Compacted-mark notification is pending manual handling

Inspect marking records and verify the unacked node's disposition before using wm marking confirm "$WP_LOG_ID" "$WP_SEG". It settles notification retries, not data corruption or segment metadata, and the record remains until truncation reaps it.

6. Verify the chosen recovery

Example / output
sleep 35
wm log readers "$WP_LOG"
# For read/metadata/replica recovery, while this segment remains retained:
wm segment probe "$WP_LOG" "$WP_SEG" --from-entry 51 --max-entries 100 \
  --bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"

For skip-range recovery, require the same reader and application checkpoint to progress through the verified tail. For metadata repair, restart affected clients and verify the resumed reader/application checkpoint plus physical reads. For replica repair, re-probe/inspect every repaired copy. For retirement, expect Truncated/reclaimed rather than readable data and verify the auditor no longer attempts compaction on the retired segment; do not require reading data intentionally reclaimed. In a controlled lab, restore original files before removing skip declarations and verifying the original full record count.

Milvus–Woodpecker Stack Examples

Install, observe and operate Milvus with Woodpecker through a complete, open-source example stack.

Why this stack exists

A running Milvus + Woodpecker deployment is easier to operate when application behavior, WAL health, resource pressure and alerts can be examined together. Users who have not yet deployed monitoring or operational tools can start here with a working example, then adapt its configuration and investigation methods to their own instances.

The default deployment contains Milvus and independent Woodpecker services with persistent dependencies. Add Prometheus, Grafana, Alertmanager, Jaeger and Loki/Alloy independently, and use wpcli for operations. A real Milvus workload exercises its WP client and the WP servers, so the example explains what to observe along the write path and how to investigate it.

Exercise the write pathMilvus workload → Milvus + WP client → WP services → object storage
Observe and investigatePrometheus → Grafana + Alertmanager · wpcli → WP services and client metadata

What you can do today

The runnable example lives in deployments/milvus-full-stack-helm/. It uses the official Milvus Helm chart, a small support chart and pinned monitoring components. Values files, dashboard JSON, alert rules, workload scripts and the notebook are included in the repository.

The monitoring and alerting guide connects these E2E views with the Milvus/WP reference dashboards, scrape verification and Alertmanager.

One operational feedback loop

  1. Install a known environment. Record chart/image versions, namespace, resource settings and connection targets.
  2. Run a real workload. Verify Milvus inserts, primary-key queries and vector search, then establish a small write-performance baseline.
  3. Check the whole chain. Compare application results with Milvus/WP metrics, dashboards, alerts, wpcli state and logs for the same time window.
  4. Improve and repeat. Investigate failures or missing signals, change one setting or instrument, and rerun the same scenario.

Commands return exit status and save structured artifacts so humans and local AI agents can follow the same loop. The current stack provides these building blocks; autonomous investigation and optimization are future extensions.

Apply the examples to your own instances

Start with a local stack to learn the expected signals and command output. For an existing Milvus + WP instance, adapt scrape selectors, dashboard variables, alert thresholds and notification routes to its topology and workload. Use that instance’s reachable WP Admin URLs and its own etcd namespace and storage paths when configuring wpcli.

Distinguish the shared WP service cluster from the metadata managed by a particular Milvus instance’s WP client. The two connection scopes are explicit in the generated configuration and operations examples.

The supplied resource sizes, single-node dependencies and example credentials are for a development environment. Use your deployment’s availability, access and storage requirements when adopting the configurations. Performance results from this local setup describe that setup.

A growing examples collection

The Helm stack, monitoring/alert checks, wpcli connection workflow and bounded write experiment are available now. Future chapters can add controlled failure and recovery drills, E2E operational diagnosis, capacity and configuration comparisons, and measured optimization examples. Each chapter should include its trigger, workload, expected signals, investigation, recovery, verification and evidence.

An Operator installation path is also planned separately. This collection will grow around concrete scenarios that help users improve the operability and observability of their own Milvus + WP instances.

Helm Stack Walkthrough

Install and operate Milvus + Woodpecker on your existing Kubernetes cluster using reviewable Helm and kubectl commands.

Prepare an existing Kubernetes context

Use configured kubectl, Helm 3 and Python 3.9+. Kubernetes, its nodes, credentials, StorageClass and registry access are managed outside this tool. This applies equally to a shared cluster, kind or local minikube. Docker is optional for offline rule tests; Jupyter needs Python 3.11+.

Commands
git clone https://github.com/zilliztech/woodpecker.git
cd woodpecker/deployments/milvus-full-stack-helm
python3 -m pip install -r scripts/requirements-connections.txt
kubectl config current-context
kubectl version
./stack.sh help

The default context is kubectl’s current context; override it with --context NAME or CONTEXT. The default namespace is milvus-stack, configurable with --namespace / NAMESPACE. An unavailable context produces configuration guidance; stack.sh never creates, starts, stops or deletes Kubernetes.

If configuration is incomplete, stack.sh distinguishes missing/unreadable kubeconfig files, an empty context list, an unselected/unknown context and API connection failures. It lists available context names and copyable commands, including ./stack.sh status --context YOUR_CONTEXT. No context is switched automatically, and credentials are not printed. See configuration diagnostics.

Review, then install

Commands
./stack.sh render
./stack.sh images                       # Image inventory only
./stack.sh install                       # Preview native commands
./stack.sh install -x                    # Execute the reviewed resource operations
./stack.sh status

Resource changes, workload writes and port forwards are previewed by default; -x executes them. Planning may read the cluster and save local values/manifests; it does not change resources. Read-only status and local rendering remain available without -x. Printed Helm/kubectl commands include the explicit target and can be copied directly.

A first deployment contains only Milvus, three independent WP services and persistent etcd/MinIO. It wraps the official Milvus Helm chart and needs no monitoring CRDs. Public defaults use Milvus v3.0.3 and WP v0.1.46. Add --local-arm64 to render/images/install for the supplied private ARM64 build; prepare its image access outside the stack.

Add optional components later

Commands
./stack.sh addons-install --with prometheus,grafana   # Preview
./stack.sh addons-install --with prometheus,grafana -x
./stack.sh addons-install --with alertmanager -x
./stack.sh addons-install --with jaeger,loki -x
./stack.sh addons-status

Use --with all for the complete observability example. Loki includes Alloy for log collection. Addon changes preserve live core overrides and previously selected components; changing Jaeger selection updates trace configuration and rolls Milvus/WP. See component dependencies and lifecycle.

Discover and open endpoints

Commands
./stack.sh forward --wpcli               # Preview native port-forward commands
./stack.sh forward --wpcli -x            # Keep running in a separate terminal
./stack.sh connections
./stack.sh connections --json

Only installed services are forwarded. --wpcli also includes per-node WP Admin and tenant etcd. Matching existing forwards are reused, and Ctrl+C closes only those started by this invocation. Only addresses marked forwarded are verified local connections; Admin HTTP forwarding does not establish SDK advertised-gRPC reachability.

Run the first verification

Commands
./stack.sh smoke -x
# Allow two 15-second scrapes when Prometheus is installed.
./stack.sh verify
./stack.sh alerts

Smoke writes real data and checks query/search. Verify requires Prometheus, with Grafana provisioning checks when selected. The alert flow uses Prometheus + Alertmanager. Core-only perf still checks writes/reads and explicitly marks absent metrics/alerts as skipped. A running Pod alone does not establish a healthy chain.

Follow the Jupyter guide

Commands
./stack.sh notebook

Open the authenticated loopback URL and select notebooks/full-stack-walkthrough.ipynb. It selects the complete observability profile. EXECUTE=False previews by default; set EXECUTE=True after reviewing the target, images, ports and load. The guide never manages Kubernetes and keeps stop/uninstall cells disabled by default.

Pause/resume workloads or remove the deployment

CommandEffect with -xData boundary
installInstall/upgrade Helm releases on the existing cluster and run the selected stack.Retain PVCs and existing addons.
stopRecord replica state and pause this stack’s workloads using native scale/patch/wait commands.Retain Helm releases/PVCs; memory/ephemeral history may disappear.
startRestore the replica counts saved by stop and wait for rollouts.Does not install a missing stack or start Kubernetes.
uninstallUninstall stack releases and delete their PVC data.Retain Kubernetes, namespace, shared CRDs and local evidence.
addons-install / addons-uninstallAdd components / remove selected components and their data.Retain the core deployment.
Commands
./stack.sh stop                  # Review only
./stack.sh stop -x               # Pause workloads; leave Kubernetes running
./stack.sh start -x              # Restore saved workload replicas
./stack.sh forward --wpcli -x
./stack.sh uninstall                 # Review the deletion plan
# ./stack.sh uninstall -x            # Executes deletion of releases AND PVC data

Stop state is stored in the namespace ConfigMap milvus-full-stack-lifecycle, so resume does not depend on one machine. Prometheus/Alertmanager are scaled through their Operator resources, and Alloy’s DaemonSet uses a reversible scheduling selector. Resume before changing Helm releases. Unsafe scale-down PVC retention policies and active autoscalers are reported before any write.

Continue with metrics, logs, traces and alerts or wpcli operations. See the separate local Kubernetes preparation example if you choose minikube.

Metrics, Logs, Traces & Alerts

Follow the same workload from Milvus through the WP client and servers, then check that the operational signals reach your dashboards and Alertmanager.

Select the signals you need

Commands
./stack.sh addons-install --with prometheus,grafana,alertmanager -x
./stack.sh addons-install --with jaeger,loki -x
./stack.sh forward --wpcli -x

Each component is optional and can be added after the core deployment. Grafana displays data from the selected backends; it does not store their metrics, logs or spans. Existing forwards need a new invocation to include newly installed services. See independent selection and removal.

Start with an E2E operational dashboard

For a specific symptom, use the E2E dashboards: Write / Append, WAL Read / Consume, Flush / Checkpoint, or Search / Query. They group the metrics by processing step and include investigation guidance. Use the three component reference dashboards below for deeper inspection.

Open the three reference dashboards

Keep ./stack.sh forward --wpcli -x running. These are the default local addresses; use ./stack.sh connections for discovered addresses when ports differ.

ViewLocal addressWhat to examine
Milvushttp://127.0.0.1:3000/d/milvus-full-stackApplication throughput, latency and component/resource health.
WP clienthttp://127.0.0.1:3000/d/woodpecker-client-k8sClient append/read behavior, latency and queues.
WP serverhttp://127.0.0.1:3000/d/woodpecker-server-k8sNode operations, flush/storage behavior and resource pressure.
Prometheushttp://127.0.0.1:9099Scrape target health, raw series, queries and pending/firing alerts.
Alertmanagerhttp://127.0.0.1:9093Received alerts, grouping, inhibition and silences.
Loki API / Grafana Explorehttp://127.0.0.1:3100 / Grafana Loki datasourceSearch collected container logs by namespace, Pod, container and text.
Jaegerhttp://127.0.0.1:16686Individual request spans and cross-service trace coverage.

Grafana’s local example login is admin / admin. Dashboards and the Prometheus data source are provisioned from repository files. Select namespace milvus-stack and a time range containing the test; allow scrapes to collect the new workload.

Check the monitoring chain

Commands
./stack.sh smoke -x
# Wait at least 30 seconds for scrapes.
./stack.sh verify

The verification report is saved to .local/verification.json. It checks target health, positive application/WP workload metrics, dashboard provisioning and query compatibility. Some panels need a particular operation or enough rate-window history before showing finite samples. An empty panel is an investigation prompt, not proof that the corresponding operation is healthy or broken.

If application writes succeed but WP activity is missing, check the deployed Milvus configuration, metric families and label selection before drawing a conclusion about the write path. The dashboard source record documents the Milvus production export and adaptations of the WP Kubernetes templates.

Inspect individual request traces

When selected, the stack deploys jaegertracing/all-in-one:1.53.0 and exports Milvus and WP spans over OTLP. Open Jaeger directly, or select the provisioned Jaeger datasource in Grafana Explore.

Commands
./stack.sh traces
./stack.sh traces-smoke -x
# Strict E2E coverage: fails on disconnected request paths.
TRACES_REQUIRE_E2E=1 ./stack.sh traces-smoke -x

The smoke probe writes 100 vectors and verifies Strong query/search, then saves exact request trace IDs, client latency and raw spans. Normal perf records also contain sampled trace links. The supplied image currently connects query/search through SN and segcore, while its user Insert trace does not continue into SN/WP. Separate WP exporter activity is not proof of an E2E write trace; the report explicitly records this gap as partial.

The historical full-stack tracing receipt records tested custom-image relationships and remaining limits. The original published Milvus-image acceptance gate is retired under the WP-only scope decision. WP instrumentation fixes remain useful; the tool does not diagnose the surrounding Milvus engine or convert asynchronous links into serial latency.

Full sampling favors small diagnostic runs and changes system overhead. Lower both server and workload sampling for sustained tests. Jaeger history is bounded and in memory, so save evidence before eviction or restart. See tracing setup, interpretation and known gaps.

Search real Pod logs

Loki stores compressed log chunks and label indexes on a 4 GiB PVC, with a 24-hour retention policy. Alloy tails this namespace’s container logs through the Kubernetes API. In Grafana Explore, select Loki and query {namespace="milvus-stack"}, then filter by Pod or text. When Jaeger is also selected, recognized trace IDs link to its trace view.

Commands
./stack.sh logs
./stack.sh logs-smoke -x

The smoke test runs a small Milvus write/query/search workload, finds its unique collection marker in the actual Proxy log and requires that marker to arrive through Alloy into Loki. It saves .local/logging-smoke.json. Collection is bounded by source-log retention and collector availability; see storage, queries and limitations.

Inspect and test alerting

Prometheus evaluates the 31 reference WP rules covering resource pressure, failures, scaling signals and client behavior. Firing alerts are sent to Alertmanager for grouping and inhibition. The default local-only receiver lets you inspect alerts in the Prometheus and Alertmanager UIs without sending email or other external notifications.

Commands
./stack.sh alerts-test   # Offline rule behavior checks; uses Docker.
./stack.sh alerts        # Live rule/receiver/AM discovery checks.
./stack.sh alerts-smoke  -x # Temporary synthetic firing/recovery/inhibition checks.

The synthetic check verifies the alerting path and removes its temporary resources. It does not create each underlying disk, object-store or quorum failure, and it does not prove that every production threshold is suitable for your workload. See the rule catalog, investigation guidance and validation details.

When external notification is needed, use the optional SMTP configuration example with your relay and credentials. Optional Mailpit testing captures email locally. Neither profile is enabled by default.

Adopt the signals in an existing deployment

  1. Match Prometheus discovery selectors and labels to your Milvus and WP Pods. Confirm the exporters expose the metric families used by each query.
  2. Adapt Grafana data-source references, namespace/instance variables and dashboard time windows.
  3. Set alert thresholds and hold periods from your workload, capacity and recovery expectations; configure your Alertmanager receiver.
  4. Exercise a known workload and a controlled alert-path test, then preserve evidence showing which checks passed and which signals remain untested.

Use wpcli to inspect the specific node, log or metadata behind a symptom, and the experiment workflow to compare behavior before and after an improvement.

E2E Operational Dashboards

Choose the symptom, then follow numbered processing steps from top to bottom. Each step explains what to inspect and where to continue.

Install the optional backends first: ./stack.sh addons-install --with prometheus,grafana -x, then keep ./stack.sh forward -x running.

Choose a chain

The Helm stack provisions four focused E2E dashboards alongside the three Milvus/WP reference dashboards. Keep the stack’s forwards running; Grafana defaults to http://127.0.0.1:3000. Use ./stack.sh connections for actual ports.

SymptomDashboard / Grafana pathOrdered steps
Slow or failed writesWrite / Append
/d/milvus-e2e-write
Proxy → Streaming Client → SN WAL → WP Client → WP services/disk → admission pressure
Consumption lag or delayed visibilityWAL Read / Consume
/d/milvus-e2e-consume
Read source/mode → historical WP reader → SN scanner → subscription → processing/visibility
Persistence/checkpoints lagFlush / Checkpoint
/d/milvus-e2e-flush
Growing data → sync scheduling → persistence → progress; separate WP background maintenance
Slow search/query requestsSearch / Query
/d/milvus-e2e-query
Proxy → Delegator/TSafe → scheduling → segment execution → result return

Every group includes visible investigation guidance, followed by throughput/status, latency or progress metrics for that step. P50/P95/P99 curves preserve actual Pod/component or node/log identities. The full guide describes each group, its metric scope, operational limits and maintenance.

Follow a slow write

Begin with Proxy completion status, mutation latency, DML queue and timestamp/primary-key preparation. Continue to producer availability and produce latency, then compare full SN WAL append with backend append and individual interceptor timers.

At the WP stages, compare pending appends, writer/replica outcomes and individual server append, buffer-wait and local-flush latency. The final group shows disk admission, usage, scheduler waiting and confirmation progress. Use wpcli and the component reference dashboards to investigate the identified node/log.

Follow delayed consumption

First distinguish real-time memory tailing from historical WAL catchup. A tailing scanner can read the SN write-ahead buffer without WP read RPCs, so zero backend read traffic alone is expected. Historical reads can use WP services or compacted segments in object storage.

Continue through reader outcomes, scanned/passed messages, ordering/transaction buffers, subscription availability and downstream processing. In the supplied Milvus build, querynode-named consumption metrics can be emitted by the SN delegator, and local subscribers bypass the remote handler. Legends retain the actual emitting process.

Normal tail EOF is displayed as an outcome. A blocking read_next can wait for new data, so its duration is not pure storage latency. The WP v0.1.46 example does not export per-reader read-frontier metrics; use wpcli for positions instead of assuming an exact backlog count.

Separate persistence and query symptoms

Use Flush / Checkpoint when consuming works but growing buffers or persistent progress lag. It separates dispatcher waiting, upload/callback execution and progress boundaries. A flusher’s “seen” TimeTick is not a durable checkpoint; idle snapshot/checkpoint age is not automatically a fault. WP compaction/offload is shown as a parallel background path.

Use Search / Query for request latency. TSafe waiting points back to consumption/visibility; scheduler waiting, segcore/segment execution, loading and result reduction describe other stages. Proxy result wait includes downstream work and transport, rather than only network latency.

Validate coverage and compare a run

Commands
./stack.sh smoke -x
# Allow two 15-second scrapes.
./stack.sh verify
./stack.sh perf -x

Verification checks seven provisioned dashboards, their queries and the required metric families in each E2E stage. The report lists optional idle/version-specific coverage gaps separately; an UP panel cannot hide a missing WAL-stage metric. Performance reports and the Jupyter guide include E2E links for the run’s time window.

Channel and WP log filters are separate identities, resolved with metadata before isolating a log. No data is not a healthy zero, histogram overflow limits quantile resolution, and overlapping timers cannot be added or subtracted. These dashboards correlate operational evidence; live read verification and controlled fault/recovery tests establish different aspects of correctness.

wpcli Operations in the Stack

Connect to the WP service cluster, attach the selected Milvus instance’s WP client metadata, and use both views during an investigation.

Prepare wpcli and local access

Install wp using the CLI setup guide, and check wp version and wp --help. The noninteractive examples work with the established operations; interactive wp shell requires a source build containing the shell addition and is not part of v0.1.47.

Run the following from deployments/milvus-full-stack-helm/, with the stack deployed and the connection-discovery Python dependency installed:

Commands
# Keep this running in a separate terminal.
./stack.sh forward --wpcli -x
Commands
./stack.sh connections
./stack.sh connections --wpcli-config > .local/wpcli.yaml
export WOODPECKER_CLI_CONFIG="$PWD/.local/wpcli.yaml"
wp ctx view

This creates a separate stack-local file. Your personal ~/.woodpecker/cli.yaml is unchanged. Export requires working WP Admin forwards; metadata readiness is reported separately. Re-run discovery after changing the deployment or restarting forwards.

Understand the two connection scopes

ScopeConfigurationPurpose
WP service clustercli.yaml: Admin endpoint, per-node Admin URL mappings and Kubernetes context/namespace.Inspect WP nodes and their runtime state. Select this connection with --context.
Milvus instance WP client metadataSeparate etcd endpoint and full metadata prefix; bucket/root path when an operation needs the storage identity.Inspect metadata managed by this Milvus instance’s WP client. Supply explicit flags or attach it inside the shell.

--wpcli-config puts the second scope in an independent comment section, with a copyable connect command. It does not put etcd, metadata-prefix or storage settings into the generated ctx.

In the default example, etcd is reachable at 127.0.0.1:32379 and the complete metadata prefix is full-stack/woodpecker, derived from Milvus’s etcd.rootPath and woodpecker.meta.prefix. The bucket is milvus-bucket; the WP WAL root is milvus/wp, including the /wp suffix added by the Milvus WAL adapter.

Use the discovered values for your instance. A shared WP service cluster can serve different clients; selecting its ctx alone does not identify which client metadata you intend to inspect.

Inspect nodes and metadata

Commands
# Service-level inspection uses only the WP context.
wp --context wp-ops-v0146/milvus-stack node list

# Select this Milvus instance's metadata explicitly.
wp --context wp-ops-v0146/milvus-stack \
  --etcd 127.0.0.1:32379 --meta-prefix full-stack/woodpecker \
  meta log list

The first result describes service nodes. The second lists logical logs in the chosen client metadata namespace. Keep actual log/node/segment identities from these results when following a diagnosis workflow; sample IDs from another environment will refer to different data.

Keep an interactive operational session

With a binary whose help lists shell, start a session using the generated context. Without --context, an interactive terminal offers the configured contexts for selection.

Commands
wp shell --context wp-ops-v0146/milvus-stack

Inside the shell, attach the Milvus instance’s WP client metadata explicitly:

Commands
connect milvus-stack --etcd 127.0.0.1:32379 --meta-prefix full-stack/woodpecker --bucket milvus-bucket --root-path milvus/wp
status --check
node list
meta log list

status --check verifies Admin and etcd access for the current session. These connection checks establish reachability; use a workload and the relevant log/data checks to establish application progress and readability.

Use the local operations workspace

Build the current wp CLI and open its local WebUI. Diagnostics finds Woodpecker problems, Operations organizes native confirmation/recovery commands, and Command Explorer exposes individual wp commands. The browser uses the same checks and commands as the terminal.

Commands
make wpcli
bin/wp webui --config /path/to/cli.yaml --listen 127.0.0.1:8795

Put Prometheus, Jaeger and Grafana connections inside the selected server context. Client metadata and storage scope remain explicit per-operation inputs. Check source access before selecting an incident interval. Successful connectivity does not establish historical data availability.

Check a Woodpecker scenario from the CLI

Commands
bin/wp diag catalog
bin/wp --context local diag scenario slow-writes --window 15m \
  --node-id YOUR_WP_NODE --details all -o json > diagnosis.json
bin/wp report inspect diagnosis.json server-disk-watermark
bin/wp report export diagnosis.json --packet > findings.json

Scenarios aggregate independent Metrics/Trace checks and summarize abnormalities, unavailable evidence and execution gaps. They do not add a second root-cause engine. The current catalog has 39 Metrics and 16 Trace checks. At least one evidence source must be configured; a missing Trace source does not block Metrics checks.

Use wpcli with the dashboards

wp diagnose uses current WP Metrics/Admin evidence. wp diag queries retained Prometheus/Jaeger evidence over a lookback or explicit From/To interval. Use native commands to confirm the observed WP condition and review the operation’s own effects before executing an intervention.

The tool diagnoses Woodpecker client/server behavior, including a WP client embedded in Milvus. It does not diagnose Milvus components, query/search, indexing, persistence stages or Loki logs. The former wpdiag source/stage commands and standalone UI are retired. Their accepted receipts remain historical records.

Saved runs and optional Agent analysis belong to Diagnostics. A model call is explicit and receives supplied findings; it cannot automatically query sources or change the cluster. See the current CLI/WebUI guide for configuration, rules, report semantics and migration.

E2E Testing & Optimization

Use a repeatable workload and its operational evidence to find gaps, investigate bottlenecks and evaluate a change.

Establish a working baseline

With the stack and forwards running, start with the integrated acceptance path:

Commands
./stack.sh acceptance -x

This always runs a Milvus insert/query/search workload, explicitly flushes the collection and verifies its complete row count and search afterward, then collects evidence. Even a small workload exercises persistence metrics without waiting for automatic flush thresholds. It also tests the selected backends: Prometheus rules and metrics, Grafana dashboards, Jaeger trace coverage, Loki log delivery and the Prometheus + Alertmanager firing/recovery/inhibition path. External email is not enabled. Each stage must succeed; keep the resulting reports if one fails so you can inspect that stage. Default trace verification reports known gaps separately; use strict E2E mode when a complete connected path is required.

Run a bounded write experiment

Commands
./stack.sh perf -x

The default run uses a 5-second warm-up, a 60-second measurement window, two workers and a total target of five 100-vector insert requests per second. It then waits 35 seconds for observation. It creates and retains its own collection, checks live read/query/search results and records acknowledged throughput, errors and insert latency.

Commands
PERF_DURATION=300 PERF_CONCURRENCY=4 PERF_BATCH_SIZE=200 PERF_RPS=10 ./stack.sh perf -x

This example raises the requested load; achieved throughput depends on the workload generator and stack. Compare one change at a time with the same images, resources and parameters. REST transport and port-forward overhead are part of the measured latency, and successful live reads do not prove durability through a crash.

Follow the evidence

Each performance run writes a unique directory under .local/perf/, including:

ArtifactUse
summary.md, report.jsonResults, parameters, correctness checks, missing signals, failures and workload/collection observations.
requests.jsonlInsert attempts, row-ID ranges, timestamps, latency and errors.
signals.jsonl, metric-*.jsonMetric/alert snapshots and exact queries over the run’s time range.
pods.json, events.txt, logsComponent images/resources, Kubernetes events and bounded component logs.

Start from failed writes or correctness checks, then compare Milvus mutation latency, WP client append/queue behavior, server RPC/flush latency and storage/resource signals. Use node/log identities and timestamps to narrow the wpcli and log investigation. Missing lazy error counters or idle histograms are coverage gaps until the corresponding behavior is exercised.

For a wider snapshot, run ./stack.sh collect. The detailed performance guide explains gates, artifact limits and interpretation. Review logs before sharing evidence.

Turn an observation into an improvement

  1. Record the symptom and baseline. Keep the workload parameters, timestamps, selected instance and observed result.
  2. Locate the affected stage. Distinguish load-generator delay, Milvus processing, WP client queues, server work and object storage.
  3. Close the relevant gap. Improve a metric, query, alert, runbook or configuration based on that evidence.
  4. Rerun and compare. Verify application results, operational checks and the new signal under the same conditions.

Use the same approach on your own Milvus + WP deployment after adapting endpoints, labels, permissions and workload limits. Treat a local result as evidence for the measured configuration, then validate the change on the target environment.

Validate the current Woodpecker diagnostic tool

The current workflow is tracked in #503: wp CLI atomic Metrics/Trace checks, scenario summaries, native Operations, Command Explorer and optional supplied-report Agent analysis. Use wp diag and wp webui; the old wpdiag execution surface is retired.

Commands
make wpcli
python3 cmd/wpdiag/tests/workspace_acceptance.py
python3 cmd/wpdiag/tests/prometheus_smoke.py
python3 cmd/wpdiag/tests/jaeger_smoke.py

Browser acceptance uses bounded HTTP fixtures and a fake Agent; source smoke tests use isolated temporary Prometheus/Jaeger containers. These are separate from real deployment receipts. Record the actual image/build, interval, scope and source coverage when checking the stack. Missing samples, idle traffic and an incomplete Trace query are not proof of health.

Confirm WP runtime observations

Use current WP Metrics/Admin commands and explicitly scoped metadata to confirm a suspected WAL problem. The earlier native acceptance guide records its bounded service fixture and limits. Do not run fault-injection harnesses against a shared deployment. Existing metadata views/checks are native operations, not a new dynamic metadata diagnosis subsystem.

Historical foundation and acceptance records

The original #450 and finite #458 acceptance program are closed. #493 supplied the final audit. The remaining Milvus published-image propagation outcome in #459 was retired by scope decision, not counted as a passed full-chain test.

These records preserve the builds, experiments and limits of earlier implementations. Their legacy commands and Milvus stages are not supported capabilities of the current WP-only tool.

Operational improvements

Use measured Woodpecker problems to improve check coverage, guidance and observability. Milvus remains a useful application workload and deployment example; it does not expand the diagnostic tool’s ownership boundary.

Welcome to Woodpecker

A cloud-native Write-Ahead Log storage engine that leverages object storage for low-cost, high-throughput, and reliable logging.

What is Woodpecker?

Woodpecker is a cloud-native WAL (Write-Ahead Log) storage implementation designed to fully utilize cloud-native infrastructure for scalability and durability. Unlike traditional on-premises WAL solutions or custom-built distributed logging systems, Woodpecker uses cloud object storage as its durable storage layer.

Woodpecker is currently under active development. Initially built for Milvus vector database and Zilliz Cloud, with broad applicability across cloud workloads.

Key Features

  • Cloud-Native WAL — Uses cloud object storage (S3, GCS, Azure, OSS) as the durable layer for scalability and cost-effectiveness
  • High-Throughput Writes — Optimized write strategies achieving 60-80% of maximum backend throughput
  • Efficient Log Reads — Memory management and prefetching strategies for optimized sequential access
  • Ordered Durability — Strict sequential ordering guarantees for log persistence
  • Flexible Deployment — Standalone service mode or embedded library mode
  • Multi-Cloud Support — AWS, GCP, Azure, Aliyun, Tencent Cloud, MinIO
  • Innovative Quorum Protocol — Configurable ensemble/write/ack quorum with AZ-aware placement
  • Condition Write Fence — Prevents split-brain with built-in fence mechanism

Core Components

ComponentDescription
ClientRead/write protocol layer for interacting with Woodpecker
EmbeddedClientClient with built-in LogStore for embedded deployments
LogStoreHandles high-speed log writes, batching, and cloud storage uploads
ObjectStorageCloud object storage backend (S3, GCS, OSS, etc.)
MetadataProvideretcd-backed metadata and coordination service
QuorumDiscoveryQuorum-based node selection and replication management

Design Benefits

  • Lightweight deployment — Easy integration with minimal dependencies
  • Decoupled compute & storage — Reduced operational complexity
  • Auto-scaling storage — Eliminates capacity planning overhead
  • Reduced local disk dependency — Ideal for cloud-native workloads
  • One-write, multiple-reads — Data-embedded metadata design enables concurrent reads without writer synchronization

Quick Start

Choose one complete path: deploy the dependencies and WAL, configure the matching client, then write and read a message.

PathWhere the application runsClient
Kubernetes · ServiceA Job in the same namespace as the LogStore clusterNewClient
Docker Compose · ServiceA container on the same Compose network as the LogStore nodesNewClient
Embedded · Go libraryYour workstation; etcd and MinIO run in DockerNewEmbedClientFromConfig

The two Service paths deploy the same architecture in different environments. Embedded runs its LogStore inside the application. Follow one tab from beginning to end; each includes its own configuration, commands and output interpretation.

These development examples use a released milvusdb/woodpecker:v0.1.46 server image and the v0.1.47 SDK API. Clone the current source checkout for the tutorial files added after that release. Development credentials, single-instance dependencies and small PVCs are for evaluation; use your own dependency, storage and lifecycle configuration for an operational deployment.

Kubernetes: deploy a WAL service and run an application

1. Prepare your cluster and checkout

Use Kubernetes 1.28+, kubectl, Docker, and a default StorageClass with at least 6Gi available for the example PVCs. Allocate resources for three LogStore nodes plus etcd, MinIO and an application Job. Commands assume the namespace wp-quickstart and the usual cluster DNS suffix cluster.local. Start at the repository root:

bash
git clone --depth 1 https://github.com/zilliztech/woodpecker.git
cd woodpecker
bash
kubectl config current-context
kubectl get storageclass

2. Deploy etcd, MinIO and three LogStore nodes

The complete manifest creates a dedicated namespace, dependency Services/Deployments/PVCs, a server ConfigMap, a headless Service and a three-replica StatefulSet. The headless Service publishes each node for direct quorum access.

bash
kubectl apply -f docs/quickstart/kubernetes.yaml
kubectl -n wp-quickstart rollout status deployment/qs-etcd --timeout=180s
kubectl -n wp-quickstart rollout status deployment/qs-minio --timeout=180s
kubectl -n wp-quickstart rollout status statefulset/wp --timeout=300s
kubectl -n wp-quickstart get pods

Continue when qs-etcd, qs-minio and wp-0/1/2 are Ready. Pending PVCs mean the StorageClass/capacity needs attention; ImagePullBackOff means the registry/image is unavailable. Inspect with kubectl -n wp-quickstart describe pod <pod>.

3. Use the Kubernetes client configuration

kubernetes-client.yaml uses names resolvable from the application Job in this namespace. Its metadata namespace, bucket and root path match the server ConfigMap:

kubernetes-client.yaml
# Development example. Replace credentials/endpoints for your environment.
woodpecker:
  storage:
    type: service
    rootPath: /woodpecker/data
  client:
    quorum:
      quorumBufferPools:
        - name: quickstart
          seeds:
            - wp-0.wp-headless:18080
            - wp-1.wp-headless:18080
            - wp-2.wp-headless:18080
etcd:
  endpoints: ["qs-etcd:2379"]
  rootPath: quickstart-k8s
minio:
  address: qs-minio
  port: 9000
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  useSSL: false
  bucketName: quickstart-k8s
  rootPath: wal
  createBucket: true
log:
  level: error
  stdout: true
A seed is discovery, not a proxy. The SDK must reach every selected node, etcd, and object storage. A single port-forward or LoadBalancer does not expose all quorum endpoints. This walkthrough runs the app inside Kubernetes; for an external SDK, provide DNS/routing to every advertised gRPC address and reachable dependencies. The wp CLI’s node-admin mappings apply to HTTP administration, not SDK gRPC traffic.

4. Build and run the application Job

The runnable client.go loads the selected YAML, creates an external etcd connection and a service client:

Client initialization · excerpt
// cfg is loaded from the path-specific client YAML below.
etcdClient, err := clientv3.New(clientv3.Config{
    Endpoints: cfg.Etcd.GetEndpoints(),
    DialTimeout: 5 * time.Second,
})
if err != nil {
    return err
}
defer etcdClient.Close()
client, err := woodpecker.NewClient(ctx, cfg, etcdClient, false)
if err != nil {
    return err
}
defer client.Close(context.Background())

It creates a fresh log, appends a message, and reads from the acknowledged position:

Write and read · excerpt
// After CreateLog / OpenLog / OpenLogWriter (see the complete program):
result := writer.Write(ctx, &wplog.WriteMessage{Payload: []byte("hello, Woodpecker")})
if result.Err != nil {
    return result.Err
}
reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
if err != nil {
    return err
}
defer reader.Close(context.Background())
message, err := reader.ReadNext(ctx)
if err != nil {
    return err
}
fmt.Println(string(message.Payload))
Complete application, including imports, configuration loading and cleanup
docs/quickstart/client.go
// Licensed to the LF AI & Data foundation under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
	"bytes"
	"context"
	"flag"
	"fmt"
	"log"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"

	"github.com/zilliztech/woodpecker/common/config"
	"github.com/zilliztech/woodpecker/woodpecker"
	wplog "github.com/zilliztech/woodpecker/woodpecker/log"
)

func main() {
	file := flag.String("config", "client.yaml", "SDK configuration file (not wp CLI cli.yaml)")
	flag.Parse()
	if err := run(*file); err != nil {
		log.Fatal(err)
	}
}

func run(file string) error {
	ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
	defer cancel()
	cfg, err := config.NewConfiguration(file)
	if err != nil {
		return fmt.Errorf("load SDK configuration: %w", err)
	}

	var client woodpecker.Client
	if cfg.Woodpecker.Storage.IsStorageService() {
		etcdClient, err := clientv3.New(clientv3.Config{
			Endpoints:   cfg.Etcd.GetEndpoints(),
			DialTimeout: 5 * time.Second,
		})
		if err != nil {
			return fmt.Errorf("create etcd client: %w", err)
		}
		defer etcdClient.Close()
		client, err = woodpecker.NewClient(ctx, cfg, etcdClient, false)
		if err != nil {
			return fmt.Errorf("create service client: %w", err)
		}
	} else if cfg.Woodpecker.Storage.IsStorageMinio() {
		client, err = woodpecker.NewEmbedClientFromConfig(ctx, cfg)
		if err != nil {
			return fmt.Errorf("create embedded client: %w", err)
		}
	} else {
		return fmt.Errorf("this example supports storage.type service or minio")
	}
	defer client.Close(context.Background())

	name := fmt.Sprintf("quickstart-%d", time.Now().UnixNano())
	if err := client.CreateLog(ctx, name); err != nil {
		return fmt.Errorf("create log: %w", err)
	}
	handle, err := client.OpenLog(ctx, name)
	if err != nil {
		return fmt.Errorf("open log: %w", err)
	}
	defer handle.Close(context.Background())
	writer, err := handle.OpenLogWriter(ctx)
	if err != nil {
		return fmt.Errorf("open writer: %w", err)
	}
	defer writer.Close(context.Background())

	payload := []byte("hello, Woodpecker")
	result := writer.Write(ctx, &wplog.WriteMessage{Payload: payload})
	if result.Err != nil {
		return fmt.Errorf("write: %w", result.Err)
	}
	fmt.Printf("WROTE log=%s position=%d:%d\n", name, result.LogMessageId.SegmentId, result.LogMessageId.EntryId)

	reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
	if err != nil {
		return fmt.Errorf("open reader: %w", err)
	}
	defer reader.Close(context.Background())
	message, err := reader.ReadNext(ctx)
	if err != nil {
		return fmt.Errorf("read: %w", err)
	}
	if !bytes.Equal(message.Payload, payload) || *message.Id != *result.LogMessageId {
		return fmt.Errorf("read result differs from acknowledged write")
	}
	fmt.Printf("READ position=%d:%d payload=%q\n", message.Id.SegmentId, message.Id.EntryId, message.Payload)
	fmt.Printf("VERIFIED mode=%s messages=1\n", cfg.Woodpecker.Storage.Type)
	return nil
}

Build the container on the same CPU architecture as the cluster nodes:

bash
docker build -f docs/quickstart/Dockerfile -t woodpecker-quickstart-client:local .

For minikube, load the image into the intended profile (replace minikube if yours has another name):

bash
minikube -p minikube image load woodpecker-quickstart-client:local

For a remote cluster, replace the example registry below with one your cluster can pull from, push the image, and set that exact image in kubernetes-job.yaml:

bash
docker tag woodpecker-quickstart-client:local registry.example.com/team/wp-quickstart-client:demo
docker push registry.example.com/team/wp-quickstart-client:demo

Then create the client ConfigMap and Job:

bash
kubectl -n wp-quickstart create configmap wp-client-config \
  --from-file=client.yaml=docs/quickstart/kubernetes-client.yaml \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f docs/quickstart/kubernetes-job.yaml
kubectl -n wp-quickstart wait --for=condition=complete job/quickstart-client --timeout=180s
kubectl -n wp-quickstart logs job/quickstart-client

5. Verify the result

Expected application output (log name varies; runtime messages may also appear):

Example output
WROTE log=quickstart-... position=0:0
READ position=0:0 payload="hello, Woodpecker"
VERIFIED mode=service messages=1

WROTE reports the acknowledged position. READ reports the entry read back. The program prints VERIFIED only after both the message ID and payload match. A nonzero exit or missing VERIFIED line requires investigation.

If the Job fails, inspect its logs and kubectl -n wp-quickstart describe job quickstart-client. Check Service DNS and node seeds for connection errors; compare metadata/bucket/root-path settings if the intended log or data is absent. To repeat the application, delete only its Job and reapply its manifest:

bash
kubectl -n wp-quickstart delete job quickstart-client
kubectl apply -f docs/quickstart/kubernetes-job.yaml

6. Clean up the evaluation environment

This deletes the dedicated namespace, its example workloads, PVCs and their data:

bash
kubectl delete namespace wp-quickstart

The manifest is a plain StatefulSet evaluation deployment. For managed Woodpecker lifecycle operations, see the Operator guide.

Docker Compose: deploy a WAL service and run an application

1. Prepare Docker and the checkout

Install Git and Docker with Compose v2.20+. Docker builds the Go application, so no host Go installation is needed for this path. Run all commands from this checkout’s repository root:

bash
git clone --depth 1 https://github.com/zilliztech/woodpecker.git
cd woodpecker

2. Start dependencies and three LogStore nodes

The tutorial-specific Compose file creates its own network and volumes, etcd, MinIO and three LogStore nodes. It advertises container DNS names so the application and nodes can reach one another; it is separate from the general deployments/ template.

bash
docker compose -f docs/quickstart/compose.yaml --profile service up -d --wait --wait-timeout 180 etcd minio wp-0 wp-1 wp-2
docker compose -f docs/quickstart/compose.yaml --profile service ps

All five services must be healthy. Etcd and MinIO also bind workstation-only ports 12379 and 19000, used by the Embedded tab. Change those bindings if occupied; the Service app uses internal names.

3. Use the Compose client configuration

compose-client.yaml is mounted into the servers and the application. It sets service mode, seed addresses, metadata namespace and object-storage location:

compose-client.yaml
# Development example. Replace credentials/endpoints for your environment.
woodpecker:
  storage:
    type: service
    rootPath: /woodpecker/data
  client:
    quorum:
      quorumBufferPools:
        - name: quickstart
          seeds:
            - wp-0:18080
            - wp-1:18080
            - wp-2:18080
etcd:
  endpoints: ["etcd:2379"]
  rootPath: quickstart-service
minio:
  address: minio
  port: 9000
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  useSSL: false
  bucketName: quickstart-service
  rootPath: wal
  createBucket: true
log:
  level: error
  stdout: true

Names such as wp-0:18080, etcd:2379 and minio:9000 are Docker-network addresses. Run the application container below on that network. Setting only a localhost seed on the workstation does not rewrite the addresses returned in quorum metadata.

4. Create the service client, write and read

The runnable client.go loads the selected YAML, creates an external etcd connection and a service client:

Client initialization · excerpt
// cfg is loaded from the path-specific client YAML below.
etcdClient, err := clientv3.New(clientv3.Config{
    Endpoints: cfg.Etcd.GetEndpoints(),
    DialTimeout: 5 * time.Second,
})
if err != nil {
    return err
}
defer etcdClient.Close()
client, err := woodpecker.NewClient(ctx, cfg, etcdClient, false)
if err != nil {
    return err
}
defer client.Close(context.Background())

It creates a fresh log, appends a message, and reads from the acknowledged position:

Write and read · excerpt
// After CreateLog / OpenLog / OpenLogWriter (see the complete program):
result := writer.Write(ctx, &wplog.WriteMessage{Payload: []byte("hello, Woodpecker")})
if result.Err != nil {
    return result.Err
}
reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
if err != nil {
    return err
}
defer reader.Close(context.Background())
message, err := reader.ReadNext(ctx)
if err != nil {
    return err
}
fmt.Println(string(message.Payload))
Complete application, including imports, configuration loading and cleanup
docs/quickstart/client.go
// Licensed to the LF AI & Data foundation under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
	"bytes"
	"context"
	"flag"
	"fmt"
	"log"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"

	"github.com/zilliztech/woodpecker/common/config"
	"github.com/zilliztech/woodpecker/woodpecker"
	wplog "github.com/zilliztech/woodpecker/woodpecker/log"
)

func main() {
	file := flag.String("config", "client.yaml", "SDK configuration file (not wp CLI cli.yaml)")
	flag.Parse()
	if err := run(*file); err != nil {
		log.Fatal(err)
	}
}

func run(file string) error {
	ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
	defer cancel()
	cfg, err := config.NewConfiguration(file)
	if err != nil {
		return fmt.Errorf("load SDK configuration: %w", err)
	}

	var client woodpecker.Client
	if cfg.Woodpecker.Storage.IsStorageService() {
		etcdClient, err := clientv3.New(clientv3.Config{
			Endpoints:   cfg.Etcd.GetEndpoints(),
			DialTimeout: 5 * time.Second,
		})
		if err != nil {
			return fmt.Errorf("create etcd client: %w", err)
		}
		defer etcdClient.Close()
		client, err = woodpecker.NewClient(ctx, cfg, etcdClient, false)
		if err != nil {
			return fmt.Errorf("create service client: %w", err)
		}
	} else if cfg.Woodpecker.Storage.IsStorageMinio() {
		client, err = woodpecker.NewEmbedClientFromConfig(ctx, cfg)
		if err != nil {
			return fmt.Errorf("create embedded client: %w", err)
		}
	} else {
		return fmt.Errorf("this example supports storage.type service or minio")
	}
	defer client.Close(context.Background())

	name := fmt.Sprintf("quickstart-%d", time.Now().UnixNano())
	if err := client.CreateLog(ctx, name); err != nil {
		return fmt.Errorf("create log: %w", err)
	}
	handle, err := client.OpenLog(ctx, name)
	if err != nil {
		return fmt.Errorf("open log: %w", err)
	}
	defer handle.Close(context.Background())
	writer, err := handle.OpenLogWriter(ctx)
	if err != nil {
		return fmt.Errorf("open writer: %w", err)
	}
	defer writer.Close(context.Background())

	payload := []byte("hello, Woodpecker")
	result := writer.Write(ctx, &wplog.WriteMessage{Payload: payload})
	if result.Err != nil {
		return fmt.Errorf("write: %w", result.Err)
	}
	fmt.Printf("WROTE log=%s position=%d:%d\n", name, result.LogMessageId.SegmentId, result.LogMessageId.EntryId)

	reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
	if err != nil {
		return fmt.Errorf("open reader: %w", err)
	}
	defer reader.Close(context.Background())
	message, err := reader.ReadNext(ctx)
	if err != nil {
		return fmt.Errorf("read: %w", err)
	}
	if !bytes.Equal(message.Payload, payload) || *message.Id != *result.LogMessageId {
		return fmt.Errorf("read result differs from acknowledged write")
	}
	fmt.Printf("READ position=%d:%d payload=%q\n", message.Id.SegmentId, message.Id.EntryId, message.Payload)
	fmt.Printf("VERIFIED mode=%s messages=1\n", cfg.Woodpecker.Storage.Type)
	return nil
}
bash
docker compose -f docs/quickstart/compose.yaml build client
docker compose -f docs/quickstart/compose.yaml run --rm client --config /examples/compose-client.yaml

5. Verify the result

Expected application output (log name varies; runtime messages may also appear):

Example output
WROTE log=quickstart-... position=0:0
READ position=0:0 payload="hello, Woodpecker"
VERIFIED mode=service messages=1

WROTE reports the acknowledged position. READ reports the entry read back. The program prints VERIFIED only after both the message ID and payload match. A nonzero exit or missing VERIFIED line requires investigation.

For startup or connection failures, inspect the dependency and LogStore logs, verify that all nodes joined, and rerun the client once they are healthy:

bash
docker compose -f docs/quickstart/compose.yaml --profile service logs --tail=80 etcd minio wp-0 wp-1 wp-2

6. Stop the example

bash
docker compose -f docs/quickstart/compose.yaml --profile service down

Named volumes retain the example data. To discard that data as well, add --volumes to the down command. Each application run creates a new log.

Embedded: run the WAL engine inside a Go application

1. Prepare Go, Docker and the checkout

Install Git, Go 1.25+ (see go.mod), and Docker with Compose v2.20+. This application runs on your workstation; only etcd and MinIO run in containers.

bash
git clone --depth 1 https://github.com/zilliztech/woodpecker.git
cd woodpecker

2. Start just etcd and MinIO

bash
docker compose -f docs/quickstart/compose.yaml up -d --wait --wait-timeout 180 etcd minio
docker compose -f docs/quickstart/compose.yaml ps

No standalone LogStore nodes are required. The two dependencies must be healthy and reachable at 127.0.0.1:12379 and 127.0.0.1:19000. This reuses the tutorial Compose dependencies if the Service example is already running.

3. Use the workstation client configuration

embedded-host.yaml selects minio storage, host-reachable dependency ports, and a separate metadata/bucket namespace. This is the SDK YAML, not wp’s cli.yaml:

embedded-host.yaml
# Development example. Replace credentials/endpoints for your environment.
woodpecker:
  storage:
    type: minio
    rootPath: ./quickstart-data
etcd:
  endpoints: ["127.0.0.1:12379"]
  rootPath: quickstart-embedded-host
minio:
  address: 127.0.0.1
  port: 19000
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  useSSL: false
  bucketName: quickstart-embedded-host
  rootPath: wal
  createBucket: true
log:
  level: error
  stdout: true

4. Start the embedded client, write and read

The runnable client.go loads the selected YAML, starts an embedded client connected to etcd and object storage:

Client initialization · excerpt
// storage.type is minio; the LogStore runs inside this application.
client, err := woodpecker.NewEmbedClientFromConfig(ctx, cfg)
if err != nil {
    return err
}
defer client.Close(context.Background())

It creates a fresh log, appends a message, and reads from the acknowledged position:

Write and read · excerpt
// After CreateLog / OpenLog / OpenLogWriter (see the complete program):
result := writer.Write(ctx, &wplog.WriteMessage{Payload: []byte("hello, Woodpecker")})
if result.Err != nil {
    return result.Err
}
reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
if err != nil {
    return err
}
defer reader.Close(context.Background())
message, err := reader.ReadNext(ctx)
if err != nil {
    return err
}
fmt.Println(string(message.Payload))
Complete application, including imports, configuration loading and cleanup
docs/quickstart/client.go
// Licensed to the LF AI & Data foundation under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
	"bytes"
	"context"
	"flag"
	"fmt"
	"log"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"

	"github.com/zilliztech/woodpecker/common/config"
	"github.com/zilliztech/woodpecker/woodpecker"
	wplog "github.com/zilliztech/woodpecker/woodpecker/log"
)

func main() {
	file := flag.String("config", "client.yaml", "SDK configuration file (not wp CLI cli.yaml)")
	flag.Parse()
	if err := run(*file); err != nil {
		log.Fatal(err)
	}
}

func run(file string) error {
	ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
	defer cancel()
	cfg, err := config.NewConfiguration(file)
	if err != nil {
		return fmt.Errorf("load SDK configuration: %w", err)
	}

	var client woodpecker.Client
	if cfg.Woodpecker.Storage.IsStorageService() {
		etcdClient, err := clientv3.New(clientv3.Config{
			Endpoints:   cfg.Etcd.GetEndpoints(),
			DialTimeout: 5 * time.Second,
		})
		if err != nil {
			return fmt.Errorf("create etcd client: %w", err)
		}
		defer etcdClient.Close()
		client, err = woodpecker.NewClient(ctx, cfg, etcdClient, false)
		if err != nil {
			return fmt.Errorf("create service client: %w", err)
		}
	} else if cfg.Woodpecker.Storage.IsStorageMinio() {
		client, err = woodpecker.NewEmbedClientFromConfig(ctx, cfg)
		if err != nil {
			return fmt.Errorf("create embedded client: %w", err)
		}
	} else {
		return fmt.Errorf("this example supports storage.type service or minio")
	}
	defer client.Close(context.Background())

	name := fmt.Sprintf("quickstart-%d", time.Now().UnixNano())
	if err := client.CreateLog(ctx, name); err != nil {
		return fmt.Errorf("create log: %w", err)
	}
	handle, err := client.OpenLog(ctx, name)
	if err != nil {
		return fmt.Errorf("open log: %w", err)
	}
	defer handle.Close(context.Background())
	writer, err := handle.OpenLogWriter(ctx)
	if err != nil {
		return fmt.Errorf("open writer: %w", err)
	}
	defer writer.Close(context.Background())

	payload := []byte("hello, Woodpecker")
	result := writer.Write(ctx, &wplog.WriteMessage{Payload: payload})
	if result.Err != nil {
		return fmt.Errorf("write: %w", result.Err)
	}
	fmt.Printf("WROTE log=%s position=%d:%d\n", name, result.LogMessageId.SegmentId, result.LogMessageId.EntryId)

	reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
	if err != nil {
		return fmt.Errorf("open reader: %w", err)
	}
	defer reader.Close(context.Background())
	message, err := reader.ReadNext(ctx)
	if err != nil {
		return fmt.Errorf("read: %w", err)
	}
	if !bytes.Equal(message.Payload, payload) || *message.Id != *result.LogMessageId {
		return fmt.Errorf("read result differs from acknowledged write")
	}
	fmt.Printf("READ position=%d:%d payload=%q\n", message.Id.SegmentId, message.Id.EntryId, message.Payload)
	fmt.Printf("VERIFIED mode=%s messages=1\n", cfg.Woodpecker.Storage.Type)
	return nil
}
bash
go run ./docs/quickstart/client.go --config docs/quickstart/embedded-host.yaml

5. Verify the result

Expected application output (log name varies; runtime messages may also appear):

Example output
WROTE log=quickstart-... position=0:0
READ position=0:0 payload="hello, Woodpecker"
VERIFIED mode=minio messages=1

WROTE reports the acknowledged position. READ reports the entry read back. The program prints VERIFIED only after both the message ID and payload match. A nonzero exit or missing VERIFIED line requires investigation.

Connection refused usually means a dependency is not healthy or the workstation port differs from the YAML. An authentication error requires matching MinIO credentials. Use the NewEmbedClientFromConfig path with storage.type: minio; service storage requires the other client constructor.

6. Stop dependencies

bash
docker compose -f docs/quickstart/compose.yaml down

Stop any Service example using these dependencies first. Named volumes retain metadata and object data; add --volumes to discard the tutorial data. For an application in its own module, install github.com/zilliztech/woodpecker@v0.1.47 and copy the complete Go source above.

Configuration

Comprehensive guide to configuring Woodpecker with all available options and their defaults.

Full Configuration Example

Woodpecker uses YAML configuration. Create a woodpecker.yaml file:

woodpecker.yaml
woodpecker:
  meta:
    type: etcd
    prefix: woodpecker
  client:
    segmentAppend:
      queueSize: 100            # admitted entries per segment through all replica outcomes and callbacks
      maxRetries: 2             # total attempts per entry and replica, including the first send
      maxBatchEntries: 1000     # queued entries per batch; no wait for more entries
      maxBatchBytes: 2000000     # payload threshold; the last entry may exceed it
      maxPendingBytes: 0        # retained payload budget in both modes; 0 uses 64 MiB
      sendTimeout: 2000         # ms per remote send attempt through its first response
    segmentRollingPolicy:
      maxSize: 100000000        # 100MB per segment
      maxInterval: 800          # seconds between rolls
      maxBlocks: 1000           # max blocks per segment
    auditor:
      maxInterval: 5            # seconds between audits
    sessionMonitor:
      checkInterval: 3          # seconds
      maxFailures: 5            # consecutive fails before invalid
    grpc:
      clientMaxSendSize: 268435456 # 256MB
      clientMaxRecvSize: 536870912 # 512MB
    # quorum: (service mode only, see Quorum section)
  logstore:
    segmentSyncPolicy:
      maxInterval: 1000         # ms between syncs
      maxIntervalForLocalStorage: 5  # ms for local backend
      maxEntries: 2000          # max entries per buffer
      maxBytes: 100000000       # 100MB buffer limit
      maxFlushRetries: 3
      retryInterval: 2000       # ms between retries
      maxFlushSize: 16000000    # 16MB per block flush
      maxFlushThreads: 8
    segmentCompactionPolicy:
      maxBytes: 32000000       # 32MB merged block size
      maxParallelUploads: 4
      maxParallelReads: 8
    segmentReadPolicy:
      maxBatchSize: 16000000   # 16MB read batch
      maxFetchThreads: 32
    retentionPolicy:
      ttl: 259200             # 72 hours for truncated segments
    fencePolicy:
      conditionWrite: "auto"   # auto | enable | disable
    grpc:
      serverMaxSendSize: 536870912   # 512MB
      serverMaxRecvSize: 268435456   # 256MB
    processorCleanupPolicy:
      cleanupInterval: 60     # seconds
      maxIdleTime: 300        # seconds
      shutdownTimeout: 15     # seconds
  storage:
    type: default             # default(=minio), minio, local, service
    rootPath: /var/lib/woodpecker

Client Configuration

The defaults below are built into the client. The repository's config/woodpecker.yaml explicitly overrides queueSize to 10000 and maxRetries to 3. Append admission limits apply per writable segment handle, not to the whole client process.

sendTimeout is not an overall append, OpenWriter or Fence deadline. After sending, each entry's final ACK read has a separate internal 30-second budget in both modes, including reads within a batch. Each retry gets fresh attempt budgets.

ParameterDescriptionDefault
segmentAppend.queueSizePositive entry-count admission limit in both modes, covering queued, sending, ACK-waiting and retrying entries until all replica outcomes and the ordered callback finish, even after quorum success.100
segmentAppend.maxRetriesTotal attempts per entry and replica, including the initial send. 0 or 1 disables retries. Both modes retry eligible send/ACK failures on the affected replica; non-retryable failure or exhaustion requests rolling, even after quorum success.2
segmentAppend.maxBatchEntriesMaximum consecutive queued entries per dispatched batch in both modes. Service mode sends one AddEntries RPC per replica; embedded mode uses local calls. Does not wait for more entries. Values ≤ 1 disable batching; retries are individual.1000
segmentAppend.maxBatchBytesPayload threshold for stopping batch collection. The last entry may exceed it; an oversized first entry is sent alone. Values ≤ 0 disable the threshold; ignored when maxBatchEntries ≤ 1. This is not a strict RPC-size limit. The YAML string 2MB means 2097152 bytes, unlike the numeric default.2000000 bytes
segmentAppend.maxPendingBytesAdmitted payload budget in both modes, counted once per entry, with the same credit lifetime as queueSize. Retries reuse credits. 0 uses 64 MiB; one oversized entry may enter when empty. Excludes transport buffers and bookkeeping.0 → 64 MiB
segmentAppend.sendTimeoutMilliseconds per remote AddEntry/AddEntries attempt, from stream opening through the first response (normally Buffered), including connection readiness waits inside the RPC. Excludes admission, client-pool lookup and final ACK reads. Embedded local calls do not use it. 0 uses 2000ms.2000
grpc.dialTimeoutTime allowed for one connection attempt to a logstore, in ms (gRPC MinConnectTimeout); also bounds a call to an address that never answers1000
grpc.connectBackoff.baseDelayFirst delay before reconnecting after a failed attempt, in ms100
grpc.connectBackoff.multiplierGrowth factor of the reconnect delay1.6
grpc.connectBackoff.jitterRandomisation of the reconnect delay0.2
grpc.connectBackoff.maxDelayLongest reconnect delay, in ms3000
quorum.selectNodesTimeoutTime allowed for one SelectNodes call to a seed, in ms; the next seed is asked when it expires2000
segmentRead.activeTimeoutTime allowed for one quorum read of an active segment from one replica, in ms; the reader moves to the next replica when it expires3000
segmentRead.settledTimeoutTime allowed for one quorum read of any other segment, which may be served from object storage, in ms20000
segmentRollingPolicy.maxSizeMaximum segment size in bytes100MB
segmentRollingPolicy.maxIntervalMaximum interval between segment rolls (seconds)800
segmentRollingPolicy.maxBlocksMaximum blocks per segment1000
auditor.maxIntervalAudit interval in seconds5
sessionMonitor.checkIntervalSession health check interval (seconds)3
sessionMonitor.maxFailuresConsecutive failures before session invalid5

LogStore Configuration

ParameterDescriptionDefault
segmentSyncPolicy.maxIntervalMax sync interval (milliseconds)1000
segmentSyncPolicy.maxIntervalForLocalStorageSync interval for local storage (ms)5
segmentSyncPolicy.maxEntriesMax entries in write buffer2000
segmentSyncPolicy.maxBytesMax write buffer size100MB
segmentSyncPolicy.maxFlushRetriesMax sync retries3
segmentSyncPolicy.retryIntervalRetry interval (ms)2000
segmentSyncPolicy.maxFlushSizeMax block flush size16MB
segmentSyncPolicy.maxFlushThreadsConcurrent flush threads8
segmentCompactionPolicy.maxBytesMax merged block size after compaction32MB
segmentCompactionPolicy.maxParallelUploadsParallel compaction upload threads4
segmentCompactionPolicy.maxParallelReadsParallel compaction read threads8
segmentReadPolicy.maxBatchSizeMax read batch size16MB
segmentReadPolicy.maxFetchThreadsConcurrent fetch threads32
retentionPolicy.ttlTTL for truncated segments (seconds)259200 (72h)
processorCleanupPolicy.cleanupIntervalIdle processor scan interval (seconds)60
processorCleanupPolicy.maxIdleTimeIdle time before cleanup (seconds)300

Storage Configuration

ParameterDescriptionDefault
storage.typeStorage backend: default (=minio), minio, local, servicedefault
storage.rootPathRoot path for log data files/var/lib/woodpecker

etcd Configuration

woodpecker.yaml
etcd:
  endpoints: [localhost:2379]
  rootPath: by-dev
  metaSubPath: meta
  kvSubPath: kv
  requestTimeout: 10000        # ms
  use:
    embed: false              # enable embedded etcd
  data:
    dir: default.etcd         # embedded etcd data dir
  ssl:
    enabled: false
    tlsCert: /path/to/cert.pem
    tlsKey: /path/to/key.pem
    tlsCACert: /path/to/ca.pem
  auth:
    enabled: false
    userName: ""
    password: ""

MinIO / S3 Configuration

woodpecker.yaml
minio:
  address: localhost
  port: 9000
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  useSSL: false
  bucketName: a-bucket
  createBucket: false
  rootPath: files
  useIAM: false
  cloudProvider: aws           # aws, gcp, aliyun, gcpnative
  region: ""
  useVirtualHost: false
  requestTimeoutMs: 10000
  listObjectsMaxKeys: 0       # 0 = unlimited
ParameterDescriptionDefault
minio.addressMinIO/S3 server addresslocalhost
minio.portMinIO/S3 server port9000
minio.accessKeyIDAccess key for authenticationminioadmin
minio.secretAccessKeySecret key for authenticationminioadmin
minio.bucketNameStorage bucket namea-bucket
minio.createBucketAuto-create bucket if not existsfalse
minio.useIAMUse IAM role authenticationfalse
minio.cloudProviderCloud provider: aws, gcp, aliyun, gcpnativeaws
minio.regionStorage region—
minio.useVirtualHostVirtual host style addressingfalse
minio.requestTimeoutMsRequest timeout in milliseconds10000

Quorum Configuration (Service Mode)

Required when using storage.type: service with a LogStore cluster.

woodpecker.yaml
woodpecker:
  client:
    quorum:
      quorumBufferPools:
        - name: "region-1"
          seeds: ["logstore1:7946", "logstore2:7946"]
      quorumSelectStrategy:
        affinityMode: "soft"   # soft | hard
        replicas: 3             # 3 or 5
        strategy: "random"      # see table below
        customPlacement:       # only for strategy: custom
          - name: "replica-1"
            region: "region-1"
            az: "az-1"
            resourceGroup: "rg-1"
Strategy ValueDescription
randomRandom node selection (default)
single-az-single-rgAll nodes in same AZ and resource group
single-az-multi-rgSame AZ, different resource groups
multi-az-single-rgMultiple AZs, same resource group
multi-az-multi-rgMultiple AZs and resource groups
cross-regionNodes across different regions (requires 2+ buffer pools)
random-groupPre-partitioned random groups
customExplicit placement rules via customPlacement

Fence Policy

ValueBehavior
autoFollow existing cluster metadata; otherwise verify condition write and fail startup if unavailable (default)
enableRequire condition write; ignore stored false metadata and fail startup if unavailable
disableForce distributed locks; skip detection

gRPC Configuration

ParameterDescriptionDefault
logstore.grpc.serverMaxSendSizeServer max send message size512MB
logstore.grpc.serverMaxRecvSizeServer max receive message size256MB
client.grpc.clientMaxSendSizeClient max send message size256MB
client.grpc.clientMaxRecvSizeClient max receive message size512MB
logstore.grpc.clientMaxSendSizeDeprecated; used with a warning only when client.grpc.clientMaxSendSize is unset.—
logstore.grpc.clientMaxRecvSizeDeprecated; used with a warning only when client.grpc.clientMaxRecvSize is unset.—

System Architecture

Understanding Woodpecker's two deployment modes and core design.

Embedded Mode

In embedded mode, Woodpecker operates as a lightweight library integrated directly into your application. It requires only etcd for metadata storage and coordination, keeping dependencies minimal.

EmbeddedClient ETCD Cloud Object Storage MinIO / S3 / Aliyun / GCS / Azure

Key characteristics of embedded mode:

  • Zero additional services required beyond etcd
  • Client directly writes to object storage
  • Condition-write fence mechanism prevents split-brain
  • Best suited for applications needing simple WAL integration

Service Mode

In service mode, WAL read/write operations and caching logic are decoupled into a dedicated LogStore cluster service. This acts as a high-performance caching layer between clients and object storage.

Client ETCD LogStore LogStore LogStore Cloud Object Storage MinIO / S3 / Aliyun / GCS / Azure

Key characteristics of service mode:

  • Dedicated LogStore cluster with gRPC communication
  • Data prefetching and read/write caching for lower latency
  • Gossip-based cluster membership and auto-discovery
  • Quorum-based replication with AZ-aware node placement
  • Best suited for high-throughput, latency-sensitive workloads

Core Components

ComponentRole
ClientRead/write protocol layer connecting applications to Woodpecker
EmbeddedClientCombined client + LogStore for embedded deployments
LogStoreHigh-speed log writes, batching, caching, and cloud storage uploads
ObjectStorageDurable storage backend using S3-compatible APIs
MetadataProvideretcd-backed metadata storage, coordination, and distributed locking
QuorumDiscoveryService mode quorum-based node selection and replication management via gossip

Data Flow

Write Path

  1. Application calls Write() or WriteAsync() on a LogWriter
  2. Writer validates the session lock is still valid (distributed lock via etcd)
  3. Gets or creates a writable segment via rolling policy (maxSize=100MB, maxInterval=800s, maxBlocks=1000)
  4. Serializes message with MarshalMessage() (payload + properties)
  5. Calls segment.AppendAsync() — data is buffered
  6. Buffer is flushed to storage when sync policy triggers (maxInterval=1000ms, maxEntries=2000, maxBytes=100MB)
  7. Segment metadata updated in etcd with revision-based optimistic locking

Read Path

  1. Application opens a LogReader with starting position and reader name (OpenLogReader(ctx, from, readerName))
  2. Reader loads entries in batches via ReadBatchAdv() (batch limit: 200 entries)
  3. Yields one entry per ReadNext() call from local cache
  4. On segment EOF, auto-advances to the next segment seamlessly
  5. Reader position updated in etcd every 30 seconds
  6. For active (still-writing) segments, polls with 200ms wait interval when no new data is available

Segment Lifecycle

Segments transition through the following states:

State Machine
Active → Completed → Sealed → Truncated → (Deleted)

Active     - Currently accepting writes
Completed  - All data flushed, no more writes
Sealed     - Metadata finalized, immutable
Truncated  - Marked for cleanup
Deleted    - Storage data removed

Quorum Protocol

Woodpecker's innovative quorum-based replication for service mode deployments.

Overview

In service mode, Woodpecker uses a configurable quorum protocol for data replication across LogStore nodes. This ensures data durability while allowing flexible tradeoffs between consistency, availability, and performance.

Quorum Parameters

The quorum is configured via the replicas setting, which determines the ensemble size (E). Write quorum (WQ) and ack quorum (AQ) are derived automatically:

ParameterSymbolFormulaE=3E=5
Ensemble SizeE= replicas35
Write QuorumWQ= E35
Ack QuorumAQ= (E / 2) + 123
Data is written to all E nodes (WQ = E), but only AQ acknowledgments are required before the write is considered durable. This allows writes to succeed even if some nodes are slow, while ensuring majority-based durability.

Buffer Pools

Buffer pools are logical groupings of LogStore nodes. Each pool is defined by a name and seed addresses used for gossip-based node discovery. Pools enable regional grouping of nodes for locality-aware placement.

woodpecker.yaml
woodpecker:
  client:
    quorum:
      quorumBufferPools:
        - name: "region-1"
          seeds: ["logstore1:7946", "logstore2:7946"]
        - name: "region-2"
          seeds: ["logstore3:7946", "logstore4:7946"]
      quorumSelectStrategy:
        affinityMode: "soft"     # soft | hard
        replicas: 3               # ensemble size (3 or 5)
        strategy: "random"        # see strategies table below
        customPlacement:         # only for strategy: custom
          - name: "replica-1"
            region: "region-1"
            az: "az-1"
            resourceGroup: "rg-1"

Node Selection Strategies

Woodpecker supports 8 node selection strategies for quorum placement:

StrategyConfig ValueDescription
RandomrandomRandom node selection from all available nodes (default)
Single AZ, Single RGsingle-az-single-rgAll nodes in the same availability zone and resource group
Single AZ, Multi RGsingle-az-multi-rgNodes in the same AZ but spread across different resource groups
Multi AZ, Single RGmulti-az-single-rgNodes across multiple AZs within the same resource group
Multi AZ, Multi RGmulti-az-multi-rgNodes spread across multiple AZs and resource groups (highest fault tolerance)
Cross Regioncross-regionNodes across different regions (requires 2+ buffer pools)
Random Grouprandom-groupPre-partitioned random groups for balanced distribution
CustomcustomExplicit placement rules via customPlacement configuration

Affinity Modes

ModeBehavior
softPrefer nodes matching the strategy constraints, but fall back to any available node if not enough matching nodes are found
hardStrictly enforce strategy constraints — fail if not enough matching nodes are available

Cluster Membership

Service mode uses HashiCorp memberlist (gossip protocol) for cluster membership management:

  • Nodes discover each other via seed addresses configured in buffer pools
  • Health status is propagated through gossip protocol
  • Failed nodes are automatically detected and excluded from quorum selection
  • New nodes can join the cluster dynamically by connecting to any seed
  • Each node exposes a bind port (gossip) and a service port (gRPC)

Object Storage

Cloud-native object storage as the durable storage layer for Woodpecker's WAL data.

Overview

Object storage serves as Woodpecker's primary durable storage layer. All WAL data is ultimately persisted to S3-compatible object storage, providing virtually unlimited capacity, high durability (11 nines), and cross-region replication capabilities.

ObjectStorage Interface

Woodpecker abstracts storage operations through a unified ObjectStorage interface:

Go Interface
type ObjectStorage interface {
    GetObject(ctx, bucket, key string, offset, size int64) (*Object, error)
    PutObject(ctx, bucket, key string, reader io.Reader, size int64) error
    PutObjectIfNoneMatch(ctx, bucket, key string, reader, size) error
    PutFencedObject(ctx, bucket, key string, reader, size, fenceId) error
    StatObject(ctx, bucket, key string) (*ObjectAttr, error)
    WalkWithObjects(ctx, bucket, prefix string, recursive bool, walkFn) error
    RemoveObject(ctx, bucket, key string) error
}

Supported Backends

Woodpecker supports multiple cloud providers via the minio.cloudProvider configuration:

ProviderConfig ValueCondition WriteNotes
Amazon S3awsSupported (If-None-Match)Default provider
Google GCSgcpSupportedS3-compatible mode
Aliyun OSSaliyunSupported (wrapped)Uses forbid-overwrite feature
Tencent COStencentSupported (wrapped)Uses forbid-overwrite feature
Azure BlobazureSupportedDedicated implementation
MinIOawsSupportedUses S3 API

Configuration

Object storage is configured via the storage and minio sections:

woodpecker.yaml
woodpecker:
  storage:
    type: default              # default(=minio), minio, local, service
    rootPath: /var/lib/woodpecker

minio:
  address: localhost
  port: 9000
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  useSSL: false
  bucketName: a-bucket
  createBucket: false
  rootPath: files
  useIAM: false
  cloudProvider: aws            # aws, gcp, aliyun, gcpnative
  region: ""
  useVirtualHost: false
  requestTimeoutMs: 10000
  listObjectsMaxKeys: 0        # 0 = unlimited

Condition Write Support

Condition write (PutObjectIfNoneMatch) is used as a fence mechanism to prevent split-brain in embedded mode. It ensures only one writer can create a given object key.

  • AWS/MinIO/GCS/Azure — Uses If-None-Match: * header for conditional PUT
  • Aliyun OSS / Tencent COS — Uses the forbid-overwrite feature (wrapped implementation)
  • Auto-detection — Woodpecker probes condition write support at startup (configurable via fencePolicy.conditionWrite)

Performance Best Practices

Write Optimization

  • The default maxFlushSize is 16MB — increase for larger batch uploads to reduce API call overhead
  • Default maxFlushThreads is 8 — increase for higher parallelism on fast networks
  • Use WriteAsync with batching for maximum throughput
  • Compaction merges small blocks into 32MB merged blocks (configurable via segmentCompactionPolicy.maxBytes)

Read Optimization

  • Read batch size is 16MB by default (segmentReadPolicy.maxBatchSize)
  • Up to 32 concurrent fetch threads for parallel reads (segmentReadPolicy.maxFetchThreads)
  • Range reads via GetObject with offset/size parameters for partial fetches

Data Durability

  • CRC32 checksums for data integrity verification on every block
  • Automatic retry on transient failures with configurable backoff (up to maxFlushRetries = 3, interval = 2s)
  • Fragment-based storage for incremental durability — data is durable as soon as each block is flushed
  • Segment compaction merges small fragments into larger objects for efficient storage and reads

Local File System Storage

High-performance disk-based log storage using memory-mapped I/O.

Overview

The local file system storage implements disk-based log storage using mmap (memory mapping) for efficient read and write performance. It maintains the Fragment concept for consistency with the object storage implementation.

File Layout

FragmentFile Layout
[Header (4K)] + [Data Area →] + [...Free Space...] + [← Index Area] + [Footer]

Header (4K)       Magic string and version information
Data Area         Actual log entries, growing forward from 4K
Free Space        Unused space between data and index
Index Area        Entry offsets, growing backward from file end
Footer            Metadata information

Data Entry Format:
[Payload Size (4B)] + [CRC32 (4B)] + [Actual Data (variable)]

Directory Structure

File Tree
/basePath/
  ├── log_[logID]/
  │   ├── fragment_[startOffset1]
  │   ├── fragment_[startOffset2]
  │   └── ...
  └── log_[logID2]/
      └── ...

Key Design Decisions

Why mmap?

  • Zero-copy read/write operations
  • Random access support
  • OS-managed page caching
  • Memory-level read/write performance

Why Fragments?

Although the local file system supports append operations, maintaining fragments provides:

  • Architecture consistency with object storage backend
  • File size limitation support
  • Segmented management and cleanup
  • Simplified concurrency control
  • Better system recovery capability

Performance Characteristics

AspectOptimization
WriteZero-copy via mmap, batch writes, async support, pre-allocated space
ReadIndex-based positioning, random reads, OS page cache, range reads
MemoryOn-demand loading via mmap, controlled fragment sizes, memory limits

Reliability

  • CRC32 checksum for data validation
  • Magic string verification in file header
  • Read-write locks for concurrent access control
  • Atomic operations for counter integrity
  • File locks to prevent concurrent writes

Staged Storage (Service Mode)

Two-tier hybrid storage combining local disk speed with object storage durability for service mode deployments.

Overview

Staged storage is the storage backend used in service mode (storage.type: service). It implements a two-tier hybrid architecture: active segments write to local disk for low-latency writes, then compaction uploads merged blocks to object storage for durable, long-term persistence.

Staged storage is automatically used when storage.type is set to service. No additional configuration is needed beyond configuring the MinIO/S3 backend for the object storage tier.

How It Works

Data Flow
Write Path:
  Client → LogStore Server → Local Disk (fast, low-latency)

Compaction:
  Local Disk → Merge Blocks → Object Storage (durable)

Read Path:
  Active segments  → Read from Local Disk
  Compacted data   → Read from Object Storage

Core Components

ComponentRole
StagedSegmentImplManages a segment's lifecycle across both tiers — coordinates writes to local disk and compaction uploads to object storage
StagedFileWriterHandles buffered writes to local disk files with block-based batching and CRC32 integrity checks
StagedFileReaderAdvReads entries from either local disk (active segments) or object storage (compacted segments) transparently

Configuration

Enable staged storage by setting storage.type: service:

woodpecker.yaml
woodpecker:
  storage:
    type: service              # enables staged storage
    rootPath: /var/lib/woodpecker # local staging directory

# Object storage tier (for compacted data)
minio:
  address: localhost
  port: 9000
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  bucketName: a-bucket
  rootPath: files

Benefits

  • Low write latency — Writes go to local disk first, avoiding object storage API overhead
  • High durability — Compaction ensures all data reaches object storage for long-term persistence
  • Efficient reads — Active data served from local disk; compacted data served from object storage with prefetching
  • Automatic lifecycle — Compaction runs automatically based on configured policies
  • Reduced API costs — Merging blocks before upload reduces the number of object storage API calls

Storage Backend Comparison

AspectObject Storage (MinIO/S3)Local File SystemStaged Storage
Config Valuedefault / miniolocalservice
ModeEmbeddedEmbedded (dev/test)Service
Write LatencyHigher (network)Lowest (local I/O)Low (local disk)
DurabilityHigh (cloud-replicated)Node-level onlyHigh (compacted to cloud)
CapacityUnlimitedDisk-limitedUnlimited (after compaction)
Best ForProduction embeddedDevelopment/testingProduction service mode

Condition Write & Fence

Split-brain prevention using condition write as a fence mechanism.

Overview

In embedded mode, Woodpecker uses condition write as a fence mechanism to prevent split-brain scenarios where a stale primary node resumes writing after being partitioned or crashed. This ensures data consistency and prevents concurrent write conflicts.

Supported Storage Backends

BackendSupportMechanism
MinIOFullIf-None-Match header
AWS S3FullConditional PUT operations
Azure BlobFullConditional writes
GCP Cloud StorageFullConditional operations
Aliyun OSSWrappedForbid overwrite feature
Tencent COSWrappedForbid overwrite feature

Detection Modes

Auto Mode (Default)

Follows existing true/false cluster metadata when present. If no metadata exists, verifies condition write support with strict enable semantics, stores true on success, and fails startup on failure.

Verification failure is surfaced immediately and is never persisted as an unsupported capability.

Enable Mode

Strict requirement: stored true is trusted, stored false is ignored, verification must succeed within 10 retries, and successful verification overwrites metadata to true.

Disable Mode

Skips detection entirely and uses distributed locks. Use when you know your backend doesn't support condition write.

Distributed Lock Fallback

When condition write is explicitly disabled, or when legacy auto metadata stores false, Woodpecker uses etcd-based distributed locks:

PropertyDetail
Lock Typeetcd concurrency.Mutex with concurrency.Session
Session TTL10 seconds (auto-renewed via keep-alive)
Lock Key/woodpecker/service/lock/{logName}
AcquisitionNon-blocking TryLock

Failover Behavior

  1. Session Expiration — etcd session expires after TTL (10s) if keep-alive fails
  2. Lock Release — Distributed lock is automatically released
  3. New Primary — New node acquires the lock
  4. Writer Switch — Applications reopen writer (up to 10s wait)

Performance Comparison

AspectCondition WriteDistributed Lock
Lock overheadNonePer-writer acquisition
Failover latencyImmediateUp to 10 seconds
ConcurrencyMultiple attempt, one winsSingle writer at a time

Configuration

woodpecker.yaml
woodpecker:
  logstore:
    fencePolicy:
      conditionWrite: "auto"   # "auto" | "enable" | "disable"

Monitoring & Metrics

Built-in Prometheus metrics and OpenTelemetry tracing for production observability.

Prometheus Metrics

Woodpecker exposes comprehensive Prometheus metrics for monitoring system health and performance. Each node exposes metrics on a dedicated port (default 9091–9094 for nodes 1–4).

Key Metrics

ModuleExample MetricsPurpose
Client Appendwoodpecker_client_append_latency, _requests_total, _bytesWrite QPS, latency P50/P99, message size distribution
Client Readwoodpecker_client_read_requests_total, _reader_bytes_readRead QPS, throughput, reader operation latency
Server LogStorewoodpecker_server_logstore_active_logs, _active_segmentsResource usage, segment lifecycle, operation success rate
Server File/Bufferwoodpecker_server_file_flush_latency, _compaction_latencyPersistence latency, compaction performance, buffer queuing
Object Storagewoodpecker_server_object_storage_operations_total, _operation_latencyS3 call frequency, latency P50/P99, bandwidth usage
System Resourceswoodpecker_server_system_cpu_usage, _memory_usage_ratioNode load, memory pressure, I/O bottleneck detection
gRPCgrpc_server_handled_total, _handling_secondsRPC QPS, success rate, error code distribution

OpenTelemetry Tracing

Woodpecker integrates with OpenTelemetry for distributed tracing. The deployment includes Jaeger (port 16686) for trace visualization. gRPC interceptors automatically propagate trace context in service mode.

Setting Up Monitoring Stack

Use the provided Docker Compose overlay to add Prometheus and Grafana to the cluster:

bash
# Start cluster with Prometheus + Grafana monitoring
cd deployments
docker compose -f docker-compose.yaml \
  -f ../tests/docker/monitor/docker-compose.monitor.yaml \
  -p woodpecker-monitor up -d

Access the monitoring services:

ServiceURLCredentials
Prometheushttp://localhost:9090—
Grafanahttp://localhost:3000Anonymous admin (no login)
Jaegerhttp://localhost:16686—
MinIO Consolehttp://localhost:9001minioadmin / minioadmin

Grafana Dashboards

Pre-built Grafana dashboard templates are available in tests/docker/monitor/grafana/templates/. These provide ready-to-use visualizations for write/read throughput, latency percentiles (P50/P95/P99), segment state distribution, storage backend performance, and error rates.

Run Monitoring Tests

Automated E2E metric verification tests are provided:

bash
# One-click: build, deploy, test, and cleanup
cd tests/docker/monitor
./run_monitor_tests.sh

# Keep cluster running after tests for manual inspection
./run_monitor_tests.sh --no-cleanup
The monitoring test suite also includes a rolling restart latency profile test that measures write/read latency across node restarts — useful as a baseline for evaluating rolling upgrade quality.