Two complementary guides for operating Woodpecker: a command-by-command reference and incident response workflows.
These pages include the local CLI additions beyond v0.1.46. Build wp from the current checkout for metadata repair, raw etcd access, truncation and the improved diagnostics.
You need a command’s syntax, parameters, examples, output interpretation or limitations. Covers environment, runtime progress, data diagnosis and metadata.
You are diagnosing read/write stalls or data corruption and need step-by-step recovery guidance.
Start with evidence
Confirm the selected context and address mappings, then distinguish runtime health, metadata consistency and actual data readability. The guides explain what each result can establish before an intervention.
CLI Handbook · Overview
Find an operation by its command group, understand its parameters, and interpret its output.
The established operations are available in v0.1.47; Interactive shell and native diagnosis are source-build additions and are not included in v0.1.47. This reference is generated from source. Check wp version and wp --help against your binary before following an example.
Each topic has command syntax, parameter explanations, a demo, illustrative output and result interpretation. The overview lists executable operations: node is a command group, while node list is an operation within it.
All operations at a glance
Search by command, purpose or parameter, then click an operation to open its topic at the matching detail. Connection parameters are explained in CLI setup.
Force compaction on a segment at an exact confirmed entry ID
<node>: Discovered node ID --expected-last-entry-id int: Exact LastEntryId from coordinator metadata; an incorrect value can publish an invalid footer (-1 only for an empty segment, required) --log int: Log ID --seg int: Segment ID
Declare an entry range unreadable so readers move past it.
<logName>: Logical log name in metadata <segmentId>: Numeric segment ID within that log --reason string: Why these entries are being given up --from-entry / --to-entry: inclusive entry range --reason: accepted-loss reason; -y commits the reviewed declaration
Run one check; no implicit execution of other checks
<name>: Local CLI context name
Connection and naming
Follow CLI setup for a Helm-installed Milvus/Woodpecker deployment, including access from outside Kubernetes.
To retain connection settings while running several operations, use Interactive shell. Select the service context first, then explicitly attach the tenant's etcd metadata and optional storage location. The shell page explains which commands are available at each stage.
Name
Meaning
logName / logId
Logical metadata name / numeric log identity. Node-local logstore operations use the numeric ID.
meta-prefix
Complete etcd namespace configured by the client, for example milvus-prod/woodpecker.
bucket + root-path
The physical storage instance. Use both on scan, inspect and probe when automatic discovery cannot locate an empty or unloaded target. The instance data command uses --root.
Admin discovery supplies quorum identities; the workstation fans out to each node. Map every target to an accessible admin origin. A working seed connection alone does not establish peer reachability.
If the metadata namespace is unknown, start with Raw etcd exploration: list literal keys, narrow the prefix, then inspect a value. Those read-only operations need an etcd connection but no admin endpoint or metadata prefix.
For decoded records, metadata consistency and controlled correction, use Metadata inspection & repair. Normal retention advancement uses log truncate; it is separate from metadata patching and skip-range declarations.
Choose the intended change
Operation
Intent / required preparation
meta log/segment patch
Correct proven metadata errors using a JSON file containing only supported fields. Apply needs stopped writers/auditors, target revision, reason and a new backup.
log skip-range add
Accept a precisely verified unreadable range when all usable alternatives fail. Requires explicit loss acceptance; it does not repair files or compaction.
log truncate
Advance verified application retention. Apply needs the log revision, reason and a new backup; later auditor rounds retire earlier finalized segments.
Preview success does not commit a change. Metadata patch/truncate previews normally exit 0; skip-range and fencing previews can exit 7 because acceptance was not given. Verify application recovery independently after any intervention.
Shared parameters
Parameter / type
Use and default
--admin-port int
Admin port for fan-out peer discovery (default 9091)
--concurrency int
Fan-out concurrency (default 8)
--context string
CLI context name (overrides current-context in cli.yaml)
--endpoint string
Admin HTTP seed endpoint (e.g. http://node:9091)
-h, --help
help for wp
--no-color
Disable color in output
--node-admin-url stringArray
Map a node ID, advertised address, or host to an admin URL (KEY=URL; repeatable; overrides context mappings)
Output formats vary by operation. Profile downloads binary data, and some diagnostic operations return findings with a nonzero exit code. Read the specific command's help and interpretation rather than treating every successful request as a health check.
The source-build wp diagnose entry point provides versioned native WP findings with explicit snapshot/live-window coverage, counter reset checks, registry limits and conflicting same-log progress. It has two implemented stall rules; exit 0 means evaluation completed, including inconclusive outcomes. Use its JSON evidence and original context/node mappings for follow-up. Use wp diag for retained Prometheus/Jaeger checks and wp webui for the local operations workspace. These inspect Woodpecker itself; Milvus-owned and Loki diagnosis are outside the tool.
Metrics scenario limits
stuck-read and stuck-write require persistent matching operations as well as zero successful completion progress during the window. An idle cluster alone is not a stall. Latency rules currently use window-average histogram latency rather than the advertised p99/baseline model. Several scenario names have only partial evaluators; use counters, operation IDs, reader positions and replica evidence together. A scenario reporting no findings is not proof that the whole cluster is healthy.
Exit codes and expected findings
Code
Meaning
0
Completed within the command's stated scope.
1 / 2 / 3 / 4
Network / usage / target absent / state conflict.
5 / 6 / 7
Wait timeout / strict partial failure / acceptance not given or aborted.
8 / 9
Yellow / red diagnostic finding.
10 / 11 / 12 / 13
Not implemented / resource absent / configuration / prerequisite error.
100 + N
Executed kubectl exit code N.
Read the exit status immediately with echo $?. Piping through another command can hide the original status unless pipefail is enabled. Example output throughout the handbook is illustrative and may omit fields; replace names and IDs with values from your own cluster.
Configure wp and run your first commands from outside Kubernetes.
0. Install wp
Assume Milvus with Woodpecker is already installed by Helm. Run wp on your workstation using a released binary or a source build. The shell examples below use bash/zsh on macOS or Linux.
Download from GitHub
Select the release and OS/CPU architecture on the GitHub Releases page. These are standalone binaries, not archives. For example, macOS on Apple Silicon:
For Linux, use wp-linux-amd64 or wp-linux-arm64; Windows uses wp-windows-amd64.exe. Use an available asset matching your machine.
Alternative: build from source
Example / output
git clone https://github.com/zilliztech/woodpecker.git
cd woodpecker
# Optionally check out the release/branch you want to build.
# git checkout v0.1.47
make wpcli
mkdir -p "$HOME/.local/bin"
cp bin/wp "$HOME/.local/bin/wp"
export PATH="$HOME/.local/bin:$PATH"
wp version
Install Git, make and the Go version required by go.mod first. You may also run ./bin/wp directly. A source build exposes the commands in that checkout; a released binary may not contain later development additions documented in this handbook. Check wp --help.
1. Connect to the Helm deployment
Use your real kubeconfig and namespace. Find the Woodpecker and etcd pods:
Example / output
export KUBECONFIG=/path/to/your/kubeconfig
kubectl -n milvus get pods -o wide
The examples below assume namespace milvus, Woodpecker pods milvus-woodpecker-0/1/2 and etcd pod milvus-etcd-0. Replace these names with the Helm deployment's actual names. Start each needed forward in its own terminal:
The seed discovers membership; it does not proxy other nodes. Inspect the advertised node IDs/addresses, then map every node you need to reach:
Example / output
wp cluster info --endpoint http://127.0.0.1:19091
2. Save the connection in cli.yaml
The usual personal location is ~/.woodpecker/cli.yaml. Create the directory and save this example YAML there, using the node IDs reported by your cluster:
endpoint is the admin seed. node_admin_urls maps discovered node IDs or advertised service addresses to reachable admin origins. current-context selects milvus-prod when --context is omitted:
wp node list
wp --context milvus-prod node list
To keep cli.yaml in another directory, select it explicitly:
Example / output
export WOODPECKER_CLI_CONFIG=/path/to/your/cli.yaml
wp ctx list
wp --context milvus-prod ctx view
Config locations and precedence
Search order: WOODPECKER_CLI_CONFIG → cli.yaml beside the resolved wp binary → $XDG_CONFIG_HOME/woodpecker/cli.yaml → ~/.woodpecker/cli.yaml. The working directory and /etc/woodpecker are not default locations. Explicit command flags override matching context fields. WOODPECKER_ENDPOINT overrides the context's endpoint unless --endpoint is given.
The k8s block supplies defaults to wp k8s helpers, not to your shell's kubectl. Operator-specific helpers target a WoodpeckerCluster CR; for a Helm deployment without that CR, use kubectl directly. Do not use the Helm release name as k8s.cluster unless it is also the actual CR name.
3. Run without cli.yaml, or add shortcuts
For a built-in interactive entry point, see Interactive shell: it requires a service context and retains an explicitly connected tenant for subsequent commands. The examples below describe ordinary bash/zsh shortcuts and one-shot CLI invocations.
A config file is optional. Supply the admin endpoint and node mappings directly:
Replace node IDs with the discovered IDs, full advertised addresses or hostnames. Repeat --node-admin-url for each peer; --endpoint alone only makes the seed reachable. CLI flags can also override an existing cli.yaml for one invocation.
Optional: source shell functions for shorter commands
After creating the milvus-prod context, save this as a file of your choice, for example ~/wp-env.sh:
Example / output
export KUBECONFIG=/path/to/your/kubeconfig
# Set this only when cli.yaml is outside the usual personal location:
# export WOODPECKER_CLI_CONFIG=/path/to/your/cli.yaml
k() { kubectl --kubeconfig "$KUBECONFIG" -n milvus "$@"; }
w() { wp --context milvus-prod "$@"; }
Example / output
source ~/wp-env.sh
k get pods
w node list
w cluster health
source defines the functions in the current shell. w forwards arguments to wp and selects milvus-prod for each invocation; it does not change current-context in the file.
4. Configure etcd for metadata queries
Current source builds keep client metadata authority outside cli.yaml. Supply reachable etcd endpoints and the full client metadata prefix explicitly, or use a private --client-parameters-file. Older context-etcd/client-profile fields are not selected; there is no Admin-config discovery fallback.
Parameters
Use
--etcd
Reachable etcd endpoints; use commas for multiple addresses.
--meta-prefix
Complete Woodpecker metadata prefix configured by the Milvus client. The example below uses milvus-prod/woodpecker; use the actual prefix, not an assumed Helm release name.
--etcd-username / --etcd-password
Etcd authentication credentials when required.
--etcd-cacert
CA certificate file on the workstation.
--etcd-cert / --etcd-key
Workstation client certificate and private-key files.
--etcd-tls-min-version
Minimum TLS version: 1.0, 1.1, 1.2 or 1.3.
For example, inspect registrations for a real Milvus log name using explicit connection settings:
Yes: they look up log/segment/quorum or operational metadata. Supply endpoint and complete prefix through explicit flags or a private client parameter file. meta and truncate need no admin connection when both etcd settings are supplied.
etcd list/get/cat
Etcd endpoints and any required TLS/auth settings only. No admin discovery or metadata prefix: keys/prefixes are literal. See Raw etcd exploration.
Certificate paths must exist on the workstation. An explicitly selected private client parameter file can contain these fields, independently of server contexts:
Example / output
endpoints: [http://127.0.0.1:32379]
meta_prefix: tenant/woodpecker
username: operator
password: YOUR_PASSWORD
tls_enabled: true
cacert: /path/to/ca.crt
# cert/key are a pair, required only for mutual TLS:
cert: /path/to/client.crt
key: /path/to/client.key
tls_min_version: "1.2"
Use a CA alone for server-authenticated TLS, or tls_enabled: true with system trust roots. Supplying cert/key enables mutual TLS; they must be supplied together. The current source default minimum TLS version is 1.2. Explicit flags override the private parameter file. ctx view shows server connection settings; client metadata parameters are supplied separately.
5. Check the connection and interpret the result
Example / output
wp ctx list
wp ctx view
wp node list
wp cluster health
Illustrative excerpts:
Example / output
NAME ENDPOINT ACTIVE
milvus-prod http://127.0.0.1:19091 *
wp: config /home/alice/.woodpecker/cli.yaml · context milvus-prod · endpoint http://127.0.0.1:19091
NAME STATE HEALTH
milvus-woodpecker-0 active OK
milvus-woodpecker-1 active OK
milvus-woodpecker-2 active OK
Cluster Health: YELLOW
active: 3 decommissioning: 0 unreachable: 0
- zero redundancy (active == N)
The star marks the saved default context; ctx view shows resolved server settings; client metadata is a separate request scope. Three OK node responses confirm all three admin origins are reachable. YELLOW with active=3 and expected-replicas=3 means no spare capacity, not a connection failure.
If the seed refuses the connection, check its port-forward. If only one peer fails, check that peer's forward and mapping. If a metadata target is absent, check the actual prefix and log name. Use node IDs from node list and log/segment IDs reported by the application or log scan.
Interactive shell
For context selection, tenant connection parameters and command availability, open the dedicated Interactive shell handbook.
wp completion bash
Generate the autocompletion script for the bash shell.
Example / output
wp completion bash
Parameters to know
Parameter / type
Use and default
--no-descriptions
disable completion descriptions
Demo
Example / output
wp completion bash > wp-completion.bash
Output and interpretation
Example / output
Generated shell completion script
Generation only writes stdout; it does not edit shell startup files. Load it using your shell’s completion mechanism. For zsh: initialize compinit, then source the generated file.
Finding or error example
Example / output
Shell setup does not load the script
Complete command help
Example / output
Generate the autocompletion script for the bash shell.
This script depends on the 'bash-completion' package.
If it is not installed already, you can install it via your OS's package manager.
To load completions in your current shell session:
source <(wp completion bash)
To load completions for every new session, execute once:
#### Linux:
wp completion bash > /etc/bash_completion.d/wp
#### macOS:
wp completion bash > $(brew --prefix)/etc/bash_completion.d/wp
You will need to start a new shell for this setup to take effect.
Usage:
wp completion bash
Flags:
-h, --help help for bash
--no-descriptions disable completion descriptions
wp completion fish
Generate the autocompletion script for the fish shell.
Example / output
wp completion fish [flags]
Parameters to know
Parameter / type
Use and default
--no-descriptions
disable completion descriptions
Demo
Example / output
wp completion fish > wp-completion.fish
Output and interpretation
Example / output
Generated shell completion script
Generation only writes stdout; it does not edit shell startup files. Load it using your shell’s completion mechanism. For zsh: initialize compinit, then source the generated file.
Finding or error example
Example / output
Shell setup does not load the script
Complete command help
Example / output
Generate the autocompletion script for the fish shell.
To load completions in your current shell session:
wp completion fish | source
To load completions for every new session, execute once:
wp completion fish > ~/.config/fish/completions/wp.fish
You will need to start a new shell for this setup to take effect.
Usage:
wp completion fish [flags]
Flags:
-h, --help help for fish
--no-descriptions disable completion descriptions
wp completion powershell
Generate the autocompletion script for powershell.
Generation only writes stdout; it does not edit shell startup files. Load it using your shell’s completion mechanism. For zsh: initialize compinit, then source the generated file.
Finding or error example
Example / output
Shell setup does not load the script
Complete command help
Example / output
Generate the autocompletion script for powershell.
To load completions in your current shell session:
wp completion powershell | Out-String | Invoke-Expression
To load completions for every new session, add the output of the above command
to your powershell profile.
Usage:
wp completion powershell [flags]
Flags:
-h, --help help for powershell
--no-descriptions disable completion descriptions
wp completion zsh
Generate the autocompletion script for the zsh shell.
Example / output
wp completion zsh [flags]
Parameters to know
Parameter / type
Use and default
--no-descriptions
disable completion descriptions
Demo
Example / output
wp completion zsh > wp-completion.zsh
Output and interpretation
Example / output
Generated shell completion script
Generation only writes stdout; it does not edit shell startup files. Load it using your shell’s completion mechanism. For zsh: initialize compinit, then source the generated file.
Finding or error example
Example / output
Shell setup does not load the script
Complete command help
Example / output
Generate the autocompletion script for the zsh shell.
If shell completion is not already enabled in your environment you will need
to enable it. You can execute the following once:
echo "autoload -U compinit; compinit" >> ~/.zshrc
To load completions in your current shell session:
source <(wp completion zsh)
To load completions for every new session, execute once:
#### Linux:
wp completion zsh > "${fpath[1]}/_wp"
#### macOS:
wp completion zsh > $(brew --prefix)/share/zsh/site-functions/_wp
You will need to start a new shell for this setup to take effect.
Usage:
wp completion zsh [flags]
Flags:
-h, --help help for zsh
--no-descriptions disable completion descriptions
wp ctx list
List all contexts defined in cli.yaml
Example / output
wp ctx list [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Demo
Example / output
wp ctx list
Output and interpretation
Example / output
NAME ENDPOINT ACTIVE
milvus-prod http://127.0.0.1:19091 *
list and view inspect local CLI configuration. use persists current-context; --context selects one invocation without changing it. No server configuration is changed.
Finding or error example
Example / output
Context does not exist / config parse or write error
Complete command help
Example / output
List all contexts defined in cli.yaml
Usage:
wp ctx list [flags]
Flags:
-h, --help help for list
wp ctx use
Switch current-context to the given name and persist to cli.yaml
Example / output
wp ctx use <name> [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Demo
Example / output
wp ctx use milvus-prod
Output and interpretation
Example / output
Switched to context "milvus-prod"
list and view inspect local CLI configuration. use persists current-context; --context selects one invocation without changing it. No server configuration is changed.
Finding or error example
Example / output
Context does not exist / config parse or write error
Complete command help
Example / output
Switch current-context to the given name and persist to cli.yaml
Usage:
wp ctx use <name> [flags]
Flags:
-h, --help help for use
wp ctx view
Print the resolved fields of the active context
Example / output
wp ctx view [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
list and view inspect local CLI configuration. use persists current-context; --context selects one invocation without changing it. No server configuration is changed.
Finding or error example
Example / output
Context does not exist / config parse or write error
Complete command help
Example / output
Print the resolved fields of the active context
Usage:
wp ctx view [flags]
Flags:
-h, --help help for view
wp report analyze
Explicitly ask the configured Agent to explain supplied findings
Example / output
wp report analyze <report.json|-> [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Demo
Example / output
wp --context local report analyze diagnosis.json --config /path/to/cli.yaml
Output and interpretation
Example / output
The explicitly configured backend-local Agent explains abnormal findings.
This explicitly invokes a model with supplied evidence. It cannot automatically query sources, run commands or repair the cluster.
Finding or error example
Example / output
No Agent configuration, no findings or a model failure leaves the original report unchanged.
Complete command help
Example / output
Explicitly ask the configured Agent to explain supplied findings
Usage:
wp report analyze <report.json|-> [flags]
Flags:
-h, --help help for analyze
wp report export
Copy original JSON or derive a compact findings packet
Example / output
wp report export <report.json|-> [flags]
Parameters to know
Parameter / type
Use and default
--packet
Export findings, supporting context, gaps and provenance; no model call
Exports a bounded findings packet with original report provenance.
Without --packet, copies the original report. It does not rerun checks.
Finding or error example
Example / output
Malformed or oversized reports are rejected.
Complete command help
Example / output
Copy original JSON or derive a compact findings packet
Usage:
wp report export <report.json|-> [flags]
Flags:
-h, --help help for export
--packet Export findings, supporting context, gaps and provenance; no model call
wp report inspect
Show recorded evidence detail without querying sources
Example / output
wp report inspect <report.json|-> <check> [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Diagnostics finds issues; Operations confirms and handles them; Command Explorer exposes native commands. No diagnostic action automatically repairs the cluster.
Finding or error example
Example / output
A missing or invalid cli.yaml prevents startup.
Complete command help
Example / output
Open a local operations and diagnosis workspace using cli.yaml
Usage:
wp webui [flags]
Flags:
--config string Shared cli.yaml (default: normal wp configuration discovery)
-h, --help help for webui
--listen string Loopback host:port (default "127.0.0.1:8794")
--metrics-rules string Metrics rules file beside cli.yaml by default
--trace-rules string Trace rules file beside cli.yaml by default
Select a service context, explicitly connect tenant metadata and storage, and run operations in one interactive session.
Source-build feature: wp shell is not included in v0.1.47. Build a checkout containing the command with make wpcli and run ./bin/wp shell --help to check availability. Examples below use wp from that build. Output is illustrative; replace names, prefixes and storage locations with those of your deployment.
Keep one terminal open while operating a multi-tenant Woodpecker service. Select the service once, then explicitly connect the tenant whose metadata you want to inspect. Inside the session, omit the leading wp; existing operation arguments, previews and confirmations still apply.
1. Configure and select the service context
A service context is required. For a Helm installation accessed from outside Kubernetes, first establish the Admin port forwards described in CLI setup. Save this example as ~/.woodpecker/cli.yaml:
endpoint is the Admin seed. node_admin_urls maps the node IDs or advertised addresses returned by your cluster to workstation-accessible Admin origins. A reachable seed alone does not make the other replicas reachable. The staging origin above is a placeholder; configure its actual mappings when required.
Example / output
wp shell --context milvus-prod
For a configuration file in another location, set the environment variable in the same terminal before starting:
Without --context, an interactive wp shell always displays a context selector, even when a current context exists. Enter a displayed number or exact name; Enter accepts a valid current-context. Piped input requires an explicit --context. Missing or invalid configuration reports the relevant configuration path and setup instructions. See CLI setup for the full configuration search order.
Example / output
Service context: milvus-prod
Config file: /Users/example/.woodpecker/cli.yaml
Admin endpoint: http://127.0.0.1:19091
Node mappings: 3
Tenant metadata: not configured
wp[context=milvus-prod | meta=disconnected]>
The prompt identifies two independent targets: context is the Woodpecker service cluster; meta is the selected tenant connection, or disconnected. The latter means no tenant is configured, not a network health verdict. Entering the shell does not contact the services or automatically attach the context's etcd settings. The shell pins its chosen service endpoint rather than inheriting WOODPECKER_ENDPOINT.
2. Connect the tenant: etcd, metadata prefix and storage
From the interactive prompt, connect using the tenant's workstation-accessible etcd endpoint and complete Woodpecker metadata prefix. Keep this example on one line when entering it:
Required operator-selected connection name, displayed as meta=tenant-a. It is an alias; it does not derive or change the actual metadata prefix.
--etcd
Required reachable etcd endpoint; comma-separate multiple endpoints. Outside Kubernetes, use the forwarded or otherwise reachable address.
--meta-prefix
Required complete metadata namespace configured by the client, for example tenant-a/woodpecker. Do not supply a particular log key or confuse it with the physical storage root.
--bucket
Physical storage bucket for this tenant. Optional for metadata-only operations; provide together with --root-path.
--root-path
Physical storage instance root, for example milvus/wp. Provide together with --bucket; it need not equal the metadata prefix.
--etcd-username / --etcd-password
Credentials when etcd authentication is enabled. Each new connection starts with fresh settings; omitted credentials are not inherited from the previous tenant.
TLS endpoint
Use an https:// etcd endpoint or supply certificate flags to enable TLS. There is no separate --etcd-tls switch in this interface.
--etcd-cacert
Local CA certificate file when needed. Certificate flags enable TLS.
--etcd-cert / --etcd-key
Local client certificate and private key for mutual TLS; provide both.
--etcd-tls-min-version
Minimum TLS version when TLS is enabled: 1.0, 1.1, 1.2 or 1.3; default 1.2.
Example / output
Service context: milvus-prod
Tenant: tenant-a
Etcd endpoints: 127.0.0.1:32379
Meta prefix: tenant-a/woodpecker
Bucket: tenant-a-bucket
Root path: milvus/wp
wp[context=milvus-prod | meta=tenant-a]>
Success verifies an etcd read under the supplied prefix, even if it is empty. It does not establish that you selected the intended tenant, that its quorum belongs to this service context, or that its data is healthy. Check status and the returned metadata before operating. connect --help lists the complete connection flags.
Metadata-only and TLS connection examples
Example / output
# Metadata-only connection; no bucket/root-path is needed.
connect tenant-a --etcd 127.0.0.1:32379 --meta-prefix tenant-a/woodpecker
# Custom CA and client certificates; use files on this workstation.
connect tenant-a --etcd etcd.example.com:2379 --meta-prefix tenant-a/woodpecker --etcd-cacert /path/to/ca.pem --etcd-cert /path/to/client.pem --etcd-key /path/to/client-key.pem
Each successful connect replaces the whole tenant connection. A metadata-only reconnect also clears a previous bucket/root-path. A failed connection retains the previous selected tenant and reports that the selection is unchanged.
3. Which commands can run now?
Session state
Available operations
Examples inside the shell
Service context only meta=disconnected
Service runtime and node diagnostics: cluster/node, environment, configuration, logging, profiles, in-flight operations, metrics, local instance inventory and node-local logstore operations. Kubernetes helpers use the service context's Kubernetes settings. Required services/tools must still be reachable.
node list cluster health config show node-0 logstore segments node-0 instance data node-0
Service context + tenant etcd and metadata prefix
All meta, etcd, and marking operations; log readers, log skip-range, log truncate; quorum-aware logstore lac and logstore fence-quorum. Mutations retain their existing validation, acceptance and backup requirements.
meta log list meta check application-log log readers application-log log skip-range list application-log etcd list tenant-a/woodpecker/
Tenant connection + bucket and root-path
log scan, segment inspect, and segment probe inherit the selected data location. In interactive mode these three require both location values, either from connect or explicit command flags.
Local session inspection, switching, help and exit. Operation help is available before a tenant is attached.
status ctx list help meta log connect --help exit
Connecting makes the required parameters available; it does not make every operation succeed. Physical reads also require reachable quorum node Admin endpoints, and mutation commands still require their operation-specific evidence and safeguards. Raw etcd commands use the literal key/prefix you supply; --meta-prefix does not automatically scope or prepend raw etcd paths.
Example / output
meta log list
log readers application-log
log scan application-log --mode raw
segment inspect application-log 0
# Override data location for this one operation only.
segment probe application-log 0 --bucket another-bucket --root-path another/root
Use actual log names from meta log list and segment IDs from meta segment list application-log. A per-command override does not modify the next command's defaults. When a tenant storage location is attached, instance data and node log-health also inherit it as a filter (their root flag is --root). They remain usable without a tenant connection. This shell's explicit storage-location rule does not change the existing non-interactive CLI's discovery behavior.
status reads local session settings without making network requests, so it works while services are down or no tenant is attached. When credentials are configured, it reports that authentication is configured without printing them. An empty bucket/root-path means physical inspection needs explicit values.
Example / output
status --check
Example / output
Admin check: OK
Etcd check: OK
The check form also prints the session configuration, then performs bounded Admin and etcd checks. Read the two results separately. When disconnected it prints Etcd check: skipped (no tenant configured); a failed Admin check does not hide the separate etcd result. Neither check verifies application progress or physical data integrity.
5. Disconnect or switch a service/tenant
Example / output
disconnect
status
use milvus-staging
connect tenant-b --etcd 127.0.0.1:42379 --meta-prefix tenant-b/woodpecker
status
disconnect clears the entire tenant connection, including credentials, TLS, metadata prefix and storage location. It retains the service context. A successful use <context> switches the service and clears any tenant attachment; connect the intended tenant again. A failed context selection leaves the current service and tenant unchanged.
Example / output
wp[context=milvus-prod | meta=tenant-a]> disconnect
Tenant metadata: not configured
wp[context=milvus-prod | meta=disconnected]> use milvus-staging
Service context: milvus-staging
Config file: /Users/example/.woodpecker/cli.yaml
Admin endpoint: https://wp-staging.example.com:9091
Node mappings: 0
Tenant metadata: not configured
wp[context=milvus-staging | meta=disconnected]>
Session-local switching does not write current-context or affect another terminal. Inside the session, ctx list and ctx view inspect saved CLI configuration; use status to inspect the active tenant attachment. Use use for session-local switching. The existing ctx use command is a persistent change to the original configuration file and does not retarget this running session; invoke it only when you intend to change the default on disk. Exit and start a new shell to reload an edited configuration file.
6. Interpret missing configuration and connection failures
If a metadata-dependent command runs while disconnected, it fails before contacting Admin or etcd, even if the service context contains etcd settings or you pass connection flags on that operation:
Example / output
wp[context=milvus-prod | meta=disconnected]> meta log list
wp shell: Prerequisite: no tenant metadata connection configured.
Connect with: connect <tenant> --etcd <host:port> --meta-prefix <prefix>.
Physical inspection also needs --bucket <bucket> --root-path <root-path>.
Run status to view the session
Result
Meaning / next step
no tenant metadata connection configured
Run connect, then retry. Service-only commands and help remain available.
physical inspection requires --bucket ... and --root-path ...
Supply the paired location flags on scan/inspect/probe, or reconnect the tenant with both. No physical inspection has started.
tenant connection failed ...; active tenant unchanged: ...
Check endpoint forwarding, authentication and local TLS files. The previous tenant remains selected, or the session stays disconnected. Run status before continuing.
Admin check: failed ... / Etcd check: failed ...
Configuration exists but that check could not succeed. A timeout or authentication failure is different from a missing connection configuration; the selected context/tenant remains intact.
7. Help, completion and exit
Example / output
help
connect --help
help segment inspect
ctx list
exit
Tab completes command/flag names and context names after use. Arrow keys recall in-memory history; connect and credential-bearing commands are excluded from history. Ctrl+C cancels the current operation and returns to the prompt. Use exit, quit or Ctrl+D to leave.
This is a command session, not bash/zsh: quoted arguments work, but environment-variable expansion, pipes and redirection are not evaluated. Run exports and port forwards in your ordinary terminal. Enter each example operation as a separate command. Existing command findings can return errors while the shell remains open for further diagnosis.
wp shell
Open an interactive operational shell. Without --context, choose a context from cli.yaml.
Example / output
wp shell [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Demo
Example / output
wp shell --context milvus-prod
# Inside the interactive session:
status
connect tenant-a --etcd 127.0.0.1:32379 --meta-prefix tenant-a/woodpecker --bucket tenant-a-bucket --root-path milvus/wp
meta log list
Output and interpretation
Example / output
wp[context=milvus-prod | meta=disconnected]>
# After connect succeeds:
wp[context=milvus-prod | meta=tenant-a]>
The service context is required; attaching a tenant is explicit and optional. Service commands work while disconnected. Metadata operations require connect, and interactive scan/inspect/probe also require bucket/root-path. status shows local selection; status --check checks reachability. Switching service contexts clears the tenant connection. This source-build feature is not included in v0.1.47.
Finding or error example
Example / output
wp shell: Prerequisite: no tenant metadata connection configured. Connect with: connect <tenant> --etcd <host:port> --meta-prefix <prefix>. Run status to view the session
Complete command help
Example / output
Open an interactive operational shell. Without --context, choose a context from cli.yaml.
Start with no tenant metadata attached; use connect to attach a tenant.
For piped input, specify --context explicitly. Type help for commands or exit to leave.
Usage:
wp shell [flags]
Flags:
-h, --help help for shell
Compare build, Go runtime and deployment environment. Differences are evidence to investigate, not automatic data corruption.
Finding or error example
Example / output
Error: TargetNotFound: node-missing
Complete command help
Example / output
Show env vars + runtime + host + build for a single node
Usage:
wp env show <node> [flags]
Flags:
-h, --help help for show
--section string section filter: env|runtime|host|build|all (default "all")
GREEN requires active > N, no unreachable/decommissioning nodes and no version skew. RED: active < N or any node unreachable. YELLOW: active == N, decommissioning or mixed versions. The default N=3 is a baseline, not inferred per-log quorum health.
Finding or error example
Example / output
Cluster Health: YELLOW
active: 3
zero redundancy (active == N)
Complete command help
Example / output
Red/yellow/green health check for the cluster
Usage:
wp cluster health [flags]
Flags:
--expected-replicas int expected replica count baseline (default 3) (default 3)
-h, --help help for health
wp cluster info
Show cluster overview (nodes, state, topology)
Example / output
wp cluster info [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Demo
Example / output
wp cluster info
Output and interpretation
Example / output
members: [node-0, node-1, node-2]
Membership is the seed’s current view, not proof of data availability. Compare peer views with gossip-diff.
Finding or error example
Example / output
Network: seed unavailable
Complete command help
Example / output
Show cluster overview (nodes, state, topology)
Usage:
wp cluster info [flags]
Flags:
-h, --help help for info
wp node cancel-decommission
Cancel an in-progress decommission (decommissioning -> active)
Example / output
wp node cancel-decommission <node> [flags]
Parameters to know
Parameter / type
Use and default
-y, --yes
skip interactive confirmation
Demo
Example / output
wp node cancel-decommission node-0
Output and interpretation
Example / output
decommission cancelled on node-0; returning to active
Cancels the lifecycle operation; re-check node state and application progress.
Finding or error example
Example / output
Error: StateConflict: ...
Complete command help
Example / output
Cancel an in-progress decommission (decommissioning -> active)
Usage:
wp node cancel-decommission <node> [flags]
Flags:
-h, --help help for cancel-decommission
-y, --yes skip interactive confirmation
wp node decommission
Trigger graceful node decommission (blocks until safe by default)
Example / output
wp node decommission <node> [flags]
Parameters to know
Parameter / type
Use and default
--async
return immediately instead of waiting for safe_to_terminate
--heartbeat-interval duration
heartbeat line interval when no progress change (default 15s)
--interval duration
progress poll interval (default 3s)
--timeout duration
overall timeout for the wait (default 30m0s)
-y, --yes
skip interactive confirmation
Demo
Example / output
wp node decommission node-0 --async
Output and interpretation
Example / output
decommission started on node-0
# --async returns before drain completion; follow with drain-status.
Changes node lifecycle state. Check capacity first; --async returns after request rather than completion. Timeout does not authorize termination.
Trigger graceful node decommission (blocks until safe by default)
Usage:
wp node decommission <node> [flags]
Flags:
--async return immediately instead of waiting for safe_to_terminate
--heartbeat-interval duration heartbeat line interval when no progress change (default 15s)
-h, --help help for decommission
--interval duration progress poll interval (default 3s)
--timeout duration overall timeout for the wait (default 30m0s)
-y, --yes skip interactive confirmation
wp node drain-status
Show (or watch) decommission progress for a node
Example / output
wp node drain-status <node> [flags]
Parameters to know
Parameter / type
Use and default
--interval duration
watch poll interval (default 3s)
--timeout duration
watch total timeout (default 30m0s)
-w, --watch
continuously watch until safe_to_terminate
Demo
Example / output
wp node drain-status node-0
Output and interpretation
Example / output
safe_to_terminate: true
Terminate only after safe_to_terminate=true. active means decommission has not completed; processors and local data independently prevent a safe drain. -w polls.
Finding or error example
Example / output
state: active
remaining_processors: 1
has_local_data: true
safe_to_terminate: false
Complete command help
Example / output
Show (or watch) decommission progress for a node
Usage:
wp node drain-status <node> [flags]
Flags:
-h, --help help for drain-status
--interval duration watch poll interval (default 3s)
--timeout duration watch total timeout (default 30m0s)
-w, --watch continuously watch until safe_to_terminate
wp node healthz
Query a node's health probe (the one readiness uses)
Example / output
wp node healthz <node> [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Demo
Example / output
wp node healthz node-0
Output and interpretation
Example / output
Health probe on node-0: ok
Process health/readiness does not establish log read/write progress or replica data integrity.
Finding or error example
Example / output
Health probe on node-0: failed
# Read individual dependency/check codes in the report.
Complete command help
Example / output
Query a node's health probe (the one readiness uses)
Usage:
wp node healthz <node> [flags]
Flags:
-h, --help help for healthz
wp node list
List all server nodes in the cluster
Example / output
wp node list [flags]
Parameters to know
Parameter / type
Use and default
--az string
filter by availability zone
--region string
filter by region
--rg string
filter by resource group
--state string
filter by state: active|decommissioning|decommissioned|unreachable|all (default "all")
Demo
Example / output
wp node list --strict
Output and interpretation
Example / output
NAME STATE HEALTH
milvus-woodpecker-0 active OK
milvus-woodpecker-1 active OK
Use node IDs from this list in node-scoped commands; distinguish membership from HTTP reachability.
Finding or error example
Example / output
milvus-woodpecker-2 unreachable
# With --strict, incomplete fan-out returns a nonzero exit code.
Complete command help
Example / output
List all server nodes in the cluster
Usage:
wp node list [flags]
Flags:
--az string filter by availability zone
-h, --help help for list
--region string filter by region
--rg string filter by resource group
--state string filter by state: active|decommissioning|decommissioned|unreachable|all (default "all")
wp node log-health
Show per-log read/write health observed by a node
Example / output
wp node log-health <node> [flags]
Parameters to know
Parameter / type
Use and default
--bucket string
Filter to one instance's bucket (requires --root)
--root string
Filter to one instance's root path (requires --bucket)
Write/read health describe client/log activity tracked on this node. Idle is inactivity, not a checksum failure. Stalled and Failed require looking at operations and last failure; this registry is not a complete metadata inventory.
Show per-log read/write health observed by a node
Usage:
wp node log-health <node> [flags]
Flags:
--bucket string Filter to one instance's bucket (requires --root)
-h, --help help for log-health
--root string Filter to one instance's root path (requires --bucket)
wp node restart
(Not implemented) Restart a node — use your orchestrator instead
Example / output
wp node restart <node> [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Demo
Example / output
wp node restart node-0
Output and interpretation
Example / output
No supported success path
This command is currently a stub. Restart through the orchestrator; first verify drain safety when doing planned replacement.
Finding or error example
Example / output
NotImplemented
Complete command help
Example / output
(Not implemented) Restart a node — use your orchestrator instead
Usage:
wp node restart <node> [flags]
Flags:
-h, --help help for restart
wp node show
Show detailed status for a single node
Example / output
wp node show <node> [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
status, logs and scale print plans by default; -x executes. scale targets a WoodpeckerCluster CR, not an arbitrary StatefulSet. Set the correct kube context and namespace. doctor checks local prerequisites.
Finding or error example
Example / output
Missing kubectl, kubeconfig or resource / kubectl exit status
Complete command help
Example / output
Tail logs from a Woodpecker pod
Usage:
wp k8s logs <node-or-pod> [flags]
Flags:
-x, --execute Execute kubectl commands (default: print only)
-f, --follow Stream logs
-h, --help help for logs
--kube-context string kubectl context name
--kubeconfig string Path to kubeconfig file
--kubectl string Path to kubectl binary
-n, --namespace string Kubernetes namespace
--since string Only show logs newer than duration (e.g. 1h)
--tail int Number of recent lines to show (default 100)
--wp-cluster string WoodpeckerCluster CR name
Scaling wp to 4 replicas...
kubectl ... patch woodpeckercluster wp --type merge -p ...
# Printed plan only; --execute runs it.
status, logs and scale print plans by default; -x executes. scale targets a WoodpeckerCluster CR, not an arbitrary StatefulSet. Set the correct kube context and namespace. doctor checks local prerequisites. This targets a WoodpeckerCluster CR, not a Helm release.
Finding or error example
Example / output
Missing kubectl, kubeconfig or resource / kubectl exit status
Complete command help
Example / output
Scale the cluster via kubectl patch
Usage:
wp k8s scale [flags]
Flags:
-x, --execute Execute kubectl commands (default: print only)
-h, --help help for scale
--kube-context string kubectl context name
--kubeconfig string Path to kubeconfig file
--kubectl string Path to kubectl binary
-n, --namespace string Kubernetes namespace
--replicas int Target replica count (required)
--wp-cluster string WoodpeckerCluster CR name
wp k8s status
Show cluster status via kubectl
Example / output
wp k8s status [flags]
Parameters to know
Parameter / type
Use and default
-x, --execute
Execute kubectl commands (default: print only)
--kube-context string
kubectl context name
--kubeconfig string
Path to kubeconfig file
--kubectl string
Path to kubectl binary
-n, --namespace string
Kubernetes namespace
--wp-cluster string
WoodpeckerCluster CR name
Demo
Example / output
wp k8s status --namespace milvus --wp-cluster woodpecker-prod
Output and interpretation
Example / output
kubectl ... get woodpeckercluster woodpecker-prod -o wide
kubectl ... describe woodpeckercluster woodpecker-prod
# Printed plan only; --execute runs it.
status, logs and scale print plans by default; -x executes. scale targets a WoodpeckerCluster CR, not an arbitrary StatefulSet. Set the correct kube context and namespace. doctor checks local prerequisites. status/scale require an operator-managed WoodpeckerCluster CR. For a Helm deployment without that CR, use kubectl get pods directly.
Finding or error example
Example / output
Missing kubectl, kubeconfig or resource / kubectl exit status
Complete command help
Example / output
Show cluster status via kubectl
Usage:
wp k8s status [flags]
Flags:
-x, --execute Execute kubectl commands (default: print only)
-h, --help help for status
--kube-context string kubectl context name
--kubeconfig string Path to kubeconfig file
--kubectl string Path to kubectl binary
-n, --namespace string Kubernetes namespace
--wp-cluster string WoodpeckerCluster CR name
Compare configurations across nodes
Usage:
wp config diff [node-a] [node-b] [flags]
Flags:
--all diff all nodes in the cluster against the first reachable node
-h, --help help for diff
--reference string explicit reference node name
wp config show
Show the resolved configuration of a single node
Example / output
wp config show <node> [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Demo
Example / output
wp config show node-0 -o json
Output and interpretation
Example / output
{"woodpecker":{"storage":{"type":"service"}}}
This is server configuration, not client SDK configuration. Compare actual values and failed respondents separately.
Finding or error example
Example / output
Error: Network: /admin/config returned status 503
Complete command help
Example / output
Show the resolved configuration of a single node
Usage:
wp config show <node> [flags]
Flags:
-h, --help help for show
Runtime logging changes affect this node. Record the old level and restore it after a short reproduction; this is separate from the log metadata command group.
Finding or error example
Example / output
node-0 (unreachable)
# In an --all report, inspect each row.
Complete command help
Example / output
Show current log level of a node (or all nodes)
Usage:
wp logging get-level [node] [flags]
Flags:
--all Show log level on all nodes
-h, --help help for get-level
wp logging set-level
Change log level of a node (or all nodes)
Example / output
wp logging set-level [node] [flags]
Parameters to know
Parameter / type
Use and default
--all
Set log level on all nodes
--level string
Target log level (debug, info, warn, error)
Demo
Example / output
wp logging set-level node-0 --level debug
Output and interpretation
Example / output
Log level set to "debug" on node-0
Runtime logging changes affect this node. Record the old level and restore it after a short reproduction; this is separate from the log metadata command group.
Finding or error example
Example / output
Error: Usage: invalid log level ...
Complete command help
Example / output
Change log level of a node (or all nodes)
Usage:
wp logging set-level [node] [flags]
Flags:
--all Set log level on all nodes
-h, --help help for set-level
--level string Target log level (debug, info, warn, error)
wp profile node-0 --type cpu --seconds 10 --output-file node0-cpu.pb.gz
Output and interpretation
Example / output
Saved: node0-cpu.pb.gz
To analyze: go tool pprof -http=127.0.0.1:8081 node0-cpu.pb.gz
CPU uses /debug/pprof/profile. Trace uses go tool trace, not pprof, and saves .trace. allocs is cumulative allocation, heap is live memory; a sparse mutex profile does not prove absence of deadlocks. Goroutine pprof aggregates stacks; it is not a raw stack dump.
Finding or error example
Example / output
pprof returned status 404 / timeout
Complete command help
Example / output
Download a Go profile or execution trace from a node
Usage:
wp profile <node> [flags]
Flags:
-h, --help help for profile
--output-file string output file path (default: ./<node>-<type>-<timestamp>.pb.gz; trace uses .trace)
--seconds int duration for cpu/trace profiles (default 30)
--type string profile type: cpu|heap|goroutine|allocs|mutex|block|threadcreate|trace (default "cpu")
Inspect running operations, their age and operation-registry pressure.
wp ops list
List in-flight operations
Example / output
wp ops list <node> [flags]
Parameters to know
Parameter / type
Use and default
--limit int
Max results
--log int
Filter by log ID
--longer-than int
Only ops running longer than N ms
--seg int
Filter by segment ID
--type string
Filter by op type (e.g. file.flush)
Demo
Example / output
wp ops list node-0 --longer-than 30000
Output and interpretation
Example / output
OP_TYPE OP_ID TRACE_ID LOG:SEG
# No rows means no matching tracked operation at this instant.
An empty table means no tracked in-flight operations at this instant, not no historical workload. Compare age and operation IDs over time.
Finding or error example
Example / output
logstore.add_entry append-op-123 ... 42:7
# The same old operation across samples needs investigation.
Complete command help
Example / output
List in-flight operations
Usage:
wp ops list <node> [flags]
Flags:
-h, --help help for list
--limit int Max results
--log int Filter by log ID
--longer-than int Only ops running longer than N ms
--seg int Filter by segment ID
--type string Filter by op type (e.g. file.flush)
Copy a real OP_ID from ops list. A completed operation disappearing is expected; registry capacity is bounded. Replace append-op-123 with an ID returned by ops list.
Finding or error example
Example / output
Error: ResourceNotFound: ...
# It may have completed or been evicted since listing.
Complete command help
Example / output
Show detail of a single in-flight operation
Usage:
wp ops show <node> [flags]
Flags:
-h, --help help for show
--op-id string Operation ID (required)
wp ops stats
Show op registry utilization and eviction statistics
Example / output
wp ops stats <node> [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Inspect Prometheus samples, trends and supported scenario findings.
wp metrics list
List all metric series exposed by a node
Example / output
wp metrics list <node> [flags]
Parameters to know
Parameter / type
Use and default
--filter string
Substring filter on metric name
Demo
Example / output
wp metrics list node-0
Output and interpretation
Example / output
NAME TYPE SERIES HELP
woodpecker_server_logstore_operations_total COUNTER 6 ...
Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.
Finding or error example
Example / output
# No matching metric: check --filter, node build and whether the series has been created.
Complete command help
Example / output
List all metric series exposed by a node
Usage:
wp metrics list <node> [flags]
Flags:
--filter string Substring filter on metric name
-h, --help help for list
Result: OK — no findings for scenario "stuck-read"
Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.
Run scenario-based metrics analysis
Usage:
wp metrics report [node] [flags]
Flags:
-h, --help help for report
--interval duration Scrape interval within window (default 1s)
--list List available scenarios
--scenario string Scenario name to run
--window duration Collection window duration (default 30s)
wp metrics snapshot
Point-in-time metric snapshot from one or all nodes
Example / output
wp metrics snapshot [node] [flags]
Parameters to know
Parameter / type
Use and default
--all
Snapshot from all nodes
--metric string
Filter to specific metric name
Demo
Example / output
wp metrics snapshot node-0
Output and interpretation
Example / output
NODE METRIC LABELS VALUE
node-0 woodpecker_server_logstore_operations_total ... 5
Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.
Finding or error example
Example / output
# Missing rows can mean absent series or a failed scrape. Check peer reachability.
Complete command help
Example / output
Point-in-time metric snapshot from one or all nodes
Usage:
wp metrics snapshot [node] [flags]
Flags:
--all Snapshot from all nodes
-h, --help help for snapshot
--metric string Filter to specific metric name
wp metrics top
Cross-node top-N comparison of a specific metric
Example / output
wp metrics top [flags]
Parameters to know
Parameter / type
Use and default
--by string
Metric name to rank by (required)
--top int
Show top N nodes (default 10)
Demo
Example / output
wp metrics top --by woodpecker_server_logstore_operations_total
Output and interpretation
Example / output
RANK NODE VALUE
1 node-0 120
2 node-1 100
Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.
Finding or error example
Example / output
# Ranking is by the requested metric value. A cumulative counter is not an interval rate.
Complete command help
Example / output
Cross-node top-N comparison of a specific metric
Usage:
wp metrics top [flags]
Flags:
--by string Metric name to rank by (required)
-h, --help help for top
--top int Show top N nodes (default 10)
Prometheus counters are cumulative; compare deltas, labels and time windows. An idle counter alone is not a stall. stuck-read/write require a persistent relevant in-flight operation and no successful/EOF progress for the same log. Some named scenarios remain heuristic or placeholders; absence of findings does not establish a healthy application. See scenario implementation limits below.
Finding or error example
Example / output
12:00:02.000 (error)
# Or (not found): inspect the scrape/metric selection.
Complete command help
Example / output
Real-time stream of a metric with trend arrows
Usage:
wp metrics watch <metric-name> <node> [flags]
Flags:
-h, --help help for watch
--interval duration Scrape interval (default 1s)
{"schema_version":"wp.diagnosis/v1","collection_status":"completed","observation_mode":"live_window","findings":[{"rule_id":"write_stall","outcome":"suspected_stall","evidence_strength":"suspected","unknown":["tenant_association_unknown"]}]}
# Abbreviated illustrative output; inspect actual observations and coverage_gaps.
Source-build capability. Reuses the WP service context and node Admin mappings. The write scenario checks append, writer-local buffer/flush trends, actual disk watermarks and optional explicitly selected log/segment quorum. Missing metadata does not fall back to another namespace. All includes these checks; focused stuck-write/read contracts remain compatible. --window samples from now; snapshot cannot prove a sustained stall. Queue pressure does not prove a disk/fsync cause, and missing replica observations do not prove quorum loss. Maintenance/object-store attribution remains an explicit evidence gap. Exit 0 means completed evaluation, not healthy; --fail-on-findings opts into exit 8 for warning/critical findings. Same-log completions cannot prove an individual op finished; resets, registry limits and missing evidence remain explicit. No external Prometheus, trace or Loki client is included.
Source-build capability. Reuses the WP service context and node Admin mappings. The read scenario separates success/EOF and pending server activity from leased reader progress and configured quorum evidence. Reader checks require explicit log/segment and WP-client etcd/prefix; there is no fallback namespace. The source client profile reports about every 30 seconds, so short unchanged-position windows are inconclusive. Fresh renewed attempts plus available data and catch-up expectation are needed for a no-progress suspicion. A missing replica is not quorum loss or corruption; a metadata end is not payload verification. Server ops lack reader identity, and duplicate numeric log IDs do not establish a tenant join. Existing focused stuck-write/read contracts remain compatible. --window samples now, not history. Exit 0 means completed evaluation, not healthy; --fail-on-findings opts into exit 8 for warning/critical findings. No external Prometheus, trace or Loki client is included.
Finding or error example
Example / output
PartialCollection (exit 6): required metrics/ops/writers were unavailable or writer details exceeded the bound; report retained.
Budget exhausted (exit 5) / cancellation (exit 7): partial evidence retained.
Complete command help
Example / output
Diagnose current WP service state or sample a live window using existing
metrics/Admin APIs. With no node argument, inspect all discovered nodes within
the target budget. Reuses cli.yaml, --context, --endpoint and node mappings.
Use --scenario write for append, flush, disk admission and explicit replica
checks, or --scenario read for activity, reader progress and quorum visibility.
Focused stuck-write/stuck-read retain their original v1 contracts. Snapshot mode
captures current evidence but cannot establish a sustained stall or buildup.
Reader progress needs the 30s client-profile reporting cadence and an explicit
read expectation. --window starts sampling now;
historical Prometheus, trace and Loki queries are available under wp diag.
Exit 0 means evaluation completed, including inconclusive/conflicting findings;
it is not a health assertion. --fail-on-findings returns 8 for warning/critical findings.
Required collection failure returns 1 (all failed) or 6 (partial), cancellation 7,
and exhausted wall-clock budget 5. JSON/YAML evidence is retained on these errors.
Usage:
wp diagnose [node] [flags]
Flags:
--budget duration Whole-command wall-clock budget including discovery (max 10m) (default 1m0s)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--fail-on-findings Return exit 8 for warning/critical findings after rendering the report
-h, --help help for diagnose
--interval duration Interval between live sampling rounds (default 1s)
--list List the implemented typed rule contracts without connecting
--log-name string Explicit WP-client metadata log; requires --segment, --etcd and --meta-prefix
--max-nodes int Maximum discovered targets; select a node if exceeded (default 8)
--max-ops int Maximum ops per node per sample (1..1000) (default 100)
--max-readers int Maximum reader records and lease checks per metadata sample (1..64) (default 16)
--max-response-bytes int Maximum bytes per discovery/source response (default 2097152)
--max-samples int Maximum sampling rounds per node (2..121) (default 31)
--max-series int Maximum retained completion-counter series per sample (default 500)
--max-writers int Maximum writer detail requests per node per sample (1..64; write/all) (default 16)
--meta-prefix string Required full client metadata prefix
--mode string Observation mode: live|snapshot; no historical retrieval (default "live")
--read-mode string Expected reader use: unknown|tail|catch-up; not historical metrics retrieval (default "unknown")
--reader string Select one reader by exact name; otherwise inspect up to --max-readers
--scenario string Native scenario: all|write|read|stuck-write|stuck-read (default "all")
--segment int Explicit metadata segment ID for replica/reader checks
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
--window duration Live observation window starting now (max 5m) (default 30s)
Inspect loaded writers, buffers, flush queues and LAC; perform targeted flush, compact or fence actions.
wp logstore buffer
Show buffer bytes summary, sorted by size
Example / output
wp logstore buffer <node> [flags]
Parameters to know
Parameter / type
Use and default
--top int
Show top N segments (default 20)
Demo
Example / output
wp logstore buffer node-0
Output and interpretation
Example / output
LOG:SEG BUFFER_BYTES BUFFER_ENTRIES
42:7 4096 4
This inspects node-local runtime state. Take repeated samples and correlate long-running operations; one nonzero queue is not automatically a stall.
Finding or error example
Example / output
# A sustained/growing value with outstanding appends requires flush-path investigation.
Complete command help
Example / output
Show buffer bytes summary, sorted by size
Usage:
wp logstore buffer <node> [flags]
Flags:
-h, --help help for buffer
--top int Show top N segments (default 20)
wp logstore compact
Force compaction on a segment at an exact confirmed entry ID
Example / output
wp logstore compact <node> [flags]
Parameters to know
Parameter / type
Use and default
--expected-last-entry-id int
Exact LastEntryId from coordinator metadata; an incorrect value can publish an invalid footer (-1 only for an empty segment, required)
Use the exact LastEntryId from verified Completed metadata, never guess from a replica’s longer tail. This acts on one replica; client auditor owns lifecycle advancement and mark distribution.
Finding or error example
Example / output
Error: StateConflict: ...
# Verify the exact coordinator last entry; inspect node logs for the rejected compaction.
Complete command help
Example / output
Force compaction on a segment at an exact confirmed entry ID
Usage:
wp logstore compact <node> [flags]
Flags:
--expected-last-entry-id int Exact LastEntryId from coordinator metadata; an incorrect value can publish an invalid footer (-1 only for an empty segment, required)
-h, --help help for compact
--log int Log ID (required)
--seg int Segment ID (required)
About to fence log 2 segment 0 on node-0. Reason: Reviewed recovery
# Preview only; no -y means no fencing and exit code 7.
Changes write acceptance on one node. One-node fencing does not necessarily block an entire quorum; use fence-quorum for deliberate quorum intervention. Re-run with -y only after reviewing the node and quorum impact.
Finding or error example
Example / output
Missing writer / fence failure
Complete command help
Example / output
Fence a segment on a single node.
On a multi-node quorum this removes one acknowledgement and the write carries on through the rest; interrupting a write takes wq-aq+1 nodes, which is what `fence-quorum` does.
Usage:
wp logstore fence <node> [flags]
Flags:
-h, --help help for fence
--log int Log ID (required)
--reason string Reason for fencing (required)
--seg int Segment ID (required)
-y, --yes Skip confirmation
wp logstore fence-quorum
Fence a segment across enough of its quorum to interrupt writes (high-risk, requires --reason)
Selected ...
# Preview only; without -y no RPC fencing is committed (exit 7).
The command previews quorum impact and requires acceptance (-y bypasses the prompt). Fencing can interrupt writes and trigger rolling/recovery; it does not repair data. Re-run the reviewed command with -y to commit.
Finding or error example
Example / output
Acceptance not given / too few selected reachable members
Complete command help
Example / output
Fence a segment across enough of its quorum to interrupt writes (high-risk, requires --reason)
Usage:
wp logstore fence-quorum <logName> <segmentId> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for fence-quorum
--meta-prefix string Required full client metadata prefix
--nodes strings Fence only these quorum members (default: every node in the quorum)
--reason string Reason for fencing (required)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
-y, --yes Skip confirmation
wp logstore flush-queue
Show flush queue depth summary, sorted by depth
Example / output
wp logstore flush-queue <node> [flags]
Parameters to know
Parameter / type
Use and default
--top int
Show top N segments (default 20)
Demo
Example / output
wp logstore flush-queue node-0
Output and interpretation
Example / output
LOG:SEG QUEUE_DEPTH QUEUE_CAP
42:7 0 16
This inspects node-local runtime state. Take repeated samples and correlate long-running operations; one nonzero queue is not automatically a stall.
Finding or error example
Example / output
42:7 16 16
# Correlate a persistently full queue with file.flush operations and backend latency.
Complete command help
Example / output
Show flush queue depth summary, sorted by depth
Usage:
wp logstore flush-queue <node> [flags]
Flags:
-h, --help help for flush-queue
--top int Show top N segments (default 20)
wp logstore force-flush
Force sync on a specific segment or all segments
Example / output
wp logstore force-flush <node> [flags]
Parameters to know
Parameter / type
Use and default
--log int
Log ID (0 = all)
--seg int
Segment ID (0 = all)
Demo
Example / output
wp logstore force-flush node-0 --log 2 --seg 0
Output and interpretation
Example / output
force-flush completed on node-0
Mutates a loaded writer. It cannot repair corrupted payloads; inspect queues and backend errors first.
Finding or error example
Example / output
Error: StateConflict: ...
Complete command help
Example / output
Force sync on a specific segment or all segments
Usage:
wp logstore force-flush <node> [flags]
Flags:
-h, --help help for force-flush
--log int Log ID (0 = all)
--seg int Segment ID (0 = all)
wp logstore lac
Show how far a segment is confirmed readable across its quorum
Example / output
wp logstore lac <logName> <segmentId> [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
LAC is confirmed progress reported per replica, not a full checksum verification. Active metadata is not a final boundary. Map every quorum member’s admin address.
Finding or error example
Example / output
quorum LAC: unknown — 1 of 3 nodes answered, fewer than aq=2
Complete command help
Example / output
Show how far a segment is confirmed readable across its quorum
Usage:
wp logstore lac <logName> <segmentId> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for lac
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp logstore segment-show
Show a segment's active writer on a node (not historical segment metadata)
Example / output
wp logstore segment-show <node> [flags]
Parameters to know
Parameter / type
Use and default
--log int
Log ID (required)
--seg int
Segment ID (required)
Demo
Example / output
wp logstore segment-show node-0 --log 2 --seg 0
Output and interpretation
Example / output
writer: ...
buffer: ...
flush_state: ...
# Illustrative excerpt; fields depend on the storage backend.
--log and --seg are numeric IDs. Historical/unloaded segments need segment inspect and log scan, not this active-writer endpoint.
Finding or error example
Example / output
Error: TargetNotFound: ...
# No loaded writer; historical data may still exist.
Complete command help
Example / output
Show a segment's active writer on a node (not historical segment metadata)
Usage:
wp logstore segment-show <node> [flags]
Flags:
-h, --help help for segment-show
--log int Log ID (required)
--seg int Segment ID (required)
Lists loaded writer objects, not every segment on disk or in etcd. An empty table does not establish no stored data; use log scan and instance data.
Finding or error example
Example / output
Header only / node unavailable
Complete command help
Example / output
List active segments on a node
Usage:
wp logstore segments <node> [flags]
Flags:
-h, --help help for segments
--log int Filter by log ID
--writable string Filter by writable state (true/false)
List bucket/root instances holding node-local data before drain or reclamation.
wp instance data
List the instances holding node-local data.
Example / output
wp instance data [node] [flags]
Parameters to know
Parameter / type
Use and default
--all
Query every node in the cluster
--bucket string
Filter to one instance's bucket (requires --root)
--root string
Filter to one instance's root path (requires --bucket)
Demo
Example / output
wp instance data --all --strict
Output and interpretation
Example / output
NODE BUCKET ROOT_PATH LOGS SEGMENTS LIVE SIZE_BYTES PROCS DELETE_STATE
node-0 milvus-bucket milvus/wp 1 2 1 1048576 1 active
Inventory local instances before deleting or draining. --bucket requires --root (this command uses --root, unlike segment --root-path). No CLI instance-delete command is currently registered; the scenario guide documents the admin API.
Finding or error example
Example / output
A node has scan_errors or is unreachable. A partial inventory does not establish that no data remains.
Complete command help
Example / output
List the instances holding node-local data.
Pair this with 'POST /admin/instance/delete' to reclaim an orphaned instance: query
first, decide, then delete only the nodes that reported the instance.
An unreachable node still holds its data, so a node shown as UNREACHABLE — or any
node reporting scan errors — means the picture is incomplete and nothing should be
deleted on the strength of it. Use --strict to turn that into a non-zero exit.
Usage:
wp instance data [node] [flags]
Flags:
--all Query every node in the cluster
--bucket string Filter to one instance's bucket (requires --root)
-h, --help help for data
--root string Filter to one instance's root path (requires --bucket)
READER OPEN_POS CURRENT LAST_REPORT OPEN_FOR STATE
reader-123 7:0 7:50 3s 5m live
Reads leased registration under readers/<logId>/. No live readers means no usable registration here, not no Milvus queries. Reports normally refresh every 30 seconds; compare two snapshots to establish progress. Lease expiry or wrong namespace can hide readers.
Finding or error example
Example / output
Log my-channel (id 42) — no live readers
# Or a stale report: check activity and namespace before declaring a stall.
Complete command help
Example / output
Show where each of a log's readers has read to
Usage:
wp log readers <logName> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for readers
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp log scan
Compare metadata with replica local coverage; raw verifies local block checksums
Example / output
wp log scan <logName> [flags]
Parameters to know
Parameter / type
Use and default
--bucket string
Storage bucket; pair with --root-path to locate unloaded or empty segments
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--from-segment int
First segment to scan
--mode string
quick reads only structure and metadata; raw reads the data through the real checksums (default "quick")
--root-path string
Storage instance root path; pair with --bucket
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
--to-segment int
Last segment to scan (default: the newest) (default -1)
SEGMENT STATE META LAST REACHES REPLICAS VERDICT
1 Active -1 (open) - 3/3 open
Per replica: local_empty / none / no_local_blocks
REACHES is the union of observed local ranges; - means none observed. 3/3 counts usable inspection responses, not three healthy data copies. quick checks structure; raw verifies local blocks. Sealed/compacted data in object storage is not verified. Explicit bucket/root-path resolves empty or unloaded targets; pending writes are not inspected.
Compare metadata with replica local coverage; raw verifies local block checksums
Usage:
wp log scan <logName> [flags]
Flags:
--bucket string Storage bucket; pair with --root-path to locate unloaded or empty segments
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--from-segment int First segment to scan
-h, --help help for scan
--meta-prefix string Required full client metadata prefix
--mode string quick reads only structure and metadata; raw reads the data through the real checksums (default "quick")
--root-path string Storage instance root path; pair with --bucket
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
--to-segment int Last segment to scan (default: the newest) (default -1)
wp log skip-range add
Declare an entry range unreadable so readers move past it.
Ranges and reasons / preview or committed declaration
A declaration makes readers skip accepted data loss; it does not fix replicas or advance compaction. Entry bounds are inclusive. add previews before confirmation (-y accepts); remove does not restore skipped/reclaimed data. Inspect all replicas and the healthy tail first.
Finding or error example
Example / output
Gate refuses because a replica can serve / acceptance not given
Complete command help
Example / output
Declare an entry range unreadable so readers move past it.
This gives up data. Entries in the range are never delivered to any reader of this log
again, and nothing restores them: withdrawing the range only exposes whatever is still on
disk. So the quorum is asked first, and a replica that can still read part of the range
refuses the declaration.
Usage:
wp log skip-range add <logName> <segmentId> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--force Declare the range even though a replica can still read part of it
--from-entry int First entry to skip (inclusive) (default -1)
-h, --help help for add
--meta-prefix string Required full client metadata prefix
--reason string Why these entries are being given up (required)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
--to-entry int Last entry to skip (inclusive) (default -1)
-y, --yes Proceed without the confirmation prompt
wp log skip-range list
Show the declared skip ranges.
Example / output
wp log skip-range list [logName] [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp log skip-range list my-channel --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker
Output and interpretation
Example / output
log 42 segment 7: 50-50
# Illustrative excerpt; inspect the recorded reason and age.
A declaration makes readers skip accepted data loss; it does not fix replicas or advance compaction. Entry bounds are inclusive. add previews before confirmation (-y accepts); remove does not restore skipped/reclaimed data. Inspect all replicas and the healthy tail first.
Finding or error example
Example / output
# No declarations is normal when no loss has been accepted.
Complete command help
Example / output
Show the declared skip ranges.
With no log name every log is listed, including ranges whose log no longer exists: log ids
are never reused, so such a range can never apply to anything again, but it stays in the
record until it is removed.
Usage:
wp log skip-range list [logName] [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for list
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp log skip-range remove
Withdraw a declared range, so readers stop skipping it.
Removed: ...
# Inspect list afterward to verify the remaining intervals.
A declaration makes readers skip accepted data loss; it does not fix replicas or advance compaction. Entry bounds are inclusive. add previews before confirmation (-y accepts); remove does not restore skipped/reclaimed data. Inspect all replicas and the healthy tail first.
Finding or error example
Example / output
Error: StateConflict: ...
# Refresh and retry after reviewing concurrent changes.
Complete command help
Example / output
Withdraw a declared range, so readers stop skipping it.
Withdrawing does not restore anything: it only lets readers try the entries again, which
is what to do once a replica has been rebuilt. A reader that already moved past the range
does not come back for it.
Usage:
wp log skip-range remove <logName> <segmentId> | --log-id N <segmentId> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--from-entry int First entry of the range to withdraw (inclusive) (default -1)
-h, --help help for remove
--log-id int Withdraw by log id instead of name, for a range whose log has been deleted (default -1)
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
--to-entry int Last entry of the range to withdraw (inclusive) (default -1)
wp log truncate
Advance the truncation frontier with the SDK's metadata-update semantics and additional operator guards. Defaults to preview. The target must be a non-empty Completed or Sealed segment. Subsequent auditor rounds mark earlier finalized segments Truncated and reclaim data according to retention and reader protection. The frontier segment is retained. Apply can cause permanent data loss; verify application checkpoints first.
Example / output
wp log truncate <logName> [flags]
Parameters to know
Parameter / type
Use and default
--apply
Advance the frontier (default: preview only)
--backup string
New pre-change backup file; never overwritten
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--entry int
Target entry ID (required) (default -1)
--expected-revision int
Log record ModRevision from preview
--reason string
Operator reason recorded in audit
--segment int
Target segment ID (required) (default -1)
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Preview requires a non-empty Completed/Sealed target and entry within 0..lastEntryId. Apply requires --apply --expected-revision (the log before.revision), --reason and a new --backup; no offline flag is required. The log and target revisions are compared and an audit record is written atomically. The frontier segment stays retained; earlier finalized segments become eligible for auditor/retention/reader-protected cleanup. Unchanged frontier means applied=false, unchanged=true with no artifacts. Verify application checkpoints first: metadata backup cannot restore reclaimed data.
Finding or error example
Example / output
Error: StateConflict: truncation cannot move backward or restore deleted data
# Active/empty targets, identity mismatches, stale revisions and concurrent updates are also rejected.
Complete command help
Example / output
Advance the truncation frontier with the SDK's metadata-update semantics and additional operator guards. Defaults to preview. The target must be a non-empty Completed or Sealed segment. Subsequent auditor rounds mark earlier finalized segments Truncated and reclaim data according to retention and reader protection. The frontier segment is retained. Apply can cause permanent data loss; verify application checkpoints first.
Usage:
wp log truncate <logName> [flags]
Flags:
--apply Advance the frontier (default: preview only)
--backup string New pre-change backup file; never overwritten
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--entry int Target entry ID (required) (default -1)
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--expected-revision int Log record ModRevision from preview
-h, --help help for truncate
--meta-prefix string Required full client metadata prefix
--reason string Operator reason recorded in audit
--segment int Target segment ID (required) (default -1)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
BLOCK ENTRIES READABLE RECOVERABLE STATUS
- - - - no_local_blocks
Walks local blocks and records to locate damage boundaries; it can report later recoverable entries despite a bad block. no_local_blocks says nothing about shared object integrity. Use actual timestamps/metadata separately; --max-blocks bounds the survey.
Finding or error example
Example / output
Block 0: checksum mismatch; entry 50 corrupt; later good entries
Complete command help
Example / output
Walk a segment's blocks on every replica and report what is wrong where
Usage:
wp segment inspect <logName> <segmentId> [flags]
Flags:
--bucket string Storage bucket; pair with --root-path (otherwise discovered on each node)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--from-block int First block to walk (default: the start of the segment)
-h, --help help for inspect
--max-blocks int Stop after this many blocks per node (default: the node's own bound)
--meta-prefix string Required full client metadata prefix
--root-path string Storage instance root path; pair with --bucket
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp segment probe
Ask every replica how far it can read a segment
Example / output
wp segment probe <logName> <segmentId> [flags]
Parameters to know
Parameter / type
Use and default
--bucket string
Storage bucket; pair with --root-path (otherwise discovered on each node)
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--from-entry int
Entry to read from (default: the start of the segment)
--max-entries int
Stop after this many entries per node (default: the node's own bound)
--root-path string
Storage instance root path; pair with --bucket
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
SOURCE object_storage SERVED 0-0 (1) OUTCOME end_of_file VERDICT served
Actually probes sequential entry reads and stops at a failure or bound. 0-0 (1) means one entry, ID 0. Object storage responses share one copy. Active tail/no local data has unknown final boundary and must not be called a permanent stall; Completed shortfall is compared to metadata. cap_reached is a bound, not EOF.
Finding or error example
Example / output
Completed expected 99, replica stops at 49 → short / RedFinding if all fail
Complete command help
Example / output
Ask every replica how far it can read a segment
Usage:
wp segment probe <logName> <segmentId> [flags]
Flags:
--bucket string Storage bucket; pair with --root-path (otherwise discovered on each node)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--from-entry int Entry to read from (default: the start of the segment)
-h, --help help for probe
--max-entries int Stop after this many entries per node (default: the node's own bound)
--meta-prefix string Required full client metadata prefix
--root-path string Storage instance root path; pair with --bucket
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Use a current source build containing wp meta and wp log truncate; these additions are not in v0.1.46. Complete CLI setup first. Supply explicit reachable etcd endpoints and the complete client metadata prefix for these commands, or select a private client parameter file. Server contexts do not select metadata authority. Pure metadata views need no node Admin connection or bucket/root path.
Example / output
wp --context milvus-prod meta log list
wp --context milvus-prod meta log show my-channel
wp --context milvus-prod meta segment list my-channel
wp --context milvus-prod meta segment show my-channel 7
wp --context milvus-prod meta check my-channel
Use the real logical log name. list includes persisted history even when node-local writer objects are unloaded. show wraps decoded metadata with its exact key, modification revision and lease; int64 fields appear as decimal strings. For modern segments, inspect the embedded quorum in segment show; separately stored quorum records may be absent.
2. Create a small JSON patch file
Pass only fields to change, not the show envelope or a full exported record. For example, create the files used by the command demos below:
Example / output
cat > ./log-policy.json <<'JSON'
{"max_segment_roll_time_seconds": 120}
JSON
# Only after independently proving that 99 is the correct finalized tail:
cat > ./segment-repair.json <<'JSON'
{"lastEntryId": "99"}
JSON
Patch target
Writable fields / constraints
log
max_segment_roll_time_seconds, max_segment_roll_size_bytes, compaction_buffer_size_bytes, max_compaction_file_count. Nonnegative; zero is allowed. These are reserved metadata fields, not live client runtime configuration.
segment
size, createTime, completionTime, sealedTime, lastEntryId, state. Times are Unix milliseconds; size is bytes. Lifecycle times must agree with state. Integers may be numbers or decimal strings.
Protected
Identity (logId/segNo), quorum and log truncation fields. Quorums are read-only; advance normal retention with log truncate.
Active requires lastEntryId=-1 and zero completion/sealed times. A verified rollback to Active must reset all three together. Completed/Sealed lastEntryId=-1 means finalized empty; Sealed needs a valid sealedTime. A key/segNo mismatch is rejected. Changing state does not generate compacted objects, repair data or execute recovery; Truncated can enable cleanup.
State-correction JSON examples
Use only the state independently verified for this segment. These are alternative patch-file contents, not automatic recovery steps. For a verified rollback to Active:
The timestamp is illustrative: replace it with the verified Unix-millisecond value, after completionTime. State names are Active, Completed, Sealed and Truncated (numbers 0, 2, 3, 4); reserved state 1 is rejected. A legacy createTime=0 must also be corrected with a verified positive creation time.
A preview returns applied=false, before/after records, changes and snapshot_revision. Review all changes. Use the target's before.revision for apply. If it is 174, stop the relevant writers/auditors and keep them stopped throughout the repair:
Example / output
wp --context milvus-prod meta segment patch my-channel 7 \
--patch-file ./segment-repair.json --apply --offline \
--expected-revision 174 --reason 'Correct verified tail after software fix' \
--backup ./segment-before-repair.json
wp --context milvus-prod meta segment show my-channel 7
wp --context milvus-prod meta check my-channel
--offline acknowledges your action; it does not stop clients. Apply checks revision and writer-lock absence, persists a new backup, and writes the correction and audit record atomically. Restart affected clients after repair so cached metadata is refreshed. applied=true/new_revision confirms the write, not data readability. Reused backup paths fail; stale revisions or concurrent changes require a fresh preview and a new backup filename.
Example / output
# Example warning on an otherwise valid scalar repair:
"warnings": [
"embedded quorum (unchanged): invalid quorum sizes or node count"
]
An existing embedded-quorum problem is preserved and reported in warnings; it does not block an independent scalar repair. meta check still reports that unresolved problem. Lifecycle and truncation-bound violations continue to block the patch. Do not mark a segment Sealed or lower a tail just to silence a physical corruption finding.
4. Interpret checks, exports and failures
Result
Meaning / next step
check: state=ok, issues=[]; exit 0
The inspected metadata constraints passed. Probe/inspect separately for physical data; verify application progress separately.
check: state=invalid; exit 9
Read the exact key/field in issues. A bad frontier, lifecycle or quorum is metadata evidence, not automatic permission to rewrite it.
TargetNotFound; exit 3
The exact record is absent. Verify context, prefix and logical IDs before diagnosing missing payloads.
Patch StateConflict; exit 4
Read the constraint, revision or writer-lock error. Re-read records and correct the verified cause; do not blindly retry apply.
export / backup
New mode-0600 file with readable known metadata and original value_base64/revisions/leases. File and parent directory must sync before patch commits. Unknown/malformed records retain their bytes. This is not a payload backup or automatic rollback command.
If commit returns a network error, re-read the target and audit records before retrying: the write outcome may be unknown. For a complete operational example, follow verified metadata repair in Troubleshooting.
wp meta check
Validate log/segment metadata constraints (does not verify physical data)
Example / output
wp meta check <logName> [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Checks log policy/frontier fields, segment identity/state/timestamps, embedded quorum structure and frontier bounds at one snapshot. Exit 9 plus issues means inconsistent metadata. state=ok establishes metadata consistency only; it does not prove payload readability or application progress.
Finding or error example
Example / output
{
"state": "invalid",
"issues": [
"milvus-prod/woodpecker/logs/my-channel/segments/7: key must use canonical segment ID 8 matching segNo"
]
}
Error: RedFinding: 1 metadata constraint violations
Complete command help
Example / output
Validate log/segment metadata constraints (does not verify physical data)
Usage:
wp meta check <logName> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for check
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp meta export
Export one metadata namespace at a consistent etcd revision
Example / output
wp meta export [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--output-file string
New snapshot file (required; never overwritten)
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Creates a new mode-0600 namespace snapshot. The file contains readable metadata for known protobuf records plus byte-exact value_base64, revisions and leases. Unknown/corrupt values retain bytes and encoding/decode errors. It is a metadata backup, not a payload backup or automatic restore command; an existing file is never overwritten.
Finding or error example
Example / output
Error: Config: create new backup file: open ./metadata-snapshot.json: file exists
Complete command help
Example / output
Export one metadata namespace at a consistent etcd revision
Usage:
wp meta export [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for export
--meta-prefix string Required full client metadata prefix
--output-file string New snapshot file (required; never overwritten)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp meta log list
list log metadata and etcd revision
Example / output
wp meta log list [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Lists top-level log records, including unloaded history. records is a snapshot, not a list of live node writers. Narrow with meta log show; use the record revision for a patch.
Finding or error example
Example / output
# records: [] means this listing has no matching records.
# An undecodable record instead returns a configuration error.
Complete command help
Example / output
list log metadata and etcd revision
Usage:
wp meta log list [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for list
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp meta log patch
Patch policy/statistics/timestamps, segment lastEntryId and state. Identity, quorum and log truncation are protected. Segment state correction does not create compacted files or repair replica contents; Truncated can enable cleanup. Applying requires stopped writers/auditors, a revision, reason, and a new backup file.
Example / output
wp meta log patch <logName> [flags]
Parameters to know
Parameter / type
Use and default
--apply
Commit the previewed correction (default: no writes)
--backup string
New pre-change backup file (required for apply; never overwritten)
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--expected-revision int
Required target ModRevision for apply
--offline
Confirm relevant writers/auditors are stopped and will remain stopped during repair
Create the patch file using the JSON examples in the metadata walkthrough before running this preview. Applying requires separately reviewed evidence and the guards described there.
Create log-policy.json using the walkthrough above. Only max_segment_roll_time_seconds, max_segment_roll_size_bytes, compaction_buffer_size_bytes and max_compaction_file_count are patchable (nonnegative; zero allowed). They are reserved metadata fields, not live runtime configuration. Preview writes nothing. Apply uses the target before.revision plus offline acknowledgement, reason and a new durable backup; unknown fields and leases are preserved.
Finding or error example
Example / output
Error: Usage: --apply requires --offline, --expected-revision, --reason, and --backup
# A stale revision or live writer lock returns StateConflict; no correction is applied.
Complete command help
Example / output
Patch policy/statistics/timestamps, segment lastEntryId and state. Identity, quorum and log truncation are protected. Segment state correction does not create compacted files or repair replica contents; Truncated can enable cleanup. Applying requires stopped writers/auditors, a revision, reason, and a new backup file.
Usage:
wp meta log patch <logName> [flags]
Flags:
--apply Commit the previewed correction (default: no writes)
--backup string New pre-change backup file (required for apply; never overwritten)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--expected-revision int Required target ModRevision for apply
-h, --help help for patch
--meta-prefix string Required full client metadata prefix
--offline Confirm relevant writers/auditors are stopped and will remain stopped during repair
--patch-file string JSON object containing supported scalar fields (required)
--reason string Operator's reason, recorded with the change
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp meta log show
show log metadata and etcd revision
Example / output
wp meta log show <logName> [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Shows the exact log key, ModRevision, lease and decoded fields. int64 values are decimal strings. A missing target means the key is absent in this namespace, not a damaged replica. Log truncation fields are changed with log truncate, not generic patch.
Finding or error example
Example / output
Error: TargetNotFound: target not found: milvus-prod/woodpecker/logs/my-channel
Complete command help
Example / output
show log metadata and etcd revision
Usage:
wp meta log show <logName> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for show
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp meta quorum list
list quorum metadata and etcd revision
Example / output
wp meta quorum list [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Lists separately stored quorum records. Modern segments embed their quorum, so an empty list can be normal; inspect meta segment show to see that topology. This group cannot patch quorum records.
Finding or error example
Example / output
# records: [] means this listing has no matching records.
# An undecodable record instead returns a configuration error.
Complete command help
Example / output
list quorum metadata and etcd revision
Usage:
wp meta quorum list [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for list
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp meta quorum show
show quorum metadata and etcd revision
Example / output
wp meta quorum show <ID> [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Shows one separately stored quorum by numeric ID; it does not fan out or test node health. es is ensemble size, wq write quorum, aq acknowledgement quorum. Modern embedded quorum is read through meta segment show.
Finding or error example
Example / output
Error: TargetNotFound: target not found: milvus-prod/woodpecker/quorums/1
Complete command help
Example / output
show quorum metadata and etcd revision
Usage:
wp meta quorum show <ID> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for show
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp meta segment list
list segment metadata and etcd revision
Example / output
wp meta segment list <logName> [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Lists persisted segments under this exact log, including unloaded history. The parent log must exist. State and lastEntryId are metadata claims; probe/inspect physical data separately.
Finding or error example
Example / output
# records: [] means this listing has no matching records.
# An undecodable record instead returns a configuration error.
Complete command help
Example / output
list segment metadata and etcd revision
Usage:
wp meta segment list <logName> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for list
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp meta segment patch
Patch policy/statistics/timestamps, segment lastEntryId and state. Identity, quorum and log truncation are protected. Segment state correction does not create compacted files or repair replica contents; Truncated can enable cleanup. Applying requires stopped writers/auditors, a revision, reason, and a new backup file.
Example / output
wp meta segment patch <logName> <segmentId> [flags]
Parameters to know
Parameter / type
Use and default
--apply
Commit the previewed correction (default: no writes)
--backup string
New pre-change backup file (required for apply; never overwritten)
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--expected-revision int
Required target ModRevision for apply
--offline
Confirm relevant writers/auditors are stopped and will remain stopped during repair
Create the patch file using the JSON examples in the metadata walkthrough before running this preview. Applying requires separately reviewed evidence and the guards described there.
Create segment-repair.json using the walkthrough above and independent data evidence. Writable fields are size, createTime, completionTime, sealedTime, lastEntryId and state. Lifecycle constraints remain strict; key identity, quorum and log truncation fields are protected. An existing invalid embedded quorum is preserved and reported in warnings rather than blocking a scalar repair. meta check will still report that unresolved quorum. A state edit does not create compacted files or repair replica contents.
Finding or error example
Example / output
Error: StateConflict: lastEntryId is below the log truncation frontier
# Other examples: key/segNo mismatch; Active segment requires lastEntryId -1; Sealed segment requires sealedTime.
Complete command help
Example / output
Patch policy/statistics/timestamps, segment lastEntryId and state. Identity, quorum and log truncation are protected. Segment state correction does not create compacted files or repair replica contents; Truncated can enable cleanup. Applying requires stopped writers/auditors, a revision, reason, and a new backup file.
Usage:
wp meta segment patch <logName> <segmentId> [flags]
Flags:
--apply Commit the previewed correction (default: no writes)
--backup string New pre-change backup file (required for apply; never overwritten)
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--expected-revision int Required target ModRevision for apply
-h, --help help for patch
--meta-prefix string Required full client metadata prefix
--offline Confirm relevant writers/auditors are stopped and will remain stopped during repair
--patch-file string JSON object containing supported scalar fields (required)
--reason string Operator's reason, recorded with the change
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp meta segment show
show segment metadata and etcd revision
Example / output
wp meta segment show <logName> <segmentId> [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Shows authoritative persisted metadata and its embedded quorum when present. Active lastEntryId=-1 is an unknown final boundary; Completed/Sealed lastEntryId=-1 means finalized empty. A key/segNo mismatch blocks patching. This read does not verify local or object-storage payloads.
Finding or error example
Example / output
Error: TargetNotFound: target not found: milvus-prod/woodpecker/logs/my-channel/segments/7
Complete command help
Example / output
show segment metadata and etcd revision
Usage:
wp meta segment show <logName> <segmentId> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for show
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
After verifying the permanently unavailable/unacked replica, settle notification retries. This preserves the record until truncation reaps it; it does not repair a segment or delete data immediately.
Finding or error example
Example / output
StateConflict for IN_PROGRESS / COMPLETED without --force
Complete command help
Example / output
Confirm one compacted-mark distribution record.
Meant for records parked as NOTIFY_PENDING_MANUAL: after verifying the unacked node
(typically permanently dead or removed from the cluster), confirm transitions the record
to NOTIFY_OPERATOR_CONFIRMED, which settles it (the auditor stops retrying and stops
counting it against the log). The etcd record itself is intentionally NOT deleted here;
it is reaped when the segment is truncated, like every other marking record.
Refuses records still IN_PROGRESS or COMPLETED unless --force is given (those are
managed automatically and normally need no operator action).
Usage:
wp marking confirm <logId> <segmentId> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
--force confirm (transition to OPERATOR_CONFIRMED) even if the record is not PENDING_MANUAL; IN_PROGRESS/COMPLETED are normally managed automatically
-h, --help help for confirm
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp marking list
List compacted-mark distribution records (default: only NOTIFY_PENDING_MANUAL)
Example / output
wp marking list [flags]
Parameters to know
Parameter / type
Use and default
--all-states
include IN_PROGRESS and COMPLETED records, not just PENDING_MANUAL
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--log int
restrict to one log id (default: all logs)
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp marking list --etcd 127.0.0.1:32379 --meta-prefix milvus-prod/woodpecker
Output and interpretation
Example / output
COMPLETED / operator-confirmed records
Records describe compacted-mark distribution, not the data payload. Investigate unacked nodes and physical cleanup before manual settlement.
Finding or error example
Example / output
NOTIFY_PENDING_MANUAL with unacked nodes
Complete command help
Example / output
List compacted-mark distribution records (default: only NOTIFY_PENDING_MANUAL)
Usage:
wp marking list [flags]
Flags:
--all-states include IN_PROGRESS and COMPLETED records, not just PENDING_MANUAL
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for list
--log int restrict to one log id (default: all logs)
--meta-prefix string Required full client metadata prefix
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Discover namespaces and inspect exact raw keys, revisions and values without writing etcd.
1. Discover keys before choosing a metadata prefix
This command group is included in v0.1.47 and later; it is not included in v0.1.46. Install a matching release or build from source, and confirm wp etcd --help before using these examples.
Use this read-only group when you can reach etcd but do not yet know the Woodpecker namespace. It connects directly to etcd: no Woodpecker admin endpoint, node mapping or metadata prefix is required. Supply a reachable endpoint explicitly, or select a private client parameter file.
Example / output
# Omit the prefix to explore all keys (first page only).
wp etcd list --etcd 127.0.0.1:32379 --limit 20
# Narrow the listing using an actual prefix found in that output.
wp etcd list milvus-prod/ --etcd 127.0.0.1:32379 --limit 20
A prefix is literal: list / lists only keys starting with a slash; it does not mean all keys. Empty results mean no matching keys in this snapshot, not an unhealthy cluster. A connection or permission error means the listing could not be established. This group never prepends a metadata prefix.
Exact key, key revision, lease, snapshot revision, encoding and value. Default get output is JSON.
cat
The value alone. It is useful for piping or saving a single value.
--encoding auto
Readable UTF-8 text is printed as text; binary/control-byte content becomes base64. get reports the selected value_encoding; cat prints a base64 notice on stderr.
--encoding text / base64 / hex
Choose the representation explicitly. text rejects invalid UTF-8. Protobuf bytes do not become decoded metadata simply by selecting text.
cat --encoding raw
Exact original bytes, without an appended newline. Redirect to a file rather than displaying binary content in a terminal.
A TargetNotFound result means that exact key is absent at the requested snapshot. Base64 is a reversible byte representation, not evidence of damaged data. This raw group does not decode, patch or restore metadata; the file above preserves one value, not a complete etcd backup.
Once you know the Woodpecker prefix, use Metadata inspection & repair for decoded log/segment records and readable namespace exports. Supply that complete prefix through context or --meta-prefix.
3. Continue a listing at the same snapshot
Use JSON output for pagination. When more is true, copy next_key_base64 and snapshot_revision from that response into the next request, keeping the same prefix:
Example / output
wp etcd list milvus-prod/ --etcd 127.0.0.1:32379 --limit 20 -o json
# Replace these placeholders with the returned cursor and snapshot revision.
wp etcd list milvus-prod/ --etcd 127.0.0.1:32379 --limit 20 -o json \
--after-key-base64 'CURSOR_FROM_PREVIOUS_RESPONSE' --revision 105
Repeat until more=false. The base64 cursor preserves binary keys. The keys array contains this page; count describes matching keys remaining in the queried range, so it can exceed the page length. --after-key is an alternative for ordinary text keys; never combine it with --after-key-base64.
Use --revision on get/cat too when comparing values with a listing. If etcd has compacted that revision, restart the listing at a new snapshot; do not silently mix pages from different revisions.
4. Reuse explicit client parameters
Keep etcd authority separate from server contexts. For repeated operations, explicitly select a private client parameter file:
wp etcd list --client-parameters-file /path/to/client-parameters.yaml --limit 20
wp etcd get tenant/woodpecker/version --etcd http://127.0.0.1:32379
Explicit flags override the private parameter file. Raw operations do not use context metadata authority or Admin discovery. See etcd connection settings for authentication/TLS fields. Recent input suggestions are independent values, not a saved tenant profile.
wp etcd cat
Print only one key's value; use --encoding raw for exact bytes
Example / output
wp etcd cat <key> [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--encoding string
Value encoding: auto|text|base64|hex; cat also supports raw (default "auto")
--revision int
Read at this etcd revision (0: latest)
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
CAEQACAu
# Encoded value only; binary auto output also explains its encoding on stderr.
cat writes the value without an envelope. auto/text/base64/hex output ends with a newline; --encoding raw preserves exact bytes and adds none. Redirect raw output to a file for byte-exact inspection. Neither cat nor get changes the key.
Finding or error example
Example / output
Error: Usage: value is not UTF-8; use --encoding base64, hex or raw
# Only cat supports --encoding raw.
Complete command help
Example / output
Print only one key's value; use --encoding raw for exact bytes
Usage:
wp etcd cat <key> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--encoding string Value encoding: auto|text|base64|hex; cat also supports raw (default "auto")
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for cat
--revision int Read at this etcd revision (0: latest)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp etcd get
Show one exact key, revisions, lease and encoded value
Example / output
wp etcd get <key> [flags]
Parameters to know
Parameter / type
Use and default
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--encoding string
Value encoding: auto|text|base64|hex; cat also supports raw (default "auto")
--revision int
Read at this etcd revision (0: latest)
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
get reads one exact key and returns key metadata plus an encoded value. auto keeps readable UTF-8 text and base64-encodes binary protobuf data; this command does not decode Woodpecker protobuf fields. Use --revision to read a retained historical snapshot and -o yaml for YAML.
Show one exact key, revisions, lease and encoded value
Usage:
wp etcd get <key> [flags]
Flags:
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--encoding string Value encoding: auto|text|base64|hex; cat also supports raw (default "auto")
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for get
--revision int Read at this etcd revision (0: latest)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
wp etcd list
List raw keys by literal prefix; omit prefix to explore all keys
Example / output
wp etcd list [prefix] [flags]
Parameters to know
Parameter / type
Use and default
--after-key string
Exclusive page start; use the previous response's next_key
--after-key-base64 string
Exclusive page start from next_key_base64; preserves binary keys
--client-parameters-file string
Private temporary etcd parameters file; not cli.yaml or a candidate profile
--limit int
Maximum keys per page (default 100)
--revision int
Read at this etcd snapshot revision (0: latest)
--standalone
Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Omit the prefix to explore all keys; list / searches only keys beginning with /. The limit controls page size, not the total count. When more=true, use the returned next_key_base64 and the same snapshot_revision for the next page. The context meta_prefix is never prepended.
Finding or error example
Example / output
Error: Usage: --after-key and --after-key-base64 are mutually exclusive
# A compacted --revision instead produces a Network error.
Complete command help
Example / output
List raw keys by literal prefix; omit prefix to explore all keys
Usage:
wp etcd list [prefix] [flags]
Flags:
--after-key string Exclusive page start; use the previous response's next_key
--after-key-base64 string Exclusive page start from next_key_base64; preserves binary keys
--client-parameters-file string Private temporary etcd parameters file; not cli.yaml or a candidate profile
--etcd string Required client etcd endpoints (comma-separated); independent of server context
--etcd-cacert string Local etcd TLS CA cert file; explicit per-run setting
--etcd-cert string Local etcd TLS client cert file; explicit per-run setting
--etcd-key string Local etcd TLS client key file; explicit per-run setting
--etcd-password string Etcd auth password; explicit per-run setting
--etcd-tls-min-version string Etcd TLS min version (1.0/1.1/1.2/1.3); explicit per-run setting
--etcd-username string Etcd auth username; explicit per-run setting
-h, --help help for list
--limit int Maximum keys per page (default 100)
--revision int Read at this etcd snapshot revision (0: latest)
--standalone Use explicit etcd settings only; requires --etcd and --meta-prefix, skips context and Admin discovery
Lists the metric, formula, rule and required inputs without querying a source.
The CLI, rules and WebUI share the same canonical check keys.
Finding or error example
Example / output
An unknown key is rejected.
Complete command help
Example / output
List implemented checks, formulas, references and scenario manifests; no source/model queries
Usage:
wp diag catalog [flags]
Flags:
--check string Check name or stable ID
-h, --help help for catalog
--source string metrics|trace
wp diag metrics check
Run one check; no implicit execution of other checks
Example / output
wp diag metrics check <name> [flags]
Parameters to know
Parameter / type
Use and default
--threshold float
Override this check threshold in native units (Trace durations use ms; error counts allow zero)
Demo
Example / output
wp --context local diag metrics check writer-latency --window 15m --details all -o json > writes.json
Output and interpretation
Example / output
A wp.diag/v1 report for the selected Writer timer.
This checks recorded WP client Writer operations. It does not infer application end-to-end latency.
Finding or error example
Example / output
Missing source configuration, failed queries and insufficient coverage remain explicit.
Complete command help
Example / output
Run one check; no implicit execution of other checks
Usage:
wp diag metrics check <name> [flags]
Flags:
-h, --help help for check
--threshold float Override this check threshold in native units (Trace durations use ms; error counts allow zero)
wp diag scenario
Run related Metrics/Trace checks and summarize their unchanged results by WAL step
Example / output
wp diag scenario <name|all> [flags]
Parameters to know
No additional operation-specific flags. Connection settings come from CLI setup.
Demo
Example / output
wp --context local diag scenario slow-writes --window 15m --node-id woodpecker-0 --details all -o json > diagnosis.json
Output and interpretation
Example / output
Independent Metrics/Trace findings summarized by WAL step.
A scenario aggregates check results without a second root-cause engine or automatic cross-source association.
Finding or error example
Example / output
Configure at least one required source; checks without a source are skipped.
Complete command help
Example / output
Run related Metrics/Trace checks and summarize their unchanged results by WAL step
Usage:
wp diag scenario <name|all> [flags]
Flags:
-h, --help help for scenario
wp diag trace check
Run one check; no implicit execution of other checks
Example / output
wp diag trace check <name> [flags]
Parameters to know
Parameter / type
Use and default
--threshold float
Override this check threshold in native units (Trace durations use ms; error counts allow zero)
Demo
Example / output
wp --context local diag trace check server-append-duration --window 15m --details all -o json > spans.json
Output and interpretation
Example / output
A source-local report for selected retained append spans.
Sampling, retention and emitted tags bound the evidence. No spans does not imply zero operation latency.
Finding or error example
Example / output
Missing spans or a query budget prevent a full assessment.
Complete command help
Example / output
Run one check; no implicit execution of other checks
Usage:
wp diag trace check <name> [flags]
Flags:
-h, --help help for check
--threshold float Override this check threshold in native units (Trace durations use ms; error counts allow zero)
Troubleshooting
Choose one of three troubleshooting paths. Each starts with a short command plan, then shows detailed steps, result interpretation and recovery verification.
CRC/read errors, short replica coverage or replica coverage disagrees with the expected boundary.
Metadata/data evidence → scan → probe/inspect → replica recovery, verified repair or accepted loss → application verification → checkpoint-backed retention.
Before running the steps
Complete CLI setup, including reachable admin origins for every quorum member. Each scenario includes a copyable connection block; replace its illustrative names and IDs with discovered values. Etcd parameters are used only by metadata-consuming commands.
Use the CLI built from the documented source checkout. Examples are illustrative excerpts; actual names, IDs and outcomes come from your deployment.
How to read a diagnostic result
Result
What it means
Idle / caught up
No new work may be available. Confirm workload and tail before declaring a stall.
Unknown / unreachable / cannot answer
Evidence is incomplete. Restore transport, mappings or target location before treating it as missing data.
Short / checksum failure / all replicas fail
Correlate with the finalized metadata boundary and identify the exact affected entry range.
Compacted / object_storage
Scan does not verify shared objects; probe can read them. Replies from three nodes may represent one shared copy.
Preview / acceptance not given
No intervention was committed. meta patch/truncate previews normally exit 0; skip-range/fencing previews can exit 7 because acceptance was not given.
meta check: invalid / exit 9
Metadata constraints failed; inspect issues. An independently verified metadata repair uses the guarded patch workflow, while physical damage needs its own recovery.
Command succeeded
Verify the application checkpoint/append result. An admin RPC succeeding is not the incident's closure.
Decide whether the reader is caught up, not reporting, unable to reach a replica, or blocked by unreadable data.
Plan: position → tail → actual reads → recovery
Step
Command
Intent
1
log readers, twice
Identify the reader and determine whether its reported position moves.
2
meta show/check; logstore lac; log scan
Compare the reader with metadata constraints, confirmed progress and physical coverage.
3
segment probe
Ask all quorum members to actually read at the suspected position.
4
segment inspect; ops list/show
Locate damaged blocks/records or a long-running request; distinguish data failures from transport failures.
5
Repeat the reader sample and application read
Verify the same reader resumes and reaches the expected tail after the appropriate recovery.
Set the example connection and log before running the commands
Complete CLI setup first. Replace the example context, log name/ID, segment, node and storage space with values from your cluster. Check your CLI version and operation help before running these examples.
Use w for admin-only operations and wm for commands that consume coordinator metadata. Add the authentication/TLS parameters from CLI setup to wm when required. Node URLs/mappings come from the selected context. Output below is illustrative and abbreviated; adapt positions to your actual evidence.
Current source builds can also save etcd settings in the selected context. The explicit flags in wm override that context for each invocation, which is useful when switching between different metadata namespaces.
Readers normally report on a 30-second cadence. A single unchanged checkpoint is not a stall. For example, the application last returned 7:49 and is waiting for 7:50; the registry can report that pending position:
Example / output
READER OPEN_POS CURRENT LAST_REPORT OPEN_FOR STATE
reader-123 7:0 7:50 3s 5m live
# A later sample still reports CURRENT 7:50.
Result
Interpretation / next step
Position advances
Reading is progressing. Compare application consumption/processing latency instead of treating this as a Woodpecker read stall.
Fresh report, unchanged position
The reader is reporting, but could simply be caught up. Compare with the confirmed tail next.
stale report
Reporting is not keeping up. Inspect client logs, blocked calls and connectivity; this alone does not prove corruption.
no read since open
No new position has been reported since opening; this is not proof of an unreadable first entry.
no live readers
No usable leased reader registration exists under this metadata namespace. Check context/prefix, client activity and leases. Milvus query traffic does not imply an open WAL reader.
CURRENT is the reader's reported position at a report tick, not a count of delivered records or an exact last-returned entry. Use application logs/checkpoints to identify the next entry actually required.
2. Compare the confirmed tail and metadata
Example / output
wm meta log show "$WP_LOG"
wm meta segment show "$WP_LOG" "$WP_SEG"
wm meta check "$WP_LOG"
wm logstore lac "$WP_LOG" "$WP_SEG"
wm log scan "$WP_LOG" --from-segment "$WP_SEG" --to-segment "$WP_SEG" \
--bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
Example / output
# Illustrative scan summary:
SEGMENT STATE META LAST REACHES REPLICAS VERDICT
7 Completed 99 0-99 3/3 ok
NODE_ID DURABLE_ENTRY STATE
milvus-woodpecker-0 99 ok
milvus-woodpecker-1 99 ok
milvus-woodpecker-2 99 ok
quorum LAC: 99 (aq=2 of 3 nodes at or past it)
If the application needs 50 and the finalized boundary is 99, there should be data to read. If an Active reader is already at the current tail, waiting can be normal. Active metadata is not a final boundary. LAC establishes confirmed progress, not checksum integrity; for Completed/Sealed segments the CLI can use metadata when no live writer remains.
check returns state=ok/issues=[] for metadata constraints only; state=invalid/exit 9 names inconsistent fields, identity or quorum. If a software bug is independently proven, use the controlled metadata repair workflow. An Active lastEntryId=-1 is normal and must not be replaced by an observed LAC to make it look finalized.
A readable Completed segment returns the requested range through its final entry. These excerpts keep only the result columns:
Example / output
# Normal: all three nodes reach the expected final entry 99.
NODE_ID SOURCE SERVED OUTCOME VERDICT
milvus-woodpecker-0 local_disk 50-99 (50) end_of_file served
milvus-woodpecker-1 local_disk 50-99 (50) end_of_file served
milvus-woodpecker-2 local_disk 50-99 (50) end_of_file served
# Abnormal: entry 50 cannot be read on any copy.
NODE_ID SOURCE SERVED OUTCOME VERDICT
milvus-woodpecker-0 local_disk nothing error cannot serve
milvus-woodpecker-1 local_disk nothing error cannot serve
milvus-woodpecker-2 local_disk nothing error cannot serve
SERVED=50-99 (50) means 50 returned entries, IDs 50 through 99 inclusive. nothing means no entry was returned from this requested start, not that the entire segment is empty. Read DETAIL for the actual error before deciding on recovery.
Probe result
Meaning / action
served, end_of_file at 99
The requested finalized tail was readable. Investigate client handling, retries or processing above Woodpecker.
cap_reached
The requested window ended; this is not EOF. Increase max-entries or move from-entry.
Active: entry_not_found / no local data
The final boundary is unknown. It may be an unwritten/caught-up tail; pending writes are not established by a no-local-data response.
One or two replicas fail; another reaches 99
Failover can cover the failed copies. Preserve the healthy copy and diagnose the unhealthy nodes.
All reachable replicas fail at required entry 50
There is no usable local alternative at that position. Inspect the exact damage and healthy tail before any data-loss decision.
unreachable / not in memberlist / HTTP 404 / cannot answer
Coverage is not established. Fix connectivity, mapping or storage-instance selection; do not interpret a failed request as confirmed data loss.
For Sealed segments, SOURCE=object_storage means the probe reads the shared compacted object. Three such replies are three reads of one shared copy, not three independent copies. See probe parameters and output fields.
4. Locate the blocker
Example / output
wm segment inspect "$WP_LOG" "$WP_SEG" --max-blocks 4096 \
--bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
w ops list "$WP_NODE" --type logstore.get_batch_entries \
--log "$WP_LOG_ID" --seg "$WP_SEG" --longer-than 30000
# Copy a real OP_ID from the list:
w ops show "$WP_NODE" --op-id REAL_OP_ID
Inspect walks beyond the first damaged block and can expose recoverable records after the bad entry. checksum_failed/record_checksum_failed differs from a bound or missing local data. Ops list is only the instantaneous in-flight registry; empty output on one node does not prove the client is idle. Check the other quorum nodes too.
5. Choose recovery and prove progress
Evidence
Recovery
Verify
Endpoint/mapping failure
Restore the relevant connection or node availability.
Probe now returns usable results; application reader progresses.
Healthy tail; no physical fault
Inspect/fix the client retry, reporting or application processing path.
Same reader resumes; application checkpoint advances.
Confirmed unrecoverable range on every copy
Follow the corruption workflow, including explicit acceptance of any skipped entries.
Same reader passes the range and reaches the verified healthy tail; record the accepted loss.
The CLI probe confirms data readability; it does not automatically advance the application's reader. The closure is the same reader progressing plus application-level read success, not just a green command.
Write Stalls
Find where append progress stops: application submission, node operations, buffering/flush, or quorum confirmation.
Confirm the app is submitting and the relevant replicas are reachable.
2
ops list/show/stats
Locate long-running append or flush operations without mistaking idle for a stall.
3
logstore segments/segment-show/buffer/flush-queue
Identify the loaded writer and its buffer or flush bottleneck.
4
logstore lac, twice
Determine whether the ack quorum's durable frontier advances.
5
Targeted fix, or reviewed fence-quorum
Restore the failing stage, then verify application append success and LAC progress.
Set the example connection and log before running the commands
Complete CLI setup first. Replace the example context, log name/ID, segment, node and storage space with values from your cluster. Check your CLI version and operation help before running these examples.
Use w for admin-only operations and wm for commands that consume coordinator metadata. Add the authentication/TLS parameters from CLI setup to wm when required. Node URLs/mappings come from the selected context. Output below is illustrative and abbreviated; adapt positions to your actual evidence.
Current source builds can also save etcd settings in the selected context. The explicit flags in wm override that context for each invocation, which is useful when switching between different metadata namespaces.
1. Confirm activity and reachability
Example / output
w node list --strict
w node healthz "$WP_NODE"
First inspect the application's append attempts, returned errors and deadlines. Healthz answers process health, not write progress. One slow replica can be absorbed by the ack quorum; a cluster health color does not identify which segment stopped.
Result
Interpretation
All nodes reachable; no application append attempts
Start with workload/submission. There is no server-side write stall to prove yet.
App submits but no relevant operation reaches any replica
Check the client's queue, RPC path and discovery/configuration before the node.
Replica unreachable
Check the address mapping/port-forward, pod state and network; then measure whether the remaining quorum still confirms writes.
2. Find the long-running operation
Example / output
w ops list "$WP_NODE" --log "$WP_LOG_ID" --seg "$WP_SEG" \
--longer-than 30000 -o json
w ops stats "$WP_NODE"
# Use an ID actually returned above:
w ops show "$WP_NODE" --op-id REAL_OP_ID
Repeat on all quorum members. Compare IDs and ages across samples: the same append persisting is different from many short-lived appends. file.flush points at the flush path. Empty output is an instant snapshot; registry evictions and app work that never reached a node can be invisible.
Output
Interpretation
ops list has no matching items; stats in_use=0
No tracked operation is in flight at this instant. With no workload this is normal; with pending client appends, check submission and other nodes.
The same OP_ID remains older than 30000ms in consecutive samples
A tracked operation is long-running. Use show to inspect its type, context and stage.
ops show returns TargetNotFound
The operation finished or was evicted before the query. Refresh list rather than treating it as a missing segment.
stats old eviction total grows
Old operations are being dropped from this bounded registry. Investigate alongside queues and errors; the cumulative total alone does not prove a current stall.
3. Inspect the loaded writer
Example / output
w logstore segments "$WP_NODE" --log "$WP_LOG_ID"
w logstore segment-show "$WP_NODE" --log "$WP_LOG_ID" --seg "$WP_SEG"
w logstore buffer "$WP_NODE"
w logstore flush-queue "$WP_NODE"
These are current occupancies, not latency measurements. Re-run while appends are outstanding: buffers draining and queue depth returning toward zero indicate progress; growing or persistently full queues need correlation with flush operations and backend errors. A single nonzero sample can be normal batching.
Pattern
Meaning / next check
Buffer/queue increases; flush operation does not finish
Likely upstream inactivity or a client/RPC issue; not evidence of a blocked flush.
segments shows only headers / segment-show is absent
These list loaded writer objects, not all persisted segments. Confirm the current segment and owner; do not infer no data.
One node behind; others advance
Measure quorum confirmation. A lagging replica need not stall the application.
Read persisted metadata when the writer registry is insufficient:
Example / output
wm meta segment list "$WP_LOG"
wm meta segment show "$WP_LOG" "$WP_SEG"
wm meta check "$WP_LOG"
wm log scan "$WP_LOG" --from-segment "$WP_SEG" --to-segment "$WP_SEG" \
--bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
meta list/show include unloaded history. check: state=ok validates metadata only; state=invalid/exit 9 identifies a constraint failure. For a proven metadata bug, use the offline repair workflow and restart affected clients afterward. Editing Active lastEntryId/state is not a substitute for writer recovery, and truncate is a retention operation rather than a fix for pending appends.
NODE_ID DURABLE_ENTRY STATE
milvus-woodpecker-0 99 ok
milvus-woodpecker-1 99 ok
milvus-woodpecker-2 70 ok
quorum LAC: 99 (aq=2 of 3 nodes at or past it)
Result
Interpretation
LAC advances
Durability confirmation progresses. Investigate client completion/latency rather than assuming every append is blocked.
LAC unchanged while appends remain outstanding
Confirmation is not advancing. Combine queues, errors and quorum-member evidence to locate the stage.
Fewer than AQ usable replies
LAC is unknown; restore reachability or replica availability. Unknown is not a confirmed smaller frontier.
no writer
Node answered but has no loaded writer for this segment. Completed/Sealed metadata may still establish a boundary; check whether the application has rolled to a newer segment.
The CLI's --timeout bounds its own admin/metadata requests, not the application's append deadline. A 2s send budget does not guarantee an end-to-end append within 2s: confirmation, retries and rolling add time.
5. Restore the stage, then re-test append
Fix verified connectivity, storage or deployment failures first. A manual flush applies to a loaded writer and does not repair corrupted data:
Example / output
w logstore force-flush "$WP_NODE" --log "$WP_LOG_ID" --seg "$WP_SEG"
If the intended recovery is to fence the old writer and let the application roll/reopen through its normal recovery path, review quorum impact first:
Example / output
# Preview: no -y means no commit and exit code 7.
wm logstore fence-quorum "$WP_LOG" "$WP_SEG" \
--reason "Reviewed recovery of stalled writer"
# After reviewing the selected nodes and impact:
wm logstore fence-quorum "$WP_LOG" "$WP_SEG" \
--reason "Reviewed recovery of stalled writer" -y
Fencing can interrupt writes. The default WQ=3/AQ=2 needs two selected members to cross the interruption gate; fencing only one does not guarantee an interrupted quorum. StateConflict/insufficient reachable members means no verified quorum intervention, not success. Check fence-quorum for selection details.
6. Verify at the application and quorum
Example / output
w ops list "$WP_NODE" --log "$WP_LOG_ID" --longer-than 30000
wm log scan "$WP_LOG" --mode quick \
--bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
# Select the actual current segment if rolling changed it:
wm logstore lac "$WP_LOG" "$WP_SEG"
Require new application appends to succeed, the durable frontier to advance under workload, and old operations/queues to clear. A fence RPC or healthy pod alone is not the recovery result. If physical reads fail or CRC errors appear, switch to the corruption workflow.
Data Corruption Diagnosis & Recovery
Establish the metadata boundary, identify readable copies and exact bad entries, then choose recovery without discarding healthy data.
Preserve decoded/raw metadata and original payloads; distinguish inconsistent metadata from physical damage.
2
log scan quick/raw
Identify suspicious segments and distinguish failed inspection from confirmed missing coverage.
3
segment probe/inspect
Find which copies read, where sequential reading stops and which later records remain recoverable.
4
Replica recovery, verified skip-range, or meta segment patch
Choose based on healthy copies, accepted loss or an independently proven metadata bug.
5
Application verification; log truncate; marking if retiring data
Verify recovery/checkpoints, then separately observe auditor/retention closure.
Set the example connection and log before running the commands
Complete CLI setup first. Replace the example context, log name/ID, segment, node and storage space with values from your cluster. Check your CLI version and operation help before running these examples.
Use w for admin-only operations and wm for commands that consume coordinator metadata. Add the authentication/TLS parameters from CLI setup to wm when required. Node URLs/mappings come from the selected context. Output below is illustrative and abbreviated; adapt positions to your actual evidence.
Current source builds can also save etcd settings in the selected context. The explicit flags in wm override that context for each invocation, which is useful when switching between different metadata namespaces.
1. Preserve evidence
Record the application checkpoint, log name/ID, segment ID, deployment version and replica locations. Preserve original replica files and relevant objects through your deployment's backup procedure before changing them. Capture the scan, probe and inspect results below; they are diagnostics, not a data backup.
Example / output
wm meta export --output-file ./metadata-before-corruption.json
wm meta log show "$WP_LOG"
wm meta segment show "$WP_LOG" "$WP_SEG"
wm meta check "$WP_LOG"
Export creates a new file containing readable known metadata and byte-exact original values, revisions and leases. It does not back up replica files/objects. check exits 0 with state=ok or 9 with state=invalid and issues; either result is metadata-only. A valid Completed boundary of 99 expects entries 0..99, but check cannot establish that those payloads exist. If independently verified data/history proves the boundary is wrong, follow metadata repair.
# Example normal summary (DETAIL column omitted):
SEGMENT STATE META LAST REACHES REPLICAS VERDICT
7 Completed 99 0-99 3/3 ok
8 Active -1 (open) - 3/3 open
# Example missing finalized tail, after a complete local survey:
SEGMENT STATE META LAST REACHES REPLICAS VERDICT
7 Completed 99 0-49 3/3 short
# Example failed inspection requests:
SEGMENT STATE META LAST REACHES REPLICAS VERDICT
7 Completed 99 - 0/3 unknown
The short example establishes a local coverage gap at 50..99; it does not explain the cause. Inspect/probe next. The unknown example establishes no coverage at all; inspect the per-replica DETAIL (for example HTTP 404, timeout or missing mapping) before deciding there is missing data.
Verdict
What was established
ok
Observed local coverage matches the metadata expectation within this survey.
short
Every relevant replica survey completed, and some expected range is absent from their combined local coverage. Any incomplete/unreachable alternative makes that missing range unknown.
unknown
Request, location, response or survey bound prevented a conclusion. Read the per-replica HTTP/error details.
open / local_empty
Active boundary remains unknown; replicas found no local durable entries. This is not automatically corruption.
compacted
Local copies are not authoritative. Scan has not verified shared object-storage data; use probe for actual reads.
reclaimed
Truncated data is intentionally eligible for cleanup; absence is not automatically loss.
REACHES is the union of observed local ranges; '-' is no observed range. REPLICAS counts usable replies, not independent healthy copies. quick checks structure; raw verifies local block checksums. Neither establishes pending writes or end-to-end object readability.
# Two failed copies and one healthy copy, abbreviated:
NODE SERVED OUTCOME VERDICT
milvus-woodpecker-0 0-49 (50) error cannot serve
milvus-woodpecker-1 0-49 (50) error cannot serve
milvus-woodpecker-2 0-99 (100) end_of_file served
The healthy copy can support reader failover. Preserve it; do not declare readable entries lost. The CLI does not offer a generic copy/repair command. Recover damaged replicas through the supported deployment/recovery path, then verify their coverage. Compaction can still fail if it requires damaged copies.
Example / output
# If all three instead stop before entry 50:
No replica can serve entry 50 ...
# Inspect may separately identify:
STATUS: checksum_failed
record stop: record_checksum_failed at entry 50
recoverable entries after the bad record: 51..99
Probe stops where sequential reading fails; inspect surveys beyond a bad block and can expose the tail. Confirm exact entry boundaries and each copy's result. A timeout, ambiguous path, 404 or cap_reached is not proof that an entry is unrecoverable. Object_storage replies inspect one shared copy, not three independent ones.
Example / output
# Inspect excerpt for one node, assuming entry 50 occupies its own block:
BLOCK ENTRIES READABLE RECOVERABLE STATUS
0 0-49 all - ok
1 50-50 none - checksum_failed
2 51-99 all - ok
READABLE=none means normal reads reject that block. RECOVERABLE describes individually verified records inside a rejected block; it is evidence for repair, not a promise that the SDK can return them. Here the later block is independently valid. If entry 50 instead shares a damaged block with other entries, verify that whole block's consequences rather than assuming only 50 is affected.
4A. Accept a confirmed unreadable range
Use this branch only when every usable copy fails and the business accepts loss of the specific entries. The example assumes only entry 50 is confirmed unrecoverable, with 51..99 verified readable:
Example / output
# Preview and check the evidence/gate; no -y means no commit (exit 7).
wm log skip-range add "$WP_LOG" "$WP_SEG" \
--from-entry 50 --to-entry 50 --reason "All copies fail at 50; verified healthy tail"
# After explicit acceptance of losing entry 50:
wm log skip-range add "$WP_LOG" "$WP_SEG" \
--from-entry 50 --to-entry 50 --reason "Accepted loss of entry 50" -y
wm log skip-range list "$WP_LOG"
Example / output
# Successful add for log ID 42, segment 7:
Declared: log 42 segment 7 now skips 50-50
Result
Meaning
Gate refuses because a copy can serve
Preserve/read that copy; do not force past healthy data just to make a command pass.
Acceptance not given
Expected preview behavior, no declaration committed.
Declaration committed
Readers can skip the inclusive range. Files remain damaged; compaction is not repaired by the declaration.
Keep the same stalled reader running. Wait for refresh/report timing and verify it crosses 50 and reaches 99. For a reader returning original entries 0..99, skipping just 50 means 99 returned records; record IDs and application checkpoints establish what was lost. Removing the declaration does not restore data.
4B. Correct a proven metadata bug
Use this branch only when independent file validation, acknowledged-write history and the responsible software fix establish the correct metadata. A replica losing acknowledged data does not justify lowering the expected boundary. Example: metadata incorrectly says lastEntryId=999, but verified history proves this finalized segment ended at 99.
Review before/after/changes, then stop relevant writers/auditors and keep them stopped. --offline only acknowledges this prerequisite. Use the target segment's before.revision (174 in this example), not the log or snapshot revision:
Example / output
wm meta segment patch "$WP_LOG" "$WP_SEG" --patch-file ./segment-repair.json \
--apply --offline --expected-revision 174 \
--reason 'Correct independently verified tail after software fix' \
--backup ./segment-before-repair.json
wm meta segment show "$WP_LOG" "$WP_SEG"
wm meta check "$WP_LOG"
Result
Interpretation / action
applied=true, new_revision, audit_key, backup
The metadata write committed. Re-read the record, restart affected clients to refresh caches, and verify physical reads/application progress.
The correction is rejected. Resolve the identity or invariant with verified evidence; do not rewrite state simply to make check pass.
warnings: embedded quorum (unchanged)
An independent scalar repair may proceed, preserving that bad quorum. check still reports the unresolved quorum; this is not a clean recovery result.
Writer lock / stale revision / concurrent update
No correction was committed by this attempt. Stop the metadata writer or refresh the preview; use a new backup filename for retry.
Commit network error
The outcome may be unknown. Inspect current metadata and audit records before retrying.
State and timestamp correction uses the same flow; follow the metadata field rules and examples. A state edit cannot create compacted files or repair replicas. Log truncation fields are protected from generic patching.
5. Close the compaction/retention loop
Reader recovery and compaction are separate. Skip-range does not fix a damaged Completed copy. Later eligible segments can be compacted independently; the damaged segment may continue failing until repaired or deliberately retired.
After verifying resumed application progress and a durable checkpoint beyond the damaged segment, advance normal retention with log truncate. The example assumes segment 8 is non-empty Completed/Sealed and its entry 0 is a verified safe boundary. Advancing into 8 permits retiring segment 7; truncating to 7:99 retains segment 7 itself.
Example / output
wm meta segment show "$WP_LOG" 8
wm log truncate "$WP_LOG" --segment 8 --entry 0
The preview has applied=false, unchanged=false, before/after and target_segment_revision. Use the log before.revision for apply (17099 in this illustrative example), not the target segment revision. No --offline flag is needed for this normal retention operation; coordinate it with the application's checkpoint policy.
applied=true/new_revision confirms the frontier update and audit record, not immediate deletion. An unchanged frontier returns unchanged=true/applied=false with no new artifacts. Active/empty targets, backward movement, bad identity, missing apply guards and stale/concurrent revisions are rejected; inspect the message and preview again rather than forcing a change.
A live application log handle/auditor marks earlier finalized segments Truncated in subsequent rounds, then applies retention and reader protection. No active auditor means the background transition will not occur. Physical deletion is asynchronous; a metadata backup cannot restore deleted files. Use scan and marking to observe the result:
Compacted-mark notification is pending manual handling
Inspect marking records and verify the unacked node's disposition before using wm marking confirm "$WP_LOG_ID" "$WP_SEG". It settles notification retries, not data corruption or segment metadata, and the record remains until truncation reaps it.
6. Verify the chosen recovery
Example / output
sleep 35
wm log readers "$WP_LOG"
# For read/metadata/replica recovery, while this segment remains retained:
wm segment probe "$WP_LOG" "$WP_SEG" --from-entry 51 --max-entries 100 \
--bucket "$WP_BUCKET" --root-path "$WP_DATA_ROOT"
For skip-range recovery, require the same reader and application checkpoint to progress through the verified tail. For metadata repair, restart affected clients and verify the resumed reader/application checkpoint plus physical reads. For replica repair, re-probe/inspect every repaired copy. For retirement, expect Truncated/reclaimed rather than readable data and verify the auditor no longer attempts compaction on the retired segment; do not require reading data intentionally reclaimed. In a controlled lab, restore original files before removing skip declarations and verifying the original full record count.
Milvus–Woodpecker Stack Examples
Install, observe and operate Milvus with Woodpecker through a complete, open-source example stack.
Why this stack exists
A running Milvus + Woodpecker deployment is easier to operate when application behavior, WAL health, resource pressure and alerts can be examined together. Users who have not yet deployed monitoring or operational tools can start here with a working example, then adapt its configuration and investigation methods to their own instances.
The default deployment contains Milvus and independent Woodpecker services with persistent dependencies. Add Prometheus, Grafana, Alertmanager, Jaeger and Loki/Alloy independently, and use wpcli for operations. A real Milvus workload exercises its WP client and the WP servers, so the example explains what to observe along the write path and how to investigate it.
The runnable example lives in deployments/milvus-full-stack-helm/. It uses the official Milvus Helm chart, a small support chart and pinned monitoring components. Values files, dashboard JSON, alert rules, workload scripts and the notebook are included in the repository.
The monitoring and alerting guide connects these E2E views with the Milvus/WP reference dashboards, scrape verification and Alertmanager.
One operational feedback loop
Install a known environment. Record chart/image versions, namespace, resource settings and connection targets.
Run a real workload. Verify Milvus inserts, primary-key queries and vector search, then establish a small write-performance baseline.
Check the whole chain. Compare application results with Milvus/WP metrics, dashboards, alerts, wpcli state and logs for the same time window.
Improve and repeat. Investigate failures or missing signals, change one setting or instrument, and rerun the same scenario.
Commands return exit status and save structured artifacts so humans and local AI agents can follow the same loop. The current stack provides these building blocks; autonomous investigation and optimization are future extensions.
Apply the examples to your own instances
Start with a local stack to learn the expected signals and command output. For an existing Milvus + WP instance, adapt scrape selectors, dashboard variables, alert thresholds and notification routes to its topology and workload. Use that instance’s reachable WP Admin URLs and its own etcd namespace and storage paths when configuring wpcli.
Distinguish the shared WP service cluster from the metadata managed by a particular Milvus instance’s WP client. The two connection scopes are explicit in the generated configuration and operations examples.
The supplied resource sizes, single-node dependencies and example credentials are for a development environment. Use your deployment’s availability, access and storage requirements when adopting the configurations. Performance results from this local setup describe that setup.
A growing examples collection
The Helm stack, monitoring/alert checks, wpcli connection workflow and bounded write experiment are available now. Future chapters can add controlled failure and recovery drills, E2E operational diagnosis, capacity and configuration comparisons, and measured optimization examples. Each chapter should include its trigger, workload, expected signals, investigation, recovery, verification and evidence.
An Operator installation path is also planned separately. This collection will grow around concrete scenarios that help users improve the operability and observability of their own Milvus + WP instances.
Install and operate Milvus + Woodpecker on your existing Kubernetes cluster using reviewable Helm and kubectl commands.
Prepare an existing Kubernetes context
Use configured kubectl, Helm 3 and Python 3.9+. Kubernetes, its nodes, credentials, StorageClass and registry access are managed outside this tool. This applies equally to a shared cluster, kind or local minikube. Docker is optional for offline rule tests; Jupyter needs Python 3.11+.
Commands
git clone https://github.com/zilliztech/woodpecker.git
cd woodpecker/deployments/milvus-full-stack-helm
python3 -m pip install -r scripts/requirements-connections.txt
kubectl config current-context
kubectl version
./stack.sh help
The default context is kubectl’s current context; override it with --context NAME or CONTEXT. The default namespace is milvus-stack, configurable with --namespace / NAMESPACE. An unavailable context produces configuration guidance; stack.sh never creates, starts, stops or deletes Kubernetes.
If configuration is incomplete, stack.sh distinguishes missing/unreadable kubeconfig files, an empty context list, an unselected/unknown context and API connection failures. It lists available context names and copyable commands, including ./stack.sh status --context YOUR_CONTEXT. No context is switched automatically, and credentials are not printed. See configuration diagnostics.
Review, then install
Commands
./stack.sh render
./stack.sh images # Image inventory only
./stack.sh install # Preview native commands
./stack.sh install -x # Execute the reviewed resource operations
./stack.sh status
Resource changes, workload writes and port forwards are previewed by default; -x executes them. Planning may read the cluster and save local values/manifests; it does not change resources. Read-only status and local rendering remain available without -x. Printed Helm/kubectl commands include the explicit target and can be copied directly.
A first deployment contains only Milvus, three independent WP services and persistent etcd/MinIO. It wraps the official Milvus Helm chart and needs no monitoring CRDs. Public defaults use Milvus v3.0.3 and WP v0.1.46. Add --local-arm64 to render/images/install for the supplied private ARM64 build; prepare its image access outside the stack.
Use --with all for the complete observability example. Loki includes Alloy for log collection. Addon changes preserve live core overrides and previously selected components; changing Jaeger selection updates trace configuration and rolls Milvus/WP. See component dependencies and lifecycle.
Discover and open endpoints
Commands
./stack.sh forward --wpcli # Preview native port-forward commands
./stack.sh forward --wpcli -x # Keep running in a separate terminal
./stack.sh connections
./stack.sh connections --json
Only installed services are forwarded. --wpcli also includes per-node WP Admin and tenant etcd. Matching existing forwards are reused, and Ctrl+C closes only those started by this invocation. Only addresses marked forwarded are verified local connections; Admin HTTP forwarding does not establish SDK advertised-gRPC reachability.
Run the first verification
Commands
./stack.sh smoke -x
# Allow two 15-second scrapes when Prometheus is installed.
./stack.sh verify
./stack.sh alerts
Smoke writes real data and checks query/search. Verify requires Prometheus, with Grafana provisioning checks when selected. The alert flow uses Prometheus + Alertmanager. Core-only perf still checks writes/reads and explicitly marks absent metrics/alerts as skipped. A running Pod alone does not establish a healthy chain.
Follow the Jupyter guide
Commands
./stack.sh notebook
Open the authenticated loopback URL and select notebooks/full-stack-walkthrough.ipynb. It selects the complete observability profile. EXECUTE=False previews by default; set EXECUTE=True after reviewing the target, images, ports and load. The guide never manages Kubernetes and keeps stop/uninstall cells disabled by default.
Pause/resume workloads or remove the deployment
Command
Effect with -x
Data boundary
install
Install/upgrade Helm releases on the existing cluster and run the selected stack.
Retain PVCs and existing addons.
stop
Record replica state and pause this stack’s workloads using native scale/patch/wait commands.
Retain Helm releases/PVCs; memory/ephemeral history may disappear.
start
Restore the replica counts saved by stop and wait for rollouts.
Does not install a missing stack or start Kubernetes.
uninstall
Uninstall stack releases and delete their PVC data.
Retain Kubernetes, namespace, shared CRDs and local evidence.
addons-install / addons-uninstall
Add components / remove selected components and their data.
Retain the core deployment.
Commands
./stack.sh stop # Review only
./stack.sh stop -x # Pause workloads; leave Kubernetes running
./stack.sh start -x # Restore saved workload replicas
./stack.sh forward --wpcli -x
./stack.sh uninstall # Review the deletion plan
# ./stack.sh uninstall -x # Executes deletion of releases AND PVC data
Stop state is stored in the namespace ConfigMap milvus-full-stack-lifecycle, so resume does not depend on one machine. Prometheus/Alertmanager are scaled through their Operator resources, and Alloy’s DaemonSet uses a reversible scheduling selector. Resume before changing Helm releases. Unsafe scale-down PVC retention policies and active autoscalers are reported before any write.
Follow the same workload from Milvus through the WP client and servers, then check that the operational signals reach your dashboards and Alertmanager.
Each component is optional and can be added after the core deployment. Grafana displays data from the selected backends; it does not store their metrics, logs or spans. Existing forwards need a new invocation to include newly installed services. See independent selection and removal.
Start with an E2E operational dashboard
For a specific symptom, use the E2E dashboards: Write / Append, WAL Read / Consume, Flush / Checkpoint, or Search / Query. They group the metrics by processing step and include investigation guidance. Use the three component reference dashboards below for deeper inspection.
Open the three reference dashboards
Keep ./stack.sh forward --wpcli -x running. These are the default local addresses; use ./stack.sh connections for discovered addresses when ports differ.
View
Local address
What to examine
Milvus
http://127.0.0.1:3000/d/milvus-full-stack
Application throughput, latency and component/resource health.
WP client
http://127.0.0.1:3000/d/woodpecker-client-k8s
Client append/read behavior, latency and queues.
WP server
http://127.0.0.1:3000/d/woodpecker-server-k8s
Node operations, flush/storage behavior and resource pressure.
Prometheus
http://127.0.0.1:9099
Scrape target health, raw series, queries and pending/firing alerts.
Alertmanager
http://127.0.0.1:9093
Received alerts, grouping, inhibition and silences.
Loki API / Grafana Explore
http://127.0.0.1:3100 / Grafana Loki datasource
Search collected container logs by namespace, Pod, container and text.
Jaeger
http://127.0.0.1:16686
Individual request spans and cross-service trace coverage.
Grafana’s local example login is admin / admin. Dashboards and the Prometheus data source are provisioned from repository files. Select namespace milvus-stack and a time range containing the test; allow scrapes to collect the new workload.
Check the monitoring chain
Commands
./stack.sh smoke -x
# Wait at least 30 seconds for scrapes.
./stack.sh verify
The verification report is saved to .local/verification.json. It checks target health, positive application/WP workload metrics, dashboard provisioning and query compatibility. Some panels need a particular operation or enough rate-window history before showing finite samples. An empty panel is an investigation prompt, not proof that the corresponding operation is healthy or broken.
If application writes succeed but WP activity is missing, check the deployed Milvus configuration, metric families and label selection before drawing a conclusion about the write path. The dashboard source record documents the Milvus production export and adaptations of the WP Kubernetes templates.
Inspect individual request traces
When selected, the stack deploys jaegertracing/all-in-one:1.53.0 and exports Milvus and WP spans over OTLP. Open Jaeger directly, or select the provisioned Jaeger datasource in Grafana Explore.
The smoke probe writes 100 vectors and verifies Strong query/search, then saves exact request trace IDs, client latency and raw spans. Normal perf records also contain sampled trace links. The supplied image currently connects query/search through SN and segcore, while its user Insert trace does not continue into SN/WP. Separate WP exporter activity is not proof of an E2E write trace; the report explicitly records this gap as partial.
The historical full-stack tracing receipt records tested custom-image relationships and remaining limits. The original published Milvus-image acceptance gate is retired under the WP-only scope decision. WP instrumentation fixes remain useful; the tool does not diagnose the surrounding Milvus engine or convert asynchronous links into serial latency.
Full sampling favors small diagnostic runs and changes system overhead. Lower both server and workload sampling for sustained tests. Jaeger history is bounded and in memory, so save evidence before eviction or restart. See tracing setup, interpretation and known gaps.
Search real Pod logs
Loki stores compressed log chunks and label indexes on a 4 GiB PVC, with a 24-hour retention policy. Alloy tails this namespace’s container logs through the Kubernetes API. In Grafana Explore, select Loki and query {namespace="milvus-stack"}, then filter by Pod or text. When Jaeger is also selected, recognized trace IDs link to its trace view.
Commands
./stack.sh logs
./stack.sh logs-smoke -x
The smoke test runs a small Milvus write/query/search workload, finds its unique collection marker in the actual Proxy log and requires that marker to arrive through Alloy into Loki. It saves .local/logging-smoke.json. Collection is bounded by source-log retention and collector availability; see storage, queries and limitations.
Inspect and test alerting
Prometheus evaluates the 31 reference WP rules covering resource pressure, failures, scaling signals and client behavior. Firing alerts are sent to Alertmanager for grouping and inhibition. The default local-only receiver lets you inspect alerts in the Prometheus and Alertmanager UIs without sending email or other external notifications.
The synthetic check verifies the alerting path and removes its temporary resources. It does not create each underlying disk, object-store or quorum failure, and it does not prove that every production threshold is suitable for your workload. See the rule catalog, investigation guidance and validation details.
When external notification is needed, use the optional SMTP configuration example with your relay and credentials. Optional Mailpit testing captures email locally. Neither profile is enabled by default.
Adopt the signals in an existing deployment
Match Prometheus discovery selectors and labels to your Milvus and WP Pods. Confirm the exporters expose the metric families used by each query.
Adapt Grafana data-source references, namespace/instance variables and dashboard time windows.
Set alert thresholds and hold periods from your workload, capacity and recovery expectations; configure your Alertmanager receiver.
Exercise a known workload and a controlled alert-path test, then preserve evidence showing which checks passed and which signals remain untested.
Use wpcli to inspect the specific node, log or metadata behind a symptom, and the experiment workflow to compare behavior before and after an improvement.
Choose the symptom, then follow numbered processing steps from top to bottom. Each step explains what to inspect and where to continue.
Install the optional backends first: ./stack.sh addons-install --with prometheus,grafana -x, then keep ./stack.sh forward -x running.
Choose a chain
The Helm stack provisions four focused E2E dashboards alongside the three Milvus/WP reference dashboards. Keep the stack’s forwards running; Grafana defaults to http://127.0.0.1:3000. Use ./stack.sh connections for actual ports.
Every group includes visible investigation guidance, followed by throughput/status, latency or progress metrics for that step. P50/P95/P99 curves preserve actual Pod/component or node/log identities. The full guide describes each group, its metric scope, operational limits and maintenance.
Follow a slow write
Begin with Proxy completion status, mutation latency, DML queue and timestamp/primary-key preparation. Continue to producer availability and produce latency, then compare full SN WAL append with backend append and individual interceptor timers.
At the WP stages, compare pending appends, writer/replica outcomes and individual server append, buffer-wait and local-flush latency. The final group shows disk admission, usage, scheduler waiting and confirmation progress. Use wpcli and the component reference dashboards to investigate the identified node/log.
Follow delayed consumption
First distinguish real-time memory tailing from historical WAL catchup. A tailing scanner can read the SN write-ahead buffer without WP read RPCs, so zero backend read traffic alone is expected. Historical reads can use WP services or compacted segments in object storage.
Continue through reader outcomes, scanned/passed messages, ordering/transaction buffers, subscription availability and downstream processing. In the supplied Milvus build, querynode-named consumption metrics can be emitted by the SN delegator, and local subscribers bypass the remote handler. Legends retain the actual emitting process.
Normal tail EOF is displayed as an outcome. A blocking read_next can wait for new data, so its duration is not pure storage latency. The WP v0.1.46 example does not export per-reader read-frontier metrics; use wpcli for positions instead of assuming an exact backlog count.
Separate persistence and query symptoms
Use Flush / Checkpoint when consuming works but growing buffers or persistent progress lag. It separates dispatcher waiting, upload/callback execution and progress boundaries. A flusher’s “seen” TimeTick is not a durable checkpoint; idle snapshot/checkpoint age is not automatically a fault. WP compaction/offload is shown as a parallel background path.
Use Search / Query for request latency. TSafe waiting points back to consumption/visibility; scheduler waiting, segcore/segment execution, loading and result reduction describe other stages. Proxy result wait includes downstream work and transport, rather than only network latency.
Verification checks seven provisioned dashboards, their queries and the required metric families in each E2E stage. The report lists optional idle/version-specific coverage gaps separately; an UP panel cannot hide a missing WAL-stage metric. Performance reports and the Jupyter guide include E2E links for the run’s time window.
Channel and WP log filters are separate identities, resolved with metadata before isolating a log. No data is not a healthy zero, histogram overflow limits quantile resolution, and overlapping timers cannot be added or subtracted. These dashboards correlate operational evidence; live read verification and controlled fault/recovery tests establish different aspects of correctness.
Connect to the WP service cluster, attach the selected Milvus instance’s WP client metadata, and use both views during an investigation.
Prepare wpcli and local access
Install wp using the CLI setup guide, and check wp version and wp --help. The noninteractive examples work with the established operations; interactive wp shell requires a source build containing the shell addition and is not part of v0.1.47.
Run the following from deployments/milvus-full-stack-helm/, with the stack deployed and the connection-discovery Python dependency installed:
Commands
# Keep this running in a separate terminal.
./stack.sh forward --wpcli -x
This creates a separate stack-local file. Your personal ~/.woodpecker/cli.yaml is unchanged. Export requires working WP Admin forwards; metadata readiness is reported separately. Re-run discovery after changing the deployment or restarting forwards.
Understand the two connection scopes
Scope
Configuration
Purpose
WP service cluster
cli.yaml: Admin endpoint, per-node Admin URL mappings and Kubernetes context/namespace.
Inspect WP nodes and their runtime state. Select this connection with --context.
Milvus instance WP client metadata
Separate etcd endpoint and full metadata prefix; bucket/root path when an operation needs the storage identity.
Inspect metadata managed by this Milvus instance’s WP client. Supply explicit flags or attach it inside the shell.
--wpcli-config puts the second scope in an independent comment section, with a copyable connect command. It does not put etcd, metadata-prefix or storage settings into the generated ctx.
In the default example, etcd is reachable at 127.0.0.1:32379 and the complete metadata prefix is full-stack/woodpecker, derived from Milvus’s etcd.rootPath and woodpecker.meta.prefix. The bucket is milvus-bucket; the WP WAL root is milvus/wp, including the /wp suffix added by the Milvus WAL adapter.
Use the discovered values for your instance. A shared WP service cluster can serve different clients; selecting its ctx alone does not identify which client metadata you intend to inspect.
Inspect nodes and metadata
Commands
# Service-level inspection uses only the WP context.
wp --context wp-ops-v0146/milvus-stack node list
# Select this Milvus instance's metadata explicitly.
wp --context wp-ops-v0146/milvus-stack \
--etcd 127.0.0.1:32379 --meta-prefix full-stack/woodpecker \
meta log list
The first result describes service nodes. The second lists logical logs in the chosen client metadata namespace. Keep actual log/node/segment identities from these results when following a diagnosis workflow; sample IDs from another environment will refer to different data.
Keep an interactive operational session
With a binary whose help lists shell, start a session using the generated context. Without --context, an interactive terminal offers the configured contexts for selection.
Commands
wp shell --context wp-ops-v0146/milvus-stack
Inside the shell, attach the Milvus instance’s WP client metadata explicitly:
Commands
connect milvus-stack --etcd 127.0.0.1:32379 --meta-prefix full-stack/woodpecker --bucket milvus-bucket --root-path milvus/wp
status --check
node list
meta log list
status --check verifies Admin and etcd access for the current session. These connection checks establish reachability; use a workload and the relevant log/data checks to establish application progress and readability.
Use the local operations workspace
Build the current wp CLI and open its local WebUI. Diagnostics finds Woodpecker problems, Operations organizes native confirmation/recovery commands, and Command Explorer exposes individual wp commands. The browser uses the same checks and commands as the terminal.
Commands
make wpcli
bin/wp webui --config /path/to/cli.yaml --listen 127.0.0.1:8795
Put Prometheus, Jaeger and Grafana connections inside the selected server context. Client metadata and storage scope remain explicit per-operation inputs. Check source access before selecting an incident interval. Successful connectivity does not establish historical data availability.
Scenarios aggregate independent Metrics/Trace checks and summarize abnormalities, unavailable evidence and execution gaps. They do not add a second root-cause engine. The current catalog has 39 Metrics and 16 Trace checks. At least one evidence source must be configured; a missing Trace source does not block Metrics checks.
Use wpcli with the dashboards
wp diagnose uses current WP Metrics/Admin evidence. wp diag queries retained Prometheus/Jaeger evidence over a lookback or explicit From/To interval. Use native commands to confirm the observed WP condition and review the operation’s own effects before executing an intervention.
The tool diagnoses Woodpecker client/server behavior, including a WP client embedded in Milvus. It does not diagnose Milvus components, query/search, indexing, persistence stages or Loki logs. The former wpdiag source/stage commands and standalone UI are retired. Their accepted receipts remain historical records.
Saved runs and optional Agent analysis belong to Diagnostics. A model call is explicit and receives supplied findings; it cannot automatically query sources or change the cluster. See the current CLI/WebUI guide for configuration, rules, report semantics and migration.
Use a repeatable workload and its operational evidence to find gaps, investigate bottlenecks and evaluate a change.
Establish a working baseline
With the stack and forwards running, start with the integrated acceptance path:
Commands
./stack.sh acceptance -x
This always runs a Milvus insert/query/search workload, explicitly flushes the collection and verifies its complete row count and search afterward, then collects evidence. Even a small workload exercises persistence metrics without waiting for automatic flush thresholds. It also tests the selected backends: Prometheus rules and metrics, Grafana dashboards, Jaeger trace coverage, Loki log delivery and the Prometheus + Alertmanager firing/recovery/inhibition path. External email is not enabled. Each stage must succeed; keep the resulting reports if one fails so you can inspect that stage. Default trace verification reports known gaps separately; use strict E2E mode when a complete connected path is required.
Run a bounded write experiment
Commands
./stack.sh perf -x
The default run uses a 5-second warm-up, a 60-second measurement window, two workers and a total target of five 100-vector insert requests per second. It then waits 35 seconds for observation. It creates and retains its own collection, checks live read/query/search results and records acknowledged throughput, errors and insert latency.
This example raises the requested load; achieved throughput depends on the workload generator and stack. Compare one change at a time with the same images, resources and parameters. REST transport and port-forward overhead are part of the measured latency, and successful live reads do not prove durability through a crash.
Follow the evidence
Each performance run writes a unique directory under .local/perf/, including:
Artifact
Use
summary.md, report.json
Results, parameters, correctness checks, missing signals, failures and workload/collection observations.
requests.jsonl
Insert attempts, row-ID ranges, timestamps, latency and errors.
signals.jsonl, metric-*.json
Metric/alert snapshots and exact queries over the run’s time range.
pods.json, events.txt, logs
Component images/resources, Kubernetes events and bounded component logs.
Start from failed writes or correctness checks, then compare Milvus mutation latency, WP client append/queue behavior, server RPC/flush latency and storage/resource signals. Use node/log identities and timestamps to narrow the wpcli and log investigation. Missing lazy error counters or idle histograms are coverage gaps until the corresponding behavior is exercised.
For a wider snapshot, run ./stack.sh collect. The detailed performance guide explains gates, artifact limits and interpretation. Review logs before sharing evidence.
Turn an observation into an improvement
Record the symptom and baseline. Keep the workload parameters, timestamps, selected instance and observed result.
Locate the affected stage. Distinguish load-generator delay, Milvus processing, WP client queues, server work and object storage.
Close the relevant gap. Improve a metric, query, alert, runbook or configuration based on that evidence.
Rerun and compare. Verify application results, operational checks and the new signal under the same conditions.
Use the same approach on your own Milvus + WP deployment after adapting endpoints, labels, permissions and workload limits. Treat a local result as evidence for the measured configuration, then validate the change on the target environment.
Validate the current Woodpecker diagnostic tool
The current workflow is tracked in #503: wp CLI atomic Metrics/Trace checks, scenario summaries, native Operations, Command Explorer and optional supplied-report Agent analysis. Use wp diag and wp webui; the old wpdiag execution surface is retired.
Commands
make wpcli
python3 cmd/wpdiag/tests/workspace_acceptance.py
python3 cmd/wpdiag/tests/prometheus_smoke.py
python3 cmd/wpdiag/tests/jaeger_smoke.py
Browser acceptance uses bounded HTTP fixtures and a fake Agent; source smoke tests use isolated temporary Prometheus/Jaeger containers. These are separate from real deployment receipts. Record the actual image/build, interval, scope and source coverage when checking the stack. Missing samples, idle traffic and an incomplete Trace query are not proof of health.
Confirm WP runtime observations
Use current WP Metrics/Admin commands and explicitly scoped metadata to confirm a suspected WAL problem. The earlier native acceptance guide records its bounded service fixture and limits. Do not run fault-injection harnesses against a shared deployment. Existing metadata views/checks are native operations, not a new dynamic metadata diagnosis subsystem.
Historical foundation and acceptance records
The original #450 and finite #458 acceptance program are closed. #493 supplied the final audit. The remaining Milvus published-image propagation outcome in #459 was retired by scope decision, not counted as a passed full-chain test.
These records preserve the builds, experiments and limits of earlier implementations. Their legacy commands and Milvus stages are not supported capabilities of the current WP-only tool.
Use measured Woodpecker problems to improve check coverage, guidance and observability. Milvus remains a useful application workload and deployment example; it does not expand the diagnostic tool’s ownership boundary.
A cloud-native Write-Ahead Log storage engine that leverages object storage for low-cost, high-throughput, and reliable logging.
What is Woodpecker?
Woodpecker is a cloud-native WAL (Write-Ahead Log) storage implementation designed to fully utilize cloud-native infrastructure for scalability and durability. Unlike traditional on-premises WAL solutions or custom-built distributed logging systems, Woodpecker uses cloud object storage as its durable storage layer.
Woodpecker is currently under active development. Initially built for Milvus vector database and Zilliz Cloud, with broad applicability across cloud workloads.
Key Features
Cloud-Native WAL — Uses cloud object storage (S3, GCS, Azure, OSS) as the durable layer for scalability and cost-effectiveness
High-Throughput Writes — Optimized write strategies achieving 60-80% of maximum backend throughput
Efficient Log Reads — Memory management and prefetching strategies for optimized sequential access
Ordered Durability — Strict sequential ordering guarantees for log persistence
Flexible Deployment — Standalone service mode or embedded library mode
Multi-Cloud Support — AWS, GCP, Azure, Aliyun, Tencent Cloud, MinIO
Innovative Quorum Protocol — Configurable ensemble/write/ack quorum with AZ-aware placement
Condition Write Fence — Prevents split-brain with built-in fence mechanism
Core Components
Component
Description
Client
Read/write protocol layer for interacting with Woodpecker
EmbeddedClient
Client with built-in LogStore for embedded deployments
LogStore
Handles high-speed log writes, batching, and cloud storage uploads
ObjectStorage
Cloud object storage backend (S3, GCS, OSS, etc.)
MetadataProvider
etcd-backed metadata and coordination service
QuorumDiscovery
Quorum-based node selection and replication management
Design Benefits
Lightweight deployment — Easy integration with minimal dependencies
The two Service paths deploy the same architecture in different environments. Embedded runs its LogStore inside the application. Follow one tab from beginning to end; each includes its own configuration, commands and output interpretation.
These development examples use a released milvusdb/woodpecker:v0.1.46 server image and the v0.1.47 SDK API. Clone the current source checkout for the tutorial files added after that release. Development credentials, single-instance dependencies and small PVCs are for evaluation; use your own dependency, storage and lifecycle configuration for an operational deployment.
Kubernetes: deploy a WAL service and run an application
1. Prepare your cluster and checkout
Use Kubernetes 1.28+, kubectl, Docker, and a default StorageClass with at least 6Gi available for the example PVCs. Allocate resources for three LogStore nodes plus etcd, MinIO and an application Job. Commands assume the namespace wp-quickstart and the usual cluster DNS suffix cluster.local. Start at the repository root:
bash
git clone --depth 1 https://github.com/zilliztech/woodpecker.git
cd woodpecker
bash
kubectl config current-context
kubectl get storageclass
2. Deploy etcd, MinIO and three LogStore nodes
The complete manifest creates a dedicated namespace, dependency Services/Deployments/PVCs, a server ConfigMap, a headless Service and a three-replica StatefulSet. The headless Service publishes each node for direct quorum access.
bash
kubectl apply -f docs/quickstart/kubernetes.yaml
kubectl -n wp-quickstart rollout status deployment/qs-etcd --timeout=180s
kubectl -n wp-quickstart rollout status deployment/qs-minio --timeout=180s
kubectl -n wp-quickstart rollout status statefulset/wp --timeout=300s
kubectl -n wp-quickstart get pods
Continue when qs-etcd, qs-minio and wp-0/1/2 are Ready. Pending PVCs mean the StorageClass/capacity needs attention; ImagePullBackOff means the registry/image is unavailable. Inspect with kubectl -n wp-quickstart describe pod <pod>.
3. Use the Kubernetes client configuration
kubernetes-client.yaml uses names resolvable from the application Job in this namespace. Its metadata namespace, bucket and root path match the server ConfigMap:
A seed is discovery, not a proxy. The SDK must reach every selected node, etcd, and object storage. A single port-forward or LoadBalancer does not expose all quorum endpoints. This walkthrough runs the app inside Kubernetes; for an external SDK, provide DNS/routing to every advertised gRPC address and reachable dependencies. The wp CLI’s node-admin mappings apply to HTTP administration, not SDK gRPC traffic.
4. Build and run the application Job
The runnable client.go loads the selected YAML, creates an external etcd connection and a service client:
Client initialization · excerpt
// cfg is loaded from the path-specific client YAML below.
etcdClient, err := clientv3.New(clientv3.Config{
Endpoints: cfg.Etcd.GetEndpoints(),
DialTimeout: 5 * time.Second,
})
if err != nil {
return err
}
defer etcdClient.Close()
client, err := woodpecker.NewClient(ctx, cfg, etcdClient, false)
if err != nil {
return err
}
defer client.Close(context.Background())
It creates a fresh log, appends a message, and reads from the acknowledged position:
Write and read · excerpt
// After CreateLog / OpenLog / OpenLogWriter (see the complete program):
result := writer.Write(ctx, &wplog.WriteMessage{Payload: []byte("hello, Woodpecker")})
if result.Err != nil {
return result.Err
}
reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
if err != nil {
return err
}
defer reader.Close(context.Background())
message, err := reader.ReadNext(ctx)
if err != nil {
return err
}
fmt.Println(string(message.Payload))
Complete application, including imports, configuration loading and cleanup
docs/quickstart/client.go
// Licensed to the LF AI & Data foundation under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"bytes"
"context"
"flag"
"fmt"
"log"
"time"
clientv3 "go.etcd.io/etcd/client/v3"
"github.com/zilliztech/woodpecker/common/config"
"github.com/zilliztech/woodpecker/woodpecker"
wplog "github.com/zilliztech/woodpecker/woodpecker/log"
)
func main() {
file := flag.String("config", "client.yaml", "SDK configuration file (not wp CLI cli.yaml)")
flag.Parse()
if err := run(*file); err != nil {
log.Fatal(err)
}
}
func run(file string) error {
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
cfg, err := config.NewConfiguration(file)
if err != nil {
return fmt.Errorf("load SDK configuration: %w", err)
}
var client woodpecker.Client
if cfg.Woodpecker.Storage.IsStorageService() {
etcdClient, err := clientv3.New(clientv3.Config{
Endpoints: cfg.Etcd.GetEndpoints(),
DialTimeout: 5 * time.Second,
})
if err != nil {
return fmt.Errorf("create etcd client: %w", err)
}
defer etcdClient.Close()
client, err = woodpecker.NewClient(ctx, cfg, etcdClient, false)
if err != nil {
return fmt.Errorf("create service client: %w", err)
}
} else if cfg.Woodpecker.Storage.IsStorageMinio() {
client, err = woodpecker.NewEmbedClientFromConfig(ctx, cfg)
if err != nil {
return fmt.Errorf("create embedded client: %w", err)
}
} else {
return fmt.Errorf("this example supports storage.type service or minio")
}
defer client.Close(context.Background())
name := fmt.Sprintf("quickstart-%d", time.Now().UnixNano())
if err := client.CreateLog(ctx, name); err != nil {
return fmt.Errorf("create log: %w", err)
}
handle, err := client.OpenLog(ctx, name)
if err != nil {
return fmt.Errorf("open log: %w", err)
}
defer handle.Close(context.Background())
writer, err := handle.OpenLogWriter(ctx)
if err != nil {
return fmt.Errorf("open writer: %w", err)
}
defer writer.Close(context.Background())
payload := []byte("hello, Woodpecker")
result := writer.Write(ctx, &wplog.WriteMessage{Payload: payload})
if result.Err != nil {
return fmt.Errorf("write: %w", result.Err)
}
fmt.Printf("WROTE log=%s position=%d:%d\n", name, result.LogMessageId.SegmentId, result.LogMessageId.EntryId)
reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
if err != nil {
return fmt.Errorf("open reader: %w", err)
}
defer reader.Close(context.Background())
message, err := reader.ReadNext(ctx)
if err != nil {
return fmt.Errorf("read: %w", err)
}
if !bytes.Equal(message.Payload, payload) || *message.Id != *result.LogMessageId {
return fmt.Errorf("read result differs from acknowledged write")
}
fmt.Printf("READ position=%d:%d payload=%q\n", message.Id.SegmentId, message.Id.EntryId, message.Payload)
fmt.Printf("VERIFIED mode=%s messages=1\n", cfg.Woodpecker.Storage.Type)
return nil
}
Build the container on the same CPU architecture as the cluster nodes:
For a remote cluster, replace the example registry below with one your cluster can pull from, push the image, and set that exact image in kubernetes-job.yaml:
bash
docker tag woodpecker-quickstart-client:local registry.example.com/team/wp-quickstart-client:demo
docker push registry.example.com/team/wp-quickstart-client:demo
Expected application output (log name varies; runtime messages may also appear):
Example output
WROTE log=quickstart-... position=0:0
READ position=0:0 payload="hello, Woodpecker"
VERIFIED mode=service messages=1
WROTE reports the acknowledged position. READ reports the entry read back. The program prints VERIFIED only after both the message ID and payload match. A nonzero exit or missing VERIFIED line requires investigation.
If the Job fails, inspect its logs and kubectl -n wp-quickstart describe job quickstart-client. Check Service DNS and node seeds for connection errors; compare metadata/bucket/root-path settings if the intended log or data is absent. To repeat the application, delete only its Job and reapply its manifest:
This deletes the dedicated namespace, its example workloads, PVCs and their data:
bash
kubectl delete namespace wp-quickstart
The manifest is a plain StatefulSet evaluation deployment. For managed Woodpecker lifecycle operations, see the Operator guide.
Docker Compose: deploy a WAL service and run an application
1. Prepare Docker and the checkout
Install Git and Docker with Compose v2.20+. Docker builds the Go application, so no host Go installation is needed for this path. Run all commands from this checkout’s repository root:
bash
git clone --depth 1 https://github.com/zilliztech/woodpecker.git
cd woodpecker
2. Start dependencies and three LogStore nodes
The tutorial-specific Compose file creates its own network and volumes, etcd, MinIO and three LogStore nodes. It advertises container DNS names so the application and nodes can reach one another; it is separate from the general deployments/ template.
bash
docker compose -f docs/quickstart/compose.yaml --profile service up -d --wait --wait-timeout 180 etcd minio wp-0 wp-1 wp-2
docker compose -f docs/quickstart/compose.yaml --profile service ps
All five services must be healthy. Etcd and MinIO also bind workstation-only ports 12379 and 19000, used by the Embedded tab. Change those bindings if occupied; the Service app uses internal names.
3. Use the Compose client configuration
compose-client.yaml is mounted into the servers and the application. It sets service mode, seed addresses, metadata namespace and object-storage location:
Names such as wp-0:18080, etcd:2379 and minio:9000 are Docker-network addresses. Run the application container below on that network. Setting only a localhost seed on the workstation does not rewrite the addresses returned in quorum metadata.
4. Create the service client, write and read
The runnable client.go loads the selected YAML, creates an external etcd connection and a service client:
Client initialization · excerpt
// cfg is loaded from the path-specific client YAML below.
etcdClient, err := clientv3.New(clientv3.Config{
Endpoints: cfg.Etcd.GetEndpoints(),
DialTimeout: 5 * time.Second,
})
if err != nil {
return err
}
defer etcdClient.Close()
client, err := woodpecker.NewClient(ctx, cfg, etcdClient, false)
if err != nil {
return err
}
defer client.Close(context.Background())
It creates a fresh log, appends a message, and reads from the acknowledged position:
Write and read · excerpt
// After CreateLog / OpenLog / OpenLogWriter (see the complete program):
result := writer.Write(ctx, &wplog.WriteMessage{Payload: []byte("hello, Woodpecker")})
if result.Err != nil {
return result.Err
}
reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
if err != nil {
return err
}
defer reader.Close(context.Background())
message, err := reader.ReadNext(ctx)
if err != nil {
return err
}
fmt.Println(string(message.Payload))
Complete application, including imports, configuration loading and cleanup
docs/quickstart/client.go
// Licensed to the LF AI & Data foundation under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"bytes"
"context"
"flag"
"fmt"
"log"
"time"
clientv3 "go.etcd.io/etcd/client/v3"
"github.com/zilliztech/woodpecker/common/config"
"github.com/zilliztech/woodpecker/woodpecker"
wplog "github.com/zilliztech/woodpecker/woodpecker/log"
)
func main() {
file := flag.String("config", "client.yaml", "SDK configuration file (not wp CLI cli.yaml)")
flag.Parse()
if err := run(*file); err != nil {
log.Fatal(err)
}
}
func run(file string) error {
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
cfg, err := config.NewConfiguration(file)
if err != nil {
return fmt.Errorf("load SDK configuration: %w", err)
}
var client woodpecker.Client
if cfg.Woodpecker.Storage.IsStorageService() {
etcdClient, err := clientv3.New(clientv3.Config{
Endpoints: cfg.Etcd.GetEndpoints(),
DialTimeout: 5 * time.Second,
})
if err != nil {
return fmt.Errorf("create etcd client: %w", err)
}
defer etcdClient.Close()
client, err = woodpecker.NewClient(ctx, cfg, etcdClient, false)
if err != nil {
return fmt.Errorf("create service client: %w", err)
}
} else if cfg.Woodpecker.Storage.IsStorageMinio() {
client, err = woodpecker.NewEmbedClientFromConfig(ctx, cfg)
if err != nil {
return fmt.Errorf("create embedded client: %w", err)
}
} else {
return fmt.Errorf("this example supports storage.type service or minio")
}
defer client.Close(context.Background())
name := fmt.Sprintf("quickstart-%d", time.Now().UnixNano())
if err := client.CreateLog(ctx, name); err != nil {
return fmt.Errorf("create log: %w", err)
}
handle, err := client.OpenLog(ctx, name)
if err != nil {
return fmt.Errorf("open log: %w", err)
}
defer handle.Close(context.Background())
writer, err := handle.OpenLogWriter(ctx)
if err != nil {
return fmt.Errorf("open writer: %w", err)
}
defer writer.Close(context.Background())
payload := []byte("hello, Woodpecker")
result := writer.Write(ctx, &wplog.WriteMessage{Payload: payload})
if result.Err != nil {
return fmt.Errorf("write: %w", result.Err)
}
fmt.Printf("WROTE log=%s position=%d:%d\n", name, result.LogMessageId.SegmentId, result.LogMessageId.EntryId)
reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
if err != nil {
return fmt.Errorf("open reader: %w", err)
}
defer reader.Close(context.Background())
message, err := reader.ReadNext(ctx)
if err != nil {
return fmt.Errorf("read: %w", err)
}
if !bytes.Equal(message.Payload, payload) || *message.Id != *result.LogMessageId {
return fmt.Errorf("read result differs from acknowledged write")
}
fmt.Printf("READ position=%d:%d payload=%q\n", message.Id.SegmentId, message.Id.EntryId, message.Payload)
fmt.Printf("VERIFIED mode=%s messages=1\n", cfg.Woodpecker.Storage.Type)
return nil
}
Expected application output (log name varies; runtime messages may also appear):
Example output
WROTE log=quickstart-... position=0:0
READ position=0:0 payload="hello, Woodpecker"
VERIFIED mode=service messages=1
WROTE reports the acknowledged position. READ reports the entry read back. The program prints VERIFIED only after both the message ID and payload match. A nonzero exit or missing VERIFIED line requires investigation.
For startup or connection failures, inspect the dependency and LogStore logs, verify that all nodes joined, and rerun the client once they are healthy:
No standalone LogStore nodes are required. The two dependencies must be healthy and reachable at 127.0.0.1:12379 and 127.0.0.1:19000. This reuses the tutorial Compose dependencies if the Service example is already running.
3. Use the workstation client configuration
embedded-host.yaml selects minio storage, host-reachable dependency ports, and a separate metadata/bucket namespace. This is the SDK YAML, not wp’s cli.yaml:
The runnable client.go loads the selected YAML, starts an embedded client connected to etcd and object storage:
Client initialization · excerpt
// storage.type is minio; the LogStore runs inside this application.
client, err := woodpecker.NewEmbedClientFromConfig(ctx, cfg)
if err != nil {
return err
}
defer client.Close(context.Background())
It creates a fresh log, appends a message, and reads from the acknowledged position:
Write and read · excerpt
// After CreateLog / OpenLog / OpenLogWriter (see the complete program):
result := writer.Write(ctx, &wplog.WriteMessage{Payload: []byte("hello, Woodpecker")})
if result.Err != nil {
return result.Err
}
reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
if err != nil {
return err
}
defer reader.Close(context.Background())
message, err := reader.ReadNext(ctx)
if err != nil {
return err
}
fmt.Println(string(message.Payload))
Complete application, including imports, configuration loading and cleanup
docs/quickstart/client.go
// Licensed to the LF AI & Data foundation under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"bytes"
"context"
"flag"
"fmt"
"log"
"time"
clientv3 "go.etcd.io/etcd/client/v3"
"github.com/zilliztech/woodpecker/common/config"
"github.com/zilliztech/woodpecker/woodpecker"
wplog "github.com/zilliztech/woodpecker/woodpecker/log"
)
func main() {
file := flag.String("config", "client.yaml", "SDK configuration file (not wp CLI cli.yaml)")
flag.Parse()
if err := run(*file); err != nil {
log.Fatal(err)
}
}
func run(file string) error {
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
cfg, err := config.NewConfiguration(file)
if err != nil {
return fmt.Errorf("load SDK configuration: %w", err)
}
var client woodpecker.Client
if cfg.Woodpecker.Storage.IsStorageService() {
etcdClient, err := clientv3.New(clientv3.Config{
Endpoints: cfg.Etcd.GetEndpoints(),
DialTimeout: 5 * time.Second,
})
if err != nil {
return fmt.Errorf("create etcd client: %w", err)
}
defer etcdClient.Close()
client, err = woodpecker.NewClient(ctx, cfg, etcdClient, false)
if err != nil {
return fmt.Errorf("create service client: %w", err)
}
} else if cfg.Woodpecker.Storage.IsStorageMinio() {
client, err = woodpecker.NewEmbedClientFromConfig(ctx, cfg)
if err != nil {
return fmt.Errorf("create embedded client: %w", err)
}
} else {
return fmt.Errorf("this example supports storage.type service or minio")
}
defer client.Close(context.Background())
name := fmt.Sprintf("quickstart-%d", time.Now().UnixNano())
if err := client.CreateLog(ctx, name); err != nil {
return fmt.Errorf("create log: %w", err)
}
handle, err := client.OpenLog(ctx, name)
if err != nil {
return fmt.Errorf("open log: %w", err)
}
defer handle.Close(context.Background())
writer, err := handle.OpenLogWriter(ctx)
if err != nil {
return fmt.Errorf("open writer: %w", err)
}
defer writer.Close(context.Background())
payload := []byte("hello, Woodpecker")
result := writer.Write(ctx, &wplog.WriteMessage{Payload: payload})
if result.Err != nil {
return fmt.Errorf("write: %w", result.Err)
}
fmt.Printf("WROTE log=%s position=%d:%d\n", name, result.LogMessageId.SegmentId, result.LogMessageId.EntryId)
reader, err := handle.OpenLogReader(ctx, result.LogMessageId, "quickstart")
if err != nil {
return fmt.Errorf("open reader: %w", err)
}
defer reader.Close(context.Background())
message, err := reader.ReadNext(ctx)
if err != nil {
return fmt.Errorf("read: %w", err)
}
if !bytes.Equal(message.Payload, payload) || *message.Id != *result.LogMessageId {
return fmt.Errorf("read result differs from acknowledged write")
}
fmt.Printf("READ position=%d:%d payload=%q\n", message.Id.SegmentId, message.Id.EntryId, message.Payload)
fmt.Printf("VERIFIED mode=%s messages=1\n", cfg.Woodpecker.Storage.Type)
return nil
}
bash
go run ./docs/quickstart/client.go --config docs/quickstart/embedded-host.yaml
5. Verify the result
Expected application output (log name varies; runtime messages may also appear):
Example output
WROTE log=quickstart-... position=0:0
READ position=0:0 payload="hello, Woodpecker"
VERIFIED mode=minio messages=1
WROTE reports the acknowledged position. READ reports the entry read back. The program prints VERIFIED only after both the message ID and payload match. A nonzero exit or missing VERIFIED line requires investigation.
Connection refused usually means a dependency is not healthy or the workstation port differs from the YAML. An authentication error requires matching MinIO credentials. Use the NewEmbedClientFromConfig path with storage.type: minio; service storage requires the other client constructor.
6. Stop dependencies
bash
docker compose -f docs/quickstart/compose.yaml down
Stop any Service example using these dependencies first. Named volumes retain metadata and object data; add --volumes to discard the tutorial data. For an application in its own module, install github.com/zilliztech/woodpecker@v0.1.47 and copy the complete Go source above.
Comprehensive guide to configuring Woodpecker with all available options and their defaults.
Full Configuration Example
Woodpecker uses YAML configuration. Create a woodpecker.yaml file:
woodpecker.yaml
woodpecker:
meta:
type: etcd
prefix: woodpeckerclient:
segmentAppend:
queueSize: 100# admitted entries per segment through all replica outcomes and callbacks
maxRetries: 2# total attempts per entry and replica, including the first send
maxBatchEntries: 1000# queued entries per batch; no wait for more entries
maxBatchBytes: 2000000# payload threshold; the last entry may exceed it
maxPendingBytes: 0# retained payload budget in both modes; 0 uses 64 MiB
sendTimeout: 2000# ms per remote send attempt through its first responsesegmentRollingPolicy:
maxSize: 100000000# 100MB per segment
maxInterval: 800# seconds between rolls
maxBlocks: 1000# max blocks per segmentauditor:
maxInterval: 5# seconds between auditssessionMonitor:
checkInterval: 3# seconds
maxFailures: 5# consecutive fails before invalidgrpc:
clientMaxSendSize: 268435456# 256MB
clientMaxRecvSize: 536870912# 512MB# quorum: (service mode only, see Quorum section)logstore:
segmentSyncPolicy:
maxInterval: 1000# ms between syncs
maxIntervalForLocalStorage: 5# ms for local backend
maxEntries: 2000# max entries per buffer
maxBytes: 100000000# 100MB buffer limit
maxFlushRetries: 3
retryInterval: 2000# ms between retries
maxFlushSize: 16000000# 16MB per block flush
maxFlushThreads: 8segmentCompactionPolicy:
maxBytes: 32000000# 32MB merged block size
maxParallelUploads: 4
maxParallelReads: 8segmentReadPolicy:
maxBatchSize: 16000000# 16MB read batch
maxFetchThreads: 32retentionPolicy:
ttl: 259200# 72 hours for truncated segmentsfencePolicy:
conditionWrite: "auto"# auto | enable | disablegrpc:
serverMaxSendSize: 536870912# 512MB
serverMaxRecvSize: 268435456# 256MBprocessorCleanupPolicy:
cleanupInterval: 60# seconds
maxIdleTime: 300# seconds
shutdownTimeout: 15# secondsstorage:
type: default# default(=minio), minio, local, service
rootPath: /var/lib/woodpecker
Client Configuration
The defaults below are built into the client. The repository's config/woodpecker.yaml explicitly overrides queueSize to 10000 and maxRetries to 3. Append admission limits apply per writable segment handle, not to the whole client process.
sendTimeout is not an overall append, OpenWriter or Fence deadline. After sending, each entry's final ACK read has a separate internal 30-second budget in both modes, including reads within a batch. Each retry gets fresh attempt budgets.
Parameter
Description
Default
segmentAppend.queueSize
Positive entry-count admission limit in both modes, covering queued, sending, ACK-waiting and retrying entries until all replica outcomes and the ordered callback finish, even after quorum success.
100
segmentAppend.maxRetries
Total attempts per entry and replica, including the initial send. 0 or 1 disables retries. Both modes retry eligible send/ACK failures on the affected replica; non-retryable failure or exhaustion requests rolling, even after quorum success.
2
segmentAppend.maxBatchEntries
Maximum consecutive queued entries per dispatched batch in both modes. Service mode sends one AddEntries RPC per replica; embedded mode uses local calls. Does not wait for more entries. Values ≤ 1 disable batching; retries are individual.
1000
segmentAppend.maxBatchBytes
Payload threshold for stopping batch collection. The last entry may exceed it; an oversized first entry is sent alone. Values ≤ 0 disable the threshold; ignored when maxBatchEntries ≤ 1. This is not a strict RPC-size limit. The YAML string 2MB means 2097152 bytes, unlike the numeric default.
2000000 bytes
segmentAppend.maxPendingBytes
Admitted payload budget in both modes, counted once per entry, with the same credit lifetime as queueSize. Retries reuse credits. 0 uses 64 MiB; one oversized entry may enter when empty. Excludes transport buffers and bookkeeping.
0 → 64 MiB
segmentAppend.sendTimeout
Milliseconds per remote AddEntry/AddEntries attempt, from stream opening through the first response (normally Buffered), including connection readiness waits inside the RPC. Excludes admission, client-pool lookup and final ACK reads. Embedded local calls do not use it. 0 uses 2000ms.
2000
grpc.dialTimeout
Time allowed for one connection attempt to a logstore, in ms (gRPC MinConnectTimeout); also bounds a call to an address that never answers
1000
grpc.connectBackoff.baseDelay
First delay before reconnecting after a failed attempt, in ms
100
grpc.connectBackoff.multiplier
Growth factor of the reconnect delay
1.6
grpc.connectBackoff.jitter
Randomisation of the reconnect delay
0.2
grpc.connectBackoff.maxDelay
Longest reconnect delay, in ms
3000
quorum.selectNodesTimeout
Time allowed for one SelectNodes call to a seed, in ms; the next seed is asked when it expires
2000
segmentRead.activeTimeout
Time allowed for one quorum read of an active segment from one replica, in ms; the reader moves to the next replica when it expires
3000
segmentRead.settledTimeout
Time allowed for one quorum read of any other segment, which may be served from object storage, in ms
20000
segmentRollingPolicy.maxSize
Maximum segment size in bytes
100MB
segmentRollingPolicy.maxInterval
Maximum interval between segment rolls (seconds)
800
segmentRollingPolicy.maxBlocks
Maximum blocks per segment
1000
auditor.maxInterval
Audit interval in seconds
5
sessionMonitor.checkInterval
Session health check interval (seconds)
3
sessionMonitor.maxFailures
Consecutive failures before session invalid
5
LogStore Configuration
Parameter
Description
Default
segmentSyncPolicy.maxInterval
Max sync interval (milliseconds)
1000
segmentSyncPolicy.maxIntervalForLocalStorage
Sync interval for local storage (ms)
5
segmentSyncPolicy.maxEntries
Max entries in write buffer
2000
segmentSyncPolicy.maxBytes
Max write buffer size
100MB
segmentSyncPolicy.maxFlushRetries
Max sync retries
3
segmentSyncPolicy.retryInterval
Retry interval (ms)
2000
segmentSyncPolicy.maxFlushSize
Max block flush size
16MB
segmentSyncPolicy.maxFlushThreads
Concurrent flush threads
8
segmentCompactionPolicy.maxBytes
Max merged block size after compaction
32MB
segmentCompactionPolicy.maxParallelUploads
Parallel compaction upload threads
4
segmentCompactionPolicy.maxParallelReads
Parallel compaction read threads
8
segmentReadPolicy.maxBatchSize
Max read batch size
16MB
segmentReadPolicy.maxFetchThreads
Concurrent fetch threads
32
retentionPolicy.ttl
TTL for truncated segments (seconds)
259200 (72h)
processorCleanupPolicy.cleanupInterval
Idle processor scan interval (seconds)
60
processorCleanupPolicy.maxIdleTime
Idle time before cleanup (seconds)
300
Storage Configuration
Parameter
Description
Default
storage.type
Storage backend: default (=minio), minio, local, service
Understanding Woodpecker's two deployment modes and core design.
Embedded Mode
In embedded mode, Woodpecker operates as a lightweight library integrated directly into your application. It requires only etcd for metadata storage and coordination, keeping dependencies minimal.
Best suited for applications needing simple WAL integration
Service Mode
In service mode, WAL read/write operations and caching logic are decoupled into a dedicated LogStore cluster service. This acts as a high-performance caching layer between clients and object storage.
Key characteristics of service mode:
Dedicated LogStore cluster with gRPC communication
Data prefetching and read/write caching for lower latency
Gossip-based cluster membership and auto-discovery
Quorum-based replication with AZ-aware node placement
Best suited for high-throughput, latency-sensitive workloads
Core Components
Component
Role
Client
Read/write protocol layer connecting applications to Woodpecker
EmbeddedClient
Combined client + LogStore for embedded deployments
LogStore
High-speed log writes, batching, caching, and cloud storage uploads
ObjectStorage
Durable storage backend using S3-compatible APIs
MetadataProvider
etcd-backed metadata storage, coordination, and distributed locking
QuorumDiscovery
Service mode quorum-based node selection and replication management via gossip
Data Flow
Write Path
Application calls Write() or WriteAsync() on a LogWriter
Writer validates the session lock is still valid (distributed lock via etcd)
Gets or creates a writable segment via rolling policy (maxSize=100MB, maxInterval=800s, maxBlocks=1000)
Serializes message with MarshalMessage() (payload + properties)
Calls segment.AppendAsync() — data is buffered
Buffer is flushed to storage when sync policy triggers (maxInterval=1000ms, maxEntries=2000, maxBytes=100MB)
Segment metadata updated in etcd with revision-based optimistic locking
Read Path
Application opens a LogReader with starting position and reader name (OpenLogReader(ctx, from, readerName))
Reader loads entries in batches via ReadBatchAdv() (batch limit: 200 entries)
Yields one entry per ReadNext() call from local cache
On segment EOF, auto-advances to the next segment seamlessly
Reader position updated in etcd every 30 seconds
For active (still-writing) segments, polls with 200ms wait interval when no new data is available
Segment Lifecycle
Segments transition through the following states:
State Machine
Active → Completed → Sealed → Truncated → (Deleted)
Active - Currently accepting writes
Completed - All data flushed, no more writes
Sealed - Metadata finalized, immutable
Truncated - Marked for cleanup
Deleted - Storage data removed
Woodpecker's innovative quorum-based replication for service mode deployments.
Overview
In service mode, Woodpecker uses a configurable quorum protocol for data replication across LogStore nodes. This ensures data durability while allowing flexible tradeoffs between consistency, availability, and performance.
Quorum Parameters
The quorum is configured via the replicas setting, which determines the ensemble size (E). Write quorum (WQ) and ack quorum (AQ) are derived automatically:
Parameter
Symbol
Formula
E=3
E=5
Ensemble Size
E
= replicas
3
5
Write Quorum
WQ
= E
3
5
Ack Quorum
AQ
= (E / 2) + 1
2
3
Data is written to all E nodes (WQ = E), but only AQ acknowledgments are required before the write is considered durable. This allows writes to succeed even if some nodes are slow, while ensuring majority-based durability.
Buffer Pools
Buffer pools are logical groupings of LogStore nodes. Each pool is defined by a name and seed addresses used for gossip-based node discovery. Pools enable regional grouping of nodes for locality-aware placement.
woodpecker.yaml
woodpecker:
client:
quorum:
quorumBufferPools:
- name: "region-1"
seeds: ["logstore1:7946", "logstore2:7946"]
- name: "region-2"
seeds: ["logstore3:7946", "logstore4:7946"]
quorumSelectStrategy:
affinityMode: "soft"# soft | hard
replicas: 3# ensemble size (3 or 5)
strategy: "random"# see strategies table below
customPlacement: # only for strategy: custom
- name: "replica-1"
region: "region-1"
az: "az-1"
resourceGroup: "rg-1"
Node Selection Strategies
Woodpecker supports 8 node selection strategies for quorum placement:
Strategy
Config Value
Description
Random
random
Random node selection from all available nodes (default)
Single AZ, Single RG
single-az-single-rg
All nodes in the same availability zone and resource group
Single AZ, Multi RG
single-az-multi-rg
Nodes in the same AZ but spread across different resource groups
Multi AZ, Single RG
multi-az-single-rg
Nodes across multiple AZs within the same resource group
Multi AZ, Multi RG
multi-az-multi-rg
Nodes spread across multiple AZs and resource groups (highest fault tolerance)
Cross Region
cross-region
Nodes across different regions (requires 2+ buffer pools)
Random Group
random-group
Pre-partitioned random groups for balanced distribution
Custom
custom
Explicit placement rules via customPlacement configuration
Affinity Modes
Mode
Behavior
soft
Prefer nodes matching the strategy constraints, but fall back to any available node if not enough matching nodes are found
hard
Strictly enforce strategy constraints — fail if not enough matching nodes are available
Cluster Membership
Service mode uses HashiCorp memberlist (gossip protocol) for cluster membership management:
Nodes discover each other via seed addresses configured in buffer pools
Health status is propagated through gossip protocol
Failed nodes are automatically detected and excluded from quorum selection
New nodes can join the cluster dynamically by connecting to any seed
Each node exposes a bind port (gossip) and a service port (gRPC)
Cloud-native object storage as the durable storage layer for Woodpecker's WAL data.
Overview
Object storage serves as Woodpecker's primary durable storage layer. All WAL data is ultimately persisted to S3-compatible object storage, providing virtually unlimited capacity, high durability (11 nines), and cross-region replication capabilities.
ObjectStorage Interface
Woodpecker abstracts storage operations through a unified ObjectStorage interface:
Condition write (PutObjectIfNoneMatch) is used as a fence mechanism to prevent split-brain in embedded mode. It ensures only one writer can create a given object key.
AWS/MinIO/GCS/Azure — Uses If-None-Match: * header for conditional PUT
Aliyun OSS / Tencent COS — Uses the forbid-overwrite feature (wrapped implementation)
Auto-detection — Woodpecker probes condition write support at startup (configurable via fencePolicy.conditionWrite)
Performance Best Practices
Write Optimization
The default maxFlushSize is 16MB — increase for larger batch uploads to reduce API call overhead
Default maxFlushThreads is 8 — increase for higher parallelism on fast networks
Use WriteAsync with batching for maximum throughput
Compaction merges small blocks into 32MB merged blocks (configurable via segmentCompactionPolicy.maxBytes)
Read Optimization
Read batch size is 16MB by default (segmentReadPolicy.maxBatchSize)
Up to 32 concurrent fetch threads for parallel reads (segmentReadPolicy.maxFetchThreads)
Range reads via GetObject with offset/size parameters for partial fetches
Data Durability
CRC32 checksums for data integrity verification on every block
Automatic retry on transient failures with configurable backoff (up to maxFlushRetries = 3, interval = 2s)
Fragment-based storage for incremental durability — data is durable as soon as each block is flushed
Segment compaction merges small fragments into larger objects for efficient storage and reads
High-performance disk-based log storage using memory-mapped I/O.
Overview
The local file system storage implements disk-based log storage using mmap (memory mapping) for efficient read and write performance. It maintains the Fragment concept for consistency with the object storage implementation.
File Layout
FragmentFile Layout
[Header (4K)] + [Data Area →] + [...Free Space...] + [← Index Area] + [Footer]
Header (4K) Magic string and version information
Data Area Actual log entries, growing forward from 4K
Free Space Unused space between data and index
Index Area Entry offsets, growing backward from file end
Footer Metadata information
Data Entry Format:
[Payload Size (4B)] + [CRC32 (4B)] + [Actual Data (variable)]
Two-tier hybrid storage combining local disk speed with object storage durability for service mode deployments.
Overview
Staged storage is the storage backend used in service mode (storage.type: service). It implements a two-tier hybrid architecture: active segments write to local disk for low-latency writes, then compaction uploads merged blocks to object storage for durable, long-term persistence.
Staged storage is automatically used when storage.type is set to service. No additional configuration is needed beyond configuring the MinIO/S3 backend for the object storage tier.
How It Works
Data Flow
Write Path:
Client → LogStore Server → Local Disk (fast, low-latency)
Compaction:
Local Disk → Merge Blocks → Object Storage (durable)
Read Path:
Active segments → Read from Local Disk
Compacted data → Read from Object Storage
Core Components
Component
Role
StagedSegmentImpl
Manages a segment's lifecycle across both tiers — coordinates writes to local disk and compaction uploads to object storage
StagedFileWriter
Handles buffered writes to local disk files with block-based batching and CRC32 integrity checks
StagedFileReaderAdv
Reads entries from either local disk (active segments) or object storage (compacted segments) transparently
Configuration
Enable staged storage by setting storage.type: service:
Split-brain prevention using condition write as a fence mechanism.
Overview
In embedded mode, Woodpecker uses condition write as a fence mechanism to prevent split-brain scenarios where a stale primary node resumes writing after being partitioned or crashed. This ensures data consistency and prevents concurrent write conflicts.
Supported Storage Backends
Backend
Support
Mechanism
MinIO
Full
If-None-Match header
AWS S3
Full
Conditional PUT operations
Azure Blob
Full
Conditional writes
GCP Cloud Storage
Full
Conditional operations
Aliyun OSS
Wrapped
Forbid overwrite feature
Tencent COS
Wrapped
Forbid overwrite feature
Detection Modes
Auto Mode (Default)
Follows existing true/false cluster metadata when present. If no metadata exists, verifies condition write support with strict enable semantics, stores true on success, and fails startup on failure.
Verification failure is surfaced immediately and is never persisted as an unsupported capability.
Enable Mode
Strict requirement: stored true is trusted, stored false is ignored, verification must succeed within 10 retries, and successful verification overwrites metadata to true.
Disable Mode
Skips detection entirely and uses distributed locks. Use when you know your backend doesn't support condition write.
Distributed Lock Fallback
When condition write is explicitly disabled, or when legacy auto metadata stores false, Woodpecker uses etcd-based distributed locks:
Property
Detail
Lock Type
etcd concurrency.Mutex with concurrency.Session
Session TTL
10 seconds (auto-renewed via keep-alive)
Lock Key
/woodpecker/service/lock/{logName}
Acquisition
Non-blocking TryLock
Failover Behavior
Session Expiration — etcd session expires after TTL (10s) if keep-alive fails
Lock Release — Distributed lock is automatically released
New Primary — New node acquires the lock
Writer Switch — Applications reopen writer (up to 10s wait)
Built-in Prometheus metrics and OpenTelemetry tracing for production observability.
Prometheus Metrics
Woodpecker exposes comprehensive Prometheus metrics for monitoring system health and performance. Each node exposes metrics on a dedicated port (default 9091–9094 for nodes 1–4).
Woodpecker integrates with OpenTelemetry for distributed tracing. The deployment includes Jaeger (port 16686) for trace visualization. gRPC interceptors automatically propagate trace context in service mode.
Setting Up Monitoring Stack
Use the provided Docker Compose overlay to add Prometheus and Grafana to the cluster:
bash
# Start cluster with Prometheus + Grafana monitoring
cd deployments
docker compose -f docker-compose.yaml \
-f ../tests/docker/monitor/docker-compose.monitor.yaml \
-p woodpecker-monitor up -d
Access the monitoring services:
Service
URL
Credentials
Prometheus
http://localhost:9090
—
Grafana
http://localhost:3000
Anonymous admin (no login)
Jaeger
http://localhost:16686
—
MinIO Console
http://localhost:9001
minioadmin / minioadmin
Grafana Dashboards
Pre-built Grafana dashboard templates are available in tests/docker/monitor/grafana/templates/. These provide ready-to-use visualizations for write/read throughput, latency percentiles (P50/P95/P99), segment state distribution, storage backend performance, and error rates.
Run Monitoring Tests
Automated E2E metric verification tests are provided:
bash
# One-click: build, deploy, test, and cleanup
cd tests/docker/monitor
./run_monitor_tests.sh
# Keep cluster running after tests for manual inspection
./run_monitor_tests.sh --no-cleanup
The monitoring test suite also includes a rolling restart latency profile test that measures write/read latency across node restarts — useful as a baseline for evaluating rolling upgrade quality.