Object Storage Guide

Cloud-native object storage implementation using MinIO/S3

Overview

Woodpecker's object storage implementation provides a cloud-native solution for durable log storage

Woodpecker supports MinIO and S3-compatible object storage as its durable storage layer. This implementation provides:

Architecture

Understanding the object storage implementation

The object storage implementation consists of several key components:

Configuration

Setting up object storage in Woodpecker

# Configuration of Woodpecker storage
storage:
  # The default value is "default", which configures Woodpecker to use MinIO-compatible storage.
  # Valid values: [default, minio, local, service]
  type: minio
  rootPath: /var/lib/woodpecker # Root path to the log data files.

# Related configuration of MinIO/S3/GCS or any other service supports S3 API
minio:
  address: localhost
  port: 9000
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  useSSL: false
  bucketName: a-bucket
  rootPath: files
  useIAM: false
  cloudProvider: aws
  region: us-east-1
  useVirtualHost: false
  requestTimeoutMs: 10000
Parameter Description Default
storage.type Storage backend type (default, minio, local, service) default
storage.rootPath Root path to the log data files /var/lib/woodpecker
minio.address IP address of MinIO or S3 service localhost
minio.port Port of MinIO or S3 service 9000
minio.accessKeyID Access key ID for authentication -
minio.secretAccessKey Secret key for authentication -
minio.bucketName Name of the bucket for storage a-bucket
minio.useIAM Whether to use IAM role for authentication false
minio.cloudProvider Cloud provider (aws, gcp, aliyun, gcpnative) aws
minio.region Storage system location region -

Performance Considerations

Optimizing object storage performance

Write Performance

Read Performance

Reliability

Ensuring data durability and consistency

Data Durability

Consistency

Usage Examples

Common patterns for using object storage

// Create a new configuration
cfg, err := config.NewConfiguration()
if err != nil {
    panic(err)
}

// Configure storage type and root path
cfg.Woodpecker.Storage.Type = "" // Use default storage type (minio-compatible)
cfg.Woodpecker.Storage.RootPath = "" // Use default storage path

// Create a new client
client, err := woodpecker.NewEmbedClientFromConfig(context.Background(), cfg)
if err != nil {
    panic(err)
}

// Create a new log
logName := "test_log" + time.Now().Format("20060102150405")
client.CreateLog(context.Background(), logName)

// Open the log
logHandle, err := client.OpenLog(context.Background(), logName)
if err != nil {
    panic(err)
}

// Open a writer
logWriter, err := logHandle.OpenLogWriter(context.Background())
if err != nil {
    panic(err)
}

// Write data asynchronously
resultChan := logWriter.WriteAsync(context.Background(),
    &log.WriterMessage{
        Payload: []byte("Hello, Woodpecker!"),
        Properties: map[string]string{
            "key": "value",
        },
    },
)

// Wait for write result
writeResult := <-resultChan
if writeResult.Err != nil {
    panic(writeResult.Err)
}

// Open a reader
earliest := log.EarliestLogMessageID()
logReader, err := logHandle.OpenLogReader(context.Background(), &earliest)
if err != nil {
    panic(err)
}

// Read data
msg, err := logReader.ReadNext(context.Background())
if err != nil {
    panic(err)
}

fmt.Printf("Read message: %s\n", string(msg.Payload))

// Clean up
logReader.Close(context.Background())
logWriter.Close(context.Background())

Best Practices

Guidelines for optimal object storage usage