S3 (`s3`)¶

The s3 connector indexes objects under a bucket prefix. It's S3-compatible, so the same connector covers AWS S3, Cloudflare R2, Google Cloud Storage (S3 interop), and MinIO — the only difference is endpoint_url.

How MFS sees it¶

The tree mirrors object keys under the configured bucket and prefix:

s3://acme-docs/
└── engineering/
    └── rfc/
        ├── rfc-001.md      document
        └── rfc-002.pdf     document

Objects are classified by extension exactly like the file connector: documents and code are converted and embedded; structured text is browse/grep only; other types are browse/export only.

Credentials¶

Pick the path for your provider:

AWS S3: prefer an instance/task role when the server runs on AWS; otherwise create a narrow IAM user or STS credential. In the AWS console, open IAM → Users (or the role used by your server) → add an inline policy with the bucket/prefix read permissions. If you create an access key, open Security credentials → Create access key, copy the access key ID and secret once, and inject them into the server environment. boto3 also reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY automatically, so you can omit them from the TOML. Minimum IAM policy:
```
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"]
  }]
}
```
Cloudflare R2: an R2 API token with Object Read; set endpoint_url = "https://<account-id>.r2.cloudflarestorage.com" and region = "auto". Create the token from the R2 dashboard, scope it to the bucket when possible, and copy the Access Key ID / Secret Access Key pair.
GCS (S3 interop): create an HMAC key for a service account with Storage Object Viewer access, then set endpoint_url = "https://storage.googleapis.com".
MinIO: create or choose a read-only access key in the MinIO console and set endpoint_url to your MinIO URL, for example https://minio.example.com.

Configuration¶

bucket = "acme-docs"
prefix = "engineering/rfc/"
region = "us-west-2"
access_key_id = "env:AWS_ACCESS_KEY_ID"
secret_access_key = "env:AWS_SECRET_ACCESS_KEY"
# endpoint_url = "https://<account-id>.r2.cloudflarestorage.com"   # R2/GCS/MinIO

Save the file as s3.toml, then probe the bucket and prefix before indexing:

mfs connector probe s3://acme-docs --config ./s3.toml
mfs add s3://acme-docs --config ./s3.toml

Sync and freshness¶

The connector uses each object's etag as its cursor, so re-syncs only re-process changed objects; deletions are caught by full_scan. Versioned buckets expose only the latest version.

Search and browse¶

mfs search "retention policy" s3://acme-docs/engineering/rfc/
mfs cat s3://acme-docs/engineering/rfc/rfc-001.md --range 1:80
mfs export s3://acme-docs/engineering/rfc/rfc-001.pdf /tmp/rfc-001.pdf

Pitfalls¶

prefix is exact — use a trailing slash when you mean a directory-like prefix.
IAM must allow both ListBucket and GetObject for the scoped bucket/prefix.
Very large PDFs or Office files can be expensive to convert.

S3 (s3)¶