Operations and upgrades

Pepper 0.2.0 adds namespace-aware readiness, signed backup manifests, verified restore, identity fencing, bounded diagnostics, and explicit replica-replacement and quorum-loss workflows.

Upgrade from 0.1.0

Stop writes and stop every 0.1.0 agent. Protocol v8 in 0.2.0 cannot form a mixed cluster with 0.1.0 protocol v4.
Back up each node separately with the 0.1.0 binary. Keep each metadata backup with that node's identity key and configuration.
Install the 0.2.0 binaries and add [namespace] and [s3] only if those services should be enabled. Namespace and S3 are opt-in in the configuration defaults.
Start all nodes in one coordinated window. Opening metadata automatically performs transactional schema 1 → 4 migrations and preserves existing pins.
Check /healthz, /readyz, node peers, admin status, admin storage, pins, and a sample object read before accepting traffic.
Create a new 0.2.0 backup for each stopped node so the backup manifest captures schema 4 and namespace recovery metadata.

Do not attempt a binary-only rollback after 0.2.0 has migrated metadata. Restore the matching stopped 0.1.0 backup and roll every node back together.

Health, readiness, and metrics

GET /healthz

Liveness only; returns ok when the HTTP process is serving.

GET /readyz

Returns 200 only when every locally hosted namespace group is running, caught up, and has a known leader; otherwise 503 with group details.

GET /metrics

Prometheus text for uptime, storage/compute, namespace commits/reads/conflicts/durability, Raft term/index/lag/quorum, staging/read leases, Merkle work, and RPC volume/errors.

pepper admin status

Human or JSON summary of configured services, queues, auth, namespace capacity, and limits.

Diagnostics HTTP API

The authenticated /v1/admin/diagnostics endpoints are bounded and redact payload content. They cover block inventory, why a CID is protected from GC, publication intents, providers, erasure shards, read-resolution routes, repair outcomes, RPC traffic, and namespace replica state. /v1/admin/dag/{cid} reports bounded DAG reachability and codec counts.

curl -H 'Authorization: Bearer <token>' http://127.0.0.1:9080/v1/admin/diagnostics/blocks
curl -H 'Authorization: Bearer <token>' http://127.0.0.1:9080/v1/admin/diagnostics/gc/<cid>
curl -H 'Authorization: Bearer <token>' http://127.0.0.1:9080/v1/admin/diagnostics/namespaces

Backup and restore

# Agent must be stopped
pepper-agent backup --config /etc/pepper/pepper.toml --output /backup/node1.redb

# Verify that node1.redb and node1.redb.manifest.json are both present
# Restore only to the same initialized node identity
pepper-agent restore --config /etc/pepper/pepper.toml --input /backup/node1.redb --force

The backup is metadata, not a copy of block payloads. Protect the identity key, block storage, backup database, and signed manifest as one recovery set. After restore, verify namespace catch-up and content durability before returning the node to service.

Replica replacement

pepper namespace status <NAMESPACE>
pepper admin namespace replace-replica <NAMESPACE> <FAILED_NODE> [--replacement-node <NODE>]
pepper namespace replicas <NAMESPACE>
pepper namespace status <NAMESPACE>

Normal replacement adds a learner, catches it up, promotes it, and fences the old membership epoch. Prefer this workflow whenever a quorum still exists.

Quorum-loss recovery

pepper admin namespace recover <NAMESPACE> <CHECKPOINT_CID> \
+  --members <NODE1> <NODE2> <NODE3> \
+  --confirm-fork-risk

Recovery can fork history if another copy of the old group can still make progress. Fence old replicas and networks, identify a CID-verified checkpoint, preserve forensic copies, and use recovery only when ordinary replacement is impossible.

Release boundaries

0.2.0 remains a developer preview for controlled private clusters, not untrusted public networks or production multi-tenant workloads.
The HTTP/S3 listener is loopback-only; use a reviewed authenticated TLS proxy or VPN for remote clients.
Pepper assumes crash-fault quorum behavior, not Byzantine peers. It does not provide confidential compute or hardware attestation.
Firecracker/KVM and WAN qualification are optional host/manual diagnostics, not required GitHub-hosted release gates.
Erasure reconstruction remains bounded and buffered; coordinate capacity, repair bandwidth, and namespace log limits for the deployment.