S3sync :: Docs

Run S3 migrations from one dashboard

Create storage connections, select an execution agent, and run a controlled copy between S3-compatible endpoints.

Core objects

How a migration is structured

Storage

A saved S3-compatible connection. A migration uses one storage as its source and another as its destination. You can check connectivity and load the source bucket list before starting.

Agent

The worker that executes a migration. Install your own agent on a reachable host over SSH or with a generated shell command, or leave agents unselected to use the shared system pool.

Migration

A runnable configuration that connects a source, destination, and optional agents. Choose included or excluded buckets, sync mode, concurrency, and copy options, then start or stop it from the migration list.

How to

Run your first migration

Create two storages. Add your source and destination endpoints in Storages, then run a connection check.
Choose an agent. Create and install your own agent in Agents, or use the shared system pool.
Create a migration. Select the source, destination, and agent; optionally limit the buckets to copy.
Start and monitor. Start the migration from its row. The list shows status, copied object count, recent errors, and timestamps.

Options

Configure each object

Storage

Name, endpoint, region, access key, secret: The connection details used to reach an S3-compatible storage.
Availability monitoring: Runs periodic connection checks and shows their result on the dashboard.
Use path style: Uses path-style S3 addressing instead of virtual-hosted-style addressing.
Insecure skip verify: Disables TLS certificate verification. Use only for endpoints you trust.
List by start after: Uses the S3 start-after pagination method instead of continuation tokens.Our testing indicates that it performs better for heavily used storages with large object counts and frequently rebuilt indexes, especially Ceph-based systems.

Agent

Name: An optional label that identifies your agent in the migration picker.
Install over SSH: Provide the host, user, and password; check the connection, then install or update the agent remotely.
Install with shell command: Generate a command, run it on the target host, and verify that the agent is running.

Migration

Name, source, destination, agents: Identify the job, select its two storages, and assign your agents or use the system pool.
Only buckets / Except buckets: Include only selected source buckets or omit named buckets. An empty include list uses all discovered source buckets.
Add suffix to buckets on destination: Adds a suffix to every destination bucket name created by the job. This helps satisfy bucket-name uniqueness requirements within a storage service or region.
Sync mode: off Completes after the copy.backup Preserves destination objects.mirror Also removes destination objects absent from the source during later sync runs.
Threads: Sets the number of parallel worker threads. More threads can improve throughput, but can also increase load and trigger storage rate limits.High thread counts can cause 4xx rate-limit responses and 5xx errors when they place excessive load on a custom S3 storage.We are working on intelligent automatic tuning for this setting.
Priority: Sets the job’s position in the queue. Higher priority runs earlier.
Multipart threshold / chunksize: Set the object size at which multipart upload is used and the size of each part, in MB.
Copy meta: Copies object metadata to the destination.
Copy bucket policy: Copies bucket policies to the destination.
Copy tags: Copies object tags to the destination.This increases request volume.
Flat listing: An experimental listing mode that can be faster in some environments.
Index by destination: Build the initial object index from the destination when destination objects must be preserved and synchronized with the source.

Ready to migrate, move, sync, or safeguard your data?

Open the migration dashboard to create and monitor a job.

Go to migration