Migrating from MinIO to Predastore

Single-Node Single/Multi-Drive - Standalone

This is the simplest path, and the closest equivalent to a single MinIO server. You'll need a Linux machine running systemd, along with Go (1.27 or newer), git, make, and openssl in order to build Predastore from source. Most distro packages of Go are a few versions behind, so you may need to grab it from go.dev. You'll also want the AWS CLI for checking Predastore once it's up.

Build and install

First up, clone the repository and build the s3d binary:

git clone --branch v1.21.0 https://github.com/mulgadc/predastore.git
cd predastore
make build

Note: This guide was written against v1.21.0. The install tooling used below (make install, predastore-keygen, and the example config) first shipped in v1.20.0, so don't go any older than that.

Install it, along with its systemd service:

sudo make install

Then create the dedicated predastore user, and the config and data directories (/etc/predastore and /var/lib/predastore respectively):

sudo systemd-sysusers && sudo systemd-tmpfiles --create

Share MinIO's drives

There's no need for new drives. Predastore can live on the ones MinIO is already using, in a directory of its own alongside MinIO's data, until you're done with MinIO in step 5. Each drive needs roughly as much free space again as MinIO is using on it, plus some headroom for re-syncs, which df -h will tell you.

The steps below assume MinIO's drives are mounted at /mnt/disk1, /mnt/disk2, and so on. If they're pooled with RAID or ZFS, treat the pool as a single drive.

For each drive, create a directory for Predastore. The leading . matters, as MinIO treats any other top-level directory on its drives as a bucket, and will delete it along with that bucket.

sudo install -d -o predastore -g predastore -m 0700 /mnt/disk1/.predastore

The service isn't permitted to write anywhere outside /var/lib/predastore, so bind-mount each directory into it. The extra options make sure the service waits for the mount, and refuses to start without it:

sudo install -d -m 0755 /var/lib/predastore/disk1
echo '/mnt/disk1/.predastore /var/lib/predastore/disk1 none bind,nofail,x-systemd.requires-mounts-for=/mnt/disk1,x-systemd.required-by=predastore.service,x-systemd.before=predastore.service 0 0' | sudo tee -a /etc/fstab

Once every drive has its line in /etc/fstab, mount them:

sudo systemctl daemon-reload
sudo mount -a

Note: Leave /var/lib/predastore/disk1 and friends owned by root. If a mount ever goes missing, that's what stops Predastore from quietly writing to the system disk in its place.

Match your MinIO setup

Predastore ships with an example config that already describes a working single-drive host, so there isn't a great deal to change. Copy it into place, and open it up in your editor of choice:

sudo cp /etc/predastore/predastore.toml.example /etc/predastore/predastore.toml
sudo $EDITOR /etc/predastore/predastore.toml

For the purposes of the migration, there are only two fields you need to set. region needs to match whatever your MinIO clients are configured with, as S3 clients sign every request for a specific region. [[auth]] holds the access key and secret your applications will use.

Keep in mind that these are root credentials, with full access to every bucket in the cluster (as mentioned in the standalone vs Spinifex discussion).

region = "us-east-1"

[[auth]]
access_key_id     = "<access-key>"
secret_access_key = "<secret-key>"
account_id        = "100000000001"

The example config also includes an example-bucket entry, which you can go ahead and delete.

Next, point the blob nodes at the drives. If MinIO only had one drive, set data_dir = "/var/lib/predastore/disk1" on the example's blob node, and the config is done.

Otherwise, replace the example's single blob node with one blob node per drive. Give each one its own id and port, and point its own data_dir at the drive's bind mount.

Leave the host's data_dir as it is, since the meta node doesn't name a directory of its own and keeps its state under /var/lib/predastore on the system disk. With three drives, the [[host]] block ends up looking like this:

[[host]]
id = 1
addr = "127.0.0.1"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 1
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 2
  role = "meta"
  port = 6660

  [[host.node]]
  id = 3
  role = "blob"
  port = 9991
  data_dir = "/var/lib/predastore/disk1"

  [[host.node]]
  id = 4
  role = "blob"
  port = 9992
  data_dir = "/var/lib/predastore/disk2"

  [[host.node]]
  id = 5
  role = "blob"
  port = 9993
  data_dir = "/var/lib/predastore/disk3"

Finally, set [rs] to match. Each object is split into data shards plus parity shards, each on a different drive, so parity is the number of drives you can lose, and data + parity can't exceed the number of drives.

RS(2,1) across three drives will survive losing any one of them (for two drives, use RS(1,1)). Your data survives, but the service won't start while a drive's bind mount is missing, so mount the replacement drive at the same path, recreate its .predastore directory, and run sudo mount -a before restarting. [rs] can't be changed once objects have been written, so make sure you're happy with it before moving any data across.

The configuration reference covers the details.

[rs]
data   = 2
parity = 1

Generate keys and a certificate

Generate the key used to encrypt data at rest, along with a self-signed TLS certificate. PREDA_SAN should list every hostname and IP address your clients will use to reach Predastore, otherwise they'll refuse to trust the certificate when you come to copy data across. It replaces the default list rather than adding to it, so include DNS:localhost and IP:127.0.0.1 if you'll also be connecting from the machine itself.

sudo PREDA_SAN="DNS:s3.example.com,IP:192.0.2.10" predastore-keygen /etc/predastore

Note: Make sure to back up /etc/predastore/master.key somewhere separate from your data backups. Without it your data is unreadable, but storing it alongside the data backups defeats the purpose of encrypting at all.

Start the service

Enable and start the service, then follow its logs to make sure it comes up without any errors:

sudo systemctl daemon-reload
sudo systemctl enable --now predastore
sudo journalctl -u predastore -f

Note: A few warnings while the nodes start up are normal, such as the gate reporting it isn't ready yet. It's errors, and the service restarting, that you're looking out for.

Check it's up

With the service running, it's worth confirming that Predastore is reachable, and that your client and cluster agree on the region.

The certificate is only readable by the predastore user, so copy it somewhere your own user can read (and over to any other machine you'll be running clients from). The rest of this guide uses this copy:

sudo install -m 0644 -o "$USER" /etc/predastore/server.pem ~/predastore-ca.pem

Point the AWS CLI at the credentials from your [[auth]] entry, and at the certificate:

export AWS_ACCESS_KEY_ID=<access-key>
export AWS_SECRET_ACCESS_KEY=<secret-key>
export AWS_CA_BUNDLE=~/predastore-ca.pem

Then create, list, and remove a test bucket. Listing buckets alone won't catch a region mismatch, which is why the check creates one.

aws s3 mb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 ls s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 rb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1

If all three commands complete without any errors, your Predastore cluster is ready for data. If you see AuthorizationHeaderMalformed ... incorrect region, the region your client is signing for doesn't match the one Predastore was set up with.

Multi-Node Multi-Drive - Standalone

This path is the equivalent of a distributed MinIO deployment. Every machine runs the same config file, and shares a single TLS certificate and encryption key.

The steps below assume three machines at 10.0.0.1, 10.0.0.2, and 10.0.0.3, but the process is the same for any number of them. You'll need Go (1.27 or newer), git, make, and openssl on each machine, as well as the AWS CLI on at least one of them for checking the cluster once it's up.

Build and install

On every machine, clone the repository and build the s3d binary:

git clone --branch v1.21.0 https://github.com/mulgadc/predastore.git
cd predastore
make build

Note: This guide was written against v1.21.0. The install tooling used below (make install, predastore-keygen, and the example config) first shipped in v1.20.0, so don't go any older than that.

Install it, along with its systemd service:

sudo make install

Create the dedicated predastore user, and the config and data directories:

sudo systemd-sysusers && sudo systemd-tmpfiles --create

make install also ships some kernel network settings. Hosts on separate machines talk to each other over QUIC, and need larger socket buffers than most distros provide by default. Without them, writes will start failing under load, with no obvious indication as to why. Apply them:

sudo sysctl --system

Share MinIO's drives

There's no need for new drives. Predastore can live on the ones MinIO is already using, in a directory of its own alongside MinIO's data, until you're done with MinIO in step 5. Each drive needs roughly as much free space again as MinIO is using on it, plus some headroom for re-syncs, which df -h will tell you.

The steps below assume each machine's MinIO drives are mounted at /mnt/disk1, /mnt/disk2, and so on. If they're pooled with RAID or ZFS, treat the pool as a single drive. The pool covers a failed drive, and Predastore covers a failed machine.

On every machine, create a directory for Predastore on each drive. The leading . matters, as MinIO treats any other top-level directory on its drives as a bucket, and will delete it along with that bucket.

sudo install -d -o predastore -g predastore -m 0700 /mnt/disk1/.predastore

The service isn't permitted to write anywhere outside /var/lib/predastore, so bind-mount each directory into it. The extra options make sure the service waits for the mount, and refuses to start without it:

sudo install -d -m 0755 /var/lib/predastore/disk1
echo '/mnt/disk1/.predastore /var/lib/predastore/disk1 none bind,nofail,x-systemd.requires-mounts-for=/mnt/disk1,x-systemd.required-by=predastore.service,x-systemd.before=predastore.service 0 0' | sudo tee -a /etc/fstab

Once every drive has its line in /etc/fstab, mount them:

sudo systemctl daemon-reload
sudo mount -a

Note: Leave /var/lib/predastore/disk1 and friends owned by root. If a mount ever goes missing, that's what stops Predastore from quietly writing to the system disk in its place.

Describe the cluster

A single config file describes the entire cluster, with one [[host]] block per machine. Each host gets its own address, its data directory, key, and certificate paths, and its own gate, meta, and blob nodes. Node ids must be unique across the whole file, not just within a host.

region needs to match whatever your MinIO clients are configured with, as S3 clients sign every request for a specific region. [[auth]] holds the access key and secret your applications will use.

Keep in mind that these are root credentials, with full access to every bucket in the cluster (as mentioned in the standalone vs Spinifex discussion).

[rs] determines how objects are spread across the machines. Each object is split into data shards plus parity shards, each on a different blob node. RS(2,1) across three machines will survive losing any one of them.

Check out the configuration reference for the full format.

version = 1
region  = "us-east-1"

[rs]
data   = 2
parity = 1

[[host]]
id   = 1
addr = "10.0.0.1"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 1
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 2
  role = "meta"
  port = 6660

  [[host.node]]
  id = 3
  role = "blob"
  port = 9991

[[host]]
id   = 2
addr = "10.0.0.2"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 4
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 5
  role = "meta"
  port = 6660

  [[host.node]]
  id = 6
  role = "blob"
  port = 9991

[[host]]
id   = 3
addr = "10.0.0.3"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 7
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 8
  role = "meta"
  port = 6660

  [[host.node]]
  id = 9
  role = "blob"
  port = 9991

[[auth]]
access_key_id     = "<access-key>"
secret_access_key = "<secret-key>"
account_id        = "100000000001"

Save it as /etc/predastore/predastore.toml on every machine.

Since every machine shares the same config file, each one needs telling which [[host]] block describes it. That lives in /etc/predastore/predastore.env. On each machine, copy the example into place, and open it up:

sudo cp /etc/predastore/predastore.env.example /etc/predastore/predastore.env
sudo $EDITOR /etc/predastore/predastore.env

Set PREDA_HOST_ID to the id of the [[host]] block that describes this machine, so 1 on 10.0.0.1, 2 on 10.0.0.2, and so on. The rest of the file can be left as it is.

PREDA_HOST_ID=1

Next, point the blob nodes at the drives. If each machine only has one drive, add data_dir = "/var/lib/predastore/disk1" to every host's blob node, and the config is done.

Otherwise, give each host one blob node per drive, each with its own id, port, and data_dir pointing at the drive's bind mount. Leave each host's data_dir as it is, since the meta node keeps its state there.

With two drives per machine, host 1 looks like this, and hosts 2 and 3 follow the same pattern with node ids 5 to 8 and 9 to 12:

[[host]]
id   = 1
addr = "10.0.0.1"
admin_port = 9099
data_dir = "/var/lib/predastore"
tls_cert = "/etc/predastore/server.pem"
tls_key = "/etc/predastore/server.key"
encryption_key = "/etc/predastore/master.key"

  [[host.node]]
  id = 1
  role = "gate"
  port = 8443
  bind_addr = "0.0.0.0"

  [[host.node]]
  id = 2
  role = "meta"
  port = 6660

  [[host.node]]
  id = 3
  role = "blob"
  port = 9991
  data_dir = "/var/lib/predastore/disk1"

  [[host.node]]
  id = 4
  role = "blob"
  port = 9992
  data_dir = "/var/lib/predastore/disk2"

Finally, set [rs] to match, which needs some care. Predastore spreads an object's shards across distinct blob nodes, but doesn't take into account which machine each blob node lives on, so two shards of the same object can land on two drives in the same machine.

To survive losing a whole machine, parity needs to be at least the number of drives per machine, and data + parity can't exceed the total number of drives. For three machines with two drives each, RS(2,2) survives losing any one machine. [rs] can't be changed once objects have been written, so make sure you're happy with it before moving any data across.

[rs]
data   = 2
parity = 2

Generate keys and a certificate

Every machine in the cluster needs to share the same TLS certificate and the same encryption key. This one is a bit of a trap; machines with their own certificates never manage to form a cluster.

So rather than running predastore-keygen on each machine straight away, generate the files once on a single machine and distribute them yourself.

On one machine, generate a self-signed certificate. Make sure subjectAltName lists every machine's address, as well as the hostname your clients will use:

sudo openssl req -x509 -newkey rsa:2048 -nodes -days 825 -subj '/CN=predastore' \
  -addext "subjectAltName=DNS:s3.example.com,IP:10.0.0.1,IP:10.0.0.2,IP:10.0.0.3" \
  -keyout /etc/predastore/server.key -out /etc/predastore/server.pem

On the same machine, generate the encryption key:

sudo sh -c 'umask 0177 && openssl rand -out /etc/predastore/master.key 32'

Copy server.pem, server.key and master.key into /etc/predastore on every other machine, using something that preserves file permissions, such as scp -p. Predastore refuses to start if either key is readable by anyone other than its owner, so if in doubt, tighten them up on each machine once they've landed:

sudo chmod 0600 /etc/predastore/master.key /etc/predastore/server.key

Then, on every machine, run predastore-keygen. It keeps the copied files, and sets their ownership:

sudo predastore-keygen /etc/predastore

Still on every machine, add the certificate to the system trust store, so each host trusts the others:

sudo cp /etc/predastore/server.pem /usr/local/share/ca-certificates/predastore.crt
sudo update-ca-certificates

Note: The trust store commands above are for Debian and Ubuntu. For RHEL and Fedora, see the Standalone TLS Trust section of the README. As with the single node setup, make sure to back up master.key somewhere separate from your data backups.

Start the service

On every machine, enable and start the service, then follow its logs:

sudo systemctl daemon-reload
sudo systemctl enable --now predastore
sudo journalctl -u predastore -f

Until every machine is up, the ones that started first will log errors about not being able to reach the others, which is expected. Once the service is running on every machine, the hosts will find each other and elect a leader, and those errors should stop. From then on, it's errors and the service restarting that you're looking out for.

Check it's up

With the cluster running, it's worth confirming that Predastore is reachable, and that your client and cluster agree on the region. Run the following from any one of the machines.

s3.example.com should point at every machine's gate, not just one, either with a DNS record per machine or a load balancer in front of them. Clients only talk to the gates that name resolves to, so if it names a single machine, losing that machine takes Predastore offline for your applications, even though the rest of the cluster carries on.

The certificate is only readable by the predastore user, so copy it somewhere your own user can read (and over to any other machine you'll be running clients from). The rest of this guide uses this copy:

sudo install -m 0644 -o "$USER" /etc/predastore/server.pem ~/predastore-ca.pem

Point the AWS CLI at the credentials from your [[auth]] entry, and at the certificate:

export AWS_ACCESS_KEY_ID=<access-key>
export AWS_SECRET_ACCESS_KEY=<secret-key>
export AWS_CA_BUNDLE=~/predastore-ca.pem

Then create, list, and remove a test bucket. Listing buckets alone won't catch a region mismatch, which is why the check creates one.

aws s3 mb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 ls s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 rb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1

If all three commands complete without any errors, your Predastore cluster is ready for data. If you see AuthorizationHeaderMalformed ... incorrect region, the region your client is signing for doesn't match the one Predastore was set up with.

Single-Node Single/Multi-Drive - Via Spinifex

A caveat before you start: Spinifex currently keeps all of Predastore's data on a single drive. While MinIO is still running, that means picking one of MinIO's drives to hold everything, which limits Predastore to that drive's capacity, with no protection against losing it. If MinIO's drives are already pooled with RAID or ZFS, the pool counts as one drive and keeps its redundancy. Support for spreading Predastore across multiple drives under Spinifex is on its way. Until then, if you need to survive a failed drive and can live without IAM, the standalone path is the better fit.

If you've gone with Spinifex, Predastore comes along for the ride. It's installed and configured as part of Spinifex, so there's no need to set it up separately.

On a single node, Spinifex runs Predastore with one blob node and RS(1,0), which means a single copy of each object with no parity.

Run the installer

Before installing, make sure the machine has a Linux bridge configured for VM networking. The bridge setup guide walks through setting one up.

Run the installer:

curl -fsSL https://install.mulgadc.com | bash

Then set up OVN networking. --management also starts OVN's central services, which a single node needs to run itself:

sudo /usr/local/share/spinifex/setup-ovn.sh --management

Initialize the node

Initializing the node generates Predastore's config, keys, and certificates for you. Just like the standalone setup, --region needs to match your MinIO clients, as Spinifex defaults to ap-southeast-2.

sudo spx admin init --node node1 --nodes 1 --region us-east-1

Share one of MinIO's drives

Predastore can live on one of the drives MinIO is already using, in a directory of its own alongside MinIO's data, until you're done with MinIO in step 5. Pick the drive with the most free space, as it needs enough room for a full copy of your data, plus some headroom for re-syncs.

The steps below assume that drive is mounted at /mnt/disk1. Create a directory for Predastore on it. The leading . matters, as MinIO treats any other top-level directory on its drives as a bucket, and will delete it along with that bucket.

sudo install -d -o spinifex-storage -g spinifex -m 0700 /mnt/disk1/.spinifex-predastore

Predastore's service isn't permitted to write anywhere outside /var/lib/spinifex/predastore, so bind-mount the directory over it. The extra options make sure the service waits for the mount, and won't write to the system disk without it:

sudo chown root:root /var/lib/spinifex/predastore
sudo chmod 0755 /var/lib/spinifex/predastore
echo '/mnt/disk1/.spinifex-predastore /var/lib/spinifex/predastore none bind,nofail,x-systemd.requires-mounts-for=/mnt/disk1,x-systemd.required-by=spinifex-predastore.service,x-systemd.before=spinifex-predastore.service 0 0' | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount -a

Start the services

Start Spinifex's services, Predastore included:

sudo systemctl start spinifex.target

Spinifex's certificate only covers the machine's own hostname and addresses. If your clients will reach Predastore by any other name, such as s3.example.com, add it to the certificate, and restart the services to pick it up:

sudo spx admin cert renew --extra-dns s3.example.com
sudo systemctl restart spinifex.target

Check it's up

With Spinifex running, it's worth confirming that Predastore is reachable, and that your client and cluster agree on the region.

Copy Spinifex's CA certificate somewhere your own user can read (and over to any other machine you'll be running clients from). The rest of this guide uses this copy:

sudo install -m 0644 -o "$USER" /etc/spinifex/ca.pem ~/predastore-ca.pem

Point the AWS CLI at the spinifex profile created during setup, which takes care of the credentials, and at your copy of the certificate:

export AWS_PROFILE=spinifex
export AWS_CA_BUNDLE=~/predastore-ca.pem

Then create, list, and remove a test bucket. Listing buckets alone won't catch a region mismatch, which is why the check creates one.

aws s3 mb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 ls s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 rb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1

If all three commands complete without any errors, your Predastore cluster is ready for data. If you see AuthorizationHeaderMalformed ... incorrect region, the region your client is signing for doesn't match the one you passed to spx admin init.

Multi-Node Multi-Drive - Via Spinifex

A caveat before you start: Spinifex currently keeps all of each machine's Predastore data on a single drive. While MinIO is still running, that means picking one of MinIO's drives on each machine, which limits each machine's share of Predastore to that drive's capacity. Predastore's erasure coding still covers a failed machine, and it treats a failed drive as a failed machine, but there's no protection against losing a drive beyond that. If MinIO's drives are already pooled with RAID or ZFS, the pool counts as one drive and keeps its redundancy. Support for spreading Predastore across multiple drives under Spinifex is on its way. Until then, if you need to survive more failed drives than failed machines and can live without IAM, the standalone path is the better fit.

Setting up a multi-node Spinifex cluster involves a fair bit more than Predastore alone, including configuring OVN networking on each server.

Rather than repeating all of it here, I'll point you to Spinifex's multi-node install guide for the full procedure, and only call out the parts that matter for the migration.

Spinifex runs one Predastore blob node per machine, so its erasure coding protects you against losing a machine.

Follow the install guide

Work through steps 1 to 3 of the multi-node guide, which cover installing Spinifex, setting the node IP variables, and configuring OVN networking on each server.

Form the cluster

Step 4 of the guide forms the cluster, with one change; --region needs to match your MinIO clients, on both the init and every join. Spinifex picks the erasure coding for you, using RS(1,1) for two machines and RS(2,1) for three or more.

On server 1, initialize the cluster:

sudo spx admin init --force --node node1 --nodes 3 \
  --bind $SPINIFEX_NODE1 --cluster-bind $SPINIFEX_NODE1 \
  --port 4432 --region us-east-1 --az us-east-1a

While init is still running, join each other server, using the token from init's output. Change --node and the bind addresses to match each server:

sudo spx admin join --force --node node2 \
  --bind $SPINIFEX_NODE2 --cluster-bind $SPINIFEX_NODE2 \
  --host $SPINIFEX_NODE1:4432 --token <token-from-init-output> \
  --region us-east-1 --az us-east-1a

Share one of MinIO's drives

On each machine, Predastore can live on one of the drives MinIO is already using, in a directory of its own alongside MinIO's data, until you're done with MinIO in step 5. Pick the drive with the most free space, as it needs roughly as much room as MinIO is using across all of that machine's drives combined, plus some headroom for re-syncs.

The steps below assume that drive is mounted at /mnt/disk1. On every server, create a directory for Predastore on it. The leading . matters, as MinIO treats any other top-level directory on its drives as a bucket, and will delete it along with that bucket.

sudo install -d -o spinifex-storage -g spinifex -m 0700 /mnt/disk1/.spinifex-predastore

Predastore's service isn't permitted to write anywhere outside /var/lib/spinifex/predastore, so on every server, bind-mount the directory over it. The extra options make sure the service waits for the mount, and won't write to the system disk without it:

sudo chown root:root /var/lib/spinifex/predastore
sudo chmod 0755 /var/lib/spinifex/predastore
echo '/mnt/disk1/.spinifex-predastore /var/lib/spinifex/predastore none bind,nofail,x-systemd.requires-mounts-for=/mnt/disk1,x-systemd.required-by=spinifex-predastore.service,x-systemd.before=spinifex-predastore.service 0 0' | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount -a

Start the services

Finish off with steps 5 and 6 of the multi-node guide to start the services and verify the cluster.

As with a single node, Spinifex's certificates only cover each server's own hostname and addresses. If your clients will reach Predastore by any other name, such as s3.example.com, add it on every server, and restart the services to pick it up:

sudo spx admin cert renew --extra-dns s3.example.com
sudo systemctl restart spinifex.target

Check it's up

With the cluster running, it's worth confirming that Predastore is reachable, and that your client and cluster agree on the region. Run the following on server 1.

Copy Spinifex's CA certificate somewhere your own user can read (and over to any other machine you'll be running clients from). The rest of this guide uses this copy:

sudo install -m 0644 -o "$USER" /etc/spinifex/ca.pem ~/predastore-ca.pem

Point the AWS CLI at the spinifex profile created during setup, which takes care of the credentials, and at your copy of the certificate:

export AWS_PROFILE=spinifex
export AWS_CA_BUNDLE=~/predastore-ca.pem

Then create, list, and remove a test bucket. Listing buckets alone won't catch a region mismatch, which is why the check creates one.

aws s3 mb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 ls s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1
aws s3 rb s3://predastore-test --endpoint-url https://s3.example.com:8443 --region us-east-1

If all three commands complete without any errors, your Predastore cluster is ready for data. If you see AuthorizationHeaderMalformed ... incorrect region, the region your client is signing for doesn't match the one you passed to spx admin init.