Migrate data from S3 and EFS to Gcore

Gcore offers S3-compatible Object Storage and NFS-based File Shares, both of which can receive data migrated from Amazon S3 or EFS using rclone, a popular CLI tool for cloud storage management.

AWS bills outgoing traffic. Before migrating large datasets, check transfer costs on the AWS pricing calculator.

The rclone remotes configured in each section can be reused across scenarios — once [aws-s3] or [gcore-os] is configured, skip that step in subsequent migrations.

Migrate data from S3 to Gcore

Two Gcore destinations are available: Object Storage for object-based access, and File Shares for NFS-based access from instances.

Gcore Object Storage

Gcore Object Storage accepts data from Amazon S3 buckets directly, using the S3-compatible API with no format conversion. This migration runs on a Linux machine with public Internet access and rclone installed.

1. Collect AWS account credentials

   Use an existing AWS CLI user or create a new one via IAM. The user must have read access to the source bucket. Required credentials: access key ID, secret access key, source bucket region, and source bucket name.

2. Create a Gcore Object Storage bucket

   Create a new Object Storage bucket following the Create S3 or SFTP storage instructions. At the end of this process, record the access key ID, secret access key, endpoint, and bucket name.

3. Configure rclone remotes

   Add rclone remotes for AWS and Gcore to the rclone config file:

   ~/.config/rclone/rclone.conf
   

   Paste the following configuration, replacing placeholders with the values from the previous steps:

   [aws-s3]
   type = s3
   provider = AWS
   access_key_id = <AWS_ACCESS_KEY_ID>
   secret_access_key = <AWS_SECRET_ACCESS_KEY>
   region = <AWS_REGION>
   acl = private
   
   [gcore-os]
   type = s3
   provider = Other
   access_key_id = <GCORE_ACCESS_KEY_ID>
   secret_access_key = <GCORE_SECRET_ACCESS_KEY>
   endpoint = <GCORE_STORAGE_ENDPOINT>
   acl = private
   
4. Migrate the data

   Sync the data from S3 to Gcore Object Storage:

   rclone sync aws-s3://<SOURCE_BUCKET> gcore-os://<DESTINATION_BUCKET>
   

   Replace SOURCE_BUCKET with the name of the Amazon S3 bucket and DESTINATION_BUCKET with the name of the Gcore Object Storage bucket.

Gcore File Share (NFS)

Gcore File Shares make S3 data accessible as a mounted NFS volume from instances or containers. Complete these steps on a Gcore instance with public Internet access, Linux, and rclone installed.

1. Collect AWS account credentials

   Gather the same AWS credentials and bucket details as described in the Object Storage migration above.

2. Create and mount a Gcore File Share

   Create a new File Share following the Configure File Shares instructions. At the end of this process, the mount path of the File Share is available.

3. Configure rclone remote

   Add the [aws-s3] remote to ~/.config/rclone/rclone.conf using the same configuration pattern as in the Object Storage migration.

4. Migrate the data

   Sync the data from S3 directly to the mounted File Share:

   rclone sync aws-s3://<SOURCE_BUCKET> <DESTINATION_PATH>
   

   Replace SOURCE_BUCKET with the name of the Amazon S3 bucket and DESTINATION_PATH with the Gcore File Share mount path from step 2.

Migrate data from EFS to Gcore

Amazon EFS does not expose a public S3 API. Object Storage receives data directly from a mounted EFS path; File Shares require an intermediate step through Object Storage.

Gcore Object Storage

Gcore Object Storage receives EFS data through a direct rclone transfer from a mounted EFS path. This migration runs on an EC2 instance with access to the EFS, public Internet access, Linux, and rclone installed.

1. Create a Gcore Object Storage bucket

   Create a new Object Storage bucket using the same steps as in the S3 migration above. The same Gcore credentials are needed: access key ID, secret access key, endpoint, and bucket name.

2. Configure rclone remote

   Add the rclone remote for Gcore to the rclone config file:

   ~/.config/rclone/rclone.conf
   

   Paste the following configuration, replacing placeholders with the credentials from step 1:

   [gcore-os]
   type = s3
   provider = Other
   access_key_id = <GCORE_ACCESS_KEY_ID>
   secret_access_key = <GCORE_SECRET_ACCESS_KEY>
   endpoint = <GCORE_STORAGE_ENDPOINT>
   acl = private
   
3. Migrate the data

   Sync the EFS data to Gcore Object Storage:

   rclone sync <SOURCE_PATH> gcore-os://<DESTINATION_BUCKET>
   

   Replace SOURCE_PATH with the Amazon EFS mount path and DESTINATION_BUCKET with the Gcore Object Storage bucket name from step 1.

Gcore File Share (NFS)

Amazon EFS and Gcore File Shares are each private to their own cloud infrastructure.

Direct migration between Amazon EFS and Gcore File Shares is not possible. The steps below describe a two-step workaround: first migrate data to Gcore Object Storage using the steps above, then copy it to the File Share.

This migration requires a Gcore instance with public Internet access, Linux, and rclone.

1. Create and mount a Gcore File Share

   Create a new File Share following the Configure File Shares instructions. Note the mount path for use in step 3.

2. Configure rclone

   If not already configured, add the [gcore-os] remote to ~/.config/rclone/rclone.conf using the same configuration from the Object Storage migration above.

3. Migrate the data

   Pull the data from Object Storage to the mounted File Share:

   rclone sync gcore-os://<SOURCE_BUCKET> <DESTINATION_PATH>
   

   Replace SOURCE_BUCKET with the name of the Gcore Object Storage bucket and DESTINATION_PATH with the Gcore File Share mount path from step 1.