Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.
Rclone is the preferred upload method for TERN data, as the upload is fast and can verify that the files uploaded to Cloudstor are the same as the original file via checksum verification.
This document describes how Rclone can be used to upload large amounts of data to the TERN CloudStor storage.
A step by step guide to setup Rclone can be found at Step by step Rclone Setup for CloudStor file upload.
Advanced users can follow the compacted Advanced Rclone for CloudStor file upload guide
Upload files to Cloudstor
The following is an example of a command you can run in the terminal / command line, from your rclone direcotry, to upload data to Cloudstor:
The following options are provided to this command to upload the files.
36 concurrent transfers are performed.
Show progress during transfer
skip files if they already exist based on checksum and size.
By default the files are skippen based on modification time and size.
This option increases the number of tasks that are checking to see if the file already exists on the remote.
It should be greater than transfers to make sure that transfers is kept at capacity.
This is the allowed Input Output idle timeout. By default this is 5 minutes.
The local directory you want to push to Cloudstor,
ie C:\tmp\images\20200103\ or external drive G:\images\20200103\ (Windows)
or /tmp/images/20200103 (Mac / Linux)
AARNET is the connection that was setup for rclone according toStep by step Rclone Setup for CloudStor file upload
The directory where you want to upload the files to in Cloudstor
In order to upload the files, you need Edit access to the folder
A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/
Example Initial Upload
For this example, the Cloudstor Access level given was edit for
Folder_1 and read for
Folder_2. Additionally, edit access was given to
Folder_2a, which is a subfolder of
Folder_2 as per below
All 3 folders show in the users’s Cloudstor Shared folder, because the folder
Folder_2a and folder
Folder_2 were shared with the user individually.
We are uploading data to the shared folder
Folder_1 using the following rclone command:
Rclone will start the copy of the files and will provide you real time feedback about the processes running. By default, each transaction is retried 10 times in case there is an issue with the transfer.
The following is an example what will display during the Rclone process.
Once the cloning is finished, the final status will be shown.
In this case, there were some initial errors with creating directories and files.
Rclone will retry the attempt 10 times to progress the copy.
In this case, the initial full clone attempt Failed (at line 12).
The retry was successfull (at line 14):
Attempt 2/3 succeeded
The Transferred file size is shown, including the checks and the Transferred file count.
Once the initial upload is executed, every sequental rclone call will only copy the files added after the last Rclone run. The outcome will be: