Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.
Rclone is the preferred upload method for TERN data, as the upload is fast and can verify that the files uploaded to Cloudstor are the same as the original file via checksum verification.
This document describes how Rclone can be used to upload large amounts of data to the TERN CloudStor storage.
A step by step guide to setup Rclone can be found at Step by step Rclone Setup for CloudStor file upload.
Advanced users can follow the compacted Advanced Rclone for CloudStor file upload guide
Upload files to Cloudstor
The following is an example of a command you can run in the terminal / command line to upload data to Cloudstor:
The following options are provided to this command to upload the files.
36 concurrent transfers are performed.
Show progress during transfer
skip files if they already exist, based on checksum and size.
By default the files are skipped based on modification time and size.
This option increases the number of tasks that are checking to see if the file already exists on the remote. This value should be greater than transfers to make sure that transfers is kept at capacity.
This is the allowed Input Output idle timeout. By default this is 5 minutes.
The local directory from which you want to push the files to Cloudstor,
ie C:\tmp\images\20200103\ or external drive G:\images\20200103\ (Windows)
or /tmp/images/20200103 (Mac / Linux)
AARNET is the connection that was setup for Rclone according toStep by step Rclone Setup for CloudStor file upload
The directory where you want to upload the files to in Cloudstor
Edit access to the Cloudstor folder is required to upload files.
A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/
Example Initial Upload
For this example, the Cloudstor Access level given was edit for
Folder_1 and read for
Folder_2. Additionally, edit access was given to
Folder_2a, which is a subfolder of
Folder_2 as per below
All 3 folders appear in the users’s Cloudstor Shared folder. This is because the folders
Folder_2 were shared with the user with specific access.
We are uploading data to the shared folder
Folder_1, from our local folder
/tmp/images using the following Rclone command:
Rclone will start the copy of the files and will provide you real time feedback about the processes running. By default, each transaction is retried 10 times in case there is an issue with the transfer.
Rclone in progress
The following is an example what will display during the Rclone process.
Rclone in finished
Once the cloning is finished, the final result of the cloning will be shown.
In this case, there were some initial errors with creating directories and files, which were resolved after retrying the command. Rclone will retry the attempt 3 times to progress the copy.
In this case, the initial full clone attempt Failed (at line 12), and the next try was successfull (at line 14):
Attempt 2/3 succeeded
The Transferred file size is shown, including the checks and the Transferred file count (line 15, 16 and 17).
Once the initial upload is executed, every sequential Rclone call will only copy the files that were added or modified after the last Rclone run. With no changes to the directory, the outcome will be: