Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.
Rclone is the preferred upload method for TERN data, as the upload is fast and can verify that the files uploaded to Cloudstor are the same as the original file via checksum verification.
This document describes how Rclone can be used to upload large amounts of data to the TERN CloudStor storage.
A step by step guide to setup Rclone can be found at Step by step Rclone Setup for CloudStor file upload.
Advanced users can follow the compacted Advanced Rclone for CloudStor file upload guide
Upload files to Cloudstor
The following is an example of a command you can run in the terminal / command line, from your rclone direcotry, to upload data to Cloudstor:
The following options are provided to this command to upload the files.
36 concurrent transfers are performed.
Show progress during transfer
skip files if they already exist based on checksum and size.
By default the files are skippen based on modification time and size.
This option increases the number of tasks that are checking to see if the file already exists on the remote.
It should be greater than transfers to make sure that transfers is kept at capacity.
This is the allowed Input Output idle timeout. By default this is 5 minutes.
The local directory you want to push to Cloudstor,
ie C:\tmp\images\20200103\ or external drive G:\images\20200103\ (Windows)
or /tmp/images/20200103 (Mac / Linux)
AARNET is the connection that was setup for rclone according toStep by step Rclone Setup for CloudStor file upload
The directory where you want to upload the files to in Cloudstor
In order to upload the files, you need Edit access to the folder
A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/
For this example, the Cloudstor Access level given was edit for
Folder_1 and read for
Folder_2. Additionally, edit access was given to
Folder_2a, which is a subfolder of
Folder_2 as per below
All 3 folders show in your Cloudstor Shared folder, because you were added as a user to
Folder_2a as well as to
We are uploading data to the shared folder
Folder_1 using the following rclone command:
What you will see in the command window, during and finally
Retries for incorrectly transferred files by Rclone
Where is my local data
Example with a real time directory