Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Rclone is the preferred upload method for TERN data, as the upload is fast and can verify that the files uploaded to Cloudstor are the same as the original file via checksum verification.

This document describes how Rclone can be used to upload large amounts of data to the TERN CloudStor storage.

A step by step guide to setup Rclone can be found at Step by step Rclone Setup for CloudStor file upload.
Advanced users can follow the compacted Advanced Rclone for CloudStor file upload guide

Upload files to Cloudstor

The following is an example of a command you can run in the terminal / command line to upload data to Cloudstor:

rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 source-dir/ AARNET:dest_dir/

The following options are provided to this command to upload the files.

A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/

Example Initial Upload

For this example, the Cloudstor Access level given was edit for Folder_1 and read for Folder_2. Additionally, edit access was given to Folder_2a, which is a subfolder of Folder_2 as per below

Folder_1
Folder_2 -> Folder_2a

All 3 folders appear in the users’s Cloudstor Shared folder. This is because the folders Folder_2a and Folder_2 were shared with the user with specific access.

We are uploading data to the shared folder Folder_1, from our local folder /tmp/images using the following Rclone command:

rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 /tmp/images/ AARNET:Shared/Folder_1/

Rclone will start the copy of the files and will provide you real time feedback about the processes running. By default, each transaction is retried 10 times in case there is an issue with the transfer.

Rclone in progress

The following is an example what will display during the Rclone process.

2020-10-09 14:14:49 ERROR : lai/core1ha/20180706/DSC00165.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:49 ERROR : lai/core1ha/20180706/DSC00178_C3.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:50 ERROR : lai/core1ha/20180706/DSC00187_B2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:52 ERROR : lai/core1ha/20180706/DSC00183_E2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:54 ERROR : lai/core1ha/20180706/DSC00168_D5.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:14:54 ERROR : lai/core1ha/20180706/DSC00167_C5.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:14:55 ERROR : lai/core1ha/20180706/DSC00193_G1.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:56 ERROR : lai/core1ha/20180706/DSC00182_F2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:15:46 ERROR : lai/core1ha/20190807/DHP LAI Boyagin 07082019.docx: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden
Transferred:   	    1.941G / 4.358 GBytes, 45%, 6.073 MBytes/s, ETA 6m47s
Errors:                 9 (retrying may help)
Transferred:          231 / 637, 36%
Elapsed time:      5m27.2s
Transferring:
 *              lai/core1ha/20180222/index 2.JPG:100% /7.906M, 1008.665k/s, 0s
 *             lai/core1ha/20180706/DSC00195.JPG:100% /5.250M, 0/s, 0s
 <...and some more lines that have been taken out...>
 *       lai/core1ha/20180222/DSC00153_R6_T9.JPG:100% /8.281M, 112.646k/s, 0s
 * lai/core1ha/20180222/D…Boyagin_22_2_2018.docx:100% /228.990k, 28.684k/s, 0s
 * lai/core1ha/20190510/D… Boyagin 10052019.docx:100% /195.322k, 48.850k

Rclone in finished

Once the cloning is finished, the final result of the cloning will be shown.
In this case, there were some initial errors with creating directories and files, which were resolved after retrying the command. Rclone will retry the attempt 3 times to progress the copy.

In this case, the initial full clone attempt Failed (at line 12), and the next try was successfull (at line 14): Attempt 2/3 succeeded

The Transferred file size is shown, including the checks and the Transferred file count (line 15, 16 and 17).

2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00569.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00547.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00552.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00577.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00546.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00573.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00571.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00554.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00570.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00568.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00557.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:44:46 ERROR : Attempt 1/3 failed with 11 errors and: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:44:52 NOTICE: webdav root 'Shared/Folder_1': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
2020-10-09 14:49:59 ERROR : Attempt 2/3 succeeded
Transferred:   	    4.477G / 4.477 GBytes, 100%, 4.153 MBytes/s, ETA 0s
Checks:               635 / 635, 100%
Transferred:          646 / 646, 100%
Elapsed time:     18m24.0s

Rclone synchronisation

Once the initial upload is executed, every sequential Rclone call will only copy the files that were added or modified after the last Rclone run. With no changes to the directory, the outcome will be:

Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:               646 / 646, 100%
Elapsed time:         2.0s