Rclone file upload to Cloudstor

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Rclone is the preferred upload method for TERN data, as the upload is fast and can verify that the files uploaded to Cloudstor are the same as the original file via checksum verification.

This document describes how Rclone can be used to upload large amounts of data to the TERN CloudStor storage.

A step by step guide to setup Rclone can be found at https://ternaus.atlassian.net/wiki/spaces/TERNSup/pages/1344274476.
Advanced users can follow the compacted https://ternaus.atlassian.net/wiki/spaces/TERNSup/pages/1344372779

Upload files to Cloudstor

The following is an example of a command you can run in the terminal / command line to upload data to Cloudstor:

1 rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 source-dir/ AARNET:dest_dir/

The following options are provided to this command to upload the files.

  • --transfers 36

    • 36 concurrent transfers are performed.

  • --progress

    • Show progress during transfer

  • --checksum

    • skip files if they already exist, based on checksum and size.
      By default the files are skipped based on modification time and size.

  • --checkers 48

    • This option increases the number of tasks that are checking to see if the file already exists on the remote. This value should be greater than transfers to make sure that transfers is kept at capacity.

  • --timeout 0

    • This is the allowed Input Output idle timeout. By default this is 5 minutes.

  • source-dir/

    • The local directory from which you want to push the files to Cloudstor,
      ie C:\tmp\images\20200103\ or external drive G:\images\20200103\ (Windows)
      or /tmp/images/20200103 (Mac / Linux)

  • AARNET:dest_dir/

A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/

Example Initial Upload

For this example, the Cloudstor Access level given was edit for Folder_1 and read for Folder_2. Additionally, edit access was given to Folder_2a, which is a subfolder of Folder_2 as per below

1 2 Folder_1 Folder_2 -> Folder_2a

All 3 folders appear in the users’s Cloudstor Shared folder. This is because the folders Folder_2a and Folder_2 were shared with the user with specific access.

We are uploading data to the shared folder Folder_1, from our local folder /tmp/images using the following Rclone command:

1 rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 /tmp/images/ AARNET:Shared/Folder_1/

Rclone will start the copy of the files and will provide you real time feedback about the processes running. By default, each transaction is retried 10 times in case there is an issue with the transfer.

Rclone in progress

The following is an example what will display during the Rclone process.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 2020-10-09 14:14:49 ERROR : lai/core1ha/20180706/DSC00165.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai: Sabre\DAV\Exception\Forbidden: 403 Forbidden 2020-10-09 14:14:49 ERROR : lai/core1ha/20180706/DSC00178_C3.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai: Sabre\DAV\Exception\Forbidden: 403 Forbidden 2020-10-09 14:14:50 ERROR : lai/core1ha/20180706/DSC00187_B2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden 2020-10-09 14:14:52 ERROR : lai/core1ha/20180706/DSC00183_E2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden 2020-10-09 14:14:54 ERROR : lai/core1ha/20180706/DSC00168_D5.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:14:54 ERROR : lai/core1ha/20180706/DSC00167_C5.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:14:55 ERROR : lai/core1ha/20180706/DSC00193_G1.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden 2020-10-09 14:14:56 ERROR : lai/core1ha/20180706/DSC00182_F2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden 2020-10-09 14:15:46 ERROR : lai/core1ha/20190807/DHP LAI Boyagin 07082019.docx: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden Transferred: 1.941G / 4.358 GBytes, 45%, 6.073 MBytes/s, ETA 6m47s Errors: 9 (retrying may help) Transferred: 231 / 637, 36% Elapsed time: 5m27.2s Transferring: * lai/core1ha/20180222/index 2.JPG:100% /7.906M, 1008.665k/s, 0s * lai/core1ha/20180706/DSC00195.JPG:100% /5.250M, 0/s, 0s <...and some more lines that have been taken out...> * lai/core1ha/20180222/DSC00153_R6_T9.JPG:100% /8.281M, 112.646k/s, 0s * lai/core1ha/20180222/D…Boyagin_22_2_2018.docx:100% /228.990k, 28.684k/s, 0s * lai/core1ha/20190510/D… Boyagin 10052019.docx:100% /195.322k, 48.850k

Rclone in finished

Once the cloning is finished, the final result of the cloning will be shown.
In this case, there were some initial errors with creating directories and files, which were resolved after retrying the command. Rclone will retry the attempt 3 times to progress the copy.

In this case, the initial full clone attempt Failed (at line 12), and the next try was successfull (at line 14): Attempt 2/3 succeeded

The Transferred file size is shown, including the checks and the Transferred file count (line 15, 16 and 17).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00569.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00547.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00552.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00577.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00546.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00573.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00571.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00554.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00570.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict 2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00568.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden 2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00557.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden 2020-10-09 14:44:46 ERROR : Attempt 1/3 failed with 11 errors and: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden 2020-10-09 14:44:52 NOTICE: webdav root 'Shared/Folder_1': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only 2020-10-09 14:49:59 ERROR : Attempt 2/3 succeeded Transferred: 4.477G / 4.477 GBytes, 100%, 4.153 MBytes/s, ETA 0s Checks: 635 / 635, 100% Transferred: 646 / 646, 100% Elapsed time: 18m24.0s

Rclone synchronisation

Once the initial upload is executed, every sequential Rclone call will only copy the files that were added or modified after the last Rclone run. With no changes to the directory, the outcome will be:

1 2 3 Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Checks: 646 / 646, 100% Elapsed time: 2.0s