Advanced Rclone for CloudStor file upload guide

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

A step by step guide to setup Rclone can be found at https://ternaus.atlassian.net/wiki/spaces/TERNSup/pages/1344274476.

A step by step guide for TERN specific Rclone uploads can be found at https://ternaus.atlassian.net/wiki/spaces/TERNSup/pages/1344340027

 

Set up

  1. Follow instructions for installation. https://ternaus.atlassian.net/wiki/spaces/TERNSup/pages/1344274476

  2. Configure a remote. For example, here are the instructions for a WebDAV remote, which is needed to configure CloudStor: https://rclone.org/webdav/. In our case it was configured using environment variables, where AARNET is the customised name given to this remote connection.

    1 2 3 4 5 export RCLONE_CONFIG_AARNET_TYPE=webdav export RCLONE_CONFIG_AARNET_URL="https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/" export RCLONE_CONFIG_AARNET_VENDOR="owncloud" export RCLONE_CONFIG_AARNET_USER="<email-address-here>" export RCLONE_CONFIG_AARNET_PASS="<your-cloudstor-password"

Commands

1 rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 source-dir/ AARNET:dest_dir/

In this command --transfers is used and set to perform 36 concurrent transfers from source-dir to dest-dir on the remote AARNET location.

The --checkers option is used to increase the number of tasks that are checking to see if the file already exists on the remote. It should be greater than transfers to make sure that transfers is kept at capacity.

The --checksum option is used to skip files if they already exist based on checksum, not mod-time and size.

You can tweak the number of transfers based on the files that are being copied, but be aware that you will hit a limit.

A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/

Caveats

  • Very small files will take a long time to copy. There’s overhead to open a file and do all of the checks against CloudStor.