Welcome to TERN Knowledge Base

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Rclone is the preferred upload method for TERN data, as the upload is fast and can verify that the files uploaded to Cloudstor are the same as the original file via checksum verification.

This document describes how Rclone can be used to upload large amounts of data to the TERN CloudStor storage.

A step by step guide to setup Rclone can be found at Step by step Rclone Setup for CloudStor file upload.
Advanced users can follow the compacted Advanced Rclone for CloudStor file upload guide

Upload files to Cloudstor

The following is an example of a command you can run in the terminal / command line, from your rclone direcotry, to upload data to Cloudstor:

rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 source-dir/ AARNET:dest_dir/

The following options are provided to this command to upload the files.

  • --transfers 36

    • 36 concurrent transfers are performed.

  • --progress

    • Show progress during transfer

  • --checksum

    • skip files if they already exist based on checksum and size.
      By default the files are skippen based on modification time and size.

  • --checkers 48

    • This option increases the number of tasks that are checking to see if the file already exists on the remote.
      It should be greater than transfers to make sure that transfers is kept at capacity.

  • --timeout 0

    • This is the allowed Input Output idle timeout. By default this is 5 minutes.

  • source-dir/

    • The local directory you want to push to Cloudstor,
      ie C:\tmp\images\20200103\ or external drive G:\images\20200103\ (Windows)
      or /tmp/images/20200103 (Mac / Linux)

  • AARNET:dest_dir/

A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/

Example

For this example, the Cloudstor Access level given was edit for Folder_1 and read for Folder_2. Additionally, edit access was given to Folder_2a, which is a subfolder of Folder_2 as per below

Folder_1
Folder_2 -> Folder_2a

All 3 folders show in your Cloudstor Shared folder, because you were added as a user to Folder_2a as well as to Folder_2.

We are uploading data to the shared folder Folder_1 using the following rclone command:

rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 /tmp/images/ AARNET:Shared/Folder_1/

 

What you will see in the command window, during and finally

Retries for incorrectly transferred files by Rclone

Where is my local data

Example with a real time directory

Issues


  • No labels