Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This document provides examples on how to upload files to Cloudstor for TERN in more detail with specific examples.

You need specific access to the upload folder to edit the files.

explain Read access vs Read/Write access for specific users

explain directories availble in Shared folder in CloudstorRclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Rclone is the preferred upload method for TERN data, as the upload is fast and can verify that the files uploaded to Cloudstor are the same as the original file via checksum verification.

This document describes how Rclone can be used to upload large amounts of data to the TERN CloudStor storage.

A step by step guide to setup Rclone can be found at Step by step Rclone Setup for CloudStor file upload.
Advanced users can follow the compacted Advanced Rclone for CloudStor file upload guide

Upload files to Cloudstor

Generic command explained in detail for the options

What you will see in the command window, during and finally

Retries for incorrectly transferred files by Rclone

Where is my local data

Example with a real time directory

IssuesThe following is an example of a command you can run in the terminal / command line, from your rclone direcotry, to upload data to Cloudstor:

Code Block
breakoutModewide

rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 source-dir/ AARNET:dest_dir/

In The following options are provided to this command to upload the files.

  • --transfers

...

  • 36

    • 36 concurrent transfers

...

    • are performed.

  • --progress

    • Show progress during transfer

  • --checksum

    • skip files if they already exist based on checksum and size.
      By default the files are skippen based on modification time and size.

  • --checkers 48

    • This option increases the number of tasks that are checking to see if the file already exists on the remote.
      It should be greater than transfers to make sure that transfers is kept at capacity.

...

  • --

...

  • timeout 0

    • This is the allowed Input Output idle timeout. By default this is 5 minutes.

  • source-dir/

    • The local directory you want to push to Cloudstor,
      ie C:\tmp\images\20200103\ or external drive G:\images\20200103\ (Windows)
      or /tmp/images/20200103 (Mac / Linux)

  • AARNET:dest_dir/

A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/

Example

For this example, the Cloudstor Access level given was edit for Folder_1 and read for Folder_2. Additionally, edit access was given to Folder_2a, which is a subfolder of Folder_2 as per below

Code Block
Folder_1
Folder_2 -> Folder_2a

All 3 folders show in your Cloudstor Shared folder, because you were added as a user to Folder_2a as well as to Folder_2.

...

We are uploading data to the shared folder Folder_1 using the following rclone command:

Code Block
breakoutModewide
rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 /tmp/images/ AARNET:Shared/Folder_1/

Code Block

What you will see in the command window, during and finally

Retries for incorrectly transferred files by Rclone

Where is my local data

Example with a real time directory

Issues