Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following is an example of a command you can run in the terminal / command line , from your rclone direcotry, to upload data to Cloudstor:

...

  • --transfers 36

    • 36 concurrent transfers are performed.

  • --progress

    • Show progress during transfer

  • --checksum

    • skip files if they already exist, based on checksum and size.
      By default the files are skippen skipped based on modification time and size.

  • --checkers 48

    • This option increases the number of tasks that are checking to see if the file already exists on the remote.
      It This value should be greater than transfers to make sure that transfers is kept at capacity.

  • --timeout 0

    • This is the allowed Input Output idle timeout. By default this is 5 minutes.

  • source-dir/

    • The local directory from which you want to push the files to Cloudstor,
      ie C:\tmp\images\20200103\ or external drive G:\images\20200103\ (Windows)
      or /tmp/images/20200103 (Mac / Linux)

  • AARNET:dest_dir/

    • AARNET is the connection that was setup for rclone Rclone according toStep by step Rclone Setup for CloudStor file upload

    • dest_dir

      • The directory where you want to upload the files to in Cloudstor

      • In order to upload the files, you need Edit access to the Cloudstor folder is required to upload files.

A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/

...

Code Block
Folder_1
Folder_2 -> Folder_2a

All 3 folders show appear in the users’s Cloudstor Shared folder, . This is because the folder folders Folder_2a and folder Folder_2 were shared with the user individuallywith specific access.

...

We are uploading data to the shared folder Folder_1, from our local folder /tmp/images using the following rclone Rclone command:

Code Block
breakoutModewide
rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 /tmp/images/ AARNET:Shared/Folder_1/

Rclone will start the copy of the files and will provide you real time feedback about the processes running. By default, each transaction is retried 10 times in case there is an issue with the transfer.

Rclone in progress

The following is an example what will display during the Rclone process.

Code Block
breakoutModefull-width
2020-10-09 14:14:49 ERROR : lai/core1ha/20180706/DSC00165.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:49 ERROR : lai/core1ha/20180706/DSC00178_C3.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:50 ERROR : lai/core1ha/20180706/DSC00187_B2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:52 ERROR : lai/core1ha/20180706/DSC00183_E2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:54 ERROR : lai/core1ha/20180706/DSC00168_D5.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:14:54 ERROR : lai/core1ha/20180706/DSC00167_C5.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:14:55 ERROR : lai/core1ha/20180706/DSC00193_G1.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:56 ERROR : lai/core1ha/20180706/DSC00182_F2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:15:46 ERROR : lai/core1ha/20190807/DHP LAI Boyagin 07082019.docx: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden
Transferred:   	    1.941G / 4.358 GBytes, 45%, 6.073 MBytes/s, ETA 6m47s
Errors:                 9 (retrying may help)
Transferred:          231 / 637, 36%
Elapsed time:      5m27.2s
Transferring:
 *              lai/core1ha/20180222/index 2.JPG:100% /7.906M, 1008.665k/s, 0s
 *             lai/core1ha/20180706/DSC00195.JPG:100% /5.250M, 0/s, 0s
 *             lai/core1ha/20180706/DSC00205.JPG:100% /8.125M, 0/s, 0s
 *             lai/core1ha/20180706/DSC00225.JPG:100% /8.125M, 0/s, 0s
 *             lai/core1ha/20190510/DSC00450.JPG:100% /5.812M, 0/s, -
 *             lai/core1ha/20190807/DSC00571.JPG:100% /6.625M, 0/s, 0s
 *  <...and some more lines that have been taken out...>
 *           lai/core1ha/20190807/DSC00574.JPG:100% /7.469M, 0/s, 0s
 *             lai/core1ha/20190807/DSC00597.JPG:100% /6.406M, 1/s, 0s
 *             lai/core1ha/20190807/DSC00627.JPG:100% /7.500M, 434/s, 0s
 *             lai/core1ha/20190807/DSC00633.JPG:100% /7.469M, 995/s, 0s
 *          lai/core1ha/20180706/DSC00242_O5.JPG:100% /9.031M, 0/s, 0s
 *       lai/core1ha/20180222/DSC00153_R6_T9.JPG:100% /8.281M, 112.646k/s, 0s
 * lai/core1ha/20180222/D…Boyagin_22_2_2018.docx:100% /228.990k, 28.684k/s, 0s
 * lai/core1ha/20190510/D… Boyagin 10052019.docx:100% /195.322k, 48.850k

Rclone in finished

Once the cloning is finished, the final status result of the cloning will be shown.
In this case, there were some initial errors with creating directories and files.
, which were resolved after retrying the command. Rclone will retry the attempt 10 3 times to progress the copy.

In this case, the initial full clone attempt Failed (at line 12).
The retry , and the next try was successfull (at line 14): Attempt 2/3 succeeded

The Transferred file size is shown, including the checks and the Transferred file count (line 15, 16 and 17).

Code Block
breakoutModefull-width
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00569.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00547.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00552.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00577.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00546.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00573.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00571.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00554.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00570.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00568.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00557.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:44:46 ERROR : Attempt 1/3 failed with 11 errors and: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:44:52 NOTICE: webdav root 'Shared/Folder_1': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
2020-10-09 14:49:59 ERROR : Attempt 2/3 succeeded
Transferred:   	    4.477G / 4.477 GBytes, 100%, 4.153 MBytes/s, ETA 0s
Checks:               635 / 635, 100%
Transferred:          646 / 646, 100%
Elapsed time:     18m24.0s

...

Rclone synchronisation

Once the initial upload is executed, every sequental rclone sequential Rclone call will only copy the files that were added or modified after the last Rclone run. The With no changes to the directory, the outcome will be:

Code Block
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:               646 / 646, 100%
Elapsed time:         2.0s

...