Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/

Example Initial Upload

For this example, the Cloudstor Access level given was edit for Folder_1 and read for Folder_2. Additionally, edit access was given to Folder_2a, which is a subfolder of Folder_2 as per below

Code Block
Folder_1
Folder_2 -> Folder_2a

All 3 folders show in your the users’s Cloudstor Shared folder, because you were added as a user to the folder Folder_2a as well as to and folder Folder_2 were shared with the user individually.

...

We are uploading data to the shared folder Folder_1 using the following rclone command:

Code Block
breakoutModewide
rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 /tmp/images/ AARNET:Shared/Folder_1/

Rclone will start the copy of the files and will provide you real time feedback about the processes running. By default, each transaction is retried 10 times in case there is an issue with the transfer.

The following is an example what will display during the Rclone process.

Code Block

...

What you will see in the command window, during and finally

Retries for incorrectly transferred files by Rclone

Where is my local data

Example with a real time directory

...

breakoutModefull-width
2020-10-09 14:14:49 ERROR : lai/core1ha/20180706/DSC00165.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:49 ERROR : lai/core1ha/20180706/DSC00178_C3.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:50 ERROR : lai/core1ha/20180706/DSC00187_B2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:52 ERROR : lai/core1ha/20180706/DSC00183_E2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:54 ERROR : lai/core1ha/20180706/DSC00168_D5.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:14:54 ERROR : lai/core1ha/20180706/DSC00167_C5.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:14:55 ERROR : lai/core1ha/20180706/DSC00193_G1.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:14:56 ERROR : lai/core1ha/20180706/DSC00182_F2.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:15:46 ERROR : lai/core1ha/20190807/DHP LAI Boyagin 07082019.docx: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden
Transferred:   	    1.941G / 4.358 GBytes, 45%, 6.073 MBytes/s, ETA 6m47s
Errors:                 9 (retrying may help)
Transferred:          231 / 637, 36%
Elapsed time:      5m27.2s
Transferring:
 *              lai/core1ha/20180222/index 2.JPG:100% /7.906M, 1008.665k/s, 0s
 *             lai/core1ha/20180706/DSC00195.JPG:100% /5.250M, 0/s, 0s
 *             lai/core1ha/20180706/DSC00205.JPG:100% /8.125M, 0/s, 0s
 *             lai/core1ha/20180706/DSC00225.JPG:100% /8.125M, 0/s, 0s
 *             lai/core1ha/20190510/DSC00450.JPG:100% /5.812M, 0/s, -
 *             lai/core1ha/20190807/DSC00571.JPG:100% /6.625M, 0/s, 0s
 *             lai/core1ha/20190807/DSC00574.JPG:100% /7.469M, 0/s, 0s
 *             lai/core1ha/20190807/DSC00597.JPG:100% /6.406M, 1/s, 0s
 *             lai/core1ha/20190807/DSC00627.JPG:100% /7.500M, 434/s, 0s
 *             lai/core1ha/20190807/DSC00633.JPG:100% /7.469M, 995/s, 0s
 *          lai/core1ha/20180706/DSC00242_O5.JPG:100% /9.031M, 0/s, 0s
 *       lai/core1ha/20180222/DSC00153_R6_T9.JPG:100% /8.281M, 112.646k/s, 0s
 * lai/core1ha/20180222/D…Boyagin_22_2_2018.docx:100% /228.990k, 28.684k/s, 0s
 * lai/core1ha/20190510/D… Boyagin 10052019.docx:100% /195.322k, 48.850k

Once the cloning is finished, the final status will be shown.
In this case, there were some initial errors with creating directories and files.
Rclone will retry the attempt 10 times to progress the copy.

In this case, the initial full clone attempt Failed (at line 12).
The retry was successfull (at line 14): Attempt 2/3 succeeded

The Transferred file size is shown, including the checks and the Transferred file count.

Code Block
breakoutModefull-width
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00569.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00547.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00552.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00577.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00546.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00573.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:33 ERROR : lai/core1ha/20190807/DSC00571.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00554.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00570.JPG: Failed to copy: Update mkParentDir failed: Parent node does not exist: Sabre\DAV\Exception\Conflict: 409 Conflict
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00568.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:31:34 ERROR : lai/core1ha/20190807/DSC00557.JPG: Failed to copy: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:44:46 ERROR : Attempt 1/3 failed with 11 errors and: Update mkParentDir failed: Could not create directory /Shared/Folder_1/lai/core1ha/20190807: Sabre\DAV\Exception\Forbidden: 403 Forbidden
2020-10-09 14:44:52 NOTICE: webdav root 'Shared/Folder_1': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
2020-10-09 14:49:59 ERROR : Attempt 2/3 succeeded
Transferred:   	    4.477G / 4.477 GBytes, 100%, 4.153 MBytes/s, ETA 0s
Checks:               635 / 635, 100%
Transferred:          646 / 646, 100%
Elapsed time:     18m24.0s

Synchronisation

Once the initial upload is executed, every sequental rclone call will only copy the files added after the last Rclone run. The outcome will be:

Code Block
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:               646 / 646, 100%
Elapsed time:         2.0s