Welcome to TERN Knowledge Base

How to Upload Files into UQRDM using the cloud web interface and Rclone

Owned by Lenn Isidore (Unlicensed)
Last updated: 17 Aug 2023 by TERN Data

The University of Queensland’s Research Data Manager (UQRDM) is a platform designed to assist researchers in managing and securely storing their research data. This service facilitates the full data management lifecycle, from planning and collection to publication and preservation. If you have received an email notification that you have been granted access as a collaborator, storage on UQRDM can be accessed as follows:

1. Using the UQRDM cloud web interface (drag and drop)

 

2. Using rclone

For detailed instructions on installing rclone, please go to the official rclone installlation page: https://rclone.org/install/.

Once it has been installed, rclone must be configured as follows:

2.1 Configure rclone (Mac users)

  1. Open your preferred web browser.

  2. Visit https://cloud.rdm.uq.edu.au/ and log in

  3. Click on your initials at the top right corner.

  4. Click on 'Settings'

    , then 'Security'.

  6. Scroll down until you find the "Devices & sessions" section.

  7. There’s a field to name your new app password. Type ‘rclone’ into this field so that you know what the password is for.

  8. Click on the 'Create new app password' button. This will generate a new password.

  9. This new password will be displayed just once. Highlight it, right-click and select 'Copy' or press 'Ctrl+C' (Windows) or 'Command+C' (Mac) to copy it. You'll need to paste this in the rclone configuration later.

  10. Open Terminal:

    • Press 'Command' and 'Spacebar' on your keyboard simultaneously. This opens Spotlight.

    • Type 'Terminal' and press 'Enter'. This opens the Terminal app.

  11. Configure rclone:

    • In the Terminal, type rclone config and press 'Enter'.

    • When asked 'e/n/d/r/c/s/q>', type n and press 'Enter'.

    • Type a name for the new remote (like 'rdm3’) and press 'Enter'.

    • When asked for the 'Storage', type webdav and press 'Enter'.

    • When asked for the 'url', type https://cloud.rdm.uq.edu.au/remote.php/dav/files/your-username/ (replace 'your-username' with the username displayed when you created the new app password in step 8) and press 'Enter'.

    • When asked for the 'vendor', type 1 (which corresponds to Nextcloud) and press 'Enter'.

    • When asked for the 'user', type the username displayed when you created the new app password in step 8 and press 'Enter'.

    • When asked ‘y/g/n>' for the ‘pass', type y and press 'Enter'. You will then be prompted to enter the password. To paste the password you copied earlier, right-click in the Command Prompt or Terminal and select 'Paste' (or use 'Ctrl+V' for Windows or 'Command+V' for Mac). Remember, even though you might not see anything appearing in the Terminal or Command Prompt when you paste your password, it's still being entered. Don't be alarmed if it seems like nothing has been pasted. Once you’ve pasted it, press 'Enter’.

    • Confirm your password by pasting it again and pressing 'Enter'.

    • Skip the 'bearer_token' by simply pressing 'Enter'.

    • When asked 'Edit advanced config?', type n and press 'Enter'.

    • To confirm the configuration is okay, type y and press 'Enter'. Then type q and press 'Enter' to quit the config.

  12. Test the Configuration:

    • Type rclone lsd rdm3: in the Terminal and press 'Enter'. It should list the RDM records that you have access to.

2.2 Configure rclone (Windows users):

  1. Create a New App Password:

    • Follow the same instructions as for Mac in section 2.1, steps 1 to 9.

  2. Open Command Prompt:

    • Press the 'Windows' key and 'R' on your keyboard at the same time to open the 'Run' dialog.

    • Type 'cmd' in the 'Run' dialog and click 'OK'. This opens the Command Prompt.

  3. Configure rclone:

    • The steps are the same as for Mac in section 2.1, step 11, but you'll be typing into the Command Prompt instead of Terminal.

  4. Test the Configuration:

    • Follow the same instructions as for Mac in section 2.1, step 12.

Please remember to replace any placeholder (like 'your-username') with your actual details. Now that rclone has been configured, it can be used to transfer files to the UQRDM.

2.3 Copying Files and Folders with rclone:

  • To copy a file or an entire folder from your local system to UQRDM, you can use the copy command:

    • If using a Mac or Linux system:

      • In the Terminal, type rclone copy /path/to/local/file_or_folder rdm3:/path/on/rdm and press 'Enter'. Replace /path/to/local/file_or_folder with the actual path of the file or folder on your local system, and rdm3:/path/on/rdm with the desired location on the UQRDM. rdm3 is the name of the remote you've configured for UQRDM.

    • If using a Windows system:

      • Open the Command Prompt, type rclone copy "C:\path\to\local\file_or_folder" rdm3:/path/on/rdm and press 'Enter'. Replace C:\path\to\local\file_or_folder with the actual path of the file or folder on your local system, and rdm3:/path/on/rdm with the desired location on the UQRDM.

Please note that it is a good practice to avoid spaces in your filenames and folder names when using rclone. If your filename or folder name has spaces, make sure to enclose the entire path in quotation marks. For example: rclone copy "/path/to/your file.txt" rdm3:/path/on/rdm".

2.4 Syncing folders with rclone:

  • To sync a local folder to a folder on UQ RDM, use the sync command. This command makes the destination folder match the source folder, altering the contents of the destination folder if necessary. rclone sync /path/to/local/folder rdm3:/path/on/rdm and press 'Enter'.

    Again, replace /path/to/local/folder with the path of the local folder, and rdm3:/path/on/rdm with the destination folder on the UQRDM.

  • If you would like real-time statistics on how much data has been transferred, and at what speed, you can use this instead:

    • For mac or Linux users: rclone sync -P /path/to/local/folder rdm3:/path/on/rdm and press 'Enter'.

    • For windows users: rclone sync -P C:\path\to\local\folder rdm3:/path/on/rdm .

2.5 Moving files with rclone:

  • If you wish to move (i.e., copy and then delete the source), you can use the move command. This is useful if you want to free up space on your local system after ensuring the files are safely stored on UQ RDM. rclone move /path/to/local/file rdm3:/path/on/rdm

    Remember, this will delete the file from the local system after it has been copied to UQ RDM.

Remember to replace the placeholders in the commands with the actual paths. You can run these commands from Terminal on Mac, Command Prompt on Windows, or the Terminal on Linux.

One more thing: rclone commands are case-sensitive, so always ensure you're using lowercase unless otherwise needed. For instance, rclone Copy will not work; you must use rclone copy.

 

2.6 Rclone checks on data copy:

To ensure correct copy of the files with rclone, options can be added to the copy command to assist you.

The following is an example of a command you can run in the terminal / command line to upload data to UQRDM:

rclone copy --transfers 36 --progress --checksum --checkers 48 --timeout 0 source-dir/ rdm3:dest_dir/

The following options are provided to this command to upload the files.

  • --transfers 36

    • 36 concurrent transfers are performed.

  • --progress

    • Show progress during transfer

  • --checksum

    • skip files if they already exist, based on checksum and size.
      By default the files are skipped based on modification time and size.

  • --checkers 48

    • This option increases the number of tasks that are checking to see if the file already exists on the remote. This value should be greater than transfers to make sure that transfers is kept at capacity.

  • --timeout 0

    • This is the allowed Input Output idle timeout. By default this is 5 minutes.

  • source-dir/

    • The local directory from which you want to push the files to UQRDM,
      ie C:\tmp\images\20200103\ or external drive G:\images\20200103\ (Windows)
      or /tmp/images/20200103 (Mac / Linux)

  • rdm3:dest_dir/

A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/

 

 

 

 

 

 

 

Provide your feedback about the experience with Knowledge base