Page Comparison

Subject: Migration to the UQ Research Data Manager (RDM) System - Upload Guidelines

Dear TERN Site Principal Investigators,

We are reaching out to you with important information regarding the migration of our data storage from Cloudstor to the University of Queensland's Research Data Manager (UQRDM) system. The UQRDM is a robust, world-leading system that aligns with the highest standards of research data management and facilitates collaboration across research communities. This transition is necessary due to the decommissioning of Cloudstor at the end of 2023.

In the interest of maintaining data integrity, improving reproducibility of research, and promoting the reuse of our valuable data, we will be moving to the UQRDM system. Below, we provide you with guidelines for uploading your data to the new system:

Access to UQRDM

Access to your site folders on the UQRDM will be granted via email. Here are the steps you can follow to upload files:

1. Check your email for a message from the UQRDM system. You will find a link to the shared folder or file and a separate email with a password.

2. Click on the link and enter the provided password when prompted. This will take you to the folder which you have been granted access to.

1. Once you're in the shared folder, navigate to the specific folder that you wish to upload and click on the "Upload" button, represented by a plus icon.

2. Navigate to the files on your computer that you wish to upload, select them, and click "Open". The files will start uploading.

3. Please wait for the upload to complete before closing the browser window.

4. Once the upload is complete, the new files will appear in the shared folder.

Data-Specific Guidelines

LAI Data

1. Create your folders for upload within EcoImages/LAI/

2. If uploading DCP data, ensure that within the site visit folder (naming convention: YYYYMMDD),ensure that you create a subfolder corresponding to the angle used to capture your LAI photos (i.e. upwards, downwards, 57_degrees). If the data set to be uploaded has photos captured from more than one of the aforementioned angles, please sort your photos according to angle and upload them to the corresponding folders.

Camera Trap Data

1. [To be added].

Photopoint Data

1. [To be added].

Please remember that accurate, complete, and well-documented datasets are crucial for the success of our research and collaborations. If you have any questions or require further assistance, don't hesitate to contact us.

Thank you for your cooperation in this important transition.

Best regards,

Lenn Isidore
Data Curation Analyst, TERNThe University of Queensland’s Research Data Manager (UQRDM) is a platform designed to assist researchers in managing and securely storing their research data. This service facilitates the full data management lifecycle, from planning and collection to publication and preservation. If you have received an email notification that you have been granted access as a collaborator, storage on UQRDM can be accessed as follows:

1. Using the UQRDM cloud web interface (drag and drop)

• Enter this address https://cloud.rdm.uq.edu.au/  into your web browser. You will be taken to the login screen below.

...

• After logging in, navigate to the folder of interest. You can now drag files and folders directly from your local computer into UQRDM.(Detailed instructions for using the UQRDM web interface can be found here: https://guides.library.uq.edu.au/for-researchers/uq-research-data-manager/using-uqrdmcloud )

2. Using rclone

For detailed instructions on installing rclone, please go to the official rclone installlation page: https://rclone.org/install/.

Once it has been installed, rclone must be configured as follows:

2.1 Configure rclone (Mac users)

1. Open your preferred web browser.

2. Visit https://cloud.rdm.uq.edu.au/ and log in

3. Click on your initials at the top right corner.

4. Click on 'Settings'

   Image Added

   , then 'Security'.

5. Image Added

6. Scroll down until you find the "Devices & sessions" section.

7. There’s a field to name your new app password. Type ‘rclone’ into this field so that you know what the password is for.

   Image Added

8. Click on the 'Create new app password' button. This will generate a new password.

   Image Added

9. This new password will be displayed just once. Highlight it, right-click and select 'Copy' or press 'Ctrl+C' (Windows) or 'Command+C' (Mac) to copy it. You'll need to paste this in the rclone configuration later.

10. Open Terminal:

    • Press 'Command' and 'Spacebar' on your keyboard simultaneously. This opens Spotlight.

    • Type 'Terminal' and press 'Enter'. This opens the Terminal app.

11. Configure rclone:

    • In the Terminal, type rclone config and press 'Enter'.

    • When asked 'e/n/d/r/c/s/q>', type n and press 'Enter'.

    • Type a name for the new remote (like 'rdm3’) and press 'Enter'.

    • When asked for the 'Storage', type webdav and press 'Enter'.

    • When asked for the 'url', type https://cloud.rdm.uq.edu.au/remote.php/dav/files/your-username/ (replace 'your-username' with the username displayed when you created the new app password in step 8) and press 'Enter'.

    • When asked for the 'vendor', type 1 (which corresponds to Nextcloud) and press 'Enter'.

    • When asked for the 'user', type the username displayed when you created the new app password in step 8 and press 'Enter'.

    • When asked ‘y/g/n>' for the ‘pass', type y and press 'Enter'. You will then be prompted to enter the password. To paste the password you copied earlier, right-click in the Command Prompt or Terminal and select 'Paste' (or use 'Ctrl+V' for Windows or 'Command+V' for Mac). Remember, even though you might not see anything appearing in the Terminal or Command Prompt when you paste your password, it's still being entered. Don't be alarmed if it seems like nothing has been pasted. Once you’ve pasted it, press 'Enter’.

    • Confirm your password by pasting it again and pressing 'Enter'.

    • Skip the 'bearer_token' by simply pressing 'Enter'.

    • When asked 'Edit advanced config?', type n and press 'Enter'.

    • To confirm the configuration is okay, type y and press 'Enter'. Then type q and press 'Enter' to quit the config.

12. Test the Configuration:

    • Type rclone lsd rdm3: in the Terminal and press 'Enter'. It should list the RDM records that you have access to.

2.2 Configure rclone (Windows users):

1. Create a New App Password:

   • Follow the same instructions as for Mac in section 2.1, steps 1 to 9.

2. Open Command Prompt:

   • Press the 'Windows' key and 'R' on your keyboard at the same time to open the 'Run' dialog.

   • Type 'cmd' in the 'Run' dialog and click 'OK'. This opens the Command Prompt.

3. Configure rclone:

   • The steps are the same as for Mac in section 2.1, step 11, but you'll be typing into the Command Prompt instead of Terminal.

4. Test the Configuration:

   • Follow the same instructions as for Mac in section 2.1, step 12.

Please remember to replace any placeholder (like 'your-username') with your actual details. Now that rclone has been configured, it can be used to transfer files to the UQRDM.

2.3 Copying Files and Folders with rclone:

• To copy a file or an entire folder from your local system to UQRDM, you can use the copy command:

  • If using a Mac or Linux system:

    • In the Terminal, type rclone copy /path/to/local/file_or_folder rdm3:/path/on/rdm and press 'Enter'. Replace /path/to/local/file_or_folder with the actual path of the file or folder on your local system, and rdm3:/path/on/rdm with the desired location on the UQRDM. rdm3 is the name of the remote you've configured for UQRDM.

  • If using a Windows system:

    • Open the Command Prompt, type rclone copy "C:\path\to\local\file_or_folder" rdm3:/path/on/rdm and press 'Enter'. Replace C:\path\to\local\file_or_folder with the actual path of the file or folder on your local system, and rdm3:/path/on/rdm with the desired location on the UQRDM.

Please note that it is a good practice to avoid spaces in your filenames and folder names when using rclone. If your filename or folder name has spaces, make sure to enclose the entire path in quotation marks. For example: rclone copy "/path/to/your file.txt" rdm3:/path/on/rdm".

2.4 Syncing folders with rclone:

• To sync a local folder to a folder on UQ RDM, use the sync command. This command makes the destination folder match the source folder, altering the contents of the destination folder if necessary. rclone sync /path/to/local/folder rdm3:/path/on/rdm and press 'Enter'.

  Again, replace /path/to/local/folder with the path of the local folder, and rdm3:/path/on/rdm with the destination folder on the UQRDM.

• If you would like real-time statistics on how much data has been transferred, and at what speed, you can use this instead:

  • For mac or Linux users: rclone sync -P /path/to/local/folder rdm3:/path/on/rdm and press 'Enter'.

  • For windows users: rclone sync -P C:\path\to\local\folder rdm3:/path/on/rdm .

2.5 Moving files with rclone:

• If you wish to move (i.e., copy and then delete the source), you can use the move command. This is useful if you want to free up space on your local system after ensuring the files are safely stored on UQ RDM. rclone move /path/to/local/file rdm3:/path/on/rdm

  Remember, this will delete the file from the local system after it has been copied to UQ RDM.

Remember to replace the placeholders in the commands with the actual paths. You can run these commands from Terminal on Mac, Command Prompt on Windows, or the Terminal on Linux.

One more thing: rclone commands are case-sensitive, so always ensure you're using lowercase unless otherwise needed. For instance, rclone Copy will not work; you must use rclone copy.

2.6 Rclone checks on data copy:

To ensure correct copy of the files with rclone, options can be added to the copy command to assist you.

The following is an example of a command you can run in the terminal / command line to upload data to UQRDM:

rclone copy --transfers 8 --progress --checksum --checkers 8 --timeout 0 source-dir/ rdm3:dest_dir/

The following options are provided to this command to upload the files.

• --transfers 8

  • 8 concurrent transfers are performed.

• --progress

  • Show progress during transfer

• --checksum

  • skip files if they already exist, based on checksum and size.
    By default the files are skipped based on modification time and size.

• --checkers 8

  • This option increases the number of tasks that are checking to see if the file already exists on the remote. This value should be greater than transfers to make sure that transfers is kept at capacity.

• --timeout 0

  • This is the allowed Input Output idle timeout. By default this is 5 minutes.

• source-dir/

  • The local directory from which you want to push the files to UQRDM,
    ie C:\tmp\images\20200103\ or external drive G:\images\20200103\ (Windows)
    or /tmp/images/20200103 (Mac / Linux)

• rdm3:dest_dir/

  • rdm3 is the connection name that was setup for Rclone according to/wiki/spaces/TERNSup/pages/1344274476

  • dest_dir

    • The directory where you want to upload the files to in UQRDM

    • Edit access to the UQRDM folder is required to upload files.

A complete list of additional options that can be passed to Rclone can be found at https://rclone.org/flags/