How to upload and access files using TERN SFTPGo portal

TERN provides a secure File Transfer Protocol (FTP) portal, SFTPGo, to support uploading, viewing, and downloading large data sets into the TERN Data repository. The TERN SFTPGo portal can be found at https://sftpgo.tern.org.au/.

To be able to login to TERN SFTPGo portal, there are 2 pre-requisites:

• users require to have an account in TERN, which can be created by logging into https://account.tern.org.au/ (the login button is on the right-hand corner of the website). Once users get a confirmation email, they can proceed with a request to access the TERN sftpgo service.

• an sftpgo user account be setup, which can be requested via email (see below for details)

Request Access to SFTPGo

The https://sftpgo.tern.org.au is a restricted service to upload, view and download data from TERN storage. Users should request access to upload data by sending an email to esupport@tern.org.au with the following information:

• Subject: sftpGo access request

• Content:

  • The email addresses of the users that need to be granted this access.

  • The level of access required: read or upload

  • The data type you are interested in, for example:

    • Acoustic, EcoImages, Bat Acoustic, drone,

    • event interested - upload, download, view

The TERN Data Team will review and process the request, inform users of the outcome of the request, and, when granted access, provide users with their SFTPGo account details (i.e. username).

1. SFTPGo Web Interface Uploads

The sftpGo web interface is an interactive data upload platform: https://sftpgo.tern.org.au. However, there are some limitations on sftpGo web-interface:

• a maximum of 30 files can be uploaded at a time

• folder upload will result in a flatten folder structure

If this does not meet your need, you can use a sftp client to upload data, such as sftp client, FileZilla, or Rclone (see the respective section below for details).

Login

Login by selecting “Sign in with OpenID”. Users should select the login preference they have used to create their TERN account from the following options “Australian Access Federation”, “CILogon” or Google.

Open

login button which will take to TERN Single Sign-On (SSO)

File interaction

Once users log in to SFTPGo, they will see all the folders they have access. If user wants to upload data, use the relevant folder to upload. The folder structure is different based on the type of data used. users may have write access to multiple folders but, only relevant folders should be used for data upload.

Open

Home folder with the list of folder users have access in SFTPGo

From the My Files menu, you can select the folder you want to use to view or upload the data (depending on the access to the location).

If users have given upload (write) access, they can create a new folder using the “New Folder” buttons to add data. You can upload files by “Drag and Drop” file or using “Upload Files”.

users can perform multiple actions on files by selecting the relevant files and clicking on the Actions button.

Setup your SFTP access in sftpGo

To allow SFTP access (SSH File Transfer Protocol), you must setup your SSH Public key in your sftpGo setting in the SFTPGo web portal. You can set up an SSH public/private key pair to do this.
An example of a MAC is/Users/users_name/.ssh/id_rsa and /Users/users_name/.ssh/id_rsa.pub.
If you have no SSH key setup yet, set up your SSH Keys as described in Understanding SSH - Key-based Authentication.

Once you have created your SSH key pair, find the Public key associated with your Private key. In our example, this would be /Users/users_name/.ssh/id_rsa.pub.

An example of a Public key content:

Copy the content of your Public key, in our example /Users/jblogs/.ssh/id_rsa.pub, and add this to “My Profile” in the “Public Keys” field and click Submit.

2. Using SFTP clients to Upload data

If you do not already have one, download and install an SFTP client. Some free desktop clients are:

• Command line sftp

  • usually comes with Linux

• FileZilla (Windows, Linux, Mac)

  • Website: https://filezilla-project.org

• Cyberduck (Windows and Mac)

  • Website: https://cyberduck.io

  • Notes: Mac users can download the software for free from the website (zipped) instead of paid version from the Mac App Store

• WinSCP (Windows)

  • Website: https://winscp.net

General SFTP settings for all clients

Use the following settings:

• Host (or hostname): sftpgo.tern.org.au

• Port: 2022

• Protocol: SFTP (SSH)

• Username: your username in sftpGo

  • This can be found in the top right corner of the sftpGo Web interface - https://sftpgo.tern.org.au/

• SSH Private Key: the entire location of your SSH private Key file
  (as per our MAC example, this would be /Users/users_name/.ssh/id_rsa)

If using FileZilla, you can follow more detailed instructions below.

2.1 Using command line sftp

• Login into the sftpGo server via the command line using:

• A one-liner to upload a folder to sftpgo (without login):

2.2 Using FileZilla

1. Open FileZilla and select from the Main Menu: `File` > `Site Manager…`

2. Click the [New Site] button and give it a name (e.g. TERN Supersites Acoustic data)

3. In the General tab, fill out the following details:

   • Host (or hostname): sftpgo.tern.org.au

   • Port: 2022

   • Protocol: SFTP (SSH)

   • Logon Type: Key file

   • Add the full “file path” of your SSH Private Key file

You do not need to worry about any of the other tabs.

1. Select your new site and click ‘Connect’ to connect to the upload site.

2. On the left side of the screen (called Local site), you will see directories in your local computer, i.e., the source. Navigate to the folder where you stored the data. When you click on a folder in the top left panel, the content of that folder is listed in the bottom left panel. 

3. On the right side of the screen (called Remote site), you will see directories in the remote connection that are specific to your SuperSite. These are your target directories. For example, if your site is KarawathaForestPark and your data is Acoustic data, you will see two directories, one for each of the sensors at Karawatha (KarawathaForestPark01 and KarawathaForestPark02). 

4. If you have Upload access, you can upload new files from your computer (Local site) to the your folders in sftpgo.tern.org.au (Remote site) by dragging and dropping the relevant files on the left-side panel to the correct folder on the right-side panel. 

2.3 Using Rclone

For detailed instructions on how to install an Rclone, please visit the official Rclone installation page: https://rclone.org/install/.

Once it has been installed, Rclone must be configured as follows:

2.3.1 Configure Rclone (Mac users)

1. Open Terminal:

   • Press 'Command' and 'Spacebar' on your keyboard simultaneously. This opens Spotlight.

   • Type 'Terminal' and press 'Enter'. This opens the Terminal app.

2. Configure Rclone:

   • In the Terminal, type rclone config and press 'Enter'.

   • When asked e/n/d/r/c/s/q>, type n and press 'Enter'.

   • Type a name for the new remote (like sftpGo) and press 'Enter'.

   • When asked for the Storage, type sftp and press 'Enter'.

   • When asked for the host, type sftpgo.tern.org.au and press 'Enter'.

   • When asked for the user, type the username in the top right corner of your sftpGo portal.

   • When asked for the port, type 2022 and press 'Enter'.

   • When asked SSH password, leave blank to use ssh-agent. , type n and press ‘Enter’

   • When asked Raw PEM-encoded private key - key_pem, press 'Enter'.

   • When asked for the key_file, enter the location of your SSH key file, for example /Users/jblogs/.ssh/id_rsa and press 'Enter'. (as setup in Setup your SFTP access in sftpGo )

   • When asked for The passphrase to decrypt the PEM-encoded private key file., type n and press ‘Enter’

   • When asked for the pubkey_file, press 'Enter'.

   • When asked for the key_use_agent, press 'Enter'.

   • When asked for the use_insecure_cipher, type 1 and press 'Enter'.

   • When asked for the disable_hashcheck, press 'Enter'.

   • When asked 'Edit advanced config?', type n and press 'Enter'.

   • To confirm the configuration is okay, type y and press 'Enter'. Then type q and press 'Enter' to quit the config.

3. Test the Configuration:

   • Type rclone lsd sftpGo: in the Terminal and press 'Enter'. It should list the directories you have been assigned access to.

2.3.2 Configure Rclone (Windows users):

1. Create a New App Password:

   • Follow the instructions for Mac in section 2.1, steps 1 to 9.

2. Open Command Prompt:

   • Press the Windows key and R on your keyboard simultaneously to open the Run dialog.

   • Type cmd in the Run dialog and click OK. This opens the Command Prompt.

3. Configure Rclone:

   • The steps are the same as Mac's in section 2.1, step 11, but you'll be typing into the Command Prompt instead of Terminal.

4. Test the Configuration:

   • Follow the same instructions as for Mac in section 2.1, step 12.

Now that Rclone has been configured, it can list and/or transfer files to the sftpGo locations.