Globus Data Transfers

Overview

Globus lets you use a web browser or command line interface to submit transfer and synchronization requests of large data sets between the Matilda HPC and other institutions, or your local workstation.

Data are transferred directly between the source and destination systems while Globus tunes performance parameters, maintains security, monitors progress, and validates correctness. You can check the transfer status at any time via the Globus activity page and will receive email when the transfer completes.

Globus provides secure and fast file/data transfer between endpoints, and is designed to facilitate research collaboration. Additionally, Globus is geared primarily towards the transfer of large data sets rather than large or small numbers of small files.

Matilda Data Collections

Accessible directories within Globus are defined as "Data Collections". Matilda is configured with the following data collections:

  • Matilda HPC - users home directory
  • Matilda Scratch - the scratch space, accessible at either the user or project level
  • Matilda Projects - the projects directory space

Getting Started

To get started using Globus on Matilda, first navigate to the Globus website, and click the "Login" button.

Next select "Oakland University" from the dropdown box, and click "Continue":

GlobusLogin.png

After clicking "Continue" you will be redirected to the Oakland University Login page. Enter your OU NetID and password and click "Sign In":

OULogin.png

After signing-in, you will be redirected to the Globus File Manager:

FileManager.png

At the top you will see a box labeled "Collection". To search for the available Matilda cluster collections, enter "Matilda" in the Collections box and you should see something like the following:

CollectionSearch.png

Now select the collection of interest. In this example, we will choose the "Matilda HPC" collection which corresponds to the user's home directory:

DirectoryList.png

(NOTE: The first time you access a new collection, you may be asked to confirm your permission to access the collection. Please grant all suggested permissions.)

This should now show you a listing of the files and folders in your home directory.

The same procedure can be used to access the other on-cluster collections: "Matilda Scratch" and "Matilda Projects".

Note that accessing "Matilda Scratch" will take you to the root "/scratch" directory. From there, click on either the "users" or "projects" folders, and then on the corresponding user or project space. Similarly, for "Matilda Projects", you will be directed to the root "/projects" directory. From there, click on the desired project space.

File Transfers

As previously mentioned, file transfers are possible between Matilda and other institutions with publicly accessible Globus endpoints. It is also possible to transfer files between your local workstation and Matilda using Globus. The following example will illustrate a transfer between Matilda and MSU's HPCC endpoints (Please note this particular example will only work if you have an MSU HPCC user account).

With your home directory pulled up by accessing the "Matilda HPC" Collection, click on the menu item "Transfer or Sync to.." in Globus. You should now see your home directory listing in the left pane, and a "Search" box in the right pane:

TransferSearch.png

Now type "MSU hpcc" in the search box in the right pane, and find iCER's MSU HPCC Collection (Note: you will be prompted to authenticate collection access using your assigned MSU NetID and password). Once selected, you should now see your Matilda home directory listing on the left, and your MSU HPCC home directory on the right:

TransferFiles.png

To transfer files, click on the directory listing of the source, select the file(s) or folder(s) you wish to transfer, and simply drag them to the destination pane.

From this directory view you can delete files or folders, create folders, upload files to the data collection from your local machine, or download files from the collection to your local machine.

PLEASE NOTE: In most cases you will be provided with access to another institution's public collection (or a private share), and would search for, or directly enter the collection name to initiate transfers to or from Matilda. At this time, OU does NOT have a subscription to Globus. This means that user generated private shares from Matilda to external institutions are not possible at this time (since private shares require the subscription service). Additionally, external users cannot access Matilda shares without an OU NetID and a Matilda HPC account.

Globus Connect Personal

Globus Connect Personal is an application that can be downloaded to your personal workstation. Globus Connect Personal essentially creates a "Collection" that corresponds to your local desktop or laptop and facilitates transfers between your local workstation and another endpoint. The Globus Connect Personal website provides installation instructions for Windows, Mac, and Linux clients.

When Globus Connect Personal is first started it will ask for your login information and a name for your workstation "Collection". This will create a personal endpoint that will be active whenever Connect Personal is running. File transfers can be initiated by clicking on the Globus Connect Personal icon in the system tray, and selecting "Web Transfers". This will open a web page to Globus with your local Collection shown in the left pane. Simply search for the Collection desired in the right hand search box to select files/folders and initiate transfers.


CategoryHPC