Moving Files

Overview

This documentation describes various methods of moving files, data, and folders from your workstation (or other server) to the Matilda Cluster, and also moving files from the cluster to another off-cluster location. Methods include the use of the command line, graphical applications, and third-party intermediary services.

Cluster Storage Locations

User data is stored in one of four (4) possible locations:

1. /home
2. /scratch/users
3. /projects
4. /scratch/projects

The examples provided below will utilize a mixture of these locations, but keep in-mind that source/destination locations on the cluster are interchangeable across all examples.

Command Line

Command line methods for moving files are broken-down by client type (Linux, Mac, Windows), depending upon the operating system of your workstation or remote server.

Linux and Mac

Since Matilda uses the Linux operating system (O/S) moving data between the cluster and other Linux or Mac machines is relatively straight-forward. In the case of Macs, OSX is a derivative of the Unix O/S "FreeBSD", so many of the commands used for file transfers are either the same, or very similar to, Linux.

The two most common commands used for file transfer are "rsync" and "scp". These commands can be used to upload or download files/directories to, or from, Matilda. To begin, if you are on a Linux or Mac machine, open the terminal. On Linux devices "terminal" can usually be found under your system menu. On Mac machines "terminal" is generally located under "Launchpad->Other".

Using rsync

To upload a file or directory to Matilda from your local machine, open the terminal and navigate to the directory that contains the directory or file(s) you wish to upload. Then use something like the following (you will be prompted for your password):

rsync -avp myDir [email protected]:/scratch/users/username/

Explanation:

• the flags "-avP" stand for "archive mode", "verbose", and "progress", respectively.
• archive mode (-a) is equivalent to "-rlptgoD". These flags generally work well for more file transfers.
• the "-P" flag shows transfer progress.
• "myDir" is the directory (or file) you wish to transfer.
• replace "username" with your actual NetID.
• In this example we are moving "myDir" to our personal scratch space.

(See rsync man for more options)

Note: If you perform an "rsync" to Matilda when you are NOT connected to the VPN, you will receive a notification on your mobile device from DUO. Please respond to this notification to complete the transfer. If you are connected already to the OU VPN, then no DUO prompt will be issued.

To download from Matilda to your local machine, you can use a very similar command by changing around the expression slightly. From your local machine:

rsync -avP [email protected]:/projects/some-lab/experiments .

Explanation:

• note in this case we start with the connection string to Matilda.
• we need to specify the exact location of the folder or file we wish to download (/projects/some-lab/experiments).
• please note the dot "." at the very end that indicates place the downloaded file/folder HERE (current directory).
• you could also replace the dot "." with another folder location on your local machine

Using scp

Secure copy or "scp" can also be used through the terminal to transfer files and directories. Like "rsync", executing "scp" outside of the OU VPN will trigger a DUO authentication request to your mobile device. An example is provided below:

scp myFile [email protected]:/home/u/username/
scp -r myDir [email protected]:~

Explanation:

• the first example transfers a file "myFile" to the user's home directory.
• the second example transfers a directory "myDir" to the user's home directory.
• note the use of the "-r" flag in the second example, which is for a "recursive" copy of the directory and its' contents.
• note the two different ways the home directory can be represented.
• to get the exact path to your home directory, on Matilda, enter "cd" followed by "pwd"
• the tilde "~" is a shortcut to your home directory

Once again, to download from Matilda to your local machine, you could use something like the following:

scp [email protected]:/scratch/projects/some-lab/dataFile .
scp -r [email protected]:/scratch/projects/some-lab/resultsDir newResults

Explanation:

• in the first example we download the file "dataFile" to the current local directory.
• in the second example we recursively download the directory "resultsDir" and rename it locally as "newResults"
• note that directory copies with "scp" require the use of the "-r" flag (in rsync recursion is built-in to the -a flag).

For more options, please see the scp man page.

rsync vs. scp

Some of the main differences between "rsync" and "scp" include:

• scp does a plain linear copy, whereas rsync "synchronizes".
• if you use scp to copy a file that already exists at the destination, it will overwrite the existing file.
• if you use rsync to copy a file that already exists at the destination, it will not overwrite unless the source version is different.
• rsync with the "-P" option is particularly useful when large transfers are disrupted. It will pick up from where it left off if restarted.
• rsync tends to be somewhat faster than scp.
• scp is particularly good for quick small or single file transfers.

Windows

MobaXterm and PuTTY

If you are a Windows user and you are using a terminal client like MobaXterm or PuTTY, you can use the exact same techniques in the command line shown above for Linux and Mac machines.

Powershell

To launch Powershell, press the "Windows Key" and enter "powershell" to locate the application. Start Powershell and you will be inside a terminal which has SSH and "scp" capabilities. You may then proceed to use "scp" commands as show above. However, "rsync" is not available in Powershell. The next section covers a method for obtaining and using "rsync" from a Windows workstation.

WSL

If you haven't already installed the "Windows Subsystem for Linux" (WSL) you may do so by following the instructions provided by Microsoft. Once installation is complete, you can access WSL by pressing the "Windows Key" and typing "wsl". Click on WSL to start the terminal. By default, you should be in a Ubuntu Linux environment. In most cases, "rsync" should already be present following installation. If it is not, you can install it using:

sudo apt install rsync openssh-client

When prompted, enter your password.

Once "rsync" is present in WSL, you can use the same commands shown in the Linux examples above to transfer files and folders. You can also use "scp" from WSL, if desired. If you are unfamiliar with WSL, you can get to a familiar Windows path in most cases using:

cd /mnt/c

This should put you in the C:\ directory of your workstation. From there you can use Linux commands to navigate to the directory where your files or folders are located. (Note that WSL creates an alternate directory tree from the standard Windows tree).

Windows Terminal

Graphical Clients

Presented below is a small selection of Graphical User Interface (GUI) based applications which can be installed and used on a variety of different platforms. These are only common examples. Many others are available.

While you are encouraged to become as familiar with the command line environment as possible (since Matilda is a command environment), using a GUI application can ease the learning curve for new users.

MobaXterm and SFTP

If you are using MobaXterm on a Windows machine, you can also use the built-in SFTP client for visual, GUI-based uploads and downloads as show in the Matilda MobaXterm documentation.

FileZilla

FileZilla is a common SFTP GUI client that can be used for transferring files between machines. It is available for Windows, Mac, and Linux. The key to using FileZilla is to configure the host properties correctly for use with Matilda.

Once downloaded and installed, start FileZilla then click "File->Site Manager->New Site". Choose a name for the site, and fill-in the boxes as illustrated below:

Fill in your correct username and password, also make sure to select "SFTP" in the "Protocol" dropdown. Now hit the "Connect" button and you should see something like the following:

NOTE: If you are connected to the OU VPN your connection will be seamless. If not, you will receive a DUO prompt which must be accepted in order to connect.

Your local drive is in the left pane, and the remote Matilda pane is on the right. You can change directories by either clicking in the panes to navigate, or enter the desirec path directly in the "site" box. For example, to change to "/scratch/users/username" enter this path in the "Remote site" box and hit enter.

Now simply drag and drop to move files.

WinSCP

WinSCP is a popular Windows-based GUI application that can be used for transferring files use SFTP or SCP protocols. Once downloaded and installed, start the program:

Fill-in the fields as shown above (substituting your actual username). For convenience, you can save this as site profile (in this case we named it Matilda), for future use. Then press the "Connect" button:

When prompted, select "Yes" to save the server key. You will now be prompted for your password. Enter it and click "OK":

If you are not on-campus or connecting through the VPN off-campus, you should then receive a DUO prompt on your mobile device. Confirm it, and you should now see the following split pane window:

You may now drag and drop files between your local machine (left pane) and Matilda (right pane).

---

CategoryHPC