TMP Space Management

Contents

Overview

Temporary (TMP) space management is used by applications and when installing software components like python modules, and by applications to store intermediate temporary files. The default directory for TMP is the "/tmp" directory on all nodes including login. On Matilda, this space is now 10GB (April 2023), but used to be relatively small (2Gb). If you find that you are encountering the error condition "No space left on device" this documentation may be helpful. The aforementioned error condition frequently occurs when "/tmp" is full or nearly so, but should be less common now that /tmp space has been increased on the nodes.

This document briefly describes how to re-designate TMP to a directory location with adequate space, should it be necessary.

Reassigning TMP

First you will need to create a directory where TMP files are to be stored. For example: 

mkdir /scratch/users/<username>/tmp

Next, in your working environment, set the appropriate environmental variable "TMPDIR". For example: 

export TMPDIR=/scratch/users/username/tmp

If using a job script, you can include this export in the job script itself. If installing python modules on login using a virtual environment, set it at the command line. Alternately, you could also add this to your ".bashrc" file so at each login TMPDIR is assigned correctly to your alternate TMP space.

MPI and TMP Warnings

CAUTION: Reassigning TMPDIR as part of a job script prior to launching an MPI job (e.g. export TMPDIR=/some/path; mpirun mympi_program ...) may generate warnings and can create communications errors and/or crashes. MPI needs to write a file in TMP for its "bookkeeping" on each node used in the job. If the reassigned TMPDIR space is located on a network mounted drive (e.g. /scratch, /projects, /home) then all nodes are attempting to track MPI overhead in the same space. At best, this may result in high I/O which can slow down your job. At worst, collisions may occur which can cause MPI process failure.

If you need a larger TMP space to store program-related files, it is better to either specify that directly in the application flags or input files (if possible), or to use redirects or other methods to place program files on a network mounted drive, while allowing MPI to keep its management files in the local TMP space (i.e. /tmp).

CategoryHPC