Scratch File Management

Contents

Overview

In order to keep the Matilda scratch space clean and maintain usable capacity, files not accessed in the last 45 days will be automatically deleted.

This page describes methods users can use to detect files at, or approaching 45 days of access time before auto-deletion occurs.

Locating files scheduled for deletion

The easiest way to find files in your scratch space that are over a certain access time is to use the built-in "lfs find" command. For example: 

cd /scratch/users/$USER
lfs find . -atime +45 -type file

or alternately: 

lfs find /scratch/users/$USER -atime +45 -type file

Of course the commands above will identify files just about to be deleted, so you may wish to alter "+45" to some other value (e.g. "+40") to provide some advanced warning of impending file deletions.

Using a cronjob

The service "cron" provides a way for users to run a job script or command at a specified time at regular intervals (e.g. every day). Users can create their own cronjobs to run the command shown above, and send them an email list of files to be deleted. To create a "crontab" (a table of your cronjobs) enter the following command on Matilda: 

crontab -e

This will open the "vi" text editor to an empty file (the first time you run it). The header below shows the column specifiers for the cronjob that should be added to your new crontab: 

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
[email protected]

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  *  command to be executed
0 1 * * * lfs find /scratch/users/<username> -atime +44 -type file

In the example above, we will run our "lfs find" command at 1:00am every day and look for files that were last accessed more than 44 days ago (this gives us a 1 day warning). The "MAILTO" command will send us an email list of any files that are found that meet this criteria. Substitute your username in the appropriate places above.

Backing up The Crontab

Since the login node periodically goes through maintenance and many directories are reimaged, it is important to backup your crontab so you can restore it after maintenance windows. To do this: 

crontab -l >my-crontab

Make sure "my-crontab" is saved somewhere in your home directory. At a later time, you can restore this crontab using: 

crontab my-crontab

CategoryHPC