/scratch is a shared resources that is critical for many peoples pipelines. If it gets filled on a node then any job that is assigned to that node will die as soon as it tries to write to /scratch. This can create extremely difficult tasks to track down bugs.
SO PLEASE DO NOT FILL UP /scratch on the cluster or master NODES!
If you are writing to /scratch you should
- Monitor your usage of it
- Cleanup after yourself
Ideally your jobs should do this automatically, however if your job exits with an error you need to manually cleanup after yourself. Everyone should create a subdirectory in /scratch with your username and write there instead of into the root. ie use
/scratch/$USER
instead of
/scratch
When the /scratch directory is full all users will experience crashing jobs so do NOT fill up /scratch.