We are aware that slurm commands such as squeue and sacct are running slowly or even timing-out this morning. We are investigating the issue. Slurm documentation seems to indicate that this may be due to a large number of concurrent jobs in addition to frequent queries to the slurm database. 

Running jobs seem to be unaffected by this issue. Though some may need to be cancelled to resolve the apparent load on slurm.

I will follow up by editing this post with the resolution.

----------------------

Update #1

I can confirm that the issue is a high number of jobs (near 50K, total) pending in the slurm queue. This generated a high load for the database when queries were made such that the remote calls over the network (from any node to the management node) timed-out before they could be completed. 

By increasing the time the network considers for a 'time-out' I was able to finally gather information from the slurm database and administratively cancel a few tens of thousands of jobs. I am working now with slurm support to determine the best way to limit the total number of jobs on the queue. This will mean that future sbatch submissions will generate errors when attempting to submit a number of jobs that would put the slurm queue above the limit. This is preferable to squeue being unavailable to all users due to high load.

Happy Monday!

Roy Prouty

Due to memory consumption on the taki login node, `taki-usr1`, we are rebooting the node to clear associated errors.

When free memory on one of our machines drops below a certain threshold, a special system process starts to kill processes based on a set of industry-set heuristics. The demise of these processes could lead to unexpected machine behavior, necessitating a reboot of the machine.

I'd like to take this opportunity to remind all taki users that the login node is used by all taki users. This node should be used to access files, write and compile code, and submit jobs to the slurm scheduler. Tests of code or workflows should be delegated to the develop partition. More information on how to use these nodes is located on the HPCF Website (https://hpcf.umbc.edu/cpu/how-to-run-programs-on-taki/).

---------------------

Choosing to look at `squeue` as an indicator on this issue. Squeue is now responding but with an added message that "MessageTimeout is too high for effective fault-tolerance". This is expected. I increased the MessageTimeout configuration parameter in an attempt to give more time for the slurm database to respond. This seems to be working, but it leaving us with an authentication issue. 

Looking at the output of squeue, there are many tens of thousands of similar jobs being run by some users. I am cancelling these and will follow-up with the users and PIs (where necessary). I will also discuss with our slurm support how to configure slurm to avoid this situation in the future.

---------------------

HPC System Administrator

Research Computing Specialist, UMBC DoIT

Office: (410) 455-6351

If you notice any issues moving forward, please do let us know.  We will be keeping a close eye on the cluster and jobs to ensure the problem does not again crop up.  

Tim Champ

Manager of Unix Infrastructure

DoIT

As of now, we aren't sure that taki will be back online tonight, but we will send another update if we do get it back online tonight.  If not, we plan to update tomorrow as we continue to work toward a solution.

Thank you for your patience, and we do apologize for the inconvenience this is causing. 

Tim 

We're currently working on infiniband networking issues on Taki that are preventing access to various storage nodes.   We hope to have another update shortly.

Thanks!

Taki Users:

The taki cluster is now back online with full access to 'home' and 'research' storage areas. Additionally, slurm is operating normally.

At this time: Users can assume normal use of the cluster.

Thank you for your patience!

Roy Prouty

Taki Users:

At this time, we've isolated the issue with taki to the networking infrastructure that facilitates the communication between nodes as well as the hosting of the 'home' and 'research' storage. The user nodes are online, but these storage areas remain unavailable.

Please continue to avoid use of the cluster until the networking issues have been resolved.

Roy Prouty

Taki Users:

We are aware of an issue rendering users unable to login to taki. We are working to resolve this issue, and we will create another post when the system is available.

Happy Monday!

Roy Prouty