Taki Users:

This coming Tuesday (September 24, 2019), we will be rebooting the user nodes on taki. These reboots will update some of our software modules on the user nodes, allowing users to get a better idea of what is available across the compute and GPU nodes.

Let us know if you have any questions!

Roy Prouty

Taki Users:

Today (Aug 27, 2019) at 5pm, we brought taki down to install some maintenance daemons. At this point (5:20p), taki is back online and no jobs should have been affected.

As always, let us know if you notice any issues or have any questions about the HPCF.

-Roy Prouty

The class MATH 627 Introduction to Parallel Computing

introduces the current HPCF cluster taki and

teaches parallel computing using it.

Starting with an efficient introduction to the Linux operating system

and the basics of the C programming language,

it introduces MPI, which is the most prevalent library for

communications on distributed-memory clusters.

This course has the goal to help you use the cluster most effectively

by understanding how the hardware and software really work

as well as to contribute to community-building by meeting other HPCF users.

The class is taught by Matthias Gobbert from Mathematics and Statistics,

the faculty lead on the HPCF initiative and liason of

the HPCF Governance Committee to the system administrators in DoIT.

See the syllabus and more information at my homepage

http://www.umbc.edu/~gobbert.

Earlier this week we upgraded slurm (our scheduling software) to try and fix several bugs that had been identified over the past few weeks through working with our support vendor. This upgrade addressed some of the reported problems but also introduced a few configuration issues that impacted user's abilities to run jobs earlier in the week. In addition to resolving these new problems, we have been working to identify and address the issues with failed jobs and nodes that users have been experiencing that were not fixed through the upgrade. 

We understand that these issue have impacted the ability to conduct research on the cluster, and we are working diligently to resolve them. To help expedite this process we have engaged an external consulting firm as well as a team of HPC experts from Dell to work closely with our staff to bring the cluster back to full capacity. 

Over the last day as we have made adjustments, we have seen an increase in the job completion and job reliability rates on Taki. We believe this to be a good indication that we are getting closer to identifying some of the root causes of these issues but we know we aren't there yet. 

We appreciate the patience and the continued communication from all of our users. If you continue to experience issues, please open tickets with all much detail as possible so we can continue to diagnose and triage these issues. The information we get from your tickets is essential to resolving these issues. 

If you have any questions or concerns, please email me and I'll be happy to discuss these issues and the status of the cluster in more detail. 

Thank You, 

Damian

--

Damian Doyle

Assistant Vice President

DoIT - UMBC

Taki Users:

We are seeing many jobs stuck in a pending or "PD" state in slurm. The reason being given by slurm is "launch failed requeued held". This is slurm's way of telling us that something is wrong with the node configuration.

While we are investigating the potential issues, we also have high-priority tickets in with slurm-support. We aim to resolve this issue ASAP.

Thank you for your patience.

Taki will go down after business hours tomorrow August 20, 2019. The downtime is expected to run from 5p-7p. Scheduled jobs will be lost.

Please be sure to save or otherwise checkpoint all progress leading up to this time.

Taki Users:

After this weekend's unscheduled downtime, much of the cluster is back in working order.

Two Important Updates:

1) We took advantage of the unscheduled downtime to make an update to slurm to resolve some issues we were seeing affecting many users. A consequence of this is that users must now specify the slurm account to which they would like to charge their slurm job. For example, a user belonging to UNIX group "pi_professor" would need to add the line:

#SBATCH --account=pi_professor

to their slurm submission scripts. Failure to make this specification will result in a sbatch submission error "sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified".

Please submit an RT ticket is you experience issues with this. (https://doit.umbc.edu/request-tracker-rt/doit-research-computing/)

The HPCF Webpage will be updated to reflect this change.

2) We have to physically investigate the 2013 GPUs before they are brought back online. These should be booting now and available in the next thirty minutes.

Taki Users:

Earlier today (Sunday, August 18) we noticed many nodes (nearly all) become unavailable. We are currently working to bring these nodes back up and identify the original issue.

A few jobs had to be administratively cancelled in this process.

At the time of this writing, more than half of the nodes are back online and the remaining half are booting. None are exhibiting behaviors indicative of any further issue.

Let us know if you notice any abnormal behavior.

1. For purposes of workflow management, note that the HPCF may have downtime scheduled every Tuesday from 5p-9p. In the event that downtime is scheduled: a posting on this page will be made.

2. We will be taking downtime next Tuesday (July 9, 2019) from 5p-9p. The purpose of this downtime is to put new software packages and file systems in place.