This lesson is in the early stages of development (Alpha version)

Exploring Remote Resources

Overview

Teaching: 25 min
Exercises: 10 min
Questions
  • How does my local computer compare to the remote systems?

  • How does the login node compare to the compute nodes?

  • Are all compute nodes alike?

Objectives
  • Survey system resources using nproc, free, and the queuing system

  • Compare & contrast resources on the local machine, login node, and worker nodes

  • Learn about the various filesystems on the cluster using df

  • Find out who else is logged in

  • Assess the number of idle and occupied nodes

Look Around the Remote System

If you have not already connected to Greene, please do so now:

[user@laptop ~]$  ssh NYUNetID@greene.hpc.nyu.edu

Take a look at your home directory on the remote system:

[NYUNetID@log-1 ~]$ ls

What’s different between your machine and the remote?

Open a second terminal window on your local computer and run the ls command (without logging in to Greene). What differences do you see?

Solution

You would likely see something more like this:

[user@laptop ~]$ ls
Applications Documents    Library      Music        Public
Desktop      Downloads    Movies       Pictures

The remote computer’s home directory shares almost nothing in common with the local computer: they are completely separate systems!

Most high-performance computing systems run the Linux operating system, which is built around the UNIX Filesystem Hierarchy Standard. Instead of having a separate root for each hard drive or storage medium, all files and devices are anchored to the “root” directory, which is /:

[NYUNetID@log-1 ~]$ ls /
afs      bin@  dev  gpfs  lib@    media  mnt  opt   root  sbin@    share  state  tmp  var
archive  boot  etc  home  lib64@  misc   net  proc  run   scratch  srv    sys    usr  vast

The “homeNYUNetID” directory is the one where we generally want to keep all of our files. Other folders on a UNIX OS contain system files and change as you install new software or upgrade your OS.

Using HPC filesystems

On Geene, you have a number of places where you can store your files. These differ in both the amount of space allocated and whether or not they are backed up.

  • Home – data stored here is available throughout the HPC system, and often backed up periodically. Please note the limit on the number of files (inodes) which can get used up easily. Use the myquota command to ensure that you are not running out of inodes!
  • Scratch – used for temporary file storage while running jobs. It is not backed up and files that are unused for over 60 days are purged. usually backed up, and should not be used for long term storage.
  • Vast – flash-drive based system that is optimal for workloads with high I/O rates. Like the scratch storage, it is also not backed up and files that are unused for over 60 days are purged.
  • RPS – provides data storage space for research projects that is easily shared amongst collaborators, backed up, and not subject to the old file purging policy. Note that it is a paid service.
  • Archive – provides a space for long-term storage of research output and is unaccessible by running jobs.

Nodes

Individual computers that compose a cluster are typically called nodes (although you will also hear people call them servers, computers and machines). On a cluster, there are different types of nodes for different types of tasks. The node where you are right now is called the login node, head node, landing pad, or submit node. A login node serves as an access point to the cluster.

As a gateway, the login node should not be used for time-consuming or resource-intensive tasks. You should be alert to this, and check with your site’s operators or documentation for details of what is and isn’t allowed. It is well suited for uploading and downloading files, setting up software, and running tests. Generally speaking, in these lessons, we will avoid running jobs on the login node.

Who else is logged in to the login node?

[NYUNetID@log-1 ~]$ who

This may show only your user ID, but there are likely several other people (including fellow learners) connected right now.

Dedicated Transfer Nodes

If you want to transfer larger amounts of data to or from the cluster, Greene offers dedicated nodes for data transfers only. The motivation for this lies in the fact that larger data transfers should not obstruct operation of the login node for anybody else. As a rule of thumb, consider all transfers of a volume larger than 500 MB to 1 GB as large. But these numbers change, e.g., depending on the network connection of yourself and of your cluster or other factors.

The real work on a cluster gets done by the compute (or worker) nodes. compute nodes come in many shapes and sizes, but generally are dedicated to long or hard tasks that require a lot of computational resources.

All interaction with the compute nodes is handled by a specialized piece of software called a scheduler (the scheduler used in this lesson is called Slurm). We’ll learn more about how to use the scheduler to submit jobs next, but for now, it can also tell us more information about the compute nodes.

For example, we can view all of the compute nodes by running the command sinfo.

[NYUNetID@log-1 ~]$ sinfo
PARTITION        AVAIL  TIMELIMIT  NODES  STATE NODELIST
cs                  up   infinite      1 drain* cs524
cs                  up   infinite      1   drng cs477
cs                  up   infinite      3  drain cs[001,478,523]
cs                  up   infinite    117    mix cs[005-009,014,023,027-028,030,093-104,109-115,124,127-128,132-133,135-148,151,155-156,159-168,170-178,181-192,194-214,274,305-306,322,338-340,346,400,436,442,444-445,488]
cs                  up   infinite      1   resv cs402
cs                  up   infinite    400  alloc cs[003-004,010-013,015-022,024-026,029,031-092,105-108,116-123,125-126,129-131,134,149-150,152-154,157-158,169,179-180,193,215-273,275-304,307-321,323-337,341-345,347-399,401,403-435,437-441,443,446-476,479-487,489-522]
cm                  up   infinite     37    mix cm[001-003,005,010,013-044]
cm                  up   infinite      7  alloc cm[004,006-009,011-012]
cl                  up   infinite      2    mix cl[003-004]
cl                  up   infinite      2  alloc cl[001-002]
v100                up   infinite      4   resv gv[001-004]
v100                up   infinite      6    mix gv[005-010]
rtx8000             up   infinite      1   resv gr030
rtx8000             up   infinite     47    mix gr[001-004,007-008,010-027,029,031,034-042,049-060]
rtx8000             up   infinite     12  alloc gr[005-006,009,028,032-033,043-048]
a100_1              up   infinite      9    mix ga[001-009]
a100_2              up   infinite      1   drng ga037
a100_2              up   infinite     18    mix ga[010-024,041-043]
a100_2              up   infinite     15  alloc ga[025-036,038-040]
cds_rtx_d           up   infinite      1  drain gr073
cds_rtx_d           up   infinite     19    mix gr[054-072]
cds_rtx_a           up   infinite      1  drain gr073
cds_rtx_a           up   infinite      1   resv gr030
cds_rtx_a           up   infinite     59    mix gr[001-004,007-008,010-027,029,031,034-042,049-072]
cds_rtx_a           up   infinite     12  alloc gr[005-006,009,028,032-033,043-048]
cilvr_a100          up   infinite      9    mix ga[001-009]
cilvr_a100_1        up   infinite      8    mix ga[001-008]
tandon_a100_1       up   infinite      1    mix ga009
cds_a100_2          up   infinite      6    mix ga[010-012,041-043]
tandon_a100_2       up   infinite     12    mix ga[013-024]
tandon_a100_2       up   infinite      2  alloc ga[025-026]
chemistry_a100_2    up   infinite      1   drng ga037
chemistry_a100_2    up   infinite     13  alloc ga[027-036,038-040]
stake_a100_1        up   infinite      9    mix ga[001-009]
stake_a100_2        up   infinite      1   drng ga037
stake_a100_2        up   infinite     18    mix ga[010-024,041-043]
stake_a100_2        up   infinite     15  alloc ga[025-036,038-040]
tandon_h100_1       up   infinite     15    mix gh[001-015]
stake_h100_1        up   infinite     15    mix gh[001-015]
h100_1              up   infinite     15    mix gh[001-015]
cpu_a100_1          up   infinite      9    mix ga[001-009]
cpu_a100_2          up   infinite      1   drng ga037
cpu_a100_2          up   infinite     18    mix ga[010-024,041-043]
cpu_a100_2          up   infinite     15  alloc ga[025-036,038-040]
cpu_gpu             up   infinite      1  drain gr073
cpu_gpu             up   infinite      5   resv gr030,gv[001-004]
cpu_gpu             up   infinite     65    mix gr[001-004,007-008,010-027,029,031,034-042,049-072],gv[005-010]
cpu_gpu             up   infinite     12  alloc gr[005-006,009,028,032-033,043-048]
mi50                up   infinite      1   resv gm011
mi50                up   infinite     16    mix gm[001-008,012,014-020]
mi50                up   infinite      3   idle gm[009-010,013]
mi100               up   infinite      3    mix gm[021-023]
mi250               up   infinite      1   resv gm025
mi250               up   infinite      1    mix gm024
gpu_misc_v100       up   infinite      1    mix gv012
chem_cpu0           up   infinite      1    mix cs488
chem_cpu0           up   infinite     22  alloc cs[489-510]
xwang               up   infinite     10  alloc cs[511-520]
short               up   infinite     38    mix cm[001-003,005,010,013-045]
short               up   infinite      7  alloc cm[004,006-009,011-012]
short               up   infinite      5   idle cm[046-050]

A lot of the nodes are busy running work for other users: we are not alone here!

There are also specialized machines used for managing disk storage, user authentication, and other infrastructure-related tasks. Although we do not typically logon to or interact with these machines directly, they enable a number of key features like ensuring our user account and files are available throughout the HPC system.

What’s in a Node?

All of the nodes in an HPC system have the same components as your own laptop or desktop: CPUs (sometimes also called processors or cores), memory (or RAM), and disk space. CPUs are a computer’s tool for actually running programs and calculations. Information about a current task is stored in the computer’s memory. Disk refers to all storage that can be accessed like a file system. This is generally storage that can hold data permanently, i.e. data is still there even if the computer has been restarted. While this storage can be local (a hard drive installed inside of it), it is more common for nodes to connect to a shared, remote fileserver or cluster of servers.

/hpc-intro/Node%20anatomy

Explore Your Computer

Try to find out the number of CPUs and amount of memory available on your personal computer.

Note that, if you’re logged in to the remote computer cluster, you need to log out first. To do so, type Ctrl+d or exit:

[NYUNetID@log-1 ~]$ exit
[user@laptop ~]$

Solution

There are several ways to do this. Most operating systems have a graphical system monitor, like the Windows Task Manager. More detailed information can be found on the command line:

  • Run system utilities
    [user@laptop ~]$ nproc --all
    [user@laptop ~]$ free -m
    
  • Read from /proc
    [user@laptop ~]$ cat /proc/cpuinfo
    [user@laptop ~]$ cat /proc/meminfo
    
  • Run system monitor
    [user@laptop ~]$ htop
    

Explore the Login Node

Now compare the resources of your computer with those of the login node.

Solution

[user@laptop ~]$ ssh NYUNetID@greene.hpc.nyu.edu
[NYUNetID@log-1 ~]$ nproc --all
[NYUNetID@log-1 ~]$ free -m

You can get more information about the processors using lscpu, and a lot of detail about the memory by reading the file /proc/meminfo:

[NYUNetID@log-1 ~]$ less /proc/meminfo

You can also explore the available filesystems using df to show disk free space. The -h flag renders the sizes in a human-friendly format, i.e., GB instead of B. The type flag -T shows what kind of filesystem each resource is.

[NYUNetID@log-1 ~]$ df -Th

Different results from df

  • The local filesystems (ext, tmp, xfs, zfs) will depend on whether you’re on the same login node (or compute node, later on).
  • Networked filesystems (beegfs, cifs, gpfs, nfs, pvfs) will be similar – but may include NYUNetID, depending on how it is mounted.

Shared Filesystems

This is an important point to remember: files saved on one node (computer) are often available everywhere on the cluster!

Explore a Worker Node

Finally, let’s look at the resources available on the worker nodes where your jobs will actually run. Try running this command to see the name, CPUs and memory available on a worker node:

 sinfo --node cs012 -o "%n %c %m"

Compare Your Computer, the Login Node and the Compute Node

Compare your laptop’s number of processors and memory with the numbers you see on the cluster login node and compute node. What implications do you think the differences might have on running your research work on the different systems and nodes?

Solution

Compute nodes have substantially more memory (RAM) installed than a personal computer. More, faster memory is key for large or complex numerical tasks.

Differences Between Nodes

Many HPC clusters have a variety of nodes optimized for particular workloads. Some nodes may have larger amount of memory, or specialized resources such as Graphics Processing Units (GPUs or “video cards”).

With all of this in mind, we will now cover how to talk to the cluster’s scheduler, and use it to start running our scripts and programs!

Key Points

  • An HPC system is a set of networked machines.

  • HPC systems typically provide login nodes and a set of compute nodes.

  • The resources found on independent (worker) nodes can vary in volume and type (amount of RAM, processor architecture, availability of network mounted filesystems, etc.).

  • Files saved on shared storage are available on all nodes.

  • The login node is a shared machine: be considerate of other users.