HPC:Main Page

High Performance Computing at PennMedicine is supported by the Enterprise Research Applications (ERA) group within Penn Medicine Academic Computing Services (PMACS). Please send requests and report problems to Jim/Rikki/Anand.

The High Performance Computing (HPC) team within the Enterprise Research Applications (ERA) group in Penn Medicine Academic Computing Services (PMACS), is a small and diverse team of individuals focused on providing HPC and Research Computing support to the faculty and staff members of the University of Pennsylvania.

PMACS ERA HPC Team:

Rikki Godshall, Associate Director, HPC and Cloud Services, ERA, PMACS

Former team member:

Anand Srinivasan, Sr. IT Project Leader/Systems Architect, HPC and Cloud Services, ERA, PMACS

About the PMACS Cluster

The PMACS/DART HPC facility opened in April 2013 to meet increasing growth in processing and storage needs for our epidemiology, biostatistics, genomics, and bioinformatics groups. The compute cluster and its storage subsystems are managed by the Enterprise Research Applications (ERA) team within Penn Medicine Academic Computing Services (PMACS), and is located at the Tierpoint Philadelphia, a Tier-3, SSAE 16/SAS 70 Type II Audit compliant colocation/data center facility.

Weekly Office Hours

We are no longer hosting regular HPC office hours. For HPC related questions, please contact us via email.

Hardware

The hardware of the PMACS Cluster comprises of:

1 dedicated master node (VM) with 16 cores & 64GB RAM
1 shadow master node (VM) with 12 cores & 24GB RAM
3176 Total Physical cores across all compute nodes (up to 6352 virtual cores with hyper-threading turned on)
7168 Total CUDA cores across GPU nodes
40TB of total RAM across all compute nodes
76 Compute nodes
- 19x Dell C6420 Quad nodes (4 nodes per enclosure; 76 compute nodes total)
  - Each node within the Dell C6420 enclosure has:
    - Two 20 core Intel Xeon Gold 6148 2.40GHz CPU, with hyper-threading turned on (so 80 threads per node)
    - 256-512 GB RAM each (6.4-12.8 GB physical core, depending on the node)
    - Mix of 56 Gb/s and 100 Gb/s InfiniBand connection to the GPFS file system
    - 1.6TB dedicated scratch space provided by local SSD or NVMe (depending on node)
1x Dell R940 "Big Memory" node
- - 4x 12-core Intel Xeon Gold 6126 2.60GHz CPUs, 96 threads total with hyper-threading turned on
  - 1.5TB RAM
  - 100 Gb/s InfiniBand connection to the GPFS file system
  - 1.6TB dedicated scratch space provided by local NVMe
2x GPU nodes; each configured with
- - 2x 22-core Intel Xeon E5-2699 v4 2.20GHz CPUs (88 threads per node, with hyperthreading turned on)
  - 512GB RAM
  - 1x Nvidia Tesla P100 16GB GPU Card (3584 CUDA cores & 16GB RAM per card)
  - 100 Gb/s InfiniBand connection to the GPFS file system
  - 1.6TB dedicated scratch space provided by local NVMe
Storage
- 4.2 PetaBytes of total usable shared IBM GPFS disk storage provided via a six-node InfiniBand connected GPFS cluster
- 1.2 PetaBytes Archival storage
All compute and storage connected via 10GigE interconnect
Dedicated 10GigE PennNet link from campus to the datacenter

Computational job scheduling/queuing and cluster management is orchestrated by the IBM Platform Computing suite of products.

Costs

The PennHPC baseline cost structure is fee-for-service based around the service-center model. Below are the costs associated with using the PMACS cluster as of 4/30/2014:

$0.035/computational/vCore slot hour
$0.055/GB/month for Disk usage
$0.015/GB/month for Archive storage
$95/hour for consulting services (excludes account setup)
No charges to maintain an account; charges are billed on an as-consumed basis only. If data is left behind by a user who longer uses the cluster, charges will be billed for disk usage until the data is deleted from the account.

Accounts

For Account requests please contact Jim Kaylor, Rikki Godshall or Anand Srinivasan

Please be sure to include the following information in your account request email:

User Info:
- User's Full Name:
- User's Email:
- User's PennKey:
- User's PennID:
- User's Status: Student/Post-Doctoral Fellow
- Lab rotation end date/Account expiration date (if applicable):
- Does the data the user intends to transmit to/from, store, or process on the PennHPC require HIPAA, FISMA, or 21 CFR Part 11 compliance?: Yes/No

PI Info:
- PI's Full Name:
- PI's Email:
- PI's PennKey (if exists):

Business Administrator (BA)/Billing info:
- BA's Name:
- BA's Email:
- 26-digit Budget code to bill HPC usage to:

Note: If the user account is not requested by the BA/PI, we will follow up directly with the BA/PI for authorization.

Usage Policies

Penn Acceptable Use Policy: http://www.upenn.edu/computing/policy/aup.html

HIPAA data

Our HPC systems have always, and continue to be, operated in a secure, Tier III off-campus data center at Tierpoint Philadelphia which is an SSAE 16/SAS 70 Type II Audit compliant co-location data center facility. Building on those physical location attributes, the compute cluster with its tightly coupled highly performant parallel file system for disk storage and the self-service tape-based archive system, are, following the requisite review of the logical environment, has been qualified to accept HIPAA data.

Users of the HPC resources who wish to process or store data subject to HIPAA compliance requirements, must inform the systems administrators of their need before loading data, and not mingle it in directories with non-compliant data. Upon receipt of the request, HPC System Administrators will create a project directory following a standard naming convention for HIPAA data with the appropriate permissions and access control (ACLs) enforced. Only users who are specifically identified by the Data Owner (usually the project's Principal Investigator) will be added to the corresponding permission group to gain access.

It is the responsibility of the Data Owner to ensure that all users granted access to a HIPAA data directory have received the required HIPAA training. For employees of the University of Pennsylvania and Penn Medicine, free training is available through the Knowledge Link system.