Difference between revisions of "HPC:Main Page"
From HPC wiki
| (37 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
| − | High Performance Computing at PennMedicine is supported by the | + | High Performance Computing at PennMedicine is supported by the HPC group within Penn Medicine Academic Computing Services (PMACS). Please send requests and report problems to [mailto:psom-pmacshpc@pennmedicine.upenn.edu ]. |
| + | === PMACS=== | ||
| + | ---- | ||
| + | The High Performance Computing (HPC) team in the Penn Medicine Academic Computing Services (PMACS), is a small and diverse team of individuals focused on providing HPC and Research Computing support to the faculty and staff members of the University of Pennsylvania. | ||
| − | + | PMACS HPC Team: | |
| − | + | * Rikki Godshall, Associate Director, HPC and Cloud Services, PMACS | |
| − | * | + | * Jason Szostek, PMACS |
| − | * | + | * Balasundaram Subramanian, PMACS |
| − | * | + | * Pavel Kovtunenko, PMACS |
| − | * | ||
| − | + | Former team member: | |
| − | + | * Anand Srinivasan, Sr. IT Project Leader/Systems Architect, HPC and Cloud Services,PMACS | |
| − | PMACS | + | === About the PMACS Cluster === |
| − | + | ---- | |
| − | + | The PMACS HPC facility opened in April 2013 to meet increasing growth in processing and storage needs for our epidemiology, biostatistics, genomics, and bioinformatics groups. The compute cluster and its storage subsystems are managed by the HPC team within Penn Medicine Academic Computing Services (PMACS), and is located at the Tierpoint Philadelphia, a Tier-3, SSAE 16/SAS 70 Type II Audit compliant colocation/data center facility. | |
| − | |||
=== Weekly Office Hours === | === Weekly Office Hours === | ||
| − | + | ---- | |
| + | We are no longer hosting regular HPC office hours. For HPC related questions, please contact us via email. | ||
| − | + | ===Hardware=== | |
| − | + | ---- | |
| − | + | <div class="mw-collapsible mw-collapsed"> | |
| − | + | The hardware of the PMACS Cluster comprises of: | |
| − | + | * '''1 dedicated master node (VM) with 16 cores & 64GB RAM''' | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | * '''1 dedicated master node ( | ||
* '''1 shadow master node (VM) with 12 cores & 24GB RAM''' | * '''1 shadow master node (VM) with 12 cores & 24GB RAM''' | ||
| − | * ''' | + | * '''3176 Total Physical cores''' across all compute nodes (up to 6352 virtual cores with hyper-threading turned on) |
* '''7168 Total CUDA cores across GPU nodes''' | * '''7168 Total CUDA cores across GPU nodes''' | ||
| − | * ''' | + | * '''40TB of total RAM across all compute nodes''' |
| − | * ''' | + | * '''76 Compute nodes''' |
| − | ** ''' | + | ** '''19x Dell C6420 Quad nodes''' (4 nodes per enclosure; 76 compute nodes total) |
*** Each node within the Dell C6420 enclosure has: | *** Each node within the Dell C6420 enclosure has: | ||
**** Two 20 core Intel Xeon Gold 6148 2.40GHz CPU, with hyper-threading turned on (so 80 threads per node) | **** Two 20 core Intel Xeon Gold 6148 2.40GHz CPU, with hyper-threading turned on (so 80 threads per node) | ||
**** 256-512 GB RAM each (6.4-12.8 GB physical core, depending on the node) | **** 256-512 GB RAM each (6.4-12.8 GB physical core, depending on the node) | ||
| − | **** 56 Gb/s InfiniBand connection to the GPFS file system | + | **** Mix of 56 Gb/s and 100 Gb/s InfiniBand connection to the GPFS file system |
**** 1.6TB dedicated scratch space provided by local SSD or NVMe (depending on node) | **** 1.6TB dedicated scratch space provided by local SSD or NVMe (depending on node) | ||
| − | + | * '''1x Dell R940 "Big Memory" node''' | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
*** 4x 12-core Intel Xeon Gold 6126 2.60GHz CPUs, 96 threads total with hyper-threading turned on | *** 4x 12-core Intel Xeon Gold 6126 2.60GHz CPUs, 96 threads total with hyper-threading turned on | ||
*** '''1.5TB RAM''' | *** '''1.5TB RAM''' | ||
| − | *** | + | *** 100 Gb/s InfiniBand connection to the GPFS file system |
*** 1.6TB dedicated scratch space provided by local NVMe | *** 1.6TB dedicated scratch space provided by local NVMe | ||
| − | |||
| − | |||
| − | |||
| − | |||
* '''2x GPU nodes'''; each configured with | * '''2x GPU nodes'''; each configured with | ||
*** 2x 22-core Intel Xeon E5-2699 v4 2.20GHz CPUs (88 threads per node, with hyperthreading turned on) | *** 2x 22-core Intel Xeon E5-2699 v4 2.20GHz CPUs (88 threads per node, with hyperthreading turned on) | ||
| − | *** | + | *** 512GB RAM |
*** '''1x Nvidia Tesla P100 16GB GPU Card (3584 CUDA cores & 16GB RAM per card)''' | *** '''1x Nvidia Tesla P100 16GB GPU Card (3584 CUDA cores & 16GB RAM per card)''' | ||
| − | *** | + | *** 100 Gb/s InfiniBand connection to the GPFS file system |
| + | *** 1.6TB dedicated scratch space provided by local NVMe | ||
* '''Storage''' | * '''Storage''' | ||
** '''4.2 PetaBytes''' of total usable shared IBM GPFS disk storage provided via a six-node InfiniBand connected GPFS cluster | ** '''4.2 PetaBytes''' of total usable shared IBM GPFS disk storage provided via a six-node InfiniBand connected GPFS cluster | ||
| Line 72: | Line 58: | ||
Computational job scheduling/queuing and cluster management is orchestrated by the IBM Platform Computing suite of products. | Computational job scheduling/queuing and cluster management is orchestrated by the IBM Platform Computing suite of products. | ||
</div> | </div> | ||
| − | === Costs === | + | |
| + | ===Costs=== | ||
| + | ---- | ||
| + | <div class="mw-collapsible mw-collapsed"> | ||
| + | |||
The PennHPC baseline cost structure is fee-for-service based around the service-center model. Below are the costs associated with using the PMACS cluster as of 4/30/2014: | The PennHPC baseline cost structure is fee-for-service based around the service-center model. Below are the costs associated with using the PMACS cluster as of 4/30/2014: | ||
| Line 80: | Line 70: | ||
* $95/hour for consulting services (excludes account setup) | * $95/hour for consulting services (excludes account setup) | ||
* No charges to maintain an account; charges are billed on an as-consumed basis only. If data is left behind by a user who longer uses the cluster, charges will be billed for disk usage until the data is deleted from the account. | * No charges to maintain an account; charges are billed on an as-consumed basis only. If data is left behind by a user who longer uses the cluster, charges will be billed for disk usage until the data is deleted from the account. | ||
| + | </div> | ||
| − | === Accounts === | + | ===Accounts=== |
| − | For Account | + | ---- |
| + | <div class="mw-collapsible mw-collapsed"> | ||
| + | For an Account Request please choose one of the following<br/> | ||
| + | [https://helpdesk.pmacs.upenn.edu/userui/hd_ticket_quick_links.php Open a '''Systems''' ticket in the PMACS Helpdesk]<br/> | ||
| + | [mailto:psom-pmacshpc@pennmedicine.upenn.edu Email PMACS HPC here: <code>psom-pmacshpc@pennmedicine.upenn.edu</code>] | ||
Please be sure to include the following information in your account request email: | Please be sure to include the following information in your account request email: | ||
| + | * User's Full Name: | ||
| + | * User's Email: | ||
| + | * User's PennKey: | ||
| + | * User's PennID: | ||
| + | * User's Status: Student/Post-Doctoral Fellow | ||
| + | * Does the data the user intends to transmit to/from, store, or process on the PennHPC require HIPAA, FISMA, or 21 CFR Part 11 compliance?: Yes/No | ||
| + | * PI's Full Name: | ||
| + | * PI's Email: | ||
| + | * BA's Name: | ||
| + | * BA's Email: | ||
| + | * 26-digit Budget code to bill HPC usage to: | ||
| − | + | '''Note:''' If the user account is not requested by the BA/PI, we will follow up directly with the BA/PI for authorization. | |
| − | + | </div> | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | ===Usage Policies=== | |
| − | + | ---- | |
| − | + | Penn Acceptable Use Policy: http://www.upenn.edu/computing/policy/aup.html | |
| − | |||
| − | + | ===HIPAA data=== | |
| − | + | Our HPC systems have always, and continue to be, operated in a secure, Tier III off-campus data center at Tierpoint Philadelphia which is an SSAE 16/SAS 70 Type II Audit compliant co-location data center facility. | |
| − | + | Building on those physical location attributes, the compute cluster with its tightly coupled highly performant parallel file system for disk storage and the self-service tape-based archive system, are, following the requisite review of the logical environment, has been qualified to accept HIPAA data. | |
| − | |||
| + | Users of the HPC resources who wish to process or store data subject to HIPAA compliance requirements, must inform the systems administrators of their need before loading data, and not mingle it in directories with non-compliant data. Upon receipt of the request, HPC System Administrators will create a project directory following a standard naming convention for HIPAA data with the appropriate permissions and access control (ACLs) enforced. Only users who are specifically identified by the Data Owner (usually the project's Principal Investigator) will be added to the corresponding permission group to gain access. | ||
| − | + | It is the responsibility of the Data Owner to ensure that all users granted access to a HIPAA data directory have received the required HIPAA training. For employees of the University of Pennsylvania and Penn Medicine, free training is available through the Knowledge Link system. | |
| − | === | + | === Other Pages === |
| − | + | ---- | |
| − | + | <div class="mw-collapsible mw-collapsed"> | |
| − | + | *[[HPC:FAQ|HPC FAQ ]] | |
| − | + | *[[HPC:Login|Connecting to the PMACS cluster]] | |
| + | *[[HPC:User_Guide|User Guide]] | ||
| + | *[[HPC:Software|Available Software]] | ||
| + | *[[HPC:Archive System|PMACS Archive System]] | ||
| + | </div> | ||
Latest revision as of 14:43, 7 July 2025
High Performance Computing at PennMedicine is supported by the HPC group within Penn Medicine Academic Computing Services (PMACS). Please send requests and report problems to [1].
Contents
PMACS
The High Performance Computing (HPC) team in the Penn Medicine Academic Computing Services (PMACS), is a small and diverse team of individuals focused on providing HPC and Research Computing support to the faculty and staff members of the University of Pennsylvania.
PMACS HPC Team:
- Rikki Godshall, Associate Director, HPC and Cloud Services, PMACS
- Jason Szostek, PMACS
- Balasundaram Subramanian, PMACS
- Pavel Kovtunenko, PMACS
Former team member:
- Anand Srinivasan, Sr. IT Project Leader/Systems Architect, HPC and Cloud Services,PMACS
About the PMACS Cluster
The PMACS HPC facility opened in April 2013 to meet increasing growth in processing and storage needs for our epidemiology, biostatistics, genomics, and bioinformatics groups. The compute cluster and its storage subsystems are managed by the HPC team within Penn Medicine Academic Computing Services (PMACS), and is located at the Tierpoint Philadelphia, a Tier-3, SSAE 16/SAS 70 Type II Audit compliant colocation/data center facility.
Weekly Office Hours
We are no longer hosting regular HPC office hours. For HPC related questions, please contact us via email.
Hardware
The hardware of the PMACS Cluster comprises of:
- 1 dedicated master node (VM) with 16 cores & 64GB RAM
- 1 shadow master node (VM) with 12 cores & 24GB RAM
- 3176 Total Physical cores across all compute nodes (up to 6352 virtual cores with hyper-threading turned on)
- 7168 Total CUDA cores across GPU nodes
- 40TB of total RAM across all compute nodes
- 76 Compute nodes
- 19x Dell C6420 Quad nodes (4 nodes per enclosure; 76 compute nodes total)
- Each node within the Dell C6420 enclosure has:
- Two 20 core Intel Xeon Gold 6148 2.40GHz CPU, with hyper-threading turned on (so 80 threads per node)
- 256-512 GB RAM each (6.4-12.8 GB physical core, depending on the node)
- Mix of 56 Gb/s and 100 Gb/s InfiniBand connection to the GPFS file system
- 1.6TB dedicated scratch space provided by local SSD or NVMe (depending on node)
- Each node within the Dell C6420 enclosure has:
- 19x Dell C6420 Quad nodes (4 nodes per enclosure; 76 compute nodes total)
- 1x Dell R940 "Big Memory" node
- 4x 12-core Intel Xeon Gold 6126 2.60GHz CPUs, 96 threads total with hyper-threading turned on
- 1.5TB RAM
- 100 Gb/s InfiniBand connection to the GPFS file system
- 1.6TB dedicated scratch space provided by local NVMe
- 2x GPU nodes; each configured with
- 2x 22-core Intel Xeon E5-2699 v4 2.20GHz CPUs (88 threads per node, with hyperthreading turned on)
- 512GB RAM
- 1x Nvidia Tesla P100 16GB GPU Card (3584 CUDA cores & 16GB RAM per card)
- 100 Gb/s InfiniBand connection to the GPFS file system
- 1.6TB dedicated scratch space provided by local NVMe
- Storage
- 4.2 PetaBytes of total usable shared IBM GPFS disk storage provided via a six-node InfiniBand connected GPFS cluster
- 1.2 PetaBytes Archival storage
- All compute and storage connected via 10GigE interconnect
- Dedicated 10GigE PennNet link from campus to the datacenter
Computational job scheduling/queuing and cluster management is orchestrated by the IBM Platform Computing suite of products.
Costs
The PennHPC baseline cost structure is fee-for-service based around the service-center model. Below are the costs associated with using the PMACS cluster as of 4/30/2014:
- $0.035/computational/vCore slot hour
- $0.055/GB/month for Disk usage
- $0.015/GB/month for Archive storage
- $95/hour for consulting services (excludes account setup)
- No charges to maintain an account; charges are billed on an as-consumed basis only. If data is left behind by a user who longer uses the cluster, charges will be billed for disk usage until the data is deleted from the account.
Accounts
For an Account Request please choose one of the following
Open a Systems ticket in the PMACS Helpdesk
Email PMACS HPC here: psom-pmacshpc@pennmedicine.upenn.edu
Please be sure to include the following information in your account request email:
- User's Full Name:
- User's Email:
- User's PennKey:
- User's PennID:
- User's Status: Student/Post-Doctoral Fellow
- Does the data the user intends to transmit to/from, store, or process on the PennHPC require HIPAA, FISMA, or 21 CFR Part 11 compliance?: Yes/No
- PI's Full Name:
- PI's Email:
- BA's Name:
- BA's Email:
- 26-digit Budget code to bill HPC usage to:
Note: If the user account is not requested by the BA/PI, we will follow up directly with the BA/PI for authorization.
Usage Policies
Penn Acceptable Use Policy: http://www.upenn.edu/computing/policy/aup.html
HIPAA data
Our HPC systems have always, and continue to be, operated in a secure, Tier III off-campus data center at Tierpoint Philadelphia which is an SSAE 16/SAS 70 Type II Audit compliant co-location data center facility. Building on those physical location attributes, the compute cluster with its tightly coupled highly performant parallel file system for disk storage and the self-service tape-based archive system, are, following the requisite review of the logical environment, has been qualified to accept HIPAA data.
Users of the HPC resources who wish to process or store data subject to HIPAA compliance requirements, must inform the systems administrators of their need before loading data, and not mingle it in directories with non-compliant data. Upon receipt of the request, HPC System Administrators will create a project directory following a standard naming convention for HIPAA data with the appropriate permissions and access control (ACLs) enforced. Only users who are specifically identified by the Data Owner (usually the project's Principal Investigator) will be added to the corresponding permission group to gain access.
It is the responsibility of the Data Owner to ensure that all users granted access to a HIPAA data directory have received the required HIPAA training. For employees of the University of Pennsylvania and Penn Medicine, free training is available through the Knowledge Link system.
