On this page you will find answers to some of the most common questions we get about the PMACS HPC System.
- 1 Costs
- 2 Billing
- 3 Accounts
- 4 General Questions
- 5 PMACS ERA Team Contact Info
- 6 Grant related questions
- 7 Accessing the PMACS HPC
- 8 Job related Questions
- 9 Software Installation
- 10 Troubleshooting Job Failures
- 11 Downloading Sequencing data from BaseSpace
- 12 Sharing Data
- 13 Technical Questions
- 14 Other Pages
- Is the PMACS HPC service free?
No. The PMACS HPC service is a pay-for-use service. See details below.
- How much does it cost to have a PMACS HPC account account
There are no costs associated with requesting or maintaining a PMACS HPC account. There are however costs associated with using system resources, be that disk or archive (for storage) or computational resources (like CPU and RAM). See below
- How much does it cost to use the PMACS HPC cluster?
- I have not used the PMACS HPC for some time, yet, I keep getting usage charges. Why?
If there is data left in an individual user's home directory or in the project and/or archive directory that is owned by the individual user, disk usage charges will apply.
- Why are disk/archive usage charges incurred when the individual user's account has been disabled?
Usage charges, be that disk or archive, are billed against UNIX groups. When accounts are disabled, the associated UNIX group is not deleted. Any data owned by that UNIX group is not deleted either. Therefore, usage charges will continue to appear in the monthly invoices. See Billing section below for details.
- I'm the PI/BA and wish to have disk/archive usage charges for a former member of my lab removed from my monthly bill. How can this be accomplished?
Having storage (disk/archive) charges for former lab members appear in monthly invoices, while a bit confusing, is usually an indication that the former lab member did not follow the correct procedure to transfer ownership/delete the data in their home directory, prior to their departure from the lab. This can be easily resolved by changing the ownership of the data in question. See Billing section below for details on how storage charges are billed.
- How is PMACS HPC usage billed?
PMACS HPC usage is billed based system resources that are used by the individual user. System resources can either be storage (disk/archive) or compute (using the PMACS HPC's available CPU cores/RAM to do analysis) or both compute and storage. See Costs section above for details.
- How are storage charges computed?
Disk and Archive storage usage for every individual PMACS HPC user is monitored daily and costs are computed for usage accrued in a given month.
- How are charges for compute calculated?
The PMACS HPC system uses a job scheduling system - an IBM product called Spectrum LSF (formerly called Platform LSF). The job scheduler keeps track of resources used when a job is run on the PMACS HPC system. Cost of computational time on the PMACS HPC system is calculated when a job has completed and resource utilization information has been logged by the job scheduler.
- Are storage charges associated with individual users or with the lab the individual user works in?
Both. Storage charges for any data placed either on the PMACS HPC's disk or archive system is associated with the UNIX group that owns the data in question. In a POSIX compliant system, each file or directory can have two "owners" - the individual user who created the file there and a UNIX group whose members are also designated to have access to the file/directory.
Example 1: $ ls -lh $HOME/my_large_file1 -rw-rw-r-- 1 asrini asrini 1.0G Nov 6 12:15 /home/asrini/my_large_file1 The above file, created within in a user's home directory, is set to be owned by the individual user and the UNIX group of that user by default. Monthly usage charges for data owned by the individual user will be listed as a line item with the individual's name. Example 2: $ ls -lh /project/asrinilab/my_large_file2 -rw-rw-r-- 1 asrini asrinilab 1.0G Nov 6 12:15 /project/asrinilab/my_large_file2 The above file, created within in a lab's project directory, is set to be owned by the UNIX group of that lab by default. Monthly usage charges for data owned by the UNIX group of the lab, will be listed as a separate line item with the lab/project directory's name.
- Why are disk usage charges for former lab members being billed when their home directory is empty?
Disk and archive usage charges are billed based on UNIX group ownership (see above). When an individual uses the UNIX move operation (the "mv" command) to move contents of their home directory, the permissions set on the lab's project directory are not respected. Therefore, the data continues to be owned by the individual's UNIX group rather than the lab's UNIX group. Thus, a line item for disk usage charges in the monthly invoice with the former lab member's name continues to appear. This can be fixed by changing ownership of the data.
- Why are archive usage charges for former lab members being billed when their home directory is empty?
Archive usage charges are billed based on UNIX group ownership. Not having data on the disk, does not impact archive usage charges. If data was placed in the archive using the UNIX move operation (the "mv" command) or if the the prescribed procedure for placing, files in the archive was not followed, the data continues to be owned by the individual's UNIX group rather than the lab's UNIX group. Thus, a line item for archive usage charges in the monthly invoice with the former lab member's name continues to appear. This can be fixed by changing ownership of the data. See Archive usage page for details on how to access the PMACS archive.
- What is a PMACS account?
A PMACS Account is a user name & password combination that grants individual users access to resources managed by the Penn Medicine Academic Computing Services (PMACS) organization.
- Is the PMACS account the same as PennKey?
No. A PennKey is a user name/password combination that grants individuals access to systems managed by UPENN's central computing division and having a PennKey alone does not grant individuals access to resources managed by PMACS. However, having a valid PennKey is a requirement for requesting a PMACS account. See Requesting Accounts section below.
- Is the PMACS HPC account different from a "regular" PMACS Account?
No. A PMACS HPC account is a PMACS account which has been granted access to the PMACS HPC system resources.
- Can I share my PMACS HPC account with other members of my lab?
NO. Sharing of PMACS accounts be that for accessing the PMACS HPC system or any other resource managed by PMACS, is not permitted. Each individual is expected to have their own account. See Requesting Accounts section below to request an account.
- What is the PMACS Password policy?
All PMACS account passwords are set to expire every 180 days. Email notifications are sent to individual users 14 days prior to password expiration. PMACS passwords can be reset via the PMACS self-service password reset portal.
- Why are PMACS accounts disabled?
PMACS Accounts can be disabled for a few different reasons: 1. Removal of valid PennKey affiliation: this is standard practice when individuals change departments or if the affiliation was set with an expiration date. 2. Account was set to be disabled on specific date: this is a special case when a request to have the account disabled was made at the time of account creation. 3. PennKey expired: when individuals leave UPENN, their PennKey is set to expire.
- Why was my PMACS HPC access disrupted?
PMACS HPC access can be disrupted for the following reasons: 1. Expired passwords (see FAQ re: PMACS Password policy, above) 2. Disabled accounts (see FAQ re: why PMACS accounts are disabled, above) 3. A PI/BA no longer wishes to sponsor the individual user and requested HPC access to be revoked.
- How do I request an account on the PMACS HPC Cluster?
- Step 0 : Get a |UPENN PennKey - Step 1 : As part of our account creation process, we routinely collect several pieces of information listed here and here. Send us all this information in an email. - Step 2 : If you are not the PI, cc your PI/BA in your account request email so we can followup with them directly for email authorization. If you are the PI/BA, you don't have to do anything else.
- I requested an account, per the instructions outlined above, how long does it take to create the account?
Typically, less than 2 business days. Sometimes, emails do get missed, so feel free to nudge us again!
- OK, I got an email confirming my account. Now what?
- How do I request to close my PMACS HPC account?
There is no prescribed procedure for closing an account. This is because PMACS accounts rely on having valid PennKeys. So when an individual's PennKey expires, their PMACS HPC access is terminated. That being said, if the intent is to ensure no additional usage charges are billed for PMACS HPC access, below are the steps that need to be taken:
If the request to close an account is from the individual user (student/post-doct/staff): - Step 1 : Delete or transfer ownership of all data stored on the PMACS HPC's disk and archive systems that is not backed up elsewhere (data stored on the PMACS HPC disk or archive systems is not backed-up and cannot be restored later). - Step 2 : Send us an email requesting to disable your PMACS HPC access and cc your PI.
If the request to close an account is from a PI/BA: - Step 1 : Delete all data stored on the PMACS HPC's disk and archive systems that is not backed up elsewhere (data stored on the PMACS HPC disk or archive systems is not backed-up and cannot be restored later). - Step 2 : Send us an email requesting to disable your PMACS HPC access. - Step 3 : Ensure that there are funds to pay for the last invoice.
- How do individual users transfer ownership of data before leaving the lab/closing accounts?
Data stored within an individual's home directory is set to be owned by them and thus inaccessible to others. This data must be copied (using rsync/cp and not mv) to the lab's project directory and subsequently deleted from their own home directory.
- Before I begin using the PMACS HPC cluster, I'd like to know how much it would cost me to do my work?
Unfortunately, there is no easy answer to this question. Cost of usage varies greatly on the kind of work you do, whether or not you have a working pipeline or if you are only now building some kind of processing pipeline, the tools you use etc.
- I have a limited amount funds available, can my usage be capped once I hit a certain limit?
No, we currently have no cap usage after compute/storage costs have reached a dollar amount.
PMACS ERA Team Contact Info
- What is the best way to ask questions about the PMACS HPC System?
Send all PMACS HPC related questions/requests to our group's email: firstname.lastname@example.org
- I'm submitting a grant application and would like to included some information about computation resources available.
We have information here that you can copy-paste directly into your application.
- Do I need to acknowledge the PMACS HPC system in my publication?
Not necessarily, but a significant portion of our HPC and Archive systems was funded through a NIH grant - 1S10OD012312 NIH. So it would be great if you do acknowledge this grant (and us!).
Accessing the PMACS HPC
- How do I access the PMACS HPC cluster?
See our Wiki page about Connecting to the PMACS Cluster
- How to check the status of my jobs?
You can use the "bjobs" command Condensed output of bjobs
$ bjobs 27002288 JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 27002288 asrini RUN interactiv consign.hpc node107.hpc bash Feb 21 21:50
- How to check if my job has stalled?
First, check the error and output files associated with the job.
If error/output files for the job don't provide the necessary information, check the long listing of bjobs a few times over the course of a few minutes, to verify if additional CPU time has accrued.
To get the long listing for the the bjobs command, run "bjobs -l <jobid>" (example below):
$ bjobs -l 27007909 Job <27007909>, User <asrini>, Project <default>, Status <RUN>, Queue <normal>, Command <sleep 1000>, Share group charged </asrini> Thu Feb 22 11:31:32: Submitted from host <consign.hpc.local>, CWD <$HOME>; Thu Feb 22 11:31:32: Started 1 Task(s) on Host(s) <node041.hpc.local>, Allocate d 1 Slot(s) on Host(s) <node041.hpc.local>, Execution Home </home/asrini>, Execution CWD </home/asrini>; Thu Feb 22 11:31:34: Resource usage collected. MEM: 0 Mbytes; SWAP: 0 Mbytes; NTHREAD: 1 PGID: 19597; PIDs: 19597 SCHEDULING PARAMETERS: r15s r1m r15m ut pg io ls it tmp swp mem loadSched - - - - - - - - - - - loadStop - - - - - - - - - - - nfsops uptime loadSched - - loadStop - - RESOURCE REQUIREMENT DETAILS: Combined: select[type == local] order[r15s:pg] span[ptile='!',Intel_EM64T:32] same[model] affinity[thread(1)*1] Effective: select[type == local] order[r15s:pg] span[ptile='!',Intel_EM64T:32] same[model] affinity[thread(1)*1]
- How to terminate a job
Use the "bkill <jobid>" command without any additional flags, first:
$ bkill 27023685 Job <27023685> is being terminated $ bjobs 27023685 JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 27023685 asrini EXIT normal consign.hpc node037.hpc *eep 10100 Feb 23 12:33
If the above fails, Try "bkill -s 9 <JOBID>"; The "-s 9" option will send a SIGTERM signal/force kill signal to the JOB but the scheduler waits for confirmation that this took effect.
$ bkill -s 9 27023688 Job <27023688> is being signaled $ bjobs 27023688 JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 27023688 asrini EXIT normal consign.hpc node012.hpc *eep 10100 Feb 23 12:39
If the above fails, you can then try the "bkill -r <JOBID>" approach. This does the same as above, but the scheduler does not wait and proceeds to remove the job from the queue.
$ bkill -r 27023755 Job <27023755> is being terminated $ bjobs 27023755 JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 27023755 asrini EXIT normal consign.hpc node037.hpc *eep 10100 Feb 23 12:41
Note: Read the man page for bkill for a more detailed explanation
- How to terminate all my jobs
$ bkill 0 Job <3797180923> is being terminated Job <3797180924> is being terminated Job <3797180925> is being terminated Job <3797180926> is being terminated
Note: Read the man page for bkill for a more detailed explanation
- How do I make sure my job does not run too long?
Set a runtime limit, in minutes, for the job (a.k.a Wall-clock). This then terminates the job using bkill, when the preconfigured runtime limit has reached and the job is still running.
$ bsub -W 1 sleep 300 Job <27576257> is submitted to default queue <normal>. $ bjobs -l Job <27576257>, User <asrini>, Project <default>, Status <RUN>, Queue <normal>, Command <sleep 300>, Share group charged </asrini> Fri Mar 2 12:12:04: Submitted from host <consign.hpc.local>, CWD <$HOME>; RUNLIMIT 1.0 min of node120.hpc.local Fri Mar 2 12:12:08: Started 1 Task(s) on Host(s) <node120.hpc.local>, Allocate d 1 Slot(s) on Host(s) <node120.hpc.local>, Execution Home </home/asrini>, Execution CWD </home/asrini>; SCHEDULING PARAMETERS: r15s r1m r15m ut pg io ls it tmp swp mem loadSched - - - - - - - - - - - loadStop - - - - - - - - - - - nfsops uptime loadSched - - loadStop - - RESOURCE REQUIREMENT DETAILS: Combined: select[type == local] order[r15s:pg] span[ptile='!',Intel_EM64T:32] same[model] affinity[thread(1)*1] Effective: select[type == local] order[r15s:pg] span[ptile='!',Intel_EM64T:32] same[model] affinity[thread(1)*1]
- How do I request more than the default 6GB RAM limit for my jobs?
Use the -M <mem_in_MB> bsub option. For example, to request 10G RAM
$ bsub -M 10240 sh test_r.sh
- Why is my job in pending (PEND) state?
There can be many reasons for this. Always check the output of "bjobs -l"
$ bjobs -l 35184277 Job <35184277>, User <asrini>, Project <default>, Status <PEND>, Queue <normal> , Command <sh myjob.sh> Tue May 8 11:48:11: Submitted from host <consign.hpc.local>, CWD <$HOME>, 16 T ask(s); PENDING REASONS: Not specified in job submission: 68 hosts; Affinity resource requirement cannot be met because there are not enough processor units to satisfy the job affinity request: 8 hosts; Job slot limit reached: 11 hosts; Load information unavailable: 5 hosts; Not enough job slot(s): 26 hosts; Closed by LSF administrator: 9 hosts; Just started a job recently: 10 hosts; Not enough hosts to meet the job's spanning requirement: 2 hosts; SCHEDULING PARAMETERS: r15s r1m r15m ut pg io ls it tmp swp mem loadSched - - - - - - - - - - - loadStop - - - - - - - - - - - nfsops uptime loadSched - - loadStop - - RESOURCE REQUIREMENT DETAILS: Combined: select[type == local] order[r15s:pg] span[ptile='!',Intel_EM64T:24] same[model] affinity[thread(1)*1] Effective: -
Notice above, under "PENDING REASONS", the reason for the job to remain in pending state is listed as "not enough processor units to satisfy the job affinity request: 8 hosts;"
- How do I setup my jobs to depend/wait on other jobs?
See the section on Job Dependency
- Am I allowed to install software in my home directory?
Yes. See the Manual Software Installation section of our wiki for some pointers.
- What are the rules for software installation?
Rule 0 : All software MUST be installed on a compute node. Use an interactive session (bsub -Is bash) for all software installation. Do NOT use the head node. Rule 1 : If you get a missing library or header file error during the software installation, make sure you've read and followed RULE 0 Rule 2 : Do NOT use "sudo" in your installation steps. This will result in a permissions error and will generate an alert to us, the Admins. Rule 3 : Read the rest of this section Rule 4 : Send us a note with details on the steps you followed and exactly which interactive session you tried to install the program on so we can investigate.
- I get a "Permission denied" error when I try to install a program in my home directory by running "make install". How do I fix it?
The reason for this error is that by default most software packages are written to be compiled and installed in system level directories like /usr/bin /usr/local/bin etc. You have two options: Option 1: Compile the program with a "prefix" flag during the "configure" step:
Option 2: install the program in a different destination after the compilation is done (using home as the default; change it if you want to):
make install DESTDIR=$HOME OR make install DESTDIR=/home/<usr_name> Replace "<usr_name>" with your user name
- OK I tried one the above options to install the program in my home directory and it worked! How do I use it?
If you followed Option 1 above, the program likely was installed in a "bin" directory in your $HOME directory. First verify if the program exists there. Using (vcftools) as an example:
$ ls $HOME/bin/vcftools /home/asrini/bin/vcftools
If you followed Option 2 above, then the program was likely installed under $HOME/usr/local/bin or $HOME/usr/bin/, in addition checking $HOME/bin, check these locations as well. Again, using (vcftools) as an example:
$ ls $HOME/bin/vcftools $ ls $HOME/usr/bin/vcftools $ ls $HOME/usr/local/bin/vcftools
Once you've determined where the location of the installed binary is, you can add this to your $PATH variable for easy use. Assuming the file is $HOME/bin:
The above line can be added to your .bashrc or .bash_profile files
- OK I tried both the above options to install the program in my home directory and still get an error. Now what?
If you are trying to install a Python package, make sure you're using a Python virtual environment. See this page for details.
If not, send us a note with details on the steps you followed and exactly which interactive session you tried to install the program on so we can investigate.
Troubleshooting Job Failures
More to come soon ...
Downloading Sequencing data from BaseSpace
- Is Illumina's BaseMount software available on the PMACS HPC system so I can download sequencing data directly from BaseSpace?
Yes. BaseMount is installed on the PMACS HPC's dedicated file transfer system: mercury.pmacs.upenn.edu. Contact us for information on how to use it.
- I have sequencing data stored in Illumina's BaseSpace. How do I download the data?
You can setup the BaseSpace Sequence Hub CLI tool to download sequencing data. Information on how to do this is available here OR You can use BaseMount
Box is an excellent option for sharing data easily with yourself or other individuals. UPenn offers unlimited Box storage for all individuals with a Pennkey. Go to https://upenn.app.box.com to access your Box.
Please refer to the HPC lftp page for instructions on how push and pull files to Box from the HPC using
- Do you have Jupyter Notebook support?
Yes we do! Please refer to the Jupyter page for instructions on how to get started.