HPC:lftp
Sharing Data
Box is an excellent option for sharing data easily with yourself or other individuals. UPenn offers unlimited Box storage for all individuals with a Pennkey. Go to https://upenn.app.box.com to access your Box.
Note: While Box offers unlimited storage to UPenn affiliates, Box limits the individual file size to 15GB.
Box also allows for FTP file transfers, so you can directly move files to and from the HPC to Box.
Please refer to following commands to lftp files to and from Box.
In order to remotely push/pull to box, please make sure that you have password set up for box that IS NOT your Pennkey credentials. You can find this under "Account Settings" and then "Authentication" to set up a secondary password.
lftp commands should either be run from the dedicated file transfer server mercury.pmacs.upenn.edu or from a interactive session on consign.pmacs.upenn.edu. Do NOT run file transfer operations on the head node
[obki@consign ~]$ bsub -Is bash << Waiting for dispatch...>> <<Starting on node059.hpc.local>> [obki@node059 ~]$
Suppose we have a directory /testfolder/ which contains a file "testfile"
[obki@node059 ~]$ mkdir testfolder [obki@node059 ~]$ cd testfolder && touch testfile [obki@node059 testfolder]$ pwd /home/obki/testfolder [obki@node059 testfolder]$ ls testfile [obki@node059 testfolder]$ cd ~ [obki@node059 ~]$
To push the contents of /testfolder/ to my Box, use the following command
lftp -p990 -e "open [your box email]@ftp.box.com; mirror -R [path to your directory on HPC] [name of directory on box]; exit"
For example
[obki@node059 ~]$ lftp -p990 -e "open obki@upenn.edu@ftp.box.com; mirror -R /home/obki/testfolder/ boxfolder; exit" Password: Total: 1 directory, 1 file, 0 symlinks New: 1 file, 0 symlinks
Replace "obki@upenn.edu" with your Box email credentials, which can be found under "Account Settings" on Box. This command will prompt you for your box password (not your pennkey password).
To pull files from box to the HPC, simply drop the "-R" flag and reverse the locations
lftp -p990 -e "open [your box email]@ftp.box.com; mirror [name of directory on box] [path to your directory on HPC] ; exit"
For example
[obki@node059 ~]$ lftp -p990 -e "open obki@upenn.edu@ftp.box.com; mirror boxfolder /home/obki/testfolder/; exit"
Again, please note that the maximum file size on box is 15GB. Push/pull operations of files larger than 15GB will fail.
