HPC:lftp

Sharing Data

Box is an excellent option for sharing data easily with yourself or other individuals. UPenn offers unlimited Box storage for all individuals with a Pennkey. Go to https://upenn.app.box.com to access your Box.

Note: While Box offers unlimited storage to UPenn affiliates, Box limits the individual file size to 15Gbs.

Box also allows for FTP file transfers, so you can directly move files to and from the HPC to Box.

Please refer to following commands to lftp files to and from Box.

In order to remotely push/pull to box, please make sure that you have password set up for box that IS NOT your Pennkey credentials. You can find this under "Account Settings" and then "Authentication" to set up a secondary password.

lftp commands should either be run from the dedicated file transfer server mercury.pmacs.upenn.edu or from a interactive session on consign.pmacs.upenn.edu. Do NOT run file transfer operations on the head node

[obki@consign ~]$ bsub -Is bash
<< Waiting for dispatch...>>
<<Starting on node059.hpc.local>>
[obki@node059 ~]$

Suppose we have a directory /testfolder/ which contains a file "testfile"

[obki@node059 ~]$ mkdir testfolder
[obki@node059 ~]$ cd testfolder && touch testfile
[obki@node059 ~]$ pwd
/home/obki/testfolder

To push the contents of /testfolder/ to my Box, use the following command lftp -p990 -e "open [your box email]@ftp.box.com; mirror -R [path to your directory on HPC] [name of directory on box]; exit"

For example

[obki@node059 ~]$ lftp -p990 -e "open obki@upenn.edu@ftp.box.com; mirror -R /home/obki/testfolder/ boxfolder; exit"
Password:
Total: 1 directory, 1 file, 0 symlinks
New: 1 file, 0 symlinks

Replace "obki@upenn.edu" with your Box email credentials, which can be found under "Account Settings" on Box. This command will prompt you for your box password (not your pennkey password).

To pull files from box to the HPC, simply drop the "-R" flag and reverse the locations lftp -p990 -e "open [your box email]@ftp.box.com; mirror [name of directory on box] [path to your directory on HPC] ; exit"

For example

[obki@node059 ~]$ lftp -p990 -e "open obki@upenn.edu@ftp.box.com; mirror boxfolder /home/obki/testfolder/; exit"

Again, please note that the maximum file size on box is 15gbs. Push/pull operations of files larger than 15gbs will fail