HPC:Rclone
This page describes the use of Rclone, a command line program to sync files and directories to and from various cloud-based storage services.
Rclone
Rclone is a command line program to sync files and directories to and from cloud-based storage services like:
- Amazon S3
- Box
- Dropbox
- Google Cloud Storage
- Google Drive
- Microsoft Azure Blob Storage
- Microsoft OneDrive
to name a few. The full list of supported services is available here.
Usage
Below are instructions to configure Rclone to sync data between the HPC and UPENN Box. A similar approach can be taken for configuring Rclone to sync data between other supported services. See the main Rclone site and adapt those instructions based on the instruction set below.
1. On the HPC, launch an interactive job:
bsub -Is bash
2. Setup a new rclone remote :
rclone config
# Within rclone
n
name> box
#(you can call it something else if you want)
Storage> box
Box App Client Id - leave blank normally.
client_id>
Box App Client Secret - leave blank normally.
client_secret>
Box App config.json location
Leave blank normally.
Enter a string value. Press Enter for the default ("").
config_json>
'enterprise' or 'user' depending on the type of token being requested. Enter a string value. Press Enter for the default ("user").
box_sub_type>
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> n
# IMPORTANT: Set the above to "n" for and leave the terminal open
3. Now, on your Windows/macOS desktop/laptop (after you installed rclone on your desktop/laptop):
Open a command prompt and type
rclone.exe authorize "box"
The above should open up a browser on your desktop and when prompted, use the username/password you have, login and then click "Grant access".
This should print a very long json string in your Windows/macOS terminal. Something that looks like:
{"access_token":"BlahBlahasdkfjalkwejrijBlahBlahBlah",,"token_type":"bearer","refresh_token":"MoreBlahBlahBlah","expiry":"2019-12-06T07:01:07.7459684311-05:00"}
4. Copy the entire string including curly braces and paste it within your terminal on the HPC (step 2. above)
5. Test (assuming on the UPENN+Box side, the files are within "DIR_ON_BOX/MY_BOX_FILES"
rclone ls box:DIR_ON_BOX/MY_BOX_FILES
To copy files to Box:
rclone ls box:DIR_ON_BOX/MY_BOX_FILES rclone copy DIR_ON_HPC box:DIR_ON_BOX/MY_BOX_FILES
Invalid refresh token
According to the docs tokens are valid only for a predetermined period of time. See the Rclone documentation for information on how this can be addressed.
