HPC:Azure Archive

We have created a container in our Azure Storage account for your lab. Access is provided via the azcopy binary available on mercury. To access your container, you have been provided with a SAS token. Keep the token secure, it provides read/write and delete authority on all files in your container.

Tape archive retirement

As of 2/1/23 we have begun uploading the contents of the tape archive to azure. Each lab that has a directory in the tape archive now has a container in the hpcarchive storage account. The process of uploading tape archive data to the azure archive is expected to take weeks, we will advise. If before the uploads have been completed, you have an urgent need to recover something from tape, please let us know. Make a systems ticket in helpdesk.pmacs.upenn.edu and we will restore the files into a subdirectory of the lab's project dir as soon as possible2. The contents of the first level of subdirectories have been tarred to maintain file attributes. A .list file with the same name as the tar is provided to get the contents of the tar.

Please Note: TIERS

To keep storage costs at their lowest all files in the archive should be set to the Archive tier as soon as they are uploaded. Note that files are automatically set to Cool upon upload. Please change tier to to Archive as soon as uploading. We will be monitoring the status of files. Only Cool tier files are available for download. Therefore, before you can download something in the archive you must set it from Archive to Cool. This can take time (depending on the size of the file). Also after downloading the file, you want to set it back to Archive ( or delete it) to ensure the lowest cost. Note that files in the Archive tier can be deleted directly and do not have to be "rehydated" first.

Examples

Here are some examples of using the azcopy command:

Here we put the access token string into a variable SAS ( e.g. SAS=`cat lab_token` ) ( the SAS token has an expiration date one year from its creation, you can see the date in the string) Be careful handling the string that is the token, it has many special characters like & and if not quoted, the shell may attempt to interpret. Here the name of the lab is in $account

List files in the container:

azcopy list "https://hpcarchive.blob.core.windows.net/${account}?${SAS}"  --properties BlobAccessTier

to upload a file ( filename ):

azcopy copy filename "https://hpcarchive.blob.core.windows.net/${account}?${SAS}"

set tier of the file to Archive:

( note the recursive, it can be done to the container and everything in it, or to folders and evereything from that path on )

to archive

azcopy set-properties "https://hpcarchive.blob.core.windows.net/${account}/${file}?${SAS}"  --block-blob-tier=archive --recursive=true

to cool

azcopy set-properties "https://hpcarchive.blob.core.windows.net/${account}/${file}?${SAS}"  --block-blob-tier=cool --recursive=true

You cannot download a file until rehydration is done, check with list command

download a fiif in cool tier):

azcopy copy "https://hpcarchive.blob.core.windows.net/${account}/${file}?${SAS}" .

( or replace . with destination path )

remove file:

azcopy remove "https://hpcarchive.blob.core.windows.net/${account}/${file}?${SAS}"

sync a directory

(not recommended, but can be used recursively copy entire directories of files instead of tar files, you will lose DATE and ownership attributes on these files ) note: you will need to do a recursive set tier on these files after the sync

azcopy sync dir "https://hpcarchive.blob.core.windows.net/${account}?${SAS}"