sftp data transfer

SFTP (SSH File Transfer Protocol) is commonly used protocol for transferring data. There are clients available for almost any platform that one can think of. It is available for retrieving data from MAX IV (as an alternative to using Globus).

Prerequisites

  • You need to have a telephone number registered in DUO where you can receive SMS messages. This is to be able to complete the two-factor authentication login at MAX IV. To verify or change the phone number, log into DUO and go to your personal information under the Account Settings. An SMS with a security code will be sent to the number listed as your primary phone.
  • Only the directories below /mxn and /data are accessible from the sftp server. If you store any files in the /home area of the HPC clusters you must copy or move them to your /mxn/visitors area in order to transfer them to your own computer. See also the section about common directories.

Log in to the sftp server from a Linux client

  1. From Linux command line, simply type sftp 'maxiv_user_name'@sftp.maxiv.lu.se on the command line, where ‘maxiv_user_name’ is the same user name you use to log in to DUO or the beamline computers.

    You will first be asked to log in using your MAX IV credentials (password).
    Next you need to provide the “otp” code, which will be sent to your mobile number via SMS.
  2. Go to the directory that you want to transfer. This will typically be under /data/visitors/’beamline’/’your_proposal’/’date’/…

    E.g. cd /data/visitors/betamax/19801111/19810101/raw/
  3.  get -R <dir>

    this command will recursively download the entire directory “dir” to the path where you started sftp.
sftp help

Typing “help” returns a list of commands

Available commands:
bye                                Quit sftp
cd path                            Change remote directory to 'path'
chgrp grp path                     Change group of file 'path' to 'grp'
chmod mode path                    Change permissions of file 'path' to 'mode'
chown own path                     Change owner of file 'path' to 'own'
df [-hi] [path]                    Display statistics for current directory or
                                   filesystem containing 'path'
exit                               Quit sftp
get [-afPpRr] remote [local]       Download file
reget [-fPpRr] remote [local]      Resume download file
reput [-fPpRr] [local] remote      Resume upload file
help                               Display this help text
lcd path                           Change local directory to 'path'
lls [ls-options [path]]            Display local directory listing
lmkdir path                        Create local directory
ln [-s] oldpath newpath            Link remote file (-s for symlink)
lpwd                               Print local working directory
ls [-1afhlnrSt] [path]             Display remote directory listing
lumask umask                       Set local umask to 'umask'
mkdir path                         Create remote directory
progress                           Toggle display of progress meter
put [-afPpRr] local [remote]       Upload file
pwd                                Display remote working directory
quit                               Quit sftp
rename oldpath newpath             Rename remote file
rm path                            Delete remote file
rmdir path                         Remove remote directory
symlink oldpath newpath            Symlink remote file
version                            Show SFTP version
!command                           Execute 'command' in local shell
!                                  Escape to local shell
?                                  Synonym for help

SFTP clients for Windows and MacOS

Some popular SFTP clients for Windows and MacOS platforms are

  • WinSCP (Windows)
  • FileZilla (Windows, MacOS, Linux)
  • CyberDuck (MacOS, Windows)
  • PuTTY (Windows command line client is part of PuTTY installation)
  • MacOS also has the standard sftp command line program available.

Transferring files to MAX IV

It is not possible to transfer files directly from an external computer to the /mxn/ or /data/visitor areas at MAX IV. Incoming files must be placed in the /data/ingest directory:

  • You can connect either via sftp.maxiv.lu.se or the Globus server.
  • Go to the directory /data/ingest and create a subdirectory (we recommend to use your user name).
  • Transfer the data from your computer to your ingest subdirectory.
  • Log in to the one of the MAX IV HPC clusters (you cannot read the ingest directory from any other machines). If you are offsite, open a VPN connection and log in to the offline HPC cluster.
  • Move or copy the files to the appropriate location. Any files left over in the ingest directory will be automatically deleted, typically within one month.