Accessing data using sftp
SFTP (SSH File Transfer Protocol) is available for retrieving data from MAX IV (as an alternative to using Globus). Click on the links below to get information on each subject:
- You need to have a telephone number registered in DUO where you can receive SMS messages in order to complete the two-factor authentication login at MAX IV. To verify or change the phone number, log into DUO and go to your personal information under Account Settings. The phone used to send SMS must be the one listed as your primary phone.
- Only the directories below /mxn and /data are accessible from the sftp server. If you store any files in the /home area of the HPC clusters you must copy or move them to your /mxn/visitors area in order to transfer them to your own computer. See also IT environment at BioMAX (“Common directories” section).
1. From Linux command line, simply type
sftp 'firstname.lastname@example.org on the command line, where ‘maxiv_user_name’ is the same user name you use to log in to DUO or the beamline computers.
You will first be asked to log in using your MAX IV credentials.
Next you need to provide the “otp” code, which will be sent to your mobile number via SMS.
2. Go to the directory that you want to transfer. This will typically be under /data/visitors/’beamline’/’your_proposal’/’date’/…
get -R <dir>
this command will recursively download the directory “dir” to the path where you started sftp.
Typing “help” returns a list of commands
Available commands: bye Quit sftp cd path Change remote directory to 'path' chgrp grp path Change group of file 'path' to 'grp' chmod mode path Change permissions of file 'path' to 'mode' chown own path Change owner of file 'path' to 'own' df [-hi] [path] Display statistics for current directory or filesystem containing 'path' exit Quit sftp get [-afPpRr] remote [local] Download file reget [-fPpRr] remote [local] Resume download file reput [-fPpRr] [local] remote Resume upload file help Display this help text lcd path Change local directory to 'path' lls [ls-options [path]] Display local directory listing lmkdir path Create local directory ln [-s] oldpath newpath Link remote file (-s for symlink) lpwd Print local working directory ls [-1afhlnrSt] [path] Display remote directory listing lumask umask Set local umask to 'umask' mkdir path Create remote directory progress Toggle display of progress meter put [-afPpRr] local [remote] Upload file pwd Display remote working directory quit Quit sftp rename oldpath newpath Rename remote file rm path Delete remote file rmdir path Remove remote directory symlink oldpath newpath Symlink remote file version Show SFTP version !command Execute 'command' in local shell ! Escape to local shell ? Synonym for help
It is not possible to transfer files directly from an external computer to the /mxn/ or /data/visitor areas at MAX IV. Incoming files must be placed in the /data/ingest directory:
- Connect to the sftp.maxiv.lu.se server.
- Go to the directory /data/ingest and create a subdirectory (we recommend to use your user name).
- Transfer the data from your computer to your ingest subdirectory.
- Log in to the one of the MAX IV HPC clusters (you cannot read the ingest directory from any other machines). If you are offsite, open a VPN connection and log in to the offline HPC cluster.
- Move or copy the files to the appropriate destiny location. Any files left over in the ingest directory will be automatically deleted, typically within one month.