Remote experiments at BioMAX
Because of the situation concerning COVID-19 we encourage BioMAX users to collect data remotely. This document describes the software required for remote connections and the current requirements for remote experiments. Click on each of the links below for information on each topic. If you need more information or help to determine if your experiment is eligible for remote access, please contact the beamline staff.
Because BioMAX is still under development and new user features are being implemented regularly, it is strongly advised that remote users have collected data at MAX IV, on site or remotely, at least once during the previous year. Remote users should be familiar with the correct procedures to ship samples to and from BioMAX, with MXCuBE3, ISPyB and the computing environment at MAX IV, and tools for handling the data and for remote access. New and non-frequent users should communicate in advance with staff to arrange training on those subjects before their beamtime (e.g. in a BAG training session or equivalent). For the time being, all the training sessions will be remote.
Fully remote access implies that the users must be able to mount and dismount samples, collect, process and backup the data without being present at MAX IV. This requires using the sample exchanger to mount and dismount the samples, which must be mounted on spine pins of the standard length and loaded in UniPucks. Standard data collection at 100 K is readily supported, and all the beamline functions and collection modes available through MXCuBE3 are also accessible during remote experiments. Data must be transferred through the internet trough SFTP or Globus only: copying data to external disks is not supported for remote users. Room temperature and serial crystallography experiments are not included in the remote program for the foreseeable future and neither are experiments from hazardous samples or any experiment that requires special instrumentation or setup. Regardless of the experiment, remote access may be granted to eligible collaborators outside MAX IV as long as one or more members of the proposal are present at the beamline.
Support staff will load and unload the UniPucks into the sample exchanger dewar and transport the dewar to the outshipping area, but will not be participate in the experiment beyond normal support duties (eg, providing basic scientific support and solving beamline issues which affect data collection). Any extensive involvement of staff in tasks which would be normally be done by users (transfer samples shipped in ESRF pucks, manual sample mounting or dealing with sample issues) should be agreed before hand and treated as a collaboration.
MAX IV users can access a dedicated BioMAX remote desktop after establishing a VPN connection to the beamline network. Users who have already installed the Pulse Secure VPN client (Linux users may be able to use recent versions of OpenConnect using the ‘pulse’ protocol instead of Pulse Secure) and the Cendio ThinLinc client to connect to the off-line cluster for data processing do not need to install any additional software. If this is not the case, follow these one time steps:
- Ensure that you enter a cell phone number in DUO where you can receive an SMS. This must be the primary phone number. When you connect to MAX IV, you will receive an otp (“one time password”) for two-step authentication.
- On a web browser, open http://vpn-white.maxiv.lu.se. Type your DUO username and password to log in. and select the realm DUO User. Then type in the otp code sent to your phone when the prompt appears.
- Download the Pulse Secure VPN application and install it on your machine. For Windows, you simply need to execute a launcher script. For Mac and Linux (Centos and Rhel), you must download the appropriate packages. Regardless of the operation system, you will need administrative permission to install the software.
- Next, download the Thinlinc client and install it.
- Test that the software installation has been successful by connecting to the offline data processing cluster front end, which is available all the time to all users listed in an active proposal:
- Open the Pulse Secure application and connect to the vpn-white server; when prompted, select the DUO User realm, enter your DUO user name and password and the otp.
- Connect to the Thinlinc client offline-fe1 by following these instructions. If the connection fails, see the “Troubleshooting” section below.
To be able to collect data at the beamline, you need to connect to the beamline network a different VPN server. Access to the beamline VPN network is granted automatically at the start of the beamtime, provided that you are listed in the DUO experimental session.
To configure the Pulse Secure client for the beamline network:
- Open the Pulse Secure client and click on the + sign to add a new connection. Choose any name for the connection (e.g., “BioMAX”). The name of the server is vpn-blue.maxiv.lu.se. Ensure that there are no connections open to any other server, including the VPN white network.
- When you click on the Connect button on the beamline network, you will get a long list of “Realms”. Scroll down to find BIOMAXremoteUser and click on the name to select it. Check the Save settings button if you want to store the connection details for future connections.
- Enter your DUO user name and password. After authenticating, type the otp code.
Once you are connected to the beamline VPN, open the Thinlinc client, and connect to the server biomax-remote.
For remote experiments, ship the dewar containing UniPucks only to MAX IV as usual. It is advisable to ship the dewar to get to MAX IV one or two days before the experiment. Please ensure that you enclose the return paperwork in the dewar, including the ISPyB generated labels, the courier return labels and, for shipments outside the EU, the proforma documentation. The beamline staff will load the UniPucks in the dewar and let you know their position so that you can enter the information in ISPyB. Staff will also inform you of when the hutch is searched and you can start the experiment. They will also provide you with contact information in case you have questions during the experiment or run into problems.
To start the experiment, open the connection to the beamline network in Pulse Secure. Then, open Thinlinc, enter the beamline server name (biomax-remote) and your log in credentials. The remote desktop is configured to look like the workstations at BioMAX and is part of the same computing environment. Links to start MXCuBE3 and ALBULA (to view diffraction images) are available as icons on the desktop. The icon named Monitor Beam provides a live plot of the beam position in the experiment hutch. See the MXCuBE documentation for more details.
Note: If the remote server is the first machine ever you log in at MAX IV under you user account, there will be no MXCuBE or other experiment software icons until you open a terminal. This will run a login script and set up the desktop environment. Next time you log in to a BioMAX machine (remote or at the beamline) you will see the icons.
To run MXCuBE3, we recommend using a screen of at least 1680×1050. Otherwise the GUI will not fit in the window, and it can become difficult to access all the buttons and links. The bottom right side of the GUI has a icon that links to video feeds from the BioMAX hutch. You can select between a wide angle view of the beamline, useful to monitor the sample mounting robot, or a closer view of the diffractometer. Because remote access enables participation in the experiment from collaborators at different locations, MXCuBE also supports features for multiuser access. Please see the MXCuBE remote access documentation.
The data will be stored in the usual location and you will be able to display the autoprocessing results on ISPyB either via the remote server or a browser on your computer. If you wish to process the data manually during the experiment, connect to the online HPC cluster from the biomax-remote server as described in the documentation. Note that you will not be able to open a connection to the offline HPC on the same machine as the remote beamline server; you can only have one VPN connection.
If you lose the connection temporarily, you will be able to reconnect to the Thinlinc session and resume the experiment; connection outages do not affect ongoing data collections or beamline tasks. Also, you can conduct the experiments at different locations (eg, you can start at work and then continue from home). If you do not log out from the Thinlinc session, you will be able to reconnect to it from any location. However, MXCuBE3 does not allow the same user to be logged in twice, so if you try to log in twice (for example, both from the beamline and remotely) you will not be able to connect unless you force-quit the first MXCuBE instance from the log in window.
When you finish the experiment, please log out from MXCuBE and from the remote server Thinlic session and let the beamline support staff know (outside working hours please use email or send a message). Staff will unload the samples and, provided that all the required paperwork is present, take the dewar to the out shipping area.
Problems with the VPN connection
- After installing the VPN client, the first time connection may fail. Reboot the machine if this happens.
- Double check that you are logging in with the correct credentials; the login is case-sensitive. If at some point you save your credentials in the connection and then change the password, you will have to create a new connection to be able to log in again.
- If you do not get an otp, make sure that the phone in your DUO profile can receive messages, and that it is in the correct format (+ sign preceding the country code and then the number)
- If you can connect to the office network vpn-white, but not the beamline network vpn-blue, check that you are included in the current beamline session; note that the permission to access the beamline network is not updated inmediately, so if you have been added to the session just before or after the beamtime starts, it can take up to half an hour until you are actually enabled to open the VPN connection. Also make sure that you select the BIOMAXremoteUser realm, not the BIOMAX realm (which is reserved for staff).
- Make sure that you do not have another VPN connection on the same computer.
- The Pulse Secure client checks for updates so at some point you will get a message about updating the application after you log in. Accepting the message will terminate your current session; it is OK to ignore this message and do the update later.
Problems with the Thinlinc server
- If the server name cannot be resolved, check that the VPN connection is still on and that you are trying to log in to the correct server (eg, if you are connected to the vpn-white server, you will not be able to log in to the biomax-remote server. Sometimes the problem can be solved by typing the entire domain name for the server. e.g. biomax-remote.maxiv.lu.se
- If you the connection fails to open an SSH tunnel to the server, you may have a corrupted old session running. Click on the “End existing session” checkbox.
- If the server has run out of licences, contact staff.
- The Thinlinc window is maximized by default. If this is a problem, use the F8 key to change this behaviour in the Thinlinc menu.
- If you close the Thinlinc window by mistake, simply log in again. You should be reconnected to the old session, as long as the “End existing session” checkbox is unchecked.
Problems during the experiment
There are some recurrent issues at MXCuBE commonly observed by users, documented here. As a remote user you may be more likely to fail to click on some menu buttons (eg, the Remote access menu, or the signout button), if you are running your remote session on a laptop or desktop with a small screen. Resizing the browser window should solve this.
The remote desktop has four working spaces by default (they are displayed, iconized, at the right bottom part of the Thinlinc window). This is useful because you can open different programs in different spaces (eg, you can run MXCuBE, Albula and ISPyB in different spaces to make more room for each GUI). If you drag a window off to a side you may move it to the adjacent space by accident. Double check this before trying to reopen or restart the window.
If the entire GUI (MXCuBE or other) becomes irresponsive, check other windows in the remote desktop and on your local computer before contacting staff. If your computer crashes or you lose the connection, you should be able to reconnect to the remote session again.