Inspecting Processing Results using ISPyB
The Data collect tab in ISPyB provides an interface to the data collection logs and the results from automated data processing pipelines launched after a data collection is complete. To access these records, follow the link to the processing run provided in MXCuBE3 or follow these directions to log in to ISPyB on a browser window. Select your role (user) and current proposal number. Click on the following links for directions to inspect the processing results.
When you click on the Data collect tab you will be directed to the table of sessions for your proposal. By default, they will be ordered in inverse order of creation, i.e. the most recent session will be listed first. Click on the date to access all the data collections in the session.
- The data collections for a session are displayed in inverse order of creation and ten per page. In order to see the first data collections in the session, you may have to scroll to the bottom of the page and use the navigation buttons there to display older data collections. For each data collection, the list shows a brief summary of the experiment parameters, the status of the processing results, an image snapshot (this will be weak for small oscillation angles) a crystal snapshot (if they have been collected) and a plot of the mean I/sigmaI as a function of resolution.
- The results summary indicates if a given pipeline has finished successfully (green) or not (red), and the bar indicates the data completeness. Since the weak high resolution can affect overall evaluation of the data, a red flag here may not be reason to worry. Note that in some cases, the pipeline may not have finished: this is often the case of the autoPROC pipeline, which takes a long time to run. However the EDNA and fastdp pipelines should finish in a few minutes, although the time depends on the number of images collected and number of reflections. If the results take longer to appear, you can see if there is an error by looking into the data processing directories as described below.
- Clicking on the data collection data set name directs to the detailed result for the data collection group (usually just one data set at MAX IV), unless you collect from ). A summary table with the results of processing with each of the three pipelines with and without the anomalous flag is shown by default. If you select a row in the table, you will get links to the output and log files for each of the programs run at each step of the pipeline. You can click on the files to display the contents or, if you are browsing from your own computer, you can also download the files.
- The experiment tab and beamline tab display relevant information about the data collection and beamline parameters.
- Clicking on the link to the data collection group in the navigation links on the upper part on the page, will display the summary of the results in the form of a table and also links to reports in different formats (PDF, and Word). In the results table, there is also a link to download the files in the last column of the table.
- Use the link to the current session to inspect other data collection groups in the same session.
If you would like to look at the results on disk, change the directory to /data/visitors/biomax/’proposal’/’date’ . This directory should be visible to any person listed in the proposal from any beamline computer or data processing HPC cluster. Under there, go to the process subdirectory and select the relevant subdirectories for the protein and crystal data group. The results are stored under the directories of the form xds_’crystal_’protein’_’run’_’point’ and are grouped by pipeline. If there are no results, look at the EDNA_’pid’.err file (it is always named EDNA regardless of the pipeline) to see the error. If in doubt what the error means, consult with the beamline staff.
Note that, even if the pipeline completes, you may be able to get better results if you process the data manually either at home or MAX IV. The XDS.INP file that can be found in each process subdirectory for each data set is a good starting point to rerun XDS with the same or different options. Please see the documentation on manual data reduction for more help on this topic.