Synchrotron experiments result in large amount of data. As a user you need to understand what kind of data you will receive in what formats in order to be able to analyze it.

beamtime directory structure

The data recorded at MAX IV is separated by proposal ID. The proposal ID is comprised of 8 digits the year the proposal was submitted (first 4 digits) and a running index showing it was the n-th proposal that year to be created (last 4 digits). For every beamtime you have, the corresponding data will be saved in a folder

/data/visitor/<beamline>/<proposalID>/

The next sub-folder is the date (and time) you started the beamtime visit in the YYYYMMDDHH (year, month, day, hour) format. If your beamtime proposal is split across multiple visits to the beamline on different days, then you will end up with a sub-folder for each of these dates. Likewise, if your beamtime proposal results just in one single visit for beamtime, you will have only one subfolder.

/data/visitor/nanomax/<proposalID>/<YYYYMMDDHH>/

The folder structure below that is identical for all beamtimes at NanoMAX:

/data/visitor/nanomax/<proposalID>/<YYYYMMDDHH>/raw/
                                               /process/
                                               /macros/
                                               /logs/
                                               /notebooks/
                                               /photos/

The sub-folder /raw/ will contain all raw data recorded by the beamline control software and all detectors. The folder is read-only for your DUO account and the beamline-user accounts to avoid the accidental deletion of recorded data. Within the raw folder you will find separate sub-folder for different samples. This is optional. We highly suggest to keep using this feature.

The sub-folder /process/ is where results from the analysis of the raw data is being saved. Any scripts and software running at the beamline to process the raw data from detectors will save their results there. We suggest that any scripts you write / software you use also utilize this folder, as it has write-permissions for the beamline-user account and any DUO account you connected to the scheduled beamtime.

We, the beamline staff, will use /macros/ to save scripts that were used to run the experiment and to analyze raw data. In /logs/ we will save an exported version of the electronic logbook at the end of your beamtime. In /notebooks/ we will place any jupyter notebooks used during the beamtime. In the folder /photos/ we keep any data taken with the optical microscopes or SEM in the support lab / sample preparation room. All these folders have read and write permission to any account connected to the beamtime visit.

raw-data structure

The data acquisition at the NanoMAX beamline is done using the in house developed control software ‘contrast’. It saves all data recorded data as h5 / hdf5 files. This Hierarchical Data Format is the default format at most synchrotron radiation facilities / beamlines. The internal structure of the files is facility or even instrument specific. There is libraries for loading h5 files for every major programming language. Many common software packages come with h5 support.

The control software saves one master scan file in the /raw/ folder for each scan you record. The file names of these master files are the scan number padded to 6 digits with zeros. So for example for scan #217 the corresponding file to look for would be:

000217.h5

h5 files (as the name suggests) come with an internal structure. On the beamline computers and on the MAX IV cluster you can use ‘silx view’ (free and open source) to explore the internal structure of the scan files. You will find the following:

/entry/description            # the used scan command as a string
/entry/contrast_version/      # details to the exact version of
                              # the control software used
/entry/snapshots/pre_scan/    # a list of all motor positions before 
                              # the scan was executed
/entry/snapshots/post_scan/   # a list of all motor positions after 
                              # the scan has finished
/entry/measurement/           # all the data taken during the scan

The substructure of /entry/measurement/ depends on which detectors were used during the scan, which motors were scanned and what type of scan it was. But in general you will find the motor positions of each scan point, each detectors readouts for each scan position and a relative time stamp for each scan point.

Larger detectors that create a lot of data, like the 2D pixel detectors and XRF detectors, write their own data file into the same directory as the master scan file. They follow the naming convention:

scan_<zero_padded_scan_number>_<detector_name>.hdf5

You do not have to dive into these files to read the raw data. The content of these files is linked through to the corresponding master scan file under /entry/measurement/. So it is sufficient to only open a scans master scan file to read all the recorded data of that scan.

example files

Exemplary raw data files for you to explore will be provided here in the future.