The HDF5 File Writer can be configured to save image frames into separate datasets within the same HDF5 data file. The selection of frame type (image frame, background/dark frame, white/flat frame) is made by use of an existing PV in area detector: $(P):cam1:FrameType
which is an mbbo record. In ophyd, the readback version of this PV: $(P):cam1:FrameType_RBV
is used to define operational values for the ophyd device. Be sure to configure the readback PV with the same values.
Both these PVs ($(P):cam1:FrameType
and $(P):cam1:FrameType_RBV
) should be configured so the values on the RHS are the HDF5 address in the file where the image frame should be stored:
PV.ZRST = /exchange/data
PV.ONST = /exchange/data_dark
PV.TWST = /exchange/data_white
# make sure to put these in autosave so they are restored when the IOC is restarted!
The HDF5 addresses shown are for the Data Exchange format. In NeXus, the detector data is stored within the instrument group, traditionally at /entry/instrument/detector/
. We then hard link that into the /entry/data
group. But, there is a big, fat, stinkin problem. The EPICS mbbo fields for the various string values (ZRST, ONST, TWST, ...) are only 25 characters long, not long enough to hold this path. So we write the data into the /entry/data
group and hard link it (in the layout.xml file) to the instrument group.
PV.ZRST = /entry/data/data
PV.ONST = /entry/data/dark
PV.TWST = /entry/data/white
# make sure to put these in autosave so they are restored when the IOC is restarted!
The area detector attributes XML file needs the selection PV included in its list. We'll call it SaveDest
so we can use the same name in the layout file:
<Attribute
name="SaveDest"
type="EPICS_PV"
source="13SIM1:cam1:FrameType"
dbrtype="DBR_STRING"
description="image, dark, or flat frame"/>
Then, the HDF5 layout XML file refers to this SaveDest
attribute in the setup (add this to the XML file just after the opening hdf5_layout
and before the first group
element)
<global name="detector_data_destination" ndattribute="SaveDest" />
The name detector_data_destination
is hard-coded in the source code of the HDF5 file writer.
In the BlueSky plan, write the frame type with 0: image, 1: dark, 2: flat before acquiring the image frame of that type. Then the HDF5 file writer will direct the image frame to the correct dataset as specified by the ZRST, ONST, or TWST field, respectively.
First, configure an instance of the sim detector.
from ophyd import Component, Device, EpicsSignal, EpicsSignalRO, EpicsSignalWithRBV
from ophyd import SingleTrigger, AreaDetector, SimDetector
from ophyd import HDF5Plugin, ImagePlugin
from ophyd.areadetector import ADComponent
from ophyd.areadetector.filestore_mixins import FileStoreHDF5IterativeWrite
image_file_path = "/tmp/simdet/%Y/%m/%d/"
_ad_prefix = "13SIM1:"
class MyHDF5Plugin(HDF5Plugin, FileStoreHDF5IterativeWrite):
"""
"""
layout_filename = ADComponent(EpicsSignal, "XMLFileName")
layout_filename_valid = ADComponent(EpicsSignal, "XMLValid_RBV", string=True)
class MySingleTriggerHdf5SimDetector(SingleTrigger, SimDetector):
image = Component(ImagePlugin, suffix="image1:")
hdf1 = Component(
MyHDF5Plugin,
suffix='HDF1:',
root='/', # for databroker
write_path_template=image_file_path, # for EPICS AD
)
adsimdet = MySingleTriggerHdf5SimDetector(_ad_prefix, name='adsimdet')
adsimdet.read_attrs.append("hdf1")
Setup Bluesky
from bluesky import RunEngine
from bluesky.utils import get_history
import bluesky.plans as bp
RE = RunEngine(get_history())
Loading metadata history from /home/mintadmin/.config/bluesky/bluesky_history.db
Set the counting time per frame to something short.
adsimdet.cam.stage_sigs["acquire_time"] = 0.02
Now, count adsimdet
using default settings.
RE(bp.count([adsimdet]))
('4e811b82-c19c-4e85-b5c9-1a0fdf3d4b71',)
Looking inside the HDF5 file, did not find the FrameType PV.
adsimdet.cam.nd_attributes_file.value
'simDetectorAttributes.xml'
We'll add the PV to this file (which is located in the IOC's startup directory) and then reload that file into the EPICS AD.
adsimdet.cam.nd_attributes_file.put("simDetectorAttributes.xml")
Count again and check if the PV is now in the HDF5 file.
Check that the attributes XML file was read and everything was OK with it. This is not in the ophyd device so we'll make a local signal to check this. Fix any errors before proceeding.
att_status = EpicsSignal("13SIM1:cam1:NDAttributesStatus", name="att_status", string=True)
att_status.value
'Attributes file OK'
RE(bp.count([adsimdet]))
('ababddb6-adaf-40bf-8d81-60c4f4aa080c',)
note: took two image captures to see the attributes ... hmmm
We've been using the default layout. We'll need our own so that we can customize it.
In the IOC directory, copy the hdf5_layout_nexus.xml
file (this is the file we have been using as a default) to a new file called layout.xml
and tell the HDF5 file write to use it. Then, check that it was read and the file content was acceptable.
adsimdet.hdf1.layout_filename.put("layout.xml")
adsimdet.hdf1.layout_filename_valid.value
'Yes'
Try it and verify SaveDest
is present in the output file.
RE(bp.count([adsimdet]))
('b0d92bb3-bf3b-48ff-9ebf-cffe0560254c',)
Configure our PV(s) for the NeXus addresses we want to use. We'll use PyEpics here since that will look more obvious. Still, we must reconnect our ophyd object after our change to the EPICS PVs to pick up this change.
import epics
# this is the PV we use as the `SaveDest` attribute
epics.caput("13SIM1:cam1:FrameType.ZRST", "/entry/data/data")
epics.caput("13SIM1:cam1:FrameType.ONST", "/entry/data/dark")
epics.caput("13SIM1:cam1:FrameType.TWST", "/entry/data/flat")
# ophyd needs this configuration
epics.caput("13SIM1:cam1:FrameType_RBV.ZRST", "/entry/data/data")
epics.caput("13SIM1:cam1:FrameType_RBV.ONST", "/entry/data/dark")
epics.caput("13SIM1:cam1:FrameType_RBV.TWST", "/entry/data/flat")
# re-connect the detector object to pick upthese changes
adsimdet = MySingleTriggerHdf5SimDetector(_ad_prefix, name='adsimdet')
adsimdet.read_attrs.append("hdf1")
We also need to modify our layout file, adding datasets in the right places for the additional image types. Here's layout.xml
after those edits are complete:
<?xml version="1.0" standalone="no" ?>
<hdf5_layout
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="../../../../../ADCore/XML_schema/hdf5_xml_layout_schema.xsd"
>
<global name="detector_data_destination" ndattribute="SaveDest" />
<group name="entry">
<attribute name="NX_class" source="constant" value="NXentry" type="string" />
<attribute name="default" source="constant" value="data" type="string" />
<group name="data">
<attribute name="NX_class" source="constant" value="NXdata" type="string" />
<dataset name="data" source="detector">
<attribute name="units" source="constant" value="counts" type="string" />
<attribute name="description" source="constant" value="image frame(s)" type="string" />
<attribute name="signal" source="constant" value="1" type="int" />
<attribute name="target" source="constant" value="/entry/data/data" type="string" />
</dataset>
<dataset name="dark" source="detector">
<attribute name="units" source="constant" value="counts" type="string" />
<attribute name="description" source="constant" value="dark (background) frame(s)" type="string" />
<attribute name="target" source="constant" value="/entry/data/dark" type="string" />
</dataset>
<dataset name="flat" source="detector">
<attribute name="units" source="constant" value="counts" type="string" />
<attribute name="description" source="constant" value="flat (white) frame(s)" type="string" />
<attribute name="target" source="constant" value="/entry/data/flat" type="string" />
</dataset>
</group> <!-- end group data -->
<group name="instrument">
<attribute name="NX_class" source="constant" value="NXinstrument" type="string" />
<group name="detector">
<attribute name="NX_class" source="constant" value="NXdetector" type="string" />
<hardlink name="data" target="/entry/data/data" />
<!-- The "target" attribute in /entry/data/data is used to
tell Nexus utilities that this is a hardlink -->
<hardlink name="dark" target="/entry/data/dark" />
<hardlink name="flat" target="/entry/data/flat" />
<group name="NDAttributes">
<attribute name="NX_class" source="constant" value="NXcollection" type="string" />
<dataset name="ColorMode" source="ndattribute" ndattribute="ColorMode">
</dataset>
</group> <!-- end group NDAttribute -->
</group> <!-- end group detector -->
<group name="NDAttributes" ndattr_default="true">
<attribute name="NX_class" source="constant" value="NXcollection" type="string" />
</group> <!-- end group NDAttribute (default) -->
<group name="performance">
<dataset name="timestamp" source="ndattribute" />
</group> <!-- end group performance -->
</group> <!-- end group instrument -->
</group> <!-- end group entry -->
</hdf5_layout>
Then, reload (and test) the XML layout file.
adsimdet.hdf1.layout_filename.put("layout.xml")
adsimdet.hdf1.layout_filename_valid.value
'Yes'
Try it and check the output file. This won't be a great test since we are only writing one image type. Just check that it seems to work.
RE(bp.count([adsimdet]))
('75293b34-b3f2-482e-8852-6434dfc04da9',)
Try this series: 4 images, 3 darks, 2 flats
import bluesky.plan_stubs as bps
import time
def frame_set(det, frame_type=0, num_frames=1):
print("type {}, frames {}".format(frame_type, num_frames))
yield from bps.mv(det.cam.frame_type, frame_type)
for frame_num in range(num_frames):
print("acquire {} frame {} of {}".format(frame_type, frame_num+1, num_frames))
yield from bps.mv(det.cam.acquire, 1)
while det.cam.acquire.value != 0:
# wait for acquisition to finish
yield from bps.sleep(0.01)
print("acquired")
def series(det, num_images=4, num_darks=3, num_flats=2):
num_frames = [num_images, num_darks, num_flats]
total = sum(num_frames)
print("total frames:", total)
print("setup")
yield from bps.mv(
det.hdf1.num_capture, total,
det.hdf1.file_write_mode, 'Capture',
det.cam.image_mode, "Multiple",
)
yield from bps.abs_set(
det.hdf1.capture, 1,
)
for i, num in enumerate(num_frames):
yield from frame_set(det, frame_type=i, num_frames=num)
print("restore")
yield from bps.mv(
det.hdf1.num_capture, 1,
det.hdf1.file_write_mode, 'Single',
det.cam.image_mode, "Single",
det.cam.num_exposures, 1,
det.cam.frame_type, 0,
)
RE(series(adsimdet))
total frames: 9 setup type 0, frames 4 acquire 0 frame 1 of 4 acquired acquire 0 frame 2 of 4 acquired acquire 0 frame 3 of 4 acquired acquire 0 frame 4 of 4 acquired type 1, frames 3 acquire 1 frame 1 of 3 acquired acquire 1 frame 2 of 3 acquired acquire 1 frame 3 of 3 acquired type 2, frames 2 acquire 2 frame 1 of 2 acquired acquire 2 frame 2 of 2 acquired restore
()
#RE.abort()
#adsimdet.cam.acquire.put(0)
#adsimdet.hdf1.capture.put(0)
#adsimdet.cam.frame_type.put(0)
Final HDF5 data file looks like this (for clarity, no attributes or array data shown):
entry:NXentry
data:NXdata
dark:NX_UINT8[3,19,33] = [ ... ]
data:NX_UINT8[4,19,33] = [ ... ]
flat:NX_UINT8[2,19,33] = [ ... ]
instrument:NXinstrument
NDAttributes:NXcollection
AcquireTime:NX_FLOAT64[9] = [ ... ]
AttributesFileNative:NX_INT8[9] = [ ... ]
AttributesFileParam:NX_CHAR[256] = simDetectorAttributes.xml
AttributesFileString:NX_CHAR[256] = simDetectorAttributes.xml
CameraManufacturer:NX_CHAR[256] = Simulated detector
CameraModel:NX_CHAR[256] = Basic simulator
E:NX_FLOAT64[9] = [ ... ]
Gettysburg:NX_CHAR[256] = Four score and seven years ago our fathers
ID_Energy:NX_FLOAT32[9] = [ ... ]
ID_Energy_EGU:NX_FLOAT32[9] = [ ... ]
ImageCounter:NX_INT32[9] = [ ... ]
MaxSizeX:NX_INT32[9] = [ ... ]
MaxSizeY:NX_INT32[9] = [ ... ]
NDArrayEpicsTSSec:NX_UINT32[9] = [ ... ]
NDArrayEpicsTSnSec:NX_UINT32[9] = [ ... ]
NDArrayTimeStamp:NX_FLOAT64[9] = [ ... ]
NDArrayUniqueId:NX_INT32[9] = [ ... ]
Pi:NX_FLOAT64[9] = [ ... ]
RingCurrent:NX_FLOAT32[9] = [ ... ]
RingCurrent_EGU:NX_FLOAT32[9] = [ ... ]
SaveDest:NX_CHAR[256] = /entry/data/data
Ten:NX_INT32[9] = [ ... ]
timestamp:NX_FLOAT64[9,5] = [ ... ]
detector:NXdetector
dark --> /entry/data/dark
data --> /entry/data/data
flat --> /entry/data/flat
NDAttributes:NXcollection
ColorMode:NX_INT32[9] = [ ... ]
performance