areaDetector Plugins

April 13, 2016

Mark Rivers

University of Chicago

Overview

A powerful feature of the EPICS areaDetector module is the concept of plugins. A plugin is code that is called by a driver that passes NDArray data in a callback. Plugins can be used to process array data in real time. Existing plugins do operations such as compute statistics (NDPluginStats), convert data to standard asyn arrays (NDPluginStdArrays), save data to disk (NDPluginFile), select regions-of-interest (NDPluginROI), as well as many more. New plugins can be written to perform specialized functions. Once a plugin is written it will work with any areaDetector driver. Plugins have the the following properties:

They can execute either in a blocking mode or a non-blocking mode. In the blocking mode the callback is executed by the driver callback thread. In this mode the callback is guaranteed to execute for each NDArray callback. However, it can slow down the driver, and does not utilize the multi-core capability of modern CPUs. In the non-blocking mode the driver callback simply places the NDArray data in a queue that is part of the plugin. The plugin then executes the callback code in one or more of its own threads. It removes NDArray data from the queue, processes it, and releases the data back to the NDArrayPool when it is done. In the non-blocking mode some additional memory is required for the NDArray objects that are in the queue. It is also possible to drop NDArray data if the queue is full when a callback occurs, i.e. some callback data will not be processed. The non-blocking mode can utilize the multi-core capabilities of modern CPUs because each plugin is executing in its own threads. The operation of the queue and the NDArrayPool class means that data never needs to be copied unless it is modified, each plugin has a pointer to the data which will continue to be valid until the last plugin is done with it.
They can be enabled or disabled at run time.
They can be throttled to only execute at a limited rate. This means, for example, that a detector can be saving data to disk at full speed, but images can be posted to EPICS at a reduced rate.
They can be throttled to limit their output data rate. This means, for example, that a detector can be saving data to disk at full speed, but images sent across the netword at a fixed network bandwidth. The number of images/s may vary with detector ROI, binning, and data compression.
They can be unplugged from one driver, and plugged into another driver at run time. For example, the NDPluginROI driver is itself a source of NDArray data callbacks, so a file saving plugin could be unplugged from a detector driver (where it would be saving the entire detector), and plugged into a particular ROI, where it would just save a portion of the detector. Similarly the NDPluginColorConvert plugin is also a source of NDArray data callbacks. A pipeline of plugins can be constructed, for example NDPluginColorConvert->NDPluginROI->NDPluginStdArrays. Each stage of this pipeline can be executing in its own threads, and on modern multi-core processors each can be executing on its own cores.
In non-blocking mode the maximum allowed number of threads is fixed when the plugin is created, but the actual number of threads to use can be changed from 1 to this upper limit at run time.
When there are multiple threads in use it is likely that the output NDArrays will not be in the correct order of ascending values of NDArray::UniqueId because each thread is processing asynchronously. All plugins therefore have an option to sort the output arrays before sending them. This comes at the expense of an increase in memory use and latency between the input and output arrays.
Plugins store the last input array so they can be processed again with different settings without waiting for a new input array to arrive.

NDPluginDriver

NDPluginDriver inherits from asynNDArrayDriver. NDPluginDriver is the class from which actual plugins are directly derived. The EPICS database NDArrayBase.template provides access to each of the parameters defined in asynNDArrayDriver, and the asynNDArrayDriver documentation describes that database. The NDPluginDriver class handles most of the details of processing NDArray callbacks from the driver. Plugins derived from this class typically need to implement the processCallbacks method, and one or more of the write(Int32, Float64, Octet) methods. The NDPluginDriver class documentation describes this class in detail.

NDPluginDriver defines parameters that all plugin drivers should implement if possible. These parameters are defined by strings (drvInfo strings in asyn) with an associated asyn interface, and access (read-only or read-write). The EPICS database NDPluginBase.template provides access to these standard plugin parameters, listed in the following table. Note that to reduce the width of this table the parameter index variable names have been split into 2 lines, but these are just a single name, for example NDPluginDriverArrayPort.

Parameter index variable	asyn interface	Access	Description	drvInfo string	EPICS record name	EPICS record type
Parameter Definitions in NDPluginDriver.h and EPICS Record Definitions in NDPluginBase.template
Information about this plugin
NDPluginDriver PluginType	asynOctet	r/o	A string describing the plugin type.	PLUGIN_TYPE	$(P)$(R)PluginType_RBV	stringin
asyn NDArray driver doing callbacks to this plugin
NDPluginDriver ArrayPort	asynOctet	r/w	asyn port name for NDArray driver that will make callbacks to this plugin. This port can be changed at run time, connecting the plugin to a different NDArray driver.	NDARRAY_PORT	$(P)$(R)NDArrayPort (P)$(R)NDArrayPort_RBV	stringout stringin
NDPluginDriver ArrayAddr	asynInt32	r/w	asyn port address for NDArray driver that will make callbacks to this plugin. This address can be changed at run time, connecting the plugin to a different address in the NDArray driver.	NDARRAY_ADDR	$(P)$(R)NDArrayAddress $(P)$(R)NDArrayAddress_RBV	longout longin
Queue size and status
NDPluginDriver QueueSize	asynInt32	r/w	The total queue size for callbacks when BlockingCallbacks=0. This can be changed at run time to increase or decrease the size of the queue and thus the buffering in this plugin. This changes the memory requirements of the plugin. When the queue size is changed the plugin temporarily stops the callbacks from the input driver and waits for all NDArrays currently in the queue to process.	QUEUE_SIZE	$(P)$(R)QueueSize $(P)$(R)QueueSize_RBV	longout longin
NDPluginDriver QueueFree	asynInt32	r/o	The number of free queue elements. This record goes into minor alarm when the queue is 75% full and major alarm when the queue is 100% full.	QUEUE_FREE	$(P)$(R)QueueFree	longin
NDPluginDriver QueueUse	N/A	r/o	The number of used queue elements.	N/A	$(P)$(R)QueueUse	calc
Number of threads
NDPluginDriver MaxThreads	asynInt32	r/o	The maximum number of threads that this plugin is allowed to use. This is defined when the plugin is created, and cannot be changed at run-time. Note that some plugins are not thread-safe for multiple threads running in the same plugin object, and these must force MaxThreads=1.	MAX_THREADS	$(P)$(R)MaxThreads_RBV	longin
NDPluginDriver NumThreads	asynInt32	r/w	The number of threads to use for this plugin. The value must be between 1 and MaxThreads.	NUM_THREADS	$(P)$(R)NumThreads $(P)$(R)NumThreads_RBV	longout longin
Sorting of output NDArrays
When using a plugin with multiple threads, or when the input plugin is NDPluginGather it is likely that the NDArray output will be slightly out of order, i.e. NDArray::uniqueId fields will not be monotonically increasing. This is because the threads are running asynchronously and at slightly different speeds. As a consequence a file plugin downstream of this plugin would write NDArrays to the file in the "wrong" order. Plugins have an option to sort the NDArrays by uniqueId to attempt to output them in the correct order. This sorting option is enabled by setting SortMode=Sorted, and works using the following algorithm: An std::multiset object is created to store the NDArray output pointers as they are received in NDArrayDriver::doNDArrayCallbacks. This is the method that all derived classes must call to output NDArrays to downstream plugins. This std::multiset also stores the time at which each NDArray was received by the NDArrayDriver::doNDArrayCallbacks method. This multiset is automatically sorted by the uniqueId of each NDArray. A worker thread is created which processes at the time interval specified by SortTime. This thread outputs the next array (NDArray[N]) in the multiset if any of the following are true: NDArray[N].uniqueId = NDArray[N-1].uniqueId. This allows for the case where multiple upstream plugins are processing the same NDArray. This may happen, for example, if NDPluginGather is being used and not all of its inputs are getting their NDArrays from from NDPluginScatter. NDArray[N].uniqueId = NDArray[N-1].uniqueId + 1. This is the normal case. NDArray[N] has been in the multiset for longer than SortTime. This will be the case if the next array that should have been output has not arrived, perhaps because it has been dropped by some upstream plugin and will never arrive. Increasing the SortTime will allow longer for out of order arrays to arrive, at the expense of more memory because the multiset will grow larger before outputting the arrays. When NDArrays are added to the multiset they have their reference count increased, and so will still be consuming memory. The multiset is limited in size to SortSize. If the multiset would grow larger than this because arrays are arriving faster than they are being removed with the specified SortTime, then they will be dropped in the same manner as when NDArrays are dropped from the normal input queue. In this case DroppedOutputArrays will be incremented. Note that because NDArrays can be stored in both the normal input queue and the multiset the total memory potentially used by the plugin is determined by both QueueSize and SortSize. If the plugin is receiving 500 NDArrays/s (2 ms period), and the maximum time the plugin threads require to execute is 20 msec, then the minimum value of SortTime should be 0.02 sec, and the minimum value of SortSize would be 10. It is a good idea to add a safety margin to these values, so perhaps SortSize=50 and SortTime=0.04 sec.
NDPluginDriver SortMode	asynInt32	r/w	Selects whether the plugin outputs NDArrays in the order in which they arrive (Unsorted=1) or sorted by UniqueId (Sorted=1).	SORT_MODE	$(P)$(R)SortMode $(P)$(R)SortMode_RBV	mbbo mbbi
NDPluginDriver SortTime	asynFloat64	r/w	Sets the minimum time that the plugin will wait for preceeding arrays to arrive before outputting array N when SortMode=Sorted.	SORT_TIME	$(P)$(R)SortTime $(P)$(R)SortTime_RBV	ao ai
NDPluginDriver SortSize	asynInt32	r/w	The maximum allowed size of the std::multiset. This can be changed at run time to increase or decrease the size of the queue and thus the buffering in this plugin. This changes the memory requirements of the plugin.	SORT_SIZE	$(P)$(R)SortSize $(P)$(R)SortSize_RBV	longout longin
NDPluginDriver SortFree	asynInt32	r/o	The number of NDArrays remaining before the std::multiset will not be allowed to grow larger and the plugin may begin to drop output frames.	SORT_FREE	$(P)$(R)SortFree	longin
NDPluginDriver DisorderedArrays	asynInt32	r/w	The number of NDArrays that have been output in the "wrong" order. The definition of the wrong order for NDArray[N] is that NDArray[N].uniqueId=NDArray[N-1].uniqueId or NDArray[N].uniqueId=NDArray[N-1].uniqueId+1. The reason for the equality test is explained above.	DISORDERED_ARRAYS	$(P)$(R)DisorderedArrays $(P)$(R)DisorderedArrays_RBV	longout longin
NDPluginDriver DroppedOutputArrays	asynInt32	r/w	Counter that increments by 1 each time an NDArray callback occurs when SortMode=1 and the std::multiset is full (SortFree=0), so the NDArray cannot be added to the std::multiset.	DROPPED_OUTPUT_ARRAYS	$(P)$(R)DroppedOutputArrays $(P)$(R)DroppedOutputArrays_RBV	longout longin
Callback enable, throttling, and statistics
NDPluginDriver EnableCallbacks	asynInt32	r/w	Enable (1) or disable (0) callbacks from the driver to this plugin. If callbacks are disabled then the plugin will normally be idle and consume no CPU resources. When disabling the plugin it will continue to process any NDArrays that are already in the queue.	ENABLE_CALLBACKS	$(P)$(R)EnableCallbacks $(P)$(R)EnableCallbacks_RBV	bo bi
NDPluginDriver BlockingCallbacks	asynInt32	r/w	0 = callbacks from the driver do not block; the NDArray data is put on a queue and the callback processes in one of the plugin threads. 1 = callbacks from the driver block; the callback processes in the driver callback thread.	BLOCKING_CALLBACKS	$(P)$(R)BlockingCallbacks $(P)$(R)BlockingCallbacks_RBV	bo bi
NDPluginDriver ProcessPlugin	asynInt32	r/w	NDPluginDriver maintains a pointer to the last NDArray that the plugin received. If the ProcessPlugin record is processed then the plugin runs again using this same NDArray. This can be used to change the plugin parameters and observe the effects on downstream plugins and image viewers without requiring the underlying detector to collect another NDArray. When the plugin is disabled the cached NDArray is released back to the NDArrayPool.	PROCESS_PLUGIN	$(P)$(R)ProcessPlugin	bo
NDPluginDriver ExecutionTime	asynFloat64	r/o	The execution time when the plugin processes. This is useful for measuring the performance of the plugin	EXECUTION_TIME	$(P)$(R)ExecutionTime_RBV	ai
NDPluginDriver MinCallbackTime	asynFloat64	r/w	The minimum time in seconds between calls to processCallbacks. Any callbacks occuring before this minimum time has elapsed will be ignored. 0 means no minimum time, i.e. process all callbacks.	MIN_CALLBACK_TIME	$(P)$(R)MinCallbackTime $(P)$(R)MinCallbackTime_RBV	ao ai
NDPluginDriver MaxByteRate	asynFloat64	r/w	The maximum data output rate in bytes/s. If the output rate would exceed this then the output array is dropped and DroppedOutputArrays is incremented. This can be useful, for example, to limit the network bandwidth from a plugin. For most plugins this logic is implemented in NDPluginDriver::endProcessCallbacks() when the plugin is finishing its operation and is doing callbacks to any downstream plugins. However, the NDPluginPva and NDPluginStdArrays plugins are treated differently because the output we generally want to throttle is not the NDArray passed to downstream plugins, but rather the size of the output for the pvaServer (NDPluginPva) or the size of the arrays passed back to device support for waveform records (NDPluginStdArrays). For these plugins the throttling logic is thus also implemented inside the plugin. If these plugins are throttled then they really do no useful work, and so ArrayCounter is not incremented. This makes the ArrayRate reflect the rate at which the plugin is actually doing useful work. For NDPluginStdArrays this is also important because clients (e.g. ImageJ) may monitor the ArrayCounter_RBV field to decide when to read the array and update the display.	MAX_BYTE_RATE	$(P)$(R)MaxByteRate $(P)$(R)MaxByteRate_RBV	ao ai
NDPluginDriver DroppedArrays	asynInt32	r/w	Counter that increments by 1 each time an NDArray callback occurs when NDPluginDriverBlockingCallbacks=0 and the plugin driver queue is full, so the callback cannot be processed.	DROPPED_ARRAYS	$(P)$(R)DroppedArrays $(P)$(R)DroppedArrays_RBV	longout longin
Debugging control
N/A	N/A	N/A	N/A	$(P)$(R)AsynIO	asyn

Guidelines and rules for plugins

The following are guidelines and rules for writing plugins

Plugins will almost always implement the processCallbacks() function. This function will be called with an NDArray pointer each time an NDArray callback occurs. This function will normally call the NDPluginDriver::processCallbacks() base class function, which handles tasks common to all plugins, including callbacks with information about the array, etc.
Plugins will generally implement one or more of the writeInt32(), writeFloat64() or writeOctet() functions if they need to act immediately on a new value of a parameter. For many parameters it is normally sufficient to simply have them written to the parameter library, and not to handle them in the writeXXX() functions. The parameters are then retrieved from the parameter library with the getIntParam(), getDoubleParam(), or getStringParam() function calls when they are needed.
If the writeInt32(), writeFloat64() or writeOctet() functions are implemented they must call the base class function for parameters that they do not handle and whose parameter index value is less than the first parameter of this class, i.e. parameters that belong to a base class.
Plugins will need to call the createParam() function in their constructor if they have additional parameters beyond those in the asynPortDriver or NDPluginDriver base classes.
Plugins may never modify the NDArray that they receive in the processCallbacks() function. The reason is that other plugins may be concurrently operating on the same NDArray, since each is passed the same pointer. This means also that when getting the attributes for this plugin that asynNDArrayDriver::getAttributes(pArray->pAttributeList) must not be called with the NDArray passed to processCallbacks(), because that will modify the NDArray attribute list, and hence the NDArray that other plugins are operating on. Plugins such as NDPluginROI and NDPluginColorConvert create new NDArrays via NDArrayPool::copy() or NDArrayPool::convert() (which copy the attributes to the new array) and then call getAttributes(pArray->pAttributeList) with the new array.
Plugins must release their mutex by calling this->unlock() when they do time-consuming operations. If they do not then they will not be able to queue new NDArrays callbacks or obtain new parameter values. Obviously they must not access memory locations that other threads could modify during this time, so they should only access local variables, not class variables (which includes the parameter library).
If plugins generate new or modified NDArrays then they must call NDPluginDriver::doNDArrayCallbacks() so that registered clients can get the values of the new arrays.

Base class medm screens

There are 2 medm screens for the NDPluginDriver. The first is NDPluginBase.adl. This exposes a subset of the EPICS PVs for the NDPluginDriver base class. It is normally included in the medm screen for every plugin. For example, the following is the medm screen for the NDPluginStdArrays plugin. Because this plugin does not have any additional records beyond those in the NDPluginDriver base class, this medm screen consists only the NDPluginBase.adl file.

NDPluginBase.adl displays only the PVs that are most commonly used, and does not expose the PVs that are intended more for expert configuration. NDPluginBaseFull.adl displays all of the PVs in NDPluginDriver base class, include those controlling the queue size, number of threads, output array sorting, etc. This display can be opened from the More related display menu in NDPluginBase.adl.

commonPlugins.cmd

The ADCore/iocBoot directory contains a file called EXAMPLE_commonPlugins.cmd. This file should be copied to commonPlugins.cmd and edited for site-specific requirements. commonPlugins.cmd is loaded by all of the example driver IOC startup scripts. It loads a set of plugins which are typically useful for detectors IOCs. Each detector medm screen has links to related displays for each of the common plugins. While this set of plugins is often useful and sufficient, users are free to add or remove plugins from this set for their own IOCs. New medm displays will typically need to be created if that is done, to have the required links to related displays.

The following medm screen shows the status of all of the common plugins at a glance, with links to bring up the detailed screen for each.

commonPlugins.adl

Performance example

The following example shows how increasing the number of threads from 1 to 5 in the NDPluginStats statistics plugins allows it to keep up with the simDetector running at about 485 frames/s. It also demonstrates the effect of changing SortMode=Sorted and SortMode=Unsorted.

The images were generated by simDetector generating 1024x1024 Float32 images at about 485 frames/s as shown in the following 2 medm screens.

The NDPluginStats plugin was configured to perform all of the statistics calculations (centroid, histogram, etc.) to maximize the time required to process each array, as shown in the following medm screen.

The statistics plugin was first run with just one thread, as shown in the NDPluginBaseFull.adl screen. This screen can be opened with the More related display from the NDPluginBase.adl screen, which is embedded on the left hand side of all plugin medm screens. Note the following on this screen:

The execution time is 8 ms.
The frame rate is 120 frames/s, which is consistent with the execution time.
The queue free size is 0, and the number of dropped frames is large because the plugin cannot keep up with the rate at which the simDetector is sending frames (485 frames/s).
SortedMode=Sorted is selected. Because there is only 1 thread SortMode does not really matter, the order of the output NDArrays will be the same whether SortMode is Sorted or Unsorted.

The following show the Linux "top" program when the plugin is running with 1 thread as above. Note that the STATS5_Plugin_1 thread is using almost 100% of a core. The simDetector is using about 58% of a core.

The NumThreads PV in the statistics plugin was then changed from 1 to 3, as shown in the following NDPluginBaseFull.adl screen. Note the following:

The execution time is still 8 ms.
The frame rate is now 384 frames/s, which is just over 3 times the value with 1 thread above.
The queue free size is 0, and the number of dropped frames is large because the plugin still cannot keep up with the rate at which the simDetector is sending frames (485 frames/s).
SortedMode=Sorted is selected. Because there are now 3 threads SortMode does matter, because the 3 threads could be producing output NDArrays in the wrong order. However, the number of disordered frames is still large, because dropped input frames will lead to missing values of NDArray::uniqueId on the output arrays, which is counted in the Disordered arrays.

The following show the Linux "top" program when the plugin is running with 3 threads as above. Note that there are now 3 STATS5_Plugin_N threads, each using almost 100% of a core.

The NumThreads PV in the statistics plugin was then changed from 3 to 5, as shown in the following NDPluginBaseFull.adl screen. Note the following:

The execution time is still about 8 ms.
The frame rate is now 482 frames/s, which is just over 4 times the value with 1 thread above.
The queue free size is 200, and the number of dropped frames is 0 because the plugin can now keep up with the rate at which the simDetector is sending frames (482 frames/s).
SortedMode=Sorted is selected. Because there are now 5 threads SortMode does matter, because the 5 threads could be producing output NDArrays in the wrong order. Now the number of disordered frames is 0, because there are no dropped input frames and the SortSize (50) and SortTime (0.05 sec) are sufficient to allow the output frames to be sorted without dropping any output frames (DroppedOutputArrays=0).

The following show the Linux "top" program when the plugin is running with 5 threads as above. Note that there are now 5 STATS5_Plugin_N threads, each using about 87% of a core.

To test sorting of output NDArrays the simDetector was configured to generate 100 arrays in Multiple mode, and the NDFileNetCDF plugin was configured to save 100 arrays in Stream mode. The netCDF plugin received its NDArrays from the STATS5 plugin running with 5 threads as shown above. The test was done 2 times, once with SortMode=Sorted, and then with SortMode=Unsorted. The files are were then read into IDL, using the read_nd_netcdf.pro file that can be found in ADCore/Viewers/IDL.

The following shows the output when reading the file that was written when SortMode=Sorted. attr[0].pvalue is the value of the UniqueId attribute for all 100 NDArrays. Note that the arrays are all in the correct UniqueId order.

IDL> t = read_nd_netcdf('thread_test_5_sorted_001.nc', attr=attr) 
IDL> u=*attr[0].pvalue
IDL> print, u 
      479298      479299      479300      479301      479302      479303      479304      479305      479306      479307
      479308      479309      479310      479311      479312      479313      479314      479315      479316      479317
      479318      479319      479320      479321      479322      479323      479324      479325      479326      479327
      479328      479329      479330      479331      479332      479333      479334      479335      479336      479337
      479338      479339      479340      479341      479342      479343      479344      479345      479346      479347
      479348      479349      479350      479351      479352      479353      479354      479355      479356      479357
      479358      479359      479360      479361      479362      479363      479364      479365      479366      479367
      479368      479369      479370      479371      479372      479373      479374      479375      479376      479377
      479378      479379      479380      479381      479382      479383      479384      479385      479386      479387
      479388      479389      479390      479391      479392      479393      479394      479395      479396      479397

The following shows the output when reading the file that was written when SortMode=Unsorted. Note that the arrays are not in the correct UniqueId order.

IDL> t = read_nd_netcdf('thread_test_5_unsorted_001.nc', attr=attr) 
IDL> u=*attr[0].pvalue
IDL> print, u 
      479398      479399      479400      479401      479402      479403      479404      479405      479406      479407
      479408      479409      479410      479411      479412      479414      479413      479415      479416      479417
      479418      479419      479420      479421      479423      479422      479424      479425      479426      479427
      479429      479428      479430      479432      479431      479435      479433      479434      479436      479437
      479438      479440      479439      479441      479443      479442      479446      479445      479444      479447
      479448      479449      479450      479452      479451      479453      479454      479456      479455      479457
      479459      479458      479460      479461      479463      479462      479464      479465      479466      479467
      479469      479468      479470      479471      479472      479473      479475      479474      479476      479477
      479478      479479      479480      479481      479482      479483      479484      479485      479486      479487
      479488      479489      479490      479491      479492      479493      479494      479495      479496      479497