The Wire-Cell team is actively and judiciously incorporating machine learning techniques into the Wire-Cell event reconstruction paradigm to pursue better physics performance in LArTPC experiments. The stringent requirements of scientific research commonly demand thorough understanding of detector performance and data analysis methods. Therefore, we view machine learning not only as an advanced technique to be incorporated, but also a process to go through in Wire-Cell. Built upon the solid domain knowledge of LArTPC as well as the extensive experience gained in developing Wire-Cell event reconstruction and performing data analysis in LArTPC experiments, we identified several areas (TPC signal processing, neutrino vertex identification, and electron neutrino flavor identification) in which the incorporation of machine learning techniques led to immediate and significant impacts.
Fig. 1 workflow of the DNN signal processing algorithm.
Fig. 2 event displays comparing 3 signal processing algorithms.
The Liquid Argon Time Projection Chamber (LArTPC) is an advanced neutrino detector technology widely used in recent and upcoming accelerator neutrino experiments. It features a low energy threshold and high spatial resolution that allow for comprehensive reconstruction of event topology. In current-generation LArTPCs, the recorded data consist of digitized waveforms on wires produced by induced signal of drifting ionization electrons, which can also be viewed as two-dimensional (2D) (time versus wire) projection images of charged-particle trajectories. For such an imaging detector, one critical step is the signal processing that reconstructs the original charge projections from the recorded 2D images. Refer to this article for more on LArTCP signal processing.
We introduce a deep neural network (DNN) in LArTPC signal processing to improve the signal region of interest (ROI) detection. By combining domain knowledge (e.g., matching information from multiple wire planes) and deep learning, this method shows significant improvements over traditional methods. Fig. 1 shows the workflow of this algorithm. We used U-Net as the network structure. We engineered the input single channel 2D image into 3 channels, one processed with filtered deconvolution, two with geometry information baked in. The bottom on the left shows several ROI detection event displays comparing 3 algorithms, traditional algorithm as reference, DNN without geometry information and DNN with geometry information. We can see the DNN with geometry information yields the best results. The inferencing utilities are available in Wire-Cell Toolkit. Refer to this article for more on the DNN LArTCP signal processing.
Accurate identification of the primary neutrino-interaction vertex in LArTPCs is critical for the neutrino flavor identification in the charge-current interactions, which is one of the keys to measure the muon neutrino to electron neutrino oscillations. This task is challenging for traditional heuristic logic type algorithms ("Human Learning") because of the wide variety and complexity of neutrino interactions.
This neutrino vertex identification problem can be viewed as a special case of the object detection task in computer vision: region proposal. We utilized a novel regressional segmentation network to predict the distance map of each Wire-Cell 3D space point to the neutrino vertex with SparseConvNet. While the regressional segmentation network increases the training efficiency, the SparseConvNet makes the training and inferencing much faster and more scalable. The DNN neutrino vertex identification leads to a 30% increase in the electron neutrino identification efficiency with Wire-Cell in MicroBooNE. The inferencing utilities are available in the Wire-Cell Prototype.
Fig. 1 illustration of the neutrino vertex identification from 3D points.
Fig. 2 Bee event display demonstrating with improved vertex identification using DNN, the connected physics object are better segmented.
Fig. 1 illustration of the boosted decision trees.
Fig. 2 Background rejection vs. signal efficiency using three algorithms: 1) traditional box cut; 2) BDT from ROOT TMVA; 3) BDT from XGBoost.
In MicroBooNE, the original neutrino-to-cosmic ratio is about 1:20k, and only about 0.5% neutrino interactions originate from electron (anti-)neutrinos. A high-performance neutrino flavor identification, e.g. electron neutrino identification, plays an important role in probing the existence of sterile neutrinos in short-baseline neutrino experiments.
Based on “human learning” algorithms, hundreds of kinematic, topological, and calorimetric variables are developed and derived to separate different flavors of neutrino events, while the performance is unstable since the optimization of the cuts on hundreds of variables is fairly inefficient using “human learning” method, e.g. hand-scanning. In order to rapidly and robustly “learn” the features (input variables) of these final state particles from different flavors of neutrino interactions and cosmic-ray muons, the machine learning method – boosted decision trees (BDT) – are utilized to take a set of the input features and split input data recursively based on these features. BDT is a much more robust method combining many weak learners (trees) into a strong classifier by maximizing the information gain on a training sample. The neutrino flavor identification is a typical classification problem of BDT. Fig. 2 shows a comparison of the ROC curves from various implemented BDTs in Wire-Cell with the original cut-based (box cut from “human learning”) result. The “Sig Efficiency” is for good reconstructed electron neutrino charged-current events, and “bkg rejection” is for all the other background from neutral-current interactions, muon neutrino charged-current interactions, and cosmic-ray muons. It is obvious the BDT introduces a significant reduction of background events. With XGBoost (eXtreme Gradient Boosting), the BDT achieves a 99.9% background rejection rate which corresponds to a 30-fold reduction compared to the cut-based result. This enables a state-of-art electron neutrino identification in MicroBooNE with a purity of about 80%.
The application of Wire-Cell machine-learned inference is sped up by two orders of magnitude when run on GPU compared to CPU. Other computationally intensive algorithms in Wire-Cell are similarly accelerated with GPU implementations. Yet other intensive Wire-Cell algorithms are resistant to the SIMD parallelization strategy required for best GPU exploitation and are best run on CPU. Meanwhile, high performance computing facilities are more and more providing a heterogeneous mix of hardware (CPU, GPU and other accelerators). This is good for Wire-Cell software but creates a new problem: efficient use of both CPU and GPU resources. Crucially, when CPU and GPU software work together they naturally imply some ideal ratio of CPU and GPU hardware in order to keep all cores working. The natural ratio of CPU and GPU depend on the particular type of processing payload. New development will lead to new natural ratios and existing ratios will evolve as CPU software is ported to GPU implementation. Current Wire-Cell jobs require order 100 CPU cores in order to provide one GPU with enough work to avoid idling. A method for running heterogeneous code on heterogeneous hardware that can adapt natural CPU/GPU ratios to ratios provided by the hardware is clearly needed. ZIO provides the infrastructure for solving the ratio-matching problem.
ZIO brings to offline production processing some of the technology which is more commonly found in distributed software-based data acquisition systems (an area that ZIO also supports). ZIO provides software and network protocols to connect disparate applications including those based on the Wire-Cell Toolkit and hardware by leveraging the power of the ZeroMQ software ecosystem. In particular, ZIO provides an extended version of the ZeroMQ Majordomo Protocol called Generaldomo which allows for multiple multi-threaded software contexts to connect to a broker while, independently, a number of GPU "worker" applications also connect to provide a particular service. This pattern is what allows for the hardware CPU/GPU ratio to be tuned to match any potentially desired software CPU/GPU ratio. If a GPU is too idle, it may be shared among more CPU-bound jobs, additional GPU services may join in order to alleviate a family of GPU-bound jobs and in general, any particular CPU/GPU ratio may be achived.
In this multi-host/process/thread environment, safe access to file resources is critical. ZIO provides for this in part with an implementation of the ZeroMQ pattern credit based flow control. It allows high throughput data transfers between contexts in a way that bounds the memory usage of buffers and assures reliable data streaming. On top of this protocol, ZIO provides a distributed file I/O method allowing multiple threads, processes and computers to share a common file resource. Initial support for the HDF5 file format exists and other file formats popular in the space, such as used by ROOT may be added as needed.
Finally, as distributed processing consists of multiple processes on multiple computers working together, ZIO provides a means to simplify forming their interconnections. To do this ZIO uses the ZeroMQ ZRE (Zyre) discovery protocol. It allows one to launch multiple processes which may then automatically "find each other" and connect over the network (or locally via IPC or shared memory queues) based on simple, abstract rules. This saves the production process manager from having to intricately specify every address and port.
Fig. 1 High-level Wire-Cell/ZIO architecture and integration.
Fig. 2 flowchart of using Wire-Cell/ZIO in DNN signal processing.
Fig. 3 DNN signal processing CPU and memory usage processing one sample w/o ZIO.