NeuNorm 2.0 Data Reduction Workflows

This directory contains detector-centric data reduction workflows for scoping NeuNorm 2.0 development.

---

Workflow Index

MARS (HFIR) - Priority 1

Workflow	Detector	TOF	Complexity
MARS CCD/CMOS	CCD/CMOS	No	Low
MARS TPX3	Timepix3	No	Medium

VENUS (SNS) - Priority 2

Workflow	Detector	TOF	Complexity
VENUS CCD/CMOS	CCD/CMOS	No	Medium
VENUS TPX1	Timepix1	Yes	Medium
VENUS TPX3	Timepix3	Yes	High
VENUS TPX4	Timepix4	Yes	High (planned)

---

Quick Reference: Key Differences

By Input Data Type

Data Type	Workflows
TIFF/FITS histogram	MARS CCD/CMOS, VENUS CCD/CMOS, VENUS TPX1
Event files (HDF5)	MARS TPX3, VENUS TPX3, VENUS TPX4

By Processing Requirements

Requirement	Workflows
Dark current correction	MARS CCD/CMOS, VENUS CCD/CMOS
Gamma filtering (critical)	MARS CCD/CMOS, MARS TPX3
Hot pixel detection	MARS TPX3, VENUS TPX3, VENUS TPX4
Beam stability correction	VENUS CCD/CMOS, VENUS TPX1, VENUS TPX3, VENUS TPX4
TOF binning	VENUS TPX1, VENUS TPX3, VENUS TPX4
Pulse reconstruction	VENUS TPX3, VENUS TPX4
Event-to-histogram	MARS TPX3, VENUS TPX3, VENUS TPX4

---

Module Requirements Summary

Shared Modules (All Workflows)

Module	Description
processing.normalizer	Compute transmission T = Sample/OB
processing.uncertainty_calculator	Error propagation
tof.pixel_detector (detect_dead_pixels)	Identify zero-count pixels
processing.roi_clipper	Apply region of interest
processing.run_combiner	Aggregate multiple acquisitions
exporters.hdf5_writer / exporters.tiff_writer	Write results (HDF5 primary; TIFF optional)

Histogram Loaders

Module	Used By
loaders.tiff_loader	CCD/CMOS, TPX1
loaders.fits_loader	CCD/CMOS, TPX1

Event Processing

Module	Used By
loaders.event_loader	TPX3, TPX4
tof.event_converter	TPX3, TPX4
tof.pulse_reconstruction	VENUS TPX3, TPX4

Corrections

Module	Used By
processing.dark_corrector	CCD/CMOS only
processing.normalizer (p_charge term)	VENUS all
filters.gamma_filter	MARS all
tof.pixel_detector (detect_hot_pixels)	TPX3, TPX4

TOF Processing

Module	Used By
tof.statistics_analyzer	TPX1, TPX3, TPX4
tof.histogram_rebinner	TPX1, TPX3, TPX4
tof.coordinate_converter	TPX1, TPX3, TPX4

---

Implementation Order Recommendation

Based on complexity and shared dependencies:

Phase 1: Core Infrastructure

1. Data models (Pydantic)

2. TIFF/FITS loaders

3. Normalizer

4. Uncertainty calculator

5. Dead pixel detector

6. Output writer

Phase 2: MARS CCD/CMOS (Simplest Complete Pipeline)

1. Dark corrector

2. Gamma filter

3. Integration: MARS CCD/CMOS end-to-end

Phase 3: MARS TPX3

1. Event loader

2. Event-to-histogram converter

3. Hot pixel detector

4. Integration: MARS TPX3 end-to-end

Phase 4: VENUS CCD/CMOS

1. Beam corrector (p_charge, ROI-based)

2. Metadata loader (DAQ)

3. Integration: VENUS CCD/CMOS end-to-end

Phase 5: VENUS TPX1

1. TOF statistics analyzer

2. Binning recommender

3. Rebinner (uniform + heterogeneous)

4. Integration: VENUS TPX1 end-to-end

Phase 6: VENUS TPX3

1. Pulse loader

2. Pulse reconstruction (JIT)

3. Integration: VENUS TPX3 end-to-end

Phase 7: VENUS TPX4

1. Validate with TPX4 data

2. Adjust loaders if needed

---

Output Summary

Workflow	Transmission	Error	TOF Edges	Dead Mask	Hot Mask
MARS CCD/CMOS	3D	3D	-	2D	-
MARS TPX3	3D	3D	-	2D	2D
VENUS CCD/CMOS	3D	3D	-	2D	-
VENUS TPX1	3D	3D	1D	2D	-
VENUS TPX3	3D	3D	1D	2D	2D
VENUS TPX4	4D	4D	1D	2D	2D

---

Document History

• Created: 2026-02-03

• Purpose: Development scoping for NeuNorm 2.0 refactoring

• Organization: Detector-centric per Jean Bilheux feedback