# MARS TPX3 Data Reduction Workflow **Beamline**: MARS (HFIR) **Detector**: Timepix3 (TPX3) **Beam Type**: Continuous (no TOF) **Applications**: nR (radiography), nCT (computed tomography), nGI (grating interferometry) --- ## Pipeline Flowchart ```mermaid flowchart TD subgraph Input["1. Event Loading"] A1[HDF5 Events] --> A[Load Sample Events] A2[HDF5 Events] --> B[Load OB Events] end subgraph EventConv["2. Event Conversion"] EC[Bin Events by x,y] EH[2D Histogram] end subgraph RunCombine["3. Run Combining"] RC1{Multiple Runs?} RC2[Average Histograms] RC3[Single Run] end subgraph ROI["4. ROI Clipping"] D{ROI Specified?} E[Apply ROI] F[Full Frame] end subgraph PixelDetect["5-6. Pixel Detection"] PD1[Dead Pixel Detection] PD2[Hot Pixel Detection] PM1[Dead Mask] PM2[Hot Mask] PM3[Combined Bad Pixel Mask] end subgraph Gamma["7. Gamma Filtering"] GF[Detect Gamma Spikes] GR[Replace with Median] end subgraph Norm["8. Normalization"] N["T = Sample / OB"] end subgraph UQ["9. Experiment Error"] UQ1[Poisson Statistics] UQ2[Error Propagation] end subgraph Output["10. Output"] O1[Transmission 3D] O2[Uncertainty 3D] O3[Dead Pixel Mask] O4[Hot Pixel Mask] O5[Metadata] end Input --> EC EC --> EH EH --> RC1 RC1 -->|Yes| RC2 RC1 -->|No| RC3 RC2 --> D RC3 --> D D -->|Yes| E D -->|No| F E --> PD1 F --> PD1 PD1 --> PM1 PD1 --> PD2 PD2 --> PM2 PM1 --> PM3 PM2 --> PM3 PM3 --> Gamma GF --> GR Gamma --> N N --> UQ1 UQ1 --> UQ2 UQ2 --> O1 UQ2 --> O2 PM1 --> O3 PM2 --> O4 O1 --> O5 O2 --> O5 O3 --> O5 O4 --> O5 style Input fill:#e1f5ff style EventConv fill:#ffe1f5 style RunCombine fill:#f5e1ff style ROI fill:#fff4e1 style PixelDetect fill:#ffe1e1 style Gamma fill:#ffe1cc style Norm fill:#e1ffe1 style UQ fill:#ffe1cc style Output fill:#f5e1ff ``` --- ## 1. Inputs | Input | Format | Required | Description | |-------|--------|----------|-------------| | Sample events | Event files (HDF5) | Yes | Neutron events (x, y, ToT) | | Open Beam events | Event files (HDF5) | Yes | Reference events without sample | | ROI | (x0, y0, x1, y1) | No | Region of interest to crop | **Metadata** (from files): - Acquisition time - Total event count **Key Differences from CCD/CMOS**: - No dark current correction (counting detector - no electronic baseline) - Event data → histogram conversion required - Hot pixel detection required (radiation damage causes false counts) --- ## 2. Processing Pipeline ``` ┌─────────────────────────────────────────────────────────────────┐ │ STEP 1: Load Event Data │ │ ─────────────────────── │ │ • Load Sample event files → event list (x, y, ToT) │ │ • Load OB event files → event list (x, y, ToT) │ │ • Extract acquisition metadata │ │ │ │ Note: TPX3 events include ToT (Time over Threshold) which │ │ correlates with deposited energy. At MARS, TOF not used. │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ │ STEP 2: Event-to-Histogram Conversion │ │ ───────────────────────────────────── │ │ Convert event lists to 2D histograms (no TOF at MARS): │ │ │ │ FOR each acquisition: │ │ • Bin events by (x, y) position │ │ • Sample_hist[i] = histogram(events_sample, bins=(x, y)) │ │ • Result: 2D count image per acquisition │ │ │ │ OB_hist = histogram(events_OB, bins=(x, y)) │ │ │ │ Output: Sample and OB each stacked 3D (N_image, x, y); │ │ OB reduced to (x, y) later by reference preparation │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ │ STEP 3: Run Combining (Optional) │ │ ──────────────────────────────── │ │ IF multiple runs provided: │ │ • Average histograms across runs (sum ÷ run count) │ │ • No metadata aggregation (metadata_keys_to_sum=[]) │ │ • Bad pixels detected once on the combined stack │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ │ STEP 4: ROI Clipping (Optional) │ │ ─────────────────────────────── │ │ IF ROI specified: │ │ • Crop all histogram arrays to ROI │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ │ STEP 5: Dead Pixel Detection │ │ ──────────────────────────── │ │ • Identify pixels with zero summed counts in the SAMPLE │ │ • dead_mask = (Sample_summed == 0) (sum over N_image) │ │ • Output: 2D boolean mask │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ │ STEP 6: Hot Pixel Detection │ │ ─────────────────────────── │ │ TPX3-specific: radiation damage causes false counts │ │ │ │ Detection methods: │ │ a) Statistical: pixels with anomalously high count rate │ │ on the SAMPLE, via a MAD threshold (default sigma=5.0): │ │ hot_mask = (Sample_summed > median + sigma×MAD×1.4826) │ │ b) Temporal: inconsistent counts across acquisitions │ │ c) ToT-based: events with abnormal ToT values │ │ │ │ Output: 2D boolean hot_pixel_mask │ │ │ │ Combined mask: bad_pixels = dead_mask | hot_mask │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ │ STEP 7: Gamma Filtering │ │ ─────────────────────── │ │ CRITICAL for MARS (SANS beamline contamination) │ │ │ │ FOR each histogram image: │ │ • Detect gamma spikes (outliers > threshold) │ │ • Replace with local median (3x3 neighborhood) │ │ │ │ Note: Gamma events may have distinct ToT signature - │ │ could filter at event level before histogramming │ │ │ │ Methods: │ │ a) Histogram-based: same as CCD/CMOS │ │ b) Event-level: filter by ToT before histogramming │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ │ STEP 8: Normalization │ │ ───────────────────── │ │ FOR each image i: │ │ │ │ T[i] = Sample_hist[i] / OB_hist │ │ │ │ Handle division: │ │ • Bad pixels carried as scipp masks (not NaN-filled) │ │ • Where OB_hist == 0: T = inf/nan (division artifact) │ │ │ │ Formula (no dark current subtraction): │ │ T = I_sample / I_OB │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ │ STEP 9: Experiment Error Propagation │ │ ──────────────────────────────────── │ │ Poisson statistics for counting detector: │ │ σ_sample = √(N_sample) │ │ σ_OB = √(N_OB) │ │ │ │ Error propagation for division: │ │ │ │ σ_T = T × √[ 1/N_sample + 1/N_OB ] │ │ │ │ Simplified (no dark current term): │ │ σ_T/T = √[ (σ_S/S)² + (σ_OB/OB)² ] │ │ = √[ 1/N_S + 1/N_OB ] │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────────────────────────────────────────────────────┐ │ STEP 10: Output │ │ ──────────── │ │ • Transmission: 3D array (N_image, x, y) for HDF5; │ │ TIFF renames N_image → z │ │ • Experiment Error: 3D array (same shape as Transmission) │ │ • Dead Pixel Mask: 2D boolean array (x, y) │ │ • Hot Pixel Mask: 2D boolean array (x, y) │ │ • Metadata: processing parameters, provenance │ └─────────────────────────────────────────────────────────────────┘ ``` --- ## 3. Output Specification | Output | Dimensions | dtype | Description | |--------|------------|-------|-------------| | Transmission | (N_image, x, y) | float32 | Normalized transmission values | | Experiment Error | (N_image, x, y) | float32 | Propagated uncertainty (1σ) | | Dead Pixel Mask | (x, y) | bool | True = dead pixel | | Hot Pixel Mask | (x, y) | bool | True = hot pixel (TPX3-specific) | | Metadata | dict | - | Processing provenance | **Metadata contents**: - Input file paths - Processing timestamp - Event-to-histogram binning parameters - Gamma filter parameters used - Hot pixel detection parameters - ROI applied (if any) - Number of runs combined (if any) - Software version --- ## 4. Decision Points | Step | Decision | Options | |------|----------|---------| | 2 | Event binning resolution | Native detector / Custom | | 3 | Multiple runs? | Combine or single run | | 4 | ROI needed? | Apply crop or full frame | | 6 | Hot pixel method | Statistical / Temporal / ToT-based | | 7 | Gamma filter level | Event-level / Histogram-level / Both | --- ## 5. Development Components ### Required Modules | Component | Purpose | Priority | |-----------|---------|----------| | `loaders.event_loader` | Load TPX3 event files | P0 | | `tof.event_converter` | Convert events to histogram | P0 | | `processing.run_combiner` | Aggregate multiple runs | P1 | | `processing.roi_clipper` | Apply ROI to arrays | P1 | | `tof.pixel_detector` | Identify dead pixels | P0 | | `tof.pixel_detector` | Identify hot pixels (TPX3) | P0 | | `filters.gamma_filter` | Remove gamma contamination | P0 | | `processing.normalizer` | Compute transmission | P0 | | `processing.uncertainty_calculator` | Error propagation | P0 | | `exporters.hdf5_writer` / `exporters.tiff_writer` | Write results (HDF5 primary; TIFF optional) | P0 | ### Data Models ``` EventData: - x: NDArray[uint16] # pixel x coordinate - y: NDArray[uint16] # pixel y coordinate - tot: NDArray[uint16] # Time over Threshold - metadata: Dict # acquisition info InputData: - sample_events: List[EventData] # per acquisition - ob_events: EventData - roi: Optional[Tuple[int, int, int, int]] - metadata: Dict ProcessedData: - transmission: NDArray[float32] # (N, y, x) - uncertainty: NDArray[float32] # (N, y, x) - dead_pixel_mask: NDArray[bool] # (y, x) - hot_pixel_mask: NDArray[bool] # (y, x) - metadata: Dict ``` --- ## 6. Key Differences from MARS CCD/CMOS | Aspect | CCD/CMOS | TPX3 | |--------|----------|------| | Input format | TIFF/FITS stacks | Event files | | Dark current | Required | Not needed | | Hot pixels | Not applicable | Required detection | | Gamma filter | Histogram only | Event or histogram level | | Error formula | Includes dark term | Simpler (no dark) | | Data conversion | None | Event → histogram | --- ## 7. Validation Criteria - [ ] Event-to-histogram conversion preserves total counts - [ ] Transmission values in expected range - [ ] inf/nan only at zero-denominator (OB) pixels; masks are preserved, not value-filled - [ ] Uncertainty > 0 for all valid pixels - [ ] Hot pixel mask identifies anomalous count pixels - [ ] Dead pixel mask identifies zero-count pixels - [ ] Gamma filtering removes spikes without affecting valid data