RESEARCH CASE STUDY
Microglomeruli Segmentation
Fine-tuning a foundation model to count synaptic boutons in a fly brain, and proving it against the state of the art
M.SC. THESIS · RWTH AACHEN · 3D INSTANCE SEGMENTATION · MICROSAM / NNU-NET V2 / SWINUNETR
A reproducible deep-learning pipeline that fine-tunes a foundation model for 3D instance segmentation of synaptic boutons in confocal Z-stacks of the Drosophila mushroom body calyx, then benchmarks it head-to-head against three SOTA 3D architectures — nnU-Net v2, Cellpose, SwinUNETR — under a physically-calibrated, instance-matching evaluation protocol.
Supervised by Prof. Dr. Abigail Morrison (Software Engineering Group, RWTH Aachen), with Prof. Dr.-Ing. Johannes Stegmaier as second examiner. Built for the Tavosanis lab, who count these structures by hand today.
What it looks like
Why this is hard
Microglomeruli sit packed against each other in the calyx. Semantic segmentation would happily paint one confluent blob across a dozen of them and score well on pixel overlap while being useless: the biological question is how many boutons there are and how big each one is. That makes this an instance problem, and it makes the interesting failure mode a merge, not a miss.
Z resolution is coarser than XY. A model that reasons in voxels rather than micrometres will systematically misjudge volume along one axis, so evaluation has to happen in physical units. Instance matching here accounts for voxel volume rather than counting voxels as if they were isotropic.
There is no public benchmark for this structure at this acquisition setting. The ground truth was hand-annotated from scratch in napari, which caps how much data there is and makes the choice of a foundation model with a strong prior a practical decision rather than a fashionable one.
The people who need these counts are biologists, not PyTorch users. If the deliverable had stopped at a weights file and a table of scores, the thesis would be complete and the work would be worthless. That is why BoutonViewer is part of this project rather than a follow-up to it.
The benchmark, and why each model is in it
Four architectures, one dataset, one evaluation protocol. The dataset is 13 volumetric stacks hand-annotated from scratch in napari across seven Drosophila brain preparations and two acquisition systems (confocal LSM and Airyscan), split nine train / two validation / two test. The point of including nnU-Net v2 and Cellpose 3D is not to have them lose. It is that a fine-tuned foundation model beating a self-configuring U-Net and the community default is a claim worth making, and beating nothing is not.
A microscopy-specialised SAM variant. Strong prior, so it can be fine-tuned on a small hand-annotated set. Trained here in 2D with Kornia augmentations and custom post-processing to assemble instances.
The default in the microscopy community, and the honest baseline to beat. If a general-purpose tool already solves this, the rest of the thesis is unnecessary.
The standard against which medical and biological segmentation is measured, precisely because it removes architecture tuning as an excuse. Included so that a win is a real win.
Tests whether native 3D attention beats a 2D foundation model with a strong prior, on a dataset this size. A genuinely open question at small n.
Each trained across four preprocessing variants
originalRawUnprocessed acquisition. The control.
dogDifference-of-GaussiansClassical blob enhancement. Tests whether a hand-designed prior still buys anything once a foundation model is doing the work.
psfPSF-deconvolvedRichardson-Lucy against the measured point-spread function. Physically motivated, and expensive.
allCombinedAll variants pooled into one training set.
Four architectures times four variants is a grid, not a single run, which is why training is launched through HTCondor submit files on the LFB cluster rather than by hand.
Results
Recall is the headline metric, not accuracy or Dice, and that is deliberate: a missed bouton is a permanent counting error, while a false positive can be filtered downstream. Evaluation prioritises recall, matched-instance mIoU, and a recall-weighted panoptic quality score. Test set: matched two-image test set · 33 ground-truth boutons.
| Model | Recall | Matched mIoU | RWPQ | False pos. |
|---|---|---|---|---|
| MicroSAM Largevit_l_lm · All-PSF | 1.000 | 0.787 | 0.787 | 16 |
| MicroSAM Basevit_b_lm · All-PSF | 1.000 | 0.771 | 0.771 | 6 |
| CellposeSAM2D XYZ · All-DoG | 0.947 | 0.768 | 0.726 | 3 |
| Swin UNETRAll-PSF | 0.947 | 0.732 | 0.693 | 1 |
| nnU-NetAll-PSF | 0.895 | 0.678 | 0.609 | 2 |
| Imarissemi-manual · current lab tool | 0.735 | 0.505 | 0.371 | 5 |
The fine-tuned MicroSAM Large checkpoint recovered every bouton — perfect recall — and posted the highest matched-instance mIoU of any model evaluated, ahead of three SOTA 3D backbones and far ahead of the lab's current Imaris workflow. The Base variant matches that recall and gives up only 0.016 mIoU while cutting false positives from 16 to 6, which is why the card recommends it when GPU memory is tight.
The honest limitation, stated on the card: MicroSAM Large trades that perfect recall for a higher false-positive count (16 on the deconvolved condition), several of them large boutons split into multiple predicted components rather than spurious detections in empty regions. The card is explicit that predictions should be reviewed before unsupervised quantification — which is exactly why BoutonViewer surfaces oversized and merged predictions instead of silently filtering them. The model was fine-tuned on a comparatively small annotated set drawn from two brain specimens, so performance under substantially different imaging or labelling has not been verified.
RWPQ = recall-weighted panoptic quality (PQ_rec). Figures are the deconvolved All-PSF condition; the full evaluation set spans 61 ground-truth instances across the two held-out specimens.
Full model cardEvaluation is instance matching, not per-pixel overlap, and it weights by physical voxel volume so that anisotropic Z spacing does not quietly inflate or deflate a score. The code is in tools/evaluate_segmentation.py.
Two decisions worth defending
The obvious move on a 3D problem is a native 3D architecture, which is why SwinUNETR and nnU-Net v2 are in the benchmark. But every training example here was annotated by hand, which puts a hard ceiling on n. A model carrying a strong microscopy prior can be fine-tuned into that regime; a 3D transformer trained from a much weaker starting point has to learn more from less. Running both was the only honest way to find out which effect dominates, rather than asserting it.
BoutonViewer runs rolling-ball background subtraction plus Richardson-Lucy deconvolution for confocal LSM stacks, and lightweight normalisation for Airyscan, which is already deconvolved by the microscope. Collapsing both into one path would have been less code and quietly wrong: you would be deconvolving already-deconvolved data. Voxel calibration is auto-derived from acquisition type and image size (LSM uses the confocal pitch 0.3 × 0.0709 × 0.0709 µm; a native super-resolution Airyscan image gets the finer 0.3 × 0.0425 × 0.0425 µm), and the stats are live-recomputed if a user overrides those fields, because the failure mode of getting this wrong is a plausible number in the wrong units rather than a crash. The Base variant on LSM also skips deconvolution entirely and takes the lighter path, so a GPU-limited run stays fast without a separate code branch.
Shipping it: BoutonViewer
A napari desktop application that runs the pipeline on a confocal or Airyscan TIFF stack, shows the raw channels and predicted labels in 3D, and reports per-bouton volume and surface area in µm³ and µm². A biologist loads a file and gets a table.
- Interactive stats table: hover or click a bouton in the viewer to inspect it, or delete a false positive without touching the model
- Prediction caching, so changing a display setting does not re-run inference
- Oversized and merged predictions are deliberately not auto-removed. A silent filter would hide exactly the failure mode that matters, so the tool surfaces them and lets a human decide
- Model and data notes shipped alongside the tool: what it was trained on, at which voxel sizes, and where it should not be trusted
Reproducibility
The thesis claim is reproducibility, so the repo has to earn it. Training, inference and evaluation are separate scripts with a --dataset flag selecting the preprocessing variant, cluster jobs are committed as submit files rather than remembered, and the selected checkpoints are published on Hugging Face with the rejected experiment runs left out. Two environment definitions exist on purpose: the annotation machine runs napari tooling under mamba, the training machine runs pip, because micro_sam is not reliably installable from PyPI and pretending otherwise would break the first person who tried to reproduce this.