Open-source performance engineering
decontX Optimization
A data-representation fix and targeted concurrency turned an impractical single-cell workflow into a routine computation.
- runtime reduction
- ~99%
- observed runtime
- 12h+ → 4–5m
on a 60k × 40k dataset
01
Overview
decontX estimates contamination in single-cell expression data. On a dataset of roughly 60,000 cells by 40,000 genes, the Python implementation required more than 12 hours to reach L1 convergence, making iteration prohibitively slow.
02
Diagnosis
Profiling traced the dominant cost to a dense-to-sparse conversion inside the iterative workflow. The representation change generated unnecessary memory and compute pressure at the scale of the input matrix.
03
My contribution
I removed the avoidable dense-to-sparse path, preserved sparse data through the relevant computation, and added concurrency where work could safely be parallelized. I then tested the revised implementation on the same large single-cell workload.
04
Result
The optimized version completed all 500 iterations in approximately four to five minutes, compared with more than 12 hours to reach convergence previously—an observed runtime reduction of about 99% on that dataset.
05
Lesson
The largest gain did not come from a more complex algorithm. It came from understanding how a scientifically natural data structure interacted with memory, conversion overhead, and iteration. The work is a useful example of profiling before optimizing.
Methods