Benchmarks: Python vs R¶

Both implementations ran every tutorial on the same machine: a shared 112-core Intel Xeon Gold 6258R server, x86-64, Linux. Each run used a fresh Docker container (docker/Dockerfile for Python, docker/Dockerfile.r for R 4.4.3 + NMF 0.28) with 32 parallel jobs (R: Number of threads, Python: n_jobs) and one BLAS/OpenMP thread per process. Python timings include numba’s just-in-time compilation (cold cache).

2,608of 2,608 output files discrete-equal to R

61%of files byte-identical

8e−11largest relative difference

1.2–7.4×Python faster than R (8 tutorials timed)

Wall time

Same server, 32 parallel jobs, one BLAS thread per process. Log scale; hover a dot for the value.

R (EcoTyper, R 4.4.3)Python (exact, default)Python fast mode

Peak memory

Summed over all processes in the container: PSS for Python (pages shared by forked workers counted once), RSS for R (separate processes).

R (EcoTyper, R 4.4.3)Python (exact, default)

Agreement with R

Largest relative difference of any numeric cell, per tutorial (log scale). Every discrete output is identical.

Results as tables¶

Tutorial	R wall	Python wall	R / Python	Python fast wall	R / fast	R peak RSS	Python peak PSS (RSS)
T1 · Bulk recovery, lung	2.4 min	20 s	7.41×	48 s	3.04×	11.5 GB	1.4 GB (3.7 GB)
T1 · Bulk recovery, lymphoma	89 s	12 s	7.12×	24 s	3.65×	7.9 GB	1.4 GB (3.9 GB)
T2 · scRNA recovery, lymphoma	48 s	17 s	2.79×	29 s	1.64×	1.2 GB	0.5 GB (3.9 GB)
T2 · scRNA recovery, CRC	3.1 min	66 s	2.86×	2.4 min	1.34×	28.6 GB	5.6 GB (31.0 GB)
T3 · Visium recovery	10.3 min	2.8 min	3.73×	5.7 min	1.80×	37.1 GB	3.8 GB (19.2 GB)
T4 · Bulk discovery	18.2 min	15.2 min	1.20×	5.5 min	3.28×	10.4 GB	3.1 GB (17.5 GB)
T5 · scRNA discovery	4.7 h	4.0 h	1.18×	3.4 h	1.38×	82.5 GB	15.0 GB (380.2 GB)
T6 · Presorted discovery	44.7 min	11.1 min	4.04×	4.8 min	9.27×	8.7 GB	3.7 GB (17.5 GB)

Tutorial	Files	Byte-identical	Discrete-equal	Max rel. diff
T1 · Bulk recovery, lung	175	112	175	8.2e-13
T1 · Bulk recovery, lymphoma	189	137	189	3.5e-13
T2 · scRNA recovery, lymphoma	13	10	13	9.1e-15
T2 · scRNA recovery, CRC	150	134	150	9.6e-15
T3 · Visium recovery	103	84	103	1.3e-14
T4 · Bulk discovery	608	313	608	3.5e-12
T5 · scRNA discovery	762	490	762	8e-11
T6 · Presorted discovery	608	313	608	3.5e-12

Note on R discovery timings. R skips every NMF restart whose estim.RData already exists, so the R discovery runs (tutorials 4–6) were repeated with an emptied cache; the rows above are from those clean runs.

Reading the results¶

Discovery (T4–T6): Python is 1.2× (bulk), 4.0× (presorted) and 1.2× (the large scRNA-seq tutorial, 4.7 h in R vs 4.0 h in Python) faster. NMF dominates the large run, and both implementations use the same algorithm.
Recovery (T1–T3): Python is 2.8–7.4× faster than R. The gain is largest on the small bulk inputs, where fixed costs dominate: R starts an Rscript process per cell type and passes results between steps through text files.
Memory: summed RSS double-counts pages shared between forked workers; on t5 Python’s summed RSS is 380 GB but the real footprint (summed PSS) is 15 GB, versus 83 GB summed RSS for R. Starting workers with ECOTYPER_MP_CONTEXT=forkserver makes summed RSS equal PSS.
Exactness costs little: these timings are in the default R-exact mode (long-double reductions, R’s number formatting and text round-trips). The opt-in fast mode (exact=False) is described on the Performance page.

Reproduce¶

On a machine with Docker and both images built:

python validation/benchmark.py --jobs 32 t1_lung t1_lymphoma t2_lymphoma t2_crc t3_visium t6_presorted t4_bulk t5_scrna
python validation/benchmark.py --table

Each run writes validation/bench/{r,py}_<tutorial>.json (wall time and peak resident memory summed over all processes in the container, sampled every 0.5 s). This page is regenerated from those files and from validation/reports/*.tsv at every documentation build.