System Design on Scientific tooling

The constraint that changes everything

Consumer SaaS can often paper over inconsistency with retries and eventual consistency. Scientific tooling cannot. When a pipeline claims an ERP peak, a spike rate, or a focus score, someone may make a research or clinical decision on that number.

So system design here starts from a different north star: reproducibility under change.

Same inputs + same code + same config should yield the same artifacts — or an explicit reason they differ (algorithm version, filter params, hardware clock skew).

Design principles I use

1. Version the science, not just the app

Treat preprocessing recipes, model weights, and feature schemas as versioned contracts.

• Store pipeline_version, model_version, and config hashes with every run
• Prefer append-only result stores over silent overwrites
• Make “re-run with vN” a first-class action in the UI

2. Separate acquisition, processing, and interpretation

Scientific systems fail when these layers share a process and a mental model.

Layer	Responsibility	Failure mode if mixed
Acquisition	Capture signals / files reliably	Lost sessions, clock issues
Processing	Deterministic transforms	Irreproducible scores
Interpretation	Human review, thresholds, notes	Blind trust in a plot

APIs between layers should be boring: object storage paths, manifests, and typed records.

3. Make latency budgets honest

A live BCI demo and an offline MEG batch job are different products that happen to share math.

• Interactive paths: stream, window, approximate, explain confidence
• Batch paths: throughput, checkpointing, resumability
• Never pretend a heavy offline method is “real-time” by hiding queue delay

4. Observability is part of scientific validity

Logs are not enough. You need:

• Per-stage timing and drop counts
• QC metrics (SNR proxies, bad-channel rates, motion / artifact flags)
• Human-visible session health before model output

If operators cannot see why a run is bad, they will ship noise with a confidence interval.

Architecture sketch (practical)

For tools I build in this space, a recurring shape is:

1. Ingest service — accepts uploads or device streams, writes raw immutable objects
2. Worker pool — pulls jobs, runs versioned pipelines, emits artifacts + QC
3. Metadata DB — subjects/sessions/runs, versions, statuses
4. Review UI — compare conditions, annotate, export
5. Auth + audit — who saw what, who exported what

The interesting part is not the boxes — it is the contracts: schemas for events, artifact naming, and idempotent job keys so retries do not duplicate science.

Performance without self-deception

Precision and performance trade off. My rules of thumb:

• Profile the pipeline before parallelizing YAML
• Cache expensive immutable stages (filtered continuous data) carefully, keyed by version
• Prefer columnar / binary formats for large arrays; keep JSON for manifests
• Isolate GPU / heavy FFT work so web traffic cannot starve science jobs

What I avoid

• Silent default changes in filters between deploys
• UI that shows model output before QC gates pass
• One mega-service that acquires, trains, and serves
• “Temporary” notebooks that become production with no tests

Closing

System design for scientific tooling is product design for trust. The architecture succeeds when a skeptical collaborator can answer: What ran? On what data? With which version? And can I reproduce it next month?

That is the standard I design toward — in FocusAnalyze and in the broader neurotech / instrumentation work around it.