# Bridging the OMR browser gaps: research notes

Follow-up to [photo-to-score-research.md](photo-to-score-research.md), which
identified two gaps between "a small-enough OMR model exists" and "mmmt-tools
could ship a photo-to-score page": an unported layer of traditional
computer-vision/classifier code around each project's neural nets, and (as
that doc stated it) a missing PyTorch→ONNX export for `homr`. Compiled
2026-08-12 from public sources (web search plus primary-source fetches of
the actual `oemer` and `homr` source trees via the GitHub API — not
summaries of summaries). As with the two prior documents, treat anything
undated as "true as of the sources cited."

**Correction to the previous document, made here rather than silently:**
gap #2 as stated — "`homr`'s weights are PyTorch `.pth` checkpoints… no one
has published an ONNX export" — is **wrong**. Reading `homr`'s actual source
(below) shows the ONNX export already exists and is `homr`'s normal,
shipped deployment path. The real gap in that area is narrower and
different: whether those already-exported graphs survive
`onnxruntime-web`'s browser execution provider, which `photo-to-score-research.md`
has now been corrected to reflect (see that file's own changelog note at
the top of its "What would actually have to happen" section).

## TL;DR

- **The ONNX-export gap doesn't exist.** `homr` ships pre-exported
  `encoder_*.onnx` / `decoder_*.onnx` files (fp32 and fp16) as its normal
  distribution mechanism, loaded via `onnxruntime`'s Python API — confirmed
  by reading `homr/transformer/configs.py`, `encoder_inference.py`, and
  `decoder_inference.py` directly. `oemer`'s models were ONNX from the
  start. Neither project needs an export step done.
- **The gap relocated, it didn't vanish.** `homr`'s own code contains a
  documented case of one of these exact ONNX graphs *breaking* on a
  non-default execution provider — the decoder "crashes on its dynamic
  KV-cache dimension" under Apple's CoreML EP, per a comment in `homr`'s
  own `onnx_providers.py`. `onnxruntime-web`'s WASM backend is a different
  non-native EP with its own narrower op/feature surface. Whether these
  specific graphs run under it is an open, testable, not-yet-tested
  question — not something either this document or the prior one can
  responsibly assert either way.
- **fp16 doesn't survive the trip to WASM.** `onnxruntime-web`'s WASM
  backend has no native fp16 support and falls back to fp32; fp16 only
  works on the WebGPU backend, which has narrower browser/device coverage.
  That revises the size budget from the previous document's ~137 MB
  (fp16) figure back toward ~280 MB (fp32) for any deployment that needs
  to work on WASM as a baseline — a real, material correction, not a
  rounding difference.
- **The autoregressive decode loop needs a small, distinct JS port.**
  `homr`'s Python decoder loop uses `io_binding` — a persistent,
  device-bound tensor API specific to native `onnxruntime`'s Python/C++
  surface. No confirmed equivalent exists in `onnxruntime-web`'s JS API.
  This is a separate, bounded porting task from the CV/heuristics layer
  below, and it's the one piece that has to be exactly right for the
  transformer to produce anything coherent at all.
- **The CV/heuristics layer is real work, but each piece has a plausible
  bridge**, verified against the actual `cv2`/`scipy`/`scikit-learn` calls
  in both projects' source: OpenCV.js for the OpenCV calls, `skl2onnx` for
  the pickled scikit-learn classifiers (running through the *same*
  `onnxruntime-web` session already needed for the neural nets), and
  plain hand-written JS for the simpler algorithmic pieces (k-means,
  linear regression, peak-finding) that don't need a heavyweight bridge
  at all.
- **So: is there a path?** Yes, a real one — sketched in [Is there a
  path?](#is-there-a-path) — but it runs through one specific, cheap,
  currently-unperformed experiment (loading `homr`'s actual `.onnx`
  graphs into `onnxruntime-web` and seeing what happens) before any of
  the porting work below is worth starting.

## Gap A, corrected: the ONNX export already exists

`homr`'s `homr/transformer/configs.py` names its own model files directly:

```python
model_name = "pytorch_model_426-b6fd20809a8dcaf10dfd39a4ca4f64c6f056e644"
self.encoder_path = os.path.join(workspace, f"encoder_{model_name}.onnx")
self.decoder_path = os.path.join(workspace, f"decoder_{model_name}.onnx")
self.encoder_path_fp16 = os.path.join(workspace, f"encoder_{model_name}_fp16.onnx")
self.decoder_path_fp16 = os.path.join(workspace, f"decoder_{model_name}_fp16.onnx")
```

and `homr/transformer/encoder_inference.py` / `decoder_inference.py` both
construct `onnxruntime.InferenceSession` directly on those `.onnx` paths —
this is `homr`'s actual, normal, shipped inference path, not a
proof-of-concept or a training artifact. (The ~286 MB PyTorch `.pth`
checkpoint found on Hugging Face in the previous document's research is a
**training** checkpoint — `configs.py` points a separate `self.checkpoint`
field at `training/architecture/…` — not the deployment artifact; that
document has been corrected to stop presenting it next to the deployment
size figures.) `oemer` never had this question in the first place — its
`1st_model.onnx` / `2nd_model.onnx` were published as ONNX from day one, and
its own `setup.py` lists a plain `onnxruntime` (or `onnxruntime-gpu`)
dependency, not PyTorch, for inference.

This also explains how [Andromr](https://github.com/aicelen/Andromr) (the
Android app running `homr` on-device, cited in the previous document) works
at all: it couldn't run a raw PyTorch checkpoint on a phone either. It's
downloading and running these same exported ONNX graphs through
onnxruntime's Android bindings — mobile-native proof that the graphs
themselves are runnable outside a full PyTorch environment, just not yet
proof that they're runnable in *this specific* browser-WASM environment.

### Where the real uncertainty sits: execution-provider fragility

`homr`'s own `homr/onnx_providers.py` carries this comment, verbatim:

> "the CoreML EP needs `ModelFormat=MLProgram` to accept any nodes of
> homr's fp16 models, and under MLProgram the decoder crashes on its
> dynamic KV-cache dimension (onnxruntime 1.26). The decoder therefore
> always stays on the CPU EP with the fp32 model, which is faster than the
> fp16 model on the CPU EP."

Read plainly: the `homr` maintainers already hit a real crash moving this
exact decoder graph off the execution provider it was tested against, and
the failure mode was specifically about a **dynamic shape in the KV-cache
dimension** — exactly the kind of thing that autoregressive, cache-using
transformer decoders are notorious for breaking non-native ONNX backends
on. `onnxruntime-web`'s WASM execution provider is itself a distinct,
narrower-surface backend from both the CPU EP the decoder is pinned to and
the CoreML EP it crashes on. Nothing in this research confirms or rules out
whether the WASM EP handles this graph correctly — and the honest way to
find out is to try it, not to reason about it from a design document. **The
recommended first step for anyone picking this up is exactly that: load
`decoder_{model}.onnx` into `onnxruntime-web` and see whether it
initializes and runs a single forward step**, before investing in any of
the porting work below. This document could not perform that test — the
GitHub release CDN was not reachable from this research environment (a
direct download attempt returned a connection failure, not a meaningful
result) — so it's reported here as the open, decisive question rather
than as an answered one.

### The fp16 budget correction

`onnxruntime-web`'s WASM backend has no native fp16 kernels and silently
falls back to fp32; genuine fp16 execution requires the WebGPU backend,
which has materially narrower device/browser coverage than WASM (WASM is
close to universal; WebGPU is still rolling out). Concretely, that means:
a deployment that has to work on the WASM backend as a baseline (i.e.
everywhere, not just WebGPU-capable browsers) should budget against
`homr`'s **fp32** weights, not the fp16 ones — reverting the previous
document's "~137 MB (fp16)" figure back toward the ~280 MB fp32 figure
that same document also reported. This isn't a new fact so much as a
correction to which of two already-reported numbers is the realistic one
to plan around.

### The piece that still needs a real JS port: the decode loop

`homr/transformer/decoder_inference.py`'s `ScoreDecoder.generate()` method
is a Python autoregressive loop: it initializes a KV cache, then for each
of up to `max_seq_len` steps, binds the previous step's output tokens plus
the running cache as inputs via `self.io_binding.bind_cpu_input(...)` /
`bind_ortvalue_input(...)`, runs the session, and reads back updated cache
tensors for the next iteration. `io_binding` — a persistent,
device-resident tensor handle that avoids re-copying the KV cache on every
step — is part of native `onnxruntime`'s Python/C++ API surface. No
directly equivalent, persistent-binding API was confirmed in
`onnxruntime-web`'s JS surface in this research; the ordinary
`InferenceSession.run()` JS call takes a plain feeds object and returns a
plain fetches object each call, which is a workable substitute (feed the
cache tensors in as regular inputs, read updated cache tensors back out of
the outputs each step) but not a drop-in translation of the Python code.
This loop — cache initialization, per-step tensor bookkeeping, EOS
handling, vocabulary decoding via the four `tokenizer_*.json` files already
shipped alongside the model — is a small, self-contained, and mechanically
well-understood porting task (maybe a few hundred lines of JS) compared to
the CV/heuristics layer below, but it's worth naming as its own item
because it's the one piece with zero tolerance for a subtly-wrong port: a
bug here doesn't degrade output quality, it produces garbage tokens.

## Gap B: the CV/heuristics layer, read from the actual source

The previous document already established that both projects pair their
neural nets with substantial non-neural code. This document went further
and read what that code actually imports and calls, in both projects, to
answer "is there a bridge for this specific code, or does someone have to
write it from nothing."

### `oemer`

- **OpenCV calls** (`import cv2`), confirmed directly in `morph.py` and
  `staffline_extraction.py`: `cv2.morphologyEx` with `MORPH_OPEN`,
  `MORPH_CLOSE`, and `MORPH_HITMISS`; `cv2.erode`; `cv2.dilate`. All four
  are standard, long-supported OpenCV functions present in
  [OpenCV.js](https://docs.opencv.org/4.13.0/d0/d43/tutorial_js_table_of_contents_contours.html),
  the official WASM/asm.js build of OpenCV maintained by the OpenCV
  project itself — not a case that needs verifying function-by-function
  from scratch, though the specific build mmmt-tools would vendor should
  still be checked against the full call list before committing, since
  OpenCV.js ships as a curated subset rather than 100% of OpenCV.
- **`scipy`/`scikit-learn` calls used directly at runtime**, not just for
  training: `staffline_extraction.py` also imports `scipy.signal.find_peaks`,
  `sklearn.cluster.KMeans`, and `sklearn.linear_model.LinearRegression` —
  these are ordinary, well-understood algorithms (peak-finding, k-means
  clustering, least-squares line fitting) called inline as part of staff
  detection, not exported models. These don't need a conversion tool at
  all — they're small enough to hand-write directly in JavaScript, which
  is exactly the pattern this repo already follows elsewhere (e.g. the
  walking-bass checker, the upper-structure tension labeling) rather than
  vendoring a numerics library for a few dozen lines of math.
- **Pickled scikit-learn classifiers**: `oemer`'s own `setup.py` packages
  `sklearn_models/*.model` and depends on `scikit-learn>=1.2`. The actual
  shipped files are `clef.model`, `rests.model`, `rests_above8.model`, and
  `sfn.model` — four small classifiers, not a large bank of them.
  `oemer/classifier.py` (the training/labeling script for these) imports
  `sklearn.svm`, `KNeighborsClassifier`, `AdaBoostClassifier`,
  `RandomForestClassifier`, `GradientBoostingClassifier`,
  `BaggingClassifier`, and `RidgeClassifier` — the module's own dedicated
  hyperparameter grid (`SVM_PARAM_GRID`) suggests SVM is the primary
  algorithm actually shipped, though this research can't confirm which
  specific algorithm backs each of the four `.model` files without
  unpickling them. It doesn't need to: **every one of those estimator
  classes is a confirmed-supported conversion target for
  [`skl2onnx`](https://onnx.ai/sklearn-onnx/supported.html)** (checked
  against sklearn-onnx's own supported-models page), the standard,
  actively maintained tool for turning a fitted scikit-learn model into an
  ONNX graph. Converting these four small classifiers with `skl2onnx` and
  running them through the *same* `onnxruntime-web` session infrastructure
  already needed for the U-Net/transformer models is the natural bridge —
  one inference runtime for the whole page, not two. The one caveat worth
  flagging: scikit-learn pickles are version-sensitive, and re-loading an
  older pickled model under a newer scikit-learn to run it through
  `skl2onnx` can fail before ONNX conversion is ever reached — worth
  checking early, not assumed away.

### `homr`

Reading `homr/staff_detection.py`, `homr/bounding_boxes.py`, and
`homr/staff_dewarping.py` directly shows the same shape of dependency:
`import cv2` in all three, plus a real, hand-rolled domain model
(`Staff`, `StaffPoint`, `Note`, `BoundingEllipse`, `RotatedBoundingBox`
classes in `homr/model.py` and `homr/bounding_boxes.py`) built on top of
those OpenCV calls to do staff-anchor detection, unit-size estimation,
perspective dewarping, and grand-staff/brace merging — the same
"OpenCV.js for the CV calls, hand-written JS for the domain logic and
simple algorithms" shape as `oemer`, just organized around richer classes
rather than free functions. `homr` doesn't appear to lean on
scikit-learn classifiers the way `oemer` does (no equivalent import found
in the modules checked), so its bridging story is simpler in one respect —
no `skl2onnx` conversion needed — and about the same in the other (still a
real OpenCV.js-plus-hand-port job for the geometry/domain layer).

## Bridging tools, assessed against what was actually found

| Tool | Bridges | Verdict |
|---|---|---|
| [OpenCV.js](https://docs.opencv.org/4.13.0/d0/d43/tutorial_js_table_of_contents_contours.html) | The `cv2.*` calls in both projects (morphology, erode/dilate, and — unverified in this research but standard OpenCV territory — contour/connected-component calls likely used elsewhere in both codebases) | Real, official, actively maintained. Costs: a real WASM download on top of the model weights (commonly cited around 8 MB) and a manual-memory-management API (every `cv.Mat` etc. needs an explicit `.delete()`) that doesn't match this repo's usual garbage-collected JS style — a real but manageable adaptation cost, not a blocker. |
| [`skl2onnx`](https://onnx.ai/sklearn-onnx/supported.html) | `oemer`'s four pickled scikit-learn classifiers | Real, maintained by the ONNX project itself, and every estimator class `oemer` imports is on its supported list. The output is an ordinary `.onnx` file runnable through the same `onnxruntime-web` session as the neural models — the strongest "actually unifies the stack" finding in this research. Caveat: this is an offline, one-time conversion step someone runs against the existing pickles, not a runtime bridge — it has to be done once, ahead of time, not shipped as live conversion logic. |
| Hand-written JS | k-means, linear regression, peak-finding, and the transformer decode loop's step-by-step orchestration | Not really a "tool" — the honest assessment is that these pieces are small and well-understood enough to just write directly, which is this repo's own established practice for exactly this kind of algorithmic code. |
| [Pyodide](https://pyodide.org/) (Python-in-WASM) | Hypothetically, all of the above at once, by running the original Python source unmodified | **Not a shortcut, and presented here with its real costs rather than as an easy way out.** Pyodide cannot run `onnxruntime` — it's a native C++ extension with no WASM build in Pyodide's package set, confirmed by Pyodide's own package-loading model (packages with C-extensions need a pre-built Pyodide wheel, and `onnxruntime` was not found among them). So a Pyodide-based approach would still need `onnxruntime-web` for the neural/classifier inference, with Pyodide only covering the `cv2`/`scipy`/`sklearn`-heuristics half — meaning it doesn't remove the JS/Python bridging problem, it relocates it to a Python↔JS FFI boundary instead of a hand-port, on top of separately downloading Pyodide's own runtime plus `opencv-python`/`scipy`/`scikit-learn` wheels (individually in the tens-of-MB range each, atop the model weights already budgeted above). Worth naming as an option; not recommended as the default path given it adds a second heavy runtime without removing the need for the first one. |

## Is there a path?

Yes — a real, technically coherent one, though not a short one:

1. **First, and before anything else: test whether `homr`'s exported
   `.onnx` graphs actually load and run under `onnxruntime-web`'s WASM
   backend**, especially the decoder given its documented fragility on a
   different non-native execution provider. This is a few hours of work,
   not a research project, and it's the fact that determines whether the
   rest of this plan is worth pursuing for `homr` specifically. (`oemer`'s
   simpler U-Net-only architecture has no equivalent dynamic-KV-cache risk
   and is the safer starting point if this check comes back negative.)
2. **Run `oemer`'s four pickled classifiers through `skl2onnx`** once,
   offline, producing four small `.onnx` files that ride in the same
   `onnxruntime-web` session pool as the segmentation/transformer models —
   no separate runtime needed for the classical-ML half of the pipeline.
3. **Vendor OpenCV.js** for the `cv2` calls, checked against the actual
   function list each project uses (this research confirmed
   `morphologyEx`/`erode`/`dilate`; a full port would need to audit every
   remaining `cv2.*` call across both projects' source, which this
   research didn't exhaustively enumerate).
4. **Hand-port the rest in JavaScript**: the simple scipy/sklearn calls
   (k-means, linear regression, peak-finding), the bbox/staff/note domain
   model classes, and — as its own carefully-scoped item — the
   autoregressive decode loop currently expressed in Python via
   `io_binding`.
5. **Re-budget the download around fp32, not fp16**, unless the plan
   explicitly accepts WebGPU-only as a requirement — a real product
   decision, not a detail to leave implicit.

None of this is small. But it is, concretely, a list of *known, bounded*
engineering tasks with a plausible tool or technique named against each
one — not an open research question the way "is a small enough OMR model
even possible" was before the previous document, or the way "what's
actually in `score.dat`" was for the Finale investigation. The one
remaining unknown that could still upend this plan — whether `homr`'s
graphs survive the WASM execution provider — is cheap to resolve and
should be resolved first.

## Sources

- [`BreezeWhite/oemer`](https://github.com/BreezeWhite/oemer) source, read
  directly via the GitHub Contents API: [`setup.py`](https://raw.githubusercontent.com/BreezeWhite/oemer/main/setup.py),
  [`oemer/morph.py`](https://raw.githubusercontent.com/BreezeWhite/oemer/main/oemer/morph.py),
  [`oemer/staffline_extraction.py`](https://raw.githubusercontent.com/BreezeWhite/oemer/main/oemer/staffline_extraction.py),
  [`oemer/classifier.py`](https://raw.githubusercontent.com/BreezeWhite/oemer/main/oemer/classifier.py),
  and the `oemer/sklearn_models/` directory listing
- [`liebharc/homr`](https://github.com/liebharc/homr) source, read directly
  via the GitHub Contents API: [`homr/onnx_providers.py`](https://raw.githubusercontent.com/liebharc/homr/main/homr/onnx_providers.py),
  [`homr/transformer/configs.py`](https://raw.githubusercontent.com/liebharc/homr/main/homr/transformer/configs.py),
  [`homr/transformer/encoder_inference.py`](https://raw.githubusercontent.com/liebharc/homr/main/homr/transformer/encoder_inference.py),
  [`homr/transformer/decoder_inference.py`](https://raw.githubusercontent.com/liebharc/homr/main/homr/transformer/decoder_inference.py),
  [`homr/download_utils.py`](https://raw.githubusercontent.com/liebharc/homr/main/homr/download_utils.py),
  [`homr/staff_detection.py`](https://raw.githubusercontent.com/liebharc/homr/main/homr/staff_detection.py),
  [`homr/bounding_boxes.py`](https://raw.githubusercontent.com/liebharc/homr/main/homr/bounding_boxes.py),
  [`homr/staff_dewarping.py`](https://raw.githubusercontent.com/liebharc/homr/main/homr/staff_dewarping.py),
  [`homr/note_detection.py`](https://raw.githubusercontent.com/liebharc/homr/main/homr/note_detection.py)
- [aicelen/Andromr](https://github.com/aicelen/Andromr) — the on-device Android app corroborating that homr's exported ONNX graphs run outside a full PyTorch environment
- [sklearn-onnx supported models](https://onnx.ai/sklearn-onnx/supported.html)
- [OpenCV.js contours/tutorial docs](https://docs.opencv.org/4.13.0/d0/d43/tutorial_js_table_of_contents_contours.html) and general OpenCV.js size/API-shape discussion surfaced in this and the prior document's research
- [Pyodide](https://pyodide.org/), its [package loading documentation](https://pyodide.org/en/stable/usage/loading-packages.html), and a [Pyodide GitHub issue on C-extension packages like `opencv-python`](https://github.com/pyodide/pyodide/issues/1627)
- ONNX Runtime Web fp16/WASM-vs-WebGPU behavior and 2026 production guidance, per current web-search results on `onnxruntime-web` FP16/WASM/WebGPU support (no single canonical Microsoft doc page was pinned down for this specific claim in this research; treat as corroborated-by-multiple-sources rather than single-sourced, and worth re-verifying directly against ONNX Runtime's own docs before relying on it for an implementation decision)
- [photo-to-score-research.md](photo-to-score-research.md) (this repo) — the document this one follows up on and partially corrects