# Photo-to-score (OMR) research

Research notes for ROADMAP.md §15's "Photo-to-score" idea. Compiled
2026-08-12 from public sources (web search + primary-source fetches
including the GitHub and Hugging Face APIs directly); no OMR software was
run to produce this. Model release lists, an active project's file
layout, and library adoption are all moving targets — treat anything
undated as "true as of the sources cited," not as current fact for all
time. No accuracy/benchmark numbers are quoted for either system profiled
below beyond what each project states about itself in qualitative terms;
a research paper's benchmark score (e.g. TrOMR's own reported SER) is not
attributed to the downstream forks that build on it, since neither
publishes a matching real-world evaluation.

**Correction (added after follow-up research):** this document originally
stated that `homr`'s weights were PyTorch checkpoints with "no public ONNX
export… found." That was wrong — reading `homr`'s actual source shows it
ships pre-exported ONNX files as its normal deployment path. See
[omr-bridging-research.md](omr-bridging-research.md) for the full
correction, what the gap actually looks like once that's fixed, and
whether there's a path to bridging it. The rest of this document has been
edited in place to match; this note exists so the correction itself is
visible, not just its result.

## TL;DR — answering the direct question

**Yes, there are open-source OMR models small enough in principle to load
into browser memory.** `oemer`'s two segmentation networks are already
published as ONNX files totaling **~104 MB** — the exact format
`onnxruntime-web` consumes, no conversion needed. `homr`'s stronger,
transformer-based pipeline (segmentation + encoder + decoder) is
**~137 MB** in its smallest published (fp16) form, and it already runs
fully on-device on a phone (via a different runtime, LiteRT/TensorFlow
Lite + ONNX Runtime, not yet a browser one) in the Android app
[Andromr](https://github.com/aicelen/Andromr) — the strongest existing
proof that device-class hardware is enough. So the ML weights are not the
blocker.

**What's missing is everything around the weights.** Neither project is a
single model call: both pair the neural net(s) with a substantial layer of
traditional computer-vision heuristics and, for `oemer`, scikit-learn
classifiers — real Python source, not a config file — that turns raw
segmentation output into an actual score. That layer would need a from-
scratch JavaScript reimplementation; nobody has published one. `homr`
**does** already ship pre-exported ONNX files for its encoder and decoder
(fp32 and fp16) as its normal deployment path — confirmed by reading its
source directly, corrected here after this document originally got that
wrong (see the note at the top of this file and
[omr-bridging-research.md](omr-bridging-research.md)). What's still
genuinely open is narrower: whether those specific graphs run correctly
under `onnxruntime-web`'s browser execution provider, which is a different,
narrower-surface backend than the ones `homr` has actually been tested
against — and `homr`'s own code documents that exact decoder graph already
breaking on one non-native execution provider (Apple's CoreML) over a
dynamic KV-cache dimension, which is a real reason for caution, not proof
of failure. No project (open-source or otherwise) was found running full
OMR client-side in a browser yet — Andromr is the closest thing, and that's
a native Android app (itself running these same exported ONNX graphs, just
through mobile bindings rather than a browser), not a webpage.

For mmmt-tools specifically: the model-size objection this repo's own
ROADMAP.md previously raised turns out to be the *wrong* objection — a
104–137 MB model download is a real UX cost (roughly 80× the ~1.3 MB OSMD
bundle already vendored for `score.html`) but not a hard technical wall,
and it doesn't compromise the local-only privacy story the way a hosted
OMR API call would. The real blocker is the unported glue code and export
tooling. See [Feasibility for mmmt-tools](#feasibility-for-mmmt-tools).

## What OMR actually is, and why it's a different problem than Finale

ROADMAP.md §15 groups this with the Finale-file-converter idea as "score
file conversion," but they're not the same kind of problem. Finale's MUSX
is a fully-specified-once-you-reverse-engineer-it **data format** — every
byte in `score.dat` means something exact, and the job is decoding a
structure (see
[finale-file-format-research.md](finale-file-format-research.md)).
**Optical music recognition (OMR)** is computer vision: a photo has no
structure to decode, only pixels, and the system has to *infer* staff
lines, noteheads, clefs, and rhythm from an image that may be skewed,
poorly lit, or camera-warped. It's the sheet-music analogue of OCR, and
it inherits OCR's characteristic failure mode — a wrong guess that looks
plausible rather than an obvious parse error. ([Wikipedia: Optical music
recognition](https://en.wikipedia.org/wiki/Optical_music_recognition) has
a general overview of the problem and its history.)

Historically the dominant open-source OMR tool has been
[**Audiveris**](https://github.com/Audiveris/audiveris) — a mature, actively
maintained Java desktop application (AGPL-3.0) that converts scanned sheet
music to MusicXML. It predates the current wave of deep-learning OMR and
still relies heavily on traditional image-processing pipelines rather than
end-to-end neural inference. It's a full JVM desktop app with a
Gradle/JDK build — not something with any realistic in-browser path — so
it's included here for landscape completeness, not as a candidate.

The more relevant recent work is **end-to-end neural OMR**: a model that
takes an image (or a dewarped staff crop) and predicts a symbol sequence
directly, the way modern OCR moved from segment-then-classify pipelines
toward sequence models. Two open-source projects in this line were
investigated in depth because both are small enough, in principle, to
matter for an in-browser use case: **`oemer`** and **`homr`**.

## Two candidate systems

| | [`oemer`](https://github.com/BreezeWhite/oemer) | [`homr`](https://github.com/liebharc/homr) |
|---|---|---|
| Author | BreezeWhite (maintained fork: meteo-team) | Christian Liebhardt (`liebharc`) |
| Pipeline | Two U-Net segmentation models → traditional CV + scikit-learn classifiers assemble the score | U-Net segmentation (adapted from oemer's) → staff reconstruction → **transformer** (based on [Polyphonic-TrOMR](https://github.com/NetEase/Polyphonic-TrOMR)) does end-to-end symbol sequence recognition per staff |
| Output | MusicXML | MusicXML 4.0 |
| Target images | "Phone-taken" sheet photos, per its own README | Explicitly camera photos, with perspective dewarping built in |
| Inference runtime | ONNX Runtime by default (`--use-tf` switches to TensorFlow) | PyTorch for training; the Android port (below) uses LiteRT + ONNX Runtime |
| Published weight format | **ONNX**, already | **ONNX**, already — `encoder_*.onnx` / `decoder_*.onnx` (fp32 and fp16), confirmed by reading `homr/transformer/configs.py` and `*_inference.py` directly; this is corrected from this document's original claim, see the note at the top of this file |
| Smallest published weight size found | `1st_model.onnx` ≈ 67.5 MB + `2nd_model.onnx` ≈ 36.7 MB ≈ **104 MB total** (exact byte counts from a project issue thread showing the download) | fp16 release assets: segnet ≈ 26.5 MB + encoder ≈ 24.1 MB + decoder ≈ 86.3 MB ≈ **~137 MB total** — but `onnxruntime-web`'s WASM backend has no native fp16 support and falls back to fp32, so a WASM-baseline deployment should budget against the **fp32** figure instead, closer to ~280 MB (a ~286 MB fp32 checkpoint was found on [Hugging Face](https://huggingface.co/homr-music/homr-transformer), 300,329,028 bytes confirmed via the HF API — though `configs.py` shows that specific file is a **training** checkpoint, not the deployment `.onnx` artifact; see [omr-bridging-research.md](omr-bridging-research.md) for the full sizing discussion) |
| License | **MIT** (confirmed by reading the repo's `LICENSE` file directly) | **AGPL-3.0** |
| Stated limitations (project's own words) | "Probably will not work on handwritten scores or other notation types"; ~3–5 minutes to process one image on GPU | "Focuses on pitch and rhythm information on the bass or treble clef, neglecting dynamics, articulation, double sharps/flats" |
| On-device precedent | None found | [**Andromr**](https://github.com/aicelen/Andromr) — a real Android app running homr's exact model stack fully on-device (segmentation + transformer encoder via **LiteRT**, transformer decoder via **ONNX Runtime**), no server involved, "your scores stay on your device," ~30 seconds per page on a contemporary phone |

Some of the release-asset sizes above don't line up cleanly with each
other — e.g. one `homr` decoder checkpoint's fp16 build is listed as
*larger* than its fp32 build, which shouldn't happen and likely reflects
packaging/versioning quirks across releases rather than a real precision
inversion. Treat the totals above as the right order of magnitude, sourced
directly from [`homr`'s GitHub Releases](https://github.com/liebharc/homr/releases)
and [`oemer`'s "checkpoints" release](https://github.com/BreezeWhite/oemer/releases/tag/checkpoints)
via the GitHub API, not as a precise, internally-consistent spec.

## The part that isn't a model: the glue code

Both projects are pitched as "end-to-end," but neither is *only* a neural
network call. Fetching `oemer`'s own source tree directly shows the shape
of what sits around its two ONNX models — real, substantial Python
modules, not thin wrappers:

```
barline_extraction.py    build_system.py         note_group_extraction.py
bbox.py                  classifier.py           notehead_extraction.py
build_label.py           dewarp.py               rhythm_extraction.py
constant.py               general_filtering_rules.py   staffline_extraction.py
constant_min.py           morph.py                symbol_extraction.py
sklearn_models/  (scikit-learn classifier pickles, not neural weights)
```

That's staffline detection, morphological filtering, bounding-box
assembly, note-group extraction, rhythm extraction, and symbol extraction
all implemented as traditional CV heuristics and small classical
classifiers, layered on top of the two U-Nets' raw pixel predictions.
`homr` similarly does staff-anchor detection, unit-size estimation, staff
reconstruction, and grand-staff/brace merging as explicit algorithmic
steps *before* the dewarped image ever reaches its transformer (see the
"Technical Details" section of its own
[README](https://github.com/liebharc/homr)). None of that logic ships as
a portable artifact — it's Python, tightly coupled to each project's own
data structures. Porting either system to the browser means reimplementing
this layer in JavaScript from reading the source, not just loading a
`.onnx` file and calling `run()`.

## What would actually have to happen to ship this in-browser

1. ~~Get `homr`'s weights into ONNX~~ **Already done.** This document
   originally listed this as a required step, based on only having found
   PyTorch `.pth` checkpoints; that was wrong (see the correction note at
   the top of this file). `homr` ships pre-exported `encoder_*.onnx` /
   `decoder_*.onnx` files as its normal deployment path, same as `oemer`.
   The real open question in this area is narrower — whether those
   specific graphs run correctly under `onnxruntime-web`'s WASM execution
   provider, given `homr`'s own code documents the decoder graph already
   breaking on a different non-native provider. See
   [omr-bridging-research.md](omr-bridging-research.md) for the full
   detail and the recommended first test.
2. **Reimplement the assembly/heuristics layer in JavaScript**, per the
   section above. This is genuine engineering, not glue — the kind of
   thing this repo's other pages (walking bass checking, upper-structure
   tension labeling) do carefully in pure JS, just at a much larger scope
   here.
3. **Wire inference through [`onnxruntime-web`](https://onnxruntime.ai/docs/get-started/with-javascript/web.html)**
   (WASM/WebGL/WebGPU backends), which is a real, actively maintained
   library — [Whisper tiny.en running fully client-side](https://onnxruntime.ai/docs/tutorials/web/deploy.html)
   is the closest existing precedent for "a real neural model, comparable
   order of magnitude, already shipping in a browser via this exact
   library." No comparable public OMR-in-the-browser demo was found in
   this research, using either `onnxruntime-web` or TensorFlow.js. Netlify
   itself is not an obstacle to this — see [Netlify hosting
   compatibility](#netlify-hosting-compatibility) below, checked directly
   against this repo's own `_headers`/`netlify.toml`.
4. **Accept the license that comes with whichever model is used.**
   `oemer` is MIT, which imposes no real constraint. `homr` — the
   stronger system, with camera-photo focus and a working on-device
   precedent — is AGPL-3.0, which for a site serving JavaScript to
   browsers is a real, non-academic obligation (AGPL's network-use clause
   is exactly the scenario "serve this over the web" triggers) and would
   need to be accepted deliberately, not incidentally.
5. **Budget for the download.** No documented hard per-file size limit was
   found for a normal Netlify static deploy; Netlify's *Large Media*
   feature (Git LFS-backed) caps individual files at 100 MB, but that
   feature only applies if a site opts into routing assets through it —
   it's not a general site-wide ceiling. Either way, 104–137 MB is a large
   first-load asset for a page to fetch, roughly 80× the ~1.3 MB OSMD
   bundle already vendored for `score.html`'s notation rendering — the
   closest existing precedent in this repo for "vendor a sizeable
   pre-built artifact instead of writing the logic by hand."

## Netlify hosting compatibility

A natural follow-up: does `onnxruntime-web` actually work on Netlify,
where mmmt-tools is hosted? **Yes — inference is 100% client-side (WASM
running in the visitor's browser), so "hosting on Netlify" just means
"can Netlify serve the JS/WASM/model files with the right headers," which
it can.** Checked directly against this repo's actual current
[`_headers`](_headers) and [`netlify.toml`](netlify.toml), rather than
Netlify's capabilities in the abstract:

- **CSP: already clear.** WebAssembly compilation under an enforced
  `script-src` CSP directive requires `'unsafe-eval'` or the narrower
  `'wasm-unsafe-eval'` — without one of the two, Chrome and other browsers
  block WASM outright ([WebAssembly/content-security-policy issue
  #7](https://github.com/WebAssembly/content-security-policy/issues/7)).
  This repo's `_headers` file **already carries `'unsafe-eval'`** in
  `script-src` (added for `js/synth-bytebeat.js`'s `new Function(...)`
  use) — so this box is already checked; no CSP loosening would be needed
  to add `onnxruntime-web`.
- **WASM MIME type: a one-line addition, not a blocker.** Browsers need a
  response header of `Content-Type: application/wasm` to use the fast
  path (`WebAssembly.instantiateStreaming`); a real, documented case of a
  Netlify site needing to fix this via a `_headers` rule was found on the
  [Netlify Support
  Forums](https://answers.netlify.com/t/what-is-the-limit-for-static-assets-a-site-built-with-a-static-site-generator/74178) —
  Netlify's default MIME mapping for `.wasm` isn't guaranteed in every
  case. Even without it, `onnxruntime-web` and browsers generally fall
  back to non-streaming `WebAssembly.instantiate()` on a plain
  `ArrayBuffer`, which still works, just compiles slightly slower. Since
  this repo's `_headers` already sets explicit per-extension rules (see
  the `/*.png`, `/*.ico` blocks), adding one for `/*.wasm` would be a
  one-line, zero-risk addition matching the file's existing pattern,
  worth doing regardless of whether Netlify's default turns out fine.
- **Threading/SIMD (performance only, not a hard requirement).**
  `onnxruntime-web`'s fastest WASM path uses multi-threading, which needs
  the page to be [cross-origin
  isolated](https://developer.mozilla.org/en-US/docs/Web/API/crossOriginIsolated) —
  `Cross-Origin-Opener-Policy: same-origin` plus
  `Cross-Origin-Embedder-Policy: require-corp` (or `credentialless`) set
  on every response. Netlify supports both headers natively through
  `_headers`/`netlify.toml` ([confirmed via the Netlify Support
  Forums](https://answers.netlify.com/t/cross-origin-headers-are-not-working-with-netlify/153447)
  and a [worked Godot-on-Netlify
  example](https://www.rafa.ee/articles/deploying-godot-4-html-exports/)),
  so this is available if wanted — but this repo's `_headers` doesn't set
  either header today, and adding `require-corp` specifically would need
  checking against the one external dependency already carried in this
  site's CSP (the tuner page's 3D-visual CDN script from
  `cdnjs.cloudflare.com` — see the CSP note at the top of `_headers`),
  since `require-corp` obligates every cross-origin subresource to opt in
  via CORP/CORS headers of its own; `credentialless` sidesteps that
  requirement at a small compatibility cost in older browsers. Without
  cross-origin isolation at all, `onnxruntime-web` still runs — just
  single-threaded, which matters more for a 104–137 MB OMR model than for
  anything this site currently ships, but isn't a functional blocker.
- **Nothing here needs Netlify Functions or any server compute.** The
  existing `netlify.toml` only configures the site's analytics functions
  (`netlify/functions`); an OMR feature would need zero additions to it,
  the same "static site, browser does all the work" shape as everything
  else mmmt-tools ships.

Net: Netlify was never the obstacle. The real, load-bearing gap is the
unported CV/heuristics glue code (both projects) plus, for `homr`
specifically, verifying its already-exported ONNX graphs actually run
under `onnxruntime-web`'s execution provider — not anything about where
the site is hosted. See [omr-bridging-research.md](omr-bridging-research.md)
for the detailed follow-up on both.

## Feasibility for mmmt-tools

The size objection in the current ROADMAP.md §15 wording ("a from-scratch
ML model shipped as a static asset... or an OMR service call") undersells
what's actually available — it isn't from-scratch-vs-hosted-API, there's a
third option: **vendor an existing, off-the-shelf model**, the same shape
of decision already made for OSMD on `score.html`. That keeps the
local-only, nothing-leaves-the-page privacy story fully intact, which a
hosted OMR API would break — this is a case where the site's DNA is
compatible with the idea, just at a much larger asset-weight cost than
anything shipped so far.

What actually gates this, based on the research above:

- **`oemer`'s path is the more tractable one to start from**: MIT-licensed,
  smaller (~104 MB), and its weights are already in the exact format
  `onnxruntime-web` needs — no export step. Its output quality is the
  weaker of the two (no transformer, heavier reliance on classical CV
  heuristics, no explicit camera-photo/dewarping focus), and reproducing
  its assembly logic in JS is still a substantial, multi-module port.
- **`homr`'s path produces the better result and has the strongest
  feasibility proof** (a real on-device Android app already shipping its
  exact exported ONNX graphs), but costs more: AGPL-3.0, larger weights
  (especially once fp16-on-WASM's fallback to fp32 is accounted for), and
  an unresolved question of whether its decoder graph — already documented
  as fragile across execution providers — survives `onnxruntime-web`'s
  WASM backend specifically. That question is cheap to answer (load the
  graph and test it) but hasn't been, by this research or, as far as this
  research found, by anyone.
- **Neither is a weekend project.** Both require porting a nontrivial
  amount of Python computer-vision/assembly logic to JavaScript by hand,
  which is real, unavoidable engineering — the "model is small enough"
  question this research was scoped to answer turns out to be the easy
  part.

If this is picked up, `oemer` (MIT, ONNX-native, smaller, no
execution-provider question mark) is the more realistic starting point
than `homr`, even though `homr` would likely produce the better
transcription — porting AGPL-licensed code and resolving an unverified
execution-provider risk are both bigger asks than reimplementing a
documented, MIT-licensed heuristics pipeline against a model architecture
with no comparable red flag. See
[omr-bridging-research.md](omr-bridging-research.md) for the detailed
breakdown of what porting either project's glue code would actually
involve, and a concrete assessment of which parts of it have a plausible
bridging tool already.

## Sources

- [Wikipedia, Optical music recognition](https://en.wikipedia.org/wiki/Optical_music_recognition)
- [Audiveris/audiveris](https://github.com/Audiveris/audiveris) (GitHub)
- [BreezeWhite/oemer](https://github.com/BreezeWhite/oemer) — [README](https://github.com/BreezeWhite/oemer/blob/main/README.md), [source tree](https://github.com/BreezeWhite/oemer/tree/main/oemer) (fetched directly via the GitHub Contents API), [LICENSE](https://raw.githubusercontent.com/BreezeWhite/oemer/main/LICENSE), [checkpoints release](https://github.com/BreezeWhite/oemer/releases/tag/checkpoints), model sizes from a project issue thread showing the actual download
- [liebharc/homr](https://github.com/liebharc/homr) — [README](https://github.com/liebharc/homr) (technical-details section), [releases](https://github.com/liebharc/homr/releases) (fetched via the GitHub Releases API for exact asset byte sizes)
- [homr-music/homr-transformer on Hugging Face](https://huggingface.co/homr-music/homr-transformer) (fetched via the Hugging Face Hub API for the exact 300,329,028-byte checkpoint size)
- [aicelen/Andromr](https://github.com/aicelen/Andromr) — the Android app running homr fully on-device
- [NetEase/Polyphonic-TrOMR](https://github.com/NetEase/Polyphonic-TrOMR) and the original [TrOMR paper (arXiv:2308.09370)](https://ar5iv.labs.arxiv.org/html/2308.09370)
- [Sheet Music Transformer (SMT)](https://github.com/antoniorv6/SMT), [SMT-plusplus](https://github.com/antoniorv6/SMT-plusplus), and the [ICDAR 2024 paper (arXiv:2402.07596)](https://arxiv.org/abs/2402.07596) — profiled as further evidence of the current OMR research direction, not evaluated as a browser candidate
- [Transcoda paper (arXiv:2605.10835)](https://arxiv.org/pdf/2605.10835) — a compact, synthetic-data-trained OMR model raised as a possibly-smaller alternative worth revisiting; no public code/weights were confirmed during this research
- [ONNX Runtime Web docs](https://onnxruntime.ai/docs/get-started/with-javascript/web.html) and its [Whisper-in-the-browser deployment tutorial](https://onnxruntime.ai/docs/tutorials/web/deploy.html), used as the closest existing precedent for a comparable-scale model already running in a browser
- Netlify Support Forums, [static asset size limit thread](https://answers.netlify.com/t/what-is-the-limit-for-static-assets-a-site-built-with-a-static-site-generator/74178) and [Large Media requirements docs](https://docs.netlify.com/build/git-workflows/large-media/requirements-and-limitations/) (100 MB cap, Large Media only)
- WebAssembly/content-security-policy, [issue #7: no way to use WebAssembly without `'unsafe-eval'`](https://github.com/WebAssembly/content-security-policy/issues/7)
- Netlify Support Forums, [cross origin headers (COOP/COEP) thread](https://answers.netlify.com/t/cross-origin-headers-are-not-working-with-netlify/153447), and [Deploying Godot 4 HTML exports with cross-origin isolation](https://www.rafa.ee/articles/deploying-godot-4-html-exports/) — worked example of COOP/COEP via a Netlify-style `_headers` file
- MDN, [`crossOriginIsolated`](https://developer.mozilla.org/en-US/docs/Web/API/crossOriginIsolated)
- This repo's own [`_headers`](_headers) and [`netlify.toml`](netlify.toml), read directly rather than assumed, to check the hosting question against mmmt-tools' actual current configuration