Offline Receipt OCR in Expo: The Hard Part Starts After Text Recognition

On a fuel receipt, 1,899 might be a unit price. 95 might be a product grade. 04 might be a pump number. The OCR engine can read all three perfectly and still leave me with the wrong fuel record.

I added receipt scanning to OdoKeep, my vehicle logbook built with React Native and Expo. Recognition runs on the phone: Apple's Vision framework on iOS and bundled Google ML Kit text recognizers on Android. The photograph and recognized text are not uploaded by this feature.

The native module handles acquisition, image preparation, text recognition, and the temporary photograph. Interpreting the result lives in TypeScript under lib/receipt-scan. Keeping that boundary lets the parser run in ordinary unit tests, independent of a camera or a particular phone.

Keep the evidence attached to the text

The native result is richer than a string. It contains lines, source boxes, available confidence information, and, where the recognizer exposes them, alternative transcriptions. The parser uses those positions to connect a field back to the place it came from on the photograph.

The pipeline is roughly:

Photograph
  -> native OCR lines and boxes
  -> numeric tokens with units and source positions
  -> ranked field candidates
  -> arithmetic corroboration
  -> reviewed draft
  -> original fuel or charging form

There is no write between text recognition and the original form. A scan produces a prefill the user can correct and save through the same route they already use for manual entry.

Rank readings before choosing a number

Numeric normalisation is the first place where the text stops being simple. Receipts can use decimal commas, decimal points, grouping spaces, Swiss apostrophes, or native digit sets. Thermal printing can also lose a separator or turn a digit into a letter.

The tokenizer can retain more than one plausible reading of a number. A token such as 1,899 does not have to be resolved before the parser has seen its label, unit, and neighbours. A possible repair has a lower standing than an ordinary reading, and its source location stays attached to it.

Candidate ranking starts with printed evidence. An explicit quantity unit is strong evidence for quantity. A rate such as EUR/L is strong evidence for unit price. A total label is evidence for the reference amount.

Dates, percentages, distance readings, and administrative labels are excluded from the amount slots. An unlabelled number does not become a missing quantity because it happens to make the calculation convenient. The search keeps at most twenty candidates per slot, which bounds the combinations it considers.

Multiplication is corroboration

The arithmetic seems like the easy part:

quantity * unitPrice = printedTotal

It is useful, but it needs a smaller claim than "the receipt is correct". A matching product corroborates a candidate reading. It does not prove that the image was read correctly or that all three fields describe the transaction I think they describe.

The parser compares the product with half a currency minor unit, plus floating point slack. For a currency with two decimal places, the base tolerance is half a cent. It does not allow a percentage of the bill, which would make larger receipts accept larger OCR errors.

This example is used in the tests:

31.93 L x 1.899 EUR/L
TOTAL 60.84 EUR

The product is approximately 60.635. A loose percentage tolerance could accept 60.84. The currency-based comparison rejects that arithmetic match.

Another test has a perfectly readable quantity and price, a subtotal that matches their product, and a lower final paid amount:

40 L x 1.80 EUR/L
Subtotal 72 EUR
Discount 4 EUR
Total 68 EUR

The parser retains 68 as the printed total reference. It does not promote 72 to the final total just because 72 makes the multiplication work. Quantity and unit price can still be useful even when the final amount includes a discount.

OdoKeep calculates the saved cost from quantity multiplied by unit price, minus an optional reviewed discount. The printed total is not copied into the saved cost. It also cannot supply a missing quantity or a missing unit price through division.

If only the total was recognized, the user still needs to enter the amounts. If quantity was recognized and price was not, the prefill leaves price empty. A partial result is allowed to remain partial.

For scans that do not pass the arithmetic check, the app can request another reading with higher contrast. The two passes are compared by arithmetic corroboration and then by the amount fields they recovered. A tie keeps the original pass. Date, time, station, and other non-amount fields can fill gaps from the losing pass because both readings refer to the same photograph.

Even that better reading goes through review. More evidence changes the ranking; it does not remove the user's opportunity to correct it.

Units and currencies need explicit choices

Units create their own class of believable errors. Litres, US gallons, imperial gallons, kilograms, cubic metres, and kilowatt-hours are all valid in the app's fuel or charging domain. They are not interchangeable.

A receipt that prints only gal cannot settle which gallon it means. The importer asks the user. If a compatible liquid unit is converted, quantity is multiplied by the volume factor and unit price is divided by the same factor:

convertedQuantity = printedQuantity * factor
convertedUnitPrice = printedUnitPrice / factor

This preserves their product before input rounding. The importer rounds converted inputs to at most three decimal places, so the original form's recalculated cost can differ slightly from the printed reference. Keeping the printed unit avoids that conversion rounding.

Physical dimension conflicts are handled more cautiously. A quantity printed in litres and a price explicitly printed per kilowatt-hour cannot be combined into a fuel cost. The parser retains the quantity and its unit and leaves price empty. It does not reinterpret one dimension as another.

Currency is equally explicit. If the receipt names a different currency, the user can keep it. If they choose the configured currency instead, the importer keeps quantity and clears unit price. No exchange rate is assumed during that choice. Displaying a saved foreign-currency cost later is a separate responsibility of the app's dated conversion layer.

Regional context comes from the device region independently of the interface language. Someone using an English interface in Portugal should not have every receipt interpreted as American. Explicit currency codes and units carry stronger evidence than regional defaults.

Recognition and photograph lifecycle

On iOS, Vision uses accurate recognition with language correction disabled. That is a deliberate setting for this input: numeric quantities and abbreviated product lines do not benefit from every assumption that helps a language model repair prose. Supported language hints are checked against the configured Vision request before being supplied.

On Android, the app bundles the Latin, Chinese, Devanagari, Japanese, and Korean text recognizers described in the ML Kit text recognition documentation. Bundling increases binary size and makes those recognizers available offline. The document-scanner acquisition path has separate Google Play services requirements and may need an initial download; the picker and bundled OCR provide a fallback. This does not establish support for every writing system.

Image handling matters just as much as field ranking. Orientation is normalised into the pixels, and the retained OCR boxes must agree with the displayed photograph. Both native implementations can try rotations when the initial reading is sparse, but a few extra fragments are not enough to justify turning the image.

Temporary copies belong to a scan session. A cancelled session cannot write a late photograph after cleanup, and one review cannot remove another review's image. A draft also belongs to the form that requested it and can be consumed once. Returning to that form again must not overwrite the user's corrections with the old scan.

Test interpretation separately from recognition

The parser tests include two real receipt transcriptions and synthetic cases for regional formats, ambiguous units, discounts, bad dates, alternate readings, and partial handoff. Those tests check what the parser does with OCR text. They are not a measured recognition rate on an image corpus. Low light, faded print, and script coverage still need device-level evaluation.

That separation makes failures easier to investigate. If the characters are wrong, I look at acquisition and recognition. If the characters are right but the record is wrong, I look at the parser's evidence and the handoff. A complete-looking JSON object would hide the difference.

Receipt scanning is part of OdoKeep. If you keep fuel or charging records, you can try the workflow in the iOS app: photograph the receipt, inspect what was recovered, and finish in the ordinary form.