Teaching an API to Read the Catalog

This Express platform is the backbone API for a wholesale operation: one Node.js service that fronts a legacy IBM DB2 system and serves everything from public product browsing to internal staff tooling, nightly reporting, marketing syncs, and data pipelines. A service with that many masters has two problems baked in from day one. The first is identity: wildly different audiences, from anonymous shoppers to B2B integrations to internal staff, all arrive at the same codebase, and each deserves a different level of trust. The second is that the business still revolves around a physical artifact: the printed product catalog.

Customers order from the printed catalog, so the website has to answer "which page is this product on" for thousands of SKUs, and the answer changes with every catalog season. Before this pipeline, page assignments were manual data entry: a person paging through a several-hundred-page catalog, SKU by SKU, for weeks. The data was late, the work was mind numbing, and mistakes surfaced as customers failing to find products they were looking at in print.

This dive covers the two designs that answer those two problems: a four-tier authentication architecture where a route's trust level is visible in the file tree, and an OCR pipeline that reads the finished catalog the way a customer does.

• Four genuinely different audiences. Public visitors need catalog data with no identity, logged-in customers carry a storefront session, B2B clients authenticate programmatically with tokens, and internal staff need operations endpoints that must never be reachable with any customer credential.
• The catalog is the source of truth. The deliverable that matters is the printed book in the customer's hands. Whatever generates page data has to agree with that artifact, not with an upstream layout file that may have changed after the proof.
• SKU formats are not uniform. Different product lines print SKUs with different shapes and conventions, some from outside vendors, so no single pattern reliably extracts all of them from raw page text.
• Cloud OCR is slow and priced per call. Text detection over hundreds of pages is an expensive, minutes-long operation that has no business running twice for the same page, and absolutely no business running inside a web request.

Option A: keep the manual process. A person with the catalog and a spreadsheet is flexible and accurate on any page a human actually reads carefully. It lost on everything else: weeks of turnaround per catalog, no repeatability, and accuracy that degrades exactly as attention does, on page 300 of 400. It did not disappear entirely, though: manual and CSV-sourced assignments remain an input lane for the cases automation should not guess at.

Option B: pull page data from the layout source. In principle the catalog's own production files know where every product sits. In practice catalog production lives in design tooling whose output is visual, not structured; SKU placement is not exported as data, and the layout can shift between the working files and the final proof. Building an extraction against the design pipeline would chase a moving, unstructured target and still not guarantee agreement with the printed result.

Option C: OCR the finished PDF. Read the artifact itself. The reason this works, where naive OCR projects fail, is a single design decision: the pipeline never asks OCR to read SKUs blind. It asks a much easier question: "which of these known SKUs appear on this page?" The complete SKU list already lives in the legacy database, so extraction becomes confirmation, and OCR noise mostly stops mattering because a smudged character no longer invents a SKU that does not exist.

The auth architecture makes trust a property of placement. Routes live in four separate route files, one per tier, each wired to its own middleware chain: tier one is public with no identity, tier two requires a customer storefront session, tier three validates a token against B2B API credentials in the legacy database, and tier four requires an internal staff session that is entirely separate from customer sessions. A separate internal-service tier with its own authentication handles machine-to-machine calls. The point of the physical separation is that a route cannot quietly skip authentication: where a route is declared is what protects it, and a reviewer can audit trust levels by reading the file tree.

The catalog pipeline runs as an internal tool, never in the request path. The finished PDF is split into page images, and each page goes through cloud text detection exactly once, with raw results cached to disk as JSON keyed by page. Every downstream step reads the cache, so reruns are free and iterating on matching logic never re-buys OCR. Matching runs the known SKU list against the page text through a set of per-product-line matchers, each tuned to how its line formats SKUs in print, including vendor lines with their own conventions. Results are reconciled: duplicates collapse, legitimate multi-page appearances keep every page, manual and CSV lanes merge in, and a difference pass against the previous run turns review into "look at what changed" instead of "re-check the whole catalog."

The output is deliberately not a direct database write. The pipeline emits a CSV of SKU-to-page assignments that a human reviews before it batch-updates the legacy database. Automation does the reading; a person keeps the judgment call on anything surprising.

Catalog page assignment went from weeks of manual entry to an automated pass plus a human review of the diff, repeatable every catalog season, and the same page data now feeds the storefront's "find it in the catalog" experience. The tier architecture outlived its first job: the same four-file structure absorbed years of new endpoints, and later services adopted the pattern, because "which tier does this belong to" turned out to be the right first question for every new route.

The honest tradeoffs: the per-line matchers are a maintenance surface. Every new product line or vendor with a novel SKU format means another matcher, and the extraction quality is only as good as the assumption that the SKU master list is complete. The disk-based OCR cache is simple and effective, but it is bound to one machine; a portable artifact store would make the pipeline runnable anywhere. And tier three's token model, one static credential per API client, is serviceable for trusted B2B partners but is the piece most worth upgrading, which is exactly the direction later platform work took with scoped, per-client credentials.

The design decision that aged best is the smallest one: matching known strings instead of reading unknown ones. It turned OCR from a fuzzy research problem into a boring lookup, and boring is what you want in a pipeline the business quietly depends on every season.