TextStruct

Problem#

Scanned textbooks are a poor fit for conventional RAG pipelines. A page of a textbook isn't a flat block of text — it's multi-column layout, tables, figures, worked examples, and definitions that all carry different pedagogical weight, and generic chunking treats them identically. Cloud OCR services also don't know or care that the document in front of them is educational material; they're tuned for general documents, not for the specific structure a textbook has.

TextStruct is a fully local, offline pipeline that turns scanned textbook PDFs into structured, RAG-ready data — without sending a single page to a cloud API.

Motivation#

TextStruct began as an experiment for AIPathshala: the question was whether classifying chunks by pedagogical role (definitions, examples, exercises, summaries, and similar) could meaningfully improve retrieval quality over conventional flat chunking. Building it as a standalone, local-first pipeline — rather than inside the production product directly — made it possible to iterate on that question independently of AIPathshala's own release cycle.

Architecture#

The pipeline runs as a sequence of local stages, each with its own module in src/:

1. PDF → images — scanned pages are extracted to images ahead of OCR.
2. Layout-aware OCR — DocTR runs locally, with a dedicated layout subsystem (multi-file: grouping, reading order, table detection, figure linking) resolving multi-column reading order rather than reading a page as one flat text stream.
3. Cleanup — OCR artifacts, headers, and footers are stripped.
4. Math refinery — deterministic, regex-based cleanup of OCR'd math, with an optional Ollama-backed LLM fallback for cases the deterministic pass can't resolve.
5. Section/chapter inference — structural boundaries are inferred from the cleaned text.
6. Semantic chunking — chunks are built and merged using embedding similarity, not fixed token windows.
7. Pedagogical role classification — a three-tier cascade (fast heuristics → an optional LLM pass → a tiebreaker) assigns each chunk one of ten roles: definition, explanation, procedure, example, question, result, reference, evidence, visual_proxy, context.
8. Vision extraction — figures and tables get CLIP ViT-B/32 embeddings (with an experimental ColPali encoder as an alternative), enabling image-aware retrieval alongside text.
9. Storage and retrieval — chunks and vision embeddings land in LanceDB, queryable by text, image, or a weighted hybrid of both, with SQL-style metadata filters.

Show source

Core Features#

• Layout-aware OCR with multi-column, table, and figure detection.
• Deterministic math cleanup, with an optional LLM fallback for harder cases.
• Ten-category pedagogical role classification, in both a fast heuristic mode and a slower, more accurate LLM-assisted mode.
• Hybrid text + vision retrieval: query by text, by image, or by a weighted combination of both.
• SQL-style metadata filtering over the vector store.
• A migration tool (with its own integrity-verification pass) for moving from an earlier pickle-based store to LanceDB.
• GPU-accelerated OCR mode alongside a forced-CPU mode, so the pipeline runs on either.
• An interactive query REPL, and debug modes that dump before/after cleanup text and annotated layout images.

Technical Decisions#

The core constraint driving TextStruct's design is that it never calls a cloud API — OCR, role classification, embeddings, and vision extraction all run locally. That constraint shaped several specific decisions: DocTR over cloud OCR services for layout-aware text extraction; a three-tier role classifier (heuristic first, LLM only when asked, a tiebreaker to resolve disagreements) rather than an LLM-only classifier, so the common case stays fast and free of network/API dependencies; and LanceDB as the vector store, chosen for its embedded, file-based operation (no separate server process) and native support for hybrid SQL-filtered vector search.

The pipeline also documents an explicit, un-fixed trade-off rather than hiding it: OCR results are not currently cached, so every run re-runs OCR from scratch even on unchanged input — a known cost of the current design, not an oversight.

Engineering Challenges#

I don't have reliable recall of the specific implementation struggles here, so rather than reconstruct a narrative that didn't happen, here's what the codebase itself documents as a real, deliberate trade-off: OCR is not cached between runs, and the vision subsystem ships two alternative encoders (CLIP and an experimental ColPali fallback) rather than one, reflecting an explicit choice to keep a faster default path while leaving room for a slower, potentially more accurate alternative on image-heavy pages.

Screenshots#

No screenshots are included yet — TextStruct is a CLI/pipeline tool without a graphical interface. A terminal capture of the query REPL, or one of the pipeline's own --debug-layout annotated bounding-box images, would be a natural fit here once supplied.

Lessons Learned#

The biggest takeaway was recognizing when experimentation has diminishing returns. Building a custom document-processing pipeline from scratch was valuable for genuinely understanding the problem — multi-column layout, math cleanup, and pedagogical structure are all harder than they look — but it also became clear that mature libraries like Docling offer a more robust, maintainable foundation for a production system than continuing to extend custom infrastructure would. That conclusion directly shaped AIPathshala's own architecture: TextStruct validated that pedagogical role classification was worth pursuing, while also demonstrating where adopting existing tooling made more sense than maintaining a bespoke pipeline.

Current Status#

Actively developed, pre-release: 31 commits since the repository was created on 2025-12-31, most recently a release-readiness pass (2026-07-09) that added an MIT license, made the Docker setup CPU-default, and added a runnable pytest configuration. No tagged releases or packaged version exist yet.

Tech Stack#

Python, with PyTorch and DocTR for OCR, sentence-transformers for text embeddings, open-clip-torch (CLIP ViT-B/32) for vision embeddings with an experimental ColPali fallback, LanceDB for vector storage, and spaCy/NetworkX for the pedagogical-bundling work in progress. Ships with both CPU and GPU Docker configurations.

Related Content#

TextStruct's central experiment — pedagogical role classification improving retrieval quality — directly informed the architecture of AIPathshala, the production education platform it was originally built to evaluate ideas for.

Browse the rest of my personal projects.