Explainable Document Classification: A Rules-Guided and OCR-Assisted Architecture for Automated File Organization

1. Introduction

Organizations rarely store documents as isolated class labels. A bank statement may belong under a financial workstream, a statement subtype, a year, a quarter, and a month. A financing filing may require a different workstream and retention policy. The useful output of classification is therefore often a structured destination plus a defensible explanation, not merely “class 17.”

Real repositories complicate this task. Files arrive as digitally generated PDFs, scans, photographs, word-processing documents, spreadsheets, and mixed-content packages. Filenames can be descriptive, generic, or actively misleading. Dates may represent execution, reporting period, filing, issue, or scan time. OCR may recover useful text but can also substitute digits and letters, fragment layout, or return nothing. A forced classifier hides these uncertainties behind a confident-looking label.

This paper proposes a hybrid architecture for explainable file organization. It uses rules as inspectable sources of domain knowledge, OCR as a conditional evidence-recovery mechanism, optional learned components as bounded evidence producers, and human review as a governed correction path. Its research questions are:

1. How can heterogeneous documents be classified into hierarchical destinations while preserving an auditable chain of evidence?
2. How should a system combine filename, native text, OCR, metadata, and rules without allowing one unreliable signal to dominate?
3. How can abstention and human review prevent uncertain decisions from contaminating a repository or its future rules?
4. How should quality be evaluated when the output includes both a category and a structured folder path?

The contribution is a reference architecture and evaluation method. The work does not claim that rules are universally superior to learned models or that OCR resolves every scan. It argues that document automation should expose its evidence, model uncertainty explicitly, and make rule evolution a controlled software change.

2. Background

2.1 Records classification

Records-management systems organize documents to support retrieval, governance, retention, and accountability. ISO 15489-1 describes concepts and principles for creating, capturing, and managing records [1]. In this context, a classifier changes the information architecture of a repository. A wrong destination can affect discoverability, access control, retention, review scope, and downstream analytics.

The target should therefore be represented as a taxonomy with stable identifiers, display names, parent-child relationships, allowed date grouping, and version history. A human-readable path is a rendered view of that taxonomy, not the identity of the category itself.

2.2 OCR as an evidence source

Tesseract’s architecture evolved from connected-component analysis toward line recognition and recurrent sequence models [2], [3]. Regardless of engine, OCR output is an interpretation of pixels, not authoritative source text. Recognition quality depends on resolution, focus, orientation, script support, compression, noise, and layout. The system must retain the provenance of each extracted token: native text, OCR text, filename, metadata, or human annotation.

OCR should be invoked when native extraction is absent or insufficient, not automatically treated as a replacement for higher-fidelity embedded text. Page sampling, rendering resolution, preprocessing, and time limits must be recorded because they affect which evidence was available to classification.

2.3 Weak supervision and rules

Weak-supervision systems demonstrate how expert-written labeling functions can combine patterns, heuristics, and distant supervision while accounting for disagreement [4]. A rules-guided document system has a related but narrower goal: make domain knowledge explicit and auditable even when a learned label model is not used.

A rule can test a filename token, phrase, regular expression, date pattern, page location, metadata property, or relationship among signals. Rules should emit evidence and a candidate label rather than mutating a file directly. Conflicts must be visible, and a rule’s priority must not be confused with empirical reliability.

2.4 Explainability and risk management

The NIST AI Risk Management Framework emphasizes validity, reliability, transparency, explainability, privacy, security, and accountability as characteristics of trustworthy AI [5]. For document classification, explainability is operational: a reviewer must be able to see why a category was selected, what contrary evidence existed, which rules fired, and why the decision was accepted or abstained.

3. Architectural Method

This paper uses architectural decomposition and failure analysis. The system is separated into ingestion, content characterization, evidence extraction, candidate generation, evidence aggregation, abstention, path rendering, persistence, human review, and rule governance. Each stage has an explicit input, output, provenance record, timeout, and failure state.

The reference architecture is informed by a production-oriented document-classification case study, but private infrastructure identifiers, customer data, and proprietary rule content are excluded. No empirical dataset is analyzed in this paper. Section 9 defines the protocol for a future controlled study.

4. Reference Architecture

4.1 Ingestion and identity

Ingestion validates file type, size, readability, and page count before expensive processing. A content fingerprint supports duplicate detection, but the fingerprint must not be used as a public identifier or cross-tenant lookup key. The original filename, detected media type, byte size, and ingestion time are recorded as metadata; none is assumed to be semantically correct.

Exact duplicates can reuse a prior decision only when the taxonomy version, tenant boundary, and policy allow it. Near-duplicate detection is a separate capability and should not silently merge documents that differ by signature, date, account, or annotation.

4.2 File characterization

The characterizer determines whether content is digitally extractable, image-dominant, mixed, encrypted, malformed, or unsupported. A PDF with one text page and several scanned pages is mixed, not simply “native.” Characterization produces a processing plan that specifies extraction method, page sample, rendering resolution, OCR language, timeout, and resource limit.

Sampling is a risk-control decision. Reading only the first page may be adequate for a cover-led document type but unsafe when signatures, schedules, or decisive clauses appear later. The plan should use taxonomy-specific sampling policies and record every omitted page range.

4.3 Evidence extraction

Evidence is normalized into typed observations:

The source may be filename, native text, OCR, embedded metadata, structural feature, or reviewer annotation. Location identifies a page and optional coordinates. Quality records extractor-specific indicators but is not automatically interpreted as probability.

Normalization includes Unicode handling, conservative whitespace repair, case folding for comparison, and preservation of the original value. Sensitive text must not enter ordinary application logs. Errors report extractor state and page identifiers without copying document content into telemetry.

4.4 OCR fallback

OCR is activated when native text is unavailable, below an evidence threshold, or inconsistent with the visible page. The fallback pipeline may deskew, normalize contrast, and render selected pages at a bounded resolution. It must preserve page aspect ratio and avoid aggressive transformations that change character shapes.

OCR can fail in four distinct ways: unavailable engine, unreadable render, successful recognition with no tokens, and low-quality or contradictory tokens. Collapsing these into “no text found” prevents diagnosis and can create false confidence. The result record therefore distinguishes execution state from recognition content.

4.5 Candidate generation

Candidate generators are independent and side-effect free. Example generators include:

• filename and extension rules;
• phrase, token, and regular-expression rules;
• document-layout or header rules;
• date-pattern and period rules;
• metadata rules;
• similarity or learned-classifier outputs;
• exact-duplicate prior decisions within the same policy boundary.

Each generator emits a candidate category, supporting evidence identifiers, contrary evidence, raw score, rule or model version, and explanation template. It does not emit a final folder path.

4.6 Evidence aggregation

Let c be a candidate category and r_j an evidence-producing rule. A transparent baseline score can be written as

where m_j represents supporting matches, q_k represents contradictions, and weights are versioned configuration. This is a ranking score, not a calibrated probability unless calibration has been demonstrated on held-out data.

Agreement across independent sources should generally carry more weight than repeated matches from the same derived text. Ten phrase matches from one OCR page are not ten independent observations. Aggregation should group correlated evidence to avoid score inflation.

If a learned classifier participates, its score remains one evidence channel. Modern neural-network confidence can be miscalibrated, and post-hoc calibration must be evaluated rather than assumed [6]. The system retains both raw and calibrated scores with model and calibrator versions.

4.7 Abstention and review

Selective classification allows a system to abstain on uncertain inputs rather than force a decision [7]. An acceptance policy can require:

where c^* is the highest-ranked category, c₂ the runner-up, τ_c a category-specific threshold, δ a minimum margin, and C an unresolved-conflict indicator. If calibrated probabilities are unavailable, the same form can use explicitly named decision scores without probability notation.

Abstention reasons include insufficient evidence, close candidates, contradictory dates, unsupported content, OCR failure, stale configuration, timeout, and policy restriction. “Unknown” must remain a first-class result with evidence and reason codes, not a miscellaneous category that absorbs failures.

4.8 Date resolution and path rendering

Document-type inference and date inference are separate. A statement may contain transaction dates, print time, due date, and reporting period. Date resolvers emit typed candidates such as issue date, effective date, filing date, period start, and period end. Category policy determines which type controls folder grouping.

After category and controlling date are accepted, a deterministic renderer maps stable taxonomy identifiers to a path:

For a monthly policy, the rendered suffix may be year/quarter/month; for a non-temporal category, date segments may be omitted. Missing dates produce an explicit undated branch only if the taxonomy permits it. Rendering is deterministic and independently testable.

4.9 Decision record

Every outcome produces an immutable decision record containing:

• source file identity and tenant scope;
• category and taxonomy version;
• rendered path and date policy;
• accepted, Unknown, or review status;
• evidence and contradiction identifiers;
• rules, models, OCR engine, and configuration versions;
• thresholds and score components;
• processing plan, page sample, timing, and failure states;
• reviewer correction and supersession links when applicable.

The record supports audit and replay. A later rule version can re-evaluate the same evidence without erasing what the earlier system decided.

5. Human Review

Human review is reserved for decisions where additional judgment has value. NIST research on human-in-the-loop technical-document annotation demonstrates a machine-assisted workflow in which human expertise remains central to domain-specific analysis [8]. A review interface should show the document, proposed destination, relevant evidence in context, competing candidates, abstention reason, and taxonomy alternatives.

The reviewer may accept, choose another category, correct the controlling date, mark a duplicate, or declare the document out of scope. A correction requires a reason code and becomes a new decision version. Review latency and disagreement are measurable system properties; they should not be hidden behind a simple “accuracy” figure.

Review prioritization should consider expected risk and information value, not only low score. A high-confidence classification into a sensitive or retention-critical category may warrant sampling. Random review samples are also needed to detect confidently wrong automation.

6. Controlled Rule Learning

Human corrections can reveal missing vocabulary, overbroad expressions, date ambiguity, or taxonomy gaps. They should generate rule proposals, not immediate production mutations. This is analogous to software change management:

1. Aggregate de-identified error patterns within an authorized tenant scope.
2. Generate a proposed rule or configuration change with supporting examples.
3. Check overlap, contradiction, and precedence against existing rules.
4. Evaluate on a frozen regression corpus, including previously correct cases.
5. Review the diff and approve it under a named owner.
6. Publish a versioned artifact with rollback metadata.
7. Monitor post-deployment acceptance, Unknown, and correction rates.

A proposal should be rejected if it solves a small error set by overmatching a broader class. Rule priority is not a substitute for resolving semantic conflicts. Tenant-specific rules must not leak examples or vocabulary into another tenant’s configuration without an explicit governance process.

7. Failure and Threat Analysis

The key threat is not only an incorrect label. It is an incorrect label that appears authoritative, is placed into the repository, and later becomes training evidence for the same mistake.

8. Multi-Tenant and Operational Design

Multi-tenant systems require configuration and evidence isolation. Each classification request resolves a tenant-specific taxonomy, rule version, date policy, storage destination, and authorization context. Cross-account or delegated access should use short-lived credentials and least privilege. Secrets and raw document content must not be copied into rule artifacts or diagnostic summaries.

Large documents require explicit resource budgets. The system may impose page, rendered-pixel, OCR-time, memory, and total-latency limits. Limits should produce a structured incomplete result, not a silent sample presented as a full-document decision. Asynchronous processing is appropriate when synchronous gateway limits cannot accommodate the required evidence plan.

Idempotency prevents retries from creating duplicate files or contradictory decisions. A request key should bind the file fingerprint, tenant, taxonomy version, and processing policy. Persistence occurs only after a decision record and destination are validated together.

9. Evaluation Protocol

9.1 Dataset and splits

Evaluation requires an authorized, de-identified corpus stratified by file type, category, scan quality, language, date pattern, page count, and document age. Near duplicates and versions must remain within one split to prevent leakage. Rules and thresholds are developed on training and validation data; the test set remains frozen until final evaluation.

The ground truth includes category identifier, controlling date type and value, expected path, duplicate relationship, and review rationale. Ambiguous documents may have an adjudicated set of acceptable labels rather than a fabricated single answer.

9.2 Classification metrics

Per-class precision, recall, and F1 expose category-specific behavior. Macro-F1 gives equal weight to classes and is therefore useful when common documents would otherwise dominate a micro average [9]. Confusion matrices should be published with class support. Exact path accuracy is reported separately because a correct category can still produce the wrong date bucket.

Metrics include:

• category accuracy, macro-F1, micro-F1, and per-class F1;
• hierarchical accuracy at workstream, category, and leaf levels;
• controlling-date type and value accuracy;
• exact rendered-path accuracy;
• duplicate-detection precision and recall;
• Unknown and human-review rates.

9.3 Selective-classification metrics

Forced-decision accuracy is insufficient for an abstaining system. Let coverage be the fraction of documents automatically accepted and risk the error rate among accepted documents. Report a risk-coverage curve as the acceptance threshold varies. Also report review precision: the fraction of routed cases that contain a real ambiguity or automation error.

Calibration is evaluated with reliability diagrams and a declared calibration error measure. Category-specific thresholds are selected only on validation data. The test set reports both quality and coverage at the frozen thresholds.

9.4 OCR fallback metrics

OCR evaluation measures fallback invocation rate, execution success, useful-evidence yield, character or word error rate on an annotated subset, and classification lift relative to the same pipeline without OCR. The study must also count harm: cases where incorrect OCR changed a previously correct or safely Unknown decision into an incorrect accepted decision.

9.5 Systems and governance metrics

Measure median and 95th-percentile latency by file type and page-count band, extraction and OCR time, timeout rate, peak memory, processed pages, and retry rate. Rule governance is evaluated through proposed-rule acceptance, regression failures, rollback frequency, correction rate before and after release, and cross-tenant isolation tests.

9.6 Ablation study

The evaluation should compare:

1. Filename-only rules.
2. Native-text rules.
3. Native text plus OCR fallback.
4. Hybrid evidence without abstention.
5. Hybrid evidence with calibrated abstention.
6. Hybrid evidence with governed human corrections.

This isolates which components improve accepted-decision quality and which merely increase coverage.

10. Discussion

10.1 Why not use only a language model?

A general model may identify semantic patterns that rules miss, but it does not remove the need for taxonomy versions, date policy, evidence provenance, abstention, access control, deterministic path rendering, or review. If used, model output should enter as versioned evidence with a constrained schema and independent validation. The architecture does not assume that generated explanations are faithful to internal model reasoning.

10.2 Why not use only rules?

Rules are inspectable but can become brittle, overlapping, and expensive to maintain. They are strongest where vocabulary and structure are stable. Learned models can improve recall for variable language, while rules can enforce policy and high-precision anchors. The correct division depends on measured error, review cost, and domain change.

10.3 Unknown as a quality feature

An Unknown state preserves repository integrity when available evidence cannot support a destination. Its value is visible only when evaluation reports coverage alongside error. A system that classifies every file can appear productive while moving uncertainty into hidden misfiling.

10.4 Explanations for different audiences

Reviewers need highlighted evidence and competing candidates. Engineers need extractor states, rule versions, and score components. Auditors need decision history and authorization context. End users need a concise reason and a correction path. One verbose trace cannot serve every audience; explanations should be generated from the same structured decision record at appropriate levels of detail.

11. Limitations

This paper presents an architecture and evaluation protocol, not a completed comparative experiment. It reports no measured classification accuracy, OCR accuracy, latency, cost, or user-study result. Weighted-rule equations are illustrative and require validation and calibration for a particular taxonomy. The design does not establish legal compliance, records-retention policy, or suitability for a regulated workflow. Handwriting, multilingual content, complex tables, adversarial documents, encrypted archives, and audio-visual records require additional study. Third-party OCR and model behavior may change by version and must be re-evaluated.

12. Conclusion

Explainable document classification is a controlled decision pipeline, not a single prediction call. The proposed architecture characterizes each file, extracts provenance-bearing evidence, invokes OCR selectively, generates independent candidates, aggregates evidence without overstating score meaning, and accepts a decision only when policy permits. Category inference, date resolution, folder rendering, human review, and rule evolution remain separate and testable.

The central safety mechanism is the ability to abstain. Unknown and review outcomes prevent incomplete or contradictory evidence from becoming an authoritative repository structure. When corrections are converted into governed, regression-tested rule proposals rather than immediate self-modification, the system can improve without turning individual mistakes into automated policy. This combination of explainability, selective automation, and controlled learning provides a practical foundation for trustworthy document organization.

References

1. International Organization for Standardization, ISO 15489-1:2016: Information and documentation—Records management—Part 1: Concepts and principles, 2016, confirmed 2021. https://www.iso.org/standard/62542.html
2. R. Smith, “An overview of the Tesseract OCR engine,” in Proceedings of the Ninth International Conference on Document Analysis and Recognition, 2007. https://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/33418.pdf
3. Tesseract OCR, “Tesseract User Manual.” https://tesseract-ocr.github.io/tessdoc/ (accessed July 23, 2026).
4. A. Ratner et al., “Snorkel: Rapid training data creation with weak supervision,” The VLDB Journal, vol. 29, pp. 709–730, 2020. https://link.springer.com/article/10.1007/s00778-019-00552-1
5. National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, Jan. 2023. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
6. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning, 2017. https://arxiv.org/abs/1706.04599
7. S. Geifman and R. El-Yaniv, “Selective classification for deep neural networks,” in Advances in Neural Information Processing Systems 30, 2017. https://papers.nips.cc/paper/7073-selective-classification-for-deep-neural-networks
8. J. Kaufman et al., “Human-in-the-loop technical document annotation: Developing and validating a system to provide machine-assistance for domain-specific text analysis,” National Institute of Standards and Technology, 2024. https://www.nist.gov/publications/human-loop-technical-document-annotation-developing-and-validating-system-provide
9. M. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Information Processing & Management, vol. 45, no. 4, pp. 427–437, 2009. https://doi.org/10.1016/j.ipm.2009.03.002

Publication Note