The document processing
pipeline

Every search begins with access control. The system reads the Passport JSON to determine which organisation the user belongs to and which documents they can access. This ensures that every response contains only documents the user is entitled to see.

The text layer of the document is produced via optical character recognition (OCR). For scanned documents this step is essential — it is the foundation for every downstream operation.

The raw OCR text is converted into structured Markdown. Noise is removed (repeated headers/footers, page numbers), producing clean, consistent text ready for AI processing.

The document is converted into a 6144-dimensional vector (fingerprint), and machine learning recognises its type. The system compares it with trained sample fingerprints and classifies with a 0.9–0.95 threshold.

Based on the detected document type, the system extracts key-value pairs: tax numbers, amounts, dates, names. Three techniques are combined: regular expressions, named-entity recognition (NER), and LLM-based extraction.

The normalised text is placed into an elastic search index. This provides classic text search: exact-phrase, partial match, filters, and similarity search on the structured fields.

Documents are chunked and vectorised: each chunk becomes a 6144-dimensional vector. These are stored in the Qdrant vector database, enabling meaning-based search — matching intent, not just words.

In the final step, elastic and semantic search results are combined. The best matches from both sources are merged and re-ranked: the first pass picks 30 hits, and the Top 10 are returned to the user.