Anthillo /blog
← All posts

Readable Doesn't Mean Usable

Mikołaj Wojciuk

Software and Data Engineer ·

Text documents, presentations and spreadsheets have always been created with one audience in mind: a human. Today, however, the same materials increasingly become a source of knowledge for AI assistants that are supposed to analyse or search them. But before a file makes it into such an agent’s knowledge base, it has to be “translated” into a format a computer can understand. And this is where the problems begin, because it turns out that things our brain doesn’t even think about while reading can be a hard obstacle for a computer.

A document is an interface for humans

When we look at a document, we rarely think about how much information we take in before reading a single word. We know where a new section begins, what is a heading and what is a footnote, which number belongs to which table row and which chart a given caption describes. So part of the meaning comes not from the words themselves, but from how they are placed on the page.

This is, of course, no accident, but the result of years of practice. Documents are designed so that a human can grasp their structure as quickly as possible. Text size, position on the page, colours, spacing, columns and even table borders act as cues that we often don’t consciously notice. To us, a table is a table because we see rows and columns. A caption under an image refers to the image because it sits right below it. A heading starts a chapter because it looks different from the rest of the text.

In a sense, a document is a bit like a user interface. It doesn’t just store content; it also hints at how that content should be read. When we strip away its visual layer, we do keep all the raw data, but we lose the information that came with it. And that is exactly why something completely obvious to a human becomes completely unreadable to an algorithm once it is “flattened” into plain text.

The computer doesn’t get the same document

This brings us to the first problem. What is a document to us doesn’t have to be one to a computer at all. We open a PDF and see a coherent report. A program sees a collection of text fragments dropped at specific positions on the page, separate images, loose lines and a mass of technical information. Only from this chaos does it have to reconstruct what was clear to us at first glance.

A lot also depends on the format itself. A Word document stores the structure of paragraphs and headings, and a spreadsheet is naturally divided into cells, rows and columns. PDF files, on the other hand, are often just a set of instructions specifying where and how to draw a given character on screen. From our human point of view, all these files may contain exactly the same table, but the way a computer has to extract it will be completely different each time.

That is why, before such material becomes useful to an AI assistant, it has to pass through an additional layer: one that extracts the text, recognises individual graphical elements and gives them a new structure. At this stage a hard question arises: what exactly should we preserve from a given document?

After all, we can extract all of its text and technically not lose a single word, yet still irreversibly lose information and turn the narrative into a flat stream of characters. It’s a bit like pouring a finished jigsaw puzzle back into the box: the logic, the relationships between pieces of data and the “navigation” that guided our eyes are gone. In theory all the pieces are still there, but let’s be honest, an assembled puzzle is much nicer to look at.

Takeaway. We can extract all the text from a document and technically not lose a single word, yet still irreversibly lose information.

Information also hides in structure

What may not be entirely obvious is that two documents that look almost identical on screen can be completely different worlds to a computer. It all depends on whether the structure we see actually exists in the file, or has only been simulated visually.

Take an ordinary heading. In a word processor it may be marked with the proper style, say Heading 2. But someone could just as well have written a new paragraph, increased the font size and made the text bold. The result? Practically the same for a human. From the computer’s perspective, though, the difference is huge: in the first case it receives hard information, “this is a section title at a certain level of the hierarchy”, while in the second it sees only a set of visual features from which it has to guess that role.

The same goes for tables. One may be a real grid of connected cells, while another may simply be a pasted image. On screen they will look identical, but from a data-processing standpoint they are entirely different challenges.

It’s even more striking in spreadsheets. We see the value 42 in a cell, while inside the file there may be a constant number, text, a complex formula or even a result fetched live from an external source. What we see is only the final representation of something much richer.

From an algorithm’s perspective, what lies “under the hood” can therefore be far more valuable than the image on screen. The file format, semantic markup and embedded formulas hand the computer ready-made answers, freeing it from having to guess the structure from the text layout.

And this is where things get really interesting. It turns out that not every way of “reading” a document can reach this hidden knowledge.

A parser and a multimodal model see differently

Until now, we assumed that for a computer to work with a document, it first had to be taken apart piece by piece. And indeed, that’s exactly what a parser does: it pulls raw text, coordinates and markup out of the file, then tries to assemble a logical structure from them.

Multimodal models, however, have opened an entirely different door. Instead of analysing the file’s code, we can render an image of the page and have the machine simply “look” at it. This brings us back to our interface metaphor: for a vision model, large text once again acts as a heading, and a grid of lines clearly means a table. The system no longer has to guess the structure from hidden tags, because it reads it directly from the visual layout.

That doesn’t mean “sight” beats parsing. A parser delivers predictable data that is easy to store in a database and quick to search. A multimodal model, in turn, preserves what the parser misses: pure visual context. So we have two different approaches: one analyses structure, the other interprets the image.

This, however, sets a new trap. Even if a powerful vision model handles the analysis of a single page brilliantly, reads the table and answers our question, that is still not enough to say we have built a knowledge base. Understanding one page doesn’t yet mean the system will be able to work with that document at scale.

”Seeing” doesn’t mean “knowing”

When a multimodal model gets an image of a single page, it can link a chart to its caption and precisely answer a question on which a classic parser would simply fail. So a tempting conclusion presents itself: since AI can “look” at documents, let’s just feed it page after page.

The problem, however, is scale. While we can drop a short manual into the model in its entirety, with thousands of pages and a rapidly growing knowledge base this becomes very hard. We can’t force the system to “read” a gigantic archive from scratch for every question, hoping it will find the right fragment.

The ability to analyse an image once is one thing, but building a system that can make lasting use of that data is a completely different story. To search the base instantly, cite exact sources or update individual files, the content has to be durably extracted and organised.

There is a trap we can fall into here: fascination with the capabilities of today’s “AI” can overshadow the fundamentals of data engineering. We still have to decide what exactly we store in the database, how we mark relationships between pieces of information, and how algorithms will later use that data.

The ability to “see” lets a model interpret a document at a given moment. But only turning that into an organised representation creates a knowledge base worthy of the name.

From document to representation

In practice, we rarely have to choose between a parser and a VLM, a vision-language model. The most powerful hybrid systems take the best of both worlds. The parser extracts text and basic structure quickly and cheaply wherever the file’s code allows it. The vision model, in turn, steps in as the expert on charts, complex tables and nested layouts that can’t be read without “looking” at the page.

The output of such a process, however, is no longer a visually pleasing document, but a raw representation optimised for machines. In the simplest version these are so-called chunks, fragments of text wrapped in metadata: information about the file, the page or the heading type. In advanced systems this structure can also store relationships between data, table structure or generated image descriptions.

{
  "text": "A multimodal model can analyse the layout of a document...",
  "source": "technical-report.pdf",
  "page": 14,
  "section": "Multimodal models",
  "type": "paragraph"
}

As a result, from the perspective of RAG systems, a document ceases to exist as a coherent file. It becomes a base of independent records, each containing a portion of knowledge, metadata and a vector representation. Thanks to this, such “atoms of knowledge” can be searched, updated and passed to a language model as precise context independently of one another.

And this is where the illusion of simply “loading a PDF” finally shatters, giving way to the hard architecture of AI systems. We have to decide on a great many things: how best to split text into fragments? How to deal with tables breaking across page boundaries? Should we keep screenshots of specific sections? How to combine semantic, vector-based search with metadata filtering?

Fortunately, we don’t have to answer all these questions in one post. In the next parts we will take a closer look at tools for processing the hardest documents, ways of building a useful knowledge representation, and how to find that knowledge effectively later, both in classic RAG systems and in modern AI assistant architectures.

Readable doesn’t mean usable

So we come back to where we started. A document can be well laid out, clear and simply pleasant to read, and still be quite a problem for an AI system. And that’s not the fault of a badly prepared file. We have simply spent years designing documents for an entirely different audience.

For us, “readable” is a synonym for good layout, a clear hierarchy and sensible tables. For a machine, “usable” means an organised set of data from which relationships can be extracted and stored in a way that makes later search easier.

Of course, this doesn’t mean that from tomorrow we should write reports in JSON files and give up on charts to make life easier for algorithms. A document still has to serve a human. But if we want it to become a source of knowledge for artificial intelligence, we need a layer that translates its visual layout into the language of data.

Takeaway. The fact that a modern model can read a PDF doesn’t yet mean we have a ready knowledge base. Recognising text is only the first step. Only a deliberate decision about what exactly to extract and how to store it gives that information real value.

Designing such mechanisms is far less spectacular work than demos showing AI answering questions about a document. But the truth is that every well-functioning assistant rests on exactly these unglamorous engineering foundations.

← All posts