It's possible to create the same PDF in many, many, many ways.

Some might lean towards exporting a layout containing text and graphics from a graphics suite.

Others might lean towards exporting text and graphics from a word processor, which is words first.

The lens of how the creating app deals with information is often something that has input on how the PDF is output.

If you're looking for an off the shelf utility that is surprisingly decent at pulling structured data from PDFs, tools like cisdem have already solved enough of it for local users. Lots of tools like this out there, many do promise structured data support but it needs to match what you're up to.

This is false. PDFs are an object graph containing imperative-style drawing instructions (among many other things). There’s a way to add structural information on top (akin to an HTML document structure), but that’s completely optional and only serves as auxiliary metadata, it’s not at the core of the PDF format.

This is what I kind of suspected but, as I said in my original comment, I'm not an expert and for the PDFs I'm reading I didn't need to delve further because that metadata simply isn't in there (although, boy do I wish it was) so I needed to use a different approach. As soon as I realised what I had was purely presentation I knew it was going to be a bit grim.

Indeed. Therein lies the rub.

Why?

Because no matter the fact that I've spent several years of my latent career crawling and parsing and outputting PDF data, I see now that pointing my LLLM stack at a directory of *.pdf just makes the invisible encoding of the object graph visible. It's a skeptical science.

The key transclusion may be to move from imperative to declarative tools or conditional to probabilistic tools, as many areas have in the last couple decades.

I've been following John Sterling's ocaml work for a while on related topics and the ideas floating around have been a good influence on me in forests and their forester which I found resonant given my own experience:

https://www.jonmsterling.com/index/index.xml

https://github.com/jonsterling/forest

I was gonna email john and ask whether it's still being worked on as I hope so, but I brought it up this morning as a way out of the noise that imperative programming PDF has been for a decade or more where turtles all the way down to the low-level root cause libraries mean that the high level imperative languages often display the exact same bugs despite significant differences as to what's being intended in the small on top of the stack vs the large on the bottom of the stack. It would help if "fitness for a particular purpose" decisions were thoughtful as to publishing and distribution but as the CFO likes to say, "Dave, that ship has already sailed." Sigh.

¯\_(ツ)_/¯

That being said, I think I'm talking about the forest of PDFs.

When I said PDFs have a "markup-like structure," I was talking from my experience manually writing PDFs from scratch using Adobe's spec.

PDFs definitely have a structured, hierarchical format with nested elements that looks a lot like markup languages conceptually.

The objects have a structure comparable to DOM-like structures - there's clear parent-child relationships just like in markup languages. Working with tags like "<<" and ">>" feels similar to markup tags when hand coding them.

This is an article that highlights what I have seen (much cleaner PDF code): "The Structure of a PDF File" (https://medium.com/@jberkenbilt/the-structure-of-a-pdf-file-...) which says:

"There are several types of objects. If you are familiar with JSON, YAML, or the object model in any reasonably modern programming language, this will seem very familiar to you... A PDF object may have one of the following types: String, Number, Boolean, Null, Name, Array, Dictionary..."

This structure with dictionaries in "<<" and ">>" and arrays in brackets really gave me markup vibes when coding to the spec (https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandard...).

While PDFs are an object graph with drawing instructions like you said, the structure itself looks a lot like markup formats.

Might be just a difference in choosing to focus on the forest vs the trees.

That hierarchical structure is why different PDF creation methods can make such varied document structures, which is exactly why text extraction is so tricky.

Learning to hand code PDFs in many ways, lets you learn to read and unravel them a little differently, maybe even a bit easier.

The general principle is that the base content is plain text, which is augmented with markup information, which may or may not have hierarchical aspects. You can simply strip away the markup again and recover just the text. That’s not at all how PDF works, however.

You cite a comparison to JSON and YAML. Those are not markup languages (despite what YAML originally was an abbreviation for, see [0]). (HTML also isn’t DOM.)

[0] https://stackoverflow.com/a/18928199

[1] https://graphmetrix.com/trinpod-server https://trinapp.com

What I really want to do is take all these docs and just reorder all the content such that I can look at page n (or section whatever) scrolling down and compare it between different years by scrolling horizontally. Ideally with changes from one year to the next highlighted.

Can your product do this?

While I'm not convinced it was viable at the business level, it feels like something platform/OS companies could focus on to have a measurable environmental and cost overhead impact.

[1] https://learn.microsoft.com/en-us/azure/ai-services/document...

The only PDF parsing scenario I would consider putting my name on is scraping AcroForm field values from standardized documents.

The article is in the context of an internet search engine, the corpus to be converted is of order 1 TB. Running that amount of data through an LLM would be extremely expensive, given the relatively marginal improvement in outcome.

For the first I can run a segmentation model + traditional OCR in a day or two for the cost of warming my office in winter. For the second you'd need a few hundred dollars and a cloud server.

Feel free to reach out. I'd be happy to have a chat and do some pro-bono work for someone building a open source tool chain and index for the rest of us.

I've found Google's Flash to cut my OCR costs by about 95+% compared to traditional commercial offerings that support structured data extraction, and I still get tables, headers, etc from each page. Still not perfect, but per page costs were less than one tenth of a cent per page, and 100 gb collections of PDFs ran to a few hundreds of dollars.

For longer PDFs I've found that breaking them up into images per page and treating each page separately works well - feeing a thousand page PDF to even a long context model like Gemini 2.5 Pro or Flash still isn't reliable enough that I trust it.

As always though, the big challenge of using vision LLMs for OCR (or audio transcription) tasks is the risk of accidental instruction following - even more so if there's a risk of deliberately malicious instructions in the documents you are processing.

The problem of getting a representative body is (surprisingly) much harder than the annotation. I know. I spent quite some time years ago doing this.

The only way to solve that is with a segmentation model followed by a regular OCR model and whatever other specialized models you need to extract other types of data. VLM aren't ready for prime time and won't be for a decade on more.

What worked was using doclaynet trained YOLO models to get the areas of the document that were text, images, tables or formulas: https://github.com/DS4SD/DocLayNet if you don't care about anything but text you can feed the results into tesseract directly (but for the love of god read the manual). Congratulations, you're done.

Here's some pre-trained models that work OK out of the box: https://github.com/ppaanngggg/yolo-doclaynet I found that we needed to increase the resolution from ~700px to ~2100px horizontal for financial data segmentation.

VLMs on the other hand still choke on long text and hallucinate unpredictably. Worse they can't understand nested data. If you give _any_ current model nothing harder than three nested rectangles with text under each they will not extract the text correctly. Given that nested rectangles describes every table no VLM can currently extract data from anything but the most straightforward of tables. But it will happily lie to you that it did - after all a mining company should own a dozen bulldozers right? And if they each cost $35.000 it must be an amazing deal they got, right?

Second, I converted pdf into pages of jpg. Gemini performed exceptional. Near perfect text extraction with intact format in markdown.

Maybe there's internal difference when processing pdf vs jpg inside the model.

Disclaimer: I'm the founder.

(https://xkcd.com/927/)

Also unlike PDF, I've never seen it actually used in the wild.

Shameless plug: I use this under the hood when you prefix any PDF URL with https://pure.md/ to convert to raw text.

First it's all the same font size everywhere, it's also got bolded "headings" with spaces that are not bolded. Had to fix my own handling to get it to process well.

This is the search engine's view of the document as of those fixes: https://www.marginalia.nu/junk/congress.html

Still far from perfect...

Heh, in my experience with PDFs that's a tautology

- The header is parsed in a way that I suspect would mislead an LLM: "BRETT GUTHRIE, KENTUCKY FRANK PALLONE, JR., NEW JERSEY CHAIRMAN RANKING MEMBER ONE HUNDRED NINETEENTH CONGRESS". Guthrie is the chairman and Pallone is the ranking member, but that isn't implied in the text. In this particular case an LLM might already know that from other sources, but in more obscure contexts it will just have to rely on the parsed text.

- It isn't converted into Markdown at all, the structure is completely lost. If you only care about text then I guess that's fine, and in this case an LLM might do an ok job at identifying some of the headers, but in the context of this discussion I think ai.toMarkdown() did a bad job of converting to Markdown and a just ok job of converting to text.

I would have considered this a fairly easy test case, so it would make me hesitant to trust that function for general use if I were trying to solve the challenges described in the submitted article (Identifying headings, Joining consecutive headings, Identifying Paragraphs).

I see that you are trying to minimize tokens for LLM input, so I realize your goals are probably not the same as what I'm talking about.

Edit: Another test case, it seems to crash on any Arxiv PDF. Example: https://pure.md/https://arxiv.org/pdf/2411.12104.

Fixed, thanks for reporting :-)

I would wager that they’re using OCR/LLM in their pipeline.

https://mistral.ai/news/mistral-ocr

The solution is OCR. Don't fuck with internal file format. PDF is designed to print/display stuff, not to be parseable by machines.

Which is different from extracting "text". Text in PDF can be encoded in many ways, in an actual image, in shapes (think, segments, quadratic bezier curves...), or in an XML format (really easy to process).

PDF viewers are able to render text, like a printer would work, processing command to show pixels on the screen at the end.

But often, paragraph, text layout, columns, tables are lost in the process. Even though, you see them, so close yet so far. That is why AI is quite strong at this task.

Regarding tables, this here https://www.npmjs.com/package/pdf-table-extractor does a very good job at table interpretation and works on top of pdf.js.

I also didn’t say what works better or worse, neither do I go into PDF being good or bad.

I simply said that a ton of problems have been covered by

The purpose of my original comment was to simply say: there’s an existing implementation so if you’re building a pdf file viewer/editor, and you need inspiration, have a look. One of the reasons why mozilla is doing this is to be a reference implementation. I’m not sure why people are upset with this. Though, I could have explained it better.

My use-case was specifically testing their performance as command-line tools, so that will skew the results to an extent. For example, PDFBox was very slow because you're paying the JVM startup cost with each invocation.

Poppler's pdftotext utility and pdfminer.six were generally the fastest. Both produced serviceable plain-text versions of the PDFs, with minor differences in where they placed paragraph breaks.

I also wrote a small program which extracted text using Chrome's PDFium, which also performed well, but building that project can be a nightmare unless you're Google. IBM's Docling project, which uses ML models, produced by far the best formatting, preserving much of the document's original structure – but it was, of course, enormously slower and more energy-hungry.

Disclaimer: I was testing specific PDF files that are representative of the kind of documents my software produces.

The PDF itself is still flawed, even if pdf.js interprets it perfectly, which is still a problem for non-pdf.js viewers and tasks where "viewing" isn't the primary goal.

Looks like you’ve found my PDF. You might want this version instead:

PDFs are often subpar. Just see the first example: standard Latex serif section title. I mean, PDFs often aren’t even well-typeset for what they are (dead-tree simulations).

[1] No sarcasm or truism. Some may just want to submit a paper to whatever publisher and go through their whole laundry list of what a paper ought to be. Wide dissemanation is not the point.

We get to learn a lot when something is new to us.. at the same time the untouchable parts of PDF to Text are largely being solved with the help of LLMs.

I built a tool to extract information from PDFs a long time ago, and the break through was having no ego or attachment to any one way of doing it.

Different solutions and approaches offered different depth or quality of solutions and organizing them to work together in addition to anything I built myself provided what was needed - one place where more things work.. than not.

pdftotext -layout input.pdf output.txt

pip install pdftotext