~hyperlinks

Right. But why not build that into the tokenizer?

> ... but maybe I am missing their insight here.

this?


> Essentially, DocLang is optimized for LLM tokenizers through markup that maps between DocLang elements and LLM tokens on a 1-to-1 basis. The spec relies on a limited XML vocabulary that aligns with LLM tokenizers to produce optimized prompts. It is lossless, so the AI conversion doesn't do away with valuable info. It's designed to support common graphical elements like tables, formulas, charts, and multimodal content. And it's an open standard.
>
> DocLang could also help keep costs under control. According to [AI Cost Check](https://aicostcheck.com/blog/ai-ocr-document-processing-costs-2026), having an AI model conduct an OCR scan on a PDF requires about 1,200 input tokens and 150 output tokens as a baseline.

0xbitcoiner

I built this pipeline with mineru last year to get markdown from pdf - it used to perform better than docling, but I haven't compared lately - and for my own meat OCR devices, markdown is better than PDF or MS Word. Easier to read, and I can grep it.

I'm not sure why they think markdown is poor for LLMs though - all the current models have been trained on it. It's the cleanest format we have? Transitioning back to XML again is a bit awful imho, but maybe I am missing their insight here. Luckily it should be easy to convert markdown to XML.

A modest proposal: Reformat everything to make documents more palatable to AI

I built this pipeline with mineru last year to get markdown from pdf - it used to perform better than docling, but I haven't compared lately - and for my own meat OCR devices, markdown is better than PDF or MS Word. Easier to read, and I can grep it.

I'm not sure why they think markdown is poor for LLMs though - all the current models have been trained on it. It's the cleanest format we have? Transitioning back to XML again is a bit awful imho, but maybe I am missing their insight here. Luckily it should be easy to convert markdown to XML.