PullMD Part 14: Wiring in Microsoft's MarkItDown — and ripping three parts back out – Simply Explained
PullMD is my little self-hosted tool that turns web pages into clean, readable text. This part of the series does not start with a plan. It starts with an itch.
I had a few HTML files sitting on my computer that I wanted to hand to an AI assistant. The tool already handles web pages. So why not local files too? That became a small new feature: drag a file in, get clean text back out.
The small itch
Most of the guardrails came from Claude: no saving uploaded files to the cache, no browser engine for uploads, and the filename goes in a hidden header instead of the web address so it never lands in the server’s visitor log. All sensible. My own reasoning was simpler: why would you ever cache a file you convert once by drag-and-drop? PullMD is first and foremost a local tool with authentication, and the privacy drama Claude used to justify those points, I did not share.
The whole thing works by drag-and-drop. While testing, the obvious follow-up hit me:
“but on mobile you could turn it into ‘or open file’, right? a downloaded html file is something you might want to convert on mobile too.”
So the drop area became tappable: desktop and mobile, same function behind it, just a different door in front.
A Microsoft library enters the picture
The upload feature ran only on my test instance at first. Two days later I stumbled across Microsoft’s freshly released MarkItDown. It is a free library, released under the MIT license (named after the university where it was created), written in Python. And it clicked. If HTML files work, why not PDF, Word, PowerPoint, Excel, EPUB? And why not YouTube transcripts? Anyone who has ever landed on one of those bloated transcript pages knows why you would rather have it inside your own tool.
I gave Claude the repository and asked: can we wire this into PullMD in a sensible way? Claude analyzed the library and came back with a plan that made immediate sense. MarkItDown handles only the non-HTML formats. For HTML it would be a step backward, because the tool already does that job well.
So another sidecar: a small helper service sitting next to the main one. PullMD is already a small cluster of containers anyway, running Trafilatura (a text extraction tool that pulls the main article out of a messy page) and Playwright (a browser running in the background, for pages that need JavaScript to show their content), both since April. What is new is a Python service that gets called only when a document format comes in that the existing system cannot handle.
So far, so clean. A library for the tedious parsing work, because going through Office formats is nobody’s favourite pastime. MarkItDown covers a whole zoo of them: PDF, DOCX, PPTX, XLSX, EPUB, CSV, JSON, XML. Exactly the kind of work you sensibly hand off rather than maintain yourself. Then I looked at Claude’s plan in detail, and “just wire it in” turned into “wire it in, but not three parts of it.”
Three parts back out
Reading through Claude’s integration plan, two things caught my eye: the YouTube converter was missing without comment, of all things the part that had tempted me the most. And for audio it listed “privacy” as the rationale while the plan still called OpenAI. I pushed back:
“why did you leave out YT transcript? … and what does ‘privacy’ mean if you use openai anyway?”
Claude’s answer walked through the reasons along the code. That became the actual work of this release: not the wiring in, but the deliberate taking back out.
Audio. MarkItDown’s default sends audio files quietly to Google’s speech recognition. For a self-hosted tool, that decision belongs to the operator, not to a silent default. So: a dedicated path where the operator picks the processing engine. Local processing for those who want it, cloud services only for those who deliberately choose them.
YouTube. The YouTube handling went out entirely. Claude’s look into its code found a fragile spot: if a video has no transcript or is blocked, the whole conversion crashes. You lose even the title and metadata, even though they were right there in the HTML. No proxy support on top of that. In its place came a dedicated handler with graceful fallback, meaning it still gives you what it can even when the ideal path fails.
Image captioning. Instead of using MarkItDown’s AI layer, the vision calls run through PullMD’s own system. Zero coupling to MarkItDown for the AI part: a MarkItDown update cannot break any of it.
The principle went into the project memory as one sentence: MarkItDown only for file types; fragile and third-party paths we build ourselves.
Keeping the helper small
The next morning, that sentence grew into a bigger architecture question. I put it on the table:
“are we building our own markitdown replacement? would that be bad design?”
Claude’s answer was no, and it was the right one. Do not rebuild the whole Office, EPUB, and spreadsheet parser zoo. Nobody wants to maintain that. But the AI layer belonged outside. Vision and speech-to-text moved entirely out of the Python sidecar and into the main server. My reasoning for it: “I don’t like making our LLM calls depend on MarkItDown like that. If someone doesn’t need documents at all, they still have to run MarkItDown alongside.”
(LLM calls are the requests to AI models, for things like describing what is in an image or turning speech into text.)
The sidecar shrank to what it does well: documents and YouTube.
At that same spot, a fourth piece nearly wandered in: Docling, IBM’s table engine. It gives the best results locally, and I was briefly thrilled, until Claude supplied the numbers: a 4.4 GB image, several gigabytes of memory per request, 60 to 140 seconds of cold start. What tipped it was not a technical trade-off. It was the picture of how that comes across on a first try:
“if there’s some endless warmup like that, I get frustrated and pin it on ‘what a bad project, this pullmd’.”
Docling is now documented only as a pluggable option, not part of the standard package. The guiding line: a light door for everyone, a heavy self-host door only for those who deliberately open it. Same logic for advanced PDF table extraction: Mistral OCR (OCR stands for Optical Character Recognition, the technology that reads text from images and scans) is available as an option, but opt-in only, so it never quietly costs money.
One adjustment came along in the same rebuild that had nothing to do with MarkItDown. It still ended up being the only real breaking change: the output now consists only of title and content. Source and fetch date live solely in the YAML frontmatter (the small structured-data section at the top of the file) instead of additionally as a line in the text. Nothing is duplicated anymore, AI assistants read fewer unnecessary words. Anyone who needs the old format gets it back via a setting. That turned the release into v3.0.0.
The Fable 5 review
That evening, with v3 finished and the release tag planned, Fable 5 appeared: Anthropic’s new flagship AI model. A brand-new model whose class is built for finding vulnerabilities, the evening before a release that had just gained a public upload path: new attack surface that nobody had yet looked at with hostile eyes.
First use, straight on the real code changes:
“please do a code review and find bugs and security issues.”
Seven real findings, one false positive. The two that gave me the most pause were security issues.
A YAML injection via a special character. The function that puts quotes around text in the frontmatter escaped one type of line break but not another. YAML (the structured-data format) treats a bare carriage return as a line break: an attacker-controlled page title could have smuggled in its own metadata line. The fix was a single extra character in the pattern. Claude had written the code, and it had slipped through every review before this one regardless.
A zip bomb, or DoS (Denial of Service: a way to overwhelm a server until it stops responding), on the public upload path. Anyone could have uploaded a malicious document that eats all the server’s memory. The fix: conversion runs in a throwaway subprocess with a time limit and a memory cap. This point first landed on the “needs real design work” list. I had it fully implemented right away anyway, because a public upload path without a resource ceiling is not something you defer to later.
Then the bugs. A missing safety net meant an error in vision or speech-to-text would have crashed the page instead of failing gracefully. A cache shortcut lost some data when a certain setting was active. An MCP (Model Context Protocol, the channel through which AI assistants talk to tools) path had drifted from the main API (the interface other programs use to talk to PullMD), so one feature was not reachable through it at all. The one false positive turned out to be harmless: the affected data gets thrown away before it ever reaches anyone.
Reading the review stays mandatory. A finding is a hypothesis, not a verdict.
The next morning I spotted a single broken image on a share page. I gave it to Claude to track down and, while at it, asked it to go over the whole release once more. A full sweep came along with it. Image URLs had not been converted from relative to absolute against the source domain. An OAuth secret had not been passed through in any compose file since an earlier version. And the opt-out flag for the breaking change was not reachable through any path: the most important gap, because a breaking change with no reachable opt-out is just a break.
After all of that: 721 Node tests plus 4 sidecar-guard tests were green, up from 644 when the branch was opened.
v3 out the door
Squash-merge, tag, release. Minutes later the community post. No triumphant tone: the thing is out, the latest version now points to v3, done. Before the push, a privacy and secrets scan ran across the whole branch. That mattered more to me than clean commit aesthetics, for obvious reasons that have already been their own topic in this series.
The division of labor was the same across the whole arc: I made the decisions, Claude analyzed and built. What sets this part apart from the previous ones is that the most important decisions were not the wiring in but the deliberate taking back out.
A third-party library spares you the parser zoo. But every path that either breaks easily or quietly ships data outward is one you had better not adopt unchecked. That one you build yourself, or you at least turn it into a deliberate choice instead of a silent default.
Part 14 of 15 in the PullMD series.
← PullMD Part 13: An Old Rule | PullMD Part 15: What Strangers See Better →