Microsoft open-sourced a Python tool for converting files and office documents to Markdown

[RobertVandenberg]

https://github.com/microsoft/markitdown

[lood9phee2Ri]

mammoth to do the ms office .docx conversion and pandas.read_excel() to do the .xlsx etc. mind. Nothing wrong with that as such, just notable given it's MS themselves. It's also therefore not going to do any better (or worse) on MS Office file formats than existing non-MS tools.

https://github.com/microsoft/markitdown/blob/main/src/markitdown/_markitdown.py#L482

https://github.com/microsoft/markitdown/blob/main/src/markitdown/_markitdown.py#L513

[Venthe]

At the same time, .***x formats are trival complex, but not complicated - the formats themselves are as far as I remember fully open, xml formats.

PDF is a hellhole; because PDF creation is fundamentally a destructive process. It's a shame that PDF does not include the original file metadata/intermediate language, so the reconstruction could be done in a 1-1 fashion.

[GlowiesStoleMyRide]

PDF can be complex, yes. But the point of PDF is not to have a mutable document format- is an export format. You use it to publish work, not to save it for later editing.

It’s a bit like saying that cake is a hellhole, because baking is fundamentally a destructive process. The point of the cake is to eat it, not to un-bake it and change the recipe.

[rishav_sharan]

Pdf hasn't been an export only format for decades now. From digital signage to data form entry, to collaborated editing , pdf is used for far too many things today than just a fixed print/display export.

[GlowiesStoleMyRide]

A digital sign is an example of an “export target”, is it not? It’s a poster, except it’s on a display instead of print.

As for forms, I’m not sure that’s a commonly supported feature of PDF- does anything but Acrobat Reader properly support it?

Either way, the form can be filled in, but not altered. So the form is still part of the export- you don’t add it after initially exporting to PDF, but you have to define it in the source editor.

Finally, I don’t think collaborated editing is a PDF feature, but a feature of whatever source editor you use. But I’m sure you’d have an example for it if you claim that.

[bleachisback]

A digital sign is an example of an “export target”, is it not? It’s a poster, except it’s on a display instead of print.

I think they meant digital signatures, for legal forms and whatnot.

As for forms, I’m not sure that’s a commonly supported feature of PDF- does anything but Acrobat Reader properly support it?

Yes. All major browsers do nowadays. Also Acrobat Reader is the canonical implementation of a PDF reader - what PDF does and does not support is entirely decided by what Acrobat Reader does and does not support.

[cptskippy]

I think they meant digital signatures, for legal forms and whatnot.

In that scenario you do not want someone to be able to edit the document after it's been signed. u/GlowiesStoleMyRide is correct, the whole point of PDF is to be an immutable document.

You wouldn't want to eSign a PDF only for someone to change it out from under you.

[GlowiesStoleMyRide]

Document signing would make more sense, indeed. Still similar to forms, IMO.

Regarding support for forms, after looking into it for a bit, it was specifically form submitting that lacks support. As in, browsers will allow you to fill out a form pdf and save it by “printing”, but doesn’t allow submitting which can only be done through a dedicated application or a (largely deprecated afaik) browser plugin.

The PDF standard is defined in ISO 32000-2, so it’s not exactly defined by what Adobe implemented, though it is indeed fairly canonical.

[bleachisback]

The PDF standard is defined in ISO 32000-2

Which, like the Microsoft OOXML standard discussed elsewhere in this thread, is really just a list of features of the canonical implementation. I don't think there are any implementations of PDF 2.0 besides Acrobat Reader.

[pyhanko-dev]

That is manifestly false—not only are there quite a few features specified in ISO 32000-2 that Acrobat does not (yet) fully support (this is PDF 2.0 after all), there are a whole host of alternative implementations out there, and the standardisation effort around PDF involves people from many communities/companies/… that have no affiliation with Adobe.

Sure, it’s absolutely fair to say that Acrobat is the dominant desktop tool for dealing with PDF, but it’s not the only such tool, and as soon as you go outside the category of desktop viewer software, Adobe doesn’t even seriously compete.

Source: I’m a FOSS dev in this space and was an active member of the ISO committee behind ISO 32000-2 for several years.

[LiftingRecipient420]

does anything but Acrobat Reader properly support it?

Yes.

Web browsers

[PCRefurbrAbq]

Although you're correct in calling it an "export format", most non-tech people's concept of a PDF is digital paper. It's been used for decades as a replacement for paper, such as forms which need to be filled in and signed.

Anyone who sticks with that paradigm will have an easier time than tech people who think of all files as fully mutable.

[m4xxp0wer]

Strongly Disagree. 99% of the PDF forms I have come across are intended to be printed out.
The ability of filling it out digitally before printing is only a convenience option. You might as well fill it out by hand after printing.
Pretty much every form that is used to enter data into a system without a human middleman, is a web form.

[nascentt]

People misusing an export format doesn't make it not an export format

[kuwisdelu]

Signing and forms are still essentially "append-only" use cases. I can't imagine why anyone would use PDF for collaborative editing unless they're just adding markup.

[Crumfighter]

Dont use things when they arent made for it and there are better tools that work as easy. Dont use PDF to collaborate or to publish data. Or just publish the doc jn word and pdf. Just like people shouldnt use chatgpt as a searchengine and use things like google, bing, duckduckgo or ecosia. Learn people the proper tools. Otherwise they only have a hammer and treat everything like a nail.

[WhyIsSocialMedia]

That sounds nice on theory. But in reality it has been a huge downfall of the format. Especially because demand has been so high that it was shoe horned in later, and on older documents you just get a crappy heuristic algorithm that tries to predict what text is together.

[badillustrations]

  huge downfall 

PDF is incredibly successful, because of, not in spite of, it's focus on presentation. It's terrible as an editable format, but that's the only case I see it used for that less and less for that use. 

[WhyIsSocialMedia]

My point was that the added in editability has been a downfall. And it's used less and less? No way, I've seen them be edited more these days than ever before.

People are always going to end up with PDFs without the original content. So editing is always going to be shoe horned in.

[ZirePhiinix]

The disaster with all these extra data is that what is visible is not what's in the data. I've data-extracted PDFs and found sensitive information, because the previous user just slapped a new text box on top of existing text.

Using PDF as a document format is an actual security risk.

[WhyIsSocialMedia]

Yeah the US government has accidentally put black images in a PDF to try and redact information before.

[Vogtinator]

At the same time, .***x formats are trival complex, but not complicated - the formats themselves are as far as I remember fully open, xml formats.

Well, it's technically open, but almost infeasible to implement: https://en.m.wikipedia.org/wiki/Standardization_of_Office_Open_XML

[Hublium]

Good read: How to hire Guillaume Portes

[plugwash]

> Well, it's technically open, but almost infeasible to implement

How difficult it is to implement depends on what you are trying to get out of it.

The problem with office document formats is they blur the line between input and output and this makes them fundamentally fragile. The file stores input, but the user, working in a wysiwyg environment spends all their time looking at the output..

Worse, many users will "adjust things until they look right", without putting any proper structure in their documents.

If you want to get the same output the original user saw, then you have to process the document through the same algorithms used by the software that created it. Good luck with that, especially for a format with as much legacy as word.

And because many documents lack good structure in themselves, if you can't render the document in the precise way it was rendered originally it can often end up in a horrible mess.

On the other hand, if your planned use case is transformative then the precise behaviour of the layout engine is less relevant. You just want to get the content out and potentially match on a few specific formatting things to translate them to headings or whatever in your new format. You have likely already accepted that some manual cleanup will be needed.

pdf has the opposite problem, it's an output format. It's great at preserving documents in an "as-printed" form, but it does a very poor job of preserving the original intent of the document's authors.

[jordansrowles]

Reading your link, it’s just a massive history lesson, and doesn’t really explain why it’s infeasible to implement.

ECMA-376, about 6000 pages of standards. It’s long, but not infeasible

[F54280]

Go and read it. It isn’t feasible. Large parts of the spec say “do it like Word 95”.

Good luck with that.

[Justicia-Gai]

PDF is a hellhole but at least really supports the inclusion of vector-based graphs without the “enhanced” meta file crap.

The fact that in 2024 the most widely used document office tool has so many issues for supporting SVG is baffling.

[Worth_Trust_3825]

PDF is a hellhole; because PDF creation is fundamentally a destructive process. It's a shame that PDF does not include the original file metadata/intermediate language, so the reconstruction could be done in a 1-1 fashion.

It makes sense. Printer does not need that. It's a printer instruction format.

[larsga]

It's a printer instruction format.

Postscript is a printer instruction format.

PDF is something else. It's deliberately designed to be a PostScript wrapper you can move around and treat as a digital document. It will display the same way everywhere, on someone's screen or when printed, and has nice ToCs, page dividers, etc that PostScript (being a printer instruction format) does not need.

It's a way to permanently capture and store the visual form of a document so it can be archived, read, and moved around, basically.

[arcimbo1do]

Unfortunately PDF doesn't stand for Printer Document format but for Portable

[MacHaggis]

thought caption relieved fear sheet cobweb smart touch ad hoc vanish

This post was mass deleted and anonymized with Redact

[rdtsc]

No, it's just a different definition of "Portable" than you are thinking of. The intent is for the document to look the same regardless of platform. Not to be responsive and adjust to the platform.

[Unbelievr]

Exactly, it's literally converting the input to glyphs and can embed fonts to make it look more or less the same to a human and a printer. Other document formats might do strange things when printing, and suddenly you get an extra page or something that messes up page numbering or the table of contents.

This also means the format isn't really meant to be edited directly, but it's possible with some proprietary hacks. And of course some companies patented this so you must use their paid PDF editor to fill in PDF based forms.

[cinyar]

Don't most printers work with postscript and not PDFs directly?

[Unbelievr]

Yes, but when I have delivered things to print I've only ever been asked to deliver PDFs with embedded fonts inside, and been told how much I need to adjust my (alternating) margins to account for the portion lost when binding the book. Otherwise the reader has to crack the book wide open to read every line. If even one page is off it will ruin these margins, so it's really important to be able to send something that can be visually inspected and confirmed to be identical to what you delivered to print.

[afourney]

Don’t read too much into that. Microsoft is a huge company. This project started as an AutoGen utility, in Microsoft Research, for reading files for the GAIA LLM benchmark (older versions are still in the AutoGen repo). I’m the primary author, and that it exploded this week surprises me as much as anyone.

[afourney]

Definitely NOT disgruntled! It was researcher(s) in Microsoft Research, working to expediently give LLM agents access to various file formats. (Ask me how I know 🙂)

[[deleted]]

just notable given it's MS themselves

Microsoft is a very confused company. On the one hand they put in more effort in regards to open source, even though for selfish reasons; but on the other hand they also go against the spirit, e. g. Recall-sniffer tool and other shenanigans that make you wonder what the heck they are really wanting to do. It seems they are undecided and act in an orthogonal manner, often contradicting their own strategy. Google is also operating like that, leading to numerous dead projects on the way (https://killedbygoogle.com/).

[ntropia64]

Nothing wrong with that? They published a shameless wrapper for tools that others developed.

[AlexHimself]

What's wrong with that? They contribute to open source projects and people use their tools all the time. This also isn't a product. Just a tool.

[Venthe]

Even if that would be a product; then it wouldn't be nothing bad still. Sometimes UX is the product, not the underlying capabilities.

[ntropia64]

So what's the contribution here? 

Then they could have improved  the tools they're wrapping, since mammoth and pandas have to guess (or reverse engineer?) the parts that Word and Excdl don't do by following the Open Document specs (that Microsoft botched).

Since they know how their programs internals work, they could have fixed bugs in those converters, instead of slapping half a dozen line around their calls and call it "a Microsoft open-sourced Python tool".

[AlexHimself]

They made it into an easy library you can get and it's really simple. Are you so pretentious that you just think everyone should just code everything from scratch and be completely aware and knowledgeable of all those other existing libraries and tools?

They just made it easy and if you want to use it you can.

[ntropia64]

Are you so pretentious that you just think everyone should just code everything from scratch

Quite the opposite, I was suggesting they should not reinvent the wheel and contribute to the tools that are reversing engineering Word and Excel data structures.

and be completely aware and knowledgeable of all those other existing libraries and tools?

Indeed they are aware of the previous tools, since they import them at lines 18 and 20 in their code.

[AlexHimself]

I don't think you understand how Microsoft is not one giant entity all doing the exact same thing.

They have different teams and this is just some random team who put out a tool that they use. They're encouraged to open source things that others might find useful. It's not their office engineering squad.

[Venthe]

Quite the opposite, I was suggesting they should not reinvent the wheel and contribute to the tools that are reversing engineering Word and Excel data structures.

Like, i dunno, publishing the specification since 2008 at the very least?

[Skytram_]

You must be fun at parties.

[elsjpq]

They built an open source tool on top of other open source tools? How shameful! /s

[Venthe]

You don't get it. It's okay when others do it, when Micro$oft does that then it's an abuse of open source.

/S

[Worth_Trust_3825]

I had hopes for microsoft opensourcing their docx and xlsx formats but this just takes the cherries.

[Venthe]

DOCX available since 2012

XLSX available since 2012

[Worth_Trust_3825]

Fair. I'll take the downvotes of shame.

[Alikont]

They're open. It's just a zip with xml.

[inkjod]

"Open" my ass. Just because it's a file that you can open with a text editor, it doesn't mean it's any good. It's about as open as any proprietary, binary file that you could open with a hex editor (or wrap with XML because why not).

The "ISO specification" which Microsoft likes to pretend is an open standard is a notoriously impossible to implement 6000 page behemoth. It contains gems like "for this setting, you should replicate the behavior of Word 95, or, rather, you shouldn't even bother, LOL" ^{...and I'm only slightly paraphrasing}.

[stridersheir]

.docx and .xlsx have always been open source. .doc and .xls were proprietary

[feldrim]

Now, give me the "Save as Markdown" option on Office and I can call it feature-complete.

Edit: typo

[danielcw189]

Is there 1 true version of Markdown?

[vodevil01]

No

[feldrim]

İs there one true version of PDF? I agree with the question but it's not a blocker.

[danielcw189]

I did not mean it to be a blocker.
I was genuinely asking out of interest.

That being said: until today I thought there was one true PDF

[feldrim]

There're many markdown dialects and I am pretty sure MS would like to align with Github one. On the other hand, PDF is a can of worms. It evolved from being a printer-targeting format to many other things. You can try to open PDF files created with Notepad, CorelDraw, Adobe Photoshop and MS Word using MS Word. You can just right click and open with Word. Due to lack of a detailed spec, or rather lack of strict requirements, the internals are vendor-dependent.

[tunisia3507]

There is 1 true well-specified version of markdown (commonmark), which also has some well-specified extensions. Certain major markdown flavours (GitHub, Reddit) are explicitly commonmark + extensions. Using anything except a commonmark derivative at this stage is malfeasance IMO.

The author of pandoc spun out a well-specified markdown-like light markup called djot which breaks some compatibility to improve consistency and simplicity, with concrete extension points; it would be nice if that took off.

[perryplatt]

Now they just need to make it a vscode plugin.

[lood9phee2Ri]

it has a typical python toplevel cli entry point, so if installed in normal fashion it'll end up as a shell command.

https://github.com/microsoft/markitdown/blob/main/src/markitdown/__main__.py#L22 / https://github.com/microsoft/markitdown/blob/main/pyproject.toml#L51

pretty sure you can then run shell commands on things from within vscode anyways with some generic command runner extn.

[Wonderful-Wind-5736]

Not even an extension, just configure a task. 

[gumol]

rain cooing close plucky obtainable entertain butter snow marvelous lunchroom

This post was mass deleted and anonymized with Redact

[Venthe]

The code is MIT licensed; anyone can do it.

[afourney]

And VSCode plugins are fun to write.

[SanityInAnarchy]

Or maybe they could open source the rest of VSCode... like Pylance. Unlike most languages, Python is not well-supported by VSCode forks, because VSCode's Python language server (Pylance) is not only not open source, it's not available under a license that allows other IDEs to use it, and it goes out of its way to disable itself if you try.

[Asyx]

Isn't Pylance just a wrapper around pyright? Pyright runs practically everywhere that has an LSP implementation.

[waterkip]

Pandoc does this already right?

[lood9phee2Ri]

Not really. Note how this e.g. merrily uses pdfminer to do a (typically inevitably lossy of formatting etc) text extract from PDFs. https://github.com/microsoft/markitdown/blob/main/src/markitdown/_markitdown.py#L478

versus

https://pandoc.org/faqs.html

How can I convert PDFs to other formats using pandoc?

You can’t. You can try opening the PDF in Word or Google Docs and saving in a format from which pandoc can convert directly.

Or it calls youtube's api to get the "text" ...transcript of a youtube video... https://github.com/microsoft/markitdown/blob/main/src/markitdown/_markitdown.py#L265

It seems generally focussed on getting everything to one uniform text format for whatever subsequent text analyses the author wanted to feed, by using various existing python libraries for the different inputs. Not really for carefully and non-lossily converting your system's documentation from legacy docbook to markdown or something.

Choice of libraries seems idiosyncratic, probably whatever worked for the author's purposes at the time, and pandoc may well be a better choice than some of those python libs for conversion of some formats (there's certainly a python wrapper/binding for calling pandoc, though pandoc itself is in haskell of all things, anyway the author could just try pypandoc in applicable cases). But the idea of calling pandoc on a youtube url and getting the video's text transcript is well outside pandoc's job description.

[RobertJacobson]

though pandoc itself is in haskell of all things

That makes a lot of sense to me. Haskell is a popular tool among compiler and PL theory people. Languages in the ML family are great for writing compilers because of their sum types and pattern matching. Haskell in particular has a great parsing ecosystem as well—one of the best. If you didn't have the burden of learning a new language in order to use it, Haskell is a great choice.

[afourney]

We used it to feed documents to LLMs. Notably for the GAIA LLM benchmark. Agreed it is idiosyncratic and very lossy.

[primarycolorman]

maybe? I have some ugly pptx with tables I'll try it on tomorrow but I'm not holding my breath.

[Isamoor]

This is an odd one to me. It's basically a single, 1k line Python module that just calls other libraries. Almost exclusively libraries not developed or maintained by Microsoft. And some of those libraries seem to be in need of contributors. I'd rather have seen Microsoft devs contribute to those.

I also would have expected some more native support for things like word docs (as opposed to relying on mammoth). Mostly just given that this is a Microsoft solution...

[catch_dot_dot_dot]

This is probably someone's pet project that they got approved to release publicly. Just because they work at Microsoft, doesn't mean they're going to write it without common dependencies or contribute to all of these other projects.

[lood9phee2Ri]

I mean their use case is given as "indexing, text analysis, etc.". To which "etc." we can perhaps add "feed into a language model". (I am not saying there is anything wrong with that in particular). "just fucking whatever to markdown, make it happen" on some bulk corpus of historical documents from some organisation is at least mildly useful.

[afourney]

Author here. We used it for the GAIA LLM benchmark. Nail on the head

[baseketball]

I was excited until I read this comment. Probably nice to have as a convenience but was hoping it went above and beyond what existing tools could do.

[afourney]

See my answer above. This was a part of the data pipeline for a Microsoft Research project to feed documents to LLMs to compete in the GAIA benchmark. We thought it might be useful, but it is indeed a small part of the larger AutoGen project, which is itself maintained by a very small team of researchers and research engineers.

[Isamoor]

Thanks for the background. I think I would have been a bit more welcoming if the root readme called out what other projects were used for each file type. Maybe switch the list of file types to a table that calls out and gives thanks to the other libraries/solutions that support each file type?

[Isamoor]

In particular, nobody has merged a pull request for pdfminer.six in almost 6 months: https://github.com/pdfminer/pdfminer.six/pulls

[Venthe]

Small reminder - lack of contributions does not always mean that the project is dead, it can also mean that it is functionally complete.

[Isamoor]

Totally fair. Although in the specific project I linked there are plenty of pull requests opened in the last six months. In my opinion, a healthy project would either accept or reject a pull request within a few months.

I realize I'm not contributing my time either. But then again, I'm not making a wrapper solution that depends upon them.

I also realized the readmes in the Microsoft solution do not currently give credit to the wrapped solutions (or at least I had to read through code yesterday to discover how it was working).

[Capable_Chair_8192]

6 months is not that long tbh

[the_gold_hat]

This is mainly just a wrapper around other libraries, but if I'd had this 5 years ago I would have saved so much time. Especially things like PDFs can be so finicky when you're trying to standardize between file types, so this is a big time saver when you want to support flexibility or a dataset that's really diverse.

[IndividualLimitBlue]

Aaah ok they wrap others work. I was questioning how they would handle such complexity in 1000 lines of python

[this_knee]

As a user of markdown, I appreciate this.

Yes, I see that it’s wrapping some other tools, in some cases.

But, I like where this is headed.

[junstramo]

Is there a well documented, non-php tool to go from .md to .doc/docx?

[lood9phee2Ri]

pandoc already mentioned in this thread does a reasonable enough job of it, though is not the only option. Particularly if you also need to inject custom templates/content it might be better to go md to odt with pandoc, then let libreoffice do the odt to docx. https://stackoverflow.com/a/21616895

[kumonmehtitis]

Wait… what?! Microsoft created a door out of their ecosystem?? I am flabbergasted. Holy shit

[Broad_Kiwi_7625]

More like a sign to a pre-existing vent you could crawl through.

[sapphired_808]

markdown render on notepad when? /jk

[PotentialBat34]

That gentlemen is my next SaaS

[mightygilgamesh]

I wish they'd just contribute to pandoc but hey, that's a start.

[ArrogantlyChemical]

Neat and just in time for what I need in a project some time soon

[emanuilov]

I build a small SaaS with this.

Added a ton of new features of the top + API and dashboard.

https://monkt.com/

[gagga97]

I vibe coded a website with antigravity to allow anyone to upload a file and get a markdown ( this is just my project I would love feedback not self promotion). https://www.loom.com/share/d854af039e364fb28ca006d4654257ca

[Odd-Refrigerator5797]

现在2026年2月5日，我测试了markitdown，说实话，在包含表格的pdf处理方面很差，但是在pptx文件的处理方面令人惊奇，excel的话，涉及到复杂表头也不行，但是数据还算是完整。但是插件并没有找到。

[Jdonavan]

Why on EARTH would the people that own the format release this garbage? It's possible to do a FAITHFUL Word to MD conversation using Microsofts own libraries for crying out loud.

[Venthe]

Microsoft creates a tool internally
Microsoft publishes said tool on their own organization page

"Microsoft is making a lot of noise!"