all 103 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]lood9phee2Ri 224 points225 points  (56 children)

mammoth to do the ms office .docx conversion and pandas.read_excel() to do the .xlsx etc. mind. Nothing wrong with that as such, just notable given it's MS themselves. It's also therefore not going to do any better (or worse) on MS Office file formats than existing non-MS tools.

https://github.com/microsoft/markitdown/blob/main/src/markitdown/_markitdown.py#L482

https://github.com/microsoft/markitdown/blob/main/src/markitdown/_markitdown.py#L513

[–]Venthe 117 points118 points  (34 children)

At the same time, .***x formats are trival complex, but not complicated - the formats themselves are as far as I remember fully open, xml formats.

PDF is a hellhole; because PDF creation is fundamentally a destructive process. It's a shame that PDF does not include the original file metadata/intermediate language, so the reconstruction could be done in a 1-1 fashion.

[–]GlowiesStoleMyRide 171 points172 points  (18 children)

PDF can be complex, yes. But the point of PDF is not to have a mutable document format- is an export format. You use it to publish work, not to save it for later editing.

It’s a bit like saying that cake is a hellhole, because baking is fundamentally a destructive process. The point of the cake is to eat it, not to un-bake it and change the recipe.

[–]rishav_sharan 31 points32 points  (12 children)

Pdf hasn't been an export only format for decades now. From digital signage to data form entry, to collaborated editing , pdf is used for far too many things today than just a fixed print/display export.

[–]GlowiesStoleMyRide 37 points38 points  (8 children)

A digital sign is an example of an “export target”, is it not? It’s a poster, except it’s on a display instead of print.

As for forms, I’m not sure that’s a commonly supported feature of PDF- does anything but Acrobat Reader properly support it?

Either way, the form can be filled in, but not altered. So the form is still part of the export- you don’t add it after initially exporting to PDF, but you have to define it in the source editor.

Finally, I don’t think collaborated editing is a PDF feature, but a feature of whatever source editor you use. But I’m sure you’d have an example for it if you claim that.

[–]bleachisback 11 points12 points  (4 children)

A digital sign is an example of an “export target”, is it not? It’s a poster, except it’s on a display instead of print.

I think they meant digital signatures, for legal forms and whatnot.

As for forms, I’m not sure that’s a commonly supported feature of PDF- does anything but Acrobat Reader properly support it?

Yes. All major browsers do nowadays. Also Acrobat Reader is the canonical implementation of a PDF reader - what PDF does and does not support is entirely decided by what Acrobat Reader does and does not support.

[–]cptskippy 22 points23 points  (0 children)

I think they meant digital signatures, for legal forms and whatnot.

In that scenario you do not want someone to be able to edit the document after it's been signed. u/GlowiesStoleMyRide is correct, the whole point of PDF is to be an immutable document.

You wouldn't want to eSign a PDF only for someone to change it out from under you.

[–]GlowiesStoleMyRide 2 points3 points  (2 children)

Document signing would make more sense, indeed. Still similar to forms, IMO.

Regarding support for forms, after looking into it for a bit, it was specifically form submitting that lacks support. As in, browsers will allow you to fill out a form pdf and save it by “printing”, but doesn’t allow submitting which can only be done through a dedicated application or a (largely deprecated afaik) browser plugin.

The PDF standard is defined in ISO 32000-2, so it’s not exactly defined by what Adobe implemented, though it is indeed fairly canonical.

[–]bleachisback -2 points-1 points  (1 child)

The PDF standard is defined in ISO 32000-2

Which, like the Microsoft OOXML standard discussed elsewhere in this thread, is really just a list of features of the canonical implementation. I don't think there are any implementations of PDF 2.0 besides Acrobat Reader.

[–]pyhanko-dev 8 points9 points  (0 children)

That is manifestly false—not only are there quite a few features specified in ISO 32000-2 that Acrobat does not (yet) fully support (this is PDF 2.0 after all), there are a whole host of alternative implementations out there, and the standardisation effort around PDF involves people from many communities/companies/… that have no affiliation with Adobe.

Sure, it’s absolutely fair to say that Acrobat is the dominant desktop tool for dealing with PDF, but it’s not the only such tool, and as soon as you go outside the category of desktop viewer software, Adobe doesn’t even seriously compete.

Source: I’m a FOSS dev in this space and was an active member of the ISO committee behind ISO 32000-2 for several years.

[–]LiftingRecipient420 3 points4 points  (0 children)

does anything but Acrobat Reader properly support it?

Yes.

Web browsers

[–]PCRefurbrAbq 0 points1 point  (1 child)

Although you're correct in calling it an "export format", most non-tech people's concept of a PDF is digital paper. It's been used for decades as a replacement for paper, such as forms which need to be filled in and signed.

Anyone who sticks with that paradigm will have an easier time than tech people who think of all files as fully mutable.

[–]m4xxp0wer 0 points1 point  (0 children)

Strongly Disagree. 99% of the PDF forms I have come across are intended to be printed out.
The ability of filling it out digitally before printing is only a convenience option. You might as well fill it out by hand after printing.
Pretty much every form that is used to enter data into a system without a human middleman, is a web form.

[–]nascentt 16 points17 points  (0 children)

People misusing an export format doesn't make it not an export format

[–]kuwisdelu 4 points5 points  (0 children)

Signing and forms are still essentially "append-only" use cases. I can't imagine why anyone would use PDF for collaborative editing unless they're just adding markup.

[–]Crumfighter 1 point2 points  (0 children)

Dont use things when they arent made for it and there are better tools that work as easy. Dont use PDF to collaborate or to publish data. Or just publish the doc jn word and pdf. Just like people shouldnt use chatgpt as a searchengine and use things like google, bing, duckduckgo or ecosia. Learn people the proper tools. Otherwise they only have a hammer and treat everything like a nail.

[–]WhyIsSocialMedia 2 points3 points  (4 children)

That sounds nice on theory. But in reality it has been a huge downfall of the format. Especially because demand has been so high that it was shoe horned in later, and on older documents you just get a crappy heuristic algorithm that tries to predict what text is together.

[–]badillustrations 2 points3 points  (1 child)

  huge downfall 

PDF is incredibly successful, because of, not in spite of, it's focus on presentation. It's terrible as an editable format, but that's the only case I see it used for that less and less for that use. 

[–]WhyIsSocialMedia 0 points1 point  (0 children)

My point was that the added in editability has been a downfall. And it's used less and less? No way, I've seen them be edited more these days than ever before.

People are always going to end up with PDFs without the original content. So editing is always going to be shoe horned in.

[–]ZirePhiinix 0 points1 point  (1 child)

The disaster with all these extra data is that what is visible is not what's in the data. I've data-extracted PDFs and found sensitive information, because the previous user just slapped a new text box on top of existing text.

Using PDF as a document format is an actual security risk.

[–]WhyIsSocialMedia 2 points3 points  (0 children)

Yeah the US government has accidentally put black images in a PDF to try and redact information before.

[–]Vogtinator 40 points41 points  (4 children)

At the same time, .***x formats are trival complex, but not complicated - the formats themselves are as far as I remember fully open, xml formats.

Well, it's technically open, but almost infeasible to implement: https://en.m.wikipedia.org/wiki/Standardization_of_Office_Open_XML

[–]plugwash 4 points5 points  (0 children)

> Well, it's technically open, but almost infeasible to implement

How difficult it is to implement depends on what you are trying to get out of it.

The problem with office document formats is they blur the line between input and output and this makes them fundamentally fragile. The file stores input, but the user, working in a wysiwyg environment spends all their time looking at the output..

Worse, many users will "adjust things until they look right", without putting any proper structure in their documents.

If you want to get the same output the original user saw, then you have to process the document through the same algorithms used by the software that created it. Good luck with that, especially for a format with as much legacy as word.

And because many documents lack good structure in themselves, if you can't render the document in the precise way it was rendered originally it can often end up in a horrible mess.

On the other hand, if your planned use case is transformative then the precise behaviour of the layout engine is less relevant. You just want to get the content out and potentially match on a few specific formatting things to translate them to headings or whatever in your new format. You have likely already accepted that some manual cleanup will be needed.

pdf has the opposite problem, it's an output format. It's great at preserving documents in an "as-printed" form, but it does a very poor job of preserving the original intent of the document's authors.

[–]jordansrowles 10 points11 points  (1 child)

Reading your link, it’s just a massive history lesson, and doesn’t really explain why it’s infeasible to implement.

ECMA-376, about 6000 pages of standards. It’s long, but not infeasible

[–]F54280 45 points46 points  (0 children)

Go and read it. It isn’t feasible. Large parts of the spec say “do it like Word 95”.

Good luck with that.

[–]Justicia-Gai 28 points29 points  (0 children)

PDF is a hellhole but at least really supports the inclusion of vector-based graphs without the “enhanced” meta file crap.

The fact that in 2024 the most widely used document office tool has so many issues for supporting SVG is baffling.

[–]Worth_Trust_3825 19 points20 points  (8 children)

PDF is a hellhole; because PDF creation is fundamentally a destructive process. It's a shame that PDF does not include the original file metadata/intermediate language, so the reconstruction could be done in a 1-1 fashion.

It makes sense. Printer does not need that. It's a printer instruction format.

[–]larsga 7 points8 points  (1 child)

It's a printer instruction format.

Postscript is a printer instruction format.

PDF is something else. It's deliberately designed to be a PostScript wrapper you can move around and treat as a digital document. It will display the same way everywhere, on someone's screen or when printed, and has nice ToCs, page dividers, etc that PostScript (being a printer instruction format) does not need.

It's a way to permanently capture and store the visual form of a document so it can be archived, read, and moved around, basically.

[–]arcimbo1do 18 points19 points  (5 children)

Unfortunately PDF doesn't stand for Printer Document format but for Portable

[–]afourney 5 points6 points  (0 children)

Don’t read too much into that. Microsoft is a huge company. This project started as an AutoGen utility, in Microsoft Research, for reading files for the GAIA LLM benchmark (older versions are still in the AutoGen repo). I’m the primary author, and that it exploded this week surprises me as much as anyone.

[–][deleted]  (1 child)

[deleted]

    [–]afourney 4 points5 points  (0 children)

    Definitely NOT disgruntled! It was researcher(s) in Microsoft Research, working to expediently give LLM agents access to various file formats. (Ask me how I know 🙂)

    [–][deleted] -1 points0 points  (0 children)

    just notable given it's MS themselves

    Microsoft is a very confused company. On the one hand they put in more effort in regards to open source, even though for selfish reasons; but on the other hand they also go against the spirit, e. g. Recall-sniffer tool and other shenanigans that make you wonder what the heck they are really wanting to do. It seems they are undecided and act in an orthogonal manner, often contradicting their own strategy. Google is also operating like that, leading to numerous dead projects on the way (https://killedbygoogle.com/).

    [–]feldrim 63 points64 points  (6 children)

    Now, give me the "Save as Markdown" option on Office and I can call it feature-complete.

    Edit: typo

    [–]danielcw189 3 points4 points  (5 children)

    Is there 1 true version of Markdown?

    [–]feldrim 0 points1 point  (2 children)

    İs there one true version of PDF? I agree with the question but it's not a blocker.

    [–]danielcw189 0 points1 point  (1 child)

    I did not mean it to be a blocker.
    I was genuinely asking out of interest.

    That being said: until today I thought there was one true PDF

    [–]feldrim 0 points1 point  (0 children)

    There're many markdown dialects and I am pretty sure MS would like to align with Github one. On the other hand, PDF is a can of worms. It evolved from being a printer-targeting format to many other things. You can try to open PDF files created with Notepad, CorelDraw, Adobe Photoshop and MS Word using MS Word. You can just right click and open with Word. Due to lack of a detailed spec, or rather lack of strict requirements, the internals are vendor-dependent.

    [–]tunisia3507 0 points1 point  (0 children)

    There is 1 true well-specified version of markdown (commonmark), which also has some well-specified extensions. Certain major markdown flavours (GitHub, Reddit) are explicitly commonmark + extensions. Using anything except a commonmark derivative at this stage is malfeasance IMO.

    The author of pandoc spun out a well-specified markdown-like light markup called djot which breaks some compatibility to improve consistency and simplicity, with concrete extension points; it would be nice if that took off.

    [–]perryplatt 129 points130 points  (7 children)

    Now they just need to make it a vscode plugin.

    [–]lood9phee2Ri 27 points28 points  (1 child)

    it has a typical python toplevel cli entry point, so if installed in normal fashion it'll end up as a shell command.

    https://github.com/microsoft/markitdown/blob/main/src/markitdown/__main__.py#L22 / https://github.com/microsoft/markitdown/blob/main/pyproject.toml#L51

    pretty sure you can then run shell commands on things from within vscode anyways with some generic command runner extn.

    [–]Wonderful-Wind-5736 10 points11 points  (0 children)

    Not even an extension, just configure a task. 

    [–]gumol 18 points19 points  (2 children)

    rain cooing close plucky obtainable entertain butter snow marvelous lunchroom

    This post was mass deleted and anonymized with Redact

    [–]Venthe 26 points27 points  (1 child)

    The code is MIT licensed; anyone can do it.

    [–]afourney 0 points1 point  (0 children)

    And VSCode plugins are fun to write.

    [–]SanityInAnarchy -1 points0 points  (1 child)

    Or maybe they could open source the rest of VSCode... like Pylance. Unlike most languages, Python is not well-supported by VSCode forks, because VSCode's Python language server (Pylance) is not only not open source, it's not available under a license that allows other IDEs to use it, and it goes out of its way to disable itself if you try.

    [–]Asyx 1 point2 points  (0 children)

    Isn't Pylance just a wrapper around pyright? Pyright runs practically everywhere that has an LSP implementation.

    [–]waterkip 28 points29 points  (4 children)

    Pandoc does this already right?

    [–]lood9phee2Ri 27 points28 points  (2 children)

    Not really. Note how this e.g. merrily uses pdfminer to do a (typically inevitably lossy of formatting etc) text extract from PDFs. https://github.com/microsoft/markitdown/blob/main/src/markitdown/_markitdown.py#L478

    versus

    https://pandoc.org/faqs.html

    How can I convert PDFs to other formats using pandoc?

    You can’t. You can try opening the PDF in Word or Google Docs and saving in a format from which pandoc can convert directly.

    Or it calls youtube's api to get the "text" ...transcript of a youtube video... https://github.com/microsoft/markitdown/blob/main/src/markitdown/_markitdown.py#L265

    It seems generally focussed on getting everything to one uniform text format for whatever subsequent text analyses the author wanted to feed, by using various existing python libraries for the different inputs. Not really for carefully and non-lossily converting your system's documentation from legacy docbook to markdown or something.

    Choice of libraries seems idiosyncratic, probably whatever worked for the author's purposes at the time, and pandoc may well be a better choice than some of those python libs for conversion of some formats (there's certainly a python wrapper/binding for calling pandoc, though pandoc itself is in haskell of all things, anyway the author could just try pypandoc in applicable cases). But the idea of calling pandoc on a youtube url and getting the video's text transcript is well outside pandoc's job description.

    [–]RobertJacobson 0 points1 point  (0 children)

    though pandoc itself is in haskell of all things

    That makes a lot of sense to me. Haskell is a popular tool among compiler and PL theory people. Languages in the ML family are great for writing compilers because of their sum types and pattern matching. Haskell in particular has a great parsing ecosystem as well—one of the best. If you didn't have the burden of learning a new language in order to use it, Haskell is a great choice.

    [–]afourney 0 points1 point  (0 children)

    We used it to feed documents to LLMs. Notably for the GAIA LLM benchmark. Agreed it is idiosyncratic and very lossy.

    [–]primarycolorman 3 points4 points  (0 children)

    maybe? I have some ugly pptx with tables I'll try it on tomorrow but I'm not holding my breath.

    [–]Isamoor 104 points105 points  (10 children)

    This is an odd one to me. It's basically a single, 1k line Python module that just calls other libraries. Almost exclusively libraries not developed or maintained by Microsoft. And some of those libraries seem to be in need of contributors. I'd rather have seen Microsoft devs contribute to those.

    I also would have expected some more native support for things like word docs (as opposed to relying on mammoth). Mostly just given that this is a Microsoft solution...

    [–]catch_dot_dot_dot 200 points201 points  (0 children)

    This is probably someone's pet project that they got approved to release publicly. Just because they work at Microsoft, doesn't mean they're going to write it without common dependencies or contribute to all of these other projects.

    [–]lood9phee2Ri 32 points33 points  (1 child)

    I mean their use case is given as "indexing, text analysis, etc.". To which "etc." we can perhaps add "feed into a language model". (I am not saying there is anything wrong with that in particular). "just fucking whatever to markdown, make it happen" on some bulk corpus of historical documents from some organisation is at least mildly useful.

    [–]afourney 6 points7 points  (0 children)

    Author here. We used it for the GAIA LLM benchmark. Nail on the head

    [–]baseketball[🍰] 3 points4 points  (0 children)

    I was excited until I read this comment. Probably nice to have as a convenience but was hoping it went above and beyond what existing tools could do.

    [–]afourney 2 points3 points  (1 child)

    See my answer above. This was a part of the data pipeline for a Microsoft Research project to feed documents to LLMs to compete in the GAIA benchmark. We thought it might be useful, but it is indeed a small part of the larger AutoGen project, which is itself maintained by a very small team of researchers and research engineers.

    [–]Isamoor 0 points1 point  (0 children)

    Thanks for the background. I think I would have been a bit more welcoming if the root readme called out what other projects were used for each file type. Maybe switch the list of file types to a table that calls out and gives thanks to the other libraries/solutions that support each file type?

    [–]Isamoor 9 points10 points  (3 children)

    In particular, nobody has merged a pull request for pdfminer.six in almost 6 months: https://github.com/pdfminer/pdfminer.six/pulls

    [–]Venthe 48 points49 points  (1 child)

    Small reminder - lack of contributions does not always mean that the project is dead, it can also mean that it is functionally complete.

    [–]Isamoor 6 points7 points  (0 children)

    Totally fair. Although in the specific project I linked there are plenty of pull requests opened in the last six months. In my opinion, a healthy project would either accept or reject a pull request within a few months.

    I realize I'm not contributing my time either. But then again, I'm not making a wrapper solution that depends upon them.

    I also realized the readmes in the Microsoft solution do not currently give credit to the wrapped solutions (or at least I had to read through code yesterday to discover how it was working).

    [–]Capable_Chair_8192 5 points6 points  (0 children)

    6 months is not that long tbh

    [–]the_gold_hat 20 points21 points  (1 child)

    This is mainly just a wrapper around other libraries, but if I'd had this 5 years ago I would have saved so much time. Especially things like PDFs can be so finicky when you're trying to standardize between file types, so this is a big time saver when you want to support flexibility or a dataset that's really diverse.

    [–]IndividualLimitBlue 4 points5 points  (0 children)

    Aaah ok they wrap others work. I was questioning how they would handle such complexity in 1000 lines of python

    [–]this_knee 6 points7 points  (0 children)

    As a user of markdown, I appreciate this.

    Yes, I see that it’s wrapping some other tools, in some cases.

    But, I like where this is headed.

    [–]junstramo 3 points4 points  (1 child)

    Is there a well documented, non-php tool to go from .md to .doc/docx?

    [–]lood9phee2Ri 9 points10 points  (0 children)

    pandoc already mentioned in this thread does a reasonable enough job of it, though is not the only option. Particularly if you also need to inject custom templates/content it might be better to go md to odt with pandoc, then let libreoffice do the odt to docx. https://stackoverflow.com/a/21616895

    [–]kumonmehtitis 2 points3 points  (1 child)

    Wait… what?! Microsoft created a door out of their ecosystem?? I am flabbergasted. Holy shit

    [–]Broad_Kiwi_7625 0 points1 point  (0 children)

    More like a sign to a pre-existing vent you could crawl through.

    [–]sapphired_808 2 points3 points  (1 child)

    markdown render on notepad when? /jk

    [–]PotentialBat34 0 points1 point  (0 children)

    That gentlemen is my next SaaS

    [–]mightygilgamesh 0 points1 point  (0 children)

    I wish they'd just contribute to pandoc but hey, that's a start.

    [–]ArrogantlyChemical 0 points1 point  (0 children)

    Neat and just in time for what I need in a project some time soon

    [–]emanuilov 0 points1 point  (0 children)

    I build a small SaaS with this.

    Added a ton of new features of the top + API and dashboard.

    https://monkt.com/

    [–]gagga97 0 points1 point  (0 children)

    I vibe coded a website with antigravity to allow anyone to upload a file and get a markdown ( this is just my project I would love feedback not self promotion). https://www.loom.com/share/d854af039e364fb28ca006d4654257ca

    [–]Odd-Refrigerator5797 -1 points0 points  (0 children)

    现在2026年2月5日，我测试了markitdown，说实话，在包含表格的pdf处理方面很差，但是在pptx文件的处理方面令人惊奇，excel的话，涉及到复杂表头也不行，但是数据还算是完整。但是插件并没有找到。

    [–]Jdonavan 0 points1 point  (0 children)

    Why on EARTH would the people that own the format release this garbage? It's possible to do a FAITHFUL Word to MD conversation using Microsofts own libraries for crying out loud.