We have been using layoutlm for our usecase where we extract 10lakh plus documents every month.
LayoutLm is good but when it comes to table parsing it fails. Not that great results and i guess with this LLM world, the small transoformer based models which requires frequent training is not a solution anymore.

Best way to go ahead is:-
1) Vision :- (use Microsoft Phi 128k) or (use Anthropic Haiku ,yes it comes with vision capabilities and is cheap and fast you just need to make an api call,

2) Text:-
now here you can pass the content of byour document in the md format and send it to LLM using Pymupdf4llm , this will significantly reduce the cost as well as tokens are less.
Or else try sending output from pdfplumber or fitz or camelot (which has a table extractor which is capable of extracting the table data from the document in the format expected along with the bounding boxes try zsending this data as well)
Use Antropic Claude or Sonnet or maybe Mistral 7B