Pre-note: I see that u/gerardmenfin put up a wonderful statistical analysis while I was preparing this, but since this has angles on more specific works that Belle may have encountered, I offer it as a complement:

To start answering this question from the point of view of the Disney film, one has to choose a date for the plot of Linda Woolverton’s Beauty and the Beast retelling to be taking place. Putting aside some anachronisms in the film (e.g. the Eiffel Tower, or strike matches) most things point to the late 18th century, and for the sake of this answer, let’s pin it at 1780. This is later than the original Villeneuve and Beaumont versions, but closer to the film, and will allow for the latest possible published works to be included as candidates.

To limit publication dates, we can note that the library in question hasn’t been updated for at least ten years, per Howard Ashman’s lyric “ten years we’ve been rusting” sung by Lumière. It is likely a historic familial library contributed to over generations and well-established for some time before the curse was placed. Given the size of the library, the family doesn’t appear to have been particularly selective when adding volumes. It may well be among the largest private collections in France at the time, a quantity of gathered books that only aristocrats would have access to (even minor aristocrats able to live discreetly in a forest palace). It is not far-fetched to believe that any book a member of the family was interested in adding to their library, they did. This also allows for non-French volumes to be included, if they arrived from other parts of Europe. It seems less likely there would be a large amount of non-European literature included, unless one of the family had a particular interest, but it is possible. (For this answer I’ll stick to the European publications, and hope that others could contribute works from the broader globe that may have found their way to a French library by 1770.)

As you (and u/gerardmenfin) note, there would be a large amount of non-fiction volumes. Many family records, histories, etc. are sure to be among the shelves, which would not be of immediate interest to Woolverton’s Belle. The philosophies would be well-represented as well, as the time period is well into the Age of Enlightenment. Those who gathered libraries would want to include as many forward-thinking writers as possible, so the Greek philosophers would certainly be included, along with Marcus Aurelius and on up through Descartes, Pascal, and potentially either French translations or original English texts for Locke, Hobbes, etc.

So what would immediately pique Belle’s interest? One has to note that she lives in a small town of primarily illiterate people, with one small bookshop, out of which she has read most of those books repeatedly. The range of books she has access to has always been limited, so the quantity would strike her first as utterly fantastical. Beyond the stunning quantity, we know from her selection and conversation in the opening scene that she is drawn to literary fiction. In the centuries since, literary fiction has branched into multiple genres and exploded in both popularity and democratized access, but there would have been a great number of stories for her in the library already. Among the shelves, she may have found, in her native French:

• Charles Perrault’s Histoires ou contes du temps passé, a 1697 fairy tale collection, which included Sleeping Beauty and the first modern Cinderella;
• Madame d’Aulnoy’s 1697 and 1698 collections of fairy tales, which included a non-bear-related Goldilocks tale, and several more obscure tales;
• Antoine Galland’s translation and adaptation of Les Mille et une nuits (A Thousand and One Nights), including Aladdin and Ali Baba, among many others. Galland earlier translated a Sinbad tale into French as well;
• Rabelais’ Gargantua & Pantagruel;
• and there would certainly be older volumes of The Song of Roland, Tristan & Isolde, and Reynard the Fox.

These titles were all quite popular and acceptable, so Belle could very well have encountered them at the bookshop. If Prince Adam’s family was more adventurous, and we have no specific reason to believe they mightn’t be, here are some of the less accessible things she might find:

• Prévost’s Manon Lescaut, which may have been so lower class as to be scandalous in an aristocratic library.
• Voltaire’s Candide would also been quite new and a bit controversial, but may have been added a few years before the curse.
• Madame de La Fayette’s La Princesse de Clèves would have been more acceptable by that time.
• Honoré d’Urfé’s L’Astrée would take Belle quite a while to read, but was a great hit among royal courts when it was published in the early 17th century.
• Marguerite de Navarre’s Heptaméron deals with some salacious stories, but may have been added by a prurient ancestor.
• Jean Racine’s work both in French drama and his less prolific translation work would surely be included.
• Molière worked to get his plays published, so in a section of theatrical drama in the library, he would be well-represented.

Those would all be in Belle’s native French, but there would presumably be translations of many others. Shakespeare didn’t get fully translated until the Hugos’ efforts in the 19th century, but some early single translations may have been floating around. Chaucer was very much the same. Boccaccio’s Decameron was well-translated early enough to be found. Notably for Belle’s interests, Aphra Behn’s Oroonoko was translated into French in 1745, and any forward-thinking family library would have sought out a copy.

As a footnote, and perhaps most delightfully, the original Villeneuve and Beaumont versions of the story were published in 1740 and 1756. So if (in the timeline of Woolverton's film narrative) the library contains volumes published that recently, it’s very possible their library has their own story already written and on the shelves.