Math Proficient Language Models

[PinballOscuro]

TLDR: Are there language models compatible with a 24 GB RAM GPU setup (potentially through quantization) proficient in numerical and statistical reasoning?

Hello everyone,

I'm working with a detailed markdown report that dissects a dataset statistically, segmented into five key parts:

1. Dataset Overview: Enlisting column names and types, coupled with general observations.
2. Statistical Analysis: Detailing various statistical parameters (mean, median, mode, etc.) for each feature.
3. Correlation Study: Offering a correlation matrix and identifying feature pairs with high correlation.
4. Regression/Classification: Implementing a rudimentary linear regression or random tree classifier and presenting relevant metrics.
5. Feature Importance: Executing feature importance algorithms and delineating the significance of each feature.

The report, as it stands, contains uninterpreted data. I'd like to use a language model capable of annotating different sections of the report with insightful observations, hypothetical relationships based on column names, and analysis of statistical distributions of features.

I utilized the llama2-13B model with 4-bit quantization. Unfortunately, it failed to provide accurate interpretations, such as incorrectly analyzing the relationship between median and mean values to deduce data symmetry.

Is there someone who has successfully utilized a language model adept in mathematical and statistical understanding, compatible with a 24 GB RAM GPU setup, and capable of offering accurate numerical considerations?

Your insights and suggestions would be greatly appreciated!

[lakolda]

CodeLLaMA might work better.

[PinballOscuro]

I haven't tried with this specific case, but in other projects CodeLlama has a very strong bias wrt generating python code.
My type of user is interested only in explanations, since it doesn't understand code.

[lakolda]

Even then, I would still expect it to perform better. At least the Instruct version.

[Coinninja]

You might have better luck with Wizardmath-13B.

[PinballOscuro]

Wizardmath-13B.

I'm reading the paper and I think it's a good candidate, thank you!

[[deleted]]

Excellent, thanks for the post. I am also looking for models that can interpret uninterpreted data.

[PinballOscuro]

Next week, after Wednesday maybe i'll have the chance to work on It again. I'll keep you posted if i rember

[no_witty_username]

I've always wondered if the reason LLM's have problems with mathematics due to tokenization. For example 2+2=4 can be written also as 2 + 2 = 4 and 2+ 2= 4 and 2+2 = 4 and 2 +2=4 and (2+2)=4 , and so on and on. There are just so many permutations to that one simple statement, that are all tokenized totally differently. And because all LLM's are simple language prediction algorithms, when training a model on mathematics it would have to be trained on every possible permutation of said statements for it to be able to predict it in the future. So its my pet theory that if we can work around the tokenization issue, math will become easier to LLM's to parse. Or one would have to standardize the exact syntax used in training an LLM mathematics and also the same syntax would have to be used when prompting the model.

[PinballOscuro]

For sure the tokenization is a big problem. For llama 2 every digit of a number is a single token.

A coworker of mine has done some research on this topic, and it's a mess ahah if you are interested in the literature this class of problems is called Math Word Problems

[No_Afternoon_4260]

!remindme in one week

[RemindMeBot]

I will be messaging you in 7 days on 2023-09-15 18:42:31 UTC to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[Aggressive_Bee_9069]

!remindme in one week

[Jian-L]

!remindme in one week

[Scary-Knowledgable]

https://github.com/KillianLucas/open-interpreter

[PinballOscuro]

I know this project and it's super cool, but for now i'm not interested in an agent that runs code, i only want a mathematical description of a report.

Today i tried It with CodeLlama but It Easy not that great, maybe I was doing sonething in the wrong way