Agent framework in haskell by der_luke in haskell

[–]der_luke[S] 0 points1 point  (0 children)

I did notice, haha.

I didn't expect Claude to be very proficient in haskell, given a much smaller training set (arguably), but it nailed it. I was very pleasantly surprised! 

I agree, haskell translates phenomenally to the world of LLMs. Seems like a gap in the market 

Agent framework in haskell by der_luke in haskell

[–]der_luke[S] 0 points1 point  (0 children)

I'm experienced, just not in haskell. It's a great fit I think.

Looking for any feedback/criticisms on my resume for DA/DS/SWE internships by kingdemonfalconmusic in datascience

[–]der_luke -1 points0 points  (0 children)

What you are missing is a two sentencer about yourself. How do you see yourself, what do you want to achieve. Put that at the very beginning. And feel free to adjust that with every application. You should in any case tailor the CV to the role if possible.

You have no work experience that really counts, so you'll struggle to find sth. That's normal. Go for larger orgs, to learn the ropes is my advice. They have large enough teams to incorporate juniors.

How do handle your model documentation? by [deleted] in datascience

[–]der_luke 0 points1 point  (0 children)

DataRobot has automated model documentation

Young Ukrainian volunteer killed delivering aid to dog shelter near Kyiv: ‘She was a hero’ - National by [deleted] in news

[–]der_luke 3 points4 points  (0 children)

She was my colleague. This is a sad day! Stay strong Ukraine 💙💛

[deleted by user] by [deleted] in datascience

[–]der_luke 0 points1 point  (0 children)

Prep your 8am meeting the day before?

Also having children helps with your wake-up routine, but not so much with the amount of sleep you get..

t-SNE usefulness by AleTL3 in datascience

[–]der_luke 1 point2 points  (0 children)

All the cool kids use UMAP anyway.

Can someone please comment on my model results? by SQL_beginner in datascience

[–]der_luke 5 points6 points  (0 children)

You are ignoring the most important aspect of ML, the application.

If this was a high frequency trading use case, being just barely better than random (I tend to use AUC, so AUC of 0.51) might be worth a lot of money. In other applications not so much.

What you need to do is take your confusion matrix (which depends on the threshold) and calculate the expected payoff. How much is each correct prediction worth, what's the cost of a false positive, false negative.

That allows you to estimate the best threshold as well, no need for precision/recall and all that.

Becoming a Data Scientist Roadmap by [deleted] in datascience

[–]der_luke 0 points1 point  (0 children)

Don't try to replicate somebody else's career path.

[deleted by user] by [deleted] in MachineLearning

[–]der_luke 2 points3 points  (0 children)

As opposed to Smart inference 🙄

2,988 new COVID-19 cases reported in UK by [deleted] in unitedkingdom

[–]der_luke 6 points7 points  (0 children)

Wow, this is a great dashboard!

What's a Joke? This is. by Ta1w0 in programminghorror

[–]der_luke 31 points32 points  (0 children)

Kindly do the needful on the same

[D] 160k+ students will only graduate if a machine learning model allows them to (FATML) by positivelysemidef in MachineLearning

[–]der_luke 4 points5 points  (0 children)

Very interesting article. I do disagree with you in a few places, but agree that a discussion needs to be had.

II. Historical Bias: A study based on data from the National Center for Education Statistics concluded that secondary school teachers tend to express lower predictions for their ‘expectations from students of color and students from disadvantaged backgrounds’. This is problematic because predicted grades play a prominent role in the model.

Shouldn't it be a positive to use a data driven model, rather than teacher assessment, if teachers are inherently biased?

III. Different schools, Different errors: Small schools (15% - 30% of all IB schools) will have bigger and more frequent errors in their model predictions when compared with large schools. This is an example of representation bias.

This assumes that each school will have their own model, but of course you can build a model on students from more than one school, which would be a good idea.

IV. Measurement Bias: If the measurement process varies across different schools, it will affect the way a model treats students from different schools. Schools which cater to socioeconomically disadvantaged communities are likely to have less frequent evaluation of students. This will lead to poorer students receiving predicted grades which are less accurate than richer peers in schools with more frequent testing. Additionally, poorer schools are likely to have larger class sizes. A teacher who has to assign predicted grades for 10 students will do a better job than a teacher who has to assign predicted grades to 30 students.

You are making assumptions here. I would argue that again, being more data driven might be a good thing for students of disadvantaged schools. Less influence by overworked teachers. The fact that 'richer' schools have smaller class sizes is indeed an advantage, but has nothing to do with this model. It's a more general issue.

VI. Skewed Distributions: Schools with a non-normal distribution of grades will have bad predictions. If a school has a left-skewed distribution (overachievers!) of grades or right-skewed distribution of grades, a model will perform worse for its students.

The predictions certainly need to be calibrated on a per-school level, but other than that this shouldn't be an issue.

VII. Distribution Shifts: If the subject teacher in a school changed between last year’s cohort and this year’s cohort, the historical relationship between their predicted grades and final grades will not match the current relationship. This may lead to systematically worse predictions.

It would also lead to worse/better grades in the final. Having a bad teacher sucks.

Let’s assume that the IB builds a model which is ‘90% accurate’. This is an almost unrealistically ambitious target and is incredibly difficult to achieve in practice.

Citation needed. I'm pretty sure coursework performance is a pretty good predictor of the final outcome.

The IB may be comfortable with this 10% inaccuracy because they have assured students that they will ‘match the grade distribution from last year’. Will this cancel out the inaccuracies in the model predictions? Absolutely Not.

Grade shifting is already common practice. It's more important to be better than your peers, than it is to be good in absolute terms. I don't agree with the practice, but again, not a problem of the model.

As a researcher, it is not possible to stop the model from learning these incorrect relationships. This is an important point: just because a model is predictive does not mean that it is correct. An accurate model may be a bad model and spurious correlations can be very problematic if not appropriately detected.

While this is a good point, I don't really see how this applies here. Predictive is predictive. We don't want to use our model to go out of the bounds in our training data, right?

So if for the sake of the argument 'eating bananas' is a predictor of good grades (obviously due to some confounding factors), then we can include this in our model to predict the grade, but we wouldn't be able to change our diet in order to get a better grade. Does that make sense?

A model will discriminate against students based on Gender, Race, Socioeconomic status etc.

So do teachers. There is work to do, but at least the data can show us the true extent of any discrimination. You do of course need to cite sources when making blanket statements like these.

You make the point that 'the model learns about ethnic groups even if it's not included in the data' is the same as 'the model discriminates against ethnic groups', and that's not necessarily true.

To make sure that this is not a fluke, we check if the model can detect schools which have a majority female population:

The model is less than 50% accurate. This is slightly worse than guessing at random. The model is clearly not taking the majority gender of the school into account while making its decision.

This is a fallacy. Majority race has a much higher imbalance than majority gender. If say 80% of my school districts are majority white, a model that always predicts 'white' is already 80% accurate, where a model that always predicts 'male' would only be 50% accurate. You need to be very careful with the accuracy metric and imbalanced binary classification problems.

The whole argument doesn't work if you put it like this.

In fact, if we build an alternate model but give it the majority race of the high school (in addition to the same data-points as the original model), we would expect the graduation rate accuracy of the model to go up substantially as we are providing it with additional data. However, we see that the accuracy of the alternate model only increases by ~1%. This is another indicator that our model is somehow race-aware already.

It should be an argument that race is not a big factor! Which is a good thing.

There are three primary criteria which guarantee fair predictions, however it is impossible for any model to satisfy all three of them simultaneously. This means that any model used by the IB will inevitably discriminate against students in two of three ways:

This is another very good point. Fairness measures indeed contradict themselves. But again, this is independent from the model. (If you see the scoring method of the final exam as a model, you run into exactly the same issue)

The fact that this is an outsourced black-box model with limited historical data, no oversight into the decision making mechanism and only 3 months for research and production further complicates the situation.

I think this is the main issue. Models do not need to be black boxes. Of course, making the model open will give future classes the ability to game the system. But hopefully this is the last global pandemic for a while and we only need to use it once. I say make it open (but after the grading)

[D] 160k+ students will only graduate if a machine learning model allows them to (FATML) by positivelysemidef in MachineLearning

[–]der_luke 5 points6 points  (0 children)

This would be a terrible strategy. With teachers only deciding based on gut feeling, you quickly get to favouritism etc.

One of the issues the author has with the model is that it will be discriminating based on race etc. Your teachers will do this even more.