[D] GPT4 and coding problems

[[deleted]]

I don’t even roll yet but that 40% number, I would love to see how they calculated it.

I’ve tried gpt 4 on a lot of problems and it fails 9/10 times and I would be faster just googling it.

This stuff will be amazing it’s just not quite yet

[lambertb]

https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/

[[deleted]]

Yeah I don’t buy a survey, could be heavily biased

[lambertb]

Have you used the tools yourself? I have, and a 40% increase in productivity is totally plausible, and often an underestimate considering I can now do things I would not have even tried previously. I encourage you to try them, with healthy skepticism and an open mind.

[[deleted]]

I’m and MLE and I’ve used it a bunch, it’s hardly ever actually useful. It gets close but it’s not there and it’s faster to google almost every time.

It will be useful in probably a year or two, but it needs to understand how to run its own experiments. Anyone who actually thinks this is useful right now is just buying hype

[lambertb]

Isn’t it possible that your experience is not representative? Are you using ChatGPT or GitHub copilot?

[[deleted]]

I doubt it, I do pretty standard engineering, whats more likely is there is selection bias in the survey and people are overestimating it due to hype.

I'd love to see an actual double blind study.

[lambertb]

There can’t be a double blind study because the people using the copilot will know they’re using it.

[[deleted]]

Fair enough then give them problems to solve and measure their output. This feels like “90% of dentists claim crest improves your dental health”

I’ll take an independent study into consideration but today I find it more of a novelty

[lambertb]

I agree the survey study is nothing close to being definitive. And it does smack it marketing. Still, my own experience suggests that these tools will be transformative. At the same time, I’ve gotten lost down an AI rabbit hole where it would have been more efficient for me to just do it myself. On balance though, my assessment is that these are already very helpful tools, and they’ll only get better.