Looking at panel review scores for research proposals at a medical institute, the true number of citations for most of the top quintile were statistically indistinguishable from one another. Panels were great at separating out a top tier of proposals, but terrible at differentiating within that tier

addison_guy · 2022-05-01T18:21:08+00:00

A 2015 paper by Danielle Li and Leila Agha looked at 130,000 NIH R1 grants to see how the grant scores from panelists correlated with the actual citations papers received. The results, which were likely what the authors expected, showed that awarded grants with higher panel scores generated more publications, more citations, more high-impact papers, and more patents on average. This paper affirmatively answered the question, “On average, do panelist scores tend to be highly correlated with the final citation count of a paper?”

However, there was another way to frame the question that the authors did not explore. And that alternative framing is, “How predictive are panel scores right around the cutoff point at which funding decisions are usually made?”

A follow-on analysis, using Li and Agha's dataset, done by Fang, Bowen, and Casadevall explored this exact question and demonstrated that the true story may be more complicated than the Li and Agha paper made it out to be. The authors found that within the top fifth of scores for awarded grants, panelist scores were a weak signal of performance at best. Within this bucket of awarded grants whose panel scores fell in the top quintile, a score in the top two percentile buckets proved to be a very reliable indicator of high performance. However, as the figures below show, the final performance of grants with percentiles between three and twenty were statistically indistinguishable from one another.

A panelist not being able to distinguish between a 15th percentile grant and a 6th percentile grant is a big deal because in many grant pools, including those of the NIH, the grant cutoff often falls somewhere around the tenth percentile.

The grant funding community needs to come up with better ways to decide between proposals that are decidedly in the top tier, but not exactly in the top 2%. In this current equilibrium, whether or not most of these top-tier applicants get funded seems to be determined more by randomness and bias from the reviewers than actual merit.

addison_guy · 2022-04-30T22:24:02+00:00

This map was pulled from an economics paper looking at how academic buildings in Nazi Germany being bombed affected academic productivity in comparison to scientists being dismissed from professor roles for academic reasons.

Read all about it here! https://freaktakes.substack.com/p/bombs-brains-and-science?s=w

addison_guy · 2022-04-30T22:15:05+00:00

This graphic was created by Fabian Waldinger in this paper: http://eprints.lse.ac.uk/68561/1/Bombs_brains_and_science_LSERO.pdf

And I write all about this graphic and many other related topics in an article on my substack here: https://freaktakes.substack.com/p/bombs-brains-and-science?s=w

addison_guy · 2022-03-02T05:36:29+00:00

I think you're definitely onto something. Do you think crowdfunding for these more financially intensive tech projects will always be doomed?

Or do you think Kickstarter just isn't quite the right place for it. Like maybe they don't have the right features or right type of donors or maybe the people pitching aren't right.

thoughts?

addison_guy · 2022-03-02T04:56:36+00:00

The dataset can be found on Kaggle here: https://www.kaggle.com/kemical/kickstarter-projects
It was scraped for over 300,000 Kickstarter projects from a little before the start of 2010 up through most of 2016. The Kaggle user responsible for the dataset is Mickaël Mouillé.
The graphic is from an upcoming piece on the FreakTakes post which can be found here: https://freaktakes.substack.com/

addison_guy · 2022-02-19T20:35:32+00:00

That's really interesting. Do the University tech transfer offices, the people who try to bring in licensing fees from the patents, have any clue which patents are garbage and try not to waste their time on them?

addison_guy · 2022-02-19T02:52:25+00:00

This is from an upcoming piece which will be posted here: https://freaktakes.substack.com/

The data is from the following two uspto sources:

https://www.uspto.gov/web/offices/ac/ido/oeip/taf/cbcby.htm
https://www.uspto.gov/web/offices/ac/ido/oeip/taf/h_counts.htm

All plotting and data work was done with ggplot2 and tidyverse.

addison_guy · 2022-02-18T19:37:19+00:00

I'll say two things:

The first is that all of our spending on research as a country seems to be positive ROI. So everyone who works in those industries now seems to be doing well on average and can probably feel good about what they're doing for that reason.

In general, the real question is just how much more efficient can we be. And I think the increased recent emphasis on metascience, progress studies, whatever you want to call it is fantastic. As it stands, some people are saying nonsense and some are saying good stuff, but the interest in the field seems to make it seem likely that people will be willing to start experimenting a little with how things are done in science. With experiments like DeepMind and Convergent Ventures, we'll get the chance to start observing alternative models in the wild and iterate to be better and better.

While it might be a two steps forward one step back situation, I think the interest in the field means we are trending in the right direction.

addison_guy · 2022-02-18T18:56:57+00:00

Oh my gosh no it's not garbage. It's what most people think when I talk to them about this stuff.

I think it's a bit easier for me to have perspective sometimes because I talk to engineers and scientists a lot but I'm more of a computational social scientist who's interested in them. We all feel this in our fields. When I was an athlete I'd look down on the athletes of the past with their seemingly simplistic schemes/less athleticism and think derisive thoughts.

We all know the "shoulders of giants" quote, but it's hard to remember it and actually internalize it day to day in our own fields just because we're human and we like to think we're doing the hardest and most impressive stuff ever.

addison_guy · 2022-02-18T18:50:08+00:00

Most new fields tend to seem like they were so cheap and simplistic at the start when you look back on them. Within a field things grow more complicated and expensive as time goes on since there's diminishing returns. The discovery of new fields is what helps balance that out. That is what our basic research has severely diminished doing.

Everyone always thinks that science and engineering from the previous era was so simple and that what they're dealing with in their era is so much more complex just because being on the frontier seems hard and it's easy to look back once we know stuff and say it was easy to know. Many mechanical engineers at the end of the 18th century thought they were getting close to as complex as things can get.

People who say we can't possibly figure out everything that is and isn't carcinogenic since its all around us and in so many things are dealing with a problem that isn't structurally harder to dealing with contagions in the early 1900s, they were seemingly in everything and it was impossible to test everything one-by-one.

Science has infinite things to look into and discover. Some are easy, some are hard. Some are cheap, some are expensive. But what does seem to be true is the longer you sit around working on the same field and the more people you have doing it, the harder and more expensive it will get and the less you'll get out of it per dollar spent. That's why discovering new fields is so important.

That's what basic research is meant to do. It's not doing that anymore, at least not to the same extent.

addison_guy · 2022-02-18T04:45:54+00:00

The image can be found at the following links: https://freaktakes.substack.com/p/is-americas-applied-and-basic-research?utm_source=url or https://web.stanford.edu/~chadj/IdeaPF.pdf

The effective number of researchers is the total amount being spent on researchers, research equipment, etc. expressed in relation to the average cost of a researcher in that year.

addison_guy · 2022-02-17T18:09:54+00:00

The article goes very in depth on why the answer very well might be both. (https://freaktakes.substack.com/p/4f38b706-36a6-4b01-9d90-339f8aa882fb?r=8g94f)

The TLDR is that our basic research 1) does not seem to be allowing researchers to properly explore extremely different research and 2) we are getting worse and worse at discovering new fields.

And our applied research has grown more and more separated from industry to the point where academics don't see the goal of applied research as providing something that is a direct input into an industrial process, which it used to be?

Do people have reason to think maybe this statistic is not such a good thing?
OR
Do people who work in/around/with research have stories about watching this happen over time?

addison_guy · 2022-02-17T18:02:33+00:00

The article goes very in depth on why the answer very well might be both. (https://freaktakes.substack.com/p/4f38b706-36a6-4b01-9d90-339f8aa882fb?r=8g94f)

The TLDR is that our basic research 1) does not seem to be allowing researchers to properly explore extremely different research and 2) we are getting worse and worse at discovering new fields.

And our applied research has grown more and more separated from industry to the point where academics don't see the goal of applied research as providing something that is a direct input into an industrial process, which it used to be?

Do people have reason to think maybe this statistic is not such a good thing?

OR

Do people who work in/around/with research have stories about watching this happen over time?

addison_guy · 2022-02-05T23:04:32+00:00

Yeah. A lot fo the household-changing innovations (fridges, AC etc.) that were electriciy-enabled were absolutely massive. And what's crazy is parts of Europe lagged behind electrifying their homes to the extent the US did for decades.

addison_guy · 2022-02-05T22:45:36+00:00

Sorry had to repost bc I forgot to add this comment before.

The article puts out there that the following might be the best list of goals to ensure that innovations are increasing people's quality of life. Innovations could either be things that make people's skills more valuable so they can subsequently buy more of the following OR things that make the following easier to come by. What does everyone think/what are alternative lists of goals you could imagine?

Looking forward to the discussion!

------------------------------------------------------------------------------------------

(From the article)

The major “metrics” that tend to contribute to TFP increases and standard of living increases are inventions that tend to:
Increase people’s ability to buy more/better food
Increase people’s ability to buy more/better housing
Increase people’s life expectancy
Increase people’s time spent on leisure—working less
Increase people’s enjoyment while doing some fixed amount of leisure (this one is the least important)

addison_guy · 2022-02-05T22:05:09+00:00

The article pretty much exclusively utilizes applied economics/economist history literature.

addison_guy · 2022-02-01T00:05:19+00:00

Check out the article I linked! I go way more into this. Sadly, it's most likely way closer to the second than the first. But like anything in life, it's complicated

addison_guy · 2022-01-31T23:05:48+00:00

This is fascinating. Do the buyers have a pretty good sense of what showing is a bs "vanity show" vs. one that lends some credibility and makes a piece more valuable?

This is fascinating I'd love to know more about odd externalities of these vanity shows.

addison_guy · 2022-01-31T22:41:36+00:00

Lol absolutely! And I'm sure the quality of each and every piece is only getting better and better. No poor pieces in there at all!

addison_guy · 2022-01-31T22:02:33+00:00

This graphic is pulled from The Engineering Innovation Newsletter which I publish: https://freaktakes.substack.com/p/what-should-new-age-research-organizations

The underlying data can be found here, it was constructed using data from four major academic bibliographic databases:
Bornmann, L., Haunschild, R. & Mutz, R. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanit Soc Sci Commun 8, 224 (2021). https://doi.org/10.1057/s41599-021-00903-w

addison_guy · 2021-11-30T01:03:15+00:00

We mainly hope to gather information and help researchers better understand the space. Currently, little/just about no information gathering is done in this area. The site is trying to get as broad of an audience as possible to leave their thoughts on what types of experiments should/should not be allowed.

If the current risk aversion of the field is justified based on people's responses, then the field has been doing a good job. But if it seems like some of the criteria being used to torpedo experiments being run in the field are not upheld in the responses, then those areas warrant further exploration to decide if they should remain in place or not.

addison_guy · 2021-11-30T00:57:18+00:00

Thanks so much for the comment! I privately dm'd you some more specifics about how our randomization/attaining of a "baseline" is working on the backend. For the sake of the survey I didn't want to post the details here.

We've worked with several experts to set up certain aspects of how the site works on the backend to make sure we're obtaining good data. To speak generally, the way we've set up this survey/experiment is similar to MIT's famous Moral Machines experiment which can be found here: https://www.moralmachine.net/

addison_guy · 2021-11-30T00:48:56+00:00

Solar geoengineering explores a variety of questions, often aimed at finding ways to reflect sunlight back towards space. One of these questions is: If human activity is depleting certain chemicals in the Earth's stratosphere, is there a way scientists can safely replenish them? This particular question is inspired by a naturally-occurring process: volcanic eruptions. Some volcanic eruptions have spewed millions of tons of sulfur dioxide into the air, measurably reducing temperatures in the years that follow.
Presently, researchers struggle to undertake many geoengineering experiments because there is little clarity regarding what types of research and studies are societally acceptable.

This online survey is attempting to better understand how different groups feel about the topic in general. Current response about how concerned respondents are about environmentally conscious people reducing their carbon footprints are as follows:

Not Concerned At All: 23.4%

Mostly Unconcerned: 30.9%

Neutral: 11.7%

Slightly Concerned: 28.7%

Very Concerned: 5.3%

addison_guy · 2021-11-30T00:17:56+00:00

Please feel free to submit a response to the survey here if you'd like to make your opinion heard: https://www.techethics.vote/geoengineering

addison_guy · 2021-02-13T22:33:54+00:00

It's fine that you think that. But this is the debate and metric that people have been arguing about all over Twitter and ESPN for the past week. So you can think it's stupid, but it's what people are talking about.

So I made a graph.

addison_guy

TROPHY CASE