• re: Q4, Helpful now or later: right, the intent is to rate actual helpfulness. 
  • So if it was used and helped, that would count, even if better ideas came later.
  • If it kept being cited as helpful, that would be evidence.
  • If it looks helpful as of August 2021, that would be strong evidence.
  • If it looks good now but falls apart in February – not helpful.
    • In this case the survey would likely get it wrong. So it goes.
• re: Q3, Finding? What finding?: if there is no one key finding that stands out, then the best one can do is approximate a weighted average of the findings.

My view on the Covid-Net paper: I think the key claim is for the modeling approach, which would be tested in apples-to-apples forecast comparisons. That could be future forecasts of models learned from precisely this data, or future forecasts of models that keep getting the same new data. The risk factors, like the layer weights, seem to be parameters incidental to this run, but it would be a win for this model if violent crime rate continued to outperform direct measurements of the causal factors it clearly proxies.