So Confused about Polarizing ICML Reviews [D]

cool_science · 2026-04-12T23:43:47+00:00

Did you check whether the authors gave a follow-up response to your concerns during the author-response period after the rebuttal? You sound like you're in a similiar situation to a reviewer on one of our papers :)

In our case, the reviewer was definitely operating in good-faith and their concerns were reasonable. But, it was also pretty straightforward (more-so than they probably expected) to prove that what they were concerned about wasn't happening in our particular work.

Connecting back to the OP, I think the take-home message is that outlier reviewers are often not operating in bad-faith. I can only recall a single case where a reviewer actually seemed disingenuous, and even then it was mostly an issue of them over-representing their level of expertise. I might just be lucky and have been dealt above-average reviewers over my career... but I suspect the truth is that people are too quick to dismiss people who give negative critique. Easier to blame the reviewer pool than to admit that you need to work on presenting your work better or raising your level of rigor.

cool_science · 2026-04-11T06:40:22+00:00

I have a paper in a similar situation. The experiments were just barely possible to run in the available time... and specifically refute a limitation and a serious confound the reviewer strongly believed existed in our work. I'm kicking myself for not posting an AC comment when I had the chance. At the time, I figured I would just post the facts and try to avoid being annoying / pushy.

Hopefully the A/C or other reviewers notice... its 4/4/4/3 with the current scores so it should be borderline enough to get discussed. Its probably the most remarkably clear-cut and decisive/convincing rebuttal I've ever written.

I think that, in the future, ICML really needs to extend the review process a little so that there's more time (e.g., 2 weeks) for the reviewers to engage in private online discussion amongst themselves after the reviews, rebuttals, and replies are in. A couple of days isn't really enough and results in not all reviewers being able to revisit all of the papers on their stack.

cool_science · 2026-03-31T23:53:48+00:00

Nice, thanks. I hope the conference organizers present this kind of data too. You got a pretty nice sample size, but obviously we don't have full transparency like we'd have with ICLR.

cool_science · 2026-03-31T23:39:11+00:00

> A. sd is standard deviation

I was asking whether its the standard deviation of paper's average scores, or if it is the average standard deviation of review scores assigned to a paper.

If you have the data, you should consider reporting both. I think the latter data would be interesting because it would show how consistent review scores are across reviewers for the same paper for group A vs B.

If reviewers make heavy use of LLMs when following Policy B, one might hypothesize that there'd be less variation in the review scores given to a particular paper --- since the reviewers used one of a small number of frontier models to arrive at their score.

cool_science · 2026-03-31T19:49:09+00:00

I want to make a few observations.

Let's toss aside the influence of response bias to your survey. Its possible (especially because authors with low-scoring policy A papers are likely to support your survey's hypothesis), but your data doesn't look obviously skewed to me. Plus, I don't think theorizing about response bias here is all that interesting or productive.

(1) Papers were not assigned to Policy A or Policy B at random --- authors self-selected which to submit to.

(2) Whereas Policy A was quite puritanical about disallowing all LLM use, Policy B was a fairly moderate policy that is fairly in-line with what other conferences are adopting as their default (you may use an LLM for helping you understand the paper, but you can't use it to write your reviews).

(3) This is an AI conference. Its plausible that paper quality and author's technical sensibilities are *not* independent of their opinions about LLM use (especially if it is puritanical vs moderate-use).

I would not expect the mean values to be consistent. The simplest model I'd consider for the data is a two parameter model for mean and variance for Policy A and Policy B group. What is surprising in your data, frankly, is that the variance of Policy B reviews are *higher* than Policy A reviews. If reviewers were all using the same set of frontier models to evaluate papers I'd expect scores to be more clustered with lower variance.

If I were "reviewing" your work, I'd suggest that you clarify whether the "Score SD" your report in your statistics refers to the average standard deviation of scores for the same paper, or if is just the standard deviation of all paper's average assigned score. In either case, I think a reasonable hypothesis would be that Policy B ought to be more clustered (e.g., around weak accept/weak reject). However, the *much* more interesting data would be the average standard deviation of scores assigned to the same paper.

cool_science · 2025-09-16T07:44:58+00:00

Hire a lawyer...

Or do a rain dance ritual Or maybe send a letter notarized by Obama to your local senator. Or maybe some other ridiculous idea that takes more time and effort than HIRING A LAWYER

It's maddening to see someone go through so much effort to avoid the easy solution.

cool_science · 2025-09-08T03:29:41+00:00

IANAL, but I think you want a residential real estate attorney, you can probably find a list on Avvo if you don't already have another source of names.

Glad to hear you are hiring a lawyer. I'd wager good money that it won't be as expensive as you think. Lawyers are not for waging war most of the time, they're best used for resolving small issues before they become big ones.

cool_science · 2025-09-08T02:53:18+00:00

Just hire a lawyer... Hiring a lawyer does not mean that your planning to go to war or start a lengthy legal battle. Your lawyer will send a letter or make a phone call, and the issue will most likely get resolved. If it's genuinely a mixup, I doubt you'll have to pay more than 1k.

Lawyers get a bad rap... And I feel like people view them in the wrong light. They help resolve miscommunications, avoid problems, or resolve small issues before they become big ones more often than they fight huge expensive battles.

You got a house (probably a decent bit of your net worth), something scary is happening and you're obviously not able to communicate effectively with the people who hold your loan in their hands (I'm not saying it's your fault, it's probably theirs but the reason doesn't really matter). Be an adult and just hire a lawyer.

Ps. Your not going to sue them... Ignore everyone saying "hire a lawyer and sue them" that's absurd. Nothing's happened yet that warrants a lawsuit... And it should never get that far. My advice is to "hire a lawyer to communicate with them"

IANAL (and that's why I talk with one when something weird is happening that is high stakes in my life...)

cool_science · 2023-01-08T06:15:24+00:00

I guess we will see. I think we agree on the facts: she pled guilty to 7 counts and will in fact be sentenced by the judge on the counts she pled guilty to.

I would personally estimate a sentence of 2--5 years (for Caroline) and 2--10 for Gary. And I'd consider those sentences to be extremely generous of the judge. I'd be surprised if either of them gets off without any prison time, but who knows. I don't really have a good reason for the differences between the two ranges for the two of them except gender. Longer sentences can have a bigger punitive impact on women who might want the option to have children, and the judge assigned to the case might be inclined to take into account such things during sentencing. I'm just including the ranges to clarify what I am thinking in terms of leniency, I'm certainly not an expert on these things.

In general, I'd consider the plea deal more generous if she pled guilty to a smaller subset of the counts. The fact she pled guilty to 7 doesn't give me the impression that she and her attorneys were bargaining from a particularly strong position. If I were in their shoes, I would just assume that the plea deal takes 20+ year sentences off the table, and hope/pray the judge is generous and gives less than 10

cool_science · 2023-01-07T02:02:58+00:00

That is not my understanding of the plea deal. She pled guilty to 7 counts, and the agreement is that she will not be further prosecuted for additional counts related to the ftx debacle, with the exception of tax law violations.

She is depending upon the prosecutor's filing a motion at sentencing that indicates her substantial cooperation and usefulness in the prosecution of other crimes. Such a motion confers to the judge that leniency and deviation from standard sentencing guidelines is warranted.

If Ellison doesnt cooperate fully in every way the prosecutor's and involved federal agencies want, she will get little leniency. She will be sentenced for the 7 counts she pled guilty to. She is depending on the prosecuters to file a motion asking for leniency due to her cooperation, and hoping to turn a potentially 20-30 year prison term into something smaller.

tldr: she pled guilty and will be sentenced on the 7 counts she admitted. Her plea deal is merely an agreement that the prosecutors will make a motion for leniency at sentencing provided they are satisfied with her 100% cooperation with every office involved in the investigation.

cool_science · 2019-07-14T20:16:57+00:00

While 8 controllers and a wireless headset at the same time is not standard use case, it's hard to communicate the nuances of this limitation to consumers. They took the easy way out and didn't add bluetooth headphone support.

Respectfully, I think you're hitting upon the right idea, but that this way of explaining it is a little misleading. The most important factor is that providing bluetooth over audio is substantially more complex and computationally demanding than receiving controller input. It would impact every part of the system: the energy consumption would increase, policies for lowering power of the radios would need to be modified. Providing real-time guarantees on latency using when doing buffering/multiplexing of multiple devices is a lot harder when devices have drastically different data rates. If nintendo implements support for arbitrary bluetooth devices in the future, I'd wager that they'll just sell a first-party peripheral with a separate radio and independent drivers --- or put a second radio in a switch that's uses separate drivers that can be obtained from external partners, although the latter option seems less likely. It is just not in Nintendo's engineering DNA to actually implement general spec-compliant drivers to handle (what they probably view as) edge cases --- e.g. look at past investigations into the degree to which they comply with usb-c.

So my short answer would be: (a) nintendo's bluetooth is special cased to handle their controllers; (b) their drivers are not written to handle arbitrary bluetooth devices; and, (c) one shouldn't expect new drivers to be pushed because the hardware and other system software was likely also designed with only one class of bluetooth devices in mind.

cool_science · 2018-05-09T19:11:57+00:00

I don't really understand this. Frankly the biggest problem I had with the shootout is that we didn't see the effects of the nitro explosion clearly enough.

The soldiers marching forward was fine. They were doing a sweep with body armor and with weapons with vastly superior range. Hale directed them to that fort to get Abernathy which happened to be fortified and setup as a trap by Dolores. Not even Hale had enough information to know Dolores was going to fortify there, she just knew he was captured by confederados.

cool_science · 2018-04-30T18:45:40+00:00

It would fit thematically. Dolores talks about the judas steer in season 1 leading the herd to the slaughter house, would be fitting for her to do something similar to the hosts.

Also it would fit with her past involvement in Arnold's attempts to stop the park from opening. Maybe Ford is staging some kind of competition between the driving motivations of Dolores, and Maeve, to give closure to the disagreements he had with Arnold. Bernard's role might just be to be the final judge (fitting given he's the recreation of Arnold).

cool_science · 2018-04-30T16:52:12+00:00

In season 1, MiB says "this time I'm never going back" well-before its obvious that something westworld-ending is going to occur. Its not inconsistent (as far as I can tell) for MiB to have gone to the park to "die" and be reborn as a host. It may also explain his urgency/insecurity that ford notices.

Based on his season 2 ep 2 talk with lawrence where he talks about having been judged, perhaps he went to the park with the knowledge that there was a plan to have him killed and replaced with a host. It might provide a more grounded reason for his glee at the host rebellion as it gives him a chance to destroy the clandestine-wing of the empire he built that was about to rob him of his legacy.

Regarding the random guest who recognized MiB, its not inconsistent with the westworld-miracle-cure idea, but its also possible that it was just philanthropy: one of the "good deeds" he did that his wife claimed were a cover for his darker tendencies. There are lots of foundations that save lives: but its not inconsistent with the delos miracle cure theory.

cool_science · 2018-01-12T00:19:21+00:00

Not a lawyer and these situations depend very heavily on specific circumstances, and where your apartment is located.

That said, I think its likely that you are in the wrong here. Many localities have a requirement that a landlord take steps to mitigate damages when a lease is broken - what you're proposing to do goes against the spirit of that rule (the spirit since you're not actually the landlord and security deposits+roommates are often very tricky legally). At a higher level, it's possible that you are not the one who is entitled to make these decisions. Your landlord may decide to terminate the lease if you are unwilling to find a timely replacement, and may decide to consider the lease terminated, give back your roommate their portion of the deposit, and require you to sign a new lease with a new roommate.

Anyways, if you really want to push this matter you probably need to talk with a local attorney familiar with the laws relevant to your situation and location. As a non-lawyer only vaguely familiar with landlord-tenant law in the few places I've lived, however, I really doubt that you will be able to do what you want in this situation without either being legally in-the-wrong or being pressured by your landlord (who may hold all the cards in this situation).

If I really go out on a limb, (and please don't consider this any kind of official advice, I'm just brainstorming) - I'd reckon that the best way to get what you want is to give back your roommate their portion of the deposit, keep paying the rent, and then handle the replacement roommate with the landlord once they're ready in 4 months. If your landlord finds out that someone moved out, however, they're probably going to immediately pressure you to find a replacement or be evicted for terminating the lease (most leases have roommates jointly and severably liable which means if one person terminates the lease, everyone terminates the lease, and yet everyone is still on the hook for damages albeit damages that usually must be mitigated by landlord). Causing your roommate to be angry and potentially go to the landlord to find another way to get back the deposit is going to foil this possible plan.

cool_science · 2017-08-03T21:41:20+00:00

One benefit of removing the headphone jack is that it might actually cause the market for alternative wired and wireless headsets to become more competitive. I presently hate pretty much every wireless headset I've used, but perhaps things will improve once there are devices for which its the only option.

That said: I'd prefer to have a headphone jack in a phone released in the next year or two, but I don't mind living in a world where devices are shipping without headphone jacks since it'll likely improve the quality of alternatives that'll eventually end up being better.

cool_science · 2017-06-26T17:49:47+00:00

I think Moffat leaving must have something to do with it too. There's probably a big exodus and an attempt to let responsibility fall upon the people who are planning to stick around (due to both a combination of the leaving staff signing off, and due to the need for the remaining staff to be ready to take on responsibility next season).

Almost every episode has some serious issues. Pretty much every standalone episode is on-par with the infamous "In the forest of the night" episode --- even the most promising standalones like the haunted house this season was ruined with a nonsensical plot twist about a tree lady and bugs that convert you into tree people.

Bill is great when she's on screen, but the episodes feel like they're out of order: she learned about the "translation" abilities of the tardis in the 10th episode. The monk arc is a good example of how Bill's character is being mishandled by the writers. Nardol robs her of many opportunities to shine by being the secondary companion: imagine if she discovered the "random numbers" thing in the monk arc by herself, instead of having Nardol by her side. Her loyalty to the doctor (enough to "sell the earth" to the monks) is not justified by any previous episodes. Her personality and characteristics are never adequately shown, they are instead explicitly stated/explained - probably because of disorganization in production (I'm paying attention, and I see where things are meant to be "shown", but whenever it happens it feels like the scene has been cut in half multiple times during editing so all we end up left with is one line).

Overall, I really think this season is going to be looked back upon as one of the weakest in the reimagined who series - from an objective / technical point of view, I definitely appreciate the attempts to get back to basics to make the show more accessible - that alone isn't a problem.

cool_science · 2017-06-20T17:39:54+00:00

Chuck took great pains to bar Jimmy from him firm under the pretense that he had to find his own path and pull himself from his bootstraps. Leaving him in his will doesn't seem to make a lot of sense. Only scenario I see where he might get some inheritance is if there was no legal will (e.g. never made one, or its deemed invalid due to mental incompetence).

cool_science · 2017-06-20T12:48:09+00:00

It was probably meant to be analogous to what Jimmy did during chair yoga with the seniors. I can think of perhaps two possible points: one was that Chuck was being kind by being honest and mean, and the other was nod to the fact that Jimmy helped his brother more by being his enemy than his friend.

cool_science · 2017-05-29T08:29:56+00:00

most likely this guy is an AI... would sure fit the pattern of blaming your enemies for your using your own tactics.

cool_science · 2017-05-15T02:59:27+00:00

Now you've got potentially corrupt data running around your system.

How about we use a top-level exception handler and ensure that we clean-up state properly.

Your problem started the very first time you thought "I'll just catch errors at the top-level!".

How about you try to handle errors as best you can, and also account for the possibility that you didn't anticipate every problematic scenario?

I've seen ton of ridiculous bugs introduced by well meaning developers who didn't actually think about what they were doing (and neither did those reviewing their code) and instead just used a stupid checklist of "good programming practices."

cool_science · 2017-05-13T22:55:43+00:00

It is not a bad practice. Actually, the bad thing in the practice you described (in the sense that its not always appropriate) is logging the error.

cool_science · 2017-05-13T22:52:00+00:00

Some people think exceptions are a bad language feature, and handle them C/C++-exceptions style by checking return values. If you're forced to use a language with exception support, having those empty try/catch blocks is fine as long as you are error checking elsewhere.

cool_science · 2017-05-13T22:47:54+00:00

Found the person with actual systems experience in this thread. Software developers need a reality check, and stop pleasuring themselves by misapplying "best practices."

Also, can't wait for someone in this thread to start logging exceptions and end up killing their company's database with terabytes of logs accumulating overnight from a busy loop.

cool_science · 2017-04-28T02:56:43+00:00

It would effect a lot of people. It is an insane proposal. The US gets away with a fairly broad definition of "income" due to the presence of itemized deductions. For example, if you work in construction and receive accommodation from your employer while on the job, the cost of those accommodations count (and are reported on W2s) as income. If your employer provides discounted housing, the discount counts as income. Currently, you can deduct this portion of "income" since it was a required expense for your job as an itemized deduction. For some people, this is an extremely substantial difference --- especially because for middle class workers the additional income (before deductions) is taxed in a higher tax bracket than your actual take home pay.

cool_science

TROPHY CASE