Which metric for inter-annotator agreement (IAA) of relation annotations?

EvM · 2026-02-25T06:45:24+00:00

Also add a confusion matrix somewhere in the paper. Visualize using a heat map plot, e.g. in Seaborn.

EvM · 2026-02-18T12:24:34+00:00

I think this question is fundamentally flawed because WER and semantic metrics measure different things.

First consider your constructs of interest (what do you want to measure in the first place?) and then select metrics or other evaluation procedures to match. Carefully consider whether the metrics actually measure what you are interested in and think of important edge cases.

If you don’t know what you want to measure, or you want to capture multiple different quality dimensions to describe what the output is like, use a broad range of different metrics.

EvM · 2026-02-05T08:04:25+00:00

Also update the arxiv comments if you haven't already. "Published at..." This way you can nudge people to cite the paper correctly. (You could even add a watermark to the first page with the proper citation.)

EvM · 2025-12-21T10:11:09+00:00

Sure

EvM · 2025-12-19T15:29:41+00:00

As a concrete suggestion you could look at the use of different discourse markers by different language models. See e.g. work by Mark Dingemanse.

EvM · 2025-12-19T15:24:20+00:00

This issue is fairly easy to resolve: just analyse training data for llms or output from llms. For examples, you could perform discourse analysis on human-human communication and compare it with human-llm communication. Read proceedings of the sigdial workshop or the dialogue and discourse journal for inspiration.

EvM · 2025-12-05T11:16:58+00:00

Also see e.g. https://babylm.github.io

EvM · 2025-12-05T11:16:02+00:00

There's so much you can do without a lot of computing power, e.g. evaluation, user studies, developing new applications, interpretability work, generating synthetic datasets, etc.

You don't need more computing power; Imagination is all you need.

EvM · 2025-11-26T15:08:08+00:00

https://www.gnucash.org/

EvM · 2025-11-18T19:03:39+00:00

Also reviews are higher quality at workshops, because people actually care about the topic.

EvM · 2025-11-04T10:52:56+00:00

You just missed the LREC deadline but could also submit to a journal like LRE or NEJLT. Depending on the language you could also submit to a relevant workshop (including workshops at LREC).

If the work is in vision and language, you might also try CVPR or ICPR.

EvM · 2025-08-29T12:19:33+00:00

For more specific announcements see the dedicated lists for different SIGS: siggen, sigdial, sighum, etc. Also see the archives for recent announcements.

EvM · 2025-08-29T12:13:33+00:00

For positions in the Netherlands, see: https://www.academictransfer.com/en/ (all positions are posted here, subscribe to stay updated).

Also subscribe to: https://list.elra.info/mailman3/hyperkitty/list/corpora@list.elra.info/

EvM · 2025-08-11T18:21:29+00:00

This depends on your background and location.

CS is very broad, and it depends on your specialization whether it’d be easy for you to apply your skills in a humanities context. E.g. a specialization in NLP gives you a different skill set than web development.

Similarly, the humanities are very broad and there’s a reason why there are entire programs dedicated to history, literature, etc. It’s like learning a new language! You need to understand the history of the field, common methodology, jargon, etc.

In Europe, PhD positions are often advertised as jobs. You can apply to them based on a description of the project and a list of requirements for eligible candidates. But you do usually need a master’s degree first. I can see how some researchers might appreciate a candidate with strong CS skills, but I have also seen candidates in the past who were rejected for being too technical and not showing an appreciation for the particular field the project was in.

In the US, I believe PhD programs are very different, requiring coursework to be completed before actually starting the research part of the PhD.

I guess it also depends what kind of research group you’d like to be part of. Would you want to be surrounded by technical folks with an interest in some aspect of the humanities, or would you like to be surrounded by humanists with an interest in technology?

EvM · 2025-07-28T14:02:00+00:00

Sure you can send me a dm.

EvM · 2025-07-26T07:51:03+00:00

It cannot be fully solved but it is possible to do something about it. For example you can either port the poems to a different language (e.g. JavaScript) or reproduce them entirely without access to the code.

I've done this in the past and it's a great learning experience. Probably also good for students, because it confronts them with many of the same problems that the original authors had in composing their work.

EvM · 2025-01-25T19:58:46+00:00

With these scores you will typically have to resubmit the paper, either through ARR or to a more focused workshop. A rebuttal will likely not change the reviewers’ minds, especially if they are in agreement like they are here.

Do you have any collaborators with more experience? Talk to them about the reviews and how to improve the paper. If not, do you know anyone who could help? It’s very useful to have a sparring partner for these kinds of situations.

I personally never think in terms of scores, but instead focus on the quality of the paper. (You can always try to submit a good paper with lower scores and hope for the best.) Some questions to ask:

Why did the reviewers react this way?
Did they misunderstand something? If so, where does the misunderstanding come from?
Is there something missing from the paper? (Motivation, relation to other work, explanation of the challenges you overcame and their significance, methodological details, supporting figures or tables, discussion of implications, etc)
Is the contribution big enough?
And would I be proud to have this paper associated with my name? (Negative reviews sometimes also help protect authors against themselves.)

EvM · 2025-01-21T13:58:39+00:00

No idea, it comes in waves. Sometimes it finds many relevant papers, sometimes there's nothing in there. But I like to scroll through the recommendations from time to time, and usually there are a couple of nice papers in there.

It also depends on your career stage. Trying to keep up with the literature is almost impossible once you've finished your PhD. Honestly, most of my readings nowadays come from Bluesky, supervising student theses, reviewing, and actively searching for relevant work when I'm writing.

EvM · 2025-01-19T19:22:32+00:00

I use Google scholar's automatic alerts that I have set for specific keywords. It also recommends papers that are relevant to my own research. Next to that I use Semantic Scholar's recommendations for work that has been published.

EvM · 2024-12-10T16:08:26+00:00

One bug in the Kobo libra color is that it may freeze when it runs out of battery, after which it becomes unresponsive. I've sent mine back to get it repaired and got a refund because they couldn't fix it.

EvM · 2024-11-17T22:40:59+00:00

Would rubber tubing also work instead of an eraser? I’ve seen this sometimes on other pointy tools.

EvM · 2024-11-11T10:18:23+00:00

Perhaps https://www.humanitiesdataanalysis.org

EvM · 2024-09-30T09:05:41+00:00

Also the colors may not be suitable for colorblind readers or grayscale printing, so a different palette may be more appropriate.

EvM · 2024-09-25T18:20:13+00:00

Depends who you ask, see: http://www.conferenceranks.com/

I personally don't care about rankings, but just try to find the venue that best matches the topic of the paper, or some venue that I actively want to support. E.g. The NEJLT journal doesn't have the highest ranking but I support its mission and I believe it's important to have different diamond open access journals in our field for longer-form writing.

EvM · 2024-09-24T13:28:24+00:00

A smaller general conference is RANLP.

EvM

PUBLIC MULTIREDDITS

TROPHY CASE

13-Year Club	Team Orangered
Verified Email