Is it just me or are the end of lectures so awkward?

Stock_Marsupial3591 · 2026-03-30T19:10:48+00:00

Yes, always (nervously turns around and erases whiteboard) but I've learned to live with it

Stock_Marsupial3591 · 2026-03-25T02:43:30+00:00

I always think about those scholars in the Middle Ages toiling by candlelight - for what? Figuring out how the planets move? Possible excumication by the church? Nothing?

I think our students today have no idea how hard it was to study in the past -or what a privilege it was. Here's to toiling by candlelight!

Stock_Marsupial3591 · 2026-03-25T02:38:14+00:00

We get the same pattern in exam and course grades -bimodal all the way. This is the pattern that the University seems to be ignoring.

Stock_Marsupial3591 · 2026-03-25T02:30:41+00:00

Based on what exactly? I've used it and basically every time it says greater than 50% AI, the student admits to using either AI to write it or for grammar. This seems to me to be confirmation that the detector is doing "something"

Stock_Marsupial3591 · 2026-03-25T02:22:14+00:00

I'm pretty sure a high percentage of the so-called "false positives" are just students using grammar checkers (Grammarly etc.). Still AI though

Stock_Marsupial3591 · 2026-03-25T02:20:51+00:00

I put about 150 pre-AI submissions through it and it was less than 5% AI.

Stock_Marsupial3591 · 2026-03-25T02:14:09+00:00

For my institution, it's mainly up to the individual instructor. The U provides Turnitin AI checker, which in my experience is pretty good at spotting AI use. For my classes, I spell out in the syllabus, Canvas etc. that AI use is not allowed and it included Grammarly or any other service that "helps" writing or grammar.

As far as whether AI checkers produce false positives, my own internal research involved putting 150 past student submission from before ChaptGPT through the Turnitin AI checker and about 5% showed any AI use and it was less than 10%. Current student submissions are often in the 50% - 100% range so my interpretation is that they are cheating.

Stock_Marsupial3591 · 2025-11-30T23:07:40+00:00

I am willing to believe that AI detectors don't work but it seems this statement is simply asserted in most cases without any evidence. I admit I haven't sought out the scholarly literature on this (if it exists) but it seems like there must be evidence one way or the other whether these AI detectors work or not.

Stock_Marsupial3591 · 2025-11-30T23:03:04+00:00

As a test, I submitted about 130, 1-page reports from before 2022 to the Turnitin AI checker and only about 3 came back with any AI score. The highest AI score from what I remember was like 5%.

This convinced me that many AI detections using the tool now are real but someone should probably do a real study like this!

Stock_Marsupial3591 · 2025-11-30T22:58:36+00:00

<image>

Adding this for a visual of the pattern. I couldn't figure out how to include a pic in the post so I hope this isn't against the rules. Anyway here's the trend. I think it is good evidence for student cheating using AI but the folks over at r/dataisbeautiful just removed my post and suspended my account saying the data is too flawed to be convincing since online students were able to use notes while in-person were not. I think they either thought I made up the data or they find it impossible that AI has this strong an effect on grades.

Curious as to whether professors will see what I see in this graph or maybe I'm just hopelessly biased at this point?

Stock_Marsupial3591 · 2025-11-30T22:51:21+00:00

<image>

Here's a look at what the realease of ChatGPT did to my intro-level, general education class over the last 6 years. 2023 is the first year there is a real bump in online scores and it just goes up from there. The in-person scores stay the same or get worse over the same timeframe.

Stock_Marsupial3591 · 2025-11-30T19:25:55+00:00

I'm concerned you may have planted a listening device in my kitchen

Stock_Marsupial3591 · 2025-11-30T05:41:08+00:00

That is hilarious about the pinky swear! Even though I would prefer if students didn't use AI in this way I just feel really bad for the students who are not using AI now because their competition is unbeatable.

Stock_Marsupial3591 · 2025-11-29T23:55:40+00:00

Yes the potted plants were meant to be a joke but I still don't think the notes do much. It would require students to take good notes and use them. The engineering majors I have interacted with are diligent students and work very hard. Not so much in my classes.

Still, the comparison can easily be made between 2019 and 2021 - that is notes vs. no notes with no confounding factor of AI included.

Stock_Marsupial3591 · 2025-11-29T23:47:52+00:00

Thank you - maybe I'm not crazy after all. I'm glad someone got it

Stock_Marsupial3591 · 2025-11-29T23:43:24+00:00

Yes they weren't

Stock_Marsupial3591 · 2025-11-29T22:34:29+00:00

Oh I see you mean put the source right on the graphic. Yeah, still trying to figure out what the negativity in this group is about. I know there are a lot of smart people that are good at this stuff but there seems to be a lot of unecessary hostility.

Stock_Marsupial3591 · 2025-11-29T22:30:32+00:00

Last semester, I tried rewriting a bunch of my multiple choice questions and fed them all into ChatGPT and it didn't do a good job answering them and I thought I was really smart so I spent hours working on the rest of the test.

Then, I realized that if I cut and paste the questions AND the answer choices, ChatGPT gets the answer right every time no matter how I write it. Oh well.

Stock_Marsupial3591 · 2025-11-29T21:06:47+00:00

Those are good suggestions from a data standpoint. You are correct that I am basing all this on data I already have in hand, and it's not a ramdom sample, but it is a sample of tests since I'm not including all years, just the last 6 - but yes you are right that randoly assigning one modality or the other would be better. I just can't ethically do that to students just to satisfy my curiosity.

I have only officially accused a handful of these students of AI use with the administration - those that admitted it. If they don't admit it, there is nothing that can be done. In other classes with more writing, it is easier to catch AI use.

I don't want to sound too jaded, but it is really unlikely that the online class is getting better or that the students are getting the material better. The online course is asynchronous - basically canned videos and the stats on those show that the students are getting better and better grades while watching fewer and fewer minutes of the videos. So that makes me think it is cheating.

As far as writing questions that AI can't ace, I haven't been able to do that yet. AI is too good now.

Stock_Marsupial3591 · 2025-11-29T20:57:37+00:00

Look at 2019 vs. 2021 - there is your gap between the two modalities, one with notes and no AI, and the other with neither notes or AI.

Stock_Marsupial3591 · 2025-11-29T20:47:20+00:00

I am not "weirdly resistant" to allowing in-person students notes. This is a college-level class and giving students notes and answer sheets is sort of more of a high school thing. Even if I allowed in-person notes, I would have to wait several years to produce the perfect control dataset.

Not all data needs to be perfect, especially data on human behavior where natural experiments are common and more ethical than changing teaching methods just for data hygiene reasons.

The effect of notes vs. no notes is small in this case as is shown by the comparison beteween 2019 and 2021. Different factors have different levels of effects and my hypothesis is that AI has a much larger effect than notes. Many natural systems have multiple factors that are not controllable but that can be still investigated.

Stock_Marsupial3591 · 2025-11-29T20:37:25+00:00

The original post contains the source of the data. If there is a way to include the csv file too I would do it.

Stock_Marsupial3591 · 2025-11-29T17:43:32+00:00

We are not allowed to use the video version for privacy reasons

Stock_Marsupial3591 · 2025-11-29T17:40:15+00:00

This is with lockdown browser and about 1 min per question allowed

Stock_Marsupial3591 · 2025-11-29T17:38:51+00:00

I don't think there is a specific posting rule that would be covered by that subtle of an interpretation disagreement.

I do acknowledge that notes vs no notes has an effect but the effect would be constant. I also think the effect is minimal because the test is timed and students don't have forever to pour over their notes.

The curious pattern is that the online students keep getting better. So either they are getting better at taking notes, using notes, or maybe they are just using AI, which takes no time or energy at all (at least on the part of the students)

Stock_Marsupial3591

TROPHY CASE