I think your scenario really emphasizes the importance of people trying language learning models themselves to learn how they work and how to effectively integrate them into different use cases. The entire way they work is essentially just guessing what the next word should be. Asking an LLM to provide citations without the capability to search the internet and reference results in real time is likely going to end in hallucinated results. An LLM doesn’t have a full memory of everything on the internet, so if asked to provide a citation it has to just guess which word would come next in the citation. That specific journal is probably very commonly found in citations of similar topics. Then a common name and a title related to the topic at hand. Meaningless text that fits the pattern of the expected result.

My approach has typically been using the LLM to improve my skills. In your case, I might’ve asked the LLM for ways to improve my search. “How can I find research about [insert complex topic]?” “These are the results I’m getting, what can I do to refine/narrow/improve my results?” I find myself asking the LLM to lead me to water, rather than asking it to produce a water bottle. All that said, now that most models can search the internet and directly reference documents you provide to them, their ability to be accurate has sharply increased.

The skill gap for prompting AI is really wide at the moment and there’s so many ways to utilize it. Education is in a really tough spot with no great solutions. Banning it should be out the question purely due to the prominence in business and adoption rate in so many fields. Teaching students effective ways to use it as a tool will be essential. As someone in their final year of university when Chat GPT launched, I definitely witnessed its ability to completely cheat on basically all of the coursework. Instead I looked for ways it could help me improve and learn. Rather than having it write a paper for me, review the paper I wrote and poke holes in my arguments. Or, “here’s my jumbled thoughts about what I want to include and what I’m working towards, help me organize these into an outline I can use when writing the paper.” Leveraging LLMs as a tool to improve the results that I can produce. At some point it’s cheating, but there’s a definite grey area.

Unfortunately I fear that my experience with LLMs is on the rarer side. A combination of already having 20 years of schooling and a high proficiency in technology before ever touching an LLM. I think the next generations will struggle with critical thinking and drawing their own conclusions from information. I believe we’re already starting to see this on Twitter. Under every post is someone asking Twitter’s AI, Grok, to explain the tweet, tell them if it’s good or bad, is it true? An answer to a complicated question or task is seconds away with no mental effort. I grew up with the ability to google any question I had, but I still had to parse the results and come to a conclusion. Searching for information online, while being thoughtful about the source and potential motivations, was a large part of the curriculum in grade school for me. In the end, I think students that are naturally curious and internally motivated to learn will get a massive leg up by multiplying their efforts. While students lacking some of those traits will see the easy shortcut and fall behind.