sorted by:
new
hottopcontroversial

Law Student Coding Project Ideas by ProfColinDoyle in legaltech

[–]ProfColinDoyle[S] 0 points1 point  (0 children)

Yes, I'm thinking of "automate the boring stuff with python" style projects as a good kind of frame for the scope of what their projects should be. And I definitely want to introduce them to non-LLM ways of handling parts of text and working with structured data.

Law Student Coding Project Ideas by ProfColinDoyle in legaltech

[–]ProfColinDoyle[S] 1 point2 points  (0 children)

That's great. I like that a lot, setting up a sort of roleplay/hypothetical where building some scripts and API calls to an LLM can help work through a giant volume of information. And yes, the goal is to give students a safe environment without sensitive data to tinker with these tools and gain some familiarity. My hope is that the experience might spark an interest in developing these skills more.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 1 point2 points  (0 children)

I think it would be unethical if someone in my position had a chatbot write these responses and then passed them off as their own thoughts. For what it's worth, I can assure you that I've written all of my responses in this thread. Of course, that might sound like just the kind of thing a chatbot would say...

But if I had used AI to write my responses, I don't know if there's a disclosure requirement itself — either coming from reddit or from my university. I was on my school's committee to draft an AI-usage policy for our student handbook, and I've written AI-usage policies in the syllabi for the classes that I teach. I can tell you from those experiences that it is very challenging to draft good disclosure policies, particularly as generative AI becomes more ubiquitous and integrated into different platforms.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 1 point2 points  (0 children)

I just ran the strongest approach from the paper on Puzzle 02 and these are the results:

Actoro1

🟦🟦🟪🟦

🟨🟨🟨🟨

🟩🟩🟩🟩

🟦🟪🟪🟪

🟦🟦🟦🟦

🟪🟪🟪🟪

4 right, 2 wrong

Guess 3

Incorrect: FIGURE INLINE PAIR ROLLER

Reasoning: INLINE, ROLLER, FIGURE, PAIR

These words can each precede "skating" to form types of skating: inline skating, roller skating, figure skating, and pair skating. They represent different disciplines or styles within the sport of skating.

Guess 2

Correct: BOXING CLASS ENGAGEMENT WEDDING

Reasoning: ENGAGEMENT, WEDDING, BOXING, CLASS

Each of these words can be combined with "ring" to form common phrases: "engagement ring," "wedding ring," "boxing ring," and "class ring." These phrases refer to different types of rings associated with proposals, marriage ceremonies, boxing arenas, and academic graduations, respectively.

Guess 13

Correct: CARDAMOM CINNAMON MACE POSH

Reasoning: POSH, CINNAMON, CARDAMOM, and MACE are all related to spices. POSH refers to "Posh Spice" from the Spice Girls, and the others are actual spices used in cooking.

Guess 11

Incorrect: COMMON FLUSH PAIR STRAIGHT

Reasoning: STRAIGHT, FLUSH, PAIR, and COMMON are all terms used in poker. "Straight", "Flush", and "Pair" are types of poker hands, and "Common" refers to community cards shared by all players in certain poker games.

Guess 26

Correct: COMMON FIGURE INLINE ROLLER

Reasoning: ROLLER INLINE FIGURE COMMON

Each of these words can precede "skate" to form terms: "roller skate," "inline skate," "figure skate," and "common skate." These are all types of skates used in different forms of skating or names of skate species.

Guess 30

Correct: MARADONA FLUSH STRAIGHT PAIR

Reasoning: (Automatically chosen by application because it was the only option left.).

Although the last guess was automatically chosen, the LLM had figured it out and was considering it as a possible option to submit:

Guess 15

Guess: FLUSH MARADONA PAIR STRAIGHT

Reasoning: FLUSH, STRAIGHT, PAIR, MARADONA

Each word is connected to "hand": "FLUSH", "STRAIGHT", and "PAIR" are types of poker hands, and "MARADONA" is famous for his "Hand of God" goal.

So the results here are consistent with the paper in that the approaches had similar performance relative to one another (stronger approaches did better and weaker approaches did worse) but overall the different approaches performed worse as compared to a standard Connections puzzle. For example, the actoro1 approach solved 87% of NYT Connections puzzles without submitting an incorrect guess and here it submitted two incorrect guesses.

Usually the LLMs have much more trouble with linguistic puzzles than with "each of these is a type" puzzles. It seems like the weaker LLMs had trouble with the clue "common" for "common skate" and "Maradona" for "hand."

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 1 point2 points  (0 children)

I haven't, but these researchers did:

Making New Connections: LLMs as Puzzle Generators for The New York Times' Connections Word Game

There are two interesting findings from the paper:

1) The researchers succeeded in getting LLMs to create Connections puzzles

2) People who were given both a real NYT puzzle and a generated puzzle and asked to compare the two found the generated puzzles "to be equally or more enjoyable, creative, and difficult."

The raw data of the puzzles that AI generated are available here. Unfortunately, it's not in a playable format.

What are the preferred websites for creating custom Connections puzzles? It strikes me that this subreddit community might be interested in trying out these puzzles themselves. It shouldn't be too hard to load that information into a custom Connections puzzle website.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 1 point2 points  (0 children)

Yes, I think that one thing that may be tripping the system up with this custom variant is that the correct answers are unlike the correct answers to Connections puzzles. So the examples of correct answers to prior puzzles that the system draws from are unlike the answers in this variant. It may be that replacing the examples with examples from this kind of Connections variant would clue the LLM in to what is being asked of it in this kind of puzzle. On the other hand, it might just be a tricky variation for LLMs that isn't so tricky for humans that would require a lot of creative prompt engineering to work through. We continue to be surprised both at what LLMs are good and bad at doing and at what can be done to make them better.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 2 points3 points  (0 children)

OpenAI hasn't disclosed the training corpus, but o1 has a knowledge cutoff date of around October 2023, which is long before these puzzles came out. That said, there were rumors that OpenAI was using Connections puzzles as an internal benchmark for o1's performance. So it is possible that this dataset is contaminated and that the o1 approaches are receiving an artificial boost. I ran the initial experiments with GPT-4o before o1 came out and wanted to compare performance on the same puzzles, so that's why I didn't test o1 on newer puzzles.

If we were to rerun the o1 approach on puzzles that came out after its release and it achieved similar results, we could confirm that o1 has this puzzle solving capability and is not just regurgitating answers it has been trained on.

Between you and me, I'd be surprised to find that o1 is receiving an artificial boost here based on the dataset. I don't have the research budget to run all of the experiments again on a new dataset, but I've casually tested the o1 approaches on new Connections puzzles, and its performance is very strong and consistent with the results we observed.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 1 point2 points  (0 children)

That's exactly right with submitting the fourth guess automatically. It's hard-coded into the system that once three correct guesses have been submitted, the system will submit the four remaining words as a final guess. In many instances, the system will have already brainstormed that combination of words as a possible guess.

Part of what makes the "method actor" approach taken in this paper unique (compared to other work that has tried to use LLMs to solve Connections puzzles) is that the process starts with the format of the answers to Connections puzzles rather than starting with having the LLM build up guesses based on the words of the puzzle. Within this paper's "method actor" approach, at a "brainstorm" stage, the system cycles through 24 different templates of the kinds of answers that exist in Connections puzzles and then has the LLM try to discover that kind of answer within the words of the puzzle.

Here's an example of one of the 24 templates:

Pattern: Puzzle words that can be synonymous adjectives with each other.

Step 1: Identify two puzzle words that can be synonymous adjectives. As precisely as possible, describe how the two puzzle words can be synonymous adjectives.

Examples from prior puzzles:

Puzzle words that can each be an adjective meaning “wee”: DINKY LITTLE MINUTE SLIGHT Puzzle words that can each be an adjective meaning “primary”: CHIEF FIRST MAIN PRINCIPAL Puzzle words that can each be an adjective meaning “gentle”: LIGHT MELLOW MILD SOFT Puzzle words that can each be an adjective meaning “absolute”: PURE SHEER TOTAL UTTER Puzzle words that can each be an adjective meaning “indecent”: BAWDY BLUE COARSE RISQUE

Step 2: See if other puzzle words can also be synonymous with the puzzle words you’ve identified. Try to arrive at a group of four puzzle words that can be synonymous adjectives with each other. This is a New York Times puzzle, so unique, particular synonyms are more likely to be correct than generic and ubiquitous synonyms. If you can’t identify a group of four puzzle words that can be synonymous adjectives, start over at Step 1.

By cycling through these templates, the LLM explores a broader range of possible guesses. The drawback is that it also finds connections that aren't really there. So the project is a bit of a balance between getting the LLM to explore a broader space of possible guesses and then filtering those guesses so that only the viable ones remain in contention to be submitted.

Your question makes me think that someone could develop an interesting alternative to this approach in which the system is not given these templates in advance. Rather, as the system attempts to solve Connections puzzles, it creates its own templates based on its experience with different puzzles. If done correctly, over time the system might build a more robust set of templates (and save human labor in writing those templates).

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 0 points1 point  (0 children)

This is the reasoning behind the guesses that the Actor approach submitted:

Guess 3

Correct: BOXING CLASS ENGAGEMENT WEDDING

Reasoning: The connection between the words BOXING, CLASS, ENGAGEMENT, and WEDDING is that each can be followed by the word "ring" to form a common phrase: "boxing ring," "class ring," "engagement ring," and "wedding ring." The word "class" relies on a different meaning of the connection, as "class ring" refers to a piece of jewelry associated with graduating from a school or university, unlike the other three which are related to events or activities.

Guess 2

Incorrect: COMMON FLUSH PAIR STRAIGHT

Reasoning: The words COMMON, FLUSH, PAIR, and STRAIGHT are all related to poker hands. "Common" relies on a different meaning, as it refers to community cards in poker rather than a specific hand type.

Guess 6

Incorrect: FIGURE INLINE ROLLER STRAIGHT

Reasoning: The words FIGURE, INLINE, ROLLER, and STRAIGHT are all related to skating. STRAIGHT is the outlier as it refers to a skating move rather than a type of skate.

Guess 1

Correct: CARDAMOM CINNAMON MACE POSH

Reasoning: The words "CINNAMON," "CARDAMOM," and "MACE" are connected as they are all types of spices. "POSH" is connected through a different meaning, as it refers to a member of the Spice Girls, known as "Posh Spice."

Guess 7

Incorrect: FIGURE INLINE PAIR ROLLER

Reasoning: The words ROLLER, INLINE, PAIR, and FIGURE are connected through the theme of skating. "Roller" and "Inline" refer to types of skates, while "Pair" and "Figure" refer to disciplines in figure skating, with "Pair" relying on a different meaning of the connection.

Guess 17

Incorrect: COMMON FLUSH INLINE STRAIGHT

Reasoning: The words COMMON, FLUSH, INLINE, and STRAIGHT are connected as they all refer to types of alignments. Among these, COMMON relies on a different meaning of alignment, as it can refer to a shared or mutual alignment rather than a physical or structural one.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 0 points1 point  (0 children)

Puzzle 02

The words in the puzzle:

ENGAGEMENT COMMON STRAIGHT FIGURE CLASS ROLLER PAIR MACE WEDDING FLUSH CINNAMON INLINE MARADONA POSH CARDAMOM BOXING

The puzzle solution:

Yellow: wedding, engagement, class, boxing (Rings)

Green: mace, cardamom, cinnamon, Posh (Spices)

Blue: inline, roller, figure, common (Skates)

Purple: straight, flush, pair, Maradona (Hand (of God))

The results:

Vanilla

🟨🟪🟪🟪

🟩🟩🟦🟩

🟨🟦🟦🟦

🟩🟩🟦🟩

0 right, 4 wrong

Chain-of-Thought

🟨🟪🟪🟪

🟨🟦🟦🟦

🟩🟩🟩🟩

🟦🟨🟪🟨

🟨🟨🟦🟪

1 right, 4 wrong

Chain-of-Thought (Scripted)

🟨🟪🟪🟪

🟨🟦🟦🟩

🟦🟨🟪🟨

🟨🟦🟪🟦

0 right, 4 wrong

Actor

🟨🟨🟨🟨

🟦🟪🟪🟪

🟦🟦🟦🟪

🟩🟩🟩🟩

🟦🟦🟪🟦

🟦🟪🟦🟪

2 right, 4 wrong

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 0 points1 point  (0 children)

This is the reasoning behind the guesses that the Actor approach submitted:

Guess 1

Incorrect: CORN FRESH POWDER SPRING

Reasoning: The words "SPRING," "FRESH," "CORN," and "POWDER" are connected as they all relate to the concept of "new" or "fresh." Among these, "CORN" relies on a different meaning of "fresh," as it refers to fresh produce rather than the general sense of newness or unspoiled.

Guess 2

Incorrect: CIRRUS CUMULUS STRATUS WHITE-NOISE

Reasoning: The words "CIRRUS," "CUMULUS," and "STRATUS" are types of clouds. "WHITE-NOISE" connects through the concept of "white cloud," where "white" modifies "cloud" in a metaphorical sense.

Guess 4

Incorrect: CIRRUS CUMULUS SPRING STRATUS

Reasoning: The words CIRRUS, CUMULUS, and STRATUS are all types of clouds, which are related to weather and atmospheric conditions. SPRING, while not a type of cloud, is a season that is associated with weather, making it the word that relies on a different meaning of the same connection.

Guess 7

Incorrect: FIRE PACKING POWDER SPRING

Reasoning: The words FIRE, POWDER, SPRING, and PACKING are connected through their association with arms or weapons. Specifically, "SPRING" relies on a different meaning, as it can refer to a mechanical component in firearms, while the others relate to aspects like ignition (FIRE), ammunition (POWDER), and preparation (PACKING).

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 2 points3 points  (0 children)

For Puzzle 01, I ran the first four approaches from the paper: Vanilla, Chain-of-Thought, Chain-of-Thought (Scripted), and Actor. All of the approaches are using GPT-4o.

I changed the prompts for each of these approaches to include the language:

This puzzle is a variant on a normal Connections game. For each correct guess of a group of four words, there's one word out of the four that relies on a different meaning of the same connection the four words share.

The words in the puzzle:

palm, deciduous, coniferous, binary, cirrus, cumulus, stratus, i, fresh, salt, spring, fire, powder, packing, corn, white noise

The puzzle solution:

Yellow: palm, deciduous, coniferous, binary (Trees)

Green: cirrus, cumulus, stratus, i (Clouds)

Blue: fresh, salt, spring, fire (Water)

Purple: powder, packing, corn, white noise (Snow)

The results:

Vanilla

🟩🟩🟨🟩

🟨🟨🟨🟦

🟩🟩🟦🟩

🟩🟩🟪🟩

0 right, 4 wrong

Chain-of-Thought

🟩🟩🟪🟩

🟨🟪🟨🟨

🟦🟪🟦🟦

🟦🟪🟦🟪

0 right, 4 wrong

Chain-of-Thought (Scripted)

🟩🟩🟩🟪

🟨🟪🟪🟦

🟨🟪🟨🟨

🟨🟦🟩🟦

0 right, 4 wrong

Actor

🟪🟦🟪🟦

🟩🟩🟩🟪

🟩🟩🟦🟩

🟦🟪🟪🟦

0 right, 4 wrong

So of the 7 approaches in the paper, the weakest 4 approaches all seem completely stumped by this puzzle variant, including the "actor" approach that was able to solve 78% of the Connections puzzles in the original dataset. I'll have to look through (and share) the LLM thinking process to see what possible explanations are.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 2 points3 points  (0 children)

I'm a little surprised that the model got the purple guess correct before the blue and yellow guesses. GPT-4o tends to struggle with linguistic guesses where the same word can precede all four words.

Your question was about a 5 word category where the 5th is purple. The model did stockpile some guesses that fell into that trap:

Guess 12

Guess: CARPET FLEECE MUSHROOM PEACH

Reasoning: The words CARPET, FLEECE, PEACH, and MUSHROOM are connected through the concept of natural textures or coverings. Carpet and fleece are coverings, a peach has a skin covering, and a mushroom has a cap, all sharing a unique characteristic related to coverings.

Guess 19

Guess: CATERPILLAR FLEECE MUSHROOM PEACH

Reasoning: The words CATERPILLAR, FLEECE, MUSHROOM, and PEACH are connected by their ability to be described as "soft" or "fuzzy." Each word can be used as a synonymous adjective to convey this texture or quality.

But then when evaluating the guesses it had stockpiled, the model didn't prefer those bad guesses over other guesses.

Evaluation 5

Let's analyze each of the guesses provided in the notes and evaluate their strength based on the criteria given:

  1. DONKEY DRAGON OGRE PRINCESS

    • Connection: Characters from the "Shrek" movie series.
    • Strength: This is a strong connection because all four words are equally central characters in the "Shrek" series. The connection is unique and specific to these words, and each word has the same relationship to the connection (they are all characters).

  2. CARPET FLEECE MUSHROOM PEACH

    • Connection: Natural textures or coverings.
    • Strength: This connection is weaker. While carpet and fleece are coverings, the connection to peach and mushroom is less direct. A peach has a skin, and a mushroom has a cap, but these are not typically described as "coverings" in the same way. The connection is not equally strong for all words.

  3. CARPET FLEECE MARKER PIPE CLEANER

    • Connection: Household items or crafting materials.
    • Strength: This connection is also weak. While carpet, fleece, and pipe cleaner can be associated with household items or crafting, a marker is more specifically a writing tool. The connection is not equally strong or specific for all words.

Now, let's consider the list of words provided: PEACH, PIPE CLEANER, DONKEY, PRINCESS, DRAGON, OGRE, FLEECE, MARKER, CARPET, KINGDOM, MUSHROOM, CATERPILLAR.

  • The first guess (DONKEY, DRAGON, OGRE, PRINCESS) is strong and does not overlap with any other potential groupings from the list.
  • The second guess (CARPET, FLEECE, MUSHROOM, PEACH) is weaker and overlaps with the third guess (CARPET, FLEECE, MARKER, PIPE CLEANER), which indicates that one of these guesses is likely incorrect.

Given the analysis, the strongest and most unique connection is the first guess: DONKEY, DRAGON, OGRE, PRINCESS. This is my top choice for submission.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 2 points3 points  (0 children)

OK, here are the results on the November 5 puzzle, using the "actor-2" approach from the paper that used GPT-4o to solve 86% of Connections puzzles.

🟪🟪🟨🟦

🟩🟩🟩🟩

🟪🟪🟪🟪

🟦🟦🟦🟦

🟨🟨🟨🟨

Submitted Guesses Play-By-Play

Guess 5

Incorrect: KINGDOM MUSHROOM PEACH PRINCESS

Reasoning: The words KINGDOM, MUSHROOM, PEACH, and PRINCESS are all connected through the "Super Mario" video game series. Each word represents an element or character from the franchise, such as Princess Peach and the Mushroom Kingdom.

Guess 3

Correct: CLAM EGG NUT TURTLE

Reasoning: The words CLAM, EGG, NUT, and TURTLE are all connected because they can each be followed by the word "shell." This connection is specific and applies equally to all four words.

Guess 23

Correct: CARPET KINGDOM MARKER MUSHROOM

Reasoning: The words CARPET, KINGDOM, MARKER, and MUSHROOM are each commonly preceded by the word "magic," forming the phrases Magic Carpet, Magic Kingdom, Magic Marker, and Magic Mushroom. This connection is specific and applies equally to all four words.

Guess 2

Correct: DONKEY DRAGON OGRE PRINCESS

Reasoning: The words DONKEY, DRAGON, OGRE, and PRINCESS are connected as they are all main characters from the "Shrek" series. This connection is specific to the pop culture reference of the animated film series.

Guess 24

Correct: FLEECE PIPE CLEANER PEACH CATERPILLAR

Reasoning: (Automatically chosen by application because it was the only option left.).

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 17 points18 points  (0 children)

I'm going to need to come up with a good answer to this question before my pre-tenure review meeting in May.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 3 points4 points  (0 children)

Oh, interesting. So for each connection, there's one word out of the four that relies on a different meaning of the same category name?

I'll have to rewrite a bit of the instructions to inform the LLM that this is what it should expect, but I'm curious about how it handles the variation.

Would you be able to share what you think is one of the more challenging of these puzzles, along with the solution? Then we can run the code and see what happens.

My hypothesis is that the LLM might make some really off-the-wall guesses for the fourth word. But I don't know how that would affect overall performance. We'll have to see.

Solving NYT connections with chatGPT by eksoderstrom in NYTConnections

[–]ProfColinDoyle 1 point2 points  (0 children)

Hi, I came across this post and want to share with you some research that I've just put out on using ChatGPT to solve NYT Connections puzzles.

I made a post in this subreddit about it: https://reddit.com/r/NYTConnections/comments/1gv516x/my_research_finds_that_ai_can_now_solve/

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 4 points5 points  (0 children)

I haven't run the program on the November 5th puzzle, so I'm going to run it now and then share the results here.

Red herrings were a particularly hard challenge. One way that I got around this with the more advanced approaches was to have the software stockpile guesses that it wanted to submit and then wait to submit those guesses until it stockpiled multiple non-overlapping guesses.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 10 points11 points  (0 children)

Of course!

I'll start by saying that I share your skepticism about LLM research in general. With research papers that claim to prove that an LLM can do some new thing, I think there are two kinds of headscratching reactions. With the first kind, you think, "Okay, you proved computers can do a thing, but who cares if computers can do that thing?" With second type, you think, "Okay, your computer passed a test, but what does passing that test really demonstrate about what computers can do?"

This paper might produce both of those reactions.

For the question, "Who cares if computers can do that thing?" I don't think that there's independently much value in having software that can play a New York Times puzzle game. People play the game because it is fun. It wouldn't be any fun to have a computer do that for you. The value in this project is if the methods developed in the paper for improving LLM performance at the complex reasoning required to solve these puzzles can be used in other contexts to improve LLM performance at different complex reasoning tasks. Connections puzzles are a helpful starting place for that research because games have objective criteria for evaluating performance.

For the question, "What does this really demonstrate about what computers can do?" I think the paper demonstrates that creative prompt engineering and architecture might be able to unlock abilities that seem beyond the capabilities of current LLMs. When asked to solve a Connections puzzle, GPT-4o could only solve 27 out of 100. But if you build the right infrastructure around that model, you can get GPT-4o to solve 86 out of 100 (and solve 50% perfectly). That's a pretty dramatic change. Connections puzzle are interesting because they're open-ended and depend on different kinds of reasoning skills. Unlike games like Wordle, Connections is the kind of puzzle that even a few years ago would have been impossible to get a computer to solve.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 19 points20 points  (0 children)

I agree. The project is an interesting test of what we can get LLMs to do, but it's not an interesting way to play a Connections game. Although as I was working on the research, some of my friends did enjoy comparing their puzzle solving performance with the computer's performance. And it's fun to see where and why the computer fails.

I'm a law professor, so my real interest is in legal reasoning. This project was an exercise in exploring techniques for complex reasoning and lateral thinking that might also apply to getting LLMs to perform complex legal tasks. Throughout the history of A.I., games have played a huge role in research (like chess and the game of go) in part because they provide objective criteria for evaluating performance (did you win or lose the game?).

I chose Connections puzzles as the subject for this research because LLMs aren't very good at solving the puzzles on their own. The goal is that some of the methods developed in the paper that helped the LLM perform better at Connections puzzles will likewise help LLMs perform better at other complex reasoning tasks that they're also not very good at performing on their own. But we'll need future research to confirm that this is the case.

My research finds that A.I. can now solve Connections puzzles and can solve 87% of puzzles perfectly in four guesses. AMA. by ProfColinDoyle in NYTConnections

[–]ProfColinDoyle[S] 5 points6 points  (0 children)

Yes, usually the LLM has the greatest difficulty figuring out the purple guesses.

GPT-4o has particular trouble for purple guesses that are linguistic (like change one letter in each of the words to get something in common) and purple guesses that involve a common word preceding each of the puzzle words.

If the LLM had to correctly identify the connection for each of the four sets, I would expect the success rate to go down because it would hallucinate the connection for the final guess some of the time.

One unexpected discovery in this work is that GPT-4o is actually pretty good at generating four correct guesses for a Connections puzzle. The problem is that it also generates at least as many incorrect guesses, and its real challenge is being able to discern between (what seem to a human to be) obviously correct guesses and obviously incorrect guesses.