Row 39161

Row ID: 39161 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 39161 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

For a long time now, I have been using a simple test for chatGPT, to determine if it has pulled its head out yet or not. To this day, it still consistently fails. For being a damn language model, I would think something like this should be an easy task -- it's something I believe a 2nd grade student could easily answer.

Here's what I ask: Give me a word where the 4th letter is a "Y".

I've used many different letters and letter positions for the prompt, I've over-simplified the prompt, and even very thoroughly explained, with examples, of what I want. Regardless of how I word the prompt it fails miserably, over and over and over, yet thinks it's correct. The types of responses I get are something like this:

Here's a word -- "Fairy". The fourth letter is a "Y"!

I then tell chatGPT that the word is incorrect, and to explain to me why. It will easily, consistently say something like "I apologize, I see where I went wrong. Fairy has an "R" in the 4th position and not a "Y". A correct answer would be "Mystery" -- the 4th letter is a "Y".

We then go round and round in circles, with repeated failures. Every time, chatGPT spits out an answer with such confidence, gets it wrong, and is easily able to explain why it's wrong. Yet it never learns from its mistakes and can never correctly answer the prompt. The closest I've been able to get is by asking it to provide this format, and asking it to repeat until it is succeasful:

Attempt 1 -Guess: Fairy -Number of letters: 5 -Letter in 4th position: "R" -Result: Failed

Attempt 2 -Guess: Puddy -Number of letters: 5 -Letter in 4th position: "D" -Result: Failed

. . .

Attempt 14 -Guess: Beryl -Number of letters: 5 -Letter in 4th position: "Y" -Result: Success

I've done this many times, over months or maybe even years of development, and still it spectacularly fails every time.

Something today that I found weird was, after reaching my frustration limit, I said something like "Okay forget all of that, let's start over with a simple request. Give me a 4 letter word". To my amazement, it says "Certainly! 'Holy' is a four letter word where the 4th letter is a Y!".... I then replied with a "wtf, I don't understand you. Give me five more" and it successfully, effortlessly, spit out 5 more 4-letter words that all ended in Y. So in some backwards-ass round-about way, it was able to come up with successful words, but not without a serious amount of effort, failed attempts, and a scrapping of all my prompts. Instead asking for just a 4 letter word with no other criteria, when it was finally like "Jk, here you go!"

Does anybody understand why it would be so terrible at such a basic prompt? It will provide words with the 3rd character a "Y", many guesses where the last character is a "Y", words that start and end with "Y", words of all different character lengths, but never a successful answer where the 4th letter is a "Y".... And again, it doesn't matter what character you ask for or what position of the word you want it, it just fails every time.

Also, does anybody have similar examples of easy prompts where chatGPT miserably fails? I know I've experienced unreliable math answers, and even inconsistent answers to the same math problem, but I don't expect it to be overly great at math (despite being a computer...). It's a language model after all, not a math wizard, so I can excuse that for now. But what else have you guys and gals seen and been shocked that chatGPT just simply cannot do correctly?

FieldValue
text For a long time now, I have been using a simple test for chatGPT, to determine if it has pulled its head out yet or not. To this day, it still consistently fails. For being a damn language model, I would think something like this should be an easy task -- it's something I believe a 2nd grade student could easily answer. Here's what I ask: Give me a word where the 4th letter is a "Y". I've used many different letters and letter positions for the prompt, I've over-simplified the prompt, and ev…
label r/chatgpt
dataType post
communityName r/ChatGPT
datetime 2024-05-22
username_encoded Z0FBQUFBQm5Lak1MV2FLZUR5UXAwanhRel9wNzlrWkVwck9Ja2c0bHdDemVBUkVEZGdpSUJzemdJMHVqU3g3RkpSa2Fjb0dNY0lmS3hNNTBabjFKaDYtUUhWSkY1dmYta1E9PQ==
url_encoded Z0FBQUFBQm5Lak9hWTllMmZJYnNxcG1KUlZmTGc0M1dCcU55VDlEdTNBdEhZZGJjTDRxQ2JkMHN4LVBnM0lKcTVWdTNvQXBZZ2U0TV9ld0hMV243Zy1PLVNrYVE1a0hiMFV0ZEZ4Y1JSWnNoamlmZ2s5MFdOcWlzel9naWI4dzVlSVBVbzhLaGo1UHZPUmhLbUd1cG44eUNudmhGNE5CYmxQOW91RFNucjNVZ2xBQVNiM19NZ3FsbERZaWd1ZThCTkRwbFN3VzBFVWlfMVY4MDhhNEhSMEswc1JsMU51ZHptZz09

Raw Record

{
  "text": "For a long time now, I have been using a simple test for chatGPT, to determine if it has pulled its head out yet or not. To this day, it still consistently fails. For being a damn language model, I would think something like this should be an easy task -- it's something I believe a 2nd grade student could easily answer. \n\nHere's what I ask:  Give me a word where the 4th letter is a \"Y\".\n\nI've used many different letters and letter positions for the prompt, I've over-simplified the prompt, and even very thoroughly explained, with examples, of what I want. Regardless of how I word the prompt it fails miserably, over and over and over, yet thinks it's correct. The types of responses I get are something like this:\n\nHere's a word -- \"Fairy\". The fourth letter is a \"Y\"!\n\nI then tell chatGPT that the word is incorrect, and to explain to me why. It will easily, consistently say something like \"I apologize, I see where I went wrong. Fairy has an \"R\" in the 4th position and not a \"Y\". A correct answer would be \"Mystery\" -- the 4th letter is a \"Y\". \n\nWe then go round and round in circles, with repeated failures. Every time, chatGPT spits out an answer with such confidence, gets it wrong, and is easily able to explain why it's wrong. Yet it never learns from its mistakes and can never correctly answer the prompt. The closest I've been able to get is by asking it to provide this format, and asking it to repeat until it is succeasful:\n\nAttempt 1\n-Guess: Fairy\n-Number of letters: 5\n-Letter in 4th position: \"R\"\n-Result: Failed\n\nAttempt 2\n-Guess: Puddy\n-Number of letters: 5\n-Letter in 4th position: \"D\" \n-Result: Failed\n\n.\n.\n.\n\n\nAttempt 14\n-Guess: Beryl\n-Number of letters: 5\n-Letter in 4th position: \"Y\"\n-Result: Success\n\nI've done this many times, over months or maybe even years of development, and still it spectacularly fails every time.\n\nSomething today that I found weird was, after reaching my frustration limit, I said something like \"Okay forget all of that, let's start over with a simple request. Give me a 4 letter word\". To my amazement, it says \"Certainly! 'Holy' is a four letter word where the 4th letter is a Y!\".... I then replied with a \"wtf, I don't understand you. Give me five more\" and it successfully, effortlessly, spit out 5 more 4-letter words that all ended in Y. So in some backwards-ass round-about way, it was able to come up with successful words, but not without a serious amount of effort, failed attempts, and a scrapping of all my prompts. Instead asking for just a 4 letter word with no other criteria, when it was finally like \"Jk, here you go!\"\n\nDoes anybody understand why it would be so terrible at such a basic prompt? It will provide words with the 3rd character a \"Y\", many guesses where the last character is a \"Y\", words that start and end with \"Y\", words of all different character lengths, but never a successful answer where the 4th letter is a \"Y\".... And again, it doesn't matter what character you ask for or what position of the word you want it, it just fails every time. \n\nAlso, does anybody have similar examples of easy prompts where chatGPT miserably fails? I know I've experienced unreliable math answers, and even inconsistent answers to the same math problem, but I don't expect it to be overly great at math (despite being a computer...). It's a language model after all, not a math wizard, so I can excuse that for now. But what else have you guys and gals seen and been shocked that chatGPT just simply cannot do correctly?\n",
  "label": "r/chatgpt",
  "dataType": "post",
  "communityName": "r/ChatGPT",
  "datetime": "2024-05-22",
  "username_encoded": "Z0FBQUFBQm5Lak1MV2FLZUR5UXAwanhRel9wNzlrWkVwck9Ja2c0bHdDemVBUkVEZGdpSUJzemdJMHVqU3g3RkpSa2Fjb0dNY0lmS3hNNTBabjFKaDYtUUhWSkY1dmYta1E9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9hWTllMmZJYnNxcG1KUlZmTGc0M1dCcU55VDlEdTNBdEhZZGJjTDRxQ2JkMHN4LVBnM0lKcTVWdTNvQXBZZ2U0TV9ld0hMV243Zy1PLVNrYVE1a0hiMFV0ZEZ4Y1JSWnNoamlmZ2s5MFdOcWlzel9naWI4dzVlSVBVbzhLaGo1UHZPUmhLbUd1cG44eUNudmhGNE5CYmxQOW91RFNucjNVZ2xBQVNiM19NZ3FsbERZaWd1ZThCTkRwbFN3VzBFVWlfMVY4MDhhNEhSMEswc1JsMU51ZHptZz09"
}
Explore Dataset Explore Row

Entry Information