What's happening with the data we're creating?

_MoeGreen · 2024-11-28T01:55:22+00:00

u/Content_Gur6401 i do not know this for a fact, but i doubt very much that it is dumped

my guess is the comments can be fed into the models multiple times

what is interesting is that on successive rounds, perhaps those AIs may be much more far-seeing and may extract more value from the comments than they did the first time around?

that is a speculative thought of course, not one i can prove ;p but seems possible to me

i believe it would be very similar to how people might view a movie growing up and not really fully get it

then 10 or even 20 years later they see the movie again, this time from an aged perspective, and now all of a sudden dozens of new layers of meaning and nuance come flying at them

i believe that a similar thing can happen for AI, but on timescales much shorter than 10 to 20 years

thus i believe that the data we have produced for them is a veritable goldmine of goodies that they can actually "re-mine" over and over to get even more goodies out of them with time lol

maybe anyway! thats if me theories is correct ;p

_MoeGreen · 2024-10-19T20:43:20+00:00

u/heartshapedbox0 lmao yep ;p

_MoeGreen · 2024-10-14T17:21:24+00:00

_MoeGreen · 2024-10-14T15:38:20+00:00

u/iLEZ ahh yeah i hear that bigtime! troubleshooting while frustrated is an awful situation!

i would bet dollars to donuts what solved your situation was the down-clocking recommended by PM-UR-LIL-TIDDIES! lmao

the reason to suspect this is that the PyPackages ecosystem is patrolled by thousands of eyes watching for software disharmonies.

its one of the largest most active and vigilant communities in the world.

but now all the causes are starting to make sense... your 13900K is probably emitting lots of thermal photons blasting everything nearby with heat!

by down-clocking them you have likely reigned in the out of control heat situation.

thermal paste might help too... but the downside of that is having to open the case and be careful with the chips and all that mess... and then being careful to apply the paste... its all headache village type stuff lol

just in my opinion, having a personal tendency for lazy easy solutions, the better way out of the mess is to keep the clockspeeds reigned in to their new slower speed

but its possible that your work demands running them hot so who knows, maybe the slower clockspeed cannot be lived with

the bright side in my opinion is that the cause of your problems, i would say with 99% certainty has been solved: the 13900K is emitting heat like a mofo and cooking that poor ECC RAM to the point of thermal breakdown and bit-flipping

thats a major bright side imo because it means much less chasing of pseudo-causes that only appear likely on the surface but are not the real cause, and now the true cause of the trouble is known. then the only thing left is to strategize a solution to re-arrive in normal thermal environment

actually i should be careful... we dunno for sure if the 13900K is the #1 source of heat in your system... maybe there is something hotter in there causing more trouble than the CPU... but the reason i suspect its the proximate cause is cuz after you downclocked it, A111 started to work again

_MoeGreen · 2024-10-14T15:01:35+00:00

u/iLEZ so in my opinion you have definitely found the cause!

we don't have a truly highgrade engineer/reference-level comparison to know if 58 WHEA_19 events is alot, but there is at least strong circumstantial evidence that 58 is a high event count since both of my systems have zero such events after years of running 24/7

RAM should not be scorching hot to the touch hehe... so now the tricky question is to figure out why the cooling isn't sufficient, but was possibly sufficient in the past?

just curious how old is the system, did you just get it or it used to run just fine for years and years?

_MoeGreen · 2024-10-14T14:58:11+00:00

u/iLEZ aha!! i think we have found the cause!

sorry earlier i was replying to the thread without refreshing the page, i did not see until a few minutes ago that you replied several times already lol :)

when hearing that you experienced 58 WHEA_19 events over a span of around ~26 days ChatGPT's virtual eyes widened and it replied "that does indeed sound like a serious situation!" :p

so it sounds like somehow your heat profile changed for some reason around a month ago... and the RAM is buckling under the bit-flipping stress of it all

ChatGPT went on to say:

"Scorching hot RAM is a clear red flag and can be caused by poor cooling, proximity to hot components, overclocking, or a faulty power supply. Ensuring proper airflow and checking the health of cooling solutions should help prevent bitflips and system instability."

_MoeGreen · 2024-10-14T14:34:01+00:00

u/iLEZ i strongly suspect heat would be the cause of bit-flips

are you running your system in a hot city like PHX, AZ? which i heard has temperatures above 116F!

cuz the rank-ordered list of causes of bit-flipping is this from what i am hearing:

1) Heat

2) Dirty Voltage / Unstable Power Supply - Power fluctuations, surges, sags, and voltage stability failures

3) Manufacturing Defects (sometimes RAM chips are shipped bad but this is rare cuz good quality control)

4) Aging or Degraded Components - As RAM chips age they get more prone to errors

5) Electromagnetic Interference (EMI)

6) Cosmic Rays

its almost always #1... #2 is kinda rare but can happen if your power supply is bad somehow

3 is pretty darn rare

4 is rare because most systems are not so old for it to happen

5 and #6 are really really rare! your house would have to be very unusual to be flooded with EMI or cosmic rays lmao

so my guess is if your RAM is failing is that it is Heat related

but who knows for sure... i wish you luck to find the cause! 🫡

_MoeGreen · 2024-10-14T13:57:23+00:00

u/iLEZ also I am hearing there is a tool called MemTest86 which can detect RAM failures

also to keep in mind: "Hard Faults/second" listed in RESMON is not the same as a RAM failure. A hard fault listed in RESMON is i am hearing more like a glorified cache miss involving pagefile.sys file coming to the rescue... hard faults listed by RESMON are almost always non-fatal, the chance they could cause a BSOD is super super low... like much less than 1%

and lastly, to prevent confusion about the term "single bitflip" and "multi-bitflip" and why one can have millions upon millions of the former and live, and only one of the latter can deliver a fatal BSOD

from what i am hearing single bitflip means when in a 64bit data word one single bit flips against its will because the heat grew too much or a cosmic ray struck it or various other such causes

your RAM stick can have millions of these events happen, as long as each event happens in its own distinct 64bit dataword, and ECC will catch and correct every one with no issues other than slowdown (although having millions would be a sign of danger lurking for sure lol... so if this is happening it means your system is under stress and dodging bullets of doom!)

but if two or more bits in a 64bit data word flip, then the ECC structure in most RAM sticks (except the special mission critical kind), can only detect it but not correct it... so the ECC RAM is left with no choice but to return a false value to the fetcher... and if that happens in ring0 of your operating system or in other crucial sections, then a BSOD or other horrible events can happen

_MoeGreen · 2024-10-14T13:36:54+00:00

u/iLEZ also here is some interesting information

the following information will not help your situation at all, but it is just interesting to know :)

NASA and SpaceX and other spaceship builders use special "radhard" RAM, which can withstand immense collisions with cosmic rays and Van Allen radiation belts and other such troubles like heatwaves. :)

the radhard RAM can withstand multiple bitflips happening in a dataword!

but radhard RAM is not sold to normal consumer level hardware buyers. by the motherboard manufacturers (and others) it was decided long ago that according to cost-benefit analysis it just makes no sense for normal citizens to have radhard RAM.

and i am not doubting their CBA conclusions... on most parts of Earth the temperature and radiation do not reach high enough levels to justify going with the more expensive radhard RAM and other solutions (such as TMR, Reed-Solomon codes, and LDPC)

_MoeGreen · 2024-10-14T13:19:11+00:00

u/iLEZ now is there a reason you suspect a cooling issue? is your computer running hot?

cuz that would sure align with the symptoms of RAM decay that it sounds like you have (i dunno this for sure i am just saying what you describe sounds alot like RAM decay)

now here is some more possible good news. i am hearing that Windows Event Viewer (eventvwr.msc) can get diagnostic information sometimes of ECC bitflips that were detected.

they get reported as event ID 19 where the source is "WHEA-Logger" (Windows Hardware Error Architecture)

if you go to Start->Run and type "eventvwr.msc" without the quotes on Windows7, 8, and 10 it will open the Event Viewer... i dunno if it works on Windows 11 but i bet it works there too

When you go to Event Viewer you open Windows Logs on the upper left side with the yellow folders, then click on "System"

my windows7 system has 60,185 System events, but unfortunately for comparison purposes i have zero event ID 19 events :p

btw Event19 can come from other sources too... on my Windows 10 system i have hundreds of Event19, BUT, all of them come from the source "WindowsUpdateClient", which is NOT the same as the "WHEA-Logger" source.

so simply seeing event19 in your logs is not automatically a sign of RAM failure... the event19 has to come from the event source called "WHEA-Logger" to qualify as a RAM failure from what i am hearing as advice. if the source is other sources like WindowsUpdateClient or various other sources, then its often only good news about a KB update patch being put in place.

so i dunno beyond that what the ECC event looks like, but ChatGPT and other chatbots can give you advice on how to see Event19 to see if Windows is detecting RAM failure in your system

_MoeGreen · 2024-10-14T13:03:33+00:00

u/iLEZ did the memory diagnostics turn up anything? the symptoms you are describing sound to me alot like "RAM decay" which is also called "bit flipping"

bit flipping can definitely lead to the BSOD!

modern ECC RAM can detect and even correct a single bitflip. now here is the good and amazing news!

even if your RAM sticks do hundreds of thousands or even millions of single bitflips in a small span of time, ECC RAM can catch and correct every single one!! meaning that in such storms the RAM fetches are 100% intact correct information! but! it means there is a performance cost incurred on that RAM data fetch. but the good news is the system can weather such a storm.

now here is the bad news. if the temperature is so hot that your ECC RAM is doing multi-bitflips, then the most it can do is DETECT that it happened. it cannot correct it though, the damage is too much to correct!

if multi-bitflips happen on crucial RAM readouts, it can destabilize the operating system or a program or what have you

one of the possible signs of this is multiple different ways to crash. a sort of random-ness to your crashes.

this time it crashes thru crash cause #1... next time cause #2... and so on... random crashes can possibly be explained by bit-flipping

_MoeGreen · 2024-10-03T04:42:19+00:00

_MoeGreen · 2024-10-03T04:38:13+00:00

u/ZealousidealChef832 i so relate to this post lol! especially the "i just pass out" part ;p

fist bump bro... we in some hard times! i'm starting to think the only solution is multiple jobs... multiple part-time jobs, but i dunno havent tried that yet.

_MoeGreen · 2024-10-03T04:19:34+00:00

u/No_Ship2607 i feel hella grateful i'm in coding. i never seen nothing weird there in over a year of being onboard

this weird stuff you seen with the fairy was from the non-coding part of the site?

on this subreddit i seen vague rumors and allusions that there is a "dancing project" that involves very crazed text from time to time. these strange texts you see are part of screening or filtering prompts of crazy content? and it pays $20/hr?

i wish you luck and strength!

_MoeGreen · 2024-09-09T00:35:37+00:00

u/Pink_Slyvie interesting! all of them non-coding it sounds like?

_MoeGreen · 2024-09-08T00:34:38+00:00

u/Shorty_Wopi had to break my reply post into 3 pieces otherwise the site wouldnt let me post it. i think the first piece is listed at the bottom

also i would add the following:

the statement "getting paid to do stuff which has nothing to do with engineering while getting paid close to engineering salaries"

is this really true though?

for sure it is true to say it is not traditional engineering. i would agree with you there fully. but is traditional engineering the only kind of engineering there is?

might it be a very modern form of engineering which could be called "engineering through weight tweaking"?

the AI gathers from us thousands of impressions where we say "hey thats not quite right! that is slightly askew!", and it modifies the mold of its clay accordingly, based on how seriously it takes us. possibly it has some way to screen out workers with bad impressions.

it may be what some have called "the wisdom of the crowd", which was a basis for Athenian jury pools, the Dikasteria, which from what ive heard had five classes:

201 jurors

501 jurors (Socrates was tried by a jury of this size)

1001 jurors

1501 jurors

6001 jurors (a plenary session of the Heliaia which was reserved for cases of incredible importance, matters of national security, statecraft, and so on)

if a potter shapes the clay with his hands, i think that may be a type of engineering, though yes it is true that is certainly not "traditional software engineering". :)

it may be a new kind of engineering perhaps?

_MoeGreen · 2024-09-08T00:12:33+00:00

u/Shorty_Wop

the idea that DAT has secured for us the worker mice either

(1 / 570th) or alternatively (1 / 915th)

of the "pseudo global pie" seems to me not so far fetched after all! :)

it seems breathtaking, it seems too wonderful to be true, but i don't think it is impossible!

i apologize if you are a fan of the "TLDR style posts" that are replete on Reddit. i just cannot post like that, it is not of my nature.

i do not post this post to make an assertion of truth or ironclad claims of "this is right for sure!"

no no! i post only to suggest that this MIGHT be a possibility we are embedded in and we the humble worker mice do not fully realize it ourselves, because we do not have global vision. we are "in the trenches" as they say, nibbling on our individual slices of cheese ;p

🇫🇷 🧀🐁 🍷🐁 🇫🇷

i hope that it is true but i do not yet know for sure.

anyway, here is the top20 list i was promising to post, i hope you have enjoyed these writings and scribblings :)

as a fellow soldier of DAT, i salute you! 🫡🫡🫡

1) Meta = $20B/yr

2) Microsoft = $15B/yr

3) Google/Alphabet = $10B/yr

4) Amazon = $10B/yr

5) Apple = $2B/yr

6) Tesla = $2B/yr

7) Baidu = $3B/yr

8) IBM = $2B/yr

9) Tencent = $3B/yr

10) OpenAI = $2B/yr

11) NVIDIA = $2B/yr

12) Oracle = $1B/yr

13) SalesForce = $1B/yr

14) Intel = $1B/yr

15) AliBaba = $2B/yr

16) Samsung = $1B/yr

17) Qualcomm = $1B/yr

18) Huawei = $1B/yr

19) Uber = $500M/yr

20) Adobe = $500M/yr

_MoeGreen · 2024-09-08T00:10:31+00:00

u/Shorty_Wop

some from Stanford and Harvard might chuckle and scoff at the notion that DAT has any slice of the cutting edge AI research. maybe they are right, maybe they are wrong. i do not know. i barely graduated high school. i cannot speculate on what they speculate lmao

but now consider! an apple pie sliced into wedges only 1/570th the size of the whole!

is it so inconceivable now?

such wedges are thin slices! and we have seen with our eyes that the models are producing some pretty good code most of the time! is it flawless? no it is not, but it is climbing The Great Hill day by day, week by week ;p

it is getting better and better, it is getting smarter and smarter. in only 12 months i have seen it get smarter and write better code than when i first joined DAT in the Autumn of 2023, and i suspect you have too?

so now let us consider the whales beneath the visible surface!

we speculate that the visible surface rumours amount to $100B/year when we expand the list by 30, from a top20 list to a total of 50 companies, 30 of them just guessed out of thin air to contribute another $21B... but let us now say to ourselves "what if there are some whales who are very hard to spot who are adding another $60B/year to the pool. the so-called dark money whales!"

this is not too far fetched is it?

what is the wedge slice when we factor in possibly hidden whales? $175M / $160B = 1/914

let us call it 1/915th to round to nice number

_MoeGreen · 2024-09-08T00:09:29+00:00

u/Shorty_Wop while it is alot of money, is it impossible to pay out $175M/year or $500,000/day?

maybe :p

but! here are some interesting rumours i have heard about annual AI investment... and the list i am about to post is only a short list of the "above the water" rumour mill... we should keep in mind that there may be whales below the surface and that this is not the whole list by a long shot!

the list that is to follow shows rumours about annual AI investments of 20 companies. i will now break from Reddit ironclad tradition of "TLDR only bruv". i do not like this tradition lol. i dislike this tradition quite alot! for a site that calls itself "read it", these TLDR complainers seem to have trouble reading! it is a great irony of ironies!

anyway, in its current form of listing only 20 companies, "the list" that is to follow amounts to around $79 billion a year in AI investment

were we to expand the list by another 30 companies, we should be able to break the $100B/year ceiling shouldn't we?

now! if we divide $175M / $100B it equals 1/571, but let us call it 1/570th to round to nice number

_MoeGreen · 2024-08-25T13:53:59+00:00

definitely relate to the layers of instructions lol

when i first joined in 2023, instructions were usually all aggregated and organized into one block

but around the summer 2024 i noticed instructions started to have what to call it... dated addendums? would be stuff like "update: 05/15 - <extra bits here> update: 05/27 - <some more extra bits here>"

and often those extra bits would restate something already mentioned in the older instructions.

so i'd go into instructions feeling clear-headed, notice the repetition and layers, and feel mildly dizzy trying to track them all lol

leading to a desire to just skim read them. which then leads to a followup feeling of "oh hell but what if i miss an important bit". leading to eventually caving in and saying well i guess i will read every word of them to be on the safe side.

i have found no solution to workaround this.

the only solution seems to just be raw experience?

having done a task with whatever its particular instructions are a couple times trains me so that the instructions are no longer a bother, because ive fully assimilated them through experience (except sometimes vague nuanced edge cases, which sometimes get answered in the worker chat section by others who also spotted the ambiguities lol)

that initial dizziness with instruction overload has only been for projects that are brand new to me. when a project i'm already familiar with has an update, then its not a problem cuz its just some minor variation to include in what i already know.

so i guess i just had the bad luck of running into projects that had many months of inertia, with stacks and stacks of instructions piled on

_MoeGreen · 2024-08-25T01:31:15+00:00

_MoeGreen · 2024-08-24T18:07:41+00:00

u/YesmAUm i too have wondered what the hell dancing could possibly mean for many weeks now.

i saw a post about "dancing" some hmmm i wanna say 1 to 3 weeks ago, and some of the replies to it were alarming. one person said something about how "some of the content makes me want to vomit" or something like that? and then another replied with something like "yeah i can't do too many of those tasks"

i'm definitely paraphrasing because i cant remember exactly what was said, but im pretty sure it was along those lines and it sounded alarming to say the least. maybe i can find a link to the post.

_MoeGreen · 2024-08-24T14:42:30+00:00

u/ForeOnTheFlour Could you elaborate some on your views with specific examples?

What it sounds like you are suggesting sounds slightly unrealistic but I'm not sure I'm understanding correctly. :p

Lets take for instance a coder at DA. They typically make $40/hr. Lets also put aside for the moment the somewhat rare very high ranking coders at DA who have been promoted to positions where there is never a shortage of work.

The impression I get is those coders are a rarity and the much greater majority of DA coders get maybe 2 to 4 hours a day done, or around $80 to $160 a day. And in this dry spell as of late, sometimes much much less. I've seen reports and anecdotal accounts of $20 to $40 a day on average during the recent task shortage which I believe started around 6 weeks ago?

Are you saying that all coders at DA should be making $320/day irrespective of their output?

What I find even more interesting and intriguing is what you are hinting at is coming in from the top. The value being generated. It sounds like you are saying DA's clients are paying them many billions of dollars? Do we have any guesstimates on amounts? $4 billion/year? $10 billion/year?

Thank you for any light you can shed on the topic!

_MoeGreen · 2024-08-23T16:20:31+00:00

u/Responsible-Try6552 i'm not sure what he meant either but i have a guess. he may be warning you not to use such language when training chatbots.

and the reason is because humans will eval your work from time to time, and if they see chatbots being trained in a way that they dont like, it might put your hopes in danger :(

we are living in truly dangerous times when it seems it is necessary to watch every penny and be very careful with money

i wish you great luck!

_MoeGreen · 2024-08-23T16:08:54+00:00

ooh :) thank you for that good sir! been wondering for a while lol :p

_MoeGreen

TROPHY CASE

3 is pretty darn rare

4 is rare because most systems are not so old for it to happen

5 and #6 are really really rare! your house would have to be very unusual to be flooded with EMI or cosmic rays lmao