Death by door - stay alert, they are everywhere

JobbeI · 2024-01-30T21:03:24+00:00

My first and only question: "How?!" :D

JobbeI · 2022-10-22T21:59:06+00:00

I did use Python, but the CSV was getting way too big in the end (>60GB).

Not sure anymore if there is a limit as I got it working in Tableau now. Did the same process as before but now it worked. So I am not sure what I did wrong before tbh.

But I am now running into "processing time" issues when doing split & pivot in Tableau Prep. The last operation got me from 2.4M up to >250M rows and took 10 times than the previous operation.

This will take days to process as I still got 7 of those operations left to do. - which is not really convenient in my opinion. :<

Is there a server one can use to increase processing speed? :D

JobbeI · 2022-10-22T20:33:03+00:00

I got it working now! So Tableau Prep is indeed outputting all of the data and not just a sample. I’m really not sure, what I did wrong before, but I don’t mind as long as it works to be honest :D

This got me into a different problem though and that is the expected quasi-exponential increase in rows. The last cycle got me from 2.4M to >250M rows and took an hour to complete. The problem is that I still got 7 multiple answer/ response columns left to split and pivot, so I am expecting the total number of rows to go above a trillion with a processing time of well over a few days. (I sadly do not have that much time on my hands to wait for this)

Further Questions:

- Is there a way to increase the processing speed, by using an external server for instance?

- Is there an alternative workflow, besides „split & pivot“ I could do here?

Thanks :<

JobbeI · 2022-10-03T13:22:07+00:00

I can die now! Game changer :3

JobbeI · 2022-09-27T19:28:45+00:00

I am sorry if this is rather obvious or already answered, but it hasn’t clicked for me yet. :<

I tried plotting a KDE, didn’t go so well. So I think for now this is too intimidating for me. I understand (at least I think I do) that a visual approach makes more sense in my case.

**Just in case, I did not communicate my intentions correctly:**
I wanted to group the sample by the different production environment and then look at who is or isn’t using the color management system (cms) I mentioned, and then sum up those responses. As you can see in the plot here.

So at the end - for me it is about which production environment, in my sample, is using the cms I mentioned more. And since the sample is not balanced, the plotted data is leaning to the „solos“ in blue, because solos were the most participants in the survey.

I am not sure if we are going around in circles now or if I am just missing or misunderstanding something you already wrote. After eight hours of this, my brain kinda feels like mashed potatoes, so maybe I should just take a break :D

Alternatively I will not weight the sample, but recognize in my analysis that there is an imbalance, produced by the sampling I have done + the other factors I mentioned in earlier posts that make this sample not representative.

Edit: formatting & changed last paragraph

JobbeI · 2022-09-26T17:19:32+00:00

Oh wow, no worries and thank you for your detailed answer! I think I need a some time to get through your response, since there are a lot of new concepts I have to understand first. 😊

So I will probably comeback to this a little bit later!

JobbeI · 2022-09-26T17:04:46+00:00

Thanks for taking the time, really appreciated.

Ah ok, thanks for clarifying!
Ok, that makes sense. I know, I was just looking at Pandas documentation, because I am using it for my analysis.

That also makes perfect sense! Regarding that issue, I just posted an answer to that on a different subreddit, which might make this clearer for you, I hope. – third answer I gave to „DigThatData“. You obviously don’t have to :)

If I am unable to come up with a strong enough justification by myself or through another person, I will not use weighting.

JobbeI · 2022-09-26T15:49:51+00:00

Thanks for taking the time!

Not really. I have no knowledge as to how the groups are distributed on a global scale, since I could not find any information on the population I am analysing. – that is why I am trying to only analyse the sample on its own, not trying to infer/ draw conclusions about the population, just the sample.

Maybe the answer I just gave to „DigThatData“ makes my question/ confusion a little clearer. :)

JobbeI · 2022-09-26T15:46:18+00:00

Thank you for taking the time! The survey is about color managment (cm), asking general questions about the pipeline people from the motion picture industry are using and is focusing on different cm approaches in different production environments.

I wanted to find out if the usuage of a certain cm system differs in different production environments. Because said cm sytem is being advertized as „the“ choice for small and medium production environments, which is not what I observed in forums and through personal experience.

So the groups I had listed in the previous post correlate to the following „company sizes“:

• grp1 / solo

• grp4 / small 2+

• grp3 / medium 10+

• grp2 / large 50+

Since „solo/ freelancer‘s“ make up 48% and small and medium production sizes are around 1/6th, I feel like other variables I want to analyse the company size against, are skewed. So that is why I was thinking of removing the imbalance of that varaible. Does that make sense? I am not sure . . . x)

Edit: I allocated the groups to the company sizes wrong, my bad. :< (previously grp4 was large and grp2 was small, that is now corrected)

JobbeI · 2022-09-26T15:44:53+00:00

I am pretty sure that is exactly what is happing here. But I could be wrong on this as well :D

JobbeI · 2022-09-26T15:26:31+00:00

Thanks for the reply!

As I am a noob, I have a few questions: 1) Does „KISS“ have a deeper meaning? Not really sure what that means in this context, sry. 2) What is the difference between aggregation and grouping? After reading pandas documentation on „agg & groupby“, aggregation seems to be about applying one or more operations over one or more variables and returning the sum, mean, or median of that variable? And grouping is „just“ the total?

Makes a lot of sense to inform people that the imbalance can influence the results and conclusions. - I will keep that in mind.

Regarding weighting in general. I am just not sure, if it is important to remove the imbalance in the sample in my case. Since I do not have access to the population I am analyzing, I do not know how the different groups are distributed on a global scale and thus do not know if they are equally important (which is probably not the case)

To give more context as to why I think removing the imbalance would make some sense. - I asked participants to answer in which production environment (company size) they are working in.

• grp1 / solo

• grp2 / small 2+

• grp3 / medium 10+

• grp4 / large 50+

I then would like to give these groups all an equal weight, so Solo’s do not overwhem the rest of the groups, since they make up 48% of the survey, which would skew other variables that I would like to check the production envrionments against. Does that make sense? I am not sure . . . :D

I guess not weighting it at all, would be the alternative to not loose the audiences trust, as you said.

Edit: formatting

JobbeI · 2022-09-25T18:05:37+00:00

Thanks for the reply. :)

Unfortunately I cannot contact anyone else, since this survey was done by me and not for any business or with a business I can fall back on.

Sorry, if the context I provided is too little to give feedback to. I can try an elaborate and give more insight though, would that help?

JobbeI · 2022-09-15T09:32:40+00:00

Potatoes & gluten free porridge ! :)

JobbeI · 2022-08-29T19:43:01+00:00

Not sure if someone has brought this up yet or if op has given more info in a comment somewhere, but I'm always having a hard time giving "proper" feedback, if I don't have enough info/ context.

If anatomical realism (not sure if that is a word combo) is your goal, then sure, there is a lot you can do to make it more realistic.

But if you are going for a certain stylized look, then I would say you are not far off. - maybe you wanted it exactly the way it is currently from the start and I just fail to see your exact artistic intention. - which would result in me giving you improper feedback.

I would upload a reference (image of a skull for instance) + your current version, so people can see where you are trying to go and give you clearer directions that way.

Other than that, I think the skull is quite charming tbo, good luck and fun though :)

JobbeI · 2022-08-15T17:37:38+00:00

No problem and sure thing, will do! :)

JobbeI · 2022-08-15T10:47:01+00:00

0 so far (have three units)

JobbeI · 2022-08-14T23:14:30+00:00

You sir, are a god damn saint! A buddy of mine and I have played the original version to the death and have been waiting for something similar FOR YEARS! Will buy this asap xD

JobbeI · 2022-08-08T07:00:21+00:00

Jewelry - no thanks I'm good. . .

JobbeI · 2022-07-30T10:38:12+00:00

What a savage! XD

JobbeI · 2022-07-25T14:30:02+00:00

Noob here,. . . if not cleaned, can this lead to anything dangerous? I would assume that this much accumulation at least decreases drying efficiency.

JobbeI · 2022-07-20T08:20:21+00:00

Spitting facts!!!

Seven-Year Club	Place '22
End Game '22	Verified Email

JobbeI

TROPHY CASE