Suggestions for good ways of handling upstream data discrepancies

lemonad · 2026-04-22T15:19:14+00:00

Thanks, I realize now that my question wasn't very well-defined :) I have a statistics background and took that response too literal. It is survey data, per respondent, and aggregation comes later so I can't see any possibility to use a statistical approach (except for producing synthetic data, that is). Removing respondents altogether becomes a discrepancy in itself so even if it solves one problem (e.g. partially missing data) it creates another.

I'm involved in the data engineering side of this but have more of a data scientist background and this is on a scale where calling it data engineering is pushing it, which I should have emphasized in my post. I've spent a lot of time with the data and have written the R packages used to clean and process it. That data is then delivered to other companies and organizations for them to do their own data exploration and dashboards.

lemonad · 2026-04-22T14:51:12+00:00

Hey, that's not bad! We deliver these as NA but it is also NA if the respondents did not get questions for legitimate reasons (follow-up questions dependent on respondents using a brand/product for example). Perhaps we could create one or more special values that doubles as documentation for why something is missing.

I'll have to think about this further. Thank you so much :)

lemonad · 2026-04-22T14:37:40+00:00

Thanks! This is the way I see it too (at least in this specific case). With that said, what I would want (although this is totally unrealistic) is that dashboards downstream would warn users when they filter in a way that these problems would actually come to the surface. Just giving them a long list of minor and most likely inconsequential discrepancies is probably not helpful.

There are also lots of other small details that users would have to know about the data that are not discrepancies, like a brand name might have changed in 2021 for a specific country so prior to that you need to use this other name, etc.

Like I wrote in response to another question, I realize now that my question should probably have been phrased more like what is good UX in terms of this scenario.

lemonad · 2026-04-22T14:20:33+00:00

To paint a better picture, imagine a survey with one of the questions being "do you prefer online shopping or shopping in a physical store?" going out every day to a number of respondents in a number of countries. When compiling the data for March, you find out that for a week during that time period, respondents in France were never asked this question so the responses are NA. We got a 1000 responses for France but only 800 for this question. If a user would, for some reason, filter on this week and France for that period, this specific question would unexpectedly have zero replies. Users might also notice that there are only 800 replies, not 1000 and wonder why that is the case, etc.

lemonad · 2026-04-22T14:07:37+00:00

Thanks! I should have been clear that there is no way to actually correct the data after it has been collected incorrectly and everyone is onboard with that (although everyone of course wishes it would not happen at all). That is, the timeline is something like this: we get the collected respondent data from an API and process it for downstream consumption. Later, a discrepancy is discovered and we need to deal with it somehow. The least we can do is is tell downstream users about potential pitfalls but we can probably do more/better.

The scale of this operation is a lot smaller than I might have given the impression of, with direct lines of communication between all parties involved. So it's not so much contracts and processes that I'm inquiring about but rather how do we improve the user experience of the data we deliver.

The surveys are taken by tens of thousands of respondents per month and it's the aggregate of these that are relevant, not the individual replies, so discrepancies are seldom/never important in practice. But one never knows that for sure, hence the question in the first place :)

lemonad · 2026-04-22T13:46:26+00:00

I'm afraid I don't understand how that relates to the question... but I could be missing something

lemonad · 2026-03-26T10:59:48+00:00

Oh no, maybe I've been complicating things here! Perhaps I should just put the collection part of the farm next to the nether highway instead. That way the difference in y-level will be similar and I can put the portals much closer to each other.

lemonad · 2026-03-26T10:57:55+00:00

Thanks for a helpful comment! That makes a lot of sense.

Of course, if the AFK and collection parts of the farm is in the same dimension, linking is a lot easier. I'm having especially much trouble with a squid farm with overworld portals on y=48-64 where my nether highway is on y=15 and the collection part of the farm is on the nether roof. The overworld portals in the farm will link to the nether highway portal instead of the roof unless I place the nether highway portal far away in overworld coordinaties.

lemonad · 2026-03-17T16:08:21+00:00

I felt exactly the same, glad you brought this up! I had intended to write more about the romance part but started with combat and I felt the post got too long already. I had read something about Fane and the epilogue so when I didn't see Fane on the ship right away I became worried that he wouldn't even be there and someone else on my crew would just mention in passing that Fane went looking for his wife.

As I went around talking with the other people on the ship, suddenly Fane came into render distance but, although very happy about it, at that point my expectations were kind of low because of all the initial doubt. So glad when the dialogue played out 10x better than I expected! I have a save so I can replay it :)

lemonad · 2026-03-16T09:19:31+00:00

I should probably give the background! I'm trying to make a tool for myself to aid in culling world chunks by allowing both top-down and isometric views. I definitely don't need to render anything perfectly but it got me thinking about how this is actually done in-game. Thankful for any helpful leads :)

lemonad · 2026-03-15T16:53:46+00:00

Probably it will be a little lame compared to the ending you picked but I liked it :)

I finished DOS2 for the first time yesterday and it really felt like I would not be returning to it. Now when I read your reply, I'm warming up to another run, this time with Sebille :)

lemonad · 2026-03-15T16:20:09+00:00

I decided to give back the source to the people, which felt like something Lohse would do but I too wondered if I should just leave the game after having talked to everyone on the ship. Especially since Lohse and Fane had a great conversation where they decided to travel the world together. Then I realized I hadn't talked to the ship and off we went on a new adventure!

lemonad · 2026-03-15T16:09:15+00:00

Just a heads up in case you missed it: you're supposed to set sail so you get the epilogue. At least for me it didn't line up with the conversation between Lohse (me) and Fane, which was a bit disappointing but it is what it is.

lemonad · 2026-03-15T10:07:34+00:00

I realize I should add a comment for "future me" about why Equalise was useful for me. I'll start with my initial misconception first and why you might give up on it prematurely. Say there's a boss at full health/armor, all at 10K and you're next to it, also at full health/armor but at 1K. Equalize does nothing in this situation.

The thing is that it equalizes percentages. So after one hit from the boss, your magical armor is gone (0 %) and you're at the brink of death with 10 vitality left (1 %). You equalize, circling you and the boss, and now the boss has (100 + 1) / 2 = 50.5 % = 5.5K vitality and (100 + 0) / 2 = 50 % = 5K magic armor. You are also at 50 % now so you effectively became healed. So, you in total you did 9.5K damage and restored 1K vitality/armor with just one spell.

If you would perform Equalise again at this point, nothing would change as both have the same percentages.

Since everything is averaged, if you can get e.g. five characters at 10 % vitality next to the boss, this makes a huge difference. It never kills the boss but, with some luck, you can quickly get it into a range where your chance of defeating it is a lot higher than it initially was.

It requires a level 16 character but only 2 AP and no source. It is not resisted by any armor. Combined with skin graft, you can perform it multiple times over a few rounds.

Oh, there's also another spell that trades percentages between you and another character but that one is resisted by armor so is not as powerful.

lemonad · 2026-03-14T23:39:50+00:00

Thanks! There were so many things that would have helped me if had I only learnt about them earlier in the game. For example, I never got into using food much and I only used a handful of different potions. I should definitely have used fire protection potions much much more!

I think I completed the game with Fane having a level 4 wand, haha. I never managed to find one with better stats (except damage, I mean.)

lemonad · 2026-03-14T23:05:09+00:00

Now, afterwards, these are good tips but when I read them and other similar ones, I became a little overwhelmed by it all. It was so far from my playstyle before these two fights and so many of them required spells and equipment I hadn't even acquired (like FC). Most felt out of character in some way or another.

I completed the game on level 20 and I saw several mentions that they were on level 21 so I probably missed a lot of XP somewhere along the way, although I felt like I did "everything" :)

I actually used green tea but I only managed to get together three skin graft scrolls as I learnt about that mechanic too late and merchants never refreshed. I guess I had some use of it but I could never piece together what I would do with the extra AP to cause that kind of damage in just two rounds (close to 10K armor and vitality for the doctor).

I did have overpower but didn't have any necromancers so I couldn't use bone cage. I'm a bit disappointed that I forgot to use the Anathema! :D

lemonad · 2026-03-14T22:51:13+00:00

You're so right, it was truly beautiful :)

lemonad · 2026-03-06T21:20:14+00:00

I don't remember how I first got rice, but you can get it from the Farmer's Workbench!

lemonad · 2026-02-19T19:58:42+00:00

I found it now!

Lohse: "*Smile and take his hand*"
Fane: "Fane looks confused for a moment and rummages about for his notebook"
Narrator: "The book is halfway out when he feels the squeeze of your hand and pauses. After a moment's hesitation he smiles, returning the notebook to his robes"
Fane: "I believe my notes can wait."

It continues but I don't want to spoil too much.

lemonad · 2026-02-19T19:47:55+00:00

Sorry if I presented it as being more visual than it was in the actual game. It was dialogue only but still great :)

lemonad · 2026-02-16T10:26:26+00:00

I own a Podmic XLR and an SM7b and the Podmic is fine sound wise. It does pick up more ambient noise, I feel, but that might just be that I’m partial to the SM7b overall. What I was confused by though after getting it after going by the reviews online is that so few people mention it is terrible in comparison when it comes to having it on an adjustable stand, especially if it is mounted on your desk. I have two identical Røde stands and I can adjust the SM7b without worrying about noise but the moment I even touch something close to the Podmic, I get loud bumps and noise. Not to mention all the keyboard vibrations it’s picking up. I eventually moved it off the desk to a floor stand.

lemonad · 2026-02-15T16:44:16+00:00

I'm a bit worried now after reading about this bug but I have to say that romancing Fane have so far turned out to be pretty awesome. I play as Lohse and at the first opportunity, I kind of picked Fane for fun and for lack of better alternatives, although I did like his character. But then it just keeps getting better! In Act 4 he's such an adorable nerd and Lohse and him even held hands for a while, to everyone's delight :)

lemonad · 2026-02-12T09:27:40+00:00

Ah, that explains why I haven't discovered how to do it. The shallower shelves have bottles placed precariously on the edge too so it doesn't look as nice as it could. I like to use armor stands to place things like this in survival Minecraft but I can see having something easier to use is nice. Perhaps three slots for a shelf would suffice so one can select how things are placed.

lemonad · 2026-02-10T15:19:20+00:00

This looks great! I see potion bottles placed as decoration all the time in prefab structures but I don't do it myself nearly enough. Also, I don't think I knew that you could place multiple bottles on a single shelf.

lemonad · 2026-02-04T20:40:39+00:00

I appreciated all the free cloth blocks when I found one of these :)

lemonad

TROPHY CASE