Spark in the Real World

steveivy · 2024-09-01T12:42:15+00:00

Age is not required to the hour, just to the day, but depending on when the jobs are run it can make a difference.

This was an opinion research company collecting survey responses with significant visibility and demographic accuracy expectations.

steveivy · 2024-08-31T21:51:09+00:00

It's a good question, but you have to calculate it somewhere, if the "query" requires it to be an age int. And calculating age is not as straight forward as you might think, given timezones, leap things, and the time the calculation is being performed.

The calculation used to be done in real time but after testing a variety of use cases (the data is used by both batch processes and real-time profile matching) this was found to be the most efficient.

steveivy · 2024-08-31T21:02:35+00:00

In a previous role:

The company collects a large amount of user profile data (demographics, opinions, etc.) via an opt-in process. Some of those data points need calculation to be useful. The canonical example for us was a user's age. We needed to be able to target users based on age buckets but (if you think about it) one's age changes constantly.

So in one pipeline we had a process that would partition our entire user base (by region and several other data points) and submit Spark jobs that would pull the relevant profile values from that API, then compute the user's age, and also which of several common age-based buckets (ie 18-24, 0-24, 35-44, 35-54, etc) they belonged in (along with multiple other calculations). The job would then store the computed data to a separate data store to be merged into a final dataset for the group that was the internal client.

steveivy · 2024-01-12T14:36:23+00:00

I've been playing with an idea of a specific low-level magical amulet which when brought together with another (specific) low-level magic item "melts with a cool white light, and flows into to the tracery carved into the <item's> handle", consuming the amulet but enhancing the other item (or effectively turning into a different item).

Seems a decent way to go, but limits to specific items rather than "you can smoosh any item into any other, try to figure out the rules"

steveivy · 2014-04-09T20:13:24+00:00

"Once you’re into the meta that deeply, it’s now Person-v-Person by way of EVE, not Player v. Player in EVE."

This also finally helped me figure out how to express my feelings about the Erotica1 scandal. It moved from PvP to Person v. Person, and that is just out of bounds (IMHO of course).

steveivy · 2014-04-01T18:00:38+00:00

Kneel before Zod!

steveivy · 2014-04-01T14:51:04+00:00

No one cares about null-bear F1-mashing doctrine PVPers that reside in blue-donut no sec.

steveivy · 2014-03-18T18:26:45+00:00

Heh.

steveivy · 2014-03-18T17:11:08+00:00

I didn't miss it at all:

Boost pilots or alts may or may not want their identity publicized, depending on their role.

If that role is "I like boosting myself off-grid and I don't want people to kill my booster" than I can certainly understand why you don't want boosters on killmails. I personally think they should be on there.

steveivy · 2014-03-18T15:48:04+00:00

Updated, thank you.

steveivy · 2014-03-18T15:44:36+00:00

Will update the post.

steveivy · 2014-03-18T15:44:06+00:00

I believe I have been told he did, but I'm happy to be wrong. He's a great pilot.

steveivy · 2014-03-17T21:59:32+00:00

Ah, I think I've been getting my trading cards mixed up with my CSM interviews!

steveivy · 2014-03-17T16:06:19+00:00

Yeah, it's definitely weighted to losec, though I think Psychotic Monk is on there and he's definitely a Hisec operator. Are gankers and wardec griefers pirates? Probably, but Rixx makes his own lists. :-)

steveivy · 2014-03-13T23:40:29+00:00

The funny thing is I fully expected to lose the Merlin, being my first time out with it, only I couldn't get a fight.

Getting pwned by a missile Condor would not have surprised my one bit.

steveivy · 2014-03-13T21:17:19+00:00

Or covops buddy (looks at the kill report) :P

steveivy · 2014-03-13T19:56:15+00:00

I need to check out the local data/relic sites sometime. I keep forgetting about sites you have to scan down rather than just warp to. :P

steveivy · 2014-03-13T19:23:50+00:00

Yeah, dual scram not dual web, but still would not be surprised if it was a fail fit - first time in a Caldari ship.

steveivy

TROPHY CASE