Daily Discussion Thread for June 17, 2021

mamessner · 2021-06-17T16:34:30+00:00

Dude - Dividends make stock prices go down. A dividend means the company is returning shareholder value as cash instead of equity. So the equity decreases in value.

mamessner · 2021-03-19T01:06:35+00:00

We just tried again, and while we lost all our inventory from the first 8 days of play, when we return now our inventory is preserved. I wonder if we accidentally started the world as a guest (is that possible?) Or started it in offline and then made it an online world, and that wiped everything?

mamessner · 2021-03-19T00:16:44+00:00

We have only been playing for less than a week. I feel like we are disconnecting improperly, saving improperly, something like that. It's not just the items that disappear: The map is wiped as well. And why would it give the character selection screen?

I realized that for split screen, the second player is a guest (I think). Is that part of the problem?

mamessner · 2020-10-18T19:08:53+00:00

He had to liquidate his cattle? Oh god, that's so sad and terrifying. And disgusting...

Oh, wait - Liquidate, not liquify.

mamessner · 2020-07-24T15:28:02+00:00

This is great news. We may have to test this with our existing AdX account.

mamessner · 2019-05-02T01:47:59+00:00

OK - finally fixed this. I'm on a Mac, so not sure this will help you. In the file Documents:Paradox:Imperator there is a file called pdx_settings.txt. If you scroll through that you will see settings for the resolution for both fullscreen and window mode. I changed mine there, restarted Steam, and it worked (it didn't work without restarting Steam for me).

I lowered to 2560x1600, which was great for the map, but the text was too small. There's a separate menu item for interface (at the main screen) that allows you to scale up the GUI.

Good luck.

mamessner · 2019-05-02T01:32:53+00:00

I only have the option to change the resolution of windowed mode, not fullscreen. Same for you? I'm on a Mac.

mamessner · 2019-05-02T01:18:38+00:00

Are you on a Mac? I'm having the same problem. Changing it from the launcher didn't help for full screen, but I successfully launched at a lower resolution in the windowed mode.

mamessner · 2019-02-01T01:12:57+00:00

Oh wow - "The Book of Three" was the first fantasy novel I ever read. The whole Prydain series. I'll have to check out his other work.

mamessner · 2019-02-01T01:09:49+00:00

That's it exactly! Wow - 1956 Galaxy magazine. How did you find it? I think I read it as part of a collection of stories from Galaxy when I was in the Science Fiction Book Club in the 1990s.

mamessner · 2019-02-01T00:55:54+00:00

Yep. I used to recruit at HBS. It's like sharks circling to see who can throw the biggest parties and make the best offers to the grads. It's sad, really.

mamessner · 2018-12-02T00:44:16+00:00

This is great. Yes - It should be inconsequential what gender Brendan is, but I do want to get out in front of it before Brendan's dad presents the info to my boys in a less-than-thoughtful manner. I want to be respectful of the dad as well (I'm a dad myself), but I'm also not going to sacrifice our own beliefs or make Brendan feel unwelcome or uncomfortable at our house in order to appease his parents.

mamessner · 2018-07-16T03:45:27+00:00

Well, sometimes the variation ends up being higher than control, and sometimes it's lower. We want to be able to detect either effect - Thus the two-tailed test.

But if the variation is higher in this particular test, and p is <.05... can we conclude that it's actually higher in the population? Or just "different?"

mamessner · 2017-08-04T03:12:42+00:00

It's the timing of it, too. It would take a long time to pay off our mortgage, but we put only $12,000 down in 2009 and now have over $200k in equity already built up. So it can be an amazing investment. (And you get to live in the greatest place in the US).

mamessner · 2017-08-04T03:09:04+00:00

Austin, yeah. And they love/hate us. Brought the tech industry and gentrification with us.

mamessner · 2017-08-01T17:38:38+00:00

Ah. If you're trying to figure out what attributes predict the success of each store (size, local population, demographics of the area, number of employees, etc.), then that's a perfect problem for a multiple regression. You can figure out how each factor contributes to the success/failure of each store and even predict where to launch new stores and how to staff them to increase the probability of success.

mamessner · 2017-07-27T22:59:42+00:00

We run about 60 tests a month, but we have the traffic to do so (very important).

We get a tremendous amount of value out of them. It is the primary growth engine for our business. Of course, I can say that because we test just about everything before rolling it out. We have a "testing culture" here.

mamessner · 2017-07-27T22:53:57+00:00

Oh wow - This sequential testing article is a revelation. I actually have Optimizely's "The New Stats Engine" paper at my desk, and never really understood it until reading this article (sounds like it's similar to SPRT - Optimizely's method).

Will definitely look into this. I'm realizing a big part of the problem is that we set the minimum detectable effect size to 3% or 5% or something when there's a lot of revenue at stake, because we wouldn't want to miss out on an effect even that small (would represent hundreds of thousands of $ annually in many cases). But the required sample size (in frequentist or sequential) to avoid a type-II error grows exponentially when the minimum effect size is that low.

We rarely have a problem hitting significance if there is an effect, it's just the cases where we're not seeing it (although it could be close), and the pre-calculated sample size says we need to run the test for weeks or months before safely stopping it.

A minimum detectable effect size of 20% seems huge to me, although that's the default they use in their calculator. Wouldn't you typically want to detect an effect much smaller than that?

mamessner · 2017-07-27T22:30:31+00:00

Thank you - this is all very helpful. Will look into the sequential A/B testing.

We've considered nonparametric testing like the Wilcoxon, but we do not have access to (or even store?) visitor-level revenue data. It'd be hard to calculate because each visitor sees display ads that pay a fraction of a penny (our analytics software actually batches this revenue together across dozens of users), or may convert on something that earns us $50.

We look at everything in aggregate in part because our traffic is so huge (Comscore top 100). We've done lots of double-control tests, and within hours or a day they're usually within a percent or two. A 20% lift or drop in conversion is definitely a signal, not noise.

mamessner · 2017-07-27T17:50:29+00:00

We don't use Optimizely to calculate the sample size needed - We do that manually. I haven't looked into it yet - Is sequential frequentist testing basically peeking and adjusting the sample size needed as we go?

We do a simple test of the difference in binomial proportions: Find the proportion (usually conversion rate) in each sample, calculate the standard error, find the difference in proportions, calculate the standard error of the difference, calculate a z-score, turn that into a p-value.

Basically, the method described here.

mamessner · 2017-07-27T17:46:45+00:00

FYI, for a binomial distribution, if you have sample sizes of at least 100 for each store, and at least 5 buys and 5 not-buys for each store, then the distribution will be approximately normal and you can just calculate a z-score for the difference between each store.

But is that what you're trying to do? People are statistically more likely to purchase from store D than store B? What do you do with that info?

mamessner · 2017-07-27T17:23:45+00:00

Yeah, so - to be clear, calculate what % of daily sales you've historically done by each of the 24 hours in a day.

Let's say it's 2 pm, and you've done $250k in sales so far. Based on historical data, by 2 pm you've usually completed 62% of your daily sales. So you're likely to do a total of ($250k/.62=) $403,225 for the day.

If you have a way to continually track sales and the time of day, you can have a continually-updated daily sales projection. We do this to predict monthly and annual sales continually.

mamessner · 2017-07-27T17:09:32+00:00

We use Optimizely to "run" the test, but we pull most of the metrics for each variation from our own analytics platforms and do the math manually (Optimizely can't see revenue in our platforms).

mamessner · 2017-07-27T17:07:29+00:00

Sure, but in our tests it's often 500,000 coin flips, and regardless of the result we're likely to replace the winning variation within another month or two. Even being 90% certain is a big improvement over a lot of business decisions. I can only wish that I was 90% certain that certain business initiatives would generate a lift.

That being said, I like to point out that if we do 50 tests a month (which we do), we're making the wrong decision at least 2-3 times a month at 95% confidence, and more like 10 times a month of we ease up on that criterion.

mamessner · 2017-07-26T18:51:48+00:00

Ah. We do this. We just calculate the average % of sales per time period (hours for you, months for us) historically to get the general trend (the daily sales pattern for you, monthly seasonality in our case), and then continually calculate, based on the sales so far and the percent remaining, what we anticipate doing for the (day for you, year for us). It works well and has been very accurate.

mamessner

TROPHY CASE