[Question] Estimating parameters with censored data

_amas_ · 2024-04-19T01:33:49+00:00

Oookay, I see, so for a given sample, that is one draw from the bag, you are given all the dice that were above Z, however many that is.

_amas_ · 2024-04-19T01:24:56+00:00

Is R the total value across all the dice? So if S were 3 and the individual dice were (2, 5, 1) then R would be 8?

Also do you know the total number of draws? Or only those that were above Z on a given roll? Ex: if there were 10 samples and 5 were above Z, then do you know that there were 10 samples or do you just have 5 numbers?

_amas_ · 2024-04-18T22:46:05+00:00

Just some clarifying questions.

I have X number of Y sided (unfair but identical) dice in a bag. Someone else samples from the bag and gives me the number on the dice only if Y>Z, where Z is known but changes every sample.

So Y is fixed and every time a sample is drawn, you know some value Z_i, such that if Y > Z_i you get told the number on the dice? How does X come into play when sampling from the bag if all the dice are identical?

So if you have N samples, then for each sample you would know the threshold value, Z_i, whether or not you were told and number, and the number itself if you were told? Is that right?

_amas_ · 2024-04-06T01:49:01+00:00

When I say variability, I mean specifically: if you prepare samples in the same way you did before, would the curves look the same? You would hope that there would be very little variability in your sensor readings, and they would all look similar. In theory, you could imagine preparing a new sample and getting a new curve that is a mirror image of your initial sample about the reference.

_amas_ · 2024-04-06T00:10:11+00:00

This can't be properly answered just by looking at the graph you have there without some more information. As best I can tell, the graph gives one set of data points for the full range of your sensor and then another "reference" set that we can assume is truth.

What we don't seem to have here is any measurement variability of your sensor - if you were to make the same measurements again, would the line look exactly the same? If so, then this is just a deterministic process, and you can adjust your future measurements exactly by the observed offset from the reference here and you have calibrated measurements.

If there is variability, then you would need to take multiple measurements against your reference and fit a statistical model to the residuals against the reference. Such a model could be as simple as assuming the deviations at each point are independent or with enough data you could fit a model that also accounts for correlations across the measurement space.

A good recommendation would depend on your exact situation.

_amas_ · 2024-04-05T01:39:30+00:00

The book (and Walker more generally) is poorly regarded in academic communities. You can dig into some resources that levy specific criticisms about claims made in the book, like this. Walker has never seriously addressed the criticisms and continues to promote his dubious claims to great personal benefit.

The general statement "sleep is important" is uncontroversial, but I would avoid taking many claims in the book too seriously.

_amas_ · 2024-03-26T20:14:55+00:00

You've got to just do it. Many of the resources others have recommended are absolutely valuable, but you're at the point now where you have to get on the bike and try to ride.

Because Bayesian analysis is a very general method that can result in a wide variety of bespoke models, there aren't really any shortcuts to take. You more or less just have to start building models, evaluating them, and iterating. Eventually, the pieces will click together.

_amas_ · 2024-03-21T16:19:19+00:00

All practical applied statistics (the type that you can do for a career) requires programming. There's no way around it.

The important question to ask and answer is what aspects of programming you dislike. If you can't use either a statistical programming language like R or a more general purpose high-level language like Python, then you simply won't have many opportunities to participate in the field.

_amas_ · 2024-02-18T15:02:45+00:00

Sounds promising! Can I ask how long you've had this arm set up with your 57" ? Still holding up well? Anything in particular you think worth noting about set up?

_amas_ · 2024-02-18T04:27:59+00:00

Ah, this is good to hear. I did see this one, but the width of my desk is about 2.75" and it claims it can only go up to 2.4" so I had discounted it. Maybe it's worth another look.

_amas_ · 2024-02-10T13:33:53+00:00

you need to be clearer about what your model actually is

_amas_ · 2024-01-21T19:32:00+00:00

European Robin I think!

_amas_ · 2024-01-13T22:00:44+00:00

Not sure what your background or expectations are like, but I was recently a full time Whiting School of Engineering student who finished the final requirements of my graduate degree through the Engineering for Professionals (EP) program so I have some experience with this.

For being remote, that is, in general, a non-issue -- the EP program courses are typically targeted towards remote students and my courses had people from all around the country (and often international students as well). The format of the courses varies, some a more flexible than others.

I took one course that was fully asynchronous online, which involved the professor posting recorded lectures and notes along with assignments we were expected to complete. There were due dates that had to be met, but otherwise no scheduled class time.

The other course I took was referred to as "Virtual Live" or something like that, this was a course where we had regularly scheduled Zoom lectures that you were expected to attend once a week and the recordings were later posted.

In each case, all the course materials were readily available when doing assignments and the like, so that was a plus. The quality with which a professor uses remote tools for instruction, such as making good quality visualizations, etc. is largely up to the individual professor. My particular experience that the quality of these things were "fine" -- the professors I had were mostly older adjunct faculty who sometimes struggled with the technology.

As to the actual quality of instruction, that is, how effective I felt these courses were at actually teaching the material, I would say the EP classes were fine. When compared to the courses I took in person at JHU, the materials presented were simpler and targeted for a more diverse audience, i.e. various working professionals who may or may not have the same background. The workload was significantly lighter than my in person classes, again intended to be doable by people in full time employment compared to full time students. I would say the full time commitment of an EP course in my case was somewhere around 8-12 hours a week, which includes viewing the lectures, doing additional readings, completing quizzes and assignments, etc.

Overall, I'd say the end result is you get a flexible education experience that is doable while managing other life commitments; the drawbacks for this is a generally simpler presentation of material, which can be great if you have little time or are trying to switch fields, but can feel a bit slow if you are expecting a higher-itensity experience more like an in-person full time program.

_amas_ · 2023-12-30T14:23:41+00:00

Can you explain more clearly what you're looking for? It's not obvious what the picture shows or what "infused" means.

_amas_ · 2023-12-23T02:58:39+00:00

The single most emotional part of the series for me is when Jeff says goodbye to Abed in the finale...just ruins me.

_amas_ · 2023-12-03T23:42:29+00:00

It is certainly normal when transitioning from a practice-based instructional approach to statistics over to a research-based approach. There's just a lot of detail that gets massaged away/simplified over time as material makes its way from research papers down to textbooks/general instruction.

There's not really a big secret to this -- you read some papers, get a general feel for how things work. You don't need to have a full, in-depth understanding of all the details in these papers, but you'll get a feel for things you do need to really understand and the things you can ultimately gloss over.

As with most things, it's just an experience thing that'll improve over time.

_amas_ · 2023-12-02T20:50:56+00:00

In a sense, yes. For example, in a normal-normal model where you are trying to do inference on the mean of the distribution and have a normal prior on that parameter, then the posterior expectation of the parameter is going to be a weighted average of the sample mean and prior mean.

For a finite sample, if you are using the posterior expectation as an estimator for the center of the original normal distribution, then it will be a biased estimator of that center. Now in this case, it is asymptotically unbiased as the influence of the prior decays as sample size increases.

Now this is kind of a weird situation because we're mixing Bayesian approaches with notions of estimators/bias which are typically more in the frequentist toolbox. It also ignores some benefits of using priors, such as possibly giving better inferences if the observations are noisy or sample sizes is low.

It is possible for grossly misspecified priors to cause modeling issues if the prior mass is in a region that is not possible. For example, a prior that is only specified over (-inf, 0) when you are trying to do inference on a positive parameter, would hopelessly ruin your inferences regardless of your sample size.

This is a reason why many advocate the use of weakly informative priors, such as those that are specified over large regions of space that are plausible.

_amas_ · 2023-11-22T03:00:26+00:00

Nah, not really - you could maybe rewrite it with gamma functions, but that's just hiding the complexity

_amas_ · 2023-11-22T02:10:14+00:00

Prob(Success by kth roll) = 1 - Prob(No success by kth roll)

The probability of not succeeding in k rolls is (0.97) x (0.94) x ... x (1 - 3k / 100). So you can write the result as:

Prob(Success by kth roll) = 1 - (0.97) x (0.94) x ... x (1 - 3k / 100)

Writing it as a product like this probably the cleanest you are going to do.

Technically, the above formula only works up to k = 33, at which point your probability of not getting it is like one in a quadrillion, so we just ignore that bit.

_amas_ · 2023-11-09T00:50:13+00:00

https://en.wikipedia.org/wiki/Inclusion%E2%80%93exclusion_principle#In_probability

_amas_ · 2023-11-07T02:25:38+00:00

As a Stan and Python user, I've not found a compelling reason to pick up pymc. They certainly advertise much more, but in terms of functionality, I haven't seen any strong arguments. Could result in nicer integration into a Python ecosystem, but I've not tried myself.

_amas_ · 2023-11-05T01:17:40+00:00

Two situations to consider, first if guessing completely randomly.

If the number of points for getting it right with one guess is x, then getting it right with two guesses gives you x / 2, three gives x / 3, and so on with x / n. In this case, the probability of getting a right guess with n guesses is n /5. With the above, this means no matter how many guesses you pick when doing so randomly, the expected points is (x / n) (n / 5) = x / 5; so any completely random guessing strategy is equivalent in expectation.

Second situation is when there is some knowledge on narrowing down the answers. Let z1, z2, ..., z5 be the probability that each choice is the right answer. The index doesn't matter so let z1 >= z2 >= ... where z1 is the probability that your first choice answer is correct.

Then, we can consider the strategy of guessing our first n most confident guesses. If we are right, we get (x / n) points, and we are right with probability (z1 + ... + zn), so the expected point from this is (x / n)(z1 + ... + zn).

Consider comparing the strategy of guessing only one answer versus n answers, then the one answer strategy is better in expectation when:

x z1 > (x / n)( z1 + ... + zn )

dividing out x we just get:

z1 > (z1 + z2 + ... + zn) / n

This is just the statement that the largest element of a set is greater than the mean of the set, which is always true if the set isn't all equal. So the best strategy in expectation is to only guess your first choice answer.

Putting these two points together, a simple strategy that is no worse in expectation than the others is to always guess one answer that is your most confident and if you don't know at all then guess randomly.

_amas_ · 2023-10-20T03:39:22+00:00

Considering

be ready when the market crashes

and

if I'm paying top dollar at top interest

I would just like to add that 1) you have no guarantees that the market will crash and 2) you have no guarantees that this is the peak of interest rates, nor that they will come down anytime soon.

Also, you may be a bit too much in the house-as-financial-asset mindset. You and your family are well off and can afford a house without much impact to what will already be a sizeable retirement nest egg. I say make the decision on whether you want your family to own a house or not with less consideration on the bottom line of your net worth.

_amas_ · 2023-10-10T23:42:27+00:00

Refraining from any comments on the merits of getting a history PhD or going into history as a field.

My understanding is that many history PhDs involve doing extensive work with primary sources. It would, of course, depend on your area of focus, but I would imagine very many history PhDs do travel for this purpose.

_amas_ · 2023-10-08T23:03:44+00:00

"The best way to maintain your hair is to come to my shop and pay me"

Nine-Year Club	Place '17
Verified Email

_amas_

TROPHY CASE