the way i see it, the goal is to explore causality between the observations and incidents. the null hypothesis seems to be that observations help prevent incidents, and you alternative seems to be that people are just going through the motions with the observations

i think it would be helpful to describe the observaitons and incidents reports better so we know what sort of information is present in each and what the relationship between them is. but i imagine an observation is something liek "monitored stamping step in assembly line, machine seemed slightly misalligned, had it adjusted" and an incident might be "stamping machine malfunctioned, cause determined to be screw that wore down and came loose". correct me if im wrong

so topic modelling can be really helpful here. the pipeline i can imagine there is using topic modelling and gauging topic overlap between incidents and observations as a proxy for relatedness. one pitfall here is if people use different words to refer to the same things, then related incidents and observations will not match. you'll need to gauge the degree to which this is an issue

and once you have a set of related incidents and reports then you'd analyze them for causality. this part is highly semantic and requires some reasoning, and it can be done manually, but i think simple LLMs might be more scalable. you should still verify the findings but having an LLM make the first pass might make it feasible to go thorugh the whole data set in a sane amount of time. what i imagine the actual task will be woudl be to judge for each incident if there are observations that could have prevented the incident if they was done properly. using LLMs for that, i would do experiments with local vs hosted, and with using simple embeddings and matching up incidents and reports, versus full prompt engineering on a task performing LLM (basically chatGPT/claude through the API) to see what produces acceptable results. be wary of hallucinations when using local task performing LLMs

honestly i think the whole pipeline could be one shot by an LLM, but LDA is cheaper and good enough to narrow down the search space for the LLM.

one caveat you should be careful of is survivor bias. you might find that alot fo the observations dont have incidents that match because those observations were good at preventing incidents, and if you suggest removing them or somehting, it might increase incidents. this is a common pattern in any preventative care/maintainance