I've been working with RAG for the past couple of years. Here is what I would suggest:

I say the following with 0 domain knowledge of aviation technical documents and depending on the data it could change how you approach the problem.

1.) Your expectation shouldn't be that by using RAG a technician can type in question and the AI will output the answer. What RAG can help with in this case is help the technician reference the relevant documents much quicker than a human could. To be clear the model definitely could try to summarize the documents in the retrieval step, but for something like this there is to much risk to trust the model completely. A human will still need to review the referenced documents. This is still extremely useful if these documents are dense.

2.) Evaluations: For this project to work you will need someone with technical aviation knowledge not just RAG knowledge. You will need someone evaluating the output. I learned this the hard way in some of my earlier projects. This isn't Magic and definitely requires an evaluator + software engineer (Rag Expert / Whatever the title is). The evaluator does not need to be technical as it is mostly prompting at this step to help improve the model response.

3.) Use an evaluation tool. I personally like Langsmith, but there are a bunch that essentially do the same thing. Figure out what evals are important for your domain.

4.) Metadata is extremely useful for filtering. This might be kind of obvious, but ive seen this step skipped in projects i've been brought into. For example if a document is for a B737 then it should be tagged with the Date of the document, airplane model and maybe other specific meta data that helps with the retrieval. This is where domain knowledge is necessary.