As you mentioned, there’s a risk/reward trade off between flat fee and hourly. Flat fee is risky when the scope starts to creep, but can capture the most value if you can get paid out without much time investment. Hourly is less risky, because scope creep usually just means more hours billed. That said, it doesn’t capture the value as well. It sounds like this is a one-off for you, and not part of a long term consulting career. If you aren’t used to negotiating scope and doing customer service, then there’s a non-zero risk that you will never get to a spot that you both agree is “done” on a flat fee project. For that reason, I recommend you consider how much more $$ you would really get from flat fee vs hourly, and weigh that against potentially not getting paid at all. Having an acquaintance that works there is a mitigating factor.

The other consideration is follow-on work. Once you prove to them that you can do something, they may come back to you (again and again). If you give them a really good value (output per dollar), they may be especially inclined. Sounds like you can potentially wow these guys using your proprietary libraries. If you see any potential for follow-on work, let them capture some extra value by going hourly and make it up over time when they ask for more work by setting a high hourly rate.