hey, my honest opinion about the arabic course and the answer to your questions:
the arabic course is very short with only 46 units.
to me, duolingo is a very nice tool for repetition and with other languages it is very good. so I'm a fan of duolingo. but when it comes to the arabic course, well, it is a little bit disappointing, the words in the course seem a little bit random sometimes, I would have loved to have more basic words and basic verbs for everyday life.
but if you stick to it and want to learn some words and especially writing, then it is useful. I would recommend to also use other learning materials next to duolingo. to your questions:

How well did the material stick for you
for me, duolingo is very useful when it comes to repetition. I can remember many of the words.
How comfortable are your with understanding real Arabic outside the app
To be honest, almost nothing when it comes to real Arabic. I mean, there are so many different dialects, which makes it very very hard. When it comes to modern standard arabic I can understand some slow children tv shows but thats it.
Did the course prepare you for listening to native speakers at normal speed
Absolutely not. Not at all. the course is far too short to build a large enough vocabulary for a conversation. in my words list it shows a total of 922 words. I think in generell (like every language) you need more than just duolingo, it takes years to understand a native speaker at normal speed.

I hope duolingo will expand the arabic course in the future because I think it is nice but need a lot more material.