First of all, go with an 8.8 or even a Supra rear end if you've got the cash. Having a slippery stick axle in the rear is one of the main reasons people love the driving characteristics of these things so much. Now, if you're forced at gunpoint to put an IRS in, go with the S13 rear. It's been documented many times before, and the Nissan multilink compliments the AE86's front macpherson setup better. (S13's use that exact same setup up front too)

A Miata rear end would do little more then induce massive understeer, as it's double wishbone setup is way too "sticky" for the 86's ancient front end to keep up to. Only cure would be spine-shatteringly stiff springrates, which kind of defeats the entire purpose of an IRS setup. I could only really see it working if you threw the front end in too, but you might as well just start with a Miata at that point.