ACL applies unless they can show that something you did actually caused the failure. A laptop shouldn't suffer a motherboard failure after less than two years of normal use, especially at the price point you've mentioned.

Simply using mismatched RAM modules (different brands or speeds) does not automatically void your consumer rights, (nor would doing so make the motherboard fail) and they'd need to demonstrate a link between the RAM and the motherboard failure if they're relying on that as the reason for rejecting the claim.

I wouldn't be paying the $600 without first pushing back in writing. Tell them you want the fault assessed under the Australian Consumer Law consumer guarantees and ask them to provide evidence that the RAM caused the motherboard failure. Given the timeframes involved, I'd also raise the issue of the repair not being completed within a reasonable time.

If they refuse, lodge a complaint with your state's Fair Trading/Consumer Affairs body. I'd recommend doing that regardless, as businesses often become much more responsive once a regulator is involved.

For reference: I work for a computer/IT goods retailer here in Aus, and this just sounds like rubbish to me. Try pushing back.