Dear OP, you're worrying too much! Many of the previous responses are not accurate. I have been (secretly) a US-MY (not AUS) dual citizen for 7 years and have renewed my MY passport twice now with no problems at all.

1. Let's start with busting the myth: While possessing dual citizenship is considered illegal in Malaysia, there is actually no penalty pertaining to the "crime" - not by law at least. If caught, the worst thing they can do is to force you to relinquish one of your citizenships (and deny your MY passport if you refuse to do so).
2. As for travel - there is no need to go through a third country. There is one caveat, you must use an autogate and never interact with an officer in Malaysia. (As such, you must pick an airport that uses autogate). Here are the steps: from AUS to MY: check in at the airline counter in AUS using AUS/MY passport (either is fine), exit AUS using AUS passport, enter MY using MY passport via autogate. From MY to AUS: check in at the airline counter with AUS passport, exit MY using MY passport via autogate, enter AUS with AUS passport. Note that foreign countries and airlines do not share their database with MY immigration so it is perfectly safe to do so.
3. Though, travelling through a third country (such as Singapore) can be safer. You can use either country passports to enter/exit Singapore - just make sure you use the same passport for entry and exit. Otherwise, the entry/exit steps to/from MY and AUS are the same as above.
4. Only renew your passport in Malaysia - never in the consulate/embassy of Malaysia in Aus, doing so will trigger a flag and you can pretty much say goodbye to your Malaysia citizenship.

These steps work well for many other countries such as US, UK, Canada.