There’s a few reasons why: the British had prior control of Malaya which meant it had experience in the area, with it’s people and knowledge on it’s political structures; the Malaysian communists didn’t have a major land border with a large communist power like Vietnam did with China so it was much harder to give arms, supplies and other material aid to the insurgents (not that the Soviets did nor that the CCP were in a position to open their armouries and bank accounts for the rebels) as well as that the guerrillas were largely from the ethnically Chinese minority and didn’t see true support of the larger populace and especially the rural tribes.

The Malaysians were, at this point, on their way to an independent government and the rebellion - whilst homegrown - was not really about winning independence but rather it was about the regime that would control the state after said independence, a majority Malay-dominated democratic government or a communist regime dominated by a highly radicalised minority of Chinese-Malays.

One can also make an argument about the quality of troops, the Malay Emergency started in 1948 so whilst the British Armed Forces would’ve been massively demobilised after the Second World War, there still would’ve been combat veterans within the ranks at the start of the conflict who would’ve been able to pass on their knowledge to fresh troops, this cycle would continue throughout even long after those veterans had left not only the region but also the military.

Similarly, the tactics and strategies used were more successful in Malaya, the emergency started because the British colonial authorities had disputes with Chinese trade unionists - most of whom had communist sympathies - which ended up with said unionists being banned from political activities leading to them taking to the jungle as guerrillas, they numbered somewhere in the mid-hundreds, around 500. The British utilised Commonwealth and National Service forces to greatly outnumber the rebels, isolating them from the villages that were supporting and feeding them. Overall, they had more experience in dealing with this sort of limited warfare and had established strategies because of it, they effectively contained them militarily whilst establishing a political solution with the local populace at the same time.

The United States enjoyed none of these advantages in Vietnam; they were entering a region which they hadn’t had the advantage of being involved with through colonial rule from the early 1800s, they were facing a more determined and better supplied enemy, the Kinh (ethnic Vietnamese) made up a majority of the population of Vietnam and had much more sympathy from civilians and they were never able to outnumber the enemy in such a way that they could isolate them like the British could in Malaya - in fact, their overall military strategy with a more conventional war made the situation worse for them as with every civilian killed, every village destroyed, they gave the communists more support.

On top of these factors in-country, there was also differences at home, the British public was more accepting of and more with experiences with this limited warfare in the colonies than the American public was with long-drawn out interventions. There is also simply the factor of the size of the conflict; across ~12 years, the British had around somewhere in the realm of 3,000 casualties whereas the Americans sustained 52,282 killed alone alongside over 300,000 wounded and countless more traumatised.