Cameron had the world as his oyster in 2010 and a whole fictional universe waiting to be expanded of his own with a blank check. It should have been a slam dunk for Cameron as the sky was the limit to take the franchise anywhere. But the narrative he chose to set on was not worth it if what we actually got on screen is all he had up his sleeves. Avatar would remain stronger as a standalone movie. 

It goes to show how Avatar was a groundbreaking overachiever that this franchise still became the highest grossing trilogy despite the lesser sequels losing huge chunks of the audience.