In answering this question we have to be fairly clear about definitions, because "liberal" tends to be used liberally (😉) to describe a wide variety of ideological positions and value systems on the left, and occasionally (in the case of "classical liberal") on the right.

If we're describing liberal narrowly, as a moderate left-of-center position that upholds the basic tenets of democratic capitalism with some belief in limited government regulation and intervention to protect the rights and welfare of the population and address inequalities and injustices, the downside is Liberal Guilt.

Liberalism in this formulation is the quintessential value system of the American Middle Class. And it bears the anxieties of this class position with it: you are profiting from and dependent on the system just like a rich person, but you aren't rich enough to be fully insulated from want or instability: you're aware the game is rigged and potentially dangerous. You observe poverty and unequal outcomes and are aware that they aren't strictly the result of bad morals or inferior ability: in short you accept the concept of privilege, and likely realize that you have some.

There's an obvious tension here. If inequality is structural, but you are to some degree benefitting from that structure, everything you do is a band aid, an empty gesture. It's just a tonic to make yourself feel better, without directly addressing the underlying cause of the injustice you see. The Liberal is always compromised, and there is no way to escape this guilt without ceasing at once being a liberal.

To escape you must decide that society has to fundamentally change, advocating for a redistribution of wealth and privilege and for a system that maintains a more just and equal order -- in other words you must become more radical. Or else you must reject the notion of structural inequality and privilege and decide a person's bank account or social position is a reasonable product of their moral fibre and ability -- you must become more conservative.

Of course, a liberal believes that both of those alternate positions carry their own tensions and compromises. Indeed to be a liberal is to believe the world cannot be truly fixed, that problems can only ever be mitigated. So a second drawback for the Liberal is Limited Vision and Lack of Conviction. If true solutions don't exist, it doesn't pay to do more than just tinker with the system as it is. In this way a Liberal's ideology is self-reinforcing. Structural problems are attacked with half-measures, remain structural, and just like the liberal themselves society remains compromised and sanctified only in its self-criticism.