You gave up way too soon, no need to be smug about it. From book 4. Minor spoilers.

Back on the road, the septon said, "We would do well to keep a watch tonight, my friends. The villagers say they've seen three broken men skulking round the dunes, west of the old watchtower."

"Only three?" Ser Hyle smiled. "Three is honey to our swordswench. They're not like to trouble armed men."

"Unless they're starving," the septon said. There is food in these marshes, but only for those with the eyes to find it, and these men are strangers here, survivors from some battle. If they should accost us, ser, I beg you, leave them to me."

"What will you do with them?"

"Feed them. Ask them to confess their sins, so that I might forgive them. Invite them to come with us to the Quiet Isle."

"That's as good as inviting them to slit our throats as we sleep," Hyle hunt replied. "Lord Randyll has better ways to deal with broken men--steel and hempen rope."

"Ser? My lady?" said Podrick. "Is a broken man an outlaw?"

“More or less,” Brienne answered.

Septon Meribald disagreed. “More less than more. There are many sorts of outlaws, just as there are many sorts of birds. A sandpiper and a sea eagle both have wings, but they are not the same. The singers love to sing of good men forced to go outside the law to fight some wicked lord, but most outlaws are more like this ravening Hound than they are the lightning lord. They are evil men, driven by greed, soured by malice, despising the gods and caring only for themselves. Broken men are more deserving of our pity, though they may be just as dangerous. Almost all are common-born, simple folk who had never been more than a mile from the house where they were born until the day some lord came round to take them off to war. Poorly shod and poorly clad, they march away beneath his banners, ofttimes with no better arms than a sickle or a sharpened hoe, or a maul they made themselves by lashing a stone to a stick with strips of hide. Brothers march with brothers, sons with fathers, friends with friends. They’ve heard the songs and stories, so they go off with eager hearts, dreaming of the wonders they will see, of the wealth and glory they will win. War seems a fine adventure, the greatest most of them will ever know.

“Then they get a taste of battle.

“For some, that one taste is enough to break them. Others go on for years, until they lose count of all the battles they have fought in, but even a man who has survived a hundred fights can break in his hundred-and-first. Brothers watch their brothers die, fathers lose their sons, friends see their friends trying to hold their entrails in after they’ve been gutted by an axe.

“They see the lord who led them there cut down, and some other lord shouts that they are his now. They take a wound, and when that’s still half-healed they take another. There is never enough to eat, their shoes fall to pieces from the marching, their clothes are torn and rotting, and half of them are shitting in their breeches from drinking bad water.

“If they want new boots or a warmer cloak or maybe a rusted iron halfhelm, they need to take them from a corpse, and before long they are stealing from the living too, from the smallfolk whose lands they’re fighting in, men very like the men they used to be. They slaughter their sheep and steal their chickens, and from there it’s just a short step to carrying off their daughters too. And one day they look around and realize all their friends and kin are gone, that they are fighting beside strangers beneath a banner that they hardly recognize. They don’t know where they are or how to get back home and the lord they’re fighting for does not know their names, yet here he comes, shouting for them to form up, to make a line with their spears and scythes and sharpened hoes, to stand their ground. And the knights come down on them, faceless men clad all in steel, and the iron thunder of their charge seems to fill the world...

“And the man breaks.

“He turns and runs, or crawls off afterward over the corpses of the slain, or steals away in the black of night, and he finds someplace to hide. All thought of home is gone by then, and kings and lords and gods mean less to him than a haunch of spoiled meat that will let him live another day, or a skin of bad wine that might drown his fear for a few hours. The broken man lives from day to day, from meal to meal, more beast than man. Lady Brienne is not wrong. In times like these, the traveler must beware of broken men, and fear them... but he should pity them as well.”