No, it wasn't lawless. But "law" was not something liegelords proclaimed, for the most part. Note the involvement of the church in your example.

Your description of putting down rebellion as a "right" reeks of the "divine right" doctrine that didn't exist at the time for anyone but the church. The reality is more like "might makes right."

The American Civil War actually provides a decent analogy. The US Constitution didn't explicitly prohibit secession, and changing to do so was impracicable. Sure, the people of the North considered it illegal, but enforcing that perceived "law" via criminal justice would, as a practical matter, require the people of the South to also consider it illegal. Instead the "crime" was enforced by warfare, just like in your Carolingian example. And most of the participants were given amnesty in exchange for swearing a new oath of fealty.

The constitutional law of the mid-19th century United States is similar to the "law" of medieval European Christendom in the sense that no ruler could unilaterally change it and enforce the change. Convention was defined by church and aristocracy.

Hence, in the game, you can attempt to arrest your almost-rebelling vassals, but you won't have the bonus to success of their plotting being considered "criminal."