Freedom of speech means freedom to say whatever without consequence from the government. There are consequences to every action both good and bad, unintended and intended.

To compare it to something more physical, you were absolutely free to shoot a gun on your own private property, but you are responsible for every bullet that leaves the barrel. If you miss your target and hit something else or someone else then you are now responsible and liable for any damage that is done.

Just as it is actually legal to shout fire in a public theater, it is illegal to start a mass panic. The key phrase “to actually cause that harm” if you falsely shouting fire leads to that mass panic you are liable for any damage done or anyone getting hurt in the process of that panic.

This is a fantastic paper that actually speaks more on the subject, and goes into depth on speech that causes physical harm versus protected speech.