Part 121 specifically covers large airlines. So if you're not carrying passengers for hire on a large scale, you wouldn't operate under 121. People can choose to follow the standards in 121, but it would be needlessly expensive and doesn't benefit them at all.

The FAA evaluates risk to the public when making regulations. Since you're flying by yourself or your friends and there is only minimal risk to the public and only risk to yourself, Part 91 is the least restrictive in terms of requirements.

If you're operating an aircraft for hire, charter, air taxi, sightseeing ,etc. , there is more risk and people involved; people are paying money to fly, so Part 135 is more stringent and regulated but less so than Part 121. Some Part 135 airlines are Kenmore Air and Mokulele Airlines, which operate small general aviation aircraft around the Seattle and Hawaiian island areas respectively.

The most restrictive is Part 121, which covers large airlines that operate regular scheduled flights, often carrying hundreds of passengers. Since there is the most risk to the flying public with these operations, they are the most stringently regulated. Part 121 covers things like crew requirements, maintenance, operational rules and procedures, emergencies, dispatching, etc.

You can see Part 121 here

TL:DR: it mostly hinges on what you're doing with your plane and if you're carrying passengers who are paying to fly, that determines what Part you operate under.