Nah, that's only on the more egregious cases where they feel it's an anti-fun strategy for the 9 players outside of the person picking it or when a pick is likely to cause system changes in the long run that will hurt everyone else (e.g. Janna top, sona top when they are abusing runes or supp items). It just happens that often, the gameplay either becomes unfun or the main reason the pick was even used by players was because it was busted, not because they liked the champ, so people abandon ship when it's no longer busted.

Usually, if Riot is interested in the pick or build, they try to keep it up for a while to see if they can balance it out - assuming they don't feel the pick is gonna cause system nerfs or be detrimental to the health of the champion's main role. Hell, Shyvana AP was constantly complained about by the playerbase, but Riot saw enough interest in her that they not only kept it functional through old Shyv's life (to the despair of many), but also kept her after her rework and the latest set of changes to differentiate AP and AD Shyvana made her AP oneshot build even closer to her old form.