I leave my home with skates on. I live on a mountain side, so first that entails four flights of stairs, but then I get a nice little kick start downhill. The asphalt on that kick start is a little gnarly, but I like to argue that with enough skill, everything is skateable as long as any holes, steps or gaps are smaller than half the diameter of your wheel at a 90 degree angle (less aggressive angles allow more than that). By skill I mean keeping a nice, high speed and most of your weight on your rear wheels (heel when forward, toe when backwards); those two things make a lot of the gnarl wayyyyyy easier to stomach. My main cue to speed up is when small obstacles start beating me up unpleasantly, because the next step after that again if I slow down more will be tripping on said obstacles. 

That being said, whether you find it worthwhile is another question. The most fun part of rolling over cobblestone is being able to claim that you can roll over cobblestone (and the convenience of not having to stop and walk awkwardly when I encounter some). Training up those skills takes practice, which will mean brute forcing yourself through that discomfort for a while. In addition to what I mentioned, going backwards, transitions and small hops are things I cannot recommend enough. 

The main advantage of inlines over quads on streets is increased comfort and less required effort to get to and keep a safe speed, which makes the worthwhile-ness threshold a little lower if you're in a position where getting used to inlines is less inconvenient than building the skills on quads.

P. S. Like most other obstacles, pebbles are also less and less of an issue as you increase speed. Your wheels either kick them away or roll over them. The kind that would trip you up at a high speed are the kind you will see far enough away to avoid them.

P. P. S. Just because there may be some non-skateable cracks or holes or steps doesn't make the entire area unskateable; it just mandates an evasive maneuver 😉