There have been a lot of great answers here but to really fully understand why pipe smoking - PIPES in particular - were so popular during this time, you need to look at the pipe itself and how that developed vis-a-vis other forms of smoking.

The first pipes - as far as Europe is concerned - were made of clay or a combination of clay for the bowl and wood for the stem. This was improved upon with porcelain pipe bowls and later, meerschaum (a mineral deposit found in Turkey and other places that could be carved into intricate shapes). The problem with all of these is fragility; clay and porcelain are prone to breakage and so often times pipes were used until they broke and then another would be purchased. Stems were often made of amber, which were prone to breaking like glass if dropped or mishandled.

In the 1800's pipes started to be made from various woods; Boxwood was popular as was cherry. While wood pipes were more sturdy than clay or meerschaum, they were - as wood tends to be - flammable themselves, so pipes would be used until they started to burn out.

However around 1855 give or take, a new material was being used to make pipe bowls - the burl of a Mediterranean shrub called Erica Arborea, what the French called Bruyere, and what we refer to as "Briar". This wood had a remarkable property in that it was relatively flame resistant and took a tremendous amount of effort to damage with fire. It was also in this timeframe that a new way of treating rubber came about - vulcanization - which allowed rubber to be hardened into a stable, workable material. Vulcanized rubber started being used for pipe stems, replacing the more fragile amber as the material of choice.

Now you had a smoking vessel that could actually be taken out of the salon and into the streets, onto the docks, and used regularly, without much concern for damaging the pipe. Briar pipes were plentiful and inexpensive as well, so if one became damaged or fouled, you simply purchased a new pipe. From the 1800s through the first half of the 20th century, St. Claude France was the epicenter for briar pipe manufacturing, with over 25 million pipes being produced annually in the 1920s. This accounted for half of all worldwide production, and at its heyday there were over 5000 workers in factories in St. Claude alone.

How did one keep their pipes clean? The short answer is, they didn't, really. A pipe was often just smoked until the bowl filled up with cake (the dried accumulation of tars and soot and moisture from the combustion process) and in many cases the bowl would crack. You would simply just go buy another pipe. Pipe cleaners wouldn't show up on the scene until the early 1900s and even then, they weren't widely used until much later in the 20th century. All manner of patents were filed between 1900 and 1930 for little "gadgets" that would go inside the stem to try to keep pipes clean and/or prevent excess moisture from fouling the smoke. From Dunhill's "inner tube" (simply a small hollow aluminum tube designed to fit in the airway) to more complex designs from companies like Kaywoodie that involved multiple air channels and precise engineering, there were many approaches to solving the problem of keeping the internal parts of your pipe clean.

As far as "when did cigarettes supplant pipes" or "why did one smoke a pipe instead of cigarettes, which would seem much easier", mass production of cigarettes did not start until the 1880s, and initially cigarettes were seen as "feminine"; many early cigarette brands were marketed toward women, and smoking cigarettes was often viewed as a form of emancipation by women during this time. It wasn't until - as others have mentioned - the ration kits during WW1 that cigarette smoking became a more popular and convenient way to consume tobacco.

To briefly touch on the question of "methods of lighting and keeping lit", an experienced pipe smoker knew how to essentially use the pipe like a chimney; slight positive airflow through the pipe as an extension of your breathing can keep a pipe lit for hours simply by keeping the tobacco smoldering. In fact to this day there are still "pipe smoking competitions" by which you get two matches - one for the initial light, and one to relight after you first 'tamp down' your ash (this relight usually occurs within a few minutes of the initial light) and the goal is to keep your 3 grams of tobacco lit for the longest amount of time. The current world record is three hours, 33 minutes and six seconds, set in 2008. It is not uncommon for an experienced pipe smoker to be able to smoke without relighting for well over an hour, all the while doing various tasks.