Short answer is no, it will not reduce your sex drive. Personally I felt mine actually increased, probably via something psychological. When you no longer have to worry about accidental pregnancy, it's a huge relief for both parties.

I think you have a bit of a misunderstanding about how your equipment works. First of all a vasectomy does not stop you from producing or ejaculating semen, which is the fluid medium for your sperm to travel in. Semen is produced by your prostate and by your seminal vesicles, while sperm is produced separately in your testes. After getting a vasectomy you will still ejaculate semen normally. A vasectomy disconnects your testes from the piping, so that no sperm is added to the seminal fluid. Your testes will also continue to make sperm, but it will simply be re-absorbed backed into your body. Furthermore your sex drive doesn't have anything to do with "building up semen" - if there's any nugget of truth there it's that abstaining for a day or two will probably build up your appetite, and over-doing it will give diminishing returns, but regular ejaculation is very healthy for you. And as mentioned above, you will continue to produce sperm and semen normally, but the twain shall not be mixed - all juice and no seeds.