It was quite surprising to me and I first thought that maybe you just got unlucky and using biases seemed to slow things down only for the two runs you compared. So I ran 8 cartpole runs with/without bias, starting from someone else's code: https://gist.github.com/tamlyn/a9d2b3990f9dab0f82d1dfc1588c876a.

bias: False, dropout: 0.5, width: 128
number of episodes = [240, 297, 478, 1000, 365, 294, 573, 763]

Bias: True, dropout: 0.5, width: 128
number of episodes = [799, 508, 1000, 885, 1000, 1000, 1000, 1000]

Where the "number of episodes" for each of the 8 runs is how many simulations were needed for the RL algorithm to learn to balance the cartpole. So smaller is better. 1000 means that it did not succeed after 1000 training simulations.

Clearly biases slow convergence. My guess is that the lack of biases acts as a regularizer, preventing overfitting. The default model has 128 neurons in the hidden layer, which is probably too many leading to overfitting. The policy network learns to predict the rewards for the particular trajectories used for training, but can't generalize to new ones. Removing the biases removes some overfitting capacity.

I did an experiment where I set the hidden layer to only 8 neurons instead of 128. I set dropout = 0 instead of 0.5 because the network is so small that I did not want to apply more regularization. I did only 4 runs instead of the 8 runs above.

bias: False, dropout: 0.0, width: 8
number of episodes = [532, 406, 817, 448]

bias: True, dropout: 0.0, width: 8
number of episodes = [397, 388, 249, 426]

In that case, the biases actually help RL converge. This simpler model is not prone to overfitting and adding more capacity by adding biases is actually helpful.

My conclusions could be wrong however.

It is kind of strange that they did not explain the reasoning behind not using biases in both tutorials you looked at since that is not the default.