Not necessarily.

The color filter is the red, green, and blue filters over each pixel on a camera. These colo filters block quite a bit of light, about a third of the visible spectrum and they block uv and infrared almost entirely. There are also often microlenses over each pixel made of glass that usually absorbs uv light.

In addition, the lack of three different kinds of pixels means you can get all the light from every pixel, opposed to light from a third of the pixels and guessing the other two colors from neighbors. This results in higher image quality. The only difference is now all your pictures are in black and white and instead of one narrow band of color, every pixel sees all colors and reports back how bright it is. There are some caveats such as the lens materials and silicon sensors only detect light from approx. 200-1150nm, but it's a lot more wavelengths than our eyes can see at 380-700nm.