If you want to be a profficient player then the answer is simply yes. I know all 12 keys but am not 100% confident with over half of them. I tend to stick to my comfort zone in the soulful sounding keys like Fmaj Eb maj and Bb Maj.
I do of course inject chords from other scales but I don’t really play who pieces in scales that I am not too familiar with.
However the more keys I leaned to play, the easier it is to improvise since you will be able to understand how and which chords lead into each other.
Sticking to just a small number of scales really limits your playing. Not just for improvisation but for playing written pieces too.
Due to technology and DAWs these days, many musicians are able to just play in one particular scale and simply transpose the key. That may be fine for production purposes however you may come unstuck in live studio session without the appropriate hardware.
I suggest you take the time to learn all 12 keys. Practice until you are able to comfortably improvise in any of the 7 chords. The first thing to start off with would be the 2-5-1 chord progression, which is the Minor, Dominant and then the Major. Once you get used to these then you can start expanding to other chords.