Human color-sense is arbitrary in that it exists for light from a particular star filtering through a particular chemical medium and illuminating stuff that is of particular interest to us and which is detectable by our particular biological tools.
Heck, even our cosmically-close sibling-species have different perceptions, like flowers and insects that use ultraviolet signals and detection. Or animals that can easily detect light polarization.
That's a pretty popular kind of star. I believe stars have surface temperature from 3K to 10K, corresponding to a pretty narrow range of black body radiation. Then it's mostly unaffected by that chemical medium, save for some scattering.
Yes, we could see some IR and UF, but that wouldn't expand that spectrum dramatically. Visible light occupies a tiny part of total EM spectrum, so expanding it arithmetically still leaves you with a tiny subset.
Humans are actually quite good at seeing light polarization. I can easily do that with my laptop screen. I've found no use for it, though.
They're not arbitrary:
* These are the ones where the energy output of Sun's black body radiation peaks, giving the scene lighting.
* These are also the ones which coincide with many excited/base energy level transitions for electrons on molecular level, giving matter a color.
Perhaps it is a quirk of our universe that these two ranges coincide. I'm not sure if that's a given.