Removing all color settings in customizer

As much as I like how GP uses the customizer, I usually prefer to put all color settings in variables in the child theme stylesheet. The stylesheet overrides the customizer settings, so everything works perfectly fine. However, I would prefer to remove all color settings so that no unnecessary styles are loaded. Is there a quick way to remove just the color settings? Or would I have to do this manually for each and every color? Alternatively, could I export the settings using "global colors" and use this to remove the color settings on other sites?