Hi Tom,
thanks for pointing me in the right direction!
The following code got me closest to what I wanted to achieve:
// GP - HIDE CUSTOM COLOR PICKER & INPUT
add_action( 'customize_controls_print_styles', function() { ?>
<style>
.wp-picker-container .iris-square,
.wp-picker-container .iris-slider,
.wp-picker-container .wp-picker-input-wrap label,
.wp-picker-container .wp-picker-input-wrap input {display: none;}
.wp-picker-container .wp-picker-input-wrap {width: 275px; height: 0;}
.wp-picker-container .iris-picker {width: auto !important; height: auto !important; padding-bottom: 0 !important; margin-top: -16px;}
.wp-picker-container .iris-picker-inner {position: relative; height: 8px;}
</style>
<?php
} );