To anybody else that runs into this problem, the simplest way to fix it is just upload the exact size image you want displayed and leave it at 100% (don’t scale the size in WP).
It means you have to create and manage multiple images but apparently the functionality to resize isn’t that good.