This usually happens when you have the homepage URL in the custom link.
Ideally, you only want the hashtag with the ID.
However, that won’t work while on your contact page, as it will look for the IDs on that page.
A plugin like this might help: https://wordpress.org/plugins/menu-swapper/
That way you can build a menu specific to your homepage without the full URLs, and keep the other menu for any additional pages.