Thanks Elvin. The issue when you select the anchor links in the menu from other page on the site. But I was able to solve it and other issues with the plugin: Page scroll to id
For anyone else who reads this:
It has options to:
Verify target position and readjust scrolling (if necessary), after scrolling animation is complete
Extend target position verification and scrolling adjustment for lazy-load scripts (images, iframes etc.) and changes in document’s length
Selecting both of those options work like a charm. It also has the offset for sticky menus.