Exclude some anchor link from smooth scrool when using function.php solution

Hello,
I've applied the following solution to have the smooth scroll behavior applied to all anchor links:

add_filter( 'generate_smooth_scroll_elements', function( $elements ) {
$elements[] = 'a[href*="#"]';

return $elements;
} );

This will generate some problems with navigation links in slider plugins (i.e. Smart Slider 3) as when the next/before navigation button is clicked, the page will in some case scroll to the top of the page.

I tried to exclude all the links not containing a particular string (i.e. #top), so I did the following:

add_filter( 'generate_smooth_scroll_elements', function( $elements ) {
$elements[] = 'a[href*="#"]:not(a[href*="#top"])';

return $elements;
} );

But this is not working (I assume it's because the href*="#"] condition is true for every "#top" anchor link.

What is the correct way to exclude some anchor links from the function? Could you please make 2 examples where:
- on the first example: all href*="#" links are targeted, excluding links containing "#top"
- on the second: all href*="#" links are targeted, excluding links containing "#top" AND "#middle"

Thank you