Hi there,
Just ran your site through Pingdom. The actual theme doesn’t seem to be taking very long to load (as you said, it’s super small).
However, 3.6MB (what the site is now) is still pretty big. Have you optimized all of your images?
Google Fonts also seem to be taking a while to load. Perhaps try the System Stack font? Or host your fonts locally: https://docs.generatepress.com/article/adding-local-fonts/
Let me know if this helps or not 🙂