Yes, that would be a good idea.
Basically, Fonts loaded by GP are from the ones you set in Appearance > Customize > Typography. Other fonts loaded are most likely not from GP.
If you have GenerateBlocks, and you load a font for any GB Block, you need to unload it too to be sure.