If you’re not going to use the plugin, you would need to follow the Adding Local Fonts article that David linked to.
That will allow you to:
1. Upload the font files to your server.
2. Add the @font-face CSS to your site. You can use Simple CSS or a child theme, just use the absolute URLs if you use Simple CSS.
3. Make it so the local fonts are available in the Customizer for you to select.