Ah, okay.
I use Teamviewer in my org, but I control access via Teamviewer itself. All of my clients have Teamviewer Host which is assigned to a management group with easy access. I just make sure all of the management accounts have 2FA. For sensitive devices, I restrict access specific hosts in Teamviewer settings.
Otherwise, I think a more elegant solution would be to set Teamviewer to allow only LAN connections, and use a site-to-site VPN. I'd imagine you have a LOT of remote sites however.