Background: recently upgraded MX100 firewall pair with MX105s. The firewall are set inline between my internet firewalls (cisco ASAs, which also act as AnyConnect VPN termination points) and core switches, setup as passthrough VPN concentrators. Intrusion Prevention is enabled, AMP is enabled, so they are analyzing all internet incoming/outbound traffic. Remote users are utilizing MX67 firewalls. Some of the users have gigabit internet (AT&T fiber, Google fiber).
Observations: The MX100s (during peak traffic times) would appear to be bottlenecked due to high CPU utilization (80-100%), causing significant slowness in remote user traffic. The CPU utilization on the MX105s doesn't even get to 20% yet my users still complain of significantly reduced performance compared to using AnyConnect. I have personally seen this from a specific user, in which image download times on AnyConnect are 50% of times when using Meraki VPN.
Setup: The MX105s default route is set to the internet firewalls, so what I think is happening is that incoming remote user traffic (encrypted) comes in through the internet firewalls, hit the MX105, is decrypted, then routed to the internet firewall, which then bounces the traffic back to the core routers, and then on to the data center servers. This seems pretty inefficient.
I would think that these newer firewalls would process the data better/faster, and that much of the slowdowns experienced before should go away, but it doesn't appear to be that way. Any ideas?