Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Register or Sign in
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
About PaulLobaugh
PaulLobaugh
Comes here often
Member since Jun 20, 2023
Jun 27 2023
Community Record
5
Posts
0
Kudos
0
Solutions
Badges
Jun 20 2023
6:45 PM
I've mentioned the problem to our account rep but they haven't followed up and I haven't pushed them on it really. The issue completely disappears if we remove the firewall from service and temporarily just connect the server(s) up directly to the internet sans firewall (which is terrifying, but was a worthwhile test), which makes it look like the cause is not on their end.
... View more
Jun 20 2023
5:00 PM
Correct, I can replicate it every time. There doesn't seem to be any pattern, sometimes you'll get 2-3 occurrences within a 20 minute span. Sometimes you'll go almost 15 minutes before you get 1. Sometimes it'll be 1 second of drop, sometimes (more rarely) it'll be 2 - 4 .The same exact thing occurs day or night, at our busiest and slowest parts of the day. The only reliable statement is that at least 1 second of drop happens at least once every 15 minutes at the bare minimum. That long running query test I keep mentioning uses a query that takes 16 minutes to complete and it has never successfully completed, making it a solid test case. If the issue was only affecting ERP clients and RDP sessions it would be harder to reproduce, as those are a bit more resilient and seem to only die with the 3-4 second drops. I had read that CPU spikes could potentially cause the issue but if you're not seeing that then that's one less possibility. We did quite a bit of testing yesterday with restarts and swapping the 75 for a 67 and whatnot. That started at about 5:15PM PST and ended towards 8PM PST. We've done no such resets today, though I did disable some ethernet power saving stuff on the NIC for the SQL Server that temporarily restarted it. Other than that, nothing else. I can't recall exactly when that was today...sometime between 12-3 PM PST.
... View more
Jun 20 2023
4:36 PM
Wow thanks for hopping in on this! We did note that on the prior firewall the MTU size was set to 1372 so we gave that a shot, no success with that unfortunately. I've done a variety of ping tests on that server (the SQL Server), including to: A local desktop at the same site My desktop at a different site The IP address of the remote SQL Server we run queries against over the internet The latter 2 experience that 1-3 seconds of dropped pings but not at the same time. For example, I will see a drop in the ping test from the SQL Server to my desktop at a different site (over an SD-WAN config) but NOT a drop in ping from the SQL Server to the remote server IP address as they're both running on the screen side by side. Strangely, in that scenario, the query I had mentioned (in which we run a long-running query from the SQL Server to the remote server over the internet) will fail despite the fact that the ping test to the remote server IP address never saw loss. That's been continuously confusing to me. Any additional data I can get for you by running similar tests? As for the topology, in an effort to rule out interference from some weird internal networking issues, we plugged the 3 servers I mentioned in the OP directly into the MX75. The 4th active port goes to a switch that then connects up to the rest of our internal networking architecture. We just wanted to isolate the 3 servers we used for testing as much as possible, so we often disconnected the rest of the internal network when running tests, just to be sure it wasn't playing a part.
... View more
Jun 20 2023
3:18 PM
We did already put a brand new MX67 in it's place to test, but we didn't take one of the 67's from a different site. That might be a good sanity test, but I'm not overly hopeful it'd fix it. Still, I can add that to the list to try. I'm not sure if we set up the MX75 as just a router...I had someone do the basic configuration for me on that test actually, is there some more basic "don't do anything to this traffic" setting that I'm unaware of that we could set?
... View more
Jun 20 2023
2:52 PM
We recently installed an MX75 at a site as the final step in an infrastructure upgrade project (several other sites got MX67's), and we immediately started to have connectivity issues with accessing SQL based apps (MS Access and an ERP client) and RDP connections from the other sites. This MX75 site has 3 servers we've focused on for problem solving. A SQL Server (ERP), a DHCP/File server (Also has the MS Access databases) and a terminal server (for RDP). After running some ping tests from one of the servers at the MX75 site, it looks like once every 10ish minutes or so (but it varies a lot) we get a "no response detected" line in the ping test. Sometimes it lasts only 1 second, and normally the RDP and SQL apps are fine. If that interruption instead lasts 3 seconds or more, the ERP/MS Access client will drop or the RDP connection will drop out. On our SQL Server we also run queries on a linked server over the internet and a query will fail if there's even 1 second of interruption, so it makes for a good sensitive test for the issue. We've tried: Plugging these servers directly into the MX75 to avoid any potential internal switching problems Disabling IPS/IDS and AMP Removing the MX75 from the site to site tunnels and just running that query to test. Changing from a fiber handoff to a copper handoff on the ISP side Removing all traffic shaping/firewall rules Upgraded the firmware to 18.107.2 We RMA'd the MX75 and tried a new one We factory reset the MX75, set up a very very basic config just to pass traffic, and still had the issue Fully replaced the MX75 with an MX67, still had the issue If we use the old firewall (or temporarily test one of the servers with no firewall) we do not have this issue. We've asked Meraki support for help, but they say the issue is on our end and the servers themselves are somehow causing the problem (I've done ping tests between regular ol' computers across sites and see the same thing, so I don't think this is true). We have no idea what else to try, has anyone experienced anything similar?
... View more
© 2024 Cisco Systems, Inc.
