@Brash   I like the idea and understand what you're saying but i'm going to need to research it and see if I can work it out.     ... View more

Bit of an update.   It looks like the MTU was part of the problem - ESX has a max supported MTU of 9000, and the E1000 warning has gone, which was expected. The performance is starting to get back to normal on the postgres db. I still want to resolve the latency warnings obviously. ... View more

Morning Everyone.    We did a couple of items last night to try and resolve. I'll explain those then give you some answers to your questions, thanks for taking an interest and your assistance!   So, we changed the MTU size on the Meraki to 9000 - this was the lesser of two evils. We also realised there is a very specific way the SAN should be connected to the stacked switches, one of 2 of the 8 cables needed a swap.    It didn't sort it though. Continuing reviewing everything, we saw the gateway ip for the vnic adapters was incorrect (it's a valid gateway ip for our network, but it's on a different vlan) would this make much difference you think?    Anyway - questions.    @Brash Brash - ISCSI data is running over L2.  Yes - I was just noting it, but wince last night the MTU was changed across the network to 9000.    @PhilipDAth No, not in the slightest. The peaks are overnight, when the backups are running and we're not seeing more than 450 Mb/s and office hours peak is about 90Mb/s.   we'll take a look at QoS thanks for the suggestion - I concur, it's worth checking but i doubt it's that cause - you never know sometimes!   duplexs are all auto/auto and 1000/full.    Interestingly, when checking the port, it shows you the device connected to it, which is one of the nics on the SAN, but doesn't resolve the ip address right away,. it has a 169.254 address, then resolves the ip.   latency warnings after making the changes...   2021-09-21T23:09:51.551Z cpu3:2218975)WARNING: SVM: 5761: scsi0:1 VMX took 1145 msecs to send copy bitmap for offset 154618822656. This is greater than expected latency. If this is a vvol disk, check with array latency. 2021-09-21T23:13:51.162Z cpu14:2218975)WARNING: SVM: 5761: scsi0:1 VMX took 1780 msecs to send copy bitmap for offset 204010946560. This is greater than expected latency. If this is a vvol disk, check with array latency. 2021-09-21T23:16:20.894Z cpu1:2218975)WARNING: SVM: 5761: scsi0:1 VMX took 1449 msecs to send copy bitmap for offset 242665652224. This is greater than expected latency. If this is a vvol disk, check with array latency. 2021-09-22T00:05:29.144Z cpu16:2097955)WARNING: vmnic5:TxI:461/461 2021-09-22T00:45:17.018Z cpu2:2097206)WARNING: ScsiDeviceIO: 1564: Device eui.560509d099ed7da86c9ce900f866ee86 performance has deteriorated. I/O latency increased from average value of 9011 microseconds to 180891 microseconds. 2021-09-22T02:04:49.023Z cpu16:2098107)WARNING: ScsiDeviceIO: 1564: Device eui.85c313515621eb466c9ce900f866ee86 performance has deteriorated. I/O latency increased from average value of 2423 microseconds to 59700 microseconds. 2021-09-22T05:52:54.503Z cpu17:2097959)WARNING: vmnic7:TxI:219/219 2021-09-22T06:04:24.672Z cpu17:2097959)WARNING: vmnic7:TxI:445/440 2021-09-22T06:14:44.691Z cpu19:2097955)WARNING: vmnic5:TxI:33/33     ... View more

Thanks for the reply -  very interesting regarding .local i'll add it to the list of changes.   You were correct also, we had to figure out why the lookup request for dns was timing out..;   and we did, about an hour ago.    There was a firewall rule blocking tcp and icmp on those vlans. serves no purpose, it was configured to stop those vlans from accessing the internet - it had the unexpected affect of blocking traffic internally.   as cmr asked about the firewall, i marked his as the solution, but i appreciate your response, thank you. ... View more

Thanks @David-H  We got a couple of MT12s (as part of a large Meraki switch upgrade) but didn’t realise they were WiFi 6 only. We have mr33s    upgraded them to v28 and had zero issues. Nice job! ... View more

Could you please tell me where I can check the firewall rules on the 210s? (My colleague is the network guy really)    its worth noting we have mx64s for the sd-wan and firewall, which are connected to stacked ms420s and all the switches are connected to that. We’ve discounted anything else than the ms210s as it’s only the VMware management network having any issues. The catalyst we replaced as I said we copied everything useful (ip addresses vlan setups etc) to minimise the number of changes we were making for troubleshooting purposes. ... View more

Hi @cmr    I’ll try to answer your questions best I can.  the hosts are connected two the stacked 210s 2 into to each for redundancy. The iscsi for the guests are also connected the same way.   management network is vlan9 server (vms) are vlan2   iscsi is vlan 8 if memory serves.   the Cisco setup replaced was catalyst 29xx switches, we took a copy of the config and basically copied it.  when we look at nic0 and the vswitch, we can see both the vcsa and one of the esx hosts, but the vcsa just will not play ball, at all. You can only login using the local admin account on the webgui and it’s very slow to login, it’s timing out basically, nothing is shown in vcsa now, not even the disconnected hosts.   we’ve been over the network setup with a fine tooth comb (me and 2 network guys) and we cannot see where the problem lies. The client and server vms seem fine. no issues with the servers or client networks. Suspecting it’s a bit of weirdness with vcsa I started diving into the vpxa logs,    2021-09-19T12:25:27Z lwsmd: [lsass] Transitioning domain 'Xxxx.local' to ONLINE state 2021-09-19T12:25:47Z lwsmd: [netlogon] DNS lookup for '_ldap._tcp.dc._msdc xxxx.local' failed with errno 110 (Connection timed out), h_errno = 2 (Host name lookup failure) 2021-09-19T12:25:47Z lwsmd: [lsass] Could not transition domain 'Xxxxx.local' to ONLINE state. Error 9502 and  2021-09-19T03:49:16.942Z cpu2:2107133)WARNING: E1000: 4325: End of the next rx packet lots of entries for both         ... View more

@Inderdeep    at least the branch can use the remaining bandwidth for cloud services and the 100Mb SD-WAN link can be more managed for now for our needs, our licence is up in 2022 so I will go back to market then.  thanks for your help. ... View more

@ww  that’s a great suggestion I’ll do that tomorrow or Monday. Thanks. ... View more

Thanks @Inderdeep I’m aware. I’ve been reviewing the documentation.   as you explained in the other thread. The mx64s have sd-wan speed of 100Mbps I have fibre links of 200Mbps, Which isn’t exactly super fast by today’s standards, the mx64 in all other areas is sufficient for our needs but to utilise all the bandwidth available I have to go to a higher model. I think 5 years ago it was fine but the range could do with a bump in capacity across the board do t you think? ... View more

    Any chance Cisco will increase the speed of the MXs across the board via firmware update? 100MB isn’t exactly good in terms of speed these days a bit old hat don’t you think?    appreciate there needs to be differences between models but come on .. ... View more

Inderdeep.   Bottleneck is network.    We run a veeam proxy at both sites as per best practice. ... View more

right ok...    bad practice I know, but checking everything (we have no traffic shaping) we're getting around half of the speed on the CBT backup in veeam.      At least now I know the target we should be after.    Ok so the branch mx is connected to a MS switch. I can see the duplex is auto 1Gbs. looks fine. The source mx connected to a catalyst, that's also 1Gbps full duplex. I don't think it's a duplexing issue but can see why you suggested it.    is there anything else I can check?        ... View more

Hi Thanks for responding.   it was 5MBs not 5Mbps sorry I changed the nomenclature.    i'll take a look but they're all set to auto-negotiate.   is 100 the max pipe we can expect on SD-WAN on MX64?  ... View more