I made a change to a port and it took 17 minutes for it to take effect. Now the port is back up, however the port is still showing as black even though the device on it is reachable. All last week we were averaging 8 minutes per change. It seems to be getting worse and worse. I think direct console access would be a godsend at this point. (By console I mean a telnet or ssh session where you can make changes in real time. If you don't know what telnet or ssh is, ignorance is bliss.)
How do you check to see which shard you are on?
Either you have a shard fault or your devices are having trouble talking to the shard (such as local firewall rules, ISP connectivity issue, etc).
So I am on shard n180. When I was on the phone with support yesterday it took 25 minutes per change. We tried several networks to confirm it wasn't specific to a site. We have the same firewall's everywhere and normally these changes take less than a minute. Support concluded that there is definitely something wrong in the back end and she kicked the ticket up. I will keep you guys informed.
Not easily as it is at a remote site.
We had a similar issue earlier this summer. While support never said they found anything wrong on our shard, it did stop after we called them.