Veeam B&R failing - Veeam say it's the MX

[JonP]

Hello team,

We are investigating an issue in our backup environment. We utilise Veeam and a job is supposed to run every night to copy our backups into PeaSoup cloud. This copy job has recently started failing.

We spoke with Veeam who said that our firewall is the likely root of the problem, and it needs to comply with their KB https://www.veeam.com/kb2140 but I can't see there is anything specific in that KB for us to "do" in order to fix the problem.

The backups have been working fine for months but recently started to fail with the error:

Failed to decrypt TLS data: (336130329) error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac Unable to retrieve next block transmission command. Number of already processed blocks: [2810]. Failed to download disk '<our server name>.vhdx'. Reconnectable protocol device was closed. Failed to upload disk '<our server name>.vhdx' Agent failed to process method {DataTransfer.SyncDisk}.

Our MSP has logged a ticket with Meraki support, but based upon my recent interactions with them I don't hold out much hope that they'll even understand the question.

Has anyone else seen anything like this before? I assume other people out there are using Veeam to copy jobs to an off-site repository!

J 🙂

[ww]

So you would need to disable  IPS/AMP (and maybe even L7 firewall &content filtering) in the MX and then test again.

You could also look for hits on any of these features before you disable anything

[JonP]

We aren't seeing anything in the event log for this, but we disabled IDS/IPS last night and we are no longer seeing the Veeam errors. However we can't risk leaving IDS/IPS disabled permenantly.

[Brash]

Second @ww 's comments.

Check the event log for hits and disable some security features.

Also double check firewall rules if any recent changes have been made.

[JonP]

We aren't seeing anything in the event log about this issue.

[PhilipDAth]

+1 to @ww .  You could also try temporarily "white-listing" the Veeam on-premise server that is talking to the cloud.

https://documentation.meraki.com/General_Administration/Cross-Platform_Content/Blocking_and_Allowing... 

Unfortunately, Meraki has now renamed this well-known and understood industry term to the more generic "Allow".

[MrMike]

We're having the same issue here the past several months.  Tonight we are going to test temporarily disabling IPS / IDS to see if that helps.  

[AlexP]

Currently, there are a few known issues with fragments not properly getting re-assembled when IPS is in use that could be at play. Depending on the forwarding path in question, you may also have positive results if you drop your MSS/MTU on the service a bit lower than 1500, especially if captures of traffic before it hits the MX indicate there is network-layer fragmentation occurring.

[Sustrans_IT]

Hi All, 

We are seeing exactly the same issue, this time with replications to 11:11. Does anyone have a resolution that doesnt involve turning off AMP/IDS? 

Cheers 

[MrMike]

To answer your question simply, no we have not.

We changed Intrusion detection and prevention setting from prevention to detection.  This was to ensure we weren't asleep at the wheel allowing threats to pass through.  Our findings were we were getting little to no benefit from this feature (as evident from the event logs).  Thus we've left it as detect only.  

Anecdotally we also observed that MS Teams meetings became more reliable with less garbled audio and dropped connections.