Ricoh Scan to Email Fails After Installing Meraki MX67

[MarcAEC]

I'm part of a team involved in a WAN upgrade project.  We've been replacing EOL Cisco routers with Meraki MX67s.  Auto-VPN is used to connect the sites to a central hub.  There is an MX100 at the hub. 

At multiple locations, immediately after installing the MX67, Ricoh scan to email started to fail. However, at some other locations, it continued to work.  Technicians have done an A:B comparison of the Ricoh settings between the sites that work and the sites that don't and haven't found any obvious differences. 

The Ricoh scan settings are configured to use an internal email server located at the hub site connecting via plain non-SSL SMTP over port 25.  The data only has to travel over the site-to-site VPN tunnel.  It never touches the internet. 

The Ricoh logs have generic errors that you'll find years worth of posts about in an internet search.  Many of those have to do with authentication problems with Office 365 or Google G-Suite.  That's not the case here.  The email server is happy to accept unencrypted, unauthenticated communication. 

Here is an example from the Ricoh log:

#[dcs_nas(104)]20/09/21 09:31:04 SMTPC: connection closed. (501) ERR:
#[dcs_nas(104)]20/09/21 09:31:04 SMTPC: connection closed. (701) ERR:
#[dcs_nas(104)]20/09/21 09:31:04 SMTPC: connection closed. (801) ERR: 

In the Exchange SmtpReceive log, we'll see something like the following:

2020-09-21T14:23:21.484Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,0,10.10.1.103:25,192.168.2.99:61118,+,,
2020-09-21T14:23:21.484Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,1,10.10.1.103:25,192.168.2.99:61118,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2020-09-21T14:23:21.484Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,2,10.10.1.103:25,192.168.2.99:61118,>,"220 EXCHANGE1.domain.local Microsoft ESMTP MAIL Service ready at Mon, 21 Sep 2020 10:23:21 -0400",
2020-09-21T14:23:21.531Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,3,10.10.1.103:25,192.168.2.99:61118,<,HELO RNP5838793490DB,
2020-09-21T14:23:21.531Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,4,10.10.1.103:25,192.168.2.99:61118,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2020-09-21T14:23:21.531Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,5,10.10.1.103:25,192.168.2.99:61118,>,250 EXCHANGE1.domain.local Hello [192.168.2.99],
2020-09-21T14:23:21.562Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,6,10.10.1.103:25,192.168.2.99:61118,<,MAIL FROM:<donotreply@domain.com>,
2020-09-21T14:23:21.562Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,7,10.10.1.103:25,192.168.2.99:61118,*,SMTPSubmit SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions
2020-09-21T14:23:21.562Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,8,10.10.1.103:25,192.168.2.99:61118,*,08D85DF4C6DA20F2;2020-09-21T14:23:21.484Z;1,receiving message
2020-09-21T14:23:21.562Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,9,10.10.1.103:25,192.168.2.99:61118,>,250 2.1.0 Sender OK,
2020-09-21T14:23:21.625Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,10,10.10.1.103:25,192.168.2.99:61118,<,RCPT TO:<john.doe@domain.com>,
2020-09-21T14:23:21.625Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,11,10.10.1.103:25,192.168.2.99:61118,>,250 2.1.5 Recipient OK,
2020-09-21T14:23:21.671Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,12,10.10.1.103:25,192.168.2.99:61118,<,DATA,
2020-09-21T14:23:21.671Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,13,10.10.1.103:25,192.168.2.99:61118,>,354 Start mail input; end with <CRLF>.<CRLF>,
2020-09-21T14:23:21.718Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,14,10.10.1.103:25,192.168.2.99:61118,*,,Proxy destination(s) obtained from OnProxyInboundMessage event
2020-09-21T14:23:21.875Z,EXCHANGE1\Ricoh Test EXCHANGE1,08D85DF4C6DA20F2,15,10.10.1.103:25,192.168.2.99:61118,-,,Remote(ConnectionReset)

We see that communication starts, the recipient and sender are accepted, and then the connection resets after the Ricoh device is prompted for the data.

In the Exchange SmtpSend log, the following is found:

2020-09-21T14:23:21.718Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,223,10.10.1.103:16585,10.10.1.104:2525,>,XPROXYFROM SID=08D85DF4C6DA20F2 IP=192.168.2.99 PORT=61118 DOMAIN=RNP5838793490DB SEQNUM=1 PERMS=1073 AUTHsrc=Anonymous,
2020-09-21T14:23:21.718Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,224,10.10.1.103:16585,10.10.1.104:2525,<,250 XProxyFrom accepted,
2020-09-21T14:23:21.718Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,225,10.10.1.103:16585,10.10.1.104:2525,*,,sending message with RecordId 0 and InternetMessageId <20200921093103GH.DCSML-S000010000.5838793490DB@192.168.2.99>
2020-09-21T14:23:21.718Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,226,10.10.1.103:16585,10.10.1.104:2525,>,MAIL FROM:<donotreply@domain.com> SIZE=0 AUTH=<>,
2020-09-21T14:23:21.718Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,227,10.10.1.103:16585,10.10.1.104:2525,>,RCPT TO:<john.doe@domain.com>,
2020-09-21T14:23:21.718Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,228,10.10.1.103:16585,10.10.1.104:2525,<,250 2.1.0 Sender OK,
2020-09-21T14:23:21.718Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,229,10.10.1.103:16585,10.10.1.104:2525,<,250 2.1.5 Recipient OK,
2020-09-21T14:23:21.718Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,230,10.10.1.103:16585,10.10.1.104:2525,>,DATA,
2020-09-21T14:23:21.718Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,231,10.10.1.103:16585,10.10.1.104:2525,<,354 Start mail input; end with <CRLF>.<CRLF>,
2020-09-21T14:23:21.875Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,232,10.10.1.103:16585,10.10.1.104:2525,*,,Proxy layer started discarding data. Acking message as failed.
2020-09-21T14:23:21.875Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20B3,233,10.10.1.103:16585,10.10.1.104:2525,-,,Local
2020-09-21T14:23:22.890Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20E9,122,10.10.1.103:16982,10.10.1.103:2525,*,,Proxying inbound session with session id 08D85DF4C6DA20F4
2020-09-21T14:23:22.890Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20E9,123,10.10.1.103:16982,10.10.1.103:2525,>,RSET,
2020-09-21T14:23:22.890Z,Inbound Proxy Internal Send Connector,08D85DF4C6DA20E9,124,10.10.1.103:16982,10.10.1.103:2525,<,250 2.0.0 Resetting,

The line of interest there is "Proxy layer started discarding data. Acking message as failed." 

So the Exchange server is failing the message after it receives what it considers junk.  Since this is all internal traffic, Meraki Advanced Malware Protection (AMP) shouldn't be involved, but could one of the Merakis be altering the traffic?  I found a suggestion online to disable "SMTP inspection" if using a Cisco ASA.  Obviously, nothing like that is present in the Meraki cloud controller, but is there a similar feature running on a Meraki MX?

I did try opening a support case with Meraki, and the support rep did packet logging while an end user ran some scans.  The technician didn't see anything of note. 

But the fact is that at some locations before Meraki MX67, scan to email worked.  And after it didn't.  Any suggestions you might have would be greatly appreciated.

[PhilipDAth]

I'm guessing you have threat protection enabled.

See if there are any SMTP signatures that appear in the drop down box for whitelisting.

https://documentation.meraki.com/MX/Content_Filtering_and_Threat_Protection/Threat_Protection 

[MarcAEC]

I don't see any SMTP rules in the whitelist dropdown.  I also checked the Security Center log and didn't find mention of the Ricoh IPs.

[PhilipDAth]

Try turning off AMP and Threat protection for a short while and see if that changes the behaviour. 

[cmr]

@MarcAEC I'd agree with @PhilipDAth, we have an MX based SD-WAN and Ricoh copiers with scan to email to local Exchange and don't see any issues.  However we have the Enterprise license that doesn't include IPS/AMP as our WAN links are all flavours of MPLS...

[MarcAEC]

I set AMP and IDP to disabled on both the hub and spoke MX, and the scan to email still failed with "Proxy layer started discarding data. Acking message as failed." on the Exchange server.

[PhilipDAth]

Okay, it's not AMP or Threat Protection, so you can turn those back on.

Is this using AutoVPN over the Internet or some kind of MPLS WAN between the copier and the Exchange server?

[MarcAEC]

The connections are via AutoVPN.  Some of the circuits may be MPLS, but when they are there is still an AutoVPN tunnel on top of it.

[cmr]

@MarcAEC Is there a correlation between the sitrs that work and the sites that are/aren't over MPLS?

[MarcAEC]

Not that I can tell.  And it gets weirder.  Templates are being used as much as possible.  There are 3 sites on one template communicating that scan to email is not working, and 1 on the same template that is scanning fine every day. 

[KarstenI]

As a test, can you change the printer from plain SMTP to SUBMISSION (tcp/587 with username/password)?

[MarcAEC · Accepted solution]

It appears that the problem has been resolved.  All of the Ricoh printers are configured with static IPs in their settings.  In Configuration > IPv4 settings, there is an option for Domain Name.  The Ricohs that weren't working either had it set to DHCP or had it set to Specfiy with a blank value.  After I entered a value and saved, scan to email started working.  Some of these printers had been set for this like years without problems.  Part of me still doesn't believe this fixed things (because why would it?)  But it's been several days and none of the users have reported problems since they were informed scan to email was working.

Thanks for all the suggestions and trying to help me resolve this.  I appreciate it.  This was a great experience.

[cmr]

@MarcAEC glad to know you fixed it, out of interest, with the ones set to DHCP, were they getting their IP address from DHCP and if so do you have the domain option set on the DHCP server?  I think it might actually be the fix as we always have that field set correctly and haven't had any issues with ours, so it does make some sense 😀

[MarcAEC]

Meraki DHCP is set to provide the domain suffix (option 15).  But none of the Ricoh printers are using DHCP.  I also noticed that some of the printers that were successfully scanning to email didn't have the correct value in the field.  It seems just being blank was a problem.