OSPF Advertising Issues with Palo Alto firewalls

[AlexG]

Hello!

I figured there's a small chance someone else may encounter something similar down the line, so hopefully this is helpful!

We've got a multi-site environment with a primary and secondary data center. We have Palo Alto 3050's in place at each site that run OSPF. We also have MX600's that act as our VPN hubs for all of our 900+ spokes. The MX600's were running firmware 12.24. The MX600 at our secondary data center simply acts as a failover and we generally don't run any traffic out of it. While trying to upgrade the firmware on these MX's, I encountered an issue where the routes were not being advertised after the upgrade. Considering I had to perform the upgrade around 1AM, and we have workers coming into these remote spoke sites around 3AM, it got my heart beating. 🙂

After an hour on the phone with Meraki support, I ended up reverting the firmware in defeat.

Fast forward a couple of weeks, and I had re-upgraded the secondary site's MX to 13.28. I wanted to do some testing, so I threw an older MX on our test bench and proceeded to block all traffic from it to our primary site via the Palo. Here's the real meat of this post:

When the Auto-VPN kicked in and I saw the LSA messages coming from the MX to our Palo (via packet captures), I thought "Hey, this is great!". Upon further inspection, the LSDB on the Palo did contain the two subnets from this test network, however they were not in the active routing table. "What the..."

I opened a ticket with Meraki, who once again said it couldn't be an issue with the MX since it's sending the LSA packet. I was skeptical, but in the end it wasn't actually their problem directly. I opened a ticket with our Palo support and found that from firmware 12.24 to 13.28, the code must have changed so that the OSPF Link Type needed to be broadcast versus p2p. Anyone with extensive knowledge into OSPF and Palo Alto's would have known they officially say to use broadcast when the interfaces are connected via Ethernet. We just missed that little detail. 🙂

I'll be upgrading our primary environment soon and will update the Link Type in the Palo along with it. I'll report back once I verify everything is all good.

[PhilipDAth · Accepted solution]

As farm as I am aware, Meraki has only ever support broadcast OSPF - it has never supported P2P (and there is no way to configure the link type).

I'm amazed it worked at all on 12.24 given what you have said.  I think you were lucky it was working in the first place then.  🙂

[AlexG]

Well, I guess I learned something new. Thanks, @PhilipDAth!

[AlexG]

Forgot to add the follow-up on this one. Oddly enough, when I switched the link type on the Palo over to broadcast from p2p prior to the MX upgrade, every single route dropped out of the table on the Palo. Once I finished upgrading the MX, the routes were learned again and popped into the active routing table. Could just be a bug within the Palo software, but it was interesting nonetheless.