Integrating VMware NSX with Ubiquiti UniFi via OSPF in My Homelab

In my homelab environment — which I often refer to as my personal Solution Center — I run a full stack of VMware Cloud Foundation (VCF), where NSX is a core component for network virtualization. One of the key challenges I’ve faced was how to properly connect NSX’s Application Virtual Networks (AVNs) to my physical network infrastructure, which is based on Ubiquiti UniFi components — specifically the UniFi Dream Machine (UDM).

Background & Motivation

For a long time, the UniFi Dream Machine only supported static routing, which posed limitations when working with dynamic environments like NSX. However, since version 8.1.113 of the UniFi Network Application, OSPF (Open Shortest Path First) is now supported. This update, together with firmware version 3.2.12 (or later) on the UniFi Gateway, opened the door for dynamic routing in my lab.

Note: OSPF support was introduced in UniFi Network Application 8.1.113. A compatible UniFi Gateway with firmware 3.2.12 or newer is required.

With this capability now available, I re-architected the integration between NSX and the physical network using OSPF, enabling a much more flexible and scalable routing setup.

Implementation Overview

NSX Configuration

In NSX, I configured a Tier-0 Gateway to advertise AVN routes using OSPF. The Tier-0 Gateway is backed by NSX Edge Nodes, which now participate as OSPF routers.

These Edge Nodes are connected to the physical network via VLAN-backed interfaces (Trunk or Routed VLANs depending on topology).

UniFi Configuration

On the UniFi side, I enabled OSPF in the UniFi Network Controller (v8.1.113+), defined an OSPF area (e.g., 0.0.0.0), and assigned the appropriate interfaces (VLANs) to participate in routing.

OSPF neighbor relationships were established successfully between the NSX Edge Nodes and the UniFi Gateway, allowing automatic route exchange.

Key Parameters to Match

To ensure proper adjacency, the following settings must align on both NSX and UniFi sides:

Area ID
Hello/Dead Intervals
Network Types (e.g., Broadcast or Point-to-Point)
Authentication (if used)
Packetsize

Beware of the pitfall I have fallen into:

The packet size of both OSFP interfaces, i.e. the interface of my UDM-Pro and the interface of NSX must be set to the same packet size. As I have always switched on jumbo packets in my environment, I naturally set an MTU of 9000 for the NSX interface.

The jumbo frames on my UDM-Pro were also activated

Nevertheless, the OSPF connection between my NSX environment and my Unifi Dream machine did not work. Where was the error? I had set everything up correctly, and both interfaces had Jumbo-Frames activated. Nevertheless, it didn’t work, and the troubleshooting started.
Fortunately, my first approach was to never trust a GUI. So I checked the packet size of the interfaces again using the CLI.
And yes, and behold, I got a hit directly on the UDM-Pro.

It quickly became clear that although I had activated “jumbo frames” on both interfaces, the MTU size was unfortunately not the same. This is because the NSX interface was configured with MTU 9000, which is also the maximum value that can be configured. In the Unifi GUI, I can only activate “Jumbo Frames” and cannot specify an exact MTU. In Unifi, however, this means an MTU of 9216.
Therefore, the MTU sizes of the two interfaces that do OSPF unfortunately did not harmonize, and my OSPF connection got stuck in the “exStart” status.

Therefore, my only chance, since I cannot set the MTU size of the OSPF interface on the UDM-Pro to a certain value, was to deactivate Jumbo Frames, since only the MTU 1500 of both interfaces was matched.

and because the OSFP connection has reached the status “Full”, it was of course possible to reach a VM located in any NSX segment (AVN), as my hardware router (UDM-Pro) now dynamically learns the routes to the NSX segments.

Conclusion

The addition of OSPF in the UniFi ecosystem was a game-changer for my homelab architecture. It enabled seamless integration between VMware NSX’s dynamic virtual networks and Ubiquiti’s physical infrastructure without static routes, improving both scalability and operational efficiency.

If you’re running a similar setup — especially with VMware Cloud Foundation in a lab or edge scenario — I highly recommend taking advantage of this new dynamic routing feature.