So what is an embedded network? I found it pretty hard to find a good name for this but I've come across them many times and created some myself. A good example here is a portable video rack. You drag a rack case around with video and network equipment and you need to connect it to the network of the venue to stream to the internet. The devices in the rack need to communicate with each other but you don't want to reconfigure their addresses every time you move to another venue because it happens to use another subnet.

The solution to this is easy! Just add a small router in the rack so you have a consistent subnet inside the rack and the NAT isolates you from the changing IP addresses outside your small network. Your rack has 10.0.0.0/24 addresses because they are easy to remember and the router gets an IP from the venue using DHCP.

This works perfectly fine until the public network in the venue also is on a 10.0.0.0/24 network. The router suddenly has the same subnet on both interfaces and addresses in your rack start conflicting with other hardware in the venue.

This is the point where I see people often picking weird subnets for portable equipment. "What are the chances the venue has 172.16.42.0/24?, or 10.11.12.0/24"? And sure this works, until you get a conflict on those because humans are simply not that great at picking random numbers. It is not actually neccesary to have network seperation by random chance, you just need router features beyond common consumer routers.

The IPv6 solution

The most official solution for this is IPv6 of course. If your private network has IPv6 internally you can just address every device by it's link local address. Due to having a router splitting the network segment between your rack and the public network you know that all the link local addresses are always your own devices.

You don't even need to have a DHCP server in your local net anymore. Every IPv6 device just gets it's own fe80:: address and you'd use network discovery protocols to find the address of the device to talk to. For example avahi can be used to have name resolution for these addresses.

In this case the router should send out route advertisements so the devices still get internet connectivity without configuring any addresses. This basically causes the devices in the rack to set the link local address of the router as the gateway address.

The massive problem with this solution is that support for IPv6 on devices that are not general-purpose computers is usually non-existent. I have a portable rack right here with a Behringer X-Air audio mixer in it and this is just modern enough to allow you to use DHCP to get an address, no IPv6 support in here whatsoever. If I grab a random video mixer here it will most likely also not support IPv6, some of them don't even do DHCP yet.

Technically the same solution is also possible on an IPv4 network. You've probably encountered it before: Getting an 169.254.0.0/16 address on your computer when you selected DHCP but the network did not have DHCP server running. This is an APIPA address and is the IPv4 equivalent of an fe80:: IPv6 address. Sadly this doesn't work that well for this though since with APIPA networking there is no way to have a gateway so you'll never have internet connectivity. Also if your machine does get a valid IP address so you can connect to the internet the APIPA address will normally be dropped on that interface since it's no longer needed. And just like IPv6 support in embedded hardware, it's most likely not implemented.

A neat trick if you are dealing with an IPv6 network though for finding the addresses of the hosts in your segment:

$ ping -c 2 -I enp3s0 ff02::1
... responses from multiple devices here ...
$ ip -6 neigh show dev enp3s0
fe80::cc55:13ff:fe9b:82a2 lladdr ce:55:13:9b:82:a2 REACHABLE 
fe80::df8c:9c93:6850:7132 lladdr b0:35:9f:5a:b2:49 REACHABLE 
fe80::ac2b:55ff:fe62:a34 lladdr ae:2b:55:63:0a:34 REACHABLE 
fe80::a0b8:78ff:fe5c:8229 lladdr a2:b8:78:5a:82:29 REACHABLE 
fe80::5660:9ff:fee1:cb4b lladdr 54:60:09:e1:cb:4b REACHABLE 
fe80::f61e:57ff:fe61:9a58 lladdr f4:1e:57:91:9a:58 router REACHABLE

This sends a broadcast ping to your local segment and then you can read out the list of neighbors known on that interface, which also conveniently tells you which device is advertising routes. The ff02::1 is one of the "notable ipv6 multicast addresses" for which there's a convenient list on Wikipedia.

A more generic solution

So instead of having more exotic addressing inside your embedded network there's also a solution that contains all the weirdness to only the router. It is possible to configure a router so that both networks it connects to have the same subnet by having separate routing tables for the interfaces.

This means your internal network can be 10.0.0.0/24 and the venue network can be 10.0.0.0/24 and it all just works. The video mixer in the rack can have the 10.0.0.4 address and there can be a 10.0.0.4 address in the venue network and nothing will conflict. This comes with a tradeoff of course and in this case is that you no longer can reach devices on the venue network, which shouldn't be a problem if you're only connected there for internet connectivity.

The magic solution here is VRFs. Normally the router has a single routing table that defines where traffic is supposed to go to based on the destination address. The routing table is the main issue in this situation because having the same subnets on the inside and outside would mean you just have two routes in the table saying 10.0.0.0/24 should go to the LAN interface and the WAN interface

$ ip route
default via 10.0.0.1 dev enp1s0 proto dhcp src 10.0.0.33 metric 100
10.0.0.0/24 dev enp1s0 proto kernel scope link src 10.0.0.33 metric 100
10.0.0.0/24 dev enp2s0 proto kernel scope link src 10.0.0.254 metric 100

This is basically telling the router that the same 10.0.0.0/24 network is reachable over both interfaces. The only way to tell the kernel that these are actually seperate is by having a VRF here.

With a VRF you can have a completely isolated routing table which you can link to your network interfaces. Traffic from those interfaces are now matched to that table instead of the main one for the routing decisions.

For my test setup I've used a Mikrotik device since it allows me to configure VRFs as I didn't have a Linux box with enough interfaces on it to make an easy test setup. But this also shows that this setup isn't so esoteric that you need to have a full PC doing the routing for it. I'm using a Mikrotik hAP mini which is a $15 3-port router which is probably the cheapest device available that just has this feature available. For testing I have the WAN port (ether1) connected to my home network and the other two ports are connected to laptops which are supposed to communicate together but still be isolated from my home network.

The basic config is already set on this device making it do NAT between the two LAN ports and the WAN port so the only thing I need to do is break it by setting the internal subnet to the same one as my home network and then configuring the VRF related things to make it actually route properly. 

# bridge-internal has the two ports for the internal devices
/interface bridge
add name=bridge-internal
/interface bridge port
add bridge=bridge-internal interface=ether2
add bridge=bridge-internal interface=ether3

# Give the router an address on the internal network
/ip address
add address=10.0.0.1/24 interface=bridge-internal

# Get an address for the WAN interface by DHCP
/ip dhcp-client
add interface=ether1

# Create the rack vrf for the bridge
/ip vrf
add interfaces=bridge-internal name=rack

# In the rack VRF tell it the default gateway is on the main VRF at the
# 10.0.0.254 address. The gateway IP _can_ be the same as the 10.0.0.1 one
# used for this router but kept seperate here to make the config easier to
# understand
/ip route
add gateway=10.0.0.254@main routing-table=rack

# The iptables rule for regular 'ol NAT for all traffic going out the WAN port
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1

# Use connection tracking to mark traffic from the internal VRF so the return
# traffic can be delivered in the same VRF. The first rule gives the
# connection the "rack" mark in the connection tracking table. The second
# rule matches traffic from the WAN port that match a connection that has the
# "rack" mark and then set the routing table to use to "rack" which is
# the VRF. This is in the prerouting chain so these rules are executed
# before anything with the routing tables are done.
/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new in-interface=bridge-internal new-connection-mark=rack
add action=mark-routing chain=prerouting connection-mark=rack in-interface=ether1 new-routing-mark=rack

And that's all that's needed. Traffic that's handled by the rack VRF get's delivered on the interfaces linked to that VRF. The traffic for the main routing table goes out the WAN port.

To make traffic destined for the internet cross the barrier between routing tables a route is added in the rack VRF that sends traffic destined to the internet to the default gateway address in the main routing table. Which will run it through the firewall rules and apply the standard NAT rule that rewrites the source address to the one of the router. There is also a connection mark added in the connection tracker that tells iptables that this connection was originated from the rack VRF.

When a packet returns from the internet the iptables connection tracker will match it up to the connection that has the connection-mark set on it. This will trigger the other mangle rule that will make sure that this packed is proccessed by the rack VRF so it gets sent back to the device in the rack that started the connection.

This all together allows the devices in the rack to have internet connectivity just like it's a regular NATted connection. It is impossible to reach other devices in my home network through it though because the routing table for the rack will just send traffic destined for 10.0.0.0/24 back out the same interface instead of the WAN port. The same thing happens when a device on my home network tries to use the router as a gateway to reach a device in the rack.

These few rules configured on the router will allow you to use your favorite subnet inside the rack without any worry about conflicts. I've seen the regular NAT setups in this situation for portable live production gear, for industrial monitoring gear and have once set this up for an ancient CNC machine that did not allow changing the subnet address.

With a few translations this should also apply to apply to the VRF commands in Linux routers and I assume VRFs are also exposed on other major router brands