I always hated and never understood network which did not make getting this to work Easier.
Luckily i already have some experience from setting up a VPN to my home with my previous shitty provider with no Public ip (behind a CGNAT).

If you want to see how it works instead of the process of me getting to this point go here:
Site-to-site vpn

Attempt One (unsuccessful):

setup:
At this point we both had our stuff behind a double nat:
Home Rotuer --> Virtualised PFSENSE --> Our vms n stuff
So we decided to setup our VPN using the PFSENSE UI.
We decided to go for IPSEC which turned out to be how should i say... Not suited for us.

We both added the Phase 1 Key exchange thingy and configured the same settings.
Under status Ipsec we found that Phase one is connected.
We then after some debugging also got Phase 2 to work.
To our surprise it was very stable and even survived reboots of the Pfsense.

Dissapointment:

After running Iperf3 we found that in both directions the Speed seems to be capped at exactl 282Mbit/s. (With normal Iperf we both have above 900 Mbit/s)
We then proceeded to waste countless hours debugging:

• Set different CPU flags in Proxmox CPU configuration eventually getting pfsense to tell us that it has fast encryption (dont remember specifics)
• Increase CPU and RAM of Pfsense VM to 4vcpu and 8192Mb ram
• Set different faster encryption algorithms
• Texting someone who understands it better than us:
  • He told us that Ipsec needs specific Optimized Hardware to even achieve close to the NIC's Limit of speed.
• ....

Attempt Two (unsuccessful)

Setup:
For our second attempt we decided to use a tool that to this day has not dissapointed us: Wireguard
My friend setup a vanilla wireguard server and sent me the configuration.
His wireguard Server was behind his pfsense and my client was behind my pfsense
With this configuration we achieved about 900 Mbit/s.
However as this was setup on a regular host and not a router with a handy gui or configuration file we decided to use Pfsense for setting up Wireguard and routing/firewall Rules between other networks of ours.

I found the gui for setting up wireguard on Pfsense very confusing but to my understanding we both setup a wireguard servers and were Peers to the other site.
Interestingly here we still got only 270 Mbit/s.
At this point we were very frustrated but sure that Pfsense was the killer of our dreams.
We did some more digging and found mostly these 2 Answers:

• This is VPN overhead and is normal (ipsec)
• This is an issue with encryption features of the cpu missing or virtualization not properly passing it to the VM

Attempt Three (Successful and surprisingly simple)

Backstory:
For a long time we knew Netbird existed but just thought of it as some Paid cloudservice that could not be selfhosted therefore not being suuited for us.
As it turns out we missed one incredibly important feature:
P2P: Netbird Clients can in optimal condidions establish peer to peer connections (directly talk to each other instead of going via the Netbird server)
This essentialy means that neither do we have to selfhost the netbird server (we still did) nor does it have to have good latency to us.
We then went back and forth if that is even possible and i basically told my friend that syncthing exists and works so we decided to try it.

Setup:
First we set up our netbird instance on a new hetzner VM on my friends account.
I did not follow the exact step but its just the quick start self host guide.
We then added Clients in our home network (not behind Pfsense) and immideatly got above 900 Mbit/s with below 1ms delay.
We finally put our devices behind pfsense and did some debugging and tinkering with options to get peer to peer to work behind double nat (pfsense) achieving our desired speed.
We soon realized that we want netbird on our Router (at this point it was pfsense) but found that this was not possible on pfsense.
So my friend then basically found out that netbird can be installed on openwrt.
This is where we setup our openwrt isntances:

wget -U "" -O expand-root.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/expand_root?codeblock=0"
. ./expand-root.sh

At this point we added both of our openwrt instances as peers and setup some routes to achieve our final results:
Iperf3 test