====== - Enable xternal traffic to baremetal cluster with Cilium pfsense BGP ======

  * Cilium can be configure to use BGP, which make metallb redundant. 
  * Before setting up cilium with bgp, bgp-configmap.yaml must be applied to k8s. The file content see cilium doc.
  * Setup pfsense bgp (FRR).
  * LB services should get ExtIPs which are announced to pfsense.
  * Setup portforwarding (NAT) from pfsense to ExtIP of, e.g., ingress.
  * Port forwarding to NodePort also works.

On Cloud Rootserver, where both public and private interfaces are available, private interfaces are used for BGP with pfsense LAN as GW. Policy based routing must be setup. See http://visrc.com:8000/dokuwiki/doku.php?id=linux:admin:network_routing:source_routing#config_with_netplan

<code>
admin@c1-node-01:~$ cat /etc/netplan/51-eth1.yaml
network:
   version: 2
   renderer: networkd
   ethernets:
     ens3:
       dhcp4: no
       dhcp6: no
       accept-ra: no
       addresses: [81.94.xx.xx/28, "2a01:xxx:xxxx:xx::xx/64"]
       gateway4: 81.94.xx.xx
       gateway6: 2a01:xxx:xxxx:xx::x
       nameservers:
         addresses: [1.0.0.1]
     ens6:
       dhcp4: no
       dhcp6: no
       accept-ra: no
       addresses: [195.16.xxx.111/25]
       routes:
         - to: 195.16.xxx.x/25
           via: 195.16.xxx.gw
           table: 102
         - to: 0.0.0.0/0
           via: 195.16.xxx.gw
           table: 102
       routing-policy:
         - from: 195.16.xxx.111
           table: 102
         - to: 195.16.xxx.111
           table: 102
</code>

Example

<code>
network:
  version: 2
  renderer: networkd
  ethernets:
    eth1:
     dhcp4: no
     dhcp6: no
     accept-ra: no
     addresses: [192.168.179.2/24]
     routes:
       - to: 192.168.179.0/24
         via: 192.168.179.224
         table: 102
       - to: default
         via: 192.168.179.224
         metric: 100
         table: 102
     routing-policy:
       - from: 192.168.179.2
         table: 102
       - to: 192.168.179.2
         table: 102
     nameservers:
       addresses: [8.8.8.8]
</code>


Somehow the br_filter module disappers so make the setting persistent. 

<code>
  315  vim /etc/sysctl.conf
  316  sysctl -p
  317  modprobe br_netfilter
  318  sysctl -p
  319  less /etc/modules-load.d/modules.conf
  320  echo br_netfilter > /etc/modules-load.d/br_netfilter.conf
</code>