K8S Plugins

• https://medium.com/kubernetes-tutorials/deploying-traefik-as-ingress-controller-for-your-kubernetes-cluster-b03a0672ae0c
• https://www.fosstechnix.com/install-traefik-ingress-controller-on-kubernetes-using-helm/
• https://mohitgoyal.co/2021/06/29/exposing-application-workloads-to-outside-world-using-ingress-in-kubernetes/

  kubectl port-forward -n traefik $(kubectl get pods -n traefik --selector "app.kubernetes.io/name=traefik" --output=name) --address 0.0.0.0 9000:9000
  

• https://blog.alexellis.io/ingress-for-your-local-kubernetes-cluster/
• https://particule.io/en/blog/k8s-no-cloud/
• https://getenroute.io/blog/ingress-controller-kubernetes-api-gateway-secure-service-jwt-oauth-oidc-network-namespace/

• https://www.thebookofjoel.com/bare-metal-kubernetes-ingress

Ingress Controller on host port

Ingress Controller using host port If you want to reach your services on ports 80 & 443 and do not want to have to set up and maintain an external load balancer to forward traffic to the ingress controller an option is to use the hostPort option on the ingress controller pod spec.

Using the pod spec port option hostPort: <port number> means that the pod will bind to the port on the host node that it is running on (it will not start if it is unable to). Using this in combination with deploying the pods as a daemonset will allow you to run a pod on every node that is listening to the port(s) in question on the host network. Binding to the host ports 80 and 443 like this is a commonly suggested way to run the Nginx Ingress Controller on a bare metal cluster. Doing this will mean that you can access your services with a hostname pointing to any cluster node (or all if you want to use round robin load balancing).

Advantages

If used in combination with an ingress controller deployed as a daemonset, you can route traffic to your services via any node using ports 80 & 443.

Disadvantages

Multiple pods cannot bind to the same host port on each node. If you want to load balance across the nodes, you will have to consider using an external load balancer or something like round-robin DNS load balancing. If pods are not run as a daemonset, they will only be reachable on the nodes they are running on.

• https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/8
• https://calvin.me/port-forward-web-servers-in-pfsense-2
• https://devopstales.github.io/kubernetes/k8s-metallb-bgp-pfsense/
• https://sue.eu/expose-loadbalanced-kubernetes-services-with-bgp-cilium/
• Metallb pfsense: https://geek-cookbook.funkypenguin.co.nz/kubernetes/loadbalancer/metallb/pfsense/

• Cilium can be configure to use BGP, which make metallb redundant.

  • Before setting up cilium with bgp, bgp-configmap.yaml must be applied to k8s. The file content see cilium doc.
  • Setup pfsense bgp (FRR).
  • LB services should get ExtIPs which are announced to pfsense.
  • Setup portforwarding (NAT) from pfsense to ExtIP of, e.g., ingress.
  • Port forwarding to NodePort also works.

  On Cloud Rootserver, where both public and private interfaces are available, private interfaces are used for BGP with pfsense LAN as GW. Policy based routing must be setup. See http://visrc.com:8000/dokuwiki/doku.php?id=linux:admin:network_routing:source_routing#config_with_netplan

Somehow the br_filter module disappers so make the setting persistent.

  315  vim /etc/sysctl.conf
  316  sysctl -p
  317  modprobe br_netfilter
  318  sysctl -p
  319  less /etc/modules-load.d/modules.conf
  320  echo br_netfilter > /etc/modules-load.d/br_netfilter.conf

• https://danmanners.com/posts/2019-02-pfsense-bgp-kubernetes/
• https://itnext.io/porter-an-open-source-load-balancer-designed-for-bare-metal-kubernetes-clusters-870e1313b7f0

• https://particule.io/en/blog/k8s-no-cloud/

• https://www.stackrox.io/blog/kubernetes-networking-demystified/

• https://komodor.com/learn/how-to-fix-crashloopbackoff-kubernetes-error/

journalctl -xeu containerd

546+02:00" level=info msg="StopPodSandbox for \"895b36244100944fd68560ced6ebb1ad2bbfb5bd4b4f471d77747d898372bb32\""
encountered while deleting endpoint" error="[DELETE /endpoint/{id}][404] deleteEndpointIdNotFound " subsys=cilium-cni
to enter namespace \"\", will not delete interface" error="failed to Statfs \"\": no such file or directory" subsys=cilium-cni
926875940+02:00" level=info msg="TearDown network for sandbox \"895b36244100944fd68560ced6ebb1ad2bbfb5bd4b4f471d77747d898372bb32\" successf>926906588+02:00" level=info msg="StopPodSandbox for \"895b36244100944fd68560ced6ebb1ad2bbfb5bd4b4f471d77747d898372bb32\" returns success

cilium-on-worker logs pod

level=info msg="Processing API request with rate limiter" name=endpoint-delete parallelRequests=17 subsys=rate uuid=49a2b5a0-b647-42be-a1a6-5cab541e1775
level=info msg="API request released by rate limiter" name=endpoint-delete parallelRequests=17 subsys=rate uuid=49a2b5a0-b647-42be-a1a6-5cab541e1775 waitDurationTotal="70.832µs"
level=info msg="API call has been processed" error="endpoint not found" name=endpoint-delete processingDuration="3.386µs" subsys=rate totalDuration="86.752µs" uuid=49a2b5a0-b647-42be-a1a6-5cab541e1775 waitDurationTotal="70.832µs"
level=info msg="Processing API request with rate limiter" name=endpoint-delete parallelRequests=17 subsys=rate uuid=c2c9d890-6966-4161-a608-d4446c204dec
level=info msg="API request released by rate limiter" name=endpoint-delete parallelRequests=17 subsys=rate uuid=c2c9d890-6966-4161-a608-d4446c204dec waitDurationTotal="56.936µs"
level=info msg="API call has been processed" error="endpoint not found" name=endpoint-delete processingDuration="3.517µs" subsys=rate totalDuration="71.604µs" uuid=c2c9d890-6966-4161-a608-d4446c204dec waitDurationTotal="56.936µs"