====== K8S Networking ======
===== - Container network =====
==== - Flannel ====
==== - Calico ====
=== - Install calicoctl ===
curl -o calicoctl -O -L "https://github.com/projectcalico/calicoctl/releases/download/v3.20.2/calicoctl"
chmod +x calicoctl
mv calicoctl /usr/local/bin
=== - Troubleshooting calico ===
== - BGP on wrong interface ==
* .https://elatov.github.io/2020/04/adding-a-node-to-a-kubernetes-cluster-with-kubeadm/
calicoctl get nodes
sudo calicoctl node status
Calico process is running.
IPv4 BGP status
+-----------------+-------------------+-------+------------+---------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+-----------------+-------------------+-------+------------+---------+
| 192.168.182.120 | node-to-node mesh | start | 2021-11-02 | Passive |
+-----------------+-------------------+-------+------------+---------+
Check pod status:
kubectl get events --sort-by='.metadata.creationTimestamp' -A | tail
kubectl describe pods -n kube-system calico-node-2v72h
...
calico/node is not ready: BIRD is not ready: BGP not established with 192.168.124.106
Solution:
specify the IP_AUTODETECTION_METHOD option to calico and it should use the appropriate interface. So after reading over the Change the [[https://docs.projectcalico.org/networking/ip-autodetection#change-the-autodetection-method|autodetection method]].
Change autodetection interface:
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=interface=eth1
Set node IP
kubectl set env daemonset/calico-node -n kube-system IP=192.168.180.122/24
== - Calico container not started ==
Check log:
kubectl logs -n kube-system -p calico-node-fjd54
...
failed to query kubeadm's config map error=configmaps "kubeadm-config" is forbidden: User "system:serviceaccount:kube-system:calico-node" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
Solution:
== - Further reading ==
* .https://medium.com/@bikramgupta/pod-network-troubleshooting-while-using-calico-on-kubernetes-ee78b731d4d8
* .https://faun.pub/kubernetes-without-kube-proxy-1c5d25786e18
===== - Accessing Services =====
* https://kubernetes.io/docs/concepts/cluster-administration/networking/
* Ingress: https://kubernetes.io/docs/concepts/services-networking/ingress/
* http://alesnosek.com/blog/2017/02/14/accessing-kubernetes-pods-from-outside-of-the-cluster/
* Local k8s: https://stackoverflow.com/questions/49219171/expose-service-on-local-kubernetes
==== - Serivce Types ====
* https://kubernetes.io/docs/concepts/services-networking/service/
* https://argus-sec.com/external-communication-with-apache-kafka-deployed-in-kubernetes-cluster/
Kubernetes allows you to define 3 types of services using the ServiceType field in its yaml file.
Valid values for the ServiceType field are:
* ClusterIP: use a cluster-internal IP only - this is the default and is discussed above. Choosing this value means that you want this service to be reachable only from inside of the cluster.
* NodePort : on top of having a cluster-internal IP, expose the service on a port on each node of the cluster (the same port on each node). You’ll be able to contact the service on any :NodePort address.
* LoadBalancer: on top of having a cluster-internal IP and exposing service on a NodePort also, ask the cloud provider for a load balancer which forwards to the Service exposed as a :NodePort for each Node.
* https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0
*
====== Ingress Service ======
* https://oteemo.com/2017/12/12/think-nodeport-kubernetes/
===== - Install Ingress =====
We are installing for baremetal k8s. For other see
* https://kubernetes.github.io/ingress-nginx/deploy/
On master node. Install Mandatory Command is required for all deployments.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
Install ingress for bare-metal:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml
Ingress should be running, check with:
kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx nginx-ingress-controller-797b884cbc-gthd2 1/1 Running 0 3m4s
To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version
POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
==== - Create ingress config for exposed services ====
* general: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
* https://matthewpalmer.net/kubernetes-app-developer/articles/kubernetes-ingress-guide-nginx-example.html
* https://danielfm.me/posts/painless-nginx-ingress.html
dang@localhost:~/.../service-directory-chariot/k8s-resource-manifests> cat sd-rest-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: sd-rest-ingress
annotations:
ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /sd-rest
backend:
serviceName: service-directory-lb
servicePort: 9000 <--- port exposed by service
kubectl create -f sd-rest-ingress.yaml
===== - Trobleshooting =====
==== - Service ingress ====
chariot-web-ui-service-lb.yaml :b2[yaml] 13,3 All
kubernetes.io/ingress.class: "nginx"
#kubernetes.io/ingress.class: "public"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- host: chariot.demo
http:
paths:
- path: /mma-web
backend:
serviceName: chariot-mma-web-lb
servicePort: 9080
==== - Health check backend services ====
kubectl describe svc chariot-mma-web-lb
Name: chariot-mma-web-lb
Namespace: default
Labels:
Annotations:
Selector: app=chariot-mma-web
Type: ClusterIP
IP: 10.102.168.200
Port: http 9080/TCP
TargetPort: 80/TCP
Endpoints: 10.244.1.25:80,10.244.2.27:80 <----------- this is it
Session Affinity: None
Events:
ssh to cluster node
ssh ubuntu@k8s-cluster-host1
curl -v -H "HOST: chariot.demo" 10.244.1.25 <--------- No Path
# HOST name of the target server, in case vhost routing is used in nginx
Rebuilt URL to: 10.244.1.25/
* Trying 10.244.1.25...
* Connected to 10.244.1.25 (10.244.1.25) port 80 (#0)
> GET / HTTP/1.1
> HOST: chariot.demo
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.15.9
< Date: Tue, 19 Mar 2019 10:05:47 GMT
< Content-Type: text/html
< Content-Length: 665
< Last-Modified: Wed, 13 Mar 2019 17:04:26 GMT
< Connection: keep-alive
< ETag: "5c89381a-299"
< Accept-Ranges: bytes
<
Chariot Web Interface
* Connection #0 to host 10.244.1.25 left intact
==== - Check ingress service ====
=== - Describe ingress service ===
kubectl describe svc -n ingress-nginx ingress-nginx
Name: ingress-nginx
Namespace: ingress-nginx
Labels: app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/par...
Selector: app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
Type: NodePort <-------------- Ingress with NodePort
IP: 10.99.242.139
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 32417/TCP <----------- the port
Endpoints: 10.244.2.28:80
Port: https 443/TCP
TargetPort: 443/TCP
NodePort: https 31665/TCP
Endpoints: 10.244.2.28:443
Session Affinity: None
External Traffic Policy: Cluster
Events:
=== - Describe nginx routing ===
kubectl describe ingresses.extensions chariot-mma-web-ingress
Name: chariot-mma-web-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 ()
Rules:
Host Path Backends
---- ---- --------
chariot.demo
/mma-web chariot-mma-web-lb:9080 ()
Annotations:
ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: false
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 3m38s nginx-ingress-controller Ingress default/chariot-mma-web-ingress
=== - Check ngin conf in ingress pod ===
ubuntu@k8s-cluster-host0:~$ kubectl exec -n ingress-nginx nginx-ingress-controller-797b884cbc-gthd2 -it bash
www-data@nginx-ingress-controller-797b884cbc-gthd2:/etc/nginx$ more /etc/nginx/nginx.conf
## start server chariot.demo
server {
server_name chariot.demo ;
listen 80;
set $proxy_upstream_name "-";
location /mma-web { <--------------- this is it
set $namespace "default";
set $ingress_name "chariot-mma-web-ingress";
set $service_name "chariot-mma-web-lb";
set $service_port "9080";
set $location_path "/mma-web";
rewrite_by_lua_block {
balancer.rewrite()
}
=== - Ingress nginx ===
kubectl logs -n ingress-nginx nginx-ingress-controller-797b884cbc-gthd2
I0319 09:44:26.010477 6 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"chariot-mma-web-ingress", UID:"9547884a-4a2b-11e9-aabf-3a215cda2552", APIVersion:"extensions/v1beta1", ResourceVersion:"7484436", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/chariot-mma-web-ingress
I0319 09:44:26.010792 6 controller.go:172] Configuration changes detected, backend reload required.
I0319 09:44:26.118037 6 controller.go:190] Backend successfully reloaded.
[19/Mar/2019:09:44:26 +0000]TCP200000.001
10.244.0.0 - [10.244.0.0] - - [19/Mar/2019:09:51:00 +0000] "GET / HTTP/1.1" 404 153 "-" "curl/7.47.0" 76 0.001 [upstream-default-backend] 127.0.0.1:8181 153 0.000 404 3cc403812d05ef43ea080771f12814e3
127.0.0.1 - [127.0.0.1] - - [19/Mar/2019:09:54:16 +0000] "GET / HTTP/1.1" 404 153 "-" "curl/7.64.0" 76 0.000 [upstream-default-backend] 127.0.0.1:8181 153 0.000 404 78949676b5a964eaca3f057c64286299
127.0.0.1 - [127.0.0.1] - - [19/Mar/2019:10:03:25 +0000] "GET /mma-web HTTP/1.1" 404 153 "-" "curl/7.64.0" 83 0.002 [default-chariot-mma-web-lb-9080] 10.244.1.25:80 153 0.000 404 11655896affbf444befaa3dd744f007b
==== - Expected operation test / Error ====
Request service using nodeport
curl -v h0.k8s.dai:32417/mma-web
curl -v -H "HOST: chariot.demo" h0.k8s.dai:32417/mma-web
* Trying 192.168.180.103...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x556ff5552e40)
* Connected to h0.k8s.dai (192.168.180.103) port 32417 (#0)
> GET /mma-web HTTP/1.1 <--------- path still there
> Host: h0.k8s.dai:32417
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.15.9
< Date: Mon, 18 Mar 2019 15:47:10 GMT
< Content-Type: text/html
< Content-Length: 153
< Connection: keep-alive
<
404 Not Found