Table of Contents
K8S Networking
1. Container network
1.1 Flannel
1.2 Calico
1.2.1 Install calicoctl
curl -o calicoctl -O -L "https://github.com/projectcalico/calicoctl/releases/download/v3.20.2/calicoctl" chmod +x calicoctl mv calicoctl /usr/local/bin
1.2.2 Troubleshooting calico
1.2.2.1 BGP on wrong interface
calicoctl get nodes
sudo calicoctl node status Calico process is running. IPv4 BGP status +-----------------+-------------------+-------+------------+---------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +-----------------+-------------------+-------+------------+---------+ | 192.168.182.120 | node-to-node mesh | start | 2021-11-02 | Passive | +-----------------+-------------------+-------+------------+---------+
Check pod status:
kubectl get events --sort-by='.metadata.creationTimestamp' -A | tail kubectl describe pods -n kube-system calico-node-2v72h ... calico/node is not ready: BIRD is not ready: BGP not established with 192.168.124.106
Solution:
specify the IPAUTODETECTIONMETHOD option to calico and it should use the appropriate interface. So after reading over the Change the autodetection method.
Change autodetection interface:
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=interface=eth1
Set node IP
kubectl set env daemonset/calico-node -n kube-system IP=192.168.180.122/24
1.2.2.2 Calico container not started
Check log:
kubectl logs -n kube-system -p calico-node-fjd54 ... failed to query kubeadm's config map error=configmaps "kubeadm-config" is forbidden: User "system:serviceaccount:kube-system:calico-node" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
Solution:
1.2.2.3 Further reading
2. Accessing Services
2.1 Serivce Types
Kubernetes allows you to define 3 types of services using the ServiceType field in its yaml file.
Valid values for the ServiceType field are:
- ClusterIP: use a cluster-internal IP only - this is the default and is discussed above. Choosing this value means that you want this service to be reachable only from inside of the cluster.
- NodePort : on top of having a cluster-internal IP, expose the service on a port on each node of the cluster (the same port on each node). You’ll be able to contact the service on any :NodePort address.
- LoadBalancer: on top of having a cluster-internal IP and exposing service on a NodePort also, ask the cloud provider for a load balancer which forwards to the Service exposed as a :NodePort for each Node.
Ingress Service
3. Install Ingress
We are installing for baremetal k8s. For other see
On master node. Install Mandatory Command is required for all deployments.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
Install ingress for bare-metal:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml
Ingress should be running, check with:
kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx nginx-ingress-controller-797b884cbc-gthd2 1/1 Running 0 3m4s
To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version
POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
3.1 Create ingress config for exposed services
dang@localhost:~/.../service-directory-chariot/k8s-resource-manifests> cat sd-rest-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: sd-rest-ingress
annotations:
ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /sd-rest
backend:
serviceName: service-directory-lb
servicePort: 9000 <--- port exposed by service
kubectl create -f sd-rest-ingress.yaml
4. Trobleshooting
4.1 Service ingress
chariot-web-ui-service-lb.yaml :b2[yaml] 13,3 All
kubernetes.io/ingress.class: "nginx"
#kubernetes.io/ingress.class: "public"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- host: chariot.demo
http:
paths:
- path: /mma-web
backend:
serviceName: chariot-mma-web-lb
servicePort: 9080
4.2 Health check backend services
kubectl describe svc chariot-mma-web-lb Name: chariot-mma-web-lb Namespace: default Labels: <none> Annotations: <none> Selector: app=chariot-mma-web Type: ClusterIP IP: 10.102.168.200 Port: http 9080/TCP TargetPort: 80/TCP Endpoints: 10.244.1.25:80,10.244.2.27:80 <----------- this is it Session Affinity: None Events: <none>
ssh to cluster node
ssh ubuntu@k8s-cluster-host1 curl -v -H "HOST: chariot.demo" 10.244.1.25 <--------- No Path # HOST name of the target server, in case vhost routing is used in nginx
Rebuilt URL to: 10.244.1.25/ * Trying 10.244.1.25... * Connected to 10.244.1.25 (10.244.1.25) port 80 (#0) > GET / HTTP/1.1 > HOST: chariot.demo > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx/1.15.9 < Date: Tue, 19 Mar 2019 10:05:47 GMT < Content-Type: text/html < Content-Length: 665 < Last-Modified: Wed, 13 Mar 2019 17:04:26 GMT < Connection: keep-alive < ETag: "5c89381a-299" < Accept-Ranges: bytes < <!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <title>Chariot Web Interface</title> <base href="/"> <meta name="viewport" content="width=device-width, initial-scale=1"> <link rel="icon" type="image/x-icon" href="favicon.ico"> </head> <body> <app-root></app-root> <script type="text/javascript" src="runtime.js"></script><script type="text/javascript" src="es2015-polyfills.js" nomodule></script><script type="text/javascript" src="polyfills.js"></script><script type="text/javascript" src="styles.js"></script><script type="text/javascript" src="vendor.js"></script><script type="text/javascript" src="main.js"></script></body> </html> * Connection #0 to host 10.244.1.25 left intact
4.3 Check ingress service
4.3.1 Describe ingress service
kubectl describe svc -n ingress-nginx ingress-nginx
Name: ingress-nginx
Namespace: ingress-nginx
Labels: app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/par...
Selector: app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
Type: NodePort <-------------- Ingress with NodePort
IP: 10.99.242.139
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 32417/TCP <----------- the port
Endpoints: 10.244.2.28:80
Port: https 443/TCP
TargetPort: 443/TCP
NodePort: https 31665/TCP
Endpoints: 10.244.2.28:443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
4.3.2 Describe nginx routing
kubectl describe ingresses.extensions chariot-mma-web-ingress
Name: chariot-mma-web-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
chariot.demo
/mma-web chariot-mma-web-lb:9080 (<none>)
Annotations:
ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: false
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 3m38s nginx-ingress-controller Ingress default/chariot-mma-web-ingress
4.3.3 Check ngin conf in ingress pod
ubuntu@k8s-cluster-host0:~$ kubectl exec -n ingress-nginx nginx-ingress-controller-797b884cbc-gthd2 -it bash www-data@nginx-ingress-controller-797b884cbc-gthd2:/etc/nginx$ more /etc/nginx/nginx.conf
## start server chariot.demo
server {
server_name chariot.demo ;
listen 80;
set $proxy_upstream_name "-";
location /mma-web { <--------------- this is it
set $namespace "default";
set $ingress_name "chariot-mma-web-ingress";
set $service_name "chariot-mma-web-lb";
set $service_port "9080";
set $location_path "/mma-web";
rewrite_by_lua_block {
balancer.rewrite()
}
4.3.4 Ingress nginx
kubectl logs -n ingress-nginx nginx-ingress-controller-797b884cbc-gthd2
I0319 09:44:26.010477 6 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"chariot-mma-web-ingress", UID:"9547884a-4a2b-11e9-aabf-3a215cda2552", APIVersion:"extensions/v1beta1", ResourceVersion:"7484436", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/chariot-mma-web-ingress
I0319 09:44:26.010792 6 controller.go:172] Configuration changes detected, backend reload required.
I0319 09:44:26.118037 6 controller.go:190] Backend successfully reloaded.
[19/Mar/2019:09:44:26 +0000]TCP200000.001
10.244.0.0 - [10.244.0.0] - - [19/Mar/2019:09:51:00 +0000] "GET / HTTP/1.1" 404 153 "-" "curl/7.47.0" 76 0.001 [upstream-default-backend] 127.0.0.1:8181 153 0.000 404 3cc403812d05ef43ea080771f12814e3
127.0.0.1 - [127.0.0.1] - - [19/Mar/2019:09:54:16 +0000] "GET / HTTP/1.1" 404 153 "-" "curl/7.64.0" 76 0.000 [upstream-default-backend] 127.0.0.1:8181 153 0.000 404 78949676b5a964eaca3f057c64286299
127.0.0.1 - [127.0.0.1] - - [19/Mar/2019:10:03:25 +0000] "GET /mma-web HTTP/1.1" 404 153 "-" "curl/7.64.0" 83 0.002 [default-chariot-mma-web-lb-9080] 10.244.1.25:80 153 0.000 404 11655896affbf444befaa3dd744f007b
4.4 Expected operation test / Error
Request service using nodeport
curl -v h0.k8s.dai:32417/mma-web curl -v -H "HOST: chariot.demo" h0.k8s.dai:32417/mma-web
* Trying 192.168.180.103... * TCP_NODELAY set * Expire in 200 ms for 4 (transfer 0x556ff5552e40) * Connected to h0.k8s.dai (192.168.180.103) port 32417 (#0) > GET /mma-web HTTP/1.1 <--------- path still there > Host: h0.k8s.dai:32417 > User-Agent: curl/7.64.0 > Accept: */* > < HTTP/1.1 404 Not Found < Server: nginx/1.15.9 < Date: Mon, 18 Mar 2019 15:47:10 GMT < Content-Type: text/html < Content-Length: 153 < Connection: keep-alive < <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.15.9</center> </body> </html> * Connection #0 to host h0.k8s.dai left intact
kubectl logs -n ingress-nginx nginx-ingress-controller-797b884cbc-gthd2 # 10.244.0.0 - [10.244.0.0] - - [19/Mar/2019:10:28:13 +0000] "GET /mma-web HTTP/1.1" 404 153 "-" <----- !!!! path should be translated???? "curl/7.64.0" 83 0.002 [default-chariot-mma-web-lb-9080] 10.244.1.25:80 <--------- Endpoint with path --> ERROR !!! 153 0.000 404 75549e81cccab68fed8c7b2dab250b95
4.4.1 Solution
- Work without host routing too!!
