我的开源
kubernetes的ingress初体验
背景
Ingress是一种灵活的服务暴露方式。
Ingress 实现
ingress有多种实现，也可以同时使用多个Ingress controllers。
官方实现
- GCE controller
- nginx controller
非官方实现
- Ambassador
- AppsCode
- Contour
- Crtrix
- F5
- Gloo
- HAProxy Technologies
- Istio
- Kong
- NGINX
- Skipper
- Traefik (go)
nginx ingress初体验
本文以最简单的nginx ingress为例。
安装配置
安装
见CentOS7上kubernetes服务初体验中使用Helm部署Nginx Ingress一节。
访问
// 查看端口
# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-controller LoadBalancer 10.109.232.28 <pending> 80:31853/TCP,443:31027/TCP 18h
nginx-ingress-default-backend ClusterIP 10.99.40.223 <none> 80/TCP 18h
// 测试 http
# curl "localhost:31853"
default backend - 404
// 测试 https
# curl "https://localhost:31027" -k
default backend - 404
暴露 http 服务
这里以 kubernetes-bootcamp 服务为例
暴露 kubernetes-bootcamp 服务
将 kubernetes-bootcamp 配置成 cluster-ip 方式
# kubectl expose deployment kubernetes-bootcamp --name=kubernetes-bootcamp --port=80 --target-port=8080
service/kubernetes-bootcamp exposed
确认cluster ip配置成功了
# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-bootcamp ClusterIP 10.105.249.152 <none> 80/TCP 5s
在容器的交互式shell中执行
root@kubernetes-bootcamp-7dc9765bf6-pn2rs:/# curl "10.105.249.152"
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-7dc9765bf6-tr8pc | v=1
root@kubernetes-bootcamp-7dc9765bf6-pn2rs:/# curl "10.105.249.152"
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-7dc9765bf6-vbppq | v=1
创建 ingress 文件 kubernetes-bootcamp.ingress.yaml [3]，根据前缀转发，内容如下：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kubernetes-bootcamp
namespace: default
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /kubernetes-bootcamp
pathType: Prefix
backend:
service:
name: kubernetes-bootcamp
port:
number: 80
# kubectl create -f kubernetes-bootcamp.ingress.yaml
ingress.extensions/kubernetes-bootcamp created
查看 ingress
# kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
kubernetes-bootcamp * 80 12m
调用
[root@172-19-120-198 ingress]# curl "localhost:31853/kubernetes-bootcamp/"
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-7dc9765bf6-fk92j | v=1
[root@172-19-120-198 ingress]# curl "localhost:31853/kubernetes-bootcamp/"
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-7dc9765bf6-tr8pc | v=1
根据域名转发，如下：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kubernetes-bootcamp
namespace: default
spec:
ingressClassName: nginx
rules:
- host: kubernetes-bootcamp.abeffect.com
http:
paths:
- path: /
pathType: Exact
backend:
service:
name: kubernetes-bootcamp
port:
number: 80
技术细节
配置的注入
进行ingress容器中
# kubectl exec -it nginx-ingress-controller-598c7fd878-smwwr -n ingress-nginx bash
查看配置
$ cat nginx.conf
可见增加的配置内容：
location /kubernetes-bootcamp {
set $namespace "default";
set $ingress_name "kubernetes-bootcamp";
set $service_name "";
set $service_port "{0 80 }";
set $location_path "/kubernetes-bootcamp";
rewrite_by_lua_block {
lua_ingress.rewrite({
force_ssl_redirect = false,
use_port_in_redirects = false,
})
balancer.rewrite()
plugins.run()
}
header_filter_by_lua_block {
plugins.run()
}
body_filter_by_lua_block {
}
log_by_lua_block {
balancer.log()
monitor.call()
plugins.run()
}
if ($scheme = https) {
more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains";
}
port_in_redirect off;
set $balancer_ewma_score -1;
set $proxy_upstream_name "default-kubernetes-bootcamp-80";
set $proxy_host $proxy_upstream_name;
set $pass_access_scheme $scheme;
set $pass_server_port $server_port;
set $best_http_host $http_host;
set $pass_port $pass_server_port;
set $proxy_alternative_upstream_name "";
client_max_body_size 1m;
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering off;
proxy_buffer_size 4k;
proxy_buffers 4 4k;
proxy_request_buffering on;
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0;
proxy_next_upstream_tries 3;
proxy_pass http://upstream_balancer;
proxy_redirect off;
}
upstream的实现
请求，会被交给 upstream_balancer 中的 balancer.balance() 来处理。
upstream upstream_balancer {
server 0.0.0.1; # placeholder
balancer_by_lua_block {
balancer.balance()
}
keepalive 32;
keepalive_timeout 60s;
keepalive_requests 100;
}
其中 balancer的逻辑在 lua/balancer.lua中。
细节见 K8S 源码探秘 之 nginx-ingress 工作原理分析和Nginx ingress 中如何透传upstream header？
ingress资源类型
ingress将 yaml 资源文件，翻译成了 nginx 配置文件。接下来看看 ingress 支持的资源类型。
单Service资源
即后端只有一个服务，如这里使用 kubernetes-bootcamp 做为默认服务。配置文件 default.ingress.yaml如下：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-default
spec:
backend:
serviceName: kubernetes-bootcamp
servicePort: 80