https://github.com/msfidelis/envoy-proxy-examples
https://github.com/yokawasa/envoy-proxy-demos
https://github.com/jansvabik/envoy-proxy
Contents
Multi server_name and multi_upstream
EDS (Endpoint Discovery Services)
Ví Dụ 1: health check bằng /path
Ví dụ 2: health dựa vào return code
Cài đặt:
https://www.envoyproxy.io/docs/envoy/latest/start/install
Bài viết về envoy: https://www.thoughtworks.com/insights/blog/building-service-mesh-envoy-0
Centos:
sudo yum install yum-utils
sudo rpm --import
'https://rpm.dl.getenvoy.io/public/gpg.CF716AF503183491.key'
curl -sL
'https://rpm.dl.getenvoy.io/public/config.rpm.txt?distro=el&codename=7'
> /tmp/tetrate-getenvoy-rpm-stable.repo
sudo yum-config-manager --add-repo
'/tmp/tetrate-getenvoy-rpm-stable.repo'
sudo yum makecache --disablerepo='*' --enablerepo='tetrate-getenvoy-rpm-stable'
sudo yum install getenvoy-envoy
git clone https://github.com/envoyproxy/envoy.git
01. Static Config /
Mô tả: Proxy sang
google, sử dụng luôn ssl của google.
[tuanda@master-node configs]$ envoy -c google_com_http3_upstream_proxy.yaml
[tuanda@master-node configs]$ cat /home/tuanda/envoy/configs/google_com_http3_upstream_proxy.yaml
admin:
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type":
type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
host_rewrite_literal:
www.google.com
cluster: service_google
http_filters:
- name: envoy.filters.http.router
clusters:
- name: service_google
connect_timeout: 30s
type: LOGICAL_DNS
# Comment out
the following line to test on v6 networks
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_google
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: www.google.com
port_value: 443
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http3_protocol_options:
{}
common_http_protocol_options:
idle_timeout: 1s
transport_socket:
name: envoy.transport_sockets.quic
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicUpstreamTransport
upstream_tls_context:
sni: www.google.com
02-1. Static Config / multi path
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/service/1"
route:
cluster: service1
- match:
prefix: "/service/2"
route:
cluster: service2
http_filters:
- name: envoy.router
config: {}
clusters:
- name: service1
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
hosts:
- socket_address:
address: service1
port_value: 80
- name: service2
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
hosts:
- socket_address:
address: service2
port_value: 80
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 8001
02-2. Multi server_name and multi_upstream
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
access_log:
- name: envoy.file_access_log
config:
path:
"/dev/stdout"
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "one.example.com"
- "www.one.example.com"
routes:
- match:
prefix: "/"
route:
cluster: targetCluster
http_filters:
- name: envoy.router
clusters:
- name: targetCluster
connect_timeout: 0.25s
type: STRICT_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
hosts: [
{ socket_address: { address: 172.18.0.3, port_value: 80 }},
{ socket_address: { address: 172.18.0.4, port_value: 80 }}
]
Giải thích:
Acess_log
access_log:
- name: envoy.file_access_log
config:
path: "/dev/stdout"
Custom log https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage
access_log:
- name: envoy.file_access_log
config:
path: "/dev/stdout"
format: "[%START_TIME%]
"%REQ(:METHOD)%
%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"
%RESPONSE_CODE% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-REQUEST-ID)%"
"%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"\n"
Hoặc log json
access_log:- name: envoy.file_access_log config: path: "/dev/stdout" json_format: {"protocol": "%PROTOCOL%", "duration": "%DURATION%", "request_method": "%REQ(:METHOD)%"}Https
Sẽ bao gồm http listen 80 và redirect to Listen 443
static_resources:
listeners:
- name: listener_http
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
virtual_hosts:
- name: backend
domains:
- "example.com"
routes:
- match:
prefix: "/"
redirect:
path_redirect: "/"
https_redirect: true
http_filters:
- name: envoy.router
config: {}
- name: listener_https
address:
socket_address: { address: 0.0.0.0, port_value: 8443 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "example.com"
routes:
- match:
prefix: "/service/1"
route:
cluster: service1
- match:
prefix: "/service/2"
route:
cluster: service2
http_filters:
- name: envoy.router
config: {}
tls_context:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: "/etc/envoy/certs/example-com.crt"
private_key:
filename: "/etc/envoy/certs/example-com.key"
clusters:
- name: service1
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.3
port_value: 80
- name: service2
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.4
port_value: 80
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 8001
File Dynamic Routing
xDS bao gồm: EDS, CDS, RDS, LDS, SDS (EndPoint, Cluster,
Route, Listener, Secret).
EDS (Endpoint Discovery Services)
Thay vì việc để confix trỏ host cứng bằng cluster dưới đây
clusters: - name: targetCluster connect_timeout: 0.25s type: STRICT_DNS dns_lookup_family: V4_ONLY lb_policy: ROUND_ROBIN hosts: [ { socket_address: { address: 172.18.0.3, port_value: 80 }}, { socket_address: { address: 172.18.0.4, port_value: 80 }} ]Ta có thể chuyển sang Endpoint discover Services bằng việc
trỏ load file eds.conf
clusters: - name: targetCluster connect_timeout: 0.25s lb_policy: ROUND_ROBIN type: EDS eds_cluster_config: service_name: localservices eds_config: path: '/etc/envoy/eds.conf'Trong file eds.conf tạm thời để 1 host như sau. Đồng thời
start luôn Envoy, để test việc tự động cập nhập host mới.
{ "version_info": "0", "resources": [{ "@type": "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment", "cluster_name": "localservices", "endpoints": [{ "lb_endpoints": [{ "endpoint": { "address": { "socket_address": { "address": "172.18.0.3", "port_value": 80 } } } }] }] }]}Sau đó ta config thêm 1 host nữa.
{ "version_info": "0", "resources": [{ "@type": "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment", "cluster_name": "localservices", "endpoints": [{ "lb_endpoints": [{ "endpoint": { "address": { "socket_address": { "address": "172.18.0.3", "port_value": 80 } } } }, { "endpoint": { "address": { "socket_address": { "address": "172.18.0.4", "port_value": 80 } } } }] }] }]}TỔNG KẾT FILE:
cat envoy.yaml
node:
id: id_1
cluster: test
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 10000 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: targetCluster
http_filters:
- name: envoy.router
clusters:
- name: targetCluster
connect_timeout: 0.25s
lb_policy: ROUND_ROBIN
type: EDS
eds_cluster_config:
service_name: localservices
eds_config:
path: '/etc/envoy/eds.conf'
cat eds.conf (#bản cũ chạy 1 node)
{ "version_info": "0", "resources": [{ "@type": "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment", "cluster_name": "localservices", "endpoints": [{ "lb_endpoints": [{ "endpoint": { "address": { "socket_address": { "address": "172.18.0.3", "port_value": 80 } } } }] }] }]}cat eds.conf #(Đây là bản sau khi thêm 1 node cluster)
{ "version_info": "0", "resources": [{ "@type": "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment", "cluster_name": "localservices", "endpoints": [{ "lb_endpoints": [{ "endpoint": { "address": { "socket_address": { "address": "172.18.0.3", "port_value": 80 } } } }, { "endpoint": { "address": { "socket_address": { "address": "172.18.0.4", "port_value": 80 } } } }] }] }]}
Thực hiện move file để Envoy tự động load cấu hình mới:
mv eds.conf tmp; mv tmp eds.conf
Kiểm tra bằng Curl, đã thấy node mới được join vào
Loadbalancer.
CDS và LDS (Cluster vaf Listener)
Có tổng 3 file, envoy.yaml là chính, cds control cluster,
lds control listen
cat envoy1.yaml
node:
id: id_1
cluster: test
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 9901
dynamic_resources:
cds_config:
path: "/etc/envoy/cds.conf"
lds_config:
path: "/etc/envoy/lds.conf"
cat cds.conf
{ "version_info": "0", "resources": [{ "@type": "type.googleapis.com/envoy.api.v2.Cluster", "name": "targetCluster", "connect_timeout": "0.25s", "lb_policy": "ROUND_ROBIN", "type": "EDS", "eds_cluster_config": { "service_name": "localservices", "eds_config": { "path": "/etc/envoy/eds.conf" } } }]}Hoặc theo yaml v3
[tuanda@localhost
configs]$ cat cds.yaml
resources:
-
"@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
name: example_proxy_cluster
type: STRICT_DNS
load_assignment:
cluster_name: example_proxy_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: service1
port_value: 8080
cat lds.conf
{ "version_info": "0", "resources": [{ "@type": "type.googleapis.com/envoy.api.v2.Listener", "name": "listener_0", "address": { "socket_address": { "address": "0.0.0.0", "port_value": 10000 } }, "filter_chains": [ { "filters": [ { "name": "envoy.http_connection_manager", "config": { "stat_prefix": "ingress_http", "codec_type": "AUTO", "route_config": { "name": "local_route", "virtual_hosts": [ { "name": "local_service", "domains": [ "*" ], "routes": [ { "match": { "prefix": "/" }, "route": { "cluster": "targetCluster" } } ] } ] }, "http_filters": [ { "name": "envoy.router" } ] } } ] } ] }]}Hoặc lds.yaml
[tuanda@localhost
configs]$ cat lds.yaml
resources:
-
"@type": type.googleapis.com/envoy.config.listener.v3.Listener
name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
name:
envoy.filters.network.http_connection_manager
typed_config:
"@type":
type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
http_filters:
- name: envoy.filters.http.router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: example_proxy_cluster
Để apply cấu hình mới (ví dụ ta change thêm node LB, sửa
cds, lds) thì chỉ cẩn move tên và move lại, Envoy sẽ reload cấu hình mới.
mv cds.conf tmp; mv tmp cds.conf; mv lds.conf
tmp; mv tmp lds.conf
API Dynamic Routing
https://www.envoyproxy.io/try/api-based-dynamic-routing-configuration
EDS server sẽ control việc add/remove cluster node. Mọi thứ
sẽ do API giải quyết.
Mô hình
Envoy Server ----(tham chiếu)----> EDS server <---- gọi
API để thay đổi cluster node
Chuẩn bị các file:
cat envoy.yaml
admin:
access_log_path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 9000
node:
cluster: mycluster
id: test-id
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 10000 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
stat_prefix: ingress_http
codec_type: AUTO
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route: { cluster: targetCluster }
http_filters:
- name: envoy.router
clusters:
- name: targetCluster
type: EDS
connect_timeout: 0.25s
eds_cluster_config:
service_name: myservice
eds_config:
api_config_source:
api_type: REST
cluster_names: [eds_cluster]
refresh_delay: 5s
- name: eds_cluster
type: STATIC
connect_timeout: 0.25s
hosts: [{ socket_address: { address: 172.18.0.4, port_value: 8080 }}]
bỏ ngỏ
Health Check
Ví Dụ 1: health check bằng /path
cat envoy.yaml
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: targetCluster
http_filters:
- name: envoy.router
clusters:
- name: targetCluster
connect_timeout: 0.25s
type: STRICT_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
hosts: [
{ socket_address: { address: 172.18.0.3, port_value: 80 }},
{ socket_address: { address: 172.18.0.4, port_value: 80 }}
]
health_checks:
- timeout: 1s
interval: 10s
interval_jitter: 1s
unhealthy_threshold: 6
healthy_threshold: 1
http_health_check:
path: "/health"
·
interval: How frequently should be health check
be performed
·
unhealthy_threshold: The number of
unhealthy health checks required before a host is marked unhealthy
·
healthy_threshold: The number of healthy
health checks required before a host is marked healthy
·
http_health_check.path: Specifies the HTTP
path that will be requested during health checking
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/health_check.proto.html
Ví dụ 2: health dựa vào return code
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: targetCluster
http_filters:
- name: envoy.router
clusters:
- name: targetCluster
connect_timeout: 0.25s
type: STRICT_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
hosts: [
{ socket_address: { address: 172.18.0.5, port_value: 80 }},
{ socket_address: { address: 172.18.0.6, port_value: 80 }}
]
outlier_detection:
consecutive_5xx: "3"
base_ejection_time: "30s"
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/outlier
Blue/Green Deploy
Route by Header
static_resources:
listeners:
- name: listener_http
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/service/1"
route:
cluster: service1
- match:
prefix: "/service/2"
headers:
-
name: "x-canary-version"
exact_match: "service2a"
route:
cluster: service2a
- match:
prefix: "/service/2"
route:
cluster: service2
#TODO Service3
http_filters:
- name: envoy.router
config: {}
clusters:
- name: service1
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.3
port_value: 80
- name: service2
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.4
port_value: 80
- name: service2a
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.5
port_value: 80
- name: service3a
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.6
port_value: 80
- name: service3b
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.7
port_value: 80
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 8001
Weigh load Balancing
static_resources:
listeners:
- name: listener_http
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/service/1"
route:
cluster: service1
- match:
prefix: "/service/2"
headers:
-
name:
"x-canary-version"
exact_match:
"service2a"
route:
cluster: service2a
- match:
prefix: "/service/2"
route:
cluster: service2
- match:
prefix: "/service/3"
route:
weighted_clusters:
clusters:
- name: service3a
weight: 80
- name: service3b
weight: 20
http_filters:
- name: envoy.router
config: {}
clusters:
- name: service1
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.3
port_value: 80
- name: service2
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.4
port_value: 80
- name: service2a
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.5
port_value: 80
- name: service3a
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.6
port_value: 80
- name: service3b
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 172.18.0.7
port_value: 80
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 8001
Debug
Load-balancing method
Weighted round robin
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 10000 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: targetCluster
http_filters:
- name: envoy.router
clusters:
- name: targetCluster
connect_timeout: 0.25s
type: STRICT_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: targetCluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 172.18.0.3
port_value: 80
load_balancing_weight: 90
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 172.18.0.4
port_value: 80
Weighted least request
Ring hash
Maglev
Random
Original destination
