Hiển thị các bài đăng có nhãn nginx. Hiển thị tất cả bài đăng
Hiển thị các bài đăng có nhãn nginx. Hiển thị tất cả bài đăng

Thứ Sáu, 28 tháng 8, 2026

Nginx mTLS

 


A. Nginx as SERVER with mTLS
Trong  hệ thống nội bộ, ca certificate thường dùng chung cả cho server và client.
Nhưng nếu bảo mật hơn, có thể tách riêng ra. Xem phần C.
Ở phần A này tôi sẽ dùng chung
1.1 Generate the CA Certificate and Key
openssl genpkey -algorithm RSA -out /etc/nginx/cert/ca.key
openssl req -days 3650 -new -x509 -key /etc/nginx/cert/ca.key -out /etc/nginx/cert/ca.crt -subj "/OU=CA-OrgUnit/CN=CA-MASTER"
openssl x509 -text -noout -in /etc/nginx/cert/ca.crt
1.2 Generate the Server Certificate and Key
openssl genpkey -algorithm RSA -out /etc/nginx/cert/server.key
openssl req -new -key /etc/nginx/cert/server.key -out /etc/nginx/cert/server.csr -subj "/OU=MyOrgUnit/CN=myserver.com.vn"
openssl x509 -req -days 3650 -in /etc/nginx/cert/server.csr -CA /etc/nginx/cert/ca.crt -CAkey /etc/nginx/cert/ca.key -CAcreateserial -out /etc/nginx/cert/server.crt
openssl x509 -text -noout -in /etc/nginx/cert/server.crt
1.3 Generate the Client Certificate and Key
openssl genpkey -algorithm RSA -out /etc/nginx/cert/client.key
openssl req -new -key /etc/nginx/cert/client.key -out /etc/nginx/cert/client.csr -subj "/OU=MyOrgUnit/CN=MyClient"
openssl x509 -req -days 3650 -in /etc/nginx/cert/client.csr -CA /etc/nginx/cert/ca.crt -CAkey /etc/nginx/cert/ca.key -CAcreateserial -out /etc/nginx/cert/client.crt
openssl x509 -text -noout -in /etc/nginx/cert/client.crt
2 Configure Nginx for SSL/TLS and mTLS
 server {
 listen 443 ssl;
    # Server certificate and key
     ssl_certificate /etc/nginx/cert/server.crt;
     ssl_certificate_key /etc/nginx/cert/server.key;

    # CA certificate for client verification
     ssl_client_certificate /etc/nginx/cert/ca.crt;
     ssl_verify_client optional;

          location / {
                  default_type text/plain;
                  if ($ssl_client_verify != SUCCESS) {
                      return 403 'blocked access to mTLS-protected resource';
                  }
                  return 200 'access to mTLS-protected resource';
          }

}
Reload nginx apply new config
nginx -s reload
3: Testing mTLS with cURL
curl -vk https://localhost
> blocked access to mTLS-protected resource
curl --cert /etc/nginx/cert/client.crt --key /etc/nginx/cert/client.key --cacert /etc/nginx/cert/ca.crt https://localhost
> access to mTLS-protected resource


openssl s_client -connect localhost:443 -cert /etc/nginx/cert/client.crt -key /etc/nginx/cert/client.key -CAfile /etc/nginx/cert/ca.crt
________________________________________
Refer: https:/medium.com/@mahernaija/new-2025-how-to-configure-mutual-tls-mtls-for-secure-nginx-206f983ba571


B. Nginx as CLIENT in mTLS
Chuẩn bị bộ client cert/key. Ngoài ra upstream_ca.crt chính là Ca-Certificate

server {
    listen 80;
    server_name my-gateway.local;

    location / {
        # 1. Đường dẫn tới Server backend yêu cầu mTLS
        proxy_pass https://secure-upstream-server.local:443;

        # 2. Bật các cấu hình SSL cho Proxy
        proxy_ssl_server_name on; # Gửi SNI (Server Name Indication) nếu cần
        # Định nghĩa tên miền sẽ gửi trong TLS Handshake đến Upstream
        # Thường sử dụng biến $host (tên miền client gửi lên) hoặc một chuỗi cố định như 'backend.com'
        proxy_ssl_name $host;

        # 3. Gửi chứng chỉ và khóa của Nginx (đóng vai trò Client) lên Server
        proxy_ssl_certificate     /etc/nginx/certs/client.crt;
        proxy_ssl_certificate_key /etc/nginx/certs/client.key;

        # 4. Xác thực chứng chỉ của Server backend để đảm bảo an toàn
        proxy_ssl_trusted_certificate /etc/nginx/certs/upstream_backend_ca.crt;
        proxy_ssl_verify             on;
        proxy_ssl_verify_depth       2;

        # Các cấu hình proxy thông thường khác
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

C. Chia riêng rẽ ca certificate client/server
Không muốn dùng chung. Ta sẽ làm như sau
1. Tạo "Upstream CA" và chứng chỉ cho Upstream Server:*8443
mkdir -p /etc/nginx/certs && cd /etc/nginx/certs

# 1.1 Tạo Upstream CA (Dùng để ký cho Server)
openssl genrsa -out upstream_ca.key 4096
openssl req -new -x509 -days 3650 -key upstream_ca.key -out upstream_ca.crt -subj "/CN=UpstreamServerRootCA"

# 1.2 Tạo chứng chỉ cho Upstream Server (Sẽ cài trên máy chủ backend)
openssl genrsa -out upstream_server.key 2048
openssl req -new -key upstream_server.key -out upstream_server.csr -subj "/CN=secure-upstream.local"
openssl x509 -req -days 365 -in upstream_server.csr -CA upstream_ca.crt -CAkey upstream_ca.key -CAcreateserial -out upstream_server.crt

2. Tạo "Client CA" và chứng chỉ cho Nginx Client:80
mkdir -p /etc/nginx/certs && cd /etc/nginx/certs

# 2.1 Tạo Client CA riêng biệt (Dùng để ký cho các Client/Nginx)
openssl genrsa -out client_ca.key 4096
openssl req -new -x509 -days 3650 -key client_ca.key -out client_ca.crt -subj "/CN=MyClientInternalCA"

# 2.2 Tạo chứng chỉ cho Nginx Client (Sẽ cài trên Nginx Proxy)
openssl genrsa -out nginx_client.key 2048
openssl req -new -key nginx_client.key -out nginx_client.csr -subj "/CN=NginxProxyClient"
openssl x509 -req -days 365 -in nginx_client.csr -CA client_ca.crt -CAkey client_ca.key -CAcreateserial -out nginx_client.crt

Cấu hình nginx
1.  Khối Server 1 (Nginx Client / Reverse Proxy): Nhận HTTP cổng 80 từ người dùng → Dùng mTLS truyền tiếp lên Backend.
2.  Khối Server 2 (Upstream Server / Backend): Chạy HTTPS cổng 8443 → Yêu cầu mTLS và chỉ tin tưởng chứng chỉ từ Client CA.
Sơ đồ: client  nginx:80 + Proxy pass Backend :8443 with mtls - Nginx:8443 with mtls
# ====================================================================
# VAI TRÒ 1: NGINX ĐÓNG VAI TRÒ CLIENT (REVERSE PROXY)
# ====================================================================
server {
    listen 80;
    server_name localhost;

    location / {
        # Chuyển tiếp tới Upstream Server chạy HTTPS cổng 8443
        proxy_pass https://secure-upstream.local:8443;
        proxy_ssl_server_name on; #send SNI

        # ------------------------------------------------------------
        # [A] XÁC THỰC SERVER: Nginx kiểm tra danh tính của Upstream Server
        # ------------------------------------------------------------
        proxy_ssl_verify             on;
        # CHỈ tin tưởng Server nếu chứng chỉ của nó do "Upstream CA" ký
        proxy_ssl_trusted_certificate /etc/nginx/certs/upstream_ca.crt;
        proxy_ssl_verify_depth       4;

        # ------------------------------------------------------------
        # [B] XÁC THỰC CLIENT: Nginx gửi chứng chỉ của mình lên Server
        # ------------------------------------------------------------
        # Chứng chỉ này được tạo và ký bởi "Client CA" riêng biệt, sẽ gửi sang server để verify
        proxy_ssl_certificate         /etc/nginx/certs/nginx_client.crt;
        proxy_ssl_certificate_key     /etc/nginx/certs/nginx_client.key;

        # Các cấu hình header cơ bản
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

# ====================================================================
# VAI TRÒ 2: UPSTREAM SERVER (GIẢ LẬP MÁY CHỦ BACKEND NHẬN KẾT NỐI mTLS)
# ====================================================================
server {
    listen 8443 ssl;
    server_name secure-upstream.local;

    # Chứng chỉ của bản thân Server (Được ký bởi Upstream CA). Như 1 ssl bình thường nhưng phơi ra để client kiểm tra.
    ssl_certificate         /etc/nginx/certs/upstream_server.crt;
    ssl_certificate_key     /etc/nginx/certs/upstream_server.key;

    # Cấu hình mTLS phía nhận: Bắt buộc phải có chứng chỉ Client hợp lệ
    ssl_verify_client       on;
    # CHỈ chấp nhận chứng chỉ Client nếu nó được ký bởi "Client CA"
    ssl_client_certificate  /etc/nginx/certs/client_ca.crt;
    ssl_verify_depth        2;

    location / {
        # Trả về phản hồi text nếu mTLS thành công tốt đẹp
        add_header Content-Type text/plain;
        return 200 "Chúc mừng! Kết nối mTLS giữa Nginx Client và Upstream Server thành công bằng 2 CA độc lập.\n";
    }
}

Khi gọi trực tiếp vào 8443 sẽ bị lỗi
Đứng ở client ta phải cung cấp đủ 3 key mới gọi được
# curl --cacert /etc/nginx/certs/upstream_ca.crt --key /etc/nginx/certs/client_ca.key --cert /etc/nginx/certs/client_ca.crt https://secure-upstream.local:8443
Chúc mừng! Kết nối mTLS giữa Nginx Client và Upstream Server thành công bằng 2 CA độc lập.

# curl localhost:80
Chúc mừng! Kết nối mTLS giữa Nginx Client và Upstream Server thành công bằng 2 CA độc lập.



# Log format for forward_proxy:
log_format tuan_forward_proxy '$http_x_real_ip $http_x_forwarded_for $remote_addr $remote_user $time_local '
'"$request" $status $body_bytes_sent '
'$request_time $upstream_response_time '
'$upstream_addr $upstream_response_length $upstream_status '
'"$http_referer" "$http_user_agent" '
'$connect_host:$connect_port $connect_addr $proxy_connect_resolve_time $proxy_connect_connect_time $proxy_connect_first_byte_time $proxy_connect_response';

# Log format for reverse_proxy:
log_format tuan_reverse_proxy escape=none '$http_x_real_ip $http_x_forwarded_for $remote_addr $remote_user [$time_local] '
'"$request_method $scheme://$host$request_uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" '
'$request_length $request_time $sent_http_content_type '
$upstream_addr $upstream_response_length $upstream_response_time $upstream_status '
'request_body: [$request_body]';



# Log format for container:
log_format tuan_pod escape-none '$remote_addr $remote_user [$time_local] '
'"$request_method $scheme://$host$request_uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" '
'$request_length $request_time $sent_http_content_type '
'- $upstream_addr $upstream_response_length $upstream_response_time $upstream_status '
'"$http_x_real_ip $http_x_forwarded_for"';

Thứ Năm, 6 tháng 8, 2026

Nginx SNI

Nginx SNI
SNI chia làm 2 loại là SNI server và SNI Client

1. Nginx reject non-SNI – server side
# Khối server mặc định để chặn mọi truy cập không khớp SNI,
yêu cầu phải gửi servername sni lên cùng
server {
    listen 443 default_server ssl;
    listen [::]:443 default_server ssl;

    # Từ chối bắt tay SSL ngay lập tức
    ssl_reject_handshake on;
}

# Cấu hình website hợp lệ của bạn
server {
    listen 443 ssl;
    server_name yourdomain.com;

    ssl_certificate     /path/to/cert.crt;
    ssl_certificate_key /path/to/cert.key;
    # ... cấu hình khác
}


2. SNI as client  side (forward proxy)
Mặc định, Nginx không tự động gửi SNI khi kết nối với Upstream.
Nếu máy chủ Upstream yêu cầu SNI để trả về chứng chỉ đúng,
bạn bắt buộc phải bật cấu hình Client-side SNI thông qua các
chỉ thị proxy_ssl_server_name và proxy_ssl_name
server {
    listen 443 ssl;
    server_name proxy.example.com;

    ssl_certificate /etc/ssl/certs/proxy.crt;
    ssl_certificate_key /etc/ssl/certs/proxy.key;

    location / {
        # 1. Chuyển tiếp request đến upstream qua HTTPS
        proxy_pass https://backend_server;

        # 2. BẬT Client-side SNI (Mặc định là off)
        proxy_ssl_server_name on;

        # 3. Định nghĩa tên miền sẽ gửi trong TLS Handshake đến Upstream
        # Thường sử dụng biến $host (tên miền client gửi lên) hoặc một chuỗi cố định như 'backend.com'
        proxy_ssl_name $host;

        # (Tùy chọn) Đảm bảo Header Host của HTTP cũng trùng khớp với SNI
        proxy_set_header Host $host;
    }
}

TLS Handshake Flow with SNI
 




Thứ Ba, 7 tháng 4, 2026

Nginx mode security


1. Cài đặt modesecurity
sudo apt-get update
sudo apt-get install gcc make build-essential autoconf automake libtool libcurl4-openssl-dev liblua5.3-dev libfuzzy-dev ssdeep gettext pkg-config libgeoip-dev libyajl-dev doxygen libpcre2-16-0 libpcre2-dev libpcre2-posix3 zlib1g zlib1g-dev -y

cd /opt && sudo git clone https://github.com/owasp-modsecurity/ModSecurity.git
cd ModSecurity
git submodule init
git submodule update
./build.sh
./configure
make
make install

2. Cài đặt nginx cho người lười compile
sudo add-apt-repository ppa:ondrej/nginx -y
sudo apt update
sudo apt install nginx -y

sudo systemctl enable nginx
sudo systemctl status nginx

sudo nginx -V
nginx version: nginx/1.28.1
built with OpenSSL 3.0.13 30 Jan 2024
TLS SNI support enabled

3. Download nginx-mod connector
cd /opt && sudo git clone https://github.com/owasp-modsecurity/ModSecurity-nginx.git

cd /opt && sudo wget https://nginx.org/download/nginx-1.28.1.tar.gz
sudo tar -xzvf nginx-1.28.1.tar.gz
cd nginx-1.28.1

sudo ./configure --with-compat --add-dynamic-module=/opt/ModSecurity-nginx
sudo make
sudo make modules

sudo cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules-enabled/
sudo cp /opt/ModSecurity/modsecurity.conf-recommended /etc/nginx/modsecurity.conf
sudo cp /opt/ModSecurity/unicode.mapping /etc/nginx/unicode.mapping

#2. Enable ModSecurity in nginx.conf
vim /etc/nginx/nginx.conf
load_module /etc/nginx/modules-enabled/ngx_http_modsecurity_module.so;

vim /etc/nginx/sites-enabled/default
modsecurity on;
modsecurity_rules_file /etc/nginx/modsecurity.conf;

#Change SecRuleEngine to On.
vim /etc/nginx/modsecurity.conf
SecRuleEngine On

#after that we can our nginx configuration and restart nginx server
sudo nginx -t
sudo systemctl restart nginx

4. Cấu hình CRS
sudo git clone https://github.com/coreruleset/coreruleset.git /etc/nginx/owasp-crs
sudo cp /etc/nginx/owasp-crs/crs-setup.conf{.example,}


sudo vim /etc/nginx/modsecurity.conf
Include owasp-crs/crs-setup.conf
Include owasp-crs/rules/*.conf

curl http://127.0.0.1/doc?s=/bin/bash #-> Return 404 FILE-NOT-FOUND

sudo nginx -t
sudo service nginx restart

# Test Modsecurity + Nginx with browser
curl http://127.0.0.1/doc?s=/bin/bash #-> Return 403 BLOCK

# Đọc log tại
cat /etc/nginx/modsecurity.conf  | grep AuditLog
tail -f /var/log/modsec_audit.log

5. Chỉ chặn SQL và XSS
sudo vim /etc/nginx/modsecurity.conf
# Nạp cấu hình cơ bản của CRS
Include /etc/nginx/owasp-crs/crs-setup.conf

# CHỈ NẠP CÁC QUY TẮC SQL INJECTION
Include /etc/nginx/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf

# CHỈ NẠP CÁC QUY TẮC XSS
Include /etc/nginx/owasp-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf

# File này giúp ModSecurity tổng hợp điểm số (score) từ các quy tắc trên để đưa ra quyết định chặn (deny) cuối cùng. Đừng quên file này.
Include /etc/nginx/owasp-crs/rules/RESPONSE-980-CORRELATION.conf

6. Bản cài ăn sẵn
https://modsecurity.digitalwave.hu/

7. Custome log
# Chọn các phần muốn ghi log (ABCDEFHJKZ)
A: Audit Log Header (Bắt buộc). Chứa thông tin cơ bản: thời gian, ID duy nhất, IP nguồn, IP đích, cổng.
B: Request Headers. Ghi lại toàn bộ các Header gửi lên (User-Agent, Cookie, Host, v.v.).
C: Request Body. Ghi lại nội dung payload gửi lên (Dữ liệu form, JSON, file upload). Chỉ hoạt động nếu SecRequestBodyAccess là On.
D: Reserved. (Dự phòng cho tương lai, hiện tại không dùng).
E: Intermediate Response Body. Ghi lại nội dung server phản hồi cho client. (Có thể làm log rất nặng).
F: Response Headers. Ghi lại các Header mà server gửi trả về cho trình duyệt.
G: Reserved. (Dự phòng).
H: Audit Log Trailer. Chứa các thông tin bổ sung như: điểm số tấn công (Score), loại engine, và các thông tin Metadata khác.
I: Replacement for C. (Ít dùng) Ghi lại Request Body nhưng dùng định dạng multipart/form-data.
J: Uploaded Files Info. Thông tin chi tiết về các file được tải lên (tên file, kích thước).
K: Matched Rules. Danh sách tất cả các Rule ID đã bị khớp (Trigger) trong request này. (Rất quan trọng để debug).
Z: Final Boundary (Bắt buộc). Đánh dấu kết thúc một bản ghi log.

vim /etc/nginx/modsecurity.conf
SecAuditLogParts ABCEFHJKZ


-----BEGIN PRIVATE KEY-----
MIGHAgExxAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQg2Wz1L+c3u314HS7l
wR253bZS7uwTvY3GP63h02mxqn+hRANCAATmy8Q+SoBpQOP1un+La/mg3WRuqR/n
pljRqM7jo80yHOjqvGtPy0aJ5YDNLXOd7pwuP75A5P5AKdMTElsMDnC0
-----END PRIVATE KEY-----

-----BEGIN CERTIFICATE-----
MIIDlzCCAx6gAwIBAgISBWe/19sRZvyEjcHQknnRGeBHMAoGCCqGSM49BAMDMDMx
CzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQDEwNZ
RTIwHhcNMjYwNzE5MjAwNjQ3WhcNMjYxMDE3MjAwNjQ2WjAZMRcwFQYDVQQDDA4q
LnR1YW4ubmFtZS52bjBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABObLxD5KgGlA
4/W6f4tr+aDdZG6pH+emWNGozuOjzTIc6Oq8a0/LRonlgM0tc53unC4/vkDk/kAp
0xMSWwwOcLSjggIqMIICJjAOBgNVHQ8BAf8EBAMCB4AwEwYDVR0lBAwwCgYIKwYB
BQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUliE0SZKTlVHs7I0jPHLT6x08
irMwHwYDVR0jBBgwFoAUuVnyjs8i8IbTN0j/dhQYuoLYVYcwMwYIKwYBBQUHAQEE
JzAlMCMGCCsGAQUFBzAChhdodHRwOi8veWUyLmkubGVuY3Iub3JnLzAnBgNVHREE
IDAegg4qLnR1YW4ubmFtZS52boIMdHVhbi5uYW1lLnZuMBMGA1UdIAQMMAowCAYG
Z4EMAQIBMC4GA1UdHwQnMCUwI6AhoB+GHWh0dHA6Ly95ZTIuYy5sZW5jci5vcmcv
NzIuY3JsMIIBDAYKKwYBBAHWeQIEAgSB/QSB+gD4AHcAyKPEf8ezrbk1awE/anoS
beM6TkOlxkb5l605dZkdz5oAAAGffDKUOwAABAMASDBGAiEA1p+PJNCidE2o7CqT
A3jH+2BwERkMHOxAQbgxXEQ8IZwCIQDXyav6sNntsCbI5yQ6xx5ynG7A3CEAyaIw
SifA8iIk0wB9ACbjZG5YaSEjvDQ/RyQ1mzeSzSRaiNgV05Mz/ZkYq0cjAAABn3wy
kQ4ACAAABQArB0I3BAMARjBEAiAdylMOiNEpiaow+JQh+FhbpWaK10ZCYSyolKfj
ZobglwIgOQENQwum/o+f8ND0glFjexoVpyWXVjmXYbLNmUPRTAswCgYIKoZIzj0E
AwMDZwAwZAIwP5C/nBghlYIbJ4qEWmgkVyLw5UbF+Oe1cCeeP010/+4W8iSEt9t/
DapTdwg52QSQAjAB8WiFKpjlwNZkShMSh/AqmSuyxMDkgoaHx/k8L+yWeook77hO
lVDIujuF/W0lA2Q=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIICjDCCAhGgAwIBAgIQTfOxXdbAeExQfNN7WObxFTAKBggqhkjOPQQDAzAuMQsw
CQYDVQQGEwJVUzENMAsGA1UEChMESVNSRzEQMA4GA1UEAxMHUm9vdCBZRTAeFw0y
NTA5MDMwMDAwMDBaFw0yODA5MDIyMzU5NTlaMDMxCzAJBgNVBAYTAlVTMRYwFAYD
VQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQDEwNZRTIwdjAQBgcqhkjOPQIBBgUr
gQQAIgNiAARxmrQzkdbEEL3MqXt3dJQttYc47axkdDTHud5TPqM2z5uSD5cmk0Wr
HlWXvnlvqBLqiB34kluxIbmMyAiq3/YD6e80/vV259K8XQIdjFXloYOa0mIU71f7
HQ09PvYDlw+jge4wgeswDgYDVR0PAQH/BAQDAgGGMBMGA1UdJQQMMAoGCCsGAQUF
BwMBMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFLlZ8o7PIvCG0zdI/3YU
GLqC2FWHMB8GA1UdIwQYMBaAFKPIJlqOoUzQNWP8myPIOq5W809WMDIGCCsGAQUF
BwEBBCYwJDAiBggrBgEFBQcwAoYWaHR0cDovL3llLmkubGVuY3Iub3JnLzATBgNV
HSAEDDAKMAgGBmeBDAECATAnBgNVHR8EIDAeMBygGqAYhhZodHRwOi8veWUuYy5s
ZW5jci5vcmcvMAoGCCqGSM49BAMDA2kAMGYCMQDIcnw5dcZLN9ffynXnnkLD/itS
JEycJPb3sRkzeqBowup7vOsAwaqoCnNn/jh9wycCMQCJM6CPlaOC4pQYYbJtVPYb
DKrIb2EKk5NpOpE6/XttQYZV/3gilB9l+Cc/DOVwmyg=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIICpjCCAiugAwIBAgIRAIchZfw0tuX7qK3Vs3BftTowCgYIKoZIzj0EAwMwTzEL
MAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2VhcmNo
IEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDIwHhcNMjYwNTEzMDAwMDAwWhcN
MzIwOTAyMjM1OTU5WjAuMQswCQYDVQQGEwJVUzENMAsGA1UEChMESVNSRzEQMA4G
A1UEAxMHUm9vdCBZRTB2MBAGByqGSM49AgEGBSuBBAAiA2IABDwS/6vhrcVqcbBo
+wgdI3fwn9x7DNJJOY/lTOti0vkwuRN87RhEhTH17E7XyFjWsPYhIPt/wzOqxTd2
b+4ZJNy9ID04YywF9U5zasDVyGSNErVNtz8uSGh5izW87j77GaOB6zCB6DAOBgNV
HQ8BAf8EBAMCAQYwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUwAwEB
/zAdBgNVHQ4EFgQUo8gmWo6hTNA1Y/ybI8g6rlbzT1YwHwYDVR0jBBgwFoAUfEKW
rt5LSDv6kviejM9ti6lyN5UwMgYIKwYBBQUHAQEEJjAkMCIGCCsGAQUFBzAChhZo
dHRwOi8veDIuaS5sZW5jci5vcmcvMBMGA1UdIAQMMAowCAYGZ4EMAQIBMCcGA1Ud
HwQgMB4wHKAaoBiGFmh0dHA6Ly94Mi5jLmxlbmNyLm9yZy8wCgYIKoZIzj0EAwMD
aQAwZgIxAMU19WCtmxVND8UHBZRoma49Z7jPs64Dma0eTu1OChVbB/2J7GV3nvYK
Ax54uk1G9QIxAO0miLVJu8PLNiXXXkiE/gsK3CTRTF/aeo4bMX42Zw40csRU6AC2
6hSW1/IWaas6dg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEcDCCAligAwIBAgIQbI8dxyfHEX97r4U6yYD5zTANBgkqhkiG9w0BAQsFADBP
MQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJuZXQgU2VjdXJpdHkgUmVzZWFy
Y2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBYMTAeFw0yNjA1MTMwMDAwMDBa
Fw0zMjA5MDIyMzU5NTlaME8xCzAJBgNVBAYTAlVTMSkwJwYDVQQKEyBJbnRlcm5l
dCBTZWN1cml0eSBSZXNlYXJjaCBHcm91cDEVMBMGA1UEAxMMSVNSRyBSb290IFgy
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEzZvVn4CDCuwJSvMWSj5cz3es3mcFDR0H
ttwW+1qLFNvicWDEukWVEYmO6gbf9yoWHKS5xcUy4APgHoIYOIvXRdgKam7mAHf7
AlF9ItgKbppbd9/w+kHsOdx1ymgHDB/qo4H1MIHyMA4GA1UdDwEB/wQEAwIBBjAd
BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUwAwEB/zAd
BgNVHQ4EFgQUfEKWrt5LSDv6kviejM9ti6lyN5UwHwYDVR0jBBgwFoAUebRZ5nu2
5eQBc4AIiMgaWPbpm24wMgYIKwYBBQUHAQEEJjAkMCIGCCsGAQUFBzAChhZodHRw
Oi8veDEuaS5sZW5jci5vcmcvMBMGA1UdIAQMMAowCAYGZ4EMAQIBMCcGA1UdHwQg
MB4wHKAaoBiGFmh0dHA6Ly94MS5jLmxlbmNyLm9yZy8wDQYJKoZIhvcNAQELBQAD
ggIBAD2/e9frmMxNpCV03qUHegg+MV2wz9644YoXdqtH8RyWYcBO7xfjjGEXdU1e
/o0OkEFiynUCOSIk/vLLo7ttz6CPAeNlWfC0XNkoGeWgK6jjXvozBaGuGH5n0Ufo
shMeWTuURqNN5G00sSXDTBrpp2+mgvdZQjb8K11TYMA25QA+YHNfbIEL0BniAhKS
2gsnJjSzrdZLI+EZ7SEyqdR2rkjd1KutLDU+n3TFyxjniZVGur4YlhMP3mY/dV95
IruAkkjOZier6hGBdEgZXXvaCz9u9iVEadsIE75pAGL8oHV5vxdARDiotRpul1IN
/UZwzAbrfUFcw1HkAcYD/mlZfnQ2ieCF2MS7j3Vhv7JPDKp45fmykmzYNSrumRW0
upFFKDBOoF7hsOb7oLyHS+Uft6jOUfOrogj8YUx38hKb2K20r42OgsSdDdxdeYWc
MS3Sb6mwJeSZEYxJ2gaXnDSPaKhhrNkYwljyVQyr4Nq+MEJytXNTnHqaAcrNwZlV
pcJL1KBnMrMjP7eanvUwL3FYj3cF17jtboLt7gLoi4+2rWZFvn+w54jmd/FIuhhZ
cEaU/wvU6BUNMtcVquVGHp7itQeDth5j+XL3j4WJ2SABwzUl6OeYdgpIt/ITZa+p
TT0mQ/r5XyA4MEAiabn7XJjvCERlF2dcn2wqJw+CreTkkQ2Rxxx
-----END CERTIFICATE-----


Thứ Tư, 19 tháng 3, 2025

Encrypt traffic with Stunnel

LAB_1 [Stunnel_Server]/[Stunnel_Client]

Diagram:



Step1: Setup Stunnel Server

#Install Stunnel & Redis
apt-get install redis-server stunnel4

#Create SSL 
openssl genrsa -out /etc/stunnel/key.pem 4096
openssl req -new -x509 -key /etc/stunnel/key.pem -out /etc/stunnel/cert.pem -days 1826
cat /etc/stunnel/key.pem /etc/stunnel/cert.pem > /etc/stunnel/private.pem
chmod 640 /etc/stunnel/key.pem /etc/stunnel/cert.pem /etc/stunnel/private.pem

cat << EOF > /etc/stunnel/stunnel-redis.conf
pid = /var/run/stunnel.pid
[redis]
client = no
cert = /etc/stunnel/private.pem
accept = 16379
connect = 127.0.0.1:6379
EOF

systemctl start stunnel4
systemctl enable stunnel4

Step2: Setup Stunnel Client

apt-get install redis-tools stunnel4
# Copy /etc/stunnel/private.pem from server -> to client

cat << EOF > /etc/stunnel/stunnel-redis.conf
pid = /var/run/stunnel.pid
[redis]
client = yes
cert = /etc/stunnel/private.pem
accept = 6379
connect = 192.168.88.12:16379
EOF

systemctl stop stunnel4
systemctl start stunnel4
systemctl enable stunnel4

Step3: Test

Run redis-cli on client machine

# redis-cli 
127.0.0.1:6379> info
redis_version:6.0.16

LAB_2 [Nginx TCP-Stream-SSL] / [Stunnel Client]

Diagram:



Step1: Setup Nginx Server

#Create SSL
openssl genrsa -out /etc/stunnel/key.pem 4096
openssl req -new -x509 -key /etc/stunnel/key.pem -out /etc/stunnel/cert.pem -days 1826
cat /etc/stunnel/key.pem /etc/stunnel/cert.pem > /etc/stunnel/private.pem
chmod 640 /etc/stunnel/key.pem /etc/stunnel/cert.pem /etc/stunnel/private.pem


# Install NGINX
apt-get install nginx libnginx-mod-stream
stream {
    server {
        listen 16382 ssl;
        proxy_pass 127.0.0.1:6379;

        # Cấu hình SSL/TLS
        ssl_certificate /etc/stunnel/cert.pem;
        ssl_certificate_key /etc/stunnel/key.pem;
        #ssl_trusted_certificate /path/to/ca_cert.crt;
    }
}

Step2: Setup Stunnel Client

apt-get install redis-tools stunnel4
#Copy /etc/stunnel/private.pem from server -> to client

cat << EOF > /etc/stunnel/stunnel-redis.conf
pid = /var/run/stunnel.pid
[redis]
client = yes
cert = /etc/stunnel/private.pem
accept = 6379
connect = 192.168.88.12:16382
EOF

systemctl stop stunnel4
systemctl start stunnel4
systemctl enable stunnel4

Step3: Test

Run redis-cli on client machine

# redis-cli 
127.0.0.1:6379> info
redis_version:6.0.16

Thứ Tư, 26 tháng 2, 2025

Nginx proxy to Kubeapi-server and act as log audit

 Bài Toán: Một ngày đẹp trời, 1 ý tưởng thật dị hợm nảy ra trong đầu:

  • Liệu nginx có thể đứng trước làm proxy reverse cho KubeAPI không?
  • Liệu cùng chung IP:port, nhưng khác domain thì có thể kết nối các cụm cluster khác nhau được không?

Thực hiện:

Phần 1: Nginx Proxy_Pass tới kube-api

Kiến trúc: Admin ---> Nginx ---> KubeAPI

Bước 1.1 Cài OpenResty (Hoặc Nginx)

Các bạn có thể cài nginx hoặc openresty đều được. Mục đích mình cài Openrestry (khi cài openresty sẽ có luôn nginx) là để sử dụng cho phần 2.

# Dành cho CenOS ( https://openresty.org/en/linux-packages.html )
cd /etc/yum.repo.d/
curl -O https://openresty.org/package/centos/openresty.repo
yum install openresty
systemctl start openresty
systemctl enable openresty

# Dành cho Ubuntu
Làm theo hướng dẫn các version của ubuntu: https://openresty.org/en/linux-packages.html#ubuntu , Ở đây tôi đang dùng 22.04
wget -O - https://openresty.org/package/pubkey.gpg | sudo gpg --dearmor -o /usr/share/keyrings/openresty.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/openresty.gpg] http://openresty.org/package/ubuntu $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/openresty.list > /dev/null
apt-get update
apt-get -y install --no-install-recommends openresty
systemctl start openresty
systemctl enable openresty

Bước 1.2 : Cấu hình nginx.conf

Ta sẽ sử dụng cert có sẵn trong kube-apiserver

cat /etc/kubernetes/manifests/kube-apiserver.yaml    
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

Thêm cầu hình proxy mới như sau:

# vim /usr/local/openresty/nginx/conf/nginx.conf

upstream kubeapi_cluster {
    server 192.168.88.12:6443 max_fails=3 fail_timeout=5s;
    server 192.168.88.13:6443 max_fails=3 fail_timeout=5s;
    server 192.168.88.14:6443 max_fails=3 fail_timeout=5s;
  }

server {
    listen 6443 ssl;
    server_name _;

    ssl_certificate /etc/kubernetes/pki/apiserver.crt;
    ssl_certificate_key /etc/kubernetes/pki/apiserver.key;

    location / {
        proxy_pass https://kubeapi_cluster/;

        proxy_http_version                 1.1;
        proxy_cache_bypass                 $http_upgrade;

        # Proxy SSL
        proxy_ssl_server_name              on;

        # Proxy headers
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Scheme $scheme;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_set_header X-Forwarded-For    $remote_addr;
        proxy_set_header X-Real-IP          $remote_addr;

        # Proxy timeouts
        proxy_connect_timeout              600s;
        proxy_send_timeout                 600s;
        proxy_read_timeout                 600s;

        proxy_ssl_certificate /etc/kubernetes/pki/apiserver-kubelet-client.crt;
        proxy_ssl_certificate_key /etc/kubernetes/pki/apiserver-kubelet-client.key;
    }
}


Reload lại OpenResty Nginx
systemctl reload openresty

Sửa kube config trỏ vào pod mới 6443 -> 6444

sed -i -e 's/6443/6444/g'  ~/.kube/config

Test kiểm tra get api và logs nginx

# kubectl get node -v=6
I1117 13:26:50.223517   36661 round_trippers.go:553] GET https://192.168.88.12:6444/api?timeout=32s 200 OK in 25 milliseconds
I1117 13:26:50.231582   36661 round_trippers.go:553] GET https://192.168.88.12:6444/apis?timeout=32s 200 OK in 5 milliseconds
I1117 13:26:50.262655   36661 round_trippers.go:553] GET https://192.168.88.12:6444/api/v1/nodes?limit=500 200 OK in 5 milliseconds
NAME           STATUS   ROLES           AGE     VERSION
master-node    Ready    control-plane   4d20h   v1.30.0
worker-node1   Ready    <none>          4d20h   v1.29.8
# tail /usr/local/openresty/nginx/logs/access.log -n 1
192.168.88.12 - - [17/Nov/2024:13:26:50 +0000] "GET /api/v1/nodes?limit=500 HTTP/1.1" 200 9242 "-" "kubectl/v1.30.0 (linux/amd64) kubernetes/7c48c2b"

*** Ý tưởng nữa hiện ra trong đầu, liệu bung luôn cả request và response của nó ra logs được không...next phần 2

Phần 2: Nginx đóng vai trò AuditLogs

Cầu hình thêm nginx.conf bổ xung log_format, access_log và lua

vim /usr/local/openresty/nginx/conf/nginx.conf

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

  #Thêm phần log format này:
    log_format log_req_resp escape=none '$remote_addr - $remote_user [$time_local] '
           ' "$request" $status $body_bytes_sent ${request_time}ms '
           '| PRINT_REQUEST_BODY: $request_body '
           '| PRINT_REQUEST_HEADER:"$req_header" '
           '| PRINT_RESPONSE_HEADER:"$resp_header" '
           '| PRINT_RESPONSE_BODY:"$resp_body" ';

    access_log  logs/access.log log_req_resp;

upstream kubeapi_cluster {
    server 192.168.88.12:6443 max_fails=3 fail_timeout=5s;
    server 192.168.88.13:6443 max_fails=3 fail_timeout=5s;
    server 192.168.88.14:6443 max_fails=3 fail_timeout=5s;
  }

   #Default 80
   server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
    
    # Cấu hình 6444 log request và response
    server {
    listen 6444 ssl;
    server_name _;

    ssl_certificate /etc/kubernetes/pki/apiserver.crt;
    ssl_certificate_key /etc/kubernetes/pki/apiserver.key;

    location / {

        #Step2: Config REPSONSE_BODY
        lua_need_request_body on;

        set $resp_body "";
        body_filter_by_lua '
          local resp_body = string.sub(ngx.arg[1], 1, 1000000)
          ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
          if ngx.arg[2] then
             ngx.var.resp_body = ngx.ctx.buffered
          end
        ';

        #Step3: Config REQUEST_HEADER, RESPONSE_HEADER
        set $req_header "";
        set $resp_header "";
        header_filter_by_lua '
          local h = ngx.req.get_headers()
          for k, v in pairs(h) do
              ngx.var.req_header = ngx.var.req_header .. k.."="..v.." "
          end
          local rh = ngx.resp.get_headers()
          for k, v in pairs(rh) do
              ngx.var.resp_header = ngx.var.resp_header .. k.."="..v.." "
          end
        ';


        proxy_pass https://kubeapi_cluster/;

        proxy_http_version                 1.1;
        proxy_cache_bypass                 $http_upgrade;

        # Proxy SSL
        proxy_ssl_server_name              on;

        # Proxy headers
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_set_header Host               $host;
        proxy_set_header X-Forwarded-Scheme $scheme;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_set_header X-Forwarded-For    $remote_addr;
        proxy_set_header X-Real-IP          $remote_addr;

        # Proxy timeouts
        proxy_connect_timeout              600s;
        proxy_send_timeout                 600s;
        proxy_read_timeout                 600s;

        proxy_ssl_certificate /etc/kubernetes/pki/apiserver-kubelet-client.crt;
        proxy_ssl_certificate_key /etc/kubernetes/pki/apiserver-kubelet-client.key;
    }
}


}



Kết quả

# kubectl get node -v=6
I1117 13:37:22.883328   39935 round_trippers.go:553] GET https://192.168.88.12:6444/api/v1/nodes?limit=500 200 OK in 13 milliseconds
NAME           STATUS   ROLES           AGE     VERSION
master-node    Ready    control-plane   4d20h   v1.30.0
worker-node1   Ready    <none>          4d20h   v1.29.8

tail -n1 /var/log/nginx/access.log
192.168.88.12 -  [17/Nov/2024:13:34:58 +0000]  "GET /api/v1/nodes?limit=500 HTTP/1.1" 200 9242 0.010ms 
| PRINT_REQUEST_BODY:  
| PRINT_REQUEST_HEADER:"host=192.168.88.12:6444 user-agent=kubectl/v1.30.0 (linux/amd64) kubernetes/7c48c2b accept=application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json accept-encoding=gzip kubectl-command=kubectl get kubectl-session=8b063f81-279c-47dc-a39b-fc42abff2dcb " 
| PRINT_RESPONSE_HEADER:"cache-control=no-cache, private content-type=application/json connection=keep-alive x-kubernetes-pf-prioritylevel-uid=c7bb2e92-7125-47ee-b8af-b0da52a06199 audit-id=d4859fdd-7e70-4791-81ea-5d5a02d02d1d x-kubernetes-pf-flowschema-uid=c2ed420f-f18f-4d80-aaf1-291dd5dc931d " 
| PRINT_RESPONSE_BODY:"{"kind":"Table","apiVersion":"meta.k8s.io/v1","metadata":{"resourceVersion":"120353"},"columnDefinitions":[{"name":"Name","type":"string","format":"name","description":"Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#names","priority":0},{"name":"Status","type":"string","format":"","description":"The status of the node","priority":0},{"name":"Roles","type":"string","format":"","description":"The roles of the node","priority":0},{"name":"Age","type":"string","format":"","description":"CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.\n\nPopulated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata","priority":0},{"name":"Version","type":"string","format":"","description":"Kubelet Version reported by the node.","priority":0},{"name":"Internal-IP","type":"string","format":"","description":"List of addresses reachable to the node. Queried from cloud provider, if available. More info: https://kubernetes.io/docs/concepts/nodes/node/#addresses Note: This field is declared as mergeable, but the merge key is not sufficiently unique, which can cause data corruption when it is merged. Callers should instead use a full-replacement patch. See https://pr.k8s.io/79391 for an example. Consumers should assume that addresses can change during the lifetime of a Node. However, there are some exceptions where this may not be possible, such as Pods that inherit a Node's address in its own status or consumers of the downward API (status.hostIP).","priority":1},{"name":"External-IP","type":"string","format":"","description":"List of addresses reachable to the node. Queried from cloud provider, if available. More info: https://kubernetes.io/docs/concepts/nodes/node/#addresses Note: This field is declared as mergeable, but the merge key is not sufficiently unique, which can cause data corruption when it is merged. Callers should instead use a full-replacement patch. See https://pr.k8s.io/79391 for an example. Consumers should assume that addresses can change during the lifetime of a Node. However, there are some exceptions where this may not be possible, such as Pods that inherit a Node's address in its own status or consumers of the downward API (status.hostIP).","priority":1},{"name":"OS-Image","type":"string","format":"","description":"OS Image reported by the node from /etc/os-release (e.g. Debian GNU/Linux 7 (wheezy)).","priority":1},{"name":"Kernel-Version","type":"string","format":"","description":"Kernel Version reported by the node from 'uname -r' (e.g. 3.16.0-0.bpo.4-amd64).","priority":1},{"name":"Container-Runtime","type":"string","format":"","description":"ContainerRuntime Version reported by the node through runtime remote API (e.g. containerd://1.4.2).","priority":1}],"rows":[{"cells":["master-node","Ready","control-plane","4d20h","v1.30.0","192.168.88.12","\u003cnone\u003e","Ubuntu 22.04.5 LTS","5.15.0-125-generic","containerd://1.7.22"],"object":{"kind":"PartialObjectMetadata","apiVersion":"meta.k8s.io/v1","metadata":{"name":"master-node","uid":"a0262311-4b3b-4261-bb97-db648705e311","resourceVersion":"119778","creationTimestamp":"2024-11-12T16:57:45Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"master-node","kubernetes.io/os":"linux","node-role.kubernetes.io/control-plane":"","node.kubernetes.io/exclude-from-external-load-balancers":""},"annotations":{"kubeadm.alpha.kubernetes.io/cri-socket":"unix:///var/run/containerd/containerd.sock","node.alpha.kubernetes.io/ttl":"0","projectcalico.org/IPv4Address":"192.168.88.12/24","projectcalico.org/IPv4IPIPTunnelAddr":"172.16.77.128","volumes.kubernetes.io/controller-managed-attach-detach":"true"},"managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-12T16:57:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}}},{"manager":"kubeadm","operation":"Update","apiVersion":"v1","time":"2024-11-12T16:57:48Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}},"f:labels":{"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2024-11-13T18:48:35Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:taints":{}}}},{"manager":"calico-node","operation":"Update","apiVersion":"v1","time":"2024-11-17T12:54:41Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}},"subresource":"status"},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-17T13:30:15Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:allocatable":{"f:ephemeral-storage":{},"f:memory":{}},"f:capacity":{"f:memory":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{}}}},"subresource":"status"}]}}},{"cells":["worker-node1","Ready","\u003cnone\u003e","4d20h","v1.29.8","192.168.88.13","\u003cnone\u003e","Ubuntu 22.04.5 LTS","5.15.0-125-generic","containerd://1.7.22"],"object":{"kind":"PartialObjectMetadata","apiVersion":"meta.k8s.io/v1","metadata":{"name":"worker-node1","uid":"604ca6f0-252d-4566-82bf-129163a96ea2","resourceVersion":"120025","creationTimestamp":"2024-11-12T16:58:04Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"worker-node1","kubernetes.io/os":"linux"},"annotations":{"kubeadm.alpha.kubernetes.io/cri-socket":"unix:///var/run/containerd/containerd.sock","node.alpha.kubernetes.io/ttl":"0","projectcalico.org/IPv4Address":"192.168.88.13/24","projectcalico.org/IPv4IPIPTunnelAddr":"172.16.180.192","volumes.kubernetes.io/controller-managed-attach-detach":"true"},"managedFields":[{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-12T16:58:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}}},{"manager":"kubeadm","operation":"Update","apiVersion":"v1","time":"2024-11-12T16:58:05Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:kubeadm.alpha.kubernetes.io/cri-socket":{}}}}},{"manager":"calico-node","operation":"Update","apiVersion":"v1","time":"2024-11-17T12:55:37Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}},"subresource":"status"},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2024-11-17T12:55:39Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2024-11-17T13:32:17Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:allocatable":{"f:cpu":{},"f:ephemeral-storage":{},"f:memory":{}},"f:capacity":{"f:cpu":{},"f:memory":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:nodeInfo":{"f:bootID":{},"f:kernelVersion":{}}}},"subresource":"status"}]}}}]}

Kết quả không đẹp đẽ như bật Kube AuditLogs trực tiếp từ Policy vì nội dung trả về khá nhiều, như vậy sẽ làm đầy ổ cứng rất nhanh. Nhưng cũng là 1 phương án thay thế có thể áp dụng.. ta có thể dùng logrotate để nén log và lưu trữ trong 1 khoảng thời gian nhất định.

(*Bài viết chỉ mang tính chất ý tưởng và chưa áp dụng vào thực tế... ^^)

(Nguồn tham khảo https://gist.github.com/gilangvperdana/2c4877c8efb729534e7f7c55e6e1e2d3

https://medium.com/@rifewang/what-happens-after-running-a-kubectl-command-8aeed20ed5c4 )


My Viblo https://viblo.asia/p/nginx-proxy-toi-kube-apiserver-va-nginx-lam-audit-logs-lai-request-response-EbNVQgoRJvR 

Nginx mTLS

  A. Nginx as SERVER with mTLS Trong  hệ thống nội bộ, ca certificate thường dùng chung cả cho server và client. Nhưng nếu bảo mật hơn, có t...