bug: lua_shared_dict prometheus-metrics overflow

[DrJSAnD]

Current Behavior

We deploy apisix in K8s cluster and have problem with prometheus metrics.
We noticed that lua_shared_dict prometheus-metrics overflows, then the number of apisix_nginx_metric_errors_total errors starts to grow and all metrics stop displaying correctly.

We try increase the prometheus-metrics parameter to 40m in the ConfigMap (config.yaml), but after 2 months this lua_shared_dict was full on all pods and errors started to occur again.

nginx_config:    # config for render the template to genarate nginx.conf
  error_log: "/dev/stderr"
  error_log_level: "warn"    # warn,error
  worker_processes: "auto"
  enable_cpu_affinity: true
  worker_rlimit_nofile: 20480  # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    enable_access_log: true
    access_log: "/dev/stdout"
    access_log_format: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'
    access_log_format_escape: default
    keepalive_timeout: "60s"
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Real-IP"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 'unix:'
    lua_shared_dict:
      prometheus-metrics: 40m

Current Apisix state

• Deployment via Helm chart: https://github.com/apache/apisix-helm-chart
• Helm Chart version: 2.10.0
• K8s pods: 3
• Pod CPU limits: 15 (usage 4%)
• Pod Memory limits: 60Gb (usage 35 GiB)
• Total requests per second: 2500 - 3000
• Active connections: 2000+
• Upstreams: 100+
• Routes: 120+
• Consumers: 60+
• Plugins: basic-auth and kafka-logger on all routes

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

1. Run apisix with default lua_shared_dict: prometheus-metrics
2. After 2-3 weeks prometheus-metrics overflows and apisix_nginx_metric_errors_total errors starts to grow and all metrics stop displaying correctly
3. Change lua_shared_dict: prometheus-metrics to 40m
4. After 2-3 months lua_shared_dict overflows again and we get a similar problem with displaying metrics

Environment

• APISIX version (run apisix version): 3.10.0
• Operating system (run uname -a): Linux apisix-69cfdc5fbf-m7k27 5.14.0-362.13.1.el9_3.x86_64 SMP PREEMPT_DYNAMIC Fri Nov 24 01:57:57 EST 2023 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.25.3.2
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.0
• APISIX Dashboard version, if relevant: 3.0.0
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[qizhendong1]

You can use curl 127.0.0.1:9091/apisix/prometheus/metrics to test the prometheus plugin endpoint if it works properly

[DrJSAnD]

Prometheus plugin works on all pods and it returns 69000+ rows from each pod.
We display metrics from Prometheus in Grafana (https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json).
When apisix_shared_dict_free_space_bytes{name="prometheus-metrics"} reached value "0" then apisix_nginx_metric_errors_total start grows up and all apisix metrics show incorrect values.

[shihuncl]

K8S部署的apisix怎么修改lua_shared_dict的大小，试了很多地方都是修改不成功

[DrJSAnD]

K8S部署的apisix怎么修改lua_shared_dict的大小，试了很多地方都是修改不成功

In ConfigMap - config.yaml

nginx_config:
    http:
        lua_shared_dict:
            prometheus-metrics: 40m

[bzp2010]

Generally this has to do with how discrete the requests are on your API service, which is related to multiple metrics such as route_id, resp_code, or even consumer_id, etc. (https://apisix.apache.org/docs/apisix/next/plugins/prometheus/#labels) Multiply them together and that's the number of record entries you'll get, which in the prometheus or observability is called a high cardinality.
If they're really large, you'll need to increase the metrics size.

[DrJSAnD]

Generally this has to do with how discrete the requests are on your API service, which is related to multiple metrics such as route_id, resp_code, or even consumer_id, etc. (https://apisix.apache.org/docs/apisix/next/plugins/prometheus/#labels) Multiply them together and that's the number of record entries you'll get, which in the prometheus or observability is called a high cardinality. If they're really large, you'll need to increase the metrics size.

Will it be okay if we increase the lua_shared_dict - prometheus-metrics parameter value to 512m or 1024m?

[jlaupa]

I have the same problem but it has been dragging since version 3.8, I have been increasing it from 40m to 100m now 200m. But in 3.10 I started to see a memory leak.

[shihuncl]

I have the same problem but it has been dragging since version 3.8, I have been increasing it from 40m to 100m now 200m. But in 3.10 I started to see a memory leak.

So what should you do if you don’t have enough memory for this indicator now? Restart apisix?

[DrJSAnD]

I have the same problem but it has been dragging since version 3.8, I have been increasing it from 40m to 100m now 200m. But in 3.10 I started to see a memory leak.

So what should you do if you don’t have enough memory for this indicator now? Restart apisix?

Yes, I restart Apisix pods and lua_shared_dict prometheus-metrics released memory. Then it starts to gradually fill up again.

[jlaupa]

@shihuncl @DrJSAnD
Well, I finally found the error.
I will share it with you in case it is your case. good luck.

Apisix in my case was taking as many workers as cores available on the node. Which is wrong, it should take from the pod according to the limitation I assign.
The solution was to set a number of workers for my traffic, I have decided based on the tests I did that 3 workers is fine. Now (1G ram, at most). If you have less memory, lower the workers and try to reuse as much as possible of the custom plugins.

The prometheus plugin, even if you set it to 10m, if you exceed this number you will only get a warning and it won't really be a problem, but it is noisy.
How did you check it?

• nproc
• top

then I saw that there were as many workers as openresty running :D
It is advised to have 1 core per worker hence I set 3.

[DrJSAnD]

@jlaupa , thanks! I will try change parameter.