cloudapi cmon endpoint causes warning / error in scrape manager?

[teutat3s]

Hello,
thank you for this nice repo and making prometheus integrate so smoothly with TDC.

After setting this up with post-setup on TDC PI 20200103T034110Z and prometheus image mantav2-prometheus release-20200423-20200422T174819Z-g4794abe
I see the following error spamming in the prometheus log. The UUID belongs to cloudapi0.

level=warn ts=2020-04-30T12:29:52.344551219Z caller=scrape.go:836 component="scrape manager"
scrape_pool=admin_dc-1 target=https://5d980efc-cd02-4690-a871-5d8b050ec7da.cmon.dc-1.cns.example.com:9163/metrics 
msg="append failed" err="expected equal, got \"INVALID\""

I'd be happy to help debugging this further by providing more logs if needed.

[bahamat]

Can you give me the output from this?

curl -k -s --cert <certfile> --key <keyfile> https://5d980efc-cd02-4690-a871-5d8b050ec7da.cmon.dc-1.cns.example.com:9163/metrics | promtool check metrics

[teutat3s]

Hm strangely the error disappeared after a few hours. I can see it was produced in the first 6 hours after initial deployment, then no more. Actually the err was saying two things:

expected equal, got \"INVALID\"
expected label name, got \"INVALID\"

Here's the output of the above command:

cpu_sys_usage counter metrics should have "_total" suffix
cpu_user_usage counter metrics should have "_total" suffix
cpu_wait_time counter metrics should have "_total" suffix
http_requests_completed counter metrics should have "_total" suffix
net_agg_bytes_in counter metrics should have "_total" suffix
tcp_retransmitted_segment_count counter metrics should have "_total" suffix
cpucap_waiting_threads_count non-histogram and non-summary metrics should not have "_count" suffix
mem_anon_alloc_fail counter metrics should have "_total" suffix
net_agg_bytes_out counter metrics should have "_total" suffix
net_agg_packets_in counter metrics should have "_total" suffix
net_agg_packets_out counter metrics should have "_total" suffix
tcp_active_open_count counter metrics should have "_total" suffix
tcp_active_open_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_duplicate_ack_count counter metrics should have "_total" suffix
tcp_failed_connection_attempt_count counter metrics should have "_total" suffix
tcp_listen_drop_Qzero_count counter metrics should have "_total" suffix
tcp_current_established_connections_total non-counter metrics should not have "_total" suffix
tcp_duplicate_ack_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_half_open_drop_count counter metrics should have "_total" suffix
tcp_listen_drop_count counter metrics should have "_total" suffix
tcp_passive_open_count counter metrics should have "_total" suffix
tcp_retransmit_timeout_drop_count counter metrics should have "_total" suffix
time_of_day counter metrics should have "_total" suffix
vfs_bytes_read_count counter metrics should have "_total" suffix
vfs_bytes_written_count counter metrics should have "_total" suffix
vfs_read_operation_count counter metrics should have "_total" suffix
vfs_run_length_time_count counter metrics should have "_total" suffix
vfs_run_time_count counter metrics should have "_total" suffix
vfs_wait_time_count counter metrics should have "_total" suffix
tcp_failed_connection_attempt_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_half_open_drop_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_listen_drop_Qzero_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_listen_drop_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_retransmitted_segment_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_bytes_read_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_bytes_written_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_run_time_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_wait_length_time_count counter metrics should have "_total" suffix
vfs_write_operation_count counter metrics should have "_total" suffix
tcp_passive_open_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_retransmit_timeout_drop_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_read_operation_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_run_length_time_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_wait_length_time_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_wait_time_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_write_operation_count non-histogram and non-summary metrics should not have "_count" suffix

[teutat3s]

I was able to reproduce this again, here is the desired output:

curl -k -s --cert ./prometheus.cert.pem --key ./prometheus.key.pem https://5d980efc-cd02-4690-a871-5d8b050ec7da.cmon.dc-1.cns.example.com:9163/metrics | promtool check metrics
error while linting: text format parsing error in line 258: unexpected end of label value %!q(*string=0xc00024cc80)

[bahamat]

I need fresh output that’s causing the the error. The previous output isn’t useful for this. We case.