Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloudapi cmon endpoint causes warning / error in scrape manager? #19

Open
teutat3s opened this issue Apr 30, 2020 · 4 comments
Open

cloudapi cmon endpoint causes warning / error in scrape manager? #19

teutat3s opened this issue Apr 30, 2020 · 4 comments

Comments

@teutat3s
Copy link
Member

teutat3s commented Apr 30, 2020

Hello,
thank you for this nice repo and making prometheus integrate so smoothly with TDC.

After setting this up with post-setup on TDC PI 20200103T034110Z and prometheus image mantav2-prometheus release-20200423-20200422T174819Z-g4794abe
I see the following error spamming in the prometheus log. The UUID belongs to cloudapi0.

level=warn ts=2020-04-30T12:29:52.344551219Z caller=scrape.go:836 component="scrape manager"
scrape_pool=admin_dc-1 target=https://5d980efc-cd02-4690-a871-5d8b050ec7da.cmon.dc-1.cns.example.com:9163/metrics 
msg="append failed" err="expected equal, got \"INVALID\""

I'd be happy to help debugging this further by providing more logs if needed.

@bahamat
Copy link
Member

bahamat commented May 2, 2020

Can you give me the output from this?

curl -k -s --cert <certfile> --key <keyfile> https://5d980efc-cd02-4690-a871-5d8b050ec7da.cmon.dc-1.cns.example.com:9163/metrics | promtool check metrics

@teutat3s
Copy link
Member Author

teutat3s commented May 2, 2020

Hm strangely the error disappeared after a few hours. I can see it was produced in the first 6 hours after initial deployment, then no more. Actually the err was saying two things:

expected equal, got \"INVALID\"
expected label name, got \"INVALID\"

Here's the output of the above command:

cpu_sys_usage counter metrics should have "_total" suffix
cpu_user_usage counter metrics should have "_total" suffix
cpu_wait_time counter metrics should have "_total" suffix
http_requests_completed counter metrics should have "_total" suffix
net_agg_bytes_in counter metrics should have "_total" suffix
tcp_retransmitted_segment_count counter metrics should have "_total" suffix
cpucap_waiting_threads_count non-histogram and non-summary metrics should not have "_count" suffix
mem_anon_alloc_fail counter metrics should have "_total" suffix
net_agg_bytes_out counter metrics should have "_total" suffix
net_agg_packets_in counter metrics should have "_total" suffix
net_agg_packets_out counter metrics should have "_total" suffix
tcp_active_open_count counter metrics should have "_total" suffix
tcp_active_open_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_duplicate_ack_count counter metrics should have "_total" suffix
tcp_failed_connection_attempt_count counter metrics should have "_total" suffix
tcp_listen_drop_Qzero_count counter metrics should have "_total" suffix
tcp_current_established_connections_total non-counter metrics should not have "_total" suffix
tcp_duplicate_ack_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_half_open_drop_count counter metrics should have "_total" suffix
tcp_listen_drop_count counter metrics should have "_total" suffix
tcp_passive_open_count counter metrics should have "_total" suffix
tcp_retransmit_timeout_drop_count counter metrics should have "_total" suffix
time_of_day counter metrics should have "_total" suffix
vfs_bytes_read_count counter metrics should have "_total" suffix
vfs_bytes_written_count counter metrics should have "_total" suffix
vfs_read_operation_count counter metrics should have "_total" suffix
vfs_run_length_time_count counter metrics should have "_total" suffix
vfs_run_time_count counter metrics should have "_total" suffix
vfs_wait_time_count counter metrics should have "_total" suffix
tcp_failed_connection_attempt_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_half_open_drop_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_listen_drop_Qzero_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_listen_drop_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_retransmitted_segment_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_bytes_read_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_bytes_written_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_run_time_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_wait_length_time_count counter metrics should have "_total" suffix
vfs_write_operation_count counter metrics should have "_total" suffix
tcp_passive_open_count non-histogram and non-summary metrics should not have "_count" suffix
tcp_retransmit_timeout_drop_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_read_operation_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_run_length_time_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_wait_length_time_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_wait_time_count non-histogram and non-summary metrics should not have "_count" suffix
vfs_write_operation_count non-histogram and non-summary metrics should not have "_count" suffix

@teutat3s
Copy link
Member Author

I was able to reproduce this again, here is the desired output:

curl -k -s --cert ./prometheus.cert.pem --key ./prometheus.key.pem https://5d980efc-cd02-4690-a871-5d8b050ec7da.cmon.dc-1.cns.example.com:9163/metrics | promtool check metrics
error while linting: text format parsing error in line 258: unexpected end of label value %!q(*string=0xc00024cc80)

@bahamat
Copy link
Member

bahamat commented Nov 16, 2021

I need fresh output that’s causing the the error. The previous output isn’t useful for this. We case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants