I can't get any metric

[misago162]

Hello

I have been tested this exporter in a machine with an openstack and in this machine, this exporter has worked fine. This openstack is a test environment with only one computer.

Then, I have installed in our production openstack environment without success, and I'm not able to see the problem. I can't get any metric. In this openstack environment we have 3 controller servers:

controller2: manage the network
controller3: manage the storage
controller1: manage the rest of the openstack components

I have installed the exporter in the controller1 with a dedicated openstack user to avoid use the admin user.

If from the own controller1 machine I try to get the metric I can see an 500 error:

# curl -v http://localhost:9183/metrics
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9183 (#0)
> GET /metrics HTTP/1.1
> Host: localhost:9183
> User-Agent: curl/7.61.1
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 500 Internal Server Error
< Server: BaseHTTP/0.6 Python/3.6.8
< Date: Thu, 28 Jul 2022 11:26:11 GMT
< 
* Closing connection 0
[root@nccontroller1 log]# 

This is the content of the prometheus-openstack-exporter.yaml file (although we have swift, it is commented just as a test to check if it was the problem):

[root@controller1 ~]# cat /etc/prometheus-openstack-exporter/prometheus-openstack-exporter.yaml 
# Example configuration file for prometheus-openstack-exporter
# Copyright (C) 2016-2019 Canonical, Ltd.
#

listen_port: 9183
cache_refresh_interval: 300  # In seconds
cache_file: /var/cache/prometheus-openstack-exporter/mycloud
cloud: mycloud
openstack_allocation_ratio_vcpu: 2.5
openstack_allocation_ratio_ram: 1.1
openstack_allocation_ratio_disk: 1.0
log_level: DEBUG

# Configure the enabled collectors here.  Note that the Swift account
# collector in particular has special requirements.
enabled_collectors:
  - cinder
  - neutron
  - nova
###  - swift
###  - swift-account-usage

# To export hypervisor_schedulable_instances metric set desired instance size
schedulable_instance_size:
    ram_mbs: 4096
    vcpu: 2
    disk_gbs: 20

# Uncomment if the cloud doesn't provide cinder / nova volumes:
#use_nova_volumes: False

## Swift

# There is no way to retrieve them using OpenStack APIs
# For clouds deployed without swift, remove this part
###swift_hosts:
###    - swift.xxx.es 

###    - export1 172.16.4.225:8080
###    - export2 172.16.4.226:8080
###    - export3 172.16.4.227:8080
###    - export4 172.16.4.228:8080

# There is no API to ask Swift for a list of accounts it knows about.
# Even if there were, Swift (in common case of Keystone auth, at
# least) only knows them by the corresponding tenant ID, which would
# be a less than useful label without post-processing.  The following
# should point to a file containing one line per tenant, with the
# tenant name first, then whitespace, followed by the tenant ID.
keystone_tenants_map:

# The reseller prefix is typically used by the Swift middleware to
# keep accounts with different providers separate.  We would ideally
# look this up dynamically from the Swift configuration.
# The Keystone middlware defaults to the following value.
reseller_prefix: AUTH_

ring_path: /etc/swift

# These will typically be read from /etc/swift/swift.conf.  If that
# file cannot be opened, then the Swift library will log an error and
# try to exit.  To run p-s-a-e as a user other than Swift, these
# settings must be set to the same values as Swift itself, and the
# above must point to an always-current readable copy of the rings.

hash_path_prefix:
hash_path_suffix:

[root@controller1 ~]# 

This is the content of the admin.novarc file

[root@controller1 ~]# cat /etc/prometheus-openstack-exporter/admin.novarc 
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
#export OS_TENANT_NAME=admin
export OS_USERNAME=prometheus-exporter
export OS_PASSWORD=XXXXXXXX
export OS_AUTH_URL=https://xxxxx.xxx.xx:5000/v3
#export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
[root@controller1 ~]# 

And this is the log info when the service is running, nothing happens when I try to get the metrics. Note that in the development openstack, we get the same problem with the list_projects but it works fine, we can get the metrics.

Jul 28 13:28:51 nccontroller1 systemd[1]: Started prometheus-openstack-exporter.
Jul 28 13:28:53 nccontroller1 python3[32042]: Starting data gather thread
Jul 28 13:28:53 nccontroller1 python3[32042]: Client setup done, keystone ver 3
Jul 28 13:28:53 nccontroller1 python3[32042]: Error getting tenants.list, continue with projects.list
Jul 28 13:28:53 nccontroller1 python3[32042]: Error getting stats: Traceback (most recent call last):
                                                File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 186, in _get_keystone_info
                                                  info["tenants"] = [x._info for x in keystone.tenants.list()]
                                                File "/usr/lib/python3.6/site-packages/keystoneclient/httpclient.py", line 893, in __getattr__
                                                  raise AttributeError(_("Unknown Attribute: %s") % name)
                                              AttributeError: Unknown Attribute: tenants
                                              
                                              During handling of the above exception, another exception occurred:
                                              
                                              Traceback (most recent call last):
                                                File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 262, in run
                                                  prodstack.update(self._get_keystone_info(keystone))
                                                File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 189, in _get_keystone_info
                                                  info["tenants"] = [x._info for x in keystone.projects.list()]
                                                File "/usr/lib/python3.6/site-packages/keystoneclient/v3/projects.py", line 142, in list
                                                  **kwargs)
                                                File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
                                                  return f(*args, **new_kwargs)
                                                File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 448, in list
                                                  list_resp = self._list(url_query, self.collection_key)
                                                File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
                                                  resp, body = self.client.get(url, **kwargs)
                                                File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get
                                                  return self.request(url, 'GET', **kwargs)
                                                File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 545, in request
                                                  resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
                                                File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request
                                                  return self.session.request(url, method, **kwargs)
                                                File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 943, in request
                                                  raise exceptions.from_response(resp, method, url)
                                              keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_projects. (HTTP 403) (Request-ID: req-c1121018-2efb-4768-bddb-cfdd6558822f)

Tell me if you need I upload more information.

Any help to find the problem would be appreciated. Thanks a lot.

[misago162]

Hello, any advice or idea about where is the problem? Thanks a lot

[jagter]

The OpenStack user prometheus-exporter doesn't have the permissions to do a list of projects.