Kepler on Kind

[emoek]

Dear Kepler-Community,

I am trying to run Kepler on my single-node Kind cluster on Docker but in Grafana all I can see are the node metrics, cgroup metrics, and kubelet metrics (although I did not expect the cgroups metrics to be all at the same level). More precisely, I can't see the metrics provided by eBPF.

I followed the instructions on https://sustainable-computing.io/installation/local-cluster/ to set up the kind cluster. Also, I checked my Ubuntu and it does provide RAPL metrics at "sys/class/powercap". As kind runs its nodes as a container I suspected that I do not get any results for eBPF and RAPL because of this virtualization as I also looked at the logs of the pods created for Kepler and saw that no "power meter" was recognized which makes me wonder because on the image of the node metric the source is set as "RAPL".

This is the log of the kepler pod in the default namespace where all other resources are:

E0928 14:22:44.919357 1 utils.go:117] getCPUArch failure: no CPU power model found for architecture Intel Skylake
I0928 14:22:44.932185 1 exporter.go:155] Kepler running on version: a7a6cb1
I0928 14:22:44.932618 1 config.go:258] using gCgroup ID in the BPF program: true
I0928 14:22:44.932868 1 config.go:260] kernel version: 5.15
I0928 14:22:44.932917 1 exporter.go:179] EnabledBPFBatchDelete: true
I0928 14:22:44.933317 1 power.go:64] Not able to obtain power, use estimate method
I0928 14:22:44.933618 1 redfish.go:169] failed to get redfish credential file path
I0928 14:22:44.933822 1 power.go:55] use acpi to obtain power
I0928 14:22:44.934257 1 acpi.go:67] Could not find any ACPI power meter path. Is it a VM?
I0928 14:22:44.950307 1 exporter.go:198] Initializing the GPU collector
I0928 14:22:50.953697 1 watcher.go:66] Using in cluster k8s config
W0928 14:22:50.991067 1 reflector.go:424] github.com/sustainable-computing-io/kepler/pkg/kubernetes/watcher.go:122: failed to list <unspecified>: pods is forbidden: User "system:serviceaccount:default:kepler" cannot list resource "pods" in API group "" at the cluster scope```

This is the log of the kepler-exporter pod in the kepler namespace:
I0928 11:45:41.810527       1 gpu.go:46] Failed to init nvml, err: could not init nvml: error opening libnvidia-ml.so.1: libnvidia-ml.so.1: cannot open shared object file: No such file or directory
I0928 11:45:41.862332       1 qat.go:35] Failed to init qat-telemtry err: could not get qat status exit status 127
I0928 11:45:42.144516       1 exporter.go:158] Kepler running on version: d87adfa
I0928 11:45:42.144561       1 config.go:274] using gCgroup ID in the BPF program: true
I0928 11:45:42.144846       1 config.go:276] kernel version: 5.15
I0928 11:45:42.145212       1 exporter.go:170] LibbpfBuilt: false, BccBuilt: true
I0928 11:45:42.145600       1 config.go:207] kernel source dir is set to /usr/share/kepler/kernel_sources
I0928 11:45:42.147621       1 exporter.go:189] EnabledBPFBatchDelete: true
I0928 11:45:42.149524       1 power.go:71] Unable to obtain power, use estimate method
I0928 11:45:42.151523       1 redfish.go:173] failed to initialize node credential: no supported node credential implementation
I0928 11:45:42.153562       1 acpi.go:67] Could not find any ACPI power meter path. Is it a VM?
I0928 11:45:46.113154       1 exporter.go:204] Initializing the GPU collector
I0928 11:45:52.126932       1 watcher.go:66] Using in cluster k8s config
modprobe: FATAL: Module kheaders not found in directory /lib/modules/5.15.49-linuxkit
chdir(/lib/modules/5.15.49-linuxkit/build): No such file or directory
I0928 11:45:52.328077       1 bcc_attacher.go:80] failed to attach the bpf program: <nil>
I0928 11:45:52.328123       1 bcc_attacher.go:159] failed to attach perf module with options [-DMAP_SIZE=10240 -DNUM_CPUS=2 -DSAMPLE_RATE=0 -DSET_GROUP_ID]: failed to attach the bpf program: <nil>, from default kernel source.
I0928 11:45:52.328544       1 bcc_attacher.go:162] trying to load eBPF module with kernel source dir /usr/share/kepler/kernel_sources/4.18.0-477.13.1.el8_8.x86_64
cannot attach kprobe, probe entry may not exist
I0928 11:45:53.862379       1 bcc_attacher.go:94] attaching kprobe to finish_task_switch failed, trying finish_task_switch.isra.0 instead
I0928 11:45:53.890531       1 bcc_attacher.go:129] Successfully load eBPF module from using bcc
I0928 11:45:53.890576       1 bcc_attacher.go:168] Successfully loaded eBPF module with options: [-DMAP_SIZE=10240 -DNUM_CPUS=2 -DSAMPLE_RATE=0 -DSET_GROUP_ID] from kernel source "/usr/share/kepler/kernel_sources/4.18.0-477.13.1.el8_8.x86_64"
I0928 11:45:53.890593       1 bcc_attacher.go:187] Successfully load eBPF module from bcc with option: [-DMAP_SIZE=10240 -DNUM_CPUS=2 -DSAMPLE_RATE=0 -DSET_GROUP_ID]
I0928 11:45:53.919034       1 container_energy.go:109] Using the Ratio/DynPower Power Model to estimate Container Platform Power
I0928 11:45:53.919084       1 container_energy.go:118] Using the Ratio/DynPower Power Model to estimate Container Component Power
I0928 11:45:53.919093       1 process_power.go:108] Using the Ratio/DynPower Power Model to estimate Process Platform Power
I0928 11:45:53.919106       1 process_power.go:117] Using the Ratio/DynPower Power Model to estimate Process Component Power
I0928 11:45:53.920321       1 node_platform_energy.go:53] Using the LinearRegressor/AbsPower Power Model to estimate Node Platform Power
I0928 11:45:53.920868       1 node_component_energy.go:54] Using the LinearRegressor/AbsPower Power Model to estimate Node Component Power
I0928 11:45:53.922190       1 exporter.go:276] Started Kepler in 11.779599684s



So I would like to know if, when using Kind, it is not possible to get RAPL metrics and also make use of eBPF or do I have to mount all that additionally and it is not provided in the installation instructions? 

I am relatively new in this field and would therefore appreciate any feedback!

[rootfs]

how did you install kepler? have you deployed ServiceMonitor?

[rootfs]

do you find RAPL readings on your machine? The sys paths are here

[emoek]

I have them in sys/class/powercap on my host but not in the Kind Cluster, the powercap folder is missing. Should I mount the powercap folder additionally or should it somehow be linked to /proc?

[rootfs]

i see, that makes sense. Can you add /sys/class from your host to kind cluster? If this works for you, can you update the doc and manifests as well? thanks

[emoek]

ok I solved it as follows:

• added mount of /sys/devices/virtual/powercap to sys/class/powercap (because sys/class/powercap locally is just a symlink)
• change permission on host of energy_uj files for all folders in sys starting with intel-rapl , I can provide a bash script for this

It seems to work as visible in the logs and the image:

I1001 18:16:45.087261 1 config.go:297] The Idle power will be exposed. Are you running on Baremetal or using single VM per node?
I1001 18:16:45.087282 1 exporter.go:170] LibbpfBuilt: true, BccBuilt: false
I1001 18:16:45.087299 1 exporter.go:189] EnabledBPFBatchDelete: true
I1001 18:16:45.088300 1 power.go:54] use sysfs to obtain power
I1001 18:16:45.088495 1 redfish.go:173] failed to initialize node credential: no supported node credential implementation
I1001 18:16:45.088524 1 power.go:56] use acpi to obtain power
I1001 18:16:45.088910 1 acpi.go:67] Could not find any ACPI power meter path. Is it a VM?
I1001 18:16:45.157917 1 exporter.go:204] Initializing the GPU collector
I1001 18:16:51.167422 1 watcher.go:66] Using in cluster k8s config

If the energy values seem valid and the solution is appropriate, I would update the docs and manifests. Where exactly should I add the manifests? under manifests/config ?

[rootfs]

@emoek that's great!

For kind patch, please find the extraMount here also cc @SamYuan1990 @vprashar2929

For kepler doc, please the KIND instruction here cc @husky-parul