| *** ysandeep|out is now known as ysandeep | 04:31 | |
| *** jpena|off is now known as jpena | 07:27 | |
| *** ysandeep is now known as ysandeep|lunch | 08:12 | |
| *** ysandeep|lunch is now known as ysandeep | 08:52 | |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: DNM Add new information for Opensearch cluster configuration https://review.opendev.org/c/openstack/ci-log-processing/+/839507 | 10:03 |
|---|---|---|
| opendevreview | Merged openstack/ci-log-processing master: Add Opensearch configuration information; change README file; enable md https://review.opendev.org/c/openstack/ci-log-processing/+/832659 | 10:14 |
| *** rlandy|out is now known as rlandy | 10:21 | |
| *** dviroel|rover|out is now known as dviroel|rover | 11:16 | |
| *** ysandeep is now known as ysandeep|afk | 12:42 | |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Send json message for performance.json file instead of string https://review.opendev.org/c/openstack/ci-log-processing/+/839522 | 12:58 |
| *** ysandeep|afk is now known as ysandeep | 13:00 | |
| dpawlik7 | dansmith: hey, by doing query ' message: "ubuntu-focal-ovh-bhs1-0029453689" and tags: performance' the performance.json file seems that it returns wrong values for api key for services like aodh_access or mistral_api_access | 13:37 |
| dpawlik7 | dansmith: the json file https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3db/837084/3/check/tacker-compliance-devstack-multinode-sol/3db2e4a/controller/logs/performance.json | 13:37 |
| *** dasm|off is now known as dasm | 13:48 | |
| dansmith | dpawlik: yeah, that's the case for at least neutron, which is mounted at the root instead of under /network like the rest of the services | 14:14 |
| dansmith | so I imagine aodh in that case is doing something similar | 14:14 |
| dansmith | I pinged gmann about the neutron case before, but never heard back (re-ping gmann) | 14:15 |
| dansmith | gmann: basically, neutron is "mounted" at the root of the tlsproxy, so we hit it as /v2.0/networks/$id, instead of /network/v2.0/networks/$id | 14:17 |
| *** ysandeep is now known as ysandeep|out | 14:21 | |
| dpawlik | dansmith: I'm wondering if it is possible to get such informations like RAM/CPU utilization for each Zuul job. I don't see such feature in Zuul and it spawn the instances... do you know if nova or qemu can provide such informations? | 14:25 |
| dansmith | dpawlik: those are highly variable for each run, and are kinda the anti-point of this work | 14:25 |
| dansmith | we've done a lot in the past to try to collect and normalize those sorts of measurements, but it's *incredibly* difficult | 14:25 |
| dansmith | we have dstat which monitors that stuff to help in looking for runaway processes and things, but otherwise it's very hard | 14:26 |
| dpawlik | dansmith: I can imagine | 14:26 |
| dansmith | even normalizing these "countable" resources is turning out harder than I expected.. I'm trying to figure out why the api and db call counts vary so much between two identical runs | 14:26 |
| dpawlik | dansmith: your tool will be not working on normal job like tox-docs, right? | 14:30 |
| dansmith | yeah, it's just devstack jobs | 14:30 |
| dpawlik | fungi: wdyt to add a simple pre job that is taking information about disk space, ram, cpu utilization and on the end post job that is calculating such information to a log file? | 14:32 |
| fungi | dpawlik: we do already put some of that in a file at the start of most jobs which we collect with the logs, though it's just unmunged output from diagnostic tools not structured data: https://zuul.opendev.org/t/openstack/build/3fe518d5d739439a994b2c096bdac3ad/log/zuul-info/zuul-info.ubuntu-focal.txt | 14:53 |
| fungi | it's more for diagnosing when a specific build goes sideways than for trending over multiple builds | 14:53 |
| fungi | what's the use case of trending that information? | 14:53 |
| fungi | there's also an ansible host info dump in yaml: https://zuul.opendev.org/t/openstack/build/3fe518d5d739439a994b2c096bdac3ad/log/zuul-info/host-info.ubuntu-focal.yaml | 14:55 |
| fungi | from that you can get things like cpu and ram counts | 14:55 |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: WIP Add logextractor tool https://review.opendev.org/c/openstack/ci-log-processing/+/839570 | 15:06 |
| dpawlik | dansmith: Poc for tool that is taking performance content and send to another index - https://review.opendev.org/c/openstack/ci-log-processing/+/839570 | 15:09 |
| dansmith | dpawlik: cool | 15:09 |
| dpawlik | fungi: aha, will check that | 15:09 |
| dpawlik | thanks | 15:09 |
| clarkb | dansmith: note that dstat was removed beacuse it is no longer maintained and the system that replaces it signiifncaly more complicated and doesn't even install reliably from distro packages | 15:19 |
| dansmith | clarkb: ah didn't realize | 15:19 |
| clarkb | performance co pilot or similar is the new tool. It installs a ton of daemons and the daemons don't reliably start causing package install to fail | 15:20 |
| clarkb | a bit sad considering how simple dstat was | 15:20 |
| dansmith | that's disappointing | 15:20 |
| dansmith | yeah | 15:20 |
| clarkb | dpawlik: re tracking this info for all zuul jobs: we've essentialyl said we don't have the help to make that happen and that is why services like the ELK service got shutdown | 15:23 |
| clarkb | at one time we did think that would be a very useful feature to tie into the CI system but not enough people agreed to help us make it happen | 15:23 |
| clarkb | The good news is nothing prevents you from doing it for the jobs you care about if that is something you want to do on a smaller scale | 15:23 |
| fungi | and also see if the validate-hosts role in zuul/zuul-jobs could be improved to include more baseline info you're interested in | 15:26 |
| fungi | that's what creates the files i pointed out | 15:26 |
| fungi | as long as the commands to get that information are lightweight and won't significantly increase job runtime, it's probably cool | 15:27 |
| fungi | (and won't destabilize jobs of course) | 15:27 |
| *** dviroel|rover is now known as dviroel|rover|lunch | 15:29 | |
| fzzf[m] | hi, nodepol use openstack provider.... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/SrNPNkNWqKYAKvcAaMhSUXBZ) | 15:40 |
| clarkb | fzzf[m]: nova should use the default security group by default | 16:02 |
| clarkb | and for networks if you can nova boot by hand without setting network info then you shouldn't need to set network info in nodepool. However some clouds do require you specify the networks to attach to an instance and if so then you need to supply them in your nodepool config | 16:02 |
| *** dviroel|rover|lunch is now known as dviroel|rover | 16:18 | |
| gmann | dansmith: sorry, I might have missed the neutron mount ping. you mean this part you are fixing right? https://review.opendev.org/c/openstack/devstack/+/839067/5/tools/get-stats.py#137 | 16:21 |
| dansmith | gmann: no.. it appears that tlsproxy does not put neutron under /network like nova is under /compute or keystone under /identity | 16:22 |
| dansmith | which means in the accounting, neutron shows up as "v2.0" instead of "network" because I cleave off the first part of the url to determine the "service name" | 16:23 |
| gmann | yeah, so in 839067 will fetch the 2nd part right. we may have such inconsistency in other services with devstack plugins. may be we can out 'url' there instead of "service name" | 16:26 |
| gmann | but not sure how different that is for other services | 16:27 |
| dansmith | gmann: well, I'm saying I think we need to fix that in devstack so neutron is under /network for consistency | 16:27 |
| gmann | yeah we can do that also. | 16:28 |
| dansmith | gmann: neutron gets its own special tls proxy config different from others: https://zuul.opendev.org/t/openstack/build/149de9e390364cb6b1d3c708be753a3c/log/controller/logs/apache_config/neutron-tls-proxy_conf.txt | 16:28 |
| dansmith | I guess because of the port, even though I don't think we see that in the log | 16:28 |
| dansmith | gmann: anyway, this is not critical right now, I just noticed it so at some point we'll need to either hack in something to the perf stats thing, or fix how neutron deploys in devstack | 16:29 |
| gmann | ok | 16:36 |
| *** jpena is now known as jpena|off | 17:01 | |
| opendevreview | Merged openstack/project-config master: Finalize ELK puppetry retirement https://review.opendev.org/c/openstack/project-config/+/839243 | 19:44 |
| *** rlandy is now known as rlandy|mtg | 20:28 | |
| *** dviroel|rover is now known as dviroel|rover|biab | 21:12 | |
| *** rlandy|mtg is now known as rlandy | 21:27 | |
| *** rlandy is now known as rlandy|bbl | 22:10 | |
| *** dasm is now known as dasm|off | 22:19 | |
| *** dviroel|rover|biab is now known as dviroel|rover | 22:31 | |
| *** dviroel|rover is now known as dviroel|rover|out | 22:54 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!