| opendevreview | Jan Gutter proposed openstack/devstack master: Update etcd version to 3.4.27 https://review.opendev.org/c/openstack/devstack/+/891353 | 07:49 |
|---|---|---|
| frickler | gmann: kopecmartin: freezer is another global_venv victim. do we want to add symlinks to devstack for non-core projects, too, or should we expect this to be done in the project's devstack plugin? https://zuul.opendev.org/t/openstack/build/8eb8ee33b26a42229a7c0f18118a8763 | 08:27 |
| frickler | makes me wonder how many of these near-dead projects we might have been missing. it also raises the question of who should be considered responsible for doing such adaptions | 08:35 |
| kopecmartin | frickler: how many projects are we talking about? .. in general, I'd say this can be used as a push to either update the projects or retire them .. if no one complains that the projects stopped working with devstack, they're in fact dead and making any changes to support them isn't cost effective | 09:35 |
| frickler | kopecmartin: how many projects are affected is difficult to tell, since they tend to not run any jobs, so no way to tell whether things are failing | 10:04 |
| frickler | if you take a look at how many zuul config errors there still are remaining one year after the queue config change, and how many I had to fix for others, maybe >50% of OpenStack? https://review.opendev.org/q/topic:fix-zuul-errors | 10:05 |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Aug 15 15:00:19 2023 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours | 15:00 |
| lpiwowar | o/ | 15:00 |
| kopecmartin | o/ | 15:01 |
| kopecmartin | let's get to it | 15:03 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:03 |
| kopecmartin | tomorrow PTL and TC nominations start | 15:03 |
| kopecmartin | #link https://governance.openstack.org/election/ | 15:03 |
| kopecmartin | let me know if anyone wanted to be PTL for QA | 15:04 |
| kopecmartin | if there won't be any candidates, i'll probably continue | 15:04 |
| kopecmartin | #topic Bobcat Priority Items progress | 15:05 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-bobcat-priority | 15:05 |
| lpiwowar | I would be definitely for keeping you as a PTL!:) | 15:05 |
| kopecmartin | :) | 15:05 |
| * kopecmartin checking the doc | 15:05 | |
| kopecmartin | last week we merged the 'Add option to install everything in global venvs' | 15:06 |
| kopecmartin | #link https://review.opendev.org/c/openstack/devstack/+/558930 | 15:06 |
| kopecmartin | that broke a few projects, e.g. freezer | 15:07 |
| kopecmartin | #link https://zuul.opendev.org/t/openstack/build/8eb8ee33b26a42229a7c0f18118a8763 | 15:07 |
| kopecmartin | however, it seems freezer isn't active, there haven't been updates in the project for some time | 15:07 |
| kopecmartin | thanks to the patch , which had impact on multiple projects, we'll see how many projects are still active and needed by users | 15:08 |
| frickler | it also broke nova, neutron and glance | 15:08 |
| frickler | but I fixed those already | 15:08 |
| kopecmartin | right, thanks for that | 15:08 |
| kopecmartin | if anyone hits a similar issue like the one above, please, let us know in this channel | 15:09 |
| kopecmartin | e.g. | 15:09 |
| kopecmartin | FileNotFoundError: [Errno 2] No such file or directory: 'freezer' | 15:09 |
| frickler | or "command not found", although we have that way too often even in passing jobs, too | 15:10 |
| kopecmartin | the patch which added debian-bookworm job to devstack was merged as well | 15:10 |
| kopecmartin | #link https://review.opendev.org/c/openstack/devstack/+/887547/7 | 15:10 |
| kopecmartin | does it mean this priority item is finished? - 'Support for Debian 12 (bookworm) in devstack' | 15:10 |
| frickler | and gmann proposed a PTI patch that will make it the default platform for 2024.1 | 15:11 |
| frickler | I think it is finished module the ongoing cleanup work, yes | 15:11 |
| frickler | *modulo | 15:12 |
| kopecmartin | nice | 15:12 |
| kopecmartin | #topic Gate Status Checks | 15:13 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:13 |
| kopecmartin | no patches there | 15:13 |
| kopecmartin | anything urgent to review? | 15:13 |
| lpiwowar | This is not urgent => https://review.opendev.org/c/openstack/tempest/+/890798 ... But if someone has time to comment and do the review it would be great:) | 15:14 |
| kopecmartin | sure | 15:15 |
| lpiwowar | thanks! | 15:15 |
| kopecmartin | #topic Bare rechecks | 15:17 |
| kopecmartin | #link https://etherpad.opendev.org/p/recheck-weekly-summary | 15:17 |
| kopecmartin | no updates since last week | 15:17 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:17 |
| kopecmartin | periodic stable full | 15:18 |
| kopecmartin | #link https://zuul.openstack.org/builds?pipeline=periodic-stable&job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-zed&job_name=tempest-full-2023-1 | 15:18 |
| kopecmartin | periodic stable slow | 15:18 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-slow-2023-1&job_name=tempest-slow-zed&job_name=tempest-slow-yoga&job_name=tempest-slow-xena | 15:18 |
| kopecmartin | periodic extra tests | 15:18 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-2023-1-extra-tests&job_name=tempest-full-zed-extra-tests&job_name=tempest-full-yoga-extra-tests&job_name=tempest-full-xena-extra-tests | 15:18 |
| kopecmartin | periodic master | 15:18 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:18 |
| kopecmartin | tempest-full-centos-9-stream is broken | 15:18 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream | 15:18 |
| kopecmartin | Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Internal Server Error (HTTP 500) | 15:19 |
| kopecmartin | although i'm not sure if that's the original cause | 15:19 |
| frickler | Aug 12 03:17:04.971803 np0034931512 devstack@keystone.service[44769]: failed to open python file /opt/stack/data/venv/bin/keystone-wsgi-public | 15:20 |
| frickler | I think global_venv is still broken on c9s, so the flag needs to be set in tempest to disable it | 15:21 |
| kopecmartin | thanks! i'll propose the patch | 15:22 |
| frickler | not sure why tempest doesn't inherit this https://review.opendev.org/c/openstack/devstack/+/558930/38/.zuul.yaml#703 | 15:22 |
| kopecmartin | hmm, i'll check the inheritance path | 15:23 |
| frickler | do we want a periodic job for bookworm, too? seems I didn't add one yet | 15:24 |
| kopecmartin | hm, do we have other distros in periodic too? | 15:25 |
| kopecmartin | that reminds me, i should add bookworm to the distros check in the agenda | 15:25 |
| frickler | well we have this centos job | 15:26 |
| kopecmartin | right, i guess it won't hurt | 15:27 |
| kopecmartin | #topic Distros check | 15:28 |
| kopecmartin | cs-9 | 15:28 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:28 |
| kopecmartin | debian | 15:28 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0 | 15:28 |
| kopecmartin | focal | 15:28 |
| kopecmartin | #link https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-ubuntu-focal&skip=0 | 15:28 |
| kopecmartin | rocky | 15:28 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx | 15:28 |
| kopecmartin | openEuler | 15:28 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-22.03-ovn-source&job_name=devstack-platform-openEuler-22.03-ovs&skip=0 | 15:28 |
| kopecmartin | all seems more or less good | 15:30 |
| kopecmartin | #topic Sub Teams highlights | 15:30 |
| kopecmartin | Changes with Review-Priority == +1 | 15:30 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:30 |
| kopecmartin | no patches | 15:30 |
| kopecmartin | #topic Open Discussion | 15:30 |
| kopecmartin | anything for the open discussion? | 15:30 |
| lpiwowar | I have one not that important question. | 15:31 |
| lpiwowar | Would it be OK to extend account-generator to generate system scoped admin? | 15:31 |
| lpiwowar | Currently, it does not generate it and it causes failure in the tempest-full-test-account-py3 job. | 15:31 |
| lpiwowar | #link https://baadd3700d0f959a2148-132839ae31c66569d89d81463638a953.ssl.cf1.rackcdn.com/890798/7/check/tempest-full-test-account-py3/1e6521b/testr_results.html | 15:31 |
| kopecmartin | hmm, system-scoped admin , isn't it deprecated? | 15:32 |
| frickler | that url looks baadd ... *scnr* | 15:32 |
| lpiwowar | So we should drop the system scouped admin in the test that is failing? | 15:32 |
| kopecmartin | :D | 15:32 |
| lpiwowar | :D | 15:33 |
| * kopecmartin trying to find the srbac doc, one sec | 15:33 | |
| frickler | maybe ask gmann as our srbac specialist | 15:33 |
| lpiwowar | frickler: +1 | 15:33 |
| kopecmartin | try to check this, maybe there's something which would suggest what to do with that | 15:35 |
| kopecmartin | #link https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html | 15:35 |
| lpiwowar | Yes, I know this doc. I will try to take a look at this again:). | 15:35 |
| kopecmartin | phase1 | 15:37 |
| kopecmartin | ... "Keeping everything as project scoped will make sure to fail the operations performed with a system scoped token (which does not have project_id) early with 403 instead of failing it with 500 in the lower layer." ... | 15:37 |
| kopecmartin | that refers to system-scoped token in general, does it include the admin one too? my assumption would be yes | 15:38 |
| kopecmartin | but better to double check | 15:38 |
| kopecmartin | oh, can't the test fail because of that ^^ | 15:38 |
| kopecmartin | ? | 15:38 |
| lpiwowar | The test is failing because of missing system scoped admin creds in the accounts.yaml. | 15:39 |
| lpiwowar | https://opendev.org/openstack/tempest/src/branch/master/tempest/scenario/test_unified_limits.py#L32 | 15:39 |
| lpiwowar | "Details: No credentials matching role: admin, scope: system specified in the accounts file" | 15:39 |
| frickler | maybe the test should be skipped in that case, instead of failing? | 15:40 |
| lpiwowar | Dynamic creds will create the system scoped admin on the go ... | 15:40 |
| lpiwowar | frickler: skipped when pre-prov creds are used? | 15:40 |
| frickler | skipped when "No credentials matching role: admin, scope: system specified in the accounts file" | 15:41 |
| lpiwowar | frickler: ok, that sounds good to me:) | 15:41 |
| frickler | that's it then? | 15:43 |
| lpiwowar | frickler, kopecmartin: +1, thanks for the help! | 15:44 |
| kopecmartin | lpiwowar: you can try to propose a wip patch to see what happens if you add 'admin' to the test's credentials | 15:44 |
| kopecmartin | and meanwhile figure out what's right, not just working, solution, i'd try follow up with gmann | 15:45 |
| kopecmartin | let's move on | 15:45 |
| kopecmartin | #topic Bug Triage | 15:45 |
| kopecmartin | #link https://etherpad.openstack.org/p/qa-bug-triage-bobcat | 15:45 |
| kopecmartin | i've recorded the numbers but haven't had time to go through the bugs :/ | 15:45 |
| kopecmartin | the numbers look stable though | 15:46 |
| kopecmartin | anyway, that's all from my side | 15:46 |
| lpiwowar | I just want to quickly mention this. Last week I talked with professor Corey we talked with at the PTG in Vancouver kopecmartin. It seems that we might have a group of students working on Tempest starting next year. I just wanted to inform everyone that we could potentially have some skilled students joining us to lend a hand :) | 15:46 |
| kopecmartin | in theory sounds good, in reality it will require quite a lot of work on management as the students will require guidance, however, it will be awesome if they help us with something | 15:49 |
| kopecmartin | we're always looking for new contributors | 15:49 |
| lpiwowar | From what I heard from the professor Corey I think that the students might help us a lot:). | 15:50 |
| kopecmartin | that'd be great, thanks for the initiative lpiwowar, i'm looking forward to some results :) | 15:50 |
| kopecmartin | let's wrap it up, i still need to propose one or 2 patches before my eod (9 minutes :D ) | 15:51 |
| lpiwowar | kopecmartin: I will keep you updated:) | 15:51 |
| kopecmartin | thanks | 15:51 |
| lpiwowar | kopecmartin: gl :D | 15:51 |
| kopecmartin | thank you everyone, see you next week | 15:51 |
| kopecmartin | #endmeeting | 15:51 |
| opendevmeet | Meeting ended Tue Aug 15 15:51:38 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:51 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-15-15.00.html | 15:51 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-15-15.00.txt | 15:51 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2023/qa.2023-08-15-15.00.log.html | 15:51 |
| gmann | lpiwowar: for RBAC, system scope are dropped except ironic and keystone. do you need that for those? | 15:58 |
| opendevreview | Martin Kopec proposed openstack/tempest master: Remove Glance v1 APIs tests https://review.opendev.org/c/openstack/tempest/+/890592 | 15:58 |
| dansmith | are we running out of disk space on workers? https://zuul.opendev.org/t/openstack/build/cb4edb50665247f99e0111f96817190f/log/job-output.txt#21519 | 16:02 |
| opendevreview | Martin Kopec proposed openstack/tempest master: Centos-9 isn't yet compatible with global_venv https://review.opendev.org/c/openstack/tempest/+/891517 | 16:03 |
| gmann | lpiwowar: we can support that in account generator file but that need more fixes than just system scope | 16:05 |
| dansmith | here's a different ENOSPC manifestation: https://zuul.opendev.org/t/openstack/build/3c3e8c2364a94ccfb155d5e9829a4b47 | 16:21 |
| elodilles | gmann kopecmartin : if you could have a quick look at this rocky-eol patch, that would be awesome: https://review.opendev.org/c/openstack/releases/+/887948 | 16:25 |
| clarkb | dansmith: those errors are on the zuul side since logs have to be copied from test machines to the executors in order to upload from there. Zuul does limit per build disk usage but I think that isn't with quotas and is instead via a monitor that will kill builds early instead so you'd get a different error in that case | 16:26 |
| dansmith | clarkb: okay I dunno what to make of that :) | 16:26 |
| clarkb | basically someone with zuul admin access needs to go look at those servers cc fungi | 16:27 |
| clarkb | looking at disk utilization of ze12 and ze10 graphs there is a definite jump in usage | 16:27 |
| clarkb | zuul itself won't have upgraded yet for the week so this is unlikely to be changes to the zuul service and instead is some sort of change on the job side using more disk | 16:27 |
| clarkb | I haven't had breakfast yet or loaded ssh keys so will be a bit before I can look more closely | 16:28 |
| clarkb | looks like a wide spread thing across all executors though | 16:28 |
| lpiwowar | gmann: The failing test works with keystone and glance. I'm not sure whether system scope is really needed there. I can check. | 16:30 |
| dansmith | clarkb: ack thanks | 16:33 |
| gmann | lpiwowar: glance does not need system scope. keystone does. | 16:50 |
| gmann | elodilles: thanks. done | 16:54 |
| -opendevstatus- NOTICE: Zuul job execution is temporarily paused while we rearrange local storage on the servers | 16:55 | |
| elodilles | gmann: thanks too! | 16:56 |
| -opendevstatus- NOTICE: Zuul job execution has resumed with additional disk space on the servers | 17:44 | |
| fungi | dansmith: hopefully no more enospc errors | 17:52 |
| dansmith | fungi: many thanks | 17:53 |
| fungi | when replacing the executors some time back we missed that we had the service data on a separate disk and didn't set the new servers up that way, so they've been chugging along putting it on the 40gb rootfs. surprisingly it only just now started to become a problem | 17:54 |
| dansmith | I'm sure we've all been there :P | 17:54 |
| fungi | now the executors have a dedicated 70gb filesystem for that data | 17:55 |
| fungi | we rejiggered all 12 in under an hour. teamwork makes the dream work | 17:55 |
| fungi | thankfully the executors have a disk utilization governor, so for the most part they avoided taking builds (which was severely limiting throughput but also at least meant very few builds that ran failed to have space) | 17:57 |
| fungi | on https://grafana.opendev.org/d/21a6e53ea4/zuul-status you can see in the executors graph where they stopped accepting most builds around 14:50 in order to avoid running out of disk | 18:00 |
| fungi | also the used hdd graph tells most of the story | 18:00 |
| fungi | in https://grafana.opendev.org/d/21a6e53ea4/zuul-status?orgId=1&viewPanel=26&from=now-90d&to=now you can see where the executors were replaced in the first two weeks of july, we just didn't notice the jump in the graph | 18:03 |
| melwitt | just saw an interesting fail on one of my patches "rsync: [receiver] write failed on "/var/lib/zuul/builds/690f066ff653490e8e0deaba0291e2a6/work/logs/controller/logs/devstack.journal.gz": No space left on device (28)" | 18:12 |
| clarkb | melwitt: ya thats the issue dansmith was talking about above. It should be better now. If you see those occuring with timestamps after ~17:37UTC let us know | 18:15 |
| melwitt | clarkb: cool, thanks. I searched for "no space" in my client and it didn't come back with anything so I thought it was new .. mah bad | 18:18 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!