Monday, 2022-12-12

opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
*** yadnesh|away is now known as yadnesh04:34
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
*** jpena|off is now known as jpena08:43
opendevreviewTim Beermann proposed zuul/zuul-jobs master: Add yamllint job.
*** rlandy|out is now known as rlandy11:16
*** dviroel_ is now known as dviroel|rover11:20
*** anbanerj is now known as frenzy_friday11:35
*** pojadhav is now known as pojadhav|afk11:39
*** rlandy is now known as rlandy|brb13:17
*** rlandy|brb is now known as rlandy13:27
*** pojadhav|afk is now known as pojadhav13:40
jm1fungi: hello :) last year you wrote this patch: today i tried to use this patch but the galaxy directory is missing on mirrors, e.g.
jm1fungi: has this galaxy mirroring been abandoned?13:44
jm1fungi: or is it simply still-work-in-progress?13:44
jm1fungi: this is what i tried to use opendev's mirror for ansible galaxy:
jm1Tengu: ^13:45
Tengujm1: you're missing the port (as answered in #tripleo)13:47
Tengujm1: I pushed a patch in order to correct the way galaxy was served - and it requires a dedicated vhost due to some sad redirect.13:48
Tengujm1: namely:
fungijm1: right, turns out that wasn't actually proxying/caching anything other than a redirect, but Tengu has replaced it with a fully functional caching proxy solution (required an alternate port number due to design choices of the galaxy site implementation)13:51
fungifor the benefit of anyone reading this channel's log, it's port 8085 for plain http or 4448 for https connections to the proxy13:52
jm1Tengu, fungi: Oh ok, thank you very much!13:52
jm1Tengu, fungi: so instead of one should be able to use e.g. ?13:52
Tenguyep, something like that13:53
Tengujm1: there's an example patch already up here:   but I need to take some time to debug it (couldn't for now, had some other issues on some other topics)13:54
jm1Tengu: does downloading the file from opendev's mirror above work for you?13:56
fungithat exact url does not work for me, no13:56
Tengujm1: you mean on my local laptop?13:56
fungilooks like that doesn't work because the proxy expects to replace that url in responses13:58
fungino, don't mind me i'm misreading the patch13:59
*** Guest1665 is now known as dasm14:00
fungilooks like that's ultimately redirected to
jm1Tengu, fungi: i was expecting to be able to download from that mirror on my local system but it redirects me to ansible galaxy Page not found error instead. 14:01
*** dasm is now known as Guest189214:01
*** Guest1892 is now known as dasm14:01
jm1Tengu, fungi: both jobs that fail with RETRY_LIMIT on also fail when downloading from galaxy14:01
fungiwhereas using the actual galaxy site ends up at
Tengu(sorry, on a call)14:02
fungilooking at the apache config, it does appear that gets forwarded to and the rest of the url seems to match (aside from differing signatures coming from the redirect, it seems)14:08
fungistrangely, when i try to directly download the url the galaxy site redirects to, that also breaks14:12
fungiahh, no i had to grab a fresh url, seems the expiration on those is very short and you get a new one each time14:15
fungihowever, substituting the first part of the url with one of our mirrors still gives me an html response full of javascript links instead14:16
fungiwith the same "page not found" if i try loading it in my browser14:17
Tenguhmmm... do they check some headers now...14:20
opendevreviewTim Beermann proposed zuul/zuul-jobs master: Add yamllint job.
jm1Tengu, fungi: do we need a dedicated vhost for galaxy s3 as well? ๐Ÿคจ14:26
Tengumaybe we can refrain re-writing the URI to galaxy-s3. still on a call.14:27
jm1Tengu: take your time, its not urgent. great that you are actually pushing that galaxy mirror forward. Thank you :)14:27
TenguI may work on that part during my off-time the 2 next weeks. depends on the load.14:28
jm1Tengu: cool ๐Ÿ˜Ž14:30
jm1fungi: thank you too for testing that out. i wasnt sure if it was my local system only and whether we can actually use opendev's mirror outside of ci.14:31
opendevreviewTim Beermann proposed zuul/zuul-jobs master: Add yamllint job.
opendevreviewTim Beermann proposed zuul/zuul-jobs master: Add yamllint job.
*** yadnesh is now known as yadnesh|away14:52
*** dviroel|rover is now known as dviroel|rover|lunch15:22
*** marios is now known as marios|out15:59
opendevreviewClark Boylan proposed opendev/system-config master: Use docker-compose exec -T with ansible tasks
clarkbautomated zuul updates broke on ^16:05
clarkbI figure we can land that and run it manually at some point16:05
clarkbalso jitsi meet just did a release we'll update to in about 12 hours16:05
fungiclarkb: all of the service-* mailing lists have pending moderator requests if you wanted to try out the interface. they're all spam of course16:07
fungiclarkb: was it the ansible upgrade with the new bridge which caused the regression 867258 is fixing?16:08
fungii guess that's implied by the commit message16:08
fungimainly just wondering if it broke longer ago than last week16:09
fungiand how far behind we are with zuul upgrades16:09
fungi(if you happen to know)16:10
*** dviroel|rover|lunch is now known as dviroel|rover16:10
*** hjensas is now known as hjensas|out16:11
clarkbfungi: the -T thing was caught in CI for db backups iirc and yes due to the recent bridge ansible update16:11
clarkbwe didn'tcatch it for the reboot playbook because we don't run that in CI16:11
clarkbwe are 9 days behind so just one cycle16:12
fungiahh, cool16:12
clarkbI'm looking at service-discuss held messages now16:12
clarkbfungi: in this case I probably want "discard" and not "reject" to avoid sending an email back to them?16:13
clarkbheh they send them with dates in the future to get better sorting?16:15
clarkbmessages have been discarded for all three lists I currently have moderator rights on16:15
clarkbI've reminded the opesntack release team about the pending Gerrit upgrade16:16
fungithe thing to watch out for (somewhat prevalent on the openstack-discuss ml) is that they'll spoof a message from a list member address and sometimes even quote parts of subjects from message archives. sneaky, sneaky spammers16:16
fungibut after a while i've gotten to where i can spot them. those in particular tend to end up held because of large attachments16:17
clarkbspam is an efficient model trainer :{16:17
clarkber :P16:17
opendevreviewClark Boylan proposed openstack/project-config master: Scale iweb back to 25% of possible quota
opendevreviewClark Boylan proposed openstack/project-config master: Set iweb max-servers to 0
opendevreviewClark Boylan proposed openstack/project-config master: Set iweb to empty labels and diskimages
opendevreviewClark Boylan proposed openstack/project-config master: Remove iweb nodepool configs
clarkbthats a series of changes to spin down iweb. I wanted to get those up and reviewed if not necessarily landed immediately just so that we can land them as things are quieter later this month16:44
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup iweb/inap provider content
clarkband that is the system-config side of cleanup17:09
clarkbI'm realizing we can't easily test project-config with tox v4 because it doesn't do speculative testing?17:25
fungithough most of its tox testenvs should be pretty straightforward to exercise locally17:27
opendevreviewClark Boylan proposed openstack/project-config master: Update tox.ini for tox v4
*** NeilHanlon_ is now known as NeilHanlon17:28
opendevreviewClark Boylan proposed opendev/system-config master: Update tox.ini for tox v4
clarkbwe should do similar for bindep and git-review and so on but I expect project-config and system-config to be the most annoying if they break17:31
*** jpena is now known as jpena|off17:32
clarkbwow updating tox.ini or is it the project entry in zuul for system-config runs all the jobs. That might need pruning17:34
fungigood way to identify unstable jobs though ;)17:37
opendevreviewClark Boylan proposed opendev/system-config master: Update tox.ini for tox v4
clarkbreskimming the gerrit 3.6 release notes the project owner implies delete perm has been removed. I didn't notice that before but I'm happy with that change18:22
clarkbWe should be on the look out for any unexpected fallout from that18:22
clarkbshould be easy enough to modify acls if necessary18:22
fungii don't expect to need to, we use explicit delete perm where we expect it18:26
opendevreviewClark Boylan proposed openstack/project-config master: Update tox.ini for tox v4
clarkb3.7 on the other hand... preparing for that upgrade will be fun18:29
fungiwhat's going to impact us with 3.7?18:38
fungithat we already know about, i mean18:38
clarkbfungi: we have to convert all of our voting rules to submit requirements18:38
fungigood thing we've normalized our acls long ago18:39
clarkb3.6 adds support for submit requirements and claims to preserve the old behavior too so it will be the transitional release while we make those changes18:39
fungishould be a pretty straightforward transformation at least18:39
clarkbthe priority labels might be hte most annoying as they aren't super consistent. But ya the others shouldn't be too bad18:39
fungiwell, our enforced ordering for sections and options and lists within options should make the converted versions easy to compare18:44
opendevreviewClark Boylan proposed opendev/system-config master: Update Gerrit to version 3.6
clarkbThere was a TODO to write that change on the ehterpad so I went ahead and did it18:49
clarkb is the etherpad18:51
ianwthanks, yeah was about to send that one, forgot to do it yesterday18:52
ianwi had a look at the submit requirements, i think we can add +2 and a +2 verified, but really it seem informational in our situation with zuul submitting things?18:54
ianwit mostly seems to affect if the "submit" button will be not greyed out18:54
clarkbianw: in 3.7 they drop compatibility with our configs that define the voting labels18:54
clarkbianw: you have to redefine them using the new submit requirements. And ya in 3.6 it is supposed to support both so we should be fine for now18:55
clarkbupgrading to 3.6 so that we have compatibility with both is the first step in upgrading to 3.7. Then we can sort out configs under 3.6 to prep for 3.718:55
ianwi'm thinking we can mostly do in CI; we already update and push the config, but we don't set it up like we have in production18:56
ianwin terms of rules/groups etc.18:56
clarkbya and we should be able to just bulk update project-config for all the one off project specific things18:57
ianw#status notice Gerrit will be unavailable for a short time as it is upgraded to the 3.6 release18:58
clarkbianw: in an hour though right?18:58
clarkbI don't think the status bot is currently here18:58
ianwoh whoops!  i've got my tz out, haha18:58
ianwnothing particularly interesting in the statusbot logs19:04
ianwi've restarted it19:06
clarkband it has rejoined19:10
fungii'll be in a meeting but around in case things go entirely sideways19:37
clarkbthe system-config tox.ini update chagne actually passes after I fixed the allowlist19:47
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
opendevreviewIan Wienand proposed zuul/zuul-jobs master: use-buildset-registry: add URL to buildset_registry_namespaces
clarkbthat is a good indicator that we don't have too much broken for toxv419:48
opendevreviewMerged opendev/system-config master: Use docker-compose exec -T with ansible tasks
opendevreviewClark Boylan proposed openstack/project-config master: Update post-review zuul pipeline definiton
ianw#status notice Gerrit will be unavailable for a short time as it is upgraded to the 3.6 release19:57
opendevstatusianw: sending notice19:57
-opendevstatus- NOTICE: Gerrit will be unavailable for a short time as it is upgraded to the 3.6 release19:57
ianwmy clock is based on "when is the weekly meeting" which works fine generally, unless you think something is happening at that time and it isn't :)19:59
ianwit's in emergency, and a screen is up and logging19:59
fungiscreen on review?19:59
opendevstatusianw: finished sending notice20:00
fungithanks, attached20:00
ianwok, will shutdown and backup 20:00
clarkbI'm attached now as well20:00
clarkbdid that do help text instead of actually downing the container?20:01
fungias noted earlier, i'm only half watching so please ping me if i'm needed20:01
clarkbfungi: noted20:01
ianwyeah, added a note, i guess we need to down and then up just mariadb20:03
ianwbackup done, looks good20:03
clarkbwe should all get in the habit of only running docker-compose from the file dir since it ignores env files (I think onyl jitsi meet uses them currently?) when they are not present in he current dir20:06
fungithere is a command-line switch to point to the specific env files if you prefer to do it that way20:08
fungii've used after realizing they were needed20:08
clarkbianw: those two shas seem to match20:08
clarkbweb ui loads for me20:11
ianwok, i see us up @ 3.6.3-26-g80d6be44e2-dirty20:11
ianwi'm watching logs in separate window20:11
ianwwe have another tight loop ssh failure from someone, but not seeing anything else20:12
clarkbwe've got another one of those unhappy ssh clients but otherwise I haven't seen anything crazy go by (I didn't catch the startup process if there were any errors there, might be worth pulling those up explicitly if you haven't et ianw)20:12
ianwyeah i watch from start, nothing i could see20:13
noonedeadpunkmy eyes started hurting badly :D20:13
jm1gerrit looks so different but i like it XD20:13
ianwwe've already got a few changes coming through20:14
noonedeadpunkOut of issues I don't see custom labels20:14
noonedeadpunkWe had backport candidate in OSA at least - it's gone now20:14
clarkbnoonedeadpunk: I'm not sure I parse that. Can you be more specific20:14
ianw went in, and has started zuul jobs20:15
noonedeadpunkOr well, better L620:16
noonedeadpunkAnd there's no such label anymore20:16
clarkbnoonedeadpunk: do you see them when you click repoy?20:17
noonedeadpunkOh, well, I do....20:17
clarkbI see them for review priority on devstack. Let me do a test vote and see how the UI renders it20:17
noonedeadpunkAh. And it appears then20:17
clarkbIts possible that the online reindex is necessary to show these? or maybe they are not shown if not set20:17
clarkbya ok20:17
clarkbits probably the case that they need to be set to be visible and that the online reindex that should be running (takes about 20 minutes iirc) may help? 20:18
noonedeadpunkActually I was quite sure they were not shown in reply as I checked that before I wrote...20:18
noonedeadpunkSo yes, might be it's re-index that's happening now20:18
clarkbnoonedeadpunk: but they are there now? and if you set the value you see it on the top level change page?20:18
noonedeadpunkOn top level page don't see them yet20:19
clarkbwell it looks like you unset it20:19
noonedeadpunkOr well20:19
clarkbif it is set is it visible?20:19
noonedeadpunkI don't see them on
noonedeadpunk(so depending on what mean undr op level20:19
clarkbok you reapplied and if set then it is visible on the top level of the change20:19
clarkbI mean on the change page20:19
noonedeadpunk*under top level20:19
noonedeadpunkyep, on the change page it's visable20:20
clarkbI think the listing does rely on the index heavily so would't be surprised if that depends on the reindex20:20
clarkbIts also possible that Gerrit is using some criteria to determine which should be shown on the listing page by default20:20
clarkbAnyway this doesn't sound fatal. As we understand it better we can provide feedback upstream to make things less weird too20:21
ianw1445 things left in the queue ... so yeah, let's let that drain ...20:21
clarkbI was just looking up how to examine that20:22
clarkb`gerrit show-queue`20:22
noonedeadpunkAlso question - is there any way to display who's voted like in old good gerrit?20:22
noonedeadpunkAsking just in case it can be configured....20:23
clarkbyes hover on the vote20:23
clarkbI don't know if it is configurable20:23
noonedeadpunkWell, hovering over vote for each change you open is quite.... time consuming?20:23
noonedeadpunkI see workaround20:23
noonedeadpunkit's shown for the list of reviewers20:24
clarkbI'm not trying to judge the method just explaining how it can be done. After the gerrit 3.2 upgrade we were very explicit that the way the team is currently staffed we cannot maintain a Gerrit fork any longer and we need our users to engage upstream (through us is fine too) for problems20:24
ianwok, from the upgrade POV, verified everything but replication.  am just waiting for indexing to finish now before moving on20:25
noonedeadpunknah, that's totally fair20:25
clarkbThe best thing is if people can file issues upstream. I'm also happy to file those too if sufficient info is shared to me. But part of why we got stuck on ancient gerrit for so long is we ended up with all these little gadgets and widgets and customizations and we realized doing all that just wasn't going to scale if we wanted to be up to date.20:25
clarkbThe good news is we've managed to engage upstream as a result and have a number of bug fixes pushed upstream through this process. The downside is we often live with issues for longer rather than holding up the entire upgrade process20:26
clarkbianw: the very end of the replication log looks clean20:27
noonedeadpunkIt's just small things that likely can be adjusted with config or smth, and I jsut lack knowledge a bit and asking as you're way more experienced and might already know how to fix that20:27
clarkbnoonedeadpunk: there is very little about the UI that is configurable unfortunately20:27
fungiwe also have a ci system which can deploy proposed config changes and obtain screenshots (and we can hold nodes from it for more interactive testing too)20:27
clarkbyou can choose a theme and diff width and how diffs are rendered, But the actual change screen and serch screens are largely unconfigurable20:27
fungiexcept through plugins20:28
noonedeadpunkI wonder why gerrit now decided to flip +W and +V places... Because I thought it's alphabetical order...20:28
clarkbright if you write java or js and install that into gerrit you can make big changes20:28
fungiwhich can add things but not really remove/replace significant parts of the us20:28
ianw(until the migration to the next javascript framework, anyway ;)20:29
clarkbianw: I didn't see the config diffing was that clean (it should be due to our testing right?)20:29
ianwyep, that was empty20:29
ianwagree replication log looks clean, it has done some replication20:30
clarkbI think the new behavior on the change screen is to show you all labels that are required to be set to merge. For us that is Code-Review, Verified, Workflow. Additional lagels are only shown if they are set to a non 0 value20:30
clarkbIt wouldn't surprise me if on the change listing screens they are also only showing you labels that must be set to merge. In order to simplify those listings20:31
clarkb heres a change that had review priority set prior to the upgrade20:31
noonedeadpunkat least these labels still work on custom dashboards, so I guess that's fine20:32
clarkbthe value shows up on the chagne screen properly but not in the listing page. I suspect we won't get these out of the listing pages20:32
clarkbnoonedeadpunk: can you link to one of those just for me own collection of examples? It helps if we ever have to talk upstream about this20:32
clarkbthose == custom dashboard20:32
ianwok, indexing should be done20:34
clarkbI think CI results for were not reported because gerrit was down. I've rechecked it and it has reenqueued20:35
clarkb(3.7 also broke/breaks rechecks and we need to followup oin the fix for that as part of the pre 3.7 upgrade prep)20:35
ianwi've approved and will take out of emergency as that merges20:36
ianwwe can fiddle the ui config to remove the "status" column, if we want.  20:37
clarkbI don't think is related to the upgrade because i Think that is checking our projects.yaml? But calling it out incase that is cheking against gerrit20:38
clarkbfungi: ^ fyi20:38
clarkbianw: I like it for the merge conflict info20:38
ianwyeah, i think we stick with it; just i did test turning it off on the held server :)20:39
ianwi'm going to afk for a brief toast break20:40
clarkbI shared that link with the openstack release team just in case20:40
ianwcurrently just waiting on gerrit config change to move through20:40
clarkbon further inspection of that openstack releases validation failure it seems to inspect project specific yaml and not the gerrit api. I think we're ok20:43
fungisounds good. thanks for the upgrade!20:44
fungii'll be around for a while still this evening in case there are any problems we haven't spotted yet20:44
clarkb is a quick approval if we want to land that20:45
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
ianwin our case, with the submit requirements, it seems more like we're "has a +2 and a +1 verified".  that would seem to more correctly show the status column as the requirements being met.  the +w is obviously required, but that's more an indication to zuul than a submit requirement, if that makes sense21:05
opendevreviewMerged openstack/project-config master: Update tox.ini for tox v4
ianwi'm going to take review out of emergency as the next thing to merge will be the config updates21:23
clarkbwe don't auto restart anyway so thats really safe21:23
noonedeadpunkclarkb: we use that But under "works" I meant  - get filtered by that.21:25
clarkbnoonedeadpunk: got it, the summary view still doesn't show the values but you can have queries that list them explicitly21:25
noonedeadpunkBut labels still are not shown even though they've got filterred by it:
clarkbya that might be a good piece of feedback. Basically if we are filtering by it then maybe show it?21:26
noonedeadpunkFor me that does make sense. But in terms of implementing that I see how it might be a bit tricky21:27
*** dviroel|rover is now known as dviroel|rover|out21:28
noonedeadpunkBut yeah, if query would be any value of label:Backport-Candidate (not only positive like we have), that it's really bad to understand what the result you get21:28
noonedeadpunkwithout renderring label values in a separate field21:29
JayFthere is an [S] in emails from gerrit since the upgrade21:31
JayFe.g. Subject: [S] Change in .....21:31
Clark[m]There are templates that control that and they are configurable but I'm pretty sure we don't override them. That must be a new piece of info they are trying to share via the template21:33
*** dasm is now known as dasm|off21:34
JayFack; as long as it was expected21:35
Clark[m] seems to indicate the change size21:38
opendevreviewMerged opendev/system-config master: Update Gerrit to version 3.6
Clark[m] is what we were running so ya that's new21:41
fungi"s" for "small" i guess21:41
clarkbya I expect those align with the size values you see in web ui listings21:46
clarkbianw: I don't see any deployed jobs queued/running at this point implying that the jobs for 867277 ran (but I haven't double checked taht). The docker compose file on review02 still seems to show 3.6 so I think that applied as expected if the jobs ran22:01
clarkbI need to pop out for a school run but will be back after22:01
ianwyep both review update and manage projects ran 22:10
ianwupdated links in etherpad22:10
ianwthat's it for the checklist22:11
fungiawesome, thanks again for getting it out of the way!22:19
ianweverything is much less scary thanks to ci!22:28
opendevreviewClark Boylan proposed openstack/project-config master: Flip jeepyb over to building Gerrit 3.6
clarkbthats a bit that I remembered22:58
clarkbinfra-root on now that it seems towork with tox v4 should I drop the ensure_tox_version arg from the change and let the zuul-jobs update switch us later? I feel like that might be best to avoid carrying this config around?23:00
clarkbI've made some edits to the meeting agenda. I'll leave that open for another half hour or so before sending it out23:10
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support
opendevreviewIan Wienand proposed zuul/zuul-jobs master: [wip] zuul-jobs-test-registry-buildset-registry-k8s-microk8s
ianwclarkb: i think probably yes take the pin out and just use the default from zuul-jobs is best23:58

Generated by 2.17.3 by Marius Gedminas - find it at!