Wednesday, 2022-01-19

opendevreviewMerged openstack/diskimage-builder master: Remove centos 9 and rhel 8 block in grub2 pkg-map
fungioh, thanks! sorry i stepped away for a bit00:43
clarkbno worries, it all went well. Yay for our testing00:44
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Don't install python3-virtualenv on redhat family
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Never call virtualenv directly, use DIB_PYTHON_VIRTUALENV
ianwfungi/clarkb: downgrade testing complete; notes @ and is on hold to play with.  if no concerns, i think update is about ready03:22
*** ysandeep|out is now known as ysandeep04:33
*** ysandeep is now known as ysandeep|brb05:57
*** ysandeep|brb is now known as ysandeep06:09
*** ysandeep is now known as ysandeep|lunch08:13
*** jpena|off is now known as jpena08:38
*** ysandeep|lunch is now known as ysandeep09:16
*** lbragstad7 is now known as lbragstad09:19
opendevreviewchandan kumar proposed zuul/zuul-jobs master: Introduce iptables_package var
*** dviroel|out is now known as dviroel10:58
*** ysandeep is now known as ysandeep|brb11:01
*** ysandeep|brb is now known as ysandeep11:15
*** rlandy|out is now known as rlandy|ruck11:15
fungiianw: thanks! and i guess holding with the fully-qualified project name worked?13:53
*** ysandeep is now known as ysandeep|away14:04
*** sboyron_ is now known as sboyron14:46
*** dviroel is now known as dviroel|lunch14:57
opendevreviewAnanya proposed opendev/elastic-recheck rdo: [DNM] Erbot with opensearch
opendevreviewMerged zuul/zuul-jobs master: Add support for RPM packages
opendevreviewJeremy Stanley proposed opendev/system-config master: Drop gitweb dependencies
opendevreviewJeremy Stanley proposed opendev/system-config master: Fix mixed spaces and hard tabs in Gerrit config
opendevreviewJeremy Stanley proposed opendev/system-config master: Use Gitea for Gerrit's code browser URLs
opendevreviewMerged zuul/zuul-jobs master: Introduce iptables_package var
*** dviroel|lunch is now known as dviroel16:15
clarkbianw: fungi: left a note about the reindex, but I think all of that is fine16:23
fungiclarkb: thanks, i expect to make a dnm change on top of those and set an autohold to poke at it some16:24
fungijust want to see if they even pass tests first16:24
clarkboh I was referring to the downgrade stuff. But the gitweb stack is also exciting :)16:25
clarkbI just haven't gotten that far this morning yet16:25
fungioh, the gerrit downgrade experiment. yep, hoping to look at that as soon as i'm done checking moderation queues16:26
clarkbI am going to hold one of the zuul streaming function test jobs to try and debug why focal python doesn't work and bionic's does16:26
opendevreviewEduardo Santos proposed openstack/diskimage-builder master: Fix openSUSE images
clarkbfungi: should be an easy one based on what I discoverd about meetpad yesterday and its mute messaging16:47
clarkbfungi: on the last chaneg in the gitweb stack it seems to be missing a root url?16:50
clarkbalso I'm not quite sure how we want to test that since we don't do jobs with gitea and gerrit in the same job. Maybe good enough to validate the urls are correct16:51
fungiclarkb: oh, yep thanks, i need to figure out where to set that, thanks16:51
fungiclarkb: on 825179, i don't suppose there's a way to fix the notice text instead? i'm not sure switchign back and forth between those behaviors is a great user experience, and presumably once the message gets corrected we'd undo that setting?16:51
clarkbI personally would prefer that I'm always muted on startup16:52
clarkbalso we've had people join before that didn't realize they were unmuted with loud background noise interrupting things. I think generally forcing people to send their audio through explicitly is nicer for calls16:53
fungiokay, i can buy that argument. if we plan to keep it this way then i'm not opposed16:54
corvusi'd like to do a rolling restart of zuul16:54
fungian announcement might be in order, but i guess the software announces it on connect anyway so maybe no need16:54
fungicorvus: sounds great! i'm happy to assist16:54
corvusi've run zuul_pull on bridge, so all containers are up to date16:55
clarkbI'm around. Looking at zuul ansible python things for the focal test node change. Ping me if I can help16:55
corvusthanks; hopefully this will just be slow and boring :)16:56
corvusi'll start by restarting the scheduler on zuul0116:57
fungiclarkb: out of curiosity, do you know why it's START_VIDEO_MUTED=0 and START_AUDIO_MUTED=0 to enable? other toggles in those configs seem to enable with =116:57
clarkbfungi: the value is the first N connections to start unmuted16:57
clarkbif we set it to 1 then the first person to connect is not muted16:57
clarkband so on16:57
fungioh, yep the comment even says so. i should try harder at reading comprehension16:58
clarkbIn the past they had a binary muted or not value. But then they changed it to this which is weird but seems to work16:58
fungiright, so =1 would leave the room leader unmuted but mute everyone else who joins16:58
corvuswhile it's coming back up, i'll go ahead and restart all the mergers16:59
corvusmergers are restarted; zuul01 is still initializing.  next i'm going to gracefully stop half the executors.17:02
corvus(i think our load is sufficiently low to permit that)17:02
fungivery cool17:05
corvusi've run into a potential issue, so i've stopped the process... (i had not yet graceful'd the executors)17:06
corvuslooking at logs now17:06
corvusi think the issue may only affect zuul01, so i don't think it's user-visible as long as zuul02 is still running17:07
fungifollowing along in the zuul matrix channel17:10
fungiclarkb: for the failure on does that mean gerrit's bazel config hard-codes an expectation the gitiles plugin will be present anyway? i guess we need to build but then disable it if we don't want to fork that?17:11
clarkbfungi: ya seems that plugin is marked core so expected to always be present17:12
fungithanks, i got quite confused reading the plugin docs, it seems there are built-in plugins in the gerrit codebase, but also apparently "core" plugins which are developed seperately17:17
opendevreviewMerged opendev/system-config master: Start meetpad meetings with muted audio
corvusokay, it looks like we're going to merge a quick fix to zuul; so i have paused the rollout.  current status is: mergers are upgraded (but are safe to continue to run at this version).  zuul01 is stopped.  zuul02 and the rest of the system are up and on the old version.17:24
corvusafter the fix merges, i will just restart zuul01 on the fixed version, restart the mergers again, and resume as before.17:25
opendevreviewEduardo Santos proposed openstack/diskimage-builder master: General improvements to the ubuntu-minimal docs
clarkbfungi: re gerrit plugins: yes, I think they made some plugins separate repos but they are still required for building gerrit as they are part of the core. Then there are additional plugins which are optional17:41
*** jpena is now known as jpena|off17:43
opendevreviewEduardo Santos proposed openstack/diskimage-builder master: Fix openSUSE images and bump them to 15.3
clarkbcorvus: any reason for me to avoid deleting my autohold while zuul is in the halfway restarted state?17:51
corvusclarkb: nope go ahead17:54
fungiclarkb: i'm at a loss to work out how to disable the gitiles plugin via configuration. it looks like there are cli and rest api options to do it if remote admin is allowed in the config, but i don't see a way to block loading a plugin at start other than renaming or deleting it on the filesystem17:55
clarkbfungi: does gitiles take a config option to disable itself maybe?17:56
fungithe scant docs available for the gitiles plugin indicate it has a config option to tell it not to take over gerrit's weblinks, which i suppose is a viable fallback, but seems silly to run the plugin at all in that case17:56
clarkbor maybe we don't disable gitiles? I think our current thing only shows us gitweb links17:56
clarkbwell I don't know that it "runs" in that case. The code will be present without any path to execute it?17:56
fungiright now gerrit's webui has the code browsing links taken over by the gitiles plugin, which effectively shadows the built-in gitweb linking. the gitiles plugin can be told not to do that, but direct gitiles urls would still be available (we could block them in apache)17:58
clarkbfungi: right now I think gitiles is replaced by gitweb links17:59
clarkbBut I see what youa re saying there isn't a way to prevent gitiles from loading if you know the urls17:59
fungiif i look at a change in gerrit, the git sha linked next to the patch set takes me to the gitiles view of that commit18:00
fungirather than to gitweb18:00
clarkbah neat. All of the other links are gitweb18:01
clarkbI bet that is a bug18:01
clarkbbelow the votes for example is a links section with gitweb and if you click on a specific file you get gitweb links there too18:01
fungioh, got it. i wasn't aware it linked anywhere else18:01
clarkboh but there is also a "browse" link adjacent to "gitweb" when you click specific files and browse == gitiles18:01
fungianyway, the gitiles docs claim it takes over gerrit's web links unless you set noWebLinks=true18:02
clarkbya I'm not sure what the best thing is. Maybe file a bug against gerrit suggesting that removal of gitiels may be desireable if using another system18:02
fungithe docs for the `gerrit plugin remove` say that it renames the plugin jars themselves:
fungimaybe the expectation is that you do that yourself if you don't want them loaded at start18:05
clarkbya that might also work. Except gerrit may expect the jar to be functional since it is a core plugin18:06
clarkbI think the fact that this is a core plugin makes it special18:06
clarkbya note there "Mandatory plugins cannot be disabled"18:06
clarkbI suspect gitiles meets that criteria based on the bazel complaints18:06
fungithat seems to also be configurable?
clarkbah then maybe it doesn't care18:08
fungianyway, testing will catch at least that much for us, i'll try the simple approach first18:08
clarkbfwiw we do a bazel build release which outputs a single war iirc. Then when you run that it extracts all the things. Not sure what the best way to not extract the gitiles plugin might be18:10
clarkbin the error log on startup you'll often see it talk about cleaning up old plugins and that is when it replaces the existing content of the plugins dir with its new contents that it extracts aiui18:11
fungioh, so the plugin jars aren't separate from the war, they're packed inside it?18:11
clarkbyes, that is my understanding. Then when the war executes it has some self extracting early step18:12
clarkband that step is what will log about cleaning up plugins on gerrit startup. Since it deletes the old versions to replace them with its new versions18:12
opendevreviewElod Illes proposed openstack/project-config master: WIP: Remove ocata branch filters
clarkbit should be safe to land the limnoria update after 2200 UTC today as there is about a 4 hour window before the nextmeeting. However, tomorrow after 19:00 should be even better so I'll plan to land the limnoria update around 19:00 UTC tomorrow18:27
opendevreviewElod Illes proposed openstack/project-config master: WIP: Remove ocata related definitions
opendevreviewElod Illes proposed openstack/project-config master: WIP: Remove publish-install-guide ocata job
clarkbfungi re meetpad muting it seems we don't restart services when those values in the .env file update. Should I go ahead and restart those now?18:47
fungiclarkb: seems fine to me. i don't know how to tell if anyone's using it at the moment18:48
clarkbI think we can dig into logs somewhere. But ya I expect it will be quick and if someone is using it they should reconnect without trouble18:49
opendevreviewJeremy Stanley proposed opendev/system-config master: Use Gitea for Gerrit's code browser URLs
clarkbhrm I suspect that feature doesn't work as advertised :(18:52
clarkbI don't join automuted and have tried in firefox and chrome18:52
clarkbmaybe down then up -d wasn't enough to rewrite teh configs?18:54
fungiwe map the configs into the containers though, right? so if they're updated on disk then down/up should be all we need18:56
opendevreviewElod Illes proposed openstack/project-config master: Remove ocata related grafana definitions
clarkbno, we set env vars and then the jitsi meet templating writes out the configs for us18:56
opendevreviewElod Illes proposed openstack/project-config master: WIP: Remove ocata related definitions
clarkbconfig.startAudioMuted = 10; is still set in config.js so it didn't template out the file as expected. I might need to move that file aside and have it rewrite it?18:57
opendevreviewElod Illes proposed openstack/project-config master: WIP: Remove publish-install-guide ocata job
fungiworth a try. maybe we should always clear the configs on restart18:57
clarkbhrm except it did update the file according to the timestamp. So something is preventing the value from going through and I think I know what it is18:58
clarkbI'll get a change up shortly18:58
opendevreviewClark Boylan proposed opendev/system-config master: Fix meetpad audio mute setting
clarkbfungi: you have to explicitly set the value, then explicitly allow the value to be passed to the container19:01
clarkbI think ^ then a restart as I did previously should fix it19:01
fungioh wow. double-book envvar accounting19:01
corvusi just used the zuul admen web api to promote the zuul change which had failed flaky tests19:03
fungivia curl or zuul-client?19:03
corvusvia clicking gui buttons19:04
* corvus uploaded an image: (25KiB) < >19:04
fungithat's slightly amazing19:05
clarkboh nice we have paginated build search results now too19:07
fungicorvus: that reminds me, is in the process of being replaced by, i guess that's set locally on the keycloak poc, not in configuration management yet?19:09
corvusfungi: yep19:56
fungicool, i'll see about updating it as time permits19:56
fungiseems like 825381 should have been enqueued into the gate 15 minutes ago19:59
fungioh, we have per-pipeline events now, seems like there's a backup in evaluating results20:00
fungirelated to restarting?20:01
fungihigher-priority results are burning down first, so i guess check will start falling once the more important pipelines clear20:02
fungiand there it goes... "Starting gate jobs."20:03
fungisomeone's got a 27-change-long series for keystone they're testing, that could be related20:05
opendevreviewMerged openstack/project-config master: Remove ocata related grafana definitions
corvuswe're running one one scheduler and suddenly delays are more noticeable.  i take that as a good sign :)20:12
corvuszuul fix just merged20:13
corvuswaiting for promote jobs20:13
fungigreat point20:21
opendevreviewMerged opendev/system-config master: Fix meetpad audio mute setting
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail our Gerrit testing for an autohold
clarkbcorvus: looks like zuul promoted images successfully20:46
clarkbLooks like the udpate to the docker-compose.yaml file did cause the service to be restart on meetpad20:56
clarkbI'll test things as soon as zuul says it is done deploying20:56
clarkbI think there might be some issue on my local system having firefox and chrome share the mic at the same time. It seems the browser that opens the meeting most recently wins.21:00
clarkbAlso it seems that when you are the first person to join a meeting it doesn't mute you. But everyone else is muted. Thats probably good enough I guess21:01
clarkbconfig.startAudioMuted = 0; is confirmed to be in the config now too21:02
corvuspulling zuul images and starting zuul01 scheduler21:04
opendevreviewGage Hugo proposed openstack/project-config master: Retire security-specs repo
corvusrestarting mergers21:06
corvusonce the schedulers are done, i'd like to just do a hard-restart of the executors instead of a graceful restart.  all the jobs will abort and restart which will delay results a bit, but the zuul rolling restart will be faster.21:10
clarkbI had missed it but one of the two backup hosts needs pruning again. I did it last time to make sure I had some comfort with it. Does anyone elsewant to do it this time around?21:10
clarkbcorvus: I think that is fine as long as we don't have any release job type deals that might not like getting reenqueued. I can give the openstack reelase team a heads up21:11
clarkbrelease pipelines are also currently empty so should be fine21:11
fungithanks, and yeah executor restarts seem like a pragmatic option to speed this up21:11
fungii can try pruning the backups. we've got that process documented, right?21:12
clarkbfungi: yup should be documented and iirc the docs were quite good when I followed them last time21:12
fungiit's not urgent, we only just started to get the 90% warning21:12
clarkbya we probably have a week or so to get to that at least21:13
fungibut i'll give it a shot after dinner21:13
fungior tomorrow worst case21:13
clarkbcorvus: I've warned the openstack release team. Should be good to go21:14
corvuscool, thx.21:14
corvuslooks like scheduler 01 is up and no tracebacks21:14
corvusi'm going to restart scheduler02 and the web ui now.  since it's not load balanced, the web ui will be unavailable, but the scheduler will continue running.21:15
opendevreviewGage Hugo proposed openstack/project-config master: Retire security-specs repo
corvuszuul02 restarting now21:17
ianwclarkb: sorry, back to your comment from hours ago :)  yeah i did try it, thinking it would reindex online, but it fails to start with that error.  i don't think it can handle it when the index version is higher21:19
clarkbianw: gotcha21:19
ianwit looks like "delta-reindex" refers to ...
ianwi'm not sure how you would revert the change index ... perhaps he is suggesting replacing with an old copy of index files from a backup you have on disk, maybe?21:23
clarkbya then if you are HA doing a delta reindex off that backup?21:24
opendevreviewMerged openstack/diskimage-builder master: Fixes for centos-9-stream efi behaviour
ianwif that is the case, it's something we should probably add to our upgrade steps to copy the indexes temporarily21:27
clarkb++ though the rename might make things weird. I guess we can do a rename and startup and wait for all reindexing that th rename needs to complete (shouldn't take too long iirc). Then do another short downtime to upgrade21:28
clarkbthat process is probably safer anyway21:28
corvuszuul02 back up and looking good so far; i'm going to hard-restart the executors now21:30
clarkbianw: when you get a chance is the limnoria update change. It got a new ps to use a tag instead of master. Looks like after 19:00 UTC tomorrow is a good time to land that due to a lack of meetings.21:36
clarkbfor the rename and upgrade we should probably start a doc for that and look at rename changes to be sure we are ready. I can probably start on that tomorrow morning21:36
*** dviroel is now known as dviroel|out21:38
clarkbgitea 1.16 just got its first RC. Doesn't look like there will be many issues for us to upgrade after a quick glance at release notes21:38
clarkbNow that I'ev said that testing will show a bunch of things :)21:39
fungiianw: clarkb: or just delete the indices and let it create new ones?21:43
clarkbfungi: that is slow (about 35 minutes on our current install iirc)21:44
clarkbbut ya that is what ianw's document says to do essentially. Just restart21:44
clarkbs/restart/start over/21:44
fungithat seems fine to me, it's only if we have to roll back anyway21:44
corvusthings look good in zuul-land21:47
corvus#status log performed (mostly) rolling restart of zuul onto commit d304f4134f05fa08aab70e9add6ec490370dc6e221:48
opendevstatuscorvus: finished logging21:48
clarkbnext stop zuul 4.12.021:48
ianwfungi: yeah, just trying to decipher :)21:48
opendevreviewGage Hugo proposed openstack/project-config master: Retire security-specs repo
clarkbI'm trying to ebtter understand the grafana deployment. It looks liek we use a simple sqlite db located in /var/lib/grafana on the container? This db also doesn't seem to be hosted from the system which implies that it is ephemeral and rebuilt from grafyaml anytime we update the image?22:00
clarkbdoes anyone know if that is the case?22:00
ianwthis seems likely22:08
ianwnot sure if it's ephemeral though22:08
ianwwell, i guess it's not.  it's ... overwrite only?  basically we run grafyaml and that adds or overwrites any graph definitions, but will not remove ones that have gone away22:09
clarkbya, and if we restart the grafana container often enough it will prune the old ones that go away on its own thatway22:09
ianwthat may be a happy accident :)  i don't think i intended that to happen but it sounds useful22:10
clarkbI'm taking notes on what is roughly required to run services with dedicated users22:10
clarkb over here as step 0 in prioritizing and planning the next steps there22:11
ianwok, cool.  corvus had something out there at one point to switch this to the raw upstream images; we're rebuilding currently to put grafyaml in the container22:12
ianwi'm not sure where we got to with that22:12
corvus is relevant22:18
clarkbthanks I'll have to process that. I wonder if that would make it hard to run grafana as a dedicated user (thinking of matrix-gerritbot). Chances are good it will be fine though22:20
opendevreviewIan Wienand proposed opendev/system-config master: Use grafyaml container image
ianwdoes service-lists actually require grafana, or is that just a typo i've managed to introduce @ ?22:27
clarkbianw: I cannot think of a relationship between grafana and lists22:42
clarkbcentos 8 stream systemd package updated in git about an hour ago to the fixed version22:49
clarkbthe packages aren't in the repos from what I can see though. Also I note that the previous update to systemd in for centos 8 stream isn't in our mirrors either. Hopefully 239-55 ends up in there though22:51
fungiclarkb: is a held node with the gitea change applied. the gitea link for the commit looks right (revision) but the link in the diff view (file) seems to put the refname in for ${commit} rather than the actual commit id22:51
fungirevision = "${project}/commit/${commit}"22:53
clarkbfungi: do they have a separate ${ref} substitution?22:53
fungifile = "${project}/src/commit/${commit}/${file}"22:53
clarkbif so then that seems like a bug22:53
fungiin the first one ${commit} is the git id and in the second one ${commit} is the refname22:53
clarkbbut maybe we can work with it anyway? the refs are pushed to gitea22:53
fungiyep, i'll have to see if i can work out the correct syntax for putting the refname in a gitea url22:53
clarkbhrm I wonder if gitea supports viewing those refs22:54
fungii think i had previously concluded gitea didn't support doing that, but it's worth revisiting22:54
clarkbya I think you may be right. The chagne that was made upstream to gitea allowed us to push them and fetch them with git but the web ui is still clueless to them iirc22:55
clarkbfungi: the gerrit docs for gitweb.file say ${commit} should be the sha-1 I think this is a bug in gerrit22:57
clarkbI wonder if gitiles wants the refname instead so internally that got changed22:57
fungithe gitea webui does *show* the refnames at least: see where it says changes/26/825326/122:58
fungii wish it included them in the commit log view too like git and cgit/gitweb do22:58
clarkbya I'm not having any luck trying to construct a url based on what I know of git and gitea. I strongly suspect they just don't have a way to expose those via the web ui23:00
clarkbbut as you point out it would work if we could get a url with the sha123:00
fungidocs say that's what it's supposed to be: "Valid replacements are ${project} for the project name in Gerrit, ${file} for the file name and ${commit} for the SHA1 hash for the commit."23:03
clarkbyup hence my expectation that this is actually a bug.23:05
clarkbThey should rename whatever they currently fill to ${ref} and have ${commit} point to the sha23:05
fungiof course, there's every chance that's a breaking change for a bunch of users who rely on the currently incorrect behavior, so they may instead leave ${commit} broken there and add a new ${no_really_commit} instead23:07
clarkbI've found the code locally let me get you a link23:07
clarkbthis might take a minute due to java's deep tree structure23:07
fungioh, thanks. java usually leaves me cross-eyed23:08
clarkb that is the utility method that does the interpolation.23:12
clarkbCall point momentarily23:12
clarkb and appear to be how we construct that value23:13
clarkbthe java lambdas still confuse me. The language has come a long way since I had to use it regularly.23:14
*** rlandy|ruck is now known as rlandy|out23:15
clarkbfungi: reading that it seems to be side A is the old side and if there is a patchset for that then get that refname otherwise the commit. Then for the new side always use the refname since it is always going to be a patchset?23:16
clarkbso ya I think it should be possible to alwys get the commit id for both sides and store them as different values then interpolate that as you suggest as a No really the commit please var23:17
clarkbmaybe we should start with an upstream bug? I think coding this shouldn't be too difficult if we want to take a stab at it. Then we can use a patch file in our CI to ensure it works too.23:17
clarkbfungi: something like private final String commitA; this.commitA = sideA.fileInfo().commitId(); then later plumb through commit as a separate var alongside revision. And then interpolate that with ${commit_sha}23:19
fungiany indication if it's still the same in master?23:22
fungibut yeah, i can probably get a bug report filed tomorrow. eod has caught up to me23:22
clarkbya it appeas to still be in master but with slightly different code23:23
clarkbI'm hacking up a thing because I already debugged most of what was necessary to write some code23:24
fungiaccording to the docs, the roottree option also takes a ${commit} so i wonder if it's correct or similarly broken... i can't find where that actually gets used in the ui23:24
clarkbthe hardest part of this is always figuring out the right git diff syntax to get a patch that patch can apply23:32
clarkbok first real issue is going to be that gitiles implements this interface23:35
clarkbfiguring out compat across the two codebases will be fun23:35
clarkbfwiw I strongly suspect that the reason it is this way is for gitiles since gitiles depends on this interface directly23:40
fungiwhich is not so great if you want to avoid serving up a code browser view in gerrit separate from your primary code browser23:42
clarkbanyway my code compiles I just have to figure out git diff really quickly then can push up a patch that builds this updated gerrit23:43
opendevreviewClark Boylan proposed opendev/system-config master: DNM this patches gerrit to do gitea links for files
clarkbfungi: ^ I think you can rebase on that and try ${hash}23:49
clarkbone thign I didn't check is that commitId is the sha1 but it really should be23:53

Generated by 2.17.3 by Marius Gedminas - find it at!