Tuesday, 2022-06-21

ianwi've closed out https://pagure.io/centos-infra/issue/814 as it seems mirrors have updated01:24
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add support for Python 3.10  https://review.opendev.org/c/openstack/diskimage-builder/+/84461301:37
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add Fedora 36 support  https://review.opendev.org/c/openstack/diskimage-builder/+/84491202:04
opendevreviewIan Wienand proposed openstack/diskimage-builder master: containerfile: warn if we don't have a Dockerfile  https://review.opendev.org/c/openstack/diskimage-builder/+/84683702:04
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Fix BLS entries for /boot partitions  https://review.opendev.org/c/openstack/diskimage-builder/+/84683802:06
opendevreviewMerged opendev/system-config master: gerrit: trigger rebuild for anonymous browsing fix  https://review.opendev.org/c/opendev/system-config/+/84680902:10
opendevreviewMerged openstack/diskimage-builder master: Revert "CentOS 9-stream : work around selinux permissions issue"  https://review.opendev.org/c/openstack/diskimage-builder/+/84607802:35
opendevreviewMerged openstack/diskimage-builder master: Stop using openstack upper constraints  https://review.opendev.org/c/openstack/diskimage-builder/+/84521802:50
opendevreviewMerged openstack/diskimage-builder master: Use plain tox jobs instead of openstack ones  https://review.opendev.org/c/openstack/diskimage-builder/+/84534902:50
opendevreviewMerged openstack/diskimage-builder master: Add a warning in satellite configuration  https://review.opendev.org/c/openstack/diskimage-builder/+/84565003:25
opendevreviewMerged openstack/diskimage-builder master: Drop tumbleweed job  https://review.opendev.org/c/openstack/diskimage-builder/+/84509703:26
opendevreviewSamuel Walladge proposed openstack/project-config master: Add Cinder Dell EMC PowerStore charm  https://review.opendev.org/c/openstack/project-config/+/84688804:18
opendevreviewMerged openstack/diskimage-builder master: Add support for Python 3.10  https://review.opendev.org/c/openstack/diskimage-builder/+/84461304:18
opendevreviewSamuel Walladge proposed openstack/project-config master: Add Cinder Dell EMC PowerStore charm  https://review.opendev.org/c/openstack/project-config/+/84688804:18
*** ysandeep|out is now known as ysandeep04:34
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Fix BLS entries for /boot partitions  https://review.opendev.org/c/openstack/diskimage-builder/+/84683804:34
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Parse block device lvm lvs size attributes  https://review.opendev.org/c/openstack/diskimage-builder/+/83982904:34
opendevreviewSteve Baker proposed openstack/diskimage-builder master: WIP Support LVM thin provisioning  https://review.opendev.org/c/openstack/diskimage-builder/+/84014404:34
opendevreviewSamuel Walladge proposed openstack/project-config master: Add cinder ibm storwize charm  https://review.opendev.org/c/openstack/project-config/+/84689904:44
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Add Fedora 36 support  https://review.opendev.org/c/openstack/diskimage-builder/+/84491204:47
opendevreviewSamuel Walladge proposed openstack/project-config master: Add Cinder IBM Storwize charm  https://review.opendev.org/c/openstack/project-config/+/84689905:04
opendevreviewSamuel Walladge proposed openstack/project-config master: Add Cinder Dell EMC PowerStore charm  https://review.opendev.org/c/openstack/project-config/+/84688805:08
opendevreviewSamuel Walladge proposed openstack/project-config master: Add Cinder IBM Storwize charm  https://review.opendev.org/c/openstack/project-config/+/84689905:08
*** pojadhav is now known as pojadhav|ruck05:11
* pojadhav|ruck brb05:29
opendevreviewDavid Ostrovsky proposed zuul/zuul-jobs master: ensure-java role: Bump default java_version to 11  https://review.opendev.org/c/zuul/zuul-jobs/+/84690906:34
*** ysandeep is now known as ysandeep|afk06:42
ianwi was going to quickly restart gerrit for 846809 but 06:51
ianw"ERROR: for gerrit  error pulling image configuration: download failed after attempts=6: received unexpected HTTP status: 500 Internal Server Error"06:51
ianwhttps://hub.docker.com/layers/gerrit/opendevorg/gerrit/3.5/images/sha256-95c8a6d4d9d3716d7ee3616d16b3699a561f26b5f8a3b98cb378a156c04815e0?context=explore looks ok06:52
*** marios_ is now known as marios06:53
ianwok, it seems to have gone away06:57
ianw# docker inspect f2711d8fdd61 | grep 95c8a606:57
ianw            "opendevorg/gerrit@sha256:95c8a6d4d9d3716d7ee3616d16b3699a561f26b5f8a3b98cb378a156c04815e0"06:57
ianwok, restarted, i no longer see the popup06:59
ianw#status log restarted gerrit @ https://review.opendev.org/c/opendev/system-config/+/84680906:59
opendevstatusianw: finished logging06:59
opendevreviewMerged opendev/statusbot master: twitter: Fix typo on OK tick  https://review.opendev.org/c/opendev/statusbot/+/84653307:13
*** jpena|off is now known as jpena07:37
*** pojadhav|ruck is now known as pojadhav|lunch07:44
fricklerinteresting long read https://anarc.at/blog/2022-06-17-matrix-notes/07:53
fricklerinfra-root: here's a case where zuul seems to have continued to run on PS1 and ignored that PS2 was submitted shortly after PS1 https://review.opendev.org/c/openstack/kolla/+/84655408:11
fricklernot sure if related to gerrit update, will try to find zuul logs08:12
*** pojadhav|lunch is now known as pojadhav|ruck08:38
fricklerzuul logs some error from gerrit right after PS2 was received, but iiuc related to a different change. nothing relevant to be seen on the gerrit side. guess we need to wait whether this happens more often08:39
noonedeadpunkfwiw we again catching post_failures due to swift upload timeouts08:42
*** ysandeep|afk is now known as ysandeep09:10
*** ykarel_ is now known as ykarel|lunch09:19
*** rlandy|out is now known as rlandy10:25
*** ykarel|lunch is now known as ykarel_11:14
*** dviroel|out is now known as dviroel11:30
bjonglez_is https://docs.openstack.org/contributors/code-and-documentation/elastic-recheck.html up-to-date? most of the links are dead12:49
bjonglez_this page is linked from https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing12:49
bjonglez_(which in turn is linked from failing Zuul tests in Gerrit)12:50
fricklerbjonglez_: it is not up-to-date, we should clean that up12:54
fungibjonglez_: thanks for pointing that out, yes a lot of that is dead wood we should be trimming. that suite of services was turned off a couple of months ago, and a separate effort is available here: https://governance.openstack.org/sigs/tact-sig.html#opensearch12:55
fungino longer being run by the opendev sysadmins, it's an effort within the openstack tact sig12:56
bjonglez_ok, thanks!12:58
*** ralonsoh_ is now known as ralonsoh13:00
fungibjonglez_: further discussion of the new opensearch.logs.openstack.org or other openstack tact sig efforts would be more on-topic in #openstack-infra13:08
fungii'll start looking into the indicated documentation updates13:08
bjonglez_fungi: right, thanks for mentioning it. since part of the doc was opendev, I thought I would raise it here13:09
fungimakes sense!13:09
*** dasm|off is now known as dasm13:33
fungii've pushed up some changes for the openstack community documentation to address those points, but as for what to do with https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing i'm open to suggestions13:48
fungii feel like we're long overdue for moving the openstackisms out of the entire document, not just that section, maybe keeping only a vestigial link to the openstack contributor guide13:49
opendevreviewSylvain Bauza proposed openstack/project-config master: update Review-Priority label for nova related projects  https://review.opendev.org/c/openstack/project-config/+/83759514:14
*** ysandeep is now known as ysandeep|afk14:19
clarkbfungi: most of that content looks fine? Its just the e-r mention and maybe the idea that a QA team exist that is the problem?14:27
clarkbfrickler: re ps1 vs ps2. I wonder if zuul did trigger the jobs but then when zuul tried to report back to gerrit it found gerrit was offline due to the restart ianw did?14:32
clarkbfrickler: thats my best guess14:33
clarkboh wait ps2 was pushed june 214:34
clarkb*june 2014:34
fungiclarkb: yeah, i mean i can do a surgical update to address the churn around elastic-recheck bit, but i mean more generally the entire developer.rst has lots of "if you're contributing to an openstack project..." all over the place14:39
fungiinformation about contributing to openstack projects should probably be in openstack's contributor guide rather than opendev's more generic manual14:39
clarkbI think that was added to help transition content out of the doc. It has probably been long enough at this point we don't need to do that anymore14:42
fungia naive grep counts 27 case-insensitive mentions of "openstack" in that file (some are things like links to wiki.o.o though so not all are necessarily candidates for removal)14:42
fungibut, e.g., sections talking about launchpad blueprints and specs repos...14:43
fungiand openstack-specific bug reporting workflows14:44
fungialso some of the stuff we link out to on the wiki should probably be moved into infra-manual14:44
fungie.g. information on formatting git commit messages and existing integration we have in gerrit/zuul for them14:45
fricklerclarkb: yes, that all happened well after the gerrit update14:52
opendevreviewClark Boylan proposed opendev/system-config master: Fix gerrit deployment dependencies  https://review.opendev.org/c/opendev/system-config/+/84703414:52
opendevreviewClark Boylan proposed opendev/system-config master: Remove Gerrit 3.4 jobs  https://review.opendev.org/c/opendev/system-config/+/84703514:52
clarkb847034 is a simple change that is safe to land now. 847035 should only be landed once we're happy with 3.5 and don't expect ot revert (though we can revert to our already built images if we really have to later)14:53
fricklerand it may as well be unrelated, just the timing makes it suspicous14:53
clarkbfrickler: ya my next best guess is that Zuul didn't see an event for that ps or that it decided no jobs should be run for some reason14:53
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.6 jobs  https://review.opendev.org/c/opendev/system-config/+/84704015:04
*** ysandeep|afk is now known as ysandeep15:12
*** dviroel is now known as dviroel|lunch15:52
opendevreviewClark Boylan proposed openstack/project-config master: Bump jeepyb gerrit image build job to 3.5  https://review.opendev.org/c/openstack/project-config/+/84705715:56
clarkb847057 joins 847034 in the list of safe to merge changes15:57
opendevreviewClark Boylan proposed opendev/system-config master: Remove Gerrit 3.4 jobs  https://review.opendev.org/c/opendev/system-config/+/84703515:59
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.6 jobs  https://review.opendev.org/c/opendev/system-config/+/84704015:59
*** marios is now known as marios|out16:11
*** pojadhav|ruck is now known as pojadhav|out16:11
*** jpena is now known as jpena|off16:24
*** dviroel|lunch is now known as dviroel16:38
*** ysandeep is now known as ysandeep|out17:37
fricklerianw: how long do you think it will take until we can do a dib release? wondering whether it still makes sense to have a workaround for wheel builds20:02
fungito clarify from the end of the meeting, shortly before the summit i was digging into more reports of audio problems for meetpad from openstack cinder folks, and discovered that one of the most effective workarounds for similar reports had actually become the default in later versions of the service config than the files we forked20:03
ianwfrickler: yeah, i merged some stuff yesterday.  but in terms of centos 9-stream, it should be working without release20:03
fungiwhich got me thinking that we should probably try to update all our meetpad configs as there are also other things we've been setting which are now set by default20:03
fricklerianw: oh, o.k., I can unpause builds and see what happens20:03
fungibut diffing them for a year+ of changes is nontrivial20:03
fungiso 1. i'd like to avoid making too much work for myself, but also 2. it would be nice to find a way to make it be less work next time we want to update configs as well20:04
ianwfrickler: yeah, so we worked through upstream fixing it, the mirrors updated, then the mirrors were corrupt for a while, but that seems cleared up now to20:04
clarkbya I don't think you need to diff them as much as just ensure the two or three setting we want are set20:04
clarkbfungi: ^20:04
ianwfrickler: yeah i was just asking about the pause status in matrix :)  but yeah, my plan today was to monitor it :)20:05
*** dviroel is now known as dviroel|afk20:06
fungiclarkb: agreed, though just noting that we've taken multiple approaches such as noting the commit from the point where we forked the files in some of them (but not all), grouping our edits together and optionally commenting the same settings out elsewhere which makes it harder to spot when the defaults change, et cetera20:06
clarkbfungi: I think that is because some things were already supported via flags and others werent so had to be added directly to configs20:07
clarkbbut ya it definitely makes it confusing20:07
fungialso some of the option names changed20:07
fungior moved to other files20:07
fricklerianw: you can see that "paused" status on http://nl01.opendev.org/dib-image-list for gentoo, I just unpaused centos-9-stream20:08
ianwfrickler: thanks!  i'll watch it through today20:09
ianwto the dib release, i was thinking of putting in f36 support which was really nothing this time with https://review.opendev.org/c/openstack/diskimage-builder/+/844912 and doing a release20:10
fricklerRamereth: quite a lot of images seem to be stuck in deleting, you can see the image ids on http://nl01.opendev.org/image-list , can you check this?20:10
ianwwe might need to manually clean them up20:11
fricklerianw: iirc the delete attemps show API failures in the log20:12
fricklerdon't think that we be different with a manual attempt on our side20:12
fricklerianw: regarding dib, my main concern were the wheel jobs, if they now work with fresh c9s, that's ok for me20:13
ianwthey *should* when the images get out there, will monitor20:13
fricklerwill check results tomorrow, cu everyone20:13
ianwi'm noticing that nb03 is not responding20:14
ianwi've done a hard reboot.  it seems like the server is up, but possibly has no network?20:17
fungithat's the one in linaro?20:17
fungiianw: check your inbox20:17
fungireply from equinix on the thread from january20:18
ianwdidn't that say things would be going away in a month or so?20:19
fungioh, end of july, yeah. so maybe not just yet20:19
ianwbut yeah, we should have that conversation because if nb03 is going away, we should move the builder to osuosl i guess20:21
ianwit looks like nodepool has also been launching nodes that can't talk.  i.e. i guess linaro networking is borked :/20:35
ianwkevinz: ^20:35
ianwi'll send mail too, as that seems to be a bit more reliable20:35
Ramerethfrickler ianw: should all of those be deleted?20:54
RamerethI haven't had time to look into why it keeps happening unfortunately20:55
ianwRamereth: yeah, all of "$ curl -s  http://nl01.opendev.org/image-list | grep osuosl | grep deleting" can be removed20:57
Ramerethianw: k, I'll work on getting those cleaned up later today or at least by tomorrow21:10
ianwfrickler: centos 9-stream lg, thanks -> Image file /opt/nodepool_dib/centos-9-stream-0000005774.qcow2 created...21:33
*** dasm is now known as dasm|off21:38
ianwclarkb / fungi: if you have sec to double check https://review.opendev.org/c/openstack/diskimage-builder/+/846837 and https://review.opendev.org/c/openstack/diskimage-builder/+/844912 that will add f36 to dib. nothing functional this time, thankfully, so should be straight-forward22:27
Ramerethianw: will that url be automatially updated?22:33
clarkbianw: trade you for https://review.opendev.org/c/openstack/project-config/+/847057 and https://review.opendev.org/c/opendev/system-config/+/847034 :)22:38
clarkbfungi: ^^22:38
fungiRamereth: yes, it will update automatically (though perhaps with a few minutes delay for nodepool to notice the change in image list)22:39
Ramerethfungi: sweet, thanks22:40
Ramerethi'm going through them now22:40
clarkbianw: there is a minor whitespace thing I noted in https://review.opendev.org/c/openstack/diskimage-builder/+/844912 but have approved it anyway because zuul and pyymal didn't seem to care22:41
Ramerethalright, I cleared those up. I'll keep an eye on it weekly and do the cleanup until I figure out what's really going on22:52
fungimuch appreciated!22:55
opendevreviewClark Boylan proposed opendev/system-config master: Run zuul cluster reboots and updates automatically  https://review.opendev.org/c/opendev/system-config/+/84619523:00
clarkbfrickler: ^ thank you for the review I think all your comments have been addressed23:01
ianwclarkb: thanks, and done on the others23:13
ianwdoesn't have to be in 846195 but it might be interesting to echo some runtime stats into statsd23:14
ianwjust with "echo <stat> | nc graphite.opendev.org"23:15
clarkbianw: I wonder if we can get timing data out of ansible directly. But then we could send states for time to restart each service?23:16
opendevreviewMerged opendev/system-config master: Fix gerrit deployment dependencies  https://review.opendev.org/c/opendev/system-config/+/84703423:18
ianwi know there's been back-and-forth with ansible and timestamps, and confusion over what the time is on the executing host v the remote host23:19
clarkbI guess the alternative is to run 'time' ourselves and measure it directly?23:20
ianwor register the output of date '+%s' in a pre-task, then do again at the end, subtract * 1000 for ms and send to graphite23:22
opendevreviewMerged openstack/project-config master: Bump jeepyb gerrit image build job to 3.5  https://review.opendev.org/c/openstack/project-config/+/84705723:22
clarkbya that might be better for tracking the whole runtime of the playbook rather than individual tasks but that alone is probably good info to trend time to restart zuul23:23
ianwi guess that mostly does the individual task tracking23:28
opendevreviewClark Boylan proposed opendev/system-config master: Remove Gerrit 3.4 jobs  https://review.opendev.org/c/opendev/system-config/+/84703523:29
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.6 jobs  https://review.opendev.org/c/opendev/system-config/+/84704023:29
clarkbianw: ya I should be able to look at that and see what we can add. One thing that always bothers me is when you have to mix the logic for metrics directly into the code for the thing you are actually trying to accomplish but probably worthwhile23:30
clarkb847040 has finally enqueued some 3.6 image builds23:30
clarkbalso gitea has a 1.17 rc now23:30
clarkbI should look into a gitea update for the rc so that we are ready once 1.17 releases23:31
ianwclarkb: has it turned out we don't need https://review.opendev.org/c/zuul/zuul-jobs/+/846201 ?23:37
clarkbianw: correct. ykarel came up with another owrkaround and I think we can proceed with your removal change instead with appropriate notice23:39
clarkbI've updated my review now23:39
ianwok, i'll send a note out 23:41
opendevreviewMerged openstack/diskimage-builder master: containerfile: warn if we don't have a Dockerfile  https://review.opendev.org/c/openstack/diskimage-builder/+/84683723:42
opendevreviewMerged openstack/diskimage-builder master: Add Fedora 36 support  https://review.opendev.org/c/openstack/diskimage-builder/+/84491223:42
clarkbthe 3.6 build fails with error: package org.apache.commons.lang does not exist23:52
clarkbthe its-base plugin tries to pull that in23:53
fungitoo-old jdk?23:53
clarkbinteresting its-base has a stable-3.6 branch I wonder if that fixes it (stable-3.3 to stable-3.5 branches do not exist)23:53
clarkbthat claims it will fix this. weird23:54
opendevreviewMerged openstack/project-config master: Re-maintenance neutron-fwaas*  https://review.opendev.org/c/openstack/project-config/+/82541223:54
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.6 jobs  https://review.opendev.org/c/opendev/system-config/+/84704023:54
clarkbAnd now I need to go kill blackberries with vinegar. I'll pick this back up tomorrow23:55
fungii thought blackberry was already long obsolete?23:55
clarkbunfortunately the plant versions manages to pop up all over. Birds and other animals do a very good job of spreading them around23:56
fungiyeah, they're all over our yard as well23:56
clarkbbut blackberries will pull moisture from both ends of their stalk so you cut an end off and stick it in a jar of vinegar and after a few days it ends up very dead23:57
fungiwhen i was growing up in the mountains, we'd just lay scrap lumber down across the thickets and walk out into the middle of them to pick the berries23:57
clarkbthings have finally dried out here enough for me to do that23:57
fungii was entirely unaware of the vinegar trick. i'll have to try that here23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!