Tuesday, 2022-02-08

clarkbdoes ansible really have no way to do a file lookup against a remote file?00:03
clarkbI'd like to generate an ssh key pair and then put the pubkey into the test gitea. But if I do that the key is on the remote and you can't read the pubkey back in to upload with the uri module without first copying that file to the host running ansible?00:04
corvusclarkb: slurp handles that00:04
corvus(it's awkward, but that's the mechanism)00:05
clarkbcorvus:thanks. I found a stackoverflow for this and its the same exact use case too (grabbing public keys)00:05
clarkbcorvus: it seems weird to have multiple mechanism for this that work in different contexts. But slurp is what I needed thanks00:05
opendevreviewIan Wienand proposed zuul/zuul-jobs master: Add introduction to the container jobs documentation  https://review.opendev.org/c/zuul/zuul-jobs/+/79896900:06
corvusclarkb: don't forget the b64decode00:06
opendevreviewClark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1  https://review.opendev.org/c/opendev/system-config/+/82818400:09
clarkbcorvus: ^ I think I got it right there. Thanks again. fungi ^ that should test that pushing over ssh works00:10
clarkb(or doesn't I suppose that would be a valid result of this. We may want to split the ssh push out into a separate change on top of 1.15.11 but one thing at a time)00:10
fungimaybe only bother to split it out if it fails00:11
opendevreviewIan Wienand proposed opendev/lodgeit master: Default to private pastes  https://review.opendev.org/c/opendev/lodgeit/+/82820100:25
opendevreviewIan Wienand proposed opendev/system-config master: lodgeit: switch pastes to be private by default  https://review.opendev.org/c/opendev/system-config/+/82820200:46
opendevreviewClark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1  https://review.opendev.org/c/opendev/system-config/+/82818401:08
opendevreviewClark Boylan proposed opendev/system-config master: Test pushes into gitea over ssh  https://review.opendev.org/c/opendev/system-config/+/82820301:08
clarkbthe test push over ssh did end up failing so I've split it in two so that we can ensure it is working on 1.15.11 before digging in too much01:08
ianychoi[m]Hi ianw, hope that you have chance to see translation job failures after https://review.opendev.org/c/zuul/zuul-jobs/+/827588 (from some investigation starting from https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-02-07.log.html#t2022-02-07T14:05:29 )01:12
ianychoi[m]While an ultimate goal should be to migrate Zanata to another new translation platform, would you help some issues regarding wheels?01:13
fungiianychoi[m]: as further investigation revealed, the various failures seem to have started at roughly the same time zuul-jobs switched the ensure-sphinx role to using python3 instead of python201:17
fungii expect explains the coincident timing for all of the various failures you found01:18
fungiit's complicated by the fact that those jobs apply openstack/requirements upper-constraints.txt for their corresponding branches, some of which dropped support for ubuntu-bionic a while ago01:19
fungii expect the lingering support for python 2.7 in constraints is the only reason they were working prior to the ensure-sphinx role change01:19
fungiwe need to brainstorm some options, though if zanata-client really won't work with any jdk version available for ubuntu focal, then that's sure to complicate the available solutions01:21
ianw(sorry, just got to finish something, bib)01:24
ianwhrm, sorry i hadn't seen that01:48
ianwperhaps we need to stick it in a continer01:54
ianwlet me start an etherpad page, as it seems we have a bit to untangle01:54
*** rlandy|ruck is now known as rlandy|out02:03
ianwfocal also has java-802:21
*** pojadhav|afk is now known as pojadhav02:36
ianwok, i at least have a root cause for what's going on with the wheel generation03:14
ianw https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/828213 wheel-cache: update openstack package for 8-stream [NEW]03:39
opendevreviewClark Boylan proposed opendev/system-config master: Test pushes into gitea over ssh  https://review.opendev.org/c/opendev/system-config/+/82820303:55
opendevreviewClark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1  https://review.opendev.org/c/opendev/system-config/+/82818403:55
*** ysandeep|out is now known as ysandeep03:57
mnasiadka_fungi: fantastic, thanks06:33
opendevreviewZhouHeng proposed openstack/project-config master: Re-maintenance neutron-fwaas*  https://review.opendev.org/c/openstack/project-config/+/82541206:55
*** ysandeep is now known as ysandeep|afk07:11
*** persia is now known as Guest220507:19
*** amoralej|off is now known as amoralej07:33
opendevreviewMerged openstack/project-config master: Retire security-specs repo - Step 3  https://review.opendev.org/c/openstack/project-config/+/82717807:35
*** jpena|off is now known as jpena08:03
*** ysandeep|afk is now known as ysandeep08:45
ianwfungi / ianychoi[m]: it looks to me like a confluence of issues.  i'll try to pick things apart, but feel free to add to https://etherpad.opendev.org/p/translation-job-failures-02-2022 and we can see what we can do09:07
opendevreviewMerged zuul/zuul-jobs master: Make CentOS9 jobs voting  https://review.opendev.org/c/zuul/zuul-jobs/+/82752909:19
opendevreviewAurelien Lourot proposed openstack/project-config master: Mirror charm-ops-interface-* to GitHub  https://review.opendev.org/c/openstack/project-config/+/82816609:33
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: [WIP] Fallback to persistent netifs names with systemd  https://review.opendev.org/c/openstack/diskimage-builder/+/82826609:59
*** dviroel|out is now known as dviroel|ruck10:05
mnasiadka_How often the CentOS mirror sync is running? http://mirror.centos.org/centos/8/storage/x86_64/ceph-octopus/ seems to be missing from OpenDev mirrors.10:31
fricklermnasiadka_: I see https://mirror.iad3.inmotion.opendev.org/centos/8-stream/storage/x86_64/ceph-octopus/ , are you using the correct URL? We don't mirror centos/8 anymore afaict10:45
mnasiadka_let me see - Status code: 404 for http://mirror-int.dfw.rax.opendev.org/centos/8/storage/x86_64/ceph-nautilus/repodata/repomd.xml10:46
mnasiadka_hmm, why centos-release-ceph-nautilus package points to 810:47
fricklermnasiadka_: IIUC RDO had to make similar adjustments. to answer your original question, mirror jobs should be running every 2 hours and https://mirror.iad3.inmotion.opendev.org/centos/timestamp.txt should tell you when the last successful sync and publish happened10:53
mnasiadka_frickler: thanks10:54
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: [WIP] Fallback to persistent netifs names with systemd  https://review.opendev.org/c/openstack/diskimage-builder/+/82826611:08
*** rlandy|out is now known as rlandy|ruck11:11
*** ysandeep is now known as ysandeep|mtg12:01
*** pojadhav is now known as pojadhav|afk12:33
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: [WIP] Fallback to persistent netifs names with systemd  https://review.opendev.org/c/openstack/diskimage-builder/+/82826612:37
*** ysandeep|mtg is now known as ysandeep12:55
*** ysandeep is now known as ysandeep|coffee13:04
*** amoralej is now known as amoralej|lunch13:18
*** pojadhav|afk is now known as pojadhav13:31
*** ysandeep|coffee is now known as ysandeep13:39
*** amoralej|lunch is now known as amoralej14:22
opendevreviewAttila Fazekas proposed opendev/elastic-recheck master: Revert "Add query for WARN: failed: route add -net bug 1857151"  https://review.opendev.org/c/opendev/elastic-recheck/+/82835514:52
*** akaha|rover is now known as akahat|dinner14:55
*** dviroel|ruck is now known as dviroel|ruck|lunch15:11
*** ysandeep is now known as ysandeep|dinner15:14
*** pojadhav is now known as pojadhav|dinner15:24
sshnaidmhow can I delete a branch in gerrit repo?15:33
sshnaidm(can I?)15:33
fungisshnaidm: if it's in an openstack project, the release managers have the necessary permissions to call the branch deletion api method in gerrit's rest api15:35
sshnaidmfungi, I'm not sure what is release manager in repo openstack/ansible-collections-openstack15:36
sshnaidmI can tag there and release..15:36
sshnaidmbut I couldn't delete a branch15:36
fungisshnaidm: the openstack release team has branch deletion permissions over all repositories in the openstack/ namespace15:37
sshnaidmfungi, ack, so I need to ask on #openstack-release?15:37
fungithat's probably the easiest way to find them, yes15:37
sshnaidmfungi, cool, thanks15:37
*** ysandeep|dinner is now known as ysandeep15:44
opendevreviewClark Boylan proposed opendev/system-config master: Test pushes into gitea over ssh  https://review.opendev.org/c/opendev/system-config/+/82820315:55
opendevreviewClark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1  https://review.opendev.org/c/opendev/system-config/+/82818415:55
clarkbif that doesn't make things clear I'll probably need to hold the nodes15:55
*** pojadhav|dinner is now known as pojadhav15:58
opendevreviewShnaidman Sagi (Sergey) proposed openstack/project-config master: Give perm to release team to delete branches  https://review.opendev.org/c/openstack/project-config/+/82837115:58
*** dviroel|ruck|lunch is now known as dviroel|ruck16:06
clarkbI've approved the pastes are private by default changes16:40
fungioh! i missed those even got pushed16:51
clarkbfungi: https://review.opendev.org/c/opendev/lodgeit/+/828201 wasthe main one16:51
clarkbthen on the system-config side we updated our test job to check it16:51
opendevreviewMerged opendev/lodgeit master: Default to private pastes  https://review.opendev.org/c/opendev/lodgeit/+/82820117:02
*** jpena is now known as jpena|off17:04
*** akahat|dinner is now known as akahat|rover17:05
clarkbinfra-root: today is EMS billing day. I didn't see the email notifying us of payment so logged in to check on it. I think everything is ok, but one weird thing is when you login it sends you straight to the setup a new home server wizard and asks for your name. I set that before I realized what it was asking for and I think it changed the string from what it was set to before. I've17:13
clarkbsince reset it back to what it should've been before. I don't expect any issues but heads up that we should double check both this month and next month are handled without trouble.17:13
clarkbI guess beware of that logging into the management system you can click on the drop down in the top right to get to the account and existing homeserver bypassing the setup thing. I think their login links are all geared towards people signing up for new service ...17:14
fungiinteresting, thanks for the heads up17:15
*** sshnaidm is now known as sshnaidm|afk17:18
*** ysandeep is now known as ysandeep|out17:21
clarkbnice my gitea ssh testing change finally passes. Once I get data from the child change I can decide if I need to clean up the extra debugging output that is likely to get in the way of typical failure debugging17:31
clarkbapparently the trick was to be very specific about which key to use to auth17:31
opendevreviewClark Boylan proposed opendev/system-config master: Test pushes into gitea over ssh  https://review.opendev.org/c/opendev/system-config/+/82820317:37
opendevreviewClark Boylan proposed opendev/system-config master: Update Gitea to 1.16.1  https://review.opendev.org/c/opendev/system-config/+/82818417:37
fungialso that pr you linked makes me glad i don't maintain any go-based software. having to worry about runtime linker differences between root and rootless containers...17:37
clarkbinfra-root the parent change in that stack should be good to go whenever we are happy with it17:37
clarkbthe child change may deserve more vetting via a held node based on what careful review of the changelog reveals17:38
clarkbbut ssh pushing does seem to pass so I must've gotten the container image updates related to that at least semi correct17:38
*** amoralej is now known as amoralej|off17:39
clarkbinfra-root and looks like today's billing notice came through and looks good. I guess we just watch out for next month but I expect it is fine17:41
clarkbluca mentioned on the mailing list that a gerrit 3.5.1 should happen soon to fix a number of issues. Just calling that out as we ramp up 3.5 investigations19:01
clarkbI'm going to send that service coordinator email now (or as soon as I can get it written)20:00
ianwfungi: ok, always trust the todo list :)20:05
ianwbackup01.ord.rax.opendev.org:/opt/bup-202007 is now mounted20:06
fungithough i went poking on the server out of curiosity and found a /var/lib/redis/dump.rdb.2020-08-21.upgrade20:06
ianwso indeed i had *not* cleaned that up, and cleaning it up is still a todo item :)20:06
ianwnow, how to extract anything from bup is an open question20:07
fungiit may be easier to use the manual db dump i found since it's from roughly the same timeframe as that bup volume?20:07
fungibut that still leaves me with being unsure how to use that dump short of booting a new server20:08
ianwthe only thing is didn't you need a 2019 version?20:08
ianwthat far back might be in the bup history20:08
fungiwell, "2019 q3" is when he said he thinks he last checked that it was intact, no idea how soon after that it was broken20:09
clarkbok email sent. Sorry for missing that previously. And now lunch20:10
fungii suppose i could have rackspace live snapshot that server, boot a copy of it at a new ip address, and then try to import the 2020-08-21 dump into its redis. that might be fairly low-effort20:11
fungiand low-risk20:12
clarkbya that would be one approach to the deploy a new ethercalc to look at the contents of the redis db20:13
clarkbits possible the redis db is human readable too. Though I doubt it20:13
fungifile says "/var/lib/redis/dump.rdb.2020-08-21.upgrade: data" so probably not, no20:13
ianwi don't think i realised ethercalc was not containerised yet20:14
fungiyeah, it's one of those remaining xenial servers i mentioned20:15
ianwhttps://hub.docker.com/r/audreyt/ethercalc looks maintained20:16
ianwtheoretically, that seems like it would just read the db dump20:17
*** prometheanfire is now known as Guest020:56
*** osmanlicilegi is now known as Guest120:56
opendevreviewMerged openstack/diskimage-builder master: Add new container element - Rocky Linux  https://review.opendev.org/c/openstack/diskimage-builder/+/82595721:24
*** Guest0 is now known as prometheanfire21:56
clarkbGerrit java 17 work has begun. Sounds like they have identified a number of things they need to address. This is good news beacuse java 17 supposedly performs quite well22:00
fungioh, very cool!22:01
*** dviroel|ruck is now known as dviroel|ruck|afk22:02
opendevreviewNeil Hanlon proposed openstack/project-config master: Add rockylinux-8 test node  https://review.opendev.org/c/openstack/project-config/+/82843522:57
opendevreviewNeil Hanlon proposed opendev/base-jobs master: Add rockylinux nodeset  https://review.opendev.org/c/opendev/base-jobs/+/82843622:58
opendevreviewNeil Hanlon proposed openstack/project-config master: Add rockylinux-8 test node  https://review.opendev.org/c/openstack/project-config/+/82843522:58
clarkbianw: NeilHanlon ^ I left a comment on that but what I realized is I think we need a dib release23:02
clarkbso that nodepool has access to the updated rocky image building stuff23:02
ianwyes we do need a release for that, and a bump in nodepool requirements.txt to pull in that release23:03
NeilHanlonah, i see thank clarkb - i was looking at https://opendev.org/openstack/project-config/commit/dce378a6b46ebcbd4a88a0306518d1d69917524f for inspiration23:04
ianwalthough the package maps will work, i think there's quite a few points in project-config/nodepool/elements that are matching on "centos" specificially23:04
clarkbNeilHanlon: adding the bits that were removed from that commit should get you most of the way there. The dib release is probably the biggest not straightforward thing, but also those updates tend to not be too bad and ianw has been driving them periodically for us for other random features23:05
clarkbianw: oh that is a good point too. Is there a way to match centos or rocky other than if centos or rocky being checked directly?23:05
ianwi don't think so, unfortunately.  i think it will mostly look like openeuler as a separate match23:06
clarkbgot it23:06
ianwi don't exactly know what openeuler is based on, but it may not be very far from what rocky is doing23:06
clarkbya I think it is also a rhel derivative so openeuler is probably a good reference as far as bits to toggle23:07
clarkbianw: maybe land the fedora-35 locale fix and then do a release? Not sure what else has landed though23:07
ianwyeah, in #openstack-dib stevebaker[m] asked me to look at some things too.  i'll do that today and i think we'll be in shape to release23:08
NeilHanlonare the nodepool/nlXX.opendev.org.yaml files auto-generated or do they need to be touched up manually?23:09
ianwNeilHanlon: they are all under manual control23:09
NeilHanlongotcha, thanks23:09
opendevreviewMerged opendev/system-config master: lodgeit: switch pastes to be private by default  https://review.opendev.org/c/opendev/system-config/+/82820223:18
clarkbianw: re ^ I think we may have to wait for the daily periodic job for lodgeit or manually pull and restart. I'm not in a hurry but thought I'd call it out23:18
opendevreviewNeil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration  https://review.opendev.org/c/openstack/project-config/+/82843523:20
*** rlandy|ruck is now known as rlandy|ruck|bbl23:24
opendevreviewNeil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration  https://review.opendev.org/c/openstack/project-config/+/82843523:25
opendevreviewIan Wienand proposed opendev/base-jobs master: centos-8 : fail if pointing to centos-8-stream image type  https://review.opendev.org/c/opendev/base-jobs/+/82843723:28
opendevreviewNeil Hanlon proposed opendev/base-jobs master: Add rockylinux nodeset  https://review.opendev.org/c/opendev/base-jobs/+/82843623:28
ianwclrkb/fungi: ^ that's where I ended up after our discussion in the meeting about failing the centos-8 node type23:28
opendevreviewNeil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration  https://review.opendev.org/c/openstack/project-config/+/82843523:28
fungiianw: oh, that's a nice lightweight solution23:29
fungihowever, testing it will be key23:30
fungia typo in there will go unnoticed until it merges, and could break every single job we run until reverted (bypassing testing, because it would break its own revert too)23:31
ianwyeah, if you agree on the general vibe, i think we can merge it to base-test and try23:31
clarkbI guess that will work as long as no one else is setting nodesets that way23:32
clarkbI'm not opposed to test it23:32
ianwthe only corner case i can think of is if someone sets centos-8 manually as centos-8-stream23:32
ianwheh, what clarkb said :)  i codesearched and couldn't find anything like that23:32
fungii too wondered about that, but suspected it was a fairly unlikely configuration23:35
opendevreviewIan Wienand proposed opendev/base-jobs master: base: fail centos-8 if pointing to centos-8-stream image type  https://review.opendev.org/c/opendev/base-jobs/+/82843723:38
opendevreviewIan Wienand proposed opendev/base-jobs master: base-test: sync with base/pre.yaml  https://review.opendev.org/c/opendev/base-jobs/+/82843923:38
opendevreviewIan Wienand proposed opendev/base-jobs master: base-test: fail centos-8 if pointing to centos-8-stream image type  https://review.opendev.org/c/opendev/base-jobs/+/82844023:38
ianwfungi: if you could check https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/828219 -- i'll try to look at the other translate failures soon and see if they sync with that being the problem23:47
fungisure thing23:55
fungiianw: were the errors coming from that venv? the failures i looked at were installing doc/source/requirements.txt into the sphinx venv23:57
fungier, doc/requirements.txt i mean23:59
ianwentirely possible i misread it23:59
ianwhttps://etherpad.opendev.org/p/translation-job-failures-02-2022 is my notes23:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!