| *** dviroel|rover is now known as dviroel|out | 00:19 | |
| ianw | well that's annoying. bionic can't mount the xfs partition on the centos-9 stream ISO | 00:48 |
|---|---|---|
| *** diablo_rojo is now known as Guest2811 | 00:53 | |
| fungi | how is it an iso if it has a partition formatted xfs instead of isofs? | 00:54 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Update centos element for 9-stream https://review.opendev.org/c/openstack/diskimage-builder/+/806819 | 00:54 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing centos 8 image builds https://review.opendev.org/c/openstack/diskimage-builder/+/813912 | 00:54 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: epel: match replacement better https://review.opendev.org/c/openstack/diskimage-builder/+/813922 | 00:54 |
| fungi | i guess they got tired of worrying about rockridge extensions and all that, and assume nobody's booting them with a physical cd-rom drive these days anyway | 01:00 |
| fungi | but still, you'd think a different term would be in order | 01:00 |
| ianw | fungi: yeah, even more confused when it does double-duty with usb keys etc. | 01:14 |
| fungi | openbsd just puts a .fs extension on its bootable install images | 01:17 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing centos 8 image builds https://review.opendev.org/c/openstack/diskimage-builder/+/813912 | 01:27 |
| ianw | "skopeo --insecure-policy copy --all docker://127.0.0.1:2021/10/14 01:17:23 socat[16] W ioctl(5, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...)" ... does this ring any bells | 01:29 |
| ianw | it looks like one of the arguments to skopeo has a bad substituion | 01:29 |
| ianw | https://zuul.opendev.org/t/openstack/build/b7efb8aea39345ff9bc09ea8f884ceee | 01:29 |
| Clark[m] | ianw we skopeo through a socat ipv4 to ipv4 proxy/tunnel because neither skopeo nor docker want to support ipv6 address literals | 01:37 |
| Clark[m] | I think we store the result of the socat listen address in an Ansible var but the socat failed and that got stored in the var instead? | 01:38 |
| Clark[m] | Maybe look earlier in the job where the socat is started and see what the result of that is? | 01:38 |
| ianw | https://zuul.opendev.org/t/openstack/build/b7efb8aea39345ff9bc09ea8f884ceee/console#4/0/2/localhost ... that appears happy | 01:41 |
| ianw | though indeed the root error is in https://zuul.opendev.org/t/openstack/build/b7efb8aea39345ff9bc09ea8f884ceee/console#4/0/2/localhost | 01:41 |
| ianw | Set socat port | 01:41 |
| Clark[m] | Hrm we did just update the zuul images to bullseye from buster with the last restart. Maybe related? | 01:43 |
| ianw | $ socat -d -d TCP-LISTEN:0,fork TCP:198.72.124.102:5000 | 01:46 |
| ianw | 2021/10/14 12:45:48 socat[488127] W ioctl(5, IOCTL_VM_SOCKETS_GET_LOCAL_CID, ...): Inappropriate ioctl for device | 01:46 |
| ianw | 2021/10/14 12:45:48 socat[488127] N listening on AF=2 0.0.0.0:47227 | 01:46 |
| ianw | this could be it. looks like it might have an extra warning line now | 01:46 |
| Clark[m] | "neat" I guess we need to filter that out? | 01:46 |
| Clark[m] | W for warning? | 01:47 |
| ianw | yep, looks like it | 01:48 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: intermediate-registry: handle socat warning out https://review.opendev.org/c/zuul/zuul-jobs/+/813924 | 02:02 |
| ianw | focal can't mount the partition in the 9-stream ISO either | 02:05 |
| ianw | https://zuul.opendev.org/t/openstack/build/3571002a75b44d22b7f0969d96417113/log/logs/centos_9-stream-build-succeeds.FAIL.log#354 | 02:05 |
| Clark[m] | Does mount need an fs type hint? Though I think it relies on whatever the equivalent of block device magic numbers are for that? | 02:09 |
| ianw | it's something deeper, and has do with checksums or something in the on-disk format aiui | 02:10 |
| Clark[m] | The release for 9-stream hasn't happened yet right? I wonder if this is the sort of thing that is open for change still | 02:11 |
| ianw | yeah, i mean the fs being mountable by anything other than the kernel on the iso is probably deep into "don't do that" territory ... | 02:12 |
| Clark[m] | But the reason we have standard filesystems is so that they are portable? | 02:14 |
| Clark[m] | Or maybe I misunderstood | 02:14 |
| Clark[m] | I mean you can even mount a windows fa on Linux now | 02:14 |
| ianw | i'm sure a more recent kernel can, but for now ... | 02:17 |
| ianw | i guess this means you can't build centos element on ubuntu. this is annoying, but perhaps inevitable | 02:18 |
| Clark[m] | Presumably if Ubuntu updates their xfs drivers it would work? | 02:20 |
| Clark[m] | Or is it not even that sort of problem? Can centos 8 mount it? | 02:20 |
| Clark[m] | Its kernel is older than focals I think | 02:20 |
| Clark[m] | But perhaps with backported xfs magic | 02:21 |
| ianw | yeah, i imagine it does, it's all checksumy stuff and i imagine is rhel focused | 02:21 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing centos 8 image builds https://review.opendev.org/c/openstack/diskimage-builder/+/813912 | 02:25 |
| ianw | https://zuul.opendev.org/t/openstack/build/c71ed47d25b34c8f95a016a1d10f47f9/log/logs/centos_9-stream-build-succeeds.FAIL.log#354 | 02:34 |
| ianw | it looks like centos-8 can't mount it either | 02:34 |
| fungi | kevinz: thanks for the cleanup help! | 02:40 |
| fungi | wow, moments ago i got notification that vorlon won't-fixed an ubuntu bug i opened more than 8 years ago, asking for an sru to get security fixes in the cacti packages for precise: https://launchpad.net/bugs/1210822 | 02:46 |
| Unit193 | There was a mass-close since precise is ESM-EOL. | 03:14 |
| ianw | i've filed https://bugzilla.redhat.com/show_bug.cgi?id=2013894 on the weird-xfs-in-image problem | 03:18 |
| *** bhagyashris is now known as bhagyashris|out | 03:30 | |
| ianw | oh, I suppose that "socat 2>&1 | grep 'foo' > /file" means that grep isn't actually flushing | 04:20 |
| fungi | Unit193: aha, i didn't realize it was still under esm until now! | 04:22 |
| Unit193 | I'm not sure it was, but I got a bug closed for pianobar too (one I didn't file.) | 04:22 |
| fungi | it was mostly a fun trip down memory lane | 04:24 |
| fungi | been ages since we had any servers running precise (thankfully) | 04:25 |
| Unit193 | Poked the maintainer about the pastebinit patches again btw, not having much luck on that. I've seen him active on IRC elsewhere, so I may have to annoy him until he does something. :/ | 04:26 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: intermediate-registry: handle socat warning out https://review.opendev.org/c/zuul/zuul-jobs/+/813924 | 04:27 |
| *** ysandeep|out is now known as ysandeep | 04:53 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: intermediate-registry: handle socat warning out https://review.opendev.org/c/zuul/zuul-jobs/+/813924 | 05:04 |
| *** ykarel|away is now known as ykarel | 05:19 | |
| ykarel | ianw, hi | 05:20 |
| ykarel | ianw, while testing c9-stream image build for centos-minimal in dib-functests | 05:21 |
| ykarel | in https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 05:21 |
| ykarel | we are hitting issues | 05:21 |
| ykarel | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d03/811392/6/check/dib-functests-bionic-python3/d034e0c/logs/centos-minimal_9-stream-build-succeeds.FAIL.log | 05:21 |
| ykarel | i checked that for c9 rpms need rpm version to be > 4.16 | 05:22 |
| ykarel | to avoid these errors | 05:22 |
| ykarel | but on bionic/focal rpm version is too old | 05:22 |
| ykarel | can you suggest on how to handle this situation or any other distro build hit similar situation yet | 05:22 |
| ykarel | i can try to build on c8-stream node if that works | 05:23 |
| ianw | ykarel: yeah, i imagined this would be the case ... it is similar with current fedora | 05:44 |
| ianw | about the only option I think we have is to use the containerfile element like the current fedora does | 05:45 |
| ianw | this takes an upstream container image and uses it as the initial chroot environment | 05:46 |
| ianw | note the best way i'd sugest to actually do this is to use the nodepool-builder image, and just call dib out of that | 05:48 |
| ianw | because this needs podman setup, and still needs other build tools etc. all that work is done in the nodepool-builder image, because it's the one we're using to create the gate images | 05:48 |
| ykarel | ianw, ok i can explore how current fedora is being built in ci and apply similar for c9-stream. | 05:49 |
| ykarel | for other options i am not sure where to look at | 05:49 |
| ianw | ykarel: see also my comments on https://review.opendev.org/c/openstack/diskimage-builder/+/806819/ | 05:50 |
| ianw | it's a similar problem, where no current distro can mount the fs in the centos 9-stream qcow2 image | 05:50 |
| ykarel | ianw, yes i am aware about this issue | 05:51 |
| ykarel | we hit that while virt-customizing c9-image on c7 host | 05:52 |
| ykarel | and to get rid of the issue we had to use custom appliance | 05:52 |
| ykarel | https://ykarel.fedorapeople.org/appliance-1.45.6.tar.xz | 05:52 |
| ykarel | similar we had hit for c8 image virt-customize on c7 but later kernel features were backported to c7 and issue went away | 05:53 |
| ykarel | and for c9 i have hear those new filesystem features will not be backported | 05:53 |
| ykarel | but yes some one should reply officially over your bz | 05:54 |
| ianw | yeah, i think it's unlikely they will agree to turn "down" the features, but at least we'll have something to point to | 05:54 |
| ianw | while there are other ways to build images obviously, everyone has there silo. i'm still unaware of anything that has as broad platform support as dib | 05:55 |
| ianw | for all its faults | 05:55 |
| ianw | "their silo" even | 05:55 |
| * ykarel_ got disconnected | 05:58 | |
| ykarel_ | ok i will explore on these options | 05:58 |
| ianw | ykarel: see https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/fedora-container for the container approach | 05:59 |
| ianw | an ingenious idea credited to corvus | 05:59 |
| ykarel_ | ianw, THanks will check in some time, in a meeting currently | 06:01 |
| ianw | heh no rush. but let me know; i certainly think the whole ecosystem will benefit by pushing in this direction | 06:02 |
| *** ykarel_ is now known as ykarel | 06:13 | |
| gthiemonge | Hi Folks, are there any known issues with the gates and stable/train? I have an octavia-tempest-plugin change that runs on stable/train and all those jobs are failing https://zuul.opendev.org/t/openstack/status#698450 | 06:53 |
| gthiemonge | There are similar backtraces on stable/train in other projects: https://zuul.opendev.org/t/openstack/build/a403d641fc7e4d689bc34032c416252a https://zuul.opendev.org/t/openstack/build/b850ddc4e2f942288c610cd99e6738e0 | 06:53 |
| frickler | gthiemonge: that looks like fallout from PyYAML-6.0 which was recently released. we might need to cap that for stable branches | 07:13 |
| frickler | but also devstack-gate should be considered eol really | 07:15 |
| frickler | oh, that role is from devstack-gate natively, which is branchless even. so cap that, but another nail in the coffin I'd say | 07:25 |
| frickler | kopecmartin: gmann: ^^ | 07:25 |
| *** jpena|off is now known as jpena | 07:32 | |
| opendevreview | yatin proposed openstack/diskimage-builder master: [WIP] Add support for CentOS Stream 9 in DIB https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 07:58 |
| frickler | for those following only here, I made a ds-gate patch and am testing it in https://review.opendev.org/c/openstack/octavia/+/813961 , from the console log it looks good so far | 08:09 |
| *** ykarel is now known as ykarel|lunch | 08:14 | |
| *** ysandeep is now known as ysandeep|lunch | 08:35 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing centos 8 image builds https://review.opendev.org/c/openstack/diskimage-builder/+/813912 | 09:07 |
| *** ykarel|lunch is now known as ykarel|lunc | 09:27 | |
| *** ykarel|lunc is now known as ykarel | 09:27 | |
| frickler | infra-root: somethings seems borked between zuul and nodepool. lots of jobs waiting, lots of ready nodes in nodepool. since ~2h | 09:38 |
| *** ysandeep|lunch is now known as ysandeep | 09:39 | |
| frickler | these seem to be about when the issue started, umpteen of them in zuul log https://paste.opendev.org/show/810000/ | 09:43 |
| frickler | not sure whether I should try to restart or give someone a chance to debug | 09:44 |
| frickler | seems this is where things went south https://paste.opendev.org/show/810001/ | 09:51 |
| frickler | looks like https://docs.opendev.org/opendev/system-config/latest/zuul.html#restarting-the-scheduler isn't up2date, but I'm trying now based on root history on zuul.o.o | 10:00 |
| frickler | #status notice zuul was stuck processing jobs and has been restarted. pending jobs will be re-enqueued | 10:01 |
| opendevstatus | frickler: sending notice | 10:01 |
| -opendevstatus- NOTICE: zuul was stuck processing jobs and has been restarted. pending jobs will be re-enqueued | 10:01 | |
| ykarel | ianw, i tried centos-minimal 9-stream on c8-stream node | 10:07 |
| ykarel | it worked fine https://ba4aa52692f4678a0a1b-3dd866e87c38d155f83264836a748517.ssl.cf5.rackcdn.com/811392/7/check/dib-functests-centos-8-stream-python3-image/bb7b147/logs/centos-minimal_9-stream-build-succeeds.PASS.log | 10:07 |
| icey | out of curiosity, does #opendev have a script used to persist and then re-enqueue jobs when Zuul gets restarted? | 10:09 |
| frickler | icey: yes, it is referenced in the above restarting doc, let me check where it is located in git | 10:15 |
| frickler | oh, the link is right in there https://opendev.org/zuul/zuul/src/branch/master/tools/zuul-changes.py | 10:16 |
| icey | :-D | 10:16 |
| frickler | so actually part of zuul proper, nothing specific to opendev | 10:17 |
| frickler | enqueue still running. now that I look at it, I maybe should have dropped the periodic jobs ... | 10:56 |
| *** dviroel|out is now known as dviroel|rover | 11:09 | |
| *** jpena is now known as jpena|lunch | 11:27 | |
| *** akahat is now known as akahat|afk | 11:54 | |
| *** ysandeep is now known as ysandeep|afk | 11:58 | |
| *** jpena|lunch is now known as jpena | 12:22 | |
| *** ysandeep|afk is now known as ysandeep | 12:55 | |
| *** ysandeep is now known as ysandeep|brb | 12:57 | |
| *** bhagyashris|out is now known as bhagyashris|mtg | 13:00 | |
| *** ysandeep|brb is now known as ysandeep | 13:03 | |
| *** akahat|afk is now known as akahat | 13:10 | |
| crohmann | Hey there. I just upgraded to OpenSSH 8.8 an noticed that I cannot connect to review.opendev.org:29418 anymore due to the rsa-sha deprecation on my end. After adding "PubkeyAcceptedAlgorithms +ssh-rsa" for this target it's back. This might be worth looking into as others might run into this sooner or later as well. | 13:27 |
| Clark[m] | crohmann: it is a known issue as gerrit's Mina sshd does not implement key exchange extensions used to negotiate sha2 hashes over the default sha1 fallback. However, the RFC does say clients may fallback to sha2 instead https://datatracker.ietf.org/doc/html/rfc8332#section-3.3 | 13:43 |
| Clark[m] | crohmann: if you can double check whether you are falling back to ssh-rsa or rsa-sha2-* that would be helpful in knowing whether or not Mina doesn't handle a sha2 fallback properly | 13:44 |
| Clark[m] | I had really hoped that the client fallbacks would be updated when sha1 was deprecated. But it seems that this is getting missed | 13:45 |
| Clark[m] | corvus: fungi: frickler: I think we should try to prioritize ianw's 813924 fix for socat stuff. Also, it seems the issue frickler noticed was related to running out of threads after a zk disconnection? Could that also be impacted by the new bullseye runtime? I think the kernel sets max thread limits and docker containers inherit ulimits from the docker daemon so that seems unlikely | 13:48 |
| Clark[m] | Also, we kept the same version of python in the debian switch. Unlikely to be a new python issue either. | 13:51 |
| crohmann | Clark[m]: I played around with it a bit now ... according to ssh -G $hostname sha2 algos are in the default list (i.e. rsa-sha2-512,rsa-sha2-256), but only if I add sha-rsa it seems to find a common cipher. | 14:05 |
| Clark[m] | crohmann: ya so it must be defaulting yo ssh-rsa which that RFC indicates is probably a but if ssh-rsa is not useable | 14:06 |
| corvus | Clark: there were a lot of tracebacks; perhaps there were a lot of threads too. frickler, infra-root if it happens again, getting a thread dump by issuing sigusr2 to the main scheduler process would be helpful. | 14:06 |
| fungi | corvus: thanks for looking, will definitely try to get a pair of thread dumps even next time, so we also get the yappi stats | 14:07 |
| crohmann | Clark[m]: I can see "remote software version GerritCodeReview_3.3.6-44-g48c065f8b3-dirty (APACHE-SSHD-2.4.0)" during the handshake. Is there anything keeping your from an upgrade to 2.7.0 ? | 14:09 |
| crohmann | seems like with 2.5.1 there was a fix for client side support of RFC 8332 --- see https://issues.apache.org/jira/browse/SSHD-1104 | 14:11 |
| crohmann | ah sorry ... I meant with 2.6.0 ... 2.5.1 was the "affected version". | 14:11 |
| fungi | crohmann: it's embedded by gerrit. we might be able to fork their bazel build scripts to force a newer mina-sshd | 14:16 |
| Clark[m] | I'd rather we wait for Gerrit to update. But also we need server side key exchange extensions support. I meant to follow-up on that bug to indicate I don't think they fixed it | 14:16 |
| Clark[m] | They made their client negotiate but for Gerrit it is the server that lacks support. They fixed the wrong end for us (good to fix both ends though) | 14:17 |
| Clark[m] | Its minimal impact to us because users can use a different key type and the impact goes away. | 14:17 |
| fungi | crohmann: it's also worth noting, depending on your preference, another workaround is to switch to ecdsa | 14:18 |
| Clark[m] | Or ed25519 | 14:18 |
| fungi | yeah | 14:18 |
| fungi | some supported ecc | 14:18 |
| corvus | looking at the zuul logs, it appears that the scheduler was quite busy after the restart. so i'm very curious what it was stuck on. but that's hard to determine after the fact considering how busy it was. | 14:20 |
| Clark[m] | But users of these platforms that have killed sha1 RSA should be asking their platforms why the default fallback wasn't changed to sha2 | 14:21 |
| corvus | sorry, not after the restart, i mean after the zk disconnection | 14:21 |
| *** ykarel is now known as ykarel|away | 14:21 | |
| Clark[m] | corvus: periodic jobs maybe? I think the timing isn't a perfect fit but possibly close enough? | 14:23 |
| corvus | 2021-10-14 08:21:40,326 DEBUG zuul.Scheduler: Run handler sleeping | 14:24 |
| corvus | that's after the reconnection, but it never woke up again | 14:24 |
| corvus | i wonder if the event watcher didn't get notified of new events | 14:25 |
| fungi | we didn't have any tickets from rackspace about outages either | 14:26 |
| corvus | swest mentioned an issue where they think that zk watches didn't survive a disconnection. that's not supposed to happen, and i haven't been able to reproduce it, but perhaps that happened here too. | 14:31 |
| Clark[m] | crohmann: https://issues.apache.org/jira/plugins/servlet/mobile#issue/SSHD-1141 is the issue that ianw filed for this. Unfortunately I think upstream misunderstood and thought the problem was using their client and not connecting to their server with openssh. We should bump that issue with questions about the server implementing the server sig algs kex. | 14:35 |
| fungi | and also mention openssh 8.8 had officially deprecated ssh-rsa now | 14:36 |
| Clark[m] | I have to do a school run but I can sort out an account if others don't have one and update that issue | 14:36 |
| fungi | yeah, i don't think i have an apache jira account, but also trying to free up to get back to adding tests to the gitea metadata update filter | 14:37 |
| corvus | i think i've gleaned all i can from the logs right now. hopefully we can get more debug info when it happens agin. | 14:49 |
| fungi | thanks, i'll try to keep a closer eye on it today | 14:50 |
| fungi | any early warning signs we should be looking for to catch it quickly? | 14:50 |
| crohmann | Clark[m]: Thanks for picking up on the ssh issue. Really appreciate that. Should I have raised an issue / bug about this somehwere or was IRC the place to come to? | 14:51 |
| crohmann | fungi: I shall be switching to another key type yes. But still you don't want contributors (think of someone wanting to push just a small documentation fix) to have a hard time debugging their ssh client ... | 14:55 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Allow gitea_create_repos always_update to be list https://review.opendev.org/c/opendev/system-config/+/813886 | 14:57 |
| fungi | crohmann: i completely agree. unfortunately solving the problem is not easy, and involves a lot of finger-pointing between the server and client implementations, with us stuck in the middle | 14:58 |
| fungi | basically the newest openssh client release is not compatible with the sshd that gerrit embeds, even though we've been warning them about it for many months | 14:59 |
| fungi | both the developers of the sshd and the openssh client developers could have made choices which would have avoided this, but neither did | 15:00 |
| crohmann | yeah - I did not want to add to that finger pointing at all. Maybe the necessity to now look at the cipher handshake side of things in SSH again by the Apache Mina devs helps in the long run. This will not be the last cipher or algo that is being deprecated or added. | 15:01 |
| fungi | openssh apparently decided that disabling ssh-rsa didn't imply switching to a fallback other than the one they deprecated, and the mina-sshd devs seemed to think that just supporting rsa2 was sufficient without adding negotiation | 15:02 |
| *** bhagyashris|mtg is now known as bhagyashris|away | 15:03 | |
| fungi | and the only real answer we've had from gerrit and mina-sshd folks is "well switch to ecc, duh, why do you still care about rsa now that quantum computers totally pwn it? oh also isn't it 2050 yet?" | 15:04 |
| crohmann | Clark[m]: I now switched to my ed25519 keypair - just works ;-) | 15:04 |
| crohmann | fungi: I am all about deprecating stuff and getting things forward. But, just like you said, this was never the issue here - it's the lack of negotiation support to find the next best alternative. And that's something that usually is part of any protocol / crypto handshake. Finding a common ground. | 15:06 |
| fungi | yup | 15:07 |
| crohmann | oh well - thanks again for revisiting this issue. | 15:08 |
| fungi | thanks for pointing out that people are actually already using openssh 8.8! | 15:13 |
| fungi | you're the first to say so, and without actual users of an official version's default configuration impacted it's been easier for gerrit and mina-sshd folks to write it off as "a fedora problem" (since fedora disabled it in a similarly broken way in advance of 8.8) | 15:14 |
| fungi | now at least we know for sure that openssh 8.8's default is exhibiting the same symptoms | 15:16 |
| crohmann | full disclose - I am "just" using Arch Linux. | 15:19 |
| clarkb | ya this came up when fedora made the switch with f33 I think. And since then its been a fun back and forth of no one with the ability to actually fix things being particularly interested in fixing things. This more so than quantum computing threats are why I'm starting to use ecc keys | 15:20 |
| clarkb | basically I'm more worried the software won't get appropriate updates than actual exploitable flaws in the algorithm for the near future | 15:20 |
| clarkb | https://issues.apache.org/jira/browse/SSHD-1141 is the actual url that I meant to link before but was on the phoen and that mangled the url to a mobile path that doesn't load iwthout auth? | 15:24 |
| clarkb | anyway ^ you can read more there now | 15:24 |
| clarkb | ok I posted a followup to that issue | 15:35 |
| fungi | thanks! | 15:39 |
| *** ysandeep is now known as ysandeep|dinner | 15:45 | |
| prometheanfire | I'm trying to find the supported python versions for openstack yoga, but can't find docs, anyone have a pointer? | 15:46 |
| clarkb | prometheanfire: https://governance.openstack.org/tc/reference/runtimes/yoga.html | 15:48 |
| clarkb | prometheanfire: I think there may be some ongoing debate for that though | 15:48 |
| fungi | prometheanfire: yeah, that's a palceholder, and what should be added is a topic on the tc's ptg agenda | 15:48 |
| fungi | for now it's effectively identical to xena | 15:49 |
| clarkb | prometheanfire: completely unrelated is gentoo still using dib for image stuff? it came up in a discussion where tripleo suggested that we just use their testing of image builds and we had to point out lots of other platforms use the tool including gentoo :) | 15:49 |
| fungi | i still need to work out how to get gentoo working again for nodepool's testing: https://review.opendev.org/771106 and its parent | 15:50 |
| clarkb | fungi: crohmann: I'll draft an email to service-discuss@lists.opendev.org as well to point out the rsa situation, the related issues, and perhaps push back on the client side too :) | 15:50 |
| clarkb | then we can point people to that if it comes up more as things bump up to openssh 8.8 | 15:50 |
| fungi | er, not nodepool testing, rather zuul-jobs testing | 15:51 |
| opendevreview | Merged zuul/zuul-jobs master: intermediate-registry: handle socat warning out https://review.opendev.org/c/zuul/zuul-jobs/+/813924 | 15:52 |
| fungi | given how long system-config-run-gitea has been going on my metadata update filter change, i think it's not actually filtering | 15:55 |
| clarkb | ha | 15:56 |
| fungi | massive console log so far too, so yeah i think it's doing all of them | 15:59 |
| clarkb | yay for testing | 15:59 |
| *** ysandeep|dinner is now known as ysandeep | 16:00 | |
| fungi | more likely the cause is how i added the role and overrode the var for it | 16:01 |
| fungi | but it's a good exercise nonetheless, since we'll be driving it from ansible too | 16:02 |
| clarkb | yup | 16:02 |
| clarkb | it actually tests things from top to bottom really well | 16:02 |
| fungi | looks like the job's wrapping up, so i'll peruse the archived logs over lunch | 16:03 |
| clarkb | I've sent that email about ssh keys to service-discuss. here is hoping I used enough keywords to have search engines pop that up for people | 16:05 |
| clarkb | fungi: also if you find time going over https://etherpad.opendev.org/p/project-renames-2021-10-15 would be good | 16:06 |
| clarkb | I copied from the previous etherpad and edited to accomodate updates we've made | 16:06 |
| fungi | yep, thanks | 16:06 |
| clarkb | I should get breakfast myself | 16:07 |
| *** marios is now known as marios|out | 16:13 | |
| fungi | interesting that the failure on https://zuul.opendev.org/t/openstack/build/093e4171a13a44b2ad791e5025c62e8d was related to cloning dib (fatal: could not read Username for 'https://localhost:3081': No such device or address)... i wonder if updating every repo crashed gitea | 16:18 |
| Clark[m] | Ya we had to back off doing the description updates all the time iirc | 16:19 |
| Clark[m] | We may need to do a throttle? | 16:20 |
| Clark[m] | The gitea logs should be in the job logs | 16:20 |
| fungi | well, the goal is to only end up refreshing metadata for one repo, so if i get that right then it presumably won't be an issue | 16:21 |
| Clark[m] | ++ | 16:23 |
| fungi | it was more that i was amused at the resulting failure mode | 16:25 |
| fungi | Clark[m]: if you get a moment, can you look at my change to test-gitea.yaml in 813886,3 and let me know if that's what you were suggesting? | 16:26 |
| fungi | oh, is the play an associative array? such that specifying import_playbook more than once overwrites the previous one? | 16:28 |
| fungi | maybe i need to add a separate play for the sync | 16:30 |
| clarkb | fungi: yes you need to split them | 16:32 |
| clarkb | I don't know which one will win in that case | 16:32 |
| fungi | thanks, that helps anyway | 16:32 |
| clarkb | fungi: you also need to update sync-gitea-repos because it hardcode true for that value | 16:33 |
| clarkb | I suspect it was updating everything and that is why (implying the sync entry in the dict won?) | 16:33 |
| clarkb | and then the error could be due to the rename not happening at all | 16:34 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Allow gitea_create_repos always_update to be list https://review.opendev.org/c/opendev/system-config/+/813886 | 16:34 |
| fungi | so importing a playbook and then overriding a var like that doesn't actually override it? | 16:36 |
| fungi | instead i need to override it everywhere we call the playbook? | 16:36 |
| *** dpawlik5 is now known as dpawlik | 16:37 | |
| clarkb | I'm not expert, but my understanding of import_playbook is basically a copy paste | 16:37 |
| clarkb | then think of the vars as setting the variable at the top of that copy paste, then inside the copy paste it gets set again to a different value | 16:38 |
| clarkb | roughly foo = 1; def bar(): foo = 2; print foo; inline bar(); | 16:39 |
| fungi | okay, so where else do we call that playbook? i can't find anywhere it's actually referenced. were we only running it by hand? | 16:39 |
| clarkb | fungi: yes it is only run by hand (and hasn't been run in a long time, it was more useful when we were bootstrapping gitea and the tooling was improvign so we would rerun to update new fields) | 16:39 |
| *** ysandeep is now known as ysandeep|out | 16:39 | |
| clarkb | basically that was our get out of jail free card for gitea updates. You ran it and it did a global update | 16:39 |
| fungi | okay, so it's as simple as just dropping the override and maybe adding a comment | 16:40 |
| clarkb | yes, "calling context should set this value" type deal maybe | 16:41 |
| clarkb | you can do it with ansible-playbook -e foo=bar or the way you've done it | 16:41 |
| clarkb | you can also do a jinja thing to have it default to another value or something. I personally find ansible vars to be complicated in how they are evaluated. having the calling context set it seems reasonable | 16:41 |
| *** jpena is now known as jpena|off | 16:42 | |
| clarkb | it defaults to true in the role too iirc which means if you don't supply it then you'll get the same behavior as the current playbook anyway | 16:42 |
| fungi | it defaults to false, actually | 16:43 |
| fungi | because the role gets added elsewhere that we don't want the metadata sync happening | 16:44 |
| clarkb | oh so it does, I wonder where I saw something making me think it defaulted to true | 16:44 |
| clarkb | always_update=dict(type='bool', default=True) <- the python lib defaults to true | 16:45 |
| clarkb | but the ansible role defaults to false | 16:45 |
| clarkb | this means if you run the python directly you get a different default than executing the role in ansible .neat | 16:45 |
| clarkb | anyway thats fine, this role isn't used often and is manually invoked. We can make it take parameters | 16:45 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Allow gitea_create_repos always_update to be list https://review.opendev.org/c/opendev/system-config/+/813886 | 16:46 |
| fungi | so as simple as that? | 16:46 |
| fungi | future improvement would be to make it able to consume the same input file as the rename playbook | 16:47 |
| fungi | but being able to pass a literal list for now should at least get us what we need | 16:47 |
| clarkb | ya Ithink that should do it | 16:48 |
| *** sboyron_ is now known as sboyron | 17:09 | |
| *** sboyron is now known as Guest2897 | 17:10 | |
| clarkb | fungi: I think there is a problem with your change looking at it again. sync-gitea-repos clones openstack/project-config to get an up to date projects.yaml file. Problem is that opendev/disk-image-builder isn't a project in upstream project-config | 17:12 |
| clarkb | hrm actually maybe that is a non issue, we're just going to clone that repo an then ignore it if grepping for project_config_src shows that we're using the /home/zuul/src location everywhere | 17:14 |
| clarkb | If that is thecase I guess a further cleanup would be to drop that clone entirely | 17:14 |
| fungi | ahh, yeah i hadn't even spotted that | 17:15 |
| opendevreview | Jeremy Stanley proposed opendev/storyboard-webclient master: Update default contact in error message template https://review.opendev.org/c/opendev/storyboard-webclient/+/814041 | 17:21 |
| *** sboyron__ is now known as sboyron_ | 17:36 | |
| fungi | clarkb: looking at https://9a2b40b9abe499183777-8788f9c43324a469e03ac2bb48dfb234.ssl.cf2.rackcdn.com/813886/5/check/system-config-run-gitea/f0cf15b/bridge.openstack.org/ara-report/results/493.html i think you were right about it using the wrong copy of projects.yaml | 18:02 |
| fungi | i don't see opendev/diskimage-builder in the output, though it went fast enough i think the filtering was at least in effect | 18:02 |
| clarkb | fungi: I see "project": "openstack/diskimage-builder" not opendev/diskimage-builder | 18:03 |
| fungi | er, should be opendev/disk-image-builder even (the rename also adds a -) | 18:04 |
| clarkb | ya implying openstack/diskimage-builder came from the old file | 18:05 |
| fungi | right | 18:05 |
| clarkb | oh wait no this is right | 18:05 |
| clarkb | because again the renames don't operate with projects.yaml | 18:05 |
| clarkb | its acompletely separate input file | 18:05 |
| fungi | yeah, but the sync seems to have used the projects.yaml which has the old repo name, pre-rename | 18:06 |
| clarkb | fungi: I think you want a new step between rename and force update that updates the projects.yaml in place to simulate us landing the thing | 18:06 |
| clarkb | fungi: right, because we aren't making an update to projects.yaml as part of this rename | 18:06 |
| clarkb | as that is a completely separate step that we land after the rename is done in production | 18:06 |
| fungi | makes sense | 18:06 |
| clarkb | I think if you edit /home/zuul/src/opendev.org/openstack/project-config/gerrit/projects.yaml to update openstack/diskimage-builder to opendev/disk-image-builder it may work | 18:07 |
| clarkb | and that would simulate us merging the project-config changes after the renames are complete and then manually running the sync repos | 18:07 |
| clarkb | so ya I think in sync-gitea-repos drop the git clone entirely then we know it isnt interferring and in the test-gitea.yaml playbook add a step to do that update in the file in place? | 18:08 |
| fungi | i guess i can call sed -i in a task, or does ansible have a sed-like builtin? | 18:08 |
| clarkb | ansible does have lineinfile | 18:09 |
| clarkb | https://docs.ansible.com/ansible/latest/collections/ansible/builtin/lineinfile_module.html | 18:09 |
| fungi | thanks, found some examples | 18:09 |
| fungi | playbooks/test-update-zuul-description.yaml is a clear one | 18:10 |
| clarkb | oh ya we already do something semi similar | 18:11 |
| clarkb | that job is getting a lot of stuff shoved into it. Might be worthwhile to think about breaking it up into two or three jobs to test different aspects of interatcing with gitea. I'll have to ponder that | 18:12 |
| clarkb | job one might be manage-projects twice to check description update and otherwise noop. job two mangae projects once then rename and do what you are working on | 18:13 |
| fungi | should the git clone in sync-gitea-projects just drop the force: yes so it will create it only if it doesn't already exist? | 18:16 |
| clarkb | fungi: it should be deleted entirely as the path is all wrong and doesn't really get in line with the new way of having zuul manage the repos for us in the jobs | 18:17 |
| clarkb | fungi: basically zuul is keeping project-config up todate in /home/zuul/src/opendev.org/openstack/project-config. We don't need an out of band update unless we're trying to override what is already merged | 18:17 |
| clarkb | before zuul synced that for us each job had to update the /opt repo. its a relic basically | 18:18 |
| *** sboyron_ is now known as sboyron | 18:21 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Allow gitea_create_repos always_update to be list https://review.opendev.org/c/opendev/system-config/+/813886 | 18:23 |
| fungi | hopefully that ^ then | 18:23 |
| clarkb | fungi: close, your new lineinfile thing needs a - hosts: "bridge"\n tasks: header thing. The import_playbooks literally copy paste the playbooks in which have that stuff in place | 18:25 |
| fungi | oh, righty | 18:25 |
| clarkb | the test zuul description update gives you all that if you want to copy it | 18:26 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Allow gitea_create_repos always_update to be list https://review.opendev.org/c/opendev/system-config/+/813886 | 18:26 |
| fungi | indeed, i had actually deleted it after copying. put back now | 18:26 |
| clarkb | I think that should do it | 18:28 |
| clarkb | https://www.phoronix.com/scan.php?page=news_item&px=XFS-Linux-5.10 is the issue the centos 9 xfs fixes I guess. Basically XFS is not Y2038 happy until very new linux. | 18:29 |
| clarkb | I'm going to go out on a limb here and suggest that is a backport worthy feature to have in your filesystem drivers | 18:29 |
| fungi | especially if you expect those systems to be in use 17 years from now | 18:30 |
| clarkb | if we completely ignore the boostrapping problem from centos 8 to 9 for building images it seems that having filesysetms not break in 2038 is a good thing (tm) | 18:30 |
| clarkb | fungi: ya and importantly an fs may long outlive its operating system | 18:30 |
| clarkb | the earlier you get support into filesystems the better imo | 18:30 |
| clarkb | *may long outlive its original operating system | 18:30 |
| fungi | right, that's what i meant by "systems" | 18:31 |
| clarkb | ++ | 18:31 |
| fungi | or at least a subset of what i meant by systems | 18:31 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Build Gerrit 3.3.7 images https://review.opendev.org/c/opendev/system-config/+/814048 | 18:47 |
| clarkb | I don't think ^ is urgent, in fact I'd prefer we just do the renames first and land and restart on that next week. But they finally got the release out the door and now we have a change to build it | 18:47 |
| fungi | yeah, we've crept pretty close to the other maintenance by now | 18:51 |
| fungi | and a gerrit patch update should just be a quick restart now that we're on 3.3 | 18:52 |
| opendevreview | Jeremy Stanley proposed opendev/storyboard-webclient master: Bindep cleanup https://review.opendev.org/c/opendev/storyboard-webclient/+/814053 | 19:22 |
| fungi | https://4d46fce2e9b1003e0d52-32bdd6c3d47a0aba5cb6656809ba5b6f.ssl.cf2.rackcdn.com/813886/7/check/system-config-run-gitea/660961b/bridge.openstack.org/ara-report/results/492.html definitely shows it using the new project name in the update now, so that much is working. i guess i need to add a test that the metadata was actually changed though | 19:39 |
| Clark[m] | fungi: A good check would be to have it change the storyboard url? | 19:44 |
| fungi | yeah, that's what i'm working on | 19:44 |
| Clark[m] | Then you can fetch the page and grep out that url in the html | 19:44 |
| fungi | though also we don't actually test that the rename worked, so adding that at the same time | 19:44 |
| fungi | basically copying the zuul description update test | 19:45 |
| clarkb | ++ | 19:50 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Allow gitea_create_repos always_update to be list https://review.opendev.org/c/opendev/system-config/+/813886 | 19:53 |
| clarkb | Lunch is consumed. The forecast I saw this morning said maybe there wouldn't be rain this afternoon. It was wrong. I'm not brave enough to get a bike ride in the rain yet | 19:53 |
| fungi | i need to switch to dinner prep while that's running | 19:54 |
| clarkb | what you've got looks good | 19:55 |
| clarkb | also when we run this for real what we can do is use a --limit to say gitea08 and make sure it is happy before going to the other 7 | 19:55 |
| fungi | great idea, yeah | 19:55 |
| clarkb | fungi: oh one issue | 19:56 |
| clarkb | fungi: you register dib_content but then check DIB_content | 19:56 |
| clarkb | I think case matters in ansible | 19:56 |
| fungi | d'oh, thanks! | 19:56 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Allow gitea_create_repos always_update to be list https://review.opendev.org/c/opendev/system-config/+/813886 | 19:57 |
| fungi | and now it's really time to make phat khii mao | 19:58 |
| fungi | a bunch of chilis on the deck have turned red | 19:59 |
| clarkb | I picked all my not yet ripened tomatos this week because we finally got proper cold nights | 20:00 |
| clarkb | I'm hoping the ripen on the counteri n paper bags otherwise we'll be eating a bunch of friend green tomatos | 20:00 |
| ianw | clarkb: i've got a few ideas for centos9 i mentioned across the two changes that i'll poke at today | 20:34 |
| ianw | (bib, have to run a errand dropping off car) | 20:34 |
| clarkb | ianw: it is interesting that the issue is related to y2038. seems like making the painful jump is the right thing to do, its just a matter of figuring out if it can be made less painful | 20:35 |
| fungi | i wonder if ubuntu will provide hwe kernels for focal | 20:44 |
| fungi | we might should consider using them in our test nodes and on our servers if so | 20:45 |
| clarkb | fungi: we don't use xfs anywhere ourselves, but I guess if that works around the builders not being able to mount stuff we should do that on the builders? | 20:46 |
| fungi | right, that | 21:00 |
| fungi | but raises the question of whether we want to run different kernels on nb hosts than everywhere else | 21:01 |
| clarkb | ya, we have done it for certain servers in the past (for afs I want to say?) | 21:01 |
| clarkb | fungi: the job failed to find the new storyboard issues url in the content. My browser is having a hard time with these logs though to understand what might have broken | 21:05 |
| fungi | yeah, i'll take a closer look once done eating | 21:06 |
| clarkb | fungi: <a class=" item" href="https://bugs.launchpad.net/disk-image-builder" target="_blank" rel="noopener noreferrer"> <- it seems to have updated but to the lp path (note the - in the name) | 21:08 |
| fungi | huh | 21:09 |
| fungi | maybe the lineinfile addition didn't work? | 21:09 |
| clarkb | maybe. The default fallthrough is lp so ya it is like we aren't matching storyboard then hitting the default which updates the name | 21:10 |
| fungi | the role output will also include a json transformation of the projects.yaml it used | 21:12 |
| clarkb | oh in ara. I'm looking on the zuul console | 21:15 |
| fungi | right | 21:15 |
| fungi | since it's nested, the zuul console view is a bit of a pain still | 21:15 |
| clarkb | ya no use-storyboard in there | 21:15 |
| fungi | mebbe i typoed | 21:16 |
| clarkb | The good news is I think this is working. but I think it would be good to have it actually switch to storyboard from lp to ensure it isn't some easy mode update | 21:16 |
| clarkb | https://778d47dd8b4c675f4bbd-a97fe3629175ad6b40107df87a4a1857.ssl.cf1.rackcdn.com/813886/9/check/system-config-run-gitea/dcd9e59/bridge.openstack.org/ara-report/results/493.html is the ara url by the way | 21:18 |
| fungi | i guess the question is why the second lineinfile task didn't seem to work | 21:19 |
| fungi | maybe the match was wrong? | 21:20 |
| clarkb | or this is some weird ansible behavior. usually when I debug these I set up a simple ansible install locally and run it against localhost and a local file | 21:20 |
| clarkb | then I can fiddle with it and see what it says | 21:20 |
| clarkb | https://778d47dd8b4c675f4bbd-a97fe3629175ad6b40107df87a4a1857.ssl.cf1.rackcdn.com/813886/9/check/system-config-run-gitea/dcd9e59/bridge.openstack.org/ara-report/results/492.html says changed is false | 21:21 |
| fungi | so likely didn't match | 21:21 |
| fungi | i assume the tasks run in sequence | 21:22 |
| fungi | does it maybe not flush the file between those? | 21:22 |
| clarkb | yes ansible is sequenced (one of the reasons people prefer it over puppet) | 21:22 |
| fungi | the second lineinfile is matching on the line changed by the first | 21:23 |
| fungi | can i embed a newline in the first lineinfile instead? | 21:23 |
| clarkb | "When modifying a line the regexp should typically match both the initial state of the line as well as its state after replacement by line to ensure idempotence." that implies to me that they would expect subsequent runs to want the new stuff | 21:24 |
| fungi | the "both" implies that it might not be though | 21:25 |
| clarkb | fungi: they say that because you want it to replace itself on the next run | 21:25 |
| clarkb | so you match the old and new state in the regex then when you run it the first time you go from old to new, then from there on its matching new to new | 21:25 |
| clarkb | I don't think that really matters here since its a one shot for each of them. But I thought maybe it indicates they do expect you to handle idempotence directly. Eg no intentional weird buffering | 21:26 |
| clarkb | I'm installing an ansible locally and will see if I can reproduce this | 21:27 |
| fungi | can i embed a newline in the first lineinfile instead? then i don't have to worry about whether the second one sees the first edit | 21:27 |
| fungi | just basically replace the old project line with the new project line plus the storyboard line | 21:29 |
| *** dviroel|rover is now known as dviroel|rover|afk | 21:30 | |
| clarkb | I don't think is the issue. I can reproduce it locally across multiple ansible invocations | 21:31 |
| fungi | am i not creating the yaml result i think i am? | 21:32 |
| clarkb | you are | 21:34 |
| clarkb | I think this is going to be "ansible is weird" when we figure it out | 21:34 |
| clarkb | fungi: dropping the second line in file and doing line: "- project: opendev/disk-image-builder\n use-storyboard: true" on the first lineinfile seems to work | 21:35 |
| clarkb | I still have no idea why the second isn't working | 21:35 |
| clarkb | but not sure how important it is to understand ansible's weirdness | 21:36 |
| fungi | yeah, i'll just do that | 21:36 |
| fungi | thanks for confirming | 21:36 |
| fungi | i'm about done with dinner now | 21:36 |
| clarkb | I have a hunch that use-storyboard: true shows up in the file later after that match and it decides that is good enough | 21:36 |
| clarkb | so insertafter is nooping as it isn't an insert immediately after maybe? | 21:36 |
| fungi | oh, like insertafter means "anywhere after" and not necessarily "immediately after" because it assumes all lines will be individually unique | 21:37 |
| clarkb | ya I'm trying to test that now | 21:39 |
| clarkb | ya that seems to be it. If I remove all instances of use-storyboard: true from projects.yaml then run what you have it updates as expected | 21:40 |
| clarkb | in that case I think the newline single replacement is about as far as I want to debug :) | 21:41 |
| ianw | clarkb: thanks for chasing up on the mina sshd thing. i ran out of steam on that | 21:43 |
| clarkb | ianw: I'll be honest I'm running out of steam myself. I have a todo on my todo list that is low priority to switch over to ed25519 keys | 21:44 |
| clarkb | I did it locally at home and haven't had any problems. I was going batch that change up in opendev when we can do a infra-root-keys rotation if anyone else wants to get onboard with me | 21:45 |
| clarkb | at this point more because I don't trust people to maintain rsa properly than a true concern that rsa will be attacked successfully in the near future | 21:45 |
| clarkb | That said MINA should totally support rsa :) | 21:46 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Allow gitea_create_repos always_update to be list https://review.opendev.org/c/opendev/system-config/+/813886 | 21:47 |
| fungi | now with less lineinfile | 21:47 |
| fungi | 50% less | 21:47 |
| clarkb | Though I guess unmaintained code is more likely to be attacked | 21:49 |
| ianw | yes i think a 5.10 kernel is the minimum to mount that xfs volume. but we wouldn't do that on nodepool-builders as i imagine we'd use the containerfile approach | 21:53 |
| ianw | but i'm thinking upgrading gate testing from bionic to debian bullseye solves a lot of problems | 21:53 |
| fungi | wfm | 21:56 |
| clarkb | pumpkin carving has begun | 22:04 |
| clarkb | is gerrit showing you very reduced context for changes without a ton of review (like 813886) new in 3.3? At first I was annoyed it wasn't showing me things but now I think I prefer it | 22:05 |
| fungi | i've not really used the gerrit webui much since gertty came to exist | 22:21 |
| fungi | so really wouldn't have noticed | 22:21 |
| clarkb | I have to say I think this is the best the UI has ever been for gerrit. WHich is saying something beacuse we went through some dark days with that one UI | 22:28 |
| fungi | i do like the inverse color theme for the new gerrit, at least | 22:31 |
| ianw | clarkb: yep. the only problem is that it works well on my phone, so i find myself looking at it at times when i really probably should be doing other things :) | 22:31 |
| clarkb | https://issues.apache.org/jira/browse/SSHD-1216 is the new MINA bug for server-sig-algs fwiw | 22:38 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: ubuntu-systemd-container: deprecate and remove jobs https://review.opendev.org/c/openstack/diskimage-builder/+/814068 | 22:38 |
| clarkb | someone else noticed they hadn't fixe dit and filed a new bug for it | 22:38 |
| clarkb | I'm trying to figure out how to subscribe to it in jiar | 22:39 |
| clarkb | Wondering what vote for this issue means | 22:44 |
| fungi | you can facebook-like it | 22:45 |
| clarkb | ya I think that is what this is. Anyway I have done so | 22:45 |
| fungi | latest round of 813886 worked as hoped | 22:50 |
| clarkb | excellent, I guess i should review it for "any reason we can't land this as is" | 22:51 |
| clarkb | I think this should be fine. the sync-gitea-repos playbook isn't run automatically. And the update made to the role library souldn't impact new project creation as it only affects updates out of band after the fact | 22:52 |
| clarkb | fungi: do you think we should edit the projects.yaml for a second change and then leave it out of the list and verify we don't update it? | 22:52 |
| clarkb | that might be overkill | 22:52 |
| fungi | we can, though i think the task runtime proves it's not updating them all | 22:53 |
| clarkb | tahts a good point | 22:53 |
| fungi | we could stand to refine the logging from that script too | 22:54 |
| fungi | like only mention a repo if it's creating it or updating its metadata (and differentiate those cases) | 22:54 |
| fungi | but also not critical | 22:54 |
| clarkb | fungi: have you had a chance to look at the etherpad yet? | 22:55 |
| clarkb | I think if we land 813886 we can update it to have steps for running the sync playbook passing it the list | 22:56 |
| fungi | i have not yet, and may have to get to it in the morning at this rate | 22:57 |
| clarkb | ok that should be fine as we haev a lot of time before starting tomorrow | 22:57 |
| clarkb | and the only thing I have tomorrow morning is taking the kids to school | 22:57 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: ubuntu: add Focal test https://review.opendev.org/c/openstack/diskimage-builder/+/814072 | 22:57 |
| fungi | i'll be around more than expected tomorrow too, the appointment for my auto safety inspection fell through | 22:57 |
| clarkb | thinking out loud here. For Monday it would be nice to land the gerrit 3.3.7 upgrade and then restart on that. Then land the gerrit.config cleanup and the gerrit vs review ansible group cleanup changes and restart again to make sure we didn't accidentally remove something we need | 23:01 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/813716 is the gerrit.config cleanup which is fun because we have a bit of stuff from like gerrit 2.8 in there | 23:02 |
| fungi | i should be able to help with those, though have at least a few ptg sessions i plan to be in as well | 23:05 |
| clarkb | ya I'm good to drive most of it if I can get reviews to make sure ther aren't any silly screwups ahead of time | 23:06 |
| clarkb | also I keep forgetting it is the PTG next week | 23:06 |
| clarkb | I should look at a rough schedule and make sure there isn't anything I absolutely need to sit in on | 23:06 |
| clarkb | fungi: maybe we should do the rename work on a meetpad call? double check there aren't any major issues there? | 23:07 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: functests: drop apt-sources https://review.opendev.org/c/openstack/diskimage-builder/+/814074 | 23:09 |
| fungi | good idea, yep! | 23:09 |
| clarkb | looks like I should probably try and be around for openstack TC discussions around centos8 EOL and python runtime selections for yoga since both of those impact us if they choose weirdly | 23:15 |
| fungi | and i have to miss the second half of the tc session on monday as it overlaps with the security sig session | 23:16 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: centos7 : drop functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/814075 | 23:19 |
| stewie925 | hi guys, I m browsing opendev repository and I selected a random project like openstack.keystone - I wanted to see how they code logging. but when I go to the search box and type "logging" nothing came out | 23:20 |
| clarkb | stewie925: yes, code search for gitea is known to be broken. We have kept https://codesearch.opendev.org running as a result | 23:21 |
| clarkb | stewie925: they use some homegrown golang text indexer thing and it doesn't seem to do a great job. If we can ever get around to running a true gitea cluster again we can try the new elasticsearch/opensearch based search instead | 23:21 |
| stewie925 | clarkb: thank you! | 23:21 |
| clarkb | stewie925: they probably use oslo.logging though | 23:23 |
| clarkb | er oslo.log I guess | 23:24 |
| clarkb | https://docs.openstack.org/oslo.log/latest/ | 23:24 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: functests: drop minimal tests in the gate https://review.opendev.org/c/openstack/diskimage-builder/+/814078 | 23:49 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Remove extras job, put gentoo job in gate https://review.opendev.org/c/openstack/diskimage-builder/+/814079 | 23:49 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Simplify functests job https://review.opendev.org/c/openstack/diskimage-builder/+/814080 | 23:49 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Run functional tests on Debian Bullseye https://review.opendev.org/c/openstack/diskimage-builder/+/814081 | 23:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!