Thursday, 2023-04-27

opendevreviewIan Wienand proposed opendev/system-config master: gerrit: update OpenDev theme CSS installation
opendevreviewMerged opendev/system-config master: Remove Gerrit 3.6 image builds and test jobs
opendevreviewMerged opendev/system-config master: Add Gerrit 3.8 image builds and test jobs
opendevreviewMerged opendev/system-config master: Add Gerrit 3.7 -> 3.8 upgrade job
opendevreviewMerged opendev/system-config master: Switch to nodepool images on
opendevreviewMerged opendev/system-config master: Update Hound to Use Python 3.11 Base Images
opendevreviewMerged opendev/system-config master: Cleanup unused nodepool-base-legacy role
opendevreviewChing Kuo proposed opendev/system-config master: Build haproxy-statsd with Python 3.11 Base Images
opendevreviewIan Wienand proposed opendev/system-config master: gerrit: update OpenDev theme CSS installation
opendevreviewIan Wienand proposed opendev/system-config master: gerrit: update OpenDev theme CSS installation for Gerrit 3.8
opendevreviewIan Wienand proposed opendev/system-config master: [dnm] trying to get some logs for openafs on centos builds
opendevreviewIan Wienand proposed opendev/system-config master: openafs-client: get logs better
ianwall the nodepool bits restarted successfully with images afaics06:17
ianw"This is a notice that your app - openstack_statusbot - has been suspended from accessing the Twitter API. However, you can self-serve reactivate your app for free." ... yeah, nah06:26
opendevreviewIan Wienand proposed openstack/project-config master: tools/ Add some human readable output
apevecwut, Lon Muck banned us ;)09:29
dpawlikfighting with bots requires sacrifices09:41
*** amoralej_ is now known as amoralej|lunch11:01
fungibut whatever, we can always make more killbots12:41
*** amoralej|lunch is now known as amoralej_13:13
clarkbianw: thank you for looking into the gerrit theming issue. looks great14:50
clarkbianw: comparing the 3.7 change 1 screenshot to the 3.8 change 1 screenshot I think that maybe the green checkmark and red x aren't working under 3.8 with the zuul summary tab. A minor detail but similar in nature to the theme issues I'm guessing14:53
clarkbinfra-root if we get in before end of day tomorrow UTC time the zuul restart process should operate on quay images for zuul instead of docker hub15:07
clarkbthe other big item I've got is the ensure-quay-repo + opendev/base-jobs + opendev/system-config stack to start publishing opendev images to quay with zuul. Reviews on that very much appreciated (and the reviews I'vegotten so far have really helped make this better I think)15:17
opendevreviewElod Illes proposed openstack/project-config master: Prevent recreate EOL'd branch
opendevreviewElod Illes proposed openstack/project-config master: Prevent recreate EOL'd branch
clarkb I think this is the zuul results summary fix. WOrking on a depends on change to check it now16:47
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing the depends on change upstream
clarkbI don't think I'm brave enough but we could theoretically upgrade to gerrit 3.8 before it even releases16:53
*** amoralej_ is now known as amoralej|off17:04
corvusfungi: can you +3 ?17:12
clarkbCSC says authoritative dns for and should be updated now17:20
fungiwhois seems to reflect that, checking tld nameservers17:23 is still returning the old ns records though17:24
fungicould take time to propagate17:24
fungier, sorry was from
fungithe tld not the root17:26 has the updated ns though17:28
fungiokay now is returning updated results too17:28
fungiseems like at least most of them probably have updated already17:29 lgtm now too17:30
fungiclarkb: here's what i saw from one of the tld nameservers a few minutes ago though:
fungisince the ttl on that response was an hour, we should probably give it some time in case any resolvers cached that17:35
fungiregistrars used to caution you to expect something like 48 hours for nameserver updates to propagate completely, though in my experience these days it's on the order of minutes or a few hours tops17:37
corvuswe expect glue records to be in place?17:37
fungigood point, i wasn't even checking that17:38
corvuswell, porkbun seems happy now.  i have switched the auth dns servers for to ns03 and 04.  so i'm guessing "yes" is the answer to that :)17:39
clarkbI don't see glue records from here but I may not be looking correctly17:39
clarkbI would expect the registrar to know they need to add those, but I also wouldn't be surprise if they don't ...17:40
fungii'm not seeing glue records returned from the tld nameservers at the moment, only ns records17:40
clarkbfungi: should we clarify with csc?17:40
fungihere's an example from my personal domain which is hosted from dns servers within itself so needs glue records:
clarkboh querying that server I see A records for ns03 and ns04 in the additional section17:43
fungiokay, now i'm seeing glue records for (ipv4 only):
clarkbso maybe we wait a bit to see if AAAA show up and if not we followup to get those added, but this should be functional in the interim17:44
corvusme too17:44
fungiokay, yeah the v4 glue was there earlier, i was just thrown by not seeing any v6 glue17:44
fungias long as v6-only machines are going through recursive resolvers that have access to the v4 internet, that ought to suffice17:45
clarkbfungi: well we know we have ipv6 only test nodes so fixing that is a good idea17:47
clarkbbut it isn't an emergency17:47
fungithose test nodes are forwarding their queries to other resolvers though17:47
fungiit's only recursive resolvers hitting the tld nameservers directly which need to be able to follow the glue17:48
clarkbah right17:48
fungiso, like, i run a (non-forwarding) caching recursive resolver on my openbsd firewalls. they do have ipv4 access. any v6-only machines on my internal network can still get the v6 addresses of our nameservers if they want them because the resolver they're going through has v4 access17:52
clarkbMy zuul results summary update worked except for I lost color indication. New ps tries to fix that but I'm quickly running up against my lack of understanding of ts and js and css and html17:56
fungithat's all word soup to me18:01
clarkbfungi: tl;dr web development and making things pretty is hard18:02
opendevreviewMerged opendev/system-config master: Switch zuul container images to
opendevreviewMerged opendev/system-config master: Switch the zuul-registry image location to
clarkbwoot my half a stab in the dark doing css stuff seems to have worked18:47
clarkbI'm sure there are other things to address before planning a 3.8 upgrade, but I think the only other thing we know about is double checking the gitweb gitea links18:50
fungiinfra-root: so we had a retired openstack repo mistakenly set to read-only because the wrong acl was picked, and needed undoing. a change to switch to the correct acl merged but jeepyb wasn't allowed to push that. i used the webui as a member of administrators to set it back to active, but rerunning manage-projects for it was a no-op because the cache indicated no change to the acl (presumably19:20
fungibecause of the earlier failure). i edited /opt/lib/jeepyb/project.cache to make the checksum for that project incorrect and ran m-p for it again, which did work as intended19:20
fungiin other news, a tentative release date for debian bookworm has been announced: 2023-06-1019:29
fungiso ~6 weeks out19:29
ianwthanks for sorting out the DNS stuff.  I think that's pretty much it, everything is pointing at the new servers now21:14
ianwclarkb: cool, update looks good.  I briefly looked at moving that plugin to Lit.  It looks like it's pretty similar to polymer conceptually but of course the way to initialize and reference everything is totally different21:16
fungiwait, is polygerrit no longer using polymer?21:32
fungithat's like the new change screen being old21:32
ianwhaha yeah it's moved to
fungii really wish webui frameworks would stop reinventing themselves every 1-2 years. this is getting tiresome21:56
fungisorry, not getting, it's been tiresome for at least a decade21:56
clarkbianw: next up is updating NS records in our zones and the SOA records? I could check the etherpad I Guess22:23
clarkbcorvus: ianw: if you get a chance can you review the trio of and these are the next steps for quay publishing in opendev22:25
clarkbianw: for the zuul application in opendevorg on was a token generate for it? and for the real account you created to act as a bot is the docker cli password saved somewhere? Those are the two bits I need to encrypt as secrets22:26
fungiclarkb: yeah, the remaining changes remove the old ns records for ns1/ns2 and update the soa records to adns0222:30
fungiit's probably safe to do that now, i just didn't want to merge them immediately after registrar updates because tld record changes tend to take time to propagate22:30
clarkbmakes sense22:31
clarkbianw: I just rebased my zuul-results-summary change because gerrit said it was in merge conflict22:31
clarkbc git had no trouble with it but it should be mergeable now I think22:31
clarkbianw: do you think we need to recheck my testing change?22:31
ianwso was just looking at that; i merged the js bundle thing as our concerns were about 3.4 branches which is long gone22:33
ianwi also tried to upload but got22:33
ianwHEAD -> refs/for/main (cannot add patch set to 371674.)22:33
clarkbianw: it let me push. Maybe they don't allow other people to update someones change on that gerrit22:34
clarkbit is a permission thing iirc and we allow it?22:34
ianwyeah "add patchset" is inherited through, refs/for/* ALLOWS it from "gerrit-trusted-users"22:34
ianwi don't know what makes people a trusted user22:34
ianwanyway, looks fine, i merged it22:35
ianwgosh it feels weird to click submit :)22:36
fungidangerous even22:36
ianwthere's nothing else in the queue; the only thing would be updating to Lit which I imagine has to happen at some point, but seems ok for now22:36
clarkbI'm happy to chip away at the least amount of effort that keeps this working22:37
clarkbI seem to recall that 3.8 is also removing robot comment support which will require zuul updates22:37
ianwclarkb: i don't think i created a docker cli password for that real-bot account22:38
fungialso it'll throw our engagement reporting out of whack since i was using that to differentiate between human-added and ci comments22:38
ianwhowever, there was a token created for it; that should be recorded in the usual place22:39
clarkbianw: cool I should be able to generate a docker cli password and reuse that token22:39
clarkbI've been punting on that aspect of my system-config change as it seems like a minor detail compared to getting all the roles and jobs sorted out22:39
ianwi don't see AAAA glue records for opendev.org22:43
ianwand still has the records -- but as discussed this might be something the registrar does and we have no control over22:43
clarkbianw: I think we can request AAAA glue records but I suspect they didn't add them. And yes they seem to have done glue records on their own22:44
clarkbfungi: is there any reason to not request AAAA glue records? I guess if they don't automatically add them then it might not be something they are familiar with and could go wrong?22:44
fungiyeah, that's the only counterargument i can think of22:45
ianwcorvus: seems similarly to have only A glue records (dig +noall +authority +additional +norecurse NS
fungii mean, it's a registrar who apparently still expects you to request nameserver changes by e-mail22:45
corvuserm, why do we think that has any glue records?22:46 shouldn't need glue records22:46
fungithe tld may be returning the glue it has from opendev.org22:46
corvusit's probably the glue records that the authoritative server is returning with a query?22:46
fungiyes, that22:47
corvusfungi: yeah that :)22:47
* fungi gets back to his very important beer22:47
ianwdig +noall +authority +additional +norecurse NS zuul-ci.org22:48
corvusso i think if we ask csc to add aaaa records then the tld will start returning them for any soa query.  or.. they will screw it up and break everything.  :)22:48
ianwdig +noall +authority +additional +norecurse NS gating.dev22:48
ianwi see A records for zuul-ci like that, but not gating.dev22:49
fungidev is a different tld than org22:49
fungiwith different operators22:49
fungithe dev tld is owned and operated by google, fwiw22:49
corvusso a recursive resolver would have to ask .org who is after getting the SOA for from google22:49
ianwahh, interesting.  my .org domain has .com nameservers, and I don't get A records22:50
corvusi'm guessing the .org tld servers don't want resolvers coming back for a second query asking who is after asking for, so it includes the glue records.22:50
ianwso I guess you get back the A/AAAA records if your nameservers are also in .org22:51
fungithat's up to the nameserver implementation usually22:51
ianwi was under the assumption that dig command was returning the configured records, if that makes sense22:52
fungisome nameservers (e.g. bind) aggressively return additional records that they think you might ask for next with your initial query, though unsolicited responses like that have also be leveraged in cache poisoning attacks so recursive resolvers have had to grow filters to make sure they only cache additional responses that make sense in the context of the original query22:52
corvusclarkb: those changes lgtm!22:55
ianwi've updated to reflect on all this23:03
ianwi'll merge first to make sure everything is ok23:04
ianwReleased volume project.tarballs successfully23:49
opendevreviewMerged zuul/zuul-jobs master: Update ensure-quay-repo to run opportunistically

Generated by 2.17.3 by Marius Gedminas - find it at!