| corvus | i feel like that one is a toss up | 00:13 |
|---|---|---|
| *** liuxie is now known as liushy | 02:00 | |
| opendevreview | Ian Y. Choi proposed openstack/project-config master: Use prepare-weblate-client job & run Weblate scripts https://review.opendev.org/c/openstack/project-config/+/961499 | 05:19 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Revert "arm64: Introduce semaphore for arm64 build jobs" https://review.opendev.org/c/opendev/zuul-providers/+/969342 | 05:48 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: nodepool-base: Add firewalld zuul-console rule https://review.opendev.org/c/opendev/zuul-providers/+/967962 | 05:50 |
| mnasiadka | The revert ^^ is no brainer I guess, and 967962 will fix lack of console streaming on Rocky OpenDev images | 08:55 |
| opendevreview | Piotr Parczewski proposed zuul/zuul-jobs master: Fix missing package https://review.opendev.org/c/zuul/zuul-jobs/+/969372 | 10:43 |
| opendevreview | Seongsoo Cho proposed openstack/project-config master: i18n: Add new Zuul job for Weblate migration testing https://review.opendev.org/c/openstack/project-config/+/961499 | 11:51 |
| *** dhill is now known as Guest32796 | 13:27 | |
| opendevreview | Merged zuul/zuul-jobs master: Remove python2-devel from bindep for Fedora https://review.opendev.org/c/zuul/zuul-jobs/+/954257 | 14:16 |
| opendevreview | Merged opendev/zuul-providers master: Revert "arm64: Introduce semaphore for arm64 build jobs" https://review.opendev.org/c/opendev/zuul-providers/+/969342 | 14:35 |
| fungi | clarkb: https://review.opendev.org/921878 has a question addressed to you (i'd try to answer it myself, but i'm not sure i fully understand the implications of that choice) | 14:47 |
| fungi | not urgent | 14:47 |
| clarkb | fungi: I think my response would be we should continue to do whatever the old zanata jobs do | 15:43 |
| clarkb | (I don't know what that is and would have to look it up) | 15:43 |
| clarkb | left a quick message about that | 15:47 |
| fungi | thanks, there's another depends-on change that was being held up by making a decision on that, which is how i spotted it | 15:47 |
| mnasiadka | Any option for reviews on https://review.opendev.org/c/opendev/zuul-providers/+/967962 ? I’d happily accept working zuul_console streaming on Rocky :-) | 16:12 |
| clarkb | mnasiadka: hrm how does that work with centos stream then? In theory iptables rules are translated by $tool to nftables? | 16:13 |
| clarkb | mnasiadka: I don't want us to have to support 3 different firewall technolofies. Can we convince rocky to respect iptables instead? | 16:14 |
| clarkb | looking at our package map we should install iptables-services on family redhat | 16:15 |
| clarkb | https://reintech.io/blog/implementing-ip-address-blocking-iptables-rocky-linux-9 implies this should work but maybe we need to enable services? | 16:17 |
| clarkb | (as a side note we're probably finally approaching a state where we could convert to nftables across the board, but I'm not positive of that) | 16:18 |
| mnaser | mnasiadka: fwiw would be nice to have this in the zuul_console module / task so in zuul-jobs so it just works(tm) like the other ones do (i guess) | 16:18 |
| mnasiadka | clarkb: centos stream installed from centos-minimal element somehow gets iptables, rocky-container installs dnf Minimal group which includes firewalld | 16:18 |
| mnasiadka | We could probably uninstall firewalld in rocky image build | 16:20 |
| mnasiadka | mnaser: that’s another option I guess | 16:20 |
| mnasiadka | clarkb: but I think sooner or later iptables is going to go - currently EL10 rather has a compatibility layer that iptables CLI creates nft rulesets | 16:22 |
| clarkb | mnasiadka: yes iptables is nftables compat layer for most modern distros now | 16:23 |
| clarkb | mnasiadka: I think the hope is that nftables support will be on all the platforms we build and then we can convert to it | 16:23 |
| clarkb | which may be the case once bionic goes away (and maybe bionic is new enough?) | 16:23 |
| clarkb | I do not think we should use firewalld at all for the same reason I don't think we should use ufw | 16:24 |
| mnasiadka | Actually just installing iptables might work in the rocky image - let me try | 16:24 |
| clarkb | they are somewhat platform specific and different platforms choose different tools by default. Instead sticking to the kernel system like iptables/nftables is more portable | 16:24 |
| clarkb | mnasiadka: I think we already install iptables on rocky | 16:24 |
| clarkb | mnasiadka: but maybe we aren't enabling it | 16:24 |
| mnasiadka | clarkb: yes, iptables-nft seems to be enabled, but somehow the rules created by nodepool-base are not getting applied | 16:26 |
| mnasiadka | Ah, it gets implemented in /etc/sysconfig | 16:27 |
| mnasiadka | So might be iptables systemd service is not enabled, let me check | 16:28 |
| clarkb | mnasiadka: zuul-providers/dib-elements/nodepool-base/post-install.d/20-iptables does appear to attempt to enable the service. But ya maybe that isn't working | 16:29 |
| clarkb | infra-root I submitted an EMS support ticket via the website which you can now see quoted in their response to the infra root email address | 16:36 |
| mnasiadka | clarkb: regarding https://review.opendev.org/c/zuul/zuul-jobs/+/966187 - I think it’s ready for review, tried to implement what is documented in zuul docs around mirrors | 16:41 |
| clarkb | mnasiadka: cool I'll try to take a look today (probably this afternoon at the earliest) | 16:44 |
| mnasiadka | No rush | 16:44 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Improve opendev's matrix gerritbot repo list https://review.opendev.org/c/opendev/system-config/+/969497 | 16:49 |
| mnasiadka | iptables.service on rocky is disabled, that explains things | 16:51 |
| clarkb | mnasiadka: so now we need to look at the build logs to see why zuul-providers/dib-elements/nodepool-base/post-install.d/20-iptables isn't enabling it as expected | 16:52 |
| mnasiadka | We might need to enable DIB_DEBUG_TRACE to get a meaningful output, I’ll try that out | 16:54 |
| mnasiadka | Well, without debug I see that symlink was created in the logs | 16:58 |
| mnasiadka | I mean in the build logs | 16:58 |
| clarkb | weird. Maybe a later element script disable its somehow? | 17:00 |
| mnasiadka | https://zuul.opendev.org/t/opendev/build/5c2e48fbbf1b4649853a788e4380970a/log/diskimage-rockylinux-10.log#7467 | 17:01 |
| mnasiadka | That’s nearly the last thing in the build, that would be somehow weird. | 17:01 |
| clarkb | maybe firewalld has a rule that takes precedence and prevents it from starting? If that is the case we could disable firewalld maybe | 17:02 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: nodepool-base: Mask firewalld unit https://review.opendev.org/c/opendev/zuul-providers/+/967962 | 17:09 |
| mnasiadka | clarkb: ^^ let’s see if that helps | 17:09 |
| clarkb | mnasiadka: does that work if the system doesn't have a firewalld service? I guess CI should tell us (a nice chnage from the old nodepool system) | 17:10 |
| mnasiadka | clarkb: tested in a fresh rocky container and masking works even if there’s no such service, it just creates a link to /dev/null without checking if the service exists | 17:12 |
| clarkb | perfect | 17:12 |
| fungi | fwiw, debian is dropping the iptables compatibility layer in forky, so in a couple of years we'll need to be using nftables directly | 17:36 |
| clarkb | it does look like bionic and bullseye do nftables so we can start converting things I think | 17:37 |
| clarkb | but like I said I think we should do iptables -> nftables for portability reasons rather than iptables -> ufw + firewalld + nftables | 17:37 |
| fungi | yeah, i switched to installing a /etc/nftables.conf on my debian systems years ago | 17:40 |
| clarkb | someone should add it to our big todo list etherpad | 17:40 |
| clarkb | I'll do it after this meeting if no one beats me to it | 17:41 |
| fungi | conveniently, it's more human-friendly than iptables/chains, though my preference is still for openbsd pf when it comes to that | 17:41 |
| clarkb | ubuntu security repos must be undergoing updates beacuse 969497 keeps hitting errors trying to fetch packages (those jobs use prod apt list configs which don't use our ci mirrors) | 17:53 |
| clarkb | I'll recheck it again in a bit to give the mirrors a chance to catch up | 17:53 |
| clarkb | I've updated the todo list with a note about nftables | 17:56 |
| Matko[m] | I'm not sure who to reach out about it. The openinfra 2025 playlist on youtube has a lot of duplicates: 2-5 times the same video: https://www.youtube.com/playlist?list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s | 17:58 |
| clarkb | Matko[m]: we can't fix that ourselves, but fungi and I should be able to get in contact with those who can. Thank you for the heads up | 17:59 |
| Matko[m] | clarkb: thanks =) | 18:00 |
| clarkb | I've let people on the foundation side know and hopefully it will get fixed soon (but haven't heard anything yet) | 18:01 |
| clarkb | Matko[m]: it is being looked into now | 18:12 |
| clarkb | I'm going to do another pass on gerrit upgrade testing shortly. This time with large h2 cache file removal included | 18:12 |
| Matko[m] | clarkb: thanks for the follow up. I already see the number of videos in the playlist decreasing. | 18:13 |
| mnasiadka | fungi: I miss times running firewalls using pf (on FreeBSD, but no difference) :-) | 18:25 |
| clarkb | apparently freebsd pf performance is much better than openbsd | 18:27 |
| clarkb | whcih doesn't matter so much at gigabit speeds but as networks get faster people have writtne about needign to switch to freebsd to keep up | 18:27 |
| fungi | mnasiadka: i used to build productized network security and acceleration appliances purely with openbsd (loved when they added their own load balancer service too, it's amazing), but i still run it at home to this day because it's so darned convenient | 18:30 |
| fungi | in particular, the fact that you can cluster and synchronize state table entries between active/active/... firewalls is still leaps beyond what you get from a lot of supposedly "enterprise" firewall solutions | 18:32 |
| clarkb | Matko[m]: cleanup should be complete now. Thanks again for the report | 18:35 |
| mnasiadka | Yup, used carp and pfsync in pre-2010 solutions ;) | 18:35 |
| fungi | also loved how they continued the fish theme with carp being their replacement for the (cisco-)patent-encumbered crrp protocol | 18:37 |
| fungi | er, vrrp | 18:37 |
| Matko[m] | clarkb: :D | 18:38 |
| fungi | if memory serves, cisco had decided that rarp infringed on their vrrp patent, so carp had to redesign rarp in such a way that it skirted the patent | 18:39 |
| fungi | though i think the vrrp patent is long since expired | 18:39 |
| fungi | ah no, too many beers ago. the internet says vrrp was a standard but cisco was claiming it infringed on their hsrp patent | 18:41 |
| fungi | regardless, yes it looks like their patent on hsrp expired 10 years ago now | 18:42 |
| clarkb | ok gerrit upgrade with caches moved aside (we copy things like indexes aside too so I figured we can move the cache aside just in case) seems to have worked on the test node. I am able to see diffs and changes after the upgrade and don't observe any complaints from the service | 19:06 |
| clarkb | I do notice that we dont' actually seem to get the Blocked users ACL update as part of the upgrade seems to be specific to new site initialization | 19:07 |
| clarkb | that is fine and I'm keeping the steps to check it in prod just in case. I think we can followup after the upgarde and add that acl at any time if we think being in sync with upstream on that is important but I don't think it affects the upgrade eitherway. Mostly we just want to know if it happens | 19:07 |
| fungi | we can always adjust that post-upgrade so long as we know | 19:08 |
| clarkb | exactly | 19:08 |
| clarkb | corvus in the upgrade document we also have a step to restart zuul schedulers to pick up the new gerrit version. Since we're upgrading on Sunday I expect eh weekly zuul upgrades to be complete by then. Should we manually restart schedulers this time given that? and just to double check only schedulers need a restart not web too right? | 19:08 |
| clarkb | and now I'll run through the downgrade again. This time I will also move the caches aside and double check that doing so is ok in a downgrade scenario | 19:09 |
| corvus | clarkb: yes i think it's worth it, though i don't expect any changes, since i don't think we've conditioned anything on any recent gerrit versions. i would do both schedulers and web for sanity/consistency. | 19:10 |
| clarkb | cool I'll update the notes to add web and indicate we should do that step this time | 19:11 |
| fungi | mainly i think it's good to have zuul restarts baked into the process in case it conditions features on newer-than-we're-running gerrit in the future (however unlikely since we'll probably be the ones to discover the version requirements, of anyone) | 19:12 |
| clarkb | fungi: yup that is why its in there but we note that depending on timing we can sometimes let weekly restarts handle it | 19:13 |
| clarkb | in this case I think we're about as far from the next weekly restart as possible so worth doing manually | 19:13 |
| fungi | but still, we might discover that a zuul feature needs newer gerrit than we have, and bake in a version check to get us back to the old behavior until we upgrade, so still a distinct possibility | 19:13 |
| clarkb | downgrade looks good with h2 cache removal as well | 19:20 |
| clarkb | I should note that the test node only has like 5 chagnes total and its caches are tiny but it is good that it loads them up anyway | 19:20 |
| clarkb | but this all continues to look good to me even with the slightly modified process | 19:20 |
| clarkb | I'll look into staging some of the post upgrade tasks soon as well | 19:21 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/969497 addresses the matrix gerritbot issue you pointed out earlier today if you want to take a look | 19:50 |
| clarkb | I'm going to grab lunch now then probably pop out for a bike ride after | 19:57 |
| opendevreview | Enzo Candotti proposed openstack/project-config master: Add SeaweedFS App to StarlingX https://review.opendev.org/c/openstack/project-config/+/969527 | 20:06 |
| opendevreview | Merged opendev/system-config master: Improve opendev's matrix gerritbot repo list https://review.opendev.org/c/opendev/system-config/+/969497 | 21:57 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!