Tuesday, 2024-01-30

opendevreviewMerged opendev/system-config master: Revert "Switch from legacy to new style keycloak container"  https://review.opendev.org/c/opendev/system-config/+/90711900:06
clarkblast call for meeting content00:07
JayFWeird question: do we generally do anything to set /bin/sh on our testing images. 00:31
JayFor do we leave it at the default (dash on ubuntu)00:31
clarkbshould be the default00:34
* elibrokeit waves to JayF00:34
JayFelibrokeit: found https://review.opendev.org/c/openstack/liberasurecode/+/907156 and we were just conjecturing how it passed CI00:34
JayFgiven afaict the defaults on ubuntu should make dash the /bin/sh provider00:35
clarkbour ubuntu and debian images are built with debootstrap and that should pull in default shell stuff. On CentOS things are built with dnf/yum and should produce similar results00:35
elibrokeitthis is actually hilariously complex00:35
elibrokeitso, debian and ubuntu as a result default to dash00:35
JayFSo basically I'm trying to ensure we haven't shipped bash-centric code elsewhere it doesn't belong 00:35
elibrokeitbut dash once did not support the basics to be a valid autoconf shell00:36
elibrokeitthey added support a decade plus ago, then a bunch of software predictably failed in debian buildbots to build from source: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=58295200:36
elibrokeitso, they passed the configure option to dash to disable valuable POSIX features00:36
elibrokeitthis restored the pre-2010 fact that a configure script will check to see if the current shell is basically capable of POSIX, and if not -> it checks for bash and tries to re-exec as bash00:37
elibrokeityou cannot really control this in configure.ac but you can control it by defacing /bin/sh itself00:37
clarkbJayF: https://paste.opendev.org/show/bjLLwdrP6DWxiHkaQRws/ jammy node on a running test I ssh'd into to check00:37
clarkbif I had to guess whatever ran the shell was selecting bash00:38
elibrokeitdebian has flipflopped several times on whether /bin/sh has $LINENO enabled, and the ultimate debian goal is to fix all packages shipped by debian's own archives and then enable $LINENO 00:38
JayFclarkb: that's basically what I was concerned about, if there's something weird that's configure-specific I think that is a small enough set of things that we don't worry about it :)00:39
JayFI'm a bash lover, but I #!/bin/bash all the things, and I may have never in my life written a configure script :)00:39
elibrokeitGNU created configure.ac technology in order to be maximally portable, so they kinda don't want configure.ac authors to be able to influence which shell it runs with :)00:40
clarkbelibrokeit: you said just above it will use bash though?00:41
JayFwell, the build is either broken or requires bash00:41
elibrokeitit will only use bash until debian's own archives are fixed, then they will re-enable $LINENO support in their dash package and thirdparty projects will find their builds failing00:41
clarkbmy point is the test environment we provide isn't divergent from the ubuntu default of dash as sh00:41
clarkbso if dash is not compatibile then something must be choosing bash00:42
JayFclarkb++ that was the only question I was asking, yeah00:42
JayFand I don't wanna get into the weeds of CI for a library I don't know much about, I just wanted to make sure we hadn't slipped a "and replace /bin/sh with real bash" into our CI jobs at some point in the last decade  :D 00:42
JayFit sounds like the answer is soundly no, something else weird was going on with that change, and I'm not interested as much in that change as I am the general case :D 00:43
elibrokeitanother debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=84224200:43
clarkbJayF: some of us run systems that use bash :)00:43
elibrokeitsomeone wanted to stop cripping /bin/sh on debian, and for a while it was un-crippled again00:43
JayFclarkb: I <3 bash, but I also like knowing our stuff works where advertised. Especially in a world where people ship/build stuff in slimmed containers00:44
elibrokeitthen they re-crippled it *again* after some more reports of debian packages failing to build from source came in00:44
JayFelibrokeit: I just hear you say we're not weird for being temporarily broken, too :) 00:44
clarkbJayF: you should not build in your final slim container00:44
elibrokeitbasically, the core issue here is: that configure.ac is not "properly good" but debian is not a good test environment for this00:44
JayFclarkb: I agree, but I'm not 100% of everyone on the internet :D 00:44
clarkbJayF: I understand the motivation there but I think it is misplaced00:44
clarkbelibrokeit: right and we intentionally do our best to provide an ubuntu system that mimics actual ubuntu installations00:45
clarkbsame for debian and centos and so on00:45
JayFMaybe, but I think about ops in terms of apis oftentimes, and if an api says "this is posix shell" we should make it posix shell, and that's what gives us the opportunity to work on platforms we don't always target00:45
clarkbreplacing dash or changing how dash is configured would be a problem for us because now our test environments don't mimic actual installations00:45
JayFthat's the primary reason I care about that :D 00:46
JayFclarkb+++++++++++++ that is 100% the heart and soul of why I asked the question00:46
elibrokeitand to be clear: this *will* break one day, once debian re-enables a feature-complete dash shell00:46
clarkbright but the fix isn't to change the shell00:46
clarkbits to change the autoconf00:46
JayFThat's what eli pushed00:46
JayFthe original linked patch00:46
elibrokeithowever, for openstack you don't really need to worry about that for now, just merge a patch if you think it is correct -- and rest easy knowing that this won't be an issue in 2 years00:47
elibrokeitand that's what my patch does00:47
JayFelibrokeit: nobody in this conversation can merge that patch fwiw :D 00:47
JayFwell, I mean, clark can do whatever he wants in gerrit, he's got a big hammer on the systems, but can and should are not the same ;) 00:47
elibrokeitno problem :)00:48
elibrokeitjust trying to clarify the risks involved00:48
JayFyeah, I'm going to ask a question but taking it to another channel, we've bugged these folks enough I think :D 00:48
JayFthanks o/00:48
clarkbalso it isn't clear to me how LINENO support affects [/test functionality? Maybe they are all bundled up into the same set of functionality that is enabeld and disabled.00:48
elibrokeitGNU autoconf has a weird suite of tests that it runs at the beginning of a configure script, to check how good your /bin/sh is, and if it detects something that isn't a very good POSIX sh (for example, the solaris 10 /bin/sh) it will also search for some other shells such as bash and zsh and try to re-exec as that00:49
elibrokeitone of its internal tests is for $LINENO support00:49
JayFso LINENO is just the canary that makes configure say "I'm going to run as bash instead, because this sh is so bad"00:50
clarkbgot it00:50
JayFI think I finally understood that for the first time across like, 2 channels and DMs of this conversation LOL00:50
clarkband [ will never work under dash so when LINENO is present it runs dash then fails00:50
JayFWhich is why the issue is exposed on Gentoo, but not on the versions of Debian we support today00:51
elibrokeitno, [ works fine, but [ foo == bar ] is not valid because /bin/sh doesn't know what "==" is00:51
JayFthat's the root I've been looking for the whole time: why eli saw it on gentoo and we didn't see it on debian/ubuntu00:51
elibrokeitwell, that does mean that the [ program emits an error :)00:52
JayFbut the build would succeed in that case anyway lol00:52
clarkbah ok I saw the original change which switched to test implying [ was the problem00:52
JayFhttps://review.opendev.org/c/openstack/liberasurecode/+/907156/1/configure.ac#167 just with all specific CPU support would be disabled00:53
clarkbfwiw you should squash those two changes and abandon the child00:53
elibrokeitit would emit scary QA warnings if your package manager catches those, then disable tons of useful features, yeah00:53
clarkboh wait it is two changes00:53
elibrokeitclarkb: they are two distinct problems, although they do both appear on the same lines00:53
clarkbI have no say but I think I'm -1 on the second00:53
clarkb[ is the same as test they aren't different not sure why it matters00:53
clarkbI guess this is specific to autoconf00:54
clarkbif it were a shell script I'd care00:54
JayFper the commit, when you run configure through M4 scripting, it changes things, and test just lets you avoid escaping00:54
elibrokeitalso yes, the first one is an actual bug today but the second one is a style issue, so I figured I would allow the maintainers to decide if they care00:54
JayFif it were a shell script, I'd -1 both changes and say "just shebang it to bash, openstack requires bash" :D 00:54
* elibrokeit knows entirely too much about autoconf, all of it bad00:55
* JayF &00:56
clarkbI can't remember who I first heard say it but its funny how `sh` is the standard but all the `sh`s are different these days so you are more portable if you actually write against bash00:58
opendevreviewMerged opendev/git-review master: Add --wip as an alias to --work-in-progress  https://review.opendev.org/c/opendev/git-review/+/90650801:04
* tkajinam didn't know the --work-in-progress option. that's nice01:54
fungitkajinam: -w and -W can also be used to set/unset the wip toggle at upload02:50
tkajinamfungi, yeah I noticed there are several options I was not aware of. it's good chance to learn these (because my previous workflow can be hugely optimized by some of these)02:55
timburkei'm glad i noticed this convo about liberasurecode! fwiw, i doubt any of the current maintainers know as much about autoconf as elibrokeit, and in general we've been pretty lax wrt style even for the C code (much less the ac stuff)04:47
elibrokeittimburke: for my sins, I'm a core developer for the meson build system :)04:47
timburkegod help me, i had commit rights to eventlet before it was cool ;-)04:48
elibrokeitthat would be the project where someone squashed all my fine-grained commits using the github merge button04:49
* elibrokeit reaches desperately for gerrit04:50
timburkeelibrokeit, as long as i've got you -- got any ideas about how i could force a more-strict build env in the libec gate? i'd love ot have another job that actually wouldn't have passed without your fix in https://review.opendev.org/c/openstack/liberasurecode/+/90715604:54
tonybfrickler: Yes, that's the current situation. There are ~28 nodes that are visible in `openstack server list` but `openstack server delete $UUID` says no such record.04:57
elibrokeittimburke: if you set `CONFIG_SHELL=/bin/sh ./configure then the configure script will understand that you "darned well want to use this specific shell" and will not try to run bash instead04:57
elibrokeitor well, I suppose you can add Gentoo CI :)04:58
elibrokeitI think setting CONFIG_SHELL is easier though04:58
timburkei like both options :-) i've mainly messed around with ubuntu or centos for my CI jobs, but especially for a non-python project, more targets seems like a good idea04:59
elibrokeitportage in particular uses this: https://github.com/gentoo/portage/blob/c82dc8c8fe92979b82df4c71bb9961973367e8f9/lib/portage/package/ebuild/doebuild.py#L224705:01
elibrokeitit will grep build logs and scan for things that are usually a suspicious sign of something wonky05:02
elibrokeitin particular: r"(.*): line (\d*): (.*): command not found$"05:02
elibrokeitr"(.*): (\d+): (.*): not found$"05:02
opendevreviewMerged openstack/project-config master: Fix wheel_volume values for centos stream wheel mirrors  https://review.opendev.org/c/openstack/project-config/+/90715005:05
JayFIsn't a base level answer just to see if you can get your thing to build an environment with no bash whatsoever?05:15
JayFIf I were approaching a CI job with this in mind, I would probably try to build inside a container that explicitly had no bash installed.05:16
timburke🎉 it's a start! https://zuul.opendev.org/t/openstack/build/978a763d413944bf9efd949165515e5d/log/job-output.txt#1942-194905:39
opendevreviewDr. Jens Harbott proposed opendev/git-review master: Add CC similarly to reviewers  https://review.opendev.org/c/opendev/git-review/+/84921907:58
jrossercould i get a held node on job openstack-ansible-deploy-aio_magnum_octavia_capi-ubuntu-jammy / 90519908:56
fricklerjrosser: sorry, but that job name is too long, can't get a grip on that. just kidding, on it ;)09:06
jrosserthankyou :)09:06
jrosserfwiw the name string is parsed and determines what runs in the job09:07
opendevreviewJan Marchel proposed openstack/project-config master: Add new components to NebulOuS project: prediction-orchiestrator, exn-middleware, overlay-network-agent  https://review.opendev.org/c/openstack/project-config/+/90706009:54
opendevreviewJan Marchel proposed openstack/project-config master: Add new components to NebulOuS project: prediction-orchiestrator, exn-middleware, overlay-network-agent  https://review.opendev.org/c/openstack/project-config/+/90706010:12
ildikovHi All, I have a quick question. If there's someone who has two OpenInfra IDs, is there a way to merge them or remove the one they don't use anymore?13:15
fungiildikov: we don't manage openinfraid, nor is it currently used to log into any of the systems we manage or host really (aside from zanata and refstack). is it maybe launchpad/ubuntuone sso openids they have two of?13:35
*** ykarel__ is now known as ykarel14:04
ildikovfungi: hmm, I might've mixed it up with the OpenInfra Foundation profile14:08
ildikovfungi: I meant the data that Bitergia is using as well14:09
fungiildikov: i see. it's probably a question for the openinfraid maintainers as to how to close down one of their ids there (if they do have multiple foundation profiles). if they contributed to a project with multiple gerrit accounts though, they ought to be able to just list the e-mail addresses for both on their foundation profile (i think it lets you include up to 3 addresses)14:11
fungimy understanding is that bitergia gets the preferred e-mail address associated with the gerrit account that owns each merged change, and then asks openinfraid (really the "summit api" at openstackid-resources.openstack.org) for the foundation profile associated with each address14:13
fungiildikov: so it's possible to have a many-to-one (well, up to three-to-one) relationship between multiple gerrit accounts and a single foundation profile14:23
fungiclarkb: i think i see where testinfra's is_listening is getting tripped up. the address getting returned from the listening sockets list is ::ffff: rather than
fungithere's a (very old) open issue about it: https://github.com/pytest-dev/pytest-testinfra/issues/28614:42
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0  https://review.opendev.org/c/opendev/system-config/+/90714114:46
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail keycloak testing for an autohold  https://review.opendev.org/c/opendev/system-config/+/90660014:46
fungirestored the is_listening test with ::ffff: prefixed on the loopback address14:46
fungianother option might be to ask it to listen on ::1 instead? but i seem to remember that docker doesn't like ipv614:48
Clark[m]I don't think docker cares here because we are using host networking. Keycloak is what would matter for ipv6 support14:49
fungiah okay, so maybe we could use v6 loopback in that case14:49
fungimariadb is listening on both and :: according to ss, so should be reachable by either, though maybe we also want to limit it to the loopback for added safety14:51
fungihttps://mariadb.com/kb/en/server-system-variables/#bind_address is what we want to tweak for that, i suppose14:54
fungialso, is there a reason we should be sticking with old mariadb versions?14:54
clarkbno reason14:55
apevec openstack-stable-maint list owners are getting: Attempting to deliver following mail to recipient(s):    <mriedem@linux.ibm.com>14:56
apevecVMSDVMA.POK.IBM.COM unable to connect for 1 days to recipient host.14:56
fungiclarkb: 10.6 is the newest we're using anywhere, but looks like they have 11.2 tagged as "latest" at the moment14:56
fungiapevec: you could set that subscription to "no delivery" or unsubscribe them14:56
clarkbfungi: ya I think whatever is the current "stable" release is fine. There may be an env var to set for the container image to set the listen addr but I'm not seeing at first glance14:56
fungihttps://hub.docker.com/_/mariadb doesn't mention a "stable" tag14:57
apevecack, I wonder why listman is not doing that after bouncing continues, it is now every day afaict ...14:57
clarkbfungi: https://mariadb.com/kb/en/mariadb-server-release-dates/14:58
fungiapevec: mailman 3 uses separate address probes rather than relying on bounces, but also i think it doesn't disable them right away in case there's a temporary problem at the recipient's mailserver. how long has it been happening for that address?14:58
clarkb10.11 is the newest long term stable release series14:58
fungiclarkb: oh! i missed it, there's an "lts" tag14:59
fungiso should we use the 10.11 tag or the lts tag?15:00
apevecfungi: in my email, I see it starting Jan 19 but maybe I deleted older15:00
clarkbfungi: I would use the 10.11 series15:00
clarkbupgrades between mariadb releases are typically easy but not free aiui15:01
apevechttps://github.com/mriedem says still at IBM15:01
apevecbut maybe IBM killed @linux.ibm.com ?15:01
apevecMX is still there15:02
opendevreviewMerged openstack/diskimage-builder master: gentoo: don't uninstall packages that aren't installed  https://review.opendev.org/c/openstack/diskimage-builder/+/90423615:03
JayFI'll note the linked github profile has an alternate contact email address and a linkedin. Might be worth pausing the sub to the list and reaching out via a different method?15:03
fungithough also, i suspect matt r. has no interest in receiving a slew of periodic stable job failure e-mails daily to that or any address15:04
clarkbya I'm not sure we have to over think this15:05
clarkbmatt knows where to find us if necessary15:05
fungithe openstack-stable-maint is only used for job failure notifications15:05
fungier, the openstack-stable-maint mailing list i mean15:05
fungiclarkb: looking at https://hub.docker.com/_/mariadb i wonder if we should be setting MARIADB_AUTO_UPGRADE for future-proofing15:14
clarkbfungi: we can cross check against what ianw did to upgrade mariadb during one of our gerrit upgrades (its documented in the etherpad for that gerrit upgrade 3.5 or 3.6 maybe?). I think in that case whathappened was a manual run of the upgrade command then a start of the container15:16
clarkbbut ya maybe we set that flag then we can simply bump the version and monitor in the future15:16
fungiif we want to limit it to listening on ::1 we'll need to install a custom my.cnf by mounting it into the container (as you observed, there's no listed envvars i can find for setting bind-address)15:32
clarkbif we do that do we override all of the other settings in the process? That may be more trouble than it is worth. If we can "mix in" a my.cnf that would be better15:34
fungimaybe it supports run-parts type inclusion dirs or something15:35
clarkblooks like ubuntu one support is interpreting fungi's message on the lp issue for openid logins as implying the bug is in the library gerrit uses for openid15:35
clarkbI'm not sure of that. I think it more likely that either bits flipped somehow or there is a record keeping error either in ubuntu one or gerrit15:35
fungii guess i don't know enough about openid protocol to figure out if gerrit or ubuntuone sso chose the association handle15:36
clarkbfungi: ubuntu one chooses it15:36
clarkbconsumers make a post to the server saying "give us an association" the server responds with the hash material to use for verification15:36
fungiso i suppose it could still be that something internal to gerrit's openid plugin didn't switch to checking things with the new handle (though from the logs we see it's at least trying to use the new handle)15:37
clarkbit is possible that either side break things by improperly or inadverdently recording the information. My point is that i don't want us thinking the issue is in java or the java lib. We have insufficient data to indicate who may have had the problem15:37
clarkbif anything failures with wiki might point to the common problemy existing on the ubuntu one side. However, we could have independent issues in the different implementations on the consumer side as well. tl;dr who knows where the problem lies :)15:46
fungiso inside the mariadb container there's a /etc/mysql/my.cnf which ends in "!includedir /etc/mysql/conf.d/"15:47
fungiwe should be able to mount a custom config stub into it15:47
fungiah, yeah, we already add a custom mount like this in the mm3 compose file:15:52
fungi- /var/lib/mailman/99-max_allowed_packet.cnf:/etc/mysql/conf.d/99-max_allowed_packet.cnf:ro15:52
clarkboh ya for the large emails :)15:55
fungimmm, we restart the mailman containers with a notify/trigger when changing that config. should we do the same with keycloak's containers?16:04
clarkbit seems unlikely to change often so that should be safe and good belts and suspenders16:04
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0  https://review.opendev.org/c/opendev/system-config/+/90714116:22
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail keycloak testing for an autohold  https://review.opendev.org/c/opendev/system-config/+/90660016:22
clarkbtree style tabs randomly stopped working in firefox16:55
clarkbthis might legitimately make firefox unusable16:55
fungithe rust-vmm community decided to move their online meetings from a proprietary/commercial videoconferencing platform to meetpad: https://lists.opendev.org/archives/list/rust-vmm@lists.opendev.org/message/IV3BDGEYUILRMWZPRYHDEA777IKSC3U7/16:59
clarkbuninstalling, restarting firefox, then starting firefox again doesn't fix it. It was working with the same version of firefox previously so it must be breaking on some sort of local state. I wonder if clearing out tabs would help17:00
clarkblooks like they updated the plugin today17:07
clarkband it doesn't work17:07
clarkbhttps://github.com/piroor/treestyletab/issues/3440 for anyone else currently suffering17:09
jrosseris there caching of container images that i can use in my jobs, for example from registry.k8s.io and docker.io?17:13
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0  https://review.opendev.org/c/opendev/system-config/+/90714117:14
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail keycloak testing for an autohold  https://review.opendev.org/c/opendev/system-config/+/90660017:14
clarkbyes for docker and quay. I think there is a role that will autoconfigure them for you17:14
jrosseri would like to be able to point this to something https://docs.openstack.org/magnum/latest/user/index.html#container-infra-prefix17:15
jrosserbut a bit of codesearch suggests that no-one does that in magnum jobs currently17:15
clarkbjrosser: zuul/zuul-jobs/roles/use-docker-mirror17:16
clarkbthat integrates with the zuul registry as well for buildset caching of images17:17
jrosserthat is a little difficult for magnum, `container_infra_prefix` gets passed into the VM it creates17:18
clarkbya its the difference between test harness and workload, but it shoudl point you at what needs to be configured17:18
clarkbyou can also look at system-config/playbooks/roles/mirror/templates/mirror.vhost.j2 if you want to see the proxy cache config directly17:19
jrossercool, thanks17:19
clarkbok I fixed tree style tabs but it lost all my config17:19
clarkbwhcih is almost as bad as not having them in the first place (why adding new tabs to the top of the list is the default over appending to the end I'll never know)17:20
jrosserclarkb: ah i see why this does not get used with magnum, when you tell it where a local registry is it assumes that all the images you need are in that one place, losing all understanding of their upstream source17:28
clarkbthat seems to be a common problem with the container world17:29
clarkbdocker for example can only configure mirrors for docker hub17:29
clarkbotherwise you have ot point it at specific locations17:29
jrosseri beleive there is more flexibility with containerd17:30
clarkbyes podman and libcontainer and so on are better about it17:30
jrosserbut regardless, magnum does not expose that in its api17:30
clarkbhowever skopeo can't talk to docker right now because of protocol version negotiation so you win some and lsoe some17:30
clarkbfungi: looks like you have two sets of held keycloak nodes. Should we cleanup the older one?18:03
fungii thought i had, but i'm about to blow them away again for another revision anyway18:04
fungiclarkb: oh! i see what happened. i accidentally blew away tonyb's meetpad autohold. sorry tonyb! would you like me to reset it and recheck the change?18:06
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0  https://review.opendev.org/c/opendev/system-config/+/90714118:11
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail keycloak testing for an autohold  https://review.opendev.org/c/opendev/system-config/+/90660018:11
clarkblooks like you found the two things I just posted comments about.18:13
clarkbfungi: if ::1 works in the wait_for I think that is a problem since it implies that java is only listening on ipv6 which I guess is ok for local proxying but you may also need to update apache?18:13
fungiyeah, i'm curious to see18:14
fungiwe could switch them to all we just need to make sure testinfra's is_listening knows the socket is tcp://::ffff: in that case18:15
fungialso mariadb's bind-address can take a list of addresses as of 10.1118:16
clarkbright. I guess :: is the one that also accepts ipv4 connections but ::1 is a specific singular address?18:16
JayF:: is the v6 equivalent to
JayFlisten on all18:16
fungiwell, for mariadb 10.11 :: no longer listens on all addresses, only ipv6 addresses, and you'd need to make it ::, instead18:16
JayF::1 is loopback, which is like
clarkbJayF: yes but it also includes all ipv6 addrs18:17
clarkber ipv4 addrs18:17
clarkbfungi: wow ok18:17
fungiJayF: depends on the socket implementation as to whether it binds to multiple address families18:17
JayFThat is standard for dual-stacked apps, they likely represent the IPs as :: in the logs, right?18:17
JayFyes, ::ffff:
fungi::ffff: is a v6-mapped v4 address18:18
JayFYeah, and I don't think it's wrong to say that most apps just expose the single socket now, yeah?18:18
JayFOr is my ops knowledge crusty and more new things do v4/v6 listens separately?18:18
fungimost apps do, but it's a sockopt18:18
clarkbI think using ::1 or is fine we just need to be consistent across the board and testing should mostly cover that (except we don't have tests for the ssl terimination proxy?)18:18
fungialso, "most apps do" *on linux* (bsd's default is the opposite)18:19
clarkbno wonder ipv6 adoptions is terrible18:19
JayFfungi: ah, a useful distinction18:19
clarkbif you have to explicitly opt into ipv6 when configured to listen on all addresses thats a whole set of things that will never update18:19
JayFAnd in many networks, v6 terminates at the edge18:20
fungibut yeah, for mariadb 10.11 and later if you want it listening on both v4 and v6 loopback you need to tell it ::1, so it opens both18:20
JayFwhich is like giving yourself all the pain of the new with none of the benefits :( whereby most v4 networks have roughly a similar shape18:20
clarkbfungi: I think as long as the local my.cnf is set properly then it won't matter too much. so docker exec -it mariadb mysql -p$PASSWORD18:21
fungianyway, we do have testing of this so it should tell us. and yes we do test the ssl termination, the testinfra-based api test is connecting to apache over https, not directly to the keycloak container18:22
clarkbfungi: 8080 is not apache though?18:22
fungibut 443 is18:22
clarkboh the api tests hit apache cool18:22
fungiand the api tests connect to 44318:23
fungiover https18:23
fungii did at least test that they work with the [::1]:443 mapping syntax18:23
clarkbcool then ya I think it doesn't matter which we choose as testing will force us to be consistent18:25
fungithat's my hope, yes18:25
fungilooks like the keycloak api is returning a "503 Service Unavailable" in the latest iteration. i bet that's your apache canary clarkb!18:53
fungi"localhost" in the vhost config likely means
fungiyep, /etc/hosts has " localhost" and nothing for ::118:54
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0  https://review.opendev.org/c/opendev/system-config/+/90714118:57
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail keycloak testing for an autohold  https://review.opendev.org/c/opendev/system-config/+/90660018:57
fungithat revision ^ worked on the held node18:57
tonybclarkb: No it's fine.19:32
clarkbI think that was for fungi 19:32
fungilooks like that last one succeeded, but now i'm going to do some more cleanup on it while i'm thinking about it, to make implementing the replacement server easier19:32
fungitonyb: yeah, i figured the held nodes were what led to the latest series of meetpad changes19:33
fungistill, sorry about that19:33
fungii'll be more careful in the future19:33
fungii should make sure to always grep the autohold list for my own nick to avoid similar accidents19:34
opendevreviewJeremy Stanley proposed opendev/system-config master: Upgrade to Keycloak 23.0  https://review.opendev.org/c/opendev/system-config/+/90714119:35
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail keycloak testing for an autohold  https://review.opendev.org/c/opendev/system-config/+/90660019:35
tonybclarkb: Yeah you're right that was meant for fungi.  Either way the hold is no longer needed19:39
fungiwell, cool, i cleaned it up for you in that case. yeah, that's my story ;)19:40
fungigiven the keycloak server "upgrade" idea is wholesale replacement and reconfiguration, what do folks think about this quick-but-ugly process: 1. put keycloak01 in the emergency disable list, 2. merge 907141, 3. boot a keycloak02, 4. add to inventory and dns switching the keycloak cname at the same time, 5. log in as generic service admin and create the zuul realm and accounts, 6. let our19:43
fungisysadmins set their credentials and associate any identities again19:43
fungii can shoot for doing all that within a day, so that anyone who does use it with some regularity doesn't have to go without for too long19:44
tonybI'm fine with that, I'm a very light user.19:46
fungiit would of course be possible to have a more atomic cut-over, or even to try to export/import our configuration, but all of those also imply more research, testing and hand-holding which add far more effort to the task, so i'm trying to be pragmatic19:48
fungiif we were already using it for something like gerrit logins, the cost/benefit analysis might tip the other direction19:49
* tonyb steps away for a couple of hours20:00
fungi23.253.164.201 is the new held keycloak99 test node if anyone wants to poke at it20:12
fungithe only other thing i'm tempted to work in is that keycloak 24.0.0 is due out in a week or so. we might want to consider targeting that version instead so we don't immediately start out a major version behind20:15
fungithere's a "nightly" container tag which probably comes close to what will end up in the official 24.0 images20:17
clarkbthat seems reasonable but also I think upgrades are supposed to be striaghtforward20:23
clarkbit is just the runtime framework swap that really broke things there20:23
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Try future Keycloak 24.0  https://review.opendev.org/c/opendev/system-config/+/90725320:24
fungihttps://github.com/keycloak/keycloak/milestone/35 technically says "Due by February 29, 2024 ... 62% complete" so maybe we just plan to upgrade, but still it'll be good to see what things are looking like there20:26
fungilooks like fido passkeys are supposed to work for v24 too20:28
fungiin case anybody cares about that20:29
fungior maybe it'll get bumped to keycloak 25, it's still waiting for https://github.com/keycloak/keycloak/pull/24305 to be revised20:33
fungiah, there's basic support for them already in v23, just not for that bit20:34
fungithey also added a high availability guide to their documentation recently, though i expect that would need an external db clustering approach similar to what we discussed for zuul20:38
fungiswapping 23.0 for nightly also passed our tests20:49
fungiso if nothing else, it probably means the upgrade won't be too onerous (basic service configuration remains functional)20:50
fungiinfra-root: rough plan outline for keycloak upgrade/replacement is https://etherpad.opendev.org/p/keycloak-refresh-202421:06
fungianything obvious missing?21:07
fungiif we do steps 8 and 9 close together we can minimize the downtime as much as possible without creating extra work21:09
fungithough now that i'm looking back over it, 6 and 7 might need local name resolution overrides on people's machines21:10
fungibut not a huge complication21:10
fungialso, anyone who doesn't care about it can personally chose to do step 7 to after step 1121:11
fungier, after step 9 i mean21:11
corvushaving multiple systems that would benefit from an ha dbms makes the idea of running a single large pxc cluster to server zuul + gerrit + keycloak more attractive22:35
clarkbnote there isn't much benefit to gerrit for that after the notedb transition23:09
clarkbthe sql db in gerrit only tracks the "reviewed" flag next to files23:09
clarkband I think gerrit even functions if the sql db does not. You just don't get that data and might get some errors you can ignore23:09
clarkbfungi: the keycloak etherpad lgtm23:22
fungithanks. if there's interest from enough folks in reviewing 907141 soon, then i can possibly get us up to step 6 there this week or early next23:27
fungidepending on how the review shakes out23:27
fungiand also prep/stage the remaining changes for the cut-over window23:28
clarkbI'll rereview it when I'm done looking at this nodepool change23:28

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!