Friday, 2023-01-13

ianwi can get the python3.11 to work by editing it's rpath in the builder container01:48
ianwi imagine a bug report along the lines of "i enable unstable on the buster container then pull in packages from there" will not be considered helpful01:48
ianwbullseye i mean01:49
*** ysandeep|out is now known as ysandeep02:17
Clark[m]Though that comment indicates this should be fine so maybe they would accept it?02:46
ianwyeah i'm trying to replicate it in a less obscure way02:47
ianwok, i believe is actionable03:44
*** ysandeep is now known as ysandeep|afk05:08
*** pojadhav- is now known as pojadhav06:12
*** arxcruz is now known as arxcruz|ruck07:54
opendevreviewCedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured
*** jpena|off is now known as jpena08:39
*** soniya29|rover is now known as soniya29|rover|lunch09:18
*** elodilles is now known as elodilles_pto09:58
*** soniya is now known as soniya29|rover10:20
*** rlandy|out is now known as rlandy11:14
*** soniya29|rover is now known as soniya29|rover|brb11:26
*** soniya29|rover|brb is now known as soniya29|rover11:54
mnasiadkaHas something changed in zuul/nodepool? Some builds of Kolla jobs have nodepool.private_ipv4 as an empty string? (And we're using it for our multi-node-vxlan-overlay role)12:18
fungimnasiadka: we do constantly upgrade zuul and nodepool from the tip of their master branches, so many somethings are always changing in zuul and nodepool. as to whether that's the reason for the behavior change you've observed, no idea. would need to go digging in recent commits12:41
mnasiadkain theory docs say private_ipv4 is optional12:42
mnasiadkabut jobs retrying in pre and failing are not something that is fun, and we haven't seen that surely in the previous days12:43
fungimnasiadka: my first guess would be "Convert openstack driver to statemachine" which merged yesterday12:44
fungibut yes, if private_ipv4 isn't guaranteed to exist, you may want to use a public address fallback12:45
mnasiadkaseems so12:45
fungicertainly not all of our providers supply a separate private ipv4 address, i wonder if the former behavior in the openstack driver was to fill it with the public address12:45
mnasiadkaprobably, let me fix that on our side, thanks for help12:46
fungiyep, the old driver had this...12:46
fungiif not self.node.private_ipv4: self.node.private_ipv4 = server.public_v412:46
fungiskimming the new driver code, i'm not seeing it do that anywhere12:47
fungigiven the documentation says there may not be a value for private_ipv4 i don't know whether this would be treated as a regression, but i'll bring it up in matrix12:48
*** ysandeep|afk is now known as ysandeep13:20
*** dasm|off is now known as dasm14:34
fricklerdon't have time to look deeper today, but devstack multinode might be seeing a similar issue
*** ysandeep is now known as ysandeep|out14:42
*** tosky_ is now known as tosky14:47
fungiit's up for discussion in the zuul matrix channel at the moment whether nodepool should make the old openstack behavior consistent across all the drivers, or add a release note14:49
fungibut regardless, we should probably just make our job definitions more robust, since we were relying on something that the documentation said wasn't reliable14:50
opendevreviewCedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured
fungifrickler: interestingly, it looks like the multi-node-bridge role in zuul-jobs does apply a fallback to public_ipv4 when private_ipv4 is not defined14:55
corvusi think we're going to just match the old behavior for now, then figure out what we want to do in the future and announce a transition period.  we should be able to restart the launchers in about an hour if we don't hit any job failures.15:00
fungicodesearch indicates that openstack/devstack and openstack/grenade do indeed use private_ipv4 exclusively for setting envvars in localrc, also openstack/openstack-zuul-jobs and openstack/tacker and openstack/kolla-ansible and openstack/skyline-console and openstack/octavia-tempest-plugin have some custom jobs that may assume it's always set, as well as the ovs_vxlan_bridge_peers role in15:01
fungiopenstack/devstack-gate, also a couple of playbooks in vexxhost/openstack-operator15:01
fungiso yeah, i have a feeling this is more widespread than we first realized15:02
fungimnasiadka: ^ just an update in case you're not glued to this channel, we've approved a change to nodepool to make the new driver backwards compatible with that behavior for now, and will come up with a transition plan to allow projects time to update their configurations without further disruption15:05
fungionce 870100 merges we can deploy updated container images with the fix and restart the launchers, so new nodes after that point will all have a private_ipv415:08
opendevreviewCedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured
*** soniya29|rover is now known as soniya29|rover|out15:43
opendevreviewClark Boylan proposed opendev/jeepyb master: Raise an error if acl pushes fail
clarkbthats a typo fix in the commit message but should also retrigger failing jobs16:23
*** marios is now known as marios|out16:39
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.7 images
opendevreviewClark Boylan proposed opendev/system-config master: Convert Verified MaxWithBlock to submit-requirement in testing
clarkbThat change illustrates the next sort of mass gerrit refs/meta/config update we'll need to do after the copycondition update16:44
clarkbI've also just asked on Gerrit discord about some confusing submit-requirement config documentation16:53
opendevreviewMerged opendev/system-config master: Override default branch name
clarkboh heh I messaged the slack room17:26
mnasiadkafungi: good to know, but I'll backport the already merged change to stable branches to be ready ;-)17:34
*** jpena is now known as jpena|off17:36
corvusi'm going to force-merge the nodepool change; it has hit its own bug 3 times in different jobs based on random provider allocation17:37
clarkbinfra-root the stack at is enough to bootsrap 3.7 image builds17:42
clarkbI haven't started on the upgrade job as I half expect that to be mpre involved. But I guess I should start that next17:43
corvusi will pull and restart launchers now17:47
corvusrestarted nl0117:49
corvus#status log restarted all nodepool launchers to pick up private_ipv4 bugfix17:51
opendevstatuscorvus: finished logging17:51
corvusfor safety, let's say as a rule of thumb that jobs started after 1800 utc should not expect to see the issue any more.17:52
clarkbcorvus: sounds good17:53
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.6 -> 3.7 upgrade test job
fungithanks corvus 17:59
fungii stepped away for lunch and missed all the excitement17:59
clarkbit looks like our zuul resuts plugin continues to work with the new js library. I'm slightly surprised by this. Our theming does too18:13
clarkbTwo fewer things to worry about pre upgrade though I suspect that both will need updates eventually due to the js framework update18:13
clarkbarg looks like compiling protobuf for gerrit 3.6 builds is broken right now18:19
clarkband this impacts my jeepyb change18:19
opendevreviewClark Boylan proposed opendev/system-config master: Fix Gerrit 3.6 image build
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.6 -> 3.7 upgrade test job
opendevreviewClark Boylan proposed opendev/system-config master: Add Gerrit 3.6 -> 3.7 upgrade test job
*** dasm is now known as dasm|off23:37

Generated by 2.17.3 by Marius Gedminas - find it at!