Monday, 2024-03-11

opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix coordination when redis used  https://review.opendev.org/c/openstack/kolla-ansible/+/91234106:32
kevkofrickler, mnasiadka: Takashi reverting stuff around auth in tooz ...06:42
mnasiadkakevko: saw the mail, thanks ;)07:00
kevkomnasiadka: I still don't understand why Takashi wants to revert auth stuff07:06
kevkomnasiadka: as per his last email and my answer ...07:07
mnasiadkakevko: because maybe there are other people using it as we have been using ;-)07:07
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix creation of ovs bridges  https://review.opendev.org/c/openstack/kolla-ansible/+/91159107:08
kevkomnasiadka: yeah - maybe .... question is if it will work also after revert ..07:08
mnasiadkahaha07:09
kevkoI really hate all those machinations with code, reverts, release exceptions, and the like.... I just want to work and  don't want to fix fix other things at night. :D 07:12
* frickler is glad that we have a CI that tells us about these issues. much better than to find out in production07:17
kevko+107:20
kevkowell, i found it firstly in my downstream CI :) 07:25
kevkoWhy do I feel like everyone is releasing new versions and everything breaks, especially during RC1 and similar important time periods? :D07:27
fricklerwell that's what rc1 is for. one could argue whether projects would need to do more cross testing before, but then issues like CI bloat and instability come up07:36
kevkomnasiadka: frickler: so, does it mean that we are frozen for now - nothing can be merged until tooz,taskflow will be fixed/released and kolla images built and available in quay ? OR we can merge stuff as we have chain against which CI works and we can for example merge kolla-ansible stuff ..07:37
fricklerwell currently I don't see a chain of patches that passes CI completely. we also need to decide whether to cap or unpin tooz07:39
kevkofrickler: this is passing https://review.opendev.org/c/openstack/kolla-ansible/+/912341 07:44
kevkofrickler: for example this can be merged for sure ...as it is k-a https://review.opendev.org/c/openstack/kolla-ansible/+/911591/10 ..and passing 07:44
fricklerkevko: but depends-on is failing07:44
kevkofrickler: it was some type of timeout if i remember correct ..recheck will work i woud say 07:46
kevkowhich is currectly in progress07:46
opendevreviewIvan Halomi proposed openstack/ansible-collection-kolla master: Bump min. version of docker-py  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/91075108:22
mnasiadkafrickler: should we cap for now? I can restore the patch in Kolla08:37
mnasiadkahttps://review.opendev.org/c/openstack/kolla/+/911703?tab=change-view-tab-header-zuul-results-summary08:38
mnasiadkawell, basically we can just merge it now08:38
mnasiadkaand revert once things are safe08:38
fricklermnasiadka: those octavia failures worry me, but they seem to be happening for > 2 weeks already, so likely unrelated. so +2 to capping and waiting for tooz to stabilize08:42
mnasiadkaI'll have a look in all failing jobs08:42
mnasiadkaubuntu-swift is failing due to some Ubuntu kernel bug, the same thing happens in Kayobe08:42
fricklerkevko: can you depend your stack on ^^ instead?08:44
kevkofrickler: yes of course, let me do it 08:45
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Implement TLS for Redis  https://review.opendev.org/c/openstack/kolla-ansible/+/90918808:45
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix coordination when redis used  https://review.opendev.org/c/openstack/kolla-ansible/+/91234108:50
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix creation of ovs bridges  https://review.opendev.org/c/openstack/kolla-ansible/+/91159108:50
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: magnum: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91235808:55
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: octavia: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91235909:07
mnasiadkafrickler: ^^ that should fix octavia, let's see09:07
mnasiadkawill continue with other services09:07
mnasiadkafrickler: should we drop Murano?09:10
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: octavia: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91235909:12
opendevreviewRoman Krček proposed openstack/kolla-ansible master: Split ipv4 and ipv6 systemctl config  https://review.opendev.org/c/openstack/kolla-ansible/+/90583109:16
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: heat: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236009:16
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: manila: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236109:16
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: neutron: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236209:16
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: zun: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236409:20
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: designate: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236509:20
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kuryr: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236609:20
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: cyborg: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236709:20
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: murano: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236809:20
mnasiadkahmm, why is https://review.opendev.org/c/openstack/kolla/+/911703 not gating?09:24
mnasiadkafrickler: another gerrit bug? ;-)09:24
mnasiadkaeven recheck doesn't work09:26
opendevreviewMichal Nasiadka proposed openstack/kolla master: openstack-base: Pin tooz to 5.0.0  https://review.opendev.org/c/openstack/kolla/+/91170309:28
kevkomnasiadka: depends on 09:30
opendevreviewMichal Nasiadka proposed openstack/kolla master: openstack-base: Pin tooz to 5.0.0  https://review.opendev.org/c/openstack/kolla/+/91170309:30
mnasiadkaholy crap09:30
mnasiadkait would be easier if zuul would send a comment that depends-on is abandoned, not doing shit09:30
mnasiadka:)09:30
kevko:D 09:31
opendevreviewMichal Nasiadka proposed openstack/kolla master: Fix aarch64 builds  https://review.opendev.org/c/openstack/kolla/+/90991509:33
opendevreviewMichal Nasiadka proposed openstack/kolla master: Fix aarch64 builds  https://review.opendev.org/c/openstack/kolla/+/90991509:34
mnasiadkathat one also needs to get in :)09:35
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: magnum: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91235809:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: octavia: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91235909:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: heat: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236009:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: manila: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236109:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: neutron: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236209:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: zun: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236409:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: designate: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236509:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: kuryr: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236609:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: cyborg: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236709:38
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: murano: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91236809:38
opendevreviewIvan Halomi proposed openstack/ansible-collection-kolla master: Bump min. version of docker-py  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/91075109:38
mnasiadkaupdated depends-on to point to tooz pin09:38
opendevreviewSven Kieske proposed openstack/kolla-ansible stable/2023.2: Keystone: Add service role  https://review.opendev.org/c/openstack/kolla-ansible/+/91206609:43
SvenKieskemnasiadka: seems we forgot to add the service role to the 2023.2 branch? at least I can't find it so far?09:44
mnasiadkaSvenKieske: because keystone started to create that role in bootstrap since 2023.209:45
SvenKieskeI'm also wondering why we didn't put the original patchset on master branch to begin with09:45
mnasiadkahttps://review.opendev.org/c/openstack/keystone/+/86342009:45
mnasiadkaso that's why it's not needed there09:45
SvenKieskemhm, weird, then it seems bootstrap isn't run on every upgrade somehow09:46
mnasiadkaif it would be the case - then the cinder/nova service role patch would fail in CI09:48
mnasiadkaand it passed nicely - https://review.opendev.org/c/openstack/kolla-ansible/+/909542?tab=change-view-tab-header-zuul-results-summary09:49
SvenKieskeseems correct as far as I can see. weird, I wonder what is causing this issue downstream then.10:03
kevkoSvenKieske: what downstream issue ...role cannot be found ? or what ? 10:05
SvenKieskeyeah, some users don't have that role, as I stated on friday when I began the investigation, in the meantime it seems we merged https://github.com/osism/container-image-kolla-ansible/pull/593/files but, I don't understand why we need it downstream on 2023.2 when upstream doesn't have it on that branch. either we are not seeing the issue upstream because we don't look hard enough or there is some other weird 10:10
SvenKieskeinteraction.10:10
SvenKieskelong lines and IRC will never be friends it seems :)10:11
mnasiadkayou keep patches as files instead of having a proper fork? brave ;-)10:11
SvenKieskeI don't know the history of that, I could imagine it's intentional to not have patching downstream being the happy path, we have very few patches actually10:15
SvenKieskethat's also why I think about this problem so much, it doesn't seem we have any downstream patch that would explain this issue.10:16
SvenKieskemnasiadka: when I grep the upgrade job (https://zuul.opendev.org/t/openstack/build/57aaffa9822e4c8f8cda493459748e98) it doesn't say that it creates the service role10:31
opendevreviewRoman Krček proposed openstack/kolla-ansible master: Fix trove module imports  https://review.opendev.org/c/openstack/kolla-ansible/+/91237810:32
SvenKieskehttps://github.com/openstack/keystone/blob/stable/2023.2/keystone/cmd/bootstrap.py#L130 is clearly being called (in the keystone bootstrap code with the service role, but the keystone log doesn't contain that. mhm10:33
mnasiadkaSvenKieske: and are you sure keystone bootstrap cli says that it creates the role on non-upgrade jobs?10:33
SvenKieskethat's what I wanted to look into now :D10:33
mnasiadkaSvenKieske: I don't think we even log the bootstrap contents anywhere - https://github.com/openstack/kolla/blob/master/docker/keystone/keystone/keystone_bootstrap.sh10:35
SvenKieskeyes it does: https://zuul.opendev.org/t/openstack/build/848d8f605b7449a38efb65f08f530a62/log/primary/logs/kolla/keystone/keystone.txt#7010:35
SvenKieskebut here it doesn't: https://zuul.opendev.org/t/openstack/build/57aaffa9822e4c8f8cda493459748e98/log/primary/logs/kolla/keystone/keystone.txt#7410:38
mnasiadkaso then it needs to work :)10:38
mnasiadkalol10:38
SvenKieskeyeah :)10:39
mnasiadkathen maybe we just need to add that in upgrade task for 2023.2, just to be sure10:39
SvenKieskeso we are thus again at my backport above :D10:39
mnasiadkabut if you run deploy with fresh 2023.1 code10:39
mnasiadkathen you'll have that role anyway10:39
SvenKieskeyeah, this is mostly a concern for older deployments I guess10:39
mnasiadkait's not backport10:39
SvenKieskeI still don't understand _why_ though :D10:39
mnasiadkait's forwardport10:40
SvenKieskeyeah10:40
SvenKieskelet me fix the merge conflict10:40
mnasiadkaand I think it should be a separate patch, because we only need to do it on upgrade10:40
mnasiadkanot a normal deploy10:40
SvenKieskeyes10:41
mnasiadkawell, in upgrade we do this: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/keystone/tasks/upgrade.yml - actually this: https://github.com/openstack/kolla-ansible/blob/5169e3bcbe556eadbe60644f7acbed6e5d6ed358/ansible/roles/keystone/handlers/main.yml#L15 - so we run keystone_manage db_sync (as per 10:42
mnasiadkahttps://github.com/openstack/kolla/blob/4b32dd50ab027e2b19c7c69f7fc5019ec3d1add7/docker/keystone/keystone/extend_start.sh#L24)10:42
mnasiadkaso we don't run bootstrap on upgrade10:42
mnasiadkaand that explains it10:42
SvenKieskeI guess I saw that, but we run service-ks-register which in theory should also create roles? I found the code split there a little confusing tbh, some roles are only created via keystone bootstrap, other's are created via the service-ks-register role10:44
mnasiadkabut we don't create those roles in 2023.2 - because we assume those are there10:44
SvenKieskeI get that we want to create user supplied roles as well, but we should maybe move the service role from service-ks-register to keystone bootstrap? not sure though.10:45
SvenKieskebecause the service role is imho a system role, not something that should be configurable by users10:45
mnasiadkawell, in register.yml we run bootstrap, but it's not called on upgrade :)10:46
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts  https://review.opendev.org/c/openstack/kolla-ansible/+/91141710:48
SvenKieskeyeah I already saw that10:49
SvenKieskenot sure if we should fix this via the extend_start.sh or via register.yml? it seems weird that all this stuff is spread over several files, I know there are many historical reasons why it is the case, but still :D10:50
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts  https://review.opendev.org/c/openstack/kolla-ansible/+/91141710:57
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Move actions to kolla_container_facts  https://review.opendev.org/c/openstack/kolla-ansible/+/91150510:59
mnasiadkaSvenKieske: I think we just need to downsize the patch you're ''forwardporting'' to add a section in upgrade.yml to call service-ks-register and create the bloody role10:59
SvenKieskemnasiadka: seems sensible, should I propose a new patch from scratch or just adjust this one?11:01
mnasiadkanew patch from scratch would be better I guess11:01
SvenKieskeokay, fine11:01
mnasiadkakevko, frickler: Can you have a look in https://review.opendev.org/c/openstack/kolla/+/909915? It fixes aarch64 builds that I broke with the sources.py patch :)11:34
kevkomnasiadka: done11:42
mnasiadkathx11:43
opendevreviewRoman Krček proposed openstack/kolla-ansible master: Fix trove module imports  https://review.opendev.org/c/openstack/kolla-ansible/+/91237811:54
opendevreviewRoman Krček proposed openstack/kolla-ansible master: Split ipv4 and ipv6 systemctl config  https://review.opendev.org/c/openstack/kolla-ansible/+/90583111:56
opendevreviewAlex Welsh proposed openstack/kolla-ansible master: WIP: Automate prometheus blackbox endpoint config  https://review.opendev.org/c/openstack/kolla-ansible/+/91242011:57
opendevreviewAlex Welsh proposed openstack/kolla-ansible master: WIP: Automate prometheus blackbox endpoint config  https://review.opendev.org/c/openstack/kolla-ansible/+/91242011:59
opendevreviewMichal Nasiadka proposed openstack/kolla master: Bump rabbitmq to 3.13  https://review.opendev.org/c/openstack/kolla/+/91109312:31
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: octavia: Use valid_interfaces instead of endpoint_type  https://review.opendev.org/c/openstack/kolla-ansible/+/91235912:49
mnasiadkaok, tooz pin is gating, we should be good tomorrow12:50
mnasiadka(unless it fails in gate)12:50
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts  https://review.opendev.org/c/openstack/kolla-ansible/+/91141712:54
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Move actions to kolla_container_facts  https://review.opendev.org/c/openstack/kolla-ansible/+/91150512:55
opendevreviewMerged openstack/kolla master: Fix aarch64 builds  https://review.opendev.org/c/openstack/kolla/+/90991513:22
opendevreviewSven Kieske proposed openstack/kolla-ansible stable/2023.2: fix creation of keystone service role on upgrade  https://review.opendev.org/c/openstack/kolla-ansible/+/91245213:29
opendevreviewSimon Dodsley proposed openstack/kolla-ansible master: Add Pure Storage FlashBlade as Manila backend  https://review.opendev.org/c/openstack/kolla-ansible/+/87984613:33
opendevreviewSven Kieske proposed openstack/kolla-ansible stable/2023.2: fix creation of keystone service role on upgrade  https://review.opendev.org/c/openstack/kolla-ansible/+/91245213:35
kevkomnasiadka: frickler: so can we merge also https://review.opendev.org/c/openstack/kolla-ansible/+/911591/11 both in chain ? 14:10
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts  https://review.opendev.org/c/openstack/kolla-ansible/+/91141714:21
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Move actions to kolla_container_facts  https://review.opendev.org/c/openstack/kolla-ansible/+/91150514:21
opendevreviewMerged openstack/kolla master: openstack-base: Pin tooz to 5.0.0  https://review.opendev.org/c/openstack/kolla/+/91170314:35
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Merge of container_facts modules  https://review.opendev.org/c/openstack/kolla-ansible/+/91246014:36
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Merge of container_facts modules  https://review.opendev.org/c/openstack/kolla-ansible/+/91246014:38
opendevreviewMichal Nasiadka proposed openstack/kolla master: Bump rabbitmq to 3.13  https://review.opendev.org/c/openstack/kolla/+/91109315:11
opendevreviewMerged openstack/kolla-ansible master: Fix coordination when redis used  https://review.opendev.org/c/openstack/kolla-ansible/+/91234115:55
SvenKieskecan this error be related to the recent tooz breakage? "TypeError: Redis.__init__() got multiple values for argument 'host'"15:57
SvenKieskeI got this in https://zuul.opendev.org/t/openstack/build/995f77b643d34228b5d41a2a69001c60/log/primary/logs/kolla/cinder/cinder-api-error.txt16:00
SvenKieskecould be related to https://review.opendev.org/c/openstack/tooz/+/911648/2/tooz/drivers/redis.py I guess16:01
kevkoSvenKieske: yes16:08
SvenKieskemhm, so I'm not sure if this tooz fix fixes our sentinel problem or made it worse? :D16:21
SvenKieskeah mnasiadka commented, thanks for the heads up! I was not aware the multinode jobs are still broken! I thought somehow the pinning was already done and effective16:22
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: rabbitmq: configuration improvements  https://review.opendev.org/c/openstack/kolla-ansible/+/90052816:23
kevkoSvenKieske: rebased ....16:23
kevkoSvenKieske: sorry ...i meant add depends-on16:23
SvenKieskekevko: thank you, but Michal commented that we need to wait for the new pinned tooz images which seem to be available only tomorrow?16:23
SvenKieskeat least for the multinode jobs16:23
kevkoSvenKieske: if you have depends-on, you will have a fresh images ..16:25
SvenKieskeah, I just reread the ML thread around tooz, I was not aware of the additional breakage in other projects, good to know16:28
opendevreviewMerged openstack/kolla-ansible master: Fix creation of ovs bridges  https://review.opendev.org/c/openstack/kolla-ansible/+/91159116:29
kevkowe have also compare container broken :D 16:51
kevkoaaaaa16:51
opendevreviewUwe Jäger proposed openstack/kolla-ansible master: Skyline configure Prometheus  https://review.opendev.org/c/openstack/kolla-ansible/+/91051417:09
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Move actions to kolla_container_facts  https://review.opendev.org/c/openstack/kolla-ansible/+/91150517:13
kevkomnasiadka: who removed docker_common_options option from kolla ? :( 17:28
kevkomnasiadka: it completly broke compare container :'( 17:29
kevkonot removed ..but now it's broken i would say 17:30
kevko:'( 17:42
kevkocompare_container functionality is just broken when environment changed 17:42
opendevreviewDoug Szumski proposed openstack/kolla-ansible master: Support custom Nova Compute Ironic host names  https://review.opendev.org/c/openstack/kolla-ansible/+/91248317:47
opendevreviewDoug Szumski proposed openstack/kolla-ansible master: Support custom Nova Compute Ironic host names  https://review.opendev.org/c/openstack/kolla-ansible/+/91248317:48
opendevreviewDoug Szumski proposed openstack/kolla-ansible master: Support custom Nova Compute Ironic host names  https://review.opendev.org/c/openstack/kolla-ansible/+/91248317:53
opendevreviewUwe Jäger proposed openstack/kolla-ansible master: Update Skyline configuration to enable SSO  https://review.opendev.org/c/openstack/kolla-ansible/+/90586018:07
kevkoWell, in the end, it's not, it's just me being silly :D :D 18:20
mnasiadkaLol18:22
SvenKieske:D18:23
opendevreviewMichal Nasiadka proposed openstack/kolla master: Revert "openstack-base: Pin tooz to 5.0.0"  https://review.opendev.org/c/openstack/kolla/+/91207018:24
mnasiadkaLeaving it here to not forget reverting :)18:24
kevkoi'm writing some code 18:27
kevkoand i've put environment: item.environment(omit)18:28
kevkoinstead of item.value.environment(omit)18:28
kevko:D :D :D 18:28
kevkoso omit did the work ..so no error ..nothing ...but also not expected data :D 18:28
kevkosh*t :D 18:28
SvenKieskeI guess we should have more prechecks to see if expected values are really there, this happens a lot imho :)18:29
kevkoSvenKieske: this was not runtime error ...this was my stupidity18:55

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!