| stan | Guys, I need some suggestions regarding Glance API hosts. Currently kolla deploys Glance API container to a single controller host if the user chooses the file backend. But it is possible that users may want to mount a shared nfs path on all their controllers, and would want a ha setup. Are we open to implementing this in kolla? If so I can give it a shot and do a patch. But before doing this I would want your opinions. | 03:33 |
|---|---|---|
| stan | SvenKieske Thoughts? | 03:33 |
| opendevreview | Antony Messerli proposed openstack/kolla-ansible master: Updates docs to fix incorrect container example https://review.opendev.org/c/openstack/kolla-ansible/+/922278 | 03:37 |
| opendevreview | Olivier Delhomme proposed openstack/kolla-ansible master: kolla-ansible now uses default inventory file https://review.opendev.org/c/openstack/kolla-ansible/+/913993 | 06:20 |
| SvenKieske | stan: let me think about it for a moment | 06:38 |
| SvenKieske | stan: I don't know much about the file backend. There seems to be little documentation about it with regards to HA scenarios like you describe. I would check first with glance if they support this kind of stuff. Any reason not to use any of the other HA backends like e.g. ceph? | 06:44 |
| SvenKieske | there is at least some IBM docs about this mentioning NFS, so it might actually work: https://www.ibm.com/docs/fr/cmwo/4.3.0.0?topic=environment-configuring-image-storage-ha-controller-nodes#taskd170021e42 | 06:46 |
| SvenKieske | mhm, do we have a list somewhere which kolla releases are SLURP? I need to bookmark that :D | 06:49 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible stable/2024.1: kolla_url: port is a string https://review.opendev.org/c/openstack/kolla-ansible/+/922285 | 06:50 |
| stan | SvenKieske In our lab, we don't have a ceph setup, it's all netapp. I would want to compare the NFS vs S3 as a glance store backed by netapp flex volume. In non ceph env, this might come in handy I suppose. | 07:13 |
| SvenKieske | if you contribute a patch I guess we would need tests for that as well | 07:14 |
| stan | Also how is ceph's performance on a decent sized cluster, say 5PB | 07:14 |
| SvenKieske | for image store more than sufficient I would say. vast majority of deployments are ceph(only) I would say > 80/90% | 07:15 |
| stan | SvenKieske Agreed about the test cases. | 07:15 |
| stan | Ceph seems to have enhanced instance spin ups by doing COW, if its performance is comparable to our netapp systems, I would like to explore it more. Waiting for hardware to arrive.. :) | 07:17 |
| SvenKieske | the netapps I've personally seen/administrated where shit compared to ceph, but ymmv :) also for prod, not just test | 07:27 |
| SvenKieske | ceph is also no free lunch and you need way more knowledge about storage and linux, but you have also more control | 07:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.2: kolla_url: port is a string https://review.opendev.org/c/openstack/kolla-ansible/+/922287 | 07:42 |
| SvenKieske | ah thx mnasiadka: I noticed the backport doesn't apply cleanly because of the base.yml I think. It's missing the haproxy-base job | 07:42 |
| mnasiadka | SvenKieske: yup, I guess we should backport the job as well | 07:43 |
| mnasiadka | let me do that | 07:43 |
| SvenKieske | fine, I was not totally sure if the backport works and then had a meeting before I could investigate deeper | 07:43 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.2: Test haproxy single external frontend https://review.opendev.org/c/openstack/kolla-ansible/+/922288 | 07:44 |
| SvenKieske | If someone has opinions on this, that would also be nice, the rmq startuptime is rather large because of a single env parameter, there are some solutions proposed, I can also put it on the whiteboard if nobody has time right now: https://review.opendev.org/c/openstack/kolla-ansible/+/921381/comment/56b11f74_f98925e8/ | 07:45 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.1: haproxy: support single external frontend https://review.opendev.org/c/openstack/kolla-ansible/+/922289 | 07:45 |
| SvenKieske | I added it to the whiteboard for completeness sake. | 07:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.2: kolla_url: port is a string https://review.opendev.org/c/openstack/kolla-ansible/+/922287 | 08:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2023.2: kolla_url: port is a string https://review.opendev.org/c/openstack/kolla-ansible/+/922287 | 08:09 |
| mnasiadka | SvenKieske: should be good now | 08:09 |
| PrzemekK | Is this kolla_url: port is a string is something crucial to implement right away? | 08:36 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Convert destroy scripts into playbooks https://review.opendev.org/c/openstack/kolla-ansible/+/920714 | 08:51 |
| sylvr | Hello! I'm about to deploy a cloud using kayobe and I have some questions. In the documentation https://docs.openstack.org/kayobe/latest/configuration/reference/network.html it says that control hosts are in the storage network by default, is it a requirement or is it ok if my control hosts are in the management network and the external network only ? (using kolla_vip to access the web-based control pane) | 08:53 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 08:55 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 08:56 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 09:09 |
| stan | I have a powerful controller 96 core and about 500gigs of ecc ram, I would like to know for what services would benefit from increasing worker count the most.. | 09:39 |
| PrzemekK | How to set neutron_ovn_availability_zones per GW server ? | 10:01 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fix fluentd log check https://review.opendev.org/c/openstack/kolla-ansible/+/918283 | 10:11 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Revert "[release] Use OpenStack 2024.1 release" https://review.opendev.org/c/openstack/kayobe/+/922300 | 10:22 |
| opendevreview | Andrew Babbitt proposed openstack/kolla-ansible master: Skyline: Fix incorrect keystone port https://review.opendev.org/c/openstack/kolla-ansible/+/922302 | 10:27 |
| opendevreview | Andrew Babbitt proposed openstack/kolla-ansible stable/2024.1: Skyline: Fix incorrect keystone port https://review.opendev.org/c/openstack/kolla-ansible/+/922303 | 10:30 |
| opendevreview | Andrew Babbitt proposed openstack/kolla-ansible stable/2023.2: Skyline: Fix incorrect keystone port https://review.opendev.org/c/openstack/kolla-ansible/+/922304 | 10:31 |
| opendevreview | Andrew Babbitt proposed openstack/kolla-ansible stable/2023.1: Skyline: Fix incorrect keystone port https://review.opendev.org/c/openstack/kolla-ansible/+/922305 | 10:32 |
| opendevreview | Merged openstack/kayobe master: CI: remove SLURP jobs for D cycle https://review.opendev.org/c/openstack/kayobe/+/922080 | 10:39 |
| PrzemekK | Ok found it - in multinode inventory neutron_ovn_availability_zones='["xxx-az"] | 10:42 |
| opendevreview | Maximilian Sesterhenn proposed openstack/kolla master: Add ovn-bgp-agent / FRR / Horizon BGPVPN dashboard https://review.opendev.org/c/openstack/kolla/+/891617 | 10:43 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: Add support for EFI booted seed/infra vms https://review.opendev.org/c/openstack/kayobe/+/921660 | 10:57 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 11:33 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: Add support for EFI booted seed/infra vms https://review.opendev.org/c/openstack/kayobe/+/921660 | 11:36 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: Add support for EFI booted seed/infra vms https://review.opendev.org/c/openstack/kayobe/+/921660 | 11:54 |
| kevko | stan: what about to just mount your nfs to some location and pass it to glance_extra_volumes variable into globals ..this should work | 11:55 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: Add support for EFI booted seed/infra vms https://review.opendev.org/c/openstack/kayobe/+/921660 | 12:21 |
| opendevreview | Merged openstack/kayobe master: remove migrate_rabbitmq_queues in D cycle https://review.opendev.org/c/openstack/kayobe/+/922081 | 12:25 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Add certificates for RabbitMQ internode TLS https://review.opendev.org/c/openstack/kolla-ansible/+/921380 | 12:27 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 12:30 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 12:35 |
| SvenKieske | mhm, do we have an opinion about the usage of community.docker vs kolla-toolbox docker-worker et al? we sometimes use community.docker.docker_container_exec and sometimes kolla-toolbox. this feels weird/complicated to me. either we don't need to maintain our own tool inside kolla-toolbox and migrate everything to the docker community module from ansible or we maybe should streamline our usage and always use | 12:41 |
| SvenKieske | our own docker module, no? | 12:41 |
| SvenKieske | see e.g. https://review.opendev.org/c/openstack/kolla-ansible/+/920714/3..4/ansible/roles/service-destroy/tasks/before_destroy.yml for a current patch where we don't use kolla-toolbox (in it's current patch iteration at least) | 12:42 |
| SvenKieske | problem I also see with using community.docker is, that I don't know if it will always be compatible with podman. | 12:42 |
| SvenKieske | simple cases like exec might work of course | 12:43 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: Add support for EFI booted seed/infra vms https://review.opendev.org/c/openstack/kayobe/+/921660 | 12:44 |
| SvenKieske | it also seems to require: docker sdk, docker cli and requests, according to the readme: https://github.com/ansible-collections/community.docker | 12:45 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 12:45 |
| mnasiadka | mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu - meeting in 15 | 12:45 |
| SvenKieske | \o/ | 12:46 |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Jun 19 13:00:14 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:00 |
| mnasiadka | o/ | 13:00 |
| SvenKieske | o/ | 13:00 |
| mmalchuk | o/ | 13:00 |
| r-krcek | o/ | 13:00 |
| mhiner | o/ | 13:00 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: seed-vm/infra-vms: Add support for EFI and q35 https://review.opendev.org/c/openstack/kayobe/+/921660 | 13:03 |
| mnasiadka | #topic agenda | 13:03 |
| mnasiadka | * CI status | 13:03 |
| mnasiadka | * Release tasks | 13:03 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 13:03 |
| mnasiadka | * Current cycle planning | 13:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:03 |
| mnasiadka | * Open discussion | 13:03 |
| mnasiadka | #topic CI status | 13:03 |
| mnasiadka | Kolla/Kolla-Ansible seems green | 13:03 |
| mnasiadka | Kayobe SLURP jobs are still failing | 13:03 |
| mnasiadka | anything I missed? | 13:03 |
| SvenKieske | kolla build failure rate seems up, at least according to https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1 ? | 13:04 |
| SvenKieske | kolla-ansible is broken in grafana (I guess for some time again, I should check more often) | 13:04 |
| mnasiadka | let me check kolla periodics | 13:06 |
| mnasiadka | last failures from 15th of June in Kolla | 13:06 |
| SvenKieske | mhm, maybe just grafana acting up | 13:07 |
| mnasiadka | anyway, let's continue | 13:07 |
| mnasiadka | #topic Release tasks | 13:07 |
| mnasiadka | R-17 has passed, according to Release Management schedule - we should switch to master branches from 2024.1 | 13:08 |
| mnasiadka | Something like https://review.opendev.org/c/openstack/kayobe/+/791764 | 13:09 |
| mnasiadka | Any volunteers to do this? | 13:09 |
| mmalchuk | I can | 13:10 |
| SvenKieske | ty :) | 13:10 |
| mnasiadka | ok the | 13:10 |
| mnasiadka | n | 13:10 |
| mnasiadka | let's move on | 13:10 |
| kevko | \o | 13:10 |
| mnasiadka | #topic Current cycle planning | 13:10 |
| mnasiadka | I started some work on the Ubuntu Noble LTS support and Ansible bump | 13:11 |
| mnasiadka | Ubuntu nodes do OOM on new Ansible, and some other fantastic things - so having fun | 13:11 |
| SvenKieske | ah we just talked about this internally today (noble). notice afaik noble doesn't support upgrades from 22.04 until 24.04.1 release | 13:12 |
| mnasiadka | Does anybody have any other high priority features that need discussing? | 13:12 |
| mnasiadka | I'm not interested in in-place upgrades now, we just need to get that working and backport this to C | 13:12 |
| frickler | the oom issue should be fixed with 2.17.1, see the comments in your patch | 13:13 |
| kevko | as always ..me :) ..just find some time to review again my patches | 13:13 |
| mnasiadka | Yeah, and hundreds of other patches :) | 13:14 |
| mnasiadka | Let's go to additional agenda | 13:14 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:14 |
| kevko | yeah, but i am actively applying changes regarding comments :) | 13:14 |
| mnasiadka | (SvenKieske: please review patch for broken keystone federation backend https://bugs.launchpad.net/kolla-ansible/+bug/2058656 | 13:14 |
| mnasiadka | trivial patch at https://review.opendev.org/c/openstack/kolla-ansible/+/913908 needs Cores <--still needs input | 13:14 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla-ansible/+/921381/comment/56b11f74_f98925e8/ (rmq startuptime high, reason found, please share your opinion) | 13:14 |
| SvenKieske | yeah the first two are rather getting old, they really don't need any discussion in the meeting imho, just feedback on the patches itself. | 13:15 |
| SvenKieske | the third one I don't know, I really would like to have some more opinions on it, but could also be discussed on the patch itself | 13:16 |
| mnasiadka | My question is - do we have ANY CI job that at least configures keystone to do federation? | 13:16 |
| SvenKieske | let me check.. | 13:16 |
| opendevreview | Maksim Malchuk proposed openstack/kolla master: Revert "[release] Use Caracal sources by default" https://review.opendev.org/c/openstack/kolla/+/922318 | 13:17 |
| mnasiadka | Because if we don't, we're going to see those issues every now and then | 13:17 |
| SvenKieske | true | 13:17 |
| SvenKieske | at least from quick grepping I don't find any tests | 13:17 |
| SvenKieske | that being said, a bugfix is a bugfix. we have many bugfixes where you don't ask for tests, even if there are none | 13:17 |
| kevko | we should add some ... | 13:18 |
| SvenKieske | and ftr I agree we should maybe focus more on tests, but I would appreciate it if this pressure would be applied equally, at least it doesn't feel to be applied equally, from my pov. | 13:18 |
| SvenKieske | but ymmv :) | 13:18 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Add support for customising Neutron physical network names https://review.opendev.org/c/openstack/kolla-ansible/+/922320 | 13:19 |
| mnasiadka | Well, I'm not going to set up an AIO with federation enabled to check if that fixes anything | 13:19 |
| mnasiadka | commented on the patch | 13:20 |
| mnasiadka | SvenKieske: what about that RMQ thing? | 13:21 |
| SvenKieske | the path thing? | 13:22 |
| SvenKieske | well the startup time is raised, because it is computed everytime rmq starts, so we should move it either to build time (in the dockerfile in kolla) or directly hardcode it | 13:22 |
| SvenKieske | not sure though how often this changes, I just had a look at the otp repo where imho the ssl libs erlang sources reside, there is already version 11 and they do frequent releases, but we do seem to have version 9.4 somewhere.. | 13:23 |
| SvenKieske | if that changes frequently we might not want to update this string manually in the dockerfile | 13:24 |
| mnasiadka | well, that's erlang version? | 13:24 |
| mnasiadka | ah, ssl lib in erlang | 13:24 |
| SvenKieske | so I think the best compromise would be to let the erl autodiscovery run at build time, so it doesn't run every rmq start, but we don't need to patch for every ssl lib minor version bump? | 13:24 |
| SvenKieske | from my pov that makes sense, but I'd really like to have some opinions of seasoned maintainers there, so we don't need to rewrite the patch in one or two weeks :) | 13:25 |
| SvenKieske | I discussed this briefly with Matus (patch author) in private. | 13:25 |
| mnasiadka | well, adding that to rabbitmq-env.conf - which we template out in kolla-ansible? | 13:26 |
| SvenKieske | this really did affect the build pipeline which sometimes would just timeout | 13:26 |
| SvenKieske | that's another option, but I'm not sure it fixes anything? doesn't this run on every start as well? or do you mean hardcoding it there? | 13:27 |
| mnasiadka | according to RMQ docs - we need to add that to rabbitmq-env.conf | 13:27 |
| mnasiadka | and if we template out the whole file in kolla-ansible | 13:27 |
| SvenKieske | let me look. wait ok if we template it, it's only run once, seems also fine | 13:27 |
| mnasiadka | then adding that in Dockerfile isn't going to get us anywhere | 13:27 |
| SvenKieske | true | 13:27 |
| SvenKieske | I'm really not an expert, this patchset is the first time I've even seen this option, just looked it up myself in upstream docs. | 13:28 |
| mnasiadka | So unless that's a big burden - I would say it should be added in kolla-ansible | 13:28 |
| mnasiadka | probably just spawning a sidecar container with rabbitmq to get that information out of it - and then use the output to be templated out to rabbitmq-env.conf - only when we enable rabbitmq tls | 13:29 |
| SvenKieske | it's fine either way :) it just doesn't make sense to compute it on every startup and make startup slower and this leads to CI pipeline timeouts in some tests :) | 13:29 |
| mnasiadka | Anyway - any volunteer to do this? | 13:29 |
| SvenKieske | I think Matus can implement it in the same patchset if he has some guidance, should be possible with kolla_toolbox, no? | 13:30 |
| SvenKieske | or should this go in a separate patch? afaik the ssl stuff is only needed for this patch? so can be developed together? | 13:30 |
| mnasiadka | Don't we want to get that backported? | 13:30 |
| mnasiadka | We have RabbitMQ TLS working today, don't we? | 13:31 |
| SvenKieske | this is only inter node tls, so the inter cluster traffic | 13:31 |
| mnasiadka | or is that only for internode TLS? | 13:31 |
| mnasiadka | ok | 13:31 |
| mnasiadka | then incorporate that in the current patch | 13:31 |
| SvenKieske | I don't know if it affects other TLS settings actually | 13:31 |
| mnasiadka | well, maybe it's good to check ;-) | 13:31 |
| SvenKieske | yeah, will do | 13:31 |
| mnasiadka | but RMQ docs rather mention inter-node | 13:31 |
| mnasiadka | anyway | 13:32 |
| mnasiadka | let's move on | 13:32 |
| mnasiadka | (r-krcek): please review https://review.opendev.org/c/openstack/kolla-ansible/+/914997 | 13:32 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: Revert "[release] Use OpenStack 2024.1 release" https://review.opendev.org/c/openstack/kayobe/+/922322 | 13:33 |
| mnasiadka | oh crap, love that kind of changes | 13:34 |
| mnasiadka | Will have a look | 13:35 |
| mnasiadka | #topic Open discussion | 13:35 |
| mnasiadka | Let's move on | 13:35 |
| mnasiadka | Actually I have a topic | 13:35 |
| mnasiadka | Since we only have two core-reviewers that are really active outside of StackHPC - I'd like to propose that if a change has one +2 from SHPC - and a second review has not landed for 2 weeks (after setting RP+1) - I'd like to merge such patches with single company approval. | 13:37 |
| mnasiadka | frickler, kevko: What do you think about that? | 13:37 |
| mmalchuk | please review https://review.opendev.org/c/openstack/kayobe/+/921627 as discussed I've made the followup | 13:38 |
| kevko | i am not sure if i understand | 13:39 |
| frickler | mnasiadka: I guess that's fine for me, even 1 week at most should be fine | 13:39 |
| mmalchuk | and new for Kayobe: https://review.opendev.org/c/openstack/kayobe/+/921628 | 13:39 |
| mmalchuk | summer lack of cores? | 13:40 |
| mmalchuk | again? | 13:40 |
| mnasiadka | kevko: currently we have a rule - if the author is from SHPC - we can't merge it with two core reviewers from SHPC | 13:40 |
| mnasiadka | We have a lot of changes, that are stalled due to missing core review from outside of SHPC | 13:40 |
| mmalchuk | mnasiadka lets add cores | 13:40 |
| mnasiadka | Those have usually RP+1 - and it does not help | 13:40 |
| mnasiadka | mmalchuk: shh | 13:40 |
| frickler | well the role is for any company, not specific to SHPC, just to avoid confusion | 13:41 |
| mnasiadka | yes | 13:41 |
| mmalchuk | frickler not only SHPC use Kolla projects | 13:41 |
| kevko | i have the same problem ... what about trade +2 so you will give me +2 for my review and I will find a time for your review :D | 13:41 |
| mnasiadka | so - in order to not let changes to rot - I'm proposing to merge patches with single company approval 2 weeks after the first +2 - if a second +2 from a separate company does not come in | 13:41 |
| mnasiadka | kevko: show me your patches that already have one +2 :) | 13:42 |
| SvenKieske | I was not asked, but maybe require at least one +1 from another company? that should be a pretty low bar, so maybe not worth spelling it out, though. ¯\_(ツ)_/¯ | 13:43 |
| mnasiadka | So, there's another thing that irritates me most - if I can | 13:43 |
| mnasiadka | Adding +1 to a patch after it gets merged - that's to you SvenKieske and mmalchuk ;-) | 13:43 |
| kevko | mnasiadka: well this is another problem ..how can I get some reviewer if you guys in SHPC just giving +2 to each other ... | 13:43 |
| kevko | but no problem ..i am ok with it btw | 13:44 |
| kevko | faster merging ..is always better ..it's still master ..it can be reverted ..it can be fixed by another patch if needed | 13:44 |
| mnasiadka | kevko: sure, everybody has the same problem - it's even hard to find two SHPC cores having time to review a patch ;-) | 13:45 |
| SvenKieske | mnasiadka: well sometimes I review a patch, give +1, then there is last minute change like typo in commit message and immediately +2 +1 workflow and merged and my review is "gone". I think it's at least a decent signal to re-review the minor change then and say "I also did look at it" | 13:45 |
| mnasiadka | But I'll try to have a look in your patches next week - last two weeks have been extremely hectic on my side | 13:45 |
| SvenKieske | if that's annoying I can leave it be I guess | 13:45 |
| mnasiadka | I think it's useful if there's a comment "you missed that", or "please follow up with a fix to this" | 13:46 |
| kevko | Or we should create some queue or something ... | 13:46 |
| mnasiadka | but just +1 is a bit irritating :) | 13:46 |
| kevko | for example here is reaaaallly trivial and important CI fix | 13:46 |
| kevko | https://review.opendev.org/c/openstack/kolla-ansible/+/919939 | 13:46 |
| mnasiadka | kevko: we have a queue, just add RP+1 :) | 13:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Use 2024.1 as previous_release https://review.opendev.org/c/openstack/kolla-ansible/+/919939 | 13:47 |
| SvenKieske | maybe a good reminder that not only mnasiadka can RP+1; not sure if everybody knows, but mere mortals (non cores) can't RP+1 (currently) | 13:47 |
| kevko | because ...without this patch for example we are testing totally something different as we should ... because now it's tested from 2023.2 to -> current uploaded patch | 13:48 |
| mnasiadka | yup | 13:48 |
| mnasiadka | I even updated that patch :) | 13:48 |
| kevko | mnasiadka: I know | 13:48 |
| mnasiadka | frickler: if you can have a look in ^^ | 13:48 |
| mnasiadka | kevko: as I said, I've been extremely busy - it should be better next week ;) | 13:48 |
| SvenKieske | this would also result in less clicks for backport voting btw, not sure how many reviews it needs, it has gotten quite some attention already: https://review.opendev.org/c/openstack/project-config/+/920472 | 13:48 |
| mnasiadka | SvenKieske: that is project-config, I have no power over that :) | 13:49 |
| kevko | SvenKieske: well, I also added some RP +1 ...but somebody just deleted it :D | 13:49 |
| SvenKieske | frickler? ;) | 13:49 |
| SvenKieske | kevko: lol ok, didn't notice, too bad :) | 13:49 |
| frickler | hmm, "branches: none" is a weird idea, I have to think about that | 13:49 |
| mnasiadka | other option is to remove the config at all | 13:49 |
| mnasiadka | and revert in E | 13:50 |
| mnasiadka | but we would need to add some docs to the release management guide | 13:50 |
| mnasiadka | Feel free to comment with a better solution | 13:50 |
| mnasiadka | we could have a CI var "master_is_slurp" | 13:50 |
| mnasiadka | maybe that's better | 13:50 |
| mnasiadka | if Zuul can honor that | 13:51 |
| SvenKieske | that sounds decent, if it works. | 13:51 |
| frickler | the thing is, the job might still run if it is defined somewhere else, like from stable/2024.1 | 13:51 |
| SvenKieske | I really would like a handy list which releases are slurp, didn't sink into my brain just yet | 13:51 |
| kevko | SvenKieske: https://releases.openstack.org/ :D ? | 13:52 |
| frickler | *.1 is slurp, *.2 isn't. slurp == odd | 13:52 |
| mnasiadka | frickler: basically I would be happier if we run slurp whole cycle | 13:52 |
| SvenKieske | frickler: ah that's a good shorthand! | 13:52 |
| SvenKieske | and I didn't know it's mentioned on the release page as well, thx! | 13:53 |
| mnasiadka | as in when it's master branch, not just when we do rc1 and find out slurp does not work | 13:53 |
| frickler | but for current master, from where would you upgrade? from 2024.1 it is just normal upgrade jobs, don't need to duplicate those | 13:55 |
| frickler | from 2023.1 is too old, we don't want to support that | 13:55 |
| frickler | from 2023.2 ... weird, might not be supported by services | 13:55 |
| mnasiadka | well, when master is 2025.1 | 13:55 |
| mnasiadka | we should be doing slurp from 2024.1 to master (to be 2025.1) | 13:55 |
| mnasiadka | right? | 13:55 |
| frickler | yes | 13:56 |
| frickler | so re-enable as soon as 2024.2 is branched | 13:56 |
| mnasiadka | yeah, was thinking if we can have jobs running when some variable is true | 13:56 |
| mnasiadka | so we would only change that variable | 13:56 |
| mnasiadka | but still setting up slurp per new release is going to take some work | 13:57 |
| frickler | I can check with zuul ppl whether that would be possible | 13:57 |
| mnasiadka | so maybe we just remove jobs from project.yaml | 13:57 |
| mnasiadka | and add them back when master is slurp | 13:57 |
| mnasiadka | and just document that in our Release Management docs | 13:57 |
| mnasiadka | I'm fine with any :) | 13:57 |
| frickler | yes, that would be the usual approach that openstack-zuul-jobs is taking | 13:58 |
| mnasiadka | ok, so let's do that | 13:58 |
| frickler | could even add them all in one project-template and just add/remove that one maybe | 13:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Use 2024.1 as previous_release https://review.opendev.org/c/openstack/kolla-ansible/+/919939 | 13:58 |
| mnasiadka | that also makes sense | 13:58 |
| mnasiadka | I'll have a go at writing the docs and trying out project template | 13:59 |
| mnasiadka | thanks | 13:59 |
| mnasiadka | ok, we're out of time :) | 13:59 |
| mnasiadka | Thank you all for coming - see you next week :) | 13:59 |
| mnasiadka | #endmeeting | 13:59 |
| opendevmeet | Meeting ended Wed Jun 19 13:59:40 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-06-19-13.00.html | 13:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-06-19-13.00.txt | 13:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-06-19-13.00.log.html | 13:59 |
| SvenKieske | If anybody has time, I proposed some backports for some bugfixes as well, which would like a review or two. | 13:59 |
| kevko | btw, if you asked for +2 and 2 weeks rule ...can I ask again my question from past ? one +2 for stable or maybe unmaintained ? | 13:59 |
| SvenKieske | thanks | 13:59 |
| SvenKieske | kevko: wasn't that even agreed to do in the past? just never done? maybe I misremember. | 14:00 |
| kevko | postponed i think :D | 14:00 |
| mmalchuk | thanks mnasiadka | 14:00 |
| kevko | thx | 14:00 |
| kevko | can I ask something even if we end already ? | 14:01 |
| SvenKieske | shoot | 14:02 |
| kevko | why we don't have ansible version in requirements in kolla-ansible ? | 14:04 |
| kevko | as ansible is the most important dependency for kolla-ansible which is in fact ansible only ? :D | 14:04 |
| mmalchuk | good question) | 14:05 |
| SvenKieske | kevko: we have it in lint-requirements.txt though ;) https://opendev.org/openstack/kolla-ansible/src/branch/master/lint-requirements.txt#L1 | 14:05 |
| kevko | SvenKieske: good joke :D | 14:06 |
| SvenKieske | I always try my very best :D | 14:06 |
| kevko | for me this is like very weird | 14:06 |
| kevko | we are trying to handle version in kolla ansible bash command | 14:06 |
| SvenKieske | I remember asking it myself, but I don't remember the answer I gave myself. | 14:07 |
| kevko | and when we need to bump ansible version ...it's again hacked in bash | 14:07 |
| kevko | i am sure that now you will have better arguments ... | 14:07 |
| kevko | if someone will reply why it is like that now | 14:08 |
| SvenKieske | it's also in test-requirements.txt btw :) | 14:08 |
| SvenKieske | but different max version.. <8 vs <10 | 14:09 |
| kevko | yeah ..because it's on several places ... | 14:09 |
| SvenKieske | kevko: let's propose a patch and see if anything breaks? | 14:09 |
| kevko | let me do it ... | 14:09 |
| SvenKieske | nice, I'm willing to debug failed CI pipelines, if there are any :) | 14:10 |
| kevko | i am 90 percent sure that everything will work | 14:11 |
| kevko | BUT | 14:11 |
| kevko | maybe it's because of zuul nodes has ansible from packages ..maybe ...i don't know | 14:11 |
| kevko | so kolla will fail if there is different version ... | 14:12 |
| kevko | but i don't think kolla-ansible as project should care about user's ansible version in some bash script ..my idea is that requirements saying ansible version should be X.Y.Z and user should resolve conflicts ... or as written on many places ...USE virtualenv | 14:13 |
| SvenKieske | yeah, maybe we should just force installation into venv? | 14:17 |
| SvenKieske | I guess that's needed for the pip install stuff anyway, isn't it? because it can't alter global packages anymore on modern distributions. | 14:18 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 14:18 |
| kevko | SvenKieske: i have everything in docker container | 14:18 |
| kevko | SvenKieske: so i have kolla-ansible container on deploy server ...mounted git as config to container and that's it ... | 14:18 |
| kevko | SvenKieske: installed from git ...so i have git blame , git log ..everything available ..it's my kolla sandbox ...and i think it should be like this ... but ..i don't know what are the opinions of others | 14:19 |
| kevko | SvenKieske: in linux everything is possible :D ^^ regarding pip and global packages | 14:20 |
| SvenKieske | seems like a nice setup, I had it also this way in the past, our downstream handles this a little bit different though. | 14:21 |
| kevko | SvenKieske: well, the main reason was this https://bugs.launchpad.net/kolla-ansible/+bug/1863510 and this https://github.com/ansible/ansible/pull/71118/files | 14:24 |
| kevko | SvenKieske: kolla never fixed this really important stuff ... :( | 14:24 |
| kevko | SvenKieske: TLDR version ... if kolla-ansible check-containers reporting changed for neutron-server ... or change some file which will trigger handler to restart neutron-server .... every neutron* containers are restarted :D | 14:26 |
| kevko | SvenKieske: which is reaaally painful under heavy load on L3 nodes for example ... | 14:27 |
| kevko | SvenKieske: so for every kolla-ansible container i am building ansible itself .... https://paste.openstack.org/show/bBwgV1mIvQwv4od0ICG3/ <<< | 14:28 |
| kevko | SvenKieske: and my handlers working as expected :) | 14:28 |
| kevko | ansible developers won't fix | 14:29 |
| SvenKieske | I just wanted to follow through why they won't fix it, but it was just decided in some ansible meeting and it seems the minutes from this 2020 meeting are gone from the server.. | 14:30 |
| SvenKieske | actually I think we hit this handler bug before without knowing about it (at least I didn't know about it) | 14:30 |
| SvenKieske | ah it's at least explained on the bug report: https://github.com/ansible/ansible/issues/22579#issuecomment-705641746 | 14:32 |
| SvenKieske | mhm, there is even a solution proposed (a workaround to achieve this in ansible itself). maybe we can put that in a wrapper for kolla? | 14:33 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 14:33 |
| SvenKieske | kevko: notify: "{{ results is changed|ternary('Restart ' ~ item.key ~ ' container', None) }} | 14:33 |
| SvenKieske | there's a documented workaround here: https://gist.github.com/yoctozepto/4408d70fee31ac3ba9d8a6491c370dfe | 14:33 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: Test haproxy single external frontend https://review.opendev.org/c/openstack/kolla-ansible/+/922288 | 14:36 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: kolla_url: port is a string https://review.opendev.org/c/openstack/kolla-ansible/+/922287 | 14:36 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default. https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 14:36 |
| SvenKieske | kevko: thanks for the pointer, I wasn't really aware of it, maybe we can fix this? | 14:38 |
| kevko | SvenKieske: i know there is a workaround ... i was there when it was actual .... but you know ... if you post XL patch ..it will lie in review for several cycles :D | 14:43 |
| SvenKieske | yes | 14:44 |
| SvenKieske | maybe patch one service at a time? :D I have no good answer to this myself, but Roman seemed also eager to tackle this, so maybe we can form a small group to work on it? | 14:44 |
| kevko | SvenKieske: well, i don't like the idea to replace built-in notify ...which is also very readable ... to some register and action ... | 14:49 |
| kevko | I just don't like it's anti-pattern | 14:50 |
| kevko | SvenKieske: and I am also afraid about edge cases ... {{ container_check.results | select('changed') | map(attribute='item') | map(attribute='key') | map('regex_replace', '^(.*)$', 'Restart \\1 container') | list }} << it looks like there can be some :D | 14:51 |
| kevko | SvenKieske: also you can (if i am correct) to specify delegate to ..or similar args and call handler ... by above approach i am not sure you can handle everything | 14:52 |
| kevko | SvenKieske: so i better patched ansible and using my container for 5 releases without problem | 14:53 |
| SvenKieske | yeah, I get that, but if upstream doesn't fix it we surely should do something, to make it less errorprone on our side? | 14:53 |
| SvenKieske | I don't know, I mean that is a nice hack, but if, for whatever reason you end up running an unmodified ansible for just one run you might have bad luck. | 14:54 |
| kevko | SvenKieske: why would I do that ? I have container and everything related to my cloud i am orchestrating ..i am exec into kolla-ansible container | 14:55 |
| kevko | SvenKieske: i have one place where I have a kolla-ansible installed ...inside container ... | 14:56 |
| SvenKieske | tada, no more enforced backport voting on already backported patches, see e.g. https://review.opendev.org/c/openstack/kolla-ansible/+/921672 | 15:13 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Add support for customising Neutron physical network names https://review.opendev.org/c/openstack/kayobe/+/922335 | 15:53 |
| SvenKieske | kevko: there is a bug in your ceph refactoring with regards to the cinder backup keyring https://review.opendev.org/c/openstack/kolla-ansible/+/907166 | 17:27 |
| kevko | SvenKieske: thanks ..but interisting | 18:31 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: seed-vm/infra-vms: Add support for EFI and q35 https://review.opendev.org/c/openstack/kayobe/+/921660 | 19:14 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: seed-vm/infra-vms: Add support for EFI and q35 https://review.opendev.org/c/openstack/kayobe/+/921660 | 19:14 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: seed-vm/infra-vms: Add support for EFI and q35 https://review.opendev.org/c/openstack/kayobe/+/921660 | 19:15 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: seed-vm/infra-vms: Add support for EFI and q35 https://review.opendev.org/c/openstack/kayobe/+/921660 | 19:17 |
| opendevreview | Andrew Babbitt proposed openstack/kolla-ansible master: Skyline: Fix incorrect keystone port https://review.opendev.org/c/openstack/kolla-ansible/+/922302 | 20:28 |
| ababbitt | In relationship to skyline, apiserver talks to the other APIs using the public endpoints by default. Any qualms changing this? | 20:35 |
| ababbitt | Also please bear with me, I haven't used Gerrit in like 10 years lol but would like to start contributing fixes and stuff as I come across them o/ | 20:37 |
| opendevreview | Merged openstack/kolla-ansible master: CI: Use 2024.1 as previous_release https://review.opendev.org/c/openstack/kolla-ansible/+/919939 | 20:49 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!