| opendevreview | Merged openstack/openstack-ansible master: Add variables to control HSTS records https://review.opendev.org/c/openstack/openstack-ansible/+/934620 | 00:47 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible master: Use mariadb-admin instead of mysqladmin https://review.opendev.org/c/openstack/openstack-ansible/+/934430 | 00:47 |
| opendevreview | Merged openstack/openstack-ansible master: Remove usage of mariadb.com infra mirror https://review.opendev.org/c/openstack/openstack-ansible/+/934037 | 00:47 |
| *** fungi is now known as Guest9249 | 01:33 | |
| *** kinrui is now known as fungi | 01:41 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Set Rocky 9 Ceph jobs as voting https://review.opendev.org/c/openstack/openstack-ansible/+/933592 | 07:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Get exact version of installed rabbitmq https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934822 | 07:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Bump RabbitMQ version to 4.0 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934060 | 07:50 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Ensure that rabbitmq-erlang and rabbitmq-server repos use the same pins https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/933724 | 07:55 |
| noonedeadpunk | some status update for myself - need to implement rabbitmq cluster bootstrap feature and in haproxy to drop extra_lb_(?tls_)vip_addresses | 07:57 |
| noonedeadpunk | then with landing mariadb 11.4 and rabbitmq 4.0 I think we can branch? | 07:58 |
| noonedeadpunk | or should we jsut backport things once what we have lands? | 07:58 |
| noonedeadpunk | and I frankly haven't looked if migration to khepri really happens.... | 08:03 |
| andrewbonney | Just checked out deployments as I accidentally moved to 3.13 with 2024.1. Looks like even without the explicit check, because that flag is experimental it doesn't get enabled so there's no issue | 08:04 |
| noonedeadpunk | on 3.13 it's experimental indeed. but on 4.0 it's not anymore | 08:04 |
| noonedeadpunk | so I guess it's jsut migrated during upgrade? | 08:05 |
| * noonedeadpunk killed all envs with 4.0 now | 08:05 | |
| andrewbonney | Yes I suspect so. I'll do another check of that this morning to see | 08:05 |
| noonedeadpunk | if that post-upgrade feature enable is executed :D | 08:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Remove cinder v2 references https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/934593 | 08:10 |
| jrosser | o/ morning | 08:33 |
| noonedeadpunk | o/ | 08:33 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_nova master: nova: template nova configuration directory https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/934933 | 09:29 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_nova master: Load vendor_data.json from versioned path https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/934934 | 09:29 |
| noonedeadpunk | andrewbonney: about that. I think we should take neutron approach instead | 09:42 |
| noonedeadpunk | I was jsut lazy to work on it for other repos | 09:42 |
| andrewbonney | I was looking at that, but wasn't sure it would actually solve the issue. It still looked like /etc/neutron/ would be missing for a period, but I may be reading it wrong | 09:42 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_neutron/commit/57638854532690b61c4abc6d735c17f31950b7e6 | 09:43 |
| noonedeadpunk | so we swap a symlink in handlers: https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/handlers/main.yml#L66-L77 | 09:43 |
| noonedeadpunk | when service is stopped | 09:43 |
| andrewbonney | Does that step also deal with removing the old symlink? | 09:44 |
| noonedeadpunk | yeah, as `force: true` | 09:44 |
| noonedeadpunk | I'd say this should be done for all services with "smart sources" but neutron was most affected back then | 09:45 |
| noonedeadpunk | (and we didn't care about failed scheduling too much during role runtime) | 09:46 |
| andrewbonney | Ok. Does https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/neutron_pre_install.yml#L48 not do anything for a symlink then? Wondering why it's there at all if it only runs for source installs | 09:46 |
| noonedeadpunk | because link will return `neutron_conf_dir_stat.stat.isdir` as false there | 09:47 |
| noonedeadpunk | I think it was just and upgrade task and probably we can drop it now at all | 09:48 |
| andrewbonney | Ok, I'll have another pass over the Nova role. If that looks reasonable it can be done to others too | 09:48 |
| noonedeadpunk | but it can be handy if moving from distro setup to source one. | 09:48 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_nova master: Change ordering of /etc/ operations to improve upgrades https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/934941 | 10:24 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Use rabbitmq_cluster_state task instead of command https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/931975 | 11:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Bump RabbitMQ version to 4.0 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934060 | 11:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Bump RabbitMQ version to 4.0 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934060 | 13:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Document requirement for RabbitMQ upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/934828 | 13:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Bump RabbitMQ version to 4.0 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934060 | 13:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Bump RabbitMQ version to 4.0 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934060 | 13:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Bump RabbitMQ version to 4.0 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934060 | 13:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Move version pinnings from distro vars to defaults https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934965 | 13:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Respect `package_state` for the role https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/934966 | 13:34 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_nova master: Change ordering of /etc/ operations to improve upgrades https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/934941 | 13:49 |
| mgariepy | why is it not happening every time ? https://bugs.launchpad.net/horizon/+bug/2045394 | 14:59 |
| jrosser | it is very annoying | 15:02 |
| mgariepy | just once in a few tmies.. (TM) | 15:03 |
| noonedeadpunk | I was never able to catch it outside of CI | 15:12 |
| noonedeadpunk | was running horizon role quite frequently recently, and it was always successfull | 15:12 |
| mgariepy | it doesnt happen too often juste enough to be annoying | 15:12 |
| noonedeadpunk | yeah | 15:12 |
| noonedeadpunk | jrosser: are you using that to get alternative versions for some things like calico? https://github.com/vexxhost/magnum-cluster-api/pull/426/files | 17:39 |
| jrosser | sort of | 17:40 |
| noonedeadpunk | or that's the tooling specifically for images preparation? | 17:40 |
| jrosser | what do you need to do? | 17:40 |
| noonedeadpunk | well, kind of that: https://github.com/vexxhost/magnum-cluster-api/pull/455 | 17:40 |
| noonedeadpunk | and it works | 17:40 |
| noonedeadpunk | but a) I don't like it's hardcoded b) it's failing unit test... | 17:41 |
| jrosser | this is for workload clusters? | 17:41 |
| noonedeadpunk | yeah | 17:41 |
| jrosser | ah ok so we have re-used the image_loader thing to prepare a registry for the control plane side | 17:41 |
| noonedeadpunk | aha | 17:41 |
| jrosser | but thats cilium, so yes makes sense that calico is for the workload side | 17:42 |
| jrosser | for our case, the workload side actually does have internet so we have not made a private registry for that | 17:42 |
| jrosser | but having said that, we do have good tooling now for the control plane side registry | 17:42 |
| noonedeadpunk | gotcha. I was more about - you dynamically load manifests and they are now not flexible enough ,so I was thinking if you was solving same issue | 17:43 |
| noonedeadpunk | as eventually k8s 1.28 needs calico 3.27 ideally | 17:44 |
| noonedeadpunk | ok, will try to figure out how to fix the unit test, as you was doing slightly different thing | 17:44 |
| jrosser | we want to dynamically load manifests yes, so that PR of ours would be applicable for both use cases | 17:45 |
| noonedeadpunk | aha | 17:45 |
| jrosser | becasue we leverage the same tool to build the registry | 17:45 |
| jrosser | and to cover upgrades you have to load N and N+1 images into the registry | 17:46 |
| jrosser | so it's all a bit wtf | 17:46 |
| noonedeadpunk | but I somehow was thinking that this not part of the image... but not sure... | 17:46 |
| jrosser | image? vm image? | 17:46 |
| noonedeadpunk | yeah | 17:47 |
| noonedeadpunk | what image is this about? lol | 17:47 |
| jrosser | the vm images have no container images in them | 17:47 |
| noonedeadpunk | yeah, so they will fetch it afterwards | 17:48 |
| jrosser | right | 17:48 |
| noonedeadpunk | but I think manifests are placed by the driver inside master node of the created workload cluster? | 17:48 |
| jrosser | image_loader prepares a registry that has all the container images that the VM images might need | 17:48 |
| jrosser | not really | 17:48 |
| noonedeadpunk | ok, yeah. and I'm trying to work with just upstream images for now... | 17:49 |
| jrosser | for calico i don't exactly know how this works tbh | 17:49 |
| jrosser | but some things get installed into the vm with charts here https://github.com/vexxhost/magnum-cluster-api/tree/main/magnum_cluster_api/charts | 17:50 |
| noonedeadpunk | ok, yeah, I'm not fully into how this all works... Just once tried to define `calico_tag` as a label realized it doesn't work "magically" like with Heat, but fails with `[Errno 2] No such file or directory: '/openstack/venvs/magnum-29.0.0/lib/python3.10/site-packages/magnum_cluster_api/manifests/calico/v3.28.2.yaml'` | 17:52 |
| noonedeadpunk | so still pretty much confusing myself here | 17:53 |
| jrosser | i am not sure that it is possible to take heat concepts across here at all | 17:53 |
| noonedeadpunk | yeah | 17:53 |
| noonedeadpunk | but defarmation, you know | 17:54 |
| noonedeadpunk | *deformation | 17:54 |
| jrosser | we do build our own images now | 17:54 |
| jrosser | and it seems ok | 17:55 |
| noonedeadpunk | was runing heat for years and close to never was looking inside of k8s | 17:55 |
| noonedeadpunk | so was expecting that if label is available - I should be able to set it to value and driver will fetch the proper image for me... | 17:57 |
| jrosser | well, labels are i think defined in magnum, not the driver | 18:00 |
| jrosser | try setting cidr on master lb fip for example :( | 18:00 |
| noonedeadpunk | they pretty much passed/handled by driver | 18:00 |
| noonedeadpunk | like LB type - ovn or amphora | 18:01 |
| noonedeadpunk | it's capi that will perform order of LB in octavia | 18:01 |
| jrosser | i have been trying to rotate passwords on bmc this week | 18:02 |
| jrosser | oh what a total nightmare :( | 18:02 |
| noonedeadpunk | haha, I've heard of such stories from our infra as well lately | 18:02 |
| noonedeadpunk | they also tried to setup "valid" certs for bmc. through API | 18:03 |
| jrosser | oh interesting | 18:03 |
| jrosser | i bet that was challenging | 18:03 |
| noonedeadpunk | Dell iDRAC is full of surprises. | 18:03 |
| noonedeadpunk | about format, length, random errors, etc... | 18:03 |
| jrosser | oh yeah i am finding similar on supermicro | 18:04 |
| jrosser | some with just broken redfish totally | 18:04 |
| noonedeadpunk | but I think they ended up with some ansible role which works in most of cases... | 18:04 |
| jrosser | some with pretty much undocumented password complexity rules | 18:04 |
| noonedeadpunk | supermicro was another fun I've heard. different kind of fun, with random reboots | 18:05 |
| noonedeadpunk | oh, yes, password length and complexity - I heard about that as well | 18:05 |
| noonedeadpunk | It's really a huge shame that in our org there's close to none of upstream contributions culture :( | 18:06 |
| noonedeadpunk | jrosser: arddennis was actually working on tls certs for redfish | 18:20 |
| noonedeadpunk | iirc it had also smth to do with hashi vault usage | 18:20 |
| jrosser | oh that sounds interesting | 18:20 |
| jrosser | pretty much what i would like to do but just did not get to look yet | 18:21 |
| noonedeadpunk | so I kinda have some hope that he might be able to share something somehow | 18:22 |
| noonedeadpunk | but I have close to no influence on that unfortunatelly | 18:22 |
| jrosser | i was thinking that if i had the hostname/fqdn set correctly on the bmc i could ask it for a CSR, push that through vault and get a certificate out | 18:23 |
| mgariepy | bmc truncating password without notice is also fun. | 18:26 |
| noonedeadpunk | like lekystone did a while back ?:) | 18:28 |
| mgariepy | never saw it in keystone when was it ? | 18:30 |
| arddennis | With the Dell servers I have examples how to do this in Ansible (gen CSR, sign it in vault and upload). I can give examples if needed. | 18:30 |
| mgariepy | i mostly use ldap for auth anyway | 18:30 |
| noonedeadpunk | mgariepy: before or even in 2023.1 | 18:32 |
| jrosser | arddennis: that would be really interesting - i don't know if it would translate to supermicro or not but just seeing how you handled it with the redfish modules (or had to resort to get_uri?!) would be very helpful | 18:32 |
| noonedeadpunk | and then it started trimming them incorrectly at some point | 18:32 |
| noonedeadpunk | mgariepy: https://review.opendev.org/c/openstack/keystone/+/890936 and https://review.opendev.org/c/openstack/keystone/+/891024 | 18:34 |
| noonedeadpunk | so eventually keystone bcrypt, which is default, was always trimming passwords.... | 18:36 |
| jrosser | noonedeadpunk: do you use the secrets key/value engine for vault at all/? | 18:37 |
| mgariepy | the bmc was triming a 8 chars irrc lol | 18:38 |
| noonedeadpunk | jrosser: nah, not really. | 18:38 |
| jrosser | ah no problem, we have made a 'helper' cli tool for that which is very cool, and i was thinking to open source that | 18:39 |
| noonedeadpunk | frnakly I had some thoughts/discussions about placing user_secrets there, but for now it's ansible-vault still | 18:40 |
| jrosser | i think we will get to that, just needs some time | 18:40 |
| noonedeadpunk | I also wanna try to play with AWX and osa integration. as when playbooks are in a separate collection.... | 18:43 |
| arddennis | jrosser: Here is example: https://paste.openstack.org/show/bH7DKXADlYLkquxvPjoa/ I can share only uri examples. I have a module for the ansible which handles most of this, but in order to share it I need permission from the company I am working for. | 18:43 |
| noonedeadpunk | arddennis: I wonder if it's worth asking, and at least publish that on our github... | 18:44 |
| noonedeadpunk | but actually that paste looks not afwul | 18:45 |
| noonedeadpunk | s/our/company/ | 18:45 |
| jrosser | i think what i was surprised at was how quickly the redfish_command module becomes not helpful | 18:47 |
| arddennis | current ansible redfish modules are a bit limited. Look at their source code. It is more like a proxy to API services with filtering on what you can send. | 18:51 |
| noonedeadpunk | I'd guess this how openstack modules might look soon when they'll get generated from openapi | 18:53 |
| mgariepy | https://zuul.opendev.org/t/openstack/build/14b37674ed884e3cb5beab65258b43f7/log/logs/host/neutron-server.service.journal-19-15-05.log.txt#6745 | 21:06 |
| mgariepy | someone seen that before ? race in ovn/ovsdb ? | 21:06 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!