| opendevreview | Takashi Kajinami proposed openstack/nova master: Replace license classifier https://review.opendev.org/c/openstack/nova/+/951226 | 01:09 |
|---|---|---|
| opendevreview | sean mooney proposed openstack/nova-specs master: Add cpu-teiring with cpu_shares spec https://review.opendev.org/c/openstack/nova-specs/+/951222 | 01:12 |
| opendevreview | sean mooney proposed openstack/nova-specs master: Add cpu-teiring with cpu_shares spec https://review.opendev.org/c/openstack/nova-specs/+/951222 | 01:13 |
| *** elodilles is now known as elodilles_ooo | 05:45 | |
| gibi | gmaan: sean-k-mooney: I replied about evacuate in https://review.opendev.org/c/openstack/nova-specs/+/937650/comment/aaf66b8d_53e159c3/ | 07:03 |
| zigo | Under Trixie, with RabbitMQ 4.0.5, I'm getting: | 08:58 |
| zigo | PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/' | 08:58 |
| zigo | Is this a miss-feature somewhere ?!? | 08:58 |
| zigo | I mean, in oslo.messaging ... | 08:58 |
| sean-k-mooney | zigo: not that im aware of but i belive kolla hit some issues with rabbit 4.x and i belive they also use debain | 10:28 |
| sean-k-mooney | zigo: all our testing for expoy and master is still useing 3.12.1 | 10:30 |
| sean-k-mooney | zigo: that what ubuntu 24.04 is shipping currently | 10:30 |
| sean-k-mooney | zigo: im not sure if oslo.messaging offically supprot rabbit 4.X | 10:34 |
| sean-k-mooney | you may be able to work around the issue by changign the cofnig default in service like chaging to quorm_queues | 10:35 |
| sean-k-mooney | zigo: i dont know if the idea of having a debian testing image has ever been raised with infra or what the eta is for trixie. we currently use debain for spice testign in the nova-ovs-hybrid-plug job | 10:36 |
| sean-k-mooney | i would be happy to see that move to trixe or sid in the future once its aviable | 10:37 |
| sean-k-mooney | although the main blocker is likely to be python 3.13 support | 10:37 |
| frickler | so far we would have to wait for trixie to be released to build an image with nodepool. with the zuul-launcher feature we might be able to start earlier, but I didn't have time yet to look into this | 10:39 |
| frickler | also, yes, the version of oslo.messaging will likely be relevant. I think this should all be working in kolla, but maybe our CI coverage is incomplete | 10:40 |
| sean-k-mooney | frickler: im wondering if the debian installer and kolla are using slightly diffent config options for the messaging transport | 10:44 |
| sean-k-mooney | frickler: for example ha queue have been deprecated for a long time they may have finally killed them in 4? | 10:45 |
| frickler | sean-k-mooney: yes, there was a lot of cleanup work to be done in the last cycle, but kolla does ship with rmq 4 for 2025.1 | 10:46 |
| sean-k-mooney | that at least promising as it implies there is a workable config that zigo could move too | 10:47 |
| zigo | sean-k-mooney: I did move to quorum queues, no choice with RabbitMQ 4.0.5, as it doesn't support HA queues anymore. Though that's not the issue. The issue is nova attempting to set not-durable, which seems to be gone from Rabbit. | 10:56 |
| zigo | I wonder how can I tell oslo.messaging not to do that. | 10:57 |
| sean-k-mooney | zigo: have you treied setting the durable_queses config option to true | 10:57 |
| zigo | I did grep not-durable in oslo.messaging and didn't find where it is. | 10:57 |
| sean-k-mooney | oslo is ment to ignore it when using quorm_queues which requrie durablity | 10:57 |
| zigo | sean-k-mooney: This creates more issues that it solves, I wouldn't recommend to anyone to do that. | 10:57 |
| sean-k-mooney | https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/amqp.py#L33-L37 | 10:58 |
| sean-k-mooney | zigo: quourm queue are unconditonally durable in rabbit | 10:58 |
| zigo | Oh ok. | 10:59 |
| sean-k-mooney | https://www.rabbitmq.com/docs/quorum-queues#feature-matrix | 10:59 |
| zigo | Let me try something then. | 10:59 |
| sean-k-mooney | so oslo should be ignoreing that config option but it may be bugged or it may be fixed but in a newer verion | 11:00 |
| zigo | I'll try setting durable_queue to true and see how it goes. | 11:00 |
| zigo | Though as much as I can tell, this doesn't fix. | 11:00 |
| zigo | Let me try. | 11:00 |
| zigo | Same thing. | 11:01 |
| zigo | I'm getting: | 11:01 |
| zigo | Queue.declare: (406) PRECONDITION_FAILED - invalid property 'non-durable' for queue 'conductor' in vhost '/' | 11:01 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: trying to figure out what broke the unit test https://review.opendev.org/c/openstack/nova/+/951393 | 11:02 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [hacking] N374 do not use time.sleep(0) to yield https://review.opendev.org/c/openstack/nova/+/950992 | 11:06 |
| zigo | sean-k-mooney: Any idea what's attempting to add non-durable and how one does that in Rabbit? | 11:08 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Cache [pci]alias parsing https://review.opendev.org/c/openstack/nova/+/951151 | 11:11 |
| sean-k-mooney | zigo: not off the top of my head sorry | 11:44 |
| frickler | zigo: did you set rabbit_quorum_queue and rabbit_transient_quorum_queue? | 11:56 |
| zigo | I did. | 11:56 |
| zigo | Both to true. | 11:56 |
| zigo | frickler: If I comment this line: | 11:57 |
| zigo | https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L447 | 11:57 |
| zigo | then daemons car start ... | 11:57 |
| frickler | for reference here's the full template that kolla uses, but maybe some rmq config is also relevant https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/aodh/templates/aodh.conf.j2#L63-L78 | 11:57 |
| zigo | frickler: And this has been tested with rabbitmq >= 4 ? | 11:58 |
| frickler | yes. are you doing a fresh deploy or an upgrade? | 11:59 |
| zigo | Fresh deploy. | 11:59 |
| zigo | Using Trixie + Epoxy. | 11:59 |
| zigo | Epoxy on Bookworm works, though I'm trying to finish my setup on Trixie, where many things just break ... :P | 12:00 |
| frickler | https://opendev.org/openstack/kolla/src/branch/master/docker/rabbitmq/apt_preferences_rabbitmq.debian | 12:00 |
| zigo | The pinning for erlang is wrong, Trixie has 27.3.3. | 12:01 |
| frickler | well that is from erlang/rmq upstream repos, not debian pkgs | 12:01 |
| zigo | Ah... ok ... | 12:02 |
| zigo | Got some same kind of issues with Neutron. :/ | 12:15 |
| zigo | oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on cl1-controller-2.infomaniak.ch:5671 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - invalid property 'non-durable' for queue 'q-plugin' in vhost '/' | 12:15 |
| zigo | I'll redo a setup from scratch, I guess. | 12:22 |
| sean-k-mooney | zigo: rabbit erqires all service on a given vhost to have the same queue config | 12:31 |
| sean-k-mooney | so if nova is uing durable qurom ques and nova and neturon share a vhost then they would both have to align | 12:31 |
| sean-k-mooney | with that said we do not recommend sharign vhosts between openstack services | 12:31 |
| zigo | Of course ! :) | 12:32 |
| zigo | I'll have a go setting durable=True everywhere, and see how that goes. | 12:32 |
| sean-k-mooney | zigo: ya we tried to do that in our new installer and hit exactly the same thing it was all or nothing | 12:33 |
| sean-k-mooney | apprently tripleo never used per service vhosts... and when rabbit supprot was added for our our new installer they said well if it was good enough for tripleo why change it... | 12:34 |
| sean-k-mooney | it makes me sad inside as that is the most basic form of isolation that you should alwasy do ... | 12:34 |
| sean-k-mooney | gibi: melwitt: just an fyi https://bugs.launchpad.net/nova/+bug/2112187 it looke like live migration after a cinder swap volume may be broken | 12:35 |
| sean-k-mooney | gibi: melwitt: this was found via our watcher testing im currently poking around in the env to see if i can find anyting useful | 12:36 |
| sean-k-mooney | its reproducable as well. we have not tried doign it by hand but this is the 3 time we have repoduced this in 2 difent envs | 12:38 |
| sean-k-mooney | gibi: melwitt: so i confiemd that it look liek we are not properly updating the migration xml. on the dest host the voluem path is /dev/sdc but we used /dev/sdd (source_path) for the migration xml | 14:13 |
| sean-k-mooney | gibi: melwitt im not conviced this si a regriesson related to the recent live migation roleback change | 14:14 |
| sean-k-mooney | however this env has that patch applied | 14:15 |
| sean-k-mooney | its effetivly master | 14:15 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 14:17 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 14:18 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 14:19 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 14:22 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 14:23 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 14:24 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user update auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 14:26 |
| gmaan | gibi: thanks | 15:10 |
| opendevreview | Ghanshyam proposed openstack/nova-specs master: Propose API policy manager role spec https://review.opendev.org/c/openstack/nova-specs/+/937650 | 15:45 |
| gmaan | sean-k-mooney: gibi ^^ updated (removed the evacuate API policy change) | 15:46 |
| *** haleyb is now known as haleyb|out | 16:59 | |
| melwitt | sean-k-mooney: hm ok, thanks for the heads up. just a thought, how hard would it be to revert the patch in your env to see if it helps? might be a fast track to getting some info | 17:36 |
| opendevreview | Ghanshyam proposed openstack/nova-specs master: Propose API policy manager role spec https://review.opendev.org/c/openstack/nova-specs/+/937650 | 18:10 |
| gmaan | sean-k-mooney: ^^ addressed your comment for admin-password | 18:28 |
| gmaan | melwitt: as you also reviewed manager role spec earlier ^^ this is up with final shape, please check and provide your feedback. | 18:29 |
| opendevreview | Ivan Anfimov proposed openstack/placement master: wip https://review.opendev.org/c/openstack/placement/+/951432 | 18:31 |
| gmaan | stephenfin: I left one comment on your favor API improvement spec, I think this is good opportunity to make additionalProperties=False for other API also https://review.opendev.org/c/openstack/nova-specs/+/940440/6/specs/2025.2/approved/flavor-search-by-name.rst#58 | 18:32 |
| gmaan | this is just a suggestion but I will not block your spec if you would like to just improve flavor API in this. let me know | 18:33 |
| melwitt | gmaan: ack will do | 18:40 |
| opendevreview | Ivan Anfimov proposed openstack/placement master: docs: start use https https://review.opendev.org/c/openstack/placement/+/951432 | 18:43 |
| gmaan | thanks | 18:45 |
| opendevreview | Ivan Anfimov proposed openstack/placement master: docs: placement service start use https https://review.opendev.org/c/openstack/placement/+/951432 | 18:45 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user update auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 18:46 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user update auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 18:52 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user update auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 19:00 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: nova service_user update auth_url https://review.opendev.org/c/openstack/nova/+/938680 | 19:03 |
| opendevreview | Ivan Anfimov proposed openstack/nova master: docs: update for services to https https://review.opendev.org/c/openstack/nova/+/938680 | 19:05 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!