| oklhost_ | morning. are there any known issues with rabbitmq after upgrading ussuri -> victoria on ubuntu-source? we got a lot "missed heartbeats from client" in rabbitmq. | 06:11 |
|---|---|---|
| *** oklhost_ is now known as oklhost | 06:11 | |
| yoctozepto | oklhost: it's the same rabbitmq between those; perhaps extra load due to upgrade itself? | 06:24 |
| oklhost | yoctozepto: hm. the upgrade was a week ago, also a 2nd fresh install facing the behaviour. yesterday we was able to fix it for a while after restarting nova-api containers | 06:26 |
| yoctozepto | oklhost: ack, interesting | 06:29 |
| yoctozepto | then perhaps a bug in nova? | 06:29 |
| ignaziocassano | Hello Everyone, with kolla wallaby we are facing another issue with cinder driver volume_driver = cinder.volume.drivers.netapp.common.NetAppDriver. I talked with netapp engineers and they do not have the same issue with a devstack installation. The cinder volume does not start in netapp cinder rott volume have .snapshotvisibility checked because cinder volume tries to change .snapshot owner | 06:45 |
| oklhost | yoctozepto: we're sill investigating. by now, it's not clear if only nova is affected. https://paste.opendev.org/show/809292/ | 06:46 |
| ignaziocassano | The workaround can be done disablinng .snapshot visibility on netapp root volume but it is only a workaround | 06:47 |
| ignaziocassano | Any suggestion please ? Do you thing I must write on cinder channel ? | 06:50 |
| ignaziocassano | Any suggestion please ? Do you think I must write on cinder channel ? | 06:50 |
| ignaziocassano | yoctozepto: what do you think about the issue I wote above ? | 06:55 |
| yoctozepto | ignaziocassano: I did not understand the issue :-( could you describe it on launchpad with more detail? | 06:59 |
| ignaziocassano | yoctozepto: yes, of course. Thanks | 07:00 |
| yoctozepto | oklhost: I recall there was some recommendation to avoid this happening too often; but I can't find it now | 07:00 |
| yoctozepto | others will surely know | 07:00 |
| oklhost | yoctozepto: that sounds hopefully :-) we're currently playing around with "heartbeat_rate" for certain services. | 07:01 |
| oklhost | https://paste.opendev.org/show/809293/ a lot services affected :D | 07:04 |
| opendevreview | Piotr Parczewski proposed openstack/kolla master: Add Prometheus libvirt exporter image https://review.opendev.org/c/openstack/kolla/+/642095 | 07:04 |
| *** hjensas is now known as hjensas|afk | 07:27 | |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Support creating roles and assigning to service users https://review.opendev.org/c/openstack/kolla-ansible/+/774452 | 08:25 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Add missing cloudkitty documentation. https://review.opendev.org/c/openstack/kolla-ansible/+/805650 | 08:40 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Sync documentation with kolla_openstack_custom_config https://review.opendev.org/c/openstack/kayobe/+/808919 | 09:36 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/wallaby: Use ansible_facts to reference facts https://review.opendev.org/c/openstack/kayobe/+/808927 | 10:49 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/victoria: Use ansible_facts to reference facts https://review.opendev.org/c/openstack/kayobe/+/808931 | 11:18 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/ussuri: Use ansible_facts to reference facts https://review.opendev.org/c/openstack/kayobe/+/808932 | 11:22 |
| *** amoralej is now known as amoralej|lunch | 12:34 | |
| *** amoralej|lunch is now known as amoralej | 13:20 | |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/victoria: Use ansible_facts to reference facts https://review.opendev.org/c/openstack/kayobe/+/808931 | 13:25 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/ussuri: Use ansible_facts to reference facts https://review.opendev.org/c/openstack/kayobe/+/808932 | 13:26 |
| opendevreview | Merged openstack/kayobe stable/ussuri: CI: Fix get_logs.sh docker logs output https://review.opendev.org/c/openstack/kayobe/+/808163 | 13:43 |
| mgoddard | headphoneJames: hi | 14:15 |
| mgoddard | headphoneJames: I added some notes to the keystone scope patch about how to split it up. Let me know your thoughts | 14:17 |
| mnasiadka | mgoddard: I might be a couple minutes late for tomorrow meeting - would you be able to start it if I won't make it to the beginning? | 14:45 |
| mgoddard | mnasiadka: sure | 14:55 |
| ignaziocassano | Hello, please take a look to this bug: https://bugs.launchpad.net/kolla-ansible/+bug/1943532 it does not seem a netapp bug but a kolla bug | 15:13 |
| mnasiadka | ignaziocassano: Can you share more about the config? what is in /var/lib/cinder/mnt and why? | 15:15 |
| ignaziocassano | mnasiadka: please let me to specify that the netapp driver we are using is cinder.volume.drivers.netapp.common.NetAppDriver. So it uses nfs. If net netapp root volume has the .snapshot visibility enabled, it hows .snapshot directories . Those are readaonly | 15:19 |
| ignaziocassano | mnasiadka: /var/lib/cinder does not exist | 15:22 |
| mnasiadka | ignaziocassano: it does exist in the container image, you are probably mounting something in /var/lib/cinder/mnt - or cinder is doing that. | 15:23 |
| ignaziocassano | mnasiadka: in docker cinder_volume container in /var/lib/cinder/mnt I have 55fcb53b884ca983ed3d6fa8aac57810 and under that all volumes I created.But it is possible only when snapshot visibility is disabled, otherwise cinder_volume remains in restarting | 15:26 |
| ignaziocassano | mnasiadka: netapp tested on devstack installation and the did not get this issue | 15:27 |
| mnasiadka | ignaziocassano: do you have cinder_extra_volumes or cinder_volume_extra_volumes set? | 15:28 |
| ignaziocassano | mnasiadka:I do not know what do you mean ....where I should set them ? | 15:30 |
| mnasiadka | ignaziocassano: in globals.yml | 15:31 |
| mnasiadka | I'm asking if you have set them, we are not mounting anything in /var/lib/cinder/mnt in cinder_volume container - at least i'm not aware | 15:32 |
| mnasiadka | Maybe the Netapp Driver does that | 15:32 |
| ignaziocassano | mnasiadka: In my cinder.conf I do not have those parameters | 15:34 |
| mnasiadka | Ok, just share your kolla-ansible globals.yml and cinder.conf in the bug | 15:34 |
| ignaziocassano | mnasiadka: If you want I cn check if netapp driver support them it I've never seen them | 15:35 |
| ignaziocassano | mnasiadka: ok, we will share what you requested | 15:38 |
| ignaziocassano | mnasiadka: I added information you requested, hope this can help | 15:53 |
| mnasiadka | ignaziocassano: will look into that later/tomorrow | 15:56 |
| ignaziocassano | mnasiadka : thanks a lot | 15:59 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/victoria: Use ansible_facts to reference facts https://review.opendev.org/c/openstack/kayobe/+/808931 | 16:14 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Fix typo in inspector.yml https://review.opendev.org/c/openstack/kayobe/+/808996 | 16:17 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Rocky Linux https://review.opendev.org/c/openstack/kolla/+/809001 | 17:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Rocky Linux https://review.opendev.org/c/openstack/kolla/+/809001 | 17:10 |
| mloza | hello, can kolla-ansible configure multi cinder backend ? | 21:19 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Use traditional interface naming scheme in IPA ramdisk https://review.opendev.org/c/openstack/kayobe/+/809015 | 21:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!