| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: Remove service-specific tags from service playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/918615 | 07:44 |
|---|---|---|
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: docs: demonstrate quick method to move between HA/Quorum queues https://review.opendev.org/c/openstack/openstack-ansible/+/919062 | 07:44 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_keystone master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/919670 | 08:14 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_barbican master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/919671 | 08:14 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_placement master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/919672 | 08:15 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_glance master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/919673 | 08:16 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_cinder master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/919674 | 08:16 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_heat master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/919675 | 08:16 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_horizon master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/919676 | 08:16 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_designate master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/919677 | 08:16 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_swift master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/919678 | 08:16 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_adjutant master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/919679 | 08:16 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_gnocchi master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/919680 | 08:16 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_ceilometer master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/919681 | 08:17 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_aodh master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/919682 | 08:17 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_cloudkitty master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/919683 | 08:17 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_ironic master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/919684 | 08:17 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_magnum master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/919685 | 08:17 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_trove master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/919686 | 08:17 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_octavia master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/919687 | 08:17 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_tacker master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/919688 | 08:17 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_blazar master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/919689 | 08:18 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_masakari master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/919690 | 08:18 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_manila master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/919691 | 08:18 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_mistral master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/919692 | 08:18 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/919693 | 08:19 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_skyline master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/919694 | 08:19 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_nova master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/918614 | 08:19 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_neutron master: Add tag to enable targeting of post-install config elements only https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/919695 | 08:21 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: Remove service-specific tags from service playbooks https://review.opendev.org/c/openstack/openstack-ansible/+/918615 | 08:27 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: docs: demonstrate quick method to move between HA/Quorum queues https://review.opendev.org/c/openstack/openstack-ansible/+/919062 | 08:27 |
| semantic | Just tried to install from master. Basic function work, but glance-api.service logs python trace ending with 'ERROR oslo_messaging.notify.messaging RuntimeError: Configuration Error: rabbit_stream_fanout need rabbit_qos_prefetch_count to be set to a value greater than 0.' | 08:43 |
| andrewbonney | Might be worth fetching the latest changes. A patch to os_glance merged at half 5 yesterday: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/919087 | 08:48 |
| andrewbonney | Unless you have done and there's some issue with that patch | 08:49 |
| semantic | Ah, I see, thank you. Probably need to fetch latest then. | 08:53 |
| noonedeadpunk | interestingly, mainly 2023.1 upgade jobs are timing out. | 08:55 |
| noonedeadpunk | what have we changed between 2023.1 and 2023.2 that can explain performance hit.... | 08:56 |
| noonedeadpunk | but then... looking at some jobs - openstack-ansible-upgrade-aio_metal-ubuntu-jammy 2h 21m 21s | 08:57 |
| noonedeadpunk | openstack-ansible-upgrade_2023.1-aio_metal-ubuntu-jammy 2h 21m 31s | 08:58 |
| noonedeadpunk | so super close.... | 08:58 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Deploy horizon by default with metal AIO scenarios https://review.opendev.org/c/openstack/openstack-ansible/+/916005 | 09:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Deploy horizon by default with metal AIO scenarios https://review.opendev.org/c/openstack/openstack-ansible/+/916005 | 09:14 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-rabbitmq_server master: Enable feature flags pre and post-upgrade https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/919701 | 09:34 |
| jrosser_ | so for manila i wonder where it tries to get nfs-ganesha-ceph=4.4-ubuntu1~jammy1 from | 09:34 |
| jrosser_ | noonedeadpunk: ^ did you try any of this locally - i will if you've not? | 09:35 |
| noonedeadpunk | yeah. I was going to make an AIO to check. | 09:35 |
| noonedeadpunk | created VM but not yet started | 09:36 |
| jrosser_ | it kind of looks like a UCA-ish package | 09:36 |
| noonedeadpunk | it looks like there's no ganesha in community repo anymore | 09:36 |
| jrosser_ | the package version there does not make sense for https://packages.ubuntu.com/jammy/nfs-ganesha-ceph | 09:36 |
| noonedeadpunk | but well - why then UCA requires dependency that is only available on focal.... | 09:37 |
| noonedeadpunk | though, it doesn't contain dependency on liburcu at all? | 09:38 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-rabbitmq_server master: Enable feature flags pre and post-upgrade https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/919701 | 09:38 |
| noonedeadpunk | ah, nfs-ganesha does | 09:38 |
| noonedeadpunk | but then it's liburcu8 https://packages.ubuntu.com/jammy-updates/nfs-ganesha | 09:38 |
| jrosser_ | i still don't understand the version 4.4 at all | 09:39 |
| jrosser_ | tbh i wonder how they even build the cephadm images for this | 09:40 |
| noonedeadpunk | cephadm images are only centos9s? | 09:41 |
| jrosser_ | oh sure, but it's a pretty weird set of distros have the nfs ganesha stuff at all https://pkgs.org/search/?q=nfs-ganesha-ceph | 09:42 |
| jrosser_ | so you kind of need a package (ideally) to install from when building the container image | 09:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update cirros image for manila tempest https://review.opendev.org/c/openstack/openstack-ansible/+/919702 | 09:48 |
| noonedeadpunk | yeah, true.... | 09:48 |
| noonedeadpunk | but that also shows more or less native repos? | 09:49 |
| noonedeadpunk | as ceph could be coming from some StoreSIG | 09:49 |
| noonedeadpunk | https://sigs.centos.org/storage/ | 09:49 |
| noonedeadpunk | which all consist of broken links.... | 09:50 |
| jrosser_ | https://github.com/ceph/ceph-container/blob/d4a0f93b06679a5c9148c9e9b367478b2a0d04e0/ceph-releases/ALL/centos/daemon-base/__DOCKERFILE_INSTALL__#L14-L19 | 09:52 |
| noonedeadpunk | so we should be looking for ganesha 5 | 09:55 |
| noonedeadpunk | but /o\, how I don't understand Docker and why ppl like it | 09:56 |
| noonedeadpunk | Like I close to never heard anything too adorable about bash being handy and convenient for developing things. But somehow Docker is... | 09:57 |
| noonedeadpunk | nayway | 09:57 |
| noonedeadpunk | just every time I look at Dockerfiles I can't stop thinking about "why" | 09:57 |
| noonedeadpunk | ok, got AIO failure | 10:17 |
| noonedeadpunk | https://paste.openstack.org/show/btKnImhzDrWqYOk5GBZE/ | 10:18 |
| noonedeadpunk | so it's not even UCA | 10:18 |
| noonedeadpunk | probably... worth trying https://launchpad.net/~nfs-ganesha/+archive/ubuntu/nfs-ganesha-5 instead... | 10:19 |
| noonedeadpunk | yeah, switching to 5 works | 10:21 |
| noonedeadpunk | also requires https://answers.launchpad.net/~nfs-ganesha/+archive/ubuntu/libntirpc-5/ | 10:22 |
| noonedeadpunk | I guess it's ceph-ansible that does provide these... | 10:22 |
| noonedeadpunk | (or well, install) | 10:22 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Add variable to globally control notifications enablement https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/918115 | 10:47 |
| jrosser_ | hmm looks like the ceph-nfs role is removed? | 10:50 |
| noonedeadpunk | in latest? | 10:50 |
| noonedeadpunk | doh | 10:50 |
| noonedeadpunk | it's not on the branch we're using yet | 10:51 |
| jrosser_ | https://github.com/ceph/ceph-ansible/commit/675667e1d60b7080dad7293f2954de23718c5042 | 10:52 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Implement variables to address oslo.messaging improvements https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/918116 | 10:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use NFS Ganesha 5 https://review.opendev.org/c/openstack/openstack-ansible/+/919714 | 10:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/898914 | 10:57 |
| noonedeadpunk | hopefully this helps... | 10:57 |
| jrosser_ | what is the status of adjutant? | 11:03 |
| jrosser_ | i dont see the messaging patches for that role | 11:03 |
| noonedeadpunk | It does not use messaging afaik | 11:10 |
| noonedeadpunk | about status - somehow maintained I guess | 11:11 |
| jrosser_ | oh well that would explaint it :) | 11:11 |
| noonedeadpunk | I tried to get it working, and it kinda did. partially... | 11:11 |
| noonedeadpunk | but I wasn't too dedicated | 11:11 |
| semantic | Ok, have reinstalled from master once again. Trying my test and facing similar problem. After i shutdown one of rabbit hosts, i cannot migrate live migrate instance from one compute host to another. nova-compute logs python trace with 'ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID fdd459b5c152476386220255539d63a8' and instance becomes 'ERROR' | 11:52 |
| semantic | statused... | 11:52 |
| semantic | Though nova-compute service now does not repeat similar log constantly but just once... | 11:52 |
| noonedeadpunk | well, it should kinda reconnect after that and retry operation... | 12:05 |
| noonedeadpunk | but it weird it does just put instance into error state.... | 12:05 |
| andrewbonney | Could you put the full log somewhere? There's a chance it's a different issue | 12:08 |
| semantic | Sure. Full log of what exactly? Nova-compute service? | 12:09 |
| andrewbonney | Yeah, any other error messages or tracebacks around the same time | 12:09 |
| jrosser_ | paste.opendev.org is good for this | 12:09 |
| semantic | https://paste.opendev.org/show/btRSEe7eP0wOwG9cj1br/ | 12:38 |
| semantic | https://paste.opendev.org/show/bTb5eJPQd2WlK7nAK0ua/ | 12:41 |
| semantic | https://paste.opendev.org/show/bOBo7mPbqfFM1Tu221mA/ | 12:42 |
| andrewbonney | I can't be certain, but that looks like what we've seen and reported in https://bugs.launchpad.net/nova/+bug/2060931 | 12:42 |
| jrosser_ | semantic: if you want to determine if you are seeing the same bug, you can add this to your user_variables.yml https://paste.opendev.org/show/b8g4PK1CoR6Kef8cXJ9r/ | 13:12 |
| jrosser_ | semantic: though the branch of oslo.messaging that points to is for bobcat, so you could create your own fork for master and revert the patch mentioned in the nova bug report | 13:13 |
| jrosser_ | the mechanism to apply a customised oslo.messaging for nova remains the same | 13:14 |
| noonedeadpunk | have you talked with nova folks on irc about that? as it seems it's been a month without any ack... | 13:28 |
| andrewbonney | I brought up three bugs yesterday. Spent most time on an online migrations one so may have to go back on that, but I'd really like to find a concrete test case as for me it takes time to occur | 13:29 |
| noonedeadpunk | I see | 13:30 |
| noonedeadpunk | I think I found another isuse with live migrations that was there forever and kinda follow-up of https://github.com/openstack/nova/commit/6a15169ed9f16672c2cde1d7f27178bb7868c41f | 13:30 |
| noonedeadpunk | but this time, when MAC contains `-` as a separator | 13:31 |
| noonedeadpunk | which is kinda okay for netaddr.EUI but not when you try to live migrate, as domain xml always contain semicolon as separator | 13:32 |
| lowercase | Good Morning, I'm encountering an issue on release `2023.1` during the `haproxy_server : | 14:08 |
| lowercase | Regenerate haproxy configuration` keeps applying `'option httpchk' : hiding headers or body at the end of the version string is deprecated. Please, consider to use 'http-check send' directive instead.` to the haproxy configuration. I know this is an issue because in haproxy 2.1 and higher, the config changes. https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/stable/2023.1/templates/service.j2 line 81 applies t | 14:08 |
| noonedeadpunk | hey | 14:10 |
| noonedeadpunk | I think, we pretty much get rid of that directive in 2023.1.... | 14:10 |
| noonedeadpunk | but this can be coming from services (or extra services) defenition | 14:11 |
| noonedeadpunk | ie: https://opendev.org/openstack/openstack-ansible/src/branch/stable/2023.1/inventory/group_vars/keystone_all/haproxy_service.yml#L24 | 14:11 |
| lowercase | Yep, that's the issue | 14:12 |
| noonedeadpunk | we probably backported some fix for the service.... | 14:12 |
| lowercase | Well tbf, and totally to make this more complex. | 14:12 |
| lowercase | you ship haproxy 2.0 for this release. | 14:12 |
| lowercase | I'm using the next lts because of some other issues we found. | 14:13 |
| lowercase | 2.4 | 14:13 |
| noonedeadpunk | we actually don't constrain proxy version. it's whatever shiped with the distro | 14:13 |
| noonedeadpunk | oh | 14:13 |
| noonedeadpunk | you mean we've adjusted that for 2023.2 only | 14:13 |
| noonedeadpunk | But I"m pretty much sure that 2023.1 should work nicely both for 20.04 and 22.04 | 14:15 |
| lowercase | Sorry, give me a moment. | 14:15 |
| noonedeadpunk | as we're having both in production right now | 14:15 |
| lowercase | I'm double checking if line 41 is the issue | 14:15 |
| lowercase | I don't think that's the issue. This is what is throwing haproxy off `reqadd X-Forwarded-Proto:\ https` | 14:17 |
| lowercase | And I modify that line to be haproxy >2.1 compliant with this line: http-request add-header X-Forwarded-Proto https | 14:17 |
| jrosser_ | lowercase: what OS do you use? btw also the end of your original message maybe truncated | 14:27 |
| lowercase | ubuntu 20.04 | 14:27 |
| lowercase | We have roadmap to take ubuntu 22.04 with Bobcat | 14:29 |
| jrosser_ | so tbh i am a bit confused here | 14:29 |
| jrosser_ | for example we have a 2023.1 job here which runs on both focal and jammy https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/917091?tab=change-view-tab-header-zuul-results-summary | 14:30 |
| jrosser_ | and those both pass / nothing terrible in the haproxy log, you can look through all the logs for those jobs | 14:30 |
| lowercase | My boss is tagging me for an issue. I will look in about a minute | 14:32 |
| jrosser_ | sure | 14:33 |
| lowercase | Sorry about that | 14:34 |
| jrosser_ | so those jobs run focal with haproxy=2.0.33-0ubuntu0.1 | 14:34 |
| lowercase | This issue isn't OS related. I took Haproxy 2.4 early due to an issue with 2.0 failing to handle the complete connection of a tcp connection. We observed the haproxy load balancer was prematurely failing to route the whole connection and dropping the connection before fin | 14:35 |
| jrosser_ | and jammy with haproxy=2.4.24-0ubuntu0.22.04.1 | 14:35 |
| jrosser_ | my point is that we are running jobs with both haproxy 2.0 and 2.4 off of the same ansible code/vars for 2023.1 | 14:36 |
| lowercase | hmm.. well that is interesting | 14:36 |
| jrosser_ | they happen to be on different OS for sure | 14:36 |
| jrosser_ | but it surprises me a bit if something is fundamentally broken if you were to put haproxy 2.4 on focal | 14:37 |
| lowercase | I think what I'm trying to communicate, is that I have focal haproxy 2.4, and the playbooks are laying down a 2.0 config. | 14:39 |
| lowercase | If the easy fix is to upgrade the os to 2.4 so the playbooks give me a 2.4 config. I'm okay with that. | 14:39 |
| jrosser_ | but they don't change though? | 14:39 |
| jrosser_ | there is nothing in the ansible code that adjusts the haproxy config written based on the version of haproxy | 14:40 |
| lowercase | The way http packet headers are added in haproxy is what changes betwen haproxy 2.0 and 2.4 | 14:40 |
| lowercase | I'll provide some documentation. | 14:40 |
| jrosser_ | please check the CI jobs | 14:41 |
| jrosser_ | you can see the config files, the haproxy logs, the haproxy config all exactly as we test it | 14:41 |
| andrewbonney | Looks like this changed well before 2023.1: https://github.com/openstack/openstack-ansible-haproxy_server/commit/ca76349e9f7c8aa9a6931222684b635a4096049c | 14:41 |
| andrewbonney | Perhaps worth checking the haproxy role isn't old? | 14:41 |
| lowercase | The dates on the haproxy role are identical to all the other roles. | 14:45 |
| lowercase | I'm still working through the ci jobs. | 14:45 |
| lowercase | I also checked ~/.ansible/* for an haproxy role as well and didn't find one | 14:45 |
| jrosser_ | is this the line you are concerned about? https://zuul.opendev.org/t/openstack/build/016ee3f4ca9f47c9b8b69a3b04fa1749/log/logs/etc/host/haproxy/haproxy.cfg.txt#45 | 14:47 |
| lowercase | and the plot thickens. | 14:47 |
| lowercase | Yes | 14:47 |
| lowercase | Okay, the issue comes from RUNNING HANDLER [haproxy_server : Regenerate haproxy configuration] *****************************************************task path: /etc/ansible/roles/haproxy_server/handlers/main.yml:37 | 14:50 |
| jrosser_ | so the haproxy role writes a whole bunch of config fragments into /etc/haproxy/conf.d/<blah> | 14:51 |
| jrosser_ | then they are all glued together to make the config file | 14:51 |
| lowercase | make sense | 14:51 |
| jrosser_ | also tbh the date on the haproxy role is not really what matters, it is the git sha that is checked out there which is critical | 14:52 |
| lowercase | * (HEAD detached at df2e7af) df2e7af Fix haproxy_stats SSL path defenition | 14:52 |
| jrosser_ | ok so that is the tip of stable/2023.1 | 14:53 |
| jrosser_ | you can check if you see the `reqadd` line in haproxy.conf, do you also see it in all the fragments in /etc/haproxy/conf.d/<...> | 14:54 |
| jrosser_ | also can i check which playbook you are running which results in `[haproxy_server : Regenerate haproxy configuration] *****************************************************task path: /etc/ansible/roles/haproxy_server/handlers/main.yml:37` | 14:55 |
| lowercase | interesting, the only o that came back with the old styl was `keystone_admin` | 14:55 |
| lowercase | - /etc/haproxy/conf.d# grep reqadd *keystone_admin: reqadd X-Forwarded-Proto:\ https | 14:56 |
| lowercase | port 35357.. im not familar with this service? | 14:57 |
| noonedeadpunk | I think it should have been dropped for a while | 14:59 |
| lowercase | It appears since Newton | 15:00 |
| lowercase | This environment can be about that age. | 15:00 |
| andrewbonney | https://github.com/openstack/openstack-ansible/commit/08dcc639eb678a167c62f14c56b0b5c76bf908c3 | 15:00 |
| noonedeadpunk | it was removed in stein from what I see | 15:01 |
| noonedeadpunk | manila failure is now more cumbersome then befre: https://877821c6012ddcb237b8-1e695c46ce38568fe2b9076122bc0b06.ssl.cf5.rackcdn.com/898914/5/check/openstack-ansible-deploy-aio_metal-ubuntu-jammy/65b132d/logs/openstack/aio1-utility/stestr_results.html | 15:02 |
| noonedeadpunk | lowercase: well, I guess I'd try to clean out /etc/haproxy/conf.d/ and re-run haproxy role | 15:02 |
| jrosser_ | woah wait | 15:03 |
| jrosser_ | 2023.1 means you need to re-run all roles for that? | 15:03 |
| noonedeadpunk | oh | 15:04 |
| noonedeadpunk | true | 15:04 |
| noonedeadpunk | I still can't get use to it... | 15:04 |
| noonedeadpunk | run setup-openstack.yml --tags haproxy-service-config | 15:05 |
| jrosser_ | it's not sufficient to just delete the out-of-date config fragment through | 15:06 |
| noonedeadpunk | oh? | 15:06 |
| jrosser_ | becasue iirc that does not trigger the handler / assemble task | 15:06 |
| noonedeadpunk | huh, I thought placing a fragment will cause change | 15:07 |
| noonedeadpunk | which will trigger assemble | 15:07 |
| jrosser_ | you have to hit this https://github.com/openstack/openstack-ansible-haproxy_server/blob/stable/2023.1/tasks/haproxy_service_config.yml#L48 | 15:07 |
| noonedeadpunk | (I was thinking to delete all fragments from /etc/haproxy/conf.d/) | 15:07 |
| jrosser_ | or line 54 | 15:07 |
| jrosser_ | ooooh sorry yes i see | 15:08 |
| jrosser_ | then re-run whole of setup-openstack with the tag, right | 15:08 |
| jrosser_ | sorry my bad | 15:08 |
| noonedeadpunk | it was ery good call about setup-openstack | 15:08 |
| jrosser_ | i also did look at manila a bit | 15:09 |
| jrosser_ | and one of the fails is from basic server ops which makes me think there is something fundamental broken with updating the tempest cirros image | 15:09 |
| noonedeadpunk | ah, yeah | 15:09 |
| jrosser_ | but that did not make much sense though | 15:11 |
| jrosser_ | one difference is the `visibility` here https://github.com/openstack/openstack-ansible-os_tempest/blob/master/vars/main.yml#L24-L30 | 15:12 |
| lowercase | Alright, nuking the /etc/haproxy/conf.d/* ended up cleaning up a lot o the errors. | 15:13 |
| lowercase | Working on the last ne. | 15:13 |
| jrosser_ | excellent! | 15:13 |
| noonedeadpunk | yeah, default is private | 15:15 |
| noonedeadpunk | which won't fly | 15:15 |
| noonedeadpunk | good catch | 15:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update cirros image for manila tempest https://review.opendev.org/c/openstack/openstack-ansible/+/919702 | 15:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use NFS Ganesha 5 https://review.opendev.org/c/openstack/openstack-ansible/+/919714 | 15:16 |
| lowercase | Wow! Thanks for keepig me going for another release | 15:19 |
| lowercase | I'm good to go. | 15:19 |
| jrosser_ | no worries - there is some cool new stuff in 2023.1 about haproxy, we introduced support for "maps" there | 15:21 |
| jrosser_ | https://www.haproxy.com/blog/introduction-to-haproxy-maps | 15:21 |
| lowercase | Wow, this could be game changing | 15:25 |
| jrosser_ | i put very very generic code into the haproxy role | 15:26 |
| jrosser_ | so i think it should be possible to construct anything that haproxy supports with maps, using the OSA vars | 15:26 |
| lowercase | I was previously really against running k8s hosting all the containers. Now with this change. I don't see why I wouldn't do that. | 15:27 |
| lowercase | I could do a whole release cycle this way | 15:27 |
| jrosser_ | so as an example, here is a regex map defined for the "base" haproxy service (this 'base' thing is a new concept too and worth understanding) https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L91-L96 | 15:28 |
| jrosser_ | and then here is a service defined for serving security.txt https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L66-L69 | 15:29 |
| jrosser_ | that declares that it wants to put an entry into the map called "base_regex" | 15:29 |
| lowercase | okay okay.. i see the pieces. | 15:32 |
| lowercase | This makes sense. | 15:32 |
| jrosser_ | you could do rate limits via maps and dynamically update them, or connect compute.example.com straight to the nova api with a map too | 15:34 |
| jrosser_ | lots of possibilities | 15:34 |
| lowercase | OH SNAP | 15:35 |
| lowercase | dude this directly solves an issue with keystone I couldn't solve. | 15:35 |
| lowercase | As you just witnessed first hand, some of these environments are old old. One of the legacy choices were that the way keystone was bound to the AD requires that we specify each user for each project. This results in a very strict method of authenticating users. We have wanted to move to group based auth, but because of the legacy method we cannot | 15:37 |
| lowercase | If I could configure a seperate instance of keystone and horizon. New keystone would bind using group objects, I could have both. and eventually move everyone off... | 15:38 |
| jrosser_ | so something that i've not yet had time to look at is some way to leverage all this maps stuff for deployments where you want all the different api on different hostnames, rather than ports | 15:40 |
| jrosser_ | and if done right it would be no problem to also have multiple keystones or horizons or whatever else | 15:41 |
| jrosser_ | (assuming of course that you don't then get a gigantic mess with service catalog etc, but thats a different matter again) | 15:43 |
| lowercase | I think each keystone would have to know about each other. i.e. use the same fennet keys. I guess im thinking about this. If keystone queries nova. Keystone places a message on the MQ. Nova picks up the message and replies. How would nova know how to reply to the correct keystone? | 15:44 |
| lowercase | Having a Blue/Green environment is easy in that everyone is self contained. But if only two services are split. How would the reverse work | 15:46 |
| lowercase | Something to look into | 15:46 |
| noonedeadpunk | 1. Kesytone does never query nova, it's vice versa | 15:47 |
| noonedeadpunk | 2. all services interact with keystone only through API - no messaging between services | 15:47 |
| noonedeadpunk | unless these are notifications | 15:47 |
| noonedeadpunk | so basically - can define a new keystone group, do integration, all kind of things, and then point haproxy frontend to it pretty much | 15:48 |
| noonedeadpunk | or do jsut some kind of ACL thingy, where only specific things go to specific backends.... | 15:50 |
| jrosser_ | i probably miss something totally obvious but you cant bind to AD twice with different settings, like having two identity providers? | 15:51 |
| jrosser_ | ^ in the same keystone | 15:51 |
| noonedeadpunk | yeah, different domains | 15:51 |
| lowercase | Well, I just discovered that when I removed all the configs in /etc/haproxy/conf.d/ it also removed all the other configs | 18:21 |
| lowercase | Any chance there is a quick method of regen all the old configs? | 18:22 |
| opendevreview | Merged openstack/openstack-ansible master: Enable rabbitmq distro installation for distro scenario https://review.opendev.org/c/openstack/openstack-ansible/+/917148 | 18:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!