| *** dviroel|afk is now known as dviroel | 00:00 | |
| *** dviroel is now known as dviroel|out | 00:24 | |
| *** dviroel|out is now known as dviroel | 00:59 | |
| *** ysandeep|rover|out is now known as ysandeep|rover | 01:19 | |
| *** dviroel is now known as dviroel|out | 01:24 | |
| *** ysandeep|rover is now known as ysandeep|afk | 02:26 | |
| *** ysandeep|afk is now known as ysandeep|rover | 04:44 | |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 05:57 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 06:15 | |
| jrosser | damiandabrowski[m]: is this no longer needed? https://github.com/openstack/openstack-ansible-os_tempest/commit/601db553f8112f6b52cf83e05f1fa935aa6491f6#diff-de116a0a771031e1702f071b78355ecf027deec07ef7d92ae3d5cc5395456953L127-L140 | 06:54 |
|---|---|---|
| jrosser | i was just looking at this https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/831640 | 06:54 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Return Erlang distribution port mgmt binding https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/830151 | 06:57 |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 07:50 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 08:03 | |
| opendevreview | Merged openstack/openstack-ansible-repo_server stable/victoria: Use /run/nginx.pid https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/836595 | 08:25 |
| admin1 | morning | 08:34 |
| damiandabrowski[m] | hey | 08:47 |
| damiandabrowski[m] | jrosser: I've removed user&role creation because I couldn't find any place where these users are used. Do You see any? | 08:48 |
| jrosser | it is possible that we do not run particularly heavy heat tests in our os_heat role | 08:49 |
| jrosser | the patch for the heat stack user was from tripleo and they almost certainly run much more extensive tests for heat | 08:49 |
| jrosser | we need to decide what to do with centos-9 | 09:08 |
| damiandabrowski[m] | hmm, tempest user creation was introduced 7 years ago by rackspace: https://opendev.org/openstack/openstack-ansible-os_tempest/commit/eaa4d699582f1a28e1be2258dd70a13d7f8170e7 | 09:09 |
| damiandabrowski[m] | so idk, maybe let's wait for the answer in https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/831640 ? | 09:10 |
| jrosser | still no cloudsmith packages for rabbitmq, and no downloads.mariadb.com for el9 either | 09:10 |
| jrosser | and the glusterfs stuff is needed becasue of no lsyncd there | 09:11 |
| admin1 | is inventory/vars/ssl.yml variabels like openstack_pki_service_intermediate_cert_name also overridable via user_variables ? | 09:50 |
| damiandabrowski[m] | admin1: yeah | 10:02 |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 10:03 | |
| jrosser | admin1: the idea is that you can override all of those openstack_pki_* variables | 10:05 |
| damiandabrowski[m] | user_*.yml is passed via '-e': https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/openstack-ansible.sh#L62 | 10:05 |
| damiandabrowski[m] | and -e has higher priority than group_vars according to this: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#understanding-variable-precedence | 10:05 |
| jrosser | hopefully from the strings there it is clear that those are placeholders and it's almost certain you want your own data in the certificates | 10:05 |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 10:34 | |
| *** dviroel_ is now known as dviroel | 11:33 | |
| opendevreview | Merged openstack/openstack-ansible master: Check for requirements file to verify repo health https://review.opendev.org/c/openstack/openstack-ansible/+/840651 | 11:40 |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 12:01 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 12:28 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Add support for centos-9 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/823983 | 13:20 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Add support for centos-9 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/823985 | 13:29 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use glusterfs to synchronise repo server contents https://review.opendev.org/c/openstack/openstack-ansible/+/837589 | 13:37 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use glusterfs to synchronise repo server contents https://review.opendev.org/c/openstack/openstack-ansible/+/837589 | 13:37 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - Centos-9 Stream support https://review.opendev.org/c/openstack/openstack-ansible/+/823417 | 13:38 |
| jrosser | noonedeadpunk: looks like unexpected side effects from this https://github.com/openstack/openstack-ansible/commit/d9636762e2dc2a16e33535b645cd98be1f651552 | 13:42 |
| jrosser | https://zuul.opendev.org/t/openstack/build/db19702ef3e74b8d9fdaed529caccb5c/log/job-output.txt#11310-11312 | 13:43 |
| jrosser | ooooh healthcheck-infrastructure.yml only runs for an infra scenario job | 13:49 |
| noonedeadpunk | oh, yes.... | 13:56 |
| noonedeadpunk | ah, ok. yes, my bad indeed :( | 13:57 |
| *** ysandeep|rover is now known as ysandeep|rover|mtg | 14:00 | |
| noonedeadpunk | Will fix now | 14:00 |
| noonedeadpunk | (just in case if was also failing but in different way) | 14:01 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Ensure requirements SHA variable is included for healthcheck playbook https://review.opendev.org/c/openstack/openstack-ansible/+/841595 | 14:02 |
| jrosser | oh i just tried this ^ | 14:02 |
| jrosser | noonedeadpunk: ^ | 14:03 |
| noonedeadpunk | oh awesome | 14:03 |
| noonedeadpunk | that's really good idea to add that job | 14:03 |
| noonedeadpunk | I love it | 14:03 |
| jrosser | the name is a bit sad - do you have a better idea | 14:05 |
| jrosser | somehow very generic | 14:05 |
| noonedeadpunk | maybe openstack-ansible-infra_health ? | 14:07 |
| noonedeadpunk | it's also bad... | 14:08 |
| noonedeadpunk | and that would be parsed as scenario I believe | 14:08 |
| jrosser | maybe its ok as it is | 14:09 |
| jrosser | easy to change later | 14:09 |
| noonedeadpunk | yeah, it's fine likely | 14:09 |
| jrosser | oh wait it's not going to work is it | 14:10 |
| noonedeadpunk | Another thing I was thinking if we should have parented from openstack-ansible-deploy-aio-infra and just define nodeset there.... But doesn't matter I guess | 14:10 |
| jrosser | becasue the job name is the wrong format | 14:10 |
| noonedeadpunk | I wonder if it should be in project anyway? | 14:11 |
| jrosser | well which project :) | 14:12 |
| jrosser | oh you mean project.yml | 14:12 |
| noonedeadpunk | yup :) | 14:12 |
| jrosser | even though its a job? | 14:13 |
| jrosser | oh it has been a long day :/ i understand what you mean | 14:15 |
| noonedeadpunk | Yeah, you defined a job, but it's not in any pipeline | 14:15 |
| noonedeadpunk | and then makes sense to make it a template and do both ubuntu/centos maybe ?:) | 14:15 |
| noonedeadpunk | But I think you're right about job name format as well | 14:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Ensure requirements SHA variable is included for healthcheck playbook https://review.opendev.org/c/openstack/openstack-ansible/+/841595 | 14:18 |
| jrosser | hah ok | 14:19 |
| noonedeadpunk | it actually fine that way as well | 14:20 |
| spatel | noonedeadpunk question, I believe file descriptor fix was part of 23.2.0 so wondering why we put that under 23.3.0 ?- https://docs.openstack.org/releasenotes/openstack-ansible/wallaby.html | 14:29 |
| noonedeadpunk | spatel: well, generally if there's feature release note - that's the reason for minor release. | 14:30 |
| spatel | last week i upgraded to 23.2.0 :) and now reading this so curious i should go to 23.3.0? | 14:30 |
| jrosser | it is because we put text in the 'feature' section of a releasenote on a stable branch | 14:32 |
| jrosser | then the release team suggest we should increment the middle version | 14:32 |
| spatel | +1 | 14:34 |
| noonedeadpunk | spatel: I don't see pthread being part of 23.2.0 | 14:35 |
| spatel | ? | 14:35 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_cinder/commit/d9c6359b02aaa695fe767895fcf5c5dce2a254e2 | 14:36 |
| spatel | I was running 23.0.0 earlier and had lots of issues related oslo, agents was dropping connection randomly then when i upgrade to 23.2.0 and its been almost month and i didn't see any single connection drop. | 14:37 |
| spatel | I think i talked to you about that few weeks ago. | 14:37 |
| spatel | you suggested go with 23.2.0 (before 23.3.0 came out) | 14:37 |
| spatel | I believe there was a bug in amqp lib or something causing random connection drop | 14:38 |
| jrosser | doing the upgrade would have never fixed that | 14:38 |
| jrosser | for 23.2.0 you could have made a config override i think | 14:38 |
| jrosser | and for 23.3.0 we add a variable as convenience | 14:39 |
| jrosser | but you have to make some setting or other either way | 14:39 |
| spatel | i didn't do anything.. let me find bug which i submitted | 14:39 |
| jrosser | if we are talking about the same thing? | 14:40 |
| spatel | all i did just minor upgrade and my issue got resolved | 14:40 |
| spatel | I am talking about agent dropping connect to rabbitMQ | 14:40 |
| jrosser | there was also a requirements update for aqmp library? | 14:40 |
| andrewbonney | You may have had a partial fix in 23.2.0. The amqp library bump happened in openstack requirements/constraints | 14:40 |
| jrosser | spatel: ^ this upgraded library is only partial fix | 14:40 |
| spatel | hmm | 14:41 |
| spatel | whatever it was but it helps me a lot, otherwise every single day i was restarting agents | 14:41 |
| jrosser | andrewbonney is correct - these rabbitmq troubles are in two parts | 14:42 |
| jrosser | one to do with the amqp library version that was moved forward in 23.3.0 | 14:42 |
| jrosser | and the other to do with the pthreads setting | 14:43 |
| spatel | This is what hurting me - https://bugs.launchpad.net/nova/+bug/1968054 | 14:45 |
| spatel | As you said - https://opendev.org/openstack/requirements/commit/887d45e86550bd5ffd25692f61063f78f85d7a2c | 14:46 |
| spatel | it was amqp===5.0.8 | 14:46 |
| spatel | i am not aware of any pthred related issue or i didn't hit yet | 14:47 |
| spatel | what is the story about pthreads setting ?? | 14:48 |
| jrosser | https://bugs.launchpad.net/oslo.messaging/+bug/1949964 | 14:48 |
| jrosser | there are two things that seem to result in file descriptor leaks | 14:49 |
| jrosser | and ultimatley agents breaking | 14:49 |
| spatel | is this a solution? heartbeat_in_pthread = False | 14:51 |
| andrewbonney | Yes, that's what is done by https://opendev.org/openstack/openstack-ansible-os_cinder/commit/d9c6359b02aaa695fe767895fcf5c5dce2a254e2 and similar patches for other services which occur in 23.3.0 | 14:53 |
| spatel | ah!! so i should upgrade to 23.3.0 then | 14:53 |
| jrosser | if that is a hassle you can do it with config overrides today in your user_variables | 14:55 |
| spatel | This setting should be apply to all service correct? neutron/nova/ etc.. | 14:55 |
| jrosser | the only thing you will get from 23.3.0 is the convenience of an already defined variable to set that per role, or everywhere | 14:55 |
| jrosser | well neutron/nova/cinder anyway | 14:56 |
| andrewbonney | We only patched nova/neutron/cinder as that's where the issues were noted. We also only applied it to the services which don't use uwsgi | 14:56 |
| spatel | Got it.. let me prepare for this upgrade now. | 14:56 |
| spatel | In that notes we should close this bug (because its still saying "New" and "Undecided") - https://bugs.launchpad.net/oslo.messaging/+bug/1949964 | 14:57 |
| jrosser | well, that is an oslo.messaging bug, not an openstack-ansible one | 14:59 |
| jrosser | and sadly no-one with responsibility for oslo seems to have commented | 14:59 |
| noonedeadpunk | magnum also needs that just i ncase | 15:23 |
| noonedeadpunk | on Y it's fixed in magnum though, so haven't created patch for that... | 15:23 |
| * jrosser wonders what has happened to the Storage SIG for centos-9 stream | 15:27 | |
| jrosser | this might be all very awkward on the glusterfs front | 15:27 |
| jrosser | they have a repo called "resilientstorage" which seems to contain pretty much everything except stuff to do with storage | 15:30 |
| jrosser | NeilHanlon: i don't suppose you have any clues on where i might find the storage sig repo for centos-9 ? | 15:31 |
| noonedeadpunk | uh.... how annoying that is.... | 15:31 |
| * NeilHanlon checks notes | 15:32 | |
| noonedeadpunk | eventually insteresting thing to test with gluster would be stacking different os/versions together.... | 15:32 |
| NeilHanlon | I want to say I recall something about it becoming default in 9 | 15:32 |
| * noonedeadpunk hopes Storage SIG were not same ppl who worked on ceph-ansible | 15:32 | |
| noonedeadpunk | (ie "deprecated") | 15:33 |
| NeilHanlon | http://mirror.stream.centos.org/SIGs/9-stream/storage/x86_64/ | 15:33 |
| NeilHanlon | i was thinking about advanced virtualization and 8.6 | 15:34 |
| NeilHanlon | which, incidentally, we'll need to work on for rocky and c8s... i'll be able to take a look at that next week I hope | 15:35 |
| mgariepy | noonedeadpunk, https://github.com/ceph/ceph-ansible --> ceph-ansible -- DEPRECATED -- | 15:38 |
| jrosser | NeilHanlon: ah cool - there used to be a centos-release-gluster8 but it looks like i have to add the repo manually now? | 15:38 |
| noonedeadpunk | mgariepy: yup, I know | 15:41 |
| noonedeadpunk | or you think we should fork it inside osa ?:) | 15:42 |
| jrosser | i was thinking about that | 15:42 |
| NeilHanlon | hmm. let me look into that jrosser, i would expect them to be provided in some extras repo | 15:43 |
| noonedeadpunk | at least it's under apache 2 license as well... | 15:43 |
| jrosser | noonedeadpunk: there is heaps and heaps of code in ceph-ansible that we never use, as it already had some docker container stuff in there | 15:43 |
| mgariepy | i'm not sure we can affort to maintaint a fork. | 15:43 |
| jrosser | i expect we could cut 75% of the code out and be left with just apt/yum and config_template | 15:43 |
| noonedeadpunk | Yeah, I'm closer to that ^ | 15:43 |
| mgariepy | anyone tried cephadm ? | 15:44 |
| * noonedeadpunk was going to point on mgariepy statement :) | 15:44 | |
| mgariepy | haha :) | 15:44 |
| jrosser | yeah, though i agree that supporting even more stuff is the opposite direction to where we need to be :) | 15:45 |
| jrosser | i was reading about cephadm a bit, but never tried it | 15:45 |
| jrosser | we would have to make docker/lxc co-exist | 15:45 |
| mgariepy | seems to be all docker | 15:45 |
| mgariepy | :sick: | 15:45 |
| jrosser | and by default the first mon becomes another "deploy" node | 15:45 |
| mgariepy | i have one docker running in a lxd container. | 15:46 |
| jrosser | this is why i was also thinking about gutting ceph-ansible for OSA | 15:46 |
| mgariepy | but.. well not really fun :/ | 15:47 |
| jrosser | as the effort to make everything-that-osa-is-not suddenly work in an osa deployment seems also pretty high | 15:47 |
| mgariepy | gutting it only to do CI stuff ? | 15:47 |
| noonedeadpunk | we have here had good old fight if we should have used ceph-ansible or cephadm... | 15:47 |
| noonedeadpunk | And since ceph-ansible got deprecated I lost it :) | 15:47 |
| jrosser | what people do for external clusters kind of doesnt matter | 15:47 |
| mgariepy | i did only 1 deploy with integrated with osa. then i discovered it was a lot easier to manage it on the side. | 15:48 |
| noonedeadpunk | well ceph-ansible now has quite simple playbook that would deploy cephadm and start dummy cluster | 15:48 |
| jrosser | personally i do not look forward to when we get people here who are doing "my first openstack" with OSA combined with cephadm | 15:48 |
| jrosser | trying to help out / debug that will be pretty tough | 15:49 |
| noonedeadpunk | https://github.com/ceph/ceph-ansible/blob/master/infrastructure-playbooks/cephadm.yml | 15:49 |
| jrosser | many uses of command: | 15:50 |
| noonedeadpunk | while I kind of like idea of forking ceph-ansible and maintaining it in some state, I'm afraid at same time about amount of work that would needs to be done to support new ceph releases... | 15:50 |
| jrosser | indeed | 15:50 |
| noonedeadpunk | I guess there was no reason to write collection as they were gonna deprecate the thing anyway | 15:52 |
| mgariepy | https://github.com/alvistack/ansible-collection-ceph | 15:55 |
| *** ysandeep|rover|mtg is now known as ysandeep|rover | 15:56 | |
| NeilHanlon | jrosser: looks like there are centos-release-gluster9 and centos-release-gluster10 meta packages available to provide those repos; No gluster8 for c9s, though | 16:07 |
| jrosser | NeilHanlon: hmmm Error: Unable to find a match: centos-release-gluster9 | 16:07 |
| jrosser | i wonder where that is | 16:07 |
| jrosser | i see that for 8-stream but not 9 | 16:09 |
| NeilHanlon | I believe it should be in Extras-Common | 16:09 |
| noonedeadpunk | hm.... | 16:09 |
| NeilHanlon | https://paste.opendev.org/show/bQxbvEgyjSsH6dznEtyy/ | 16:09 |
| jrosser | i wonder if i have a very old cloud image here | 16:10 |
| NeilHanlon | that is possible. it appears there are two .repo files in c9s, a centos.repo and a centos-addons.repo. the -addons one provides the extras | 16:11 |
| jrosser | yes that was it | 16:18 |
| jrosser | i ended up with centos-addons.repo.rpmnew | 16:18 |
| jrosser | NeilHanlon: thanks for the tips - i'm making some progress now | 16:27 |
| *** ysandeep|rover is now known as ysandeep|rover|out | 16:28 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Pass valid cert regen variable to pki role https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/841617 | 16:37 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Add upgrade path from lsyncd to shared filesystem. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/839411 | 16:39 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Remove all code for lsync, rsync and ssh https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837588 | 16:39 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Clean up legacy lsycnd, rsync and ssh key config https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837859 | 16:39 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Use the same vars file for all versions of centos https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/841618 | 16:39 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Use distro packages for nginx on centos. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/841619 | 16:39 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add support for centos-9 https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/841620 | 16:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - Centos-9 Stream support https://review.opendev.org/c/openstack/openstack-ansible/+/823417 | 16:45 |
| opendevreview | Merged openstack/ansible-role-python_venv_build master: Split venv_rebuild functionality https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/773984 | 16:59 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Add support for centos-9 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/823985 | 17:28 |
| jrosser | mgariepy: jamesdenton damiandabrowski[m] we broke some stuff :( need to merge this https://review.opendev.org/c/openstack/openstack-ansible/+/841595 | 17:40 |
| damiandabrowski[m] | checking | 17:41 |
| jrosser | good example there of how to conditionally run a particular zuul job only when specific files are modified | 17:41 |
| damiandabrowski[m] | ouh, looks like mgariepy was faster :D | 17:48 |
| mgariepy | sorry | 17:58 |
| noonedeadpunk | jrosser: I think we'd need same for healthcheck-hosts.yml | 18:32 |
| noonedeadpunk | as well | 18:33 |
| jrosser | i think we will | 18:34 |
| jrosser | i thought about that but decided we should do a seperate patch | 18:34 |
| noonedeadpunk | yes, totally | 18:34 |
| jrosser | so this says centos-9 is supported https://www.rabbitmq.com/install-rpm.html#cloudsmith | 18:40 |
| jrosser | but i am having repo-blindness today and can't see it :( | 18:41 |
| noonedeadpunk | don't see either.... | 18:43 |
| noonedeadpunk | not even in packagecloud | 18:45 |
| jrosser | oh there is instrutions on that page for centos9 | 18:46 |
| jrosser | it is very confusing | 18:47 |
| noonedeadpunk | But actually script should not work | 18:48 |
| noonedeadpunk | As it should retrieve repo from here https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/config_file.repo?os=centos&dist=9&source=script | 18:48 |
| noonedeadpunk | So that's super confusing indeed | 18:49 |
| jrosser | "This example assumes the CentOS Stream 8 version of the package, suitable for Red Hat 8, CentOS Stream 9, CentOS Stream 8 and modern Fedora releases." | 18:49 |
| jrosser | wtf | 18:49 |
| noonedeadpunk | oh.... | 18:49 |
| noonedeadpunk | that explains everything.... | 18:49 |
| noonedeadpunk | WHich is super dumb at same time.... | 18:49 |
| noonedeadpunk | I should have read carefully... | 18:50 |
| jrosser | ok, so we probably need to adjust this https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/master/vars/redhat.yml#L22 | 18:51 |
| jrosser | to make it just be '8' | 18:51 |
| jrosser | with a REALLY BIG COMMENT :) | 18:52 |
| mgariepy | or add if major == 9 do major - 1 :P | 18:52 |
| jrosser | maybe is rabbitmq basically depends on erlang, and erlang on libc or something, then it's pretty portable | 18:53 |
| jrosser | yes thats how it seems to be rabbitmq-server has almost no dependancies except erlang | 18:55 |
| jrosser | and erlang only really wants the libc/libstdc++ and systemd | 18:56 |
| noonedeadpunk | yup, I guess comment is what we need here.... | 19:03 |
| noonedeadpunk | to revise also in the future, as I bet that things will drift one day, considering how stream is developed | 19:03 |
| jrosser | the good news is that my centos-9 job has got through all the repo / gluster stuff and this is the next thing that breaks | 19:04 |
| noonedeadpunk | btw centos 8 no longer supported in Zed | 19:07 |
| noonedeadpunk | Well, at least because of py3.6 being dropped | 19:07 |
| noonedeadpunk | and ppl replace el8 jobs with el9 right now | 19:08 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Add support for centos-9 https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/823985 | 19:13 |
| jrosser | yes i saw some of that | 19:13 |
| jrosser | we kind of have no overlap really | 19:13 |
| jrosser | i still don't know what the deal is with lxc for centos-9 | 19:13 |
| opendevreview | Merged openstack/openstack-ansible master: Ensure requirements SHA variable is included for healthcheck playbook https://review.opendev.org/c/openstack/openstack-ansible/+/841595 | 20:08 |
| admin1 | tag 4.2.0 .. that SSL haproxy bug still exists .. where haproxy setup fails 3 times .. one for each contoller and finally works on the 4th try | 20:22 |
| admin1 | i am going to document each tries and open a bug report | 20:22 |
| admin1 | https://bugs.launchpad.net/openstack-ansible/+bug/1973242 | 20:42 |
| admin1 | 24.2.0 .. i have a blocker .. on multi controllers, keepalived starts on none .. error is => r2c1 Keepalived_vrrp[130901]: (Line 34) *** Configuration line starting `auth_pass` is missing a parameter after keyword `auth_pass` at word position 2 | 20:47 |
| admin1 | something changed | 20:47 |
| jrosser | admin1: you have all the repos, you can see whats changed | 20:54 |
| jrosser | you can see the template for that config file here https://github.com/evrardjp/ansible-keepalived/blob/master/templates/keepalived.conf.j2#L114 | 20:56 |
| jrosser | what do you have? | 20:56 |
| jrosser | here is the value that should be set https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/haproxy/keepalived.yml#L62 | 20:58 |
| jrosser | which comes from user_secrets https://opendev.org/openstack/openstack-ansible/src/branch/master/etc/openstack_deploy/user_secrets.yml#L201 | 20:59 |
| jrosser | admin1: do you have your user_secrets setup properly? | 20:59 |
| admin1 | i do | 21:02 |
| jrosser | so, does the value of haproxy_keepalived_authentication_password appear in your keepalived config file as auth_pass? | 21:03 |
| admin1 | i will redo the user_secrets | 21:03 |
| admin1 | how about the other keepalived bug .. it has been since a few releases .. i think from the time we moved to pki | 21:04 |
| jrosser | well i see errors from keepalived there | 21:04 |
| admin1 | cat: /etc/haproxy/ssl/haproxy_r2c1-172.29.236.9-ca.crt: No such file or directory"] | 21:04 |
| jrosser | imho you have not shown what the actual error is | 21:05 |
| jrosser | handlers run at the end of the play | 21:05 |
| jrosser | my guess is that the actual error is before what you've pasted into the bug report | 21:06 |
| admin1 | i have a new build next week .. i will try to do a full log that time | 21:06 |
| admin1 | this one, i just recorded the error outputs | 21:06 |
| jrosser | i'm trying to say that there would have been a previously "failed" task | 21:07 |
| jrosser | but becasue handlers always run at the end of the play, they are running and also failing | 21:07 |
| jrosser | but the root cause is earlier in the log | 21:07 |
| admin1 | i get it | 21:07 |
| admin1 | i ran it inside tmux :( | 21:07 |
| jrosser | you have the ansible log in /openstack | 21:07 |
| admin1 | will you be in the summit btw ? | 21:08 |
| jrosser | anyway, i bould believe that there is some issue with the PKI role changes and user supplied certificates | 21:08 |
| admin1 | or still undecided .. | 21:08 |
| jrosser | *could | 21:08 |
| jrosser | but that keepalived thing is totally unrelated to pki role and you should check through those variables i showed you | 21:09 |
| *** dviroel is now known as dviroel|afk | 21:09 | |
| jrosser | the keepalived role and haproxy role are pretty tightly coupled becasue they run in the same playbook | 21:10 |
| jrosser | so if the auth_pass stuff goes wrong in keepalived i could also see giving you those certificate based errors from your LP bug | 21:11 |
| admin1 | thanks jrosser | 21:24 |
| *** prometheanfire is now known as Guest0 | 22:26 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!