noonedeadpunk | doh, gerrit got upgraded again | 06:57 |
---|---|---|
noonedeadpunk | I almost get used to 3.0 :p | 06:58 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Set infra_lxc job to centos-8-stream https://review.opendev.org/c/openstack/openstack-ansible/+/826135 | 07:07 |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd master: Add integrated linters test https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/799038 | 07:13 |
noonedeadpunk | morning everyone \o/ | 08:04 |
damiandabrowski[m] | hey! | 08:04 |
jrosser | good morning | 08:12 |
*** dviroel|out is now known as dviroel | 11:20 | |
opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd master: Add integrated linters test https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/799038 | 11:29 |
*** sshnaidm|afk is now known as sshnaidm | 12:28 | |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add ssh_keypairs role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/825113 | 13:00 |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add ssh_keypairs role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/825113 | 14:23 |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Create ssh certificate authority https://review.opendev.org/c/openstack/openstack-ansible/+/825292 | 14:23 |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Create ssh certificate authority https://review.opendev.org/c/openstack/openstack-ansible/+/825292 | 14:51 |
noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
opendevmeet | Meeting started Tue Jan 25 15:00:28 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
noonedeadpunk | #topic rollcall | 15:00 |
noonedeadpunk | \o/ | 15:00 |
damiandabrowski[m] | hey! | 15:00 |
*** sshnaidm is now known as sshnaidm|afk | 15:01 | |
jrosser | hello | 15:02 |
spatel | \o/ | 15:02 |
noonedeadpunk | #topic bug triage | 15:04 |
noonedeadpunk | we have several new bugs lately | 15:04 |
noonedeadpunk | they seem pretty simple though | 15:04 |
noonedeadpunk | admin16: sorry for stupid question, but aren't you admin0 for co-incedence ?:) | 15:05 |
noonedeadpunk | as regarding https://bugs.launchpad.net/openstack-ansible/+bug/1955676 would be great to have a descision if it's go or no go | 15:06 |
noonedeadpunk | Other then that, there's other interesting bug | 15:07 |
noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/1958645 | 15:07 |
noonedeadpunk | I'm not sure I see why comma should be added there? | 15:07 |
noonedeadpunk | as we put it inside horizon_local_settings, which is regular python code | 15:08 |
spatel | seems typo | 15:09 |
noonedeadpunk | I'm not sure it does correct thing this way though https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/templates/horizon_local_settings.py.j2#L92-L93 | 15:09 |
jrosser | huh https://opendev.org/openstack/horizon/src/branch/master/doc/source/configuration/settings.rst#openstack_keystone_domain_choices | 15:11 |
jrosser | what is that all about | 15:11 |
noonedeadpunk | I haven't used horizon for quite a while, but are choices even a required thing? | 15:11 |
noonedeadpunk | I mean - that could be even avoided by default, or...? | 15:12 |
jrosser | same thing appears to be the case for AVAILABLE_THEMES and another few things in there | 15:12 |
noonedeadpunk | `option can be used` | 15:12 |
noonedeadpunk | And if it's not defined - would all domains appear by default.... | 15:13 |
noonedeadpunk | I guess not, as default setting would prevent it I believe... | 15:13 |
jrosser | well - seems our code does not match the horizon docs either way | 15:13 |
noonedeadpunk | well that fully worked for me on T though... | 15:14 |
noonedeadpunk | But yes, agree, we need to change that | 15:15 |
jrosser | https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/horizon/templates/local_settings.j2#L102 | 15:15 |
noonedeadpunk | yeah, fair | 15:16 |
noonedeadpunk | ANother trivial bug | 15:17 |
noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/1958434 | 15:17 |
noonedeadpunk | I think that's also fair solution suggested? | 15:17 |
spatel | how does CI pass then? because we don't have that role for CI job? | 15:19 |
jrosser | probably becasue we do a metal deploy and it picks it up off the host | 15:19 |
noonedeadpunk | Yeah | 15:19 |
jrosser | theres quite a few lxc<>metal bugs occur that way | 15:19 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Fix default multidomain choises https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/826310 | 15:22 |
noonedeadpunk | #topic office hours | 15:22 |
noonedeadpunk | Oh, backport candidate label has just been merged | 15:23 |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_aodh master: Ensure libxml2 is installed on debian systems https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/826311 | 15:24 |
jrosser | do we abandon these https://review.opendev.org/q/topic:reno-ocata ? | 15:24 |
noonedeadpunk | yep | 15:25 |
noonedeadpunk | it was really nasty side effect of EOLing integrated repo... | 15:25 |
jrosser | andrewbonney: we need this for centos-8 removal https://review.opendev.org/c/openstack/openstack-ansible/+/826135 | 15:25 |
noonedeadpunk | So back in days integrated and role repos were defined in single file in releases | 15:25 |
noonedeadpunk | So roles were eoled, and I decided to eol integrated as follow-up, once a-r-r will be switched to eol tag as well | 15:26 |
noonedeadpunk | but when adjusting list, CD decided to create branch on every repo, and then delete it.... | 15:27 |
noonedeadpunk | spawning tons of rubbish | 15:27 |
noonedeadpunk | btw returning back to horizon, there's one patch that is interesting I guess https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/820928 | 15:28 |
noonedeadpunk | I tested it in aio both for distro and source with some extra panel (like heat?). Source worked nicely, but for distro iirc panel wasn't added for $reason. It actually wasn't added even with current state of role, so might be issue with panel itself that is packaged... | 15:30 |
noonedeadpunk | but not sure | 15:30 |
noonedeadpunk | And I wasn't able to take a look on CI stuff and how to re-use that efficiently during previous week | 15:31 |
jrosser | no worries | 15:32 |
jrosser | i have been out for a few days too | 15:32 |
jrosser | just managed to pick up the keypairs stuff again today | 15:32 |
noonedeadpunk | Oh, would be awesome to merge https://review.opendev.org/q/topic:integrated_lintersbtw | 15:32 |
jrosser | andrewbonney: looks like more for you ^ | 15:32 |
noonedeadpunk | Yeah, I saw you pushed changes, will hopefully review them tomorrow morning | 15:32 |
andrewbonney | :) | 15:32 |
noonedeadpunk | as well as damiandabrowski[m] to tempest | 15:33 |
jrosser | so on the keypairs stuff - i *think* it will work for nova on a new deployment, though i've only tested AIO | 15:33 |
noonedeadpunk | will it for upgraded one ?:) | 15:33 |
jrosser | there is code to undo all the old config, but i need to add handling for existing sshd config which may conflict with what this new role wants to do | 15:33 |
jrosser | i did add stuff to remove the old ssk keys, so it will probably work | 15:34 |
jrosser | but similar to SSL stuff you probably need to have run the play against all the hosts | 15:34 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Define variables for tempest plugins https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/826121 | 15:36 |
*** dviroel is now known as dviroel|lunch | 15:37 | |
noonedeadpunk | I think it's quite fair to distribute across all computes. | 15:37 |
noonedeadpunk | Thankfully this should be done one time and could be run with limit afterwards? | 15:38 |
jrosser | yes, i think with this method there is no coupling between the compute hosts at all | 15:38 |
jrosser | where are we with the centos-8 removal | 15:38 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Define variables for tempest plugins https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/826121 | 15:39 |
jrosser | xena is broken on tempest vars? | 15:39 |
jrosser | vars/requirements | 15:39 |
noonedeadpunk | X is broken with neutron plugin version | 15:40 |
noonedeadpunk | Eventually what I realized is that starting from X tempest is not in u-c anymore | 15:40 |
noonedeadpunk | So I got fed with that and decided that it's worth fixing plugins versions as well | 15:41 |
noonedeadpunk | With removal as far as I got, chain looks like - we merge https://review.opendev.org/c/openstack/openstack-ansible/+/826135, then we should able to merge https://review.opendev.org/c/openstack/openstack-ansible/+/824567 and https://review.opendev.org/c/openstack/openstack-ansible/+/824570 likely go last | 15:43 |
jrosser | i need to look again at the centos-9 patches | 15:47 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest stable/xena: Define variables for tempest plugins https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/826060 | 15:47 |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Convert xinetd clustercheck to systemd socket service https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/824042 | 15:48 |
jrosser | ^ this is important if the proxysql stuff might take a while | 15:48 |
noonedeadpunk | It's in merge conflict now | 15:51 |
jrosser | ok no worries | 15:51 |
jrosser | i think i need to test it locally on a fresh AIO | 15:51 |
jrosser | something wierd is not working | 15:51 |
noonedeadpunk | Main issue with proxysql, that I'm a bit struggling with, is that it requires users to be created for itself as well https://proxysql.com/documentation/users-configuration/ and all this layering of config that is applied... | 15:52 |
noonedeadpunk | So it's a bit fishy and would require big changes and some migration... | 15:52 |
noonedeadpunk | But honestly time is main thing that I'm struglling for real :) | 15:53 |
*** sshnaidm|afk is now known as sshnaidm | 15:55 | |
noonedeadpunk | And now I can finally work on updating dashboard to include BC | 15:59 |
spatel | noonedeadpunk I am working on W - X upgrade in my lab | 16:01 |
noonedeadpunk | #endmeeting | 16:01 |
opendevmeet | Meeting ended Tue Jan 25 16:01:50 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:01 |
opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-01-25-15.00.html | 16:01 |
opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-01-25-15.00.txt | 16:01 |
opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-01-25-15.00.log.html | 16:01 |
spatel | Should i be worried of anything? like mysql SSL stuff? (i am using F5 loadbalancer so thinking to disable it during upgrade otherwise it will break some stuff) | 16:02 |
noonedeadpunk | I think no - it's lab after all :) | 16:03 |
noonedeadpunk | but as I said, there were some improvement from 24.0.0 that should have landed with https://review.opendev.org/c/openstack/openstack-ansible/+/825391 | 16:04 |
spatel | in lab i am not worried but i am going to create process from lab so in production i don't see any hiccups | 16:04 |
spatel | agreed about 24.1.0 but i am just testing upgrade process in multinode environment so make sure nothing missed | 16:05 |
spatel | not worried about improvement | 16:06 |
spatel | what is the deal with mysql connection? | 16:07 |
spatel | i meant connection_recycle_time ? | 16:08 |
opendevreview | Merged openstack/ansible-role-qdrouterd master: Use integrated linter jobs https://review.opendev.org/c/openstack/ansible-role-qdrouterd/+/825839 | 16:08 |
noonedeadpunk | Um, and what's with it? | 16:09 |
noonedeadpunk | you mean that we implemented new variable to have better control over it? | 16:10 |
spatel | I am asking what value i should be using in production? | 16:10 |
spatel | I have 300 compute nodes so its kinda large deployment | 16:10 |
noonedeadpunk | well computes are not connecting to galera ;) | 16:11 |
spatel | but does it impact with message rate? | 16:11 |
noonedeadpunk | But it was only an issue when haproxy restarted a lot of times in a row, you could get up to max_connections easily | 16:11 |
spatel | my question is why does that setting came in picture ? | 16:11 |
spatel | hmm! i have F5 ... | 16:12 |
noonedeadpunk | I'm not sure it matters as it's still l3 balancer? | 16:13 |
spatel | yes | 16:13 |
spatel | i have seen lots of folks using very large value here like connection_recycle_time = 1500 | 16:13 |
noonedeadpunk | yeah, so connections are not killed properly, and with previous default value if you restart haproxy like 5 times in an hour, you will catch stuck mysql because of not having spare connections | 16:14 |
noonedeadpunk | But it was always like that. So it's more an improvement | 16:14 |
spatel | copy that | 16:15 |
noonedeadpunk | And giving controll over value in a nice way | 16:15 |
spatel | so let me not touch it | 16:15 |
spatel | I am keeping SSL off for mysql just for safely | 16:15 |
noonedeadpunk | while generally it's decreased, you now can set `openstack_db_connection_recycle_time` and it would result if adjusting value for all services | 16:15 |
spatel | nice universal variable :) | 16:16 |
jrosser | spatel: at some point we might default to ssl everywhere | 16:16 |
jrosser | and then start to make it the only option | 16:16 |
jrosser | so changing the defaults maybe only stores up trouble for later | 16:17 |
jrosser | like with rabbitmq, you just have no choice now | 16:17 |
noonedeadpunk | (it wasn't our call to have that said) | 16:17 |
noonedeadpunk | eventually role still provides choice? | 16:18 |
jrosser | no, but i am concerned about how we might support a mixture of SSL and non-SSL, and the migration between them in the long term for everything | 16:18 |
noonedeadpunk | once it's matter of including or not including pki role? | 16:19 |
noonedeadpunk | ah, yes, we have configs as well.. | 16:19 |
jrosser | yes, it's pretty widespread change | 16:19 |
noonedeadpunk | but yes, I agree that in general having mysql covered with TLS is a good thing overall | 16:19 |
jrosser | this may be a good PTG topic | 16:20 |
jrosser | james gibson has a spec for making the internal VIP https | 16:20 |
jrosser | but that is looking like it might be something that you do during an upgrade, as the intermediate http&https setup in parallel is pretty ugly | 16:21 |
spatel | jrosser :) ssl is good but bad for troubleshooting but i agreed for security | 16:26 |
spatel | I still need to work on OVN SSL stuff which i totally drop but i am back now | 16:26 |
spatel | One more question currently if someone want to deploy OVN then they need to do export SCENARIO='aio_ovn_lxc' can we create variable or something like neutron_agent_plugin=ovn or openvswitch or linuxbridge ? | 16:29 |
jrosser | SCENARIO is only ever used for automatically configuring the AIO | 16:30 |
jrosser | for real deployments then variables need to be set | 16:30 |
spatel | I am talking about production | 16:30 |
spatel | 3 node | 16:31 |
spatel | we don't have any good variable for that | 16:31 |
noonedeadpunk | um, there's such variable `neutron_plugin_type: ml2.ovn` | 16:31 |
jrosser | we do | 16:31 |
jrosser | https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/templates/user_variables_neutron_ovn.yml.j2 | 16:31 |
jrosser | ^ that is what the ovn AIO scenario does, it applys that set of vars | 16:31 |
spatel | hmm oh | 16:32 |
jrosser | check out what is in that directory | 16:32 |
noonedeadpunk | It's even documented here https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html#openstack-ansible-user-variables | 16:32 |
jrosser | all the different scenarios we test in CI just grab one of those files | 16:32 |
spatel | i misunderstood that :) sorry | 16:32 |
spatel | Can we add small snippet about OVN in aio build? like if someone want to build OVN then do export SCENARIO='aio_ovn_lxc' | 16:33 |
spatel | export SCENARIO='aio_ovn_lxc' | 16:33 |
spatel | sorry here - https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html | 16:34 |
spatel | i don't think other new folks know that we do have ovn support | 16:35 |
noonedeadpunk | we have network scenarios listed here https://docs.openstack.org/openstack-ansible-os_neutron/latest/ | 16:37 |
noonedeadpunk | but we probably want to drop this red warning and experimental support one day | 16:38 |
noonedeadpunk | *about experimental | 16:38 |
*** dviroel|lunch is now known as dviroel | 16:38 | |
spatel | Yes, if someone visit openstack-ansible and try to build AIO so just have option on AIO page that you can build with OVN for experimental | 16:39 |
noonedeadpunk | well... we kind of need to maintain list of all scenarios then I'd say | 16:40 |
noonedeadpunk | which I'm not sure how to achieve right now | 16:40 |
spatel | otherwise we never every going to get bug report if folks not going to try :( | 16:40 |
noonedeadpunk | I do agree here | 16:41 |
spatel | all put all senario in one page and create symlink to AIO offical page | 16:42 |
spatel | look like we need some cleanup on AIO page, because some of stuff they are just extra and old like this note - configure: error: cannot run C compiled programs. | 16:42 |
spatel | I never ever get that error in AIO | 16:43 |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add ssh_keypairs role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/825113 | 17:09 |
opendevreview | Merged openstack/ansible-role-systemd_networkd master: Add integrated linters test https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/799038 | 17:13 |
opendevreview | Merged openstack/ansible-role-systemd_mount master: Add integrated linters test https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/799036 | 17:24 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use sysctl ini-like config file https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826338 | 17:26 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use sysctl ini-like config file https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826338 | 17:27 |
opendevreview | Merged openstack/openstack-ansible-os_horizon master: Refactor use of include_vars https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/824331 | 17:30 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use journald logging for RabbitMQ https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826345 | 17:40 |
opendevreview | Merged openstack/openstack-ansible-ceph_client master: Add integrated linters test https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/798816 | 17:41 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not run rsyslog against RabbitMQ https://review.opendev.org/c/openstack/openstack-ansible/+/826347 | 17:42 |
opendevreview | Merged openstack/openstack-ansible stable/wallaby: Set infra_lxc job to centos-8-stream https://review.opendev.org/c/openstack/openstack-ansible/+/826135 | 17:53 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Define variables for tempest plugins https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/826121 | 18:10 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest stable/xena: Define variables for tempest plugins https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/826060 | 18:11 |
opendevreview | Merged openstack/openstack-ansible master: Fix definition of ssl_protocol https://review.opendev.org/c/openstack/openstack-ansible/+/823990 | 18:19 |
opendevreview | Merged openstack/openstack-ansible master: Add defaults for TLS 1.3 and rename TLS <= 1.2 variable https://review.opendev.org/c/openstack/openstack-ansible/+/823943 | 18:19 |
opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Change default mode while creating directories https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/824774 | 18:21 |
jamesdenton | does OSA have the ability to specify an FQDN for external_lb_vip_address that is compatible with a NAT? | 18:27 |
jamesdenton | maybe haproxy_bind_external_lb_vip_address is the trick | 18:28 |
jrosser | i think i have done this | 18:42 |
jrosser | hmm | 18:42 |
jrosser | yes, haproxy_bind_external_lb_vip_address was the natted address | 18:45 |
jrosser | and external_lb_vip_address was the floating ip | 18:45 |
jamesdenton | yep, or in my case, the FQDN that points to the public IP | 18:46 |
jrosser | (this is an AIO on a private network in an openstack project with a neutron floating IP) | 18:46 |
jamesdenton | haproxy was trying to bind to that | 18:46 |
jamesdenton | i see | 18:46 |
jrosser | it even managed to get itself an LE certificate | 18:46 |
jamesdenton | haproxy_bind_external_lb_vip_address worked, thank you | 18:46 |
jamesdenton | reeeeeally | 18:46 |
jrosser | yup | 18:47 |
jrosser | real dns entry required somewhere, but yes it worked | 18:47 |
jamesdenton | aye | 18:47 |
jamesdenton | do you know if we have the ability to drop root CAs onto the host? for corp-signed certs? | 18:48 |
jrosser | https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/defaults/main.yml#L171-L175 | 18:48 |
jamesdenton | my hero, thank you | 18:49 |
jrosser | np | 18:49 |
opendevreview | Merged openstack/openstack-ansible-os_tempest master: Deprecate congress and lxd tempest plugins https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/826111 | 18:56 |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Convert xinetd clustercheck to systemd socket service https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/824042 | 19:08 |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Convert xinetd clustercheck to systemd socket service https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/824042 | 21:12 |
*** dviroel is now known as dviroel|afk | 21:38 | |
opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_horizon master: Fix default multidomain choices https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/826310 | 22:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!