| noonedeadpunk | so infra upgrade jobs seem to fail for $reason.... | 07:44 |
|---|---|---|
| noonedeadpunk | which I bet is related to the plugins repo and some env that is set | 07:44 |
| jrosser | morning | 07:54 |
| damiandabrowski[m] | hey! | 07:54 |
| jrosser | noonedeadpunk: do you mean for everything or for https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/824042 ? | 07:54 |
| jrosser | becasue i think i know what is wrong with the xinetd patch | 07:54 |
| noonedeadpunk | I saw some failurte for lxc | 07:54 |
| jrosser | ah ok | 07:54 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/824345 | 07:55 |
| noonedeadpunk | ah, yes, it's different indeed | 07:57 |
| jrosser | we had to fix that already didnt we | 07:57 |
| jrosser | this https://zuul.opendev.org/t/openstack/build/7b49679fc0514ccb92edf4e98b1e5efb/log/job-output.txt#24105-24109 | 07:57 |
| jrosser | feels like we have old plugins with new ansible, or something like that | 07:57 |
| noonedeadpunk | But rabbit fails same way https://zuul.opendev.org/t/openstack/build/15bffe62909843bc9c75ef934e0e74cb/log/job-output.txt#23834 | 07:58 |
| noonedeadpunk | yes, exactly. And I can blame several things. First of all, we changed ANSIBLE_TRANSPORT https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/openstack-ansible.rc#L50 | 07:58 |
| noonedeadpunk | And we should unset this during run_upgrade https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/run-upgrade.sh#L173 | 07:59 |
| jrosser | it no longer has this https://github.com/openstack/openstack-ansible-plugins/commit/5b2b38ea1cf2554b081e9638423d3d6b06308ec4 | 08:00 |
| noonedeadpunk | As I guess that /etc/ansible/roles/plugins is used as transport and not collection | 08:01 |
| noonedeadpunk | And we stopped managing that path | 08:02 |
| noonedeadpunk | Likely we should just drop it during upgrade, but well, not sure it will fix issue | 08:02 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use openstack.cloud.config module instead of deprecated os_client_config https://review.opendev.org/c/openstack/openstack-ansible/+/826423 | 08:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Test infra upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/826424 | 08:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Test infra upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/826424 | 08:07 |
| noonedeadpunk | and eventually, we just ran whole setup-infrastructure without any issues... | 08:13 |
| jrosser | though the play exits and the first task fails | 08:20 |
| jrosser | so it may pick up different environment / config for ansible at that point | 08:20 |
| jrosser | *previous play | 08:20 |
| noonedeadpunk | so like ENV is persistant in scope of gate-check-commit.sh? | 08:21 |
| noonedeadpunk | and unset inside included script doesn't really affect it? | 08:21 |
| jrosser | if we set environment in a shell called by gate-check-commit, once that completes, it wont persist | 08:22 |
| jrosser | you can't change the parent environment | 08:22 |
| noonedeadpunk | we;re trying to unset instead. But I guess it still applies | 08:23 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Convert xinetd clustercheck to systemd socket service https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/824042 | 08:25 |
| noonedeadpunk | damiandabrowski[m]: where did you catched this error you mentioned in https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826345/1 ? | 08:26 |
| noonedeadpunk | *have you catched | 08:27 |
| noonedeadpunk | ah, bullseye | 08:27 |
| damiandabrowski[m] | sorry, i should have be more specific ;) | 08:27 |
| noonedeadpunk | yeah, for it we use special case as no external repo existed... | 08:28 |
| noonedeadpunk | likely that is not the case anymore | 08:29 |
| noonedeadpunk | oh.... So if you run unset inside script, it's not really applied to ENV.... | 08:31 |
| noonedeadpunk | ie https://paste.opendev.org/show/812361/ | 08:32 |
| noonedeadpunk | well, I didn't know that... | 08:32 |
| noonedeadpunk | somehow | 08:32 |
| noonedeadpunk | but it works if you source instead | 08:33 |
| *** anbanerj is now known as frenzyfriday | 08:35 | |
| jrosser | source executes it in the current shell i think? | 08:37 |
| noonedeadpunk | yeah, I guess it's the difference indeed | 08:38 |
| starborn | Time appropiate greeetings. I am very very late to the party and really hate to ask this: Why has the nspawn support been deprecated? Any technical reasons? | 08:40 |
| noonedeadpunk | starborn: not compatible with centos and lack of contributors/maintainers for it | 08:45 |
| jrosser | i think we maybe had one person we know of try it out | 08:46 |
| starborn | I see. Thanks for the information. | 08:48 |
| jrosser | it really is a long time ago but wasn't there also an issue that nspawn containers had much more limited networking | 08:50 |
| jrosser | like you couldn't describe that it had multiple interfaces in the config | 08:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Test infra upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/826424 | 09:09 |
| opendevreview | Merged openstack/openstack-ansible-os_aodh master: Ensure libxml2 is installed on debian systems https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/826311 | 10:05 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Use ssh_keypairs role to generate cold migration ssh keys https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/825306 | 10:08 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_aodh stable/xena: Ensure libxml2 is installed on debian systems https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/826378 | 10:09 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_aodh stable/wallaby: Ensure libxml2 is installed on debian systems https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/826379 | 10:10 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_aodh stable/victoria: Ensure libxml2 is installed on debian systems https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/826380 | 10:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Test infra upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/826424 | 11:00 |
| *** dviroel|afk is now known as dviroel | 11:21 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use cloudsmith repo for rabbit and erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826444 | 11:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use cloudsmith repo for rabbit and erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826444 | 11:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Allow different install methods for rabbit/erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826445 | 11:40 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Update used RabbitMQ and Erlang https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826446 | 11:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use journald logging for RabbitMQ https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826345 | 11:46 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: DNM - Add support for centos-9 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/823983 | 12:05 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - Centos-9 Stream support https://review.opendev.org/c/openstack/openstack-ansible/+/823417 | 12:06 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add ssh_keypairs role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/825113 | 12:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Test infra upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/826424 | 12:26 |
| *** odyssey4me is now known as Guest652 | 12:40 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: DNM - Add support for centos-9 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/823983 | 12:41 |
| mathlin | hi, on a Rocky + Ubuntu 18.04 installation and did an regular update on one of the controlplane nodes, now the lxcbr0 has status unknown, Any ideas, seems systemd+udev was updated (still running ifupdown) | 13:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server master: Use systemd_service role for overrides https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/826463 | 13:21 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible stable/xena: Fix definition of ssl_protocol https://review.opendev.org/c/openstack/openstack-ansible/+/826381 | 13:25 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible stable/wallaby: Fix definition of ssl_protocol https://review.opendev.org/c/openstack/openstack-ansible/+/826382 | 13:26 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible stable/victoria: Fix definition of ssl_protocol https://review.opendev.org/c/openstack/openstack-ansible/+/826383 | 13:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Fix infra scenario repo server cluster https://review.opendev.org/c/openstack/openstack-ansible/+/826468 | 13:43 |
| jrosser | mathlin: you could re-run playbooks/lxc-hosts-setup.yml and watch what happens with these tasks https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/rocky/tasks/lxc_net.yml | 14:02 |
| jrosser | i would advise in general to get more up to date, we no longer test anything as old as rocky | 14:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [DNM] Test infra upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/826424 | 14:22 |
| noonedeadpunk | damn, I really no idea why in the world it's not working as expected ^ | 14:23 |
| mathlin | jrosser: we are reinstalling in March, latest LTS and OSA. playbook ran without errors but device still in unknown | 14:45 |
| jrosser | as far as i can see lxcbr0 is managed with ifupdown | 14:47 |
| jrosser | a bridge is not up if there is nothing connected to it | 14:48 |
| jrosser | so you could use `brctl show` to see how that is | 14:48 |
| mathlin | so the lxc-containers can't start, maybe i am going the wrong way about this | 14:49 |
| jrosser | there should be per-container logs in /var/log/lxc/.... | 14:56 |
| mathlin | only very old logs in there | 15:00 |
| *** dviroel is now known as dviroel|lunch | 15:01 | |
| NeilHanlon | was doing an install last night and noticed there appears to be a regression in rhel-like systems due to the lxc3.0 COPR repository providing the python3-lxc package for python3.6, but OSA is using python3.8 for ansible-core 2.12. Is that copr owned by someone in the community? | 15:35 |
| jrosser | it is yes | 15:42 |
| jrosser | oh well "rhel-like community" is what i mean :) | 15:42 |
| jrosser | we are only using python3.8 in the /opt/ansible-runtime virtualenv for ansible | 15:43 |
| jrosser | tasks which are run against localhost should be using the default interpreter, which will be 3.6, precisely because of things like the lxc and selinux bindings | 15:43 |
| jrosser | interesting if thats not happening though | 15:44 |
| noonedeadpunk | we could drop smth vital for that to work with centos-8 removal... | 15:45 |
| noonedeadpunk | but centos-8-stream should still work | 15:45 |
| jrosser | and i think we test this exact setup today on centos-8-stream for both lxc and metal | 15:46 |
| noonedeadpunk | yes, I think indeed we must have 3.8 just for osa venv | 15:47 |
| jrosser | NeilHanlon: ^^ | 15:47 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon master: Adjust default configuration to support TLS v1.3 https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/823946 | 15:47 |
| jrosser | did we ever merge a patch to do 3.8 everywhere and then have to undo it partially? | 15:48 |
| jrosser | i would be surprised if that had passed the tests though | 15:48 |
| NeilHanlon | the problem is that in that venv, there is no lxc available | 15:56 |
| jrosser | there should not need to be | 15:56 |
| NeilHanlon | so you end up with `"The `lxc` module is not importable. Check the requirements." centos` | 15:57 |
| NeilHanlon | oh, probably important to mention.. this was an AIO | 15:57 |
| jrosser | what should happen is ansible tasks from the controller (localhost) will ssh to the target (localhost) and use the normal python outside the venv | 15:57 |
| jrosser | the thing to do will be to check what is happening in our existing CI | 15:57 |
| jrosser | becasue that uses the *exact* same config as a default setup AIO | 15:58 |
| NeilHanlon | gotcha. i will debug it a bit | 15:58 |
| jrosser | it might be interesting to just start an interactive python on the host and try `import lxc` for a real quick sanity check | 15:58 |
| NeilHanlon | https://rpa.st/L7GA | 16:00 |
| jrosser | that looks reasonable | 16:00 |
| NeilHanlon | from what I can tell, when it ssh's to the host to run the lxc module, it's running it under python3.8, which doesn't have the lxc module installed | 16:01 |
| mathlin | jrosser: i was way off, turned out that my /var/log/journal/* entries where gone, a quick restore and we are up and running again. Thank you for setting me straight regarding dependencies here. Now to investigate why they where deleted | 16:05 |
| jrosser | NeilHanlon: here is the first attempt we had to use python3.8 on centos-8-stream https://review.opendev.org/c/openstack/openstack-ansible/+/822260/1/scripts/bootstrap-ansible.sh | 16:08 |
| jrosser | that didnt work, for reasons like the ones you describe | 16:08 |
| jrosser | what got merged was this https://review.opendev.org/c/openstack/openstack-ansible/+/822260/4/scripts/bootstrap-ansible.sh | 16:09 |
| jrosser | which does not have `alternatives --set python3 /usr/bin/python3.8` | 16:09 |
| noonedeadpunk | that also required switcxhing to venv from virtualenv | 16:10 |
| jrosser | yes | 16:10 |
| opendevreview | Merged openstack/openstack-ansible master: Use openstack.cloud.config module instead of deprecated os_client_config https://review.opendev.org/c/openstack/openstack-ansible/+/826423 | 16:10 |
| jrosser | there is of course also ansible python interpreter discovery at play here | 16:10 |
| jrosser | `ansible aio1 -m debug -a var=ansible_python` | 16:11 |
| jrosser | NeilHanlon: this is on Rocky? | 16:13 |
| NeilHanlon | jrosser, yeah | 16:15 |
| NeilHanlon | i see what's happening now | 16:15 |
| NeilHanlon | ANSIBLE_PYTHON_INTERPRETER is set to auto, and it's auto-discovering the python3.8 that's installed over the 3.6 | 16:15 |
| jrosser | it is not a surprise | 16:16 |
| jrosser | there is spaghetti code inside ansible to choose the interpreter | 16:16 |
| jrosser | and if it doesnt specifically know about the OS then things can be wierd, or do an unexpected fallback | 16:16 |
| NeilHanlon | yeah, for sure | 16:17 |
| jrosser | we had to deal with this for a previous debian until ansible learned about it properly | 16:17 |
| NeilHanlon | going to try redoing this with that set explicitly to 3.6 and see how that goes | 16:18 |
| noonedeadpunk | I think we can leverage this somehow https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/bootstrap-ansible.sh#L157 | 16:18 |
| jrosser | there are clues in here about how we did it for debian buster https://github.com/openstack/openstack-ansible/commit/de16d1434b8529ba0d868baf023be7f39864b488#diff-4506379276173b431eda2adc244386d3d684511adb19e0b65f592ab77eed66b5 | 16:19 |
| noonedeadpunk | so likely `OSA_ANSIBLE_PYTHON_INTERPRETER` could be defined somewhere near https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/bootstrap-ansible.sh#L80 | 16:20 |
| noonedeadpunk | as we set it to auto explicitly at the beginning of the script (on L58) | 16:21 |
| *** dviroel|lunch is now known as dviroel | 16:23 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Fix infra scenario repo server cluster https://review.opendev.org/c/openstack/openstack-ansible/+/826468 | 16:26 |
| NeilHanlon | thank you noonedeadpunk, jrosser. Would it be ok to include that as part of my existing patch for rocky, or should I make a new patch you think? | 16:28 |
| jrosser | you can either keep a giant patch, or if you want to 'stack' a series of patches on your master branch locally thats also completely possible | 16:29 |
| noonedeadpunk | I don't think why not to update existing one? | 16:29 |
| noonedeadpunk | I'd say without this it's unlikely that CI will pass anyway | 16:30 |
| NeilHanlon | yep, true | 16:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix infra upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/826424 | 16:48 |
| jrosser | noonedeadpunk: what do you make of this https://paste.opendev.org/show/812375/ | 16:59 |
| jrosser | prototype of lsyncd converted into native systemd | 17:00 |
| noonedeadpunk | oh, PathChanged ? | 17:01 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon master: Fix default multidomain choices https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/826310 | 17:01 |
| jrosser | yeah | 17:01 |
| noonedeadpunk | I didn't know that was ever a case... | 17:01 |
| jrosser | i was going to lift the rsync command from the lsyncd debug log and see if it works | 17:01 |
| jrosser | see this https://man7.org/linux/man-pages/man5/systemd.path.5.html | 17:02 |
| jrosser | i expect there is all sorts of corner case handling in lsyncd, but this seems kind of simple | 17:02 |
| noonedeadpunk | and instead script.sh we run rsync? | 17:03 |
| jrosser | we would need to write bash/python thing to do the rsync | 17:04 |
| jrosser | or in the most trivial case just call it direct | 17:04 |
| jrosser | i don't know if it need to repeatedly rsync until there are no changes | 17:04 |
| jrosser | as during a venv build i expect that the path unit will get triggered many many times | 17:05 |
| noonedeadpunk | `it is activated if the file which was open for writing gets closed` And I don't see any timeout there before running command. Maybe we can define that in repo_sync@.service though | 17:06 |
| noonedeadpunk | Like TimeoutStartSec | 17:06 |
| jrosser | i expect the biggest issue will be race between files being created/changed and the time it takes to rsync | 17:08 |
| jrosser | and if systemd queues up the running of the unit, or just does it one-off | 17:08 |
| noonedeadpunk | Also I'm not sure I understood part with protection against busy looping | 17:19 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest master: Define variables for tempest plugins https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/826121 | 17:19 |
| noonedeadpunk | So they enforce StartLimitIntervalSec and that results in basically stopping of file tracking as well? | 17:20 |
| jrosser | thats a bit unclear isnt it | 17:21 |
| jrosser | maybe i try that | 17:21 |
| noonedeadpunk | but other then that it feels more convenient to manage at least) | 17:24 |
| jrosser | https://paste.opendev.org/show/812377/ | 17:24 |
| jrosser | seems sensible | 17:24 |
| noonedeadpunk | and likely we can make this "multi-master" by splitting directories to sync for wheels | 17:24 |
| noonedeadpunk | so it keeps track of files that are changed while it's failing? | 17:25 |
| jrosser | i put StartLimitIntervalSec=5, ran it 10 times quickly, then waited, and ran it once more | 17:25 |
| jrosser | it is a different test to see if if tracks whilst failing | 17:26 |
| noonedeadpunk | ah, I see | 17:26 |
| jrosser | which i think it does not do | 17:27 |
| jrosser | but nothing stops us adding a timer unit to this as well | 17:27 |
| jrosser | and you are right, we can use multi master very easily | 17:27 |
| jrosser | and in the 'top level' service file we can specify target IP and architecture, you can have as many parameters as you need i believe | 17:28 |
| noonedeadpunk | so with lsyncd we have delay on running rsync in 15 sec https://opendev.org/openstack/openstack-ansible-repo_server/src/branch/master/templates/lsyncd.lua.j2#L607 | 17:28 |
| jrosser | it does say that upon success the path unit will immediatley check again | 17:29 |
| jrosser | it's not so easy to make these simplified unit files with the systemd_service role | 17:30 |
| jrosser | it puts a ton of extra stuff in there | 17:30 |
| noonedeadpunk | well... we already have templated services.... | 17:30 |
| noonedeadpunk | what we miss is Path | 17:31 |
| noonedeadpunk | or I'm not seeing hidden troubles? | 17:32 |
| jrosser | i see this with the galera xinetd changes | 17:32 |
| jrosser | where i have a socket service that need to call a trivial unit which runs the mysql status check | 17:32 |
| jrosser | but it does include all sorts of stuff in the j2 template that you can't remove | 17:32 |
| noonedeadpunk | I actually wonder if `ExecStartPre: /usr/bin/sleep 15` will jsut add same delay as lsyncd did? | 17:33 |
| jrosser | oh interesting, yes | 17:33 |
| jrosser | you can't omit ExecStart | 17:33 |
| jrosser | so the /usr/bin/true is just a hack to keep it there | 17:33 |
| jrosser | what i really wanted was some sort of noop unit which just had dependancies on the others | 17:34 |
| noonedeadpunk | well, it's just matter of `| default([])` here https://opendev.org/openstack/ansible-role-systemd_service/src/branch/master/templates/systemd-service.j2#L35 ? | 17:35 |
| jrosser | oh sorry no, it's an invalid `simple` type unit without it | 17:35 |
| jrosser | from systemd POV | 17:35 |
| noonedeadpunk | ah | 17:35 |
| jrosser | i've not yet tried to use the systemd_service role to set this up, it's just hand crafted atm | 17:36 |
| noonedeadpunk | but must it be simple? | 17:36 |
| noonedeadpunk | hm, maybe idle will even fit us.... | 17:39 |
| noonedeadpunk | But seems every type except notify requires ExecStart | 17:39 |
| noonedeadpunk | ah, no, everything needs execstart | 17:42 |
| *** sshnaidm is now known as sshnaidm|afk | 17:49 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Clean-up systemd overrides removal task https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/826503 | 17:51 |
| jrosser | noonedeadpunk: commit message here is interesting https://review.opendev.org/c/openstack/swift/+/821192 | 17:52 |
| jrosser | i wonder if that is also related to glance+uwsgi issues | 17:52 |
| jrosser | iirc that was chunked encoding trouble too? | 17:52 |
| noonedeadpunk | iirc yes, it was | 17:54 |
| noonedeadpunk | `transformation_chunked` huh | 17:54 |
| jrosser | indeed | 17:54 |
| noonedeadpunk | this commit message should be put into uwsgi docs.... | 17:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Drop cell1 upgrade to template format https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/826504 | 17:59 |
| NeilHanlon | anyone seen this before? https://rpa.st/6PAA | 18:14 |
| jrosser | NeilHanlon: if you run the (sub)playbook again with -vvv you'll see the parameters that were passed | 18:18 |
| noonedeadpunk | I bet it was just empty string in a list or empty element? | 18:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix infra upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/826424 | 18:48 |
| spatel | noonedeadpunk 24.0.0 vs 24.0.0.0rc2 ? | 18:50 |
| spatel | which one i should give it a try | 18:51 |
| spatel | going with 24.0.0 | 18:53 |
| noonedeadpunk | they are exactly same just in case | 19:00 |
| noonedeadpunk | btw we need to merge https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/826060 to proceed with dropping centos-8 | 19:00 |
| noonedeadpunk | oh, its backport, i think I can vote then?:) | 19:01 |
| jrosser | NeilHanlon: is your patch current? I might have a go with a Rocky cloud image tomorrow…… | 19:06 |
| noonedeadpunk | also - let's maybe merge https://review.opendev.org/c/openstack/openstack-ansible/+/782557 for real? :) | 19:39 |
| noonedeadpunk | NeilHanlon: also you might want to review as we might drop there smth important for Rocky 8 ^ | 19:39 |
| NeilHanlon | jrosser: not super urgent. i have a patch into dib for adding a rocky container, though it appears to be failing in CI due to changes unrelated to mine | 19:40 |
| NeilHanlon | noonedeadpunk - thank you! will crosscheck that with my patch and add back in if needed | 19:40 |
| NeilHanlon | actually, looks good at a glance. | 19:43 |
| spatel | noonedeadpunk why do i need to run - openstack-ansible certificate-authority.yml ? | 19:45 |
| jrosser | ok lets merge it | 19:45 |
| spatel | To generate new CA, you will need to run the following command: | 19:45 |
| spatel | why do i need to generate new CA for upgrade? | 19:46 |
| jrosser | spatel: do you have some context - i am guessing this is upgrade notes? | 19:48 |
| spatel | Yes - I am following this steps to upgrade from W->X - https://docs.openstack.org/openstack-ansible/latest/admin/upgrades/major-upgrades.html | 19:48 |
| spatel | jrosser my question is why do i need to Re-generate CA just for upgrade process? I already have CA running from previous upgrade. | 19:49 |
| jrosser | it is becasue the next command in the instructions is run with --limit | 19:51 |
| jrosser | but i think that you may be right that this is only relevant for V->W upgrades | 19:51 |
| spatel | Yes, i can understand that this step is required before PKI role introduce but now we already have running CA in wallaby so we don't need this | 19:52 |
| spatel | not sure if this will generate SSL cert for Galera ... i doubt | 19:53 |
| jrosser | well its simple | 19:53 |
| jrosser | if we don't take stuff like this out of the docs then it will stay for the next release] | 19:54 |
| jrosser | so just a mistake really | 19:54 |
| jrosser | and you are right it will not generate an SSL cert for galera, thats expected | 19:54 |
| jrosser | it just sets up the CA | 19:54 |
| spatel | Lets me ignore this step and move forward, i will submit patch and see if noonedeadpunk has any objection :) | 19:54 |
| jrosser | the PKI role is called again in the galrea role to generate the cert from the CA | 19:54 |
| jrosser | the pki role is multi-purpose | 19:55 |
| spatel | Yes that is what i thought galera will just generate cert from your existing CA | 19:55 |
| jrosser | it can manage the CA, and it can generate certs | 19:55 |
| jrosser | depending on what variables you set it will do one/other/both | 19:55 |
| jrosser | it's used right at the start of setup-hosts to make the CA | 19:55 |
| jrosser | and then the roles like galera, rabbit and haproxy use it again as needed | 19:56 |
| spatel | that is cool | 19:56 |
| jrosser | that means there is no gigantic logic needed up front to work out what all the certs you need are for the whole deployment | 19:57 |
| jrosser | it's on-demand | 19:57 |
| spatel | currently i have set galera_use_ssl: false (just for safe play because i have F5 in front door instead of haproxy) | 19:58 |
| * noonedeadpunk already away | 20:00 | |
| spatel | jrosser what is this for -e package_state=latest ? | 20:03 |
| jrosser | osa defaults to using package_state=present | 20:03 |
| jrosser | so that you don't get surprise upgrades when running the playbooks day to day | 20:03 |
| jrosser | forcing state=latest when you specifically want an upgrade makes upgrades happen when you expect them | 20:04 |
| spatel | are these packages for distro binary correct? | 20:05 |
| jrosser | for things like rabbit or whatever | 20:06 |
| jrosser | apt/rpm stuff | 20:06 |
| spatel | i thought we have pinned version of packages like rabbit/mariadb/memcache etc | 20:06 |
| jrosser | you can see how it is used here https://codesearch.opendev.org/?q=package_state&i=nope&literal=nope&files=&excludeFiles=&repos= | 20:07 |
| spatel | copy that | 20:11 |
| spatel | This is going to do true upgrade not just upgrade openstack binaries :) | 20:12 |
| spatel | jrosser what is your method to upgrade major release? do you upgrade everything on same time or in pieces ? | 20:16 |
| jrosser | basically follow the instructions | 20:16 |
| spatel | like day1 infra nodes only and day2 some compute and day3 rest of compute... etc.. | 20:16 |
| jrosser | but yes, we've got a schedule on a confluence page that gives the order we've done it previously, and notes of whats important | 20:17 |
| spatel | I am asking before when last time i upgraded from V->W (it took 24 hour on 250 compute nodes) | 20:18 |
| jrosser | but if nothing goes wrong and it's not also an operating system upgrade then it might well get done in a day | 20:18 |
| jrosser | but like you we do it all in the lab first for no big surprises | 20:19 |
| spatel | I found openstack-ansible stopped all service on compute node and then started them parellel (that created big mess) | 20:19 |
| spatel | i wish osa do work in serial and not stop start compute service all in one shot that kind of scary | 20:22 |
| jrosser | spatel: you can do it however you like https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-nova-install.yml#L27 | 20:23 |
| jrosser | everything is configurable ;) | 20:24 |
| jrosser | and soon it will be much easier to use --limit as well | 20:24 |
| spatel | oh! wait.. so how do i use it ? nova_compute_serial ? | 20:25 |
| opendevreview | Merged openstack/openstack-ansible-os_zun master: Enable recursion in combine() filter https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/824219 | 20:25 |
| spatel | what is 100% ? | 20:25 |
| jrosser | https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html#setting-the-batch-size-with-serial | 20:26 |
| spatel | sweet! so i can use - nova_compute_serial: 10% | 20:27 |
| spatel | we should create doc for upgrade cheat-sheet where we can put all these cool stuff and method for smooth upgrade. | 20:30 |
| spatel | coding is easy but documentation is hardest part :( | 20:30 |
| jrosser | you'd have to test all of this | 20:35 |
| NeilHanlon | TIL about codesearch. that will be helpful | 20:41 |
| * NeilHanlon bookmarks | 20:41 | |
| spatel | lol i do have bookmark but always forgot to use when need it :) | 20:42 |
| spatel | jrosser do you prefer this - openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e package_state=latest | 20:42 |
| spatel | or one at a time like first galera and then rabbitmq and then next one etc.. | 20:43 |
| jrosser | I can’t remember - not on the right system just now to look at our docs | 20:46 |
| spatel | no worry, but please share your experience when you get a time :) | 20:49 |
| *** dviroel is now known as dviroel|afk | 21:06 | |
| opendevreview | Merged openstack/openstack-ansible-os_tempest stable/xena: Define variables for tempest plugins https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/826060 | 21:10 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Adjust default configuration to support TLS v1.3 https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/823945 | 21:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix infra upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/826424 | 21:24 |
| jrosser | what happens to all these xena patches that fail wanting a rebase | 21:30 |
| jrosser | odd | 21:30 |
| noonedeadpunk | hm it really is actually... | 21:31 |
| noonedeadpunk | well maybe it's because of https://review.opendev.org/c/openstack/openstack-ansible/+/824567 but unlikely though | 21:32 |
| noonedeadpunk | but well, tomorrow is a new day :) | 21:33 |
| jrosser | indeed | 21:34 |
| jrosser | zuul restart just before as well so it could all be from that | 21:34 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Fix ordering error enabling/disabling Apache modules https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/824000 | 21:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone stable/xena: Fix ordering error enabling/disabling Apache modules https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/826550 | 21:46 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone stable/wallaby: Fix ordering error enabling/disabling Apache modules https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/826551 | 21:46 |
| opendevreview | Merged openstack/openstack-ansible master: Gather additional facts for haproxy playbook https://review.opendev.org/c/openstack/openstack-ansible/+/823655 | 22:11 |
| opendevreview | Merged openstack/openstack-ansible master: Remove references to unsupported operating systems https://review.opendev.org/c/openstack/openstack-ansible/+/782557 | 22:33 |
| prometheanfire | do I need to increase sync priority in galera? having galera/mariadb constantly returning bad healthchecks and coming back | 23:30 |
| prometheanfire | max_connections strikes again | 23:34 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!