| jamesdenton | noonedeadpunk FWIW, here is a comparison of a standard AIO w/ the PAM patch you posted earlier, 899286. https://paste.openstack.org/show/b2fOoYTPBlPeqMTeH3c5/ | 00:53 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM Disable wheels build https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 05:14 |
| opendevreview | Merged openstack/ansible-role-pki master: Simplify PKI host directory creation https://review.opendev.org/c/openstack/ansible-role-pki/+/899269 | 05:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM Disable wheels build https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 05:16 |
| noonedeadpunk | that looks like quite some difference | 06:21 |
| noonedeadpunk | jamesdenton: was that with the first patchset, which had UsePAM no (https://review.opendev.org/c/openstack/ansible-hardening/+/899286/1/vars/main.yml#433) or my second try https://review.opendev.org/c/openstack/ansible-hardening/+/899286/2/tasks/rhel7stig/sshd.yml ? | 06:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM: Try to tune SSH in pre-step. https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 06:28 |
| noonedeadpunk | I can't see what we're doing in our ssh connection plugin that can make it slower then original one for the bare metal hosts | 06:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM: Try to tune SSH in pre-step. https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 06:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Disable GSSAPIAuthentication for SSH https://review.opendev.org/c/openstack/ansible-hardening/+/899286 | 07:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM: Try to tune SSH in pre-step. https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 07:21 |
| mnasiadka | NeilHanlon: thanks for doing that ;) | 07:31 |
| jrosser | good morning | 07:46 |
| noonedeadpunk | o/ | 07:52 |
| jrosser | zk role may be broken for bookworm https://zuul.opendev.org/t/openstack/build/f95e123a2f02478d80f34e526513371f/log/job-output.txt#10758 | 07:53 |
| noonedeadpunk | yeah... | 07:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Disable dynamic motd message https://review.opendev.org/c/openstack/ansible-hardening/+/899370 | 08:25 |
| damiandabrowski | hey folks. Sorry that I couldn't attend to PTG, had a tough openstack upgrade this week :/ | 08:48 |
| noonedeadpunk | jrosser: btw I didn't catch issue with rgw and your patch for tempest in AIO | 08:50 |
| noonedeadpunk | and tempest.scenario.test_object_storage_basic_ops passed | 08:51 |
| jrosser | noonedeadpunk: interesting, we see it fail in CI though don't we | 08:54 |
| noonedeadpunk | yup | 08:56 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/zed: Fix logic of discovering hosts by service https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/898780 | 09:01 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/yoga: oidc: fix recognition of x forwarded headers from v2.4.11 https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/899046 | 09:17 |
| noonedeadpunk | and yes, I also see disabling dynamic mode saving like 7% of time alike to what jamesdenton posted | 09:35 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Cleanup upgrade to ssh_keypairs step https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/898802 | 09:54 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Add nova_libvirt_live_migration_inbound_addr to compute SAN https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/898751 | 09:58 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Apply rate limit for journald in AIO builds https://review.opendev.org/c/openstack/openstack-ansible/+/898770 | 09:58 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Cleanup upgrade to ssh_keypairs step https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/898801 | 09:59 |
| opendevreview | Merged openstack/openstack-ansible-os_masakari master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/898936 | 10:02 |
| opendevreview | Merged openstack/openstack-ansible-os_senlin master: Fix linters for example playbook https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/899248 | 10:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Replace deprecated httpchk with send https://review.opendev.org/c/openstack/openstack-ansible/+/899383 | 10:09 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Add security rule for octavia healthmanager https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/897316 | 10:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Ensure tempest include and exclude lists all use unique names https://review.opendev.org/c/openstack/openstack-ansible/+/893968 | 10:10 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Drop Neutron oslomsg configuration https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/898923 | 10:17 |
| opendevreview | Merged openstack/openstack-ansible-os_heat master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/899249 | 10:17 |
| noonedeadpunk | So... Looks like not using our connection plugin is really improving things _a lot_ | 10:17 |
| noonedeadpunk | in conjuction with disabling wheels we're really down to 1:10 for metal jobs | 10:18 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 10:18 |
| anskiy | noonedeadpunk: well, I'm either blind or stupid, but it seems that I only need to add `neutron_availability_zones` to the appropriate configs for OVS/LXB: dhcp_agent.ini or l3_agent.ini, as there are no other mentions of AZs in os_neutron except those I've added. | 10:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM: Try to tune SSH in pre-step. https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 10:19 |
| noonedeadpunk | anskiy: well, pretty much yes. this what we have to do to make AZs working as expected (for us) with ovs: https://paste.openstack.org/show/bGmyxv08ImtHSIcRa0dw/ | 10:22 |
| noonedeadpunk | But I guess I was concerned that you can't suply really a list to the config | 10:22 |
| noonedeadpunk | it should be 1 specific AZ which neutron agent serves in | 10:23 |
| noonedeadpunk | not sure though | 10:23 |
| noonedeadpunk | so I just didn't want to introduce variables with wrong types or without way forward for other options | 10:24 |
| noonedeadpunk | but also didn't have time to think thoroughly about that :( | 10:24 |
| anskiy | noonedeadpunk: okay, so I just need to check that thing, thank you! | 10:26 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create master: Use FQCN for lxc_container module https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/899128 | 10:37 |
| jrosser | i would expect the connection plugin to be almost a no-op for a metal deploy as it just should call through to the base SSH class | 10:45 |
| jrosser | so if it's hurting performance in metal jobs then we have some fundamental thing to look at | 10:45 |
| noonedeadpunk | yes, I am exactly under same impression looking at code | 10:45 |
| noonedeadpunk | I was going to play now in multi-node sandbox a bit | 10:46 |
| noonedeadpunk | just re-running setup-hosts against metal only - even though it doesn't make changes it should show some difference if it's there | 10:47 |
| jrosser | the thing to be careful of too is that with pipelining not all modules use put_file / fetch_file | 10:52 |
| jrosser | need to be using -vvvvv to see whats actually happening | 10:52 |
| jrosser | and for something like copy: it might use stat: first over the controlpersist connection to see if the file is already there | 10:52 |
| jrosser | so doing a benchmark with something like copy: where you make the content be a loop: index perhaps, so it's forced to change every iteration | 10:53 |
| noonedeadpunk | I'm actually not sure we're using pipelining at all. | 10:55 |
| noonedeadpunk | That what I've spotted early in the morning | 10:55 |
| noonedeadpunk | https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/ssh.py#L584 we don't have that | 10:56 |
| opendevreview | Merged openstack/openstack-ansible master: Drop ssh_keypairs_install_authorized_keys reference https://review.opendev.org/c/openstack/openstack-ansible/+/898804 | 10:56 |
| opendevreview | Merged openstack/openstack-ansible master: Bump ansible version to 2.15.5 https://review.opendev.org/c/openstack/openstack-ansible/+/899257 | 10:56 |
| noonedeadpunk | And I guess this part we override in the module | 10:56 |
| noonedeadpunk | Next thing is potentially use their retry.... | 10:57 |
| noonedeadpunk | maybe that's where we also loose time... But our version looks way more simple... | 10:57 |
| jrosser | we should also check that the behaviour of ControlPath is correct for our LXC case | 10:58 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/yoga: oidc: fix overloading of redirect_uri for cli client https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/899041 | 10:59 |
| noonedeadpunk | yeah | 10:59 |
| jrosser | as we could be in a sitaution where every physical_host/container_name gets it's own connection | 10:59 |
| jrosser | vs. one connection per physical_host | 10:59 |
| noonedeadpunk | I'm pretty sure it's what happening | 11:00 |
| noonedeadpunk | But if we see penalty just for metal - I would try to fix that first | 11:00 |
| jrosser | absolutely | 11:00 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/2023.1: Use internal endpoint for barbican API https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/899042 | 11:02 |
| jrosser | question maybe if this is actually necessary with more modern ansible https://github.com/openstack/openstack-ansible-plugins/commit/621c552e233473bca9ce220abe210025052c9ada | 11:05 |
| noonedeadpunk | yeah, so I guess we might be able to use original decorator for that. But I kinda not sure it's faster looking at it's code either | 11:13 |
| jamesdenton | noonedeadpunk it was patchset 2 - https://review.opendev.org/c/openstack/ansible-hardening/+/899286/2/ | 11:13 |
| noonedeadpunk | jamesdenton: yeah, I also see like 6-9% improvement in execution speed | 11:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Disable GSSAPIAuthentication for SSH https://review.opendev.org/c/openstack/ansible-hardening/+/899286 | 11:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: Disable dynamic motd message https://review.opendev.org/c/openstack/ansible-hardening/+/899370 | 11:15 |
| noonedeadpunk | these should be covering it now ^ | 11:16 |
| jamesdenton | nice find | 11:17 |
| jrosser | it's like two layers of retry though | 11:21 |
| jrosser | we have one around exec_command in our plugin | 11:21 |
| jrosser | then inside the native SSH plugin, exec_command uses _run which retries also.... https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/ssh.py#L1223 | 11:21 |
| noonedeadpunk | oh, yes, true | 11:28 |
| noonedeadpunk | That indeed looks wrong then | 11:28 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Add gate_log_requirements function https://review.opendev.org/c/openstack/openstack-ansible/+/899158 | 11:37 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/898924 | 11:58 |
| opendevreview | Merged openstack/openstack-ansible-os_trove master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/898931 | 11:58 |
| opendevreview | Merged openstack/openstack-ansible-os_cloudkitty master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/898816 | 11:59 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Switch codename to Bobcat https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/899294 | 12:00 |
| opendevreview | Merged openstack/openstack-ansible-os_senlin master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/898926 | 12:02 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Calculate if target is a container only once https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/899162 | 12:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper master: Use jdk 17 for Zookeeper https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/899386 | 12:07 |
| opendevreview | Merged openstack/openstack-ansible-os_masakari master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/898912 | 12:08 |
| noonedeadpunk | well, on tasks that are not chaning things there's no difference with our connection plugin | 12:08 |
| opendevreview | Merged openstack/openstack-ansible-os_heat master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/898908 | 12:10 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/898909 | 12:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Remove retries decorator from ssh plugin https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/899393 | 12:30 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/zed: Use internal endpoint for barbican API https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/899043 | 12:33 |
| opendevreview | Merged openstack/ansible-hardening master: Disable GSSAPIAuthentication for SSH https://review.opendev.org/c/openstack/ansible-hardening/+/899286 | 13:45 |
| jrosser | noonedeadpunk: i guess we should backport some of these performance things so they take effect for the whole of an upgrade job? | 14:00 |
| noonedeadpunk | yeah | 14:01 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-hardening stable/2023.1: Disable GSSAPIAuthentication for SSH https://review.opendev.org/c/openstack/ansible-hardening/+/899326 | 14:02 |
| noonedeadpunk | I kinda really thinking about diabling building wheels for metal deployments | 14:12 |
| noonedeadpunk | and test that in LXC only, where we can afford to run LXC | 14:13 |
| noonedeadpunk | as wheels on metal is kinda useless as well - some other role can easily provide requirements that not present in another one | 14:13 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Remove nspawn container support https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/899163 | 14:27 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/yoga: Use internal endpoint for barbican API https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/899044 | 14:38 |
| opendevreview | Merged openstack/ansible-hardening master: Disable dynamic motd message https://review.opendev.org/c/openstack/ansible-hardening/+/899370 | 14:50 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-hardening stable/2023.1: Disable dynamic motd message https://review.opendev.org/c/openstack/ansible-hardening/+/899328 | 14:52 |
| noonedeadpunk | These results look amazing to have that said: https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 14:53 |
| jamesdenton | how much did you shave off? | 14:57 |
| noonedeadpunk | looks like around 20mins | 14:58 |
| jamesdenton | not too shabby | 15:00 |
| jrosser | it's likley worth another pass across all the utility type roles to see if we can filter loops / remove tasks / reduce skipping etc | 15:00 |
| jrosser | systemd_service was bothering me as it deals with service / socket / timer in the same code path | 15:03 |
| noonedeadpunk | yeah, that is actually another thing to look into... | 15:06 |
| noonedeadpunk | But I wonder how to get to the same result without DNMs.... | 15:08 |
| noonedeadpunk | Well. If we revert 1 patch that dropped SSH from all containers... | 15:09 |
| noonedeadpunk | We could drop our connection plugin.... | 15:09 |
| jrosser | this must be able to be collapsed https://github.com/openstack/ansible-role-systemd_service/blob/master/tasks/main.yml#L16-L69 | 15:09 |
| jrosser | we can filter the list rather than when: https://github.com/openstack/ansible-role-systemd_service/blob/master/tasks/main.yml#L100 | 15:10 |
| jrosser | same https://github.com/openstack/ansible-role-systemd_service/blob/master/tasks/main.yml#L118 | 15:10 |
| jrosser | again :) https://github.com/openstack/ansible-role-systemd_service/blob/master/tasks/main.yml#L132 | 15:10 |
| noonedeadpunk | ah, yeah, so have just 1 task placing templates | 15:15 |
| noonedeadpunk | or well. filter | 15:16 |
| noonedeadpunk | I was just thinking that list must for sure have at least 1 of these templates, so we can just loop with_together or smth like that... And then place tempalte only if condition is satisfied from the other one... But yeah, I guess filter is cleaner | 15:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM: Try to tune SSH in pre-step. https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 15:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Tune SSH in pre-step setup https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 15:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Tune SSH in pre-step setup https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 15:30 |
| opendevreview | Merged openstack/ansible-hardening stable/2023.1: Disable GSSAPIAuthentication for SSH https://review.opendev.org/c/openstack/ansible-hardening/+/899326 | 15:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Track stable/2023.2 SHAs for upstream projects https://review.opendev.org/c/openstack/openstack-ansible/+/897434 | 15:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/898818 | 16:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898073 | 16:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898074 | 16:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898075 | 16:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898074 | 16:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Map default value of rabbitmq_management_ssl to haproxy_ssl https://review.opendev.org/c/openstack/openstack-ansible/+/899416 | 16:28 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Cosmetic tidy up of pid lookup function https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/899164 | 16:47 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Remove extra container check https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/899165 | 16:47 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Retrieve container name and physical host via get_options https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/899166 | 16:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Tune SSH in pre-step setup https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 16:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Disable wheels build for metal AIO deployments https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 17:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Disable wheels build for metal AIO deployments https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 17:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Disable wheels build for metal AIO deployments https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 17:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Disable wheels build for metal AIO deployments https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 17:08 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server stable/2023.1: Apply haproxy-service-config tag on include https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/898500 | 17:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper master: Add upgrade jobs for zookeeper https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/897754 | 17:32 |
| opendevreview | Merged openstack/ansible-hardening stable/2023.1: Disable dynamic motd message https://review.opendev.org/c/openstack/ansible-hardening/+/899328 | 17:33 |
| opendevreview | Merged openstack/openstack-ansible master: Track stable/2023.2 SHAs for upstream projects https://review.opendev.org/c/openstack/openstack-ansible/+/897434 | 23:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!