| noonedeadpunk | mornings | 06:52 |
|---|---|---|
| noonedeadpunk | Today in the morning I realized that likely we made a mistake by using openstack namespace for our collections... I am really thinking if it's worth to change that | 06:53 |
| noonedeadpunk | and then, it will open doors for couple of things | 06:53 |
| noonedeadpunk | 1. We can create osa.playbooks collection and move there all our playbooks, and maybe even dynamic inventory? And then users shouldn't care about where they've cloned osa to - they will just run `openstack-ansible osa.playbook.nova-setup` for example | 06:54 |
| noonedeadpunk | then we will be able to re-name the playbooks to more reflect that it's not only installation, but also about day2 | 06:55 |
| noonedeadpunk | And in integrated repo we can actually leave current playbooks for compatability, but they will just contain import of osa.playbooks.nova | 06:56 |
| noonedeadpunk | 2. We can create a collection where to pack all systemd things and make an independent release model for it, as they pretty much don't change. And have like osa.systemd.service role and osa.systemd.networkd, etc | 06:57 |
| noonedeadpunk | And current one will be just osa.plugins.* | 06:57 |
| noonedeadpunk | but then it's open question where to draw line between independent roles and collections | 07:03 |
| noonedeadpunk | but I have feeling that smth like osa.playbooks would be really beneficial | 07:05 |
| noonedeadpunk | since we're in PTG week we can gather quickly this week to potentially discuss that and raise opinions:) | 07:15 |
| jrosser | morning | 08:28 |
| jrosser | is the collection issue that we only get to have one currently (openstack.osa) ? | 08:28 |
| noonedeadpunk | Well. We probably can have more, as we don't publish (as of today), but I guess then it's a bit question of naming | 08:33 |
| noonedeadpunk | like openstack.osa-ops can be a thing | 08:33 |
| noonedeadpunk | and then openstack.osa_playbooks (sorry, there can't be `-` in collection names, only underscores) | 08:34 |
| noonedeadpunk | but naming will pretty much suck | 08:43 |
| noonedeadpunk | or well. we can do ofc openstack.ansible for playbooks... | 08:44 |
| noonedeadpunk | like `openstack-ansible openstack.ansible.nova`. That looks cool. A bit ambitious though, but well :) | 08:45 |
| noonedeadpunk | catchy | 08:45 |
| noonedeadpunk | So well. We might be fine with namespace if we kinda find a good way of naming things in it | 08:47 |
| noonedeadpunk | and we actually can even place playbooks in oopenstack-ansible-plugins | 08:47 |
| noonedeadpunk | though I'd rather manage these things independently I guess | 08:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Refactor log collection from lxc containers https://review.opendev.org/c/openstack/openstack-ansible/+/877382 | 09:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not try to install packages on each log_instance_info run https://review.opendev.org/c/openstack/openstack-ansible/+/866372 | 09:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not try to install packages on each log_instance_info run https://review.opendev.org/c/openstack/openstack-ansible/+/866372 | 09:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tacker master: Fix linters for example playbook https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/899241 | 09:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tacker master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/898930 | 09:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_mistral master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/899242 | 09:47 |
| jrosser | huh https://git.afpy.org/mdk/ansible-parallel | 09:49 |
| noonedeadpunk | that;s interesting:) | 09:54 |
| noonedeadpunk | except unreadable symbols in code.... | 09:54 |
| noonedeadpunk | and with thinking about smth like that to be frank, though was also interested in trying out https://ansible.readthedocs.io/projects/runner/en/latest/ instead of exec ansible-playbook directly | 10:01 |
| noonedeadpunk | like you have run_asycn already there: https://ansible.readthedocs.io/projects/runner/en/latest/python_interface/#run-async-helper-function | 10:01 |
| noonedeadpunk | as that should help you out with parsing exit codes a lot | 10:02 |
| jrosser | interesting - looks like a quite few options here | 10:05 |
| noonedeadpunk | interesting would be to get osa running in AWX actually.... | 10:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Refactor log collection from lxc containers https://review.opendev.org/c/openstack/openstack-ansible/+/877382 | 10:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_mistral master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/898913 | 10:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Fix linters for example playbook https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/899246 | 10:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/898929 | 10:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/898929 | 10:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/898922 | 10:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Add example playbook and linters job to the role https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/899247 | 10:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Fix linters for example playbook https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/899248 | 10:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/898926 | 10:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/898908 | 10:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/899249 | 10:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/898908 | 10:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ceilometer master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/898814 | 10:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ceilometer master: Add quorum support for service https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/895696 | 10:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ceilometer master: Enable Ceilometer resource cache https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/888032 | 10:34 |
| noonedeadpunk | So, mds setup in ceph-ansible is broken for ansible-core 2.15: https://github.com/ceph/ceph-ansible/pull/7466/files | 12:13 |
| noonedeadpunk | thus, manila is failing there | 12:13 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Bump ansible version to 2.15.5 https://review.opendev.org/c/openstack/openstack-ansible/+/899257 | 12:14 |
| jrosser | i also saw that we have ceph failing elsewhere | 12:14 |
| jrosser | https://review.opendev.org/c/openstack/openstack-ansible/+/893968 | 12:15 |
| jrosser | due to the way we constantly overwrote the tempest test vars (fixed by that patch) it might be this is broken for a while | 12:16 |
| jrosser | thats getting 503 on rgw but i've not been able to look at that yet | 12:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Refactor log collection from lxc containers https://review.opendev.org/c/openstack/openstack-ansible/+/877382 | 12:20 |
| noonedeadpunk | I can spawn an aio and have a look | 12:27 |
| opendevreview | Merged openstack/openstack-ansible-plugins master: Simplfy addition of keystone users to roles https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/896017 | 12:55 |
| jamesdenton | maybe this is a dumb question, but is there anything preventing running multiple roles/playbooks in parallel? Especially if there's no dependencies between them? | 13:01 |
| jrosser | did you read back up there ^^^^ ? | 13:01 |
| jamesdenton | no | 13:01 |
| jrosser | some discussion already today about that, and seems theres a bunch of approaches | 13:01 |
| jamesdenton | oh hah | 13:02 |
| jamesdenton | i will scrollback | 13:02 |
| jrosser | theres not really a reason we can't | 13:02 |
| jrosser | so long as we observe things like haproxy must go first, keystone must go first etc etc | 13:02 |
| jamesdenton | right, setup-infra + keystone, then spawn a few concurrent playbook runs | 13:03 |
| jrosser | even withing setup infra i think we can parallelise the roles | 13:03 |
| jamesdenton | yeah, potentially. i wouldn't know where to start but glad to know there's been some thought about that | 13:03 |
| jrosser | so long as haproxy is done before we try to do galera, for example | 13:03 |
| noonedeadpunk | I think we also discussed that yesterday in zoom | 13:03 |
| jamesdenton | maybe this is my subconscious bubbling up | 13:04 |
| jamesdenton | or this is the manager in me acting as though your ideas are my own :D | 13:04 |
| jrosser | we discussed performance also today here | 13:06 |
| jrosser | and went over again how there are conflicting requirements, like for CI its all about control plane | 13:06 |
| jrosser | but for real deployments what you actually want better is nova+neutron | 13:06 |
| jamesdenton | speaking of CI... if i wanted to test OSA builds against a new pool of servers (such as an internal cloud) what would be needed to do that? | 13:07 |
| jrosser | noonedeadpunk: check this out https://review.opendev.org/q/topic:bgp-exp (you too jamesdenton) | 13:08 |
| jrosser | relevant to discussions from PTG of ampora BGP sessions | 13:09 |
| jamesdenton | interesting. is there a spec? | 13:09 |
| jrosser | i think so, yes | 13:09 |
| noonedeadpunk | ugh | 13:09 |
| jamesdenton | was this it? https://docs.openstack.org/octavia/latest/contributor/specs/version1.1/active-active-l3-distributor.html | 13:09 |
| noonedeadpunk | means we're late with our patch that we were supposed to push like couple of weeks ago | 13:10 |
| jamesdenton | iirc you weren't thrilled with the implementation, right? because it required peering from tenant network? | 13:10 |
| jamesdenton | n/m it says peering from lb-mgmt-net | 13:11 |
| jrosser | yeah, i guess i prefer the approach like neutron bgp agent | 13:11 |
| jrosser | looking at the ascii diagram the peering is "external" | 13:12 |
| jamesdenton | yeah, that's at least predictable. here it looks like you need to enable dynamic peering by subnet cidr | 13:12 |
| jrosser | oh actually, [P2] looks different to what i remember | 13:13 |
| jrosser | i would kind of expect to see some BFD thrown in there somewhere in order to get routes withdrawn real quick when an amphora goes away for whatever reason | 13:16 |
| jrosser | jamesdenton> speaking of CI... if i wanted to test OSA builds against a new pool of servers (such as an internal cloud) what would be needed to do that? | 13:17 |
| jrosser | ^ you mean more than just running an AIO? | 13:18 |
| jamesdenton | well, i guess replicating a zuul would be the first step? | 13:18 |
| jamesdenton | yeah, i'm not really sure what i'm asking TBH. | 13:18 |
| jrosser | depends if you want to precisely replicate the actual CI | 13:19 |
| jrosser | or just automatically run an AIO which is a bit different | 13:19 |
| jrosser | we have people here who use github actions with on-prem runners in openstack VM, which is also a form of CI too | 13:20 |
| jamesdenton | really i just need to load test some new regions i'm spinning up, thinking of the best way to do that. Might just be to use existing TF stuff i have | 13:20 |
| jamesdenton | maybe thats the way to go, i've never done that | 13:20 |
| jrosser | but yeah, load test might be more like TF as you say | 13:20 |
| jrosser | as you'd be able to parameterise that pretty easily | 13:21 |
| NeilHanlon | So, I started trying to package incus yesterday because I was... well, not bored, but, procrastinating.. | 13:40 |
| NeilHanlon | it didn't go well 😂 | 13:40 |
| noonedeadpunk | I see :D | 13:43 |
| noonedeadpunk | sounds not super promising | 13:43 |
| mgariepy | it's version 0.1 ;p | 13:43 |
| mgariepy | i guess the ubuntu/debian build works as expected. | 13:44 |
| mgariepy | haven't had time to play with it tho. only tested some arch user pkg and some config were missing. | 13:45 |
| noonedeadpunk | they have a repo for ubuntu already... So I guess it does | 13:45 |
| mgariepy | Stéphane Grabber is building it. and it's running on the incus demo website. | 13:46 |
| mgariepy | https://linuxcontainers.org/incus/try-it/ | 13:46 |
| jrosser | looks like we might be able to save some tasks in the PKI role | 13:51 |
| jrosser | ensuring that dirs exist way more times than really needed | 13:51 |
| mgariepy | i'm a bit too ctrl+w happy to use a linux shell inside web browser lol. | 13:51 |
| noonedeadpunk | oh yes | 13:52 |
| noonedeadpunk | that's always a case for me with IPMI consoles :D | 13:52 |
| noonedeadpunk | can't really resist from doing ctrl+w | 13:53 |
| mgariepy | glad i'm not alone :) haha if you find a cure please tell me. | 13:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not try to install packages on each log_instance_info run https://review.opendev.org/c/openstack/openstack-ansible/+/866372 | 13:57 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-pki master: Simplify PKI host directory creation https://review.opendev.org/c/openstack/ansible-role-pki/+/899269 | 13:57 |
| opendevreview | Merged openstack/openstack-ansible master: Add CI jobs for debian bookworm https://review.opendev.org/c/openstack/openstack-ansible/+/894561 | 14:02 |
| NeilHanlon | noonedeadpunk: yeah, i think it will get better.. the build process right now is a bit "wtf" to me so, i am gonna hold off on trying to build a proper package for now... lol | 14:05 |
| mgariepy | NeilHanlon, you probably can poke de dev on libera-chat #lxc | 14:05 |
| NeilHanlon | mgariepy: good call, thanks | 14:06 |
| mgariepy | poke stgraber ;) | 14:06 |
| noonedeadpunk | the last patch left for finishing debian 12:) https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/898724 | 14:48 |
| noonedeadpunk | (or about the last) | 14:49 |
| noonedeadpunk | and also some ready patches for quorum queues https://review.opendev.org/q/topic:osa/quorum_queues+status:open+label:Verified | 15:31 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-pki master: Only create certificate destination directories once. https://review.opendev.org/c/openstack/ansible-role-pki/+/899279 | 15:32 |
| noonedeadpunk | that patch doesn't looks like having any significant decrease of runtime: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/899166 :( | 15:58 |
| noonedeadpunk | as it was quite promising | 15:58 |
| noonedeadpunk | unless it's not properly tested.... | 15:59 |
| jrosser | no sadly it does not seem to have much impact | 16:15 |
| jrosser | but actually the connection plugin code size is really reduced with these patches | 16:15 |
| jrosser | which is understandability++ | 16:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Adopt for usage openstack_resources role https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/899284 | 16:20 |
| noonedeadpunk | yeah. true, that is still good improvement, I just had some hopes, that doing if/else 1M times instead of 5M times will save at least some time | 16:22 |
| jrosser | i did also try removing the ssh roundtrip to get the container PID and instead mount /root from each container into /openstack/roots/<container_name> | 16:30 |
| jrosser | but somehow that didnt really give a noticeable improvement ether | 16:30 |
| noonedeadpunk | oh, well, I think I have found smth | 16:32 |
| noonedeadpunk | https://paste.openstack.org/show/bEyZD3APh1X264o7zXSY/ | 16:33 |
| noonedeadpunk | And I made literally 1 change | 16:33 |
| noonedeadpunk | and I bet you might easily guess what the change it was... | 16:33 |
| noonedeadpunk | will try to come with smth relatively sustainable... | 16:38 |
| noonedeadpunk | so 400ms on each connection might have dramatical difference | 16:40 |
| jamesdenton | you disable reverse dns lookup? | 16:46 |
| jamesdenton | curious :) | 16:46 |
| jrosser | its always dns | 16:46 |
| jrosser | i wonder if whatever that is also makes a difference with controlpersist | 16:47 |
| noonedeadpunk | MOTD | 16:49 |
| jamesdenton | hah | 16:49 |
| noonedeadpunk | the dynamic one which comes from pam.d | 16:49 |
| noonedeadpunk | static one we put there has no difference at all | 16:50 |
| noonedeadpunk | I'm not sure if upstream images do have that kind of thing though | 16:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: DNM Check if disabling PAM speedup things https://review.opendev.org/c/openstack/ansible-hardening/+/899286 | 17:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: DNM https://review.opendev.org/c/openstack/openstack-ansible/+/899287 | 17:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM https://review.opendev.org/c/openstack/openstack-ansible/+/899288 | 17:03 |
| noonedeadpunk | I can imagine that motd is indeed not part of the CI images. Though looks like in real world it can speedup things a lot | 17:04 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Use relevant osbpo repository for Debian 12 https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/898724 | 17:20 |
| NeilHanlon | btw rocky just built the centos-release-openstack-bobcat packages and it should be available on mirrors shortly (for any distro-type jobs we might have whenever) | 17:47 |
| NeilHanlon | (h/t mnasiadka for the request) | 17:48 |
| noonedeadpunk | NeilHanlon: yeah, I know, there was a ML regarding that yesterday. But we kinda need to have UCA as well before switching to that | 18:07 |
| noonedeadpunk | oh, they already have bobcat, sweet | 18:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Switch codename to Bobcat https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/899294 | 18:11 |
| noonedeadpunk | here we go ^ | 18:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not try to install packages on each log_instance_info run https://review.opendev.org/c/openstack/openstack-ansible/+/866372 | 18:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Add gate_log_requirements function https://review.opendev.org/c/openstack/openstack-ansible/+/899158 | 18:24 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/899288/ gives actually interesting results - it's failing quite randomly on SSH connection | 18:26 |
| noonedeadpunk | but then they go to retry somehow | 18:27 |
| noonedeadpunk | as I guess zuul uses pam? but super werd as rhel have it commented out... and only keypairs should be used | 18:31 |
| noonedeadpunk | but things fail on image upload in tempest at worst | 18:32 |
| noonedeadpunk | sad I won;t get any time which it took to fail even... | 18:32 |
| jrosser | we have retry in the connection plugin | 18:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-hardening master: DNM Check if disabling PAM speedup things https://review.opendev.org/c/openstack/ansible-hardening/+/899286 | 18:48 |
| noonedeadpunk | well. it shouldn't retry zuul jobs after task fails :) | 18:49 |
| noonedeadpunk | so it indeed looks like slow VMs in CI: https://paste.openstack.org/show/bMYzMdECdOeeZqGVjcOb/ | 19:39 |
| noonedeadpunk | Like 1 minute wasted here literally nowhere | 19:39 |
| noonedeadpunk | unless these were wheels that built.... | 19:39 |
| nixbuilder | Hello... I am installing from the 27.1.0 scripts and am getting an haproxy deprecation error... the error is "'option httpchk' : hiding headers or body at the end of the version string is deprecated. Please, consider to use 'http-check send' directive instead." So I get deprecation errors and the install fails because of errors found in configuration. Has anyone else seen this? | 19:40 |
| noonedeadpunk | nixbuilder: do you have any custom haproxy endpoints defined? Or that is clean installation which is pretty much default one? | 19:41 |
| noonedeadpunk | Also what OS are you running? | 19:41 |
| noonedeadpunk | As I don't see "option httpchk" anywhere in our code for 27.1.0 | 19:42 |
| noonedeadpunk | ah, sorry I do see them | 19:42 |
| nixbuilder | OS is Rocky 9 and I do not have any custom haproxy endpoints defined. This is a clean bare metal install. | 19:43 |
| noonedeadpunk | ok, I see. So we have patched that on master: https://review.opendev.org/c/openstack/openstack-ansible/+/887285 | 19:44 |
| noonedeadpunk | But I don't think it would raise an error to be frank - it's a noise but it should not hurt I believe... | 19:44 |
| noonedeadpunk | So can you kindly paste error? | 19:44 |
| noonedeadpunk | you can use https://paste.openstack.org/ for that | 19:45 |
| nixbuilder | https://paste.openstack.org/show/bjIvLlf8vqEwdWF0LXN4/ | 19:47 |
| jrosser | that is just a deprecation warning from haproxy | 19:48 |
| jrosser | it's not a failure | 19:48 |
| noonedeadpunk | ` 'bind 10.255.60.2:15671' : unable to stat SSL certificate from file '/etc/haproxy/ssl/haproxy_infra01-10.255.60.2.pem' : No such file or directory` | 19:48 |
| noonedeadpunk | that is the failure | 19:48 |
| noonedeadpunk | I think, that could be some leftover from previous runs, if you happen to change IPs or some services? | 19:49 |
| * noonedeadpunk writing some suggestion | 19:51 | |
| nixbuilder | No... fresh install. Hmmm... the '/etc/haproxy/ssl/' is empty :-( | 19:52 |
| noonedeadpunk | try smth like: 1. ansible -m file -a "path=/etc/haproxy/conf.d state=absent" haproxy_all | 19:53 |
| noonedeadpunk | 2. openstack-ansible playbooks/haproxy-install.yml --tags haproxy_server-config,haproxy-config | 19:54 |
| noonedeadpunk | 3. openstack-ansible playbooks/setup-infrastructure.yml --tags haproxy-service-config | 19:56 |
| noonedeadpunk | nixbuilder: certificates should be stored in /etc/openstack_deploy/pki folder | 19:56 |
| noonedeadpunk | so they're generated there and then placed on destination hsots | 19:57 |
| nixbuilder | noonedeadpunk: I double-checked my user_variables.yml file and I have 'haproxy_ssl: false'. So I didn't think certificates would have been required. | 19:57 |
| noonedeadpunk | ah. | 19:58 |
| noonedeadpunk | well... it should not fail then.... | 19:59 |
| noonedeadpunk | I think we have... imperfectess... | 20:00 |
| nixbuilder | noonedeadpunk: The only certs in '/etc/openstack_deploy/pki' are for galera | 20:00 |
| noonedeadpunk | nixbuilder: can you try adding `rabbitmq_management_ssl: false`? | 20:00 |
| nixbuilder | Sure. | 20:01 |
| noonedeadpunk | somehow we don't respect haproxy_ssl here for rabbitmq: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/rabbitmq_all.yml#L41-L42 | 20:03 |
| noonedeadpunk | nixbuilder: and rpobably you can disregard my previous commands | 20:04 |
| noonedeadpunk | just define rabbitmq_management_ssl and re-run rabbitmq-install.yml playbook | 20:04 |
| noonedeadpunk | jrosser: really weird things I see when I attach to zuul consoles. Like things being stuck between tasks | 20:08 |
| noonedeadpunk | and pretty much always like for 69 seconds or smth like that | 20:09 |
| noonedeadpunk | or it's renderring weirdly... or I am missing smth... | 20:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM check if having ARA or not affects job runtime https://review.opendev.org/c/openstack/openstack-ansible/+/899309 | 20:13 |
| noonedeadpunk | or there's really smth off with SSH | 20:14 |
| nixbuilder | noonedeadpunk: Thanks... that fixed it! | 20:36 |
| noonedeadpunk | ok, awesome. I guess we need to patch that though... | 20:37 |
| jrosser | noonedeadpunk: on a metal job i guess we can also check without our connection plugin? | 20:38 |
| noonedeadpunk | I think we did that | 20:39 |
| noonedeadpunk | (I guess it was damiandabrowski tbh) | 20:39 |
| noonedeadpunk | and there was no difference iirc | 20:39 |
| jrosser | interesting | 20:39 |
| jrosser | i made a "lots.yml" which did copy: N times | 20:40 |
| jrosser | and there was a difference between our plugin and not with that | 20:40 |
| jrosser | but it was not huge, like 40s vs 35s or something | 20:40 |
| jrosser | that was with zero vars as well | 20:41 |
| noonedeadpunk | yeah, I guess in CI there was no visible difference after all | 20:41 |
| noonedeadpunk | Though I'm looking at this DNM patch and it feels like being slightly faster on average... | 20:41 |
| noonedeadpunk | But I do really see things just being stuck for 60sec... | 20:42 |
| jrosser | which one? | 20:42 |
| damiandabrowski | I only removed openstack.osa.linear plugin: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/874425 | 20:42 |
| noonedeadpunk | ah, ok, righy | 20:42 |
| noonedeadpunk | jrosser: basically this https://review.opendev.org/c/openstack/openstack-ansible/+/899288 (but changes are in dependent one) | 20:43 |
| jrosser | ah ok | 20:43 |
| noonedeadpunk | but config like that looks like... smth that can cause such 1m hold on connection: https://zuul.opendev.org/t/openstack/build/4e911b59ed7340d9bb098b3904fd2a01/log/logs/etc/host/ssh/sshd_config.txt#156-157 | 20:44 |
| jrosser | i wonder if sshd logs any of that | 20:45 |
| noonedeadpunk | but like this job was indeed fast https://zuul.opendev.org/t/openstack/build/7990e26401c945499fae2f2f1c4c2cee | 20:45 |
| noonedeadpunk | but we didn't have bookworm jobs to compare... | 20:46 |
| noonedeadpunk | But I barely saw jobs that finish less then 1:20 or smth in a while... | 20:47 |
| jrosser | grafana will be insightful for that | 20:47 |
| jrosser | we need to update that again too | 20:47 |
| noonedeadpunk | well, there're some port scanning happening on the vm in CI | 20:53 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/f7fb69eeaa6e488d8f64a914eb0d82a7/log/logs/host/auth.log.txt#1512-1517 | 20:56 |
| noonedeadpunk | but probably not too much.... | 20:56 |
| noonedeadpunk | disabled pam in zuul looked really fast btw. | 21:03 |
| noonedeadpunk | though it was failing as well... | 21:04 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/897350 | 21:08 |
| noonedeadpunk | but what I know for sure is that gssapi is enabled in CI images that for sure slow down things a lot | 21:09 |
| noonedeadpunk | As i've seen on some failures today that auth failed with gssapi,publickey,password | 21:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM test metal CI speed without connection plugin https://review.opendev.org/c/openstack/openstack-ansible/+/899312 | 21:16 |
| jrosser | those random ssh attempts could be using up auth slots in the sshd config | 21:38 |
| noonedeadpunk | Yeah, I was kinda thinking the same... there was not lot of them though... but maybe just because of limiting, given that ansible kinda already consumes quite some | 21:39 |
| jrosser | look also in the connection plugin, if we get failure to connect it has exponential back off retries | 21:41 |
| noonedeadpunk | do we log these though? | 21:41 |
| jrosser | that might both hide things and cause delay at the same time | 21:42 |
| noonedeadpunk | I'm looking now in https://zuul.opendev.org/t/openstack/stream/5780246de5c3496ca1345222d01bb047?logfile=console.log which jsut was stuck for 10 mins already | 21:42 |
| jrosser | only with lots of -v | 21:42 |
| noonedeadpunk | https://paste.openstack.org/show/bzX1GB5o7BoRPmkbHmsi/ | 21:42 |
| noonedeadpunk | ah lol | 21:42 |
| noonedeadpunk | it was executing tempest | 21:42 |
| noonedeadpunk | I guess I'm jsut getting confused with that task name and result are show together | 21:43 |
| noonedeadpunk | while i nreal world you will see task header and then status will follow | 21:43 |
| noonedeadpunk | but then, I've spotted that https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L110-L113 takes quite some time on each run | 21:44 |
| jrosser | maybe something is horribly inefficient to delete many files | 21:45 |
| noonedeadpunk | ugh, no, forget about it, it's again output that confused me... | 21:47 |
| noonedeadpunk | next task is building wheels and that what takes time | 21:47 |
| jrosser | those time stamps are confusing | 21:50 |
| noonedeadpunk | I'm not 100% sure, but that looks slightly better then last couple of results on avergae https://review.opendev.org/c/openstack/openstack-ansible/+/899288 | 21:50 |
| noonedeadpunk | but again - we could be lucky with the load in CI... | 21:50 |
| noonedeadpunk | like 1 upgrade is less then 2h.. | 21:51 |
| noonedeadpunk | but I don't think it has smth to do with the patch | 21:51 |
| noonedeadpunk | and then 1 llxc is almost 3h | 21:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM: Try to tune SSH in pre-step. https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 22:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM Disable wheels build https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 22:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM: Try to tune SSH in pre-step. https://review.opendev.org/c/openstack/openstack-ansible/+/899318 | 22:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM Disable wheels build https://review.opendev.org/c/openstack/openstack-ansible/+/899319 | 22:26 |
| noonedeadpunk | huh, not using our connection plugin looks like a visable improvement.... | 22:38 |
| noonedeadpunk | maybe too earl to judge though... | 22:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!