| pwickham72 | Hi further to my previous message, i found the cause. I had set the transport to paramiko in ansible.cfg for some reason. When i removed it, it worked :-) | 00:36 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Switch to ansible-core and bump up to 2.17 https://review.opendev.org/c/openstack/kolla/+/924246 | 06:53 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Switch to ansible-core and bump up to 2.17 https://review.opendev.org/c/openstack/kolla/+/924246 | 06:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 06:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Add ansible-core as a dependency https://review.opendev.org/c/openstack/kolla-ansible/+/922369 | 06:59 |
| sylvr | Good morning #openstack-kolla ! | 07:51 |
| opendevreview | Maximilian Stinsky proposed openstack/kolla master: Add Dockerfile for neutron-ovn-vpn-agent https://review.opendev.org/c/openstack/kolla/+/924302 | 07:51 |
| sylvr | jovial: I think it's possible that kayobe doesn't provide the default ipmi credentials when writing the introspection rules for Bifrost | 07:54 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 07:56 |
| sylvr | to be more thorough, I think that the info in inspector.yml aren't used when configuring Bifrost (inspector.yml should be used by Bifrost and the Ironic Overcloud right?) | 07:56 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Implement TLS for Redis https://review.opendev.org/c/openstack/kolla-ansible/+/909188 | 08:18 |
| opendevreview | Maximilian Stinsky proposed openstack/kolla master: Add Dockerfile for neutron-ovn-vpn-agent https://review.opendev.org/c/openstack/kolla/+/924302 | 08:55 |
| SvenKieske | mhmm | 08:59 |
| SvenKieske | _maybe_ this new neutron code is problematic, because it assumes IPv6 to be always present, I didn't test it myself just yet (I need to read up on the neutron rootwrap implementation): https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353/36/etc/neutron/rootwrap.d/vpnaas.filters#16 | 09:00 |
| SvenKieske | it basically sets sysctl values unconditionally for ipv6 and ipv4, which might throw an error, if some neutron magic doesn't catch it | 09:01 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default. https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 09:29 |
| SvenKieske | alright, the code handles it: https://opendev.org/openstack/neutron-vpnaas/src/commit/256464aea691f8b4957ba668a117963353f34e4c/neutron_vpnaas/services/vpn/device_drivers/ovn_ipsec.py#L194 | 09:37 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 10:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 10:05 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix mariadb recovery https://review.opendev.org/c/openstack/kolla-ansible/+/920304 | 10:12 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL https://review.opendev.org/c/openstack/kolla-ansible/+/909912 | 10:21 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add ansible-core as a dependency https://review.opendev.org/c/openstack/kolla-ansible/+/922369 | 10:32 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible stable/2023.1: [2023.1 only] Add upgrade to target version of RMQ https://review.opendev.org/c/openstack/kolla-ansible/+/918976 | 10:41 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible stable/2024.1: Add a precheck to catch RMQ SLURP upgrades https://review.opendev.org/c/openstack/kolla-ansible/+/918977 | 10:45 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible stable/2024.1: Add ca_path for module uri in opensearch role https://review.opendev.org/c/openstack/kolla-ansible/+/924320 | 10:47 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible stable/2023.2: Add ca_path for module uri in opensearch role https://review.opendev.org/c/openstack/kolla-ansible/+/924321 | 10:48 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible stable/2023.1: Add ca_path for module uri in opensearch role https://review.opendev.org/c/openstack/kolla-ansible/+/924322 | 10:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 10:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: test interface names with dashes https://review.opendev.org/c/openstack/kolla-ansible/+/794545 | 11:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 11:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Switch to Ubuntu 24.04 https://review.opendev.org/c/openstack/kolla/+/907589 | 11:35 |
| kevko | hmm, does anybody has currently installed 2024.1 ? | 11:46 |
| kevko | i don't know why but my rabbitmq reporting unhealthy ..but it seems it's everything ok | 11:47 |
| mnasiadka | not yet, we'll be rather upgrading sometime in fall this year | 11:47 |
| kevko | (rabbitmq)[rabbitmq@controller0 /]$ rabbitmq-diagnostics -t 5 -q status | 11:47 |
| kevko | Error: operation status on node rabbit@controller0 timed out. Timeout value used: 5000 | 11:47 |
| kevko | weird | 11:48 |
| kevko | but everything working as expected :D | 11:48 |
| opendevreview | Merged openstack/kolla stable/2024.1: fix: apple silicon architecture selection https://review.opendev.org/c/openstack/kolla/+/924249 | 11:55 |
| opendevreview | Merged openstack/kolla master: [docs] remove not working dev setup https://review.opendev.org/c/openstack/kolla/+/922079 | 11:55 |
| sylvr | do anyone else experience broken autodiscovery on 2024.1 with IPMI driver ? | 11:55 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 11:59 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Switch to Ubuntu 24.04 https://review.opendev.org/c/openstack/kolla/+/907589 | 12:02 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Add mechanism for patching files in containers https://review.opendev.org/c/openstack/kolla/+/829295 | 12:26 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Add macro for patch files to all images https://review.opendev.org/c/openstack/kolla/+/829296 | 12:26 |
| SvenKieske | mnasiadka: bbezak: meeting? :) | 13:00 |
| SvenKieske | or is it canceled? | 13:00 |
| SvenKieske | mhm I didn't see a mail at least, checking.. | 13:01 |
| SvenKieske | kevko: you happen to know if we have a meeting today? :D | 13:01 |
| mnasiadka | sorry | 13:02 |
| mnasiadka | #startmeeting kolla | 13:02 |
| opendevmeet | Meeting started Wed Jul 17 13:02:52 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:02 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:02 |
| mnasiadka | #topic rollcall | 13:02 |
| mnasiadka | o/ | 13:02 |
| SvenKieske | o/ | 13:02 |
| mhiner | o/ | 13:03 |
| frickler | \o | 13:03 |
| mnasiadka | #topic agenda | 13:04 |
| mnasiadka | * CI status | 13:04 |
| mnasiadka | * Release tasks | 13:04 |
| mnasiadka | * Current cycle planning | 13:04 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:04 |
| mnasiadka | * Open discussion | 13:04 |
| mnasiadka | #topic CI status | 13:04 |
| mnasiadka | I haven't seen any serious issues | 13:04 |
| mnasiadka | #topic Release tasks | 13:04 |
| mnasiadka | It's R-15 - nothing related to us | 13:04 |
| mnasiadka | #topic Current cycle planning | 13:05 |
| mnasiadka | I re-started work on Ubuntu 24.04 and Ansible bump | 13:05 |
| mnasiadka | Realised we've pinned bifrost Ansible version in Yoga and never bumped it up... | 13:05 |
| mnasiadka | Trying to do it in a way we don't need to manage the Ansible pin in Bifrost image | 13:05 |
| SvenKieske | is that a long term need, or is bifrost going to be fixed eventually? I don't recall the details why it's necessary, sorry. a link where I can read up would also suffice :) | 13:06 |
| mnasiadka | well, bifrost pins ansible versions each cycle but only in their env-setup.sh (and bifrost-cli) | 13:07 |
| mnasiadka | we can't use bifrost-cli, because that requires things like network_interface to be set | 13:07 |
| mnasiadka | and we're skipping bootstrap on image build | 13:07 |
| SvenKieske | ah, alright | 13:07 |
| SvenKieske | ty | 13:07 |
| mnasiadka | and env-setup.sh is failing because it removes epel first, and then tries to install it - and we try to use our sources | 13:07 |
| mnasiadka | I'll work it out somehow | 13:07 |
| mnasiadka | nothing else on the HAVE TO DO list on the whiteboard | 13:08 |
| mnasiadka | let's move on | 13:08 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:08 |
| mnasiadka | (r-krcek) on behalf of m.hiner please check if the docs are sufficient https://review.opendev.org/c/openstack/kolla-ansible/+/836941 | 13:08 |
| mnasiadka | I'm a bit worried about some things that are now worked around, and should be done in separate patches | 13:09 |
| mhiner | mnasiadka: if you have time, could you please go reply to my replies on the patch? | 13:09 |
| mnasiadka | I've already commented on this | 13:09 |
| mnasiadka | Yeah, I'll reply there | 13:10 |
| mhiner | thank you | 13:10 |
| mnasiadka | but basically we don't need to launch the migration CI job every time | 13:10 |
| SvenKieske | docs themselves seem fine to me. there's new commentary not concerned with docs :) | 13:10 |
| mnasiadka | I assume it should be only run when some critical things for the migration itself change | 13:10 |
| mnasiadka | e.g. migration playbooks | 13:10 |
| mnasiadka | I would prefer to have more coverage than one distro | 13:10 |
| mnasiadka | And I would prefer if we have patches for things like https://review.opendev.org/c/openstack/kolla-ansible/+/836941/comment/3f2bfb23_9910ff49/ and https://review.opendev.org/c/openstack/kolla-ansible/+/836941/comment/204e192d_d764d8d5/ to merge everything at the same time | 13:11 |
| SvenKieske | yes, mhm, can we also run them when docker/podman are updated? might be hard to detect in CI though | 13:11 |
| mnasiadka | instead of reworking this later | 13:11 |
| mnasiadka | because I have a hunch it's going to stay like this for a long time ;-) | 13:11 |
| mhiner | Ivan has started to work on those patches | 13:12 |
| SvenKieske | afaik listing all containers was already pretty mature? | 13:12 |
| mhiner | yes but it was made to work in kolla_container but I think the functionality is better suited for kolla_container_facts module | 13:13 |
| mnasiadka | I'll reply on the patch after the meeting - but would like to see the final solution patches and merge everything together | 13:13 |
| mhiner | alright | 13:13 |
| mnasiadka | ok then | 13:14 |
| mnasiadka | no more topics on the whiteboard | 13:14 |
| mnasiadka | #topic Open discussion | 13:14 |
| chembervint | Hi, I just wanted to up my patch, do you have any questions on it? https://review.opendev.org/c/openstack/kolla-ansible/+/920377 | 13:15 |
| mhiner | Additionally, can you please review this patch: https://review.opendev.org/c/openstack/kolla-ansible/+/911417 | 13:16 |
| mhiner | The container names patch will be dependent on it as it adds actions to kolla_container_facts | 13:16 |
| mnasiadka | mhiner: will have a look | 13:17 |
| mhiner | thanks again | 13:17 |
| mnasiadka | chembervint: does it make any sense to not make 30minutes a default? | 13:17 |
| SvenKieske | if anybody cares about unmaintained/zed you might want to check the publish jobs, they could be broken: https://zuul.openstack.org/builds?project=openstack%2Fkolla&pipeline=periodic-weekly&result=FAILURE&result=RETRY_LIMIT&result=POST_FAILURE&result=NODE_FAILURE&result=SKIPPED&skip=0 | 13:17 |
| SvenKieske | kevko maybe? :) | 13:17 |
| mnasiadka | it's unmaintained, we don't care | 13:18 |
| mnasiadka | but it seems the new password hasn't been backported to zed | 13:18 |
| SvenKieske | regarding https://review.opendev.org/c/openstack/kolla-ansible/+/911417 I'd like to have some more senior opinions if we should backport it, not quite sure. | 13:18 |
| mnasiadka | I would rather stop publishing | 13:19 |
| mnasiadka | since it's unmaintained | 13:19 |
| mnasiadka | any volunteer to stop publish jobs on yoga and zed? | 13:19 |
| chembervint | mnasiadka: I've proposed it at the beginning, and then regarding on your comment moved it into variable, and has added Docs around it. but I don't see any concernes, if we will make default=30mins, but do it through var, to make customer able to tune it | 13:20 |
| mnasiadka | well, I'm not using this at all | 13:20 |
| mnasiadka | SvenKieske/frickler - I think you're using that - can you review? | 13:20 |
| SvenKieske | I would prefer a variable there, on not that huge deployments this will stretch upstart time if something is wrong with the service unit | 13:20 |
| SvenKieske | I'll review it, yes :) | 13:21 |
| mnasiadka | nice, thanks | 13:21 |
| chembervint | thank you! :) | 13:21 |
| SvenKieske | one advice: did you talk to upstream neutron-openvswitch-agent people if they think they can speed it up? | 13:22 |
| SvenKieske | because this is really a workaround for a process taking 3 minutes to start | 13:22 |
| SvenKieske | smells to me like there could be some code improvement be made on the neutron side? | 13:22 |
| SvenKieske | s/start/sync/ | 13:23 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/924332 - for stopping publish jobs on zed | 13:23 |
| mnasiadka | ok then | 13:24 |
| mnasiadka | should we wrap up? | 13:24 |
| kevko | SvenKieske: i am building my own images ..so i don't care | 13:24 |
| mnasiadka | So I guess we should wrap up :) | 13:25 |
| mnasiadka | Thank you all for coming - see you next week | 13:25 |
| mnasiadka | #endmeeting | 13:25 |
| opendevmeet | Meeting ended Wed Jul 17 13:25:13 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:25 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-17-13.02.html | 13:25 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-17-13.02.txt | 13:25 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-17-13.02.log.html | 13:25 |
| kevko | btw, anybody to re-review ? https://review.opendev.org/c/openstack/kolla-ansible/+/907166 | 13:25 |
| frickler | IMO the correct solution for the ohm0 interface is to configure it with static IP address | 13:25 |
| mnasiadka | frickler: that would be way better I guess | 13:27 |
| mnasiadka | comment on the patch ;-) | 13:28 |
| SvenKieske | I mean that should also be possible, but why on earth does a 2k network port sync take 3 minutes to finish? that smells like a really inefficient algorithmic design. | 13:28 |
| mnasiadka | kevko: done ;) | 13:28 |
| mnasiadka | SvenKieske: rabbitmq, rabbitmq, rabbitmq :) | 13:29 |
| SvenKieske | that's 45ms per port. | 13:29 |
| kevko | mnasiadka: that was quick :D | 13:29 |
| kevko | mnasiadka: :D :D :D | 13:29 |
| mnasiadka | kevko: had comments in draft, but as usual I forgot to click the reply button :) | 13:29 |
| frickler | SvenKieske: 30m, not 3m. I guess that's one of the reasons why ovn was made, ovs doesn't scale too well | 13:30 |
| SvenKieske | so roughly half a second per port..yeah well.. | 13:31 |
| chembervint | SvenKieske: our client's case is about ~3k ovs ports per node - they are really take a time to be synced by neutron-openvswitch-agent. yes, definitely it will be better, if neutron could be faster, but I'm not sure that it is possible .. | 13:31 |
| chembervint | here I propose simple and tested WA for such deployments | 13:31 |
| SvenKieske | chembervint: hint them at ovn I guess ;) | 13:31 |
| SvenKieske | chembervint: sure! a good workaround is always a good short term solution. but the readme doesn't match the actual code :) | 13:31 |
| chembervint | we will test such cases on OVN I think in couple of months :) | 13:31 |
| kevko | mnasiadka: replied | 13:31 |
| SvenKieske | and we need to make sure we don't drown in workarounds | 13:31 |
| *** Ale is now known as Guest455 | 13:36 | |
| oli | Hey guys, I am seeing a similar bug to this https://bugs.launchpad.net/kolla-ansible/+bug/2057847 when trying to add an arm64 compute server to our existing x86_64 deployment, we have created a group var for arm to use the `openstack_tag_suffix: "-aarch64"` any ideas how to fix? | 13:41 |
| sylvr | TheJulia: hello ! | 13:42 |
| sylvr | wrong chat, sorry | 13:47 |
| kevko | oli: give a log | 13:51 |
| kevko | oli: you can set the tag per host | 13:52 |
| kevko | oli: via inventory | 13:52 |
| oli | I get the same output from kolla-ansible as the one linked , is there a particular log you want | 13:52 |
| oli | kevko: I can try change it from a group level var to a host var | 13:53 |
| kevko | oli: i think it will work | 13:54 |
| oli | kevko: will report back shortly | 13:54 |
| opendevreview | Victor Chembaev proposed openstack/kolla-ansible master: Fix octavia-interface timeout https://review.opendev.org/c/openstack/kolla-ansible/+/920377 | 13:58 |
| opendevreview | Merged openstack/kolla-ansible master: Fix mariadb recovery https://review.opendev.org/c/openstack/kolla-ansible/+/920304 | 14:02 |
| oli | kevko: no luck, same error | 14:09 |
| oli | kevko: https://paste.opendev.org/show/824768/ | 14:15 |
| kevko | oli: you don't have a problem with tag ..but with execution of container | 14:41 |
| oli | kevko: the exec format error seems to be because the arch difference, but it looks like that task is delegated to control1a which is x64 | 14:48 |
| kevko | oli: so set the right tag for that host | 14:53 |
| oli | kevko: does it not take it from the gobals.yml ? or do i need to define that for every host/group | 14:54 |
| kevko | oli: if you set to globals ..it is widely used ...as it is ansible's extra-vars which has precedence before everything else ... | 14:55 |
| kevko | oli: in that case you need delete everything from globals ...so it will be not overriden ..and set it in inventory as host_var for every host as you need | 14:56 |
| oli | kevko: i don't define the suffix in the globals jus that specific host | 14:56 |
| kevko | oli: run kolla-ansible and run on host where it is failing command following >> for i in $(seq 1 1000);do sleep 0.5 ; docker run -v /var/run/docker.sock:/var/run/docker.sock nexdrew/rekcod nova_list_cells >> /tmp/my_log;done | 14:58 |
| kevko | oli: by this approach you will catch exact docker command which is actually runned on that host ...then ..send the output | 14:59 |
| oli | https://paste.opendev.org/show/buYHJTDeSfvC3Zy9v2nU/ I think setting the suffix for group_vars/control/all to will sort it | 15:03 |
| oli | as the control1a is trying to start the aarc64 container by the looks of it | 15:04 |
| kevko | oli: did you try mu suggestion of above command ? | 15:04 |
| kevko | oli: you need to remove suffix also from group vars i woull say | 15:05 |
| oli | kevko: using podman for our deployment, had issues with mariadb with docker | 15:05 |
| kevko | for i in $(seq 1 1000);do sleep 0.5 ; docker run -v /var/run/podman.sock:/var/run/docker.sock nexdrew/rekcod nova_list_cells >> /tmp/my_log;done | 15:05 |
| kevko | for i in $(seq 1 1000);do sleep 0.5 ; podman run -v /var/run/podman.sock:/var/run/docker.sock nexdrew/rekcod nova_list_cells >> /tmp/my_log;done | 15:06 |
| kevko | sorry | 15:06 |
| SvenKieske | always a good link to be reminded of :) https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html#understanding-variable-precedence | 15:06 |
| kevko | yep | 15:06 |
| kevko | SvenKieske: but i don't understand why the image is problem here ..as aarch64 and amd64 are same images ...there is no difference :D | 15:07 |
| SvenKieske | so _if_ it's supplied via extra vars you can't overwrite those, because extra vars have highest precedence :) | 15:07 |
| kevko | SvenKieske: we also should refactor kolla and start to use group vars and host vars ..instead extra vars | 15:07 |
| oli | SvenKieske: so our current set up is nothing defined in globals or group_vars, and 1 hostvar for the 1 arm node | 15:08 |
| kevko | oli: run the command i've sent to you | 15:09 |
| kevko | oli: while kolla-ansible runnning ...when it will fail ....cat /tmp/my_log | 15:09 |
| SvenKieske | kevko: are you sure dumb-init is arch agnostic? :D I didn't look into the paste or the bugreport above, just was reading along, but I have compiled enough gentoo to know that most "arch independent" packages really are not independent | 15:09 |
| kevko | give me an outptu | 15:09 |
| kevko | SvenKieske: i am not sure ...but i am sure that if i will have command to run from hand i can advice better | 15:10 |
| kevko | nexdrew/rekcod is the way | 15:10 |
| oli | WARNING: IPv4 forwarding is disabled. Networking will not work. | 15:10 |
| oli | Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? | 15:10 |
| kevko | oli: what did you run | 15:11 |
| kevko | oli: i said /var/run/podman.sock:/var/run/docker.sock | 15:11 |
| oli | sorry, just seen the second command | 15:11 |
| oli | replacing docker with podman | 15:11 |
| kevko | nexdrew/rekcod will check /var/run/docker.sock inside container ...but it works even if you bind podman sock ... | 15:12 |
| kevko | so please, try it again ..then paste here a command | 15:12 |
| oli | root@control1a:~# for i in $(seq 1 1000);do sleep 0.5 ; podman run -v /var/run/podman.sock:/var/run/docker.sock nexdrew/rekcod nova_list_cells >> /tmp/my_log;done | 15:12 |
| oli | Error: short-name "nexdrew/rekcod" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf" | 15:12 |
| kevko | aaaaaa, podman :D | 15:13 |
| kevko | oli: you need to set full image path | 15:14 |
| kevko | docker.io/nexdrew/rekcod:latest | 15:16 |
| SvenKieske | most of the time I find podman to be more correct - you could call it pedantic - than docker, but it leads to some difference in behaviour :) | 15:16 |
| kevko | for i in $(seq 1 1000);do sleep 0.5 ; podman run -v /var/run/podman.sock:/var/run/docker.sock docker.io/nexdrew/rekcod:latest nova_list_cells >> /tmp/my_log;done | 15:16 |
| oli | kevko: the command doesn't work get the Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? | 15:27 |
| kevko | oli: what is the command you tried ? | 15:28 |
| oli | for i in $(seq 1 1000);do sleep 0.5 ; podman run -v /var/run/podman.sock:/var/run/podman.sock docker.io/nexdrew/rekcod:latest nova_list_cells >> /tmp/my_log;done | 15:29 |
| oli | for i in $(seq 1 1000);do sleep 0.5 ; podman run -v /var/run/podman.sock:/var/run/docker.sock docker.io/nexdrew/rekcod:latest nova_list_cells >> /tmp/my_log;done | 15:29 |
| oli | both of these with the same result | 15:29 |
| kevko | oli: okay, this is the right one >> for i in $(seq 1 1000);do sleep 0.5 ; podman run -v /var/run/podman/podman.sock:/var/run/docker.sock docker.io/nexdrew/rekcod:latest nova_list_cells >> /tmp/my_log;done | 15:31 |
| kevko | oli: i can't know everything :D ...podman paths are slightly different | 15:31 |
| kevko | oli: this time it will work | 15:32 |
| oli | kevko: now seeing Error: No such object: nova_list_cells which is a different error but have kolla-ansible -i multinode delpoy --limit arm_compute running | 15:33 |
| kevko | oli: yes, but you need it to run in for loop i provided ...because it is not running .... but in moment there will be container starting ...it will catch it and write to file ..got it ? | 15:34 |
| kevko | oli: then we can debug and try it from hand | 15:34 |
| oli | kevko: yes, thanks, appreciate your time with this | 15:35 |
| oli | kevko: kolla-ansible failed but didnt produce any file as the container doesn't get started | 15:37 |
| kevko | kevko: hmmmmmmm | 15:39 |
| SvenKieske | if I'm not mistaken you could redirect podman logs in general to a file and debug that instead. podman should at least throw some error why the container doesn't start, did you check podman logs? | 15:40 |
| kevko | so , it failed on module level i would say | 15:40 |
| kevko | check the podman log for nova_list_cells container name | 15:40 |
| kevko | oli: you can also try it again ...it's possible there was a 0.5 gap and you didn't take it | 15:42 |
| kevko | *caught it ... | 15:42 |
| oli | will try again | 15:45 |
| oli | still nothing | 15:49 |
| oli | I don't think its able to start the container as its trying to start the -aarch64 container on the x86_64 machine | 15:50 |
| oli | if the containers are the same then it should work but the kolla -vvv output seems to indicate the dumb_init is the issue | 15:50 |
| kevko | oli: yeah, that's probably it | 15:57 |
| kevko | oli: w8 a second | 15:57 |
| kevko | oli: but i still don't understand - if you correctly set the tag for that host ...it should be OK | 16:00 |
| oli | I only set the option openstack_tag_suffix: "-aarch64" as a host_var for compute-1-arm64 , in globals.yml this var is commented out | 16:03 |
| oli | is there any other host_vars i should be setting? as other containers on the arm server are up | 16:04 |
| oli | https://paste.opendev.org/show/824771/ | 16:05 |
| kevko | oli: give me globals, and inventory | 16:28 |
| oli | kevko: sure | 16:33 |
| oli | https://paste.opendev.org/show/bSGR5gPeXot0JsyUbO81/ | 16:36 |
| kevko | oli: i can't see openstack_tag_suffix | 16:37 |
| oli | its not in globals , i set a host_var for that | 16:37 |
| oli | host_var/compute-1-arm64/all.yml > openstack_tag_suffix: "-aarch64" | 16:38 |
| kevko | oli: where did you config host_var ? i normally set host_var to inverntory file | 16:38 |
| kevko | for example control1a foo=bar | 16:38 |
| kevko | oli: try it to define in inventory file | 16:39 |
| oli | kevko: will give it a go | 16:42 |
| kevko | oli: ok, i found your issue probably | 16:43 |
| kevko | oli: if you set openstack_tag_suffix - you effectivelly changed every tag ---- because of openstack_tag: "{{ openstack_release }}-{{ kolla_base_distro }}-{{ kolla_base_distro_version }}{{ openstack_tag_suffix }}" | 16:43 |
| oli | but if i define it as a host_var it should only apply to that host right? | 16:44 |
| kevko | oli: hmm, yeah ..trye | 16:44 |
| kevko | true | 16:44 |
| kevko | kevko: who knows how it is delegated | 16:45 |
| oli | tried adding the openstack_tag_suffix to every host and setting it per host there and still same error | 16:46 |
| oli | i deleted all host/group vars with tag_suffix in | 16:46 |
| kevko | oli: also, remove from group vars | 16:46 |
| oli | kevko: still same error | 16:52 |
| oli | will do some more digging , im sure im missing some variables somewhere but cant find any example multiarch deployment set up | 16:53 |
| kevko | oli: try to override the image entire full path | 16:54 |
| kevko | oli: nova_conductor_image_full | 16:55 |
| kevko | oli: override this one | 16:55 |
| oli | for which host the compute one? or the control | 16:58 |
| kevko | oli: control one ... and non aarch64 | 17:01 |
| kevko | oli: for that host where it's failing | 17:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 18:00 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 18:35 |
| opendevreview | Dawud proposed openstack/kolla-ansible master: Add size limits to Fluentd buffers https://review.opendev.org/c/openstack/kolla-ansible/+/924359 | 19:43 |
| samcat116 | Hi folks, on my Bobcat based stack I'm seeing QueuePool limit of size 1 overflow 50 reached, connection timed out, timeout 30s on my neutron server container on one of my 3 controllers, resulting in 504 errors to the neutron API. Restarting that container seems to solve it (but not sure if thats a temporary fix. Any advice on some config to try here if I see this again? It seems that db pools were intentionally removed from the | 20:00 |
| samcat116 | config here | 20:00 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Add exclude for permissions in kolla config.json file https://review.opendev.org/c/openstack/kolla/+/655620 | 20:05 |
| kevko | samcat116: modify sqlalchemy/oslo.db stuff | 20:22 |
| kevko | samcat116: experiment with that | 20:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 20:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 20:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: bifrost: Fix ansible and deps installation https://review.opendev.org/c/openstack/kolla/+/924246 | 20:31 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Add script for distributed cron https://review.opendev.org/c/openstack/kolla/+/924365 | 20:34 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!