| opendevreview | Merged openstack/openstack-ansible master: Set bullseye jobs to voting https://review.opendev.org/c/openstack/openstack-ansible/+/805172 | 00:55 |
|---|---|---|
| spatel | I have noticed issue with mysql 10.6.4 with focal, it stuck during mysql upgrade. | 02:44 |
| spatel | how did this pass CI and got merge ? | 02:44 |
| spatel | jrosser noonedeadpunk ? | 02:45 |
| spatel | 10.5.12 works great! without issue. look like 10.6.4 has serious issue as you mentioned in today meeting. | 02:55 |
| *** rpittau|afk is now known as rpittau | 07:06 | |
| noonedeadpunk | spatel: https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/805822 | 07:23 |
| noonedeadpunk | Dunno how it passed, but it happens not always. but kind of frequently | 07:23 |
| *** mgoddard- is now known as mgoddard | 07:47 | |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Revert "Update galera to 10.6.4" https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/805822 | 08:36 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/victoria: Improve speed of permissions changes https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/805767 | 09:10 |
| kleini_ | OSA deploys HAproxy in front of all service APIs and furthermore terminates SSL. Unfortunately, if you look into the response of https://<external_vip>:9696 it says: "href": "http://<external_vip>:9696/v2.0/". The vagrant OpenStack provider relies on this information to determine available API versions. But the connection then to HTTP will fail as HAproxy is only listening on HTTPS. I see now furthermore that secure_proxy_ssl_ | 10:35 |
| kleini_ | header configuration option in Nova is deprecated. How can I configure OSA to have the protocol in those version responses correct? | 10:35 |
| kleini_ | sorry, I was wrong. the deprecated secure_proxy_ssl_header is in the oslo_message section and the correct one in the wsgi section | 10:38 |
| kleini_ | hmm, seems to need more investigation as there seems to be support on the Nova service for the X-Forwarded-Proto header. so checking next, whether HAproxy sets this or not. | 10:40 |
| *** kleini_ is now known as kleini | 10:48 | |
| noonedeadpunk | kleini: yeah, services should kind of respect X-Forwarded-Proto but not every service does :( | 10:48 |
| noonedeadpunk | we should have set X-Forwarded-Proto in cases where ssl is set | 10:49 |
| kleini | okay, it works for all services correctly out of the box but Neutron | 10:53 |
| noonedeadpunk | I don't think we can do anything about it. | 10:54 |
| noonedeadpunk | I can recall asking neutron folks about that several years ago, but they didn't saw a problem in that. | 10:54 |
| kleini | the problem is, that the answer is wrong and clients fail relying on the answer | 10:55 |
| noonedeadpunk | their argument was that this answer shouldn't be relied or smth like that... | 10:56 |
| noonedeadpunk | But worth checking again :) | 10:56 |
| noonedeadpunk | like you have keystone catalog and it should be used at any times anyway | 10:56 |
| noonedeadpunk | But I agree that's a bug imo... | 10:57 |
| noonedeadpunk | Btw, also we don't use uwsgi for neutron atm.. Maybe it's fixed with uwsgi - dunno... | 10:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Implement uWSGI for neutron-api https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/486156 | 10:58 |
| noonedeadpunk | or maybe https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/defaults/main.yml#L135-L138 breaks it? | 10:59 |
| noonedeadpunk | you can actually try setting _neutron_api_paste_ini_overrides to empty dict jsut to check things out | 11:00 |
| kleini | I set it to an empty dict, checked multiple times for any typos, but it does not have an effect on generated /etc/neutron/api-paste.ini | 11:26 |
| kleini | I have to confess, that I have no idea, what api-paste.ini does | 11:28 |
| mgariepy | noonedeadpunk, https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/801910 typo | 11:59 |
| noonedeadpunk | um, no, it's as designed | 12:02 |
| mgariepy | um.. .. wow. | 12:03 |
| noonedeadpunk | truthy/falsy were added in 2.10 as they treat both empty lists and strings | 12:04 |
| noonedeadpunk | As I guess we have some weird value for haproxy_service_enabled | 12:05 |
| noonedeadpunk | hm, but not in this case... | 12:05 |
| mgariepy | well it does make sense i wasn't aware of it . | 12:05 |
| noonedeadpunk | as we do |bool filter... | 12:05 |
| noonedeadpunk | well, I think here I could really use false... | 12:06 |
| noonedeadpunk | I can't see now why I picked up using falsy instead | 12:06 |
| mgariepy | it was a typo ;p haha | 12:07 |
| noonedeadpunk | I guess I didn't want to use `== False` and is False is not valid test ? | 12:07 |
| mgariepy | i need to open my laptop to cleanup the fan.. | 12:08 |
| mgariepy | i'll be back in a bit. | 12:08 |
| mgariepy | there is always a screw that doesn't want to cooperate .. | 12:54 |
| depasquale | ciao guys. I have a question for you: looking at the following online official documentation (https://docs.openstack.org/neutron/wallaby/install/install-ubuntu.html) there are configurations for Neutron that I am not able to see after an ansible run... | 12:57 |
| depasquale | in particular the following paragraph... https://docs.openstack.org/neutron/wallaby/install/controller-install-option1-ubuntu.html | 12:58 |
| depasquale | and https://docs.openstack.org/neutron/wallaby/install/controller-install-option2-ubuntu.html | 12:58 |
| depasquale | can someone suggest the best way to replicate "exactly" the output of the two paragraphs in a runnable ansible config? | 12:59 |
| depasquale | thanks in advance | 12:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Use management_address by default https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/805989 | 13:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use cinder defaults for cinder_management_address https://review.opendev.org/c/openstack/openstack-ansible/+/805991 | 13:04 |
| spatel | noonedeadpunk around? | 13:11 |
| spatel | I am seeing very odd issue when deploying OVN and i believe its related this - https://paste.opendev.org/show/808304/ | 13:13 |
| spatel | getting this error when installing openvswitch on ovn container - https://paste.opendev.org/show/808305/ | 13:14 |
| spatel | openvswitch looking for kernel module directory in container which doesn't exist, /lib/modules/ how do i mount that in container ? | 13:15 |
| noonedeadpunk | no idea honestly | 13:15 |
| noonedeadpunk | you can't add modules in containers | 13:15 |
| spatel | by default LXC doesn't have /lib/modules | 13:16 |
| spatel | hmmm | 13:16 |
| noonedeadpunk | it's just not allowed because of security reasons | 13:16 |
| halali_- | hi, when I try to deploy Memcached as high availability using https://docs.openstack.org/openstack-ansible-memcached_server/latest/ I got an OSA playbook failure https://paste.opendev.org/show/808306/ | 13:16 |
| spatel | then what is the for - https://paste.opendev.org/show/808304/ | 13:16 |
| noonedeadpunk | these are containers so kernel-based virtualization, ie all containers and host have same kernel | 13:17 |
| spatel | i thought this snippet does that | 13:17 |
| noonedeadpunk | so you can't do any adjustments of kernel from inside the continer | 13:17 |
| spatel | i got it but i can clearly see open you install openvswitch inside container its looking for kernel module and which doesn't exist. | 13:18 |
| noonedeadpunk | halali_-: how does haproxy_extra_services look like? | 13:18 |
| mgariepy | spatel, https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/neutron.yml#L67 | 13:18 |
| noonedeadpunk | I have a feeling these might be a wrong type for some reason | 13:18 |
| spatel | openstack_host_specific_kernel_modules: this is magic code but not sure how does it work | 13:18 |
| noonedeadpunk | I think we add module on host and then pass them to containers with that | 13:19 |
| mgariepy | i guess to use a module inside a container it needs 1- to be loaded on the host. and then added to the lxc config. | 13:19 |
| spatel | mgariepy you are right.. | 13:20 |
| halali_- | haproxy_extra_services is https://paste.opendev.org/show/808307/ | 13:21 |
| spatel | mgariepy how do i fix that? i have https://paste.opendev.org/show/808304/ code in user_variables.yml | 13:21 |
| noonedeadpunk | halali_-: yeah. so it's wrong:) | 13:21 |
| spatel | i did run setup-hosts.yml but still it didn't mount /lib/modules | 13:21 |
| mgariepy | on focal ? | 13:22 |
| spatel | yes | 13:22 |
| spatel | 20.04 | 13:22 |
| mgariepy | with agent inside a lxc container? | 13:22 |
| noonedeadpunk | oh, ouch. see issue with our doc | 13:23 |
| spatel | with agent? | 13:23 |
| mgariepy | you want the neutron-agent inside a lxc container? | 13:23 |
| spatel | i am deploying OVN with multi-node and when i was running os-neutron-install.yml playbook on ovn container i hit that error related openvswitch module didn't found | 13:24 |
| spatel | the i found we need read doc and i found this code - https://paste.opendev.org/show/808304/ | 13:25 |
| spatel | hold on... | 13:25 |
| spatel | i found it | 13:25 |
| spatel | what was missing | 13:25 |
| spatel | i can see /lib/modules directory inside LXC | 13:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-memcached_server master: [doc] Fix haproxy_extra_services layout https://review.opendev.org/c/openstack/openstack-ansible-memcached_server/+/805996 | 13:26 |
| noonedeadpunk | halali_-: does that make sense to you ? ^ | 13:27 |
| spatel | earlier i was running setup-hosts.yml with --limit and it wasn't working | 13:27 |
| halali_- | yeah, thanks | 13:27 |
| mgariepy | on focal i think it's /usr/lib/modules not /lib/module | 13:28 |
| mgariepy | modules** | 13:28 |
| spatel | mgariepy can you explain what this code does - https://paste.opendev.org/show/808304/ | 13:28 |
| spatel | i am not seeing any mount point here - /var/lib/lxc/ovn-lab-infra-1_neutron_ovn_northd_container-cb55f5ef/config | 13:29 |
| noonedeadpunk | spatel: it does modprobe modules on bare metal hosts | 13:29 |
| noonedeadpunk | during setup-hosts | 13:29 |
| spatel | oh! | 13:29 |
| spatel | i thought it will does on LXC | 13:29 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/tasks/configure_metal_hosts.yml#L60-L66 | 13:30 |
| mgariepy | no. you cannot load a module inside the container. | 13:30 |
| mgariepy | you need to load it on the host then pass it to the container. | 13:30 |
| spatel | does that means now /lib/modules will be available on each container? | 13:30 |
| spatel | how to pass that to container ? is there any special command or config to pass module to LXC container? | 13:31 |
| spatel | trying to understand flow how does this process work | 13:31 |
| noonedeadpunk | I think it might be enough to modprobe on host. As modules should be bind-mounted anyway | 13:33 |
| spatel | you are saying just loading module on host machine is enough and no need to add any bind or mount point in lxc config right? | 13:34 |
| noonedeadpunk | well, it's hard to guess for me. You should see if container has this folder mounted or not | 13:34 |
| noonedeadpunk | if yes - jsut modprobe should help | 13:35 |
| spatel | i can see /lib/modules present only on ovn container not any other so it has to be setting somewhere in config file, just try to find where is that setting related bind | 13:36 |
| spatel | nothing here - https://paste.opendev.org/show/808309/ | 13:36 |
| noonedeadpunk | do you need it on some other containers? | 13:36 |
| noonedeadpunk | it's present for neutron becuase of https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/common-playbooks/neutron.yml#L70-L72 | 13:38 |
| spatel | no i don't need on other containers.. | 13:38 |
| noonedeadpunk | bind should present in /var/lib/lxc/$name/config iirc | 13:39 |
| spatel | that is what i thought but its not there :( | 13:39 |
| noonedeadpunk | some include maybe... | 13:40 |
| spatel | no in include also | 13:40 |
| noonedeadpunk | well, it must be there... | 13:40 |
| noonedeadpunk | since that what included task does https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/common-tasks/os-lxc-container-setup.yml#L49-L61 | 13:41 |
| mgariepy | https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/neutron.yml#L71 | 13:41 |
| spatel | This is include file - https://paste.opendev.org/show/808310/ | 13:41 |
| noonedeadpunk | mgariepy: can I ask for your vote in https://review.opendev.org/c/openstack/openstack-ansible/+/804976 ?:) | 13:42 |
| spatel | This is very odd none of entry found which say mount that module in OVN container. i did search in whole LXC directory - cat /var/lib/lxc/*/config | grep lxc.mount.entry | grep module | 13:45 |
| spatel | its gotta be somewhere else | 13:46 |
| mgariepy | noonedeadpunk, done. | 13:48 |
| mgariepy | what are the groups for your ovn container? | 13:49 |
| mgariepy | when : "'neutron_agent' in group_names" | 13:49 |
| spatel | neutron_ovn_northd | 13:50 |
| mgariepy | so you have your answer ? | 13:50 |
| spatel | ? | 13:50 |
| mgariepy | https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/neutron.yml#L79 | 13:50 |
| mgariepy | it doesn't match the condition. | 13:50 |
| mgariepy | so it's not added. | 13:51 |
| spatel | how does /lib/modules appeared on ovn container without any config ? | 13:51 |
| noonedeadpunk | well, shouldn't neutron_ovn_northd be amoung neutron_agent? | 13:51 |
| mgariepy | scripts/inventory-manage.py -g | 13:52 |
| spatel | https://paste.opendev.org/show/808311/ | 13:52 |
| mgariepy | -g != -G ;p | 13:52 |
| noonedeadpunk | looking at https://opendev.org/openstack/openstack-ansible/src/branch/master/etc/openstack_deploy/env.d/neutron_ovn.yml.example it's not | 13:52 |
| spatel | https://paste.opendev.org/show/808312/ | 13:53 |
| spatel | there is no group name neutron_agent anywhere in inventory | 13:53 |
| mgariepy | interesting.. i guess it would need to be added ? | 13:54 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/env.d/neutron.yml#L57 | 13:54 |
| spatel | mgariepy that is what i am trying to understand without config how does it load module in OVN and what if i reboot host it will disappeared right? | 13:55 |
| noonedeadpunk | But I think in container_skel it should belong_to network-agent_containers ? | 13:55 |
| noonedeadpunk | most likely it will ,yes | 13:56 |
| spatel | doesn't neutron-agent will be part of neutron_all ? | 13:56 |
| noonedeadpunk | yes, but neutron_ovn_northd_container shoudl be part of neutron-agent I guess | 13:56 |
| noonedeadpunk | OR we should adjust that condition in playbook | 13:57 |
| noonedeadpunk | because we explicitly overwrite neutron_agents_container in example | 13:57 |
| spatel | add neutron_ovn_northd in that playbook? | 13:57 |
| mgariepy | the module is loaded on the host? | 13:58 |
| noonedeadpunk | The problem is that I have pretty vague understanding what northd agent is and if it can be considered as some kind of agent... | 13:58 |
| spatel | Yes its its loaded | 13:59 |
| spatel | root@ovn-lab-infra-1:~# lsmod | grep openvswitch | 13:59 |
| spatel | openvswitch 135168 0 | 13:59 |
| noonedeadpunk | and what to do best - consider it as agent and adjust docs or adjust playbook by adding northd to condition | 13:59 |
| mgariepy | and you don't see it in the container? | 13:59 |
| spatel | its also loaded on LXC container | 14:00 |
| spatel | root@ovn-lab-infra-1-neutron-ovn-northd-container-cb55f5ef:~# lsmod | grep openvswitch | 14:00 |
| spatel | openvswitch | 14:00 |
| mgariepy | can you paste the config of that container? | 14:00 |
| noonedeadpunk | but unlikely to survive reboot) | 14:00 |
| spatel | because i have /lib/modules present on container, which wasn't there before | 14:00 |
| spatel | root@ovn-lab-infra-1-neutron-ovn-northd-container-cb55f5ef:~# ls -l /lib/modules | 14:00 |
| spatel | total 4 | 14:00 |
| spatel | drwxr-xr-x 5 root root 4096 Aug 24 22:50 5.4.0-42-generic | 14:00 |
| mgariepy | the lxc config. | 14:00 |
| spatel | https://paste.opendev.org/show/808309/ | 14:01 |
| spatel | my big question is how does that appear there after adding this code and re-run playbook - https://paste.opendev.org/show/808304/ | 14:01 |
| noonedeadpunk | according to config it should not be there | 14:03 |
| jrosser | depasquale: can you be a bit more specific? the provider_networks section of openstack_user_config is used to set up the neutron networks | 14:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Bump OpenStack-Ansible Victoria https://review.opendev.org/c/openstack/openstack-ansible/+/806003 | 14:08 |
| spatel | noonedeadpunk can we add ovn_northd here to fix this issue - https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/neutron.yml#L79 | 14:10 |
| spatel | i think that is easy way, otherwise OVN is still broken if someone reboot :) | 14:10 |
| spatel | i would like to ask jamesdenton also how did he do that? | 14:11 |
| mgariepy | did you test the reboot ? | 14:11 |
| halali_- | noonedeadpunk using allowlist failed, success using whitelist instead on service configuration user_variables.yml https://paste.openstack.org/show/808313/ | 14:11 |
| spatel | not yet... | 14:11 |
| spatel | i can reboot | 14:11 |
| spatel | let me reboot hold on | 14:11 |
| mgariepy | haha :D | 14:11 |
| mgariepy | just to confirm the issue is there. | 14:11 |
| spatel | doing it, rebooting whole host not just container | 14:12 |
| opendevreview | Merged openstack/openstack-ansible master: Revert "set non-voting for broken tox-doc test" https://review.opendev.org/c/openstack/openstack-ansible/+/803832 | 14:12 |
| spatel | while we are rebooting jrosser did you create patch to fix mysql version from 10.6.4 to 10.5.12 ? | 14:14 |
| spatel | last night i got stuck with 10.6.4 | 14:14 |
| jrosser | it's merged i think | 14:14 |
| spatel | :) | 14:14 |
| jrosser | but you are on W, right? | 14:14 |
| jrosser | or master? | 14:14 |
| spatel | playing with master because all OVN patches are there | 14:14 |
| jrosser | here is the revert https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/805822 | 14:15 |
| spatel | sweet! | 14:15 |
| * jrosser meetings again | 14:15 | |
| noonedeadpunk | halali_-: ah, yes, fair | 14:21 |
| noonedeadpunk | we merged switch to allowlist only on W | 14:21 |
| noonedeadpunk | spatel: well, as I said - we can either do that or add northbridge to agents group depending on what is more appropriate thing | 14:22 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Exclude neutron from venv constraints https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/798960 | 14:23 |
| spatel | mgariepy after reboot ovn container still has /lib/modules directory so look like it works.. | 14:26 |
| spatel | now where the heck that coming from we need to find :) | 14:26 |
| spatel | may be lxc-hooks ? | 14:26 |
| mgariepy | hmm. | 14:26 |
| spatel | BRB | 14:30 |
| noonedeadpunk | maybe we somewhere duplicated functionality of bind mounting modules... | 14:33 |
| opendevreview | Merged openstack/openstack-ansible master: Switch ceph-ansible to Pacific https://review.opendev.org/c/openstack/openstack-ansible/+/804610 | 14:35 |
| opendevreview | Merged openstack/openstack-ansible master: Switch CentOS upgrade jobs to Stream ones https://review.opendev.org/c/openstack/openstack-ansible/+/805350 | 14:36 |
| mgariepy | spatel, quick one. does the horizon container have the module also ? | 14:47 |
| mgariepy | maybe the openvswitch service does some magic. | 14:48 |
| spatel | checking | 14:54 |
| spatel | mgariepy no horizon doesn't have /lib/module directory inside container | 14:55 |
| spatel | only and only ovn has that directory | 14:55 |
| mgariepy | so it's magic lol. | 15:04 |
| mgariepy | not sure what's going on there. | 15:04 |
| spatel | mgariepy i am digging into it.. to find out what the heck going on :) | 15:50 |
| spatel | but its clean we need /lib/module inside OVN container otherwise it won't work | 15:51 |
| jrosser | spatel: mgariepy did you see this? https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/common-playbooks/neutron.yml#L67-L79 | 15:55 |
| spatel | yes we did | 15:55 |
| spatel | Question is who mount /lib/modules inside ovn container? | 15:56 |
| spatel | if not seeing any entry telling lxc to do that | 15:56 |
| spatel | so how magically it getting mounted inside only and only ovn container | 15:57 |
| spatel | even after reboot host that entry automatically appeared, how? | 15:57 |
| jrosser | maybe it's a stupid question but why is the openvswitch module needed there? | 15:58 |
| spatel | because ovn_northd container runs ovsdb and it need openvswitch-switch | 15:58 |
| spatel | may be dependency tree issue | 15:58 |
| jrosser | spatel: do you know which package it is trying to install when you get that error | 16:03 |
| spatel | openvswitch-switch package looking for /lib/modules path to find openvswitch module | 16:04 |
| *** rpittau is now known as rpittau|afk | 16:04 | |
| jrosser | no, i mean which pacakge out of the ones we list in the os_neutron role | 16:04 |
| spatel | you are asking which task was failing in os_neutron? sorry i am not following you.. | 16:06 |
| jrosser | the only place we specify openvswitch-switch is here https://github.com/openstack/openstack-ansible-os_neutron/blob/ea9dae41ad61195ae25f7673d016bcf2486204c8/vars/debian.yml#L18 | 16:08 |
| jrosser | the packages for northd are listed here https://github.com/openstack/openstack-ansible-os_neutron/blob/ea9dae41ad61195ae25f7673d016bcf2486204c8/vars/debian.yml#L62 | 16:08 |
| jrosser | so what i am trying to understand is if there is a logic error, that we try to install too much stuff in the ovn_northd container and accidentally also include the openvswitch-switch package when actually it's not needed | 16:09 |
| spatel | hmm | 16:13 |
| jrosser | i am looking at this https://github.com/openstack/openstack-ansible-os_neutron/blob/ea9dae41ad61195ae25f7673d016bcf2486204c8/vars/source_install.yml#L27-L33 | 16:13 |
| jrosser | and that installs openvswitch-switch everywhere, when maybe we only need that in the place that the neutron agent is running | 16:13 |
| jrosser | but then there is also the case like mgariepy has, where the agent is containerised | 16:14 |
| *** sshnaidm is now known as sshnaidm|afk | 16:14 | |
| jrosser | jamesdenton: ^ any thoughts on this, if we install openvswitch-switch in too many places? | 16:14 |
| spatel | in OVN case we don't need neutron-agent anymore | 16:14 |
| jrosser | well you know what i mean, the place where the vswitch is | 16:15 |
| spatel | that is whole point of OVN to remove all neutron-agent | 16:15 |
| jrosser | which is mostly only the host | 16:15 |
| spatel | openvswitch only required on compute nodes (not infra) | 16:15 |
| jrosser | for DVR type applications, yes, not for centralised | 16:16 |
| spatel | let me try to remove openvswitch-switch from ovn_northd container and see if it works | 16:16 |
| jrosser | for that container only try setting neutron_ovs_distro_packages: [] | 16:16 |
| spatel | If infra node is network node then we mostly run network node in metal so no need required inside container | 16:16 |
| jrosser | thats why i question the current logic | 16:18 |
| spatel | jrosser you have a very good point | 16:18 |
| jrosser | for metal deploy this is all finew | 16:18 |
| spatel | why do we need ovs switch inside OVN lxc | 16:18 |
| jrosser | but LXC deploy often reveals errors like this in the code as things have to be more correct | 16:18 |
| spatel | can i override neutron_ovs_distro_packages: [] just for ovn lxc ? | 16:19 |
| jrosser | is neutron-ovn-controller the group of all things which should be running the vswitch? | 16:19 |
| spatel | Yes neutron-ovn-controller run on compute host so we need vswitch there | 16:20 |
| jrosser | so there is this flag https://github.com/openstack/openstack-ansible-os_neutron/blob/77f9b16967069fa1041c252543a1c1d861b1a86e/vars/main.yml#L21 | 16:24 |
| jrosser | but it seems that controls maybe too many things, the installation of the ovs repo which you need for more things that just openvswitch-switch | 16:24 |
| jrosser | and also it determines the actual installation of openvswitch-switch | 16:25 |
| spatel | that stanza telling install switch to each of that group | 16:32 |
| jrosser | but it does more than that, like install the repo | 16:38 |
| jrosser | like here https://github.com/openstack/openstack-ansible-os_neutron/blob/77f9b16967069fa1041c252543a1c1d861b1a86e/tasks/neutron_pre_install.yml#L79-L87 | 16:39 |
| spatel | in short it mess :) | 16:54 |
| spatel | no easy to remove openvswitch-switch from ovn container | 16:55 |
| jrosser | mess can bit tidied :) | 16:56 |
| jrosser | but i have no OVN deployment so i can only guess/advise | 16:56 |
| jrosser | another question though, why don't we see this error on OVN AIO builds? | 16:57 |
| jrosser | as we run OVN jobs in CI with LXC | 16:58 |
| spatel | jrosser because i missed this code in my multi-node deployment - https://paste.opendev.org/show/808317/ | 17:09 |
| spatel | as document saying to do - https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html | 17:09 |
| spatel | and i got error in ovn saying where is /lib/modules directory but after adding that snippet lxc mount /lib/modules inside container and everything works | 17:10 |
| spatel | jrosser i did some experiment and turn out we need openvswitch-switch package inside ovn container because of ovsdb-server | 19:03 |
| spatel | ovn used ovsdb-server for database and that binaries comes from openvswitch-switch package | 19:03 |
| spatel | https://paste.opendev.org/show/808320/ | 19:04 |
| spatel | I think we are good nothing to worry :) | 19:06 |
| jrosser | but the ovsdb-server binary is in openvswitch-common | 19:06 |
| jrosser | tbh I still have no idea what the actual issue is | 19:07 |
| spatel | issue is me.. i missed one step that created this chaos :) | 19:07 |
| spatel | https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html | 19:07 |
| spatel | I forgot to add this section - openstack_host_specific_kernel_modules: | 19:08 |
| spatel | so i asked question why do we need that step but look like it required for ovn_northd container to install openvswitch-switch package (without that you can't install openvswitch-switch package inside container) | 19:09 |
| jrosser | and what happens in the CI jobs? | 19:09 |
| spatel | we should add little wording around that section, to explain why do we need that snippet | 19:09 |
| spatel | I believe that is part of CI job.. otherwise it won't work | 19:10 |
| spatel | or may be we are installing compute on same box and that is why openvswitch module is loaded on host vm | 19:10 |
| spatel | in short bottom line is you have to do modprobe openvswitch on host machine to install openvswitch-switch inside container | 19:11 |
| spatel | Do you mind if i add some more doc around this section to make it clear for other folks - https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html | 19:12 |
| spatel | # Ensure the openvswitch kernel module is loaded - this is little confusing, like where to load module :( | 19:12 |
| mgariepy | https://paste.opendev.org/show/808321/ | 19:13 |
| mgariepy | openvswitch-commons has ovsdb-server. | 19:13 |
| mgariepy | openvswitch-common** | 19:14 |
| spatel | does common need switch package as a dependency? | 19:14 |
| spatel | let me try to install common and see if it auto install switch | 19:15 |
| mgariepy | openvswitch-switch depends on -common | 19:15 |
| spatel | good so it doesn't have dependency | 19:16 |
| spatel | we can try that, just install common on container and leave switch package that should work | 19:16 |
| spatel | again that required re-arranging bunch of stuff to make it work that way | 19:17 |
| spatel | i will give it a try and see if we can remove that deps | 19:18 |
| spatel | mgariepy look like we don't need /lib/module binding to ovn container.. it was mis-understanding :) | 19:20 |
| spatel | my bad | 19:20 |
| spatel | during last night experiment i copied /lib/module directory to lxc rootfs to see my issue go away :( | 19:21 |
| spatel | i have rebuild everything/ destroy and now i can't see /lib/module anywhere mounted and everything working | 19:22 |
| mgariepy | you probably need it to start openvswitch-switch.service.\ | 19:22 |
| opendevreview | Merged openstack/openstack-ansible master: Replace deprecated collection names https://review.opendev.org/c/openstack/openstack-ansible/+/805585 | 19:28 |
| spatel | mgariepy try to configure SRIOV with OVN so wondering if we need neutron_sriov_nic_agent ? | 19:38 |
| spatel | any idea | 19:38 |
| mgariepy | no clue. i didn't had time to play that much with ovn in the past couple of months | 19:39 |
| spatel | no worry let me play and figure out | 19:39 |
| jamesdenton | jrosser openvswitch-switch really only needs to exist on neutron agent nodes and compute nodes | 22:53 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!