| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_skyline master: Add EL distro support https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/912370 | 09:31 |
|---|---|---|
| f0o | any way to specify the MTU in neutron through ansible? I checked https://docs.openstack.org/openstack-ansible-os_neutron/latest/ but no hits :/ | 10:12 |
| jrosser | f0o: is there something specific you want to set in a neutron config file? | 10:12 |
| f0o | I noticed all my VMs have an mtu of 1442 which would be fine for vxlan over 1500 links but all links are 9000 so it seems like a bit of a waste | 10:12 |
| f0o | I also noticed, perhaps coincidentally, that network performance from vm<>vm is about 5Gbit/s while host<>host is 10Gbit/s | 10:13 |
| f0o | in linuxbridge you'd just set global_physnet_mtu and path_mtu to 9000 in neutron and it would be "done" | 10:14 |
| f0o | but on OVN I've honestly not the slightest idea | 10:14 |
| jrosser | f0o: did you come across the `config_template` ansible module we use yet? | 10:16 |
| f0o | nope | 10:16 |
| jrosser | ok, so, there are a ton of openstack services, each of which have a ton of options | 10:17 |
| jrosser | so it's completely impractical to have a defaults/main.yml variable for each of them, and expose a variable directly | 10:17 |
| jrosser | there is some documentation here for how we allow this to happen in a manageable way https://docs.openstack.org/openstack-ansible/latest/reference/configuration/using-overrides.html | 10:18 |
| f0o | so before I go into this rabbithole - does it even make sense to bother about the vif/ovn MTU here? | 10:18 |
| noonedeadpunk | I'm pretty sure we should have an example somewhere in docs | 10:18 |
| noonedeadpunk | or a bug report realted to this :D | 10:19 |
| noonedeadpunk | sec | 10:19 |
| f0o | :D | 10:19 |
| jrosser | for the vars you mentioned already it would be this https://paste.opendev.org/show/bo6MGQbuL5cBuJTUNJ8W/ | 10:19 |
| f0o | magic | 10:19 |
| f0o | well gonna plug it in and cross my fingers | 10:20 |
| jrosser | but it's totally worth understanding the what/how of config_template and how you're able to make any config you like for any service, even when there does not appear to be a var for it | 10:20 |
| f0o | yeah will read into it | 10:20 |
| f0o | thanks for the pointer! | 10:20 |
| jrosser | those structures i gave you are key/value to include into specific sections of the various config files for neutron | 10:21 |
| jrosser | but if this works for OVN, i have actually no idea | 10:21 |
| f0o | I got a feeling I will have quite many of those hah | 10:21 |
| jrosser | right - this is the primary mechanism in OSA for customising the services beyond the most commonly used things we put in defaults/main.yml of the ansible roles | 10:22 |
| noonedeadpunk | f0o: https://bugs.launchpad.net/openstack-ansible/+bug/1825012 | 10:22 |
| noonedeadpunk | ok, yes, same :) | 10:23 |
| f0o | hah glad to see I'm not the only one asking for those things | 10:23 |
| jrosser | well, remember we start with "sensible defaults" | 10:24 |
| jrosser | theres a bunch of people who can't/won't change from those so it's tricky | 10:24 |
| f0o | makes sense | 10:24 |
| f0o | I dont think I would've bothered looking into it if I had somewhere near 8Gbit/s on the vm<>vm traffic. I would just call it "good enough" - the 4-5G range was just what upset me. So I'm just going through the low-hanging fruits like MTU | 10:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_skyline master: Add EL distro support https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/912370 | 10:26 |
| f0o | I hope that fixes it because if I end up digging into OVS/OVN I will likely just get lost | 10:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_skyline master: Do not define a random password for each run https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/912332 | 10:26 |
| f0o | OVS is a deeeep rabbit hole | 10:26 |
| jrosser | so is getting the datapath accellerated | 10:26 |
| f0o | no havent yet touched DPDK | 10:27 |
| noonedeadpunk | regarding skyline - I guess we'd need to land https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/912333/5 regardless which doesn't have any testing at all, as it's a circular dependency with https://review.opendev.org/c/openstack/openstack-ansible/+/859446 anyway | 10:27 |
| jrosser | which way is the stack of patches there? | 10:28 |
| jrosser | gerrit UI confuses me with the order | 10:28 |
| jrosser | i guess i mean we can break the circularness by making the patch that brings the testing back being the last one we merge | 10:29 |
| jrosser | as if the order is right we will see the results be good on that one | 10:30 |
| noonedeadpunk | well, we can't merge anything without any jobs as they are today | 10:31 |
| noonedeadpunk | So I guess first is https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/912333 and then some more work potentially needed on 859446 | 10:31 |
| noonedeadpunk | (like dropping nginx) | 10:31 |
| noonedeadpunk | and couple of nits potentially | 10:32 |
| noonedeadpunk | like https://review.opendev.org/c/openstack/openstack-ansible-os_skyline/+/912370 afterwards | 10:32 |
| jrosser | i was thinking that the nginx thing might turn into a haproxy map | 10:33 |
| noonedeadpunk | yeah | 10:33 |
| noonedeadpunk | totally | 10:33 |
| jrosser | but it really was not completely clear what the nginx setup was actually doing | 10:33 |
| jrosser | it seemed almost redundant | 10:33 |
| jrosser | so i felt i was missing something important | 10:33 |
| noonedeadpunk | nah | 10:34 |
| noonedeadpunk | it's really just passing url by regexp towards corresponsive apis | 10:34 |
| noonedeadpunk | including skyline api | 10:34 |
| jrosser | right - so the one thing that haproxy cant do is any kind or rewrite iirc | 10:35 |
| jrosser | *kind of | 10:35 |
| noonedeadpunk | the only nasty part is that there's no way fastcgi will work with uwsgi, | 10:35 |
| noonedeadpunk | or well... what nginx is actually doing - serving static content | 10:35 |
| noonedeadpunk | there're jsut proxy-passes all along... | 10:36 |
| noonedeadpunk | that;s the full resulting config: https://paste.openstack.org/show/bi4wWquWDNGfvMkeBHGX/ | 10:36 |
| noonedeadpunk | so the "only" important thing, is "root /openstack/venvs/skyline-28.1.0.dev29/lib/python3.10/site-packages/skyline_console/static" I guess | 10:37 |
| noonedeadpunk | but maybe we wanna move that to apache just for consistency.... | 10:37 |
| noonedeadpunk | as haproxy won't be able to handle that | 10:37 |
| jrosser | oh i see it's mapping the ports to backend urls right that makes sense | 10:38 |
| jrosser | oh no it doesnt | 10:39 |
| jrosser | yes i don't understand this | 10:40 |
| gokhan__ | did victori/wallaby/xena/yoga upgrade affected by unmaintained branch creation ? | 10:49 |
| jrosser | gokhan__: https://bugs.launchpad.net/openstack-ansible/+bug/2055417 | 10:50 |
| noonedeadpunk | gokhan__: in a way, yes | 10:52 |
| noonedeadpunk | but for now - mostly the Yoga is | 10:53 |
| noonedeadpunk | though, it's fixed already | 10:53 |
| gokhan__ | ok now the only for yoga untamaintainted branch is created. | 10:54 |
| gokhan__ | before creation of unmaintained branchs of vwx ı need to complete my envs upgrade | 10:55 |
| gokhan__ | noonedeadpunk, it seems there is no fwaas integration in osa :( | 10:57 |
| noonedeadpunk | I think there was | 11:00 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/defaults/main.yml#L478-L483 ? | 11:01 |
| noonedeadpunk | gokhan__: https://docs.openstack.org/openstack-ansible-os_neutron/latest/configure-network-services.html#firewall-service-optional | 11:01 |
| noonedeadpunk | never tested it | 11:01 |
| jrosser | there was deprecation / undeprecation around fwaas i think? | 11:03 |
| jrosser | by neutron | 11:03 |
| noonedeadpunk | yeah | 11:06 |
| noonedeadpunk | there was | 11:06 |
| noonedeadpunk | I don't think we were fast enough to drop it... Or jsut reverted it once it was undeprecated | 11:06 |
| noonedeadpunk | so it can be absent indeed for some release | 11:06 |
| noonedeadpunk | gokhan__: can you please explain me the point/reason of fwaas? As I really not getting it... | 11:07 |
| noonedeadpunk | And how it's different from security groups, except that it works on net nodes rather then computes | 11:07 |
| gokhan__ | noonedeadpunk, I tested it but it didn't work. ı | 11:13 |
| gokhan__ | for me it is customer requirement. I am trying to understand how it works fistly | 11:13 |
| gokhan__ | neutron-fwaas is revived again | 11:14 |
| gokhan__ | also someone s working for ovn driver https://review.opendev.org/c/openstack/neutron-fwaas/+/845756 | 11:16 |
| noonedeadpunk | I guess it depends what the requirement actually is | 11:16 |
| noonedeadpunk | what is also meant under it not working? As it was quite abandoned at some point, and broken on it's own to the point of deprecation... | 11:17 |
| noonedeadpunk | But indeed there's might be smth off with the config as well | 11:18 |
| noonedeadpunk | As basically, default port security does cover all usecases for our customers at least... | 11:18 |
| noonedeadpunk | but indeed, might be easier to have a thing in front of the router... But then I'd really wonder how this is gonna work with ovn and distributed fips for instance... As trafic reaches computes regardless in this case | 11:20 |
| gokhan__ | I am trying how differs it from security groups. | 11:20 |
| noonedeadpunk | I guess it's mainly about where traffic gets filtered | 11:21 |
| noonedeadpunk | security groups are per VM, and this I guess on a router level | 11:22 |
| gokhan__ | noonedeadpunk, with fwaas you can block, drop and accept traffic with on router port, with security groups you can have allow rules | 11:26 |
| gokhan__ | ı will try if fwaas is working as expected | 11:27 |
| noonedeadpunk | security groups drop what is not allowed | 11:29 |
| noonedeadpunk | so I just see them as firewall with default drop rule, which is fair enough, I guess | 11:30 |
| noonedeadpunk | main benefit of fwaas probably - not to let traffic reach computes, and drop it earlier. But it makes sense to me only without DVR on a standalone net nodes | 11:30 |
| f0o | jrosser: FWIW setting the MTUs to 9k has indeed solved the network performance. It's now 10G VM<>VM cross hosts which is what the VIF QoS is set to. Thanks again! | 11:31 |
| noonedeadpunk | f0o: I guess you have multiqueue enabled as well? | 11:31 |
| f0o | you ring a bell but unless that's automagic no - I'm right now on a very "stock" setup what OSA produces. Just now getting my toes wet with full-blown overrides and config adjustments | 11:32 |
| f0o | noonedeadpunk: shouldnt hw:vif_multiqueue_enabled be in the "libvirt Driver Options" metadata collection? | 11:40 |
| jrosser | f0o: so on top of that config overrides concept, you should also be able to use `--tags neutron-config` to only run the ansible tasks which write out the config files / restart the services | 11:45 |
| jrosser | that should make adjusting config quicker to deploy, though if something doesnt work please ask as we have had bugs with this before | 11:45 |
| noonedeadpunk | um, well, it should be assigned to flavors or images. But it's surprising then you can get 10g on single core basically | 11:45 |
| noonedeadpunk | Unless today this is sorted out by different means with ovn... | 11:45 |
| jrosser | noonedeadpunk: i think andrewbonney tried multiqueue for us on linuxbridge and it somehow made things slower rather than faster | 11:46 |
| noonedeadpunk | heh | 11:46 |
| noonedeadpunk | it depends on amount of vm cores at least... | 11:46 |
| noonedeadpunk | not beneficial for 1 vcpu for sure | 11:47 |
| f0o | noonedeadpunk: hrm I see no difference in the perf between having it enabled or not on a 4vcpu instance - both hit 10G | 11:49 |
| jrosser | also dpdk is pretty misunderstood | 11:50 |
| jrosser | we had someone try really very hard to make it all work, with all the downsides that come with it | 11:50 |
| jrosser | only to finally understand that without a dpdk enabled workload in the guest it's kind of not worth it | 11:51 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon master: Address Django Deprecations for 4.1 https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/912107 | 12:52 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/2023.2: Determine if upgrade source branch is stable/ or unmaintained/ https://review.opendev.org/c/openstack/openstack-ansible/+/912434 | 12:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/2023.1: Determine if upgrade source branch is stable/ or unmaintained/ https://review.opendev.org/c/openstack/openstack-ansible/+/912435 | 13:02 |
| opendevreview | Merged openstack/ansible-role-uwsgi stable/zed: Remove undefined bionic linters job https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/910188 | 13:41 |
| nixbuilder | Got a question... evidently I need to set 'glance_rootwrap_conf_overrides' but not sure how to set it. In my 27.4.0 installation, I need to add '/openstack/venvs/glance-27.4.0/bin' to the exec_dirs in rootwrap.conf. But I am not sure how to do that in the user_variables.yml file. | 13:50 |
| nixbuilder | I guess what I am asking is how do I set it in user_variables.yml so that if I install this again it picks up the correct path (i.e. glance-27.4.0) | 13:51 |
| jrosser | nixbuilder: what specific problem do you have? | 13:56 |
| jrosser | i ask because there is this fix which seems to be related https://review.opendev.org/q/I4ee3fc33fdbeb50fc7b102bf62d6134f83c5925f | 13:57 |
| jrosser | that patch does exactly what you describe, add the venv/bin to exec_dirs automatically | 13:58 |
| nixbuilder | jrosser: After installing the patch did not seem to work... https://paste.openstack.org/show/bPpXOQOcr2zi4H1Y7HUT/ | 14:01 |
| nixbuilder | jrosser: So I had to manually add in the extra search directory. | 14:02 |
| jrosser | nixbuilder: when i look at one of our CI jobs, the path looks correct https://zuul.opendev.org/t/openstack/build/07f21b685d9c4de3839a786ab01c8562/log/logs/etc/openstack/aio1_glance_container-5e87a20e/glance/rootwrap.conf.txt | 14:04 |
| nixbuilder | jrosser: Don't know why mine turned out wrong... that's why I asked what to put in the 'glance_rootwrap_conf_overrides' in user_variables.yml. | 14:07 |
| jrosser | well, thats a bit catch-ww | 14:07 |
| jrosser | catch-22 even | 14:07 |
| jrosser | because the variable `glance_rootwrap_conf_overrides` was added by the patch that is not working for you | 14:08 |
| jrosser | but you can see from the patch what the default would be https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/901562/1/vars/main.yml | 14:08 |
| jrosser | here is the same CI result from a 27.x release https://zuul.opendev.org/t/openstack/build/b66378a8eb184380af0db820eff72b23/log/logs/etc/openstack/aio1_glance_container-3ba3f915/glance/rootwrap.conf.txt | 14:11 |
| nixbuilder | jrosser: I can see what you are saying... I see it in my '/etc/ansible/roles/os_glance/vars/main.yml'. But it still didn't work. I'll just have to check it after installation if I install it again. Weird. | 14:14 |
| jrosser | you should be able to see from the ansible output if it is templating the rootwrap file | 14:15 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_cloudkitty master: Add variable to configure storage options https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/912292 | 16:38 |
| f0o | neutron-ovn-metadata-agent is very fragile wow | 16:50 |
| f0o | it feels like it doesnt do reconnect well (or at all?) | 16:51 |
| f0o | once the connection drops it will just spam TLS errors over and over again, after you restart the service it's all fine | 16:51 |
| f0o | non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>: ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'> | 16:53 |
| f0o | this actually reads like a code-issue even | 16:53 |
| f0o | anyone else seen this before? | 16:53 |
| f0o | my guess, when I rebooted one of the network nodes (which hosts northd and gateway) neutron-ovn-metadata-agent went haywire and didnt fallback to the backup northd. Probably because those arent going through HAProxy.. So it's a bit unclear how it selects which northd instance it wants to connect to | 16:56 |
| f0o | what baffles me is that even when the orginial northd came back online the neutron-ovn-metadata-agent was still complaining and error-looping until it was manually restarted | 16:58 |
| f0o | https://bugzilla.redhat.com/show_bug.cgi?id=2115035 oh hey look at that | 16:59 |
| f0o | alright I'll stop buggering openstack-ansible and take a look at neutron and see if they can include https://github.com/ovsrobot/ovs/commit/f09a55946cc83583c2e93be632e50f51ea830322 | 17:04 |
| f0o | or I'm a bit confused who's actually responsible... It might be openstack-ansible since OSA decides which version of what is installed right? | 17:05 |
| jrosser | f0o: version of what? | 17:06 |
| jrosser | (sorry lots of moving parts and i'm not sure which you mean) | 17:07 |
| f0o | no worries | 17:09 |
| f0o | version the python package "ovs" | 17:10 |
| jrosser | and which release are you using? | 17:10 |
| f0o | tag stable/2023.2 | 17:10 |
| f0o | it installs /openstack/venvs/neutron-28.0.2.dev6/lib/python3.10/site-packages/ovs/ - so I'm guessing 28.0.2dev6 for OpenStack | 17:11 |
| f0o | and ovs is...ovs-2.17.1.post1 | 17:11 |
| jrosser | that is the openstack-ansible release/version | 17:11 |
| f0o | ovs-2.17.2 seems to contain the fix | 17:11 |
| f0o | jrosser: stable/2023.2 | 17:11 |
| jrosser | and what ends up in that venv is controlled prettyty much totally by https://github.com/openstack/requirements/blob/stable/2023.2/upper-constraints.txt#L189 | 17:12 |
| jrosser | so for those python package versions, that is something that OSA installes, but under the constrains defined in the openstack global upper-constraints | 17:12 |
| f0o | so 2024.1 (guessing that's the next version) will contain the ovs fix? | 17:13 |
| jrosser | you can see that here https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L192 | 17:13 |
| f0o | I think we can bump ovs to ovs-2.17.2 as a backport fix for the bug (https://bugzilla.redhat.com/show_bug.cgi?id=2115035 / https://bugs.launchpad.net/openvswitch/+bug/1985062) | 17:13 |
| jrosser | if the openstack requriements team agree that is acceptable, yes | 17:14 |
| jrosser | the best thing to do is to propose a change to the requirements repo, citing those bugs and see what feedback you get | 17:14 |
| f0o | alrighty let's see if gerrit still remembers me and hates me | 17:15 |
| jrosser | however, as this is openstack-ansible, there is usually a way to override these things too | 17:15 |
| jrosser | let me check | 17:15 |
| jrosser | f0o: if you look in your deploy node you should find /etc/openstack_deploy/upper-constraints/.... | 17:16 |
| jrosser | during the neutron install, it needs to get the constraints to use from somewhere, https://github.com/openstack/openstack-ansible-os_neutron/blob/master/defaults/main.yml#L84C6-L84C26 | 17:18 |
| jrosser | and the default is this https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/defaults/source_install.yml#L38 | 17:18 |
| f0o | oh so I can just edit the file in /etc/openstack_deploy/upper-constraint/... ? | 17:18 |
| jrosser | or even you can copy the file and hack it up a bit | 17:19 |
| jrosser | with a new name, and point neutron_upper_constraints_url to your modified file | 17:19 |
| jrosser | this does then become a bit of tech-debt that you are carrying and would need to be removed when you upgrade or that constraint gets fix for real | 17:20 |
| f0o | true | 17:20 |
| jrosser | so yes its totally possible to edit it in place | 17:20 |
| f0o | first things first - proposing the fix so it can be done the right way | 17:20 |
| f0o | then do a hackup to get my test-env working | 17:20 |
| jrosser | or there are enough hooks in the vars to allow you to have a completely customised constraints file for the whole deployment, or per service, as you need | 17:21 |
| jrosser | just beware that a new copy of the constraints file will be downloaded, possibly overwriting your adjustments if you edit in-place the existing file and re-run bootstrap-ansible | 17:23 |
| jrosser | so a copy would be safer in that respect | 17:23 |
| f0o | https://review.opendev.org/c/openstack/requirements/+/892892/1/upper-constraints.txt | 17:23 |
| f0o | why didnt this make it into my requirements tho? | 17:23 |
| f0o | shows as merged into 2023.2 | 17:23 |
| jrosser | huh | 17:23 |
| f0o | because it turns out they had it already fixed in January with https://review.opendev.org/c/openstack/requirements/+/892892 | 17:24 |
| f0o | and that was merged into 2023.1 | 17:24 |
| f0o | was it merged into the wrong branch? shouldnt it be 2023.2? | 17:25 |
| f0o | my gerrit-fu is really not that good, I just do random buttons and found this | 17:25 |
| jrosser | urgh thats not great | 17:25 |
| f0o | (: | 17:26 |
| jrosser | noonedeadpunk: did this really want to be on 2023.2 as well? https://review.opendev.org/c/openstack/requirements/+/892892 | 17:27 |
| opendevreview | Jimmy McCrory proposed openstack/openstack-ansible-os_neutron master: Use ansible_facts['processor_vcpus'] instead of fact variable https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/912481 | 17:35 |
| nixbuilder | jrosser: What openstack process starts the dnsmasq? | 17:35 |
| jrosser | nixbuilder: possibly the neutron dhcp agent? | 17:36 |
| nixbuilder | jrosser: OK... thanks! | 17:37 |
| opendevreview | Jimmy McCrory proposed openstack/openstack-ansible-galera_server master: Additional TLS configuration options https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/911009 | 18:03 |
| opendevreview | Jimmy McCrory proposed openstack/openstack-ansible master: Add check_hostname option to db healthcheck tasks https://review.opendev.org/c/openstack/openstack-ansible/+/911150 | 18:08 |
| opendevreview | Jimmy McCrory proposed openstack/openstack-ansible master: Remove obsolete nova_force_config_drive variable from docs https://review.opendev.org/c/openstack/openstack-ansible/+/912486 | 18:24 |
| *** jamesdenton__ is now known as jamesdenton | 20:05 | |
| opendevreview | Jimmy McCrory proposed openstack/openstack-ansible-os_neutron master: Use ansible_facts['processor_vcpus'] instead of fact variable https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/912481 | 20:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!