| *** chandankumar is now known as chkumar|rover | 05:57 | |
| *** akahat is now known as akahat|ruck | 07:25 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use the python venv module to build the ansible runtime venv https://review.opendev.org/c/openstack/openstack-ansible/+/822273 | 07:53 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build master: Use the native python module to create virtual environments https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/822274 | 07:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: [WIP] Update ansible-core to 2.12.1 https://review.opendev.org/c/openstack/openstack-ansible/+/822063 | 07:55 |
| noonedeadpunk | mornings | 08:26 |
| noonedeadpunk | we would need to squash https://review.opendev.org/c/openstack/openstack-ansible/+/822273 and https://review.opendev.org/c/openstack/openstack-ansible/+/822260/ anyway I guess :( | 08:26 |
| noonedeadpunk | but https://review.opendev.org/c/openstack/openstack-ansible/+/822063 looks very promising :) | 08:27 |
| jrosser_ | i think i have the order of the patches wrong atm | 08:28 |
| noonedeadpunk | or that) | 08:29 |
| jrosser_ | this is blocking cleaning out centos-8 https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/820679 | 08:31 |
| andrewbonney | Possibly related to the lock dir changes, but not sure why we didn't see it before: [Errno 13] Permission denied: '/run/lock/zun/os-brick-connect_volume' | 08:36 |
| noonedeadpunk | yep, seems like related... but I bet I saw that and somehow were fixing... | 08:43 |
| jrosser_ | i think python_venv_build needs a bit of an overhaul | 08:45 |
| jrosser_ | interesting what we do for centos with two possible versions of python | 08:45 |
| jrosser_ | though i think we have code elsewhere to determine the python version | 08:46 |
| noonedeadpunk | eventually e can use 3.8 just for deploy host | 08:46 |
| noonedeadpunk | Not sure if openstack would require 3.8 for Yoga... | 08:47 |
| noonedeadpunk | as 3.6 EOLing in several month I guess? | 08:47 |
| noonedeadpunk | ah, even in 3 days :) | 08:47 |
| noonedeadpunk | so preferable would be to use 3.8 for venvs as well I believe | 08:48 |
| jrosser_ | theres stuff here to work out the version https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_install_symlink.yml | 08:49 |
| noonedeadpunk | um, I guess it's more to place rbd libraries inside venvs | 08:57 |
| noonedeadpunk | but yeah, for centos it will be an issue... | 08:57 |
| jrosser_ | where is the right place to ensure we have python3-venv installed everywhere | 08:59 |
| noonedeadpunk | but yeah, for centos it will be an issue... | 09:50 |
| noonedeadpunk | damn it, posted wrong msg | 09:50 |
| noonedeadpunk | so, right now we do it here https://opendev.org/openstack/openstack-ansible-openstack_hosts/src/branch/master/vars/ubuntu-20.04.yml#L53 | 09:50 |
| noonedeadpunk | I'm not sure if we need it outside of the python_venv_build though | 09:50 |
| noonedeadpunk | so I'd say we should add it to venv_build_base_distro_package_list | 09:51 |
| noonedeadpunk | we also have calico broken for sure :( it used to work though when we were merging stuff | 09:59 |
| noonedeadpunk | jrosser_: should we try to merge this https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/822238 and do rc2 based on that? | 10:03 |
| noonedeadpunk | or jsut merge https://review.opendev.org/c/openstack/openstack-ansible/+/822259 asap? | 10:03 |
| noonedeadpunk | (we still need rc2 as backported some bugfixes) | 10:04 |
| opendevreview | Merged openstack/openstack-ansible master: Bump cinder git hash to work around bug https://review.opendev.org/c/openstack/openstack-ansible/+/822108 | 10:09 |
| jrosser_ | it comes down to this https://github.com/openstack/ansible-role-python_venv_build/blob/master/defaults/main.yml#L39 | 10:14 |
| jrosser_ | that the packages to install on the venv install host is an empty list, all of the package definitions in that role are for the build host, not the target | 10:15 |
| admin1 | \o | 10:25 |
| andrewbonney | jrosser_: I can't replicate the issue for https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/820679 in an AIO. All permissions look fine and tests pass | 10:46 |
| *** sshnaidm|afk is now known as sshnaidm | 11:07 | |
| *** dviroel__ is now known as dviroel | 11:14 | |
| ygk_12345 | hi all | 12:37 |
| ygk_12345 | i have a train version of OSA | 12:37 |
| ygk_12345 | when I cheked the virsh defs of the vms on the hypervisors, i see its showing as custom cpu mode | 12:38 |
| ygk_12345 | but I cant find any configuration option in the nova.conf file | 12:38 |
| ygk_12345 | can someone explain how the cpu_mode is set to "custom" automatically and what models is it emulating ? | 12:39 |
| noonedeadpunk | if it's not set I believe it should be hypervisor model tbh | 12:44 |
| noonedeadpunk | but virsh xml should anyway disclose model that it emulates | 12:45 |
| ygk_12345 | noonedeadpunk: from the link https://docs.openstack.org/nova/latest/admin/cpu-models.html , custom should be set explicitly. then how my OSA is picking it up and the models it is emulating ? | 12:46 |
| noonedeadpunk | there was tricky thing about it... | 12:48 |
| ygk_12345 | noonedeadpunk: what is it ? | 12:48 |
| noonedeadpunk | so eventually, if you reference to https://docs.openstack.org/nova/train/configuration/config.html#libvirt.cpu_mode it will say that default is none. | 12:49 |
| noonedeadpunk | then description of none says that `default CPU model from QEMU will be used` which I guess triggers that "custom" model in xml | 12:50 |
| noonedeadpunk | I guess some common effect was for host-model maybe... when in xml it was ending up as custom. | 12:51 |
| noonedeadpunk | but not sure here | 12:51 |
| ygk_12345 | noonedeadpunk: why is it showing as custom , if its "none" in the xml def ? | 12:51 |
| ygk_12345 | noonedeadpunk: hmm sounds interesting | 12:51 |
| noonedeadpunk | wait. it's none in nova.conf | 12:51 |
| noonedeadpunk | nova.conf != xml def | 12:51 |
| ygk_12345 | noonedeadpunk: they both set the cpu mode. isn't it ? | 12:52 |
| noonedeadpunk | It doesn't mean they're necessarily interpreted 1 to 1 | 12:53 |
| ygk_12345 | noonedeadpunk: and for cusotm mode, we have to specify the list of models from old to new | 12:53 |
| ygk_12345 | noonedeadpunk: according to the doc | 12:53 |
| noonedeadpunk | oh, well, according to https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L5089 when cpu_mode is none it should be "host-model" | 12:57 |
| ygk_12345 | noonedeadpunk: i got it now. so its defaulting to host-model. but its showing as "custom" in the xml | 13:00 |
| ygk_12345 | noonedeadpunk: strange | 13:00 |
| noonedeadpunk | I think it comes to https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html | 13:02 |
| noonedeadpunk | As host model is kind of specific thing in libvirt and kind of "custom" thing anyway | 13:04 |
| noonedeadpunk | `Libvirt supports a third way to configure CPU models known as “Host model”. This uses the QEMU “Named model” feature, automatically picking a CPU model that is similar the host CPU, and then adding extra features to approximate the host model as closely as possible.` | 13:04 |
| noonedeadpunk | So it's custom :) | 13:04 |
| noonedeadpunk | ygk_12345: ^ | 13:04 |
| ygk_12345 | noonedeadpunk: oh I see it now. thanks bro for your timely help :) | 13:05 |
| noonedeadpunk | np) | 13:06 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/xena: Make calico non voting https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/822239 | 13:19 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/xena: Add Support for DPDK Bonding https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/822238 | 13:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Bump OpenStack-Ansible for Xena RC2 and release https://review.opendev.org/c/openstack/openstack-ansible/+/822259 | 13:56 |
| jamesdenton | good morning, all | 14:04 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_ironic master: Update Ironic Inspector for Metal Deployments https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/821503 | 14:44 |
| spatel | morning J | 14:49 |
| noonedeadpunk | o// | 15:01 |
| spatel | jamesdenton was little busy dealing with other stuff, i still need to test this - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/822238 | 15:42 |
| spatel | how urgent this is? | 15:42 |
| spatel | looking at code i am 100% its going to work because its pretty straight forward | 15:43 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: Remove testing on Centos-8 https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/820679 | 15:43 |
| jamesdenton | it should work? :D | 15:45 |
| jamesdenton | test at your convenience | 15:45 |
| spatel | +1 :) | 15:45 |
| jamesdenton | when we're confident that it works as expected, then it can be extended to non-dpdk OVS, too | 15:45 |
| spatel | i have lab box but it has only 2 nic and for this test i need min 3 nic | 15:45 |
| spatel | perfect! | 15:46 |
| jamesdenton | yes. although, you *might* be able to get away with just listing a single bond member | 15:46 |
| jamesdenton | if you wanted to test the mechanics of it | 15:46 |
| spatel | hmm that is also a option | 15:46 |
| spatel | i was thinking i can build compute no VMware vm which i can add multiple NIC and use uio driver to bind DPDK | 15:47 |
| spatel | jamesdenton question on sriov compute node we don't need neutron-linuxbridge-agent service correct? | 16:03 |
| spatel | I wonder because of both agent running on compute node it double RabbitMQ connections and load | 16:07 |
| spatel | because of group_binds: its installing linuxbridge_agent on my SRIOV compute nodes also - https://paste.opendev.org/show/811774/ | 16:41 |
| spatel | can i set this variable for host specific instead of global ? | 16:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Update supported operating systems for Yoga release https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/822343 | 16:52 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: [WIP] Update ansible-core to 2.12.1 https://review.opendev.org/c/openstack/openstack-ansible/+/822063 | 16:53 |
| jrosser_ | something breaks oddly on the centos jobs for ^ that patch regarding creating the keystone db | 16:56 |
| jrosser_ | which would be the first interaction we have with the database i guess | 16:56 |
| noonedeadpunk | smth related to delegate and missing libselinux ? | 17:17 |
| spatel | My neutron-server stein chewing up memory like hell i believe it has memory leak | 18:43 |
| noonedeadpunk | spatel: it really does hehe | 19:01 |
| noonedeadpunk | they published even security patch for that iirc | 19:01 |
| spatel | related memory leak? | 19:02 |
| spatel | i am trying to search but didn't find anything | 19:02 |
| noonedeadpunk | https://security.openstack.org/ossa/OSSA-2021-006.html | 19:02 |
| spatel | oh | 19:02 |
| noonedeadpunk | not sure if that's what you see, but well... | 19:03 |
| spatel | This looks interesting - https://launchpadlibrarian.net/556474193/0001-Don-t-use-singleton-in-routes.middleware.RoutesMiddl.patch | 19:05 |
| spatel | is there a way to find out its really the case for me? | 19:06 |
| spatel | I have Queens deployment and i never ever had any single issue on queens, memory utilization is pretty low for all openstack components | 19:08 |
| spatel | Stein is full of crap.. i am seeing very high memory utilization for neutron and because of that its impacting other components like rabbitMQ etc.. | 19:09 |
| spatel | currently all i can do is restart neutron-server everyday | 19:09 |
| spatel | If this bug is real then it should impact queens deployment also | 19:11 |
| noonedeadpunk | yeah, that's true. but eventually it depends on the "user" actions, as it's not appers on its own... | 19:12 |
| noonedeadpunk | but it could be unrelated as well, that's true | 19:13 |
| spatel | can i hand edit this patch and watch few days? | 19:13 |
| noonedeadpunk | eventually neutron have some issues I think anyway, as otherwise uwsgi would work just nicely | 19:13 |
| noonedeadpunk | yes, you can, but eventually there were more security issues found in neutron in 2021 | 19:13 |
| noonedeadpunk | ie https://security.openstack.org/ossa/OSSA-2021-001.html or https://security.openstack.org/ossa/OSSA-2021-005.html | 19:14 |
| noonedeadpunk | so tbh I'd just updated neutron to latest stein SHA | 19:15 |
| spatel | in that case do i need to update db? | 19:16 |
| noonedeadpunk | um, no? But role would trigger that anyway I guess | 19:16 |
| noonedeadpunk | it shouldn't change anything though | 19:16 |
| spatel | hmm! let me first do this - https://review.opendev.org/c/openstack/neutron/+/807335/2/neutron/api/extensions.py#204 | 19:17 |
| spatel | hand edit this file and restart neutron and watch for memory utilization.. | 19:18 |
| spatel | if this patch fix my memory leak then i don't need to poke beast :) | 19:18 |
| spatel | I am doing this on only single neutron-server so i can compare with other neutron-servers | 19:22 |
| spatel | if memory continue creep up then i will revert that change | 19:22 |
| jrosser_ | looks like just centos to fix now with ansible 2.12 patch, the debian-alikes are passing | 19:25 |
| spatel | noonedeadpunk after that patch i can see lower utilization on patched node compare to other still its too early to say. i should wait for 24 hours | 19:43 |
| *** dviroel is now known as dviroel|afk | 20:06 | |
| jamesdenton | spatel you don't need linuxbridge agent on the compute w/ sr-iov if that's all its doing | 22:03 |
| jamesdenton | and yes, it's the group bindings causing that. you can create new groups and use this: https://docs.openstack.org/openstack-ansible/latest/user/prod/provnet_groups.html | 22:04 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_ironic master: Update Ironic Inspector for Metal Deployments https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/821503 | 22:12 |
| spatel | jamesdenton thank!! i will look into it.. | 22:14 |
| spatel | after removing lxb from sriov nodes it reduce rabbit connection count 50% | 22:14 |
| jamesdenton | oh nice - how many did you have? | 22:14 |
| spatel | i have 200 compute nodes | 22:18 |
| spatel | each one was creating 10 TCP connection | 22:18 |
| spatel | total connection was 4000k around but after removing lxb from sriov now its showing 2000 | 22:19 |
| spatel | sorry 4k i meant | 22:19 |
| spatel | Look like time to deploy ironic :) | 22:38 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_ironic master: Update Ironic Documentation https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/822203 | 22:41 |
| jamesdenton | yes, please :) | 22:41 |
| jamesdenton | i've got it all working pretty well, but the key to a nice setup is using an ML2 plugin (like networking-generic-switch), and i don't know how popular that idea is | 22:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!