| *** ysandeep|out is now known as ysandeep | 04:49 | |
| *** ysandeep is now known as ysandeep|ruck | 04:49 | |
| noonedeadpunk | ElDuderino: hey. keepalived config in l3 namespaces are done by neutron only, while on controller it's configured by OSA. So you should not really compare these 2 configs as they are different in many parts and not supposed to match | 08:09 |
|---|---|---|
| noonedeadpunk | However, it's totally not ok, that all of l3 agents are shown as active for the router | 08:10 |
| noonedeadpunk | I've double cheked on my env and all keepalived in neutron have same weight | 08:13 |
| noonedeadpunk | Though from what I can tell it should not be an issue on it's own | 08:13 |
| noonedeadpunk | ElDuderino: what operating system and openstack version you run on your net nodes? | 08:14 |
| noonedeadpunk | As there was bunch of mess on neutron side with keepalived once ubuntu 20.04 has been released | 08:14 |
| noonedeadpunk | or better say - compatability between keepalived 1.8 and 2.0 which are defaults for 18.04 and 20.04 corresponsively | 08:15 |
| jrosser | keepalived will use the Mac or ip values to determine priority if everything else is equal I think? | 08:36 |
| noonedeadpunk | Yeah, I think it's some sort of hash based on that... | 09:06 |
| noonedeadpunk | Also checking on my keepalived confs for neutron, it seems that it's using multicast. Which means you must have l2 connection between net nodes by default | 09:06 |
| noonedeadpunk | I think unicast should be possible with some config setting | 09:07 |
| opendevreview | Marcus Klein proposed openstack/openstack-ansible-os_neutron master: Allow to set dnsmasq configuration options https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/864872 | 09:18 |
| admin1 | those using ovs + osa .. ( very common) .. how do you monitor your outbound traffic .. say you see a traffic of 2 gbit increase suddenly .. how do you map out to a specific vm ? | 10:30 |
| noonedeadpunk | I'd check sflow - then you will see source IP which can you map quite easily | 10:32 |
| noonedeadpunk | rather then check traffic on interface3s | 10:33 |
| admin1 | do you use the slow to something like hostmon ? | 10:34 |
| noonedeadpunk | Well, I used Wanguard previously, have no idea what's being used as of today | 10:38 |
| noonedeadpunk | kind of not my problem anymore :D | 10:39 |
| noonedeadpunk | but yes, mapping interface to vm is nightmare in ovs for me | 10:40 |
| noonedeadpunk | I think you need to check ovs flows or smth for that | 10:40 |
| damiandabrowski | admin1: if you don't have any tool in place and you just see traffic spike on some compute node: long time ago i written short guide "How to find a VM(s) to which the VXLAN traffic from br-vxlan tcpdump belongs" | 10:52 |
| damiandabrowski | it's quite nasty and there's definitely better way to achieve it but it saved me once or twice: https://paste.openstack.org/show/bMypmA8JJpNXQgwsyAKz/ | 10:53 |
| opendevreview | Jorge San Emeterio proposed openstack/openstack-ansible-os_tempest master: [DNM] Restarting glance before running tempest https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/862304 | 10:57 |
| *** dviroel|afk is now known as dviroel | 11:20 | |
| admin1 | damiandabrowski, thanks . i am more going in this direction => https://blog.sflow.com/2010/01/open-vswitch.html | 11:29 |
| admin1 | but instead of using conf, doing this https://mail.openvswitch.org/pipermail/ovs-dev/2010-July/165245.html | 11:29 |
| damiandabrowski | yeah, that's definitely more convenient method :D | 11:31 |
| *** dviroel_ is now known as dviroel | 11:38 | |
| dok53 | Hi all, quick one. When I spin up an instance (local storage or backend storage I have no migrate/live migrate option on the dropdown to move it between compute hosts.Should that be in OSA by default? | 12:04 |
| dok53 | It's ok, I found it :) | 12:07 |
| *** dviroel_ is now known as dviroel | 12:16 | |
| admin1 | damiandabrowski, all i can say is .. implementation was swift and its awesome :D | 12:18 |
| admin1 | full br-vlan and br-vxlan traffic monitoring and graphing | 12:18 |
| admin1 | was able to find out who and where easily in a graph | 12:19 |
| noonedeadpunk | yeah, sflow is awesome indeed. not 100% accurate but really great on trends and possible to react quickly and detect suspicious things (like too much incoming traffic on dns inside vm or smth like that) | 12:20 |
| noonedeadpunk | and quite a few tools to process it as well | 12:22 |
| admin1 | i had someone pushing 4.6g on a single interface .. | 12:22 |
| admin1 | i quiet like it :) | 12:22 |
| admin1 | the fact that he was able to push 4.6g in a single port | 12:22 |
| noonedeadpunk | I'm quite surprised... Does you have multiqueue enabled? | 12:26 |
| noonedeadpunk | *Do | 12:26 |
| admin1 | yep | 12:32 |
| noonedeadpunk | ah, ok, then it explains 4.6g :) | 12:46 |
| dok53 | Me again, live migrate works over the cli fine but is there a way I can enable it from horizon? | 12:47 |
| dok53 | Sorry my bad, needed to go to the admin section | 12:55 |
| opendevreview | Jorge San Emeterio proposed openstack/openstack-ansible-os_tempest master: [DNM] Restarting glance before running tempest https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/862304 | 14:05 |
| opendevreview | Jorge San Emeterio proposed openstack/openstack-ansible-os_tempest master: [DNM] Restarting glance before running tempest https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/862304 | 14:07 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_neutron master: add ovn ssl config https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/862403 | 14:29 |
| *** frenzy_friday is now known as frenzy_friday|doc | 14:43 | |
| mgariepy | admin1, do you have an ovn deployment >? | 14:44 |
| ElDuderino | @noonedeadpunk thanks a bunch. We're on on 18.04 and are running Rocky (I know, ugh). Our provisioning codebase works in other environments, but this is a diff switch fabric and server vendor (so I'll go digging on the l2 side to see if they are blocking multicast). | 14:56 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Nov 22 15:00:33 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| NeilHanlon | o/ | 15:03 |
| noonedeadpunk | #topic bug triage | 15:04 |
| noonedeadpunk | We have fresh bug regarding debian, pbr and our new Yoga release (25.2.0) | 15:05 |
| noonedeadpunk | https://bugs.launchpad.net/openstack-ansible/+bug/1997365 | 15:05 |
| noonedeadpunk | PBR should be constrained in u-c so I won't assume issue because of that | 15:06 |
| mgariepy | hmm | 15:06 |
| noonedeadpunk | Ubuntu is using py3.10 by default, Debian 3.9 which is by far only difference. | 15:07 |
| noonedeadpunk | Oh damn | 15:08 |
| noonedeadpunk | setuptools is not more constrained in u-c | 15:08 |
| damiandabrowski | sorry i'm not available today but i'd appreciate some input in internal tls changes when you have some time | 15:12 |
| damiandabrowski | https://review.opendev.org/q/topic:tls-backend | 15:12 |
| noonedeadpunk | https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/864783 is affacted by this bug | 15:13 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/5cdb8c6ede164be9abd29e8743fcd2f3/log/job-output.txt#5232 | 15:13 |
| noonedeadpunk | ok, I will push a fix for that | 15:14 |
| noonedeadpunk | #topic office hours | 15:15 |
| noonedeadpunk | Well, we've released 25.2.0 that includes rocky 9 support in Yoga | 15:15 |
| noonedeadpunk | so Yoga now supports both rocky 8 and 9 | 15:15 |
| noonedeadpunk | It's a bit weird as there's no reason to get 8 deployed - it's first release where R support was added and we're dropping 8 in Zed | 15:16 |
| noonedeadpunk | But anyway | 15:16 |
| noonedeadpunk | Zookeeper role is in fair shape I would say, except TLS part. I tried to follow what infra folks are doing to encrypt zookeeper, except using our pki role and failed with java trace that it can't read certs or smth | 15:17 |
| noonedeadpunk | I will spend more time to sort this out | 15:18 |
| noonedeadpunk | mgariepy: how's ovn tls is going? | 15:21 |
| noonedeadpunk | ah, you've jsut uploaded new patchset :) | 15:21 |
| mgariepy | it doing well :) | 15:21 |
| mgariepy | was working in my vm but needs some review | 15:22 |
| noonedeadpunk | ok, awesome | 15:22 |
| mgariepy | is your zookeeper patch in gerrit ? | 15:23 |
| noonedeadpunk | yup | 15:23 |
| noonedeadpunk | https://review.opendev.org/q/topic:osa%252Fzookeeper | 15:24 |
| noonedeadpunk | it even is passing ci | 15:24 |
| noonedeadpunk | but without TLS yet | 15:24 |
| noonedeadpunk | I'm going to add tls as a follow up patch | 15:24 |
| mgariepy | ha ok | 15:24 |
| noonedeadpunk | or well, almost passing it | 15:25 |
| noonedeadpunk | but ready to be tested :) | 15:25 |
| noonedeadpunk | or reviewed at least | 15:25 |
| mgariepy | yours is way bigger than mine ;p hahah | 15:25 |
| noonedeadpunk | btw skyline repo still has not been merged | 15:25 |
| noonedeadpunk | sorry for that hehe | 15:26 |
| mgariepy | hmm intermitent fail are no fun :/ | 15:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Constrain setuptools https://review.opendev.org/c/openstack/openstack-ansible/+/865297 | 15:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Constrain setuptools https://review.opendev.org/c/openstack/openstack-ansible/+/865160 | 15:31 |
| mgariepy | for zookeeper patch do we need additional logs / configs ? | 15:32 |
| mgariepy | meyba add them in there : https://github.com/openstack/openstack-ansible/blob/master/scripts/log-collect.sh | 15:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Constrain setuptools https://review.opendev.org/c/openstack/openstack-ansible/+/865160 | 15:33 |
| noonedeadpunk | mgariepy: oh, yes, good catch | 15:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add zookeeper deployment https://review.opendev.org/c/openstack/openstack-ansible/+/864750 | 15:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add zookeeper deployment https://review.opendev.org/c/openstack/openstack-ansible/+/864750 | 15:37 |
| mgariepy | awesome | 15:37 |
| noonedeadpunk | So I guess that zookeeper is by far the only thing out ones that we defined as required for Zed on PTG | 15:39 |
| noonedeadpunk | So I'm thinking to make beta release once we merge this plus ovn and some bugfixes | 15:39 |
| noonedeadpunk | Hopefully I will end tls part for zookeeper this week | 15:40 |
| mgariepy | if you need some help i can probably take a couple hours to help you with the tls stuff on zookeeper. | 15:40 |
| opendevreview | Jorge San Emeterio proposed openstack/openstack-ansible-os_tempest master: [DNM] Restarting glance before running tempest https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/862304 | 15:43 |
| opendevreview | Jorge San Emeterio proposed openstack/openstack-ansible-os_tempest master: [DNM] Restarting glance before running tempest https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/862304 | 15:44 |
| noonedeadpunk | Well, I think I followed infra scripts right in terms of pki role. But for some reason zookeeper doesn't like it | 15:49 |
| noonedeadpunk | Maybe I need to re-create containers... | 15:49 |
| noonedeadpunk | As if it's added to java storage, maybe it just refuses to add simmilar but a bit different key... Or with new key with same name... Hm | 15:49 |
| noonedeadpunk | But it felt super close. As actually I made to the point where client auth worked but stuck on cluster trafic encryption... | 15:50 |
| noonedeadpunk | I will ping you though for help if got completely stuck ) | 15:50 |
| noonedeadpunk | As I get some code but seems I'm passing smth wrong still... | 15:51 |
| mgariepy | okie | 15:53 |
| noonedeadpunk | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Tue Nov 22 16:00:42 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-11-22-15.00.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-11-22-15.00.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-11-22-15.00.log.html | 16:00 |
| mgariepy | hmm. | 16:13 |
| mgariepy | noonedeadpunk, https://zuul.opendev.org/t/openstack/build/f2bbed72811c448183318935bac01479 | 16:14 |
| mgariepy | same isssue with setuptool i guess ? | 16:14 |
| jrosser | sorry to miss the meeting | 16:15 |
| noonedeadpunk | yup | 16:15 |
| noonedeadpunk | no worries jrosser | 16:15 |
| noonedeadpunk | we've got gates broken because of setuptools | 16:15 |
| *** dviroel is now known as dviroel|lunch | 16:19 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Add variable for user defined list of deploy images https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/865310 | 16:22 |
| *** dviroel|lunch is now known as dviroel | 17:00 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update self version fetching mechanism https://review.opendev.org/c/openstack/openstack-ansible/+/865312 | 17:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Update self version fetching mechanism https://review.opendev.org/c/openstack/openstack-ansible/+/865166 | 17:10 |
| *** frenzy_friday|doc is now known as frenzy_friday | 17:36 | |
| mgariepy | anyone here have some experience with changing some policy ? | 18:19 |
| mgariepy | i wonder how bad it can get when upgrading. | 18:19 |
| damiandabrowski | we have a small nova policy change in one of our private clouds which allows users with custom role to create flavors | 18:20 |
| damiandabrowski | i haven't noticed any issues with that so far | 18:20 |
| mgariepy | how do you test that your change is not affecting anything else ? | 18:27 |
| damiandabrowski | besides reading release notes and running tempest tests before and after upgrade, I'm afraid we don't :/ mainly because it's a small change | 18:28 |
| damiandabrowski | maybe we should write some custom tempest tests in the future to cover that | 18:28 |
| mgariepy | some stake holder for a project want to have quite fine grained roles i am not 100% sure it worth all the effort tbh | 18:29 |
| *** dviroel is now known as dviroel|afk | 20:37 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!