| *** ysandeep|out is now known as ysandeep|rover | 05:04 | |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 06:15 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 06:48 | |
| noonedeadpunk | mornings | 07:51 |
|---|---|---|
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:00 | |
| jrosser | good morning | 08:03 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Move slow_query_log_file to location handled by logrotation https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/841415 | 09:05 |
| jrosser | noonedeadpunk: did you understand that nfv/openvswitch thing yet? i was just looking at reviews and saw this https://opendev.org/openstack/openstack-ansible-galera_server/src/branch/master/templates/my.cnf.j2#L47 | 09:10 |
| jrosser | argh | 09:10 |
| jrosser | this https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/803630 | 09:11 |
| noonedeadpunk | I tried to get to spawn aio to check that yestarday but didn't managed to due to meetings and other crap | 09:11 |
| noonedeadpunk | but I thought that nfv-openvswitch being installed on hosts and countainers should be neough | 09:12 |
| jrosser | iirc this was to do with the yum config being copied from the host onto the container fs | 09:13 |
| jrosser | but some of it is in those vars files, and got left behind | 09:13 |
| noonedeadpunk | well, yes, but why.... | 09:13 |
| jrosser | not sure if this is what you're seeing though | 09:13 |
| noonedeadpunk | so answering question - no, I don't understand that yet | 09:14 |
| * noonedeadpunk bootstrapping aio finally | 09:19 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 09:21 | |
| noonedeadpunk | issue reproduced | 09:34 |
| jrosser | cool | 09:35 |
| noonedeadpunk | So basically it misses /etc/dnf/vars/nfvsigdist with content `8-stream`.... | 09:38 |
| noonedeadpunk | As mirrorlist defined as `mirrorlist=http://mirrorlist.centos.org/?release=$nfvsigdist&arch=$basearch&repo=nfv-openvswitch-2` | 09:38 |
| noonedeadpunk | jrosser: do you think we need doing this? https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/vars/redhat.yml#L36 | 10:03 |
| noonedeadpunk | As we don't copy sources.list.d for deb.... | 10:04 |
| noonedeadpunk | But I'm not sure actually if any repos will be present without it... let me check... | 10:04 |
| noonedeadpunk | SO the only thing we need from host - epel repo for systemd-networkd | 10:24 |
| jrosser | noonedeadpunk: probably the idea there is if an environment has custom repos (like internal mirrors or whatever) then those get copied into the lxc base image | 10:37 |
| jrosser | otherwise i think we would have a problem when those containers first boot as that kind of config is out of the scope of OSA | 10:37 |
| jrosser | you are right it's not sources.list.d but we do have this https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/vars/debian.yml#L23 | 10:38 |
| noonedeadpunk | jrosser: it's kind of confusing. As when preparing chroot we install upstream repos https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/vars/redhat.yml#L17 | 10:42 |
| noonedeadpunk | so basically, we should be good with them anyway | 10:42 |
| noonedeadpunk | or env prep would fail on build command? | 10:42 |
| jrosser | tbh this is all kind of confusing | 10:43 |
| noonedeadpunk | so based on that I'd say it makes sense to to not copy yum.repos.d from host, but just add epel to container just for networkd | 10:43 |
| jrosser | becasue i just made a centos-8 aio and it didnt break like this | 10:44 |
| jrosser | maybe i did it wrong :/ | 10:44 |
| noonedeadpunk | it's distro install only that fails :) | 10:44 |
| noonedeadpunk | what is confusing is that later on we cleaning out repos copied from host https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/templates/prep-scripts/redhat_prep.sh.j2#L13 | 10:45 |
| noonedeadpunk | So there're 2 solutions I see: 1. drop copying yum.repos.d and add epel manually inside chroot. 2. In prep script drop nfv repo that was copied | 10:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Clean up NFV OpenvSwitch repo for container build https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/842086 | 10:53 |
| noonedeadpunk | I'm a bit afraid if https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/842086 and https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/840353 arte circular... | 10:54 |
| * jrosser just broke keyboard, back now.... | 10:54 | |
| jrosser | drink + keyboard = bad | 10:54 |
| noonedeadpunk | that's fun and sad at the same time :) | 10:56 |
| * noonedeadpunk really hopes it wasn't some fancy mechanical one | 10:56 | |
| jrosser | i've put it outside in the sun to dry :( | 10:56 |
| jrosser | no just old apple wired USB one | 10:56 |
| noonedeadpunk | then good opportunity to have smth better :D | 10:58 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Remove all code for lsync, rsync and ssh https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837588 | 11:06 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Clean up legacy lsycnd, rsync and ssh key config https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/837859 | 11:06 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Use the same vars file for all versions of centos https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/841618 | 11:06 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-repo_server master: Use distro packages for nginx on centos. https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/841619 | 11:06 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Limit cloudkitty workers https://review.opendev.org/c/openstack/openstack-ansible/+/841939 | 11:11 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/wallaby: Limit cloudkitty workers https://review.opendev.org/c/openstack/openstack-ansible/+/841940 | 11:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Limit mistral worker https://review.opendev.org/c/openstack/openstack-ansible/+/841941 | 11:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/wallaby: Limit mistral worker https://review.opendev.org/c/openstack/openstack-ansible/+/841942 | 11:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/835717 | 11:14 |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 11:18 | |
| jrosser | something very wrong with dns_domain here https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/827251 | 11:19 |
| jrosser | feels specific to ovs? | 11:19 |
| noonedeadpunk | `Unrecognized attribute(s) 'dns_domain'` -> I can recall that | 11:23 |
| noonedeadpunk | and that I saw that nowhere in confs... and likely tighten to neutron-lib version, but not sue | 11:23 |
| noonedeadpunk | At least I put time on that some month ago without any result | 11:24 |
| *** dviroel|out is now known as dviroel | 11:28 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - Centos-9 Stream support https://review.opendev.org/c/openstack/openstack-ansible/+/823417 | 11:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - Centos-9 Stream support https://review.opendev.org/c/openstack/openstack-ansible/+/823417 | 11:43 |
| damiandabrowski[m] | jrosser: today I'm going to focus on testing gluster implementation, do You have any scenarios in mind that You would like me to test? | 11:44 |
| jrosser | i think that the aio_lxc_infra is most insteresting as you get 3x repo containers | 11:45 |
| jrosser | and maybe we use this to write some docs for create/delete containers, or replace a host? | 11:46 |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 11:51 | |
| damiandabrowski[m] | that's already done on focal and seems to be working. I'll also try to test what happens when one(or more) containers go down and and transition lsyncd->gluster during and ugrade | 11:52 |
| jrosser | ah yes, i have a whole bunch of patches which should managed the upgrade, and also clean up the lsyncd / ssh config | 11:53 |
| damiandabrowski[m] | If I have enough time, I'll also try to test multi-distro gluster scenario | 11:53 |
| damiandabrowski[m] | great | 11:53 |
| damiandabrowski[m] | hmm, don't we already have something about containers recreation in our docs? let me have a look | 11:54 |
| jrosser | i just added one more depends-on to those and hopefully we should see that in CI if it works too | 11:54 |
| jrosser | though i'm worried about that being circular dependancy | 11:54 |
| jrosser | the extra /openstack/gluster mount is important, and if the contents get kept or not | 11:54 |
| damiandabrowski[m] | i think it's not that crucial to keep existing repo content when switching to gluster, right? | 11:56 |
| jrosser | my patch should do that at upgrade time | 11:56 |
| damiandabrowski[m] | it can be easily recreated anyway, and during openstack upgrade it has to be recreated AFAIK | 11:57 |
| damiandabrowski[m] | okok | 11:57 |
| jrosser | there is a whole bunch of stuff here for that https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/839411/13/tasks/repo_pre_install.yml | 11:58 |
| damiandabrowski[m] | great | 11:59 |
| damiandabrowski[m] | regarding container recreation in our docs, we have it but only in context of galera: https://docs.openstack.org/openstack-ansible/latest/admin/maintenance-tasks.html#rebuild-a-container | 12:00 |
| damiandabrowski[m] | i wonder if it's enough or we want to have something more universal? | 12:00 |
| jrosser | oh interesting, we can't uninstall rsync as that breaks metal jobs | 12:36 |
| jrosser | the zuul log collection requires that | 12:36 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Return Erlang distribution port mgmt binding https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/830151 | 12:39 |
| jrosser | i wonder if it is reasonable to uninstall rsync - particularly for a metal job we don't know if it is needed for something else | 12:40 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Support dns_domain_keywords extension https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/842102 | 13:10 |
| jrosser | spatel: could you make a launchpad bug for the lxc/btrfs thing we talked about yesterday? | 13:11 |
| spatel | jrosser sure, i can but let me ask this question. why OSA can't override that file? | 13:12 |
| jrosser | i think right now there is a fight between OSA placing the file and it coming from the distro package | 13:12 |
| spatel | how do i explain this in launchpad that its a bug? | 13:13 |
| jrosser | i think just describe that the contents of the file on your system is different to what we think that the lxc_hosts role is placing | 13:14 |
| jrosser | something is really not right becasue i just look at an AIO i made this morning and still the file is incorrect there | 13:15 |
| jrosser | yet at the same time the mount exists and is ext4 | 13:15 |
| spatel | ok let me put some wording around | 13:16 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Move slow_query_log_file to location handled by logrotation https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/841415 | 13:16 |
| spatel | is this correct place to open bug? - https://bugs.launchpad.net/bugs/bugtrackers/lxc | 13:16 |
| jrosser | oh no, for OSA | 13:17 |
| spatel | ohhh | 13:17 |
| jrosser | https://bugs.launchpad.net/openstack-ansible | 13:17 |
| spatel | copy that | 13:17 |
| spatel | jrosser https://bugs.launchpad.net/openstack-ansible/+bug/1973762 | 13:26 |
| mgariepy | fun a couple of mariadb cve.. https://nvd.nist.gov/vuln/detail/CVE-2022-27376%20https://nvd.nist.gov/vuln/detail/CVE-2022-27377%20https://nvd.nist.gov/vuln/detail/CVE-2022-27380 | 13:30 |
| spatel | Page Not Found | 13:32 |
| mgariepy | arf. stupid %20 :/ | 13:32 |
| mgariepy | https://nvd.nist.gov/vuln/detail/CVE-2022-27376 | 13:33 |
| mgariepy | https://nvd.nist.gov/vuln/detail/CVE-2022-27377 | 13:33 |
| mgariepy | https://nvd.nist.gov/vuln/detail/CVE-2022-27380 | 13:33 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-galera_server master: Bump mariadb version. https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/842106 | 13:39 |
| mgariepy | noonedeadpunk, jrosser should we backport this one or shoud we just do the change in all the branches ? | 13:41 |
| noonedeadpunk | We can backport that to X I believe and would need to change in other ones | 13:42 |
| noonedeadpunk | As they have also different major versions | 13:42 |
| noonedeadpunk | But mariadb hasn't release 10.6.8 yet? | 13:43 |
| noonedeadpunk | As I don't see anything in https://mariadb.org/download/?t=mariadb&p=mariadb&r=10.6.7&os=Linux&cpu=x86_64&pkg=tar_gz&i=systemd&m=nextlayer | 13:43 |
| mgariepy | hmm, ok let's wait a day or 2.. | 13:46 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: haproxy: fix csp for glance image uploads via horizon https://review.opendev.org/c/openstack/openstack-ansible/+/842111 | 13:47 |
| opendevreview | Merged openstack/openstack-ansible master: Remove workaround for pre-release debian bullseye https://review.opendev.org/c/openstack/openstack-ansible/+/823491 | 14:01 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Be even more explicit about lb_vip_address https://review.opendev.org/c/openstack/openstack-ansible/+/842132 | 14:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Add passwd for rhel8. https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/840353 | 14:59 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue May 17 15:00:40 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| damiandabrowski[m] | hi! | 15:01 |
| jrosser | o/ hello | 15:04 |
| noonedeadpunk | #topic hours | 15:04 |
| noonedeadpunk | so, evenctually, as you might all guess, I haven't done anything from what I promised to do previous week which upsets me a lot.... | 15:05 |
| noonedeadpunk | Like 10 mins before meeting I started looking into PKI stuff for octavia as that seemed pretty close | 15:06 |
| noonedeadpunk | Didn't have chance to play with gluster stuff though | 15:07 |
| noonedeadpunk | but from codebase I don't see any big issues right now | 15:07 |
| jrosser | i nearly have the whole stack passing CI now https://review.opendev.org/q/topic:osa-gluster | 15:08 |
| jrosser | just problem with uninstalling rsync on metal jobs breaks the CI log upload :) | 15:08 |
| jrosser | but only for ubuntu /o\ | 15:09 |
| noonedeadpunk | I'm not sure I actually also realized whole mess with phased packages :( https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/840518 | 15:10 |
| noonedeadpunk | Do we need to enable phased updates also for Focal? | 15:11 |
| jrosser | oh yes there was a big mess with that | 15:11 |
| jrosser | phasing is only after Focal | 15:11 |
| noonedeadpunk | then we should add some conditional inside script for os release? | 15:12 |
| noonedeadpunk | as now it's a global prep script | 15:12 |
| noonedeadpunk | (which also targets Debian) | 15:13 |
| jrosser | we could do, though it does seem to be OK | 15:13 |
| noonedeadpunk | ok then | 15:13 |
| noonedeadpunk | I guess it's last patch to land for yammy? | 15:14 |
| jrosser | APT appears to disregard that config on the other OS | 15:14 |
| jrosser | yes, expect for this https://review.opendev.org/c/openstack/openstack-ansible/+/839483 | 15:14 |
| jrosser | and i don't know how many jobs we want to introduce right now | 15:14 |
| jrosser | that only puts jammy in the check pipeline and NV as well | 15:15 |
| jrosser | which might be reasonable for 'experimental' support then it gets switched to voting/gate for Z? | 15:15 |
| noonedeadpunk | yup, I guess it's fair atm | 15:16 |
| jrosser | i expect that the patches generated during branching will throw up a bunch of role failures for jammy that we've not tried yet | 15:16 |
| noonedeadpunk | And indeed we should switch to voting in Z | 15:16 |
| noonedeadpunk | I won't be surprised :) | 15:17 |
| noonedeadpunk | And indeed NV sounds quite suitable right now for it | 15:17 |
| noonedeadpunk | considering there's no maria/rabbit at very least | 15:17 |
| jrosser | then we have centos-9 i guess | 15:18 |
| jrosser | https://review.opendev.org/c/openstack/openstack-ansible/+/823417 \o/ | 15:18 |
| jrosser | and a few small things left in here https://review.opendev.org/q/topic:osa%252Fcentos-9 | 15:19 |
| noonedeadpunk | And eventually we should make it voting kind of.... | 15:19 |
| noonedeadpunk | as centos-8 is being dropped in Z | 15:19 |
| noonedeadpunk | (or well, py3.6 is) | 15:19 |
| jrosser | right - though we have to decide what to do about LXC | 15:19 |
| noonedeadpunk | let me guess - it's absent for it? | 15:20 |
| jrosser | i just don't know, other than have some experimental code path that uses a snap, and just say "don't use this" | 15:20 |
| jrosser | i could add code to lxc_hosts to install it via snap | 15:20 |
| jrosser | use at own risk :) | 15:20 |
| noonedeadpunk | so it's provided only by lxd? | 15:21 |
| noonedeadpunk | well yeah, lxc for centos 8 that we're using also not maintained actually | 15:22 |
| jrosser | oh hmm | 15:23 |
| jrosser | maybe i misunderstand that lxc can't be installed on it's own with a snap | 15:23 |
| *** ysandeep|rover is now known as ysandeep|out | 15:24 | |
| noonedeadpunk | oh, so you mean there's only lxd option now? | 15:26 |
| jrosser | snapcraft.io only seems to show a LXD snao | 15:28 |
| jrosser | snap | 15:28 |
| jrosser | thats going to bring in a whole load of stuff we don't want to manage, i think | 15:28 |
| jrosser | annoyingly there is lxc4.x in fedora | 15:28 |
| jrosser | i wonder how hard it is to use copr to build what we want | 15:32 |
| jrosser | as really thats what we already use for c8 i think | 15:32 |
| damiandabrowski[m] | i wonder what long-term solutions do we have to make our lives easier...switch to LXD? maybe systemd-nspawn? | 15:33 |
| jrosser | NeilHanlon: would you have any tips on how we could get lxc packages for centos9? | 15:33 |
| jrosser | given that we already do this https://github.com/openstack/openstack-ansible-lxc_hosts/blob/9a4004169450a0bb68ffe0fec31e6887a9c075f3/vars/redhat-host.yml#L19-L20 | 15:33 |
| jrosser | damiandabrowski[m]: i made some initial start on LXD support maybe a year ago | 15:33 |
| jrosser | but it would suffer from the same problem that the only viable installation on all platforms is a snap | 15:34 |
| jrosser | and thats got it's own problems | 15:34 |
| noonedeadpunk | damiandabrowski[m]: we jsut got rid of nspawn as it's support is absent on centos because of dropped btrfs | 15:35 |
| noonedeadpunk | everything leads to containerd lol | 15:35 |
| damiandabrowski[m] | jrosser: ahh :/ have we noted these problems somewhere? | 15:35 |
| jrosser | fwiw we use LXD here for a bunch of things outside OSA, and before we "deliberately broke" the mandatory auto-update thing we had * break simultaneously because of new buggy lxd versions then get pushed onto you | 15:35 |
| damiandabrowski[m] | noonedeadpunk: fair point, thanks | 15:36 |
| noonedeadpunk | yeah, auto-update of snap is nightmare | 15:36 |
| noonedeadpunk | I can imagine that for galera as example.... | 15:37 |
| noonedeadpunk | jrosser: you;re right, fedora has even lxc4.... | 15:37 |
| jrosser | i understand zero about copr but i am guessing we can use that to compile the .spec file for fedora into a centos9 package | 15:37 |
| noonedeadpunk | Basically, if sombody would volunteer to maintain lxc in epel for centos/rocky that would be awesome | 15:38 |
| jrosser | or something similar | 15:38 |
| noonedeadpunk | I also just found that https://koschei.fedoraproject.org/package/lxc?collection=epel8 - is it same thing? | 15:39 |
| jrosser | that looks like where it comes from i think | 15:40 |
| jrosser | that looks like we should even find lxc3 for centos8 in epel | 15:40 |
| noonedeadpunk | Oh, ok I even submitted a bug there.... https://bugzilla.redhat.com/show_bug.cgi?id=2034709 | 15:40 |
| jrosser | it's all here https://src.fedoraproject.org/rpms/lxc/blob/f36/f/lxc.spec | 15:44 |
| jrosser | anyway maybe we should look at bugs, we're going to be out of time | 15:45 |
| noonedeadpunk | #topic bug triage | 15:46 |
| noonedeadpunk | I'm looking at https://bugs.launchpad.net/openstack-ansible/+bug/1967270 at the moment | 15:46 |
| noonedeadpunk | I think it's quite fair | 15:47 |
| jrosser | indeed | 15:48 |
| * jrosser fixes | 15:48 | |
| noonedeadpunk | Another one that bothers me was https://bugs.launchpad.net/openstack-ansible/+bug/1971179 | 15:49 |
| noonedeadpunk | but I don't find it _that_ trivial | 15:49 |
| noonedeadpunk | andrewbonney wrote that related fix is not really covering issue | 15:50 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_octavia master: Fix condition for deleting old amp images https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/842145 | 15:50 |
| noonedeadpunk | but I'm didn't have time to have a deep dive into CSP. And X default worked for us nicealy, but we don't use horizon in fact | 15:50 |
| jrosser | i think andrewbonney may have addresses the CSP stuff | 15:51 |
| jrosser | i have seen other reports of that RBD size mismatch too and never understood what was actually happening there | 15:52 |
| noonedeadpunk | damiandabrowski[m]: was playing with that for a while | 15:53 |
| noonedeadpunk | But basically it's indeed all about chunking | 15:54 |
| noonedeadpunk | disabling uwsgi helps here if anybody facing it | 15:54 |
| noonedeadpunk | basically glanceclient doesn't support chunking, while openstackclient does. | 15:54 |
| noonedeadpunk | So there can be great confusion/mess between services who does use what | 15:55 |
| damiandabrowski[m] | yeah, but i haven't found out yet why uwsgi is causing these issues :/ | 15:55 |
| jrosser | it is a shame there is no activity from the glance people on that bug for > 1 year | 15:55 |
| jrosser | there was some stuff from the debian packaging which was about enabling chunking for uwsgi? | 15:55 |
| noonedeadpunk | well... they were not very active in fixing uwsgi at all | 15:55 |
| noonedeadpunk | well yes, but client should also support that | 15:56 |
| noonedeadpunk | (I guess) | 15:56 |
| jrosser | "If you plan to put uWSGI behind a proxy/router be sure it supports chunked input requests (or generally raw HTTP requests)." | 15:57 |
| noonedeadpunk | At least in openstacksdk I see that https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/image/iterable_chunked_file.py | 15:57 |
| noonedeadpunk | And there's nothing close to that in glanceclient. And it has own implementation of client object as well | 15:58 |
| damiandabrowski[m] | but on the other hand, if i recall correctly, i noticed these issues only when using uwsgi+ceph backend | 15:58 |
| damiandabrowski[m] | uwsgi+file backend worked fine | 15:58 |
| damiandabrowski[m] | so i don't really understand what's going on | 15:59 |
| noonedeadpunk | so damiandabrowski[m] was able to reproduce issue nicely using glance image-create, but not with openstack image create | 15:59 |
| noonedeadpunk | well yeah, maybe ceph adds some extra thing here as well.... | 16:00 |
| noonedeadpunk | As then you upload client -> haproxy -> uwsgi -> ceph and each should chunk correctly | 16:01 |
| noonedeadpunk | maybe we're somewhere out of sync in that chain.... | 16:01 |
| jrosser | that bug is talking about snapshots written to ceph backend isnt it? | 16:01 |
| noonedeadpunk | if you take snapshot from instance - it's kind of same? | 16:02 |
| noonedeadpunk | as you have a file you need to upload to glance as an image | 16:02 |
| *** dviroel is now known as dviroel|lunch | 16:02 | |
| jrosser | but potentially a different client code there? | 16:03 |
| noonedeadpunk | yeah | 16:03 |
| damiandabrowski[m] | +1 | 16:03 |
| jrosser | right now i'm not totally clear what is / is not working | 16:03 |
| noonedeadpunk | and I guess horizon still uses glance client as well | 16:03 |
| jrosser | might be worth updating the bug with what we know | 16:04 |
| noonedeadpunk | damiandabrowski[m]: can you do that? | 16:04 |
| jrosser | like if `openstack` cli is fine then thats at least one data point | 16:04 |
| damiandabrowski[m] | yeah, i plan to come back to this issue when I'll have some time | 16:04 |
| damiandabrowski[m] | our internal ticket about that is still open | 16:04 |
| noonedeadpunk | yeah, I know :) But it's good point we should put that we're working and what's the progress (even if it's close to none) | 16:05 |
| noonedeadpunk | and we're out of time... | 16:06 |
| noonedeadpunk | #endmeeting | 16:06 |
| opendevmeet | Meeting ended Tue May 17 16:06:10 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:06 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-05-17-15.00.html | 16:06 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-05-17-15.00.txt | 16:06 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-05-17-15.00.log.html | 16:06 |
| jrosser | i was going to take a quick look at this lxc btrfs mount thing | 16:06 |
| jrosser | dont know whats happening there yet | 16:06 |
| jrosser | and also i still have a *very* beta skyline role (pretty broken tbh) and i see they are adding new stuff all the time there (magnum and zun this week) | 16:07 |
| noonedeadpunk | so. do you want to get that role to opendev? I can push change to create a repo and fork from yours | 16:09 |
| jrosser | we can, though i think it needs really a lot of work | 16:10 |
| jrosser | the amount of memory required to build skyline-console is gigantic | 16:10 |
| jrosser | and right now the apache config is just wrong, and it needs a temporary setup with nginx until that can be understood | 16:11 |
| jrosser | spatel: are you still interested in using skyline? | 16:11 |
| spatel | Yes | 16:12 |
| spatel | currently i am running inside docker outside OSA deployment | 16:12 |
| spatel | I have also tested it to deploy it without Docker but that process is ugly so next step to try our whl package which is provided by CI-CD | 16:13 |
| noonedeadpunk | all kind of mess.... | 16:14 |
| noonedeadpunk | I'll try to hunt down skyline ppl in Berlin :D | 16:15 |
| jrosser | well, i have had it "working" with my ansible role | 16:15 |
| jrosser | all deployed from source in an LXC | 16:15 |
| jrosser | but i am not enough of a web developer to understand how to debug it | 16:16 |
| noonedeadpunk | btw it doesn't seem that dns patch worked... | 16:16 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/842102 still failing badly.... | 16:16 |
| noonedeadpunk | Maybe it's must be enabled by default now? | 16:17 |
| jrosser | oh well i don't think that patch addresses the failure we saw earlier | 16:17 |
| jrosser | i just worked on a couple of bugs today and that was one | 16:17 |
| jrosser | so something fundamental breaks even without that on all ovs jobs, which is odd | 16:17 |
| noonedeadpunk | Doesn't `dns_domain_keywords` sound related to `Unrecognized attribute(s) 'dns_domain'`? | 16:18 |
| jrosser | agreed it sounds like it | 16:18 |
| jrosser | but this patch is similarly fubar https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/827251 | 16:18 |
| jrosser | so i think that os_neutron generally is a bit broken right now | 16:19 |
| spatel | noonedeadpunk did you say that you are coming to berlin ? | 16:19 |
| noonedeadpunk | I meant - meaybe we should add dns_domain_keywords to https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/defaults/main.yml#L339 ? | 16:19 |
| noonedeadpunk | spatel: yup | 16:19 |
| spatel | boy! i will see you there :) | 16:20 |
| noonedeadpunk | sure thing :) | 16:20 |
| jrosser | noonedeadpunk: oh i see | 16:20 |
| noonedeadpunk | jrosser: or well... at least try... Not sure in that :D | 16:20 |
| jrosser | doesnt sound right, or it wouldnt be a config option | 16:20 |
| jrosser | unless there is an incorrect assumption in tempest that it is always there | 16:21 |
| noonedeadpunk | oh, wait | 16:21 |
| noonedeadpunk | it calls for test_create_update_network_dns_domain | 16:21 |
| jrosser | https://opendev.org/openstack/neutron-tempest-plugin/src/branch/master/neutron_tempest_plugin/api/test_networks.py#L98-L112 | 16:23 |
| noonedeadpunk | yeah | 16:23 |
| jrosser | that code is untouched for a very long time | 16:23 |
| noonedeadpunk | tempest code was touched ;) | 16:24 |
| noonedeadpunk | * os_tempest | 16:24 |
| noonedeadpunk | Likely we dropped some option (damiandabrowski[m]) | 16:24 |
| jrosser | is it this https://zuul.opendev.org/t/openstack/build/5b3f4309df5b41db9e7744377ccdf6fe/log/logs/etc/host/openstack_deploy/user_variables_neutron_ovs.yml.txt#8 | 16:24 |
| jrosser | well, no | 16:25 |
| noonedeadpunk | Like https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/stable/xena/templates/tempest.conf.j2#L106 | 16:25 |
| jrosser | ooooh that might do it | 16:25 |
| noonedeadpunk | question why it works forlxb but not ovs. | 16:28 |
| *** andrewbonney_ is now known as andrewbonney | 16:29 | |
| noonedeadpunk | ah, we likely don't run this test for lxb ) | 16:29 |
| damiandabrowski[m] | yeah, looks like default extension list is much longer than we had previously set: https://github.com/openstack/neutron-tempest-plugin/blob/master/zuul.d/yoga_jobs.yaml | 16:34 |
| damiandabrowski[m] | trying to find out how can we fix this `"Unrecognized attribute(s) 'dns_domain'"` error | 16:34 |
| noonedeadpunk | Already pushed https://review.opendev.org/c/openstack/openstack-ansible/+/842153 jsut in case | 16:35 |
| noonedeadpunk | And made https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/842102 dependant on it | 16:35 |
| damiandabrowski[m] | great, i was just thinking about it as a temporary solution | 16:35 |
| noonedeadpunk | gerritbot jsut crashed so... | 16:35 |
| damiandabrowski[m] | shouldn't we enable neutron's dns plugin? https://github.com/openstack/openstack-ansible-os_neutron/blob/master/defaults/main.yml#L337 | 16:37 |
| damiandabrowski[m] | ah, we've already tried it | 16:38 |
| jrosser | damiandabrowski[m]: enabling it there would mean we enable it on everyones deployment by default | 16:48 |
| jrosser | these things are usually kept minimal | 16:48 |
| jrosser | if we want to specifically test it in CI then the AIO config should be adjusted to include that | 16:48 |
| jrosser | via an override | 16:49 |
| NeilHanlon | jrosser: i will request to get it into epel for 9 | 16:49 |
| NeilHanlon | s/it/lxc3.0/ | 16:49 |
| damiandabrowski[m] | yeah I agree with You | 16:50 |
| *** dviroel_ is now known as dviroel | 16:52 | |
| jrosser | reviews here if anyone has a moment https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/842086 | 16:53 |
| damiandabrowski[m] | done | 16:58 |
| noonedeadpunk | NeilHanlon: would be great to get lxc 4.0 for el9 ;) as that's I guess what fedora has? | 17:10 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Template out zuul mirror info variables for later use by embedded ansible https://review.opendev.org/c/openstack/openstack-ansible/+/842158 | 17:14 |
| NeilHanlon | noonedeadpunk: for sure! :) I found your ticket, going to follow up on it | 17:24 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Limit cloudkitty workers https://review.opendev.org/c/openstack/openstack-ansible/+/841939 | 17:34 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Limit mistral worker https://review.opendev.org/c/openstack/openstack-ansible/+/841941 | 17:44 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: Limit cloudkitty workers https://review.opendev.org/c/openstack/openstack-ansible/+/841940 | 17:44 |
| *** melwitt_ is now known as melwitt | 18:08 | |
| NeilHanlon | jrosser, noonedeadpunk: i'm trying a build over here until I get some movement on EPEL itsellf: https://copr.fedorainfracloud.org/coprs/neil/lxc4.0/builds/ | 18:27 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Clean up NFV OpenvSwitch repo for container build https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/842086 | 18:29 |
| jrosser | NeilHanlon: nice! | 18:29 |
| *** ianw_ is now known as ianw | 19:11 | |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: [DNM] Add dns_domain to neutron_plugin_base https://review.opendev.org/c/openstack/openstack-ansible/+/842222 | 20:08 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Template out zuul mirror info variables for later use by embedded ansible https://review.opendev.org/c/openstack/openstack-ansible/+/842158 | 20:49 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Add centos-9 support https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/842236 | 20:53 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: WIP - Centos-9 Stream support https://review.opendev.org/c/openstack/openstack-ansible/+/823417 | 20:57 |
| jrosser | NeilHanlon: ^ pulled in your copr build there hopefully | 20:58 |
| jrosser | ahha "No package python3-lxc available." | 21:27 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Add centos-9 support https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/842236 | 21:30 |
| NeilHanlon | jrosser: added python3-lxc and lxcfs copied over from thm/lxc3.0 | 21:36 |
| NeilHanlon | I *might* need to modify python3-lxc to run a different python; right now it's for the system python 36 | 21:36 |
| jrosser | i guess thats 3.8 on c-9? | 21:36 |
| NeilHanlon | ah.. yep, would be | 21:37 |
| NeilHanlon | brain thought el8 for some reason | 21:37 |
| NeilHanlon | i guess if we want this for el8 as well at some point, would need to make it modular to support a different python | 21:39 |
| jrosser | no worries, it's pretty late here so i'll leave it, but if you think it's worth hitting recheck later on this just go for it https://review.opendev.org/c/openstack/openstack-ansible/+/823417 | 21:39 |
| NeilHanlon | ack, thanks! | 21:39 |
| *** dviroel is now known as dviroe|out | 22:12 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!