16:00:39 <noonedeadpunk> #startmeeting openstack_ansible_meeting
16:00:40 <openstack> Meeting started Tue Feb 23 16:00:39 2021 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:43 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:00:48 <noonedeadpunk> o/
16:00:53 <gshippey> o/
16:01:16 <noonedeadpunk> I as usual haven't prepared any agenda for the meeting... Should do better in future
16:01:27 <noonedeadpunk> checking for new bug reports
16:02:43 <gshippey> @jrosser (if he is about) how is https://etherpad.opendev.org/p/osa-ci-failures going?
16:02:48 <noonedeadpunk> #topic bug triage
16:02:49 <noonedeadpunk> https://bugs.launchpad.net/openstack-ansible/+bug/1915618
16:02:51 <openstack> Launchpad bug 1915618 in openstack-ansible "gnocchi file missing" [Undecided,New]
16:03:05 <jrosser> o/ hello
16:03:26 <noonedeadpunk> I finally need to make fresh deployemnty of all telemetry and check what;'s going on there
16:03:42 <noonedeadpunk> as I saw weird stuff in logs in CI
16:03:57 <noonedeadpunk> but no idea what the result of these
16:04:13 <jrosser> i did more work on the repo server / u-c stuff https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/774517
16:04:28 <jrosser> redid that pretty totally to make it take u-c files from the deploy host
16:05:03 <jrosser> not passing a test job though https://review.opendev.org/c/openstack/openstack-ansible/+/774523
16:05:48 <jrosser> this is ready for one of the ceph repo CI fixes https://review.opendev.org/c/openstack/openstack-ansible/+/775695
16:06:32 <noonedeadpunk> 774523 seems more like passing then not
16:06:43 <noonedeadpunk> except distro installs
16:06:50 <jrosser> yeah its mostly there, havent looked at why it's broken
16:07:23 <jrosser> could be a corner case there for utility and things where we don't have a repo server for distro installs?
16:07:36 <jrosser> utility/repo etc
16:08:07 <jrosser> this is also ready https://review.opendev.org/c/openstack/openstack-ansible/+/776464
16:08:33 <noonedeadpunk> well, it fails on tempest
16:08:44 <noonedeadpunk> https://c6ba2e2bae7e431e3d06-3245c124accefaca4c820513e2930ad5.ssl.cf5.rackcdn.com/774523/7/check/openstack-ansible-deploy-aio_distro_metal-ubuntu-focal/f1e316a/logs/ara-report/results/1259.html
16:08:46 <jrosser> and maybe most importantly the mariadb 10.5.9 bump needs attention
16:09:43 <noonedeadpunk> 776464 is neat indeed
16:10:37 <noonedeadpunk> regarding mariadb, dunno. that feels like a bug for galera
16:10:54 <noonedeadpunk> and probasbly we should report it?
16:11:00 <jrosser> looked like the upgrade jobs fail?
16:11:04 * jrosser looks
16:11:39 <noonedeadpunk> it's failed on users creation...
16:12:08 <noonedeadpunk> on 10.5.9
16:12:45 <noonedeadpunk> and what if rebase it on top of the https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/775893
16:12:46 <jrosser> oh no https://zuul.opendev.org/t/openstack/build/0af1b2a37646470dbae747523eecffc3/log/job-output.txt#16473-16490
16:12:57 <jrosser> yes fails in a similar way
16:13:20 <jrosser> yeah, we should at least try that rebase as they did say we should not mess with the root user
16:14:11 <noonedeadpunk> well, montywi said that to me a year ago I guess when there was a mess with 10.4.13 iirc
16:14:28 <noonedeadpunk> so worth trying I guess
16:15:07 <noonedeadpunk> hm
16:15:15 <noonedeadpunk> https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/777060 failed on uypgrade as well
16:15:47 <noonedeadpunk> have we merged smth breaking upgrade jobs?
16:15:59 <noonedeadpunk> in terms of the galera/utility?
16:17:21 <noonedeadpunk> ah https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/777060/3..4/defaults/main.yml
16:17:34 <noonedeadpunk> that explains it :p
16:17:46 <jrosser> doh
16:17:52 <jrosser> curse of git add -A
16:18:08 <noonedeadpunk> can't agree more :)
16:18:33 <noonedeadpunk> #topic office hours
16:18:42 <openstackgerrit> Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/777060
16:19:01 <jrosser> so i did a ton of work on lxc_hosts
16:19:02 <gshippey> in regards to that gnocchi issue https://bugs.launchpad.net/kolla-ansible/+bug/1780245 https://opendev.org/openstack/openstack-ansible-os_gnocchi/src/branch/master/tasks/main.yml#L146
16:19:04 <openstack> Launchpad bug 1780245 in kolla-ansible train "gnocchi-metricd container missconfigured on second and third nodes" [Medium,Triaged]
16:19:23 <openstackgerrit> Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Bump MariaDB version to 10.5.9  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/777088
16:19:48 <noonedeadpunk> we also should do smth with octavia
16:19:57 <jrosser> oh, andrewbonney, you around?
16:19:58 <noonedeadpunk> since it's broken for a while and we dunno why...
16:20:15 <andrewbonney> yes. I'm still looking at it as I can't replicate the centos failure
16:20:27 <juanoterocas> Hi is there any explicit tutorial on how to setup networking before boostraping?
16:20:28 <andrewbonney> debian might be https://bugzilla.redhat.com/show_bug.cgi?id=1832710 though as it uses a very old libvirt
16:20:29 <openstack> bugzilla.redhat.com bug 1832710 in libvirt "vhost-user multiple queues hot plug fail:error: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter type for 'queues', expected: integer" [High,Closed: errata] - Assigned to pkrempa
16:20:41 <openstackgerrit> Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Bump MariaDB version to 10.5.9  https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/777088
16:21:06 <noonedeadpunk> I thought we're installing libvirt from debian-updates?
16:21:18 <noonedeadpunk> or it's jsut qemu...
16:21:23 <noonedeadpunk> as nova complains otherwise
16:21:36 <andrewbonney> qemu comes from debian-backports, and there was an update around when it started failing for a CVE
16:21:44 <jrosser> juanoterocas: we're just in our weekly meeting right now, but you can use whichever method you like, nothing is really fixed. There are examples in the etc/ directory of openstack-ansible.
16:22:01 <juanoterocas> ok thx
16:22:02 <andrewbonney> nova on W looks like it will need libvirt >= 6 anyway so we'll need to find it from somewhere
16:22:34 <noonedeadpunk> doh...
16:22:52 <noonedeadpunk> Eventually I believe debian folks will do smth regarding that
16:24:19 <jrosser> wonder if there is a W variant for here https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/vars/debian.yml#L88-L94
16:25:05 <noonedeadpunk> not yet at least
16:25:36 <jrosser> zigo: can you give us any pointers if we should be able to run W on buster re. libvirt version?
16:25:39 <noonedeadpunk> btw awesome job with centos stream
16:25:47 <jrosser> yeah i am pleased with that
16:26:01 <jrosser> kind of related to the lxc_hosts stuff actually
16:26:13 <noonedeadpunk> Would be really great to move all images to be built from chroot
16:26:20 <jrosser> we need to decide if we bump the existing centos to 8.3 and backport to V
16:26:31 <jrosser> becasue that determines what patches we merge and in what order
16:26:57 <jrosser> moving to chroot would be a big change and then backports are not possible any more
16:27:13 <noonedeadpunk> yeah, agree
16:27:22 <noonedeadpunk> you meant to make use of stream on V?
16:27:37 <jrosser> oh no, its why i did this https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/775778
16:27:51 <jrosser> so that we have the opportunity to fix up the current method to get 8.3 and backport that
16:28:03 <jrosser> then tear it all up straight away after that on master to use chroots
16:28:33 <noonedeadpunk> tbh dunno
16:29:12 <noonedeadpunk> eventually on deployemnts we do dnf update before packing them, which will update stuff to 8.3 anyway
16:29:13 <jrosser> i guess we don't run many CI jobs really on victoria so the failure i was trying to work around there is kind of minimal
16:29:32 <noonedeadpunk> yeah. and for CI it's not so critical as on master
16:29:47 <jrosser> the issue was that the 8.1 container gets the infra repo setup, but dnf update blows that all away in an unfortunate way during 8.1>8.3
16:29:59 <noonedeadpunk> I was thinking more if we should give 8.3 -> Stream upgrade path
16:30:00 <jrosser> but really corner case
16:30:11 <noonedeadpunk> and might be that do this on V is relevant
16:31:46 <jrosser> anyway, just wanted to point that the situation with lxc_hosts is kind of complicated depending on what we choose to do
16:32:02 <noonedeadpunk> as otherwise we will need to keep 8.3 during W and drop it afterwards. And I have no intention to carry such super simmilar distros
16:32:46 <jrosser> i think if we were to merge the changes for 8.3 containers to master then the opportunity to backport is there should we need it
16:32:58 <jrosser> even if then we pretty soon change the code to chroots on master
16:34:28 <noonedeadpunk> ok, fair
16:34:50 <noonedeadpunk> but it's also if we're going to backport chroot and stream to V
16:35:03 <noonedeadpunk> because then uit's just +1 patch to backport :)
16:35:55 <jrosser> the whole upgrade situation is tricky
16:36:08 <jrosser> i don't know how long the infra people will keep a non stream image
16:36:25 <noonedeadpunk> we agreed till W at least iirc
16:36:48 <jrosser> right, but that does mean that our U/V centos jobs will stop working anyway at that point potentially
16:37:11 <jrosser> which will make testing upgrades hard
16:37:47 <noonedeadpunk> well, we kind of never tested os upgrades? neither performed them?
16:38:00 <noonedeadpunk> but yeah, I got the point
16:38:25 <jrosser> perhaps not to worry too much about it
16:38:51 <jrosser> swapping centos-8 for centos-8 stream on master should be a priority
16:39:00 <noonedeadpunk> yes, agree
16:39:05 <noonedeadpunk> let's leave V as is
16:39:09 <noonedeadpunk> in terms of centos
16:39:15 <noonedeadpunk> at least for now
16:39:23 <noonedeadpunk> maybe we will reconsider later...
16:39:46 <jrosser> this ansible_facts stuff is massive patches too
16:39:49 <jrosser> thanks for looking at them
16:40:26 <noonedeadpunk> I really want to start pki stuff, but I have to research trove due to inner prio right now
16:40:55 <noonedeadpunk> But I still aio to grab both...
16:41:06 <jrosser> i may have someone who can help with the pki stuff
16:42:06 <noonedeadpunk> btw regarding trove. I know that it uses rabbit to talk to agents (which are actually vms). And I'm not sure how good it will be to use general rabbit. And thinking if it makes sense to deploy another one on trove containers
16:42:19 <jrosser> that would be ++
16:42:46 <noonedeadpunk> gotcha
16:42:55 <jrosser> the rabbit role is quite well formed so that should be ok i expect to reuse it
16:43:37 <noonedeadpunk> yeah, I also think it should be fine. Just pass another secrets as vars, but shouldn't be too hard
16:43:58 <jrosser> oh also andrewbonney has started to look at fixing the neutron role
16:44:15 <noonedeadpunk> yeah, I saw the patch
16:44:33 <noonedeadpunk> really didn't have time to add integrated roles :(
16:44:58 <noonedeadpunk> oh, btw https://review.opendev.org/c/openstack/openstack-ansible/+/776581
16:45:10 <noonedeadpunk> bumped ansible and collections
16:45:42 <noonedeadpunk> they moved posix out of general collection
16:46:30 <noonedeadpunk> btw, I was thinking about creating openstack-ansible-collections to place there our modules for git, and, maybe, config-template. But in case of config -template that damn bad repo name
16:46:44 <noonedeadpunk> since it's used by tripleo and etc
16:47:36 <jrosser> config_template is so cool it almost deserves it's own
16:47:52 <noonedeadpunk> well, agree...
16:48:07 <noonedeadpunk> but I think we can't jsut convert that repo into collection?
16:48:24 <jrosser> tbh i'm not sure, but basically 'no'
16:48:32 <jrosser> theres other stuff to do i think
16:48:51 <jrosser> you need metadata and to build locally some package to upload to galaxy
16:49:10 <noonedeadpunk> I just really use config_template internally more and more and get fed of the way I need to install it..
16:49:34 <noonedeadpunk> well, we could convert it, but I think we probably should not...
16:49:43 <jrosser> nsible-galaxy collection init my_namespace.my_collection
16:50:52 <jrosser> for internal stuff here we have a role clone script that populates my-project/roles
16:51:04 <jrosser> and also puts other things like config-template in my-project/repos
16:51:12 <jrosser> which is also an ugly hack
16:51:48 <noonedeadpunk> yeah. and when you have pretty much independant repos that have just requirements.yml for external dependencies...
16:53:16 <noonedeadpunk> ok, well, need to think about best way to do that. maybe ask cloudnull opinion as well :)
16:58:56 <noonedeadpunk> #endmeeting