15:00:38 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:39 <opendevmeet> Meeting started Tue Apr 19 15:00:38 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:39 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:39 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:00:44 <noonedeadpunk> #topic rollcall
15:00:51 <spatel> \o/
15:01:05 <damiandabrowski[m]> hey!
15:01:08 <noonedeadpunk> o/
15:01:37 <mgariepy> hey ! o/
15:01:42 <jrosser> hello
15:02:24 <noonedeadpunk> #topic office hours
15:02:26 <damiandabrowski[m]> and thanks for nominating me! (i just saw the message on ML)
15:03:38 <noonedeadpunk> so. seems we have issue with ceph-ansible version... Will need to take a look at the actuall thing first time tomorrow in the morning....
15:04:08 <noonedeadpunk> but that sounds really weird, I havn't heard a thing that they're replacing cinder-volume with anything else...
15:04:19 <noonedeadpunk> *ceph-volume
15:04:51 <damiandabrowski[m]> FYI, i'm not sure if it's related, but I've hit this error when deploying AIO+ceph today https://paste.openstack.org/show/bRMsII6P2s1YyjugFuWk/
15:05:00 <damiandabrowski[m]> but haven't looked much into this yet
15:05:22 <noonedeadpunk> that's interesting actually.
15:05:42 <noonedeadpunk> I can recall we were having apt pinning for ceph somehow....
15:05:43 <jrosser> we should have apt pins to make sure that the ceph version comes from where we expect
15:06:00 <jrosser> which iirc on ubuntu should always be download.ceph.com
15:06:03 <noonedeadpunk> but I think we did that only for ceph-client role?
15:06:14 <jrosser> could be, yes
15:08:04 <jrosser> https://github.com/openstack/openstack-ansible-ceph_client/blob/master/defaults/main.yml#L43
15:08:42 <noonedeadpunk> huh and we don't really include role anywhere
15:09:01 <noonedeadpunk> unless we rely on that? https://opendev.org/openstack/openstack-ansible-ceph_client/src/branch/master/meta/main.yml#L40-L45
15:09:10 <jrosser> yes it comes from meta
15:09:11 <noonedeadpunk> but, um....
15:09:32 <noonedeadpunk> but anyway failure is during ceph-ansible runtime
15:09:49 <noonedeadpunk> (in CI at least)
15:10:25 <noonedeadpunk> Also I spent time previous week for unplanned activity of fixing run of roles in check mode
15:10:35 <noonedeadpunk> was interesting how much doable is that...
15:11:07 <noonedeadpunk> hosts and infra seems quite fine, openstack services are tricky
15:11:49 <noonedeadpunk> I also thought it would be easy to have CI job but then realized that to run in check mode it has plenty of dependencies in terms of missing services and certificates.
15:12:12 <noonedeadpunk> Still doable but appeared a bit harder then expected
15:12:33 <noonedeadpunk> Will likely continue that after more important things are done:)
15:12:52 <noonedeadpunk> BTW we kind of need to make last release of V and move it to EM
15:14:01 <noonedeadpunk> jrosser: do we need wip here? https://review.opendev.org/c/openstack/openstack-ansible-tests/+/836335
15:15:01 <jrosser> no - i think it was just to try to get things merged in order down the stable branches
15:15:22 <jrosser> this has all been a quite big mess with not really any cherry-picks that work
15:15:35 <jrosser> so i had to patch each branch pretty much individually
15:16:32 <jrosser> and really the patch we need is https://review.opendev.org/c/openstack/openstack-ansible-tests/+/837368
15:16:57 <jrosser> there is a bunch of broken things due to that ^
15:18:59 <jrosser> the gluster patches are nearly ready for proper review
15:19:14 <jrosser> i have to make a hack^W fix for rocky
15:20:39 <noonedeadpunk> But hopefully NeilHanlon would ping for some packaging changes one day?
15:20:51 <jrosser> hopefully yes
15:21:25 <noonedeadpunk> ok great. And then I can pick CentOS 9 deployment on top of gluster :)
15:22:01 <NeilHanlon> Yes, i am working on fixing up those release packages for, e.g., gluster
15:22:47 <jrosser> i just need to decide if we lineinfile/regex the repo file or vendor a copy of it and just copy: it into place
15:22:49 <noonedeadpunk> great)
15:24:45 <jrosser> do you think we should be more specific about pinning the ceph repo on ceph hosts?
15:25:16 <jrosser> if we pin the client to ceph.com but not the server, we still have potential conflicts between ubuntu / uca / ceph.com repos
15:26:15 <noonedeadpunk> jrosser: Um, can you point to that Rocky hack as I can't find it quickly :)
15:26:29 <jrosser> no i still try to write it :)
15:26:39 <noonedeadpunk> ah, ok then :)
15:27:06 <jrosser> but i believe we have to adjust mirrorlist=http://mirrorlist.centos.org/?release=$stream&arch=$basearch&repo=ResilientStorage&infra=$infra
15:27:33 <noonedeadpunk> regarding ceph pinning - that kind of make sense... Despite that should be part of ceph-ansible to be fair...
15:27:47 <jrosser> so that always release=8-stream
15:29:23 <damiandabrowski[m]> i need to spend some time trying to figure out why i get this downgrade error, then I'll have a better view on how do we want to pin ceph repo. Planning to look into this during the evening
15:29:33 <noonedeadpunk> ceph-ansible already has requirements.yml in place, so technically we can push them a change to handle pinning and add another dependency on our roles :)
15:29:41 <jrosser> NeilHanlon: do you have a bug link i can put in my patch for glusterfs?
15:31:36 <noonedeadpunk> don't think they're gonna like it though...
15:32:06 <noonedeadpunk> I kind of more and more thinking that integration of ceph-ansible should be more for CI purposes only.
15:33:04 <noonedeadpunk> while it's good to have playbooks for that and some sort of integration, I'm not really sure if anybody should do that in production :)
15:33:12 <noonedeadpunk> (at least as is)
15:34:17 <damiandabrowski[m]> I'm going to use it in production in a few weeks, i can share my thoughts then :D
15:34:43 <noonedeadpunk> the questions come when you're trying to upgrade stuff :)
15:35:34 <noonedeadpunk> as we pin specific ceph-ansible version that is tighten to ceph releases quite hardly
15:35:37 <SiavashSardari> i have it in production. (he types with shaking hands :D )
15:36:18 <noonedeadpunk> as then when you upgrade osa you should either override ceph-ansible or do upgrade ceph at same time kind of.
15:36:29 <mgariepy> i had it with osa for exactly 1 deploy then i move the ceph-ansible on the side right after
15:37:23 <SiavashSardari> we are thinking about replacing it with ceph-adm, not starting the procedure though
15:37:37 <jrosser> ceph-ansible is fine if you use it on its own
15:37:51 <jrosser> i wouldnt copy the approach from OSA CI into a production deployment
15:38:26 <noonedeadpunk> nah, we're not talking about ceph-adm, but more about at what level we should provide ceph deployment
15:38:40 <noonedeadpunk> and if we explicit enough about how to use that properly
15:41:11 <SiavashSardari> tbh I like the idea of having one repo for both but at the end of the day we are using them kinda separately and will override the role-requirement if needed
15:41:55 <damiandabrowski[m]> i just think that ceph integration is a huge advantage of openstack-ansible project(especially for a newcomer who is is comparing different deployment tools and trying to choose one)
15:42:14 <damiandabrowski[m]> so maybe we can list our main issues with it and think what we can do about them?
15:42:31 <jrosser> ceph-ansible sometimes has very specific requirements for the ansible version (though this is less important with modern ansible)
15:42:52 <jrosser> and versions of ceph-ansible only support very specific releases of ceph
15:43:19 <mgariepy> specific major rel
15:43:36 <jrosser> so you ended up in very difficult places at upgrade time when the openstack, ansible and ceph (and maybe also OS) versions all had to change simultaneously
15:43:42 <jrosser> and thats a total nightmare for upgrades
15:44:37 <damiandabrowski[m]> but You don't have to run `ceph-install.yml` in the middle of openstack upgrade, right? :D
15:45:00 <noonedeadpunk> no, if you're not running /run-upgrade.sh :p
15:45:11 <noonedeadpunk> (which what newcommers will do)
15:45:27 <jrosser> and we test exactly "zero" of that approach :)
15:45:48 <noonedeadpunk> and you will get newer clients anyway unless explicitly set in user_variables
15:46:37 <noonedeadpunk> I'm not talking about dropping support, but likely more being more explicit in documentation about possible caveats by doing that
15:46:44 <damiandabrowski[m]> so maybe what we need to do is to show a warning message when running run-upgrade.sh with ceph enabled?
15:46:50 <damiandabrowski[m]> ah yeah, i was just going to say that
15:47:00 <damiandabrowski[m]> that another option is put some information in docs
15:47:05 <jrosser> well - back to where we started there is really only benefit in pinning to ceph.com
15:47:36 <jrosser> the UCA packages didnt used to have debug symbols and that was a problem for us when things were broken
15:47:38 <damiandabrowski[m]> saying that OSA upgrades may be tricky when it manages ceph as well
15:48:38 <noonedeadpunk> I can recall some early on bionic distro provided ceph version was treated as newer one comparing to one that was coming from ceph.com
15:48:50 <jrosser> yes, i remember having to fix that
15:49:01 <noonedeadpunk> I guess that's when we added pinning :)
15:49:16 <jrosser> i think it was the local patch that ubuntu applied made the version number sematically greater
15:49:26 <noonedeadpunk> yup
15:49:54 <noonedeadpunk> while in fact it wasn't and dependencies were broken
15:50:26 <noonedeadpunk> anyway - let's see what the current issues are.
15:51:27 <jrosser> damiandabrowski[m]: you might want to look at this as it addresses similar errors to those you were seeing https://github.com/openstack/openstack-ansible-os_cinder/blob/9f2bf29db8ef921cfad7857dcb7652436d0d887b/tasks/main.yml#L183-L209
15:51:35 <noonedeadpunk> ofc we can add pinning in playbook alone
15:51:56 <damiandabrowski[m]> thanks, I'll have a look
15:52:17 <jrosser> i had to split the list of packages into those to install before, and after the ceph_client role is run
15:53:54 <damiandabrowski[m]> btw. we already managed to merge few of my tempest patches, but many of them still remain unreviewed.
15:53:59 <damiandabrowski[m]> it would be awesome to close this topic soon ;) https://review.opendev.org/q/topic:tempest-damian-2021-12
15:55:36 <mgariepy> i'll take a look
15:56:57 <noonedeadpunk> damiandabrowski[m]: aren't most of them need rebase?
15:57:36 <noonedeadpunk> see tons of `Indirect ancestor` in gerrit
15:57:52 <noonedeadpunk> likely they won't merge cleanly but not sure
15:58:32 <damiandabrowski[m]> that's right, but the plan was to get initial reviews ->  apply Your suggestions -> rebase -> merge
15:58:49 <damiandabrowski[m]> by doing it like this, we'll avoid running too much unnecessary zuul jobs
15:58:53 <damiandabrowski[m]> but I can rebase them now if You want :D
16:00:09 <damiandabrowski[m]> but main drawback of this plan is that You'll need to leave Your reviews twice on these patches(before and after rebase)
16:01:14 <noonedeadpunk> well, rebase in gerrit doesn't remove Code-Review label
16:01:20 <noonedeadpunk> it would remove jsut Workflow
16:01:33 <noonedeadpunk> but ok, indeed:)
16:01:34 <damiandabrowski[m]> ahhh okay
16:01:40 <noonedeadpunk> #endmeeting