16:03:59 <tiffanie> #startmeeting openstack_ansible_meeting
16:04:00 <openstack> Meeting started Tue Sep  3 16:03:59 2019 UTC and is due to finish in 60 minutes.  The chair is tiffanie. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:04:01 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:04:03 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:04:07 <mnaser> Hi all
16:04:08 <noonedeadpunk> o/
16:04:11 <tiffanie> #topic office hours
16:05:12 <mnaser> How's everyone
16:05:26 <evrardjp> so, I haven't got the chance to continue my work on cleaning up the 42.3, but it's still pending
16:05:45 <evrardjp> I see noonedeadpunk guilhermesp mnaser jrosser helped, so thanks!
16:06:03 <evrardjp> I also still have the job state matrix to finish
16:06:13 <evrardjp> but that will have to wait a little
16:06:45 <guilhermesp> o/
16:06:46 <evrardjp> that's all I had for today.
16:07:13 <mnaser> i wanted to bring up killin old branches
16:07:16 <evrardjp> mnaser: what's up on your side? ptl letter ready?
16:07:22 <mnaser> its already sent out hehe
16:07:22 <evrardjp> oh that's a good topic
16:07:22 <noonedeadpunk> So I have linters check done for integrated tests https://review.opendev.org/#/c/679101/ It was failing only linters check before issue with gastes
16:07:31 <evrardjp> mnaser: great!
16:07:49 <evrardjp> oh yeah I also wanted to discuss killing magnum queens
16:07:56 <evrardjp> but go ahead mnaser :)
16:08:16 <mnaser> ok so
16:08:16 <evrardjp> (FYI kolla and triple O have started killing branches, we can do too, there is precedent)
16:08:39 <mnaser> right now we have
16:08:43 <mnaser> stable/ocata-stable/stein
16:08:46 <mnaser> thats a lot of branches
16:08:58 <mnaser> i think the only reason we cared about ocata was rax folks mentioned it
16:09:05 <mnaser> but no one is really stepping up at this point
16:09:34 <evrardjp> you want me to propose to tag everything and mark everything EOL for Ocata?
16:09:35 <noonedeadpunk> I'm wondering if we still need to carry pike as well...
16:09:47 <evrardjp> I would say Pike and Ocata are in the same boat
16:09:51 <evrardjp> I would wait for queens
16:09:56 <mnaser> yeah i was gonna go release by release
16:09:58 <evrardjp> it's not even EM yet
16:10:06 <jamesdenton> Is there harm in carrying the branch?
16:10:20 <mnaser> ci is broken for it
16:10:29 <mnaser> so really unless someone is gonna push a patch and start fixing that
16:10:37 <mnaser> and even then tbh i have no interest in reviewing patches to fix stable/ocata.
16:10:40 <evrardjp> it's kinda setting a wrong expectation
16:11:06 <evrardjp> because it's still under extended "maintenance" where we actually don't do any maintenance
16:11:14 <jamesdenton> Yeah Ocata, IDGAF. But Pike might be important. I'll have to ask
16:11:26 <evrardjp> :D
16:11:42 <mnaser> if someone is out here pushing patches and making it happy, id be happy
16:11:50 <mnaser> but so far, i havent seen anything
16:11:59 <jamesdenton> how frequently does CI run for those old things? Only on patches?
16:12:14 <evrardjp> I can start by killing ocata, and wait for your answer. In the meantime, we can disable functional testing in Pike if it doesn't pass anymore (but in that case, I would prefer going for EOL sooner rather than later)
16:12:20 <mnaser> https://review.opendev.org/#/q/projects:openstack/openstack-ansible+is:open+branch:stable/pike
16:12:22 <mnaser> and there is a lot of them
16:12:35 <mnaser> ok by a lot its 27 patches
16:12:39 <evrardjp> For killing the branch I will close all of those
16:12:44 <mnaser> but they demonstrate that everything is clearly broken
16:12:52 <jamesdenton> fair enough
16:13:01 <jamesdenton> I am asking now and will get back to ya
16:13:02 <mnaser> this is why i wanted to go release by release
16:13:09 <mnaser> seems like we have consensus that ocata can go
16:13:20 <noonedeadpunk> mnaser: humm, I'd say mainly suse42 is broken there
16:13:23 <jamesdenton> yeah, sorry to interject
16:13:37 <evrardjp> jamesdenton: you're right to do so
16:13:53 <evrardjp> noonedeadpunk: which links to my first topic... I am removing it :p
16:13:54 <noonedeadpunk> If we drop suse for pike, I'd say it will be passing ci
16:14:05 <mnaser> suse42 broken in pike?
16:14:06 <evrardjp> noonedeadpunk: we are passing CI in the integrated in pike that's for sure
16:14:15 <mnaser> or ocata?
16:14:16 <evrardjp> for ubuntu*
16:14:20 <noonedeadpunk> mnaser: pike
16:14:26 <mnaser> ah ok
16:14:33 <evrardjp> but it doesn't mean that some projects are fine
16:14:43 <noonedeadpunk> yeah, agree
16:14:47 <noonedeadpunk> like magnum:)
16:15:02 <evrardjp> it's not the only one, I don't want to do finger pointing in pike right now
16:15:06 <evrardjp> but the state is bad
16:15:43 <evrardjp> let's work on the cleanup of 42.3 for pike, and recheck all of those subporjects
16:15:46 <evrardjp> projects*
16:15:50 <noonedeadpunk> ++
16:16:33 <evrardjp> if they don't work, we'll reduce the testing there and write that down somewhere in an etherpad. Then we send a summary message on the ML saying "If you care about Pike, here is what should be done", else we'll just drop it in x weeks
16:16:47 <evrardjp> is that fine for everyone?
16:16:57 <jamesdenton> Alright - so I think our official line is "feel free to purge Pike whenever"
16:17:00 <noonedeadpunk> I think it fair
16:17:04 <logan-> o/
16:17:22 <evrardjp> jamesdenton: that or be ready to fix it :)
16:17:22 <jrosser> o/
16:17:29 <jamesdenton> :D
16:17:36 * jamesdenton looks around - not it
16:17:42 <evrardjp> hahaha
16:17:46 <logan-> yeah ocata and pike seem reasonable to drop if no one is maintaining them or using them
16:18:28 <evrardjp> when tagging EOL ppl will review if there are pending patches, so it makes sense to clean the complete state before asking the EOL tag
16:19:02 <jamesdenton> We simply need the 16.x tags
16:19:15 <mnaser> yeah those arent going anywhere
16:19:19 <jamesdenton> k
16:19:21 <mnaser> jamesdenton: we = platform9?
16:19:25 <jamesdenton> :|
16:19:28 <jamesdenton> we = RAX
16:19:31 <mnaser> oh
16:19:32 <jamesdenton> long story
16:19:33 <mnaser> lol
16:19:34 <mnaser> okay
16:19:40 <mnaser> well good to know
16:19:47 <mnaser> didn't seem to add up :p
16:19:52 <jamesdenton> hehe
16:20:03 <mnaser> alright
16:20:13 <mnaser> something that concerns me is that sustainability of the project
16:20:38 <mnaser> there's a lot of users but not a lot of poeple pushing patches and reviewing etc
16:21:37 <mnaser> Do people not want to use Ansible anymore? Do they prefer other tooling (aka containers?) Over this?
16:21:38 <noonedeadpunk> also I'd say we have pretty big docs debt right now...
16:21:52 <noonedeadpunk> it's in addition
16:21:59 <noonedeadpunk> not the root cause:)
16:22:24 <mnaser> I've spoken to some million core scale cloud users and they didn't seem that interested by trying to use it more. Containers made a lot of sense to them.
16:23:00 <jamesdenton> These were former OSA users?
16:23:08 <jamesdenton> or operators
16:23:09 <mnaser> Other distros have done a lot of platform evolution like moving the control plane to containers like tripleo
16:23:19 <mnaser> No just operators looking to change tooling for deployment
16:24:22 <mnaser> I just don't want us to be stuck in where we started forever until the project is irrelevant.. or we can just decide to stick to it till the very end
16:24:53 <mnaser> I'd love to hear maybe other operators like jrosser or logan- chime in too from a deployers pov
16:25:15 <evrardjp> I think the simplest tech make it very appealing.
16:25:24 <noonedeadpunk> I'd say that it's pretty controversial  and really depends... And this might be really good thing to ask on some opendev summit...
16:25:43 <evrardjp> (And I think there are other tools doing other way of deploying, like kolla or osh... it's just different dna)
16:26:19 <jrosser> OSA does what I need it to do
16:26:21 <evrardjp> mnaser: shouldn't the foundation poll data help here?
16:26:29 <noonedeadpunk> I kinda remember really big interest to OSA in Berin
16:26:56 <evrardjp> I think OSA is boring for many, but it's good boring :)
16:26:59 <logan-> i'd say the same as jrosser
16:27:01 <jrosser> i'm not sure what 'irellevant' means in this context, just cause we don't use a currently fashoinable texh isnt a bad thing
16:27:09 <mnaser> Right but is this a sustainable DNA?  I'm worried that with time as technology evolves it won't make sense and be a crutch
16:27:21 <mnaser> Say like: I want to do ci/cd to deliver my cloud
16:27:49 <mnaser> Raise your hand if you'd trust running OSA every X minutes to ensure state. It's just so many moving parts
16:27:51 <evrardjp> then git clone logan-'s work
16:28:42 <mnaser> I get it. It's different DNA. I agree to an extent. This isn't a comfortable conversation
16:29:00 <miloa> Not all organisations want/could adopt the new tech as "docker" containers. And LXC is easier for some operators than docker for example, so it make OSA rlevant :)
16:29:06 <mnaser> But as a project do we want to "ride it out till the end" or do we want to start looking at ways to improve things.
16:29:16 <evrardjp> I think openstack is hard, but if you're saying that you don't trust to ci/cd osa, maybe that's what we need to fix.
16:29:57 <guilhermesp> I agree about following the tendency of the IT marked for example: k8s is becoming a preferable way for lots of dev/ops and I've been hearing this since ocata release... but tbh, we need more concrete data to analyze OSA role in the openstack ecosystem
16:30:44 <noonedeadpunk> For example I know companies, that still do not trust docker, that's was the root for choosing osa
16:30:45 <guilhermesp> market*
16:31:06 <mnaser> I agree to an extent. Not trusting docker and containers was a thing many years ago
16:31:09 <evrardjp> also I don't think it's a good idea to change the DNA of OSA -- if we want to do something else, maybe we should add efforts to the other projects already in place, instead of us changing, except if we do something not existing
16:31:35 <mnaser> That's fair enough
16:31:52 <jamesdenton> In the survey, Is OSA called out specifically, or does it fall under the generic "Ansible" umbrella? I also found past surveys not to reflect OSA usage accurately - especially when managed hosting providers have hundreds of clouds under management but count as "One Deployment" for survey purposes.
16:31:53 <evrardjp> mnaser: I kinda like the idea of docker for some part of the infra -- for example, tempest -- why do we need to care
16:31:53 <guilhermesp> yes indeed
16:32:19 <evrardjp> jamesdenton: I think I fixed that
16:32:27 <evrardjp> but that might have changed
16:32:32 <jamesdenton> Great, evrardjp! Been a while
16:32:39 <evrardjp> we first call out the techs involved (so Ansible)
16:32:42 <evrardjp> then the projects
16:32:50 <evrardjp> (OSA/Kolla/whatever)
16:32:59 <evrardjp> tbc
16:33:22 <evrardjp> To summarize I understand the concern of mnaser. We are a very small team
16:33:37 <mnaser> I think that the decreased amount of contributions and traction means we're sitting in a very niche thing
16:33:41 <jrosser> i think that is the issue more than any particular tech choice
16:33:48 <evrardjp> it seems by being working and boring, we don't convert new ppl to be contributors
16:34:15 <evrardjp> jrosser: go on? :D
16:34:23 <mnaser> It works for us, the group of people who are running and maintaining it, but we can't grab new users cause they have many other options that they'd prefer which work for them
16:34:31 <mnaser> Which is slowly putting us in a corner
16:34:34 <jrosser> evrardjp: the active group having shrunk so much
16:35:30 <noonedeadpunk> probably they don't what to cotnribute much, because we have really big codebase and it needs more time to cover it? Moreover, if everything is working as expected there might bew no need for contributions
16:36:01 <guilhermesp> that's exactly the case of the lab I was working on in the past noonedeadpunk
16:36:15 <mnaser> But there is a big need. Our upgrades to train don't work and that has to be fixed. There is a lot of technical debt (uwsgi work for example), there's many tiny little quirks that come up
16:36:46 <mnaser> Heck we don't even know if our Stein upgrade scripts work
16:36:54 <mnaser> Jesse used to be the one who took care of all of that
16:37:02 <jrosser> it's a problem that the rax folks put a disproportionate amount of effort in
16:37:08 <noonedeadpunk> btw, can you remind me what's wrong with upgrade to train? Since we didn't backport standalone placement iirc...
16:37:12 <jrosser> and it's hard for the rest of us to fill that space
16:37:37 <noonedeadpunk> disregard, just will take a look
16:37:40 <BjoernT> Seems like https://review.opendev.org/#/c/672078 was back-ported in the stable stein branch that breaks db online migration. I had to manually move all volumes so new service_uuids as the cinder-manage  volume update_host does not update the service uuids
16:37:41 <mnaser> Train upgrade is because you need to have placement deployed externally so all the upgrade code that exports the placement db and imports it into new placement doesn't exist
16:37:59 <mnaser> Because it uses a seperate db
16:38:08 <mnaser> And by no means is that an easy upgrade step either
16:38:33 <openstackgerrit> Vadim Kuznetsov proposed openstack/openstack-ansible-os_octavia master: Octavia certificate distribution  https://review.opendev.org/657662
16:38:34 <mnaser> jrosser: true but I feel like that goes to contribute to the idea that were in this little niche that served a certain company wel
16:38:41 <noonedeadpunk> I thought that built-in placement still exist for stein... Ok...
16:39:01 <mnaser> Now that they're gone, there's a few other users but for the main part it doesn't seem like where people want to go
16:39:10 <mnaser> noonedeadpunk: it is in Stein but in train it's gone
16:39:13 <mnaser> Aka master
16:39:22 <mnaser> So Stein to master/train is broken
16:39:36 <mnaser> And I don't know if rocky to Stein is tested much..
16:39:39 <noonedeadpunk> ah, yep, that's absolutelly true
16:40:11 <evrardjp> looking at the stats of review diversity on stein for our deployment project, we don't have to be ashamed
16:40:29 <mnaser> Review diversity is easy
16:40:31 <mnaser> It's the commits
16:40:45 <jrosser> i would like there to be a more obvious todo list
16:40:48 <mnaser> And the sustainability
16:41:24 <jrosser> PTG doesnt really address having a living backlog of stuff that needs doing, or allow prioritisation
16:41:38 <jrosser> so for example, if i hire someone to work on OSA full time - what do they do?
16:42:16 <noonedeadpunk> centos-8? :p
16:42:23 <mnaser> I think we can totally start something like that.. but I also don't want us to sit and spend a whole document on backlog
16:42:27 <noonedeadpunk> yeah, and python3
16:42:29 <mnaser> Only to kinda have it sit there
16:42:31 <jrosser> btw i don't expect an answer - just i think we can do better at that
16:42:37 <evrardjp> jrosser: there are so many things to make it sustainable: like having multi node jobs and upgrade jobs
16:42:44 <evrardjp> that's the thing you care about
16:42:45 <mnaser> evrardjp: like the lists we have in the tc
16:43:09 <mnaser> It's a list that doesn't get used.. I don't think making a work list will make people come and pick it up imho
16:43:31 <mnaser> OSA isn't cool enough for people to contribute to it on their free time. The only people contributing are users
16:43:35 <noonedeadpunk> btw, we have really full bugtracker and I'm not sure if it's being reviewed
16:43:54 <evrardjp> that's fair but here it's more on the battlefield, so I would expect that ppl would have an incentive on making things work upstream before having to do the battle testing in their environment :)
16:43:55 <noonedeadpunk> Since I don't feel like some work is happening there
16:44:44 <evrardjp> It used to be a very active thing, but as soon as you don't answer to bugs, the ppl start to disappear indeed
16:45:18 <noonedeadpunk> yeah, that what I did some time at horizon
16:45:18 <miloa> evradjp: +1
16:45:41 <noonedeadpunk> I just opened bug tracker and decided even not to fill another bug there
16:46:00 <guilhermesp> yeah that's true... maybe we need to plan another bug track day :P
16:46:12 <jrosser> these things have to be continuous though
16:46:15 <evrardjp> We might want to clean up stuff there, and return to the regular bug triage + bug squash day
16:46:16 <jrosser> like review
16:46:23 <mnaser> Right but its just us that go and fix bugs with no one stepping up
16:46:25 <noonedeadpunk> I'd say it stopped being active right after we stopped doing bug triage
16:46:41 <mnaser> Anyone remember what bug triage was like? As someone doing meetings it was just killing the whole thing
16:46:48 <mnaser> Struggled to get an answer from anyone in the meeting...
16:47:00 <evrardjp> it always was like that
16:47:04 <mnaser> And to be honest, there's plenty of stuff to fix that we need.
16:47:11 <evrardjp> from day 0 I joined the project
16:47:37 <mnaser> I'm all for.helping others but do we really have time to sit and fix reported bugs by people when we struggle to get stuff that we know is fundamental for the project to work
16:47:54 <mnaser> Aka fixing some bug vs fixing the ticking placement time bomb for example
16:47:59 <evrardjp> yeah we got an interrupt based way of working that's on IRC. That works but that's not perfect for an outsider. But I am not sure ppl fixing bugs will bring new contributors here. But neither is a new tech
16:48:30 <mnaser> It would bring new users, imho. But that's just an opinion. I could be completely wrong
16:48:35 <noonedeadpunk> I'd say that at least we need to look through them and prioritise
16:48:48 <noonedeadpunk> Since there  might be more bomb than placement one
16:48:59 <mnaser> That's pretty valid tio
16:49:01 <mnaser> Too
16:49:37 <jrosser> personally i dont think things are as bad as they seem
16:49:47 <evrardjp> I agree with jrosser
16:49:51 <jrosser> a lot of other openstack components are in worse way
16:49:59 <jrosser> in terms of review, irc activity and so on
16:50:03 <jrosser> we do quite well imho
16:50:22 <noonedeadpunk> Almost all of opensatck projects have problems with new contributors now afaik
16:50:28 <evrardjp> looking at heat and glance, which are core parts of openstack IMO, we are not that bad :)
16:50:47 <noonedeadpunk> yeah, agree with jrosser
16:50:55 <evrardjp> which is why I think we should focus on a few things, making sure they are working fine
16:50:57 <jrosser> this contains wise words https://docs.openstack.org/openstack-ansible/latest/contributor/core-reviewers.html
16:51:14 <evrardjp> jrosser: I know, right? :p
16:51:39 <mnaser> In full transparency, personally I think an architecture running OpenStack on top of kuberenetes opens up the door for a lot of things (nest monitoring powered by daemonsets), simplified ci and cd, simplified network architecture and overall mgmt
16:52:02 <evrardjp> it's not that green in OSH world, I can tell you :)
16:52:18 <evrardjp> we are by far in a better shape, due to our simplicity
16:52:32 <mnaser> While I think we'll continue in OSA for a while, I think eventually that's where we will go.. and I just worry that if we do that then the project gets a tough hit..
16:53:05 <mnaser> I don't think we will deploy on OSA forever. Kuberenetes is def the future because of the scale of deployments and advantages it brings operationally
16:53:38 <mnaser> And I personally feel that if that's how I think, maybe there might be others that think of kuberenetes. And rather than perhaps losing more, we can adopt a new direction. Or we dont
16:54:41 <noonedeadpunk> maybe we can add things like an extra with nspawn?
16:54:45 <evrardjp> what's missing in the k8s openstack world is operators to deal with things
16:54:52 <noonedeadpunk> * like nspawn
16:55:22 <jrosser> evrardjp: i agree completely - theres no-one here who could even start with that
16:55:40 <mnaser> Brb driving (I already got s ticket for using my phone a while back hah)
16:56:30 <evrardjp> there are ppl who technically can, but the question is "why would they?" if there is something like OSA already working :p
16:57:26 <miloa> Diversity is important :) for any ecosystems LXC/Docker/kubernetes. Let users the choice :) Because their environnement and background are differents.
16:57:52 <evrardjp> I think as an action point, it would be nice to know why ppl love OSA, focus on this to make it stronger.
16:57:55 <miloa> Their is already many deployement tools using Kubernetes.
16:58:06 <evrardjp> miloa: totally agreed :)
16:58:13 <jrosser> i chose it becasue there was no magical element that would jump up and fubar my stuff randomly
16:58:25 <jrosser> everything is understandable and debuggable
16:58:32 <noonedeadpunk> +1
16:58:38 <evrardjp> ok, so, is there still black magic in OSA, like we used to have? :p
16:58:46 <jamesdenton> If anyone is gonna FUBAR my environment, it's gonna be me!
16:58:52 <jrosser> too right :)
16:58:54 <evrardjp> jamesdenton: :)
16:59:09 <evrardjp> jamesdenton: with the mighty strength of my PEBKACs!
16:59:20 <jamesdenton> :D
17:00:11 <tiffanie> #endmeeting