16:02:14 <evrardjp> #startmeeting openstack_ansible_meeting
16:02:16 <openstack> Meeting started Tue May 21 16:02:14 2019 UTC and is due to finish in 60 minutes.  The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:02:19 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:02:25 <mnaser> #topic office hours
16:02:36 <mnaser> oh im not cool enough to do that right now
16:02:37 <evrardjp> #chair mnaser
16:02:37 <openstack> Current chairs: evrardjp mnaser
16:02:41 <mnaser> #topic office hours
16:02:44 <mnaser> ok now im cool enough
16:02:50 <guilhermesp> o/
16:03:01 <logan-> o/
16:03:01 <mnaser> evrardjp: btw, awesome patches on the integrated stuff on top of odyssey4me work
16:03:13 <evrardjp> https://docs.google.com/spreadsheets/d/1coiPHGqaIKNgCGYsNhEgzswqwp4wedm2XoBCN9WMosY/edit#gid=752070695
16:03:32 <jrosser> o/
16:03:32 <mnaser> welp
16:03:35 <evrardjp> that's the map of where we are. Focusing on metal. then metal distro. then lxc.
16:03:37 <mnaser> I gotta vpn for that link :)
16:03:54 <mnaser> ethercalc's aren't blocked :p
16:04:02 <evrardjp> I guessed
16:04:16 <mnaser> all good, I have it open here now
16:04:18 <evrardjp> It doesn't matter for the conversation, basically there are reviews up, and things needs improvement
16:04:42 <evrardjp> I see a few red dots in the horizon, for example, ironic, congress, blazar
16:04:59 <evrardjp> octavia, rally, tacker, trove, zun
16:05:16 <evrardjp> those are either not ready for distro installs, or they are not ready for multi distro, or both
16:05:33 <evrardjp> (there are others I didn't mention)
16:05:41 <mnaser> hmm, I think its mostly just adding some vars/<os>.yaml file?
16:05:49 <mnaser> or is it quite a bit more complicated than that
16:05:50 <evrardjp> some of it is just that.
16:06:03 <mnaser> obviously I'm making it far more trivial but yeaeh
16:06:04 <evrardjp> some are not just that, for example the packages don't exist for that distro
16:06:26 <evrardjp> or need to refactor the code to get to a point that's similar to other roles
16:06:35 <evrardjp> anyway this deserves love, and help is mostly welcomed
16:07:05 <evrardjp> there is a "if you want to help" on top of this page, so it should help ppl understand what's going on
16:07:09 <mnaser> ill try to do my part in reviewing, and anything failing debian/centos I can try and make a patch 'underneath'
16:07:22 <guilhermesp> evrardjp: I will take a look carefully
16:07:33 * raukadah didnot getting time to look on python-httplib2 failure on centos
16:07:40 <guilhermesp> I will also continue mnaser 's work of db refactor https://review.opendev.org/#/q/topic:osa/db-refactor
16:07:59 <evrardjp> mnaser: I think the most annoying is not the patch underneath for metal support for distro x, but more getting to support distro package isntalls
16:08:43 <mnaser> yeah.  that's quite a lot more work to be done there
16:08:44 <evrardjp> anyway, sometimes there are patches "underneath" those links, so don't hesitate to look at the "related" patches
16:09:03 <evrardjp> that's all I have on that topic
16:09:30 <evrardjp> the other topic I have is linked to reviews, should I just continue?
16:10:12 <logan-> may as well. thanks for putting up the spreadsheet and patches for the integrated tests stuff!
16:10:20 <mnaser> sure. I think what you discussed in on-going anyways and doesn't have anything actionable in specific other than: pay attention to reviews, try to help land some of these changes
16:10:38 <evrardjp> yes
16:10:46 <evrardjp> we'll need a map, basically, for another thing too:
16:12:16 <evrardjp> We'll have to use the constraints url file that are published by releases. I guess we'll discuss whether to use /master or /train for master (I don't have an opinion yet). But for stable branches, we're gonna have to do it too.
16:12:50 <evrardjp> The thing is... We shouldn't use that url everywhere, because that will always represent the head of the branch. So we'll have to be careful in the reviews.
16:12:56 <mnaser> don't we already pin commit hash for constraints. right now?
16:13:04 <evrardjp> correct.
16:13:14 <evrardjp> so those shouldn't change.
16:13:29 <evrardjp> the ones who can change are the test ones basically
16:13:35 <evrardjp> (tox, etc.)
16:14:03 <mnaser> so on release, we would freeze that commit, but in CI, we point to tip of requirements branch matching the release
16:14:04 <evrardjp> I just want to raise awareness we shouldn't just change everything :)
16:14:05 <mnaser> right?
16:14:31 <evrardjp> I think we can keep the things frozen for requirements/repo build and many
16:14:39 <evrardjp> and bump regularily
16:14:51 <evrardjp> but the tox and all should be using the new urls.
16:14:55 <noonedeadpunk> o/ missed start
16:15:01 <evrardjp> So basically don't use the new urls everywhere :p
16:15:01 <mnaser> ah yes, I agree, that's fine
16:15:14 <mnaser> use the new urls in tox only because we want to keep our pinned requirements
16:15:27 <evrardjp> yes.
16:15:37 <mnaser> ok, makes sense to me!
16:15:43 <evrardjp> there might be some other places we need adapting, but we need to do it smartly basically :)
16:15:54 <evrardjp> that's all I had on this topic
16:16:18 <mnaser> fairz
16:16:23 <evrardjp> the last topic I had was releases. I am late on releasing due to summit and travels and stuff.
16:16:32 <evrardjp> and broken jobs
16:16:34 <evrardjp> and many others
16:16:35 <logan-> one other note on the new integrated testing rework stuff, you can actually implement scenarios in the roles downstream while still inheriting the integrated repo jobs
16:16:38 <logan-> https://opendev.org/openstack/openstack-ansible-os_nova/commit/8f55c68d8371a50625c3fdb5b82cbeb62f2808c4
16:17:15 <logan-> any pre-run playbooks in the job will run after the AIO has already been bootstrapped
16:17:15 <evrardjp> but logan helped on fixing things in Rocky/Queens, so thanks to him! I hope we'll have new releases soon
16:17:36 <mnaser> ack, yes, stein patch I had to recheck now
16:17:42 <mnaser> I don't think we back ported the placement integration stuff, did we?
16:17:43 <evrardjp> logan-: yeah that's a cool feature
16:17:44 <logan-> yes thanks for pushing the reviews through.. queens is almost unblocked as of now
16:18:02 <logan-> mnaser: no it is not backported
16:18:09 <mnaser> yeah, that's all in master right now..
16:18:10 <guilhermesp> no we didn't mnaser
16:18:16 <mnaser> what a mess it was to land it :<
16:18:23 <logan-> yeah
16:18:40 <guilhermesp> and I think we haven't worked in the upgrade stuff for placement
16:18:55 <logan-> ^ right
16:19:07 <mnaser> I'm thinking: we land most recent bump and push rc2, backport the placement stuff (so we at least have Greenfield placement in stable/stein)
16:19:14 <guilhermesp> i still have a picture of odyssey4me 's pseudo code :P
16:19:28 <mnaser> and then we can work on the upgrade stuff afterwards.. as we usually don't release with the expectation of functional upgrades (generally)
16:19:42 <mnaser> thoughts.. complaints..? :p
16:20:10 <evrardjp> we have until june to do so
16:20:10 <guilhermesp> I'd be ok as first step just try to have stable/stein greenfield for placement
16:20:17 <evrardjp> or july?
16:20:18 <noonedeadpunk> sounds reasonable
16:20:21 <evrardjp> can't remember let me check
16:20:22 <logan-> i think we should focus on getting stein released and then revisit the placement backport since it is optional for stein
16:20:54 <guilhermesp> I can take care of this backport anyways
16:20:57 <evrardjp> 11 July, 2019
16:21:09 <mnaser> logan-: if we release stein without placement, then we don't backport placement
16:21:24 <evrardjp> I think we could, as a community, agree on backporting afterwards
16:21:37 <logan-> backporting placement in its current state would probably set us back quite a bit on R->S upgrade readiness right?
16:21:39 <evrardjp> as long as we are clear in the ML/renos/versions
16:21:58 <mnaser> it is a very invasive upgrade step so as someone who is going to update openstack, I'd be in for a mean surprise
16:22:11 <mnaser> we do things like create new databases, shut off services and migrate data from one db to another
16:22:12 <logan-> yeah I think a backport is more palatable when we have code at least proposed if not landed to enable upgrades
16:22:13 <guilhermesp> yes, you're right evrardjp logan- mnaser
16:22:15 <mnaser> many things can go wrong™
16:22:47 <mnaser> I think odyssey4me suggestion at the time was that we never release OSA with the expectation of upgrades working from day 1, so that was kind at the story there.
16:23:33 <mnaser> the placement changes are pretty major, I would either release with them or without, I would not think making this back portable is good for our users
16:24:09 <evrardjp> I agree with you, I just insist on deadlines
16:24:21 <evrardjp> :)
16:24:29 <mnaser> if we agree that we can release without upgrades, then I think we should be (relatively) ok
16:24:44 <mnaser> if we agree that a we must release with upgrades support, we might have to drop it
16:25:10 <mnaser> I know odyssey4me mentioned that he volunteered to help with that upgrade effort.. so it'd be nice to hear if he can still do it (or not)
16:25:35 <evrardjp> If we don't provide the upgrade, it will be a hard mess for many, and I am not sure who will make the effort for doing so
16:25:50 <evrardjp> (or when)
16:26:00 <jrosser> can we have a "placement is present but dormant" situation then if the migration tooling appears then we are good
16:26:23 <mnaser> nope, because it uses an entirely different database
16:26:37 <mnaser> so if it does get deployed, it'll probably want to take over the service catalog entry too
16:27:10 <mnaser> running the playbook will effectively update the service catalog entry to point @ the new placement
16:27:18 <evrardjp> it makes sense to release stein with it, right?
16:27:21 <openstackgerrit> Merged openstack/openstack-ansible stable/rocky: Use opendev links  https://review.opendev.org/659933
16:27:27 <evrardjp> it's in that cycle that it was extracted
16:27:39 <mnaser> I think so.  we would save a lot of trouble for ourselves by doing it now than later
16:27:41 <evrardjp> so it's easier for us to remember that's when you'll see it appear
16:28:16 <mnaser> I can't remember but Jesse had a *very* good reason for including it in stein too
16:28:21 <mnaser> I really don't remember it off the top of my head now
16:28:21 <logan-> it seems like we're not saving ourselves trouble, it is the same amount of trouble but we're accelerating it for <some gain> (i'm not sure what the gain is yet)
16:28:45 <mnaser> yes, that <some gain> was something Jesse mentioned that got me thinking at the PTG, not remembering though :(
16:28:51 <mnaser> hopefully when he reads buffer, he can chime in
16:29:04 <logan-> ++
16:29:32 <evrardjp> odyssey4me: ^
16:30:21 <evrardjp> I guess I will propose an rc2 soon, but wait for the final release, and we can discuss this later.
16:30:33 <evrardjp> that's all I had for today
16:30:35 <logan-> thanks evrardjp
16:30:39 <mnaser> evrardjp: yes, I rechecked your most recent bump
16:30:45 <mnaser> it passed check but failed gate so it should be ok.
16:31:12 <logan-> on a separate topic, this is not urgent at all but I would like to get some thoughts on credential scoping in roles, I pushed some POC patches for nova. if these are adopted, this concept would likely apply to a number of other roles as well.
16:31:12 <logan-> #link https://review.opendev.org/#/q/topic:fix-credentials-scoping
16:31:37 * mnaser looks
16:32:03 <mnaser> logan-: I looked over that, I was hoping if you took maybe sometime to ask the nova/neutron team about possible side effects
16:32:14 <mnaser> I'm thinking places where the resource was owned by X and now it is owned by Y
16:32:43 <mnaser> generally all openstack install guides have kinda historically mentioned to use users of the other service (but I totally agree with your approach tbh)
16:32:44 <logan-> yep, gotcha, I will do that
16:33:22 <mnaser> but I think that's really neat, reduce the # of vars and it just makes sense that nova uses its own user to talk to neutron, rather than the neutron user
16:33:36 <evrardjp> logan-: gosh this is amazing
16:34:03 <evrardjp> all those vars will go away
16:34:04 <noonedeadpunk> can't agree more, looks nice:)
16:34:04 <mnaser> -54 in group_vars/all is a win
16:35:05 <evrardjp> imagine if the rest would be in a simple etcd? :p
16:35:16 <evrardjp> or local facts?
16:35:17 <evrardjp> wow
16:35:23 <mnaser> #thanks logan- awesome efforts in cleaning up and unblocking OSA gates
16:35:25 <openstackstatus> mnaser: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks)
16:35:45 <mnaser> #thanks evrardjp dealing with OSA's painful release process!
16:35:46 <openstackstatus> mnaser: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks)
16:36:11 <guilhermesp> ++
16:36:28 <logan-> thanks for looking at those. glad the concept makes sense, it seems like the main concerns are upgrade impact, so I'll look into that some more
16:36:37 <mnaser> this should be interesting to watch - https://review.opendev.org/#/q/topic:osa/db-refactor
16:36:48 <mnaser> I don't think I have time to push more on those but the 'general' concept is there
16:37:15 <guilhermesp> yeah I started t look at the topic today
16:37:28 <guilhermesp> just did a small fix in linters but I haven't look yet at the failures
16:37:32 <guilhermesp> gonna be working on this
16:37:48 <mnaser> just needs a little bit of cleanup and then needs to be done across all roles, then we can use the proposal bot to keep it in sync :)
16:37:49 <mnaser> big win
16:38:01 <mnaser> it also re-orders things so the db and rabbit are ready before we deploy
16:38:03 <guilhermesp> ++
16:38:11 <mnaser> no more crash error looping forever while we configure stuff
16:38:47 <evrardjp> mnaser: commented on https://review.opendev.org/#/c/657029/2
16:38:50 <logan-> yeah that will be nice
16:39:15 <mnaser> I'm going to be a bit 'more' back in service when I come back home on the 29th
16:39:36 <mnaser> apologies for my MIA-ness, dealing with this fairly complex deployment and being in a different timezone/space/world
16:39:50 <evrardjp> I think the reason we used different names for those tasks was to easily compare those, and see what this task was doing. But with ARA we don't need it anymore
16:41:43 <mnaser> yupp
16:41:59 <guilhermesp> I'd rename this as glance_db_sync https://review.opendev.org/#/c/657029/2/tasks/glance_db_setup.yml
16:42:15 <guilhermesp> if this is a pattern across other roles btw
16:42:29 <evrardjp> guilhermesp: +1 for me
16:43:32 <guilhermesp> but that's not the case for nova tough https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_db_setup.yml :P
16:43:56 <openstackgerrit> Merged openstack/openstack-ansible stable/rocky: Add Calico networking AIO scenario  https://review.opendev.org/659712
16:43:57 <openstackgerrit> Merged openstack/openstack-ansible stable/queens: Use opendev links  https://review.opendev.org/659936
16:44:03 <mnaser> yeah id work on the more simpler ones first and then clean up those other ones
16:44:06 <evrardjp> guilhermesp: but we could be consistent in the future
16:44:13 <evrardjp> more consistent*
16:44:30 <guilhermesp> yeah evrardjp I agree refactoring stuff that makes more sense is a great first step
16:46:23 <jrosser> ansible 2.8.0 for master is very close to done, it needs the ceph-ansible 4.0.0 work to merge first though
16:46:33 <evrardjp> jrosser: how is that last part?
16:46:56 <evrardjp> I think it would be pretty early in the cycle to move to 2.8, so that's a good move
16:47:40 <jrosser> the ceph patch is looking reasonable, except that it fails tempest on some swift stuff
16:47:58 <evrardjp> :(
16:48:33 <jrosser> it wasnt a very obvious failure, i bring this up in case anyone has an idea where to look next http://logs.openstack.org/03/656503/10/check/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/3bfd9d8/logs/openstack/aio1_utility_container-78e37705/utility/stestr_results.html
16:50:08 <openstackgerrit> Merged openstack/openstack-ansible stable/queens: Restore linters job to voting  https://review.opendev.org/660018
16:50:38 <logan-> jrosser: same ceph version as we currently run in master right?
16:52:13 <openstackgerrit> Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_keystone master: Update uw_apache to run against bionic  https://review.opendev.org/660455
16:52:35 <jrosser> logan-: that patch brings it up to nautilus
16:53:10 <jrosser> which is guess could be optional, perhaps i should try the same patch deploying mimic to separate the problems
16:53:11 <logan-> if 4.0 supports mimic maybe we should split up the ceph-ansible upgrade from the ceph ugprade
16:53:34 <openstackgerrit> Merged openstack/openstack-ansible stable/queens: Bump etcd role to v3 capable SHA  https://review.opendev.org/659868
16:53:41 <logan-> because i suspect its a m->n radosgw keystone auth issue that will need to be worked out, probabyl unrelated to ceph-ansible v4
16:54:34 <jrosser> yes i think thats quite possible, i'll redo the patch just to bring ceph-ansible up to 4.0.0
16:57:11 <mnaser> Neat
16:58:23 <openstackgerrit> Jonathan Rosser proposed openstack/openstack-ansible master: Update to ceph-ansible 4.0.0  https://review.opendev.org/656503
17:01:09 <logan-> thanks jrosser, if we can get ceph-ansible 4 in that will at least unblock the ansible 2.8 stuff and then we can work out the ceph upgrade separately. ill look thru the nautilius release notes to see if I can spot the change
17:01:54 <logan-> last thing i've got is https://review.opendev.org/#/q/status:open+owner:logan2211%2540gmail.com+(topic:calico3+OR+topic:osa/opendev) -- only a few more reviews needed to finish up fixing the gates + getting calico both working and tested from Q->master (previously calico was working on P and earlier but broken Q->master)
17:13:48 <logan-> jrosser: i wonder if the broken pipe requests in radosgw line up with the tempest fails http://logs.openstack.org/03/656503/10/check/openstack-ansible-deploy-aio_ceph-ubuntu-bionic/3bfd9d8/logs/openstack/aio1_ceph-rgw_container-2cdc0de2/ceph/ceph-rgw-aio1-ceph-rgw-container-2cdc0de2.rgw0.log.txt.gz
17:15:01 <logan-> hopefully there is a debug mode where we can get some more output there because that's useless :/
17:15:07 <jrosser> yes that’s possibly it
17:15:11 <jrosser> Agreed!
17:15:36 <jrosser> That should be on by default in for these tests if there is one
17:15:53 <mnaser> #endmeeting