18:01:07 <SumitNaiksatam> #startmeeting networking_policy 18:01:08 <openstack> Meeting started Thu Sep 7 18:01:07 2017 UTC and is due to finish in 60 minutes. The chair is SumitNaiksatam. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:01:11 <openstack> The meeting name has been set to 'networking_policy' 18:01:18 <SumitNaiksatam> rkukura: thanks for chairing the meeting last time 18:01:33 <SumitNaiksatam> tbachman: thanks for shepherding the discussion on the DB migrations 18:01:46 <SumitNaiksatam> annakk: thanks for bringinging up the UI patches 18:01:51 <tbachman> SumitNaiksatam: not sure I shepherded as much as watched 18:01:57 <tbachman> the sheep moved themselves ;) 18:02:05 <SumitNaiksatam> and my apologies for going MIA at the last minute 18:02:07 <rkukura> baa 18:02:07 <tbachman> SumitNaiksatam: +1k to annakk 18:02:11 <SumitNaiksatam> tbachman: lol! 18:02:15 <tbachman> rkukura: lol 18:02:19 <annakk> :) 18:02:31 <SumitNaiksatam> #info agenda https://wiki.openstack.org/wiki/Meetings/GroupBasedPolicy#Sept_7th_2017 18:02:36 <SumitNaiksatam> we have a bit to cover today 18:02:39 <SumitNaiksatam> lets get rolling 18:02:50 <SumitNaiksatam> #topic mitaka, newton DB migration backport strategy 18:02:57 <SumitNaiksatam> #link https://review.openstack.org/#/c/499185/ 18:03:03 <SumitNaiksatam> #link https://review.openstack.org/#/c/499240 18:03:34 <SumitNaiksatam> i read the transcript of the last meeting, and i meant to respond last thursday itself, somehow didnt happen, and then fast forward to today :-) 18:03:51 <tbachman> SumitNaiksatam: no worries. Good to get it firsthand here 18:04:01 <SumitNaiksatam> i agree with the rkukura and the general point that we have a short term problem to fix and then a longer term decision to make 18:04:03 <tbachman> (and thanks for catching up via transcript) 18:04:11 <SumitNaiksatam> tbachman: np 18:04:40 <SumitNaiksatam> we cannot do any more releases without solving the current mess we are in (regardless of what longer term strategy we use) 18:04:52 <SumitNaiksatam> so lets first try to resolve that 18:05:19 <SumitNaiksatam> would it make more sense to start with the mitaka to newton migration issue first? 18:05:33 <rkukura> SumitNaiksatam: Can we assume master or stable branch trunk chasing use cases are not a concern? 18:05:35 <SumitNaiksatam> (the discussion will most likely apply to newton to ocata as well) 18:05:46 <SumitNaiksatam> rkukura: yes, we never claimed to support them 18:05:54 <SumitNaiksatam> rkukura: i mean we support them 18:06:03 <SumitNaiksatam> but we did not claim that they would not break 18:06:08 <rkukura> ok, just wanted to make that explicit 18:06:22 <SumitNaiksatam> rkukura: very good point, thanks for stating that upfront 18:07:29 <SumitNaiksatam> so rkukura and tbachman you idenfied that there are a couple of migrations that are skipped between mitaka and newton? 18:07:36 <rkukura> for mitaka->newton upgrade, I see two migraitions that will be missed 18:07:51 <SumitNaiksatam> (these being different from the ones that are missing between newton and ocata) 18:08:03 <rkukura> 5239 tenant_id->project_iod 18:08:16 <rkukura> c460 NFP devstack 18:08:24 <SumitNaiksatam> rkukura: okay 18:08:31 <tbachman> SumitNaiksatam: here was the list I’d compiled: https://pastebin.com/sstdM9ZZ 18:08:34 <SumitNaiksatam> i also want to make a general comment regarding NFP 18:08:41 <SumitNaiksatam> tbachman: great thanks 18:08:47 <tbachman> SumitNaiksatam: np! 18:09:02 * tbachman doesn’t have #link powers 18:09:08 <tbachman> or maybe I do? 18:09:08 <SumitNaiksatam> oh 18:09:15 <tbachman> #link https://pastebin.com/sstdM9ZZ <= DB migrations 18:09:20 <SumitNaiksatam> #chairs rkukura tbachman annakk 18:09:29 <SumitNaiksatam> #chair rkukura tbachman annakk 18:09:30 <openstack> Current chairs: SumitNaiksatam annakk rkukura tbachman 18:09:32 <tbachman> I guess I didn’t have topic powers 18:09:40 <tbachman> but I did have link 18:10:00 <rkukura> tbachman: I don’t see 5239 on your list 18:10:11 <tbachman> hmmmm 18:10:20 * tbachman tries to remember why 18:10:29 <SumitNaiksatam> so my comment on NFP, to the extent i am aware of, we dont have a user currently deploying NFP on mitaka or newton 18:10:57 <rkukura> tbachman: is it because tenant_id wasn’t renamed project_id in the DB until newton? 18:11:23 * tbachman is looking 18:11:33 <SumitNaiksatam> so while we should try our best to maintain the integrity of that component, we can make changes to that component without fear of disruption 18:11:34 * rkukura has not looked at when neutron renamed it 18:12:33 <tbachman> rkukura: which patch introduced that? 18:12:41 <rkukura> tbachman: to gbp? 18:12:46 <tbachman> yeah 18:12:48 <SumitNaiksatam> i think the change has been going in neutron for a while 18:12:57 <SumitNaiksatam> but i think we were forced to make the change in newton? 18:13:14 <SumitNaiksatam> annakk did the change in GBP if recall correctly 18:13:19 <annakk> yes 18:13:27 <SumitNaiksatam> and we didnt want to backport it then 18:13:43 <tbachman> ah 18:14:02 <tbachman> rkukura: I think my list was based on the most recent migration in stable/mitaka 18:14:06 <tbachman> so it likely doesn’t cover that one 18:14:10 <tbachman> (and good point) 18:14:24 <rkukura> yes it was annakk’s newton patch, so we don’t want it on mitaka 18:15:06 <rkukura> but it needs to run when upgrading from mitaka->newton, and it will not because mitaka’s HEAD is beyond it in newton’s chain 18:15:10 <tbachman> #link https://github.com/openstack/group-based-policy/commit/f7f1ce349e2ed7579f64430c8cf8f0f5137c8eec#diff-9ae7c407a57ed5bb5e916dbbf4530e8b <= patch with DB migration 18:15:17 <tbachman> rkukura: ack 18:15:24 <SumitNaiksatam> rkukura: ack 18:16:26 <SumitNaiksatam> so can we introduce conditional migrations at the HEAD that adds those changes only if they are missing? 18:16:36 <rkukura> seems like, on stable/newton, we need to make that migration idempotent and move closer to HEAD 18:16:51 <rkukura> SumitNaiksatam: I think we are saying the same thing 18:16:53 <annakk> SumitNaiksatam 18:16:55 <SumitNaiksatam> rkukura: right 18:17:06 <annakk> I think its the best way to go 18:17:27 <annakk> if we care about same-branch migrations 18:17:31 <rkukura> same thing for the NFP patch? 18:18:04 <SumitNaiksatam> rkukura: i would vote in favor of doing it for the NFP patch as well, just to maintain the overall consistency 18:18:28 <rkukura> SumitNaiksatam: right, even if someone isn’t using NFP, the DB schema needs to be consistent with the code 18:18:32 <SumitNaiksatam> the migration patch would however have to be introduced in master, and would have to backport its way to newton 18:18:58 <SumitNaiksatam> rkukura: agree, depending on the schema change 18:19:11 <rkukura> SumitNaiksatam: I agree the same issue with the same two patches will be issues on stable/ocata and master 18:19:16 <SumitNaiksatam> i guess thats what you mean by “consistent" 18:19:34 <SumitNaiksatam> rkukura: ack 18:20:12 <rkukura> so there are two more migrations that will be missed on upgrade from mitaka or newton to ocata: 18:20:18 <SumitNaiksatam> hmmm 18:20:22 <rkukura> da6a QoS 18:20:31 <rkukura> bff1 NFP 18:20:59 <SumitNaiksatam> okay, rkukura thanks for digging those out 18:22:50 <rkukura> and my notes have one that at that time was HEAD on master but not back-ported to ocata: 4c0c APIC 18:24:10 <rkukura> so should our strategy be to do one patch that fixes everyhing on master that might be missed when upgrading from mitaka, and then manually back-porting the needed parts of that to stable/ocata and stable/newton? 18:24:43 <SumitNaiksatam> rkukura: ack on one patch 18:24:44 <tbachman> rkukura: seems like a sound approach 18:24:54 <annakk> agree 18:24:59 <SumitNaiksatam> stable/ocata would be identical to master 18:25:29 <rkukura> SumitNaiksatam: I agree it should be, until there is something new on master that doesn’t get back-ported 18:25:55 <SumitNaiksatam> rkukura: right 18:26:15 <SumitNaiksatam> there is a some difference between newton and ocata 18:26:24 <SumitNaiksatam> namely, QoS and the NFP changes 18:26:30 <rkukura> SumitNaiksatam: right 18:26:33 <SumitNaiksatam> okay so assuming we follow this plan 18:26:56 <SumitNaiksatam> we will be left with different chains on different branches 18:27:00 <rkukura> I could do these patches, but probably wouldn’t get to them til mid next week. I’m flying BOS->SFO tomorrow (might be able to do them on the plane) and taking PTO Monday 18:27:03 <tbachman> should we revert the ip_pool changes the? 18:27:17 <tbachman> s/the/the/ 18:27:18 <tbachman> dang 18:27:21 <tbachman> s/the/then/ 18:27:27 * tbachman checks his “n” key 18:27:42 <rkukura> tbachman: 27b7 is on master, ocata, and newton, right? 18:27:47 <tbachman> rkukura: ack 18:27:57 <tbachman> but placement was the question 18:28:10 <tbachman> just wasn’t sure if it made things simpler 18:28:15 * tbachman is fine with leaving it as-is 18:28:29 <tbachman> but — I’ll revert the other 2 DB migration patches that I have 18:28:30 <SumitNaiksatam> tbachman: you are thinking that we should put this change also in the “uber” backport patch? 18:28:32 <rkukura> is there a reason to have that change on newton, or not? 18:28:57 <tbachman> the IPv6 feature is all the way back to stable/mitaka 18:29:08 <SumitNaiksatam> rkukura: ideally i think it should be all the branches 18:29:09 <tbachman> the question is whether we want the same behavior of ip_pool 18:29:29 <tbachman> my thinking is that it should be the same 18:29:41 <SumitNaiksatam> tbachman: ack 18:29:45 <rkukura> tbachman: ack 18:30:04 <tbachman> which leaves the stable/mitaka patch for that feature 18:30:27 <tbachman> #link https://review.openstack.org/#/c/499185/ <= backport of ip_pool size increase to stable/mitaka 18:32:25 <rkukura> so if we back-port this to stable/mitaka and 27b7 becomes HEAD on stable/mitaka, we will have more patches that would be missed upgrading from mitaka to newton or ocata 18:32:37 <rkukura> s/patches/migrations/ 18:32:51 <SumitNaiksatam> rkukura: so thats the longer term issue 18:33:00 <rkukura> guess so 18:33:11 <SumitNaiksatam> my earlier comment - “we will be left with different chains on different branches” 18:33:29 <SumitNaiksatam> so what do we do for future backports once we have cleaned up the mess 18:33:47 <rkukura> SumitNaiksatam: good transition 18:33:56 <SumitNaiksatam> (ip_pools falling into that future backports category though we already have the patch ready) 18:34:16 <annakk> why do we backport agressively? 18:34:25 <SumitNaiksatam> annakk: to support users :-) 18:34:29 <rkukura> one option is to try to keep the DB schema identical across the active branches 18:34:38 <SumitNaiksatam> rkukura: right 18:35:07 <annakk> users are reluctant to go forward in release? 18:35:09 <SumitNaiksatam> annakk: typically the users, once they deploy a particular release, dont move as quickly 18:35:20 <SumitNaiksatam> annakk: within the same release, yes 18:35:30 <SumitNaiksatam> but moving forward to a new release takes more doing 18:35:41 <SumitNaiksatam> i know people still using liberty 18:36:10 <SumitNaiksatam> i mean legit users in production 18:36:29 <rkukura> annakk: also, GBP was evolving rapidly, so users wanted the latest GBP 18:36:40 <SumitNaiksatam> rkukura: ack 18:36:55 <annakk> i think there's a general consensus in openstack to avoid backporting features that require schema changes 18:37:09 <SumitNaiksatam> annakk: completely agree and understand 18:37:24 <SumitNaiksatam> annakk: but in those cases, the distro vendors are still doing the backporting 18:38:11 <SumitNaiksatam> albeit selectively and perhaps on a per customer basis 18:38:18 * tbachman notes there are 2 classes of openstack users. 1) pre-production, who want the latest openstack release; 2) production users, who never want to move to a new release ;) 18:38:27 <SumitNaiksatam> tbachman: lol 18:38:33 <annakk> :) 18:38:52 <SumitNaiksatam> tbachman: and on (2) not only do they dont want to move, they still want all the latest features :-) 18:39:05 <rkukura> tbachman: Also, 1.5) want newest GBP on older OpenStack 18:39:08 <SumitNaiksatam> and that latter part makes it really difficult to support 18:39:08 <tbachman> lol 18:39:25 <SumitNaiksatam> rkukura: i categorized that as 2.5 18:39:53 <SumitNaiksatam> anyway, annakk to your question, we decided to take the simple way out - 18:39:59 <SumitNaiksatam> that is try to keep all branches consistent 18:40:10 <rkukura> so should GBP start playing by the rules at some point? 18:40:20 <SumitNaiksatam> that is feature wise, to the extent possible 18:41:02 <annakk> SumitNaiksatam: sounds like its only possible if new features don't conflict with neutron db all the way back 18:41:09 <SumitNaiksatam> rkukura: i think if it was left to us, it would be much easier for us to just play by the rules since it reduces this headache of backporting 18:41:27 <SumitNaiksatam> annakk: yes sure, and mostly we havent hit issues in that regard (luckily) 18:42:16 <rkukura> if GBP’s official stable branches followed OpenStack backporting rules, it would be possible to maintain different branches (unstable/mitaka, unstable/newton, unstable/ocata, …) that more aggressively back-port, but that is more work 18:42:53 <SumitNaiksatam> rkukura: exactly i was just saying that, we dont have a “distro” vendor like entity doing that “unstable” branch work for us 18:43:07 <SumitNaiksatam> for other openstack projects its being done by those distro vendors 18:43:15 <rkukura> SumitNaiksatam: right 18:43:42 <annakk> thanks for the 101, makes sense 18:43:42 <rkukura> but even then, new features are typically not back-ported, are they? 18:43:58 <rkukura> s/are/aren’t/ 18:44:00 <SumitNaiksatam> rkukura: i agree, mostly not features 18:44:55 <SumitNaiksatam> but i guess its easier for them to put the stake in the ground for those projects 18:46:12 <SumitNaiksatam> of course we can decide to the same thing here, but i am not sure if this is really beyond our control in the sense that if a user asks for a feature be backported, would we actually be able to say no! 18:46:42 * tbachman wonders if we’re any closer to a path forward here 18:47:01 <SumitNaiksatam> tbachman: thanks for the reality check :-) 18:47:11 <tbachman> sorry — not trying to be negative 18:47:25 <tbachman> seems like we have several options 18:47:29 <SumitNaiksatam> tbachman: no, didnt mean to suggest that you were implying that 18:47:31 <rkukura> it would be good to keep some flexibility 18:48:23 <SumitNaiksatam> one thing is for sure, we need to be much more selective in backporting features/patches that involve DB migrations (assuming we even decide that we will allow those and have a strategy around it) 18:48:26 <SumitNaiksatam> rkukura: ack 18:48:41 <rkukura> I think a good case to think about is 5239 (tenant->project), which should definitely not be back-ported to stable/mitaka, but is definitely needed when upgrading from mitaka to anything newer 18:48:44 <SumitNaiksatam> this current mess is a good opener in that sense 18:49:30 <SumitNaiksatam> rkukura: so that would be the argument against making all migration chains identical? 18:50:17 <tbachman> SumitNaiksatam: you had mentioned the idea of backporting with a no-op 18:50:48 <SumitNaiksatam> tbachman: thanks, i did, but that is not the ultimate solution, because that introduces the issue of skipping that migration all together 18:50:57 <SumitNaiksatam> when migrating from one branch to the next 18:51:16 <SumitNaiksatam> i meant the no-op only for upgrades witthin the same branch 18:51:41 <SumitNaiksatam> instead of no-op we could do conditional, but i dont think that would solve the problem 18:51:49 <SumitNaiksatam> completely 18:51:59 <SumitNaiksatam> we have 9 mins remaining 18:52:05 <rkukura> I don’t see how no-op solves anything 18:52:38 * tbachman nods 18:53:06 <SumitNaiksatam> no-op could potentially solve the problem of temporarily get the chains to be temporarily identical across the branches 18:53:41 <annakk> but they can't stay identical forever, can they? 18:53:43 <rkukura> I think it might help to keep the migrations that are not backported to older branches in front of (closer to HEAD than) migrations that are back-ported 18:53:54 <SumitNaiksatam> but we are solving the short term problem in a better way here by introducing the idempotent migrations at the head 18:54:12 <SumitNaiksatam> annakk: agree 18:54:18 <rkukura> no-op makes the chains look identical, but they still are not really identical 18:54:31 <SumitNaiksatam> annakk: but we would have more control over what we merge in the future 18:54:46 <SumitNaiksatam> rkukura: right, schema is not identical 18:55:08 <annakk> how do we handle situation when feature X is not backported, but after that comes feature Y that is backported? 18:55:16 <SumitNaiksatam> rkukura: regarding your earlier point, it can still get us to same situation we are currently in 18:55:18 <annakk> no-op for X? 18:55:49 * tbachman understands why the OpenStack policy is not to backport DB migrations :( 18:55:55 <SumitNaiksatam> i dont think we should discuss no-op since it doesnt solve the longer term problem 18:56:05 <SumitNaiksatam> sorry that i brought it up :-( 18:56:15 <tbachman> SumitNaiksatam: I did actually — sorry 18:56:24 <tbachman> (in this meeting, that is) 18:56:40 <SumitNaiksatam> tbachman: i take the responsibility 18:56:41 <SumitNaiksatam> okay so in the interest of time 18:56:49 <SumitNaiksatam> we will have to go back to emails 18:56:54 <SumitNaiksatam> lets try to close this by tomorrow 18:57:07 <SumitNaiksatam> perhaps we can take a break and think again with some more clarity 18:57:16 <SumitNaiksatam> #topic UI patches 18:57:21 <SumitNaiksatam> i have tried the ocata UI 18:57:32 <SumitNaiksatam> annakk: does your patch fix the issues you are seeing? 18:57:34 <rkukura> I think a policy on ordering the migrations when back-porting could at least prevent the need to shuffle them and make them idempotent going forward 18:57:53 <SumitNaiksatam> rkukura: definitely 18:58:07 <rkukura> lets cover UI 18:58:13 <SumitNaiksatam> sorry i meant i *have not* tried Ocata UI 18:58:16 <annakk> SumitNaiksatam: it fixes one issue, and I'm not sure about the rest since I discovered issues linger after horizon restart 18:58:20 <SumitNaiksatam> i have tested newton recently 18:58:31 <SumitNaiksatam> annakk: okay, i will give it a try 18:58:56 <SumitNaiksatam> sorry that you ran into this after we released ocata, should have been caught before release 18:59:19 <SumitNaiksatam> annakk: thanks also for starting to look at the UI patchers 18:59:24 <SumitNaiksatam> tbachman: thanks to you as well 18:59:32 <SumitNaiksatam> #topic py27 logs 18:59:39 <tbachman> SumitNaiksatam: np! 18:59:43 <annakk> I'll continue testing it above the fix 18:59:53 <SumitNaiksatam> please take look here #link https://wiki.openstack.org/wiki/Meetings/GroupBasedPolicy#Sept_7th_2017 18:59:56 <SumitNaiksatam> annakk: thanks 19:00:06 <SumitNaiksatam> in the past we had cleaned up 19:00:25 <SumitNaiksatam> but again we have a huge amount of repetitive logging 19:00:39 <SumitNaiksatam> makes it very difficult to debug test failures 19:00:49 <SumitNaiksatam> we need to clean up/supress 19:01:00 <SumitNaiksatam> and check if there are any real issues hidden here in the error message 19:01:04 <SumitNaiksatam> *messages 19:01:08 <SumitNaiksatam> thanks all for joining today 19:01:10 <SumitNaiksatam> #endmeeting