15:01:31 <ihrachys> #startmeeting neutron_upgrades
15:01:32 <openstack> Meeting started Mon Aug  8 15:01:31 2016 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:35 <openstack> The meeting name has been set to 'neutron_upgrades'
15:01:46 <electrocucaracha> o/
15:01:48 <rossella_s> hi
15:01:49 <jlibosva> \o
15:01:49 <korzen> hello
15:01:53 <ihrachys> hello my friends! it's good to see you.
15:02:07 <korzen> good to see you too :)
15:02:09 <rossella_s> it's good to see you
15:02:16 <asingh> Hello
15:02:27 <ihrachys> #link https://wiki.openstack.org/wiki/Meetings/Neutron-Upgrades-Subteam Agenda
15:02:53 * ihrachys looks at bare agenda and thinks I should populate it with some info not to forget everything that we happen to track.
15:02:54 <sayalilunkad> hello
15:03:04 <ankur-gupta-f> o/
15:03:09 <ihrachys> #action ihrachys to update agenda with some major topics we cover
15:03:17 <ihrachys> #topic Announcements
15:03:53 <ihrachys> I have two things: one is N3 is coming, some time at the end of August. The more things we close now the better.
15:04:23 <ihrachys> and second thing, there is a midcycle next week, and we should have some extensive discussions there on upgrades, specifically objects strategy.
15:04:54 <ihrachys> there will probably be a short 'seminar' that armax wants to be delivered on objects, to spread the word to other contributors.
15:05:15 <ihrachys> they say it's learning curve is steep on objects, so we should try to preach the gospel :)
15:05:38 <ankur-gupta-f> :)
15:05:53 <ihrachys> the etherpad for the event captures some upgrades topic:
15:05:58 <ihrachys> #link https://etherpad.openstack.org/p/newton-neutron-midcycle-workitems
15:06:10 <ihrachys> line 27+
15:06:40 <ihrachys> korzen: you go to the event right? me too. anyone else?
15:06:47 <korzen> yeap
15:07:00 <korzen> electrocucaracha, is going
15:07:02 <dasm> electrocucaracha: and me are also going.
15:07:11 <jlibosva> I'll be also there
15:07:32 <ihrachys> good representation it seems
15:07:43 <korzen> ihrachys, anything to prepare for the upgrades discussion?
15:08:08 <korzen> stories, code examples? pop-corn?
15:08:15 <ihrachys> we'll update the team on how it went. if you have anything missing in the agenda that you think should be covered there too, please tell us so that we have a chance to raise your flag.
15:09:21 <ihrachys> korzen: this week I should talk to armax on particularities. after that I will know more. generally, the more specific stuff we bring the better. patches, plans, proposals
15:09:43 <ihrachys> there should be time for general discussions on direction as well as code reviews/hacking
15:10:45 <ihrachys> #topic Partial Multinode Grenade
15:11:14 <ihrachys> grenade jobs are currently busted, but hopefully not our fault :)
15:11:28 <ihrachys> on serious note, for linuxbridge we still wait for https://review.openstack.org/#/c/345707/ and friends to land.
15:11:39 <ihrachys> that should get us closer to solving mtu issues in linuxbridge gate.
15:12:14 <ihrachys> it may take a while for that to land since the trust we had before was broken by a gate breakage from the previous attempt... ;)
15:12:53 <ihrachys> sc68cal even said once neutron-legacy should not be touched much, but then I see +2 from him on the patch, so probably we are good :)
15:13:17 <ihrachys> some neutron grenade jobs were added to devstack experimental queue: https://review.openstack.org/#/c/351450/
15:13:25 <sc68cal> I only do it when absolutely required.
15:13:38 <ihrachys> so in case we need to validate a devstack fix we don't need to build a whole depends-on scaffolding
15:13:40 <sc68cal> ihrachys: are you saying I should change my +2 ? ;)
15:13:59 <ihrachys> sc68cal: may I revoke my comment? :D
15:14:04 <sc68cal> :)
15:14:43 <ihrachys> so all in all, not much progress on that front, but we have a plan. moving on.
15:14:53 <ihrachys> #topic Object implementation
15:15:00 <ihrachys> ...that section should be split...
15:15:40 <ihrachys> ok, first thing first. push-notifications feature by kevinbenton depends on us delivering three core objects so that rpc callbacks can be used for those new messages
15:15:41 <electrocucaracha> we have more people who wants to participate on this
15:16:12 <ihrachys> electrocucaracha: cool, I think models refactoring is a good start? but let's touch it a tad later.
15:16:23 <ihrachys> three objects to deliver are: ports, networks, subnets.
15:16:30 <korzen> ihrachys, subnets should be ready
15:16:35 <korzen> like now in master
15:16:37 <ihrachys> note that kevinbenton really needs objects, not necessarily them adopted in db code.
15:16:41 <ihrachys> right
15:16:53 <ihrachys> subnets are good as korzen said! so that's one.
15:17:04 <ihrachys> for ports, I spinned out a separate patch that includes just the object:
15:17:21 <ihrachys> #link https://review.openstack.org/#/c/351368/
15:17:28 <ihrachys> that one does not contain all needed extensions, but I am close
15:17:42 <ihrachys> one of extensions are secgroups
15:17:45 <ihrachys> #link https://review.openstack.org/#/c/351101/
15:17:48 <manjeets_> test
15:17:53 <ihrachys> ^ again, spinned off the previous patch
15:17:57 <ihrachys> manjeets_: passed :)
15:18:28 <ihrachys> there is some work on the ports object patch, but reviews are welcome, I plan to tackle remaining bits tomorrow.
15:18:36 <ihrachys> and then, the last one is network.
15:18:49 <manjeets_> how important is external network ?
15:18:59 <ihrachys> I see a WIP: https://review.openstack.org/#/c/269658/ for networks
15:19:02 <korzen> network is kind of WIP: https://review.openstack.org/#/c/269658/
15:19:22 <manjeets_> i refactored it and today was going to work for object conversion of external network
15:19:27 <ihrachys> manjeets_: not P1, routers are not in focus right now.
15:19:50 <korzen> I can continue working for network this week
15:19:55 <ihrachys> manjeets_: I think the more viable thing to work on right now is arranging models and objects as per previous discussions. this should prepare the tree for next steps.
15:20:15 <ihrachys> korzen: depending on how long it will take you to get the job done (as in reviewable)
15:20:18 <manjeets_> ok i'll send more patches today for relocation
15:20:45 <ihrachys> korzen: I originally told kevinbenton I will have something for him this week, so if it will take more on your side, I should better step in
15:21:20 <korzen> ihrachys, the problem with network is extesions
15:21:33 <ihrachys> manjeets_: we may not land all of them in one go, we'll need to be strategic so that we avoid git conflicts for other ongoing work.
15:21:43 <ihrachys> korzen: too many of them? weird ones
15:21:44 <ihrachys> ?
15:22:05 <korzen> it is like not OVO extension fields
15:22:36 <korzen> if we drop all of extension main object would be ready very fast
15:23:00 <korzen> but we should define all fields in OVO and then work on loading the extensions properly
15:23:01 <ankur-gupta-f> korzen: that was my thought
15:23:11 <electrocucaracha> but those are required when we are going to integrate the network obj
15:23:12 <tonytan4ever> I have one for agent: https://review.openstack.org/#/c/330870/
15:23:26 <ihrachys> korzen: so it's 'just' work to do? no blockers apart time and effort?
15:23:27 <korzen> and until then, the synthetic fields without any loading methods should not bother much
15:24:31 <ihrachys> tonytan4ever: this should be reworked as per late resolution on intended tree structure: https://review.openstack.org/#/c/351780/
15:24:36 <korzen> ihrachys, I guess that most case has been fixed for subnet so network should not require to have extra fixes in objects API
15:24:55 <tonytan4ever> I see. Will do.
15:25:06 <ihrachys> cool. as long as we at least can get_object() and get a proper object with needed extensions, that's good enough.
15:25:14 <ihrachys> even if the resource is created by other means
15:25:28 <korzen> that is how I'm thinking too
15:25:54 <ihrachys> korzen: ok, let's sync later this week, if I will get time on that before you, I will take over
15:25:55 <manjeets_> ihrachys, your patch is going to send my refactors in conflict
15:26:13 <ihrachys> manjeets_: which patch. README plus empty directory?
15:26:32 <manjeets_> yes
15:26:40 <ihrachys> nope, gerrit does not show it.
15:26:45 <manjeets_> I had created that dir in first relocation
15:26:57 <ihrachys> I guess git is smart enough to converge.
15:27:26 <ihrachys> ok, for those working on model relocation, a note on how to move them in compatible way
15:27:58 <ihrachys> HenryG came with a smart way of doing it, by using _DeprecateSubset as in https://review.openstack.org/#/c/351793/4/neutron/db/allowed_address_pairs/models.py
15:27:59 <manjeets_> i also added model dir https://review.openstack.org/#/c/348562/
15:28:05 <ihrachys> this pattern should be used for all moves.
15:29:02 <electrocucaracha> new model files are going to be in singular or plural, e.g. allowed_address_pair
15:29:05 <ihrachys> manjeets_: ok cool. if that one goes in quick, I will abandon my patch. I just wanted to set the stage quick for other patches to start popping up in paralle.
15:29:28 <manjeets_> yea relocate l3 db models is ready
15:29:29 <ihrachys> electrocucaracha: probably singular, but yeah, we should set it right away.
15:29:38 <manjeets_> its is following henry's approach
15:30:14 <ihrachys> manjeets_: one issue I see with the patch is it has lots of conflicts, 18 patches. may need to wait a while to land so that we don't break others' work in unfortunate time.
15:30:41 <ihrachys> reviewers will need to take a look at conflicting patches to see whether there is anything more critical.
15:31:13 <manjeets_> yea we ned to be careful, l3 touches so many files
15:32:07 <ihrachys> manjeets_: one thing we could do is temporarily leaving model names in consuming patches the same
15:32:30 <ihrachys> like in https://review.openstack.org/#/c/348562/15/neutron/db/external_net_db.py, we could leave the model as l3_db and then changes to the code won't be needed.
15:32:41 <ihrachys> that should reduce the number of conflicts.
15:33:53 <ihrachys> it won't help in some cases where you need to import both modules, but at least some.
15:34:10 <manjeets_> yea that was my concern
15:34:34 <manjeets_> I am trying to  keep dependency to avoid conflict
15:34:58 <manjeets_> 2 other relocations are dependent on first one
15:35:03 <ihrachys> one related thing is, now that we'll have a single place for all models, we can automatically load them in head.py as we do for objects in test_objects
15:35:12 <ihrachys> see https://review.openstack.org/#/c/351793/4/neutron/db/migration/models/head.py the bottom
15:35:34 <ihrachys> ideally tools.import_modules_recursively(os.path.dirname(models.__file__)) will eventually be the only thing in the file
15:36:23 <electrocucaracha> ihrachys:  is that for testing all the models?
15:36:39 <ihrachys> manjeets_: thanks for the work. I should get to your patches later this week once we are closer to solving the story for push-notifications
15:36:59 <manjeets_> ok thanks
15:37:02 <ihrachys> electrocucaracha: the file is used in functional tests to load all models and initialize database schema based on those models.
15:37:13 <ihrachys> electrocucaracha: then the test runs alembic scripts and compares schema
15:37:20 <ihrachys> if it's different, then the test fails.
15:37:25 <ihrachys> because those should stay in sync
15:37:45 <manjeets_> ihrachys, we had one problem with schema testing which I think henry was working on
15:38:03 <ihrachys> manjeets_: link/log would help to understand the context
15:38:04 <electrocucaracha> got it, and the idea is to reduce the number of imports in that file later, right?
15:38:32 <manjeets_> https://review.openstack.org/#/c/294183/
15:38:36 <ihrachys> electrocucaracha: yeah, we will be able to remove them from the file one by one as we move models under neutron/db/models
15:38:46 <manjeets_> we need that fix for above kind of testing ^^
15:38:51 <electrocucaracha> nice  :)
15:39:06 <ihrachys> electrocucaracha: see how we avoid tracking each file for objects: https://github.com/openstack/neutron/blob/master/neutron/tests/unit/objects/test_objects.py#L55
15:40:02 <ihrachys> manjeets_: I was thinking it's already done, at least for some backends. but maybe I don't grasp something.
15:40:42 <manjeets_> there is race condition when different test tries populates database
15:40:51 <manjeets_> with a same backend
15:40:58 <ihrachys> ok, one thing for everyone to be aware of is, kevinbenton raised a good point that we sneaked API change for plugins when we started passing objects into extension methods (thru _make_*_dict)
15:41:13 <ihrachys> previously, plugins were expecting for db models
15:41:27 <ihrachys> we will need to eventually switch to objects, but it should not happen ad hoc
15:41:45 <ihrachys> so we have a patch to revert to passing models where we incorrectly exposed objects before
15:41:46 <ihrachys> #link https://review.openstack.org/#/c/348279/
15:41:55 <ihrachys> and https://review.openstack.org/#/c/348987/
15:42:07 <ihrachys> this will need to happen till N
15:43:01 <ihrachys> finally, korzen caught an issue with subnet api tests that neutron api controller passes all filters into plugins, not just for those extensions enabled in the setup.
15:43:28 <ihrachys> we talked it thru with kevinbenton and the idea is to capture and filter out those unknown filter (pun!) on api layer
15:43:36 <ihrachys> so that plugins don't receive any random crap
15:43:53 <ihrachys> I was thinking I will find time for that the prev week. obviously I haven't :P
15:44:17 <ihrachys> if anyone aware of api layer is reading, that would be a great opportunity to step in to help with that.
15:44:31 <ihrachys> the reason why we need it is because objects are strict in the list of filters they accept.
15:44:42 <ihrachys> so we raise InvalidInput on unknown filter.
15:45:03 <ihrachys> instead of silently swallowing it
15:45:15 <ihrachys> korzen: btw does it return nothing? or just ignores somehow?
15:45:32 <korzen> nothing
15:45:53 <ihrachys> right, query builder will construct WHERE clause that won't match
15:46:04 <korzen> or sorry, it is ignoring the unknown filters returning all the resources
15:46:12 <korzen> yea
15:46:17 <ihrachys> probably the right compatible behaviour would be to return nothing from api layer right away.
15:46:33 <ihrachys> aha
15:46:36 <manjeets_> korzen, its this happening while get_object ?
15:46:53 <ihrachys> then we filter out unknown queries and that should be the same behaviour as it's now
15:46:55 <ihrachys> manjeets_: yeah, get_object(s)
15:47:01 <ihrachys> we self.validate_filters() there
15:47:21 <ihrachys> and it raises on if filter is not a field and not in extra_filter_names.
15:47:32 <korzen> https://github.com/openstack/neutron/blob/master/neutron/db/common_db_mixin.py#L203 there is the matching filter inserted into sql query
15:48:23 <ihrachys> korzen: seems like it just does nothing then
15:48:33 <korzen> ihrachys, so let me repeat, currently the API is returning full list of resources
15:48:44 <ihrachys> because no clauses (lines 209 and 220) will match
15:48:46 <korzen> when you pass invalid filter
15:49:14 <manjeets_> what do you exactly mean by full resources you mean all rows ?
15:49:21 <ihrachys> to be pedant, not full list, but list with other valid filters applied, if there are any
15:49:21 <manjeets_> korzen,
15:49:23 <korzen> 2 API calls with invalid filters will return all rows
15:49:46 <korzen> ihrachys, yes, with other filters applied :P
15:49:53 <ihrachys> :)
15:50:29 <ihrachys> ok, we are on the same page. again, if anyone is willing to step in and make history and is not scared by api router, that's the piece to bite.
15:50:38 * ihrachys looks at jschwarz ;)
15:50:53 * jschwarz looks at ihrachys
15:50:57 * jschwarz understands what is going on
15:51:11 <jschwarz> I'll be happy to help out
15:51:11 <ihrachys> haha
15:51:23 <ihrachys> jschwarz: ok, let me poke you later this week.
15:51:36 <jschwarz> sounds good
15:52:07 <ihrachys> another general topic for everyone's attention is plan in regards to dasm's keystone-v3 effort
15:52:17 <ihrachys> you probably know we rename tenant_id to project_id everywhere
15:52:36 <ihrachys> actually models have the new project_id attribute since lately
15:52:53 <ihrachys> and tenant_id is just an alias (that also triggers a warning?)
15:53:06 <korzen> ihrachys, nope, no warning
15:53:06 <manjeets_> yes HasProject instead of HasTenant
15:53:06 <dasm> ihrachys: it suppose to, but HenryG mentioned, that it doesn't.
15:53:18 <dasm> korzen: yep.
15:53:25 <ihrachys> so for places where we currently transform object's project_id into db's tenant_id, we can kill the transformation.
15:53:40 <ihrachys> korzen: ack. I guess dasm will get it to resolution ;)
15:53:54 <dasm> ihrachys: indeed. i have it on my plate.
15:54:04 <ihrachys> I started pushing some patches to convert existing objects to project_id directly
15:54:06 <ihrachys> like https://review.openstack.org/#/c/350991/
15:54:23 <ihrachys> but then armax is concerned there is no clear plan on how to get it globally.
15:54:25 <HenryG> dasm: I have a patch in progress to improve deprecation warnings
15:54:37 <dasm> HenryG: great. Thank you for your help.
15:54:54 <ihrachys> though I don't agree it should be a single patch, I guess there is some things to write up and plan for, so that other folks are aware of the direction.
15:55:14 <ihrachys> some objects are not exposed on RPC wire, which is cool since we can just rename fields
15:55:22 <ihrachys> others are, like qos policy with its tenant_id field
15:56:30 <ihrachys> so for qos, we need transition rules as well as a new version, like in https://review.openstack.org/#/c/351071/2/neutron/objects/qos/policy.py
15:57:26 <ihrachys> there is also a question on how we translate from tenant_id filter to project_id for plugin-to-objects path, and back, but that will probably need discussion with dasm and friends
15:58:23 <ihrachys> I see the time is approaching to end the meeting
15:58:35 <ihrachys> #topic Other stuff
15:58:53 <ihrachys> next meeting is on Monday, no changes in regards to midcycles
15:59:11 <ihrachys> I will travel to Cork on Tue only, so will handle the chairing.
15:59:12 <korzen> there is public Holiday in Poland next week
15:59:14 <manjeets_> ihrachys, i have devref up for review
15:59:21 <ihrachys> korzen: which day
15:59:25 <ihrachys> manjeets_: link
15:59:25 <korzen> Monday
15:59:40 <korzen> 15th
15:59:43 <manjeets_> https://review.openstack.org/#/c/351438/
15:59:57 <manjeets_> devref for relocating models ^^
16:00:11 <ihrachys> korzen: ack, we will miss you :) but we'll meet on the event, so good.
16:00:14 <korzen> I will be OOO from Friday to Tues
16:00:29 <ihrachys> manjeets_: I see, maybe that's the better patch to contain the scaffolding dir, not mine.
16:00:33 <ihrachys> manjeets_: let me review
16:00:34 <korzen> I will be in Cork Tue evening
16:00:41 <ihrachys> ack.
16:00:46 <ihrachys> ok we need to close the meeting
16:00:46 <electrocucaracha> korzen: me too
16:00:51 <manjeets_> ok thanks
16:00:53 <ihrachys> thanks everyone!
16:00:53 <ihrachys> #endmeeting