18:04:18 #startmeeting networking_policy 18:04:19 Meeting started Thu Apr 24 18:04:18 2014 UTC and is due to finish in 60 minutes. The chair is SumitNaiksatam. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:04:20 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:04:22 The meeting name has been set to 'networking_policy' 18:04:37 Hello team GBP 18:04:43 #info Group Policy wiki: https://wiki.openstack.org/wiki/Neutron/GroupPolicy 18:05:06 we were overloading the the meeting page with other design and code information 18:05:22 so, thanks to banix, we have moved those contents to the wiki 18:05:48 going forward lets use the meeting wiki for meeting specific information, and the GP wiki for all other stuff 18:05:57 banix: thanks for populating the wiki! 18:06:01 Added the pointer to neutron-specs review as well 18:06:04 sure no problem 18:06:20 banix: yes, noticed that, thats our first item 18:06:22 topic 18:06:34 #topic GP spec gerrit review 18:06:45 #link https://review.openstack.org/#/c/89469 18:06:57 hopefully everyone noticed this by now 18:07:08 SumitNaiksatam: Neat work. Thank you 18:07:11 SumitNaiksatam: Thanks! That was phenomenal work 18:07:13 thanks to the entire team to get the content ready 18:07:23 i'm a bit ignorant of the spec requirements, is there a reason to provide the python code at the end? 18:07:26 +1 to that 18:07:28 rms_13_ mandeep: thanks you guys had as much a part 18:07:29 yes great specs and also great cleaned up google doc 18:07:57 spec != implemenatation in my book 18:08:15 marun: my understanding is that the attribute map infromation can be expressed either as a table or the attribute map itself 18:08:17 the gdoc is very comprehensive 18:08:23 yuck 18:08:31 let's hope we change that and soon 18:08:50 it's way too easy to miss the forest for the trees with the current api definition mechanism 18:09:14 marun: it would have been much easier to represent in tables, if we did not have a hard requirement to put this in ascii tables 18:09:17 or maybe I'm the only one whose eyes glaze over at blocks of largely undifferentiated data? 18:09:33 marun: we can change and put in tables 18:10:01 It's fine if that's the current way of doing things. I'm just hoping that we evolve the spec requirements before too long to make them more readable. 18:10:21 marun: I added a sample table to the example spec for this sort of thing 18:10:34 makes sense and as things get reviewed we will clean up the specs 18:10:58 ok any techincal questions on the spec? 18:11:06 questions -> discussions 18:11:22 there were a good number of questions in emails; Ronak are you here? 18:11:29 Yes sir 18:11:35 marun: This development on this spec has been in progress for some time, and we are trying to retrofit what we have done into a new process. Given the history, IMO it makes sense to continue with this as it is complete and correct 18:11:35 i think getting the spec in was a good start (because earlier the criticism was that we were dealing with a google doc and not the gerrit review) 18:12:09 SumitNaiksatam: I agree 18:12:16 rms_13_ hi, was not sure about the alias. just wondering if you plan to make your comments on the review board 18:12:20 SumitNaiksatam: agreed 18:12:30 banix: ye 18:12:33 *yes 18:12:39 By end of today. 18:12:39 rms_13_: go ahead 18:12:48 as i understand it if there are shortcoming wrt presentation or substance we can address them throughout the review process 18:12:57 rms_13_: thanks 18:13:01 banix: yes 18:13:03 I apologize if this is a dumb question, but has there been any consideration as to traceability/troubleshooting? 18:13:16 I would also like to capture that somewhere in GP docs. Suggestions? Reason is that there would be multiple patchsets and the comments can get lost very easily 18:13:25 i.e. being able to trace low-level primitives to their source policy? 18:13:26 marun: i would imagine that is not specific to this feature 18:13:46 marun: ah ok, i typed over you 18:13:48 marun: Agree with Sumit. Shoudlnt that be a generic feature? 18:13:52 and vise-versa 18:13:52 one good thing is we do not have the 3rd party CIs voting on neutron-specs 18:14:07 banix: :) 18:14:09 saying it should be generic is kind of valid... 18:14:11 except 18:14:38 if this is the first abstraction of its kind, and depends on this so-called 'generic' capability....? 18:14:39 marun: yes, the current model should will allow you to trace the tie in from the new policy constructs to the classical neutron constructs 18:15:19 Is there a reason it's not included in the spec? Or would that be a future spec? 18:15:36 SumitNaiksatam: And that was an explicit goal (and constraint) for the model design 18:15:36 what is not included in the spec? 18:15:38 SumitNaiksatam: can a contract_scope wild-card contrac-id with a specic label, for eg: <*, label-secure-web-server> 18:16:09 hemanthravi: Yes 18:16:11 being able to trace the impact of a policy across primitives 18:16:16 SumitNaiksatam: ^ 18:16:24 marun: the mapping from policy to classical neutron constructs is included in the spec 18:16:28 marun: don't think we have spent much time on this beyond the fact that resources can be traces through the standard resource relationships; so nothing beyond that that need to be specified in the spec 18:16:46 so there is no way to query that data through the api? 18:17:07 i understand that those relationships have to be represented, but there doesn't seem to be any provision for their visibility 18:17:27 I would like to know if that's intentional because it will be done in the future, or whether it's not considered important. 18:17:33 marun: the current model has the mapping ... waht more did you need? 18:17:44 uh 18:17:49 ^^ 18:17:55 see 'visibility' 18:18:16 marun: The mapping extension extends the group policy extension with attributes linking the group resources to the core resources. 18:18:19 marun: #link see https://review.openstack.org/#/c/89469/5/specs/juno/group-based-policy-abstraction.rst L642 onwards 18:18:21 i'm not saying something is wrong, i'm saying I'm confused 18:18:27 marun: And that can be quried 18:18:44 you want tp querry a contract and see for example the services used? that kind of info is there 18:19:18 what about the reverse? 18:19:28 query a port and know what policies are affecting it 18:20:11 marun: a port belongs to an EP and an EP has policies applied on it via an EPG, you can query all that. 18:20:13 I think I liked the google doc better :/ 18:20:38 marun: (not belongs - associated with) 18:21:56 marun: And you definitely do not want to duplicate that information in a second resource in the model - that will lead to other problems ;-) 18:22:04 I'm not suggesting duplication 18:22:08 marun: I think you’d be able to query to find the EP associated with a given port ID, and navigate from their. 18:22:36 rkukura: yes, that is correct 18:22:42 anything beyond the above, then can be built on top of the basic model. Special client side commands maybe? 18:22:49 from EP you can get to EPG, contract, etc 18:22:58 banix: yes agree 18:23:02 SumitNaiksatam: do you need to add a extended attr on port for that? 18:23:02 marun: In that case I misunderstood. As you can query that information as Port -> EP -> EPG -> Contract -> Policy rule 18:23:25 ah, so visiblity is on primitives only 18:23:29 fair enough 18:23:30 marun: perhaps you are asking if there is an easy way to do the above ^^^ 18:23:49 SumitNaiksatam: I think so 18:23:49 marun: are you suggesting we have cli commands that do provide the info in a single call or something along that line? 18:23:50 marun: and that can probably added as a follow up, if it adds avalue 18:24:00 marun: The assumption being that the tools/code can do the appropriate level of resolution (as needed) 18:24:14 hemanthravi: extended attribute to do what? 18:24:30 I think that's a bad assumption from a performance perspective, but as SumitNaiksatam says optimization can happen where needed 18:24:35 to find the ep associated with a port 18:24:55 marun: I think that belongs to the driver / renderer (for performance) 18:25:06 marun: I agree. There are many solution for optimization - but first we need to get our data model correct. 18:25:23 SumitNaiksatam: to find the ep associated with a port 18:25:36 hemanthravi: you should be able to query the endpoints and filter on the port_id 18:25:40 hemanthravi: that is there 18:25:57 hemanthravi: I don't think it is appropriate to add EP information on a generic neutron port object 18:25:58 SumitNaiksatam: ok 18:25:59 hemanthravi: the way SumitNaiksatam suggests above 18:26:17 s3wong: yes, we are trying to avoid that 18:26:46 SumitNaiksatam: unlike security-groups? 18:27:04 hemanthravi: yes, unlike security-groups 18:27:11 s3wong: agree; the question becomes if a querry EPs and filtering become prohibitively costly in large systems but may be not to worry about now 18:27:17 The get API does have filters for this purpose, and the SQL queries shoudn’t be that bad performance-wise 18:27:30 rkukura banix: agree 18:27:36 marun: One of the drivers of the group policy is the scale. And by dealing at larger groupos we make the system a lot more efficient for normal use-case. Do not read the the query chain that I showed you as the normal use case. That is possible, but the implementation at aggregate levels can be far more effecient 18:27:41 banix: agreed 18:27:52 marun hemanthravi: good points to bring up! 18:28:11 gets our thinking oriented towards that goal as well 18:28:16 thanks for bringing it up 18:28:23 mandeep: I'll believe the assertion about scale when I see the scale testing ;) 18:28:32 marun: ;-) 18:28:45 thats a good segue 18:28:58 #topic Functional tests 18:29:07 not quite scale tests 18:29:19 and we dont even have the funcationality implemented :-) 18:29:33 SumitNaiksatam: long term thinking? :) 18:29:34 heh 18:29:36 Perfect time to do functional testing 18:29:38 but i wanted to bounce this off, to get started on this 18:29:48 I'm going to be pushing for test plan inclusion in specs at summit 18:30:05 marun: ok 18:30:08 rms_13_ seemed to be interested in taking this up for GP 18:30:10 We'll need some education and framework to make it work, of course. Not everyone has a QA background. 18:30:18 rms_13_: +1 18:30:24 marun: yes, we are definitely looking forward to you for that 18:30:36 forward -> up 18:30:44 not me, but we have some QA folks here at Red Hat that have offered to bootstrap the effort 18:30:54 marun: ok you can facilitate 18:31:04 or whoever 18:31:17 marun: Is there some specific framework that is being suggested? 18:31:18 rms_13_: you still interested in looking at this? 18:31:20 There is a place that folks here could start at on their own, though 18:31:27 SumitNaiksatam: Sure 18:31:34 rms_13_: ok 18:31:35 I can take a look 18:31:36 thanks 18:31:37 mandeep: framework in terms of 'how to write a good test plan' instructions and templates 18:31:39 rms_13_: thanks!!! 18:31:44 not a runtime framework 18:31:53 anyone else interested, please let us know as well 18:32:04 marun: Got it 18:32:15 the starting point would be 'what's the simplest operational test that we could write against the implemented spec' 18:32:17 marun: are we planning to have a framework? 18:32:36 SumitNaiksatam: For functional testing, it will be stuff we write to simulate whatever we need 18:32:44 SumitNaiksatam: for integration testing, as per usual, Tempest 18:32:48 marun: i dont have an opinion one way or the other, but i just want to know what are our dependencies 18:32:54 marun: How would you proceed on writing ft for feature like this? 18:33:15 rms_13_: The starting point is functional api tests 18:33:29 rms_13_: ensuring that the api spec works in practice 18:33:36 rms_13_: as simple as 'create resource, resource is created' 18:33:43 rms_13_: delete resource, resource is deleted 18:34:00 rms_13_: i.e. not actually checking that any network state changes happen 18:34:04 marun: you have a patch for retargetable functional tests 18:34:24 marun: for the benefit of everyone here, can you summarize what it is meant for? 18:34:27 marun: got it. some of these we might already have as part of ut. Dumb question but can we inherit those? 18:34:36 SumitNaiksatam: I do. That's a good starting point for api tests. I'll aim to have that merged before summit, but it can be the basis of efforts pre-merge if folks are eager 18:34:47 SumitNaiksatam: sure 18:35:00 marun: thx. will take a look 18:35:22 So, retargetable functional api tests are written against an abstraction of the neutron api 18:35:23 marun: Is there a gerrit for for that? 18:35:41 (gerrit review that is ;) 18:35:41 #link https://review.openstack.org/#/c/72585/ 18:35:43 the abstraction ensures that we can target both the python api (no running service required) and a running service 18:35:44 marun:any document for this 18:36:07 the idea is that it's easier to develop a test against an api 18:36:23 and then nice not to have to repeat the work to test a live deployment 18:36:58 there is an example of a retargetable test a the bottom of this file: https://review.openstack.org/#/c/72585/12/neutron/tests/api/base_v2.py 18:37:39 it uses a 'client' attribute that is provided at runtime, and in this patch can be either targeting the plugin api or a rest client that tempest defines 18:37:44 but the test is the same in either case 18:38:02 the specifics are a bit tricky, but hopefully the test itself is straightforward to everyone 18:38:13 I don't think an understanding of all the machinery will be necessary to make use of it 18:38:19 marun: thanks. Ya it is. 18:38:35 marun: okay, i think it will take some time for people to absorb, and we can hopefully get back to you with questions 18:39:13 SumitNaiksatam: sounds good. I'll also be working on docs and instructions to present in a summit session. 18:39:20 marun: nice 18:39:44 since this is a large feature (GP), certain things will land before others 18:40:07 if, as a community, we can converge on the object model, it makes it easy to make progress 18:40:15 easy -> easier 18:40:29 #topic PoC Status Update 18:40:49 on the model side, i made some incremental progress 18:41:01 but the model was evolving based on the feedback 18:41:19 rkukura helped me with some of the UTs 18:41:28 so at this point, EPG and EP is in there 18:41:37 along with the resource definition of everything else 18:42:28 banix has also been looking at the db schema and reviewing what i have been putting 18:42:57 if no questions for me or banix, then i can hand over to rkukura s3wong for the driver update 18:43:05 rkukura: ? 18:43:08 OK 18:43:19 SumitNaiksatam: Thanks. I have expressed some comments/suggestions on that via email. That is my start of doing review at the noiro branch. 18:43:38 rms_13_: sure, you can comment on the gerrit review 18:43:44 rms_13_: thanks, btw 18:43:46 I’m working now on bridge domain, since that maps to neutron network. 18:44:20 So no real progress on the mapping driver itself yet, but should be getting their soon 18:44:53 rkukura: ok thanks 18:45:04 One question is whether the mappings to network, subnet, and port should be established immediately when resources are created, or deferred until they are actually needed? 18:45:06 I started looking at rkukura 's code under rkukura/mapping, but can't make much progress until the service model is defined 18:45:22 s3wong: i agree 18:45:53 s3wong: Yes, so far I’ve really just been helping SumitNaiksatam with the API, model, and UTs 18:46:29 SumitNaiksatam, rkukura: Is the mapping mandatory? 18:46:44 rms_13_: its a part of the reference implementation 18:47:00 but its an extension to the policy model itself 18:47:12 so in theory, you could do your own mapping 18:47:32 rkukura: perhaps a late binding 18:47:36 rms_13_: In theory, you could have a pure GroupPolicy implementation that doesn’t provide the mapping extension. 18:47:37 got it 18:47:38 yes for PoC definitely need the mapping 18:47:43 rkukura: when they are actually needed? meaning that only when groups are attached to a contract? 18:47:56 rkukura: but i am thinking that for the PoC it might be easier to do an immediate (default) binding 18:48:07 SumitNaiksatam: +1 18:48:35 s3wong: i agree, policy construct binding is late 18:48:48 i think rkukura is asking the mapping to classical neutron 18:48:52 Immediate is probably less work, but might not be too difficult to defer mapping to network and subnet until the Endpoint is created and a neutron port has to be created 18:49:06 rkukura: agree 18:49:17 rkukura: yes 18:50:27 hemanthravi: this is an unfair question, but i will ask nevertheless - any update on the client, CLI? :-) 18:50:35 Will it work though in scaled environment? Lets say you try to boot 50 VMs together. You are getting 50 create_port(). You start creating network and subnet on 1st. Other 49 needs to wait. 18:50:50 SumitNaiksatam: pushed a branch with inital commit for ep, epg 18:50:58 hemanthravi: oh sweet 18:51:07 hemanthravi: as long as you have a good handle on it 18:51:14 hemanthravi: nice 18:51:23 SumitNaiksatam: create/delete should are done...update wip 18:51:36 hemanthravi: thanks! 18:51:38 rms_13_: Good point, but the trade-off is that once mapped to neutron ports, pretty much nothing is mutable. 18:51:47 hemanthravi: the model might change a bit as we receive feedback, so be prepared (i know you dont like me already :-P) 18:52:14 rkukura: yes, nothing is mutable with the classical neutron mapping, hence late binding is better 18:52:24 binding -> mapping 18:52:25 SumitNaiksatam: do realize that...but should be easy to make the changes..once code for the existign model is in 18:52:33 hemanthravi: sweet 18:52:51 hemanthravi: what about Horizon, you said you would check with prasadv? 18:52:57 rkukura: correct. The more I think the more late binding becomes complex. To give it enough thought, should we start with immediate mapping till summit for PoC? 18:52:58 oh prasadv is also around 18:53:16 rms_13_: yes we need to balance it out 18:53:32 hemanthravi prasadv: any thoughts on the Horizon work? 18:53:33 SumitNaiksatam: i thought we commited to Heat resources 18:53:51 prasadv: ah ok, i was trying my luck 18:53:57 :) 18:54:00 prasadv: Heat is next on the agenda 18:54:04 rms_13_: I’ll start with the path of least resistence ;) 18:54:07 prasadv: GREAT! 18:54:10 I have not done yet 18:54:22 prasadv: do you think someone like Subra might have time to look at Horizon? 18:54:24 rkukura: make sense 18:54:56 SumitNaiksatam: when does horizon work need to start. I guess CLI need to be done right 18:55:13 prasadv: yes client needs to be ready first 18:55:25 prasadv: then CLI (though both go hand in hand) 18:55:29 i mean horizon calls CLI o the back end 18:55:30 prasadv: horizon after that 18:55:44 prasadv: but i just wanted to get some folks thinking about that as well 18:55:55 prasadv: else it will slip through the cracks 18:56:08 I’d appreciate feedback on some comments I posted in https://review.openstack.org/#/c/89469/ regarding navigability between EPGs, BDs, and RDs. 18:56:09 anyone else here interested in pitching in on the horizon work? 18:56:16 A good UI model for this is going to be challenging. But we can get started with something simple 18:56:16 rkukura: sure 18:56:20 SumitNaiksatam: I understand. Am crunched for resrouces here. hence hesitating 18:56:42 prasadv: absolutely, and please dont misunderstand, dont mean to put you on the spot 18:56:47 prasadv: We get that. No problem 18:56:49 I will get back toyou end of day today ok? 18:57:00 prasadv: like i said, i am just trying my luck :-) 18:57:04 prasadv: no worries 18:57:15 Heat is certainly the more important piece in comparison with Horizon 18:57:16 we will get started on heat 18:57:20 if we dont have an owner for this, all of us will need to sit in a room, and do this 18:57:28 else we cant get the feature out 18:57:46 prasadv: thanks 18:57:59 SumitNaiksatam: are you thinking to have it done by the summit? 18:58:04 SumitNaiksatam: I agree. I will take the responsibility to get some resource on it 18:58:06 SumitNaiksatam: horizon piece 18:58:14 mandeep: ok, thanks 18:58:28 same thing goes for functional tests and tempest 18:58:37 if we cant do those, we cant get the feature out 18:58:55 banix: yes, for the summit 18:59:03 I will look into FT 18:59:18 Do you see FT completed before summit? 18:59:39 SumitNaiksatam: what level of resource is needed for FT? 18:59:41 rms_13_: thanks 18:59:50 Given that the unit tests as written are largely functional.... 18:59:50 rms_13_ prasadv: its not clear 19:00:04 They could be rewritten without too much effort. 19:00:07 banix: let’s say its highly desirable 19:00:22 SumitNaiksatam -> PLM 19:00:29 :) 19:00:43 rms_13_: ha 19:00:59 rms_13_: actually the community is the PLM here 19:01:15 oh we are already out of time 19:01:16 cool. are we over time? 19:01:18 rms_13_: :-) 19:01:24 since we have not yet been kicked 19:01:31 couple of other items 19:01:38 #topic Summit 19:01:48 our design summit session was accepted 19:01:54 Yay 19:02:00 Cool 19:02:03 we have a slot on thursday at around noon i believe 19:02:09 so we will get 40 mins 19:02:18 much of what we discuss here goes to discussions there 19:02:33 regading the conference presentation slot (also on thursday) 19:02:41 banix was leading that charge 19:02:44 banix: any updates? 19:02:45 working on the slide; should have a first cut soon 19:02:52 banix: thanks 19:03:00 s3wong: did you get a chance to think about it? 19:03:03 banix: thanks! 19:03:04 checking the time for the general session talk; i think it is Thursday afternoon 19:03:10 banix: yes 19:03:27 SumitNaiksatam: thinking about the presentation? only we need to demo something :-) 19:03:57 s3wong: if we have it ready :-) 19:03:59 we do not *have to* have a demo but having it would be great 19:04:08 s3wong: i think that was our original goal 19:04:09 stage intended behaviour in a video ;) 19:04:20 marun: that is a good idea 19:04:31 lets keep it as a goal 19:04:31 i think we should have the demo recorded 19:04:32 SumitNaiksatam: and as banix wrote the original abstract, we have to have a story on service 19:04:37 banix: agree 19:04:38 We do need the PoC purely as a spike thru the design and validate that it "hangs well together" ;-) 19:05:02 I need to run folks. Sorry. See you all at next meeting. 19:05:06 s3wong: agree 19:05:11 rms_13_: thaks 19:05:18 alrighty, anything else folks? 19:05:24 we are well over time 19:05:34 s3wong: yes i think that will be more interseting imho 19:05:43 s3wong: we need another services discussion :-) 19:05:59 SumitNaiksatam: I think that is a given :-) 19:06:06 ok thanks everyone! 19:06:11 #endmeeting