17:00:49 <arosen> #startmeeting CongressTeamMeeting
17:00:50 <openstack> Meeting started Tue Nov 11 17:00:49 2014 UTC and is due to finish in 60 minutes.  The chair is arosen. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:54 <openstack> The meeting name has been set to 'congressteammeeting'
17:01:03 <arosen> Hi, pballand and tim  are out today so I'm going to be running todays IRC meeting.
17:01:09 * arosen FYI: I'm talking this meeting from the train so I might get disconnected a few times :/
17:01:29 <arosen> Hope everyone had a good time at the sumit and a good trip back home.
17:01:34 <jasonsb> hello sir
17:01:43 <arosen> hi jasonsb
17:01:45 <jasonsb> just made the flight
17:02:15 <arosen> anyone else around yet?
17:02:36 <arosen> I think daylights savings time might have thrown a few people off since we're meeting an hour earlier now.
17:02:39 <jasonsb> did sunny say he was going to join?
17:03:08 <arosen> not sure
17:03:22 <arosen> anyways: Here is todays meeting agenda:
17:03:30 <arosen> #link https://wiki.openstack.org/wiki/Meetings/Congress
17:03:48 <arosen> Here's the etherpad from the summit. We had lots of good discussion there
17:03:53 <arosen> #link https://etherpad.openstack.org/p/par-kilo-congress-design-session
17:04:10 <arosen> i guess we can wait a few minutes and see if others join in.
17:05:39 <arosen> Does anyone have anything they want to discuss at this point or give a status update?
17:06:30 <arosen> I guess I can go first :)
17:07:00 <jasonsb> the floor is yours
17:07:01 <arosen> At the sumit the Murano group was pretty interested in integrating with congress so yesterday I spent some time getting murano up and running.
17:07:38 <arosen> they also have devstack integration simlar to how congress does with devstack so that made it easy to set up
17:07:59 <arosen> Still in the process of reading though their docs to try and figure out how it use it though. Hopefully I'll figure more things out this week.
17:08:28 <arosen> We also merged the oslo.db patches into congress yesterday so congress now has a persistence layer  :)
17:08:28 <jasonsb> congress as a store of murano acl's?
17:08:58 <arosen> jasonsb: I think the first step would be to implement a datasource driver for murano
17:09:16 <arosen> then there was talk of murano consuming the congress api some how to do proactive enforcement.
17:09:54 <jasonsb> i like the idea
17:10:12 <arosen> I believe the murano team is going to help us out with this integration
17:10:22 <jasonsb> i wonder if you made congress pluggable then it would be easy sell
17:10:29 <arosen> jasonsb: me too. it will be existing to see something consume the congress api :)
17:10:37 <jasonsb> (if congress not installed then the check gets nooped)
17:11:08 <arosen> jasonsb:  I agree.
17:12:06 <arosen> i think it would be nice if we could provide some easy integration hooks for murano to do their policy checks. I wonder if the datasource driver could expose some additional hooks to make things easier.
17:12:58 <jasonsb> this is probably naive
17:13:06 <jasonsb> but its nice to have a use-case which you can kind of fit in your head
17:13:08 <arosen> currently though I'm just sorting through how their api and murano works.
17:13:13 <jasonsb> GBP isn't one of those for me
17:13:34 <jasonsb> or QoS for that matter
17:13:38 <arosen> jasonsb:  do you have any blueprints you're interested in contributing to this release?
17:14:02 <jasonsb> I do
17:14:15 <arosen> jasonsb:  do you wanna talk about those now?
17:14:21 <jasonsb> but i'm currently trying to wrap my head around how it might work
17:15:07 <jasonsb> I guess UI is integral
17:15:15 <arosen> what type of thing are you trying to expose?
17:15:26 <jasonsb> as well as workflow.  in my version of QoS there is interaction with user
17:15:40 <arosen> jasonsb:  currently there is some horizon integration in the tree in contrib/horizon. the devstack script handles setting that up if you wanna try it out.
17:15:56 <arosen> jasonsb:  what type of qos? Network?
17:16:14 <jasonsb> actually, if you are still giving status
17:16:22 <jasonsb> i'm very curious about ceilometer meeting
17:16:22 <arosen> nope i'm done.
17:16:51 <jasonsb> (2pm friday?)
17:17:02 <arosen> Sorry I'm not sure off hand when it is.
17:17:24 <arosen> Some of us did talk to the ceilometer team at the openstack summit and they had some good insight for us.
17:17:26 <jasonsb> oh, i thought all of you had a meeting with them
17:17:40 <arosen> ah on friday at the sumit you mean?
17:17:43 <jasonsb> yes
17:17:48 <arosen> ah okay :)
17:18:11 <arosen> cool, so one of the use cases we presented them with was trying to find idle vms that and been idle for the last month
17:18:58 <arosen> they said unfortunately in the current standing of ceilometer it was really able to provide us they level of historical data currently (or could put would be pretty slow and consume a lot of space on their back end to do it)
17:19:24 <jasonsb> able or unable?
17:19:33 <arosen> they said they are much better at providing this type of data for 30 minute periods which most people use for auto scaling.
17:19:39 <arosen> *unable
17:20:04 <arosen> also they said that they only persist the data for status only if they have a specific alarm already setup for it
17:20:36 <arosen> so i don't think we're able to get historical data unless a meter was configured to save that data i believe.
17:20:54 <jasonsb> sounds like a nice value-add
17:20:57 <arosen> they said they are working on a new backend that's back by swift that should help solve these types of issues.
17:21:18 <jasonsb> things which congress is interested in get saved
17:21:40 <arosen> they also gave us some suggestions on how we could go about implementing this use case using their current implementation of ceilometer
17:22:09 <arosen> basically having congress poll ceilometer and doing the saving of the data so we could then query over it later.
17:22:34 <arosen> though I don't think this is really a good solution for us as it will make the datasource driver a lot more complex
17:23:14 <arosen> and it will also put us in the business of storing this  data which i don't think we wnat to do at this point
17:23:25 <jasonsb> i wonder if you will have to save data for other use cases
17:23:41 <jasonsb> sounds like compliance history would have to be saved somewhere
17:24:01 <jasonsb> unless it itself was a ceilometer source
17:24:09 <arosen> jasonsb: I think we probably will but the ceilometer data seems like an awful large amount of data.
17:24:30 <jasonsb> agreed
17:24:33 <arosen> also the ceilometer team is already working on fixing these issues in ceilometer so i think it might make more sense to just wait in the meantime.
17:24:48 <arosen> and also help them out as well if we get some spare cycles
17:25:20 <arosen> it would probably be good for us to be involved there since we'll be consuming their apis and data when they're ready.
17:26:00 <jasonsb> did you discuss at what rate ceilometer data could be pulled?
17:26:01 <arosen> jasonsb:  did you want to talk about your QoS blueprint?
17:26:09 <jasonsb> for my QoS use case it might be pretty quick
17:26:23 <arosen> jasonsb:  i think they said they had alarms which we could leverage to avoid having to do a lot of polling.
17:26:39 <jasonsb> ok, i'll check into the details
17:26:53 <jasonsb> sure, I have a couple of questions regarding QoS
17:27:03 <arosen> jasonsb:  the meeting was a lot about describing what congress was and getting large time series of data from cielometer.
17:27:18 <arosen> we'll definitely want to follow up with them again :)
17:27:28 <arosen> shoot
17:27:51 <jasonsb> So there is a UI component.  namely, how does the user specify QoS parameters
17:28:04 <jasonsb> in my specific case it is storage related #'s
17:28:31 <arosen> jasonsb:  In datalog that congress uses?
17:28:38 <arosen> as in as part of the policy?
17:28:49 <jasonsb> exactly
17:29:06 <jasonsb> my hope is that it would not be raw datalog
17:29:14 <jasonsb> but thats just a hope
17:29:40 <arosen> jasonsb:  to be honest I think Tim is the best person to ask. I believe there is some concept of > <= there. I think i remember seeing some policy that said ceilmeter:statistics(load>1)
17:29:57 <jasonsb> if i understand congress intent on UI then i'm aligned with that
17:30:09 <jasonsb> ok, my question is
17:30:43 <jasonsb> given some datalog snippit which relates to QoS, is it within congress philosophy to allow the enforcement
17:30:58 <jasonsb> engine to give a thumbs up or thumbs down vote if it will accept?
17:31:20 <jasonsb> ie: is a round-trip back to user possible
17:31:45 <arosen> jasonsb:  okay so you're saying will congress expose the functionality to ask if a specific thing is out of policy before it does it?
17:32:10 <arosen> so the user can ask if this is out of policy?
17:32:21 <jasonsb> will congress expose functionality which makes sure both user and enforcement agree to the contract before it is set in stone
17:32:36 <jasonsb> only then does it become policy
17:32:58 <jasonsb> basically, for QoS, is the user asking something which can physically be done?
17:32:58 <arosen> jasonsb:  i see. This is a good idea though I don't think we've talked much in the past about this type of contract.
17:33:02 <jasonsb> if yes, then agree to the terms
17:33:05 <jasonsb> if not then complain
17:33:39 <jasonsb> it might be more complication than you want
17:33:58 <jasonsb> but it would be a way for central ISO to delegate some things to department level for instance
17:34:07 <jasonsb> so maybe its a useful pattern?
17:35:31 <jasonsb> i think this is the only thing which i can foresee needing which congress isn't already going to do (from what i understand)
17:36:08 <jasonsb> i can probably get around it by throwing an error of (policy not accepted)
17:36:11 <arosen1> jasonsb:  sorry got disconnected *
17:36:15 <jasonsb> and let the user fix it until i agree to it
17:36:45 <jasonsb> np
17:36:54 <arosen1> jasonsb: cool, yea i think that works
17:37:03 <jasonsb> which version?
17:37:15 <arosen1> ingeneral i think we'll have to deal with this same type of thing for other sources
17:37:26 <jasonsb> or is it the same?
17:37:43 <arosen1> sorry i might have missed a messsage what do you mean by which version?
17:38:20 <jasonsb> i guess there isn't much difference
17:38:51 <jasonsb> version 1: policy engine does not accept policy so user is requested to make changes
17:38:55 <jasonsb> until policy engine accepts
17:39:26 <jasonsb> version 2: user specifies policy which policy engine does not accept
17:39:49 <jasonsb> policy engine throws an exception which is shown to user and user can fix their parameters
17:40:38 <arosen1> jasonsb:  i see.
17:40:41 <jasonsb> the only real distinction i am making between the two is in version 1, the policy is not accepted and saved in persistent store until both
17:40:48 <jasonsb> enforcement and user agree to it
17:41:06 <arosen1> jasonsb:  and  by user you more mean the backend system that is doing the enforcement?
17:41:15 <arosen1> as in if the enforcement is possible.
17:41:31 <jasonsb> user is human here
17:41:41 <jasonsb> (or a proxy for a human)
17:42:32 <jasonsb> engine==enforcement
17:42:36 <jasonsb> sorry for sloppy language
17:42:40 <arosen1> so in your example the human would want to accept if they want this policy enforced on them?
17:43:09 <jasonsb> in my use-case human is driving the policy
17:43:16 <arosen1> jasonsb:  sorry mind giving a quick example of the work flow for this?
17:43:18 <jasonsb> she is asking for a particular QoS parameter
17:43:26 <arosen1> i think i missed a few lines of the chat when i disconnected.
17:43:29 <jasonsb> those parameters are being imposed on the enforcement
17:44:02 <arosen1> jasonsb:  would she be asking congress first or the system that is providing the function that qos is applied to ?
17:44:04 <jasonsb> user specifies max of 10ms latency for IO
17:44:17 <jasonsb> (round trip time)
17:44:46 <jasonsb> asking congress first
17:44:53 <arosen1> k
17:45:05 <jasonsb> enforcement gets an up or down vote
17:45:13 <jasonsb> is it possible to meet 10ms?
17:45:28 <jasonsb> if yes, then QoS policy is accepted and becomes persistent
17:45:30 <arosen1> jasonsb:  gotcha i see what you're saying.
17:45:49 <jasonsb> so its policy imposed by user on system
17:45:53 <jasonsb> not the other way round
17:46:10 <jasonsb> I think data retention policies would be similar
17:46:44 <arosen1> jasonsb:  i see. Hrm this sounds kinda tricky to implement. I guess the thing that congress is talking to would have to expose an api to know what type of quota or guarantees are available to do this though.
17:47:21 <arosen1> i'm wondering how congress is able to learn that 10ms is to low of a number or not.
17:47:34 <jasonsb> i think congress would call enforcement
17:47:47 <jasonsb> accept_policy_yes_or_not()
17:48:12 <arosen1> jasonsb:  though congress would only want to accept it if the other back end system can implement this QoS policy right?
17:48:13 <jasonsb> if yes then save in persistent store
17:48:23 <jasonsb> yes
17:48:45 <jasonsb> you could use it to make sure the policy made sense perhaps
17:48:59 <jasonsb> and only the enforcement code knew for certain
17:49:04 <arosen1> right -- so that's the part that i'm wondering. How would congress learn this information to do enforcement.
17:49:22 <arosen1> the datasource doing QoS would have to expose whats allowed
17:49:29 <arosen1> if it does that i think congress should totally do this.
17:49:36 <jasonsb> i think only enforcement knows
17:49:56 <arosen1> jasonsb:  and by enforcement you mean the thing doing the QoS? or?
17:50:02 <jasonsb> yes
17:50:12 <jasonsb> the code which is watching and making adjustments
17:50:16 <jasonsb> (if necessary)
17:50:30 <arosen1> gotcha
17:50:35 <jasonsb> in my use-case it plays an active role in making sure system is in compliance
17:50:45 <arosen1> do you have a system that exposes QoS that you want to integrate with congress
17:51:04 <arosen1> unfortunately  neutron doesn't really have one today besides the nsx plugin
17:51:39 <arosen1> which just exposes queues which allows you to cap throughput on neutron ports.
17:52:06 <jasonsb> mine is greenfield
17:52:13 <arosen1> jasonsb:  there has been some work to have a reference qos api in neutron but i don't think it's made much progress
17:52:18 <jasonsb> so i'm searching for best match for what congress wants to do
17:53:44 <jasonsb> maybe this is a way out of difficulty on GBP
17:53:58 <jasonsb> in GBP meeting there was potential for conflicting policies
17:54:30 <jasonsb> maybe with transactions such as this, if all of the policies (expresssed in datalog) pass the conflict test
17:54:37 <jasonsb> then they get baked in stone
17:54:44 <jasonsb> otherwise, round trip to user
17:54:52 <jasonsb> (or proxy for user)
17:56:30 <arosen1> jasonsb: I think i understand. It's more if you're writing a policy that says , "I only want this deployed if the backend can give me exactly this"
17:56:49 <jasonsb> yes
17:56:52 <jasonsb> and also
17:57:18 <jasonsb> i only want this deployed if backend parses and agrees that it is sane
17:57:24 <jasonsb> (no conflicts etc)
17:57:59 <jasonsb> eventually i think this would end up someplace in a heat stack or something
17:58:19 <jasonsb> where the heat job would not be able to start if the QoS policy couldn't be met
17:58:29 <jasonsb> (unless best-effort or something was turned on)
17:58:55 <jasonsb> the desired result would be hadoop job which completed in predictable amount of time if the job was run
17:59:05 <arosen1> jasonsb:  my thought is that one would make the request to provision some QoS amount then that system would call back to congress to see if this QoS amount was out of policy
17:59:33 <arosen1> if the QoS amount couldn't be implemented by that system it would just return an error before calling to congress.
17:59:41 <arosen1> This is how heat works today.
17:59:56 * glebo sorry I missed today. Failed to alter calendar item to adjust for daylight savings clock change relative to UTC meeting time
18:00:01 <arosen1> if it tries to deploy a template and you don't have enough quota it will fail once it gets an over quota error.
18:00:12 <arosen1> glebo: no worries
18:00:34 <arosen1> jasonsb:  we sorta  of running out of time but I believe i see what you're saying.
18:00:44 * glebo thankful there is irc log ;-)
18:00:48 <jasonsb> i didn't quite get your version
18:00:51 <jasonsb> #congress?
18:01:00 <arosen1> i think congress should also have some way to validate policy that is told to it to confirm that it's even a valid policy
18:01:02 <arosen1> jasonsb: sure
18:01:06 <arosen1> #endmeeting