14:00:14 <edleafe> #startmeeting nova_scheduler
14:00:14 <openstack> Meeting started Mon May 23 14:00:14 2016 UTC and is due to finish in 60 minutes.  The chair is edleafe. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:18 <openstack> The meeting name has been set to 'nova_scheduler'
14:00:21 <edleafe> Anyone around this morning?
14:00:22 <claudiub> o/
14:00:24 <mlavalle> o/
14:00:26 <cdent> edleafe: is it memorial day?
14:00:27 <rlrossit> o/
14:00:31 <takashin> o/
14:00:35 <Yingxin> o/
14:00:36 <edleafe> cdent: next monday
14:00:36 <rlrossit> cdent: that's next week
14:00:36 <sudipto> o/
14:00:49 <cdent> ah, of course
14:01:00 * cdent can't do calendars or math
14:01:45 <edleafe> let's give 'em another minute or so
14:01:58 <bauzas> tick tack
14:02:30 <edleafe> #topic Specs
14:02:39 <edleafe> First up is: Improve Scheduler Logging - https://review.openstack.org/#/c/306647/
14:02:45 <edleafe> SOme discussion on this one
14:02:57 <jaypipes> o
14:03:03 <edleafe> The idea was to keep it simple at first, and then add complexity if needed
14:03:09 * jaypipes bleary eyed on PST timezone
14:03:15 <edleafe> poor jaypipes
14:03:36 <edleafe> Anyone have anything to add on that spec?
14:03:42 <johnthetubaguy> so I think we all agree with the need for the logging on error, which is good
14:03:58 <edleafe> Nit: those aren't errors
14:04:15 * cdent gives edleafe a nickel
14:04:16 <johnthetubaguy> true, scheduler fails
14:04:49 <edleafe> So when NoValidHost is the result, we log every scheduler decision that led up to it
14:04:54 <johnthetubaguy> what do folks think here, the simple approach, would that be enough?
14:05:02 <bauzas> so, MHO is that we should approve that spec and discuss on the implementation by the series reviews
14:05:17 <bauzas> given it's like a top prio for operators
14:05:31 <edleafe> With no option to turn that reporting on if a host is selected, but not the one that the operator expected
14:06:08 <johnthetubaguy> edleafe: so if you turn on full debug, I think you would get that, but obviously at a cost
14:06:36 <edleafe> johnthetubaguy: that wasn't my understanding
14:06:56 <edleafe> johnthetubaguy: the logs would be cached, and only emitted if NoValidHost
14:07:18 <johnthetubaguy> right, I guess I was assuming it wrapped the existing logs, not a new system
14:07:20 <bauzas> what I'd like is to make sure we have something in Newton for providing an accumaled failures log
14:07:24 <edleafe> Full debug won't record every scheduler decision
14:07:29 <johnthetubaguy> edleafe: thats my main concern
14:07:32 <johnthetubaguy> oops
14:07:40 <johnthetubaguy> I mean bauzas: thats my main concern
14:08:11 <bauzas> Matt proposed to split the implementation in two phases, one for Newton and one for Ocata
14:08:20 <edleafe> OK, so let's push this spec forward, and we can always add to the logging in a later change
14:08:24 <bauzas> I tend to agree with that, but I was more thinking of having a general spec
14:08:31 <johnthetubaguy> I guess we wait till mirdem is online, and resolve the questions
14:08:47 <bauzas> johnthetubaguy: if so, we could miss the deadline :/
14:09:16 <bauzas> johnthetubaguy: I think I'm okay with matt, there are things we can possibly do now and some others to be done later
14:09:34 <edleafe> Is there a need to hold the spec approval back?
14:09:53 <edleafe> (for the simplest case)
14:10:03 <johnthetubaguy> what I am saying, is we don't have the people here to resolve the discussion, so lets follow up with them when they are on line
14:10:06 <bauzas> or, we could just clarify what we expect
14:10:23 <bauzas> johnthetubaguy: yeah, agreed
14:10:39 <rlrossit> johnthetubaguy: remember mriedem is out for the week
14:10:49 <edleafe> ok, so we'll table this until matt is back from holiday
14:10:52 <johnthetubaguy> rlrossit: ah, I totally forgot that, oops
14:11:02 <johnthetubaguy> so waiting a week doesn't seem overly wise
14:11:18 <johnthetubaguy> can we merge the basic spec instead?
14:11:25 <bauzas> johnthetubaguy: hence my proposal to accept the spec with some clear comment that the implementation phase should be a bit more discussed  ?
14:11:32 <edleafe> uh, that's what bauzas and I have been asking :)
14:11:42 <bauzas> we could lately revise the spec
14:11:46 <johnthetubaguy> well, thats not the same thing here
14:11:56 <edleafe> or make a follow-up spec
14:11:59 <johnthetubaguy> we could revise the spec to be more vague, I would happy merge that
14:12:25 <edleafe> I'd really like to see the implementation and the load on memory
14:12:35 <johnthetubaguy> edleafe: yup, I think we all would
14:12:36 <bauzas> johnthetubaguy: that looks an acceptable trade-off :)
14:13:03 <johnthetubaguy> who is going to chase up on getting that done?
14:13:05 <bauzas> asking cfriesen to provide a new PS just for clarifying that
14:13:09 <johnthetubaguy> the spec revision that is?
14:13:17 <bauzas> I could
14:13:27 <edleafe> There will be detailed logging for every request stored in memory until it completes
14:13:31 <johnthetubaguy> bauzas: awesome, ping me when the new thing is up
14:14:00 <edleafe> #action bauzas to ping cfriesen on a new PS for the scheduler logging spec
14:14:13 <bauzas> is cfriesen on PST ?
14:14:21 <edleafe> no idea
14:14:33 <bauzas> edleafe: I was rather thinking of proposing a new PS to clarify our agreement here
14:14:39 <johnthetubaguy> I think he is, but not 100% sure
14:14:58 <edleafe> bauzas: ah, I was reading what johnthetubaguy asked
14:15:02 <edleafe> #undo
14:15:03 <openstack> Removing item from minutes: <ircmeeting.items.Action object at 0x9fdc410>
14:15:22 <bauzas> I guess we can move on
14:15:23 <bauzas> :)
14:15:24 <edleafe> #action bauzas to push a new PS for the scheduler logging spec clarifying what we discussed here
14:15:32 <edleafe> Yeah, let's move on
14:15:38 <edleafe> Next up: WIP Resource providers: dynamic resource classes - https://review.openstack.org/#/c/312696/ (Jay)
14:15:46 <edleafe> still awake, jaypipes?
14:16:03 <jaypipes> edleafe: yup but no real stuff to talk about on that.
14:16:20 <jaypipes> edleafe: I need to push a new rev of resource-providers-allocations today and get dansmith's signoff on it.
14:16:31 <edleafe> ok, sounds good
14:16:37 <edleafe> Next up: Add newton spec for ordered filter scheduler - https://review.openstack.org/#/c/256323/ (John)
14:17:20 <edleafe> johnthetubaguy: want to discuss any of the comments here?
14:18:03 <johnthetubaguy> yeah, not sure I get your question, only just spotted it right now
14:18:13 <bauzas> I'll review that spec asap
14:18:28 <bauzas> I have a pretty decent backlog of specs to look at
14:18:37 <edleafe> bauzas: I think we all do :)
14:18:42 <bauzas> and I was a bit off those days
14:18:55 <johnthetubaguy> edleafe: whats your question about soft filter? you need an example?
14:19:13 <bauzas> soft filter is an oxymoron, right ? :)
14:19:20 <jaypipes> bauzas: yes :)
14:19:26 <edleafe> johnthetubaguy: well, I get the "I'd prefer if this host had X" case
14:19:46 <johnthetubaguy> ah, you mean the quantitive ones
14:19:49 <edleafe> but not how "I prefer a host with more X than a host with less X" case
14:19:49 <bauzas> nit, I tend to prefer "soft/hard constraints" and get rid of the notion of weighters/filters
14:20:26 <johnthetubaguy> edleafe: so they might be why we never remove weights, but I was hoping to add a threshold to turn that into a yes/no
14:20:34 <edleafe> bauzas: 'contraints' has too many letters to type
14:20:50 <jaypipes> edleafe: so does constraints.
14:21:00 <bauzas> I reviewed the spec in the early days so it probably diverged, but IIRC, it was about providing a consistent way to ponder our constraints
14:21:03 <edleafe> jaypipes: :-P
14:21:06 <bauzas> graaaah
14:21:29 <bauzas> as a French guy, I'm used to be under constraints, like those days
14:21:34 <jaypipes> requires vs. prefers...
14:21:43 <bauzas> yup
14:22:02 <bauzas> and tbh, I'm very worried of any weigher inflation we could have
14:22:27 <edleafe> johnthetubaguy: I guess if we can't remove weights, I don't see significant gain in this approach
14:22:35 <bauzas> having some way to consider that filters and weighers are two sides of the coin
14:22:44 <bauzas> is +1 to me
14:22:52 <bauzas> edleafe: why ?
14:23:03 <johnthetubaguy> yeah, the soft/hard transition seems a useful thing
14:23:04 <bauzas> edleafe: why a filter can't be giving more than just a boolean ?
14:23:10 <edleafe> bauzas: because now we have *3* types of things
14:23:29 <edleafe> filters, preferences, and weights
14:23:34 <johnthetubaguy> so I hope we could get rid of weighers by adding the threshold
14:23:44 <johnthetubaguy> I just haven't done enough of that to prove it will work nicely or not
14:23:56 <bauzas> so, I need to look again at the spec before further commenting it
14:24:08 <Yingxin> edleafe: what's the difference between preferences and weights?
14:24:11 <johnthetubaguy> hard preference, soft preference, maybe then weighers
14:24:32 <johnthetubaguy> Yingxin: the preference is always a yes/no answer
14:24:41 <edleafe> Yingxin: a preference says "I prefer that a host has X, but it's OK if it doesn't"
14:25:04 <edleafe> a weight says "given two hosts, prefer the one that has more/less of X"
14:25:20 <edleafe> qualitative and quantitative, as johnthetubaguy put it
14:25:25 <johnthetubaguy> Yingxin: your question on per request preferences seems orthogonal to this spec, I actually quite like the sharding approach for different settings, but I know jaypipes was coming up with some better ideas
14:25:27 <Yingxin> johnthetubaguy: edleafe: boolean vs valued right?
14:25:40 <johnthetubaguy> so any valued one can be converted to boolean with a threshold
14:25:52 <johnthetubaguy> the bigger question, is does that actually prove practical or not
14:26:02 <johnthetubaguy> I haven't tried that enough
14:26:08 <edleafe> johnthetubaguy: the even bigger question is how to set that threshold
14:26:28 <johnthetubaguy> edleafe: so thats the problem today, with *ALL* preferences
14:26:44 <edleafe> johnthetubaguy: not for qualitative stuff
14:26:50 <edleafe> boolean works fine with those
14:27:02 <johnthetubaguy> booealn works, we just don't support that
14:27:10 <johnthetubaguy> in a soft way
14:27:27 <johnthetubaguy> so you have to convert the boolean to a number
14:27:36 <johnthetubaguy> and add it with other numbers, and its very, very messy
14:28:00 <edleafe> johnthetubaguy: which is why I'm not seeing the advantage of this approach
14:28:10 <edleafe> it's much cleaner in some cases
14:28:15 <edleafe> but it makes the others messier
14:28:19 <bauzas> but Python accepts True = 1.0 so we're good :p
14:28:26 * bauzas jk
14:28:48 <edleafe> bauzas: not in Python 3
14:28:55 <edleafe> :)
14:29:07 <bauzas> edleafe: shhhhhht
14:29:11 <johnthetubaguy> so today, soft->hard means writing code, which sucks
14:29:11 <edleafe> OK, let's continue discussion on the spec itself
14:29:41 <bauzas> ++
14:29:59 <johnthetubaguy> its also worth noting, you don't have to use the new soft requirements, if you don't need them
14:30:13 <johnthetubaguy> right now I can't work out how to configure the scheduler
14:30:28 <johnthetubaguy> if people know how, then thats cool, but we need to somehow share that
14:31:13 <edleafe> johnthetubaguy: you mean writing a "how-to" guide?
14:31:22 <edleafe> with different use cases?
14:31:23 <johnthetubaguy> yep, that would help folks
14:31:40 <johnthetubaguy> I can't get the weighers to stop miss-behaving in strange ways
14:31:58 <edleafe> hmmm... maybe we could recruit operators of openstack clouds to find out what they do
14:32:06 <edleafe> and compile them
14:32:45 <Yingxin> johnthetubaguy: I had some comments in your spec, about how people want the "preference": they may want to specify them in the flavor.
14:32:50 <johnthetubaguy> just to be clear, I have my operator hat on here
14:33:18 * cdent reckons johnthetubaguy has many dashing hats
14:33:21 <johnthetubaguy> Yingxin: I like the feature, it just seems orthogonal (and I cover it in the sharding spec, slightly)
14:33:23 <edleafe> johnthetubaguy: which is why it's hard for us non-ops to understand all the use cases
14:33:34 <Yingxin> johnthetubaguy: ok
14:33:45 <johnthetubaguy> I tried to give examples in the spec of it working and not
14:33:49 <Yingxin> I'll read that
14:33:57 <edleafe> Moving on...
14:34:01 <edleafe> Pass Qualitative Requirements separately in a Request - https://review.openstack.org/#/c/313784/ (Ed)
14:34:02 <johnthetubaguy> +1
14:34:14 <edleafe> Only a few comments so far
14:34:27 <edleafe> I didn't get much response on the ML post
14:34:36 <johnthetubaguy> so we have another spec (previously) for over-riding image properties
14:35:02 <edleafe> So I have no idea what the impact would be on billing for moving from flavor-only
14:35:06 <johnthetubaguy> it seems like image properties, and whatever this API are, should have the same allowed things
14:35:29 <claudiub> question: are the specs for qualitative requirements / host capabilities / enum capabilities bps going to be approved for N?
14:35:33 <johnthetubaguy> ah...
14:35:35 <edleafe> johnthetubaguy: not following what you mean
14:35:38 <johnthetubaguy> so a flavor has to allow this API
14:35:47 <jaypipes> claudiub: good question :)
14:35:59 <bauzas> edleafe: I thought https://review.openstack.org/#/c/313784/ was dependent on another jaypipes's spec about enumerating capabilities ?
14:36:01 <edleafe> Enums are a non-starter for this
14:36:08 <claudiub> ty, I want a good, positive answer for it. :)
14:36:13 <jaypipes> claudiub: I need to review all of them this afternoon. I'm hoping edleafe can have a special session on the qualitative side of things.
14:36:14 <edleafe> We could have *some* common ones pre-defined
14:36:26 <jaypipes> bauzas: yeah, but edleafe hates my ideas :)
14:36:41 <bauzas> jaypipes: which leaves me as a sad panda then :)
14:36:46 <jaypipes> heh
14:36:50 <jaypipes> it's ok...
14:37:00 <jaypipes> I'll comment back on the standardize-capabilities one, edleafe.
14:37:01 <bauzas> jaypipes: because I was liking your idea that you exposed in the design session
14:37:02 <edleafe> The need for dynamic resources proves my argument :)
14:37:35 <edleafe> Things will always be changing, and new stuff being added
14:37:45 <edleafe> Having to support those in a code release doesn't cut it
14:37:50 <panda> ...
14:37:53 <johnthetubaguy> so I thought we said dynamic flavors would be opt in, i.e. a flavors by default don't allow them?
14:38:32 <jaypipes> edleafe: dynamic resource *classes*, not qualitative capabilities.
14:38:42 <jaypipes> edleafe: but in any case, I will respond on the spec.
14:38:47 <bauzas> okay, so I should review https://review.openstack.org/#/c/313784/ and https://review.openstack.org/#/c/309762/ concurrently then
14:39:04 <edleafe> jaypipes: it's the pattern: spell out all the classes as an enum
14:39:09 <edleafe> oops! we need more
14:39:21 <edleafe> implement a work-around
14:39:38 <edleafe> It's better to design for flexibility from the start
14:39:55 <edleafe> Sure, there will be some common classes/capabilities across all clouds
14:40:06 <jaypipes> edleafe: I'm fine with that, but how would you standardize capability strings without either constants or enums or something?
14:40:26 <bauzas> that's my point
14:40:32 <edleafe> jaypipes: you standardize the common ones
14:40:48 <bauzas> before providing a nicer API, we should discuss on the semantics on that API
14:40:56 <edleafe> any deployment-specific capabilities are added by the operator
14:41:00 <bauzas> of* that API
14:41:25 <edleafe> We can have a local way of doing that, but not an openstack-wide code change
14:41:46 <Yingxin> such as dynamic capabilities added into api database?
14:42:02 <jaypipes> edleafe: right, totes agreed.
14:42:17 <edleafe> Yingxin: that's one possibility, yes
14:42:30 <jaypipes> edleafe: but what is your proposed implementation of the "standardization of the common ones"?
14:43:01 <edleafe> jaypipes: I think the way you moved the "standard" classes from enum is a good example
14:43:31 <edleafe> IOW, define the common ones in code, but allow for dynamic ones later
14:44:02 <edleafe> Only the common ones would be guaranteed to be cross-cloud available
14:44:29 <edleafe> Anyway, time is ticking... continue arguments on the spec
14:44:34 <cdent> how do you inspect for "common"
14:44:48 <edleafe> Next: Report host memory bandwidth as a metric in Nova -  (sudipto)
14:44:56 <edleafe> sudipto: around?
14:45:04 <sudipto> edleafe, yeah...
14:45:28 <edleafe> oops - lost the URL
14:45:38 <sudipto> https://review.openstack.org/319960 - here's the review. Basically this was approved in the last 2 cycles...and finally I have the pcp code merged - hence wanted to get a re-approval on the same.
14:45:54 <sudipto> I have answered the queries you had edleafe
14:46:15 <edleafe> OK, I'll re-review
14:46:18 <sudipto> johnthetubaguy, wanted to know if the scheduler team thinks there's any impact of this being in the monitor code from the scheduler team per say.
14:46:24 <sudipto> hence i updated the agenda with it.
14:46:25 <edleafe> It would be great to get more eyes on this
14:46:41 <jaypipes> edleafe: got it. makes sense.
14:47:03 <edleafe> Any other specs to discuss?
14:47:08 <mlavalle> how about the routed networks spec: https://review.openstack.org/#/c/263898/
14:47:24 <edleafe> ah yes
14:47:25 <mlavalle> we got a +1 from jaypipes thanks
14:47:28 <edleafe> thanks mlavalle
14:47:44 <edleafe> oh, then I'm opposed :)
14:47:44 <mlavalle> can we get more people reviewing it?
14:47:52 <mlavalle> lol
14:48:15 <johnthetubaguy> yeah, I need to get jump on that one again soon
14:48:20 <edleafe> me too
14:48:31 <mlavalle> johnthetubaguy, edleafe: thanks!
14:48:37 <edleafe> got it in my unending list of open tabs
14:49:00 <edleafe> #topic Code Reviews
14:49:07 <edleafe> I had two listed on the agenda
14:49:14 <bauzas> mlavalle: in my pipe too
14:49:30 <edleafe> cdent and claudiub: want to discuss them?
14:49:30 * mlavalle thanks bauzas
14:49:41 <bauzas> 10 mins before calling it a meeting
14:50:11 <cdent> edleafe: which ones?
14:50:40 <cdent> the g-r-p related stuff is now in need of a major rebase to deal with the change of table location, which is my project for the next few hours
14:51:36 <edleafe> ugh - network dropped
14:51:42 <edleafe> sorry
14:51:45 <edleafe> Add Allocation and AllocationList objects - https://review.openstack.org/#/c/282442 (cdent)
14:51:56 <edleafe> Adds HostCapabilities object model - https://review.openstack.org/#/c/285856 (Claudiu)
14:52:09 <bauzas> cdent: we first need to merge dansmith's series, amirite ?
14:52:23 <cdent> bauzas: the part that matters, the db change, is merged
14:52:27 <bauzas> cdent: not talking about the API DB migration which merged
14:52:35 <cdent> and I can slot my stuff in under the object change
14:52:46 <cdent> it's all already several patches high, one more won't hurt
14:52:52 <bauzas> cdent: okay, just needed to make sure you don't need the object interface change
14:53:06 * edleafe thinks juggling mutiple DBs is so much fun!
14:53:14 <claudiub> yeah, looking for some feedback on the host capabilities thingy
14:53:31 <bauzas> given non-prio spec freeze is one week away, I'll be honest and claim that my top reviews will be for specs those days
14:53:52 <claudiub> I've introduced dependencies between capabilities: e.g. os_shielded_vms cannot be requested without also requesting os_secure_boot.
14:53:53 <edleafe> bauzas: +1
14:54:32 <Yingxin> Scheduler claim implementation: https://review.openstack.org/#/c/262938/
14:54:38 <Yingxin> it's from last release
14:55:28 <edleafe> Yingxin: added to my list
14:55:34 <edleafe> 5 minutes to go
14:55:37 <Yingxin> edleafe: thanks
14:55:42 <edleafe> #topic Opens
14:55:59 <edleafe> Question: should we meet next week?
14:56:07 <bauzas> Yingxin: I thought we agreed to not duplicate the Claim model ?
14:56:09 <edleafe> It's a US holiday and the start of PyCon
14:56:20 * cdent has no preference
14:56:26 * bauzas shrugs
14:56:28 <claudiub> it could be on a different day?
14:56:32 * edleafe hopes everyone is going to PyCon, too
14:56:50 <bauzas> I pass :)
14:56:55 <edleafe> claudiub: I'll be out most of the week
14:57:00 <claudiub> ok
14:57:16 <Yingxin> np
14:57:17 <edleafe> We can have the meeting, but we'll need someone to run it
14:57:34 <bauzas> FWIW, I'm totally okay to *not* run that meeting :)
14:57:38 <edleafe> *cough* cdent *cough*
14:57:51 <cdent> oh, I'll run it if people want to have it, no problem
14:58:08 <cdent> but I'm also happy to skip it if people will be away
14:58:47 <edleafe> who want to have a meeting next week?
14:58:59 <cdent> -1
14:59:30 <edleafe> Unless there's a decent number, let's just skip it
14:59:46 <edleafe> Time running out
15:00:00 <edleafe> The crickets have it - no meeting next week!
15:00:05 <edleafe> #endmeeting