17:02:25 <jimbaker> #startmeeting craton
17:02:26 <openstack> Meeting started Thu Feb  9 17:02:25 2017 UTC and is due to finish in 60 minutes.  The chair is jimbaker. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:02:26 <sulo> sure, i dont mind
17:02:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:02:29 <openstack> The meeting name has been set to 'craton'
17:02:49 <jimbaker> #chair sigmavirus sulo jimbaker
17:02:50 <openstack> Current chairs: jimbaker sigmavirus sulo
17:02:55 <jimbaker> #link https://etherpad.openstack.org/p/craton-meetings
17:03:00 <jimbaker> #topic Roll Call
17:03:15 <thomasem> o/
17:03:19 <jovon> o/
17:03:20 <Syed__> o/
17:03:32 * sigmavirus can't chair this today
17:03:40 <jimbaker> sigmavirus, np, i will chair
17:03:47 <jimbaker> and hopefully keep short
17:03:52 <git-harry> hi
17:03:55 <tojuvone> hi
17:04:22 <sulo> hello
17:05:04 <jimbaker> so at some point we agreed that we are not supposed to have this meeting if we didn't have an agenda in advance...
17:05:26 <jimbaker> but given lots of stuff happening... i believe we should disregard and have a short meeting regardless :)
17:06:02 <jimbaker> everyone ok with that?
17:06:15 <Syed__> yep
17:06:36 <jovon> indeed
17:06:50 <jimbaker> awesome. let's just move to our standing agenda, and fit in what we need to communicate
17:06:51 <thomasem> yep
17:07:08 <jimbaker> #topic Action Items from Last Meeting
17:08:06 <jimbaker> http://eavesdrop.openstack.org/meetings/craton/2017/craton.2017-02-06-15.00.log.html
17:09:00 <jimbaker> bunch of action items. so i haven't taken care of doing the dusty's req doc as launchpad
17:09:02 <jimbaker> bug
17:09:17 <thomasem> I have not gotten around to the BP yet this week. I took the bulk resources endpoint bug, but that's lowered in priority right now.
17:09:50 <thomasem> Had some good discussion around the latter, though, and will continue once we get some more critical things done.
17:10:22 <jimbaker> thomasem, agreed. we will want to figure out deployment specifics, and if you want to continue to drive point on that, that would be great
17:10:55 <thomasem> You bet
17:11:09 <jimbaker> re dusty's doc, we got some very useful triaging in our discussion with toan and antonym
17:11:42 <jimbaker> which to recap for others here (such as tojuvone)
17:12:15 <jimbaker> we are trying to figure out for rackspace if we can immediately use craton for some rackspace support needs
17:12:47 <jimbaker> separately i discussed with farid getting these into launchpad so we can track
17:13:46 <jimbaker> i believe as your project lead :) that the most important thing we can do is see craton get adopted for real usage. it will get us in front of real problems, which this project is all about solving
17:13:58 <jimbaker> *so earlier is better here*
17:14:46 <jimbaker> so that's my action item and thomasem's
17:14:57 <jimbaker> sigmavirus is out this point, but we can discuss that later
17:15:31 <jimbaker> sulo, it was mentioned as info, but it could be considered an implicit action item as well given priority
17:15:32 * sigmavirus is here
17:15:33 <sigmavirus> just also in another meeting
17:15:33 <thomasem> Yeah. We have some discussions that need be had with another contender for these use-cases.
17:15:34 <sigmavirus> and can't focus on chairing this one :)
17:15:58 <jimbaker> sigmavirus, no worries. do you want to report on your action item at this time?
17:16:38 <jimbaker> sulo, also please give us feedback on Parent/Child Relationship work is the highest priority work for this week (was marked as #info)
17:16:39 <sigmavirus> Just constantly spinning between projects. Everytime I come back to craton I'm working on testing things
17:16:46 <sigmavirus> (Things I have in flight)
17:17:21 <jimbaker> sigmavirus, and that's your action item right, testing the CLI - hopefully will reduce some of that spin as well
17:18:41 <Syed__> cool
17:19:30 <jimbaker> ok, let me report on the last high priority item, that might have been marked as #action; more work on the getters/setters in the client/CLI, mostly by clean up; and addition of delete support
17:20:02 <jimbaker> i will push that up momentarily. did find that our DELETE on variables doesn't actually do anything, so i have a patch for that as well
17:20:21 <Syed__> Sounds good
17:20:41 <Syed__> for me, i have been looking into https://bugs.launchpad.net/craton/+bug/1662842
17:20:43 <openstack> Launchpad bug 1662842 in craton "Allow search by labels" [Critical,New]
17:20:49 <jimbaker> i like the curious symmetry of "addition of delete" :)
17:21:06 <Syed__> and have some patches in the review, need to write functional test for update calls of projects and users and that patch will be good to go
17:21:12 <jimbaker> Syed__, that's cool, but we are currently focused on action items
17:21:37 <Syed__> Sounds good, i guess once bugs are raised and referenced properly, helps a lot in picking those up
17:22:38 <jimbaker> Syed__, right, that was part of my conversation with farid is everyone doing a better job there. which seems to have improved. and more importantly, making it easier to use the bug info to get at useful reporting
17:23:48 <jimbaker> so we will work on that, and that's very much related to the "requirements as a bug/feature" (or better, bugs) action item i started off with
17:24:08 <Syed__> yeap and that would be great once we have that in place
17:24:12 <Syed__> +1
17:24:44 <jimbaker> awesome. i think that's it for action items, which all seem to be in flight at this point
17:25:12 <jimbaker> #topic Stand-up
17:25:32 <jimbaker> #topic Stand-up - Syed
17:26:13 <jimbaker> (let it be noted that the meetbot seems currently unresponsive. but it's in the log)
17:26:21 <Syed__> Sounds good
17:26:32 <jimbaker> Syed__, anything you want to add to your earlier comments?
17:27:00 <Syed__> No i guess thats all i am doing. Will be looking into picking some CLI bug apart from other things i am doing
17:27:09 <jimbaker> Syed__, that would be great
17:27:33 <jimbaker> #topic Stand-up - git-harry
17:28:43 <jimbaker> fwiw, git-harry, i'm still thinking about my response to your spec on variables in https://review.openstack.org/#/c/428190/
17:28:55 <jimbaker> and i see you have some changes now for review
17:29:09 <git-harry> I've got some patches in review for tidying up the variables a bit. Currently I'm going through the review queue. If there's high priority stuff that needs picking up, point me in the direction of it and I'll take a look.
17:29:25 <git-harry> jimbaker: that stuff is just clean up to make any future changes easier
17:29:44 <git-harry> done
17:30:12 <jimbaker> git-harry, got it. dupe removal is good. i do have concerns about some of the API changes you proposed, namely around PUT
17:30:16 <jimbaker> for variables
17:30:33 <jimbaker> we can discuss further
17:30:41 <git-harry> okay
17:30:50 <jimbaker> #topic Stand-up - jovon
17:31:44 <jimbaker> we will circle back to jovon
17:31:50 <jimbaker> #topic Stand-up - thomasem
17:31:52 <jovon> currently working on documenting users, projects, and devices.
17:32:03 <jimbaker> jovon, +1
17:32:13 <thomasem> Need reviews:
17:32:14 <thomasem> https://review.openstack.org/427777
17:32:14 <thomasem> https://review.openstack.org/428996
17:32:22 <thomasem> Been keeping those tidy
17:32:40 <thomasem> Currently working on wiring up Cloud resource in between Project and Region
17:32:49 <thomasem> Going through the review queue as well
17:32:57 <jimbaker> thomasem, awesome, sounds like you're keeping busy!
17:33:13 <thomasem> I try!
17:33:39 <jimbaker> i expect any number of conflicts here, but we will get the changes in
17:33:42 <thomasem> Other than that, had a lot of meetings distracting me this week, but that won't continue much.
17:34:35 <thomasem> Need to write up a bug for the issue git-harry found this morning with an API change that went in, so putting that on my list right now.
17:34:45 <thomasem> done
17:34:55 <jovon> question. are we planning to create a separate definition for devices? as it is currently nested within net devices?
17:35:03 <jimbaker> thomasem, thanks for reminding us of that part of standup protocol
17:35:25 <thomasem> You bet!
17:35:37 <jimbaker> jovon, consider it like an abstract class
17:35:51 <jovon> understood
17:35:55 <jimbaker> it should not be directly instantiated
17:36:00 <jovon> just curious
17:36:12 <jimbaker> not certain if sqlalchemy would prevent, but our apis certainly do not support
17:36:55 <jimbaker> jovon, no worries. but let's move to open discussion if more. everyone, please remember your open discussion topics!
17:37:16 <jimbaker> #topic Stand-up tojuvone
17:37:27 <tojuvone> ok..
17:37:35 <tojuvone> I want to make "planned host maintenance" using Craton
17:37:49 <tojuvone> Adding namespace, something like:  http://${CRATON_IP}:8080/v1/hosts/{id}/host_details
17:37:56 <tojuvone> I want to add this url link to Nova
17:38:09 <tojuvone> Now in Craton namespace host_details is added by "admin"
17:38:17 <tojuvone> but in Nova side also owner of server/VM will see the same link
17:38:26 <tojuvone> I guess we can handle the policy somehow in Craton to make this possible?
17:38:44 <tojuvone> I want to update my Nova spec with different url to link to Craton than it currently state there: https://review.openstack.org/428070/
17:38:47 <jimbaker> tojuvone, so we should have the level of support necessary for this in rbac
17:38:58 <jimbaker> but rbac + namespaces is not something we have considered just yet
17:39:25 <jimbaker> having said that: i think namespaces are exactly the way one would want to organize some set of variables for rbac
17:40:13 <jimbaker> we have considered separately that the prefix of a variable is something we would directly control with rbac, and is mentioned briefly in the rbac bp that is posted
17:40:49 <jimbaker> (this follows similar practice in tooling like zookeeper or consul)
17:41:02 <sulo> tojuvone: your nova service permissions are different from craton .. unless you are using all within the same keystone domain/context
17:41:13 <jimbaker> so that is the extended answer to yes
17:41:29 <jimbaker> sulo, and that's a great question, what that cross service permission model means
17:41:47 <sulo> tojuvone: i am not sure why you are worried about policy right now .. i think what should matter here is how does nova know to create the link
17:41:56 <sulo> tojuvone: the answer is possibly, it doesnt
17:42:07 <tojuvone> We need to set it there
17:42:18 <tojuvone> when making http://${CRATON_IP}:8080/v1/hosts/{id}/host_details
17:42:23 <sulo> so unless this url is something you are manually looking to set in nova .. i dont see how this will work
17:42:34 <sulo> tojuvone: what is id ?
17:42:49 <sulo> id here is craton id .. nova doesnot know that
17:42:56 <jimbaker> sulo, so we see some similar sort of bidirectionality required in webhooks
17:43:07 <tojuvone> "id" I guess we have in Craton host the "id"
17:43:15 <sulo> jimbaker: how do you mean ?
17:43:25 <jimbaker> but can i ask we move this to the open discussion?
17:43:30 <sulo> tojuvone: yes, but how does nova know that id ?
17:43:41 <sulo> tojuvone: are you going to manually push the url for each host ?
17:43:43 <jimbaker> please remember this topic, we can get back to it
17:44:01 <jimbaker> sulo, since you are here
17:44:04 <tojuvone> yes, need to push it to Nova
17:44:12 <jimbaker> #topic Stand-up sulo
17:44:16 <sulo> jimbaker: iirc nova does not have any webhooks, neither do we
17:44:38 <jimbaker> sulo, no, but it's in the same idea. what can we learn?
17:44:41 <jimbaker> :
17:44:43 <jimbaker> :)
17:44:49 <sulo> I am just chasing a sqlalchemy bug thats causign search by labels to fail
17:45:04 <sulo> its on pagination function
17:45:19 <sulo> but looks like something weird on sqla
17:45:29 <jimbaker> sulo, ok, is this something we can ignore for now
17:45:42 <sulo> not unelss we want search by labels to work
17:45:44 <jimbaker> by avoiding that join?
17:45:52 <sulo> join ?
17:46:06 <jimbaker> to the labels table
17:46:17 <jimbaker> but  back up
17:46:29 <jimbaker> the relevant question is search by labels
17:46:33 <sulo> not sure i follow, why woud that matter ?
17:46:37 <jimbaker> so sure, that needs to work :)
17:47:22 <sulo> although, it is freaking out on labels join .. we do join on variables too .. which seems to work
17:47:37 <sulo> well, its not exactly join its freakign out on
17:47:39 <jimbaker> sulo, earlier we were discussing unnecessary joins and how that was impacting pagination. specifically variable details when such details were not requested (not even implemented!)
17:47:45 <sulo> its the part where pagination is adding limit
17:48:08 <sulo> jimbaker: thats different ? .. we are eager loading
17:48:15 <sulo> we can chagne that to select load
17:48:20 <sulo> but that wont fix this problem
17:48:22 <jimbaker> sulo, and eager loading can be great. but not here
17:49:07 <jimbaker> let's back up. i wanted to find out earlier about parent/child and supporting cabinet to container views
17:49:28 <jimbaker> since this is the highest priority task we have right now, other than variable support down to the CLI
17:49:52 <sulo> parent_id is a easy fix, ill push a review soon
17:50:02 <sulo> i need this pagination work to get done for that
17:50:10 <jimbaker> sulo, ok, just want to make sure we are addressing
17:50:13 <sulo> since it piggybacks on the links support added by ian
17:50:19 <jimbaker> ah, ok
17:50:28 <jimbaker> and not so separable
17:50:33 <jimbaker> as coded now
17:50:36 <sulo> yeah its tied to this code
17:50:46 <sulo> my change is based on this work
17:51:01 <sulo> everything works .. except the seach by labels test is failing
17:51:01 <jimbaker> got it. i was hearing pagination, and thinking, that's not in any of our immediate reqs
17:51:13 <jimbaker> even though we know it's super important for the future
17:51:47 <sulo> anyway need to figure out why this is happening
17:51:59 <sulo> as it seems there might be other conditions this can cause pain with
17:52:03 <jimbaker> yes. let's spend some more time on this
17:52:24 <jimbaker> but we may need to separate out the link code and defer pagination if too much
17:52:29 <jimbaker> just want to give us that out
17:52:45 <sulo> yeah so the search function i had added while back
17:52:54 <sulo> and without the pagination bit it works
17:52:59 <jimbaker> got it
17:53:02 <sulo> its a much have feature righ tnow
17:53:09 <sulo> example: seach for all compute nodes
17:53:12 <sulo> or api nodes etc
17:53:19 <sulo> by labels
17:53:20 <jimbaker> in terms of vars
17:53:36 <sulo> more than one service on a host
17:54:05 <sulo> identified by host labels
17:54:21 <jimbaker> right, which is more efficient with labels than going into json
17:54:54 <jimbaker> #info clearly we have gone to open discussion...
17:55:45 <jimbaker> sulo, let me state something provocative/time saving: i do wonder if we will end up not caring about labels after all
17:56:17 <sulo> how so ?
17:56:25 <jimbaker> what would happen if we just removed labels?
17:56:45 <sulo> we wont have a nice way to tag hosts ? vars are not well suited for this ?
17:56:46 <thomasem> Yes... what WOULD happen if we just removed labels?
17:56:50 <jimbaker> could we do the same sort of queries, by using variables?
17:56:58 <sulo> maybe
17:57:02 <sulo> it might be ugly
17:57:20 <jimbaker> sulo, especially given namespaces, which we will be implementing
17:57:20 <sulo> example: host1 needs 3 labels "compute", "api", "scheduler"
17:57:30 <sulo> even with namespace
17:57:50 <jimbaker> so let's say we have the labels namespace
17:57:59 <sulo> labels (namespace) will have which k:v pair
17:58:05 <jimbaker> labels/compute=true
17:58:47 <jimbaker> basically we get everything we need
17:59:11 <sulo> yeah, i dunno how that seach will look like
17:59:22 <jimbaker> yes, right now our variables search is expensive, because it's not indexed. but it should be cheap with the right json indexing
17:59:50 <jimbaker> in this case, it might be still be fine, since we only care about the presence of keys, not even the values
18:00:32 <thomasem> It would be weird to have labels/compute=false
18:00:35 <thomasem> and treat it as a compute.
18:00:39 <sulo> yeah
18:00:50 <sulo> i mean we can make it work if desired
18:01:03 <jimbaker> thomasem, but that's why we have the labels namespace
18:01:22 <jimbaker> i should point out that the usual representation of a set is a dict...
18:01:31 <sulo> yeah i dunno
18:01:38 <sulo> what happens when someone does compute=false
18:01:48 <sulo> it wil mean we have to query on k and filter on value
18:02:01 <jimbaker> cpython may have an alternative implementation; but that's how it's done in java for example: key=true
18:02:09 <jimbaker> implicitly
18:02:24 <jimbaker> sulo, so we constrain with the namespace
18:02:36 <jimbaker> we can use convention for now
18:02:42 <sulo> like ?
18:03:24 <jimbaker> but one thing tojuvone and i discussed is that if we had a "maintenance/" namespace, we could constrain the allowed transitions
18:03:25 <sulo> so what do we gain from doing k:v labels, vs doing labels the way it is now ?
18:03:34 <sulo> i think if we gain more .. then its a no brainer
18:03:41 <jimbaker> so one can just set keys to arbitrary values
18:03:55 <jimbaker> sulo, so we get k:v labels for free
18:04:09 <jimbaker> and we implement less db code, and complex queries
18:04:13 <sulo> i get that .. but whats the use case for that
18:04:14 <jimbaker> which are currently breaking
18:04:30 <jimbaker> also, labels don't work on anything but devices
18:04:45 <sulo> right
18:04:56 <sulo> then its the equivalent of getting rid of labels
18:04:58 <tojuvone> Actually might extend /maintenance to more generic /host_details
18:05:02 <sulo> k:v is just vars
18:05:15 <jimbaker> yes!
18:05:17 <sulo> we can still keep the endpoint etc ..
18:05:30 <sulo> not saying i am convinced this is good yet
18:05:32 <jimbaker> sulo, because it knows about the labels/ namespace
18:05:40 <sulo> yeah
18:05:56 <sulo> i am mostly worried about perf hit when there is large data seach on vars
18:06:02 <sulo> but if we dont think that will be a probl
18:06:06 <sulo> then its fine
18:06:19 <jimbaker> sulo, i think it's less of a problem than a multitable join
18:06:32 <tojuvone> I need to go, but be back in as hour or so
18:06:40 <sulo> jimbaker: you will have that join regardless .. because varxmixin
18:06:44 <jimbaker> tojuvone, thanks for your participation
18:06:50 <sulo> jimbaker: currently we can get rid of vars in labels seach
18:06:53 <sulo> thats a differnt issue
18:07:06 <jimbaker> sulo, there are joins regardless :)
18:07:11 <jimbaker> just fewer joins
18:07:14 <jimbaker> in a complex query
18:07:15 <sulo> heh sure
18:07:33 <jimbaker> so consider labels + variables in the search query. with pagination. uh-oh
18:08:15 * thomasem needs to go investigate the labels implementation
18:08:29 <sulo> jimbaker: yeah, i think we should experiement for sure
18:08:30 <jimbaker> thomasem, it is simple, and a delight to read
18:08:42 <jimbaker> but it's too simple imho
18:08:43 <sulo> this is somethign we've talked about for a while anyway
18:08:51 <sulo> maybe its time to just try it out
18:08:55 <jimbaker> sulo, yeah, we kept on simplifying variables
18:09:01 <jimbaker> sorry, simplifying *labels*
18:09:09 <jimbaker> ultimate simplification: remove them
18:09:36 <sulo> well anyway, in terms of priority work to get done
18:09:40 <sulo> we needs labels right now
18:09:58 <jimbaker> but again if we can just use convention now
18:10:04 <jimbaker> labels/compute=true
18:10:18 <jimbaker> then figure out how to make that cleaner later...
18:10:45 <jimbaker> then we can get this done now
18:11:05 <sulo> thats a much bigger change right now ?
18:11:18 <sulo> that needs change on all layers including api maybe
18:11:21 <jimbaker> but vars work now
18:11:23 <sulo> will have to look
18:11:33 <sulo> labels works too
18:11:40 <jimbaker> just not enough
18:11:50 <jimbaker> anyway... good discussion
18:11:58 <sulo> well its pretty bad if we are getting rid of label because somethig else is breaking it
18:12:06 <sulo> lets figure out why paginaiton is breaking it
18:12:12 <sulo> rather than getting rid of it
18:12:17 <jimbaker> sulo, so i would first relax the requirement on sqlalchemy
18:12:24 <jimbaker> and use 1.1.5
18:12:29 <jimbaker> as we earlier discussed
18:12:37 <jimbaker> on this channel
18:12:51 <jimbaker> thomasem mentioned that he saw a similar bug
18:13:28 <jimbaker> i suggest we wrap up our meeting. we are well over
18:13:40 <jimbaker> discussion can continue of course
18:13:51 <jimbaker> seeing no objections....
18:13:52 <thomasem> That bug might have been fixed, btw. I wanted to throw out an idea of where to look, but I haven't gone and dug in very deep on it.
18:13:58 <jimbaker> #endmeeting