15:00:30 <jimbaker> #startmeeting craton
15:00:31 <openstack> Meeting started Mon Mar 27 15:00:30 2017 UTC and is due to finish in 60 minutes.  The chair is jimbaker. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:31 <jimbaker> #chair sigmavirus sulo jimbaker thomasem
15:00:32 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:33 <jimbaker> #link https://etherpad.openstack.org/p/craton-meetings
15:00:34 <openstack> The meeting name has been set to 'craton'
15:00:35 <jimbaker> #topic Roll Call
15:00:36 <openstack> Current chairs: jimbaker sigmavirus sulo thomasem
15:00:43 <jimbaker> o/
15:00:48 <thomasem> o/
15:00:53 <anonymike> o/
15:00:56 <thomasem> Lol jimbaker I was about to paste that, too.
15:01:07 <thomasem> If I'm going to chair, mind letting me do that paste?
15:01:09 <jimbaker> thomasem, i just get impatient, that's all...
15:01:21 <jimbaker> ;)
15:01:35 <antonym> o/
15:01:45 <git-harry> hi
15:02:08 <thomasem> #topic Action Items from Last Meeting
15:02:09 <thomasem> #link http://eavesdrop.openstack.org/meetings/craton/2017/craton.2017-03-20-14.59.html
15:02:34 <thomasem> Pushing mine out again. Got bogged down in nuance with the nested vars stuff and didn't get to it.
15:02:45 <thomasem> #action thomasem to write BP regarding deployment as a starting point for iterating on a suggested deployment model
15:03:05 <thomasem> and jimbaker, was there anything more to do regarding translating Dusty's stuff?
15:03:13 <thomasem> Or giving feedback on that, or anything?
15:03:22 <jimbaker> re action item to do the mapping... i think this was a good idea, but not certain if we really need
15:03:24 <thomasem> That didn't get covered last week, but I know it was an action item we had pending for you.
15:03:30 <thomasem> Okay, then we'll skip that one.
15:03:45 <jimbaker> cool
15:03:51 <thomasem> #topic Stand-up
15:03:51 <thomasem> #info each team member briefly describes what they are working on this week, and describes blockers (if there are any)
15:03:58 <thomasem> #topic Stand Up :: jimbaker
15:04:38 <jimbaker> need to finalize alembic. it's been hanging out there for a while with last "small bit" of work needed
15:05:11 <jimbaker> i was planning to revisit rbac, but pushing that out to the next sprint
15:05:16 <jimbaker> so later this week
15:05:47 <thomasem> done?
15:05:49 <jimbaker> lastly tojuvone and i have a conference proposal on maintenance. i looked into more of these issue with respect to OPNFV
15:05:51 <thomasem> Ah
15:05:59 <jimbaker> done
15:06:09 <thomasem> Cool, thanks!
15:06:17 <thomasem> #topic Stand Up :: thomasem
15:06:21 <thomasem> Doing finishing touches on nested var search patch, mostly some additional functional tests. Solved a few bugs that came up from Harry and some things I saw over the weekend uncovered by the functional tests I already wrote. Seems to be in a much better place now and would like some eyes on the patch. I've refactored it a bit to use the jsonpath-rw library, which should be more reliabl
15:06:21 <thomasem> e parsing than my earlier attempts. Other than that, going through review queue and going to write documentation for nested vars search and put that up for review in a separate patch (the current one is large enough).
15:06:21 <thomasem> done
15:06:29 <thomasem> #topic Stand Up :: anonymike
15:06:33 <anonymike> I've been working on adapting some helpful quick start and wrapper functions to craton. I submitted code and document updates last night. I've also been working on a proof of concept project to get some craton data into elastic search. I didn't get as far as I would have liked over the weekend, but I should have something to demo in the next day or so.
15:06:36 <anonymike> done
15:06:47 <thomasem> #topic Stand Up :: antonym
15:07:59 <antonym> continue working with the provisioning pieces and getting the json searching stuff working the way i want to for server discovery, will also steal some spinnets of ansible code for docs
15:08:06 <antonym> done
15:08:18 <thomasem> #topic Stand Up :: git-harry
15:09:28 <git-harry> Current WIP: ddressing comments on https://review.openstack.org/#/c/447580/ and reviewing https://review.openstack.org/#/c/443941/
15:09:29 <git-harry> done
15:10:16 <thomasem> #topic This Week's Priorities
15:10:44 <thomasem> Tentatively (let's discuss these if anyone disagrees)
15:10:48 <thomasem> 1. https://review.openstack.org/441644
15:10:53 <thomasem> 2. https://review.openstack.org/443941
15:11:04 <thomasem> 3. https://bugs.launchpad.net/python-cratonclient/+bug/1675410
15:11:05 <openstack> Launchpad bug 1675410 in Craton's Python Client "Add fields selector to <resource>-show commands" [Undecided,New]
15:11:19 <jimbaker> #3 is hugely important
15:11:25 <thomasem> 4. https://review.openstack.org/447580
15:11:30 <thomasem> And... what else?
15:11:47 <thomasem> Yeah, I'd prefer we get someone focusing on #3 sooner rather than later.
15:11:55 <thomasem> If anyone has bandwidth.
15:12:03 <fsaad> agree on #3
15:12:12 <jimbaker> i would add the outstanding stuff sulo has for client support of network resources
15:12:22 <jimbaker> at least https://review.openstack.org/#/c/427245/
15:12:28 <jimbaker> so that can be #5
15:12:35 <thomasem> https://review.openstack.org/427249, https://review.openstack.org/427245, and https://review.openstack.org/427237
15:12:44 <thomasem> Merge conflicts and needs some tests, iirc.
15:13:05 <jimbaker> yeah, pretty straightforward, just finish up some missing capabilities
15:13:10 <jimbaker> at client level
15:13:15 <thomasem> So, in summary, our priorities this week are:
15:13:23 <thomasem> 1. https://review.openstack.org/441644
15:13:28 <thomasem> 2. https://review.openstack.org/443941
15:13:35 <thomasem> 3. https://bugs.launchpad.net/python-cratonclient/+bug/1675410
15:13:35 <openstack> Launchpad bug 1675410 in Craton's Python Client "Add fields selector to <resource>-show commands" [Undecided,New]
15:13:42 <thomasem> 4. https://review.openstack.org/447580
15:13:48 <thomasem> 5. https://review.openstack.org/#/c/427245/
15:14:09 <thomasem> 6 and 7 (if we have time) are the other network-* client patches that are in merge conflict.
15:14:21 <thomasem> Who can start work on #3?
15:14:28 <jimbaker> +1
15:14:39 <thomasem> 1 & 2 are already in progress and have owners.
15:14:44 <thomasem> and #4
15:14:58 <anonymike> I can look at #3
15:15:19 <thomasem> Excellent. I'd like to discuss it in Open Discussion, so we can be sure all of the context is shared.
15:15:48 <anonymike> Great
15:16:01 <thomasem> Cool. So those are our priorities. :)
15:16:05 <thomasem> #topic Open Discussion
15:16:19 <jimbaker> so #3 is a fairly interesting bug
15:17:02 <jimbaker> we need to implement a comprehensive select functionality that's easy to use
15:17:09 <anonymike> seems pretty straight forward :/  what are the issues?
15:17:10 <anonymike> ah
15:17:36 <jimbaker> so things we might want to select for
15:18:02 <jimbaker> for now: some set of vars, labels, network relationships
15:18:13 <jimbaker> parent(s), children
15:18:28 <jimbaker> in the future: rbac role assignments
15:18:54 <jimbaker> however, regardless of what this is abstractly, we need to make sure it handles
15:19:03 <jimbaker> https://bugs.launchpad.net/python-cratonclient/+bug/1675410/comments/3
15:19:03 <openstack> Launchpad bug 1675410 in Craton's Python Client "Add fields selector to <resource>-show commands" [Undecided,New]
15:19:15 <jimbaker> so bjoern's specific comments
15:19:34 <thomasem> anonymike: So, some asks from our last demo were around more flexibility in what fields are displayed for <resource>-show, and <resouce>-list. They wanted to be able to view some of the data that's actually in variables, without showing ALL of the variables (which can be cumbersome). Ultimately, if we have to way to do things like --fields id, cloud_id, var:host_facts.release.distribut
15:19:34 <thomasem> ion, var:host_facts.hardware.core_count
15:19:38 <thomasem> Yeah
15:19:43 <thomasem> as an example
15:19:45 <thomasem> ^^
15:19:56 <jimbaker> thomasem, yeah, and are those json paths?
15:20:13 <thomasem> Yeah
15:20:37 <jimbaker> so that's why i wanted to point out it's a fairly interesting bug
15:20:39 <thomasem> Now, the other ask (and I'm not sure if we want to do this, exactly) was to have the ability to alias the var selectors
15:20:53 <jimbaker> oh, nice
15:20:55 <jimbaker> makes sense
15:20:57 <anonymike> yeah
15:20:59 <thomasem> As you can imagine... heh paths can get pretty long
15:21:10 <thomasem> So, I wanted to get thoughts here before entering all of this into the bug. :)
15:21:15 <thomasem> Is why it's not there yet.
15:21:17 <jimbaker> so again, it is "select"
15:21:18 <anonymike> what would that look like in the command?
15:21:23 <anonymike> ah yeah
15:21:29 <thomasem> That's what I'm not sure of yet.
15:21:42 <thomasem> It's all comma-separated
15:21:49 <thomasem> (or at least that's a suggestion)
15:21:56 <jimbaker> and this "select" may imply joining
15:22:02 <thomasem> Right now, actually, it's space delimited and I think it has to be the last argument in the command.
15:22:15 <jimbaker> or --field=spec --field=another-spec
15:22:19 <anonymike> jimbaker: so we want to intelligently select the fields from the db rather than just apply our own filter on what is returned
15:22:19 <thomasem> And --fields is only supported for list calls
15:22:44 <jimbaker> anonymike, so the object graph stuff in sqlalchemy is pretty smart for this
15:22:58 <jimbaker> although we need to a better job in limiting queries
15:23:11 <jimbaker> one reason why current queries can get slow
15:23:27 <jimbaker> https://bugs.launchpad.net/craton/+bug/1664707
15:23:27 <openstack> Launchpad bug 1664707 in craton "SQLAlchemy queries are not lazy enough" [High,New]
15:23:44 <thomasem> (did anyone ever write a bug for the --limit issues that Tim was experiencing during the demo?)
15:24:03 <thomasem> Ah, I have that in my notes to do. Nevermind. I will. :)
15:24:06 <jimbaker> thomasem, does not appear to be the case
15:24:19 <thomasem> #action thomasem to write up client bugs related to --limit from Demo
15:24:22 <thomasem> Thanks
15:24:26 <jimbaker> cool. consensus was to do this in the client
15:24:44 <jimbaker> and support --limit=1 to limit=all
15:24:48 <thomasem> Righ
15:24:50 <thomasem> t
15:25:03 <thomasem> Sorry, didn't mean to interrupt. We're good to continue chatting about --fields :)
15:25:17 <jimbaker> thomasem, it's good, that was an important usability issue
15:25:29 <jimbaker> also i sneaked in the spec of all
15:25:35 <jimbaker> --limit=all
15:25:38 <thomasem> Sneaky
15:25:44 <fsaad> thomasem: maybe relevant https://bugs.launchpad.net/craton/+bug/1642532
15:25:44 <openstack> Launchpad bug 1642532 in craton "Limit functionality only partially implemented for listing hosts" [Medium,Confirmed]
15:25:44 <jimbaker> better than --limit=-1
15:25:56 <thomasem> fsaad: thank you!
15:26:02 <fsaad> o/
15:26:12 <jimbaker> just a reasonable value for that spec, that's all
15:26:36 <thomasem> agreed
15:26:45 <thomasem> or maybe --limit=yes
15:26:51 <jimbaker> :)
15:26:52 * thomasem is just being silly now
15:27:06 <jimbaker> so https://bugs.launchpad.net/craton/+bug/1642532 is pretty out of date
15:27:06 <openstack> Launchpad bug 1642532 in craton "Limit functionality only partially implemented for listing hosts" [Medium,Confirmed]
15:27:25 <jimbaker> i'm going to close it as is, given work on pagination
15:27:53 <anonymike> +1
15:28:11 <thomasem> anonymike: So, you're wondering how much of the API can be leveraged for the filtering?
15:28:19 <thomasem> and specification of fields
15:28:32 <anonymike> correct
15:28:33 <thomasem> Or whether we ought to leverage the API for that.
15:28:56 <thomasem> I think that'd be the most efficient... no need to uselessly pass data that's not going to be used.
15:29:06 <anonymike> but that's probably taking the easy way out, if we can get that done on the backend we SHOULD correct?
15:29:27 <anonymike> yeah
15:29:36 <jimbaker> related to discussion we have had on graphql
15:29:51 <thomasem> Oh yes
15:30:13 <anonymike> ha my old team was just beginning to look into graphql
15:30:18 <thomasem> http://graphql.org/
15:30:30 <jimbaker> my initial take on this: let's do it inefficiently for now
15:30:40 <jimbaker> because it's not so inefficient
15:31:14 <jimbaker> but we will want to revisit. especially with respect to bulk
15:31:26 <jimbaker> or with respect to any UI
15:31:53 <jimbaker> btw, speaking of graphql, would this imply we would want to frontend with react?
15:32:01 <jimbaker> just curious...
15:32:06 <anonymike> +10
15:32:13 <thomasem> The moment we find ourselves duplicating the logic for interacting with the API, we need to put it in the API.
15:32:22 <jimbaker> +1
15:33:00 <jimbaker> thomasem, and this gets into pushing the specific selection critertia
15:33:17 <jimbaker> right now, we have this details=all
15:33:31 <jimbaker> but that's a placeholder for a more interesting spec
15:33:39 <jimbaker> of what the user wants
15:33:42 <thomasem> Wonder how aliases could look
15:34:57 <thomasem> ?fields:path:$.variables.foo.bar|alias:"foo_bar",path:$.id|alias:"my_id"
15:35:06 <jimbaker> --field foo->bar
15:35:25 <jimbaker> the best part of using | is the chance to confuse bash
15:35:34 <thomasem> Yeah, it's my favorite.
15:35:35 <jimbaker> if not quoted...
15:35:46 <thomasem> Haha, that's fair.
15:35:56 <jimbaker> we already can do that with & of course
15:36:21 <fsaad> lol
15:36:45 <jimbaker> thomasem, so at some point i think the answer is json
15:36:51 <thomasem> I think the success of this solution increases exponentially with the number of bash special characters that are used.
15:36:56 <jimbaker> vs trying to build a mini language here
15:37:02 <jimbaker> thomasem, :)
15:37:04 <thomasem> in the syntax
15:37:09 <thomasem> jimbaker: yeah
15:37:23 <thomasem> I probably need to be careful when trolling.
15:37:24 <jimbaker> maybe repurpose emoji?
15:37:25 <thomasem> Might actually happen.
15:37:45 <anonymike> ha
15:38:22 <thomasem> So, yeah, I'm open to suggestions.
15:38:36 <thomasem> Ultimately, though, we probably need to be able to alias these selections, lest we get ridiculous column names.
15:38:46 <jimbaker> yeah
15:39:03 <jimbaker> so -> might be good
15:39:11 <jimbaker> or comparable
15:39:24 <anonymike> I like that
15:39:36 <jimbaker> as some nice bash meaning of course
15:40:09 <jimbaker> has
15:40:29 <jimbaker> anyway, we are in a bikeshedding moment i think
15:40:35 <thomasem> Yes, I think so.
15:40:39 <jimbaker> we can conclude
15:40:59 <jimbaker> 1. selection of fields is a critical feature for support
15:41:18 <jimbaker> (and we enumerated some examples, but at least support bjoern's list)
15:41:32 <jimbaker> 2. aliasing is something we need to do
15:42:12 <thomasem> cool
15:42:21 <thomasem> So, anonymike, do you feel like you have enough to get going on that?
15:42:30 <thomasem> Of course, we're always here to chat about the nuances as you go.
15:42:32 <anonymike> thomasem: I think so
15:42:35 <thomasem> Awesome
15:42:36 <jimbaker> anonymike, also you are going to rely on thomasem and me here
15:42:50 <jimbaker> given you are new to the codebase
15:43:24 <thomasem> Sounds good to me. Happy to help however I can!
15:43:25 <jimbaker> sulo and git-harry are also resources - basically any core can advise on how to build something end-to-end in craton
15:43:41 <jimbaker> anonymike, hint: this is how one becomes core...
15:43:52 <anonymike> :)
15:44:01 <thomasem> That and pizza bribes go a long way.
15:44:09 <jimbaker> :)
15:44:14 <anonymike> haha I can totally swing pizza bribes
15:44:15 <thomasem> :P
15:44:17 <thomasem> LOL!
15:44:33 <anonymike> any pointers to spots in the code I should dive into for this?
15:45:03 <jimbaker> anonymike, i think you want to look at jsonpath-rw
15:45:17 <jimbaker> since it will be client side selection
15:45:23 <fsaad> one question regarding field selection, is the thought to still by default show all ?
15:45:40 <jimbaker> fsaad, yes
15:45:54 <thomasem> anonymike: https://github.com/openstack/python-cratonclient/blob/master/cratonclient/shell/v1/hosts_shell.py#L81 and, yeah, jsonpath-rw will probably be your friend here.
15:45:57 <jimbaker> but we could have something where the user can specify this through config
15:46:11 <jimbaker> thomasem was proposing this last week
15:46:35 <jimbaker> my pushback here was that probably wanted something that is not going to be an env var for every possibility
15:46:40 <jimbaker> of resource and detail
15:47:19 <jimbaker> maybe json/yaml specified, with env var pointing to it?
15:47:29 <fsaad> would be nice to have that configurable somehow, will defer to y'all on the details
15:47:45 <jimbaker> fsaad, right - that's where we left it off
15:48:05 <fsaad> is right now a good time?
15:48:15 <thomasem> As long as it follows a convention, it could be fairly abstract and not all that cumbersome.
15:48:19 <jimbaker> we know it should be configurable, we are just going to do it in a second phase
15:48:29 <thomasem> <RESOURCE>_
15:48:31 <thomasem> Ooops
15:48:33 <jimbaker> possibly immediately after this bug
15:48:44 <fsaad> ok sounds good to me!
15:49:01 <fsaad> feel like once you have the capability to use selection, making it configurable should be pretty straightforward right
15:49:02 <jimbaker> but let's not do it in one bug at least. big enough i think to do select and alias
15:49:13 <thomasem> fsaad: oh, yeah.
15:49:22 <fsaad> sounds like a plan then. Thanks guys!
15:49:32 <thomasem> It's really a matter at that point of how people want to interact with it.
15:49:51 <thomasem> sure thing fsaad
15:49:57 <jimbaker> which we will figure out by demoing and actual use
15:50:02 <thomasem> Yerp
15:50:16 <thomasem> Alright, cool. We have 10 minutes left
15:50:19 <jimbaker> including by ourselves
15:50:29 <thomasem> Anything else, or do we want to recover 10 minutes of our lives?
15:50:36 <jimbaker> +1 on recovery
15:50:53 <thomasem> #endmeeting