16:00:09 <thomasem> #startmeeting craton
16:00:09 <thomasem> #chair sigmavirus sulo jimbaker thomasem
16:00:09 <thomasem> #link https://etherpad.openstack.org/p/craton-meetings
16:00:09 <openstack> Meeting started Thu Mar 23 16:00:09 2017 UTC and is due to finish in 60 minutes.  The chair is thomasem. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:11 <thomasem> #topic Roll Call
16:00:13 <openstack> The meeting name has been set to 'craton'
16:00:14 <openstack> Current chairs: jimbaker sigmavirus sulo thomasem
16:00:19 <thomasem> o/
16:00:33 <anonymike> o/
16:02:24 <thomasem> jimbaker: sigmavirus:?
16:02:30 <sigmavirus> o/
16:02:34 <thomasem> git-harry: ?
16:02:36 <jimbaker> o/
16:02:40 <fsaad> o/
16:03:01 <git-harry> hi
16:03:10 <thomasem> tojuvone: ?
16:03:15 <tojuvone> hi
16:03:36 <thomasem> #topic Stand Up
16:03:43 <thomasem> #info each team member briefly describes what they are working on this week, and describes blockers (if there are any)
16:03:51 <thomasem> #topic Stand Up :: thomasem
16:04:36 <thomasem> Working on JSON Path some more. Really far along, I'm keeping it up-to-date here: https://review.openstack.org/#/c/443941/. I see git-harry reviews and appreciate his comments (and agree, I was on the fence anyway about the JSON approximation)
16:05:20 <thomasem> Adding handling for invalid JSON path expressions so we're not returning 500s over it and finishing up testing. Looking into the bug that git-harry found in that patch, too.
16:06:10 <thomasem> done
16:06:17 <thomasem> #topic Stand Up :: anonymike
16:06:39 <anonymike> Working on setting up my environment and getting familiar with the code. I've submitted a few updates to the installation docs and I'm currently working on touching up the "wrapper functions" script to quick launch craton  and easily hit the api.
16:06:42 <anonymike> done
16:06:54 <thomasem> #topic Stand Up :: sigmavirus
16:07:29 <sigmavirus> Wrapping up cratonclient docs/betamax work for tomorrow (which is my last day)
16:07:33 <sigmavirus> Trying to do reviews for others
16:07:37 <sigmavirus> </fin>
16:07:43 <fsaad> :'(
16:07:51 <jimbaker> we will miss sigmavirus
16:07:57 <sigmavirus> nah, I'm a pain in the toe
16:08:00 <thomasem> indeed
16:08:07 <anonymike> :(
16:08:08 <sigmavirus> thanks for taking my side thomasem
16:08:09 <sigmavirus> =P
16:08:17 <jimbaker> minor pain, still can run with that pain in toe
16:08:18 <thomasem> sigmavirus: see, it could be an answer to both.
16:08:27 <thomasem> :P
16:08:37 <thomasem> #topic Stand Up :: jimbaker
16:08:38 <antonym> :/
16:08:47 <sigmavirus> clever
16:08:49 <jimbaker> reviews, trying to understand betamax
16:09:09 <jimbaker> (and why it's failing for me - sigmavirus and i were just going through a debug on that)
16:09:22 <jimbaker> need to finish up alembic
16:09:25 <jimbaker> done
16:09:39 <thomasem> #topic Stand Up :: fsaad
16:10:20 <fsaad> 'grats on yesterday's demo I think it went well regardless of what it seemed like :)
16:10:32 <antonym> +1
16:10:42 <thomasem> :) Thank you!
16:10:45 <fsaad> I'll be poking support for that subset of variables that would be useful to have in the host show
16:11:01 <git-harry> fsaad: I didn't attend that, are there any action items from it?
16:11:02 <fsaad> and also procuring hardware for the craton server today
16:11:08 <fsaad> done
16:11:17 <fsaad> git-harry: I think so, we can chat after standup!
16:11:23 <thomasem> #topic Stand Up :: git-harry
16:12:00 <git-harry> Mostly review related stuff.
16:12:01 <git-harry> done
16:12:15 <thomasem> #topic Stand Up :: tojuvone
16:12:21 <tojuvone> OPNFV summit CFP and Nova side of the planned host maintenance.
16:12:28 <tojuvone> done
16:12:34 <thomasem> #topic Stand Up :: antonym
16:13:42 <antonym> did some doc work and working on getting my provisioning env with craton set up in physical server lab (has graduated from running on fusion), also thought about putting together some docs with ansible use cases like using the uri module with craton for people to have some examples to start from
16:13:55 <antonym> done
16:14:03 <thomasem> +1!
16:14:36 <thomasem> #topic git-harrry's question regarding action items from demo yesterday
16:14:49 <thomasem> Wow, Harry, you got an extra "r" on the house.
16:15:14 <jimbaker> blame autorepeat setting...
16:15:19 <sigmavirus> I bet harry thought to himself, "arrrrrr matey"
16:15:23 <thomasem> I have mine set pretty high.
16:15:34 <thomasem> or low... as the case may be.
16:15:44 <jimbaker> just giving you cover that's all
16:15:55 <thomasem> Anyway, the question was "11:11 git-harry: fsaad: I didn't attend that, are there any action items from it?"
16:15:59 <thomasem> Thanks jimbaker! :P
16:16:20 <jimbaker> so two items
16:16:39 <jimbaker> 1. get the exact data loaded that support requested
16:16:54 <jimbaker> 2. be able to show a subset of variables
16:17:30 <jimbaker> it also prompted some side conversations i had with thomasem re possibility of adding elasticsearch support to craton earlier than later
16:18:05 <git-harry> I thought we'd already sourced the right data, what was wrong with it this time?
16:18:10 <fsaad> I'd say we focus on #2
16:18:16 <fsaad> can work with Tim on #1
16:18:38 <thomasem> git-harry: it was missing, like, package information, iirc.
16:18:43 <jimbaker> right, the craton's team focus is on #2
16:18:45 <fsaad> but if one of you feel like working on data loading while he's out, his script is publicly accessible too
16:18:50 <fsaad> yeah
16:19:26 <jimbaker> there were are also some other questions raised by tim - such as how to work with large result sets in the client
16:19:27 <sigmavirus> I'm glad we're at the point where we recognize this isn't craton's problem per-say
16:19:35 <jimbaker> basically he was using --limit, but not --marker
16:19:45 <thomasem> sigmavirus: which part?
16:20:08 <thomasem> jimbaker: I think he was expecting it to just paginate automatically to the specified limit.
16:20:17 <jimbaker> thomasem, exactly
16:20:26 <sigmavirus> thomasem: the part of loading in package info
16:20:31 <jimbaker> which the client could readily do if we wanted to have that support
16:20:36 <sigmavirus> cratonclient's shell can autopaginate if we want
16:20:42 <jimbaker> given underlying generator
16:20:44 <sigmavirus> that's like a 1 line change
16:21:00 <jimbaker> yeah, sounds about right
16:21:02 <sigmavirus> or maybe 5 (one for each -list command, idk)
16:21:16 <antonym> #link https://github.com/pwnall1337/ops-generic/tree/master/osops_common
16:21:38 <thomasem> He also ran into some bug with --limit when it was kind of low? I don't know what the boundaries there are, but he was getting 400s all of a sudden.
16:21:49 <thomasem> I don't think he filed a bug about it yet, though?
16:21:56 <jimbaker> no bug
16:22:01 <thomasem> Like --limit 2 was giving 400s?
16:22:13 <jimbaker> right, that's a real bug
16:22:28 <jimbaker> given --limit passes straight through
16:22:32 <git-harry> No, I think the schema enforces a minimum
16:22:34 <thomasem> Yeah, it was weird.
16:23:00 <jimbaker> ahh, good point git-harry. still might be worth having limit have a min of say 1
16:23:05 <git-harry> Yeah, minimum is 10
16:23:05 <thomasem> Then we need better messaging around that, or to remove the minimum... why have a minimum?
16:23:08 <thomasem> Ahhhh
16:23:18 <jimbaker> that's a useful limit
16:23:25 <git-harry> The client doesn't expose error messages
16:23:27 <thomasem> min of 1 is, yes.
16:23:33 <jimbaker> i just want ONE item
16:23:35 <thomasem> min of 10 I'm not sure I understand.
16:23:51 <jimbaker> so two bugs here, plus a feature change
16:23:59 <jimbaker> or something like that
16:24:05 <thomasem> lol
16:24:47 <jimbaker> 1. client should report errors precisely from api server; 2. limit of 1 is fine; 3. limit of anything is also fine, and should use underlying autopaginate
16:26:42 <jimbaker> so that will fix up some usability issues
16:26:53 <thomasem> For the issue with <resource>-show: https://bugs.launchpad.net/python-cratonclient/+bug/1675410 and I'll be creating another (or adding to this one) regarding affording for specifying variables to extract, and possibly RegEx support later on.
16:26:53 <openstack> Launchpad bug 1675410 in Craton's Python Client "Add fields selector to <resource>-show commands" [Undecided,New]
16:27:13 <jimbaker> thomasem, +1, this is the big ask from support
16:27:30 <jimbaker> and easy enough to support
16:28:08 <thomasem> Alright. Any other questions on that topic?
16:28:11 <jimbaker> i should mention that one more thing that came up related to variables is that loading from ansible is not always going to produce highly useful things to search against
16:28:15 <thomasem> or comments/concerns
16:28:53 <jimbaker> i forget the exact one, but this was with respect to a list of processor strings. or was it even a list - could have been one big string
16:28:57 <thomasem> And, not sure about what the environment looked like, but the search was pretty slow over a really small amount of data with the nested var search.
16:29:37 <git-harry> jimbaker: that sounds like a user problem
16:29:39 <thomasem> Would be good to figure out JSON column indexing and how that might help us. Again, this is where I think Elasticsearch would outshine our nested var searching.
16:29:45 <jimbaker> thomasem, yeah, because we have no indexing on that json stuff. it's even worse because the alembic change i have under review did add a useful index for k/v search
16:29:58 <jimbaker> to at least make that part work better
16:30:19 <jimbaker> so must get alembic change in, even subsetting with respect to keys would be far better
16:30:32 <thomasem> I still think at some point we're going to have to set some defined boundaries on what this nested stuff can do and say that for everything else, we need to use ES.
16:30:37 <thomasem> Gotcha
16:30:39 <jimbaker> thomasem, agreed about elasticsearch
16:30:55 <jimbaker> i think we could readily do the following for craton
16:31:03 <jimbaker> elasticsearch + kibana
16:31:09 <jimbaker> as a dashboard
16:31:48 <jimbaker> just requires a small amount of work to aggregate docs into elasticsearch
16:32:15 <jimbaker> there are some issues here, such as how do we integrate our security model with elasticsearch
16:32:25 <jimbaker> but it would demo well before we get to that point
16:32:38 <anonymike> +1
16:33:07 <jimbaker> but it would be a fairly simple matter of programming to iterate over resources; dump into elasticsearch with the python interface it supports (use bulk mode)
16:33:36 <jimbaker> i bet anonymike could get this going in an afternoon...
16:33:40 <jimbaker> :)
16:33:43 <anonymike> ;)
16:33:50 <thomasem> Yeah, we can address the "how" more when we get someone dedicated to getting that running and documenting it.
16:34:08 <jimbaker> i suggest we just do a little skunkworks to try out something
16:34:13 <thomasem> Okay
16:34:17 <jimbaker> then we can go back & figure out proper integration
16:34:44 <jimbaker> given we know that elasticsearch has everything we need for security integration, plus keeping stuff live via its replication model
16:35:04 <thomasem> jimbaker: I believe git-harry mentioned that the variable bloat from loading useless things from Ansible was a user concern.
16:35:20 <jimbaker> thomasem, but that's more of a question of filtering i think
16:35:58 <jimbaker> nothing i have seen is even close to something that would be expensive to manage in a database, assuming proper indexing
16:36:12 <jimbaker> but it's super painful right now from the CLI
16:36:56 <thomasem> The comment was specifically about loading variables from Ansible, though. "related to variables is that loading from ansible is not always going to produce highly useful things to search against"
16:36:58 <jimbaker> it only takes a few kilobytes to be too much on the CLI. but megabytes... mysql has no problem
16:37:12 <jimbaker> thomasem, agreed. lots of chaff
16:37:16 <thomasem> Which I believe is where git-harry's comment came from. Basically: Cool, don't load variables you don't care about.
16:37:34 <sigmavirus> got distracted, fwiw, the pagination minimum amount was agreed upon in the specification
16:37:46 <jimbaker> right. and at least know what you want to filter on/search for/and be reasonable about you organize
16:37:50 <thomasem> sigmavirus: interesting. I'll have to go look and see the rationale.
16:37:51 <sigmavirus> so y'all need to update the spec in addition to updating the code =)
16:38:14 <jimbaker> thomasem, and it could be more of the CLI should just the smart thing
16:38:32 <sigmavirus> jimbaker: showing better messages from the CLI is absolutely a good goal too =)
16:38:33 <jimbaker> you want --limit=1 - fine, i will just give you the first item
16:38:40 <sigmavirus> But yeah, I think ES is also necessary for everything we're talking about
16:39:09 <sigmavirus> also, I wanted to do the limitations on 'limit' in a pre-processing function instead of the schema but was directed to use the schema
16:39:18 <sigmavirus> doing it in code would have allowed us to just return 10 instead of returning a 400
16:39:47 <jimbaker> yep, so anonymike, sound like a fun task for you? seems like right up your alley
16:39:53 <jimbaker> in terms of elasticsearch
16:40:01 <anonymike> jimbaker: elasticsearch?
16:40:04 <anonymike> yezzir
16:40:09 <thomasem> Excellent!
16:40:10 <jimbaker> awesome
16:40:22 <sigmavirus> anonymike++
16:40:23 <jimbaker> new guy, gets to work on fun stuff. what can be better? :)
16:40:31 <anonymike> :D
16:40:54 <jimbaker> seriously, it's pretty good, because we need more eyeballs on that python client
16:41:09 <jimbaker> and this is a good place to try it out
16:41:42 <jimbaker> and if by chance we have something to demo next week, well i would never complain about that...
16:42:15 <thomasem> Okay, cool. So, we have some bugs to write, it sounds like.
16:42:18 <anonymike> I better get started :) haha
16:42:20 <thomasem> And BPs
16:42:26 * sigmavirus nods
16:42:36 <jimbaker> yep
16:42:40 <sigmavirus> might be worth updating the spec template to include a place to log updates
16:42:59 <jimbaker> except for you anonymike - you can just figure stuff out, and report back to us
16:43:00 <thomasem> some amendments section
16:43:02 <sigmavirus> (since it sounds like you'll be updating the pagination-spec and something like that might be useful to track the documents changes for people without their hands on the git repo)
16:43:21 <sigmavirus> thomasem: exactly
16:43:33 <anonymike> jimbaker: sure, sounds good
16:45:36 <thomasem> jimbaker: you want to take the BP for ES/Kibana and feature change for "limit", and I'll take writing up the client bugs?
16:45:38 <jimbaker> anonymike, at some point, we will also want to look here - https://github.com/noplay/python-mysql-replication
16:45:52 <jimbaker> thomasem, no BP yet for ES/kibaba
16:45:57 <thomasem> kk
16:46:06 <jimbaker> we are going to skunkworks this first, just to get an understanding of what's there
16:46:46 <thomasem> Lol, okay. It's either/or, to me. I just wanted something to communicate what anonymike is working on, is all, and a place to put notes and such for what's found.
16:47:12 <thomasem> Anyway, however we wish to run this deal.
16:47:19 <anonymike> jimbaker: something like this? https://github.com/scharron/elasticsearch-river-mysql
16:47:46 <jimbaker> anonymike, it is, but rivers are now deprecated in ES
16:48:03 <anonymike> ha yeah, last update 5 yrs
16:48:20 <jimbaker> in general, we have to do this at a higher level
16:48:35 <jimbaker> the notification can say - the underlying doc needs to be updated
16:48:45 <jimbaker> but it has to understand the model of the system
16:48:55 <jimbaker> vs just raw table stuff
16:49:34 <jimbaker> so i think this starts getting more into blueprint level considerations - and why pushing this aspect out a bit
16:49:56 <thomasem> Alright, we're coming up on 10 minutes left. Any other topics folks want to bring up before we run out of time?
16:50:01 <jimbaker> but a huge advantage of ES/kibana is we see changes over time
16:50:08 <jimbaker> thomasem, i'm good
16:50:41 <thomasem> #topic Open Discussion
16:51:20 <thomasem> Alright. Sounds like we've got a plan and a lot of good ideas came out of the demo.
16:51:36 <jimbaker> yep
16:52:22 <thomasem> Priorities are modifying the loading script and affording for the client to configure `--fields` with some variable selection, too?
16:52:45 <thomasem> Once existing priorities are sorted, of course, being the Alembic stuff, nested var search, and...
16:53:08 <fsaad> +1
16:53:12 <thomasem> I think that's it that's remaining?
16:53:21 <thomasem> From Monday's meeting.
16:53:34 <thomasem> So let's finish it up.
16:53:41 <thomasem> And move on to the priorities from our demo
16:53:56 <fsaad> thomasem / all: so does https://bugs.launchpad.net/python-cratonclient/+bug/1675410 mean we don't need to wait for support to provide a list of default fields to show in host-list ?
16:53:56 <openstack> Launchpad bug 1675410 in Craton's Python Client "Add fields selector to <resource>-show commands" [Undecided,New]
16:53:58 <jimbaker> right such as fix client up with respect to --limit=1 to whatever - no change needed in rest api
16:53:59 <fsaad> or still would be nice to have ?
16:54:25 <jimbaker> fsaad, i think we can skip that for the moment
16:54:34 <thomasem> fsaad: it'd be nice to have, but I'm hoping it'll make it so they can get whatever variables they care about themselves.
16:54:47 <thomasem> whatever fields/variables
16:54:49 <jimbaker> thomasem and i were discussing that this gets into a more general conf system
16:54:57 <jimbaker> for client expectations
16:55:10 <jimbaker> otherwise it feels sort of ad hoc
16:55:53 <thomasem> Imagine how bad --details will look in a host-list, though?
16:56:02 <thomasem> assuming that causes it to include the variables...
16:56:07 <jimbaker> thomasem, indeed, megabytes!
16:56:20 <jimbaker> looks great with --format=json
16:56:24 <jimbaker> however
16:56:25 <thomasem> Haha, right
16:56:30 <thomasem> tabled, though... ouch
16:56:57 <jimbaker> there is no good answer here. but again i think the idea you proposed of putting stuff in columns by --field is a great one
16:57:03 <thomasem> cool
16:57:05 <jimbaker> (does it imply details btw)
16:57:29 <thomasem> I suppose... but isn't all that --details does is include variables?
16:57:33 <jimbaker> for now
16:57:36 <thomasem> Gotcha
16:57:37 <thomasem> Okay
16:57:40 <jimbaker> it should also include labels etc
16:57:47 <thomasem> Ahhh right
16:57:48 <thomasem> okay
16:58:00 <jimbaker> the underlying thing is ?details=all - with the implication that it could be mean less
16:58:07 <thomasem> Wells, I'll finish fleshing out the bug(s) and we can discuss from there
16:58:10 <jimbaker> or some interesting subset
16:58:21 <jimbaker> what are my role assignments? etc etc
16:58:45 <thomasem> --fields id, name, roles, var:host_facts.os
16:58:48 <jimbaker> children...
16:58:55 <jimbaker> lots of possible stuff to show
16:59:10 <jimbaker> anyway, probably wrap up. not going to do all that today
16:59:15 <thomasem> Alright, coming up on close of meeting.
16:59:27 <thomasem> 3...
16:59:29 <thomasem> 2...
16:59:34 <thomasem> 1...
16:59:37 <thomasem> #endmeeting