17:00:38 <thomasem> #startmeeting craton
17:00:40 <openstack> Meeting started Thu Mar  2 17:00:38 2017 UTC and is due to finish in 60 minutes.  The chair is thomasem. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:00:43 <openstack> The meeting name has been set to 'craton'
17:00:56 <thomasem> #chair sigmavirus sulo jimbaker thomasem
17:00:57 <thomasem> #link https://etherpad.openstack.org/p/craton-meetings
17:00:58 <openstack> Current chairs: jimbaker sigmavirus sulo thomasem
17:01:14 <thomasem> #topic Roll Call
17:01:16 <jimbaker> o/
17:01:19 <thomasem> o/
17:01:20 <Syed__> o/
17:01:25 <sigmavirus> git-harry: the root cause of that bug is that the columns are not fixed so we sometimes sort on columns that have none in them
17:01:26 <sigmavirus> o/
17:01:37 <sigmavirus> if we had consistent column ordering, that would be better
17:01:42 <git-harry> sigmavirus: no, that is related but not the root cause
17:01:50 <jovon> o/
17:02:18 <thomasem> sulo: you have time for this meeting?
17:02:24 <sulo> yes .. iam here
17:02:32 <thomasem> awesome
17:03:24 <thomasem> I think we can skip action items. All are being punted.
17:03:32 <jimbaker> so some topics: demo monday, and its support (data ingest, tmux/other access)
17:03:36 <openstackgerrit> git-harry proposed openstack/python-cratonclient master: Add table formatter sort key  https://review.openstack.org/440617
17:03:46 <jimbaker> plus critical bugs
17:03:56 <Syed__> jimbaker: would appreciate if you can send the invite for demo
17:04:05 <jimbaker> farid, ^^^
17:04:08 * thomasem would really like for us to make a habit of adding topics to the etherpad
17:04:15 <jimbaker> thomasem is entirely right!
17:04:24 * farid ack's
17:04:27 <openstackgerrit> git-harry proposed openstack/python-cratonclient master: Add devices-list to support /v1/devices  https://review.openstack.org/438561
17:05:08 <thomasem> #topic Demo Monday
17:06:20 <thomasem> So, as I understand it we're going to go down the list of use-cases from Bjoern and map them to a demonstrable thing in Craton
17:06:27 <jimbaker> thomasem, ack
17:06:50 <thomasem> We have farid's ppt in flight
17:06:58 <sigmavirus> thomasem: to be fair, I don't oft have topics to add ahead of time =P
17:07:04 <sigmavirus> But I agree
17:07:21 <thomasem> That is fair, indeed. :P
17:07:34 <jimbaker> it's the nature of collab - we figure this out as we go at times
17:07:43 <thomasem> #topic bug  https://bugs.launchpad.net/python-cratonclient/+bug/1668221
17:07:44 <openstack> Launchpad bug 1668221 in Craton's Python Client "Random NoneType() < int() error from cli" [High,In progress] - Assigned to git-harry (git-harry)
17:08:02 <thomasem> So sigmavirus and git-harry were chatting about this one, let's get that chat resolved
17:08:43 <sigmavirus> So this is related to (at least) the fact taht we don't have a consistent column ordering
17:08:48 <sigmavirus> what that means is that we always sort on the first column
17:09:08 <sigmavirus> so there are times where the first column has the potential to have None in it as well as another type.
17:09:12 <jimbaker> ahh. and why are we sorting in the client?
17:09:36 <jimbaker> shouldn't this respect what the api produces, given pagination?
17:09:45 <sigmavirus> If we pick a consistent column ordering and consistent column to sort on, then that negates the need to take into consideration None
17:09:58 <sigmavirus> jimbaker: so this formatter is a reimplementation of something ripped off from the rest of openstack
17:10:05 <git-harry> sigmavirus: --fields
17:10:08 <sigmavirus> We can default it to order based on what's returned
17:10:12 <jimbaker> +1
17:10:15 <sigmavirus> git-harry: --of-gold
17:10:19 <jimbaker> :)
17:10:57 <jimbaker> i feel like the stuff that has been ripped out from the rest of openstack has been of unfortunate dubious quality. sorry
17:11:10 <sigmavirus> jimbaker: no offense taken. I didn't rip it off
17:11:12 <sigmavirus> =)
17:11:13 <git-harry> sigmavirus: you can order the columns to try and work around the issue and that will likely work most of the time but if someone specifies a set of fields that will nullify the workaround
17:11:14 <jimbaker> indeed
17:11:31 <sigmavirus> git-harry: which is why jimbaker and I agree that no ordering on anything by default makes sense
17:11:40 <jimbaker> exactly, this problem just goes away
17:12:10 <jimbaker> and py2 vs py3 sorting issues
17:12:15 <jimbaker> magically gone
17:12:17 <git-harry> wait, are you suggesting stripping the client-side sorting and and just taking it from the api?
17:12:22 <jimbaker> yes
17:12:38 <jimbaker> given pagination, does it even make sense? no
17:12:45 <git-harry> okay, then yes I agree
17:13:11 <sigmavirus> git-harry: the entire concern about sortby_index goes away
17:13:22 <jimbaker> it can mean we should default some cols to sort on, etc, in the api req
17:13:29 <jimbaker> so the api does the right thing
17:13:35 <sigmavirus> jimbaker: the API defaults to [id, created_at]
17:13:58 <jimbaker> sigmavirus, got it, otherwise pagination would just NOT work
17:14:03 <jimbaker> as it is now
17:14:06 <sigmavirus> jimbaker: correct
17:14:06 <jimbaker> so yeah, we are good
17:14:28 <thomasem> On that topic, for projects we probably need to just sort on created_at or name or something other than ID, lol.
17:15:03 <jimbaker> thomasem, well that's a separate question: maybe we should default to name, ..., instead of id
17:15:05 <sigmavirus> thomasem: right, we should update the API to that default
17:15:06 <jimbaker> in general
17:15:14 <thomasem> yes
17:15:16 <sigmavirus> jimbaker: meh
17:15:28 <sigmavirus> ordering on id makes enough sense for most
17:15:29 <jimbaker> ok, a bikesheddable moment it seems :)
17:15:32 <sigmavirus> lol
17:15:38 <sigmavirus> I agree it's silly for UUIDs
17:15:41 <thomasem> Ruh roh, Scoob. Just a passing comment!
17:15:45 <jimbaker> moving on!
17:16:09 <thomasem> #topic vars support in CLI
17:16:11 <thomasem> jimbaker: you had questions?
17:16:20 * sigmavirus will brb
17:16:30 <jimbaker> yes, just wanted to find out if code is ready to try out
17:16:40 <jimbaker> sans testing and all of course
17:17:00 <thomasem> Oh, just for projects. I will be adding the support to the others sometime soon here.
17:17:02 <jimbaker> i did take a look yesterday, and it was very much WIP, which is fine
17:17:11 <jimbaker> ok, that works for me
17:17:38 <jimbaker> thomasem, also - could you update the commit message if you haven't already for specific usage
17:17:51 <sigmavirus> jimbaker: that seems like a bit too much to ask =P
17:17:52 <thomasem> Yes, mind making a note of that on the review?
17:18:07 <jimbaker> that's the one place it's been documented. other docs elsewhere are fine - but stripped down in the commit msg is fine
17:18:35 * thomasem should have requested acceptance criteria
17:18:49 <jimbaker> thomasem, we changed the acceptance criteria yesterday
17:18:55 <jimbaker> midflight, love it
17:19:05 <thomasem> How can we change what does not exist?
17:19:08 <thomasem> ;)
17:19:51 <thomasem> Anywho, please note on the review things you'd like to change. Keep in mind time constraints, though, please. I don't wish to scope creep this thing into oblivion.
17:19:54 <jimbaker> there was some degree of acceptance criteria in the bug, or something. but regardless: just want to ensure we communicate how we use the var manager to work with vars. that's all
17:20:22 <thomasem> Sure
17:20:28 <jimbaker> thomasem, well, all i ask is get the ability to get/set/delete vars with respect to a resource. the hows, i don't care too much about
17:20:52 <thomasem> Sounds good
17:20:57 <jimbaker> ok, i will try out after this meeting and ask thomasem if i run into problems
17:21:02 <jimbaker> done?
17:21:05 <thomasem> yeppers!
17:21:22 <thomasem> Lemme know what blows up
17:21:39 <jimbaker> will do
17:21:43 <thomasem> excellent
17:21:47 <thomasem> Any other topics?
17:21:54 <thomasem> folks wish to discuss
17:22:02 <jimbaker> data ingest
17:22:07 <thomasem> #topic data ingest
17:22:38 <jimbaker> antonym, cloudnull, sulo, zz_pwnall1337, this is relevant to your work
17:23:42 <thomasem> So, where does this all live right now? The scripts.
17:23:58 <jimbaker> that's my first question
17:24:02 <jimbaker> as well
17:24:36 <jimbaker> thomasem, my ideal is we can have a rosetta stone of sorts
17:24:37 <cloudnull> I'd appreciate a way to POST with a list of devices for ingest.
17:24:49 <cloudnull> that way i can do imports a cab at a time
17:24:50 <jimbaker> three different ways to accomplish this import
17:24:52 <cloudnull> or something similar
17:25:11 <cloudnull> iterating through the client is error prone and clunky.
17:25:22 <jimbaker> cloudnull, yeah, so we will have that in https://bugs.launchpad.net/craton/+bug/1661714
17:25:22 <openstack> Launchpad bug 1661714 in craton "Bulk endpoint for working with resources" [High,Confirmed] - Assigned to Thomas Maddox (thomas-maddox)
17:25:42 <jimbaker> thomasem, still plan to work on that? (after other work?)
17:26:09 <thomasem> jimbaker: Absolutely, but I do not want to prevent someone else from working on it if others' have bandwidth and no higher priority.
17:26:22 <thomasem> s/others'/others/
17:26:42 <jimbaker> re rosetta stone and 3 ways: 1. REST API; 2. python client; 3. CLI. and of course the bulk endpoint is sort of the real way to do this
17:27:19 <cloudnull> do we need an endpoint for this specifically though ?
17:27:20 <jimbaker> anyway, just wanted to make sure we look at this usage from a completeness perspective
17:27:31 <thomasem> Some questions around that, actually.
17:27:39 <thomasem> So, bulk endpoint in API - we wouldn't want it to block would we?
17:27:42 <cloudnull> if we call the hosts endpoint i should be a post a list and the api do the right thing
17:27:49 <jimbaker> cloudnull, this gets into overloading with respect to v1/hosts
17:27:56 <jimbaker> etc
17:28:05 <cloudnull> overloading ?
17:28:30 <jimbaker> does POST v1/hosts (etc) take a single resource; or it can also a take a list of resources
17:28:34 <jimbaker> ?
17:28:51 <cloudnull> I
17:29:02 <jimbaker> or am i misconstruing your request?
17:29:23 <cloudnull> **imo it should take a list?
17:29:52 <cloudnull> it can be a list of 1
17:30:04 <cloudnull> or many
17:30:07 <jimbaker> i think the opposite consideration is, we have a number of collections that require to be updated; and that's why we were looking at a bulk endpoint
17:30:35 <jimbaker> eg, you will want to post a cloud of 1-of-more-regions, etc, ... down to the vars
17:30:50 <cloudnull> yes.
17:30:57 <thomasem> Yeah, I understood it as someone wanted to import an entire, say, project.
17:31:25 <jimbaker> thomasem, very valid way of thinking about it from a scoping perspective
17:31:37 <cloudnull> but if im thinking of this like a DC, I'd like to POST a cab at a time
17:31:47 <cloudnull> so something like importing 22 nodes
17:31:51 <cloudnull> **hosts
17:31:54 <jimbaker> so maybe v1/projects is the bulk endpoint....
17:32:03 <cloudnull> then the network gear for tha cab
17:32:04 <cloudnull> etc
17:32:10 <thomasem> Hmmmmm
17:32:25 <jimbaker> right. it should be easy to add a new cab
17:32:42 <cloudnull> IE if i had to create the project, cell, etc, with a single api call... so be it.
17:32:45 <thomasem> So, here's my problem with this
17:32:55 <cloudnull> but not having the option to POST many hosts at a time is bummer
17:33:06 <jimbaker> sure, we get that
17:33:15 <thomasem> Bulk won't scale well and if we're uploading a huge cloud, it could be a problem if we're expecting to block on that operation.
17:33:29 <jimbaker> thomasem, well i look at this as the same problem as pagination
17:33:31 <thomasem> Unless we want to use more like "job" semantics around it.
17:33:55 <jimbaker> we don't let you download the whole cloud now either, so to speak
17:34:01 <thomasem> jimbaker: The problem is we want an import to be atomic.. no?
17:34:01 <cloudnull> thomasem: the api should return 202 once the payload has been recieved and get to work behind the scenes
17:34:14 <cloudnull> there should be no need to block on the client
17:34:26 <thomasem> Correct, but what if something fails halfway through?
17:34:30 <jimbaker> right, no need to block on the client. but we are just doing db ops
17:34:39 <jimbaker> these are not long running tasks of bring stuff up
17:35:02 <jimbaker> as it is, we already assume that long running stuff is treated differently, in workflows
17:35:14 <thomasem> So this would be long-running
17:35:15 <jimbaker> thomasem, i didn't say bulk was easy :)
17:35:33 <jimbaker> thomasem, i very much disagree with that premise. here's an alternative way we could have done pagination
17:35:38 <jimbaker> we could have kept a cursor open
17:35:40 <cloudnull> thomasem: if it forked and moved the request over to a transaction id which could be queied then we could pull to see when the op was completed.
17:35:43 <jimbaker> then page through it
17:35:52 <thomasem> Exactly
17:35:54 <jimbaker> that is a terrible idea
17:35:59 <cloudnull> if it failed we would know .
17:36:04 <git-harry> can anyone point me to the script that has been written for doing this on the client side?
17:36:17 <thomasem> cloudnull: that's exactly my point... We want to communicate the success/failure of the import
17:36:20 <thomasem> and handle it accordingly
17:36:22 <jimbaker> git-harry, right, we just asked about this
17:36:32 <thomasem> if something goes wrong, we'll want to undo what we've already done.
17:36:35 <git-harry> jimbaker: yes, and I didn't see an answer
17:36:48 <jimbaker> i know that we discussed putting this in a osic ops repo
17:37:08 <thomasem> But, then, why not submit to a /v1/imports resource?
17:37:12 <cloudnull> couldn't we just not commit the insert and report back via the transaction id that the pOST failed ?
17:37:19 <thomasem> That will create an import "job" and give you back a place you can watch
17:37:52 <jimbaker> thomasem, so the question is: what's the maximum size of the submitted resource for that import job?
17:38:16 <cloudnull> i mean we can control the payload size within the api
17:38:45 <jimbaker> we are going to do it in some batch. how do we size this batch?
17:38:56 <thomasem> I don't imagine the client needs to care?
17:39:09 <jimbaker> thomasem, currently it does with pagination, however...
17:39:24 <jimbaker> just asking - what's the difference?
17:39:29 <thomasem> How do you break up an entire cloud import?
17:39:31 <thomasem> What would that look like?
17:39:52 <jimbaker> i didn't say it was trivial :)
17:39:58 <thomasem> jimbaker: I know
17:40:01 <thomasem> I never expected it was
17:40:30 <sigmavirus> heh
17:40:43 <thomasem> The difference is in pagination it's a list of resources of the same type
17:40:58 <jimbaker> anyway... here's the takeaway: we have to think about this more
17:41:32 <thomasem> Rather, they're equal in the hierarchy of the response.
17:42:08 <jimbaker> i did like our discussion re graphql
17:42:37 <jimbaker> although i don't know if it solves for import. it certainly helps with export
17:42:51 <cloudnull> not to say we should do exactly this but Miguel (he a racker) has a fairly straight forward approach to doing async w/ flask -- https://blog.miguelgrinberg.com/post/using-celery-with-flask -- in that way we could have a simple-ish solution to the issue of bulk import and being able to release a client call.
17:43:16 <cloudnull> which would should scale fine .
17:43:32 <cloudnull> without also needing us to reinvent this wheel
17:43:43 <jimbaker> i doubt the problem turns on async or not
17:44:22 <cloudnull> jimbaker: re: [11:33] <thomasem> Bulk won't scale well and if we're uploading a huge cloud, it could be a problem if we're expecting to block on that operation.
17:44:38 <thomasem> cloudnull: I totally agree it should be async.
17:44:39 <jimbaker> and hence my discussion of pagination...
17:44:55 <jimbaker> and analogs of
17:45:01 <thomasem> But I don't think pagination solves this. Anyway, we can get into the details of that design.
17:45:08 <thomasem> But it sounds like that's not today?
17:45:14 <jimbaker> most certainly not
17:45:30 <cloudnull> yea, im not sure pagination is what we're looking for here.
17:46:17 <thomasem> Alrightyo. Any other topics folks want to discuss?
17:46:21 <thomasem> Or right into Open Discussion?
17:46:40 <jimbaker> +1
17:46:51 <thomasem> #topic Open Discussion
17:48:59 <thomasem> crickets
17:49:04 <thomasem> Okay, heads down, everyone?
17:49:08 <jimbaker> yes
17:49:09 <farid> zz_pwnall1337: around ?
17:49:14 <thomasem> #endmeeting