| cloudnull | in building the deployment framework do i need to worry about zookeeper / redis ? | 00:16 |
|---|---|---|
| cloudnull | looking at http://craton.readthedocs.io/en/latest/architecture.html | 00:17 |
| *** VW has joined #craton | 00:49 | |
| *** Syed__ has quit IRC | 00:55 | |
| *** VW_ has joined #craton | 02:59 | |
| *** VW has quit IRC | 02:59 | |
| openstackgerrit | Merged openstack/craton master: Fake data: Generate child container per host https://review.openstack.org/439846 | 03:36 |
| *** VW_ has quit IRC | 04:34 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/craton master: Updated from global requirements https://review.openstack.org/440034 | 05:26 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-cratonclient master: Updated from global requirements https://review.openstack.org/440094 | 05:35 |
| sulo | cloudnull: you dont have to worry about zk right now .. and possibly never. There might be some other msg broker server involved though. It will come into play when we start doing workflows (automated audits/remediation). | 10:00 |
| sulo | cloudnull: antonym: also re: https://github.com/openstack/craton/blob/master/tools/docker_run.sh#L38-L46 ... yes its mostly for testing/demo, and yes the need for local project/user etc is when not using keystone. | 10:11 |
| sulo | cloudnull: antonym: we do have a chicken and egg problem when it comes to bootstrapping the project in the sense that a root user is expected. We do that right now though extenal script .. but work in underway to make that a part of the init migration. | 10:12 |
| *** VW has joined #craton | 11:33 | |
| *** VW has quit IRC | 12:03 | |
| *** VW_ has joined #craton | 12:03 | |
| *** VW_ has quit IRC | 12:03 | |
| *** VW has joined #craton | 12:04 | |
| *** VW has quit IRC | 12:27 | |
| *** VW has joined #craton | 12:36 | |
| *** VW has quit IRC | 13:00 | |
| *** VW has joined #craton | 13:02 | |
| cloudnull | sulo: understood, thanks for the follow up. makes sense now. | 13:34 |
| cloudnull | should we expect admin user management to be done within the craton-manage cli in the future? | 13:34 |
| *** VW_ has joined #craton | 13:40 | |
| *** VW_ has quit IRC | 13:40 | |
| *** VW_ has joined #craton | 13:41 | |
| *** VW has quit IRC | 13:41 | |
| sulo | cloudnull: yes, project mgt is actually in cratonclient already, just need to work out the right way to do so with the root user that gets created when the project is bootstrapped | 13:53 |
| cloudnull | cool. thanks sulo | 13:57 |
| openstackgerrit | Ian Cordasco proposed openstack/craton master: Update schema definition for network interfaces port https://review.openstack.org/439832 | 13:59 |
| openstackgerrit | Ian Cordasco proposed openstack/craton master: Update schema definition for network interfaces port https://review.openstack.org/439832 | 14:15 |
| sigmavirus | sulo: maybe +2 things after the tests finish running? :P | 14:17 |
| sigmavirus | Although I appreciate your enthusiasm | 14:17 |
| sulo | sigmavirus: hah, yeah i relise no test result after, the patch looked good to me locally though | 14:20 |
| sulo | sigmavirus: did it fail ? | 14:20 |
| sigmavirus | Yeah I did a stupid in the functional tests :) | 14:20 |
| farid | hi all | 14:22 |
| sulo | sigmavirus: rgr, ill be more careful | 14:23 |
| sulo | hi farid | 14:23 |
| * farid never claimed powerpoint skills | 14:23 | |
| sigmavirus | farid: who said otherwise? | 14:23 |
| * sigmavirus shakes fist at zuul | 14:23 | |
| cloudnull | o/ farid | 14:23 |
| sigmavirus | I want my test results for Patchset 3 | 14:23 |
| farid | lol, just messing, I put together a set of slides for the monday demo, still needs work though | 14:23 |
| sigmavirus | farid: I hear spit, elbow grease, and shoe shine help | 14:23 |
| farid | totally | 14:24 |
| thomasem | farid: Reviewing those slides right now, actually. Got a question or two queued up that might help us clarify some things. :P | 14:26 |
| thomasem | Will reply to the thread, though. | 14:26 |
| sigmavirus | hm | 14:26 |
| thomasem | Or I can ask here, I suppose | 14:27 |
| farid | thomasem: awesome, yeah either way works | 14:27 |
| farid | sigmavirus: I'll send those over | 14:27 |
| thomasem | farid: "Enable ADM and Support/Ops Rackers to quickly obtain an inventory listing of devices, Openstack version and Projects on the account" seems a little confusing. I thought Account == Project? | 14:27 |
| thomasem | Do we mean to communicate Clouds in a Project rather than Projects in an Account? | 14:27 |
| thomasem | s/in/on/ | 14:28 |
| openstackgerrit | Ian Cordasco proposed openstack/craton master: Update schema definition for network interfaces port https://review.openstack.org/439832 | 14:28 |
| farid | I think yeah clouds in an account, the way we've been working it | 14:30 |
| farid | think there were comments in that word document regarding same, lemme check | 14:31 |
| thomasem | Gotcha | 14:35 |
| thomasem | Yeah, just want to be sure we're not communicating some additional concept that doesn't quite map. | 14:36 |
| sigmavirus | okay sulo +2, to your heart's content | 14:36 |
| *** wirehead_ has quit IRC | 14:49 | |
| *** wirehead_ has joined #craton | 14:50 | |
| *** VW_ has quit IRC | 14:52 | |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: WIP: CLI and client support for get/set/delete of resource vars https://review.openstack.org/427032 | 14:54 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: WIP: CLI and client support for get/set/delete of resource vars https://review.openstack.org/427032 | 14:56 |
| openstackgerrit | Sulochan Acharya proposed openstack/python-cratonclient master: Add vars filtering to region-list https://review.openstack.org/440611 | 14:59 |
| *** VW has joined #craton | 15:00 | |
| *** VW has quit IRC | 15:01 | |
| *** VW has joined #craton | 15:01 | |
| openstackgerrit | git-harry proposed openstack/python-cratonclient master: Add table formatter sort key https://review.openstack.org/440617 | 15:06 |
| openstackgerrit | git-harry proposed openstack/python-cratonclient master: Add devices-list to support /v1/devices https://review.openstack.org/438561 | 15:08 |
| *** valw has joined #craton | 15:10 | |
| *** valw has quit IRC | 15:11 | |
| *** valw has joined #craton | 15:11 | |
| farid | thomasem: changed project to clouds in the doc, said you were editing I hope that doesn't step over any changes. | 15:12 |
| thomasem | farid: Nope, I had to open it to copy/paste. Not making edits. :P | 15:12 |
| thomasem | It wouldn't let me copy/paste from the rendered presentation. | 15:12 |
| farid | :| | 15:14 |
| openstackgerrit | Merged openstack/craton master: Updated from global requirements https://review.openstack.org/440034 | 15:23 |
| thomasem | farid: Looking at UC 5 where it references the hierarchy being country region > customer > cloud > cloud region > ... Where does "country region" fit into the overall scheme of things as it relates to Craton? | 15:24 |
| thomasem | We don't have that concept since the top-most is Project (customer). | 15:24 |
| farid | we had discussed of using labels for that stuff iirc thomasem | 15:26 |
| farid | since we didn't have that in the hierarchy proper | 15:26 |
| thomasem | Gotcha | 15:26 |
| thomasem | Thanks farid! | 15:31 |
| farid | yeppers, hope jimbaker / sulo can confirm but I'm pretty sure that was the discussion. | 15:37 |
| jimbaker | farid, thomasem, or perhaps more likely through variables. incidentally i have the underlying code now working, will work on making it possible to use in the overall query filter that is built, including with pagination | 15:49 |
| jimbaker | specifically the code for https://bugs.launchpad.net/craton/+bug/1661226 | 15:50 |
| openstack | Launchpad bug 1661226 in craton "Filter by variables lacks resolved variables" [Critical,Confirmed] - Assigned to Jim Baker (jimbaker) | 15:50 |
| *** valw has quit IRC | 15:57 | |
| git-harry | jimbaker: fyi https://bugs.launchpad.net/craton/+bug/1669493 | 16:06 |
| openstack | Launchpad bug 1669493 in craton "Filtering with multiple variables returns an empty list" [Undecided,New] | 16:06 |
| thomasem | sigmavirus: good feedback, I'll fix this up. Thanks! | 16:10 |
| thomasem | sigmavirus: but is that more the approach we were going for overall? | 16:12 |
| sigmavirus | I think so | 16:12 |
| thomasem | sans my silliness | 16:12 |
| thomasem | Okay, great. Thanks! | 16:12 |
| *** Syed__ has joined #craton | 16:15 | |
| *** jovon has joined #craton | 16:17 | |
| jimbaker | git-harry, not surprising. i pretty much thought the existing vars would not work in that way | 16:18 |
| jimbaker | fortunately my code does the right thing in this case | 16:19 |
| sulo | https://bugs.launchpad.net/python-cratonclient/+bug/1669508 | 16:36 |
| openstack | Launchpad bug 1669508 in Craton's Python Client "Resource GET calls need to allow filtering" [Undecided,New] | 16:36 |
| git-harry | thomasem: https://bugs.launchpad.net/python-cratonclient/+bug/1668221 | 16:54 |
| openstack | Launchpad bug 1668221 in Craton's Python Client "Random NoneType() < int() error from cli" [High,In progress] - Assigned to git-harry (git-harry) | 16:54 |
| thomasem | Ahhhh | 16:55 |
| sigmavirus | thomasem: git-harry I still haven't reproduced that but I think I know what the caus eis | 16:58 |
| sigmavirus | Oh is that what you're trying to fix with that patch git-harry ? | 16:58 |
| sigmavirus | that's a very indirect way to fix that | 16:58 |
| git-harry | ? | 16:58 |
| thomasem | #startmeeting craton | 17:00 |
| openstack | Meeting started Thu Mar 2 17:00:38 2017 UTC and is due to finish in 60 minutes. The chair is thomasem. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:00 |
| openstack | The meeting name has been set to 'craton' | 17:00 |
| thomasem | #chair sigmavirus sulo jimbaker thomasem | 17:00 |
| thomasem | #link https://etherpad.openstack.org/p/craton-meetings | 17:00 |
| openstack | Current chairs: jimbaker sigmavirus sulo thomasem | 17:00 |
| thomasem | #topic Roll Call | 17:01 |
| jimbaker | o/ | 17:01 |
| thomasem | o/ | 17:01 |
| Syed__ | o/ | 17:01 |
| sigmavirus | git-harry: the root cause of that bug is that the columns are not fixed so we sometimes sort on columns that have none in them | 17:01 |
| sigmavirus | o/ | 17:01 |
| sigmavirus | if we had consistent column ordering, that would be better | 17:01 |
| git-harry | sigmavirus: no, that is related but not the root cause | 17:01 |
| jovon | o/ | 17:01 |
| thomasem | sulo: you have time for this meeting? | 17:02 |
| sulo | yes .. iam here | 17:02 |
| thomasem | awesome | 17:02 |
| thomasem | I think we can skip action items. All are being punted. | 17:03 |
| jimbaker | so some topics: demo monday, and its support (data ingest, tmux/other access) | 17:03 |
| openstackgerrit | git-harry proposed openstack/python-cratonclient master: Add table formatter sort key https://review.openstack.org/440617 | 17:03 |
| jimbaker | plus critical bugs | 17:03 |
| Syed__ | jimbaker: would appreciate if you can send the invite for demo | 17:03 |
| jimbaker | farid, ^^^ | 17:04 |
| * thomasem would really like for us to make a habit of adding topics to the etherpad | 17:04 | |
| jimbaker | thomasem is entirely right! | 17:04 |
| * farid ack's | 17:04 | |
| openstackgerrit | git-harry proposed openstack/python-cratonclient master: Add devices-list to support /v1/devices https://review.openstack.org/438561 | 17:04 |
| thomasem | #topic Demo Monday | 17:05 |
| thomasem | So, as I understand it we're going to go down the list of use-cases from Bjoern and map them to a demonstrable thing in Craton | 17:06 |
| jimbaker | thomasem, ack | 17:06 |
| thomasem | We have farid's ppt in flight | 17:06 |
| sigmavirus | thomasem: to be fair, I don't oft have topics to add ahead of time =P | 17:06 |
| sigmavirus | But I agree | 17:07 |
| thomasem | That is fair, indeed. :P | 17:07 |
| jimbaker | it's the nature of collab - we figure this out as we go at times | 17:07 |
| thomasem | #topic bug https://bugs.launchpad.net/python-cratonclient/+bug/1668221 | 17:07 |
| openstack | Launchpad bug 1668221 in Craton's Python Client "Random NoneType() < int() error from cli" [High,In progress] - Assigned to git-harry (git-harry) | 17:07 |
| thomasem | So sigmavirus and git-harry were chatting about this one, let's get that chat resolved | 17:08 |
| sigmavirus | So this is related to (at least) the fact taht we don't have a consistent column ordering | 17:08 |
| sigmavirus | what that means is that we always sort on the first column | 17:08 |
| sigmavirus | so there are times where the first column has the potential to have None in it as well as another type. | 17:09 |
| jimbaker | ahh. and why are we sorting in the client? | 17:09 |
| jimbaker | shouldn't this respect what the api produces, given pagination? | 17:09 |
| sigmavirus | If we pick a consistent column ordering and consistent column to sort on, then that negates the need to take into consideration None | 17:09 |
| sigmavirus | jimbaker: so this formatter is a reimplementation of something ripped off from the rest of openstack | 17:09 |
| git-harry | sigmavirus: --fields | 17:10 |
| sigmavirus | We can default it to order based on what's returned | 17:10 |
| jimbaker | +1 | 17:10 |
| sigmavirus | git-harry: --of-gold | 17:10 |
| jimbaker | :) | 17:10 |
| jimbaker | i feel like the stuff that has been ripped out from the rest of openstack has been of unfortunate dubious quality. sorry | 17:10 |
| sigmavirus | jimbaker: no offense taken. I didn't rip it off | 17:11 |
| sigmavirus | =) | 17:11 |
| git-harry | sigmavirus: you can order the columns to try and work around the issue and that will likely work most of the time but if someone specifies a set of fields that will nullify the workaround | 17:11 |
| jimbaker | indeed | 17:11 |
| sigmavirus | git-harry: which is why jimbaker and I agree that no ordering on anything by default makes sense | 17:11 |
| jimbaker | exactly, this problem just goes away | 17:11 |
| jimbaker | and py2 vs py3 sorting issues | 17:12 |
| jimbaker | magically gone | 17:12 |
| git-harry | wait, are you suggesting stripping the client-side sorting and and just taking it from the api? | 17:12 |
| jimbaker | yes | 17:12 |
| jimbaker | given pagination, does it even make sense? no | 17:12 |
| git-harry | okay, then yes I agree | 17:12 |
| sigmavirus | git-harry: the entire concern about sortby_index goes away | 17:13 |
| jimbaker | it can mean we should default some cols to sort on, etc, in the api req | 17:13 |
| jimbaker | so the api does the right thing | 17:13 |
| sigmavirus | jimbaker: the API defaults to [id, created_at] | 17:13 |
| jimbaker | sigmavirus, got it, otherwise pagination would just NOT work | 17:13 |
| jimbaker | as it is now | 17:14 |
| sigmavirus | jimbaker: correct | 17:14 |
| jimbaker | so yeah, we are good | 17:14 |
| thomasem | On that topic, for projects we probably need to just sort on created_at or name or something other than ID, lol. | 17:14 |
| jimbaker | thomasem, well that's a separate question: maybe we should default to name, ..., instead of id | 17:15 |
| sigmavirus | thomasem: right, we should update the API to that default | 17:15 |
| jimbaker | in general | 17:15 |
| thomasem | yes | 17:15 |
| sigmavirus | jimbaker: meh | 17:15 |
| sigmavirus | ordering on id makes enough sense for most | 17:15 |
| jimbaker | ok, a bikesheddable moment it seems :) | 17:15 |
| sigmavirus | lol | 17:15 |
| sigmavirus | I agree it's silly for UUIDs | 17:15 |
| thomasem | Ruh roh, Scoob. Just a passing comment! | 17:15 |
| jimbaker | moving on! | 17:15 |
| thomasem | #topic vars support in CLI | 17:16 |
| thomasem | jimbaker: you had questions? | 17:16 |
| * sigmavirus will brb | 17:16 | |
| jimbaker | yes, just wanted to find out if code is ready to try out | 17:16 |
| jimbaker | sans testing and all of course | 17:16 |
| thomasem | Oh, just for projects. I will be adding the support to the others sometime soon here. | 17:17 |
| jimbaker | i did take a look yesterday, and it was very much WIP, which is fine | 17:17 |
| jimbaker | ok, that works for me | 17:17 |
| jimbaker | thomasem, also - could you update the commit message if you haven't already for specific usage | 17:17 |
| sigmavirus | jimbaker: that seems like a bit too much to ask =P | 17:17 |
| thomasem | Yes, mind making a note of that on the review? | 17:17 |
| jimbaker | that's the one place it's been documented. other docs elsewhere are fine - but stripped down in the commit msg is fine | 17:18 |
| * thomasem should have requested acceptance criteria | 17:18 | |
| jimbaker | thomasem, we changed the acceptance criteria yesterday | 17:18 |
| jimbaker | midflight, love it | 17:18 |
| thomasem | How can we change what does not exist? | 17:19 |
| thomasem | ;) | 17:19 |
| thomasem | Anywho, please note on the review things you'd like to change. Keep in mind time constraints, though, please. I don't wish to scope creep this thing into oblivion. | 17:19 |
| jimbaker | there was some degree of acceptance criteria in the bug, or something. but regardless: just want to ensure we communicate how we use the var manager to work with vars. that's all | 17:19 |
| thomasem | Sure | 17:20 |
| jimbaker | thomasem, well, all i ask is get the ability to get/set/delete vars with respect to a resource. the hows, i don't care too much about | 17:20 |
| thomasem | Sounds good | 17:20 |
| jimbaker | ok, i will try out after this meeting and ask thomasem if i run into problems | 17:20 |
| jimbaker | done? | 17:21 |
| thomasem | yeppers! | 17:21 |
| thomasem | Lemme know what blows up | 17:21 |
| jimbaker | will do | 17:21 |
| thomasem | excellent | 17:21 |
| thomasem | Any other topics? | 17:21 |
| thomasem | folks wish to discuss | 17:21 |
| jimbaker | data ingest | 17:22 |
| thomasem | #topic data ingest | 17:22 |
| jimbaker | antonym, cloudnull, sulo, zz_pwnall1337, this is relevant to your work | 17:22 |
| thomasem | So, where does this all live right now? The scripts. | 17:23 |
| jimbaker | that's my first question | 17:23 |
| jimbaker | as well | 17:24 |
| jimbaker | thomasem, my ideal is we can have a rosetta stone of sorts | 17:24 |
| cloudnull | I'd appreciate a way to POST with a list of devices for ingest. | 17:24 |
| cloudnull | that way i can do imports a cab at a time | 17:24 |
| jimbaker | three different ways to accomplish this import | 17:24 |
| cloudnull | or something similar | 17:24 |
| cloudnull | iterating through the client is error prone and clunky. | 17:25 |
| jimbaker | cloudnull, yeah, so we will have that in https://bugs.launchpad.net/craton/+bug/1661714 | 17:25 |
| openstack | Launchpad bug 1661714 in craton "Bulk endpoint for working with resources" [High,Confirmed] - Assigned to Thomas Maddox (thomas-maddox) | 17:25 |
| jimbaker | thomasem, still plan to work on that? (after other work?) | 17:25 |
| thomasem | jimbaker: Absolutely, but I do not want to prevent someone else from working on it if others' have bandwidth and no higher priority. | 17:26 |
| thomasem | s/others'/others/ | 17:26 |
| jimbaker | re rosetta stone and 3 ways: 1. REST API; 2. python client; 3. CLI. and of course the bulk endpoint is sort of the real way to do this | 17:26 |
| cloudnull | do we need an endpoint for this specifically though ? | 17:27 |
| jimbaker | anyway, just wanted to make sure we look at this usage from a completeness perspective | 17:27 |
| thomasem | Some questions around that, actually. | 17:27 |
| thomasem | So, bulk endpoint in API - we wouldn't want it to block would we? | 17:27 |
| cloudnull | if we call the hosts endpoint i should be a post a list and the api do the right thing | 17:27 |
| jimbaker | cloudnull, this gets into overloading with respect to v1/hosts | 17:27 |
| jimbaker | etc | 17:27 |
| cloudnull | overloading ? | 17:28 |
| jimbaker | does POST v1/hosts (etc) take a single resource; or it can also a take a list of resources | 17:28 |
| jimbaker | ? | 17:28 |
| cloudnull | I | 17:28 |
| jimbaker | or am i misconstruing your request? | 17:29 |
| cloudnull | **imo it should take a list? | 17:29 |
| cloudnull | it can be a list of 1 | 17:29 |
| cloudnull | or many | 17:30 |
| jimbaker | i think the opposite consideration is, we have a number of collections that require to be updated; and that's why we were looking at a bulk endpoint | 17:30 |
| jimbaker | eg, you will want to post a cloud of 1-of-more-regions, etc, ... down to the vars | 17:30 |
| cloudnull | yes. | 17:30 |
| thomasem | Yeah, I understood it as someone wanted to import an entire, say, project. | 17:30 |
| jimbaker | thomasem, very valid way of thinking about it from a scoping perspective | 17:31 |
| cloudnull | but if im thinking of this like a DC, I'd like to POST a cab at a time | 17:31 |
| cloudnull | so something like importing 22 nodes | 17:31 |
| cloudnull | **hosts | 17:31 |
| jimbaker | so maybe v1/projects is the bulk endpoint.... | 17:31 |
| cloudnull | then the network gear for tha cab | 17:32 |
| cloudnull | etc | 17:32 |
| thomasem | Hmmmmm | 17:32 |
| jimbaker | right. it should be easy to add a new cab | 17:32 |
| cloudnull | IE if i had to create the project, cell, etc, with a single api call... so be it. | 17:32 |
| thomasem | So, here's my problem with this | 17:32 |
| cloudnull | but not having the option to POST many hosts at a time is bummer | 17:32 |
| jimbaker | sure, we get that | 17:33 |
| thomasem | Bulk won't scale well and if we're uploading a huge cloud, it could be a problem if we're expecting to block on that operation. | 17:33 |
| jimbaker | thomasem, well i look at this as the same problem as pagination | 17:33 |
| thomasem | Unless we want to use more like "job" semantics around it. | 17:33 |
| jimbaker | we don't let you download the whole cloud now either, so to speak | 17:33 |
| thomasem | jimbaker: The problem is we want an import to be atomic.. no? | 17:34 |
| cloudnull | thomasem: the api should return 202 once the payload has been recieved and get to work behind the scenes | 17:34 |
| cloudnull | there should be no need to block on the client | 17:34 |
| thomasem | Correct, but what if something fails halfway through? | 17:34 |
| jimbaker | right, no need to block on the client. but we are just doing db ops | 17:34 |
| jimbaker | these are not long running tasks of bring stuff up | 17:34 |
| jimbaker | as it is, we already assume that long running stuff is treated differently, in workflows | 17:35 |
| thomasem | So this would be long-running | 17:35 |
| jimbaker | thomasem, i didn't say bulk was easy :) | 17:35 |
| jimbaker | thomasem, i very much disagree with that premise. here's an alternative way we could have done pagination | 17:35 |
| jimbaker | we could have kept a cursor open | 17:35 |
| cloudnull | thomasem: if it forked and moved the request over to a transaction id which could be queied then we could pull to see when the op was completed. | 17:35 |
| jimbaker | then page through it | 17:35 |
| thomasem | Exactly | 17:35 |
| jimbaker | that is a terrible idea | 17:35 |
| cloudnull | if it failed we would know . | 17:35 |
| git-harry | can anyone point me to the script that has been written for doing this on the client side? | 17:36 |
| thomasem | cloudnull: that's exactly my point... We want to communicate the success/failure of the import | 17:36 |
| thomasem | and handle it accordingly | 17:36 |
| jimbaker | git-harry, right, we just asked about this | 17:36 |
| thomasem | if something goes wrong, we'll want to undo what we've already done. | 17:36 |
| git-harry | jimbaker: yes, and I didn't see an answer | 17:36 |
| jimbaker | i know that we discussed putting this in a osic ops repo | 17:36 |
| thomasem | But, then, why not submit to a /v1/imports resource? | 17:37 |
| cloudnull | couldn't we just not commit the insert and report back via the transaction id that the pOST failed ? | 17:37 |
| thomasem | That will create an import "job" and give you back a place you can watch | 17:37 |
| jimbaker | thomasem, so the question is: what's the maximum size of the submitted resource for that import job? | 17:37 |
| cloudnull | i mean we can control the payload size within the api | 17:38 |
| jimbaker | we are going to do it in some batch. how do we size this batch? | 17:38 |
| thomasem | I don't imagine the client needs to care? | 17:38 |
| jimbaker | thomasem, currently it does with pagination, however... | 17:39 |
| jimbaker | just asking - what's the difference? | 17:39 |
| thomasem | How do you break up an entire cloud import? | 17:39 |
| thomasem | What would that look like? | 17:39 |
| jimbaker | i didn't say it was trivial :) | 17:39 |
| thomasem | jimbaker: I know | 17:39 |
| thomasem | I never expected it was | 17:40 |
| sigmavirus | heh | 17:40 |
| thomasem | The difference is in pagination it's a list of resources of the same type | 17:40 |
| jimbaker | anyway... here's the takeaway: we have to think about this more | 17:40 |
| thomasem | Rather, they're equal in the hierarchy of the response. | 17:41 |
| jimbaker | i did like our discussion re graphql | 17:42 |
| jimbaker | although i don't know if it solves for import. it certainly helps with export | 17:42 |
| cloudnull | not to say we should do exactly this but Miguel (he a racker) has a fairly straight forward approach to doing async w/ flask -- https://blog.miguelgrinberg.com/post/using-celery-with-flask -- in that way we could have a simple-ish solution to the issue of bulk import and being able to release a client call. | 17:42 |
| cloudnull | which would should scale fine . | 17:43 |
| cloudnull | without also needing us to reinvent this wheel | 17:43 |
| jimbaker | i doubt the problem turns on async or not | 17:43 |
| cloudnull | jimbaker: re: [11:33] <thomasem> Bulk won't scale well and if we're uploading a huge cloud, it could be a problem if we're expecting to block on that operation. | 17:44 |
| thomasem | cloudnull: I totally agree it should be async. | 17:44 |
| jimbaker | and hence my discussion of pagination... | 17:44 |
| jimbaker | and analogs of | 17:44 |
| thomasem | But I don't think pagination solves this. Anyway, we can get into the details of that design. | 17:45 |
| thomasem | But it sounds like that's not today? | 17:45 |
| jimbaker | most certainly not | 17:45 |
| cloudnull | yea, im not sure pagination is what we're looking for here. | 17:45 |
| *** ediardo has left #craton | 17:46 | |
| thomasem | Alrightyo. Any other topics folks want to discuss? | 17:46 |
| thomasem | Or right into Open Discussion? | 17:46 |
| jimbaker | +1 | 17:46 |
| thomasem | #topic Open Discussion | 17:46 |
| thomasem | crickets | 17:48 |
| thomasem | Okay, heads down, everyone? | 17:49 |
| jimbaker | yes | 17:49 |
| farid | zz_pwnall1337: around ? | 17:49 |
| thomasem | #endmeeting | 17:49 |
| openstack | Meeting ended Thu Mar 2 17:49:14 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/craton/2017/craton.2017-03-02-17.00.html | 17:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/craton/2017/craton.2017-03-02-17.00.txt | 17:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/craton/2017/craton.2017-03-02-17.00.log.html | 17:49 |
| jimbaker | thanks everyone! | 17:49 |
| thomasem | Danke! | 17:49 |
| jimbaker | http://www.reactivemanifesto.org/ | 17:50 |
| jimbaker | with concrete specs here: http://www.reactive-streams.org/ | 17:51 |
| jimbaker | i don't know if anyone has looked at this specifically from a python perspective, but certainly related work with respect to async/await | 17:52 |
| farid | git-harry: zz_pwnall1337 has the script, trying to find that | 17:53 |
| jimbaker | anyway, just wanted to put this out there, because it has a great body of work that can offer ideas. eg, backpressure | 17:53 |
| jimbaker | i personally don't think it applies to craton fwiw... | 17:54 |
| farid | git-harry: https://gist.github.com/pwnall1337/ef023192ebe7eeb5e41c692035513154 | 17:54 |
| farid | don't think that's the latest though | 17:54 |
| jimbaker | ok, hopefully we will get visibility on that | 17:54 |
| jimbaker | it's all about making things reproducible with respect to this import work | 17:55 |
| *** VW_ has joined #craton | 17:55 | |
| *** VW has quit IRC | 17:55 | |
| git-harry | farid: thanks | 17:57 |
| farid | got hold of zz_pwnall1337 , he said this one has parent/child import but is missing labels which he's working on still | 17:58 |
| farid | the one I gisted earlier | 17:58 |
| farid | git-harry: sent e-mail with the code sulo had replied with as well for comparison | 18:02 |
| farid | he said in the message his has labels | 18:02 |
| farid | should be in a repo somewhere. | 18:02 |
| sulo | git-harry: that script didnt work for me unfortunately | 18:04 |
| sulo | assuming we are talkign about the import script | 18:04 |
| * sulo wasnt paying attention here | 18:04 | |
| git-harry | sulo: farid just sent me your version | 18:05 |
| sulo | git-harry: i dont have a version ... i commented out the bits that didnt work .. and just added some temp stuff to get containers imported | 18:05 |
| git-harry | ah | 18:06 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: WIP: CLI and client support for get/set/delete of resource vars https://review.openstack.org/427032 | 18:06 |
| jimbaker | sulo, so if we can see that script somewhere... just want to have visibility into work on data ingest | 18:07 |
| jimbaker | and hopefully collaboration | 18:07 |
| farid | so what was the decision on the nested vars btw ? | 18:08 |
| sulo | jimbaker: need to ask Tim to post it somewhere | 18:09 |
| sulo | farid: nested vars ? | 18:10 |
| farid | yeah like ansible facts that are nested | 18:10 |
| sulo | what about it ? ... sorry wasnt paying attention here .. missing context | 18:11 |
| farid | tim is thinking of inserting all ansible facts as a single json blob | 18:11 |
| farid | which I don't think will be useful for filtering for example | 18:11 |
| sulo | yeah thats not useful | 18:11 |
| farid | cloudnull: think you started looking at this too ? | 18:11 |
| sulo | you would want to do k:v .. if the value is nested then thats that | 18:11 |
| farid | correct me if wrong plz | 18:11 |
| sulo | if you do one k:v then its not searchable | 18:12 |
| farid | sulo: so if the value is nested it would be k.k.k:v ? | 18:12 |
| cloudnull | farid: yes. | 18:12 |
| sulo | yes ... and thats not searchable right now | 18:12 |
| sulo | json support right now is limited | 18:12 |
| farid | cloudnull: cool thanks | 18:13 |
| cloudnull | according to jimbaker the vars backend is simple json so we should be able to store "all the json" as is. | 18:13 |
| farid | sulo: yeah but k.k.k:v would be searchable right ? | 18:13 |
| cloudnull | as for search idk if it can dive into the nested structure | 18:13 |
| sulo | yes thats true .. but searching is different | 18:13 |
| sulo | farid: no .. we sont have that right now .. mysql does support it (and getting better with 8) | 18:14 |
| *** VW_ has quit IRC | 18:14 | |
| sulo | farid: not sure we will never though | 18:14 |
| sulo | not saying | 18:15 |
| sulo | we should improve that as we go | 18:15 |
| cloudnull | sulo: you may be able to implement something like this https://github.com/cloudnull/cruton/blob/master/cruton/data_store/drivers/cassandra/utils.py#L95-L140 | 18:15 |
| cloudnull | allowing the app to recursively search the data | 18:16 |
| sulo | cloudnull: yes, if we are doing it client side (client to db) we can | 18:16 |
| sulo | cloudnull: maybe we can steal some of that :) | 18:17 |
| cloudnull | please do | 18:17 |
| jimbaker | craton is indifferent about the nesting; but searching is currently limited to k=v. the next step would be add to json path support, which is straightforward, i think the biggest concern was how to represent with a url | 18:24 |
| jimbaker | in terms of any desired query | 18:25 |
| jimbaker | also this doesn't consider searches that look at this from full text. but a separate issue, presumably to handled by using searchlight | 18:25 |
| jimbaker | thomasem, so a couple of things on https://review.openstack.org/#/c/427032/ | 18:31 |
| thomasem | jimbaker: mhmm? | 18:32 |
| thomasem | jimbaker: can we note them inline? | 18:32 |
| jimbaker | 1. python client update and get work well. the interface is nice. i haven't figured out delete yet, but will look at project-vars-delete to see if i can figure | 18:32 |
| jimbaker | thomasem, yes, i will also do that there | 18:33 |
| jimbaker | don't worry, dude! | 18:33 |
| thomasem | jimbaker: awesome | 18:33 |
| thomasem | haha okey doke :P | 18:33 |
| jimbaker | just wanted to go to irc first to see if i can get clarification | 18:33 |
| thomasem | craton project-vars-delete <id> <key> <key> <key> was what I thought we were going for there? | 18:33 |
| thomasem | Oh, yeah. No worries. | 18:33 |
| jimbaker | so should it be project.variables.delete('foo', 'bar'), or something else | 18:33 |
| thomasem | Ahhh, I see what you mean. | 18:34 |
| jimbaker | right now that just causes crazy url construction | 18:34 |
| thomasem | Unfortunately that is project.variables.delete(json=['foo', 'bar']) | 18:34 |
| jimbaker | also project-vars-set key=value calls DELETE on v1/projects/<id>/variables? not certaun | 18:35 |
| thomasem | jimbaker: You know what? It probably should only do that if there are any to delete. | 18:36 |
| thomasem | Good call. | 18:36 |
| thomasem | jimbaker: ^^ | 18:36 |
| thomasem | Some of this is awkwardness with passing a list as a body to DELETE | 18:36 |
| thomasem | And trying not to break the interface for all of the other delete calls | 18:37 |
| thomasem | Where you could do resource.delete(id) | 18:37 |
| thomasem | Maybe if we change DELETE to accept something like {"keys": [...]}? | 18:38 |
| jimbaker | thomasem, next up. let's get __repr__'s written for VariableManager, Variables, Variable | 18:38 |
| thomasem | Okay. Is there a particular pattern you would prefer I follow for those __repr__'s? | 18:39 |
| jimbaker | thomasem, possibly | 18:39 |
| jimbaker | well just be informative :) | 18:40 |
| jimbaker | easy enough, right? | 18:40 |
| thomasem | "THIS IS A VARIABLE OBJECT, SHEESH!" | 18:40 |
| jimbaker | actually there's something similar on the backend | 18:40 |
| jimbaker | in the models | 18:40 |
| thomasem | We'll see when I write them and get feedback to change them. :P | 18:40 |
| thomasem | Cool, I'll check 'em out. | 18:40 |
| thomasem | Thanks | 18:40 |
| jimbaker | but in general, for Variables i would expect that it repr as a dict, with its wrapper | 18:40 |
| thomasem | Right | 18:41 |
| jimbaker | you know, the usual - give me how i could be constructed | 18:41 |
| thomasem | +1 | 18:41 |
| jimbaker | but awesome work, looks great | 18:41 |
| thomasem | Thanks!! | 18:42 |
| jimbaker | let's hammer out project first, then we can add in the rest on the next pass | 18:42 |
| thomasem | Sounds good | 18:42 |
| jimbaker | ok, i will add my comments to the latest iter of the patch | 18:43 |
| thomasem | Danke mein Freund | 18:43 |
| thomasem | Oh wow... didn't realize what time it wsa | 18:44 |
| jimbaker | btw, back to the delete stuff - my ideal would be, update and delete should be halves of just doing JSON PATCH | 18:44 |
| jimbaker | so if at some point we implemented JSON PATCH | 18:44 |
| jimbaker | it could just transparently work with existing code | 18:45 |
| jimbaker | pipe dream? just an ask | 18:45 |
| thomasem | Right, delete would be a patch call on update? | 18:46 |
| thomasem | for variables | 18:46 |
| jimbaker | update/delete really are just parts of patch, right? | 18:46 |
| thomasem | Unless you're deleting the whole resource | 18:46 |
| jimbaker | but then it's done at the resource level | 18:47 |
| thomasem | PATCH /v1/<id>/variables makes sense for deleting variables | 18:47 |
| thomasem | Right | 18:47 |
| jimbaker | assuming a certain bug is fixed by yours truly.... :) | 18:47 |
| jimbaker | anyways, just something to think about. i don't know the answers, just wanted to ask this question | 18:48 |
| thomasem | Gotcha | 18:48 |
| thomasem | Okay, food! | 18:49 |
| jimbaker | later! | 18:50 |
| *** VW has joined #craton | 18:53 | |
| *** valw has joined #craton | 19:11 | |
| *** valw_ has joined #craton | 19:14 | |
| *** valw has quit IRC | 19:15 | |
| sigmavirus | git-harry: should I pick up https://bugs.launchpad.net/python-cratonclient/+bug/1668221? | 19:18 |
| openstack | Launchpad bug 1668221 in Craton's Python Client "Random NoneType() < int() error from cli" [High,In progress] - Assigned to git-harry (git-harry) | 19:18 |
| * sigmavirus interprets silence as acquiesence | 19:21 | |
| *** valw_ has quit IRC | 19:23 | |
| openstackgerrit | Merged openstack/python-cratonclient master: Updated from global requirements https://review.openstack.org/440094 | 19:27 |
| jimbaker | sigmavirus, +1 | 19:31 |
| jimbaker | i believe that was the net of our earlier meeting, given that sorting is out of the client | 19:31 |
| * sigmavirus nods | 19:33 | |
| sigmavirus | jimbaker: can you look at 439832 ? sulo had +2'd it but I fubar'd the tests (not the fix) | 19:33 |
| thomasem | Beautiful day outside. My goodness. | 19:33 |
| jimbaker | yeah, i had noticed :) | 19:34 |
| farid | a walk out there may be necessary | 19:41 |
| jimbaker | best part of my ack response is that it could apply to either sigmavirus or thomasem... | 19:46 |
| jimbaker | sigmavirus, +2'ed that minor, thanks! | 19:46 |
| openstackgerrit | Merged openstack/craton master: Update schema definition for network interfaces port https://review.openstack.org/439832 | 19:52 |
| openstackgerrit | Syed Ahsan Shamim Zaidi proposed openstack/craton master: Documentation Fix for Region Create https://review.openstack.org/440790 | 20:03 |
| jimbaker | thomasem, here it is, early access to the underlying query we need for search by resolved vars, https://gist.github.com/jimbaker/3c2d3acdc49940444417914ae0ad951a | 20:39 |
| jimbaker | next steps: 1. do obvious generalization; 2. write as a subquery, much like https://github.com/openstack/craton/blob/master/craton/db/sqlalchemy/api.py#L262 ; 3. pull it all together; 4. tests somewhere in that process | 20:40 |
| jimbaker | thomasem, one reason i like to do this in the context of a gist is that it shows how we can explore with the existing API, before committing to some specific impl details | 20:41 |
| thomasem | +1 | 20:50 |
| openstackgerrit | Sulochan Acharya proposed openstack/craton master: Allow resolved vars in details list calls https://review.openstack.org/439175 | 20:58 |
| openstackgerrit | Ian Cordasco proposed openstack/python-cratonclient master: WIP: Make column ordering consistent and predictable https://review.openstack.org/440810 | 21:01 |
| *** VW has quit IRC | 21:04 | |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: WIP: CLI and client support for get/set/delete of resource vars https://review.openstack.org/427032 | 21:16 |
| *** VW has joined #craton | 21:17 | |
| thomasem | jimbaker: added __repr__'s and made it so VariableManager.delete(...) can take just a list of keys to delete (it wraps crud.CRUDClient's delete(...)) | 21:18 |
| jimbaker | nice! | 21:19 |
| jimbaker | and before i could comment on that stuff on the review | 21:19 |
| jimbaker | it's ok | 21:19 |
| jimbaker | :) | 21:19 |
| thomasem | So, now you'd do something like cc.projects.get(id).variables.delete('key1', 'key2') | 21:19 |
| thomasem | Verrrrry funny | 21:20 |
| thomasem | :P | 21:20 |
| jimbaker | thomasem, ok, just a quick mention - take a look at the __repr__ for Variable, because i get stuff like this Variable(name="baz", value="88") | 21:24 |
| jimbaker | i checked the variables from the rest api and it's actually 88 | 21:24 |
| jimbaker | so probably want to use %r or something like that | 21:25 |
| thomasem | Oh gotcha, good point | 21:25 |
| jimbaker | also want to be able to support nested json in general. but that's just a __repr__ question. it's fine in terms of variable.value | 21:26 |
| jimbaker | in terms of maintaining its type | 21:26 |
| thomasem | Oh gotcha, good point | 21:26 |
| thomasem | Oops, mt | 21:26 |
| thomasem | Lemme test. I'm pretty sure that'll work, though. | 21:27 |
| thomasem | <Variables test=Variable(name=u'test', value=2), thomas=Variable(name=u'thomas', value={u'what': Variable(name=u'what', value={u'on': Variable(name=u'on', value=u'earth')})}), foo=Variable(name=u'foo', value=u'bar'), zoo=Variable(name=u'zoo', value=u'ba | 21:28 |
| thomasem | Shows nested | 21:28 |
| thomasem | jimbaker: ^^ | 21:28 |
| thomasem | There are sigmavirus's comments here, too: https://review.openstack.org/#/c/427032/15/cratonclient/common/cliutils.py | 21:36 |
| thomasem | I am converting some of those to using oslo_utils.strutils, but I can't find one that does "none" or "null" to None, also that's a good question regarding escaping quotes... we might want to consider if someone wishes to actually set these values to the string "none", for instance. | 21:37 |
| thomasem | I have to do none4='"NONE"' to get "none4": "\"NONE\"", | 21:40 |
| thomasem | But, that does work. | 21:40 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: WIP: CLI and client support for get/set/delete of resource vars https://review.openstack.org/427032 | 21:47 |
| *** wirehead_ has quit IRC | 21:55 | |
| *** wirehead_ has joined #craton | 22:02 | |
| *** wirehead_ has quit IRC | 22:06 | |
| *** mhayden has left #craton | 22:08 | |
| *** wirehead_ has joined #craton | 22:15 | |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: WIP: CLI and client support for get/set/delete of resource vars https://review.openstack.org/427032 | 22:38 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: WIP: CLI and client support for get/set/delete of resource vars https://review.openstack.org/427032 | 22:46 |
| thomasem | Alright, I need to run. Have a lovely evening, everyone!! | 22:50 |
| jimbaker | thomasem, take care! | 23:02 |
| farid | well ajaxterm doesn't look all that fabulous for a demo | 23:53 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!