| *** VW has joined #craton | 00:15 | |
| *** VW has quit IRC | 00:36 | |
| *** VW has joined #craton | 00:38 | |
| *** VW has quit IRC | 01:59 | |
| *** VW has joined #craton | 01:59 | |
| *** VW has quit IRC | 02:08 | |
| *** VW has joined #craton | 02:21 | |
| *** VW has quit IRC | 02:26 | |
| *** VW has joined #craton | 03:44 | |
| *** VW has quit IRC | 03:44 | |
| *** VW has joined #craton | 03:44 | |
| *** VW has quit IRC | 03:47 | |
| *** VW has joined #craton | 03:47 | |
| *** tojuvone has joined #craton | 04:14 | |
| *** VW has quit IRC | 05:36 | |
| sulo | o/ | 09:47 |
|---|---|---|
| *** openstackgerrit has quit IRC | 10:33 | |
| *** openstackgerrit has joined #craton | 11:26 | |
| openstackgerrit | Sulochan Acharya proposed openstack/python-cratonclient master: Include parent_id by default in devices https://review.openstack.org/441888 | 11:26 |
| sulo | really simple review pls ^^^ | 11:27 |
| thomasem | o/ | 13:27 |
| *** VW has joined #craton | 13:41 | |
| *** VW has quit IRC | 13:45 | |
| antonym | if i wanted to track a list of all of the macs for a device, is that something that i could put under the vars for a host? and could i search for that list given just the mac? also, where would the best place to store the other attributes that belong with the ip address, like netmask, gateway, dns, etc ? | 14:05 |
| sulo | antonym: heh, so i did start modeling a network interface / network device and networks separately a while bac | 14:06 |
| sulo | https://github.com/openstack/craton/blob/master/craton/db/sqlalchemy/models.py#L446 | 14:06 |
| sulo | but it needs a bit more work tbh | 14:06 |
| antonym | yeah, i noticed that but it didn't look like it had any linkage from the host | 14:07 |
| sulo | the idea was to link host to interface | 14:07 |
| sulo | i think thats missing | 14:07 |
| sulo | but i wants sure what properties we wanted either | 14:07 |
| sulo | i did try to ping a few network folks to get some feedback but didnt get too far | 14:08 |
| sulo | antonym: if you havea req .. i can work on that | 14:08 |
| sulo | its somehting i wanted to clean up anyway | 14:08 |
| antonym | yeah, main use case is to blindly discover a host booting, push it's macs up and create a record with all attributes from a server, and then the next time it boots, it could use the mac to relook up the exist record | 14:08 |
| sulo | gotcha | 14:09 |
| antonym | ip info so that i can pull everything to configure an interface given that host record | 14:09 |
| antonym | could probably shove all the networking info in as vars for now, but if i want to search by macs, it'd probably be nice to search for a list of vars so i don't need mac1, mac2, etc | 14:11 |
| thomasem | sulo: how does the commit message need to be updated here: https://review.openstack.org/#/c/427032? | 14:11 |
| thomasem | I thought I had updated, wondering what I missed? | 14:12 |
| sulo | antonym: ack | 14:12 |
| sulo | thomasem: ah, i saw craton host-vars-get 3 x=true y=47 z=foo/bar w=3.14159 so i thought that was not implemented and from old patch ? | 14:13 |
| sulo | unless thats set ? | 14:13 |
| sigmavirus | sulo: that looks like set | 14:25 |
| thomasem | sulo: Ohhh, I didn't see that. Thank you, lemme update. | 14:26 |
| fsaad | morning guys! | 14:27 |
| thomasem | hey fsaad! | 14:27 |
| fsaad | hi thomasem | 14:27 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: Add variables support for remaining resources https://review.openstack.org/441627 | 14:27 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: CLI and client support for get/set/delete of resource vars https://review.openstack.org/427032 | 14:27 |
| thomasem | sigmavirus: jimbaker: Took away that nested/recursive conversion for the Variables dict and turned it into just the top-level being wrapped in a Variable object. | 14:28 |
| thomasem | sulo: updated commit message | 14:29 |
| sigmavirus | thomasem: +1 | 14:29 |
| thomasem | Lol whatta patch | 14:30 |
| thomasem | Truly a team effort. | 14:30 |
| *** VW has joined #craton | 14:49 | |
| *** VW has quit IRC | 14:49 | |
| openstackgerrit | Merged openstack/python-cratonclient master: Include parent_id by default in devices https://review.openstack.org/441888 | 14:50 |
| *** VW has joined #craton | 14:50 | |
| *** VW_ has joined #craton | 14:51 | |
| *** VW has quit IRC | 14:51 | |
| *** VW_ has quit IRC | 14:53 | |
| *** VW has joined #craton | 14:53 | |
| *** VW has quit IRC | 14:54 | |
| *** VW has joined #craton | 14:55 | |
| *** VW has quit IRC | 14:56 | |
| *** VW has joined #craton | 14:57 | |
| *** VW has quit IRC | 14:58 | |
| *** VW has joined #craton | 14:58 | |
| jimbaker | thomasem, thanks so much for taking on that vars patch. looking just about there! | 14:59 |
| jimbaker | and yes, truly a team effort | 14:59 |
| sigmavirus | Courtesy reminder, meeting in #openstack-meeting-4 in 1 minute | 14:59 |
| *** VW has quit IRC | 14:59 | |
| sigmavirus | thomasem: want me to chair this one? | 14:59 |
| thomasem | Up to you. I just got the etherpad set up. | 15:00 |
| thomasem | Are you eager to chair? | 15:00 |
| jimbaker | just waiting for the previous meeting to end it looks | 15:00 |
| sigmavirus | thomasem: eager to chair? not really =P but I'm happy to do it if you have feedback from jimbaker to addres | 15:00 |
| thomasem | Oh, none that's come through yet. | 15:01 |
| thomasem | Appreciate it! | 15:01 |
| jimbaker | sigmavirus, thomasem - not yet, i just took the briefest of looks | 15:01 |
| jimbaker | will do so in depth once we are done w/ the upcoming meeting | 15:01 |
| openstackgerrit | Sulochan Acharya proposed openstack/craton master: Fake Data: Add labels to devices https://review.openstack.org/441981 | 15:14 |
| *** david-lyle has quit IRC | 15:20 | |
| sulo | fsaad: demo dry run ? | 15:34 |
| fsaad | yeah just go over it before the actual demo | 15:34 |
| sulo | so i was going to set it up in that box | 15:34 |
| sulo | but for some reason i couldnt get in | 15:34 |
| fsaad | wut | 15:34 |
| sulo | but setup is as easy as running the docker file | 15:34 |
| fsaad | root pw I handed didn't work ? | 15:34 |
| sulo | commands | 15:34 |
| sulo | fsaad: no | 15:34 |
| jimbaker | fsaad, sulo, i will let you take that on | 15:35 |
| sulo | fsaad: also | 15:35 |
| sulo | where does the presentation fit here | 15:35 |
| sulo | fsaad: do you want to do that first then do the demo ? | 15:35 |
| jimbaker | sulo, btw, just so you know: user construction is stricter with the schema change i made in https://review.openstack.org/#/c/441644/ - see https://review.openstack.org/#/c/441644/2/tools/docker_run.sh | 15:36 |
| sulo | fsaad: not sure i followed what format we wanted | 15:36 |
| jimbaker | thomasem, ^^^ | 15:36 |
| fsaad | the intent of the presentation as I understood it was just to go over each UC slide and demo the functionaliity | 15:36 |
| fsaad | is what I understood really | 15:36 |
| jimbaker | trying to avoid future bugs where the data model is out of sync with the database | 15:36 |
| fsaad | the slides don't have to have any meat to them, it's just a title for us to use as guideline on what to demo | 15:36 |
| sulo | jimbaker: stricter in what way ? | 15:37 |
| jimbaker | as in the same bug we saw in projects if it didn't have a variable association | 15:37 |
| sulo | gotcha | 15:37 |
| jimbaker | we also mix in variables with users | 15:37 |
| sulo | i am running master for demo so i havet hit any new patch | 15:38 |
| jimbaker | anything that mixes in variables MUST have a corresponding variable association. the foreign key constraints now enforce this | 15:38 |
| jimbaker | sulo, no worries, just in case you try it out. alembic change is only for production stuff, doesn't change anything demo wise | 15:38 |
| sulo | jimbaker: thats good. we just need to smooth the bootstrap user/project thing | 15:39 |
| sulo | or atleast document it on the intended usage etc | 15:39 |
| sulo | i believe project-create or user-create does the right things | 15:39 |
| sulo | so that first user should really be use and throw | 15:39 |
| jimbaker | sulo, yeah, i think if we do something like https://review.openstack.org/#/c/441644/2/craton/tests/functional/__init__.py | 15:39 |
| jimbaker | but instead it generates high quality crypto credentials, and reports back, that would suffice | 15:39 |
| thomasem | jimbaker: yeah, an initial data migraiton should set that up | 15:40 |
| jimbaker | but not dbsync itself | 15:40 |
| thomasem | was the idea we had discussed previously | 15:40 |
| thomasem | Why? | 15:40 |
| sigmavirus | brb | 15:40 |
| thomasem | Where else are you going to bootstrap that? | 15:40 |
| thomasem | If you make it a part of the base version of the DB, you won't accidentally re-bootstrap on start or anything... otherwise you have to do that logic in the code. | 15:41 |
| jimbaker | thomasem, i suppose we could add some state info to dbsync | 15:41 |
| jimbaker | and get at what you want | 15:41 |
| sulo | it should be aapart of the carton-dbsync command | 15:41 |
| thomasem | Yeah | 15:41 |
| sulo | and to remove them too | 15:41 |
| jimbaker | anyway, the important thing is: only use the model code to construct objects. done. | 15:41 |
| thomasem | sulo: how will we then be able to create a root user for legitimate root purposes? | 15:42 |
| thomasem | Like adding projects | 15:42 |
| thomasem | If we bootstrap, set up, and then remove said bootstrap user | 15:42 |
| thomasem | You don't want an admin user to have the privileges to create root users | 15:42 |
| sulo | create a root user using project create or user craete commands | 15:42 |
| sulo | using the initial root user | 15:42 |
| thomasem | Right | 15:42 |
| jimbaker | for all of this, we should only use the rest api | 15:42 |
| thomasem | I thought you were saying to create the bootstrap user do some migration stuff, then delete it | 15:43 |
| thomasem | I misunderstood, sulo | 15:43 |
| jimbaker | so the initial bootstrap user is constructed | 15:43 |
| jimbaker | we get back its token with a high entropy secret | 15:43 |
| thomasem | jimbaker: correct, and to do that, we need to apply some RBAC around the user creation and add support for creating a root user with that API. | 15:43 |
| jimbaker | everything else can go from there | 15:43 |
| *** Syed__ has joined #craton | 15:44 | |
| jimbaker | if you want, you as the craton deployer can delete this bootstrap user | 15:44 |
| jimbaker | anyway... may i suggest this is a topic for tomorrow? :) | 15:44 |
| thomasem | Sure | 15:44 |
| jimbaker | sorry, very interesting stuff | 15:44 |
| jimbaker | adding to tomorrow's topics... | 15:44 |
| thomasem | It's cool. You linked the review, man. :P | 15:44 |
| sulo | fsaad: so do you want ot quickly go throughthose at first then i will do the demo | 15:45 |
| thomasem | awesome, looking forward to it | 15:45 |
| fsaad | sulo: can do that, all we wanted was to stick to the subject and go in order iirc from last demo, which happened to start jumping through subjects and didn't allow us to get much done | 15:46 |
| fsaad | so I think it'd be good to make sure questions are asked at the end, and we demo each one of the use cases at a time in order that's all | 15:46 |
| jimbaker | right, so just make sure we do the classic presentation: | 15:46 |
| fsaad | will the bulk export be demo-able? https://bugs.launchpad.net/python-cratonclient/+bug/1667109 ? | 15:47 |
| openstack | Launchpad bug 1667109 in Craton's Python Client "Allow environment export from client" [Undecided,New] - Assigned to Ian Cordasco (icordasc) | 15:47 |
| jimbaker | tell me what we are going to say (or in this case, demo), specifically with respect to workflow; demo; tell them what we have demoed | 15:47 |
| *** VW has joined #craton | 15:47 | |
| jimbaker | sigmavirus, looks we should unassign that to you | 15:47 |
| jimbaker | fsaad, just fixed up the labels on https://bugs.launchpad.net/python-cratonclient/+bug/1667109 | 15:48 |
| openstack | Launchpad bug 1667109 in Craton's Python Client "Allow environment export from client" [High,New] | 15:48 |
| jimbaker | (not going to demo today) | 15:49 |
| fsaad | boo | 15:49 |
| jimbaker | sorry | 15:49 |
| fsaad | it's all good ;) | 15:50 |
| * fsaad silently removes slide from presentation hoping no one notices | 15:50 | |
| sulo | heh, there no bulk anything ;) | 15:51 |
| *** VW has quit IRC | 15:51 | |
| jimbaker | not yet. let me add to tomorrow's meeting | 15:51 |
| thomasem | lol fsaad | 15:53 |
| thomasem | jimbaker: you have a lot of stuff assigned to you at once... are those all small changes, or are you feeling like you could use some cover there? | 15:53 |
| thomasem | Asking, because, I'll probably be looking to snag things in the next day or so. | 15:54 |
| * thomasem is overusing commas again | 15:54 | |
| thomasem | https://launchpad.net/craton/+milestone/v0.1.0 | 15:55 |
| thomasem | Looking there | 15:55 |
| jimbaker | thomasem, brb | 15:55 |
| thomasem | okay.gif | 15:55 |
| fsaad | does the bulk export really belong in the client ? thinking it's simpler to do it there but if we're gonna put the work might as well be in the server, unless there's some technical reason I'm oblivious to now | 15:56 |
| thomasem | fsaad: there is this https://bugs.launchpad.net/craton/+bug/1661714 | 15:57 |
| openstack | Launchpad bug 1661714 in craton "Bulk endpoint for working with resources" [High,Confirmed] | 15:57 |
| thomasem | I think we're going to address that server-side. I suspect the client implementation is to get something a bit quicker while we design and implement something server-side. | 15:57 |
| thomasem | At least, that was my understanding. I have been wrong many times before. | 15:58 |
| fsaad | yeah same thomasem thanks | 15:58 |
| fsaad | sulo: wanna do a quick chat ? | 15:58 |
| *** VW has joined #craton | 15:58 | |
| sulo | fsaad: sure, in 5 mins .. just going to get a cup of tea | 15:59 |
| sulo | ? | 15:59 |
| thomasem | Mmm, and biscuits? | 15:59 |
| fsaad | yeah will do the same | 15:59 |
| fsaad | y'all welcome to join just want to chat about the demo and if I can help with anything btw | 15:59 |
| *** VW has quit IRC | 15:59 | |
| sulo | thomasem: yes, for sure | 15:59 |
| *** VW has joined #craton | 16:00 | |
| thomasem | So awesome | 16:00 |
| *** VW has quit IRC | 16:01 | |
| *** VW has joined #craton | 16:01 | |
| *** VW has quit IRC | 16:02 | |
| *** VW has joined #craton | 16:02 | |
| sulo | fsaad: i am in your room | 16:07 |
| *** VW_ has joined #craton | 16:07 | |
| *** VW has quit IRC | 16:10 | |
| jimbaker | thomasem, just removed myself from https://bugs.launchpad.net/craton/+bug/1668308 | 16:16 |
| openstack | Launchpad bug 1668308 in craton "orphan variables on host delete" [High,Confirmed] | 16:16 |
| openstackgerrit | git-harry proposed openstack/craton master: Remove networks validation duplicate details key https://review.openstack.org/442015 | 16:17 |
| jimbaker | i was naively thinking that my alembic work could also support on that, but it should be done at a higher level in model code | 16:17 |
| jimbaker | it should be a straightforward fix however | 16:17 |
| *** david-lyle has joined #craton | 16:17 | |
| thomasem | jimbaker: gotcha. Okey doke! Responsibility is one of my top strengths, so I'm especially sensitive to someone taking on too much for some period of time. :P | 16:17 |
| jimbaker | thomasem, other than that, i do own all of those other bugs | 16:17 |
| thomasem | If you can do all of that at the same time, cool. But, yeah. | 16:18 |
| jimbaker | although if someone had some spare cycles to implement testing for https://review.openstack.org/#/c/440929/, i wouldn't mind | 16:18 |
| fsaad | cloudnull: around to join a vidyo call ? | 16:18 |
| fsaad | s/around/available/ | 16:18 |
| jimbaker | but it's on top of that existing review of course | 16:18 |
| jimbaker | (this time, i don't think it's also, completely refactor the WIP code, as we saw with vars in the client...) | 16:19 |
| cloudnull | i am. is it now ? | 16:19 |
| jimbaker | and speaking of that, time to do that review | 16:19 |
| cloudnull | and where? | 16:19 |
| fsaad | cloudnull: we're in vidyo in my channel, if you can make it | 16:19 |
| cloudnull | joining | 16:19 |
| *** david-lyle has quit IRC | 16:22 | |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: Add variables support for remaining resources https://review.openstack.org/441627 | 16:39 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: DRY out Variables shell tests https://review.openstack.org/442026 | 16:39 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: Adds remaining shell tests for variables https://review.openstack.org/442027 | 16:39 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: DRY out Variables shell tests https://review.openstack.org/442026 | 16:46 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: Adds remaining shell tests for variables https://review.openstack.org/442027 | 16:46 |
| thomasem | Ohhh yay! It's getting a little sunny outside. I think I know a furball that wants a walk. bbiab for lunch. | 16:53 |
| openstackgerrit | Merged openstack/craton master: Remove networks validation duplicate details key https://review.openstack.org/442015 | 17:05 |
| *** david-lyle has joined #craton | 17:18 | |
| *** david-lyle has quit IRC | 17:24 | |
| *** david-lyle has joined #craton | 17:30 | |
| *** jovon has joined #craton | 17:44 | |
| sulo | hey fsaad can you give me that link to the scrip again | 17:44 |
| sulo | vidyo sucks with keeping history | 17:44 |
| fsaad | no probs, sec | 17:45 |
| fsaad | https://github.com/osic/ops-generic/blob/master/tools/osa_import_for_craton/osa_import.py | 17:45 |
| fsaad | hey guys I'm going to change the meeting room to allow Syed to join externally | 17:45 |
| thomasem | back | 17:47 |
| thomasem | thanks jimbaker!! | 17:47 |
| fsaad | wb thomasem | 17:47 |
| thomasem | wty fsaad | 17:47 |
| jimbaker | thomasem, thanks for all that hard work on the vars stuff, nice to see it work so well! | 18:01 |
| jimbaker | noticed we need to update our README.rst on the client, https://github.com/openstack/python-cratonclient (was just checking commit & mirroring progress) | 18:01 |
| sigmavirus | jimbaker: yeah, we need to set up the infra jobs to publish docs I think | 18:02 |
| jimbaker | yep, sounds good to me | 18:04 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: Ensure consistent variables behavior across https://review.openstack.org/442099 | 18:10 |
| openstackgerrit | Thomas Maddox proposed openstack/python-cratonclient master: Ensure consistent vars shell behavior across resources https://review.openstack.org/442099 | 18:11 |
| thomasem | ^^ uses the same test code across all resource for variables shell tests | 18:12 |
| thomasem | jimbaker: you bet! | 18:15 |
| jimbaker | thomasem, doing a fast follow! :) | 18:15 |
| thomasem | lol, yes indeed | 18:16 |
| thomasem | On my walk I said to myself, "Wow... that was pretty crummy copying all of those test around. Should be the same behavior everywhere, let's ensure that." | 18:16 |
| jimbaker | which sounds good to me. to be honest, i did some opportunities for refactoring. but i also decided, working and pretty good code now, better code in the future! :) | 18:17 |
| jimbaker | *did see* | 18:17 |
| thomasem | Agreed. This one was so straightforward it seemed silly not to just take the 5 minutes. | 18:17 |
| thomasem | Better to start off on the right foot | 18:17 |
| thomasem | See several things I'd change even in that patch, but I could iterate all day. We have other priorities. | 18:18 |
| jimbaker | indeed. the good before the perfect. but perfectibility, i'm all in favor of that. *it is a process* | 18:18 |
| thomasem | mhmm | 18:18 |
| thomasem | Alright, onward! | 18:19 |
| openstackgerrit | Ian Cordasco proposed openstack/python-cratonclient master: Add testing plan to specifications https://review.openstack.org/442102 | 18:19 |
| *** VW_ has quit IRC | 18:19 | |
| openstackgerrit | Ian Cordasco proposed openstack/python-cratonclient master: Add testing plan to specifications https://review.openstack.org/442102 | 18:28 |
| sigmavirus | jimbaker: you have feedback on I1f84c29610127de12c292a210fd003ae07bd6462 | 18:29 |
| jimbaker | sigmavirus, yeah, responding. i don't know about 64 or 50 (or maybe 45) for ipaddr fields | 18:32 |
| jimbaker | i didn't actually change that value with this update | 18:33 |
| sigmavirus | jimbaker: just wondering why you chose 64 | 18:33 |
| sigmavirus | ah | 18:33 |
| jimbaker | yeah, sorry, that was potentially a separate change. but we have a mix of automated script building + manual changes | 18:33 |
| jimbaker | and i wanted to make it consistent to the extent i could with how it's done in models.py | 18:34 |
| jimbaker | the other thing you mention is respect to breaking out the sql changes in docker_run.sh | 18:35 |
| jimbaker | so to summarize a comment i'm making, plus a bug report - we should not do this in sql at all | 18:35 |
| openstackgerrit | Merged openstack/python-cratonclient master: Ensure consistent vars shell behavior across resources https://review.openstack.org/442099 | 18:35 |
| jimbaker | it should be done in one level up, using model code | 18:35 |
| jimbaker | the sql in docker_run.sh is not portable, and it's not good long run. we need this done with something that dbsync can call, and maybe it's just exposed by dbsync | 18:36 |
| jimbaker | but cryptographically sound credentials are generated for the bootstrap user, etc | 18:36 |
| jimbaker | so that change in docker_run.sh was necessary purely because it was violating the contract for the users table, as implied by models.User | 18:38 |
| * sigmavirus wasn't aware we cared about portability at this point in time | 18:39 | |
| openstackgerrit | Merged openstack/craton master: Fake Data: Add labels to devices https://review.openstack.org/441981 | 18:39 |
| jimbaker | oh sure. except this is the only usage that's not portable | 18:39 |
| sigmavirus | Sure, but portability isn't something OpenStack in general is concerned about | 18:39 |
| jimbaker | more importantly, the other concerns about gen credentials are more immediately germane | 18:39 |
| sigmavirus | Given MySQL/Maria are the (far and away) dominant databases in use by OS operators, I don't think we should care | 18:40 |
| jimbaker | brb, upcoming demo & all | 18:40 |
| openstackgerrit | Merged openstack/craton master: Documentation Fix for Region Create https://review.openstack.org/440790 | 18:40 |
| openstackgerrit | Merged openstack/craton master: Minor Documentation Cleanup https://review.openstack.org/429797 | 18:41 |
| jimbaker | sigmavirus, postgresql has better json support, but weaker clustering, than mysql. i like to keep us competitive by not locking in against a specific db as they evolve over time | 19:02 |
| sigmavirus | jimbaker: I personally quite prefer Postgres over MySQL/Maria. That said, I expect operators to leverage existing infrastructure in their clouds, which is overwhelmingly MySQL/Maria | 19:03 |
| sigmavirus | I'm basing all of this on the User-Survey results | 19:03 |
| sigmavirus | So I think, until we have actual Postgres users, we should document our existing mysql/maria nonsense prominently so people can actually bootstrap their installs. | 19:04 |
| jimbaker | sure. anyway, the portability of sql is the weaker arg. the more important thing is to go against the models level; and to use python for that | 19:04 |
| *** VW has joined #craton | 19:41 | |
| sigmavirus | Always blame dusty | 19:42 |
| jimbaker | sigmavirus, sounds like a standing agenda item for our meetings: #topic blame dusty | 19:56 |
| sigmavirus | It'd be funnier if dusty were in those | 19:56 |
| jimbaker | +1. need to make that change | 19:57 |
| jimbaker | :) | 19:57 |
| fsaad | hey guys wanted to chat about the inventory load script, and together with it the querying capabilities of craton for those imported variables | 20:26 |
| fsaad | I sort of chatted with thomas a bit already, and will create a ticket as well | 20:26 |
| fsaad | but I think what will be required from the import tool is that all the ansible facts are loaded into craton in a way that are searchable, I used as an example hard drive manufacturer with him | 20:27 |
| fsaad | using as example, the import tool should load the ansible facts like this "hosts.host[0].hardware_profiles.disks.disk[0].manufacturer=>'seagate'" | 20:28 |
| thomasem | I think we can store that, just can't yet query specifically for that? | 20:28 |
| fsaad | such that the user can query "which of my customers have hard drive==seagate" in order to plan for maintenances and the like | 20:28 |
| sigmavirus | fsaad: so, ansible doesn't have that information by default | 20:28 |
| fsaad | across clouds of course | 20:28 |
| fsaad | sigmavirus: the manufacturer is an example, but I think the full info gathered by the setup module would be a great start | 20:29 |
| sigmavirus | thomasem: jimbaker how does one query vars that are nested? | 20:29 |
| sigmavirus | fsaad: sure | 20:29 |
| thomasem | That's the problem | 20:29 |
| sigmavirus | And that information would be discoverable | 20:29 |
| thomasem | the data model doesn't exactly support it | 20:29 |
| sigmavirus | thomasem: that's what I thought | 20:29 |
| sigmavirus | so the variable/metadata info will need to be rather flat for queries like that to be useful | 20:29 |
| thomasem | Yeah | 20:30 |
| fsaad | zz_pwnall138: ^ relevant to our earlier conversation btw | 20:31 |
| sigmavirus | fsaad: I think zz_pwnall138 is sleeping on the job | 20:31 |
| thomasem | Lol | 20:31 |
| fsaad | sigmavirus: well the import script could flatten the environment as such right, like flatten the nested environment name to something like key1.key2.key3? | 20:31 |
| fsaad | instead of key1:{key2:value{key3:value}}} | 20:32 |
| * fsaad hopes he closed all '{'s | 20:32 | |
| fsaad | oh I closed more | 20:32 |
| sigmavirus | lol | 20:32 |
| fsaad | don't know, up for discussion obviously | 20:33 |
| sigmavirus | fsaad: depends on who writes the import script =) | 20:33 |
| fsaad | true, well end goal is to be able to search for those values independent of each other right, so which structure makes more sense ? | 20:33 |
| fsaad | like right now zz_pwnall138 wrote it in a way that all facts are dumped as a single key=>v, where v is the json blob of all facts | 20:34 |
| fsaad | which meets requirement of storing those in craton, but doesn't allow for the customer/support rep to search / filter across customers that meet a particular set of values | 20:34 |
| sigmavirus | fsaad: right | 20:34 |
| sigmavirus | well the other thing, is how the data about the harddrive manufacturer gets there | 20:34 |
| sigmavirus | because let's say we have some little piece of junk that is run by cron on thet customer server that detects that info and updates craton | 20:35 |
| sigmavirus | if it does that, I'd expect it to do something like harddrive.sda.manufacturer = seagate | 20:35 |
| sigmavirus | the problem still occurs that you won't be able to say 'harddrive.manufacturer = seagate' | 20:36 |
| thomasem | The problem is, if you have other keys somewhere along there, it won't match correctly... You couldn't search for {key1: {key2: {key3: value}}} and get back what you expect if the stored data is: {key1: {key2: {key3: value, key4: value}}}, for instance. | 20:36 |
| sigmavirus | to query on * | 20:36 |
| thomasem | Another problem is* | 20:36 |
| sigmavirus | thomasem: right | 20:36 |
| sigmavirus | you'd need to be able to do filter like harddrive.*.manufacturer=seagate | 20:36 |
| sigmavirus | that's *plausible* but probably ill-advised | 20:36 |
| thomasem | So, we see a lot of problems with it as it's currently implemented. Now, what do we do aboot it? | 20:37 |
| antonym | yeah, same situations with macs, if i wanted to capture them all and get a result based on one of them | 20:37 |
| fsaad | this sounds like a problem for elasticsearch dammit | 20:37 |
| fsaad | :) | 20:37 |
| thomasem | Lol | 20:38 |
| sigmavirus | fsaad: so | 20:38 |
| sigmavirus | I mentioned a long time ago that craton + searchlight could do a lot | 20:38 |
| sigmavirus | because searchlight already understands notifications and we could emit notifications so searching through stuff inc raton would be as nice as using elasticsearch | 20:38 |
| sigmavirus | (hint, searchlight uses elasticsearch to do this but does the shim work of taking notifications and putting them in ES and providng multitenancy + rbac) | 20:38 |
| sigmavirus | so technically, there exists something that could make this easier for us | 20:39 |
| sigmavirus | but we also have the problem of craton needing to be small | 20:39 |
| sigmavirus | and self-contained | 20:39 |
| fsaad | yeah, interesting though. | 20:39 |
| sigmavirus | which is why we keep rejecting the idea of using something actually secure to store secrets | 20:39 |
| sigmavirus | so if we can figure out exactly how many external services we want to integrate with/rely on for certain features we can avoid development years | 20:40 |
| sigmavirus | Obviously, we don't want to make deploying Craton a nightmare but I think we need to balance our priorities | 20:41 |
| sigmavirus | Do we want secure secret storage for customers? Do we want highly flexible searching? | 20:42 |
| sigmavirus | And does craton have to do all of those things or can we rely on ohter thigns? | 20:42 |
| sigmavirus | but yeah | 20:44 |
| sigmavirus | we have some options | 20:45 |
| sigmavirus | I suspect that cassandra could solve the searching problem too | 20:45 |
| sigmavirus | (I think that's partly why cloudnull picked it for his poc) | 20:45 |
| fsaad | oh I see | 20:47 |
| *** zz_pwnall138 is now known as pwnall1337 | 20:55 | |
| thomasem | added to https://etherpad.openstack.org/p/craton-meetings | 21:02 |
| thomasem | I think it'd be good for us to set aside some time to truly discuss the future of Craton in that respect. Because, yeah, we could wind up spending a really long time solving things that have already been solved. | 21:03 |
| thomasem | And probably solved in a better way :P | 21:03 |
| sigmavirus | Anyway | 21:05 |
| sigmavirus | I've been hacking since around 6:30 this morning, so I'm going to go for a walk and make dinner | 21:06 |
| sigmavirus | Have a good evening y'all | 21:06 |
| thomasem | sigmavirus: take it easy! Catch ya tomorrow. | 21:07 |
| jimbaker | sigmavirus, two approaches. first full text searching (so searchlight). second json path queries | 21:09 |
| jimbaker | neither is currently implemented, but both are certainly implementable with what we have | 21:10 |
| jimbaker | with no changes to the model | 21:10 |
| jimbaker | https://dev.mysql.com/doc/refman/8.0/en/json-search-functions.html#operator_json-column-path | 21:10 |
| jimbaker | also the current query scheme of conjunctive filters is the smallest possible query implementation that could be useful; and easy enough to express in a url. clearly we can do more, as we figure out what "more" should be | 21:12 |
| -openstackstatus- NOTICE: restarting gerrit to address performance problems | 21:14 | |
| openstackgerrit | Ian Cordasco proposed openstack/python-cratonclient master: WIP: Add Betamax for testing https://review.openstack.org/442165 | 21:17 |
| thomasem | sigmavirus: your bot doing work for you? | 21:17 |
| jimbaker | https://dev.mysql.com/doc/refman/8.0/en/json.html#json-paths | 21:30 |
| thomasem | Let's pull down 8.0 and give it a go? Having a hard time finding when that's supposed to be considered stable? I guess that's a question. Do we care? | 21:31 |
| jimbaker | thomasem, so i look at this longterm | 21:33 |
| thomasem | Sure. I guess the question is more around how soon do we need to support what? Is this some how blocking our short-term goals? | 21:34 |
| jimbaker | at some point, both mysql and mariadb will reach feature parity with postgresql on json - it's clearly not just "roadmap" | 21:34 |
| thomasem | Oh, correct | 21:34 |
| thomasem | Yeah, I get that. | 21:35 |
| jimbaker | postgresql for json btw - https://www.postgresql.org/docs/9.6/static/datatype-json.html - specifically GIN indexes | 21:35 |
| jimbaker | thomasem, in the short term, we can work around | 21:35 |
| thomasem | What workaround are you proposing? :) | 21:36 |
| jimbaker | specifically for short term: so 5.7 also has json support, specifically virtual column indexing. it's not as great as GIN, but it probably will work for us | 21:36 |
| jimbaker | sorry, apparently i need to write this faster... :) | 21:36 |
| thomasem | Not a problem at all. | 21:36 |
| jimbaker | the disadvantage of virtual column indexing is that we need to use some form of structuring to take advantage of it | 21:37 |
| jimbaker | but probably works well enough for "common cases". let's see | 21:37 |
| thomasem | Aha, yeah. It looks like there's a fair amount of support in 5.7. I didn't realize. | 21:38 |
| jimbaker | also there's nothing wrong with scanning records for small data sets. if we are just looking for a matching set of keys first, that probably works | 21:38 |
| jimbaker | as a first pass filter, not to mention the other filters | 21:38 |
| jimbaker | small data sets certainly include 1000 records.... | 21:39 |
| jimbaker | or maybe 10000 | 21:39 |
| jimbaker | or maybe larger | 21:39 |
| jimbaker | given that the records are most likely to be in memory anyway | 21:39 |
| jimbaker | so hopefully that gives context of why went down this path | 21:40 |
| jimbaker | why *we* went | 21:40 |
| thomasem | Yeah, it helps. | 21:40 |
| thomasem | Thank you | 21:40 |
| jimbaker | np | 21:41 |
| *** jovon has quit IRC | 21:52 | |
| fsaad | should I file a ticket with what we've got so far on variable store/retrieval or have a meeting all together and build it up ? | 21:55 |
| fsaad | or do we have one already? :) | 21:55 |
| fsaad | is that https://bugs.launchpad.net/craton/+bug/1661226 ? | 21:56 |
| openstack | Launchpad bug 1661226 in craton "Filter by variables lacks resolved variables" [Critical,In progress] - Assigned to Jim Baker (jimbaker) | 21:56 |
| pwnall1337 | btw https://github.com/osic/ops-generic/tree/master/tools/osa_import_for_craton imports ansible vars | 21:58 |
| pwnall1337 | host facts | 21:59 |
| pwnall1337 | re working it right now to import each host facts as it's own key | 21:59 |
| pwnall1337 | spoke with fsaad | 21:59 |
| thomasem | pwnall1337: So, we support what you need there for the import. The only missing thing is nice queries for nested variables afterwards? | 22:01 |
| thomasem | Is that true? | 22:02 |
| jimbaker | we could look into providing that sooner than later via json path | 22:03 |
| jimbaker | http://docs.sqlalchemy.org/en/latest/core/type_basics.html#sqlalchemy.types.JSON | 22:05 |
| jimbaker | we may need to relax this constraint first in requirements.txt: SQLAlchemy<1.1.0,>=1.0.10 # MIT, and upgrade to SA latest | 22:07 |
| thomasem | Gotcha | 22:09 |
| thomasem | Oh, yeah! How was everyones' weekend? | 22:09 |
| thomasem | Forgot to ask with all of the goings on. | 22:09 |
| fsaad | great! relaxing thankfully. Got a run in a natural area park I like to go on saturday morning, rest was filled with family stuff but mostly hanging out at home ;) | 22:10 |
| thomasem | That sounds awesome! | 22:11 |
| fsaad | I was schooled by a couple 40+'s (I'm 3 years short of that group) | 22:11 |
| fsaad | :) | 22:11 |
| fsaad | thomasem: you ? | 22:11 |
| thomasem | LOL | 22:11 |
| thomasem | :P | 22:11 |
| jimbaker | nice weekend. i did fix up a few things workwise, notably the alembic stuff (and need to respond to sigmavirus on his comments...) | 22:12 |
| fsaad | I meant your weekend not your age btw hehe | 22:12 |
| jimbaker | but also got in some good biking and hiking in beautiful weather | 22:12 |
| fsaad | nice ... weather was pretty meh here, drizzly mostly but not show stopper. Good to hear you had nice weather there | 22:13 |
| thomasem | Mine was nice! Let's see dinner party on Friday (ended up chatting and being silly until like 5am Saturday), Saturday evening met up with some friends at Salt Lick (ohhh man that was good), and Sunday I got to go visit an old friend that I went to college with and catch up over whiskey/cigars. I am SO not used to socializing like that. So, my social meter's on empty about now. :P | 22:14 |
| thomasem | jimbaker: that's excellent! | 22:14 |
| fsaad | haha, that's awesome thomasem | 22:14 |
| * fsaad envies on that salt lick visit | 22:14 | |
| thomasem | I'd never been. They have one hell of an operation there. | 22:15 |
| fsaad | now I want some bbq | 22:15 |
| thomasem | Hahaha | 22:15 |
| jimbaker | i like running at salt lick gulch here in colorado.... | 22:15 |
| jimbaker | great trail runs | 22:15 |
| antonym | man salt lick sounds good right now | 22:15 |
| fsaad | yeah, I've been just once but completely agree, big place, quick service, real good food | 22:15 |
| jimbaker | salt lick bbq is also good | 22:15 |
| jimbaker | and convenient airport location too | 22:15 |
| thomasem | Ohhh jimbaker, that's one of their satellite places. There's the original one in Driftwood, TX that's this complex including a huge restaurant, vineyard, booze shop, etc. super rustic and inviting. | 22:17 |
| jimbaker | yep, i know of the original as well | 22:18 |
| thomasem | Ohhh gotcha, okay, cool. | 22:18 |
| thomasem | I didn't. So, I got surprised when we drove up. | 22:18 |
| fsaad | that once I was there I saw it was so packed I thought it'd be a franklins-like experience of standing in line, but was pleasantly surprised | 22:18 |
| jimbaker | the AUS location is somewhat more convenient, i have only driven slow route between san antonio and austin once, 3 years ago | 22:19 |
| thomasem | Gotcha | 22:19 |
| thomasem | fsaad: yeah, I was surprised by that, also. And the staff didn't even charge us for the kids at the table, which was kind of nice of them. | 22:19 |
| thomasem | Good stuff. Glad folks had good weekends. | 22:20 |
| thomasem | Alrightyo, I need to run for an appointment. Have a lovely evening, everyone!! | 22:26 |
| fsaad | laters thomasem | 22:32 |
| fsaad | I'm gonna transfer home, hopefully avoid some traffic ... bbiab | 22:32 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!