| *** VW has quit IRC | 00:02 | |
| jimbaker | thomasem, ok, sounds good | 00:03 |
|---|---|---|
| jimbaker | looking forward to that conversation, and i will take a look at your code as well | 00:03 |
| jimbaker | but in general the algorithm itself would not care, just need to resolve any leaf issues like this | 00:03 |
| *** jovon has quit IRC | 00:22 | |
| *** VW has joined #craton | 01:32 | |
| *** valw has joined #craton | 03:18 | |
| *** valw has quit IRC | 03:40 | |
| *** zz_pwnall1337 is now known as pwnall1337 | 03:46 | |
| *** pwnall1337 is now known as zz_pwnall1337 | 05:34 | |
| openstackgerrit | Jim Baker proposed openstack/craton master: WIP: Variable search for resources now uses resolved variables. https://review.openstack.org/440929 | 07:26 |
| *** VW_ has joined #craton | 08:21 | |
| *** VW has quit IRC | 08:21 | |
| sulo | what was this openstack thing that we wanted to use as scrum board ? | 10:21 |
| git-harry | sulo: storyboard? | 10:24 |
| sulo | git-harry: thanks | 10:24 |
| *** openstackgerrit has quit IRC | 10:33 | |
| *** VW_ has quit IRC | 13:13 | |
| *** VW has joined #craton | 13:13 | |
| *** valw has joined #craton | 14:18 | |
| *** valw has quit IRC | 14:23 | |
| *** valw has joined #craton | 14:41 | |
| *** valw_ has joined #craton | 14:58 | |
| *** valw has quit IRC | 15:00 | |
| thomasem | Happy Friday, folks. | 15:28 |
| *** Syed__ has joined #craton | 15:32 | |
| thomasem | jimbaker: I think I figured out the problem I was having yesterday. :) | 15:40 |
| thomasem | I'm doing some manual testing, but would appreciate your thoughts when I throw it up in review. | 15:40 |
| thomasem | Essentially, I had to drop the initial matches dict since kv_pairs doesn't map exactly to the result of (variable.key, variable.value) anymore, since the key could be JSON path. So, instead, I build the matches as I got and ensure that len(set(kv_pairs)) == len(matches.keys()), to ensure there's a match for every one. | 15:42 |
| thomasem | That way the intersection computation at the end won't just evaluate the matches that were found, but make sure it's an intersection according to all criteria that was passed in. | 15:43 |
| thomasem | At least, that's how it works in my head. We shall see! | 15:43 |
| thomasem | For the matches dict, I went with a string representation of the dict, so matches[(variable.key, str(variable.value))]. Thinking through whether that will be lossy in any fashion enough to cause a problem here. | 15:45 |
| jimbaker | thomasem, cool. just something to realize: i fixed up the var searching algorithm in the branch you are depending on | 15:48 |
| jimbaker | let's discuss further | 15:48 |
| jimbaker | i did move to using "or" clauses, given that sqlite does not support "in" searches with tuples | 15:49 |
| thomasem | oi | 15:50 |
| thomasem | okay | 15:50 |
| jimbaker | but there should be no performance impact here | 15:50 |
| thomasem | I mean, I did the same thing, because of the JSON column limitation there, too. | 15:50 |
| jimbaker | i understand | 15:50 |
| jimbaker | re the matches dict - it really should not matter what the "key" is | 15:50 |
| jimbaker | the point is to collect the matches that are required for future intersection | 15:51 |
| thomasem | As long as it's unique to the kv_pair? Because the goal appears to be ensuring matches for all kv_pairs that are passed in and then doing an intersection computation on the results | 15:51 |
| jimbaker | but it sounds like the problem is that we cannot do an or clause | 15:51 |
| thomasem | Where'd you get that from? | 15:52 |
| jimbaker | because the matched variables are used to populate the matches | 15:52 |
| thomasem | I'm able to do an or clause just fine | 15:52 |
| jimbaker | no, just thinking about the counting calculation you make. but maybe that works as well | 15:53 |
| jimbaker | the or clause generation is not the interesting part. the interesting piece is determining the intersection | 15:54 |
| jimbaker | with respect to resolution | 15:54 |
| thomasem | Right, and that was having snags because variable.value is now a dict | 15:54 |
| thomasem | correct | 15:54 |
| thomasem | variable.value can now be a dict* | 15:55 |
| thomasem | I mean to say | 15:55 |
| jimbaker | brb, need to transport kid | 15:55 |
| thomasem | k | 15:55 |
| jimbaker | back | 16:11 |
| *** openstackgerrit has joined #craton | 16:17 | |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: JSON Path-like querying for variables https://review.openstack.org/443941 | 16:17 |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: Move to MySQL 5.7 and SQLAlchemy>=1.1.0 https://review.openstack.org/443186 | 16:17 |
| jimbaker | thomasem, taking a look | 16:20 |
| thomasem | jimbaker: cool, I noted a few shortcomings that I'm ideating around right now. | 16:21 |
| thomasem | like properly handling 'true' vs. '"true"' | 16:21 |
| thomasem | and other non-string types | 16:21 |
| thomasem | But, the string case appears to work fine now. :) | 16:22 |
| thomasem | I know it needs bracket support added | 16:22 |
| jimbaker | thomasem, yeah, we need to do it the right way | 16:22 |
| thomasem | Lol, what is the right way? :P | 16:23 |
| thomasem | "right" | 16:23 |
| jimbaker | sure. i like the CLI quoting that the http command does | 16:23 |
| jimbaker | but the right way means, no lossiness | 16:23 |
| jimbaker | so "true" is distinguished from true, etc, | 16:23 |
| thomasem | Yeah, that's the tricky part. I think I'm going to come up with a list of queries that I'd expect to work. | 16:23 |
| thomasem | Right | 16:23 |
| jimbaker | we can expand on the right way. a lot | 16:24 |
| jimbaker | but i think that's a good starting point :) | 16:24 |
| *** valw_ has quit IRC | 16:24 | |
| thomasem | Oh, definitely. I just want a simple starting point that gives folks a way to query any type of value we're storing here. | 16:25 |
| jimbaker | thomasem, and one possibility is that true matches true and "true" | 16:25 |
| thomasem | the leaves of it, at least. | 16:25 |
| thomasem | That's the problem. One may want to store the literal "true" | 16:25 |
| thomasem | or the boolean true | 16:25 |
| jimbaker | unless stricter is desired | 16:25 |
| jimbaker | in the search | 16:25 |
| jimbaker | of course. it's json | 16:25 |
| thomasem | Yep | 16:26 |
| thomasem | Now, to your point, the question may be "who cares?" | 16:26 |
| thomasem | Chances are they just want the one that says "true", regardless of whether it's bool or string | 16:26 |
| thomasem | or ones that say* | 16:27 |
| thomasem | And trying to distinguish may be adding power that confuses more than helps | 16:27 |
| thomasem | Assuming that's what you were getting at. Don't want to put words in your mouth. | 16:28 |
| thomasem | Or hands, as the case may be. | 16:28 |
| jimbaker | yeah, my words are: let's match both the string and the non-string, if valid. but we have to count properly of course | 16:29 |
| jimbaker | so 3.14159 or "3.14159" etc | 16:29 |
| thomasem | Hmmm, okay. | 16:30 |
| jimbaker | although maybe that also means using https://www.python.org/dev/peps/pep-0485/ - anyway, we will see if necessary | 16:31 |
| thomasem | Alright, so, then, why store as anything other than a string? | 16:32 |
| jimbaker | we are not doing calculations, so my understanding is that json presumes a specific IEEE repr for floats | 16:32 |
| jimbaker | thomasem, because it's json | 16:32 |
| jimbaker | i think it's ok. it only comes up in what, a couple of places? | 16:33 |
| jimbaker | i think we are fine | 16:33 |
| thomasem | I mean, why do we do the value conversion if we're only ever going to query it as a string? | 16:33 |
| thomasem | To me, the answer would be so we're not lossy. | 16:34 |
| thomasem | So we can eventually move to distinguishing queries, if desired. | 16:34 |
| thomasem | Without having to migrate the data | 16:34 |
| openstackgerrit | Merged openstack/python-cratonclient master: Add testing plan to specifications https://review.openstack.org/442102 | 16:35 |
| jimbaker | ok, with these thoughts to ruminate on | 16:35 |
| jimbaker | let me look at your branch | 16:35 |
| thomasem | mmkay | 16:35 |
| *** jimbaker has quit IRC | 16:39 | |
| thomasem | One way around this is to essentially make the API expect a JSON value in vars queries | 16:40 |
| thomasem | So, you'd do like vars=neutron_l2_population:true,nova_console_type:"novnc",glance_default_store:"swift" | 16:41 |
| thomasem | vars=neutron_l2_population:true,nova_console_type:"novnc",glance_default_store:"swift",child-nested.bumbleywump.cucumberpatch:"haha" | 16:41 |
| *** jimbaker has joined #craton | 16:42 | |
| *** jimbaker is now known as Guest67010 | 16:42 | |
| *** Guest67010 is now known as jimbaker | 16:43 | |
| *** jimbaker has joined #craton | 16:43 | |
| *** ChanServ sets mode: +o jimbaker | 16:43 | |
| thomasem | jimbaker: did you lose what I said? | 16:44 |
| jimbaker | sorry bounced bouncer. will check log | 16:45 |
| thomasem | No problemo | 16:45 |
| jimbaker | thomasem, so i think this is fine. i was just suggesting with my ref to httpie that we could have a special syntax for a string (or perhaps a non-strict string or non-string match) | 16:47 |
| jimbaker | but this is more of a CLI problem in any event | 16:47 |
| jimbaker | the rest api should be reasonably friendy | 16:47 |
| jimbaker | but it need not be crazy either on our side | 16:47 |
| jimbaker | make sense? we are making it up as we go of course! | 16:47 |
| jimbaker | :) | 16:48 |
| jimbaker | but i like the dotted notation on the key side, that looks nice | 16:48 |
| thomasem | I mean, I could write the logic to do the approximate equality and json.dumps when necessary, too. Just a matter of the UX we want. :) I'd really like to understand what the customers would want for this. | 16:50 |
| thomasem | Do they ever see themselves caring about the difference between 3 and "3" or true and "true"? | 16:50 |
| thomasem | If not, we can make the API easier. If so, it's probably better to go the more strict route so they get EXACTLY what they asked for. | 16:51 |
| thomasem | Could also add a strict flag, but... I dunno. Feels like we're overcomplicating the situation at that point. | 16:52 |
| *** logan_ has joined #craton | 16:56 | |
| *** logan_ is now known as Guest29589 | 16:57 | |
| *** logan- has quit IRC | 16:59 | |
| *** Guest29589 is now known as logan- | 17:03 | |
| jimbaker | thomasem, gave some thought to this | 17:12 |
| jimbaker | thomasem, gave some thought to this | 17:15 |
| jimbaker | let's have a simple expression language, which is non strict for strings vs nonstrings, only supports , for conjunction | 17:15 |
| jimbaker | this is the language we are working out right now. seems to be fine | 17:15 |
| jimbaker | then we reserve the following for a future query expression language | 17:15 |
| jimbaker | vars={....} | 17:15 |
| jimbaker | so curly braces indicates this reserved future expr lang | 17:15 |
| jimbaker | it will allow for full quoting, json paths, the use of & and | for conj and disj | 17:15 |
| jimbaker | etc | 17:15 |
| jimbaker | so someone could say vars={foo->json_path | bar:"foo" & "you-want-a-var-with-hyphens-sure-you-can-do-that": true} | 17:16 |
| jimbaker | or maybe "or" and "and" | 17:17 |
| jimbaker | of course all this would be url quoted, as expected | 17:17 |
| thomasem | And what portion of that do you feel falls in scope on this work I'm doing right now? | 17:18 |
| jimbaker | vars={foo->json_path or bar:"foo" and "you-want-a-var-with-hyphens-sure-you-can-do-that": true} | 17:18 |
| jimbaker | none of this, except to possibly reserve {...} | 17:18 |
| jimbaker | :) | 17:18 |
| thomasem | Okay, so need a regex filter for our validators | 17:18 |
| jimbaker | i just want to limit scope now by pointing at this exciting new feature we may in the future develop :) | 17:18 |
| thomasem | Right. Yeah, I see nothing wrong with evolving this into something far more powerful. | 17:19 |
| thomasem | rather, I agree that it'd be nice. But, yeah, let's drive that with use-cases | 17:19 |
| jimbaker | this also lets me off the hook on the vars should resolve or not | 17:19 |
| jimbaker | for search | 17:19 |
| thomasem | How do you mean? | 17:19 |
| jimbaker | because we can just push that into the future expr language | 17:19 |
| jimbaker | so let's just say, and i'm just making it up right now/brainstorming | 17:20 |
| jimbaker | vars={foo=:bar and baz:42} | 17:21 |
| jimbaker | foo= means do not resolve | 17:21 |
| jimbaker | baz implies resolution | 17:22 |
| jimbaker | or maybe it is | 17:22 |
| thomasem | Wait... why would we not want resolved by default? | 17:22 |
| jimbaker | vars={foo:bar and baz^:42} | 17:22 |
| jimbaker | thomasem, my point entirely | 17:22 |
| jimbaker | i want to push out into the future expr lang how to indicate "i don't want it resolved, which should be done on a var usage by var usage basis btw" | 17:23 |
| thomasem | What? I'm interpreting what you're saying as: let's abandon https://review.openstack.org/#/c/440929/4 | 17:23 |
| jimbaker | sometimes face to face is more effective :) | 17:23 |
| jimbaker | but never | 17:23 |
| jimbaker | in terms of abandoning | 17:23 |
| jimbaker | (not never to "face to face" !!!) | 17:24 |
| thomasem | Right | 17:24 |
| jimbaker | (see ambiguity is everywhere. just got to get used to it online!) | 17:24 |
| thomasem | So, for your patch https://review.openstack.org/#/c/440929/, you were going to add a way to tell it not to resolve? | 17:24 |
| jimbaker | i like not having to do something today that can be done tomorrow | 17:24 |
| jimbaker | thomasem, i was | 17:24 |
| thomasem | but now you're suggesting that part isn't necessary? | 17:24 |
| jimbaker | not anymore | 17:24 |
| thomasem | Okay, I think I gotcha now. Cool. | 17:25 |
| thomasem | Okay, A/C guy's here, so I may disappear unexpectedly. | 17:25 |
| thomasem | Hooray for pre-summer maintenance. | 17:25 |
| jimbaker | exactly. it's the "I'll gladly pay you Tuesday for a hamburger today" | 17:25 |
| jimbaker | hopefully my roof guy shows up | 17:26 |
| jimbaker | it's been an excessively windy winter, and spring is almost here. got to get my battered roof fixed. (had roofer cancel multiple times, due to being too windy) | 17:26 |
| thomasem | Don't let the thing come off your house! | 17:27 |
| thomasem | I've seen Twister. | 17:27 |
| thomasem | Why do I imagine this kludge of bent nails and bungie cords keeping it all together? :P | 17:27 |
| jimbaker | fortunately we are just west of the line where tornadoes can occur in colorado. literally just a few miles in fact | 17:27 |
| thomasem | Oh man, that's good. | 17:28 |
| jimbaker | but the flip side is, we live close enough to the mountains to get battered by winds coming off of them | 17:28 |
| jimbaker | so you win some, you lose some | 17:28 |
| jimbaker | anyway, supposedly i have hurricane resistant shingles, whatever that means. but installation is everything, and the builder just pumped out roofs when they were going through the subdivision dev | 17:29 |
| jimbaker | that, and 100 mph winds, and stuff flies | 17:30 |
| jimbaker | so my wife, who is wfh today, informed me they have started the roof repair. (obviously not above me yet) | 17:33 |
| jimbaker | (i would hear the hammers) | 17:33 |
| thomasem | Lol, I see | 17:59 |
| thomasem | We never get winds like that hear... that's got to be crazy. | 17:59 |
| thomasem | Alright, implemented some JSON approximation that'll handle these values a bit better. :) | 18:02 |
| thomasem | vars=neutron_l2_population:true,test:"true",nova_console_type:novnc,child-nested.bumbleywump.cucumberpatch:"haha"' | 18:04 |
| thomasem | it'll handle novnc and "haha" the same way, just json.dumps(...) the novnc one. | 18:04 |
| thomasem | So, basically, if you want the value as a string literal, you can wrap it in quotes. That way you can distinguish "true" from true or 1 from "1" | 18:05 |
| thomasem | for example. | 18:05 |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: JSON Path-like querying for variables https://review.openstack.org/443941 | 18:06 |
| thomasem | Alright, time to start messing with brackets | 18:07 |
| thomasem | Wow, it appears to "just work" | 18:14 |
| thomasem | Okay, let me make this a bit more complicated | 18:14 |
| sulo | hehe | 18:15 |
| thomasem | so root-level arrays bomb. Need to handle those. | 18:17 |
| thomasem | Naturally because of the way the first item in the path is actually the key_ in the data modedl. | 18:17 |
| thomasem | model* | 18:17 |
| jimbaker | thomasem, nice | 18:23 |
| jimbaker | sulo, so what do you think of my proposal for a future expression language for queries? for now we just reserve vars={...} for it | 18:23 |
| jimbaker | or maybe it can be more like q={...} | 18:23 |
| jimbaker | hmmm, that might be much nicer | 18:24 |
| jimbaker | another possibility is we have discussed elasticsearch (possibly through searchlight, although i'm not sure exactly what we get with searchlight) | 18:28 |
| jimbaker | the q={} query might possibly work with elasticsearch, although it's unclear to me how we would teach it about var resolution | 18:29 |
| jimbaker | it does know about json docs however, https://www.elastic.co/guide/en/elasticsearch/reference/1.4/mapping-object-type.html | 18:29 |
| thomasem | Yeah, i do have to wonder at what point it's better to offload that sort of concern. | 18:37 |
| thomasem | There are other solutions with far more powerful query languages, and it'd be cumbersome for us to re-invent them. | 18:37 |
| jimbaker | so the point is, we can put in this escape valve to consider for later | 18:37 |
| jimbaker | having faceted, flexible search is a huge win | 18:37 |
| thomasem | Yeah, however we wish to get it. I do think this nested query'ing capability will take care of a large portion of that use-case, though. At least the immediate one. | 18:38 |
| jimbaker | and it's easy enough to sync with elasticsearch. the only question is adding extras like resolution, etc, but i suspect we just denormalize as necessary | 18:38 |
| jimbaker | for sure | 18:38 |
| thomasem | So... apparently I run my A/C too much during the summer and my motor is probably going in the next 2 years. | 18:39 |
| thomasem | for my outside unit. | 18:39 |
| * thomasem sighs | 18:39 | |
| thomasem | The list of lessons learned from buying this house is enormous. | 18:39 |
| thomasem | Gut reaction is just build my own, but of course that has its own problems. | 18:40 |
| tojuvone | Evening deeper level of the continent or how it is described | 18:45 |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: JSON Path-like querying for variables https://review.openstack.org/443941 | 19:19 |
| thomasem | Alrighty I can now specify paths to any of these values: {"foo": [1,2,true,"true",{"dangling": "doc"}]} | 19:24 |
| thomasem | and get back the hosts that match | 19:24 |
| thomasem | like vars=foo[*]:1 or vars=foo[2]:true or vars=foo[*].dangling:doc | 19:25 |
| thomasem | Oh dear... found a bug | 19:28 |
| thomasem | Need to be able to support multiples like vars=foo[3]:true,foo[*].dangling:doc | 19:28 |
| thomasem | where it returns the same variable to match each. | 19:28 |
| thomasem | That's where simply counting and comparing lengths won't work. | 19:29 |
| openstackgerrit | Merged openstack/craton master: Add http_codes decorator where missing https://review.openstack.org/443565 | 19:43 |
| openstackgerrit | Merged openstack/craton master: Validate parent_id is neither id nor a descendant https://review.openstack.org/443566 | 19:43 |
| jimbaker | i have reached out to diablo rojo (kendall nelson), and we should be next in line to move to storyboard, after monasca and refstack. so in a couple of weeks | 20:02 |
| jimbaker | however they have the process automated so that it will load up the launchpad bugs/blueprints and put into storyboard. so we just need to cut over at the right time, and it should be good | 20:02 |
| jimbaker | note that storyboard has ubuntu login integration, so it should just transfer over in terms of being able to use existing creds | 20:05 |
| thomasem | Oh, there's a line? | 20:11 |
| thomasem | So, I guess we can't move to it next week, then. | 20:12 |
| jimbaker | thomasem, nope | 20:13 |
| jimbaker | but it's ok, because we can continue to use LP, and it will transfer over. so that's actually even better - no manual work on our part | 20:14 |
| jimbaker | i'm sure adjustments will need to be made. and more tagging. | 20:14 |
| thomasem | Ideally! Depending on how that all maps. :P | 20:14 |
| thomasem | Yep | 20:14 |
| jimbaker | yeah, but raw text is pretty good already | 20:14 |
| thomasem | k | 20:14 |
| jimbaker | and it sounds like that they do better than that | 20:15 |
| jimbaker | we shall see | 20:15 |
| thomasem | Excellent | 20:17 |
| thomasem | huzzah, unicode support | 21:01 |
| thomasem | http 'http://127.0.0.1:8080/v1/hosts?details=all&vars=schön[*]:schön' | 21:01 |
| thomasem | works | 21:01 |
| thomasem | Boy time flies when you're ideating a million nuances. | 21:16 |
| jimbaker | nice! and yes it does | 21:21 |
| thomasem | jimbaker: I'm about to push a rebased revision of your patch up | 21:25 |
| jimbaker | thomasem, you beat me to it. sounds good, please go ahead | 21:25 |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: JSON Path-like querying for variables https://review.openstack.org/443941 | 21:25 |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: Move to MySQL 5.7 and SQLAlchemy>=1.1.0 https://review.openstack.org/443186 | 21:25 |
| openstackgerrit | Thomas Maddox proposed openstack/craton master: WIP: Variable search for resources now uses resolved variables. https://review.openstack.org/440929 | 21:25 |
| jimbaker | it is the most minor rebasing | 21:25 |
| thomasem | Yeah, just some displaced tests | 21:26 |
| thomasem | :P | 21:26 |
| jimbaker | need to get back & add some more in fact | 21:26 |
| jimbaker | ok, will do that now | 21:26 |
| jimbaker | brb | 21:26 |
| thomasem | Agh, sorry Syed__. I lost track of time. :\ | 21:28 |
| thomasem | This needs +2 https://review.openstack.org/#/c/443170/ | 21:53 |
| thomasem | and workflow | 21:53 |
| thomasem | Could we get eyes on it so we can move forward with the Alembic changes, too? | 21:54 |
| thomasem | Since that makes some of the Alembic changes unnecessary | 21:54 |
| thomasem | Here's an updated deploy script https://gist.github.com/jimbaker/e74a7b98bc60519033fd455a22163ad2#gistcomment-2023979 | 21:56 |
| thomasem | if you'd like to pull it down and check it out | 21:56 |
| thomasem | Time for me to run. Have a lovely weekend, everyone! | 22:08 |
| jimbaker | thomasem, i will review | 22:42 |
| jimbaker | also i'm going to fix up your json path patch | 22:42 |
| jimbaker | have a good one! | 22:42 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!