Thursday, 2016-01-21

*** bpokorny_ has joined #openstack-searchlight00:00
*** bpokorny has quit IRC00:04
*** lei-zh- has quit IRC00:04
*** lakshmiS has joined #openstack-searchlight00:11
openstackgerritTravis Tripp proposed openstack/searchlight: Add Devstack install support for searchlightclient  https://review.openstack.org/26835800:20
openstackgerritTravis Tripp proposed openstack/searchlight: Add Devstack install support for searchlightclient  https://review.openstack.org/26835800:21
openstackgerritLi Yingjun proposed openstack/python-searchlightclient: Add facet list support  https://review.openstack.org/25502700:37
openstackgerritLi Yingjun proposed openstack/python-searchlightclient: Added search resource client and cli  https://review.openstack.org/26089900:37
openstackgerritLi Yingjun proposed openstack/python-searchlightclient: Add resource type list CLI  https://review.openstack.org/24907600:37
openstackgerritMerged openstack/python-searchlightclient: Add resource type list CLI  https://review.openstack.org/24907600:50
openstackgerritMerged openstack/python-searchlightclient: Add facet list support  https://review.openstack.org/25502700:52
*** bpokorny_ has quit IRC00:55
*** bpokorny has joined #openstack-searchlight00:55
*** bpokorny has quit IRC01:01
*** yingjun has joined #openstack-searchlight01:07
*** lakshmiS has quit IRC01:10
*** TravT has quit IRC01:51
*** TravT has joined #openstack-searchlight01:52
*** TravT has quit IRC01:52
*** TravT has joined #openstack-searchlight01:52
*** TravT has quit IRC01:57
*** bpokorny has joined #openstack-searchlight02:48
*** TravT has joined #openstack-searchlight02:58
*** TravT has quit IRC03:06
*** bpokorny_ has joined #openstack-searchlight03:17
*** bpokorny has quit IRC03:21
*** bpokorny_ has quit IRC03:22
*** TravT has joined #openstack-searchlight03:31
*** TravT has quit IRC03:36
*** TravT has joined #openstack-searchlight03:40
*** TravT has quit IRC03:49
*** TravT has joined #openstack-searchlight03:54
*** TravT has quit IRC04:08
*** TravT has joined #openstack-searchlight04:13
*** TravT has quit IRC04:17
*** TravT has joined #openstack-searchlight04:19
*** TravT has quit IRC04:25
*** TravT has joined #openstack-searchlight04:28
*** TravT has quit IRC04:37
*** bpokorny has joined #openstack-searchlight04:38
*** TravT has joined #openstack-searchlight04:42
*** TravT has quit IRC04:47
*** TravT has joined #openstack-searchlight04:52
*** TravT has quit IRC04:57
*** TravT has joined #openstack-searchlight04:57
*** TravT has quit IRC05:02
*** david-lyle has quit IRC05:02
*** david-lyle has joined #openstack-searchlight05:03
*** TravT has joined #openstack-searchlight05:07
*** TravT has quit IRC05:16
*** TravT has joined #openstack-searchlight05:21
*** TravT_ has joined #openstack-searchlight05:35
*** TravT has quit IRC05:35
*** TravT_ has quit IRC05:44
*** TravT has joined #openstack-searchlight05:49
*** TravT has quit IRC05:54
*** TravT has joined #openstack-searchlight06:08
*** bpokorny has quit IRC06:12
*** TravT has quit IRC06:12
*** TravT has joined #openstack-searchlight06:17
*** TravT has quit IRC06:22
*** TravT has joined #openstack-searchlight06:27
*** TravT has quit IRC06:31
*** TravT has joined #openstack-searchlight06:32
*** TravT has quit IRC06:37
*** TravT has joined #openstack-searchlight06:41
*** TravT has quit IRC06:46
*** TravT has joined #openstack-searchlight06:51
*** TravT has quit IRC06:55
openstackgerritAndreas Jaeger proposed openstack/python-searchlightclient: Remove argparse from requirements  https://review.openstack.org/27039006:56
*** TravT has joined #openstack-searchlight07:00
*** lakshmiS has joined #openstack-searchlight07:06
*** lakshmiS has quit IRC07:12
*** TravT has quit IRC07:19
*** TravT has joined #openstack-searchlight07:23
*** TravT has quit IRC07:28
*** TravT has joined #openstack-searchlight07:33
*** TravT has quit IRC07:37
*** TravT has joined #openstack-searchlight08:25
*** TravT has quit IRC08:39
*** TravT has joined #openstack-searchlight08:44
*** TravT has quit IRC08:48
*** exploreshaifali has joined #openstack-searchlight09:31
*** yingjun has quit IRC09:33
*** TravT has joined #openstack-searchlight09:36
*** TravT has quit IRC09:41
*** TravT has joined #openstack-searchlight09:45
*** TravT has quit IRC09:50
*** TravT has joined #openstack-searchlight09:55
*** TravT has quit IRC10:00
*** openstackgerrit has quit IRC10:02
*** openstackgerrit has joined #openstack-searchlight10:03
*** TravT has joined #openstack-searchlight10:05
*** TravT has quit IRC10:10
*** TravT has joined #openstack-searchlight10:29
*** TravT has quit IRC10:38
*** TravT has joined #openstack-searchlight10:58
*** TravT has quit IRC11:02
*** TravT has joined #openstack-searchlight11:07
*** TravT has quit IRC11:21
*** TravT has joined #openstack-searchlight11:25
*** TravT has quit IRC11:30
*** TravT has joined #openstack-searchlight11:31
*** TravT has quit IRC11:35
*** yingjun has joined #openstack-searchlight11:37
*** TravT has joined #openstack-searchlight11:40
*** TravT has quit IRC11:44
*** TravT has joined #openstack-searchlight11:45
*** TravT has quit IRC11:54
*** TravT has joined #openstack-searchlight12:18
*** TravT has quit IRC12:23
*** TravT has joined #openstack-searchlight12:28
*** TravT has quit IRC12:42
*** TravT has joined #openstack-searchlight12:47
*** TravT has quit IRC12:56
*** TravT has joined #openstack-searchlight13:00
*** TravT has quit IRC13:05
*** TravT has joined #openstack-searchlight13:10
*** TravT has quit IRC13:24
*** TravT has joined #openstack-searchlight13:25
*** TravT has quit IRC13:29
*** TravT has joined #openstack-searchlight13:34
*** TravT has quit IRC13:38
*** TravT has joined #openstack-searchlight13:44
*** TravT has quit IRC13:48
*** TravT has joined #openstack-searchlight13:53
*** exploreshaifali has quit IRC13:54
*** TravT has quit IRC13:58
*** TravT has joined #openstack-searchlight14:03
*** TravT has quit IRC14:12
*** TravT_ has joined #openstack-searchlight14:12
*** TravT_ has quit IRC14:17
*** lakshmiS has joined #openstack-searchlight14:30
*** TravT has joined #openstack-searchlight14:42
*** TravT has quit IRC14:51
*** TravT has joined #openstack-searchlight14:52
TravTCourtesy Searchlight meeting reminder in #openstack-meeting-4: lakshmiS, nikhil_k, rosmaita, TravT, david-lyle, kragniz, sjmc7, ekarlso, abhijeetm, itisha, briancline, lei-zh, yingjun14:59
sjmc7ta15:01
rosmaitao/15:01
TravThey rosmaita: perhaps you meant to wave o/ in the other room ;)15:01
rosmaitaoops15:02
*** sigmavirus24_awa is now known as sigmavirus2415:03
sjmc7you’re just so happy!15:14
*** TravT_ has joined #openstack-searchlight15:21
*** TravT has quit IRC15:25
*** TravT_ is now known as TravT15:25
*** TravT_ has joined #openstack-searchlight15:34
*** TravT has quit IRC15:37
*** TravT_ is now known as TravT15:38
*** lakshmiS has quit IRC16:00
*** dhellmann has joined #openstack-searchlight16:07
openstackgerritLi Yingjun proposed openstack/python-searchlightclient: Remove dash for resource-type in cli  https://review.openstack.org/27089416:20
*** yingjun has quit IRC16:25
*** bpokorny has joined #openstack-searchlight16:44
*** bpokorny has quit IRC16:51
*** RickA-HP has joined #openstack-searchlight16:51
*** bpokorny has joined #openstack-searchlight16:52
*** bpokorny has quit IRC16:53
sjmc7TravT: RickA-HP : rather than playing gerrit tag on https://review.openstack.org/#/c/245222/8/specs/mitaka/zero-downtime-reindexing.rst, i think i’m still a little confused around assigning aliases to resource types. if two resource types, ‘A’ and ‘B’ are both on the same logical index ‘index-1’ which has an alias ‘index’, i don’t think it’s possible to reindex and reassign the alias for type ‘A’ but not ‘B'17:00
sjmc7which is what i mean when i say that i don’t think it makes sense to have an alias per resource type17:00
TravTi think we need a diagram or whiteboard or something17:01
sjmc7i didn’t state the example terribly well, re-reading it17:01
sjmc7ok, so ‘A’ and ‘B’ has their aliases, a-index and b-index17:01
sjmc7i decide to reindex A (or both)17:01
sjmc7and at some popint, a-index and b-index end up oiinting at different logical indices17:02
sjmc7now there are two copies of ‘A’ and ‘B’s data17:02
sjmc7it’s not possible to run a query against a-index and b-index and differentiate which copies you’ll query17:02
RickA-HPToo make sure I understand this, I'm going to rename thing. Resource Type (RT) A has an alias "aliasA" which points to index "index1". RT B has an alias "aliasB" which also points to index "index1". Is this correct so far?17:04
sjmc7right17:04
RickA-HPNow RT A decides to re-index. We create a new index "index2" We update "aliasA" to point to both "index1" and "index2".17:05
sjmc7umm.. no17:05
sjmc7we flip aliasA to point at index2 once it’s done17:05
RickA-HPI haven't got that far yet :)17:05
sjmc7these are search indexes17:06
sjmc7not listener17:06
sjmc7sorry - search aliases17:07
RickA-HPOk. RT A listener has alias "aliasA1" pointing to index "index1". RT A search has alias "aliasA2" pointing to "index1". Same with RT B. Listener has alias "aliasB1" pointing to index "index1" and Search has alias "aliasB2" pointing to "index1".17:08
RickA-HPWHen re-syncing RT A, we update "aliasA1" (listener) to point to "index1" and the newly created index "index2".17:09
RickA-HP"aliasA2" still points to "index1".17:09
sjmc7umm.. ok, although i don’t think it’s important to have separate aliases for listeners either. but for the sake of your example, yes17:09
TravTokay, for fun, i started drawing this17:10
TravThttps://docs.google.com/presentation/d/1T35UXexvafxTaJ6Mw8JXrtSvmToZT5yqvufugsXxJ-4/edit?usp=sharing17:10
RickA-HPThe updates from RT A go into both "index1" anmd "index2". The re-sync for RT A also goes into "index1" and "index2".17:10
RickA-HPRT A searches still go to "index1".17:11
sjmc7ok17:11
RickA-HPSince RT B is associated with "index1", we update "aliasB1" (listener) to incldue "index1" and "index2" and start a resync.17:11
sjmc7ok, so this is where my question 1 comes, but i’ll let you finish17:12
RickA-HPNow new updates from RT  B go into "index1" and "index2" and the re-sync of RT B goes into "index1" and "index2".17:12
RickA-HPThe RT A search is still going to "index1".17:12
RickA-HPWhen both RT A and RT B re-sync have compelted. We change "aliasA2" (Search) to "index2".17:12
sjmc7ok17:12
sjmc7this is where question 2 cdomes in17:13
sjmc7btu carry on17:13
RickA-HPWe finally switch "aliasA1" and "aliasB1" to "index2" only.17:13
RickA-HPWhew!17:13
RickA-HPTravt: Good job keeping up with the Google Docs!17:13
RickA-HPThe assumption also is that the duplicate data already in index1 from the resync gets tossed by ES due to versioning.17:14
sjmc7ok, so here’s question 117:15
sjmc7what beenfit is there having separate aliases for A’s listener and B’s listener? why couldn’t we create index2 and the mappings, add index2 to a single alias, then have A and B start pushing data into it?17:16
sjmc7if it’s easier we can do voice. i can update the spec review afterwards17:17
RickA-HPLogically, there is no benefit and "aliasA1" == "aliasB1". In terms of the current implementation does each listener store it's own index?17:17
sjmc7i am enjoying watching travis draw though. it’s like the Bob Ross of Visio17:17
RickA-HPTravis doesn't have the hair for being Bob Ross. No on the other hand myself...17:18
sjmc7each plugin has a configuration option for the index it uses17:18
RickA-HPSo in my example, "aliasA1" is the variable within the plugin that stores the alias value. My assumption that values of the variables "aliasA1" and "aliasB1" would be the same.17:19
RickA-HPI actually diagrammed most of this in Visio on Tuesday. I will add it to the blueprint.17:20
sjmc7so there is NOT an alias per resource type17:20
sjmc7there’s an alias per index, that multiple RTs will use17:20
RickA-HPYes.17:20
RickA-HPSounds like the blueprint needs some nomenclature!17:21
sjmc7well, you say that we have aliasA1 include index2, but that’s not correct17:21
sjmc7we have the alias to which aliasA1 refers to include index217:21
sjmc7the reason i’m asking is that you and travis both disagreed wiht me on the review, and i think it’s an important distinction17:22
sjmc7and we need to get him one of those drawing tablets17:22
RickA-HPYes, "aliasA1" the variable inside of the plug-in for RT A points to the physcial "aliasXX" in ES. The physical alias "aliasXX" contains a reference to "index1"17:23
RickA-HPThe value of "aliasA1" should almost never change.17:24
RickA-HPInstead, we will change the physical ES alias "aliasXX".17:24
RickA-HPsjmc7: Do you still prefer a phone conferece or continue?17:25
sjmc7no, i think we’re agreeing17:25
TravThmm... so, rick, you are saying the alias in the config file is not an ES alias?17:25
sjmc7it’ll be like now - the nova plugin and glance plugin both default to index_name=searchlight17:26
TravTlet's do a hangout17:26
sjmc7except it’ll be an alias, not a logical index17:26
sjmc7ok17:26
TravThttps://hangouts.google.com/call/gpavaaggm44jlsra3j6rhk2yima17:27
*** sigmavirus24 is now known as sigmavirus24_awa17:27
*** lakshmiS has joined #openstack-searchlight18:11
*** lakshmiS has quit IRC18:29
*** bpokorny has joined #openstack-searchlight18:44
*** lakshmiS has joined #openstack-searchlight18:51
*** sigmavirus24_awa is now known as sigmavirus2419:02
*** TravT has quit IRC19:46
*** TravT has joined #openstack-searchlight19:56
*** TravT has quit IRC20:00
*** TravT has joined #openstack-searchlight20:14
TravTsjmc7 lakshmiS RickA-HP (if you aren't too busy)... just looking at the final search query CLI patch21:04
TravThttps://review.openstack.org/#/c/260899/21:04
TravTi'm a little torn on whether this is an example like we spoke this morning where we could push it in and then iterate on formatting later21:05
sjmc7with this one, to me it looks kind of broken21:05
TravTi don't want to lose the --details part21:05
sjmc7the —detail output is terribly confusing21:05
TravTbut the formatting is quite bad21:06
sjmc7yeah, me neither21:06
sjmc7maybe just a json blob?21:06
sjmc7a massive json blob21:06
sjmc7i don’t have a good answer, unfortunately. it’s sort of a problem inherent to unstructured search21:07
sjmc7perhaps the plugin could define a list of fields?21:07
sjmc7rather than the client?21:07
TravTstill a lot of problematic maintenance21:07
TravTi don't have this solved on horizon side either21:07
sjmc7yeah.. but at least it’s on the server side where the plugins live21:07
sjmc7i agree it’s not ideal. i think having each client define it is a pain21:08
TravTbut what does it define?21:09
TravTpresentation is typically left to the presenter21:09
sjmc7dunno. a list of ‘useful’ default fields21:09
sjmc7yeah, i know :(21:09
sjmc7maybe we do that then. have a list of mappings of type -> fields displayed21:09
sjmc7back in a sec. this has been a problem since the early prototypes, how you use what is, to the client, unstructured data21:10
lakshmiSalmost feel like having a cli interaction tool that can generate this command line for easy users21:10
TravTwell and easier input is definitely a goal, but a a problem we have is the output right now. http://pasteboard.co/WM13caw.png21:12
lakshmiShow about a summary output and dig deeper with another query to get the details21:14
RickA-HPThe output is not good. Cassandra has the same type of issue. At least the "pretty" option on ES helps.21:14
TravTlet me grab a couple more example outputs to show you21:15
RickA-HPCould we have a "format" option to display the output in a clean way (like pretty=true in ES). But still retain the raw output. The raw output is good if trying to parse it with tools or in a pipe.21:16
TravThttp://paste.openstack.org/show/484614/21:17
TravThmm, looks like other openstack client commands are pretty crappy too21:18
TravTopenstack hypervisor list21:18
TravTthen openstack hypervisor show21:18
TravTwill provide copy paste21:19
TravThttp://pasteboard.co/Y3tEI8r.png21:20
lakshmiSthats a comprise for showing the whole source output. alternatively we could provide a link which returns the source but that's one more hop to server21:20
TravTwell see, we do provide simply output21:20
TravTopenstack search query "" is simple and readable21:20
TravTadding --details makes it ugly21:20
TravTokay, this is already solved actually21:22
sjmc7if you want the raw output, you’d probably use the python client itself21:22
lakshmiSwell, --details is optional , so even though its ugly admin is asking for it21:22
TravTit is partially solved21:22
TravTosc supports max width21:23
TravTsearch query "" --details --max-width 5021:23
TravTthere is also a column and json format argument...21:23
TravTsearch query "" --details --max-width 50 -c Source -f json21:24
TravTkind og21:25
TravTkind of...21:25
sjmc7—json kind of demonstrates the problem21:26
sjmc7the json’s actually broken21:26
sjmc7because of the comma->newline substitution21:26
TravTyeah... it isn't valid json21:28
TravTttripp@ubuntu:~$ openstack search query "" --details -c Source -f value | python -m json.tool21:28
TravTExpecting property name: line 1 column 2 (char 1)21:28
sjmc7something that’ll work, for now at least, is to have —details dump the entire _source out as unmodified json21:31
TravTyeah, so line 89 here21:32
TravThttps://review.openstack.org/#/c/260899/7/searchlightclient/osc/v1/search.py21:32
sjmc7yeah, and that’s the one i commented on i think, although i wasn’t forceful enough21:32
TravTlet me see what that does locally21:32
TravTline 102 actually...21:35
sjmc7it’s replacing all commas with newlines, and un-quoting (since json.dumps is putting quotes around everything)21:37
TravTwell, that helped, but21:38
TravTthis still doesn't work because the output doesn't provide a surrounding brace21:38
sjmc7?21:38
TravTttripp@ubuntu:~$ openstack search query "name: cirros" --details -c Source -f value | python -m json.tool21:39
TravTExtra data: line 2 column 1 - line 5 column 1 (char 444 - 1876)21:39
sjmc7the whole thing’s one giant string!21:40
sjmc7take that function out entirely and it’ll do what you expect21:41
TravTyep21:43
TravTand i'd argue its fine21:43
TravTopenstack search query "name: cirros" --details --max-width 5021:43
TravTyou can see it is json21:44
TravTso then you can do openstack search query "name: cirros" --details -c Source -f json21:44
TravTand it looks fine21:44
sjmc7right, but the non-jsoned output looks a little weird21:44
sjmc7having a look how this is done elsewhere21:45
TravTi think that is how it is done21:45
sjmc7it’s sort of broken for glance image show, for instance; the properties list is output as a big string21:46
sjmc7i.e. any formatting you do on column values applies to all formats21:47
TravTyeah same for hyervisor show21:48
TravTbut that is a bit different21:48
TravTi mean this is a query21:48
TravTand -details gives you native result21:48
TravTthe query is native21:48
sjmc7right. what i am saying is, if you apply formatting to the _source column, it applies in both cases21:49
sjmc7so if you turn it into a readable string in the table case, that’s how it’ll be presented in the json21:49
sjmc7i’m not sure i’m making sense any more21:51
* TravT trying to make brain work... just a sec21:52
TravTgot distracted21:52
TravTsjmc7: i know what you are saying21:53
sjmc7that’s good. i’m not sure i do any more :)21:53
TravTi think it is better for it to output the json in table case21:53
sjmc7what do we want to happen?21:53
TravTso that if you are using this in a pipeline of commands you can actually use the json in the -c (column only) case21:54
sjmc7you mean in the —f json case?21:54
TravTyeah, what he said21:54
sjmc7the -c just removes the id, score etc21:54
TravTwell -c and -f case21:54
sjmc7ok. in that case, we accept that in the tabular output, you’ll get a string representation of the _source field21:55
sjmc7as in, str(_source)21:56
sjmc7unicodes and all21:56
TravTbetter idea?21:57
sjmc7the (only) alternative is to format _source in some way, and accept it’s not consumable with —json21:58
TravTmaybe if the argument was --source (not --details), then it would make more sense21:59
sjmc7yeah21:59
TravTyou get a source column and it gives you source data that you can then operate on programatically21:59
sjmc7ONLY in -f json mode21:59
sjmc7not the tabular data21:59
sjmc7i’m ok with that, i can’t think of a good way to get readable table and proper json format in the sonstraints of osc22:00
sjmc7maybe we can get some input later22:00
sjmc7i’ll put a review up22:00
TravTok, i also have some comments with these commands in draft state22:01
TravTsjmc7: added some comment, please hop on with your own to +/- the suggestions.22:08
sjmc7i did22:09
sjmc7simultaneously22:09
TravTif --source optionally took arguments, they could be the fields to return...22:32
TravTe.g. --source returns all source22:32
TravT--source name,boo, it could return just those columns.22:33
sjmc7yeah. maybe a future amendment?22:38
TravTyeah, not required for this patch22:55
*** sigmavirus24 is now known as sigmavirus24_awa23:34
*** bpokorny_ has joined #openstack-searchlight23:58
*** bpokorny_ has quit IRC23:58
*** bpokorny_ has joined #openstack-searchlight23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!