| *** david-lyle has joined #openstack-searchlight | 01:03 | |
| openstackgerrit | Travis Tripp proposed openstack/searchlight: More efficient result filtering https://review.openstack.org/207682 | 05:43 |
|---|---|---|
| *** lakshmiS has joined #openstack-searchlight | 09:43 | |
| *** sballe has joined #openstack-searchlight | 10:45 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:41 | |
| sjmc7 | TratT_ - for the plugin session patch, try 'username' instead of 'user_name'. there's a confusing deprecation warning from the keystone session config parser about it preferring 'user-name' but neither a dash nor underscore work | 14:26 |
| *** lakshmiS has quit IRC | 14:53 | |
| *** lakshmiS has joined #openstack-searchlight | 14:57 | |
| openstackgerrit | Steve McLellan proposed openstack/searchlight: Refactor clients handling to use Sessions. https://review.openstack.org/219271 | 15:59 |
| *** TravT_ is now known as TravT | 16:22 | |
| openstackgerrit | Lakshmi N Sampath proposed openstack/searchlight: Fix for Property mapping for Metadef properties fails https://review.openstack.org/228531 | 16:28 |
| openstackgerrit | Travis Tripp proposed openstack/searchlight: Add --force when doing initial index sync https://review.openstack.org/228535 | 16:50 |
| TravT | smc7: sessions patch now connects properly to glance, nova, designate. just checking through a few other things on it. | 17:00 |
| sjmc7 | ok. the glance connection's not using sessions, so it's still using the reauthorization decorator. in mitaka we'll be able to neaten it up further | 17:01 |
| sjmc7 | lakshmiS - you there? | 17:03 |
| sjmc7 | i'm looking at https://review.openstack.org/#/c/228531. you've changed 'name' to 'property' in _serialize_property, but the API for properties returns keys 'description', 'name', 'title', 'type' | 17:05 |
| lakshmiS | yes | 17:05 |
| sjmc7 | after your patch the elasticsearch data remaps 'name' -> 'property' | 17:05 |
| lakshmiS | yeah it does return name but we mapped it as property | 17:06 |
| sjmc7 | we have property_mapping which has name,type,title,description | 17:07 |
| lakshmiS | even that might have been modified in some previous patch. let me check. its probably working right now due to dynamic mapping | 17:08 |
| sjmc7 | right - we end up with 'property' as a dynamically mapped field | 17:09 |
| sjmc7 | yeah, it's definitely likely i blundered when i converted this from the raw database access, and your patch does fix the problem, but it may be incomplete | 17:10 |
| sjmc7 | it also creates an API difference with glance client | 17:10 |
| lakshmiS | property is a dictionary key in glance server API but glance client modified it as name | 17:13 |
| sjmc7 | ok. and we definitely want to map it as property? | 17:14 |
| sjmc7 | in which case, we should change the defined mapping to remove 'name' | 17:14 |
| lakshmiS | i think we used the key as "property" to make it easier to understand since there were so many "name" fields. I dont have any preference either way | 17:14 |
| sjmc7 | although... my data now has both | 17:15 |
| sjmc7 | 'name' and 'property' | 17:15 |
| lakshmiS | either we keep it as "property" or change everything including listener to use "name" | 17:15 |
| lakshmiS | which data? | 17:15 |
| sjmc7 | {"property": "os_version", "type": "string", "description": "Operating system version as specified by the distributor. (for example, '11.10')", "name": "os_version", "title": "OS Version"} | 17:15 |
| lakshmiS | can you try deleting the index and sync it again | 17:15 |
| sjmc7 | yeah, i did | 17:16 |
| sjmc7 | one sec, doing it again | 17:16 |
| sjmc7 | still getting both | 17:17 |
| lakshmiS | do you see that for all property objects? | 17:17 |
| sjmc7 | no, doesn't look like it | 17:18 |
| lakshmiS | hmm, i will checkout the code again | 17:19 |
| sjmc7 | without the patch everything just has name | 17:21 |
| sjmc7 | with it.. some stuff seems to have 'property' as well, but not all | 17:21 |
| lakshmiS | ok i will see whats going on | 17:21 |
| sjmc7 | ta | 17:21 |
| lakshmiS | ;) | 17:21 |
| TravT | lakshmiS: are the functional tests running in zuul yet? | 17:24 |
| lakshmiS | no its skipping. checked with ceilometer and they are not running it either | 17:25 |
| TravT | ok, i was just running 'em locally to be sure. | 17:26 |
| TravT | Ran 125 tests in 0.250s | 17:26 |
| TravT | PASSED (id=0, skips=28) | 17:26 |
| lakshmiS | but their tests are not really running es. | 17:26 |
| TravT | ^ sessions patch | 17:26 |
| TravT | i think that one looks okay | 17:26 |
| TravT | not seeing any reason to hold it up at this point. | 17:26 |
| lakshmiS | any functional tests use fake client | 17:31 |
| openstackgerrit | Lakshmi N Sampath proposed openstack/searchlight: Fix for Property mapping for Metadef properties fails https://review.openstack.org/228531 | 17:44 |
| lakshmiS | sjmc7: seems like there could be a bug with glanceclient. metadef get property in a namespace shouldn't return 'name' field but i think whenever a property is updated from glanceclient command line it is adding 'name' fields for those properties | 17:45 |
| lakshmiS | added another patch. let me know if that works for you | 17:46 |
| sjmc7 | ok, that looks like it'll work | 17:48 |
| TravT | lakshmiS: i'm not following this totally, but will what you are doing ensure that the data still looks like what we get out of native API (even if there are extra fields)? | 17:50 |
| lakshmiS | the extra fields "name" doesnt exist when data is loaded into glance but during updates from glance client command line they are getting added | 17:52 |
| lakshmiS | data from glance should look like this | 17:53 |
| lakshmiS | "properties": { | 17:53 |
| lakshmiS | "hw_watchdog_action": { | 17:53 |
| lakshmiS | "enum": [ | 17:53 |
| lakshmiS | "disabled", | 17:53 |
| lakshmiS | "reset", | 17:53 |
| lakshmiS | "poweroff", | 17:53 |
| lakshmiS | "pause", | 17:53 |
| lakshmiS | "none" | 17:53 |
| lakshmiS | ], | 17:53 |
| lakshmiS | "type": "string", | 17:53 |
| lakshmiS | "description": "For the libvirt driver, you can enable and set the behavior of a virtual hardware watchdog device for each flavor. Watchdog devices keep an eye on the guest server, and carry out the configured action, if the server hangs. The watchdog uses the i6300esb device (emulating a PCI Intel 6300ESB). If hw_watchdog_action is not specified, the watchdog is disabled. Watchdog behavior set using a specific image's prope | 17:53 |
| lakshmiS | rties will override behavior set using flavors.", | 17:53 |
| lakshmiS | "title": "Watchdog Action" | 17:53 |
| lakshmiS | } | 17:53 |
| lakshmiS | } | 17:53 |
| sjmc7 | right.. the difference is that we have to flatten that into a list of objects | 17:53 |
| sjmc7 | so it's whether 'hw_watchdog_action' becomes 'name' or a 'property | 17:54 |
| lakshmiS | which it does during initial load but i think there might be a bug in glance client which is adding this | 17:54 |
| lakshmiS | "properties": { | 17:54 |
| lakshmiS | "hw_watchdog_action": { | 17:54 |
| lakshmiS | "name": "hw_watchdog_action", | 17:54 |
| lakshmiS | "enum": [ | 17:54 |
| lakshmiS | "disabled", | 17:54 |
| lakshmiS | "reset", | 17:54 |
| lakshmiS | "poweroff", | 17:54 |
| lakshmiS | "pause", | 17:54 |
| lakshmiS | "none" | 17:54 |
| lakshmiS | ], | 17:54 |
| lakshmiS | "type": "string", | 17:54 |
| lakshmiS | "description": "For the libvirt driver, you can enable and set the behavior of a virtual hardware watchdog device for each flavor. Watchdog devices keep an eye on the guest server, and carry out the configured action, if the server hangs. The watchdog uses the i6300esb device (emulating a PCI Intel 6300ESB). If hw_watchdog_action is not specified, the watchdog is disabled. Watchdog behavior set using a specific image's prope | 17:54 |
| lakshmiS | rties will override behavior set using flavors.", | 17:54 |
| lakshmiS | "title": "Watchdog Action" | 17:54 |
| lakshmiS | } | 17:54 |
| lakshmiS | } | 17:54 |
| lakshmiS | there is an extra "name" field which shouldn't be there | 17:54 |
| TravT | okay gimme just a second to read this. also interacting with somebody on the search panel and a magic search widget bug. | 17:55 |
| lakshmiS | well if this looks complex to you guys then the other way to do fix this is to keep the "name" field and fix the listener. either way the bug on glance client doesnt go away but since we use same names its not a problem | 17:58 |
| TravT | lakshmiS: looking through code now. | 17:59 |
| TravT | lakshmiS: | 18:06 |
| lakshmiS | yes | 18:06 |
| TravT | i'm leaning towards saying we should go to name, since that is what comes out of glance client... | 18:06 |
| TravT | i mean no matter how we slice it this creates a data structure different than actual API, right? | 18:07 |
| TravT | because the properties objects got flattened (for search reasons, i believe) | 18:07 |
| TravT | what do you think? | 18:08 |
| lakshmiS | yeah i think glance client didnt flatten it the right way we wanted, so we ended with "name" field | 18:08 |
| TravT | well, should we fix glance client? | 18:08 |
| lakshmiS | i would like to do that but not sure how long that will take | 18:09 |
| lakshmiS | we might as well change ours :) | 18:09 |
| TravT | i'm trying to think if we'll run into issues with "name" | 18:09 |
| lakshmiS | i dont know if you still recollect the reason behind using "property" as the key instead of "name" but if i still remember it was done to make it more readable i guess | 18:10 |
| TravT | we use "title" for display_name to align with json schema... | 18:10 |
| TravT | by above, do you mean that | 18:11 |
| TravT | "properties": { | 18:11 |
| TravT | "cpu_sockets": { | 18:11 |
| TravT | "title": "vCPU Sockets", | 18:11 |
| TravT | "description": "Preferred number of sockets to expose to the guest.", | 18:11 |
| TravT | "type": "integer" | 18:11 |
| TravT | }, | 18:11 |
| TravT | cpu_sockets | 18:11 |
| lakshmiS | yes that was before flattening | 18:11 |
| TravT | i mean we did the metadefs structure internally to match json schema as much as possible. | 18:11 |
| lakshmiS | right | 18:12 |
| TravT | i don't even remember this flattening thing | 18:12 |
| lakshmiS | i think we initially did this in elasticsearch | 18:12 |
| lakshmiS | "properties": [ | 18:12 |
| lakshmiS | { | 18:12 |
| lakshmiS | "property": "architecture",....} | 18:12 |
| lakshmiS | instead of "properties": [ | 18:12 |
| lakshmiS | { | 18:12 |
| lakshmiS | "name": "architecture", | 18:12 |
| TravT | yeah, i believe that was because then you can do a search like: perties.name = "architecture" | 18:13 |
| TravT | otherwise when doing dynamic mapping, we'd end up with a field for every incoming property | 18:13 |
| lakshmiS | so i dont know which is better anymore. do you also think its better to have "name"? | 18:14 |
| TravT | cpu_sockets above would end up as its own mapped object | 18:14 |
| TravT | you could then search properties.cpu_sockets.title = 'vCPU', but i'm not sure what the value is | 18:14 |
| TravT | i don't know why you'd ever search for that. | 18:15 |
| lakshmiS | no i think you are talking about having dynamic property name as field which is not what we want | 18:15 |
| TravT | the property mapping as is appears to be missing some of the other fields like enum, min value, etc | 18:16 |
| lakshmiS | it whether we should have properties.property = "architecture" or properties.name ="architecture" | 18:16 |
| TravT | so, i don't understand glance client? | 18:16 |
| TravT | it is adding a name property? | 18:16 |
| lakshmiS | yeah i think thats a bug | 18:16 |
| TravT | maybe wayne added that at some point | 18:17 |
| lakshmiS | until now it went undetected since we were using the same field name | 18:17 |
| lakshmiS | i will checkout the history | 18:17 |
| lakshmiS | keeping it same as glance client is one way to look at it but more importantly how users use elasticsearch query is also important so lets decide which looks good | 18:18 |
| lakshmiS | properties.property = "architecture" or properties.name ="architecture"? | 18:18 |
| TravT | so, objects gets the same thing | 18:19 |
| lakshmiS | yes | 18:19 |
| TravT | so, objects.name make sense | 18:20 |
| lakshmiS | right | 18:20 |
| TravT | probably easier if properties.name | 18:20 |
| TravT | more symetrical | 18:20 |
| lakshmiS | ok lets go with it then. | 18:21 |
| sjmc7 | sorry, my grilled cheese caught fire | 18:21 |
| TravT | i do think the way they both are being indexed needs the same the same treatment on the name field that is a problem for the other resource types name field. | 18:21 |
| sjmc7 | yeah, the flattening is so you can search; the API format doesn't lend itself well to discovery | 18:21 |
| TravT | https://review.openstack.org/#/c/226378/2/searchlight/elasticsearch/plugins/designate/recordsets.py | 18:22 |
| lakshmiS | once we have a release, these changes will have impact on horizon | 18:23 |
| TravT | yes, they will. | 18:23 |
| sjmc7 | renaming fields will, yeah | 18:23 |
| TravT | right now, i'm actually filtering out metadefs in the search | 18:23 |
| TravT | because there is some other problem... i'll dig into that today. | 18:23 |
| lakshmiS | ok so vote is to keep the field as "name". will update the patch | 18:24 |
| sjmc7 | ok. i'll test it when you're done | 18:24 |
| sjmc7 | the fix looked good other than that confusion | 18:24 |
| TravT | yeah. i think so. | 18:24 |
| lakshmiS | yeah it uncovered a bug too | 18:24 |
| TravT | sjmc7: i wonder if name should be same treatment as name in other places or should stay not_analyzed. | 18:25 |
| TravT | it will never contain a space... | 18:25 |
| TravT | it actually is more like an id | 18:25 |
| sjmc7 | it'll still get tokenized | 18:25 |
| sjmc7 | if it's an id, it should be not_analyzed i think | 18:25 |
| TravT | yeah, probably we need to do this on properties.title | 18:26 |
| TravT | and objects.display_name | 18:26 |
| lakshmiS | yeah name shouldn't be tokenized | 18:26 |
| TravT | so, in that case | 18:26 |
| TravT | is it really id? | 18:26 |
| lakshmiS | yes in metadef space | 18:26 |
| sjmc7 | but... even if the field's not analyzed, some searches still won't work as you expect | 18:26 |
| sjmc7 | we maybe need to look at this later | 18:27 |
| sjmc7 | and come up with some advice because it is confusing. i think in general, id type fields should be non_analyzed | 18:28 |
| TravT | i think so as well. | 18:28 |
| sjmc7 | even with e.g. nova servers, the 'name' field is problematic; you want to be able to do partial searches but you might also want to search for exact matches | 18:29 |
| TravT | there is a part of me that wants to pull metadefs from being indexed for now. | 18:29 |
| TravT | i'm just not sure if there is a huge value to indexing them. but let's at least fix the name field and i'll think a bit more on it. | 18:35 |
| TravT | sjmc7: lakshmiS: i have an eye doctor apt in 20 mins. so, i'll be away for a short while. | 18:38 |
| lakshmiS | ok | 18:39 |
| TravT | lakshmiS: this should be an easy review for you: https://review.openstack.org/#/c/228535/ | 18:39 |
| TravT | i know you need to get to bed. | 18:39 |
| sjmc7 | i feel like i need to get to bed :) | 18:39 |
| lakshmiS | will check it out after my patch | 18:39 |
| TravT | ok | 18:39 |
| TravT | bbiab will try to check in on irc. | 18:40 |
| openstackgerrit | Lakshmi N Sampath proposed openstack/searchlight: Fix for Property mapping for Metadef properties fails https://review.openstack.org/228531 | 19:12 |
| sjmc7 | lakshmiS - you want me to give that a quick go? | 19:14 |
| lakshmiS | forgot something to revert. one last patch | 19:14 |
| sjmc7 | ok. let me know. i have an hour's worth of meetings in 15 minutes | 19:15 |
| openstackgerrit | Lakshmi N Sampath proposed openstack/searchlight: Fix for Property mapping for Metadef properties fails https://review.openstack.org/228531 | 19:15 |
| lakshmiS | ok try with latest patch | 19:15 |
| sjmc7 | will do | 19:15 |
| openstackgerrit | Merged openstack/searchlight: Add --force when doing initial index sync https://review.openstack.org/228535 | 19:16 |
| sjmc7 | looks good | 19:16 |
| lakshmiS | ok | 19:17 |
| sjmc7 | you can go to bed now :) | 19:18 |
| lakshmiS | ttyl | 19:26 |
| *** lakshmiS has quit IRC | 19:26 | |
| openstackgerrit | Steve McLellan proposed openstack/searchlight: Add additional facet fields https://review.openstack.org/228063 | 19:28 |
| *** david-lyle has quit IRC | 20:14 | |
| openstackgerrit | Merged openstack/searchlight: More efficient result filtering https://review.openstack.org/207682 | 20:21 |
| openstackgerrit | Merged openstack/searchlight: Fix for Property mapping for Metadef properties fails https://review.openstack.org/228531 | 20:31 |
| openstackgerrit | Travis Tripp proposed openstack/searchlight: Add additional facet fields https://review.openstack.org/228063 | 20:40 |
| sjmc7 | TravT - i was trying to reproduce 1499805 earlier without any luck. i was missing the magic step | 20:55 |
| TravT | hey, just a sec. | 20:55 |
| sjmc7 | i can put a patch up for that shortly, although i think a 500 if there's no mapping is actually pretty reasonable | 20:56 |
| TravT | on a hangout debugging magic search | 20:56 |
| sjmc7 | good luck! | 20:56 |
| TravT | it would be nice if we can somehow | 20:56 |
| TravT | still return some facets. | 20:56 |
| TravT | here's the way i got 500 to reproduce | 20:56 |
| sjmc7 | yep. but no terms | 20:56 |
| TravT | basically not have a valid mapping for a requested aggregate | 20:57 |
| sjmc7 | yeah, i'll see what we get back | 20:58 |
| *** david-lyle has joined #openstack-searchlight | 21:09 | |
| TravT | sjmc7: did you see the other fields i mentioned on the additional facets patch? | 21:14 |
| TravT | https://review.openstack.org/#/c/228063/2/searchlight/elasticsearch/plugins/nova/servers.py | 21:14 |
| sjmc7 | yeah. sorry, i did reply but forgot the stupid review button | 21:14 |
| sjmc7 | i can keep adding more and more but at some point i'm not sure how useful it is | 21:15 |
| TravT | well, those are the last few i spotted. | 21:15 |
| sjmc7 | ok, i can add them | 21:15 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:23 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 21:23 | |
| openstackgerrit | Steve McLellan proposed openstack/searchlight: Correct facet error when no mapping defined https://review.openstack.org/228639 | 21:55 |
| openstackgerrit | Steve McLellan proposed openstack/searchlight: Rename GLANCE_TEST_SOCKET_FD_STR https://review.openstack.org/228642 | 22:01 |
| sjmc7 | you still around, TravT? | 22:22 |
| TravT | sjmc7: hey | 22:28 |
| sjmc7 | busy afternoon! looking at those extra fields - i agree with the network ones, not so much with the others, but if you think there's a practical use for them i can add them | 22:28 |
| sjmc7 | the cardinality on hostID is potentially quite large | 22:29 |
| sjmc7 | since it's a function of host + tenant | 22:29 |
| sjmc7 | the disk config ones i don't really know what they are, nor whether anyone would ever want to filter by them | 22:29 |
| TravT | that's right... i keep forgetting about that. | 22:29 |
| sjmc7 | i don't feel that strongly aobut it, but i don't want to add stuff for the sake of it | 22:30 |
| TravT | just a sec, looking again at facet returned | 22:31 |
| TravT | will vm_state always map to status? | 22:32 |
| sjmc7 | i think so - status must be derived from it somehow | 22:32 |
| sjmc7 | the extensions are a bit dodgy, not everyone has them | 22:32 |
| TravT | well right now, i am getting a few fields back that i have no idea what to enter in the UI | 22:33 |
| TravT | which i don't think should be coming back. | 22:33 |
| TravT | hmmm | 22:33 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:33 | |
| TravT | no never mind on that | 22:33 |
| sjmc7 | we can also exclude some :) ultimately i think we may want to make these configuration driven | 22:33 |
| TravT | ok. let's not do the "other" ones i mentioned | 22:34 |
| TravT | but go with the network ones | 22:34 |
| TravT | at least version | 22:34 |
| sjmc7 | yep, i've added those. ok, got to update the tests, will push it up in a few minute | 22:34 |
| TravT | maybe the IPS type isn't so interesting? | 22:34 |
| sjmc7 | probably not | 22:34 |
| sjmc7 | maybe you want to find any servers with floating ips tho | 22:35 |
| TravT | well, i'm already getting Networks.OS-EXT-IPS:type: | 22:35 |
| TravT | in the retruned facets, but no options | 22:35 |
| sjmc7 | yep | 22:35 |
| TravT | that's why i suggested providing options for it | 22:35 |
| sjmc7 | short of providing options for everything, or excluding much more, that'll always happen | 22:36 |
| TravT | let me switch from admin to demo | 22:36 |
| TravT | I'm thinking IPS type might be admin only as well. | 22:37 |
| TravT | right now it isn't | 22:37 |
| sjmc7 | it's not just fixed/floating? | 22:37 |
| TravT | well what is that giving back? i was thinking it might be talking about the diff between fixed, vxlan, vlan, etc | 22:38 |
| TravT | oh wait, was confusing fixed and flat | 22:38 |
| sjmc7 | hmm.. | 22:39 |
| sjmc7 | pretty sure it's fixed v floating | 22:39 |
| TravT | yeah, it is | 22:39 |
| TravT | i suppose that is still a pretty nice query to have | 22:40 |
| TravT | be able to find all servers that are using a floating ip | 22:40 |
| sjmc7 | can't hurt | 22:40 |
| TravT | i'm going crazy right now... one of my monitors is flashing on and off every 5 seconds. got to figure that out... | 22:41 |
| sjmc7 | :D | 22:42 |
| openstackgerrit | Steve McLellan proposed openstack/searchlight: Add additional facet fields https://review.openstack.org/228063 | 22:56 |
| openstackgerrit | Travis Tripp proposed openstack/searchlight: Improve documentation info for devstack config https://review.openstack.org/228664 | 23:42 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!