15:01:06 <sjmc7> #startmeeting openstack search
15:01:06 <openstack> Meeting started Thu Mar 30 15:01:06 2017 UTC and is due to finish in 60 minutes.  The chair is sjmc7. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:08 <lei-zh> o/
15:01:10 <openstack> The meeting name has been set to 'openstack_search'
15:02:02 <Kevin_Zheng> I
15:02:05 <Kevin_Zheng> I
15:02:12 <TravT> o/
15:02:16 <Kevin_Zheng> 0/
15:02:34 <sjmc7> morning/evening all
15:03:45 <sjmc7> probably going to be quite a short one unless anyone else has stuff they want to talk about
15:03:50 <sjmc7> #topic nova versioned notifications
15:04:09 <Kevin_Zheng> I have one :)
15:04:11 <sjmc7> I haven’t had as much time as I’d like, but looking at them, switching to versioned payloads for instances shouldn’t be a big deal
15:05:49 <sjmc7> Kevin_Zheng: you were able to receive instance.update events correctly, right?
15:05:59 <sjmc7> i think i must have failed to restart one of the nova components
15:06:00 <Kevin_Zheng> Yeah
15:06:17 <Kevin_Zheng> Probably API
15:06:55 <sjmc7> ah, yes, possibly
15:07:05 <Kevin_Zheng> Because suspend notify sent from compute but update sent from API
15:07:08 <sjmc7> i’ll retry. it’s nice that the payloads are complete for all the notification times
15:07:42 <sjmc7> that should mean we can get everything we need from the notifications. the network information looked a bit different from what we currently have but i think it should be possible to make it fit
15:08:32 <Kevin_Zheng> Sure
15:08:50 <sjmc7> i’m hoping to get a patch up by this time next week but my time’s still really restricted
15:09:33 <sjmc7> at the moment, leaving nova to send out unversioned notifications should be sufficient since the payloads are very similar
15:10:57 <sjmc7> #topic Overridable default filters
15:11:07 <sjmc7> the other part of this is handling nova’s “deleted” case
15:11:26 <sjmc7> i’ve put up a spec https://review.openstack.org/#/c/451846/
15:11:30 * gibi is lurking late
15:12:06 <Kevin_Zheng> Cool will check tomorrow
15:12:10 <sjmc7> to allow plugins to define filters that are applied to all queries unless the user specifies otherwise, to handle cases like nova’s ‘NOT deleted’ and glance’s community images
15:12:46 <sjmc7> we can do something that’s “good enough” by looking at the query i think
15:13:45 <sjmc7> so if “deleted” is present as a key in a query, we won’t apply that default filter
15:14:36 <Kevin_Zheng> Sounds good
15:15:01 <sjmc7> the other option is to have an explicit field for ‘don’t apply these defaults’ which is a bit safer but requires knowledge of what the defaults are. not sure the best approach
15:15:02 <lei-zh> looks some more work needed for query_string
15:15:27 <sjmc7> yeah. the use cases though are quite limited so i don’t think it has to be perfect
15:16:54 <lei-zh> maybe with regular expression we can deal with that case
15:17:41 <sjmc7> what’s the phrase? ‘Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.’
15:19:04 <sjmc7> anyway, that is something we’ll need to support in some form i think
15:19:06 <lei-zh> or just looking for the exact key is enough : )
15:19:13 <lei-zh> yep
15:19:19 <sjmc7> yeah, that would be my first thought, keep it really simple
15:19:38 <sjmc7> that was all i had on the agenda for today; like i say, i’ve been really short on time the last few weeks
15:19:40 <sjmc7> #topic open discussion
15:19:46 <sjmc7> anyone else? Kevin_Zheng ?
15:20:00 <Kevin_Zheng> Yeah
15:20:25 <Kevin_Zheng> My colleague asked one interesting question
15:20:47 <Kevin_Zheng> I will just paste the whole thing here
15:21:27 <Kevin_Zheng> Searchlight to support multi-region deployments.
15:21:27 <Kevin_Zheng> Telecom operators want to deploy lightweight OpenStack into small site, for example uCPE, the number of small sites may reach to 100k. Cloud administrator wants to have a flexible search capability for all resources provision in these distributed OpenStack instances.
15:21:27 <Kevin_Zheng> In fact, this is a typical scenrio for searchlight to support OpenStack multi-region deployments. In one region, administrator can do the search in Nova, Cinder or Neutron. But if there are many regions, then no tool currently to provide cross region search capability.
15:21:28 <Kevin_Zheng> The idea is to deploy the listener in each region, but deploy the searchlight API and elasticsearch in a centralized way, all data collected from listeners will be put into centralized elasticsearch.
15:21:28 <Kevin_Zheng> The risk to serialize the data into elasticsearch directly from listener is lack of authorization. So it's better to provide data input API in searchlight API, so that only authorized access can be allowed, this will be critical important in multi-region deployments. And listener can be configured to serialize data via elasticsearch API directly or via
15:21:28 <Kevin_Zheng> searchlight API.
15:22:06 <Kevin_Zheng> So it is about supporting SL over multiple regions
15:22:11 <sjmc7> This came up before, one sec
15:22:46 <sjmc7> we actually did a PoC at one of the summits (tokyo i think)
15:23:02 <Kevin_Zheng> Cool
15:23:12 <sjmc7> you can set up elasticsearch to act as a proxy/multiplexer
15:23:34 <sjmc7> it’s in the docs as ‘tribe nodes’ https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-tribe.html
15:23:56 <TravT> See https://docs.openstack.org/developer/searchlight/plugins.html?highlight=include_region_name#global-configuration-options
15:24:01 <TravT> include_region_name
15:24:15 <TravT> and https://docs.openstack.org/developer/searchlight/plugins.html?highlight=include_region_name#inheritable-common-configuration-options
15:24:19 <TravT> override_region_name
15:24:20 <sjmc7> tribe nodes don’t store any data, they just proxy queries out. you can point searchlight-api at a tribe node and it’ll then search multiple clusters, and yeah, with that option TravT just mentioned you can separate data by region
15:24:29 <sjmc7> searchlight-listener runs on each site to a single cluster
15:24:42 <Kevin_Zheng> Cool
15:25:06 <Kevin_Zheng> Will check it
15:25:16 <sjmc7> that doesn’t deply searchlight centrally, and doesn’t aggregate the data physically, but it does let you reach out to your various sites
15:25:39 <sjmc7> you could also set up replication from the regions to a central point
15:25:51 <sjmc7> but that’s not something elasticsearch has native, you’d need a tool to do it
15:25:51 <Kevin_Zheng> Hmm
15:26:02 <TravT> http://specs.openstack.org/openstack/searchlight-specs/specs/newton/cross-region-search.html
15:26:47 <Kevin_Zheng> Thanks, this is cool
15:26:48 <sjmc7> clustering across data centers isn’t recommended because of the risk of network connectivity causing brain farts in the cluster
15:27:16 <sjmc7> so you’d need something more like mysql replication to shunt data periodically from each cluster to a central one if you wanted to do that
15:27:21 <sjmc7> the tribe thing worked really well
15:27:25 <TravT> https://review.openstack.org/#/c/353681/
15:27:40 <sjmc7> obviously it requires keystone-to-keystone federation so you have shared users & projects
15:28:09 <Kevin_Zheng> Yeah, they have already done that part
15:29:02 <breton> keystone-to-keystone federation probably doesn't give you shared projects
15:29:14 <breton> (if it does, i'd love to see how)
15:29:33 <sjmc7> hmm, ok. that might prove a bit tricky then
15:29:42 <breton> oh well, on the second thought, it does
15:29:54 <breton> but only with Ocata
15:30:48 <breton> but still, not project ids, but only project names
15:30:49 <Kevin_Zheng> Thanks for the info thought, might be a good chance to use SL in real commercial deployments
15:31:33 <sjmc7> yeah. i’d look at that first. if project ids aren’t consistent across sites then it might need a bit more work because it’d become very hard to collate information in different regions
15:32:44 <Kevin_Zheng> I will provide them the info and maybe they can do some tests and provide feedbacks
15:32:55 <TravT> that'd be great!
15:33:17 <sjmc7> yeah. they can always ask on the list or in IRC if they want some help
15:33:42 <Kevin_Zheng> Yeah, I will leave notes there
15:35:17 <sjmc7> ok
15:35:35 <sjmc7> thanks for the discussion, good people are thinking about it. anyone got anything else?
15:36:30 <lei-zh> nope
15:39:23 <sjmc7> ok, in that case i guess we’ll call it here
15:39:25 <sjmc7> thanks all
15:40:30 <sjmc7> #endmeeting