15:00:46 <sjmc7> #startmeeting openstack search
15:00:50 <openstack> Meeting started Thu Mar  9 15:00:46 2017 UTC and is due to finish in 60 minutes.  The chair is sjmc7. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:51 <sjmc7> haha, good start
15:00:52 <sjmc7> morning all
15:00:54 <openstack> The meeting name has been set to 'openstack_search'
15:01:04 <rosmaita> i wonder how many "marketing committee" meeting logs there are
15:01:10 <rosmaita> probably a lot
15:01:34 <sjmc7> sounds like a front for a spy agency
15:01:45 <yingjun> o/
15:01:47 <TravT> o/
15:01:55 <lei-zh> o/
15:01:55 <Kevin_Zheng_Hex> o/
15:01:56 <sjmc7> you have to get up early for the glance one - aren’t you west coast rosmaita ?
15:03:08 <sjmc7> we’ll get going in one more minute
15:03:09 <rosmaita> no, virginia
15:03:16 <rosmaita> otherwise, i would have moved the meeting time!
15:03:22 <sjmc7> :)
15:04:19 <sjmc7> okey dokey
15:04:25 <sjmc7> #topic pike open
15:04:34 <sjmc7> Ocata’s now all officially closed
15:05:17 <TravT> i noticed that launchpad doesn't have a pike series in it
15:05:27 <sjmc7> Yeah.. not sure why that is yet
15:05:36 <TravT> well, i always created them myself
15:05:45 <sjmc7> Ah, then that would be why: I’m lazy
15:05:49 <TravT> including ocata
15:05:59 <sjmc7> So I’ll do that following this
15:06:08 <rosmaita> sounds like TravT volunteered to do it?
15:06:20 <TravT> nah, i was trying to make sjmc7 feel guilty
15:06:31 <sjmc7> :)  Mission accomplished! We’ve got a huge laundry list of blueprints at https://blueprints.launchpad.net/searchlightthat needs some curating
15:06:34 <TravT> ;)
15:07:10 <sjmc7> Which i’ve also been lazy about. I’m going it curate it a bit today, but if there’s work people want to see in Pike please raise a BP or tag it for Pike
15:07:46 <sjmc7> We got most of the core stuff in over the last couple of releases so there are fewer big ticket items, which is good
15:09:10 <sjmc7> The UI could do with some love, and there are some more architectural things in there. I’d definitely like to solidify ES 5 support
15:09:57 <sjmc7> Dunno if anyone has any ideas for stuff they’d like to add we could discuss?
15:10:06 <sjmc7> Otherwise mailing list or launchpad is fine
15:10:07 <TravT> there was an email this morning from richard
15:10:31 <sjmc7> Yeah, I was gonna reply to that after this too - I’ll suggest they put that code in horizon which i think was the intent all along
15:10:35 <TravT> about moving our registered resource types in the ui into horizon.
15:10:47 <TravT> yeah, the panel started as a horizon patch
15:11:13 <TravT> but since it was following the typical trajectory of taking multiple years to merge there, we broke it into its own panel
15:12:04 <TravT> but david-lyle all along said that he thought that was a good way to get code sort of incubated (breaking it out) and then bringing back in when it makes sense
15:12:37 <sjmc7> ok. this is a good test of that statement :)
15:12:57 <sjmc7> We’ll need to coordinate a bit so that we don’t duplicate it but that should be fine
15:13:14 <TravT> yeah, i think it will overall benefit to have some of those resource panels go back into horizon... because we have not had the time to get all the actions finished for some of them.
15:13:36 <TravT> moving it into horizon will ironically likely help that now.
15:14:13 <sjmc7> Yeah. It makes sense generally too, so i don’t think it’s very controversial
15:14:44 <sjmc7> Ok, if there’s no features/pike things to discuss i’ll move on to the feature/pike thing i wanted to discuss
15:15:03 <sjmc7> #topic Nova cells
15:15:25 <sjmc7> There was a mailing list discussion following the PTG from Matt in Nova - http://lists.openstack.org/pipermail/openstack-dev/2017-February/112996.html
15:15:47 <sjmc7> They’re serious this release about beginning integration with Searchlight. They have a spec at https://review.openstack.org/441692 and matt’s asked some questions in IRC
15:16:16 <TravT> yeah, shout out the lei-zh and Kevin_Zheng_Hex for jumping in on that.
15:16:24 <sjmc7> Yeah, thanks both for that
15:16:56 <sjmc7> I don’t know if we can exclusively switch to versioned or we’ll have to maintain both; that might be a question for nova, but any changes will only be made to versioned notifications
15:17:05 <Kevin_Zheng_Hex> turns out many problems occoured
15:17:11 <sjmc7> Ah.
15:17:35 <sjmc7> Not entirely unexpected - are there any that’ll require work on searchlight?
15:18:15 <Kevin_Zheng_Hex> not much, I think the major problem is in Nova
15:18:32 <sjmc7> Excellent!
15:18:33 <sjmc7> :)
15:18:42 <sjmc7> Although i suspect there will be stuff for us later
15:18:58 <lei-zh> I see Kevin_Zheng_Hex starts to work on a POC
15:19:02 <sjmc7> I saw the discussion about being able to force list instances to go to the DB
15:19:05 <sjmc7> Was there anything else?
15:19:39 <Kevin_Zheng_Hex> we have to find a way to get trusty data IMO
15:19:57 <Kevin_Zheng_Hex> otherwise it will be a dead loop
15:20:15 <Kevin_Zheng_Hex> lei-zh find out this
15:20:26 <sjmc7> trusty data?
15:20:28 <Kevin_Zheng_Hex> if we re-sync, nova will call SL
15:20:34 <sjmc7> Ah, yes
15:21:03 <sjmc7> My thought was (as i put in the review) if they already need a code path that supports not having searchlight, there should be a flag to force that path
15:22:57 <sjmc7> I don’t know if that makes sense/is possible, but I think it will be necessary not just for initial indexing but for reliability if SL becomes unavailable or needs resyncing/updating
15:23:10 <sjmc7> Since we’re a secondary data store i would not want nova to be totally reliant on SL
15:23:10 <Kevin_Zheng_Hex_> oops, disconnected
15:24:00 <Kevin_Zheng_Hex_> yeah, according to the current spec, there will be one config to set wheter use SL or not
15:24:45 <Kevin_Zheng_Hex_> but it will be stop nova -> unset flag -> start nova -> re-sync -> stop nova -> set flag -> start nova
15:24:49 <Kevin_Zheng_Hex_> very ugly
15:24:51 <sjmc7> urgh
15:24:57 <sjmc7> yeah, an API flag seems much more sensible
15:25:19 <Kevin_Zheng_Hex_> but seems API sub team don't like it
15:25:28 <sjmc7> it’s not just at initial openstack deployment we need to sync
15:25:34 <sjmc7> after major releases we probably will
15:25:56 <sjmc7> we don’t have the same rolling upgrade capabilities nova does
15:26:49 <Kevin_Zheng_Hex_> and nova keeps read from it's own DB while listing ... due to its extension mechanism
15:26:58 <Kevin_Zheng_Hex_> but it's a nova problem
15:27:07 <sjmc7> Yeah. Anyway, i guess anyone who’s interested keep an eye on that review (https://review.openstack.org/441692)
15:27:15 <sjmc7> They do seem quite serious about making progress on it
15:27:16 <lei-zh> yep, if you consider SL as kind of cache for nova DB, a way to refresh at anytime is necessary
15:27:41 <sjmc7> Our versioned notification BP is at https://blueprints.launchpad.net/searchlight/+spec/nova-versioned-notifications
15:28:02 <sjmc7> I see Kevin_Zheng has assigned that to himself, but let us know if you need help
15:28:14 <sjmc7> The ironic plugin uses versioned notifications if you have any trouble getting it set up
15:28:58 <Kevin_Zheng_Hex_> ah... I didn't assign it...
15:29:38 <sjmc7> Hahaha, maybe TravT did in a fit of wishful thinking
15:29:40 <lei-zh> lol
15:29:44 <sjmc7> I’ll unassign you for now
15:29:52 <Kevin_Zheng_Hex_> I thought we already adopt versioned noifiation?
15:30:04 <sjmc7> Not for the nova plugin
15:30:11 <lei-zh> flavor did?
15:30:12 <sjmc7> They were still writing them until Ocata
15:30:14 <Kevin_Zheng_Hex_> flavor versioned notification
15:30:19 <sjmc7> Ah, yes
15:30:21 <lei-zh> but not for instance
15:30:29 <Kevin_Zheng_Hex_> yeah, thats true
15:30:30 <sjmc7> Ok, so it may be easy to get set uip
15:30:35 <sjmc7> The payloads are very different
15:30:36 <TravT> ah, yeah, i did assign kevin.... my bad
15:30:59 <Kevin_Zheng_Hex_> I don't know I have enough time
15:31:15 <Kevin_Zheng_Hex_> intergration seems tons of work
15:31:27 <Kevin_Zheng_Hex_> nova intergration
15:32:06 <Kevin_Zheng_Hex_> I mean for the versioned notification thing
15:32:43 <sjmc7> Yeah, it may be. I’ll try to dig out some time to do it
15:33:32 <Kevin_Zheng_Hex_> But I would help if intergration went smooth :)
15:33:59 <Kevin_Zheng_Hex_> Just need to figure out the plan and then it might be easier
15:34:19 <sjmc7> Yep. This cycle’s longer than Ocata so hopefully we’ve got a decent shot at it
15:35:00 <lei-zh> Is supporting nova delete also on the agenda?
15:35:10 <sjmc7> The soft delete thing?
15:35:14 <lei-zh> yep
15:35:35 <sjmc7> Yeah, that’s a good point.. Apparently nova needs the capability to list “deleted” instances
15:35:52 <Kevin_Zheng_Hex_> yeah, and there is also soft_delete
15:36:02 <Kevin_Zheng_Hex_> we should do both
15:36:56 <sjmc7> Yeah. Historically the problem is that we can add filters to a plugin (“.. AND NOT deleted”) but allowing the user to override it isn’t supported
15:37:33 <sjmc7> Looking at the code though, it seems like it would be possible to add limited support where we’d look for {“term”: } type query clauses to override default filters like that
15:38:01 <sjmc7> It wouldn’t apply to RBAC filters, just data ones. This came up with glance a while back for some reason i can’t remember. community images i think
15:38:29 <sjmc7> I’ll add a blueprint for it
15:38:43 <lei-zh> you mean the problem is  for query_string or for all queries
15:39:08 <sjmc7> yeah, i think we can’t cope with query_string but that’s ok
15:40:01 <sjmc7> if you want to override it, you have to provide a structured query with {“term”: {“deleted”: [true, false]}}
15:41:10 <lei-zh> yeah, this seems more complicated than versioned notifications
15:41:40 <lei-zh> I'm often in trouble fighting with ES queries
15:42:27 <sjmc7> Yeah, the syntax is a bit overwhelming
15:42:38 <sjmc7> I keep meaning to look to see if they can be simplified a bit
15:43:22 <Kevin_Zheng> can we have a easier client？
15:43:40 <sjmc7> Our client does allow simplified queries
15:44:11 <sjmc7> We could add a flag for this, although it would be a bit special case
15:44:50 <Kevin_Zheng> And the response looked too nested
15:45:16 <sjmc7> The hits.[0]._source ?
15:45:18 <Kevin_Zheng> That's just what I feel while working on pic
15:45:24 <Kevin_Zheng> Poc
15:45:27 <Kevin_Zheng> Yeah
15:45:47 <Kevin_Zheng> But that no big deal :)
15:45:53 <sjmc7> Yeah.. I have wondered whether we should have abstracted away from Elasticsearch more
15:46:04 <sjmc7> There are/were good reasons we did it this way, but can always change
15:46:19 <sjmc7> Feel free to file BPs or bugs if tehre are things that get in your way
15:47:12 <Kevin_Zheng> Maybe we could have a param in client
15:47:40 <Kevin_Zheng> To choice whether nested result or simple one to return?
15:47:44 <sjmc7> To transform the result? Yeah
15:47:54 <sjmc7> That’d be easy to do
15:48:08 <Kevin_Zheng> Good , I will try tomorrow
15:48:19 <sjmc7> You’d lose some of the information (like total results, timing)
15:48:41 <sjmc7> But if all you want is the _source that’s ok
15:49:00 <sjmc7> Ok, will open it up for a few more minutes
15:49:04 <sjmc7> #topic open discussion
15:49:16 <Kevin_Zheng> Yeah, feels like do it in nova didn't look nice
15:49:30 <sjmc7> Yeah, no objection to making that an option on the client
15:49:36 <sjmc7> Any more on this issue? Or any others?
15:49:47 <Kevin_Zheng> :)
15:50:29 <rosmaita> i have a question from out of left field, so i'll wait until we're sure there's no follow-up from the above discussion
15:50:41 <sjmc7> i’m drooling with anticipatin
15:51:00 <rosmaita> sounds like one of those new statin drugs
15:51:09 <sjmc7> hahahaha
15:51:30 <sjmc7> once there are no more FDA regulations i’ll try marketing it! what was your question?
15:51:59 <rosmaita> someone asked me whether you need keystone to run searchlight, and i said yes because searchlight needs to know your roles so it can expose info appropriately
15:52:13 <rosmaita> but nikhil pointed out htat therere's a paste option for noauth
15:52:24 <rosmaita> so i was wondering what hte correct answer is
15:52:52 <sjmc7> you don’t *need* it, but you do need the headers indicating your tenant
15:53:08 <sjmc7> tenant and/or admin roles
15:53:17 <rosmaita> so basically, it's simpler to run with keystone in place
15:53:19 <sjmc7> so same as any other openstack service
15:53:32 <rosmaita> gotcha
15:53:45 <sjmc7> if you handle auth some other way, you can take out keystone
15:54:06 <sjmc7> but with the noauth middleware you’re saying “anyone can access anything”
15:54:25 <sjmc7> so you’d need to hide the API behind something
15:54:39 <rosmaita> ok, thanks
15:54:45 <sjmc7> i’m somewhat curious now :)
15:54:55 <sjmc7> our API code is very similar to glance’s since we branched it
15:56:10 <rosmaita> yeah, it looked pretty familiar
15:57:18 <sjmc7> if he has details on what he wants to do he can come ask
15:57:32 <sjmc7> ok, gonna call it here. i’ve created the pike series and milestones in launchpad so stuff can be scoped to it
15:57:36 <sjmc7> thanks everyone!
15:57:43 <rosmaita> thank you
15:58:04 <sjmc7> good night/morning/afternoon all
15:58:07 <sjmc7> #endmeeting