15:01:55 <TravT> #startmeeting openstack search
15:01:56 <openstack> Meeting started Thu Aug  6 15:01:55 2015 UTC and is due to finish in 60 minutes.  The chair is TravT. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:00 <openstack> The meeting name has been set to 'openstack_search'
15:02:02 <nikhil_k> and stepping on toes of people, so good things to make the most of time I sup
15:02:19 <TravT> o/
15:02:24 <nikhil_k> o/
15:02:36 <lakshmiS> o/
15:02:45 <nikhil_k> TravT: your hi-5 is very hip ;)
15:02:56 <TravT> lol
15:03:00 <sjmc7> o/
15:03:19 <rosmaita> o/
15:03:28 <TravT> Ok, i know sigmavirus24 is around
15:03:41 <TravT> here's our agenda:
15:03:42 <TravT> https://etherpad.openstack.org/p/search-team-meeting-agenda
15:03:46 <TravT> add anything you want to it
15:03:59 <TravT> i don't have any general status updates today.
15:04:07 * sigmavirus24 neither
15:04:10 <sigmavirus24> except webob is awful
15:04:14 <TravT> lol
15:04:24 <TravT> so, on to topics
15:04:30 <TravT> #topic Testing
15:04:57 <TravT> Python 3 https://review.openstack.org/#/c/209939/
15:05:01 <sjmc7> yay!
15:05:14 <TravT> svilgelm just put that patch up...
15:05:43 <TravT> let's take a look at it and if it looks good, this should mean we can enable another set of gate tests for python, I believe
15:06:05 <lakshmiS> thats great. no need to test on case by case basis
15:06:40 <TravT> FYI if you use POSTMAN: https://www.getpostman.com/collections/8c0b1e05875c7c58e967
15:07:03 <TravT> I just have a few POSTMAN tests that I use for manual testing.  If you use postman, thought I'd share them
15:07:22 <lakshmiS> i think we should link that to documentation under examples/tools
15:07:39 <TravT> yeah, i was thinking maybe a wiki
15:07:44 <TravT> not sure that should be in source
15:08:15 <TravT> since they'll change...
15:08:35 <TravT> but the manual testing leads to the next topic.
15:08:37 <TravT> Functional Tests - https://blueprints.launchpad.net/searchlight/+spec/set-up-functional-tests
15:08:56 <TravT> we are missing functional / integration tests / whatever you want to call them.
15:09:06 <TravT> and this is starting to be more problematic i think
15:09:20 <sjmc7> yes... it's really easy to break stuff at the moment
15:09:32 <lakshmiS> I have started working on glance plugin tests, but getting elasticsearch up is the key
15:09:50 <sjmc7> lakshmiS - was the unmerged patch to glance any help, or not?
15:09:51 <TravT> does it make sense to build a mock ES layer?
15:10:06 <ekarlso> o/
15:10:13 <nikhil_k> oh boy
15:10:15 <sjmc7> TravT - i did consider that
15:10:17 <lakshmiS> sjmc7: yes i did take whatever i can
15:10:20 <nikhil_k> that's a black hole
15:10:37 <sjmc7> but yeah, it's potentially a lot of work. unfortunately there's no 'fake es driver' i could find
15:10:41 <lakshmiS> nikhil_k is assigned to build mock ES layer
15:10:50 <nikhil_k> :P
15:11:29 <TravT> lakshmiS: do you want to talk a bit more on what you are working on?
15:12:08 <nikhil_k> so, I thought we ere going with #link https://review.openstack.org/#/c/157209/ to begin with?
15:12:39 <lakshmiS> for timebeing i am spinning up ES and testing the glance plugin code for indexing image data
15:13:40 <TravT> how are you spinning up ES?
15:13:49 <TravT> i mean are you doing that manually?
15:13:54 <lakshmiS> manually :)
15:14:52 <TravT> okay, so, what can we do to help out here?
15:14:58 <sjmc7> i did do some work on that
15:15:10 <sjmc7> in that patch
15:15:49 <nikhil_k> sjmc7: wondering if the issue is that we need stronger gating?
15:15:50 <lakshmiS> sjmc7: i haven't created a new patch but I have code which I can also add to that patch
15:16:31 <nikhil_k> is the breaking stuff due to missing in-tree functional tests or is for cross project updates testing?
15:16:37 <sjmc7> nikhil_k - the problem is that the unit tests don't cover everything
15:16:40 <sjmc7> no, it's in-tree
15:16:45 <nikhil_k> gotcha
15:17:13 <nikhil_k> yeah, I was under the impression that the patch would land and give us more direction
15:17:35 * david-lyle slips in late
15:17:51 <TravT> so, this one in glance needs to be abandoned obviously
15:17:57 <sjmc7> that patch on its own can't land
15:18:42 <lakshmiS> I will create a new patch and merge with my code as startup then
15:19:02 <TravT> sjmc7, i know this is something you worked on early.  what do we need to with infra to make this work?
15:19:09 <sjmc7> i've never worked with infra..
15:19:54 <TravT> ok, so lakshmiS will port this over to searchlight
15:20:11 <TravT> and then maybe we should have follow up discussion / even a hangout if necessary
15:20:15 <nikhil_k> TravT: why infra?
15:20:30 <TravT> nikhil_k, i don't know... steve has that in his notes on the BP
15:20:35 <nikhil_k> ah
15:20:53 <nikhil_k> sjmc7: may be that's after we have done gating work, iiuc?
15:21:11 <sjmc7> i think my note was about getting it gating - i'm not sure elasticsearch is typiucally available
15:21:25 <TravT> well, it isn't typically available.
15:21:27 <sjmc7> the WIP glance patch assumed e-s was available on the system
15:21:33 <TravT> but our devstack installer installs it
15:21:33 <sjmc7> ok, so that's maybe what the note about infra was about
15:21:47 <TravT> when we install searchlight via our plugin
15:21:56 <TravT> thanks to ekarlso
15:22:21 <TravT> infra has a ES installation library that both we and ceilometer depend on
15:22:32 <TravT> is that what you mean?
15:23:02 <sjmc7> yeah, that sounds like it would be useful, so maybe it's a solved problem. like i say, i know very little about infra
15:23:19 <TravT> ok.  well, i can say that with devstack, you get a working env with ES
15:24:09 <lakshmiS> ceilometer might already have gating jobs with dependency on ES. I could talk with them on how they do it
15:24:36 <TravT> lakshmiS, that'd be great.
15:25:09 <TravT> we have some good patches in the pipeline and I would feel a lot better with these integration level tests in the gate.
15:25:46 <TravT> okay, so it seems like we have next steps there.
15:26:18 <TravT> lakshmiS, if you want to have a time setup to talk sooner, then please do.  I know with the timezone diff it can be hard.
15:26:35 <lakshmiS> sure
15:26:48 <TravT> #topic Planning a liberty 3 prioritization hangout (TravT)
15:27:04 <TravT> So, i'd really like to have a release at end of liberty 3.
15:27:21 <TravT> and we have a number of blueprints in the pipeline.
15:27:30 <TravT> some of them have patches, some don't.
15:27:58 <TravT> but i think we should start focusing most efforts on a few of them...
15:28:10 <david-lyle> lakshmiS: there is a gate-tempest-dvsm-ceilometer-es job in project-config/zuul/layout.yaml that looks promising
15:28:21 <lakshmiS> thx david-lyle
15:29:13 <TravT> functionally, I think nova plugin, designate plugin should be the functional target.
15:29:17 <sjmc7> from my perspective, the most important BPs i've done work on are swapping out 'index' and 'type' for an unchanging 'resource name'; nova; plugin config
15:30:12 <TravT> IMO we should focus on helping get through what has been started.
15:31:01 <TravT> And there are a number of items like you just mentioned sjmc7 that are about solidifying the api and code
15:31:22 <sjmc7> yeah. i'm working on your comments on the nova one; i'd like reviews on the 'resource name' one
15:31:32 <TravT> i did provide a review on the resource one
15:31:40 <TravT> there is some room for discussion in it.
15:31:49 <sjmc7> ok
15:31:55 <TravT> we have topics for those in just a sec.
15:31:55 <sjmc7> i'll try to spend some time today, been a busy week
15:31:58 <sjmc7> k
15:33:12 <TravT> so, maybe next week in this meeting or as a separate meeting after next week, we should have a walkthrough of the BPs and check their target series.
15:33:19 <TravT> that's all.
15:33:39 <TravT> #topic bug review
15:33:40 <nikhil_k> TravT: do you think we shoud have a action item for such things? or wait for the video conferencing stuff?
15:34:14 <TravT> nikhil_k: it might be a lot easier to do a video conference to walk through the blueprints and talk about them
15:34:24 <TravT> probably take less time and provide richer forum for discussion.
15:34:52 <TravT> I could send out a doodle poll for times.
15:34:57 <TravT> what do you think?
15:34:59 <rosmaita> TravT: +1
15:35:18 <TravT> ok
15:35:22 <sjmc7> yup
15:35:40 * TravT forgets the command for action items.
15:35:41 <lakshmiS> are you planning for next week?
15:35:59 <TravT> it can be next week or the week after.
15:36:03 <TravT> are you around next week?
15:36:15 <nikhil_k> TravT: works for me
15:36:26 <lakshmiS> yes. earlier the better to finalize on what BP's will go in
15:36:50 <TravT> #action TravT will setup a BP prioritization review meeting for Liberty 3
15:36:58 <nikhil_k> TravT: might worth doing some prep work on our sides for the meeting next week
15:37:00 <TravT> guess that wasn't it
15:37:24 <TravT> nikhil_k, ok
15:37:26 <nikhil_k> TravT: that's it!
15:37:46 <TravT> ok, i'll work on that.
15:37:52 <TravT> on to bug reviews.
15:38:11 <nikhil_k> thanks
15:38:14 <TravT> there are a few that already have some review.
15:38:17 <TravT> one of them is critical
15:38:32 <TravT> Already one +2
15:38:32 <TravT> Critical - Ceilometer / Notifications: https://review.openstack.org/#/c/202392
15:38:32 <TravT> High - Glance indexing https://review.openstack.org/207719
15:38:34 <TravT> Medium - Doc update https://review.openstack.org/208750
15:39:01 <TravT> Please try to look at them.
15:39:53 <TravT> #topic Blueprint review
15:40:14 <TravT> sjmc7, as we just discussed, i did put some comments on the nova patch.
15:40:23 <TravT> anything about that which we should discuss in larger forum?
15:40:39 <sjmc7> yep, working on them
15:40:52 <TravT> ok, cool.
15:40:56 <sjmc7> just the issue of denormalizing
15:41:04 <sjmc7> specifically what you called out with glance image names, flavor names
15:41:54 <TravT> yeah, what occurred to me looking through that patch was that we might have to think about the concept of plugins affecting other plugins.
15:42:07 <TravT> so, to explain briefly to everybody here.
15:42:09 <sjmc7> right - the nova plugin can't listen to glance notifications, for instance
15:42:27 <sjmc7> well.. i take that back, we can assign multiple handlers
15:42:40 <sjmc7> it does create that mass-update scenario though
15:42:51 <TravT> in the nova data, sjmc7 was putting things like image name in there when the instance is indexed.
15:43:08 <TravT> however image name can change independent of the instance
15:43:16 <nikhil_k> but that's from the instance response for the image-meta info
15:43:28 <nikhil_k> right?
15:43:31 <TravT> but the nova plugin which indexes instances doesn't get notifications related to that
15:43:36 <nikhil_k> right
15:43:42 <sjmc7> it can do, i can add a handler
15:44:00 <nikhil_k> thta was my concern with nova notifications in our call last week
15:44:31 <nikhil_k> I wonder if for nova we should make a call to the api on notification trigger?
15:44:44 <nikhil_k> if we do not want to go down the route of direct access to DB
15:44:55 <TravT> sjmc7, so, you are thinking that the nova plugin would include an image notification handler as well?
15:45:10 <sjmc7> no, but it needs to hook into glance notifications
15:45:17 <TravT> ok.
15:45:37 <TravT> i was trying to think how that would affect message consumption and acknowledgment
15:45:40 <lakshmiS> may there should be link from image updates independend of instance updates and link them in ES index
15:45:55 <nikhil_k> interesting
15:46:27 <nikhil_k> TravT: I think one more issue is
15:46:43 <nikhil_k> nova has static image-meta where as the one on the image itself might change
15:46:55 <nikhil_k> nova basically copies over that info
15:47:11 <TravT> and does nova ever update that data?
15:47:16 <rosmaita> no
15:47:40 <nikhil_k> I think nova admin api might
15:47:42 <TravT> is it even updated when an instance is first created?
15:47:45 <nikhil_k> need to double check
15:47:51 <rosmaita> how are we getting the image_ properties from nova?
15:48:00 <nikhil_k> API call
15:48:23 <rosmaita> i didn't think those were exposed in the API?
15:48:27 <nikhil_k> TravT: it's copied from glance when instance is (first) booted
15:48:59 <nikhil_k> rosmaita: some are, right? you are correct mostly all are hidden
15:49:02 <rosmaita> the novaclient displays image name, but it does a request to get the image detail and constructs it client'side
15:49:29 <TravT> so novaclient call's glance anyway?
15:49:44 <rosmaita> well, by proxying through the compute api
15:49:44 <TravT> or is it calling the nova server for image info and nova server is just proxying?
15:49:53 <nikhil_k> second
15:49:54 <TravT> (v1 / v2 fun)
15:50:03 <nikhil_k> nah
15:50:47 <nikhil_k> just bad separation
15:51:08 <nikhil_k> it might be worth investigating proposing a standard immutable prop in glance to avoid such things
15:51:20 <rosmaita> ideally the entries for servers in the index would contain only permanent info about images, probably just uuid
15:51:29 <rosmaita> but, what happens if hte image is deleted?
15:51:35 <rosmaita> much sadness
15:51:42 <nikhil_k> :)
15:51:44 <sjmc7> one option is we don't try to answer this
15:51:50 <nikhil_k> ++
15:51:54 <sjmc7> and we don't have image name, flavor name indexed with servers
15:52:00 <nikhil_k> rosmaita: I think it's worth having deleted images in ES index
15:52:01 <sjmc7> but that makes the searching much less impressive
15:52:11 <sjmc7> images that are deleted get deleted
15:52:24 * TravT misses relational joins sometimes.
15:52:51 <lakshmiS> well there are parent child relationships in ES we can explore
15:53:04 <nikhil_k> well I think we are going too far away. whatever we do it might be worth keep some options open for changing design later
15:53:08 <rosmaita> lakshmiS: that sounds cool
15:53:25 <lakshmiS> https://www.elastic.co/blog/managing-relations-inside-elasticsearch
15:54:01 <nikhil_k> lakshmiS: are there horizontal inheritance capabilities, I might be thinking something crazy to do here but it might work
15:54:04 <nikhil_k> ?
15:54:05 <sjmc7> for the sake of getting something working then, that is an option - don't try to denormalize now
15:54:16 <nikhil_k> cool
15:54:24 <TravT> i'm okay with that as starting point, i think.
15:54:29 <nikhil_k> sjmc7: works for me for now
15:54:37 <TravT> but we can talk through queries further
15:54:55 <sjmc7> ok. it does mean that cross-resource searching will be largely not-useful, i think
15:55:14 <nikhil_k> for now
15:55:17 <nikhil_k> :)
15:55:38 <TravT> the parent child isn't for nesting virtual docs, i think.
15:55:56 <TravT> i mean if image is index as a doc in index a type b
15:56:22 <TravT> the instance can't contain it as a reference, i thought.
15:56:30 <sjmc7> i would urge us to stay away from it
15:56:50 <TravT> ok... so, we won't have it in first pass.
15:57:07 <TravT> running out of time.
15:57:30 <TravT> the doc type review takes a little thought.
15:57:35 <TravT> so also please look at that one.
15:57:40 <TravT> i already added some comments
15:57:52 <TravT> https://review.openstack.org/#/c/207672/
15:58:01 <sjmc7> thanks, will take a look today
15:58:05 <sjmc7> other comments welcome
15:58:08 <TravT> and finally, designate plugin.
15:58:25 <sjmc7> yeah, let's get that in as well
15:58:32 <rosmaita> i thought they were all hidden since they are in instance_system_metadata table
15:58:36 <sjmc7> i haven't tested ekarlso's latest patch
15:58:44 <TravT> i did seem to finally get a working devstack with designate so that i could have some intelligence when reviewing the patch.
15:59:08 <TravT> but yesterday got clobbered with meetings.
15:59:21 <lakshmiS> lakshmiS is stilly unlucky with devstack install for designate
15:59:31 <TravT> #topic 12 seconds of open discussion
15:59:52 <sjmc7> :)  think we covered everything i wanted to
15:59:52 <nikhil_k> :)
16:00:06 <TravT> okay, that's it.  thanks everybody
16:00:11 <nikhil_k> THanks!
16:00:16 <TravT> #endmeeting