Tuesday, 2016-01-26

*** TravT has quit IRC00:13
*** TravT has joined #openstack-searchlight00:14
*** yingjun has joined #openstack-searchlight00:25
*** david-lyle has joined #openstack-searchlight00:42
*** bpokorny has quit IRC01:38
*** yingjun has quit IRC01:47
*** david-lyle has quit IRC02:05
*** yingjun has joined #openstack-searchlight02:41
*** lakshmiS has quit IRC02:43
*** david-lyle has joined #openstack-searchlight03:25
*** GB21 has joined #openstack-searchlight07:05
*** GB21 has quit IRC07:59
*** GB21 has joined #openstack-searchlight08:27
*** yingjun has quit IRC10:11
*** yingjun has joined #openstack-searchlight11:00
*** yingjun has quit IRC11:12
*** yingjun has joined #openstack-searchlight11:13
*** yingjun has quit IRC11:17
*** yingjun has joined #openstack-searchlight11:42
*** yingjun has quit IRC12:43
*** yingjun has joined #openstack-searchlight12:50
*** yingjun has quit IRC13:29
*** yingjun has joined #openstack-searchlight13:30
*** yingjun_ has joined #openstack-searchlight13:34
*** yingjun has quit IRC13:34
*** yingjun has joined #openstack-searchlight13:37
*** yingjun_ has quit IRC13:38
*** yingjun has quit IRC14:01
*** yingjun has joined #openstack-searchlight14:10
*** yingjun has quit IRC14:26
*** yingjun has joined #openstack-searchlight14:32
*** yingjun has quit IRC14:45
*** RickA-HP has joined #openstack-searchlight14:58
*** yingjun has joined #openstack-searchlight14:59
TravThello15:00
yingjunhi15:01
sjmc7morning15:01
rosmaitaRickA-HP: will you be insulted if i begin referring to you as "mr. erudite" ?15:01
rosmaitao/15:01
RickA-HPGood morning/evening15:01
sjmc7:D15:01
RickA-HP:)15:01
TravThow is everybody doing today15:01
TravT?15:01
RickA-HPNot too bad for 8:00am!15:02
rosmaitaanother snow day here, so wfh15:02
TravTi wasn't sure if lakshmiS would make it...15:02
TravTstill a bit early in California and I think he has to get his kids to school15:03
TravTnikhil: david-lyle: FYI we're gonna be talking specs right now15:03
TravTAllright, let's start with what I hope is an easier one.15:04
RickA-HP:)15:05
TravTThe notification forwarding one.  It already has 3 +2's and a +1 from Fei Long who is the zaqar PTL.15:05
TravThttps://review.openstack.org/#/c/246220/15:05
TravTunless there are any objections, i was going to go ahead an +A it.15:05
rosmaitalooks like work is already underway on it15:06
RickA-HPIt looked good to me15:07
TravTYes there is an initial patch up15:07
TravThttps://review.openstack.org/#/c/271958/15:07
TravTwhich is very early15:07
TravTbut it is good to see progress15:07
TravTOkay, going once15:08
rosmaitai don't see any reason not to approve, unless there's a question of reviewer bandwith or something15:08
TravTwell, there's always some question of reviewer bandwidth, but I'd rather get this in prior to going to 1.0 with searchlight.15:09
TravTbefore we have to take deprecation more seriously15:09
rosmaitagood point15:09
TravTok going twice15:09
rosmaitalet's do it15:09
TravTSold to the man from Pennsylvania15:09
TravTok, +A applied15:10
*** sigmavirus24_awa is now known as sigmavirus2415:11
TravTok, so now on to the zero downtime:15:12
TravThttps://review.openstack.org/#/c/245222/15:12
openstackgerritMerged openstack/searchlight-specs: This BP is implementing a notification-forwarding for SearchLight index  https://review.openstack.org/24622015:13
TravTRickA-HP: I added some comments, but they are mostly just minor cleanup items.15:14
sjmc7i need to go through the current patchset15:14
rosmaitame too15:14
TravTso, want to take 10 - 15 minutes now and we'll pick chat up?15:15
RickA-HPTravT: I saw your comments. I was waiting for more before cleaning up and submitting a new patchset.15:15
rosmaitaTravT: +115:16
*** yingjun has quit IRC15:22
*** david-lyle_ has joined #openstack-searchlight15:27
sjmc7TravT, RickA-HP : https://www.elastic.co/guide/en/elasticsearch/reference/1.7/indices-aliases.html#filtered (the line before that section header) throws a wrench in the works15:34
TravTreading15:34
RickA-HPHmmmm....15:35
TravTok, this is the filtered aliases we discussed before possibly as part of role based filtering (but you didn't use)15:35
sjmc7above that15:35
*** yingjun has joined #openstack-searchlight15:35
sjmc7the sentence before the section header15:35
sjmc7“It is an error to index to an alias which points to more than one index."15:35
TravTooh15:35
sjmc7gonna do a quick test15:36
RickA-HPIs this referring to the glob pattern for specifying multiple indexes?15:37
sjmc7don’t think so, checking15:37
TravTthat seems extraordinarily odd15:37
TravTgiven that above that it explicitly talk about adding multiple indexes to an alias15:37
sjmc7the common use for aliases is for logstash type applications15:38
RickA-HPIt seems that they are implying that multiple indexes in an alias is to be use only for querying.15:38
sjmc7where you’re using them to search15:38
sjmc7they’re not implying it :)15:38
TravTwould this just mean that the listeners and sync job would just retrieve the indexes in an alias and then push to both indexes directly15:38
sjmc7could do. that introduces a race condition15:39
TravTanything we can do to avoid all the overhead i originally put in the spec (now in the alternatives sections)15:39
sjmc7curl localhost:9200/test-alias/test-type/1 -XPUT '{"a": "b"}'15:40
sjmc7{"error":"ElasticsearchIllegalArgumentException[Alias [test-alias] has more than one indices associated with it [[test-2, test-1]], can't execute a single index op]","status":400}15:40
TravTallright, back to the drawing board!15:40
sjmc7yeah, sorry :(  i should’ve thought to check earlier15:41
RickA-HPSteve, good catch!15:41
sjmc7better late than later15:41
* TravT seems to remember discussion between us of running this test a month ago15:41
rosmaitathat is a ... bummer15:41
TravTso, alias switching is still required15:42
sjmc7it is a bit. the good news is it’d raise an error :)15:42
TravTwe just can't publish to the alias15:42
sjmc7the brute-force method would be to pull the index names out of the error and retry...15:42
TravTyes, the could serve as the communication mechanism I suppose.15:43
sjmc7might be safer than a half-assed home-grown solution for a case tht’s in use for a tiny percentage of time15:43
TravTsync job adds the multiple indexes15:43
TravTerror trapped15:43
RickA-HPOr the listener would have two aliases, one corresponding to the old index and one to the new index.15:43
TravTpublishes to all indexes15:43
sjmc7though you know me, i LOVE half-assed homegrown solutions15:43
RickA-HPOnce again, Searchlight would manage the aliases.15:43
TravTRickA-HP, that gets back to the alternative section15:44
sjmc7and it’d publish to both aliases?15:44
sjmc7i think actually the error trapping is simplest/safest, if not terribly elegant15:44
RickA-HPTravis, maybe it's time we rename the "alternative" section!15:44
TravTwe have to either put APIs and a bunch of management communication on talking to them15:45
TravTor try to do something like error trapping15:45
rosmaitai'm with sjmc7 on the error trapping approach15:45
sjmc7it’s times like this i really miss Visual Basic’s On Error Resume Next15:45
TravTwhich I don't think is that bad15:45
sjmc7everything was so much simpler15:45
sjmc7yeah, ruminate on it15:45
sjmc7other than that i have a few minor nits, but mostly looks good15:46
sjmc7and kind of agree it looks like a patent application :)15:46
TravTsjmc7: except Exception: pass15:46
rosmaitaRickA-HP: did you do the diagrams? nice work15:47
sjmc7OERN was like a global version of that. it was glorious15:47
sjmc7try: <do everything> except: pass15:47
sjmc7yeah, the diagrams are nice15:47
TravTsjmc7: ES docs say it is atomic15:48
TravTRenaming an alias is a simple remove then add operation within the same API. This operation is atomic, no need to worry about a short period of time where the alias does not point to an index:15:48
sjmc7ah, my mistake - i thought it was like bulk indexing15:49
sjmc7man, i’m off form today15:49
TravTi'd say you are quite on form, finding that little tidbit in the docs about aliases15:49
sjmc7think i’m batting about 0.100 for the morning!15:50
TravTokay, my vote is for switching this up a bit to use error trapping... would be nice to optimize it somehow, but that's something Rick can explore in code15:52
TravTnot sure you'd have anything faster than ES giving an instant response of more than one index in an alais15:52
sjmc7nope15:53
TravTthe error includes the indexes15:53
sjmc7right. i don’t think it’s that ugly15:53
TravTso no additional query needed15:53
rosmaitajust need to hope no one "optimizes" the error message upstream15:53
sjmc7:D  that’d be unfortunate15:54
*** GB21 has quit IRC15:54
RickA-HPOr changes the format of the error messages.15:54
TravTwhat are you talking about? error messages and APIs never change15:54
TravT;)15:54
rosmaitaanyway, i like the plan15:54
* TravT glares at rosmaita for trying to throw more monkey wrenches into our hacking15:54
* rosmaita is glare-proof15:55
TravTtime for a beer, gentleman15:55
rosmaitabesides, we don't index Glare yet, do we?15:55
TravTno, what is the status on that anyway?15:55
TravTglare, that is15:55
sjmc7and puns15:55
rosmaitait's going to be its own endpoint, i think15:56
RickA-HPMaybe there is another apporach. What if the search alias had  multiple indexes and the listener alias only has the "new" index.15:56
TravTRickA-HP: please expand?15:56
RickA-HPLet me get it straight in my mind first :)15:57
RickA-HPThe Glance listener starts indexing into indexA. The Glance search is querying indexA.15:58
RickA-HPWhen we get a reindex request for Glance, the GLance listener switches to a new index indexB.15:58
RickA-HPThe Glance search will now query on IndexA and IndexB.15:59
sjmc7then you’ll get two results for everything?15:59
TravTyou'd have duplicate results15:59
RickA-HPIs there a way to filter out dups?15:59
TravTand if we are re-indexing because A contains things that should have been deleted15:59
TravTyou'll still get the things in A that should not be returned16:00
*** yingjun has quit IRC16:00
TravTas a note the docs should have the same ID16:00
TravTso maybe they'd filter out16:00
sjmc7nope, it’d give you both16:00
TravTbecause diff index?16:00
sjmc7yep16:00
RickA-HPWe would still get the deleted docs in indexA with our current solution.16:00
TravTtrue16:01
*** GB21 has joined #openstack-searchlight16:01
TravTwe'd have to filter all results for dupes16:01
TravTand prefer results from new index16:01
TravTrick that isn't quite true if listener is going to both on deletions16:02
TravTif listener is still going to both, then deletes would be removed in A16:03
TravTunless your deletion journal concept goes in16:03
TravTand the filtering includes handling that filtering as well16:03
TravT(deleted doc in index B means filter out of results from index A)16:04
TravTthis seems a little fragile.16:04
RickA-HPYes, it seems overly complicated.16:04
TravTi think it is good idea to put some into the alternative section at least16:04
RickA-HPIt feels like defects are just waiting to be written with this approach.16:04
TravTby the way, i like that you brought up an inverse idea16:05
TravTfor consideration16:05
TravTok, so I guess Rick will publish updates to the spec today. if any other grand idea comes up, we'll go over it.16:06
*** yingjun has joined #openstack-searchlight16:06
rosmaitasounds good!16:06
TravTwill look for that, but let's also talk about deletion journal16:06
RickA-HPWe're over time for the meeting. Should we still continue or schedule another meeting to discuss deletions.16:06
TravTdo you all still have a little time?16:07
rosmaitayes, let's keep going16:07
sjmc7let’s do the deletions one since there was some confusion16:07
TravTyingjun: sorry it is late for you16:07
RickA-HPI have time to continue the discussion.16:07
TravTif you need to drop, we understand16:07
TravTthe irc room is logged so you can catch up if needed16:07
TravThttps://review.openstack.org/#/c/269783/16:07
rosmaita+1 for deletions spec16:08
rosmaitaand sorry for the confusion16:08
TravTrosmaita: no need to be sorry at all. very much appreciate calling out questions.16:09
sjmc7itwas good to clarify it16:09
RickA-HPYes, if it was confusing it could have meant that it was incomplete.16:09
TravTi actually found myself confused yesterday trying to remember why it was needed16:10
TravTas evidenced by my current comments on it16:10
RickA-HPWe also uncovered some subtleties in the design that need careful thought when implementing.16:11
rosmaitaso what's really weird is that a PUT to a deleted doc can re-create it?16:11
rosmaitaor am i completely misunderstanding?16:11
sjmc7rosmaita: yeah, because we use the openstack id for documents16:11
rosmaitagotcha16:11
sjmc7to avoid having to remove old copies on updates/deletes16:12
sjmc7it’s always a problem in eventually-consistent systems16:13
sjmc7or rather, it’s always a problem with asynchronous operations16:14
sjmc7i think even if we were using SQL we’d have to do this because we’ll never guarantee we process everything every event order, even if we receive them in order16:14
RickA-HPAre there any unresolved issues with this spec (as opposed to some more clean up)?16:15
TravTWell, i think my comment on line 54 is a case that SQL, nosql, anything can't handle on its own16:15
rosmaitai'm a bit vague on how the TTL will be coordinated with the reindexing16:15
TravTthe fact you have multiple processes using different routes (one via API callbacks and one via notifications) to get data16:16
*** yingjun has quit IRC16:16
TravTRick, re-reading16:17
TravTbut where does it catch the fact that we actually STORE deleted documents16:17
TravTe.g. we get a deleted document and we actually need to store it and set it to deleted16:18
TravTwith TTL16:18
TravTwithout that, I don't see how it handles the race condition.16:18
*** david-lyle_ has quit IRC16:18
rosmaitai wonder whether instead of deleted: true it would be worth having deleted: version-num-of-doc-when-deleted16:19
sjmc7the deleted doc’s version will still get updated. but yeah, we could do that. i don’t think it’d make querying much more expensive (test for the presence/absence of the field)16:20
sjmc7what would we gain, rosmaita ?16:20
rosmaitasjmc7: not sure16:20
sjmc7actually, that is a point; on a deletion, we’ll no longer be able to generate a version (timestamp)16:21
TravTcan i -1 RickA-HP for using funny and appropriate phrases that I have to google?16:21
rosmaitai think nova uses the epoch time for the 'deleted' column in the databases16:21
rosmaitaTravT: no, but you can call him "mr. erudite"16:22
TravTok, maybe we can vote that he has to change his irc nick to that16:22
sjmc7i’m not sure how much information we get in notifications. maybe i’m misremembering; it’d make sense if we got a notification that included the time16:22
*** GB21 has quit IRC16:23
TravTi'm not opposed to the deleted field having more info if we have it16:25
sjmc7we have to have the info; we need to supply a version number16:25
TravTmaybe filtering is faster if it is only a boolean?16:26
* TravT just trying to think of pros / cons16:26
rosmaitayeah, i'm a "just in case" kind of guy, but there's no sense having extra overhead if we don't really need it16:26
sjmc7can use an existence check. i dont’ think that’d be a significant concern. i’m more worried we don’t have enough information to put the right version on ‘deleted’ docs. i’d need to look at the notificaitons16:27
RickA-HPI was starting to look at notifications, but it will take longer than this meeting.16:27
RickA-HPIt sounds like there is still some investigation needed for the delete field.16:28
TravTexistence check would work no matter what is stored.16:29
rosmaitai have to drop for next 1/2 hour16:29
TravTRickA-HP: i don't feel like what is actually stored in the field is worth holding up the BP over... more concerned on just a bit more clarity, on the use case..16:29
RickA-HPOk16:30
sjmc7right, i think that’s an implementation thing16:30
sjmc7i do think it’s worth making sure we can correctly assigna  version to ‘deleted’ docs, since it doesn’t work if we can't16:31
TravTideally, it would be best to use explain it in the first paragraph or so in the least amount of words as possible.16:31
sjmc7starting to wonder if there’s a timestamp rabbitmq assigns we can use instead of anything specific to the resource16:32
TravTwhich I think might be done in the problem description section16:32
sjmc7anywho16:32
TravTsjmc7: i think we had those questions on the review from lei16:32
TravTneed to check that16:32
TravToslo timestamp16:32
TravThttps://review.openstack.org/#/c/255751/16:33
TravTok, so, it seems like just a few minor updates to the deletion spec and it should be good to g16:33
RickA-HPTravT: I'll update the problem description to be more focused on zero downtime.16:34
TravTno more nitpicks after next update.  only "THIS CAN'T BE DONE" kind of comments.  Everybody agree to that?16:34
RickA-HPI do :)16:34
sjmc7yeahyep16:34
rosmaitaok16:34
RickA-HPI'll update both the zero downtime and deletion specs today and push them out this afternoon.16:34
TravTcool, thanks guys.  Great focus.16:34
TravTsjmc7: I'll try to dig more into the role based patch today16:35
TravTstarted yesterday16:35
TravTlots of files16:35
sjmc7so many files16:35
RickA-HPWhat other reviews do need more attention?16:35
TravThttps://review.openstack.org/#/c/257516/16:35
TravTrosmaita: sjmc7: we should probably just push the global requirements patches through.16:36
sjmc7yeah16:37
TravTI'll also test out this one: https://review.openstack.org/#/c/267881/16:37
rosmaitaTravT: ok16:38
TravTRickA-HP: this URL should help you16:38
TravThttps://review.openstack.org/#/q/project:%255E.*searchlight.*+status:open,n,z16:38
TravTAll right guys, thanks for the time this morning.16:38
RickA-HPTravT: Travis, yeah I know the open reviews, I was just wondering if there were some higher priority than others.16:39
TravTSteve' role based one is highest priority16:40
sjmc7ALL my patches are the highest priority!16:48
TravTsjmc7: actually at least 3 of them are...16:49
sjmc7but yes, that one has ramifications in other places16:49
TravTthree cheers to lakshmi, made progress on the glance notifications16:49
TravThttps://review.openstack.org/#/c/221307/16:49
sjmc7https://review.openstack.org/#/c/267864/ is a little simpler but is sort of a dependency for other plugins16:49
sjmc7ah, nice16:49
TravThe's not here to receive the cheers16:49
TravT:(16:49
*** RickA-HP has quit IRC16:50
sjmc7he snoozed and losed16:50
sjmc7or did real life things, like taking children to school16:50
*** lakshmiS has joined #openstack-searchlight16:59
*** RickA-HP has joined #openstack-searchlight17:01
*** bpokorny has joined #openstack-searchlight17:06
*** openstackgerrit has quit IRC17:17
*** openstackgerrit has joined #openstack-searchlight17:18
*** sigmavirus24 is now known as sigmavirus24_awa17:59
*** sigmavirus24_awa is now known as sigmavirus2418:03
*** sigmavirus24 is now known as sigmavirus24_awa18:04
openstackgerritOpenStack Proposal Bot proposed openstack/searchlight: Updated from global requirements  https://review.openstack.org/27165218:14
openstackgerritLakshmi N Sampath proposed openstack/searchlight: WIP - Swift plugin Pending RBAC and Code tuning.  https://review.openstack.org/27162218:33
sjmc7lakshmiS: middleware’s going to be problematic for swift notifications. if you POST metadata, any existing metadata won’t be visible short of loading the objects from the metadata store. i’ll see how difficult it would be to inject notifications into the swift code itself18:50
lakshmiShmm18:55
lakshmiSin the worst case i can do incremental metadata update by lookup from es first18:56
*** sigmavirus24_awa is now known as sigmavirus2418:57
sjmc7i’ll see how tough it would be to do the notifications inside swift, like other services do19:03
lakshmiSi find it easier to create notifications from inside a service; atleast in the case of glance19:14
sjmc7the plumbing was mostly there in glance19:25
sjmc7and also middleware’s easily installable if it’s not in-tree19:25
*** bpokorny_ has joined #openstack-searchlight19:30
*** bpokorny_ has quit IRC19:31
*** RickA-HP has quit IRC19:31
*** bpokorny_ has joined #openstack-searchlight19:31
sjmc7yeah, maybe stick with updating metadata19:33
*** bpokorny has quit IRC19:34
lakshmiSok19:36
TravTlakshmiS: great job on the metadef notifications20:10
TravT12:33 openstackgerrit: Merged openstack/glance: Fix for Image members not generating notifications  https://review.openstack.org/22130720:10
TravT12:36 flaper87: lakshmiS: TravT ^20:10
TravT12:36 flaper87: sorry it took so long20:10
*** nikhil has quit IRC20:27
*** nikhil has joined #openstack-searchlight20:28
lakshmiSyeah finally. thanks to glance team20:41
openstackgerritTravis Tripp proposed openstack/searchlight: Added Keystone and RequestID headers to CORS middleware  https://review.openstack.org/26539620:50
TravTkrotscheck: I'll probably regret asking you this because I have to join a meeting in 2 minutes and won't be around, but maybe you can help me out here.20:58
TravTgetting a CORS error when going against searchlight on local host20:59
TravThttp://paste.openstack.org/show/485077/20:59
TravT(POSTMAN) -> Searchlight (localhost) -> ElasticSearch on VM @ 192.168.200.20021:00
TravTThat fials21:00
TravTfails21:00
TravTwith the error I pasted21:00
TravTAny ideas?21:00
sjmc7lakshmiS: stuffed the swift middleware up at https://github.com/sjmc7/swift-messaging-middleware21:03
lakshmiSok thanks21:04
*** RickA-HP has joined #openstack-searchlight21:09
sjmc7also lakshmiS , i’ve changed my mind - i’d be ok merging the swift indexing patch before notifications if it makes it easier to test/review21:24
sjmc7as long as we get the notifications in by mitaka21:24
*** bpokorny_ has quit IRC21:26
*** bpokorny has joined #openstack-searchlight21:27
*** bpokorny has quit IRC21:28
*** bpokorny has joined #openstack-searchlight21:29
lakshmiSgood to see searchlight is python3 compatible at https://wiki.openstack.org/wiki/Python3#Other_OpenStack_Applications_and_Projects21:35
sjmc7:)21:35
sjmc7yep, test run in 3.421:35
*** lakshmiS has quit IRC22:19
*** sigmavirus24 is now known as sigmavirus24_awa23:28
*** lakshmiS has joined #openstack-searchlight23:30
openstackgerritRick Aulino proposed openstack/searchlight-specs: Zero Downtime Reindexing Proposal  https://review.openstack.org/24522223:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!