| *** sballe has joined #openstack-searchlight | 01:30 | |
| *** TravT has quit IRC | 02:53 | |
| *** sigmavirus24 has quit IRC | 03:27 | |
| *** sigmavirus24 has joined #openstack-searchlight | 03:29 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 03:39 | |
| *** lakshmiS has joined #openstack-searchlight | 03:48 | |
| *** sballe has quit IRC | 03:56 | |
| *** lakshmiS has quit IRC | 06:00 | |
| *** lakshmiS has joined #openstack-searchlight | 06:03 | |
| *** GB21 has joined #openstack-searchlight | 06:27 | |
| *** lakshmiS has quit IRC | 08:03 | |
| *** lakshmiS has joined #openstack-searchlight | 08:03 | |
| *** GB21 has quit IRC | 09:30 | |
| *** GB21 has joined #openstack-searchlight | 09:45 | |
| *** GB21 has quit IRC | 10:31 | |
| ekarlso | sjmc7: u on ? | 12:43 |
|---|---|---|
| *** sballe has joined #openstack-searchlight | 12:43 | |
| ekarlso | lakshmiS: u around ? | 13:53 |
| *** TravT has joined #openstack-searchlight | 13:58 | |
| lakshmiS | hey ekarlso: | 14:01 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:03 | |
| openstackgerrit | Endre Karlson proposed openstack/searchlight: Add Designate plugin. https://review.openstack.org/199099 | 14:24 |
| ekarlso | TravT: so in the latest PS i'm wonder why the delete_by_query deletes all the recordsets when deleting a domain? seems that the filter for zone_id doesn't have any effect ? | 14:25 |
| TravT | ekarlso: let me take a look | 14:26 |
| TravT | ok, i'm gonna pull it and try it out. | 14:30 |
| sjmc7 | hmm, it's also been deprecated as of v1.5.3 | 14:33 |
| TravT | it'll be nice when this is in and I don't have to rebuild tox | 14:34 |
| sjmc7 | you shouldn't have to rebuild it - .tox/<env>/bin/pip install -e . should get all the dependencies and entrypoints | 14:35 |
| TravT | sjmc7: yep, see that in the docs | 14:35 |
| TravT | https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html | 14:35 |
| sjmc7 | it looks from the docs that the 'q=' parameter takes a querystring, ekarlso | 14:37 |
| sjmc7 | so q='zone_id:%s' % zone_id | 14:37 |
| sjmc7 | or you can use body={'query': {'term': {'zone_id': zone_id}}} | 14:37 |
| TravT | yeah, but it looks like we should do the bulk delete | 14:38 |
| ekarlso | TravT: bulk delete ? | 14:46 |
| TravT | from elasticsearch import helpers | 14:47 |
| TravT | actions = [] | 14:47 |
| TravT | for document in documents: | 14:47 |
| TravT | action = { | 14:47 |
| TravT | '_id': document.get(self.document_id_field), | 14:47 |
| TravT | } | 14:47 |
| TravT | actions.append(action) | 14:47 |
| TravT | helpers.bulk( | 14:47 |
| TravT | client=self.engine, | 14:47 |
| TravT | index=self.index_name, | 14:47 |
| TravT | doc_type=self.document_type, | 14:47 |
| TravT | chunk_size=self.chunk_size, | 14:47 |
| TravT | actions=actions) | 14:47 |
| TravT | but do a search first using the search_type of scan, I think... looking at python client api for that | 14:47 |
| sjmc7 | think you need a 'delete' in the action | 14:47 |
| TravT | yep, somehow missed that | 14:48 |
| sjmc7 | if we're expecting thousands of records per zone, a scan is probably worth doing | 14:48 |
| sjmc7 | also in helpers | 14:49 |
| TravT | helpers.bulk( | 14:50 |
| TravT | action='delete', | 14:50 |
| TravT | client=self.engine, | 14:50 |
| TravT | index=self.index_name, | 14:50 |
| TravT | doc_type=self.document_type, | 14:50 |
| TravT | chunk_size=self.chunk_size, | 14:50 |
| TravT | actions=actions) | 14:50 |
| TravT | fudge | 14:50 |
| TravT | just throwing random code out to ekarlso, probably does more harm than good | 14:51 |
| ekarlso | :P | 14:51 |
| sjmc7 | :) yeah.. i'm wondering whether we should maybe get the base plugin in without the parent-child relationship. if this is the last issue, then let's try and solve it, but we need a cut-off | 14:51 |
| TravT | that's what i was thinking yesterday... but seemed like everything else is working? | 14:52 |
| ekarlso | well parent / child stuff works ? | 14:52 |
| ekarlso | just deleting the recordsets atm that's bork | 14:52 |
| sjmc7 | notifications have some issues when i tested yesterday evening | 14:52 |
| sjmc7 | yeah, ok. if that's the only hting we can get that done | 14:52 |
| ekarlso | sjmc7: I think that's you having a old version of designate... | 14:52 |
| sjmc7 | :) that might well be, it's a few weeks old | 14:52 |
| TravT | i've got 2 day old devstack, i'll try it all out. | 14:52 |
| TravT | BTW, with designate added to the list of services i'm running, my poor laptop hits a limit. | 14:53 |
| TravT | i'm now having to kill off other services for it to even function | 14:54 |
| sjmc7 | how much RAM does your VM have? | 14:54 |
| TravT | 9 | 14:54 |
| sjmc7 | whoa | 14:54 |
| TravT | but i'm running full ubuntu in it | 14:54 |
| TravT | and do dev in it... | 14:54 |
| sjmc7 | i run a server OS which saves at least 2 gb | 14:54 |
| TravT | not just server | 14:54 |
| sjmc7 | ah | 14:54 |
| sjmc7 | ah right, i forgot your weird setup :) | 14:55 |
| TravT | i like to be able to blow stuff away and screw things up at will | 14:55 |
| TravT | but it is coming at a price | 14:55 |
| sjmc7 | yeah.. i can't imagine running pycharm in a VM | 14:55 |
| TravT | worked fine until recently. | 14:56 |
| TravT | but i swear the services are eating up more and more | 14:56 |
| sjmc7 | all them features! you could turn some off | 14:56 |
| TravT | anwyay, let's see... | 14:57 |
| TravT | do a scan query by zone_id, i believe | 14:57 |
| TravT | then do bulk delete | 14:57 |
| TravT | ok, thought I almost had it ready to try, but time for our meeting | 15:00 |
| TravT | Courtesy Searchlight meeting reminder in #openstack-meeting-4: lakshmiS, nikhil_k, rosmaita, sigmavirus24, TravT, krykowski, david-lyle, wokuma, kragniz, sjmc7, ekarlso, | 15:01 |
| sjmc7 | i filed https://bugs.launchpad.net/searchlight/+bug/1493958 with patch https://review.openstack.org/#/c/221903/ to allow use of parent-child in the designate plugin yesterday. if you're testing designate, please stick a comment on that too (it's a dependency of the designate patch) | 16:00 |
| openstack | Launchpad bug 1493958 in OpenStack Search (Searchlight) "Patch needed to plugin/base to support parent-child relationships" [Undecided,New] - Assigned to Steve McLellan (sjmc7) | 16:00 |
| ekarlso | u guys got any clues on the last blockers or ? | 16:45 |
| TravT | ekarlso, i got pulled into a call, will be out of it shortly | 16:48 |
| TravT | sorry | 16:48 |
| ekarlso | ;P | 16:48 |
| ekarlso | TravT: no worries : p | 16:48 |
| *** TravT_ has joined #openstack-searchlight | 17:07 | |
| *** TravT has quit IRC | 17:08 | |
| *** lakshmiS has quit IRC | 17:11 | |
| *** nikhil_k_ is now known as nikhil_k-bbiab | 17:14 | |
| *** TravT_ has quit IRC | 17:14 | |
| *** TravT has joined #openstack-searchlight | 17:16 | |
| *** nikhil_k-bbiab is now known as nikhil_k | 18:17 | |
| TravT | ekarlso, i've been looking at the delete issue | 19:38 |
| TravT | i have code that should work, but sadly throws an error which i think is due to ES bug. | 19:39 |
| sjmc7 | :O | 19:39 |
| TravT | sjmc7 | 19:40 |
| TravT | : | 19:40 |
| TravT | from elasticsearch import helpers | 19:40 |
| TravT | query = { | 19:40 |
| TravT | 'fields': '_id', | 19:40 |
| sjmc7 | heresy! | 19:40 |
| TravT | 'query': { | 19:40 |
| TravT | 'term': { | 19:40 |
| TravT | 'zone_id': id_ | 19:40 |
| TravT | } | 19:40 |
| TravT | } | 19:40 |
| TravT | } | 19:40 |
| TravT | from recordsets import RecordSetIndex | 19:40 |
| TravT | documents = helpers.scan( | 19:40 |
| TravT | client=self.engine, | 19:40 |
| TravT | index=self.index_name, | 19:40 |
| TravT | doc_type="OS::Designate::RecordSet", | 19:40 |
| TravT | query=query) | 19:40 |
| TravT | actions = [] | 19:40 |
| TravT | for document in documents: | 19:40 |
| TravT | action = { | 19:40 |
| TravT | '_id': document['_id'], | 19:40 |
| TravT | '_op_type': 'delete', | 19:41 |
| TravT | '_index': self.index_name, | 19:41 |
| TravT | '_type': 'OS::Designate::RecordSet' | 19:41 |
| TravT | } | 19:41 |
| TravT | actions.append(action) | 19:41 |
| TravT | if actions: | 19:41 |
| TravT | helpers.bulk( | 19:41 |
| TravT | client=self.engine, | 19:41 |
| TravT | actions=actions) | 19:41 |
| TravT | that actually deletes the recordsets we want | 19:41 |
| TravT | but it is trying to re-index at the end it seems | 19:41 |
| TravT | BulkIndexError: ('2 document(s) failed to index.', [{u'delete': {u'status': 404, u'_type': u'OS::Designate::RecordSet', u'_index': u'searchlight', u'_version': 1, u'found': False, u'_id': u'5dd8ce4c-e9bb-4958-9293-f56156e46012'}}, {u'delete': {u'status': 404, u'_type': u'OS::Designate::RecordSet', u'_index': u'searchlight', u'_version': 1, u'found': False, u'_id': u'84335944-7ac1-4966-8136-1bca74b56e56'}}]) | 19:41 |
| TravT | and can't find them | 19:41 |
| david-lyle | paste.openstack.org is good too :) | 19:41 |
| sjmc7 | :D | 19:41 |
| TravT | i'm too hungry to not be lazy | 19:41 |
| david-lyle | fair enough | 19:42 |
| sjmc7 | it may just be te bulk syntax but that almost looks like it's trying to index documents that look like {'delete': {some stuff}} | 19:42 |
| TravT | yeah, but it is actually deleting the documents | 19:43 |
| TravT | i've been through the ES test code and the code itself for helpers... | 19:43 |
| sjmc7 | maybe just a misleading exceoption name then? | 19:43 |
| TravT | seems right? | 19:43 |
| TravT | https://github.com/elastic/elasticsearch-py/blob/master/elasticsearch/helpers/__init__.py#L69 | 19:43 |
| sjmc7 | hmm.. you MAY need the zone_id, because of the way partent-child works | 19:44 |
| sjmc7 | the _id alone isn't enough for it to know which shard the document lives on | 19:44 |
| TravT | hmm... that's true. | 19:44 |
| TravT | shard wise... | 19:45 |
| sjmc7 | i am guessing a bit though | 19:45 |
| TravT | but again | 19:45 |
| TravT | it is deleting the docs | 19:45 |
| TravT | when i go back and query directly, they are gone | 19:45 |
| sjmc7 | can you print the 'actions' before running the delete? | 19:46 |
| sjmc7 | how many in total are there? | 19:46 |
| TravT | 2 | 19:46 |
| TravT | i've stepped through debugger a couple times on this. | 19:46 |
| TravT | i have to manually re-index each time i create... | 19:47 |
| TravT | the notification listener piece isn't working with the parent child thing | 19:47 |
| TravT | just haven't gotten back to that | 19:47 |
| sjmc7 | yeah. it looks like the exception is ALWAYS BulkIndexError | 19:47 |
| sjmc7 | regardless of operation type | 19:47 |
| TravT | yeah, it is like it is querying back... | 19:48 |
| TravT | not the status is 404 | 19:48 |
| TravT | the typical status when the doc isn't found | 19:48 |
| TravT | s/not/note | 19:49 |
| sjmc7 | yeah... something fishy is definitely up | 19:49 |
| TravT | i think designate might have some other issues | 19:50 |
| TravT | not the plugin, but designate itself | 19:50 |
| TravT | no, nevermind on that. | 19:51 |
| TravT | just realized it is rbac. | 19:51 |
| TravT | and horizon doesn't remove the domain entry from the screen without a refresh after you delete... | 19:53 |
| TravT | sjmc7: just stepping into bulk. | 20:00 |
| TravT | no error | 20:05 |
| TravT | hmmm... | 20:06 |
| TravT | sjmc7: i'm thinking there is a pycharm debug issue | 20:12 |
| TravT | when I run it not through debugger, no errors | 20:12 |
| sjmc7 | :( | 20:12 |
| TravT | everything deleted | 20:12 |
| TravT | bug when pausing at breakpoints to look at the actions, get errors. | 20:13 |
| TravT | maybe a cache issue. | 20:13 |
| TravT | so, fwiw, what i posted above works. | 20:13 |
| TravT | with one issue | 20:13 |
| TravT | i hard coded the recordset document type | 20:13 |
| sjmc7 | ok, well that's sort of good news | 20:13 |
| TravT | how do i get a handle to the RecordSet instance from within the Domain notification handler? | 20:14 |
| sjmc7 | oh, hmm. not easily | 20:14 |
| TravT | i can't statically referrence recordset | 20:14 |
| sjmc7 | the plugins don't really know about each other | 20:14 |
| TravT | get_index and get_document_type require self reference. | 20:15 |
| sjmc7 | right. umm... we may need to register dependencies | 20:15 |
| sjmc7 | when the plugins load? | 20:15 |
| sjmc7 | further down the rabbit hole | 20:15 |
| TravT | well, i think we can live with the doc type string | 20:16 |
| sjmc7 | yeah, for now i think | 20:16 |
| TravT | pretty lame that they are deprecating the delete by query | 20:17 |
| TravT | i'll post these comments back for ekarlso... | 20:17 |
| sjmc7 | yeah... deleting in e-s is not a good thing | 20:17 |
| TravT | now to eat lunch and then see what is missing in the notifications for parent mapping. | 20:18 |
| sjmc7 | yeah, i've not eaten all day eitehr :( | 20:18 |
| TravT | i did drink an energy drink for breakfast... | 20:18 |
| sjmc7 | your five hours are up | 20:18 |
| * TravT goes to look for sustenance | 20:18 | |
| *** TravT_ has joined #openstack-searchlight | 20:23 | |
| *** TravT has quit IRC | 20:25 | |
| sjmc7 | TravT_, you back? or did hunger claim you? | 22:17 |
| TravT_ | i'm back... just running through designate plugin now | 22:17 |
| *** TravT_ is now known as TravT | 22:17 | |
| TravT | I'm not seeing indexing errors on notifications. | 22:17 |
| sjmc7 | ok, endre was right then - older version of designate | 22:17 |
| TravT | and has_parent query seems to work. | 22:17 |
| sjmc7 | i will remove my objection | 22:17 |
| TravT | but | 22:18 |
| TravT | "status": "PENDING", | 22:18 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:18 | |
| sjmc7 | not sure what changes that | 22:18 |
| TravT | first notification has that status | 22:18 |
| TravT | never get a notification that changes it to ACTIVE | 22:19 |
| TravT | if I re-run index sync | 22:19 |
| TravT | it is "status": "ACTIVE", | 22:19 |
| sjmc7 | aww :( | 22:19 |
| sjmc7 | maybe designate just doesn't send one? | 22:19 |
| TravT | so, missing some notification there... | 22:19 |
| TravT | perhaps, I'm leaving that comment now on patch. | 22:19 |
| TravT | was just about to re-test RBAC | 22:19 |
| TravT | on the bright side, at the end of this i will know more about designate | 22:20 |
| TravT | primarily that it is made of zones and recordsets | 22:21 |
| TravT | with strings here and there | 22:21 |
| TravT | :P | 22:21 |
| sjmc7 | :D | 22:22 |
| TravT | on the bright side, has_parent query works regardless of notification based indexing or manual index sync | 22:23 |
| sjmc7 | that's something! | 22:24 |
| TravT | it is! | 22:24 |
| TravT | so, i think that means I can +2 the parent patch. | 22:24 |
| sjmc7 | yeah, it's pretty straightforward | 22:25 |
| TravT | can we test it? | 22:25 |
| sjmc7 | unit tests? yeah. but probably not today | 22:25 |
| TravT | i guess there aren't tests for similar fields. | 22:25 |
| * TravT poor excuse | 22:25 | |
| sjmc7 | we need to improve test coverage all round | 22:26 |
| TravT | probably best to add a functional test | 22:26 |
| TravT | was looking at bulk helpers | 22:26 |
| TravT | probably some test code we can borrow from it | 22:26 |
| TravT | for functional tests | 22:26 |
| TravT | ok, now onto RBAC testing. | 22:27 |
| * TravT crosses fingers | 22:27 | |
| openstackgerrit | Steve McLellan proposed openstack/searchlight: WIP Add faceting https://review.openstack.org/222388 | 22:28 |
| sjmc7 | that patch ^ needs some work but it's close to what we had for the summit prototype | 22:29 |
| TravT | ok, cool, thanks! | 22:29 |
| sjmc7 | don't spend time reviewing it properly now but if you have a change to glance at it. i'll try and add to the docs to show usage in a little bit | 22:30 |
| TravT | i'll probably try to integrate to it today, since like I mentioned earlier, i need it for horizon plugin | 22:30 |
| sjmc7 | ok. i just tested it works, but it'll likely fail pep8 etc | 22:30 |
| TravT | i'm thinking i'll add static facets first in horizon | 22:31 |
| TravT | and then do dynamic... | 22:31 |
| sjmc7 | yeah, that's what kelly did for the prototype | 22:32 |
| TravT | well, admin token in a admin project is seeing demo project zone | 22:34 |
| sjmc7 | that's good! | 22:34 |
| sjmc7 | i think i did test RBAC yesterday and looked ok | 22:34 |
| TravT | demo logged into demo project only sees demo zone | 22:37 |
| sjmc7 | hurrah | 22:37 |
| TravT | no recordsets... | 22:37 |
| sjmc7 | boo | 22:37 |
| TravT | there's no project_it or tenant_id on any of the recordsets | 22:39 |
| sjmc7 | i think i did note that on my review | 22:40 |
| TravT | i'm pretty sure i've noted it before as well. | 22:40 |
| TravT | you know, i think we need to update the docs or change the is_admin handling on query_params | 22:43 |
| TravT | https://github.com/openstack/searchlight/blob/b4bd7665d84712f66e684bf8a897c512251a661b/searchlight/api/v1/search.py#L261 | 22:43 |
| TravT | perhaps add a check for all_tenants or all_projects there | 22:44 |
| TravT | ok, project_id comes through with notifications | 22:48 |
| sjmc7 | hmm | 22:48 |
| TravT | but not index sync | 22:48 |
| TravT | if my mumbling to myself as I walk through it is bothersome, let me know | 22:49 |
| sjmc7 | :) | 22:49 |
| TravT | ok, i think i have the fix. will add comment to review | 22:51 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!