| openstackgerrit | Li Yingjun proposed openstack/python-searchlightclient: Add optional fields for source to query cli https://review.openstack.org/271152 | 01:19 |
|---|---|---|
| *** yingjun has joined #openstack-searchlight | 01:19 | |
| openstackgerrit | Lei Zhang proposed openstack/searchlight: WIP - Zaqar Forwarder plugin https://review.openstack.org/271958 | 09:17 |
| *** yingjun has quit IRC | 09:40 | |
| *** yingjun has joined #openstack-searchlight | 09:40 | |
| *** yingjun has quit IRC | 09:45 | |
| *** yingjun has joined #openstack-searchlight | 11:35 | |
| *** yingjun has quit IRC | 14:16 | |
| *** nikhil_k has quit IRC | 14:33 | |
| *** nikhil has joined #openstack-searchlight | 14:34 | |
| *** yingjun has joined #openstack-searchlight | 14:37 | |
| *** yingjun has quit IRC | 14:38 | |
| *** yingjun has joined #openstack-searchlight | 14:38 | |
| *** yingjun has quit IRC | 14:43 | |
| *** yingjun has joined #openstack-searchlight | 14:43 | |
| *** GB21 has joined #openstack-searchlight | 15:13 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:20 | |
| *** exploreshaifali has joined #openstack-searchlight | 15:35 | |
| *** yingjun has quit IRC | 15:38 | |
| *** bpokorny has joined #openstack-searchlight | 16:50 | |
| *** lakshmiS has joined #openstack-searchlight | 16:55 | |
| sjmc7 | lakshmiS: i’m gonna modify the swift notification middleware to better support your patch. assuming we want to support accounts, containers, objects, we need create/update/delete operations for all of them; i’ll put as many of the metadata fields your patch is looking for in | 17:03 |
| lakshmiS | sjmc7: sounds good | 17:09 |
| *** GB21 has quit IRC | 17:17 | |
| *** TravT has joined #openstack-searchlight | 17:23 | |
| openstackgerrit | Lakshmi N Sampath proposed openstack/searchlight: WIP - Swift plugin Pending RBAC and Code tuning. https://review.openstack.org/271622 | 18:25 |
| *** pcaruana has joined #openstack-searchlight | 18:51 | |
| *** bpokorny has quit IRC | 18:59 | |
| *** openstackgerrit has quit IRC | 19:02 | |
| *** openstackgerrit has joined #openstack-searchlight | 19:03 | |
| openstackgerrit | Rick Aulino proposed openstack/searchlight-specs: Spec for changes to ES document deletion. https://review.openstack.org/269783 | 19:16 |
| openstackgerrit | Travis Tripp proposed openstack/searchlight: Add Devstack install support for searchlightclient https://review.openstack.org/268358 | 19:18 |
| *** bpokorny has joined #openstack-searchlight | 19:18 | |
| *** RickA-HP has joined #openstack-searchlight | 19:19 | |
| RickA-HP | rosmaita: Hi Brian. Thanks for the thoughtful feedback on the ES Deletion blueprint. I had changes and add more concrete examples. I look forward to more comments! | 19:21 |
| openstackgerrit | Rick Aulino proposed openstack/searchlight-specs: Spec for changes to ES document deletion. https://review.openstack.org/269783 | 19:26 |
| *** exploreshaifali has quit IRC | 19:30 | |
| rosmaita | RickA-HP: no problem, i'll take a look soon as i have some free time | 19:30 |
| *** pcaruana has quit IRC | 19:32 | |
| *** pcaruana has joined #openstack-searchlight | 19:36 | |
| *** pcaruana has quit IRC | 19:40 | |
| *** pcaruana has joined #openstack-searchlight | 19:43 | |
| *** pcaruana has quit IRC | 19:56 | |
| openstackgerrit | Merged openstack/searchlight: Add Devstack install support for searchlightclient https://review.openstack.org/268358 | 20:22 |
| sjmc7 | lakshmiS: some of the swift fields aren’t going to be available in middleware, unfortunately | 21:13 |
| sjmc7 | i’m going to see how hard it’d be to send them from within the code instead, like nova etc do | 21:14 |
| lakshmiS | some of them are created by searchlight itself to link the parent | 21:15 |
| sjmc7 | yeah. middleware pretty much only has access to the request and response headers | 21:15 |
| sjmc7 | and you don’t get many headers for PUT/POST operations | 21:16 |
| sjmc7 | fields like bytes used, container-object-count, those sorts of things won’t be available | 21:17 |
| sjmc7 | i’ll see how many i can get hold of | 21:19 |
| sjmc7 | maybe we can ignore the others for now | 21:19 |
| sjmc7 | also, i can’t find any distinction between container-id and container-name (same for accounts) | 21:20 |
| lakshmiS | id and name are same | 21:27 |
| openstackgerrit | Rick Aulino proposed openstack/searchlight-specs: Zero Downtime Reindexing Proposal https://review.openstack.org/245222 | 21:27 |
| lakshmiS | id is searchlight created field to uniquely get the swift url by further rest request | 21:27 |
| sjmc7 | ok, just storing those for searching. and it looks like swift doesn’t support rename, so that’s good | 21:27 |
| sjmc7 | i can’t figure out when/where new accounts are created | 21:28 |
| lakshmiS | thats the tricky part | 21:28 |
| lakshmiS | account is a keystone account, so it has to come from keystone | 21:28 |
| sjmc7 | there’re POST and PUT handlers for them, but i’m not sure where they’re called | 21:28 |
| lakshmiS | swift doesnt have an api to list accounts | 21:28 |
| sjmc7 | right, just get stats for an account you know the name of | 21:29 |
| sjmc7 | i shall experiment | 21:29 |
| sjmc7 | i wonder if it’s necessary to store Accounts at all | 21:29 |
| lakshmiS | admin of an account can add a member to an account to give read/write rights by using POST | 21:29 |
| sjmc7 | do they have any useful information? | 21:29 |
| lakshmiS | we need accounts to get to containers | 21:29 |
| sjmc7 | we don’t need them in e-s though | 21:29 |
| sjmc7 | unless they control rbac | 21:30 |
| lakshmiS | could be useful to get summary | 21:30 |
| lakshmiS | of number of containers and objects on an account | 21:30 |
| sjmc7 | the summary won’t ever get updated | 21:30 |
| lakshmiS | no api for that unless we keep it updated on container/object change | 21:30 |
| sjmc7 | urgh | 21:31 |
| sjmc7 | e-s lets you run sums on child documents | 21:31 |
| lakshmiS | that could also be useful if we dont have to update any other fields on account | 21:32 |
| notmyname | FYI in swift there is currently work going on to enable an operator to get a list of all accounts found in the cluster. it's a current outreachy project | 21:32 |
| lakshmiS | notmyname: it would have saved a lot of time for me :) | 21:33 |
| sjmc7 | notmyname: at current, is account creation implicit when you first create a container? | 21:33 |
| lakshmiS | based on policy yes | 21:33 |
| lakshmiS | proxy-server.conf account_autocreate = True | 21:34 |
| notmyname | right. that. by default, authorized requests will lazily create an account if it doesn't exist | 21:36 |
| sjmc7 | ok. it’s not clear to me why we’d want to maintain account information in elasticsearch | 21:37 |
| notmyname | what account information? just that it exists? or metadata on it? | 21:37 |
| sjmc7 | metadata | 21:38 |
| lakshmiS | summayr on container count and metadata | 21:38 |
| sjmc7 | or that it exists | 21:38 |
| lakshmiS | i think it doesnt harm having it in es. | 21:38 |
| notmyname | how so? | 21:38 |
| sjmc7 | it is much harder to update | 21:38 |
| notmyname | users can set arbitrary metadata on their account, too. | 21:39 |
| sjmc7 | ah, ok. is that kind of thing useful in a searching scenario? | 21:39 |
| sjmc7 | i am a little blind, so i appreciate the help :) trying to figure out how much we need to do to reliably keep things up to date | 21:39 |
| notmyname | what do you mean by up to date? | 21:40 |
| sjmc7 | if ew’re storing account metadata in elasticsearch | 21:40 |
| lakshmiS | and can have objects on account too i guess(x-account-object-count) | 21:40 |
| sjmc7 | we need to make sure we know when it changes. i was hoping to do that with middleware, but there’s a limit to how much information is available in the request/response headers | 21:41 |
| notmyname | multi-tenancy in swift is done at the swift account level. in many cases, the user will only have access to their own account, so it doesn't make sense from a searching perspective (eg "show me the accounts where foo=bar" will be their own account or no results b/c/ permissions) | 21:41 |
| sjmc7 | right. and even for an admin, searching over user-supplied metadata maybe isn’t very useful because it won’t mean anything | 21:42 |
| notmyname | but other deployment models have an account per "other thing" (ie not a user). so you might have an account per department or project or something. in that case, yes a user would have access to more than one account and searching makes sense | 21:42 |
| lakshmiS | notmyname: what happesn if user has reselleradmin role. can he not search across accounts? | 21:42 |
| sjmc7 | i guess the question is: what is a user searching for | 21:42 |
| sjmc7 | object metadata yes, definitely | 21:42 |
| sjmc7 | container metadata… maybe | 21:43 |
| notmyname | lakshmiS: yes, that would work. but reselleradmin should be used like root. ie not an active thing :-) | 21:43 |
| sjmc7 | account metadata, not so sure | 21:43 |
| lakshmiS | notmyname: do users normally store metadata on account in real world | 21:43 |
| notmyname | I don't have data on that. it's a part of the API so I must assume "yes" | 21:44 |
| sjmc7 | ok. then i guess we should assume yes | 21:44 |
| sjmc7 | i do think we should drop any derived fields, like object counts | 21:45 |
| sjmc7 | otherwise every object change will require a load of changes to accounts and containers | 21:45 |
| lakshmiS | object count is part of account info swift REST api | 21:45 |
| sjmc7 | i know | 21:45 |
| sjmc7 | when you upload an object, what information would you expect in the notification? | 21:46 |
| notmyname | IMO, indexing account metadata is a silly question. as in, that's not even near the top of things to worry about. in large clusters, you'll have order of 10k to 100k accounts. and 3 or 4 (or 5!) orders of magnitude more containers possible. and another 4-6 orders of magnitude more objects than accounts | 21:46 |
| sjmc7 | notmyname: yeah, that’s why i’d prefer to focus on objects/containers | 21:46 |
| notmyname | eg 100k accounts, the largest of which have 100k containers, the largest of which have 1000k+ objects | 21:47 |
| sjmc7 | i was once one of those users :) | 21:47 |
| notmyname | someone will ask for searching account metadata. eg "show me accounts with more than 1M objects in storage policy Foo but less thank 1PB in the same policy" | 21:48 |
| notmyname | even if it's "just" an operator, that type of query will be very very useful | 21:49 |
| sjmc7 | yep - but i don’t think we need to incur the overhead of storing those counts for every object request | 21:49 |
| sjmc7 | versus aggregating at query time | 21:49 |
| notmyname | you've lost me. I don't know enough about your system to know what that means | 21:49 |
| lakshmiS | for that kind of object count it can only be periodical if we need it | 21:49 |
| notmyname | right. and it's only updated on the account asynchronously | 21:50 |
| lakshmiS | so swift get account info looks at cached object count? | 21:50 |
| notmyname | "cached". what a tricky work ;-) | 21:50 |
| sjmc7 | :) | 21:50 |
| notmyname | so the object count is stored on disk in the account. and it's also cached in memcache (with something like a 30s ttl) | 21:51 |
| notmyname | but it's not a up-to-the-moment 100% accurate count. never has been, never will | 21:51 |
| notmyname | so you could poll the accounts. that's ugly. | 21:52 |
| notmyname | or... | 21:52 |
| notmyname | as part of the "list all the accounts" thing that's being worked on, we're developing a way you can hook into the auditors that walk the disk | 21:52 |
| notmyname | ie we're crawling the disk to see what's there. we check for bit rot. we're adding the functionality to call an external method when we find something | 21:53 |
| lakshmiS | so there "will be" a call back hook to notify on account update | 21:53 |
| notmyname | object version is https://review.openstack.org/#/c/212824/ with an example at https://github.com/smerritt/basic-audit-watcher | 21:54 |
| notmyname | and here's the account version https://review.openstack.org/#/c/268830/ | 21:54 |
| notmyname | note that neither have landed yet. but I expect them to | 21:54 |
| lakshmiS | sjmc7: if you want to wait until we can recieve account updates, I can disable the account indexing for now | 21:56 |
| sjmc7 | no, we can get updates on metadata POSTs etc | 21:56 |
| sjmc7 | not so sure about the aggregation data (object counts, byte sums) | 21:56 |
| lakshmiS | right | 21:56 |
| lakshmiS | if we index it will out of sync soon without them(kind of bad to have than disable it) | 21:57 |
| TravT | sorry for missing this, but sjmc7, are you saying that we would still be able to do things like get object and container counts per account even if we don't index the account by aggregating? | 22:01 |
| sjmc7 | in a fashion, yeah. there’s no direct equivalent of SQL’s ‘HAVING’ clause, but there are various aggregation functions | 22:02 |
| sjmc7 | it wouldn’t be as straightforward as a stored number | 22:02 |
| sjmc7 | i guess we could make it optional until we know what impact it’ll have trying to store it | 22:03 |
| TravT | well, if i'm reading this correctly, if we receive notifications for all these counts on the account level, we'll be getting double notifications right? | 22:04 |
| TravT | once when the object change notification comes in and once on the account level. Or am i missing something? | 22:05 |
| TravT | i guess i need to read the patch lakshmi put up late Friday night | 22:05 |
| lakshmiS | they will be different notifications. object count on container and account | 22:06 |
| lakshmiS | ideally it should be CRUD notifications each for account/container/object | 22:07 |
| TravT | lakshmiS sjmc7: are you on the current prototype discussions for the new swift panel in horizon? | 22:07 |
| TravT | i can add you if you aren't | 22:08 |
| lakshmiS | yes i am on | 22:08 |
| TravT | ok, good | 22:08 |
| lakshmiS | i have a hard stop in few mins. will be back later | 22:10 |
| lakshmiS | notmyname: off the topic question. do you know if swift will have support for keystone "session" this release? | 22:11 |
| lakshmiS | i meant swiftclient | 22:12 |
| *** RickA-HP has quit IRC | 22:12 | |
| notmyname | passing in a keystone session and using that as an auth context? | 22:14 |
| lakshmiS | yes | 22:14 |
| notmyname | certainly been talked about, and desired. https://review.openstack.org/#/c/270045/ seems to be a start | 22:14 |
| lakshmiS | yeah sjmc7 mentiond about it, but wasnt sure how serious is the change to be landed in near future | 22:15 |
| lakshmiS | anyway thanks for the link. will keep a watch | 22:17 |
| *** lakshmiS has quit IRC | 22:19 | |
| *** lakshmiS has joined #openstack-searchlight | 23:05 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:07 | |
| *** david-lyle has quit IRC | 23:44 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!