14:00:33 <pdeore> #startmeeting glance 14:00:33 <opendevmeet> Meeting started Thu May 19 14:00:33 2022 UTC and is due to finish in 60 minutes. The chair is pdeore. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:33 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:33 <opendevmeet> The meeting name has been set to 'glance' 14:00:33 <pdeore> #topic roll call 14:00:34 <pdeore> #link https://etherpad.openstack.org/p/glance-team-meeting-agenda 14:00:43 <pdeore> o/ 14:00:47 <dansmith> o/ 14:01:23 <mrjoshi> o/ 14:01:35 <pdeore> lets wait few minutes for everyone to join 14:01:35 <jokke_> o/ 14:02:46 <pdeore> abhishekk, and rosmaita are busy in resolving the broken gate issue 14:03:03 <rosmaita> well, "resolving" is a bit strong 14:03:13 <rosmaita> more like trying to figure out WTF 14:03:52 <abhishekk> yeah :/ 14:04:07 <pdeore> yeah, of course.. 14:04:09 <abhishekk> unfortunately last couple of days we are hitting with very strange errors 14:04:55 <abhishekk> if anyone interested to have a look, this is the patch which reported the error, https://review.opendev.org/c/openstack/glance/+/842400 14:05:14 <pdeore> ack 14:05:26 <pdeore> can we start ? 14:05:34 <croelandt> yes! 14:05:38 <pdeore> #topic release/periodic jobs 14:05:57 <pdeore> It's M1 Release Week and we have released glance-store 4.0.0 yesterday 14:06:08 <pdeore> #link https://review.opendev.org/c/openstack/glance/+/842400 is causing failure in glance gate 14:06:25 <pdeore> so, we might skip glance M1 release if gate is not fixed today 14:07:53 <abhishekk> correction: above patch is not causing failure, above patch has reported the failure 14:08:28 <pdeore> ack 14:09:11 <pdeore> Periodic jobs all green except some oslo tips jobs failure for python3.6 14:09:19 <pdeore> #link https://review.opendev.org/c/openstack/glance/+/841350 14:09:19 <pdeore> #link https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/841368 14:09:30 <pdeore> Still waiting to get these patches merged 14:10:03 <pdeore> I have requested the infra people to look after the openstack-zuul-jobs patch as I'm not much familiar with those job changes and the comments received on the patch 14:10:45 <pdeore> moving ahead 14:10:50 <pdeore> #topic cache-update response code 14:11:32 <pdeore> #link https://review.opendev.org/c/openstack/glance/+/841278/1#message-456096e48b28e5b866deb8bf53e9258ee08219a0 14:11:39 <abhishekk> jokke_, there are some suggestions on the above backport, so can you reconsider this vote 14:12:41 <jokke_> abhishekk: I noticed 14:13:07 <abhishekk> ack 14:13:22 <jokke_> rosmaita: just a point to what you said, the spec you linked indeed called the 202, the _documentation_ is changed to that with the patch and stated 200 14:13:40 <rosmaita> yeah, but the original doc, the spec, said 202 14:13:56 <rosmaita> which is documentation 14:14:24 <rosmaita> and actually is supposed to be the design doc 14:16:35 <jokke_> yes I totally agree that there's been oversight on implementation and reviews. Just a note that we've been blocking these fixes even on master in past ... Like if we want to break the stable that discussion needs to happen in the mailing list 14:16:57 <jokke_> it's not something we should just go and do no matter how convenient it is for us 14:19:12 <dansmith> I have no problem expressing my support for this route to resolution on the mailing list, if it will help 14:19:38 <dansmith> not sure it's really necessary, but if it helps to just have the same conversation we've had on the ML that's fine with me, but I think we should try to do it quick 14:19:51 <dansmith> as the longer the wait the worse it is 14:19:55 <abhishekk> +1 14:21:31 <rosmaita> exactly 14:21:38 <jokke_> so if the stable maint team greenlights that api change, we also should bump the api version and add reno for it as I mentioned in my original review. just silently changing it is really bad practise 14:23:51 <rosmaita> you mean bump in yoga? 14:24:07 <dansmith> I thought it was just bump in master, 14:24:12 <jokke_> well bump in master and squash them together for backporting 14:24:15 <dansmith> but still backport to yoga 14:24:28 <dansmith> so bump the api in yoga too? 14:24:49 <jokke_> well if we backport the change, then yes 14:25:04 <dansmith> if we were making some real change that people were likely to struggle with I'd agree with much more caution, but I think we can be rational about this specific change and need not overreact, IMHO 14:25:15 <jokke_> the api version glance is reporting is there to indicate changes in the api 14:26:40 <jokke_> dansmith: I just think it's quite slippery slope to not treat every API breaking change at same severity 14:27:31 <dansmith> it is a slippery slope, which is why we should be cautious and critical about exceptions.. however, this seems like a good candidate for an exception to me 14:28:22 <jokke_> agreed, and there is documented process for it ;) 14:28:54 <dansmith> rosmaita: you wanna send an email to the list detailing what we discussed, ask for comment and we can go from there? 14:29:06 <rosmaita> i can do that 14:29:49 <jokke_> cheers 14:30:37 <pdeore> can we move ahead to next topic? 14:30:43 <jokke_> ++ 14:30:47 <pdeore> #topic new location APIs 14:30:55 <pdeore> #link https://review.opendev.org/c/openstack/glance-specs/+/840882 14:31:09 <pdeore> the spec is submitted by whoami-rajat 14:31:09 <whoami-rajat> hey 14:31:34 <whoami-rajat> so we discussed this idea at the PTG at glance cinder cross project session 14:31:39 * jokke_ is piling up for today's meeting ;) 14:31:52 <whoami-rajat> and as discussed i proposed a spec implementing the same idea 14:32:04 <whoami-rajat> but there are some concerns by jokke_ , which I've replied to 14:32:15 <whoami-rajat> just wanted to bring some attention to it 14:32:31 <whoami-rajat> I'm happy to continue the discussion either here or review but just want to get things moving 14:33:38 <jokke_> I expressed the concern for admin scoped api for this purpose already during the PTG discussion and after reading through the proposal I think it's very bad idea. 14:34:44 <rosmaita> can you explain that some more? 14:34:50 <jokke_> dansmith: I don't know if you have read through the spec yet. How do you feel about getting Nova storing adming credentials in configs and moving to use the new API for the ceph stuff? Will it ever happen? 14:34:51 <whoami-rajat> we need this API for both service-service interaction and also for operators, I'm not sure what a good alternative is 14:35:20 <dansmith> jokke_: I haven't read this spec, but will after the meeting 14:35:47 <whoami-rajat> we've kept the credentials for nova cinder interaction since forever and it doesn't have any side effects, as far as I've seen 14:35:56 <dansmith> sounds like this might be a thing for the service role, but yeah currently nova has no special creds for glance and obviously it'd be a change to start doing that 14:36:07 <whoami-rajat> and the credentials can be removed once service role is in place 14:36:29 <dansmith> right, service role would be ideal for this sort of thing 14:36:49 <dansmith> (just based on a quick skim) 14:36:53 <whoami-rajat> but again, I'm not sure what the timeline of that being available is, I already have a feature dependent on it stuck since yoga ... 14:37:03 <whoami-rajat> dependent on the service role 14:37:58 <dansmith> I'm not sure either as we just had some keystone discussion about it in the last week 14:38:32 <jokke_> rosmaita: my main concerns are storing the admin credentials all over the place as we very well know OS has no limitation for them. We're expecting to use them for API that's main purpose is to go around gapi on image data handling and thus we also loose any authorization in glance side for the location operations 14:38:40 <dansmith> adding admin credentials in the meantime would definitely be a big (and unfortunate) hammer, but I really haven't read the spec so ... 14:39:01 <whoami-rajat> that's my concern, we can't halt all work for keystone to make it available and can try to adapt current alternatives until then, but that's just my thinking 14:39:38 <jokke_> And we're aiming this thing to replace the current locations calls, meaning that every change on them needs to be coordinated between Nova (Never easy to get changes in), Cinder and Glance 14:39:46 <rosmaita> well, we could implement it for a 'service' role, and if people want to use it, they'll have to add it themselves 14:40:08 <dansmith> yeah, we really don't have to wait for keystone for service role stuff I think, 14:40:21 <dansmith> especially for specific configs like shared ceph 14:40:46 <rosmaita> so the locations API will be admin or service 14:40:53 <rosmaita> because actual admins will want to use it 14:40:59 <jokke_> Well the service role would still mean that we need to have some level of double authentication from Keystone side 14:41:18 <rosmaita> why double auth? 14:41:22 <whoami-rajat> jokke_, we will be keeping compatibility until all consumers have moved to new APIs, but the current code is not easy to follow and understand as of now 14:41:36 <dansmith> it means nova has a separate set of credentials for using that api, yes, but it does for pretty much every other downstream service already 14:42:30 <jokke_> rosmaita: isn't the whole idea that there is 2 tokens (or double purpose token) issued which effectively states "Service X is doing this operation on behalf of user Y" 14:42:38 <dansmith> no 14:43:09 <whoami-rajat> we've custom commands for locations in glanceclient which calls image-update and we've bunch of nested checks in image update, one of such check is ``show_multiple_locations`` 14:43:28 <dansmith> for things like this (again, haven't read spec, but interpolating) nova would use the user's token for things on behalf of the user, but then use its own creds to peek behind the curtain for things like locations 14:43:59 <dansmith> glance wouldn't change other than its policy, and nova would just use a differnent ksclient to make the location calls, 14:44:06 <dansmith> like we do for neutron port bindings and such 14:44:16 <jokke_> dansmith: that's unfortunate 14:44:29 <whoami-rajat> yes, similar to what cinder does to call nova for certain operations 14:44:35 <dansmith> jokke_: unfortunate that this is how it works for every other service? 14:44:57 <dansmith> (genuine question) 14:45:07 <jokke_> dansmith: yes 14:45:28 <dansmith> the unfortunate thing is that right now those are all all-powerful admin users, which is very very unfortunate 14:45:40 <whoami-rajat> just an example of code implementation, priviledged_user is the key parameter here https://github.com/openstack/cinder/blob/master/cinder/compute/nova.py#L84 14:45:42 <jokke_> cause if the intention is not to expose the originating user, it still breaks the audit chain 14:45:53 <dansmith> with the service role being the thing endowed to do specific things, I don't think it's that unfortunate and I'm not sure what the realistic alternative is, based on how our stuff works 14:46:04 <jokke_> and removes the authorisation from the owning service 14:47:01 <dansmith> yeah, which is the purpose of the dual-token thing I think you were referring to earlier (although pejoratively, I thought), but AFAIK, that's not how we do most things today 14:47:49 <jokke_> Cause I think the locations here is pretty darn good example of why you would want the double authentication. You want to confirm that user is not poking the internals on their own, but you're after all changing the project owned resource and want to ensure who ever is requesting that having the rights to do so 14:48:03 <dansmith> anyway, apologies for not having read the spec, but perhaps we can move on and I'll comment on it after this and maybe we can circle back? 14:48:40 <whoami-rajat> I'm ok with it unless the spec gets stuck ... 14:48:46 <whoami-rajat> but i can always bring back the topic next week 14:48:56 <dansmith> jokke_: I dunno, if that api is restricted to admins and services, the audit chain doesn't seem terribly concerning to me.. audit chains are always nice, but nova is really the thing servicing the user and using glance's locations api to do things 14:49:22 <abhishekk> whoami-rajat, also I have concern related to spec 14:49:33 <jokke_> I think this might warrant wider discussion as well and sorry dansmith for throwing you right in. I just thought you might have better insight of how posible it would be to even get nova adobting this 14:49:38 <abhishekk> for new location APIs you are going to add new policies,right? 14:49:50 <whoami-rajat> yes 14:50:15 <dansmith> jokke_: I don't think nova adoption will be a problem because the spec for that will be "do for glance as we do for everything else" but still worth careful consideration, no argument there 14:50:15 <abhishekk> ok, I will also comment my thoughts on the spec 14:50:57 <dansmith> jokke_: I really like that nova->glance is all user-token at the moment :) 14:51:08 <jokke_> dansmith: agreed 14:51:24 <rosmaita> well, that does prompt a question from me 14:51:32 <rosmaita> is OSSN-0065 still an issue? 14:51:45 <jokke_> and the easy way to not expose it to the und users is to deploy service facing gapi and user facing gapi with separate configs and policies 14:51:53 * abhishekk apologies for me being not so active in today's meeting 14:52:06 <whoami-rajat> sure, but if your concern is regarding the policies can be modified for non-admins, the catch here is that these policies won't be shared unlike the current ones which are shared at other places hence the default is allowing members 14:52:23 <whoami-rajat> but we can continue discussion in review, don't want to take up all meeting time 14:52:43 <rosmaita> my question is whether in these modern times, is it still a security risk to expose locations? 14:53:03 <whoami-rajat> rosmaita, as discussed during the PTG, yes and it will be when we starting adding more features that allows more glance cinder cross project dependencies 14:53:14 <whoami-rajat> like rbd cow cloning when uploading volume to image 14:53:38 <jokke_> I just think requiring admin crdentials saved somewhere to be used for API that is by far mostly used for service-service interactions and expecting the consuming service to handle the authorisation what the uer should or shoul not be able to do is very very bad idea 14:53:46 <dansmith> rosmaita: well as jokke_ noted, you currently should be only exposing them to users via your internal endpoint, 14:53:58 <whoami-rajat> but my answer is based on glance team's points ^ 14:54:02 <dansmith> so unless you've done your networking poorly... 14:54:15 <dansmith> it's still "host-based auth" which is not great either, but.. 14:55:57 <pdeore> I think we can continue this discussion in review, as just 5 mins left 14:56:15 <jokke_> And I have hard time understanding how this would fit to the RBAC model, like what scope it should be 14:56:23 <jokke_> pdeore: ++ ... thanks 14:56:26 <dansmith> project 14:56:27 <whoami-rajat> sure 14:56:39 <abhishekk> Just an update, next week I will not be able to join the meeting as I have conflict which I can not avoid 14:57:00 <pdeore> Thanks, let's move to open discussion 14:57:01 <pdeore> #topic Open Discussion 14:57:01 <abhishekk> also pdeore will be going on vacation and will be back after next week 14:57:04 <jokke_> dansmith: but the back-end details are for sure system matter, not project matter and if it's project admin, it's not solving 0065 14:57:26 <abhishekk> so is there any volunteer to chair the meeting or we can skip it for next week? 14:57:28 <pdeore> yeah, I will also not be around for entire week 14:57:51 <croelandt> It's a holiday next Thursday here 14:58:00 <dansmith> maybe just punt then? 14:58:03 <jokke_> if both of you are away, should we postpone at least the locations discussion for the following week ... I'd say just cancel the meeting for next week all togeher 14:58:26 <abhishekk> sounds good to me, meantime we can discuss on the spec as well 14:58:29 <rosmaita> cancel works for me 14:58:41 <whoami-rajat> i think this is sort of the same problem when a volume is a project level resource but host attribute is a system level thing -- image (project level) , locations (system level) 14:58:52 <jokke_> if 3 of ye are away I'm sure we can coordinate any possibly urgent stuff in normal glance channel between myself dansmith and rosmaita 14:58:56 <dansmith> whoami-rajat: agree 14:59:09 <abhishekk> cool 14:59:41 <pdeore> great! 15:00:15 <abhishekk> pdeore, plz send the mail to ML saying next weeks meeting is canceled and we will be meeting a week after 15:00:23 <jokke_> as all of us are off tomorrow, I'll keep my eyes on ML next week for the backporting matter 15:00:32 <abhishekk> we can conclude now, thank you all 15:00:35 <pdeore> abhishekk, sure, I will do that 15:00:38 <whoami-rajat> thanks for the discussion everyone! 15:00:40 <jokke_> if agreed with stable maint team, will lift my -2 then 15:00:49 <pdeore> Thanks everyone for joining !! 15:00:54 <jokke_> thanks all! 15:01:05 <pdeore> #endmeeting