Thursday, 2022-05-19

*** akekane_ is now known as abhishekk06:46
*** dasm|off is now known as dasm13:21
pdeore#startmeeting glance14:00
opendevmeetMeeting started Thu May 19 14:00:33 2022 UTC and is due to finish in 60 minutes.  The chair is pdeore. Information about MeetBot at
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'glance'14:00
pdeore#topic roll call14:00
pdeorelets wait few minutes for everyone to join14:01
pdeoreabhishekk, and rosmaita are busy in resolving the broken gate issue14:02
rosmaitawell, "resolving" is a bit strong14:03
rosmaitamore like trying to figure out WTF14:03
abhishekkyeah :/14:03
pdeoreyeah, of course.. 14:04
abhishekkunfortunately last couple of days we are hitting with very strange errors14:04
abhishekkif anyone interested to have a look, this is the patch which reported the error,
pdeorecan we start ?14:05
pdeore#topic release/periodic jobs14:05
pdeoreIt's  M1 Release Week and we have released glance-store 4.0.0 yesterday14:05
pdeore#link is causing failure in glance gate14:06
pdeoreso, we might skip glance M1 release if gate is not fixed today14:06
abhishekkcorrection: above patch is not causing failure, above patch has reported the failure14:07
pdeorePeriodic jobs all green except some oslo tips jobs failure for python3.6 14:09
pdeoreStill waiting to get these patches merged14:09
pdeoreI have requested the infra people to look after the openstack-zuul-jobs patch as I'm not much familiar with those job changes and the comments received on the patch14:10
pdeoremoving ahead14:10
pdeore#topic cache-update response code14:10
abhishekkjokke_, there are some suggestions on the above backport, so can you reconsider this vote14:11
jokke_abhishekk: I noticed14:12
jokke_rosmaita: just a point to what you said, the spec you linked indeed called the 202, the _documentation_ is changed to that with the patch and stated 20014:13
rosmaitayeah, but the original doc, the spec, said 20214:13
rosmaitawhich is documentation14:13
rosmaitaand actually is supposed to be the design doc14:14
jokke_yes I totally agree that there's been oversight on implementation and reviews. Just a note that we've been blocking these fixes even on master in past ... Like if we want to break the stable that discussion needs to happen in the mailing list 14:16
jokke_it's not something we should just go and do no matter how convenient it is for us14:16
dansmithI have no problem expressing my support for this route to resolution on the mailing list, if it will help14:19
dansmithnot sure it's really necessary, but if it helps to just have the same conversation we've had on the ML that's fine with me, but I think we should try to do it quick14:19
dansmithas the longer the wait the worse it is14:19
jokke_so if the stable maint team greenlights that api change, we also should bump the api version and add reno for it as I mentioned in my original review. just silently changing it is really bad practise14:21
rosmaitayou mean bump in yoga?14:23
dansmithI thought it was just bump in master,14:24
jokke_well bump in master and squash them together for backporting14:24
dansmithbut still backport to yoga14:24
dansmithso bump the api in yoga too?14:24
jokke_well if we backport the change, then yes14:24
dansmithif we were making some real change that people were likely to struggle with I'd agree with much more caution, but I think we can be rational about this specific change and need not overreact, IMHO14:25
jokke_the api version glance is reporting is there to indicate changes in the api14:25
jokke_dansmith: I just think it's quite slippery slope to not treat every API breaking change at same severity14:26
dansmithit is a slippery slope, which is why we should be cautious and critical about exceptions.. however, this seems like a good candidate for an exception to me14:27
jokke_agreed, and there is documented process for it ;)14:28
dansmithrosmaita: you wanna send an email to the list detailing what we discussed, ask for comment and we can go from there?14:28
rosmaitai can do that14:29
pdeorecan we move ahead to next topic?14:30
pdeore#topic new location APIs 14:30
pdeorethe spec is submitted by whoami-rajat 14:31
whoami-rajatso we discussed this idea at the PTG at glance cinder cross project session14:31
* jokke_ is piling up for today's meeting ;)14:31
whoami-rajatand as discussed i proposed a spec implementing the same idea14:31
whoami-rajatbut there are some concerns by jokke_ , which I've replied to14:32
whoami-rajatjust wanted to bring some attention to it14:32
whoami-rajatI'm happy to continue the discussion either here or review but just want to get things moving14:32
jokke_I expressed the concern for admin scoped api for this purpose already during the PTG discussion and after reading through the proposal I think it's very bad idea. 14:33
rosmaitacan you explain that some more? 14:34
jokke_dansmith: I don't know if you have read through the spec yet. How do you feel about getting Nova storing adming credentials in configs and moving to use the new API for the ceph stuff? Will it ever happen?14:34
whoami-rajatwe need this API for both service-service interaction and also for operators, I'm not sure what a good alternative is14:34
dansmithjokke_: I haven't read this spec, but will after the meeting14:35
whoami-rajatwe've kept the credentials for nova cinder interaction since forever and it doesn't have any side effects, as far as I've seen14:35
dansmithsounds like this might be a thing for the service role, but yeah currently nova has no special creds for glance and obviously it'd be a change to start doing that14:35
whoami-rajatand the credentials can be removed once service role is in place14:36
dansmithright, service role would be ideal for this sort of thing14:36
dansmith(just based on a quick skim)14:36
whoami-rajatbut again, I'm not sure what the timeline of that being available is, I already have a feature dependent on it stuck since yoga ...14:36
whoami-rajatdependent on the service role14:37
dansmithI'm not sure either as we just had some keystone discussion about it in the last week14:37
jokke_rosmaita: my main concerns are storing the admin credentials all over the place as we very well know OS has no limitation for them. We're expecting to use them for API that's main purpose is to go around gapi on image data handling and thus we also loose any authorization in glance side for the location operations14:38
dansmithadding admin credentials in the meantime would definitely be a big (and unfortunate) hammer, but I really haven't read the spec so ...14:38
whoami-rajatthat's my concern, we can't halt all work for keystone to make it available and can try to adapt current alternatives until then, but that's just my thinking14:39
jokke_And we're aiming this thing to replace the current locations calls, meaning that every change on them needs to be coordinated between Nova (Never easy to get changes in), Cinder and Glance14:39
rosmaitawell, we could implement it for a 'service' role, and if people want to use it, they'll have to add it themselves14:39
dansmithyeah, we really don't have to wait for keystone for service role stuff I think, 14:40
dansmithespecially for specific configs like shared ceph14:40
rosmaitaso the locations API will be admin or service14:40
rosmaitabecause actual admins will want to use it14:40
jokke_Well the service role would still mean that we need to have some level of double authentication from Keystone side14:40
rosmaitawhy double auth?14:41
whoami-rajatjokke_, we will be keeping compatibility until all consumers have moved to new APIs, but the current code is not easy to follow and understand as of now14:41
dansmithit means nova has a separate set of credentials for using that api, yes, but it does for pretty much every other downstream service already14:41
jokke_rosmaita: isn't the whole idea that there is 2 tokens (or double purpose token) issued which effectively states "Service X is doing this operation on behalf of user Y"14:42
whoami-rajatwe've custom commands for locations in glanceclient which calls image-update and we've bunch of nested checks in image update, one of such check is ``show_multiple_locations``14:43
dansmithfor things like this (again, haven't read spec, but interpolating) nova would use the user's token for things on behalf of the user, but then use its own creds to peek behind the curtain for things like locations14:43
dansmithglance wouldn't change other than its policy, and nova would just use a differnent ksclient to make the location calls,14:43
dansmithlike we do for neutron port bindings and such14:44
jokke_dansmith: that's unfortunate14:44
whoami-rajatyes, similar to what cinder does to call nova for certain operations14:44
dansmithjokke_: unfortunate that this is how it works for every other service?14:44
dansmith(genuine question)14:44
jokke_dansmith: yes14:45
dansmiththe unfortunate thing is that right now those are all all-powerful admin users, which is very very unfortunate14:45
whoami-rajatjust an example of code implementation, priviledged_user is the key parameter here
jokke_cause if the intention is not to expose the originating user, it still breaks the audit chain14:45
dansmithwith the service role being the thing endowed to do specific things, I don't think it's that unfortunate and I'm not sure what the realistic alternative is, based on how our stuff works14:45
jokke_and removes the authorisation from the owning service14:46
dansmithyeah, which is the purpose of the dual-token thing I think you were referring to earlier (although pejoratively, I thought), but AFAIK, that's not how we do most things today14:47
jokke_Cause I think the locations here is pretty darn good example of why you would want the double authentication. You want to confirm that user is not poking the internals on their own, but you're after all changing the project owned resource and want to ensure who ever is requesting that having the rights to do so14:47
dansmithanyway, apologies for not having read the spec, but perhaps we can move on and I'll comment on it after this and maybe we can circle back?14:48
whoami-rajatI'm ok with it unless the spec gets stuck ...14:48
whoami-rajatbut i can always bring back the topic next week14:48
dansmithjokke_: I dunno, if that api is restricted to admins and services, the audit chain doesn't seem terribly concerning to me.. audit chains are always nice, but nova is really the thing servicing the user and using glance's locations api to do things14:48
abhishekkwhoami-rajat, also I have concern related to spec14:49
jokke_I think this might warrant wider discussion as well and sorry dansmith for throwing you right in. I just thought you might have better insight of how posible it would be to even get nova adobting this14:49
abhishekkfor new location APIs you are going to add new policies,right?14:49
dansmithjokke_: I don't think nova adoption will be a problem because the spec for that will be "do for glance as we do for everything else" but still worth careful consideration, no argument there14:50
abhishekkok, I will also comment my thoughts on the spec14:50
dansmithjokke_: I really like that nova->glance is all user-token at the moment :)14:50
jokke_dansmith: agreed14:51
rosmaitawell, that does prompt a question from me14:51
rosmaitais OSSN-0065 still an issue?14:51
jokke_and the easy way to not expose it to the und users is to deploy service facing gapi and user facing gapi with separate configs and policies14:51
* abhishekk apologies for me being not so active in today's meeting14:51
whoami-rajatsure, but if your concern is regarding the policies can be modified for non-admins, the catch here is that these policies won't be shared unlike the current ones which are shared at other places hence the default is allowing members14:52
whoami-rajatbut we can continue discussion in review, don't want to take up all meeting time14:52
rosmaitamy question is whether in these modern times, is it still a security risk to expose locations?14:52
whoami-rajatrosmaita, as discussed during the PTG, yes and it will be when we starting adding more features that allows more glance cinder cross project dependencies14:53
whoami-rajatlike rbd cow cloning when uploading volume to image14:53
jokke_I just think requiring admin crdentials saved somewhere to be used for API that is by far mostly used for service-service interactions and expecting the consuming service to handle the authorisation what the uer should or shoul not be able to do is very very bad idea14:53
dansmithrosmaita: well as jokke_ noted, you currently should be only exposing them to users via your internal endpoint,14:53
whoami-rajatbut my answer is based on glance team's points ^14:53
dansmithso unless you've done your networking poorly...14:54
dansmithit's still "host-based auth" which is not great either, but..14:54
pdeoreI think we can continue this discussion in review, as just 5 mins left 14:55
jokke_And I have hard time understanding how this would fit to the RBAC model, like what scope it should be14:56
jokke_pdeore: ++ ... thanks14:56
abhishekkJust an update, next week I will not be able to join the meeting as I have conflict which I can not avoid14:56
pdeoreThanks, let's move to open discussion14:57
pdeore#topic Open Discussion14:57
abhishekkalso pdeore will be going on vacation and will be back after next week14:57
jokke_dansmith: but the back-end details are for sure system matter, not project matter and if it's project admin, it's not solving 006514:57
abhishekkso is there any volunteer to chair the meeting or we can skip it for next week?14:57
pdeoreyeah, I will also not be around for entire week 14:57
croelandtIt's a holiday next Thursday here14:57
dansmithmaybe just punt then?14:58
jokke_if both of you are away, should we postpone at least the locations discussion for the following week ... I'd say just cancel the meeting for next week all togeher14:58
abhishekksounds good to me, meantime we can discuss on the spec as well14:58
rosmaitacancel works for me14:58
whoami-rajati think this is sort of the same problem when a volume is a project level resource but host attribute is a system level thing -- image (project level) , locations (system level)14:58
jokke_if 3 of ye are away I'm sure we can coordinate any possibly urgent stuff in normal glance channel between myself dansmith and rosmaita 14:58
dansmithwhoami-rajat: agree14:58
abhishekkpdeore, plz send the mail to ML saying next weeks meeting is canceled and we will be meeting a week after15:00
jokke_as all of us are off tomorrow, I'll keep my eyes on ML next week for the backporting matter15:00
abhishekkwe can conclude now, thank you all15:00
pdeoreabhishekk, sure, I will do that15:00
whoami-rajatthanks for the discussion everyone!15:00
jokke_if agreed with stable maint team, will lift my -2 then15:00
pdeoreThanks everyone for joining !!15:00
jokke_thanks all!15:00
opendevmeetMeeting ended Thu May 19 15:01:05 2022 UTC.  Information about MeetBot at . (v 0.1.4)15:01
opendevmeetMinutes (text):
*** dasm is now known as dasm|off21:26
shelrkWe're closed. You guys have to leave.23:43

Generated by 2.17.3 by Marius Gedminas - find it at!