14:00:16 <rosmaita> #startmeeting glance
14:00:17 <openstack> Meeting started Thu Oct 26 14:00:16 2017 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:21 <openstack> The meeting name has been set to 'glance'
14:01:02 <rosmaita> #topic roll call
14:01:05 <rosmaita> hello everyone
14:01:16 <jokke_> o/
14:01:17 <McClymontS> o/
14:01:31 <abhishek_k> o/
14:02:38 <croelandt> \o
14:03:02 <rosmaita> ok, i'll get started
14:03:12 <rosmaita> #topic updates
14:03:33 <rosmaita> we *will* have a glance meeting next week
14:03:49 <rosmaita> but not the week after (that's the sydney summit)
14:04:29 <rosmaita> #topic updates - devstack situation
14:05:02 <rosmaita> quick update on this ... as you are hopefully aware, tasks do not work in the stable/pike or master devstack
14:05:16 <rosmaita> this means that image import does not work, either
14:05:42 <abhishek_k> yes
14:05:45 <rosmaita> the workaround is to use stable/ocata devstack, but update glance to master before running
14:05:56 <rosmaita> the problem has something to do with eventlet
14:06:13 <rosmaita> it is possible that an alternative taskflow engine will fix this
14:06:24 <rosmaita> or moving the task stuff to a separate service
14:06:37 <rosmaita> but we don't want to do the latter until the registry is gone
14:06:53 <rosmaita> and i haven't had time to experiment much with the former
14:07:12 <rosmaita> this is bug  https://bugs.launchpad.net/glance/+bug/1712463 , by the way
14:07:13 <openstack> Launchpad bug 1712463 in Glance "failure when running under wsgi configuration" [High,In progress] - Assigned to Brian Rosmaita (brian-rosmaita)
14:07:13 * smcginnis wanders by
14:07:28 <rosmaita> so, i want to run this plan by everyone
14:08:03 <rosmaita> namely, modify the systemd stuff in devstack so that it does *not* run glance under uwsgi
14:08:21 <rosmaita> that way, at least we'll be able to test everything we're working on in devstack
14:08:36 <rosmaita> and continue to figure out a better solution as time allows
14:08:50 <smcginnis> ++
14:08:51 <rosmaita> (i will now stop to let people read and respond)
14:09:25 <McClymontS> Sounds good to me
14:09:28 <abhishek_k> sounds good, what I have done is instead of calling taskflow async, I am executing it synchronously for testing purpose
14:09:55 <rosmaita> abhishek_k: how exactly are you doing that?
14:10:30 <abhishek_k> instead of calling pool.spawn_n(new_task.run, task_executor) I am calling new_task.run(task_executor)
14:10:39 <rosmaita> gotcha
14:11:01 <rosmaita> (was hoping it was a config option or something)
14:11:17 <rosmaita> ok, i will work on that today, hopefully it won't be too bad
14:11:33 <rosmaita> are there any systemd mavens here who i could direct questions to?
14:11:34 <jokke_> I'd rather have devstack not deploying under uwsgi than need to patch Glance every time one want's to run tasks
14:11:45 <rosmaita> jokke_ ++
14:11:55 <abhishek_k> jokke_: ++
14:12:04 <rosmaita> yes, but abhishek_k does give us devs a good workaround
14:12:24 <rosmaita> ok, one more update i just thought of
14:12:35 <rosmaita> #topic updates - python-glanceclient test situation
14:12:59 <rosmaita> you may have noticed that one of the functional tests (actually 2, but one is n-v) is failing for the glanceclient
14:13:18 <rosmaita> the problem is that it's an autoconverted "legacy" test being run by zuul3
14:13:42 <rosmaita> we are supposed to bring that test into our own repo and zuul3 will run it from there
14:14:22 <rosmaita> the problem right now is that the legacy test tries to create a hard link from some stuff owned by zuul and bring it into devstack user space, but the permissions fail
14:14:56 <rosmaita> i looked quickly at the zuul3 migration docs, and that particular action won't be necessary once the job has been rewritten
14:15:05 <rosmaita> which is great, but the job must be rewritten
14:15:17 <rosmaita> i'm willing to take a crack at this
14:15:26 <rosmaita> but, i will work on the devstack thing first
14:15:38 <rosmaita> so that means failing python-glanceclient tests for a while longer
14:15:59 <rosmaita> which isn't necessarily a big deal ... unless someone has a key fix they want to get in?
14:16:12 <rosmaita> (does anyone?)
14:16:22 <abhishek_k> no
14:17:15 <rosmaita> ok, then, this is mostly an awareness thing ... don't bother to recheck glanceclient patches until we get this fixed
14:18:16 <rosmaita> (although, there is a patch up to zuul for a workaround, but it hasn't been accepted yet, i think because they want to encourage people to migrate these legacy jobs)
14:19:17 <rosmaita> here's the patch i'm talking about: https://review.openstack.org/#/c/512194/1
14:19:44 <jokke_> yeiii ... once more "Lets break everyone and not fix them so they will do the work for us"
14:19:46 <rosmaita> if that ^^ merges, then i believe our glanceclient functional tests will work again
14:20:34 <rosmaita> ok, that's all i've got for updates, if anyone else has an update, please hold for open discussion
14:20:53 <rosmaita> #topic implement update_capabilities for stores
14:20:58 <rosmaita> croelandt you have the floor
14:21:13 <croelandt> thanks
14:21:21 <croelandt> so, I have two users who face  the same issue
14:21:44 <croelandt> they use Glance and NFS, and they have a race condition between nfsd and Glance
14:22:01 <croelandt> Glance boots up, sees taht it cannot write to /path/to/nfs/directory
14:22:13 <croelandt> then decides it will *never* be able to write there
14:22:33 <croelandt> after that, NFS becomes usable, but Glance still will not write to /path/to/nfs/directory
14:22:44 <rosmaita> that is not nice
14:22:47 <croelandt> there is a whole infractructure available to update the capabilities of a store
14:23:03 <croelandt> but the function taht actually implements the update is empty/does not exist
14:23:16 <croelandt> so users can say "ok, update the capabilities every 5 minutes" using a config option
14:23:20 <rosmaita> i didn't think that had been implemented, but your mention of store_capabilities_update_min_interval suggests otherwise
14:23:27 <croelandt> and every 5 minutes, the process is being run
14:23:29 <rosmaita> or are you proposing that new config option?
14:23:36 <croelandt> only to end up in a function that does "pass"
14:23:38 <croelandt> it exists
14:23:42 <McClymontS> I'm certain it exists
14:23:44 <McClymontS> ^^
14:23:48 <rosmaita> gotcha, you just explained
14:23:52 <croelandt> I sent a patch https://review.openstack.org/#/c/455454/
14:23:59 <croelandt> so it's a bit weird to test "in the real world"
14:24:12 <croelandt> because you basically have to wait for a bit before being able to actually write to the directory
14:24:26 <croelandt> so you might want to grab some coffee before retrying
14:24:46 <croelandt> but it worked on my test setup when I wrote the patch, and the whole thing seems reasonable to me
14:24:55 <croelandt> what do you think?
14:25:17 <rosmaita> i'm trying to remember a long discussion i had with zhi when he proposed this
14:25:47 <rosmaita> i'll have to go back and look at the original gerrit review, i guess, and see if it's relevant
14:26:35 <croelandt> my point is mostly that we have a feature that seems only half implemented
14:26:35 <rosmaita> let's take a note that we need to review the config option help text around the store update config option, though, to make it clear that it doesn't work for most stores
14:26:43 <croelandt> so we should either implement it completely, or remove it
14:26:55 <rosmaita> you are reading my mind!
14:26:57 <croelandt> because the store_capabilities_update_min_interval seems kind of useless right now (at least with NFS)
14:27:07 <croelandt> hehe
14:27:12 <rosmaita> is there a better way to handle the NFS race condition?
14:27:25 <croelandt> rosmaita: well, the workaround is to restart glance once nfsd is up
14:27:31 <rosmaita> i'm pretty sure none of the stores implement the update-capabilities
14:27:36 <jokke_> or start it only after
14:27:38 <croelandt> but well, sometimes people reboot services, etc.
14:27:41 <McClymontS> right or wait to start
14:27:48 <croelandt> jokke_: in that case, let's remove the whole feature
14:27:54 <croelandt> it's confusing right now
14:27:57 <McClymontS> I agree
14:28:07 <croelandt> jokke_: and you'd be the one closing my downstream bugs, btw :D
14:28:16 * croelandt does not want to deal with angry users :D
14:28:39 <rosmaita> that whole capabilities with all the bitmasking is pretty obscure
14:29:15 <jokke_> basically the "right" way to do that is to tell what-ever-your-init-is that glance depends nfsd to work and that way glance will follow nfsd (for example if admin restarts nfsd) instead of blowing up
14:29:51 <jokke_> do we have any stores that are actually utilizing that?
14:30:06 <jokke_> I can't recall ever even stumbling upon any of that code
14:30:11 <rosmaita> jokke_ : that == update capabilities ?
14:30:13 <McClymontS> I don't believe so
14:30:18 <jokke_> rosmaita: yeah
14:30:30 <rosmaita> i don't think so, but we should definitely check
14:30:41 <rosmaita> someone might have slipped one past the goalie
14:30:48 <McClymontS> lol
14:30:57 <jokke_> so was that a) some half implemented thing that never got finished or was it for some single usecase on single store and nothing else got ever touched?
14:31:22 <rosmaita> i believe that it was overengineered
14:31:26 <croelandt> yeah, I'm not sure whether other stores use this feature
14:31:30 <jokke_> 'cause we can't really remove it if it's actually implemented somewhere
14:31:40 <rosmaita> zhi had some use case for capabilities, but i dont' remember why it was important
14:31:58 <jokke_> and in that case, no matter how horrible it sounds we should look after croelandt's patch
14:32:14 <rosmaita> ok, sounds like we need some quick research here
14:32:20 <rosmaita> any volunteer?
14:32:31 <rosmaita> need to (a) make sure the capability is unused, and
14:32:44 <jokke_> but if only thing ever that was implemented was the config option, I'd say we get rid of it and document what should be done to achieve success
14:32:45 <rosmaita> (b) go back and look over the original patches to see what the use case was for this
14:33:12 <rosmaita> and (c) what jokke_ just said
14:33:25 <McClymontS> I agree with jokke_ and I think we are all on the same page here
14:33:41 <McClymontS> figure out whats going on with it, then decide if we want to implement it or get rid of it
14:34:03 <abhishek_k> rosmaita: what is the timeline for this?
14:34:10 <rosmaita> croelandt ?
14:34:23 <rosmaita> abhishek_k (i don't have a priority for this yet)
14:34:32 <jokke_> I think croelandt wanted this fixed 2 weeks ago :P
14:34:37 <rosmaita> croelandt is the one with customers on his back
14:34:38 <croelandt> rosmaita: taht's a good summary :D
14:34:41 <croelandt> hehe
14:34:50 <croelandt> well, they're not putting that much pressure on me, to be honest
14:35:08 <croelandt> so it's not top priority
14:35:12 <rosmaita> croelandt do you have time to research this and we can discuss next week?
14:35:33 <croelandt> rosmaita: I'll take a look at it, sure, though I wuold have loved some help from someone more knowledgeable
14:35:34 <abhishek_k> might be I can put some time a week after on this
14:36:01 <rosmaita> croelandt you could ask flavio, i remember him commenting on the original patch for the capabilities
14:36:08 <rosmaita> zhi unfortunately has moved on
14:36:18 <jokke_> croelandt: ping me if you need help and I'll try to dig it out with you
14:36:19 <croelandt> rosmaita: ok
14:36:23 <croelandt> jokke_: oh, fantastic
14:36:40 <jokke_> croelandt: and yeah, defo ask Flavio :D
14:36:49 <croelandt> yeah, but asking Flavio about anything is cheating
14:37:02 <rosmaita> #action croelandt to research glance_store update_capabilities as outlined (a)-(c) above
14:37:02 <jokke_> he loves to dig in old history he has been involved in
14:37:08 <croelandt> haha
14:37:18 <rosmaita> ok, anything else on this?
14:37:22 <croelandt> nope, I'm good
14:37:50 <rosmaita> cool, thanks for putting this on the agenda ... we will either fix something, or simplify the code base!
14:38:01 <rosmaita> #topic proposed backport
14:38:08 <jokke_> croelandt: but tbh I rather make the time this/next week than get pulled in during summit because it just became huge customer priority
14:38:17 <rosmaita> #link https://review.openstack.org/#/c/440876/
14:38:50 <rosmaita> background on this one is that the e-m-c upgrades were introduced in ocata for zero-downtime database upgrades
14:38:54 <croelandt> jokke_: yeah, they have a workaround, so... hope it wont be a priority
14:39:01 <abhishek_k> yup, this is required to fix upgrade in stable/ocata
14:39:27 <rosmaita> i thought the regular upgrade worked in ocata?
14:39:34 <rosmaita> but i may be confusing myself
14:40:48 <abhishek_k> rosmaita: if you run db sync then it works, but if you try e-m-c then it fails during migrate phase
14:41:05 <rosmaita> ok, gotcha
14:41:15 <rosmaita> well, this is a pretty safe change
14:41:29 <rosmaita> jokke_ what do you think (in your capacity of release czar)?
14:41:32 <jokke_> so I've bit mixed feelings on this
14:41:43 <rosmaita> here's the related bug: https://bugs.launchpad.net/glance/+bug/1726237
14:41:44 <openstack> Launchpad bug 1726237 in Glance "Installing stable/ocata using expand/migrate/contract for the first time raises TypeError" [Undecided,New]
14:42:17 <jokke_> it's fairly isolated fix, but it's also a fix for experimental feature in Phase II branch
14:42:54 <jokke_> which means that I'm tempted to say we leave it as it's fixed in Pike (where the feature is as well experimental)
14:43:23 <rosmaita> #link https://docs.openstack.org/releasenotes/glance/ocata.html
14:43:35 <rosmaita> those are the ocata release notes, saying it's experimental
14:44:02 <jokke_> would we have done that in March when the release was fresh, sure it would have made sense
14:44:10 <abhishek_k> what will be the consequences if it is accepted for stable/ocata?
14:44:18 <jokke_> but atm. with my stable hat on, I'd say we leave it
14:45:00 <smcginnis> Yeah, experimental in phase II is probably not going to fly with the stable team.
14:45:02 <jokke_> abhishek_k: depending of what tonyb thinks it might block us releasing
14:45:14 <nikhil> ./
14:45:27 <smcginnis> It would not be aligned with stable:follows-policy that glance asserts.
14:45:54 <abhishek_k> jokke_: ok, Could you please add your concerns on the patch as well?
14:45:59 <rosmaita> i'm looking at the db docs, we need to update them during queens
14:46:08 <rosmaita> #link https://docs.openstack.org/glance/latest/admin/db.html
14:46:34 <rosmaita> we also need to get the upgrade-in-gate jobs set up so that we can un-experimentalize this featurew
14:48:01 <jokke_> smcginnis: well as said it's pretty messy ... it is kind of critical feature blocking bug, although on experimental feature so it's more or less depending who reads the policy if it's acceptable or not and thus I'd say we don't as a) I don't want to start that fight with anyone b) as it went totally under radar for good 7 months, it's likely not impacting that many people either
14:48:15 <rosmaita> #action jokke_ add concerns about patching stable/ocata to https://review.openstack.org/#/c/440876/
14:48:50 <rosmaita> smcginnis if you have a minute to add your comment about not being aligned with stable:follows-policy, that would be helpful too
14:48:51 <jokke_> rosmaita: IIRC that was on of the Queens priorities
14:49:09 <smcginnis> Sure, got it in a tab and should get to it soon.
14:49:19 <rosmaita> jokke_ yeah, but i probably should've put a spec-lite in for it
14:49:23 <abhishek_k> jokke_, smcginnis: makes sense
14:50:21 <rosmaita> i've been holding off on priorities patch until ossn-0075 spec is worked out, but that is a bad idea
14:50:32 <rosmaita> i will put up a priorities patch later today
14:50:57 <rosmaita> #action rosmaita propose queens priorities patch to glance-specs
14:51:12 <rosmaita> ok, anything else about the backport proposal?
14:51:12 <nikhil> do we have quotas in Q priority?
14:51:18 * nikhil waits
14:51:42 <rosmaita> #topic open discussion
14:51:52 <rosmaita> nikhil: nope
14:52:05 <rosmaita> i think the plan is to research and be ready to do them in R
14:53:35 <nikhil> k, ty
14:54:04 <rosmaita> we do need someone to take on the task of making sure the research actually happens, though
14:54:31 <rosmaita> it looks like the people who proposed the quotas changes during the pike cycle are not interested in following up
14:55:02 <rosmaita> but i personally think some kind of quotas implementation is important for glance
14:55:23 <rosmaita> we've held off so that we can be consistent with all of openstack, but the cross-project effort appears to have stalled, too
14:55:44 <smcginnis> There has been some recent rumbling there though.
14:56:07 <smcginnis> Maybe worth seeing if anyone is pursuing that before doing anything glance-specific with quotas.
14:56:22 <rosmaita> does anyone here have sufficient interest to become our point person on the glance quotas initiative?
14:56:31 <rosmaita> smcginnis ++
14:56:46 <nikhil> how  cross project efforts are driven?
14:57:10 <nikhil> ad-hoc or some centralized way?
14:57:19 <jokke_> nikhil: that's the thing ... they really aren't anymore
14:57:22 <rosmaita> i think it's pretty ad-hoc
14:57:32 <nikhil> jokke_: ha ;)
14:57:47 <rosmaita> i think i saw on the ML that the cross-project specs repo is being decomissioned? or am i misremembering?
14:57:59 <smcginnis> sdague had some thoughts there, but I know he isn't able to follow through on it. Might be worth pinging him.
14:58:04 <jokke_> like since the x-project meeting was killed it has been super difficult to follow up any x-proj activities
14:58:09 <smcginnis> IIRC, there was also a ML post or two recently.
14:58:16 <nikhil> rosmaita: well I can't commit to pushing it but I can be the go-to person for discussions/quota meetings
14:58:39 <nikhil> smcginnis: checking
14:58:48 <rosmaita> ok, i will mention this at the summit and see if we get any takers
14:58:54 <nikhil> k
14:59:02 <rosmaita> maybe a product person or operator would be willing to coordinate
14:59:17 <jokke_> I'm also interested to see during the forum session how much the audience has interest to have quotas
14:59:31 <rosmaita> jokke_ ++
14:59:39 <rosmaita> we need to remember to mention it
14:59:44 <nikhil> from cern
14:59:44 <rosmaita> ok, just about out of time
14:59:46 <smcginnis> I have gotten feedback from operators that it is important to them.
15:00:11 <rosmaita> ok, our hour is up ... thanks everyone!
15:00:15 <jokke_> smcginnis: I've got that feedback from one or two
15:00:19 <jokke_> chher everyone1
15:00:26 <abhishek_k> thank you all
15:00:35 <rosmaita> #endmeeting