16:00:00 <thingee> #startmeeting cinder
16:00:00 <openstack> Meeting started Wed Feb 25 16:00:00 2015 UTC and is due to finish in 60 minutes.  The chair is thingee. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:04 <openstack> The meeting name has been set to 'cinder'
16:00:08 <thingee> hello everyone!
16:00:09 <hemna> morning
16:00:10 <dulek> o/
16:00:13 <e0ne> hi!
16:00:14 <thangp> o/
16:00:15 <scottda> hi
16:00:15 <bswartz> hi
16:00:20 <smcginnis> o/
16:00:23 <sigmavirus24> 'ello
16:00:25 <DuncanT> lo
16:00:27 <thingee> some announcements before we get started...
16:00:29 <asselin> hi
16:00:30 <jnrao> hi
16:00:46 <yuriy_n17> hi everyone
16:00:47 <thingee> Third party CI http://lists.openstack.org/pipermail/openstack-dev/2015-January/054614.html
16:00:50 <rhe00> hi
16:00:51 <mtanino> hi
16:00:54 <thingee> it's required for Kilo for all volume drivers
16:00:55 <cebruns> hi
16:01:12 <jungleboyj> o/
16:01:19 <xyang1> Hi
16:01:22 <thingee> if your volume driver, and all protocols it supports are not covered by a CI by the end of Kilo, you risk the driver being removed.
16:01:48 <thingee> please don't wait and get help in the third party ci meetings
16:01:56 <thingee> https://wiki.openstack.org/wiki/Meetings/ThirdParty
16:02:11 <smcginnis> thingee: End of K3, right? Not end of Kilo.
16:02:18 <thingee> smcginnis: sure
16:02:37 <thingee> speaking of deadlines, Cinder has deadlines for code.
16:02:39 <thingee> http://lists.openstack.org/pipermail/openstack-dev/2015-February/056964.html
16:02:52 <thingee> March 1st, code must be in for all BPs. Passing jenkins. Not WIP
16:03:15 <thingee> I will untarget BPs otherwise. The reviewers need time to, well, review!
16:03:46 <thingee> March 10th, code must be merged for BPs. We will do a freeze. Bug patches are OK though!
16:04:05 <thingee> Kilo priorities...anyone doing reviews should be following https://etherpad.openstack.org/p/cinder-k3-priorities
16:04:12 <thingee> These are things targeted.
16:04:17 <thingee> for k-3
16:04:25 <thingee> Ok lets gets started!
16:04:33 <thingee> Agenda for today
16:04:38 <thingee> #link https://wiki.openstack.org/wiki/CinderMeetings#Next_meeting
16:04:51 <thingee> #topic Display dates with tz info with API change
16:04:53 <thingee> yuriy_n17: hi
16:05:01 <thingee> #link https://review.openstack.org/#/c/156552/
16:05:07 <yuriy_n17> hi everyone
16:05:23 <thingee> yuriy_n17: have you spoke to the API WG? https://wiki.openstack.org/wiki/API_Working_Group
16:05:49 <yuriy_n17> no
16:06:06 <thingee> Ok, general project consensus would be great.
16:06:15 <thingee> members of each project, including myself, are part of this
16:06:36 <thingee> also to discuss the specifics of changing this in current API versions
16:06:50 <thingee> which I think is what the current convtroversy is
16:08:14 * thingee taps the microphone
16:08:29 * flip214 turns up the volume to 11
16:08:52 * jungleboyj breaks off the nob.
16:08:54 <yuriy_n17> thingee: To have dates displayed in Cinder with tz info I offered that
16:09:14 <thingee> yuriy_n17: understood. I would appreciate project consensus on this though. Not just Cinder doing this
16:09:23 <thingee> I don't have a project with it though.
16:09:26 <thingee> problem*
16:10:04 <DuncanT> thingee: I do. It breaks client code
16:10:22 <DuncanT> thingee: That needs to be managed somehow
16:10:25 <thingee> DuncanT: agreed. Let me be more clear...I don't have a problem with the idea.
16:10:38 <thingee> DuncanT: I want the API WG to decide on when this should be introduced to a project.
16:10:46 <DuncanT> thingee: Ah, me neither, in fact it is an important thing to fix
16:11:00 <sigmavirus24> thingee: as a member, I agree with the idea
16:11:07 <e0ne> DuncanT: it works for me but i'll recheck it once more after meeting
16:11:15 <sigmavirus24> however, I'm more of the opinion that every timestamp should be stored in UTC+0 for simplicity
16:11:29 <winston-d> DuncanT: agree, everytime I look at the log, I need to convert the timezone, twice.
16:11:37 <thingee> alright no disagreements... yuriy_n17 can I count on you to bring this to the API WG?
16:11:43 <DuncanT> sigmavirus24: +1 That solves the current issue without any change to the API
16:12:34 <yuriy_n17> thingee: sure
16:12:38 <sigmavirus24> DuncanT: at worst cinder could alter itself to accept any timestamp with tz info and convert it to UTC+0 before storing or just assume it's already UTC+0
16:12:44 <kmartin> yuriy_n17, https://wiki.openstack.org/wiki/Meetings/API-WG
16:12:45 <sigmavirus24> yuriy_n17: I believe our next meeting is in a few hours
16:13:03 <thingee> #action yuriy_n17 to talk to API WG about TZ time in APIs and when they should be introduced.
16:13:11 <thingee> sigmavirus24, yuriy_n17: thanks!
16:13:28 <DuncanT> sigmavirus24: Tomorrow, according to that link
16:13:42 <thingee> #topic Notice only: 3rd Party CI - Important Action required
16:13:45 <thingee> asselin: hi!
16:13:48 <asselin> hi
16:13:52 <sigmavirus24> DuncanT: correct. The alternating schedule messes me up =P
16:13:52 <thingee> #link http://lists.openstack.org/pipermail/openstack-dev/2015-February/056508.html
16:14:10 <asselin> so last week I mentioned that -infra is changing the ip address of the gerrit server
16:14:19 <asselin> this impacts 3rd party ci. please read the e-mail
16:14:28 <asselin> #link http://lists.openstack.org/pipermail/openstack-dev/2015-February/056508.html
16:14:49 <asselin> yesterday, in the infra meeting they agreed to setup a netcat hello world service to help test firewall rules
16:15:05 <asselin> that completes my action item from last week. :)
16:15:29 <xyang2> asselin: when will the hello world service be ready?
16:15:47 <anteaya> when pleia2 sets it up, we didn't set a time for it
16:15:53 <asselin> xyang2, not sure. hopefully this week. pleia2 has the action to do it.
16:16:13 <xyang2> asselin, anteaya: ok, thanks
16:16:38 <asselin> if not other questions, that's it from me
16:16:48 <thingee> #link http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-02-24-19.01.txt
16:17:06 <thingee> asselin: thank you! anything else?
16:17:13 <asselin> nope
16:17:26 <thingee> #topic Approach to bug 1409012
16:17:28 <openstack> bug 1409012 in Cinder "Volume becomes in 'error' state after scheduler starts" [High,In progress] https://launchpad.net/bugs/1409012 - Assigned to Michal Dulko (michal-dulko-f)
16:17:32 <thingee> DuncanT, dulek: hi!
16:17:36 <dulek> hi, let me explain background quickly
16:17:39 <thingee> #link https://bugs.launchpad.net/cinder/+bug/1409012
16:17:57 <dulek> Problem is scheduler doesn't have metrics at the start so it fails requests in 60-seconds-wide time window.
16:18:10 <dulek> winston-d proposed a patch to shrink this window to a few seconds, but this doesn't solve the bug because still requests sent before metrics update will be received first and scheduler will fail
16:18:19 <dulek> (but change is of course great)
16:18:29 <dulek> now I can take two approaches in the patch that's delaying requests:
16:18:40 <dulek> 1. delay requests for 60 seconds or less (manager will count down the delay passed to further request flows)
16:18:53 <dulek> this makes delays longer but scheduler will know whole situation when making a decision
16:19:05 <dulek> 2. ask scheduler_driver every second till we have valid response or delay passes
16:19:17 <dulek> this will make delays shorter but scheduler may not have all the metrics when making a decision which may lead to a poor choice
16:19:27 <dulek> So the question is - which one is better?
16:19:48 <hemna> shouldn't the scheduler wait to d scheduling until it's collected stats from every c-vol instance ?
16:20:09 <DuncanT> The second approach IMO, otherwise you get unnecessary stalls
16:20:19 <dulek> hemna: That's option 1 I guess.
16:20:20 <e0ne> hemna: we can't do it with currunt oslo.messaging
16:20:21 <winston-d> hemna: scheduler doesn't know how many vol servies are up
16:20:47 <hemna> winston-d, the db table lists the c-vol services
16:20:57 <hemna> and their state
16:21:22 <thingee> hemna: does that get propagated by the scheduler though? :)
16:21:27 <winston-d> hemna: there's no guarantee active c-vol in DB will continute to be active
16:21:36 <dulek> winston-d: +1
16:21:51 <DuncanT> thingee: The scheduler can connect to the db just fine
16:21:58 <dulek> winston-d: but we should make it possible to send metrics for every volume to be fair
16:22:26 <dulek> winston-d: not use the one we're getting first - that's my opinion
16:22:36 <thingee> DuncanT: right, I'm saying if the scheduler uses the db to tell what c-vols are active, but it's setting that itself...not really useful :)
16:22:46 <winston-d> dulek: i agree, that's why I think waiting for one periodic_interval is acceptable.
16:23:00 <DuncanT> dulek: Does total 'fairness' make sense for a small time window though? They're already unfair enough on a busy system  due to update lag anyway
16:23:03 <hemna> thingee, shouldn't the c-vol services be updating their status in the services table?
16:23:05 <hemna> not the scheduler
16:23:20 <thingee> hemna: well that's why it was a question. I don't know what is.
16:23:25 <hemna> oh ok :)
16:23:29 <hemna> there is a report_count field as well
16:23:40 <dulek> hemna: this isn't done because of db performance issues
16:24:10 <dulek> DuncanT has more info on that i think
16:24:34 <DuncanT> dulek: I forget, sorry, It vaguely rings bells, but I can't remember the details
16:24:37 <winston-d> hemna: query DB for once, is fine, but the worst case of current fix is to wait one periodic interval.
16:25:00 <winston-d> hemna: which doesn't need to query DB.
16:25:20 <thingee> dulek: can you explain #2 again?
16:25:38 <dulek> It's DuncanT's idea, maybe he'll explain it better?
16:25:46 <DuncanT> Ok
16:25:53 <dulek> DuncanT: thanks :)
16:26:22 <winston-d> dulek: my opinion for your change is, we can move the sleep waiting out of taskflow, into scheduler manager, then we have great change to make sure *every* request doesn't have to be block for 60 secs.
16:26:25 <DuncanT> So basically we just keep looping the scheduler call every second until we find a host that can take it, then send it. We don't care if there is a better host coming along later
16:27:04 <DuncanT> It means the normal case scheduling resumes as soon as at least one host has reported in
16:27:16 <thingee> DuncanT: so we're just leaving things in the queue?
16:27:46 <winston-d> DuncanT: but it may still fail even scheduler already have one set of stats.
16:27:56 <dulek> thingee: no, looping the scheduler means looping FilterScheduler calls
16:28:03 <dulek> thingee: not RPC calls
16:28:13 <winston-d> DuncanT: if the request unfortunately needs more than that backend can offer.
16:28:16 <DuncanT> winston-d: Keep trying it until you hit 60 seconds or it passes
16:28:38 <jungleboyj> DuncanT: That seems to make sense.
16:28:39 <DuncanT> winston-d: The worst case is the same as just waiting 60 seconds, but the average case is better
16:28:43 <thingee> winston-d: +1 yea that's what I was thinking
16:29:12 <dulek> I agree with winston-d here
16:29:29 <winston-d> DuncanT: when this change lands: https://review.openstack.org/#/c/158623/, I am pretty confident 95% cases, the scheduler will have all stats from c-vol(s) in a few seconds.
16:29:30 <jungleboyj> DuncanT: So, worst case scenario is you have to wait 60 seconds to find out there isn't a backend that satisfies your needs?
16:29:38 <hemna> so the c-vol service updates a report_count field every report interval
16:29:38 <DuncanT> jungleboyj: Correct
16:29:48 <thingee> I think I'm fine with this, but I need to know what looping means in the filter call. It seems to me if the scheduler has not valid hosts available, it shouldn't be taking anything from the MQ
16:30:04 <hemna> can't the scheduler keep track of those at startup waiting for each of them to report
16:30:14 <hemna> and have some timeout for misbehaving/down c-vol services
16:30:17 <DuncanT> winston-d: the scheduler can look in the db and if it has received all the reports then it can terminate early, no problem
16:30:26 <hemna> like wait 2 reporting intervals
16:30:35 <thingee> DuncanT: is what I said earlier possible?
16:30:36 <hemna> and ignore the c-vol services that haven't updated in 2 intervals
16:30:38 <DuncanT> hemna: The timeout should be twice the interval, yup
16:31:00 <DuncanT> hemna: There's no way to stop the scheduler consuming incoming request at the moment though
16:31:10 <DuncanT> hemna: So you have a create stalled for a full minute
16:31:19 <DuncanT> hemna: That doesn't need to be
16:31:19 <e0ne> thingee: we are limited in messaging impl to take or not to take something accroding to our needs
16:31:29 <jungleboyj> hemna: +2
16:31:34 <hemna> well I think it should be stalled until the scheduler recognizes that c-vol services have reported.
16:31:50 <hemna> if they don't report, then it's hosed anyway
16:31:55 <thingee> seems like messaging impl should be updated to stall taking?
16:31:55 <DuncanT> thingee: We can't make it not consume though, the olso messaging library won't let up
16:32:11 <thingee> were there push backs on that from the oslo team?
16:32:16 <thingee> or was it even proposed?
16:32:36 <e0ne> thingee: yep, if we or somebody fix it, cinder patch will be very small and clean
16:32:40 <DuncanT> thingee: If we can get that into oslo, then sure, but I can't see how to do that given the way the rabbit connections are handled at the moment
16:32:43 <hemna> delay the start of listening to the API message queues
16:32:54 <hemna> until the scheduler sees c-vol services reporting
16:32:59 <hemna> then register the listeners
16:33:15 <dulek> hemna: Nope, then scheduler won't get metrics
16:33:22 <e0ne> hemna: +1. that is what we need
16:33:31 <hemna> dulek, metrics?
16:33:36 <hemna> it has the connection to the DB
16:33:38 <dulek> hemna: scheduler need to listen for metrics, but not listen to requests
16:33:39 <DuncanT> hemna: You need to get metrics on a different queue to the create RPC
16:33:42 <winston-d> hemna: scheduler needs to broadcast a call to all c-vol when it starts 'hey, guys, send me your latest stats'.
16:33:57 <dulek> hemna: but scheduler gets volume metrics/stats from RPC instead of DB
16:33:59 <thingee> Ok, so if I'm understanding things correctly, this only allows for X amount of time that things come in to resume them if the scheduler wasn't ready
16:34:03 <hemna> winston-d, isn't that on different message queues?
16:34:05 <thingee> and right not we're saying 60 seconds
16:34:06 <hemna> than API requests ?
16:34:19 <DuncanT> thingee: Correct
16:34:32 <dulek> hemna: no, it's the same
16:34:32 <winston-d> hemna: current RPC handling code takes care of all xchange/queues in one call, IIRC.
16:34:53 <dulek> winston-d: true
16:35:01 <hemna> dang :(
16:35:13 <winston-d> hemna: so to be able to broadcast, you open the door for 'create', 'resize', all requests.
16:35:36 <dulek> we explored all these ideas you mention
16:35:43 <thingee> DuncanT, dulek: seems fine to me to wait X amount of time, when the first available host reports, start accepting
16:35:58 <dulek> that's why I presented just 2 alternatives - rest won't work
16:36:16 <thingee> everyone in agreemnt?
16:36:18 <dulek> thingee: hm, you're getting it wrong I think
16:36:23 <DuncanT> thingee: You can't just wait for the first host, since it might not be able to take that volume
16:36:40 <dulek> thingee: so we're unable to stop accepting requests for X seconds
16:36:44 <e0ne> thingee: +1. and we need to raise this topic to oslo.messaging team
16:36:50 <DuncanT> thingee: And waiting X time every time means a stalled create (or many) every time a scheduler comes up
16:36:54 <winston-d> thingee: to be more accurate, we accepting once scheduler starts, dulek's change is to block those requests until schedule have, hopefully, all c-vol stats
16:37:08 <dulek> winston-d: that's true
16:37:27 <thingee> winston-d: that's what I was trying to convey.
16:37:29 <thingee> dulek: ^
16:37:48 <thingee> but how do you know when *all* have reported?
16:38:15 <dulek> thingee: We can hope that all of them should report in periodic_interval
16:38:29 <dulek> Or even faster with winston-d's patch
16:38:38 <winston-d> thingee: sadly, we don't. scheuler has no idea what periodic_interval are set for each up&running c-vol.
16:38:57 <thingee> dulek: periodic interval to me seems like solution #1. which we were against because of unnecessary wait time
16:39:22 <winston-d> thingee: scheduler crosses its fingers and hope c-vol shares the same value for 'periodic_interval' as itself and wait for one or two interval.
16:39:30 <DuncanT> winston-d: I think it is fair to require the periodic_interval in the cinder.conf on the scheduler to be right
16:39:31 <e0ne> :)
16:39:43 <hemna> how about a new c-vol column called ready
16:39:54 <hemna> that signals to the scheduler that the c-vol service is ready to reprot
16:39:55 <hemna> report
16:40:15 <DuncanT> hemna: no value, it can just send its stats RPC
16:40:26 <DuncanT> hemna: Remember you might have N shedulers
16:40:28 <dulek> hemna: that column would have to mean - stats sent after scheduler startup
16:40:28 <hemna> well, meaning if the stop the scheduler from listening
16:40:36 <hemna> the scheduler can look for that column
16:40:41 <hemna> prior to listening
16:40:53 <jungleboyj> How big a deal is this really?  This is just at start time?  How often is the delay going to be an issue?
16:41:05 <hemna> the c-vol services can push the messages into the queue even if the scheduler isn't listening
16:41:29 <dulek> hemna: yes, but there's no guarantee that it did before the request
16:41:32 <hemna> dunno, I'm just trying to come up with ideas.
16:41:38 <winston-d> DuncanT: unless we have ways to force cfg sync between c-sch and c-vol, the best case is to hope for the best.
16:41:41 <thingee> jungleboyj: +1 mostly I'm a little confused and we gain a little time to resume
16:41:43 <dulek> hemna: request can still come before stats
16:42:01 <DuncanT> winston-d: We can start looking at putting config in the db... we've talked about that a few times
16:42:09 <hemna> dulek, yah that's fine, the scheduler should be able to reject requests before it's ready
16:42:13 <jungleboyj> thingee: I don't want to see a complicated solution for a small window.  Glad I am not the only one thinking it.
16:42:16 <bswartz> it sounds to me like the scheduler is just useless until 60 seconds after it comes up
16:42:18 <hemna> but the volume shouldn't be put in error state
16:42:34 <dulek> hemna: But it will fetch it from the MQ and fail the volume creation
16:42:34 <winston-d> jungleboyj: exactly, this is not really a big deal, at least to me.
16:42:51 <bswartz> but even in a HA config you should have multiple schedulers running, so if one needs to go down and come back up, it's just useless until 60 seconds later
16:43:06 <DuncanT> bswartz: ++
16:43:11 <thingee> bswartz: +1
16:43:15 <hemna> winston-d, jungleboyj well except in a busy deployment, you might get a bunch of volume create requests queued up as you bounce c-vol
16:43:34 <thingee> I would really like to see things fixed up in oslo message.
16:43:37 <jungleboyj> dulek: Can you answer why the delay is such a big concern?
16:43:39 <winston-d> hemna: true
16:43:49 <DuncanT> hemna: If you bounce the scheduler right now then you get a bunch of volumes created in error
16:43:50 <jungleboyj> hemna: I suppose.
16:43:56 <dulek> dulek: I'm all for the delay - my current implementation in the patch
16:44:05 <dulek> jungleboyj: ^
16:44:27 <DuncanT> jungleboyj: The delay always being 60 seconds is my complaint... it should only be as long as it takes for the first suitable backend to report in
16:44:42 <DuncanT> jungleboyj: i.e. approach (2) in the initial problem statement
16:44:45 <hemna> DuncanT, which should be 2 reporting cycles
16:44:47 <jungleboyj> DuncanT: Ok, I can see that argument.
16:44:58 <winston-d> dulek: +1, all my suggestions were about how to make the window smaller without too much trouble, if possible
16:45:14 <DuncanT> jungleboyj: That can break fairness in the scheduler though, which some people don't like
16:45:26 <DuncanT> jungleboyj: I'll take the unfairness over the stall, personally
16:45:41 <jungleboyj> DuncanT: Again, it seems like a small window we are talking about.
16:46:08 <winston-d> DuncanT: problem is, that's a bit too ideal to me, 'first suitable backend' means scheduler will have to try many times.
16:46:09 <hemna> so the messaging approach is to leave the API requests on the queue until the scheduler is ready?
16:46:13 <DuncanT> jungleboyj: A minute stall on volume creates will show up in our SLA for sure
16:46:16 <dulek> So can I just ask a question - fairness (scheduler making good decisions in that small time window) or stall
16:46:19 <dulek> ?
16:46:29 <DuncanT> hemna: Yes, but that requires bit messaging changes
16:46:36 <winston-d> Cory may be pissed given they have so many backends?
16:46:43 <hemna> DuncanT, seems like a good feature to have
16:46:53 <bswartz> DuncanT: what if you have redundant schedulers though -- can't one that's working take the message while the other is waiting for stats?
16:46:55 <flip214> DuncanT: if a minute stall is too long, you'll need active/active services.
16:47:00 <hemna> like a listening mask
16:47:02 <DuncanT> hemna: Definitely, but we need to fix this bug now.
16:47:07 <hemna> yah :(
16:47:13 <DuncanT> flip214: Active/active makes this worse
16:47:18 <dulek> bswartz: It cannot, scheduler need to connect to get the stats
16:47:32 <dulek> bswartz: once it connects - it gets everything, even requests
16:47:37 <DuncanT> flip214: Active/active means when the second one comes up, it takes half the live requests but stalls them all for a minutes
16:47:43 <jungleboyj> dulek: Sounds like having less stall is preferred.
16:47:52 <DuncanT> jungleboyj: ++
16:48:01 <bswartz> so the scheduler needs a way to start accepting stats updates without accepting create requests
16:48:04 <dulek> jungleboyj: thank you, I was getting more and more confused
16:48:05 <winston-d> dulek: correctness and eventually/overall fairnes
16:48:13 <thingee> I think this needs to be more thought out honestly. I agree with jungleboyj that I don't really see what this is fixing. And it's as other people raised making other scheduler cases of active/active worse.
16:48:33 <dulek> thingee: which solution?
16:48:36 <jungleboyj> Wheee!
16:49:00 <thingee> this delay thing. dulek, jungleboyj asked you what are we actually noticing being reported up as a problem here.
16:49:03 <winston-d> dulek: i'm not too concerned about a few requests not being placing to the 'best' backend it can be.
16:49:04 <thingee> I didn't see an answer to that.
16:49:11 <DuncanT> winston-d: ++
16:49:27 <jungleboyj> winston-d: ++
16:49:45 <cebruns> winston-d: +1 Perfect is the enemy of good.
16:49:56 <jungleboyj> That is the feeling I am getting here.  I wouldn't want to wait 60 seconds every time we are restarted if it can be avoided.
16:49:58 <DuncanT> thingee: Start a new scheduler (active/active for example). For 30 seconds, right now, half your volumes go into error
16:50:10 <dulek> thingee: So the problem is - if there are requests in the queue and scheduler starts - it will fail all of them
16:50:14 <DuncanT> thingee: That is the current bug
16:50:19 <hemna> winston-d, except it might fail via capabilities masking before the right backend reports
16:51:22 <thingee> dulek: can you check if something is in the queue, if so, do we have a backend reported? no, ok delay
16:51:26 <thingee> but don't delay by default
16:51:42 <winston-d> hemna: that's why DuncanT proposed scheudler only starts processing request when the first 'acceptable' backend is available.
16:51:43 <dulek> thingee: No, oslo.messaging won't allow that now
16:51:53 <thingee> the bug again seems in oslo.messaging.
16:52:09 <dulek> thingee: yes, we're talking about solution for kilo
16:52:14 <hemna> thingee, or, we separate the queues for API -> scheduler and c-vol -> scheduler
16:52:28 <dulek> thingee: as we agreed on the bug discussion that it's too late to request changes in oslo
16:52:29 <thingee> dulek: you mean L right? think a release already went out
16:52:32 <hemna> and we don't listen to the API topic until the scheduler is ready.
16:52:39 <winston-d> hemna: we have separate queues.
16:52:51 <thingee> dulek: I wouldn't want this in K for cinder though. it's too late without seeing potential issues.
16:52:54 <hemna> winston-d, there is only 1 scheduler_topic
16:53:09 <hemna> I'm suggestion 2 scheduler topics
16:53:11 <e0ne> hemna: the problem is we can't listen only on queue
16:53:15 <DuncanT> thingee: This is a real bug in K right now
16:53:16 <dulek> Okay, did we agreed that the patch should follow DuncanT's idea?
16:53:27 <hemna> 7 minutes left
16:53:46 <DuncanT> thingee: We hit it in release testing, it is trivial to trigger
16:53:47 <hemna> can we continue to hack out ideas in cinder channel and move on ?
16:53:59 <thingee> hemna: there is nothing else
16:54:02 <hemna> ok
16:54:08 <dulek> So that would be - if we're in the 60-seconds-time-windows then we should ask scheduler_driver every second if it fails?
16:54:09 * anteaya is hoping for open discussion
16:54:37 <thingee> ok we'll continue to #openstack-cinder after the meeting
16:54:41 <thingee> #topic open discussion
16:54:50 <anteaya> thanks
16:55:03 <anteaya> so these are the jobs jenkins is running on cinder: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n1066
16:55:18 <anteaya> does everyone know what these mean or how to find out?
16:55:40 <anteaya> because if not, I would like to help you so you can know
16:55:54 <bswartz> anteaya: I promise you not everyone knows
16:56:08 <anteaya> since when I asked about jobs running on cinder, or about to last week noone seemed to know about sheepdog
16:56:17 <anteaya> bswartz: I dont' take your word for it
16:56:24 <anteaya> they are here and can speak for themselves
16:56:29 <anteaya> bswartz: are you saying you know?
16:56:42 <anteaya> bswartz: because if so, that's great
16:56:49 <cebruns> anteaya: I'll be honest and say I don't know.  :)
16:56:52 <bswartz> I know where to find this stuff when I need to
16:56:56 <anteaya> cebruns: thanks
16:57:00 <anteaya> bswartz: great
16:57:15 * jungleboyj doesn't
16:57:18 <anteaya> I'm not pointing fingers, if anyting this is a failing from my end
16:57:21 <anteaya> which I want to fix
16:57:25 <anteaya> jungleboyj: great thank you
16:57:56 <anteaya> okay so I'd like to fix this so all folks in cinder who want to know know how to figure out what is running on jenkins against cinder repos
16:58:08 <anteaya> I was thinking maybe a q and a tutorial?
16:58:15 <anteaya> what do folks think of that?
16:58:49 <anteaya> nothing?
16:58:55 <anteaya> am I invading your space?
16:59:02 * flip214 turns up the volume to 11
16:59:02 <anteaya> if this isn't helpful I can find the door
16:59:09 <smcginnis> anteaya: Sounds good, but think we are out of time here.
16:59:14 <flip214> oh, it would be very interesting!
16:59:14 <dulek> some of guys are already on openstack-cinder
16:59:15 <anteaya> fair enough
16:59:17 <winston-d> anteaya: tutorial would be nice
16:59:19 <cebruns> anteaya: I think it's good.
16:59:26 <dulek> anteaya: tutorial would be great
16:59:36 <anteaya> okay thanks let me work on that
16:59:39 <anteaya> I'm done
16:59:40 <jungleboyj> anteaya: I am fine with that.
16:59:42 <jungleboyj> Thanks.
17:00:06 <thingee> thanks everyone
17:00:07 <winston-d> anteaya: thx
17:00:11 <thingee> thanks anteaya
17:00:14 <thingee> #endmeeting