21:01:27 #startmeeting project 21:01:27 Meeting started Tue Oct 1 21:01:27 2013 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:28 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:28 o/ 21:01:30 The meeting name has been set to 'project' 21:01:31 o/ 21:01:32 #link http://wiki.openstack.org/Meetings/ProjectMeeting 21:01:39 #topic General stuff 21:01:52 We are still struggling with RC1 publication, mostly due to the bugs creating gate instability and slowing down merges 21:02:02 If this continues we might need extraordinary measures 21:02:13 Like limiting APRV to RC1-targeted fixes (and gate fixes), at least until we get a few RC1s out 21:02:30 but there have been some improvement today, so we'll revisit the need for that later this week 21:02:43 and we should have glance and keystone out anytime now 21:02:59 sdague: Could you give us a quick status on the 3 top gate-f*cking offenders ? 21:03:18 o/ 21:03:25 so they are moving around a little, but here is the latest elastic recheck results - http://paste.openstack.org/show/47784/ 21:03:58 realistically we had #1224001 go away mostly this morning 21:04:13 which was the neutron bug that was masking everything else yesterday 21:04:22 there is one fix in flight with potential for the simultaneous read thing ... https://review.openstack.org/#/c/49210 21:04:25 sdague: bug is still there but it doesn't affect non-neutron that much, right ? 21:04:29 it's gone away because we stopped running neutron under tenanant issolation 21:04:44 which means basically neutron is getting extra kid gloves in the gate 21:04:58 ok 21:05:16 sdague: what would be your new favorite friend ? 21:05:16 I'd say the simultaneous read issue that russellb is pointing out is in the top 3 21:05:25 markmcclain: ^^^ (my link) 21:05:41 #1229475 - what russellb linked for a review 21:05:49 * markmcclain looking 21:05:56 and #1226337 21:06:15 we don't have a bot in here that urls these? 21:06:28 sdague: #1226337 just delayed keystone RC1 cut by several hours, so it's my personal favorite 21:06:28 markmcclain: patch is a bit brute force, but we're sort of grasping here 21:06:35 bug 1226337 21:06:38 Launchpad bug 1226337 in cinder "tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern flake failure" [Undecided,Fix committed] https://launchpad.net/bugs/1226337 21:06:44 bug 1229475 21:06:45 Launchpad bug 1229475 in python-neutronclient "terminate_instance(): RuntimeError: Second simultaneous read on fileno 16 detected" [Undecided,In progress] https://launchpad.net/bugs/1229475 21:07:12 russellb: understood.. this is code that has been around for sometime, so wondering why it is suddenly causing so many failures recently 21:07:17 so 1226337 is basically a general case of volumes don't get created reliably 21:07:31 sdague: do we have an idea of how to fix that one ? 21:07:37 sdague: ttx yes 21:07:40 jgriffith: has been working on it 21:07:40 markmcclain: well, i'm nervous to merge it honestly, because we don't have a good explanation 21:07:45 sdague: ttx proposed fix in the queue 21:07:50 jgriffith: ok 21:07:57 jgriffith: ok, I thought that was just the debug fix? 21:08:00 sdague: ttx and to be specific it's the backing lun for the target isn't created 21:08:04 we've made a dent in it 21:08:04 markmcclain: all we know is that neutronclient ends up with a bad fd 21:08:20 sdague: no, that's an actual attempt to recreate the backing lun if it's missing 21:08:25 ok, great 21:08:29 as well as provide more debug info 21:08:30 so it looks like we have stuff in progress to improve the sutuation on the top 3 offenders... which might fix those 21:08:50 russellb: from a risk standpoint it can't hurt 21:08:53 sdague: is that a fair summary or do you have any issue flying under the radar ? 21:09:01 markmcclain: yeah, does *seem* safe at least 21:09:04 ttx: I think that's fine for now 21:09:15 russellb: only likely to increase request latency 21:09:24 sdague: ok 21:09:28 long term I do want to move to request too 21:09:32 annegentle, jeblair: other news from Docs/Infra programs ? 21:09:43 affecting release, that is ? 21:09:47 markmcclain: cool was wondering about that ... guess we should be quiet now :) 21:10:06 In other news, it looks like we won't have phone landlines to call in in design summit rooms in Hong-Kong. 21:10:16 If we want to hook in critical remote participants we'll have to get creative using modern stuff like "the Internet" 21:10:40 ttx: what's "Internet" 21:10:55 jgriffith: somethig they apparently have in HongKong 21:11:04 we don't normally have such modern stuff 21:11:05 world of wonder 21:11:21 OK, let's go into project-specific details now, if annegentle and jeblair don't have anything to add 21:11:37 #topic Oslo status 21:11:43 dhellmann: hi! 21:11:47 hi! 21:11:52 dhellmann is standing in for markmc 21:11:56 #link https://launchpad.net/oslo/+milestone/havana-rc1 21:12:05 One bug left before stable/havana cut: 21:12:08 Bug 1225099 21:12:09 Launchpad bug 1225099 in oslo "Encoding hard-coded in gettextutils.Message" [High,In progress] https://launchpad.net/bugs/1225099 21:12:20 we've got a 2 part fix for that 21:12:22 Impacts most other projects. Could you elaborate on the impact of that one ? Given how many projects are affected I'd like to make sure it's actually RC1-critical 21:12:50 read the bug but it just confused me 21:12:50 part 1 is to disable "lazy" translation in the projects currently using it, because the implementation is sketchy 21:13:07 part 2 is to fix the translation issue in logging, and we have a patch in process for that 21:13:17 https://review.openstack.org/#/c/49230/3 21:14:00 that fix will let us re-enable lazy evaluation, but given that this is happening so late in the cycle most of us feel more comfortable leaving it turned off and focusing on it early in icehouse 21:14:06 dhellmann: so those fixes will need to be synced in the consuming projects ? 21:14:10 especially because this touches logging, which is used everywhere 21:14:33 ttx: we will need to patch the projects to avoid configuring logging with lazy evaluation enabled (there are 2 ways that's done) 21:14:50 and then the other changes will be worked on, but don't need to be merged into the other projects 21:15:17 dhellmann: any chance we can have those patches proposed today ? some projects will block their RC1 on that quite soon 21:15:29 ceilometer comes to mind 21:15:35 heat too 21:15:40 ttx: doubtful, but early tomorrow may be possible 21:16:06 (I'm talking about the lazy-disabling patch, not the long-term fix) 21:16:09 ok 21:16:28 yes, me, too (it's the end of the day here and I have a long commute ahead of me today) 21:16:43 I'll see if some of the other folks involved can start on that, though 21:16:51 dhellmann: could you explain the impact ? How many translations use double byte stuff ? 21:17:17 that's not entirely clear, but my impression is many asian languages will be impacted 21:17:33 but *only* for logging 21:17:48 so a work-around is to not log using the locale-specific language 21:18:34 dhellmann: ok, my understanding is that the patch will be simple enough, so maybe we'll find contributors to propose them earlier and in all cases they should be reviewed fast enough 21:18:35 which is, obviously, not ideal 21:18:40 right 21:18:52 so that will not create too much of a delay 21:19:02 dhellmann: that's all I had. Questions about Oslo ? 21:19:10 dhellmann: anything you wanted to mention ? 21:19:17 nothing else from me 21:19:32 #topic Keystone status 21:19:36 dolphm: hi! 21:19:36 o/ 21:19:38 #link https://launchpad.net/keystone/+milestone/havana-rc1 21:19:44 RC1 pending merge of the icehouse version bump: https://review.openstack.org/#/c/48971/ 21:19:53 yay! 21:19:55 dolphm: luckily you don't seem to use this lazy thing 21:20:05 I'll cut the release branch and tag RC1 if no bug gets targeted to havana-rc1 when the version bump merges. 21:20:09 :) 21:20:18 which should happen during my sleep, hopefully 21:20:25 dolphm: Anything else that may affect us ? 21:20:33 nope! 21:20:53 Other news / questions about Keystone ? 21:21:28 #topic Ceilometer status 21:21:33 jd__: hey 21:21:36 yop 21:21:37 #link https://launchpad.net/ceilometer/+milestone/havana-rc1 21:21:43 Last review in gate: https://review.openstack.org/#/c/47542/ 21:21:53 Then we are affected by Bug 1225099 21:21:54 Launchpad bug 1225099 in oslo "Encoding hard-coded in gettextutils.Message" [High,In progress] https://launchpad.net/bugs/1225099 21:22:03 just for 6 hours already :) 21:22:07 jd__: could you push a patch for that, with dhellmann quick guidance ? 21:22:15 sure 21:22:30 Then we can push and approve the version bump (in setup.cfg, bump 2013.2 to 2014.1) tomorrow in our common morning 21:22:48 that would work for me 21:22:57 we should be all set for RC1 tomorrow 21:23:03 I just need to know what the patch should like since I've no idea yet :) 21:23:07 unless you have a surprise up your sleeve 21:23:21 jd__: so I'll let you sync with dhellmann now 21:23:23 Other news / questions about Ceilometer ? 21:24:09 #topic Swift status 21:24:12 hi! 21:24:15 yay swift 21:24:24 #link https://launchpad.net/swift/+milestone/1.9.3-rc1 21:24:25 notmyname: o/ 21:24:36 notmyname: How far are you ? Would be nice to get to RC1 this week for you too, leaving 2 weeks for RC testing and any necessary respin 21:24:50 from what I've seen diskfile-databasebroker-as-apis is still very much in progress ? 21:25:03 while the only review I could find earlier today about multiple-domains-in-domain-remap is abandoned 21:25:13 so I can't really tell an ETA 21:25:16 tracking 2 patches for RC: https://review.openstack.org/#/c/48538/ and https://review.openstack.org/#/c/45134/. Those are the requirements. I'd love to see some more of the DiskFile, but getting reviews has been slow (although the codes is pretty much written) 21:25:43 * ttx adds to starred list 21:25:57 notmyname: i'll ping you again when those two land 21:26:09 see where we are at that poin 21:26:09 the memcache pool got hit with gate failures, so it should be landing soon, I think 21:26:10 t 21:26:14 kk, sounds good 21:26:32 but later this week sounds definitely possible here 21:26:46 notmyname: Other news ? 21:26:51 that would be nice :-) 21:27:03 no other news from my end right now, but i have a question 21:27:11 go for it 21:27:38 are the updated global requirements (that ae now being auto added to the patch queue) required for RC, too? ie are we getting all on the same page before havana? 21:28:37 notmyname: I'm not totally sure if it's required or just nice to have 21:28:42 mordred: ? 21:28:44 jeblair: mordred: ^^? 21:29:02 what did I do? 21:29:04 i.e. should MOST projects or ALL projects have it for it to work 21:29:16 I think it's nice to have 21:29:24 ok, good to know for the review queue 21:29:26 thanks 21:29:26 o/ 21:29:29 mordred: thx 21:29:37 ttx: ok, I'm good 21:29:41 #topic Glance status 21:29:43 live from Bend, OR 21:29:44 I know swift has concerns around the lower bars - so I'd say that being in conflict with them is bad 21:29:53 markwash: o/ 21:29:54 mordred: we'll figure something out 21:29:58 #link https://launchpad.net/glance/+milestone/havana-rc1 21:30:08 RC1 pending merge of the icehouse version bump: https://review.openstack.org/#/c/49029/ 21:30:12 all done* 21:30:15 now on top of gate 21:30:17 *some bumping entailed 21:30:28 #link https://bugs.launchpad.net/glance/?field.tag=havana-rc-potential 21:30:31 those were the bumps 21:30:35 ETA 7min 21:30:40 or 6 hours, depending on the results 21:30:53 markwash: Any last-minute issue ? 21:30:53 * markwash bites nails 21:31:10 I'll cut the release branch and tag RC1 after meeting if it passes 21:31:12 there are those havana-rc-potential issues but they don't look very serious 21:31:16 tomorrow morning if not 21:31:32 if any of those rc potentials land, does that screw anything up? 21:31:37 markwash: we'll keep potentially RC2-openers in that list for regular review 21:31:53 markwash: stuff that lands post-version-bump are icehouse 21:31:58 k cool thanks 21:32:05 markwash: we can still backport them though if needed 21:32:12 Other news / questions about Glance ? 21:32:19 Nothing from me 21:32:31 markwash: you don't use the lazy translation stuff, right ? 21:32:55 I don't know what that means, so probably not? 21:33:27 we'll assume markmc did his affected analysis right :) 21:33:35 #topic Neutron status 21:33:40 markmcclain: hi! 21:33:46 hi 21:33:46 #link https://launchpad.net/neutron/+milestone/havana-rc1 21:33:59 5 RC1-targeted bugs left 21:34:13 bug 1224967: https://review.openstack.org/48261 needs core reviews 21:34:14 Launchpad bug 1224967 in neutron "port down after live migration" [Medium,In progress] https://launchpad.net/bugs/1224967 21:34:40 You are affected by common issue bug 1225099, so we'll need a patch disabling the lazy mode here too 21:34:41 Launchpad bug 1225099 in oslo "Encoding hard-coded in gettextutils.Message" [High,In progress] https://launchpad.net/bugs/1225099 21:34:54 yeah.. was just discussing with dhellmann 21:34:54 Then the last 3 things are the flaky issues affecting the gate 21:35:17 At this point we should cut RC1 as soon as the first two issues are merged and keep working on the other 3 21:35:32 and do a RC2 if we do significant progress on them 21:35:44 markmcclain: Anything else you'd like to throw at RC1 ? 21:36:21 ok.. I just want to make sure we understand the first three before release 21:36:39 agreed 21:36:47 softening the tempest checks while nice is concerning to me, but we can work towards a fix in RC2 21:36:59 Other news / questions about Neutron ? 21:37:12 nothing else from me 21:37:14 #topic Cinder status 21:37:18 jgriffith: hola! 21:37:25 #link https://launchpad.net/cinder/+milestone/havana-rc1 21:37:28 howdy 21:37:38 5 bugs left 21:37:52 bug 1230176 still needs once extra core review 21:37:54 Launchpad bug 1230176 in cinder "[taskflow] Quotas are not rolled back on failure of create volume " [High,In progress] https://launchpad.net/bugs/1230176 21:38:13 on https://review.openstack.org/#/c/49103/ 21:38:23 Then we are left with three bugs with uncertain status: 21:38:29 + the lazy thing 21:38:33 bug 1230189 21:38:34 Launchpad bug 1230189 in cinder "[taskflow] Incorrect state transition on failure of create volume API" [High,New] https://launchpad.net/bugs/1230189 21:38:44 did you make progress on that one ? 21:38:56 Not yet, but it's number is up 21:39:01 bug 1229894 21:39:02 Launchpad bug 1229894 in cinder "brick has duplicate conf entries in iser and iscsi" [High,In progress] https://launchpad.net/bugs/1229894 21:39:03 after this meeting I'll jumpon it 21:39:19 in progres, just need the review 21:39:25 review's 21:39:28 at https://review.openstack.org/#/c/48528/ 21:39:44 the error code 22 I think might be able to go away 21:39:44 jgriffith: will that solve all or just half the bug ? 21:40:03 all 3 areas for that bug are in progress 21:40:18 (I see iser in addition to iscsi) 21:40:23 there's mine, Walts and cburgess they're all in 21:40:32 ok 21:40:35 jgriffith: Mine needs to be re-based actually. 21:40:41 bug 1223469 21:40:41 cburgess: boooo 21:40:42 Launchpad bug 1223469 in cinder "Volume create tgtadmin fails with error code 22" [High,Confirmed] https://launchpad.net/bugs/1223469 21:40:51 Was waiting for Walt to get his fixed. 21:40:52 I was just looking at that and I think I can close it 21:40:53 I need his changes. 21:41:05 ttx: it turns out that my retry is working :) 21:41:10 jgriffith: what do you mean by "might be able to go away" 21:41:10 ttx: http://paste.openstack.org/show/47798/ 21:41:54 ttx: so the error still occurs but the recovery change I made seems to be working 21:42:09 ttx: at least based on queiries in logstash 21:42:17 jgriffith: so we could move that to havana-rc-potential ? 21:42:37 ttx: I think I'm just going to close it an open it back up if something different comes up 21:42:45 jgriffith: sounds good 21:43:00 jgriffith: you'll need a patch up for bug 1225099 too 21:43:02 Launchpad bug 1225099 in oslo "Encoding hard-coded in gettextutils.Message" [High,In progress] https://launchpad.net/bugs/1225099 21:43:12 maybe wait and see how ceilo fixes it 21:43:27 Other news / questions about Cinder ? 21:43:27 errmmm... TBH haven't looked at that yet but heard dhellmann talking about it I think 21:43:45 Nope... you all may be rid of me after this week :) 21:43:45 jgriffith: yes, the idea is to disable lazy translations until properly ficed 21:43:48 fixed 21:43:49 but I'm not going anywhere :) 21:44:10 I'll still be a PITA 21:44:12 jgriffith: thanks! 21:44:14 #topic Nova status 21:44:19 o/ 21:44:21 russellb: hi! 21:44:25 #link https://launchpad.net/nova/+milestone/havana-rc1 21:44:30 All in gate except: 21:44:30 looking a bit better 21:44:33 bug 1229475 21:44:36 Launchpad bug 1229475 in python-neutronclient "terminate_instance(): RuntimeError: Second simultaneous read on fileno 16 detected" [Undecided,In progress] https://launchpad.net/bugs/1229475 21:44:51 yep, still pushing hard on that one, potential patch up against neutronclient 21:45:00 and the neutron folks are jumping in to help look too 21:45:03 ok 21:45:21 and will need to dsiable "lazy" for common bug 1225099 21:45:22 Launchpad bug 1225099 in oslo "Encoding hard-coded in gettextutils.Message" [High,In progress] https://launchpad.net/bugs/1225099 21:45:31 that's the last two things 21:45:38 yeah, was just going to follow the lead of what gets done across all affected projects there 21:45:46 jgriffith: Is walter done rebasing his stuff? 21:45:49 translations in the gate 21:46:05 translations were not that stale in your case anyway 21:46:15 tomorrow seems possible, but optimistic, but should definitely be ready thursday, unless something new happens 21:46:37 I'm traveling on Thursday.Friday but will do my best syncing with you 21:46:44 oh, sorry 21:46:54 russellb: well, my faulty 21:47:08 would you want to just wait for monday at that point? 21:47:10 I thought we'd be done with RC1s by then. I should know better 21:47:18 heh 21:47:22 russellb: that conference better have decent wifi 21:47:29 ha ok. 21:47:37 so we'll still aim for this week asap then 21:47:49 yes, definitely 21:47:54 Other news / questions about Nova ? 21:48:25 #topic Heat status 21:48:29 shardy: o/ 21:48:32 o/ 21:48:33 #link https://launchpad.net/heat/+milestone/havana-rc1 21:48:46 3 bugs left, one gating 21:48:49 bug 1227901 - where are you now on this one ? 21:48:50 Launchpad bug 1227901 in heat "CFN signal URLs don't work when trusts is enabled" [High,In progress] https://launchpad.net/bugs/1227901 21:49:27 ttx: I think we've decided to go ahead and push the change, and I've reached agreement with ayoung re the tempest tests for keystoneclient 21:49:35 shardy: cool 21:49:49 then we'll be pending our common friend bug 1225099 21:49:50 ttx: I just addressed some (minor) review comments, so hopefully we'll get that patch in the gate soon 21:49:51 Launchpad bug 1225099 in oslo "Encoding hard-coded in gettextutils.Message" [High,In progress] https://launchpad.net/bugs/1225099 21:50:19 would be great to find someone in some pacific tz to push a patch disabling the lazy stuff 21:50:36 then we could cut RC1 tomorrow, maybe 21:50:38 Sure, I'll speak to stevebaker and asalkeld 21:50:59 yup, tomorrow looks possible if we can get things through the gate.. 21:51:02 shardy: i'll miss a european PTL 21:51:22 Other news / questions about Heat ? 21:51:23 ttx: Haha, I'll still be around to pass on messages ;) 21:52:02 #topic Horizon status 21:52:05 \o 21:52:05 gabrielhurley: o/ 21:52:07 We're down to five bugs, all with patches that have been approved, so we're waiting on the gate. translations are merged to both Horizon and django_openstack_auth packages. Released 1.1.3 of django_openstack_auth to PyPI with the latest. Once these last bugs are merged I'll push the version bump to gerrit and we'll cut our RC1. I'd expect that within the next 24 hours depending on the gate. 21:52:10 #link https://launchpad.net/horizon/+milestone/havana-rc1 21:52:35 * ttx looks at https://bugs.launchpad.net/horizon/+bug/1211535 21:52:36 Launchpad bug 1211535 in horizon "HorizonMiddleware class causes excessive session creation" [Medium,Fix committed] 21:52:57 ohn the django_openstack_auth fix fixed it 21:53:27 ok, looks like you might be all set for tomorrow 21:53:27 yeah. there was a change that needed to be released there, so I did that along with the translations in one fell swoop 21:53:47 tomorrow's the goal 21:54:12 Other news / questions about Horizon ? 21:54:19 I got nothin' 21:54:26 I have one 21:54:31 markwash: damn 21:54:37 go for it 21:54:43 can we make the jobs for django_openstack_auth voting? I think they all pass but most are non voting 21:54:52 absolutely 21:54:53 the jenkins gate jobs 21:54:59 at least fine by me 21:54:59 gabrielhurley: cool, I will propose that change 21:56:09 other questions ? 21:56:44 #topic Incubated projects 21:56:50 devananda, hub_cap: around ? 21:57:22 aye 21:57:24 hub_cap: Still no need for an RC2 ? 21:57:29 well......... 21:57:45 https://github.com/openstack/trove/blob/master/etc/trove/api-paste.ini#L22 21:57:54 im thinking we should correct that and backport it 21:58:03 yes 21:58:10 i think zigo_ said there was a security issue w/ teh last one 21:58:21 or the last time that was "fudged" in another project 21:58:25 yes, that was fixed in other projects 21:58:26 thx to zigo_ for finding it for us :) 21:58:38 that alone will constitute our rc2 :) 21:58:43 hub_cap: please file a bug about it 21:58:54 get it fixed in icehouse 21:59:03 i think zigo_ did let me look 21:59:04 then we'll open RC2 and backport it 21:59:16 https://bugs.launchpad.net/trove/+bug/1233305 21:59:17 Launchpad bug 1233305 in trove "Symlink attack with signing_dir = /tmp/keystone-signing-nova" [Undecided,New] 21:59:21 but first file a bug and get it fixed into master 21:59:25 ok 21:59:48 tagged 21:59:59 Any question ? 22:00:06 ill fix now and we can backport 22:00:09 nope 22:00:15 just hugs and lots of em 22:00:27 ack, talk to you tomorrow on opening that RC2 window 22:00:36 #endmeeting