Tuesday, 2015-04-28

*** openstack has joined #openstack-relmgr-office00:08
*** dims has joined #openstack-relmgr-office00:27
*** dims has quit IRC01:46
*** dims has joined #openstack-relmgr-office02:47
*** dims has quit IRC02:52
*** asalkeld has quit IRC04:18
*** asalkeld has joined #openstack-relmgr-office04:30
ttxredrobot: RC2 done @ https://launchpad.net/barbican/+milestone/kilo-rc207:03
asalkeldttx: do we have a 1-1?07:50
ttxasalkeld: we do07:51
ttxI'll grab coffee first07:51
ttxmeet you in 507:51
ttx#startmeeting ptl_sync07:56
openstackMeeting started Tue Apr 28 07:56:46 2015 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.07:56
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.07:56
openstackThe meeting name has been set to 'ptl_sync'07:56
ttx#topic Heat07:56
ttxasalkeld: o/07:56
asalkeldare we looking at bugs that might cause another rc?07:57
ttxasalkeld: so at this point we only respin in case of a major regression, blatant security hole or stuff that can't be fixed post-release07:57
asalkeldI don't see any like that07:57
ttxDo you have anythign that meets that description there ?07:58
asalkeldnot yet07:58
ttxhttps://bugs.launchpad.net/heat/+bug/1446252 sounds like a typical bug07:58
openstackLaunchpad bug 1446252 in heat "Can't rollback update with nested stack in progress" [High,Fix committed] - Assigned to Zane Bitter (zaneb)07:58
ttxcan land in stable/kilo post-release alright07:58
ttxQuick look at proposed stable/kilo things07:59
ttxhttps://review.openstack.org/#/c/177602/ is interesting07:59
ttxis it something we can change at any time ?08:00
asalkeldttx: it is not a security issue08:00
ttxRight, this SHA1 is not actually brute-forceable enough08:01
asalkeldso yeah, can be done any time08:01
ttxok, looks like you're good (which is good news, since some others are not in such good shape)08:01
asalkeldit's just to make a dynamic version number08:01
ttxYou should work on release notes on the remaining time08:01
asalkeldok will do08:02
ttx#info At RC2, no need for a RC3 at this stage08:02
ttxWe need release notes completed by Thursday when we unleash the thing to the world08:02
ttxWe usually have more time to work on them but we were late this cycle08:02
ttxso the sooner the better (as you get closer you'll get edit conflicts)08:03
ttxasalkeld: questions?08:03
asalkeldttx: no all good08:03
ttxasalkeld: unless I hear from you I'll retag the RC2 as final on release day and play LP tricks08:03
asalkeldsounds good08:03
ttxawesome, thanks !08:04
asalkeldi'll let you know if there is something critical08:04
asalkeldbut i haven't seen any08:04
ttxasalkeld: right08:04
ttxthe point is, we'll find significant bugs next week too08:04
ttxso the question at this stage is... can this wait08:05
asalkeld... yes we can08:06
*** zz_johnthetubagu is now known as johnthetubaguy09:17
ttxjohnthetubaguy: o/ talk to you in a minute09:18
johnthetubaguyttx: ack09:20
ttx#topic Nova09:20
ttxSo we have a RC3 window opened, and it's not that easy to close with a busy gate09:20
ttxWe opened it for https://bugs.launchpad.net/nova/+bug/144807509:21
openstackLaunchpad bug 1448075 in OpenStack Compute (nova) kilo "Recent compute RPC API version bump missed out on security group parts of the api" [Critical,In progress]09:21
ttxwhich didn't merge in master yet09:21
ttxand is failing tests again on the retry09:21
ttxso I'm a bit worried we won't be able to close this one today09:22
ttxOne of the reasons I don't want to consider anything not merged in master for RC09:22
johnthetubaguyoh dear, agreed with must be merged in master09:23
johnthetubaguytaking a look09:23
ttxAt this point on current load, that patch won't merge in the next 14 hours09:23
ttxwhich places the RC3 closing, at best, at 26 hours09:24
ttxnot sure we can afford that09:24
johnthetubaguyah, sounds like gate has gone unstable :(09:24
johnthetubaguyaweful timing, as ever with these things09:25
ttxwell, I don't agree. We shouldn't depend on last week for anything09:25
johnthetubaguyvery true09:25
ttxRCs from last week should have been ok09:25
ttxSo I blame lack of test coverage for things that are considered RC09:25
johnthetubaguythats totally fair09:26
ttxanyway, two things trying to slip in the same RC window09:26
johnthetubaguyI have schedued a session for this09:26
openstackLaunchpad bug 1447132 in OpenStack Compute (nova) kilo "nova-manage db migrate_flavor_data doesn't do instances not in instance_extra" [High,New]09:26
ttxand bug 144474509:26
openstackbug 1444745 in OpenStack Compute (nova) "Updates RPC version alias for kilo" [High,Fix committed] https://launchpad.net/bugs/1444745 - Assigned to Alex Xu (xuhj)09:26
ttxThat last one is concerning, sounds like something that should be done prerelease09:26
ttx(but hasn't been)09:27
johnthetubaguyyeah, thats the one I am worried about09:27
ttxBoth are actually closer to merging than 144807509:27
ttxsince they are in master already09:27
johnthetubaguybasically, it causes us some upgrade issues for liberty, but I think we could work around it09:27
ttxso two options09:27
ttx1/ cancel RC3 while nothing has merged in it yet09:28
ttx2/ Do RC3 (in which case we could add both bugs in)09:28
ttxI'm leaning toward doing it with at least 1448075 and 1444745 in09:29
ttxNot totally sure about 144713209:29
ttxThat probably means closing RC3 tomorrow09:29
ttxwhich means it won't see any testing09:29
ttxso this fixes must be 100% safe09:29
ttxand I have a hard time understanding what those RPC things actually change09:30
ttxwhat's your take on the options ?09:30
johnthetubaguyso my take is, its all about this one: https://bugs.launchpad.net/nova/+bug/144807509:30
openstackLaunchpad bug 1448075 in OpenStack Compute (nova) kilo "Recent compute RPC API version bump missed out on security group parts of the api" [Critical,In progress]09:30
johnthetubaguythats the only real trigger for RC309:31
ttx...and we just added another hour to the lead in, thanks to a patch that was top of line and -2ed09:31
ttxNow at 12:4209:31
johnthetubaguyif that doesn't make it, then RC3 doesn't matter, I think09:31
ttx12 hours and 42 minutes09:31
johnthetubaguythats kinda crazy09:32
johnthetubaguyso lets just revisit the other ones a second09:32
ttxand that's European morning, so as low as it will be09:32
johnthetubaguybug 144474509:32
openstackbug 1444745 in OpenStack Compute (nova) "Updates RPC version alias for kilo" [High,Fix committed] https://launchpad.net/bugs/1444745 - Assigned to Alex Xu (xuhj)09:32
johnthetubaguythat one can be backported09:32
johnthetubaguywe usually forget to do that, it only really gets used by those using liberty anyway09:33
johnthetubaguynot too major09:33
johnthetubaguybug 144713209:33
openstackbug 1447132 in OpenStack Compute (nova) kilo "nova-manage db migrate_flavor_data doesn't do instances not in instance_extra" [High,New] https://launchpad.net/bugs/144713209:33
johnthetubaguythis is a utility that helps force the database to upgrade the data09:34
johnthetubaguynow we plan to force people to run that before some point in liberty, so we can drop the compat code09:34
johnthetubaguythat could be first step during a liberty upgrade, if it really has to be, so we could backport that OK, possibly09:34
johnthetubaguyits fairly risk free, as its changing a util not core code09:35
johnthetubaguyI would like to jam that in, if we had time, but we don't, so I am OK without that I think09:35
ttxSo... how about we cancel the RC3 until we know if we have a chance to make one09:35
ttxWe get that other fix in master, and once it's in, we see what the lead time is for the stable/kilo patch09:36
johnthetubaguyttx: OK, so we reopen RC3 if/when bug 1448075 lands09:36
openstackbug 1448075 in OpenStack Compute (nova) kilo "Recent compute RPC API version bump missed out on security group parts of the api" [Critical,In progress] https://launchpad.net/bugs/144807509:36
ttxif it lands and the lead time on the gate still lets us do RC3 before Thursday09:36
johnthetubaguyworst case, we give are selves some more compat code to keep during liberty, I think we can keep just those methods, but I need to check09:36
johnthetubaguyttx: yes true09:37
ttxso let's keep an eye on that patch (https://review.openstack.org/#/c/177186/) and gate queue status today09:38
ttxjohnthetubaguy: we can talk at the end of the day09:39
johnthetubaguyttx: so I think the mistake we made was to be way too late trying to raise the compute RPCAPI, its a long time since we did that, and turns out we didn't really remember everything that was needed09:39
ttxIn the mean time I'll kill the RC3 window09:39
johnthetubaguyttx: OK, thanks09:39
johnthetubaguyttx: so we have a first draft of Nova summit sessions in an etherpad now09:39
johnthetubaguyttx: waiting for cross project ones, and finding session leaders before we upload all of those to sched09:40
johnthetubaguyttx: thanks sorry for the delay, stupid trains09:42
ttxjohnthetubaguy: oh, we have option 3 too. Merge stuff in kilo that are not in master yet09:42
ttxNot sure that's a good idea though09:42
johnthetubaguyttx: yeah, I am not sure I like that idea09:42
ttxok, RC3 deleted for now09:43
johnthetubaguyttx: tahnks09:43
ttxjohnthetubaguy: I'll let you explain the situation to the nova crowd09:45
johnthetubaguyttx: will do09:45
ttxWe could ask peopel to stop approving things, but I know it's useless09:46
ttxjohnthetubaguy: oh, and while you wait on gate... https://wiki.openstack.org/wiki/ReleaseNotes/Kilo09:46
ttxWe need those done by tomorrow09:47
ttxthe earlier, the better to avoid edit conflicts09:47
johnthetubaguyah, by tomorrow, oops, yes09:48
johnthetubaguyits on the agenda for the meeting on thursday, which is no good09:48
johnthetubaguywill push on that09:48
ttxyeah, at that point release will be out already09:49
*** dims has joined #openstack-relmgr-office10:12
ttxgate lead-in down to 4h55min11:38
*** dims has quit IRC11:44
*** dims has joined #openstack-relmgr-office11:51
*** eglynn has joined #openstack-relmgr-office11:58
eglynnttx: knock, knock ... ready when you are12:00
eglynnttx: re. the ceilometerclient stable/kilo release, I need to discuss that a bit with gordc12:01
eglynnttx: (on the agenda for our call at 13:30UTC today)12:01
ttxeglynn: ohai12:02
ttxeglynn: it's been published yesterday12:02
ttx#topic Ceilometer12:02
eglynna-ha, I see https://pypi.python.org/pypi/python-ceilometerclient/1.0.1412:02
ttxeglynn: so at this point we only respin in case of a major regression, blatant security hole or stuff that can't be fixed post-release12:03
ttxespecially with the gate being on fire12:03
eglynnyep, got it12:03
ttxlooking at ceilometer, you seem to be on the good side12:03
eglynnyep, nothing major on the horizon12:03
ttxEven https://wiki.openstack.org/wiki/ReleaseNotes/Kilo#OpenStack_Telemetry_.28Ceilometer.29 seems to be in good shape12:04
* ttx gives out the "Good Citizen" badge to Ceilometer team12:04
eglynnttx: LOL :) ... finally!12:04
ttxeglynn: that doesn't mean you get the "webscale" badge :P12:05
eglynnttx: touche!12:05
ttxBut yeah, I appreciate you guys not being late :)12:05
ttxso, no remark on my side. Will retag RC2 to final on release day, unless you scream before then12:06
eglynncool, sounds good!12:06
ttxHave a nice day!12:06
eglynnyou too, thanks for your time!12:07
ttxSergeyLukjanov: ready when you are12:08
ttx#info No RC3 on the horizon12:08
ttx#topic Sahara12:08
*** johnthetubaguy is now known as zz_johnthetubagu12:11
ttxwell, no Sergey12:25
openstackRemoving item from minutes: <ircmeeting.items.Topic object at 0x8cc3350>12:25
ttxdhellmann: you there ?12:25
SergeyLukjanovttx, hey12:25
SergeyLukjanovttx, sorry for the delay12:25
ttx#topic Sahara12:25
dhellmannttx: here, but take SergeyLukjanov first I'm in the middle of a release for oslo.messaging12:25
ttxSergeyLukjanov: as I told eglynn, so at this point we only respin in case of a major regression, blatant security hole or stuff that can't be fixed post-release12:26
ttxespecially with gate not behaving12:26
SergeyLukjanovso, no new issues in Sahara, we're ready for release12:26
ttxright, can't spot any such issues in shara backlog12:26
ttxso we are goot to go, will retag on Thursday as final12:26
SergeyLukjanovand I think we already fully finished testing and switched to L dev12:26
ttxSergeyLukjanov: nice. Will you be around to push the "other" shara pieces around release time ?12:27
SergeyLukjanovttx, yup12:27
ttxSergeyLukjanov: I can do Sahara in the EU morning, so ping me then12:27
SergeyLukjanovttx, okay12:27
ttxSergeyLukjanov: you need to work on https://wiki.openstack.org/wiki/ReleaseNotes/Kilo#OpenStack_Data_Processing_service_.28Sahara.2912:28
ttxneeds ot be finalized before EOD tomorrow12:28
ttxwell, not "finalized"12:28
ttxbut at least be made presentable12:28
ttxso people reading them don't feel it's WIP12:28
SergeyLukjanovttx, ack, we have a draft already, will finalize it tomorrow EU morning12:28
ttxthat is all12:28
ttxSergeyLukjanov: have a great day!12:29
ttx#topic Oslo12:29
SergeyLukjanovwhat's about client release from stable/kilo?12:29
SergeyLukjanovwith fresh requirements12:29
dhellmannSergeyLukjanov: we can't change the requirements of stable branch clients12:29
openstackRemoving item from minutes: <ircmeeting.items.Topic object at 0x8cac710>12:30
SergeyLukjanovI mean https://github.com/openstack/python-saharaclient/compare/0.8.0...stable/kilo12:30
ttxSergeyLukjanov: is there any critical bug that would warrant a kilo point release ? Or could you just make a liberty release ?12:30
SergeyLukjanovwe have requirements synced to stable/kilo branch of python-saharaclient12:30
SergeyLukjanov(Kilo release reqs)12:30
SergeyLukjanovdo we need to have a release in this branch that includes them?12:30
ttxSergeyLukjanov: yeah, I don't think that is needed12:31
SergeyLukjanovno bugs12:31
dhellmannSergeyLukjanov: releasing an update of the library with requirements changes would require changing the minor version of the library, which would put it outside of the stable branch release series12:31
dhellmannyou should probably revert that commit12:31
dhellmannthat way when you have a bug fix later, you can release safely12:31
SergeyLukjanovdhellmann, in master we have 0.9.0 and in stable/kilo the latest one is 0.8.012:31
ttxSergeyLukjanov: also I wouldn't backport random fixes to the lib branch, that sounds overkill12:32
dhellmannthe old minimums without caps are compatible with the stable branch12:32
SergeyLukjanovokay, so, no need to release client from stable branch12:32
ttxSergeyLukjanov: confirmed12:32
ttx#topic Oslo12:32
ttxdhellmann: so oslo.messaging release was my only topic12:33
dhellmannI just tagged it, and I'm looking for my sandbox with the script for generating the release note mail12:33
ttxok, so I think we are good. Maybe we can tag stable/kilo oslo-incubator12:33
ttxand release too12:34
dhellmannyes, let's go ahead and do that12:34
ttxi.e. I ca push a 2015.1.0 tag and cut stable/kilo from same commit12:34
dhellmannyou mean branch, not tag, right?12:34
dhellmannyeah, ok, both12:34
ttxhistorically there was a tag too12:34
ttxnot that it changes anything, but consistency ftw12:34
dhellmannright, I was a little behind and meant for the stable/kilo name12:34
dhellmannit should be safe to do that from master, let me check12:35
*** mordred has quit IRC12:35
*** mordred has joined #openstack-relmgr-office12:35
dhellmannI have master as 0db1430757e1a33de7d3a20f4dd7048b4374d024 and that should be OK12:36
ttxack that is what I have12:36
ttxall set12:38
ttxI think we are kilo-done as far as Oslo is concerned12:38
ttxdhellmann: anything on your mind ?12:38
dhellmannwe may want to talk about centralizing library releases, since the semver thing continues to be a source of confusion12:39
ttxyes, I have a working session oin RelMgt track for that12:39
dhellmanndid you see lifeless' post on requirements management?12:39
ttxI'd like to take a page off oslo book and see how wer can reorg the "release team" to do that12:39
ttxdhellmann: still on my todo12:40
dhellmann#link https://rbtcollins.wordpress.com/2015/04/28/dealing-with-deps-in-openstack/12:40
dhellmannyes, "central" would need to include several people12:40
ttxI fear that's the only way to enforce some amount of process12:41
ttxThe trick is to do it without getting in the way12:41
dhellmannat least until we have enough community knowledge, which I fear we won't be able to build until we stop tweaking the process :-)12:41
ttxbut the current process inconsistency is not really looking good anyway12:41
dhellmannI'm thinking of ways to automate the process, but it keeps coming back to having the signed git tags12:42
ttxand as we rely more and more on lib stuff... starts to become a clear issue12:42
ttxdhellmann: if it all boils down to having a way to review tags.... we could push for that support in Gerrit12:42
-openstackstatus- NOTICE: Gate is experiencing epic failures due to issues with mirrors, work is underway to mitigate and return to normal levels of sanity12:42
*** ChanServ changes topic to "Gate is experiencing epic failures due to issues with mirrors, work is underway to mitigate and return to normal levels of sanity"12:42
ttxI like mordred's announcements.12:43
dhellmannyeah, I have no idea what would be involved in making it possible for gerrit to review tags. My thought was to have a file that lists the tags, and review that, and then have a bot apply the tag after the review12:43
ttxanyway, meat for that discussion12:43
dhellmannbut we would have to trust the bot's gpg signature, which I suppose isn't a huge issue12:43
ttxdhellmann: https://groups.google.com/forum/#!topic/repo-discuss/VV_xj6ftvoY12:44
dhellmannI need to learn more about gpg's interaction with git12:44
dhellmannoh, interesting, I'll put that on my reading list12:44
ttxwell, it's just other people complaining about that feature gap12:44
ttxnot really interesting reading12:44
ttxjust proving it's not completely an openstack thing12:45
mordreddhellmann: I would like very  much to get someone to add tag submission to gerrit directly12:45
mordredit really just takes someone being willing to write java12:45
ttxmordred: I'll write Haskell before I'm back to Java code12:46
mordredmaybe we can convince mirantis to let SergeyLukjanov do it - he likes java12:46
* SergeyLukjanov reading12:46
ttxAlthough writing it is less painful than trying to package it, trust me.12:46
ttxdhellmann: so all that is left of https://etherpad.openstack.org/p/the-big-thaw is the external dep pinning on stable/kilo12:47
ttxdhellmann: I think we can postpone that until YVR12:47
ttxno point in doing that if we rebuild a new house of cards12:48
dhellmannyes, I would like to put that off as long as we can12:48
SergeyLukjanovmordred, I like JVM, not Java, I've been writing on pure Java about 3 or 4 years ago last time ;)12:48
mordredSergeyLukjanov: :)12:48
mordredSergeyLukjanov: but you can do ANYTHING! ;)12:48
ttxSergeyLukjanov: no no you like writing Java code and even more having on Gerrit code.12:48
dhellmannSergeyLukjanov: I'll bet it's like riding a bike. You'll pick it up again in no time12:49
ttxdhellmann: it's more like riding a bike without a saddle12:49
ttxdhellmann: you can do it, but only if that's the only way to travel12:49
SergeyLukjanovgerrit in build using GWT and that's the main reason IMO to avoid looking in its code12:50
* ttx throws up in a remote corner of the office12:50
SergeyLukjanovbut from our PoV gerrit with tags support is very usefull12:50
ttxSergeyLukjanov: right!12:51
*** eglynn has quit IRC13:08
*** zz_johnthetubagu is now known as johnthetubaguy13:12
*** eglynn has joined #openstack-relmgr-office13:31
ttxmestery: ready when you are13:43
mesteryttx: Here13:44
ttx#topic Neutron13:44
ttxmestery: struggling a lot to close that RC3 window13:44
mesteryYEs, I see that https://review.openstack.org/177789 is having issues in the check queue13:44
ttxbut at least the change is in master13:44
ttxso I still hope we'll get it through13:45
ttxbut shows the danger of late respins13:45
ttxyou're at the mercy of gate weather13:45
mesteryTotally agree, but on the other hand, I'm glad testing found these issues, so :)13:45
ttxwell, I would even more glad if automated testing found those issues 2 months ago13:45
mesteryCompletely agree13:46
ttxanyway, I think we should get it in by EOD13:46
ttxand we have patches merged for RC3 already so backpedaling is not really a good option13:46
ttxso... full speed ahead!13:46
ttxhttps://wiki.openstack.org/wiki/ReleaseNotes/Kilo#OpenStack_Network_Service_.28Neutron.29  looks in good shape13:47
mesteryI've had people working on it, I'll go through it again and update as needed13:47
ttxmestery: if by EOD we don't have that last patch in we have two choices13:48
ttxwe can delay RC3 until tomorrow (leaving little time to catch weird errors like corrupted tarballs)13:48
ttxand we can cut without that fix in13:49
mesteryWell, lets just see if we can get that in13:49
mesteryAnd not have to look at those options13:49
mesteryThough I agree, those are the options should we not succeed13:49
ttxfull speed ahead!13:49
ttxsuffice to say only really odd stuff will trigger a RC addition at this point13:50
mesteryttx: Just curious, how many other RC3s are in the works? Are we the only one?13:50
ttxThere was a Nova one but the fix isn't in master yet so I cancelled it13:50
mestery\++ to that13:50
ttxwe may revive it if the fix finally makes it, since it's a pretty critical thing13:50
ttxbut in the mean time we wait13:51
ttxthere may be a swift one13:51
ttxonly news so far13:51
* mestery puts on his zuul monitoring glasses for the day13:52
ttxthe gate os completely desintegrating before my eyes13:52
mesteryI know :(13:52
mesteryLots of red on there13:52
ttxlet's hope they figure it out13:52
ttxat least we cut on the lead time. We seem to fail faster now13:52
mesteryPositive spin I guess13:53
ttxthis morning the lead time was 15 hours13:53
ttxso it took 15 hours to fail13:53
mesteryIf it was like that now, we'd just as well spin RC3 without that last one.13:53
ttxOK, I'll be talking to you later13:54
mesteryYes, talk to you later13:54
*** dims has quit IRC14:15
*** dims has joined #openstack-relmgr-office14:15
ttxnikhil_k_: o/14:20
nikhil_k_ttx: hi14:21
ttx#topic Glance14:21
ttxso at this point we only respin in case of a major regression, blatant security hole or stuff that can't be fixed post-release14:21
ttxespecially with the gate on fire14:21
ttxnikhil_k_: do you have anything that fits that description14:21
nikhil_k_makes sense14:21
ttxI don't see anything in the kilo-rc-potential list or the proposed backports14:22
ttxnikhil_k_: in the remaining time (before tomorrow EOD) you should fill out https://wiki.openstack.org/wiki/ReleaseNotes/Kilo#OpenStack_Image_Service_.28Glance.2914:27
ttxso that it's ready for release day14:27
nikhil_k_ttx: all three?14:27
nikhil_k_Key New Features, Known Issues and Upgrade Notes ?14:27
nikhil_k_just trying to make sure if ground is covered14:28
ttxYes. If you don't have anything to say in the latter sections, just remove them14:28
ttxIf you have significant issues that won't get fixed in release, good idea to mention them in "Known Issues"14:28
nikhil_k_ttx: ok. I don't have anything else for now in Kilo.14:28
ttxall good14:29
nikhil_k_ttx: so about the upgrade notes14:29
nikhil_k_ttx: we have tried to apply as many UpgradeImpact flags as possible.14:29
nikhil_k_Where do those flags go?14:29
nikhil_k_or you get some notification to include them if misses?14:30
ttxThey don't automatically go anywhere. But you should be able to grep them in the git log and mention the most significant glitches14:30
nikhil_k_I hoped some digest is already created14:30
ttxThey are supposed to help in redacting that section14:30
ttxnot that I know of14:30
nikhil_k_anyways, grepping is always fun14:30
ttxnikhil_k_: OK, so unless we talk again, I'll be releasing RC2 as final on Thursday. You get those release notes filled before then.14:31
*** notmyname has quit IRC14:31
ttxnikhil_k_: talk to you later!14:33
ttxthingee: around?14:33
thingeettx: hi14:33
ttx#topic Cinder14:33
thingeebug came up with config generator. Unfortunately nova also has it14:34
*** notmyname has joined #openstack-relmgr-office14:34
openstackLaunchpad bug 1447380 in OpenStack Compute (nova) "wrong cinder.conf.sample generation: missing directives for keystone_authtoken (at least)" [Critical,In progress] - Assigned to John Griffith (john-griffith)14:34
ttxthingee: that was fixed in RC2 for Cinder right14:34
ttxthingee: Nova dismissed it as being broken for a long time14:34
thingeeok sorry missed that on there side.14:35
nikhil_k_(sorry, got disconnected from IRC. talk to you later)14:35
ttxas I told nikhil; at this point we only respin in case of a major regression, blatant security hole or stuff that can't be fixed post-release14:35
ttxI don't see any such things in the pipe for Cinder14:35
ttxAlso the gate is on fire so we can't really respin anything at the moment14:35
thingeeI've been watching the incoming bugs pretty closely and caught up. nothing important came in14:35
* thingee throws a cup of water at it14:36
ttxSo I'll retag RC2 as final on Thursday, unless we speak again (or you send me an email)14:36
ttxin the mean time, you should apply final polish to https://wiki.openstack.org/wiki/ReleaseNotes/Kilo#OpenStack_Block_Storage_.28Cinder.2914:36
thingeettx: btw, documentation with release note items. oh my :)14:36
thingeeso happy about this14:37
ttxthingee: anything else ?14:37
thingeegit log --grep=UpgradeImpact --since=6.month ... I think that does it?14:38
thingeenikhil_k_: ^14:38
nikhil_k_nice, thanks14:38
thingeethat's it for me. gearing up for sessions and liberty specs :)14:38
thingeettx: thanks14:38
ttxawesome, ttyl14:38
ttxdavid-lyle: hi!14:38
david-lylettx: o/14:46
ttx#topic Horizon14:46
ttxAs I tell everyone today... at this point we only respin in case of a major regression, blatant security hole or stuff that can't be fixed post-release14:46
ttxespecially since the gate decided to burn all day14:46
ttxLooking at Horizon I see one thing that could have qualified14:46
openstackLaunchpad bug 1447288 in OpenStack Dashboard (Horizon) "create volume from snapshot using horizon error" [Critical,In progress] - Assigned to Masco Kaliyamoorthy (masco)14:47
ttxBut since it's not even fixed in master yet, I  think that will have to live as a known release bug and get fixed post-release14:47
david-lyleI don't think that's worth a new spin14:47
ttxdavid-lyle: at least not a very late spin.14:47
ttxdavid-lyle: ok, so RC2 is still good to go at this point14:48
david-lylewe can backport for the next stable kilo release14:48
ttxTime to focus on https://wiki.openstack.org/wiki/ReleaseNotes/Kilo#OpenStack_Dashboard_.28Horizon.2914:48
ttxThat should be completed before EOD tomorrow, since the release will happen before you get up on Thursday14:49
david-lylewill do14:49
ttxUnless we talk again, will retag RC2 as final on release day. Thanks for your help!14:49
david-lylethank you14:49
*** johnthetubaguy is now known as zz_johnthetubagu15:22
-openstackstatus- NOTICE: gerrit has been restarted to clear an issue with its event stream. any change events between 14:43-15:30 utc should be rechecked or have their approval votes reapplied to trigger jobs15:37
ttxmorganfainberg: around?15:48
morganfainbergTtx sure am15:48
ttx#topic Keystone15:48
ttxAs I tell everyone today... at this point we only respin in case of a major regression, blatant security hole or stuff that can't be fixed post-release15:48
morganfainbergNothing I've seen qualifies for that in keystone.15:49
ttxright me neither15:49
ttxSo I'll retag RC2 as final on thursday, unless we speak again15:49
morganfainbergSounds good.15:49
morganfainbergRelease notes are under way15:49
ttxYep you should complete those by EOD tomorrow15:50
ttxsince the release will be out by the time you get up on thursday15:50
ttxso you can wake up to campagne15:50
morganfainbergI will work on them today. But I am unavailable all tomorrow.15:50
ttxmorganfainberg: ok, so TODAY. Or delegate15:50
morganfainbergWill be traveling. I shall delegate as well to another core.15:50
morganfainbergTo finalize at our meeting.15:50
ttxThanks for everything !15:51
morganfainbergYay Kilo release!15:51
ttxI'll be seeing you soon.15:51
ttxnotmyname: around now if you are15:51
notmynamettx: I'm here15:51
ttx#topic Swift15:51
ttxnotmyname: did you come to a conclusion on what we discussed yesterday ?15:52
notmynamelooking at it right now15:52
*** ChanServ changes topic to "OpenStack Release Managers office - Where weekly 1:1 sync points between release manager and PTLs happen - Logged at http://eavesdrop.openstack.org/irclogs/%23openstack-relmgr-office/"15:53
ttxso for the record, we may consider a RC3, but we'll know more later today15:58
notmynamealso, I've started working on the release notes wiki page15:59
notmynameand hope to get to summit scheduling later this week15:59
ttxok, as far as release notes go, try to get them completed by EOD tomorrow16:00
ttxRelease will happen before you get up on Thursday16:00
notmynamewhat is the release time?16:00
notmynameor is there a set time?16:00
ttxI starttagging stuff at 0800 UTC and usually finish around 13:00 utc16:00
ttxPR goes off at 14:00 utc I think16:01
ttxbut I just send email when I'm done16:01
ttxAlright, I'll be talking to you later today on that RC316:01
ttxdevananda: raedy when you are16:01
ttxhmm, actually I'll be back in 10min16:04
devanandattx: mmkay16:05
devanandai'll be here16:05
ttx#topic Ironic16:16
ttxdevananda: o/16:17
ttxdevananda: As I tell everyone today... at this point we only respin in case of a major regression, blatant security hole or stuff that can't be fixed post-release16:17
ttxI see no such thing in Ironic kilo-rc-potential or proposed backports16:17
devanandattx: nor do I16:17
devanandaone of those seems to be a bug that affects a lot of projects16:18
devanandasome of htem have done backports?16:18
devanandaso I'm unclear about it16:18
ttxwhich one ?16:18
openstackLaunchpad bug 1384379 in OpenStack Compute (nova) "versions resource uses host_url which may be incorrect" [High,In progress] - Assigned to shihanzhang (shihanzhang)16:18
ttxNo they don't16:18
devanandaah. or they fixed it earlier in the cycle16:19
ttxThat leaves https://wiki.openstack.org/wiki/ReleaseNotes/Kilo16:19
ttxyou should add a Ironic section to that16:19
ttxand fill it before EOD tomorrow16:19
devanandahow's that look?16:19
ttxLooks good16:19
ttxmaybe just copy/paste16:20
devanandacool, will do now16:20
devanandai think that's it :)16:20
ttxa few TODOs still in there16:20
devanandaooh. right.16:20
devanandawill do once those are done :)16:20
ttxAlso might want to map to the other format, but I don't mind people being original there :)16:21
ttxOK, so unless we speak again I'll retag RC2 as final on release day and make things happen in LP and tarballs and stuff16:21
ttxdevananda: ttyl !16:22
devanandattx: cheers!16:22
ttxWe seem to be out of SlickNiks16:22
ttxlet's wait a few16:22
ttx#topic Trove16:22
ttxFor the record, nothing critical seems to have crept up there16:23
ttxSo that leaves us with a RC3 for Neutron, and potential ones for Swift and Nova.16:23
ttxmestery: that damn change is really not playing ball, failing again in check16:26
ttxthat said the gate is now empty so retries might be an option16:26
mesteryttx: Lets give it anotehr spin, I noticed the same.16:26
mesteryttx: But it is being difficult, this time failing in some random way in the functional job unrelated to the change.16:26
ttxzz_johnthetubagu: given that gate seems to be more amenable now, I'm willing to consider the Nova RC3 if https://review.openstack.org/#/c/177186/ makes it to master over the next hours16:27
sdaguemestery: what's the pass rate of the neutron functional job? I've seen that failing a lot in the gate16:28
mesterysdague: One second, in another thread I'm unwiding16:29
ttxOK, I'll be abck in a couple of hours to take the temperature on those RC3s again16:30
openstackMeeting ended Tue Apr 28 16:30:17 2015 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:30
openstackMinutes:        http://eavesdrop.openstack.org/meetings/ptl_sync/2015/ptl_sync.2015-04-28-07.56.html16:30
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/ptl_sync/2015/ptl_sync.2015-04-28-07.56.txt16:30
openstackLog:            http://eavesdrop.openstack.org/meetings/ptl_sync/2015/ptl_sync.2015-04-28-07.56.log.html16:30
mesterysdague: http://goo.gl/nzlfV716:31
mesterysdague: Looks like it spiked a bit but has come back down since last week,.16:32
sdagueyeh, still kind of high, I'd expect the per project jobs to be shooting for much lower fail rates16:32
sdaguegiven the smaller number of moving parts16:32
ttxzz_johnthetubagu: bah, the critical Nova RC3 change fails again in check16:34
mesterysdague: Agreed16:35
ttxmestery: I trust you will recheck it when it finally fails ?16:38
mesteryttx: I am on it. Go and enjoy your evening and we'll see what happens with it on it's next time through the loop :)16:39
sdaguettx: so... keystone middleware is basically spewing hundreds of warning lines all over the place which seems unhelpful16:42
sdagueit's actually making it really hard to figure out where real issues might be for these fails16:42
sdaguemorganfainberg: ^^^16:43
morganfainbergsdague: ack16:44
morganfainbergsdague: stable or 1.6.x?16:44
sdaguehttp://logs.openstack.org/86/177186/3/check/check-tempest-dsvm-full/38a8915//logs/screen-c-api.txt.gz?level=TRACE#_2015-04-28_16_15_09_574 - kilo16:44
sdagueI think both16:44
morganfainberglet me check wondering if something else changed16:45
sdagueso, I brought this up somewhere at least a week ago, and I guess it got lost16:45
sdagueright, so keystonemiddleware.auth_token has emitted 3.5 million warning messages in the last 7 days of gate runs - http://logstash.openstack.org/#eyJzZWFyY2giOiJsb2dsZXZlbDpXQVJOSU5HIEFORCBtb2R1bGU6a2V5c3RvbmVtaWRkbGV3YXJlLmF1dGhfdG9rZW4iLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzAyMzk2NTQ3MTV916:48
morganfainbergsdague: yeah looking at it now for you.16:48
sdagueapproximately 0% seem to be useful16:48
morganfainbergsdague: seeing when this was introduced so i can figure out why this needs to be a log warn16:48
morganfainbergsdague: or why it originally was marked as such16:48
morganfainbergsdague: i'm inclined to say this could be INFO16:49
sdaguehonestly, it should be deleted16:49
morganfainbergeh, mixed feelings on that16:49
sdagueI don't know anything useful that's being presented with that information16:49
morganfainbergit is useful to know a request didn't pass a x-subject-token16:49
morganfainbergerm x-auth-token16:49
morganfainbergespecially in inter-service work16:50
sdagueto an operator?16:50
sdagueon every single request?16:50
morganfainbergit's not every request16:50
morganfainbergsdague: it is only if we don't have an x-auth-token16:50
morganfainbergsdague: in the request16:50
sdagueit looks like it's in most of these requests16:50
morganfainbergsdague: like i said, looking now to see why this was introduced.16:50
sdagueand what exactly will an operator do with that information?16:51
morganfainbergsdague: ah crud middelware is pre-request id16:51
morganfainbergdebug might be the right place16:52
morganfainbergsdague: i was thinking if it showed the req-id, then you could at lease correlate people trying to access w/o a token16:52
morganfainbergbut i'm wrong. this is likely a debug statement mis-marked as warn16:52
sdagueyes, you can not, but even if you could, what would an operator be expected to do? email people and tell them what?16:52
morganfainbergsdague: if someon was complaining about XXX not working when i try and access nova boot16:53
morganfainberg"well your app isn't passing x-auth-token"16:53
morganfainbergit's an edge case16:53
morganfainberglike i said, this is likely a debug thing mismarked as warn16:53
sdagueit's a super edge case, given that we do it 3.5 million times and it results in 0 failures16:53
sdagueon it's own16:53
morganfainbergsdague wow...16:57
morganfainberggrizzly level of old code here16:57
morganfainbergsdague: so this should be downgraded to debug.16:58
morganfainbergsdague: at most.16:58
morganfainbergsdague: if not removed.16:58
morganfainbergsdague: it could have value at debug but likely not too much16:58
sdagueyeh, in it's current form it doesn't seem all that useful. Anyway, it would be good to cut new libs soonish, because this will look like a regression to people given the noise level16:59
morganfainbergsdague: we've been issuing these logs at in the same way (same message) since grizzly17:00
sdaguemorganfainberg: no, this is new17:00
morganfainbergsdague: something else has changed not providing an x-auth-token header17:00
morganfainberga bunch of requests17:00
sdagueI have definitely not seen this17:00
morganfainbergpossibly related to discovery?17:00
morganfainbergsdague: it's not middleware changing in this case trust me. it's something else has changed.17:01
sdagueit isn't in juno17:01
morganfainbergsdague: because whatever else change hadn't happened in juno. juno doesn't issue the requests w/o the x-auth-token header17:01
morganfainbergsdague: it's not a big deal filing a bug and getting this committed17:02
sdagueso... keystone client changing things?17:02
morganfainbergsdague: or other clients?17:02
sdaguewhat could change that behavior17:02
morganfainbergsdague: any client.17:02
sdaguemorganfainberg: it's multiple cross services it looks like17:02
morganfainbergsdague: so might be ksc related, might be other client changes as well17:03
sdaguewhich means keystone client is more likely culprit17:03
morganfainbergsdague: my point was just "it isn't keystonemiddleware"17:03
sdaguethe session work got pushed into all the clients17:03
morganfainbergbut the fix is clearly to just downgrade that in middleware17:03
morganfainbergso we have less spewing.17:03
sdaguewe have a bunch of keystoneclient warnings as well - http://logs.openstack.org/86/177186/3/check/check-tempest-dsvm-full/38a8915//logs/screen-n-api.txt.gz?level=WARNING17:06
sdagueI wonder if there is coupling17:06
morganfainberglet me bug jamielennox w17:06
sdagueso 2.2 million of those in 7 days17:07
morganfainbergsdague: https://review.openstack.org/#/q/Iab569a14a9f3268abef6bc36e46cf15dc16564b9,n,z17:08
sdagueyeh, basically it looks like the bulk of the WARNINGs in the system are from keystone.* components17:09
morganfainbergsdague: i'll corner jamielennox about the ksc stuff17:13
morganfainbergbut at least middleware i have reviews up and as soon as they merge i'll do the shuffle.17:13
morganfainbergso we can have them out on pypi17:14
sdaguesounds good17:14
sdagueyeh, because the keystone stuff gets consumed so many other places, I think we need a lot more care to when it starts raising ugly red flags17:14
*** eglynn has quit IRC18:00
*** eglynn has joined #openstack-relmgr-office18:30
*** mestery has quit IRC19:34
*** mestery has joined #openstack-relmgr-office19:39
ttxmestery: alright, I think NeutronRC3 is a go19:40
mesteryttx: Was just going to say the same to you :)19:40
ttxtagging in progress19:41
ttxtakes forever to clone git though19:44
mesteryI know :(19:48
*** openstackstatus has quit IRC19:52
*** openstackstatus has joined #openstack-relmgr-office19:54
*** ChanServ sets mode: +v openstackstatus19:54
*** eglynn has quit IRC19:55
sdaguettx, so with the nova rc3, we decided not to add https://review.openstack.org/#/c/176806/ to it?20:56
ttxsdague: it wasn't in John's list this morning20:57
ttxat this stage the lesser the better, but it might have been oversight20:57
sdagueyeh, that's fine, it would have been nice to have, but I didn't realize the rc3 was happening, and I was firefighting this morning20:58
ttxsdague: yeah, in the morning when we had the discussion the gate was pretty backlogged so we were pretty pessimistic on our RC3 abilities21:02
ttxhence the very conservative approach21:03
sdagueyeh, no worries21:03
ttxsdague: but now that you pushed that "alias tox=true" on nodes, it works better21:03
sdaguewell, if neutron doesn't rando kill top of gate, we'll land a bunch here shortly21:05
ttxdansmith: c21:07
ttxcross those appendage things again21:07
ttxworked well last time21:07
dansmithttx: crossing21:08
dimssdague: an actual review with that alias?21:09
dansmithdims: you're too easy21:09
dimsi went looking for it dansmith!21:09
dansmithsdague: looks like that test might be hung?21:10
dansmithoh nevermind21:10
dansmithread the timestamp wrong21:10
dansmithit sure has been taking its sweet time21:10
dansmithttx: I'm going to run off for a bit, but rest assured my appendages will remain crossed until I return21:18
ttxlooks like we have a winner again21:30
ttxdansmith: RC3 on its way, thanks for your help21:51
dansmithttx: likewise! :)21:51
dansmithttx: shall I unalias tox now or do you have more to wait for? :P21:51
ttxdansmith: we have swift coming up21:52
dansmithokay, better leave it for a bit longer then :)21:52
*** dims_ has joined #openstack-relmgr-office23:29
*** dims has quit IRC23:31

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!