Thursday, 2015-04-16

*** dims_ has quit IRC00:40
fungi174074 has finally merged (the clouds were not with us this day), so i'm just waiting for puppet to get it applied on the gerrit server00:46
notmynamettx: dhellmann: FYI, swift will need an RC2. some stuff discovered in testing today. working on patches. I want to talk either (my) tonight or tomorrow moring on what the right schedule for that is00:52
* notmyname goes on kid duty at home00:53
fungi174074 is live on gerrit now, stable/kilo branches properly locked down to release teams01:19
*** jogo has quit IRC01:24
*** dims has joined #openstack-relmgr-office01:40
*** dims has quit IRC01:47
*** dims has joined #openstack-relmgr-office02:48
*** dims has quit IRC02:53
*** dims has joined #openstack-relmgr-office05:45
*** dims has quit IRC05:50
ttxnotmyname: unless the bug creates data-loss or catastrophic failure, I'd rather have the RC1 survive a little more so that we potentially discover other issues in it. Like target a RC2 for Monday/Tuesday next week06:32
notmynamettx: oh, yes.06:32
notmynamettx: thanks for the clarification on date. that's what I was looking for06:33
ttxcheers06:33
notmynamettx: but, yes, we will need one06:33
ttxnoted06:34
ttxdhellmann: having a look, but EDGE data link seems to struggle with Gerrit page loading06:37
notmynamettx: I think I overheard that other projects were doing rc2s on monday?06:37
notmynamedoes it make it easier to do ours then too?06:38
ttxnotmyname: not necessarily. Some of them will happen earlier and some later. I prefer to stagger them based on RC1 release date06:38
notmynameok06:38
ttxbut yeah, I epxect a pack of them around Monday06:38
notmynamettx: note that one of the issues we'd like to resolve is a FFE for global requirements to bump pyeclib. we've submitted patches and I've sent a message to the ML06:40
ttxack06:40
notmynameI'm not sure what the process is on that, but any help you can give is appreciated06:40
ttxwe usually try to measure the pain for packagers at this point06:41
ttxtraveling for meetings today and tomorrow so on limited connectivity06:41
ttxsdague, fungi, jeblair: So... on https://review.openstack.org/#/c/173842/ it looks like we have the same exact failure with everything on stable/* compared to when we had proposed/*06:44
ttxI see the same errors06:45
*** dansmith has quit IRC07:04
*** dansmith has joined #openstack-relmgr-office07:08
*** dansmith is now known as Guest6767207:08
*** zz_johnthetubagu is now known as johnthetubaguy08:29
*** mestery has joined #openstack-relmgr-office08:40
*** mestery_ has quit IRC08:43
ttxor maybe that's not the same error... arh can't do anything on this spotty wifi09:19
*** dims has joined #openstack-relmgr-office09:19
ttxsdague: would welcome your eagle eyes checking that the test at https://review.openstack.org/#/c/173842/ does what it's supposed to do now that we switched everything to using "stable/kilo"09:20
*** dims has quit IRC09:24
ttxI see the same kind of errors but maybe those are expected09:27
sdaguettx: ok, just got my first cup of coffee, let me look09:41
sdagueso https://review.openstack.org/#/c/173842/ is a stable/kilo proposed change09:42
sdagueno, this looks good - http://logs.openstack.org/42/173842/1/check/check-tempest-dsvm-full/8b0b195/logs/devstack-gate-setup-workspace-new.txt.gz#_2015-04-15_22_50_15_463 trove in on stable/kilo09:43
sdaguethe clients are still maybe weird, right?09:43
sdagueok, so the thing to look for is the following 2 strings09:46
sdagueactually, no... it's simpler09:47
sdaguesearch "git_checkout"09:48
sdaguethat will show you the branches everything is set to09:48
sdaguethis looks right, I think the only thing that's maybe up for grabs is tempest-lib and a couple others09:49
sdaguelet me make a list09:49
sdaguehttp://paste.openstack.org/show/204070/09:50
sdagueso sahara-dashboard seems like it should have a branch09:51
sdagueI don't know about the ironic agent / libs09:51
sdagueand tempest lib is still a ?09:51
sdaguebut the thing is, the fall back logic will just use master, so we can branch later on those if it's a problem09:52
*** dims has joined #openstack-relmgr-office10:07
sdaguettx: so https://review.openstack.org/#/c/173924 is going to be "fun" to unwind10:36
sdagueand by fun, I mean *not fun at all*10:36
sdagueall openstack libraries have to uncap their requirements and release simultaneously10:37
sdaguethis is one of those issues with libraries getting g-r syncs10:37
sdaguewhich I don't think was anticipated10:37
*** dims has quit IRC12:14
*** dims__ has joined #openstack-relmgr-office12:17
fungiyeah, for the future we should probably branch the clients during requirements freeze and not cap master requirements entries for them?12:44
fungia lot of the original discussion around it was to introduce the caps on requirements stable/kilo after the kilo release. the concern was not releasing with capped versions, but rather protecting the stable branch with caps once the release was over12:50
fungihowever that does necessarily require client/library devs to not introduce breaking change releases during the rc period12:51
fungiso maybe something we can't bank on without taking away their tagging privileges temporarily12:51
ttxo/12:51
ttxtemporarily available12:51
ttxwaiting for my contacts to show up for meeting, may drpo any moment12:52
ttxso we should probably not have capped deps in master before the branch12:52
fungimaybe. it seems like a lesson learned now, but we might find unexpected drawbacks with that plan next time12:53
dhellmannyeah, it may have been simpler to branch and then cap :-/12:56
dhellmannttx, fungi, sdague : do you know what the job "gate-tempest-dsvm-neutron-src-python-heatclient-juno" is? that same pattern is showing up as a failure in a few places in the stable/kilo branches12:57
dhellmannttx, fungi, sdague: is that the backwards-compatibility test that we don't need any more because we're capping requirements?12:58
fungidhellmann: yeah, that's a backward-compatibility job to run master branch tip or changes proposed to master of heatclient against everything else from juno12:58
fungifrom when we needed to make sure heatclient (and others) didn't break stable/juno because they didn't have stable/juno branches12:59
dhellmannright12:59
dhellmannso we should be able to turn that job off now, I think?13:00
fungias we add stable branches to the clients/libraries, those .*-src-.*-(icehouse|juno) jobs should be removed13:00
fungithe idea is that things like clients that have to maintain compatible with the rest apis of older server releases should have functional testsuites which can be pointed at older release servers as needed to keep them backward-compatible in the ways we still care about (user-facing backward compatibility)13:01
dhellmannright, that's what I thought. Do you know if anyone is working on those functional tests?13:02
fungiseveral of the clients already have them13:02
sdaguedhellmann: it's all very project specific13:02
sdaguettx: so... the root issue, capping requirements in libraries is probably *always* the wrong thing to do13:02
dhellmannso right now we have a bunch of clients that can't land patches in their kilo branches because those test jobs fail13:03
dhellmannit's not clear if we should spend a bunch of time debugging that, or just remove the test jobs13:03
dhellmannor at least turn them off for the branch13:03
sdaguedhellmann: yeh, all the stable compat jobs are obsolete if the client in question has stable branches13:03
dhellmannsdague: maybe we should make the requirements update job "smarter"? have it look for a flag in the repo or the project list or something?13:04
sdagueI feel like we keep trying to make things smarter one patch at a time, and they just wedge in new ways13:04
dhellmannok, I'll throw together a patch to remove the jobs from the libraries that are failing right now13:04
sdagueI think realistically, g-r only should apply to top level projects, delete most of what's in projects.txt13:05
dhellmannsdague: well, if we don't make that tool smarter we have to remember not to approve requirements updates to libs in stable branches, right?13:05
dhellmannhmm, that's likely to lead to competing requirements for second-level dependencies, isn't it?13:05
sdaguedhellmann: no worse than external libraries13:06
sdagueI think the only thing we should probably enforce on libraries in their requirements is they can't cap anything13:06
dhellmannhow do we resolve the conflict when oslo.db wants a newer sqlalchemy than is in the global requirements list?13:06
dhellmannexcept hacking, and whatever other special cases we have like that13:07
sdagueI guess the enforcement would really be "libraries must be wider than gr, and not cap"13:08
sdaguedhellmann: looking at the logs on ttx's attempt to uncap master13:09
sdagueI think we need to release versions of all libraries that had caps to uncapped versions all at the same time13:09
dhellmannok, I can do that in a bit -- I have to drive in to atlanta today13:09
sdaguedhellmann: well, you can't without fungi I don't think13:09
sdaguebecause gr sync13:09
sdagueand requirements check won't let you13:10
dhellmannwe can remove the libs from the project list for a little while today -- you and I can approve that13:10
sdaguedhellmann: this is going to include all the clients as well13:10
dhellmannok13:11
dhellmannsdague: can you tl;dr why we need versions without caps?13:12
sdaguedhellmann: http://logs.openstack.org/24/173924/1/check/check-tempest-dsvm-full/6c09e06/logs/devstacklog.txt.gz#_2015-04-15_23_02_58_57513:13
dhellmannsdague: https://review.openstack.org/17437113:13
sdaguebecause lifting the cap for oslo.config brings in a version of oslo.config that many things using oslo config wedge on13:14
dhellmannsdague: ah, because we released libs while there were caps in place :-/13:14
sdaguedhellmann: those are the wrong jobs13:14
dhellmannugh13:14
sdaguestable-compat-jobs13:14
dhellmannah, I looked for the "neutron-src" part and thought that was the right macro13:15
dhellmannreworking...13:15
dhellmannsdague: updated13:22
fungishould we also remove the definition of that job template from jenkins/jobs/tempest-jobs.yaml and drop all its instantiations from jenkins/jobs/projects.yaml or save that cleanup for later?13:28
fungiif there's some desire to keep it around for now, i'll go ahead and approve 174371 as-is13:29
fungidhellmann: sdague: ^ ?13:31
sdaguefungi: so, I don't know if other things want to keep it13:32
sdagueI'd say move forward for now, and we can clean up things later13:32
fungidhellmann: aha, i think saharaclient has it added directly to their project entry in layout.yaml rather than using a project-template13:34
fungiahh, nope they just have a stale override for it in the jobs list13:36
fungiit was apparently set nonvoting for them13:36
fungiwe can clean that up later too13:36
dhellmannsdague, fungi : here's the patch to drop libraries from requirements updates so we can change their requirements and do releases: https://review.openstack.org/17438313:36
*** Guest67672 is now known as dansmith13:36
*** dansmith is now known as Guest1445313:37
dhellmannI'm going to throw my laptop in my bag and head to the city. I should be back online in a couple of hours. If you want to shepherd that patch through, and start submitting the changes to the library requirements files, I think I can use my release team powers to approve them when I'm back online13:37
dhellmannand if not then we'll hassle the ptls to do it13:38
fungii think that's going to require some extra care. there are possibly a lot of places where we're installing dependencies before the things depending on them, and pinning library dependencies in global-requirements because they're broken versions of things13:38
fungiso if we just stop syncing requirements to those projects now, we're probably going to wind up in a world of hurt as they start changing them, until we have a plan for making sure jobs always install the things with specific constraints before things with none13:39
dhellmannfungi: yeah, I think we can limit it to removing caps from libs we manage13:39
dhellmannand I intend them to only be out of sync until we get the releases done, and then we'll revert this patch13:39
fungioh, got it13:40
dhellmannof course if you have another option, I'm open to suggestions, this was just a way to achieve the releases without asking you to ninja merge a bunch of requirements changes to the libs :-)13:40
funginah, this gives us a good test too13:40
fungibrb13:41
* dhellmann disconnects for his commute13:41
*** Guest14453 is now known as dansmith_13:43
sdaguedhellmann: so, I'm chasing a release critical nova bug atm, might be limitted in response13:44
*** dansmith_ is now known as dansmith13:58
jeblairi wonder if, since the immediate problem we were working on yesterday is past, maybe we could move some of this discussion back to #openstack-infra?14:07
jeblairi think there's quite a bit here that it would have been useful for other people to see, especially with respect to project-config changes, etc...14:08
fungiagreec14:09
fungiagreed too14:09
*** bswartz has joined #openstack-relmgr-office14:46
*** nikhil_k is now known as flaper87-214:49
*** flaper87-2 is now known as nikhil_k14:50
bswartzttx: ping14:52
nikhil_kttx: hi14:55
bswartzttx: you answered my main question with your ML post, and so far we have no RC potential bugs, so it's possible that we don't need to meet today14:56
nikhil_kttx: Just wated to give a quick update on RCs. There are a few reviews/bugs blocking us on that. I've listed them in https://trello.com/c/CoZ9g13d/45-kilo-rc2 . We can sync on Tuesday or earlier if you need more info.14:56
redrobotttx o/15:07
*** russellb has quit IRC15:32
*** russellb has joined #openstack-relmgr-office15:35
ttxbswartz, redrobot: I'm stuck in a meeting15:38
ttxso we'll skip15:38
ttxredrobot: are you getting closer to a RC1 ?15:38
bswartzttx: no worries, I am too15:38
ttxdhellmann: o/ how are things going ?15:42
redrobotttx yep, all the stuff we want in is in review.   I'm hoping RC1 will be ready by the end of the week.15:43
ttxredrobot: ok, just approve the open-liberty review and I'll make it happen15:48
ttxbrb15:49
*** russellb has quit IRC15:57
*** russellb has joined #openstack-relmgr-office16:01
ttxfungi, jeblair, sdague, dhellmann: Anything urgent you need me for ? I'll be at my hotel all evening and tomorrow morning and able to spend some time on this again16:09
fungittx: not that i'm aware, and dhellmann's https://review.openstack.org/174383 seems to pass jenkins so i'm approving16:10
fungidhellmann: i'll go ahead and propose a wip revert for that too16:11
fungialso moving this conversation to #openstack-infra16:12
ttxok16:13
notmynamettx: re https://review.openstack.org/#/c/174171/ "sort out testing", what's the status. seems that from the ML email this morning, it's good? or is there more to do?16:23
ttxnotmyname: bit more to do, especially on master to unfreeze it16:35
notmynameok. is there something I can track from my side?16:35
ttxhttps://etherpad.openstack.org/p/the-big-thaw16:37
ttxtracking status there ^16:37
ttxbasically currently stable/kilo is 75% good and master requirements still 0% good16:38
*** russellb has quit IRC16:55
*** bswartz has quit IRC16:58
*** russellb has joined #openstack-relmgr-office17:00
* dhellmann finally makes it back online17:36
dhellmannfungi, ttx: ok, heading to infra17:36
*** johnthetubaguy is now known as zz_johnthetubagu18:11
ttxohai19:40
dhellmannttx: I'm still seeing requirements check failures in stable for some of the clients: https://jenkins02.openstack.org/job/gate-python-heatclient-requirements/128/console19:40
ttxdhellmann: so we have a number of libraries that want to do a point release in stable/kilo19:41
dhellmannttx: no one should release *anything* until we have this sorted out19:41
dhellmannhaving more releases just makes this problem harder19:41
ttxright, tat is what I wanted to check19:41
ttxI kind of wanted them in before I cut RC2 so that requirements could be synced, BUT actually they should not trigger a requirements update19:42
ttxso they can land after RC219:42
ttxthe only requirements update blocking a RC2 is swift's https://review.openstack.org/#/c/174171/19:43
dhellmannit looks like we updated requirements in stable/kilo, and that's what's causing the problems with the lib repos19:43
dhellmannI'd like to get to the point where we can land changes in the stable/kilo branches before we make any other changes to anything19:43
ttxso we should still hold on htat one19:43
dhellmannbecause keeping up with what is working and what is broken is getting hairy19:43
ttxchecking if pyeclib is used anywhere else bu swift19:44
ttxdhellmann: I thought we could land changes already (my point (1) in #infra)19:44
dhellmannnot everywhere19:44
dhellmannthe first couple of jobs on https://review.openstack.org/#/q/owner:doug-hellmann+branch:stable/kilo+is:open,n,z are still failing19:45
dhellmannpatches, that is19:45
ttxok, got it19:45
dhellmannheatclient's stable/kilo requirement list is out of date, so I think we can fix that19:45
dhellmannthat may be the same problem everywhere19:45
ttxso the only RC2 blocked atm would be swift's19:45
dhellmannI rechecked https://review.openstack.org/#/c/174338/1 but zuul hasn't picked it up yet19:45
dhellmannttx: fungi just said that he's restarting gerrit so we'll want to recheck again when it is back up19:46
ttxok19:46
fungiyep, it's back up now19:46
dhellmannfungi: thanks19:47
-openstackstatus- NOTICE: gerrit has been restarted to clear a problem with its event stream. any gerrit changes updated or approved between 19:14 and 19:46 utc will need to be rechecked or have their approval reapplied for zuul to pick them up19:47
ttxnikhil_k, morganfainberg: we'll still have to wait a bit before cutting new client libs, stable/kilo branches still acting up19:50
nikhil_kttx: gotcha, thanks. I think Stuart McLaren pointed out a good bug that we should get fixed in glanceclient and to avoid breaking Nova, Cinder, Ironic in some situations :)19:52
ttxnikhil_k: one question is.. should we bump the requirements as a result of that19:52
nikhil_kah glad you saw that19:53
nikhil_kttx: yes we should19:53
ttxdepends how broken the current version is19:53
nikhil_kso if people use ssl19:53
ttxbecause the bump needs to propagate to release, which make it a costly thing19:53
nikhil_kNova won't be able to boot VM19:53
nikhil_kSame goes for Cinder etc19:54
ttxnikhil_k: which versions are affected ? Is that a recent regression ?19:54
nikhil_kttx: this might help https://review.openstack.org/#/c/172453/319:55
ttxthx19:55
ttxso anything since 0.16.119:55
ttxlooks like people have been living with that for some time :)19:56
ttxok, we'll consider a bump when we'll be able to make a new version of the lib19:56
ttxdepending on when that is, may be cheap or costly19:56
dhellmannttx: it looks like the bot-submitted requirements update to heatclient passed the requirements test, so that's progress20:00
ttxdhellmann: cool20:02
ttxnotmyname: around?20:02
ttxdhellmann: hmm.. ;looks like we never pushed a .gitreview update for openstack/swift20:06
ttxI'll propose one now, good way to check we can land stuff there20:06
notmynamettx: here20:07
ttxnotmyname: was looking into the need/possibility to open a swift RC2 window20:07
notmynamewhat do you mean?20:07
ttxbut then we can't land the requirements change just now so probably no hurry20:07
ttxi.e. list the patches we want in a RC2, land them and tag the result RC220:07
notmynameoh, you mean today instead of monday/tuesday?20:08
ttxyeah, but that was before I saw we didn't land the .gitreveiw bump there20:08
ttxwe should start by that, to check that the branch is operational20:08
notmynamewe don't have all the patches ready for an rc220:09
ttxnotmyname: ok, so no hurry20:09
ttxcome on hotel wifi20:10
notmyname.gitreview is a convenience. it's nice to have, but totally optional. what do you mean by making sure the branch is "operational"?20:10
ttxchecking that the branch successfully runs tests using a gratuitous change that doesn't even affect the release20:11
nikhil_kttx: thanks (got pulled away earlier)20:11
notmynamegot it20:11
ttxi.e. the .gitreview change20:11
ttxI'd welcome your +2 on https://review.openstack.org/#/c/174582/20:11
ttxnotmyname: ^20:11
notmynamettx: done20:12
ttxthx20:12
dhellmannttx: I only pushed those changes for the libs where we were making branches20:13
ttxdhellmann: ah.Hm. Should probably push them for every project then. Checked for nova and assumed we did that everywhere20:14
notmynamettx: so after .gitreview lands, then we'll be able to land the RC patches20:14
notmyname?20:14
ttxnotmyname: I'd rather have them all ready before we start merging them. The idea is to not kill the RC1 until we are ready to make a RC220:15
ttxas soon as we statr doing a RC2 people stop testing RC120:15
ttxso the shorter the window the better20:15
notmynameok, makes sense. so we'll queue up the stuff to land, then open it (ie say "let's do rc2"), then land it all20:17
ttxyep20:17
notmynameok, and we'll target early next week for that20:17
notmynamethaks20:17
ttxalso helps seeing how risky the bundle is (not that important right now, but important when a few days from release)20:18
ttxnotmyname: ack20:18
notmynamettx: so we do have one change for rc2 that is dependent on the global requirements change. that's the only thing I'm worried about20:18
ttxagreed, that's why we can't do the RC2 fr swift until we get thigs all straighten out with requirements20:19
dhellmannttx: I have a script ready to look for other projects that need their .gitreview updated; let me know when we're at a point where it's safe to run it20:29
ttxdhellmann: let's see tomorrow when we have the test results of the probes I launched20:30
ttx(keystone and swift)20:30
dhellmannttx: ok, I'll push it up as a review for release-tools since I won't be online tomorrow20:30
ttxok thx20:30
ttxok, I'm a bit tired now so I'll call it a day20:32
ttxdhellmann: so you won't be around at all tomorrow ?20:32
dhellmannttx: https://review.openstack.org/17459420:34
dhellmannttx: no, it's a travel day :-/20:34
ttxOK, I'll try to make progress where i can.20:34
ttxttyl!20:35
dhellmannttx: I'll be here for another 45 minutes or so20:36
dhellmannttx: fungi has https://review.openstack.org/#/c/174595/1 ready to remove the requirements check jobs for libs in master so we can land the needed updates20:38
fungilooks like i should tweak the commit message there a teeny bit, just a sec20:39
fungicommit message improved20:40
*** dims__ has quit IRC21:23
*** dims__ has joined #openstack-relmgr-office22:08
*** dims__ has quit IRC22:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!