| *** maxbit has quit IRC | 00:02 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Install newer requests package for twine https://review.openstack.org/97962 | 00:03 |
|---|---|---|
| fungi | jhesketh: ^ | 00:03 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Move variables in manifests/site.pp to top of file https://review.openstack.org/97659 | 00:04 |
| *** camunoz has quit IRC | 00:04 | |
| jhesketh | fungi: LGTM :-) | 00:05 |
| *** signed8bit has quit IRC | 00:05 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Make import checks properly detect blank lines and comments https://review.openstack.org/97988 | 00:05 |
| *** maxbit has joined #openstack-infra | 00:06 | |
| *** salv-orlando has quit IRC | 00:07 | |
| *** Ivo has quit IRC | 00:07 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Add templated uploading jobs for doc8 project https://review.openstack.org/96631 | 00:08 |
| *** signed8bit has joined #openstack-infra | 00:09 | |
| *** signed8bit has quit IRC | 00:10 | |
| *** praneshp_ has joined #openstack-infra | 00:10 | |
| *** wenlock has quit IRC | 00:10 | |
| *** signed8bit has joined #openstack-infra | 00:11 | |
| *** praneshp has quit IRC | 00:11 | |
| *** praneshp_ is now known as praneshp | 00:11 | |
| *** signed8bit has quit IRC | 00:11 | |
| *** dims__ has joined #openstack-infra | 00:12 | |
| *** signed8bit has joined #openstack-infra | 00:12 | |
| *** jaypipes has joined #openstack-infra | 00:13 | |
| *** mmaglana has quit IRC | 00:13 | |
| *** dims_ has quit IRC | 00:14 | |
| *** dims__ is now known as dims | 00:15 | |
| clarkb | sdague: https://review.openstack.org/#/c/97845/ failed in the gate | 00:16 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Bug 1324932 is a duplicate of 1286635 https://review.openstack.org/97992 | 00:16 |
| uvirtbot | Launchpad bug 1324932 in openstack-ci "devstack setup fails with "E: There are problems and -y was used without --force-yes" (dup-of: 1286635)" [Undecided,New] https://launchpad.net/bugs/1324932 | 00:16 |
| uvirtbot | Launchpad bug 1286635 in openstack-ci ""Packages cannot be authenticated" when doing an apt-get in the gate" [Undecided,Won't fix] https://launchpad.net/bugs/1286635 | 00:16 |
| clarkb | so did 97757 | 00:16 |
| lifeless | clarkb: saddddface | 00:18 |
| mtreinish | clarkb: the 97757 fail looks like a new one, I'm working on a query | 00:19 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Updated zake now that zake 0.0.20 is released https://review.openstack.org/93156 | 00:19 |
| clarkb | mtreinish: thanks | 00:19 |
| *** signed8bit has quit IRC | 00:21 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Suppress the graph for bug 1260654 https://review.openstack.org/97993 | 00:22 |
| uvirtbot | Launchpad bug 1260654 in openstack-ci "Could not initialize class jenkins.model.Jenkins$MasterComputer" [Critical,Fix released] https://launchpad.net/bugs/1260654 | 00:22 |
| *** apevec has quit IRC | 00:24 | |
| openstackgerrit | A change was merged to openstack-infra/config: Install newer requests package for twine https://review.openstack.org/97962 | 00:24 |
| *** ramashri has joined #openstack-infra | 00:26 | |
| *** zhiyan_ is now known as zhiyan | 00:27 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Bug 1324932 is a duplicate of 1286635 https://review.openstack.org/97992 | 00:29 |
| uvirtbot | Launchpad bug 1324932 in openstack-ci "devstack setup fails with "E: There are problems and -y was used without --force-yes" (dup-of: 1286635)" [Undecided,New] https://launchpad.net/bugs/1324932 | 00:29 |
| uvirtbot | Launchpad bug 1286635 in openstack-ci ""Packages cannot be authenticated" when doing an apt-get in the gate" [Undecided,Won't fix] https://launchpad.net/bugs/1286635 | 00:29 |
| *** yamahata has joined #openstack-infra | 00:30 | |
| harlowja | hi guys, is it possible to get https://review.openstack.org/#/c/93156/ checked, it has 2 +2's and then i had to rebase and adjust 1 line (nothing else changed) | 00:31 |
| *** unicell has joined #openstack-infra | 00:33 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Use the correct name for apache https://review.openstack.org/97994 | 00:34 |
| *** UtahDave has quit IRC | 00:36 | |
| *** maxbit has quit IRC | 00:36 | |
| *** maxbit has joined #openstack-infra | 00:36 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add query for bug 1304184 https://review.openstack.org/97995 | 00:39 |
| uvirtbot | Launchpad bug 1304184 in nova "instance stuck into rebuild state when nova-compute unexpected restart" [Undecided,In progress] https://launchpad.net/bugs/1304184 | 00:39 |
| *** annegentle has quit IRC | 00:40 | |
| *** mwagner_lap has joined #openstack-infra | 00:40 | |
| *** jhesketh has quit IRC | 00:41 | |
| *** homeless has quit IRC | 00:46 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add query for bug 1304184 https://review.openstack.org/97995 | 00:47 |
| uvirtbot | Launchpad bug 1304184 in nova "instance stuck into rebuild state when nova-compute unexpected restart" [Undecided,In progress] https://launchpad.net/bugs/1304184 | 00:47 |
| *** ramashri has quit IRC | 00:48 | |
| *** rwsu has quit IRC | 00:51 | |
| *** gokrokve has joined #openstack-infra | 00:54 | |
| *** aconrad has quit IRC | 01:00 | |
| *** jhesketh_ has joined #openstack-infra | 01:00 | |
| *** jhesketh_ has quit IRC | 01:01 | |
| *** jhesketh has joined #openstack-infra | 01:01 | |
| *** mrodden has quit IRC | 01:01 | |
| *** camunoz has joined #openstack-infra | 01:03 | |
| *** mriedem has joined #openstack-infra | 01:04 | |
| *** maxbit has quit IRC | 01:04 | |
| *** maxbit has joined #openstack-infra | 01:05 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1304184 https://review.openstack.org/97995 | 01:06 |
| uvirtbot | Launchpad bug 1304184 in nova "instance stuck into rebuild state when nova-compute unexpected restart" [Undecided,In progress] https://launchpad.net/bugs/1304184 | 01:06 |
| *** amcrn has quit IRC | 01:07 | |
| *** asettle has quit IRC | 01:13 | |
| *** asettle has joined #openstack-infra | 01:14 | |
| *** maxbit has quit IRC | 01:15 | |
| *** maxbit has joined #openstack-infra | 01:16 | |
| *** yaguang has joined #openstack-infra | 01:19 | |
| *** maxbit has quit IRC | 01:24 | |
| *** maxbit has joined #openstack-infra | 01:25 | |
| *** aconrad has joined #openstack-infra | 01:25 | |
| *** crc32 has quit IRC | 01:25 | |
| *** yamahata has quit IRC | 01:26 | |
| *** yamahata has joined #openstack-infra | 01:27 | |
| jogo | fungi: https://review.openstack.org/#/c/97845/ | 01:27 |
| jogo | can that be bumped to top of gate? | 01:27 |
| clarkb | it was already bumped | 01:27 |
| clarkb | and I need to make dinner | 01:27 |
| jogo | clarkb: it failed out once | 01:27 |
| clarkb | yup | 01:27 |
| *** maxbit has quit IRC | 01:27 | |
| jogo | clarkb: because of https://review.openstack.org/#/c/97912/ | 01:28 |
| clarkb | right so I think we need to reprioritize | 01:28 |
| clarkb | get fixes for actual fails in first | 01:28 |
| clarkb | I doubt there are a lot of icehouse changes right now | 01:28 |
| *** maxbit has joined #openstack-infra | 01:28 | |
| jogo | https://etherpad.openstack.org/p/June-gate has two patches for actual fails so far | 01:29 |
| fungi | well, apevec is trying to cut 2014.1.1 tomorrow | 01:29 |
| clarkb | oh is it that time already? | 01:29 |
| clarkb | that seems early | 01:29 |
| jogo | fungi: and this should be in 2014.1.1 right? | 01:29 |
| clarkb | fungi: feel free to promote but I can't do that right now | 01:29 |
| clarkb | I am dinnering | 01:29 |
| clarkb | but if things keep failing the priority should be fixing the underlying issues | 01:30 |
| fungi | jogo: that's more what i'm asking... did this get a freeze exception? | 01:30 |
| clarkb | like 97912 and the ntpdate thing | 01:30 |
| jogo | fungi: hmm not sure I don't see any review from apevec | 01:31 |
| fungi | jogo: i'm actually curious how there are non-stable-maint +2s on it | 01:31 |
| * jogo tries to +2 it | 01:31 | |
| jogo | seems I cannot | 01:31 |
| jogo | anyway I don't really see how cutting 2014.1.1 can block stablizing the gate | 01:32 |
| jogo | that doesn't make sense to me | 01:32 |
| fungi | jogo: oh, they are in stable-maint-core | 01:33 |
| fungi | er, openstack-stable-maint | 01:33 |
| fungi | i forgot they were members | 01:33 |
| fungi | but it does look like all nova reviewers on that | 01:34 |
| fungi | probably would be good to double-check since the call for testing has already gone out on that branch earlier today | 01:34 |
| clarkb | the ntpdate change didn't work | 01:34 |
| fungi | oh? what did it do? | 01:35 |
| clarkb | fungi: http://logs.openstack.org/24/97924/2/check/check-tempest-dsvm-full/7344c4f/console.html#_2014-06-04_22_20_11_688 | 01:35 |
| *** lcostantino has quit IRC | 01:35 | |
| *** rfolco has quit IRC | 01:36 | |
| jogo | dtroyer: https://review.openstack.org/#/c/97912/ | 01:36 |
| clarkb | fungi: I can reproduce that with `ntpdate You` | 01:36 |
| anteaya | jogo: is he back? | 01:36 |
| *** maxbit has quit IRC | 01:36 | |
| anteaya | jogo: he has been away | 01:36 |
| jogo | anteaya: didn't know he was gone | 01:36 |
| clarkb | fungi: so I think the grep for the ntp server is wrong | 01:36 |
| jogo | anteaya: looking for some devstack cores | 01:36 |
| fungi | russellb: if you're around, did https://review.openstack.org/97845 get a stable freeze exception? call for testing already went out and point release was supposed to be tagged tomorrow i think | 01:36 |
| clarkb | fungi: maybe we hardcode debian ntp servers as the default then we can override using that variable | 01:36 |
| anteaya | jogo: last week he was and the first of this, sdague said he should be back end of this week | 01:36 |
| anteaya | jogo: I haven't seen him around though | 01:37 |
| *** fbo_away is now known as fbo | 01:37 | |
| jogo | jeblair: ^ | 01:37 |
| jogo | your devstack core | 01:37 |
| anteaya | jogo: you are hitting all the away folks | 01:37 |
| jogo | along with jessie andrews and vish | 01:37 |
| clarkb | fungi: left in a comment on that change | 01:38 |
| clarkb | now time for food | 01:38 |
| *** gokrokve has quit IRC | 01:38 | |
| jogo | what devstack cores are around? | 01:38 |
| fungi | clarkb: hah. i think it's parsing a comment because it should grep for ^server | 01:38 |
| jogo | https://review.openstack.org/#/admin/groups/50,members | 01:39 |
| fungi | jogo: jeblair is away for a week or more | 01:39 |
| * jogo thinks we should make fungi an honerary devstack core | 01:39 | |
| anteaya | jogo: wow, sdague | 01:39 |
| jogo | honorary | 01:40 |
| anteaya | yeah teh devstack core group needs some updating | 01:40 |
| jogo | anteaya: badly | 01:40 |
| * fungi doesn't need more spinning plates | 01:40 | |
| anteaya | are jesse and vish around? | 01:40 |
| anteaya | I rarely see them on irc | 01:40 |
| anteaya | chmouel is but wrong time zone | 01:40 |
| clarkb | fungi: oh if its that simple then we can update the grep | 01:40 |
| anteaya | and garyk is active on reviews but I don't often see him about on irc | 01:41 |
| clarkb | jogo: he is already honorary everything core when he feels like it | 01:41 |
| jogo | clarkb: an offical honorary core in this case | 01:41 |
| clarkb | fungi: this may be a case for unofficial honorary approving, but I think we can let sdague self approve instead | 01:41 |
| anteaya | so hopefully dtroyer is coming to the qa/infra meetup and can bootstrap some folks on the path to devstack-core | 01:42 |
| jogo | anteaya: that reminds me I have to tell monty I am gonna go | 01:43 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Force ntpdate before we start the timer https://review.openstack.org/97924 | 01:43 |
| fungi | clarkb: updated ^ | 01:43 |
| *** aysyd has quit IRC | 01:44 | |
| clarkb | fungi: thanks, I can approve it once tests get beyond that first bit of d-g | 01:44 |
| fungi | jogo: looking forward to seeing you there | 01:44 |
| *** jcooley has quit IRC | 01:44 | |
| clarkb | jogo: we have to tell monty? :P | 01:45 |
| clarkb | I hope he knows I am going | 01:45 |
| anteaya | jogo: sign up: https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014 | 01:45 |
| anteaya | clarkb: I'm just going to send him my expense report | 01:45 |
| clarkb | fungi: also good catch and woo comments | 01:45 |
| anteaya | well he was there when we were talking about it | 01:46 |
| *** nosnos has joined #openstack-infra | 01:46 | |
| jogo | mtreinish: ping https://review.openstack.org/#/c/97842/ | 01:46 |
| jogo | clarkb: well tell him is a code for book the tiket | 01:47 |
| anteaya | the golden tiket | 01:47 |
| clarkb | fungi: 5 Jun 01:47:21 ntpdate[1886]: the NTP socket is in use, exiting I was worried about that | 01:47 |
| fungi | clarkb: crap. so we need to stop and start ntp around it i guess | 01:48 |
| fungi | adding | 01:48 |
| clarkb | either that or convince ntpd to do a 4 hour update (which it won't do) | 01:48 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: [WIP] Modularized StoryBoard Module https://review.openstack.org/98007 | 01:48 |
| jogo | jeblair: works for HP? | 01:48 |
| anteaya | yeah I saw that too | 01:48 |
| anteaya | that also was my response | 01:48 |
| anteaya | if you look at the history corvus made that change | 01:49 |
| anteaya | so I don't understand | 01:49 |
| clarkb | fungi: not sure what the ntp appropriate response here is | 01:49 |
| krotscheck | mordred: Finally got around to making all the versions play happy ^ | 01:49 |
| jogo | anteaya: I figured a cut and pase error? | 01:49 |
| clarkb | fungi: ntp hates me | 01:49 |
| jogo | mtreinish: got some tempest patches for you | 01:49 |
| anteaya | jogo: that makes sense | 01:49 |
| krotscheck | Still need to add more docs. | 01:49 |
| *** jcooley has joined #openstack-infra | 01:49 | |
| jogo | fungi: ^ | 01:49 |
| anteaya | jogo: yeah cause it is the same as what mtreinish said | 01:49 |
| jogo | mtreinish: oh you already +1ed it https://review.openstack.org/#/c/97842/1 | 01:50 |
| clarkb | fungi: maybe we need to prepend our ntpd init script with ntpdate? | 01:50 |
| anteaya | whereas we who work on monty's team all have a different name for it | 01:50 |
| clarkb | fungi: hax | 01:50 |
| jogo | mtreinish: err https://review.openstack.org/#/c/97940/ | 01:50 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Force ntpdate before we start the timer https://review.openstack.org/97924 | 01:51 |
| clarkb | fungi is `service` something centos/fedora speak? | 01:52 |
| fungi | yeah, which is why i used it (though i don't know for sure that the servicename is the same, or even that ntpdate is installed on them by default) | 01:53 |
| *** maxbit has joined #openstack-infra | 01:53 | |
| ianw | is anyone with under-the-hood knowledge of hpcloud here? | 01:54 |
| clarkb | ianw: tteggel and gilliard | 01:54 |
| ianw | i wonder if https://bugs.launchpad.net/devstack/+bug/1326539 is related to the connection issues i've seen | 01:54 |
| uvirtbot | Launchpad bug 1326539 in devstack "devstack setup fails to get pip" [Undecided,In progress] | 01:54 |
| ianw | described here http://lists.openstack.org/pipermail/openstack-infra/2014-June/001299.html | 01:54 |
| ianw | tteggel / gilliard : ^ | 01:54 |
| clarkb | ianw: could be but we shouldn't be talking to pypi.python.org at all | 01:54 |
| clarkb | ianw: so its a bug on our end for sure | 01:54 |
| ianw | clarkb: yeah, there's sdague's change out https://review.openstack.org/#/c/97912/ | 01:55 |
| ianw | but in the mean time, these timeouts seem semi-frequent | 01:55 |
| ianw | i was also seeing it to bootstrap.pypa.io anyway | 01:56 |
| jogo | fungi: so thoughts on bumping https://review.openstack.org/#/c/97845/ it has had 43 hits in last 24 hours | 01:56 |
| jogo | it almost got merged once before anyway | 01:56 |
| jogo | also what do you think of approving https://review.openstack.org/#/c/97912/ ? | 01:57 |
| *** melwitt has quit IRC | 01:57 | |
| clarkb | ianw: we cache the bootstrap.pypa.io file | 01:57 |
| clarkb | ianw: which is why that method is prefered | 01:57 |
| clarkb | fungi: ntpdate change lgtm | 01:58 |
| clarkb | fungi: did you want to check logs before I approve it? | 01:58 |
| fungi | looking | 01:58 |
| jogo | that failure has 51 hits in 48 hours | 01:59 |
| jogo | message:"err_trap" AND tags:"console" | 01:59 |
| clarkb | jogo: I think sdague should approve | 01:59 |
| fungi | hrm... no experimental f20 jobs on d-g | 02:00 |
| jogo | clarkb: he is done for the day | 02:00 |
| clarkb | jogo: I know | 02:00 |
| clarkb | he will be awake really early tomorrow | 02:00 |
| jogo | haha I thought we could have a present for him thats all | 02:00 |
| clarkb | I don't think ninja approving a change with no core votes is a present | 02:01 |
| clarkb | more like invite to yell on ml | 02:01 |
| jogo | clarkb: fair enough | 02:01 |
| jogo | but the lack of devstack cores around is scary | 02:02 |
| jogo | almost as bad as grenade | 02:02 |
| jogo | clarkb: so the problem with sdague approving his patch early in the morning, is what infra person will be around to bump it up to the top of the queue? | 02:03 |
| clarkb | jogo: SergeyLukjanov | 02:03 |
| fungi | clarkb: yeah, logs in progress for 97924 look right | 02:03 |
| clarkb | approved | 02:04 |
| fungi | and yeah, SergeyLukjanov already has lots of experience enqueuing and promoting changes in the gate, so should be fine | 02:04 |
| clarkb | jogo: honestly I don't think its that critical | 02:04 |
| jogo | clarkb: ahh excellent | 02:04 |
| clarkb | if it happens at 9am pst vs 5am pst meh | 02:04 |
| jogo | clarkb: well every fix to the gate we can do is a bug step forward | 02:05 |
| fungi | if it waits until 1300z (9am edt) i'll be around too | 02:05 |
| jogo | btw is there abug filed with rax about there mirror? | 02:05 |
| fungi | is it permanently broken? | 02:05 |
| jogo | fungi: transiently | 02:06 |
| jogo | or to be more specific: apt-get update randmly fails | 02:06 |
| jogo | anyay dinner time o/ | 02:06 |
| clarkb | me too | 02:08 |
| ianw | fungi: yeah, was thinking of adding f20 jobs to d-g | 02:08 |
| clarkb | I will probably pop in later to see how 97924 is doing | 02:08 |
| clarkb | ianw: hopefully 97924 doesn't break you >_> | 02:08 |
| clarkb | ianw: though that point in d-g things can fail without failing the job | 02:09 |
| ianw | fungi: get it checking on devstack first | 02:09 |
| clarkb | so we should be ok | 02:09 |
| clarkb | worst case time doesn't update but test will continue to run | 02:09 |
| fungi | right, worst case is no worse than previous except a few benign errors in the console log | 02:10 |
| *** marcoemorais has quit IRC | 02:10 | |
| *** trinaths has joined #openstack-infra | 02:15 | |
| ianw | fungi: benign errors are the worst though, because we have to find the problem all over again on fedora before realising it needs fixing. anyway, just checking if ntp is installed by that point on f20... | 02:15 |
| trinaths | Good Morning All. | 02:16 |
| *** fbo is now known as fbo_away | 02:16 | |
| fungi | ianw: it's entirely probable that we need to add ntpdate to the puppet manifests. i think it just happens to be on ubuntu images in our providers by chance | 02:17 |
| *** dkranz has quit IRC | 02:17 | |
| ianw | fungi / clarkb : ok, for a start it's "ntpd" not "ntp" on f20 | 02:18 |
| *** zz_gondoi is now known as gondoi | 02:19 | |
| ianw | it's not in the cloud image i booted by default, but not sure how that translates to upstream | 02:20 |
| ianw | anyway, i'll propose the tests for upstream soonish and then i can do a fixup easy enough | 02:21 |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack/requirements: Add rfc3987 library https://review.openstack.org/98012 | 02:26 |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack/requirements: Add rfc3987 library https://review.openstack.org/98012 | 02:28 |
| *** gondoi is now known as zz_gondoi | 02:30 | |
| *** Ryan_Lane has quit IRC | 02:30 | |
| *** pcrews has quit IRC | 02:32 | |
| mtreinish | jogo: I actually have concerns on the second patch. I think it'll cause more problems | 02:32 |
| mtreinish | because we don't know how long the volumes will be lying around | 02:32 |
| mtreinish | instead of falling apart because of slow deletes I think running out of space in the backing lvm volume is a possibility | 02:32 |
| mtreinish | and also leaking state between admin tests | 02:33 |
| mtreinish | I want to talk to sdague and jgriffith about it tomorrow to see if we can figure out a better fix | 02:33 |
| *** trinaths has quit IRC | 02:33 | |
| *** signed8bit has joined #openstack-infra | 02:37 | |
| *** wenlock has joined #openstack-infra | 02:38 | |
| mtreinish | and on the first patch mriedem brings up a good point about doing it for the v3 tests too. (even if it has a +2 from me already) | 02:38 |
| mtreinish | jogo: anyway I'm out for tonight, I'll be back tomorrow | 02:38 |
| fungi | ianw: thanks! in part, the _existence_ of running ntp is the problem we're working around (having an instance boot with incorrect time isn't so bad if ntp doesn't then change it during the job) | 02:38 |
| mordred_phone | fungi: morning | 02:43 |
| fungi | mordred_phone: landed? | 02:44 |
| *** signed8bit has quit IRC | 02:45 | |
| mordred_phone | yup. taxiing | 02:45 |
| mordred_phone | there was no power on this flight, so I did not accomplish work | 02:45 |
| anteaya | mordred_phone: where are you? | 02:45 |
| anteaya | palo alto? | 02:45 |
| mordred_phone | just landed in nyc | 02:46 |
| anteaya | cool | 02:46 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Fix grab_console_log variable typo https://review.openstack.org/98015 | 02:47 |
| anteaya | I have to go to sleep | 02:48 |
| anteaya | I hope no more people get shot in Moncton | 02:48 |
| jhesketh | clarkb, fungi: If you guys wouldn't mind merging this trivial patch I'd appreciate it (I'm poking at the swift pushing stuff) https://review.openstack.org/#/c/98015/ | 02:50 |
| *** dims has quit IRC | 02:53 | |
| *** dims has joined #openstack-infra | 02:55 | |
| clarkb | jhesketh: done | 02:56 |
| jhesketh | thanks clarkb :-) | 02:56 |
| *** gokrokve has joined #openstack-infra | 03:02 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix grab_console_log variable typo https://review.openstack.org/98015 | 03:03 |
| *** lcheng_ has joined #openstack-infra | 03:03 | |
| *** aconrad has quit IRC | 03:07 | |
| *** arnaud has quit IRC | 03:10 | |
| *** mriedem has quit IRC | 03:11 | |
| *** sballe has joined #openstack-infra | 03:11 | |
| *** asettle has quit IRC | 03:13 | |
| *** alexandra has joined #openstack-infra | 03:13 | |
| *** RuiChen has joined #openstack-infra | 03:15 | |
| *** alexandra is now known as asettle | 03:17 | |
| *** Ryan_Lane has joined #openstack-infra | 03:18 | |
| *** otherwiseguy has joined #openstack-infra | 03:23 | |
| RuiChen | hi stackers, I have a feature need to be tested in libvirt 1.1.1+ and qemu1.5+, the current ci infrastructure support it? | 03:23 |
| *** Ryan_Lane has quit IRC | 03:24 | |
| RuiChen | if not, what I can do? | 03:24 |
| clarkb | RuiChen: not yet | 03:24 |
| clarkb | when we switch to trusty I think so | 03:24 |
| clarkb | I am hoping to have trusty slaves available by end of next week. Then we will have to test and so on so it may be a couple more weeks | 03:26 |
| jesusaurus | clarkb: what needs to happen to get trusty slaves going? | 03:26 |
| clarkb | jesusaurus: nodepool dib | 03:26 |
| *** sballe has quit IRC | 03:26 | |
| mgagne | clarkb: and puppet? | 03:26 |
| clarkb | we want to switch to uploading glance images as the current image builds are clunky and don't scale | 03:27 |
| clarkb | mgagne: puppet is done | 03:27 |
| mgagne | clarkb: awesome | 03:27 |
| clarkb | (assuming we haven't regressed) | 03:27 |
| *** sballe has joined #openstack-infra | 03:27 | |
| clarkb | I was going to start working on it yesterday but things were still sideways | 03:29 |
| clarkb | fungi: oh we didn't restart nodepool today :( | 03:29 |
| * clarkb will try to do that early tomorrow | 03:29 | |
| fungi | clarkb: i'm still up for it, but it can wait too | 03:31 |
| clarkb | if you want to do it I can hang around but machine with keys is off | 03:32 |
| clarkb | I can turn it on if anything goes sideways or we can wiat | 03:32 |
| fungi | i suppose i could just do it now... was there anything in particular we wanted to double-check? | 03:32 |
| clarkb | fungi: yes we need to make sure puppet is running and the version we want is installed | 03:32 |
| RuiChen | changbl: thank you, I look forward to trusty :) | 03:32 |
| clarkb | fungi: then we need to check that we don't leak nodes again | 03:33 |
| *** talluri has joined #openstack-infra | 03:33 | |
| *** jaypipes has quit IRC | 03:33 | |
| fungi | clarkb: okay, so stop nodepool, delete cruft nodes, enable puppet agent, trigger puppet run from ci-puppetmaster, check installed nodepool version, start nodepool... | 03:35 |
| fungi | missing anything obvious? | 03:35 |
| clarkb | fungi: I think you can run puppet first and check the version before stopping | 03:36 |
| clarkb | otherwise lgtm | 03:36 |
| fungi | ahh, good point | 03:36 |
| fungi | starting now | 03:36 |
| *** dims_ has joined #openstack-infra | 03:39 | |
| *** dims has quit IRC | 03:39 | |
| clarkb | basically we want to make sure that jeblair's fix gets in for that node leakage and double check it worked | 03:40 |
| fungi | yep | 03:41 |
| *** cody-somerville has quit IRC | 03:42 | |
| fungi | of course we've broken nodepool --version, but pip list sez 68860d9 is installed | 03:42 |
| fungi | which looks like the tip of master | 03:42 |
| *** nosnos has quit IRC | 03:43 | |
| *** sballe has quit IRC | 03:43 | |
| fungi | no, wait | 03:43 |
| fungi | it's not | 03:43 |
| clarkb | ya that must be the old broken version beacuse its not in my gitl og | 03:43 |
| clarkb | b6539f9cdd19a71d8e9e7dec49f21ac15876950b is what we want | 03:43 |
| fungi | should be b6539f9 yeah | 03:44 |
| fungi | checking logs | 03:44 |
| fungi | hrm | 03:45 |
| clarkb | fungi: I am going to guess the repo isn't on the master branch in /opt and puppet won't change it there to do an update | 03:45 |
| clarkb | fungi: I think if you git checkout master then run puppet it will be ok | 03:45 |
| clarkb | make sure you do that as the owner of the repo though otherwise files may end up with funny perms | 03:46 |
| fungi | # On branch master | 03:46 |
| clarkb | oh what if puppet is ff only | 03:46 |
| clarkb | might need to just git reset --hard origin/master | 03:46 |
| fungi | the puppet log mentioned no problem updating to master and refreshing the nodepool install exec | 03:46 |
| fungi | well, the repo in /opt matches the origin state, no problem | 03:47 |
| fungi | but pip reports a different installed nodepool version | 03:47 |
| fungi | is what i was trying to say | 03:47 |
| clarkb | oh its one of those situations woo pip | 03:47 |
| *** reed has quit IRC | 03:47 | |
| fungi | i wonder if the puppet exec doesn't work the way we think | 03:48 |
| clarkb | or if pip is confused | 03:48 |
| fungi | oh, maybe | 03:48 |
| fungi | with --version broken it's hard to tell | 03:49 |
| clarkb | ya | 03:49 |
| clarkb | fungi: well if nodepool is still up and running in its current state maybe we just deal with it tomorrow | 03:50 |
| clarkb | (it is pretty late for both of us more so you) | 03:50 |
| fungi | drwxr-sr-x 2 root staff 4096 Jun 5 03:40 /usr/local/lib/python2.7/dist-packages/nodepool-b6539f9-py2.7.egg-info | 03:51 |
| fungi | so maybe pip is confused, yes | 03:51 |
| clarkb | I think that is a case where pip uninstall ; rm -rf dist-packages/nodepool* ; pip install --upgrade cures the problem | 03:52 |
| clarkb | but again maybe just deal with that tomorrow | 03:52 |
| fungi | agreed. puppet is still enabled... leave it that way, yeah? | 03:53 |
| clarkb | I think so, everything is up to date from a config and nodepool git perspective so pupept should be fine | 03:53 |
| fungi | cool. have a good night! | 03:53 |
| clarkb | you too | 03:53 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Fetch graphitejs zuul dependency https://review.openstack.org/98018 | 03:55 |
| ianw | clarkb / fungi: is min-ram 30720 just supposed to be the default for hpcloud instances? | 03:58 |
| fungi | ianw: it's the minimum flavor you need to get 8 cpus | 04:00 |
| clarkb | which after testing is what we hae found to be the sweet spot for performance to resources ratios | 04:00 |
| clarkb | pleia2: when did the e-r on status.o.o change merge? | 04:00 |
| lifeless | also the point at which tempest doesn't kill devstack | 04:00 |
| clarkb | I completely missed that in the mess of broken clouds | 04:00 |
| pleia2 | clarkb: yesterday I think | 04:00 |
| *** basha has joined #openstack-infra | 04:00 | |
| clarkb | lifeless: well it worked fine on 4 cpu nodes | 04:00 |
| ianw | ah cool, thanks | 04:00 |
| fungi | and roughly matches the performance we get from an 8g rax instance | 04:01 |
| clarkb | lifeless: I have some rough unscientific numbers if you wan them for why this is how we do it | 04:01 |
| lifeless | clarkb: yes but see sdague's point about more cpu counts above ;> | 04:01 |
| clarkb | fungi: but still slower than 1.0 | 04:01 |
| fungi | indeed | 04:01 |
| clarkb | lifeless: right but ^ 4vcpu 1.0 was still better | 04:01 |
| lifeless | clarkb: once I dig myself out of this mess, yes. | 04:01 |
| *** RuiChen has quit IRC | 04:04 | |
| *** eharney has quit IRC | 04:04 | |
| *** harlowja is now known as harlowja_away | 04:07 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: tests for JJB command processing https://review.openstack.org/97908 | 04:08 |
| *** basha has quit IRC | 04:10 | |
| mordred | lifeless: you haz mess? | 04:11 |
| mordred | krotscheck: neat! | 04:11 |
| *** lcheng_ has quit IRC | 04:12 | |
| lifeless | mordred: yes, Ironic was being, uhm, ironic with me. | 04:13 |
| clarkb | and 97757 failed | 04:13 |
| clarkb | mordred: you know if you aren't dead tired and ready to fall asleep | 04:13 |
| clarkb | mordred: 97924,4 and 97757,1 need promotion in the gate | 04:14 |
| mordred | lifeless: _excellent_ | 04:14 |
| clarkb | in that order | 04:14 |
| mordred | clarkb: I'm actually not dead tired | 04:14 |
| *** CSheedy has joined #openstack-infra | 04:14 | |
| clarkb | I am not either but the machine with keys is over there off so that I don't work a whole lot | 04:14 |
| clarkb | this strategy would work if I didn't have another computer over here with a connection to freenode | 04:15 |
| mordred | clarkb: should I wait for a reset? or just promote them? | 04:15 |
| *** renlt has joined #openstack-infra | 04:15 | |
| clarkb | probably wait for reset as we may actually clear out ~20 changes in a half hour | 04:15 |
| mordred | ok. I'll keep my eye on it | 04:15 |
| mordred | clarkb: I had a terrible idea, btw ... | 04:16 |
| *** Sukhdev has joined #openstack-infra | 04:16 | |
| *** otherwiseguy has quit IRC | 04:16 | |
| mordred | clarkb: what if we wrote a filter/function/something for zuul that would allow something to enqueue a change with a different pipeline config | 04:16 |
| mordred | clarkb: oh wait - nevermind | 04:17 |
| *** yfried has quit IRC | 04:17 | |
| *** gokrokve has quit IRC | 04:17 | |
| mordred | clarkb: thanks, it was helpful to try to write out the thought in my head | 04:17 |
| clarkb | ok :) | 04:17 |
| clarkb | mordred: also looks like a small group of us may be leaning towards staying in darmstadt and not frankfurt | 04:18 |
| clarkb | to maximize sleep in time | 04:18 |
| clarkb | and simplicity | 04:18 |
| clarkb | but don't let me ruin your dreams of commuting every morning from frankfurt westin to darmstadt | 04:18 |
| lifeless | whats in darmstadt | 04:19 |
| clarkb | infra/qa mid cycle meetup | 04:19 |
| openstackgerrit | A change was merged to openstack-infra/config: Add Designate DevStack/Requirements/Docs Jobs https://review.openstack.org/97348 | 04:19 |
| clarkb | ooh I wonder if darmstadt is situated in a ttx friendly location | 04:20 |
| *** dims_ has quit IRC | 04:20 | |
| *** Hunner has quit IRC | 04:22 | |
| mordred | clarkb: I think he does not want to join due to proximity to oscon | 04:23 |
| *** trinaths has joined #openstack-infra | 04:23 | |
| * tchaypo learns where is ttx? | 04:24 | |
| * tchaypo glares at weechat | 04:24 | |
| tchaypo | lifeless: the other thing that's in darmstadt is a deutsche telekom office | 04:25 |
| clarkb | mordred: :( that is understandable I guess | 04:25 |
| tchaypo | lifeless: not unrelatedly | 04:25 |
| pleia2 | clarkb: staying in darmstadt is probably my preference as well, because yay sleep | 04:25 |
| *** sballe has joined #openstack-infra | 04:25 | |
| clarkb | pleia2: ya I hope to have 20 minutes tomorrow to book the trip and will look for hotel in darmstadt | 04:26 |
| jogo | pleia2: won't jetleg mess that up anyway | 04:26 |
| clarkb | tchaypo: you guys have something there too? | 04:26 |
| *** lcheng_ has joined #openstack-infra | 04:26 | |
| clarkb | jogo: that is why every minute counts | 04:26 |
| pleia2 | clarkb: cool, please lmk what you end up with | 04:26 |
| jogo | clarkb: haha | 04:27 |
| clarkb | adding ~1.5 hours of roundtrip every day commuting is time I could sleep | 04:27 |
| pleia2 | jogo: I'm still a bit under the weather, so being able to return to hotel quickly/by myself is helpful :) | 04:27 |
| clarkb | or try to sleep | 04:27 |
| clarkb | pleia2: will do | 04:27 |
| clarkb | last I looked condor and lufthansa/united have direct flights from seatac but condor is cheaper and not united so I may giev them a try | 04:27 |
| *** plars is now known as plars-away | 04:27 | |
| pleia2 | lufthansa is nice, that's what I'm taking to croatia next weekend | 04:28 |
| clarkb | then I was going to try and do one of the two hotels on the wiki page | 04:28 |
| pleia2 | but united co-chairs with them, so have to be careful :) | 04:28 |
| pleia2 | (can tell when you book, *fine print) | 04:28 |
| clarkb | I will have to look closely then | 04:28 |
| *** harlowja_at_home has joined #openstack-infra | 04:28 | |
| *** Longgeek has joined #openstack-infra | 04:28 | |
| tchaypo | clarkb: no | 04:31 |
| *** sballe has quit IRC | 04:32 | |
| *** CSheedy has quit IRC | 04:32 | |
| *** cody-somerville has joined #openstack-infra | 04:35 | |
| clarkb | mordred: tminus 8 minutes (but no big deal if you have fallen asleep) | 04:35 |
| mordred | clarkb: I'm still here | 04:35 |
| *** jcoufal has joined #openstack-infra | 04:36 | |
| *** matsuhashi has joined #openstack-infra | 04:39 | |
| *** harlowja_at_home has quit IRC | 04:40 | |
| clarkb | mordred: now is a good time | 04:41 |
| clarkb | gate just reset | 04:41 |
| *** nosnos has joined #openstack-infra | 04:41 | |
| mordred | clarkb: going for it | 04:41 |
| mordred | clarkb: zuul promote 97924,4 97757,1 yeah? | 04:41 |
| clarkb | mordred: zuul promot --pipeline gate --changes 97924,4 97757,1 | 04:42 |
| clarkb | as root too | 04:42 |
| clarkb | so that it can read the zuul.confr | 04:42 |
| clarkb | er zuul promote | 04:42 |
| mordred | done | 04:42 |
| clarkb | that has an e at the end | 04:42 |
| clarkb | woot thanks | 04:42 |
| mordred | yeah - i was mainly just making sure those were the changes | 04:42 |
| mordred | and the order | 04:42 |
| clarkb | yup thats the order | 04:43 |
| clarkb | 97924 fixes the time in the future bug and 97757 fixes lifeless' bug | 04:43 |
| *** renlt has quit IRC | 04:46 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Add devstack-f20 to hpcloud https://review.openstack.org/98024 | 04:46 |
| *** markwash has quit IRC | 04:56 | |
| SergeyLukjanov | oh, my ... | 04:59 |
| SergeyLukjanov | >1000 msgs in scrollback | 04:59 |
| SergeyLukjanov | morning/evening folks | 04:59 |
| clarkb | SergeyLukjanov: o/ | 05:01 |
| SergeyLukjanov | anteaya, yeah, I like the proposed name pattern too | 05:01 |
| clarkb | SergeyLukjanov: they gate is a bit cranky. There is a devstack change that we semi expect sdague to merge early in the morning his time that may need promotion | 05:01 |
| clarkb | SergeyLukjanov: apparently he is the only devstack core that isn't on vacation and he wrote the change so we will see what happens :) | 05:02 |
| SergeyLukjanov | clarkb, okay, got it | 05:03 |
| *** Longgeek has quit IRC | 05:06 | |
| *** dkehn has quit IRC | 05:06 | |
| *** Longgeek has joined #openstack-infra | 05:07 | |
| *** yfried has joined #openstack-infra | 05:07 | |
| *** basha has joined #openstack-infra | 05:07 | |
| *** dkehn has joined #openstack-infra | 05:08 | |
| *** marcoemorais has joined #openstack-infra | 05:17 | |
| *** marcoemorais1 has joined #openstack-infra | 05:19 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Fix up status page dependencies https://review.openstack.org/98028 | 05:19 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Use the latest jquery on zuul https://review.openstack.org/98029 | 05:20 |
| clarkb | SergeyLukjanov: jhesketh if 97910 looks good to you guys I think that would help reduce some of the test slave pressure we have had the past few days and make our gate more robust | 05:21 |
| *** marcoemorais has quit IRC | 05:21 | |
| SergeyLukjanov | clarkb, ack | 05:22 |
| *** hunner has joined #openstack-infra | 05:22 | |
| *** hunner is now known as Hunner | 05:22 | |
| jhesketh | clarkb: +2 | 05:22 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove markdown-docbook https://review.openstack.org/97106 | 05:23 |
| SergeyLukjanov | clarkb, +A | 05:23 |
| clarkb | cool thnaks | 05:23 |
| clarkb | jogo: ^ if you are still around | 05:23 |
| clarkb | but that is two less tests per nova patch | 05:23 |
| clarkb | also fyi I am going to approve 96907 which won't go into affect until after tomorrow morning (PDT) image builds | 05:26 |
| clarkb | so it should be safish :) | 05:26 |
| clarkb | that should clean up a lot of the logs around sudo and be less confusing to people reading them | 05:28 |
| openstackgerrit | A change was merged to openstack-infra/config: Bump LARGE_OPS number from 50=>100 https://review.openstack.org/97910 | 05:28 |
| *** marun_afk is now known as marun | 05:29 | |
| * SergeyLukjanov brb in 3 hours | 05:30 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add the new hostname to /etc/hosts in prepare_node https://review.openstack.org/96907 | 05:35 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Force ntpdate before we start the timer https://review.openstack.org/97924 | 05:41 |
| clarkb | woot | 05:41 |
| clarkb | though 97757 failed again | 05:41 |
| clarkb | but at least one gate bug killed now I can fall asleep | 05:41 |
| *** Sukhdev has quit IRC | 05:41 | |
| mordred | clarkb: woot | 05:44 |
| lifeless | mordred: so you'll handle 97757 ? :) | 05:47 |
| *** mdenny has quit IRC | 05:51 | |
| *** arnaud has joined #openstack-infra | 05:51 | |
| *** mdenny has joined #openstack-infra | 05:51 | |
| *** miqui has quit IRC | 05:54 | |
| *** jcoufal has quit IRC | 05:56 | |
| *** oomichi has quit IRC | 06:02 | |
| *** gyee has quit IRC | 06:04 | |
| *** andreykurilin_ has joined #openstack-infra | 06:05 | |
| *** johnthetubaguy has quit IRC | 06:05 | |
| *** jraim has quit IRC | 06:05 | |
| *** mancdaz has quit IRC | 06:06 | |
| *** johnthetubaguy has joined #openstack-infra | 06:06 | |
| *** simonmcc has quit IRC | 06:06 | |
| *** jraim has joined #openstack-infra | 06:06 | |
| *** mancdaz has joined #openstack-infra | 06:08 | |
| *** simonmcc has joined #openstack-infra | 06:08 | |
| *** flaper87|afk is now known as flaper87 | 06:08 | |
| mattoliverau | Night all, I'm off for the day! | 06:09 |
| *** srenatus has quit IRC | 06:11 | |
| *** srenatus has joined #openstack-infra | 06:11 | |
| mordred_phone | clarkb: I pushed up a new still not working nodepool patch | 06:13 |
| *** marun is now known as marun_afk | 06:14 | |
| *** renlt has joined #openstack-infra | 06:16 | |
| *** trinaths has quit IRC | 06:20 | |
| clarkb | jesusaurus: can you propose your hostname fix to d-g as we | 06:23 |
| clarkb | *as well | 06:23 |
| clarkb | mordred: cool | 06:23 |
| *** harlowja_at_home has joined #openstack-infra | 06:27 | |
| harlowja_at_home | qq, whats up with the gate zuul job queue situation, is it getting better? | 06:27 |
| *** Longgeek has quit IRC | 06:27 | |
| clarkb | not really. I think grenade/tempesr are having trouble | 06:28 |
| harlowja_at_home | hmmm, durn | 06:28 |
| harlowja_at_home | out of curosity, are there less workers in the gate queue (interesting to see that only like 5 workers seem to be processing jobs?) | 06:29 |
| * harlowja_at_home watching status.openstack.org/zuul/ | 06:30 | |
| harlowja_at_home | is there anyway to help to make it better? | 06:31 |
| harlowja_at_home | *excluding beating with stick, haha | 06:31 |
| SlickNik | Can I get some eyeballs on https://review.openstack.org/#/c/97608/ when folks get a chance? Thanks much! | 06:33 |
| *** hashar has joined #openstack-infra | 06:33 | |
| *** rgerganov has joined #openstack-infra | 06:34 | |
| *** rdopiera has joined #openstack-infra | 06:38 | |
| *** tkelsey has joined #openstack-infra | 06:39 | |
| *** lttrl has quit IRC | 06:40 | |
| *** andreykurilin_ has quit IRC | 06:40 | |
| *** andreykurilin_ has joined #openstack-infra | 06:41 | |
| jesusaurus | clarkb: i see https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L153 | 06:43 |
| jesusaurus | clarkb: what change were you thinking? | 06:44 |
| *** srenatus has quit IRC | 06:44 | |
| *** srenatus has joined #openstack-infra | 06:44 | |
| clarkb | hrm is that called early? | 06:48 |
| clarkb | I want to say maybe it isnt early enough to prevent sudo from complaining | 06:49 |
| clarkb | harlowja that is intentional. the gate operates on a window | 06:49 |
| mordred_phone | clarkb: we really do need to write a bunch more about concepts and whys | 06:52 |
| harlowja_at_home | *for curious folks like me ;) | 06:52 |
| *** lcheng_ has quit IRC | 06:52 | |
| jesusaurus | mordred_phone: hopefully some of that will end up in infra-specs | 06:53 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Keep the hierarchy of tripleo files copied to log server https://review.openstack.org/98043 | 06:53 |
| mordred_phone | jesusaurus: ++ | 06:53 |
| mordred_phone | and infra-manuals | 06:54 |
| *** freyes has joined #openstack-infra | 06:54 | |
| harlowja_at_home | clarkb, window being used so that only X number of changes can cause problems at the same time (if the window is small then its easier to identify what root caused an issue) | 06:54 |
| *** andreykurilin_ has quit IRC | 06:56 | |
| *** Longgeek has joined #openstack-infra | 06:58 | |
| *** camunoz has quit IRC | 07:00 | |
| jesusaurus | harlowja_at_home: well, a window to try to minimize the number of vms active (not that we can minimize the number of vms needed) | 07:00 |
| *** freyes has quit IRC | 07:01 | |
| harlowja_at_home | jesusaurus, gotcha, what happens if more vms become active? (bad things?) | 07:01 |
| *** Longgeek has quit IRC | 07:02 | |
| jesusaurus | well, some people have to pay more for that, but generally no cloud is limitless | 07:04 |
| ttx | clarkb: Germany is never a ttx-friendly location :) | 07:05 |
| ttx | I need to look up if i can join you all but it's a bit unlikely as i have a conference in Montpellier just befor | 07:06 |
| ttx | e | 07:06 |
| *** AaronGr has quit IRC | 07:06 | |
| *** bradm has quit IRC | 07:07 | |
| *** wenlock has quit IRC | 07:11 | |
| *** AaronGr has joined #openstack-infra | 07:13 | |
| *** maxbit has quit IRC | 07:14 | |
| *** harlowja_at_home has quit IRC | 07:15 | |
| *** oomichi has joined #openstack-infra | 07:16 | |
| *** mattt has quit IRC | 07:17 | |
| *** bradm has joined #openstack-infra | 07:22 | |
| hashar | at worth you could fly from montpellier to germany http://www.montpellier.aeroport.fr/fr/destinations/carte-des-destinations :D | 07:23 |
| lifeless | mordred_phone: ping | 07:24 |
| *** jlibosva has joined #openstack-infra | 07:25 | |
| mordred_phone | lifeless: I don't believe in ping | 07:25 |
| *** jp_at_hp has joined #openstack-infra | 07:25 | |
| lifeless | mordred_phone: 97757 needs to be requeued up the top - | 07:25 |
| lifeless | mordred_phone: its blocking ironic and tripleo checks | 07:25 |
| lifeless | mordred_phone: (its the one clarkb handed over to you :)) | 07:26 |
| mordred_phone | lifeless: k. I'm at a bar. SergeyLukjanov , around? | 07:26 |
| mordred_phone | harlowja_at_home: the window is adaptive to the gate health | 07:27 |
| *** doude has joined #openstack-infra | 07:28 | |
| lifeless | mordred_phone: at 1pm ? :) | 07:30 |
| mordred_phone | lifeless: at 3:30am | 07:30 |
| lifeless | mordred_phone: oh, you're stateside again? | 07:30 |
| mordred_phone | lifeless: this bar has the best deviled eggs | 07:31 |
| lifeless | mmm | 07:31 |
| mordred_phone | lifeless: hell yes | 07:31 |
| mordred_phone | lifeless: in the NY cizzile | 07:31 |
| hashar | they look good https://en.wikipedia.org/wiki/Deviled_egg#mediaviewer/File:Deviled_Eggs_-_3-23-08.jpg | 07:32 |
| hashar | jhesketh: if you are around, I got a Zuul patch to loosen its dependencies to python-swiftclient ( https://review.openstack.org/#/c/97933/ ) | 07:33 |
| *** flaper87 is now known as flaper87|afk | 07:39 | |
| *** ihrachyshka has joined #openstack-infra | 07:42 | |
| *** arnaud has quit IRC | 07:42 | |
| *** Guest22618 has joined #openstack-infra | 07:44 | |
| *** mrda is now known as mrda-away | 07:44 | |
| *** skolekonov has joined #openstack-infra | 07:48 | |
| *** andreaf has joined #openstack-infra | 07:50 | |
| *** salv-orlando has joined #openstack-infra | 07:51 | |
| *** apevec has joined #openstack-infra | 07:52 | |
| *** jistr has joined #openstack-infra | 07:53 | |
| apevec | ugh, top of gate is >28h | 07:54 |
| apevec | not a good start for 2014.1.1 release day | 07:54 |
| *** Longgeek has joined #openstack-infra | 07:55 | |
| *** _nadya_ has joined #openstack-infra | 07:59 | |
| *** pelix has joined #openstack-infra | 08:01 | |
| *** tcammann has joined #openstack-infra | 08:04 | |
| *** e0ne has joined #openstack-infra | 08:06 | |
| *** Longgeek has quit IRC | 08:10 | |
| *** basha has quit IRC | 08:11 | |
| *** Longgeek has joined #openstack-infra | 08:11 | |
| *** dizquierdo has joined #openstack-infra | 08:13 | |
| *** Longgeek has quit IRC | 08:16 | |
| *** ominakov has joined #openstack-infra | 08:17 | |
| boris-42 | jhesketh hey there | 08:18 |
| *** Longgeek has joined #openstack-infra | 08:19 | |
| *** e0ne has quit IRC | 08:19 | |
| *** viktors|afk has joined #openstack-infra | 08:19 | |
| ttx | apevec: it certainly seems to be struggling a bit | 08:20 |
| *** jcoufal has joined #openstack-infra | 08:27 | |
| *** jcoufal has quit IRC | 08:27 | |
| *** jcoufal has joined #openstack-infra | 08:28 | |
| mordred | hashar: they're very tasty | 08:31 |
| *** mkoderer has quit IRC | 08:32 | |
| apevec | ttx, btw please proof read relnote draft https://wiki.openstack.org/wiki/ReleaseNotes/2014.1.1 | 08:32 |
| apevec | tristanC, ^ please double check I included all OSSAs | 08:33 |
| hashar | mordred: i can imagine :) | 08:33 |
| mordred | hashar: the ones at ward iii come with a slice of olive and a slice of jalapeño on top | 08:34 |
| * hashar looks up jalapeño | 08:35 | |
| ttx | apevec: on it | 08:35 |
| openstackgerrit | Christian Berendt proposed a change to openstack-dev/hacking: Add H804: enforce title case in commit messages https://review.openstack.org/98059 | 08:35 |
| *** trinaths has joined #openstack-infra | 08:36 | |
| hashar | mordred: if I ever come to NYC, I will definitely attempt to meet you up in that bar to taste those devil eggsç | 08:36 |
| ttx | apevec: I'd say "80 bugs" rather than "80 launchpad bugs" -- otherwise looks good | 08:38 |
| ttx | apevec: also checked the OSSA list and it looks good | 08:38 |
| apevec | thanks | 08:40 |
| *** prad has quit IRC | 08:44 | |
| apevec | ttx, looks like "launchpad bugs" was copied since https://wiki.openstack.org/wiki/ReleaseNotes/2011.3.1 | 08:45 |
| apevec | but no time like now to change the tradition | 08:45 |
| apevec | s/tradition/copy&paste/ | 08:45 |
| *** viktors|afk is now known as viktors | 08:47 | |
| ttx | apevec: I can update it if you aren't in the middle of an edit | 08:47 |
| ttx | oh, you fixed it already | 08:47 |
| jhesketh | boris-42: Hey | 08:50 |
| boris-42 | jhesketh could you explain your comment sorry | 08:50 |
| apevec | ttx, yeah, was just doing wiki archaeology :) | 08:50 |
| boris-42 | jhesketh I didn't get it.. | 08:50 |
| jhesketh | hashar: is that swiftclient ready to review? | 08:50 |
| boris-42 | jhesketh this one https://review.openstack.org/#/c/97778/ | 08:50 |
| jhesketh | boris-42: the format of your gerrit ACL is wrong | 08:51 |
| *** basha has joined #openstack-infra | 08:51 | |
| boris-42 | jhesketh I mean about tag | 08:51 |
| jhesketh | boris-42: I don't follow sorry | 08:51 |
| boris-42 | jhesketh this comment https://review.openstack.org/#/c/97778/2/modules/openstack_project/files/review.projects.yaml | 08:52 |
| boris-42 | jhesketh not first one | 08:52 |
| jhesketh | oh right, sorry | 08:52 |
| jhesketh | boris-42: there is a git tag on that repository | 08:52 |
| *** hashar_ has joined #openstack-infra | 08:52 | |
| jhesketh | it will be imported into the cloned repository | 08:52 |
| boris-42 | jhesketh so probably I should just remove it? | 08:52 |
| *** hashar has quit IRC | 08:52 | |
| boris-42 | jhesketh with push --force | 08:52 |
| jhesketh | tags generate releases from the tarball jobs | 08:52 |
| jhesketh | (I'm not sure if it would retroactively, I'd have to check) | 08:53 |
| jhesketh | boris-42: well it depends if you want the tag or not. I'd say it probably wouldn't hurt to leave it, but I was point it out to you | 08:53 |
| boris-42 | jhesketh I mean it's not a big deal to remove osprofiler from pypi | 08:53 |
| boris-42 | jhesketh and remove tag | 08:53 |
| jhesketh | that shouldn't be necessary | 08:53 |
| boris-42 | jhesketh btw what about osprofiler on pypi | 08:54 |
| boris-42 | jhesketh should I remove it, to be able to publish tags? | 08:54 |
| *** unicell has quit IRC | 08:54 | |
| boris-42 | jhesketh and make new versions via openstack ci | 08:54 |
| *** Alexei_987 has quit IRC | 08:54 | |
| jhesketh | boris-42: to be honest I'm not sure. I don't think it matters. I'd have to double check what the tarball jobs do, but I'm late for meeting some people so I don't have time at the moment sorry | 08:57 |
| jhesketh | Perhaps somebody else in here knows? | 08:57 |
| jhesketh | otherwise I'll take a look tomorrow | 08:57 |
| boris-42 | jhesketh I'll just get rid of tag & lib | 08:57 |
| boris-42 | jhesketh it will work 100% lol=) | 08:57 |
| boris-42 | jhesketh and it will take just a couple of minute | 08:58 |
| *** achuprin has quit IRC | 08:58 | |
| *** oomichi has quit IRC | 09:00 | |
| *** jooools has joined #openstack-infra | 09:01 | |
| *** marcoemorais1 has quit IRC | 09:05 | |
| *** achuprin has joined #openstack-infra | 09:13 | |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Finally add osprofiler to stackforge https://review.openstack.org/97778 | 09:15 |
| *** hashar_ is now known as hashar | 09:16 | |
| *** zhiyan is now known as zhiyan_ | 09:18 | |
| *** flaper87|afk is now known as flaper87 | 09:19 | |
| *** e0ne has joined #openstack-infra | 09:21 | |
| *** dizquierdo is now known as dizquierdo_out | 09:21 | |
| *** _nadya_ has quit IRC | 09:29 | |
| SergeyLukjanov | I'm here now | 09:31 |
| SergeyLukjanov | lifeless, mordred_phone | 09:31 |
| SergeyLukjanov | I presume that 97757 is still needed | 09:32 |
| SergeyLukjanov | so, promoting it to the top of queue | 09:32 |
| lifeless | thanks | 09:32 |
| *** oomichi has joined #openstack-infra | 09:32 | |
| SergeyLukjanov | lifeless, oh, you're still not sleeping ;) | 09:32 |
| lifeless | I should be soon:) | 09:33 |
| *** jgallard has joined #openstack-infra | 09:33 | |
| SergeyLukjanov | lifeless, done | 09:34 |
| SergeyLukjanov | sdague, do you have a world-fixing CR to devstack? :) | 09:35 |
| *** derekh_ has joined #openstack-infra | 09:37 | |
| *** flaper87 is now known as flaper87|afk | 09:38 | |
| *** srenatus has quit IRC | 09:41 | |
| *** denis_makogon has joined #openstack-infra | 09:42 | |
| *** srenatus has joined #openstack-infra | 09:42 | |
| Kiall | Hey guys - This commit to add a few new jobs landed recently, but the gate-designate-requirements jobs that were enabled, and which exist in jenkins, are not being triggered. Thoughts? https://github.com/openstack-infra/config/commit/33d7410017284916fb7f8a7ab2bcabb0eef64adf | 09:45 |
| Kiall | The other jobs it added (devstack and -docs) have both been enabled. | 09:46 |
| sdague | SergeyLukjanov: just getting coffee ... is there a specific bug to fix, or all the bugs? | 09:50 |
| SergeyLukjanov | sdague, there were some msgs re you is the last one devstack core and will self approve some fix today's morning :) | 09:51 |
| SergeyLukjanov | lifeless, 97757,1 just failed the py26 test | 09:52 |
| sdague | SergeyLukjanov: so there is this one - https://review.openstack.org/#/c/97912/ | 09:52 |
| sdague | which actually has some +2s on it | 09:52 |
| Kiall | Ignore me - There is a path filter on the requirements jobs - I just spotted it :) | 09:52 |
| SergeyLukjanov | lifeless, https://jenkins04.openstack.org/job/gate-nova-python26/2753/console | 09:52 |
| *** mkoderer has joined #openstack-infra | 09:53 | |
| SergeyLukjanov | sdague, nice, +1 from me too | 09:54 |
| SergeyLukjanov | sdague, sounds like it should be promoted too | 09:54 |
| sdague | SergeyLukjanov: I'll start listing stuff here - https://etherpad.openstack.org/p/gatetriage-june2014 | 09:54 |
| SergeyLukjanov | sdague, could you please take a look at nova unit tests failure? | 09:55 |
| SergeyLukjanov | 97757,1 has been promoted to unblock ironic/tripleo | 09:56 |
| SergeyLukjanov | but it fails the py26 btest | 09:56 |
| sdague | SergeyLukjanov: sure thing | 09:57 |
| *** basha has left #openstack-infra | 09:59 | |
| sdague | SergeyLukjanov: it appears to be a longish standing unit test issue - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIk1lc3NhZ2luZ1RpbWVvdXQ6IE5vIHJlcGx5IG9uIHRvcGljIHNjaGVkdWxlclwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDAxOTYyMjg4MzEyfQ== | 09:59 |
| *** dkehn has quit IRC | 10:00 | |
| *** basha has joined #openstack-infra | 10:01 | |
| sdague | SergeyLukjanov: it's https://bugs.launchpad.net/nova/+bug/1311778 | 10:02 |
| uvirtbot | Launchpad bug 1311778 in nova "Unit tests fail with MessagingTimeout errors" [Critical,New] | 10:02 |
| *** dkehn has joined #openstack-infra | 10:02 | |
| *** rlandy has joined #openstack-infra | 10:04 | |
| *** jgallard has quit IRC | 10:06 | |
| *** ildikov has quit IRC | 10:07 | |
| *** ildikov has joined #openstack-infra | 10:07 | |
| SergeyLukjanov | sdague, so, we could just re-promote | 10:07 |
| SergeyLukjanov | sdague, am I right? | 10:07 |
| sdague | we could | 10:09 |
| sdague | so, honestly, I'm back to my theory that we should dump the gate queue | 10:10 |
| sdague | because I realized something else with the grenade fix | 10:10 |
| sdague | we've now upped concurrency on all the jobs | 10:10 |
| sdague | as we no longer have 4 core nodes | 10:10 |
| *** yamahata has quit IRC | 10:10 | |
| sdague | and we have a set of things in the gate that passed on a 4 core node | 10:11 |
| sdague | but have a race | 10:11 |
| SergeyLukjanov | heh | 10:11 |
| sdague | once there are tests from this - https://review.openstack.org/97966 we'll want to promote | 10:11 |
| *** ihrachyshka has quit IRC | 10:12 | |
| sdague | SergeyLukjanov: so realistically ironic is failing it's own unit tests right now | 10:12 |
| SergeyLukjanov | bah | 10:12 |
| sdague | which means I don't think we should bother with a promote on 97757 until that's resolved | 10:13 |
| sdague | unless it's the same thing | 10:13 |
| sdague | is there a bug or any context that explains this? | 10:13 |
| sdague | lifeless: do you know about the ironic unit tests and if they are linked to this? | 10:14 |
| SergeyLukjanov | I don't see anything related | 10:14 |
| SergeyLukjanov | to it | 10:14 |
| *** hashar has quit IRC | 10:14 | |
| srenatus | is there a way to increase the timeout for a job? with chef cookbooks and extensive chefspec runs, the 30m limit is hit quite often | 10:14 |
| sdague | those ironic unit test breaks are at least a day old, as I was manually pulling all their changes out of the queue | 10:15 |
| SergeyLukjanov | so, we have an option to make a zuul reset, promote all important changes, send email with excuse | 10:15 |
| sdague | SergeyLukjanov: yeh, if you start etherpad drafting things I'll help with the email | 10:15 |
| sdague | srenatus: yes, it's configured in the job definition in the openstack-infra/config repo | 10:16 |
| sdague | SergeyLukjanov: because we kind of need an all hands on deck here as well | 10:16 |
| *** denis_makogon has quit IRC | 10:17 | |
| SergeyLukjanov | sdague, " if you start etherpad drafting things" which things? | 10:17 |
| SergeyLukjanov | you mean CRs that should be promoted? | 10:17 |
| sdague | SergeyLukjanov: no, the email explaining to people why we've dumped the gate | 10:19 |
| SergeyLukjanov | sdague, oh, got it | 10:19 |
| srenatus | sdague: thanks, found that devstack-gate.yaml serves as an example how to override it | 10:23 |
| sdague | srenatus: that's probably not quite the right thing | 10:23 |
| sdague | let me get you a more specific link | 10:23 |
| *** renlt has quit IRC | 10:24 | |
| srenatus | sdague: oh ok... thanks | 10:25 |
| sdague | srenatus: actually, maybe you did find it | 10:25 |
| sdague | I didn't realize that was only defined in the wrappers part | 10:26 |
| sdague | yeh, I'd suggest proposing a patch with the fix you think based on reading devstack-gate.yaml, and SergeyLukjanov will correct it if need be :) | 10:27 |
| SergeyLukjanov | ack | 10:27 |
| SergeyLukjanov | sdague, I'm started writing email draft in your etherpad | 10:30 |
| sdague | ok | 10:30 |
| SpamapS | sdague: so is the gate just so far gone that we need to dump it? | 10:31 |
| sdague | SpamapS: yeh, I think so | 10:31 |
| *** matsuhashi has quit IRC | 10:31 | |
| sdague | especially with the concurency change | 10:32 |
| sdague | a bunch of stuff made it over a 4 ft bar, and ran into an 8 ft brick wall | 10:32 |
| sdague | it's a side effect of hp cloud 1.0 => 1.1 that we didn't quite notice | 10:33 |
| *** matsuhashi has joined #openstack-infra | 10:34 | |
| sdague | though in 12 minutes all the heat changes will dump out anyway, because the 17 patch series has a merge conflict | 10:34 |
| sdague | actually that's a 21 patch series? | 10:35 |
| *** Guest22618 has quit IRC | 10:36 | |
| sdague | SergeyLukjanov: once the trove test lands, and the heat series falls out, we should do the promotes. We can then work on the email while those are churning. | 10:36 |
| SpamapS | sdague: oh, what if I were to push a rebase of that? | 10:36 |
| sdague | SpamapS: it would take it out of the gate early | 10:37 |
| SpamapS | I knew that series would be trouble. :-P | 10:37 |
| sdague | which would also be fine | 10:37 |
| SergeyLukjanov | sdague, okay | 10:37 |
| sdague | honestly, that series will consume all the gate resources we've got for a couple of hours | 10:37 |
| sdague | long series like that are a mild dos attack | 10:38 |
| sdague | though we have the nodes now | 10:38 |
| SpamapS | sdague: I don't think zaneb though we'd approve the whole thing in 2 days | 10:38 |
| SpamapS | thought | 10:38 |
| sdague | yeh, that's fair. It's just funny how different different teams work some times :) | 10:39 |
| sdague | the heat team loves everything in long series | 10:39 |
| sdague | when reviewing their changes to tempest I usually have to ask them to collapse a few things | 10:39 |
| SergeyLukjanov | sdague, 97966 already failed it's grenade job :( | 10:42 |
| sdague | SergeyLukjanov: ok, let me figure out what I did wrong this time | 10:42 |
| *** praneshp has quit IRC | 10:42 | |
| *** praneshp has joined #openstack-infra | 10:44 | |
| *** praneshp has quit IRC | 10:45 | |
| SpamapS | oh ugh | 10:45 |
| SpamapS | that merge conflict is _nasty | 10:45 |
| SpamapS | I'm going to let ZaneB work that one out | 10:45 |
| SergeyLukjanov | sdague, trove tests just merged | 10:46 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: set concurrency for grenade https://review.openstack.org/98080 | 10:47 |
| *** jooools has quit IRC | 10:47 | |
| sdague | SergeyLukjanov: turns out we need that one too | 10:47 |
| sdague | can you look at that? | 10:48 |
| SergeyLukjanov | sdague, yup, sure, but it was lgtm last time for me | 10:48 |
| sdague | the grenade one failed because it was running 8 way again. Basically it's not passing the control var | 10:48 |
| sdague | that's the d-g change to pass it | 10:48 |
| SergeyLukjanov | sdague, oh, yeah, see it | 10:48 |
| *** jooools has joined #openstack-infra | 10:49 | |
| sdague | so the grenade fail was unrelated to the grenade change | 10:49 |
| sdague | it's just that 8 way testing doesn't work very much | 10:49 |
| *** basha has quit IRC | 10:50 | |
| *** jooools has quit IRC | 10:50 | |
| SergeyLukjanov | sdague, it's lgtm | 10:50 |
| sdague | so, honestly, I'd say promote the top 4 in the list | 10:51 |
| sdague | even though these 2 haven't gotten through check yet | 10:51 |
| SergeyLukjanov | "To Promote" section to promote ;) | 10:51 |
| sdague | yep | 10:51 |
| SergeyLukjanov | sdague, ^^ | 10:51 |
| sdague | which means moving these 2 to gate queue as well | 10:51 |
| *** basha has joined #openstack-infra | 10:51 | |
| *** om has joined #openstack-infra | 10:52 | |
| SergeyLukjanov | sdague, yup, working on promote commands | 10:54 |
| *** e0ne_ has joined #openstack-infra | 10:54 | |
| sdague | awesome, thanks | 10:55 |
| *** talluri has quit IRC | 10:55 | |
| sdague | also... it's incredibly awesome to have someone that can do all these things when I'm up in the morning :) | 10:55 |
| sdague | \o/ for global infra team | 10:56 |
| *** talluri has joined #openstack-infra | 10:56 | |
| *** talluri has quit IRC | 10:56 | |
| sdague | as I'm looking at the things which are bouncing of random bugs, these 4 should address a bunch of them | 10:56 |
| *** talluri has joined #openstack-infra | 10:56 | |
| sdague | so we might start picking up speed in the gate after them | 10:56 |
| sdague | so I'll reserve my gate dump idea for now | 10:57 |
| sdague | we might just have to pound these 4 through though | 10:57 |
| SpamapS | sdague: what was the hpcloud 1.1 change that we're not dealing with btw? | 10:57 |
| * SpamapS should be asleep.. stupid indian jet lag won't let go 5 days later. | 10:58 | |
| *** e0ne has quit IRC | 10:58 | |
| SergeyLukjanov | sdague, I'm glad to help ;) | 10:58 |
| SergeyLukjanov | sdague, so, all changes are presented in gate now, promoting them in the order like in etherpad | 10:58 |
| SergeyLukjanov | you have ~90 secs to stop me | 10:59 |
| sdague | SpamapS: mon 1.0 shut off | 10:59 |
| sdague | SergeyLukjanov: let me look | 10:59 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Name fields checked with regex https://review.openstack.org/95763 | 10:59 |
| SergeyLukjanov | sdague, sudo zuul promote --pipeline gate --changes 97912,2 97845,2 98080,1 97966,2 | 11:00 |
| sdague | yeh, that seems right | 11:00 |
| SpamapS | sdague: OH | 11:00 |
| sdague | SpamapS: so we're on 1.1 | 11:00 |
| SergeyLukjanov | sdague, probably we could move grenade fix to the top to avoid grenade failures | 11:00 |
| SpamapS | sdague: so just "scramble oops firedrill new things go" | 11:00 |
| sdague | SpamapS: well that was mon/tues | 11:00 |
| SergeyLukjanov | sdague, or it's not so bad to fail every time? | 11:00 |
| sdague | SergeyLukjanov: honestly, lets try it as is | 11:00 |
| SergeyLukjanov | ack | 11:00 |
| SpamapS | sdague: right, and Wed was "find and extinguish embers" | 11:00 |
| sdague | SpamapS: well, it led to an interesting new edge case | 11:01 |
| sdague | we were running at half capacity for 2 days | 11:01 |
| SpamapS | sdague: more interesting than "the spamaps hole"? | 11:01 |
| sdague | so the check queue got to 500 changes | 11:01 |
| sdague | and people just kept approving things | 11:01 |
| sdague | so when nodes showed up, in combination with the "the spamaps hole", we got this giant tidal wave into the gate | 11:02 |
| SpamapS | ahh | 11:02 |
| SergeyLukjanov | sdague, promoted | 11:02 |
| sdague | SergeyLukjanov: thanks | 11:02 |
| SergeyLukjanov | sdague, np | 11:03 |
| SergeyLukjanov | I'd like to see gate alive as well as you :) | 11:03 |
| SergeyLukjanov | apevec, do we have a lot of pending CRs in gate to release 2014.1.1? | 11:03 |
| SpamapS | sdague: I'm going to go out on a limb and say tidal waves are bad. | 11:04 |
| SergeyLukjanov | apevec, we have 6 in sahara btw | 11:04 |
| *** dprince has joined #openstack-infra | 11:04 | |
| SpamapS | ok.. time to attempt sleep again | 11:05 |
| *** matsuhashi has quit IRC | 11:06 | |
| *** palar has joined #openstack-infra | 11:06 | |
| *** matsuhashi has joined #openstack-infra | 11:06 | |
| *** matsuhashi has quit IRC | 11:10 | |
| SergeyLukjanov | sdague, time to have a short meeting and big coffee | 11:12 |
| *** yamahata has joined #openstack-infra | 11:15 | |
| *** om has quit IRC | 11:15 | |
| *** ihrachyshka has joined #openstack-infra | 11:19 | |
| *** lcostantino has joined #openstack-infra | 11:19 | |
| *** dizquierdo_out is now known as dizquierdo | 11:19 | |
| *** nosnos has quit IRC | 11:20 | |
| *** talluri has quit IRC | 11:22 | |
| apevec | SergeyLukjanov, there are only 2 Neutron 2014.1.1 exceptions in the gate: 96919 and its child 96920 | 11:24 |
| *** matsuhashi has joined #openstack-infra | 11:24 | |
| *** matsuhashi has quit IRC | 11:24 | |
| apevec | 96919 is still all queued only | 11:24 |
| apevec | for >19h | 11:24 |
| *** matsuhashi has joined #openstack-infra | 11:24 | |
| *** mmaglana has joined #openstack-infra | 11:27 | |
| *** ihrachyshka has quit IRC | 11:27 | |
| *** ihrachyshka has joined #openstack-infra | 11:27 | |
| SergeyLukjanov | apevec, heh, I'm not sure that it'll be merged today :( | 11:28 |
| apevec | :( | 11:29 |
| *** matsuhashi has quit IRC | 11:29 | |
| apevec | SergeyLukjanov, fix the gate :) | 11:30 |
| *** HenryG has joined #openstack-infra | 11:32 | |
| *** om has joined #openstack-infra | 11:34 | |
| *** talluri has joined #openstack-infra | 11:35 | |
| sdague | SpamapS: enjoy the coffee | 11:37 |
| sdague | apevec: people need to fix openstack :) | 11:37 |
| *** lbragstad has quit IRC | 11:38 | |
| *** dprince has quit IRC | 11:39 | |
| sdague | SergeyLukjanov: 98080,1 ended up failing because the other grenade patch wasn't there | 11:39 |
| sdague | we might just want to ninja merge those 2 | 11:39 |
| sdague | but that can wait until after coffee | 11:40 |
| SergeyLukjanov | I'm here now | 11:40 |
| *** amotoki has quit IRC | 11:41 | |
| SergeyLukjanov | sdague, I can ninja merge ;) | 11:41 |
| sdague | well, lets let the stuff above it try to get through first | 11:42 |
| SergeyLukjanov | sdague, yup, agree | 11:43 |
| SergeyLukjanov | sdague, so, you're proposing to ninja merge 98080 and 97966 after first two merged | 11:43 |
| sdague | yeh | 11:43 |
| SergeyLukjanov | good | 11:43 |
| *** rfolco has joined #openstack-infra | 11:44 | |
| * SergeyLukjanov just like to save context :) | 11:44 | |
| sdague | from my reading of the failure 98080 is basically just failing on the 8 way concurency | 11:44 |
| sdague | let me double check that assumption | 11:44 |
| openstackgerrit | Stephan Renatus proposed a change to openstack-infra/config: Increase timeout for chef-related jobs, 30min is not enough time https://review.openstack.org/98084 | 11:47 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326754 https://review.openstack.org/98085 | 11:47 |
| uvirtbot | Launchpad bug 1326754 in tripleo "Ubuntu CI failures : Unable to locate package ubuntu-cloud-keyring" [High,Triaged] https://launchpad.net/bugs/1326754 | 11:47 |
| sdague | oh, ffs, the top of that failed on something else | 11:47 |
| sdague | which is actually this other thing I was just fixing | 11:48 |
| *** afazekas has joined #openstack-infra | 11:48 | |
| sdague | SergeyLukjanov: so I'd say ninja merge 97912,2 regardless | 11:50 |
| sdague | it's top of gate, but failed on what I think https://review.openstack.org/#/c/97842/ will help with | 11:51 |
| SergeyLukjanov | sdague, yup | 11:52 |
| SergeyLukjanov | I'm going to ninja merge it right now | 11:52 |
| *** julim has joined #openstack-infra | 11:52 | |
| sdague | so I'm going to send a different email which is just appealing to people to work on fixing gate reseting bugs | 11:53 |
| SergeyLukjanov | sdague, ack | 11:56 |
| SergeyLukjanov | sdague, https://review.openstack.org/#/c/97912/ submited | 11:56 |
| *** PaulMurray has joined #openstack-infra | 11:57 | |
| SergeyLukjanov | fungi, clarkb, mordred, fyi I've submitted https://review.openstack.org/#/c/97912/ | 11:59 |
| *** jgallard has joined #openstack-infra | 11:59 | |
| *** dprince has joined #openstack-infra | 12:01 | |
| *** flaper87|afk is now known as flaper87 | 12:02 | |
| *** hashar has joined #openstack-infra | 12:06 | |
| *** pdmars has joined #openstack-infra | 12:06 | |
| *** pdmars has quit IRC | 12:08 | |
| *** Longgeek has quit IRC | 12:12 | |
| *** Longgeek has joined #openstack-infra | 12:12 | |
| *** mriedem has joined #openstack-infra | 12:15 | |
| boris-42 | sdague around? | 12:16 |
| boris-42 | sdague or better in qa | 12:17 |
| Alex_Gaynor | Hey all, OpenSSL security release on this fine morning just dropped, packages should be out shortly. | 12:17 |
| sdague | boris-42: if you aren't fixing race conditions in the gate, I don't have time for it | 12:18 |
| Alex_Gaynor | (https://www.openssl.org/news/secadv_20140605.txt) | 12:18 |
| boris-42 | sdague I would like to speak about | 12:18 |
| boris-42 | sdague one bug | 12:18 |
| sdague | boris-42: ok, shoot | 12:18 |
| boris-42 | sdague timeout one | 12:18 |
| sdague | bug url? | 12:19 |
| boris-42 | sdague one sec | 12:19 |
| boris-42 | Bug 1254890 - "Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures | 12:19 |
| uvirtbot | Launchpad bug 1254890 in nova ""Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures" [High,Confirmed] https://launchpad.net/bugs/1254890 | 12:19 |
| boris-42 | sdague ^ | 12:19 |
| boris-42 | sdague it may be cause by default setup of keystone | 12:19 |
| boris-42 | sdague lemme show what is happening with keystone performance under load | 12:19 |
| boris-42 | sdague http://logs.openstack.org/63/91463/1/check/check-rally-dsvm-rally/30e0b0c/rally-plot/results.html.gz | 12:20 |
| boris-42 | sdague after 2k iterations operation get tokens start working for a lot of time | 12:20 |
| boris-42 | sdague and let me show how it affects heat | 12:20 |
| boris-42 | sdague http://logs.openstack.org/45/98045/2/check/check-rally-dsvm-rally/f2d4c29/rally-plot/results.html.gz | 12:21 |
| boris-42 | sdague so as tempest is running a lot of stuff during 1hrs | 12:21 |
| boris-42 | sdague there is a lot of interaction to keystone | 12:21 |
| boris-42 | sdague at some point keystone may just suck | 12:21 |
| boris-42 | sdague and it produce super long request to get tokens -> which produce timeout in client | 12:22 |
| sdague | have you looked at the logs in those failures | 12:22 |
| sdague | because I've never seen that be an issue | 12:22 |
| boris-42 | sdague I am not 100% that it is issue, but it can be | 12:23 |
| boris-42 | sdague so probably just running under apache keystone in dsvm gate will reduce amount of timeouts | 12:23 |
| sdague | boris-42: please actually look into the gate logs to find evidence for that | 12:23 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Remove trailing spaces in debug log https://review.openstack.org/97488 | 12:23 |
| sdague | I really doubt it | 12:23 |
| *** om has quit IRC | 12:23 | |
| boris-42 | sdague so how can I found that operation get/validate token inside any of project started working slow? | 12:23 |
| boris-42 | sdague cause it won't fail, it will just work for a really long amount of time | 12:25 |
| boris-42 | sdague e.g. couple of minutes.. | 12:25 |
| sdague | we log all our calls here - http://logs.openstack.org/65/98065/1/check/check-grenade-dsvm-partial-ncpu/18541b3/logs/tempest.txt.gz | 12:25 |
| sdague | the token calls are taking about 0.1s | 12:25 |
| sdague | even at the end of the run | 12:25 |
| sdague | that's a log for something that failed on this issue | 12:26 |
| *** trinaths has quit IRC | 12:26 | |
| *** trinaths has joined #openstack-infra | 12:27 | |
| boris-42 | sdague btw what and where is specified timeout? | 12:27 |
| boris-42 | sdague it's time-out in tempest as I understand? | 12:28 |
| sdague | boris-42: you don't understand | 12:28 |
| sdague | the issue is the resource never makes the transition | 12:28 |
| *** PaulMurray has left #openstack-infra | 12:29 | |
| *** _nadya_ has joined #openstack-infra | 12:30 | |
| *** palar has quit IRC | 12:30 | |
| sdague | boris-42: I'm happy if you'd like to help on gate things. But that means realizing that rally isn't really relevant here, and instead digging in and reading the failure logs for these issues to get to the bottom of the actual fails. | 12:30 |
| boris-42 | sdague i just thought that that stuff can be related to this one | 12:30 |
| *** unicell has joined #openstack-infra | 12:30 | |
| *** HenryG has quit IRC | 12:31 | |
| sdague | but you did that before reading any of the logs of the failures, which just makes it a distraction. There is a ton of data that we capture on these fails. | 12:31 |
| boris-42 | sdague so I'll investigate those logs | 12:31 |
| sdague | awesome, that would be great | 12:31 |
| *** pdmars has joined #openstack-infra | 12:31 | |
| boris-42 | sdague but to safe my time a bit | 12:31 |
| *** hdd_ has joined #openstack-infra | 12:31 | |
| boris-42 | sdague timeout is happening inside projects? | 12:31 |
| boris-42 | sdague or there is timeout in tempest | 12:32 |
| boris-42 | ? | 12:32 |
| sdague | here is the test that fails - http://logs.openstack.org/42/97442/1/check/check-grenade-dsvm/362736f/console.html#_2014-06-04_12_35_32_155 | 12:32 |
| sdague | and the streams we capture for it | 12:32 |
| sdague | it did an operation on that resource to make to go to a different state | 12:32 |
| sdague | and after polling once a second, for 196s, the resource didn't get there | 12:33 |
| *** Hal has joined #openstack-infra | 12:33 | |
| sdague | you can see the stream of GET calls polling for status | 12:33 |
| *** Hal is now known as Guest93241 | 12:33 | |
| sdague | you can cross reference what the test was actually doing by jumping back to the test itself, to figure out which state transitions it was trying to make | 12:34 |
| *** _nadya_ has quit IRC | 12:34 | |
| sdague | also corrolate the server logs during those time windows, especially because we've got uuids for resources | 12:34 |
| boris-42 | Server 64ee89ec-f9cc-474d-9c43-5013bb93e1ca failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: ACTIVE. | 12:36 |
| boris-42 | sdague ^ OHH? | 12:36 |
| boris-42 | what that means | 12:36 |
| *** talluri has quit IRC | 12:36 | |
| boris-42 | ah got it | 12:36 |
| *** talluri has joined #openstack-infra | 12:36 | |
| *** weshay has joined #openstack-infra | 12:37 | |
| *** talluri has quit IRC | 12:41 | |
| *** mbacchi has joined #openstack-infra | 12:41 | |
| *** dkranz has joined #openstack-infra | 12:42 | |
| SergeyLukjanov | sdague, yay! https://review.openstack.org/#/c/97912/ is merged | 12:43 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Refresh token support https://review.openstack.org/95478 | 12:43 |
| mtreinish | sdague: is there not already an e-r query for the pip thing? | 12:43 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/elastic-recheck: Refine queries for tripleo jobs https://review.openstack.org/98096 | 12:43 |
| mriedem | mtreinish: there isn't a good fingerprint for the pip failure | 12:44 |
| mriedem | mtreinish: i have a query in the bug report though | 12:44 |
| *** HenryG has joined #openstack-infra | 12:44 | |
| mtreinish | mriedem: oh yeah I can see that now, it just kind of disappears in the log without an error message | 12:44 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: set concurrency for grenade https://review.openstack.org/98080 | 12:44 |
| mriedem | mtreinish: there is err_trap at the bottom | 12:44 |
| mriedem | so that's what i based a logstash query on | 12:45 |
| mriedem | but it's probably too generic | 12:45 |
| *** adalbas has joined #openstack-infra | 12:45 | |
| sdague | mtreinish: it's not really easy to fingerprint | 12:46 |
| mtreinish | wouldn't that just catch any failure in devstack though? (unless I'm misunderstanding what it is) | 12:46 |
| sdague | mtreinish: it would | 12:46 |
| mriedem | mtreinish: that's why i didn't post that for an e-r query | 12:46 |
| mtreinish | ok, makes sense | 12:46 |
| mriedem | i'll have a new unrelated one coming up soon though... :) | 12:46 |
| sdague | I actually have a change to help debug it in the future | 12:46 |
| mriedem | or two | 12:46 |
| sdague | which I was holding so I didn't accidently rebase the change itself | 12:47 |
| SergeyLukjanov | btw get-pip change is already ninja-landed | 12:47 |
| sdague | SergeyLukjanov: right, great | 12:47 |
| sdague | https://review.openstack.org/98098 will make it possible to fingerprint in future | 12:47 |
| *** lbragstad has joined #openstack-infra | 12:47 | |
| SergeyLukjanov | sdague, all 4 changes are now landed | 12:48 |
| SergeyLukjanov | yay! | 12:48 |
| sdague | SergeyLukjanov: awesome | 12:49 |
| *** BAKfr has joined #openstack-infra | 12:50 | |
| *** _nadya_ has joined #openstack-infra | 12:51 | |
| *** aburaschi has joined #openstack-infra | 12:52 | |
| *** radez_g0n3 is now known as radez | 12:53 | |
| *** BobBallAway is now known as BobBall | 12:55 | |
| *** miqui has joined #openstack-infra | 12:57 | |
| *** flaper87 is now known as flaper87|afk | 13:02 | |
| *** liam__ has joined #openstack-infra | 13:02 | |
| *** liam__ has quit IRC | 13:02 | |
| *** jamespage_ has joined #openstack-infra | 13:03 | |
| *** jamespage_ has quit IRC | 13:04 | |
| *** zz_gondoi is now known as gondoi | 13:07 | |
| *** jergerber has joined #openstack-infra | 13:07 | |
| *** HenryG has quit IRC | 13:08 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova resize/migrate bug 1326778 https://review.openstack.org/98106 | 13:09 |
| *** om has joined #openstack-infra | 13:09 | |
| uvirtbot | Launchpad bug 1326778 in nova "test_list_migrations_in_flavor_resize_situation fails with 400 after VERIFY_RESIZE state" [Medium,Confirmed] https://launchpad.net/bugs/1326778 | 13:09 |
| *** smarcet has joined #openstack-infra | 13:09 | |
| *** dizquierdo is now known as dizquierdo_afk | 13:14 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for nova resize/migrate bug 1326778 https://review.openstack.org/98106 | 13:14 |
| uvirtbot | Launchpad bug 1326778 in nova "test_list_migrations_in_flavor_resize_situation fails with 400 after VERIFY_RESIZE state" [Medium,Confirmed] https://launchpad.net/bugs/1326778 | 13:14 |
| *** rpodolyaka has joined #openstack-infra | 13:14 | |
| *** om has quit IRC | 13:18 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Suppress the graph for bug 1260654 https://review.openstack.org/97993 | 13:18 |
| uvirtbot | Launchpad bug 1260654 in openstack-ci "Could not initialize class jenkins.model.Jenkins$MasterComputer" [Critical,Fix released] https://launchpad.net/bugs/1260654 | 13:18 |
| *** IvanBerezovskiy has joined #openstack-infra | 13:19 | |
| *** niska has quit IRC | 13:21 | |
| *** jergerber has quit IRC | 13:22 | |
| *** samuelmz has quit IRC | 13:24 | |
| *** luqas has joined #openstack-infra | 13:24 | |
| *** niska has joined #openstack-infra | 13:26 | |
| *** mfer has joined #openstack-infra | 13:26 | |
| *** bknudson has joined #openstack-infra | 13:27 | |
| *** dkliban_afk is now known as dkliban | 13:28 | |
| *** doug-fish has joined #openstack-infra | 13:29 | |
| anteaya | I'm going to look at this list gate-ceilometer-python27 : 17 Uncategorized Fails. 15.0% Classification Rate (20 Total Fails) | 13:29 |
| *** HenryG has joined #openstack-infra | 13:29 | |
| sdague | SergeyLukjanov: what's the log server look like? | 13:31 |
| *** yfried has quit IRC | 13:31 | |
| sdague | anteaya: thanks | 13:31 |
| sdague | SergeyLukjanov: http://logs.openstack.org/42/97442/1/check/check-grenade-dsvm-partial-ncpu/3d0a15a/logs/old/screen-n-cpu.txt.gz is suspiciously truncated | 13:31 |
| sdague | as are other logs on that run | 13:31 |
| SergeyLukjanov | sdague, me looking | 13:31 |
| *** wenlock has joined #openstack-infra | 13:32 | |
| SergeyLukjanov | sdague, at least, we have enough disk space | 13:33 |
| SergeyLukjanov | >> /dev/mapper/main-logs 12T 9.7T 2.2T 83% /srv/static/logs | 13:34 |
| sdague | hm | 13:34 |
| *** om has joined #openstack-infra | 13:34 | |
| sdague | I wonder if we are running out of space on these new hp nodes during the gzip | 13:35 |
| sdague | because it's only the big ones that are truncated | 13:35 |
| SergeyLukjanov | sdague, no, enough space and the file itself isn't truncated on server | 13:35 |
| sdague | though / is 30 G | 13:35 |
| *** homeless has joined #openstack-infra | 13:36 | |
| fungi | wow, that's a fair amount of scrollback i lack time to read. is there anything urgent which needs my attention? | 13:36 |
| SergeyLukjanov | fungi, I've ninja-submitted get-pip fix | 13:37 |
| SergeyLukjanov | and 4 more changes has been promoted to make gate a bit healthy | 13:37 |
| sdague | fungi: yeh, using https://etherpad.openstack.org/p/gatetriage-june2014 as our message bus | 13:37 |
| sdague | so the things that have been done should be in there | 13:38 |
| SergeyLukjanov | sdague, oh, I'm wrong, it's truncated on server too - last line is "2014-06-04 13:11:24.245 AUDIT nova.compute.claims [req-2fd8e9ec-f180-465e-829d-b105433c07a1 ServersTestXML" | 13:38 |
| Kiall | Is it normal for a +2 to "stick around" after a new patchset is uploaded? https://review.openstack.org/#/c/98093/ | 13:38 |
| russellb | fungi: no, i missed the freeze | 13:38 |
| fungi | sdague: SergeyLukjanov: thanks! | 13:39 |
| russellb | apevec: sounds like we merged https://review.openstack.org/#/c/97845/ without an exception | 13:39 |
| BAKfr | I've a question for the gerrit developers | 13:40 |
| anteaya | sdague: so the 5 errors on the bottom of the list are all multiple db errors, the other 15 are events related to 3 failed tests, I'll work on fingerprints of the 3 failed tests first | 13:40 |
| sdague | fungi: if you have any thoughts on why - http://logs.openstack.org/42/97442/1/check/check-grenade-dsvm-partial-ncpu/3d0a15a/logs/old/screen-n-cpu.txt.gz is truncated | 13:40 |
| sdague | anteaya: thanks | 13:40 |
| BAKfr | Is it normal that some commits from "openstack" branches are not used ? | 13:40 |
| fungi | BAKfr: if by "the gerrit developers" you mean the ones at google who write gerrit, you can find them in #gerrit on freenode | 13:40 |
| luqas | hi, I would like to change the e-mail contact for our 3rd party gerrit account but I can't register a new e-mail, any idea? | 13:41 |
| fungi | BAKfr: oh, a question for openstack's gerrit admins. can you elaborate? i'm not sure what you mean | 13:41 |
| fungi | sdague: looking | 13:41 |
| *** signed8bit has joined #openstack-infra | 13:41 | |
| *** otherwiseguy has joined #openstack-infra | 13:42 | |
| sdague | 3 of the larger log files in that run are truncated | 13:42 |
| BAKfr | Somes commits done for old versions are not used | 13:42 |
| BAKfr | like this: https://review.openstack.org/#/c/35447/ | 13:42 |
| sdague | before I write it off to sun spots, I want to make sure it's not systemic | 13:43 |
| fungi | sdague: all screen logs? maybe screen sessions weren't terminated properly before log copying began and buffered writes were lost? (completely off-the-wall theory) | 13:43 |
| anteaya | luqas: hold on that thought, in a bit of time we will have a system where you will have a wiki page and can edit that contact email until your heart's content | 13:43 |
| BAKfr | We've a few commits in an unused "openstack" branch . | 13:43 |
| sdague | fungi: yeh, I was only looking at screen logs | 13:43 |
| *** om has quit IRC | 13:43 | |
| *** dangers_away is now known as dangers | 13:43 | |
| luqas | anteaya: thanks :-) | 13:43 |
| anteaya | luqas: I will find you if there is an issue with midokura | 13:44 |
| fungi | BAKfr: ahh, you're talking about custom patches to gerrit. a couple months ago upstream gerrit finally got to the point where it was mostly usable for us. there are a few things like that ugly hack you pointed out which we couldn't really find a good upstream solution for. point is we're trying not to actually modify the gerrit source code for the instance we're running now | 13:45 |
| fungi | luqas: yeah, we're about to blank out the e-mail addresses in gerrit for all those service accounts and move them to a wiki | 13:45 |
| fungi | luqas: partly to work around a limitation in gerrit where two accounts can't share the same e-mail address, and it would be nice for service account administrators to be able to list their normal e-mail address as a point of contact but still use the same address for their normal gerrit user too | 13:46 |
| BAKfr | fungi, it's answers my interrogations. thanks | 13:47 |
| sdague | fungi / SergeyLukjanov: after the cinder change lands, I'd say promote 98108 | 13:47 |
| sdague | that's what killed nova change #2 | 13:47 |
| sdague | and the tempest change | 13:47 |
| sdague | the cinder change just landed | 13:47 |
| sdague | so it's good promoting time | 13:47 |
| *** habib has joined #openstack-infra | 13:47 | |
| fungi | BAKfr: probably the best way for us to get gerrit 2.9 or later to solve that problem would be a patch that could (with some configuration) pass through the response from the remote api and embed it into the on-screen error message | 13:48 |
| *** sandywalsh_ has joined #openstack-infra | 13:48 | |
| luqas | anteaya: fungi: ok, I'll wait then it wasn't something critical, thanks for your clarification | 13:48 |
| SergeyLukjanov | sdague, okay, I'll do it | 13:48 |
| sdague | SergeyLukjanov: it needs to be moved to the gate queue | 13:48 |
| fungi | BAKfr: but "soon" we anticipate having our own separate openid provider, at which point we can provide a much tighter and lighter-weight integration between foundation system accounts and gerrit accounts, so we can drop the hackish way we're using the gerrit contactstore feature to do that today | 13:49 |
| *** dmakogon_ has joined #openstack-infra | 13:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add dib-utils project https://review.openstack.org/90281 | 13:49 |
| *** otherwiseguy has quit IRC | 13:49 | |
| SergeyLukjanov | sdague, is cinder change already landed? | 13:50 |
| sdague | SergeyLukjanov: yeh | 13:50 |
| sdague | it was in front of the failing nova change | 13:50 |
| SergeyLukjanov | sdague, okay, promoting 98108 | 13:50 |
| SergeyLukjanov | sdague, I was too slow to see it ;) | 13:50 |
| sdague | :) | 13:50 |
| *** prad has joined #openstack-infra | 13:50 | |
| sdague | fungi: pypi.openstack.org is hosted at rax? | 13:51 |
| fungi | sdague: yeah, it's just an apache vhost on static.openstack.org | 13:51 |
| fungi | it's in rax dfw | 13:51 |
| fungi | sdague: looking into connectivity issues for it? | 13:52 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcInNvY2tldC50aW1lb3V0OiB0aW1lZCBvdXRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwMTk3NjMwNzk4NX0= | 13:52 |
| SergeyLukjanov | sdague, 98108 promoted | 13:52 |
| sdague | it's just a couple, but hp region b3 => pypi.openstack.org | 13:52 |
| sdague | failures to pip install | 13:52 |
| *** otherwiseguy has joined #openstack-infra | 13:52 | |
| sdague | fungi: is there already an appropriate bug for that? | 13:53 |
| fungi | sdague: ianw reported a bug yesterday about stray tcp/rst on connections in hpcloud 1.1 and was able to get the service tech who picked it up to reproduce it consistently | 13:53 |
| fungi | (in hp cloud services) | 13:53 |
| fungi | though that says socket timeout, not connection reset by peer | 13:54 |
| fungi | might be a separate issue | 13:54 |
| sdague | yeh, this is the best way to fingerprint pip just choking | 13:54 |
| *** pcrews has joined #openstack-infra | 13:54 | |
| fungi | sdague: i think we have blanket bugs for "the internet sucks" which are perpetually open or replaced by new ones when lp auto-closes them for inactivity. don't know the bug numbers off the top of my head | 13:54 |
| sdague | well, I'd like to track this one. It's a little different than just internet sucks | 13:55 |
| sdague | because it's pip to our servers from our servers | 13:55 |
| fungi | indeed | 13:55 |
| sdague | I'll just open a new one | 13:55 |
| *** yaguang has quit IRC | 13:55 | |
| *** sballe has joined #openstack-infra | 13:55 | |
| fungi | actually, it seems far less terrible than hpcloud 1.0 az2 was, from a network stability standpoint | 13:56 |
| fungi | connections from there used to crap out very frequently | 13:56 |
| dmakogon_ | hello, guys, i caught a problem with launching keystone, https://rdjenkins.dyndns.org/job/Trove-Gate/3974/console, timestamp 13:20:45 | 13:56 |
| *** arnaud__ has joined #openstack-infra | 13:57 | |
| *** d0ugal has joined #openstack-infra | 13:57 | |
| fungi | dmakogon_: ERROR: openstackclient.shell Exception raised: six>=1.6.0 | 13:58 |
| fungi | that or is it breaking down earlier? | 13:58 |
| dmakogon_ | fungi, but before that keystone wasn't able to start | 13:58 |
| dmakogon_ | fungi, key failed to start | 13:58 |
| dmakogon_ | fungi, ah, get it, seems that it's true, six and client problem | 13:59 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: tracking bug for connection issues between HP and RAX https://review.openstack.org/98112 | 13:59 |
| fungi | dmakogon_: i only see that string in the xtrace output from the screen command | 13:59 |
| *** otherwiseguy has quit IRC | 13:59 | |
| fungi | dmakogon_: it doesn't actually get echoed, so keystone appears to have started right up | 14:00 |
| dmakogon_ | fungi, than it seems that the problem is in openstackclient | 14:00 |
| *** crc32 has joined #openstack-infra | 14:00 | |
| fungi | dmakogon_: yeah, looks like it | 14:01 |
| BobBall | Any idea my nodepool doc change fails unit tests? Doesn't seem like something that could be a race, but it passes in my local env: http://logs.openstack.org/35/97835/2/check/gate-nodepool-python27/55c3a93/console.html | 14:01 |
| *** gokrokve has joined #openstack-infra | 14:01 | |
| *** markwash has joined #openstack-infra | 14:03 | |
| dmakogon_ | fungi, could you please tell who's on charge of openstackclient ? | 14:03 |
| fungi | dmakogon_: dtroyer | 14:03 |
| BobBall | Perhaps I don't understand the test, but it seems to be doing the same read-only(!) thing twice, with the second time it gets a different answer... :) | 14:03 |
| fungi | BobBall: huh... No handlers could be found for logger "nodepool.NodePool" | 14:03 |
| *** _nadya_ has quit IRC | 14:03 | |
| BobBall | oh - I saw that one before... let me look for something to do with that. I was just focused on the final error | 14:03 |
| BobBall | the mismatch | 14:04 |
| BobBall | 2014-04-01T16:53:01 <fungi> jeblair: are the 'No handlers could be found for logger "nodepool.NodePool"' errors during test_subnodes on your radar? | 14:04 |
| *** _nadya_ has joined #openstack-infra | 14:04 | |
| BobBall | :) I was _just_ looking at that... | 14:04 |
| BobBall | (from http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-04-01.log) | 14:04 |
| fungi | BobBall: so the MismatchError: 3 != 2 | 14:04 |
| fungi | BobBall: oh, that seems to have cropped up after the timeout | 14:04 |
| dmakogon_ | dtroyer, ping | 14:05 |
| anteaya | dmakogon_: he has been away since last week, I haven't seen him around yet | 14:05 |
| fungi | BobBall: so it looks like it threw an exception in test_subnodes, hung, then got killed via timeout about 35 minutes later | 14:06 |
| anteaya | though I am told he is expected back by week's end, which could be today, tomorrow or Monday | 14:06 |
| dmakogon_ | anteaya, thanks | 14:06 |
| anteaya | np | 14:06 |
| *** pblaho has joined #openstack-infra | 14:07 | |
| *** Mithrandir has quit IRC | 14:08 | |
| *** yfried has joined #openstack-infra | 14:10 | |
| fungi | BobBall: also that traceback looks slightly unpossible... i think it's actually truncated and then interrupted by the "no loggers" error | 14:10 |
| fungi | maybe subunit crossed the streams | 14:10 |
| BobBall | Indeed... | 14:10 |
| BobBall | never cross the streams! | 14:10 |
| fungi | i'm going to recheck no bug that change just to see if this is repeatable or some stray weirdness | 14:11 |
| BobBall | ok, thanks. I don't like bugs that don't give any details but couldn't figure out what to put in a bug report! | 14:11 |
| *** wenlock has quit IRC | 14:11 | |
| fungi | ran in rax dfw, which aside from boot errors has been pretty solid lately, so seems marginally unlikely that the disk fell out from under it or something | 14:11 |
| anteaya | sdague: I'm doing something wrong - I should have 5 hits for the logstash query: message:"FAIL: tests.alarm.test_rpc" | 14:12 |
| *** Mithrandir has joined #openstack-infra | 14:12 | |
| fungi | anteaya: did you change the query timeframe to something other than the default "last 15 minutes"? | 14:14 |
| anteaya | sdague: the oldest is May 30th | 14:14 |
| anteaya | fungi: yes, All Time | 14:14 |
| *** e0ne_ has quit IRC | 14:14 | |
| anteaya | which is the last 2 weeks, I do believe | 14:14 |
| *** e0ne has joined #openstack-infra | 14:14 | |
| sdague | anteaya: log ? | 14:14 |
| fungi | "all time" also beats up the cluster pretty badly, i recall clarkb saying | 14:14 |
| sdague | yeh, 7 days is probably the right thing to do | 14:15 |
| anteaya | http://logs.openstack.org/81/96781/3/gate/gate-ceilometer-python27/6e8b3a7/console.html | 14:15 |
| anteaya | I can switch to 7 days | 14:15 |
| *** andreaf has quit IRC | 14:16 | |
| anteaya | last 7 days, that query, no results returned | 14:16 |
| sdague | anteaya: I think '.' is not tokenized | 14:16 |
| sdague | so you have to search for the whole test | 14:17 |
| sdague | otherwise it's not matching it as a word | 14:17 |
| sdague | message:"FAIL: tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_presence" | 14:17 |
| anteaya | sorry june 3 for that one | 14:17 |
| anteaya | that failed on me as well | 14:17 |
| anteaya | I will try again though | 14:17 |
| sdague | tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_presence is treated as a single word | 14:17 |
| sdague | that one just returned for me | 14:17 |
| anteaya | oh | 14:17 |
| anteaya | cool | 14:17 |
| sdague | fungi: oh, that was the thing Alex_Gaynor pinged about. openssl patch | 14:18 |
| fungi | sdague: yeah, we'll have it when ubuntu auto updates. i'll double-check but if it's not ready the alternative is that we all take a holiday | 14:18 |
| anteaya | 28 hits, I wonder if there is a bug open for it yet | 14:19 |
| *** viktors is now known as viktors|afk | 14:19 | |
| sdague | fungi: :) | 14:19 |
| fungi | sdague: http://www.ubuntu.com/usn/usn-2232-1/ | 14:19 |
| sdague | well could just turn off +A rights in gerrit, and tell everyone to go away for 2 days | 14:19 |
| sdague | :) | 14:19 |
| fungi | spot checking systems, but i'll also see if centos has patches out yet | 14:20 |
| fungi | on precise it looks like we want to be using libssl1.0.0 1.0.1-4ubuntu5.14 | 14:20 |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack/requirements: Add rfc3987 library https://review.openstack.org/98012 | 14:22 |
| *** aysyd has joined #openstack-infra | 14:22 | |
| *** unicell has quit IRC | 14:22 | |
| apevec | russellb, fixes for gate issues have free ride | 14:22 |
| apevec | russellb, I'd just appreciate setting the target in LP | 14:23 |
| fungi | apevec: good point. broken stable gate means no stable point releases ;) | 14:23 |
| russellb | apevec: k | 14:23 |
| *** primemin1sterp is now known as primeministerp | 14:23 | |
| *** primeministerp has quit IRC | 14:24 | |
| apevec | fungi, indeed | 14:24 |
| *** primeministerp has joined #openstack-infra | 14:24 | |
| fungi | fixed package version for rhel 6 seems to be openssl-1.0.1e-16.el6_5.14.i686.rpm so will probably be the same on centos once they get it ported in | 14:25 |
| fungi | but i don't see an announcement for it yet | 14:25 |
| fungi | or probably openssl-1.0.1e-16.el6_5.14.x86_64.rpm in our case | 14:26 |
| fungi | ugh. i wish rackspace wouldn't configure their ubuntu base images for security updates from their mirror. it hasn't updated yet and so lacks fixed packages | 14:28 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication https://review.openstack.org/98123 | 14:28 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326820 https://review.openstack.org/98124 | 14:28 |
| uvirtbot | Launchpad bug 1326820 in ceilometer "FAIL: tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_presence" [Undecided,New] https://launchpad.net/bugs/1326820 | 14:28 |
| anteaya | sdague: my line length is probably wrong in that one, I'm not sure how to fix it | 14:29 |
| dstufft | oh yea | 14:29 |
| dstufft | I need to update some OpenSSL | 14:29 |
| *** basha has quit IRC | 14:29 | |
| fungi | dstufft: and talk to your base image people about not configuring customers to use an outdated security repository | 14:30 |
| *** gokrokve has quit IRC | 14:30 | |
| dstufft | I'm not sure who those people are | 14:30 |
| *** basha has joined #openstack-infra | 14:30 | |
| dstufft | I could probably figure it out | 14:30 |
| fungi | yeah, me neither :/ | 14:30 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication https://review.openstack.org/98123 | 14:30 |
| fungi | worth noting, this is why debian (even well before ubuntu) configured a separate repository for security updates... so that users could get release packages from a very laggy mirror but still get the latest security updates straight from the distro | 14:31 |
| *** Longgeek has quit IRC | 14:33 | |
| fungi | restarting apache on review.o.o, so there could be a blip from the webui | 14:33 |
| fungi | seems to be still working | 14:34 |
| sdague | fungi: I know DinaBelova was looking into it, but there is a way to get back reseting votes on commit message changes right? | 14:34 |
| fungi | sdague: i *think* it's a config option. looking real fast | 14:34 |
| sdague | because the lack of that behavior is killing me. About 1/3 of feedback follow up I have to do manually because people's votes are sticky | 14:34 |
| DinaBelova | fungi, oh, thank you! | 14:34 |
| sdague | and their feedback was about adding a bug | 14:34 |
| *** jgrimm has joined #openstack-infra | 14:35 | |
| sdague | or they get mad at me that I have a -1 on their patch after they updated a thing in the message, and I never knew it | 14:35 |
| *** Lingo_ has joined #openstack-infra | 14:36 | |
| *** Lingo_ is now known as TravelingBear | 14:37 | |
| *** maxbit has joined #openstack-infra | 14:37 | |
| *** skolekonov has quit IRC | 14:37 | |
| *** gokrokve has joined #openstack-infra | 14:39 | |
| *** TravelingBear is now known as BadCub01 | 14:39 | |
| *** wenlock has joined #openstack-infra | 14:40 | |
| *** pblaho has quit IRC | 14:40 | |
| *** mburned is now known as mburned_out | 14:42 | |
| fungi | sdague: DinaBelova: https://review.openstack.org/Documentation/config-labels.html#label_copyAllScoresIfNoCodeChange | 14:42 |
| *** dims_ has joined #openstack-infra | 14:44 | |
| fungi | strangely, that doesn't seem editable in the All-Projects acl within the gerrit webui | 14:44 |
| fungi | we may need to edit and push an update to it via git | 14:44 |
| *** jaypipes has joined #openstack-infra | 14:45 | |
| fungi | sdague: DinaBelova: https://review.openstack.org/gitweb?p=All-Projects.git;a=blob;f=project.config;h=ca6b31f;hb=HEAD#l100 | 14:46 |
| DinaBelova | fungi, oh, really strange! | 14:46 |
| *** timrc-afk is now known as timrc | 14:46 | |
| *** thedodd has joined #openstack-infra | 14:47 | |
| *** mburned_out is now known as mburned | 14:47 | |
| fungi | but yeah, docs say it defaults to false, so we could just remove that one line. i'll wait for confirmation from clarkb and zaro that it's sane/desirable, but seems like a reasonable change to me | 14:48 |
| fungi | gets us closer to the earlier behavior we had in 2.4 | 14:48 |
| *** doddstack has joined #openstack-infra | 14:49 | |
| DinaBelova | fungi, well - it's not only close behaviour | 14:49 |
| DinaBelova | that's more logical really | 14:49 |
| *** trinaths has quit IRC | 14:49 | |
| fungi | BobBall: it's doing it again! https://jenkins06.openstack.org/job/gate-nodepool-python27/18/console | 14:49 |
| *** rdopiera has quit IRC | 14:49 | |
| DinaBelova | as in the open projects keeping code/commit messages clean is also important | 14:50 |
| DinaBelova | and sometimes that's good reason for -1 | 14:50 |
| fungi | BobBall: succeeded in check this time, but now hanging in the gate instead | 14:50 |
| BobBall | That sucks! :( | 14:50 |
| fungi | BobBall: so it looks like we probably have a race condition in the test | 14:50 |
| BobBall | quite a common one if it's hitting me 2 out of 3 times... | 14:51 |
| *** thedodd has quit IRC | 14:51 | |
| fungi | DinaBelova: i completely agree. that's why we didn't carry scores over on commit message changes before, just pointing out that this gets us closer to that behavior again | 14:51 |
| BobBall | d'oh | 14:52 |
| fungi | BobBall: yes, probably introduced in a recent patch. we've had a lot of fixes go in which required new tests or changes to existing ones for nodepool | 14:52 |
| BobBall | it's a parallel tox thing fungi | 14:52 |
| DinaBelova | fungi, yeah, that's true :) | 14:52 |
| BobBall | I bet you anything :D we're polling the DB for nodes to see how many are ready | 14:53 |
| BobBall | but at some point another test will be reserving one of those nodes surely | 14:53 |
| fungi | BobBall: oho, probably right! | 14:53 |
| BobBall | which means a 3rd node is built | 14:53 |
| fungi | BobBall: well, or holding a lock or something | 14:53 |
| *** jgrimm has quit IRC | 14:53 | |
| fungi | BobBall: though i thought we were using per-runner databases for these tests to avoid that sort of thing | 14:54 |
| BobBall | oh... | 14:54 |
| * fungi looks at the fixtures | 14:54 | |
| BobBall | dburi: '{dburi}' | 14:54 |
| *** BAKfr has left #openstack-infra | 14:54 | |
| BobBall | no | 14:54 |
| BobBall | we're always using openstack_citest aren't we? | 14:55 |
| fungi | but regardless, if you're not running this tox env with 4-way parallelization then that could explain why we hit it in ci but you're unable to reproduce locally | 14:55 |
| BobBall | I am - but even then, it's a race condition rather than guaranteed | 14:55 |
| BobBall | and I've got a shiny new machine... so I hope it's faster than the gate ;) | 14:55 |
| BobBall | oh hang on - I see it now | 14:57 |
| BobBall | we do create a custom DB for each test | 14:57 |
| BobBall | poo - bang goes my theory | 14:57 |
| fungi | yeah, overriding TestNodepool.dburi | 14:57 |
| *** achuprin has quit IRC | 14:58 | |
| fungi | also, this time it broke in test_node_az, so entirely different test | 14:58 |
| *** jistr has quit IRC | 14:58 | |
| fungi | and a very recently added one (came in with nova os-ext-az support for dealing with hpcloud 1.1 this week) | 14:59 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added sort parameters to API https://review.openstack.org/95959 | 15:00 |
| *** jistr has joined #openstack-infra | 15:00 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Refresh token support https://review.openstack.org/94363 | 15:00 |
| fungi | i think i probably want clarkb's eyes on this first thing once he's around, because we were planning to restart nodepool but this could imply a new bug, in which case we're better off identifying it in some way other than running it in production | 15:00 |
| *** hdd_ has quit IRC | 15:01 | |
| *** otherwiseguy has joined #openstack-infra | 15:02 | |
| *** timrc is now known as timrc-afk | 15:03 | |
| *** timrc-afk is now known as timrc | 15:06 | |
| *** atiwari has joined #openstack-infra | 15:08 | |
| *** eharney has joined #openstack-infra | 15:09 | |
| *** achuprin has joined #openstack-infra | 15:11 | |
| sdague | fungi / SergeyLukjanov - we need a repromote of - https://review.openstack.org/#/c/98108/ - it got hit by another thing | 15:12 |
| *** radez is now known as radez_g0n3 | 15:13 | |
| *** dprince has quit IRC | 15:13 | |
| fungi | Details: Service cinder-backup could not be found. | 15:13 |
| fungi | eek | 15:13 |
| SergeyLukjanov | sdague, fungi, I'm able to promote something | 15:14 |
| SergeyLukjanov | is it still needed? | 15:14 |
| fungi | SergeyLukjanov: sure | 15:14 |
| SergeyLukjanov | promoting | 15:15 |
| fungi | i haven't really started yet | 15:15 |
| fungi | go for it | 15:15 |
| sdague | fungi: yeh, we need the failure logs that mtreinish has most of a patch for | 15:15 |
| sdague | I'm also going to try to add some better sanity checks on grenade | 15:15 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for bug 1323524 https://review.openstack.org/98146 | 15:16 |
| SergeyLukjanov | fungi, sdague, done | 15:16 |
| uvirtbot | Launchpad bug 1323524 in ceilometer "tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate failed for timeout" [Medium,Confirmed] https://launchpad.net/bugs/1323524 | 15:16 |
| sdague | SergeyLukjanov: thanks | 15:16 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Refresh token support https://review.openstack.org/95478 | 15:16 |
| SergeyLukjanov | heh, today is the promoting day | 15:17 |
| *** ihrachyshka has quit IRC | 15:18 | |
| mtreinish | fungi, sdague: I think that patch already merged: https://review.openstack.org/#/c/91965/ | 15:18 |
| devananda | morning, folks | 15:19 |
| anteaya | morning devananda | 15:19 |
| sdague | mtreinish: ok, that would indicate that it didn't create that file | 15:20 |
| devananda | fungi: any chance we can bump 97757 to the top of the gate queue? ironic and tripleo are both broken/blocked until this lands -- we tried to land it yesterday, i'm not sure what happened (I had an appt to run to and lifeless / clarkb were working on it when I left) | 15:20 |
| devananda | it looks like it got rechecked and is now at the tail of the queue | 15:20 |
| mtreinish | sdague: or that I screwed something up in the patch | 15:20 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1321826 https://review.openstack.org/98149 | 15:21 |
| uvirtbot | Launchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress] https://launchpad.net/bugs/1321826 | 15:21 |
| *** patelna has joined #openstack-infra | 15:23 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1321826 https://review.openstack.org/98149 | 15:23 |
| uvirtbot | Launchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress] https://launchpad.net/bugs/1321826 | 15:23 |
| devananda | maybe i'm misinterpreting the zuul status page, but why does it look like non-integrated projects are gating in series with integrated projects? | 15:26 |
| *** virmitio has joined #openstack-infra | 15:26 | |
| fungi | devananda: possibly because they share a job they shouldn't | 15:26 |
| anteaya | sdague: in this set of 8 failures, 5 patches failed with the same 8 failures, should I pick one failure and create a fingerprint from that? http://logs.openstack.org/76/93576/4/gate/gate-ceilometer-python27/338daab/testr_results.html.gz | 15:27 |
| sdague | anteaya: yep | 15:28 |
| anteaya | kk | 15:28 |
| fungi | devananda: or maybe that's intentional | 15:28 |
| fungi | devananda: which ones? | 15:28 |
| devananda | fungi: eg, if "gate-tempest-dsvm-full" is in an incubated project's gate check, would taht serialize it with, say, nova? | 15:29 |
| *** gargola has quit IRC | 15:29 | |
| fungi | devananda: yes. which incubated project are you seeing that for? | 15:29 |
| *** gargola has joined #openstack-infra | 15:30 | |
| devananda | fungi: i observed that for ironic yesterday | 15:30 |
| anteaya | sdague: there have been no hits since May 30th on that fingerprint, categorize it anyway? | 15:30 |
| devananda | fungi: also some projects in related programs, es taskflow (in oslo) and tripleo-image-elements (in tripleo) are also serialized | 15:31 |
| anteaya | message:"FAIL: tests.api.v2.test_acl_scenarios.TestAPIACL.test_authenticated(sqlite)" | 15:31 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/elastic-recheck: Treat tripleo builds as openstack projects https://review.openstack.org/98154 | 15:32 |
| fungi | devananda: ahh, i don't immediately see jobs for ironic which would have that effect | 15:32 |
| devananda | fungi: none are running at the moment -- we can't land anything until 97757 merges in nova | 15:32 |
| devananda | fungi: so we aren't even approving patches right now | 15:32 |
| fungi | devananda: yeah, more that i was looking at the jobs defined for it in zuul's layout.yaml file | 15:33 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova pep8 gate fail bug 1326848 https://review.openstack.org/98156 | 15:34 |
| uvirtbot | Launchpad bug 1326848 in nova "pep8 F481 checks got caught in the gate" [Undecided,New] https://launchpad.net/bugs/1326848 | 15:34 |
| sdague | anteaya: yeh, if it pulls stuff off the uncategorized list | 15:34 |
| *** jgallard has quit IRC | 15:34 | |
| anteaya | sdague: k | 15:35 |
| phschwartz | jeblair: fungi: mordred: Sorry I was incapacitated the past few days due to my back. I will take a look at the iad region and also have a member of ops take a look. Are you still seeing a large number of ERROR state instances? | 15:35 |
| fungi | phschwartz: not sure about error state... yesterday's interesting issue was that you seem to have at least one compute node in iad starting instances with time offset by 4 hours (probably configured to edt rather than utc or something) | 15:36 |
| phschwartz | fungi: hmm, weird. Do you have a uuid for an instance that came up that way that I can look at? They should all be utc so that would need to be fixed. | 15:37 |
| sdague | devananda: the relationships are transitive | 15:37 |
| fungi | phschwartz: i can get you a uuid, though the instance associated with it is long burned and buried | 15:37 |
| fungi | just a sec | 15:37 |
| sdague | so if there is any path, anywhere, that all these changes can connect | 15:37 |
| phschwartz | We still have logs that we can see what hv it was on | 15:37 |
| sdague | then they are serialized | 15:38 |
| *** arnaud__ has quit IRC | 15:38 | |
| *** weshay has quit IRC | 15:39 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326754 https://review.openstack.org/98085 | 15:39 |
| uvirtbot | Launchpad bug 1326754 in tripleo "Ubuntu CI failures : Unable to locate package ubuntu-cloud-keyring" [High,Triaged] https://launchpad.net/bugs/1326754 | 15:39 |
| mordred_phone | phschwartz: do you have a copy of our sekrit graphs? | 15:39 |
| phschwartz | mordred_phone: No, I do not. | 15:39 |
| devananda | sdague: "can connect" as defined by the matrix of tests that each job runs? | 15:39 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for bug 1324885 https://review.openstack.org/98159 | 15:40 |
| uvirtbot | Launchpad bug 1324885 in ceilometer/icehouse "py26, py27 gate jobs are failing due to inaccurate mocks of keystone tokens" [Critical,Fix committed] https://launchpad.net/bugs/1324885 | 15:40 |
| sdague | devananda: correct | 15:40 |
| devananda | sdague: ack, thanks | 15:40 |
| *** mrodden has joined #openstack-infra | 15:41 | |
| devananda | so, back to my first question :) any way we can bump 97757 to the top of the queue, so that ironic and tripleo aren't totally blocked for another >24hrs? | 15:41 |
| anteaya | I'm moving on to: gate-ceilometer-python26 : 21 Uncategorized Fails. 19.2% Classification Rate (26 Total Fails) | 15:41 |
| sdague | devananda: there is an open question on that | 15:42 |
| sdague | are the ironic unit tests failed related or not? | 15:42 |
| sdague | because ironic is also failing unit tests | 15:42 |
| sdague | https://etherpad.openstack.org/p/gatetriage-june2014 - | 15:42 |
| devananda | oh? hrm | 15:42 |
| sdague | that's our message bus for gate failures | 15:42 |
| sdague | so I nixed the reenqueue this morning because of that until there is an answer | 15:43 |
| devananda | sdague: where is the failed unit test? | 15:44 |
| sdague | on any ironic change in the last 24 hrs | 15:44 |
| fungi | wow... novaclient 95426,2 is a serious mess (failing all integration jobs). must be the victim of another api change | 15:45 |
| sdague | devananda: I pulled 5 changes manually out of the gate yesterday because of it | 15:45 |
| *** unicell has joined #openstack-infra | 15:45 | |
| sdague | so please get the ironic house in order first, then we can talk about promotes from other projects | 15:45 |
| devananda | sdague: ack, looking into it. i'm not aware of any *unit* test failures in Ironic -- only the brokenness caused by the nova change that 97757 is reverting | 15:46 |
| fungi | oh, or maybe the victim of more rax mirror issues... "Failed to fetch http://mirror.rackspace.com/ubuntu/pool/main/o/openssl/openssl_1.0.1-4ubuntu5.14_amd64.deb 404 Not Found" | 15:46 |
| *** jistr has quit IRC | 15:46 | |
| sdague | devananda: then you haven't been looking at jobs in the last 24 hrs :) | 15:46 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for nova pep8 gate fail bug 1326848 https://review.openstack.org/98156 | 15:46 |
| uvirtbot | Launchpad bug 1326848 in nova "pep8 F481 checks got caught in the gate" [Undecided,New] https://launchpad.net/bugs/1326848 | 15:46 |
| fungi | (that's the security update related to today's openssl usn) | 15:46 |
| sdague | there is one in check run now that's failed | 15:46 |
| *** afazekas has quit IRC | 15:47 | |
| devananda | sdague: last 12 - sure. but I was looking during the 12 before that | 15:47 |
| *** jistr has joined #openstack-infra | 15:47 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer test_ordination_assign timeout bug 1326855 https://review.openstack.org/98160 | 15:47 |
| uvirtbot | Launchpad bug 1326855 in ceilometer ""tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_assign" race in the gate" [Undecided,New] https://launchpad.net/bugs/1326855 | 15:47 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Capture tempest failure files https://review.openstack.org/98162 | 15:48 |
| fungi | phschwartz: do you have any involvement with the package mirrors in rax? | 15:48 |
| dstufft | Hey, can someone approve https://review.openstack.org/#/c/97616/ ? :D | 15:48 |
| phschwartz | fungi: I do not, but I could find someone that deals with them | 15:48 |
| *** gokrokve has quit IRC | 15:49 | |
| fungi | phschwartz: i think i have a note for someone's irc nick or e-mail address, though it could be cold now | 15:49 |
| mordred | sdague: the thing I find fascinating is that the gate wasn't in bad shape before setuptools broke the world | 15:49 |
| *** jcoufal has quit IRC | 15:49 | |
| fungi | mordred: i think that was coincidental timing | 15:49 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/elastic-recheck: Treat tripleo builds as openstack projects https://review.openstack.org/98154 | 15:49 |
| sdague | mordred: I have a new theory | 15:49 |
| mordred | yeah? | 15:49 |
| sdague | most of what's in the gate queue has never seen an hp 1.1 node | 15:50 |
| sdague | because it passed on rax | 15:50 |
| fungi | this is entirely probable | 15:50 |
| sdague | when we were down a cloud | 15:50 |
| sdague | so the timing differences might be shaking things out | 15:50 |
| devananda | sdague: that unit test failure in the check queue IS because ofthe nova change - =https://jenkins01.openstack.org/job/gate-ironic-python26/1654/console | 15:50 |
| devananda | 2014-06-05 15:36:28.836 | TypeError: __init__() takes at most 4 arguments (5 given) | 15:50 |
| sdague | devananda: ok, that's why I asked the question | 15:50 |
| devananda | :) | 15:50 |
| sdague | and... wat | 15:51 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/elastic-recheck: Refine queries for tripleo jobs https://review.openstack.org/98096 | 15:51 |
| mordred | sdague: ah. fascinating | 15:51 |
| sdague | why are you importing nova in unit tests? | 15:51 |
| devananda | sdague: heh. cause we want to run unit tests on the noav.virt.ironic driver | 15:51 |
| fungi | phschwartz: my notes say carl thompson (all i have for him is an e-mail address) | 15:51 |
| *** che-arne has joined #openstack-infra | 15:51 | |
| phschwartz | Let me see if that is still correct | 15:51 |
| phschwartz | might take a few | 15:51 |
| *** radez_g0n3 is now known as radez | 15:51 | |
| devananda | sdague: there was a long conversation during I3 about how we could test that out of nova's tree. this is what we're doing to maintain an out of tree driver | 15:52 |
| sdague | devananda: ok, can you update the etherpad with that piece of information | 15:52 |
| devananda | sdague: it sucks gating on a nova driver not in nova ... really | 15:52 |
| *** ominakov has quit IRC | 15:52 | |
| fungi | phschwartz: looks like the mirror index updated before the package to which it referred was present, so it's possible your mirroring algorithm doesn't grok sane update ordering for apt repositories | 15:52 |
| sdague | devananda: you could always make that part a non voting job | 15:52 |
| sdague | devananda: also, I think the transitive dependency is oslotest | 15:53 |
| sdague | tripleo and ironic cross test with it | 15:53 |
| sdague | and it cross tests with oslo.messaging | 15:53 |
| *** rcarrillocruz has quit IRC | 15:53 | |
| *** yolanda has joined #openstack-infra | 15:53 | |
| sdague | which gets you back to a tempest-dsvm-full universe | 15:53 |
| *** ayoung has joined #openstack-infra | 15:53 | |
| ayoung | mordred, Zuul merge queue is at 34 hr 34 min. I assume this is not news? | 15:54 |
| ayoung | wha | 15:54 |
| fungi | ayoung: depends on the size of the rock you live under | 15:54 |
| sdague | ayoung: apparently you don't read email | 15:54 |
| yolanda | hi, mordred, taking a look at your patches about dib and nodepool. But i don't see in nodepool where images-dir and elements-dir are used, do we need some specific version of nodepool? | 15:54 |
| * NobodyCam needs a bigger rock | 15:54 | |
| ayoung | sdague, I went to be with an empty mailbox. I woke up with 250 message in it...still clearing | 15:55 |
| * sdague also wants a bigger rock | 15:55 | |
| mordred | yolanda: I have a half-finished patch to nodepool | 15:55 |
| fungi | NobodyCam: let me know when you find one, and if you have room for me too | 15:55 |
| mordred | yolanda: that does not work | 15:55 |
| NobodyCam | :) | 15:55 |
| yolanda | mordred, what's missing in nodepool? is there so much work pending? | 15:56 |
| mordred | yolanda: https://review.openstack.org/46482 | 15:56 |
| mordred | yolanda: that patch ^^ needs to be finished | 15:56 |
| mordred | which is mainly about doing sane things with glanceclient | 15:56 |
| *** ihrachyshka has joined #openstack-infra | 15:58 | |
| *** rcarrillocruz has joined #openstack-infra | 15:58 | |
| *** doude has quit IRC | 16:00 | |
| *** ihrachyshka has quit IRC | 16:00 | |
| *** ihrachyshka has joined #openstack-infra | 16:00 | |
| openstackgerrit | A change was merged to openstack-infra/config: Adding barbican-specs to infra configs https://review.openstack.org/97616 | 16:00 |
| *** dizquierdo_afk is now known as dizquierdo | 16:01 | |
| *** praneshp has joined #openstack-infra | 16:02 | |
| fungi | sdague: interestingly, we got a burp of nodes in many places (mostly hpcloud) with time which was waaay off between 20:48 and 20:52 yesterday, right as the ntpdate fix went in... "Job timeout set to: -23365133 minutes" http://logs.openstack.org/24/97924/1/check/check-tempest-dsvm-full/a7ba6ef/console.html message:"timeout: invalid option -- '2'" AND tags:console | 16:02 |
| *** marcoemorais has joined #openstack-infra | 16:02 | |
| *** marun_afk is now known as marun | 16:02 | |
| *** IvanBerezovskiy has left #openstack-infra | 16:02 | |
| *** praneshp has quit IRC | 16:02 | |
| sdague | fungi: that was me screwing up that patch | 16:03 |
| sdague | that's the fail log for that patch | 16:03 |
| sdague | in the bad version | 16:03 |
| fungi | sdague: oh! right, those were the check tests | 16:03 |
| fungi | heh | 16:03 |
| fungi | nevermind! | 16:03 |
| NobodyCam | https://www.youtube.com/watch?v=V3FnpaWQJO0 | 16:04 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer keystone mock bug 1324885 https://review.openstack.org/98159 | 16:05 |
| uvirtbot | Launchpad bug 1324885 in ceilometer/icehouse "py26, py27 gate jobs are failing due to inaccurate mocks of keystone tokens" [Critical,Fix committed] https://launchpad.net/bugs/1324885 | 16:05 |
| fungi | NobodyCam: heh | 16:05 |
| *** starmer has joined #openstack-infra | 16:05 | |
| *** weshay has joined #openstack-infra | 16:06 | |
| NobodyCam | that may show my real age.. but I had to :-p | 16:06 |
| devananda | sdague: are there any other concerns prior to promoting 97757 ? | 16:06 |
| fungi | NobodyCam: nothing to be embarrassed about there... we were lucky and got to watch snl in the couple of years before it began to suck | 16:06 |
| sdague | devananda: not if you are sure it's all the same issue | 16:07 |
| sdague | devananda: but before we do that, how will ironic and nova get over this hump? | 16:07 |
| *** _afazekas is now known as afazekas | 16:08 | |
| devananda | sdague: a) the patch which 97757 reverts broke ironic's gate, therefor b) it's not possible any other broken code landed in ironic after that, and c) we know reverting 97757 fixes ironic, because we've all been doing that for the last ~24 hours locally | 16:08 |
| *** lcheng_ has joined #openstack-infra | 16:08 | |
| *** lcheng_ has quit IRC | 16:08 | |
| sdague | devananda: right I get that | 16:09 |
| devananda | sdague: well, some of us have been applying 97447, which is a local fix to ironic, too, but it actually conflicts with the 97757 revert, so we've since blocked it ... | 16:09 |
| sdague | but is there a coming change to nova that's going to do this again? | 16:09 |
| sdague | or is there a non disruptive way to get the behavior that's desired | 16:09 |
| devananda | sdague: the first patch that broke things actually is doing mostly the right thing | 16:10 |
| devananda | sdague: but it did so by changing the function signature of HostManager __init__ | 16:10 |
| devananda | sdague: which neither nova reviewers nor I caught | 16:10 |
| *** starmer has quit IRC | 16:10 | |
| dstufft | Anyone able to review/approve https://review.openstack.org/#/c/97616/ ? :D It's for adding barbican-specs to the infra configs stuff | 16:10 |
| *** mmaglana has quit IRC | 16:10 | |
| devananda | sdague: so AFAIK, the plan is to revert it (unbreaks nova_bm and ironic), then redo the original patch in a way that doesn't break things | 16:11 |
| *** sandywalsh_ has quit IRC | 16:11 | |
| sdague | gotcha. So there some unit test proposed to prevent that change on the nova side with a scary "this is a contract, don't change it"? | 16:11 |
| *** basha has quit IRC | 16:11 | |
| devananda | jogo: around? want to chime in? | 16:11 |
| *** mmaglana has joined #openstack-infra | 16:11 | |
| *** p5ntangle has joined #openstack-infra | 16:12 | |
| devananda | sdague: not afaik, but others (jogo and lifeless) were working on this stuff when I had to run yesterday, and i haven't been able to sync with them yet today | 16:13 |
| *** e0ne has quit IRC | 16:13 | |
| sdague | fungi: so top of gate, which is the nova issue that's killing lots of stuff, is bouncing off one of the generic neutron network failures | 16:13 |
| sdague | can we just ninja that in, I've got another alleviating fix which can't seem to pass until that one is in | 16:14 |
| *** p5ntangle has quit IRC | 16:14 | |
| fungi | sdague: as long as nova cores are confident the only failures are due to bugs entirely unrelated to that change, then sure | 16:15 |
| SergeyLukjanov | I have highlighting for word "ninja" :) | 16:15 |
| sdague | fungi: it only fails on the neutron job | 16:15 |
| sdague | I'm confident, and still a nova core :) | 16:15 |
| fungi | sdague: 98108,1? | 16:15 |
| sdague | yeh | 16:15 |
| sdague | it's the #2 bug on - http://status.openstack.org/elastic-recheck/ | 16:16 |
| *** mmaglana has quit IRC | 16:16 | |
| mordred | sdague: that's a good question above - we should figure out a way to test some of these interface contracts | 16:16 |
| mordred | sdague: it's code, afterall, we should be able to figure that out | 16:16 |
| sdague | mordred: yeh, if ironic team is depending on certain behavior in nova not to change, the thing to do is build unit tests for that | 16:16 |
| sdague | in nova | 16:16 |
| sdague | with big "don't touch" on it | 16:16 |
| fungi | sdague: once gate-tempest-dsvm-postgres-full completes on it, i'll elevate my privs and submit it to merge | 16:17 |
| sdague | fungi: sounds good | 16:17 |
| *** gyee has joined #openstack-infra | 16:17 | |
| mordred | sdague: ++ | 16:17 |
| devananda | right. fwiw, ironic is depending on several /internal/ interfaces in Nova | 16:18 |
| mordred | sdague: someone could always change the unittest in concert - but that would be more likely to be caught in review | 16:18 |
| sdague | mordred: sure, but put them in test_ironic_contract.py | 16:18 |
| mordred | sdague: ++ | 16:18 |
| *** timrc is now known as timrc-afk | 16:18 | |
| sdague | and then have the ironic team put gerrit watch on it | 16:18 |
| mordred | yup | 16:18 |
| devananda | ++ | 16:18 |
| mordred | if someone really needs to change it - then at least everyone is aware and can plan for it | 16:19 |
| *** openstackgerrit has quit IRC | 16:20 | |
| *** starmer has joined #openstack-infra | 16:20 | |
| *** jgrimm has joined #openstack-infra | 16:21 | |
| *** openstackgerrit has joined #openstack-infra | 16:21 | |
| sdague | right, and you also get to have a very specific conversation about which interfaces everyone agrees on in review | 16:21 |
| sdague | that should reduce the surprises | 16:21 |
| *** Fabio has joined #openstack-infra | 16:22 | |
| *** weshay has quit IRC | 16:22 | |
| *** tkelsey has quit IRC | 16:23 | |
| *** Guest93241 has quit IRC | 16:23 | |
| *** luqas has quit IRC | 16:25 | |
| *** starmer has quit IRC | 16:26 | |
| *** hashar has quit IRC | 16:26 | |
| *** ramashri has joined #openstack-infra | 16:26 | |
| *** sballe has quit IRC | 16:26 | |
| *** ayoung is now known as ayoung-lunch | 16:28 | |
| *** om has joined #openstack-infra | 16:28 | |
| *** che-arne has quit IRC | 16:29 | |
| sdague | fungi: ninja merge on https://review.openstack.org/#/c/98108/ ? | 16:31 |
| sdague | it's done and reported | 16:31 |
| *** jistr has quit IRC | 16:31 | |
| Shrews | sdague: i'll work on the "scary contract change" patch | 16:31 |
| *** doddstack has quit IRC | 16:32 | |
| fungi | sdague: yeah, was just doing that--it's merged now | 16:32 |
| sdague | Shrews: awesome | 16:32 |
| sdague | fungi: sweet | 16:32 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for ceilometer test_ordination_allocate bug 1323524 https://review.openstack.org/98146 | 16:32 |
| uvirtbot | Launchpad bug 1323524 in ceilometer "tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate failed for timeout" [Medium,Confirmed] https://launchpad.net/bugs/1323524 | 16:32 |
| *** thedodd has joined #openstack-infra | 16:33 | |
| sdague | fungi: so I think we should see if the sahara stable/icehouse stuff lands after the current nova patch | 16:33 |
| sdague | then once that resolves ~ 1hr, do the next promote set | 16:34 |
| *** mmaglana has joined #openstack-infra | 16:34 | |
| * fungi also needs to grab lunch once done putting this list of uuids together for phschwartz | 16:34 | |
| sdague | fungi: sure, so an hour respit would be useful :) | 16:34 |
| fungi | ideal | 16:34 |
| sdague | ok, I'm going to do the lunch thing as well, will check in ~ 1:30 EST on any more promotes | 16:35 |
| *** starmer has joined #openstack-infra | 16:35 | |
| SergeyLukjanov | I'll be happy to see sahara stable/icehouse stuff landed for today's 2014.1.1 | 16:35 |
| *** lcheng_ has joined #openstack-infra | 16:36 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer periodic notifier ut bug 1321826 https://review.openstack.org/98149 | 16:37 |
| uvirtbot | Launchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress] https://launchpad.net/bugs/1321826 | 16:37 |
| *** rcarrillocruz has quit IRC | 16:37 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/elastic-recheck: Refine queries for tripleo jobs https://review.openstack.org/98096 | 16:37 |
| zaro | morning | 16:37 |
| anteaya | morning zaro | 16:38 |
| *** asselin_ has quit IRC | 16:38 | |
| *** starmer has quit IRC | 16:40 | |
| *** zzelle has joined #openstack-infra | 16:40 | |
| *** praneshp has joined #openstack-infra | 16:40 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 16:41 | |
| fungi | phschwartz: here are the uuids of the 226 instances where we saw boot with time off by 4 hours, and the times when we deleted them... http://paste.openstack.org/show/82978 | 16:41 |
| *** weshay has joined #openstack-infra | 16:41 | |
| *** Ryan_Lane has joined #openstack-infra | 16:41 | |
| phschwartz | fungi: ty, I will pull up the logs. | 16:41 |
| fungi | phschwartz: note that we have no idea whether this is persisting now, given that we merged a workaround to devstack-gate to run ntpdate before storing start times for timeout delta calculations | 16:42 |
| fungi | which is why there are none after 20:00 utc yesterday | 16:42 |
| *** rcarrillocruz has joined #openstack-infra | 16:42 | |
| * fungi disappears off to lunch | 16:42 | |
| *** asselin has joined #openstack-infra | 16:43 | |
| *** andreaf has joined #openstack-infra | 16:44 | |
| *** sballe has joined #openstack-infra | 16:46 | |
| clarkb | o/ | 16:47 |
| *** cyli has quit IRC | 16:48 | |
| *** praneshp has quit IRC | 16:48 | |
| anteaya | hey clarkb | 16:49 |
| clarkb | fungi: ready to do nodepool whenever you are (assuming it hasn't been done yet) | 16:49 |
| clarkb | mordred: you are the resident python package weirdness expert. want to poke around on nodepool.o.o to see if you can figure out why the nodepool install is so funny? (it is on an older version despite having installed a current version according to pip freeze) | 16:50 |
| *** harlowja_away is now known as harlowja | 16:50 | |
| mordred | clarkb: yeah | 16:50 |
| *** jaypipes has quit IRC | 16:50 | |
| clarkb | mordred: fungi and I were just going to pip uninstall and rm -rf dist-packages/nodepool* and pip install to work around it | 16:50 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326312 https://review.openstack.org/98175 | 16:50 |
| uvirtbot | Launchpad bug 1326312 in ironic "Broken HostManager: TypeError: __init__() takes at most 4 arguments (5 given)" [Critical,In progress] https://launchpad.net/bugs/1326312 | 16:50 |
| *** starmer has joined #openstack-infra | 16:50 | |
| mordred | clarkb: that's probably fine | 16:50 |
| *** cyli has joined #openstack-infra | 16:51 | |
| *** dprince has joined #openstack-infra | 16:54 | |
| clarkb | mordred: thanks | 16:54 |
| *** starmer has quit IRC | 16:55 | |
| *** ihrachyshka has quit IRC | 16:56 | |
| *** ayoung-lunch is now known as ayoung | 16:56 | |
| *** habib has quit IRC | 16:56 | |
| clarkb | fungi: sdague DinaBelova yes I think we want to disable copyallscoresifnocodechange and only copy on trivial rebase | 16:57 |
| *** aconrad has joined #openstack-infra | 16:57 | |
| *** radez is now known as radez_g0n3 | 16:57 | |
| clarkb | BobBall: fungi nodepool uses per test databases | 16:58 |
| clarkb | it is the first project to properly do so | 16:58 |
| clarkb | which is a bit annoying since everyone was screaming they wanted that functionality and I don't think anyone really uses it other than ourselves | 16:58 |
| *** sarob has joined #openstack-infra | 16:58 | |
| *** jlibosva has quit IRC | 16:59 | |
| *** praneshp has joined #openstack-infra | 17:02 | |
| *** isviridov is now known as isviridov|away | 17:03 | |
| *** sandywalsh_ has joined #openstack-infra | 17:03 | |
| *** smarcet has quit IRC | 17:05 | |
| clarkb | fungi: BobBall it looks like the nodepool deficit calculation decided that it needed to add a third node | 17:05 |
| clarkb | which could be a race between test assert and nodepools deficit loop calculation | 17:05 |
| *** markwash_ has joined #openstack-infra | 17:05 | |
| *** smarcet has joined #openstack-infra | 17:06 | |
| *** derekh_ has quit IRC | 17:06 | |
| *** unicell has quit IRC | 17:07 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Add an option to mark uploaded patchsets as WIP https://review.openstack.org/96528 | 17:07 |
| YorikSar | zzelle, fungi: ^ | 17:08 |
| zzelle | YorikSar, hello | 17:08 |
| YorikSar | zzelle, fungi: I finally got what was wrong with "git show" - Git is too old in Ubuntu. | 17:09 |
| *** markwash has quit IRC | 17:09 | |
| *** markwash_ is now known as markwash | 17:09 | |
| YorikSar | And there finally are some docs. | 17:09 |
| zzelle | YorikSar, git show ? | 17:09 |
| clarkb | if you think it is old on precise don't use it on centos6 | 17:09 |
| YorikSar | So please take a look - probably we'll be able to land it this week. | 17:09 |
| YorikSar | zzelle: git show doesn't support --no-patch for that version. | 17:10 |
| YorikSar | zzelle: So I changed it to "git log -n1" for every commit... | 17:10 |
| zzelle | clarkb, in a previous life, i was developping in python 2.4 on centos5 (sniff) | 17:10 |
| morganfainberg | clarkb, there are per-test dbs? | 17:10 |
| morganfainberg | actually... i think keystone has some changes pending to use that. | 17:10 |
| YorikSar | clarkb: Oh... And what's Git version there? | 17:10 |
| clarkb | YorikSar: 1.7.1 | 17:11 |
| YorikSar | clarkb: I'm sitting on hot versions with my Gentoo, so it's very sad to see such denial in the gate | 17:11 |
| sdague | clarkb: can we get that defaulted globally? | 17:11 |
| YorikSar | clarkb: Ok, 12.04 has 1.7.9 | 17:12 |
| clarkb | sdague: get what defaulted? | 17:12 |
| YorikSar | clarkb: I guess, difference is not that big :) | 17:12 |
| sdague | disable copyallscoresifnocodechange and only copy on trivial rebase | 17:12 |
| clarkb | sdague: yeah I think one of us will have to push a change to All-Projects directly according to fungis report on it not being guiable | 17:13 |
| YorikSar | sdague: By the way, about gate... May b it would make sense to throw changes that are too old off "gate" queue to the "check" one and then take them back if they pass? | 17:14 |
| clarkb | YorikSar: we do that | 17:15 |
| YorikSar | sdague: So those changes that are there for 30h - it's 30h in Zuul not in the "gate" queue in particular? | 17:15 |
| clarkb | fungi: BobBall that may be related to the soft locking around listing nodes | 17:15 |
| clarkb | could have introduced a race in the test | 17:15 |
| YorikSar | clarkb: ^ | 17:16 |
| clarkb | YorikSar: no its 30 hours in the gate queue | 17:17 |
| clarkb | but before they got into the gate queue if they had tests more than 72 hours old they had to be retested in check first | 17:17 |
| *** MarkAtwood has joined #openstack-infra | 17:17 | |
| clarkb | since the gate is self checking it is in theory not necessary to bump those out | 17:18 |
| YorikSar | clarkb: Oh, really. | 17:18 |
| *** reed has joined #openstack-infra | 17:18 | |
| clarkb | yes its an ordered list nothing jumps the queue (except for when we manually do that to get fixes to the head of the queue) | 17:18 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for cinder out-of-date conf.sample bug 1326890 https://review.openstack.org/98180 | 17:18 |
| uvirtbot | Launchpad bug 1326890 in cinder "cinder.conf.sample is not up to date" [Undecided,New] https://launchpad.net/bugs/1326890 | 17:18 |
| YorikSar | clarkb: Thanks for setting my mind straight :) | 17:19 |
| *** hashar has joined #openstack-infra | 17:19 | |
| *** starmer has joined #openstack-infra | 17:20 | |
| clarkb | fungi: BobBall: ya I think this is a relationship between watermark_sleep in the test and the sleep() that is performed to wait for nodes | 17:21 |
| *** p5ntangle has joined #openstack-infra | 17:21 | |
| *** hashar has quit IRC | 17:22 | |
| clarkb | http://logs.openstack.org/35/97835/2/check/gate-nodepool-python27/55c3a93/console.html#_2014-06-04_16_49_32_423 is where the race happens I think. Basically node 2 comes on line right after nodepool decides it has a deficit of 1 | 17:23 |
| clarkb | I am going to try bumping watermark sleep to 1, which doubles the amount of time nodepool has to mark that node ready before it checks again | 17:24 |
| *** starmer has quit IRC | 17:25 | |
| *** sballe has quit IRC | 17:25 | |
| *** BadCub01 has quit IRC | 17:26 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Increase watermark sleep in tests for reliability. https://review.openstack.org/98183 | 17:26 |
| clarkb | thats a bit of a hack and we should get away from sleeping if possible but meh | 17:26 |
| *** gokrokve has joined #openstack-infra | 17:27 | |
| apevec | ttx, adam_g - so given gate situation, is pushing 2014.1.1 today realistic - or do tag pushing jobs go to a separate queue from gate? | 17:29 |
| apevec | we also have approved Neutron exceptions 96919 and 96920 still queued in gate | 17:29 |
| clarkb | apevec: the tag pushing jobs are separate | 17:29 |
| clarkb | but if you need to merge things before tagging that is in the gate | 17:29 |
| apevec | yeah, we have those two, I'll ask again on stable-maint if we should wait for them | 17:30 |
| *** marcoemorais has quit IRC | 17:32 | |
| clarkb | apevec: we can promote them if they are critical though we try to reserve that for gate fixes | 17:32 |
| *** marcoemorais has joined #openstack-infra | 17:32 | |
| clarkb | (promotion bumps them to the head of the queue) | 17:32 |
| *** marcoemorais has quit IRC | 17:33 | |
| *** Ryan_Lane1 has joined #openstack-infra | 17:33 | |
| *** marcoemorais has joined #openstack-infra | 17:33 | |
| apevec | agreement was they're critical enough as freeze exception, but not sure if | 17:33 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326312 https://review.openstack.org/98175 | 17:34 |
| uvirtbot | Launchpad bug 1326312 in ironic "Broken HostManager: TypeError: __init__() takes at most 4 arguments (5 given)" [Critical,In progress] https://launchpad.net/bugs/1326312 | 17:34 |
| *** Ryan_Lane has quit IRC | 17:34 | |
| apevec | promotion is justified | 17:34 |
| *** sarob has quit IRC | 17:34 | |
| apevec | I guess, only if there are not current gate fixes in the gate? | 17:34 |
| SlickNik | Hey guys | 17:35 |
| clarkb | apevec: ya and we can always put the gate fixes at the very front :) fwiw I am comfortable promoting those changes if you ened them but would be good to get fungi and sdague's input too | 17:35 |
| clarkb | (and SergeyLukjanov + jhesketh feedback if they are awake) | 17:35 |
| *** starmer has joined #openstack-infra | 17:36 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for setuptools py26 unicode bug 1326878 https://review.openstack.org/98187 | 17:36 |
| uvirtbot | Launchpad bug 1326878 in pbr "Race "TypeError: expecting a bytes object, got unicode" on py26 setup" [Undecided,New] https://launchpad.net/bugs/1326878 | 17:36 |
| clarkb | I am going to reboot for local updates | 17:36 |
| SlickNik | I'm seeing devstack installs failing to install keystone; during create_keystone_accounts it bails out with ERROR: openstackclient.shell Exception raised: six>=1.6.0 | 17:36 |
| clarkb | back in a bit (assuming that new kernel works) | 17:36 |
| SlickNik | Is this a known issue? | 17:36 |
| apevec | ok, let's wait for +1 from more infra folks | 17:36 |
| devananda | jogo: won't https://review.openstack.org/#/c/97369/ prevent E-R from being useful for ironic? | 17:37 |
| clarkb | SlickNik: I am not aware of it but still trying to catch up on scrollback | 17:37 |
| devananda | jogo: "Non-voting jobs are non-voting because the are unstable" -- except for when they're not. Like tempest-dsvm-virtual-ironic | 17:37 |
| clarkb | devananda: aren't your jobs voting? | 17:37 |
| clarkb | or will be voting | 17:37 |
| devananda | clarkb: not on nova. they can't be. remember? | 17:37 |
| devananda | "will be" | 17:37 |
| clarkb | devananda: no the whole point of jhesketh's change is to make them voting.. | 17:37 |
| SlickNik | clarkb: Okay, thanks! Will try and dig in some more. | 17:38 |
| *** dosaboy has quit IRC | 17:38 | |
| devananda | clarkb: gotcha. so that change to E-R won't exclude voting-but-not-gating-because-its-third-party jobs | 17:39 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for glance _authorized_upload_ bug 1326899 https://review.openstack.org/98189 | 17:39 |
| uvirtbot | Launchpad bug 1326899 in glance "FAIL: glance.tests.unit.v1.test_api.TestGlanceAPI.test_add_copy_from_image_authorized_upload_image_authorized" [Undecided,New] https://launchpad.net/bugs/1326899 | 17:39 |
| devananda | ? | 17:39 |
| *** e0ne has joined #openstack-infra | 17:39 | |
| *** Sukhdev has joined #openstack-infra | 17:39 | |
| *** starmer has quit IRC | 17:40 | |
| openstackgerrit | Fathi Boudra proposed a change to openstack-infra/jenkins-job-builder: ssh publisher: add always-publish-from-master support https://review.openstack.org/98190 | 17:40 |
| clarkb | devananda: correct | 17:40 |
| clarkb | I wouldn't expect it to at least | 17:40 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Added query for cinder out-of-date conf.sample bug 1326890 https://review.openstack.org/98180 | 17:41 |
| uvirtbot | Launchpad bug 1326890 in cinder "cinder.conf.sample is not up to date" [Undecided,New] https://launchpad.net/bugs/1326890 | 17:41 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: tracking bug for connection issues between HP and RAX https://review.openstack.org/98112 | 17:41 |
| clarkb | reboot complete. xfce has decided I need a giant mouse pointer | 17:41 |
| *** dosaboy has joined #openstack-infra | 17:41 | |
| clarkb | we are trying to make those jobs voting for a variety of reasons and if this isn't one of them then we should fix e-r :) | 17:42 |
| *** srenatus has quit IRC | 17:42 | |
| *** thedodd has quit IRC | 17:42 | |
| *** ramashri has quit IRC | 17:42 | |
| *** ildikov has quit IRC | 17:42 | |
| *** srenatus has joined #openstack-infra | 17:43 | |
| fungi | clarkb: apevec: i think not delaying stable point releases is probably worth the relatively minor hit from promoting the changes needed for them | 17:43 |
| fungi | YorikSar: we briefly discussed git review -W in the infra meeting this week and want to make sure we don't make it too openstack-configuration-specific... i'm starting to wonder if we shouldn't instead make the behavior of git review -D be configurable in .gitreview files, defaulting to draft behavior but overridable to set label votes (and in projects for review.openstack.org we just patch all ~300 | 17:43 |
| fungi | .gitreview files to set workflow -1 for -D) | 17:43 |
| *** Ajaeger has joined #openstack-infra | 17:43 | |
| fungi | YorikSar: i wouldn't expect that to change the mechanics of the patch too substantially, but we probably ought to go ahead and tag a 1.24 and then we can do 1.25 soon after once the work in progress vs draft idea is sorted | 17:44 |
| clarkb | fungi: ++ | 17:44 |
| *** SlickNik has quit IRC | 17:45 | |
| *** spiffxp_ has quit IRC | 17:45 | |
| fungi | clarkb: i like when someone just generally ++'s me. i assume that's transitive to all my past (and future) ideas | 17:45 |
| *** masayukig has quit IRC | 17:46 | |
| clarkb | seems reasonable | 17:46 |
| pleia2 | hehe | 17:46 |
| * fungi is nothing if not reasonable | 17:46 | |
| *** SlickNik has joined #openstack-infra | 17:46 | |
| clarkb | fungi: so I think nodepool's test issues are just test issues | 17:46 |
| *** basha has joined #openstack-infra | 17:46 | |
| clarkb | fungi: if the patch I pushed comes back clean I am happy to continue restarting nodepool | 17:46 |
| *** pelix has quit IRC | 17:47 | |
| fungi | clarkb: i am willing to accept that. we can stop it, blow away the pip install, manually re-install it from tip in /opt and do the other cleanup, then start it again | 17:47 |
| clarkb | yup | 17:47 |
| clarkb | but we seem to be hurting for nodes again... | 17:47 |
| * fungi reviews said patch | 17:47 | |
| *** spiffxp has joined #openstack-infra | 17:48 | |
| apevec | clarkb, fungi - ok, so we have +1 from you both - one more from sdague for promoting 96919 and 96920 (latter depends on the former) | 17:48 |
| apevec | mestery, ^ your patches | 17:48 |
| fungi | btw, we need to get the non-profit question settled for the foundation and then get it added to https://org.amazon.com/ref=smi_ge_sc_cc_cc so that i can appreciate the irony of amazon donating to a project which is targeted at replacing their cloud services business | 17:48 |
| clarkb | and we are ~24 minutes away from a gate reset | 17:48 |
| apevec | mestery, be ready for sdague's questions :) | 17:49 |
| clarkb | fungi: :) | 17:49 |
| clarkb | sdague: question from me, does that nova backport to icehouse need to be promoted to get the neutron patches to merge? | 17:49 |
| SlickNik | fungi: that's awesome :) | 17:50 |
| *** ramashri has joined #openstack-infra | 17:50 | |
| *** starmer has joined #openstack-infra | 17:50 | |
| apevec | clarkb, which nova backport? | 17:50 |
| *** ctracey has quit IRC | 17:51 | |
| *** masayukig has joined #openstack-infra | 17:51 | |
| clarkb | apevec: let me find it in scrollback | 17:51 |
| *** ctracey has joined #openstack-infra | 17:51 | |
| *** markmcclain has joined #openstack-infra | 17:52 | |
| clarkb | apevec: 97845 | 17:52 |
| apevec | 97845 was merged today | 17:52 |
| clarkb | oh hey it merged | 17:52 |
| clarkb | cool | 17:52 |
| *** basha has quit IRC | 17:53 | |
| *** _nadya_ has quit IRC | 17:53 | |
| *** Sam-I-Am has quit IRC | 17:53 | |
| *** p5ntangle has quit IRC | 17:54 | |
| *** starmer has quit IRC | 17:55 | |
| *** marun has quit IRC | 17:56 | |
| *** marun has joined #openstack-infra | 17:56 | |
| * fungi is catching up on lunchtime scrollback a bit... | 17:57 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for ceilometer test_ordination_presence bug 1326820 https://review.openstack.org/98124 | 17:58 |
| uvirtbot | Launchpad bug 1326820 in ceilometer "FAIL: tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_presence (dup-of: 1323524)" [Undecided,New] https://launchpad.net/bugs/1326820 | 17:58 |
| uvirtbot | Launchpad bug 1323524 in ceilometer "tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate failed for timeout" [Medium,Confirmed] https://launchpad.net/bugs/1323524 | 17:58 |
| fungi | SlickNik: dmakogon_ mentioned that six-related failure a few hours ago. looks like a problem in openstackclient? | 17:58 |
| fungi | i think maybe it's only impacting trove, as i don't think anybody else is seriously using openstackclient in the gate yet | 18:00 |
| *** ramashri has quit IRC | 18:00 | |
| SlickNik | fungi: Yes. Seems like it. | 18:00 |
| *** arnaud__ has joined #openstack-infra | 18:00 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326312 https://review.openstack.org/98175 | 18:00 |
| uvirtbot | Launchpad bug 1326312 in ironic "Broken HostManager: TypeError: __init__() takes at most 4 arguments (5 given)" [Critical,In progress] https://launchpad.net/bugs/1326312 | 18:01 |
| *** ramashri has joined #openstack-infra | 18:01 | |
| clarkb | so we have had similar problems when people pin six | 18:01 |
| clarkb | is openstackclient or trove pinning six? | 18:01 |
| SlickNik | fungi: Looks like devstack's using it in HEAD. | 18:01 |
| fungi | huh | 18:01 |
| clarkb | SlickNik: do you have a link to the failure | 18:01 |
| fungi | right on. maybe that means it'll be production grade and web scale in no time ;) | 18:01 |
| *** Sam-I-Am has joined #openstack-infra | 18:01 | |
| fungi | clarkb: https://rdjenkins.dyndns.org/job/Trove-Gate/3974/console | 18:02 |
| fungi | that was dmakogon_'s example anyway | 18:02 |
| *** signed8bit_ZZZzz is now known as signed8bit | 18:02 | |
| SlickNik | fungi: yes, thanks for pulling that out. | 18:02 |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:02 | |
| * fungi uses openstackclient for his own personal accounts/servers, but has workarounds to deal with things like rackspace's glance endpoint not being correct in their keystone service catalog | 18:03 | |
| clarkb | openstackclient requires >= 1.6.0 | 18:03 |
| fungi | ...and the lack of account password protection | 18:03 |
| clarkb | I am guessing something else pins it | 18:03 |
| *** amcrn has joined #openstack-infra | 18:03 | |
| clarkb | do you have this failure somewhere with a pip freeze capture? | 18:03 |
| clarkb | hard to debug without logs | 18:03 |
| *** denis_makogon has joined #openstack-infra | 18:04 | |
| fungi | i should push up my "really ugly shell script" which will decrypt your account credentials for you on the fly from a blowfish blob using a separate passphrase prompted for at invocation time | 18:04 |
| SlickNik | clarkb: working on that. | 18:05 |
| *** timrc-afk is now known as timrc | 18:05 | |
| *** Sukhdev has quit IRC | 18:05 | |
| *** starmer has joined #openstack-infra | 18:05 | |
| clarkb | fungi: nice | 18:05 |
| fungi | it was a reaction to the incomplete/broken keychain implementation in openstackclient | 18:06 |
| clarkb | fungi: https://review.openstack.org/#/c/98183/ bam +1 | 18:06 |
| fungi | i spent more time trying to get the keychain integration working than i did just putting a wrapper in place to do basically the same thing | 18:06 |
| fungi | 2m 21s | 18:07 |
| clarkb | for personal things I just use the web ui | 18:07 |
| clarkb | I only have two servers | 18:07 |
| clarkb | clearly I am using the cloud wrong | 18:07 |
| fungi | we ought to recheck this several times... we saw it happening at random | 18:07 |
| clarkb | fungi: can do | 18:07 |
| *** vhoward has left #openstack-infra | 18:08 | |
| *** nati_ueno has joined #openstack-infra | 18:08 | |
| fungi | very rough estimate was about 50% occurrence | 18:08 |
| *** signed8bit_ZZZzz is now known as signed8bit | 18:09 | |
| fungi | 97835 is the current example | 18:09 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: removed job configuration duplication in docs https://review.openstack.org/98197 | 18:09 |
| clarkb | ah yup ~ 50% | 18:09 |
| *** starmer has quit IRC | 18:10 | |
| fungi | given that change only touches documentation, it made for an obvious canary | 18:10 |
| *** signed8bit has quit IRC | 18:11 | |
| fungi | i was working to get unit tests running on my dev box, but they seem to want a working mysql configuration so i stopped | 18:11 |
| sdague | fungi: https://etherpad.openstack.org/p/gatetriage-june2014 so I think two promotes should go in | 18:11 |
| *** Sukhdev has joined #openstack-infra | 18:12 | |
| clarkb | sdague: did you see the questions about promoting icehouse neutron changes for release? | 18:12 |
| fungi | sdague: 97842,2 and 97757,1 in that order? | 18:12 |
| clarkb | fungi: yes, 97757 only affects ironic so should go behind the other change | 18:12 |
| clarkb | fungi: and the icehouse changes can go behind those t2 | 18:12 |
| fungi | sdague: and yes, i think the stable branch release managers would appreciate if i promoted remaining bits they need for today's tagging along with these | 18:13 |
| fungi | i can add them to the etherpad as a note | 18:13 |
| sdague | clarkb: I didn't | 18:14 |
| sdague | at this point if they are not in that etherpad, I don't believe in their existance :) | 18:14 |
| clarkb | sdague: see comments from apevec in recent sb | 18:14 |
| clarkb | sdague: well they aren't gate issues | 18:14 |
| clarkb | so separate | 18:14 |
| fungi | sdague: in the etherpad now under "possible" | 18:15 |
| sdague | yeh, if they aren't gate issues I was ignoring them | 18:15 |
| *** reed has quit IRC | 18:15 | |
| *** reed has joined #openstack-infra | 18:16 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: removed job configuration duplication in docs https://review.openstack.org/98197 | 18:16 |
| sdague | are they breaking stable/icehouse gate? | 18:16 |
| sdague | or nice to have | 18:16 |
| fungi | already part of the drafted release notes i think | 18:16 |
| sdague | because I sort of feel mixed jumping the queue for a release | 18:16 |
| clarkb | sdague: I think they need to merge before release can happen | 18:16 |
| clarkb | sdague: right that is why we are asking you :) | 18:16 |
| sdague | :) | 18:17 |
| fungi | technically, 96919 is security-related | 18:17 |
| *** dkliban is now known as dkliban_bbiab | 18:18 | |
| sdague | so consider me a soft -1 on philosophical grounds but feel free to override me on that | 18:18 |
| apevec | I've listed options on stable-maint http://lists.openstack.org/pipermail/openstack-stable-maint/2014-June/002323.html | 18:18 |
| apevec | no replies from stable-maint audience yet | 18:19 |
| *** maxbit_ has joined #openstack-infra | 18:19 | |
| apevec | mestery, ^ what do you say? | 18:19 |
| *** signed8bit has joined #openstack-infra | 18:19 | |
| *** starmer has joined #openstack-infra | 18:20 | |
| *** sarob has joined #openstack-infra | 18:21 | |
| fungi | with vmt hat on i'm inclined to push 96919 up to the front anyway | 18:22 |
| sdague | fungi: I think that's fine | 18:22 |
| sdague | so, honestly, if your are promoting, just go for it. | 18:22 |
| *** maxbit has quit IRC | 18:23 | |
| fungi | one of the two remaining patches for neutron 2014.1.1 addresses a possible security hole, so yeah bringing the other along for the ride is fairly non-impacting | 18:23 |
| *** markmcclain has quit IRC | 18:24 | |
| fungi | so i'm proposing we promote (in sequence) 97842,2 97757,1 96919,1 96920,1 | 18:24 |
| *** starmer has quit IRC | 18:25 | |
| clarkb | sounds good | 18:25 |
| SergeyLukjanov | fungi, all sahara changes failed in gate too | 18:25 |
| SergeyLukjanov | because of neutron | 18:25 |
| SergeyLukjanov | job failed on the first one | 18:25 |
| SergeyLukjanov | could we include them too? | 18:26 |
| fungi | SergeyLukjanov: for 2014.1.1? | 18:26 |
| SergeyLukjanov | fungi, yup | 18:26 |
| *** maxbit has joined #openstack-infra | 18:26 | |
| SergeyLukjanov | https://review.openstack.org/#/q/status:open+project:openstack/sahara+branch:stable/icehouse+topic:sahara-2014.1.1,n,z | 18:26 |
| openstackgerrit | Marc Abramowitz proposed a change to stackforge/python-jenkins: Fix remaining PEP8 errors in docstrings https://review.openstack.org/98204 | 18:26 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Configure Timeline from Story details https://review.openstack.org/93563 | 18:27 |
| *** sarob has quit IRC | 18:27 | |
| fungi | SergeyLukjanov: that's a tougher call. sahara wasn't integrated in icehouse so it technically doesn't have to release point releases in tandem with the integrated projects from icehouse | 18:28 |
| *** maxbit_ has quit IRC | 18:28 | |
| apevec | SergeyLukjanov, yeah, your not about by stable-maint schedule | 18:29 |
| *** andreykurilin_ has joined #openstack-infra | 18:29 | |
| apevec | for stable/icehouse, you'll be when integrated in stable/juno | 18:29 |
| SergeyLukjanov | fungi, I'm ok if you'd like to not doing it | 18:29 |
| e0ne | hi all! i'm tring to setup local CI environment using http://ci.openstack.org/puppet.html. manifests https://github.com/openstack-infra/config are hard-coded to use HTTPS for jenkins, but it isn't necessary for all cases. will be it useful for anybody from a community? i'm ready to implement and propose fix for it | 18:29 |
| SergeyLukjanov | apevec, yup, but we're following the process since the icehouse cycle | 18:29 |
| fungi | and since sahara's icehouse branch isn't covered by the vmt, i can't really advocate for an exception for any security fixes there either | 18:29 |
| *** sarob_ has joined #openstack-infra | 18:29 | |
| apevec | that was bad Enlish. s/your not about/you're not bound/ | 18:29 |
| *** andreaf has quit IRC | 18:30 | |
| apevec | SergeyLukjanov, that's great for training your team so you're ready for stable/juno ! | 18:30 |
| SergeyLukjanov | fungi, apevec, okay, let's keep sahara stable changes as is | 18:30 |
| clarkb | e0ne: is there a problem with using https? | 18:30 |
| clarkb | e0ne: I would argue it should just be used (our implementation uses snake oil certs so not the greatest but better than nothing) | 18:30 |
| fungi | SergeyLukjanov: thanks for understanding! i know it's tough with gate delays impacting release timelines | 18:30 |
| *** sarob_ has quit IRC | 18:31 | |
| e0ne | clarkb: no, but i don't need to use ssl in my local private infra. it's an overhead for me | 18:31 |
| SergeyLukjanov | fungi, np, it's not so critical for now | 18:31 |
| clarkb | e0ne: but it isn't overhead... | 18:32 |
| *** dizquierdo has quit IRC | 18:32 | |
| clarkb | it should just work | 18:32 |
| *** otherwiseguy has quit IRC | 18:32 | |
| fungi | the 1990s called and they want their extra couple of ssl round-trips worth of bandwidth consumption back | 18:32 |
| *** sarob_ has joined #openstack-infra | 18:32 | |
| fungi | (okay, that joke kinda fell flat) | 18:32 |
| e0ne | ok, maybe something went wrong and it doesn't works out-of-box for me | 18:33 |
| fungi | but yes, if ssl/tls processing overhead is straining your jenkins master or clients, you're in the wrong decade | 18:33 |
| SergeyLukjanov | clarkb, "(and SergeyLukjanov + jhesketh feedback if they are awake)" sorry, but I'm not sure that I understand the target for feedback :) | 18:33 |
| clarkb | e0ne: you may need to specify the snakeoil certs as we do (but I thought they were by default) | 18:33 |
| e0ne | clarkb, thanks! | 18:34 |
| jerryz | clarkb, fungi: i met with the same problem pabelanger had a couple of months ago. http://pastebin.com/91S3eAju | 18:35 |
| *** Alexei_987 has joined #openstack-infra | 18:35 | |
| jerryz | turned out that system install of libzmq overrides pip installed pyzmq | 18:36 |
| jerryz | but it was because i followed the instruction in readme of nodepool | 18:36 |
| e0ne | and what about hard-coded git_repos and hostnames? e.g. i want ti use custom jenkins-job-builder repo and/or revision. https://github.com/e0ne/config/commit/b1b6c2d4503c78a354d6f1330345850d505d40eb - a very very draft example of changes. shuold i create bug/blueprint for it? | 18:36 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Added query for cinder out-of-date conf.sample bug 1326890 https://review.openstack.org/98180 | 18:36 |
| uvirtbot | Launchpad bug 1326890 in cinder "cinder.conf.sample is not up to date" [Undecided,New] https://launchpad.net/bugs/1326890 | 18:37 |
| mestery | apevec: Sorry, was afk, here now, what can I help with regarding my patches for stable? | 18:37 |
| fungi | jerryz: would BobBall's documentation patch at https://review.openstack.org/#/c/97835/2/doc/source/installation.rst have helped you avoid that? | 18:37 |
| e0ne | puppetmaster_host is also should be changed for a local env. e.g. https://github.com/e0ne/config/commit/5621c06fa5a6d4a6c989a7bcfb61466a69ad34bb | 18:37 |
| e0ne | my changes is a very draft for now | 18:37 |
| fungi | e0ne: i think asselin may have some changes proposed already to make git repository urls and default branches configurable | 18:38 |
| apevec | mestery, justify their importance, but we have +1 now | 18:38 |
| jerryz | fungi: yeah, it would. but i would also be great to update readme | 18:38 |
| mestery | apevec: OK, thanks! Much appreciated for the help here! | 18:38 |
| fungi | e0ne: but i don't know if you need a bug for it... you're welcome to just propose a patch into our gerrit without filing a bug too | 18:38 |
| apevec | SergeyLukjanov, btw at least one sahara patch didn't fail in neutron but in devstack-gate setup? http://logs.openstack.org/58/97458/1/check/check-tempest-dsvm-neutron-heat-slow/6ee7bbf/console.html | 18:39 |
| fungi | jerryz: do you mind proposing a patch to the readme? | 18:39 |
| sdague | apevec: that was yesterday | 18:39 |
| apevec | ah yes, didn't look at timestamp | 18:40 |
| *** nati_ueno has quit IRC | 18:40 | |
| SergeyLukjanov | apevec, yup, yesterday's one | 18:40 |
| jerryz | fungi: remove libzmq-dev from the installation of dependency part or have the same explanation as in BobBall's patch | 18:40 |
| apevec | just for my education where do you look further for such devstack-gate-setup failure? | 18:40 |
| fungi | mestery: apevec: i've got the remaining neutron icehouse patches ready to push to the front along with a couple gate fixes as soon as we see another tip reset | 18:40 |
| *** om has quit IRC | 18:40 | |
| jerryz | fungi: i think not installing libzmq-dev would do | 18:41 |
| apevec | SergeyLukjanov, nothing in devstack-gate-setup-workspace-new.txt.gz afaict? | 18:41 |
| SlickNik | clarkb / fungi: So I looked at the pip freeze output and it's surprising but "six==1.6.1" | 18:41 |
| e0ne | fungi: i saw only https://review.openstack.org/#/c/96539 from asselin. it doesn't cover all cases | 18:41 |
| mestery | fungi: Thanks! | 18:41 |
| clarkb | SlickNik: huh | 18:41 |
| fungi | e0ne: sounds like additional patches may be called for in that case | 18:41 |
| e0ne | agree | 18:42 |
| rkukura | Anyone have any experience (good, bad, impossible) with gerrit patches that depend on more than one other patch? | 18:42 |
| clarkb | pleia2: fungi mtreinish just booked trip. staying at maritim konferenz hotel in darmstadt | 18:42 |
| fungi | rkukura: tons | 18:42 |
| e0ne | fungi, you're right, thanks! | 18:42 |
| clarkb | rkukura: it is not possible to have multiple parents without a merge commit | 18:42 |
| clarkb | rkukura: you can serialize them though | 18:42 |
| rkukura | fungi: Good, bad? | 18:42 |
| *** gyee has quit IRC | 18:42 | |
| fungi | clarkb: i was leaning toward the intercityhotel since it looked cheaper, but will go with the flow ;) | 18:43 |
| clarkb | fungi: well maritim is HP preferred | 18:43 |
| clarkb | and I could book without a deposit :) | 18:43 |
| rkukura | clarkb: We’ve basically got a matrix of patches, split into layers (API, DB, plugin, …) and columns (sets of resources) | 18:43 |
| fungi | clarkb: given the number of hp'ers coming, that's a good reason | 18:43 |
| rkukura | clarkb: So a typical patch will depend on the layer below it in the same column, and the same layer of the previous column | 18:44 |
| *** om has joined #openstack-infra | 18:44 | |
| fungi | rkukura: well, good experience mostly, but the dependent patches need to be for the same git repository and branch, and need to be sequenced | 18:44 |
| jogo | devananda: only if ironic jobs have " (non-voting)" next to them in the jenkins comment. and if they do we can make a special exception | 18:45 |
| pleia2 | clarkb: cool, thanks | 18:45 |
| clarkb | jogo: we won't have to make an exception shortly we are going to make them voting | 18:45 |
| clarkb | pleia2: and the lufthansa/united flight appears to be lufthansa operated so I took a chance on that | 18:45 |
| jogo | clarkb: even better | 18:45 |
| pleia2 | clarkb: hooray :) | 18:46 |
| rkukura | fungi: Same git repo definitely | 18:46 |
| clarkb | keep in mind you lose a day flying that direction so will have to leave saturday | 18:46 |
| clarkb | (or I had to anyways) | 18:46 |
| pleia2 | clarkb: direct to frankfurt? | 18:46 |
| clarkb | yup | 18:47 |
| pleia2 | cool | 18:47 |
| pleia2 | I've done that trip a few times, but never left the airport, so exciting! | 18:47 |
| fungi | clarkb: pleia2: arriving in frankfurt via united at noon local on sunday | 18:47 |
| rkukura | fungi, clarkb: If I have a merge commit as part of my patch, when I do “git review”, will gerrit treat that merge commit as a separate dependency? | 18:47 |
| clarkb | rkukura: yes the merge commit will have its own change. keep in mind we don't allow merge commits upstream | 18:48 |
| fungi | rkukura: gerrit should treat merge commits just like any other commit, but your account would need to have the necessary access permissions allowing it to push a merge commit to refs/for | 18:48 |
| *** smarcet has quit IRC | 18:48 | |
| rkukura | clarkb, fungi: Is it reasonable to push initial patch sets with the merge commit for review, and rebase after the dependency has been merged? | 18:49 |
| * fungi has to fly norfolk->washington dc->frankfurt... bleagh | 18:49 | |
| clarkb | fungi: our hp and rax error rates look really similar | 18:50 |
| clarkb | kind of scary | 18:50 |
| *** nati_ueno has joined #openstack-infra | 18:50 | |
| clarkb | rkukura: I think you can just push them serially | 18:50 |
| clarkb | rkukura: I think merge commits and rebases just complicate | 18:50 |
| fungi | rkukura: i don't push merge commits for review. i use git rebase to order all the commits and then push that series | 18:50 |
| pleia2 | fungi: heh, I have austin > dc > munich > zagreb next weekend | 18:50 |
| fungi | ew | 18:50 |
| pleia2 | yeah | 18:51 |
| *** thedodd has joined #openstack-infra | 18:51 | |
| clarkb | fungi: I arrive ~9am frankfurt local time on sunday | 18:51 |
| rkukura | clarkb: I was concerned about how to rebase on an update of the branch that I’ve merged | 18:51 |
| clarkb | fungi: depending on how tired I am and light I pack it may be tempting to spend that day in frankfurt before heading to darmstadt | 18:52 |
| fungi | rkukura: if you're developing for code review in gerrit, generally you want to keep your topic branch rebased on the tip of the remote branch it's targeting | 18:52 |
| rkukura | clarkb, fungi: So are you suggesting that I stick with a single chain of dependencies, where the API layer of one column depends on the top (plugin) layer of the previous column, even though it really only depends on the API layer of that column? | 18:52 |
| *** e0ne has quit IRC | 18:53 | |
| devananda | jogo: one does now, pending the change from jhesketh to enable a separate pipeline for it | 18:53 |
| clarkb | rkukura: you can have them branch but a serial list is simple | 18:53 |
| clarkb | rkukura: you just can't have more than one parent | 18:53 |
| fungi | rkukura: yes, basically | 18:53 |
| *** ildikov has joined #openstack-infra | 18:53 | |
| *** e0ne has joined #openstack-infra | 18:53 | |
| rkukura | clarkb, fungi: OK, I’m convinced! | 18:54 |
| fungi | gerrit deals well with changes with more than one child commit, but merges (a change with multiple parent commits) are not handled well... the code review interface is fairly messy | 18:54 |
| fungi | and also needs special permission to push those anyway | 18:54 |
| jogo | devananda: when that lands ping me, we can add a non-voting whitelist to elastic-recheck | 18:55 |
| rkukura | fungi, clarkb: Thanks! | 18:55 |
| *** andreykurilin_ has quit IRC | 18:56 | |
| *** andreykurilin_ has joined #openstack-infra | 18:56 | |
| sdague | fungi: did you do the promote? | 18:57 |
| fungi | sdague: was waiting for a nasty gate reset | 18:58 |
| sdague | ah ok | 18:58 |
| fungi | there are a few changes in there which are ~10 minutes from clearing out of the gate | 18:58 |
| *** e0ne has quit IRC | 18:58 | |
| *** sarob_ has quit IRC | 18:58 | |
| fungi | so i don't want to reset their timers unnecessarily | 18:58 |
| devananda | jogo: we should add the non-voting whitelist for [check,gate]-tempest-dsvm-ironic-nv now, and remove it once we're reporting as a third-party CI to nova | 18:59 |
| *** arnaud__ has quit IRC | 19:00 | |
| jogo | devananda: yes, although that won't happen until the gate issues are sorted out | 19:00 |
| devananda | jogo: to recap: there's an ironic*nv job now that we'd like to have included in recheck. several patches are in flight, and some more to be written, which will make it voting (as a third party) sometime this cycle | 19:00 |
| devananda | jraim: absolutely :) gate block comes first | 19:00 |
| devananda | jogo: ^ | 19:00 |
| *** sarob has joined #openstack-infra | 19:01 | |
| clarkb | ok I am going to go find food | 19:01 |
| *** smarcet has joined #openstack-infra | 19:02 | |
| *** ramashri_ has joined #openstack-infra | 19:03 | |
| *** ramashri has quit IRC | 19:03 | |
| *** adalbas has quit IRC | 19:04 | |
| jogo | sdague: https://review.openstack.org/#/c/98000/ wondering if you think that would help at all or not | 19:04 |
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/nodepool: Remove libzmq-dev from dependency list https://review.openstack.org/98218 | 19:04 |
| anteaya | SergeyLukjanov: if you are still around, did you find anytime to answer any new third party ci requests? | 19:05 |
| sdague | jogo: shouldn't rwe already be bailing there? | 19:05 |
| sdague | || /bin/true | 19:05 |
| * fungi has a bucket in hand for bailing | 19:05 | |
| *** starmer has joined #openstack-infra | 19:05 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: tracking bug for connection issues between HP and RAX https://review.openstack.org/98112 | 19:05 |
| sdague | fungi: or a tractor, if they are made of hay | 19:06 |
| *** ramashri has joined #openstack-infra | 19:06 | |
| fungi | bailing wire. that's always had a great connotation | 19:06 |
| *** nati_ueno has quit IRC | 19:07 | |
| sdague | fungi: also, I'm going to backport the get-pip fixes to icehouse and havana | 19:07 |
| sdague | or more importantly, I did, and pushed those | 19:07 |
| fungi | okay, awesome | 19:08 |
| *** marcoemorais1 has joined #openstack-infra | 19:08 | |
| sdague | those might be sensible promotes | 19:08 |
| fungi | already approved? | 19:08 |
| *** james_li has joined #openstack-infra | 19:08 | |
| *** ramashri_ has quit IRC | 19:08 | |
| fungi | if so, i can tack them on | 19:08 |
| *** marcoemorais has quit IRC | 19:08 | |
| sdague | dtroyer's not back yet. If you want to review them, I'll self approve if there are a couple of sane +1s on them | 19:09 |
| sdague | https://review.openstack.org/#/c/98211/ | 19:09 |
| sdague | https://review.openstack.org/#/c/98210/ | 19:09 |
| *** jaypipes has joined #openstack-infra | 19:09 | |
| sdague | I realized I put this one under it - https://review.openstack.org/#/c/98209, let me restack that | 19:10 |
| jogo | sdague: I think we used to continue since we had 3rd part apt resources | 19:10 |
| *** starmer has quit IRC | 19:10 | |
| sdague | jogo: no, I'm saying, we do now as well | 19:10 |
| sdague | fungi: ok, I restacked so the icehouse get-pip doesn't rely on anything | 19:11 |
| *** sarob has quit IRC | 19:11 | |
| james_li | Hi clarkb , fungi we (Solum team) wants to enable docker on our gate, we are aware that F20 supports docker fine. We also like to know when Trusty will be an option for gating? | 19:11 |
| jogo | sdague: ahh but do you think a retry would help? | 19:11 |
| *** sarob has joined #openstack-infra | 19:11 | |
| sdague | jogo: why? | 19:11 |
| jogo | sdague: I was thinking the apt mirror failures could be very transient | 19:12 |
| jogo | and a retry may just fix it. (but this is just a shot in the dark) | 19:12 |
| sdague | sure, but why is retrying the package index grab going to help? | 19:12 |
| sdague | because it's not fatal | 19:12 |
| dtroyer | sdague: fwiw, I'm kinda back now…just getting started looking at backlog stuff. anything specific I should start with? | 19:13 |
| jogo | sdague: ahh, because apt-get install fails in those cases | 19:13 |
| jogo | sdague: so the failure happens in the next step | 19:13 |
| sdague | dtroyer: well the gate is kind of backed up, but the 3 patches above would be good | 19:13 |
| sdague | jogo: log link? | 19:14 |
| devananda | sdague: think its worth adding one unit test for each public method of nova.virt.driver.virt_driver:ComputeDriver that we're using in ironic.nova.virt.ironic.driver.IronicDriver? | 19:14 |
| anteaya | hey dtroyer, welcome back | 19:14 |
| sdague | devananda: sounds fair to me | 19:14 |
| fungi | sdague: 98209,2 is an icehouse backport of a gate stability fix as well? | 19:15 |
| devananda | got the path wrong (nova.virt.driver:ComputeDriver) | 19:15 |
| devananda | sdague: ack, tanks | 19:15 |
| sdague | fungi: no, it's a make grenade work better for me backport | 19:15 |
| fungi | ahh | 19:15 |
| sdague | which is why it's on top | 19:15 |
| *** stevebaker has quit IRC | 19:15 | |
| *** stevebaker has joined #openstack-infra | 19:15 | |
| fungi | right, got it | 19:15 |
| sdague | if we consider the bottom to be closest to master | 19:15 |
| sdague | it can land the normal way | 19:16 |
| fungi | yup | 19:16 |
| *** sarob has quit IRC | 19:16 | |
| *** homeless has quit IRC | 19:17 | |
| zzelle | hi everyone | 19:17 |
| *** W00die has quit IRC | 19:17 | |
| *** gargola has quit IRC | 19:17 | |
| *** gargola has joined #openstack-infra | 19:17 | |
| *** w00dy has joined #openstack-infra | 19:17 | |
| zzelle | is it normal that no jobs are run in Zuul post pipeline ? | 19:17 |
| *** w00dy is now known as Guest43801 | 19:18 | |
| jogo | sdague: is it safe to clse https://bugs.launchpad.net/openstack-ci/+bug/1298929 no e-r hits? | 19:18 |
| uvirtbot | Launchpad bug 1298929 in openstack-ci "certificate errors downloading pip or installing pip packages" [Undecided,New] | 19:18 |
| *** homeless has joined #openstack-infra | 19:18 | |
| sdague | jogo: yeh, I think so | 19:19 |
| SergeyLukjanov | anteaya, I can do it now I think | 19:19 |
| anteaya | SergeyLukjanov: awesome, thank you | 19:20 |
| SergeyLukjanov | anteaya, do we just asking them to wait for some more time? | 19:20 |
| anteaya | are we asking who to wait? | 19:20 |
| SergeyLukjanov | (I'm afraid I could skip something about it in logs) | 19:20 |
| *** tkelsey has joined #openstack-infra | 19:20 | |
| SergeyLukjanov | anteaya, 4rd party testing acc requesters | 19:20 |
| mtreinish | clarkb: cool | 19:20 |
| SergeyLukjanov | heh, 3rd* | 19:20 |
| *** maxbit_ has joined #openstack-infra | 19:21 | |
| sdague | I want 5th party testing | 19:21 |
| SergeyLukjanov | sdague, it could help | 19:21 |
| *** sarob has joined #openstack-infra | 19:21 | |
| jogo | sdague: log link http://logs.openstack.org/38/96738/2/gate/gate-tempest-dsvm-neutron/f15f60b/logs/devstacklog.txt.gz | 19:21 |
| SergeyLukjanov | *probably* | 19:21 |
| anteaya | SergeyLukjanov: no, as long as you and I and fungi are happy with the naming scheme I outlined yesterday, we have no cause to wait for account creation | 19:21 |
| anteaya | sdague: buckaroo bonzai and the 8th dimension | 19:21 |
| anteaya | I want testing in 8 dimensions | 19:22 |
| SergeyLukjanov | anteaya, okay, I'll check request and return back with names | 19:22 |
| anteaya | SergeyLukjanov: thanks | 19:22 |
| anteaya | you can add your suggestions to the bottom of that etherpad if you like | 19:22 |
| fungi | zzelle: the post pipeline is backed up about an hour because of an inrush in the check pipeline. once zuul finishes dealing with check changes older than the post commits, it will assign workers to those jobs | 19:22 |
| anteaya | https://etherpad.openstack.org/p/automated-gerrit-account-naming-format | 19:23 |
| anteaya | I wish I could get that pretty colour of pink back, but I can't seem to find it | 19:23 |
| sdague | jogo: we have a deb expert around? | 19:23 |
| jogo | fungi: ^ ? | 19:23 |
| sdague | because I don't feel like doing the update would help there | 19:23 |
| fungi | sdague: depends on what kind of expertise you're expecting | 19:23 |
| SergeyLukjanov | anteaya, heh, the first requester I see ask for "jenkins-cicd" account ;) | 19:23 |
| anteaya | yeah, so that's no | 19:24 |
| sdague | jogo: I think this is the issue - http://logs.openstack.org/38/96738/2/gate/gate-tempest-dsvm-neutron/f15f60b/logs/devstacklog.txt.gz#_2014-06-05_16_33_50_393 | 19:24 |
| sdague | which triggers the error | 19:24 |
| anteaya | SergeyLukjanov: did they give a company name? | 19:24 |
| zzelle | fungi,fine, just checking | 19:24 |
| sdague | we could always --force-yes | 19:24 |
| sdague | and see what happens | 19:24 |
| *** maxbit has quit IRC | 19:24 | |
| SergeyLukjanov | anteaya, nope, /me writing an email | 19:24 |
| anteaya | cool | 19:25 |
| anteaya | hopefully you can also get them to tell you what their thing does | 19:25 |
| anteaya | that might help with the naming too | 19:25 |
| fungi | zzelle: yeah, there's a heat patch series 24 changes deep starting at https://review.openstack.org/96927 which absorbed all our quota a little over an hour ago | 19:25 |
| *** andreykurilin_ has quit IRC | 19:25 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove some queries with no hits https://review.openstack.org/98224 | 19:26 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added field restrictions and error messages to project forms https://review.openstack.org/95873 | 19:26 |
| jogo | sdague: right, so the question is will a working pass at apt-get update help with that or not. | 19:26 |
| *** gokrokve_ has joined #openstack-infra | 19:26 | |
| fungi | jogo: sdague: so that looks like a broken mirror | 19:27 |
| zzelle | fungi, fine | 19:27 |
| *** otherwiseguy has joined #openstack-infra | 19:27 | |
| sdague | fungi: which means nothing will help, right? | 19:28 |
| jogo | fungi: well the apt-get update command fails half way through | 19:28 |
| sdague | jogo: no, it fails all the way through | 19:28 |
| sdague | I think the error messages a just confusing in apt | 19:29 |
| sdague | but I don't think it did anything successfully | 19:29 |
| *** e0ne has joined #openstack-infra | 19:29 | |
| *** _nadya_ has joined #openstack-infra | 19:30 | |
| fungi | sdague: jogo: the last thing it did successfully was get http://mirror.rackspace.com precise-security Release.gpg | 19:30 |
| *** gokrokve has quit IRC | 19:30 | |
| fungi | then it ignores a bunch of stuff which is already new enough to stay cached | 19:30 |
| fungi | then fails to get http://mirror.rackspace.com precise/main Sources | 19:30 |
| fungi | and everything it wants after that also fails | 19:30 |
| SergeyLukjanov | anteaya, hp-cinder-ci vs hp-storage-ci? | 19:31 |
| sdague | isn't there like an apt bt client | 19:31 |
| *** tkelsey has quit IRC | 19:31 | |
| fungi | so the issue there is apt-get update didn't really work and ran into connectivity issues partway through, then we proceeded to try and apt-get upgrade anyway which predictably fails due to signature/checksum mismatches | 19:32 |
| anteaya | SergeyLukjanov: they fit the format, as long as hp is only going to have one 3rd party ci account testing cinder | 19:32 |
| jogo | sdague: that would be pretty neat | 19:33 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add puppet-openstack project puppet-openstack_extras https://review.openstack.org/97647 | 19:33 |
| jogo | fungi: so perhaps a simple retry may help? but I wonder if using bt would be better | 19:33 |
| mtreinish | sdague, jogo: https://wiki.debian.org/DebTorrent | 19:33 |
| anteaya | SergeyLukjanov: we can create that as requested and if hp asks for a second account to test cinder, they have to name it something else | 19:33 |
| * jogo looks at the clock and realizes he hasn't eaten today and wonders off to lunch | 19:33 | |
| sdague | mtreinish: yeh, it looks pretty unfinished | 19:33 |
| SergeyLukjanov | anteaya, they are asking for hp-cinder-ci "HP Storage CI", I'll ask them to confirm one of the names | 19:34 |
| fungi | jogo: well, i _suspect_ that retrying apt-get update there probably wouldn't have helped. there was probably a somewhat prolonged period of time where that slave was failing to reach the mirror (the mirror might have even been offline) | 19:34 |
| sdague | fungi: would it make sense to put multiple mirrors in the definitions? | 19:35 |
| anteaya | SergeyLukjanov: kk, I wouldn't do HP Storage CI since that team probably doens't know about swift or trove or any of the other storage services | 19:35 |
| sdague | or will that just lead to more explody | 19:35 |
| *** starmer has joined #openstack-infra | 19:35 | |
| fungi | sdague: it would need some testing. i believe the issue which would arise is if the mirror which was unreachable was the one the slave thought had the most recent version of some package | 19:36 |
| anteaya | SergeyLukjanov: what happens if we give them hp-cinder-ci HP Cinder CI and tell them that is what it is? | 19:36 |
| anteaya | the username has to match the Full Name anyway | 19:36 |
| sdague | fungi: it looks like we just stared over again | 19:36 |
| sdague | if you were looking for a rebase window | 19:36 |
| fungi | sdague: yup | 19:36 |
| fungi | promoting now | 19:36 |
| SergeyLukjanov | anteaya, I think we should do it | 19:37 |
| fungi | did those two devstack backports get a thumbs-up from dtroyer? | 19:37 |
| fungi | looks like it | 19:37 |
| *** afazekas is now known as _afazekas_slp | 19:37 | |
| fungi | enqueuing those and then promoting all together | 19:37 |
| dtroyer | fungi: the get-pip.py ones? yes | 19:37 |
| fungi | moved 97842,2 97757,1 98210,2 96919,1 96920,1 98211,1 to the front | 19:38 |
| *** gokrokve_ has quit IRC | 19:38 | |
| anteaya | SergeyLukjanov: let'd do it then | 19:38 |
| sdague | fungi: thanks | 19:39 |
| *** gokrokve_ has joined #openstack-infra | 19:40 | |
| *** starmer has quit IRC | 19:40 | |
| dhellmann | sdague: I don't understand this requirements check failure error message, and would appreciate your interpretation if you have a minute: http://logs.openstack.org/39/96239/1/check/gate-oslo.i18n-requirements/4944868/console.html | 19:40 |
| sdague | dhellmann: so this was a thing that someone was looking at yesterday | 19:41 |
| sdague | that enforcement is supposed to only happen on openstack/requirements | 19:41 |
| Ajaeger | dhellmann: I think that was reverted yesterday | 19:41 |
| sdague | not on the projects | 19:41 |
| dhellmann | ah, ok, so maybe a recheck will let it pass? | 19:41 |
| sdague | so we need the script to change | 19:41 |
| sdague | Ajaeger: I don't think it did | 19:41 |
| sdague | because the proposal was to delete everything | 19:41 |
| dhellmann | or, oh, I need to wait for a job change | 19:42 |
| *** om has quit IRC | 19:42 | |
| sdague | fungi: who was looking at that? | 19:42 |
| * Ajaeger tries to figure out what he saw flying by ;) | 19:42 | |
| *** krotscheck has left #openstack-infra | 19:43 | |
| *** maxbit has joined #openstack-infra | 19:44 | |
| Ajaeger | dhellmann: https://review.openstack.org/#/c/97893/ - not reverted yet | 19:45 |
| *** arnaud__ has joined #openstack-infra | 19:45 | |
| *** gyee has joined #openstack-infra | 19:46 | |
| *** maxbit_ has quit IRC | 19:47 | |
| *** krotscheck has joined #openstack-infra | 19:50 | |
| SergeyLukjanov | anteaya, http://lists.openstack.org/pipermail/openstack-infra/2014-June/001302.html | 19:50 |
| * anteaya clicks | 19:50 | |
| SergeyLukjanov | anteaya, what we think about personal name CI? | 19:50 |
| SergeyLukjanov | anteaya, IMO it's bad | 19:50 |
| *** starmer has joined #openstack-infra | 19:50 | |
| *** nati_ueno has joined #openstack-infra | 19:51 | |
| anteaya | okay great, email the person who requested the account and ask for additional details | 19:51 |
| *** maxbit has quit IRC | 19:52 | |
| SergeyLukjanov | anteaya, acl | 19:52 |
| SergeyLukjanov | ack* | 19:52 |
| *** maxbit has joined #openstack-infra | 19:52 | |
| *** apevec has quit IRC | 19:52 | |
| fungi | sdague: i replied to your -1 saying that the "fix" isn't just a matter of changing that script. the change was made to the wrong project | 19:53 |
| SergeyLukjanov | fungi, anteaya, re 3rd party accs: one account created (hp-cinder-ci) and three name clarification requests sent | 19:53 |
| *** ramashri has quit IRC | 19:53 | |
| dhellmann | Ajaeger: thanks | 19:53 |
| SergeyLukjanov | fungi, anteaya, no more pending requests | 19:54 |
| fungi | sdague: regarding the revert proposed in https://review.openstack.org/97893 | 19:55 |
| *** starmer has quit IRC | 19:55 | |
| *** _nadya_ has quit IRC | 19:56 | |
| *** gokrokve_ has quit IRC | 19:57 | |
| fungi | sdague: and then i reopened the original bug 1274731 which hasn't actually been fixed | 19:57 |
| uvirtbot | Launchpad bug 1274731 in openstack-ci "non overalapping change to requirements repo broke gate" [Medium,Confirmed] https://launchpad.net/bugs/1274731 | 19:57 |
| sdague | fungi: ok, but that's all the right logic to do it on gr | 19:57 |
| *** dkranz has quit IRC | 19:57 | |
| anteaya | SergeyLukjanov: great, thank you | 19:57 |
| sdague | so shouldn't we instead make that script take a flag about which mode to run in | 19:57 |
| sdague | basically I actually think we should consider fixing the base bug atomic with this | 19:58 |
| sdague | because otherwise the base bug is never going to be fixed | 19:58 |
| *** otherwiseguy has quit IRC | 19:58 | |
| fungi | i'm not sure why, since the base bug was never fixed at all and this was merely a new bug introduced | 19:58 |
| fungi | sdague: well... the project-requirements-change.py slave script is one which gets run projects consuming global requirements. it isn't used for checking the openstack/requirements project changes | 19:59 |
| *** otherwiseguy has joined #openstack-infra | 19:59 | |
| *** masayukig has quit IRC | 19:59 | |
| fungi | and the functions which were added to it really don't have much else to do with that script | 19:59 |
| sdague | fungi: ok | 19:59 |
| sdague | looks like yolanda handled that patch the first time, maybe we can convince her to pick up the base bug for this :) | 20:00 |
| fungi | the functions can probably be mostly extracted into a script committed to openstack/requirements and run from a tox env as a new job | 20:00 |
| fungi | since they're only relevant to that one project, openstack-infra/config is the wrong place for that logic to reside | 20:01 |
| fungi | as ir can't be self-testing there | 20:01 |
| *** esker has joined #openstack-infra | 20:02 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for ceilometer keystone mock bug 1324885 https://review.openstack.org/98159 | 20:02 |
| uvirtbot | Launchpad bug 1324885 in ceilometer/icehouse "py26, py27 gate jobs are failing due to inaccurate mocks of keystone tokens" [Critical,Fix committed] https://launchpad.net/bugs/1324885 | 20:02 |
| sdague | fungi: ok, I cede | 20:03 |
| sdague | looks like the nova revert is failing again | 20:03 |
| fungi | sdague: i've updated bug 1274731 with additional guidance on addressing it | 20:03 |
| uvirtbot | Launchpad bug 1274731 in openstack-ci "non overalapping change to requirements repo broke gate" [Medium,Confirmed] https://launchpad.net/bugs/1274731 | 20:03 |
| *** masayukig has joined #openstack-infra | 20:04 | |
| *** markwash has quit IRC | 20:04 | |
| sdague | ok, I'm stepping away from 'puter for a bit. Need to take a break otherwise will brain mush :) | 20:04 |
| *** weshay has quit IRC | 20:04 | |
| *** starmer has joined #openstack-infra | 20:05 | |
| *** ramashri has joined #openstack-infra | 20:06 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1326312 https://review.openstack.org/98175 | 20:06 |
| uvirtbot | Launchpad bug 1326312 in ironic "Broken HostManager: TypeError: __init__() takes at most 4 arguments (5 given)" [Critical,In progress] https://launchpad.net/bugs/1326312 | 20:06 |
| clarkb | ok food coma acquired | 20:06 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer periodic notifier ut bug 1321826 https://review.openstack.org/98149 | 20:07 |
| uvirtbot | Launchpad bug 1321826 in ceilometer "periodic notifier unit test failure" [Medium,In progress] https://launchpad.net/bugs/1321826 | 20:07 |
| openstackgerrit | Adrian Otto proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge https://review.openstack.org/98237 | 20:07 |
| anteaya | clarkb: well done | 20:08 |
| clarkb | james_li: we are hoping to have trusty real soon now | 20:08 |
| fungi | it's out old friend, linux ephemeral ports! "error: [Errno 98] Address already in use" http://logs.openstack.org/57/97757/1/gate/gate-tempest-dsvm-neutron-large-ops/c591f0e/logs/screen-key.txt.gz#_2014-06-05_19_48_00_625 | 20:08 |
| clarkb | james_li: but we keep fighting old fires | 20:08 |
| fungi | s/out/our/ | 20:08 |
| clarkb | s/old/old and new/ | 20:08 |
| *** nati_ueno has quit IRC | 20:08 | |
| fungi | clarkb: what's old is new again | 20:08 |
| fungi | clarkb: new is the new old | 20:08 |
| anteaya | wasn't that a kernel bug last time? | 20:08 |
| clarkb | anteaya: no | 20:09 |
| james_li | clarkb: cool. | 20:09 |
| anteaya | or am I mis-remembering | 20:09 |
| clarkb | anteaya: very much a keystone and IANA sillyness | 20:09 |
| anteaya | kk, I'm wrong | 20:09 |
| anteaya | ah | 20:09 |
| clarkb | anteaya: well not completely | 20:09 |
| fungi | anteaya: s/bug/intentional design choice/ | 20:09 |
| clarkb | anteaya: certain people tried to frame it that way. I think linux is actually sane here | 20:09 |
| anteaya | kk | 20:09 |
| *** starmer has quit IRC | 20:10 | |
| james_li | clarkb: fungi when you are done, I think it will be announced via ML? | 20:10 |
| james_li | meaning trusty | 20:10 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: WIP: Add various translation jobs https://review.openstack.org/98238 | 20:10 |
| fungi | james_li: i'm pretty sure there will be much discussion on the ml as soon as people are ready to start porting jobs to 14.04, yes | 20:10 |
| fungi | james_li: there are lots of improvements waiting on us being able to run things on newer ubuntu lts | 20:11 |
| james_li | ok, thanks! | 20:11 |
| sdague | james_li: helpers would probably be appreciated | 20:11 |
| james_li | :) | 20:11 |
| *** adalbas has joined #openstack-infra | 20:11 | |
| *** Sukhdev has quit IRC | 20:11 | |
| fungi | james_li: certainly an easy way to help is to try to run devstack, tempest, unit tests for various projects, et cetera on trusty yourself and report bugs/propose fixes | 20:12 |
| sdague | fungi: it mostly works, I'm on trusty for all my normal development | 20:12 |
| fungi | the more things "just work" when we fire them up on trusty, the easter that will go | 20:12 |
| *** weshay has joined #openstack-infra | 20:12 | |
| james_li | fungi: yeah, I can definitly help on that, | 20:12 |
| *** dkliban_bbiab is now known as dkliban | 20:13 | |
| fungi | sdague: not sure if you saw, but the bug which bounced the nova revert out this time was the keystone vs. ephemeral ports cage match | 20:14 |
| sdague | oh, nice | 20:14 |
| *** lcheng_ has quit IRC | 20:14 | |
| sdague | well morganfainberg is going to fix that by getting apache working :) | 20:14 |
| fungi | very much still a problem. very much still no good solution | 20:14 |
| fungi | oh, apache front-end. yes | 20:14 |
| sdague | yep | 20:14 |
| clarkb | sdague: I think he mentioned that the test would have to go bcak to experimental though | 20:14 |
| clarkb | because lolnoonefixesanything | 20:14 |
| * fungi fixes nothing | 20:15 | |
| sdague | heh | 20:15 |
| openstackgerrit | Adrian Otto proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge https://review.openstack.org/98237 | 20:16 |
| fungi | sdague: oh, and your havana backport for get-pip is being stomped on by "BuildErrorException: Server %(server_id)s failed to build and is in ERROR status" | 20:17 |
| *** adrian_otto has joined #openstack-infra | 20:17 | |
| sdague | fungi: nice | 20:17 |
| *** rlandy has quit IRC | 20:17 | |
| sdague | oh havana | 20:17 |
| sdague | ok, walking away from computer for real now | 20:17 |
| * fungi is thrilled that we agreed at the summit to support havana for even longer... sigh | 20:17 | |
| *** hemna_ has joined #openstack-infra | 20:17 | |
| sdague | I thought it was icehouse? | 20:18 |
| fungi | oh, right, icehouse | 20:18 |
| sdague | but regardless, I want that conversation back on the ML | 20:18 |
| fungi | yeah, that's right. we agreed that the keeping havana working ship had already sailed | 20:18 |
| sdague | because the additional load on infra and qa is to the point where I'm not agreed on anything unless more helpers show up | 20:18 |
| openstackgerrit | Adrian Otto proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge https://review.openstack.org/98237 | 20:19 |
| adrian_otto | ok, done revising. | 20:19 |
| *** devananda has quit IRC | 20:19 | |
| mgagne | dhellmann: are you working on #1326811 ? | 20:19 |
| *** HenryG has quit IRC | 20:19 | |
| mgagne | bug 1326811 | 20:19 |
| uvirtbot | Launchpad bug 1326811 in trove "Client failing with six =>1.6 error" [Undecided,New] https://launchpad.net/bugs/1326811 | 20:19 |
| *** godp1301 has joined #openstack-infra | 20:20 | |
| *** slamont has joined #openstack-infra | 20:20 | |
| *** mat128 has joined #openstack-infra | 20:20 | |
| *** starmer has joined #openstack-infra | 20:20 | |
| mgagne | dhellmann: we found a very unfortunate timing. get-pip.py installs 1.5.6 by default now instead of what I suspect was 1.4.1 before: https://github.com/pypa/pip/commit/2b0b2f8ff3d5fc8674665c847cf9b1947f70418b | 20:21 |
| Ajaeger | fungi: for how long do we want to support icehouse? I missed that one | 20:21 |
| dhellmann | mgagne: yeah, I was looking at that | 20:22 |
| mgagne | dhellmann: and devstack used to be pinned to 1.4.1 before this change got merged: https://github.com/openstack-dev/devstack/commit/7b63c5ec9e9aea54024105a6521a754efc9f9983 | 20:22 |
| *** Samolo has joined #openstack-infra | 20:22 | |
| james_li | fungi: clarkb : just curious about if any openstack projects already use Trusty for their gates ? | 20:23 |
| fungi | Ajaeger: https://etherpad.openstack.org/p/StableIcehouse "ACTION: Agreed on 15 months" | 20:23 |
| dhellmann | mgagne: what does pip 1.5.6 do differently? | 20:23 |
| fungi | mgagne: dhellmann: right, we're back to consuming latest pip there | 20:24 |
| james_li | fungi: clarkb basically if I test everthing runs fine for Solum, can I switch to Trusty for our gate? | 20:24 |
| fungi | mgagne: dhellmann: there are still a few places where we're stuck on 1.4.1 due to needing to wait for a tox 1.8 release | 20:24 |
| *** starmer has quit IRC | 20:24 | |
| Ajaeger | thanks, fungi | 20:25 |
| dhellmann | how does the newer pip trigger the problem with six being downgraded? | 20:25 |
| fungi | james_li: the current blocker is that we haven't tried actually adding ubuntu 14.04 lts workers to our nodepool yet, and until very recently none of our providers had base images for it nor glance services we could use to upload | 20:25 |
| mgagne | dhellmann: people are reporting that installing pip 1.4.1 instead works (by reverting 7b63c5ec9e9aea54024105a6521a754efc9f9983) | 20:25 |
| dhellmann | mgagne: I understand that. I don't understand why. :-) | 20:25 |
| dhellmann | mgagne: is there a review for devstack to revert that change? | 20:26 |
| fungi | mgagne: dhellmann: right, it's almost certainly either something unorthodox in the six packaging, or a dependency ordering issue which is being exposed through happenstance in later pip | 20:26 |
| mgagne | dhellmann: pip 1.5.6 does not require setuptools anymore. dependencies look to be handled differently | 20:26 |
| * dhellmann sigh | 20:27 | |
| mgagne | fungi: one of my coworker has similar issues with dependency resolutions when upgrading to pip 1.5.6 | 20:27 |
| james_li | fungi: sorry “our providers” mean what? | 20:27 |
| mgagne | fungi: I'll check with him | 20:27 |
| clarkb | mordred: I have reviewed the nodepool dib change. Please see cover comments and inline comments | 20:27 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for oslo.messaging unit test race bug 1283926 https://review.openstack.org/98244 | 20:27 |
| uvirtbot | Launchpad bug 1283926 in oslo.messaging "test_rabbit.TestRacyWaitForReply failed with a not related change" [Low,Fix committed] https://launchpad.net/bugs/1283926 | 20:27 |
| *** andreaf has joined #openstack-infra | 20:27 | |
| fungi | dhellmann: mgagne: i don't expect that commit to be reverted in its entirety, as it was a fix to a pretty severe gate-blocking bug (manifested as workers trying to connect to pypi.python.org to get pip 1.4.1 so they could downgrade the one already preinstalled by get-pip on the server image) | 20:28 |
| clarkb | james_li: our cloud providers that we run all of our tests on | 20:28 |
| fungi | james_li: currently primary rackspace and hpcloud | 20:28 |
| fungi | er, primarily | 20:28 |
| dhellmann | fungi: that's fine; is there another fix? | 20:28 |
| clarkb | we should ask dstufft | 20:29 |
| james_li | clarkb: fungi ic, thanks! | 20:29 |
| *** gokrokve has joined #openstack-infra | 20:29 | |
| dstufft | clarkb: what's up? | 20:30 |
| fungi | dhellmann: we could probably cache a copy of pip 1.4.1 on the images and then have devstack install that, but certainly trying to work out why six is breaking on upgrade with modern pip would be good (and so far i've only seen reports of it affecting trove's third-party ci) | 20:30 |
| clarkb | dhellmann: mgagne I really expect that something is pinning six though | 20:30 |
| clarkb | dstufft: we are using pip 1.5.6 in the devstack gate now and it breaks around six apparently | 20:30 |
| Shrews | sdague: WIP for now until the fix is merged: https://review.openstack.org/98201 | 20:30 |
| dstufft | can you define breaks around six | 20:30 |
| clarkb | dstufft: sorry its a very nebulous error because ENOLOGS | 20:30 |
| clarkb | dstufft: but SlickNik can probably help that | 20:30 |
| fungi | dstufft: it looks like https://rdjenkins.dyndns.org/job/Trove-Gate/3974/console | 20:31 |
| fungi | "ERROR: openstackclient.shell Exception raised: six>=1.6.0" | 20:31 |
| mgagne | dhellmann: ok, I was wrong, pip still requires setuptools | 20:31 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for heat assert isinstance bug 1326946 https://review.openstack.org/98245 | 20:31 |
| uvirtbot | Launchpad bug 1326946 in heat "assert isinstance(definition, rsrc_defn.ResourceDefinition) failing gate due to merge conflict" [Low,In progress] https://launchpad.net/bugs/1326946 | 20:31 |
| *** lcheng_ has joined #openstack-infra | 20:32 | |
| clarkb | * correction devstack gate is fine. Third parties having trouble | 20:32 |
| fungi | mgagne: it vendors in a copy on its own now though, as opposed to expecting it to be there or retrieving one from somewhere | 20:32 |
| dhellmann | dstufft: there may be some useful logs on https://bugs.launchpad.net/python-openstackclient/+bug/1326811 and https://bugs.launchpad.net/devstack/+bug/1326925 | 20:32 |
| uvirtbot | Launchpad bug 1326811 in trove "Client failing with six =>1.6 error" [Undecided,New] | 20:32 |
| clarkb | I love that 97757 has failed again | 20:32 |
| anteaya | clarkb: sorry am I missing context? | 20:32 |
| anteaya | I am switching windows a lot today | 20:32 |
| clarkb | anteaya: context for which subject? | 20:33 |
| anteaya | Third parties having trouble | 20:33 |
| clarkb | anteaya: trove third party tests are broken | 20:33 |
| anteaya | ah | 20:33 |
| anteaya | thanks | 20:33 |
| fungi | anteaya: trove's ci is experiencing some issues around python-openstackclient and six when installing with current versions of pip, but not when using pip 1.4.1 | 20:33 |
| anteaya | kk, thanks | 20:34 |
| *** maxbit_ has joined #openstack-infra | 20:34 | |
| *** julim has quit IRC | 20:34 | |
| *** HenryG has joined #openstack-infra | 20:34 | |
| anteaya | that knits together the bits nicely | 20:34 |
| clarkb | fungi: I think the hypothesis of dep resolution is a good one | 20:34 |
| * fungi tries to reproduce with a virtualenv | 20:34 | |
| *** gokrokve has quit IRC | 20:34 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Added query for ceilometer test_ordination_allocate bug 1323524 https://review.openstack.org/98146 | 20:35 |
| uvirtbot | Launchpad bug 1323524 in ceilometer "tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate failed for timeout" [Medium,Confirmed] https://launchpad.net/bugs/1323524 | 20:35 |
| dhellmann | mgagne: when I just ran devstack, with the change to use pip 1.5.6, I got six==1.6.1 | 20:35 |
| clarkb | fungi: we are still going to restart nodepool yes? do you want my test change to merge before we do that? It has no affect on running nodepool | 20:35 |
| mgagne | dhellmann: In bug 1326925, it is reported that manually installing setuptools 3.6 fixes the issue | 20:35 |
| fungi | clarkb: yeah, i think it's safe. i'm a little uneasy about restarting it while we're resource-starved again, but i suppose it'll recover soon enough | 20:35 |
| uvirtbot | Launchpad bug 1326925 in devstack "stack.sh fails when creating openstack admin project" [Undecided,Confirmed] https://launchpad.net/bugs/1326925 | 20:36 |
| *** starmer has joined #openstack-infra | 20:36 | |
| clarkb | fungi: hrm good point | 20:36 |
| godp1301 | dhellmann: also have six=1.6.1 using pip 1.4.1 | 20:36 |
| clarkb | the cloud have been hurting today | 20:36 |
| Ajaeger | I see now jobs failing with " sudo: /usr/sbin/apache2ctl: command not found" | 20:36 |
| *** gokrokve has joined #openstack-infra | 20:36 | |
| dstufft | dep resolution hasn't really changed | 20:36 |
| fungi | Ajaeger: that's an unrelated error which seems to crop up when devstack dies abruptly in other ways | 20:36 |
| dstufft | Issuing the command `sudo pip install setuptools==3.6` fixes the issue. | 20:37 |
| dstufft | I bet | 20:37 |
| fungi | Ajaeger: pretend you never saw that error message and go looking in the devstack log for the actual failure | 20:37 |
| godp1301 | dstufft is does i confirm | 20:37 |
| dstufft | clarkb: fungi I bet you're getting six from a wheel in pip 1.5 | 20:37 |
| dstufft | but from a sdist in 1.4 | 20:37 |
| Ajaeger | fungi: Argh ;( It hit one of your promoted patches - 97757 - ok, will scroll further up... | 20:37 |
| mat128 | “Issuing the command `sudo pip install setuptools==3.6` fixes the issue.” | 20:37 |
| *** Ryan_Lane1 has quit IRC | 20:37 | |
| *** otherwiseguy has quit IRC | 20:37 | |
| dstufft | and an old setuptools is being used | 20:38 |
| *** maxbit has quit IRC | 20:38 | |
| Ajaeger | fungi: please double check - there's nothing else going on in https://jenkins03.openstack.org/job/gate-tempest-dsvm-neutron-large-ops/16520/console | 20:38 |
| dstufft | old setuptools that doesn't recognize the dist-info directories | 20:38 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove some queries with no hits https://review.openstack.org/98224 | 20:38 |
| dstufft | which wheel uses | 20:38 |
| Ajaeger | What do I miss? | 20:38 |
| fungi | Ajaeger: yeah, that one was clobbered by keystone trying to listen on its iana-assigned port which happens to be within the linux default ephemeral ports range and had something already bound on it | 20:38 |
| fungi | Ajaeger: the nova revert i mean | 20:38 |
| clarkb | dstufft: I thought pip bundles setuptools though | 20:38 |
| Ajaeger | Ah. Hit this again here https://jenkins06.openstack.org/job/gate-requirements-integration-dsvm/155/console | 20:38 |
| clarkb | dstufft: is that setuptools not installed globally? | 20:39 |
| Ajaeger | fungi: yeah, one was the nova revert | 20:39 |
| *** Samolo has quit IRC | 20:39 | |
| dstufft | clarkb: pip bundles pkg_resources froms etuptools for our own internal uses, however that doesn't control the pkg_resources that is installed globally | 20:39 |
| clarkb | gotcha | 20:39 |
| dstufft | clarkb: get-pip.py will also install setuptools *if* setuptools isn't already installed | 20:39 |
| fungi | Ajaeger: on the nova revert, follow that link at the top of the console log to the location of the detailed logs on logs.o.o, then look in the devstack log and you'll see keystone failed to start | 20:40 |
| clarkb | dstufft: is there a way to force it to install setuptools regardless? | 20:40 |
| fungi | Ajaeger: then if you look in the screen-key log you'll see keystone complaining that its port is already in use | 20:40 |
| dstufft | clarkb: No, but you could just do pip install --upgrade pip in stack.sh? | 20:40 |
| clarkb | dstufft: if not I suppose we can just do pip install --upgrade setuptools | 20:40 |
| clarkb | dstufft: or that | 20:40 |
| dstufft | er | 20:40 |
| clarkb | dhellmann: mgagne ^ | 20:40 |
| *** starmer has quit IRC | 20:40 | |
| dstufft | pip install --upgrade setuptools I mean | 20:40 |
| dstufft | not pip | 20:40 |
| clarkb | ok | 20:40 |
| clarkb | dstufft: thanks | 20:40 |
| dhellmann | clarkb: I haven't been able to reproduce the issue | 20:40 |
| dhellmann | a simple devstack run with nothing special in localrc doesn't exhibit the problem, so I wonder if it depends on a particular service being enabled | 20:41 |
| *** lcheng_ has quit IRC | 20:41 | |
| mgagne | dhellmann: our use case includes ironic | 20:42 |
| dstufft | dhellmann: assuming you're talking about the six thing, it probably depends on what Os you're using | 20:42 |
| mat128 | dhellmann: https://gist.github.com/anonymous/40530a9b8d281738c4e7 | 20:42 |
| clarkb | so I think we can just have devstack do pip install --upgrade setuptools after installing pip | 20:42 |
| fungi | dhellmann: do we run openstackclient for anything in normal devstack/tempest jobs? because it's openstackclient.shell that i saw complaining in trove's logs | 20:42 |
| *** Ryan_Lane has joined #openstack-infra | 20:42 | |
| clarkb | in the gate that should hit our mirror | 20:42 |
| dstufft | clarkb: Looking at the changelog, you need at least setuptools 0.6.28 | 20:42 |
| dstufft | to get dist-info support | 20:43 |
| dhellmann | fungi: I think dtroyer has been updating devstack to use openstackclient | 20:43 |
| mat128 | fungi: creating projects, roles and stuff | 20:43 |
| Ajaeger | thanks, fungi! | 20:43 |
| fungi | dhellmann: mat128: okay, so that theory's out | 20:43 |
| dstufft | clarkb: precise has 0.6.24 | 20:43 |
| clarkb | mgagne: SlickNik can you guys test with an explicit pip install --upgrade setuptools after get pip is installed | 20:44 |
| fungi | the example where it's failing claims to be ubuntu, but no idea what release | 20:44 |
| dhellmann | mat128: thanks, what OS are you on? | 20:44 |
| mgagne | fungi: Ubuntu 12.04.3 is used | 20:44 |
| mat128 | Ubuntu 12.04.3 | 20:44 |
| fungi | oh, looks like ubuntu precise actually | 20:44 |
| mat128 | I was able to reproduce it on 12.04.4 | 20:45 |
| mat128 | using an Ubuntu-provided vagrant box | 20:45 |
| fungi | since its apt-get update is hitting a precise repository | 20:45 |
| mriedem | how is the gate looking now? better? | 20:45 |
| mat128 | stack@devstack:/usr/local/lib/python2.7/dist-packages/six-1.6.1.dist-info$ cat WHEEL | 20:46 |
| *** UtahDave has joined #openstack-infra | 20:46 | |
| mat128 | looks like six is a wheel | 20:46 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Add query for heat assert isinstance bug 1326946 https://review.openstack.org/98245 | 20:46 |
| uvirtbot | Launchpad bug 1326946 in heat "assert isinstance(definition, rsrc_defn.ResourceDefinition) failing gate due to merge conflict" [Low,In progress] https://launchpad.net/bugs/1326946 | 20:46 |
| clarkb | mriedem: you are funny | 20:46 |
| mriedem | just coming in late | 20:46 |
| fungi | oh! i wonder if we're shielded from this in the upstream gate by not mirroring wheels? | 20:46 |
| dhellmann | mat128: six 1.6.1 should be fine, though, right? | 20:46 |
| dhellmann | mat128: is there a second copy of six that is *not* a wheel? | 20:47 |
| fungi | nevermind... http://pypi.openstack.org/openstack/six/six-1.6.1-py2.py3-none-any.whl | 20:47 |
| *** gokrokve has quit IRC | 20:47 | |
| mat128 | there's a six.py file directly in there | 20:47 |
| *** marcoemorais1 has quit IRC | 20:47 | |
| clarkb | fungi: could be | 20:47 |
| dstufft | mat128: you have setuptools 0.6.24 don't you | 20:47 |
| *** yjiang5 has quit IRC | 20:47 | |
| clarkb | fungi: hrm we shouldn't have wheels there unless hrm | 20:47 |
| clarkb | fungi: is run-mirror smart enough to put non platform specific wheels in the regular mirror? | 20:48 |
| *** otherwiseguy has joined #openstack-infra | 20:48 | |
| *** marcoemorais has joined #openstack-infra | 20:48 | |
| dstufft | PyPI doens't allow you to upload linux wheels anyways | 20:48 |
| dstufft | that are platform specific to linux that is | 20:48 |
| fungi | clarkb: i think run-mirror generates wheels | 20:48 |
| clarkb | fungi: it does but it should only put them in the wheel mirror | 20:48 |
| clarkb | that wheel above shouldn't be there | 20:48 |
| *** yjiang5 has joined #openstack-infra | 20:49 | |
| mat128 | re-running stack with the old code created a six-1.6.1.egg-info | 20:49 |
| mat128 | I have to leave but slamont has the same setup, he will be answering questions | 20:49 |
| mat128 | he's right next to me :P | 20:49 |
| * dhellmann is having network trouble and can't reach his remote server | 20:50 | |
| *** andreykurilin_ has joined #openstack-infra | 20:50 | |
| *** starmer has joined #openstack-infra | 20:51 | |
| *** weshay has quit IRC | 20:51 | |
| slamont | dstufft, mat128 : 0.6.24-1ubuntu1 | 20:51 |
| dstufft | yea | 20:51 |
| dstufft | that's the problem then | 20:51 |
| *** lcheng_ has joined #openstack-infra | 20:52 | |
| dstufft | setuptools prior to 0.6.28 didn't understand .dist-info, so it doesn't see six as installed at all (becuase it's installed with a .dist-info directory) and the pkg_resources check is failing. | 20:52 |
| fungi | ahh, i *believe* we kill distro packages of pip, setuptools, et cetera with fire before running get-pip | 20:52 |
| fungi | on our nodepool images | 20:52 |
| dstufft | the reason the 1.4 -> 1.5 upgrade triggered this, is 1.5 installs wheels by default | 20:52 |
| fungi | so when devstack shows up at the party, old setuptools is already out the door | 20:52 |
| clarkb | fungi: we do | 20:53 |
| godp1301 | python-pip is uninstalled by install_pip: uninstall_package python-pip | 20:53 |
| godp1301 | nothing else | 20:53 |
| dstufft | you often times can't unintall python-setuptools becuase things depend on it | 20:54 |
| godp1301 | mgagne and I are trying the pip install —update setuptools fix now | 20:54 |
| fungi | godp1301: so it sounds like devstack's install_pip should also uninstall_package python-setuptools | 20:54 |
| dstufft | fungi: that might not work | 20:54 |
| dhellmann | fungi: are we seeing this problem in our gate, or just on developer systems? | 20:54 |
| dstufft | see: dependencies | 20:54 |
| fungi | dhellmann: we haven't seen it in our gate | 20:54 |
| fungi | dstufft: yep, good point | 20:54 |
| dhellmann | ok, that's good at least | 20:54 |
| *** Samolo has joined #openstack-infra | 20:54 | |
| dstufft | fungi: but luckily pip gives no fucks and will happily stomp all over the apt-get installed setuptools :V | 20:54 |
| dstufft | "luckily" | 20:54 |
| clarkb | fungi: it just needs ot upgrade setuptools | 20:55 |
| *** starmer has quit IRC | 20:55 | |
| dstufft | clarkb: fungi I'm going to poke some contacts in cannonical to see if maybe I can convince them to SRU setuptools to patch in dist-info | 20:56 |
| dstufft | patch in dist-info support* | 20:56 |
| fungi | heh... in devstack lib/infra there's a function unfubar_setuptools | 20:56 |
| fungi | dstufft: that sounds helpful | 20:57 |
| Alex_Gaynor | dstufft: do you recognize https://jenkins01.openstack.org/job/gate-python-swiftclient-python33/207/console ? | 20:57 |
| mgagne | fungi: looks to never be called =) | 20:58 |
| fungi | it looks like nothing ever calls it, yeah | 20:58 |
| fungi | http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/infra#n29 | 20:58 |
| godp1301 | fungi: you mean removing python-setuptools from /files/* ? If so, I'll have a look at what needs to be done | 20:58 |
| dstufft | Alex_Gaynor: nope | 20:58 |
| Alex_Gaynor | dstufft: oh boy | 20:58 |
| godp1301 | fungi: slightly similar to what if in /lib/infra | 20:59 |
| godp1301 | fungi: uninstall_package python-setuptools | 20:59 |
| godp1301 | pip_install -U setuptools | 20:59 |
| godp1301 | pip_install -U pip | 20:59 |
| clarkb | you won't need to upgrade pip as get pip should do that for you | 20:59 |
| dstufft | maybe get-pip.py should put a lower bounds on the setuptools it accepts | 21:00 |
| dstufft | hmm | 21:00 |
| *** resker has joined #openstack-infra | 21:00 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Create zuul template translation-jobs https://review.openstack.org/98252 | 21:00 |
| *** stevebaker has quit IRC | 21:00 | |
| *** stevebaker has joined #openstack-infra | 21:00 | |
| *** esker has quit IRC | 21:01 | |
| *** aysyd has quit IRC | 21:02 | |
| fungi | mgagne: looks like dtroyer removed the last call to it in https://review.openstack.org/39827 when he brought in get-pip | 21:02 |
| fungi | mgagne: prior to that it was being called from stack.sh | 21:02 |
| *** prad has quit IRC | 21:03 | |
| fungi | Alex_Gaynor: looks like get_script_header() returns a bytes type when run under 3.3 | 21:04 |
| Alex_Gaynor | fungi: why did it just start showing up though | 21:04 |
| dstufft | because computer | 21:04 |
| fungi | Alex_Gaynor: my next step would be to git blame around that function | 21:05 |
| fungi | git blamethrower | 21:05 |
| dstufft | have you tried turning it off and on again | 21:05 |
| *** starmer has joined #openstack-infra | 21:05 | |
| fungi | Alex_Gaynor: was gate-python-swiftclient-python33 nonvoting recently? | 21:06 |
| Alex_Gaynor | fungi: it's been voting for a week or two | 21:07 |
| *** rfolco has quit IRC | 21:07 | |
| fungi | Alex_Gaynor: in that case, i'd suggest you find the most recent successful run you can and diff the pip freeze from them | 21:07 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for oslo.messaging unit test race bug 1283926 https://review.openstack.org/98244 | 21:07 |
| uvirtbot | Launchpad bug 1283926 in oslo.messaging "test_rabbit.TestRacyWaitForReply failed with a not related change" [Low,Fix committed] https://launchpad.net/bugs/1283926 | 21:07 |
| *** smarcet has quit IRC | 21:08 | |
| fungi | Alex_Gaynor: when things spontaneously start to break, more often than not it's a new release of an outside dependency doing it | 21:08 |
| dstufft | aren't dependencies awesome | 21:08 |
| fungi | they are when they work and keep you from needing to write as much code | 21:09 |
| Alex_Gaynor | fungi: pytz and keystoneclient are the only things that upgraded | 21:09 |
| zaro | jhesketh: ping | 21:10 |
| *** starmer has quit IRC | 21:10 | |
| * dhellmann wonders if his chef-solo config is "fixing" this before he even runs devstack | 21:10 | |
| dtroyer | tools/install_pip.sh used to also manually install a reasonable setuptools, maybe it's time to re-introduce that? | 21:11 |
| fungi | Alex_Gaynor: oh... this is breaking down inside easy_install.py... i on;y just spotted that | 21:12 |
| dstufft | here's a question | 21:12 |
| *** e0ne has quit IRC | 21:12 | |
| dstufft | how come devstack doesn't just use a virtualenv | 21:12 |
| Alex_Gaynor | fungi: yeah, easy_install.py is a real bad place :/ | 21:12 |
| Alex_Gaynor | I've reproduced this locally, I wonder if it effects other packages | 21:13 |
| Alex_Gaynor | l;et me get a bug filed | 21:13 |
| dstufft | woe to thee who gazes upon easy_install.py | 21:13 |
| fungi | Alex_Gaynor: so python-swiftclient is uninstallable in a python 3.3 venv i guess, looking closer at the traceback | 21:13 |
| *** andreykurilin_ has quit IRC | 21:14 | |
| *** pdmars has quit IRC | 21:14 | |
| *** aburaschi has quit IRC | 21:15 | |
| fungi | Alex_Gaynor: actually, installing tip of master python-swiftclient in a python3.3 venv seems to work for me... checking what pip freeze claims there | 21:15 |
| *** Ryan_Lane has quit IRC | 21:16 | |
| fungi | Alex_Gaynor: what are your minimal steps to reproduce? | 21:16 |
| *** slamont has quit IRC | 21:16 | |
| fungi | looks like i'm getting python-swiftclient==2.1.0.9.g3d0de79 requests==2.3.0 simplejson==3.5.2 six==1.6.1 | 21:16 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for heat assert isinstance bug 1326946 https://review.openstack.org/98245 | 21:17 |
| Alex_Gaynor | fungi: rm -rf .tox/py33; tox -e py33 | 21:17 |
| uvirtbot | Launchpad bug 1326946 in heat "assert isinstance(definition, rsrc_defn.ResourceDefinition) failing gate due to merge conflict" [Low,In progress] https://launchpad.net/bugs/1326946 | 21:17 |
| fungi | k | 21:17 |
| *** sarob has quit IRC | 21:17 | |
| *** andreaf has quit IRC | 21:17 | |
| zaro | jhesketh: got myself onto a ubuntu dev env. ran turbo-hipster tests from master (tox -epy27), got the following error: http://paste.openstack.org/show/83003/ | 21:17 |
| *** devananda has joined #openstack-infra | 21:17 | |
| *** sarob has joined #openstack-infra | 21:18 | |
| *** Ryan_Lane has joined #openstack-infra | 21:18 | |
| fungi | Alex_Gaynor: yep, reproduced on debian/testing... seeing if i can whittle down the reqs | 21:19 |
| Alex_Gaynor | fungi: https://bugs.launchpad.net/python-swiftclient/+bug/1326972 is an issue I filed for this | 21:20 |
| uvirtbot | Launchpad bug 1326972 in python-swiftclient "swiftclient fails to install under tox with Python3" [Undecided,New] | 21:20 |
| *** andreykurilin_ has joined #openstack-infra | 21:20 | |
| *** starmer has joined #openstack-infra | 21:20 | |
| *** nati_ueno has joined #openstack-infra | 21:22 | |
| *** sarob has quit IRC | 21:22 | |
| *** Ajaeger has quit IRC | 21:23 | |
| fungi | Alex_Gaynor: unable to reproduce without tox in a python3.3 virtualenv using './foo/bin/pip install -U -r requirements.txt -r test-requirements.txt .' | 21:24 |
| fungi | so looks fairly tox-specific. hopefully i can narrow it down further | 21:24 |
| *** starmer has quit IRC | 21:25 | |
| mgagne | dhellmann: looks like updating setuptools after installing pip fixes the issue | 21:25 |
| *** apevec has joined #openstack-infra | 21:25 | |
| Alex_Gaynor | fungi: it's weird though, my tox version didn't change, this si going to be subtle | 21:25 |
| dhellmann | mgagne: yeah, and my chef setup installs setuptools and pip for me, so I never saw this at all | 21:26 |
| fungi | mgagne: dhellmann: so probably just adding a pip install -U setuptools immediately after running get-pip in devstack is the best way forward | 21:26 |
| apevec | ttx, fungi - looks like I lost priv to push tags: ! [remote rejected] 2014.1.1 -> 2014.1.1 (prohibited by Gerrit) | 21:26 |
| apevec | error: failed to push some refs to 'ssh://apevec@review.openstack.org:29418/openstack/nova.git' | 21:26 |
| fungi | apevec: checking into it now | 21:26 |
| mgagne | fungi, dhellmann: what I tested with: https://github.com/mgagne/devstack/commit/ff40b037faf6b960bd36150f195feec00345a6b9 | 21:26 |
| dhellmann | fungi: that seems reasonable | 21:27 |
| apevec | fungi, or could be b/c I changed my key to alan.pevec@redhat.com ? | 21:27 |
| apevec | I had apevec@redhat.com before | 21:27 |
| fungi | apevec: you've got membership in a group with access to push signed tags to refs/* for all projects | 21:27 |
| *** slamont has joined #openstack-infra | 21:28 | |
| apevec | fungi, is that checking "tagger" field? | 21:29 |
| fungi | apevec: is your alan.pevec@redhat.com address listed as an additional address in your gerrit settings? | 21:29 |
| apevec | it is not | 21:29 |
| * apevec fixes | 21:29 | |
| *** mfer has quit IRC | 21:29 | |
| fungi | yeah, that might be needed first | 21:29 |
| fungi | it'll want to send you a test e-mail message to activate that additional address too | 21:29 |
| *** andreykurilin_ has quit IRC | 21:30 | |
| dtroyer | mgagne: can you try that as 'pip_install —upgrade setuptools' | 21:30 |
| dtroyer | that will let it work from behind a proxy and other nice things | 21:30 |
| fungi | Alex_Gaynor: aha! it's setup.py develop which triggers this, not installing | 21:30 |
| mgagne | dtroyer: is pip_install -U setuptools ok? | 21:31 |
| SlickNik | clarkb: Sorry was away tending to something else. Catching up on scrollback now… | 21:31 |
| mgagne | dtroyer: looks to be the syntax used elsewhere | 21:31 |
| dtroyer | sure, I'm just used to using the long opts | 21:31 |
| *** Guest43801 is now known as Woodie | 21:32 | |
| apevec | fungi, yep, that was it, thanks! | 21:32 |
| fungi | apevec: you're welcome | 21:32 |
| *** prad has joined #openstack-infra | 21:33 | |
| openstackgerrit | Ferrante proposed a change to openstack-infra/jenkins-job-builder: Adds support for system-groovy https://review.openstack.org/98257 | 21:33 |
| *** lbragstad has quit IRC | 21:34 | |
| mgagne | dtroyer: alright, gimme a couple of minutes to test it | 21:34 |
| fungi | Alex_Gaynor: further whittling, if you only install requirements.txt into the virtualenv, then setup.py develop works fine. the breakage is when you also have test-requirements.txt installed, so it's some interaction there | 21:34 |
| fungi | i'll try to bisect the culprit out of test-requirements.txt | 21:34 |
| *** mrda-away is now known as mrda | 21:35 | |
| *** arnaud__ has quit IRC | 21:35 | |
| *** starmer has joined #openstack-infra | 21:36 | |
| *** nati_uen_ has joined #openstack-infra | 21:39 | |
| *** nati_ueno has quit IRC | 21:39 | |
| Alex_Gaynor | fungi: bizzare | 21:39 |
| jogo | fungi clarkb: so AFAIK zuul know dynamically calculates the integrated gate? can we get that data into ER? | 21:39 |
| jogo | ES* | 21:40 |
| clarkb | jogo: no | 21:40 |
| jogo | clarkb: no to which part? | 21:40 |
| clarkb | both | 21:40 |
| *** starmer has quit IRC | 21:40 | |
| *** sarob has joined #openstack-infra | 21:40 | |
| jogo | http://status.openstack.org/zuul/ has the word integrated | 21:40 |
| clarkb | its not dynamic :) | 21:40 |
| *** masayukig has quit IRC | 21:40 | |
| clarkb | its a zuul layout entry | 21:40 |
| jogo | oh | 21:40 |
| clarkb | and getting that to ER is tricky because ER only knows what jenkins tells it | 21:41 |
| clarkb | and jenkins has no concept of gates and stuff | 21:41 |
| clarkb | we could in theory pass through that as a variable but it wouldn't be used for anything else | 21:41 |
| jogo | zuul doesn't talk to ES directly? | 21:41 |
| *** Ryan_Lane has quit IRC | 21:41 | |
| clarkb | jogo: no | 21:41 |
| *** Ryan_Lane has joined #openstack-infra | 21:41 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for setuptools py26 unicode bug 1326878 https://review.openstack.org/98187 | 21:41 |
| jogo | clarkb: so the issue I want to address here is: status.openstack.org/elastic-recheck/data/uncategorized.html | 21:42 |
| uvirtbot | Launchpad bug 1326878 in pbr "Race "TypeError: expecting a bytes object, got unicode" on py26 setup (dup-of: 1325514)" [Undecided,New] https://launchpad.net/bugs/1326878 | 21:42 |
| uvirtbot | Launchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,Fix released] https://launchpad.net/bugs/1325514 | 21:42 |
| fungi | Alex_Gaynor: hacking! | 21:42 |
| jogo | has no concept of what is integrated and what isn't | 21:42 |
| SlickNik | clarkb / mgagne: I can confirm that an explicit upgrade of setuptools fixes the issue. | 21:42 |
| clarkb | jogo: it could ask zuul maybe | 21:42 |
| jogo | clarkb: ohh, hmm that may work | 21:43 |
| clarkb | and cache that for a day or so before asking again | 21:43 |
| jogo | give it a job name and get a yes or no back? | 21:43 |
| fungi | Alex_Gaynor: working commands... ./foo/bin/pip install -U -r requirements.txt ; . foo/bin/activate ; python setup.py develop | 21:43 |
| mgagne | SlickNik: yep, testing with suggested approach by dtroyer | 21:43 |
| jogo | well the uncategorized page is just a cron job | 21:43 |
| jogo | as you know | 21:43 |
| Alex_Gaynor | fungi: yup, new hacking release yesterday | 21:43 |
| jogo | Alex_Gaynor: have another version to release today | 21:43 |
| clarkb | jogo: I was thinking grabbing the list of projects for the integrated gate via zuul status.json | 21:43 |
| jogo | waiting for a review | 21:43 |
| fungi | Alex_Gaynor: but this breaks as seen in the job... ./foo/bin/pip install -U -r requirements.txt 'hacking>=0.8.0,<0.9' ; . foo/bin/activate ; python setup.py develop | 21:44 |
| *** gokrokve has joined #openstack-infra | 21:44 | |
| clarkb | jogo: all of the projects should be listed in the json file | 21:44 |
| mtreinish | jogo: why not just pull infra/config and parse the zuul yaml... | 21:44 |
| jogo | Alex_Gaynor: https://review.openstack.org/#/c/97988/ | 21:44 |
| *** dkliban is now known as dkliban_afk | 21:44 | |
| mtreinish | since it's infrequent | 21:44 |
| dhellmann | woo, I finally reproduced the six failure | 21:44 |
| Alex_Gaynor | jogo: have you been folloing this disussion between fungi and myself? | 21:44 |
| *** denis_makogon has quit IRC | 21:44 | |
| openstackgerrit | Adrian Otto proposed a change to openstack-infra/config: Add a solum-spec repo to stackforge https://review.openstack.org/98237 | 21:44 |
| jogo | Alex_Gaynor: no I just ssaw the words hacking and I cut a new one yesterday | 21:44 |
| fungi | jogo: installing hacking into a py33 venv causes subsequent setup.py develop to fail for swiftclient | 21:45 |
| Alex_Gaynor | jogo: might want to read back, it looks like the newest hacking on py3k causes some weird breakages | 21:45 |
| jogo | mtreinish: hmm that may work want to work on that ? | 21:45 |
| mgagne | dhellmann: you cloned devstack from master and stack.sh ? nothing else? | 21:45 |
| fungi | jogo: my gut says it's one of the newer deps in the latest hacking release | 21:45 |
| slamont | dhellmann: I put a pip freeze output in LP 1326811 | 21:45 |
| uvirtbot | Launchpad bug 1326811 in trove "Client failing with six =>1.6 error" [Undecided,New] https://launchpad.net/bugs/1326811 | 21:45 |
| *** sarob has quit IRC | 21:45 | |
| fungi | jogo: but i haven't started bisecting those yet to figure out which | 21:46 |
| mtreinish | jogo: actually I think clarkb's idea makes more sense. But I can do the yaml pull idea too (it shouldn't be hard) | 21:46 |
| *** maxbit has joined #openstack-infra | 21:46 | |
| *** masayukig has joined #openstack-infra | 21:46 | |
| jogo | mtreinish: cool I nominate you to make it happen ;) | 21:46 |
| jogo | fungi: let me read backlog. not sure what I broke | 21:46 |
| jogo | Alex_Gaynor: so I am a little lost. nothing should be using hacking 0.9.0 yet | 21:48 |
| *** maxbit_ has quit IRC | 21:48 | |
| fungi | jogo: i suspect it may be a new release of something hacking depends on | 21:48 |
| fungi | jogo: i'm just working my way down the dependency chain | 21:48 |
| jogo | fungi: we pin all of those | 21:48 |
| fungi | hacking was the first-order dep introducing this behavior | 21:49 |
| jogo | so all uses of hackign should pin to <0.9.0 | 21:49 |
| jogo | as 0.9.0 adds new rules | 21:49 |
| jogo | fungi: have a link to the failed job? | 21:49 |
| jogo | fungi: also we gate py33 on new hacking | 21:49 |
| fungi | jogo: https://jenkins01.openstack.org/job/gate-python-swiftclient-python33/207/console (though i doubt it will help) | 21:50 |
| *** starmer has joined #openstack-infra | 21:50 | |
| *** dims__ has joined #openstack-infra | 21:51 | |
| *** Samolo has quit IRC | 21:51 | |
| SlickNik | mgagne: Thanks! Let me know if there's something I can help with (testing or otherwise) to speed things up! | 21:51 |
| *** g_306396236 has joined #openstack-infra | 21:51 | |
| *** marcoemorais has quit IRC | 21:52 | |
| jogo | fungi: so why do you think its hacking? | 21:52 |
| fungi | jogo: Alex_Gaynor: yeah, i've ruled out hacking by only installing its deps, and still reproduce | 21:52 |
| *** marcoemorais has joined #openstack-infra | 21:52 | |
| jogo | fungi: phew | 21:52 |
| mgagne | SlickNik, dhellmann: I don't know what to do from now. I know how to fix it but don't know the technical reason as to WHY updating setuptools fixes the issue. | 21:52 |
| Alex_Gaynor | jogo: sorry for the noice | 21:52 |
| Alex_Gaynor | noise* | 21:52 |
| jogo | Alex_Gaynor: haha no worries glad I didn't break things (this time) :) | 21:52 |
| fungi | jogo: i don't think it's hacking, but i bisected test-requires.txt down to only the hacking entry and could still reproduce. now i'm moving down into hacking's deps to see which is responsible | 21:53 |
| dhellmann | slamont: thanks | 21:53 |
| dhellmann | mgagne: yeah, I turned off the bits of my chef-solo setup that were pre-installing setuptools and pip, which hid the problem | 21:53 |
| SlickNik | mgagne: I believe the reason is that six is installed using wheels, and the older setuptools doesn't understand the .dist-info format, so it doesn't think six is installed. | 21:53 |
| *** dims_ has quit IRC | 21:53 | |
| clarkb | SlickNik: according to dstufft that is correct | 21:54 |
| jogo | fungi: I am rerunning a check on a hacking patch | 21:54 |
| jogo | to see what happens | 21:54 |
| *** g_306396236 has quit IRC | 21:54 | |
| clarkb | in other news I have dib running on precise to build a nodepool dib image | 21:54 |
| clarkb | testing mordred's dib stuff | 21:55 |
| clarkb | if I get this image built next step is trusty | 21:55 |
| *** starmer has quit IRC | 21:55 | |
| mgagne | dhellmann: still have your "broken" devstack around? | 21:55 |
| dhellmann | mgagne: yeah | 21:55 |
| mgagne | dhellmann: what's the version of setuptools installed? | 21:55 |
| dhellmann | mgagne: oh, wait, no, I fixed it :-/ | 21:56 |
| *** g_306396236 has joined #openstack-infra | 21:56 | |
| jogo | fungi: so the latest hacking worked with py33 | 21:56 |
| *** g_306396236 has quit IRC | 21:56 | |
| dstufft | mgagne: someone with a broken devstack said they had setuptools 0.6.24 | 21:56 |
| dhellmann | when I updated setuptools it was 0.6.something | 21:56 |
| dstufft | you need 0.6.28 | 21:56 |
| dhellmann | yeah, a dev version of 0.6.24 | 21:56 |
| dstufft | for wheels to not be broken | 21:56 |
| SlickNik | mgagne / dhellmann: I have results of a pip freeze from a borked instance | 21:56 |
| dhellmann | now I have 4.0.1 | 21:56 |
| jogo | FWIW we didn't gate hacking 0.8.1 on py33 | 21:56 |
| jogo | Alex_Gaynor: ^ | 21:56 |
| dhellmann | is that right? | 21:56 |
| SlickNik | I can double check | 21:56 |
| dstufft | dhellmann: that's the latest version yea | 21:57 |
| dhellmann | how did we get from 0.6 to 4.1? | 21:57 |
| dhellmann | wow, ok, I had no idea | 21:57 |
| mgagne | dhellmann: 0.6.24 <<<<<< 4.0.1 | 21:57 |
| dstufft | setuptools is doing the semver thing | 21:57 |
| dhellmann | makes sense | 21:57 |
| dstufft | so any little backwards cinompat gets a mjor bump | 21:57 |
| fungi | Alex_Gaynor: jogo: narrowed it down to... ./foo/bin/pip install -U -r requirements.txt 'flake8==2.0' ; . foo/bin/activate ; python setup.py develop | 21:57 |
| mgagne | 0.6.24 comes from the system (Ubuntu) | 21:57 |
| Alex_Gaynor | dstufft: s/semver/chrome/firefox/ | 21:57 |
| dstufft | i'm not entirely sure it's a good idea, because you can't really reasonably pin setuptools | 21:57 |
| *** jerryz has quit IRC | 21:57 | |
| dhellmann | ok, I'm not sure I have anything else useful to add to the setuptools conversation, so I'm going to see about delicious dinner. | 21:57 |
| Alex_Gaynor | fungi: no new flake8 realses since last october though? | 21:57 |
| fungi | Alex_Gaynor: yeah, which is hella strange | 21:58 |
| jogo | fungi: ahh that makes sense, latest hacking uses a newer one | 21:58 |
| jogo | Alex_Gaynor: we pin that | 21:58 |
| jogo | so things don't berak | 21:58 |
| jogo | break | 21:58 |
| jogo | heh | 21:58 |
| fungi | Alex_Gaynor: so installing requirements.txt and flake8 into the virtualenv and then trying to setup.py develop goes sideways | 21:58 |
| jogo | fungi: what happens if you bump hackign to 0.9.0? | 21:59 |
| jogo | does that work? | 21:59 |
| fungi | trying | 21:59 |
| *** g_306396236 has joined #openstack-infra | 21:59 | |
| *** g_306396236 has quit IRC | 21:59 | |
| fungi | jogo: Alex_Gaynor: so the issue is exhibited with flake8=2.0 installed but not with flake8==2.1.0 | 22:00 |
| jogo | fungi: thats what I expect as hacking 0.9 works | 22:02 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add puppet-openstack project puppet-openstack_extras https://review.openstack.org/97647 | 22:02 |
| *** markwash has joined #openstack-infra | 22:02 | |
| *** stevebaker has quit IRC | 22:03 | |
| *** stevebaker has joined #openstack-infra | 22:03 | |
| fungi | jogo: Alex_Gaynor: what's strange is that the only difference in pip freeze output between them is flake8==2.0 vs flake8==2.1.0 | 22:03 |
| *** resker has quit IRC | 22:04 | |
| jogo | hmm is this a newer pip or something? | 22:04 |
| fungi | smoking gun! | 22:05 |
| mgagne | working on patch | 22:05 |
| fungi | pip freeze is incomplete | 22:05 |
| *** starmer has joined #openstack-infra | 22:05 | |
| *** devananda has quit IRC | 22:05 | |
| fungi | pip list exposes a difference in setuptools versions | 22:05 |
| *** sarob has joined #openstack-infra | 22:06 | |
| fungi | working venv has flake8 (2.1.0) and setuptools (3.4.4) | 22:06 |
| *** devananda has joined #openstack-infra | 22:06 | |
| fungi | broken venv has flake8 (2.0) and setuptools (4.0.1) | 22:06 |
| fungi | jogo: Alex_Gaynor: ^ | 22:06 |
| *** markwash_ has joined #openstack-infra | 22:06 | |
| *** signed8bit has quit IRC | 22:06 | |
| *** markwash has quit IRC | 22:06 | |
| *** markwash_ is now known as markwash | 22:06 | |
| jogo | fungi: odd | 22:07 |
| *** devananda has quit IRC | 22:07 | |
| SlickNik | mgagne: Thanks! | 22:07 |
| openstackgerrit | Brant Knudson proposed a change to openstack/requirements: Add a tox env to check requirements overlap https://review.openstack.org/98262 | 22:08 |
| mestery | infra folks, I replied to the gate failures thread with what I hope is a workable plan around the "ssh timeouts" issues with neutron: http://lists.openstack.org/pipermail/openstack-dev/2014-June/036865.html | 22:08 |
| mestery | It involves a revert of a patch from armax which we think is making this worse, which will help lessen the load while we continue working the root cause. | 22:08 |
| jogo | mestery: woot | 22:09 |
| fungi | Alex_Gaynor: so the end of this story (and the beginning of the next) is "setuptools 4.0.1 release broke setup.py develop for python-swiftclient under python 3.3" | 22:09 |
| *** slamont has quit IRC | 22:09 | |
| mestery | jogo: Hopefully this can relieve pressure while we continue root causing. | 22:09 |
| mestery | jogo: Here is the revert: https://review.openstack.org/#/c/97245/ | 22:09 |
| anteaya | mestery: I'm for reverting what caused the spike | 22:09 |
| anteaya | mestery: what more do you need to keep getting to the root issue? | 22:10 |
| jogo | sdague: ^ | 22:10 |
| *** starmer has quit IRC | 22:10 | |
| mestery | anteaya: Well, we think it made it worse by uncovering something, but covering it up for now will move things back to where they were. :) | 22:10 |
| Alex_Gaynor | fungi: wahoo, can I convince you to file a bug with setuptools? | 22:10 |
| mestery | anteaya: salv-orlando has a patch to dump console logs into the tempest logs, that would help a bit I think. | 22:10 |
| *** sarob has quit IRC | 22:11 | |
| mestery | anteaya: https://review.openstack.org/#/c/97269/, but I think mtreinish indicated it was superseded by a few patches from mriedem. | 22:11 |
| anteaya | mestery: awesome | 22:11 |
| mestery | anteaya: See the comments at the bottom. | 22:11 |
| SlickNik | mestery: Nice! :) | 22:11 |
| *** MarkAtwood has quit IRC | 22:12 | |
| *** gondoi is now known as zz_gondoi | 22:12 | |
| anteaya | mestery: can you scare up some more neutron reviews for mriedem's patches please? | 22:12 |
| jogo | mtreinish: ^ thoughts on the revert | 22:12 |
| fungi | Alex_Gaynor: you bet, but first i need to cook dinner | 22:12 |
| anteaya | mestery: another 2 neutron dev reviews would be great | 22:13 |
| mgagne | SlickNik, dhellmann, clarkb: https://review.openstack.org/#/c/98263/ | 22:13 |
| Alex_Gaynor | fungi: thanks /cc dstufft | 22:13 |
| mestery | anteaya: I will do it. | 22:13 |
| *** signed8bit has joined #openstack-infra | 22:13 | |
| anteaya | mestery: thanks | 22:13 |
| mestery | anteaya: Thanks! | 22:13 |
| *** godp1301 has quit IRC | 22:13 | |
| jogo | mestery: can you update teh commit msg on https://review.openstack.org/#/c/97245/ | 22:13 |
| jogo | make it a little more verbose | 22:13 |
| mestery | jogo: Looking ... | 22:14 |
| mestery | jogo: Yes, let me do that now quickly. | 22:14 |
| jogo | mestery: thanks. | 22:14 |
| mtreinish | mestery: I'm fine with reverting it if the commit msg explains why | 22:15 |
| mestery | mtreinish: Cool! Working on that now. | 22:15 |
| jogo | mestery: faster ;) | 22:15 |
| mtreinish | but will you be able to reproduce it outside of the gate? | 22:15 |
| *** sarob has joined #openstack-infra | 22:15 | |
| mestery | jogo: :P | 22:15 |
| mestery | mtreinish: We haven't had much luck, we plan to resubmit the patch which broke it and debug it that way. | 22:15 |
| mgagne | I have to go now, I'll address comments on my change (if any) later tonight | 22:15 |
| *** bknudson has quit IRC | 22:16 | |
| mtreinish | mestery: ok makes sense | 22:16 |
| *** patelna has quit IRC | 22:16 | |
| *** jgrimm has quit IRC | 22:16 | |
| SlickNik | mgagne: Can I co-author the patchset with you? I can take over any comments. | 22:17 |
| anteaya | so yeah, I think mriedem's logging changes look a bit more specific and I'm for reverting the thing that created the spike | 22:17 |
| SlickNik | mgagne: Trying to get the fix out as soon as I can, so that the trove tests can run green again. :) | 22:17 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for setuptools py26 unicode bug 1326878 https://review.openstack.org/98187 | 22:18 |
| uvirtbot | Launchpad bug 1326878 in pbr "Race "TypeError: expecting a bytes object, got unicode" on py26 setup (dup-of: 1325514)" [Undecided,New] https://launchpad.net/bugs/1326878 | 22:18 |
| mestery | jogo: resubmitted | 22:18 |
| uvirtbot | Launchpad bug 1325514 in openstack-ci "'unicode' does not have the buffer interface" [Undecided,Fix released] https://launchpad.net/bugs/1325514 | 22:18 |
| mestery | jogo mtreinish: I have to take off for a bit now, I'll be back later tonight though, PM if you need anthing else for when I return. | 22:19 |
| fungi | Alex_Gaynor: btw i narrowed it down to being introduced between setuptools 3.8 and 4.0 and have added a simpler reproduction in a comment to bug 1326972 | 22:19 |
| uvirtbot | Launchpad bug 1326972 in python-swiftclient "swiftclient fails to install under tox with Python3" [Undecided,New] https://launchpad.net/bugs/1326972 | 22:19 |
| * mestery goes afk. | 22:19 | |
| mtreinish | mestery, jogo: ok +A on the revert | 22:20 |
| *** starmer has joined #openstack-infra | 22:21 | |
| jogo | mestery: nice commit msg | 22:21 |
| *** Woodie is now known as W00die | 22:21 | |
| jogo | fungi: clarkb ^ can we get that patch bumped? | 22:21 |
| jogo | hmm I think fungi is eating | 22:22 |
| clarkb | jogo: which one? | 22:22 |
| jogo | https://review.openstack.org/#/c/97245/ | 22:22 |
| clarkb | jogo: once it makes it into the gate sure | 22:22 |
| *** marcoemorais has quit IRC | 22:22 | |
| anteaya | jenkins is unhappy on one of mriedem's add logging patches: https://review.openstack.org/#/c/95831/4 | 22:22 |
| jogo | clarkb: thanks WFM | 22:22 |
| *** marcoemorais has joined #openstack-infra | 22:23 | |
| *** starmer has quit IRC | 22:25 | |
| anteaya | ERROR: the main setup script run by this job failed - exit code: 1 | 22:28 |
| clarkb | well that will ruin your day | 22:28 |
| anteaya | but devstacklog finished with an ip | 22:28 |
| *** stevebaker has quit IRC | 22:28 | |
| *** stevebaker has joined #openstack-infra | 22:28 | |
| anteaya | my day? | 22:28 |
| clarkb | whoever got the error :) | 22:28 |
| *** maxbit_ has joined #openstack-infra | 22:28 | |
| clarkb | looks like the tempest revert failed | 22:29 |
| anteaya | the failure was here: https://review.openstack.org/#/c/95831/4 | 22:29 |
| anteaya | the one I'm looking at | 22:29 |
| *** maxbit has quit IRC | 22:30 | |
| anteaya | blast it did so (the tempest revert failed) | 22:30 |
| jogo | clarkb: 2014-06-05 22:27:54.866 | 2014-06-05 22:27:54.848 | python: can't open file '/opt/stack/old/devstack/files/get-pip.py': [Errno 2] No such file or directory | 22:30 |
| clarkb | jogo: it will be in new/ | 22:31 |
| clarkb | jogo: is that a grenade job? | 22:31 |
| jogo | clarkb: yes | 22:31 |
| jogo | clarkb: https://jenkins03.openstack.org/job/check-grenade-dsvm-icehouse/386/console | 22:31 |
| clarkb | jogo: did grenade fail because of that or continue and download get pip itself? | 22:31 |
| clarkb | because its supposed to fall back on the internet | 22:31 |
| jogo | clarkb: hmm it failed hard | 22:32 |
| jogo | see console | 22:32 |
| clarkb | so probably a bug in grenade and d-g | 22:33 |
| jogo | clarkb: whee | 22:33 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1326754 https://review.openstack.org/98085 | 22:34 |
| uvirtbot | Launchpad bug 1326754 in tripleo "Ubuntu CI failures : Unable to locate package ubuntu-cloud-keyring" [High,Triaged] https://launchpad.net/bugs/1326754 | 22:34 |
| jogo | didn't we just land a pip change to stable branches of devstack? | 22:34 |
| jogo | clarkb: or are those still in flight | 22:34 |
| clarkb | jogo: maybe I haven't been following | 22:34 |
| *** slamont has joined #openstack-infra | 22:34 | |
| clarkb | we landed the master one | 22:34 |
| jogo | clarkb: https://review.openstack.org/#/c/98211/ | 22:34 |
| sdague | jogo: looks solid | 22:34 |
| jogo | https://review.openstack.org/#/c/98210/ | 22:35 |
| clarkb | so the problem is the file is curled without being written to disk | 22:35 |
| clarkb | if you look at he log the zip file is just blasted out to stdout | 22:35 |
| *** starmer has joined #openstack-infra | 22:36 | |
| clarkb | so that is grenade/devstack bug | 22:36 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Modularized StoryBoard Module https://review.openstack.org/98007 | 22:36 |
| clarkb | then we have to figure out why the file was not copied into the old/ dir | 22:36 |
| clarkb | and I know why | 22:36 |
| zaro | clarkb: do you happen to know how to debug a jjb test case? | 22:36 |
| anteaya | I'm going to get something to eat and will continue groking logs when I return | 22:36 |
| clarkb | zaro: what do you mean by 'debug'. Use the debugger? | 22:36 |
| sdague | clarkb: what's up with the get-pip thing now? | 22:37 |
| clarkb | sdague: jogo d-g does a mv from the cache-files location instead of a cp (because we have disk constraints) and it copies new/ first so new/ has all the cached files but old/ doesn't | 22:37 |
| zaro | clarkb: i'm trying to run a jjb test in eclipse. i think it's using nose and it's not working. | 22:37 |
| clarkb | sdague: two things. d-g isn't putting the cached files in the old/ dir for grenade for the reason above ^ | 22:37 |
| sdague | clarkb: ah... | 22:37 |
| clarkb | sdague: second thing https://jenkins03.openstack.org/job/check-grenade-dsvm-icehouse/386/consoleFull the fallback of grabbing the file directly fails because the file is downloaded with curl and spit to stdout instead of disk | 22:37 |
| sdague | clarkb: hmmm | 22:38 |
| clarkb | I can work a fix for d-g | 22:38 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1326754 https://review.openstack.org/98085 | 22:38 |
| uvirtbot | Launchpad bug 1326754 in tripleo "Ubuntu CI failures : Unable to locate package ubuntu-cloud-keyring" [High,Triaged] https://launchpad.net/bugs/1326754 | 22:38 |
| *** apevec has quit IRC | 22:39 | |
| sdague | clarkb: oh, yeh, havana | 22:40 |
| *** starmer has quit IRC | 22:40 | |
| sdague | yep, havana is completely wrong | 22:40 |
| sdague | ... so that's why I merge conflicted | 22:40 |
| fungi | clarkb: sdague: jogo: when did this first manifest (why didn't we see it on the devstack change which started relying on get-pip)? | 22:40 |
| fungi | oh, only havana | 22:40 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add a real check for the integrated gate jobs https://review.openstack.org/98264 | 22:40 |
| sdague | fungi: yeh, it's the icehouse grenade job | 22:41 |
| fungi | right | 22:41 |
| *** eharney has quit IRC | 22:41 | |
| sdague | so old is havana | 22:41 |
| fungi | argh | 22:41 |
| jogo | sdague: heh | 22:41 |
| fungi | dinnering, then will pitch in if it's still full of fail | 22:41 |
| *** thedodd has quit IRC | 22:41 | |
| sdague | yep, my havana change is still wrong | 22:41 |
| sdague | ok, let's me get to fixing | 22:42 |
| sdague | you know, I have another cleanup I'm going to merge with this | 22:42 |
| *** doug-fish has left #openstack-infra | 22:43 | |
| *** prad has quit IRC | 22:43 | |
| Alex_Gaynor | fungi: if you haven't already, I'll file a bug against setuptools with your reproduction steps? | 22:44 |
| *** zzelle has quit IRC | 22:44 | |
| fungi | Alex_Gaynor: feel free! i was going to look first to see whether there was already one open for this | 22:44 |
| *** devananda has joined #openstack-infra | 22:45 | |
| *** devananda has quit IRC | 22:45 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Put cached files in old/ dirs too https://review.openstack.org/98265 | 22:45 |
| clarkb | jogo: sdague ^ | 22:45 |
| *** james_li has quit IRC | 22:46 | |
| *** lcostantino has quit IRC | 22:46 | |
| *** jhesketh has quit IRC | 22:47 | |
| fungi | Alex_Gaynor: looking over their tracker, nothing jumps out at me | 22:47 |
| Alex_Gaynor | fungi: filed https://bitbucket.org/pypa/setuptools/issue/218/setuppy-develop-fails-with-some-packages | 22:47 |
| *** devananda has joined #openstack-infra | 22:48 | |
| fungi | Alex_Gaynor: coolio | 22:48 |
| *** starmer has joined #openstack-infra | 22:50 | |
| *** dangers is now known as dangers_away | 22:52 | |
| sdague | clarkb: https://review.openstack.org/#/c/98269/ - that should fix havana | 22:52 |
| sdague | clarkb: you want to hardlink them? or are we jumping filesystems | 22:53 |
| clarkb | sdague: jumping filesystemms | 22:54 |
| sdague | ok +2 from me | 22:54 |
| clarkb | sdague: /opt is the big fs mounted on the ephemeral drive /home/jenkins/cache/files is the root fs | 22:54 |
| *** jamielennox|away is now known as jamielennox | 22:54 | |
| sdague | gotcha | 22:54 |
| clarkb | we can probably symlink | 22:54 |
| clarkb | but we are trying to remove disk usage on / | 22:54 |
| clarkb | because its only 10G | 22:55 |
| sdague | yep, no worries | 22:55 |
| *** starmer has quit IRC | 22:56 | |
| clarkb | mordred: I have questions about dib | 22:56 |
| *** lcheng_ has quit IRC | 22:56 | |
| clarkb | mordred: I am now at the point of wondering whether you tested these elements >_> | 22:56 |
| *** maxbit_ has quit IRC | 22:56 | |
| *** maxbit has joined #openstack-infra | 22:57 | |
| pleia2 | my favorite part about jboss is how it makes me feel like I don't know how to use computers | 22:58 |
| pleia2 | go forth and do magic things (you better work) | 22:58 |
| pleia2 | ((it doesn't work)) | 22:59 |
| clarkb | pleia2: that sounds like dib | 22:59 |
| clarkb | it isn't running puppet | 22:59 |
| pleia2 | haha | 22:59 |
| clarkb | it should run puppet | 22:59 |
| clarkb | I have no idea what is going on | 22:59 |
| pleia2 | boo | 22:59 |
| morganfainberg | clarkb, so what you're saying is they successfully ported the jboss feature pleia2 just described to dib... or vice versa | 22:59 |
| krotscheck | Anyone know if there’s a dependency-and/or-other reason that would prevent me from adding/using puppetlabs/rabbitmq to /config? | 23:00 |
| morganfainberg | i dislike "go do magic things" instructions. | 23:00 |
| mattoliverau | Morning | 23:01 |
| clarkb | krotscheck: not that I know of | 23:01 |
| *** zns has joined #openstack-infra | 23:01 | |
| clarkb | morganfainberg: in this case its deceptive and I think I understand it because its just bash | 23:01 |
| pleia2 | morganfainberg: heh, it's just complicated and I've never gotten to deep into it so the screens of launching stuff is greek (well, java) to me :) | 23:01 |
| clarkb | but in reality I have no idea what is going on | 23:01 |
| morganfainberg | clarkb, :( well if i can help (why do i say these things) let me know. happy to add a spare pair of eyes if it'd help you at all | 23:02 |
| pleia2 | mattoliverau: good morning | 23:02 |
| krotscheck | clarkb: Awesome, thanks :) | 23:02 |
| clarkb | morganfainberg: thanks but so far I have been able to hit it with a hammer until things go | 23:02 |
| clarkb | most recent hammer was commenting things out | 23:03 |
| clarkb | >_> | 23:03 |
| morganfainberg | clarkb, hehe | 23:03 |
| *** amcrn has quit IRC | 23:03 | |
| morganfainberg | pleia2, i think that describes most of my recent interactions with Java based apps. | 23:03 |
| pleia2 | I'm glad it's not just me | 23:04 |
| mattoliverau | Hammers: the only tool your ever really need :P | 23:04 |
| mattoliverau | s/your/you'll | 23:04 |
| *** starmer has joined #openstack-infra | 23:05 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-dev/pbr: Check for git before querying it for a version https://review.openstack.org/98273 | 23:07 |
| jogo | clarkb: I would review but I don't have enough context to about that part o fthe system https://review.openstack.org/#/c/98265/ | 23:08 |
| clarkb | jogo: when fungi gets back he should be able to review | 23:09 |
| jogo | cool | 23:10 |
| fungi | looking | 23:10 |
| clarkb | in fairness to mordred his dib elements have just drifted away from some of the refactoring around image builds we have done | 23:10 |
| *** starmer has quit IRC | 23:10 | |
| jogo | good news is it looks like things aren't getting worse right now | 23:11 |
| clarkb | but that coupled with my lack of working dib knowledge is confusing. Trial by fire is good for me | 23:11 |
| anteaya | morning mattoliverau | 23:14 |
| fungi | clarkb: i'm a little weirded out that $copy_cache is treated as a bool switch based on whether you put arbitrary string contents in the parameter, but it looks like it should work | 23:14 |
| clarkb | fungi: ya its clunky. Have a better idea? | 23:14 |
| clarkb | fungi: could pass in mv or cp | 23:14 |
| clarkb | and get rid of the if | 23:14 |
| *** jhesketh has joined #openstack-infra | 23:14 | |
| clarkb | would allow you to use rsync and whatever else too | 23:15 |
| fungi | meh. i don't feel strongly enough to suggest anything else | 23:15 |
| clarkb | ok | 23:15 |
| mattoliverau | Morning anteaya, how you this fine day? | 23:16 |
| sdague | yeh, I'd say lets just go with it. It's not the squireliest thing we've got | 23:16 |
| anteaya | mattoliverau: tired but still typing | 23:16 |
| jhesketh | Morning | 23:16 |
| anteaya | mattoliverau: how are you? | 23:16 |
| anteaya | morning jhesketh | 23:16 |
| jhesketh | zaro: Interesting.. that was just running tox? | 23:16 |
| jhesketh | heya anteaya | 23:16 |
| mattoliverau | anteaya: lol, don't stay up too late, we will all need you refreshed for tomorrow :) I'm really good. It's Friday before a long weekend! And the wife and I are going to visit Tasmania this weekend, so rather excited! | 23:17 |
| fungi | clarkb: sdague: out of curiosity why wasn't this bothering us before? is get-pip.py the first file grenade's really expected in there (by way of havana devstack)? | 23:18 |
| fungi | does this go away after havana is no more? | 23:18 |
| clarkb | fungi: because devstack never used get pip before | 23:18 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Added query for glance _authorized_upload_ bug 1326899 https://review.openstack.org/98189 | 23:18 |
| uvirtbot | Launchpad bug 1326899 in glance "FAIL: glance.tests.unit.v1.test_api.TestGlanceAPI.test_add_copy_from_image_authorized_upload_image_authorized" [Undecided,New] https://launchpad.net/bugs/1326899 | 23:18 |
| clarkb | fungi: it was using the pip 1.4.1 sdist | 23:18 |
| anteaya | mattoliverau: awesome!! | 23:18 |
| clarkb | fungi: that and the fallbacks usually work | 23:18 |
| anteaya | pleia2: I'm lost what is the point of zanata again? | 23:19 |
| sdague | fungi: right | 23:19 |
| pleia2 | anteaya: one of the options to replace transifex | 23:19 |
| anteaya | mattoliverau: and thanks for your kind words :D | 23:19 |
| sdague | or more clarkb right | 23:19 |
| anteaya | pleia2: what other the other options | 23:19 |
| pleia2 | anteaya: pootle is the other leader | 23:19 |
| *** maxbit_ has joined #openstack-infra | 23:19 | |
| fungi | sdague: clarkb: but this doesn't affect icehouse->juno/master grenade? | 23:19 |
| anteaya | pleia2: zanata sounds fraught with obstacles | 23:19 |
| pleia2 | anteaya: the devs offered to help us, so I'm giving them a few chances :) | 23:20 |
| anteaya | pleia2: are you also working on the pootle? | 23:20 |
| sdague | fungi: no, because we -O in icehouse and juno devstack | 23:20 |
| anteaya | pleia2: you generous soul, you | 23:20 |
| pleia2 | anteaya: yeah, we've had a pootle dev server up since october | 23:20 |
| sdague | the get_pip.py fix did a clean cherry to icehouse | 23:20 |
| anteaya | pleia2: ha ha ha, my decision would already be made | 23:20 |
| sdague | and a merge conflict to havana | 23:20 |
| *** maxbit has quit IRC | 23:20 | |
| *** praneshp has quit IRC | 23:20 | |
| sdague | but I didn't pick up on why | 23:20 |
| sdague | this was why | 23:20 |
| *** starmer has joined #openstack-infra | 23:21 | |
| *** maxbit has joined #openstack-infra | 23:21 | |
| sdague | fungi: and this was the road not take | 23:21 |
| sdague | not taken | 23:21 |
| sdague | we always defaulted through the other path | 23:21 |
| sdague | part of the issue with the fact that this code had 2 parallel paths | 23:21 |
| sdague | and the one we wanted no one was running until today | 23:21 |
| sdague | so ... it was wrong | 23:21 |
| fungi | sdague: aha! got it. so is there some way we can decorate this to be removable come havana eol? otherwise there isn't a lot of indication to future generations why it's doing that | 23:22 |
| sdague | fungi: ?? | 23:22 |
| sdague | devstack has branches | 23:22 |
| sdague | it just dies at some point | 23:22 |
| *** praneshp has joined #openstack-infra | 23:22 | |
| fungi | sdague: yeah, but devstack-gate has no branches, so it's accommodating behavior in devstack stable/havana | 23:23 |
| sdague | fungi: no it isn't | 23:23 |
| * fungi is confused again then | 23:23 | |
| sdague | it's accomodating grenade generally | 23:23 |
| sdague | and providing cached files in old and new | 23:23 |
| sdague | instead of just new | 23:23 |
| fungi | so were not testing grenade icehouse->master? | 23:23 |
| sdague | and making old go to the network | 23:23 |
| sdague | no, we went to the network | 23:23 |
| sdague | but icehouse was correct | 23:24 |
| fungi | oh, or this is actually broken for icehouse->havana grenade too | 23:24 |
| fungi | right, now i get it | 23:24 |
| *** radez_g0n3 is now known as radez | 23:24 | |
| fungi | the master change merged because the network didn't hiccup | 23:24 |
| sdague | yep | 23:24 |
| sdague | and the icehouse change merged because the havana one hadn't yet | 23:25 |
| sdague | so we went to the network the old way | 23:25 |
| sdague | then the havana change merged | 23:25 |
| *** maxbit_ has quit IRC | 23:25 | |
| *** starmer has quit IRC | 23:25 | |
| fungi | so we keep this mechanism around indefinitely. wfm | 23:25 |
| sdague | but we don't forward upgrade | 23:25 |
| sdague | so we didn't see the break | 23:25 |
| sdague | this is actually a place where the forward grenade job would have potentially expoded it | 23:25 |
| fungi | separately, havana devstack needs to curl -O i guess | 23:26 |
| sdague | actually, no it would have 100% expoded it | 23:26 |
| devananda | clarkb, sdague: looks like 97757 hit a neutron failure and got rechecked, and is now at the *bottom* of the queue again | 23:26 |
| sdague | yep | 23:26 |
| sdague | fungi: see - https://review.openstack.org/#/c/98269/ | 23:26 |
| clarkb | devananda: welcome to the jungle | 23:26 |
| devananda | clarkb: \o/ | 23:26 |
| sdague | devananda: right, this was part of the "all hands on deck" call. Because until we can get enough races out of openstack as a whole, this is going to keep happening. | 23:27 |
| devananda | ack | 23:27 |
| *** mrodden has quit IRC | 23:28 | |
| *** spiffxp has quit IRC | 23:28 | |
| *** SlickNik has quit IRC | 23:28 | |
| fungi | ianw: can i assume red hat ci failures for f20 and rhel7 on havana changes are expected? | 23:28 |
| *** SlickNik has joined #openstack-infra | 23:29 | |
| *** miqui has quit IRC | 23:30 | |
| sdague | ok, I'm going to get some dinner. My brain is pretty toast anyway. Will probably just work on some debuggability fixes for grenade in the morning so those fails will be a little easier to sort out. | 23:30 |
| *** SlickNik has quit IRC | 23:30 | |
| ianw | fungi: yeah, don't think i've fixed them up for that branch | 23:30 |
| ianw | we don't see a lot of havana changes | 23:30 |
| ianw | redhatci is is a pretty sorry state all over until a few changes get merged ... i'm keeping an eye out here if i can help, let me know ... | 23:31 |
| clarkb | our devstack precise images are 2.6GB qcow2 | 23:31 |
| *** ociuhandu has joined #openstack-infra | 23:32 | |
| *** marcoemorais has quit IRC | 23:33 | |
| *** marcoemorais has joined #openstack-infra | 23:33 | |
| fungi | clarkb: sounds about right. bare-precise and bare-centos6 were larger according to glanceclient, if memory serves | 23:34 |
| *** virmitio has quit IRC | 23:36 | |
| *** lcheng_ has joined #openstack-infra | 23:37 | |
| *** hemna_ has quit IRC | 23:37 | |
| *** aconrad has quit IRC | 23:38 | |
| clarkb | fungi: yeah its all the cached data | 23:38 |
| clarkb | mordred so I mostly have it working now. I am trying to tighten up the errors cases and use set -e everywhere | 23:39 |
| clarkb | mordred: puppet poses a problem though because puppet will always fail on restarting iptables persistent and rsyslog | 23:39 |
| clarkb | but in general we want a puppet failure to not result in a new image | 23:39 |
| clarkb | not sure how to tackle that yet | 23:39 |
| fungi | clarkb: in what way does puppet fail restarting iptables? is this because it's calling the initscript with restart instead of start (since all that initscript does is reapply the rules files)? | 23:42 |
| *** UtahDave has quit IRC | 23:42 | |
| *** mrodden has joined #openstack-infra | 23:43 | |
| anteaya | would 2014-06-05 21:10:52.334 24090 ERROR oslo.messaging.notify.dispatcher [-] Exception during message handling cause check-tempest-dsvm-neutron to fail? | 23:46 |
| anteaya | http://logs.openstack.org/31/95831/4/check/check-tempest-dsvm-neutron/3c7af2e/logs/screen-ceilometer-anotification.txt.gz?level=ERROR | 23:47 |
| clarkb | fungi: it is because it runs in a chroot | 23:47 |
| clarkb | fungi: and iirc dib disables toggling of services | 23:47 |
| clarkb | I am sure there is a way around this and I will hop over to #tripleo to ask | 23:47 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Modularized StoryBoard Module https://review.openstack.org/98007 | 23:47 |
| *** markwash has quit IRC | 23:48 | |
| fungi | clarkb: oh! this probably needs policy.d gymnastics then | 23:48 |
| fungi | er, policy-rc.d | 23:49 |
| fungi | http://people.debian.org/~hmh/invokerc.d-policyrc.d-specification.txt | 23:49 |
| * clarkb reads | 23:49 | |
| fungi | i can't recall if that made it into ubuntu, or if there's a newer more recommended solution to handling initscripts in chroots nowadays | 23:49 |
| *** dprince has quit IRC | 23:49 | |
| *** starmer has joined #openstack-infra | 23:50 | |
| anteaya | well I found a bug for it and have sent 95831,4 back via recheck, going for a walk now | 23:52 |
| *** jaypipes has quit IRC | 23:52 | |
| pleia2 | anteaya: enjoy | 23:52 |
| fungi | clarkb: oh, though policy-rc.d will only end up being relevant for deb maintscripts and other things which call invoke-rc.d, but puppet very well may be doing less distro-specific things to call initscripts | 23:53 |
| clarkb | fungi: probabl | 23:54 |
| fungi | clarkb: an alternative... temporarily symlink all initscripts to bin/true during the build and then restore them after ;) | 23:54 |
| *** mmaglana has quit IRC | 23:54 | |
| clarkb | fungi: that is terrible but I might just do it | 23:55 |
| * fungi has more terrible ideas if you run out | 23:55 | |
| *** zns has quit IRC | 23:55 | |
| *** starmer has quit IRC | 23:55 | |
| clarkb | fungi: well I am tempted to have a global CHROOT flag | 23:55 |
| clarkb | and puppet can be smart based on that | 23:55 |
| fungi | i think the trick is going to be if puppet installs a package and then tries to manipulate the initscript for it directly or something | 23:56 |
| clarkb | its a bit clunky but at least contained to the source of the problem | 23:56 |
| fungi | well, it won't apply inside third-party modules we're using though | 23:56 |
| fungi | so no idea if that will present a problem | 23:56 |
| *** atiwari has quit IRC | 23:57 | |
| clarkb | oh good point | 23:57 |
| clarkb | eg mysql | 23:57 |
| clarkb | on the on devstack nodes | 23:57 |
| fungi | clarkb: dib really needs kvm instead of just a chroot, i'm betting, for this | 23:57 |
| clarkb | fungi: yes that is what I told lifeless and he said no | 23:57 |
| clarkb | because this is a specific instance of a generic problem | 23:57 |
| clarkb | unfortunately it is the only instance that bites us | 23:58 |
| clarkb | and the rest of the generic bits don't matter | 23:58 |
| lifeless | fungi: so, debootstrap, same situation - don't run services when preparing operating systems | 23:58 |
| lifeless | fungi: run them when running operating systems | 23:58 |
| fungi | lifeless: agreed. downside is we have to avoid puppet then because... not designed to actually care | 23:58 |
| lifeless | the puppet scripts in question need to be taught the difference between prepare and operate | 23:59 |
| clarkb | fungi: we could use packer | 23:59 |
| clarkb | just don't tell lifeless | 23:59 |
| *** SlickNik has joined #openstack-infra | 23:59 | |
| clarkb | >_> | 23:59 |
| clarkb | lifeless: so we can do that for our stuff, but not for the third party things | 23:59 |
| clarkb | mysql and postgres being the complicated ones | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
