| *** dkranz has joined #openstack-infra | 00:00 | |
| fungi | i too worry that we're not keeping ahead of the project growth curve enough to keep it from getting painful | 00:00 |
|---|---|---|
| notmyname | proof by existence. it's already painful ;-) | 00:01 |
| * fungi nods | 00:01 | |
| *** thuc has joined #openstack-infra | 00:01 | |
| fungi | the uptick in activity since devs returned from winter holidays has been tough to accommodate | 00:02 |
| clarkb | fungi: I think it is more tha last minute milestone stuff | 00:02 |
| notmyname | clarkb: than or the? | 00:02 |
| clarkb | we are all great at procrastinating | 00:02 |
| clarkb | notmyname: the sorry silly tyops | 00:02 |
| notmyname | heh, both work but give very different meanings | 00:03 |
| notmyname | eg here's a way devs could work around gate instability: push all patches to another repo and then have someone commit the changes and babysit them through the gate. that would allow most devs to continue to contribute and take into account otherwise approved patches and it only keeps one person on babysitting duty | 00:04 |
| notmyname | to be clear, I think this is a _very_ bad long-term thing | 00:04 |
| notmyname | but it works | 00:04 |
| *** herndon has quit IRC | 00:05 | |
| clarkb | how does that help the dev? they get zero testing that way | 00:05 |
| clarkb | maybe that is the idea :) | 00:05 |
| notmyname | how are they not doing any testing before they submit patches??!?!?! | 00:05 |
| * notmyname wanted to have all caps there | 00:05 | |
| clarkb | notmyname: I would guess about 99.9999% of our developers do not test anything before pushing | 00:06 |
| clarkb | notmyname: you are the .0001% | 00:06 |
| clarkb | which comes back to fungi's point that the machien encourages this behavior | 00:06 |
| clarkb | folks don't know how to do their own testing and debugging and so on | 00:06 |
| notmyname | clarkb: I'm not at all claiming to do things right. in what I do I want to get better. but testing your code before submitting it and testing it before approving it seems to be basic "I am a programmer" skills | 00:07 |
| fungi | well, to be honest, i see this fairly often in security fixes too... the devs run unit tests and style checks, but integration testing is hard and we quite often find that a security fix breaks interop with something we're testing | 00:07 |
| dkranz | mtreinish: One of us should +A the whitelist change when if passes check | 00:07 |
| clarkb | notmyname: I completely agree. But I also see the reality of how often check tests fail for trivial things that would be caught by simple local testing | 00:07 |
| fungi | so it's unfair to say they don't test things. we implemented the integration testing machinery to make it easier for them to be fairly sure without having to run a matrix of 10 different devstack configurations by hand for every proposed change | 00:08 |
| fungi | a lot of devs definitely don't test things rigorously before proposing for review, granted. i don't have any idea what percentage that is though | 00:09 |
| fungi | but that aside, we try to pre-run all the tests anyway before they get approved | 00:09 |
| fungi | so gating should ideally just be catch-up testing to make sure the results don't change between when they were last obtained and merging to the repo | 00:10 |
| clarkb | fungi: just glacning at the check queue ~5 changes there have trivial problems that would have been detected by `tox` on a local box | 00:11 |
| fungi | yep | 00:12 |
| clarkb | but ya that is why we have the check queue | 00:12 |
| fungi | my point was more that even with security fixes where we ask the patch submitter and core reviewers to test the change, we still quite frequently find that we catch issues in automated testing that they didn't | 00:13 |
| *** sarob has quit IRC | 00:13 | |
| clarkb | I see | 00:13 |
| fungi | so while it's nice for devs to test their changes before submitting, its not sufficient | 00:14 |
| fungi | the "have someone shepherd the changes in" suggestion is to some extent one of the things the vmt ends up doing in that case, and it's definitely time-consuming | 00:16 |
| *** CaptTofu has joined #openstack-infra | 00:17 | |
| clarkb | fungi: ya, the qa team (and to a certain extent myself) have had to do that when particular bugs need fixing right away to avoid much more pain | 00:18 |
| *** jgillies has quit IRC | 00:18 | |
| *** ^demon|away is now known as ^d | 00:18 | |
| *** ^d has left #openstack-infra | 00:18 | |
| fungi | clarkb: highlights are hopefully recorded in yesterday's meeting summary on eavesdrop, and i think i may have given you some action items. i honestly don't remember. i think i probably just gave most of them to myself but feel free to take over any which pique your interest | 00:19 |
| *** esker has quit IRC | 00:20 | |
| clarkb | ok, I should look at that summary | 00:20 |
| fungi | i tried to capture the state of all the broken as accurately as i could | 00:20 |
| *** beagles has quit IRC | 00:21 | |
| fungi | also, keep in mind that i've not had sufficient bandwidth to keep up with mailing lists and bug triage, so you are almost certainly going to find things i'm as of yet unaware of | 00:21 |
| fungi | just keeping up with the issues people have been bringing up in irc has been sufficient to keep my nighttime nap breaks brief | 00:22 |
| clarkb | :( | 00:23 |
| clarkb | fungi: quick question, what are we using the org.openstack group on maven nexus for? | 00:24 |
| fungi | that rackspace docs thingy | 00:24 |
| clarkb | but that is a rackspace stackforge thingy | 00:25 |
| clarkb | not an org.openstack thing | 00:25 |
| clarkb | are they going to change it? and if so is that something that needs to be TC'd? | 00:25 |
| fungi | is it? i thought it was infra. that's how scattered i am | 00:25 |
| clarkb | fungi: I can tackle the jenkins01 upgrade, and will work with zaro to sort out the scp plugin race with logstash | 00:26 |
| fungi | it was something infra was working to get extracted from rackspace and publication automation built for, so seems odd we'd go to those lengths for a stackforge something | 00:26 |
| clarkb | fungi: we were doing that because we had it all sorted, then they said wait you are doing it wrong and things got weird | 00:26 |
| fungi | but yeah, if it's not an official openstack effort, i agree that org.openstack is the wrong place for it | 00:27 |
| *** xchu has joined #openstack-infra | 00:30 | |
| clarkb | fungi: oh and I am still good to help with project rename on saturday, is there a scheduled time for that? | 00:32 |
| clarkb | zaro: I can talk SCP plugin race now if you are around | 00:32 |
| *** mrodden has joined #openstack-infra | 00:33 | |
| fungi | clarkb: i guessed at 1900utc while in the meeting i think? but really i don't care. whatever time works for you to be around | 00:33 |
| clarkb | 1900utc sounds good to me | 00:33 |
| fungi | also probably not captured in the meeting, i've asked mark collier to provide some guidance/confirmation on the term "openstack" in short names of stackforge projects, just so we don't get anybody in trademark hot water by continuing to approve them | 00:34 |
| clarkb | oh good idea | 00:35 |
| mtaylor | hey it's clarkb! | 00:35 |
| clarkb | mtaylor: its an mtaylor | 00:35 |
| clarkb | mtaylor: I got bumped to business on my last flight. It helped a little but not much. Not sure how you manage | 00:36 |
| *** mtaylor is now known as mordred | 00:36 | |
| *** mordred has quit IRC | 00:36 | |
| *** mordred has joined #openstack-infra | 00:36 | |
| mordred | clarkb: years of practice | 00:38 |
| mordred | also, callouses | 00:38 |
| *** CaptTofu has quit IRC | 00:38 | |
| clarkb | fungi: so to recap on what I have caught up on so far. zaro is interested in scp plugin race. I can upgrade jenkins01, and will assist project rename on saturday 1900utc | 00:39 |
| fungi | sounds great--thanks! | 00:39 |
| fungi | also, new today, i've been aggressively doing passes to retry nodepool deletes of nodes already in a delete state and it seems to help increase capacity. apparently rackspace is bad about ignoring deletes | 00:41 |
| fungi | clarkb: oh, also. reviewing all the things. i haven't been doing nearly as much of that as we need, so there are lots of people asking why their patches are being ignored. i feel bad, but there's only so much time in the day | 00:42 |
| clarkb | got it. I won't review anything today. Too zombie like, but will put that on the list for tomorrow | 00:43 |
| fungi | yeah, i wasn't meaning to dump stuff on you, just pointing out where the current pain points are for when you're rested and able | 00:44 |
| clarkb | you should dump stuff on me :) I just don't trust myself to do proper code review at the moment | 00:44 |
| fungi | yeah, try to recover. you spent a while on the opposite side of a sphere 10k miles in diameter | 00:45 |
| StevenK | fungi: It's hard work trying to avoid dropbears ... | 00:46 |
| fungi | StevenK: not if you're any good at riding the 'roos | 00:46 |
| *** dripton has quit IRC | 00:46 | |
| clarkb | StevenK: I am slightly disappointed that I didn't come face to face with one of the many deadly creatures of that continent | 00:46 |
| clarkb | fungi: 'skippy' was the term used on the kebab shack that I got kangaroo kebabs from :) | 00:47 |
| *** mrda has joined #openstack-infra | 00:47 | |
| *** lcestari has quit IRC | 00:47 | |
| StevenK | clarkb: http://en.wikipedia.org/wiki/Skippy_the_Bush_Kangaroo | 00:48 |
| clarkb | oh man I ate skippy | 00:48 |
| StevenK | Hahahaha | 00:49 |
| *** dripton has joined #openstack-infra | 00:49 | |
| StevenK | clarkb: I am happy enough to see said deadly creatures from behind thick glass. | 00:50 |
| *** mrodden has quit IRC | 00:52 | |
| *** sarob has joined #openstack-infra | 00:58 | |
| *** boris-42 has quit IRC | 01:05 | |
| *** CaptTofu has joined #openstack-infra | 01:05 | |
| *** dkliban has quit IRC | 01:06 | |
| *** jcoufal has quit IRC | 01:08 | |
| *** wenlock has quit IRC | 01:11 | |
| *** alexpilotti has quit IRC | 01:11 | |
| *** thuc_ has joined #openstack-infra | 01:14 | |
| zaro | clarkb: sorry, about to head out for today. will be online later though. | 01:14 |
| clarkb | zaro: ok, I don't expect to be awake for much longer, can talk about it tomorrow | 01:15 |
| *** dstanek has joined #openstack-infra | 01:15 | |
| *** thuc has quit IRC | 01:17 | |
| *** thuc_ has quit IRC | 01:18 | |
| fungi | mmmskippy | 01:18 |
| *** jasondotstar has joined #openstack-infra | 01:21 | |
| *** Loquacity is now known as loquacity | 01:26 | |
| *** loq_mac has joined #openstack-infra | 01:27 | |
| *** jerryz has quit IRC | 01:29 | |
| *** loquacity is now known as loquacities | 01:29 | |
| *** loq_mac has quit IRC | 01:29 | |
| *** nosnos has joined #openstack-infra | 01:33 | |
| *** gothicmindfood has quit IRC | 01:34 | |
| *** reed has quit IRC | 01:35 | |
| mordred | clarkb: also, I'm working through project add patches, running manage-projects by hand debug mode | 01:43 |
| mordred | clarkb: so that I can potentially catch what's going wrong | 01:43 |
| *** gokrokve has quit IRC | 01:43 | |
| *** dims has quit IRC | 01:43 | |
| clarkb | mordred: any ideas yet? I wonder if we are just exposing new bugs by fixing old ones | 01:43 |
| *** gokrokve has joined #openstack-infra | 01:43 | |
| mordred | clarkb: unsure. it's so far worked fine running by hand | 01:44 |
| mordred | so I'm wondering if it's a weird puppetness | 01:44 |
| *** jgriffit2 has quit IRC | 01:45 | |
| *** jgriffith has joined #openstack-infra | 01:46 | |
| *** gokrokve has quit IRC | 01:48 | |
| fungi | that has crossed my mind | 01:49 |
| *** dcramer_ has joined #openstack-infra | 01:50 | |
| fungi | i will note that before clarkb's last patch it did seem to need to be run once for each new group appearing in the acl, but the other day when i reran it, all new groups were created in one shot | 01:50 |
| *** praneshp has joined #openstack-infra | 01:51 | |
| *** sarob has quit IRC | 01:54 | |
| *** julim has quit IRC | 01:54 | |
| *** sarob has joined #openstack-infra | 01:54 | |
| *** dims has joined #openstack-infra | 01:57 | |
| *** sarob has quit IRC | 01:58 | |
| *** jasondotstar has quit IRC | 02:01 | |
| *** yaguang has joined #openstack-infra | 02:06 | |
| *** jasondotstar has joined #openstack-infra | 02:08 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Explicitly document requirements for 3rd party testing https://review.openstack.org/63478 | 02:12 |
| *** prad has joined #openstack-infra | 02:14 | |
| *** rakhmerov has quit IRC | 02:16 | |
| *** jasondotstar has quit IRC | 02:17 | |
| *** rakhmerov has joined #openstack-infra | 02:18 | |
| fungi | does anybody know enough about config sample generation to triage bug 1269641? i'm pretty sure the assertion in that bug about them being automatically generated by an infra system is incorrect, but mainly just because i don't know about any such automated mechanism | 02:20 |
| uvirtbot | Launchpad bug 1269641 in cinder "cinder.conf.sample out of date" [Undecided,Invalid] https://launchpad.net/bugs/1269641 | 02:20 |
| clarkb | fungi: I kicked it out of openstack-ci and marked it cinder | 02:22 |
| clarkb | fungi: its a project there | 02:22 |
| clarkb | s/there/thing/ | 02:22 |
| fungi | yeah, that's what i thought. thanks! | 02:22 |
| clarkb | and jgriffith updated the bug saying all changes failing that test need a rebase as the fix went in | 02:22 |
| fungi | ahh | 02:22 |
| fungi | i hadn't pulled it up yet to notice it had already been tackled, just saw the initial e-mail | 02:23 |
| fungi | clearly i should wind down for the evening | 02:23 |
| clarkb | ya I have been trying to snipe those as soon as I see them in email | 02:24 |
| clarkb | because ugh | 02:24 |
| *** krotscheck has quit IRC | 02:24 | |
| * fungi nods | 02:24 | |
| *** herndon has joined #openstack-infra | 02:30 | |
| *** rnirmal has joined #openstack-infra | 02:31 | |
| *** esker has joined #openstack-infra | 02:35 | |
| *** rfolco has quit IRC | 02:37 | |
| *** esker has quit IRC | 02:41 | |
| *** esker has joined #openstack-infra | 02:42 | |
| lifeless | mordred: fungi: https://review.openstack.org/#/c/66262/3 plox | 02:46 |
| *** esker has quit IRC | 02:46 | |
| *** fifieldt has joined #openstack-infra | 02:47 | |
| *** gokrokve has joined #openstack-infra | 02:48 | |
| *** obondarev_ has joined #openstack-infra | 02:48 | |
| *** nicedice has quit IRC | 02:51 | |
| *** nicedice has joined #openstack-infra | 02:52 | |
| *** bnemec has quit IRC | 03:01 | |
| *** AaronGr is now known as AaronGr_Zzz | 03:04 | |
| lifeless | fungi: where are we at with https://review.openstack.org/#/c/66491/ ? | 03:08 |
| *** nati_ueno has joined #openstack-infra | 03:13 | |
| fungi | i've simply not had time to set up a glanceclient environment with the appropriate credentials and educate myself on how to upload with it | 03:17 |
| fungi | hopefully with clarkb and mordred back i won't be quite so swamped tomorrow | 03:19 |
| openstackgerrit | A change was merged to openstack-infra/config: Export a GEARDSERVER to tripleo-gate. https://review.openstack.org/66262 | 03:20 |
| *** bradm has joined #openstack-infra | 03:21 | |
| *** CaptTofu has quit IRC | 03:23 | |
| *** rossella_s has joined #openstack-infra | 03:24 | |
| *** zhiwei has joined #openstack-infra | 03:26 | |
| mordred | fungi: bah | 03:26 |
| *** CaptTofu has joined #openstack-infra | 03:26 | |
| dkranz | fungi: I guess you are perhaps gone but https://review.openstack.org/# is ready to be pushed through the gate. | 03:28 |
| mordred | dkranz: that's not a change | 03:28 |
| mordred | :) | 03:28 |
| pleia2 | everything! | 03:28 |
| mordred | pleia2: neat! I'm glad everythung is ready | 03:28 |
| fungi | dkranz: if you can specify the change number, i can stick around to promote it on the next reset | 03:29 |
| fungi | ahh, you did specify over in -qa | 03:30 |
| *** jasondotstar has joined #openstack-infra | 03:31 | |
| *** obondarev_ has quit IRC | 03:41 | |
| *** SlickNik has quit IRC | 03:41 | |
| *** SlickNik has joined #openstack-infra | 03:42 | |
| *** rakhmerov has joined #openstack-infra | 03:43 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/devstack-gate: Enable Neutron metering service plugin https://review.openstack.org/66142 | 03:44 |
| EmilienM | fungi: after FWaaS plugin merge in devstack-gate, I rebased for metering plugin ^ | 03:45 |
| *** prad has quit IRC | 03:48 | |
| *** krotscheck has joined #openstack-infra | 03:48 | |
| *** zhiyan has joined #openstack-infra | 03:49 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: DO NOT MERGE: test out eatmydata https://review.openstack.org/66960 | 03:50 |
| *** rossella_s has quit IRC | 03:50 | |
| *** CaptTofu has quit IRC | 03:54 | |
| *** krotscheck has quit IRC | 03:54 | |
| dkranz | fungi: Thanks. Sorry wrong channel. It's at the top now. I will kick the devstack change first thing in the morning if no one else has done it yet. | 03:57 |
| dkranz | mordred: Huh. Paste fail. | 03:58 |
| *** dkliban has joined #openstack-infra | 03:58 | |
| *** dkranz has quit IRC | 04:03 | |
| *** dkliban has quit IRC | 04:09 | |
| *** rnirmal has quit IRC | 04:12 | |
| *** weshay has quit IRC | 04:13 | |
| *** markmcclain has joined #openstack-infra | 04:14 | |
| *** a7ndrew has quit IRC | 04:14 | |
| *** jasondotstar has quit IRC | 04:15 | |
| *** jasondotstar has joined #openstack-infra | 04:18 | |
| *** herndon has quit IRC | 04:19 | |
| mordred | fungi: I know it's late ... but are we expecting to see zuul missing gerrit events in our current state? | 04:26 |
| *** reed has joined #openstack-infra | 04:26 | |
| clarkb | mordred: I don't think so, is it missing events? there is a rather large event queue that it needs to chew through | 04:26 |
| mordred | clarkb: I approved https://review.openstack.org/#/c/66443/ over an hour ago | 04:27 |
| mordred | and I don't see it in the status page at all | 04:27 |
| mordred | how do I tell if it missed the event | 04:27 |
| clarkb | I am guessing it is stuck in the event queue | 04:27 |
| mordred | ok. just making sure that there wasn't a larger issue | 04:27 |
| clarkb | mordred: see the queue lengths on the status page, they are long | 04:27 |
| clarkb | mordred: if you are really curious you can go grep for that change in the zuul log files | 04:27 |
| *** markmcclain has quit IRC | 04:28 | |
| mordred | clarkb: just mildly curious | 04:28 |
| *** gokrokve has quit IRC | 04:28 | |
| *** gokrokve has joined #openstack-infra | 04:30 | |
| *** boris-42 has joined #openstack-infra | 04:30 | |
| *** pcrews_ has quit IRC | 04:31 | |
| *** sandywalsh has quit IRC | 04:33 | |
| *** reed has quit IRC | 04:36 | |
| *** sandywalsh has joined #openstack-infra | 04:45 | |
| *** markwash has quit IRC | 04:45 | |
| *** coolsvap has joined #openstack-infra | 04:55 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 04:59 | |
| *** dkliban has joined #openstack-infra | 05:00 | |
| *** markwash has joined #openstack-infra | 05:01 | |
| *** markwash has quit IRC | 05:04 | |
| *** jasondotstar has quit IRC | 05:10 | |
| *** gyee__ has quit IRC | 05:13 | |
| *** adam_g has quit IRC | 05:16 | |
| *** ryanpetrello has quit IRC | 05:27 | |
| *** nosnos_ has joined #openstack-infra | 05:27 | |
| *** nosnos has quit IRC | 05:28 | |
| *** nicedice has quit IRC | 05:29 | |
| *** ArxCruz has quit IRC | 05:33 | |
| *** markwash has joined #openstack-infra | 05:37 | |
| *** esker has joined #openstack-infra | 05:38 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Explain howto use maven's pre/postbuilders https://review.openstack.org/66035 | 05:41 |
| *** gokrokve has quit IRC | 05:48 | |
| *** loquacities has quit IRC | 05:52 | |
| *** CaptTofu has joined #openstack-infra | 05:55 | |
| *** loquacities has joined #openstack-infra | 05:57 | |
| *** CaptTofu has quit IRC | 05:59 | |
| *** mrda has quit IRC | 06:00 | |
| *** loq_mac has joined #openstack-infra | 06:01 | |
| *** loq_mac has quit IRC | 06:03 | |
| *** nosnos_ has quit IRC | 06:10 | |
| *** nosnos has joined #openstack-infra | 06:10 | |
| *** coolsvap_away has joined #openstack-infra | 06:15 | |
| *** coolsvap has quit IRC | 06:16 | |
| *** coolsvap_away is now known as coolsvap | 06:20 | |
| *** denis_makogon has joined #openstack-infra | 06:22 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 06:22 | |
| *** chandankumar has joined #openstack-infra | 06:23 | |
| *** chmouel has quit IRC | 06:24 | |
| *** senk has quit IRC | 06:24 | |
| *** chmouel has joined #openstack-infra | 06:27 | |
| *** loquacities has quit IRC | 06:34 | |
| *** mattoliverau has quit IRC | 06:35 | |
| *** mikal has quit IRC | 06:35 | |
| *** mikal has joined #openstack-infra | 06:35 | |
| *** mattoliverau has joined #openstack-infra | 06:35 | |
| *** loquacities has joined #openstack-infra | 06:36 | |
| *** esker has quit IRC | 06:39 | |
| *** slong- has joined #openstack-infra | 06:55 | |
| *** slong has quit IRC | 06:55 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Add Periodic Translation Updates to ops-guide https://review.openstack.org/67022 | 07:02 |
| *** salv-orlando_ has joined #openstack-infra | 07:02 | |
| *** salv-orlando has quit IRC | 07:02 | |
| *** salv-orlando_ is now known as salv-orlando | 07:02 | |
| *** yolanda has joined #openstack-infra | 07:15 | |
| *** adam_g has joined #openstack-infra | 07:18 | |
| *** adam_g has quit IRC | 07:18 | |
| *** adam_g has joined #openstack-infra | 07:18 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Add manuals-upstream-translation-update to api-site https://review.openstack.org/66519 | 07:21 |
| *** gokrokve has joined #openstack-infra | 07:24 | |
| *** gokrokve has quit IRC | 07:29 | |
| *** dkliban has quit IRC | 07:30 | |
| *** flaper87|afk is now known as flaper87 | 07:35 | |
| *** boris-42 has quit IRC | 07:43 | |
| *** denis_makogon has quit IRC | 07:43 | |
| *** jamielennox is now known as jamielennox|away | 07:48 | |
| *** gokrokve has joined #openstack-infra | 07:54 | |
| *** CaptTofu has joined #openstack-infra | 07:56 | |
| *** sarob has joined #openstack-infra | 07:56 | |
| *** praneshp has quit IRC | 07:59 | |
| *** CaptTofu has quit IRC | 08:01 | |
| *** harlowja_ is now known as harlowja_away | 08:02 | |
| *** sarob has quit IRC | 08:08 | |
| *** jcoufal has joined #openstack-infra | 08:08 | |
| *** odyssey4me has joined #openstack-infra | 08:09 | |
| *** markwash has quit IRC | 08:16 | |
| openstackgerrit | Sascha Peilicke proposed a change to openstack-dev/pbr: Sync with global requirements https://review.openstack.org/67041 | 08:17 |
| *** dcramer_ has quit IRC | 08:23 | |
| *** hashar has joined #openstack-infra | 08:27 | |
| *** dcramer_ has joined #openstack-infra | 08:28 | |
| *** sarob has joined #openstack-infra | 08:34 | |
| *** luqas has joined #openstack-infra | 08:35 | |
| *** fbo has joined #openstack-infra | 08:38 | |
| *** sarob_ has joined #openstack-infra | 08:40 | |
| *** sarob has quit IRC | 08:42 | |
| *** sarob_ has quit IRC | 08:45 | |
| *** pblaho has joined #openstack-infra | 08:46 | |
| *** vkozhukalov has joined #openstack-infra | 08:48 | |
| *** dizquierdo has joined #openstack-infra | 08:59 | |
| *** _ruhe is now known as ruhe | 09:03 | |
| *** luqas has quit IRC | 09:06 | |
| *** luqas has joined #openstack-infra | 09:07 | |
| *** luqas has quit IRC | 09:08 | |
| *** zhiwei has quit IRC | 09:10 | |
| *** mancdaz_away is now known as mancdaz | 09:11 | |
| *** luqas has joined #openstack-infra | 09:11 | |
| *** dpyzhov has joined #openstack-infra | 09:16 | |
| *** markmc has joined #openstack-infra | 09:16 | |
| *** yassine has joined #openstack-infra | 09:18 | |
| *** Ryan_Lane has quit IRC | 09:19 | |
| *** locke105 has quit IRC | 09:19 | |
| *** salv-orlando_ has joined #openstack-infra | 09:21 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Add openstack-doc-tools https://review.openstack.org/65361 | 09:22 |
| *** salv-orlando has quit IRC | 09:24 | |
| *** salv-orlando_ is now known as salv-orlando | 09:24 | |
| *** afazekas has joined #openstack-infra | 09:25 | |
| *** nati_ueno has quit IRC | 09:33 | |
| *** jhesketh__ has quit IRC | 09:34 | |
| *** sarob has joined #openstack-infra | 09:34 | |
| hashar | :( | 09:35 |
| hashar | I am missing CLA | 09:35 |
| *** fbo has quit IRC | 09:36 | |
| *** derekh has joined #openstack-infra | 09:37 | |
| *** yamahata has quit IRC | 09:38 | |
| *** jcoufal_ has joined #openstack-infra | 09:38 | |
| *** sarob has quit IRC | 09:39 | |
| *** pblaho has quit IRC | 09:39 | |
| *** jishaom has joined #openstack-infra | 09:40 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: maven-deploy: don't require id https://review.openstack.org/66033 | 09:41 |
| *** jcoufal has quit IRC | 09:41 | |
| *** DinaBelova has joined #openstack-infra | 09:42 | |
| *** jcoufal_ is now known as jcoufal | 09:43 | |
| *** salv-orlando has quit IRC | 09:47 | |
| *** salv-orlando has joined #openstack-infra | 09:47 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/devstack-gate: Timestamp setup logs https://review.openstack.org/67086 | 09:49 |
| *** nosnos has quit IRC | 09:51 | |
| *** bookwar has joined #openstack-infra | 09:52 | |
| *** pblaho has joined #openstack-infra | 09:55 | |
| *** CaptTofu has joined #openstack-infra | 09:57 | |
| *** DinaBelova has quit IRC | 09:59 | |
| *** DinaBelova has joined #openstack-infra | 09:59 | |
| *** ruhe is now known as ruhe_away | 09:59 | |
| *** che-arne has joined #openstack-infra | 10:01 | |
| *** CaptTofu has quit IRC | 10:01 | |
| *** che-arne has quit IRC | 10:02 | |
| *** boris-42 has joined #openstack-infra | 10:03 | |
| *** rakhmerov has quit IRC | 10:03 | |
| *** rakhmerov has joined #openstack-infra | 10:04 | |
| *** jamielennox|away has quit IRC | 10:04 | |
| *** che-arne has joined #openstack-infra | 10:04 | |
| *** mestery_ has joined #openstack-infra | 10:05 | |
| *** fbo has joined #openstack-infra | 10:05 | |
| *** ianw has quit IRC | 10:06 | |
| *** mestery has quit IRC | 10:08 | |
| *** ruhe_away is now known as _ruhe | 10:09 | |
| *** ianw has joined #openstack-infra | 10:09 | |
| *** jamielennox|away has joined #openstack-infra | 10:09 | |
| *** max_lobur_afk is now known as max_lobur | 10:11 | |
| *** jooools has joined #openstack-infra | 10:13 | |
| *** sarob has joined #openstack-infra | 10:13 | |
| openstackgerrit | Alexis Lee proposed a change to openstack-infra/elastic-recheck: Augment the e-r query for bug 1258682 https://review.openstack.org/67094 | 10:14 |
| uvirtbot | Launchpad bug 1258682 in openstack-ci "timeout causing gate-tempest-dsvm-full to fail" [High,In progress] https://launchpad.net/bugs/1258682 | 10:15 |
| *** luqas has quit IRC | 10:15 | |
| *** jeblair has quit IRC | 10:17 | |
| *** sarob has quit IRC | 10:18 | |
| *** jeblair has joined #openstack-infra | 10:19 | |
| *** xchu has quit IRC | 10:22 | |
| *** xchu has joined #openstack-infra | 10:23 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 10:25 | |
| *** michchap_ has joined #openstack-infra | 10:26 | |
| *** michchap has quit IRC | 10:26 | |
| *** xchu has quit IRC | 10:28 | |
| *** jishaom has quit IRC | 10:30 | |
| *** Ryan_Lane has joined #openstack-infra | 10:33 | |
| *** sarob has joined #openstack-infra | 10:34 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 10:35 | |
| *** sarob has quit IRC | 10:39 | |
| *** xchu has joined #openstack-infra | 10:41 | |
| *** obondarev_ has joined #openstack-infra | 10:41 | |
| *** luqas has joined #openstack-infra | 10:42 | |
| openstackgerrit | Ivan Melnikov proposed a change to openstack-infra/config: Add pylint job for anvil https://review.openstack.org/67103 | 10:42 |
| *** alexpilotti has joined #openstack-infra | 10:44 | |
| *** salv-orlando_ has joined #openstack-infra | 10:48 | |
| *** salv-orlando has quit IRC | 10:50 | |
| *** salv-orlando_ is now known as salv-orlando | 10:50 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 10:51 | |
| *** _ruhe is now known as ruhe | 10:52 | |
| *** matel has joined #openstack-infra | 10:53 | |
| matel | Hello, I would like to create a localrc for Xenapi testing. | 10:54 |
| matel | I am following the devstack-gate readme: https://github.com/openstack-infra/devstack-gate/blob/master/README.rst#simulating-devstack-gate-tests | 10:54 |
| matel | Why does the doc say to export ZUUL_URL=/home/jenkins/workspace-cache | 10:55 |
| matel | Who's gonna populate the workspace-cache? | 10:56 |
| *** blamar has quit IRC | 10:58 | |
| *** flaper87 is now known as flaper87|afk | 10:59 | |
| *** yamahata has joined #openstack-infra | 10:59 | |
| *** rfolco has joined #openstack-infra | 11:02 | |
| *** mrda has joined #openstack-infra | 11:03 | |
| *** ArxCruz has joined #openstack-infra | 11:05 | |
| *** xchu has quit IRC | 11:13 | |
| *** mrda has quit IRC | 11:18 | |
| *** mdenny has quit IRC | 11:19 | |
| *** tma996 has joined #openstack-infra | 11:30 | |
| *** lcestari has joined #openstack-infra | 11:33 | |
| *** sarob has joined #openstack-infra | 11:34 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Upgrade manila's devstack-job to tempest-job https://review.openstack.org/66846 | 11:39 |
| *** sarob has quit IRC | 11:39 | |
| *** weshay has joined #openstack-infra | 11:41 | |
| *** yaguang has quit IRC | 11:48 | |
| kiall | matel: devstack-gate is intended for Jenkins running DevStack, rather than normal dev's running DevStack .. It will have lots of Jenkins/Zuul/Gerrit specific quirks in how it configures things | 11:50 |
| matel | I understand that, but as I want to integrate to devstack-gate, it's important for me to reproduce that thing. | 11:52 |
| matel | kiall: Are you saying that the documentation (https://github.com/openstack-infra/devstack-gate/blob/master/README.rst#simulating-devstack-gate-tests) is outdated? | 11:53 |
| kiall | matel: ah - if your trying to do that, then devstack-gate and it's docs make sense :) | 11:54 |
| kiall | I thought you we're trying to use it as an example for getting XenApi working under normal devstack! | 11:54 |
| matel | No, it's okay, we are working on integrating the xenapi testing to the official gate. | 11:55 |
| kiall | Anyway - I don't have any specific knowledge on devstack-gate, other than knowing it's purpose :) | 11:55 |
| kiall | The infra folks are almost all US based, and fast asleep right about now.. It usually picks up around here in a few hours.. Someone who knows it will probably be around then | 11:55 |
| matel | kiall: Thanks, I am debugging the errors, and will propose a patch to the docs - as I got everything up and running. | 11:56 |
| *** katyafervent_ is now known as katyafervent_awa | 11:56 | |
| *** CaptTofu has joined #openstack-infra | 11:58 | |
| openstackgerrit | Sascha Peilicke proposed a change to openstack/requirements: Require testrepository>=0.0.18 https://review.openstack.org/67124 | 11:59 |
| *** jp_at_hp has joined #openstack-infra | 12:03 | |
| *** CaptTofu has quit IRC | 12:03 | |
| *** hashar has quit IRC | 12:04 | |
| *** sirushti has quit IRC | 12:05 | |
| *** sirushti has joined #openstack-infra | 12:05 | |
| *** weshay has quit IRC | 12:06 | |
| *** coolsvap has quit IRC | 12:08 | |
| openstackgerrit | Sascha Peilicke proposed a change to openstack/requirements: Require testrepository>=0.0.18 https://review.openstack.org/67124 | 12:10 |
| *** weshay has joined #openstack-infra | 12:12 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack-infra/config: Upload pre-release oslo.messaging tags to pypi https://review.openstack.org/67131 | 12:16 |
| *** katyafervent_awa is now known as katyafervent | 12:19 | |
| *** DinaBelova has quit IRC | 12:23 | |
| *** DinaBelova has joined #openstack-infra | 12:25 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Add Periodic Translation Updates to ops-guide https://review.openstack.org/67022 | 12:28 |
| *** derekh has quit IRC | 12:29 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Upgrade manila's devstack-job to tempest-job https://review.openstack.org/66846 | 12:29 |
| *** sarob has joined #openstack-infra | 12:34 | |
| *** DinaBelova has quit IRC | 12:38 | |
| *** sarob has quit IRC | 12:39 | |
| *** xchu has joined #openstack-infra | 12:40 | |
| *** DinaBelova has joined #openstack-infra | 12:42 | |
| *** derekh has joined #openstack-infra | 12:45 | |
| *** heyongli has joined #openstack-infra | 12:49 | |
| *** smarcet has joined #openstack-infra | 12:50 | |
| *** ruhe is now known as ruhe_away | 12:53 | |
| *** ruhe_away is now known as ruhe | 12:54 | |
| *** obondarev_ has quit IRC | 12:55 | |
| *** xchu has quit IRC | 12:58 | |
| *** prad has joined #openstack-infra | 12:59 | |
| *** dizquierdo has quit IRC | 13:00 | |
| *** prad has quit IRC | 13:04 | |
| *** alexpilotti has quit IRC | 13:06 | |
| *** flaper87|afk is now known as flaper87 | 13:06 | |
| *** salv-orlando has quit IRC | 13:08 | |
| *** zhiyan has quit IRC | 13:12 | |
| *** yamahata has quit IRC | 13:13 | |
| *** yamahata has joined #openstack-infra | 13:15 | |
| *** ArxCruz has quit IRC | 13:18 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/config: Add logging to #openstack-meeting-3 https://review.openstack.org/67152 | 13:18 |
| *** ArxCruz has joined #openstack-infra | 13:21 | |
| *** CaptTofu has joined #openstack-infra | 13:23 | |
| katyafervent | Hi all! Is something wrong with Zuul/Jenkins right now? | 13:28 |
| *** prad has joined #openstack-infra | 13:29 | |
| *** yamahata has quit IRC | 13:29 | |
| ttx | katyafervent: not that I know of... it's just built a backlog it has trouble to resorb | 13:30 |
| ttx | katyafervent: or did you notice anything special ? | 13:31 |
| *** yamahata has joined #openstack-infra | 13:31 | |
| *** fifieldt has quit IRC | 13:31 | |
| katyafervent | I have a review https://review.openstack.org/#/c/66546/ and there is no comment from Jenkins almost for a whole day | 13:33 |
| *** yamahata has quit IRC | 13:33 | |
| katyafervent | What do I need to check first in that case? | 13:33 |
| * ttx checks | 13:33 | |
| AJaeger | katyafervent, look at http://status.openstack.org/zuul/ | 13:34 |
| *** dstanek has quit IRC | 13:34 | |
| AJaeger | ttx, the numbers look wrong - a very long queue and only 5 in check | 13:34 |
| *** prad has quit IRC | 13:34 | |
| ttx | AJaeger: her change is not showing up there | 13:34 |
| *** yamahata has joined #openstack-infra | 13:34 | |
| *** sarob has joined #openstack-infra | 13:34 | |
| AJaeger | ttx "Queue lengths: 816 events, 58 results." - the 816 scarces me and I guess katyafervent's is in there | 13:35 |
| ttx | AJaeger: the very long queue is due to the backlog. It's quite normal to come up with a near-empty check queue at this time of the day | 13:35 |
| ttx | the events take time to be processed, but 90min sound a bit too much | 13:35 |
| AJaeger | katyafervent, FYI: I got comments from Jenkins on my requests but with a long delay | 13:36 |
| ttx | there might be some issue in event processing yesz | 13:36 |
| ttx | fungi will tell us how "normal" this actually is when he arrives | 13:36 |
| katyafervent | Ok) Thanks a lot! | 13:37 |
| ttx | katyafervent: your change is probably stuck in that long event queue | 13:37 |
| ttx | like AJaeger said | 13:37 |
| mordred | yes. the queue backlog is extra slow right now | 13:39 |
| *** sarob has quit IRC | 13:39 | |
| AJaeger | katyafervent, "recheck" will not help - it is approved. it will be picked up eventually. | 13:39 |
| AJaeger | If not, ask somebody to approve it again - but wait until the queue is down again | 13:40 |
| ttx | mordred: IIRC there was some change recently that made it far faster... but maybe it's brittle ? | 13:40 |
| ttx | (event queue processing) | 13:40 |
| *** ruhe is now known as ruhe_away | 13:41 | |
| mordred | yeah - I think there is further work jeblair wants to do in zuul, iirc | 13:41 |
| *** hashar has joined #openstack-infra | 13:43 | |
| *** ruhe_away is now known as ruhe | 13:44 | |
| *** coolsvap has joined #openstack-infra | 13:45 | |
| sdague | yeh, something isn't right on the event ratio right now | 13:45 |
| sdague | there are so few in use nodes, that seems really weird | 13:46 |
| *** salv-orlando has joined #openstack-infra | 13:47 | |
| *** mfink has joined #openstack-infra | 13:49 | |
| *** afazekas_ has joined #openstack-infra | 13:49 | |
| *** NikitaKonovalov has joined #openstack-infra | 13:50 | |
| *** markwash has joined #openstack-infra | 13:51 | |
| *** lucasagomes has joined #openstack-infra | 13:52 | |
| *** jaypipes has joined #openstack-infra | 13:53 | |
| fungi | ttx: perhaps | 13:54 |
| *** dpyzhov has quit IRC | 13:54 | |
| fungi | sdague: i'll see if we have any stale, but i think we just get a lot failing to build or delete getting in the way and chweing up quota | 13:54 |
| *** DinaBelova has quit IRC | 13:55 | |
| *** zhiyan has joined #openstack-infra | 13:55 | |
| sdague | fungi: ok | 13:55 |
| AJaeger | anteaya, +1 to your email on -dev about "Hat Tip to fungi" | 13:55 |
| *** burt1 has joined #openstack-infra | 13:55 | |
| fungi | AJaeger: i sent an e-mail to -dev? | 13:55 |
| anteaya | AJaeger: yeah | 13:55 |
| * fungi has been too busy to read e-mail | 13:55 | |
| *** markmcclain has joined #openstack-infra | 13:55 | |
| anteaya | fungi yeah I sent to to -dev since I didn't want to clog the channel | 13:56 |
| anteaya | you have been too busy to read email | 13:56 |
| AJaeger | fungi: http://lists.openstack.org/pipermail/openstack-dev/2014-January/024550.html | 13:56 |
| fungi | oh, well thanks! | 13:56 |
| *** lucasagomes has quit IRC | 13:56 | |
| anteaya | fungi: :D | 13:56 |
| *** emagana has joined #openstack-infra | 13:56 | |
| AJaeger | Now I'm clogging the channel since I didn't want to clog the mail archives ;) | 13:56 |
| anteaya | I figured you would see it when you had the time to consume | 13:56 |
| anteaya | didn't want to make your job harder than it is | 13:57 |
| anteaya | AJaeger: hahaha | 13:57 |
| *** dpyzhov has joined #openstack-infra | 13:57 | |
| fungi | all current nodes have been in their state in the nodepool table for less than 2 hours (1.12 hours is the oldest state for one currently). i think it's just that we don't retry deletes aggressively at the moment, and since adding lots of rackspace nodes we've found that they often don't delete on the first try so stick around until the cleanup pass kicks off | 13:58 |
| fungi | i'll see if i can clean some out and boost teh number in use | 13:58 |
| *** dolphm has joined #openstack-infra | 13:58 | |
| dolphm | was the gate paused or something overnight? | 13:59 |
| fungi | dolphm: nope. it's been moving all night | 13:59 |
| fungi | people also approve things on the other side of the world | 13:59 |
| dolphm | maybe i didn't sleep as much as i thought i did lol | 13:59 |
| *** ruhe is now known as ruhe_away | 13:59 | |
| dolphm | queue positions of the patches i'm watching just look very familiar | 14:00 |
| *** DinaBelova has joined #openstack-infra | 14:00 | |
| *** dizquierdo has joined #openstack-infra | 14:00 | |
| *** heyongli has quit IRC | 14:01 | |
| fungi | judging from the approval time on the top nova change, it's taking about 22 hours to pass through the integratedgate | 14:02 |
| *** nati_ueno has joined #openstack-infra | 14:04 | |
| dolphm | fungi: eek, good to know | 14:05 |
| *** salv-orlando_ has joined #openstack-infra | 14:06 | |
| kiall | ttx: BTW - re the #openstack-meeting-3 thread .. I believe you can ask the FreeNode guys to set a redirect from #openstack-meeting-alt to a new #openstack-meeting-2 | 14:07 |
| *** dcramer_ has quit IRC | 14:07 | |
| *** lucasagomes has joined #openstack-infra | 14:07 | |
| ttx | kiall: interesting | 14:07 |
| ttx | kiall: would that somehow move everyone to the new one ? | 14:08 |
| kiall | It would redirect upon join as far as I know, I don't think it will shift people who are already there.. | 14:08 |
| *** obondarev_ has joined #openstack-infra | 14:08 | |
| kiall | But that's probably not as big a pain as asking everyone to update their auto join lists, wiki pages etc etc | 14:09 |
| *** emagana has quit IRC | 14:09 | |
| *** thuc has joined #openstack-infra | 14:09 | |
| *** emagana has joined #openstack-infra | 14:09 | |
| kiall | e.g. try /join #php and you'll be redirected to ##php - and get a message like "3:57 [470 (?) #php ##php] Forwarding to another channel" | 14:10 |
| *** salv-orlando has quit IRC | 14:10 | |
| *** salv-orlando_ is now known as salv-orlando | 14:10 | |
| *** markvoelker has joined #openstack-infra | 14:10 | |
| *** NikitaKonovalov has quit IRC | 14:12 | |
| anteaya | dolphm: also currently I am seeing 869 events in the zuul status page, which are incoming messages to zuul from gerrit which zuul has not looked at yet | 14:13 |
| fungi | yeah, that nova change at the head of the gate just failed on a tempest-postgres-full job | 14:13 |
| anteaya | many patches to check and approved patches are undoubtedly in the 869 events | 14:13 |
| *** emagana_ has joined #openstack-infra | 14:14 | |
| fungi | it'll be recalculating this for a few more minutes still | 14:14 |
| dolphm | for whoever did the work: i love the collapsed view on the zuul page - thank you! | 14:15 |
| *** emagana has quit IRC | 14:15 | |
| fungi | dolphm: that was mostly russellb's holiday fun | 14:15 |
| dolphm | russellb: thanks! | 14:15 |
| russellb | :-D | 14:16 |
| russellb | was fun | 14:16 |
| *** nati_ueno has quit IRC | 14:19 | |
| *** nati_ueno has joined #openstack-infra | 14:20 | |
| EmilienM | fungi: Hi, could you re +2 this one ? https://review.openstack.org/66142 I did the rebase like we said yesterday | 14:21 |
| fungi | EmilienM: sure. i was watching it last night to see whether the rebase passed check jobs, but then i fell asleep before it finished | 14:21 |
| *** dkliban has joined #openstack-infra | 14:22 | |
| kiall | ttx: actually.. you don't even need freenode to do it.. `/mode #openstack-meeting-alt +if #openstack-meeting-2` i.e. Set #openstack-meeting-alt to invite only, and forward uninvited users to #openstack-meeting-2 | 14:22 |
| *** julim has joined #openstack-infra | 14:22 | |
| * kiall gets back to work | 14:22 | |
| *** DinaBelova has quit IRC | 14:23 | |
| fungi | kiall: neat! i didn't realize channel modes alone could do that | 14:23 |
| kiall | I had an inkling they could.. but couldn't remember, hence I looked it up ;) | 14:23 |
| AJaeger | fungi, can you approve this, please? - or when will others be around to approve it? https://review.openstack.org/#/c/64795/ We really want finally gating on the api projects... | 14:24 |
| *** lucasagomes has quit IRC | 14:24 | |
| *** obondarev__ has joined #openstack-infra | 14:26 | |
| AJaeger | thanks, fungi! | 14:29 |
| *** obondarev_ has quit IRC | 14:29 | |
| EmilienM | fungi: oh cool, thank you! | 14:29 |
| fungi | AJaeger: yw. i double-checked it, and it seems safe enough | 14:30 |
| fungi | EmilienM: you bet | 14:30 |
| *** thuc has quit IRC | 14:30 | |
| sdague | fungi: hey, how do I get from a build_uuid to a log server url? | 14:30 |
| *** thuc has joined #openstack-infra | 14:31 | |
| AJaeger | fungi, openstack-doc-tools works for the operations-guide already but this is indeed a bit more complex. Thanks! | 14:31 |
| AJaeger | If this works for a day or two, I'll gate some more - I found too many problems with broken repos that I like to see this in ;) | 14:31 |
| fungi | sdague: good question. i think it incorporates other bits too... i'm digging for the function now | 14:31 |
| sdague | ok | 14:32 |
| fungi | sdague: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/openstack_functions.py#n16 | 14:33 |
| fungi | it uses a custom function (basically a plugin) | 14:34 |
| sdague | ok | 14:34 |
| *** sarob has joined #openstack-infra | 14:34 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 14:35 | |
| *** thuc has quit IRC | 14:35 | |
| *** blamar has joined #openstack-infra | 14:35 | |
| AJaeger | fungi, do you have a few minutes to help with manual translations? I'd like your guidance on how to move forward. IF you don't, I can catch you next week or write an email to the infra list. | 14:37 |
| fungi | AJaeger: that's probably complicated enough it would benefit from when there are more people around | 14:37 |
| AJaeger | fungi, will ask again next week... | 14:37 |
| AJaeger | fungi, fine with me - thanks | 14:38 |
| *** sarob has quit IRC | 14:39 | |
| *** dkranz has joined #openstack-infra | 14:41 | |
| *** prad has joined #openstack-infra | 14:42 | |
| fungi | AJaeger: thank you for your patience... i'm still spread a little thin at the moment | 14:42 |
| AJaeger | Yeah, I know... | 14:43 |
| *** thuc has joined #openstack-infra | 14:43 | |
| *** dstanek has joined #openstack-infra | 14:43 | |
| *** dizquierdo has quit IRC | 14:45 | |
| *** salv-orlando_ has joined #openstack-infra | 14:46 | |
| *** salv-orlando has quit IRC | 14:46 | |
| *** salv-orlando_ is now known as salv-orlando | 14:46 | |
| *** sandywalsh has quit IRC | 14:47 | |
| *** salv-orlando has quit IRC | 14:49 | |
| *** eharney has quit IRC | 14:49 | |
| *** salv-orlando has joined #openstack-infra | 14:49 | |
| *** obondarev__ has quit IRC | 14:49 | |
| *** obondarev_ has joined #openstack-infra | 14:51 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Create gate for api-site https://review.openstack.org/64795 | 14:53 |
| AJaeger | fungi, sorry - just noticed that I had to rebase the change so that it can get merged. | 14:53 |
| *** CaptTofu has quit IRC | 14:53 | |
| AJaeger | There was a small conflict that I had to resolve ^ | 14:53 |
| fungi | okay | 14:53 |
| *** AaronGr_Zzz has quit IRC | 14:54 | |
| AJaeger | can I ping you later for re-approving it? | 14:54 |
| fungi | sure | 14:54 |
| AJaeger | ok, then until later - thanks! | 14:55 |
| *** bnemec has joined #openstack-infra | 14:55 | |
| *** bnemec has quit IRC | 14:55 | |
| *** bnemec has joined #openstack-infra | 14:56 | |
| *** ryanpetrello has joined #openstack-infra | 14:56 | |
| *** thuc has quit IRC | 14:56 | |
| *** thuc has joined #openstack-infra | 14:57 | |
| *** senk has joined #openstack-infra | 14:57 | |
| *** boris-42 has quit IRC | 15:00 | |
| *** senk1 has joined #openstack-infra | 15:00 | |
| *** sandywalsh has joined #openstack-infra | 15:00 | |
| *** thuc has quit IRC | 15:01 | |
| *** senk has quit IRC | 15:02 | |
| *** SergeyLukjanov_a is now known as SergeyLukjanov | 15:02 | |
| *** oubiwann has joined #openstack-infra | 15:03 | |
| *** AaronGr_Zzz has joined #openstack-infra | 15:03 | |
| max_lobur | Hi Folks! Could someone please help me to review https://review.openstack.org/#/c/66349/ (Add futures library to global requirements). We need this on Ironic project | 15:05 |
| *** rwsu has joined #openstack-infra | 15:07 | |
| *** zhiyan has left #openstack-infra | 15:11 | |
| fungi | max_lobur: you really want to try to get reviews from some of the explicit members of https://review.openstack.org/#/admin/groups/131,members | 15:11 |
| fungi | max_lobur: the release and infra teams are mainly only included there so we can approve emergency requirements changes affecting the infrastructure or release process | 15:12 |
| mordred | is anyone interested in disucssing why I've been awake already for an hour and a half? | 15:12 |
| fungi | mordred: not so much. i assume your branes are still on nyc time | 15:12 |
| *** nati_uen_ has joined #openstack-infra | 15:14 | |
| *** mestery_ is now known as mestery | 15:14 | |
| *** NikitaKonovalov has joined #openstack-infra | 15:15 | |
| *** CaptTofu has joined #openstack-infra | 15:15 | |
| mordred | fungi: I haven't been there in like 4 weeks or something - I'm just going to blame clarkb | 15:15 |
| fungi | feel free. he's not around to defend himself yet | 15:15 |
| fungi | easy target | 15:15 |
| max_lobur | fungi, thanks, I see familiar there, I'll ask directly | 15:17 |
| *** nati_ueno has quit IRC | 15:17 | |
| *** kraman1 has joined #openstack-infra | 15:19 | |
| *** AJaeger has quit IRC | 15:20 | |
| *** nati_uen_ has quit IRC | 15:21 | |
| *** marun has joined #openstack-infra | 15:21 | |
| *** nati_ueno has joined #openstack-infra | 15:21 | |
| *** dcramer_ has joined #openstack-infra | 15:22 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Adding configuration for mistral-extra project https://review.openstack.org/66443 | 15:25 |
| *** morganfainberg is now known as needscoffee | 15:25 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 15:26 |
| *** mrodden has joined #openstack-infra | 15:31 | |
| chmouel | is there an issue with jenkins/zuul atm ? my reviews was uploaded here https://review.openstack.org/#/c/65704/ and it does not seem to show on the zuul status, http://status.openstack.org/zuul/ nor in the status.json curl -s http://zuul.openstack.org/status.json | python -mjson.tool |grep 65704 | 15:32 |
| mordred | chmouel: the event queue is very long at the moment | 15:33 |
| chmouel | mordred: ah ok, fair enough | 15:34 |
| fungi | it's not helping that we have several changes near the head of the gate failing basic unit tests at the moment | 15:35 |
| *** herndon has joined #openstack-infra | 15:36 | |
| fungi | argh, and people approving stable branch changes while grenade is still broken for havana | 15:36 |
| fungi | (really, it's devstack being broken for grizzly) | 15:37 |
| chmouel | fungi: anything i can help with? | 15:37 |
| *** b3nt_pin has joined #openstack-infra | 15:37 | |
| *** b3nt_pin is now known as beagles | 15:37 | |
| fungi | chmouel: bug 1266094 would be awesome ;) | 15:37 |
| uvirtbot | Launchpad bug 1266094 in devstack "check_grenade_dsvm failed - aggregates tests failed" [Undecided,Confirmed] https://launchpad.net/bugs/1266094 | 15:37 |
| marun | fungi: for your review :) https://review.openstack.org/#/c/66967/ | 15:38 |
| fungi | marun: on first brush, i'm thinking this will compare similarly to swift's functional tests, yeah? | 15:39 |
| marun | fungi: Yes, and I based it on that job. | 15:39 |
| marun | fungi: experimental for now, so we can work out the kinks. | 15:39 |
| fungi | makes sense | 15:39 |
| chmouel | fungi: cool will give it a try | 15:39 |
| fungi | chmouel: you will be the hero of the stable branches if you fix that ;) | 15:40 |
| fungi | grizzly and havana changes have basically been blocked for a month or more | 15:40 |
| *** dprince has joined #openstack-infra | 15:41 | |
| *** dizquierdo has joined #openstack-infra | 15:41 | |
| mordred | fungi: stable branches are important | 15:41 |
| *** che-arne is now known as che-arne_afk | 15:41 | |
| mordred | fungi: at least, people keep telling me that they are | 15:42 |
| fungi | i've heard the same thing ;) | 15:42 |
| *** che-arne_afk is now known as che-arne | 15:42 | |
| *** vkozhukalov has quit IRC | 15:42 | |
| *** smarcet has left #openstack-infra | 15:43 | |
| *** dpyzhov has quit IRC | 15:43 | |
| *** markmc has quit IRC | 15:45 | |
| fungi | yeah, most of the fast-failing changes in the gate right now seem to be either stable branch patches which should have just been held until stable jobs are working again, or changes whose unit tests are failing (but weren't shortly before they were approved, according to check results) | 15:46 |
| *** gokrokve has quit IRC | 15:46 | |
| *** gokrokve_ has joined #openstack-infra | 15:47 | |
| *** dpyzhov has joined #openstack-infra | 15:47 | |
| *** krotscheck has joined #openstack-infra | 15:47 | |
| *** wenlock has joined #openstack-infra | 15:47 | |
| chmouel | shame :( i am rekicking a clean vm to test this | 15:48 |
| mordred | fungi: do we perhaps have a requirement that bumped and is killing unittest? | 15:49 |
| sdague | fungi: so how are people approving stable branch issues when they don't pass jenkins in the first place? | 15:49 |
| sdague | fungi: what's the grizzly devstack issue? | 15:49 |
| sdague | I had apparently missed that one | 15:49 |
| fungi | sdague: apparently the one i'm looking at now passed grenade on december 26th and got approved yesterday | 15:50 |
| fungi | bit of a gap | 15:50 |
| *** kraman1 has left #openstack-infra | 15:50 | |
| sdague | fungi: yeh, so that's the old results problem | 15:50 |
| fungi | sdague: it started with aggregates exercises failing on grizzly devstack, update in the bug alludes to needing to backport a requirements support change to cinder maybe | 15:51 |
| *** jgrimm has joined #openstack-infra | 15:51 | |
| fungi | but in the interim, who knows what other bitrot has crept in while it was untestable | 15:51 |
| sdague | yep | 15:51 |
| fungi | so there may nw be multiple bugs to sort out | 15:51 |
| *** krotscheck has quit IRC | 15:52 | |
| anteaya | mordred and others, can we get some reviews on https://review.openstack.org/#/c/66967/1 (related to NT code sprint) ? | 15:52 |
| *** krotscheck has joined #openstack-infra | 15:53 | |
| mordred | lgtm | 15:54 |
| anteaya | mordred: thank you | 15:54 |
| *** marun has quit IRC | 15:55 | |
| sdague | fungi: ok, I'll kick off a grizzly job | 15:55 |
| sdague | and figure out how it looks | 15:56 |
| *** dpyzhov has quit IRC | 15:56 | |
| fungi | sdague: thanks! | 15:56 |
| fungi | more eyes make the bug shallow | 15:56 |
| *** marun has joined #openstack-infra | 15:58 | |
| *** thedodd has joined #openstack-infra | 15:58 | |
| *** boris-42 has joined #openstack-infra | 15:59 | |
| openstackgerrit | Evgeny Fadeev proposed a change to openstack-infra/askbot-theme: fixed a bug of misused function call https://review.openstack.org/67197 | 15:59 |
| openstackgerrit | Evgeny Fadeev proposed a change to openstack-infra/askbot-theme: some css changes https://review.openstack.org/67198 | 15:59 |
| *** rcleere has joined #openstack-infra | 16:00 | |
| *** gothicmindfood has joined #openstack-infra | 16:01 | |
| *** odyssey4me has quit IRC | 16:02 | |
| *** dprince has quit IRC | 16:03 | |
| *** eharney has joined #openstack-infra | 16:03 | |
| *** ruhe_away is now known as ruhe | 16:04 | |
| *** dprince has joined #openstack-infra | 16:04 | |
| fungi | nice. my isp's smarthost can't resolve lists.openstack.org now | 16:05 |
| *** senk1 has quit IRC | 16:07 | |
| fungi | SMTP error from remote mail server after RCPT TO:<openstack-dev@lists.openstack.org>: host smtp-server.nc.rr.com [75.180.132.33]: 450 4.1.8 - Domain of recipient does not resolve [R0309001] | 16:07 |
| *** andreaf has quit IRC | 16:09 | |
| fungi | if https://jenkins04.openstack.org/job/gate-tempest-dsvm-postgres-full/2201/console succeeds, we've got 11 changes popping off the gate all at once | 16:10 |
| anteaya | fungi: woohoo, more fun for you | 16:11 |
| anteaya | and hoping that job succeeds | 16:11 |
| *** yaguang has joined #openstack-infra | 16:11 | |
| *** senk has joined #openstack-infra | 16:11 | |
| *** IzikP_Radware has joined #openstack-infra | 16:11 | |
| fungi | yeah, so i see no issues with any of the authoritative dns in the chain to lists.openstack.org. my isp is just exhibiting a new variety of flaky behavior today | 16:12 |
| fungi | and it's fixed now i guess | 16:13 |
| anteaya | yay for random failure | 16:14 |
| *** nati_uen_ has joined #openstack-infra | 16:15 | |
| *** yaguang has quit IRC | 16:16 | |
| *** nati_uen_ has quit IRC | 16:17 | |
| *** nati_uen_ has joined #openstack-infra | 16:18 | |
| *** nati_ueno has quit IRC | 16:19 | |
| fungi | success... and there they go | 16:19 |
| SpamapS | is zuul dead right now or something? | 16:19 |
| SpamapS | https://review.openstack.org/#/c/66248/9 <-- not going? | 16:19 |
| *** AaronGr_Zzz is now known as AaronGr | 16:19 | |
| fungi | Queue lengths: 940 events, 5 results. | 16:19 |
| fungi | you're one of those 940 events it hasn't looked at yet | 16:20 |
| SpamapS | ahhhhhh | 16:20 |
| fungi | it's spending most of its time dealing with the results of jobs which are already running | 16:20 |
| dansmith | fungi: ouch.. what's going on? | 16:21 |
| SpamapS | is this the queue problem that jeblair was talking about solving with "moar gearman" on the ML? | 16:21 |
| fungi | SpamapS: yes | 16:21 |
| fungi | SpamapS: well, recalculating the merge refs for all the dependent queue jobs anyway | 16:22 |
| SpamapS | right | 16:22 |
| fungi | which is a lot of what it spends its time spinning on when things get this heavy | 16:22 |
| fungi | it just got upgraded last week to a rackspace performance node, and the load average graph for it is still pretty crazy... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=388&rra_id=all | 16:23 |
| IzikP_Radware | Hi, could someone help with updating ssh pub key of our 3rd party testing account? | 16:24 |
| fungi | IzikP_Radware: probably in a bit. what happened to the old key? | 16:24 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1269864 https://review.openstack.org/67213 | 16:24 |
| anteaya | I also think it didn't help that nodepool took a time out for about 8 hours day before yesterday | 16:24 |
| uvirtbot | Launchpad bug 1269864 in neutron "error: [Errno 113] No route to host" [Undecided,New] https://launchpad.net/bugs/1269864 | 16:24 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 16:24 | |
| fungi | anteaya: yes, we've not really recovered from that backlog | 16:24 |
| fungi | was a bad way to start the week | 16:25 |
| anteaya | yes | 16:25 |
| fungi | SpamapS: right, so refreshing my memory by looking at top on the zuul server, it's spending a lot of its time acting as a git server too, which is also part of what we would be able to scale out with moar gearman | 16:26 |
| *** thuc has joined #openstack-infra | 16:28 | |
| *** thuc has quit IRC | 16:28 | |
| IzikP_Radware | jenkins VM crushed, we backed up the new key, should remain the same from now on, let me know when to send you the details. | 16:29 |
| *** thuc has joined #openstack-infra | 16:29 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Explicitly document requirements for 3rd party testing https://review.openstack.org/63478 | 16:29 |
| fungi | IzikP_Radware: is it the one you sent to the openstack-infra list a week ago? if so, I'll go ahead and change it to that now | 16:30 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 16:30 | |
| fungi | it's been mostly just me the past week, so i've not been doing a great job of keeping on top of mailing list posts or bug reports | 16:30 |
| IzikP_Radware | yes should be it, ends with YSYb6gCap+aQJ openstack3rdpartytesting@radware.com. you're doing great. | 16:32 |
| *** markmc has joined #openstack-infra | 16:33 | |
| *** mfer has joined #openstack-infra | 16:33 | |
| *** rossella_s has joined #openstack-infra | 16:34 | |
| *** pblaho has quit IRC | 16:34 | |
| fungi | IzikP_Radware: okay, i've updated it | 16:36 |
| IzikP_Radware | Great, working ! thank you very much. | 16:38 |
| anteaya | IzikP_Radware: please subscribe to https://review.openstack.org/#/c/63478/ and weigh in if you can | 16:40 |
| anteaya | this is our direction for 3rd party testing system requirements | 16:40 |
| *** gyee_ has joined #openstack-infra | 16:40 | |
| *** mdenny has joined #openstack-infra | 16:42 | |
| jog0 | fungi: you were right about eatmydata https://review.openstack.org/#/c/66960/2 | 16:44 |
| fungi | jog0: i wish i weren't. it was definitely worth trying again | 16:45 |
| fungi | but proof that there's probably no magic bullet for optimizing our current workload | 16:45 |
| jog0 | :/ yeah | 16:46 |
| *** nicedice has joined #openstack-infra | 16:47 | |
| chmouel | so I have been struggly testing grenade, can i just spin up a saucy image checking grendae and run grende.sh to make it works or there is more? | 16:54 |
| jog0 | fungi: btw any update on the libvirt 1.x stuff? | 16:55 |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Add local-branch option https://review.openstack.org/65369 | 16:56 |
| anteaya | chmouel: sdague feels precise is a better starting point | 16:56 |
| anteaya | he is sitting beside me so I asked first | 16:56 |
| fungi | jog0: no, i discovered that the recent refactoring to make devstack-gate self-testing rexec correctly has implications on my test script which i need to track down | 16:56 |
| *** pcrews_ has joined #openstack-infra | 16:56 | |
| chmouel | anteaya: ah nice thank you | 16:57 |
| fungi | er, re-exec | 16:57 |
| chmouel | say hello to sylvain and EmilienM from me | 16:57 |
| anteaya | chmouel: and then yes grenade.sh is the script to run | 16:57 |
| anteaya | chmouel: I can do that, they are doing great work, very busy | 16:57 |
| anteaya | so is nicholasp as well | 16:58 |
| jog0 | mordred: https://review.openstack.org/#/q/owner:jenkins+project:openstack/python-novaclient,n,z where is the global reqs sync bot? | 17:00 |
| jog0 | fungi: ack | 17:00 |
| jog0 | thanks | 17:00 |
| *** ruhe is now known as ruhe_away | 17:00 | |
| *** dkliban is now known as dkliban_lunch | 17:00 | |
| *** rnirmal has joined #openstack-infra | 17:01 | |
| *** krotscheck is now known as kro_commute | 17:01 | |
| EmilienM | chmouel: hello! | 17:01 |
| *** sarob has joined #openstack-infra | 17:01 | |
| mordred | jog0: I beleive we turned it off due to an issue? fungi? wasn't there a stable/master issue? | 17:01 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 17:01 | |
| *** gothicmindfood has quit IRC | 17:01 | |
| fungi | mordred: it got fixed and reenabled | 17:02 |
| *** sarob has quit IRC | 17:02 | |
| jog0 | fungi: hrm then where is it? | 17:02 |
| jog0 | https://review.openstack.org/#/q/owner:jenkins++is:open,n,z | 17:02 |
| *** sarob has joined #openstack-infra | 17:02 | |
| jog0 | does it work for test-requirements ? | 17:03 |
| mordred | it should, yes | 17:03 |
| *** sarob has quit IRC | 17:04 | |
| jog0 | this patch https://review.openstack.org/#/c/67048/1 raised the question | 17:04 |
| *** sarob has joined #openstack-infra | 17:04 | |
| fungi | jog0: we only just got requirements changes landable again yesterday (since two weeks) | 17:06 |
| *** nati_ueno has joined #openstack-infra | 17:06 | |
| *** nati_ueno has quit IRC | 17:06 | |
| *** jcoufal has quit IRC | 17:07 | |
| *** pcrews_ has quit IRC | 17:07 | |
| fungi | the "bot" isn't a bot. just a jenkins job which runs when a global requirements change merges. if the previous patchset it proposed wasn't passing check jobs or someone downvoted it, the two weeks where we couldn't land any changes to that project would have been sufficient to see those changes auto-abandoned | 17:07 |
| *** gothicmindfood has joined #openstack-infra | 17:07 | |
| *** nati_ueno has joined #openstack-infra | 17:08 | |
| *** nati_ueno has left #openstack-infra | 17:08 | |
| *** NikitaKonovalov has quit IRC | 17:08 | |
| *** gokrokve_ has quit IRC | 17:09 | |
| *** nati_uen_ has quit IRC | 17:09 | |
| *** ruhe_away is now known as _ruhe | 17:10 | |
| *** IzikP_Radware has quit IRC | 17:10 | |
| *** markwash has quit IRC | 17:10 | |
| jog0 | fungi: there are no open novaclient jenkins patches https://review.openstack.org/#/q/owner:jenkins++project:openstack/python-novaclient,n,z | 17:12 |
| *** esker has joined #openstack-infra | 17:12 | |
| fungi | jog0: i'm not surprised. why is that unusual? | 17:13 |
| mordred | fungi: as in - there are none proposed in that period of time | 17:13 |
| fungi | right, we couldn't update openstack/requirements for over two weeks | 17:13 |
| *** sarob has quit IRC | 17:14 | |
| jog0 | fungi: the patch that this is based on was over two weeks old | 17:14 |
| fungi | it's been more than two weeks since it could have proposed a sync patch | 17:14 |
| *** gokrokve has joined #openstack-infra | 17:14 | |
| mordred | ah. duh. I undersatnd the whole conversation now | 17:14 |
| *** gokrokve has quit IRC | 17:14 | |
| *** sarob has joined #openstack-infra | 17:14 | |
| *** gokrokve has joined #openstack-infra | 17:14 | |
| jog0 | https://review.openstack.org/#/c/58362/4 | 17:14 |
| jog0 | is the patch that is trying to be synced over | 17:15 |
| fungi | until i landed that pypi-mirror change yesterday, nothing would have merged to global requirements to trigger an updates patchset. so either the previous patchsets merged or were auto-abandoned in that timeframe, i would guess | 17:15 |
| mordred | fungi: https://review.openstack.org/#/c/58362/ | 17:15 |
| mordred | fungi: landed on 12-30 | 17:15 |
| mordred | the most recent patch proposed to novaclient was on 12/10 | 17:15 |
| *** jooools has quit IRC | 17:16 | |
| *** max_lobur is now known as max_lobur_afk | 17:16 | |
| *** rfolco has quit IRC | 17:17 | |
| jog0 | mordred: exactly, so huh? | 17:17 |
| mordred | I have reverified a requirements change - let's see if when it lands tomorrow if it proposes a requirements change to novaclient | 17:17 |
| jog0 | cool | 17:18 |
| *** sarob has quit IRC | 17:18 | |
| *** sarob has joined #openstack-infra | 17:19 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 17:19 | |
| fungi | novaclient doesn't seem to require keyring. i can't recall if the job is smart enough only to propose the merged change even when it's not for a requirement of that project | 17:20 |
| *** kro_commute has quit IRC | 17:20 | |
| mordred | it doesn't work that way | 17:21 |
| mordred | it operates on the fact that requirements changed at all | 17:21 |
| mordred | and then it does an update.py from requirements to each project - and if _that_ produces any results, it will propose the change | 17:21 |
| mordred | so if novaclient is out of sync with global, a change even to a req that isn't relevant will cause a patch to get it back in sync | 17:21 |
| fungi | looks like it proposed https://review.openstack.org/64464 on the 30th | 17:22 |
| pleia2 | jog0: let me know when you're back in town, I have your LCA bag :) | 17:22 |
| *** markwash has joined #openstack-infra | 17:23 | |
| jog0 | pleia2: sweet, next week | 17:23 |
| mordred | excellent | 17:23 |
| jog0 | fungi: novaclient has keyring in test not reqs | 17:23 |
| *** nati_ueno has joined #openstack-infra | 17:24 | |
| *** hashar has quit IRC | 17:24 | |
| *** tma996 has quit IRC | 17:25 | |
| *** sarob has quit IRC | 17:27 | |
| *** sarob has joined #openstack-infra | 17:28 | |
| *** sarob_ has joined #openstack-infra | 17:30 | |
| *** rfolco has joined #openstack-infra | 17:30 | |
| *** mancdaz has quit IRC | 17:31 | |
| fungi | jog0: mordred: short answer, job needs work if anyone wants to file a bug... http://logs.openstack.org/df/df444f161d085478bc32ac529062afeabcfc3faa/post/propose-requirements-updates/dfeac29/console.html.gz | 17:31 |
| *** sarob has quit IRC | 17:33 | |
| *** nati_ueno has quit IRC | 17:34 | |
| *** rcleere has quit IRC | 17:34 | |
| fungi | (log url determined from the merge commit sha of the corresponding openstack/requirements merge commit) | 17:34 |
| *** sdake has quit IRC | 17:35 | |
| *** rakhmerov has quit IRC | 17:36 | |
| *** nati_ueno has joined #openstack-infra | 17:36 | |
| *** mancdaz_away has joined #openstack-infra | 17:37 | |
| *** mancdaz_away is now known as mancdaz | 17:37 | |
| *** pballand has joined #openstack-infra | 17:38 | |
| *** HenryG has joined #openstack-infra | 17:38 | |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Add local-branch option https://review.openstack.org/65369 | 17:39 |
| *** chandankumar has quit IRC | 17:41 | |
| *** praneshp has joined #openstack-infra | 17:42 | |
| *** sarob_ has quit IRC | 17:42 | |
| lifeless | fungi: ping | 17:42 |
| lifeless | fungi: https://review.openstack.org/#/c/66491/ | 17:42 |
| *** nati_ueno has quit IRC | 17:43 | |
| *** luqas has quit IRC | 17:43 | |
| *** nati_ueno has joined #openstack-infra | 17:43 | |
| *** luqas has joined #openstack-infra | 17:44 | |
| jog0 | fungi: heh jenkins has an old git-review | 17:44 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 17:44 | |
| jog0 | fun fodder: http://lists.openstack.org/pipermail/openstack-docs/2014-January/003644.html | 17:44 |
| jog0 | copyrights in source | 17:44 |
| jog0 | on docs ML | 17:45 |
| fungi | lifeless: yep, reading the manpage for glanceclient now | 17:45 |
| *** luqas has quit IRC | 17:46 | |
| lifeless | fungi: why? I did the upload... | 17:46 |
| *** derekh has quit IRC | 17:46 | |
| *** sarob has joined #openstack-infra | 17:46 | |
| *** sarob has quit IRC | 17:49 | |
| fungi | lifeless: oh, i just now saw your comment on the review. doing that was still in my backlog | 17:50 |
| fungi | thanks! | 17:50 |
| *** yolanda has quit IRC | 17:50 | |
| *** dprince has quit IRC | 17:51 | |
| *** yolanda has joined #openstack-infra | 17:51 | |
| fungi | at least now i've got it set up and should be able to do it next time around | 17:51 |
| *** luqas has joined #openstack-infra | 17:52 | |
| anteaya | we can't seem to get below 950 zuul events on the status page | 17:53 |
| anteaya | results (messages to zuul from Jenkins via gearman) has hit 0 a few times | 17:54 |
| *** luqas has quit IRC | 17:54 | |
| anteaya | just an observation, not a call to action | 17:54 |
| fungi | lifeless: are we going to need to pass a region name? | 17:55 |
| *** Ajaeger has joined #openstack-infra | 17:55 | |
| *** luqas has joined #openstack-infra | 17:55 | |
| fungi | anteaya: it's a fifo. zuul is processing them, but they're arriving as fast as it's servicing them more or less, so it's not really making much ground there | 17:55 |
| fungi | it took over 2 hours for us to get an approved change to show up in the gate a little while ago | 17:56 |
| *** kmartin has joined #openstack-infra | 17:57 | |
| *** nati_uen_ has joined #openstack-infra | 17:57 | |
| fungi | lifeless: nevermind. seems to work for me as long as i don't specify a region | 17:57 |
| *** tian has quit IRC | 17:57 | |
| Ajaeger | fungi: https://review.openstack.org/64795 - the api-site gate - has passed the check gate. Could you check and approve again, please? | 17:58 |
| *** tian has joined #openstack-infra | 17:58 | |
| lifeless | fungi: you may need to eventually, for now, no. | 17:58 |
| fungi | got it | 17:59 |
| anteaya | fungi: yeah | 17:59 |
| openstackgerrit | Evgeny Fadeev proposed a change to openstack-infra/askbot-theme: added the launchpad importer code https://review.openstack.org/67251 | 17:59 |
| *** derekh has joined #openstack-infra | 17:59 | |
| fungi | lifeless: lgtm. glance image-list and nova image-list show "Ubuntu Precise 12.04 LTS Server 64-bit" being there | 17:59 |
| pleia2 | \o/ | 17:59 |
| mordred | woot | 18:00 |
| *** luqas has quit IRC | 18:00 | |
| anteaya | 946 woohoo | 18:00 |
| Ajaeger | Thanks, fungi! | 18:00 |
| pleia2 | thanks fungi | 18:00 |
| notmyname | mordred: fungi: jeblair: clarkb: average number of gate queue resets now graphed at http://not.mn/gate_status.html | 18:00 |
| anteaya | thanks fungi | 18:00 |
| lifeless | there is something terribly wrong with networking atm, it gets about 12kbps to the internet | 18:00 |
| *** dkliban_lunch is now known as dkliban | 18:00 | |
| lifeless | but it is reliable, just slow | 18:00 |
| *** nati_ueno has quit IRC | 18:00 | |
| *** mancdaz is now known as mancdaz_away | 18:01 | |
| anteaya | notmyname: nice | 18:01 |
| *** mattoliverau has quit IRC | 18:01 | |
| jog0 | https://review.openstack.org/#/c/67094/1//COMMIT_MSG | 18:02 |
| jog0 | if you view the commit message as rendered by gerrit | 18:02 |
| jog0 | the URL lost the 2 at the end | 18:02 |
| *** reed has joined #openstack-infra | 18:02 | |
| *** nati_uen_ has quit IRC | 18:02 | |
| *** markmcclain has quit IRC | 18:02 | |
| mordred | oh neat - that's from an HP person I don't know | 18:02 |
| *** rakhmerov has joined #openstack-infra | 18:03 | |
| kmartin | would it be possible for a core to look at https://review.openstack.org/#/c/65179/ now that gate has ungated itself. It's holding up geting a a new HP LeftHand cinder driver from landing....thanks | 18:03 |
| *** harlowja_away is now known as harlowja_ | 18:03 | |
| *** mrodden has quit IRC | 18:03 | |
| jog0 | mordred: yeah, I wonder how he stumbled across this and why gerrit is breaking thigns | 18:03 |
| *** pcrews_ has joined #openstack-infra | 18:04 | |
| *** harlowja_ has quit IRC | 18:04 | |
| *** mrodden has joined #openstack-infra | 18:04 | |
| *** markmcclain has joined #openstack-infra | 18:05 | |
| *** sarob has joined #openstack-infra | 18:05 | |
| *** nati_ueno has joined #openstack-infra | 18:05 | |
| *** nati_ueno has quit IRC | 18:05 | |
| *** dprince has joined #openstack-infra | 18:06 | |
| fungi | jog0: we still have something not quite right with the bug link matching in the gerrit webui, yes. seems to gobble the last digit when it renders the link | 18:06 |
| *** derekh has quit IRC | 18:08 | |
| jog0 | fungi: cool long as someone is aware | 18:08 |
| fungi | jog0: i think there is a proposed fix somewhere in the untamed wilds of the openstack-infra/config change review backlog | 18:08 |
| fungi | take plenty of ammo... there are tigers running wild in there too | 18:09 |
| *** vipul is now known as vipul-away | 18:09 | |
| *** vipul-away is now known as vipul | 18:09 | |
| lifeless | pleia2: so, I suspect nodepool needs to be taught about us having two nics | 18:09 |
| *** sarob has quit IRC | 18:10 | |
| lifeless | pleia2: (I'm betting when fungi tries again that it's going to die flaming :)) | 18:10 |
| *** luqas has joined #openstack-infra | 18:10 | |
| *** krotscheck has joined #openstack-infra | 18:10 | |
| fungi | that will potentially need to be addressed in the prep script | 18:12 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Improve tripleo nodepool image build efficiency. https://review.openstack.org/67255 | 18:12 |
| HenryG | For everyone's information, I found that an extension I had in Chrome was automatically adding lines (about 1 per second) to any etherpad that I had open in my browser. :( | 18:12 |
| HenryG | The offending extension is ¡ÅççéñţšPłüş! (AccentsPlus) | 18:12 |
| fungi | lifeless: pleia2: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/nodepool/scripts/prepare_tripleo.sh | 18:13 |
| fungi | HenryG: awesome! glad you all figured out who it was | 18:13 |
| anteaya | HenryG: thank you for telling us | 18:13 |
| *** fbo is now known as fbo_away | 18:13 | |
| HenryG | We should warn people not to use this extension. | 18:13 |
| *** harlowja has joined #openstack-infra | 18:14 | |
| *** vkozhukalov has joined #openstack-infra | 18:14 | |
| *** gokrokve has quit IRC | 18:14 | |
| *** mattoliverau has joined #openstack-infra | 18:14 | |
| *** gokrokve has joined #openstack-infra | 18:14 | |
| *** dims has quit IRC | 18:15 | |
| lifeless | fungi: thats too low level | 18:16 |
| *** jerryz has joined #openstack-infra | 18:16 | |
| lifeless | fungi: its the nova api calls that have to alter | 18:16 |
| pleia2 | hmm | 18:16 |
| fungi | lifeless: how are you expecting it to blow up if the vm has more than one nic? | 18:17 |
| fungi | i think our other providers give us multi-homed virtual machines too | 18:17 |
| lifeless | fungi: nova boot without a specified list of nics will error | 18:17 |
| zaro | **** review-dev.o.o will be up and down for testing today **** | 18:18 |
| fungi | zaro: thanks! | 18:18 |
| fungi | lifeless: ahh, i wonder why that doesn't error for our other providers. feature of newer nova api? | 18:19 |
| lifeless | fungi: you're not using multiple nics | 18:19 |
| clarkb | morning | 18:19 |
| *** gokrokve has quit IRC | 18:19 | |
| clarkb | sorry just got roped into an hp thing I didnt see on calendar until now... | 18:19 |
| lifeless | fungi: floating IPs are nat, not additional networks you can choose to use | 18:19 |
| fungi | lifeless: i guess i don't get what you mean by "using" in this case. the virtual machines seem to boot fine with configuration on more than one vnic | 18:20 |
| clarkb | will be work working after lunch I hope | 18:20 |
| fungi | clarkb: morning | 18:20 |
| lifeless | fungi: I don't see a running vm from you; just the two we've brought up ourselves | 18:20 |
| fungi | lifeless: no, i mean on our other providers. we have virtual machines with configuration on eth0 and eth1 | 18:21 |
| *** gokrokve has joined #openstack-infra | 18:21 | |
| lifeless | fungi: hmm; and you've not explicitly asked for that? | 18:21 |
| fungi | lifeless: rackspace virtual machines (i just checked) come up by default with both an eth0 and an eth1 and we don't have to pass nic options to nova boot to tell it about that | 18:21 |
| fungi | but i'll accept that it may just be a question of how their nova is configured | 18:22 |
| lifeless | fungi: verra strange. Nova trunk will barf if there is more than one network and you haven't selected what nics you want. | 18:22 |
| fungi | very well might be they're running older nova in tat case | 18:23 |
| *** krotscheck has quit IRC | 18:23 | |
| lifeless | fungi: no; its been like this for a long time | 18:23 |
| fungi | i assume you're talking about the --nic <net-id=net-uuid,v4-fixed-ip=ip-addr,port-id=port-uuid> | 18:24 |
| lifeless | yes | 18:24 |
| *** vipul is now known as vipul-away | 18:25 | |
| *** _ruhe is now known as ruhe | 18:25 | |
| lifeless | oh | 18:25 |
| lifeless | I bet they are running nova-network | 18:25 |
| lifeless | not neutron | 18:25 |
| fungi | on rackspace when we nova boot, we get a vm with two interfaces. nova list refers to one as "public" and one as "private" (i hate those terms) | 18:26 |
| fungi | ahh, good point! | 18:26 |
| *** CaptTofu has quit IRC | 18:26 | |
| *** gokrokve has quit IRC | 18:26 | |
| fungi | lifeless: perhaps we need SergeyLukjanov_'s awesome patch? https://review.openstack.org/66394 | 18:27 |
| *** gokrokve has joined #openstack-infra | 18:27 | |
| *** beagles has quit IRC | 18:27 | |
| *** oubiwann has quit IRC | 18:27 | |
| lifeless | fungi: possibly | 18:27 |
| fungi | it seems to deal explicitly with nic selection | 18:28 |
| lifeless | fungi: with nova cli its just nova boot --flavor m1.large --image "Ubuntu Precise 12.04 LTS Server 64-bit" --key-name lifeless test-slave --nic net-id=854b0c8c-6665-4f0c-a2f3-bd94c458c98b --nic net-id=d452f993-2fd0-432d-9d03-6e4b57f6147b | 18:28 |
| lifeless | note the use of *two* nic parameters | 18:28 |
| *** markmc has quit IRC | 18:28 | |
| *** loquacities has quit IRC | 18:28 | |
| *** wayneeseguin has quit IRC | 18:28 | |
| *** primeministerp has quit IRC | 18:28 | |
| fungi | lifeless: SergeyLukjanov_: okay, so perhaps we need to be able to pass a list there | 18:29 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Configure eth1 for DHCP in tripleo-gate instances https://review.openstack.org/67260 | 18:29 |
| fungi | i'm betting his environment had just one neutron-managed network connected to the vm | 18:29 |
| lifeless | so with one neutron manageed network you don't need his patch | 18:30 |
| lifeless | just boot, default will DTRT | 18:30 |
| mtreinish | fungi: I found a 3rd party CI leaving build started messages: https://review.openstack.org/#/c/66970/ | 18:30 |
| *** primeministerp has joined #openstack-infra | 18:31 | |
| mtreinish | although it's only one | 18:31 |
| fungi | mtreinish: yeah, they just had me reset their ssh key for them earlier today | 18:31 |
| *** dims has joined #openstack-infra | 18:31 | |
| fungi | it was IzikP_Radware but he seems to be gone now | 18:31 |
| mtreinish | off to a good start I guess... | 18:31 |
| *** gokrokve has quit IRC | 18:31 | |
| fungi | also it's leaving urls to an unreachable system | 18:32 |
| *** b3nt_pin has joined #openstack-infra | 18:32 | |
| *** yassine has quit IRC | 18:33 | |
| *** bknudson has quit IRC | 18:33 | |
| *** wayneeseguin has joined #openstack-infra | 18:33 | |
| *** dizquierdo has quit IRC | 18:33 | |
| *** loquacities has joined #openstack-infra | 18:34 | |
| *** b3nt_pin is now known as beagles | 18:35 | |
| *** beagles is now known as beagles_brb | 18:35 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Add info about browser incompatibility to etherpad.rst https://review.openstack.org/67262 | 18:35 |
| anteaya | HenryG: ^ | 18:36 |
| fungi | anteaya: thanks for that | 18:36 |
| *** dims has quit IRC | 18:36 | |
| HenryG | anteaya: thanks | 18:37 |
| *** oubiwann has joined #openstack-infra | 18:38 | |
| *** mrodden1 has joined #openstack-infra | 18:39 | |
| openstackgerrit | A change was merged to openstack-infra/config: Adding configuration for mistral-extra project https://review.openstack.org/66443 | 18:39 |
| anteaya | fungi: markmcclain is in favour of revoking that ssh key from radware | 18:40 |
| anteaya | fungi: welcome | 18:40 |
| *** jcoufal has joined #openstack-infra | 18:40 | |
| anteaya | the tolerance for 3rd party cruft is diminishing rapidly | 18:40 |
| *** mrodden has quit IRC | 18:41 | |
| *** harlowja_ has joined #openstack-infra | 18:41 | |
| *** harlowja has quit IRC | 18:41 | |
| *** nati_ueno has joined #openstack-infra | 18:42 | |
| *** markmc has joined #openstack-infra | 18:42 | |
| *** CaptTofu has joined #openstack-infra | 18:42 | |
| *** NikitaKonovalov has joined #openstack-infra | 18:42 | |
| *** markwash has quit IRC | 18:44 | |
| *** harlowja has joined #openstack-infra | 18:45 | |
| *** ruhe is now known as ruhe_away | 18:45 | |
| mtreinish | fungi: did you see the followup message from Radware? | 18:46 |
| mtreinish | "<LOG_URL>" | 18:46 |
| *** harlowja_ has quit IRC | 18:46 | |
| *** markwash has joined #openstack-infra | 18:46 | |
| fungi | mtreinish: i do now. that's pretty awful | 18:47 |
| *** odyssey4me has joined #openstack-infra | 18:47 | |
| *** NikitaKonovalov has quit IRC | 18:47 | |
| fungi | anteaya: i'll fire off a quick heads up to the person who was asking for the ssh key update and let them know i've been asked to temporarily revoke all access to their service account | 18:48 |
| *** ruhe_away is now known as ruhe | 18:48 | |
| *** nati_ueno has quit IRC | 18:48 | |
| *** nati_ueno has joined #openstack-infra | 18:49 | |
| *** rcleere has joined #openstack-infra | 18:50 | |
| *** NikitaKonovalov has joined #openstack-infra | 18:50 | |
| *** vipul-away is now known as vipul | 18:50 | |
| *** dims has joined #openstack-infra | 18:52 | |
| *** openstackgerrit has quit IRC | 18:52 | |
| *** openstackgerrit has joined #openstack-infra | 18:53 | |
| *** carl_baldwin has joined #openstack-infra | 18:54 | |
| *** coolsvap has quit IRC | 18:54 | |
| *** bknudson has joined #openstack-infra | 18:54 | |
| fungi | mtreinish: anteaya: markmcclain: http://lists.openstack.org/pipermail/openstack-infra/2014-January/000654.html | 18:55 |
| markmcclain | fungi: thanks | 18:56 |
| *** NikitaKonovalov has quit IRC | 18:56 | |
| *** markwash has quit IRC | 18:56 | |
| anteaya | fungi: thank you | 18:57 |
| lifeless | pleia2: in the nodepool README.rst it grants privileges to nodepool@localhost; should that be `whoami`? or is it hardcoded ? | 18:58 |
| *** hashar has joined #openstack-infra | 18:59 | |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: tests: Allow to test maven project parameters https://review.openstack.org/67265 | 19:00 |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: project_maven: Don't require artifact-id and group-id https://review.openstack.org/66036 | 19:00 |
| *** markwash has joined #openstack-infra | 19:00 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:00 | |
| *** luqas has quit IRC | 19:01 | |
| anteaya | 700 events now in zuul, better than 900 | 19:01 |
| anteaya | yay progress | 19:01 |
| lifeless | fungi: is nodepool project == tenant ? | 19:02 |
| *** luqas has joined #openstack-infra | 19:02 | |
| fungi | lifeless: it seems to correspond to the OS_TENANT_NAME envvar | 19:02 |
| lifeless | thanks | 19:03 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 19:03 | |
| *** derekh has joined #openstack-infra | 19:03 | |
| *** mrodden has joined #openstack-infra | 19:03 | |
| *** mrodden1 has quit IRC | 19:03 | |
| *** hashar has quit IRC | 19:04 | |
| *** NikitaKonovalov has quit IRC | 19:05 | |
| *** vipul is now known as vipul-away | 19:05 | |
| lifeless | right | 19:06 |
| lifeless | BadRequest: Multiple possible networks found, use a Network ID to be more specific. (HTTP 400) (Request-ID: req-610cf435-3005-4f5f-ae2d-3dc43725eb8a) | 19:06 |
| lifeless | ^ thats what nodepool does atm | 19:07 |
| lifeless | working up a fix | 19:07 |
| *** NikitaKonovalov has joined #openstack-infra | 19:07 | |
| fungi | lifeless: noted. and 66394 doesn't fix that? | 19:07 |
| *** NikitaKonovalov has quit IRC | 19:11 | |
| anteaya | there is a horizon patch failing in post | 19:11 |
| lifeless | fungi: well it would, but we need two nics | 19:12 |
| lifeless | fungi: that being the point of the config :) | 19:12 |
| *** NikitaKonovalov has joined #openstack-infra | 19:12 | |
| fungi | lifeless: got it, so we need the yaml to be able to take a list there (and adjust the option handler to do the right thing with it) | 19:12 |
| kiall | Hey guys - I couldn't convince someone with gerrit admin to take a look at https://bugs.launchpad.net/openstack-ci/+bug/1268570 for me? | 19:13 |
| uvirtbot | Launchpad bug 1268570 in openstack-ci "Service Account Request" [Undecided,New] | 19:13 |
| fungi | kiall: saw the request, just haven't had time to grab it yet | 19:13 |
| kiall | Ah - Fair enough so :) I wasn't sure it was seen.. No worries then | 19:14 |
| mfer | fungi hi. | 19:14 |
| fungi | hi mfer | 19:15 |
| mfer | fungi i was wondering if you'd heard back about the naming pattern for openstack sdks? | 19:15 |
| *** wenlock has quit IRC | 19:15 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add uncategorized failure generation code https://review.openstack.org/67267 | 19:15 |
| *** herndon has quit IRC | 19:15 | |
| *** NikitaKonovalov has quit IRC | 19:16 | |
| *** luqas has quit IRC | 19:16 | |
| sdague | fungi: 66869 is a target for promote | 19:17 |
| sdague | I just approved it | 19:17 |
| sdague | it brings in enhanced console logging for when guests fail to start in the gate, so hopefully we can figure out why they do | 19:17 |
| *** jooools has joined #openstack-infra | 19:17 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:18 | |
| *** luqas has joined #openstack-infra | 19:18 | |
| carl_baldwin | Hi openstack-infra, I would like to discuss whether there is potential to run the gate VMs with some kernel fixes (would be back-ported from 3.8 kernel) that I have found improve the performance of running commands in network namespaces: something that neutron does frequently. | 19:18 |
| carl_baldwin | http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=665e205c16c1f902ac6763b8ce8a0a3a1dcefe59 | 19:18 |
| fungi | mfer: not yet. i e-mailed the foundation ceo on tuesday after we talked, since he wanted specific details before deciding whether to check with our legal counsel | 19:18 |
| carl_baldwin | http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=32263dd1b43378b4f7d7796ed713f77e95f27e8a | 19:18 |
| *** sdake has joined #openstack-infra | 19:18 | |
| carl_baldwin | ^ Canonical has back ported these to their 3.5.0-46 kernel package but they don't seem to have back ported them to the 3.2.0 stream. | 19:19 |
| SergeyLukjanov | fungi, lifeless, I see you talked about https://review.openstack.org/#/c/66394/ | 19:19 |
| fungi | sdague: the last change i needed to promote took 2 hours from approval to showing up in the gate and then another hour before the reset from the promote completed for testing to start on it (still being tested now, ~30 minutes to go) | 19:20 |
| SergeyLukjanov | I'm really looking for any feedback | 19:20 |
| lifeless | SergeyLukjanov: I'm about to push an improved version if thats ok with you | 19:20 |
| fungi | sdague: so expect it to not be ready right away | 19:20 |
| SergeyLukjanov | lifeless, sure, I could re-iterate on it if you want | 19:21 |
| SergeyLukjanov | lifeless, is it needed for tripleo cluster support? | 19:21 |
| fungi | carl_baldwin: short answer is that we probably can if the kernels are officially security-support maintained in ubuntu precise-backports, or soon possibly in ubuntu cloud archive for precise (12.04 lts) | 19:21 |
| lifeless | SergeyLukjanov: yes | 19:21 |
| *** NikitaKonovalov has quit IRC | 19:22 | |
| SergeyLukjanov | lifeless, btw we're using this patching for savanna-ci, so, it's tested in semi-prod | 19:22 |
| fungi | SergeyLukjanov: short reason is the need for it to handle multiple network ids | 19:22 |
| SergeyLukjanov | patch* | 19:22 |
| fungi | (booting a server multi-homed on more than one nettron network) | 19:23 |
| SergeyLukjanov | fungi, yup, I've planned to improve it later, but looks like lifeless will do it earlier ;) | 19:23 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Permit specifying instance networks to use. https://review.openstack.org/66394 | 19:23 |
| fungi | s/nettron/neutron/ | 19:23 |
| *** NikitaKonovalov has joined #openstack-infra | 19:23 | |
| lifeless | SergeyLukjanov: cool - ^ its a generalised version of it | 19:23 |
| *** luqas has quit IRC | 19:23 | |
| carl_baldwin | fungi: The 3.5.0 version should be in the supported channels but the 3.2.0 back port doesn't exist yet in any channel afaik. | 19:23 |
| *** esker has quit IRC | 19:24 | |
| lifeless | SergeyLukjanov: in this version you just list the network details | 19:24 |
| lifeless | like | 19:24 |
| lifeless | networks: | 19:24 |
| lifeless | - net-id: '854b0c8c-6665-4f0c-a2f3-bd94c458c98b' | 19:24 |
| lifeless | - net-id: 'd452f993-2fd0-432d-9d03-6e4b57f6147b' | 19:24 |
| lifeless | no use-neutron setting (if networks is set, we set it, if its not, we don't) | 19:24 |
| SergeyLukjanov | lifeless, awesome, thank you for tip and fix | 19:24 |
| fungi | carl_baldwin: i suspect we would be able to upgrade to 3.5.0 as long as it's in precise-backports or uca | 19:25 |
| SergeyLukjanov | lifeless, correct, just my personal preference to explicitly specify some important stuff | 19:25 |
| lifeless | if you want to control the ip v4 address etc, you just add that into the dicts | 19:25 |
| lifeless | SergeyLukjanov: sure, so the reason I took the explicit setting out is that its actually incorrect | 19:25 |
| SergeyLukjanov | I had some experience of coding on Scala and I hate anything implicit after that, heh ;) | 19:25 |
| lifeless | SergeyLukjanov: since this is all about how we request things from nova | 19:25 |
| lifeless | its not that I object to explicit; its that it wasn't accurate | 19:26 |
| jog0 | mordred: here is another patch that I think should be automated https://review.openstack.org/#/c/67053/1 | 19:26 |
| fungi | carl_baldwin: otherwise, we're probably looking at trusty/14.04 lts in the openstack juno development cycle | 19:26 |
| jog0 | mordred: re: global reqs | 19:26 |
| SergeyLukjanov | lifeless, agreed | 19:26 |
| *** harlowja has quit IRC | 19:26 | |
| *** resker has joined #openstack-infra | 19:26 | |
| *** odyssey4me has quit IRC | 19:27 | |
| lifeless | fungi: so that patch works for me; can we land it and the config change please ?:) | 19:28 |
| lifeless | | 10fd9f71-2feb-4da8-92d3-bc9f01511a71 | precise-1389900310.template.openstack.org | ACTIVE | None | Running | default-net=10.0.0.9, 138.35.77.21; tripleo-bm-test=192.168.1.25 | | 19:29 |
| carl_baldwin | fungi: What next step would you recommend I take to get the ball rolling? (I understand your team is running a bit thin at the moment) | 19:29 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: only run on openstack gate projects https://review.openstack.org/67273 | 19:30 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs https://review.openstack.org/66565 | 19:30 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: move to static LOG https://review.openstack.org/66564 | 19:30 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: create more sane logging for the er bot https://review.openstack.org/66435 | 19:30 |
| *** NikitaKonovalov has quit IRC | 19:30 | |
| *** gyee_ has quit IRC | 19:31 | |
| *** harlowja has joined #openstack-infra | 19:31 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added route tests for authentication https://review.openstack.org/66911 | 19:32 |
| *** beagles_brb is now known as beagles | 19:33 | |
| fungi | lifeless: nova network-list gives me three networks there. which two do we want? (ext-net, default-net, tripleo-bm-test) | 19:34 |
| fungi | lifeless: sorry if i missed it when you said earlier or something | 19:34 |
| *** derekh has quit IRC | 19:34 | |
| *** odyssey4me has joined #openstack-infra | 19:35 | |
| lifeless | fungi: | 19:35 |
| lifeless | 08:24 < lifeless> networks: | 19:35 |
| lifeless | 08:24 < lifeless> - net-id: '854b0c8c-6665-4f0c-a2f3-bd94c458c98b' | 19:35 |
| lifeless | 08:24 < lifeless> - net-id: 'd452f993-2fd0-432d-9d03-6e4b57f6147b' | 19:35 |
| fungi | default-net and tripleo-bm-test i guess | 19:35 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs https://review.openstack.org/66565 | 19:35 |
| *** odyssey4me has quit IRC | 19:35 | |
| lifeless | fungi: which is default-net and tripleo-bm-test, yes | 19:35 |
| fungi | okay, i see those interface labels in your nova list above | 19:35 |
| *** NikitaKonovalov has joined #openstack-infra | 19:35 | |
| lifeless | fungi: (nova takes uuids, not names) | 19:35 |
| fungi | right | 19:35 |
| fungi | i'll get 66491 updated with that real quick | 19:36 |
| *** odyssey4me has joined #openstack-infra | 19:36 | |
| *** nati_uen_ has joined #openstack-infra | 19:37 | |
| *** praneshp has quit IRC | 19:37 | |
| *** dkliban is now known as dkliban_afk | 19:37 | |
| *** elasticio has quit IRC | 19:38 | |
| *** nati_uen_ has quit IRC | 19:38 | |
| *** sdake has quit IRC | 19:38 | |
| *** sdake has joined #openstack-infra | 19:38 | |
| *** sdake has quit IRC | 19:38 | |
| *** sdake has joined #openstack-infra | 19:38 | |
| *** dcramer_ has quit IRC | 19:38 | |
| *** praneshp has joined #openstack-infra | 19:38 | |
| *** nati_uen_ has joined #openstack-infra | 19:39 | |
| lifeless | pleia2: / mordred: so when I test nodepool locally, and something goes wrong, HTF can I tell what went wrong? I mean, it spawns ok but is deleting the node and redoing it subsequently | 19:39 |
| *** NikitaKonovalov has quit IRC | 19:39 | |
| fungi | lifeless: nodepool.log is where i end up looking | 19:40 |
| fungi | or image.log if it's cycling on image creation still | 19:40 |
| *** odyssey4me has quit IRC | 19:40 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:41 | |
| *** nati_ueno has quit IRC | 19:41 | |
| *** marun has quit IRC | 19:43 | |
| lifeless | fungi: ah, this has it all on stdout | 19:43 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Update TripleO Cloud API endpoint for Nodepool https://review.openstack.org/66491 | 19:43 |
| *** marun has joined #openstack-infra | 19:43 | |
| zaro | fungi: testing gerrit upgrade script, but got a question on one part. | 19:44 |
| fungi | zaro: what's that? | 19:44 |
| zaro | fungi: https://etherpad.openstack.org/p/gerrit_upgrade_script | 19:44 |
| zaro | fungi: last step there using the set-members command to users to group. | 19:44 |
| zaro | fungi: this assumes the user that running the script is a gerrit admin with ssh key setup | 19:45 |
| zaro | fungi: apparently it doesn't work for me even though i've setup my ssh key on gerrit. don't know why. it says permissioned denied. | 19:45 |
| fungi | zaro: i think you need to be a member of a group which is an owner of that group | 19:46 |
| fungi | zaro: have you checked that? | 19:46 |
| ruhe | lifeless: the problem might be caused by the authorized_keys which end up in the nodes built by nodepool | 19:46 |
| ruhe | modules/openstack_project/manifests/init.pp has hardcoded public key which is probably different from what you have in your setup | 19:46 |
| zaro | fungi: i can add users using the UI, no problem. | 19:47 |
| lifeless | ruhe: I exported NODEPOOL_SSH_KEY | 19:47 |
| *** gokrokve has joined #openstack-infra | 19:48 | |
| *** odyssey4me has joined #openstack-infra | 19:48 | |
| zaro | fungi: it's not gerrit at all. the error i get is an ssh thing. it just says my key is not valid. | 19:48 |
| fungi | lifeless: ruhe: possibly related to https://review.openstack.org/62066 ? | 19:48 |
| ruhe | lifeless: a have a feeling that it might not work; i'm testing the same thing right now | 19:48 |
| ruhe | fungi: that's exactly what i need! | 19:49 |
| *** dkliban_afk is now known as dkliban | 19:49 | |
| fungi | zaro: your public ssh key and the username specified on the command line correspond to the right account? can you run any ssh api commands against it with that key/username? ls-projects for example? | 19:49 |
| ruhe | fungi: but still it exports NODEPOOL_SSH_KEY; let me try it; i'll get back with results | 19:50 |
| fungi | carl_baldwin: it looks like precise-updates has linux-image-3.5.0-18-generic==3.5.0-18.29~precise1 for amd64. will that do what you're thinking we need? | 19:50 |
| *** dcramer_ has joined #openstack-infra | 19:51 | |
| fungi | ruhe: lifeless: that was a patch to support us setting up a separate nodepool instance running on our jenkins-dev server | 19:51 |
| fungi | since we needed to be able to override the ssh key | 19:52 |
| *** NikitaKonovalov has quit IRC | 19:52 | |
| *** krotscheck has joined #openstack-infra | 19:52 | |
| lifeless | ruhe: so the key in init.pp sets the ssh key for jenkins | 19:52 |
| *** hogepodge has joined #openstack-infra | 19:52 | |
| lifeless | ruhe: I don't believe it replaces the root/ubuntu ssh key for admins | 19:52 |
| zaro | fungi: maybe i'm getting the sytax wrong. what should i specify for the host? | 19:54 |
| *** vipul-away is now known as vipul | 19:54 | |
| mordred | fungi: wow. we should probably be running precise-updates anyway | 19:54 |
| lifeless | fungi: thats without using NODEPOOL_SSH_KEY though | 19:54 |
| zaro | fungi: i tried localhost which repsonded with permission denied (publickey) | 19:54 |
| fungi | carl_baldwin: oh, actually linux-image-3.5.0-45-generic==3.5.0-45.68~precise1 is the latest 3.5 in precise-updates | 19:54 |
| fungi | zaro: did you pass the gerrit service port number with -p? | 19:55 |
| zaro | fungi: ohh, hold on. my account on the server doesn't match the one on gerrit. sorry. let me try again. | 19:55 |
| fungi | zaro: ssh -p 29418 username@localhost gerrit ls-projects | 19:56 |
| fungi | zaro: and that assumes that the .ssh/id_rsa.pub in the calling account on that server matches what's associated with the username you're trying to authenticate as in gerrit | 19:56 |
| *** NikitaKonovalov has joined #openstack-infra | 19:57 | |
| SergeyLukjanov | fungi, mordred, could you please take a look on a patch that adds savanna-ci-config repo? https://review.openstack.org/#/c/66521/ | 19:57 |
| SergeyLukjanov | probably I missed something, but is everything ok with zuul? Queue lengths: 822 events, 973 results. | 19:59 |
| mordred | SergeyLukjanov: it's ... behind | 19:59 |
| SergeyLukjanov | I see the same the whole day today | 19:59 |
| *** CaptTofu has quit IRC | 19:59 | |
| mordred | the event rate is coming in as fast as zuul can process the events | 19:59 |
| *** dcramer_ has quit IRC | 20:00 | |
| mordred | fungi: the big cost in event processing is the api callback to gerrit, right? | 20:00 |
| SergeyLukjanov | mordred, looks like zuul trying to create a time machine and enslave the world :) | 20:00 |
| mordred | SergeyLukjanov: it is | 20:00 |
| mordred | :) | 20:00 |
| *** hashar has joined #openstack-infra | 20:00 | |
| mordred | fungi: because it needs to call back to check to see if an aprv'd patch is 'ready for merge' and also to get the patch history and make sure each of those is 'ready for merge' right? | 20:01 |
| carl_baldwin | fungi: There are more recent kernels available in precise-updates. Up to linux-image-3.5.0-45-generic is available. | 20:01 |
| *** resker has quit IRC | 20:01 | |
| *** NikitaKonovalov has quit IRC | 20:02 | |
| SergeyLukjanov | mordred, I think yes, it's an api callback | 20:02 |
| *** esker has joined #openstack-infra | 20:02 | |
| SergeyLukjanov | mordred, through the cli - https://github.com/openstack-infra/zuul/blob/master/zuul/lib/gerrit.py#L115 | 20:02 |
| carl_baldwin | However, now that I'm looking for the package with these patches, it appears that they will be in the next version (-46): http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-precise.git;a=commit;h=Ubuntu-lts-3.5.0-46.70 | 20:02 |
| mordred | fungi: I wonder if it would be worth throwing zaro to the wolves and having him write a thing inside of gerrit that emits a new event when all of those things have been met | 20:02 |
| *** NikitaKonovalov has joined #openstack-infra | 20:02 | |
| *** odyssey4me has quit IRC | 20:02 | |
| fungi | mordred: i wonder if a gerrit plugin would be able to do that | 20:03 |
| anteaya | zaro: if you want to create a user for me with username anteaya and ssh key: http://paste.openstack.org/show/61391/ I can try too | 20:05 |
| *** esker has quit IRC | 20:06 | |
| *** mrodden has quit IRC | 20:07 | |
| *** wenlock has joined #openstack-infra | 20:07 | |
| *** mrodden has joined #openstack-infra | 20:08 | |
| *** luqas has joined #openstack-infra | 20:09 | |
| *** odyssey4me has joined #openstack-infra | 20:10 | |
| *** jooools has quit IRC | 20:10 | |
| *** nati_ueno has joined #openstack-infra | 20:11 | |
| *** wenlock has quit IRC | 20:12 | |
| *** dcramer_ has joined #openstack-infra | 20:13 | |
| notmyname | fungi: https://jenkins01.openstack.org/job/gate-tempest-dsvm-postgres-full/ hasn't successfully run in nearly 6 hours | 20:14 |
| fungi | notmyname: exciting | 20:14 |
| *** krtaylor has quit IRC | 20:14 | |
| *** nati_uen_ has quit IRC | 20:15 | |
| fungi | sdague: mtreinish: anybody else in qa: interested in looking into gate-tempest-dsvm-postgres-full being completely broken? | 20:15 |
| mtreinish | fungi: it's completely broken? | 20:15 |
| *** nati_ueno has quit IRC | 20:15 | |
| fungi | mtreinish: notmyname says it hasn't run successfully in the last 6 hours | 20:16 |
| notmyname | mtreinish: see the link above: Last successful build (#5672), 5 hr 47 min ago | 20:16 |
| fungi | though https://jenkins02.openstack.org/job/gate-tempest-dsvm-postgres-full/ has a successful run from 4 hours ago | 20:17 |
| mtreinish | notmyname: that's just on one jenkins | 20:17 |
| *** jooools has joined #openstack-infra | 20:18 | |
| fungi | oh, the successful one from 4 hours ago was a stable/havana run | 20:18 |
| *** dims has quit IRC | 20:18 | |
| notmyname | fungi: is there a place to see aggregated stats across all jenkins boxes? | 20:19 |
| notmyname | fungi: ie for exactly this question | 20:19 |
| *** NikitaKonovalov has quit IRC | 20:19 | |
| fungi | notmyname: closest thing we have is the statsd reporting back to graphite.o.o | 20:19 |
| notmyname | ok | 20:20 |
| zaro | mordred: what could be worse than wip? | 20:20 |
| notmyname | fungi: ok. I'm familiar with that :-) looks pretty bad right now http://not.mn/gate_status.html | 20:20 |
| zaro | fungi: gerrit command works now. | 20:20 |
| *** AaronGr is now known as aarongr_afk | 20:20 | |
| fungi | notmyname: agreed | 20:21 |
| fungi | zaro: excellent | 20:21 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add project for savanna-ci configs https://review.openstack.org/66521 | 20:22 |
| *** jooools has quit IRC | 20:23 | |
| *** yolanda has quit IRC | 20:23 | |
| fungi | i should not have looked at the openstack-dev ml either. i apparently have a thousand unread messages from it | 20:23 |
| anteaya | yeah, don't look at that | 20:24 |
| anteaya | anything really important someone will let you know | 20:24 |
| *** NikitaKonovalov has joined #openstack-infra | 20:24 | |
| fungi | thanks | 20:24 |
| mattoliverau | Morning all! | 20:26 |
| SlickNik | Hey guys! | 20:29 |
| SlickNik | Could you please review https://review.openstack.org/#/c/65040/, and https://review.openstack.org/#/c/65065/ when you get a chance? | 20:29 |
| SlickNik | They're both related to changes regarding being able to test trove with tempest. Thanks! | 20:29 |
| *** jooools has joined #openstack-infra | 20:29 | |
| *** dcramer_ has quit IRC | 20:30 | |
| akscram | hi guys, please pay attention to the change https://review.openstack.org/#/c/66929/ | 20:31 |
| anteaya | mattoliverau: hey there | 20:32 |
| anteaya | so glad you are here | 20:32 |
| openstackgerrit | A change was merged to openstack-infra/askbot-theme: fixed a bug of misused function call https://review.openstack.org/67197 | 20:32 |
| openstackgerrit | A change was merged to openstack-infra/askbot-theme: some css changes https://review.openstack.org/67198 | 20:32 |
| openstackgerrit | A change was merged to openstack-infra/askbot-theme: added the launchpad importer code https://review.openstack.org/67251 | 20:32 |
| anteaya | akscram: what you do mean by pay attention? | 20:32 |
| anteaya | are you asking for a review? | 20:33 |
| akscram | anteaya: yep, please review | 20:33 |
| anteaya | okay, note that fungi is alone right now and this list of tasks is large | 20:35 |
| anteaya | patience is appreciated | 20:35 |
| zaro | fungi: another question. | 20:35 |
| notmyname | fungi: FYI, I also now have http://not.mn/nova_gate_status.html and http://not.mn/neutron_gate_status.html for their specific tests graphed independently of the 6 common gate jobs | 20:35 |
| zaro | fungi: the gerrit upgrade script executes sudo which requires manually entering password. what do you do to not require that? | 20:36 |
| *** melwitt has joined #openstack-infra | 20:36 | |
| *** odyssey4me has quit IRC | 20:37 | |
| jog0 | fungi: whats your thoughts on not running non-voting jobs in the gate? | 20:37 |
| anteaya | notmyname: so you are calculating a 100% chance of a patch passing nova gate and a 0% chance of a patch passing neutron gate | 20:37 |
| anteaya | is that correct? | 20:37 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Augment the e-r query for bug 1258682 https://review.openstack.org/67094 | 20:38 |
| uvirtbot | Launchpad bug 1258682 in openstack-ci "timeout causing gate-tempest-dsvm-full to fail" [High,In progress] https://launchpad.net/bugs/1258682 | 20:38 |
| fungi | zaro: on our servers we have it set to not require a password | 20:38 |
| notmyname | anteaya: yes, for the most recent 12 hour time bucket. if things start passing, that number will go up | 20:38 |
| anteaya | mattoliverau: do you have things to work on, can I help you with anything? | 20:38 |
| fungi | notmyname: cool (or unfortunate depending on how you look at it) | 20:38 |
| anteaya | notmyname: can we get any more detail on which test or jobs are failing for neutron? | 20:38 |
| fungi | jog0: i usually omit them unless they're gate-specific jobs for some particular reason | 20:38 |
| anteaya | sitting here in a room of neutron folks, it would be helpful for us to know what to look at | 20:39 |
| fungi | jog0: other possible reason it to collect gate-specific results on them | 20:39 |
| notmyname | anteaya: looking at the tiny colored lines? ;-) | 20:39 |
| anteaya | I'm trying | 20:39 |
| fungi | jog0: (i.e. not polluted by people testing their proposed changes) | 20:39 |
| zaro | fungi: ok. how about for the gerrit ssh command? when i run it asks me to enter my passphrase? | 20:39 |
| EmilienM | dtroyer: ping | 20:39 |
| *** mrodden1 has joined #openstack-infra | 20:39 | |
| anteaya | the thick red line covers a lot of them | 20:39 |
| *** mrda has joined #openstack-infra | 20:39 | |
| jog0 | fungi: cool so we can remove them on a case by case basis, thanks | 20:40 |
| anteaya | so is that the python27 jobs? | 20:40 |
| dtroyer | EmilienM: yo | 20:40 |
| *** dims has joined #openstack-infra | 20:40 | |
| fungi | zaro: probably on the server for local connections we could authenticate with the magic "Gerrit Code Review" ssh account which recognizes the gerrit ssh host key as the authentication key | 20:40 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Log how long it takes to run a query when collecting metrics https://review.openstack.org/65514 | 20:40 |
| fungi | zaro: unless they've taken that out since 2.4 | 20:40 |
| notmyname | anteaya: looks like py27 on my screen | 20:41 |
| notmyname | anteaya: FWIW the red line is actually graphed behind the others, but ya it's hard to read details, even on my high-res screen | 20:41 |
| zaro | fungi: let me try to find it. | 20:42 |
| *** sarob has joined #openstack-infra | 20:42 | |
| anteaya | notmyname: okay so neutron python27 jobs took a recent turn for the worse, according to your graph | 20:43 |
| *** mrodden has quit IRC | 20:43 | |
| notmyname | anteaya: yes | 20:43 |
| markmcclain | notmyname: re the neutron graphs | 20:44 |
| *** odyssey4me has joined #openstack-infra | 20:44 | |
| fungi | notmyname: anteaya: the main reason for the red line sitting at 0 a good chunk of today is probably because of the event queue throughput being at capacity. it's causing a massive swing between servicing all check jobs and then servicing the gate following each reset, so i basically waited two hours for one change to come through the event queue and get added to the gate, promoted it (which trigger a | 20:44 |
| notmyname | anteaya: it's important to note that the graph is only data, not interpretation. eg many (not all) of the failures on the 9th were due to the jenkins issues last week, I think | 20:44 |
| fungi | gate reset) then waited another hour or more for it to get assigned workers, then an hour to test it, then it failed and got dequeued, and the gate is still waiting on new workers for the change which was immediately behind it | 20:44 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add uncategorized failure generation code https://review.openstack.org/67267 | 20:44 |
| notmyname | markmcclain: what's up? | 20:45 |
| jog0 | fungi: so we want to better track the infra failures so we can clearly track unclassified failures | 20:45 |
| jog0 | so we want to classify something like http://logs.openstack.org/00/62300/4/gate/gate-nova-pep8/6083f02/console.html | 20:45 |
| *** dkliban is now known as dkliban_afk | 20:45 | |
| jog0 | should we have a catch all infra failure bug or do them separately? | 20:45 |
| notmyname | fungi: partially, yes. but since I'm using 12 hour graphs it would have to be a _significant_ -infra issue to not allow anything to pass | 20:46 |
| openstackgerrit | A change was merged to openstack-infra/config: Create gate for api-site https://review.openstack.org/64795 | 20:46 |
| fungi | jog0: i think it depends on how we intend to treat those bugs. they're almost always going to be situation-specific recurrent events. if we treat them as actual infra bugs we're incented to close them once the event and its fallout are resolved, but it sounds like you need a persistent bucket for certain classes of events | 20:47 |
| anteaya | fungi: yeah not an optimal situation | 20:47 |
| notmyname | but i'm hoping that we aren't getting mentally anchored (http://en.wikipedia.org/wiki/Anchoring) to something that's too low. I'd like the red line on the top graph to be >95%. that's starting to get close to acceptable | 20:47 |
| jog0 | fungi: take message:"SEVERE ERROR occurs: java.lang.InterruptedException" for example | 20:47 |
| fungi | notmyname: i completely agree. i seriously don't want anyone thinking the current performance is even remotely acceptable | 20:48 |
| jog0 | is that a persistent or should that be its own bug? | 20:48 |
| fungi | jog0: that's like having a bug for "python raises an exception" | 20:49 |
| *** luqas has quit IRC | 20:49 | |
| jog0 | fungi: a severe exception ;) | 20:50 |
| *** pcrews_ has quit IRC | 20:50 | |
| *** luqas has joined #openstack-infra | 20:50 | |
| jog0 | this a little more descriptive message:"[EnvInject] - [ERROR] - SEVERE ERROR occurs: java.lang.InterruptedException" | 20:50 |
| fungi | jog0: for the most part, i can see benefit to categorizing the ways in which various pieces of infrastructure topple over. some of those will get permanently fixed, others may not | 20:50 |
| *** bknudson has quit IRC | 20:50 | |
| zaro | fungi: i can't seem to find that magic ssh account in either the gerrit docs or the gerrit db. | 20:51 |
| jog0 | fungi: so file seperate bugs for now and we can merge em or whatnot later? | 20:51 |
| fungi | jog0: well, that's been the status quo so far | 20:51 |
| jog0 | fungi: thanks | 20:51 |
| fungi | jog0: i'm not saying it's teh right thing to do, just saying we can either track them in elastic-recheck *or* we can treat them as discrete infrastructure events, but not really both | 20:52 |
| fungi | not unless we do both separately anyway | 20:52 |
| *** yamahata has quit IRC | 20:52 | |
| jog0 | fungi: we want to track them in elastic-recheck and can have a catch all bug with a query with many ORs in it. or many infra bugs | 20:53 |
| *** luqas has quit IRC | 20:53 | |
| fungi | jog0: sure. just suggesting that most infra "bugs" will already be in a state we consider resolved/closed by the time someone adds a query for them to elastic-recheck | 20:54 |
| jog0 | fungi: right, thats fine. | 20:54 |
| jog0 | this is just for record keeping | 20:54 |
| jog0 | to identify more bugs in openstack | 20:54 |
| clarkb | fungi: ok I am actually really back now I hope | 20:55 |
| zaro | fungi: could we not just use an ssh agent ? | 20:55 |
| *** soleblaze has joined #openstack-infra | 20:55 | |
| *** homeless has quit IRC | 20:56 | |
| *** homeless has joined #openstack-infra | 20:56 | |
| *** denis_makogon_ has joined #openstack-infra | 20:56 | |
| *** homeless has quit IRC | 20:56 | |
| *** homeless has joined #openstack-infra | 20:56 | |
| fungi | zaro: we could. and/or keychain | 20:57 |
| *** CaptTofu has joined #openstack-infra | 20:57 | |
| fungi | zaro: try this as the gerrit2 user on the server... | 20:58 |
| fungi | ssh -i ~/review_site/etc/ssh_host_rsa_key -l 'Gerrit Code Review' -p 29418 localhost gerrit --help | 20:58 |
| fungi | zaro: that authenticates as the internal master account | 20:58 |
| fungi | (secret gerrit voodoo) | 20:58 |
| fungi | clarkb: my savior! | 20:58 |
| fungi | clarkb: go fix all the things | 20:58 |
| clarkb | fungi: what is broken? | 21:00 |
| fungi | clarkb: just so you don't have to read scrollback, current most annoying situation, which i suspect is not something we'll be fixing this week, is the volume of gerrit events is coming about as fast as zuul can keep up with them. so it has multi-hour swings between servicing all check requests, and then having a shot at the gate | 21:00 |
| clarkb | :/ | 21:00 |
| *** jooools has quit IRC | 21:00 | |
| *** aarongr_afk is now known as AaronGr | 21:00 | |
| clarkb | firefox has decided it should fullscreen me by default. and I can't figure out how to turn it off | 21:00 |
| fungi | then the top change in the gate fails, quite probably (the one it tried in the past five hours did), and it goes back to dealing with all the pending check pipeline changes | 21:01 |
| zaro | fungi: that command doesn't work with ls-projects | 21:01 |
| *** rfolco has quit IRC | 21:02 | |
| fungi | zaro: actually it does. one odd quirk about that account is that gerrit eats all output from commands run by it | 21:02 |
| fungi | zaro: so you get no feedback except on things like --help or api errors | 21:02 |
| zaro | ahh.. ok. let me try again. | 21:02 |
| *** gothicmindfood has quit IRC | 21:03 | |
| *** denis_makogon_ is now known as denis_makogon | 21:04 | |
| *** obondarev_ has quit IRC | 21:05 | |
| lifeless | pleia2: ping? | 21:05 |
| *** vkozhukalov has quit IRC | 21:05 | |
| *** vipul is now known as vipul-away | 21:05 | |
| zaro | fungi: hmm. i ran the command to add user to group but failed with 'fatal: internal server error'. same command worked when i run using my account. | 21:05 |
| *** krtaylor has joined #openstack-infra | 21:07 | |
| zaro | clarkb: F11 to toggle full screen | 21:07 |
| fungi | zaro: ahh, okay. i've used it for gsql calls, but there are definitely specific operations it doesn't support for various reasons, as i've discovered | 21:07 |
| clarkb | zaro: right, I do that then it goes back if I switch windows it is annoying | 21:07 |
| *** dizquierdo has joined #openstack-infra | 21:07 | |
| *** bknudson has joined #openstack-infra | 21:08 | |
| *** vipul-away is now known as vipul | 21:09 | |
| zaro | clarkb: ohh, probably an OS thing then. | 21:09 |
| *** dprince has quit IRC | 21:09 | |
| lifeless | fungi: / clarkb: why does nodepool create per-image keypairs when NODEPOOL_SSH_KEY is set? | 21:09 |
| lifeless | surely it should just use the supplied keypair in that case? | 21:10 |
| clarkb | lifeless: NODEPOOL_SSH_KEY is for the user account not the root account | 21:11 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: parse the failed jobs in stream https://review.openstack.org/65789 | 21:11 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: moving readiness checks into stream https://review.openstack.org/65961 | 21:11 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: refactor bot to be based on argparse https://review.openstack.org/66401 | 21:11 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: fix bot to actually use super() https://review.openstack.org/66405 | 21:11 |
| lifeless | clarkb: yeah, but it makes debugging super hard when one can't log in at all on a failed deploy | 21:12 |
| zaro | fungi: ok.since i don't know how to get that working, i've noted it in the script. So it's basically tested and ready to go whenever we decide to pull the trigger. | 21:12 |
| *** marun_ has joined #openstack-infra | 21:13 | |
| clarkb | lifeless: I am not arguing it should be that way, just explaining the behavior | 21:13 |
| *** yamahata has joined #openstack-infra | 21:13 | |
| lifeless | clarkb: ack | 21:13 |
| *** mrda_ has joined #openstack-infra | 21:13 | |
| fungi | zaro: sure, i think for having a repeatable process to do the upgrade, that's good enough. we can probably just run those ssh commands from our workstations instead of locally on the server too if we'd prefer | 21:13 |
| zaro | fungi: excellent! | 21:14 |
| *** dkliban_afk is now known as dkliban | 21:14 | |
| *** marun has quit IRC | 21:14 | |
| *** marun_ is now known as marun | 21:14 | |
| *** jcoufal has quit IRC | 21:15 | |
| fungi | and as we spot anything else which needs to be evolved in the upgrade process through subsequent testing, or from failed upgrade attempts which we roll back, we can update that etherpad accordingly | 21:15 |
| zaro | not sure if etherpad is the right place for the script but will leave it there until there's a need to move it somewhere else. | 21:15 |
| fungi | zaro: also having the steps to roll back to its pre-upgrade state would be helpful to note in there as well | 21:15 |
| zaro | i've added the link to the gerrit-2.8-upgrade blueprint. | 21:16 |
| zaro | fungi: ok. will add that. | 21:16 |
| fungi | which i assume is mostly just reload the mysqldump we take immediately prior to the upgrade, and same for a klast-minute copy of the gerrit2 homedir | 21:16 |
| mattoliverau | anteaya: sorry, been reading scrollback, takes a while when everything in the channel seems to happen while you sleep :P Damn time difference ;) | 21:16 |
| *** NikitaKonovalov has quit IRC | 21:16 | |
| fungi | mattoliverau: i've found the solution to that problem. don't sleep. ever again | 21:17 |
| mattoliverau | anteaya: I'm good for the moment, taking a look at a few things, my cacti redirect patch has been sent for review. Working on puppetifying turbo-hipster so it aligns with Infra better. | 21:17 |
| *** wenlock has joined #openstack-infra | 21:17 | |
| clarkb | zaro: scp plugin race thing, I can talk about that in a minute | 21:17 |
| mattoliverau | fungi: you must be on *alot* of coffee :P | 21:17 |
| zaro | fungi: yep. | 21:17 |
| zaro | clarkb: whenver your ready | 21:18 |
| clarkb | zaro: what is happening is jenkins emits the onFinished event before the console.html file is even created on the log server. I think we should add a synchronization point for the job thread and console.html writer thread that ensures the file is always created before the job completes | 21:19 |
| clarkb | zaro: the console.html may be incomplete but at least it will exist and apache won't return a 404 | 21:19 |
| fungi | mattoliverau: not enough. i end up taking a nap every morning around 3am | 21:20 |
| * fungi clearly can't follow his own advice | 21:20 | |
| *** che-arne has quit IRC | 21:21 | |
| *** sarob has quit IRC | 21:21 | |
| clarkb | zaro: is that someone you want to try patching? | 21:22 |
| clarkb | s/someone/something/ | 21:22 |
| *** sarob has joined #openstack-infra | 21:22 | |
| fungi | patch all the people | 21:22 |
| zaro | clarkb: don't quite understand yet. log server is logstash correct? | 21:22 |
| mattoliverau | fungi: lol, you deserve a medal.. or a holiday! | 21:23 |
| clarkb | zaro: no log server is where the console.html file is being copied | 21:23 |
| fungi | zaro: log server in this case means the logs.openstack.org website | 21:23 |
| fungi | mattoliverau: i want a metal holiday | 21:23 |
| mattoliverau | fungi: I believe that's called mining... and it isn't really a holiday, but you'd probably get more sleep then you do now :P | 21:24 |
| zaro | clarkb: ok. i'll take a crack at it. will let you know if i have further questions. | 21:25 |
| clarkb | zaro: basically the job thread needs to wait for the console copy thread to say it has written something | 21:26 |
| clarkb | and something may be creating a 0 byte file | 21:26 |
| *** sarob has quit IRC | 21:26 | |
| mordred | clarkb, fungi: so far, I've run in storyboard-webclient and mistral-extra and neither have produced errors - but neither added new groups | 21:26 |
| *** jhesketh_ has joined #openstack-infra | 21:27 | |
| fungi | mordred: that really makes me suspect it's something to do with running under the context of the puppet agent | 21:27 |
| mordred | yeah. | 21:27 |
| mordred | I'm going to try the savana-ci one from SergeyLukjanov | 21:27 |
| *** krotscheck has quit IRC | 21:28 | |
| SergeyLukjanov | mordred, thank you | 21:28 |
| fungi | mordred: i think we have logger support in the script, maybe we would benefit from a patch to configure it to log messages somewhere we can inspect so that when it runs under puppet and breaks we get a little more info | 21:28 |
| dstufft | mordred: I'll probably be starting the wheels for 1.5.1 / 1.11.1 RCs later tonight | 21:29 |
| dstufft | the last thing we were waiting on just dropped | 21:29 |
| dstufft | fungi: and whoever else | 21:29 |
| dstufft | ^ | 21:29 |
| *** julim has quit IRC | 21:30 | |
| jog0 | fungi: any bug for http://logs.openstack.org/19/66619/2/gate/gate-nova-pep8/81e93af/console.html | 21:30 |
| *** jcoufal has joined #openstack-infra | 21:31 | |
| anteaya | mattoliverau: welcome to the world of -infra, infinite scrollback and a good first habit to pick up | 21:31 |
| anteaya | mattoliverau: can you hit me again with a url for the catci patch? | 21:31 |
| *** obondarev_ has joined #openstack-infra | 21:31 | |
| fungi | jog0: yeah, i guess we could keep (re/ab)using bug 1267364 for that | 21:32 |
| uvirtbot | Launchpad bug 1267364 in openstack-ci "Recurrent jenkins slave agent failures" [Critical,In progress] https://launchpad.net/bugs/1267364 | 21:32 |
| *** harlowja has quit IRC | 21:32 | |
| fungi | jog0: no, wait, wrong bug | 21:33 |
| anteaya | cacti even | 21:33 |
| jog0 | bug https://bugs.launchpad.net/openstack-ci/+bug/1260311 | 21:33 |
| uvirtbot | Launchpad bug 1260311 in openstack-ci "hudson.Launcher exception causing build failures" [Undecided,New] | 21:33 |
| jog0 | bug 1260311 | 21:33 |
| uvirtbot | Launchpad bug 1260311 in openstack-ci "hudson.Launcher exception causing build failures" [Undecided,New] https://launchpad.net/bugs/1260311 | 21:33 |
| jog0 | that looks like it | 21:33 |
| fungi | jog0: i think maybe we don't have one for the traceback you have there... it's probably a case of the vm going offline after being registered as a jenkins slave but prior to a job getting assigned | 21:34 |
| *** hashar_ has joined #openstack-infra | 21:34 | |
| fungi | oh, looks like we do | 21:34 |
| jog0 | with a better query | 21:34 |
| fungi | of course, seems to be one i missed triaging | 21:34 |
| *** hashar has quit IRC | 21:34 | |
| *** hashar_ is now known as hashar | 21:34 | |
| *** sarob has joined #openstack-infra | 21:35 | |
| sdague | fungi: any idea about this - http://logs.openstack.org/92/61892/10/check/check-dg-tempest-dsvm-full-reexec/93cccb9/console.html#_2014-01-16_06_00_15_762 | 21:36 |
| sdague | which is preventing the libvirt log collection from passing | 21:36 |
| *** smurugesan has joined #openstack-infra | 21:36 | |
| *** smurugesan has left #openstack-infra | 21:37 | |
| *** smurugesan has joined #openstack-infra | 21:37 | |
| *** wenlock has quit IRC | 21:37 | |
| fungi | sdague: the missing /var/lib/jenkins/logs/slaves/devstack-precise-hpcloud-az1-1098385/slave.log | 21:38 |
| *** wenlock has joined #openstack-infra | 21:38 | |
| fungi | sdague: you get it consistently for that one change? | 21:38 |
| fungi | sdague: oh, "Looks like the node went offline during the build." | 21:39 |
| fungi | sdague: so basically the master lost contact with the slave while the job was running | 21:39 |
| fungi | weird | 21:39 |
| fungi | it got very close to completion there | 21:39 |
| *** hogepodge_ has joined #openstack-infra | 21:40 | |
| russellb | sdague: fungi: i would be a super happy camper if we could try this: https://review.openstack.org/#/c/65805/ :-) | 21:41 |
| *** wenlock has quit IRC | 21:41 | |
| *** wenlock has joined #openstack-infra | 21:42 | |
| clarkb | fungi: I am thinking jenkins01 upgrade can ahppen this weekend after the gerrit restart as things should be quiet then | 21:42 |
| * zaro is afk, will bbl | 21:42 | |
| clarkb | fungi: is there any urgent reason to do it sooner? | 21:42 |
| *** hogepodge has quit IRC | 21:43 | |
| *** hogepodge_ is now known as hogepodge | 21:43 | |
| fungi | russellb: i'll leave that up to sdague and mtreinish... i'm happy to +2/approve on a technical basis, but it's more of a testing decision | 21:43 |
| russellb | also depends on https://review.openstack.org/#/c/66379/ | 21:43 |
| *** wenlock has quit IRC | 21:43 | |
| *** wenlock has joined #openstack-infra | 21:43 | |
| clarkb | russellb: looks like that bumps test run time up quite a bit | 21:43 |
| clarkb | into the hour range | 21:43 |
| russellb | clarkb: yes, but it also makes them work | 21:43 |
| fungi | clarkb: i don't see any obvious urgency for it, no | 21:43 |
| fungi | clarkb: weekend seems fine | 21:43 |
| fungi | clarkb: they're already into the hour range | 21:43 |
| mordred | clarkb: "should be quiet then" | 21:43 |
| * mordred enjoys clarkb sense of humour | 21:44 | |
| fungi | yeah, last weekend and the weekend before were anything but quiet, just quieter than weekdays | 21:44 |
| mattoliverau | anteaya: http://is.gd/XftgIj | 21:44 |
| *** needscoffee is now known as morganfainberg | 21:45 | |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: tests: Allow to test maven project parameters https://review.openstack.org/67265 | 21:45 |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: project_maven: Don't require artifact-id and group-id https://review.openstack.org/66036 | 21:45 |
| fungi | okay, sorry to abandon everyone in their hour of need, but i'm supposed to be going out to dinner with family. i'll be back around later though | 21:46 |
| *** markmc has quit IRC | 21:46 | |
| *** Ajaeger has quit IRC | 21:47 | |
| mordred | fungi: you dinner | 21:48 |
| * mattoliverau is heading home from the coffee shop before it gets too hot outside. BBS | 21:49 | |
| sdague | clarkb / mordred: so honestly, we should have a pretty good test on whether the fix by russellb is going to help, which is to merge it and see if it empties zuul overnight | 21:50 |
| russellb | talking about this in -nova too ... | 21:51 |
| russellb | couple of smoking guns | 21:51 |
| russellb | 1) non-linear slowdown when this patch is in | 21:51 |
| russellb | 2) look at sysstat and how maxed out the node is - http://logs.openstack.org/81/66081/1/gate/gate-tempest-dsvm-postgres-full/57dd449/logs/screen-sysstat.txt.gz | 21:51 |
| sdague | clarkb: are you in the state to do a baby sit on something like this? | 21:52 |
| sdague | yeh, the sysstat system time is definitely an indicator, that high system time seems very odd | 21:52 |
| mordred | I can help - fixing the gate is quite important | 21:52 |
| russellb | sdague: scroll down a ways and see how it's close to 0% idle for a long time | 21:53 |
| clarkb | sdague: ya I will be around to revert it if necessary (I suppose that is the babysitting that would be needed) | 21:53 |
| *** odyssey4me has quit IRC | 21:53 | |
| russellb | i think that's causing use failures all over the place | 21:53 |
| sdague | mordred: so if we merge this, will it change the concurrency on jobs in the reset? | 21:53 |
| mordred | also - this is a bit inline with a conversation jog0 and I had a while back - which is that if you consider what we're trying to make these test clouds do - it's AMAZING that it works at all | 21:53 |
| mordred | sdague: yes. it should. tip of d-g is grabbed on each change | 21:54 |
| sdague | mordred: ok, lets do it | 21:54 |
| russellb | need the infra-config change first | 21:54 |
| russellb | https://review.openstack.org/#/c/66379/ | 21:54 |
| mordred | honest to god - tests taking twice as long but passing is better than tests resetting constantly | 21:54 |
| sdague | mordred: agreed | 21:54 |
| russellb | mordred: but it's only 1.2-1.4x longer, since it was maxed out | 21:55 |
| dstufft | mordred: "first make it correct, then make it fast" | 21:55 |
| dstufft | :D | 21:55 |
| sdague | mordred: so please +A - https://review.openstack.org/#/c/66379/ | 21:55 |
| *** dkliban is now known as dkliban_afk | 21:55 | |
| mordred | if there are real russellb things that can be fixed subsequently | 21:55 |
| sdague | russellb: it's more like 1.5 - 1.6 | 21:55 |
| sdague | jeblair: gave those numbers on a really fast run | 21:55 |
| sdague | but we'll get lots of data overnight | 21:55 |
| mordred | is 125 high enough? | 21:55 |
| russellb | sdague: ah ok | 21:55 |
| dansmith | 1.6 is as compelling as 1.2 to me, FWIW | 21:56 |
| sdague | mordred: .... we could always go on the safe side and bump them to 185 | 21:56 |
| mordred | sure. just making sure we're bumping the timeout properly | 21:56 |
| sdague | why don't we do that | 21:56 |
| *** luqas has joined #openstack-infra | 21:57 | |
| russellb | highest i've seen is close to 90 minutes | 21:57 |
| sdague | go to 3 hrs, we haven't had a real runaway problem in a while | 21:57 |
| mordred | sdague: do we have any fear of being held hostage by a true runaway test? | 21:57 |
| sdague | I haven't seen one in tempest in forever | 21:57 |
| sdague | each test has a timeout as well | 21:57 |
| mordred | good point | 21:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Ensure taskflow upload affects the mirrors https://review.openstack.org/66051 | 21:57 |
| mordred | russellb says 90 is high-water though | 21:57 |
| russellb | happy to make it much higher so it's not in the way, whatever | 21:58 |
| sdague | mordred: we actually hit the 90 timeout on one of those | 21:58 |
| russellb | true | 21:58 |
| mordred | ok. let's bump it then. russellb, you wanna do it? | 21:58 |
| russellb | sure, happy to | 21:58 |
| *** harlowja has joined #openstack-infra | 21:59 | |
| *** harlowja has quit IRC | 21:59 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1269940 https://review.openstack.org/67303 | 21:59 |
| uvirtbot | Launchpad bug 1269940 in openstack-ci "[EnvInject] - [ERROR] - SEVERE ERROR occurs: java.lang.InterruptedException" [Undecided,New] https://launchpad.net/bugs/1269940 | 21:59 |
| *** harlowja has joined #openstack-infra | 21:59 | |
| sdague | I'd just hate to do this, then be killing ourselves in timeouts | 22:00 |
| mordred | yah | 22:01 |
| russellb | good point | 22:01 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Increase timeouts for jobs doing tempest runs https://review.openstack.org/66379 | 22:01 |
| russellb | so now timeout is doubled to 3 hours | 22:01 |
| russellb | from 90 | 22:01 |
| mordred | ok. that's +A | 22:01 |
| *** odyssey4me has joined #openstack-infra | 22:02 | |
| mordred | and I've +A'd the d-g patch too | 22:02 |
| *** pelix has joined #openstack-infra | 22:03 | |
| russellb | \o/ | 22:03 |
| *** jcoufal has quit IRC | 22:03 | |
| russellb | tempting to move it to the front of the gate .... | 22:03 |
| russellb | just had a reset anyway not long ago it seems | 22:03 |
| *** ArxCruz has quit IRC | 22:04 | |
| *** nati_ueno has joined #openstack-infra | 22:04 | |
| sdague | mordred: so as soon as config merges we should promote the d-g change | 22:04 |
| sdague | russellb: yeh, I agree, but we need config to reload first | 22:04 |
| russellb | sounds great | 22:04 |
| russellb | thanks a bunch guys | 22:04 |
| russellb | fwiw, we've been working really hard on nova-network performance this week too | 22:04 |
| sdague | othwerwise we have a reasonable chance of reseting ourselves... which would be bad :) | 22:04 |
| *** hashar has quit IRC | 22:05 | |
| sdague | russellb: awesome | 22:05 |
| *** pcrews_ has joined #openstack-infra | 22:05 | |
| *** ruhe is now known as _ruhe | 22:06 | |
| *** ArxCruz has joined #openstack-infra | 22:07 | |
| mordred | sdague, russellb: clarkb says there's a CVE we need to bump to the front too | 22:07 |
| chmouel | fungi: https://review.openstack.org/#/c/67311/ | 22:08 |
| mordred | but really, we need the event to be processed by zuul first, so that there is a thing we can bump | 22:08 |
| chmouel | fungi: i think that would make things better | 22:08 |
| chmouel | fungi: it fails on my vm but it seems unrelated to the problems others have seen | 22:08 |
| chmouel | (boto version being to new for grizzly euca2ool is the failure i am getting but i wonder if it's some mess up on my system) | 22:09 |
| sdague | mordred: honestly, if this is increasing our chances of things passing, then it should go in front | 22:09 |
| *** gyee has quit IRC | 22:09 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 22:09 | |
| *** marun has quit IRC | 22:09 | |
| *** SergeyLukjanov_a is now known as SergeyLukjanov_ | 22:10 | |
| *** MarkAtwood has joined #openstack-infra | 22:10 | |
| clarkb | sdague: I don't think anyone is disagreeing just that we have several things to bump all at once | 22:10 |
| clarkb | sdague: so we should approve the change now | 22:10 |
| mordred | clarkb: change is approved | 22:10 |
| wenlock | question for gerritbot gurus, anyone know where the logfile lives? | 22:11 |
| wenlock | im trying to debug a startup issue on my end | 22:11 |
| clarkb | wenlock: you have to specify the location in the config file | 22:12 |
| clarkb | or it goes to stderr | 22:12 |
| *** prad has quit IRC | 22:12 | |
| wenlock | clarkb: ahh | 22:12 |
| wenlock | clarkb: ok, so i just need to go figure out what it's called... it's not set | 22:13 |
| *** thuc has quit IRC | 22:13 | |
| *** nati_ueno has quit IRC | 22:13 | |
| *** vipul is now known as vipul-away | 22:13 | |
| *** vipul-away is now known as vipul | 22:13 | |
| clarkb | wenlock: under ircbot it is log_config | 22:13 |
| wenlock | clarkb: thanks! | 22:13 |
| clarkb | oh that expects a python logging config file which can specify locations | 22:13 |
| *** thuc has joined #openstack-infra | 22:13 | |
| clarkb | so its an extra level of indirection :/ | 22:14 |
| *** sarob has quit IRC | 22:14 | |
| wenlock | ic | 22:14 |
| *** gyee has joined #openstack-infra | 22:14 | |
| *** nati_ueno has joined #openstack-infra | 22:14 | |
| wenlock | like a log4j file | 22:14 |
| wenlock | ok | 22:14 |
| *** sarob has joined #openstack-infra | 22:14 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1260311 https://review.openstack.org/67314 | 22:15 |
| uvirtbot | Launchpad bug 1260311 in openstack-ci "hudson.Launcher exception causing build failures" [Low,Triaged] https://launchpad.net/bugs/1260311 | 22:15 |
| anteaya | mattoliverau: commented | 22:15 |
| anteaya | mattoliverau: also we are used to posting urls as they are in this channel, like so: https://review.openstack.org/#/c/66181/ | 22:16 |
| anteaya | some patches are familiar enough we recognize them but their number, in this case 66181 and we refer to them that way | 22:16 |
| * mattoliverau is back | 22:16 | |
| anteaya | *by their number | 22:16 |
| anteaya | :D | 22:17 |
| anteaya | welcome back | 22:17 |
| *** luqas has quit IRC | 22:17 | |
| *** thuc has quit IRC | 22:18 | |
| anteaya | mattoliverau: I'll introduce you around when we are not so busy | 22:19 |
| mattoliverau | anteaya: thanks for the welcome and the comments :) BTW that hot Perth weather decided to follow me home to Melbounre, all week has been 40+ :( | 22:19 |
| anteaya | if we ever get to a point when we are not so busy | 22:19 |
| mordred | mattoliverau: welcome! I've heard of you already I think | 22:19 |
| anteaya | I have read that, they cancelled the tennis due to the broiling temps | 22:19 |
| mordred | WHAT??? | 22:20 |
| anteaya | yeah | 22:20 |
| mordred | they cancelled the tennis? | 22:20 |
| clarkb | "the tennis" <- I have no idea why it is called that but I laugh just about every time I hear it | 22:20 |
| clarkb | mordred: there are 176 nodes in the delete state in nodepool | 22:21 |
| mordred | clarkb: what else would you call The Australian Open Tennis Tournament? | 22:21 |
| mordred | clarkb: wow | 22:21 |
| anteaya | http://www.bbc.com/sport/0/tennis/25741891 | 22:21 |
| mattoliverau | clarkb: LOL. yeah, we aussies love our sport... apparently | 22:21 |
| clarkb | mordred: I am going to slowly start deleting them | 22:21 |
| clarkb | mordred: not going to one shot it to make sure I don't break anything | 22:21 |
| *** jaypipes has quit IRC | 22:21 | |
| mordred | ++ | 22:21 |
| mattoliverau | anteaya: I'll make sure I'm in channel everyday I'm working.. and maybe even when I'm not, so no rush :) | 22:21 |
| * mordred has been to the tennis twice | 22:22 | |
| anteaya | mattoliverau: awesome, thank you | 22:22 |
| mattoliverau | mordred: Only good things I hope :P | 22:22 |
| anteaya | mordred: hope it was less than 40C when you went | 22:22 |
| mordred | mattoliverau: I hear all. I see all. | 22:22 |
| anteaya | there was a story today where I read they cancelled the tennis but I can't find it atm | 22:23 |
| mattoliverau | mordred: so all the not so good things then :P | 22:23 |
| anteaya | mattoliverau: he is omnipresent that way | 22:23 |
| mattoliverau | lol | 22:23 |
| *** sandywalsh has quit IRC | 22:23 | |
| anteaya | mattoliverau: so just get a sense of the flow | 22:23 |
| anteaya | read stuff, don't worry about what you don't understand yet | 22:24 |
| anteaya | click links and start to get familiar with our workflow | 22:24 |
| anteaya | don't worry about what doesn't make sense yet, it will | 22:24 |
| anteaya | just be patient | 22:24 |
| mordred | we should turn the in-use/deleting graph into sound and build some dubstep on top of it | 22:25 |
| anteaya | waiting devs can dance while waiting for patches to show up in the queue | 22:25 |
| anteaya | good idea | 22:25 |
| *** HenryG has quit IRC | 22:26 | |
| anteaya | that graph does have a consistent rhythm to it | 22:26 |
| mattoliverau | anteaya: cool, will do :) And hopefully once I get up to speed I can help take the load of you guys, especially in my timezone :) | 22:26 |
| anteaya | awesome | 22:26 |
| anteaya | start small | 22:27 |
| anteaya | being able to answer questions, like "why doesn't my patch show up in the zuul status page" is a good question to be able to answer right now | 22:27 |
| anteaya | we get that one about 6 times a day right now | 22:27 |
| anteaya | mattoliverau: do you remember the answer to that one? | 22:27 |
| mattoliverau | I noticed that in the scroll back :) | 22:28 |
| anteaya | yup | 22:28 |
| anteaya | how would you answer that? | 22:28 |
| *** vipul is now known as vipul-away | 22:29 | |
| anteaya | I can give a hint if you need it | 22:29 |
| mattoliverau | anteaya: The queue length is quite high, so although it has been submitted to zuul it hasn't been processed and started in the check pipeline yet. | 22:31 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1269940 https://review.openstack.org/67303 | 22:31 |
| anteaya | woohoo | 22:31 |
| uvirtbot | Launchpad bug 1269940 in openstack-ci "[EnvInject] - [ERROR] - SEVERE ERROR occurs:" [Undecided,New] https://launchpad.net/bugs/1269940 | 22:31 |
| anteaya | yes | 22:31 |
| anteaya | you are hired | 22:31 |
| anteaya | those questions are all yours now | 22:31 |
| *** esker has joined #openstack-infra | 22:31 | |
| anteaya | thank you | 22:32 |
| anteaya | you are also welcome to point them to this information on the status.openstack.org/zuul page: Queue lengths: 543 events, 1 results. | 22:32 |
| anteaya | but answer however you like | 22:32 |
| mattoliverau | anteaya: sure, and thanks :) | 22:33 |
| jog0 | openstack/openstack points to no commints in last 6 hours | 22:33 |
| mattoliverau | anteaya: I did learn from the best ;) | 22:33 |
| anteaya | mattoliverau: flatterer | 22:34 |
| mordred | jog0: we're about to fix everything | 22:34 |
| mattoliverau | anteaya: gets you everywhere :) | 22:34 |
| chmouel | i should find a way how to put the status./zuul page as screensaver | 22:34 |
| sdague | mordred: well it's taking 2 hrs to get on a unit test node | 22:34 |
| anteaya | mattoliverau: eh, not so much, here patches get you further | 22:34 |
| mattoliverau | lol | 22:34 |
| anteaya | :D | 22:35 |
| anteaya | chmouel: heh | 22:35 |
| *** sandywalsh has joined #openstack-infra | 22:35 | |
| anteaya | chmouel meet mattoliverau | 22:35 |
| chmouel | hello mattoliverau | 22:35 |
| anteaya | mattoliverau this is chmouel | 22:35 |
| mattoliverau | chmouel: hi | 22:35 |
| *** esker has quit IRC | 22:36 | |
| anteaya | mattoliverau is from oz and chmouel is in france | 22:36 |
| *** esker has joined #openstack-infra | 22:36 | |
| anteaya | mattoliverau is learning our infra system | 22:36 |
| chmouel | two good country for wine i might say | 22:36 |
| anteaya | chmouel is a very helpful chap | 22:36 |
| mattoliverau | chmouel: indeed :) | 22:36 |
| anteaya | since you will both be awake for some portion of your days, I figured it does hurt for you to meet now | 22:36 |
| chmouel | nice thanks anteaya | 22:37 |
| mattoliverau | anteaya: he may get bombared with questions from me then ;) | 22:37 |
| anteaya | chmouel: sure | 22:37 |
| anteaya | mattoliverau: I'm counting on it | 22:37 |
| anteaya | chmouel loves questions | 22:37 |
| anteaya | :D | 22:37 |
| chmouel | anteaya: ;) | 22:37 |
| chmouel | mattoliverau: for the hours i will be sleepless feel free to ask :) | 22:37 |
| mattoliverau | chmouel: cheers, I appreciate that :) | 22:38 |
| *** CaptTofu has quit IRC | 22:39 | |
| sdague | clarkb: so if you want something else to ponder, I think zuul might just be a sad panda right now. https://review.openstack.org/#/c/67267/ was submitted 2hrs ago, and hasn't found a py27 node yet. Which I think isn't really right | 22:39 |
| clarkb | sdague: its node thrash | 22:39 |
| clarkb | nodepool can't keep up with the demand | 22:40 |
| clarkb | sdague: I am manually deleting nodes to give nodepool more room to grow into | 22:40 |
| *** CaptTofu has joined #openstack-infra | 22:41 | |
| *** sarob has quit IRC | 22:41 | |
| *** mfink has quit IRC | 22:42 | |
| *** sarob has joined #openstack-infra | 22:42 | |
| *** jamielennox|away is now known as jamielennox | 22:43 | |
| *** emagana_ has quit IRC | 22:43 | |
| *** dkranz has quit IRC | 22:44 | |
| *** thuc has joined #openstack-infra | 22:44 | |
| *** denis_makogon has quit IRC | 22:44 | |
| *** krotscheck has joined #openstack-infra | 22:45 | |
| sdague | clarkb: but it's a py27 node... not a devstack node | 22:46 |
| *** sarob has quit IRC | 22:46 | |
| clarkb | sdague: a bunch of our py27 jobs are on single use slaves now iirc | 22:47 |
| clarkb | let me double check on the elasticrecheck jobs to see if that do that | 22:48 |
| *** vipul-away is now known as vipul | 22:48 | |
| sdague | ok | 22:48 |
| sdague | yeh, it looks like it just hit | 22:48 |
| clarkb | sdague: doesn't look like elasticrecheck is using single use nodes yet | 22:49 |
| pleia2 | lifeless: oops, was at lunch, just saw your ping from earlier now | 22:49 |
| clarkb | could still be general node starvation due to thrash | 22:49 |
| lifeless | pleia2: no worries, I'm now recovering a trashed cloud :P | 22:50 |
| lifeless | there may be a short delay, but I will come back to it | 22:50 |
| pleia2 | doh :) | 22:51 |
| *** thuc has quit IRC | 22:52 | |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Adding apache redirect for the cacti url. https://review.openstack.org/66181 | 22:53 |
| *** dizquierdo has quit IRC | 22:53 | |
| *** ryanpetrello has quit IRC | 22:54 | |
| *** obondarev_ has quit IRC | 22:56 | |
| mattoliverau | anteaya: I have addressed your commants in 66181. That's what I get for just modifying the package maintainers file and not changing it to follow our style guide. | 22:58 |
| mordred | mattoliverau: yes please | 22:59 |
| *** carl_baldwin has quit IRC | 22:59 | |
| *** simonmcc has quit IRC | 23:00 | |
| *** mfer has quit IRC | 23:00 | |
| *** carl_baldwin has joined #openstack-infra | 23:00 | |
| *** ewindisch is now known as zz_ewindisch | 23:01 | |
| anteaya | mattoliverau: thank you | 23:01 |
| *** mfink has joined #openstack-infra | 23:02 | |
| anteaya | there is only one way to learn and you are doing it | 23:02 |
| anteaya | :D | 23:02 |
| *** nati_uen_ has joined #openstack-infra | 23:02 | |
| mattoliverau | anteaya: thats true, and I do want to do it properly, so now I know the sytle required for apache config files :) | 23:04 |
| *** jgrimm has quit IRC | 23:06 | |
| *** nati_ueno has quit IRC | 23:06 | |
| *** smarcet has joined #openstack-infra | 23:06 | |
| *** smarcet has left #openstack-infra | 23:06 | |
| *** zz_ewindisch is now known as ewindisch | 23:08 | |
| anteaya | mattoliverau: great | 23:09 |
| *** thuc has joined #openstack-infra | 23:10 | |
| *** carl_baldwin has quit IRC | 23:10 | |
| *** carl_baldwin has joined #openstack-infra | 23:10 | |
| anteaya | mattoliverau: I am winding down for the evening and am soon off to dinner | 23:11 |
| anteaya | follow your nose, be helpful, ask questions (you never know who might be available to answer) | 23:12 |
| anteaya | talk to you tomorrow some time, unless this is your Friday in which case talk to you next week | 23:12 |
| clarkb | zaro: is the scp plugin something that you plan on working on soon? that is a reasonable high priority item and I can take a stab at it if you are busy with other things | 23:13 |
| mattoliverau | anteaya: OK, will do. Thanks for everything! It is Friday for me, so probably next week. Unless I jump online during the weekend :) | 23:13 |
| *** ewindisch is now known as zz_ewindisch | 23:14 | |
| *** markmcclain has quit IRC | 23:15 | |
| *** salv-orlando has quit IRC | 23:16 | |
| *** sarob has joined #openstack-infra | 23:16 | |
| *** carl_baldwin has quit IRC | 23:17 | |
| *** carl_baldwin has joined #openstack-infra | 23:17 | |
| anteaya | mattoliverau: your call, I'm betting there will be activity this weekend but part of this work is you have to decide how to best use your time | 23:17 |
| anteaya | so I'll see you when I see you | 23:18 |
| mattoliverau | anteaya: have a great night and enjoy your weekend, when it reaches you :) | 23:18 |
| anteaya | thanks | 23:18 |
| *** ryanpetrello has joined #openstack-infra | 23:19 | |
| *** flaper87 is now known as flaper87|afk | 23:19 | |
| anteaya | :D | 23:19 |
| *** rossella_s has quit IRC | 23:19 | |
| *** zz_ewindisch is now known as ewindisch | 23:20 | |
| *** dims has quit IRC | 23:22 | |
| *** eharney has quit IRC | 23:23 | |
| *** carl_baldwin has quit IRC | 23:23 | |
| *** dstanek has quit IRC | 23:24 | |
| *** ewindisch is now known as zz_ewindisch | 23:25 | |
| *** rcleere has quit IRC | 23:25 | |
| *** gokrokve has quit IRC | 23:26 | |
| jhesketh_ | Howdy | 23:26 |
| *** sarob has quit IRC | 23:26 | |
| *** gokrokve has joined #openstack-infra | 23:26 | |
| *** burt1 has quit IRC | 23:26 | |
| clarkb | jhesketh_: hey there | 23:26 |
| sdague | clarkb: so is there any thought that a zuul restart might help? It seems odd that - https://review.openstack.org/#/c/66379/ hasn't gone anywhere in 1.5 hrs | 23:27 |
| mattoliverau | jhesketh_: Good morning | 23:28 |
| sdague | it's processing events *super* slowly | 23:28 |
| clarkb | sdague: yes it is processing events super slowly. I don't think a restart would help as it would just get back into the current state | 23:29 |
| clarkb | sdague: zuul has no DDOS protection and we are handily DDOSing it | 23:29 |
| sdague | clarkb: so the total number of things in the queue doesn't seem worse than monday morning | 23:30 |
| sdague | when nodepool was off for 8 hrs | 23:30 |
| *** gokrokve has quit IRC | 23:30 | |
| sdague | but it's definitely behaving a lot worse | 23:30 |
| clarkb | sdague: its a combination of the number of jobs running (which hit zuuls git repos) and the processing of events to get there | 23:31 |
| sdague | ok | 23:31 |
| sdague | so zuul's going to see 2x this load next week on i2 | 23:31 |
| clarkb | isn't this week i2? I thought it was | 23:32 |
| sdague | no, next week | 23:32 |
| clarkb | I see | 23:32 |
| sdague | this is actually just normal non vacation week | 23:32 |
| sdague | there really hasn't been much added to the gate today | 23:32 |
| sdague | it's just moving through everything like molases | 23:32 |
| clarkb | right but the gate and check queues have been pertty large | 23:32 |
| clarkb | notice the jobs launched per hour graph it is well above typical week | 23:32 |
| sdague | clarkb: on monday we started with a check of 250 and gate of 50 | 23:32 |
| clarkb | 400-500 is where we float during a normal week we are about 2x that today | 23:33 |
| sdague | ok, well, it's only going to get worse :) | 23:33 |
| clarkb | we know... | 23:33 |
| sdague | so I guess the question is do we have a bigger box for zuul? | 23:33 |
| russellb | lots of people picking up steam post-vacation/holidays though | 23:34 |
| russellb | lots of extra energy :) | 23:34 |
| clarkb | sdague: we can do that if necessary (we did it during lca and it helped), however we have to go to a giant machine to get anymore cpus than we currently have | 23:34 |
| clarkb | sdague: have you seen jeblairs email on scaling zuul? | 23:34 |
| sdague | clarkb: I have | 23:34 |
| sdague | but is that going to be ready in 5 weeks? | 23:34 |
| clarkb | sdague: that is going to be a priority of his when he gets back and I think it will help quite a bit | 23:34 |
| sdague | because i3 is going to be 3x this load I'd expect | 23:34 |
| clarkb | sdague: I don't see why it cant be | 23:34 |
| sdague | ok, cool if it's that quick to get out there that's awesome | 23:35 |
| clarkb | the other thing which has been talked about without much enthusiasm until recently is a rate limiter for zuul | 23:35 |
| sdague | yeh, that would probably really help | 23:35 |
| clarkb | I think rate limiting zuul will help quite a bit because we can avoid the situation we are in now and only process consumable chunks at any given time | 23:35 |
| sdague | yep, definitely | 23:36 |
| sdague | because we're definitely event thrashing | 23:36 |
| clarkb | ya there is a compounding problem | 23:36 |
| notmyname | rate limit == only process eg the first 15 patches at a time? | 23:36 |
| *** dcramer_ has joined #openstack-infra | 23:37 | |
| clarkb | notmyname: sort of. It needs to rate limit behind some of the event processing because most gerrit events are just noise. But yes only have so many changes in flight at a given time so that failures don't bring everything to a halt | 23:37 |
| clarkb | jeblair has been opposed to this in the past but I think he may be more on board after talking about it briefly at LCA | 23:37 |
| *** thedodd has quit IRC | 23:37 | |
| clarkb | notmyname: my current thinking is to do something like tcp slow start and have the system self limit | 23:37 |
| *** dims has joined #openstack-infra | 23:38 | |
| notmyname | clarkb: just as long as it's not "system is full. approve this patch in XX minutes" :-) | 23:38 |
| clarkb | notmyname: no no, it would happen transparent to the user | 23:38 |
| clarkb | for example gate queue is 90 deep right now. if first change fails that means we have to restart 90 changes, that means recalculating zuul merges for 90 commits then serving those to 900 jobs and hoping you don't do it all again when the second fails | 23:39 |
| clarkb | that puts a lot of strain on everything. but a queue depth of say 20 may not | 23:39 |
| *** ok_delta has joined #openstack-infra | 23:40 | |
| clarkb | but I think rate limiting is somethign we will need to talk more about with jeblair as he has been wary of the idea in the past | 23:40 |
| *** lcestari has quit IRC | 23:41 | |
| *** ryanpetrello has quit IRC | 23:43 | |
| *** nati_ueno has joined #openstack-infra | 23:50 | |
| *** melwitt has quit IRC | 23:51 | |
| *** nati_ueno has quit IRC | 23:51 | |
| *** nati_ueno has joined #openstack-infra | 23:52 | |
| *** krtaylor has quit IRC | 23:52 | |
| *** nati_uen_ has quit IRC | 23:53 | |
| clarkb | I wonder if zuuls simple event handling mechanism (handle all results then handle events) is partially to blame here. | 23:54 |
| clarkb | if there are enough jobs returning results and honestly 900 over an hour seems like it could do this, then zuul would be starved of its ability to process gerrit events | 23:54 |
| *** vipul is now known as vipul-away | 23:55 | |
| *** CaptTofu has quit IRC | 23:58 | |
| *** dkliban_afk has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!