| clarkb | jeblair: ya I am tempted to get in a day before but I will have just done the trip back from hawaii and I trust you and mordred will do the right thing :) | 00:00 |
|---|---|---|
| *** saschpe has quit IRC | 00:00 | |
| Shrews | fungi: i'm just glad i finally don't have to fly the entire day for once | 00:00 |
| jeblair | clarkb: you can trust us to eat bbq! | 00:00 |
| *** jhesketh__ has joined #openstack-infra | 00:00 | |
| jeblair | Shrews: i'm road-tripping it from nc | 00:01 |
| clarkb | fungi: omni appears to be sold out and I am not sure how to get HP to do the room block if there are rooms in the block left | 00:01 |
| Shrews | jeblair: oh? from raleigh? | 00:01 |
| * clarkb looks for hilton | 00:02 | |
| fungi | clarkb: wow, that fast? | 00:02 |
| clarkb | fungi: well it is this carlson wagon thing | 00:02 |
| fungi | clarkb: the westin is apparently several blocks from the conference center, from what i'm told (but i don't trust it without consulting a map) | 00:02 |
| clarkb | fungi: who knows what additional restrictions they place on my booking | 00:02 |
| fungi | clarkb: only your hair dresser knows for sure | 00:02 |
| *** julim has quit IRC | 00:03 | |
| jeblair | Shrews: probably from winston; i'm thinking of spending a couple days in raleigh early in the week, then a couple with the folks in winston, then heading out from there | 00:03 |
| fungi | oh, wait, i'm thinking "who knows what evil lurks in the hearts of men?" (the shadow knows!) | 00:03 |
| Shrews | fungi: i'm not sure clarkb is old enough to get that previous reference ;) | 00:03 |
| clarkb | westin is a ways away but looks doable. The hilton near the conference center doesn't seem terrible | 00:03 |
| clarkb | Shrews: I didn't get it :/ | 00:04 |
| Shrews | lol | 00:04 |
| jeblair | ha! | 00:04 |
| fungi | Shrews: agreed, but the shadow is of course timeless | 00:04 |
| clarkb | damnit hilton isn't available either | 00:04 |
| * clarkb wonders if there is another massive conference happening at the same time | 00:04 | |
| jeblair | apparently the westin is under renovation and theoretically should be finished by the time of the summit | 00:04 |
| *** julim has joined #openstack-infra | 00:05 | |
| clarkb | they do have rail though right? I suppose being more core then catching that to conference center won't be terrible | 00:05 |
| lifeless | fungi: I'm keen to get that patch in so we can assess NobodyCam's Ironic support patches for tripleo-incubator | 00:05 |
| fungi | clarkb: it's possible we've already got a ton of room blocks set up for conferencegoers, but i haven't heard | 00:06 |
| clarkb | fungi: there are a couple room blocks omni is one location | 00:06 |
| clarkb | fungi: but I don't know how to make hps thing use the room blocks | 00:06 |
| clarkb | lifeless: ^ do you know? | 00:06 |
| dims | sdague, jeblair - the libvirt 0.9.8+ is ready for review - https://review.openstack.org/#/c/79816/ | 00:06 |
| *** saschpe has joined #openstack-infra | 00:06 | |
| *** medieval1 has joined #openstack-infra | 00:07 | |
| lifeless | clarkb: no idea; have we the goahead to book now ? | 00:10 |
| *** atiwari has quit IRC | 00:10 | |
| Shrews | lifeless: not that i've heard. i'm still wondering if i'm going to have to pay my own way | 00:10 |
| fungi | hmmm... zuul gearman server logs are not collected for yesterday... jeblair: why does zuul log when a build is added that it's on <Worker Unknown> (is that just a placeholder for future use or does it get updated or something?) | 00:12 |
| fungi | trying to track down Alex_Gaynor's unstable job result, but i can't seem to figure out from the zuul log which jenkins master was associated with the worker which grabbed it | 00:13 |
| jeblair | fungi: it's beceause gearman-plugin doesn't send information back about the worker; that was a jhesketh enhancement, i think turbo-hipster does use it | 00:13 |
| fungi | oh, okay | 00:13 |
| jeblair | fungi: that would be exactly the sort of information it would pass back though. :) | 00:13 |
| *** gokrokve_ has quit IRC | 00:13 | |
| jeblair | shouldn't be too hard to add | 00:13 |
| fungi | got it. so right now we don't actually have a way to map back from a job result to a jenkins master, i take it | 00:13 |
| jeblair | fungi: oh, see if the job url shows up in the debug log | 00:14 |
| *** gokrokve has joined #openstack-infra | 00:14 | |
| *** mrmartin has quit IRC | 00:14 | |
| jeblair | oh, huh, even the worker name shows up there | 00:15 |
| *** hogepodge has quit IRC | 00:15 | |
| jeblair | 2014-03-20 00:00:00,470 DEBUG zuul.Gearman: Build <gear.Job 0x7f95acb9ead0 handle: H:127.0.0.1:5731904 name: build:gate-grenade-dsvm unique: 01dd55a900f8445aaf7e4b55732c0605> update {u'url': u'https://jenkins02.openstack.org/job/gate-grenade-dsvm/7291/', u'manager': u'jenkins02.openstack.org', u'worker': u'devstack-precise-hpcloud-az3-2879669_exec-0', u'name': u'gate-grenade-dsvm', u'number': 7291} | 00:15 |
| *** rhsu has quit IRC | 00:15 | |
| jeblair | fungi: so you should be able to find a line like that ^ by searching for the build uuid | 00:16 |
| *** melwitt has quit IRC | 00:16 | |
| fungi | yeah, looking for the one associated with Build 7807fcef99a744ffbc77e55b5b7303e7 | 00:16 |
| fungi | was trying to spot what mapped those to one another | 00:17 |
| jeblair | i don't see that in the last 3 days of logs | 00:17 |
| jeblair | fungi: (oops, nm) | 00:18 |
| fungi | 2014-03-19 22:20:38,168 DEBUG zuul.IndependentPipelineManager: Adding build <Build 7807fcef99a744ffbc77e55b5b7303e7 of check-tempest-dsvm-full on <Worker Unknown>> of job check-tempest-dsvm-full to item <QueueItem 0x7f9565f80850 for <Change 0x7f95d11168d0 81319,1> in check> | 00:18 |
| *** gokrokve has quit IRC | 00:19 | |
| jeblair | 2014-03-19 22:41:30,294 DEBUG zuul.Gearman: Build <gear.Job 0x7f95d109e950 handle: H:127.0.0.1:5702811 name: build:check-tempest-dsvm-full unique: 7807fcef99a744ffbc77e55b5b7303e7> update {u'url': u'https://jenkins05.openstack.org/job/check-tempest-dsvm-full/4128/', u'manager': u'jenkins05.openstack.org', u'worker': u'devstack-precise-hpcloud-az2-2877013_exec-0', u'name': u'check-tempest-dsvm-full', u'number': 4128} | 00:19 |
| fungi | https://jenkins05.openstack.org/job/check-tempest-dsvm-full/4128/ | 00:19 |
| jeblair | yep | 00:19 |
| fungi | yeop | 00:19 |
| jogo | neutron has a fix to the top gate bug: Bug 1283522 - | 00:19 |
| uvirtbot | Launchpad bug 1283522 in neutron "DB lock timeout errors with parallel operations" [Critical,In progress] https://launchpad.net/bugs/1283522 | 00:19 |
| jogo | marun: has been working on https://review.openstack.org/#/c/80413/ | 00:19 |
| fungi | com.jcraft.jsch.JSchException: Session.connect: java.net.SocketException: Connection reset | 00:19 |
| fungi | Alex_Gaynor: https://jenkins05.openstack.org/job/check-tempest-dsvm-full/4128/consoleText | 00:19 |
| jogo | and it may be worth bumping this up to the top of the gate queue | 00:19 |
| jogo | gate is 13 hours behind | 00:20 |
| fungi | checking jenkins master log on jenkins05 for that worker now | 00:20 |
| jogo | with 29 failures due to this bug in last 24 hours | 00:20 |
| jogo | or roughly 30% of classified gate failures | 00:20 |
| jogo | jeblair fungi: thoughts | 00:21 |
| fungi | jogo: it passes tests and got approved? i'll promote it along with sdague's change he requested a bit ago | 00:21 |
| marun | fungi: yes, it passed check and is approved. | 00:21 |
| *** hogepodge has joined #openstack-infra | 00:24 | |
| *** masayukig has joined #openstack-infra | 00:24 | |
| *** hogepodge has quit IRC | 00:25 | |
| *** mrodden has quit IRC | 00:25 | |
| *** thuc has quit IRC | 00:26 | |
| *** thuc has joined #openstack-infra | 00:26 | |
| fungi | Alex_Gaynor: the scp publisher seems to have gotten a connection refusal (tcp/rst) connecting to static.openstack.org to upload job artifacts. haven't seen this one before, though we did see some intermittent network strangeness in rax-dfw in the past few hours | 00:27 |
| *** dkliban has quit IRC | 00:27 | |
| jogo | fungi: thanks | 00:27 |
| fungi | though https://status.rackspace.com/ doesn't mention anything odd | 00:27 |
| jogo | fungi: which patch did sdague request | 00:27 |
| fungi | jogo: 81603,2 | 00:28 |
| *** medieval1 has quit IRC | 00:28 | |
| *** matsuhashi has joined #openstack-infra | 00:29 | |
| *** krotscheck has quit IRC | 00:29 | |
| fungi | so stacking 80413,4 in with that on the next opportunity | 00:29 |
| Alex_Gaynor | fungi: awesome. Any way to rekick the build? | 00:30 |
| * clarkb did hilton doubletree fwiw which is probably not the nicest thing but was closer than some of the other options | 00:30 | |
| fungi | Alex_Gaynor: you can remove and re-add your approval vote | 00:30 |
| *** thuc has quit IRC | 00:31 | |
| fungi | Alex_Gaynor: oh, or i can approve it if maybe you're not core on requirements | 00:31 |
| Alex_Gaynor | fungi: I don't have an approval vote side it's my patch | 00:31 |
| fungi | Alex_Gaynor: done | 00:31 |
| Alex_Gaynor | Thanks | 00:31 |
| fungi | oh, that neutron job at the helm is about to take a phaser blast to the thorax. i'll just pre-empt the impending gate reset there with the queued promotes now | 00:33 |
| lifeless | redeploying tripleo testenvs | 00:33 |
| fungi | marun: jogo: sdague: promoted 80413,4 81603,2 now | 00:33 |
| lifeless | fungi: so - any chance you can look at https://review.openstack.org/#/c/81608/ | 00:33 |
| fungi | lifeless: maybe. is it more urgent than getting pypy and py33 jobs running? | 00:34 |
| marun | fungi: danke! | 00:34 |
| *** yaguang has joined #openstack-infra | 00:34 | |
| *** salv-orlando has joined #openstack-infra | 00:35 | |
| fungi | lifeless: reviewing now | 00:35 |
| marun | lifeless: or more important than approving the fix for the neutron functional job? https://review.openstack.org/#/c/81400/ | 00:35 |
| fungi | marun: neutron is functional? ;) | 00:35 |
| marun | fungi: only if we get this job fixed! ;) | 00:35 |
| fungi | heh | 00:35 |
| lifeless | fungi: ugh, didn't realise there was a firedrill on, but its pretty important since we can't tell if Ironic is working without these | 00:36 |
| fungi | i think it's always fire drill any more | 00:36 |
| fungi | lifeless: lgtm. marun: looking at yours now | 00:37 |
| marun | danke! | 00:37 |
| fungi | gotta keep my review stats up somehow | 00:38 |
| marun | heh | 00:38 |
| clarkb | jeblair: https://github.com/gitpython-developers/GitPython/issues/72 is the issue affecting the HP zuul merger best I can tell | 00:39 |
| clarkb | jeblair: I did manage to catch it in progress. straced the git process and it is writing to fd 0 and stuck there | 00:39 |
| clarkb | er sorry reading 0 | 00:39 |
| fungi | marun: lgtm | 00:40 |
| clarkb | killing the git process made the zuul merger continue on with its life. I think that we must not see this as a result of running on different hardware | 00:40 |
| marun | yay! | 00:41 |
| marun | danke! | 00:41 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Fix pip on py3k/pypy nodes https://review.openstack.org/75213 | 00:41 |
| clarkb | and with that debugging done time for me to walk home | 00:43 |
| *** rhsu has joined #openstack-infra | 00:43 | |
| *** asalkeld has left #openstack-infra | 00:44 | |
| jogo | fungi: thanks | 00:44 |
| fungi | okay, new wrinkle. if i remove system-installed pkg-resources *or* setuptools, then system-installed tox is nonfunctional | 00:45 |
| *** fifieldt has quit IRC | 00:47 | |
| *** yaguang has quit IRC | 00:47 | |
| *** yaguang has joined #openstack-infra | 00:47 | |
| dstufft | fungi: yea fucking about with that stuff breaks random shit :/ | 00:47 |
| *** CaptTofu has joined #openstack-infra | 00:48 | |
| fungi | dstufft: no kidding | 00:48 |
| *** yamahata has joined #openstack-infra | 00:48 | |
| dstufft | fungi: there's a reason I suggested just using a venv :) | 00:48 |
| * dstufft has been down this road more than once | 00:48 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add Ironic check jobs via tripleo-ci https://review.openstack.org/81608 | 00:52 |
| *** unicell has quit IRC | 00:53 | |
| *** unicell has joined #openstack-infra | 00:54 | |
| *** andreaf has quit IRC | 00:54 | |
| openstackgerrit | A change was merged to openstack-infra/config: Run neutron functional job as tempest user https://review.openstack.org/81400 | 00:55 |
| *** rpodolyaka1 has joined #openstack-infra | 00:55 | |
| *** hogepodge has joined #openstack-infra | 00:56 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Fix pip on py3k-precise nodes https://review.openstack.org/75213 | 00:59 |
| fungi | okay, ^ that seems to solve the current python33 job issues | 00:59 |
| *** rpodolyaka1 has quit IRC | 01:00 | |
| *** mgagne has joined #openstack-infra | 01:03 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters. https://review.openstack.org/70636 | 01:04 |
| *** amcrn has quit IRC | 01:06 | |
| marun | fungi: I'm afraid I screwed up in approving that patch, we need to update the commit message. | 01:06 |
| marun | fungi: at some point it drifted away from what is actually done. | 01:06 |
| *** khyati has quit IRC | 01:09 | |
| fungi | marun: well, either unapprove/-2 it and it will fail to merge (which will cause a gate reset), or upload a new patchset which will snipe it out of the gate (and cause a gate reset), or let it merge in its current state (if all its jobs succeed) | 01:09 |
| marun | fungi: I've unapproved/-2, will let you know when it gets updated. | 01:10 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow merge failures to have unique reporters. https://review.openstack.org/70636 | 01:12 |
| *** sandywalsh has quit IRC | 01:17 | |
| *** mgagne1 has joined #openstack-infra | 01:19 | |
| *** mgagne has quit IRC | 01:21 | |
| *** fifieldt has joined #openstack-infra | 01:22 | |
| *** nosnos has joined #openstack-infra | 01:25 | |
| *** yaguang has quit IRC | 01:25 | |
| *** yaguang has joined #openstack-infra | 01:26 | |
| jeblair | fungi, marun: i really wanted sdague to review https://review.openstack.org/81400 | 01:27 |
| *** alexpilotti_ has joined #openstack-infra | 01:28 | |
| marun | jeblair: ah, sorry. well, we can revert if it's not to his liking. I think he approved it in the first place, though. | 01:28 |
| jeblair | i believe he has some domain expertise that is relevant | 01:28 |
| jeblair | i don't see him anywhere on the review | 01:28 |
| marun | jeblair: hmmm, sad to say he didn't look at the original change either: https://review.openstack.org/#/c/66967/ | 01:29 |
| marun | jeblair: I thought he had for some reason. He was the person I discussed this with at the tempest/neutron meetup in montreal in January. | 01:30 |
| marun | jeblair: The approach I took was entirely at his direction. | 01:30 |
| *** alexpilotti has quit IRC | 01:30 | |
| *** alexpilotti_ is now known as alexpilotti | 01:30 | |
| jeblair | marun: ok; mostly i know there was some weird stuff at some point in tempest regarding sudo and neutron. i wasn't sure if any of it is relevant, so i wanted him to see that to make those connections since i didn't have the relevant info in my brain | 01:31 |
| jeblair | marun: it's not urgent or anything, i just want to make sure that someone with all of that context sees it, and i know it's not me :) | 01:31 |
| jeblair | sdague: a post-hoc review of https://review.openstack.org/#/c/81400/2 would still be useful i think | 01:32 |
| marun | jeblair: it's not about tempest, though. | 01:32 |
| marun | jeblair: (the commit message is sadly outdated) | 01:32 |
| marun | jeblair: but yes, he should take a look. | 01:33 |
| *** salv-orlando has quit IRC | 01:35 | |
| jeblair | fungi: pip change lgtm | 01:36 |
| jeblair | clarkb: does your repo have submodules? | 01:37 |
| clarkb | jeblair: no, it was nova and ceilometer | 01:37 |
| *** thuc has joined #openstack-infra | 01:37 | |
| clarkb | I guess someone could've stuck submodules in a branch... | 01:37 |
| clarkb | ugh | 01:37 |
| jeblair | clarkb: would be good to confirm since the issue mentions them | 01:37 |
| clarkb | yeah I will look at the various branches tomorrow | 01:38 |
| *** SumitNaiksatam has quit IRC | 01:39 | |
| jeblair | clarkb: do you have a stacktrace of the zuul merger when it was stuck? | 01:39 |
| jeblair | clarkb: exactly what git python was doing at the time might be interesting | 01:40 |
| clarkb | jeblair: I do, and I have ps output and strace stuff from a couple git processes, I can put that all together in the morning and paste it | 01:40 |
| clarkb | currently doing dinner at home and probably won't manage to get on the node with the data until tomorrow | 01:42 |
| *** thuc has quit IRC | 01:42 | |
| *** alexpilotti has quit IRC | 01:44 | |
| jeblair | clarkb: possibly it needs to use something like http://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate | 01:45 |
| jeblair | clarkb: which apparently uses threads on windows to work around the winsock issue | 01:46 |
| *** chandan_kumar has joined #openstack-infra | 01:53 | |
| *** Ryan_Lane has quit IRC | 01:54 | |
| *** sandywalsh has joined #openstack-infra | 01:57 | |
| *** maxbit has quit IRC | 01:57 | |
| *** dcramer_ has quit IRC | 01:59 | |
| *** zehicle_at_dell has quit IRC | 01:59 | |
| *** zehicle_at_dell has joined #openstack-infra | 02:00 | |
| *** wenlock has quit IRC | 02:00 | |
| *** shakamunyi has joined #openstack-infra | 02:00 | |
| clarkb | well we aren't using windows. Is that bug a windows only thing? I noticed one of the testers was windowsing but I didn't htink all of them were | 02:01 |
| clarkb | huh it is windows there so maybe this is a different issue, but similar | 02:02 |
| *** sdake_ has quit IRC | 02:04 | |
| *** _alexandra_ is now known as alex-luncheon | 02:07 | |
| *** thuc has joined #openstack-infra | 02:12 | |
| *** nosnos_ has joined #openstack-infra | 02:16 | |
| *** nosnos has quit IRC | 02:19 | |
| *** gyee has quit IRC | 02:20 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:22 | |
| *** thuc_ has joined #openstack-infra | 02:25 | |
| *** mburned has quit IRC | 02:26 | |
| *** mburned has joined #openstack-infra | 02:27 | |
| *** derekh has quit IRC | 02:27 | |
| *** mburned has quit IRC | 02:27 | |
| *** jepoy_ has quit IRC | 02:27 | |
| *** thuc has quit IRC | 02:28 | |
| *** miqui has quit IRC | 02:28 | |
| *** che-arne has quit IRC | 02:29 | |
| *** thuc_ has quit IRC | 02:31 | |
| *** saschpe has quit IRC | 02:32 | |
| *** thuc has joined #openstack-infra | 02:32 | |
| *** saschpe has joined #openstack-infra | 02:33 | |
| *** sdake_ has joined #openstack-infra | 02:35 | |
| *** sdake_ has quit IRC | 02:35 | |
| *** sdake_ has joined #openstack-infra | 02:35 | |
| *** sabari2 has joined #openstack-infra | 02:36 | |
| *** mwagner__ is now known as mwagner__zzzz | 02:37 | |
| kevinbenton | fungi, clarkb: can you rebump https://review.openstack.org/#/c/80413/ to the top? | 02:37 |
| kevinbenton | or jeblair ^^ | 02:38 |
| kevinbenton | not sure who to ask | 02:38 |
| kevinbenton | it's a fix for some of the gate db lock issues | 02:38 |
| *** caleb_ has joined #openstack-infra | 02:42 | |
| *** jepoy has joined #openstack-infra | 02:44 | |
| *** khyati has joined #openstack-infra | 02:46 | |
| *** mriedem has quit IRC | 02:48 | |
| *** jpeeler has quit IRC | 02:49 | |
| *** dcramer_ has joined #openstack-infra | 02:50 | |
| mattoliverau | jeblair, clarkb, fungi or any other cores still up, I have some friends from rackspace in the US wondering about this change 79058, would you guys be able to take a look? | 02:52 |
| *** mriedem has joined #openstack-infra | 02:56 | |
| *** rpodolyaka1 has joined #openstack-infra | 02:57 | |
| *** thuc has quit IRC | 02:59 | |
| *** thuc has joined #openstack-infra | 03:00 | |
| *** rpodolyaka1 has quit IRC | 03:02 | |
| *** thuc has quit IRC | 03:04 | |
| *** mburned has joined #openstack-infra | 03:07 | |
| *** jepoy has quit IRC | 03:17 | |
| *** jepoy has joined #openstack-infra | 03:19 | |
| *** chandan_kumar has quit IRC | 03:22 | |
| *** harlowja is now known as harlowja_away | 03:23 | |
| *** thuc has joined #openstack-infra | 03:24 | |
| *** gtt116 has joined #openstack-infra | 03:25 | |
| *** pcrews has quit IRC | 03:25 | |
| *** jepoy has quit IRC | 03:29 | |
| *** jepoy has joined #openstack-infra | 03:33 | |
| *** CaptTofu has quit IRC | 03:39 | |
| *** CaptTofu has joined #openstack-infra | 03:39 | |
| *** CaptTofu_ has joined #openstack-infra | 03:41 | |
| *** CaptTofu has quit IRC | 03:41 | |
| *** dcramer_ has quit IRC | 03:45 | |
| *** matsuhashi has quit IRC | 03:47 | |
| *** alex-luncheon is now known as _alexandra_ | 03:48 | |
| *** david-lyle has joined #openstack-infra | 03:50 | |
| *** rpodolyaka1 has joined #openstack-infra | 03:58 | |
| *** dcramer_ has joined #openstack-infra | 03:59 | |
| *** rpodolyaka1 has quit IRC | 04:03 | |
| *** mriedem has quit IRC | 04:04 | |
| *** mspreitz has joined #openstack-infra | 04:05 | |
| openstackgerrit | Martin Konrad proposed a change to openstack-infra/jenkins-job-builder: SCM module: Add support for multiple Git remotes. https://review.openstack.org/80084 | 04:06 |
| mspreitz | What's wrong with the gate? | 04:07 |
| *** sdake_ has quit IRC | 04:08 | |
| *** jepoy has quit IRC | 04:09 | |
| *** _nadya_ has joined #openstack-infra | 04:09 | |
| *** gokrokve has joined #openstack-infra | 04:09 | |
| *** CaptTofu_ has quit IRC | 04:12 | |
| *** saju_m has joined #openstack-infra | 04:12 | |
| *** CaptTofu has joined #openstack-infra | 04:12 | |
| *** CaptTofu has quit IRC | 04:16 | |
| *** thuc has quit IRC | 04:20 | |
| *** thuc has joined #openstack-infra | 04:21 | |
| *** sweston has joined #openstack-infra | 04:21 | |
| *** thuc has quit IRC | 04:25 | |
| *** mihgen has joined #openstack-infra | 04:28 | |
| *** matsuhashi has joined #openstack-infra | 04:31 | |
| *** thuc has joined #openstack-infra | 04:41 | |
| *** gokrokve has quit IRC | 04:49 | |
| *** thuc has quit IRC | 04:49 | |
| *** gokrokve has joined #openstack-infra | 04:55 | |
| *** _nadya_ has quit IRC | 04:58 | |
| *** khyati has quit IRC | 05:08 | |
| *** _nadya_ has joined #openstack-infra | 05:09 | |
| *** chandan_kumar has joined #openstack-infra | 05:10 | |
| *** yaguang has quit IRC | 05:13 | |
| *** mspreitz has quit IRC | 05:16 | |
| *** chandan_kumar has quit IRC | 05:17 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Only attempt to rename tarballs in tarball job. https://review.openstack.org/81702 | 05:18 |
| clarkb | mordred: ^ | 05:18 |
| *** yongli has joined #openstack-infra | 05:19 | |
| *** nosnos has joined #openstack-infra | 05:19 | |
| *** sdake_ has joined #openstack-infra | 05:21 | |
| *** nosnos_ has quit IRC | 05:23 | |
| *** jepoy has joined #openstack-infra | 05:23 | |
| *** nicedice has quit IRC | 05:25 | |
| *** ildikov_ has quit IRC | 05:26 | |
| *** rpodolyaka1 has joined #openstack-infra | 05:28 | |
| *** amotoki has joined #openstack-infra | 05:29 | |
| *** amcrn has joined #openstack-infra | 05:31 | |
| *** vkozhukalov_ has joined #openstack-infra | 05:33 | |
| *** gokrokve_ has joined #openstack-infra | 05:34 | |
| kevinbenton | how do things make it into the openstack/common/lockutils.py folder? | 05:36 |
| kevinbenton | s/folder/module/ | 05:36 |
| clarkb | kevinbenton: they come from oslo-incubator and get synced into the various projects | 05:37 |
| kevinbenton | clarkb: so is the best way to get something in to just put it somewhere in the project first to prove it's useful? | 05:37 |
| clarkb | no I think you have to get it into oslo first | 05:37 |
| *** jepoy_ has joined #openstack-infra | 05:38 | |
| clarkb | the idea being we want to avoid snowflakes not created them | 05:38 |
| *** gokrokve has quit IRC | 05:38 | |
| kevinbenton | so how do you prove it's useful first? | 05:38 |
| clarkb | I dunno dhellman can probably explain it much better than I (and maybe they do wnat it in individual projects first but last I heard that was a no no) | 05:39 |
| kevinbenton | clarkb: ok | 05:39 |
| clarkb | kevinbenton: see http://lists.openstack.org/pipermail/openstack-dev/2014-March/029720.html | 05:40 |
| *** jepoy has quit IRC | 05:41 | |
| kevinbenton | clarkb: thanks | 05:42 |
| *** nosnos_ has joined #openstack-infra | 05:49 | |
| *** mgagne1 has quit IRC | 05:50 | |
| mordred | clarkb: looking | 05:50 |
| *** nosnos has quit IRC | 05:51 | |
| *** yaguang has joined #openstack-infra | 05:54 | |
| *** talluri has joined #openstack-infra | 05:56 | |
| *** thuc has joined #openstack-infra | 05:57 | |
| *** _nadya_ has quit IRC | 05:57 | |
| *** gokrokve_ has quit IRC | 05:57 | |
| *** thuc has quit IRC | 06:02 | |
| *** saju_m has quit IRC | 06:02 | |
| *** mrda is now known as mrda_away | 06:02 | |
| *** nati_ueno has joined #openstack-infra | 06:03 | |
| *** saju_m has joined #openstack-infra | 06:04 | |
| anteaya | mattoliverau: that is the satori gerritbot patch, it just needs a core to +A it, 5 +1 votes and a happy jenkins | 06:05 |
| *** saju_m has quit IRC | 06:07 | |
| *** saju_m has joined #openstack-infra | 06:08 | |
| kevinbenton | clarkb: do you want to push 81025,4 to the top on the next reset? | 06:09 |
| *** vkozhukalov_ has quit IRC | 06:19 | |
| *** matsuhashi has quit IRC | 06:28 | |
| *** jepoy has joined #openstack-infra | 06:29 | |
| *** matsuhashi has joined #openstack-infra | 06:30 | |
| *** jepoy_ has quit IRC | 06:32 | |
| *** _alexandra_ is now known as alex-gone | 06:33 | |
| mattoliverau | anteaya: Hey! Welcome back btw, you've been missed :) | 06:34 |
| anteaya | mattoliverau: ha ha ha, thanks | 06:34 |
| anteaya | how are you? | 06:34 |
| *** saju_m has quit IRC | 06:35 | |
| mattoliverau | I'm good, been busy flying to the states and back, it's good to be home and working on fun things again :) | 06:35 |
| anteaya | I hear that | 06:35 |
| * mattoliverau says that while sitting in an airport about to fly to a different city | 06:35 | |
| anteaya | :D | 06:36 |
| anteaya | typical | 06:36 |
| anteaya | how goes the groking of -infra? | 06:36 |
| mattoliverau | It's going well, I get involved in all the config related questions and discussions and now that I'm hacking nodepool to create TH instances, jeblair is kind enough to let me, or rather asks me sometimes to look at node pool patchs, which is nice of him for thinking about me. I keep an eye on the infra code reviews and try and help out there where I can. | 06:38 |
| *** alff has joined #openstack-infra | 06:38 | |
| anteaya | woohoo, well done | 06:38 |
| anteaya | I'm glad to hear you feel included | 06:38 |
| mattoliverau | so all in all going well. I tend to shadow some of the infra meetings and insert my 2 cents when dealing with things I have experience with, like Linux system stuff :) | 06:39 |
| anteaya | that is the way to do it | 06:39 |
| anteaya | way to go | 06:39 |
| mattoliverau | Very, infra is a very inclusive group, great culture, me linking alot :) | 06:39 |
| anteaya | so happy to hear that, yay! | 06:39 |
| mattoliverau | anteaya: Were you on holiday this time? or travelling for work again? | 06:40 |
| anteaya | holiday | 06:40 |
| mattoliverau | nice :) | 06:40 |
| anteaya | I needed some time to recharge | 06:40 |
| anteaya | I made a great choice | 06:40 |
| anteaya | I have a friend who is thai and she spends a few months in her home village every year | 06:41 |
| anteaya | they have a new house and a spare room | 06:41 |
| anteaya | the whole village wanted to meet me | 06:41 |
| anteaya | many of them had never seen a white woman before | 06:41 |
| anteaya | all the women wanted to touch my skin | 06:41 |
| anteaya | it got to the point when greeting a group of them I just offered them an arm | 06:41 |
| anteaya | felt like I was the attraction at a petting zoo | 06:42 |
| anteaya | but they were very kind and gentle | 06:42 |
| mattoliverau | Wow, that sounds incredible! I havne't quite made it to thai land, been to vietnam, so close, but I should go, I'm not that far away :) | 06:42 |
| anteaya | taught pronounciation at the local school | 06:42 |
| anteaya | thailand was great for me, would not have been so much fun if I hadn't stayed with my friend | 06:42 |
| anteaya | the children can read english but they can't pronounce the words | 06:43 |
| anteaya | or recognized them when they hear them | 06:43 |
| anteaya | so I talked english at them | 06:43 |
| anteaya | and helped them learn L | 06:43 |
| anteaya | they have trouble with L, T, R, ch, sh, V | 06:44 |
| mattoliverau | cool, yeah I think you need to stay with someone who can take you to see the real people, being a simple tourest isn't quite the same. | 06:44 |
| *** rpodolyaka1 has quit IRC | 06:44 | |
| anteaya | so we got the group of them to learn L and some got T as well, the aventurous ones | 06:44 |
| anteaya | agreed | 06:44 |
| anteaya | I watched the cows walk past the kitchen window every morning and then return at the end of the day | 06:45 |
| mattoliverau | Ha ha, sounds like a real out of the box experience! | 06:45 |
| anteaya | my kind of experience | 06:45 |
| anteaya | boxes don't last long for me | 06:45 |
| mattoliverau | So you couldn't get much further from your normal life :) | 06:45 |
| anteaya | which was exactly what I wanted | 06:45 |
| mattoliverau | I'm impressed! That sounds amazing! | 06:46 |
| anteaya | her partner was a little miffed at me since I wouldn't fix his phone and tablet, but after a day or so he got over it | 06:46 |
| anteaya | thanks, I really needed it | 06:46 |
| anteaya | it was great | 06:46 |
| mattoliverau | Haha, the IT curse, your everyone's tech support, good on you for holding your ground and honouring your holiday :) | 06:47 |
| anteaya | thanks | 06:47 |
| anteaya | he didn't talk to me for a few days, but that was fine with me | 06:48 |
| mattoliverau | lol, you had children to teach anyway | 06:48 |
| anteaya | cows, skinny chickens, more fruit and herbs than I could imagine | 06:48 |
| anteaya | the children were fantastic, they worked so hard | 06:48 |
| anteaya | I could see it in their eyes | 06:48 |
| mattoliverau | Well now I want to go on holidays somewhere exotic, and I'm lucky, I'm quite close to asia already :) | 06:49 |
| anteaya | they know how important it is to learn how to pronounce english | 06:49 |
| anteaya | it was so gratifying to help them, even to learn one letter | 06:49 |
| anteaya | bonus | 06:49 |
| anteaya | bangkok is fun, I was there one night, stayed with the daughter of a friend of my friend, so again the local experience | 06:50 |
| anteaya | but my preference is to spend time in the country | 06:50 |
| *** _nadya_ has joined #openstack-infra | 06:50 | |
| anteaya | they have outdoor education centers, which I would recommend if you want something genuine | 06:51 |
| *** _nadya_ has quit IRC | 06:51 | |
| *** nati_ueno has quit IRC | 06:51 | |
| mattoliverau | So away from the city and into the villages, the real people :) | 06:51 |
| anteaya | yes | 06:51 |
| anteaya | this one is the funniest: http://www.cabbagesandcondoms.net/Index.htm | 06:52 |
| anteaya | it is a series of educational resorts designed to teach people about agriculture, public health and family planning | 06:52 |
| anteaya | tons of fruit trees in the properties | 06:52 |
| mattoliverau | lol, what a combination | 06:53 |
| anteaya | yeah, I laughed | 06:53 |
| anteaya | but apparently it is effective | 06:53 |
| *** masayukig has quit IRC | 06:53 | |
| mattoliverau | That's what matters | 06:53 |
| anteaya | the condom icon makes them friendly and approachable so people are willing to ask questions | 06:53 |
| *** wchrisj has quit IRC | 06:54 | |
| anteaya | I was really surprised when I saw the sign though | 06:54 |
| anteaya | we stopped into one to book a room for a family friend | 06:54 |
| anteaya | for the wedding | 06:54 |
| anteaya | went to a wedding and an interment service while I was there | 06:54 |
| anteaya | I had no idea | 06:54 |
| anteaya | as soon as I arrived my friend told me so I had to get clothes for the wedding | 06:55 |
| anteaya | fortunately the village is filled with silk weavers | 06:55 |
| anteaya | :D | 06:55 |
| mattoliverau | lol, so your were along for the ride. Nice, so I assume you own alot of silk now? | 06:55 |
| anteaya | woohooo, and cotton and is it beautiful | 06:56 |
| anteaya | once the village found out I like textiles, they started to gift me with them | 06:56 |
| anteaya | so they could have the honour of making sure I had something from them | 06:56 |
| anteaya | blew me away | 06:56 |
| anteaya | methinks I need to tidy my house though, since if any of them visit I will be putting them up for a few days | 06:57 |
| anteaya | but that is pretty easy to take | 06:57 |
| *** rlandy has joined #openstack-infra | 06:57 | |
| mattoliverau | Wow, that's very genrous of them! Sounds like an amazing adventure, hopefully you made some friends for life! | 06:58 |
| anteaya | I think i have | 06:58 |
| mattoliverau | Anyway time to board the plane. Welcome back anteaya | 06:58 |
| anteaya | thanks | 06:59 |
| anteaya | safe travels | 06:59 |
| *** rpodolyaka1 has joined #openstack-infra | 06:59 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Tidy up app.js - limit lines to 80chars https://review.openstack.org/81716 | 07:01 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Build status page more jquery like https://review.openstack.org/81717 | 07:01 |
| *** sweston has quit IRC | 07:02 | |
| *** caleb_ has quit IRC | 07:08 | |
| *** caleb_ has joined #openstack-infra | 07:08 | |
| *** yolanda has joined #openstack-infra | 07:10 | |
| *** jlibosva has joined #openstack-infra | 07:12 | |
| *** shakamunyi has quit IRC | 07:15 | |
| *** jlibosva has quit IRC | 07:16 | |
| *** rhsu has quit IRC | 07:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Only attempt to rename tarballs in tarball job. https://review.openstack.org/81702 | 07:18 |
| *** rpodolyaka1 has quit IRC | 07:19 | |
| mordred | clarkb: ^^ merged. thanks | 07:23 |
| *** unicell has quit IRC | 07:24 | |
| *** sabari2 has quit IRC | 07:26 | |
| *** lukego has joined #openstack-infra | 07:27 | |
| *** talluri has quit IRC | 07:30 | |
| *** talluri_ has joined #openstack-infra | 07:31 | |
| *** yjiang has joined #openstack-infra | 07:32 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: [WIP] Comments controller https://review.openstack.org/81505 | 07:32 |
| *** AJaeger has quit IRC | 07:38 | |
| *** dstanek has quit IRC | 07:38 | |
| *** nati_ueno has joined #openstack-infra | 07:39 | |
| *** rhsu has joined #openstack-infra | 07:41 | |
| *** shakamunyi has joined #openstack-infra | 07:41 | |
| *** jlibosva has joined #openstack-infra | 07:43 | |
| *** skraynev_afk is now known as skraynev | 07:45 | |
| *** shakamunyi has quit IRC | 07:45 | |
| *** vkozhukalov_ has joined #openstack-infra | 07:49 | |
| *** rhsu has quit IRC | 07:49 | |
| *** flaper87|afk is now known as flaper87 | 07:50 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add PuppetBoard to openstack infra https://review.openstack.org/77928 | 07:52 |
| *** jepoy_ has joined #openstack-infra | 07:53 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: [WIP] Comments controller https://review.openstack.org/81505 | 07:55 |
| *** saju_m has joined #openstack-infra | 07:56 | |
| *** jepoy has quit IRC | 07:56 | |
| *** caleb_` has joined #openstack-infra | 07:56 | |
| *** andreaf has joined #openstack-infra | 07:56 | |
| *** caleb_ has quit IRC | 07:58 | |
| *** caleb_` has quit IRC | 08:06 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Reuse floating ips https://review.openstack.org/80893 | 08:07 |
| nibalizer | anteaya: awesome! +1ing before jenkins | 08:10 |
| anteaya | the benefits of jet lag | 08:10 |
| anteaya | I'm awake | 08:10 |
| anteaya | also there is a cascade effect about to happen once puppetboard is in | 08:11 |
| anteaya | I do believe | 08:11 |
| anteaya | a pip patch then becomes viable | 08:11 |
| anteaya | which gives breathing room to oh so many people | 08:12 |
| anteaya | such as olso | 08:12 |
| anteaya | according to my most recent skim of the backscroll | 08:12 |
| anteaya | well oslo isn't people, but I think you know what I mean | 08:13 |
| nibalizer | im actually not sure what oslo is, but okay | 08:15 |
| anteaya | ah sorry | 08:15 |
| anteaya | allow me to fill in the blanks then | 08:16 |
| nibalizer | ok | 08:16 |
| anteaya | olso is common libraries and code | 08:16 |
| anteaya | they start in oslo-incubator: http://git.openstack.org/cgit/openstack/oslo-incubator/tree/README.rst | 08:17 |
| anteaya | the movitation is code reuse across projects | 08:17 |
| nibalizer | ok that makes sense | 08:18 |
| anteaya | so if code is in a project and another project wants to use the same functionality, the idea is to put it into olso-incubator to get the functionaly correct | 08:18 |
| anteaya | now olso is rapidly expanding | 08:18 |
| nibalizer | how does puppetboard help that? or is it the greater control of pip that does that? | 08:18 |
| anteaya | and has many other repos associated with it: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n137 | 08:19 |
| anteaya | nibalizer: well after puppetboard goes in, we are going to soft freeze puppet changes | 08:19 |
| *** jcoufal has joined #openstack-infra | 08:19 | |
| anteaya | allowing for a pip patch to merge, which keeps having to be rebased | 08:20 |
| anteaya | due to puppet changes | 08:20 |
| nibalizer | yea, i was following that at the meeting on tuseday | 08:20 |
| nibalizer | I hope I haven't oversold puppetboard, its not perfect | 08:20 |
| anteaya | https://review.openstack.org/#/c/51425/ | 08:20 |
| anteaya | it is better than what we have currently | 08:20 |
| nibalizer | for instance if catalog compilation fails puppetboard doesn't get to know about that | 08:20 |
| nibalizer | anteaya: thats certainly true | 08:21 |
| anteaya | mostly we want it in before the soft freeze on puppet changes | 08:21 |
| * nibalizer wonders if anyone reads /var/log/puppet | 08:21 | |
| anteaya | since it is so close | 08:21 |
| anteaya | how do I access /var/log/puppet? | 08:21 |
| anteaya | it is on the puppet master server? | 08:21 |
| nibalizer | yes | 08:21 |
| anteaya | if yes, I don't have access | 08:21 |
| nibalizer | its more lkie /var/log/puppet<tab> because I don't remember the exact log file right now | 08:22 |
| anteaya | so only 4 people can access that info | 08:22 |
| *** dizquierdo has joined #openstack-infra | 08:22 | |
| anteaya | close enough | 08:22 |
| anteaya | so puppetboard patch is simply the trigger for a cascade of other actions | 08:22 |
| anteaya | the next action being the soft freeze on puppet changes until 51425 gets in | 08:23 |
| nibalizer | yea 51425 is a huge change | 08:23 |
| anteaya | which will make many people happy, who are currently less so now | 08:23 |
| anteaya | yes | 08:23 |
| anteaya | and having puppetboard available with help with the debugging once it goes in | 08:23 |
| anteaya | s/with/will | 08:24 |
| *** rpodolyaka1 has joined #openstack-infra | 08:24 | |
| *** amcrn has quit IRC | 08:26 | |
| nibalizer | I agree | 08:27 |
| anteaya | great | 08:27 |
| anteaya | nothing is perfect | 08:27 |
| anteaya | but waiting for it to be so hinders other action | 08:27 |
| *** johnthetubaguy has joined #openstack-infra | 08:27 | |
| nibalizer | I'm noticing some tailing whitespace on 51425, is it correct ediquitte to review on this? | 08:27 |
| anteaya | yes | 08:28 |
| anteaya | do review | 08:28 |
| nibalizer | do i -1 ? | 08:28 |
| *** jgallard has joined #openstack-infra | 08:28 | |
| anteaya | I haven't looked at 51425 myself at all, just aware of it's existance | 08:28 |
| anteaya | hmmmm, that is certainly a choice you can make | 08:28 |
| *** rpodolyaka1 has quit IRC | 08:29 | |
| nibalizer | i'll noscore it, just to all it to the author's attention | 08:29 |
| anteaya | I don't have a hard and fast rule for -1'ing on whitespace errors | 08:29 |
| anteaya | yeah | 08:29 |
| anteaya | I think I go by the author's familiarity with the process | 08:29 |
| nibalizer | not errors, but gerrit makes them RED and obvious so might as well fix it | 08:29 |
| anteaya | for a new contributor, I would -1 to ensure I had their attention | 08:29 |
| anteaya | for a regular contributor who I think just had typo I would just no score | 08:30 |
| anteaya | right, might as well fix it | 08:30 |
| *** jgallard has quit IRC | 08:33 | |
| *** jgallard has joined #openstack-infra | 08:33 | |
| *** nosnos has joined #openstack-infra | 08:33 | |
| *** SumitNaiksatam has quit IRC | 08:35 | |
| *** nosnos_ has quit IRC | 08:37 | |
| *** shakamunyi has joined #openstack-infra | 08:42 | |
| *** hashar has joined #openstack-infra | 08:44 | |
| *** shakamunyi has quit IRC | 08:46 | |
| *** nati_ueno has quit IRC | 08:47 | |
| marun | if someone could bump this job, we might avoid some unnecessary failures: https://review.openstack.org/#/c/80413/ | 08:48 |
| marun | it was bumped earlier but we needed to fix the commit message | 08:48 |
| marun | it addresses part of the reason https://bugs.launchpad.net/neutron/+bug/1283522 is occuring | 08:49 |
| uvirtbot | Launchpad bug 1283522 in neutron "DB lock timeout errors with parallel operations" [Critical,In progress] | 08:49 |
| marun | anteaya: ^ ? | 08:49 |
| *** mihgen has quit IRC | 08:50 | |
| *** morganfainberg is now known as morganfainberg_Z | 08:54 | |
| anteaya | hi marun, I would bump if I could | 08:57 |
| anteaya | alas I have no such power | 08:57 |
| *** ihrachys|afk is now known as ihrachys | 08:57 | |
| anteaya | mordred do you have the power of the bump? | 08:57 |
| *** nati_ueno has joined #openstack-infra | 08:58 | |
| *** nati_uen_ has joined #openstack-infra | 08:59 | |
| *** bada_ has quit IRC | 08:59 | |
| *** bada_ has joined #openstack-infra | 09:00 | |
| *** apevec has joined #openstack-infra | 09:02 | |
| *** yassine has joined #openstack-infra | 09:03 | |
| *** nati_ueno has quit IRC | 09:03 | |
| apevec | ttx, adam_g, ping - I'm doing stable/grizzly 2013.1.5 release today, the final one! I'd like to see if Ubuntu has CI result from stable/grizzly before I push tags? | 09:04 |
| *** jp_at_hp has joined #openstack-infra | 09:04 | |
| ttx | apevec: no idea, maybe jamespage knows ? | 09:04 |
| jamespage | ttx, lemme check | 09:05 |
| *** lttrl has joined #openstack-infra | 09:08 | |
| jamespage | ttx, apevec: I don't right now - I'm mid upgrade to trusty on the CI lab and I broke MAAS re-ip'ing the main network last night | 09:10 |
| jamespage | apevec, so don't block on Ubuntu CI please | 09:10 |
| apevec | jamespage, ok thanks | 09:10 |
| jamespage | apevec, sorry - not ideal I know | 09:10 |
| jamespage | I will run it as soon as I have a working network again :-) | 09:11 |
| apevec | I won't push tags before EOD today, so there's a chance :) | 09:12 |
| *** hashar has quit IRC | 09:18 | |
| *** derekh has joined #openstack-infra | 09:20 | |
| * anteaya is experiencing a wave of tired and is going to give sleep another try | 09:24 | |
| *** rpodolyaka1 has joined #openstack-infra | 09:25 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: make neutron not voting in the gate https://review.openstack.org/81741 | 09:27 |
| *** lttrl has quit IRC | 09:28 | |
| *** lttrl has joined #openstack-infra | 09:28 | |
| *** markmc has joined #openstack-infra | 09:29 | |
| *** rpodolyaka1 has quit IRC | 09:29 | |
| *** roeyc has joined #openstack-infra | 09:36 | |
| *** hashar has joined #openstack-infra | 09:38 | |
| *** psedlak_clone is now known as psedlak | 09:39 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added db_api for comments https://review.openstack.org/81232 | 09:41 |
| *** dizquierdo is now known as dizquierdo_afk | 09:41 | |
| sdague | SergeyLukjanov: do you have access to do promotes? | 09:43 |
| *** dpyzhov has joined #openstack-infra | 09:45 | |
| *** mihgen has joined #openstack-infra | 09:47 | |
| *** e0ne has joined #openstack-infra | 09:48 | |
| *** SumitNaiksatam has joined #openstack-infra | 09:53 | |
| *** dpyzhov has quit IRC | 09:56 | |
| *** dpyzhov has joined #openstack-infra | 09:56 | |
| *** nosnos has quit IRC | 09:59 | |
| *** ominakov has joined #openstack-infra | 09:59 | |
| *** dpyzhov has quit IRC | 10:01 | |
| *** yamahata has quit IRC | 10:02 | |
| *** yjiang has quit IRC | 10:03 | |
| *** jcoufal has quit IRC | 10:06 | |
| *** nosnos has joined #openstack-infra | 10:09 | |
| *** lttrl has quit IRC | 10:09 | |
| *** bada_ has quit IRC | 10:15 | |
| *** rwsu has quit IRC | 10:18 | |
| *** dpyzhov has joined #openstack-infra | 10:20 | |
| *** rpodolyaka1 has joined #openstack-infra | 10:25 | |
| *** e0ne has quit IRC | 10:28 | |
| *** rpodolyaka1 has quit IRC | 10:30 | |
| *** afazekas has joined #openstack-infra | 10:31 | |
| *** e0ne has joined #openstack-infra | 10:32 | |
| *** rwsu has joined #openstack-infra | 10:33 | |
| *** hdd has quit IRC | 10:33 | |
| *** rlandy has quit IRC | 10:40 | |
| *** lttrl has joined #openstack-infra | 10:41 | |
| *** nati_uen_ has quit IRC | 10:43 | |
| *** nati_ueno has joined #openstack-infra | 10:44 | |
| *** hdd has joined #openstack-infra | 10:48 | |
| *** nati_ueno has quit IRC | 10:48 | |
| *** fifieldt has quit IRC | 10:49 | |
| *** rlandy has joined #openstack-infra | 10:52 | |
| *** yamahata has joined #openstack-infra | 10:54 | |
| *** yassine has quit IRC | 10:56 | |
| *** chandankumar_ has quit IRC | 11:00 | |
| *** chandankumar has joined #openstack-infra | 11:01 | |
| *** matsuhashi has quit IRC | 11:01 | |
| *** Kiall has quit IRC | 11:02 | |
| *** ihrachys is now known as ihrachys|lunch | 11:02 | |
| *** homeless has quit IRC | 11:03 | |
| *** jhesketh_ has quit IRC | 11:03 | |
| *** kiall has joined #openstack-infra | 11:04 | |
| *** ProfFalken has quit IRC | 11:04 | |
| *** homeless has joined #openstack-infra | 11:05 | |
| *** ProfFalken has joined #openstack-infra | 11:05 | |
| *** roeyc has quit IRC | 11:08 | |
| *** jhesketh__ has quit IRC | 11:08 | |
| *** hashar has quit IRC | 11:16 | |
| *** dizquierdo_afk is now known as dizquierdo | 11:20 | |
| *** e0ne_ has joined #openstack-infra | 11:20 | |
| *** lcostantino has joined #openstack-infra | 11:20 | |
| *** jhesketh__ has joined #openstack-infra | 11:22 | |
| *** jhesketh__ has quit IRC | 11:22 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add /sbin to the net-info job builder path https://review.openstack.org/81641 | 11:22 |
| *** CaptTofu has joined #openstack-infra | 11:23 | |
| *** e0ne has quit IRC | 11:24 | |
| *** matsuhashi has joined #openstack-infra | 11:24 | |
| *** jgallard has quit IRC | 11:25 | |
| *** mbacchi has joined #openstack-infra | 11:25 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack-infra/config: Enable Gearman as default on Jenkins slaves https://review.openstack.org/73214 | 11:26 |
| *** rpodolyaka1 has joined #openstack-infra | 11:26 | |
| *** mflobo has joined #openstack-infra | 11:27 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack-infra/config: Enable Gearman as default on Jenkins slaves https://review.openstack.org/73214 | 11:28 |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack-infra/config: Enable Gearman as default on Jenkins slaves https://review.openstack.org/73214 | 11:29 |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Update gerrit-git-prep.sh to the /p/ when cloning from directly from Gerrit https://review.openstack.org/81772 | 11:29 |
| *** rpodolyaka1 has quit IRC | 11:30 | |
| *** dpyzhov has quit IRC | 11:32 | |
| *** markmcclain has joined #openstack-infra | 11:35 | |
| *** markmcclain1 has joined #openstack-infra | 11:35 | |
| *** markmcclain1 has quit IRC | 11:35 | |
| openstackgerrit | A change was merged to openstack-infra/config: Only initialize transifex if there's no .tx directory https://review.openstack.org/81599 | 11:36 |
| *** markmcclain1 has joined #openstack-infra | 11:36 | |
| *** alexpilotti has joined #openstack-infra | 11:37 | |
| *** markmcclain1 has quit IRC | 11:37 | |
| *** ociuhandu has quit IRC | 11:38 | |
| *** markmcclain has quit IRC | 11:39 | |
| *** markmcclain has joined #openstack-infra | 11:41 | |
| markmcclain | would it be possible to promote something to the top of the gae? | 11:42 |
| markmcclain | s/gae/gate/ | 11:42 |
| sdague | markmcclain: not until someone with infra root wakes up | 11:43 |
| openstackgerrit | A change was merged to openstack-infra/config: Add titles to the times on zuul status's page https://review.openstack.org/80160 | 11:43 |
| markmcclain | sdague: ok.. hopefully soon..have another address to deadlock issue | 11:45 |
| sdague | markmcclain: in addition to the one that was pulled from the gate queue to fix the commit message last night? | 11:46 |
| sdague | or the same one | 11:46 |
| markmcclain | same one | 11:46 |
| markmcclain | sdague: https://review.openstack.org/#/c/80413/ | 11:47 |
| sdague | yeh, my plan was to ask fungi to promote that as soon as he was awake | 11:47 |
| markmcclain | awesome | 11:47 |
| sdague | and if we still see deadlocks happening behind that, turn off neutron voting | 11:47 |
| sdague | because it's been 8 days | 11:47 |
| *** salv-orlando has joined #openstack-infra | 11:47 | |
| sdague | and we really need to let other projects merge their release critical bugs | 11:47 |
| markmcclain | yeah this one has not been fun as it has been hard to really ensure the other proposals fully fix the deadlock issue | 11:49 |
| sdague | it really feels like there is a fundamentals problem here | 11:50 |
| sdague | there was some talk of an agent rewrite to solve this at a fundamental level at one point, right? | 11:50 |
| *** fifieldt has joined #openstack-infra | 11:51 | |
| *** david-lyle has quit IRC | 11:52 | |
| markmcclain | yes there was some work on separate workers, but issue can be triggered with concurrent http requests | 11:52 |
| *** rfolco has joined #openstack-infra | 11:53 | |
| markmcclain | sdague: already working through postmortem on the bug handling as it seems there was social issues that prevented the best solution from being presented first | 11:58 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add missing neutron logs https://review.openstack.org/81782 | 11:59 |
| *** dpyzhov has joined #openstack-infra | 11:59 | |
| *** dpyzhov has quit IRC | 12:03 | |
| sdague | also, lbaas seems to be failing a ton in the pile of neutron gate resets not categorized | 12:03 |
| *** dpyzhov has joined #openstack-infra | 12:04 | |
| *** eharney has quit IRC | 12:04 | |
| *** dpyzhov has quit IRC | 12:04 | |
| sdague | we have 10 resets currently uncategorized in the last 24 for the neutron job, which means even after the deadlock fix, there are a bunch of other issues | 12:04 |
| *** dpyzhov has joined #openstack-infra | 12:05 | |
| *** ociuhandu has joined #openstack-infra | 12:06 | |
| *** mburned has quit IRC | 12:06 | |
| mordred | sdague: I have access to do promotes, although I've never performed one | 12:07 |
| sdague | mordred: https://review.openstack.org/#/c/80413/ | 12:07 |
| sdague | top gate reset issue is hopefully addressed by that | 12:07 |
| russellb | fungi: can you add me to nova-docker-core ? | 12:08 |
| *** matsuhashi has quit IRC | 12:08 | |
| mordred | russellb: no core for you | 12:08 |
| russellb | mordred: or you! | 12:08 |
| russellb | :) | 12:08 |
| mordred | russellb: that's pretty safe :) | 12:08 |
| *** saju_m has quit IRC | 12:09 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Comments controller https://review.openstack.org/81505 | 12:10 |
| *** fifieldt has quit IRC | 12:10 | |
| mordred | sdague: ok - it seems like I just run "zuul promote --pipeline --changes 80413" | 12:11 |
| *** yaguang has quit IRC | 12:12 | |
| mordred | sdague: I still need to wait fora reset, yeah? | 12:12 |
| sdague | mordred: you might need patch name | 12:12 |
| sdague | patch # | 12:12 |
| *** dpyzhov has quit IRC | 12:12 | |
| *** matsuhashi has joined #openstack-infra | 12:12 | |
| mordred | ah | 12:12 |
| *** yassine has joined #openstack-infra | 12:12 | |
| sdague | zuul promote --pipeline --changes 80413,5 ? | 12:12 |
| sdague | honestly I don't know | 12:12 |
| salv-orlando | jlibosva: can I ask you something about https://review.openstack.org/#/c/58695/ | 12:12 |
| sdague | mordred: honestly, I'd just promote now | 12:13 |
| sdague | it will force a reset | 12:13 |
| mordred | sdague: ok. things that bad then? | 12:13 |
| sdague | but the cost of it is nothing if this actually fixes things | 12:13 |
| mordred | ++ | 12:13 |
| sdague | you see that top of gat is 16hrs... right? :) | 12:13 |
| mordred | oh - yeah | 12:13 |
| mordred | sdague: done | 12:14 |
| *** dpyzhov has joined #openstack-infra | 12:14 | |
| sdague | I also have the plan B queued up | 12:15 |
| mordred | awesome | 12:15 |
| sdague | which is to turn off voting on neutron | 12:15 |
| sdague | if this doesn't seem to solve the issue | 12:15 |
| mordred | I will wait for jeblair to show up before I promote that | 12:16 |
| sdague | it's not a promote, it's an infra change | 12:16 |
| sdague | and I agree | 12:16 |
| mordred | since that soudns like one of those things more of us should chat about :) | 12:16 |
| sdague | absolutely | 12:16 |
| sdague | for reference, here is that change with the justification - https://review.openstack.org/#/c/81741/ | 12:17 |
| *** dpyzhov has joined #openstack-infra | 12:17 | |
| *** matsuhashi has quit IRC | 12:18 | |
| *** dcramer_ has quit IRC | 12:18 | |
| *** mikeburns has joined #openstack-infra | 12:18 | |
| *** pdmars has joined #openstack-infra | 12:19 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Setup the tripleo-test-cloud as a provider for Fedora instances https://review.openstack.org/81792 | 12:19 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Add the check-tripleo-overcloud-f20 jenkins job https://review.openstack.org/81793 | 12:19 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Add tripleo overcloud job to check queue https://review.openstack.org/81794 | 12:19 |
| dims | sdague, still think this is a good idea? (patched libvirt) https://review.openstack.org/#/c/79816/ if so, it's ready | 12:20 |
| salv-orlando | sdague: I have some rough numbers regarding whether 80413... | 12:21 |
| salv-orlando | mostly whether it will fix the issue or not | 12:21 |
| sdague | salv-orlando: and.... :) | 12:22 |
| sdague | dims: +A | 12:23 |
| salv-orlando | sdague: the lock wait timeout causing the gate issues shows up in 706 neutron checks out of a total of 7,052 | 12:23 |
| salv-orlando | which is roughly 10% | 12:23 |
| dims | sdague, thanks! | 12:23 |
| sdague | ok, so that represents about 2/3 of the fails? | 12:24 |
| salv-orlando | for patch 80413, 68 neutron jobs where executed while rechecking and the lock wait timeout on update ports was never seen | 12:24 |
| sdague | as I think I saw neutron success rate was about 85% right now | 12:24 |
| salv-orlando | lock wait timeout is not always a reason for failure | 12:24 |
| sdague | salv-orlando: great | 12:24 |
| sdague | salv-orlando: ok | 12:24 |
| *** matsuhashi has joined #openstack-infra | 12:24 | |
| salv-orlando | sdague: the timeout might occur but go unnoticed since it does not always timeout a nova call | 12:25 |
| *** saju_m has joined #openstack-infra | 12:26 | |
| *** rpodolyaka1 has joined #openstack-infra | 12:27 | |
| *** ildikov_ has joined #openstack-infra | 12:27 | |
| pleia2 | good morning | 12:29 |
| pleia2 | so much snow | 12:29 |
| SergeyLukjanov | pleia2, morning | 12:29 |
| *** dpyzhov has quit IRC | 12:31 | |
| sdague | pleia2: you should have been in the NE last month | 12:31 |
| *** rpodolyaka1 has quit IRC | 12:31 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix pip on py3k-precise nodes https://review.openstack.org/75213 | 12:32 |
| mordred | ttx: ^^ | 12:33 |
| *** unicell has joined #openstack-infra | 12:33 | |
| ttx | kewl | 12:34 |
| *** mikeburns has quit IRC | 12:34 | |
| sdague | hmmm... something else is up | 12:34 |
| pleia2 | sdague: hah, yeah, I hear it was a pretty brutal winter | 12:35 |
| sdague | there really aren't that many nodes running right now | 12:35 |
| *** mburned_out has joined #openstack-infra | 12:35 | |
| *** mburned_out is now known as mburned | 12:36 | |
| sdague | I only see about 20 devstack nodes allocated across everything in zuul | 12:36 |
| sdague | mordred: do you have enough view into nodepool to figure out if something's up? | 12:37 |
| *** mikeburns has joined #openstack-infra | 12:37 | |
| *** dpyzhov has joined #openstack-infra | 12:38 | |
| *** weshay has joined #openstack-infra | 12:39 | |
| *** CaptTofu has quit IRC | 12:39 | |
| *** roeyc has joined #openstack-infra | 12:41 | |
| *** mihgen has quit IRC | 12:41 | |
| mordred | sdague: looking | 12:42 |
| *** dpyzhov has quit IRC | 12:42 | |
| *** vkozhukalov_ has quit IRC | 12:42 | |
| *** nosnos has quit IRC | 12:46 | |
| mordred | sdague: there are many nodes in deleting and used state | 12:46 |
| *** matsuhashi has quit IRC | 12:47 | |
| mordred | sdague: 81 deleting 262 used 93 building | 12:48 |
| fungi | what's the concern? | 12:49 |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack-infra/config: Enable Gearman as default on Jenkins slaves https://review.openstack.org/73214 | 12:50 |
| *** CaptTofu has joined #openstack-infra | 12:50 | |
| fungi | i was looking at the zuul status page a little while ago and it seemed to be processing | 12:50 |
| fungi | there were some changes in the check queue which apparently got jobs restarted | 12:50 |
| mordred | fungi: expand the things in the gate | 12:50 |
| mordred | like, the top one | 12:50 |
| fungi | neutron change running jobs | 12:51 |
| mordred | oh - nm | 12:51 |
| *** aysyd has joined #openstack-infra | 12:51 | |
| fungi | looks like we're fresh out of a gate reset | 12:51 |
| mordred | they were showing queued in my browser - but they're showing properly now | 12:51 |
| fungi | check pipeline gets starved while we assign nodes to gate changes, especially after a reset | 12:52 |
| mordred | nod | 12:52 |
| *** saju_m has quit IRC | 12:53 | |
| fungi | that one at the top is from the promote you ran, i think | 12:53 |
| mordred | yah | 12:53 |
| mordred | that it is | 12:53 |
| fungi | which explains the difference in enqueued times | 12:53 |
| *** CaptTofu has quit IRC | 12:55 | |
| fungi | oh, good, that's the one i meant to check in on this morning. marun had me promote it last night, but then decided at the last minute it needed a better commit message and -2'd it before it could merge. glad to see it back in now | 12:55 |
| *** mburned has quit IRC | 12:57 | |
| *** mburned_out has joined #openstack-infra | 12:57 | |
| *** mburned_out is now known as mburned | 12:57 | |
| sdague | fungi: so check shouldn't be that starved right | 12:57 |
| *** bknudson has quit IRC | 12:57 | |
| sdague | because we shouldn't be able to consume all the d-g nodes in our 20 job window | 12:57 |
| fungi | well, plus any which were already running on jobs in other pipelines | 12:58 |
| *** vkozhukalov has joined #openstack-infra | 12:58 | |
| *** mburned is now known as mburned_out | 12:58 | |
| *** lttrl has quit IRC | 12:58 | |
| fungi | remember we're already running at near capacity, then the gate dumps 200+ nodes and says "gimme more" so any new ones which get spun up keep going to the gate until it's satiated, then they start going to waiting check jobs again | 12:59 |
| sdague | ok, I guess maybe we were rest calculating so much that they keep not getting assigned | 12:59 |
| *** jgallard has joined #openstack-infra | 13:00 | |
| *** IvanBerezovskiy1 has joined #openstack-infra | 13:00 | |
| fungi | mordred: i assume you picked up on what i changed in 75213 and why? testing showed that the system-installed tox wouldn't run if we removed the system-installed pkg-resources or setuptools, unfortunately, even if we pulled new ones in via pip | 13:00 |
| fungi | mordred: also, ez_setup.py seemed not to run at all on our nodes (neither py3k-precise nor bare-precise) complaining about zipimport being unable to parse the header on the setuptools 3.3 egg | 13:01 |
| fungi | mordred: but according to dstufft the latest versions of get-pip.py vendor in a new setuptools with pip, so apparently ez_setup.py isn't strictly necessary anyway | 13:02 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add missing neutron logs https://review.openstack.org/81782 | 13:02 |
| mordred | yes. this is true - and I've tested that emperical | 13:02 |
| sdague | it's helpful when I don't stack up unrelated patches | 13:03 |
| sdague | fungi: once that passes tests, could you take it through? | 13:03 |
| sdague | we're missing logs for 3 of the neutron agents in logstash | 13:03 |
| *** thuc has joined #openstack-infra | 13:03 | |
| sdague | and at least one of them is being problematic in the gate | 13:03 |
| *** dpyzhov has joined #openstack-infra | 13:04 | |
| fungi | sdague: sure thing, if it'll help debug more gate failures, top priority | 13:04 |
| *** mburned_out has quit IRC | 13:04 | |
| *** mwagner__zzzz is now known as mwagner_ | 13:04 | |
| *** mburned_out has joined #openstack-infra | 13:04 | |
| *** thuc_ has joined #openstack-infra | 13:04 | |
| sdague | fungi: definitely give us a better fingerprint | 13:04 |
| sdague | otherwise I'm matching off a failed tempest test | 13:04 |
| *** flaper87 is now known as flaper87|afk | 13:05 | |
| *** sdake has quit IRC | 13:05 | |
| fungi | sdague: yep, logs good, guesses bad. got it ;) | 13:05 |
| sdague | :) | 13:05 |
| *** sdake has joined #openstack-infra | 13:05 | |
| *** sdake has quit IRC | 13:05 | |
| *** sdake has joined #openstack-infra | 13:05 | |
| *** mburned_out has quit IRC | 13:05 | |
| *** saju_m has joined #openstack-infra | 13:06 | |
| *** thuc has quit IRC | 13:07 | |
| sdague | we should have HULK COMMIT day some time | 13:11 |
| fungi | hulk smash gate | 13:11 |
| *** flaper87|afk is now known as flaper87 | 13:12 | |
| * fungi can't participate--caps lock permanently broken ;) | 13:12 | |
| *** zul has quit IRC | 13:12 | |
| *** adalbas has joined #openstack-infra | 13:14 | |
| *** mburned_out has joined #openstack-infra | 13:15 | |
| *** zul has joined #openstack-infra | 13:15 | |
| *** mriedem has joined #openstack-infra | 13:15 | |
| *** mburned_out is now known as mburned | 13:15 | |
| *** changbl has quit IRC | 13:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: add missing neutron logs https://review.openstack.org/81782 | 13:16 |
| *** lttrl has joined #openstack-infra | 13:17 | |
| *** ihrachys|lunch is now known as ihrachys | 13:17 | |
| fungi | with the current gate reset rate, we seem to be getting about 1.5 changes per hour merged to master branches of things in the main integration test queue | 13:18 |
| jlibosva | salv-orlando: hi, sure. Sorry, I was afk | 13:18 |
| fungi | but that neutron fix is very, very close to merging now. hopefully that'll make a huge impact on our throughput | 13:18 |
| salv-orlando | jlibosva: I was just curious what's the status of that patch… do you think we can get grenade to pass soon? | 13:18 |
| *** dcramer_ has joined #openstack-infra | 13:19 | |
| fungi | neutron python26 jobs take upwards of an hour to complete these days? | 13:19 |
| ttx | jeblair: when you're up and have 5 min., would like to check our facts on using fun requirements branches to handle dep freeze | 13:19 |
| fungi | oic, centos6 nodes in hpcloud seem to take more than twice as long on the neutron jobs as those in rax for some reason. awesome | 13:21 |
| jlibosva | salv-orlando: I hope so, currently tempest tests are failing after upgrade, probably due to not configured values for nova notifiers.So far I got rid of all troubles during upgrade. I'm doing small steps and it takes a lot of time to get upgraded environment. I'm not happy about my speed either :( | 13:21 |
| *** bknudson has joined #openstack-infra | 13:21 | |
| *** vhoward has left #openstack-infra | 13:21 | |
| jlibosva | salv-orlando: anyway, I think it's review ready, so I can work on making it pass while getting comments on the code. | 13:22 |
| salv-orlando | thanks jlibosva - we will review the patch then. Give a shout to anybody from the core team if you need assistance | 13:22 |
| *** saju_m has quit IRC | 13:23 | |
| jlibosva | salv-orlando: are core reviewers from openstack-qa or infra? I'm not sure in which component grenade belongs | 13:23 |
| markmcclain | sdague, fungi: concerned that logs for the py26 unit test for 80413 haven't updated in nearly an hour | 13:23 |
| fungi | jlibosva: qa | 13:23 |
| markmcclain | https://jenkins03.openstack.org/job/gate-neutron-python26/618/console | 13:23 |
| jlibosva | I can find it in gerrit, right... | 13:23 |
| jlibosva | fungi: thanks | 13:23 |
| fungi | markmcclain: it looks like that job takes about an hour to tun on hpcloud nodes, roughly twice as long as on rackspace nodes | 13:23 |
| fungi | er, to run | 13:23 |
| markmcclain | crazy | 13:24 |
| markmcclain | difference is the way cores are counted? | 13:24 |
| markmcclain | s/is/in/ | 13:24 |
| fungi | i agree, and the timeout for that job is an hour, so not a good situation | 13:24 |
| fungi | looks like it's about to report, hopefully just under the wire | 13:24 |
| sdague | markmcclain: we're running 4 cores in HP | 13:25 |
| sdague | markmcclain: honestly, the neutron unit tests need a diet, we keep increasing the timeout on them | 13:25 |
| fungi | it's uploading job artifacts now | 13:25 |
| fungi | it passed, just barely | 13:25 |
| markmcclain | sdague: planned for Juno | 13:25 |
| fungi | and merged, looks like | 13:25 |
| sdague | cool | 13:25 |
| markmcclain | the double testing of json and xml causes lots of unnecessary bloat | 13:26 |
| sdague | trove uses wsgi_intercept as an interesting test tool | 13:26 |
| sdague | which would help not actually starting the wsgi stack fully | 13:26 |
| sdague | which neutron does a lot in the unit tests IIRC | 13:26 |
| *** rpodolyaka1 has joined #openstack-infra | 13:27 | |
| markmcclain | sdague: we're moving to pecan in Juno | 13:27 |
| markmcclain | so we'll get away from the full http stack setup we've got now | 13:28 |
| markmcclain | the full http stack test leaks lots of objects when the APIRouter is destroyed on tearDown which causes the high mem usage | 13:28 |
| fungi | should we bump the timeout further on gate-neutron-python26 in that case, to reduce false resets because of long runs in hpcloud, or look for another quick alternative? | 13:29 |
| *** thomasem has joined #openstack-infra | 13:30 | |
| markmcclain | fungi: we don't plan on merging any new plugins which contributes the most bloat | 13:31 |
| markmcclain | I can hold the plugins out until the new API layer | 13:31 |
| *** dkranz has joined #openstack-infra | 13:31 | |
| markmcclain | otherwise all py26 tests will get a larger timeout right? | 13:31 |
| *** rpodolyaka1 has quit IRC | 13:32 | |
| fungi | well, my point being we've got that job regularly coming within a minute of the timeout, so all it takes is a sneeze to push it over the edge | 13:32 |
| markmcclain | btw.. it's looking better since that patch landed… 14 jobs behind it have passing neutron tempest runs | 13:32 |
| markmcclain | yeah.. I'm fine with a large timeout wasn't sure if that was going to cause problems elsewhere | 13:32 |
| fungi | mmm... that keystone change in slot #2 just lost a full tempest run to | 13:33 |
| fungi | "BuildErrorException: Server 4b4a44c8-157e-4c59-ac81-a9e60d2147d3 failed to build and is in ERROR status" | 13:33 |
| fungi | libvirt? | 13:33 |
| markmcclain | yeah :( | 13:34 |
| fungi | is this one of the ones we're hoping the patched libvirt addresses? | 13:34 |
| sdague | fungi: link? | 13:34 |
| fungi | ahh, further up was "Details: Time Limit Exceeded! (196s)while waiting for active, but we got queued. | 13:34 |
| fungi | sdague: http://logs.openstack.org/85/81385/4/gate/gate-tempest-dsvm-full/8fd64e8/console.html | 13:34 |
| sdague | fungi: yeh, that's too generic of an issue to tag on | 13:34 |
| *** dstanek has joined #openstack-infra | 13:34 | |
| fungi | tanked in test_create_backup | 13:35 |
| fungi | ServerActionsV3Test.test_create_backup, so i guess that's a nova v3-specific api test? | 13:35 |
| sdague | fungi: no, this is a different issue, the fix for which is in the queue somewhere | 13:36 |
| sdague | http://logs.openstack.org/85/81385/4/gate/gate-tempest-dsvm-full/8fd64e8/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-03-20_13_06_04_124 | 13:36 |
| sdague | oh, no, never mind | 13:36 |
| sdague | yes it's the libvirt issue | 13:36 |
| sdague | http://logs.openstack.org/85/81385/4/gate/gate-tempest-dsvm-full/8fd64e8/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-03-20_13_10_53_075 | 13:37 |
| sdague | fungi: so we have that change in the gate | 13:37 |
| sdague | we could promote that | 13:37 |
| *** mburned has quit IRC | 13:37 | |
| sdague | fungi: https://review.openstack.org/#/c/79816/ | 13:37 |
| fungi | absolutely | 13:37 |
| fungi | now's a good time since we basically just reset anyway | 13:37 |
| sdague | yep | 13:38 |
| *** salv-orlando has quit IRC | 13:38 | |
| fungi | it's in the top spot now | 13:38 |
| *** saju_m has joined #openstack-infra | 13:39 | |
| sdague | cool | 13:40 |
| fungi | eta 1 hour, sayeth the jenkins masters | 13:40 |
| sdague | yep | 13:42 |
| *** pbelanyi has joined #openstack-infra | 13:42 | |
| sdague | those estimates are all over the place though with the 4 vs. 8 core nodes | 13:42 |
| sdague | fungi: if you want to review some css - https://review.openstack.org/#/c/80843/ :) | 13:42 |
| sdague | that will make me happy, because it means no more horizontal scrolling (except when something dumps a keystone token to the screen) | 13:43 |
| *** wchrisj has joined #openstack-infra | 13:43 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/storyboard: Remove Branch and Milestone legacy tables https://review.openstack.org/81562 | 13:44 |
| fungi | i take it having the spans partially overlap other block elements (as in your modification to test_html_file_filters()) works fine in most browsers? | 13:45 |
| *** eharney has joined #openstack-infra | 13:45 | |
| *** dpyzhov has quit IRC | 13:45 | |
| *** azneita has joined #openstack-infra | 13:46 | |
| *** andreaf has quit IRC | 13:46 | |
| *** zehicle_at_dell has quit IRC | 13:48 | |
| pleia2 | hm, I'm clearly missing something, how does /opt/nodepool-scripts get populated? | 13:48 |
| *** yamahata has quit IRC | 13:48 | |
| *** saju_m has quit IRC | 13:48 | |
| pleia2 | I know where the contents come *from* just not how they get there | 13:49 |
| *** yamahata has joined #openstack-infra | 13:49 | |
| *** jswarren has quit IRC | 13:50 | |
| *** hashar has joined #openstack-infra | 13:51 | |
| openstackgerrit | A change was merged to openstack-infra/os-loganalyze: wrap log lines https://review.openstack.org/80843 | 13:52 |
| sdague | fungi: yes, it seems to | 13:52 |
| fungi | pleia2: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py#n568 | 13:53 |
| *** medieval1 has joined #openstack-infra | 13:53 | |
| pleia2 | fungi: aha, thank you | 13:53 |
| fungi | pleia2: the nodepool server injects them via scp | 13:54 |
| * pleia2 nods | 13:54 | |
| fungi | from its local copy | 13:54 |
| pleia2 | I'm doing testing by manually spinning up hpcloud VMs, so that step is absent | 13:54 |
| *** ildikov_ has quit IRC | 13:55 | |
| jamespage | apevec, I have a network again - running tests now | 13:57 |
| jamespage | apevec, the quantum tests passed OK _ just running the other two topologies we test with | 13:57 |
| apevec | jamespage, cool, thanks! | 13:58 |
| *** jergerber has joined #openstack-infra | 13:58 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/storyboard: Remove Branch and Milestone legacy tables https://review.openstack.org/81562 | 13:58 |
| sdague | the call stack when we hit cleanup in a test is interesting, and less helpful in figuring out what's going on than I thought | 14:00 |
| *** e0ne_ has quit IRC | 14:01 | |
| *** mrodden has joined #openstack-infra | 14:02 | |
| *** dpyzhov has joined #openstack-infra | 14:03 | |
| *** andreaf has joined #openstack-infra | 14:05 | |
| *** dcramer_ has quit IRC | 14:05 | |
| *** dcramer_ has joined #openstack-infra | 14:06 | |
| *** ildikov_ has joined #openstack-infra | 14:06 | |
| *** mburned has joined #openstack-infra | 14:08 | |
| *** e0ne has joined #openstack-infra | 14:08 | |
| *** dcramer_ has quit IRC | 14:11 | |
| *** mfer has joined #openstack-infra | 14:11 | |
| *** flaper87 is now known as flaper87|afk | 14:13 | |
| *** dstanek_afk has joined #openstack-infra | 14:14 | |
| *** dstanek has quit IRC | 14:15 | |
| *** dstanek_afk is now known as dstanek | 14:15 | |
| *** jcoufal has joined #openstack-infra | 14:16 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/os-loganalyze: use padding instead of margins for highlighting https://review.openstack.org/81817 | 14:16 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/os-loganalyze: be safe about the selector https://review.openstack.org/81818 | 14:16 |
| sdague | fungi: a couple small fix ups, found some odd interactions between highlighted regions vs the console logs | 14:17 |
| sdague | which mostly are visually confusing | 14:17 |
| *** sandywalsh has quit IRC | 14:18 | |
| sdague | I've also been trying to figure out a crazy hack to provide equiv highlighting on cgit | 14:18 |
| *** nati_ueno has joined #openstack-infra | 14:19 | |
| fungi | sdague: started to approve 81818 and then on re-reviewing it spotted a typo, i think | 14:21 |
| sdague | fungi: good catch | 14:21 |
| fungi | feels like old (c) times, looking for missing semicolons | 14:22 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/os-loganalyze: be safe about the selector https://review.openstack.org/81818 | 14:22 |
| sdague | plus js is weird | 14:22 |
| sdague | in that the semi colon in mostly optional | 14:22 |
| sdague | except when it isn't | 14:22 |
| sdague | that's also what I get for not seperating this out into a template which emacs would pick up as js | 14:23 |
| fungi | heh | 14:23 |
| sdague | my html multi mode stuff works super slick | 14:24 |
| sdague | but it has to think the base is html | 14:24 |
| *** pcrews has joined #openstack-infra | 14:24 | |
| jeblair | ttx: good time of day to you! | 14:24 |
| ttx | jeblair: ohai! | 14:24 |
| jeblair | fungi, sdague: good morning! | 14:25 |
| openstackgerrit | A change was merged to openstack-infra/os-loganalyze: use padding instead of margins for highlighting https://review.openstack.org/81817 | 14:25 |
| jeblair | mordred: congratulations on your (zuul) promotion. :) | 14:25 |
| sdague | jeblair: good morning | 14:25 |
| fungi | howdy and welcome to the land of the light | 14:25 |
| sdague | markmcclain: the lbaas neutron service is failing a lot in the gate as well (the current reset in the gate is because of it) | 14:26 |
| sdague | my instinct is to turn off that service until it can be proven more reliable | 14:27 |
| jeblair | ttx: what are your thoughts around requirements and branches? | 14:27 |
| *** rpodolyaka1 has joined #openstack-infra | 14:28 | |
| *** azneita has quit IRC | 14:28 | |
| ttx | jeblair: so.. IIUC the idea would be to cut a milestone-proposed branch next week on Tuesday, before we start creating MP branches for RC1s in all projects ? | 14:28 |
| *** e0ne_ has joined #openstack-infra | 14:28 | |
| ttx | jeblair: one of my questions is... what happens for projects that would still not have a MP branch by then | 14:29 |
| ttx | i.e. nova continuing on master for a few more days after we set a MP branch for requirements | 14:29 |
| ttx | would master "use" the MP branch ? | 14:29 |
| *** dprince has joined #openstack-infra | 14:30 | |
| *** thuc_ has quit IRC | 14:30 | |
| *** sandywalsh has joined #openstack-infra | 14:31 | |
| *** malini has joined #openstack-infra | 14:31 | |
| *** thuc has joined #openstack-infra | 14:31 | |
| malini | jeblair: hello! Can you take a look at the new patchset in https://review.openstack.org/#/c/81249/ ? Thanks! | 14:31 |
| jeblair | ttx: ah, no it would not | 14:31 |
| *** e0ne has quit IRC | 14:32 | |
| jeblair | ttx: i'm trying to think if that's something we should encode into devstack-gate... | 14:32 |
| ttx | jeblair: so that means, Dep freeze would actually not be "enforced" until that MP branch is cut in the project itself | 14:32 |
| *** alff has quit IRC | 14:32 | |
| *** rpodolyaka1 has quit IRC | 14:33 | |
| ttx | jeblair: we can also have a "soft freeze" on requirements/master between the first RC1 and the last RC1 | 14:33 |
| fungi | is there any reason we don't just flat out freeze the requirements repository in the interim (except for temporary < or != things we have to do to it to un-wedge gating)? | 14:33 |
| ttx | i.e. cut the branch but make sure master = MP | 14:33 |
| ttx | that should only last a week or so | 14:34 |
| fungi | that seems safest to me | 14:34 |
| jeblair | should d-g fall back on m-p if it exists and then master? i don't think that would work because the reality is that you usually want master to be free to continue development. | 14:34 |
| openstackgerrit | A change was merged to openstack-infra/os-loganalyze: be safe about the selector https://review.openstack.org/81818 | 14:34 |
| ttx | jeblair: yeah, that seems a bit too much | 14:35 |
| jeblair | ttx: so yeah, freeze while you're branching the projects seems safest | 14:35 |
| jeblair | ttx: then when all the branches are created, unfreeze master | 14:35 |
| ttx | jeblair: takes a bit of education, but requirements +2ers should be educatable | 14:35 |
| *** thuc has quit IRC | 14:35 | |
| ttx | So we'd softfreeze requirements next Tuesday, and once all projects have done a RC1 (and therefore have a milestone-proposed branch) we can cut a requirements MP branch | 14:36 |
| ttx | and unfreeze requirements master | 14:37 |
| jeblair | sounds right | 14:37 |
| ttx | jeblair: projects MP branches would still consume requirements master, right | 14:37 |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack-infra/devstack-gate: Update Readme to setup jenkins accout https://review.openstack.org/81823 | 14:37 |
| jeblair | ttx: yes, as long as requirements has no MP branch, MP changes on projects will fall back to requirements master | 14:37 |
| ttx | jeblair: OK, I /think/ that can work | 14:38 |
| ttx | it may actually be how we did it last time | 14:38 |
| ttx | informally | 14:38 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Temporary workaround for Time out errors from libvirt https://review.openstack.org/79816 | 14:39 |
| jeblair | sdague, dims ^ | 14:40 |
| mordred | jeblair: thanks! I was pretty excited | 14:40 |
| openstackgerrit | Solly Ross proposed a change to openstack/requirements: Limit psutil to <2.0.0 https://review.openstack.org/81373 | 14:40 |
| dansmith | is that workaround to use the new libvirt package? | 14:42 |
| jeblair | dansmith: yes, the one that should show up in ubuntu soon | 14:43 |
| dansmith | jeblair: cool | 14:43 |
| *** Ryan_Lane has joined #openstack-infra | 14:44 | |
| dims | jeblair, thanks! | 14:44 |
| dims | cross my fingers and toes now :) | 14:44 |
| *** dkliban has joined #openstack-infra | 14:44 | |
| sdague | jeblair: yep, great | 14:44 |
| *** salv-orlando has joined #openstack-infra | 14:45 | |
| markmcclain | sdague: just got back from lunch | 14:46 |
| markmcclain | looking at it now, the error seems new | 14:46 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add match for lbaas failure https://review.openstack.org/81825 | 14:48 |
| sdague | markmcclain: it's been around for at least a week | 14:48 |
| sdague | more frequent recently | 14:48 |
| markmcclain | sdague: was looking at the history.. did we reactivate the check I had disabled in Feb? | 14:49 |
| sdague | yes, it's been reactivated | 14:52 |
| *** nati_ueno has quit IRC | 14:52 | |
| *** mrmartin has joined #openstack-infra | 14:53 | |
| *** CaptTofu has joined #openstack-infra | 14:54 | |
| fungi | jeblair: do you happen to know off the top of your head whether nodepool updates its scheduling on the fly when there are config changes or requires a restart for them to take effect? and if the latter, we would prefer a patch which made it do the former i assume? | 14:55 |
| *** mwagner is now known as mwagner_notHere | 14:55 | |
| *** thuc has joined #openstack-infra | 14:56 | |
| jeblair | fungi: scheduling like the periodic tasks? should be on the fly | 14:56 |
| fungi | jeblair: okay, in that case the image rebuild schedule was updated in the nodepool.conf several days ago but nodepoold is still using the old start times from what i can tell on image ages | 14:57 |
| fungi | so probably a bug | 14:57 |
| *** atiwari has joined #openstack-infra | 14:57 | |
| jeblair | sounds like it :/ | 14:58 |
| fungi | i'll see if i can spot it in the source | 14:58 |
| *** maxbit has joined #openstack-infra | 14:59 | |
| fungi | well, first i'll double-check the logs from when that file changed and see if maybe it's having a hard time parsing it due to some typo | 14:59 |
| *** miqui has joined #openstack-infra | 14:59 | |
| *** thuc has quit IRC | 15:02 | |
| *** thuc has joined #openstack-infra | 15:02 | |
| *** thedodd has joined #openstack-infra | 15:02 | |
| sdague | ok, so this lbaas thing is raising a really important question | 15:03 |
| sdague | we've got a ton of services we turn on in d-g | 15:03 |
| *** nati_ueno has joined #openstack-infra | 15:03 | |
| sdague | which we don't test in any real way | 15:03 |
| sdague | and I think that once we have a way to make tempest changes depend on d-g changes, we don't let people add services that don't have some sane tempest coverage | 15:03 |
| markmcclain | sdague: yeah so there's the good check to make sure we don't have regression from activating new services | 15:03 |
| markmcclain | but I agree the coverage is not ideal | 15:04 |
| markmcclain | it is kind of chicken and egg | 15:04 |
| sdague | if we can cross depend changes, it's not chicken and egg | 15:04 |
| sdague | we can also build experimental jobs that do things differently | 15:04 |
| *** sabari has joined #openstack-infra | 15:04 | |
| ttx | jeblair: now that https://review.openstack.org/#/c/75213/, how long should it take for py33 jobs to start working ? | 15:05 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add match for lbaas failure https://review.openstack.org/81825 | 15:05 |
| markmcclain | sdague: not sure how cross dep would help here | 15:05 |
| jeblair | ttx: i think that's probably what prompted fungi to ask about image builds... | 15:05 |
| *** mwagner_ has quit IRC | 15:06 | |
| sdague | because you could build tempest tests which test the service on a d-g change that enables the service | 15:06 |
| ttx | jeblair: merged at 13:32 but a test started at 13:36 still promtly failed | 15:06 |
| markmcclain | ok so mutually dependent cross project tests | 15:06 |
| sdague | so then you could build a patch stack that was d-g - enable + tempest changes .... | 15:06 |
| fungi | jeblair: ttx: correct. images rebuilt around 0600 utc before that was merged | 15:06 |
| ttx | (reference: https://review.openstack.org/#/c/72093/) | 15:06 |
| ttx | fungi: ok, so need to wait tomorrow before recheck | 15:06 |
| sdague | which would demonstrate that the service would work and would be tested | 15:06 |
| jeblair | fungi: shall i kick off a py3k rebuild? | 15:06 |
| markmcclain | sdague: of the 30ish tests in the proposed tempests we do have tests for LBaaS and VPNaaS | 15:06 |
| fungi | jeblair: well, 6 py3k rebuilds i guess | 15:07 |
| fungi | jeblair: but sure, that would help | 15:07 |
| sdague | markmcclain: the fact that the scenario test fails this often is not a good sign | 15:07 |
| jeblair | ttx: that change has to make it onto a new image, so it will be a few hours probably | 15:07 |
| sdague | markmcclain: those are all API tests, right? | 15:07 |
| markmcclain | sdague: yes | 15:07 |
| *** wenlock has joined #openstack-infra | 15:07 | |
| sdague | so they aren't actually seeing if the thing works | 15:07 |
| ttx | jeblair: tomorrow is fine. Want to tag a new oslo.rootwrap before eow | 15:07 |
| markmcclain | I'm guessing we haven't made progress on a good area to have test for bake for awhile before they become gating | 15:08 |
| *** sabari has quit IRC | 15:08 | |
| * ttx takes a break to mow the lawn before it rains tomorrow | 15:09 | |
| zul | so...im just wondering whats the timeline for you guys moving to trusty after its out? | 15:10 |
| fungi | zul: no idea. right now none of our providers have beta images in their catalog for it so we can't get much of a head start | 15:11 |
| jeblair | zul: depends on either how fast our providers get trusty images, or if they fix their glance support | 15:11 |
| zul | fungi/jeblair: have you guys been tracking whats going on with trusty? | 15:11 |
| fungi | zul: also, presumably we'll continue testing icehouse on precise but start testing master/juno on trusty | 15:12 |
| *** talluri_ has quit IRC | 15:12 | |
| jeblair | fungi: ++ | 15:12 |
| fungi | zul: tracking in what way? | 15:12 |
| *** dstanek has quit IRC | 15:12 | |
| zul | fungi: like which software versions its going to have (python/libvirt/etc), etc | 15:12 |
| jamespage | apevec, OK - all three topologies smoked ok for grizzly stable | 15:13 |
| fungi | zul: nope--it's mostly irrelevant to us until we can actually try running our tests on it i think | 15:13 |
| zul | fungi: python3.4 looks like is going to be the default for python3 at least | 15:14 |
| zul | fungi: https://bugs.launchpad.net/ubuntu/+source/python3.3/+bug/1295153 fyi | 15:14 |
| uvirtbot | Launchpad bug 1295153 in python3.3 "[FFe] Remove Python 3.3 from Trusty" [Undecided,New] | 15:14 |
| fungi | zul: neat--i assume we'll start testing that on bare-trusty images then, and we can retire the py3k-precise node type when we eol icehouse | 15:15 |
| zul | fungi: yeah i think im going to do a ods session that we should be moving to python3.4 for python3 testing | 15:16 |
| *** david-lyle has joined #openstack-infra | 15:16 | |
| fungi | i would imagine some time fairly early in the juno dev cycle we'd switch from running gate-.*-python33 to gate-.*-python34 jobs and the latter would just go on our bare-trusty nodes | 15:16 |
| zul | fungi: ack | 15:16 |
| fungi | and mask those jobs to master branch and limit python33 jobs to stable/havana and stable/icehouse | 15:17 |
| *** dstanek has joined #openstack-infra | 15:17 | |
| fungi | i'm more worried about what all will be necessary to get devstack/tempest integration jobs running on trusty | 15:18 |
| *** changbl has joined #openstack-infra | 15:18 | |
| fungi | i suspect most of the time will be spent first on puppet/nodepool tweaks for trusty, and then on devstack/tempest/grenade | 15:18 |
| fungi | grenade brings up an interesting question for distro release changes too | 15:19 |
| sdague | yeh, this is not going to be quick | 15:19 |
| zul | fungi: i ran them last night in a vm and only had like 8 failures but i think thats due to my limitations | 15:19 |
| fungi | i don't think we were gating on grenade the last time we switched distro releases | 15:19 |
| sdague | fungi: no, we weren't | 15:19 |
| fungi | so we may need to also get havana devstack working on trusty | 15:19 |
| zul | fungi: btw we have mysql 5.5 in main and mysql 5.6 in universe as well | 15:20 |
| jeblair | fungi: i don't think grenade was a twinkle in dtroyer's eye yet at that time. :) | 15:20 |
| sdague | fungi: well, we could run a separate image for grenade in stable | 15:20 |
| fungi | or run grenade for juno on precise and switch grenade to trusty for the k release | 15:20 |
| sdague | jeblair: yeh, grenade showed up as nebula finding issues in the last 2 weeks of the folsom release | 15:20 |
| fungi | jeblair: right | 15:20 |
| fungi | jeblair: the quantal switch aborted before we got that far anyway | 15:21 |
| jeblair | so today it's rax-iad being persnickety | 15:21 |
| fungi | anyway, grenade means that either juno will need to work on precise or havana will need to work on trusty | 15:22 |
| fungi | unless grenade wants to learn about interweaving a distro upgrade, and that makes our upgrade story a lot less vendor-agnostic if so | 15:22 |
| jeblair | fungi: well, i don't think grenade is vendor agnostic | 15:23 |
| jeblair | i mean, inasmuch as devstack does dpkg/rpm actions | 15:23 |
| *** mgagne has joined #openstack-infra | 15:23 | |
| *** gyee has joined #openstack-infra | 15:25 | |
| fungi | jeblair: true, but if grenade starts encoding tweaks to handle distro upgrades to work around us being unable to straddle an ubuntu release with two openstack integrated releases, it becomes even less so | 15:26 |
| *** krtaylor has quit IRC | 15:27 | |
| sdague | fungi: I so don't want to do release jumping in grenade | 15:28 |
| *** rpodolyaka1 has joined #openstack-infra | 15:28 | |
| sdague | my assumption is we'd run havana -> icehouse grenade on precise | 15:29 |
| sdague | and icehouse -> on trusty | 15:29 |
| sdague | once we can get that to work | 15:29 |
| *** rpodolyaka1 has quit IRC | 15:30 | |
| fungi | sdague: makes sense | 15:32 |
| fungi | sdague: that would still allow us to remove precise from our corral once havana is eol | 15:33 |
| fungi | er, once icehouse is eol | 15:33 |
| fungi | so we'd try to get icehouse working on both precise and trusty, and use it as our hinge | 15:35 |
| *** pbelanyi has quit IRC | 15:35 | |
| *** azneita has joined #openstack-infra | 15:36 | |
| fungi | i suppose the bigger question is whether we run two sets of integration tests on icehouse changes, once to keep it working on precise and one to keep it working on trusty? | 15:36 |
| openstackgerrit | A change was merged to openstack-infra/config: Add PuppetBoard to openstack infra https://review.openstack.org/77928 | 15:36 |
| *** IvanBerezovskiy1 has quit IRC | 15:37 | |
| fungi | that could get ugly as well | 15:38 |
| fungi | btw i'll keep an eye on the puppet master to make sure that ^ isn't breaking install_modules.sh | 15:39 |
| *** gokrokve has joined #openstack-infra | 15:42 | |
| sdague | fungi: actually, once havana is eol | 15:47 |
| *** jnoller has joined #openstack-infra | 15:47 | |
| sdague | because then we'd stop the pull up upgrade jobs | 15:47 |
| fungi | sdague: but we'd still be testing icehouse on precise because that's what we released it when testing | 15:47 |
| sdague | fungi: ok, I assumed we were going to get rid of those jobs | 15:48 |
| sdague | because if not, we need to go ask hp and rax to double quota ASAP | 15:48 |
| sdague | as that would be a ton of jobs on all the icehouse stable | 15:48 |
| fungi | if we stop testing icehouse on precise while we're still supporting icehouse, we run the risk of backporting a change which causes it to stop working on precise | 15:48 |
| fungi | unless we want to make the call that everyone running icehouse on precise should immediately upgrade to trusty as soon as we've figured out how to make icehouse (not just juno) work on trusty | 15:50 |
| fungi | and does that mean non-stable-safe changes to the already released icehouse to make that possible? | 15:50 |
| *** jcoufal has quit IRC | 15:50 | |
| jeblair | ord and dfw py3k images are ready | 15:52 |
| jeblair | iad, az1, and az2 failed, az3 is running | 15:52 |
| fungi | is this instability pretty much entirely just the facter/metadata issue still? | 15:53 |
| sdague | fungi: yeh, I think we probably want to just make that call | 15:53 |
| jeblair | in az2 yes. i think there's an ssh daemon start timing issue in rax we can probably work around in nodepool | 15:54 |
| *** signed8bit has joined #openstack-infra | 15:54 | |
| jeblair | (also az2 for the metadata) | 15:54 |
| *** rossella_s has quit IRC | 15:54 | |
| *** dcramer_ has joined #openstack-infra | 15:55 | |
| *** rossella_s has joined #openstack-infra | 15:56 | |
| jeblair | fungi, sdague: our current process is to test openstack releases on the operating system they were released on | 15:56 |
| jeblair | so i would expect that we would continue to test icehouse on precise for as long as we support icehouse | 15:56 |
| sdague | ok, interesting | 15:56 |
| jeblair | that's the assumption our platform testing policy is based on, and what we've been telling everyone | 15:57 |
| sdague | well we'll definitely need a ton more nodes for that | 15:57 |
| sdague | because that last transition tempest was 76 tests | 15:57 |
| fungi | that was my base assumption as well. which brings us back to grenade being able to upgrade icehouse->juno on precise and juno->k on trusty | 15:57 |
| sdague | and ran in one configuration | 15:57 |
| jeblair | sdague: i don't understand "because that last transition tempest was 76 tests" | 15:58 |
| fungi | i parsed that as "tempest only had 76 tests when we switched from oneiric to precise" | 15:59 |
| sdague | yes | 15:59 |
| sdague | basically the set of things we had to make and keep working across 2 ltses was very small | 15:59 |
| sdague | and ran very quickly | 16:00 |
| jeblair | *nod* | 16:00 |
| *** medieval1 has quit IRC | 16:00 | |
| jeblair | anyway, i mentioned our current assumptions/policy not to indicate that i think it's set in stone or anything... | 16:00 |
| *** krotscheck has joined #openstack-infra | 16:00 | |
| sdague | because, I'll be completely honest, I don't see us pulling off working grenade icehouse -> juno on both precise and trusty quickly | 16:00 |
| jeblair | just to say i think that's the current understanding, and as you've indicated before sdague, we'll probably need to get all the right people in a room at ods to figure it out | 16:00 |
| sdague | yep | 16:01 |
| *** medieval1 has joined #openstack-infra | 16:01 | |
| sdague | it was half way through i-2 before we were upgrading from havana stable instead of grizzly stable | 16:01 |
| sdague | as a data point | 16:01 |
| *** dcramer_ has quit IRC | 16:02 | |
| jeblair | fungi: what jobs are we looking for to see if the py3k fix worked? | 16:03 |
| fungi | sdague: would it help if we added a stub master->master upgrade job which we only ran on changes to grenade, so you could easily bifurcate stable->master as soon as it emerges? | 16:04 |
| *** njd001 has joined #openstack-infra | 16:04 | |
| njd001 | I have a stupid question, why is review.openstack.org's IP address running a port scan against my DMZ? | 16:05 |
| njd001 | my ASA just SHUN'ed your access | 16:05 |
| sdague | fungi: it's unclear if the infrastructure to make that work would save that much effort | 16:06 |
| jeblair | njd001: do you have any logs that could help us look into that? | 16:07 |
| fungi | njd001: you (or someone in your network) likely opened up a bunch of connections to review.openstack.org and the source ports were allocated sequentially, making it look like a linear port scan | 16:07 |
| njd001 | This DMZ is doing Neutron-CI by the way... | 16:07 |
| jeblair | njd001: oh, that combined with what fungi said is suggestive... | 16:08 |
| njd001 | for Cisco's implimentation... oh okay makes sense so thats the only IP? | 16:08 |
| fungi | njd001: what do you show as the "source port" for the scan coming from review.openstack.org? is it something like 22 or 80 or 443? | 16:08 |
| *** markvoelker has joined #openstack-infra | 16:08 | |
| fungi | or 29318? | 16:08 |
| jeblair | or 29418 | 16:08 |
| fungi | that | 16:08 |
| fungi | i never can remember that port number | 16:08 |
| fungi | you'd think i'd have it memorized after nearly 2 years | 16:09 |
| njd001 | yes most likely 443,80 | 16:09 |
| njd001 | i just saw the scan hit, and the ASA notified it as a port-scan attack I'll mark this IP as safe. | 16:09 |
| *** nicedice has joined #openstack-infra | 16:10 | |
| fungi | njd001: yep, sounds like your firewall or nids is misinterpreting outbound sockets on sequential source ports as an inbound port scan | 16:10 |
| *** pmathews has joined #openstack-infra | 16:10 | |
| fungi | that's not at all uncommon, especially for stateless scanners looking for tcp/ack scans | 16:10 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Fix Fedora Puppet failure to use pip https://review.openstack.org/81841 | 16:10 |
| pleia2 | ]o/ | 16:10 |
| * anteaya notes there are 18 in post | 16:10 | |
| fungi | jeblair: i rechecked 72093 which should tell us | 16:10 |
| fungi | jeblair: that was oslo.rootwrap, but we were also hitting it on git-review changes | 16:11 |
| njd001 | i think it was the return port that it shun'ed on because it tried 3340-3389 and i have 3389 blackholed. Cool I'll mark it as safe, is that the ONLY IP though for review.openstack.org? | 16:11 |
| fungi | njd001: it has two global ip addresses currently... 198.101.231.251 and 2001:4800:780d:509:3bc3:d7f6:ff04:39f0 | 16:12 |
| njd001 | no review-ci.openstack.org or anything? I just want to make sure the ASA does not block CI and Review processes. | 16:12 |
| jeblair | btw, storyboard meeting has started | 16:12 |
| fungi | njd001: but those are subject to change if we have to rebuild teh server for any reason | 16:12 |
| njd001 | fungi got it thanks! Yes its why I am using a DNS name | 16:12 |
| fungi | njd001: nope, just those two ip addresses for the gerrit server currently | 16:12 |
| njd001 | fungi :o) perfect! | 16:13 |
| fungi | njd001: one for ipv4 and one for ipv6 (dual stack config) | 16:13 |
| jeblair | njd001: they don't change often, but they probably will change withen the next few months, possibly in conjunction with the gerrit upgrade | 16:14 |
| *** blamar-away is now known as blamar | 16:14 | |
| njd001 | we do have a nuetron-ci running there talking to Gerrit etc. | 16:16 |
| njd001 | noted, thank you jeblair! | 16:17 |
| *** mrodden has quit IRC | 16:17 | |
| *** jnoller has quit IRC | 16:20 | |
| *** hashar has quit IRC | 16:22 | |
| *** thuc has quit IRC | 16:23 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add post test hook to copy marconi logs https://review.openstack.org/81249 | 16:23 |
| *** thuc has joined #openstack-infra | 16:23 | |
| *** CaptTofu has quit IRC | 16:23 | |
| *** CaptTofu has joined #openstack-infra | 16:24 | |
| *** nithyag has quit IRC | 16:24 | |
| *** gokrokve_ has joined #openstack-infra | 16:25 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: wip: DO NOT MERGE gather dstat data on ceilometer free env https://review.openstack.org/81843 | 16:27 |
| *** gokrokve has quit IRC | 16:27 | |
| *** thuc has quit IRC | 16:28 | |
| *** thuc has joined #openstack-infra | 16:28 | |
| *** SumitNaiksatam has quit IRC | 16:28 | |
| *** CaptTofu has quit IRC | 16:28 | |
| *** hogepodge has quit IRC | 16:28 | |
| *** rpodolyaka1 has joined #openstack-infra | 16:29 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack-infra/devstack-gate: Update Readme to setup jenkins accout https://review.openstack.org/81823 | 16:31 |
| *** mrodden has joined #openstack-infra | 16:31 | |
| *** CaptTofu has joined #openstack-infra | 16:32 | |
| *** rhsu has joined #openstack-infra | 16:33 | |
| *** rpodolyaka1 has quit IRC | 16:34 | |
| clarkb | morning | 16:34 |
| *** mrmartin has quit IRC | 16:35 | |
| *** hogepodge has joined #openstack-infra | 16:35 | |
| anteaya | morning clarkb | 16:35 |
| *** mgagne has quit IRC | 16:36 | |
| *** dims has quit IRC | 16:37 | |
| *** mgagne has joined #openstack-infra | 16:37 | |
| *** caleb_ has joined #openstack-infra | 16:38 | |
| *** UtahDave has joined #openstack-infra | 16:39 | |
| *** pdmars has quit IRC | 16:39 | |
| *** thuc has quit IRC | 16:43 | |
| *** thuc_ has joined #openstack-infra | 16:43 | |
| *** SumitNaiksatam has joined #openstack-infra | 16:43 | |
| *** bhuvan has joined #openstack-infra | 16:43 | |
| *** e0ne_ has quit IRC | 16:44 | |
| *** e0ne has joined #openstack-infra | 16:46 | |
| *** moted has quit IRC | 16:47 | |
| *** mgagne has quit IRC | 16:49 | |
| *** e0ne_ has joined #openstack-infra | 16:49 | |
| *** dims has joined #openstack-infra | 16:49 | |
| *** UtahDave has left #openstack-infra | 16:51 | |
| *** dripton has quit IRC | 16:51 | |
| *** e0ne has quit IRC | 16:52 | |
| fungi | clarkb: nibalizer: install_modules.sh is complaining that 'nibalizer-puppetboard' (v2.3.0) requires 'puppetlabs-vcsrepo' (>= 0.1.2) http://paste.openstack.org/show/73922/ ideas on a path forward there? | 16:54 |
| clarkb | fungi: the no install deps thing was supposed to work around that | 16:55 |
| fungi | yep. i read the commit message too ;) | 16:55 |
| clarkb | but I guess that isn't strong enough of a no really puppet | 16:55 |
| clarkb | fungi: is it failing to install it? | 16:55 |
| nibalizer | :( | 16:55 |
| clarkb | or is it just warning us about the mismatch? | 16:55 |
| fungi | perhaps it's just being noisy... the script did return 0 | 16:56 |
| nibalizer | i expect a warning w/ continue ( also I thought i > /dev/null >&1 on that) | 16:56 |
| * nibalizer very excited that pboard is going live! | 16:56 | |
| fungi | i do have a /etc/puppet/modules/puppetboard/ so i guess it worked | 16:56 |
| *** MarkAtwood has joined #openstack-infra | 16:56 | |
| clarkb | fungi: yeah I have a hunch it is just telling us we are silly | 16:56 |
| fungi | i think *it's* the one who is being the sillyness | 16:57 |
| *** azneita has quit IRC | 16:58 | |
| *** e0ne_ has quit IRC | 16:58 | |
| *** esker has joined #openstack-infra | 16:59 | |
| *** dripton has joined #openstack-infra | 16:59 | |
| nibalizer | ah crap, yea I >/dev/null 2>&1 on the 'upgrade' line but not the 'install' line | 16:59 |
| *** roeyc has quit IRC | 16:59 | |
| nibalizer | i don't like having to supress all information though | 16:59 |
| nibalizer | it would be great if --without-dependencies implied --we-know-what-we're-doing | 17:00 |
| clarkb | nibalizer: lol but puppet is enterprise | 17:00 |
| *** krotscheck has quit IRC | 17:00 | |
| *** eharney has quit IRC | 17:00 | |
| jeblair | clarkb: --void-service-contract ? | 17:01 |
| fungi | jeblair: --pay-us-more | 17:01 |
| *** mgagne has joined #openstack-infra | 17:01 | |
| jeblair | wow. i so want to add that to something. | 17:01 |
| nibalizer | --customer-id=483BEACDF83721DEADBEEF | 17:01 |
| fungi | --heres-my-wallet | 17:01 |
| *** mrmartin has joined #openstack-infra | 17:02 | |
| jeblair | fungi: did we ever add --bitcoin-address to the zuul promote command? :) | 17:02 |
| nibalizer | we should paste this scrollback to a puppet bug on this haha | 17:02 |
| fungi | jeblair: first we'd need to open that through a public api | 17:02 |
| *** changbl has quit IRC | 17:02 | |
| * jeblair muses on the possibility of an application's public api actually being bitcoin transactions themselves... | 17:03 | |
| * fungi considers the possibility that stackforge unit test jobs could just be mining bitcoins | 17:03 | |
| nibalizer | jeblair: i guess namecoin does DNS through the coin mesh ... maybe thats like what you're thinking | 17:03 |
| *** mrmartin has quit IRC | 17:04 | |
| *** SnowDust has joined #openstack-infra | 17:04 | |
| *** nkinder has joined #openstack-infra | 17:04 | |
| nibalizer | fungi: my classic line if i want to DOS someone is to write an openstack test to fire the openstack testing infra at the victim | 17:04 |
| fungi | nibalizer: all you really need to do is configure devstack to start git-cloning from their website | 17:05 |
| fungi | we've done that to ourselves before ;) | 17:05 |
| *** mwagner_ has joined #openstack-infra | 17:07 | |
| nibalizer | hahahhaha | 17:07 |
| nibalizer | the devstack cannon | 17:07 |
| nibalizer | i love it | 17:07 |
| *** nati_ueno has quit IRC | 17:08 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Only enable nodepool precise-backports on Ubuntu https://review.openstack.org/81852 | 17:08 |
| *** hogepodge has quit IRC | 17:08 | |
| *** nati_ueno has joined #openstack-infra | 17:08 | |
| annegentle | clarkb: fungi: is an upstream required to be defined when making a new repo? Thinking of the security notes repo creation: https://review.openstack.org/#/c/73157/ | 17:09 |
| fungi | annegentle: only if you already have an existing repository you want to import | 17:10 |
| *** ominakov has quit IRC | 17:10 | |
| fungi | annegentle: otherwise it creates an empty initial repository with just a .gitreview files for you to start using | 17:10 |
| annegentle | clarkb: fungi: the idea is that the security notes will be housed in git, so is it ok to import from nkinder first? | 17:10 |
| *** sabari has joined #openstack-infra | 17:10 | |
| annegentle | nkinder: then we can use yours for the initial import no problem | 17:10 |
| *** skraynev is now known as skraynev_afl | 17:10 | |
| *** skraynev_afl is now known as skraynev_afk | 17:10 | |
| fungi | annegentle: sure, if he's already got them in a git repository published somewhere else, then that's fine. otherwise they'd need to be added as initial commits after the gerrit project creation happens. either is a valid means to the same end | 17:11 |
| annegentle | nkinder: then the only other wording question is whether the repo itself can be named security-notes or openstack-security-notes | 17:11 |
| annegentle | fungi: ok thanks for the 'splainin' | 17:11 |
| clarkb | jeblair: http://paste.openstack.org/show/73925/ zuul merger oddness. However after looking at that again I notice the git fetch pack ref spec looks fishy | 17:12 |
| *** hogepodge has joined #openstack-infra | 17:12 | |
| nkinder | annegentle: for naming, I preferred "openstack-security-notes" since that maps to the OSSN abbreviation that refers to them | 17:12 |
| clarkb | jeblair: so you might scroll down and look at the ps output first | 17:12 |
| *** ominakov has joined #openstack-infra | 17:12 | |
| nkinder | annegentle: there were some other openstack/openstack-* repos already too | 17:12 |
| *** caleb_ has quit IRC | 17:12 | |
| *** nati_ueno has quit IRC | 17:12 | |
| fungi | nibalizer: so... what am i supposed to be connecting to so i can see the puppetboard interface? the firewall configuration only specified 8081/tcp open but whatever's listening on that closes the connection as soon as i try a get request on it | 17:13 |
| fungi | nibalizer: i see something listening on 80/tcp too, and if i manually open that in iptables i get a 404 response on / | 17:13 |
| fungi | seems to be an apache running on it | 17:14 |
| annegentle | nkinder: yeah I'm the worst, making openstack/openstack-manuals :) | 17:15 |
| annegentle | nkinder: hadn't considered the ossn | 17:15 |
| annegentle | nkinder: can you put that in a reply on the review so we have a record of the decision reasoning? | 17:15 |
| annegentle | nkinder: then I'll +1 | 17:15 |
| nkinder | annegentle: yep, will do that now | 17:15 |
| annegentle | nkinder: with openstack/openstack-security-notes if you like it | 17:15 |
| nibalizer | fungi: puppetboard should be listening on 80 | 17:17 |
| jeblair | clarkb: that fetch is funny lookin | 17:17 |
| fungi | nibalizer: hmmm... perhaps apache is grabbing it instead | 17:17 |
| *** vkozhukalov has quit IRC | 17:17 | |
| nibalizer | well apache should be giving it to wsgi | 17:18 |
| clarkb | jeblair: google seems to show that efs/heads/master is a thing | 17:18 |
| nibalizer | whats in /etc/httpd/conf.d ? | 17:18 |
| nibalizer | is there another vhost than puppet thats taking precedence? | 17:18 |
| jeblair | there's nothing in sites-enabled | 17:20 |
| *** sabari has quit IRC | 17:20 | |
| fungi | nibalizer: http://paste.openstack.org/show/73928/ | 17:20 |
| jeblair | oh, it's in conf.d | 17:20 |
| clarkb | jeblair: http://code.google.com/p/chromium/issues/detail?id=90675 seems similar | 17:21 |
| fungi | jeblair: no, i think you're right--we seem to be missing a vhost definition | 17:21 |
| clarkb | though with older versions of git | 17:21 |
| jeblair | fungi: yeah | 17:21 |
| *** nicedice has quit IRC | 17:21 | |
| *** sabari has joined #openstack-infra | 17:21 | |
| clarkb | oh but server side would be old because centos | 17:21 |
| jeblair | clarkb: oh it _is_ using communicate, interesting | 17:21 |
| clarkb | but in upstream land we serve that stuff off of gerrit which is not older centos stuff | 17:22 |
| clarkb | jeblair: yeah, I am beginning to think gitpython may not be at fault directly and instead this is another centos git + precise git oddity | 17:22 |
| clarkb | jeblair: I will switch the protocol to git:// and see if that makes a difference | 17:23 |
| jeblair | clarkb: ack | 17:23 |
| clarkb | the git code should be different between http and git | 17:23 |
| fungi | nibalizer: so i think we want an apache vhost definition and also a firewall rule to open 80/tcp (and maybe close 8081/tcp?) | 17:23 |
| *** morganfainberg_Z is now known as morganfainberg | 17:23 | |
| *** yassine has quit IRC | 17:23 | |
| markmcclain | fungi: this hit the hour limit: https://review.openstack.org/#/c/81025/ | 17:24 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Delete created keypairs if nova boot fails https://review.openstack.org/81647 | 17:24 |
| fungi | nibalizer: the vhost definition could probably just set docroot to /srv/puppetboard/puppetboard and that would work, i suspect | 17:24 |
| markmcclain | fungi: sadly the UT passed, but the job failed before log uploaded | 17:25 |
| fungi | markmcclain: i'm not surprised. i'm actually sort of surprised we weren't noticing more of them. we've been running those on hpcloud for a while, so unless the neutron unit tests recently got a good bit slower i can't think of another likely trigger | 17:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add queue-name parameter to job https://review.openstack.org/81320 | 17:26 |
| markmcclain | yeah.. salv-orlando and I were looking at the speed of a few tests | 17:26 |
| nibalizer | fungi: sounds plausible | 17:26 |
| nibalizer | honestly my apache is a bit weak so if you know how to fix it, you should go for it | 17:26 |
| jeblair | fungi: py3k in all regions but az1 and az2 is ready | 17:26 |
| nibalizer | we need 8081/tcp open so that the puppet master can hit puppetdb | 17:27 |
| fungi | markmcclain: at this point our options are to go back to not running python26 jobs in hpcloud, increase the overall time limit for everyone's python26 jobs, make neutron python26 jobs special in such a way that we can set a longer timeout for them, or find a way to get them to run faster | 17:27 |
| fungi | jeblair: thanks! | 17:27 |
| *** krtaylor has joined #openstack-infra | 17:27 | |
| jeblair | fungi: --"not running python26 jobs in hpcloud"; ambivalent on others | 17:27 |
| fungi | jeblair: agreed, mainly just enumerating potential solutions. i agree i'd rather not go back to limiting what we run where | 17:28 |
| jeblair | clarkb: i haven't seen any email about it so i assume there are no updates on any of the 3 hpcloud issues? (dns, metadata, quota) | 17:29 |
| clarkb | jeblair: none | 17:29 |
| fungi | i'd also rather not make one project's python test jobs broken out specially to be able to set a separate timeout since that seems fiddly... i'm leaning toward just raising the gate-{name}-python26 job template timeout to 90 minutes | 17:29 |
| jeblair | oh wait i just now see an email about quota! | 17:29 |
| markmcclain | fungi, jeblair: so we could one slightly blunt quick fix.. disable XML unit testing in py26 | 17:29 |
| *** rpodolyaka1 has joined #openstack-infra | 17:30 | |
| jeblair | clarkb: tldr: quota==done | 17:30 |
| clarkb | jeblair: wow I expected a bunch of pusback but yay \o/ | 17:30 |
| fungi | markmcclain: but xml is enterprise! i can imagine how anyone could live without it | 17:30 |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: add query for bug 1295261 https://review.openstack.org/81855 | 17:30 |
| fungi | er, can't | 17:30 |
| uvirtbot | Launchpad bug 1295261 in keystone "test_v3_os_revoke.OSRevokeTests: invalid event issued_before time; Too early" [Critical,New] https://launchpad.net/bugs/1295261 | 17:30 |
| jeblair | clarkb: we didn't ask for enough! | 17:30 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Fix elasticsearch URL for gate queue https://review.openstack.org/81856 | 17:30 |
| salv-orlando | enterprise as in star trek's ship? | 17:30 |
| *** afazekas has quit IRC | 17:30 | |
| *** ildikov_ has quit IRC | 17:31 | |
| jogo | jeblair: ^^ | 17:32 |
| jeblair | salv-orlando: yeah, but it's the 1701-B | 17:32 |
| fungi | salv-orlando: yes, but one of the early ones which looked like a salad plate with prongs | 17:32 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Require gear 0.5.4 https://review.openstack.org/81628 | 17:32 |
| jeblair | fungi wins this round | 17:32 |
| *** markmc has quit IRC | 17:33 | |
| * fungi is a big fan of tos, but you have to admit the effects have not aged well, and the recent remaster with "new" cgi effects look kinda out of place | 17:33 | |
| *** rpodolyaka1 has quit IRC | 17:34 | |
| *** changbl has joined #openstack-infra | 17:34 | |
| salv-orlando | fungi: so was it actually called <federation><planets><ship name="1701-b"/></planets></federation> ? | 17:34 |
| *** khyati has joined #openstack-infra | 17:35 | |
| fungi | salv-orlando: right, except in the future they've learned how to make xml less readable than that | 17:35 |
| jeblair | fungi: yeah, tbh i'd rather they not have done the cgi. as long as i'm wearing my imagination hat, it's better if i don't have to take it off. | 17:35 |
| salv-orlando | fungi: I see! I might carry this on forever. But since I think we should all go back to work I won't mention they will be doing XML in klingon | 17:36 |
| jeblair | salv-orlando: xml in _the original_ klingon | 17:36 |
| *** nkinder has left #openstack-infra | 17:36 | |
| *** caleb_ has joined #openstack-infra | 17:37 | |
| *** dcramer_ has joined #openstack-infra | 17:40 | |
| *** derekh has quit IRC | 17:40 | |
| *** dizquierdo has quit IRC | 17:42 | |
| *** _nadya_ has joined #openstack-infra | 17:43 | |
| jogo | clarkb: logstash is cought up right? | 17:43 |
| clarkb | jogo: yup | 17:43 |
| jogo | whoa then it looks like the libvirt bug fix worked | 17:43 |
| *** gokrokve has joined #openstack-infra | 17:44 | |
| *** gokrokve_ has quit IRC | 17:44 | |
| *** SnowDust has quit IRC | 17:46 | |
| *** wchrisj_ has joined #openstack-infra | 17:50 | |
| *** gokrokve has quit IRC | 17:50 | |
| *** dprince_ has joined #openstack-infra | 17:50 | |
| *** gokrokve has joined #openstack-infra | 17:50 | |
| *** dprince has quit IRC | 17:50 | |
| *** jooools has quit IRC | 17:55 | |
| *** gokrokve has quit IRC | 17:55 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Fix elasticsearch URL for gate queue https://review.openstack.org/81856 | 17:55 |
| jeblair | clarkb: what is hpcloud region-a.geo-1 ? | 17:57 |
| clarkb | jeblair: it is the 1.1 counterpart to region-b in the west region. I was told not to use it yet as our capacity in 1.0 west is apparently shared with region-a or some such | 17:58 |
| clarkb | jeblair: once we are in region-b we can probably ask about putting half our stuff in region-a and half in region-b | 17:58 |
| jeblair | clarkb: gotcha | 17:58 |
| *** yolanda has quit IRC | 17:58 | |
| *** SnowDust has joined #openstack-infra | 17:58 | |
| *** yolanda has joined #openstack-infra | 17:59 | |
| *** dprince_ has quit IRC | 18:00 | |
| *** dprince has joined #openstack-infra | 18:00 | |
| jeblair | oh! we still aren't useng pvhvm for devstack-precise! | 18:00 |
| fungi | not for bare-precise are we? | 18:01 |
| fungi | er, nor | 18:01 |
| jogo | 2014-03-20 12:35:44.624 | Downloading/unpacking pyelasticsearch (from elastic-recheck==51d2840) | 18:01 |
| clarkb | right only that one check image uses it | 18:01 |
| clarkb | we should fix that | 18:01 |
| jogo | 2014-03-20 12:35:44.624 | Some externally hosted files were ignored (use --allow-external pyelasticsearch to allow). | 18:02 |
| *** Ryan_Lane has quit IRC | 18:02 | |
| jeblair | correct, only devstack-precise-check and bare-centos6 | 18:02 |
| clarkb | jogo: ugh did they go externally hosted? | 18:02 |
| *** mriedem has quit IRC | 18:02 | |
| fungi | https://pypi.python.org/simple/pyelasticsearch/ has tarballs up through 0.6.1 | 18:03 |
| jogo | hmm it worked locally with pip 1.5.2 | 18:03 |
| fungi | jogo: got a link to the failure log? might have been a network blip looking at the simple index or something | 18:05 |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: add query for bug 1294971 https://review.openstack.org/81871 | 18:06 |
| uvirtbot | Launchpad bug 1294971 in keystone "gate-grenade-dsvm-partial-ncpu Failure in upgrade-keystone" [Critical,Triaged] https://launchpad.net/bugs/1294971 | 18:06 |
| *** caleb_` has joined #openstack-infra | 18:06 | |
| jogo | fungi: http://logs.openstack.org/49/81649/1/gate/gate-elastic-recheck-pep8/84e6c09/console.html | 18:06 |
| jogo | running a recheck to see whats going on | 18:06 |
| *** signed8bit has quit IRC | 18:07 | |
| clarkb | jogo: fwiw your change to fix the AND string appendage did pass and merge | 18:07 |
| *** Ryan_Lane has joined #openstack-infra | 18:08 | |
| *** caleb_ has quit IRC | 18:09 | |
| fungi | "openstack-infra/elastic-recheck will not use the internal openstack pypi mirror" | 18:09 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add all images and increase usage of region-b https://review.openstack.org/81873 | 18:09 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Stop using hpcloud az1-az3 in favor of region-b https://review.openstack.org/81874 | 18:09 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove hpcloud az1-az3 https://review.openstack.org/81875 | 18:09 |
| fungi | so that's good at least | 18:09 |
| fungi | oh... it failed to install, i think? | 18:11 |
| *** rpodolyaka1 has joined #openstack-infra | 18:11 | |
| fungi | Downloading/unpacking pyelasticsearch (from elastic-recheck==51d2840) | 18:11 |
| fungi | No distributions at all found for pyelasticsearch in ./.tox/pep8/lib/python2.7/site-packages (from elastic-recheck==51d2840) | 18:11 |
| fungi | we should maybe start collecting /home/jenkins/.pip/pip.log on those jobs | 18:12 |
| *** rpodolyaka1 has quit IRC | 18:13 | |
| fungi | i think that slave might have been experiencing some intermittent network issues though... note at the beginning of the log it fails to fetch origin from git.o.o and has to retry | 18:13 |
| fungi | 2014-03-20 12:31:11.435 | git.openstack.org[0: 192.237.223.224]: errno=Connection timed out | 18:14 |
| *** mriedem has joined #openstack-infra | 18:14 | |
| *** rpodolyaka1 has joined #openstack-infra | 18:17 | |
| jogo | fungi: ahh anyway the recheck should sort that out | 18:17 |
| *** stack555 has joined #openstack-infra | 18:18 | |
| ryanpetrello | anybody know the status of publishing wheels to pypi via openstackci ? | 18:18 |
| clarkb | ryanpetrello: tox is waffling about the best way to fix the problem in 1.7.0... | 18:19 |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: add query for bug 1295261 https://review.openstack.org/81855 | 18:19 |
| uvirtbot | Launchpad bug 1295261 in keystone "test_v3_os_revoke.OSRevokeTests: invalid event issued_before time; Too early" [Undecided,Triaged] https://launchpad.net/bugs/1295261 | 18:19 |
| clarkb | ryanpetrello: that said last I checked latest pip and virtualenv did work for us | 18:19 |
| clarkb | ryanpetrello: we can in theory move to latest pip and virtualenv with older tox and produce and consume wheels that way | 18:20 |
| ryanpetrello | I see | 18:20 |
| *** Ajaeger has joined #openstack-infra | 18:20 | |
| ryanpetrello | just loosely following https://review.openstack.org/#/c/57176/ | 18:20 |
| stack555 | Hello I am working on the folllowing review of adding test cases for nuage plugin https://review.openstack.org/#/c/81673/ .. I have added a decorator requires_ext='net-parition' in my test case so that it runs only when nuage plugin is enabled but I noticed that when i checked out code the tempest.conf by default has api_extension set to all. | 18:20 |
| ryanpetrello | but figured there wasn't much point in merging this until I can actually produce/publish them since we release w/ the CI tools | 18:20 |
| clarkb | ryanpetrello: the biggest problems have been reconsuming wheels sanely so that we know we aren't releasing badness | 18:21 |
| stack555 | Because of this my test case is run by jenkins all the time | 18:21 |
| stack555 | Is there a way I can get around this? | 18:21 |
| clarkb | stack555: you may have better luck talking tempest over in #openstack-qa | 18:21 |
| *** gokrokve has joined #openstack-infra | 18:21 | |
| *** SnowDust has left #openstack-infra | 18:21 | |
| nibalizer | fungi: to be clear, I'm at work right now so I can't help debug | 18:21 |
| nibalizer | i'll let you know when I'm free | 18:21 |
| stack555 | sure thank you | 18:21 |
| *** zehicle_at_dell has joined #openstack-infra | 18:22 | |
| fungi | nibalizer: it's not urgent, just giving you a heads up on the list of what we know to be missing now | 18:22 |
| *** rpodolyaka1 has quit IRC | 18:22 | |
| nibalizer | okay sweet | 18:22 |
| fungi | nibalizer: and thanks, as always! | 18:22 |
| nibalizer | fungi: happy to help | 18:22 |
| *** packet has joined #openstack-infra | 18:26 | |
| *** jlibosva has quit IRC | 18:26 | |
| *** ominakov has quit IRC | 18:30 | |
| *** gokrokve has quit IRC | 18:31 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1294874 https://review.openstack.org/81649 | 18:31 |
| uvirtbot | Launchpad bug 1294874 in postgresql "Sub-process /usr/bin/dpkg returned an error code" [Undecided,New] https://launchpad.net/bugs/1294874 | 18:31 |
| kevinbenton | can the timeout be increased for UTs in neutron? | 18:32 |
| kevinbenton | http://logs.openstack.org/25/81025/4/gate/gate-neutron-python26/11dbe19/console.html | 18:32 |
| *** rcleere has joined #openstack-infra | 18:32 | |
| kevinbenton | gate reset because of it | 18:32 |
| kevinbenton | and it was very close to finishing | 18:32 |
| clarkb | kevinbenton: sure, anyone can propose the change | 18:32 |
| kevinbenton | where to? | 18:32 |
| *** thuc_ has quit IRC | 18:32 | |
| *** thuc has joined #openstack-infra | 18:33 | |
| *** jnoller has joined #openstack-infra | 18:33 | |
| *** rpodolyaka1 has joined #openstack-infra | 18:33 | |
| clarkb | kevinbenton: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml#n53 that said it would also be nice to see neutron speed up their tests if possible. markmcclain and salv-orlando were looking at that earlier | 18:33 |
| *** johnthetubaguy has quit IRC | 18:34 | |
| *** markmcclain has quit IRC | 18:34 | |
| arosen | Just curious is there a page that shows the history of merges and gate failures? | 18:34 |
| arosen | #openstack-merges just shows the merges not the failures. | 18:35 |
| Ajaeger | Anybody around that can help me with a tox.ini question? | 18:35 |
| clarkb | Ajaeger: sure | 18:35 |
| Ajaeger | https://jenkins05.openstack.org/job/openstack-install-deploy-guide-ja/43/console shows "could not find executable 'cd'" | 18:35 |
| Ajaeger | clarkb: great, thanks! | 18:35 |
| clarkb | arosen: https://github.com/openstack/openstack/graphs/commit-activity is another handy place to consume merges | 18:35 |
| Ajaeger | clarkb: line is "cd generated; cd {posargs} ; cp ../doc/pom.xml ." | 18:35 |
| *** jgallard has quit IRC | 18:35 | |
| clarkb | arosen: the gate failures get posted to #openstack-qa via elastic-recheck | 18:36 |
| Ajaeger | I had previously " cp doc/pom.xml generated/{posargs}/pom.xml" but that failed with tox 1.7 | 18:36 |
| clarkb | Ajaeger: yup tox 1.7.0 is fun that way | 18:36 |
| clarkb | Ajaeger: all the other projects currently require 1.6.1 | 18:36 |
| kevinbenton | clarkb: i agree. but existing patches should not be declined | 18:36 |
| Ajaeger | and so we changed to work with both. It worked fine for us locally with the changed line - with both 1.6.1 and 1.7. | 18:36 |
| arosen | clarkb: thanks. | 18:36 |
| clarkb | kevinbenton: well it could be that those particular patches are the problem patches and should be declined | 18:36 |
| Ajaeger | Should we just hard-require 1.6.1? I see "minversion=1.6" but not maxversion - does that exist? | 18:37 |
| *** thuc has quit IRC | 18:37 | |
| kevinbenton | clarkb: i've had several patches now fail because of this though and subsequently pass on retry. the current state of trunk runs very close to 60 minutes | 18:37 |
| clarkb | Ajaeger: maxversion does not exist, so it is an unenforceable requirement | 18:38 |
| Ajaeger | Or do you have a better idea on how to make this work with both 1.6 and 1.7 - as I've said, it worked locally for us but fails only on jenkins. | 18:38 |
| Ajaeger | clarkb: I wanted to support users that run 1.7... | 18:39 |
| *** zehicle_at_dell has quit IRC | 18:39 | |
| *** yolanda has quit IRC | 18:39 | |
| clarkb | Ajaeger: right, I think that is a good thing to do if you can swing it. The rest of openstack doesn't currently do that though | 18:39 |
| *** zehicle_at_dell has joined #openstack-infra | 18:39 | |
| Ajaeger | clarkb: I failed with it as you noticed - and wonder why it works locally but fails in the post job ;( | 18:40 |
| clarkb | Ajaeger: are you using tox 1.6.1 locally? | 18:40 |
| Ajaeger | Is something wrong in the command line with cd? | 18:40 |
| Ajaeger | clarkb: yes, I do | 18:41 |
| clarkb | Ajaeger: no, best guess is since cd is a builtin tox may be trying to exec cd | 18:41 |
| clarkb | Ajaeger: do you have the change handy that updated the tox.ini? | 18:41 |
| Ajaeger | clarkb: it fails locally now as well for me. | 18:41 |
| fungi | kevinbenton: i agree it's unfortunate. we set a time limit on how long those jobs are allowed to run so that they don't get stuck and run indefinitely. at one time an hour seemed like a long time, but people keep piling things in and the job takes longer and longer to run, so eventually they bump up against that limit | 18:41 |
| Ajaeger | Ok, back to debugging locally. ;( | 18:41 |
| * Ajaeger thought it was working locally. | 18:41 | |
| clarkb | Ajaeger: my hunch is for some reason cd is being exec'd directly rather than in a shell | 18:41 |
| Ajaeger | Sorry for not double checking and bothering... | 18:42 |
| Ajaeger | might be... | 18:42 |
| *** sarob has joined #openstack-infra | 18:42 | |
| clarkb | Ajaeger: you can try something like bash -c 'cd ja; do next thing; and next thing' | 18:42 |
| clarkb | and see if that fixes it | 18:42 |
| fungi | kevinbenton: the line you want to propose a change for is http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml#n58 | 18:42 |
| Ajaeger | I'll give it a try, thanks | 18:42 |
| fungi | kevinbenton: 90 minutes might be a sane next step | 18:42 |
| kevinbenton | fungi: that won't even be necessary because i agree that it needs to stop ballooning | 18:43 |
| kevinbenton | fungi: it's very close to being ok | 18:44 |
| *** ildikov_ has joined #openstack-infra | 18:45 | |
| fungi | kevinbenton: agreed. i watched several this morning running in hpcloud (where we're stuck using only 4 vcpus on a test node) complete right at 59 minutes | 18:46 |
| openstackgerrit | Kevin Benton proposed a change to openstack-infra/config: Increase timeout for py26/27 unit tests https://review.openstack.org/81885 | 18:47 |
| fungi | clarkb: jeblair: if one of you is okay with 81885, that will likely cut down on some unnecessary gate resets in the near term | 18:48 |
| *** ryanpetrello has quit IRC | 18:48 | |
| jeblair | wfm aprvd | 18:48 |
| *** miarmak has quit IRC | 18:49 | |
| *** miarmak has joined #openstack-infra | 18:49 | |
| fungi | thx | 18:49 |
| *** ryanpetrello has joined #openstack-infra | 18:49 | |
| *** amcrn has joined #openstack-infra | 18:51 | |
| openstackgerrit | A change was merged to openstack-infra/config: Enable Neutron for Ironic Devstack jobs https://review.openstack.org/81611 | 18:51 |
| *** gokrokve has joined #openstack-infra | 18:54 | |
| Ajaeger | clarkb, I'll create a script now that does the job - ugly ;( | 18:57 |
| fungi | grrr... my recheck hit hpcloud-az1 where the py3k-precise image hasn't built yet (image-list says there's one in a building state there for nearly 3 hours) | 18:58 |
| *** caleb_` has quit IRC | 18:58 | |
| fungi | er, nearly 2 hours | 18:59 |
| fungi | looks like that's the only one still building too... luck of the draw | 18:59 |
| *** peddamat has quit IRC | 18:59 | |
| fungi | oh, hpcloud-az2 still as well | 19:01 |
| fungi | right, same ones jeblair said earlier | 19:01 |
| rcarrillocruz | helo helo, could any core reviewer +2 https://review.openstack.org/#/c/81234/ please? | 19:02 |
| jeblair | fungi: az2 failed, i just rekicked it. az1 is still building and may have a chance of completing | 19:03 |
| fungi | jeblair: given my luck on the past few rechecks, i'll just stop wasting resources rechecking until they both complete | 19:03 |
| *** thuc has joined #openstack-infra | 19:04 | |
| *** jeckersb is now known as jeckersb_gone | 19:05 | |
| *** signed8bit has joined #openstack-infra | 19:06 | |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: add query for bug 1294971 https://review.openstack.org/81871 | 19:09 |
| uvirtbot | Launchpad bug 1294971 in keystone "gate-grenade-dsvm-partial-ncpu Failure in upgrade-keystone" [Critical,Triaged] https://launchpad.net/bugs/1294971 | 19:09 |
| *** thuc_ has joined #openstack-infra | 19:12 | |
| *** thuc has quit IRC | 19:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: Increase timeout for py26/27 unit tests https://review.openstack.org/81885 | 19:15 |
| *** vkozhukalov has joined #openstack-infra | 19:15 | |
| zaro | jeblair: can you tell me what starts the gearman-job-server process on zuul.o.o? | 19:16 |
| *** lcostantino has quit IRC | 19:16 | |
| jeblair | zaro: zuul has an internal gear server | 19:17 |
| *** lcostantino has joined #openstack-infra | 19:17 | |
| *** jeckersb_gone is now known as jeckersb | 19:17 | |
| zaro | jeblair: oh. i see, thanks. | 19:17 |
| *** caleb_ has joined #openstack-infra | 19:18 | |
| zaro | jeblair: i was just wondering if gearman was running on zuul-dev as well so that answers my question. | 19:18 |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: provide more specific feedback to query authors https://review.openstack.org/81891 | 19:18 |
| *** _nadya_ has quit IRC | 19:18 | |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: add query for bug 1295261 https://review.openstack.org/81855 | 19:19 |
| uvirtbot | Launchpad bug 1295261 in keystone "test_v3_os_revoke.OSRevokeTests: invalid event issued_before time; Too early" [Undecided,In progress] https://launchpad.net/bugs/1295261 | 19:19 |
| *** alff has joined #openstack-infra | 19:23 | |
| zaro | jeblair: also i noticed a few zuul_merger services got added to review.o.o, but not to zuul-dev. Should those also get added to zuul-dev? | 19:25 |
| jeblair | zaro: do you mean they got added to zuul.o.o? (you said review.o.o) | 19:26 |
| zaro | jeblair: looks like there are new servers zm01.openstack.org & zm02.o.o in site.pp | 19:27 |
| jeblair | zaro: yes; still not sure what you're asking in regard to review.o.o. | 19:28 |
| zaro | jeblair: they appear as gearman workers on zuul.o.o, but not in zuul-dev.o.o | 19:29 |
| jeblair | zaro: right. so i'm going to assume you're not talking about review.o.o at all because that doesn't make sense. | 19:29 |
| zaro | jeblair: should those also be workers in zuul-dev? | 19:29 |
| *** yolanda has joined #openstack-infra | 19:29 | |
| jeblair | zaro: zuul-dev should run a zuul merger on it locally | 19:29 |
| *** mrmartin has joined #openstack-infra | 19:30 | |
| mrmartin | re | 19:30 |
| *** _nadya_ has joined #openstack-infra | 19:31 | |
| *** _nadya_ has quit IRC | 19:31 | |
| zaro | jeblair: ok, gota be afk now. will look into making a change for that. | 19:31 |
| *** andreaf has quit IRC | 19:31 | |
| zaro | thanks. | 19:31 |
| *** _nadya_ has joined #openstack-infra | 19:32 | |
| *** caleb_ has quit IRC | 19:33 | |
| *** _nadya_ has joined #openstack-infra | 19:33 | |
| *** packet has quit IRC | 19:33 | |
| *** packet has joined #openstack-infra | 19:34 | |
| *** JoshNang has left #openstack-infra | 19:34 | |
| *** malini is now known as malini_afk | 19:34 | |
| *** _nadya_ has quit IRC | 19:36 | |
| *** caleb_ has joined #openstack-infra | 19:37 | |
| *** rpodolyaka1 has quit IRC | 19:38 | |
| *** njd001 has quit IRC | 19:39 | |
| *** rpodolyaka1 has joined #openstack-infra | 19:44 | |
| *** signed8bit has quit IRC | 19:45 | |
| *** yolanda has quit IRC | 19:48 | |
| *** bhuvan has quit IRC | 19:48 | |
| *** jamielennox is now known as jamielennox|away | 19:48 | |
| *** rpodolyaka1 has quit IRC | 19:49 | |
| *** zhiyan_ is now known as zhiyan | 19:50 | |
| *** wchrisj_ has quit IRC | 19:51 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Call nodepool devstack deps with bash for TripleO https://review.openstack.org/81899 | 19:52 |
| pleia2 | ^^ that should be the last of the trio of patches I have up | 19:52 |
| pleia2 | | c8fe411f-4de9-4d1e-87ff-ae90afb6d4e9 | tripleo-fedora-tripleo-test-cloud-1.slave.openstack.org | ACTIVE | - | Running | default-net=10.0.72.85, 138.35.77.72 | | 19:53 |
| pleia2 | :) | 19:53 |
| pleia2 | (tested on my own account on the tripleo cloud) | 19:53 |
| *** mrmartin has quit IRC | 19:55 | |
| openstackgerrit | A change was merged to openstack-infra/config: Openstackid track site version https://review.openstack.org/80996 | 19:55 |
| *** caleb_` has joined #openstack-infra | 19:55 | |
| *** signed8bit has joined #openstack-infra | 19:56 | |
| *** caleb_ has quit IRC | 19:57 | |
| *** thuc_ has quit IRC | 19:58 | |
| *** bhuvan has joined #openstack-infra | 19:58 | |
| *** thuc has joined #openstack-infra | 19:58 | |
| *** thuc_ has joined #openstack-infra | 19:59 | |
| *** caleb_` has quit IRC | 20:00 | |
| *** markwash has joined #openstack-infra | 20:01 | |
| jeblair | fungi: az1 is ready | 20:01 |
| jeblair | (only az2 remains) | 20:01 |
| *** hashar has joined #openstack-infra | 20:02 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Always create .venv for upstream_translation_horizon https://review.openstack.org/81902 | 20:02 |
| *** markwash has quit IRC | 20:02 | |
| *** markwash has joined #openstack-infra | 20:02 | |
| *** thuc has quit IRC | 20:03 | |
| *** caleb_ has joined #openstack-infra | 20:04 | |
| *** caleb_ has quit IRC | 20:05 | |
| *** caleb_ has joined #openstack-infra | 20:05 | |
| *** e0ne has joined #openstack-infra | 20:06 | |
| clarkb | ok back from lunch | 20:08 |
| clarkb | pleia2: why call it with bash? is there a permissions problem we need to fix instead? | 20:10 |
| pleia2 | clarkb: permissions seem fine, I'm really not sure what Fedora is grumping about | 20:10 |
| boris-42 | clarkb hey there | 20:10 |
| clarkb | pleia2: huh | 20:10 |
| pleia2 | works fine to call ./prepare_node.py from the first script | 20:10 |
| pleia2 | happy to have other solutions to test, I know my fix is not optimal | 20:11 |
| boris-42 | clarkb I would like to take a part in infra/* stuff and help with reviewing for start=) | 20:11 |
| clarkb | pleia2: I think you need to +x it in the git repo | 20:11 |
| clarkb | boris-42: awesome | 20:11 |
| boris-42 | clarkb is there any way to get reviews for all openstack-infra/* patches?) | 20:12 |
| pleia2 | clarkb: seems to be when I clone it | 20:12 |
| boris-42 | clarkb I mean in gerrit | 20:12 |
| boris-42 | clarkb patches* | 20:12 |
| clarkb | boris-42: I think most of us set up gerrit watches for all projects with the openstack-infra/ prefix | 20:12 |
| clarkb | pleia2: mine is 644 | 20:12 |
| pleia2 | clarkb: ah, interesting, I'll fix | 20:13 |
| *** bhuvan has quit IRC | 20:13 | |
| *** bhuvan has joined #openstack-infra | 20:13 | |
| *** thuc has joined #openstack-infra | 20:14 | |
| *** rlandy has quit IRC | 20:14 | |
| boris-42 | clarkb should I add one by one or there is simpler way?) | 20:16 |
| clarkb | boris-42: one by one is the only way gerrit allows us to do it I think. the key ones would probably be openstack-infra/config /zuul /nodepool /gear | 20:17 |
| clarkb | those make up the bulk of the reviews right now | 20:17 |
| boris-42 | clarkb btw what is the simplest for start?) | 20:17 |
| *** dprince has quit IRC | 20:17 | |
| clarkb | the config repo is probably the most important one because it gives you an overview of everything | 20:17 |
| clarkb | the zuul repo would be the next best one to watch | 20:17 |
| boris-42 | clarkb so I should start from it?) | 20:18 |
| boris-42 | clarkb btw one question is there any way to test patches?) | 20:18 |
| *** thuc_ has quit IRC | 20:18 | |
| boris-42 | clarkb I mean locally | 20:18 |
| *** thuc has quit IRC | 20:18 | |
| *** bhuvan has quit IRC | 20:18 | |
| fungi | though leaf projects also could use some love and don't necessarily require as much holistic knowledge... for example git-review or jenkins-job-builder (though the latter already has its own separate core reviewer group who are doing a great job) | 20:19 |
| *** njd001 has joined #openstack-infra | 20:19 | |
| clarkb | I would start with config and zuul. There are ways to test. zuul has a pretty decent built in functional test suite. just run `tox` to run those tests. For config things are trickier and usually oyu need to spin up a VM you don't care about somewhere and apply the puppet changes. http://ci.openstack.org/sysadmin.html#making-a-change-in-puppet documents a way to test | 20:20 |
| Ajaeger | has anybody an idea why many of our translation jobs fail right now? I see for example in https://jenkins.openstack.org/job/keystone-upstream-translation-update/1354/consoleText: | 20:20 |
| Ajaeger | Error uploading file: There is a syntax error in your file. | 20:20 |
| Ajaeger | Line 1972: duplicate message definition... | 20:20 |
| Ajaeger | Line 7: ...this is the location of the first definition | 20:20 |
| Ajaeger | msgfmt: found 1 fatal error | 20:20 |
| boris-42 | clarkb okay I will try to start with config and zuul | 20:20 |
| boris-42 | clarkb and then we will see | 20:21 |
| Ajaeger | And the same message in nova as well: https://jenkins.openstack.org/job/nova-upstream-translation-update/lastStableBuild/consoleText | 20:21 |
| fungi | Ajaeger: i think it's been that nobody's been working on keeping them in shape (some of those projects may not have many/any actual translated strings anyway) | 20:21 |
| clarkb | Ajaeger: I think that means the po file update is creating duplicates | 20:21 |
| *** mflobo_ has joined #openstack-infra | 20:22 | |
| anteaya | boris-42: status:open project:^openstack-infra/.* | 20:22 |
| anteaya | https://review.openstack.org/#/q/status:open+project:%255Eopenstack-infra/.*,n,z | 20:22 |
| boris-42 | anteaya oh thanks a lot=) | 20:22 |
| anteaya | np | 20:22 |
| boris-42 | anteaya i tried both at then end /.* and in "^../.*" but didn't try without ""=) | 20:23 |
| Ajaeger | what a mess ;( | 20:23 |
| boris-42 | lol | 20:23 |
| fungi | Ajaeger: that console log you linked to succeeded, i think because that part is being run under set +e | 20:23 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Fix install_devstack_dependencies.sh permissions https://review.openstack.org/81899 | 20:23 |
| anteaya | boris-42: yeah, took me a while to get the regex to work for myself, then saved it as a tweet for my own reference | 20:24 |
| * Ajaeger sees the failure now, there's indeed a duplicate. Just wonder where it comes from. | 20:24 | |
| pleia2 | hrm | 20:24 |
| Ajaeger | Ok, will ponder about this some more tomorrow. Just got one step further today: https://review.openstack.org/81902 | 20:24 |
| *** mflobo has quit IRC | 20:25 | |
| Ajaeger | thanks and bye for today! | 20:25 |
| *** e0ne has quit IRC | 20:25 | |
| *** markvoelker has left #openstack-infra | 20:25 | |
| *** mrda_away is now known as mrda | 20:25 | |
| *** flaper87|afk is now known as flaper87 | 20:25 | |
| *** sarob has quit IRC | 20:26 | |
| *** Ajaeger has quit IRC | 20:26 | |
| *** sarob has joined #openstack-infra | 20:26 | |
| *** bhuvan has joined #openstack-infra | 20:27 | |
| fungi | pleia2: your 81899 change adds a distro portability conditional around the precise backports addition, but isn't mentioned in the changelog (which only talks about executable bit on the other file) | 20:28 |
| *** julim has quit IRC | 20:28 | |
| fungi | oh, you abandoned anyway | 20:28 |
| *** e0ne has joined #openstack-infra | 20:29 | |
| pleia2 | fungi: yeah, going to just add it to the https://review.openstack.org/#/c/81852 | 20:29 |
| *** dstanek_afk has joined #openstack-infra | 20:30 | |
| *** sarob has quit IRC | 20:31 | |
| *** dstanek has quit IRC | 20:32 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Only enable nodepool precise-backports on Ubuntu https://review.openstack.org/81852 | 20:34 |
| *** caleb_ has quit IRC | 20:35 | |
| fungi | heading out to grab a bite--back in a while | 20:37 |
| pleia2 | enjoy | 20:37 |
| *** rwsu has quit IRC | 20:37 | |
| *** dkliban has quit IRC | 20:38 | |
| *** caleb_ has joined #openstack-infra | 20:39 | |
| *** mkoderer has quit IRC | 20:41 | |
| *** che-arne has joined #openstack-infra | 20:41 | |
| boris-42 | clarkb are you still there?) | 20:42 |
| clarkb | boris-42: yes | 20:42 |
| boris-42 | clarkb I am just about how "nerd" should I be?) | 20:43 |
| boris-42 | clarkb https://review.openstack.org/#/c/70636/8/zuul/scheduler.py this is okay to be merged?) | 20:43 |
| boris-42 | is this* | 20:43 |
| *** MarkAtwood has quit IRC | 20:44 | |
| *** FallenPegasus has joined #openstack-infra | 20:44 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add query for bug 1295261 https://review.openstack.org/81855 | 20:45 |
| *** rpodolyaka1 has joined #openstack-infra | 20:45 | |
| uvirtbot | Launchpad bug 1295261 in keystone "test_v3_os_revoke.OSRevokeTests: invalid event issued_before time; Too early" [Critical,In progress] https://launchpad.net/bugs/1295261 | 20:45 |
| clarkb | boris-42: suggestions like that are certainly helpful. We don't tend to get too picky about the code as long as it is correct and readable | 20:45 |
| clarkb | boris-42: I think both versions (your suggestion and the existing code) fit both criteria | 20:45 |
| *** dims has quit IRC | 20:46 | |
| boris-42 | clarkb okay so we may just +1 and left comment =) | 20:46 |
| clarkb | yup sounds good | 20:46 |
| boris-42 | ok | 20:46 |
| boris-42 | so now I should learn how to test =) it and what it actually does lol=) | 20:46 |
| *** mwagner_ has quit IRC | 20:46 | |
| *** rpodolyaka1 has quit IRC | 20:49 | |
| maurosr | hey guys, question, should this bug https://bugs.launchpad.net/tempest/+bug/1289218 go to infra instead of tempest? | 20:52 |
| uvirtbot | Launchpad bug 1289218 in tempest "Subunit discover error's aren't shown in logs" [Undecided,New] | 20:52 |
| *** marun has quit IRC | 20:53 | |
| clarkb | maurosr: its an upstream testrepository bug, fixed in testrepository but unreleased | 20:53 |
| *** dstanek_afk is now known as dstanek | 20:54 | |
| *** alff has quit IRC | 20:54 | |
| maurosr | clarkb: ok cool, I will mention that and mark it as invalid, tks! | 20:54 |
| *** jamielennox|away is now known as jamielennox | 20:55 | |
| *** lcostantino has quit IRC | 20:55 | |
| *** flaper87 is now known as flaper87|afk | 20:56 | |
| *** krtaylor has quit IRC | 20:57 | |
| *** dcramer_ has quit IRC | 21:00 | |
| *** dims has joined #openstack-infra | 21:02 | |
| pleia2 | heading out for dinner, I'll be back later | 21:03 |
| *** thuc has joined #openstack-infra | 21:03 | |
| pleia2 | will probably be out most of the east coast afternoon tomorrow (family lunch) | 21:04 |
| *** rwsu has joined #openstack-infra | 21:04 | |
| *** rfolco has quit IRC | 21:05 | |
| *** sarob has joined #openstack-infra | 21:06 | |
| *** aysyd has quit IRC | 21:07 | |
| *** jaypipes has quit IRC | 21:07 | |
| *** mdenny has quit IRC | 21:07 | |
| *** mdenny has joined #openstack-infra | 21:07 | |
| clarkb | pleia2: have fun | 21:08 |
| *** thomasem has quit IRC | 21:09 | |
| *** rpodolyaka1 has joined #openstack-infra | 21:09 | |
| *** lukego has quit IRC | 21:09 | |
| *** hogepodge has quit IRC | 21:10 | |
| *** sarob has quit IRC | 21:10 | |
| *** greghaynes has quit IRC | 21:11 | |
| *** sarob has joined #openstack-infra | 21:11 | |
| *** hogepodge has joined #openstack-infra | 21:12 | |
| *** greghaynes has joined #openstack-infra | 21:12 | |
| *** mdenny has quit IRC | 21:13 | |
| *** jaypipes has joined #openstack-infra | 21:14 | |
| *** rpodolyaka1 has quit IRC | 21:14 | |
| *** rpodolyaka1 has joined #openstack-infra | 21:15 | |
| *** thuc has quit IRC | 21:15 | |
| SlickNik | clarkb: is there a reason not to switch to git.openstack.org? (in a trove non-infra integration-test job). Github is acting flaky again. | 21:15 |
| *** thuc has joined #openstack-infra | 21:16 | |
| clarkb | SlickNik: nope, you should definitely switch | 21:16 |
| *** dkranz has quit IRC | 21:16 | |
| *** dkliban has joined #openstack-infra | 21:16 | |
| SlickNik | clarkb: Sounds good, will do. Thanks! | 21:16 |
| *** jamielennox is now known as jamielennox|away | 21:22 | |
| *** rpodolyaka1 has quit IRC | 21:24 | |
| *** vkozhukalov has quit IRC | 21:25 | |
| *** mfer has quit IRC | 21:25 | |
| *** flaper87|afk is now known as flaper87 | 21:27 | |
| *** caleb_ has quit IRC | 21:27 | |
| *** e0ne has quit IRC | 21:28 | |
| *** FallenPegasus has quit IRC | 21:34 | |
| *** flaper87 is now known as flaper87|afk | 21:34 | |
| *** afazekas has joined #openstack-infra | 21:36 | |
| *** esker has quit IRC | 21:36 | |
| *** krotscheck has joined #openstack-infra | 21:36 | |
| *** david-lyle has quit IRC | 21:37 | |
| *** weshay has quit IRC | 21:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Only enable nodepool precise-backports on Ubuntu https://review.openstack.org/81852 | 21:41 |
| *** branen_ has joined #openstack-infra | 21:44 | |
| *** CaptTofu has quit IRC | 21:46 | |
| *** dstanek has quit IRC | 21:46 | |
| *** branen has quit IRC | 21:47 | |
| clarkb | fungi: jeblair https://review.openstack.org/#/c/81841/ if we can get that through I think we will be ready to nodepool fedora | 21:48 |
| *** MarkAtwood has joined #openstack-infra | 21:48 | |
| JayF | Is there anything else I need to do for https://review.openstack.org/#/c/79088/ to ensure it's ready for 'new project Friday'? | 21:50 |
| jeblair | clarkb: lgtm | 21:50 |
| JayF | It's already been reviewed by two core reviewers, just wanted to ensure everything is in line so we don't have to wait another week | 21:50 |
| clarkb | JayF: you should be set unless someone else catches something between now and tomorrow | 21:51 |
| clarkb | usually with last minute things like that we will just update it for you | 21:51 |
| JayF | Cool. I'll be in IRC all day tomorrow (working day PST, but my bouncer is always on) and will gladly assist with anything as needed. | 21:51 |
| clarkb | JayF: sounds good | 21:51 |
| *** gokrokve has quit IRC | 21:52 | |
| *** gokrokve has joined #openstack-infra | 21:52 | |
| *** caleb_ has joined #openstack-infra | 21:52 | |
| *** signed8bit has quit IRC | 21:53 | |
| *** jhesketh_ has joined #openstack-infra | 21:53 | |
| *** mriedem has quit IRC | 21:53 | |
| clarkb | jeblair: thank you | 21:53 |
| stevebaker | jeblair: hey, any chance for some love on https://review.openstack.org/#/c/81375/ ? | 21:53 |
| jhesketh_ | Morning | 21:53 |
| clarkb | jhesketh_: hi there | 21:54 |
| *** sandywalsh has quit IRC | 21:54 | |
| *** zhiyan is now known as zhiyan_ | 21:55 | |
| *** gokrokve has quit IRC | 21:56 | |
| jhesketh_ | clarkb: since you reviewed my first javascript patch, could I get your opinion on my second please: https://review.openstack.org/#/c/81717/ | 21:56 |
| clarkb | jhesketh_: uh sure. I really don't know what I am doing there though | 21:57 |
| jhesketh_ | sure, no worries | 21:57 |
| jhesketh_ | I'm mostly looking for clarification on if this is something I should be doing | 21:57 |
| *** dkranz has joined #openstack-infra | 21:57 | |
| jhesketh_ | jeblair: your opinion on it might be help too if you have time please | 21:57 |
| jhesketh_ | I'm working towards getting zuul's app.js to feature parity with infra's status page so I can remove the duplication | 21:58 |
| *** jgrimm has quit IRC | 21:59 | |
| clarkb | jhesketh_: what is the significance of vars with $ in their names? | 22:00 |
| clarkb | arethey things associated with the jquery $? | 22:00 |
| jhesketh_ | clarkb: correct | 22:00 |
| jhesketh_ | denotes a jquery object | 22:00 |
| *** caleb_ has quit IRC | 22:00 | |
| *** CaptTofu has joined #openstack-infra | 22:02 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints https://review.openstack.org/79757 | 22:03 |
| *** masayuki_ has joined #openstack-infra | 22:05 | |
| *** yjiang5 has quit IRC | 22:05 | |
| *** mikeburns is now known as mikeburns_afk | 22:07 | |
| *** mikeburns_afk is now known as mikeburns | 22:07 | |
| *** marun has joined #openstack-infra | 22:07 | |
| *** mikeburns is now known as mikeburns_afk | 22:07 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints https://review.openstack.org/79757 | 22:07 |
| *** atiwari has quit IRC | 22:08 | |
| *** ociuhandu has quit IRC | 22:08 | |
| *** mikeburns_afk has quit IRC | 22:09 | |
| clarkb | http://logs.openstack.org/73/81873/1/check/gate-config-puppet-syntax/32394d9/console.html huh | 22:09 |
| clarkb | apparently git doesn't like that file either. it is the killer of processes. poor vim | 22:10 |
| mattoliverau | Morning all | 22:12 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Remove Branch and Milestone legacy tables https://review.openstack.org/81562 | 22:13 |
| *** mburned has quit IRC | 22:13 | |
| clarkb | fungi: nibalizer where did we end up on puppetboard? /me tries to be useful with that again | 22:14 |
| fungi | JayF: also, you were saying in #-dev the other day you wanted to help with openpgp key signing arrangements at the summit, right? if so, we'll work out some details with ttx next week now that he's had some time to settle in and revover from vacation ;) | 22:14 |
| fungi | clarkb: i think we need to open 80 (and maybe close 8081) in the firewall and add a vhost definition (probably setting the docroot to the path which the apache conf snipped for puppetboard it handing off to the wsgi interface) | 22:15 |
| fungi | s/snipped/snippet/ | 22:15 |
| clarkb | fungi: does that mean puppetboard is running? I thought it came with a vhost definition | 22:16 |
| clarkb | or is that vhost incomplete causing it to not work? | 22:16 |
| fungi | clarkb: it's running on puppetdb and i opened 80 manually in the fw to test, but apache is returning a 404 | 22:16 |
| fungi | and i didn't have time to take it further | 22:16 |
| clarkb | roger | 22:17 |
| clarkb | huh no vhost /me looks at the change again | 22:18 |
| *** rcleere has quit IRC | 22:18 | |
| *** hashar has quit IRC | 22:18 | |
| nibalizer | clarkb: hi im at work, so can't be very useful right now, sorry | 22:19 |
| fungi | nibalizer: you're fine, get back to $dayjob ;) | 22:19 |
| clarkb | nibalizer: its fine /me should learn how some of this works too | 22:19 |
| JayF | fungi: absolutely! | 22:19 |
| *** caleb_ has joined #openstack-infra | 22:19 | |
| *** khyati has quit IRC | 22:20 | |
| *** Sukhdev has joined #openstack-infra | 22:20 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Use correct path to puppetboard vhost. https://review.openstack.org/81929 | 22:20 |
| clarkb | fungi: nibalizer ^ that may help | 22:21 |
| *** khyati has joined #openstack-infra | 22:21 | |
| *** bhuvan has quit IRC | 22:21 | |
| Sukhdev | Folks can I seek help in relaunching jenkins for this https://review.openstack.org/#/c/66921/ | 22:21 |
| *** rpodolyaka1 has joined #openstack-infra | 22:21 | |
| clarkb | Sukhdev: sure, you need to reverify with a bug number | 22:22 |
| fungi | JayF: great. i put a bug in his ear about it at the foundation staff meeting and aain in one of our vmt round-ups, so i'll loop you in when he's available to think about scheduling | 22:22 |
| clarkb | Sukhdev: you will need to figure out why http://logs.openstack.org/21/66921/14/gate/gate-tempest-dsvm-full/2744788/ failed then reverify with that bug number | 22:22 |
| *** thomasem has joined #openstack-infra | 22:22 | |
| *** caleb_` has joined #openstack-infra | 22:22 | |
| *** azneita has joined #openstack-infra | 22:22 | |
| *** gokrokve has joined #openstack-infra | 22:23 | |
| *** caleb_` has quit IRC | 22:23 | |
| Sukhdev | clarkb: it is failing for non related test failures. I normally do recheck no bug. But, I thought for the merge, I have to reverify - But, I can not see a bug number to supply | 22:23 |
| *** bhuvan has joined #openstack-infra | 22:23 | |
| clarkb | Sukhdev: you need to determine the bug that caused it to fail | 22:24 |
| clarkb | Sukhdev: http://status.openstack.org/rechecks/ is a good place to look for current bugs | 22:24 |
| fungi | Sukhdev: the test which failed was tempest.api.compute.v3.servers.test_server_actions.ServerActionsV3Test.test_create_backup if that helps | 22:25 |
| fungi | (according to the console log) | 22:25 |
| *** mburned_out has joined #openstack-infra | 22:25 | |
| *** mburned_out is now known as mburned | 22:25 | |
| *** rpodolyaka1 has quit IRC | 22:25 | |
| *** caleb_ has quit IRC | 22:25 | |
| Sukhdev | clarkb, fungi: thanks let me look. | 22:25 |
| fungi | Sukhdev: i see some potentially relevant tracebacks in http://logs.openstack.org/21/66921/14/gate/gate-tempest-dsvm-full/2744788/logs/screen-n-api.txt.gz?level=TRACE | 22:26 |
| *** nati_ueno has joined #openstack-infra | 22:26 | |
| *** ociuhandu has joined #openstack-infra | 22:27 | |
| fungi | Sukhdev: also http://logs.openstack.org/21/66921/14/gate/gate-tempest-dsvm-full/2744788/logs/screen-n-cpu.txt.gz?level=TRACE | 22:27 |
| fungi | "ERROR nova.virt.libvirt.driver [req-7bfc0ecc-ea99-466d-9b9e-b39f8669baa4 ServerActionsV3Test-874187406 ServerActionsV3Test-536249989] An error occurred while trying to launch a defined domain with xml..." | 22:28 |
| clarkb | after reading the latest at https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror I don't expect a fix from upstream that won't require us to change all tox.ini files | 22:28 |
| fungi | i am disappoint | 22:28 |
| *** dizquierdo has joined #openstack-infra | 22:29 | |
| clarkb | perhaps we should consider doing that to work around the problem today then. We can go back to doing the small bash script inline in tox.ini | 22:29 |
| fungi | i was hoping there would be new noise on that in the past 9 days | 22:29 |
| Sukhdev | fungi: the dilemma is that this failed test (and traces) are not related or relevent to my patch. | 22:29 |
| clarkb | fungi: well consensus seems to be they need a new {posargs-strings} thing | 22:29 |
| clarkb | because var substitutions shouldn't be string substitutions but argv substitutions | 22:29 |
| clarkb | so the string substitution method should be special cased | 22:30 |
| clarkb | but that would require us to update the tox.ini files | 22:30 |
| fungi | Sukhdev: right, instead it means that openstack broke in subtle ways when your change was being tested. if you want openstack to be less broken, it helps us all to track down some details on what broke and maybe why, so we can prioritize what need the most attention on fixing | 22:30 |
| *** jergerber has quit IRC | 22:30 | |
| clarkb | which means there is less reason to not do that now | 22:30 |
| *** krotscheck has quit IRC | 22:30 | |
| Sukhdev | fungi: I see - good point. Let me do some digging | 22:31 |
| *** thuc_ has joined #openstack-infra | 22:31 | |
| *** krotscheck has joined #openstack-infra | 22:31 | |
| clarkb | jeblair: fungi: mordred SergeyLukjanov et al I can write the little bash script thing and propose it to zuul for example and we can see how horrible it looks | 22:32 |
| fungi | clarkb: sure, if we gotta, we gotta | 22:33 |
| clarkb | well we don't gotta do the nasty bash thing | 22:33 |
| clarkb | we do gotta update tox.ini at some point | 22:33 |
| clarkb | worst thing about updating tox.ini like that is it makes it not backward compatible even more | 22:33 |
| clarkb | ugh | 22:33 |
| jogo | is bug 1272417 still considered incomplete? | 22:33 |
| uvirtbot | Launchpad bug 1272417 in openstack-ci "Unable to install software dependencies due to: "Cannot fetch index base URL"" [Medium,Incomplete] https://launchpad.net/bugs/1272417 | 22:34 |
| *** thuc has quit IRC | 22:34 | |
| *** dizquierdo has quit IRC | 22:35 | |
| *** khyati has quit IRC | 22:35 | |
| *** thuc_ has quit IRC | 22:35 | |
| fungi | jogo: it's incomplete insofar as every time it's come up so far we've been unable to find matching requests in the server log, so the alternative is that we mark it "wont fix" and assume it's internet packet loss or work on ways to set up a local pypi mirror on every slave filesystem | 22:36 |
| *** thuc has joined #openstack-infra | 22:36 | |
| clarkb | fungi: I marked it traiged so that the in progress patch(es) don't end up applying to a bug that gets killed on us | 22:36 |
| clarkb | we can always edit that status as necessary if the patch(es) don't help | 22:36 |
| *** adalbas has quit IRC | 22:37 | |
| fungi | clarkb: oh, we have in progress patches? | 22:37 |
| clarkb | fungi: yeah derekh is proposing changes to increase the pip timeout | 22:38 |
| *** david-lyle has joined #openstack-infra | 22:38 | |
| fungi | so we do | 22:38 |
| fungi | ahh, right, that | 22:38 |
| clarkb | I think we only need one change and have -1'd them with suggestions | 22:38 |
| fungi | awesome. i missed that turn of events | 22:38 |
| *** wenlock has quit IRC | 22:39 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints https://review.openstack.org/79757 | 22:39 |
| *** ociuhandu has quit IRC | 22:39 | |
| *** e0ne has joined #openstack-infra | 22:39 | |
| *** cyeoh is now known as cyeoh_away | 22:39 | |
| fungi | worth consideration... our entire pypi mirror (including plenty of cruft) is a mere 1.6g on disk | 22:39 |
| *** maxbit has quit IRC | 22:40 | |
| fungi | too bad hpcloud thinks 10g of root filesystem is more than you should ever want | 22:40 |
| sdague | 64k is enough for anyone | 22:41 |
| *** e0ne has quit IRC | 22:42 | |
| jogo | sdague: haha | 22:43 |
| fungi | that was 640k if you're talking about the supposed bill gates quote, but i did have a computer with only 64k early on | 22:43 |
| jesusaurus | i think 10g roots are reasonable, if you are running a service that needs more than that you should be attaching a volume | 22:43 |
| jesusaurus | i thought it was 680k | 22:43 |
| fungi | no. no it was not 680k ;) | 22:43 |
| sdague | 68020? | 22:45 |
| clarkb | jesusaurus: that is true of long lived things not true of jenkins slaves | 22:45 |
| clarkb | jesusaurus: so we end up going through the same hoops for a thing that will live 45 minutes | 22:45 |
| *** thuc has quit IRC | 22:45 | |
| jesusaurus | ahh. ugh. | 22:45 |
| *** thuc has joined #openstack-infra | 22:46 | |
| clarkb | it does make sense from the hosting side though especially for image snapshots | 22:46 |
| clarkb | 10GB max size for snapshots -> win | 22:46 |
| *** krotscheck has quit IRC | 22:46 | |
| fungi | jesusaurus: to speed up testing, we want to make any resources which the test might need (and which we reasonably can) available on the local filesystem so that jobs complete quickly with the reduced latency/increased bandwidth and also so as not to subject us to random failures from network connectivity problems | 22:47 |
| *** krotscheck has joined #openstack-infra | 22:47 | |
| jesusaurus | fungi: oh, in that case what we really want are read-only volumes that can be attached to multiple hosts simultaneously... | 22:47 |
| *** changbl has quit IRC | 22:47 | |
| * jesusaurus hopes | 22:48 | |
| apevec | ttx, adam_g, fungi - upload_release.py renamed quantum-2013.1.5.tar.gz https://launchpad.net/neutron/grizzly/2013.1.5/+download/neutron-2013.1.5.tar.gz/+md5 | 22:48 |
| clarkb | jesusaurus: sort of, it would be cool if they were copy on write | 22:48 |
| clarkb | jesusaurus: because we need a place to do writes too | 22:48 |
| mattoliverau | Or used unionfs or something | 22:48 |
| apevec | how can I fix that manually in LP? | 22:49 |
| adam_g | apevec, i think this happened to me once | 22:49 |
| adam_g | apevec, i believe you can delete the tarball @ https://launchpad.net/neutron/grizzly/2013.1.5 | 22:49 |
| *** khyati has joined #openstack-infra | 22:49 | |
| jeblair | clarkb, jhesketh_: reading scrollback | 22:49 |
| openstackgerrit | A change was merged to openstack-infra/config: Always create .venv for upstream_translation_horizon https://review.openstack.org/81902 | 22:50 |
| adam_g | apevec, then re-upload by hand or munge the script a bit to upload it as quantum-* | 22:50 |
| apevec | adam_g, thanks! | 22:50 |
| *** flaper87|afk is now known as flaper87 | 22:50 | |
| mattoliverau | ro volume + aufs/unionfs would work like COW, in the sense you can write to the filesystem on watch node, but not effect other nodes. | 22:50 |
| mattoliverau | s/watch/each/ | 22:50 |
| apevec | adam_g, btw, you're available for next stable/havana 2013.2.3 ? | 22:50 |
| mattoliverau | Sorry, coming in on the end of a conversation :P I'll get back to reading scroll back :) | 22:51 |
| adam_g | apevec, yup. should be | 22:51 |
| clarkb | mattoliverau: we could just skip ahead and mount an afs filesystem on all slaves | 22:51 |
| apevec | adam_g, cool, then please take care announcing freeze next week | 22:51 |
| *** thomasem has quit IRC | 22:51 | |
| apevec | it's coming fast :) | 22:51 |
| fungi | apevec: so today's the final grizzly release, correct? if so, i'll go ahead and eol the grizzly branches and start ripping out ci jobs next week | 22:52 |
| clarkb | oh man is it happening? | 22:52 |
| apevec | fungi, yes, final one! | 22:53 |
| fungi | clarkb: at the summit we actually said i-3 would be when we eol'd grizzly, but forgot at the time that we'd already set a point release schedule to do a .5 release today | 22:53 |
| fungi | apevec: awesome--thanks! | 22:54 |
| mattoliverau | clarkb: I haven't used afs, but sure, as long as you don't acutally writeback to the server updating all node's filesystems :) | 22:55 |
| fungi | mattoliverau: afs solves every problem ever, haven't you heard? | 22:55 |
| clarkb | mattoliverau: mordred and jeblair apparently built an afs timemachine in NC somewhere and now they try to use it to fix all of the problems | 22:55 |
| mattoliverau | lol | 22:55 |
| *** alex-gone is now known as _alexandra_ | 22:56 | |
| fungi | i used afs plenty in my salad days, but never had the joy of running a node | 22:56 |
| jeblair | fungi, clarkb: indeed! in this case, use afs, warm the cache, then set it to offline mode, and you've got copy on write! | 22:56 |
| mattoliverau | I see, so afs is the global and definitive answer to all things.. got it :) | 22:56 |
| jeblair | but actually for this case, i think just a simple read-only volume would be what we'd want for the pypi mirror | 22:56 |
| jeblair | why did we want cow? | 22:57 |
| fungi | probably just for scratch space. not really necessary i don't thinkl | 22:57 |
| clarkb | jeblair: I was thinking additional stuff like git repos | 22:57 |
| clarkb | if we are going to cache one thing we should cache all the things | 22:57 |
| fungi | clarkb: we can still clone from the ro volume in that case and write in the clone | 22:57 |
| clarkb | fungi: right but then you hit the 10G problem | 22:57 |
| fungi | s/640k/10g/ right | 22:58 |
| *** signed8bit has joined #openstack-infra | 22:58 | |
| *** bhuvan has quit IRC | 22:58 | |
| jeblair | also, ultimately, this is not that different than using a pypi mirror, except you're counting on afs retrying lost udp packets instead of pip retrying lost tcp connections | 22:58 |
| fungi | i don't think we're actually hitting the 10g ceiling though with the current workarounds. it's been suggested that jenkins mistakes other sorts of failures for a full filesystem | 22:59 |
| jeblair | oh, except you could pre-warm the afs cache on the image | 22:59 |
| jeblair | anyway | 22:59 |
| jeblair | jhesketh_: yeah, getting the app.js stuff so we can use it is good, i'll try to review in depth soon | 22:59 |
| jhesketh_ | jeblair: so that's kinda the next step.. the intermediate step that I've taken was to make app.js more jquery like. Then I realised that might not be something we want to do for performance reasons (see my inline comment on that review) | 23:00 |
| *** gokrokve has quit IRC | 23:00 | |
| jeblair | jhesketh_: oh interesting. ok. | 23:00 |
| jhesketh_ | but otherwise I hope to work on bringing app.js up to feature-parity with infra's status | 23:00 |
| *** gokrokve has joined #openstack-infra | 23:01 | |
| fungi | clarkb: hp support just got back to you/us on the quota bump! | 23:01 |
| jeblair | jhesketh_: yeah, mainly we need to be able to customize the look a bit, which is mostly just header/footer, so really just having a page that has a div in the middle, and then loading app.js and saying "take control of that div" is probably fine | 23:01 |
| jesusaurus | clarkb: but if you have your git repos in a ro volume then unionfs on top, then you can write to them and only pay for disk for changes | 23:01 |
| fungi | "The Instance, RAM, and Floating IPs quotas...have been increased to 600, 18,000GB, and 600 respectively in US East v13.5..." | 23:01 |
| clarkb | jesusaurus: ya there are definitely smart ways to deal with this | 23:02 |
| *** khyati has quit IRC | 23:02 | |
| *** masayuki_ is now known as masayukig | 23:02 | |
| clarkb | fungi: I think that happened this morning | 23:02 |
| clarkb | jeblair has a few patches up to migrate us | 23:02 |
| jeblair | fungi: long time ago so i wrote this series: https://review.openstack.org/#/c/81873/ | 23:02 |
| fungi | well, "just" for me is between when i found time to read e-mail yesterday and when i started reading it again just now ;) | 23:02 |
| jeblair | jhesketh_: we do have those sparklines in there, and i don't know what to do with that, but we should be able to come up with something | 23:03 |
| jeblair | jhesketh_: (i don't think graphite should be required for all users) | 23:03 |
| clarkb | jeblair: did you see my comments about tox in sb too? | 23:03 |
| clarkb | jeblair: starts around 2228UTC | 23:03 |
| jhesketh_ | jeblair: app.js doesn't have any of the train-tracks or expand/collapse features either | 23:04 |
| *** apevec has quit IRC | 23:04 | |
| jeblair | clarkb: yeah... is it worth waiting to see if they go with the posargs-string? | 23:04 |
| jeblair | clarkb: so maybe we avoid updating all tox.ini files twice? | 23:04 |
| clarkb | jeblair: maybe. My gripe with posargs-string is it isn't backward compat to 1.6 | 23:05 |
| clarkb | jeblair: we can bump the min version in tox.ini though | 23:05 |
| openstackgerrit | A change was merged to openstack-infra/config: Open ports 8003, 8004 for heat API calls from compute https://review.openstack.org/81375 | 23:05 |
| *** gokrokve has quit IRC | 23:05 | |
| jeblair | jhesketh_: yeah, all that should go in app.js. i think the only thing we do in the middle of the page (between the header and graphs at the bottom) that shouldn't go in app.js is the sparklines (at least, they shouldn't be required) | 23:05 |
| *** jnoller has quit IRC | 23:05 | |
| jhesketh_ | sure, I haven't looked at how they are filled | 23:06 |
| jhesketh_ | it might not hurt for app.js to support them if the data exists | 23:06 |
| jeblair | jhesketh_: you'll love it. :) | 23:06 |
| *** salv-orlando has quit IRC | 23:06 | |
| jhesketh_ | heh, I'm not that great at javascript so I'm not sure why I'm doing this :-P | 23:06 |
| jeblair | jhesketh_: yeah, we could add a graphite host parameter | 23:06 |
| jhesketh_ | I just want to overlay zuul results in gerrit because I think it'll be cool | 23:07 |
| *** salv-orlando has joined #openstack-infra | 23:07 | |
| *** flaper87 is now known as flaper87|afk | 23:07 | |
| jeblair | jhesketh_: and if it's there, do sparklines, and skip if not | 23:07 |
| *** bhuvan has joined #openstack-infra | 23:07 | |
| jhesketh_ | yep, that was what I was thinking | 23:07 |
| *** zzelle has joined #openstack-infra | 23:08 | |
| *** jamielennox|away is now known as jamielennox | 23:08 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add SubNodes and the ability to delete them https://review.openstack.org/81938 | 23:09 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add the ability to create subnodes https://review.openstack.org/81939 | 23:09 |
| jeblair | that's 2 out of 3 steps needed to add multi-node support to nodepool | 23:09 |
| clarkb | jeblair: I really think we should use heat for that but have no heat yet... | 23:10 |
| clarkb | jeblair: in the subnode nodepool stuff, how would things like vpn between nodes work? (assuming we need a vpn between the nodes) | 23:11 |
| jeblair | clarkb: yes. there are lots of great ways that problem could be solved. the solution i'm working on has the notable advantage of being possibly and relatively simple with our current tools. | 23:11 |
| clarkb | jeblair: agreed | 23:11 |
| jeblair | clarkb: so, if our providers support neutron (they don't) we could hook it up that way | 23:11 |
| clarkb | jeblair: region-b does but not rax | 23:11 |
| jeblair | clarkb: otherwise, we could set up an openvpn between them | 23:11 |
| *** ociuhandu has joined #openstack-infra | 23:12 | |
| clarkb | support neutron | 23:12 |
| jeblair | clarkb: or, maybe for some things we don't need a vpn and could just open the public ips to each other | 23:12 |
| jeblair | clarkb: step 3 of 3 (not yet complete) is to add a script point where nodepool ssh's into each of the nodes and does what needs to be done to make sure they can talk to each other | 23:12 |
| jeblair | clarkb: at a base, that will be generating an ssh key and putting it on each host | 23:13 |
| fungi | i think the bigger thing is that we need a vpn not because of security/provacy/authenticity but just because we want to be able to make layer-2 peerings between nodes in potentially dofferent broadcast domains | 23:13 |
| jeblair | clarkb: it could also do whatever other stuff | 23:13 |
| clarkb | jeblair: gotcha, that is basically where I figured heat would be helpful | 23:13 |
| clarkb | fungi: right, for neutron to do fun things :) | 23:13 |
| fungi | my i and o keys are apparently tii clise to ine anither these days | 23:14 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Delete created keypairs if nova boot fails https://review.openstack.org/81647 | 23:14 |
| jeblair | this is interesting: http://docs.rackspace.com/servers/api/v2/cn-devguide/content/index.html | 23:15 |
| *** bknudson has quit IRC | 23:16 | |
| jeblair | it says "OpenStack Networking API v2" | 23:16 |
| jeblair | but that does not look like the neutron api | 23:16 |
| jeblair | compare http://docs.rackspace.com/servers/api/v2/cn-devguide/content/list_networks.html | 23:16 |
| jeblair | with http://docs.openstack.org/api/openstack-network/2.0/content/GET_os-networks-v2.0_listNetworks_v2.0_networks_networks.html | 23:16 |
| *** thedodd has quit IRC | 23:17 | |
| clarkb | two different uris | 23:17 |
| fungi | "powered by the OpenStack Networking code base" what else could it be? (i'm going to regret asking, i'm sure) | 23:17 |
| jeblair | clarkb: yep. "v2.0/networks" is neutron, "/os-networksv2" is rackspace | 23:18 |
| jeblair | what is rackspace running? | 23:18 |
| openstackgerrit | A change was merged to openstack-infra/config: Add all images and increase usage of region-b https://review.openstack.org/81873 | 23:18 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix Fedora Puppet failure to use pip https://review.openstack.org/81841 | 23:18 |
| jeblair | pvo: do you know why the rackspace cloud network api doesn't seem to be neutron? (i'm really confused) | 23:20 |
| *** khyati has joined #openstack-infra | 23:21 | |
| fungi | i thought "powered by the OpenStack Networking code base" meant i could interface with it using the real openstack networking api, but the available methods don't look at all similar | 23:21 |
| *** rpodolyaka1 has joined #openstack-infra | 23:22 | |
| *** derekh has joined #openstack-infra | 23:23 | |
| pleia2 | o/ | 23:23 |
| pleia2 | derekh: I brought up a fedora vm on tripleo cloud :) | 23:24 |
| fungi | oh, it is neutron with extensions, but their docs still say "quantum" | 23:24 |
| pleia2 | derekh: so your change should be ready to go when we have time to keep an eye on it | 23:24 |
| fungi | "Rackspace has also deployed certain extensions as permitted by the OpenStack Quantum API contract" | 23:24 |
| derekh | pleia2: sweet :-) | 23:24 |
| pleia2 | derekh: yeah! just tested with my user account for now, but infra should be fine | 23:25 |
| *** azneita has quit IRC | 23:25 | |
| clarkb | derekh: did you see my comments about the pip thing? | 23:25 |
| derekh | clarkb: yup, will fix that up in a few minutes | 23:26 |
| *** sarob has quit IRC | 23:26 | |
| *** rpodolyaka1 has quit IRC | 23:26 | |
| *** sarob has joined #openstack-infra | 23:26 | |
| jeblair | jhesketh_: do you know anyone at rax who might have an answer to our cloud networking questions? ^ | 23:27 |
| *** azneita has joined #openstack-infra | 23:27 | |
| *** pmathews has quit IRC | 23:27 | |
| jhesketh_ | jeblair: I'm not sure to be honest (I'm just checking) | 23:30 |
| jhesketh_ | I believe it's some kind of mixed model | 23:30 |
| jhesketh_ | so maybe there's a proxy layer but I'm only guessing hre | 23:31 |
| *** sarob has quit IRC | 23:31 | |
| *** david-lyle has quit IRC | 23:31 | |
| *** che-arne has quit IRC | 23:31 | |
| fungi | the wording in the rax networking api developer intro suggests to me that normal (maybe not entirely recent?) quantum/neutron methods should work too | 23:32 |
| jeblair | fungi: they don't, i've checked | 23:32 |
| fungi | poop | 23:32 |
| fungi | then the documentation is also *woefully* misleading in that regard | 23:32 |
| jeblair | fungi: where do you see that? | 23:32 |
| fungi | ""powered by the OpenStack Networking code base...deployed certain extensions as permitted by the OpenStack Quantum API contract" http://docs.rackspace.com/servers/api/v2/cn-devguide/content/index.html | 23:33 |
| jeblair | ah, that | 23:33 |
| fungi | lipservice apparently | 23:34 |
| fungi | i guess that could be interpreted as "we've got quantum running under the hood, but you can't get to its api, here use this thing instead it's just as good" | 23:34 |
| fungi | tastes like chicken | 23:35 |
| *** mrodden has quit IRC | 23:35 | |
| fungi | pyrax, network edition ;) | 23:36 |
| *** mwagner_notHere has quit IRC | 23:37 | |
| jeblair | i'm waiting for someone at rackspace to tell me what i'm missing... because if this is what it looks like, i'm going to be furious. | 23:37 |
| clarkb | jeblair: can you see my comment in https://review.openstack.org/#/c/81938/ to make sure I am understadning that | 23:43 |
| jeblair | jhesketh_, fungi: i've sent email to jnoller and pvo; i'll let you know what i hear | 23:44 |
| pvo | jeblair: because it isn't neutron yet. Change over is happening in the coming weeks. | 23:45 |
| *** _alexandra_ is now known as alex-cafehackwla | 23:45 | |
| jeblair | pvo: that's excellent news! | 23:46 |
| pvo | looking for emails. | 23:46 |
| *** dstanek has joined #openstack-infra | 23:46 | |
| pvo | jeblair: did you just send it? | 23:46 |
| jeblair | pvo: yes just | 23:46 |
| pvo | ok, I'll be patient. | 23:46 |
| jeblair | pvo: basically i was completely befuddled, but your answer explains all i need to know. :) | 23:46 |
| pvo | ok, cool. | 23:46 |
| fungi | jeblair: did you send it through our foundation e-mail service? the provider on that one has slow servers or something ;) | 23:47 |
| clarkb | does that mean the docs that say it is quantum are wrong but will be made correct? | 23:47 |
| clarkb | that doesn't explain the difference in api though | 23:47 |
| pvo | clarkb: the docs are correct as of today. | 23:47 |
| pvo | melange+quantum | 23:47 |
| clarkb | I see | 23:47 |
| jeblair | pvo: but when you switch to neutron, the real neutron api will be exposed? | 23:48 |
| fungi | pvo: want to adopt melange? free to a good home! | 23:48 |
| pvo | jeblair: yessir | 23:48 |
| pvo | fungi: ha...haha... hahaha. please lets just kill it. | 23:48 |
| pvo | its been doing better. | 23:48 |
| *** mrodden has joined #openstack-infra | 23:48 | |
| *** CaptTofu has quit IRC | 23:49 | |
| fungi | does anybody know whatever happened with burrow? we've still got a dns address record for its docs pointed at an empty vhost on the old wiki server | 23:49 |
| fungi | i'm thinking that's safe to delete | 23:50 |
| fungi | http://burrow.openstack.org/ | 23:50 |
| jeblair | fungi: yes, it was never made an official project | 23:50 |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Add neutron functional job to check (non-voting) https://review.openstack.org/81943 | 23:50 |
| jeblair | clarkb: responded | 23:50 |
| pvo | jeblair: responded. | 23:50 |
| clarkb | jeblair: thank you | 23:50 |
| marun | clarkb, fungi: …slowly moving towards having a working functional job for neutron ^ | 23:51 |
| fungi | marun: i see. it's working okay on your experimental runs i take it? | 23:51 |
| marun | fungi: yup, it checks out. | 23:51 |
| fungi | too cool | 23:51 |
| marun | fungi: it's just a subset of the unit tests for now, so getting it working with sudo was the hard part. | 23:51 |
| marun | (for some small value of hard) | 23:52 |
| jeblair | fungi, clarkb: so we should be able to plan on having neutron in both providers soon; nodepool can hook up l2 networks between nodes then if we want | 23:52 |
| pleia2 | fungi: thanks for getting to my bugdaystats change :) | 23:54 |
| *** alexpilotti has quit IRC | 23:55 | |
| *** zzelle has quit IRC | 23:56 | |
| *** nati_ueno has quit IRC | 23:56 | |
| *** nati_ueno has joined #openstack-infra | 23:57 | |
| fungi | jeblair: i think that's the right solution and worth waiting/designing for where l2 is considered | 23:57 |
| *** azneita has quit IRC | 23:57 | |
| clarkb | we may still end up seeing traffic we need getting filtered with neutron but we won't know for sure until we try | 23:58 |
| fungi | tripleo-ci already exposes neutron to us too, when it gets to the point where we want to start doing general-purpose things with it | 23:58 |
| *** mriedem has joined #openstack-infra | 23:58 | |
| *** cyeoh_away is now known as cyeoh | 23:58 | |
| fungi | clarkb: agreed, having neutron doesn't necessarily mean we can expect ip protocol 47 will pass unimpeded between providers | 23:59 |
| fungi | (for example) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!