| *** reed_ has quit IRC | 00:00 | |
| jog0 | jeblair: I don't think elastic-recheck picked up on 1224001 | 00:06 |
|---|---|---|
| jog0 | and its one a very frequent bug righ tnow | 00:07 |
| clarkb | jog0: is that the new one? | 00:10 |
| jog0 | clarkb: yeah | 00:16 |
| jog0 | I ran elasticsearch locally and it worked | 00:16 |
| jog0 | clarkb: so something is funny on the server side | 00:16 |
| clarkb | jog0 how does the bot know to reread the file? | 00:16 |
| clarkb | does it just reread each time? | 00:17 |
| *** dkliban has joined #openstack-infra | 00:17 | |
| jog0 | clarkb: yup | 00:19 |
| *** mrodden has joined #openstack-infra | 00:20 | |
| *** dcramer_ has quit IRC | 00:20 | |
| openstackgerrit | Chris Hoge proposed a change to openstack-infra/config: Adding puppet-vswitch project to Stackforge https://review.openstack.org/49020 | 00:27 |
| clarkb | jog0: puppet isn't running on the server | 00:28 |
| clarkb | jog0: I am going to run it in noop mode to see how much stuff it iwll hcnage then run it if it comes back clean | 00:28 |
| clarkb | jeblair: fungi: any opinion on whether or not puppet can be started? | 00:28 |
| fungi | clarkb: i think it should be caught up, but didn't want to start it out from under jeblair while he might still be looking at it | 00:30 |
| jog0 | clarkb: cool thanks | 00:30 |
| fungi | last i looked he was still tailing logs | 00:30 |
| jog0 | is it possible to manually update the git repo | 00:30 |
| clarkb | fungi: I see that --noop shows log configs will be changed | 00:31 |
| clarkb | fungi: from INFO to DEBUG | 00:31 |
| clarkb | jog0: it is, I will do that instead | 00:31 |
| jog0 | actually with EOB on west coast shouldn't matter to much if we wait | 00:31 |
| openstackgerrit | Chris Hoge proposed a change to openstack-infra/config: Adding puppet-vswitch project to Stackforge https://review.openstack.org/49020 | 00:31 |
| jog0 | whoa gate is 49 haha | 00:31 |
| jeblair | clarkb, fungi: i'm clear for you to start puppet | 00:31 |
| fungi | clarkb: ahh, yeah that patch merged since my last test apply | 00:31 |
| fungi | sounds fine to restart to me then | 00:32 |
| clarkb | jeblair: ok thanks, I will restart it | 00:32 |
| clarkb | I also manually updated the repo | 00:32 |
| jeblair | cool | 00:32 |
| clarkb | jog0: should work now | 00:33 |
| pabelanger | Gents, do you mind giving a once over to https://review.openstack.org/49020 when you have a moment? It is a stackforge import of a puppet module. Thanks in advance | 00:33 |
| pabelanger | and gals too | 00:33 |
| jog0 | clarkb: thanks | 00:34 |
| *** sarob has quit IRC | 00:34 | |
| *** sarob has joined #openstack-infra | 00:35 | |
| clarkb | jeblair: we seem to be spending a lot more time with nodepool VMs in deleting state recently. is that all rackspace? | 00:35 |
| jeblair | clarkb: primarily gate flapping, secondarily nodepool has a bug in the cleanup cron so if it hits an error deleting, they can stay around for days | 00:37 |
| jeblair | clarkb: i've been running "for x in `nodepool list |grep "delete | [2-9][0-9]\\."|awk '{print $2}'`; do echo $x; nodepool delete $x; done" about once a day to keep up with that until i fix it. | 00:38 |
| *** davidhadas has quit IRC | 00:38 | |
| jeblair | (or some variant) | 00:38 |
| *** sarob has quit IRC | 00:39 | |
| *** davidhadas has joined #openstack-infra | 00:39 | |
| *** hogepodge_ has quit IRC | 00:42 | |
| *** nosnos has joined #openstack-infra | 00:50 | |
| dstufft | Was there any PyPI timeouts today? | 00:53 |
| jeblair | dstufft: https://jenkins02.openstack.org/job/gate-requirements-install/259/console is the only one i see with a quick glance | 00:55 |
| jeblair | dstufft: https://jenkins01.openstack.org/job/gate-requirements-install/ and https://jenkins02.openstack.org/job/gate-requirements-install/ being the best places to look for that | 00:55 |
| dstufft | jeblair: you might want to try bumping the pip timeout btw | 00:56 |
| dstufft | by default it's 15s | 00:56 |
| *** anteaya has quit IRC | 00:56 | |
| jeblair | dstufft: where's that set? | 00:57 |
| jeblair | dstufft: (and that affects "SSLError: The read operation timed out" ?) | 00:57 |
| dstufft | jeblair: same as any pip variable, --timeout on the cli, PIP_TIMEOUT =, or ~/.pip/pip.conf timeout = | 00:57 |
| dstufft | jeblair: I don't know for sure, it's an idea | 00:58 |
| dstufft | jeblair: i'm still trying to trace the problem that's affecting thrift | 00:59 |
| dstufft | and a few others | 00:59 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Set pip timeout to 60s when using pypi https://review.openstack.org/49093 | 01:00 |
| jeblair | dstufft: ^ | 01:00 |
| dstufft | jeblair: cool, hopefully it helps. I'm still gonna keep poking at it though incase | 01:01 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Set pip timeout to 60s when using pypi https://review.openstack.org/49093 | 01:01 |
| *** rockyg has quit IRC | 01:01 | |
| *** amotoki has joined #openstack-infra | 01:03 | |
| *** emagana has quit IRC | 01:04 | |
| *** thingee is now known as thingee_zzz | 01:05 | |
| *** sarob has joined #openstack-infra | 01:05 | |
| *** sarob has quit IRC | 01:10 | |
| *** changbl has joined #openstack-infra | 01:12 | |
| *** MarkAtwood has quit IRC | 01:13 | |
| *** sarob has joined #openstack-infra | 01:13 | |
| *** ryanpetrello has quit IRC | 01:14 | |
| mordred | dstufft, jeblair: I am in favor of your products. can you send me some brochures? | 01:16 |
| mordred | jeblair: btw - while your head is in that space | 01:16 |
| mordred | jeblair: there is another change related to consuming select_mirror.sh in d-g | 01:17 |
| *** sarob has quit IRC | 01:18 | |
| *** ryanpetrello has joined #openstack-infra | 01:20 | |
| *** UtahDave has quit IRC | 01:20 | |
| *** MarkAtwood has joined #openstack-infra | 01:20 | |
| *** esker has joined #openstack-infra | 01:21 | |
| *** dkliban has quit IRC | 01:22 | |
| *** nati_ueno has quit IRC | 01:22 | |
| *** dguitarbite has joined #openstack-infra | 01:28 | |
| *** melwitt1 has quit IRC | 01:30 | |
| *** boris-42 has quit IRC | 01:33 | |
| *** ryanpetrello has quit IRC | 01:39 | |
| *** ryanpetrello has joined #openstack-infra | 01:42 | |
| *** MarkAtwood has quit IRC | 01:46 | |
| Alex_Gaynor | So something seems to be broken on the python2.6 builders | 01:48 |
| Alex_Gaynor | If you look at teh gate pipeline, they're all failing | 01:49 |
| *** tizzo has quit IRC | 01:49 | |
| Alex_Gaynor | mordred, clarkb, fungi: ^ | 01:50 |
| Alex_Gaynor | Also, zuul doesn't seem to respond super well to this many failures, 1285 results in teh queue right now | 01:50 |
| *** rockyg has joined #openstack-infra | 01:54 | |
| *** ryanpetrello has quit IRC | 01:56 | |
| *** sarob has joined #openstack-infra | 02:06 | |
| *** UtahDave has joined #openstack-infra | 02:10 | |
| *** sarob has quit IRC | 02:10 | |
| *** mrda has quit IRC | 02:15 | |
| dims | Anyone know if zuul "stuck"? 5 reviews stuck in "check" though all the jenkins jobs have finished | 02:20 |
| *** sarob has joined #openstack-infra | 02:26 | |
| *** sarob has quit IRC | 02:31 | |
| *** AlexF has joined #openstack-infra | 02:31 | |
| *** schwicht has quit IRC | 02:31 | |
| *** jerryz has quit IRC | 02:34 | |
| *** dims has quit IRC | 02:40 | |
| jgriffith | new one I think: https://jenkins02.openstack.org/job/gate-cinder-python26/1591/console | 02:40 |
| *** dguitarbite has quit IRC | 02:40 | |
| jgriffith | So just curious, is there any sort of plan for getting the gate issues resolved at this point? | 02:42 |
| jgriffith | ie do I still need to try and keep +A frozen? | 02:42 |
| *** sarob has joined #openstack-infra | 02:44 | |
| *** dcramer_ has joined #openstack-infra | 02:47 | |
| *** sarob has quit IRC | 02:49 | |
| *** dolphm has joined #openstack-infra | 02:55 | |
| *** AlexF has quit IRC | 03:00 | |
| dolphm | is something dead? "reverify" doesn't seem to be doing anything | 03:00 |
| dolphm | and a couple changes failed with this in the past hour http://logs.openstack.org/13/48413/2/gate/gate-horizon-python26/f79f074/console.html | 03:00 |
| *** dguitarbite has joined #openstack-infra | 03:01 | |
| markmcclain | looks like there's a problem with py26 tests | 03:05 |
| markmcclain | many of them seem to be failing in the gate | 03:05 |
| * mordred looking | 03:07 | |
| *** rcleere has joined #openstack-infra | 03:08 | |
| *** yaguang has joined #openstack-infra | 03:08 | |
| mordred | dolphm, Alex_Gaynor, jgriffith, markmcclain - it looks like it may be moving again? | 03:09 |
| dolphm | mordred: i can't get https://review.openstack.org/#/c/46193/ gating again | 03:09 |
| mordred | oh - wait | 03:09 |
| mordred | it looks like it may be slaves off of jenkins02 | 03:10 |
| mordred | clarkb, fungi: you guys around? | 03:10 |
| mordred | I think it may have all been one slave | 03:11 |
| mordred | that was unhappy | 03:11 |
| mordred | but was failing quickly | 03:11 |
| dolphm | so it probably chewed through a lot of jobs? | 03:12 |
| mordred | so consuming a LOT of the jobs and failing them | 03:13 |
| jgriffith | :( | 03:13 |
| jgriffith | long day for sir Jenkins | 03:14 |
| mordred | yah | 03:14 |
| mordred | srrsly | 03:14 |
| mordred | I gotta tell you - he gets sad with the resets | 03:14 |
| jgriffith | mordred: perhaps he's going on strike | 03:15 |
| jgriffith | jenkins says: "I've had enough of this abuse...." | 03:15 |
| *** che-arne has quit IRC | 03:15 | |
| mordred | jgriffith: I don't blame him | 03:18 |
| jgriffith | mordred: me neither... I'd suggest a night off and I'll buy him a drink | 03:19 |
| * hub_cap sneaks up to the bar | 03:19 | |
| * mordred throws a goat at hub_cap | 03:19 | |
| * hub_cap flogs mordred with a blotched axe | 03:19 | |
| hub_cap | i have /slap too ;) | 03:19 |
| *** DennyZhang has joined #openstack-infra | 03:20 | |
| *** sandywalsh has quit IRC | 03:23 | |
| *** senk has joined #openstack-infra | 03:24 | |
| *** yaguang has quit IRC | 03:25 | |
| *** basha has joined #openstack-infra | 03:26 | |
| dolphm | is new stuff getting into the gate queue? | 03:27 |
| *** dguitarbite has quit IRC | 03:29 | |
| jeblair | Alex_Gaynor: the result queue doesn't mean very much these days; it should be removed soon; it's little more than an indication that zuul should run the queue processor, so it could really be a flag. at any rate, as long as there is work to do, it will do it, and once things settle down, it burns through the 'queue' very quickly. | 03:38 |
| dolphm | (disregard my last comment, the delay before starting jobs just seems longer than i'm used to) | 03:39 |
| *** bnemec has quit IRC | 03:39 | |
| jeblair | mordred: which slave? | 03:41 |
| dolphm | jeblair: example- http://logs.openstack.org/93/46193/19/gate/gate-keystone-python26/3f95a7f/console.html | 03:42 |
| markmcclain | Is there an easy to express a revert through? | 03:42 |
| *** jhesketh_ has joined #openstack-infra | 03:42 | |
| *** bnemec has joined #openstack-infra | 03:43 | |
| markmcclain | reverting should stop this https://bugs.launchpad.net/neutron/+bug/1224001 | 03:43 |
| uvirtbot | Launchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [Critical,In progress] | 03:43 |
| jeblair | markmcclain: no, it's on our wishlist | 03:43 |
| *** senk has quit IRC | 03:46 | |
| * markmcclain loves logstash right now | 03:49 | |
| *** sandywalsh has joined #openstack-infra | 03:49 | |
| jeblair | markmcclain: have you seen the bot in #openstack-qa? the elastic-recheck script is announcing both failures that it classifies as well as unknown ones | 03:50 |
| markmcclain | no.. I forgot to add that room to my auto join list | 03:51 |
| jeblair | markmcclain: http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2013-10-01.log | 03:51 |
| markmcclain | jeblair: I think my favorite thing about logstash is that it made finding the problem commit super easy | 03:52 |
| jeblair | markmcclain: ++ | 03:54 |
| jeblair | clarkb: ^ fan mail! | 03:54 |
| *** AlexF has joined #openstack-infra | 03:55 | |
| clarkb | woot | 03:56 |
| *** esker has quit IRC | 03:58 | |
| *** esker has joined #openstack-infra | 04:04 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files https://review.openstack.org/48783 | 04:07 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Provide default ConfigParser object https://review.openstack.org/48790 | 04:07 |
| *** DennyZhang has quit IRC | 04:08 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Add repo scm https://review.openstack.org/45165 | 04:17 |
| *** AlexF has quit IRC | 04:22 | |
| *** AlexF has joined #openstack-infra | 04:31 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Raise exceptions when no entrypoint, macro or template found https://review.openstack.org/49102 | 04:36 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Raise exceptions when no entrypoint, macro or template found https://review.openstack.org/49102 | 04:37 |
| *** Ryan_Lane has quit IRC | 04:38 | |
| *** darraghb has joined #openstack-infra | 04:38 | |
| *** crank has quit IRC | 04:41 | |
| *** tvb has joined #openstack-infra | 04:46 | |
| *** senk has joined #openstack-infra | 04:47 | |
| mordred | jeblair: it was centos6-10 | 04:49 |
| darraghb | query on https://review.openstack.org/#/c/45165/, should the JJB code default to generating all XML elements even if the default is blank/no-value, or should it default to the plugin behaviour on which XML tags to set? | 04:50 |
| *** senk has quit IRC | 04:51 | |
| *** esker has quit IRC | 04:53 | |
| *** tvb has quit IRC | 04:58 | |
| mordred | darraghb: there was a discussion around that a little while ago and I don't remember the outcome | 04:59 |
| mordred | I tihnk mgagne and zaro were talking about it maybe? | 04:59 |
| darraghb | mordred: any irc logs kept online to search? | 05:02 |
| mordred | darraghb: yup! | 05:03 |
| darraghb | nvrmind found them I think: 'eavesdrop' | 05:03 |
| mordred | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/ | 05:03 |
| darraghb | nuts, google isn't indexing the openstack-infra logs | 05:05 |
| darraghb | mordred: any guestimate on how long ago that conversation came up? | 05:05 |
| *** afazekas has quit IRC | 05:05 | |
| *** jhesketh_ has quit IRC | 05:06 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:08 | |
| mordred | darraghb: gah... no :) | 05:08 |
| mordred | :( | 05:08 |
| *** Ryan_Lane has joined #openstack-infra | 05:08 | |
| darraghb | right so, wget the lot and grep it is | 05:10 |
| mordred | nice choice! | 05:11 |
| mordred | also, I'll try to remember to poke mgagne and zaro in the morning | 05:11 |
| *** Ryan_Lane has quit IRC | 05:13 | |
| *** SergeyLukjanov has quit IRC | 05:14 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:15 | |
| *** pmoosh has quit IRC | 05:16 | |
| *** AlexF has quit IRC | 05:24 | |
| *** thingee_zzz is now known as thingee | 05:32 | |
| *** afazekas has joined #openstack-infra | 05:35 | |
| *** slong has quit IRC | 05:35 | |
| *** slong has joined #openstack-infra | 05:35 | |
| *** jeremyb has quit IRC | 05:36 | |
| *** jeremyb has joined #openstack-infra | 05:36 | |
| *** Ryan_Lane has joined #openstack-infra | 05:39 | |
| darraghb | Can't find anything concrete on whether should default to sending blank XML tags where plugin normally doesn't set them in the UI unless a specific value is set by the user | 05:43 |
| *** jeremyb has quit IRC | 05:46 | |
| *** jeremyb has joined #openstack-infra | 05:46 | |
| *** senk has joined #openstack-infra | 05:47 | |
| *** Ryan_Lane has quit IRC | 05:47 | |
| *** afazekas is now known as afazekas_no_irq | 05:49 | |
| *** senk has quit IRC | 05:54 | |
| *** tvb|afk has joined #openstack-infra | 05:54 | |
| *** tvb|afk has quit IRC | 05:54 | |
| *** tvb|afk has joined #openstack-infra | 05:54 | |
| *** darraghb has quit IRC | 05:59 | |
| *** pblaho has joined #openstack-infra | 06:00 | |
| *** UtahDave has quit IRC | 06:14 | |
| *** SergeyLukjanov has quit IRC | 06:15 | |
| *** vipul is now known as vipul-away | 06:16 | |
| *** odyssey4me has joined #openstack-infra | 06:18 | |
| *** blamar has quit IRC | 06:20 | |
| *** AlexF has joined #openstack-infra | 06:29 | |
| *** vipul-away is now known as vipul | 06:30 | |
| *** slong has quit IRC | 06:33 | |
| *** Ryan_Lane has joined #openstack-infra | 06:39 | |
| *** flaper87|afk is now known as flaper87 | 06:43 | |
| *** Ryan_Lane has quit IRC | 06:46 | |
| *** senk has joined #openstack-infra | 06:50 | |
| ttx | mordred: translations get updated faster than the gate can merge them. | 06:50 |
| *** AlexF has quit IRC | 06:54 | |
| *** senk has quit IRC | 06:54 | |
| ttx | clarkb, jebalir: in case you're still around, Zuul status page looks broken to me (nothing but the graphs in there) | 06:54 |
| ttx | jeblair: ^ | 06:54 |
| *** jcoufal has joined #openstack-infra | 07:11 | |
| *** thingee is now known as thingee_zzz | 07:11 | |
| *** salv-orlando has joined #openstack-infra | 07:15 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:18 | |
| *** DinaBelova has joined #openstack-infra | 07:19 | |
| *** markmc has joined #openstack-infra | 07:35 | |
| *** Ryan_Lane has joined #openstack-infra | 07:40 | |
| *** fbo_away is now known as fbo | 07:42 | |
| *** Ryan_Lane has quit IRC | 07:44 | |
| *** jpich has joined #openstack-infra | 07:50 | |
| *** hashar has joined #openstack-infra | 07:54 | |
| *** dizquierdo has joined #openstack-infra | 07:55 | |
| *** johnthetubaguy has joined #openstack-infra | 07:57 | |
| *** DinaBelova has quit IRC | 08:00 | |
| *** AlexF has joined #openstack-infra | 08:04 | |
| *** salv-orlando has quit IRC | 08:05 | |
| *** DinaBelova has joined #openstack-infra | 08:08 | |
| *** yassine has joined #openstack-infra | 08:11 | |
| *** tvb|afk has quit IRC | 08:14 | |
| *** tvb|afk has joined #openstack-infra | 08:14 | |
| *** tvb|afk has quit IRC | 08:14 | |
| *** tvb|afk has joined #openstack-infra | 08:14 | |
| *** kpavel has joined #openstack-infra | 08:16 | |
| *** derekh has joined #openstack-infra | 08:18 | |
| *** AlexF has quit IRC | 08:19 | |
| *** kpavel has quit IRC | 08:19 | |
| AJaeger | Is jenkins or Zuul down? | 08:20 |
| BobBall | Probably | 08:21 |
| BobBall | or possibly not - why do you think they are? | 08:21 |
| AJaeger | BobBall, https://review.openstack.org/#/q/status:open+project:openstack/openstack-manuals,n,z | 08:22 |
| AJaeger | There's no progress since 90 mins - the validation jobs normally need less than 10 minutes each | 08:22 |
| BobBall | gate queue is very long ATM | 08:22 |
| AJaeger | BobBall, for examle https://review.openstack.org/#/c/48945/ | 08:22 |
| BobBall | Maybe it is broken | 08:23 |
| BobBall | That job isn't showing in Zuul | 08:23 |
| BobBall | but the gate is very long as I said | 08:23 |
| AJaeger | YEah, it's slower than usual for sure even earlier | 08:25 |
| AJaeger | 5hours ago it took 40 mins until "starting gating jobs " appeared at https://review.openstack.org/#/c/48738/ | 08:25 |
| AJaeger | this sometimes takes seconds... | 08:25 |
| *** dizquierdo has quit IRC | 08:27 | |
| AJaeger | Any idea why openstack-manuals are not shown at all on http://status.openstack.org/zuul/ ? | 08:40 |
| *** Ryan_Lane has joined #openstack-infra | 08:40 | |
| ttx | status/zuul up again | 08:41 |
| *** Ryan_Lane has quit IRC | 08:44 | |
| *** tkammer has joined #openstack-infra | 08:45 | |
| *** AlexF has joined #openstack-infra | 08:47 | |
| *** flaper87 is now known as flaper87|afk | 08:55 | |
| *** flaper87|afk is now known as flaper87 | 08:56 | |
| *** Clabbe has quit IRC | 09:03 | |
| *** dizquierdo has joined #openstack-infra | 09:04 | |
| *** Ryan_Lane has joined #openstack-infra | 09:10 | |
| *** AlexF has quit IRC | 09:12 | |
| *** Ryan_Lane has quit IRC | 09:15 | |
| ttx | but looks like events and results are not really processed | 09:18 |
| ttx | things merged 30 minutes ago but did not reach the post pipeline yet | 09:19 |
| ttx | approved reviews are not being queued to the gate either | 09:20 |
| *** senk has joined #openstack-infra | 09:22 | |
| *** senk has quit IRC | 09:26 | |
| *** dizquierdo has quit IRC | 09:33 | |
| *** dizquierdo has joined #openstack-infra | 09:37 | |
| ttx | hmm, looks like it suddenly unstuck | 09:50 |
| *** luhrs1 has joined #openstack-infra | 09:51 | |
| *** luhrs1 is now known as che-arne | 09:51 | |
| *** DinaBelova has quit IRC | 09:59 | |
| *** sandywalsh has quit IRC | 10:10 | |
| *** Ryan_Lane has joined #openstack-infra | 10:11 | |
| *** Ghe_Holidays is now known as GheRivero | 10:14 | |
| *** Ryan_Lane has quit IRC | 10:15 | |
| *** flaper87 is now known as flaper87|afk | 10:17 | |
| *** pcm_ has joined #openstack-infra | 10:19 | |
| *** pcm_ has quit IRC | 10:21 | |
| *** pcm_ has joined #openstack-infra | 10:21 | |
| *** senk has joined #openstack-infra | 10:22 | |
| *** sandywalsh has joined #openstack-infra | 10:23 | |
| *** schwicht has joined #openstack-infra | 10:23 | |
| *** SergeyLukjanov has quit IRC | 10:25 | |
| *** senk has quit IRC | 10:27 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:27 | |
| *** flaper87|afk is now known as flaper87 | 10:29 | |
| *** DinaBelova has joined #openstack-infra | 10:30 | |
| *** darraghb has joined #openstack-infra | 10:33 | |
| *** AlexF has joined #openstack-infra | 10:35 | |
| darraghb | jeblair: need an indication on whether the assumption in https://review.openstack.org/#/c/49102/ that there should only be 1 or no entry points returned is valid | 10:41 |
| AJaeger | ttx, indeed it's moving forward now | 10:41 |
| *** Ryan_Lane has joined #openstack-infra | 10:41 | |
| *** Ryan_Lane has quit IRC | 10:45 | |
| *** basha has quit IRC | 10:54 | |
| *** sergmelikyan has joined #openstack-infra | 10:54 | |
| sergmelikyan | Hi guys, not so long ago mordred updated repos with PBR and removed d2to1 from setup_requires section. | 10:56 |
| sergmelikyan | Also this commit added #!/usr/bin/env python hashbang, it's used/required by CI? | 10:56 |
| *** hashar has quit IRC | 11:00 | |
| sergmelikyan | And what is GLOBAL REQUIREMENTS REPO ? | 11:01 |
| *** AlexF has quit IRC | 11:04 | |
| sergmelikyan | last question is obsolete :( Next time I will search first :( | 11:04 |
| *** AlexF has joined #openstack-infra | 11:05 | |
| *** hashar has joined #openstack-infra | 11:05 | |
| *** boris-42 has joined #openstack-infra | 11:07 | |
| *** hashar_ has joined #openstack-infra | 11:13 | |
| *** hashar has quit IRC | 11:13 | |
| *** hashar_ is now known as hashar | 11:13 | |
| *** ojacques has quit IRC | 11:14 | |
| *** ojacques_ has joined #openstack-infra | 11:14 | |
| *** ojacques_ is now known as OlivierSanchez | 11:14 | |
| *** che-arne has quit IRC | 11:16 | |
| *** flaper87 is now known as flaper87|afk | 11:16 | |
| *** dims has joined #openstack-infra | 11:18 | |
| *** luhrs1 has joined #openstack-infra | 11:19 | |
| *** luhrs1 is now known as che-arne | 11:19 | |
| *** darraghb has left #openstack-infra | 11:21 | |
| *** flaper87|afk is now known as flaper87 | 11:21 | |
| *** senk has joined #openstack-infra | 11:23 | |
| sdague | ttx: I think that with the amount of work rescheduling resets, things are a little slower than you might imagine | 11:26 |
| ttx | sdague: ack... it's a lot slower than when the gate was so much loaded around FPF, though | 11:27 |
| *** senk has quit IRC | 11:27 | |
| sdague | yeh, but nnfi means it reschedules on the reset immediately | 11:27 |
| ttx | sdague: right. That creates weird artifacts, like the post pipeline only being fed once every 4 hours | 11:28 |
| ttx | I don't really think that's healthy | 11:28 |
| ttx | if you look at the post pipeline trends graph on zuul status page, you can see how weird it is | 11:29 |
| *** michchap has quit IRC | 11:29 | |
| sdague | yeh, though there were also a spike of check jobs at the same tim | 11:30 |
| ttx | nothing got there for like 5 hours and then we got 30 changes fed to it at once | 11:30 |
| *** michchap has joined #openstack-infra | 11:30 | |
| ttx | the check line was pretty much empty all morning | 11:30 |
| ttx | (european morning) | 11:30 |
| *** primemin1sterp has joined #openstack-infra | 11:31 | |
| ttx | so I wouldn't blame the check line. | 11:31 |
| ttx | the check line got fed at around the same moment as the post pipeline | 11:31 |
| sdague | honestly, it's probably worth figuring out how to instrument zuul for behavior we think is aberant | 11:31 |
| *** primeministerp has quit IRC | 11:31 | |
| sdague | because just changing scheduling again will push up some other issue (like the check starve) | 11:32 |
| sdague | we're actually running in a weird config right now because of reacting to the check starve | 11:32 |
| ttx | sdague: probably boils down to the way the events get processed | 11:32 |
| sdague | and the implications for when priorities are set | 11:32 |
| sdague | which is only when something enters a queue | 11:32 |
| ttx | I don't think it really resulted in slowing things down, just "looks" weird | 11:33 |
| ttx | also we lost zuul status page for about 3 hours | 11:33 |
| *** nosnos has quit IRC | 11:33 | |
| ttx | the page was still fetching json but would not display queues | 11:33 |
| ttx | no clear error in js consoles | 11:33 |
| ttx | I blamed the complexity of the gate tree | 11:34 |
| ttx | sdague: i'm considering to only allow gate fixes and RC1-blocking fixes, until we get a few RC1s out. I.e. stop random fixes from preventing us to issue rc1 | 11:35 |
| ttx | sdague: do you think that could help ? | 11:35 |
| sdague | ttx: are you sure that post did just get pushed to the bottom of the page/ | 11:36 |
| sdague | when the train map gets wide, the 3rd column drops | 11:36 |
| sdague | below it | 11:36 |
| ttx | sdague: the page was pretty empty. Only showed graphs | 11:36 |
| ttx | and the "Queue lengths:" line with empty counts | 11:37 |
| sdague | ttx: oh, yeh, so that means zuul stopped responding to json requests | 11:37 |
| *** matsuhashi has quit IRC | 11:37 | |
| sdague | why... no idea | 11:37 |
| ttx | sdague: the json was fetched alright | 11:37 |
| sdague | the json was empty? | 11:37 |
| ttx | Hmmm... all I can tell is that it was returning 200. That said I could fetch it myself. Used Dan Smith's tool to parse the json | 11:38 |
| sdague | ok | 11:39 |
| ttx | I blamed some timeout in the page processing that would not generate any error in console | 11:39 |
| sdague | yeh, could be | 11:39 |
| ttx | and then you fetch the next one before you could fully display the previous one | 11:39 |
| sdague | oh, yeh, that could be | 11:39 |
| sdague | we refresh rather more than we need | 11:40 |
| ttx | sdague: another fun fail: translations being re-imported (creating another patchset) while the gate is almost done processing the change | 11:40 |
| ttx | gets refreshed every day and those days, one day is not enough to get through the gate | 11:41 |
| ttx | oops. | 11:41 |
| sdague | ttx: so on the gate fix issue, the challenge is we've been having is that teams aren't prioritizing the project races that are causing the issues | 11:41 |
| *** Ryan_Lane has joined #openstack-infra | 11:41 | |
| *** Ryan_Lane has joined #openstack-infra | 11:41 | |
| ttx | sdague: i'm looking through the bugs right now | 11:42 |
| ttx | see if I can help to prioritize correctly. The trick is the bug status is often not reflecting reality | 11:42 |
| *** OlivierSanchez is now known as osanchez | 11:42 | |
| *** markmcclain has quit IRC | 11:42 | |
| sdague | well, the neutron bugs that are bouncing the gate I've set to Critical and rc1 | 11:42 |
| *** markmcclain has joined #openstack-infra | 11:43 | |
| ttx | Looking at https://bugs.launchpad.net/tempest/+bug/1226337 right now, trying to understand if it needs more than the cinder fix | 11:43 |
| uvirtbot | Launchpad bug 1226337 in cinder "tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern flake failure" [Undecided,Fix committed] | 11:43 |
| *** markmcclain1 has joined #openstack-infra | 11:44 | |
| *** markmcclain has quit IRC | 11:44 | |
| *** osanchez has quit IRC | 11:45 | |
| ttx | sdague: was wondering if the "Last seen" was calculated from hte recheck time or the test failure time :) | 11:45 |
| sdague | ttx: recheck time I think | 11:45 |
| *** Ryan_Lane has quit IRC | 11:46 | |
| sdague | if we have a good logstash requery we can track it down | 11:46 |
| *** ryanpetrello has joined #openstack-infra | 11:46 | |
| *** dcramer_ has quit IRC | 11:46 | |
| ttx | sdague: would be nice to know if that cinder fix closed it | 11:46 |
| ttx | since that was our biggest offender | 11:46 |
| sdague | that's not our biggest offender | 11:46 |
| sdague | and I suspect that's probably good now | 11:47 |
| ttx | ok, one of our 180+ retries offenders | 11:47 |
| *** sandywalsh has quit IRC | 11:47 | |
| *** ryanpetrello has quit IRC | 11:47 | |
| *** OlivierSanchez has joined #openstack-infra | 11:47 | |
| sdague | this is our biggest offender - https://bugs.launchpad.net/neutron/+bug/1224001 | 11:48 |
| uvirtbot | Launchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [Critical,In progress] | 11:48 |
| *** OlivierSanchez is now known as osanchez | 11:48 | |
| ttx | sdague: I was going by the retry count | 11:48 |
| sdague | 165 instances in last 24 hrs - http://logstash.openstack.org/#eyJzZWFyY2giOiJcInRlbXBlc3Quc2NlbmFyaW8udGVzdF9uZXR3b3JrX2Jhc2ljX29wcyBBc3NlcnRpb25FcnJvcjogVGltZWQgb3V0IHdhaXRpbmcgZm9yXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDQ1NzY5NzMxOX0= | 11:48 |
| sdague | so the problem is, with neutron issues, everyone just does a recheck no bug | 11:49 |
| ttx | ha. ha. | 11:49 |
| *** markmc has quit IRC | 11:49 | |
| *** yassine_ has joined #openstack-infra | 11:52 | |
| *** yassine has quit IRC | 11:56 | |
| *** dolphm has quit IRC | 11:58 | |
| *** sandywalsh has joined #openstack-infra | 12:01 | |
| *** dkranz has joined #openstack-infra | 12:01 | |
| *** MoXxXoM has quit IRC | 12:01 | |
| ttx | sdague: for 1224001, apparently it was narrowed down to enabling tenant isolation in order to get tempest neutron tests to run in parallel (bug 1216076)... any idea if the original disease was actually worse than the cure ? | 12:03 |
| uvirtbot | Launchpad bug 1216076 in tempest "Neutron jobs won't pass tempest running in parallel" [Critical,Fix released] https://launchpad.net/bugs/1216076 | 12:03 |
| *** tkammer has quit IRC | 12:03 | |
| sdague | ttx: yeh, I was just working through that | 12:04 |
| ttx | sdague: ok, then I'll stop distracting you :) | 12:04 |
| sdague | so if neutron can't run under tenant issolation... that's kind of a problem. Basically means that it can't take simultaneous requests | 12:04 |
| sdague | which, realistically is what we are seeing | 12:04 |
| sdague | so my current idea | 12:04 |
| sdague | revert that patch | 12:04 |
| ttx | sdague: yes, that would still be a bug in Neutron alright. Just trying to unfuck the gate | 12:05 |
| sdague | add 2 neutron tests to the neutron jobs that run under isolation, so the neutron team could still work through it | 12:05 |
| sdague | which would also have the added benefit of getting us 4 tempest jobs on neutron changes | 12:05 |
| *** dims has quit IRC | 12:07 | |
| ttx | sdague: sounds like a plan | 12:07 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tenant issolation jobs to neutron patches https://review.openstack.org/49144 | 12:07 |
| sdague | so the only issue with that plan.... | 12:08 |
| *** dims has joined #openstack-infra | 12:08 | |
| sdague | it will take a full day for something to move up through the gate | 12:08 |
| ttx | sdague: unless we kill it. If that solves like 50% of resets that's worth it | 12:09 |
| *** dizquierdo has left #openstack-infra | 12:09 | |
| sdague | well - https://review.openstack.org/#/c/49143/ is for review | 12:09 |
| *** Ryan_Lane has joined #openstack-infra | 12:12 | |
| sdague | ttx: also https://review.openstack.org/#/c/49144/ - which would be the zuul config changes | 12:12 |
| *** mriedem has joined #openstack-infra | 12:15 | |
| *** Ryan_Lane has quit IRC | 12:16 | |
| *** hashar has quit IRC | 12:18 | |
| *** flaper87 is now known as flaper87|afk | 12:20 | |
| *** esker has joined #openstack-infra | 12:21 | |
| mriedem | mordred: i think this has to be pushed through the gate since it's quantumclient - it's blocking all stable branch patches: https://review.openstack.org/#/c/49006/ | 12:22 |
| *** tkammer has joined #openstack-infra | 12:22 | |
| *** senk has joined #openstack-infra | 12:23 | |
| ttx | yeah, a pile of 4 just got through | 12:24 |
| *** esker has quit IRC | 12:26 | |
| *** senk has quit IRC | 12:27 | |
| sdague | ttx: well the neutron job passed on that devstack one | 12:29 |
| sdague | fungi: you up yet? | 12:29 |
| ttx | people need to stop on this sleeping habit | 12:30 |
| dims | lol | 12:32 |
| *** dkliban has joined #openstack-infra | 12:32 | |
| *** rfolco has joined #openstack-infra | 12:32 | |
| *** flaper87|afk is now known as flaper87 | 12:33 | |
| *** tizzo has joined #openstack-infra | 12:35 | |
| mriedem | sdague: am i wrong about this? ttps://review.openstack.org/#/c/49006/ | 12:36 |
| mriedem | in that it won't pass tempest because it's quantumclient | 12:37 |
| mriedem | oops https://review.openstack.org/#/c/49006/ | 12:37 |
| sdague | mriedem: yeh, I think anything on the quantumclient branch needs special handling | 12:38 |
| * sdague can't wait until we drop grizzly support | 12:38 | |
| mriedem | sdague: has to wait 6 months | 12:38 |
| mriedem | you could hibernate! | 12:39 |
| *** odyssey4me has quit IRC | 12:39 | |
| ttx | can't wait until we drop folsom support. | 12:39 |
| mriedem | ttx: speaking of: https://review.openstack.org/#/c/48788/ | 12:40 |
| mriedem | i thought folsom only got security fixes now, is that right? | 12:40 |
| *** thomasm has joined #openstack-infra | 12:40 | |
| mriedem | thomasm: good morning, are you going to abandon this? https://review.openstack.org/#/c/48299/ | 12:41 |
| ttx | mriedem: it no longer gets point releases. That doesn't prevent critical bugfixes from being proposed to the branch | 12:41 |
| ttx | not convinced that would be one of them though | 12:41 |
| mriedem | ttx: ok. once icehouse is released though then people are on their own for folsom patches right? | 12:42 |
| thomasm | mriedem, Oh, man, I forgot to do that. sorry about that! | 12:42 |
| ttx | yes, definitely | 12:42 |
| *** markmc has joined #openstack-infra | 12:42 | |
| mriedem | thomasm: np, just cleaning up my review backlog :) | 12:42 |
| thomasm | =] | 12:42 |
| thomasm | done | 12:42 |
| *** Ryan_Lane has joined #openstack-infra | 12:43 | |
| mriedem | thanks | 12:43 |
| thomasm | Sure thing. Thanks for reminding me! | 12:43 |
| *** pblaho has quit IRC | 12:44 | |
| AlexF | a na samom dele? | 12:47 |
| *** Ryan_Lane has quit IRC | 12:47 | |
| AlexF | sorry… please disregard... | 12:47 |
| thomasm | Hey jog0, would you mind checking out https://review.openstack.org/#/c/48351/ when you get a change? It's that devstack fix for RPC notifier when CM is enabled. | 12:47 |
| thomasm | Fixed up the commit mesage, like asked. | 12:47 |
| thomasm | message* | 12:48 |
| thomasm | chance* | 12:48 |
| thomasm | whoa can't type this morning | 12:48 |
| sdague | thomasm: jog0's on the west coast at a conference, so I'd expect he'll be on a little later | 12:48 |
| thomasm | sdague, Ohhh okay. Cool. Thanks! =] | 12:49 |
| fungi | sdague: 'sup? | 12:50 |
| * fungi peruses scrollback | 12:50 | |
| sdague | fungi: so this https://review.openstack.org/#/c/49143/ has a good chance of alleviating the gate resets | 12:51 |
| sdague | but if I just put it into the gate, it probably won't bounce to the top until late this afternoon | 12:52 |
| sdague | so what kind of magic should we do to get that to the top of the queue | 12:53 |
| * fungi looks and thinks | 12:56 | |
| koolhead17 | mordred: around | 12:57 |
| fungi | also, i've only read about half the scrollback so far... did we get any more problem slaves like centos6-10 throwing java exceptions in the log and not running tests? | 12:57 |
| sdague | fungi: not that I've noticed this morning | 12:57 |
| *** jcoufal has quit IRC | 12:58 | |
| *** dprince has joined #openstack-infra | 12:58 | |
| fungi | sdague: okay, so there are two possible ways for jumping the queue with 49143 (maybe more than that, but these are the options i know)... | 12:58 |
| fungi | one is to manually merge that change to tempest and bypass the gate, risking the potential that it's not actually gate-worthy even though check jobs succeeded | 12:59 |
| sdague | so it's a revert for devstack, so I think manual merge would be fine | 13:00 |
| fungi | the other is to stop zuul and dump the queue, then reenqueue all pending changes with that one at the front of the list | 13:00 |
| fungi | er, right devstack. not tempest | 13:00 |
| * fungi is still on coffee #2 | 13:00 | |
| sdague | no worries | 13:00 |
| sdague | so because it's a revert, I think direct merge is ok | 13:00 |
| sdague | because we know we worked with the old code, as we did until thursday | 13:01 |
| fungi | i can take care of that, in that case | 13:01 |
| sdague | great, the sooner the better | 13:01 |
| fungi | gimme a few minutes to make sure i do it "correctly" and don't just cause myself new problems to spend my morning fixing | 13:01 |
| *** julim has joined #openstack-infra | 13:01 | |
| sdague | yep, no worries :) | 13:01 |
| *** rockyg has quit IRC | 13:02 | |
| sdague | then a couple of gate resets later we should have everyone on the code, and the gate will flow again | 13:02 |
| *** anteaya has joined #openstack-infra | 13:03 | |
| fungi | yeah, just thinking through that to make sure there's not something i'm missing. it might cause other queued changes to lib/tempest to fail to merge because of conflicts, but seems generally safe | 13:03 |
| *** sandywalsh has quit IRC | 13:03 | |
| *** weshay has joined #openstack-infra | 13:04 | |
| sdague | yeh, it should be pretty issolated | 13:05 |
| fungi | sdague: are there any other devstack cores around to get a second +2 just for cya? | 13:05 |
| sdague | I can live with merge conflicts | 13:05 |
| sdague | dtroyer is the only other one | 13:05 |
| sdague | really | 13:05 |
| fungi | k | 13:05 |
| sdague | honestly, we +A stuff through in emergencies, so consider it +A | 13:05 |
| *** davidhadas has quit IRC | 13:06 | |
| sdague | I just didn't want to actually send it to the gate | 13:06 |
| *** tizzo has quit IRC | 13:06 | |
| fungi | here goes... | 13:08 |
| fungi | and it's merged | 13:08 |
| sdague | great, thank you | 13:08 |
| sdague | oh, sweet, the top of the queue reset | 13:09 |
| sdague | which means we might get this all in one go on the new code | 13:09 |
| sdague | oh, never mind, it reset 7 mins ago | 13:10 |
| * fungi resumes catching up on scrollback to see if there are any other fires which need putting out | 13:10 | |
| sdague | I think that's the only fire | 13:10 |
| *** adalbas has joined #openstack-infra | 13:10 | |
| ttx | fungi: agree that's theonly fire | 13:11 |
| ttx | a few weird things like the zuul status page not showing queues, or weird sudden processing of results resulting in spikes in the post pipeline after hours of inactivity | 13:12 |
| ttx | but that was not strictly blocking | 13:12 |
| *** Ryan_Lane has joined #openstack-infra | 13:13 | |
| *** yassine has joined #openstack-infra | 13:15 | |
| fungi | right, sounds like the "refresh faster than we can draw" theory is a good possibility there | 13:15 |
| sdague | fungi: when you get a chance - https://review.openstack.org/#/c/49144/ is sort of the 2nd half of 49143 | 13:16 |
| ttx | fungi: all I can say is that it was fetching the status.json alright... and no JS error in the console | 13:16 |
| sdague | to bring the jobs back for neutron | 13:16 |
| fungi | also, on translation import derpage, we've been trying to get up with tom because something seems to have changed with transifex accounts for cinder and nova such that we haven't been successfully sending the updated potfiles back for no idea how long now | 13:16 |
| sdague | so we don't just make RC by turning off all the tests | 13:16 |
| *** dcramer_ has joined #openstack-infra | 13:16 | |
| sdague | ok, 47984,2 just reset | 13:17 |
| fungi | hopefully i'll catch him at tonight's conference call if nothing else | 13:17 |
| sdague | which means everything in the gate will now be on the new code for neutron tests (or already passed their neutron tests) | 13:17 |
| *** hashar has joined #openstack-infra | 13:17 | |
| *** Ryan_Lane has quit IRC | 13:18 | |
| sdague | so this is the moment of truth | 13:18 |
| *** zul has quit IRC | 13:18 | |
| *** dkliban has quit IRC | 13:19 | |
| *** sandywalsh has joined #openstack-infra | 13:19 | |
| *** yassine_ has quit IRC | 13:19 | |
| *** AlexF has quit IRC | 13:20 | |
| *** esker has joined #openstack-infra | 13:22 | |
| *** hashar has quit IRC | 13:24 | |
| *** che-arne has quit IRC | 13:25 | |
| *** ryanpetrello has joined #openstack-infra | 13:28 | |
| *** alexpilotti has joined #openstack-infra | 13:30 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tenant isolation jobs to neutron patches https://review.openstack.org/49144 | 13:30 |
| sdague | thanks jd__ for spelling comments :) | 13:31 |
| *** AlexF has joined #openstack-infra | 13:31 | |
| jd__ | that's all I can do :) | 13:32 |
| sdague | ok, first gate reset since that revert is the pypi CDN 'slode | 13:32 |
| sdague | so that's good | 13:32 |
| *** zul has joined #openstack-infra | 13:32 | |
| *** hashar has joined #openstack-infra | 13:33 | |
| *** acabrera has joined #openstack-infra | 13:33 | |
| *** acabrera is now known as alcabrera | 13:34 | |
| *** blamar has joined #openstack-infra | 13:34 | |
| *** luhrs1 has joined #openstack-infra | 13:41 | |
| *** davidhadas has joined #openstack-infra | 13:41 | |
| *** luhrs1 is now known as che-arne | 13:42 | |
| *** jcoufal has joined #openstack-infra | 13:42 | |
| clarkb | ttx: delay in post processing possibly due to gearman queue priority. the backlog of check and gate must be serviced first | 13:42 |
| ttx | clarkb: the numbers in "queue length" were definitely looking bad. Like 150 events and 90 results or something | 13:43 |
| *** Ryan_Lane has joined #openstack-infra | 13:44 | |
| ttx | clarkb: you can still see te calm then the peak on the post pipe trend line on zuul status page | 13:44 |
| sdague | ttx: first neutron reset in the gate, but it's a different bug | 13:45 |
| ttx | clarkb: do we have any leverage to throw more muscle into gearman queue processing ? | 13:45 |
| openstackgerrit | David Peraza proposed a change to openstack/requirements: Adding sqlalchemy db2 dialect dependencies https://review.openstack.org/48745 | 13:45 |
| ttx | sdague: so far so good | 13:45 |
| sdague | so I think one of the NNFI failings right now is the fact that failing jobs get automatically readded if jobs above them reset | 13:46 |
| *** tkammer has quit IRC | 13:47 | |
| *** dkranz has quit IRC | 13:47 | |
| sdague | the current issue is we've got 3 requirements changes in the gate | 13:47 |
| ttx | sdague: if there is a way to anticipate failure, https://review.openstack.org/#/c/49029/ was -2ed so we already know it won't merge | 13:47 |
| sdague | ttx: that doesn't keep it out of the gate | 13:48 |
| sdague | just prevents it from merging | 13:48 |
| ttx | can we kill it and win 12 min ? | 13:48 |
| *** Ryan_Lane has quit IRC | 13:48 | |
| sdague | ttx: that won't be a reset though | 13:48 |
| sdague | it will just stall | 13:49 |
| *** rcleere has quit IRC | 13:49 | |
| sdague | so right now the requirements changes have a 50% chance of failing because the pypi CDN is broken, and it's going to take a week or two for them to roll out fixes | 13:50 |
| clarkb | sdague: ya the assumption is the failure closest to the head of the queue causes subsequent failures. this doesn't apply well to a flaky gate | 13:50 |
| sdague | clarkb: it's not a flakey gate! | 13:50 |
| sdague | it's bugs in the servers | 13:50 |
| sdague | I'm going to hit anyone that uses the terms flakey gate or flakey tests from here on out :) | 13:50 |
| clarkb | sdague right less deterministic failures somewhere | 13:51 |
| * anteaya removes flakey from her list of words, except to describe pie crust | 13:51 | |
| clarkb | that better? :) | 13:51 |
| sdague | :) | 13:51 |
| sdague | yes | 13:51 |
| sdague | so the assumption that we made previously was that the race in the server that caused a reset was introduced in the change that reset | 13:51 |
| sdague | but in experience we know that's rarely the case | 13:52 |
| sdague | these are almost always races that were introduced way before the patch that failed | 13:52 |
| jgriffith | +1 | 13:53 |
| fungi | well, i think the ordering is still tuned for deterministic failures, really | 13:53 |
| sdague | fungi: right | 13:53 |
| fungi | and honestly i have trouble imagining how we could tune it more efficiently for non-deterministic failure modes | 13:53 |
| sdague | but the system is now at a complexity, and the testing is now actually starting to evolve to something closer to a real world env, that strict determinism is gone | 13:54 |
| sdague | fungi: when something fails, kick it out entirely | 13:54 |
| *** dkliban has joined #openstack-infra | 13:54 | |
| *** changbl has quit IRC | 13:54 | |
| fungi | that still suggests an assumption that the race in the server that caused a reset was introduced in the change that reset | 13:54 |
| sdague | basically the problem is we effectively have "auto recheck" on things in the queue if they are still running any test job when something ahead of them fails | 13:55 |
| sdague | which actually means we aren't keeping track of at least 50% of our races | 13:55 |
| sdague | because they are hidden in the fact that the job auto rechecked multiple times in the gate | 13:56 |
| *** dcramer_ has quit IRC | 13:57 | |
| sdague | if we kicked them, then we'd make folks go through the normal process of registering the race | 13:57 |
| fungi | i expect the end result to be the same, statistically | 13:57 |
| sdague | well, the bias is actually that the longer a test you have, the more likely you are to get autorechecked | 13:58 |
| sdague | so neutron actually gets a lot of autorechecks | 13:58 |
| sdague | because of the long py26 job | 13:58 |
| *** AlexF has quit IRC | 13:59 | |
| fungi | yes, but it's only the recheck which happens when all other changes ahead succeed which counts | 13:59 |
| fungi | and that set of jobs runs only once for any given change | 14:00 |
| sdague | so take those requirements jobs, they are going to keep failing | 14:00 |
| sdague | we've got 3 in a row in the gate | 14:00 |
| sdague | and at 50% fail rate, everything behind it is basically "give up, wait until dinner" | 14:00 |
| sdague | and they've failed a lot already | 14:00 |
| fungi | right, so the net result is that we should have some automated means of preventing approvals for requirements changes? that will be the end result anyway | 14:01 |
| sdague | but we are auto rechecking them, vs. kicking them out and making a manual call to put them back in | 14:01 |
| *** tkammer has joined #openstack-infra | 14:01 | |
| sdague | or, if you fail, stick you at the end of the queue | 14:01 |
| sdague | the next time a reset happens | 14:02 |
| fungi | basically, the longer the integrated gate queue is, the greater the chance that changes tested more heavily against jobs with nondeterministic failures will require manual intervention (because they get more opportunities to be kicked out) | 14:02 |
| *** markmc has quit IRC | 14:03 | |
| sdague | fungi: sure, but then we can have some human control over patches | 14:03 |
| fungi | however, when the queue is short, the same changes don't get kicked out as aggressively because they have those same jobs run fewer times, at least under your proposal | 14:03 |
| sdague | fungi: yes | 14:04 |
| *** jcoufal has quit IRC | 14:04 | |
| sdague | however, on average, the queue will be shorter | 14:04 |
| sdague | because we are shedding things out | 14:04 |
| sdague | and making people put them back in | 14:04 |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Add tests to the assignedNode option https://review.openstack.org/49158 | 14:04 |
| sdague | which would also let a gate fixing bug fix have a better chance of moving through the queue | 14:04 |
| fungi | i think that shorter queue comes at the expense of a lot of additional developer time and a slowing down of the overall pace of development. i guess that could be argued as a positive outcome, but i'm unconvinced | 14:05 |
| *** markmcclain1 has quit IRC | 14:05 | |
| mriedem | hmm, speaking of translations... | 14:06 |
| mriedem | babel.core.UnknownLocaleError: unknown locale 'tl_PH' | 14:06 |
| mriedem | that's coming from cinder using babel 0.9.6 | 14:06 |
| fungi | and moving them to the back of the queue on failure risks that the developers will never get feedback if the gate is long enough that it never empties, but maybe that will happen less often the fewer failing changes are closer to the head | 14:07 |
| *** thedodd has joined #openstack-infra | 14:07 | |
| sdague | part of the issue is our entire categorization system of races, rechecks, doesn't work unless it leaves the queue | 14:08 |
| *** davidhadas_ has joined #openstack-infra | 14:09 | |
| sdague | unrelated precise31 looks bad | 14:09 |
| sdague | https://jenkins01.openstack.org/job/gate-python-cinderclient-pep8/148/console | 14:09 |
| fungi | checking | 14:09 |
| fungi | ahh, yep | 14:10 |
| *** AlexF has joined #openstack-infra | 14:10 | |
| fungi | i've offlined it while i prod it a littole | 14:10 |
| fungi | little | 14:10 |
| *** davidhadas has quit IRC | 14:11 | |
| sdague | what about an intermediate solution where you only get to be auto-rechecked if you fail a shared job? | 14:12 |
| sdague | or more specifically if you fail a job unique to you, you don't get to be put back into the gate automatically on a reset above you. I think that could be figured out programatically | 14:13 |
| mriedem | ttx: given your discussion with russellb about translations, i think this is going to be a problem for rc1: https://bugs.launchpad.net/cinder/+bug/1233694 | 14:14 |
| uvirtbot | Launchpad bug 1233694 in cinder "babel 0.9.6 can't compile tl_PH locale" [Undecided,New] | 14:14 |
| sdague | mriedem: we're trying to get babel 1.1 into requirements, it's in the gate :) | 14:14 |
| mriedem | sdague: link to review? | 14:15 |
| sdague | https://review.openstack.org/#/c/48739/ | 14:15 |
| *** jcoufal has joined #openstack-infra | 14:15 | |
| *** mkoderer has quit IRC | 14:15 | |
| fungi | sdague: agreed, i think zuul could fairly easily confirm whether there were no changes ahead of you which are being tested with the same job | 14:15 |
| mriedem | sdague: thanks | 14:15 |
| *** Alex_Gaynor has quit IRC | 14:15 | |
| *** tvb|afk has quit IRC | 14:18 | |
| sdague | fungi: so it would be really nice to have something that auto downs nodepool nodes on a Java stack trace | 14:19 |
| sdague | so it doesn't kill 6 changes before we notice | 14:19 |
| mordred | sergmelikyan: the #!/usr/bin/env python is not used/required - it's just in the boilerplate setup.py we copy around now | 14:20 |
| mordred | sergmelikyan: is it problematic for you? | 14:20 |
| *** tizzo has joined #openstack-infra | 14:21 | |
| fungi | sdague: nodepool already does that, effectively | 14:22 |
| fungi | sdague: the two slaves mentioned today are static/reused virtual machines, not under nodepool's control | 14:23 |
| sdague | ok, so precise31 failed at least 3 patches | 14:23 |
| sdague | oh, ok, so how about those? | 14:23 |
| mordred | mriedem: ok | 14:23 |
| fungi | sdague: jenkins would need some way of interpreting the failure and taking that node out of service. would probably be a jenkins plugin | 14:24 |
| sdague | fungi: how do you do it now? | 14:24 |
| sdague | manually | 14:24 |
| fungi | sdague: i log into the jenkins webui and click a button | 14:24 |
| *** dcramer_ has joined #openstack-infra | 14:24 | |
| sdague | so jenkins has a REST interface | 14:24 |
| sdague | I wonder if we could look at the static jobs and automate that | 14:24 |
| fungi | sdague: it's certainly possible, but would probably fail up to several jobs before it was taken out of service | 14:25 |
| sdague | ok, still probably faster then human eyes :) | 14:25 |
| fungi | and the current plan, i think (last i heard) was to switch the static slaves to nodepool management anyway | 14:26 |
| fungi | which is probably a better use of our time | 14:26 |
| sdague | ok, cool | 14:26 |
| *** tizzo has quit IRC | 14:27 | |
| anteaya | mordred: could you take a peek at a pm i sent you? | 14:27 |
| sdague | agreed :) just trying to figure out all the fails that are doing us damage now | 14:27 |
| fungi | there's a non-zero chance right now that the recent rise in static node failures are due to us pushing jenkins too hard, so having each master start taking all of its slaves offline without a human in the loop to go fix them would potentially take us in the wrong direction as available testing resources go | 14:27 |
| *** pblaho has joined #openstack-infra | 14:28 | |
| fungi | we're taking a bit of a wait-and-see approach since the jenkins master restarts over the weekend, but may decide this is a sign we need more masters and have scaled the current ones to too many slaves apiece | 14:28 |
| mordred | eek. that's a great scrollback to wake up to | 14:29 |
| fungi | mordred: it was mostly about you | 14:29 |
| fungi | (okay, not really) | 14:30 |
| mordred | I think I might be leaning towards sdague in terms of kicking things out once they report failure | 14:30 |
| mordred | although I'm baffled about how to get more priority on these | 14:30 |
| *** dizquierdo has joined #openstack-infra | 14:31 | |
| mordred | sdague, ttx: perhaps a strongly worded email similar to the above thing sdague said about it not being a flaky gate or flaky tests? | 14:31 |
| ttx | mordred: we discussed it earlier. The correct wording is "f*cked gate" and "gate-f*cking bugs" | 14:32 |
| ttx | the gate is perfectly fine and honest, it's just getting abused | 14:32 |
| mordred | the gate is, in fact, showing us the current state of the software | 14:33 |
| mordred | it's just doing it in aggregate | 14:33 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tenant isolation jobs to neutron patches https://review.openstack.org/49144 | 14:33 |
| ttx | mordred: I think those issues are getting some attention. It's just tricky issues so everyone prefer to look at another problem | 14:33 |
| mtreinish | sdague: ^^^ so greenlighting tenat isolation on neutron caused some of the issues? | 14:34 |
| sdague | mtreinish: that's the guess | 14:34 |
| ttx | clarkb: did you just throw some muscle at events processing ? the q is down to 0 | 14:34 |
| ttx | for the first time in the last 10 hours at least | 14:34 |
| sdague | based on temporal corolation | 14:34 |
| clarkb | ttx nothing from me. I am not actually here this is an illusion | 14:35 |
| ttx | soething definitely happened | 14:35 |
| anteaya | clarkb: ghost tooth | 14:35 |
| *** markmcclain has joined #openstack-infra | 14:37 | |
| mriedem | sdague: looks like babel 1.1 isn't going to cut it, there is a bug in there for zh_CN | 14:37 |
| soren | mordred: devpi is working out quite well. Thanks for the recommendation. | 14:37 |
| mriedem | babel.core.UnknownLocaleError: unknown locale 'zh_CN' | 14:37 |
| mriedem | http://babel.pocoo.org/docs/changelog/#version-1-3 | 14:37 |
| jeblair | ttx: if you send me the status.json i can try to figure out why the status page didn't work | 14:38 |
| jeblair | the post queue is bursty because the gate queue is being reset so much, zuul spends most of its time processing that. | 14:38 |
| mtreinish | sdague: that's pathetic, and we expect it to have a full gate at some point in the future? | 14:38 |
| ttx | jeblair: I happened to have saved it | 14:39 |
| mordred | soren: w00t! | 14:39 |
| *** davidhadas has joined #openstack-infra | 14:39 | |
| mordred | mriedem: so you're saying we're going to need Babel 1.3? | 14:40 |
| ttx | jeblair: sent | 14:40 |
| jeblair | ttx: event queues build up becaues the gate queue is being reset so much, zuul spends most of its time processing that. :) as soon as it settles for a bit, zuul works through the backlog quickly. | 14:40 |
| mriedem | mordred: yes | 14:40 |
| mriedem | mordred: added a note here: https://review.openstack.org/#/c/48739/ | 14:40 |
| mriedem | i'll update my cinder bug | 14:40 |
| *** Alex_Gaynor has joined #openstack-infra | 14:41 | |
| jeblair | sdague, mordred: if you're very concerned with the pypi failures and the requirements changes, perhaps this would help: https://review.openstack.org/#/c/49093/ | 14:42 |
| *** davidhadas_ has quit IRC | 14:43 | |
| mriedem | mordred: i just confirmed that babel 1.3 compiles zh_CN | 14:43 |
| *** tkammer has quit IRC | 14:43 | |
| *** tkammer has joined #openstack-infra | 14:44 | |
| *** mgagne has joined #openstack-infra | 14:44 | |
| *** mgagne has joined #openstack-infra | 14:44 | |
| mordred | jeblair: yes. I'm hoping that it will | 14:44 |
| jeblair | so let's merge it already? :) | 14:44 |
| fungi | jeblair: i already approved | 14:44 |
| jeblair | fungi: thanks | 14:45 |
| *** Ryan_Lane has joined #openstack-infra | 14:45 | |
| fungi | it was on top of my pile to look at from overnight gate-things | 14:45 |
| fungi | also, https://review.openstack.org/49144 should probably get a high review priority so that neutron folk don't spend too long thinking their issues are resolved | 14:47 |
| mriedem | mordred: also confirmed that babel 1.3 compiles tl_PH, so i'll push up a patch to openstack-requirements for bumping babel to 1.3 | 14:47 |
| mordred | +2 from me - I'll let jeblair +A it | 14:47 |
| mordred | mriedem: you wannt just hijack the 1.1 patch? | 14:48 |
| mriedem | mordred: hijack how? | 14:48 |
| mriedem | push another patch to it? | 14:48 |
| mordred | yah | 14:48 |
| mordred | sdague: in our earlier conversation about ejecting things from the gate | 14:48 |
| mriedem | sdague: mordred: i'm not sure if my CLA will allow that - sdague - do you know? | 14:49 |
| mriedem | (pushing a change to a non-ibm patch) | 14:49 |
| *** Ryan_Lane has quit IRC | 14:49 | |
| mordred | sdague: we could, if we happen to see patches that have failed in the middle of the gate, just push commit message amending patches to them which will get them ejected (evil hack) | 14:49 |
| sdague | mordred: yeh, I know, but I haven't fully scripted that hack yet :) | 14:50 |
| mordred | sdague still doens't really help with the recheck machinery though | 14:50 |
| mordred | sdague: :) | 14:50 |
| sdague | gate-snipe | 14:50 |
| sdague | mriedem: you should be fine | 14:51 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Move savanna under openstack org https://review.openstack.org/48491 | 14:51 |
| fungi | mordred: having zuul notice cdrv=-2 or aprv=0 comments in the gerrit stream on changes it's got in a dependent pipeline and dequeue them might be a good feature request | 14:51 |
| mriedem | sdague: ok, i'll save the transcript :_ | 14:51 |
| mriedem | :) | 14:51 |
| sdague | mriedem: isn't not even code ;) | 14:51 |
| mriedem | sdague: yeah, i feel i have to err on the side of caution though | 14:52 |
| *** changbl has joined #openstack-infra | 14:52 | |
| *** rcleere has joined #openstack-infra | 14:54 | |
| sdague | yeh, no worries. I can snag it if you like. | 14:57 |
| openstackgerrit | Matt Riedemann proposed a change to openstack/requirements: Raise Babel requirements to >= 1.3 https://review.openstack.org/48739 | 14:59 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Raise Babel requirements to >= 1.3 https://review.openstack.org/48739 | 14:59 |
| sdague | doh | 14:59 |
| sdague | well, that was a thing | 14:59 |
| mriedem | sdague: jinks | 14:59 |
| mriedem | *jinx | 14:59 |
| sdague | well, it looks live I've got patch 3, it will be funny to see how jenkins handles all of that | 15:00 |
| mriedem | sdague: if you don't mind, i'd like to push mine up so it addresses the cinder bug | 15:00 |
| mriedem | in the commit message | 15:00 |
| *** matty_dubs|gone is now known as matty_dubs | 15:00 | |
| sdague | mriedem: sure | 15:00 |
| sdague | so pull the review again and do another --amend | 15:00 |
| sdague | then it should sit on top | 15:01 |
| *** jcoufal has quit IRC | 15:02 | |
| *** pblaho has quit IRC | 15:02 | |
| *** boris-42 has quit IRC | 15:02 | |
| sdague | jeblair: I don't understand the jenkins fail on my layout patch - http://logs.openstack.org/44/49144/3/check/gate-config-layout/8a686d1/console.html | 15:03 |
| *** DinaBelova has quit IRC | 15:04 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack/requirements: Raise Babel requirements to >= 1.3 https://review.openstack.org/48739 | 15:04 |
| sdague | as it looks like I have all the symbols cross listed right, unless i've gone blind | 15:04 |
| openstackgerrit | A change was merged to openstack-infra/config: Set pip timeout to 60s when using pypi https://review.openstack.org/49093 | 15:04 |
| jeblair | sdague: needs to be added to the devstack-jobs job group at the end of devstack-gate.yaml | 15:05 |
| jeblair | sdague: (so that they are instantiated with check- and gate- variants) | 15:05 |
| *** dkranz has joined #openstack-infra | 15:05 | |
| jeblair | sdague: left comment | 15:06 |
| *** yamahata has joined #openstack-infra | 15:06 | |
| sdague | gotcha | 15:06 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add tenant isolation jobs to neutron patches https://review.openstack.org/49144 | 15:07 |
| sdague | ok, fixed | 15:07 |
| sdague | jeblair: so what do you think about the idea in zuul that if you've failed a job which no one else above you in the queue is running, you don't get automattically restarted when a reset above you happens | 15:08 |
| fungi | gah, yeah i overlooked that when i pointed out the other issues with the patch | 15:09 |
| *** yamahata has quit IRC | 15:09 | |
| fungi | i guess we all did. thank you gate-config-layout job! ;) | 15:10 |
| sdague | heh | 15:10 |
| sdague | fungi: https://jenkins02.openstack.org/job/gate-ceilometer-python26/1282/console - crc fail? | 15:11 |
| fungi | fun. i'll see if that tarball's still busted | 15:12 |
| *** basha has joined #openstack-infra | 15:13 | |
| fungi | possible that we have a race when a tarball is being uploaded while another job tries to download it | 15:13 |
| jeblair | sdague: i'm still thinking about it, but so far i'm not a big fan; it reverses a lot of the understanding about what zuul should be doing, so i don't want to embark on that lightly. | 15:13 |
| ttx | mordred, jeblair: I was discussing with sdague the need to temporarily restrict the gate to RC1 and gate fixes, but it seems things are flowing slightly better now ? | 15:13 |
| freyes | whois mgagne | 15:13 |
| mgagne | I am me | 15:14 |
| sdague | jeblair: so here's the thing | 15:14 |
| freyes | mgagne: sorry, I was trying to do /whois foo to check if you where available | 15:14 |
| jeblair | sdague: we _do_ see patches affecting following patches from time to time, and in the case of flakey failures, i'm not sure ejecting and reporting them more often is going to help. it just means that people roll the dice more often than a machine. | 15:15 |
| sdague | neutron is currently only passing about 4% of the tempest load a nova to land a change | 15:15 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add jobs for extra and dib repos https://review.openstack.org/49167 | 15:15 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove unused savanna rtfd jobs https://review.openstack.org/49168 | 15:15 |
| *** therve has quit IRC | 15:15 | |
| sdague | oh, different thing :) | 15:15 |
| freyes | mgagne: I updated the change 48661 (add git shallow clone support) in case you wanna take a second look :) | 15:15 |
| fungi | sdague: that tarball's fine, so either it got corrupted in transit or it got downloaded while it was being overwritten | 15:15 |
| jeblair | sdague: parse error on "neutron is currently only passing about 4% of the tempest load a nova to land a change" | 15:15 |
| sdague | jeblair: it's not related to this conversation | 15:15 |
| *** Ryan_Lane has joined #openstack-infra | 15:16 | |
| jeblair | oh. stricken from record then. :) | 15:16 |
| sdague | jeblair: so basically right now patches have a window of "auto recheck" | 15:16 |
| sdague | while the are auto rechecking, developers don't get notified of the races that are being exposed in their test run | 15:16 |
| sdague | elastic-recheck doesn't get to see that data | 15:16 |
| sdague | all that race behavior is hidden | 15:17 |
| jeblair | i don't really look at it that way; it's the same behavior we've always had it zuul, which is that your patch is rejected only if it fails on its own merit | 15:17 |
| sdague | but I think it made the assumption that the reason for a fail was related to something in the queue | 15:17 |
| jeblair | we had a time when for one small case, patches were rejected _not_ based on their own merit (when there were merge conflicts), and we got, rightly so i think, complaints because of that | 15:17 |
| jeblair | sdague: that does not have to be hidden from e-r | 15:17 |
| jeblair | sdague: i suggested a while back that e-r should consume the zmq event stream | 15:18 |
| jeblair | sdague: and then it can look at all the jobs it wants | 15:18 |
| sdague | but it can't look at their logs | 15:18 |
| jeblair | why not? | 15:18 |
| jeblair | _all_ logs are saved | 15:18 |
| sdague | even the reset ones? | 15:18 |
| jeblair | yep | 15:18 |
| *** therve has joined #openstack-infra | 15:19 | |
| sdague | we just don't link them anywhere? | 15:19 |
| jeblair | correct. you can still see them in the logs.o.o directory listings | 15:19 |
| *** Ryan_Lane has quit IRC | 15:20 | |
| fungi | and i believe you will still find them when querying logstash as well | 15:20 |
| sdague | ok, so can we change that, and actually report to gerrit when resets happen, and the location of the logs? | 15:20 |
| jeblair | sdague: i think we would be lynched. | 15:20 |
| fungi | i think that would generate waaaay lots of gerrit comment spam unless we held that until the end, in which case we'd end up with potentially over-limit comments instead | 15:21 |
| jeblair | the number of comments in gerrit would make it unusable... | 15:21 |
| sdague | it's not spam, it's real failures | 15:21 |
| jeblair | it's possible real failures, based on a future set of changes which may not merge... | 15:21 |
| sdague | we have to get out of this mindset that the fails need to be swept under the rug | 15:21 |
| sdague | at this point in time it's 90% real failures at least | 15:21 |
| jeblair | we're not sweeping them under the rug; they just aren't confirmed | 15:22 |
| sdague | they are made invisible | 15:22 |
| sdague | you can't confirm invisible things | 15:22 |
| sdague | and it makes people think the system is more reliable than it is | 15:23 |
| jeblair | i don't think anyone thinks the system is reliable | 15:23 |
| sdague | so attacking races doesn't get prioritized | 15:23 |
| mrodden | so Elastic Recheck is my new best buddy | 15:23 |
| *** Ryan_Lane has joined #openstack-infra | 15:24 | |
| jeblair | sdague: consider it this way: currently the only test run that matters as far as reporting to gerrit goes is the "last" one -- the one where the patch is at the head of the queue. with non-deterministic behavior, it has a certain % chance of merging. | 15:24 |
| *** senk has joined #openstack-infra | 15:25 | |
| *** basha has quit IRC | 15:25 | |
| *** datsun180b has joined #openstack-infra | 15:26 | |
| jeblair | sdague: your proposal is to make the _first_ test run the one that matters. if that first run hits a failure, the patch gets ejected. the same percentage chance applies there... | 15:26 |
| jeblair | sdague: except that if your patch has the bad luck to be at the end of the queue, it may get a chance to fail over and over again | 15:26 |
| fungi | this is why i was saying earlier that i don't think it will change the end result, statistically-speaking, other than to make the outcome less accurate | 15:27 |
| *** julim has quit IRC | 15:27 | |
| jeblair | sdague: so basically, you start with the same chance of failing or merging, and your proposal increases the chance of failure due to the length of the queue | 15:27 |
| jeblair | the length of the queue is not an indication of the quality of your patch | 15:27 |
| jeblair | so it's being tied to an irrelevant variable | 15:27 |
| fungi | increasingly so the longer the queue is ahead of you | 15:27 |
| fungi | yup, agreed | 15:27 |
| *** Cyril2 has joined #openstack-infra | 15:27 | |
| sdague | ok, fair | 15:27 |
| *** senk has quit IRC | 15:29 | |
| ttx | come on 42640, you can do it | 15:30 |
| ttx | we need some channel to place bets on merges in the gate | 15:30 |
| ttx | funnier that snail races | 15:30 |
| sdague | so I realize the crux of my problem is the lack of a promotion feature in the gate | 15:30 |
| sdague | if we have 50 changes in the gate queue, and we have a race fix, we're going to wait 8 hrs for a merge | 15:31 |
| jeblair | sdague: move this patch to top? | 15:31 |
| *** basha has joined #openstack-infra | 15:31 | |
| sdague | jeblair: yes | 15:31 |
| jeblair | sdague: yeah, that's high on my list | 15:31 |
| sdague | if we kicked fast, at least the gate queue would turn over in order quicker | 15:31 |
| sdague | so things would progress | 15:31 |
| sdague | or if we had some probably matrix, on a gate reset reorder based on chance to succeed | 15:31 |
| sdague | or lacking that, on smallest lines of change | 15:32 |
| jeblair | sdague: i think that an infra-root controlled method of promoting changes could be done relatively easily | 15:32 |
| *** fungi has quit IRC | 15:33 | |
| jeblair | creating a ui to deal with that is slightly more complicated (zuul's web ui is read-only at the moment; dealing with authn/z etc is a bit of overhead that needs to be written) | 15:33 |
| jeblair | we could continue down the 'magic incantations in gerrit comments' road, but that just feels weird. | 15:33 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add jobs for extra and dib repos https://review.openstack.org/49167 | 15:33 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove unused savanna rtfd jobs https://review.openstack.org/49168 | 15:33 |
| *** fungi has joined #openstack-infra | 15:33 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add support for multiple bug matches in a failure https://review.openstack.org/49171 | 15:34 |
| sdague | jeblair: what about another gerrit column | 15:34 |
| jeblair | i really wish we had gerrit buttons that just emitted events | 15:34 |
| jeblair | maybe that's worth thinking about as a gerrit plugin. | 15:34 |
| zul | mordred: ping so we are getting rid of d2to1 now? | 15:34 |
| mordred | zul: god yes. we've been geting rid of it for months | 15:36 |
| mordred | zul: upstream became unresponsive and there was a critical bug | 15:36 |
| *** basha has quit IRC | 15:36 | |
| mordred | zul: so we merged it into pbr and took over | 15:36 |
| zul | mordred: ah | 15:36 |
| jeblair | sdague: the column might work (i'm not sure if you can have a fully-optional column), but that feels weirder than magic comments (it's there all the time, unlike comments which are there only when you need them) | 15:36 |
| mordred | jeblair: I think magic comments sounds great | 15:36 |
| mordred | jeblair: also, yes to exploring gerrit plugins that make buttons that emit events -especially if we can acl those | 15:37 |
| jeblair | okay, i can try to get over my aversion to a magic-comment-based ui. :) | 15:38 |
| *** dkranz has quit IRC | 15:39 | |
| *** AlexF has quit IRC | 15:40 | |
| *** julim has joined #openstack-infra | 15:42 | |
| sdague | :) | 15:42 |
| sdague | so before adding more magic comments.... | 15:43 |
| sdague | we really need acls on magic comments | 15:43 |
| jeblair | sdague: i believe they exist | 15:43 |
| sdague | because with the abuse of recheck no bug, I really don't want to open up "jump the queue" to anyone | 15:43 |
| jeblair | sdague: i believe wikimedia added support for having zuul only process certain comments from certain email addresses | 15:44 |
| jeblair | sdague: also, i think you just picked the comment text. :) | 15:44 |
| openstackgerrit | A change was merged to openstack/requirements: Raise/Relax WebTest requirement to match Keystone https://review.openstack.org/48898 | 15:47 |
| openstackgerrit | A change was merged to openstack/requirements: os/requirements should be in sync with itself https://review.openstack.org/49064 | 15:47 |
| mordred | OMG. A change merged to requirements | 15:48 |
| mordred | and the WebTest change merged! | 15:48 |
| mordred | woot! | 15:48 |
| * mordred goes to see if a bunch of patches just got submitted by jenkins | 15:48 | |
| *** primemin1sterp has quit IRC | 15:49 | |
| jeblair | mordred: i don't think the post jobs have run for that yet | 15:49 |
| *** thomasbiege has joined #openstack-infra | 15:49 | |
| annegentle | mordred: ttx: no technical committee meeting today, right? | 15:49 |
| annegentle | and jeblair that was a funny buzzkill :) | 15:49 |
| *** svarnau has joined #openstack-infra | 15:50 | |
| annegentle | I could hear mordred's excitement deflating from here | 15:50 |
| *** primeministerp has joined #openstack-infra | 15:50 | |
| clarkb | poor mordred | 15:50 |
| ttx | annegentle: right | 15:50 |
| annegentle | ttx: ok thanks | 15:51 |
| ttx | heyy.. we might actually have a rc1 today. #20 and #31 in queue are icehouse openings. | 15:52 |
| ttx | (keystone & glance) | 15:52 |
| *** yassine has quit IRC | 15:52 | |
| *** dkranz has joined #openstack-infra | 15:52 | |
| fungi | finally got around to checking out what was going on with precise31, but was able to relaunch the slave agent on it and it ran jobs successfully, so no idea what was going on there | 15:53 |
| fungi | but apparently precise27 and 39 are dead, dead, deadski | 15:53 |
| fungi | gonna have to hard reboot them i think | 15:53 |
| fungi | ssh timeout | 15:53 |
| *** SergeyLukjanov has quit IRC | 15:54 | |
| *** dprince has quit IRC | 15:54 | |
| *** thomasbiege has quit IRC | 15:54 | |
| *** senk has joined #openstack-infra | 15:54 | |
| *** senk has quit IRC | 15:56 | |
| *** emagana has joined #openstack-infra | 15:58 | |
| jeblair | mordred: propose-requirements-updates: LOST | 15:58 |
| *** senk has joined #openstack-infra | 15:58 | |
| sdague | jeblair: so now that this is actually working - can you re +A it - https://review.openstack.org/#/c/49144/ ? | 15:59 |
| sdague | then I'll run off to lunch / biking for a bit and stop annoying folks for a while :) | 15:59 |
| jeblair | sdague: done! | 16:00 |
| *** matty_dubs is now known as matty_dubs|lunch | 16:01 | |
| *** adalbas has quit IRC | 16:02 | |
| jeblair | mordred: i think gearman-plugin may have missed it for some reason. i hit 'save' on the job in the jenkins ui and it's registered now. | 16:03 |
| mordred | jeblair: blast! | 16:03 |
| mordred | jeblair: can we inject a post event for that job? I'd love to see it run (and I really do need to actually learn how to do job injection) | 16:04 |
| jeblair | mordred: you could use the script in zuul/tools to manually submit the gearman job with the correct parameters | 16:04 |
| mordred | funny, you answered the question as I asked it | 16:04 |
| mordred | jeblair: do I connect to zuul.o.o and run it from there? | 16:04 |
| openstackgerrit | A change was merged to openstack-infra/config: add tenant isolation jobs to neutron patches https://review.openstack.org/49144 | 16:04 |
| fungi | mordred: from the shell on zuul, yes | 16:04 |
| jeblair | mordred: yeah, do it on-host on zuul.o.o; the script has --help to tell you the params to use | 16:04 |
| jeblair | mordred: and just look here for the values: https://jenkins.openstack.org/job/post-mirror-python33/130/parameters/? | 16:04 |
| fungi | mordred: you can look in my .bash_history to see recent invocations too | 16:05 |
| jeblair | mordred: and https://jenkins.openstack.org/job/post-mirror-python33/131/parameters/? | 16:05 |
| jeblair | (since there were 2 reqs changes) | 16:05 |
| mordred | ./trigger-job.py --job propose-requirements-updates -project openstack/requirements --pipeline post --refname master --oldrev 910f4f2d58bf3811d889d38b3e2aae575f83a13f --newrev 921c2f99afda3287d3530d5c45c93b598563f907 | 16:06 |
| mordred | does that look right? | 16:06 |
| mordred | --project | 16:06 |
| mordred | I probably don't need oldrev and newrev do it since it'sa post job | 16:07 |
| *** crank has joined #openstack-infra | 16:08 | |
| jeblair | mordred: i'd include them all | 16:11 |
| *** dkliban has quit IRC | 16:12 | |
| *** boris-42 has joined #openstack-infra | 16:12 | |
| mordred | jeblair: kk | 16:12 |
| jeblair | i'm pretty sure newrev is required | 16:12 |
| Ryan_Lane | https://review.openstack.org/#/c/47933/ <-- automatically abandoned :( | 16:12 |
| jeblair | Ryan_Lane: sorry about that; i'm not a fan of auto-abandon myself; the script never seems to know when people are really busy. | 16:14 |
| Ryan_Lane | heh | 16:14 |
| * Ryan_Lane nods | 16:14 | |
| Ryan_Lane | I'm turning into a true californian. I'm using passive aggressiveness to poke for reviews rather than a poke | 16:14 |
| *** thomasbiege has joined #openstack-infra | 16:15 | |
| *** adalbas has joined #openstack-infra | 16:15 | |
| *** osanchez has quit IRC | 16:15 | |
| *** mkoderer_ has joined #openstack-infra | 16:18 | |
| *** thomasbiege has quit IRC | 16:19 | |
| mordred | Ryan_Lane: watch out, they're going to take your Nola card away | 16:19 |
| *** SergeyLukjanov has joined #openstack-infra | 16:20 | |
| *** DinaBelova has joined #openstack-infra | 16:20 | |
| *** DinaBelova has quit IRC | 16:20 | |
| *** flaper87 is now known as flaper87|afk | 16:21 | |
| portante | mordred: do you know why "Jenkins" proposed a change in the Swift code base for various requirements? | 16:23 |
| jeblair | mordred: ! :) | 16:24 |
| mordred | portante: I do! | 16:24 |
| fungi | portante: success! | 16:24 |
| portante | oh good | 16:24 |
| mordred | portante: it's a new magical feature and this is its first incarnation of success! | 16:24 |
| portante | magical, oh, my, dragons, and castles, too? | 16:24 |
| portante | ;) | 16:24 |
| fungi | portante: url? want to see how it came out | 16:24 |
| mordred | portante: we have rolled out automation of proposing syncs with the global requirements list any time a change lands to said list | 16:24 |
| portante | neat | 16:25 |
| portante | sure, url would be great | 16:25 |
| mordred | https://review.openstack.org/#/q/topic:openstack/requirements,n,z | 16:25 |
| fungi | ahh, even better | 16:25 |
| jeblair | swift devs don't seem to be receiving it very well | 16:25 |
| mordred | nope | 16:25 |
| mordred | I guess we should have expected that | 16:25 |
| mordred | I think "LOL... no" pretty much sums it up | 16:26 |
| jeblair | mordred: it's always great to cap of months of collaborative work with someone laughing at you. | 16:26 |
| portante | ;) | 16:26 |
| portante | these things take time | 16:26 |
| mordred | no they don't | 16:26 |
| mordred | these things have been in exactly this state since the project began | 16:27 |
| *** DennyZhang has joined #openstack-infra | 16:28 | |
| *** DennyZhang has quit IRC | 16:28 | |
| portante | can you give "it" a name other than Jenkins when it posts a review, and possibly have it add an initial comment that describes what triggered it? | 16:29 |
| mordred | portante: it's pretty much identical to the translations update | 16:29 |
| mordred | I don't really think it needs a bunch of explanation - there is a global requirements list that the project has decided it will align to. this is a sync from that list. jenkins proposed the change. what more information would you want? | 16:30 |
| portante | when you say, "that the project" do you mean the individual project like swift, or the project meaning the greater openstack project? | 16:31 |
| mordred | OpenStack | 16:32 |
| mordred | swift is a sub-project of OpenStack | 16:32 |
| *** DennyZhang has joined #openstack-infra | 16:32 | |
| portante | okay | 16:32 |
| fungi | both precise27 and 39 wound up needing a hard reboot. 27 came up and was able to launch the slave agent and run jobs successfully, but i couldn't get the agent to launch on 39 until i deleted the slave from jenkins and re-added it from scratch | 16:35 |
| fungi | now it's running jobs successfully, and we're back up to our full compliment | 16:35 |
| jeblair | mordred: the requirements propose job starts at the same time that the mirror update job starts, yeah? that might cause a problem if proposed requirements changes can't actually fech their new requirements yet... | 16:36 |
| openstackgerrit | Diane Fleming proposed a change to openstack-infra/config: Updated api.yaml for the netconn-api v1.0 and v2.0 refs - pom location changed https://review.openstack.org/49030 | 16:36 |
| fungi | unless those jobs are exempted from the mirror selection | 16:36 |
| jeblair | fungi: i'm increasingly of the opinion that nodepool for everything is the way to deal with this... | 16:37 |
| fungi | jeblair: for slave failures? yes, agreed | 16:37 |
| jeblair | fungi: i don't think we want that; because we'd have to exempt _all_ jobs... and all jobs for all changes after them, etc... | 16:37 |
| * fungi nods... i wonder if we can tell that all three mirror jobs have completed before proposing the change | 16:38 | |
| *** DinaBelova has joined #openstack-infra | 16:38 | |
| *** Ajaeger1 has joined #openstack-infra | 16:38 | |
| fungi | job dependencies in zuul are limited to a tree for the moment, right? | 16:39 |
| mordred | I quit | 16:39 |
| mordred | portante, notmyname: I'm not going to talk to the other swift devs anymore. I will happily talk to you | 16:39 |
| mordred | it makes me too angry | 16:39 |
| mordred | jeblair: good point | 16:40 |
| fungi | portante had great constructive comments on https://review.openstack.org/49206 | 16:42 |
| mordred | awesome | 16:42 |
| mordred | I honestly do prefer constructive comments. thank you portante | 16:42 |
| fungi | which ultimately should be addressed with a change to openstack/requirements and another to openstack/swift i think | 16:42 |
| fungi | also i think this underscores a strength of this update mechanism (as we saw with translations to some degree too)... control over how and when to approve the change continues to rest with the devs on each of the projects being integrated | 16:44 |
| mordred | fungi: ++ | 16:44 |
| fungi | so it's not like release or infra are cramming these commits down any project's throat | 16:45 |
| fungi | ceilometer already approved their update | 16:48 |
| *** matty_dubs|lunch is now known as matty_dubs | 16:49 | |
| fungi | i'd be a little worried that there could be some resistance to the 4-line setup.py getting an hp copyright entry, though it doesn't particularly bother me | 16:52 |
| *** derekh has quit IRC | 16:52 | |
| *** tkammer has quit IRC | 16:52 | |
| *** dkliban has joined #openstack-infra | 16:53 | |
| *** emagana has quit IRC | 16:55 | |
| hub_cap | fungi: mordred one less thing for me to do, +2! | 16:58 |
| *** hashar has quit IRC | 16:58 | |
| zaro | seems like the ci-puppetmaster.o.o and puppet-dashboard.o.o are down? | 16:58 |
| hub_cap | and itll show me when i have requirements that arent in global (mordred i thought u pushed reviews for pexpect?) | 16:59 |
| hub_cap | https://review.openstack.org/#/c/49207/1/requirements.txt | 16:59 |
| *** emagana has joined #openstack-infra | 17:00 | |
| *** davidhadas has quit IRC | 17:00 | |
| *** julim_ has joined #openstack-infra | 17:00 | |
| *** Ryan_Lane has quit IRC | 17:01 | |
| *** davidhadas has joined #openstack-infra | 17:01 | |
| *** DennyZhang has quit IRC | 17:02 | |
| *** julim has quit IRC | 17:03 | |
| mordred | hub_cap: I did - but it hasn't landed | 17:04 |
| hub_cap | okey mordred | 17:05 |
| mordred | hub_cap: (so don't +2 it yet) | 17:05 |
| *** afazekas_no_irq has quit IRC | 17:05 | |
| hub_cap | looks like the review caused some issues w/ our shiz and cant be merged, so 1) ill fix that code in question, and 2) ill update the requirements as well | 17:05 |
| mordred | hub_cap: that said - I am hoping this will bring to people's attention when openstack/requirements is incompatible with their project | 17:05 |
| mordred | hub_cap: thank you! | 17:05 |
| hub_cap | is ther a way to retrigger the jenkins job? | 17:05 |
| hub_cap | once ive cleaned up | 17:05 |
| hub_cap | and yes im very happy that i dont have to see that my shits not inline w/ global- anymore | 17:06 |
| hub_cap | ^ ^ by some manual process | 17:06 |
| uvirtbot | hub_cap: Error: "^" is not a valid command. | 17:06 |
| mordred | portante: do you think adding a paragraph to the commit message about where the list came from and to propose changes to it if an entry here seems wrong would be helpful? | 17:06 |
| hub_cap | u virt bot i hate u | 17:06 |
| *** fbo is now known as fbo_away | 17:06 | |
| mordred | hub_cap: yah. just recheck no bug the change | 17:06 |
| hub_cap | sweet! and itll regen the diffs too??? | 17:08 |
| *** reed has joined #openstack-infra | 17:08 | |
| hub_cap | mordred: im renaming my project openstack-trove ;) | 17:08 |
| zaro | fungi: do you know why those urls are not resolving? they are referenced in the openstack wiki. | 17:08 |
| mordred | hub_cap: nope. the diffs will be the diffs | 17:09 |
| mordred | hub_cap: but if you update global requirements, it will submit a new patch to that change | 17:09 |
| hub_cap | ok cool, that makes sense for the pexpect thing | 17:09 |
| hub_cap | but wrt to my failures cuz maybe we arent using teh right lib (lets say the mock version, which is a issue w/ our client), and i go and make that change in my review to fix it | 17:10 |
| fungi | zaro: the puppet-dashboard is being rebuilt (anteaya is still working on it). the puppet master has not had a reachable web interface for as long as i've been on the project | 17:11 |
| *** mrmartin has joined #openstack-infra | 17:11 | |
| mordred | hub_cap: so, in that case, just put the relevant requirement fix in your requirements file in your change | 17:11 |
| mordred | hub_cap: git should be able to figure out the merge | 17:11 |
| hub_cap | oh duh | 17:12 |
| * hub_cap overlooks the obvious | 17:12 | |
| anteaya | zaro: you can see the beta puppet-dashboard here: http://15.185.153.222/ | 17:12 |
| anteaya | I'm working on puppeting it to bring it into the fold | 17:13 |
| anteaya | fold as in a place to keep sheep safe | 17:13 |
| fungi | clearly we need more sheep | 17:13 |
| hub_cap | flock: openstack sheep-aas | 17:14 |
| hub_cap | lol flock? wtf i need breakfast | 17:14 |
| anteaya | hub_cap: and some vowels | 17:14 |
| * anteaya hands hub_cap some vowels | 17:14 | |
| hub_cap | <3 | 17:15 |
| anteaya | :D | 17:15 |
| * fungi imagines a flock of sheep flying overhead | 17:16 | |
| anteaya | with no limits | 17:16 |
| fungi | flock of pigs works better, i think | 17:16 |
| *** gyee has joined #openstack-infra | 17:17 | |
| jog0 | the bots are taking over: https://review.openstack.org/#/c/49195/ | 17:18 |
| jog0 | mordred: ^ | 17:19 |
| Alex_Gaynor | `hah | 17:19 |
| anteaya | maybe but if the pigs are flying, ttx is where a Hawaiian shirt and posing for a portrait | 17:20 |
| anteaya | autograph pen at the ready | 17:20 |
| pleia2 | flying sheep? it's only tuesday :) | 17:20 |
| hub_cap | http://3.bp.blogspot.com/_moh5MaCXCBw/SVBGf63ZJWI/AAAAAAAAAt4/GlI0pes8LXE/S1600-R/gir.pig.flying.jpg | 17:20 |
| anteaya | *wearing | 17:21 |
| clarkb | jog0: I think that means we can just drink beer all day. THe bots are doing the real work | 17:22 |
| mordred | jog0: haha | 17:22 |
| *** hogepodge has joined #openstack-infra | 17:23 | |
| *** sandywalsh has quit IRC | 17:23 | |
| fungi | ahh, gir, you never fail to amuse me | 17:24 |
| jeblair | jog0: that is a riot! | 17:24 |
| fungi | it's almost as if they're having a conversation | 17:25 |
| fungi | some projects may have been too eager to approve the requirements sync... https://review.openstack.org/49189 | 17:27 |
| *** prad has joined #openstack-infra | 17:29 | |
| jog0 | we have created a monster | 17:29 |
| *** nati_ueno has joined #openstack-infra | 17:33 | |
| *** Cyril2 has quit IRC | 17:34 | |
| *** ArxCruz has quit IRC | 17:34 | |
| zaro | anteaya: thnx. looks very nice. | 17:35 |
| *** ArxCruz has joined #openstack-infra | 17:38 | |
| anteaya | thansk | 17:40 |
| anteaya | thanks | 17:40 |
| *** jerryz has joined #openstack-infra | 17:41 | |
| hub_cap | jog0: its only a monster when it starts -1'ing | 17:44 |
| openstackgerrit | Matthew Treinish proposed a change to openstack/requirements: Add tempest to projects list https://review.openstack.org/49225 | 17:46 |
| jog0 | hub_cap: as long as we don't have +2 bots ... | 17:47 |
| fungi | jog0: i may be a +2 bot ;) | 17:48 |
| dims | lol | 17:49 |
| *** hogepodge has quit IRC | 17:54 | |
| *** DinaBelova has quit IRC | 17:55 | |
| *** hogepodge has joined #openstack-infra | 17:56 | |
| *** tvb|afk has joined #openstack-infra | 17:57 | |
| *** chuckieb has joined #openstack-infra | 17:59 | |
| *** chuckieb|2 has joined #openstack-infra | 18:02 | |
| *** sarob has joined #openstack-infra | 18:03 | |
| *** DinaBelova has joined #openstack-infra | 18:04 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: don't look for things only in FAILURE logs https://review.openstack.org/49228 | 18:04 |
| *** chuckieb has quit IRC | 18:05 | |
| mriedem | sdague: i had to read that commit message a few times ^ :) | 18:05 |
| sdague | heh | 18:05 |
| *** gyee has quit IRC | 18:11 | |
| *** ericw has quit IRC | 18:11 | |
| mriedem | mordred: hey, hate to bug you, just wondering about this? https://review.openstack.org/#/c/49006/ | 18:12 |
| mriedem | maybe automate force merge for the quantumclient branch? :) | 18:13 |
| *** chuckieb has joined #openstack-infra | 18:16 | |
| openstackgerrit | David Peraza proposed a change to openstack/requirements: Adding sqlalchemy db2 dialect dependencies https://review.openstack.org/48745 | 18:17 |
| mordred | mriedem: sorry - I'll get to it in just a little bit :( | 18:19 |
| mriedem | mordred: no problem, i saw something about beer drinking and i'm assuming hell raising | 18:19 |
| *** davidhadas has quit IRC | 18:20 | |
| *** chuckieb|2 has quit IRC | 18:20 | |
| Ajaeger1 | Hi Infra team, is there a chance to get https://review.openstack.org/#/c/47691/ and https://review.openstack.org/#/c/47897/ reviewed and approved, please? | 18:26 |
| *** mrmartin_ has joined #openstack-infra | 18:28 | |
| *** mrmartin has quit IRC | 18:30 | |
| clarkb | Ajaeger1: reviewed, thank you for the new patchset | 18:30 |
| *** rnirmal has joined #openstack-infra | 18:31 | |
| Ajaeger1 | clarkb: thanks for the review | 18:31 |
| *** chuckieb has quit IRC | 18:32 | |
| *** mrmartin__ has joined #openstack-infra | 18:36 | |
| *** mrmartin_ has quit IRC | 18:39 | |
| *** Ryan_Lane has joined #openstack-infra | 18:40 | |
| *** qba73 has joined #openstack-infra | 18:41 | |
| Ryan_Lane | mordred: heh, for real. I need to be way more aggressive to keep that | 18:41 |
| *** DinaBelova has quit IRC | 18:44 | |
| *** osanchez has joined #openstack-infra | 18:52 | |
| *** ryanpetrello has quit IRC | 18:53 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct unattended upgrades origins assignment https://review.openstack.org/49235 | 18:53 |
| *** thingee_zzz is now known as thingee | 18:54 | |
| annegentle | pleia2: around? Could I get the text/slides of your workshop you did? Looking for more inspiration | 18:55 |
| pleia2 | annegentle: the only slides I had were the generic ones I pulled from /marketing so folks knew what openstack was | 18:56 |
| *** osanchez has quit IRC | 18:57 | |
| *** OlivierSanchez has joined #openstack-infra | 18:57 | |
| pleia2 | annegentle: http://princessleia.com/journal/?p=8526 has some more details about what I did, plus the handout that had some manual nova commands I had people try after I walked them through basics of starting a VM in horizon | 18:57 |
| pleia2 | http://princessleia.com/presentations/CodeChixworkshopOpenStackDocumentation.pdf is the handout, you're welcome to use/modify/whatever | 18:58 |
| *** derekh has joined #openstack-infra | 18:58 | |
| jeblair | it's meeting time! | 18:59 |
| lifeless | in 60 seconds :P | 18:59 |
| annegentle | pleia2: thank you! | 18:59 |
| pleia2 | annegentle: you're welcome, happy to help | 18:59 |
| *** jerryz has quit IRC | 18:59 | |
| *** jerryz has joined #openstack-infra | 19:00 | |
| * pleia2 puts on 2 meetings hat | 19:00 | |
| annegentle | pleia2: oh how cool you put the PDF of the Ops Guide on there, great thinking | 19:00 |
| pleia2 | annegentle: yeah, I figured it was the best "hit the ground running" docs we have | 19:00 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Disable the salt minion data cache on master https://review.openstack.org/49238 | 19:04 |
| *** melwitt has joined #openstack-infra | 19:05 | |
| *** che-arne has quit IRC | 19:06 | |
| *** ryanpetrello has joined #openstack-infra | 19:09 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add support for multiple bug matches in a failure https://review.openstack.org/49171 | 19:10 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Make the salt master also a minion https://review.openstack.org/49240 | 19:10 |
| *** davidhadas has joined #openstack-infra | 19:10 | |
| *** DinaBelova has joined #openstack-infra | 19:11 | |
| *** saper_ has joined #openstack-infra | 19:18 | |
| *** locke1051 has joined #openstack-infra | 19:19 | |
| *** annegentle_ has joined #openstack-infra | 19:21 | |
| *** fboz has joined #openstack-infra | 19:22 | |
| *** fboz is now known as fbo | 19:22 | |
| *** jeblair_ has joined #openstack-infra | 19:22 | |
| *** niska` has joined #openstack-infra | 19:24 | |
| *** jeblair has quit IRC | 19:24 | |
| *** DinaBelova has quit IRC | 19:24 | |
| *** jeblair_ is now known as jeblair | 19:24 | |
| *** GheRiver1 has joined #openstack-infra | 19:25 | |
| *** ttx` has joined #openstack-infra | 19:25 | |
| *** adam_g` has joined #openstack-infra | 19:26 | |
| *** lillie has joined #openstack-infra | 19:26 | |
| *** lillie is now known as Guest74104 | 19:26 | |
| *** locke105 has quit IRC | 19:27 | |
| *** adam_g has quit IRC | 19:27 | |
| *** annegentle has quit IRC | 19:27 | |
| *** GheRivero has quit IRC | 19:27 | |
| *** olaph has quit IRC | 19:27 | |
| *** oubiwann has quit IRC | 19:27 | |
| *** niska has quit IRC | 19:27 | |
| *** fbo_away has quit IRC | 19:27 | |
| *** dtroyer has quit IRC | 19:27 | |
| *** EmilienM has quit IRC | 19:27 | |
| *** saper has quit IRC | 19:27 | |
| *** Guest31783 has quit IRC | 19:27 | |
| *** ttx has quit IRC | 19:27 | |
| *** Adri2000 has quit IRC | 19:27 | |
| *** davidhadas has quit IRC | 19:27 | |
| *** DinaBelova has joined #openstack-infra | 19:28 | |
| *** davidhadas has joined #openstack-infra | 19:28 | |
| *** ttx` is now known as ttx | 19:28 | |
| *** adam_g` is now known as adam_g | 19:28 | |
| *** ttx has quit IRC | 19:28 | |
| *** ttx has joined #openstack-infra | 19:28 | |
| *** oubiwann has joined #openstack-infra | 19:28 | |
| *** oubiwann is now known as Guest49600 | 19:29 | |
| *** EmilienM has joined #openstack-infra | 19:30 | |
| *** dtroyer has joined #openstack-infra | 19:31 | |
| *** dizquierdo has left #openstack-infra | 19:32 | |
| *** vipul is now known as vipul-away | 19:35 | |
| *** davidhadas has quit IRC | 19:35 | |
| *** davidhadas has joined #openstack-infra | 19:36 | |
| *** Ryan_Lane has quit IRC | 19:37 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: don't look for things only in FAILURE logs https://review.openstack.org/49228 | 19:37 |
| *** olaph has joined #openstack-infra | 19:38 | |
| *** dcramer_ has quit IRC | 19:43 | |
| *** hashar has joined #openstack-infra | 19:43 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add check_success tool https://review.openstack.org/49246 | 19:44 |
| *** davidhadas has quit IRC | 19:45 | |
| *** davidhadas has joined #openstack-infra | 19:45 | |
| *** w_ has joined #openstack-infra | 19:52 | |
| *** olaph has quit IRC | 19:54 | |
| ttx | that "elastic recheck" dude is smarter than most our devs | 19:54 |
| fungi | at least they haven't been replaced by a small shell script. more like a modest-sized python script | 19:54 |
| *** emagana has quit IRC | 19:54 | |
| anteaya | ttx just handing out the compliments today | 19:54 |
| ttx | anteaya: I get more grumpy as we get closer to release | 19:55 |
| anteaya | oh goody | 19:55 |
| anteaya | I have more to look forward to | 19:55 |
| ttx | anteaya: my contract says I should be like that | 19:55 |
| anteaya | ha ha ha | 19:55 |
| pleia2 | hah | 19:55 |
| jog0 | ttx: https://review.openstack.org/#/c/49195/ the bots are taking over | 19:55 |
| anteaya | good thing you are complying with your contract | 19:55 |
| *** cody-somerville has quit IRC | 19:56 | |
| ttx | jog0: now if the bots could fix the gate-fcking issues, we'd live in a perfect world | 19:56 |
| jog0 | amen! | 19:57 |
| ttx | Keystone RC1 just delayed another 5/6 hours, thanks to bug 1230407 | 19:58 |
| uvirtbot | Launchpad bug 1230407 in neutron "VMs can't progress through state changes because Neutron is deadlocking on it's database queries, and thus leaving networks in inconsistent states" [Critical,Confirmed] https://launchpad.net/bugs/1230407 | 19:58 |
| *** boris-42 has quit IRC | 19:58 | |
| anteaya | :( | 19:58 |
| ttx | Glance RC1 might hit in 35min in the best case scenario | 19:59 |
| *** boris-42 has joined #openstack-infra | 19:59 | |
| anteaya | go go Glance RC1 | 20:00 |
| * anteaya hands ttx a cat for petting as good luck | 20:00 | |
| *** DinaBelova has quit IRC | 20:01 | |
| *** thingee is now known as thingee_zzz | 20:02 | |
| anteaya | ttx this would be your Glance patch you are waiting on? https://review.openstack.org/#/c/49029/ | 20:03 |
| ttx | anteaya: don't name it ! Bad luck ! | 20:05 |
| *** vipul-away is now known as vipul | 20:05 | |
| * anteaya throws salt over her shoulder and swings a chicken | 20:06 | |
| *** cody-somerville has joined #openstack-infra | 20:06 | |
| *** cody-somerville has quit IRC | 20:06 | |
| *** cody-somerville has joined #openstack-infra | 20:06 | |
| *** enikanorov-w has quit IRC | 20:06 | |
| *** derekh has quit IRC | 20:07 | |
| *** enikanorov-w has joined #openstack-infra | 20:08 | |
| *** rnirmal has quit IRC | 20:08 | |
| lifeless | ttx: hey | 20:09 |
| lifeless | ttx: I will send a mail on this but | 20:09 |
| lifeless | ttx: whats your view on an API (currently on stackforge) doing releases | 20:09 |
| lifeless | ttx: can we do it, does it interfere with incubation, what do we need to do over and above what we'd do for regular projects? | 20:10 |
| ttx | lifeless: you should be able to do whatever you want | 20:10 |
| ttx | lifeless: devil is in the details of course :) | 20:10 |
| jeblair | lifeless: what api do you have on stackforge? didn't we move tripleo to openstack/ ? | 20:10 |
| lifeless | jeblair: tuskar | 20:11 |
| lifeless | jeblair: I plan to ask for the those projects to move once they have solid CI in place | 20:11 |
| lifeless | jeblair: (which means tempest etc IMO, and that probably comes behind the bm testing story given what tuskar does) | 20:12 |
| jeblair | lifeless: did tripleo have a scope expansion? | 20:13 |
| lifeless | jeblair: Not AFAIK | 20:13 |
| jeblair | lifeless: but tuskar repos are now part of tripleo? | 20:13 |
| ttx | jeblair: tuskar could actually be seen as part of the tripleO mission. | 20:13 |
| lifeless | jeblair: so tuskar fits under the original tripleo mission | 20:14 |
| lifeless | jeblair: deployment isn't 'get the code on machines' - thats not hard and not that interesting | 20:14 |
| jeblair | lifeless: it's okay, you don't have to convince me. :) | 20:14 |
| lifeless | jeblair: sure :) just answering the questions | 20:15 |
| mordred | jeblair: answer is "yes, tuskar is managed by tripleo program now" | 20:15 |
| lifeless | jeblair: we did propose and then announce after discussion on the -dev list | 20:15 |
| lifeless | jeblair: should we have copied -infra? | 20:15 |
| jeblair | i guess i didn't expect a program to absorb a new project without a tc discussion, but that was clearly a lack of imagination on my part. :) | 20:16 |
| lifeless | jeblair: (I'd like to avoid surprising you with this sort of thing) | 20:16 |
| jeblair | lifeless: well, we do normally like to help :) | 20:16 |
| jeblair | lifeless: we should get the launchpad projects normalized and go ahead and move the git repos | 20:16 |
| jeblair | lifeless: we just scheduled a repo move this weekend for savanna | 20:16 |
| lifeless | jeblair: ok, cool - that would be great | 20:16 |
| mordred | jeblair: if I was smarter, I would have brought up tuskar when you asked that question | 20:17 |
| lifeless | we have no plans to rename them at the moment | 20:17 |
| lifeless | just shift their parent dior | 20:17 |
| lifeless | dir | 20:17 |
| mordred | right. that's a rename for us | 20:17 |
| jeblair | we could probably lump tuskar in too, if it's ready; but i don't want to rush you if you want to think about renaming them or anything. but hey you just answered that. :) | 20:17 |
| lifeless | sure, I was heading off questions about the final name | 20:17 |
| lifeless | will you be sending announcements about the renames happening, or should I inform folk specifically ? | 20:18 |
| lifeless | and thanks | 20:18 |
| jeblair | lifeless: i'll send an announcement | 20:18 |
| mordred | lifeless: what would be great is if someone from tripleo could make a patch to infra renaming from stackforge/tuskar* to openstack/tuskar* - so that we make sure that we get everything that's appropriate | 20:19 |
| *** julim_ has quit IRC | 20:19 | |
| mordred | lifeless: keep in mind that once the repos move under openstack/ they are subject automatically to the restricted mirror and openstack/requirements | 20:19 |
| jeblair | lifeless: oct 5 1600 utc is the current scheduled time; devs will need to update git remotes then | 20:19 |
| jeblair | mordred, lifeless: oh, yes, if the mirror/reqs would be a problem, we can delay until that's ready. | 20:20 |
| *** dmakogon_ has joined #openstack-infra | 20:20 | |
| *** johnthetubaguy has quit IRC | 20:20 | |
| SergeyLukjanov | btw the only requirement that we have in savanna that is not in openstack/requirements is python-savannaclient | 20:22 |
| *** vipul is now known as vipul-away | 20:24 | |
| lifeless | jeblair: I think we should wear the pain up front | 20:24 |
| lifeless | I will mail the list now to alert folk | 20:24 |
| freyes | hashar: hi, if you have some time to review my updated patch set (https://review.openstack.org/#/c/48661) would be great | 20:25 |
| mordred | SergeyLukjanov: we should probably go ahead and add a patch for that to o/r | 20:25 |
| *** vipul-away is now known as vipul | 20:25 | |
| SergeyLukjanov | mordred, yeah, I'll do it now | 20:25 |
| hashar | freyes: not tonight sorry :/ | 20:25 |
| mordred | hashar, mgagne, zaro, jeblair: did you see darraghb's question about jjb from last night? | 20:25 |
| freyes | hashar: no worries :) | 20:25 |
| ryanpetrello | so in light of last week's pecan/wsme fiasco | 20:25 |
| mordred | 20:22:52 mordred | having them have relationships, and also being able to upload things to places and then promote them | 20:26 |
| mordred | 20:23:11 mordred | so, I think it might be worthwhile thinking about, especially as we think about multi-branch mirrors | 20:26 |
| mordred | 20:23:31 mordred | it also has an offline mode where it's fine serving what it's already got if it can't talk to the network upstream | 20:26 |
| ryanpetrello | we were wanting to set up something in CI that would gate pecan and wsme against eachother | 20:26 |
| hashar | freyes: I noticed some activity, I might have commented one or two time but not that much :/ Been quite a busy day :) | 20:26 |
| mgagne | mordred: no, I was out of the office. IRC? review? | 20:26 |
| ryanpetrello | I know we do this for official openstack projects | 20:26 |
| ryanpetrello | anybody know how this works? | 20:26 |
| mordred | mgagne: hashar 04:50:15 darraghb | query on https://review.openstack.org/#/c/45165/, should the JJB code default to generating all XML elements even if the default is | 20:27 |
| mordred | | blank/no-value, or should it default to the plugin behaviour on which XML tags to set? | 20:27 |
| mgagne | mordred: checking | 20:27 |
| hashar | freyes: ah your is actually quite simple. let me look | 20:27 |
| clarkb | ryanpetrello: setup a job that tehy share, when you do that zuul treats them as dependent (the job needs to be added to the gate pipeline) | 20:27 |
| clarkb | ryanpetrello: you can look at gate-tempest-devstack-vm-full for an example | 20:27 |
| freyes | hashar: indeed :) only trailing white spaces removal | 20:27 |
| hashar | mordred: regarding job, I am not sure whether we want to output empty XML element. I think there is a bunch of existing modules that do that already. | 20:27 |
| mordred | ryanpetrello: the trick is basically to have a snippet of code which checks out multiple repos of code appropriately - let me point you at the relevent bit from devstack-gate... | 20:28 |
| hashar | freyes: look at my cover message :-) The code from tests/base.py is copy pasted from tests/publishers/test_publishers.py | 20:28 |
| hashar | freyes: so you will want to make tests/publishers/test_publishers.py to rely on the base class you introduced. | 20:29 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Allow use of python-savannaclient-0.3a3 https://review.openstack.org/49252 | 20:29 |
| freyes | hashar: yes, I want to make it reusable, I have another patch to add some tests | 20:29 |
| jeblair | mordred, ryanpetrello: having that bit of code be a generic slave script that accepted a project list as arguments might be useful. | 20:29 |
| mordred | jeblair: I was just thinking the same thing | 20:30 |
| ryanpetrello | +1 | 20:30 |
| freyes | hashar: ok, I'll refactor it, but I'll split my patch, one to add base.py, another one for git shallow clone | 20:30 |
| mordred | ryanpetrello: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n71 | 20:30 |
| mordred | there is the non-generic part | 20:30 |
| hashar | freyes: that would be ideal! | 20:30 |
| mordred | but even it isn't bad | 20:30 |
| freyes | hashar: super, thanks for reviewing it | 20:30 |
| mordred | ryanpetrello: $PROJECTS is pre-set to the list of projects to clone | 20:30 |
| mordred | ryanpetrello: in your case, it would be PROJECTS="stackforge/wsme stackforge/pecan" | 20:30 |
| ryanpetrello | right | 20:31 |
| ryanpetrello | okay, thanks! | 20:31 |
| hashar | freyes: and thank you for enhancing the tests! | 20:31 |
| *** tvb|afk has quit IRC | 20:31 | |
| mordred | ryanpetrello: I'm running around a little crazy today - but would be happy to help you sort out doing that - because I agree with jeblair having a more cookie-cutter answer to this question would be a nice building block | 20:31 |
| *** sarob has quit IRC | 20:31 | |
| ryanpetrello | mordred: yea, that would be awesome | 20:32 |
| *** sarob has joined #openstack-infra | 20:32 | |
| *** flaper87|afk is now known as flaper87 | 20:32 | |
| hashar | freyes: I will have a look at JJB changes tomorrow. Thx :) | 20:33 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document bootstrapping of Gerrit ACLs. https://review.openstack.org/45011 | 20:33 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add jobs for extra and dib repos https://review.openstack.org/49167 | 20:36 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Move savanna under openstack org https://review.openstack.org/48491 | 20:36 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove unused savanna rtfd jobs https://review.openstack.org/49168 | 20:36 |
| *** sarob has quit IRC | 20:36 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Docs on bringing up Jenkins in new infrastructures. https://review.openstack.org/45216 | 20:38 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document spinning up a derived zuul. https://review.openstack.org/45164 | 20:38 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document running custom slaves in ones own infra. https://review.openstack.org/45346 | 20:38 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Non-openstack-ci support for launch/dns.py. https://review.openstack.org/44980 | 20:38 |
| *** Ryan_Lane has joined #openstack-infra | 20:38 | |
| lifeless | ok thats super weird | 20:39 |
| lifeless | I pushed up a change | 20:39 |
| *** dcramer_ has joined #openstack-infra | 20:39 | |
| lifeless | I9bb037c31e192f1d719cfdd3a1d59053eba0c5b0 | 20:39 |
| lifeless | and gerrit doesn't see it | 20:39 |
| lifeless | but it's at the bottom of the stack I just updated | 20:39 |
| *** emagana has joined #openstack-infra | 20:40 | |
| *** thomasm has quit IRC | 20:40 | |
| fungi | lifeless: supposed to be the parent of Ic1d5a511 i take it? | 20:42 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Add savanna to tracked projects list https://review.openstack.org/49254 | 20:42 |
| lifeless | yeah | 20:42 |
| fungi | 35cfe04c4d82fc89cadd5e7ff73690d7d1858aa8Move tuskar to openstack. | 20:42 |
| *** qba73 has quit IRC | 20:42 | |
| lifeless | thats the one | 20:43 |
| fungi | gerrit doesn't report a change with that sha... a draft perhaps? | 20:43 |
| lifeless | NFI | 20:44 |
| lifeless | I'll make a spurious change to it and repush | 20:44 |
| fungi | i'll see if i can find it in the db | 20:44 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Move tuskar to openstack. https://review.openstack.org/49255 | 20:45 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Docs on bringing up Jenkins in new infrastructures. https://review.openstack.org/45216 | 20:45 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document spinning up a derived zuul. https://review.openstack.org/45164 | 20:45 |
| lifeless | sorry about the noise.. | 20:45 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document running custom slaves in ones own infra. https://review.openstack.org/45346 | 20:45 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Non-openstack-ci support for launch/dns.py. https://review.openstack.org/44980 | 20:45 |
| fungi | normally gerrit wouldn't allow you to parent a change on a ref it doesn't know | 20:45 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document bootstrapping of Gerrit ACLs. https://review.openstack.org/45011 | 20:45 |
| lifeless | there it goes | 20:45 |
| lifeless | I think I ticked a bug | 20:45 |
| lifeless | I will describe it in LP for you to consider | 20:45 |
| fungi | you wouldn't be the first ;) | 20:45 |
| fungi | yeah, says patchset 1, so wasn't a draft, at least not with that change-id | 20:46 |
| * mordred blames SpamapS | 20:47 | |
| *** thingee_zzz is now known as thingee | 20:47 | |
| mordred | SpamapS is better than anyone else at doing bizarre things with patch uploads | 20:47 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Docs on bringing up Jenkins in new infrastructures. https://review.openstack.org/45216 | 20:48 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document spinning up a derived zuul. https://review.openstack.org/45164 | 20:48 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document running custom slaves in ones own infra. https://review.openstack.org/45346 | 20:48 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Non-openstack-ci support for launch/dns.py. https://review.openstack.org/44980 | 20:48 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document bootstrapping of Gerrit ACLs. https://review.openstack.org/45011 | 20:48 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Move tuskar to openstack. https://review.openstack.org/49255 | 20:48 |
| *** gyee has joined #openstack-infra | 20:50 | |
| openstackgerrit | A change was merged to openstack-infra/config: Use Jenkins templates for old manual jobs https://review.openstack.org/47691 | 20:51 |
| lifeless | https://bugs.launchpad.net/openstack-ci/+bug/1233846 | 20:51 |
| lifeless | fungi: ^ | 20:51 |
| uvirtbot | Launchpad bug 1233846 in openstack-ci "pushing a new commit under an abandonded one lead to a phantom base" [Undecided,New] | 20:51 |
| lifeless | I think that should be reproducable | 20:51 |
| fungi | haha, yes possibly | 20:52 |
| fungi | this is likely something reported/fixed upstream after 2.4 | 20:52 |
| *** sarob has joined #openstack-infra | 20:52 | |
| *** dkliban has quit IRC | 20:53 | |
| mordred | fungi: one of these days we'll get to upgrade, and it's going to be sexy | 20:54 |
| lifeless | mordred: jeblair: anyhow, https://review.openstack.org/49255 - the tuskar move for you | 20:55 |
| *** Ryan_Lane has quit IRC | 20:56 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Publish atom.xml for manuals https://review.openstack.org/47897 | 20:58 |
| *** dizquierdo has joined #openstack-infra | 20:58 | |
| *** w_ is now known as olaph | 21:00 | |
| hashar | lifeless: thanks for the doc update :-] | 21:01 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Create new gates for openstack-manuals https://review.openstack.org/48128 | 21:01 |
| *** thomasm has joined #openstack-infra | 21:02 | |
| *** harlowja has quit IRC | 21:02 | |
| * hashar waves | 21:02 | |
| lifeless | hashar: o/ | 21:03 |
| *** weshay has quit IRC | 21:04 | |
| *** dcramer_ has quit IRC | 21:04 | |
| lifeless | sorry | 21:04 |
| *** hashar has quit IRC | 21:04 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Docs on bringing up Jenkins in new infrastructures. https://review.openstack.org/45216 | 21:04 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document spinning up a derived zuul. https://review.openstack.org/45164 | 21:04 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document running custom slaves in ones own infra. https://review.openstack.org/45346 | 21:04 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Non-openstack-ci support for launch/dns.py. https://review.openstack.org/44980 | 21:04 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Document bootstrapping of Gerrit ACLs. https://review.openstack.org/45011 | 21:04 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Move tuskar to openstack. https://review.openstack.org/49255 | 21:04 |
| *** Ajaeger1 has quit IRC | 21:06 | |
| *** rfolco has quit IRC | 21:06 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Bug 1230407 was at least two problems, split apart. https://review.openstack.org/49261 | 21:08 |
| uvirtbot | Launchpad bug 1230407 in neutron "VMs can't progress through state changes because Neutron is deadlocking on it's database queries, and thus leaving networks in inconsistent states" [Critical,Confirmed] https://launchpad.net/bugs/1230407 | 21:08 |
| *** gyee has quit IRC | 21:08 | |
| *** OlivierSanchez has quit IRC | 21:10 | |
| *** gyee has joined #openstack-infra | 21:11 | |
| *** mrmartin__ has quit IRC | 21:11 | |
| *** OlivierSanchez has joined #openstack-infra | 21:15 | |
| *** julim has joined #openstack-infra | 21:18 | |
| *** jcoufal has joined #openstack-infra | 21:19 | |
| *** weshay has joined #openstack-infra | 21:19 | |
| jog0 | looking for a review for https://review.openstack.org/#/c/49261/ | 21:21 |
| jog0 | elastic-recheck | 21:21 |
| jog0 | sdague clarkb mtreinish ^ | 21:21 |
| *** jpich has quit IRC | 21:22 | |
| sdague | jog0: got my +2 | 21:22 |
| mtreinish | jog0: where are the flake8 results :) | 21:22 |
| jog0 | mtreinish: good question | 21:26 |
| *** Adri2000 has joined #openstack-infra | 21:27 | |
| *** ryanpetrello has quit IRC | 21:29 | |
| *** che-arne has joined #openstack-infra | 21:30 | |
| *** flaper87 is now known as flaper87|afk | 21:30 | |
| jog0 | mordred: https://review.openstack.org/#/c/49179/1 | 21:32 |
| jog0 | looks like that needs to be pushed everywhere | 21:32 |
| mordred | jog0: ++ | 21:33 |
| jog0 | thats your bread and butter want to blast it out | 21:33 |
| *** dkranz has quit IRC | 21:33 | |
| *** alcabrera has quit IRC | 21:36 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add the extra neutron jobs to neutronclient changes https://review.openstack.org/49263 | 21:36 |
| sdague | mordred, jeblair can you look at that ^^^ | 21:37 |
| sdague | it would be helpful in knowing if we fixed the neutronclient race issue with nova vs. neutron | 21:38 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add mox fixture to baste TestCase https://review.openstack.org/49264 | 21:38 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Make test_required_files.py real unit tests https://review.openstack.org/49265 | 21:38 |
| sdague | otherwise we are rechecking a lot and getting a small amount of data each go | 21:38 |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Moved get_scenarios() function to a base module to make it easier to reuse https://review.openstack.org/48661 | 21:42 |
| *** harlowja has joined #openstack-infra | 21:42 | |
| mriedem | can anyone tell me if this is a 'recheck no bug'? looks like a pypi hiccup: https://review.openstack.org/#/c/48739/ | 21:43 |
| fungi | mtreinish: really? we have baste test cases now? ;) (returning the nitpick favor) | 21:43 |
| *** changbl has quit IRC | 21:43 | |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Moved get_scenarios() to base module to make it easier to reuse https://review.openstack.org/48661 | 21:43 |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Moved get_scenarios() function to a base module to make it easier to reuse https://review.openstack.org/49267 | 21:44 |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Added support for Git shallow clone parameter https://review.openstack.org/49268 | 21:44 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make test_queries save stdout in tox testr https://review.openstack.org/49080 | 21:44 |
| mordred | fungi: you should ALWAYS test the base | 21:44 |
| mordred | damn | 21:44 |
| mordred | couldn't even do the joke | 21:44 |
| mordred | fungi: you should ALWAYS test the BASTE | 21:44 |
| mtreinish | fungi: heh, oops | 21:45 |
| fungi | mriedem: the devstack failures on that change do not look like pypi download problems | 21:46 |
| mtreinish | although I like baste vs base, it's grown on me | 21:46 |
| mriedem | fungi: sdague pointed out it's this: https://bugs.launchpad.net/openstack-ci/+bug/1232592 | 21:46 |
| uvirtbot | Launchpad bug 1232592 in openstack-ci "pypi CDN bugs cause SSL errors when trying to install packages" [Undecided,New] | 21:46 |
| *** dizquierdo has left #openstack-infra | 21:46 | |
| fungi | mriedem: yes, tehre's a cdn problem which causes trouble retrieving thrift, from what we've seen, but it's only one of the three failures on that test run | 21:47 |
| fungi | the two devstack-tempest job failures on it are unrelated to that issue | 21:47 |
| mriedem | fungi: yeah, the others are this: https://bugs.launchpad.net/tempest/+bug/1233743 | 21:48 |
| uvirtbot | Launchpad bug 1233743 in tempest "TestStampPattern.test_stamp_pattern fails with timeout" [Undecided,New] | 21:48 |
| mriedem | i didn't really want to recheck on that since i know that's not the issue | 21:48 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Make test_required_files.py real unit tests https://review.openstack.org/49265 | 21:48 |
| fungi | mriedem: i'd probably recheck with that bug number rather than no bug, since at least one of the failures causing you to recheck does have an actual bug associated with it | 21:48 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add mox fixture to base TestCase https://review.openstack.org/49264 | 21:48 |
| mriedem | i mean, given the change | 21:48 |
| mriedem | fungi: yeah, definitely, it just wasn't the best | 21:49 |
| mriedem | which is why i asked | 21:49 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Bug 1230407 was at least two problems, split apart. https://review.openstack.org/49261 | 21:49 |
| uvirtbot | Launchpad bug 1230407 in neutron "VMs can't progress through state changes because Neutron is deadlocking on it's database queries, and thus leaving networks in inconsistent states" [Critical,Confirmed] https://launchpad.net/bugs/1230407 | 21:49 |
| *** matty_dubs is now known as matty_dubs|gone | 21:51 | |
| *** SergeyLukjanov has quit IRC | 21:54 | |
| *** johnthetubaguy has joined #openstack-infra | 21:55 | |
| anteaya | so send this one through the queue again with reverify bug #1233743? | 21:57 |
| uvirtbot | Launchpad bug 1233743 in tempest "TestStampPattern.test_stamp_pattern fails with timeout" [Undecided,New] https://launchpad.net/bugs/1233743 | 21:57 |
| anteaya | https://review.openstack.org/#/c/49029/ | 21:57 |
| *** cody-somerville has quit IRC | 21:57 | |
| *** SergeyLukjanov has joined #openstack-infra | 21:58 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Remove gate-django_openstack_auth-noop job. https://review.openstack.org/49274 | 21:59 |
| openstackgerrit | A change was merged to openstack-infra/config: add the extra neutron jobs to neutronclient changes https://review.openstack.org/49263 | 21:59 |
| anteaya | well I did it, hope it was right | 22:00 |
| *** pcm_ has quit IRC | 22:00 | |
| *** johnthetubaguy has quit IRC | 22:01 | |
| *** SergeyLukjanov has quit IRC | 22:01 | |
| anteaya | ttx :( | 22:01 |
| *** thomasm has quit IRC | 22:01 | |
| ttx | <- sad panda | 22:02 |
| anteaya | yeah | 22:02 |
| *** nati_ueno has quit IRC | 22:02 | |
| anteaya | no kidding | 22:02 |
| anteaya | ttx so you are traveling the end of this week? | 22:03 |
| ttx | I'm at a conference in Paris Thursday afternoon /Friday morning | 22:04 |
| anteaya | nice | 22:04 |
| ttx | and in a train Thusrday morning and Friday afternoon | 22:04 |
| anteaya | okay so you will have time to send an email kicking off the TC nomination process on the Friday? | 22:05 |
| ttx | anteaya: I can certainly do it, but wouldn't mind if you did | 22:05 |
| anteaya | I can do it | 22:05 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Make test_required_files.py real unit tests https://review.openstack.org/49265 | 22:05 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add mox fixture to base TestCase https://review.openstack.org/49264 | 22:05 |
| ttx | anteaya: that would be one less thing I need to think about. would be great :) | 22:06 |
| anteaya | can you email me the salient points and I'll do a first draft? | 22:06 |
| anteaya | np | 22:06 |
| anteaya | or just pull from the wikipage? | 22:06 |
| annegentle_ | hi all, is there a sandbox repo for practicing with git review or did I imagine it? | 22:06 |
| ttx | anteaya: it's pretty much the same as the PTL election, pull from the wiki page and send me a draft if you want to be doubleplussure | 22:06 |
| anteaya | I want to be doubleplussure so I will | 22:07 |
| anteaya | and the following week I will basically be offline from Wednesday afternoon onward | 22:07 |
| anteaya | flying to Budapest and then first day of the conf on the Friday | 22:07 |
| anteaya | so starting the elections is all you, my friend | 22:08 |
| ttx | anteaya: will do | 22:08 |
| *** boris-42 has quit IRC | 22:08 | |
| anteaya | great | 22:09 |
| *** esker has quit IRC | 22:09 | |
| *** mriedem has quit IRC | 22:09 | |
| anteaya | annegentle_: I do think review-dev is the server you are looking for: http://git.openstack.org/cgit/openstack-infra/config/tree/manifests/site.pp#n45 | 22:10 |
| *** esker has joined #openstack-infra | 22:10 | |
| *** cody-somerville has joined #openstack-infra | 22:11 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Cleanup tests https://review.openstack.org/48764 | 22:13 |
| *** esker has quit IRC | 22:14 | |
| *** nati_ueno has joined #openstack-infra | 22:15 | |
| *** Reapster_ is now known as Reapster | 22:16 | |
| *** rcleere has quit IRC | 22:17 | |
| *** michchap has quit IRC | 22:17 | |
| annegentle_ | anteaya: SWEET | 22:17 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Enforce E501 max-line-length at 100 chars https://review.openstack.org/48763 | 22:18 |
| *** michchap has joined #openstack-infra | 22:18 | |
| anteaya | annegentle_: \o/ | 22:20 |
| *** datsun180b has quit IRC | 22:20 | |
| *** MarkAtwood has joined #openstack-infra | 22:23 | |
| *** thedodd has quit IRC | 22:25 | |
| *** flaper87|afk is now known as flaper87 | 22:27 | |
| fungi | annegentle_: openstack-dev/sandbox | 22:28 |
| fungi | also | 22:28 |
| *** luhrs1 has joined #openstack-infra | 22:30 | |
| *** jerryz has quit IRC | 22:31 | |
| *** emagana has quit IRC | 22:31 | |
| *** emagana has joined #openstack-infra | 22:32 | |
| *** che-arne has quit IRC | 22:32 | |
| *** mriedem has joined #openstack-infra | 22:32 | |
| *** markmcclain has quit IRC | 22:33 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1232748 https://review.openstack.org/49280 | 22:34 |
| uvirtbot | Launchpad bug 1232748 in nova "[Postgres] DB error: (OperationalError) could not translate host name "localhost" to address" [Undecided,Confirmed] https://launchpad.net/bugs/1232748 | 22:34 |
| *** ryanpetrello has joined #openstack-infra | 22:39 | |
| *** nati_ueno has quit IRC | 22:45 | |
| *** vipul is now known as vipul-away | 22:47 | |
| *** vipul-away is now known as vipul | 22:47 | |
| *** OlivierSanchez has quit IRC | 22:48 | |
| *** ryanpetrello has quit IRC | 22:50 | |
| *** dmakogon_ has quit IRC | 22:50 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Cleanup README https://review.openstack.org/49281 | 22:51 |
| *** sarob_ has joined #openstack-infra | 22:54 | |
| *** emagana has quit IRC | 22:55 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add support for multiple bug matches in a failure https://review.openstack.org/49171 | 22:55 |
| openstackgerrit | A change was merged to openstack/requirements: Align our setup.py with ourselves https://review.openstack.org/49059 | 22:55 |
| *** shardy is now known as shardy_afk | 22:55 | |
| *** emagana has joined #openstack-infra | 22:55 | |
| *** sarob has quit IRC | 22:58 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add requirements-install to watch test runs https://review.openstack.org/49283 | 22:58 |
| *** sarob_ has quit IRC | 22:59 | |
| clarkb | fyi I am restarting elastic-recheck on logstash.o.o at jog0's requiest. it apparently got stuck | 22:59 |
| sdague | cool | 22:59 |
| clarkb | jog0 is working on a work around and I am aware of a larger issue in how we are (not) indexing console.html completely all the time | 22:59 |
| sdague | we probably need a heartbeating mechanism on the bot so puppet can know it's alive | 23:00 |
| *** emagana has quit IRC | 23:00 | |
| *** sarob has joined #openstack-infra | 23:01 | |
| *** jcoufal has quit IRC | 23:05 | |
| *** nati_ueno has joined #openstack-infra | 23:08 | |
| clarkb | restarted. This looks like potentially a larger problem (due to asynchronous console.html copies) I will keep brainstorming for a better solution to it (current hacks are work arounds) | 23:12 |
| *** dims has quit IRC | 23:13 | |
| *** reed has quit IRC | 23:13 | |
| *** sarob has quit IRC | 23:15 | |
| *** sarob has joined #openstack-infra | 23:15 | |
| *** MarkAtwood has quit IRC | 23:16 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Stop using While True to wait for ElasticSearch https://review.openstack.org/48766 | 23:18 |
| *** slong has joined #openstack-infra | 23:21 | |
| *** prad has quit IRC | 23:22 | |
| *** alexpilotti has quit IRC | 23:23 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Increase log worker retry timeout length. https://review.openstack.org/49289 | 23:23 |
| *** dims has joined #openstack-infra | 23:27 | |
| *** flaper87 is now known as flaper87|afk | 23:30 | |
| *** vipul is now known as vipul-away | 23:38 | |
| *** sarob has quit IRC | 23:38 | |
| *** sarob has joined #openstack-infra | 23:39 | |
| *** vipul-away is now known as vipul | 23:41 | |
| *** AlexF has joined #openstack-infra | 23:42 | |
| *** sarob has quit IRC | 23:43 | |
| *** rfolco has joined #openstack-infra | 23:47 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Inspect the Gearman queue for immediate demand https://review.openstack.org/48517 | 23:50 |
| jeblair | clarkb, fungi, mordred: it's possible that's slightly mind-bending. | 23:51 |
| *** DennyZhang has joined #openstack-infra | 23:52 | |
| jeblair | clarkb, fungi, mordred: i'm sorry about that if it is. | 23:52 |
| *** nati_ueno has quit IRC | 23:54 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Inspect the Gearman queue for immediate demand https://review.openstack.org/48517 | 23:54 |
| *** rfolco has quit IRC | 23:56 | |
| *** jerryz has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
