| *** yamamoto has joined #openstack-infra | 00:01 | |
| *** unicell1 has joined #openstack-infra | 00:02 | |
| *** davideagnello has joined #openstack-infra | 00:02 | |
| *** unicell has quit IRC | 00:03 | |
| *** markvoelker has quit IRC | 00:03 | |
| *** yamamoto has quit IRC | 00:05 | |
| *** davideagnello has quit IRC | 00:07 | |
| *** sputnik13 has joined #openstack-infra | 00:07 | |
| *** sarob has joined #openstack-infra | 00:14 | |
| *** hamzy has joined #openstack-infra | 00:16 | |
| *** yamamoto has joined #openstack-infra | 00:22 | |
| *** emagana has joined #openstack-infra | 00:23 | |
| *** dboik has joined #openstack-infra | 00:25 | |
| *** dboik_ has joined #openstack-infra | 00:26 | |
| *** sarob has quit IRC | 00:27 | |
| *** Krinkle is now known as Krinkle|detached | 00:27 | |
| *** emagana has quit IRC | 00:29 | |
| *** emagana has joined #openstack-infra | 00:29 | |
| *** dboik has quit IRC | 00:30 | |
| *** emagana has quit IRC | 00:35 | |
| *** teran has joined #openstack-infra | 00:38 | |
| *** teran has quit IRC | 00:44 | |
| *** Guest46804 has quit IRC | 00:48 | |
| *** yamamoto has quit IRC | 00:50 | |
| *** tiswanso has joined #openstack-infra | 00:56 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Switch check-functional-dsvm-magnum on for every check review https://review.openstack.org/170755 | 00:58 |
|---|---|---|
| *** achanda has joined #openstack-infra | 00:59 | |
| *** tiswanso_ has joined #openstack-infra | 01:00 | |
| *** markvoelker has joined #openstack-infra | 01:00 | |
| *** xianghui has quit IRC | 01:02 | |
| *** tiswanso has quit IRC | 01:02 | |
| *** Hal has joined #openstack-infra | 01:03 | |
| *** baoli has joined #openstack-infra | 01:03 | |
| *** Hal is now known as Guest27148 | 01:03 | |
| *** xianghui has joined #openstack-infra | 01:04 | |
| *** markvoelker has quit IRC | 01:04 | |
| *** baoli has quit IRC | 01:14 | |
| *** baoli has joined #openstack-infra | 01:14 | |
| *** dangers_away is now known as dangers | 01:16 | |
| *** zwq432 has quit IRC | 01:17 | |
| *** emagana has joined #openstack-infra | 01:18 | |
| *** YorikSar has quit IRC | 01:21 | |
| *** emagana has quit IRC | 01:23 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Switch check-functional-dsvm-magnum on for every check review https://review.openstack.org/170755 | 01:25 |
| *** baoli has quit IRC | 01:27 | |
| *** achanda has quit IRC | 01:33 | |
| *** achanda has joined #openstack-infra | 01:38 | |
| *** yamamoto_ has joined #openstack-infra | 01:39 | |
| *** teran has joined #openstack-infra | 01:39 | |
| *** paul-- has quit IRC | 01:39 | |
| *** teran has quit IRC | 01:43 | |
| *** otter768 has joined #openstack-infra | 01:44 | |
| *** dimsum__ has quit IRC | 01:46 | |
| *** sputnik13 has quit IRC | 01:46 | |
| *** otter768 has quit IRC | 01:48 | |
| *** jogo has joined #openstack-infra | 01:53 | |
| *** jogo has quit IRC | 01:54 | |
| *** julim_ has quit IRC | 01:55 | |
| *** sputnik13 has joined #openstack-infra | 01:55 | |
| *** jogo has joined #openstack-infra | 01:56 | |
| *** sarob has joined #openstack-infra | 01:57 | |
| *** jogo has quit IRC | 01:57 | |
| *** markvoelker has joined #openstack-infra | 02:00 | |
| *** sarob has quit IRC | 02:02 | |
| *** baoli has joined #openstack-infra | 02:04 | |
| *** markvoelker has quit IRC | 02:05 | |
| *** koolhead17 has joined #openstack-infra | 02:06 | |
| *** flashgordon is now known as jogo | 02:08 | |
| *** baoli has quit IRC | 02:09 | |
| *** jogo has quit IRC | 02:10 | |
| *** jogo has joined #openstack-infra | 02:10 | |
| *** jogo is now known as flashgordon | 02:11 | |
| *** jogo has joined #openstack-infra | 02:12 | |
| *** jogo has quit IRC | 02:12 | |
| *** jogo has joined #openstack-infra | 02:12 | |
| *** emagana has joined #openstack-infra | 02:12 | |
| *** amotoki has joined #openstack-infra | 02:13 | |
| *** emagana has quit IRC | 02:17 | |
| *** sputnik13 has quit IRC | 02:19 | |
| *** woodster_ has joined #openstack-infra | 02:20 | |
| *** sputnik13 has joined #openstack-infra | 02:23 | |
| *** sputnik13 has quit IRC | 02:26 | |
| *** sarob has joined #openstack-infra | 02:32 | |
| *** sarob has quit IRC | 02:37 | |
| *** baoli has joined #openstack-infra | 02:37 | |
| openstackgerrit | Boris Bobrov proposed openstack/requirements: Limit version of python-memcached https://review.openstack.org/170762 | 02:37 |
| openstackgerrit | Boris Bobrov proposed openstack/requirements: Limit version of python-memcached https://review.openstack.org/170762 | 02:38 |
| *** dkranz has quit IRC | 02:39 | |
| *** teran has joined #openstack-infra | 02:40 | |
| *** sarob has joined #openstack-infra | 02:42 | |
| *** baoli has quit IRC | 02:42 | |
| *** sarob has quit IRC | 02:42 | |
| *** teran has quit IRC | 02:44 | |
| *** dimsum__ has joined #openstack-infra | 02:46 | |
| *** sarob has joined #openstack-infra | 02:49 | |
| *** vinkman has joined #openstack-infra | 02:50 | |
| *** dimsum__ has quit IRC | 02:52 | |
| *** otter768 has joined #openstack-infra | 02:53 | |
| *** emagana has joined #openstack-infra | 02:54 | |
| *** vinkman has quit IRC | 02:54 | |
| *** sarob has quit IRC | 02:56 | |
| *** jogo has quit IRC | 02:56 | |
| *** jogo has joined #openstack-infra | 02:57 | |
| *** jogo has quit IRC | 02:57 | |
| *** jogo has joined #openstack-infra | 02:57 | |
| *** vinkman has joined #openstack-infra | 02:57 | |
| *** jogo has quit IRC | 02:57 | |
| *** emagana has quit IRC | 02:59 | |
| *** chlong has joined #openstack-infra | 03:00 | |
| *** markvoelker has joined #openstack-infra | 03:01 | |
| *** vinkman has quit IRC | 03:02 | |
| *** koolhead17 has quit IRC | 03:04 | |
| *** jogo has joined #openstack-infra | 03:04 | |
| *** markvoelker has quit IRC | 03:06 | |
| *** tiswanso_ has quit IRC | 03:20 | |
| *** emagana has joined #openstack-infra | 03:23 | |
| *** coolsvap|afk is now known as coolsvap | 03:29 | |
| *** emagana has quit IRC | 03:32 | |
| *** changbl has joined #openstack-infra | 03:32 | |
| *** achanda has quit IRC | 03:33 | |
| *** otter768 has quit IRC | 03:34 | |
| *** teran has joined #openstack-infra | 03:41 | |
| *** teran has quit IRC | 03:46 | |
| *** koolhead17 has joined #openstack-infra | 03:47 | |
| *** sputnik13 has joined #openstack-infra | 03:52 | |
| *** markvoelker has joined #openstack-infra | 04:02 | |
| *** markvoelker has quit IRC | 04:07 | |
| *** ajmiller has quit IRC | 04:08 | |
| *** koolhead17 has quit IRC | 04:16 | |
| *** achanda has joined #openstack-infra | 04:24 | |
| openstackgerrit | Clark Laughlin proposed openstack-infra/puppet-jenkins: Change installation of the JDK to use the architecture of the host https://review.openstack.org/168222 | 04:29 |
| *** dangers is now known as dangers_away | 04:40 | |
| *** changbl has quit IRC | 04:40 | |
| *** imcsk8 has quit IRC | 04:41 | |
| *** imcsk8 has joined #openstack-infra | 04:41 | |
| *** teran has joined #openstack-infra | 04:41 | |
| *** teran has quit IRC | 04:47 | |
| *** unicell has joined #openstack-infra | 05:01 | |
| *** unicell1 has quit IRC | 05:01 | |
| *** markvoelker has joined #openstack-infra | 05:03 | |
| *** changbl has joined #openstack-infra | 05:05 | |
| *** cody-somerville has joined #openstack-infra | 05:09 | |
| *** hdd has joined #openstack-infra | 05:10 | |
| *** Longgeek has joined #openstack-infra | 05:14 | |
| *** davideagnello has joined #openstack-infra | 05:19 | |
| *** deepakcs has joined #openstack-infra | 05:25 | |
| *** mriedem1 has joined #openstack-infra | 05:28 | |
| *** wenlock has joined #openstack-infra | 05:30 | |
| *** mriedem has quit IRC | 05:31 | |
| *** coolsvap is now known as coolsvap|afk | 05:33 | |
| *** mriedem has joined #openstack-infra | 05:34 | |
| *** changbl has quit IRC | 05:34 | |
| *** otter768 has joined #openstack-infra | 05:35 | |
| *** mriedem1 has quit IRC | 05:36 | |
| *** otter768 has quit IRC | 05:39 | |
| *** woodster_ has quit IRC | 05:40 | |
| *** davideag_ has joined #openstack-infra | 05:44 | |
| *** davideagnello has quit IRC | 05:45 | |
| *** coolsvap has joined #openstack-infra | 05:47 | |
| *** Longgeek has quit IRC | 05:47 | |
| openstackgerrit | Clark Laughlin proposed openstack-infra/system-config: Install puppet from the Ubuntu apt repository for aarch64/arm64 https://review.openstack.org/170607 | 05:54 |
| openstackgerrit | Clark Laughlin proposed openstack-infra/system-config: Do not install linux-headers-virtual for trusty or later https://review.openstack.org/170776 | 05:55 |
| *** mase_home has joined #openstack-infra | 05:57 | |
| *** emagana has joined #openstack-infra | 06:06 | |
| *** yamamoto_ has quit IRC | 06:07 | |
| *** yamamoto_ has joined #openstack-infra | 06:07 | |
| *** davideag_ has quit IRC | 06:10 | |
| *** emagana has quit IRC | 06:11 | |
| *** hdd has quit IRC | 06:19 | |
| *** Longgeek has joined #openstack-infra | 06:24 | |
| *** Abhi_ has joined #openstack-infra | 06:24 | |
| Abhi_ | Hi all, in my ci setup the zuul hangs after 10 min, can anyone help me to resolve this issue?? | 06:25 |
| *** coolsvap has quit IRC | 06:25 | |
| *** tnovacik has joined #openstack-infra | 06:34 | |
| *** coolsvap has joined #openstack-infra | 06:38 | |
| *** dhritishikhar_ has joined #openstack-infra | 06:40 | |
| *** dhritishikhar_ is now known as dhritishikhar___ | 06:40 | |
| *** jamielennox|away is now known as jamielennox | 06:48 | |
| *** subscope has joined #openstack-infra | 06:50 | |
| *** teran has joined #openstack-infra | 06:53 | |
| *** dhritishikhar___ has quit IRC | 06:58 | |
| *** jpeeler has joined #openstack-infra | 07:02 | |
| *** markvoelker has quit IRC | 07:04 | |
| *** dizquierdo has joined #openstack-infra | 07:10 | |
| *** YorikSar has joined #openstack-infra | 07:13 | |
| *** mase_home has quit IRC | 07:14 | |
| *** HeOS has joined #openstack-infra | 07:16 | |
| *** sergsh has joined #openstack-infra | 07:17 | |
| *** andreykurilin_ has joined #openstack-infra | 07:20 | |
| *** andreykurilin_ has quit IRC | 07:24 | |
| *** andreykurilin_ has joined #openstack-infra | 07:24 | |
| *** yfried is now known as yfried|afk | 07:26 | |
| *** achanda has quit IRC | 07:28 | |
| *** vlaza has joined #openstack-infra | 07:30 | |
| *** yfried|afk is now known as yfried | 07:32 | |
| *** e0ne has joined #openstack-infra | 07:35 | |
| *** otter768 has joined #openstack-infra | 07:36 | |
| *** achanda has joined #openstack-infra | 07:39 | |
| *** otter768 has quit IRC | 07:40 | |
| *** dhritishikhar_ has joined #openstack-infra | 07:45 | |
| *** yfried is now known as yfried|afk | 07:45 | |
| *** dizquierdo has quit IRC | 07:50 | |
| *** yfried|afk is now known as yfried | 07:50 | |
| *** _nadya_ has joined #openstack-infra | 07:51 | |
| *** _nadya_ has quit IRC | 07:53 | |
| *** teran_ has joined #openstack-infra | 07:54 | |
| *** emagana has joined #openstack-infra | 07:54 | |
| *** teran has quit IRC | 07:55 | |
| *** davideagnello has joined #openstack-infra | 07:58 | |
| *** emagana has quit IRC | 07:59 | |
| *** tkelsey has joined #openstack-infra | 07:59 | |
| *** davideagnello has quit IRC | 08:03 | |
| *** markvoelker has joined #openstack-infra | 08:03 | |
| *** teran_ has quit IRC | 08:04 | |
| *** tkelsey has quit IRC | 08:04 | |
| *** markvoelker has quit IRC | 08:08 | |
| *** sergsh has quit IRC | 08:08 | |
| *** yfried is now known as yfried|afk | 08:09 | |
| *** wicope has joined #openstack-infra | 08:10 | |
| *** wicope has joined #openstack-infra | 08:11 | |
| *** wicope has joined #openstack-infra | 08:11 | |
| *** e0ne has quit IRC | 08:12 | |
| *** wicope has joined #openstack-infra | 08:12 | |
| *** kiseok7_ has joined #openstack-infra | 08:16 | |
| *** andreykurilin_ has quit IRC | 08:16 | |
| *** e0ne has joined #openstack-infra | 08:17 | |
| *** ajo has joined #openstack-infra | 08:17 | |
| *** e0ne has quit IRC | 08:20 | |
| *** sergsh has joined #openstack-infra | 08:21 | |
| *** therve has quit IRC | 08:21 | |
| *** pal has joined #openstack-infra | 08:23 | |
| *** therve has joined #openstack-infra | 08:25 | |
| *** dhritishikhar___ has joined #openstack-infra | 08:27 | |
| *** achanda has quit IRC | 08:27 | |
| *** yfried|afk is now known as yfried | 08:30 | |
| *** Guest27148 has quit IRC | 08:31 | |
| *** Abhi_ has quit IRC | 08:33 | |
| *** dhritishikhar_ has quit IRC | 08:34 | |
| *** dhritishikhar___ has quit IRC | 08:34 | |
| *** dhritishikhar_ has joined #openstack-infra | 08:38 | |
| *** dizquierdo has joined #openstack-infra | 08:44 | |
| *** yfried has quit IRC | 08:48 | |
| *** emagana has joined #openstack-infra | 08:49 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Independent provider manager for slow tasks https://review.openstack.org/170788 | 08:52 |
| *** emagana has quit IRC | 08:53 | |
| *** teran has joined #openstack-infra | 08:54 | |
| *** hichihara has quit IRC | 08:54 | |
| *** chlong_ has joined #openstack-infra | 08:54 | |
| *** coolsvap has quit IRC | 08:55 | |
| *** YorikSar has quit IRC | 08:57 | |
| *** coolsvap has joined #openstack-infra | 08:57 | |
| *** chlong has quit IRC | 08:57 | |
| *** tkelsey has joined #openstack-infra | 08:58 | |
| *** e0ne has joined #openstack-infra | 09:00 | |
| *** e0ne is now known as e0ne_ | 09:01 | |
| *** markvoelker has joined #openstack-infra | 09:04 | |
| *** dhritishikhar_ has quit IRC | 09:05 | |
| *** dhritishikhar_ has joined #openstack-infra | 09:05 | |
| *** e0ne_ is now known as e0ne | 09:08 | |
| *** markvoelker has quit IRC | 09:08 | |
| *** skolekonov has joined #openstack-infra | 09:10 | |
| *** gulic has joined #openstack-infra | 09:11 | |
| *** Hal has joined #openstack-infra | 09:12 | |
| dhritishikhar_ | asettle, ping | 09:12 |
| *** dizquierdo has quit IRC | 09:12 | |
| *** Hal is now known as Guest41672 | 09:12 | |
| *** chandankumar_ has joined #openstack-infra | 09:14 | |
| *** dhritishikhar_ has quit IRC | 09:17 | |
| *** dhritishikhar_ has joined #openstack-infra | 09:18 | |
| *** YorikSar has joined #openstack-infra | 09:21 | |
| *** dizquierdo has joined #openstack-infra | 09:24 | |
| *** coolsvap has quit IRC | 09:30 | |
| *** Guest41672 has quit IRC | 09:34 | |
| *** otter768 has joined #openstack-infra | 09:37 | |
| *** dizquierdo has quit IRC | 09:39 | |
| *** otter768 has quit IRC | 09:41 | |
| *** emagana has joined #openstack-infra | 09:43 | |
| *** Longgeek has quit IRC | 09:43 | |
| *** Longgeek has joined #openstack-infra | 09:43 | |
| *** coolsvap has joined #openstack-infra | 09:45 | |
| *** emagana has quit IRC | 09:47 | |
| *** yamamoto_ has quit IRC | 09:50 | |
| *** dizquierdo has joined #openstack-infra | 09:51 | |
| *** chandankumar_ has quit IRC | 09:53 | |
| *** chandankumar has joined #openstack-infra | 09:53 | |
| *** coolsvap has quit IRC | 09:55 | |
| *** dimsum__ has joined #openstack-infra | 09:56 | |
| *** Hal has joined #openstack-infra | 10:03 | |
| *** Hal is now known as Guest6333 | 10:04 | |
| *** markvoelker has joined #openstack-infra | 10:04 | |
| *** tkelsey has quit IRC | 10:06 | |
| *** coolsvap has joined #openstack-infra | 10:07 | |
| *** Guest6333 has quit IRC | 10:09 | |
| *** markvoelker has quit IRC | 10:09 | |
| *** pc_m has joined #openstack-infra | 10:13 | |
| *** yamamoto has joined #openstack-infra | 10:26 | |
| *** Hal has joined #openstack-infra | 10:27 | |
| *** Hal is now known as Guest66209 | 10:28 | |
| *** Guest66209 has quit IRC | 10:32 | |
| *** ZZelle_ has joined #openstack-infra | 10:34 | |
| *** dboik_ has quit IRC | 10:37 | |
| *** wenlock has quit IRC | 10:40 | |
| *** chlong_ has quit IRC | 10:41 | |
| *** mattymo has joined #openstack-infra | 10:50 | |
| mattymo | ping infra:) https://review.openstack.org/#/c/155309/ was marked +2 and workflow +1, but it didn't get merged | 10:50 |
| *** _nadya_ has joined #openstack-infra | 10:53 | |
| *** chlong_ has joined #openstack-infra | 10:58 | |
| *** _nadya_ has quit IRC | 10:58 | |
| *** dhritishikhar_ has quit IRC | 10:59 | |
| *** yamamoto has quit IRC | 11:04 | |
| *** chlong_ has quit IRC | 11:05 | |
| *** markvoelker has joined #openstack-infra | 11:05 | |
| *** markvoelker has quit IRC | 11:10 | |
| samueldmq | morning | 11:10 |
| *** marcusvrn has joined #openstack-infra | 11:11 | |
| *** yamamoto has joined #openstack-infra | 11:14 | |
| *** e0ne is now known as e0ne_ | 11:14 | |
| *** e0ne_ is now known as e0ne | 11:16 | |
| sdague | fungi: when you get up, it looks like zuul isn't queueing things at the moment | 11:17 |
| sdague | I just pushed a few patches up, and they are aren't showing up in zuul | 11:18 |
| *** Longgeek has quit IRC | 11:20 | |
| *** doug-fish has joined #openstack-infra | 11:22 | |
| *** dhritishikhar_ has joined #openstack-infra | 11:22 | |
| *** chlong has joined #openstack-infra | 11:24 | |
| *** eantyshev has joined #openstack-infra | 11:25 | |
| eantyshev | Hi! We use Jenkins 1.580.2 & JJB in our third-party CI and it doesn't publish artifacts in case the build fails. I have no clue how to change this | 11:28 |
| eantyshev | Openstack Jenkins has version of 1.565.3, and there is no such problem there, AFAIU | 11:30 |
| fungi | sdague: thanks and ugh. betting it's gerrit again | 11:31 |
| *** emagana has joined #openstack-infra | 11:31 | |
| sdague | fungi: ok, it stopped working about 45 minutes ago IIRC | 11:33 |
| sdague | I was in the middle of doing things with the trove complex depends-on chain, don't know if that is related or not | 11:33 |
| sdague | but the coincidence is probably worth noting | 11:33 |
| fungi | 2015-04-06 10:56:24,267 DEBUG gerrit.GerritWatcher: Received data from Gerrit event stream | 11:34 |
| fungi | that's the last thing its seen on the gerrit debug stream | 11:34 |
| *** emagana has quit IRC | 11:35 | |
| *** pal has quit IRC | 11:36 | |
| fungi | 100+ stream-events tasks waiting since 10:56:30 in the gerrit show-queue output | 11:36 |
| fungi | restarting gerrit now | 11:36 |
| sdague | fungi: cool | 11:37 |
| *** otter768 has joined #openstack-infra | 11:37 | |
| fungi | #status notice gerrit has been restarted to restore event streaming. any change events missed by zuul (between 10:56 and 11:37 utc) will need to be rechecked or have new approval votes set | 11:38 |
| openstackstatus | fungi: sending notice | 11:38 |
| *** ldnunes has joined #openstack-infra | 11:38 | |
| -openstackstatus- NOTICE: gerrit has been restarted to restore event streaming. any change events missed by zuul (between 10:56 and 11:37 utc) will need to be rechecked or have new approval votes set | 11:38 | |
| *** Longgeek has joined #openstack-infra | 11:39 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard-webclient: Added possibility to scroll events https://review.openstack.org/170820 | 11:40 |
| openstackstatus | fungi: finished sending notice | 11:40 |
| sdague | fungi: does that mean a bunch of rechecks ? | 11:41 |
| sdague | oh, never mind | 11:41 |
| *** otter768 has quit IRC | 11:42 | |
| openstackgerrit | Merged openstack-infra/storyboard: Fixed import https://review.openstack.org/168467 | 11:43 |
| *** dhritishikhar___ has joined #openstack-infra | 11:47 | |
| *** dhritishikhar_ has quit IRC | 11:49 | |
| *** dizquierdo has quit IRC | 11:49 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Fix teams docs. https://review.openstack.org/170822 | 11:49 |
| *** jamielennox is now known as jamielennox|away | 11:49 | |
| *** Hal has joined #openstack-infra | 11:50 | |
| *** Hal is now known as Guest98422 | 11:50 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Independent provider manager for slow tasks https://review.openstack.org/170788 | 11:52 |
| *** dhritishikhar_ has joined #openstack-infra | 11:54 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-cgit: Add configurable settings for httpd.conf https://review.openstack.org/160885 | 11:54 |
| *** dkranz has joined #openstack-infra | 11:56 | |
| *** marcusvrn has quit IRC | 11:56 | |
| *** marcusvrn has joined #openstack-infra | 11:56 | |
| *** dhritishikhar___ has quit IRC | 11:58 | |
| *** jschwarz has joined #openstack-infra | 11:58 | |
| *** woodster_ has joined #openstack-infra | 12:00 | |
| *** dizquierdo has joined #openstack-infra | 12:01 | |
| *** baoli has joined #openstack-infra | 12:05 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Fixed reference to experimental API https://review.openstack.org/169458 | 12:06 |
| *** markvoelker has joined #openstack-infra | 12:06 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Added commandline commands to package.json https://review.openstack.org/168488 | 12:06 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Removed grunt:server https://review.openstack.org/163165 | 12:06 |
| *** kgiusti has joined #openstack-infra | 12:08 | |
| *** rkukura has joined #openstack-infra | 12:09 | |
| *** markvoelker has quit IRC | 12:10 | |
| *** chandankumar has quit IRC | 12:11 | |
| *** dhritishikhar_ has quit IRC | 12:12 | |
| *** dhritishikhar_ has joined #openstack-infra | 12:12 | |
| *** dhritishikhar_ has quit IRC | 12:14 | |
| *** dhritishikhar_ has joined #openstack-infra | 12:15 | |
| *** dhritishikhar_ has quit IRC | 12:17 | |
| *** yamamoto has quit IRC | 12:17 | |
| *** bswartz has quit IRC | 12:19 | |
| *** xyang1 has joined #openstack-infra | 12:19 | |
| *** dhritishikhar_ has joined #openstack-infra | 12:19 | |
| *** bradjones_ has joined #openstack-infra | 12:22 | |
| *** katyafervent is now known as katyafervent_awa | 12:23 | |
| *** asilenkov_ has joined #openstack-infra | 12:23 | |
| *** ajo is now known as ajo|lunch | 12:23 | |
| *** bradjones has quit IRC | 12:24 | |
| *** bradjones_ is now known as bradjones | 12:24 | |
| *** bradjones has joined #openstack-infra | 12:24 | |
| *** markvoelker has joined #openstack-infra | 12:24 | |
| *** dhritishikhar_ has quit IRC | 12:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch check-functional-dsvm-magnum on for every check review https://review.openstack.org/170755 | 12:25 |
| *** coolsvap has quit IRC | 12:25 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Add Docker and Kubernetes requirements needed by Magnum https://review.openstack.org/170830 | 12:25 |
| *** rfolco has joined #openstack-infra | 12:25 | |
| *** dprince has joined #openstack-infra | 12:25 | |
| *** emagana has joined #openstack-infra | 12:25 | |
| *** dimsum__ has quit IRC | 12:25 | |
| *** wenlock has joined #openstack-infra | 12:26 | |
| *** weshay has joined #openstack-infra | 12:28 | |
| *** chandankumar has joined #openstack-infra | 12:28 | |
| *** emagana has quit IRC | 12:30 | |
| *** wenlock has quit IRC | 12:30 | |
| *** tnovacik has quit IRC | 12:30 | |
| *** ayoung has joined #openstack-infra | 12:32 | |
| *** dhritishikhar_ has joined #openstack-infra | 12:33 | |
| *** aysyd has joined #openstack-infra | 12:33 | |
| *** pal has joined #openstack-infra | 12:36 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard-webclient: Added possibility to scroll events https://review.openstack.org/170820 | 12:36 |
| *** Krinkle|detached is now known as Krinkle | 12:37 | |
| *** deepakcs has quit IRC | 12:38 | |
| *** marcusvrn has quit IRC | 12:41 | |
| *** marcusvrn1 has joined #openstack-infra | 12:41 | |
| *** tkelsey has joined #openstack-infra | 12:44 | |
| *** _nadya_ has joined #openstack-infra | 12:44 | |
| *** dkliban_afk is now known as dkliban | 12:45 | |
| *** rlandy has joined #openstack-infra | 12:45 | |
| *** e0ne is now known as e0ne_ | 12:46 | |
| *** koolhead17 has joined #openstack-infra | 12:49 | |
| *** e0ne_ is now known as e0ne | 12:49 | |
| *** annegentle has joined #openstack-infra | 12:50 | |
| *** tkelsey has quit IRC | 12:50 | |
| *** dimsum__ has joined #openstack-infra | 12:51 | |
| *** annegentle has quit IRC | 12:52 | |
| *** annegentle has joined #openstack-infra | 12:52 | |
| *** dizquierdo has quit IRC | 12:53 | |
| *** koolhead_ has joined #openstack-infra | 12:55 | |
| *** claudiub has joined #openstack-infra | 12:57 | |
| *** dboik has joined #openstack-infra | 12:57 | |
| *** bswartz has joined #openstack-infra | 12:57 | |
| *** hdd has joined #openstack-infra | 12:58 | |
| *** sdake has joined #openstack-infra | 12:58 | |
| *** amotoki has quit IRC | 12:58 | |
| *** koolhead17 has quit IRC | 12:59 | |
| *** sdake_ has joined #openstack-infra | 12:59 | |
| *** dboik_ has joined #openstack-infra | 13:00 | |
| *** mriedem1 has joined #openstack-infra | 13:00 | |
| *** mriedem has quit IRC | 13:02 | |
| *** dboik has quit IRC | 13:02 | |
| *** dizquierdo has joined #openstack-infra | 13:03 | |
| *** Krinkle is now known as Krinkle|detached | 13:03 | |
| *** sdake has quit IRC | 13:03 | |
| *** dimsum__ is now known as dims | 13:03 | |
| *** adalbas has joined #openstack-infra | 13:05 | |
| *** mriedem1 has quit IRC | 13:06 | |
| *** bcourt has joined #openstack-infra | 13:08 | |
| *** julim has joined #openstack-infra | 13:09 | |
| *** julim has quit IRC | 13:11 | |
| *** katyafervent_awa is now known as katyafervent | 13:11 | |
| *** rkukura has quit IRC | 13:14 | |
| *** dboik_ has quit IRC | 13:14 | |
| *** tiswanso has joined #openstack-infra | 13:18 | |
| *** ajo|lunch is now known as ajo | 13:18 | |
| *** emagana has joined #openstack-infra | 13:20 | |
| *** medieval1 has joined #openstack-infra | 13:20 | |
| *** medieval1 has quit IRC | 13:22 | |
| *** annegentle has quit IRC | 13:22 | |
| *** hdd has quit IRC | 13:22 | |
| *** medieval1 has joined #openstack-infra | 13:23 | |
| *** emagana has quit IRC | 13:24 | |
| *** davideagnello has joined #openstack-infra | 13:25 | |
| *** dustins has joined #openstack-infra | 13:29 | |
| *** davideagnello has quit IRC | 13:29 | |
| *** teran_ has joined #openstack-infra | 13:31 | |
| *** e0ne is now known as e0ne_ | 13:31 | |
| *** dboik has joined #openstack-infra | 13:33 | |
| *** Longgeek has quit IRC | 13:33 | |
| *** dboik has quit IRC | 13:33 | |
| *** dboik has joined #openstack-infra | 13:34 | |
| *** teran has quit IRC | 13:34 | |
| *** teran_ has quit IRC | 13:35 | |
| *** subscope has quit IRC | 13:35 | |
| *** teran has joined #openstack-infra | 13:36 | |
| *** mriedem has joined #openstack-infra | 13:37 | |
| *** otter768 has joined #openstack-infra | 13:38 | |
| *** wenlock has joined #openstack-infra | 13:39 | |
| *** wenlock_ has joined #openstack-infra | 13:39 | |
| *** otter768 has quit IRC | 13:43 | |
| fungi | i wonder how https://github.com/saltstack/raet would stack up against rabbit for the openstack backend communication layer. that might address a lot of the server-to-server communication security we're lacking (where ssh key management and api calls over https cover the rest) | 13:44 |
| *** chandankumar has quit IRC | 13:46 | |
| *** dustins_ has joined #openstack-infra | 13:47 | |
| *** bknudson has joined #openstack-infra | 13:48 | |
| *** eharney has joined #openstack-infra | 13:48 | |
| *** esker has joined #openstack-infra | 13:49 | |
| *** dustins has quit IRC | 13:49 | |
| *** ddieterly has joined #openstack-infra | 13:49 | |
| *** annegentle has joined #openstack-infra | 13:52 | |
| *** e0ne_ is now known as e0ne | 13:53 | |
| *** bradjones has quit IRC | 13:53 | |
| *** wenlock_ has quit IRC | 13:53 | |
| *** wenlock has quit IRC | 13:53 | |
| *** mriedem has quit IRC | 13:53 | |
| *** tiswanso has quit IRC | 13:53 | |
| *** marcusvrn1 has quit IRC | 13:53 | |
| *** dprince has quit IRC | 13:53 | |
| *** markvoelker has quit IRC | 13:53 | |
| *** niska has quit IRC | 13:53 | |
| *** hogepodge has quit IRC | 13:53 | |
| *** sc68cal has quit IRC | 13:53 | |
| *** sdague has quit IRC | 13:53 | |
| *** briancline has quit IRC | 13:53 | |
| *** bcourt has quit IRC | 13:53 | |
| *** dizquierdo has quit IRC | 13:53 | |
| *** dhritishikhar_ has quit IRC | 13:53 | |
| *** skolekonov has quit IRC | 13:53 | |
| *** e0ne has quit IRC | 13:53 | |
| *** therve has quit IRC | 13:53 | |
| *** xianghui has quit IRC | 13:53 | |
| *** yamahata has quit IRC | 13:53 | |
| *** david-lyle has quit IRC | 13:53 | |
| *** gilliard has quit IRC | 13:53 | |
| *** JoshNang has quit IRC | 13:53 | |
| *** isq has quit IRC | 13:53 | |
| *** juggler_ has quit IRC | 13:53 | |
| *** mwhudson has quit IRC | 13:53 | |
| *** toabctl has quit IRC | 13:53 | |
| *** raginbajin has quit IRC | 13:53 | |
| *** sgordon has quit IRC | 13:53 | |
| *** Vivek has quit IRC | 13:53 | |
| *** _qb has quit IRC | 13:53 | |
| *** spiffxp- has quit IRC | 13:53 | |
| *** Clint has quit IRC | 13:53 | |
| *** dstufft has quit IRC | 13:53 | |
| *** Apsu has quit IRC | 13:53 | |
| *** d0ugal has quit IRC | 13:53 | |
| *** timrc has quit IRC | 13:53 | |
| *** dkliban has quit IRC | 13:53 | |
| *** mkoderer has quit IRC | 13:53 | |
| *** laurelm has quit IRC | 13:53 | |
| *** begal-sop_ has quit IRC | 13:53 | |
| *** spiffxp has quit IRC | 13:53 | |
| *** ZZelle has quit IRC | 13:53 | |
| *** dtroyer has quit IRC | 13:53 | |
| *** hockeynut has quit IRC | 13:53 | |
| *** krtaylor has quit IRC | 13:53 | |
| *** dtantsur|afk has quit IRC | 13:53 | |
| *** mordred has quit IRC | 13:53 | |
| *** cyeoh has quit IRC | 13:53 | |
| *** freyes has quit IRC | 13:53 | |
| *** jvrbanac has quit IRC | 13:53 | |
| *** masayukig has quit IRC | 13:53 | |
| *** zz_dimtruck has quit IRC | 13:53 | |
| *** mgagne has quit IRC | 13:53 | |
| *** adam_g_out has quit IRC | 13:53 | |
| *** comstud has quit IRC | 13:53 | |
| *** rainya has quit IRC | 13:53 | |
| *** lbragstad has quit IRC | 13:53 | |
| *** gus has quit IRC | 13:53 | |
| *** gondoi has quit IRC | 13:53 | |
| *** ryanpetrello has quit IRC | 13:53 | |
| *** adreznec has quit IRC | 13:53 | |
| *** maurosr has quit IRC | 13:53 | |
| *** mattoliverau has quit IRC | 13:53 | |
| *** anthonyper has quit IRC | 13:53 | |
| *** ekarlso has quit IRC | 13:53 | |
| *** primeministerp has quit IRC | 13:53 | |
| *** Mithrandir has quit IRC | 13:53 | |
| *** mbacchi-pi has quit IRC | 13:53 | |
| *** jklare has quit IRC | 13:53 | |
| *** mikal has quit IRC | 13:53 | |
| *** pfallenop has quit IRC | 13:53 | |
| *** darvon has quit IRC | 13:53 | |
| *** coolsvap|afk has quit IRC | 13:53 | |
| *** Trozz has quit IRC | 13:53 | |
| *** jroll has quit IRC | 13:53 | |
| *** dolphm has quit IRC | 13:53 | |
| *** eglute has quit IRC | 13:53 | |
| *** d34dh0r53 has quit IRC | 13:53 | |
| *** pleia2 has quit IRC | 13:53 | |
| *** clarkb has quit IRC | 13:53 | |
| *** SergeyLukjanov has quit IRC | 13:53 | |
| *** NikitaKonovalov has quit IRC | 13:53 | |
| *** dmitryme has quit IRC | 13:53 | |
| *** alazarev has quit IRC | 13:53 | |
| *** Daviey has quit IRC | 13:53 | |
| *** dims has quit IRC | 13:53 | |
| *** vlaza has quit IRC | 13:53 | |
| *** TheJulia has quit IRC | 13:53 | |
| *** radez_g0n3 has quit IRC | 13:53 | |
| *** lascii has quit IRC | 13:53 | |
| *** ianw has quit IRC | 13:53 | |
| *** fabo has quit IRC | 13:53 | |
| *** jgriffith has quit IRC | 13:53 | |
| *** davidlenwell has quit IRC | 13:53 | |
| *** purp has quit IRC | 13:53 | |
| *** SpamapS has quit IRC | 13:53 | |
| *** beecee has quit IRC | 13:53 | |
| *** sirushti has quit IRC | 13:53 | |
| *** vipul has quit IRC | 13:53 | |
| *** GheRivero has quit IRC | 13:53 | |
| *** phschwartz has quit IRC | 13:53 | |
| *** antonym has quit IRC | 13:53 | |
| *** frobware has quit IRC | 13:53 | |
| *** gothicmindfood has quit IRC | 13:53 | |
| *** jeblair has quit IRC | 13:53 | |
| *** larsks has quit IRC | 13:53 | |
| *** ibiris_away has quit IRC | 13:53 | |
| *** jhesketh has quit IRC | 13:53 | |
| *** sdake_ has quit IRC | 13:53 | |
| *** wicope has quit IRC | 13:53 | |
| *** HeOS has quit IRC | 13:53 | |
| *** imcsk8 has quit IRC | 13:53 | |
| *** enikanorov has quit IRC | 13:53 | |
| *** harlowja_away has quit IRC | 13:53 | |
| *** trey has quit IRC | 13:53 | |
| *** Krinkle|detached has quit IRC | 13:53 | |
| *** skraynev has quit IRC | 13:53 | |
| *** flaper87 has quit IRC | 13:53 | |
| *** markvan has quit IRC | 13:53 | |
| *** Ryan_Lane has quit IRC | 13:53 | |
| *** frickler has quit IRC | 13:53 | |
| *** ondergetekende has quit IRC | 13:53 | |
| *** BobBall_AWAY has quit IRC | 13:53 | |
| *** waynr has quit IRC | 13:53 | |
| *** jkt has quit IRC | 13:53 | |
| *** zz_zz_ja| has quit IRC | 13:53 | |
| *** clayton has quit IRC | 13:53 | |
| *** SlickNik has quit IRC | 13:53 | |
| *** filler has quit IRC | 13:53 | |
| *** DinaBelova has quit IRC | 13:53 | |
| *** ruhe has quit IRC | 13:53 | |
| anteaya | fungi: if I see techhat at pycon I can ask him | 13:53 |
| *** hdd has joined #openstack-infra | 13:54 | |
| fungi | cool! | 13:55 |
| *** ZZelle has joined #openstack-infra | 13:55 | |
| *** dtroyer has joined #openstack-infra | 13:55 | |
| *** hockeynut has joined #openstack-infra | 13:55 | |
| *** krtaylor has joined #openstack-infra | 13:55 | |
| *** dtantsur|afk has joined #openstack-infra | 13:55 | |
| *** mordred has joined #openstack-infra | 13:55 | |
| *** cyeoh has joined #openstack-infra | 13:55 | |
| *** freyes has joined #openstack-infra | 13:55 | |
| *** jvrbanac has joined #openstack-infra | 13:55 | |
| *** zz_dimtruck has joined #openstack-infra | 13:55 | |
| *** masayukig has joined #openstack-infra | 13:55 | |
| *** mgagne has joined #openstack-infra | 13:55 | |
| *** gondoi has joined #openstack-infra | 13:55 | |
| *** adam_g_out has joined #openstack-infra | 13:55 | |
| *** rainya has joined #openstack-infra | 13:55 | |
| *** lbragstad has joined #openstack-infra | 13:55 | |
| *** comstud has joined #openstack-infra | 13:55 | |
| *** gus has joined #openstack-infra | 13:55 | |
| *** ryanpetrello has joined #openstack-infra | 13:55 | |
| *** adreznec has joined #openstack-infra | 13:55 | |
| *** maurosr has joined #openstack-infra | 13:55 | |
| *** mattoliverau has joined #openstack-infra | 13:55 | |
| *** anthonyper has joined #openstack-infra | 13:55 | |
| *** ekarlso has joined #openstack-infra | 13:55 | |
| *** primeministerp has joined #openstack-infra | 13:55 | |
| *** Mithrandir has joined #openstack-infra | 13:55 | |
| *** mbacchi-pi has joined #openstack-infra | 13:55 | |
| *** jklare has joined #openstack-infra | 13:55 | |
| *** mikal has joined #openstack-infra | 13:55 | |
| *** pfallenop has joined #openstack-infra | 13:55 | |
| *** darvon has joined #openstack-infra | 13:55 | |
| *** coolsvap|afk has joined #openstack-infra | 13:55 | |
| *** Trozz has joined #openstack-infra | 13:55 | |
| *** eglute has joined #openstack-infra | 13:55 | |
| *** jroll has joined #openstack-infra | 13:55 | |
| *** dolphm has joined #openstack-infra | 13:55 | |
| *** d34dh0r53 has joined #openstack-infra | 13:55 | |
| *** pleia2 has joined #openstack-infra | 13:55 | |
| *** clarkb has joined #openstack-infra | 13:55 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:55 | |
| *** NikitaKonovalov has joined #openstack-infra | 13:55 | |
| *** dmitryme has joined #openstack-infra | 13:55 | |
| *** alazarev has joined #openstack-infra | 13:55 | |
| *** Daviey has joined #openstack-infra | 13:55 | |
| anteaya | eantyshev: if we would like to join #openstack-meeting one of the weekly third party meetings is about to begin, you might find someone able to help you | 13:55 |
| *** dims has joined #openstack-infra | 13:56 | |
| *** vlaza has joined #openstack-infra | 13:56 | |
| *** sirushti has joined #openstack-infra | 13:56 | |
| *** TheJulia has joined #openstack-infra | 13:56 | |
| *** radez_g0n3 has joined #openstack-infra | 13:56 | |
| *** lascii has joined #openstack-infra | 13:56 | |
| *** ianw has joined #openstack-infra | 13:56 | |
| *** fabo has joined #openstack-infra | 13:56 | |
| *** jgriffith has joined #openstack-infra | 13:56 | |
| *** davidlenwell has joined #openstack-infra | 13:56 | |
| *** purp has joined #openstack-infra | 13:56 | |
| *** SpamapS has joined #openstack-infra | 13:56 | |
| *** larsks has joined #openstack-infra | 13:56 | |
| *** beecee has joined #openstack-infra | 13:56 | |
| *** ibiris_away has joined #openstack-infra | 13:56 | |
| *** vipul has joined #openstack-infra | 13:56 | |
| *** GheRivero has joined #openstack-infra | 13:56 | |
| *** phschwartz has joined #openstack-infra | 13:56 | |
| *** antonym has joined #openstack-infra | 13:56 | |
| *** frobware has joined #openstack-infra | 13:56 | |
| *** gothicmindfood has joined #openstack-infra | 13:56 | |
| *** jeblair has joined #openstack-infra | 13:56 | |
| *** dizquierdo has joined #openstack-infra | 13:56 | |
| *** amitgandhinz has joined #openstack-infra | 13:56 | |
| *** nelsnelson has joined #openstack-infra | 13:58 | |
| *** niska has joined #openstack-infra | 13:58 | |
| *** wenlock_ has joined #openstack-infra | 13:58 | |
| *** wenlock has joined #openstack-infra | 13:58 | |
| *** mriedem has joined #openstack-infra | 13:58 | |
| *** tiswanso has joined #openstack-infra | 13:58 | |
| *** marcusvrn1 has joined #openstack-infra | 13:58 | |
| *** dprince has joined #openstack-infra | 13:58 | |
| *** markvoelker has joined #openstack-infra | 13:58 | |
| *** hogepodge has joined #openstack-infra | 13:58 | |
| *** sc68cal has joined #openstack-infra | 13:58 | |
| *** sdague has joined #openstack-infra | 13:58 | |
| *** briancline has joined #openstack-infra | 13:58 | |
| *** bcourt has joined #openstack-infra | 13:58 | |
| *** dhritishikhar_ has joined #openstack-infra | 13:58 | |
| *** skolekonov has joined #openstack-infra | 13:58 | |
| *** e0ne has joined #openstack-infra | 13:58 | |
| *** therve has joined #openstack-infra | 13:58 | |
| *** xianghui has joined #openstack-infra | 13:58 | |
| *** yamahata has joined #openstack-infra | 13:58 | |
| *** david-lyle has joined #openstack-infra | 13:58 | |
| *** gilliard has joined #openstack-infra | 13:58 | |
| *** JoshNang has joined #openstack-infra | 13:58 | |
| *** isq has joined #openstack-infra | 13:58 | |
| *** juggler_ has joined #openstack-infra | 13:58 | |
| *** mwhudson has joined #openstack-infra | 13:58 | |
| *** toabctl has joined #openstack-infra | 13:58 | |
| *** raginbajin has joined #openstack-infra | 13:58 | |
| *** sgordon has joined #openstack-infra | 13:58 | |
| *** Vivek has joined #openstack-infra | 13:58 | |
| *** _qb has joined #openstack-infra | 13:58 | |
| *** spiffxp- has joined #openstack-infra | 13:58 | |
| *** Clint has joined #openstack-infra | 13:58 | |
| *** dstufft has joined #openstack-infra | 13:58 | |
| *** Apsu has joined #openstack-infra | 13:58 | |
| *** d0ugal has joined #openstack-infra | 13:58 | |
| *** timrc has joined #openstack-infra | 13:58 | |
| *** dkliban has joined #openstack-infra | 13:58 | |
| *** mkoderer has joined #openstack-infra | 13:58 | |
| *** laurelm has joined #openstack-infra | 13:58 | |
| *** begal-sop_ has joined #openstack-infra | 13:58 | |
| *** spiffxp has joined #openstack-infra | 13:58 | |
| *** jhesketh has joined #openstack-infra | 13:58 | |
| *** spzala has joined #openstack-infra | 13:58 | |
| *** dvorak has joined #openstack-infra | 13:58 | |
| *** bradjones has joined #openstack-infra | 13:58 | |
| *** sdake_ has joined #openstack-infra | 13:58 | |
| *** wicope has joined #openstack-infra | 13:58 | |
| *** HeOS has joined #openstack-infra | 13:58 | |
| *** imcsk8 has joined #openstack-infra | 13:58 | |
| *** enikanorov has joined #openstack-infra | 13:58 | |
| *** harlowja_away has joined #openstack-infra | 13:58 | |
| *** trey has joined #openstack-infra | 13:58 | |
| *** Krinkle|detached has joined #openstack-infra | 13:58 | |
| *** skraynev has joined #openstack-infra | 13:58 | |
| *** flaper87 has joined #openstack-infra | 13:58 | |
| *** markvan has joined #openstack-infra | 13:58 | |
| *** Ryan_Lane has joined #openstack-infra | 13:58 | |
| *** frickler has joined #openstack-infra | 13:58 | |
| *** ondergetekende has joined #openstack-infra | 13:58 | |
| *** BobBall_AWAY has joined #openstack-infra | 13:58 | |
| *** waynr has joined #openstack-infra | 13:58 | |
| *** jkt has joined #openstack-infra | 13:58 | |
| *** zz_zz_ja| has joined #openstack-infra | 13:58 | |
| *** SlickNik has joined #openstack-infra | 13:58 | |
| *** filler has joined #openstack-infra | 13:58 | |
| *** DinaBelova has joined #openstack-infra | 13:58 | |
| *** ruhe has joined #openstack-infra | 13:58 | |
| *** flaper87 has quit IRC | 13:59 | |
| *** flaper87 has joined #openstack-infra | 13:59 | |
| *** wicope has quit IRC | 13:59 | |
| *** wicope has joined #openstack-infra | 14:00 | |
| *** lascii is now known as alaski | 14:00 | |
| *** nelsnelson has quit IRC | 14:00 | |
| *** niska has quit IRC | 14:00 | |
| *** wenlock_ has quit IRC | 14:00 | |
| *** wenlock has quit IRC | 14:00 | |
| *** mriedem has quit IRC | 14:00 | |
| *** tiswanso has quit IRC | 14:00 | |
| *** marcusvrn1 has quit IRC | 14:00 | |
| *** dprince has quit IRC | 14:00 | |
| *** markvoelker has quit IRC | 14:00 | |
| *** hogepodge has quit IRC | 14:00 | |
| *** sc68cal has quit IRC | 14:00 | |
| *** sdague has quit IRC | 14:00 | |
| *** briancline has quit IRC | 14:00 | |
| *** niska has joined #openstack-infra | 14:00 | |
| *** nelsnelson has joined #openstack-infra | 14:01 | |
| *** wenlock_ has joined #openstack-infra | 14:01 | |
| *** wenlock has joined #openstack-infra | 14:01 | |
| *** mriedem has joined #openstack-infra | 14:01 | |
| *** tiswanso has joined #openstack-infra | 14:01 | |
| *** marcusvrn1 has joined #openstack-infra | 14:01 | |
| *** dprince has joined #openstack-infra | 14:01 | |
| *** markvoelker has joined #openstack-infra | 14:01 | |
| *** hogepodge has joined #openstack-infra | 14:01 | |
| *** sc68cal has joined #openstack-infra | 14:01 | |
| *** sdague has joined #openstack-infra | 14:01 | |
| *** briancline has joined #openstack-infra | 14:01 | |
| ctlaugh | krtaylor: Thank you for finding my typo :) | 14:03 |
| krtaylor | ctlaugh, hehheh, sorry for such a nit :) | 14:04 |
| *** annegentle has quit IRC | 14:05 | |
| *** bhunter71 has joined #openstack-infra | 14:05 | |
| ctlaugh | krtaylor: np :) | 14:05 |
| ctlaugh | It was late last night when I finally sent them all up | 14:06 |
| *** ajmiller has joined #openstack-infra | 14:07 | |
| openstackgerrit | Clark Laughlin proposed openstack-infra/system-config: Do not install linux-headers-virtual for trusty or later https://review.openstack.org/170776 | 14:08 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:08 | |
| *** tnovacik has joined #openstack-infra | 14:10 | |
| anteaya | fungi: if you are up for a silly question: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/ I'm looking at the April logs though there may be other examples | 14:12 |
| anteaya | the April 1 log has 02 April | 14:12 |
| anteaya | and so does April 2 | 14:12 |
| anteaya | April 3 have 04 April | 14:13 |
| anteaya | and so does April 4 | 14:13 |
| anteaya | is this a known thing or noone has looked at it before? | 14:13 |
| anteaya | April 3 has 04 April | 14:15 |
| fungi | anteaya: not so much a bug as just it rotates at midnight utc and sometimes the file was last updated close enough to midnight that the write buffer might not get flushed until a few seconds after. that's merely the file modification date/time | 14:15 |
| *** emagana has joined #openstack-infra | 14:18 | |
| anteaya | ah | 14:18 |
| anteaya | okay I understand now | 14:18 |
| *** emagana has quit IRC | 14:18 | |
| *** btully has quit IRC | 14:18 | |
| anteaya | thanks | 14:19 |
| *** emagana has joined #openstack-infra | 14:19 | |
| *** hdd has quit IRC | 14:19 | |
| *** miqui has joined #openstack-infra | 14:19 | |
| ameade | Hey folks, this patch doesnt seem to be getting picked up for the gate queue: https://review.openstack.org/#/c/169781/ | 14:21 |
| anteaya | ameade: I just got it back in to check | 14:22 |
| *** nelsnelson has quit IRC | 14:22 | |
| *** amotoki has joined #openstack-infra | 14:23 | |
| *** amotoki has quit IRC | 14:23 | |
| anteaya | ameade: fungi had to restart gerrit a while ago, you are welcome to check the backscroll, the queue was backed up and changed had not been picked up | 14:23 |
| *** mtanino has joined #openstack-infra | 14:23 | |
| anteaya | ameade: if it passes check and doesnt' get into the gate you can let us know and I can look again, but if that happens you probably need a fresh +A | 14:23 |
| anteaya | ameade: let's see what happens | 14:24 |
| *** sputnik13 has quit IRC | 14:24 | |
| ameade | anteaya: kk cool | 14:24 |
| fungi | i hope that's not related. the events we lost were between 10:56 and 11:37 utc | 14:24 |
| fungi | oh, i see it was approved at 11:09 utc so yes that falls within the problem window | 14:25 |
| *** tonytan4ever has joined #openstack-infra | 14:25 | |
| fungi | rechecks aren't going to help there because the change already has a verify +1 from jenkins | 14:25 |
| fungi | it needs to get reapproved (replace or add a workflow +1 vote) and it should pop right into the gate pipeline at that point | 14:25 |
| anteaya | great, thanks fungi | 14:25 |
| ameade | fungi: thanks, i'll ping some cinder cores then | 14:26 |
| *** mtanino has quit IRC | 14:27 | |
| *** mtanino has joined #openstack-infra | 14:27 | |
| *** zz_dimtruck is now known as dimtruck | 14:29 | |
| *** dangers_away is now known as dangers | 14:29 | |
| *** sputnik13 has joined #openstack-infra | 14:30 | |
| *** deepakcs has joined #openstack-infra | 14:32 | |
| *** achanda has joined #openstack-infra | 14:35 | |
| *** funzo has quit IRC | 14:35 | |
| *** pal has quit IRC | 14:37 | |
| *** nelsnelson has joined #openstack-infra | 14:40 | |
| *** _nadya_ has quit IRC | 14:40 | |
| *** ivar-lazzaro has joined #openstack-infra | 14:42 | |
| *** achanda has quit IRC | 14:43 | |
| *** ZZelle has quit IRC | 14:43 | |
| *** flaviosr has joined #openstack-infra | 14:44 | |
| *** ZZelle has joined #openstack-infra | 14:44 | |
| *** AJaeger has joined #openstack-infra | 14:44 | |
| *** AJaeger has quit IRC | 14:44 | |
| *** dhritishikhar_ has quit IRC | 14:46 | |
| *** eantyshev has quit IRC | 14:46 | |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Sync starred changes regardless of subscription https://review.openstack.org/168508 | 14:47 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Sync own changes regardless of subscription https://review.openstack.org/168181 | 14:47 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Add permalink to change view https://review.openstack.org/169766 | 14:47 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Support searching by URL https://review.openstack.org/169371 | 14:47 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Add checkout and cherry-pick to change list https://review.openstack.org/168510 | 14:47 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Add held changes https://review.openstack.org/167838 | 14:47 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Highlight starred changes in list https://review.openstack.org/166572 | 14:47 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Fix changeset fetching https://review.openstack.org/124239 | 14:47 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Try git protocol last when fetching. https://review.openstack.org/170860 | 14:47 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Sync change when missing refs https://review.openstack.org/170861 | 14:47 |
| *** MarkAtwood has joined #openstack-infra | 14:47 | |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr: Add TODO file to os-testr https://review.openstack.org/170864 | 14:49 |
| *** annegentle has joined #openstack-infra | 14:49 | |
| *** julim has joined #openstack-infra | 14:52 | |
| *** zul has quit IRC | 14:54 | |
| *** teran has quit IRC | 14:54 | |
| *** teran has joined #openstack-infra | 14:54 | |
| *** signed8bit has joined #openstack-infra | 14:56 | |
| *** jaypipes has joined #openstack-infra | 14:56 | |
| *** ociuhandu has joined #openstack-infra | 14:56 | |
| *** hdd has joined #openstack-infra | 14:56 | |
| *** teran_ has joined #openstack-infra | 14:57 | |
| *** vlaza has quit IRC | 14:58 | |
| *** AJaeger has joined #openstack-infra | 14:59 | |
| *** teran has quit IRC | 14:59 | |
| anteaya | fungi: so confirming the ask.o.o data migration is happening at 1700 utc, yes? http://lists.openstack.org/pipermail/community/2015-March/001102.html | 14:59 |
| *** esker has quit IRC | 15:00 | |
| *** chlong has quit IRC | 15:00 | |
| *** unicell1 has joined #openstack-infra | 15:00 | |
| *** pal has joined #openstack-infra | 15:02 | |
| fungi | anteaya: that's when i planned to start, though reed's later announcements said "9am pdt" which would be 16:00 utc | 15:02 |
| *** e0ne is now known as e0ne_ | 15:02 | |
| *** asselin has joined #openstack-infra | 15:02 | |
| *** unicell has quit IRC | 15:02 | |
| fungi | http://lists.openstack.org/pipermail/openstack/2015-April/012172.html | 15:03 |
| fungi | so i suppose i could start earlier if i want | 15:03 |
| anteaya | oh goodness | 15:03 |
| anteaya | well since it is just you, I guess so | 15:03 |
| *** rkukura has joined #openstack-infra | 15:03 | |
| *** e0ne_ is now known as e0ne | 15:03 | |
| fungi | i didn't have lunch plans anyway, so it makes no difference to me | 15:04 |
| anteaya | fungi: let me know when you are finished as you had said you needed some testers | 15:04 |
| anteaya | fungi: well there's a silver lining | 15:04 |
| fungi | you bet--thanks! | 15:04 |
| fungi | double-checking the ttl on the dns record now | 15:04 |
| anteaya | kk | 15:04 |
| anteaya | fungi: did you have an etherpad for this so I can following the bouncing ball? | 15:04 |
| *** esker has joined #openstack-infra | 15:04 | |
| anteaya | or do you just want to ping when there is something I can do? | 15:05 |
| fungi | probably no need for an etherpad since it's just me performing the actual steps, and they're all documented at http://specs.openstack.org/openstack-infra/infra-specs/specs/migrate_askbot.html | 15:05 |
| *** dkranz has quit IRC | 15:06 | |
| jeblair | fungi: let me know if you need a hand | 15:06 |
| *** packet has joined #openstack-infra | 15:07 | |
| fungi | thanks jeblair. shouldn't be necessary but i'll let you know | 15:07 |
| anteaya | fungi: k, thanks | 15:07 |
| anteaya | jeblair: morning, so last week I was away Wednesday for boiling down syrup as I had planned and posted to channel Tuesday night | 15:08 |
| anteaya | then Thursday right after nova meeting, the power went out, took out half the town and didn't come back on until the end of the day | 15:09 |
| anteaya | so sorry for not being around | 15:09 |
| *** radez_g0n3 is now known as radez | 15:09 | |
| *** davideagnello has joined #openstack-infra | 15:13 | |
| * SpamapS shakes off the rust from 3 days of PTO+weekend | 15:13 | |
| *** dannywilson has joined #openstack-infra | 15:14 | |
| anteaya | rust is good | 15:14 |
| SpamapS | hah, yes yes it is | 15:14 |
| jeblair | anteaya: no worries; i'm enjoying following your adventures in syrup making :) | 15:15 |
| sdague | jeblair: ok, I have the devstack plugin working for trove | 15:15 |
| anteaya | fungi: should a third party ci be having to specify aio in their job? http://paste.openstack.org/show/198549/ | 15:15 |
| anteaya | jeblair: cool, thanks :) | 15:15 |
| anteaya | it has been an unbelieveable year | 15:15 |
| jeblair | sdague: yay! i didn't end up with as much time on friday to look at that as i had hoped. i'll take a look now | 15:16 |
| *** dannywilson has quit IRC | 15:16 | |
| *** dannywilson has joined #openstack-infra | 15:17 | |
| sdague | I also disconnected the grenade change from the series and merged that, because it wasn't really strictly relevant, and made understanding all the depends-on graph a bit easier | 15:17 |
| fungi | anteaya: i would think no since aio should be the default DEVSTACK_GATE_TOPOLOGY already? it's at least worth asking the question | 15:18 |
| jeblair | sdague: ok | 15:18 |
| mriedem | where does the log collection happen again for what gets archived in a job run? looked in devstack, d-g, system-config and project-config | 15:18 |
| *** marcusvrn has joined #openstack-infra | 15:19 | |
| sdague | mriedem: devstack-gate | 15:19 |
| anteaya | fungi: yes this was what I was thinking, trying to debug an issue in -meeting, thanks | 15:19 |
| anteaya | fungi: a newcomer so many things could be happening here | 15:19 |
| fungi | anteaya: devstack-vm-gate-wrap.sh exports DEVSTACK_GATE_TOPOLOGY=${DEVSTACK_GATE_TOPOLOGY:-aio} so unless they're using a modified version of that file it should be unnecessary (but fine) | 15:19 |
| anteaya | fungi: ah okay thanks, as long as no harm I'll leave it for now | 15:19 |
| mriedem | sdague: cleanup_host, found it, thanks | 15:20 |
| *** sdake has joined #openstack-infra | 15:20 | |
| *** packet has quit IRC | 15:20 | |
| *** dkranz has joined #openstack-infra | 15:21 | |
| *** koolhead_ has quit IRC | 15:21 | |
| jeblair | sdague: did 167412 hit the bug we were trying to fix with the 5 sec delay in zuul? | 15:21 |
| *** sputnik13 has quit IRC | 15:22 | |
| sdague | jeblair: not sure, it may be an actual merge conflict | 15:22 |
| sdague | at the end of the day we don't need the patch though | 15:22 |
| jeblair | sdague: why's that? | 15:23 |
| fungi | jeblair: sdague: looks like it was triggered by a new patchset in its crd dependency | 15:23 |
| *** dprince has quit IRC | 15:23 | |
| sdague | jeblair: because won't we land the trove bit, then flip the project-config? | 15:23 |
| jeblair | fungi: okay, that sounds non-buggy then. maybe not ideal, but at least halfway sensible. | 15:24 |
| sdague | or maybe I still don't have the graph fully in my head | 15:24 |
| *** sdake_ has quit IRC | 15:24 | |
| fungi | 167412 got the "depends on a change that failed to merge" error at 12:33 and it declares a crd on 167047 which got a patchset #5 at that same moment while it was still in the check pipeline | 15:24 |
| fungi | so the only saner things it might could have done would be to immediately requeue 167412 in the check pipeline to recalculate the dependency set, or return a less confusing error | 15:25 |
| reed | hello folks | 15:25 |
| *** hodos|3 has joined #openstack-infra | 15:25 | |
| fungi | hi-diddly-ho neighbor reed | 15:25 |
| jeblair | sdague: so _technically_ we don't need it, but it makes the patch series self-testing and also means that we don't race when making the change to project-config | 15:25 |
| sdague | jeblair: ok | 15:26 |
| sdague | so a couple of things that took a bit of time to sort out | 15:26 |
| sdague | 1) *when* you add enable_plugin into the localrc is important, it has to be *after* DEST is defined, because it uses DEST to do things | 15:27 |
| *** deepakcs has quit IRC | 15:27 | |
| sdague | so I had to reorder the devstack-gate bits | 15:27 |
| jeblair | sdague: basically the process is: 1) land (unused) trove plugin, 2) always ensure trove plugin is used regardless of configuration, 3) enable trove plugin via configuration, 4) remove safety from step (2) | 15:27 |
| fungi | reed: you had originally proposed 10am pdt today (17:00 utc) to start the ask.o.o outage, but your ml announcements said 9am pdt (which is 16:00 utc). i'm fine with either as long as mrmartin and evgeny are around to try it out as soon as i finish | 15:27 |
| sdague | when defined in project-config, we're already late enough, this isn't an issue | 15:28 |
| jeblair | sdague: ah, good to know | 15:28 |
| sdague | 2) using lib/... in plugins is really tricky because you aren't always where you think you are (pwd wise), so I just integrated it into a single plugin.sh file for simplicity | 15:28 |
| reed | fungi, let's start sooner, if possible... I'm checking with marton and evgeny | 15:28 |
| fungi | reed: fine by me. i've got everything ready to go anyway, so 1600 utc is cool with me | 15:29 |
| *** dizquierdo has quit IRC | 15:30 | |
| jeblair | sdague: why no git_clone in the plugin? | 15:31 |
| sdague | jeblair: because enable_plugin clones the tree | 15:32 |
| sdague | or do you mean the pythonclient bit? | 15:32 |
| jeblair | sdague: oooh, got it. that's kind of obvious, sorry. :) | 15:32 |
| *** mrmartin has joined #openstack-infra | 15:33 | |
| jeblair | sdague: what are you thinking wrt to troveclient? | 15:33 |
| *** GheRivero_ has joined #openstack-infra | 15:34 | |
| *** vlaza has joined #openstack-infra | 15:35 | |
| jeblair | sdague: i'm guessing the right thing to do is have the trove tempest job enable both trove and troveclient devstack plugins? | 15:35 |
| jeblair | sdague: 'cause i'm guessing trove itself doesn't use troveclient | 15:35 |
| *** sergsh has quit IRC | 15:36 | |
| sdague | basically do the same kind of plugin for troveclient, it has it's own job defs | 15:37 |
| *** rwsu has joined #openstack-infra | 15:37 | |
| sdague | yeh, and for that enable both | 15:38 |
| sdague | though, honestly, I don't know if troveclient even has fullstack tests | 15:38 |
| *** carl_baldwin has joined #openstack-infra | 15:38 | |
| *** otter768 has joined #openstack-infra | 15:39 | |
| sdague | yeh, actually, the troveclient tempest job is kind of pointless - http://logs.openstack.org/55/162855/5/check/gate-tempest-dsvm-neutron-src-python-troveclient/6b8d3ba/console.html.gz | 15:39 |
| *** vlaza has quit IRC | 15:39 | |
| *** carl_baldwin has quit IRC | 15:40 | |
| sdague | I do not know if the trove functional tests actually use troveclient | 15:40 |
| sdague | my guess is they don't | 15:40 |
| sigmavirus24 | So when I try to view the contents of https://review.openstack.org/#/c/105989/ I get http://paste.openstack.org/show/198579/ as an error message. Any ideas on why that might be? | 15:41 |
| *** carl_baldwin has joined #openstack-infra | 15:41 | |
| *** pal has quit IRC | 15:42 | |
| *** dprince has joined #openstack-infra | 15:43 | |
| *** pal has joined #openstack-infra | 15:43 | |
| *** otter768 has quit IRC | 15:44 | |
| jeblair | sigmavirus24: no, and wfm with ff37 | 15:44 |
| sigmavirus24 | jeblair: I'm on 36. Let me upgrade | 15:44 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/devstack-gate: Archive ceph logs and config https://review.openstack.org/170900 | 15:44 |
| *** flaviosr has quit IRC | 15:45 | |
| *** gyee has joined #openstack-infra | 15:45 | |
| anteaya | I too can open that url, ff37 | 15:45 |
| anteaya | and chrome 41 | 15:46 |
| sigmavirus24 | Other person having this problem is using chrome | 15:46 |
| *** david-lyle_ has joined #openstack-infra | 15:46 | |
| * sigmavirus24 wonders if it's an extension causing this | 15:46 | |
| jeblair | sigmavirus24, sdague: we recently added some js that fetches data from another host. that seems like entirely the wrong kind of url to trigger an error if it's related to that. however, it's worth entertaining the possibility it's related. | 15:47 |
| fungi | yeah, i would have expected a much different error message in that case | 15:48 |
| jeblair | also, have we reverted clippy yet? | 15:48 |
| sdague | jeblair: I don't think so, but the load is hidden behind a date check | 15:49 |
| sigmavirus24 | clippy worked fine for me | 15:50 |
| sigmavirus24 | and I can view other diffs on other changes | 15:50 |
| sigmavirus24 | so I'm not sure what's going on with that specific review | 15:50 |
| jeblair | sigmavirus24: that's consistent? | 15:50 |
| sigmavirus24 | With FF37 it still doesn't work | 15:50 |
| sdague | https://review.openstack.org/#/c/105989/8/specs/kilo/mtu-selection-and-advertisement.rst,cm is the url? | 15:50 |
| sigmavirus24 | correct | 15:50 |
| *** yamamoto has joined #openstack-infra | 15:50 | |
| sdague | ok, I see the same issue there | 15:51 |
| fungi | oh, i didn't think to test the new change screen | 15:51 |
| sigmavirus24 | So in Chrome, I get "Server Unavailable" | 15:51 |
| sigmavirus24 | Consistently | 15:51 |
| sdague | sigmavirus24: heh... check your adblocker | 15:52 |
| sigmavirus24 | ugh | 15:52 |
| sigmavirus24 | yeah | 15:52 |
| sigmavirus24 | that fixes it in Chrome | 15:52 |
| fungi | i wouldn't be surprised if some adblock plugins just knee-jerk filter anything with long strings of hex | 15:52 |
| sdague | yeh, the url has the word advertisement in it, uBlock decides that's evil and blocks it | 15:52 |
| zaro | morning | 15:52 |
| sigmavirus24 | I wonder what about that specific page, AdBlock Plus doesn't like | 15:52 |
| sigmavirus24 | HAH | 15:52 |
| fungi | oh, hah so a word match | 15:52 |
| sigmavirus24 | Sorry about that | 15:53 |
| *** davideagnello has quit IRC | 15:53 | |
| jeblair | interesting that only happens with the new screen | 15:53 |
| * sigmavirus24 reminds himself again why he's not a web-developer | 15:53 | |
| jeblair | sdague: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=397&rra_id=3&view_type=&graph_start=1425657038&graph_end=1428335438 | 15:53 |
| *** ZZelle_ has quit IRC | 15:54 | |
| jeblair | sdague: zuul seems to have gone from sending ~10Mbps peak to ~60Mbps with the gerrit js change | 15:54 |
| sdague | jeblair: what's the url in the old screen | 15:54 |
| sdague | jeblair: yeh, I think it's still fetching the entire status.json | 15:54 |
| *** sputnik13 has joined #openstack-infra | 15:55 | |
| sdague | did the patch to provide a per change url land ? | 15:55 |
| *** adalbas has quit IRC | 15:56 | |
| *** emagana has quit IRC | 15:56 | |
| jeblair | sdague: - /status/change/X,Y: return status just for gerrit change X,Y | 15:58 |
| jeblair | i think so | 15:58 |
| sdague | yeh | 15:58 |
| sdague | so hideci.js can be optimized to use that instead, right now it's pulling status.json fully | 15:59 |
| sdague | and just pulling out the changes it cares about | 15:59 |
| jeblair | sdague: also, when i look at firebug, i see it stacking up queries every few seconds even though they aren't finished | 15:59 |
| reed | fungi, shall we use this channel or community? | 15:59 |
| jeblair | reed: this one please | 15:59 |
| fungi | reed: this | 15:59 |
| reed | cool | 16:00 |
| reed | we can start, mrmartin is here too | 16:00 |
| jeblair | sdague: so i have a backlog of 12 http requests | 16:00 |
| *** jschwarz has quit IRC | 16:00 | |
| *** harlowja_at_home has joined #openstack-infra | 16:00 | |
| *** emagana has joined #openstack-infra | 16:00 | |
| fungi | reed: mrmartin: i'll go ahead and stop services on both the old (166.78.49.245) and new (104.239.140.123) machines now | 16:01 |
| reed | ACK | 16:01 |
| sdague | jeblair: it's using setTimeout so that's probably expected under this circumstance. Could be optimized. | 16:01 |
| mrmartin | fungi re | 16:01 |
| *** asselin has quit IRC | 16:02 | |
| jeblair | sdague: would you please take on the tasks of (a) optimizing that and (b) using the /status/change url? | 16:02 |
| jeblair | (it looks like zuul's load average has also doubled) | 16:02 |
| *** stevemar has joined #openstack-infra | 16:02 | |
| jeblair | sdague: (also, if it isn't already, it would be good to use the jquery visibility plugin so that it doesn't query when the page isn't visible) | 16:03 |
| *** annegentle has quit IRC | 16:03 | |
| sdague | jeblair: I'm not going to be able to get to it for at least a couple of days, I'm in the middle of some grenade refactoring here. I think jhesketh had taken over the patch, so maybe he'd have time to poke at it? | 16:04 |
| krotscheck | Can I get someone to look at the storyboard database and check to see if a table named 'may_mutate-to' exists? | 16:04 |
| krotscheck | 'may_mutate_to' | 16:04 |
| krotscheck | We've apparently got some broken migrations. | 16:04 |
| *** tqtran has joined #openstack-infra | 16:04 | |
| openstackgerrit | Stefano Maffulli 'reed' proposed openstack-infra/infra-manual: Creates a sandbox repo tutorial https://review.openstack.org/138195 | 16:07 |
| jeblair | krotscheck: ack | 16:09 |
| fungi | mrmartin: okay, i stopped nginx and tomcat7 but ended up having to kill supervisord to get rid of the uwsgi processes | 16:09 |
| *** marun has joined #openstack-infra | 16:09 | |
| fungi | i couldn't figure out what initscript controlled that (if indeed there was one at all) | 16:09 |
| mrmartin | fungi: ok. | 16:10 |
| *** Evgeny has joined #openstack-infra | 16:10 | |
| fungi | mrmartin: how have you been manually creating the database dumps? don't want to lose changes from the db since 05:09 utc today | 16:11 |
| *** marcusvrn2 has joined #openstack-infra | 16:11 | |
| *** unicell1 has quit IRC | 16:11 | |
| mrmartin | simple pg_dump | 16:11 |
| jeblair | krotscheck: no tables with "mut" in their names. (also, a hyphen in a table name is awkward.) | 16:11 |
| jeblair | krotscheck: alembic version 042 | 16:11 |
| *** adam_g_out is now known as adam_g | 16:12 | |
| Evgeny | mrmartin: it's good you can also add --exclude-table django_session | 16:12 |
| *** armax has joined #openstack-infra | 16:12 | |
| fungi | Evgeny: thanks! | 16:12 |
| Evgeny | mrmartin: this will reduce size of dump | 16:12 |
| fungi | trying to map the command-line arguments to the ones we're using for the psql cli | 16:13 |
| Evgeny | mrmartin: but on the new machine one should do 'python manage.py syncdb' to rebuild the table | 16:13 |
| *** marcusvrn1 has quit IRC | 16:13 | |
| *** e0ne has quit IRC | 16:13 | |
| mrmartin | Evgeny, yeap, it is included in the spec. | 16:13 |
| krotscheck | jeblair: Thanks | 16:13 |
| mrmartin | fungi: /srv/ask/node2/cron/backup.sh | 16:14 |
| krotscheck | jeblair: We're looking for alembic version 47. | 16:14 |
| fungi | mrmartin: thanks, that's much easier. i discovered from the manpage they're basically the same except you just give the database name as an argument rather than passing it via the -d option | 16:14 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Added possibility to use non-existent keys https://review.openstack.org/101580 | 16:16 |
| *** zul has joined #openstack-infra | 16:16 | |
| fungi | as soon as this database dump finishes, i'll copy it to the new machine and start the import steps we've got documented | 16:17 |
| fungi | there we go | 16:18 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Initial work for supporting downstream-ext https://review.openstack.org/158117 | 16:18 |
| Evgeny | fungi: there is also directory with uploaded files /srv/ask/upfiles | 16:18 |
| fungi | Evgeny: yep, tarring that up too per the backup script | 16:18 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Tasks may now be un-assigned. https://review.openstack.org/159948 | 16:18 |
| reed | thinking of this, we could have put a redirect on ask.openstack.org :/ | 16:19 |
| fungi | reed: a redirect to www.openstack.org? | 16:20 |
| reed | fungi, wrong word redirect... a 'pardon our dust' page | 16:21 |
| fungi | the dust will clear shortly | 16:21 |
| reed | yeah :) | 16:22 |
| *** koolhead17 has joined #openstack-infra | 16:23 | |
| *** alexpilotti has joined #openstack-infra | 16:23 | |
| fungi | okay, database dump and static files tarball have been copied to the new server, getting them loaded now | 16:25 |
| *** _nadya_ has joined #openstack-infra | 16:26 | |
| *** _nadya_ has quit IRC | 16:26 | |
| jeblair | krotscheck: what's the storyboard sitrep? | 16:27 |
| *** btran has joined #openstack-infra | 16:28 | |
| clarkb | morning | 16:28 |
| krotscheck | jeblair: Storyboard's up, some migrations are missing, I was just able to run the migrations locally, so that's not an issue. I suspect something in the puppet module borked. | 16:28 |
| krotscheck | jeblair: And by 'up' I mean it's running. | 16:28 |
| krotscheck | jeblair: But apparently not functional | 16:28 |
| jeblair | krotscheck: yeah agreed | 16:29 |
| clarkb | sdague: have you had a chance to look at https://review.openstack.org/#/c/107267/ ? | 16:29 |
| krotscheck | jeblair: So, I can go look at the puppet module and see if something there made the migrations not go. | 16:29 |
| fungi | database dump has been successfully imported on the new ask.openstack.org server. sync'ing and running migrations on it now | 16:29 |
| jeblair | krotscheck: do you want to try to fix forward or should we pin storyboard to a previous git revision? | 16:29 |
| krotscheck | But since I don't have access to the server I can't see what's going on there. | 16:29 |
| *** tiswanso has quit IRC | 16:30 | |
| krotscheck | jeblair: Depends on how easy the fix is. It's not my code either ;/ | 16:30 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Don't run check-tempest-dsvm-full-ceph job on stable icehouse/juno https://review.openstack.org/170912 | 16:30 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Make check-tempest-dsvm-full-ceph voting for cinder/glance/nova https://review.openstack.org/170913 | 16:30 |
| jeblair | krotscheck: when do you think it broke? | 16:30 |
| *** ivar-lazzaro has quit IRC | 16:30 | |
| *** tiswanso has joined #openstack-infra | 16:30 | |
| reed | fungi, woot | 16:31 |
| jeblair | clarkb: how do i convince puppetboard to show me older reports for a node? | 16:31 |
| krotscheck | jeblair:3AM this morning | 16:31 |
| *** sputnik13 has quit IRC | 16:31 | |
| clarkb | jeblair: I don't think you can | 16:31 |
| krotscheck | jeblair: or sometime around then. | 16:31 |
| jeblair | krotscheck: utc? | 16:31 |
| *** baoli has quit IRC | 16:31 | |
| fungi | making sure puppet is up to date on the new ask server too before i go starting services | 16:31 |
| *** dboik_ has joined #openstack-infra | 16:31 | |
| *** dboik_ has quit IRC | 16:31 | |
| fungi | mrmartin: btw we're still getting this error from puppet... | 16:32 |
| fungi | Error: /Stage[main]/Openstack_project::Ask/File[/usr/share/solr/WEB-INF/lib/lucene-analyzers-smartcn-4.7.2.jar]: Could not evaluate: Could not retrieve information from environment production source(s) file:/tmp/solr-4.7.2/contrib/analysis-extras/lucene-libs/lucene-analyzers-smartcn-4.7.2.jar | 16:32 |
| *** dboik_ has joined #openstack-infra | 16:32 | |
| mrmartin | fungi: I know | 16:32 |
| fungi | presuambly not urgent to address | 16:32 |
| openstackgerrit | Merged openstack-infra/system-config: Ensure the latest version of bandersnatch https://review.openstack.org/167382 | 16:32 |
| *** pal has quit IRC | 16:32 | |
| mrmartin | it is the smartcn analyzer copy script, and if don't have the extracted solr in tmp, it fails | 16:32 |
| krotscheck | jeblair: ~2000... I think? | 16:33 |
| *** ZZelle_ has joined #openstack-infra | 16:33 | |
| krotscheck | jeblair: https://review.openstack.org/#/c/163412/9 | 16:33 |
| mrmartin | but the /usr/share/solr/WEB-INF/lib/lucene-analyzers-smartcn-4.7.2.jar must be exist | 16:33 |
| krotscheck | Wait. | 16:33 |
| krotscheck | I'm wrong. If 42 is the last we have, then 43 never migrated either. | 16:33 |
| fungi | mrmartin: Evgeny: reed: services are up and i'm rebuilding the solr indices now | 16:34 |
| jeblair | clarkb: we really need to be able to see older runs :( | 16:34 |
| reed | yay | 16:34 |
| krotscheck | jeblair: So sometime before march 30th | 16:34 |
| clarkb | jeblair: this was one of the documented loss of functionalities that we knew about when moving to puppetboard | 16:34 |
| clarkb | but the old dashboard was inoperable so it was seem as an improvement | 16:34 |
| jeblair | clarkb: i believe the alternative was "nothing", so we still made the right choice there. but we still need it. | 16:34 |
| *** dboik has quit IRC | 16:35 | |
| *** baoli has joined #openstack-infra | 16:35 | |
| nibalizer | jeblair: infinity older runs or just > 10? | 16:36 |
| nibalizer | because we get >10 as soon as we upgrade it | 16:36 |
| nibalizer | we just never have | 16:36 |
| jeblair | nibalizer: how many do we get when we upgrade? | 16:36 |
| nibalizer | its a config option iirc | 16:37 |
| fungi | mrmartin: Evgeny: reed: i went ahead and updated the static files from the tarball while waiting on solr reindexes to finish. english just finished and chinese is going now | 16:37 |
| nibalizer | https://github.com/puppet-community/puppetboard/blob/master/puppetboard/default_settings.py#L13 | 16:37 |
| jeblair | nibalizer: i would say that our requirement is satisfied as N approaches infinity :) | 16:37 |
| fungi | oh, we had very little chinese to reindex | 16:37 |
| fungi | search test looks okay | 16:38 |
| fungi | restarting celeryd and apache2 one last time | 16:38 |
| jeblair | nibalizer: i think we could live with setting it to, say, 115792089237316195423570985008687907853269984665640564039457584007913129639936... | 16:38 |
| fungi | mrmartin: Evgeny: reed: want to check anything before i update dns to the new addresses? | 16:38 |
| *** adalbas has joined #openstack-infra | 16:38 | |
| jeblair | krotscheck: http://paste.openstack.org/show/198620/ | 16:39 |
| reed | fungi, checking | 16:39 |
| nibalizer | jeblair: heh, we could deff tune it, at some point you'll be waiting so long for jinja and puppetdb that you won't want it that high | 16:40 |
| nibalizer | but i think i see what you're saying | 16:40 |
| *** thingee has joined #openstack-infra | 16:40 | |
| nibalizer | I usually use puppetboard as a 'how are things now' and you want to be able to use it to get old reports to figure out what happened | 16:40 |
| krotscheck | jeblair: whoooooaaaa.a... | 16:41 |
| clarkb | puppet runs once every 45 minutes, thats 32 runs per day. 5760 reports is approximately 6 months of reports per node | 16:41 |
| mrmartin | search works for me | 16:41 |
| jeblair | nibalizer: yes, i want anyone to be able to see why something broke. we probably need to be able to access several days worth of reports for that. | 16:41 |
| *** unicell has joined #openstack-infra | 16:42 | |
| jeblair | nibalizer: (specifically, krotscheck should have been able to see the thing i just had to go look up for him) | 16:42 |
| *** ddieterly has quit IRC | 16:42 | |
| jeblair | nibalizer: (but it happened more than 10 puppet runs ago, so it was invisible) | 16:42 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: bogus commit just for testing https://review.openstack.org/170918 | 16:42 |
| reed | fungi, looks like it works | 16:42 |
| *** ddieterly has joined #openstack-infra | 16:42 | |
| clarkb | if we do a week thats ~225 repots | 16:42 |
| mrmartin | fungi, reed: both english and chinese search works well | 16:42 |
| *** ddieterly has quit IRC | 16:42 | |
| fungi | okay, any last objections before i switch the dns records to bring the new server live? | 16:42 |
| reed | fungi, Evgeny, mrmartin all seems to be fine. Ship it! | 16:43 |
| *** ddieterly has joined #openstack-infra | 16:43 | |
| *** david-lyle_ has quit IRC | 16:43 | |
| Evgeny | reed: I think here I can only help you test specific malfunctions, mostly waiting to respond to any of those | 16:44 |
| reed | Evgeny, thanks, :) | 16:44 |
| fungi | mrmartin: Evgeny: reed: dns updated to 104.239.140.123 and 2001:4800:7819:104:be76:4eff:fe05:b1be (ttl has been 5 minutes for a while on those so should be taking effect for you shortly) | 16:44 |
| nibalizer | jeblair: the situation is we need a newer puppetboard for that, and the module doesn't know how to update puppetboard | 16:44 |
| clarkb | fungi: woot | 16:45 |
| jeblair | fungi: wfm | 16:45 |
| fungi | i'll keep the ttl on them down at 5 minutes until at least tomorrow in case we discover that we need to switch back for some reason | 16:45 |
| reed | already switched for me | 16:45 |
| fungi | accepting the new ssh host key for the replacement ask.openstack.org server on our puppetmaster now as well | 16:46 |
| Evgeny | reed: quite soon we'll need to update the site to migrate to new Google authentication | 16:46 |
| jeblair | nibalizer: cool. so maybe we can file a story for that when storyboard is back up. | 16:46 |
| reed | Evgeny, yes, indeed | 16:46 |
| Evgeny | reed: by next week the theme update will be ready | 16:46 |
| mrmartin | fungi, reed: DNS switched here too | 16:46 |
| reed | Evgeny, mrmartin let's talk about the upgrade, what do we need to do? | 16:47 |
| Evgeny | reed: I think I might need to contribute to your puppet repo, depending on how it all works | 16:47 |
| mrmartin | reed, what you like to upgrade? the theme? | 16:47 |
| reed | mrmartin, Evgeny: I'd like also to add OpenStackID to the mix of auth sources | 16:47 |
| *** chandankumar has joined #openstack-infra | 16:48 | |
| fungi | i've tested out a handful of questions i already had in my history and they seem to be loading fine | 16:48 |
| Evgeny | mrmartin: the theme will have to be updated (or at least checked) to use the next version. | 16:48 |
| reed | mrmartin, the code, too, I think, to support google openidconnect | 16:48 |
| Evgeny | mrmartin: also I'll actually need to make a new release of Askbot off what's in the repo now | 16:48 |
| reed | google dropped openid2 | 16:48 |
| mrmartin | Evgeny: this links to master github repo: https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/manifests/ask.pp#L97 | 16:49 |
| mrmartin | so be careful here | 16:49 |
| clarkb | fungi: its loading here too | 16:49 |
| mrmartin | https://github.com/openstack-infra/puppet-askbot/blob/master/manifests/init.pp#L18 | 16:49 |
| Evgeny | mrmartin, ok looks like it's reading the existing theme repo | 16:49 |
| mrmartin | and if we have a new release for pip, than the version must be applied to system-config/../ask.pp | 16:50 |
| Evgeny | mrmartin, ok looks like it's reading the existing theme repo. Will it be then enough to only update the askbot version in the puppet spec? | 16:50 |
| Evgeny | ok | 16:50 |
| mrmartin | need to update in the system-config repo | 16:50 |
| Evgeny | ok, then possibly there will be commits to the theme repo + update of askbot version | 16:51 |
| Evgeny | mrmartin: is there staging instance? | 16:51 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Make (check|gate)-tempest-dsvm-full-ceph voting for cinder/glance/nova https://review.openstack.org/170913 | 16:51 |
| clarkb | nibalizer: what would need to be done tohave the puppet module update puppetboard? | 16:52 |
| mrmartin | Evgeny, before you do any updates in the theme, we should pin the theme github on ask.o.o to a specific commit | 16:52 |
| *** sarob has joined #openstack-infra | 16:52 | |
| clarkb | nibalizer: could we have the module accept a version and when that chagnes perform an upgrade to that new version? | 16:52 |
| Evgeny | mrmartin, sounds good | 16:52 |
| *** davideagnello has joined #openstack-infra | 16:52 | |
| mrmartin | so you can freely contribute to the master branch | 16:52 |
| anteaya | fungi: got pulled away, https://ask.openstack.org/en/questions/ renders for me | 16:53 |
| krotscheck | jeblair: Can you run this against the storyboard db for me? http://paste.openstack.org/show/198636/ | 16:53 |
| fungi | anteaya: no worries. we're only just now at the point where it should be resolving properly in dns for everyone | 16:53 |
| fungi | thanks for testing | 16:53 |
| nibalizer | clarkb: ya something like that | 16:53 |
| krotscheck | jeblair: The projects where the second name column is 'null' are the ones I care about. | 16:53 |
| anteaya | fungi: navigation and sign in seem to be working as I expect | 16:54 |
| *** annegentle has joined #openstack-infra | 16:55 | |
| *** sarob has quit IRC | 16:55 | |
| fungi | awesome. so reed's announced 3-4 hour window seems to have completed in well under 1 hour. let me know if anyone turns up a problem indicating we need to go back | 16:55 |
| reed | fungi, so far so good | 16:55 |
| fungi | but at this point is seems to have been completely successful | 16:55 |
| reed | thank you | 16:55 |
| fungi | er, it | 16:55 |
| fungi | you're welcome | 16:56 |
| mrmartin | awesome | 16:56 |
| reed | mrmartin, Evgeny: I'd like to lay down a plan for the upgrade | 16:56 |
| reed | maybe move to #openstack-community to discuss this with less noise and come back here with a proposal? | 16:56 |
| mrmartin | sure. | 16:57 |
| *** yamamoto has quit IRC | 16:57 | |
| *** sarob has joined #openstack-infra | 16:57 | |
| jeblair | krotscheck: ack | 16:58 |
| clarkb | cinerama: were you still planning on updating 164011 with my suggestion? | 16:58 |
| *** dizquierdo has joined #openstack-infra | 16:58 | |
| clarkb | cinerama: gah looks like you did, but my local gertty doesn't see it for some reason | 16:59 |
| cinerama | clarkb: i posted another patch on friday that incorporates it | 16:59 |
| clarkb | cinerama: ya, sorry for the noise, reviewing it now | 16:59 |
| jeblair | krotscheck: http://paste.openstack.org/show/198637/ | 16:59 |
| *** harlowja_at_home has quit IRC | 17:00 | |
| Evgeny | Reed, mrmartin: I will do theme update and release of Askbot, then let you know by Monday next week the latest | 17:00 |
| jeblair | krotscheck, fungi, clarkb: i think we missed a step in project renames? | 17:00 |
| krotscheck | Ah-hah | 17:00 |
| jeblair | krotscheck, fungi, clarkb: oh, actually, those probably all appeared as new projects for storyboard. nevermind. | 17:00 |
| mrmartin | Evgeny, join to #openstack-community plz. | 17:00 |
| Evgeny | reed, mrmartin: meanwhile pinning to specific commit of theme can be done | 17:00 |
| *** hdd has quit IRC | 17:01 | |
| krotscheck | Gut reaction tells me that the project import script doesn't create new branches. | 17:01 |
| jeblair | ah | 17:01 |
| *** pal has joined #openstack-infra | 17:02 | |
| clarkb | cinerama: commented I think there is one small bug in your erb loop | 17:04 |
| clarkb | cinerama: please let me know if I am wrong :) | 17:04 |
| krotscheck | jeblair: Yep. It doesn't. Oversight in code review. | 17:04 |
| cinerama | clarkb: guilty as charged | 17:05 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add checking of multijob phase condition. https://review.openstack.org/164669 | 17:05 |
| krotscheck | jeblair: No. I lied. Dagnabbit | 17:06 |
| jeblair | clarkb: are you subscribed to puppet-zanata? | 17:06 |
| clarkb | cinerama: but otherwise looks good, and I agree we can sort out the http listener in a different change | 17:06 |
| clarkb | jeblair: yes | 17:06 |
| jeblair | clarkb: (or more to the point, were you before friday?) | 17:06 |
| clarkb | I had to do a hard refresh to get the latest patchset to show up | 17:06 |
| clarkb | jeblair: yes I was | 17:06 |
| greghaynes | If things are slowish https://review.openstack.org/#/c/165240/ and https://review.openstack.org/#/c/164927/ (resurrect wheel building) have been waiting for reviews for a while | 17:06 |
| jeblair | clarkb: bummer. i wonder how that slipped through. perhaps we need to increase the window of overlap. | 17:07 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Handle NullChange "merges" gracefully. https://review.openstack.org/85217 | 17:07 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 17:07 |
| jeblair | clarkb: any chance you are still running the same gertty process you were running on friday and it's debug logging? :) | 17:07 |
| clarkb | jeblair: I am but without debug logging | 17:08 |
| *** tonytan4ever has quit IRC | 17:08 | |
| *** baoli has quit IRC | 17:09 | |
| krotscheck | jeblair: One last thing, can you run this? "select count(id) from storyboard.branches where project_id = NULL" to see if you get any results? | 17:09 |
| jeblair | clarkb: oh well. don't think i could do much without a debug log. | 17:09 |
| clarkb | ok | 17:09 |
| jeblair | krotscheck: 0 results | 17:09 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Enable a check job for networking-ovn https://review.openstack.org/170927 | 17:09 |
| krotscheck | jeblair: Hrmph. | 17:09 |
| krotscheck | jeblair: Thanks | 17:09 |
| clarkb | greghaynes: reviewed, one small thing to consider inline | 17:12 |
| *** baoli has joined #openstack-infra | 17:13 | |
| mattymo | ping infra:) https://review.openstack.org/#/c/155309/ was marked +2 and workflow +1, but it didn't get merged | 17:13 |
| greghaynes | clarkb: ah, ok | 17:13 |
| mattymo | clarkb, jeblair fungi ^? | 17:13 |
| clarkb | mattymo: that may have been lost due to gerrit stopping its event stream | 17:13 |
| clarkb | mattymo: try rechecking it | 17:14 |
| fungi | mattymo: looking. if it happened between 10:56 and 11:37 utc it may need its review vote readded | 17:14 |
| clarkb | or that^ | 17:14 |
| *** baoli has quit IRC | 17:14 | |
| fungi | er, its workflow vote | 17:14 |
| *** tiswanso has quit IRC | 17:14 | |
| mattymo | ok I'll try that. thanks! | 17:14 |
| anteaya | mattymo: no, it was +1'd on March 30th | 17:14 |
| *** achanda has joined #openstack-infra | 17:15 | |
| anteaya | I don't know why it didn't merge | 17:15 |
| fungi | oh, it has no verify vote from jenkins | 17:15 |
| fungi | a recheck comment will fix that | 17:15 |
| mattymo | simply recheck on its own in a comment? | 17:16 |
| *** MarkAtwood has quit IRC | 17:16 | |
| openstackgerrit | Stephanie Miller proposed openstack-infra/puppet-zanata: Add apache configuration for zanata https://review.openstack.org/164011 | 17:16 |
| fungi | patchset 4 was uploaded when we were having issues back on march 26 (similar problem) and so zuul never saw it | 17:16 |
| fungi | mattymo: yes, just a review comment that says recheck | 17:16 |
| anteaya | fungi: ah | 17:17 |
| *** YorikSar has quit IRC | 17:17 | |
| fungi | or at least as long as it starts wit the work recheck (whatever you write after the work recheck is irrelevant to triggering zuul) | 17:17 |
| *** tiswanso_ has joined #openstack-infra | 17:17 | |
| fungi | er, with the word | 17:17 |
| * fungi sighs. typing is hard | 17:17 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Enable a check job for networking-ovn https://review.openstack.org/170927 | 17:18 |
| mattymo | didn't seem to help | 17:18 |
| anteaya | mattymo: it is queued | 17:19 |
| JayF | What's a reasonable way to troubleshoot post jobs? I had a patch merge on a project, wanted to ensure post jobs ran properly, but I haven't seen them show up yet in status.openstack.org/zuul even after it's been merged for 5m+ | 17:19 |
| *** baoli_ has joined #openstack-infra | 17:19 | |
| mattymo | anteaya, thanks. I'll just hang tight | 17:19 |
| *** Evgeny has left #openstack-infra | 17:19 | |
| anteaya | mattymo: status.openstack.org/zuul | 17:19 |
| JayF | Hmm. Searching using the box on the page showed it to me when I couldn't see it before | 17:19 |
| anteaya | mattymo: if it is on that page it is in the works | 17:19 |
| clarkb | JayF: those wouldn't be the jobs that just showed up as queued for IPA would they? | 17:19 |
| clarkb | JayF: I think zuul was just processing a large set of events all at once | 17:20 |
| anteaya | JayF: Queue lengths: 33 events, 123 results | 17:20 |
| *** sdake_ has joined #openstack-infra | 17:20 | |
| JayF | clarkb: Probably were those; I was trying to find it and couldn't. | 17:20 |
| JayF | Queue length is queue for zuul to actually process the job needs to be started? | 17:20 |
| JayF | Aaah | 17:20 |
| anteaya | JayF: look at the queue lenths, top right corner | 17:20 |
| *** skolekonov is now known as _skolekonov | 17:20 | |
| anteaya | yes | 17:20 |
| anteaya | or do something with them | 17:20 |
| fungi | yeah, if something's not showing up there right away, wait for the events/results queues to get back around 0 | 17:20 |
| krotscheck | jeblair: Ok, so this should unblock the migration. I'm going to poke NikitaKonovalov to ask arpininen to triage why it's broken in the first place, as that's not really a code block I'm that familiar with. http://paste.openstack.org/show/198639/ | 17:20 |
| JayF | That makes perfect sense, I never realized that's what those numbers were for. Thanks! | 17:21 |
| anteaya | events are things taht happened that zuul ahsn't done something with yet, results are things coming back that haven't been posted yet | 17:21 |
| *** yamahata has quit IRC | 17:22 | |
| *** sdake has quit IRC | 17:23 | |
| krotscheck | jeblair: This would be the less skynetty verion of that: http://paste.openstack.org/show/198640/ | 17:23 |
| jeblair | krotscheck: there's no restricted column | 17:24 |
| krotscheck | jeblair: Ah. That one's in migration 46 | 17:25 |
| jeblair | insert into storyboard.branches (project_id, name, created_at, expired, autocreated) SELECT pb.id, 'master', NOW(), 0, 0 from (SELECT p.id, p.name, b.name as branch FROM storyboard.projects as p left outer join storyboard.branches as b on b.project_id = p.id) as pb where pb.branch IS NULL; | 17:25 |
| jeblair | krotscheck: so i'll do that ^ ? | 17:25 |
| krotscheck | jeblair: Yep, that should do it. | 17:25 |
| krotscheck | jeblair: At least, that'll unblock the migrations. | 17:25 |
| * krotscheck hopes. | 17:25 | |
| jeblair | krotscheck: done; the earlier query looks good now | 17:26 |
| krotscheck | jeblair: So the migration ran? | 17:27 |
| jeblair | krotscheck: no, i'm going to run it manually now | 17:27 |
| *** HeOS has quit IRC | 17:27 | |
| *** patrickeast has joined #openstack-infra | 17:28 | |
| jeblair | krotscheck: done, no output | 17:28 |
| krotscheck | jeblair: Ex ept storyboard is back up now :) | 17:28 |
| jeblair | krotscheck: yep, seems to be working | 17:28 |
| krotscheck | jeblair: Cool. I'll try to figure out what actually happened. | 17:29 |
| jeblair | krotscheck: thanks | 17:29 |
| *** rm_work is now known as rm_work|away | 17:29 | |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: use optimized url for zuul status https://review.openstack.org/170935 | 17:30 |
| sdague | jeblair: so... I think that addresses both the network optimization and the stacking up of requests | 17:30 |
| jeblair | sdague: oh cool, thanks! | 17:30 |
| *** stevemar has quit IRC | 17:30 | |
| sdague | however, I'm getting timeouts hitting zuul status from my computer, so I can't test that it actually works correctly | 17:30 |
| *** stevemar has joined #openstack-infra | 17:31 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:31 | |
| jeblair | sdague: well, does it at least not stack up requests while it's waiting for one to timeout? i guess you mean you've tested the negative case, just not the positive one? :) | 17:31 |
| sdague | jeblair: right, the negative case is right now | 17:32 |
| sdague | no idea if it still does the correct things :) | 17:32 |
| jeblair | sdague: yeah, zuul's LA is >6 now | 17:32 |
| *** julim has quit IRC | 17:34 | |
| sdague | oh, hey, some are returning | 17:34 |
| sdague | XMLHttpRequest cannot load https://zuul.openstack.org/status/change/170885,2. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://review.openstack.org' is therefore not allowed access. The response had HTTP status code 404. | 17:35 |
| sdague | ok, is there a new thing needed in zuul for that? | 17:35 |
| jeblair | sdague: probably an apache config update | 17:35 |
| clarkb | you need to update zuul's apache vhost to allow CORS requests | 17:35 |
| jeblair | clarkb: it already does | 17:35 |
| clarkb | oh | 17:35 |
| sdague | clarkb: right, we did for status.json right? | 17:36 |
| sdague | that's the part I'm confused on | 17:36 |
| jeblair | clarkb: it just may be fairly specific | 17:36 |
| clarkb | I bet its url specific | 17:36 |
| jeblair | so the new url may not match | 17:36 |
| clarkb | ya | 17:36 |
| jeblair | so how about we pull this from production now | 17:36 |
| jeblair | so we stop killing zuul | 17:36 |
| sdague | jeblair: wfm | 17:36 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Pass through git revision to triggered jobs https://review.openstack.org/163149 | 17:36 |
| fungi | yeah, our apache config is all specific to status.json so far | 17:37 |
| jeblair | then add it back when we have all of these problems fixed | 17:37 |
| jeblair | i'll work on the revert | 17:37 |
| fungi | RewriteRule ^/status.json$ http://127.0.0.1:8001/status.json [P] | 17:37 |
| *** julim has joined #openstack-infra | 17:37 | |
| fungi | and so on | 17:37 |
| *** baoli_ has quit IRC | 17:38 | |
| *** baoli has joined #openstack-infra | 17:38 | |
| fungi | and yes, i second reverting for the moment. i believe it's responsible for me having to restart my browser if i leave it on a gerrit review for an hour or more | 17:38 |
| *** marcusvrn has quit IRC | 17:39 | |
| jeblair | fungi: OH THAT'S what's doing that! | 17:39 |
| fungi | indeed | 17:39 |
| fungi | i started coming around to that realization just this morning | 17:40 |
| *** otter768 has joined #openstack-infra | 17:40 | |
| rcarrillocruz | may I please get some reviews on https://review.openstack.org/#/c/168572/2 ? | 17:41 |
| rcarrillocruz | clarkb: ^ you reviewed a previous patchset | 17:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Revert zuul status in gerrit https://review.openstack.org/170936 | 17:42 |
| sdague | jeblair: instead of that, could we just comment out L337 ? | 17:43 |
| sdague | which is the entry point to the new code | 17:43 |
| sdague | just to make working on fixing it a bit simpler | 17:44 |
| anteaya | and so you are aware, I never saw any zuul status on my gerrit ui pages | 17:44 |
| anteaya | it looks like I was supposed to | 17:44 |
| *** otter768 has quit IRC | 17:45 | |
| fungi | anteaya: at https://review.openstack.org/170936 i see "review currently being tested (full status) ... in between pipelines, status should update shortly" | 17:45 |
| fungi | anteaya: it's appearing where the results table would normally be | 17:46 |
| *** ashleighfarnham has joined #openstack-infra | 17:46 | |
| fungi | anteaya: that text doesn't show up for you? | 17:46 |
| anteaya | fungi: http://imgur.com/KkBYh8d | 17:46 |
| sdague | anteaya: also, right now, it's timing out a lot | 17:47 |
| anteaya | I do not see that text | 17:47 |
| clarkb | rcarrillocruz: I have a slight preference for setting the puppetmaster to not undef in puppetmaster.pp | 17:47 |
| anteaya | perhaps I am in the timeout crowd | 17:47 |
| *** yamahata has joined #openstack-infra | 17:47 | |
| clarkb | rcarrillocruz: then anyone else can update their calls to that class to pass in an override | 17:47 |
| clarkb | rcarrillocruz: we don't need to do a hiera lookup for that | 17:47 |
| rcarrillocruz | so, same change but requiring the param | 17:48 |
| rcarrillocruz | ? | 17:48 |
| *** flaviosr has joined #openstack-infra | 17:48 | |
| jeblair | sdague: that would work too, but you can revert this change and stack your change on top of it fairly easily, right? | 17:48 |
| anteaya | fwiw I'm not opposed to sdague's suggestion of how to proceed but I am for reducing load on zuul | 17:48 |
| clarkb | rcarrillocruz: you don't need to require the param just s/undef/puppetmaster.openstack.org/ | 17:48 |
| clarkb | rcarrillocruz: then remove your site.pp changes | 17:48 |
| sdague | jeblair: well the inbrowser development is simpler if the infrastructure for it is mostly fully loaded. | 17:49 |
| sdague | anyway, it's a suggestion | 17:49 |
| *** YorikSar has joined #openstack-infra | 17:49 | |
| *** mrmartin has quit IRC | 17:51 | |
| rcarrillocruz | ok | 17:51 |
| jeblair | fungi, clarkb, zaro: so i think we're seeing https://issues.apache.org/jira/browse/SSHD-348 | 17:51 |
| *** sputnik13 has joined #openstack-infra | 17:51 | |
| jeblair | fungi: have you looked at thread stack traces in gerrit before restarting? | 17:51 |
| jeblair | i looked just now, and i see one that appears to be waiting | 17:52 |
| jeblair | SSH-Stream-Worker-9 | 17:52 |
| clarkb | looks like we want to build a gerrit with version 0.14 of that lib? | 17:53 |
| jeblair | did we accidentally change something during the upgrade? | 17:53 |
| clarkb | jeblair: we would've changed the java libs we install from the system pacakges | 17:53 |
| clarkb | but that was intentional | 17:53 |
| jeblair | clarkb: yeah, but that shouldn't include the ssh lib, right? | 17:54 |
| jeblair | clarkb: that should be in the gerrit build itself? | 17:54 |
| zaro | jeblair: it's in gerrit build | 17:54 |
| clarkb | it does not include the ssh lib but does include bouncy castle | 17:54 |
| clarkb | which the ssh lib may use | 17:54 |
| zaro | this ssh fix was patched for 2.9 | 17:55 |
| clarkb | bcpg, bcpkix, bcprov, and mysql-connector are the libs installed that way | 17:55 |
| jeblair | zaro: right, istr talking about this wrt to 2.9. but we're seeing the problem (i think) on 2.8 | 17:55 |
| *** MarkAtwood has joined #openstack-infra | 17:56 | |
| jeblair | clarkb: did we get new versions of those when we installed the new server? (and do we not update those automatically?) | 17:56 |
| *** baoli has quit IRC | 17:56 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Expose puppetmaster_server param on openstack_project::puppetmaster https://review.openstack.org/168572 | 17:56 |
| clarkb | jeblair: we get new versions of them because they come from trusty not precise, and they are updated automatically by the system package manager | 17:56 |
| fungi | jeblair: i didn't realize you could capture thread stack traces for those. i've definitely seen stream-events tasks piling up in the show-queue output when we get into a bad way | 17:57 |
| rcarrillocruz | clarkb: ^ | 17:57 |
| rcarrillocruz | thx | 17:57 |
| *** tonytan4ever has joined #openstack-infra | 17:57 | |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: use optimized url for zuul status https://review.openstack.org/170935 | 17:57 |
| jeblair | fungi: https://review.openstack.org/monitoring then mouseover SSH-Stream-Worker-9 | 17:57 |
| jeblair | fungi: or https://review.openstack.org/monitoring?part=threadsDump | 17:57 |
| fungi | oh neat | 17:57 |
| *** sarob has quit IRC | 17:59 | |
| *** dboik_ has quit IRC | 17:59 | |
| *** baoli_ has joined #openstack-infra | 18:00 | |
| *** dboik has joined #openstack-infra | 18:00 | |
| fungi | though i thought the java ssh lib was baked-in, not externally linked? | 18:00 |
| jeblair | fungi, clarkb: yeah, i think it is | 18:00 |
| jeblair | see: https://gerrit-review.googlesource.com/#/c/65696/ | 18:00 |
| fungi | though maybe we're running a war we built? | 18:00 |
| fungi | and somehow updated to a different build than we had running on the precise server? | 18:01 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Project Loader creates default branches. https://review.openstack.org/170942 | 18:01 |
| sdague | well, if someone else figures out how to put jquery-visibility in place on the gerrit system, and the cors thing, https://review.openstack.org/170935 should be the entire set of hideci.js fixes. | 18:01 |
| krotscheck | jeblair: ^^ That fixes it. | 18:01 |
| jeblair | fungi, clarkb, zaro: http://git.openstack.org/cgit/openstack-infra/gerrit/tree/lib/mina/BUCK?h=openstack/2.8.4 | 18:03 |
| jeblair | id = 'org.apache.sshd:sshd-core:0.9.0.201311081', | 18:03 |
| jeblair | so that _should_ have an sshd that's not affected by that bug | 18:03 |
| *** sdake has joined #openstack-infra | 18:05 | |
| jeblair | actually, we're 1 commit behind that, but it still doesn't change that. | 18:07 |
| *** VijayTripathi has joined #openstack-infra | 18:07 | |
| fungi | [2015-04-06 11:37:16,178] INFO com.google.gerrit.sshd.SshDaemon : Started Gerrit SSHD-CORE-0.9.0.201311081 on *:29418 | 18:07 |
| fungi | so confirmed that's what gerrit thinks we're running | 18:07 |
| jeblair | nice | 18:07 |
| * fungi is flummoxed | 18:08 | |
| *** sdake_ has quit IRC | 18:09 | |
| *** esker has quit IRC | 18:09 | |
| clarkb | which is the version that _david_ points at downgrading too as a possible work around | 18:10 |
| *** yamahata has quit IRC | 18:10 | |
| zaro | hard to tell if going to ssh 0.14 requires any of that code change as well. | 18:10 |
| clarkb | we are affected by https://issues.apache.org/jira/browse/SSHD-330 though according to that thread | 18:11 |
| *** harlowja_away is now known as harlowja | 18:11 | |
| *** gulic has quit IRC | 18:11 | |
| clarkb | but that appears to be a completely unrelated issue | 18:12 |
| *** rhe00 has left #openstack-infra | 18:13 | |
| jeblair | i wish i knew what commit fixed that bug | 18:14 |
| jeblair | maybe this one: https://git-wip-us.apache.org/repos/asf?p=mina-sshd.git;a=commitdiff;h=964e76890cf56da4491199860d0ea8276fbd26a6 | 18:14 |
| clarkb | jeblair: which bug? 348 or 330? | 18:15 |
| jeblair | 348 | 18:15 |
| clarkb | https://git-wip-us.apache.org/repos/asf?p=mina-sshd.git;a=commit;h=2aed686bdb21681a421033c6ee5997e5cd8a9a83 is 330, https://git-wip-us.apache.org/repos/asf?p=mina-sshd.git;a=commitdiff;h=964e76890cf56da4491199860d0ea8276fbd26a6 appears to be 348 | 18:15 |
| jeblair | was looking to see if it suggested something that may have changed to trigger it. | 18:15 |
| zaro | it's it this one? https://gerrit-review.googlesource.com/#/c/62070/ | 18:16 |
| *** pal has quit IRC | 18:16 | |
| zaro | this one ^ fixes 330? | 18:17 |
| jeblair | did they ever figure out what was introduced to sshd to cause the bug? | 18:17 |
| zaro | not that i'm aware of. just confirmed that downgrade fixed it | 18:17 |
| zaro | https://gerrit-review.googlesource.com/#/c/65696/ fixed 348 ? | 18:18 |
| jeblair | i'm starting to wonder if the sshd bug already existed in 0.90, and something in trusty triggers it | 18:18 |
| jeblair | zaro: that doesn't fix 348, that uses the new version of sshd which contains the fix to 348 | 18:18 |
| jeblair | or perhaps the network path is different to the new server and causes stuck connections more often | 18:19 |
| jeblair | after all, we did see this once before on the old server | 18:19 |
| jeblair | it's just now we see it every few days on the new one | 18:19 |
| jeblair | it's also possible that a recently added third-party ci system is more apt to trigger the bug, and the timing near the server move is coincidence | 18:20 |
| *** AJaeger has quit IRC | 18:20 | |
| jeblair | unfortunately, i don't know a way to associate a connection with that stuck thread | 18:20 |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:22 | |
| *** dboik has quit IRC | 18:23 | |
| *** rm_work|away is now known as rm_work | 18:24 | |
| *** e0ne has joined #openstack-infra | 18:26 | |
| *** dboik has joined #openstack-infra | 18:26 | |
| *** rm_work is now known as rm_work|away | 18:28 | |
| *** hdd has joined #openstack-infra | 18:29 | |
| *** flaviosr has quit IRC | 18:30 | |
| Shrews | SpamapS: were you the one that added dogpile caching support to shade? | 18:33 |
| *** rm_work|away is now known as rm_work | 18:34 | |
| Shrews | SpamapS: ah, git log says yolanda | 18:34 |
| fungi | jeblair: also i've tried killing pretty much ever task in the queue and new stream-events calls still end up waiting forever | 18:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/gerrit: Add username to stream-events queue entries https://review.openstack.org/170948 | 18:34 |
| fungi | er, every | 18:34 |
| jeblair | fungi: oh, that is interesting | 18:34 |
| jeblair | fungi, clarkb, zaro: i'm hoping that https://review.openstack.org/170948 will tell us the account name of stuck stream event tasks | 18:35 |
| jeblair | fungi, clarkb, zaro: and we can see if there is a pattern there (eg, if one account in particular generates stuck tasks) | 18:35 |
| jeblair | that change is untested. uncompiled, even. | 18:35 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add NPM Command Execution as a Build Target https://review.openstack.org/169829 | 18:35 |
| *** teran_ has quit IRC | 18:37 | |
| *** teran has joined #openstack-infra | 18:37 | |
| *** teran has quit IRC | 18:42 | |
| clarkb | jeblair: I think you can map the connection id to a user already but let me double check on that | 18:42 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Project Group detail page improvements. https://review.openstack.org/169038 | 18:43 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Typo in Project Groups. https://review.openstack.org/169037 | 18:43 |
| *** dboik has quit IRC | 18:43 | |
| openstackgerrit | Merged openstack-infra/system-config: Revert zuul status in gerrit https://review.openstack.org/170936 | 18:44 |
| fungi | jeblair: clarkb: does the 4-byte hex string after the timestamp in each line of the sshd_log correspond to the 4-byte hex task id in the show-queue listing? | 18:45 |
| clarkb | fungi: I think it may be the 4 byte connection session id from show-connections | 18:45 |
| clarkb | trying to confirm that now | 18:45 |
| fungi | ahh | 18:45 |
| jeblair | clarkb: i was unable to confirm that, but i hope you do | 18:46 |
| fungi | yep, confirmed d76eebfe for murano-ci in show-connections matches a log line for the same account | 18:46 |
| clarkb | 8ac6b821 parallels-pcs-ci shows up in both too | 18:47 |
| jeblair | yeah, but that doesn't tie the connection to the queue | 18:47 |
| fungi | right, so dead end | 18:48 |
| greghaynes | If infra-ites want to review, I just made an image that booted in both rax and hp using https://review.openstack.org/#/c/167003/ | 18:48 |
| clarkb | jeblair: fungi correct | 18:48 |
| greghaynes | so in theory its getting close | 18:48 |
| SpamapS | Shrews: yolanda added caching, but I refactored her patch to use dogpile. | 18:48 |
| Shrews | SpamapS: doesn't seem to be working correctly, afaict: http://paste.openstack.org/show/198700/ | 18:49 |
| *** medieval1 has quit IRC | 18:50 | |
| *** dboik has joined #openstack-infra | 18:50 | |
| Shrews | SpamapS: no caching specified, so should be null driver. but the volume seems to be cached since it's status never goes from 'created' to 'available' | 18:50 |
| Shrews | SpamapS: and only a single call to _get_volumes_from_cloud() which retrieves it from the cloud | 18:50 |
| Shrews | SpamapS: i'm just confused and looking for dogpile experts | 18:50 |
| *** adalbas has quit IRC | 18:51 | |
| *** kutija has quit IRC | 18:52 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add NPM Command Execution as a Build Target https://review.openstack.org/169829 | 18:53 |
| clarkb | cinerama: +2 thanks! | 18:53 |
| jeblair | clarkb, fungi, mordred: i may be on a plane to pycon tomorrow, can one of you chair the infra meeting? | 18:54 |
| fungi | jeblair: sure, be my pleasure | 18:54 |
| cinerama | clarkb: thank you | 18:54 |
| jeblair | also, need me to do anything at pycon? :) | 18:54 |
| *** signed8bit_ZZZzz is now known as signed8bit | 18:55 | |
| SpamapS | Shrews: I'll take a look | 18:56 |
| *** bcourt has quit IRC | 18:56 | |
| SpamapS | clarkb: btw, is this Salt's answer to 0mq being not-good? https://github.com/saltstack/raet | 18:56 |
| greghaynes | SpamapS: I think their initial answer was the ssh transport | 18:57 |
| SpamapS | Shrews: can you share voltest.py ? | 18:57 |
| greghaynes | after they got burned by zmq and pyzmq folks pointing fingers at each other going "pyzmq3 will fix it all!" | 18:57 |
| greghaynes | but thats neat | 18:57 |
| clarkb | ya it appaers to target specific use cases that salt has | 18:58 |
| *** dizquierdo has quit IRC | 18:58 | |
| Shrews | SpamapS: it's in that paste | 18:59 |
| fungi | SpamapS: saw the debian itp for that today and was wondering. baked-in crypto for auth/privacy at least | 18:59 |
| clarkb | fungi: which is a bit annoying because they already made a big mistake with that once | 19:00 |
| fungi | true | 19:00 |
| fungi | that was un-fun to update | 19:00 |
| *** adalbas has joined #openstack-infra | 19:00 | |
| anteaya | greghaynes: nicely done | 19:01 |
| SpamapS | Shrews: oh duh | 19:01 |
| morganfainberg | Solution: ansible. | 19:01 |
| morganfainberg | Oh sorry /me stops being snarky. | 19:01 |
| SpamapS | fungi: the first rule of crypto is do not do your own crypto | 19:01 |
| SpamapS | morganfainberg: IMO it's not snark, it's reality. | 19:01 |
| fungi | this is true. however adding crypto to a message queue has not been done efficiently in the past | 19:02 |
| SpamapS | But I'm sure there are good reasons to use Salt that I'm not aware of, and thus, my reality is (as usual) flawed. | 19:02 |
| morganfainberg | SpamapS: and I think the second rule of crypto is "even if you don't use your own crypto, it'll be wrong" | 19:02 |
| fungi | afaik | 19:02 |
| clarkb | SpamapS: salt does in theory scale better | 19:02 |
| fungi | fireball! | 19:02 |
| *** baoli_ has quit IRC | 19:02 | |
| clarkb | SpamapS: but its also more complicated | 19:02 |
| greghaynes | nah, salt is super simple | 19:02 |
| SpamapS | morganfainberg: 2nd rule of crypto is pay somebody to audit your crypto | 19:02 |
| clarkb | greghaynes: not compared to ansible | 19:02 |
| greghaynes | its all pluggable | 19:02 |
| greghaynes | thats kind of the beauty | 19:02 |
| greghaynes | even the language you define your states in is just a plugable layer | 19:03 |
| morganfainberg | 3rd rule: it'll still be wrong. | 19:03 |
| *** Rockyg has joined #openstack-infra | 19:03 | |
| clarkb | greghaynes: you need syndics and consistent zmq and the way you do orchestration across salt nodes is funky | 19:03 |
| greghaynes | yea, dont use it for that | 19:03 |
| greghaynes | :p | 19:03 |
| clarkb | greghaynes: thats what you use ansible for | 19:03 |
| greghaynes | yerp | 19:03 |
| mordred | greghaynes: when what do you use it for? | 19:03 |
| morganfainberg | SpamapS: poodle, freak, heart bleed, etc etc export-quality crypto. | 19:03 |
| greghaynes | mordred: its pretty good at apply state to single node | 19:04 |
| clarkb | morganfainberg: export quality crypto was intentional fwiw | 19:04 |
| greghaynes | especially if you want to roll your own magic around that | 19:04 |
| clarkb | morganfainberg: because the code you write to encrypt stuff is a weapon | 19:04 |
| fungi | morganfainberg: pretty much everything crossing the internet... poor-quality crypto | 19:04 |
| morganfainberg | clarkb: not the issue with it that happened the last few months :P | 19:04 |
| krotscheck | Woo, my npm job finally passes! | 19:04 |
| SpamapS | morganfainberg: oh, and rule 0 of crypto is don't listen to a blow-hard like SpamapS to tell you how to do crypto ;) | 19:04 |
| morganfainberg | clarkb: and I'm sure that was audited. | 19:05 |
| * krotscheck gripes at jjb a bit. | 19:05 | |
| morganfainberg | krotscheck: you porting keystone to nodejs for me next? | 19:05 |
| morganfainberg | krotscheck: I hear it's not only webscale but fast >.> | 19:05 |
| *** dboik has quit IRC | 19:06 | |
| morganfainberg | so on paper salt says all the right things. | 19:06 |
| morganfainberg | Always has. | 19:06 |
| krotscheck | morganfainberg: As soon as you fix pip :) | 19:06 |
| morganfainberg | I've been burned enough by it I have an issue with using it when ansible does what I need. | 19:06 |
| morganfainberg | krotscheck: rm -rf | 19:07 |
| morganfainberg | krotscheck: done | 19:07 |
| krotscheck | morganfainberg: What a coincidence, that was my suggestion for how to fix keystone ;) | 19:07 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Typo in Project Groups. https://review.openstack.org/169037 | 19:07 |
| clarkb | morganfainberg: it shouldn't be surprising that clients and servers will accept export ciphers when they were intentionally engineered to do so 20 years ago | 19:08 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Project Group detail page improvements. https://review.openstack.org/169038 | 19:08 |
| clarkb | morganfainberg: it should be treated as a bug today, BUT this was all intentional at one time so not surprising that it works | 19:08 |
| morganfainberg | clarkb: not surprised but was using as my case of "you're doing crypto wrong" no matter what you do. Doesn't mean you should try to do it right. Just know it'll be wrong in subtle and different ways each time. | 19:09 |
| morganfainberg | Doesn't mean you shouldn't try and do it correctly* | 19:10 |
| * morganfainberg kicks autocorrect. | 19:10 | |
| clarkb | I guess it depends on your definition of wrong, according to 90s programmers this was correct :) | 19:10 |
| morganfainberg | When it becomes "treat it like a bug today" it is then wrong. | 19:10 |
| clarkb | which unfortuantely was the case due to non technical reasons :( | 19:11 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-jeepyb: Parameterize git revision of jeepyb https://review.openstack.org/170960 | 19:12 |
| clarkb | jhesketh: when you are around please see comments on https://review.openstack.org/#/c/107267/ I think we can go ahead and move forward with that unless sdague has any input | 19:12 |
| *** mrmartin has joined #openstack-infra | 19:12 | |
| fungi | not so fond memories of the debian non-us archives. also why openbsd still develops any of its own crypto exclusively outside usa borders by non-usa citizens | 19:13 |
| *** chandankumar has quit IRC | 19:13 | |
| *** wicope has quit IRC | 19:13 | |
| *** sarob has joined #openstack-infra | 19:16 | |
| *** mrmartin has quit IRC | 19:17 | |
| *** boris-42 has quit IRC | 19:18 | |
| zaro | clarkb: ohh wait. you can test jeblair change on review-dev | 19:19 |
| krotscheck | clarkb: So, remember that xvfb conversation we had the other day? is this more of what you were thinking of? https://review.openstack.org/#/c/169829/6/jenkins/jobs/macros.yaml | 19:20 |
| zaro | the review-dev db has already been migrated to 2.9 | 19:20 |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr: Add TODO file to os-testr https://review.openstack.org/170864 | 19:21 |
| clarkb | krotscheck: yup, though you may want to record the pid of the xvfb process to make it easy to kill later? probably easy enough to look it up by the display number | 19:21 |
| *** dhritishikhar_ has joined #openstack-infra | 19:21 | |
| krotscheck | clarkb: jeblair's suggestion was to just let it go, since they're single use instances. | 19:22 |
| clarkb | ya it should be fine | 19:22 |
| krotscheck | clarkb: Also, I seem to remember that you can kill an xvfb by display id. | 19:22 |
| *** HeOS has joined #openstack-infra | 19:23 | |
| *** eharney has quit IRC | 19:24 | |
| *** koolhead17 has quit IRC | 19:26 | |
| *** _nadya_ has joined #openstack-infra | 19:27 | |
| openstackgerrit | Merged openstack/os-testr: Add TODO file to os-testr https://review.openstack.org/170864 | 19:28 |
| *** teran has joined #openstack-infra | 19:29 | |
| *** _nadya_ has quit IRC | 19:31 | |
| anteaya | mtreinish: os-testr is a qa repo, yeah? | 19:31 |
| mtreinish | anteaya: yeah | 19:31 |
| anteaya | so do we want it reporting in here? | 19:32 |
| mtreinish | anteaya: https://review.openstack.org/169933 | 19:32 |
| mtreinish | it was a point of debate :) | 19:32 |
| anteaya | its nice to know what you are doing and all | 19:32 |
| *** zul has quit IRC | 19:32 | |
| zaro | does anyone know whether you need to do anything special to get zuul tests working? it always fails for me, Content-Type: text/plain;charset="utf8" \n traceback \n E \n returncode 1420 | 19:32 |
| mtreinish | anteaya: see but I got your attention now, so you can start helping on those todo items :) | 19:33 |
| anteaya | mtreinish: right after my syrup hits 225 | 19:33 |
| zaro | oops, failure is actually "failure: process-returncode [ multipart Content-Type: text/plain;charset="utf8" \n traceback \n E returncode 1420 \n]" | 19:34 |
| *** dhritishikhar_ has quit IRC | 19:34 | |
| SpamapS | Shrews: looking at your shade example.. I wonder if you've tried pdb'ing in and seeing what the responses are from the server? | 19:34 |
| anteaya | mtreinish: 221 won't be long now | 19:35 |
| Shrews | SpamapS: i did not. i just put in print statements (that output i showed you) to look at the volume object being tested | 19:36 |
| *** ajmiller has quit IRC | 19:39 | |
| jeblair | zaro: are you running zuul tests locally on osx? | 19:40 |
| jeblair | zaro: if so, they don't work on osx | 19:41 |
| *** otter768 has joined #openstack-infra | 19:41 | |
| jeblair | zaro: can you test out 170948 locally really quickly? | 19:41 |
| zaro | no, i'm runing on trusty | 19:42 |
| zaro | jeblair: will give that change a try. | 19:42 |
| jeblair | zaro: oh, that's weird then. maybe paste the full output? | 19:43 |
| clarkb | fungi: is the bindep stack ready for rechecks? looks like the cookiecutter sync merged | 19:43 |
| mtreinish | anteaya: what does the number signify? volume of syrup? | 19:44 |
| fungi | clarkb: yeah, i'm currently hacking on the centos-6 node label addition, but those should be ready to go. i don't think any of them actually merge-conflict | 19:45 |
| fungi | i'll ping them all real quick | 19:45 |
| zaro | jeblair: http://paste.openstack.org/show/198744/ | 19:45 |
| SpamapS | Shrews: oh I think I might see the bug | 19:45 |
| jeblair | mtreinish: i assumed temperature | 19:45 |
| *** otter768 has quit IRC | 19:46 | |
| *** dboik has joined #openstack-infra | 19:47 | |
| fungi | clarkb: hrm, i guess i need to rebase all these because their parent got a new patchset before it merged. that's annoyingly silly | 19:47 |
| *** dboik has quit IRC | 19:47 | |
| *** dboik has joined #openstack-infra | 19:47 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Workflow documentation is now in infra-manual https://review.openstack.org/139436 | 19:48 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Add CentOS RPM support https://review.openstack.org/157978 | 19:48 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Add codename and release platform profiles https://review.openstack.org/157979 | 19:48 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Update .gitreview for project rename https://review.openstack.org/168525 | 19:48 |
| Shrews | SpamapS: awesome | 19:48 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Support for older dpkg https://review.openstack.org/170189 | 19:48 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Accept old dpkg-query errors https://review.openstack.org/170244 | 19:48 |
| *** alexpilotti has quit IRC | 19:48 | |
| anteaya | mtreinish: temp | 19:49 |
| anteaya | mtreinish: though I have to default to 'when it starts to boil over' | 19:49 |
| anteaya | which it did this time | 19:49 |
| jeblair | nibalizer: afayk, are we ready to try releasing puppet httpd again? | 19:50 |
| anteaya | I'm learning though, I have moved to a taller pot | 19:50 |
| mtreinish | jeblair: you win | 19:50 |
| nibalizer | jeblair: afaik | 19:50 |
| *** kgiusti has left #openstack-infra | 19:50 | |
| mtreinish | anteaya: heh, yeah I can imagine that gets messy | 19:50 |
| anteaya | yup, still mopping the floor this morning from when we filtered it | 19:51 |
| anteaya | we'll do better next year | 19:51 |
| *** baoli has joined #openstack-infra | 19:51 | |
| *** baoli has quit IRC | 19:52 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Add a --brief option https://review.openstack.org/164854 | 19:52 |
| *** baoli has joined #openstack-infra | 19:52 | |
| *** aysyd has quit IRC | 19:53 | |
| fungi | clarkb: okay, i think that's all my rebases in bindep | 19:54 |
| fungi | thanks for the reminder | 19:54 |
| *** sdake_ has joined #openstack-infra | 19:54 | |
| *** aysyd has joined #openstack-infra | 19:55 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Document openstackid deployment https://review.openstack.org/170974 | 19:55 |
| *** Guest48074 is now known as redrobot | 19:55 | |
| jeblair | nibalizer: what version? 0.0.1? | 19:56 |
| jeblair | 0.1.0? | 19:57 |
| nibalizer | doesn't matter to me | 19:57 |
| jeblair | 0.1.0 it is | 19:57 |
| *** sdake has quit IRC | 19:58 | |
| jeblair | 2015-04-06 19:58:50,746 DEBUG zuul.Gearman: Function build:puppet-httpd-forge-upload:bare-trusty is not registered | 20:00 |
| jeblair | i deleted the tag so we can try again later | 20:01 |
| *** annegentle has quit IRC | 20:01 | |
| *** alexpilotti has joined #openstack-infra | 20:04 | |
| krotscheck | Any core other than clarkb able to look at a new take on javascript builds for me? https://review.openstack.org/#/c/169829/ | 20:08 |
| anteaya | krotscheck: we prefer if non-voting jobs are just in check, not in the gate pipeline | 20:10 |
| *** SumitNaiksatam has joined #openstack-infra | 20:11 | |
| krotscheck | anteaya: Any comments other than that? | 20:11 |
| anteaya | krotscheck: since you have to respin might as well incorporate clarkb's comment | 20:12 |
| *** dboik has quit IRC | 20:12 | |
| krotscheck | anteaya: Anything other than that? | 20:12 |
| krotscheck | anteaya: I want to get all the comments in so I can minimize churn. | 20:13 |
| anteaya | krotscheck: I don't have an opinion on the javascript.yaml other than that but will a non-voting job you can assess whether your job template is any good without shooting yourself in the foot | 20:13 |
| anteaya | right | 20:13 |
| anteaya | I was typing | 20:13 |
| anteaya | and making mistakes | 20:13 |
| anteaya | I have high blood sugar atm | 20:13 |
| anteaya | see syrup ^^ | 20:13 |
| anteaya | s/will/with | 20:14 |
| *** alexpilotti has quit IRC | 20:16 | |
| *** ociuhandu has quit IRC | 20:16 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add NPM Command Execution as a Build Target https://review.openstack.org/169829 | 20:16 |
| *** emagana has quit IRC | 20:19 | |
| *** dprince has quit IRC | 20:19 | |
| *** baoli has quit IRC | 20:21 | |
| anteaya | krotscheck: thank you | 20:21 |
| *** tonytan4ever has quit IRC | 20:22 | |
| *** ajmiller has joined #openstack-infra | 20:24 | |
| *** emagana has joined #openstack-infra | 20:24 | |
| *** tonytan4ever has joined #openstack-infra | 20:24 | |
| *** dboik has joined #openstack-infra | 20:26 | |
| krotscheck | anteaya: No worries :) | 20:26 |
| *** dboik has quit IRC | 20:27 | |
| *** dboik has joined #openstack-infra | 20:27 | |
| krotscheck | Is something up with flake8 at the moment? One of my jobs is suddenly (unexpectedly) failing on pep8, the error is not reproducible locally, and the console dump is not helpful. | 20:27 |
| krotscheck | http://logs.openstack.org/60/170260/11/check/gate-oslo.middleware-pep8/ad34907/console.html | 20:27 |
| *** dhritishikhar_ has joined #openstack-infra | 20:27 | |
| *** krtaylor has quit IRC | 20:28 | |
| *** andreykurilin_ has joined #openstack-infra | 20:28 | |
| *** dhritishikhar_ has quit IRC | 20:28 | |
| *** dhritishikhar_ has joined #openstack-infra | 20:29 | |
| *** dkehn has quit IRC | 20:29 | |
| *** dkehn has joined #openstack-infra | 20:30 | |
| SpamapS | Shrews: this is actually kind of nasty | 20:32 |
| *** krtaylor has joined #openstack-infra | 20:33 | |
| SpamapS | Shrews: writing a pile of mock to write a test. :-/ | 20:33 |
| jeblair | SpamapS: that's kind of a red flag, right? :) | 20:36 |
| SpamapS | jeblair: always | 20:36 |
| SpamapS | state is hard | 20:36 |
| SpamapS | caching is hard | 20:36 |
| stevemar | anteaya dropped the hammer | 20:36 |
| stevemar | thanks for that, i wasn't looking forward to all that email | 20:37 |
| anteaya | stevemar: I mean really people, look around | 20:38 |
| anteaya | every single one thinks it is just them, have you taken a look at the seating in this stadium | 20:38 |
| SpamapS | jeblair: I'm extremely suspicious about "wins" with mock. But I'm not devoting enough time to this to evaluate cinderclient's fakes. ;) | 20:38 |
| *** jckasper has joined #openstack-infra | 20:38 | |
| anteaya | stevemar: I have a lot of room in my heart to forgive jungleboyj, because he is awesome | 20:39 |
| anteaya | but yeah, the crowd | 20:39 |
| jckasper | simple question: Is "Toggle CI" option no longer available on code reviews ? | 20:39 |
| jckasper | no longer seeing the button on code reviews anymore | 20:40 |
| *** yamahata has joined #openstack-infra | 20:40 | |
| anteaya | jckasper: it should be there | 20:40 |
| anteaya | jckasper: when did it disappear for you? | 20:40 |
| anteaya | do you recall? | 20:40 |
| jckasper | quite a few of us have noticed it is gone ... on a page refresh | 20:40 |
| jckasper | not sure. fairly recently I think | 20:40 |
| anteaya | mine is gone too | 20:41 |
| anteaya | and yes on a refresh | 20:41 |
| jckasper | can we have Toggle CI back ? please. | 20:41 |
| anteaya | what merged in the last few hours that might have taken out 'toggle ci'? | 20:41 |
| stevemar | anteaya, understandable, jungleboyj is pretty awesome | 20:41 |
| anteaya | jckasper: it was a mistake to be usre | 20:41 |
| anteaya | stevemar: :) | 20:41 |
| anteaya | jckasper: thanks for bringing it to my attention, not sure if others were already aware | 20:41 |
| krotscheck | heeeeeeeeeey clarkb! wanna review the diff since your last review? :D https://review.openstack.org/#/c/169829/ | 20:42 |
| anteaya | within the last hour actually | 20:42 |
| jckasper | "NetworkError: 404 Not Found - https://review.openstack.org/static/hideci.js?e=cd91875ddc755c2ff03db41aea179f3a" in the console | 20:42 |
| *** spzala has quit IRC | 20:42 | |
| *** dhritishikhar_ has quit IRC | 20:43 | |
| *** sdake_ has quit IRC | 20:44 | |
| anteaya | jeblair: looks like this patch somehow took out the toggle ci button: https://review.openstack.org/#/c/170936/ | 20:44 |
| anteaya | I do believe | 20:44 |
| jeblair | anteaya: we may just need to restart gerrit | 20:45 |
| *** jogo has quit IRC | 20:45 | |
| *** zul has joined #openstack-infra | 20:45 | |
| *** jogo has joined #openstack-infra | 20:45 | |
| jeblair | yeah, i think so | 20:45 |
| anteaya | ah | 20:46 |
| anteaya | that would be easiest | 20:46 |
| anteaya | jckasper: stand by | 20:46 |
| jckasper | anteaya: standing | 20:47 |
| *** marcusvrn2 has quit IRC | 20:49 | |
| *** hdd has quit IRC | 20:52 | |
| *** Rockyg has quit IRC | 20:56 | |
| *** salv-orlando has joined #openstack-infra | 20:56 | |
| zaro | jeblair: generally 170948 works just fine. not sure how i would force that message though. | 20:57 |
| *** jogo has quit IRC | 20:57 | |
| *** aysyd has quit IRC | 20:57 | |
| *** jogo has joined #openstack-infra | 20:59 | |
| *** jogo has quit IRC | 20:59 | |
| *** jogo has joined #openstack-infra | 20:59 | |
| *** aysyd has joined #openstack-infra | 20:59 | |
| *** hdd has joined #openstack-infra | 20:59 | |
| fungi | anteaya: jeblair: any better? i just touched ~gerrit2/review_site/etc/GerritSiteHeader.html to get gerrit to reload that | 21:00 |
| fungi | seems to be working for me | 21:01 |
| *** btully has joined #openstack-infra | 21:01 | |
| jckasper | Toggle CI is back for me | 21:01 |
| *** ZZelle_ has quit IRC | 21:02 | |
| *** hichihara has joined #openstack-infra | 21:03 | |
| *** bswartz has quit IRC | 21:05 | |
| jesusaurus | clarkb: nibalizer: you may have not noticed that your comments on 161165 have been responded to | 21:07 |
| *** ZZelle_ has joined #openstack-infra | 21:09 | |
| jeblair | fungi: i thought puppet did that? | 21:09 |
| nibalizer | jesusaurus: i've been in a hole sorry, checking now | 21:10 |
| jeblair | zaro: run 'gerrit stream-events' and then in another terminal, run 'gerrit show-queue' | 21:10 |
| *** marun has quit IRC | 21:10 | |
| *** emagana has quit IRC | 21:10 | |
| * clarkb catches up after lunch | 21:12 | |
| *** marun has joined #openstack-infra | 21:12 | |
| *** emagana has joined #openstack-infra | 21:13 | |
| *** ajmiller_ has joined #openstack-infra | 21:14 | |
| *** e0ne has quit IRC | 21:14 | |
| *** Bsony has joined #openstack-infra | 21:14 | |
| *** julim has quit IRC | 21:14 | |
| *** weshay has quit IRC | 21:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Run module-forge-upload jobs on pypi https://review.openstack.org/170982 | 21:14 |
| fungi | jeblair: i'm starting to think that we need to insert a delay for that, because when i manually change its javascript and then touch that file immediately after it doesn't help, but if i wait a few seconds (10? 30?) it picks it up fine after that | 21:14 |
| jeblair | nibalizer: https://review.openstack.org/170982 | 21:14 |
| openstackgerrit | Merged openstack-infra/system-config: Updates to running-your-own CI docs: Changes required https://review.openstack.org/162268 | 21:14 |
| jeblair | fungi: oh weird | 21:14 |
| asselin_ | pip freeze | grep tempest | 21:16 |
| fungi | i haven't followed up with any controlled experiments yet to confirm, but that would definitely explain the behavior we're seeing | 21:17 |
| *** ldnunes has quit IRC | 21:17 | |
| *** ajmiller has quit IRC | 21:17 | |
| clarkb | krotscheck: approved | 21:18 |
| krotscheck | clarkb: grazie, multo grazie | 21:19 |
| *** dustins_ has quit IRC | 21:19 | |
| *** ajmiller__ has joined #openstack-infra | 21:19 | |
| mtreinish | asselin_: are you in your tox venv? | 21:20 |
| asselin_ | mtreinish, ?? | 21:20 |
| clarkb | jesusaurus: the problem is that it doesn't fix the problem for port 443 | 21:20 |
| clarkb | jesusaurus: it may work just fine if you only have one thing listenign on 443 but it isn't generally fixing the problem | 21:20 |
| clarkb | jesusaurus: my -1 is still valid as written | 21:20 |
| zaro | jeblair: hmm, i'm not seeeing it. i thought you see this on a stack trace? | 21:20 |
| mtreinish | asselin_: 'pip freeze | grep tempest' might not show anything if your just running tempest in tox :) | 21:21 |
| mtreinish | or it might be different than what you're running with | 21:21 |
| jeblair | zaro: no it's supposed to show up in the 'gerrit show-queue' list | 21:21 |
| asselin_ | mtreinish, :) I see...didn't even realize it went here :) | 21:21 |
| zaro | jeblair: hmm let me double check that the code is in there. | 21:22 |
| clarkb | jesusaurus: I left a comment | 21:22 |
| jeblair | zaro: oh, it would probably only show up on a stuck connection or if you happen to catch it at just the right time, sorry | 21:22 |
| jeblair | zaro: i'm not sure of a good way to test this... | 21:22 |
| *** ajmiller_ has quit IRC | 21:22 | |
| *** radez is now known as radez_g0n3 | 21:23 | |
| jeblair | zaro: maybe adding a sleep to writeEvents or something... | 21:24 |
| jesusaurus | clarkb: yes, but I agree with Bob that a missing NameVirtualHost *:443 is orthogonal to the change being proposed | 21:25 |
| clarkb | jesusaurus: it isn't because the change says "this allows jenkins vhost to coexist with other vhosts" which it does not | 21:25 |
| clarkb | jesusaurus: it only does for port 80 | 21:26 |
| *** dkranz has quit IRC | 21:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Add NPM Command Execution as a Build Target https://review.openstack.org/169829 | 21:27 |
| *** tnovacik has quit IRC | 21:28 | |
| *** Bsony has quit IRC | 21:28 | |
| anteaya | fungi: sorry keep getting pulled away, family in the house | 21:29 |
| * krotscheck steeples his fingers. | 21:29 | |
| krotscheck | mwa. ha. ha. | 21:29 |
| *** tiswanso_ has quit IRC | 21:29 | |
| * anteaya sees the toggle ci button | 21:29 | |
| *** sdake has joined #openstack-infra | 21:29 | |
| krotscheck | I'm sorry to ask this, because I know this has happeend before, but what does NOT_REGISTERED mean in zuul? | 21:31 |
| *** dboik has quit IRC | 21:33 | |
| *** sdake_ has joined #openstack-infra | 21:33 | |
| anteaya | it means that the gearman layer isn't working as expected | 21:33 |
| anteaya | zuul talks to gearman talks to jenkins | 21:33 |
| anteaya | someone isn't doing what the others think they are | 21:33 |
| anteaya | krotscheck: where do you see it? | 21:33 |
| *** dboik has joined #openstack-infra | 21:34 | |
| *** sdake has quit IRC | 21:34 | |
| jeblair | krotscheck: it means that no workers have registered the job. that might be because jjb has not run to create the jobs in jenkins yet. | 21:35 |
| *** annegentle has joined #openstack-infra | 21:35 | |
| clarkb | fungi: reviewing bindep change snow | 21:35 |
| jeblair | krotscheck: you can either check the jenkins masters or puppetboard to see | 21:36 |
| openstackgerrit | Merged openstack-infra/bindep: Workflow documentation is now in infra-manual https://review.openstack.org/139436 | 21:39 |
| fungi | clarkb: thanks! i sort of got sidetracked. got a new grill and had to test it out by making bison hot dogs | 21:39 |
| *** bhunter71 has quit IRC | 21:39 | |
| *** jckasper has quit IRC | 21:40 | |
| krotscheck | anteaya: A new job I just landed. I'm guessing I was too quick on the trigger for the recheck | 21:40 |
| clarkb | fungi: in 157978 I do no understand where EPOCH and VERSION and NAME are coming from | 21:40 |
| clarkb | fungi: what am I missing there? | 21:40 |
| *** sputnik13 has quit IRC | 21:41 | |
| fungi | clarkb: those are format string replacements which get performed by rpm --qf | 21:41 |
| *** otter768 has joined #openstack-infra | 21:42 | |
| fungi | clarkb: see the rpm manpage for a list of the various format string variables --qf takes | 21:42 |
| *** sputnik13 has joined #openstack-infra | 21:42 | |
| jogo | I take it you guys are aware of the whole slew of http://logs.openstack.org/08/170808/2/check/check-grenade-dsvm-neutron/0a71fd4/logs/grenade.sh.txt.gz#_2015-04-06_20_43_04_556 issues | 21:43 |
| clarkb | oh huh I thought it was python interpolation. Thanks | 21:43 |
| fungi | i can mention that in a code comment if you think it helps, though it's basically parity to what dpkg-query is doing in the Dpkg() class | 21:43 |
| *** pc_m has quit IRC | 21:43 | |
| *** baoli has joined #openstack-infra | 21:43 | |
| jogo | I count 24 hits for that in past 24 hours | 21:43 |
| *** melwitt has joined #openstack-infra | 21:43 | |
| jeblair | jogo: what providers are the failing nodes in? | 21:44 |
| *** tonytan4ever has quit IRC | 21:44 | |
| clarkb | fungi: ya its not necessary, just my unfamiliarity with `rpm` | 21:44 |
| *** aysyd has quit IRC | 21:44 | |
| jeblair | jogo: all rax-dfw? | 21:45 |
| fungi | jogo: first i've seen it | 21:45 |
| jogo | jeblair: query message:" HTTPConnectionPool(host=" AND message:", port=80): Read timed out." | 21:45 |
| jogo | jeblair: yeah all dfw | 21:45 |
| *** annegentle has quit IRC | 21:46 | |
| fungi | jogo: i can retrieve it, so betting there are network problems in dfw | 21:46 |
| jogo | there was an older set from pypi.region-b.geo-1.openstack.org as well | 21:46 |
| *** hdd has quit IRC | 21:46 | |
| fungi | though status.rackspace.com doesn't mention any issues | 21:46 |
| *** otter768 has quit IRC | 21:46 | |
| jogo | fungi: https://tinyurl.com/nxapcog | 21:46 |
| *** unicell has left #openstack-infra | 21:46 | |
| jogo | past 4 hours, only a few hits | 21:47 |
| *** btully has quit IRC | 21:47 | |
| clarkb | fungi: did you want to add fedora to that change or in a followup chnage? | 21:47 |
| *** andreykurilin__ has joined #openstack-infra | 21:47 | |
| jogo | must have been some tiny hickup or two there | 21:47 |
| fungi | cpu usage has picked up there starting around 1700 utc: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2489&rra_id=all | 21:47 |
| jeblair | jogo: the dfw ones seem to be clustered by time. so probably individual network events there. | 21:47 |
| *** ivar-lazzaro has quit IRC | 21:47 | |
| *** andreykurilin_ has quit IRC | 21:47 | |
| *** Alexandra_ has joined #openstack-infra | 21:47 | |
| anteaya | krotscheck: ah | 21:47 |
| jogo | jeblair: yeah, I wonder if its a neutron issue underneath :) | 21:48 |
| fungi | clarkb: yeah, just hadn't done so yet. mostly focused on adding the platforms we need first, and there are no bare-f21 nodes we need to worry about migrating (yet anyway?) | 21:48 |
| *** ivar-lazzaro has joined #openstack-infra | 21:48 | |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror https://review.openstack.org/165240 | 21:48 |
| greghaynes | clarkb: ^ | 21:48 |
| clarkb | fungi: correct, just devstack nodes | 21:48 |
| fungi | jeblair: the dfw cpu spikes also seem to be clustered by time. looking now to see if i can tell what that is | 21:48 |
| jeblair | fungi: 5 cents it's bandersnatch | 21:48 |
| fungi | wouldn't surprise me in the least. that' | 21:48 |
| fungi | s a sucker bet | 21:49 |
| jeblair | fungi, clarkb: i think i approved the 'upgrade bandersnatch' change earlier | 21:49 |
| clarkb | greghaynes: +2 | 21:49 |
| greghaynes | \O/ | 21:49 |
| anteaya | jogo: hickup is a redneck on a ladder, hiccup? | 21:49 |
| *** dboik has quit IRC | 21:49 | |
| clarkb | jeblair: bandersnatch affecting neutron grenade? | 21:49 |
| jogo | hahaha hiccup* | 21:49 |
| anteaya | jogo: :) | 21:49 |
| jeblair | clarkb: bandersnatch eating cpu | 21:49 |
| clarkb | sorry too many trains of thought | 21:49 |
| jeblair | 853 root 20 0 375m 126m 4724 S 100 3.4 3:54.01 bandersnatch | 21:49 |
| fungi | jeblair: bingo. bandersnatch process eating an entire coy for 4 hours | 21:49 |
| clarkb | silly bandersnatch | 21:49 |
| fungi | s/coy/cpu/ | 21:49 |
| jeblair | yum. tasty coy. | 21:49 |
| *** dboik has joined #openstack-infra | 21:49 | |
| fungi | it's playing coy with us | 21:50 |
| *** mriedem is now known as mriedem_away | 21:50 | |
| fungi | strace shows it in a select loop | 21:50 |
| fungi | the 16:55 run of the wrapper script threw an exception and that's the last thing in the log | 21:51 |
| jeblair | yeah, and unfortunately, process output isn't logged :/ | 21:51 |
| jeblair | fungi: i'm going to kill it and start it in screen | 21:52 |
| clarkb | jeblair: sounds good | 21:52 |
| *** dims has quit IRC | 21:52 | |
| fungi | thanks. another nickel says it won't do it again | 21:52 |
| *** dboik has quit IRC | 21:53 | |
| *** dboik has joined #openstack-infra | 21:54 | |
| jeblair | http://paste.openstack.org/show/198838/ | 21:54 |
| *** dims has joined #openstack-infra | 21:54 | |
| *** dims has quit IRC | 21:54 | |
| *** mrda-away is now known as mrda | 21:55 | |
| jeblair | new bandersnatch not backwards compatible with files old bandersnatch left laying around? | 21:55 |
| *** dims has joined #openstack-infra | 21:55 | |
| *** baoli has quit IRC | 21:55 | |
| jeblair | also, just cause: hundreds of InsecurePlatformWarnings all over the output | 21:56 |
| jeblair | fungi: so maybe it's just repeatedly hitting the timeout we set for it? | 21:57 |
| jeblair | since it thinks it needs to do a full sync | 21:57 |
| clarkb | jeblair: that does look like it intends on doing a full sync | 21:57 |
| *** doug-fish has quit IRC | 21:58 | |
| jeblair | yeah. it's at CairoSVG right now ;) | 21:58 |
| *** ddieterly has quit IRC | 21:59 | |
| openstackgerrit | Merged openstack-infra/bindep: Update .gitreview for project rename https://review.openstack.org/168525 | 22:00 |
| fungi | i wonder, did new bandersnatch just release today? | 22:01 |
| fungi | nope | 22:01 |
| fungi | maybe this is the pypi data change dstufft was warning about that we needed newer bandersnatch for | 22:02 |
| clarkb | fungi: that would be my guess | 22:02 |
| fungi | oh, we're running 1.7 there | 22:02 |
| clarkb | fungi: jeblair and its doing a full sync to catch things it may have missed previously that the new protocol allows it to catch | 22:02 |
| fungi | so sayeth the logs | 22:02 |
| *** teran_ has joined #openstack-infra | 22:03 | |
| fungi | 2015-04-06 16:55:01,917 INFO: bandersnatch/1.7 (CPython 2.7.3-final0, Linux 3.2.0-77-generic x86_64) | 22:03 |
| fungi | but pip list says we have 1.8 installed | 22:03 |
| fungi | i bet the upgrade got confused | 22:04 |
| *** nelsnelson has quit IRC | 22:04 | |
| fungi | same situation in iad | 22:05 |
| clarkb | fungi: what version of pip is there? | 22:05 |
| *** teran has quit IRC | 22:05 | |
| fungi | er, actually no it's not hung in iad, though it did stop running | 22:05 |
| fungi | pip 1.5.6 | 22:06 |
| clarkb | ok so that pip should be new enough to properly upgrade things | 22:06 |
| clarkb | I think | 22:06 |
| fungi | yeah, all four of our mirrors stopped running successfully as of 16:55 utc | 22:07 |
| fungi | so that probably corresponds to a change on pypi | 22:07 |
| fungi | also all of them seem to still be running bandersnatch 1.7 | 22:07 |
| clarkb | or when my change to update bandersnatch merged? | 22:08 |
| *** teran has joined #openstack-infra | 22:08 | |
| *** achanda has quit IRC | 22:08 | |
| clarkb | it was 9:32 PDT, now to do maths | 22:08 |
| fungi | too 16:32 pdt | 22:09 |
| clarkb | 1655 would be 9:55 PDT | 22:09 |
| fungi | yup | 22:09 |
| fungi | close enough to be damning | 22:09 |
| *** teran_ has quit IRC | 22:09 | |
| clarkb | I wouldn't be surprised if the upgrade failed in such a way that bandersnatch had a lol | 22:09 |
| clarkb | considering the versions that are reported | 22:09 |
| clarkb | maybe we need to do that upgrade while bandersnatch is not running? | 22:09 |
| fungi | ooh, maybe pip failed to delete files which were open for read? digging in puppet logs to see if there are any clues | 22:10 |
| clarkb | in other news I checked the trusty python3 package yesterday and still don't have a new version there | 22:11 |
| fungi | no errors from puppet when it ensured bandersnatch changed '1.7' to '1.8' | 22:12 |
| fungi | clarkb: we can probably proceed with the "run py34 jobs on debian" idea even without dib now that it's a few weeks away from the officially announced release date | 22:13 |
| dstufft | fungi: nothings changed in pypi yet | 22:13 |
| fungi | clarkb: the reason bare-.* dib was a blocker there was lack of jessie images | 22:13 |
| dstufft | probably won't change until warehouse goes live | 22:13 |
| *** achanda has joined #openstack-infra | 22:14 | |
| *** andreykurilin__ has quit IRC | 22:14 | |
| fungi | dstufft: thanks, and yeah we've now figured out that it seems to be due to a bandersnatch upgrade gone wrong | 22:14 |
| *** ddieterly has joined #openstack-infra | 22:15 | |
| dstufft | fungi: cool | 22:15 |
| fungi | dstufft: when we run bandersnatch it reports as 1.7 but pip says it's got 1.8 | 22:15 |
| dstufft | did bandersnatch not get restarted afer an upgrade? | 22:16 |
| dstufft | er | 22:16 |
| dstufft | nvm me | 22:16 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added NPM executions for StoryBoard Tests. https://review.openstack.org/170995 | 22:16 |
| dstufft | I'm tired :( | 22:16 |
| krotscheck | Step one passed, step d! | 22:16 |
| krotscheck | step 2 I mean | 22:16 |
| dstufft | fungi: the .pycs might be wrong maybe | 22:16 |
| dstufft | if the .py files are right | 22:16 |
| *** amitgandhinz has quit IRC | 22:18 | |
| anteaya | krotscheck: I'm interested in step d! | 22:18 |
| fungi | dstufft: ahh, nope, bad theory. the logs said 1.7 up to the point where it broke, which is when we tried to upgrade to 1.8 | 22:20 |
| clarkb | fungi: so is it just breaking due to the timeout then? I suppose jeblairs run will tell us | 22:20 |
| *** dboik has quit IRC | 22:21 | |
| *** ddieterly has quit IRC | 22:21 | |
| fungi | and hpcloud does indeed seem to be in a similar situation to dfw | 22:21 |
| *** ddieterly has joined #openstack-infra | 22:21 | |
| fungi | i suspect the timeout in cron for calling the wrapper script terminates the wrapper without terminating the bandersnatch child process | 22:22 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:22 | |
| krotscheck | anteaya: That's the one after PROFIT! | 22:23 |
| krotscheck | I'm still stuck on b: ??? | 22:23 |
| fungi | so i'm going with clarkb's change merged, bandersnatch upgraded, 1.8 decided it needed to do a full mirror refresh (because possibly not backward-compatible with the 1.7 mirror state files?) | 22:24 |
| fungi | and the timeout does look like it's doing the right thing, because on the mirrors we haven't touched yet the bandersnatch process has the same start time in the process list as the wrapper script | 22:25 |
| fungi | which is from just a few minutes agoi | 22:26 |
| *** ociuhandu has joined #openstack-infra | 22:29 | |
| clarkb | fungi: so do you think jeblair's screen run is something we need to do everywhere? | 22:30 |
| clarkb | I can grab some nodes and run bandersnatch if so | 22:30 |
| *** ddieterly has quit IRC | 22:30 | |
| fungi | clarkb: yeah, chances are that will allow it to recover. i'd disable puppet agent, comment out the cronjob, then run bandersnatch mirror as root in a screen session | 22:32 |
| fungi | doing that on region-b-geo-1 now | 22:32 |
| fungi | oh, also kill whatever bandersnatch process is running or you'll collide on its lockfile | 22:33 |
| clarkb | I think you can just run it with the lockfile and get the lock in your screen run | 22:33 |
| fungi | true, though that will cause it to log errors. i guess that's tolerable | 22:33 |
| fungi | it's in progress on pypi.region-b.geo-1.o.o now | 22:34 |
| *** Bsony has joined #openstack-infra | 22:34 | |
| fungi | same for pypi.iad.o.o | 22:35 |
| clarkb | looks like someone ya ok thats you | 22:35 |
| clarkb | fungi: were yougoing to get ord as well? | 22:36 |
| fungi | in progress | 22:36 |
| clarkb | thanks | 22:36 |
| fungi | running there now too | 22:36 |
| jeblair | fungi, clarkb: yeah, i did the puppet disable route on dfw, but i needed to see output. we probably don't need that on the others | 22:36 |
| fungi | i've detached from those screen sessions but i'll check back in on those three in a few hours | 22:36 |
| jeblair | dfw is at lowercase a now: /simple/apidev-sanza/ | 22:37 |
| clarkb | https://review.openstack.org/#/c/168408/ fixes a bug with ec2-api tests in tempest with aiopcpu jobs | 22:38 |
| clarkb | apparently i386 is not a valid machine type with the ec2-api | 22:38 |
| * mtreinish can't wait to remove those tests from tempest... | 22:39 | |
| openstackgerrit | Stephanie Miller proposed openstack-infra/puppet-zanata: Add zanata.ini file https://review.openstack.org/171000 | 22:39 |
| jeblair | clarkb: that seems so harsh and judgemental | 22:39 |
| *** __afazekas has quit IRC | 22:39 | |
| *** Bsony has quit IRC | 22:39 | |
| clarkb | jeblair: well, its just an interesting decision that you can make when you only ever intend to run one deployment of a thing | 22:40 |
| mordred | jeblair, clarkb: if you want i386 you want the wrong thing and should be judged | 22:40 |
| clarkb | mordred: that too (its why I wrote that d-g change) | 22:40 |
| greghaynes | where are our ppc macine types | 22:41 |
| mtreinish | greghaynes: or m68k support, that's what it's really missing | 22:43 |
| greghaynes | Yes! | 22:44 |
| mtreinish | totally wants an m68k cloud now | 22:44 |
| mtreinish | jeblair: ^^^ infra cloud? :) | 22:44 |
| *** annegentle has joined #openstack-infra | 22:44 | |
| *** annegentle_ has joined #openstack-infra | 22:44 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 22:45 | |
| fungi | when i moved last year i discarded a perfectly functional mac quadra running debian/m68k | 22:46 |
| jeblair | mtreinish: you're just making me want it to be i386 only. (with an option for i387 for an extra fee) | 22:47 |
| fungi | (for definitions of "perfectly functional which included 48mb of ram, a scsi-2 hard drive and 10base-t network connectivity anyway) | 22:47 |
| clarkb | mriedem_away: if you are not away, any chance you have run 170900 as a depends-on nova patch so we can see the logs getting collected? | 22:48 |
| *** bswartz has joined #openstack-infra | 22:48 | |
| fungi | jeblair: can't squander those fpus after all | 22:48 |
| mtreinish | jeblair: lol, yeah if they need floats make them pay :) | 22:49 |
| *** dimtruck is now known as zz_dimtruck | 22:49 | |
| *** ashleighfarnham has quit IRC | 22:50 | |
| *** markvoelker has quit IRC | 22:52 | |
| *** sarob has quit IRC | 22:54 | |
| *** claudiub has quit IRC | 22:55 | |
| jhesketh | Morning | 22:57 |
| mtreinish | jeblair: wait if hp is providing the hardware that means there is a chance of making a hppa cloud, right? | 22:57 |
| anteaya | jhesketh: morning | 22:57 |
| mtreinish | I'm sure there's a warehouse somewhere with a bunch of pa-risc boxes sitting there | 22:57 |
| anteaya | krotscheck: ah see I never get to step d! myself, I should work harder at that | 22:57 |
| clarkb | jhesketh: good morning. I commented on https://review.openstack.org/#/c/107267/ you are right we should rebase that change and maybe address my comments there | 22:58 |
| clarkb | jhesketh: curious to know what you think about my suggestions | 22:58 |
| jhesketh | anteaya: hey, hope you had a good Easter :-) | 22:58 |
| anteaya | jhesketh: thanks, I did, outside for most of it | 22:58 |
| anteaya | jhesketh: how about yourself? | 22:58 |
| *** devlaps has joined #openstack-infra | 22:59 | |
| *** annegentle_ has quit IRC | 22:59 | |
| *** annegentle has quit IRC | 22:59 | |
| jhesketh | clarkb: awesome, thanks. I saw that tick by over the weekend but was too busy eating chocolate to look closely. Your suggestions all looked good so I'll give it a go today, thanks :-) | 22:59 |
| anteaya | jhesketh: mmmm chocolate | 23:00 |
| *** annegentle has joined #openstack-infra | 23:00 | |
| *** annegent_ has joined #openstack-infra | 23:00 | |
| *** ivar-lazzaro has quit IRC | 23:00 | |
| jhesketh | anteaya: nice. Very good thanks. Lots of family time and way too much bad eating | 23:00 |
| *** annegentle is now known as Guest48668 | 23:00 | |
| clarkb | dougwig: commented on 164874 | 23:00 |
| fungi | mtreinish: when i moved i also tossed a working hp model 720 running debian/hppa... i miss my menagerie | 23:00 |
| clarkb | jhesketh: awesome, looking forward to it, I thinke once we hvae that the coverage reports will be hosted properly by swift via os-loganalyze | 23:01 |
| jhesketh | Neat | 23:01 |
| clarkb | jhesketh: also the footers look great, so we are very near t obeing able to switch to swift for all the things | 23:01 |
| mtreinish | fungi: heh, your collection was definitely nicer than mine was :) | 23:02 |
| anteaya | jhesketh: well done, isn't that what easter is all about | 23:02 |
| fungi | mtreinish: at one point i was only a couple of very obscure or expensive machines away from having representatives of every officially supported debian architecture | 23:03 |
| jhesketh | anteaya: it's one aspect ;-) | 23:03 |
| *** Krinkle|detached is now known as Krinkle | 23:03 | |
| *** dangers is now known as dangers_away | 23:03 | |
| fungi | mtreinish: i was tempted to pick up a suitable as/400 to run s390 but even old and broken those were a bit out of my price range for what i was doing | 23:04 |
| fungi | so i was unfortunately using the hercules emulator for that platform | 23:05 |
| mtreinish | fungi: heh, I was barely able able to get an s390 vm for a week at ibm when I was writing kernel code intended to be used on it. | 23:07 |
| fungi | but i had working hardware for sparc, alpha, mips, mipsel, hppa, m69k, ppc | 23:07 |
| fungi | er, m68k | 23:07 |
| *** jamielennox|away is now known as jamielennox | 23:07 | |
| fungi | others i'm probably forgetting | 23:07 |
| fungi | this was before they added arm, though i did effectively have it too | 23:07 |
| fungi | i was short s390 and sh | 23:08 |
| mtreinish | fungi: you should have bought a dreamcast :) | 23:08 |
| fungi | yeah, it was on my to do list for sh | 23:08 |
| fungi | i had a dreamcast but hadn't gotten debian running on it | 23:09 |
| *** MarkAtwood has quit IRC | 23:10 | |
| fungi | (well, still have a dreamcast--i wouldn't abandon a working game console after all) | 23:10 |
| *** bknudson has quit IRC | 23:10 | |
| *** chlong has joined #openstack-infra | 23:10 | |
| mtreinish | fungi: you definitely put me to shame, I only ever had ppc, sparc, arm, mips, and ia64 (the last 2 were in my lab though, not really mine) | 23:10 |
| *** Guest48668 has quit IRC | 23:10 | |
| fungi | ahh, yep. ia64... i hadn't gotten my hands on a cheap ia64 either. i don't think there were any | 23:11 |
| *** baoli has joined #openstack-infra | 23:11 | |
| mtreinish | fungi: heh, that just means you need a 2nd one. I have 2 ps3s, one for linux (it used to be my mail server, that was a bad idea) | 23:11 |
| fungi | hah | 23:11 |
| *** sarob has joined #openstack-infra | 23:11 | |
| *** annegent_ has quit IRC | 23:13 | |
| lifeless | fungi: ia64 .... good heaters | 23:15 |
| *** jaypipes has quit IRC | 23:16 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:17 | |
| *** Alexandra_ has quit IRC | 23:18 | |
| mtreinish | lifeless: heh, we used the ia64 box as a music server in the lab, old usb drive farm, and had a serial terminal plugged into it. It was all we could think to do with it | 23:19 |
| mtreinish | it was surprisingly capable at running cmus | 23:19 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:21 | |
| lifeless | mtreinish: it was EPIC | 23:23 |
| fungi | i don't think i ever knew anyone using them in production | 23:23 |
| *** zul has quit IRC | 23:24 | |
| mtreinish | lifeless: hahaha | 23:26 |
| *** markvoelker has joined #openstack-infra | 23:27 | |
| *** ZZelle_ has quit IRC | 23:35 | |
| *** annegentle has joined #openstack-infra | 23:35 | |
| *** annegent_ has joined #openstack-infra | 23:36 | |
| *** annegentle is now known as Guest37267 | 23:36 | |
| *** carl_baldwin has quit IRC | 23:37 | |
| *** sdake has joined #openstack-infra | 23:42 | |
| *** otter768 has joined #openstack-infra | 23:43 | |
| *** sdake_ has quit IRC | 23:45 | |
| *** Guest37267 has quit IRC | 23:47 | |
| *** otter768 has quit IRC | 23:48 | |
| *** annegent_ has quit IRC | 23:48 | |
| *** achanda has quit IRC | 23:51 | |
| *** rlandy has quit IRC | 23:53 | |
| *** markvoelker has quit IRC | 23:53 | |
| *** baoli has quit IRC | 23:53 | |
| *** emagana has quit IRC | 23:54 | |
| *** rfolco has quit IRC | 23:54 | |
| *** emagana has joined #openstack-infra | 23:55 | |
| *** SumitNaiksatam has quit IRC | 23:56 | |
| *** dannywilson has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!