| *** roxanaghe has quit IRC | 00:00 | |
| *** thorst_ has joined #openstack-infra | 00:00 | |
| *** dingyichen has joined #openstack-infra | 00:02 | |
| *** zhurong has quit IRC | 00:03 | |
| *** raunak has quit IRC | 00:03 | |
| *** baoli has quit IRC | 00:04 | |
| *** raunak has joined #openstack-infra | 00:05 | |
| *** thorst_ has quit IRC | 00:09 | |
| *** thorst_ has joined #openstack-infra | 00:10 | |
| *** baoli has joined #openstack-infra | 00:12 | |
| *** fguillot_ has quit IRC | 00:13 | |
| *** fguillot_ has joined #openstack-infra | 00:13 | |
| *** thorst_ has quit IRC | 00:19 | |
| *** sflanigan has joined #openstack-infra | 00:22 | |
| *** tqtran has joined #openstack-infra | 00:24 | |
| *** tqtran has quit IRC | 00:28 | |
| *** thorst_ has joined #openstack-infra | 00:34 | |
| *** pahuang has joined #openstack-infra | 00:35 | |
| *** raunak has quit IRC | 00:37 | |
| *** thorst_ has quit IRC | 00:39 | |
| *** fguillot_ has quit IRC | 00:42 | |
| *** fguillot_ has joined #openstack-infra | 00:43 | |
| *** sarob has joined #openstack-infra | 00:45 | |
| *** greghaynes has quit IRC | 00:48 | |
| *** greghaynes has joined #openstack-infra | 00:49 | |
| *** sarob has quit IRC | 00:49 | |
| openstackgerrit | Craige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio https://review.openstack.org/342481 | 00:49 |
|---|---|---|
| *** amitgandhinz has joined #openstack-infra | 00:51 | |
| *** thorst_ has joined #openstack-infra | 00:53 | |
| *** thorst_ has quit IRC | 00:53 | |
| *** amitgandhinz has quit IRC | 00:56 | |
| *** mriedem has quit IRC | 00:58 | |
| *** Hal has quit IRC | 00:59 | |
| *** fguillot_ has quit IRC | 01:02 | |
| *** bswartz has joined #openstack-infra | 01:02 | |
| *** tphummel has joined #openstack-infra | 01:10 | |
| *** adrian_otto has joined #openstack-infra | 01:12 | |
| *** bswartz has quit IRC | 01:13 | |
| *** roxanaghe has joined #openstack-infra | 01:25 | |
| *** aeng has quit IRC | 01:26 | |
| *** roxanaghe has quit IRC | 01:29 | |
| *** gildub has joined #openstack-infra | 01:32 | |
| *** adrian_otto1 has joined #openstack-infra | 01:33 | |
| *** adrian_otto has quit IRC | 01:34 | |
| *** sflanigan has quit IRC | 01:34 | |
| *** yamahata has joined #openstack-infra | 01:41 | |
| *** tonytan4ever has joined #openstack-infra | 01:42 | |
| *** aeng has joined #openstack-infra | 01:42 | |
| *** yanyanhu has joined #openstack-infra | 01:44 | |
| *** sflanigan has joined #openstack-infra | 01:46 | |
| *** tonytan4ever has quit IRC | 01:47 | |
| openstackgerrit | Craige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio https://review.openstack.org/342481 | 01:51 |
| *** amitgandhinz has joined #openstack-infra | 01:52 | |
| *** thorst_ has joined #openstack-infra | 01:52 | |
| *** amitgandhinz has quit IRC | 01:56 | |
| *** baoli has quit IRC | 01:56 | |
| openstackgerrit | Craige McWhirter proposed openstack-infra/puppet-phabricator: Vagrant files for puppet-phabricator https://review.openstack.org/355273 | 01:58 |
| *** aeng has quit IRC | 01:59 | |
| *** adrian_otto1 has quit IRC | 01:59 | |
| *** sflanigan has quit IRC | 02:00 | |
| *** tonytan4ever has joined #openstack-infra | 02:01 | |
| *** thorst_ has quit IRC | 02:07 | |
| *** thorst_ has joined #openstack-infra | 02:08 | |
| *** jamielennox is now known as jamielennox|away | 02:10 | |
| *** aeng has joined #openstack-infra | 02:12 | |
| *** sflanigan has joined #openstack-infra | 02:12 | |
| *** sflanigan has joined #openstack-infra | 02:12 | |
| *** thorst_ has quit IRC | 02:16 | |
| *** jamielennox|away is now known as jamielennox | 02:30 | |
| *** gongysh has joined #openstack-infra | 02:30 | |
| openstackgerrit | kyle liu proposed openstack-infra/project-config: Add new project networking-zte https://review.openstack.org/355278 | 02:35 |
| *** nwkarsten has joined #openstack-infra | 02:36 | |
| *** gongysh has quit IRC | 02:37 | |
| *** apetrich has joined #openstack-infra | 02:37 | |
| *** gothicmindfood has quit IRC | 02:40 | |
| *** nwkarsten has quit IRC | 02:40 | |
| *** sarob has joined #openstack-infra | 02:46 | |
| *** sarob has quit IRC | 02:50 | |
| *** gildub has quit IRC | 02:51 | |
| *** amitgandhinz has joined #openstack-infra | 02:52 | |
| *** amotoki has quit IRC | 02:54 | |
| *** amotoki has joined #openstack-infra | 02:55 | |
| *** amotoki has quit IRC | 02:56 | |
| *** amitgandhinz has quit IRC | 02:57 | |
| *** tphummel has quit IRC | 02:58 | |
| *** zhurong has joined #openstack-infra | 02:59 | |
| *** nwkarsten has joined #openstack-infra | 03:04 | |
| *** nwkarsten has quit IRC | 03:08 | |
| *** thorst_ has joined #openstack-infra | 03:15 | |
| *** thorst_ has quit IRC | 03:21 | |
| *** yamamoto has joined #openstack-infra | 03:22 | |
| *** baoli has joined #openstack-infra | 03:24 | |
| *** gothicmindfood has joined #openstack-infra | 03:37 | |
| *** gothicmindfood has quit IRC | 03:37 | |
| *** baoli has quit IRC | 03:39 | |
| *** amotoki has joined #openstack-infra | 03:39 | |
| *** vikrant has joined #openstack-infra | 03:40 | |
| *** tonytan4ever has quit IRC | 03:46 | |
| *** ramishra has quit IRC | 03:49 | |
| *** vikrant has quit IRC | 03:51 | |
| *** ramishra has joined #openstack-infra | 03:51 | |
| *** vikrant has joined #openstack-infra | 03:52 | |
| *** amitgandhinz has joined #openstack-infra | 03:53 | |
| *** kzaitsev_mb has quit IRC | 03:53 | |
| *** amitgandhinz has quit IRC | 03:57 | |
| *** amotoki has quit IRC | 04:01 | |
| *** aeng has quit IRC | 04:03 | |
| *** sflanigan has quit IRC | 04:05 | |
| *** vikrant is now known as vikrant|brb | 04:08 | |
| *** amotoki has joined #openstack-infra | 04:09 | |
| *** amotoki has quit IRC | 04:13 | |
| *** amotoki has joined #openstack-infra | 04:17 | |
| *** thorst_ has joined #openstack-infra | 04:18 | |
| *** aeng has joined #openstack-infra | 04:20 | |
| *** vikrant|brb is now known as vikrant | 04:20 | |
| *** chlong has joined #openstack-infra | 04:21 | |
| *** esberglu has joined #openstack-infra | 04:22 | |
| *** armax has joined #openstack-infra | 04:24 | |
| *** thorst_ has quit IRC | 04:26 | |
| *** tqtran has joined #openstack-infra | 04:26 | |
| Jeffrey4l_ | any guys can review this? https://review.openstack.org/355132 | 04:28 |
| *** armax has quit IRC | 04:29 | |
| *** tqtran has quit IRC | 04:30 | |
| Jeffrey4l_ | fungi, anteaya this totally blocked kolla-kube project. | 04:31 |
| Jeffrey4l_ | https://review.openstack.org/355132 | 04:31 |
| *** gouthamr has quit IRC | 04:33 | |
| *** shashank_hegde has joined #openstack-infra | 04:40 | |
| *** _nadya_ has joined #openstack-infra | 04:42 | |
| *** zhurong has quit IRC | 04:43 | |
| *** zhurong has joined #openstack-infra | 04:44 | |
| *** sarob has joined #openstack-infra | 04:46 | |
| *** jaosorior has joined #openstack-infra | 04:47 | |
| *** roxanaghe has joined #openstack-infra | 04:49 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:50 | |
| *** sarob has quit IRC | 04:51 | |
| *** roxanaghe has quit IRC | 04:52 | |
| *** amitgandhinz has joined #openstack-infra | 04:54 | |
| *** psachin has joined #openstack-infra | 04:56 | |
| *** amitgandhinz has quit IRC | 04:58 | |
| *** gildub has joined #openstack-infra | 04:58 | |
| *** aeng has quit IRC | 05:06 | |
| *** bswartz has joined #openstack-infra | 05:07 | |
| *** gildub has quit IRC | 05:10 | |
| *** esberglu has quit IRC | 05:11 | |
| *** _nadya_ has quit IRC | 05:13 | |
| *** pabelanger has quit IRC | 05:14 | |
| *** wfoster has quit IRC | 05:15 | |
| *** pabelanger has joined #openstack-infra | 05:15 | |
| *** lucas-dinner has quit IRC | 05:15 | |
| *** senk_ has joined #openstack-infra | 05:16 | |
| *** aeng has joined #openstack-infra | 05:18 | |
| *** wfoster has joined #openstack-infra | 05:19 | |
| *** lucasagomes has joined #openstack-infra | 05:19 | |
| *** rbergeron has quit IRC | 05:24 | |
| *** rbergeron has joined #openstack-infra | 05:24 | |
| *** dmsimard has quit IRC | 05:25 | |
| *** rcernin has joined #openstack-infra | 05:25 | |
| *** thorst_ has joined #openstack-infra | 05:25 | |
| *** dmsimard has joined #openstack-infra | 05:26 | |
| *** thorst_ has quit IRC | 05:31 | |
| *** gildub has joined #openstack-infra | 05:36 | |
| *** kzaitsev_mb has quit IRC | 05:40 | |
| *** rbuzatu has quit IRC | 05:45 | |
| *** r-mibu has quit IRC | 05:46 | |
| *** r-mibu has joined #openstack-infra | 05:46 | |
| *** baoli has joined #openstack-infra | 05:51 | |
| *** roxanaghe has joined #openstack-infra | 05:52 | |
| *** amitgandhinz has joined #openstack-infra | 05:55 | |
| *** baoli has quit IRC | 05:56 | |
| *** roxanaghe has quit IRC | 05:57 | |
| *** amitgandhinz has quit IRC | 05:59 | |
| *** florianf has joined #openstack-infra | 06:02 | |
| *** rbuzatu has joined #openstack-infra | 06:03 | |
| *** jmccrory is now known as jmccrory_away | 06:12 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject undercloud's CA into python-ironic-agent image https://review.openstack.org/355312 | 06:15 |
| *** zhurong has quit IRC | 06:15 | |
| *** zhurong has joined #openstack-infra | 06:16 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test https://review.openstack.org/355316 | 06:17 |
| *** yamamoto has quit IRC | 06:18 | |
| *** _nadya_ has joined #openstack-infra | 06:20 | |
| *** aviau has quit IRC | 06:22 | |
| *** aviau has joined #openstack-infra | 06:22 | |
| *** rbuzatu has quit IRC | 06:26 | |
| *** armax has joined #openstack-infra | 06:26 | |
| *** arif-ali has quit IRC | 06:27 | |
| *** thorst_ has joined #openstack-infra | 06:29 | |
| *** armax has quit IRC | 06:31 | |
| *** arif-ali has joined #openstack-infra | 06:31 | |
| *** vinaypotluri has quit IRC | 06:31 | |
| *** shashank_hegde has quit IRC | 06:34 | |
| *** rbuzatu has joined #openstack-infra | 06:34 | |
| *** senk_ has quit IRC | 06:35 | |
| *** david-lyle has joined #openstack-infra | 06:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:36 | |
| *** pcaruana has joined #openstack-infra | 06:36 | |
| *** thorst_ has quit IRC | 06:36 | |
| *** shashank_hegde has joined #openstack-infra | 06:37 | |
| *** david-lyle_ has quit IRC | 06:39 | |
| *** kzaitsev_mb has quit IRC | 06:41 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject undercloud's CA into python-ironic-agent image https://review.openstack.org/355312 | 06:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test https://review.openstack.org/355316 | 06:43 |
| *** javeriak has joined #openstack-infra | 06:43 | |
| *** rwsu has joined #openstack-infra | 06:44 | |
| *** _nadya_ has quit IRC | 06:44 | |
| *** martinkopec has joined #openstack-infra | 06:46 | |
| *** tonytan4ever has joined #openstack-infra | 06:47 | |
| openstackgerrit | Merged openstack-infra/zuul: Move other-requirements.txt to bindep.txt https://review.openstack.org/354869 | 06:47 |
| *** esikachev has joined #openstack-infra | 06:48 | |
| *** tonytan4ever has quit IRC | 06:52 | |
| *** rbuzatu has quit IRC | 06:53 | |
| *** amitgandhinz has joined #openstack-infra | 06:55 | |
| *** nmagnezi has joined #openstack-infra | 06:56 | |
| *** skraynev has joined #openstack-infra | 06:56 | |
| *** rbuzatu has joined #openstack-infra | 06:58 | |
| *** yamamoto has joined #openstack-infra | 06:59 | |
| *** amitgandhinz has quit IRC | 07:00 | |
| *** javeriak has quit IRC | 07:00 | |
| *** Thelo has joined #openstack-infra | 07:02 | |
| *** chlong has quit IRC | 07:08 | |
| *** Thelo has quit IRC | 07:10 | |
| *** senk_ has joined #openstack-infra | 07:11 | |
| *** Thelo has joined #openstack-infra | 07:12 | |
| *** Thelo has left #openstack-infra | 07:17 | |
| *** shashank_hegde has quit IRC | 07:20 | |
| *** _nadya_ has joined #openstack-infra | 07:20 | |
| *** jpich has joined #openstack-infra | 07:21 | |
| *** savihou has joined #openstack-infra | 07:21 | |
| *** skraynev has quit IRC | 07:24 | |
| *** skraynev has joined #openstack-infra | 07:25 | |
| *** chlong has joined #openstack-infra | 07:25 | |
| *** ifarkas_afk is now known as ifarkas | 07:26 | |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config: Replace DVR multinode full nv job with DVR scenario tests https://review.openstack.org/355344 | 07:26 |
| *** shashank_hegde has joined #openstack-infra | 07:26 | |
| *** matrohon has joined #openstack-infra | 07:28 | |
| *** ihrachys has joined #openstack-infra | 07:31 | |
| *** amotoki_ has joined #openstack-infra | 07:32 | |
| *** thorst_ has joined #openstack-infra | 07:34 | |
| *** amotoki has quit IRC | 07:35 | |
| *** yamahata has quit IRC | 07:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:37 | |
| *** javeriak has joined #openstack-infra | 07:41 | |
| *** roxanaghe has joined #openstack-infra | 07:41 | |
| *** thorst_ has quit IRC | 07:41 | |
| *** kzaitsev_mb has quit IRC | 07:42 | |
| *** matthewbodkin has joined #openstack-infra | 07:42 | |
| *** roxanaghe has quit IRC | 07:45 | |
| *** bkero-pto is now known as bkero | 07:56 | |
| *** amitgandhinz has joined #openstack-infra | 07:56 | |
| *** dimtruck is now known as zz_dimtruck | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** amitgandhinz has quit IRC | 08:01 | |
| *** sdake has joined #openstack-infra | 08:01 | |
| *** zzzeek has joined #openstack-infra | 08:02 | |
| *** tonytan4ever has joined #openstack-infra | 08:03 | |
| *** savihou has quit IRC | 08:04 | |
| *** chlong has quit IRC | 08:07 | |
| *** tonytan4ever has quit IRC | 08:08 | |
| *** gildub has quit IRC | 08:11 | |
| *** e0ne has joined #openstack-infra | 08:15 | |
| *** sdake has quit IRC | 08:15 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:16 | |
| *** dtantsur|afk is now known as dtantsur | 08:18 | |
| *** dingyichen has quit IRC | 08:24 | |
| *** dingyichen has joined #openstack-infra | 08:25 | |
| *** strigazi is now known as strigazi_AFK | 08:26 | |
| *** ccamacho has joined #openstack-infra | 08:26 | |
| *** ccamacho has quit IRC | 08:26 | |
| *** ccamacho has joined #openstack-infra | 08:26 | |
| *** tqtran has joined #openstack-infra | 08:27 | |
| *** Na3iL has joined #openstack-infra | 08:28 | |
| *** Na3iL has quit IRC | 08:28 | |
| *** dingyichen has quit IRC | 08:31 | |
| *** javeriak_ has joined #openstack-infra | 08:31 | |
| *** tqtran has quit IRC | 08:31 | |
| *** esikachev has quit IRC | 08:31 | |
| *** javeriak has quit IRC | 08:32 | |
| *** javeriak has joined #openstack-infra | 08:32 | |
| *** esikachev has joined #openstack-infra | 08:33 | |
| *** Thelo has joined #openstack-infra | 08:34 | |
| *** e0ne has quit IRC | 08:35 | |
| *** sdake has joined #openstack-infra | 08:35 | |
| *** javeriak_ has quit IRC | 08:36 | |
| *** e0ne has joined #openstack-infra | 08:37 | |
| *** thorst_ has joined #openstack-infra | 08:39 | |
| *** yaume has joined #openstack-infra | 08:40 | |
| *** markvoelker has joined #openstack-infra | 08:41 | |
| *** sdake has quit IRC | 08:43 | |
| *** javeriak has quit IRC | 08:44 | |
| *** javeriak has joined #openstack-infra | 08:44 | |
| *** markvoelker has quit IRC | 08:45 | |
| *** thorst_ has quit IRC | 08:47 | |
| *** Thelo has quit IRC | 08:52 | |
| *** Gibi has joined #openstack-infra | 08:53 | |
| *** Goneri has joined #openstack-infra | 08:55 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test https://review.openstack.org/355378 | 08:55 |
| *** amitgandhinz has joined #openstack-infra | 08:57 | |
| *** electrofelix has joined #openstack-infra | 08:58 | |
| *** rbuzatu has quit IRC | 08:58 | |
| *** tkelsey has joined #openstack-infra | 09:00 | |
| *** yaume has quit IRC | 09:01 | |
| *** amitgandhinz has quit IRC | 09:02 | |
| *** kzaitsev_mb has quit IRC | 09:02 | |
| *** acoles_ is now known as acoles | 09:03 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:06 | |
| *** Hal has joined #openstack-infra | 09:07 | |
| *** kzaitsev_mb has quit IRC | 09:11 | |
| openstackgerrit | Alexey Stepanov proposed openstack-infra/project-config: fuel-qa: stable-mu branches for maintenance and stable for upgrades https://review.openstack.org/355382 | 09:14 |
| *** shashank_hegde has quit IRC | 09:18 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Fix type in test_attr_list handling https://review.openstack.org/355385 | 09:20 |
| *** yaume has joined #openstack-infra | 09:22 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Fix typo in test_attr_list handling https://review.openstack.org/355385 | 09:24 |
| *** savihou has joined #openstack-infra | 09:24 | |
| *** oanson has joined #openstack-infra | 09:28 | |
| *** armax has joined #openstack-infra | 09:28 | |
| *** roxanaghe has joined #openstack-infra | 09:29 | |
| *** sshnaidm has quit IRC | 09:29 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test https://review.openstack.org/355378 | 09:30 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject undercloud's CA into python-ironic-agent image https://review.openstack.org/355312 | 09:30 |
| *** armax has quit IRC | 09:33 | |
| *** roxanaghe has quit IRC | 09:33 | |
| *** jaosorior is now known as jaosorior_brb | 09:38 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Remove the test_attr_prefix before injecting https://review.openstack.org/355393 | 09:43 |
| *** asselin has joined #openstack-infra | 09:45 | |
| *** thorst_ has joined #openstack-infra | 09:45 | |
| *** sdake has joined #openstack-infra | 09:46 | |
| *** yaume has quit IRC | 09:46 | |
| *** asselin_ has quit IRC | 09:48 | |
| Jeffrey4l_ | any guys can review this? https://review.openstack.org/355132 | 09:48 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Remove the test_attr_prefix before injecting https://review.openstack.org/355393 | 09:49 |
| *** yamamoto has quit IRC | 09:50 | |
| *** thorst_ has quit IRC | 09:52 | |
| *** tosky has joined #openstack-infra | 09:55 | |
| openstackgerrit | amrith proposed openstack-infra/project-config: Revert "[trove] Promote scenario tests to voting and gating" https://review.openstack.org/355397 | 09:56 |
| *** gildub has joined #openstack-infra | 09:57 | |
| *** oanson has quit IRC | 09:58 | |
| *** amitgandhinz has joined #openstack-infra | 09:58 | |
| *** rbuzatu has joined #openstack-infra | 09:58 | |
| *** amitgandhinz has quit IRC | 10:02 | |
| *** rbuzatu has quit IRC | 10:03 | |
| *** vgridnev has joined #openstack-infra | 10:03 | |
| vgridnev | hello team, could you please review https://review.openstack.org/#/c/354700/ ? | 10:04 |
| odyssey4me | We have a bot called 'ops-bot' in #openstack-ansible. Before I kick it for being super annoying (it repeats every link pasted in the channel) I'd like to know if the bot is an infra test of some sort? | 10:04 |
| *** tonytan4ever has joined #openstack-infra | 10:04 | |
| openstackgerrit | Merged openstack-infra/release-tools: if we fail to send mail, the job should fail https://review.openstack.org/351860 | 10:05 |
| openstackgerrit | Merged openstack-infra/release-tools: Move other-requirements.txt to bindep.txt https://review.openstack.org/354863 | 10:05 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/subunit2sql: Remove the test_attr_prefix before injecting https://review.openstack.org/355393 | 10:05 |
| *** tonytan4ever has quit IRC | 10:08 | |
| *** zz_dimtruck is now known as dimtruck | 10:18 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:18 | |
| *** rbuzatu has joined #openstack-infra | 10:19 | |
| *** yanyanhu has quit IRC | 10:22 | |
| openstackgerrit | Nadya Shakhat proposed openstack-infra/project-config: Add fuel-plugin-openstack-telemetry https://review.openstack.org/355406 | 10:23 |
| *** Na3iL has joined #openstack-infra | 10:25 | |
| *** asettle has joined #openstack-infra | 10:25 | |
| *** dimtruck is now known as zz_dimtruck | 10:28 | |
| *** jaosorior_brb is now known as jaosorior | 10:31 | |
| *** sdague has joined #openstack-infra | 10:31 | |
| *** yamamoto_ has joined #openstack-infra | 10:32 | |
| *** javeriak has quit IRC | 10:36 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test https://review.openstack.org/355378 | 10:42 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject undercloud's CA into python-ironic-agent image https://review.openstack.org/355312 | 10:42 |
| *** markvoelker has joined #openstack-infra | 10:42 | |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/project-config: designate: Add a non-voting job with postgres as DB backend https://review.openstack.org/354141 | 10:42 |
| *** pblaho has joined #openstack-infra | 10:44 | |
| *** markvoelker has quit IRC | 10:47 | |
| *** Hal has quit IRC | 10:47 | |
| *** amitgandhinz has joined #openstack-infra | 10:58 | |
| *** javeriak has joined #openstack-infra | 11:00 | |
| *** amitgandhinz has quit IRC | 11:03 | |
| *** thorst_ has joined #openstack-infra | 11:04 | |
| *** zhurong has quit IRC | 11:05 | |
| *** skraynev is now known as skraynev__ | 11:05 | |
| *** lucasagomes is now known as lucas-hungry | 11:16 | |
| *** roxanaghe has joined #openstack-infra | 11:17 | |
| *** roxanaghe has quit IRC | 11:22 | |
| *** ldnunes has joined #openstack-infra | 11:25 | |
| *** armax has joined #openstack-infra | 11:30 | |
| *** amotoki_ has quit IRC | 11:31 | |
| *** jkilpatr has joined #openstack-infra | 11:32 | |
| *** amotoki has joined #openstack-infra | 11:34 | |
| *** jaosorior has quit IRC | 11:34 | |
| *** armax has quit IRC | 11:35 | |
| *** jaosorior has joined #openstack-infra | 11:35 | |
| *** asettle has quit IRC | 11:36 | |
| *** amotoki has quit IRC | 11:37 | |
| *** rodrigods has quit IRC | 11:38 | |
| *** rodrigods has joined #openstack-infra | 11:38 | |
| *** rhallisey has joined #openstack-infra | 11:38 | |
| *** rhallisey_ has joined #openstack-infra | 11:39 | |
| *** vgridnev has quit IRC | 11:40 | |
| DuncanT | Can anybody help me understand why the gate-cinder-python27-db-ubuntu-xenial job is marked as failed on https://review.openstack.org/#/c/337061/ please? All the tests seem to list status 'ok' | 11:46 |
| *** amotoki has joined #openstack-infra | 11:48 | |
| *** weshay_afk is now known as weshay | 11:48 | |
| *** asettle has joined #openstack-infra | 11:50 | |
| *** furlongm has joined #openstack-infra | 11:51 | |
| *** furlongm_ has quit IRC | 11:51 | |
| *** rhallisey has quit IRC | 11:51 | |
| *** rfolco has joined #openstack-infra | 11:51 | |
| *** dprince has joined #openstack-infra | 11:52 | |
| *** Na3iL has quit IRC | 11:54 | |
| *** amitgandhinz has joined #openstack-infra | 11:59 | |
| *** baoli_ has joined #openstack-infra | 12:00 | |
| *** vgridnev has joined #openstack-infra | 12:01 | |
| *** sdake has quit IRC | 12:02 | |
| *** apetrich has quit IRC | 12:02 | |
| *** rhallisey has joined #openstack-infra | 12:03 | |
| *** kgiusti has joined #openstack-infra | 12:03 | |
| *** amitgandhinz has quit IRC | 12:04 | |
| *** edmondsw has joined #openstack-infra | 12:04 | |
| *** tonytan4ever has joined #openstack-infra | 12:05 | |
| *** apetrich has joined #openstack-infra | 12:05 | |
| *** yamamoto_ has quit IRC | 12:06 | |
| *** sdake has joined #openstack-infra | 12:06 | |
| *** yamamoto has joined #openstack-infra | 12:07 | |
| *** yamamoto has quit IRC | 12:07 | |
| *** yamamoto has joined #openstack-infra | 12:07 | |
| *** bethwhite has quit IRC | 12:08 | |
| *** tonytan4ever has quit IRC | 12:09 | |
| *** gouthamr has joined #openstack-infra | 12:10 | |
| *** sigmavirus|away is now known as sigmavirus | 12:10 | |
| *** asettle has quit IRC | 12:11 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Implement LXD hypervisor experimental check https://review.openstack.org/355434 | 12:12 |
| *** tonytan4ever has joined #openstack-infra | 12:13 | |
| *** moravec has quit IRC | 12:13 | |
| *** moravec has joined #openstack-infra | 12:14 | |
| mordred | odyssey4me: nope. none of our bots, or our trolls, are named ops-bot | 12:14 |
| odyssey4me | mordred ok awesome, I've banned it for a week - hopefully it won't return | 12:15 |
| mordred | odyssey4me: you don't find repeating links to be useful? | 12:15 |
| mordred | :) | 12:15 |
| *** amotoki has quit IRC | 12:16 | |
| *** bethwhite- has quit IRC | 12:17 | |
| *** roxanaghe has joined #openstack-infra | 12:18 | |
| *** jkilpatr has quit IRC | 12:18 | |
| *** moravec has quit IRC | 12:18 | |
| *** zz_dimtruck is now known as dimtruck | 12:18 | |
| *** gordc has joined #openstack-infra | 12:19 | |
| *** Zara has quit IRC | 12:20 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/diskimage-builder: Change DIB_IPA_CERT resulting file name https://review.openstack.org/355440 | 12:20 |
| *** SotK has quit IRC | 12:20 | |
| *** moravec has joined #openstack-infra | 12:20 | |
| *** amotoki has joined #openstack-infra | 12:23 | |
| *** roxanaghe has quit IRC | 12:23 | |
| *** bethwhite has joined #openstack-infra | 12:24 | |
| *** sshnaidm|afk has joined #openstack-infra | 12:26 | |
| *** lucas-hungry is now known as lucasagomes | 12:27 | |
| *** psilvad has joined #openstack-infra | 12:28 | |
| *** xarses has quit IRC | 12:28 | |
| *** amotoki has quit IRC | 12:28 | |
| *** dimtruck is now known as zz_dimtruck | 12:28 | |
| pleia2 | good morning | 12:29 |
| pleia2 | (east coast again this week) | 12:29 |
| *** markvoelker has joined #openstack-infra | 12:30 | |
| mordred | pleia2: enjoy philly! | 12:31 |
| pleia2 | mordred: thanks :) | 12:31 |
| pleia2 | it was really hot yesterday, but looks like the heat wave broke last night | 12:31 |
| *** woodster_ has joined #openstack-infra | 12:32 | |
| *** pradk has joined #openstack-infra | 12:33 | |
| *** jkilpatr has joined #openstack-infra | 12:33 | |
| *** rhallisey_ has quit IRC | 12:34 | |
| *** mtanino has joined #openstack-infra | 12:35 | |
| mordred | pleia2: good to know it's hot somewhere - it's gotten chilly here - it's only 83 right now! | 12:36 |
| * mordred shivers | 12:36 | |
| *** rlandy has joined #openstack-infra | 12:36 | |
| pleia2 | hah | 12:37 |
| *** sdake has quit IRC | 12:38 | |
| *** amitgandhinz has joined #openstack-infra | 12:39 | |
| mordred | pleia2: if you're bored/waking up - wanna poke an easy project-config patch? https://review.openstack.org/#/c/354795/ | 12:39 |
| pleia2 | mordred: sure | 12:40 |
| mordred | \o/ | 12:41 |
| pleia2 | easy indeed, lgtm | 12:41 |
| * mordred likes to be friendly early in the morning | 12:42 | |
| *** kzaitsev_mb has quit IRC | 12:43 | |
| *** julim has joined #openstack-infra | 12:44 | |
| *** vikrant has quit IRC | 12:46 | |
| *** raildo has joined #openstack-infra | 12:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Remove python3 jobs from nodepool https://review.openstack.org/355449 | 12:46 |
| *** SotK has joined #openstack-infra | 12:47 | |
| *** esberglu has joined #openstack-infra | 12:47 | |
| *** Zara has joined #openstack-infra | 12:47 | |
| *** amoralej|off has quit IRC | 12:50 | |
| *** amoralej has joined #openstack-infra | 12:53 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 12:53 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 12:56 |
| *** xyang1 has joined #openstack-infra | 12:56 | |
| *** devkulkarni has joined #openstack-infra | 13:00 | |
| *** julim has quit IRC | 13:01 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:01 | |
| *** zz_dimtruck is now known as dimtruck | 13:04 | |
| *** julim has joined #openstack-infra | 13:05 | |
| *** ccamacho has quit IRC | 13:05 | |
| *** esberglu has quit IRC | 13:05 | |
| *** esberglu has joined #openstack-infra | 13:05 | |
| *** mdrabe has joined #openstack-infra | 13:08 | |
| *** moravec has quit IRC | 13:08 | |
| *** moravec has joined #openstack-infra | 13:09 | |
| *** esberglu has quit IRC | 13:09 | |
| *** jcoufal has joined #openstack-infra | 13:13 | |
| *** vgridnev has quit IRC | 13:14 | |
| *** asettle has joined #openstack-infra | 13:17 | |
| *** gildub has quit IRC | 13:18 | |
| *** javeriak has quit IRC | 13:18 | |
| *** javeriak has joined #openstack-infra | 13:19 | |
| *** amotoki has joined #openstack-infra | 13:19 | |
| *** devkulkarni has quit IRC | 13:20 | |
| *** javeriak has quit IRC | 13:24 | |
| *** kaisers1 has left #openstack-infra | 13:24 | |
| *** _ari_ has joined #openstack-infra | 13:24 | |
| *** esberglu has joined #openstack-infra | 13:25 | |
| *** matt-borland has joined #openstack-infra | 13:27 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: aclmanager: Reuse releasetools.governance code https://review.openstack.org/355464 | 13:31 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: Remove Release Managers from post-release groups https://review.openstack.org/355465 | 13:31 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: Authenticate before doing group membership tests https://review.openstack.org/355466 | 13:31 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: Use os.path functions instead of string slices https://review.openstack.org/355467 | 13:31 |
| *** signed8bit has joined #openstack-infra | 13:33 | |
| *** tonytan4ever has quit IRC | 13:33 | |
| *** mriedem has joined #openstack-infra | 13:34 | |
| *** _ari_ has quit IRC | 13:36 | |
| odyssey4me | Are the DNS resolvers for nodepool nodes set by something in infra? We're seeing configurations like http://logs.openstack.org/05/350305/4/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-trusty/c61f729/logs/instance-info/host_dns_info_11-47-20.log in failed jobs, and http://logs.openstack.org/01/353701/5/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-trusty/b5cdf99/logs/instance-info/host_dns_info_ | 13:37 |
| odyssey4me | 20-15-16.log in successful ones. | 13:37 |
| *** _nadya_ has quit IRC | 13:37 | |
| *** amitgandhinz has quit IRC | 13:37 | |
| *** amitgandhinz has joined #openstack-infra | 13:38 | |
| *** Shrews has quit IRC | 13:38 | |
| *** rbergeron has quit IRC | 13:43 | |
| *** rbergeron has joined #openstack-infra | 13:43 | |
| *** nwkarsten has joined #openstack-infra | 13:44 | |
| *** bethwhite_ has joined #openstack-infra | 13:44 | |
| dansmith | so I think I just saw zuul's dependency thing do something wrong | 13:45 |
| *** dotplus has joined #openstack-infra | 13:45 | |
| dansmith | if you look at 354265, it depends-on something that was un-merged.. I rechecked it and now it's in check and gate at the same time, | 13:46 |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config: Add scenarios from Neutron to multinode dvr full job https://review.openstack.org/355344 | 13:46 |
| dansmith | wait, nevermind | 13:46 |
| dansmith | the bottom one just got +Wd, so nevermind :) | 13:46 |
| openstackgerrit | Merged openstack-infra/nodepool: Add ZooKeeper connection listener https://review.openstack.org/351910 | 13:47 |
| cloudnull | just as a note to what odyssey4me said, the failed job was OSIC and the success was RAX. so this may be something related to the V6 network update however it looks like the resolvers are getting setup correctly on a new instance built w/ that network -- http://cdn.pasteraw.com/s955knknh6z887pcwn22og7b4b9vnr8 | 13:47 |
| *** martinkopec has quit IRC | 13:47 | |
| odyssey4me | pabelanger mordred ^ | 13:48 |
| *** camunoz has joined #openstack-infra | 13:49 | |
| *** martinkopec has joined #openstack-infra | 13:50 | |
| *** gothicmindfood has joined #openstack-infra | 13:50 | |
| *** devkulkarni has joined #openstack-infra | 13:51 | |
| *** jheroux has joined #openstack-infra | 13:51 | |
| *** yamamoto has quit IRC | 13:52 | |
| *** chlong has joined #openstack-infra | 13:52 | |
| *** valderrv_ has joined #openstack-infra | 13:52 | |
| *** mikeym has joined #openstack-infra | 13:52 | |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| *** ayoung has joined #openstack-infra | 13:55 | |
| *** dprince has quit IRC | 13:55 | |
| *** yamamoto has quit IRC | 13:57 | |
| *** dims has quit IRC | 13:59 | |
| *** adrian_otto has joined #openstack-infra | 14:01 | |
| *** oanson has joined #openstack-infra | 14:01 | |
| *** rvasilets_ has left #openstack-infra | 14:01 | |
| openstackgerrit | Timothy R. Chavez proposed openstack-infra/jenkins-job-builder: Add support for the random string parameter https://review.openstack.org/351384 | 14:02 |
| *** mhickey has joined #openstack-infra | 14:04 | |
| *** dims has joined #openstack-infra | 14:05 | |
| *** roxanaghe has joined #openstack-infra | 14:06 | |
| *** rbrndt has joined #openstack-infra | 14:07 | |
| *** Julien-zte has joined #openstack-infra | 14:08 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add osc-lib and os-client-config to PROJECTS https://review.openstack.org/354795 | 14:08 |
| *** mtanino has quit IRC | 14:09 | |
| jroll | hi, does someone mind looking at a one line project-config change to unbreak ironic stable jobs? https://review.openstack.org/#/c/354608/1 | 14:09 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Remove EPEL usage https://review.openstack.org/347499 | 14:10 |
| *** yamamoto has joined #openstack-infra | 14:10 | |
| *** roxanaghe has quit IRC | 14:10 | |
| *** sdake has joined #openstack-infra | 14:10 | |
| *** devkulkarni has quit IRC | 14:10 | |
| zaro | morning | 14:12 |
| *** adrian_otto has quit IRC | 14:13 | |
| anteaya | morning zaro | 14:13 |
| anteaya | jroll: +2 | 14:13 |
| anteaya | jroll: I was ircing in my dream last night | 14:13 |
| cloudnull | fungi anteaya: too RE: timeouts in the OSIC and the resolvers being set to "127.0.0.1". | 14:13 |
| anteaya | jroll: and I was cleaning up a channel for something but you were still using it | 14:14 |
| cloudnull | ive been hunting aroung however I don't see where the resolvers are being written | 14:14 |
| anteaya | jroll: it is interesting to remember ircing and seeing your username in my dream last night | 14:14 |
| cloudnull | maybe something in the instance setup scripts that I'm just not seeing | 14:14 |
| cloudnull | ? | 14:14 |
| anteaya | cloudnull: I don't know | 14:14 |
| cloudnull | that makes two of us :) | 14:14 |
| *** annegentle has joined #openstack-infra | 14:14 | |
| anteaya | mordred: was up earlier so was pleia2 ^^ | 14:14 |
| cloudnull | good morning btw :) | 14:14 |
| anteaya | cloudnull: at least you are not alone | 14:15 |
| anteaya | good morning to you | 14:15 |
| cloudnull | ++ | 14:15 |
| anteaya | thanks for being so attentive to osic cloud | 14:15 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Move other-requirements.txt to bindep.txt https://review.openstack.org/354857 | 14:15 |
| anteaya | much appreciation to you | 14:15 |
| *** savihou has quit IRC | 14:15 | |
| *** adrian_otto has joined #openstack-infra | 14:15 | |
| jroll | anteaya: heh. I was editing terrible release notes in a dream last night :| | 14:15 |
| cloudnull | its been fun. now to make it even better. | 14:15 |
| jroll | anteaya: also, thanks for the review :) | 14:16 |
| anteaya | hi DuncanT all the -db tests on that patch are failing | 14:17 |
| anteaya | please ask huyang to stop rechecking that patch | 14:17 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test https://review.openstack.org/346949 | 14:17 |
| anteaya | jroll: ha ha, you must have been online in the dream time same time as me | 14:18 |
| anteaya | cloudnull: awesome | 14:18 |
| DuncanT | anteaya: Sure, but if you read the log, there's no indication at all of what failed or why | 14:18 |
| anteaya | jroll: I wonder if I am more efficent working online in the dreamtime than I am when I'm awake | 14:18 |
| DuncanT | anteaya: Every test reports 'ok' | 14:18 |
| anteaya | DuncanT: okay I am looking | 14:18 |
| anteaya | DuncanT: sure sometimes the job fails if something goes wrong in test teardown | 14:19 |
| anteaya | the job and the test are two different things | 14:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Added Gem Mirror to Infra https://review.openstack.org/253616 | 14:19 |
| anteaya | the job runs the test | 14:19 |
| anteaya | and the job does other things | 14:19 |
| anteaya | the job has to succeed for jenkins to report success, not just the test | 14:19 |
| DuncanT | anteaya: And outputs logs. None of which appear to tell me why jenkins is unhappy | 14:19 |
| anteaya | right, looking | 14:20 |
| *** adrian_otto has quit IRC | 14:20 | |
| anteaya | but let's stop rechecking in the meantime | 14:20 |
| anteaya | 2016-08-15 05:42:28.552244 | [Zuul] Job complete, result: FAILURE | 14:20 |
| anteaya | DuncanT: the last ansible command run was TASK [copy] http://logs.openstack.org/61/337061/8/check/gate-cinder-python27-db-ubuntu-xenial/3df65b1/_zuul_ansible/ansible_log.txt | 14:21 |
| anteaya | which recieved no output | 14:21 |
| *** jtomasek is now known as jtomasek|biab | 14:23 | |
| *** burgerk has joined #openstack-infra | 14:23 | |
| *** tonytan4ever has joined #openstack-infra | 14:23 | |
| anteaya | DuncanT: from what I can tell it was able to successfully install python http://logs.openstack.org/61/337061/8/check/gate-cinder-python27-db-ubuntu-xenial/3df65b1/tox/ | 14:24 |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Fix syntax error in ironic-python-agent post job https://review.openstack.org/355487 | 14:24 |
| DuncanT | anteaya: It must have been able to, or the tests passing in console.log would not be passing | 14:24 |
| sbezverk | infra team, please review https://review.openstack.org/355132, this issue completely blocking development on kolla-kubernetes project | 14:26 |
| *** dprince has joined #openstack-infra | 14:27 | |
| *** annegentle has quit IRC | 14:27 | |
| anteaya | DuncanT: sure, I am justing pointing out the place in the log where that is documented | 14:28 |
| anteaya | mordred: can you explain what is happening here in the ansible log? 2016-08-15 05:42:28,386 p=28725 u=zuul | fatal: [node]: FAILED! => {"async_result": {"ansible_job_id": "47026230230.7362" | 14:28 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Implement Swift pypy experimental check https://review.openstack.org/355491 | 14:28 |
| anteaya | mordred: my sense is the words failed and fatal aren't terrific but it is in the middle of the log | 14:28 |
| anteaya | so how fatal is it? | 14:29 |
| *** _nadya_ has joined #openstack-infra | 14:29 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Implement Swift pypy experimental check https://review.openstack.org/355491 | 14:29 |
| *** dimtruck is now known as zz_dimtruck | 14:29 | |
| *** tqtran has joined #openstack-infra | 14:30 | |
| *** oanson has quit IRC | 14:30 | |
| *** pcaruana has quit IRC | 14:30 | |
| *** _nadya_ has quit IRC | 14:30 | |
| *** devkulkarni has joined #openstack-infra | 14:30 | |
| *** thiagop has joined #openstack-infra | 14:31 | |
| *** _nadya_ has joined #openstack-infra | 14:31 | |
| anteaya | DuncanT: from what I can see something in the build didn't do what it was supposed to do for zuul to finish with status success | 14:32 |
| anteaya | DuncanT: also I have not been able to find in the logs, with confidence, anything that shows what that thing was | 14:33 |
| *** armax has joined #openstack-infra | 14:33 | |
| anteaya | we may have to wait for fungi | 14:33 |
| DuncanT | anteaya: That's pretty much where I was too. Thanks. | 14:33 |
| anteaya | DuncanT: thank you, and the person doing rechecks is asleep right now I take it? | 14:33 |
| *** zz_dimtruck is now known as dimtruck | 14:34 | |
| DuncanT | Given his timezone, probably, yes | 14:34 |
| *** tqtran has quit IRC | 14:34 | |
| anteaya | wonderful | 14:34 |
| *** admcleod_ has joined #openstack-infra | 14:35 | |
| anteaya | do you know them, or should I comment on the patch that rechecking over and over isn't the best approach? | 14:35 |
| anteaya | I don't know if they know that | 14:35 |
| *** admcleod has quit IRC | 14:35 | |
| *** _nadya_ has quit IRC | 14:35 | |
| *** cody-somerville has joined #openstack-infra | 14:35 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test. https://review.openstack.org/346949 | 14:36 |
| *** armax has quit IRC | 14:37 | |
| *** zhurong has joined #openstack-infra | 14:37 | |
| *** mdrabe has quit IRC | 14:37 | |
| anteaya | DuncanT: thank you | 14:38 |
| DuncanT | anteaya: For those who don't want to dig into the guts of infra stuff, it actually is actually the best method to get a patch through in practice in cases like this, though it's usually best to give it twelve hours or so between runs | 14:39 |
| *** zhurong has quit IRC | 14:39 | |
| *** bethwhite_ has quit IRC | 14:39 | |
| anteaya | DuncanT: or at least look at the logs and comment saying the logs don't show me a failure | 14:40 |
| *** Julien-zte has quit IRC | 14:40 | |
| anteaya | are programmers unwilling to at least open a log? | 14:40 |
| anteaya | stdout is sharing some input in that logfile | 14:41 |
| anteaya | I'm uncertain if what it is saying is enough to result in failure | 14:42 |
| *** devkulkarni has quit IRC | 14:42 | |
| DuncanT | He did look at console.log, saw all successes and emailed me confused | 14:42 |
| *** krtaylor has quit IRC | 14:42 | |
| *** bethwhite_ has joined #openstack-infra | 14:42 | |
| mordred | 2016-08-15 05:42:28,386 p=28725 u=zuul | fatal: [node]: FAILED! => {"async_result": {"ansible_job_id": "47026230230.7362", "changed": false, "finished": 0, "invocation": {"module_args": {"jid": "47026230230.7362", "mode": "status"}, "module_name": "async_status"}, "started": 1}, "changed": false, "failed": true, "msg": "async task produced unparseable results"} | 14:43 |
| mordred | http://logs.openstack.org/61/337061/8/check/gate-cinder-python27-db-ubuntu-xenial/3df65b1/_zuul_ansible/ansible_log.txt | 14:43 |
| DuncanT | I looked at console.log, saw all success and was confused too. I looked at the end of the other logs, didn't see anything suggesting they failed, was even more confused and so came here | 14:43 |
| mordred | is where the problem is | 14:43 |
| *** devkulkarni has joined #openstack-infra | 14:43 | |
| anteaya | mordred: what is the problem | 14:43 |
| anteaya | I saw that but am unable to understand what it is trying to convey to me | 14:44 |
| mordred | well - there is the problem and then the problem that's causing the problem | 14:44 |
| anteaya | mordred: do expand | 14:44 |
| * anteaya draws up a chair | 14:44 | |
| anteaya | DuncanT: I'm glad you looked, thank you | 14:44 |
| mordred | the direct problem is that the task execution bit of ansible, in this case we use the async task runner, has received additional output from something that is supposed to be only json | 14:44 |
| *** mdrabe has joined #openstack-infra | 14:45 | |
| mordred | the indirect problem is whatever is causing that additional output | 14:45 |
| anteaya | should a test fail in this scenario? | 14:45 |
| mordred | I do not konw the current state of investigation into when this happens | 14:45 |
| *** jdennis1 has quit IRC | 14:45 | |
| anteaya | mordred: so far this is the first incident I have seen | 14:45 |
| *** kzaitsev_mb has quit IRC | 14:46 | |
| anteaya | but I will say I am not up on all the backscroll | 14:46 |
| mordred | well, it is to - as ansible essentially has broken in its ability to read the return code from the test process | 14:46 |
| mordred | so from ansibe's pov all it knows is "something broke" | 14:46 |
| jeblair | mordred: i think investigation needs to be re-opened | 14:46 |
| mordred | jeblair: I agree | 14:46 |
| mordred | jeblair: I thought I remembered pabelanger saying something about library warnings and had a hypothesis | 14:46 |
| mordred | jeblair: luckily - it seems we have a change that consistently fails :) | 14:47 |
| jeblair | mordred: i think you and Shrews landed 2 changes in ansible to address this, right? i don't remember what they were supposed to do though | 14:47 |
| *** sdake has quit IRC | 14:48 | |
| stevemar | can someone help me in getting me added to keystone-release? apparently we're going to be using them for the upcoming release, but i'm not in the group (am in stable release fwiw) | 14:48 |
| *** hongbin has joined #openstack-infra | 14:50 | |
| *** nwkarsten has quit IRC | 14:50 | |
| *** nwkarsten has joined #openstack-infra | 14:51 | |
| DuncanT | mordred: There doesn't seem to be any logging anywhere that lets anybody not intimately familiar with all this debug what actually output the wrong thing though... or even for that matter for somebody who is familiar. Might there be some benefit to adding -vvv to the ansible execution? It gets sent to a separate log, so it won't be noise in the normal case | 14:52 |
| *** jbernard1 has joined #openstack-infra | 14:52 | |
| *** xarses has joined #openstack-infra | 14:53 | |
| *** ociuhandu has joined #openstack-infra | 14:55 | |
| *** nwkarsten has quit IRC | 14:55 | |
| mordred | DuncanT: well, unfortunately in this particular case there is no additional information available that would be any more useful to anyone than the error message that's there... and it's a bug that such a behavior is surfacing to the user at all | 14:56 |
| jeblair | mordred: i found 229d8f6b21109e4180e457d95765379d07af384e | 14:57 |
| jeblair | mordred: wasn't there another one? | 14:57 |
| mordred | DuncanT: I say that not to discount the idea, which is good - but more to give you a sense of where we're at with debugging when this happens - as soon as we can characterize what's actually happening (which we don't know) we'll be able to respond and be resilient - and/or add user facing messages that would help a user deal with it | 14:57 |
| *** dtantsur is now known as dtantsur|mtg | 14:57 | |
| jeblair | mordred: was there a change to actually output what it's failing to parse? | 14:58 |
| *** Julien-zte has joined #openstack-infra | 14:58 | |
| DuncanT | mordred: Fair enough. I'm no kind of ansible expert, but running -vvv is my usual first port of call in debugging things | 14:58 |
| mordred | jeblair: looking | 14:59 |
| jeblair | DuncanT: does running with -vvv output the data that it fails to parse? | 14:59 |
| *** jimbaker has joined #openstack-infra | 14:59 | |
| *** xarses has quit IRC | 14:59 | |
| *** apetrich has quit IRC | 14:59 | |
| *** bin_ has joined #openstack-infra | 15:00 | |
| DuncanT | jeblair: In this specific case, I don't know - I don't know how to repro this environment to find out. In many cases, it prints the command line being run and any unexpected stderr | 15:00 |
| jeblair | DuncanT: the issue in this case is that there is an internal error parsing the json that the ansible async module passes around | 15:01 |
| jeblair | DuncanT: in previous versions of ansible, there was no way to see the data that caused the parse error | 15:01 |
| jeblair | DuncanT: *that* is what we need to proceed in debugging | 15:01 |
| *** spzala has joined #openstack-infra | 15:01 | |
| jeblair | DuncanT: i'm trying now to ascertain if there is such a way in the newly released version | 15:01 |
| DuncanT | jeblair: I know precisely nothing about the async module, but I'd be surprised if it has changed. It's just python code though, right, so we could, in theory, patch it (or, more sensibly, create our own async module with better verbose output and send the patch upstream in the hope we can drop it in future) | 15:02 |
| *** elo has quit IRC | 15:02 | |
| jeblair | DuncanT: well, it was *supposed* to have changed with some patches from mordred and Shrews to address this problem | 15:03 |
| jeblair | mordred: was the other one 4e239f6ce0d8ed96d734ef6ca75fa745c3925045 ? | 15:03 |
| DuncanT | jeblair: Ah, got you. Ok, I'll sit back and wait for a while, clearly I don't have enough history to be more than noise at this point, since I've not got any time to really dig in | 15:03 |
| mordred | jeblair: I do not see that commit? | 15:03 |
| Jeffrey4l_ | any guys can review this? https://review.openstack.org/355132 | 15:04 |
| Jeffrey4l_ | it block kolla-kubernets project now. | 15:04 |
| openstackgerrit | Merged openstack/os-testr: Remove discover from test-requirements https://review.openstack.org/325876 | 15:05 |
| openstackgerrit | Merged openstack/os-testr: Delete openstack/common in flake8 exclude list https://review.openstack.org/355175 | 15:05 |
| *** devkulkarni has quit IRC | 15:05 | |
| *** devkulkarni has joined #openstack-infra | 15:06 | |
| *** devkulkarni has quit IRC | 15:06 | |
| *** senk_ has quit IRC | 15:06 | |
| anteaya | DuncanT: thanks for bringing this to our attention | 15:06 |
| *** rcernin has quit IRC | 15:06 | |
| mordred | jeblair: it looks like we do return "async_result" ... but that does not include the raw thing | 15:06 |
| jeblair | DuncanT: well, if you know anything about how the async module works or have suggestions on how to debug this, that would be great. however, at our volume of work, we can't afford to run with -vvv except for just a few minutes, so we need to know it's going to help. | 15:06 |
| jeblair | mordred: second commit was in modules/core | 15:07 |
| *** nwkarsten has joined #openstack-infra | 15:07 | |
| jeblair | mordred: next step: have ansible log the failed-to-parse data? | 15:08 |
| mordred | jeblair: see it | 15:08 |
| *** Shrews has joined #openstack-infra | 15:09 | |
| mordred | jeblair: so - it didn't fail in utilities/logic/async_wrapper.py best I can tell | 15:11 |
| *** dprince has quit IRC | 15:12 | |
| mordred | jeblair: becuase both of the json parsing exception handlers there set failed=1 in the result dict | 15:12 |
| mordred | but | 15:12 |
| mordred | 2016-08-15 05:42:28,386 p=28725 u=zuul | fatal: [node]: FAILED! => {"async_result": {"ansible_job_id": "47026230230.7362", "changed": false, "finished": 0, "invocation": {"module_args": {"jid": "47026230230.7362", "mode": "status"}, "module_name": "async_status"}, "started": 1}, "changed": false, "failed": true, "msg": "async task produced unparseable results"} | 15:12 |
| mordred | oh. wait. gah | 15:12 |
| mordred | nevermind | 15:12 |
| *** karthik__ has joined #openstack-infra | 15:12 | |
| *** nwkarsten has quit IRC | 15:13 | |
| *** yamamoto has quit IRC | 15:14 | |
| *** yamamoto has joined #openstack-infra | 15:14 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:15 | |
| sbezverk | jeblair: Curios why infra team keeps ignoring the issue we are trying to bring to your team attention for past three days? | 15:17 |
| *** nwkarste_ has joined #openstack-infra | 15:18 | |
| mordred | sbezverk: well, the last two days were the weekend | 15:18 |
| *** javeriak has joined #openstack-infra | 15:18 | |
| sbezverk | mordred: It sounds like a reply I would get from IT if 1990's | 15:18 |
| jeblair | sbezverk: back when people didn't work on weekends? | 15:19 |
| jeblair | sbezverk: i don't work on weekends | 15:19 |
| jeblair | sbezverk: you are welcome to join the infra team and work on weekends if you like | 15:19 |
| *** nwkarst__ has joined #openstack-infra | 15:19 | |
| *** nwkarst__ has quit IRC | 15:19 | |
| sbezverk | if I could +2 I would | 15:19 |
| sbezverk | but I cannot and I have to rely on existing cores | 15:20 |
| jeblair | sbezverk: start by +1ing | 15:20 |
| jeblair | sbezverk: eventually it turns into +2 | 15:20 |
| *** yamamoto has quit IRC | 15:20 | |
| *** jtomasek|biab is now known as jtomasek | 15:20 | |
| jeblair | sbezverk: though the -1 is more important for that | 15:20 |
| *** phschwartz has quit IRC | 15:20 | |
| *** nwkarsten has joined #openstack-infra | 15:20 | |
| sbezverk | jeblair: we have kolla-kube projecy on hold because of the gate | 15:20 |
| *** nwkarsten has quit IRC | 15:20 | |
| sbezverk | it is surprising to see this being ignored.. | 15:21 |
| anteaya | sbezverk: I reviewed the patch | 15:21 |
| anteaya | sbezverk: have you read my review yet? | 15:21 |
| anteaya | 355132 | 15:21 |
| *** nwkarst__ has joined #openstack-infra | 15:22 | |
| odyssey4me | anteaya if you have a moment, reviews of https://review.openstack.org/355491 & https://review.openstack.org/355434 would be appreciated | 15:22 |
| *** nwkarste_ has quit IRC | 15:22 | |
| *** nwkarst__ has quit IRC | 15:22 | |
| anteaya | odyssey4me: sure I'm chairing a meeting | 15:22 |
| anteaya | if I get a moment after that I will look, thank you | 15:23 |
| odyssey4me | sure, once you're done of course | 15:23 |
| *** nwkarste_ has joined #openstack-infra | 15:23 | |
| *** nwkarste_ has quit IRC | 15:23 | |
| anteaya | thank you | 15:23 |
| sbezverk | anteaya: done, I posted | 15:23 |
| sbezverk | agreement | 15:23 |
| *** nwkarste_ has joined #openstack-infra | 15:24 | |
| *** nwkarste_ has quit IRC | 15:24 | |
| anteaya | thank you | 15:24 |
| anteaya | once I finish chairing this meeting I will look again | 15:24 |
| sbezverk | anteaya: thank you | 15:24 |
| anteaya | your welcome | 15:25 |
| *** nwkarste_ has joined #openstack-infra | 15:25 | |
| anteaya | and if you want to start reviewing infra patches, let me know if you want any guidance on that | 15:25 |
| anteaya | happy to have more reviewers | 15:25 |
| * mordred waves at Shrews and hopes he's excited about this morning's ansible issue | 15:26 | |
| Shrews | SO excited | 15:27 |
| Shrews | and i just can't hide it | 15:27 |
| *** nwkarst__ has joined #openstack-infra | 15:27 | |
| *** nmagnezi has quit IRC | 15:28 | |
| *** nwkarst__ has quit IRC | 15:29 | |
| Shrews | jeblair: 354419 seems to have a silly typo in the string format :) | 15:29 |
| *** nwkarst__ has joined #openstack-infra | 15:29 | |
| *** esikachev has quit IRC | 15:29 | |
| *** nwkarste_ has quit IRC | 15:29 | |
| *** nwkarste_ has joined #openstack-infra | 15:30 | |
| *** nwkarste_ has quit IRC | 15:30 | |
| jeblair | Shrews: %i is a thing :) https://docs.python.org/2/library/stdtypes.html#string-formatting-operations | 15:30 |
| jeblair | Shrews: the problem we're looking at today is: | 15:31 |
| jeblair | 15:12 < mordred> 2016-08-15 05:42:28,386 p=28725 u=zuul | fatal: [node]: FAILED! => {"async_result": {"ansible_job_id": "47026230230.7362", "changed": false, "finished": 0, "invocation": {"module_args": {"jid": "47026230230.7362", "mode": "status"}, "module_name": "async_status"}, "started": 1}, "changed": false, "failed": true, "msg": "async task produced unparseable results"} | 15:31 |
| Shrews | jeblair: ugh, my bad. i was going off of this: https://docs.python.org/2/library/string.html#format-specification-mini-language | 15:31 |
| *** nwkarsten has joined #openstack-infra | 15:31 | |
| *** nwkarsten has quit IRC | 15:32 | |
| jeblair | though i almost never use %i because %s is easier than thinking. | 15:32 |
| *** nwkarsten has joined #openstack-infra | 15:33 | |
| Shrews | it's neat that they have that spec in 2 different places and they aren't the same | 15:33 |
| *** armax has joined #openstack-infra | 15:33 | |
| fungi | sbezverk: i was working quite a lot through the weekend (much to my wife's annoyance) but it wasn't obvious to me what was going on there either, nor that it was imminently urgent for you. i'm sorry about that, but i also have to say i find your choice of words rather offensive so please keep discussion constructive in here in the future | 15:33 |
| *** nwkarsten has quit IRC | 15:33 | |
| *** Julien-zte has quit IRC | 15:33 | |
| *** edtubill has joined #openstack-infra | 15:33 | |
| *** nwkarsten has joined #openstack-infra | 15:34 | |
| *** nwkarst__ has quit IRC | 15:34 | |
| *** mdrabe has quit IRC | 15:35 | |
| *** thiagop has quit IRC | 15:35 | |
| *** mdrabe has joined #openstack-infra | 15:35 | |
| *** baoli_ has quit IRC | 15:35 | |
| Shrews | jeblair: yeah, mordred's already brought that to my attention. when i've seen that in the past (i think), it was b/c the results were nothing (not something and that something couldn't be parsed) | 15:35 |
| *** nwkarste_ has joined #openstack-infra | 15:35 | |
| *** nwkarste_ has quit IRC | 15:36 | |
| jeblair | Shrews, mordred: i think the weakness here is that they are not logged. | 15:36 |
| jeblair | so we're still just guessing | 15:36 |
| *** thiagop has joined #openstack-infra | 15:36 | |
| *** nwkarste_ has joined #openstack-infra | 15:36 | |
| mordred | Shrews: could that be another race? like, when you say "results were nothing" - you mean the async result file was empty, yeah? | 15:36 |
| Shrews | mordred: yeah | 15:36 |
| Shrews | mordred: dunno if it's another race though. just stating what i saw before | 15:37 |
| anteaya | fungi: mordred I just skimmed backscroll but thank you for the weekend gerrit reindex | 15:37 |
| *** armax has quit IRC | 15:37 | |
| anteaya | I haven't had a chance to read why it was required yet | 15:38 |
| *** nwkarst__ has joined #openstack-infra | 15:38 | |
| *** nwkarst__ has quit IRC | 15:38 | |
| anteaya | and zaro thanks for your help on the weekend | 15:38 |
| anteaya | and if I missed anyone else | 15:38 |
| *** nwkarsten has quit IRC | 15:38 | |
| *** nwkarsten has joined #openstack-infra | 15:38 | |
| fungi | anteaya: because i missed that puppet was going to care about the ownership on the gerrit build i downloaded and treat correcting it as a gerrit upgrade (and the manifest still tells it to restart gerrit and run an offline reindex when that happens) | 15:39 |
| *** nwkarst__ has joined #openstack-infra | 15:39 | |
| anteaya | ah :( | 15:39 |
| zaro | actually we all forgot about that | 15:39 |
| anteaya | oh my | 15:39 |
| fungi | anteaya: now that https://review.openstack.org/355194 has merged, it shouldn't happen again | 15:39 |
| sbezverk | fungi: Appologies for being offensive, but please understand my frustration too, since friday none of the ready patches got merged and bunch of new PS are all failing and we need to produce working demo in 1 week | 15:40 |
| *** edtubill has quit IRC | 15:40 | |
| *** edtubill has joined #openstack-infra | 15:40 | |
| *** harlowja_at_home has joined #openstack-infra | 15:40 | |
| anteaya | I'm glad you fixed the problem, fungi and zaro and mordred | 15:40 |
| *** nwkarste_ has quit IRC | 15:40 | |
| *** elo has joined #openstack-infra | 15:40 | |
| anteaya | thank you | 15:40 |
| *** nwkarste_ has joined #openstack-infra | 15:41 | |
| *** nwkarste_ has quit IRC | 15:41 | |
| pabelanger | morning | 15:41 |
| *** devkulkarni has joined #openstack-infra | 15:41 | |
| anteaya | sbezverk: what you describe is typical of every group we work with | 15:41 |
| anteaya | we do our very best, every day, all day long | 15:41 |
| *** nwkarste_ has joined #openstack-infra | 15:42 | |
| anteaya | and some folks past that | 15:42 |
| *** nwkarste_ has quit IRC | 15:42 | |
| anteaya | we didn't agree to your timeline, that is your doing | 15:42 |
| anteaya | we have over 100 projects to address and deal with | 15:42 |
| anteaya | our work can be very draining and tiring | 15:42 |
| anteaya | we do best fueled by gratitude | 15:42 |
| anteaya | which is always appreciated | 15:42 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add blurb about communication to docs landing page https://review.openstack.org/355533 | 15:42 |
| fungi | luckily this doesn't feel like a job to me, so i don't mind doing it 60-80 hours a week but we all need to sleep sometime ;) | 15:43 |
| *** nwkarsten has quit IRC | 15:43 | |
| anteaya | now if you need more from us that what we provide on a daily basis, please discuss this with us in advance | 15:43 |
| anteaya | fungi: or mow the lawn as the case may be :) | 15:43 |
| *** nwkarst__ has quit IRC | 15:43 | |
| wznoinsk | lennyb: lennyb: the urllib problem was caused us using wrong version of devstack (hence devstack/lib/tempest) | 15:43 |
| *** nwkarste_ has joined #openstack-infra | 15:44 | |
| *** nwkarste_ has quit IRC | 15:44 | |
| *** nwkarste_ has joined #openstack-infra | 15:44 | |
| *** nwkarste_ has quit IRC | 15:44 | |
| anteaya | wznoinsk: lennyb ah you found the issue? | 15:45 |
| *** weshay is now known as weshay_brb | 15:45 | |
| wznoinsk | wznoinsk: yes, not sure whether that's the same as for others because our issue was caused by using wrong devstack commit | 15:45 |
| *** nwkarste_ has joined #openstack-infra | 15:46 | |
| *** nwkarste_ has quit IRC | 15:46 | |
| wznoinsk | which shouldn't happen when you follow master or stable branches for devstack | 15:46 |
| *** asettle has quit IRC | 15:46 | |
| lennyb | wznoinsk, urllib3==1.14 I guess so, also I've got answer that tempest uses virtual environment I am not sure how/if it works | 15:46 |
| *** vhosakot has joined #openstack-infra | 15:47 | |
| *** nwkarste_ has joined #openstack-infra | 15:47 | |
| *** asettle has joined #openstack-infra | 15:47 | |
| *** nwkarste_ has quit IRC | 15:47 | |
| pabelanger | cloudnull: odyssey4me: We setup a local unbound service, which then forwards to google ipv4: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/finalise.d/89-unbound | 15:47 |
| lennyb | wznoinsk,logs #link http://13.69.151.247/Nova-ML2-Sriov/5408_cloudx-23/ | 15:47 |
| *** nwkarste_ has joined #openstack-infra | 15:48 | |
| jeblair | Shrews, mordred: how should we proceed? | 15:48 |
| pabelanger | cloudnull: odyssey4me: I _think_ we can add forward-addr: google ipv6 dns just about the existing statement on line 29 and hope unbound does the right thing | 15:48 |
| *** elo has quit IRC | 15:48 | |
| pabelanger | I just need to test it on both ipv4 / ipv6 networks | 15:48 |
| lennyb | wznoinsk, I export .._BRANCH=stable/mitaka , also for devstack branch | 15:48 |
| fungi | lennyb: wznoinsk: right, tempest in our jobs is run from within a virtualenv so that it can depend on different versions of things than openstack services/libraries | 15:48 |
| lennyb | fungi, when I've updated urllib3 after devtsack install it worked | 15:49 |
| fungi | particularly necessary since it's branchless so would be hard to make work with teh constrained versions of any of its dependencies from every supported stable branch | 15:49 |
| mtreinish | fungi: well in most cases, there is one job that uses a venv with system site-packages. But there is no reason to use that at all since you can install the plugin very easily in a real venv | 15:49 |
| lennyb | fungi, what do you mean in your jobs? how do you run it? | 15:49 |
| fungi | lennyb: from devstack-gate's default gate_hook | 15:50 |
| lennyb | fungi in our CI, I just cd /opt/tempest; testr run | 15:50 |
| *** nwkarsten has joined #openstack-infra | 15:50 | |
| cloudnull | pabelanger: is that something new-ish ? | 15:50 |
| mordred | jeblair: I do not yet have a good idea - still reading through things | 15:50 |
| *** nwkarst__ has joined #openstack-infra | 15:51 | |
| cloudnull | because we're seeing nameservers set in the resolv.conf file of instances built on other providers, like rax | 15:51 |
| jeblair | mordred: okay, i'll wait till you're done | 15:51 |
| *** Apoorva has joined #openstack-infra | 15:51 | |
| pabelanger | cloudnull: nameserver should be 127.0.0.1 on all images | 15:52 |
| openstackgerrit | greghaynes proposed openstack-infra/project-config: Add notifications for dib changes to openstack-dib https://review.openstack.org/355543 | 15:52 |
| *** nwkarste_ has quit IRC | 15:52 | |
| pabelanger | cloudnull: been that way for a while, IIRC | 15:52 |
| mordred | cloudnull: but the interaction between the unbound and the ipv6-only instances is new ... so that might be where something is going strange? | 15:52 |
| *** vhosakot has quit IRC | 15:52 | |
| *** weshay_brb is now known as weshay | 15:52 | |
| mordred | it might not be - just mainly pointing out that that's the thing that changed | 15:52 |
| *** nwkarste_ has joined #openstack-infra | 15:52 | |
| *** nwkarste_ has quit IRC | 15:53 | |
| anteaya | <-- offline for a bit | 15:53 |
| fungi | lennyb: under tox as the tempest user, from the look of things (sudo -H -u tempest tox) http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n754 | 15:53 |
| *** vhosakot has joined #openstack-infra | 15:53 | |
| *** asettle has quit IRC | 15:53 | |
| *** Sukhdev has joined #openstack-infra | 15:53 | |
| *** nwkarste_ has joined #openstack-infra | 15:53 | |
| *** asettle has joined #openstack-infra | 15:54 | |
| fungi | lennyb: followed by options defining which set of tests to run based on the name of the tox env invoked | 15:54 |
| *** martinkopec has quit IRC | 15:54 | |
| Shrews | jeblair: mordred: we're running the most recent ansible, yeah? | 15:54 |
| *** nwkarsten has quit IRC | 15:55 | |
| jeblair | pabelanger, cloudnull, mordred: if resolv.conf in rax nodes is being set to rax ns, that is unexpected behavior | 15:55 |
| jeblair | Shrews: yes | 15:55 |
| lennyb | fungi: thanks, I will check this approach | 15:55 |
| pabelanger | jeblair: agreed | 15:55 |
| *** nwkarsten has joined #openstack-infra | 15:55 | |
| pabelanger | confirming that now | 15:55 |
| jeblair | pabelanger, cloudnull, mordred: a spot check of an idle rax instance confirms they are being set to rax ns | 15:55 |
| *** nwkarsten has quit IRC | 15:55 | |
| cloudnull | failure in OSIC: http://logs.openstack.org/05/350305/4/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-trusty/c61f729/logs/instance-info/host_dns_info_13-15-18.log | 15:56 |
| cloudnull | success in RAX: http://logs.openstack.org/01/353701/5/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-trusty/b5cdf99/logs/instance-info/host_dns_info_21-37-02.log | 15:56 |
| pabelanger | jeblair: cloudnull: mordred: confirmed. I can work on a patch | 15:56 |
| *** nwkarst__ has quit IRC | 15:56 | |
| *** Na3iL has joined #openstack-infra | 15:56 | |
| *** nwkarsten has joined #openstack-infra | 15:56 | |
| fungi | ouch. i wonder what's updating our resolvers in rax? i thought we were explicitly overriding that because their resolvers were unreliable for our use case | 15:56 |
| jeblair | pabelanger: thanks | 15:56 |
| jeblair | fungi: yes, i am very interested to see how that slipped through. again. | 15:56 |
| fungi | maybe a new glean feature or something | 15:57 |
| *** nwkarsten has quit IRC | 15:57 | |
| *** Apoorva has quit IRC | 15:57 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update M2 Release plugin to use convert xml https://review.openstack.org/346072 | 15:57 |
| *** xarses has joined #openstack-infra | 15:57 | |
| pabelanger | fungi: let me check that first | 15:57 |
| *** mtanino has joined #openstack-infra | 15:57 | |
| cloudnull | fungi: I can also poke about at other regions too to see whats being set. its not total bust for us, we're just seeing some slow gates in the osic and that was a change that stood out. | 15:57 |
| *** vhosakot has quit IRC | 15:57 | |
| cloudnull | 's/change/difference/' | 15:58 |
| *** nwkarsten has joined #openstack-infra | 15:58 | |
| *** nwkarste_ has quit IRC | 15:58 | |
| mordred | fungi: we're fully on glean in rax now, right? | 15:58 |
| *** gothicmindfood has quit IRC | 15:58 | |
| *** vhosakot has joined #openstack-infra | 15:58 | |
| openstackgerrit | Merged openstack-infra/project-config: Make the kolla-kubernetes relate jobs non-voting https://review.openstack.org/355132 | 15:58 |
| fungi | mordred: everywhere, right | 15:58 |
| *** nwkarste_ has joined #openstack-infra | 15:59 | |
| fungi | mordred: and they don't have dhcp and we're not installing nova-agent, so the only likely answer is that glean is setting it based on resolver info in the configdrive metadata | 15:59 |
| pabelanger | I can see DNS servers in network_data.json for rax, check what glean is doing now | 15:59 |
| cloudnull | but to pabelanger point if we could get ipv6 DNS in the unbound forwarder that would be fantastic too | 15:59 |
| fungi | cloudnull: yes, we need to do that | 16:00 |
| *** nwkarst__ has joined #openstack-infra | 16:00 | |
| pabelanger | cloudnull: yup | 16:00 |
| pabelanger | I think we could use glean for that too | 16:00 |
| *** nwkarst__ has quit IRC | 16:00 | |
| pabelanger | oh wai | 16:00 |
| *** jpich has quit IRC | 16:00 | |
| pabelanger | no, that is not correct | 16:00 |
| lennyb | wznoinsk, thanks, I will update you if/when I have more data | 16:00 |
| fungi | clarkb mentioned late last week (friday?) that we were likely hardcoding ipv4 resolvers in our unbound forwarders | 16:00 |
| *** nwkarst__ has joined #openstack-infra | 16:00 | |
| jeblair | yes, probably so | 16:00 |
| *** elo has joined #openstack-infra | 16:01 | |
| fungi | which would need adjusting for v6-only environmentsd | 16:01 |
| *** yamahata has joined #openstack-infra | 16:01 | |
| *** vinaypotluri has joined #openstack-infra | 16:01 | |
| jeblair | 2001:4860:4860::8888 2001:4860:4860::8844 | 16:01 |
| *** ifarkas is now known as ifarkas_afk | 16:01 | |
| *** mat128 is now known as mat128|afk | 16:02 | |
| jeblair | are the v6 addrs for google | 16:02 |
| *** dtantsur|mtg is now known as dtantsur | 16:02 | |
| *** adrian_otto has joined #openstack-infra | 16:02 | |
| *** nwkarsten has quit IRC | 16:02 | |
| *** apetrich has joined #openstack-infra | 16:02 | |
| *** nwkarsten has joined #openstack-infra | 16:03 | |
| *** nwkarsten has quit IRC | 16:03 | |
| fungi | yeah, i think the suggestions were we could check at boot whether we have a global route for ipv6 and if so default to the v6 resolver addresses falling back to the current configuration for v4-only servers, or that we could set it in our nodepool ready scripts based on the provider where we're booting (though that would mean services starting at boot have no working name resolution i guess) | 16:03 |
| *** nwkarste_ has quit IRC | 16:03 | |
| *** nwkarsten has joined #openstack-infra | 16:04 | |
| *** martinkopec has joined #openstack-infra | 16:04 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make side bar the same length as navbar https://review.openstack.org/355554 | 16:04 |
| wznoinsk | lennyb: it looks like a different issue, you get the urllib 3.16 installed for tempest http://13.69.151.247/Nova-ML2-Sriov/5408_cloudx-23/logs/stack.sh.log.gz at 2016-08-11 13:48:49.916 | 16:04 |
| *** wznoinsk has left #openstack-infra | 16:04 | |
| *** wznoinsk has joined #openstack-infra | 16:04 | |
| *** nwkarste_ has joined #openstack-infra | 16:05 | |
| *** elo has quit IRC | 16:05 | |
| fungi | i think things like time synchronization would probably be broken unfortunately so that second idea is probably not viable | 16:05 |
| *** nwkarste_ has quit IRC | 16:05 | |
| *** nwkarst__ has quit IRC | 16:06 | |
| *** nwkarste_ has joined #openstack-infra | 16:06 | |
| pabelanger | jeblair: fungi: mordred: yes, looks like glean is setting up DNS in rax: http://paste.openstack.org/show/557589/ | 16:06 |
| lennyb | wznoinsk but pip freeze shows urllib3=1.14 http://13.69.151.247/Nova-ML2-Sriov/5408_cloudx-23/env/pip-freeze.txt.gz | 16:07 |
| *** martinkopec has quit IRC | 16:07 | |
| wznoinsk | 3.16 was in a venv, if you're running tempest out of venv youre using 3.16, if not then 3.14 | 16:07 |
| *** nwkarst__ has joined #openstack-infra | 16:07 | |
| *** tqtran has joined #openstack-infra | 16:08 | |
| openstackgerrit | Merged openstack-infra/system-config: Add firehose.o.o to cacti https://review.openstack.org/354489 | 16:08 |
| *** nwkarst__ has quit IRC | 16:08 | |
| mordred | pabelanger: yah. are we not re-overwriting that in our ready scripts? | 16:08 |
| jeblair | mordred: we don't set up networking in the ready script | 16:08 |
| pabelanger | right | 16:08 |
| jeblair | mordred: we set up dns in the image | 16:08 |
| *** nwkarsten has quit IRC | 16:08 | |
| *** nwkarst__ has joined #openstack-infra | 16:08 | |
| jeblair | so glean is undoing that | 16:08 |
| lennyb | wznoinsk, I am running tempest from the shell. so I guess it uses 3.14 | 16:09 |
| jeblair | we need to tell glean not to touch resolv.conf | 16:09 |
| *** nwkarst__ has quit IRC | 16:09 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for Fingerprint plugin https://review.openstack.org/345726 | 16:09 |
| pabelanger | it looks like glean.sh can parse something and set switches on glean | 16:09 |
| pabelanger | maybe a /etc/defaults/glean file? | 16:10 |
| *** jtomasek is now known as jtomasek|afk | 16:10 | |
| *** nwkarsten has joined #openstack-infra | 16:10 | |
| fungi | there has been some back and forth on whether glean should have a config file | 16:10 |
| *** ihrachys has quit IRC | 16:10 | |
| *** nwkarsten has quit IRC | 16:10 | |
| *** dims has quit IRC | 16:10 | |
| mordred | yah - and how it should know that a thing like resolv.conf should not be modified | 16:10 |
| *** nwkarsten has joined #openstack-infra | 16:11 | |
| fungi | chattr +i? ;) | 16:11 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make side bar the same length as navbar https://review.openstack.org/355554 | 16:11 |
| jeblair | sadly, we figured out how to convince dhclient and friends not to modify it | 16:11 |
| pabelanger | fungi: nice | 16:11 |
| jeblair | fungi: basically, yes, we did that | 16:11 |
| jeblair | now we have to figure it out again with glean | 16:11 |
| *** nwkarste_ has quit IRC | 16:11 | |
| jeblair | or write a new network bootstrapping system with even *fewer* features | 16:11 |
| wznoinsk | lennyb: tempest log would tell you that, i.e.: http://intel-openstack-ci-logs.ovh/84/352884/1/check/tempest-dsvm-ovsdpdk-nfv-networking/eb36b91/logs/tempest.txt.gz | 16:12 |
| fungi | "glee" | 16:12 |
| *** gyee has joined #openstack-infra | 16:12 | |
| pabelanger | fungi: I lol'd more then I should have on that | 16:12 |
| jeblair | or, i dunno just give up | 16:13 |
| jeblair | and use the cloud dns systems | 16:13 |
| jeblair | i admit, i'm frustrated | 16:13 |
| *** nwkarste_ has joined #openstack-infra | 16:13 | |
| jeblair | because we spent so long getting this right before glean, and then we undid it. | 16:13 |
| *** admcleod has joined #openstack-infra | 16:13 | |
| *** admcleod has joined #openstack-infra | 16:13 | |
| *** nwkarste_ has quit IRC | 16:13 | |
| *** nwkarste_ has joined #openstack-infra | 16:14 | |
| *** admcleod_ has quit IRC | 16:14 | |
| jeblair | i want to say it took us a few months | 16:14 |
| lennyb | wznoinsk, ok, I've got it, I will run tempest from virt env | 16:14 |
| jeblair | because each change is an image rebuild | 16:14 |
| pabelanger | jeblair: what are you thoughts on setting up google ipv6 dns for osic-cloud1? When should we write them to /etc/unbound/forwarding.conf? | 16:14 |
| *** gothicmindfood has joined #openstack-infra | 16:15 | |
| pabelanger | struggling to find the right solution | 16:15 |
| mordred | well, I am going to go back to thinking about the other problem, because I'm finding the tone of dealing with this problem to be quite unpleasant and unproductive | 16:15 |
| jeblair | mordred: thanks | 16:15 |
| *** nwkarst__ has joined #openstack-infra | 16:15 | |
| *** nwkars___ has joined #openstack-infra | 16:16 | |
| *** nwkarsten has quit IRC | 16:16 | |
| *** e0ne has quit IRC | 16:17 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update xvnc to use convert xml https://review.openstack.org/346120 | 16:17 |
| *** nwkarsten has joined #openstack-infra | 16:17 | |
| *** vhosakot has quit IRC | 16:17 | |
| greghaynes | chattr +i seems like a great idea IMO | 16:17 |
| *** nwkarsten has quit IRC | 16:17 | |
| jeblair | pabelanger: i would say we should add the v6 addrs in the same place we add the v4. maybe we can add them both | 16:17 |
| *** lucasagomes is now known as lucas-afk | 16:17 | |
| greghaynes | and if glean doesnt handle chattr +i I'd consider it a glean bug | 16:18 |
| *** nwkarsten has joined #openstack-infra | 16:18 | |
| *** vhosakot has joined #openstack-infra | 16:18 | |
| *** nwkarsten has quit IRC | 16:18 | |
| pabelanger | jeblair: okay, that is DIB element today. I'll continue testing that path, thanks | 16:18 |
| *** nwkarste_ has quit IRC | 16:18 | |
| *** nwkarsten has joined #openstack-infra | 16:18 | |
| *** nwkarst__ has quit IRC | 16:19 | |
| *** karthik__ has quit IRC | 16:19 | |
| *** nwkarste_ has joined #openstack-infra | 16:20 | |
| *** nwkars___ has quit IRC | 16:20 | |
| *** _nadya_ has joined #openstack-infra | 16:21 | |
| jeblair | greghaynes: if we go with chattr we will have *literally* gone full circle with glean: https://review.openstack.org/#/c/90764/ | 16:21 |
| jeblair | greghaynes: https://review.openstack.org/#/c/90423/ | 16:22 |
| jeblair | so, i mean, if that's the interface we want to go with, we do have the patches already written. | 16:22 |
| *** nwkarst__ has joined #openstack-infra | 16:22 | |
| krotscheck | Oh wow, AJaeger isn't around. Is the world ending? | 16:23 |
| *** nwkarst__ has quit IRC | 16:23 | |
| pabelanger | krotscheck: PTO for the next 10days I believe | 16:23 |
| jeblair | greghaynes: otoh, as a user, i don't think it's a good interface, and istr we had many problems with it. | 16:23 |
| krotscheck | pabelanger: Aaaah | 16:23 |
| krotscheck | pabelanger: Smart man to sign out of IRC :) | 16:23 |
| krotscheck | pabelanger: Did the new bindep images get uploaded? | 16:23 |
| *** nwkarst__ has joined #openstack-infra | 16:23 | |
| pabelanger | krotscheck: I believe all clouds are using it | 16:23 |
| *** nwkarsten has quit IRC | 16:24 | |
| greghaynes | jeblair: Do you remember any of the issues with it? My thinking is simply that its a lot less complexity for glean to detect failure when writing to the file as opposed to config parsing | 16:24 |
| *** dprince has joined #openstack-infra | 16:24 | |
| *** lbeliveau has quit IRC | 16:24 | |
| greghaynes | and regardless glean should handle a failure there | 16:24 |
| greghaynes | I lack the context on why you all switched off chattr +i, I thought you always wanted to override resolv.conf to be nameserver 127.0.0.1 because of unbound, so no matter what glean wouldnt be writing the correct thing there | 16:25 |
| jeblair | greghaynes: one of the issues is highlighted in the comments: "Of course this means Puppet won't be able to update it either after this, but we don't plan on changing it." | 16:25 |
| mordred | greghaynes: I think I'm leaning more towards a glean config | 16:26 |
| *** nwkarste_ has quit IRC | 16:26 | |
| *** nwkarste_ has joined #openstack-infra | 16:26 | |
| fungi | don't see a problem with glean supporting its own dedicated configuration, so long as it has sane default behaviors when there is no glean config present | 16:26 |
| mordred | because honestly, inferring whether or not the resolv.conf that came in the image is more valid thatn the metadata provided by the cloud via config-drive or dhcp ... is likely never going to happen | 16:26 |
| mordred | fungi: yah | 16:27 |
| mordred | config should be highly optional | 16:27 |
| Shrews | jeblair: so, i *think* i have identified an issue with the code we added to ansible before for this | 16:27 |
| * mordred concurs with Shrews theory | 16:27 | |
| *** nwkarst__ has quit IRC | 16:28 | |
| *** lbeliveau has joined #openstack-infra | 16:28 | |
| *** harlowja_at_home has quit IRC | 16:28 | |
| *** nwkarst__ has joined #openstack-infra | 16:28 | |
| fungi | i mean, you could do it with envvars passed from the calling startup script or command-line options/arguments, but those are basically just configuration supplied in different ways | 16:28 |
| *** nwkarst__ has quit IRC | 16:28 | |
| greghaynes | Yea, I'd go configuration over env vars. I'm not super opposed to env vars, jsut trying to find the path of least resistence | 16:28 |
| greghaynes | sounds like config file might be that | 16:28 |
| *** hockeynut has joined #openstack-infra | 16:28 | |
| Shrews | jeblair: https://github.com/ansible/ansible/blob/devel/lib/ansible/executor/task_executor.py#L597 | 16:28 |
| greghaynes | er, sorry, not super opposed to a config file | 16:29 |
| fungi | cloud-init and dhclient support configuration files on disk that can tell them to leave resolv.conf alone | 16:29 |
| *** nwkarst__ has joined #openstack-infra | 16:29 | |
| Shrews | jeblair: we should be doing async_result.get('parsed', False) there | 16:29 |
| *** nwkarst__ has quit IRC | 16:29 | |
| *** yamahata has quit IRC | 16:29 | |
| jeblair | greghaynes: it looks like using chattr had a cascading failure effect and broke everything in rackspace, which is why we reverted it over the weekend. it was probably rackspace-specific stuff which doesn't apply now. but i think it caused me to think of the approach as fragile. http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-04-28.log.html | 16:30 |
| Shrews | jeblair: that job that failed seemed to be taking longer than normal (based on past jobs i looked at) which likely gave it more time to fail in that way we are trying to catch there | 16:30 |
| *** nwkarsten has joined #openstack-infra | 16:30 | |
| *** nwkarsten has quit IRC | 16:30 | |
| Shrews | but we failed at catching it | 16:30 |
| *** nwkarste_ has quit IRC | 16:31 | |
| *** dkehn_ has quit IRC | 16:31 | |
| krotscheck | pabelanger: Excellent! | 16:31 |
| *** _nadya_ has quit IRC | 16:31 | |
| greghaynes | ok, so the world just hates readonly files it sounds like | 16:31 |
| *** tqtran has quit IRC | 16:31 | |
| *** dkehn has quit IRC | 16:31 | |
| greghaynes | seems plausible given that glean probably explodes right now with them due to the same reasons, its not somthing most folks consider | 16:31 |
| jeblair | Shrews: thinking... | 16:31 |
| krotscheck | infra-core: according to pabelanger, Ajaeger's comment on https://review.openstack.org/#/c/346130/ has now been resolved - can anyone step in and give it the missing +A? (already has 2x+2) | 16:31 |
| *** Apoorva has joined #openstack-infra | 16:31 | |
| *** nwkarste_ has joined #openstack-infra | 16:31 | |
| greghaynes | mordred: config file SGTM - theres another TODO of making glean support not re-asserting state based on machine-id which needs some similar code I think | 16:32 |
| Shrews | jeblair: tl;dr, if 'parsed' isn't in the response, we don't want to quit | 16:32 |
| *** xarses has quit IRC | 16:32 | |
| *** dkehn has joined #openstack-infra | 16:32 | |
| *** Jeffrey4l_ has quit IRC | 16:33 | |
| clarkb | good morning | 16:33 |
| cloudnull | o/ clarkb | 16:33 |
| *** nwkarst__ has joined #openstack-infra | 16:33 | |
| *** nwkarst__ has quit IRC | 16:33 | |
| *** nwkarst__ has joined #openstack-infra | 16:34 | |
| *** Hal has joined #openstack-infra | 16:34 | |
| openstackgerrit | Merged openstack-infra/git-review: Clarify that submitting multiple commits is OK https://review.openstack.org/351888 | 16:34 |
| *** nwkars___ has joined #openstack-infra | 16:35 | |
| jeblair | Shrews, mordred: i agree, defaulting the getter to true does not match the behavior in the comment. we have probably effectively changed many of the "timeout" errors to "unparseable" errors with that change. | 16:35 |
| mordred | jeblair: ++ | 16:35 |
| jeblair | Shrews, mordred: could probably just be "async_result.get('parsed')" | 16:35 |
| mordred | jeblair: we want to bail from the loop when we have parsed a failure | 16:35 |
| mordred | yah | 16:35 |
| *** nwkarsten has joined #openstack-infra | 16:36 | |
| *** nwkarsten has quit IRC | 16:36 | |
| jeblair | (to match the rest of the getters) | 16:36 |
| Shrews | mordred: trying to find your original PR for that... you remember it? | 16:36 |
| mordred | Shrews: I can look | 16:36 |
| clarkb | sbezverk: fwiw you could have made those tox targets return success if things are really that pressing | 16:36 |
| jeblair | Shrews, mordred: https://github.com/ansible/ansible/pull/16458 | 16:36 |
| mordred | https://github.com/ansible/ansible/pull/16458 | 16:37 |
| mordred | gah | 16:37 |
| Shrews | https://github.com/ansible/ansible/pull/16458 | 16:37 |
| mordred | beat me | 16:37 |
| *** nwkarsten has joined #openstack-infra | 16:37 | |
| Shrews | apparently | 16:37 |
| clarkb | sbezverk: its not typically a great idea to have your tests unconditionally pass... but if you are indeed in such a place where you need stuff changed over the weekend that is an option available to you | 16:37 |
| *** nwkarste_ has quit IRC | 16:37 | |
| *** tosky has quit IRC | 16:37 | |
| jeblair | have we ever said that's okay? | 16:38 |
| jeblair | i guess we have now | 16:38 |
| *** nwkarst__ has quit IRC | 16:38 | |
| *** nwkarste_ has joined #openstack-infra | 16:38 | |
| anteaya | morning clarkb | 16:39 |
| *** nwkars___ has quit IRC | 16:39 | |
| zaro | would any infra-core be wiling to help enable gerrit/storyboard integration today? referenced instructions are referenced in commit message: https://review.openstack.org/#/c/347486/ | 16:39 |
| *** sputnik13 has joined #openstack-infra | 16:40 | |
| anteaya | clarkb: I hope you had a wonderful time mostly offline | 16:40 |
| *** tqtran has joined #openstack-infra | 16:40 | |
| clarkb | jeblair: I think it would be nice to have projects use the PTI and not set things nonvoting and just stub stuff out if they have to. Not sure if that is the situation that the kolla folks are in | 16:40 |
| clarkb | jeblair: there is a ton of project-config chrun just to handle "we don't have tests yet" when its trivial to have a single tests that passes | 16:40 |
| clarkb | same thing with docs and pep8 | 16:40 |
| *** florianf has quit IRC | 16:40 | |
| *** nwkarsten has quit IRC | 16:41 | |
| *** devkulkarni1 has joined #openstack-infra | 16:41 | |
| *** nwkarst__ has joined #openstack-infra | 16:41 | |
| *** dims has joined #openstack-infra | 16:42 | |
| Zara | (:D I'm distracted with storyboard js meetup but still excitedly watching gerrit things) | 16:42 |
| *** dkehn has quit IRC | 16:43 | |
| *** nwkarsten has joined #openstack-infra | 16:43 | |
| *** rbuzatu has quit IRC | 16:44 | |
| *** devkulkarni has quit IRC | 16:44 | |
| *** tqtran has quit IRC | 16:44 | |
| *** nwkarste_ has quit IRC | 16:44 | |
| *** rbuzatu has joined #openstack-infra | 16:44 | |
| *** nwkarste_ has joined #openstack-infra | 16:45 | |
| sdague | hmmm... even with wheels, it seems like it's taking us 5 minutes to do pip installs on standard runs, possibly because of bw to our mirrors? | 16:46 |
| pmalik | Hello dear Infra cores. We (Trove/DBaaS) are looking to gather more data on some of our other supported datastores. It would be really helpful to see the tests as 'nv'. Could you possibly review at your discretion: https://review.openstack.org/#/c/354881/ Thanks. | 16:46 |
| sdague | http://logs.openstack.org/81/354981/6/check/gate-novaclient-dsvm-functional/de0c6c4/logs/devstacklog.txt.gz#_2016-08-15_14_38_32_810 - that looks like the download is happening at 4Mbps for numpy | 16:46 |
| *** nwkarst__ has quit IRC | 16:46 | |
| *** nwkarst__ has joined #openstack-infra | 16:46 | |
| *** nwkarst__ has quit IRC | 16:46 | |
| *** dims has quit IRC | 16:47 | |
| *** nwkarst__ has joined #openstack-infra | 16:47 | |
| *** dkehn has joined #openstack-infra | 16:48 | |
| *** nwkarsten has quit IRC | 16:48 | |
| fungi | sdague: does it seem worse in rackspace than elsewhere? we get some pretty terrible network behaviors in their ord region in particular | 16:48 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: increase novaclient functional timeout. https://review.openstack.org/355566 | 16:49 |
| sdague | fungi: I don't have a systemic view here | 16:49 |
| *** nwkarsten has joined #openstack-infra | 16:49 | |
| *** nwkarsten has quit IRC | 16:49 | |
| fungi | yeah, that's not an easy thing to query for | 16:49 |
| *** dkehn_ has joined #openstack-infra | 16:49 | |
| Shrews | jeblair: https://github.com/ansible/ansible/pull/17091 | 16:49 |
| sdague | but I've failed on job timouts twice on rax ord and was blown away by the pip_install time | 16:49 |
| sdague | pip_install 375 | 16:50 |
| sdague | on that job | 16:50 |
| *** nwkarsten has joined #openstack-infra | 16:50 | |
| *** nwkarsten has quit IRC | 16:50 | |
| *** nwkarste_ has quit IRC | 16:50 | |
| fungi | sdague: i think we need bigger servers... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=3063&rra_id=all | 16:50 |
| fungi | i bet that flavor has a 100mbps bw cap | 16:50 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make side bar the same length as navbar https://review.openstack.org/355554 | 16:50 |
| sdague | vs. pip_install 184 on different provider | 16:50 |
| sdague | fungi: ah, yeh, probably | 16:51 |
| *** nwkarsten has joined #openstack-infra | 16:51 | |
| *** beagles has joined #openstack-infra | 16:51 | |
| sdague | and there are a bunch more nodes in that region than others, right? | 16:51 |
| jeblair | wow, we only recently hit that | 16:51 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review: Use hash of test ID to pick Gerrit ports in tests https://review.openstack.org/285620 | 16:52 |
| sdague | yeh, 195 servers in rax-ord | 16:52 |
| fungi | sdague: yep, so other rackspace regions are likely not seeing this due to lower instance quotas, but also other providers likely don't enforce the same bw limit | 16:52 |
| *** tphummel has joined #openstack-infra | 16:52 | |
| *** nwkarste_ has joined #openstack-infra | 16:52 | |
| *** nwkarste_ has quit IRC | 16:52 | |
| fungi | as jeblair points out, we only just started hitting it in the past few weeks ourselves | 16:52 |
| sdague | so 1/3 of our capacity is hitting that mirror | 16:52 |
| *** nwkarst__ has quit IRC | 16:53 | |
| *** dims has joined #openstack-infra | 16:53 | |
| *** nwkarste_ has joined #openstack-infra | 16:53 | |
| *** adrian_otto has quit IRC | 16:54 | |
| sdague | yeh, though the trend line was there for a while, so it was inevitable | 16:54 |
| *** tonytan4ever has quit IRC | 16:54 | |
| jeblair | it is a disturbing trend line | 16:54 |
| *** nwkarst__ has joined #openstack-infra | 16:55 | |
| *** nwkarst__ has quit IRC | 16:55 | |
| jeblair | to double in 3 months | 16:55 |
| *** nwkarsten has quit IRC | 16:56 | |
| sdague | yeh, it's also the slamming portion of the cycle | 16:56 |
| *** nwkarsten has joined #openstack-infra | 16:56 | |
| *** nwkarsten has quit IRC | 16:56 | |
| sdague | anyway, I guess the question is, are there sensible relief valves here? | 16:56 |
| jeblair | yeah, we can spin up a new mirror | 16:56 |
| *** nwkarsten has joined #openstack-infra | 16:56 | |
| jeblair | infra-root: ^ any volunteers? | 16:56 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists https://review.openstack.org/354730 | 16:57 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Create timeline events for boards and worklists https://review.openstack.org/350146 | 16:57 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API https://review.openstack.org/354729 | 16:57 |
| sdague | how hard would it be to pre cache into the images ? do a pip install upper-constraints.txt in a venv, then delete venv? Then we'd skip a lot of the downloads from the mirror. | 16:57 |
| *** adrian_otto has joined #openstack-infra | 16:57 | |
| anteaya | if the social contracts in a given project are such that a contributor offers a patch to project-config to change tests and noone else in the project was aware of the patch, I think the solution is better socialization within the project regarding change, not changing the behaviour of tests to report false status | 16:58 |
| *** nwkarste_ has quit IRC | 16:58 | |
| *** lucas-afk is now known as lucasagomes | 16:58 | |
| odyssey4me | sdague IIRC you can actually just tell pip to download, not install | 16:58 |
| *** jerryz has joined #openstack-infra | 16:59 | |
| mordred | so ... | 16:59 |
| mordred | we started the entire pre-cache/mirror game with doing caching downloads into the images | 16:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add IPv6 DNS support https://review.openstack.org/355570 | 16:59 |
| mordred | it has almost never worked like expected | 16:59 |
| sdague | mordred: because? | 17:00 |
| *** signed8bit is now known as signed8bit_Zzz | 17:00 | |
| pabelanger | jeblair: clarkb: cloudnull: ^ Some testing on both ipv4 / ipv6 clouds shows that should work for unbound^ | 17:00 |
| *** nwkarste_ has joined #openstack-infra | 17:00 | |
| mordred | sdague: the reasons are varied and I have forgotten many of them - but it was consistently bad enough that we built mirrors instead | 17:00 |
| *** sdake has joined #openstack-infra | 17:00 | |
| sdague | because during a devstack run, we only ever download things once, just through pip's internal cache | 17:01 |
| clarkb | anteaya: definitely. I just know that for many projects in this situation the only reason they fail is tehy haven't configured tox | 17:01 |
| sdague | so if that was already populated with "recently" then it would at least relieve preasure | 17:01 |
| *** nwkarst__ has joined #openstack-infra | 17:01 | |
| clarkb | anteaya: so if we can just get them to do that instead they have tests that work and its less burden on project-config | 17:01 |
| jeblair | clarkb: that wasn't this situation at all | 17:02 |
| sdague | upper-constraints changes would still go through | 17:02 |
| clarkb | jeblair: ok | 17:02 |
| *** hrubi has quit IRC | 17:02 | |
| *** nwkars___ has joined #openstack-infra | 17:03 | |
| *** nwkars___ has quit IRC | 17:03 | |
| *** nwkars___ has joined #openstack-infra | 17:03 | |
| *** _nadya_ has joined #openstack-infra | 17:03 | |
| *** nwkars___ has quit IRC | 17:04 | |
| sdague | anyway, slightly related to that, we need a timeout bump on novaclient functional tests - https://review.openstack.org/#/c/355566/ - which is how I discovered this mirror constraint | 17:04 |
| *** nwkarsten has quit IRC | 17:04 | |
| *** nwkarste_ has quit IRC | 17:04 | |
| *** nwkars___ has joined #openstack-infra | 17:04 | |
| *** signed8bit_Zzz is now known as signed8bit | 17:04 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/project-config: Make the kolla-kubernetes jobs non-voting and experimental https://review.openstack.org/355199 | 17:04 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review: Refactor Isolated Env to use in unit tests https://review.openstack.org/308476 | 17:04 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/git-review: Set author and committer explicitly https://review.openstack.org/222601 | 17:04 |
| mordred | sdague, jeblair: so - it might be worth re-trying. the pip caching code has gotten much better. and we also have the constraints files - so doing a "pip install -d . -c upper-constraints.txt global-requirements.txt" in the image build might work better now than it did a few years ago | 17:05 |
| sdague | mordred: yeh, I wouldn't want to do anything more complicated that pip itself | 17:06 |
| mordred | when we did it last time, the newer pip download cache had not yet been implemented | 17:06 |
| *** jaosorior has quit IRC | 17:06 | |
| *** nwkarst__ has quit IRC | 17:06 | |
| clarkb | mordred: sdague is that still per user? | 17:06 |
| sdague | clarkb: yeh | 17:06 |
| sdague | so just do it as the stack user | 17:06 |
| jeblair | sdague: stack user does not exist | 17:06 |
| fungi | oh, right, the last time we tried there was no such thing as a pip cache or a wheelhouse | 17:06 |
| clarkb | which doesn't actually exist there | 17:06 |
| clarkb | ya | 17:06 |
| sdague | ah... | 17:06 |
| mordred | and stack user would not help non-devstack changes | 17:06 |
| jeblair | sdague: jenkins/zuul is the only user | 17:06 |
| jeblair | sdague: so you'd need to sudo move the cache | 17:07 |
| sdague | mordred: it would not, however devstack changes are probably the biggest consumers | 17:07 |
| *** Goneri has quit IRC | 17:07 | |
| jeblair | (which i believe we also did) | 17:07 |
| mordred | jeblair: ++ | 17:07 |
| mordred | jeblair: I agree with you | 17:07 |
| fungi | presumably devstack-gate could mv/cp/rsync the cache from ~jenkins to ~stack | 17:07 |
| odyssey4me | can the cache path be configured in the global pip.conf perhaps? | 17:07 |
| *** Hal has quit IRC | 17:07 | |
| fungi | odyssey4me: not easily since pip wants it writeable | 17:07 |
| *** mat128|afk is now known as mat128 | 17:07 | |
| *** Hal has joined #openstack-infra | 17:08 | |
| odyssey4me | fungi something like /opt/pip_cache - and just make it writable for anyone/everyone? | 17:08 |
| fungi | and if memory serves it also checks ownership of the cachedir directly, so globally-writeable is probably not a solution | 17:08 |
| *** harlowja has joined #openstack-infra | 17:08 | |
| odyssey4me | ugh | 17:08 |
| electrofelix | YorikSar: I wonder if you might review the response I left on https://review.openstack.org/#/c/222601/ a while back and see if it's acceptable for you? | 17:08 |
| mordred | yah. it does check ownership | 17:08 |
| *** kzaitsev_mb has quit IRC | 17:08 | |
| sdague | ok, I guess this is why we can't have nice things :) | 17:09 |
| jeblair | mordred, sdague: if someone wants to give that a shot, i'm not opposed. it will increase our image sizes of course and consume root filesystem space. it is also probably worth doing a quick test against an unsaturated mirror to find out how much faster we're actually talking about. | 17:09 |
| sdague | never mind then | 17:09 |
| odyssey4me | perhaps an extension of z-c then, which can move the folder appropriately and set the appropriate rights? | 17:09 |
| *** dprince has quit IRC | 17:09 | |
| *** nwkars___ has quit IRC | 17:09 | |
| jeblair | oh, well, never mind then | 17:09 |
| mordred | is bandwidth cached on the private network? and if not, is it viable to try to do config to use private network to hit mirror instead of public? | 17:09 |
| mordred | s/cached/capped/ | 17:09 |
| *** yamahata has joined #openstack-infra | 17:09 | |
| fungi | though having devstack-gate rsync ~jenkins/.cache/pip into ~stack/.cache and ~tempest/.cache when it's also rsync'ing git repos from /opt/git to ~stack/new may make sense? | 17:10 |
| sdague | so the numbers I've got just by poking is that internap is doing the pip installs in < 1/2 the time of rax-ord | 17:10 |
| odyssey4me | mordred that sounds like a nifty idea - it should also kill the L3 interaction which should speed it up | 17:10 |
| sdague | and I think internap nodes are otherwise slower | 17:10 |
| clarkb | odyssey4me: its still L3ing on private net iirc | 17:10 |
| sdague | so back of the envelope, we're probably adding 3 - 4 minutes to every rax job because of the bw constriction | 17:10 |
| clarkb | glean gets a list of nets to route through that interface | 17:11 |
| sdague | rax dsvm job | 17:11 |
| jeblair | sdague: i would like to discount the rax-ord times because the solution to that is easy, get a new server | 17:11 |
| fungi | odyssey4me: mordred: that would also be a fairly rax-centric choice, since we're relying on their rfc-1918 flat net spanning tenants/projects | 17:11 |
| jeblair | sdague: the reason to use a local pip cache, in my mind, is if it's faster than our best-case times on an unsaturated mirror | 17:11 |
| odyssey4me | bah, this is why we can't have nice things :p | 17:12 |
| *** rajinir has joined #openstack-infra | 17:12 | |
| mgagne | sdague: I don't know about RAX but we have a lower number of instances and therefore nodes dedicated (not shared) for ci infra. At this point, you could be your own noisy neighbours. but I didn't fully read backlog =) | 17:12 |
| clarkb | mordred: rereading the bw details for rax the private net can do 2x the public net | 17:12 |
| *** ihrachys has joined #openstack-infra | 17:12 | |
| clarkb | since public net can only utilize 50% of total bandwidth allocation | 17:12 |
| fungi | mgagne: in this specific case it's rackspace's flavor-based bandwidth rate limits | 17:12 |
| * anteaya buys many things at the thrift store as she has accepted she can't have nice things | 17:13 | |
| mordred | clarkb: this: https://support.rackspace.com/how-to/cloud-networks-faq/ says there is no charge for traffic on servicenet - but it does not indicate if there are bandwidth caps | 17:13 |
| clarkb | and 200mbps is the limit for the 2GB flavor and 50% of that is 100mbps which we are seeing | 17:13 |
| fungi | mgagne: the flavor we used for mirror.ord.rax..o.o only gets 100mbps bw, and we're topping out there under load | 17:13 |
| clarkb | mordred: https://www.rackspace.com/cloud/servers/pricing footnote 4 | 17:13 |
| jeblair | are we seriously thinking that we should try to work around this rather than just launch a new server? | 17:14 |
| fungi | yeah, their "200mbps" is 100mbps egress + 100mbps ingress if memory serves | 17:14 |
| clarkb | jeblair: no I think we should make an 8GB instance with 800mbps | 17:14 |
| fungi | jeblair: i think we should just boot a replacement mirror.ord.rax..o.o but i don't personally have time to do it for a few more hours | 17:14 |
| jeblair | clarkb: not a 4g with 400? | 17:14 |
| mgagne | sdague: "I think internap nodes are otherwise slower" are we talking about jobs execution time? (not network) ? | 17:15 |
| clarkb | jeblair: maybe start there and go bigger if necessary | 17:15 |
| pabelanger | fungi: jeblair: I can boot the replacement if needed. | 17:15 |
| fungi | i can get to it later today if we settle on a preferred flavor to replace the current one | 17:15 |
| fungi | pabelanger: oh, thank you! | 17:15 |
| odyssey4me | the simplest solution is certainly the best, although the creative exercise of looking at alternative solutions is also interesting and can sometimes spawn unrelated ideas | 17:15 |
| *** asettle has quit IRC | 17:15 | |
| *** vhosakot has quit IRC | 17:15 | |
| *** sarob has joined #openstack-infra | 17:16 | |
| mordred | odyssey4me: agree. in this case, I think it served to underscore why booting a new server is absolutely the right choice | 17:16 |
| *** Na3iL has quit IRC | 17:16 | |
| *** vhosakot has joined #openstack-infra | 17:16 | |
| *** oanson has joined #openstack-infra | 17:16 | |
| fungi | as to sdague's other request, pre-warming the new afs cache before putting it into production, i don't think we've done that before. it seems probably doable, but it would also be very quickly self-correcting anyway | 17:16 |
| *** sarob has quit IRC | 17:17 | |
| clarkb | fungi: should be as easy as pip installing constraints against the ip addr of the new host | 17:17 |
| pabelanger | so, performance1-4 or performance1-8? Sounds like that is up for debate currently | 17:17 |
| *** sarob has joined #openstack-infra | 17:17 | |
| clarkb | 4GB is fine with me | 17:17 |
| fungi | seems like performance1-4 should be fine | 17:17 |
| pabelanger | okay | 17:17 |
| *** _sarob has joined #openstack-infra | 17:18 | |
| fungi | we've only started hitting 100mbps egress a few weeks ago, so doubling that to 200mbps egress should satisfy us for a while (perhaps indefinitely unless we get a quota bump there) | 17:18 |
| pabelanger | just ord for now? | 17:18 |
| mordred | I don't even see performance1-4 on the pricing list | 17:18 |
| *** matthewbodkin has quit IRC | 17:18 | |
| clarkb | pabelanger: you can probably check the other cacti graphs to see if other instances exhibit the same capped bw behavior | 17:19 |
| fungi | though since we have so much more quota in ord, it's unlikely we're hitting it elsewhere | 17:20 |
| clarkb | dfw and iad don't come close to 100mbps according to cacti | 17:20 |
| fungi | but i agree it deserves being checked | 17:20 |
| *** bethwhite_ has quit IRC | 17:20 | |
| cloudnull | mordred: performance.* flavors are now general.* i believe | 17:21 |
| clarkb | OVH and internap look fine too | 17:21 |
| clarkb | so yes, I think just ord for now | 17:21 |
| * cloudnull assuming your talking about rax | 17:21 | |
| *** tqtran has joined #openstack-infra | 17:21 | |
| sdague | jeblair: ok, I believe that it is, though you'd have to instrument pip maybe to figure out | 17:21 |
| sdague | or add up the size of .pip/cache and do some back of the envelope there | 17:21 |
| *** sarob has quit IRC | 17:22 | |
| *** krtaylor has joined #openstack-infra | 17:22 | |
| mordred | cloudnull: yah | 17:22 |
| clarkb | cloudnull: are there bw limits in osic? | 17:23 |
| cloudnull | nope | 17:23 |
| fungi | pabelanger: i just reviewed all our mirrors, and while some (mirror.bhs1.ovh.o.o) exceed the volume in rax-ord, none of the graphs besides that one show an envelope indicative of a bandwidth cap getting hit | 17:24 |
| pabelanger | fungi: great, thanks | 17:24 |
| pabelanger | new server launching now | 17:24 |
| fungi | oh, though while it sounds like osic is probably fine, it's not in cacti right now | 17:24 |
| *** lucasagomes is now known as lucas-dinner | 17:25 | |
| zaro | fungi: forgot about this one. identified another duplicate cron job for gerrit git gc https://review.openstack.org/#/c/334715/ | 17:25 |
| *** ayoung has quit IRC | 17:25 | |
| *** asettle has joined #openstack-infra | 17:26 | |
| *** dtantsur is now known as dtantsur|afk | 17:26 | |
| cloudnull | clarkb: do you want / need bw limits setup? we could do qos'ing via neutron or setup "tc" rule if needed. | 17:26 |
| *** kzaitsev_mb has joined #openstack-infra | 17:26 | |
| cloudnull | but we're not doing anything as of now | 17:27 |
| clarkb | cloudnull: no I don't think we do :) just double checking we don't need to be aware of that like we have to be in rax | 17:27 |
| anteaya | zaro: does root own that cron job? | 17:27 |
| cloudnull | nope. | 17:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Shut down gearman client in tests https://review.openstack.org/355109 | 17:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Remove testresources https://review.openstack.org/354441 | 17:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Make ZK fixture more robust https://review.openstack.org/355131 | 17:27 |
| *** cody-somerville has quit IRC | 17:28 | |
| zaro | anteaya: it looks like it to me. | 17:29 |
| *** ihrachys has quit IRC | 17:29 | |
| *** tqtran has quit IRC | 17:29 | |
| anteaya | zaro: where are you looking, review-dev? | 17:29 |
| *** cody-somerville has joined #openstack-infra | 17:29 | |
| fungi | zaro: to anteaya's point, it said user=>'gerrit2' before, and that needs to be retained when doing ensure=>absent | 17:29 |
| mordred | clarkb: have a sec and feel like +A on 355131 there? (it makes tests not be flaky) | 17:29 |
| *** rbuzatu has quit IRC | 17:29 | |
| *** vhosakot has quit IRC | 17:30 | |
| clarkb | mordred: trying to catch up on email but I can take a look | 17:30 |
| mordred | clarkb: email is the worst | 17:30 |
| zaro | fungi: ohh right. will fix that. | 17:30 |
| anteaya | zaro: fungi the code says owner gerrit2 on line 374 | 17:30 |
| *** degorenko is now known as _degorenko|afk | 17:30 | |
| *** rbuzatu has joined #openstack-infra | 17:30 | |
| dstufft | new pip cache is awesome, but make sure you have Etags and Cache-Control headers | 17:30 |
| anteaya | is that enough for the cron jobs? | 17:31 |
| dstufft | it needs those | 17:31 |
| fungi | zaro: however i think we also aren't using that particular cronjob in production as it's wrapped in if (!defined(File[$local_git_dir])) | 17:31 |
| *** esikachev has joined #openstack-infra | 17:31 | |
| *** adrian_otto1 has joined #openstack-infra | 17:31 | |
| *** shashank_hegde has joined #openstack-infra | 17:31 | |
| bkero | fungi: So the gerritbot2 work we discussed last week is a bit troublesome. The way that the gerritbot puppet class is made makes it a singleton-per-host. We can switch it from a class to a defined type, but that's going to be nasty to merge -- each bot would have it's own init script, logging config, channel config, maybe ssh keys. | 17:32 |
| mordred | jeblair, pabelanger, Shrews, DuncanT: the presumptive fix for the ansible async issue has been merged upstream | 17:32 |
| bkero | Alternatively we might be able to run it on a different host without modifying it. | 17:32 |
| mordred | of course, for us to pick it up, we'll need to go back to running from git instead of a release | 17:32 |
| anteaya | mordred: wonderful | 17:32 |
| zaro | fungi: local_git_dir is the local replication correct? | 17:32 |
| *** tqtran has joined #openstack-infra | 17:32 | |
| fungi | bkero: shouldn't need separate ssh keys, but it will need separate versions of the rest of that yes. i figured a lot of it would have to become erb templates | 17:33 |
| jeblair | mordred: should we look into running it locally like we did before? | 17:33 |
| zaro | aren't we repicating to local_git_dir on review.o.o? | 17:33 |
| bkero | fungi: Yeah, I have a ~200 line patch to do that | 17:33 |
| bkero | I just don't know how it's ever going to get merged | 17:33 |
| mordred | it has been applied to the stable branch as well - so we could just run off of the upstream stable branch instead of Shrews branch which is based off of tip of devel | 17:33 |
| mordred | jeblair: ^^ | 17:33 |
| fungi | zaro: yes, so that's saying if the local replication directory is not defined then add this cron resource. but as we have local replication set up that file resource already exists so the cron resource never gets added | 17:34 |
| DuncanT | mordred: thanks for the update | 17:34 |
| *** vhosakot has joined #openstack-infra | 17:34 | |
| fungi | zaro: i don't know why it's written that way (looks to me like someone put a } in the wrong place, but there are no comments explaining so maybe it's intentional and i'm just not able to come up with the reasoning) | 17:34 |
| *** sambetts is now known as sambetts|afk | 17:35 | |
| *** adrian_otto has quit IRC | 17:35 | |
| jeblair | pabelanger: can you refresh https://review.openstack.org/355197 ? | 17:36 |
| *** dprince has joined #openstack-infra | 17:36 | |
| jeblair | mordred, pabelanger: let's land that, then we can manually install the ansible upstream stable branch and restart launchers to pick up both changes | 17:36 |
| *** rcernin has joined #openstack-infra | 17:37 | |
| pabelanger | jeblair: looking | 17:38 |
| mordred | jeblair: agree | 17:38 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix link to findbugs minimal example https://review.openstack.org/347602 | 17:38 |
| *** e0ne has joined #openstack-infra | 17:38 | |
| mordred | jeblair: it is confirmed to be in stable-2.1 branch | 17:38 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update HTML Publisher plugin to use convert xml https://review.openstack.org/347605 | 17:39 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Simplify zuul_console port binding logic https://review.openstack.org/355197 | 17:40 |
| *** edtubill has quit IRC | 17:40 | |
| pabelanger | jeblair: updated per your comments | 17:40 |
| *** ayoung has joined #openstack-infra | 17:41 | |
| clarkb | jeblair: mordred for 355131 I wonder if we can tell it to bind on port 0 then get the actual port back sanely (using /proc maybe?) | 17:42 |
| *** tonytan4ever has joined #openstack-infra | 17:45 | |
| *** senk_ has joined #openstack-infra | 17:45 | |
| zaro | fungi: just took a closer look and it seems to me that whole section is just duplicating cron.pp in puppet-gerrit. i think it should be completely removed | 17:46 |
| *** rbrndt has quit IRC | 17:46 | |
| fungi | zaro: i agree. i think its vestigial dead code | 17:46 |
| pabelanger | clarkb: jeblair: fungi: mordred: Can we land https://review.openstack.org/#/c/326649/ so we can use non-root permissions for launch-node.py? | 17:47 |
| *** inc0 has joined #openstack-infra | 17:47 | |
| fungi | pabelanger: was that the only missing piece? | 17:48 |
| *** raunak has joined #openstack-infra | 17:48 | |
| pabelanger | fungi: I believe so | 17:48 |
| clarkb | pabelanger: fungi it will also update the ansible cache iirc. So that needs to be writeable too | 17:48 |
| zaro | fungi: i'm surprised that puppet lint didn't pick up that missing } | 17:48 |
| pabelanger | clarkb: ah, yes. | 17:49 |
| pabelanger | also | 17:49 |
| Shrews | mordred: jeblair: so, this is new in stable-2.1 (https://github.com/ansible/ansible/pull/17003) but i don't immediately see any issues with it. just FYI | 17:49 |
| pabelanger | OS_CLOUD=openstackci-rax OS_REGION=ORD openstack server list is not returning servers from ORD, but DFW | 17:50 |
| fungi | zaro: it's not missing, just several resources after the file resource. in retrospect, i think that was probably added when we moved local mirror handling to the gerrit module and just never cleaned up after | 17:50 |
| Shrews | it also has the fix for the temp dir race | 17:50 |
| pabelanger | I don't know why atm | 17:50 |
| mordred | pabelanger: on puppetmaster? | 17:50 |
| pabelanger | mordred: yes | 17:50 |
| Shrews | jeblair: i think you found that one ^^^^ (re: tmp dir race) | 17:50 |
| mordred | pabelanger: looking | 17:50 |
| clarkb | I always use the openstack flags not the env vars | 17:50 |
| clarkb | fwiw | 17:50 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add mirror.regionone.osic-cloud1.o.o to cacti https://review.openstack.org/355580 | 17:50 |
| mordred | pabelanger: OS_REGION_NAME | 17:51 |
| mordred | not OS_REGION | 17:51 |
| pabelanger | haha | 17:51 |
| pabelanger | launch/README is wrong | 17:51 |
| pabelanger | mordred: thanks | 17:51 |
| *** _nadya_ has quit IRC | 17:51 | |
| fungi | pabelanger: launch/README isn't "wrong" per se. it's just using different envvars in its example shell script than what openstackclient would use | 17:53 |
| jeblair | clarkb: i'm not sure -- i didn't think to ask proc. however, i'm just about convinced that with the zookeeper chroot option, we can drop the per-test fixture and just expect a locally running zk... | 17:54 |
| fungi | it's not passing those to osc | 17:54 |
| *** vhosakot has quit IRC | 17:54 | |
| pabelanger | fungi: Ah, right. That explains it | 17:55 |
| beagles | pabelanger, got some weird stuff happening in some puppet-neutron CI for mitaka where a bunch of ubuntu jobs are failing (see https://review.openstack.org/#/c/355235/) | 17:55 |
| beagles | pabelanger, who should I bug about that? :) | 17:56 |
| *** _nadya_ has joined #openstack-infra | 17:56 | |
| *** _nadya_ has quit IRC | 17:57 | |
| *** kzaitsev_mb has quit IRC | 17:58 | |
| *** Sukhdev has quit IRC | 17:58 | |
| anteaya | beagles: EmilienM is the ptl for puppet-openstacklib: http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml#n4116 | 17:58 |
| anteaya | he might be able to help | 17:59 |
| EmilienM | don't bug me at every bug in puppet modules :) | 17:59 |
| beagles | anteaya, actually thanks for the correction - that's puppet-openstacklib | 17:59 |
| fungi | beagles: those look like they all hit a one-hour timeout running in osic, which i believe is related to the ipv6 dns discussion which was going on in here earlier | 17:59 |
| beagles | anteaya, I was sent in this direction | 17:59 |
| pabelanger | async task produced unparseable results | 17:59 |
| beagles | fungi, interesting | 17:59 |
| pabelanger | http://logs.openstack.org/35/355235/1/check/gate-puppet-openstacklib-puppet-beaker-rspec-ubuntu-trusty/149ed66/_zuul_ansible/ansible_log.txt | 18:00 |
| beagles | pabelanger, yup | 18:00 |
| pabelanger | looks like ansible is failing | 18:00 |
| pabelanger | I think we are working on patching zuul | 18:00 |
| mordred | pabelanger: we just landed a patch upstream for that | 18:00 |
| pabelanger | mordred: ++ | 18:00 |
| sdague | jeblair: as a data point, with a primed cache on my NUC the pip_install time is 72s. So even in our best cases, my guess is that 2/3rds of the pip install time is spent on network | 18:00 |
| mordred | and will roll out the fix to infra at the same time as your other patch | 18:00 |
| fungi | were the job timeouts in osic directly related to the ansible json parsing errors? | 18:00 |
| sdague | basically we've got a fixed cost of ~ 1 minute to install for dsvm runs, and 2 - 6 minutes of network time | 18:01 |
| pabelanger | mordred: great | 18:01 |
| pabelanger | beagles: sounds like fix is in progress | 18:01 |
| beagles | pabelanger, thanks man! | 18:01 |
| *** inc0 has quit IRC | 18:01 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Support lazy resolving of include yaml tags https://review.openstack.org/63580 | 18:07 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Remove duplicate code to setup gerrit local replication https://review.openstack.org/355587 | 18:07 |
| openstackgerrit | Ben Kero proposed openstack-infra/puppet-gerritbot: Refactor bot into defined types to allow multiple bots https://review.openstack.org/355588 | 18:08 |
| bkero | greghaynes: ^ | 18:08 |
| bkero | fungi: ^ | 18:08 |
| bkero | That's also going to need a transition plan :/ | 18:08 |
| jeblair | bkero: quick thought experiment -- how hard to make gerritbot support 2 connections? | 18:09 |
| openstackgerrit | Henry Gessau proposed openstack-infra/project-config: Use python-db-jobs for networking-sfc https://review.openstack.org/354358 | 18:09 |
| bkero | jeblair: the gerritbot project itself? I have no idea, never looked at the source | 18:10 |
| *** elo has joined #openstack-infra | 18:10 | |
| *** ihrachys has joined #openstack-infra | 18:13 | |
| bkero | jeblair: You'd have to do some multiprocess/threaded python, since these just run/spin by themselves: http://git.openstack.org/cgit/openstack-infra/gerritbot/tree/gerritbot/bot.py#n407 | 18:14 |
| *** e0ne has quit IRC | 18:15 | |
| *** nwkarsten has joined #openstack-infra | 18:15 | |
| jeblair | bkero: yeah, i'd imagine it would just end up looking a lot like running 2 bots inside of one process. running 2 processes is probably the better way, just wanted to throw that out there in case it looked too gnarley | 18:15 |
| *** vhosakot has joined #openstack-infra | 18:15 | |
| bkero | jeblair: It's going to look gnarly either way. The easiest way would be to run on a different host. | 18:16 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Allow using lockfile per jenkins master https://review.openstack.org/293631 | 18:16 |
| bkero | but I'm sure that's also fraught with inheritance nightmares | 18:16 |
| *** e0ne has joined #openstack-infra | 18:17 | |
| *** apetrich has quit IRC | 18:18 | |
| jeblair | it also has other drawbacks :) | 18:18 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Output additional info when exceptions occur https://review.openstack.org/309735 | 18:18 |
| *** Apoorva_ has joined #openstack-infra | 18:20 | |
| *** bknudson has joined #openstack-infra | 18:21 | |
| *** inc0 has joined #openstack-infra | 18:21 | |
| greghaynes | bkero: nice | 18:21 |
| *** xyang1 has quit IRC | 18:22 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Refactor base test classes inheritance for reuse https://review.openstack.org/336090 | 18:22 |
| sdague | could I get some reviews on https://review.openstack.org/#/c/355566/ to increase timeouts on novaclient jobs? | 18:24 |
| *** Apoorva has quit IRC | 18:24 | |
| *** vhosakot_ has joined #openstack-infra | 18:24 | |
| openstackgerrit | Ben Kero proposed openstack-infra/puppet-gerritbot: Refactor bot into defined types to allow multiple bots https://review.openstack.org/355588 | 18:24 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Improve logger output for expanding templates https://review.openstack.org/336091 | 18:25 |
| *** vhosakot has quit IRC | 18:25 | |
| *** xyang1 has joined #openstack-infra | 18:26 | |
| *** electrofelix has quit IRC | 18:27 | |
| beagles | pabelanger, mordred: what should I be watching for a heads up that the expected fixes are in? | 18:27 |
| *** senk_ has quit IRC | 18:27 | |
| mordred | beagles: we'll just ping you | 18:28 |
| beagles | mordred, thanks! | 18:28 |
| *** apetrich has joined #openstack-infra | 18:29 | |
| *** acoles is now known as acoles_ | 18:29 | |
| *** csomerville has joined #openstack-infra | 18:30 | |
| *** rbrndt has joined #openstack-infra | 18:30 | |
| *** cody-somerville has quit IRC | 18:33 | |
| *** vhosakot_ has quit IRC | 18:33 | |
| *** ayoung has quit IRC | 18:37 | |
| clarkb | sdague: is that related to the pip bw thing? where are we spending the other 40 minutes? | 18:37 |
| rajinir | Gate seems to be broken. No hosts found to map to cell, exiting. Any ETA? | 18:37 |
| sdague | clarkb: well, there is 7 minutes not in any log files before setup workspace, no idea why | 18:38 |
| sdague | clarkb: but regardless, we've been pushing up towards our time alotment | 18:38 |
| clarkb | rajinir: is there more context for that? like a log file? | 18:38 |
| clarkb | sdague: ya I am ok with bumping it just want to amke sure we don't focus on 4 mintues of extra pip time when we have 40 minutes of setup elsewhere that may actually be the problem | 18:39 |
| sdague | and we need to land code before freeze otherwise basically the nova cli will stop working | 18:39 |
| sdague | clarkb: well, that also impacts dpkg installs | 18:39 |
| pabelanger | rajinir: where did you see that? | 18:39 |
| clarkb | sdague: those are cached though | 18:39 |
| *** tqtran has quit IRC | 18:39 | |
| sdague | clarkb: ok, the biggest mystery to me right now is the missing 7 minutes here - http://logs.openstack.org/81/354981/6/check/gate-novaclient-dsvm-functional/de0c6c4/console.html#_2016-08-15_14_20_49_382031 | 18:40 |
| sdague | because the setupworkspace first log entry is at 27 and change | 18:40 |
| sdague | http://logs.openstack.org/81/354981/6/check/gate-novaclient-dsvm-functional/de0c6c4/logs/devstack-gate-setup-workspace-new.txt.gz | 18:40 |
| *** asettle has quit IRC | 18:41 | |
| rajinir | https://www.irccloud.com/pastebin/9cdm8DQ3/Gate-broken-Aug15 | 18:41 |
| *** jimbaker has quit IRC | 18:41 | |
| *** rcernin has quit IRC | 18:42 | |
| rajinir | clarkb: https://www.irccloud.com/pastebin/9cdm8DQ3/Gate-broken-Aug15 | 18:42 |
| rajinir | pabelanger: I was watching the gate on my thirdparty CI | 18:42 |
| clarkb | looks like we lost time due to ntp | 18:43 |
| * clarkb grumps that ntp isn't more sane | 18:43 | |
| jeblair | clarkb: how can you tell? | 18:44 |
| clarkb | jeblair: http://logs.openstack.org/81/354981/6/check/gate-novaclient-dsvm-functional/de0c6c4/console.html#_2016-08-15_14_19_55_777925 due to that line I am ssuming that the logs don't jump forward in time due to a time update but instaed actually took that long | 18:45 |
| *** jimbaker has joined #openstack-infra | 18:46 | |
| jeblair | clarkb: so you're thinking because it said it failed to sync earlier, it jumped later? | 18:46 |
| *** jimbaker has quit IRC | 18:46 | |
| *** jimbaker has joined #openstack-infra | 18:46 | |
| pabelanger | rajinir: I cannot comment on that, but the gate is not broken. As other projects are passing properly | 18:46 |
| *** Apoorva_ has quit IRC | 18:46 | |
| clarkb | jeblair: ya thats one possibility | 18:46 |
| *** karthik__ has joined #openstack-infra | 18:46 | |
| *** Apoorva has joined #openstack-infra | 18:47 | |
| *** vhosakot has joined #openstack-infra | 18:47 | |
| *** vhosakot has quit IRC | 18:47 | |
| jeblair | what's the 10 minutes before ntp-wait? | 18:47 |
| clarkb | jeblair: I think tahts the 10 minutes of ntp-wait waiting | 18:47 |
| rajinir | pabelanger>: On the ironic channel, a couple of folks are also seeing it | 18:47 |
| jeblair | clarkb: oh, all output at the ned | 18:47 |
| jeblair | end | 18:47 |
| clarkb | ya | 18:47 |
| *** vhosakot has joined #openstack-infra | 18:48 | |
| *** spzala has quit IRC | 18:48 | |
| *** spzala has joined #openstack-infra | 18:48 | |
| clarkb | rajinir: pabelanger I don't see an error in that paste either? looks like just debug logs? | 18:48 |
| *** tqtran has joined #openstack-infra | 18:49 | |
| jeblair | clarkb, sdague: ianw and pabelanger have been looking into ntp issues | 18:49 |
| rajinir | https://www.irccloud.com/pastebin/1OBWSsBL/GateBroken-Aug15 | 18:50 |
| jeblair | sdague: so if we're spending 10 real minutes waiting for ntp to sync, failing, and then losing 7 fake minutes when it eventually comes around, that's going to have an impact. :) | 18:51 |
| clarkb | rajinir: it looks like it is trying to configure cells but the config doesn't exist. You might just be able to run without cells? | 18:51 |
| *** spzala has quit IRC | 18:51 | |
| *** hockeynut has quit IRC | 18:51 | |
| *** spzala has joined #openstack-infra | 18:52 | |
| *** javeriak has quit IRC | 18:52 | |
| *** bstinson has quit IRC | 18:53 | |
| mordred | clarkb: while we're looking at timing things - this one might be related to bandwidth caps and stuff ... | 18:53 |
| mordred | clarkb: but: http://logs.openstack.org/05/351905/7/check/check-osc-plugins/71038e2/console.html#_2016-08-15_17_59_59_241935 | 18:53 |
| *** bstinson has joined #openstack-infra | 18:54 | |
| *** javeriak has joined #openstack-infra | 18:54 | |
| mordred | clarkb: if you scan the log, it looks like every time that tries to touch git.o.o it takes 4 minutes | 18:54 |
| mordred | http://logs.openstack.org/05/351905/7/check/check-osc-plugins/71038e2/console.html#_2016-08-15_17_51_18_401054 | 18:54 |
| mordred | http://logs.openstack.org/05/351905/7/check/check-osc-plugins/71038e2/console.html#_2016-08-15_17_55_42_239654 | 18:54 |
| mordred | http://logs.openstack.org/05/351905/7/check/check-osc-plugins/71038e2/console.html#_2016-08-15_18_04_16_348305 | 18:55 |
| clarkb | I doubt that is related to ntp if it happens more than once. Probably something related to the git mirrors and/or networking and/or git | 18:55 |
| mordred | http://logs.openstack.org/05/351905/7/check/check-osc-plugins/71038e2/console.html#_2016-08-15_18_08_33_496567 | 18:55 |
| mordred | yah | 18:55 |
| mordred | it's always a remote update - and it's always roughly 4 minutes | 18:55 |
| sdague | clarkb: I don't think it's ntp | 18:56 |
| fungi | bandwidth utilization on git.o.o seems to be nearing/reaching 400mbps egress traffic at times http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=862&rra_id=all | 18:56 |
| sdague | syslog has regular logging through the whole window | 18:56 |
| sdague | http://logs.openstack.org/81/354981/6/check/gate-novaclient-dsvm-functional/de0c6c4/logs/syslog.txt.gz | 18:56 |
| sdague | ansible is doing something that's not logging | 18:56 |
| sdague | http://logs.openstack.org/81/354981/6/check/gate-novaclient-dsvm-functional/de0c6c4/logs/syslog.txt.gz#_Aug_15_14_20_51 | 18:57 |
| sdague | oh, it's the filesystem rebuilds | 18:57 |
| sdague | do we still need to do that on nodes? | 18:57 |
| fungi | what's the bw cap for rax's 30gb performance flavor? | 18:57 |
| *** pt_15 has joined #openstack-infra | 18:57 | |
| *** rbuzatu has quit IRC | 18:57 | |
| *** itisha has joined #openstack-infra | 18:58 | |
| clarkb | sdague: we do need swap, and the / is tiny iirc so likely yes we need to make /opt large there | 18:58 |
| sdague | clarkb: ok, that takes 7 minutes | 18:58 |
| sdague | http://logs.openstack.org/81/354981/6/check/gate-novaclient-dsvm-functional/de0c6c4/logs/syslog.txt.gz#_Aug_15_14_27_21 | 18:58 |
| rajinir | clarb: This could be something to with ironic plugin. Discussion happening in ironic channel to revert. thanks | 18:58 |
| jeblair | perhaps it's copying the git repos to the new device that is slow? | 18:59 |
| openstackgerrit | Eddie Ramirez proposed openstack-infra/project-config: Add craton-dashboard repository (Horizon Plugin) https://review.openstack.org/354274 | 18:59 |
| *** _nadya_ has joined #openstack-infra | 18:59 | |
| sdague | jeblair: no, this is the mkfs | 18:59 |
| clarkb | jeblair: looking at sdague's log links it is the mkfs | 18:59 |
| clarkb | since it doesn't mount until 7 minutes later | 18:59 |
| mordred | I concur with sdague | 18:59 |
| sdague | and there are no other logs in this window | 18:59 |
| clarkb | it is possible we want to not utilize the full disk there and make a smaller but large enough fs | 18:59 |
| *** tkelsey has quit IRC | 18:59 | |
| *** tqtran has quit IRC | 18:59 | |
| *** e0ne has quit IRC | 18:59 | |
| jeblair | that is a very long mkfs | 18:59 |
| sdague | I agree, that seems super long | 18:59 |
| fungi | could try -E lazy_itable_init ? | 19:00 |
| jeblair | where's the mkfs command? | 19:00 |
| jeblair | fungi: rxtx_factor is 2500.0 | 19:00 |
| clarkb | oh wait it mounts twice | 19:00 |
| clarkb | the first mount is fast so I don't think it is the mkfs | 19:00 |
| openstackgerrit | Eddie Ramirez proposed openstack-infra/project-config: Add craton-dashboard repository (Horizon Plugin) https://review.openstack.org/354274 | 19:00 |
| mordred | actually - it seems to be the mount | 19:00 |
| mordred | yah - what clarkb said | 19:00 |
| clarkb | jeblair: I think you are right, it mounts first in other location, copies, then chagnes mount | 19:00 |
| clarkb | the copy being the slow bit? | 19:00 |
| sdague | clarkb: oh, yeh, could be | 19:01 |
| fungi | jeblair: okay, so we're nowhere near the bw cap there i guess | 19:01 |
| mordred | how horrible would it be to do this dance in the ready-script rather than in d-g? | 19:01 |
| jeblair | fungi: i forget the rax math needed to get to 'upstream bandwidth' though | 19:01 |
| jeblair | mordred: not every job needs it | 19:02 |
| clarkb | jeblair: fungi its divide that number by 2 and thats your mbps iirc | 19:02 |
| mordred | jeblair: bother | 19:02 |
| clarkb | so 1250mbps for public interface | 19:02 |
| jeblair | clarkb: that means we have 200mbit for our 2gb mirror? | 19:02 |
| *** _sarob has quit IRC | 19:02 | |
| sdague | clarkb: yeh, with the 2 mounts I agree | 19:02 |
| mordred | fungi, clarkb: the "disk" optimized flavors at rackspace have a much higher bandwidth number | 19:02 |
| sdague | this is the find / copy | 19:02 |
| *** sarob has joined #openstack-infra | 19:03 | |
| *** psachin has quit IRC | 19:03 | |
| sdague | https://github.com/openstack-infra/devstack-gate/blob/88a41dab7a56dd96b7abb4f8fcc986d2aeb65cf0/functions.sh#L363 - is the line that seems to take ~7 minutes | 19:03 |
| clarkb | jeblair: hrm ya it should be 200mbps but thats not what we are seeing there. Weird. | 19:03 |
| mordred | scuse me - "I/O Optimized" | 19:03 |
| fungi | so i'm guessing the umount is flushing the write cache | 19:03 |
| fungi | how about we mv the contents of /opt somewhere else on the rootfs, mount the ephemeral disk at /opt, then mv the files into it? | 19:04 |
| mordred | oh - but nevermind- those have huge amounts of cpu and are way more pricey - just a bigger general would meet expanded needs much simpler | 19:04 |
| fungi | then we don't umount and mount it again | 19:04 |
| jeblair | mordred: not always -- io1-30==performance2-30==2500.0 | 19:04 |
| *** sarob has quit IRC | 19:04 | |
| mordred | jeblair: yah - sorry, I was looking at the first table entry and missing the fact that it was a 15G instance | 19:05 |
| jeblair | ya | 19:05 |
| clarkb | fungi: would be easy enough to push a patch that does that and compare times | 19:05 |
| mordred | that seems like a mildly strange definitoin of the smallest "I/O Optmized" flavor | 19:05 |
| clarkb | also need to figure out why ntp-wait is so cranky | 19:05 |
| mordred | clarkb: sync with ianw/pabelanger on that | 19:05 |
| sdague | fungi: ok, while that is going on, anyone want to +A - https://review.openstack.org/#/c/355566/ so we can make forward progress with novaclient? :) | 19:05 |
| mordred | there was a bunch of stuff on that topic towards the end of last week | 19:05 |
| jeblair | mordred: i already mentioned that :) | 19:06 |
| fungi | clarkb: same-fs mv should be atomic and basically instantaneous, so i expect it's a performance improvement to not umount and mount again regardless... just a question of how much | 19:06 |
| mordred | jeblair: yup - it's just been chatty so didn't want clarkb to miss it :) | 19:06 |
| jeblair | clarkb, mordred: some more background reading: https://bugzilla.redhat.com/show_bug.cgi?id=1361382 | 19:06 |
| openstack | bugzilla.redhat.com bug 1361382 in ntp "ntp-wait hangs after boot for a long time, unless ntpd is restarted" [Unspecified,Closed: notabug] - Assigned to mlichvar | 19:06 |
| jeblair | sdague: ^ | 19:06 |
| *** fifieldt has quit IRC | 19:07 | |
| *** edtubill has joined #openstack-infra | 19:08 | |
| *** asselin_ has joined #openstack-infra | 19:08 | |
| jeblair | sdague, clarkb, fungi: is it the case that we need to move the data off of / in order to free up space there for all the installs? | 19:08 |
| *** sarob has joined #openstack-infra | 19:09 | |
| clarkb | jeblair: yes I think so | 19:09 |
| openstackgerrit | Scott DAngelo proposed openstack-infra/project-config: Add experimental Cinder job for multibackend https://review.openstack.org/330678 | 19:09 |
| fungi | jeblair: right, that's why we mv rather than cp | 19:09 |
| clarkb | jeblair: VMs and mysql and friends all need disk | 19:09 |
| sdague | right, but wasn't a bunch of that for hp pathelogical flavors? | 19:10 |
| *** Swami has joined #openstack-infra | 19:10 | |
| clarkb | reading this seems like we could use ntpd -qg ? | 19:10 |
| * fungi wishes ntp.org's ntpd worked like openntpd at startup | 19:10 | |
| clarkb | sdague: rax and hp were basicaly the same | 19:10 |
| jeblair | mordred: when i clone python-aodhclient locally from our git mirrors, it takes 1 second; so i don't know what's taking 4 minutes in that job you linked. | 19:10 |
| clarkb | sdague: tiny / huge ephemeral disk | 19:10 |
| *** asselin has quit IRC | 19:10 | |
| mordred | jeblair: me either - it was the consistency of it across multiple invocations that had me the most concerned | 19:10 |
| fungi | clarkb: i'd have to reread, but it sounds like ntpd -qg can still take 10+ minutes to stabilize | 19:11 |
| clarkb | fungi: -g says "This option allows the time to be set to any value without restriction" | 19:11 |
| sdague | actually, I'm super confused, that log line is here - http://logs.openstack.org/81/354981/6/check/gate-novaclient-dsvm-functional/de0c6c4/logs/devstack-gate-setup-workspace-new.txt.gz#_2016-08-15_14_27_22_667 ? | 19:11 |
| sdague | is ansible just buffering this whole thing and throwing away all the useful timestamp info? | 19:12 |
| *** tqtran has joined #openstack-infra | 19:12 | |
| clarkb | sdague: no I think we do that timestamping outside of ansible | 19:13 |
| clarkb | sdague: with the tooling pulled out of devstack | 19:13 |
| sdague | well, those mount timestamps don't line up with the ones in syslog | 19:13 |
| sdague | and would state that the mv took 0.003s | 19:13 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Clarify OVERWRITE_OLD_IMAGE docs https://review.openstack.org/355607 | 19:14 |
| jeblair | i have to run to lunch now. bbl | 19:14 |
| clarkb | it wouldn't surprise me if it is a buffering issue in the timestamping, just not related to ansible I don't think | 19:14 |
| *** elo has quit IRC | 19:14 | |
| sdague | ok | 19:14 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Clarify OVERWRITE_OLD_IMAGE docs https://review.openstack.org/355607 | 19:15 |
| openstackgerrit | Merged openstack-infra/project-config: increase novaclient functional timeout. https://review.openstack.org/355566 | 19:15 |
| *** asettle has joined #openstack-infra | 19:16 | |
| *** devkulkarni has joined #openstack-infra | 19:17 | |
| *** devkulkarni1 has quit IRC | 19:17 | |
| sdague | ok, well, for right now, we need to get these novaclient bits sorted. So I'm going to switch gears back over to that. | 19:18 |
| clarkb | fungi: looks like we could also just start ntpd with the -g flag | 19:19 |
| *** fifieldt has joined #openstack-infra | 19:19 | |
| *** asettle has quit IRC | 19:19 | |
| fungi | clarkb: yep, i'm trying to see if i can figure out why that's not configurable for the initscript/systemd unit | 19:19 |
| fungi | because if that were generally useful, you'd think it would be a startup option | 19:20 |
| clarkb | ntpd -g for the normal daemon should work if we start within +/-68 years of current time from my reading of docs | 19:21 |
| clarkb | 1970 is only 46 years ago so we should be fine even if we start at the epoch | 19:21 |
| *** Sukhdev has joined #openstack-infra | 19:25 | |
| *** asselin__ has joined #openstack-infra | 19:25 | |
| clarkb | fungi: my tumbleweed system uses -g, but it also has a force set option that will run sntp first | 19:25 |
| clarkb | so -g may not be sufficient? | 19:25 |
| clarkb | fungi: trusty has it set to -g in /etc/default ntp too | 19:27 |
| clarkb | but trusty has no sntp option | 19:27 |
| *** asselin_ has quit IRC | 19:28 | |
| *** asettle has joined #openstack-infra | 19:29 | |
| *** signed8bit is now known as signed8bit_Zzz | 19:30 | |
| *** asettle has quit IRC | 19:31 | |
| *** adrian_otto1 has quit IRC | 19:31 | |
| *** sean-k-mooney has joined #openstack-infra | 19:33 | |
| *** oanson has quit IRC | 19:34 | |
| fungi | clarkb: indeed, my debian systems have NTPD_OPTS='-g' too | 19:36 |
| clarkb | fungi: I think we could do a quick survey of our images and see if they use -g by default and if they do try removing any ntp machinery from our jobs? | 19:38 |
| fungi | "When the initial offset is larger than 0.128s, ntpd will step the clock and then it will wait for at least 900 seconds (in default configuration) before it reports it's in the synchronized state." | 19:38 |
| clarkb | the ntp machinery in our jobs was there to calculate job timeouts, but since ntpd can't skew things drastically those timeouts shouldnb't be terribly affected adn the -g should get us fairly close | 19:38 |
| *** signed8bit_Zzz is now known as signed8bit | 19:38 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for cells v2 setup bug 1613417 https://review.openstack.org/355619 | 19:40 |
| openstack | bug 1613417 in devstack "gate-tempest-dsvm-cells broken with cell v2 setup: "No hosts found to map to cell, exiting."" [Undecided,In progress] https://launchpad.net/bugs/1613417 | 19:40 |
| clarkb | I think we would have to worry about scheduling jobs on insatnces fast enough that -g isn't done doing its thing but I don't expect it to take a ton of time since its supposed to ignore all those pesky limits | 19:40 |
| *** coreyob has quit IRC | 19:40 | |
| fungi | i'm hunting for code or documentation to back up the assertion in that bug report | 19:41 |
| *** jimbaker has quit IRC | 19:41 | |
| *** tonytan4ever has quit IRC | 19:41 | |
| fungi | the implication is that -g will avoid ntpd freaking out and exiting if the initial offset is significant, but won't actually cause it to synchronize to that new time any faster | 19:43 |
| *** amitgandhinz has quit IRC | 19:43 | |
| *** amitgandhinz has joined #openstack-infra | 19:44 | |
| clarkb | ah | 19:44 |
| clarkb | we could stop ntpd, run sntp, start ntpd | 19:45 |
| clarkb | which is similar to how the old ntpdate stuff worked | 19:45 |
| *** jimbaker has joined #openstack-infra | 19:45 | |
| *** jimbaker has quit IRC | 19:45 | |
| *** jimbaker has joined #openstack-infra | 19:45 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:46 | |
| fungi | "Under conditions of extreme network congestion, the roundtrip delay jitter can exceed three seconds and the synchronization distance, which is equal to one-half the roundtrip delay plus error budget terms, can become very large. The ntpd algorithms discard sample offsets exceeding 128 ms, unless the interval during which no sample offset is less than 128 ms exceeds 900s. The first sample after that, | 19:46 |
| fungi | no matter what the offset, steps the clock to the indicated time." | 19:46 |
| fungi | http://doc.ntp.org/4.1.0/ntpd.htm | 19:46 |
| fungi | so i think that means that even at start, if the local time is off by more than 128ms, ntpd won't actually synchronize the clock for 900s | 19:47 |
| clarkb | which is certainly long enough to race job starts | 19:47 |
| fungi | and -g simply keeps ntpd from freaking out at startup if that >128ms skew is large enough to be >1000s | 19:48 |
| *** ayoung has joined #openstack-infra | 19:48 | |
| *** yamahata has quit IRC | 19:49 | |
| *** senk_ has joined #openstack-infra | 19:49 | |
| *** yamahata has joined #openstack-infra | 19:49 | |
| fungi | so, i agree, this seems to be the reason for suggesting sntp | 19:50 |
| *** senk_ has quit IRC | 19:50 | |
| fungi | and centos 7 still has an "ntpdate" service which ntpd depends on for taking acre of that, but in more recent fedora releases they seem to have replaced it with an sntp "service" to do basically the same | 19:52 |
| *** rbuzatu has joined #openstack-infra | 19:52 | |
| clarkb | fungi: are they enabled by default or opt in? | 19:54 |
| clarkb | on suse I Have to set some flag to force sntp | 19:54 |
| pabelanger | fungi: clarkb: jeblair: Took longer then expected, but new mirror server in ord is online: 104.130.70.63 | 19:55 |
| mordred | pabelanger: woot! | 19:55 |
| pabelanger | fungi: clarkb: jeblair: going to enroll into ansible and update DNS | 19:55 |
| clarkb | fungi: I am thinking the simplest thing is to undo ntp-wait and replace ntpdate with sntp | 19:55 |
| clarkb | fungi: in d-g | 19:55 |
| clarkb | fungi: or possibly make sntp part of the ready script | 19:56 |
| clarkb | so that all jobs have sane ntp | 19:56 |
| clarkb | pabelanger: great thank you for getting that up | 19:56 |
| *** rbuzatu has quit IRC | 19:56 | |
| fungi | clarkb: it got discussed in last week's meeting. maybe skim the minutes from here to the end of the topic http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-08-09-19.04.log.html#l-70 | 19:57 |
| fungi | clarkb: basically ntpd is no longer the default time sync solution on rh-based platforms, so we likely want to go with each distro's default implementations | 19:58 |
| *** tonytan4ever has joined #openstack-infra | 19:58 | |
| jeblair | fungi: the new info for me is that apparently ntp-wait is hanging on ubuntu test nodes | 19:58 |
| mordred | same here | 19:59 |
| fungi | which to me means we could add an sntp call in debian/ubuntu, but switch centos/fedora to chrony | 19:59 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Fix template filename https://review.openstack.org/355626 | 19:59 |
| fungi | and probably just drop ntp-wait from d-g altogether? | 19:59 |
| clarkb | fungi: a simple which sntp || which chrony type switch would be fine | 19:59 |
| clarkb | ya | 19:59 |
| *** Goneri has joined #openstack-infra | 20:00 | |
| fungi | basically rely on time sync to become a forced part of node bootup, and let jobs just assume that is a solved problem | 20:00 |
| pabelanger | dns updated, will take 60mins | 20:00 |
| clarkb | fungi: ya, we might also want to talk to debian and ubuntu about supporting a forced thing out of the box | 20:01 |
| clarkb | since from what I can see that doesn't exist currently (but I may be missing some pacakge that adds it) | 20:01 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Add gate-novaclient-dsvm-functional-neutron-nv job https://review.openstack.org/355148 | 20:01 |
| fungi | back to my earlier wistfulness of ntp.org having something akin to openntpd's -s option | 20:02 |
| *** oomichi_ has joined #openstack-infra | 20:02 | |
| clarkb | hrm ubuntu says they have a thing called timedatectl | 20:02 |
| fungi | "-s: Try to set the time immediately at startup, as opposed to slowly adjusting the clock. ntpd will stay in the foreground for up to 15 seconds waiting for one of the configured NTP servers to reply." | 20:02 |
| clarkb | so now we have ntpdate, sntp, chrony, and timedatectl | 20:03 |
| *** sigmavirus is now known as sigmavirus|away | 20:03 | |
| *** oomichi_ is now known as oomichi | 20:03 | |
| fungi | openntpd is packaged on debian/ubuntu as well if you're making a list ;) | 20:03 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Raised max instance in the OSIC https://review.openstack.org/355628 | 20:03 |
| clarkb | but timedatectl won't run if you ahve ntp installed | 20:03 |
| mordred | of course it won't | 20:03 |
| clarkb | I wonder if we just removed our ntp setup completely if things would just work (tm) | 20:03 |
| mordred | why would you ever make a utility that would run if you asked it to run | 20:04 |
| cloudnull | ^ idk if infra core folks want to let my max-instance change in quite yet but i figured i'd put it up. | 20:04 |
| pabelanger | woah | 20:04 |
| fungi | mordred: clearly they think they've put a safety on their foot-cannon | 20:04 |
| *** coreyob has joined #openstack-infra | 20:05 | |
| fungi | cloudnull: you won't find me disagreeing | 20:05 |
| mordred | fungi: I'm pretty sure that the piece of paper tape across the opening on the front of the cannon that says "danger" will keep me from shooting myself | 20:05 |
| * clarkb noms on more tasty VMs | 20:05 | |
| anteaya | cloudnull: what might we be waiting for? | 20:05 |
| mordred | cloudnull: we like your max-instance change | 20:06 |
| *** Apoorva has quit IRC | 20:06 | |
| pabelanger | should we land IPv6 dns first? | 20:06 |
| cloudnull | IDK if there was need to wait on DNS things or what now | 20:06 |
| cloudnull | *not | 20:06 |
| pabelanger | https://review.openstack.org/#/c/355570/ | 20:06 |
| jeblair | we might wait on the zuul telnet fix, or dns | 20:06 |
| anteaya | the crowd hath spoken | 20:06 |
| cloudnull | ha! | 20:06 |
| clarkb | oh I approved it, I can remove the approval | 20:06 |
| jeblair | i don't know that we should, just saying those are the things to consider | 20:06 |
| fungi | cloudnull: what did the dns solution end up being? are our queries to ipv4 resolver addresses going through a pat? | 20:06 |
| cloudnull | IDK if my cloud will cry, but i have a name to live up to. | 20:06 |
| anteaya | ha ha ha | 20:06 |
| clarkb | fungi: they are NAT'd by the neutron router | 20:07 |
| anteaya | cloudnull: and we will help you get there | 20:07 |
| fungi | k | 20:07 |
| cloudnull | ++ | 20:07 |
| jlvillal | For 'gertty'. When looking at a diff. Is there a search the diff feature? | 20:07 |
| cloudnull | fungi: what clarkb said | 20:07 |
| fungi | jlvillal: ctrl-s | 20:07 |
| pabelanger | cloudnull: we are seeing some failures to launch in osic-cloud1: http://grafana.openstack.org/dashboard/db/nodepool-osic but I was going to wait until we landed dns patch to start looking why | 20:07 |
| mordred | cloudnull: we can always increase the level of pain we inflict on your cloud any time you feel like you need to prove your skills as a leet operator | 20:08 |
| fungi | jlvillal: at least by default, but as with any keybindings in gertty you can set that to something else | 20:08 |
| *** nmagnezi has joined #openstack-infra | 20:08 | |
| * cloudnull enjoys pain | 20:08 | |
| jlvillal | fungi: Thanks. Strange a few moments ago on some diff it was showing it searching for a patch. But now it works. Odd. | 20:08 |
| *** _sarob has joined #openstack-infra | 20:08 | |
| fungi | i expect dns, while needing to get solved, may be fine through pat for now. zuul-launcher ipv6 console streaming support on the other hand could be something we want to solve quickly | 20:08 |
| pabelanger | fungi: 355570 was my attempt at fixing dns | 20:09 |
| mordred | pabelanger: https://review.openstack.org/#/c/355048/ btw | 20:09 |
| fungi | is there a zuul console patch for ipv6 url support? | 20:09 |
| cloudnull | pabelanger: I've been monitoring / watching the logs and such. IDK what is causing the "Error Node Launch Attempts" as neutron || nova aren't stacking or really throwing any errors. | 20:09 |
| cloudnull | but i'm actively trying to hunt things down. | 20:09 |
| jeblair | fungi: https://review.openstack.org/355197 | 20:10 |
| fungi | aha, thanks | 20:10 |
| jlvillal | fungi: That search is a bit odd. It doesn't move the page down if the search result is outside the view. | 20:10 |
| cloudnull | it may simply be an issue with neutron programing th einterface in time. but i've not proven that at this point | 20:10 |
| mordred | cloudnull: oh - also, I don't know if you saw, but one of the things I was considering a problem with ipv6/shade/nodepool on osic is now at least understood ... but i don't think it's generally fixable at the moment | 20:10 |
| fungi | jlvillal: keep hitting ctrl-s to advance | 20:10 |
| jlvillal | fungi: Ah sweet :) Thanks. | 20:11 |
| jeblair | jlvillal: ah, yeah, it doesn't look like it jumps to the initial match if outside the view. however, repeated ctrl-s will get it there | 20:11 |
| *** sarob has quit IRC | 20:11 | |
| cloudnull | mordred: i had not seen that. something we might be able to help out with ? | 20:11 |
| jlvillal | jeblair: Thanks | 20:11 |
| pabelanger | mordred: Great, +1 since I haven't done much shade yet | 20:11 |
| jeblair | probably it should jump to the first one | 20:11 |
| mordred | cloudnull: the basic jist is that the single network with a public ipv6 and a private ipv4 is confusing to shade's concept of inferring what you want to do with your networks ... but it's not preventing us from launching nodes or using them so I'm not going to fix it until we find a way in which it breaks and can imagine a general solution | 20:11 |
| jlvillal | jeblair: I would vote for that behavior :) | 20:11 |
| mordred | cloudnull: I think it's just a deficiency in the neutron data model, and if we try to work around it TOO much in this case I think it'll lead to more not less confusion | 20:12 |
| *** vhosakot has quit IRC | 20:12 | |
| pabelanger | cloudnull: So, It think we are not resolving DNS in our nodepool ready-script, we do host git.openstack.org, and if that fails we delete the server and launch again | 20:12 |
| clarkb | jeblair: mordred is there a reason that that zuul patch hasn't been approved yet? can I go ahead and approve it? | 20:12 |
| *** amitgandhinz has quit IRC | 20:12 | |
| mordred | clarkb: nope. just waiting on a second +2 | 20:12 |
| jeblair | clarkb: no reason i know of. i think pabelanger local-tested it. | 20:12 |
| *** amitgandhinz has joined #openstack-infra | 20:12 | |
| *** _sarob has quit IRC | 20:13 | |
| pabelanger | jeblair: clarkb: Yes, I tested it locally with a simple python app | 20:13 |
| clarkb | jeblair: mordred pabelanger though thinking about it, does that work if for some reaosn a host doesn't have a working ipv6 stack? do we care about such hosts? | 20:13 |
| cloudnull | mordred: :'( at least things are still working | 20:13 |
| mordred | clarkb: I do not personally care about such hosts at the moment | 20:13 |
| fungi | it does of course mean that people without ipv6 connectivity can't get to some of the log streams, but... join us in the new era. hurricane electric tunnels for everyone! | 20:13 |
| jeblair | clarkb: fungi assured us that should be fine for any linux post 1997 or something. | 20:13 |
| *** rbuzatu has joined #openstack-infra | 20:14 | |
| pabelanger | fungi: Yes! my lack of ipv6 at home is becoming a problem now | 20:14 |
| clarkb | fungi: thats me now that I changed ISPs | 20:14 |
| clarkb | I should fix that | 20:14 |
| clarkb | pabelanger: one trick is to ssh tunnel | 20:14 |
| * sc68cal wishes FiOS would get their shit together | 20:14 | |
| mordred | sc68cal: ++ | 20:14 |
| clarkb | you can v6 to v4 or v4 to v6 pretty easily with ssh | 20:14 |
| clarkb | sc68cal: ya thats who I changed to | 20:14 |
| mordred | sc68cal: oh - that reminds me - I need to call frontier to see if their Gig service is available for me | 20:14 |
| pabelanger | clarkb: cool, I haven't looked how to yet | 20:15 |
| jeblair | clarkb: that is, it should listen on v4 and v6 for dual stack hosts, which is all of them. of course some of our nodes now are not *routable* over v4. | 20:15 |
| sc68cal | mordred: lol humblebrag | 20:15 |
| jeblair | fungi: right^ | 20:15 |
| clarkb | jeblair: yup | 20:15 |
| *** sarob has joined #openstack-infra | 20:15 | |
| *** Goneri has quit IRC | 20:15 | |
| clarkb | jeblair: and even if you don't have a global ipv6 addr you should have a link local addr and loopback to listen on for v6 | 20:15 |
| anteaya | sc68cal: it is nice to see you, have a frowny face | 20:15 |
| * sc68cal thinks he needs a REST API to POST things he needs downloaded, and ship hard drives to mordred :) | 20:15 | |
| fungi | clarkb: pabelanger: my home ipv6 is via an he tunnel from my firewall. even have a /48 and reverse dns delegated for it | 20:15 |
| clarkb | jeblair: so not a problem on the bind side I don't think unless running ancient linux as fungi said | 20:15 |
| clarkb | fungi: ya I just know that after having native v6 with comcast very little stuff functions properly with it. Thought that may be related to the giant bitbuckets in seattel and denver in comcast land and HE is happier | 20:16 |
| clarkb | I have approved the zuul change | 20:16 |
| clarkb | fungi: I have had to disable v6 in order to get working internets more than once | 20:17 |
| *** Apoorva has joined #openstack-infra | 20:17 | |
| fungi | clarkb: the ancient behavior isn't so much lack of linklocal addressing, as older system-wide "v6only" socket behavior (which you can still set via sysctl or explicit socketopts) | 20:17 |
| openstackgerrit | Merged openstack-infra/nodepool: Make ZK fixture more robust https://review.openstack.org/355131 | 20:17 |
| mordred | \o/ | 20:18 |
| clarkb | fungi: I want to say ubuntu of the 2005 ish era didn't have v6 enabled at all? but thats ancient so meh | 20:18 |
| fungi | basically, binding a socket on :: used to only listen on all ipv6 addresses, not any ipv4 addresses | 20:18 |
| mordred | robust test fixtures are great | 20:18 |
| *** rbuzatu has quit IRC | 20:18 | |
| jeblair | i'm a fan | 20:19 |
| jeblair | mordred, pabelanger, Shrews: i'll work on getting ansible manually installed on launchers | 20:19 |
| jeblair | see if i can find my old playbooks for that | 20:19 |
| *** javeriak has quit IRC | 20:19 | |
| *** inc0 has quit IRC | 20:19 | |
| Shrews | alrighty then | 20:20 |
| mordred | jeblair: cool | 20:20 |
| mordred | jeblair, Shrews: next time you're bored: https://review.openstack.org/#/c/355048/ ... I added tests and a release note even | 20:20 |
| Shrews | mordred: i could of swore i reviewed that already. perhaps i forgot to vote | 20:21 |
| fungi | clarkb: controllable through the IPV6_V6ONLY sockopt (since Linux 2.4.21 and 2.6) and /proc/sys/net/ipv6/bindv6only system default | 20:21 |
| fungi | jeblair: ^ | 20:21 |
| *** valderrv_ has quit IRC | 20:21 | |
| *** karthik__ has quit IRC | 20:21 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test https://review.openstack.org/346949 | 20:21 |
| fungi | or net.ipv6.bindv6only via sysctl | 20:22 |
| openstackgerrit | Merged openstack-infra/zuul: Simplify zuul_console port binding logic https://review.openstack.org/355197 | 20:22 |
| fungi | it was somewhat hotly debated on debian-devel ~7 years ago https://lists.debian.org/debian-devel/2009/10/msg00542.html | 20:23 |
| fungi | so that's what i mean by "relatively modern" | 20:23 |
| mordred | fungi: so as a cya, we could set net.ipv6.bindv6only to false with sysctl | 20:23 |
| mordred | maybe in the zuul puppet | 20:23 |
| fungi | i think we add that if someone complains that their 7-year-old server isn't running our experimental zuul-launcher correctly? | 20:24 |
| fungi | the one we haven't documented much nor encouraged others to switch to? | 20:24 |
| jeblair | it's on the test nodes too | 20:24 |
| jeblair | so 'testing on a 7 year old platform' | 20:24 |
| fungi | ahh, yeah. i'll check centos 7 | 20:24 |
| pabelanger | mordred: fungi: if you have time today, would not object to a review of 354818. Start mirroring source packages for debian / ubuntu for zigo | 20:24 |
| * jeblair hopes it's not called centos '7' because it's 7 years old | 20:25 | |
| fungi | net.ipv6.bindv6only = 0 already on centos 7 | 20:25 |
| sean-k-mooney | mmedvede: are you about? | 20:25 |
| fungi | good thing we're not still on centos 10! | 20:25 |
| jeblair | if so, i'm not sure about 'upgrading' | 20:25 |
| fungi | also net.ipv6.bindv6only = 0 on ubuntu precise | 20:26 |
| *** valderrv has joined #openstack-infra | 20:26 | |
| fungi | so we should be fine | 20:26 |
| mmedvede | sean-k-mooney: I am here | 20:26 |
| clarkb | fungi: ianw pabelanger ok I think I have caught up on the ntp meeting discussion. From my reading of that and ubuntu docs I think we might be ok to completely drop ntp packages and services from our test images | 20:27 |
| sean-k-mooney | mmedvede: i tried to set up my own instance of ciwatch but the ci_id are always null so it does not render correctly | 20:27 |
| *** kgiusti has left #openstack-infra | 20:27 | |
| sean-k-mooney | mmedvede: is the most uptoday code in the gitub? | 20:27 |
| clarkb | fungi: ianw pabelanger we just have to make sure that the distro defaults of chrony and timedatectl end up in place | 20:27 |
| *** tonytan4ever has quit IRC | 20:27 | |
| clarkb | fungi: ianw pabelanger I can try booting some ubuntu-minimal and fedora-minimal images once I am otherwise caught up on post vacation things to see if those just work | 20:28 |
| cloudnull | also, just a shout out: thanks everyone for helping the OSIC get to gating on IPv6! its really quite awesome to see all of this getting done and rolling into production. | 20:28 |
| cloudnull | at the next ops-meetup/summit: beers on me :) | 20:28 |
| clarkb | cloudnull: its pretty neat on our end too (we have long said ipv6 should mostly work and it looks like it does \o/) | 20:28 |
| clarkb | cloudnull: thank you ! | 20:28 |
| *** asselin__ has quit IRC | 20:28 | |
| *** asselin has joined #openstack-infra | 20:29 | |
| *** xyang1 has quit IRC | 20:29 | |
| mmedvede | sean-k-mooney: yes. I have a script I can share that should setup ciwatch for you (using puppet-ciwatch module) | 20:29 |
| *** xyang1 has joined #openstack-infra | 20:29 | |
| sean-k-mooney | mmedvede: well i have it running in a docker container https://ciwatch.seanmooney.info/project?project=neutron&time=7+days | 20:29 |
| sean-k-mooney | but it looks like i missed something | 20:30 |
| *** devkulkarni has quit IRC | 20:31 | |
| clarkb | oh heh it looks like timedatectl may be a systemd realted thing that configures chronyd? | 20:31 |
| *** ociuhandu has quit IRC | 20:31 | |
| clarkb | this isn't convoluted and confusing at all | 20:31 |
| clarkb | and may not be part of precise but is available on trusty looks like | 20:31 |
| *** sdake has quit IRC | 20:32 | |
| *** asselin_ has joined #openstack-infra | 20:32 | |
| mtreinish | clarkb: yeah that's a systemd thing | 20:32 |
| sean-k-mooney | mmedvede: if you can point me towrad the script though i would be happy to compare and see what i missed | 20:32 |
| mtreinish | clarkb: or at least I think it is, because that's what I've had to use for time settings on my arch boxes for a while | 20:33 |
| clarkb | mtreinish: on ubuntu the systemd/systemd-services packages provide it | 20:33 |
| *** _nadya_ has quit IRC | 20:33 | |
| jeblair | #status log Installed ansible stable-2.1 branch on zuul launchers to pick up https://github.com/ansible/ansible/commit/d35377dac78a8fcc6e8acf0ffd92f47f44d70946 | 20:34 |
| openstackstatus | jeblair: finished logging | 20:34 |
| *** asselin has quit IRC | 20:35 | |
| mmedvede | sean-k-mooney: it is pretty much just using puppet module to deploy it http://paste.openstack.org/show/557627/ | 20:36 |
| *** nmagnezi has quit IRC | 20:37 | |
| sean-k-mooney | mmedvede: thanks the only real difference i can see between the puppet deployment and my manually deployment is i used the default sqlite conenction string instead of useing mysql | 20:38 |
| sean-k-mooney | mmedvede: ill give the puppet aproch a shot though and see if that works form me. thanks for the help | 20:39 |
| pabelanger | eep, 200 nodes just got deleted by nodepool | 20:39 |
| pabelanger | checking why now | 20:39 |
| mmedvede | sean-k-mooney: ok. I'll try deploying from scratch myself when I get some free time. I'll let you know if I see the same problem you are seeing | 20:40 |
| fungi | pabelanger: gate reset? | 20:40 |
| pabelanger | fungi: I think because ansible was reinstalled | 20:41 |
| pabelanger | http://logs.openstack.org/88/329788/7/check/gate-tempest-dsvm-full-ceph-plugin-src-glance_store/a5b3c07/_zuul_ansible/ansible_log.txt is a new failure | 20:41 |
| pabelanger | lets see if it happens again | 20:41 |
| fungi | looks like devstack-gate cells jobs are probably hitting the same problem rajinir was seeing in a third-party ci | 20:41 |
| jeblair | pabelanger: yes, i just reinstalled ansible | 20:42 |
| *** esikachev has quit IRC | 20:43 | |
| clarkb | fungi: yup they pushed a fix | 20:43 |
| jeblair | pabelanger: probably should have incorporated it into a graceful shutdown/reinstall/start playbook | 20:43 |
| pabelanger | jeblair: Ya, failures line up with that. replacement nodes back online | 20:43 |
| fungi | k | 20:43 |
| pabelanger | jeblair: np | 20:43 |
| pabelanger | jeblair: I think you said you have a potential fix for inplace upgrades for ansible a while back? | 20:44 |
| jeblair | pabelanger: in-place upgrades of zuul, and that's there | 20:44 |
| *** adrian_otto has joined #openstack-infra | 20:44 | |
| jeblair | pabelanger: not ansible though. we need to stop/upgrade/start for ansible | 20:44 |
| pabelanger | okay | 20:44 |
| jeblair | but that doesn't happen often | 20:44 |
| jeblair | i hope | 20:44 |
| pabelanger | ya | 20:44 |
| *** tonytan4ever has joined #openstack-infra | 20:45 | |
| pabelanger | Starting to see traffic on the new mirror.ord.rax.openstack.org server | 20:47 |
| jeblair | on a (perhaps related) note, i enqueued 355628 into the gate | 20:47 |
| pabelanger | cacti.o.o confirms too | 20:47 |
| clarkb | fungi: ianw pabelanger looks like newer ubuntu may run https://www.freedesktop.org/software/systemd/man/systemd-timesyncd.service.html by default | 20:48 |
| clarkb | unfortuantelky the docs for that don't say anything about how it handles skew | 20:48 |
| anteaya | jeblair: so DuncanT should be able to recheck that patch? | 20:48 |
| anteaya | and beagles too? | 20:48 |
| jeblair | anteaya: yep | 20:49 |
| anteaya | thank you | 20:49 |
| anteaya | thank you mordred | 20:49 |
| jeblair | #status log gracefully restarting all zuul-launchers | 20:49 |
| openstackstatus | jeblair: finished logging | 20:50 |
| openstackgerrit | Merged openstack-infra/project-config: Raised max instance in the OSIC https://review.openstack.org/355628 | 20:50 |
| cloudnull | woot! | 20:50 |
| *** gouthamr has quit IRC | 20:51 | |
| jeblair | in a few hours, we should have v6 telnet links working there | 20:51 |
| clarkb | jeblair: does that depend on new images in osic? | 20:51 |
| clarkb | I can babysit that if you think it will help | 20:51 |
| jeblair | clarkb: no, it's zuul-console component copied over by ansible from zuul-launcher | 20:51 |
| clarkb | ah | 20:52 |
| jeblair | clarkb: the few hours is the zuul-launcher global graceful restart i just kicked off | 20:52 |
| clarkb | kk | 20:52 |
| pabelanger | fungi: already up to 140 Mbps http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=184 | 20:52 |
| jeblair | (we *can* hard-restart the launchers, but it would burn more nodes) | 20:52 |
| fungi | pabelanger: that's a great indication of how terrible things were before, if we were wanting 40% more than our bw cap there | 20:53 |
| pabelanger | fungi: indeed cc sdague ^ | 20:53 |
| fungi | we probably need to keep an eye on it and maybe replace it again with an even bigger flavor if we get closer to 200mbps | 20:53 |
| pabelanger | ya | 20:54 |
| mordred | ++ | 20:54 |
| pabelanger | or setup load balanceers | 20:54 |
| *** raildo has quit IRC | 20:54 | |
| mordred | me | 20:54 |
| mordred | meh | 20:54 |
| mordred | bigger vms | 20:54 |
| mordred | more power | 20:54 |
| mordred | mmmm | 20:54 |
| clarkb | load balancers have a similar problem | 20:54 |
| clarkb | since they are restricted to the same bw constraints | 20:54 |
| sdague | mordred: max powers! | 20:54 |
| pabelanger | clarkb: that is true | 20:54 |
| clarkb | so in this case its simpler to just go bigger | 20:54 |
| mordred | sdague: so much powers | 20:54 |
| pabelanger | go big or go home | 20:55 |
| sdague | https://www.youtube.com/watch?v=7P0JM3h7IQk | 20:55 |
| sdague | simpsons ^^^ | 20:55 |
| fungi | one of my favorite episodes | 20:56 |
| fungi | i've thought from time to time it would have made an amusing online handle/pseudonym | 20:57 |
| mordred | sdague: so - if I could bother you for a sec ... http://logs.openstack.org/94/352594/1/experimental/gate-grenade-dsvm-neutron-libs-ubuntu-xenial-nv/0c53e25/ - I added an experimental grenade job to neutronclient so that we can show that the combo of the latest os-client-config and the patch I wrote appropriately works | 20:57 |
| *** sdake has joined #openstack-infra | 20:57 | |
| fungi | clarkb: if only rackspace had a network-heavy flavor. we don't need more ram/cpu/disk but we end up with it anyway to get more bandwidth | 20:58 |
| mordred | sdague: BUT - it has the sads | 20:58 |
| sdague | mordred: ok, you have about 3 minutes to explain the sads before I call it a day. | 20:58 |
| sdague | but you should do that, because I'll look first thing in the morning | 20:58 |
| anteaya | sdague: thank you | 20:58 |
| mordred | sdague: it complains about xenial | 20:58 |
| mordred | sdague: which makes me think job config issue | 20:58 |
| mordred | sdague: but I thought I copied all of the goo from other people | 20:59 |
| sdague | right, grenade doesn't run on xenial | 20:59 |
| sdague | because stack.sh comes from mitaka | 20:59 |
| sdague | which shipped before xenial | 20:59 |
| *** dprince has quit IRC | 20:59 | |
| sdague | and we typically don't backport that support change | 21:00 |
| mordred | hrm. ok, then I think my original version of that patch was potentially more correcter | 21:00 |
| *** jkilpatr has quit IRC | 21:00 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Bump tempest version to latest https://review.openstack.org/355641 | 21:00 |
| sdague | so... you should probably just move this to run on trusty | 21:00 |
| sdague | we talked about just doing the backport, but clarkb didn't think it was needed when he was rolling jobs over | 21:01 |
| clarkb | right we decided to run mitaka to newton/master on trusty | 21:01 |
| * mordred grumps | 21:01 | |
| mordred | https://review.openstack.org/#/c/354664/2/jenkins/jobs/projects.yaml,unified | 21:01 |
| mordred | sdague: cool. thnaks. super helpful | 21:02 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Bump tempest version to latest https://review.openstack.org/355641 | 21:02 |
| sdague | mordred: ok, great. If you need other things, feel free to send an email. Heading out for the day. | 21:02 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE TESTING https://review.openstack.org/316436 | 21:02 |
| *** mhickey has quit IRC | 21:03 | |
| fungi | i think grenade pull-up jobs for newton make more sense on trusty since mitaka was only tested on trusty and newton was tested on trusty for ~half of its development | 21:03 |
| *** hrubi has joined #openstack-infra | 21:03 | |
| clarkb | yup. Thought I also think that grenade should not be so forceful about what platform I run it on | 21:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Run neutronclient experimental grenade job on trusty https://review.openstack.org/355642 | 21:03 |
| clarkb | if I want to run it on tumbleweed please let me ... | 21:03 |
| *** julim has quit IRC | 21:03 | |
| mordred | clarkb, anteaya: ^^ per the conversation just now | 21:04 |
| anteaya | mordred: okey dokey | 21:05 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for cells v2 setup bug 1613417 https://review.openstack.org/355619 | 21:06 |
| openstack | bug 1613417 in devstack "gate-tempest-dsvm-cells broken with cell v2 setup: "No hosts found to map to cell, exiting."" [Undecided,In progress] https://launchpad.net/bugs/1613417 | 21:06 |
| *** spzala has quit IRC | 21:07 | |
| *** spzala has joined #openstack-infra | 21:07 | |
| *** ihrachys has quit IRC | 21:08 | |
| pabelanger | fungi: big spike now, 175Mbps | 21:08 |
| pabelanger | how high will it go | 21:08 |
| pabelanger | nobody knows | 21:08 |
| *** sdake has quit IRC | 21:08 | |
| mordred | wow. we produce some traffic! | 21:09 |
| fungi | and that's just local mirror access for rax-ord | 21:09 |
| fungi | makes me wonder if we have very stale caches on images there | 21:09 |
| *** yamamoto has joined #openstack-infra | 21:10 | |
| fungi | or whether that's something other than distro packages | 21:10 |
| *** yamamoto has quit IRC | 21:11 | |
| fungi | le sigh. i'm getting "yaml.reader.ReaderError: unacceptable character #x009b: special characters are not allowed" trying to parse openstack/governance:reference/projects.yaml | 21:11 |
| pabelanger | fungi: so, I created a new volume for the new server since I didn't want to break jobs running. So, cache does need to warm up there | 21:11 |
| anteaya | fungi: :( | 21:11 |
| fungi | pabelanger: yep, but that would show up as ingress not egress | 21:12 |
| jeblair | we just had a logistical discussion in #zuul which ended up with the idea that we should make a feature branch for nodepool for the zk work. even though we want to land that and start using it soon, it will take multiple changes to implement, and coexistence with the current builder design is difficult. | 21:12 |
| pabelanger | fungi: Yes, that is true | 21:12 |
| fungi | jeblair: seems reasonable to me | 21:12 |
| jeblair | clarkb: i guess we should have gone with your pick for server size. :) | 21:12 |
| pabelanger | cloudnull: your patch just went live | 21:12 |
| *** adrian_otto has quit IRC | 21:12 | |
| pabelanger | incoming 250 nodes on osic-cloud1 | 21:12 |
| mordred | fungi: how ar eyou reading it? | 21:13 |
| cloudnull | wooo! | 21:13 |
| * cloudnull goes home for the day | 21:13 | |
| cloudnull | :) | 21:13 |
| fungi | mordred: yaml.safe_load(requests.get(PROJECTS_LIST % ref).text) | 21:13 |
| jeblair | cloudnull: oh wait, you almost forgot your pager! | 21:13 |
| * cloudnull runs | 21:13 | |
| *** spzala has quit IRC | 21:13 | |
| mordred | fungi: ah - a=yaml.safe_load(open('reference/projects.yaml', 'r').read()) works for me | 21:13 |
| mordred | I will try your version | 21:13 |
| *** rhallisey has quit IRC | 21:14 | |
| mordred | fungi: you don't have an expansion of PROJECTS_LIST % ref handy do you? | 21:14 |
| fungi | mordred: i think it may be with how/where i'm retrieving it from. digging deeper | 21:14 |
| fungi | mordred: https://review.openstack.org/gitweb?p=openstack/governance.git;a=blob_plain;f=reference/projects.yaml;hb=master | 21:15 |
| mordred | ah | 21:15 |
| mordred | a=yaml.safe_load(requests.get('http://git.openstack.org/cgit/openstack/governance/plain/reference/projects.yaml').text) works for me | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| *** rbuzatu has joined #openstack-infra | 21:15 | |
| *** tqtran has quit IRC | 21:15 | |
| mordred | fungi: I can re-create your error with the review.o.o url | 21:16 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Fix template filename https://review.openstack.org/355626 | 21:16 |
| fungi | mordred: yep. i'm finding what position 25352 is next | 21:16 |
| mordred | fungi: fun! | 21:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add tripleo-test-clouds AFS mirrors to cacti.o.o https://review.openstack.org/355644 | 21:16 |
| mordred | fungi: I blame jgit | 21:16 |
| *** sdake has joined #openstack-infra | 21:17 | |
| * Shrews blames j<anything> | 21:17 | |
| *** _ari_ has joined #openstack-infra | 21:17 | |
| *** fitoduarte has joined #openstack-infra | 21:17 | |
| *** adduarte has joined #openstack-infra | 21:17 | |
| *** _ari_ has quit IRC | 21:17 | |
| fungi | u' - name: Zbyn\xc4\x9bk Schwarz\n' | 21:18 |
| *** adduarte has quit IRC | 21:18 | |
| mordred | fungi: from git.o.o I get: u' name: Zbyn\u011bk Schwarz\n ' right around there | 21:18 |
| mordred | fungi: I wonder if you need a header for the requests.get to set a language or something | 21:19 |
| mordred | or encoding I mean | 21:19 |
| fungi | mordred: possibly | 21:19 |
| clarkb | yaml is utf8 by default iirc | 21:20 |
| clarkb | so if you are somehow getting the bits in not utf8 that may make it mad | 21:21 |
| mordred | yah - but gitweb might be encoding over the wire | 21:21 |
| mordred | or decoding | 21:21 |
| mordred | or something | 21:21 |
| *** elo has joined #openstack-infra | 21:21 | |
| *** rbuzatu has quit IRC | 21:21 | |
| mordred | yup | 21:22 |
| mordred | my browser tells me that that link is being served as "Western (Windows-1252)" | 21:22 |
| fungi | mordred: requests.get(blah).encoding indeed says 'ISO-8859-1' | 21:23 |
| fungi | looks like it might be a gitweb fallback behavior | 21:24 |
| pabelanger | okay, stepping away to run some family errands. I think our original mirror.ord.rax.openstack.org can be deleted now. Last hit to apache logs in 15/Aug/2016:20:47:05 +0000. I'll do that when I get back this evening just to be safe | 21:25 |
| *** matt-borland has quit IRC | 21:25 | |
| clarkb | pabelanger: thanks again | 21:25 |
| clarkb | fungi: mordred fallback for when you don't set an accepts encoding? | 21:26 |
| clarkb | ok ntp is making me go blind | 21:26 |
| clarkb | ianw: pabelanger: any other feedback on not using our ntp mdoule on the test images at all? | 21:26 |
| *** adrian_otto has joined #openstack-infra | 21:27 | |
| fungi | clarkb: likely. i'm just reading through requests docs now | 21:28 |
| *** edtubill has quit IRC | 21:28 | |
| *** jkilpatr has joined #openstack-infra | 21:28 | |
| *** apetrich has quit IRC | 21:29 | |
| mordred | fungi: ok. SO ... | 21:30 |
| mordred | response = requests.get('https://review.openstack.org/gitweb?p=openstack/governance.git;a=blob_plain;f=reference/projects.yaml;hb=master') | 21:31 |
| mordred | response.encoding = 'utf-8' | 21:31 |
| mordred | a=yaml.safe_load(response.text) | 21:31 |
| mordred | fungi: that ^^ works | 21:31 |
| fungi | mordred: yep, found that gem | 21:31 |
| mordred | fungi: so I think what may be happening is that gitweb is returning utf8 data but setting the header wrong | 21:31 |
| fungi | so requests is assuming the response is in latin1 when it's actually utf8 all along | 21:31 |
| mordred | yah | 21:31 |
| mordred | yah: 'Content-Type': 'text/plain; charset=ISO-8859-1' | 21:32 |
| mordred | that's in the respnse headers from gitweb | 21:32 |
| fungi | .headers does indeed sat that | 21:32 |
| fungi | er, say | 21:32 |
| fungi | that's where i just went as well | 21:32 |
| mordred | I think we can consider that to be independently verified results then! :) | 21:32 |
| fungi | mordred: i think it's actually not setting an encoding, which rfc 2616 says means latin1 | 21:34 |
| mordred | fungi: lovely | 21:34 |
| *** adrian_otto has quit IRC | 21:34 | |
| fungi | i'd need to use a packet sniffer to confirm whether requests is faking that in the headers dict, or apache is actually passing it | 21:35 |
| *** jheroux has quit IRC | 21:35 | |
| *** jcoufal has quit IRC | 21:35 | |
| fungi | could be we need apache on review.o.o configured differently | 21:35 |
| fungi | there's "AddDefaultCharset UTF-8" as one possibility | 21:36 |
| *** yamahata has quit IRC | 21:38 | |
| *** sdake has quit IRC | 21:39 | |
| karthikp_ | clarkb: Hi | 21:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Disable ntp services on single use test instances https://review.openstack.org/355651 | 21:40 |
| clarkb | fungi: ianw pabelanger ^ I am going to WIP that until I can do more testing of the distro boot time defaults to make sure tehy do set something sane (manpages for ubuntu claim they do and I think its all systemd related so the other distros should too) | 21:40 |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config: Rename bifrost integration test job https://review.openstack.org/355652 | 21:41 |
| clarkb | karthikp_: hello | 21:41 |
| *** edtubill has joined #openstack-infra | 21:41 | |
| karthikp_ | clarkb: got a question for you regarding grenade.... any idea why this step is necessary? | 21:42 |
| karthikp_ | https://github.com/openstack-dev/grenade/blob/2a213d4e644f939e26bae82d11b8b4961e7ab65b/projects/70_cinder/upgrade.sh#L67 | 21:42 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Fix template links for split uncategorized https://review.openstack.org/354491 | 21:43 |
| clarkb | karthikp_: I don't know for sure but guessing that if the services really fail to start then the lgos won't exist | 21:43 |
| clarkb | karthikp_: you might have better luck checking the git logs for that line | 21:43 |
| anteaya | TheJulia: does bifrost have more than one job for ipa? | 21:46 |
| anteaya | TheJulia: if not, how about removing the adjective and going with ipa | 21:46 |
| TheJulia | anteaya: two, we build IPA with debian as well | 21:46 |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects https://review.openstack.org/347047 | 21:46 |
| anteaya | just trying to see if we can avoid needing to rename the job when you change the image | 21:47 |
| fungi | mordred: i've investigated some apache-side workarounds, but on deeper investigation it seems that gitweb has a history of returning incorrect content types, including for blob_plain http://git.661346.n2.nabble.com/PATCH-1-4-gitweb-Fix-utf8-encoding-for-blob-plain-blobdiff-plain-commitdiff-plain-and-patch-td7582051.html | 21:47 |
| TheJulia | anteaya: renaming the job was like the last thing I wanted to do though :\ | 21:47 |
| fungi | mordred: so i'll just set the encoding in our script as a workaround | 21:47 |
| anteaya | TheJulia: yeah, how about ipa-cirros | 21:47 |
| anteaya | that is different from ipa-debian, yeah? | 21:48 |
| TheJulia | not as descriptive, yeah, it also fires up debian in that job, so in theory that still works | 21:48 |
| *** amotoki has quit IRC | 21:48 | |
| * TheJulia likes it | 21:48 | |
| anteaya | well the description should be in the log, right? | 21:48 |
| anteaya | yay you like it | 21:48 |
| anteaya | thanks | 21:48 |
| *** burgerk has quit IRC | 21:48 | |
| TheJulia | :) I'll update it in a little bit | 21:49 |
| anteaya | yup, thanks | 21:49 |
| anteaya | going offline soon, I'll look tomorrow | 21:49 |
| *** adrian_otto has joined #openstack-infra | 21:51 | |
| clarkb | ipa works on cirros? | 21:52 |
| * clarkb wonders if thats another image in the ironic ramdisk image list | 21:52 | |
| karthikp_ | clarkb: i see .. git logs? | 21:52 |
| clarkb | karthikp_: the revision control history for that repo may tell you why that line was added | 21:52 |
| *** devkulkarni has joined #openstack-infra | 21:52 | |
| *** sdake has joined #openstack-infra | 21:54 | |
| anteaya | clarkb: https://review.openstack.org/#/c/355652/1 | 21:54 |
| anteaya | I have to assume the answer to your question is yes, based on the existing name of the job | 21:55 |
| *** tqtran has joined #openstack-infra | 21:55 | |
| *** harlowja has quit IRC | 21:55 | |
| TheJulia | clarkb: more like we deploy cirros as fast lightweight reliable test | 21:55 |
| anteaya | I'm that patch is all I am using for my assertion | 21:55 |
| anteaya | s/I'm/but | 21:56 |
| anteaya | wow | 21:56 |
| clarkb | ah you boot cirros using tinycore ramdisk | 21:56 |
| clarkb | gotcha | 21:56 |
| karthikp_ | clarkb: Oh ya that was added for all the projects by sdague..i iwlll chekc with him | 21:56 |
| karthikp_ | clarkb: thanks | 21:56 |
| anteaya | JayF: is thinking like me | 21:56 |
| *** tonytan4ever has quit IRC | 21:57 | |
| * TheJulia lets there be a little chatter and goes to start cooking dinner :) | 21:57 | |
| JayF | anteaya: that's the nicest thing you've ever said to me \o/ :) | 21:57 |
| JayF | My thought was just, if I were graphing this job, I'd wanna see how the default changed it w/o having to change the name | 21:58 |
| JayF | if you have >1 of something, sure, specify, but maybe leave it out if it's only 1 | 21:58 |
| *** tkelsey has joined #openstack-infra | 21:58 | |
| anteaya | JayF: ha ha ha :) | 21:59 |
| anteaya | JayF: I agree with you thinking | 21:59 |
| anteaya | your* | 21:59 |
| anteaya | I'm so glad I could math in school, the spelling gods never looked my way | 22:00 |
| *** amotoki has joined #openstack-infra | 22:00 | |
| *** matrohon has quit IRC | 22:00 | |
| *** tqtran has quit IRC | 22:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Run neutronclient experimental grenade job on trusty https://review.openstack.org/355642 | 22:00 |
| *** gordc has quit IRC | 22:00 | |
| *** amotoki has quit IRC | 22:00 | |
| *** yamahata has joined #openstack-infra | 22:01 | |
| *** xarses has joined #openstack-infra | 22:01 | |
| anteaya | <-- offline | 22:02 |
| *** camunoz has quit IRC | 22:02 | |
| *** mtanino has quit IRC | 22:03 | |
| *** onovy has quit IRC | 22:03 | |
| *** tkelsey has quit IRC | 22:03 | |
| openstackgerrit | Merged openstack-infra/nodepool: Shut down gearman client in tests https://review.openstack.org/355109 | 22:04 |
| openstackgerrit | Merged openstack-infra/nodepool: Remove testresources https://review.openstack.org/354441 | 22:04 |
| *** peterlisak has quit IRC | 22:05 | |
| *** esberglu has quit IRC | 22:06 | |
| *** thorst_ has quit IRC | 22:08 | |
| *** tqtran has joined #openstack-infra | 22:08 | |
| mmedvede | sean-k-mooney: around? I tested a fresh install of ciwatch, it works fine | 22:08 |
| *** mdrabe has quit IRC | 22:09 | |
| *** mriedem has quit IRC | 22:09 | |
| * clarkb is working on building dib -minimal images without an explicit ntp install to see what we end up with | 22:09 | |
| clarkb | ianw: pabelanger ^ hopefully that shows us we can get away with just not doing stuff on the sinlge use nodes | 22:09 |
| *** valderrv has quit IRC | 22:10 | |
| *** edtubill has quit IRC | 22:10 | |
| *** beagles is now known as beagles_brb | 22:11 | |
| *** thorst_ has joined #openstack-infra | 22:13 | |
| *** tqtran has quit IRC | 22:15 | |
| *** bswartz has quit IRC | 22:15 | |
| *** tqtran has joined #openstack-infra | 22:16 | |
| scottda | yolanda: Would you re-approve https://review.openstack.org/#/c/330678/ when you have a chance? The dependent patch has merged an it needed a rebase. | 22:16 |
| *** thorst_ has quit IRC | 22:18 | |
| *** jistr has quit IRC | 22:18 | |
| *** peterlisak has joined #openstack-infra | 22:19 | |
| *** jistr has joined #openstack-infra | 22:19 | |
| *** onovy has joined #openstack-infra | 22:19 | |
| *** sdake has quit IRC | 22:20 | |
| *** vhosakot has joined #openstack-infra | 22:25 | |
| *** netsin has quit IRC | 22:25 | |
| *** yamamoto has joined #openstack-infra | 22:26 | |
| *** hockeynut has joined #openstack-infra | 22:29 | |
| *** jkilpatr has quit IRC | 22:29 | |
| *** weshay has quit IRC | 22:32 | |
| *** nwkarsten has quit IRC | 22:32 | |
| *** nwkarsten has joined #openstack-infra | 22:32 | |
| *** signed8bit is now known as signed8bit_Zzz | 22:34 | |
| *** fguillot_ has joined #openstack-infra | 22:34 | |
| *** sdake has joined #openstack-infra | 22:35 | |
| *** nwkarsten has quit IRC | 22:37 | |
| *** krtaylor has quit IRC | 22:38 | |
| *** rbuzatu has joined #openstack-infra | 22:38 | |
| *** netsin has joined #openstack-infra | 22:38 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/project-config: Add new/update existing projects https://review.openstack.org/347047 | 22:38 |
| pabelanger | clarkb: ack | 22:39 |
| pabelanger | clarkb: I haven't really been following the ntp issues from today. Will try and catch up on backscroll here in a bit | 22:40 |
| clarkb | pabelanger: tl;dr is after seeing the meeting notes from last week I saw you all mentioned just using the defaults on the distros. and on further investigation I think at least for systemd distros it may just work if we stop explicuitly installing ntp | 22:40 |
| clarkb | pabelanger: so building images locally to test that theory | 22:41 |
| pabelanger | clarkb: Ah, yes. I remember that | 22:41 |
| clarkb | pabelanger: since systemd has some built in time syncing stuff that should update the time on boot if I am reading things correctly | 22:42 |
| clarkb | but want to test that first | 22:42 |
| clarkb | and figure out what trusty and precise do | 22:42 |
| *** beagles_brb is now known as beagles | 22:42 | |
| JayF | timesyncd is pretty nuts though. it just does a tls connection to something and steals the timestamp iirc | 22:42 |
| JayF | like if that's good enough, it's good enough, just a strange way of doing things | 22:43 |
| JayF | ah it apparently talks to real ntp servers now, that's an improvement | 22:43 |
| pabelanger | #status log mirror.ord.rax.openstack.org upgraded to performance1-4 to address network bandwidth cap. | 22:43 |
| pabelanger | and original server now deleted | 22:44 |
| *** weshay has joined #openstack-infra | 22:44 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow https://review.openstack.org/355666 | 22:44 |
| *** rbuzatu has quit IRC | 22:44 | |
| *** pabelanger has quit IRC | 22:45 | |
| *** pabelanger has joined #openstack-infra | 22:45 | |
| pabelanger | #status log mirror.ord.rax.openstack.org upgraded to performance1-4 to address network bandwidth cap. | 22:45 |
| openstackstatus | pabelanger: finished logging | 22:45 |
| *** signed8bit_Zzz is now known as signed8bit | 22:46 | |
| *** fguillot_ has quit IRC | 22:47 | |
| clarkb | JayF: ya for our long lived servers we will probably continue to ntp or similar | 22:47 |
| clarkb | JayF: but on the test instances we really just need a mostly correct timestamps in logs that won't jump halfway through a job | 22:48 |
| pabelanger | okay, just starting to look into osic-cloud1 lauch node errors, first issue: http://paste.openstack.org/show/557761/ | 22:49 |
| clarkb | pabelanger: that needs to use iptables6 I think | 22:50 |
| * mordred lookie | 22:50 | |
| pabelanger | I think so too | 22:50 |
| pabelanger | Oh, can we land https://review.openstack.org/#/c/355047/ | 22:50 |
| pabelanger | help reduce debug logs in nodepool | 22:51 |
| pabelanger | when we cannot host git.o.o | 22:51 |
| mordred | oh piddle | 22:51 |
| *** edmondsw has quit IRC | 22:51 | |
| cloudnull | pabelanger: anything you need from me ? | 22:51 |
| cloudnull | or any way I can help ? | 22:51 |
| mordred | the 'bug' in shade (it currently doesn't do enough magic WRT IPv4/IPv6 addresses) _may_ bite us with multi-node | 22:52 |
| pabelanger | cloudnull: I don't think so. We just need to update some nodepool scripts I think | 22:52 |
| * mordred goes to look through nodepool real quick | 22:52 | |
| pabelanger | mordred: oh? | 22:52 |
| *** vhosakot has quit IRC | 22:53 | |
| mordred | yeah. blast | 22:53 |
| mordred | that means I _am_ going to have to fix that | 22:53 |
| * mordred cries | 22:53 | |
| mordred | actually ... | 22:53 |
| *** fguillot_ has joined #openstack-infra | 22:53 | |
| pabelanger | cloudnull: actually, I do see an SSH timeout for osic-cloud1 | 22:53 |
| pabelanger | cloudnull: let me see if I can get the instance ID | 22:53 |
| mordred | clarkb: multinode testing networking ... | 22:53 |
| clarkb | ya thats the setup for allowing all traffic between multinode right? | 22:54 |
| mordred | clarkb: we don't actually need subnodes_private to have things in it, right? because we have clouds with only public? | 22:54 |
| clarkb | should be simple to just check the ip and use the right iptables command | 22:54 |
| pabelanger | cloudnull: http://paste.openstack.org/show/557762/ timeout waiting for ssh access | 22:55 |
| clarkb | mordred: last time I tried to use public only on clouds with both priovate and and public openstack didn't work | 22:55 |
| clarkb | mordred: clouds like osic when fip and bluebox | 22:55 |
| clarkb | I think NAT is or was creating problems for us there | 22:55 |
| mordred | clarkb: k. so - what if one of the things in subnodes_public was a 10. address | 22:55 |
| clarkb | then other random stuff wouldn't work I would expect | 22:56 |
| cloudnull | pabelanger: looking | 22:56 |
| mordred | clarkb: the tl;dr here is that on osic we detect the 10. ipv4 address as being "public" | 22:56 |
| clarkb | mordred: we should put the ipv6 addr in there no? | 22:56 |
| cloudnull | mordred: does it make your life easier if i change that to a 192 address ? | 22:56 |
| mordred | cloudnull: nope | 22:56 |
| cloudnull | ok | 22:56 |
| mordred | clarkb: well, we put the ipv6 address into interface_ip and will use it correctly for most things | 22:56 |
| *** nwkarsten has joined #openstack-infra | 22:57 | |
| mordred | clarkb: but nodepool multi-node is the one place where we might look explicitly for public/private and expect themto be correct | 22:57 |
| *** sdake has quit IRC | 22:57 | |
| mordred | (most of the rest of the cases it all just works because interface_ip has the ipv6 address and everything is happy) | 22:57 |
| *** mriedem has joined #openstack-infra | 22:57 | |
| *** tonytan4ever has joined #openstack-infra | 22:58 | |
| clarkb | mordred: multinode d-g wants to use the private addrs for most stuff (I think everything) due to the presumed nat issues | 22:58 |
| mordred | clarkb: ok. I'll work on a fix then | 22:58 |
| clarkb | mordred: so I wouldn't expect that to break with 10 net addr in public | 22:58 |
| clarkb | but you need to have it in private list too | 22:58 |
| mordred | clarkb: well, the 10. will not be in private | 22:59 |
| mordred | only in public | 22:59 |
| clarkb | I think nodepool puts it in both | 22:59 |
| mordred | neat | 22:59 |
| clarkb | if there is no private addr then it writes the public to private | 22:59 |
| mordred | I'll go read through that code more | 22:59 |
| clarkb | so that things relying on "private" continue to work | 22:59 |
| mordred | woot! | 22:59 |
| mordred | oh good | 22:59 |
| mordred | (this is me really not wanting to try to solve the problem right now) | 22:59 |
| mordred | clarkb: for slightly more wordy context- the underlying problem is that we currently determine "does this route packets off the cloud" with the Network object. (and to be fair, that's where the router:external property which does not mean routes externally sits) | 23:01 |
| *** nwkarsten has quit IRC | 23:01 | |
| mordred | clarkb: but it turns out you can have a subnet that routes externally and a subnet that does not route externally both attached to the same Network | 23:01 |
| mordred | clarkb: so the _real_ question that needs to be asked is "is the port that provides this IP address attached to a subnet that can route externally" | 23:02 |
| mordred | but that's a bunch more data model trolling to get consistent and right every time - and most of the time it's an extra level of complexity that doesn't show up | 23:02 |
| *** rbrndt has quit IRC | 23:02 | |
| pabelanger | clarkb: did we want to land 355570 now? So we can have the dns fix for tomorrows image builds | 23:02 |
| clarkb | mordred: fun | 23:03 |
| mordred | clarkb: yah. | 23:03 |
| *** tonytan4ever has quit IRC | 23:03 | |
| pabelanger | mordred: can we remove the autohold for Automatically held after failing gate-shade-dsvm-functional-neutron ? | 23:04 |
| pabelanger | or is that still needed | 23:04 |
| mordred | pabelanger: yes. absolutely can remove | 23:04 |
| clarkb | pabelanger: does that work in clouds with no v6? does unbound know to do the right thing in that situation? | 23:04 |
| *** devkulkarni has quit IRC | 23:04 | |
| *** asettle has joined #openstack-infra | 23:04 | |
| mordred | that's a good question | 23:04 |
| pabelanger | clarkb: I tested with both ovh and osic and it worked. | 23:05 |
| pabelanger | I can confirm with each other cloud too | 23:05 |
| clarkb | pabelanger: and you made sure that it was using unbound not the cloud provided resolvers? | 23:06 |
| clarkb | I am not sure if that happens in ovh like in rax | 23:06 |
| *** markvoelker has quit IRC | 23:06 | |
| pabelanger | clarkb: yup, nslookup used 127.0.0.1 | 23:07 |
| pabelanger | same with dig +trace | 23:07 |
| mtreinish | fungi, pabelanger, clarkb: hmm did I miss a step in adding firehose.o.o to cacti: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=300 is all blank | 23:07 |
| pabelanger | welp, internap is also using DNS from cloud provider | 23:07 |
| openstackgerrit | Abhishek Raut proposed openstack-infra/project-config: Use python-db-jobs for tap-as-a-service https://review.openstack.org/355670 | 23:08 |
| jeblair | mtreinish: for starters, isn't the server 'firehose01.openstack.org'? | 23:08 |
| *** xyang1 has quit IRC | 23:08 | |
| *** Goneri has joined #openstack-infra | 23:08 | |
| mtreinish | jeblair: ah, yep that'd probably do it | 23:08 |
| fungi | ahh, yep, need to fix that at http://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera/common.yaml#n290 | 23:09 |
| fungi | i missed that | 23:09 |
| jeblair | i'm not sure if that's the actual cause, but i'm not certain it's not. | 23:09 |
| jeblair | cacti says 'udp ping success / snmp error' | 23:09 |
| *** hongbin has quit IRC | 23:09 | |
| *** asettle has quit IRC | 23:09 | |
| jeblair | (but i think our convention is actual hostnames in cacti, so i think we should change it regardless) | 23:10 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Fix firehose hostname on cacti hiera https://review.openstack.org/355671 | 23:10 |
| mtreinish | jeblair, fungi: ^^^ | 23:10 |
| fungi | as for why it's not showing up, i don't see snmpd running on the server | 23:10 |
| fungi | Active: active (exited) since Mon 2016-08-01 15:48:50 UTC; 2 weeks 0 days ago | 23:11 |
| fungi | sayeth `service snmpd status` | 23:11 |
| jeblair | that would do it fer shure | 23:11 |
| clarkb | pabelanger: approved | 23:12 |
| clarkb | my local xenial host without ntp is definitely running the systemd thing | 23:12 |
| clarkb | trusty doesnt' seem to do much with time though | 23:12 |
| fungi | i'll refrain from restarting snmpd on it until the hiera change makes it onto the cacti host | 23:12 |
| jeblair | fungi: good plan | 23:12 |
| jeblair | less to delete that way | 23:12 |
| fungi | laziness is next to godliness | 23:12 |
| fungi | or something like that | 23:13 |
| mtreinish | pleia2: it looks like puppet updated the stuff, but the cron job is still not happy: http://status.openstack.org/elastic-recheck/data/others.html | 23:13 |
| clarkb | ya I think older non systemd distros are going to be a problem here | 23:14 |
| clarkb | definitely doesn't do anything on trusty | 23:14 |
| clarkb | there goes that idea :P | 23:14 |
| *** asselin has joined #openstack-infra | 23:14 | |
| pabelanger | clarkb: I think we are going to be good, all clouds appear to have inet6 address on eth0 and lo0. And unbound seems to do the right think if ipv6 entry is not accessible, fails to the next entry which is ipv4 | 23:15 |
| *** asselin_ has quit IRC | 23:15 | |
| *** asselin_ has joined #openstack-infra | 23:15 | |
| clarkb | pabelanger: most of those clouds just have link local addrs though | 23:16 |
| *** baoli has joined #openstack-infra | 23:16 | |
| cloudnull | pabelanger: interestingly I'm seeing this on the compute node where that instance was spawned. http://cdn.pasteraw.com/b8ivpgerqk2si66honrodwt9vryxp30 | 23:16 |
| clarkb | pabelanger: which won't get them to gogole dns. The exceptions are osic, rax, and vexxhost | 23:16 |
| cloudnull | however no other errirs | 23:16 |
| cloudnull | *errors | 23:16 |
| clarkb | in any case if it falls back to ipv4 without ridiculously long timeouts we should be fine | 23:16 |
| pabelanger | cloudnull: right, I've tested on both bluebox and ovh, if I force ipv6 dns, it fails. If I add both, ipv6 and ipv4, dns works as expected | 23:17 |
| pabelanger | clarkb: Ya, it is pretty fast | 23:17 |
| pabelanger | surprisingly | 23:17 |
| cloudnull | pabelanger: was talking about that instance you noted as having ssh timeouts | 23:18 |
| * clarkb is beginning to wonder if the simplest thing would be to install our own init script for sntp and jsut run that once at boot on all platforms | 23:18 | |
| clarkb | probably going to run into dependency hell with the existing distro stuff though | 23:18 |
| *** asselin has quit IRC | 23:19 | |
| pabelanger | cloudnull: Oh, neat. So you are seeing something | 23:19 |
| cloudnull | yea i may need to do some iptables munging or neutron tweaking to make that happier. | 23:20 |
| cloudnull | idk quite yet | 23:20 |
| * pabelanger nods | 23:20 | |
| cloudnull | but yes. | 23:20 |
| openstackgerrit | Merged openstack-infra/project-config: Add IPv6 DNS support https://review.openstack.org/355570 | 23:21 |
| *** xarses has quit IRC | 23:22 | |
| cloudnull | found a bug that was patched but it looks like its just a warning: https://bugs.launchpad.net/neutron/+bug/1565705 | 23:23 |
| openstack | Launchpad bug 1565705 in neutron "iptables duplicate rule warning on ports with multiple security groups" [Medium,Fix released] - Assigned to Kevin Benton (kevinbenton) | 23:23 |
| *** shashank_hegde has quit IRC | 23:23 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics https://review.openstack.org/263971 | 23:24 |
| fungi | anteaya: zaro: ^ latest gerrit upgrade allowed some serious simplification there on multiple fronts | 23:24 |
| zaro | fungi: ahh nice! | 23:25 |
| zaro | fungi: i'm testing online index but sorta hit a snag. not enough memory on review-dev now! | 23:26 |
| fungi | dropped more than 50 loc | 23:26 |
| fungi | zaro: oh, ouch! | 23:26 |
| fungi | we can rebuild it bigger if needed | 23:26 |
| clarkb | and it looks like on fedora and centos we would have to explicitly install something to set the time so they are more like ubuntu trusty | 23:27 |
| zaro | fungi: yeah may need to if we want to test multiple users hitting it while it's reindexing. | 23:27 |
| clarkb | I will need to fiddle with these VMs a bit more when its not almost the end of the day | 23:28 |
| clarkb | figure out what magic is needed to make things happen | 23:28 |
| zaro | fungi: on the bright side it seems to be working great with just me poking at it. | 23:28 |
| *** devkulkarni has joined #openstack-infra | 23:29 | |
| *** gyee has quit IRC | 23:30 | |
| fungi | anteaya: dhellmann: you _should_ be able to use https://review.openstack.org/263971 on your own to generate the electoral rolls now, though with the coming round of technical elections i think i should generate a set too and then election officials can confirm the lists they have match mine just to be on the safe side. if it works out though, our gerrit admins can get completely out of involvement in | 23:33 |
| *** kzaitsev_mb has quit IRC | 23:33 | |
| fungi | future elections unless troubleshooting becomes necessary | 23:33 |
| *** xarses has joined #openstack-infra | 23:34 | |
| dhellmann | fungi : excellent | 23:34 |
| dhellmann | though I won't be an election official since I'll be up for election | 23:35 |
| fungi | ahh, yup ;) | 23:35 |
| *** harlowja has joined #openstack-infra | 23:36 | |
| *** devkulkarni has quit IRC | 23:40 | |
| clarkb | the mroe I dig the more I think we might have to do our own equivalent to ntpdate at boot using system appropriate tools | 23:40 |
| clarkb | since everything seems to do the gentle update to avoid making processes unhappy | 23:40 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test. https://review.openstack.org/346949 | 23:40 |
| *** hockeynut has quit IRC | 23:43 | |
| jhesketh | Morning | 23:44 |
| *** sdague has quit IRC | 23:44 | |
| *** pahuang has quit IRC | 23:45 | |
| *** jerryz has quit IRC | 23:46 | |
| mordred | it's a jhesketh ! | 23:47 |
| mordred | clarkb: yah - when what we want is "MAKE IT GOOD NOW" | 23:47 |
| *** sarob has quit IRC | 23:48 | |
| * clarkb is happy his suse system already comes with this feature | 23:48 | |
| clarkb | but I can't find anything like ti on ubuntu | 23:48 |
| *** zhurong has joined #openstack-infra | 23:49 | |
| *** gyee has joined #openstack-infra | 23:49 | |
| jhesketh | mordred: indeed :-) | 23:50 |
| *** dingyichen has joined #openstack-infra | 23:51 | |
| *** baoli has quit IRC | 23:54 | |
| cloudnull | pabelanger: sadly, yet again, I can't find anything specifically wront with the environment that would produce an ingress ssh timeout. If we can identify one of these instances and keep it online I can troubleshoot it further. | 23:55 |
| cloudnull | Now that I have LOTS of IPs to play with I'll try to reproduce it on my own but for now, IDK :'( | 23:56 |
| *** asselin_ has quit IRC | 23:56 | |
| clarkb | reading chrony init scripts for ubuntu it will do a burst on interface startup but not a step | 23:56 |
| *** jimbaker has quit IRC | 23:57 | |
| *** zhurong has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!