| *** zz_dimtruck is now known as dimtruck | 00:01 | |
| boris-42 | fungi: hi | 00:01 |
|---|---|---|
| boris-42 | fungi: could you help to get this change in requirments in https://review.openstack.org/#/c/301108/ | 00:02 |
| boris-42 | fungi: we have really bad bug in rally | 00:02 |
| boris-42 | fungi: thanks | 00:03 |
| *** jamesmcarthur has quit IRC | 00:03 | |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder: Remove ssh host keys when using simple init https://review.openstack.org/301982 | 00:04 |
| *** flepied has quit IRC | 00:04 | |
| *** asettle has joined #openstack-infra | 00:05 | |
| *** ZZelle_ has quit IRC | 00:06 | |
| clarkb | boris-42: are we still in a requirements freeze? you may need to coordinate with the release team | 00:06 |
| boris-42 | clarkb: so not sure | 00:06 |
| boris-42 | dhellmann: ^ | 00:06 |
| boris-42 | dims: ^ | 00:07 |
| fungi | boris-42: clarkb: dims had already +2'd so i assumed master reqs is no longer frozen | 00:07 |
| boris-42 | fungi: clarkb thanks | 00:07 |
| dims | fungi : yes, master requirements is open | 00:07 |
| fungi | but i can abort it if someone says it's still supposed to be frozen for release | 00:07 |
| fungi | thanks dims! | 00:08 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Upload the ironic-python-agent images to cache https://review.openstack.org/301698 | 00:08 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Use the cached ironic-python-agent images https://review.openstack.org/301699 | 00:08 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Fix image caching logic https://review.openstack.org/299537 | 00:08 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image https://review.openstack.org/299538 | 00:08 |
| dims | fungi : thanks! should be good | 00:08 |
| *** marg7175 has joined #openstack-infra | 00:10 | |
| *** gildub has quit IRC | 00:11 | |
| *** tqtran has quit IRC | 00:13 | |
| *** xarses has joined #openstack-infra | 00:15 | |
| *** ddieterly has joined #openstack-infra | 00:16 | |
| *** yamamoto_ has quit IRC | 00:18 | |
| *** sridhar_ram has quit IRC | 00:19 | |
| *** xarses has quit IRC | 00:19 | |
| *** nwkarsten has joined #openstack-infra | 00:23 | |
| *** flepied has joined #openstack-infra | 00:23 | |
| *** mtanino has quit IRC | 00:25 | |
| *** claudiub|2 has quit IRC | 00:27 | |
| *** Qiming has quit IRC | 00:27 | |
| *** yamamot__ has joined #openstack-infra | 00:31 | |
| *** sdake has joined #openstack-infra | 00:32 | |
| *** jamesmcarthur has joined #openstack-infra | 00:34 | |
| *** sdake_ has quit IRC | 00:34 | |
| *** nwkarsten has quit IRC | 00:34 | |
| *** shashank_hegde has quit IRC | 00:35 | |
| *** Sukhdev has quit IRC | 00:35 | |
| *** nwkarsten has joined #openstack-infra | 00:35 | |
| *** yamamot__ has quit IRC | 00:35 | |
| *** jamesmcarthur has quit IRC | 00:38 | |
| *** nwkarsten has quit IRC | 00:39 | |
| lifeless | just a heads up | 00:40 |
| lifeless | funcsigs and mock are getting releases today | 00:40 |
| lifeless | constrants protected projects will be fine :).... its possible (however unlikely) that others may get disrupted | 00:40 |
| *** jamesmcarthur has joined #openstack-infra | 00:41 | |
| *** gildub has joined #openstack-infra | 00:42 | |
| *** esker has quit IRC | 00:43 | |
| *** dimtruck is now known as zz_dimtruck | 00:43 | |
| *** jcooley_ has quit IRC | 00:44 | |
| *** xarses has joined #openstack-infra | 00:45 | |
| *** xarses has quit IRC | 00:46 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-openstackci: Order periodic job results by date https://review.openstack.org/301989 | 00:46 |
| *** Jeffrey4l has joined #openstack-infra | 00:47 | |
| *** esker has joined #openstack-infra | 00:47 | |
| *** xarses has joined #openstack-infra | 00:47 | |
| *** jamesmcarthur has quit IRC | 00:48 | |
| *** zz_dimtruck is now known as dimtruck | 00:49 | |
| *** jcooley_ has joined #openstack-infra | 00:49 | |
| *** flepied has quit IRC | 00:50 | |
| *** sdake_ has joined #openstack-infra | 00:51 | |
| *** sdake has quit IRC | 00:51 | |
| *** jamesmcarthur has joined #openstack-infra | 00:51 | |
| *** baoli has joined #openstack-infra | 00:52 | |
| *** knikolla has quit IRC | 00:56 | |
| *** lakshmiS has joined #openstack-infra | 00:57 | |
| *** yamamoto has joined #openstack-infra | 01:00 | |
| *** yuanying has quit IRC | 01:01 | |
| *** yuanying has joined #openstack-infra | 01:02 | |
| *** mylu has quit IRC | 01:03 | |
| *** rossella_s has quit IRC | 01:03 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:03 | |
| *** rossella_s has joined #openstack-infra | 01:03 | |
| *** sflanigan has quit IRC | 01:03 | |
| *** yuanying has quit IRC | 01:04 | |
| *** flepied has joined #openstack-infra | 01:04 | |
| *** jamesmcarthur has quit IRC | 01:04 | |
| *** zhurong has joined #openstack-infra | 01:05 | |
| *** yamamoto has quit IRC | 01:05 | |
| *** kgiusti has quit IRC | 01:06 | |
| *** roxanaghe has joined #openstack-infra | 01:07 | |
| *** erlon has quit IRC | 01:07 | |
| *** EricGonczer_ has joined #openstack-infra | 01:08 | |
| *** pfallenop has quit IRC | 01:09 | |
| *** zzzeek_ has quit IRC | 01:10 | |
| *** darvon has quit IRC | 01:10 | |
| *** pfallenop has joined #openstack-infra | 01:11 | |
| *** darvon has joined #openstack-infra | 01:11 | |
| *** roxanaghe has quit IRC | 01:12 | |
| *** yuanying has joined #openstack-infra | 01:13 | |
| *** zzzeek has joined #openstack-infra | 01:13 | |
| *** amotoki has joined #openstack-infra | 01:13 | |
| *** amotoki_ has joined #openstack-infra | 01:15 | |
| *** tiswanso has joined #openstack-infra | 01:15 | |
| *** yamahata has joined #openstack-infra | 01:15 | |
| *** kzaitsev_mb has quit IRC | 01:16 | |
| *** pvaneck has quit IRC | 01:17 | |
| *** tiswanso has quit IRC | 01:17 | |
| *** Swami has quit IRC | 01:17 | |
| dims | lifeless : oh dear :) | 01:17 |
| *** tiswanso has joined #openstack-infra | 01:18 | |
| *** amotoki has quit IRC | 01:18 | |
| *** sridhar_ram has joined #openstack-infra | 01:19 | |
| *** bpokorny_ has joined #openstack-infra | 01:20 | |
| *** mriedem has quit IRC | 01:20 | |
| *** akshai has joined #openstack-infra | 01:21 | |
| *** flepied has quit IRC | 01:21 | |
| *** sflanigan has joined #openstack-infra | 01:22 | |
| *** sflanigan has joined #openstack-infra | 01:22 | |
| *** akshai_ has joined #openstack-infra | 01:22 | |
| *** thorst_ has quit IRC | 01:22 | |
| *** bpokorny has quit IRC | 01:23 | |
| *** asettle has quit IRC | 01:23 | |
| *** thorst has joined #openstack-infra | 01:23 | |
| *** bpokorny_ has quit IRC | 01:24 | |
| *** esker has quit IRC | 01:24 | |
| *** yuanying has quit IRC | 01:25 | |
| *** lakshmiS has quit IRC | 01:25 | |
| *** akshai has quit IRC | 01:25 | |
| *** yuanying has joined #openstack-infra | 01:28 | |
| *** lakshmiS has joined #openstack-infra | 01:29 | |
| *** yuanying has quit IRC | 01:30 | |
| *** Qiming has joined #openstack-infra | 01:30 | |
| *** thorst has quit IRC | 01:31 | |
| *** salv-orlando has joined #openstack-infra | 01:32 | |
| *** sridhar_ram1 has joined #openstack-infra | 01:33 | |
| *** sridhar_ram has quit IRC | 01:34 | |
| *** Adri2000_ has quit IRC | 01:34 | |
| *** Jeffrey4l has quit IRC | 01:34 | |
| *** apoorvad has quit IRC | 01:35 | |
| *** EricGonczer_ has quit IRC | 01:35 | |
| *** ybathia has quit IRC | 01:37 | |
| *** mylu has joined #openstack-infra | 01:38 | |
| lifeless | dims: should be fine... should :) | 01:38 |
| *** flepied has joined #openstack-infra | 01:39 | |
| *** yuanying has joined #openstack-infra | 01:39 | |
| *** Jeffrey4l has joined #openstack-infra | 01:42 | |
| *** yuanying has quit IRC | 01:42 | |
| *** kengo_sakai has joined #openstack-infra | 01:42 | |
| *** jamielennox is now known as jamielennox|away | 01:43 | |
| *** yuanying has joined #openstack-infra | 01:43 | |
| *** yamamot__ has joined #openstack-infra | 01:46 | |
| *** Adri2000 has joined #openstack-infra | 01:46 | |
| *** marg7175 has quit IRC | 01:47 | |
| *** woodster_ has quit IRC | 01:47 | |
| *** salv-orlando has quit IRC | 01:47 | |
| *** yonglihe has joined #openstack-infra | 01:48 | |
| *** tphummel has quit IRC | 01:50 | |
| *** yamamot__ has quit IRC | 01:51 | |
| *** jamielennox|away is now known as jamielennox | 01:53 | |
| *** sdake has joined #openstack-infra | 01:55 | |
| *** amrith is now known as _amrith_ | 01:56 | |
| *** sdake_ has quit IRC | 01:56 | |
| *** asettle has joined #openstack-infra | 02:01 | |
| *** EricGonczer_ has joined #openstack-infra | 02:01 | |
| *** yuanying has quit IRC | 02:02 | |
| *** Qiming has quit IRC | 02:03 | |
| *** Qiming has joined #openstack-infra | 02:03 | |
| *** yuanying has joined #openstack-infra | 02:05 | |
| *** asettle has quit IRC | 02:06 | |
| *** xarses has quit IRC | 02:12 | |
| *** xarses has joined #openstack-infra | 02:12 | |
| *** sflanigan has quit IRC | 02:13 | |
| *** xarses has quit IRC | 02:13 | |
| *** xarses has joined #openstack-infra | 02:14 | |
| *** EricGonczer_ has quit IRC | 02:14 | |
| *** xarses has quit IRC | 02:14 | |
| *** xarses has joined #openstack-infra | 02:15 | |
| *** ddieterly is now known as ddieterly[away] | 02:15 | |
| *** ddieterly[away] is now known as ddieterly | 02:15 | |
| *** mylu has quit IRC | 02:16 | |
| *** EricGonczer_ has joined #openstack-infra | 02:16 | |
| *** EricGonczer_ has quit IRC | 02:17 | |
| *** sridhar_ram1 has quit IRC | 02:18 | |
| *** lakshmiS has quit IRC | 02:18 | |
| *** mylu has joined #openstack-infra | 02:18 | |
| *** ddieterly is now known as ddieterly[away] | 02:19 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/300845 | 02:20 |
| *** zzzeek has quit IRC | 02:21 | |
| *** zzzeek has joined #openstack-infra | 02:23 | |
| *** jamesmcarthur has joined #openstack-infra | 02:23 | |
| *** sflanigan has joined #openstack-infra | 02:25 | |
| *** sflanigan has joined #openstack-infra | 02:25 | |
| *** EricGonczer_ has joined #openstack-infra | 02:26 | |
| *** jamesmcarthur has quit IRC | 02:26 | |
| *** amotoki_ is now known as amotoki | 02:28 | |
| *** marg7175 has joined #openstack-infra | 02:29 | |
| *** edmondsw has quit IRC | 02:29 | |
| *** dimtruck is now known as zz_dimtruck | 02:29 | |
| *** thorst has joined #openstack-infra | 02:29 | |
| *** yamamoto has joined #openstack-infra | 02:31 | |
| *** EricGonczer_ has quit IRC | 02:31 | |
| *** achanda has joined #openstack-infra | 02:34 | |
| *** rhochmuth has left #openstack-infra | 02:34 | |
| *** tphummel has joined #openstack-infra | 02:34 | |
| *** yamamoto has quit IRC | 02:35 | |
| *** thorst has quit IRC | 02:36 | |
| *** jamesmcarthur has joined #openstack-infra | 02:38 | |
| *** asettle has joined #openstack-infra | 02:39 | |
| *** mylu has quit IRC | 02:44 | |
| *** ramishra has joined #openstack-infra | 02:46 | |
| *** ybathia has joined #openstack-infra | 02:46 | |
| *** jamesmcarthur has quit IRC | 02:49 | |
| *** mylu has joined #openstack-infra | 02:49 | |
| *** Keedya has joined #openstack-infra | 02:49 | |
| *** yuanying has quit IRC | 02:50 | |
| *** ybathia has quit IRC | 02:51 | |
| *** Keedya has quit IRC | 02:53 | |
| *** Keedya has joined #openstack-infra | 02:53 | |
| *** Keedya has quit IRC | 02:58 | |
| *** sdake_ has joined #openstack-infra | 03:01 | |
| *** Keedya has joined #openstack-infra | 03:02 | |
| *** salv-orlando has joined #openstack-infra | 03:03 | |
| *** binbincong has quit IRC | 03:04 | |
| *** sdake has quit IRC | 03:04 | |
| *** Keedya has quit IRC | 03:05 | |
| *** marg7175 has quit IRC | 03:08 | |
| *** Keedya has joined #openstack-infra | 03:10 | |
| *** mylu has quit IRC | 03:10 | |
| *** jamesmcarthur has joined #openstack-infra | 03:11 | |
| *** mylu has joined #openstack-infra | 03:13 | |
| *** salv-orlando has quit IRC | 03:13 | |
| *** mylu has quit IRC | 03:14 | |
| *** mylu has joined #openstack-infra | 03:15 | |
| *** Keedya has quit IRC | 03:15 | |
| *** yamamot__ has joined #openstack-infra | 03:16 | |
| *** roxanaghe has joined #openstack-infra | 03:16 | |
| *** jamesmcarthur has quit IRC | 03:17 | |
| *** fedexo has joined #openstack-infra | 03:17 | |
| *** lakshmiS has joined #openstack-infra | 03:19 | |
| *** ddieterly[away] has quit IRC | 03:19 | |
| *** amotoki has quit IRC | 03:20 | |
| *** yamamot__ has quit IRC | 03:21 | |
| *** lakshmiS has quit IRC | 03:23 | |
| *** markvoelker has quit IRC | 03:26 | |
| *** achanda has quit IRC | 03:27 | |
| *** _ari_ has joined #openstack-infra | 03:28 | |
| *** roxanaghe has quit IRC | 03:29 | |
| *** amotoki has joined #openstack-infra | 03:30 | |
| *** _ari_ has quit IRC | 03:33 | |
| *** thorst has joined #openstack-infra | 03:34 | |
| *** esker has joined #openstack-infra | 03:34 | |
| *** binbincong has joined #openstack-infra | 03:38 | |
| *** mylu has quit IRC | 03:41 | |
| *** thorst has quit IRC | 03:41 | |
| *** mylu has joined #openstack-infra | 03:41 | |
| *** mylu has joined #openstack-infra | 03:42 | |
| *** ramishra has quit IRC | 03:42 | |
| *** baoli has quit IRC | 03:43 | |
| *** tiswanso has quit IRC | 03:43 | |
| *** roxanaghe has joined #openstack-infra | 03:44 | |
| *** jamesmcarthur has joined #openstack-infra | 03:45 | |
| *** ramishra has joined #openstack-infra | 03:45 | |
| *** yuanying has joined #openstack-infra | 03:46 | |
| *** jamesmcarthur has quit IRC | 03:48 | |
| *** tqtran has joined #openstack-infra | 03:49 | |
| *** links has joined #openstack-infra | 03:52 | |
| *** esker has quit IRC | 03:52 | |
| *** resker has joined #openstack-infra | 03:52 | |
| *** esker has joined #openstack-infra | 03:57 | |
| *** jamesmcarthur has joined #openstack-infra | 03:58 | |
| *** resker has quit IRC | 03:59 | |
| *** shashank_hegde has joined #openstack-infra | 04:00 | |
| *** cody-somerville has quit IRC | 04:00 | |
| *** yamamot__ has joined #openstack-infra | 04:01 | |
| *** tqtran has quit IRC | 04:02 | |
| *** esker has quit IRC | 04:05 | |
| *** yamamot__ has quit IRC | 04:05 | |
| *** esker has joined #openstack-infra | 04:07 | |
| *** cody-somerville has joined #openstack-infra | 04:09 | |
| *** AJaeger has quit IRC | 04:09 | |
| *** AJaeger has joined #openstack-infra | 04:10 | |
| *** yamamot__ has joined #openstack-infra | 04:17 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/reviewstats: Filter out members without usernames https://review.openstack.org/302005 | 04:17 |
| openstackgerrit | Tony Breeds proposed openstack-infra/reviewstats: Indicate the branch when generating stable checks https://review.openstack.org/302006 | 04:17 |
| *** lakshmiS has joined #openstack-infra | 04:19 | |
| *** asettle has quit IRC | 04:20 | |
| *** oanson has joined #openstack-infra | 04:20 | |
| *** asettle has joined #openstack-infra | 04:20 | |
| openstackgerrit | Merged openstack/requirements: Bump alembic version to 0.8.4 https://review.openstack.org/301108 | 04:21 |
| *** jamielennox is now known as jamielennox|away | 04:21 | |
| *** lakshmiS has quit IRC | 04:24 | |
| *** markvoelker has joined #openstack-infra | 04:26 | |
| *** jamesmcarthur has quit IRC | 04:29 | |
| *** jamielennox|away is now known as jamielennox | 04:29 | |
| *** shashank_hegde has quit IRC | 04:30 | |
| *** salv-orlando has joined #openstack-infra | 04:31 | |
| *** markvoelker has quit IRC | 04:32 | |
| *** mrmartin has joined #openstack-infra | 04:36 | |
| *** zhurong has quit IRC | 04:39 | |
| *** salv-orlando has quit IRC | 04:39 | |
| *** thorst has joined #openstack-infra | 04:39 | |
| *** woodster_ has joined #openstack-infra | 04:44 | |
| *** ramishra has quit IRC | 04:45 | |
| *** thorst has quit IRC | 04:47 | |
| *** zhurong has joined #openstack-infra | 04:48 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's download protocols https://review.openstack.org/300596 | 04:50 |
| *** jamesmcarthur has joined #openstack-infra | 04:51 | |
| *** mylu has quit IRC | 04:51 | |
| *** abregman has joined #openstack-infra | 04:52 | |
| *** Kiall has quit IRC | 04:52 | |
| *** jcooley_ has quit IRC | 04:53 | |
| *** hichihara has quit IRC | 04:53 | |
| *** jaosorior has joined #openstack-infra | 04:55 | |
| *** Kiall has joined #openstack-infra | 04:56 | |
| *** jamesmcarthur has quit IRC | 04:58 | |
| *** tobiash has quit IRC | 04:58 | |
| *** shashank_hegde has joined #openstack-infra | 04:59 | |
| *** kengo_sa_ has joined #openstack-infra | 05:00 | |
| *** achanda has joined #openstack-infra | 05:00 | |
| *** armax has quit IRC | 05:01 | |
| *** jamielennox is now known as jamielennox|away | 05:01 | |
| *** rossella_s has quit IRC | 05:03 | |
| *** rossella_s has joined #openstack-infra | 05:03 | |
| *** Sukhdev has joined #openstack-infra | 05:03 | |
| *** kushal has joined #openstack-infra | 05:06 | |
| *** isaacb has joined #openstack-infra | 05:07 | |
| *** mylu has joined #openstack-infra | 05:08 | |
| *** tobiash has joined #openstack-infra | 05:11 | |
| *** jamielennox|away is now known as jamielennox | 05:11 | |
| *** mylu has quit IRC | 05:12 | |
| *** nadya has joined #openstack-infra | 05:21 | |
| *** jamielennox is now known as jamielennox|away | 05:22 | |
| *** [1]Thelo has joined #openstack-infra | 05:23 | |
| *** Thelo has quit IRC | 05:25 | |
| *** [1]Thelo is now known as Thelo | 05:25 | |
| *** ramishra has joined #openstack-infra | 05:25 | |
| *** akshai_ has quit IRC | 05:28 | |
| *** krtaylor has quit IRC | 05:29 | |
| *** jamielennox|away is now known as jamielennox | 05:30 | |
| *** jcooley_ has joined #openstack-infra | 05:30 | |
| *** sekrit has quit IRC | 05:30 | |
| *** mrmartin has quit IRC | 05:30 | |
| *** krtaylor has joined #openstack-infra | 05:31 | |
| *** fedexo has quit IRC | 05:36 | |
| *** xarses has quit IRC | 05:38 | |
| *** Thelo has quit IRC | 05:39 | |
| *** nmagnezi has joined #openstack-infra | 05:39 | |
| *** xarses has joined #openstack-infra | 05:39 | |
| *** sdake has joined #openstack-infra | 05:40 | |
| *** roxanaghe has quit IRC | 05:41 | |
| *** exploreshaifali has joined #openstack-infra | 05:41 | |
| *** roxanaghe has joined #openstack-infra | 05:42 | |
| *** asettle has quit IRC | 05:42 | |
| *** sdake_ has quit IRC | 05:42 | |
| *** xarses_ has joined #openstack-infra | 05:44 | |
| *** thorst has joined #openstack-infra | 05:44 | |
| *** asettle has joined #openstack-infra | 05:44 | |
| *** salv-orlando has joined #openstack-infra | 05:46 | |
| *** tphummel has quit IRC | 05:46 | |
| *** xarses has quit IRC | 05:47 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Remove ssh host keys when using simple init https://review.openstack.org/301982 | 05:47 |
| openstackgerrit | vikram.choudhary proposed openstack-infra/project-config: Adding neutron-dynamic-routing as neutron's stadium project. https://review.openstack.org/268726 | 05:47 |
| *** sdake_ has joined #openstack-infra | 05:49 | |
| *** ramishra has quit IRC | 05:51 | |
| *** kengo_sa_ has quit IRC | 05:51 | |
| *** thorst has quit IRC | 05:52 | |
| *** sdake has quit IRC | 05:52 | |
| *** claudiub|2 has joined #openstack-infra | 05:53 | |
| *** salv-orlando has quit IRC | 05:55 | |
| *** binbincong has quit IRC | 05:57 | |
| *** nadya has quit IRC | 05:58 | |
| *** rcernin has joined #openstack-infra | 05:59 | |
| AJaeger | aeng: are you still around? | 05:59 |
| aeng | yup | 06:00 |
| AJaeger | Is there anything you need from me? | 06:00 |
| AJaeger | Haven't read my emails completely yet... | 06:00 |
| aeng | ahhaha... well.. for the merge translation, its a bug in Zanata | 06:00 |
| AJaeger | And for the pushing? | 06:01 |
| AJaeger | Is that a result? Should we restart the service? | 06:01 |
| aeng | for the failure in Jenkins..we might need to get hold of the logs when the job happens | 06:01 |
| AJaeger | should I push manually something? | 06:02 |
| AJaeger | we can then ask an infra-root for log files | 06:02 |
| *** fawadkhaliq has joined #openstack-infra | 06:02 | |
| aeng | yup, keep the timestamp of when you doing it | 06:03 |
| aeng | not sure what timezone the server runs in | 06:03 |
| AJaeger | it should be UTC | 06:03 |
| AJaeger | zanata-cli -B -e push ? Or any extra arguments? | 06:03 |
| aeng | yup | 06:04 |
| aeng | thats the command | 06:04 |
| AJaeger | jhesketh, yolanda : can either of you help us, please? We need an infra-root... | 06:05 |
| AJaeger | aeng: http://paste.openstack.org/show/493102/ is the failure | 06:05 |
| *** sridhar_ram has joined #openstack-infra | 06:06 | |
| aeng | AJaeger, yup. so i need the server log to see what happens | 06:06 |
| aeng | suspect its a bug too in Zanata | 06:07 |
| *** roxanaghe has quit IRC | 06:07 | |
| *** Sukhdev has quit IRC | 06:08 | |
| AJaeger | aeng: we have the time 6:04 UTC for it. Once an infra-root shows up, I'll ask them to send the log files to you. | 06:09 |
| *** sridhar_ram1 has joined #openstack-infra | 06:09 | |
| yolanda | AJaeger, i am around for 15 minutes, then i need to go to school. What's up? | 06:10 |
| yolanda | zanata failures? | 06:11 |
| AJaeger | yolanda: can you get us log files from translate.openstack.org, please? | 06:11 |
| *** sridhar_ram has quit IRC | 06:11 | |
| yolanda | yep | 06:11 |
| AJaeger | /opt/wildfly/standalone/log/server.log is the file, we need the last 10 minutes... | 06:11 |
| AJaeger | yolanda: I just gave you aeng's email address in private window | 06:12 |
| yolanda | ok, collecting it | 06:12 |
| AJaeger | aeng: should we afterwards just try restarting zanata? | 06:13 |
| AJaeger | aeng: First have a look at the log file ;), please | 06:13 |
| aeng | well.. lets see the log first | 06:13 |
| *** mikelk has joined #openstack-infra | 06:13 | |
| yolanda | AJaeger, aeng, sent | 06:15 |
| AJaeger | thanks, yolanda ! I guess, we'll ping you later again ;) | 06:15 |
| yolanda | sure, no problem. I need to take children to school and do some paperwork, but will be back in an hour as much | 06:16 |
| AJaeger | no stress, thanks! | 06:17 |
| *** abregman has quit IRC | 06:20 | |
| *** sridhar_ram1 has quit IRC | 06:20 | |
| *** scheuran has joined #openstack-infra | 06:20 | |
| *** abregman has joined #openstack-infra | 06:20 | |
| *** fawadkhaliq has quit IRC | 06:20 | |
| jaosorior | Hey, I'm seeing some errors starting rabbitmq in an F23 node running devstack, for the barbican gate. Does someone know if there are any problems like that around? http://logs.openstack.org/53/301353/8/check/gate-barbican-dogtag-devstack-dsvm-f23/9a7b2db/logs/syslog.txt.gz#_Apr_05_21_16_07 | 06:22 |
| *** cody-somerville has quit IRC | 06:22 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: DNM: Attempt to fix pypy https://review.openstack.org/302045 | 06:27 |
| AJaeger | jaosorior: nothing I'm aware of, best ask some of the F23 experts like ianw or pabelanger. | 06:28 |
| ianw | jaosorior: i'm not aware of known issues around that | 06:28 |
| *** markvoelker has joined #openstack-infra | 06:29 | |
| ianw | jaosorior: but, looking closer, yeah, there may be an issue | 06:30 |
| aeng | AJaeger, which project you're trying to push? | 06:30 |
| openstackgerrit | Merged openstack-infra/project-config: Add install-distro-packages to chef jobs https://review.openstack.org/301904 | 06:32 |
| ianw | jaosorior: yep, looks like -> https://bugzilla.redhat.com/show_bug.cgi?id=1323610 has hit | 06:33 |
| openstack | bugzilla.redhat.com bug 1323610 in rabbitmq-server "Broken since 3.6.1" [Unspecified,Assigned] - Assigned to lemenkov | 06:33 |
| jaosorior | ianw: that looks like it. Is there anything we can do about it? | 06:35 |
| jaosorior | something I should modify from the gate definition in project-config or something of the sort? | 06:35 |
| *** markvoelker has quit IRC | 06:35 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/300845 | 06:35 |
| ianw | jaosorior: hmm, i don't really think so. i'm not sure we can tell dnf to install an older version | 06:36 |
| *** cody-somerville has joined #openstack-infra | 06:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Add notification for stable branches in trove IRC channel https://review.openstack.org/301903 | 06:36 |
| AJaeger | aeng: barbican | 06:37 |
| *** claudiub has joined #openstack-infra | 06:37 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Ubuntu binary build and deploy gate for kolla https://review.openstack.org/292942 | 06:38 |
| *** jamielennox is now known as jamielennox|away | 06:38 | |
| *** claudiub|2 has quit IRC | 06:40 | |
| ianw | jaosorior: hmm, dnf downgrade might work? | 06:41 |
| jaosorior | ianw: Might be | 06:42 |
| ianw | jaosorior: just let me test that out on an image here, if so we can stick that in devstack quickly | 06:42 |
| jaosorior | alright | 06:42 |
| *** flepied has quit IRC | 06:42 | |
| aeng | AJaeger, we might need more information | 06:42 |
| *** flepied has joined #openstack-infra | 06:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Add post job for building and uploading tinyipa https://review.openstack.org/299485 | 06:44 |
| openstackgerrit | Merged openstack-infra/project-config: Ironic uses grenade plugin. Setup GRENADE_PLUGINRC https://review.openstack.org/299011 | 06:44 |
| *** mrmartin has joined #openstack-infra | 06:46 | |
| *** jcooley_ has quit IRC | 06:46 | |
| *** woodster_ has quit IRC | 06:47 | |
| *** cody-somerville has quit IRC | 06:47 | |
| *** cody-somerville has joined #openstack-infra | 06:47 | |
| *** nadya has joined #openstack-infra | 06:47 | |
| *** achanda has quit IRC | 06:47 | |
| *** shashank_hegde has quit IRC | 06:48 | |
| openstackgerrit | Merged openstack-infra/project-config: Migrate groups to ubuntu-trusty https://review.openstack.org/301678 | 06:48 |
| *** jcooley_ has joined #openstack-infra | 06:48 | |
| aeng | AJaeger, yolanda, can you get the zanata.xml generated in https://jenkins.openstack.org/job/aodh-upstream-translation-update/290/console | 06:48 |
| *** thorst has joined #openstack-infra | 06:49 | |
| AJaeger | aeng: I can generate it manually | 06:50 |
| aeng | yup, | 06:50 |
| AJaeger | I have the barbican one here - is that one ok as well? | 06:50 |
| aeng | AJaeger, its alright.. .we might need the database from infra | 06:51 |
| AJaeger | http://paste.openstack.org/show/493105/ | 06:51 |
| aeng | doesnt seems to be the content in text | 06:51 |
| AJaeger | let me generate aodh... | 06:51 |
| aeng | we need to db to debug | 06:52 |
| aeng | not much we can see from here | 06:52 |
| aeng | i will send email to eliz | 06:52 |
| AJaeger | aeng: let's wait for yolanda to return, hope she can help. please include her in the email | 06:52 |
| ianw | jaosorior: let's see how https://review.openstack.org/302055 goes | 06:53 |
| AJaeger | aodh : http://paste.openstack.org/show/493107/ | 06:53 |
| *** achanda has joined #openstack-infra | 06:53 | |
| *** mrmartin has quit IRC | 06:54 | |
| jaosorior | ianw: thanks. Lets see how that goes | 06:54 |
| jaosorior | I'll make a test change in barbican to see if this fixes the issue | 06:55 |
| *** zhurong has quit IRC | 06:55 | |
| *** nadya has quit IRC | 06:56 | |
| *** zhurong has joined #openstack-infra | 06:57 | |
| *** thorst has quit IRC | 06:57 | |
| *** vgridnev has joined #openstack-infra | 06:59 | |
| *** achanda has quit IRC | 07:00 | |
| *** jaosorior has quit IRC | 07:00 | |
| jhesketh | AJaeger, aeng: were you two able to get help with the translation thing (re email) | 07:02 |
| *** openstackgerrit has quit IRC | 07:02 | |
| aeng | jhesketh, which email? | 07:02 |
| *** openstackgerrit has joined #openstack-infra | 07:03 | |
| jhesketh | aeng: "merging of translations" | 07:03 |
| aeng | jhesketh, its a bug in Zanata. So the workaround I suggest is to do zanata:pull from existing version, and push it to the new version | 07:03 |
| AJaeger | jhesketh: yes, yolanda helped us. thanks! | 07:03 |
| *** hferenc has joined #openstack-infra | 07:04 | |
| *** ifarkas has joined #openstack-infra | 07:04 | |
| aeng | ah... jhesketh, need a copy of Zanata database | 07:04 |
| jhesketh | aeng: you still need a copy? | 07:04 |
| aeng | yup | 07:04 |
| AJaeger | aeng: so, copying in the webui will not help? (instead of merging) | 07:04 |
| *** claudiub|2 has joined #openstack-infra | 07:04 | |
| aeng | AJaeger, if you're copying to new version, then use it | 07:05 |
| aeng | otherwise, if to already an existing version.. then you will need to do the manual pull push | 07:05 |
| aeng | jhesketh, yes, if you can get a copy of the database dump and send to me email: aeng@redhat.com | 07:05 |
| hferenc | hi all | 07:06 |
| AJaeger | aeng: it's to an already existing version ;( | 07:06 |
| aeng | I need to go now guys... continue tomorrow | 07:06 |
| AJaeger | aeng: thanks | 07:06 |
| *** pcaruana has joined #openstack-infra | 07:06 | |
| *** aeng is now known as aeng_away | 07:06 | |
| *** ihrachys has joined #openstack-infra | 07:06 | |
| jhesketh | aeng_away: I'll take a look.. | 07:06 |
| *** jaosorior has joined #openstack-infra | 07:07 | |
| *** claudiub has quit IRC | 07:08 | |
| *** roxanaghe has joined #openstack-infra | 07:08 | |
| *** rcernin has quit IRC | 07:08 | |
| *** rcernin has joined #openstack-infra | 07:09 | |
| *** asettle has quit IRC | 07:10 | |
| *** fhubik has joined #openstack-infra | 07:12 | |
| *** salv-orlando has joined #openstack-infra | 07:12 | |
| *** roxanaghe has quit IRC | 07:13 | |
| jhesketh | aeng_away: I'm not very familiar with zanata so since you're away until tomorrow I might ask pleia2 to look at exporting the db.. however ping me when you're back and if you haven't got what you need I can help from there :-) | 07:14 |
| *** rlandy has quit IRC | 07:15 | |
| *** Hal has joined #openstack-infra | 07:18 | |
| *** Hal is now known as Guest17603 | 07:19 | |
| hferenc | i'd like to add a test to tempest that depends on an image/instance in which the watchdog device is activated | 07:20 |
| hferenc | how can i make this work? | 07:21 |
| *** jaosorior has quit IRC | 07:22 | |
| *** jaosorior has joined #openstack-infra | 07:22 | |
| ianw | jaosorior: http://logs.openstack.org/55/302055/1/check/gate-tempest-dsvm-platform-fedora23-nv/5b953aa//logs/devstacklog.txt.gz | 07:23 |
| ianw | "no match available for package" | 07:23 |
| ianw | the only thing i can think is maybe we can install the old version from koji. cause if it's gone from the mirror, well it's just not there | 07:23 |
| jaosorior | ianw: do you know when the fix for the rabbitmq package will be available? | 07:24 |
| *** exploreshaifali has quit IRC | 07:24 | |
| ianw | jaosorior: no, just what's in the bug | 07:24 |
| *** mrmartin has joined #openstack-infra | 07:26 | |
| *** esikachev has joined #openstack-infra | 07:26 | |
| *** flepied has quit IRC | 07:27 | |
| *** hferenc has quit IRC | 07:27 | |
| *** mrmartin has quit IRC | 07:28 | |
| ianw | jaosorior: https://review.openstack.org/#/c/302055/2/tools/fixup_stuff.sh might have more luck | 07:29 |
| *** vgridnev has quit IRC | 07:29 | |
| *** |-paul-| has joined #openstack-infra | 07:30 | |
| openstackgerrit | Omer Anson proposed openstack-dev/pbr: Use testrepository's get_base method to decide if test repository exists. https://review.openstack.org/302066 | 07:30 |
| *** Nikolay_St has joined #openstack-infra | 07:31 | |
| *** jlanoux has joined #openstack-infra | 07:32 | |
| jaosorior | ianw: I did this change to test that configuration https://review.openstack.org/#/c/302057/ | 07:32 |
| *** yamamot__ has quit IRC | 07:32 | |
| Nikolay_St | hi infra team. on murano-ci we have in issue with zuul. it described here: https://storyboard.openstack.org/#!/story/2000532 | 07:33 |
| Nikolay_St | I know that sahara team has the same issue | 07:33 |
| Nikolay_St | so, my question is: do you see it before? if yes, how did you fix it | 07:33 |
| *** hferenc has joined #openstack-infra | 07:34 | |
| *** fhubik has quit IRC | 07:34 | |
| *** yamamot__ has joined #openstack-infra | 07:34 | |
| yolanda | AJaeger, i'm back, do you need help? | 07:35 |
| *** yaume has joined #openstack-infra | 07:35 | |
| AJaeger | yolanda: we need a database dump, hope that pleia2 can give that later, she should know the system better - aeng_away is now away (end of day in Australia) | 07:36 |
| AJaeger | yolanda: So, thanks for asking but nothing needed from you now. | 07:36 |
| yolanda | ok just ping me if you need help | 07:36 |
| AJaeger | will do - hope that pleia2 takes over once both aeng_away and pleia2 are awake ;) | 07:37 |
| *** arxcruz has joined #openstack-infra | 07:37 | |
| Nikolay_St | AJaeger: can you take at the issue I mentioned? If you're not too busy | 07:38 |
| *** cody-somerville has quit IRC | 07:43 | |
| *** mrmartin has joined #openstack-infra | 07:43 | |
| openstackgerrit | Yuval Brik proposed openstack-infra/project-config: Smaug: add python3 CI jobs https://review.openstack.org/302072 | 07:44 |
| *** vincentll has joined #openstack-infra | 07:45 | |
| *** jaosorior has quit IRC | 07:47 | |
| *** jaosorior has joined #openstack-infra | 07:47 | |
| *** shardy has joined #openstack-infra | 07:51 | |
| *** dtantsur|afk is now known as dtantsur | 07:51 | |
| openstackgerrit | KATO Tomoyuki proposed openstack-infra/project-config: Enable release notes translation for python projects https://review.openstack.org/300923 | 07:51 |
| *** shardy has quit IRC | 07:52 | |
| *** mrmartin has quit IRC | 07:53 | |
| *** shardy has joined #openstack-infra | 07:53 | |
| *** mrmartin has joined #openstack-infra | 07:54 | |
| openstackgerrit | KATO Tomoyuki proposed openstack-infra/project-config: [i18n] update reno i18n filenames for consistency https://review.openstack.org/302076 | 07:54 |
| *** mrmartin has quit IRC | 07:54 | |
| *** thorst has joined #openstack-infra | 07:55 | |
| *** jordanP has joined #openstack-infra | 07:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:58 | |
| *** sfinucan has joined #openstack-infra | 07:58 | |
| *** achanda has joined #openstack-infra | 08:00 | |
| AJaeger | Nikolay_St: I cannot help with that one | 08:00 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Add missing sudo to deb packaging job https://review.openstack.org/302082 | 08:01 |
| *** thorst has quit IRC | 08:02 | |
| *** daemontool has joined #openstack-infra | 08:04 | |
| *** placko_ has joined #openstack-infra | 08:04 | |
| *** achanda has quit IRC | 08:05 | |
| *** hferenc has quit IRC | 08:06 | |
| *** flepied has joined #openstack-infra | 08:07 | |
| robcresswell | Hello! Over the past few days we've seen constant npm-* test failures in Horizon. One of our contributors pointed out https://github.com/npm/npm/issues/12196 | 08:07 |
| *** akaszuba has joined #openstack-infra | 08:07 | |
| robcresswell | The suggestions there indicte issues with old npm versions. Does anyone happen to know how NPM is installed on the test nodes? | 08:08 |
| Nikolay_St | AJaeger: so, I just noticed that we don't have issue of this type with Jenkins jobs | 08:09 |
| *** yaume_ has joined #openstack-infra | 08:09 | |
| *** hashar has joined #openstack-infra | 08:09 | |
| AJaeger | robcresswell: see krotscheck's comment there - best discuss further once he comes online again | 08:10 |
| *** yaume has quit IRC | 08:10 | |
| robcresswell | AJaeger: Sure, will do. | 08:11 |
| eliqiao | hello, anyone help to check gate, it fails now http://logs.openstack.org/31/301431/2/check/gate-functional-dsvm-magnum-api/7cd902b/console.html | 08:12 |
| AJaeger | eliqiao: is there anything specific in there that makes you suspect a failure on the infra side? If yes, please point it out. | 08:14 |
| *** roxanaghe has joined #openstack-infra | 08:14 | |
| zigo | AJaeger: Hi there! Seems like... https://review.openstack.org/#/c/302082/ <--- We missed a sudo call... | 08:14 |
| AJaeger | zigo: later... I need to do other stuff now, was just signing off until later... | 08:15 |
| zigo | No pb. | 08:16 |
| *** hferenc has joined #openstack-infra | 08:18 | |
| *** roxanaghe has quit IRC | 08:18 | |
| *** exploreshaifali has joined #openstack-infra | 08:18 | |
| * eliqiao checking logs. | 08:19 | |
| *** esikachev has quit IRC | 08:20 | |
| eliqiao | AJaeger: I get root cause, it's in post_script, not infra's fault. | 08:20 |
| eliqiao | AJaeger: thanks, sorry for the bother :( | 08:21 |
| *** jistr has joined #openstack-infra | 08:22 | |
| *** _jcooley has joined #openstack-infra | 08:22 | |
| *** jcooley_ has quit IRC | 08:24 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Add grenade job https://review.openstack.org/301818 | 08:24 |
| *** aeng has joined #openstack-infra | 08:24 | |
| *** netsin has quit IRC | 08:25 | |
| *** gildub has quit IRC | 08:28 | |
| *** esikachev has joined #openstack-infra | 08:29 | |
| *** mrmartin has joined #openstack-infra | 08:29 | |
| *** markvoelker has joined #openstack-infra | 08:30 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Add election alias to governance.openstack.org https://review.openstack.org/292675 | 08:32 |
| *** Thelo has joined #openstack-infra | 08:33 | |
| jaosorior | ianw: I think it did the trick | 08:33 |
| *** vgridnev has joined #openstack-infra | 08:34 | |
| *** markvoelker has quit IRC | 08:35 | |
| *** esikachev has quit IRC | 08:35 | |
| *** e0ne has joined #openstack-infra | 08:35 | |
| *** esikachev has joined #openstack-infra | 08:36 | |
| *** isaacb has quit IRC | 08:36 | |
| *** exploreshaifali has quit IRC | 08:40 | |
| *** salv-orlando has quit IRC | 08:45 | |
| *** derekh has joined #openstack-infra | 08:46 | |
| rossella_s | yolanda, hi! thanks for looking at my patch. I'd appreciate if you could have a look at https://review.openstack.org/298779 too when you have time. clarkb gave a +2 already. thanks a lot!! | 08:49 |
| yolanda | hi rossella_s , sure | 08:50 |
| rossella_s | yolanda, thanks a lot! :) | 08:50 |
| *** kzaitsev_mb has quit IRC | 08:51 | |
| openstackgerrit | Arie Bregman proposed openstack-infra/shade: Add internal_network filter to floating IP creation https://review.openstack.org/302104 | 08:52 |
| *** exploreshaifali has joined #openstack-infra | 08:53 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Fix timeout on crm_resource --wait https://review.openstack.org/301624 | 08:53 |
| *** placko_ has quit IRC | 08:55 | |
| *** vgridnev has quit IRC | 08:57 | |
| openstackgerrit | Merged openstack/diskimage-builder: Remove ssh host keys when using simple init https://review.openstack.org/301982 | 08:58 |
| *** thorst has joined #openstack-infra | 09:00 | |
| *** e0ne has quit IRC | 09:00 | |
| *** sambetts|afk is now known as sambetts | 09:00 | |
| *** rossella_s has quit IRC | 09:03 | |
| *** rossella_s has joined #openstack-infra | 09:03 | |
| *** _amrith_ is now known as amrith | 09:04 | |
| *** abregman is now known as abregman|afk | 09:04 | |
| Thelo | Hi all, when I start zuul I see 2016-04-06 "17:02:46,770 DEBUG zuul.MergeClient: Waiting for gearman" then nothing, does anyone know what I can do? | 09:05 |
| *** electrofelix has joined #openstack-infra | 09:05 | |
| *** thorst has quit IRC | 09:06 | |
| *** skolekonov|away is now known as skolekonov | 09:07 | |
| openstackgerrit | Saverio Proto proposed openstack/diskimage-builder: Debian: dont set always the hostname to debian https://review.openstack.org/205047 | 09:07 |
| *** placko_ has joined #openstack-infra | 09:07 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Fix search_domains when not passing filters https://review.openstack.org/302113 | 09:10 |
| *** Daviey_ has quit IRC | 09:10 | |
| *** dizquierdo has joined #openstack-infra | 09:11 | |
| *** flongo has joined #openstack-infra | 09:11 | |
| *** Daviey has joined #openstack-infra | 09:12 | |
| *** amrith is now known as _amrith_ | 09:12 | |
| *** cdent has joined #openstack-infra | 09:13 | |
| hferenc | hi all, could somebody please help me regarding this (https://bugs.launchpad.net/tempest/+bug/1566719) bug? i already have a scenario test working with a workaround but i'd appreciate if someone could instruct me how to make it work with a test image | 09:14 |
| openstack | Launchpad bug 1566719 in tempest "Test for watchdog related feature is missing" [Undecided,New] - Assigned to Ferenc Horváth (hferenc) | 09:14 |
| *** _degorenko|afk is now known as degorenko | 09:14 | |
| *** salv-orlando has joined #openstack-infra | 09:15 | |
| *** ramishra has joined #openstack-infra | 09:15 | |
| openstackgerrit | vikram.choudhary proposed openstack-infra/project-config: Adding networking-huawei project. https://review.openstack.org/302120 | 09:17 |
| *** _amrith_ is now known as amrith | 09:18 | |
| ianw | jaosorior: hmm, centos7 failed but i don't think it's related | 09:19 |
| *** salv-orlando has quit IRC | 09:19 | |
| *** claudiub has joined #openstack-infra | 09:21 | |
| *** chem has quit IRC | 09:23 | |
| *** claudiub|2 has quit IRC | 09:24 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Refactor, moved mysql part out of init.pp https://review.openstack.org/299661 | 09:25 |
| *** amrith is now known as _amrith_ | 09:25 | |
| *** _amrith_ is now known as amrith | 09:26 | |
| *** amrith is now known as _amrith_ | 09:27 | |
| *** chem has joined #openstack-infra | 09:27 | |
| *** _jcooley has quit IRC | 09:27 | |
| *** _amrith_ is now known as amrith | 09:28 | |
| *** aeng has quit IRC | 09:28 | |
| igorbelikov | hi folks, looks like all nmp-lint jobs are failing right now, is infra aware of that? | 09:29 |
| openstackgerrit | Merged openstack-infra/python-storyboardclient: Update Docs https://review.openstack.org/298910 | 09:29 |
| igorbelikov | https://storyboard.openstack.org/?#!/story/2000541 - looks like a story about the issue | 09:30 |
| *** zhurong has quit IRC | 09:30 | |
| *** flongo has quit IRC | 09:32 | |
| *** zeih has joined #openstack-infra | 09:32 | |
| igorbelikov | yolanda: hi, do you know something about npm-lint issue ^^ by any chance? | 09:33 |
| igorbelikov | http://logs.openstack.org/58/221758/3/check/gate-horizon-npm-run-lint/7e77386/console.html#_2016-04-06_04_57_16_840 - example of fresh failure | 09:33 |
| *** achanda has joined #openstack-infra | 09:33 | |
| yolanda | igorbelikov, no but i can take a look | 09:34 |
| *** ramishra has quit IRC | 09:37 | |
| igorbelikov | yolanda: thanks. this seems to be blocking all projects using npm-lint jobs | 09:38 |
| *** amrith is now known as _amrith_ | 09:38 | |
| *** achanda has quit IRC | 09:38 | |
| *** _amrith_ is now known as amrith | 09:38 | |
| yolanda | strange error | 09:39 |
| yolanda | so it happens on all npm installs? | 09:39 |
| yolanda | can you reproduce locally? | 09:39 |
| *** kushal has quit IRC | 09:39 | |
| yolanda | igorbelikov, do you have some change i can recheck, to hold the node running it? | 09:40 |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Refactor, moved mysql part out of init.pp https://review.openstack.org/299661 | 09:40 |
| *** flongo has joined #openstack-infra | 09:40 | |
| igorbelikov | yolanda: we’re trying to land this one https://review.openstack.org/#/c/295257/, you can recheck it | 09:41 |
| yolanda | thx | 09:41 |
| igorbelikov | yolanda: I didn’t try to reproduce it locally yet, but vkramskikh says he couldn’t reproduce it. | 09:42 |
| yolanda | ok going to hold a node to debug | 09:43 |
| AJaeger | igorbelikov: see scrollback, a known issue | 09:43 |
| AJaeger | igorbelikov: krotscheck has been looking at it... | 09:43 |
| igorbelikov | AJaeger: oh, thanks | 09:43 |
| AJaeger | igorbelikov: see also https://github.com/npm/npm/issues/12196 | 09:44 |
| *** zeih has quit IRC | 09:44 | |
| yolanda | AJaeger, i missed that as well | 09:44 |
| vkramskikh | it says it should be fixed a few hors ago, but the issue is still here :( | 09:44 |
| AJaeger | vkramskikh: read krotscheck's comment about stale caches... | 09:45 |
| *** placko_ has quit IRC | 09:47 | |
| vkramskikh | yet, but yesterday someone said that the jobs are run on fresh new vms so local npm cache shouldn't be an issue. or is it some other cache? | 09:49 |
| *** kushal has joined #openstack-infra | 09:52 | |
| igorbelikov | AJaeger: after reading krotscheck’s comments in scrollback looks like this was supposed to be working as of now. I’m not really sure if this is a cache issue or something is still broken | 09:54 |
| *** vgridnev has joined #openstack-infra | 09:54 | |
| AJaeger | igorbelikov: best to wait for krotscheck to wake up, there's nothing I can do... | 09:55 |
| *** ramishra has joined #openstack-infra | 09:57 | |
| *** BobBall has joined #openstack-infra | 09:57 | |
| persia | Is static.OpenStack.org part of the same CDN? I would expect not, which could be the stale cache. | 09:58 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Temporary disable npm-lint jobs for fuel-ui https://review.openstack.org/302145 | 09:59 |
| *** yamamot__ has quit IRC | 10:00 | |
| igorbelikov | AJaeger: can we use this ^^ as a temporary measure to unblock fuel-ui? | 10:00 |
| yolanda | i'm looking at the conversation and i cannot ensure that this should be fixed, last bits from krotscheck are to retry in a while... | 10:01 |
| rcarrillocruz | igorbelikov: how about making them non-voting? | 10:01 |
| rcarrillocruz | AJaeger: ? | 10:01 |
| *** tobiash has quit IRC | 10:01 | |
| bookwar | rcarrillocruz: we have duplicated lint via third-party Fuel CI anyway | 10:01 |
| yolanda | so we better wait for him, npm is installing from registry.npmjs.org , so we don't have control over it | 10:02 |
| bookwar | and it is voting but not blocking | 10:02 |
| *** salv-orlando has joined #openstack-infra | 10:02 | |
| *** sdake_ has quit IRC | 10:02 | |
| igorbelikov | rcarrillocruz: if I understand correctly we’ll need to put noop as a gate job for this to work, right? | 10:03 |
| *** salv-orlando has quit IRC | 10:03 | |
| igorbelikov | rcarrillocruz: don’t mind to make it non-voting, just not sure how to do it config-wise if it’s the only job | 10:03 |
| rcarrillocruz | i think either noop or non-voting would work | 10:03 |
| *** salv-orlando has joined #openstack-infra | 10:03 | |
| *** yamamot__ has joined #openstack-infra | 10:03 | |
| *** Qiming has quit IRC | 10:04 | |
| *** thorst has joined #openstack-infra | 10:04 | |
| *** tobiash has joined #openstack-infra | 10:05 | |
| *** nadya has joined #openstack-infra | 10:05 | |
| openstackgerrit | Nikolay Starodubtsev proposed openstack-infra/project-config: Switch murano dashboard npm-lint job non-voting https://review.openstack.org/302147 | 10:06 |
| *** e0ne has joined #openstack-infra | 10:06 | |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/puppet-gerrit: Order of the classes parameters is refactored https://review.openstack.org/294728 | 10:07 |
| persia | Switching each project non-voting seems an awkward solution. Is there a way to force the check to use an updated cache for retrieval? | 10:07 |
| yolanda | AJaeger, from our rules, non-voting jobs cannot be on gate, just on check. Do we make exceptions on these cases? | 10:08 |
| *** placko_ has joined #openstack-infra | 10:08 | |
| *** yamamot__ has quit IRC | 10:08 | |
| *** tosky has joined #openstack-infra | 10:09 | |
| *** ekarlso- has quit IRC | 10:09 | |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Switch npm-lint to non-voting for fuel-ui https://review.openstack.org/302145 | 10:09 |
| *** ramishra has quit IRC | 10:09 | |
| *** sdague has joined #openstack-infra | 10:10 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:10 | |
| *** yaume_ has quit IRC | 10:11 | |
| igorbelikov | yolanda, AJaeger I tried to obey rules here ^ :) | 10:11 |
| *** thorst has quit IRC | 10:11 | |
| *** dizquierdo has quit IRC | 10:11 | |
| yolanda | ++ | 10:12 |
| *** yaume has joined #openstack-infra | 10:12 | |
| yolanda | going to send an status update to the channel, until we can talk with krotscheck | 10:12 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Switch npm-lint to non-voting for fuel-ui https://review.openstack.org/302145 | 10:13 |
| *** abregman|afk is now known as abregman | 10:15 | |
| yolanda | #status alert npm lint jobs are failing due to a problem with npm registry. The problem is under investigation, and we will update once the issue is solved. | 10:15 |
| openstackstatus | yolanda: sending alert | 10:15 |
| bookwar | igorbelikov: yes, let's go with this non-voting check, we will be able to debug npm cache issues but will have unblocked merge for the repo | 10:15 |
| *** yaume_ has joined #openstack-infra | 10:17 | |
| -openstackstatus- NOTICE: npm lint jobs are failing due to a problem with npm registry. The problem is under investigation, and we will update once the issue is solved. | 10:17 | |
| *** ChanServ changes topic to "npm lint jobs are failing due to a problem with npm registry. The problem is under investigation, and we will update once the issue is solved." | 10:17 | |
| Zara | yolanda: did you see the conversation in #storyboard about the npm stuff last night? can paste if useful | 10:17 |
| yolanda | Zara nope, i just read the backlog on the infra channel | 10:17 |
| Zara | http://paste.openstack.org/show/493130/ | 10:18 |
| Zara | only skimmed backscroll in here, sounds like the same thing | 10:18 |
| *** salv-orlando has quit IRC | 10:18 | |
| *** claudiub|2 has joined #openstack-infra | 10:18 | |
| yolanda | ah yes, i saw that on infra as well. So wait seems to be the answer? | 10:18 |
| Zara | ahh, yeah | 10:18 |
| yolanda | but well, few hours since that conversation happened, and we are still hitting problems | 10:19 |
| *** mrmartin has quit IRC | 10:20 | |
| openstackstatus | yolanda: finished sending alert | 10:20 |
| *** yaume has quit IRC | 10:20 | |
| *** ramishra has joined #openstack-infra | 10:20 | |
| *** claudiub has quit IRC | 10:21 | |
| *** kushal has quit IRC | 10:21 | |
| hashar | Zara: yolanda npm 1.x is crippled with bunch of bugs. Trusty still comes with 1.3 , Debian Jessie had 1.4.21 + patches | 10:21 |
| *** ekarlso- has joined #openstack-infra | 10:21 | |
| Thelo | Hi I have been away for a while, did anything change with the vendors CIs recheck? When I type recheck Vendor Product in gerrit it I don't see my comment into the history | 10:21 |
| *** ekarlso- has quit IRC | 10:22 | |
| hashar | Zara: yolanda: quoting Wikimedia Javascript guru, node 0.10 shipped with 1.x and you might want to use npm 1.4.29 instead (latest in 1.4.x) ( https://phabricator.wikimedia.org/T124474#2135319 ) | 10:22 |
| *** ekarlso has joined #openstack-infra | 10:22 | |
| *** asettle has joined #openstack-infra | 10:23 | |
| hashar | Zara: yolanda: on Wikimedia CI we end ups pinning npm 2.15.2 for each of nodejs 0.8, 0.10 and 4.3 . That is done by install npm from the .deb package then use npm to update itself with npm install -g npm@2.15.2 relevant puppet magic is https://github.com/wikimedia/operations-puppet/blob/production/modules/contint/manifests/packages/javascript.pp | 10:24 |
| *** asettle has quit IRC | 10:24 | |
| rcarrillocruz | jhesketh: much cleaner putting the empty dict as default | 10:24 |
| rcarrillocruz | thanks :-) | 10:24 |
| jhesketh | no trouble :-) | 10:25 |
| *** mrmartin has joined #openstack-infra | 10:25 | |
| *** salv-orlando has joined #openstack-infra | 10:26 | |
| yolanda | hashar, i'm pretty sure i saw this pattern as well in some infra components | 10:26 |
| yolanda | going to check how is that installed on ubuntu-trusty | 10:27 |
| *** dizquierdo has joined #openstack-infra | 10:27 | |
| *** dizquierdo has quit IRC | 10:28 | |
| hashar | maybe that is done in the diskimage | 10:29 |
| Zara | hashar: ahh, I think I've left sb's npm alone since I started; I don't even know if this error is a problem for sb atm (though probably), so I'll check that. I'm going to ping krotscheck and hide. | 10:29 |
| *** salv-orlando has quit IRC | 10:30 | |
| *** cdent has quit IRC | 10:30 | |
| *** kushal has joined #openstack-infra | 10:30 | |
| igorbelikov | AJaeger: can you check this when you’ll have a minute https://review.openstack.org/#/c/302145/? | 10:30 |
| hashar | ah | 10:30 |
| hashar | yolanda: jenkins/jobs/macros.yaml and it just : apt-get install -y nodejs nodejs-legacy npm | 10:31 |
| hashar | so that is the Jenkins job installing whatever npm is shipped by the distro | 10:31 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Fix search_domains when not passing filters https://review.openstack.org/302113 | 10:31 |
| *** markvoelker has joined #openstack-infra | 10:31 | |
| hashar | Zara: krotscheck: yolanda: I am 100% sure the npm version shipped in Trusty has a bunch of nasty bugs. We have hit them regularly like 2 years ago and first bumped to 1.4.x and eventually to 2.15.2 which works fine on nodejs 0.8, 0.10 and 4.3. Maybe that npm can be included in the diskimage | 10:32 |
| *** mrmartin has quit IRC | 10:34 | |
| *** pblaho has quit IRC | 10:34 | |
| *** achanda has joined #openstack-infra | 10:35 | |
| *** markvoelker has quit IRC | 10:36 | |
| *** weshay has quit IRC | 10:37 | |
| yolanda | hashar, i see an apt-get install npm, but then failing on npm install, so even if we ran npm install to update npm itself it will fail? | 10:40 |
| Zara | hashar: thanks for the tip. I think it's probably more secure if updated, too. I get scared of updating node/npm because I associate it with everything breaking and spending an afternoon deleting everything from node_modules and retrying... | 10:41 |
| *** achanda has quit IRC | 10:41 | |
| Zara | but I think I shall have to face my fear. | 10:41 |
| hashar | yolanda: if npm install npm is hit by the gzip bug, yeah most probably it is going to fail :( | 10:42 |
| hashar | another way is to have the result of npm install to be send to a git repo (like openstack-infra/npm-mirror ) and install it from that (via git-archive or clone) | 10:43 |
| openstackgerrit | Nikolay Starodubtsev proposed openstack-infra/project-config: Switch murano dashboard npm-lint job non-voting https://review.openstack.org/302147 | 10:43 |
| *** tpsilva has joined #openstack-infra | 10:43 | |
| hashar | but that is not so nice to maintain :/ | 10:43 |
| openstackgerrit | Nikolay Starodubtsev proposed openstack-infra/project-config: Switch murano dashboard npm-lint job non-voting https://review.openstack.org/302147 | 10:43 |
| hashar | another way is to backport some npm.deb package and tweak the images apt.configuration to provide a specific apt repo / release and pin the package | 10:44 |
| *** salv-orlando has joined #openstack-infra | 10:47 | |
| *** salv-orlando has quit IRC | 10:48 | |
| yolanda | is there some trusted ppa for that? | 10:48 |
| *** zeih has joined #openstack-infra | 10:51 | |
| *** hieulq has joined #openstack-infra | 10:52 | |
| *** rodrigods has quit IRC | 10:52 | |
| *** rodrigods has joined #openstack-infra | 10:53 | |
| *** tqtran has joined #openstack-infra | 10:56 | |
| *** salv-orlando has joined #openstack-infra | 10:57 | |
| *** salv-orlando has quit IRC | 10:57 | |
| *** salv-orlando has joined #openstack-infra | 10:57 | |
| *** salv-orlando has quit IRC | 10:57 | |
| *** salv-orlando has joined #openstack-infra | 10:58 | |
| *** salv-orlando has quit IRC | 10:58 | |
| *** cdent has joined #openstack-infra | 10:58 | |
| *** claudiub has joined #openstack-infra | 10:59 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Switch ironic-inspector jobs to iPXE https://review.openstack.org/302172 | 11:00 |
| *** exploreshaifali has quit IRC | 11:00 | |
| *** tqtran has quit IRC | 11:01 | |
| *** nadya has quit IRC | 11:01 | |
| *** asettle has joined #openstack-infra | 11:01 | |
| *** claudiub|2 has quit IRC | 11:02 | |
| *** asettle has quit IRC | 11:08 | |
| *** thorst has joined #openstack-infra | 11:09 | |
| *** yamamoto_ has joined #openstack-infra | 11:09 | |
| *** yamamoto_ has quit IRC | 11:10 | |
| *** yamamoto_ has joined #openstack-infra | 11:10 | |
| *** yamamoto_ has quit IRC | 11:10 | |
| jroll | AJaeger: anteaya: is this "all voting jobs must run in gate" documented somewhere? https://review.openstack.org/#/c/300989/3 | 11:13 |
| AJaeger | jroll: that is one of the basic concepts of how we test things in OpenStack ;) | 11:15 |
| AJaeger | jroll: not sure where it's mentioned explicit. | 11:15 |
| AJaeger | jroll: I expect http://docs.openstack.org/infra/zuul/gating.html | 11:15 |
| *** hieulq_ has joined #openstack-infra | 11:15 | |
| jroll | AJaeger: a quick check of zuul/layout shows many projects not following this policy, so I guess I'm curious why it's just now being asked for | 11:16 |
| AJaeger | jroll: show me one or two of your quick check, please | 11:16 |
| *** thorst has quit IRC | 11:16 | |
| jroll | unless I'm missing something | 11:16 |
| jroll | AJaeger: glance: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L6200 | 11:16 |
| hashar | yolanda: re trusted ppa for npm: at Wikimedia we usually backport a .deb package from Ubuntu or Debian and push it to an apt repository | 11:17 |
| AJaeger | jroll: I always review with this assumption, it just might be that I missed something before... | 11:17 |
| jroll | AJaeger: keystone: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L6677 | 11:17 |
| jroll | AJaeger: nova: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L7738 | 11:17 |
| jroll | AJaeger: oslo.messaging https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L8551 | 11:17 |
| AJaeger | gate-tempest-dsvm-full-devstack-plugin-ceph-nv is non-voting - the "-nv" stands for non-voting | 11:17 |
| * jroll will stop there | 11:17 | |
| *** marg7175 has joined #openstack-infra | 11:17 | |
| AJaeger | And non-voting jobs only run in check queue | 11:17 |
| *** hieulq has quit IRC | 11:17 | |
| jroll | gate-rally-dsvm-glance is not | 11:18 |
| AJaeger | Line 1903 has it as non-voting | 11:18 |
| jroll | aha, right, I wonder if all of these are declared that way | 11:18 |
| * jroll doesn't like two ways to do things | 11:19 | |
| AJaeger | jroll: 99 % of these are ;) | 11:19 |
| jroll | sigh, okay | 11:19 |
| dtantsur | jroll, I wonder if we should make the old ramdisk gates non-voting on master, as both are pending deprecation | 11:19 |
| *** claudiub|2 has joined #openstack-infra | 11:20 | |
| AJaeger | jroll: I might have missed one or the other - and there are some where we had some good discussions with the teams and then left them voting in check only. | 11:20 |
| jroll | AJaeger: so ironic has known issues where a slow test node can easily timeout the job, which is why we don't want to run every voting job in the gate queue, it tends to just trigger more false negatives | 11:20 |
| *** thorst has joined #openstack-infra | 11:20 | |
| AJaeger | jroll: I suggest to get this fixed instead - and ironic is not in the integrated gate. | 11:21 |
| jroll | AJaeger: sure, if you have suggestions on avoiding test nodes where nested virt is slow, I'm all ears. the alternative is likely doubling our timeouts. | 11:22 |
| AJaeger | jroll: but I would not -1 a change that gives a proper explanation why the job is only in the check and has broad support from the project team. | 11:22 |
| jroll | AJaeger: well, integrated gate or not, it's frustrating and wasteful to have that many false negatives | 11:22 |
| AJaeger | With a job only in the check queue, you risk merging stuff that does not work anymore. | 11:22 |
| jroll | I understand | 11:22 |
| *** weshay has joined #openstack-infra | 11:22 | |
| *** claudiub has quit IRC | 11:22 | |
| jroll | I believe the benefits outweigh the risks | 11:23 |
| AJaeger | jroll: On the other hand I'll -1 everything that comes without a reason ;) | 11:23 |
| jroll | in any case, I guess in the future we'll add reasons where needed | 11:23 |
| jroll | AJaeger: thanks for the chat | 11:23 |
| AJaeger | jroll: I suggest you discuss later with the infra team, perhaps somebody has ideas on how to help you. | 11:23 |
| *** Qiming has joined #openstack-infra | 11:24 | |
| jroll | AJaeger: oh, I have :) | 11:24 |
| AJaeger | And if in the end you (ironic) decides that the benefirts outweight the risks and propose a documented change... | 11:24 |
| jroll | other than finding some bare metal in the gate, or infinite timeouts, I don't think there are many options | 11:24 |
| AJaeger | "the infra team" -> the rest of the team. Some of them have far more experience than I have ;) | 11:24 |
| AJaeger | jroll: Yeah ;( | 11:25 |
| bookwar | AJaeger: sorry for pinging yet again, https://review.openstack.org/#/c/302145/ we have patch for enabling noop in gate and non-voting lint in check pipelines. Could you please take a look? the npm issue affects us a lot, and we need version bump patch to land to proceed with branching routines | 11:25 |
| *** Goneri has quit IRC | 11:25 | |
| AJaeger | bookwar: propose a revert of the change directly and I'll +2 - I consider it a temporary measure... | 11:25 |
| AJaeger | bookwar: +2ed | 11:27 |
| AJaeger | bbl | 11:27 |
| bookwar | AJaeger: thanks | 11:28 |
| *** dtantsur is now known as dtantsur|brb | 11:28 | |
| *** amotoki has quit IRC | 11:30 | |
| *** zeih has quit IRC | 11:31 | |
| jhesketh | amrith: ping | 11:31 |
| amrith | jhesketh, hello | 11:32 |
| *** zeih has joined #openstack-infra | 11:33 | |
| jhesketh | amrith: just looking at some gerrit json, and it looks like you somehow have two accounts | 11:33 |
| amrith | jhesketh, I do | 11:33 |
| jhesketh | amrith: one lacking a username somehow | 11:33 |
| amrith | yes | 11:33 |
| amrith | that one should not be used | 11:33 |
| amrith | I'll give you the correct ID | 11:33 |
| jhesketh | amrith: that's okay, mostly curious if you know how you ended up with multiple accounts (it broke one of the review stats tooling) | 11:34 |
| amrith | sent you the ID in PM | 11:34 |
| jhesketh | thanks :-) | 11:35 |
| *** ldnunes has joined #openstack-infra | 11:35 | |
| *** gildub has joined #openstack-infra | 11:35 | |
| jhesketh | amrith: do you know how you ended up with two accounts per chance? | 11:36 |
| jhesketh | or why one is missing a username | 11:36 |
| amrith | so people don't use it :) | 11:36 |
| amrith | I do know how I ended up with two accounts but it is a long story and has to do with peculiarities in LP and its mechanism for recovering lost accounts. | 11:37 |
| jhesketh | amrith: okay no worries | 11:37 |
| jhesketh | thanks :-) | 11:37 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors https://review.openstack.org/302192 | 11:38 |
| *** yamamoto has joined #openstack-infra | 11:41 | |
| *** ramishra has quit IRC | 11:41 | |
| *** flongo has quit IRC | 11:41 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch npm-lint to non-voting for fuel-ui https://review.openstack.org/302145 | 11:42 |
| *** gordc has joined #openstack-infra | 11:42 | |
| *** Thelo has quit IRC | 11:44 | |
| *** nadya has joined #openstack-infra | 11:46 | |
| *** yamamoto has quit IRC | 11:48 | |
| *** amrith is now known as _amrith_ | 11:48 | |
| *** roxanaghe has joined #openstack-infra | 11:49 | |
| *** doug-fish has joined #openstack-infra | 11:50 | |
| *** ramishra has joined #openstack-infra | 11:50 | |
| *** hferenc has quit IRC | 11:50 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/300845 | 11:50 |
| openstackgerrit | Merged openstack-infra/project-config: [Manila] Add grenade job https://review.openstack.org/301818 | 11:51 |
| *** |-paul-| has quit IRC | 11:52 | |
| *** ramishra has quit IRC | 11:54 | |
| *** roxanaghe has quit IRC | 11:54 | |
| *** aysyd has joined #openstack-infra | 11:55 | |
| *** amotoki has joined #openstack-infra | 11:56 | |
| *** exploreshaifali has joined #openstack-infra | 11:57 | |
| *** mrmartin has joined #openstack-infra | 11:58 | |
| *** yaume has joined #openstack-infra | 12:03 | |
| *** |-paul-| has joined #openstack-infra | 12:04 | |
| *** hieulq_ has quit IRC | 12:05 | |
| *** rfolco has joined #openstack-infra | 12:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Smaug: add python3 CI jobs https://review.openstack.org/302072 | 12:06 |
| *** yaume_ has quit IRC | 12:07 | |
| *** tiswanso has joined #openstack-infra | 12:07 | |
| *** tiswanso has quit IRC | 12:07 | |
| *** tiswanso has joined #openstack-infra | 12:08 | |
| *** Jeffrey4l has quit IRC | 12:09 | |
| mnaser | my nova patch has failed twice now for "nova-dvsm-full-tempest", "dsvm-tempest-kvm" and "check-tempest-compute" | 12:09 |
| mnaser | and it seemed to be ery quick 2 minute failures, anyway to recheck for those specific CIs only | 12:10 |
| *** salv-orlando has joined #openstack-infra | 12:10 | |
| yolanda | morgan, so the issue i had is that betamax only records one entry per url that matches | 12:10 |
| morgan | Ahhh | 12:11 |
| yolanda | i talked with sigmavirus24 yesterday, and he suggested a fix | 12:11 |
| yolanda | https://github.com/sigmavirus24/betamax/compare/master...yrobla:master | 12:11 |
| morgan | Great! | 12:11 |
| yolanda | i have that working on my shade tests, but my concern here, is that i'm changing old behaviour | 12:11 |
| yolanda | so if there are people relying on a single entry being present from several url calls, this change will break them. i wanted to talk with sigmavirus24 to see what should be the best approach | 12:12 |
| *** Douhet has quit IRC | 12:12 | |
| *** asettle has joined #openstack-infra | 12:12 | |
| *** Douhet has joined #openstack-infra | 12:13 | |
| *** e0ne has quit IRC | 12:13 | |
| fungi | mnaser: recheck reruns all jobs. the situation we'd get with allowing rerunning only specific jobs would be like picking a pin and tumbler lock to get race conditions or other nondeterministic bugs merged (individually retry each job until you get them all to show a passing result) | 12:14 |
| morgan | No one in keystone land will rely on that | 12:14 |
| morgan | Ksa betamax is super new | 12:15 |
| mnaser | fungi: that's fair and i agree with what you're saying. i'll do another recheck and if the failures persist, i'll try to see who i can contact | 12:15 |
| morgan | So, if ya need anything fixed on our end please let me know. We can push that through. | 12:15 |
| *** markvoelker has joined #openstack-infra | 12:15 | |
| fungi | mnaser: nondeterministic failures exposed by the existing jobs unfortunately already encourage people to keep rechecking until they get new nondeterministic bugs merged | 12:15 |
| fungi | so the problem is a self-perpetuating one. we definitely don't want to make it even worse | 12:16 |
| mnaser | fungi: actually, jenkins gave a +1 even though third party CI (Intel NFV and Virtuozzo storage) failed | 12:16 |
| *** asettle has quit IRC | 12:16 | |
| mnaser | are third party CI's non-voting which is why? | 12:16 |
| *** hughsaunders has quit IRC | 12:16 | |
| *** baoli has joined #openstack-infra | 12:16 | |
| mnaser | re: https://review.openstack.org/#/c/300134/ | 12:16 |
| fungi | mnaser: oh, that's somewhat different. yes core reviewers very well may ignore third-party ci system results under various conditions | 12:16 |
| fungi | they're advisory and not at all enforced by the upstream ci system | 12:17 |
| *** e0ne has joined #openstack-infra | 12:17 | |
| mnaser | fungi: i see, thanks, this patch and another one in libvirt will hopefully let us bring back our instances | 12:18 |
| fungi | mnaser: some of them do have specific comment patterns you can leave on your change to get them to rerun their jobs, but it's inconsistent and often not well documented on their part | 12:18 |
| yolanda | morgan, not actually on the keystone fixture, but on the betamax project itself | 12:18 |
| *** hughsaunders has joined #openstack-infra | 12:18 | |
| morgan | Right. But if we can do something to smooth it out let me know | 12:18 |
| mnaser | yeah, i noticed that, i'll stick to generic recheck if anything | 12:18 |
| *** flongo has joined #openstack-infra | 12:19 | |
| yolanda | morgan, one thing i hit, is that the keystone fixture is masking the record modes that are not 'all'. I'm not currently using it, but if i wanted to use new_episodes, for example, the keystone fixture won't support it | 12:19 |
| morgan | Ahh. OK I'll see what we can do about that | 12:20 |
| *** ramishra has joined #openstack-infra | 12:21 | |
| *** baoli_ has joined #openstack-infra | 12:21 | |
| *** trown|outtypewww is now known as trown | 12:23 | |
| *** baoli has quit IRC | 12:24 | |
| *** kgiusti has joined #openstack-infra | 12:25 | |
| *** pradk_ has joined #openstack-infra | 12:26 | |
| *** salv-orlando has quit IRC | 12:26 | |
| *** Qiming_ has joined #openstack-infra | 12:27 | |
| *** Liuqing has joined #openstack-infra | 12:27 | |
| *** Qiming has quit IRC | 12:28 | |
| *** andymaier has joined #openstack-infra | 12:30 | |
| openstackgerrit | Arie Bregman proposed openstack-infra/shade: Add internal_network filter to floating IP creation https://review.openstack.org/302104 | 12:33 |
| *** matt-borland has joined #openstack-infra | 12:34 | |
| *** markmcd has quit IRC | 12:36 | |
| *** markmcd1 has quit IRC | 12:36 | |
| *** Liuqing has quit IRC | 12:37 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks https://review.openstack.org/291393 | 12:37 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Add common bash functions to help track metrics. https://review.openstack.org/291392 | 12:37 |
| *** Liuqing has joined #openstack-infra | 12:38 | |
| *** achanda has joined #openstack-infra | 12:38 | |
| fungi | just a heads up to the team, i'm going to be away from the screen for a good chunk of my afternoon and tomorrow morning (~1500 utc today to ~1700 utc tomorrow) | 12:38 |
| *** flongo has quit IRC | 12:38 | |
| *** julim has joined #openstack-infra | 12:39 | |
| fungi | i'll try to check in periodically, but have to drive a bit and visit with family | 12:39 |
| *** ociuhandu has quit IRC | 12:39 | |
| *** markmcd has joined #openstack-infra | 12:40 | |
| *** Liuqing has quit IRC | 12:42 | |
| *** ramishra has quit IRC | 12:42 | |
| *** Liuqing has joined #openstack-infra | 12:43 | |
| *** achanda has quit IRC | 12:43 | |
| openstackgerrit | Merged openstack-infra/shade: Fix search_domains when not passing filters https://review.openstack.org/302113 | 12:44 |
| *** dprince has joined #openstack-infra | 12:44 | |
| rossella_s | fungi hi! I know you are busy but when you have time could you please check https://review.openstack.org/301255 and https://review.openstack.org/298779 ...it's for the neutron review dashboard, it should be quick, thanks a lot! | 12:45 |
| *** Liuqing has quit IRC | 12:45 | |
| *** julim has quit IRC | 12:45 | |
| *** tiswanso has quit IRC | 12:46 | |
| *** amotoki has quit IRC | 12:46 | |
| *** Liuqing has joined #openstack-infra | 12:49 | |
| *** nwkarsten has joined #openstack-infra | 12:53 | |
| openstackgerrit | Merged openstack/requirements: Add wheel to UNCONSTRAINABLE https://review.openstack.org/301569 | 12:54 |
| *** sfinucan has left #openstack-infra | 12:56 | |
| *** links has quit IRC | 12:56 | |
| *** binbincong has joined #openstack-infra | 12:57 | |
| cbader | anyone else see issues with pypi.python.org returning Hash Sum mismatch that last few days. | 12:58 |
| *** rlandy has joined #openstack-infra | 12:58 | |
| cbader | Sorry apt-get from nova.clouds.archive.ubuntu.com not pypi | 12:59 |
| *** akshai has joined #openstack-infra | 13:01 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Enable src and non-src tinyipa jobs against Ironic https://review.openstack.org/302242 | 13:02 |
| *** ociuhandu has joined #openstack-infra | 13:02 | |
| *** rossella_s has quit IRC | 13:03 | |
| *** rossella_s has joined #openstack-infra | 13:03 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 13:04 |
| openstackgerrit | Arie Bregman proposed openstack-infra/shade: Add internal_network filter to floating IP creation https://review.openstack.org/302104 | 13:06 |
| *** Nikolay_St has quit IRC | 13:06 | |
| *** larsbutler has quit IRC | 13:07 | |
| *** burgerk has joined #openstack-infra | 13:09 | |
| *** salv-orlando has joined #openstack-infra | 13:09 | |
| *** edmondsw has joined #openstack-infra | 13:09 | |
| fungi | cbader: we started running our own package mirrors some time ago because of flaky distro mirrors | 13:10 |
| fungi | so probably not seeing it in our ci jobs now | 13:10 |
| pabelanger | morning | 13:11 |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Enable src and non-src tinyipa jobs against Ironic https://review.openstack.org/302242 | 13:11 |
| yolanda | fungi, pabelanger, do you have more light over npm issue ? people had to disable their npm-lint jobs | 13:12 |
| pabelanger | yolanda: didn't know there was an issue | 13:12 |
| yolanda | reading backlog from krotscheck, it looked as a resolved issue, but we got failures all this morning | 13:12 |
| fungi | yolanda: nope. the javascript ecosystem is a big black box to me | 13:12 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Add routing info to host_info.txt https://review.openstack.org/302246 | 13:12 |
| *** exploreshaifali has quit IRC | 13:13 | |
| yolanda | i sent an alert this morning explaining the issue and we've approved a pair of changes this morning to turn these tests to non-voting | 13:13 |
| fungi | what i've seen mentioned on the internet is basically that npm is terrible and people should stop using it, but i'm sure there are other sides to the story i've simply not seen | 13:13 |
| *** _ari_ has joined #openstack-infra | 13:13 | |
| *** amitgandhinz has joined #openstack-infra | 13:13 | |
| cbader | fungi, I am setting up mirror now. | 13:13 |
| *** kengo_sakai has quit IRC | 13:13 | |
| *** kgalanov has quit IRC | 13:13 | |
| yolanda | i'll wait for krotscheck to come online | 13:14 |
| *** kozhukalov has quit IRC | 13:14 | |
| *** EricGonczer_ has joined #openstack-infra | 13:14 | |
| jroll | fungi: well, it has "sometimes distributes node packages" going for it, but other than that you're pretty much spot on :) | 13:14 |
| *** EricGonczer_ has quit IRC | 13:14 | |
| pabelanger | cbader: see https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh for how we setup our nodepool images | 13:15 |
| *** sdake has joined #openstack-infra | 13:15 | |
| *** kgalanov has joined #openstack-infra | 13:15 | |
| *** kozhukalov has joined #openstack-infra | 13:16 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 13:16 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with periodic jobs https://review.openstack.org/297038 | 13:16 |
| fungi | yolanda: the explanation yesterday was that clearing your local npm cache should solve it, except we start with an empty cache presumably? | 13:16 |
| *** mriedem has joined #openstack-infra | 13:16 | |
| fungi | at least i'm not aware of any mechanism currently building an npm cache on our worker images anyway | 13:16 |
| yolanda | fungi, but we've seen that errors on new ubuntu-trusty nodes, so how can cache affect to that? | 13:16 |
| fungi | yolanda: basically the same question i have, yes | 13:17 |
| cbader | pabelanger, great thanks | 13:17 |
| Kiall | Any project-config core have a few mins to review a repo addition? https://review.openstack.org/#/c/301219/ | 13:17 |
| *** salv-orlando has quit IRC | 13:17 | |
| fungi | yolanda: is the upstream npm bug report still getting any updates? new details maybe? i haven't looked | 13:17 |
| *** Goneri has joined #openstack-infra | 13:18 | |
| pabelanger | infra-root: EmilienM: I've just approved the puppet-jobs migration to centos-7. I'll be monitoring this morning. | 13:18 |
| yolanda | i looked this morning, but didn't get anything clear, just mentions to that cache problem | 13:18 |
| *** zz_dimtruck is now known as dimtruck | 13:18 | |
| *** sdake_ has joined #openstack-infra | 13:19 | |
| fungi | oh, looks like their issue tracker is twitter? | 13:19 |
| pabelanger | neat | 13:19 |
| *** sdake has quit IRC | 13:19 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Enable src and non-src tinyipa jobs against Ironic https://review.openstack.org/302242 | 13:20 |
| fungi | at least that's the implication i get from krotscheck's comment, "...consulted with @npm/registry-team about the current CDN configuration for the primary npm registry..." | 13:20 |
| fungi | i assume the @ there means twitterspace | 13:20 |
| *** jsavak has joined #openstack-infra | 13:21 | |
| *** dprince has quit IRC | 13:22 | |
| *** bdemers has joined #openstack-infra | 13:23 | |
| *** tiswanso has joined #openstack-infra | 13:23 | |
| SamYaple | fungi: idk could be systemd stuff | 13:23 |
| SamYaple | hard to tell | 13:23 |
| *** sdake_ has quit IRC | 13:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Add missing sudo to deb packaging job https://review.openstack.org/302082 | 13:24 |
| *** Liuqing has quit IRC | 13:26 | |
| *** cloudtrainme has joined #openstack-infra | 13:26 | |
| *** annegentle has joined #openstack-infra | 13:27 | |
| *** sdake has joined #openstack-infra | 13:27 | |
| fungi | https://github.com/npm/npm/issues/12196 looks like the best source of details on this problem | 13:27 |
| openstackgerrit | Merged openstack-infra/project-config: Migrate puppet jobs to centos-7 dib https://review.openstack.org/301251 | 13:27 |
| yolanda | yep, i read the thread with krotscheck, but i'm not fully understanding it, and how that can apply to our case | 13:28 |
| *** dprince has joined #openstack-infra | 13:29 | |
| fungi | looks like it's still a waiting game | 13:29 |
| fungi | the caches, as i interpret those last comments, refer to caching within the cdn not in npm | 13:29 |
| fungi | no indication of how long they tell their cdn (whoever is providing that) to cache data | 13:30 |
| *** |-paul-| has quit IRC | 13:32 | |
| *** Liuqing has joined #openstack-infra | 13:33 | |
| *** alex_xu has quit IRC | 13:33 | |
| krotscheck | githubspace, actually | 13:35 |
| *** alex_xu has joined #openstack-infra | 13:35 | |
| krotscheck | I'm not quite online yet, but will be in about 20 minutes. GOod to know that things aren't working yet. | 13:36 |
| fungi | aha. i didn't know @ also meant github | 13:36 |
| * krotscheck made some progres on our registry mirror, but he ran into some problems around baby bedtime so punted to today. | 13:36 | |
| *** ddieterly has joined #openstack-infra | 13:36 | |
| *** jamesmcarthur has joined #openstack-infra | 13:36 | |
| *** mtanino has joined #openstack-infra | 13:37 | |
| *** roxanaghe has joined #openstack-infra | 13:37 | |
| fungi | though this highlights that we should probably consider making newer npm standard on our images the same way we make newer pip standard. the ecosystems for non-distro package installers move at a much faster clip (understandably) than long-term stable linux distros do | 13:38 |
| fungi | and as such, so do their respective package managers | 13:38 |
| *** matrohon has joined #openstack-infra | 13:39 | |
| yolanda | hi krotscheck, good morning. So you confirm is a matter of waiting? also, there have been some interesting conversation about how to properly consume latest versions of npm | 13:39 |
| *** amotoki has joined #openstack-infra | 13:39 | |
| yolanda | although i think in this particular case, even npm install to update npm properly should have failed | 13:39 |
| fungi | yolanda: we already have a pattern established for bootstrapping newer npm, but yes that's the impression i get as well (using old npm to install newer npm was hitting this same bug) | 13:40 |
| *** pblaho has joined #openstack-infra | 13:41 | |
| *** eharney has quit IRC | 13:41 | |
| *** roxanaghe has quit IRC | 13:42 | |
| *** esker has quit IRC | 13:42 | |
| *** Jeffrey4l has joined #openstack-infra | 13:42 | |
| krotscheck | The correct way to land newer npm is, I believe, to change the 'manage-registry' flag on this patch to true: https://review.openstack.org/#/c/242161/ | 13:42 |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Refactor, moved mysql part out of init.pp https://review.openstack.org/299661 | 13:43 |
| *** tonytan4ever has joined #openstack-infra | 13:43 | |
| krotscheck | That will, at least temporarily, make npm come from the nodesource registry, which is likely to stick around long enough for us to move the npm jobs to xenial | 13:43 |
| krotscheck | xenial has LTS npm on it. | 13:43 |
| krotscheck | Also, it'll buy us enough time to get our mirror working, which should shield us further from issues like this. | 13:44 |
| *** _amrith_ is now known as amrith | 13:44 | |
| *** jaosorior has quit IRC | 13:44 | |
| fungi | we do have xenial(-ish!) nodes booting and available in nodepool now, but yeah we should avoid relying on them for any voting jobs until xenial is officially released, at a minimum | 13:45 |
| pabelanger | ++ | 13:45 |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Refactor, moved mysql part out of init.pp https://review.openstack.org/299661 | 13:46 |
| pabelanger | clarkb: when your online today, could use your help setting up experimental devstack-gate job for xenial | 13:46 |
| fungi | i expect though, barring any significant issues with actually running those jobs on xenial, it will be very quick to make that transition once xenial is official | 13:46 |
| krotscheck | Am I go to update the single-use-slave with sane npm? I can add switches that will make it behave properly for xenial | 13:46 |
| krotscheck | i.e. is that something we can reach consensus on before I dive in? | 13:46 |
| *** dimtruck is now known as zz_dimtruck | 13:47 | |
| fungi | krotscheck: i'm taking a look at the nodejs puppet module now to see if i can decipher the things it does. want to make sure it's not going to create a mess on our images for non-node jobs to trip over | 13:47 |
| krotscheck | fungi You go tit | 13:47 |
| fungi | but in general i'm in favor of the idea of providing newer language-specific package managers on our images than the distros are themselves packaging, as long as there are no conflicts | 13:48 |
| *** pradk_ has quit IRC | 13:49 | |
| *** cbader has quit IRC | 13:50 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Refactor, moved mysql part out of init.pp https://review.openstack.org/299661 | 13:51 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on initial json https://review.openstack.org/302269 | 13:52 |
| *** ramishra has joined #openstack-infra | 13:52 | |
| Shrews | rcarrillocruz: yolanda: jhesketh: have a moment to discuss https://review.openstack.org/302113 ? | 13:52 |
| *** smarcet has joined #openstack-infra | 13:53 | |
| *** matrohon has quit IRC | 13:53 | |
| krotscheck | Ok, building a trusty node with the new puppet manifest now, will let you know if it updates node. | 13:53 |
| yolanda | Shrews, sure | 13:53 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on initial json https://review.openstack.org/302269 | 13:53 |
| *** matrohon has joined #openstack-infra | 13:53 | |
| krotscheck | Anyone have a quick switch somewhere for "Hey only run this on xenial?" | 13:53 |
| fungi | krotscheck: put "node: ubuntu-xenial" in the job definition | 13:54 |
| *** eharney has joined #openstack-infra | 13:55 | |
| Shrews | rcarrillocruz: yolanda: jhesketh: so, first, i LOVE that you all are reviewing shade more! thanks so much for that. but that should not have merged so quickly | 13:55 |
| pabelanger | fungi: krotscheck: then check experimental in the review | 13:55 |
| fungi | krotscheck: if there's an experimental job for it, then leaving a "check ex... | 13:55 |
| fungi | yeah, what pabelanger said | 13:55 |
| *** ddieterly is now known as ddieterly[away] | 13:55 | |
| *** yfried has joined #openstack-infra | 13:56 | |
| Shrews | rcarrillocruz: yolanda: jhesketh: first of all, the default value of {} instead of None is dangerous, and that is not the best way to handle that change | 13:56 |
| *** ramishra has quit IRC | 13:56 | |
| rcarrillocruz | so what's your opinion on that? the first patchset or something else | 13:57 |
| *** knikolla has joined #openstack-infra | 13:57 | |
| rcarrillocruz | " | 13:57 |
| rcarrillocruz | ? | 13:57 |
| Shrews | rcarrillocruz: yolanda: jhesketh: secondly, and there is no way you could have known this without either myself or mordred first telling you, the gate-shade-dsvm-ansible-dev-nv should ALWAYS pass even though it is non-voting | 13:57 |
| *** nadya has quit IRC | 13:57 | |
| *** gildub has quit IRC | 13:58 | |
| yolanda | Shrews, why is that set to nv? | 13:58 |
| Shrews | looks like it timed out, but someone should have done a recheck to verify | 13:58 |
| mordred | yolanda: because we don't have a symmetric gate on the ansible repo | 13:58 |
| Shrews | yolanda: that job tests shade against the unreleased development version of ansible. it helps us catch when we break ansible, or when they break us | 13:58 |
| *** oanson has quit IRC | 13:58 | |
| Shrews | yolanda: the other ansible job tests against the released version of ansible | 13:59 |
| *** mriedem is now known as mriedem_meeting | 14:00 | |
| Shrews | yolanda: jhesketh: lastly, that change needs to be verified against the os_keystone_domain module since shade does not yet test that one automatically (that stuff usually falls to me since i'm more familiar with that area) | 14:00 |
| *** xarses_ has quit IRC | 14:01 | |
| pabelanger | okay, JJB manually running on jenkins04-07 for puppet-jobs update | 14:01 |
| Zara | jroll: oh, meant to say the other day and forgot, thanks for the vim-etherpad link, I think I'm going to keep an eye on how that project evolves. :) | 14:01 |
| Shrews | yolanda: jhesketh: rcarrillocruz: this is all context that mordred and myself have done a bad job of sharing with other folks, so this is really a bad on us | 14:01 |
| *** ddieterly[away] is now known as ddieterly | 14:02 | |
| *** cdent has quit IRC | 14:02 | |
| yolanda | Shrews, mordred, good to know, i was not aware of that review guidelines | 14:02 |
| *** claudiub|2 has quit IRC | 14:02 | |
| *** electrofelix has quit IRC | 14:02 | |
| Shrews | yolanda: yeah, unless you're dealing with shade everyday, hard to know | 14:03 |
| Shrews | rcarrillocruz: i'd first want to know why that breaks for that one API method and works everywhere else | 14:04 |
| yolanda | Shrews, i wonder if we can share that guidelines somewhere in shade, maybe in some README section? | 14:04 |
| Shrews | yolanda: great idea! i will work on that for us | 14:05 |
| fungi | i always forget how to navigate from something like https://forge.puppetlabs.com/puppetlabs/nodejs to a browseable source code repository. whoever designs puppetlabs webpages doesn't design them for the way my brain works, i think | 14:05 |
| fungi | nevermind! it's apparently a smallish link at the top for "Project URL" | 14:06 |
| *** Guest17603 has quit IRC | 14:06 | |
| fungi | no wonder i never can remember... such a vague term | 14:06 |
| rcarrillocruz | Shrews: | 14:06 |
| *** jamesmcarthur has quit IRC | 14:06 | |
| rcarrillocruz | http://paste.openstack.org/show/493163/ | 14:06 |
| Shrews | rcarrillocruz: ok. maybe the way to handle that is 'if filters is None: filters={}' or something. defaulting in the param list is bad news if it ever gets modified | 14:08 |
| *** mtanino has quit IRC | 14:08 | |
| fungi | krotscheck: looks like that's going to break on xenial maybe? i don't see anything that'll match 16.04 at https://github.com/voxpupuli/puppet-nodejs/blob/master/manifests/init.pp#L27 | 14:08 |
| rcarrillocruz | that's what i put in the first patchset if you see the change | 14:08 |
| *** Hal has joined #openstack-infra | 14:08 | |
| rcarrillocruz | you want me to amend | 14:08 |
| rcarrillocruz | ? | 14:08 |
| rcarrillocruz | i mean | 14:08 |
| rcarrillocruz | pushing other change | 14:08 |
| rcarrillocruz | as that's already merged | 14:08 |
| *** Hal is now known as Guest99028 | 14:08 | |
| *** jamesmcarthur has joined #openstack-infra | 14:09 | |
| Shrews | rcarrillocruz: i think mordred might have a fix ready to push? check with him. i need to leave now for a doctor appointment | 14:09 |
| *** zz_dimtruck is now known as dimtruck | 14:09 | |
| rcarrillocruz | sure | 14:10 |
| fungi | krotscheck: i guess if we want to use a nodesource package we may need to explicitly set the repo for it | 14:10 |
| fungi | for xenial anyway | 14:10 |
| krotscheck | fungi: That's why I wanted that in-puppet swtich. For xenial, just use apt repo. | 14:11 |
| fungi | oh! right, you did already say that ;) | 14:11 |
| *** eharney has quit IRC | 14:11 | |
| *** jaosorior has joined #openstack-infra | 14:12 | |
| krotscheck | It wouls be nice if the instructions on this puppet module actually worked. | 14:14 |
| *** rbrndt_ has joined #openstack-infra | 14:15 | |
| fungi | krotscheck: the other thing i'm concerned about is that it looks like this will indefinitely add the nodesource package repo to the apt sources list/yum registry and also add its package signing keys as trusted | 14:15 |
| fungi | so as a means of bootstrapping a new npm executable, it's a bit heavyweight | 14:15 |
| fungi | we'd likely need to go in behind it and remove/invalidate those so jobs didn't inadvertently install packages from them | 14:16 |
| krotscheck | fungi: I understand. Is that enough of a concern to have us investigate other options? | 14:16 |
| fungi | krotscheck: not necessarily. it's just a question of whether the cleanup is more or less effort than reinventing the minimal bits we need | 14:16 |
| fungi | in an npm dib element for example | 14:16 |
| krotscheck | fungi: That's juju I am unfamiliar with, and unlikely to be able to unblock the gate in a reasonable timeframe. Is this something someone else would be willing to do? | 14:17 |
| krotscheck | Because, well, horizon mitaka builds are borked. | 14:18 |
| fungi | krotscheck: oh, you're looking at this as a means of unblocking jobs for the current npm cdn issue? well, it would also require waiting for new worker images everywhere | 14:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Don't use singleton dicts unwittingly https://review.openstack.org/302280 | 14:19 |
| fungi | puppet is only being applied at image creation | 14:19 |
| mordred | Shrews, rcarrillocruz: ^^ | 14:19 |
| krotscheck | fungi: I get that. So right now, I have horizon, fuel, murano, storyboard, and the entire suite of horizon plugins blocked on npm jobs. | 14:19 |
| krotscheck | That's objective 1 | 14:19 |
| krotscheck | Because mitaka | 14:20 |
| *** cbader has joined #openstack-infra | 14:20 | |
| fungi | krotscheck: any clue what the cache invalidation timeout is on the npm cdn? | 14:20 |
| krotscheck | fungi: It was _supposed_ to be 24 hours. | 14:20 |
| fungi | looks like they only rolled back the encoding change 19 hours ago, so in theory should clear within the next 5? | 14:21 |
| *** Murad has joined #openstack-infra | 14:21 | |
| *** xarses_ has joined #openstack-infra | 14:21 | |
| fungi | at least that's the timeline i'm able to piece together from the github issue | 14:21 |
| sc` | pabelanger: thanks for kicking a recheck of that job. still failing on python-requests, which is being pulled in as a dependency of keystone, it seems | 14:22 |
| Murad | I had my new project commit merged, what is next : https://review.openstack.org/#/c/295260/ | 14:22 |
| *** ramishra has joined #openstack-infra | 14:22 | |
| pabelanger | sc`: Ya, it does look to be unrelated to the migration to centos-7 | 14:22 |
| *** cdent has joined #openstack-infra | 14:22 | |
| sc` | i don't have visibility to see from where that package is coming, if it's the deps repo or the mitaka repo itself | 14:23 |
| *** dizquierdo has joined #openstack-infra | 14:24 | |
| *** zeih has quit IRC | 14:26 | |
| *** achanda has joined #openstack-infra | 14:26 | |
| *** mrmartin has quit IRC | 14:26 | |
| *** ramishra has quit IRC | 14:27 | |
| *** mrmartin has joined #openstack-infra | 14:28 | |
| *** claudiub has joined #openstack-infra | 14:28 | |
| AJaeger | Murad: you're now at http://docs.openstack.org/infra/manual/creators.html#wait-here - please ask an admin to add you to the ACLs and then you can add others for approving changes. You can start sending changes in already... | 14:29 |
| *** diana_clarke has left #openstack-infra | 14:29 | |
| *** yamamoto has joined #openstack-infra | 14:29 | |
| *** mrmartin has quit IRC | 14:30 | |
| *** eharney has joined #openstack-infra | 14:30 | |
| krotscheck | fungi: Still waiting to hear back from npm support on specifics for their CDN configuration. | 14:31 |
| *** matrohon has quit IRC | 14:31 | |
| *** mriedem_meeting is now known as mriedem | 14:31 | |
| *** Swami has joined #openstack-infra | 14:31 | |
| *** Goneri has quit IRC | 14:32 | |
| *** Swami_ has joined #openstack-infra | 14:32 | |
| *** Swami_ has quit IRC | 14:32 | |
| *** Swami has quit IRC | 14:32 | |
| fungi | krotscheck: could the npm-install macro pull down newer npm debs instead? | 14:32 |
| fungi | that might be a faster path to resolution | 14:32 |
| *** Swami has joined #openstack-infra | 14:32 | |
| krotscheck | ......maaaaybe. | 14:33 |
| fungi | are we only running that on trusty now anyway? | 14:33 |
| krotscheck | yep | 14:33 |
| fungi | if so the logic could be remarkably simple | 14:33 |
| *** cloudtrainme has quit IRC | 14:33 | |
| fungi | wget the deb from nodesource, dpkg -i it, then proceed with the other stuff | 14:33 |
| fungi | or, though a few more steps, add the nodesource apt repo to the sources list, add their signing key to apt-key, apt-get update and then apt-get install npm or whatever | 14:34 |
| *** zul has quit IRC | 14:35 | |
| pabelanger | 2016-04-06 14:31:29.528 | liberasurecode-devel x86_64 1.0.9-3.el7 openstack-liberty 19 k | 14:35 |
| fungi | point being, npm-install is already doing these things as root, so the overall job logic doesn't change much | 14:35 |
| pabelanger | fungi: ^ does that look right to you for bindep-fallback? | 14:35 |
| fungi | (if at all) | 14:35 |
| *** flwang1 has quit IRC | 14:35 | |
| fungi | pabelanger: i need more context | 14:36 |
| pabelanger | https://jenkins01.openstack.org/job/gate-project-config-bindep-fallback-centos-7/1/consoleFull | 14:36 |
| *** zeih has joined #openstack-infra | 14:36 | |
| pabelanger | fungi: sorry, trying to figure out why our bindep jobs pass but seeing failures on puppet for liberasurecode-devel | 14:37 |
| pabelanger | looks like we have a mismatch of yum repos | 14:37 |
| fungi | pabelanger: where's the failure example? | 14:37 |
| pabelanger | https://jenkins06.openstack.org/job/gate-puppet-cinder-puppet-syntax-3-centos-7/1/console | 14:37 |
| fungi | that job looks like it passes | 14:37 |
| pabelanger | that is the failure | 14:37 |
| fungi | yep, i see it | 14:38 |
| pabelanger | so, looking at http://nodepool.openstack.org/dib.centos-7.log we seem to be subscribed to openstack-liberty repo | 14:38 |
| fungi | my guess is we're missing whatever provides liberasurecode-devel on centos-7 as an available rhn package "channel" or whatever yum calls it | 14:38 |
| pabelanger | which, I assume is RDO? | 14:38 |
| fungi | ahh, yep i bet it is | 14:38 |
| pabelanger | Didn't realize we add them about dib time | 14:39 |
| *** eharney has quit IRC | 14:39 | |
| pabelanger | https://github.com/openstack-infra/system-config/blob/master/install_puppet.sh#L155 | 14:39 |
| pabelanger | guess that is the line | 14:39 |
| *** eharney has joined #openstack-infra | 14:40 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Use nodesource scripts to install more recent version of node https://review.openstack.org/302292 | 14:40 |
| krotscheck | fungi ^^ | 14:40 |
| krotscheck | That works locally. No idea if it works on infra. | 14:40 |
| *** electrofelix has joined #openstack-infra | 14:40 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:40 | |
| fungi | krotscheck: at the moment, it couldn't work _worse_ than the current jobs ;) | 14:41 |
| *** sdake_ has joined #openstack-infra | 14:42 | |
| *** sdake has quit IRC | 14:42 | |
| AJaeger | krotscheck: I would prefer downloading a deb instead of running sudo bash -e | 14:42 |
| mugsie | anyone have a spare +2 for https://review.openstack.org/301219 :) ? it is a basic repo addition | 14:42 |
| *** sdake_ is now known as sdake | 14:43 | |
| *** placko_ has quit IRC | 14:43 | |
| pabelanger | fungi: okay, I see the issue. ---> Package rdo-release.noarch 0:liberty-2 will be erased | 14:43 |
| krotscheck | AJaeger: I'm iterating. | 14:43 |
| pabelanger | which contains the package | 14:43 |
| AJaeger | mugsie: it needs designate PTL+1 | 14:43 |
| pabelanger | fungi: so, the question is, is bindep smart enough to match on a installed repo? | 14:43 |
| mugsie | AJaeger: I am the designate PTL | 14:43 |
| fungi | pabelanger: bindep just calls the package manager utilities and parses their output | 14:44 |
| AJaeger | mugsie: I only checked reviewers but not committer, sorry ;( | 14:44 |
| krotscheck | Do our nodes have apt-transport-https, lsb-release and curl installed? | 14:44 |
| AJaeger | mugsie: ok, let me double check and search for a spare +2 ;) | 14:44 |
| mugsie | AJaeger: thanks :) | 14:44 |
| AJaeger | mugsie: I added a spare +A on top of it;) | 14:45 |
| pabelanger | fungi: okay, maybe call bindep before they uninstall rdo package | 14:45 |
| *** Murad has quit IRC | 14:46 | |
| *** dprince has quit IRC | 14:46 | |
| fungi | pabelanger: yeah, right now bindep doesn't know anything about adding/removing package channels nor does it inherently have multi-phase logic | 14:46 |
| *** Goneri has joined #openstack-infra | 14:46 | |
| *** thiagop has quit IRC | 14:47 | |
| *** zeih has quit IRC | 14:47 | |
| mugsie | AJaeger: ty | 14:47 |
| fungi | pabelanger: but also, i wonder why we're removing rdo if the job needs packages from it? | 14:47 |
| *** zeih has joined #openstack-infra | 14:47 | |
| fungi | pabelanger: or rather, where does the rdo removal occur? in the job itself? | 14:47 |
| *** sdake_ has joined #openstack-infra | 14:48 | |
| *** ericksonsantos has quit IRC | 14:48 | |
| *** flwang has joined #openstack-infra | 14:49 | |
| fungi | krotscheck: so the jobs themselves don't need nodejs-legacy (they call nodejs by its debian-given name instead)? | 14:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: puppet-prepare-node after bindep for puppet jobs https://review.openstack.org/302298 | 14:50 |
| pabelanger | fungi: that's the current proposed fix ^ | 14:50 |
| krotscheck | fungi: I've found that to be inconsistent - occasionally, an npm package will invoke node directly, and will bork because it doesn't realize it's on debian | 14:50 |
| pabelanger | fungi: Ya, they clean up the repos so puppet can manage them | 14:50 |
| *** thiagop has joined #openstack-infra | 14:50 | |
| krotscheck | The nodesource package installs node on the right path | 14:50 |
| *** sdake has quit IRC | 14:51 | |
| *** salv-orlando has joined #openstack-infra | 14:51 | |
| fungi | krotscheck: only asking because you're removing the installation of the nodejs-legacy package | 14:51 |
| *** zeih has quit IRC | 14:51 | |
| *** sc68cal has joined #openstack-infra | 14:51 | |
| *** nwkarsten has quit IRC | 14:51 | |
| *** vikram___ has joined #openstack-infra | 14:51 | |
| krotscheck | fungi: Right - the nodejs-legacy package is there only because the debian packages have that name conflic tthing going. The nodesource ones (and I think the xenial ones) don't. | 14:51 |
| *** jsavak has quit IRC | 14:52 | |
| *** nwkarsten has joined #openstack-infra | 14:52 | |
| *** nadya has joined #openstack-infra | 14:52 | |
| *** esikachev has quit IRC | 14:52 | |
| krotscheck | I have a new revision, just a minute while I test it. | 14:52 |
| *** jsavak has joined #openstack-infra | 14:53 | |
| *** achanda has quit IRC | 14:53 | |
| *** jaosorior has quit IRC | 14:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Create openstack/designate-tempest repository https://review.openstack.org/301219 | 14:54 |
| fungi | krotscheck: cool. also i just finished verifying we only run that macro on jobs using ubuntu-trusty workers, and so aren't running it in anything sensitive | 14:54 |
| *** mtanino has joined #openstack-infra | 14:54 | |
| pabelanger | sc`: okay, here is the issue with python-requests: https://github.com/openstack-infra/system-config/blob/master/install_puppet.sh#L106 | 14:55 |
| pabelanger | we do pip install requests into centos-7 dibs now | 14:55 |
| fungi | about the worst upshot would be the possibility of a nodesource.com compromise causing us to embed a backdoor when building tarballs of javascript-based projects for publication | 14:55 |
| pabelanger | sc`: I will need to talk with ianw about this, but a work-around is to pip uninstall requests first | 14:56 |
| *** nwkarsten has quit IRC | 14:56 | |
| sc` | so i've been told | 14:56 |
| pabelanger | fungi: clarkb: I could use some thoughts about ^ | 14:56 |
| krotscheck | ANyone know why sudo echo 'deb https://deb.nodesource.com/node_4.x ${DISTRO} main' > /etc/apt/sources.list.d/nodesource.list is giving me permission denied? | 14:56 |
| vikram___ | AJaeger: Ping | 14:57 |
| pabelanger | fungi: clarkb: as we many need to rollback to devstack-centos7 because of it | 14:57 |
| sc` | pip uninstalling stuff seems like a messy solution to a not very straightforward problem | 14:57 |
| vikram___ | AJaeger: Could you please provide some pointers on your comment "Use git-filter to create a repository that contains just the files you want. Oslo-incubator has some example on how to do it." | 14:57 |
| jroll | Zara: re: etherpad-vim, glad you liked it :) | 14:57 |
| jroll | krotscheck: because your > is not running as root, try echo 'foo' | sudo tee /etc/apt/sources.list.d/nodesource.list | 14:58 |
| fungi | vikram___: the tool you're looking for is "git filter-branch" | 14:58 |
| pabelanger | sc`: well, that is for devstack jobs. I think we could move that into devstack some how but need to hash it out | 14:58 |
| vikram___ | fungi: Can you plz provide an example | 14:58 |
| *** esikachev has joined #openstack-infra | 14:58 | |
| krotscheck | jroll: Thans, will try | 14:58 |
| jroll | np | 14:58 |
| vikram___ | fungi: Or comment here "https://review.openstack.org/#/c/268726/" | 14:59 |
| fungi | vikram___: i don't have time to hold anyone's hand learning git right now, but i think there was an example of doing it in our puppet module split spec | 14:59 |
| *** yamahata has quit IRC | 14:59 | |
| *** eharney has quit IRC | 14:59 | |
| sc` | pabelanger: indeed. if it makes more sense to pip uninstall for my one use case, then fine | 14:59 |
| vikram___ | fungi: Thanks for your helpp | 14:59 |
| *** roxanaghe has joined #openstack-infra | 15:00 | |
| fungi | vikram___: actually, for that we ended up using "git subtree split" instead, but example is at http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet-modules.html | 15:00 |
| openstackgerrit | Travis Tripp proposed openstack-infra/project-config: Make searchlight-ui javascript tests non-voting https://review.openstack.org/302303 | 15:00 |
| sc` | i'm not doing anything particularly fancy with how i'm getting to the python-requests problem | 15:00 |
| *** TravT has joined #openstack-infra | 15:00 | |
| *** pfallenop has quit IRC | 15:00 | |
| vikram___ | fungi: I found it ;) | 15:00 |
| fungi | pabelanger: so is the underlying problem that some base system tools in newer red hat distros depend on the python requests module being installed? | 15:01 |
| *** Jeffrey4l has quit IRC | 15:01 | |
| *** Nikolay_St has joined #openstack-infra | 15:01 | |
| *** roxanaghe has quit IRC | 15:01 | |
| *** zul has joined #openstack-infra | 15:01 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 15:02 | |
| pleia2 | AJaeger: I haven't read full backscroll, but still need a db dump? | 15:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Uninstall python-requests for puppet-jobs https://review.openstack.org/302305 | 15:02 |
| fungi | pabelanger: just trying to figure out why we're installing requests on the images at all | 15:02 |
| pabelanger | fungi: ^ current proposed work around | 15:02 |
| pabelanger | fungi: but yes, ianw add python-requests as a pip install for some devstack issues I believe | 15:02 |
| fungi | pabelanger: on ubuntu images we go to a lot of trouble to make sure we're not installing python libs in the images, particularly requests | 15:03 |
| *** Qiming_ has quit IRC | 15:03 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Use nodesource scripts to install more recent version of node https://review.openstack.org/302292 | 15:03 |
| krotscheck | AJaeger, fungi: Version 2, now with less trusting of foreign bash scripts ^^ | 15:03 |
| pabelanger | fungi: Since it is specific to devstack, I think we can make the argument to move that logic into devstack some how | 15:03 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:03 | |
| AJaeger | pleia2: yes, aeng_away still needs a db dump | 15:03 |
| fungi | pabelanger: is it impacting jobs right now, or only some proposed changes? | 15:03 |
| AJaeger | pleia2: if you could provide one, please, it would be great. | 15:04 |
| *** burgerk_ has joined #openstack-infra | 15:04 | |
| sc` | fungi: python-requests impacts chef integration | 15:04 |
| pabelanger | fungi: yes, we have some gate failures for chef and puppet because of it, now that we are using centos-7 dibs | 15:04 |
| pabelanger | fungi: I have proposed work arounds for puppet currently | 15:04 |
| pabelanger | fungi: otherwise, we'll have to roll back to devstack-centos7 | 15:04 |
| sc` | that same workaround would work for me | 15:05 |
| sc` | unless there's a Better Way to resolve it | 15:05 |
| pabelanger | better way is to remove it from base image, but that is not the fastest as we need new image builds and uploads | 15:05 |
| * sc` nods | 15:06 | |
| *** burgerk has quit IRC | 15:06 | |
| pabelanger | or we rollback to devstack-centos7 | 15:06 |
| pleia2 | AJaeger: ok, on it | 15:06 |
| AJaeger | pleia2: all pushing to zanata currently fails - so I'd like us to have this fixed today. So, please check with aeng_away whether he has enough information and what we can do to become functional again. Should we restart zanata? Or do we need to do something else? | 15:06 |
| *** roxanaghe has joined #openstack-infra | 15:06 | |
| *** julim has joined #openstack-infra | 15:06 | |
| AJaeger | pleia2: thanks. I'll be offline now for a couple of hours, will check in later briefly. | 15:07 |
| *** nadya has quit IRC | 15:07 | |
| *** Jeffrey4l has joined #openstack-infra | 15:07 | |
| *** roxanaghe has quit IRC | 15:08 | |
| anteaya | AJaeger: pleia2 can I suggest you create an etherpad or something for this issue? | 15:08 |
| pabelanger | https://review.openstack.org/#/c/285876/ is for python-requests on centos7 | 15:08 |
| *** jogo has quit IRC | 15:09 | |
| anteaya | AJaeger: pleia2 I'm reading the backscroll on this and I can understand both the confusion and why it is taking so long to address | 15:09 |
| pabelanger | which is actually fedora-23 | 15:09 |
| *** ramishra has joined #openstack-infra | 15:09 | |
| anteaya | AJaeger: pleia2 I don't know if an etherpad will help but might be better than the current system? | 15:09 |
| openstackgerrit | Merged openstack-infra/reviewstats: Filter out members without usernames https://review.openstack.org/302005 | 15:09 |
| openstackgerrit | Merged openstack-infra/reviewstats: Indicate the branch when generating stable checks https://review.openstack.org/302006 | 15:10 |
| *** jogo has joined #openstack-infra | 15:10 | |
| *** cloudtrainme has joined #openstack-infra | 15:10 | |
| *** yaume has quit IRC | 15:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Uninstall python-requests for puppet-jobs https://review.openstack.org/302305 | 15:11 |
| *** cdent has quit IRC | 15:12 | |
| *** Jeffrey4l has quit IRC | 15:12 | |
| pabelanger | yolanda: do you mind helping land https://review.openstack.org/#/c/302298/ and https://review.openstack.org/#/c/302305/ | 15:12 |
| yolanda | sure | 15:12 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Install more recent version of node https://review.openstack.org/302292 | 15:12 |
| krotscheck | Sorry, had to update the commit message | 15:12 |
| *** roxanaghe has joined #openstack-infra | 15:12 | |
| krotscheck | Because really | 15:12 |
| *** eharney has joined #openstack-infra | 15:14 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: tripleo.sh --delorean-build handle oslo.* package builds https://review.openstack.org/302312 | 15:14 |
| *** pblaho has quit IRC | 15:14 | |
| *** vgridnev has quit IRC | 15:15 | |
| *** Liuqing has quit IRC | 15:15 | |
| *** nwkarsten has joined #openstack-infra | 15:16 | |
| *** vgridnev has joined #openstack-infra | 15:17 | |
| *** sridhar_ram has joined #openstack-infra | 15:18 | |
| *** jcoufal has joined #openstack-infra | 15:19 | |
| *** dtantsur|brb is now known as dtantsur | 15:19 | |
| *** asettle has joined #openstack-infra | 15:19 | |
| *** annegentle has quit IRC | 15:19 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Install more recent version of node https://review.openstack.org/302292 | 15:20 |
| *** bpokorny has joined #openstack-infra | 15:20 | |
| *** annegentle has joined #openstack-infra | 15:21 | |
| *** sridhar_ram has quit IRC | 15:21 | |
| *** claudiub has quit IRC | 15:21 | |
| *** kaisers has left #openstack-infra | 15:22 | |
| *** salv-orlando has quit IRC | 15:22 | |
| *** esikachev has quit IRC | 15:23 | |
| *** Jeffrey4l has joined #openstack-infra | 15:24 | |
| *** mylu has joined #openstack-infra | 15:24 | |
| *** roxanaghe has quit IRC | 15:24 | |
| *** asettle has quit IRC | 15:24 | |
| *** EricGonczer_ has joined #openstack-infra | 15:25 | |
| *** Nikolay_St has quit IRC | 15:26 | |
| *** armax has joined #openstack-infra | 15:27 | |
| *** zul_ has joined #openstack-infra | 15:27 | |
| *** roxanaghe has joined #openstack-infra | 15:27 | |
| *** zul has quit IRC | 15:27 | |
| *** jsavak has quit IRC | 15:29 | |
| *** sdake has joined #openstack-infra | 15:29 | |
| *** esker has joined #openstack-infra | 15:29 | |
| *** jsavak has joined #openstack-infra | 15:30 | |
| *** sdake_ has quit IRC | 15:30 | |
| *** bpokorny has quit IRC | 15:30 | |
| *** nwkarsten has quit IRC | 15:31 | |
| *** shardy has quit IRC | 15:31 | |
| *** nwkarsten has joined #openstack-infra | 15:31 | |
| *** esikachev has joined #openstack-infra | 15:32 | |
| *** rajinir has joined #openstack-infra | 15:32 | |
| pleia2 | anteaya: we have all the data we need already, we're just struggling with time zones, nothing an etherpad will help with | 15:34 |
| *** annegentle has quit IRC | 15:34 | |
| krotscheck | fungi: The build failed with "ImportError: No module named requestsexceptions" | 15:34 |
| *** annegentle has joined #openstack-infra | 15:35 | |
| fungi | okay, need to get on the road, but will check in some later today and tomorrow morning, just not planning to be at the computer again much until around 1800 utc tomorrow | 15:35 |
| krotscheck | ok | 15:35 |
| openstackgerrit | Merged openstack-infra/project-config: puppet-prepare-node after bindep for puppet jobs https://review.openstack.org/302298 | 15:35 |
| *** nwkarsten has quit IRC | 15:35 | |
| *** mikelk has quit IRC | 15:36 | |
| *** ddieterly is now known as ddieterly[away] | 15:36 | |
| anteaya | pleia2: oh okay, I've been following the conversation in backscroll, but you know best | 15:37 |
| *** lakshmiS has joined #openstack-infra | 15:37 | |
| anteaya | pleia2: hope you find a solution | 15:38 |
| anteaya | fungi: happy family | 15:39 |
| *** sdake_ has joined #openstack-infra | 15:40 | |
| *** ddieterly[away] is now known as ddieterly | 15:40 | |
| Zara | Hi, I'm trying to read gerrit stream events using a python script that imports gerritlib. I'm getting an error, `paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted` | 15:40 |
| Zara | I may be doing absolutely everything wrong | 15:40 |
| *** cloudtrainme has quit IRC | 15:40 | |
| Zara | for now, I'm just using my own account, and I only want to read events, not send any sort of change to gerrit | 15:41 |
| *** Swami_ has joined #openstack-infra | 15:41 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Add routing info to host_info.txt https://review.openstack.org/302246 | 15:41 |
| *** armax_ has joined #openstack-infra | 15:41 | |
| Zara | if there's a guide to this kind of thing anywhere, could someone point me at it? | 15:41 |
| *** armax has quit IRC | 15:42 | |
| *** armax_ is now known as armax | 15:42 | |
| *** julim has quit IRC | 15:43 | |
| *** sdake has quit IRC | 15:43 | |
| *** cloudtrainme has joined #openstack-infra | 15:43 | |
| anteaya | Zara: hi, paramiko is a python ssh library: http://www.paramiko.org/ | 15:43 |
| *** nwkarsten has joined #openstack-infra | 15:43 | |
| pleia2 | Zara: broadly speaking, this might help you http://docs.openstack.org/infra/system-config/third_party.html#reading-the-event-stream | 15:44 |
| anteaya | Zara: I think if you look at where paramiko is looking for your ssh keys and in what form that may address the error | 15:44 |
| *** jogo has quit IRC | 15:44 | |
| *** bpokorny has joined #openstack-infra | 15:44 | |
| pleia2 | Zara: I've seen that error when ssh key has a password but it's not being provided | 15:45 |
| *** Swami has quit IRC | 15:45 | |
| anteaya | Zara: here is the key handling section of the paramiko docs: http://docs.paramiko.org/en/1.16/api/keys.html | 15:46 |
| *** jogo has joined #openstack-infra | 15:46 | |
| Zara | pleia2: yeah, I think that might be it and it's just not giving me an opportunity to provide one | 15:46 |
| anteaya | Zara: and yes as pleia2 points out reading the docs on the gerrit event stream would be great too | 15:46 |
| Zara | anteaya: thank you :) in this case I have a thing importing gerritlib which then imports paramiko, so it's all a bit abstracted... fun! :D | 15:46 |
| *** vincentll has quit IRC | 15:47 | |
| anteaya | wooooo | 15:47 |
| anteaya | sounds like the abstracts can't find all the key bits they want | 15:47 |
| thingee | Zara: from_private_key_file() does appear to have a password param | 15:47 |
| *** nwkarsten has quit IRC | 15:48 | |
| Zara | (I had a look at the docs, but either I missed the bit which would help with this specific problem, or it wasn't there. I'll look again, though, just in case) | 15:48 |
| Zara | thingee: ahh, hm, where is that used? (a gerritlib thing or a paramiko thing?) | 15:49 |
| Zara | seems like a paramiko thing | 15:49 |
| pleia2 | thingee: I seem to have you to thank for the git&gerrit lunch being so well organized according to our documentation, so thank you :) (I had a call with Tamara yesterday to make sure everything looked good, it did!) | 15:49 |
| thingee | Zara: the docs anteaya linked to mention it. | 15:49 |
| *** yamahata has joined #openstack-infra | 15:49 | |
| *** pblaho has joined #openstack-infra | 15:50 | |
| *** andymaier has quit IRC | 15:50 | |
| thingee | pleia2: oh neat, happy to hear that documentation is being used more. | 15:50 |
| thingee | pleia2: yeah mentoring group is using it, upstream university, etc. happy it's helpful | 15:51 |
| *** nwkarsten has joined #openstack-infra | 15:51 | |
| *** Sukhdev_ has joined #openstack-infra | 15:51 | |
| *** claudiub has joined #openstack-infra | 15:52 | |
| *** salv-orlando has joined #openstack-infra | 15:53 | |
| *** dizquierdo has quit IRC | 15:53 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use split cirros image for ping test https://review.openstack.org/301951 | 15:53 |
| *** tonytan4ever has quit IRC | 15:54 | |
| *** fhubik has joined #openstack-infra | 15:54 | |
| *** tonytan4ever has joined #openstack-infra | 15:54 | |
| *** nwkarsten has quit IRC | 15:56 | |
| *** electrofelix has quit IRC | 15:56 | |
| *** salv-orlando has quit IRC | 15:56 | |
| *** Rockyg has joined #openstack-infra | 15:56 | |
| *** xyang1 has joined #openstack-infra | 15:57 | |
| Zara | ah, so I think gerritlib uses paramiko's ssh client, but not its ssh agent (which is the bit containing from_private_key_file | 15:57 |
| openstackgerrit | Merged openstack-infra/project-config: Uninstall python-requests for puppet-jobs https://review.openstack.org/302305 | 15:57 |
| *** salv-orlando has joined #openstack-infra | 15:58 | |
| Zara | I'm going by https://github.com/openstack-infra/gerritlib/blob/master/gerritlib/gerrit.py | 15:58 |
| *** asettle has joined #openstack-infra | 15:58 | |
| *** yamahata has quit IRC | 15:59 | |
| *** yamahata has joined #openstack-infra | 15:59 | |
| *** fedexo has joined #openstack-infra | 15:59 | |
| pabelanger | stepping out for some lunch, relaunched JJB to grab the 2 puppet fixes we just merged. Will be back shortly | 15:59 |
| *** claudiub|2 has joined #openstack-infra | 16:00 | |
| *** jamesmcarthur has quit IRC | 16:02 | |
| *** kzaitsev_mb has quit IRC | 16:02 | |
| *** tphummel has joined #openstack-infra | 16:02 | |
| *** claudiub has quit IRC | 16:02 | |
| *** jsavak has quit IRC | 16:03 | |
| *** cdent has joined #openstack-infra | 16:03 | |
| *** pblaho has quit IRC | 16:03 | |
| *** asettle has quit IRC | 16:03 | |
| Zara | so I still don't really know what to do; seems like there must be some option more sensible than locally patching gerritlib (and I'm not sure how to go about that anyway) | 16:05 |
| *** jsavak has joined #openstack-infra | 16:05 | |
| *** austin81 has joined #openstack-infra | 16:06 | |
| *** apoorvad has joined #openstack-infra | 16:06 | |
| *** erlon has joined #openstack-infra | 16:07 | |
| *** cloudtrainme has quit IRC | 16:08 | |
| *** claudiub has joined #openstack-infra | 16:08 | |
| *** claudiub|2 has quit IRC | 16:08 | |
| *** annegentle has quit IRC | 16:09 | |
| *** akaszuba has quit IRC | 16:10 | |
| *** jistr has quit IRC | 16:10 | |
| anteaya | Zara: I wish I had something helpful to tell you, but the best I have is "go you!" | 16:10 |
| anteaya | Zara: go you! | 16:10 |
| *** kzaitsev_mb has joined #openstack-infra | 16:11 | |
| *** Nikolay_St has joined #openstack-infra | 16:12 | |
| Zara | \o/ | 16:12 |
| *** tphummel has quit IRC | 16:12 | |
| *** jamesmcarthur has joined #openstack-infra | 16:12 | |
| *** EricGonc_ has joined #openstack-infra | 16:12 | |
| * Zara is tempted to hardcode password in paramiko/client.py ; what could possibly go wrong? =D | 16:13 | |
| Zara | more seriously, I wonder how other people do it | 16:14 |
| thingee | Zara: when I was running a ci, I just didn't use a password for the ssh key | 16:14 |
| *** annegentle has joined #openstack-infra | 16:14 | |
| *** mylu has quit IRC | 16:14 | |
| morgan | thingee: tha is what i've done as well. | 16:14 |
| zaro | morning | 16:14 |
| zaro | anybody know why i get failure without a log to view? http://logs.openstack.org/96/300596/6/check/gate-infra-puppet-apply-devstack-centos7/f0a2746/ | 16:16 |
| *** EricGonczer_ has quit IRC | 16:16 | |
| anteaya | morning zaro | 16:16 |
| anteaya | zaro: how old is the failure? | 16:17 |
| anteaya | zaro: if it is beyond a certain age (30 days?) the logs are deleted to make space for new logs | 16:17 |
| *** smarcet has quit IRC | 16:18 | |
| zaro | it's from yesterday | 16:19 |
| zaro | https://review.openstack.org/#/c/300596/6 | 16:19 |
| anteaya | zaro: that is odd | 16:19 |
| zaro | rechecked and same result both times | 16:19 |
| *** HeOS has quit IRC | 16:19 | |
| Zara | thingee: hm, how would I try that? (if it's sensible to try it) | 16:20 |
| *** mylu has joined #openstack-infra | 16:20 | |
| anteaya | zaro: I'd say the puppet-apply-devstack-centos7 job isn't doing the write thing with logging on failure | 16:20 |
| anteaya | zaro: can you find any other failures for that job recently? | 16:21 |
| anteaya | Zara: create a key with no password, then tell your script to use the new password-less key | 16:21 |
| *** hashar has quit IRC | 16:23 | |
| Shrews | mordred: your search_domains() fix has also failed the ansible-nv test with another timeout on the os_object test. methinks there to be a problem somewhere. i'll dig into that. | 16:23 |
| *** keedya has joined #openstack-infra | 16:24 | |
| *** kzaitsev_mb has quit IRC | 16:24 | |
| *** nadya has joined #openstack-infra | 16:25 | |
| *** vgridnev has quit IRC | 16:25 | |
| *** EricGonc_ has quit IRC | 16:25 | |
| *** woodster_ has joined #openstack-infra | 16:25 | |
| *** EricGonczer_ has joined #openstack-infra | 16:25 | |
| Zara | hm... 'gerritymcgerritson' seemed like a sensible name, but I'm probably going to grow to hate typing it | 16:25 |
| Zara | anteaya: thank you :) | 16:25 |
| *** asettle has joined #openstack-infra | 16:26 | |
| Zara | I guess I should add that to my gerrit account, too | 16:26 |
| anteaya | Zara: ha ha ha | 16:27 |
| anteaya | Zara: yes whatever ssh key you create is what the script will use to verify against your gerrit account | 16:28 |
| *** roxanaghe has quit IRC | 16:28 | |
| *** daemontool has quit IRC | 16:28 | |
| *** jistr has joined #openstack-infra | 16:28 | |
| *** yfried has quit IRC | 16:29 | |
| wznoinsk | hi infra | 16:29 |
| anteaya | hello wznoinsk | 16:29 |
| Zara | IT WORKED | 16:30 |
| Zara | thank you all | 16:30 |
| anteaya | Zara: yay! | 16:30 |
| anteaya | Zara: well done | 16:30 |
| wznoinsk | I see openstack/designate-tempest-plugin in https://git.openstack.org/cgit/openstack-infra/project-config/plain/gerrit/projects.yaml but not https://git.openstack.org/cgit/openstack/?q=designate | 16:30 |
| wznoinsk | hi anteaya | 16:30 |
| *** trown is now known as trown|lunch | 16:30 | |
| *** mrmartin has joined #openstack-infra | 16:30 | |
| Zara | I should've asked hours ago, but yaaaaaaaaay! | 16:30 |
| *** asettle has quit IRC | 16:30 | |
| *** jlanoux has quit IRC | 16:30 | |
| *** ddieterly is now known as ddieterly[away] | 16:30 | |
| anteaya | Zara: many of us were asleep hours ago | 16:31 |
| wznoinsk | is it waiting for someone to upload that repo to git.openstack.org or automatic refresh/creation hasn't run yet? | 16:31 |
| Zara | true :) | 16:31 |
| *** claudiub has quit IRC | 16:31 | |
| zaro | anteaya: rechecked a different puppet-gerrit change so we'll see, https://review.openstack.org/#/c/300596/6 | 16:32 |
| anteaya | wznoinsk: I have no idea | 16:32 |
| zaro | anteaya: opps, meant this one https://review.openstack.org/#/c/300572 | 16:32 |
| anteaya | wznoinsk: there doesn't seem to be a designate-tempest-plugin repo in git.o.o | 16:32 |
| wznoinsk | anteaya, I know it breaks nodepool dib | 16:32 |
| *** jtomasek has quit IRC | 16:33 | |
| zaro | anteaya: actually, rechecked both. | 16:33 |
| anteaya | wznoinsk: I would suggest you use git blame on project-config to find the patch that added it while there is no repo | 16:33 |
| anteaya | wznoinsk: if the repo doesn't exist, the patch to add it shouldn't have been merged | 16:33 |
| *** Swami_ has quit IRC | 16:34 | |
| *** jistr is now known as jistr|off | 16:34 | |
| *** dprince has joined #openstack-infra | 16:34 | |
| anteaya | zaro: I get devstack-centos7 logs for 300572, but it did pass | 16:34 |
| wznoinsk | anteaya, https://review.openstack.org/#/c/301219/4/gerrit/projects.yaml | 16:34 |
| anteaya | perhaps there is a logging issue on failure | 16:34 |
| *** roxanaghe has joined #openstack-infra | 16:34 | |
| wznoinsk | anteaya, so should I chase the approver or the commiter? | 16:35 |
| *** zul_ has quit IRC | 16:35 | |
| anteaya | no if there is a patch that created the repo that merged but the repo wasn't created that is a problem with manage projects | 16:35 |
| anteaya | wznoinsk: you need to find one of the infra roots to find out why that project is supposed to exist on git.o.o but doesn't yet | 16:36 |
| *** jordanP has quit IRC | 16:36 | |
| zaro | anteaya: that one last ran on 4/5 so i wanted to recheck to make sure it still provides logs | 16:37 |
| anteaya | wznoinsk: now I understand your original question | 16:37 |
| anteaya | wznoinsk: your original question was that a project creation patch was merged but the project hasn't yet been created | 16:37 |
| wznoinsk | anteaya, I was thinking the change will need to be approved first for the project to appear under git repo list but it's probably the other way round as per what you say | 16:38 |
| anteaya | zaro: yup, fair enough | 16:38 |
| anteaya | wznoinsk: no, the project creation patch has to merge | 16:38 |
| anteaya | then the project has to be created | 16:38 |
| anteaya | step two is automated and obviously hasn't worked yet | 16:38 |
| anteaya | someone needs to address step two | 16:38 |
| *** claudiub has joined #openstack-infra | 16:39 | |
| anteaya | your approach and ascertion that it breaks dib led me to believe that it had been added to a job without having a project creation patch | 16:39 |
| *** annegentle has quit IRC | 16:39 | |
| anteaya | yours was the third conversation I was juggling at the time | 16:39 |
| *** doug-fish has quit IRC | 16:40 | |
| anteaya | context really helps me | 16:40 |
| *** doug-fish has joined #openstack-infra | 16:40 | |
| pleia2 | wznoinsk: that patch is still in the post queue, search for c21baae in status.openstack.org/zuul | 16:40 |
| wznoinsk | anteaya, it breaks dib as it takes the list of projects from gerrit/projects.yaml which is not what git.openstack.org has (dib tries to clone these) | 16:40 |
| *** annegentle has joined #openstack-infra | 16:41 | |
| wznoinsk | pleia2, ah ok, thanks | 16:41 |
| anteaya | pleia2: thanks | 16:41 |
| anteaya | wznoinsk: that may be, but the breaking of dib isn't the issue, the issue is that the patch merged but the repo isnt' created | 16:42 |
| anteaya | wznoinsk: which as pleia2 pointed out is due to the patch being in post | 16:42 |
| clarkb | so much to catch up on | 16:43 |
| pleia2 | hopefully that's what it is, but I'm reluctant to go poking around while it's still making it's way through jobs | 16:43 |
| *** scheuran has quit IRC | 16:43 | |
| pleia2 | clarkb: good morning | 16:43 |
| wznoinsk | anteaya, my original message was clear imho, dib imapcted was the result of it | 16:43 |
| *** claudiub has quit IRC | 16:44 | |
| *** amitgandhinz has quit IRC | 16:44 | |
| anteaya | morning clarkb | 16:44 |
| *** roxanaghe has quit IRC | 16:44 | |
| clarkb | morning | 16:44 |
| anteaya | pleia2: yes it would make sense to me that if the patch is in post, the repo wouldn't be created yet | 16:44 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config: Ironic grenade jobs do not work until after Mitaka https://review.openstack.org/300082 | 16:45 |
| *** doug-fish has quit IRC | 16:45 | |
| *** roxanaghe has joined #openstack-infra | 16:46 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 16:47 |
| *** tosky has quit IRC | 16:47 | |
| *** Swami has joined #openstack-infra | 16:47 | |
| *** amotoki has quit IRC | 16:48 | |
| cinerama | bkero: ping | 16:49 |
| pleia2 | cinerama: I haven't seen you in forever, hello! | 16:49 |
| cinerama | hi pleia2! | 16:49 |
| *** claudiub has joined #openstack-infra | 16:50 | |
| *** derekh has quit IRC | 16:50 | |
| clarkb | pabelanger: is there a tl;dr on the centos stuff? | 16:51 |
| *** zul has joined #openstack-infra | 16:51 | |
| *** ddieterly[away] is now known as ddieterly | 16:52 | |
| *** vikram___ has quit IRC | 16:52 | |
| *** achanda has joined #openstack-infra | 16:53 | |
| *** vgridnev has joined #openstack-infra | 16:53 | |
| *** burgerk has joined #openstack-infra | 16:53 | |
| bkero | cinerama: pong | 16:54 |
| clarkb | pabelanger: I am still trying to catch up on everything but ya can help with xenial things | 16:54 |
| *** claudiub has quit IRC | 16:54 | |
| *** mylu has quit IRC | 16:54 | |
| *** sc68cal has quit IRC | 16:55 | |
| sc` | clarkb: python-requests from rpm conflicts with a pip installed one from the base image. a proposed workaround is to do the pip uninstall and let the rpm pave over | 16:55 |
| *** burgerk_ has quit IRC | 16:55 | |
| clarkb | why not use the pip one? | 16:56 |
| *** EricGonczer_ has quit IRC | 16:56 | |
| clarkb | I think thats what we do on the other images | 16:56 |
| *** scheuran has joined #openstack-infra | 16:56 | |
| sc` | because yum pulls in the package as a hard dependency | 16:56 |
| *** scheuran has quit IRC | 16:56 | |
| sc` | can't break that dependency | 16:57 |
| clarkb | ya I think the dep is still there but the pip one is actually on disk? | 16:57 |
| cinerama | bkero, hey, it would be great if you can update https://review.openstack.org/#/c/294760/ | 16:58 |
| clarkb | the underlying issue is that unlike debian rhel does not have pip intall to /usr/local | 16:58 |
| sc` | yup. so the package comes in and tries to lay stuff down, but finds someone else moved in, and barfs | 16:58 |
| sc` | it affects both puppet and chef | 16:58 |
| *** ayoung has quit IRC | 16:59 | |
| cinerama | bkero: would like to get something that addresses this issue landed | 16:59 |
| *** davideagnello has joined #openstack-infra | 16:59 | |
| bkero | cinerama: Ah, sure thing | 16:59 |
| clarkb | sc`: oh is it because yum/requiests are trying to update? | 17:00 |
| clarkb | sc`: we install pip after yum is already installed | 17:00 |
| sc` | yum is trying to install over what pip installed | 17:00 |
| clarkb | right I guess the question is why | 17:00 |
| sc` | and can't, so builds fail | 17:00 |
| clarkb | it should all already be installed and functional | 17:01 |
| clarkb | because we do the pip install after the base system install | 17:01 |
| *** skolekonov is now known as skolekonov|away | 17:01 | |
| *** pfallenop has joined #openstack-infra | 17:01 | |
| *** pfallenop has joined #openstack-infra | 17:01 | |
| sc` | seems to be directory vs. symlink. i think the better solution is to remove it from the base image entirely, like pabelanger suggested, but it'll take more time | 17:01 |
| clarkb | remove what from the base image? | 17:02 |
| sc` | python-requests | 17:02 |
| *** Guest99028 has quit IRC | 17:02 | |
| clarkb | hrm is that not a dep of yum? | 17:02 |
| sc` | it comes in as a package from rdo, as a dependency of keystone | 17:02 |
| clarkb | basically I am confused because we have put a fair bit of work into making this all function, but maybe everywhere else uses pip for requests too so never notices | 17:03 |
| *** rossella_s has quit IRC | 17:03 | |
| clarkb | including on the devstack-centos7 images | 17:03 |
| clarkb | which should be functionally equivalent to the centos-7 images when it comes to that? | 17:03 |
| sc` | i don't have visibility to see just which repo holds python-requests, but jenkins build logs don't lie | 17:03 |
| clarkb | I guess I may need to compare the output of some filesytem listings | 17:03 |
| clarkb | I don't disagree that it is failing | 17:03 |
| *** rossella_s has joined #openstack-infra | 17:03 | |
| clarkb | I juts want to understand why before we go making changes | 17:03 |
| sc` | of course | 17:03 |
| *** piet has joined #openstack-infra | 17:03 | |
| sc` | a workaround for puppet has already been merged | 17:04 |
| clarkb | are they forcefully reinstalling requests via yum? | 17:04 |
| *** jsavak has quit IRC | 17:04 | |
| sc` | likely, since puppet and rdo share the same package set for centos | 17:04 |
| *** fedexo has quit IRC | 17:05 | |
| sc` | puppet and chef | 17:05 |
| sc` | sorry | 17:05 |
| sc` | E_COFFEE | 17:05 |
| *** ramishra has quit IRC | 17:05 | |
| *** esker has quit IRC | 17:05 | |
| clarkb | also we enable rdo in our package builds because it is the only way to get packages that we need to cache for openstack like ovs | 17:05 |
| clarkb | I think there was confusion on why we enable that, and that is why | 17:05 |
| *** esker has joined #openstack-infra | 17:06 | |
| *** vgridnev has quit IRC | 17:06 | |
| *** nwkarsten has joined #openstack-infra | 17:06 | |
| *** angdraug has joined #openstack-infra | 17:08 | |
| *** dprince has quit IRC | 17:08 | |
| *** dprince has joined #openstack-infra | 17:09 | |
| *** vgridnev has joined #openstack-infra | 17:09 | |
| *** vgridnev has quit IRC | 17:09 | |
| *** mrmartin has quit IRC | 17:10 | |
| *** sputnik13 has joined #openstack-infra | 17:11 | |
| clarkb | infra-root does anyone have an active ssh connection to nodepool? | 17:11 |
| pleia2 | I do not | 17:11 |
| pleia2 | and stepping away for a few hours for activities with family in town | 17:11 |
| clarkb | https://review.openstack.org/#/c/301982/2 broke us when the script moved from install.d to cleanup.d beacuse cleanup.d does not run in the chroot | 17:11 |
| pleia2 | :\ | 17:12 |
| openstackgerrit | Tim Burke proposed openstack-infra/infra-manual: Fix "contributing on behalf of a company" formatting https://review.openstack.org/302372 | 17:12 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix ssh key cleanup to run in chroot https://review.openstack.org/302373 | 17:12 |
| *** salv-orlando has quit IRC | 17:12 | |
| krotscheck | I've got +2 and a passing gate on our NPM build issue, can I get a second core to look at this? https://review.openstack.org/#/c/302292/ | 17:12 |
| krotscheck | clarkb, AJaeger ^^? | 17:12 |
| clarkb | I think our options are force reboot via nova and hope ubuntu + nova agent initing is smart enough to give us new keys, stop instance and recover mode drop some files in, make a new instance (excet we have no quota to do that) | 17:12 |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/300845 | 17:13 |
| clarkb | the service is running (for now) so I don't think its end of the world important but this should be corrected today | 17:13 |
| greghaynes | sorry! | 17:13 |
| greghaynes | thats pretty bad | 17:13 |
| greghaynes | I W+1'd the fix | 17:13 |
| bkero | greghaynes: does pkg-map handle distro versions? | 17:13 |
| bkero | or is it just family (debian) and distro (ubuntu)? | 17:13 |
| *** sputnik13 has quit IRC | 17:13 | |
| *** jtomasek has joined #openstack-infra | 17:14 | |
| clarkb | krotscheck: somewhat busy trying to reason about recovering nodepool.openstack.org | 17:14 |
| greghaynes | it does not do distro versions AFAIK | 17:14 |
| clarkb | hopefully someone has a persistent ssh connection up | 17:14 |
| bkero | ok | 17:14 |
| krotscheck | clarkb: No prob. | 17:14 |
| *** sputnik13 has joined #openstack-infra | 17:14 | |
| bkero | greghaynes: consider it a feature request that I'd like to get around to implementing | 17:14 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:14 | |
| * krotscheck hopes that AJaeger takes a look. He usually has good comments. | 17:14 | |
| greghaynes | bkero: Yea, that seems reasonable to me | 17:15 |
| *** jsavak has joined #openstack-infra | 17:15 | |
| *** nwkarsten has quit IRC | 17:16 | |
| clarkb | I am going to tes tthe reboot option by making a ubuntu trusty instance in rax, killing its sh host keys, and rebooting | 17:16 |
| *** BobBall is now known as BobBall-AWOL | 17:16 | |
| anteaya | clarkb: according to cacti nodepool seems to be working on the service | 17:16 |
| anteaya | clarkb: but not being able to ssh in is a concern | 17:16 |
| clarkb | anteaya: yes the service is still running, we just can't puppet it or access the server | 17:16 |
| *** nwkarsten has joined #openstack-infra | 17:16 | |
| greghaynes | clarkb: so the package updating aspect of nodepool is triggered over ssh? | 17:16 |
| anteaya | clarkb: what do you think caused the ssh issue? | 17:16 |
| clarkb | greghaynes: yes beacuse thats puppet which is triggered by ansible | 17:16 |
| greghaynes | womp womp | 17:16 |
| *** e0ne has quit IRC | 17:17 | |
| clarkb | anteaya: https://review.openstack.org/#/c/301982/2 | 17:17 |
| clarkb | the first patchset was safe but the second was not | 17:17 |
| greghaynes | clarkb: oh, actually | 17:17 |
| greghaynes | clarkb: reboot might work | 17:17 |
| anteaya | clarkb: ah, :( | 17:17 |
| clarkb | greghaynes: ya I am going to test it | 17:17 |
| clarkb | one a different instance | 17:17 |
| clarkb | s/one/on/ | 17:17 |
| anteaya | shouldn't we revert 301982 before a reboot? | 17:17 |
| anteaya | lest we have the same code running after a reboot? | 17:17 |
| clarkb | anteaya: the fix has already been approved | 17:18 |
| clarkb | and is being tested independently | 17:18 |
| greghaynes | clarkb: is there cloud-init on nodepool? | 17:18 |
| *** eil397 has joined #openstack-infra | 17:18 | |
| *** ddieterly is now known as ddieterly[away] | 17:18 | |
| greghaynes | because thats probably the best hope | 17:18 |
| anteaya | clarkb: ah okay, thanks | 17:19 |
| clarkb | greghaynes: depends on what rax puts in their base image | 17:19 |
| anteaya | clarkb: what is the fix? | 17:19 |
| clarkb | greghaynes: I think so | 17:19 |
| greghaynes | heh | 17:19 |
| *** annegentle has quit IRC | 17:19 | |
| clarkb | greghaynes: I am sorting out booting a small instance on the same image to test | 17:19 |
| clarkb | but hvae to figure out what the image and flavor and all that are | 17:20 |
| greghaynes | yea, now we get to find out if cloud-init re-triggers hostkey gen based on the file existing or whether it has already been run | 17:20 |
| anteaya | clarkb: sorry I haven't been able to follow the backscroll if this has been clearly stated prior to my questions | 17:20 |
| *** marg7175 has quit IRC | 17:20 | |
| clarkb | greghaynes: you should maybe also add a gurad there to make sure target root is sset | 17:20 |
| *** marg7175 has joined #openstack-infra | 17:20 | |
| *** nwkarsten has quit IRC | 17:20 | |
| clarkb | anteaya: basically patchset1 ran in the context of the dib chroot whcih meant the path that was deleted was in the image. The second patchset ran outside of the chroot so the build host (nodepool) had its keys deleted | 17:20 |
| anteaya | oh wow | 17:21 |
| clarkb | everything still works except for ssh into the machine. theory is that if we reboot the init system will generate new keys for us and it will function | 17:21 |
| openstackgerrit | Merged openstack/diskimage-builder: Revert "Skip centos functional testing" https://review.openstack.org/274938 | 17:21 |
| clarkb | I am going to work on testing this theory now | 17:21 |
| *** sdake_ is now known as sdake | 17:21 | |
| *** Rockyg has quit IRC | 17:21 | |
| greghaynes | clarkb: Yea, although its -u so if its unset itll just error | 17:21 |
| greghaynes | which is kind of what we want | 17:21 |
| clarkb | also if anyone has an existing connection they can generate new keys for us | 17:21 |
| anteaya | clarkb: okay thank you for clarifying so I can follow along | 17:21 |
| anteaya | clarkb: okay well hopefully someone has and comes online soon | 17:22 |
| *** ddieterly[away] is now known as ddieterly | 17:22 | |
| *** tosky has joined #openstack-infra | 17:22 | |
| krotscheck | mordred, jeblair: Y'all aren't around by any chance? We have a patch for the NPM issue that is blocking horizon's mitaka gate. https://review.openstack.org/#/c/302292 | 17:23 |
| *** flepied has quit IRC | 17:24 | |
| *** ddieterly is now known as ddieterly[away] | 17:24 | |
| *** ddieterly[away] is now known as ddieterly | 17:24 | |
| *** doug-fish has joined #openstack-infra | 17:24 | |
| *** MarkAtwood has joined #openstack-infra | 17:24 | |
| greghaynes | clarkb: fix worked locally | 17:25 |
| *** jsavak has quit IRC | 17:26 | |
| zaro | anteaya: still no log https://review.openstack.org/#/c/300596 | 17:26 |
| zaro | but maybe that's just only on failure | 17:26 |
| anteaya | zaro: yes, but on failure we should have logs | 17:26 |
| *** abregman has quit IRC | 17:26 | |
| anteaya | zaro: I think you have found a bug for that job | 17:26 |
| anteaya | in that job | 17:26 |
| *** jsavak has joined #openstack-infra | 17:26 | |
| *** achanda has quit IRC | 17:27 | |
| zaro | because PS 5 passed and there was a log | 17:27 |
| *** esikachev has quit IRC | 17:27 | |
| krotscheck | AJaeger: Thanks :D | 17:27 |
| anteaya | zaro: indeed there is | 17:27 |
| zaro | but odd that it's the only one that failed. you happen to know what's different in that test besides it be 'devstack'? | 17:27 |
| anteaya | I do not | 17:28 |
| krotscheck | AJaeger: I'll get rid of it as soon as we get xenial things :) | 17:28 |
| *** Sukhdev_ has quit IRC | 17:28 | |
| greghaynes | clarkb: not sure if you want to try and promote the fix? | 17:29 |
| greghaynes | I am a bit worried about downstream CI folks | 17:29 |
| zaro | clarkb: was wondering what is the difference between job gate-infra-puppet-apply-devstack-centos and the other gat-infra-puppet-.. jobs? | 17:29 |
| *** claudiub has joined #openstack-infra | 17:29 | |
| *** sdake_ has joined #openstack-infra | 17:29 | |
| *** tonytan4ever has quit IRC | 17:29 | |
| *** sdake has quit IRC | 17:29 | |
| krotscheck | How long do project-config changes take to propagate? I'm guessing 15-30 minutes, yes? | 17:29 |
| zaro | clarkb: i'm seeing that job fail but no log | 17:29 |
| clarkb | greghaynes: I can stick it straight into the gate in a minute | 17:30 |
| zaro | krotscheck: my last job took 10 mins | 17:30 |
| greghaynes | clarkb: awesome | 17:30 |
| clarkb | zaro: they run on different base images which causes different puppet behavior | 17:30 |
| krotscheck | zaro: Neat | 17:30 |
| clarkb | greghaynes: initial testing says reboot is no good | 17:31 |
| zaro | krotscheck: nope, i take that back it took 1hr | 17:31 |
| krotscheck | NOOOOooooo | 17:31 |
| greghaynes | clarkb: womp womp. You could boot it in single user via rax gui | 17:31 |
| clarkb | ok will promote fix now | 17:31 |
| clarkb | or arther stick it in gate | 17:31 |
| greghaynes | ok | 17:31 |
| *** esikachev has joined #openstack-infra | 17:31 | |
| zaro | clarkb: so if it only fails on `devstack`, like this https://review.openstack.org/#/c/300596 is taht a valid failure? | 17:32 |
| anteaya | zaro: they seem to be defined in http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n142 | 17:32 |
| zaro | clarkb: also no log so i have no idea why | 17:32 |
| *** tqtran has joined #openstack-infra | 17:32 | |
| clarkb | greghaynes: thats done | 17:32 |
| *** mylu has joined #openstack-infra | 17:32 | |
| clarkb | zaro: sorry I am busy trying to unbreak nodepool, I can take a look once I have a plan | 17:32 |
| greghaynes | clarkb: ty | 17:33 |
| *** nadya has quit IRC | 17:33 | |
| *** ayoung has joined #openstack-infra | 17:33 | |
| anteaya | clarkb greghaynes can I ask what the patch is that you are considering the fix? | 17:33 |
| greghaynes | anteaya: https://review.openstack.org/#/c/302373/ | 17:34 |
| openstackgerrit | Merged openstack-infra/project-config: Install more recent version of node https://review.openstack.org/302292 | 17:34 |
| clarkb | greghaynes: I am going to boot a new test instance on that imag eand see if there is nayhting I can take advantage of | 17:34 |
| clarkb | but guess is no, its either recover mode it, or rebuild it | 17:34 |
| greghaynes | clarkb: Yea, I cant think of any other ideas | 17:34 |
| anteaya | greghaynes: thank you | 17:34 |
| anteaya | it is a shame noone else got a chance to review and weigh in on the patch | 17:35 |
| clarkb | anteaya: the fix? | 17:35 |
| anteaya | yes | 17:35 |
| clarkb | I don't thin kwe want to wait... | 17:35 |
| clarkb | this will make dib users incredibly unhappy | 17:35 |
| * clarkb looks at himself. Confirmed | 17:35 | |
| greghaynes | Ya, this is bad for anyone using the release, the best hope is we can get this in before anyone else pulls down the release | 17:35 |
| *** jamesmcarthur has quit IRC | 17:35 | |
| krotscheck | Merged. And now, we wait. | 17:35 |
| greghaynes | even if its broken, it cant be worse | 17:36 |
| anteaya | well you and greg seem to have this in hand then, I'll stay out as I'm obviously slowing you down | 17:36 |
| anteaya | sorry for that | 17:36 |
| *** esikachev has quit IRC | 17:36 | |
| clarkb | greghaynes: cloud-init is installed | 17:37 |
| clarkb | but I gess it won't kick ubuntu hard enough to make new keys | 17:37 |
| clarkb | ugh | 17:37 |
| greghaynes | clarkb: hrm, I guess it decides wheter to gen host keys based on whether it needs to re-assert state | 17:37 |
| greghaynes | clarkb: if you change some metadata about the noe it might | 17:37 |
| greghaynes | node | 17:37 |
| clarkb | ya, will have to look more but at some point it just becomes easier to rebuild | 17:38 |
| greghaynes | agreed | 17:38 |
| Kiall | Q - a repo creation change landed about 3 hrs ago, but the repo doesn't exist yet.. Is that expected? https://review.openstack.org/#/c/301219/4 | 17:38 |
| *** ayoung has quit IRC | 17:38 | |
| *** ayoung has joined #openstack-infra | 17:39 | |
| *** sambetts is now known as sambetts|afk | 17:39 | |
| *** esker has quit IRC | 17:39 | |
| AJaeger | Kiall: that is not normal. I can confirm that openstack/designate-tempest is not on git.openstack.org ;( | 17:39 |
| Kiall | check github and gerrrit too, not showing | 17:40 |
| clarkb | ssh_deletekeys: True is in the rax provided cloud config | 17:40 |
| Kiall | I checked* | 17:40 |
| clarkb | which I think means delete the keys on first start? if very much did not handle them having been deleted under it :( | 17:40 |
| AJaeger | Kiall: You need an infra-root to help - clarkb seems to be the only one around now and he's firefighting already. | 17:40 |
| AJaeger | Kiall: please ask later again | 17:41 |
| clarkb | greghaynes: so the idea of changing metadata has some possibility | 17:41 |
| greghaynes | clarkb: yea, its probably worth a shot | 17:41 |
| Kiall | unacceptable, we demand prompt support for our $0 - kidding, there is no particular rush ;) | 17:41 |
| greghaynes | I could also read the code | 17:41 |
| *** amitgandhinz has joined #openstack-infra | 17:42 | |
| *** sdake has joined #openstack-infra | 17:42 | |
| clarkb | greghaynes: looks like dleeting /var/lib/cloud/instances will do it too | 17:42 |
| clarkb | but I cna't easily delete that :) | 17:42 |
| greghaynes | clarkb: is that the cached metadata? | 17:42 |
| *** tosky_ has joined #openstack-infra | 17:42 | |
| clarkb | ya its where cloud init records what it has done | 17:42 |
| *** tosky_ has quit IRC | 17:42 | |
| greghaynes | yea, so that supports the theory | 17:43 |
| clarkb | I am going to try the metadata update route on my test box | 17:43 |
| clarkb | as soon as I figure out how to update the metadata :) | 17:43 |
| *** david-lyle has quit IRC | 17:44 | |
| *** sdake_ has quit IRC | 17:44 | |
| *** tonytan4ever has joined #openstack-infra | 17:45 | |
| *** ddieterly is now known as ddieterly[away] | 17:46 | |
| *** degorenko is now known as _degorenko|afk | 17:46 | |
| clarkb | nova meta clarkb-test foo=bar | 17:46 |
| clarkb | I worry that cloud init will be smart enough to know that doesn't affect ssh :/ | 17:47 |
| greghaynes | that took me way too long to find | 17:47 |
| clarkb | results in a minute | 17:47 |
| greghaynes | yep, possible | 17:47 |
| clarkb | ya connection still closed by remote end | 17:47 |
| clarkb | apparently the user adta script thing you can shove in only runs on first boot too | 17:48 |
| clarkb | but I am going to test that next I think | 17:48 |
| *** marg7175 has quit IRC | 17:48 | |
| *** ihrachys has quit IRC | 17:49 | |
| *** shashank_hegde has joined #openstack-infra | 17:51 | |
| *** cloudtrainme has joined #openstack-infra | 17:51 | |
| clarkb | harlowja: you are cloud init person. Can user data scripts run on not the first boot without changing anything in the instance? | 17:52 |
| clarkb | harlowja: is there some flag I can set in metadata that says no really run this every time? | 17:52 |
| clarkb | or maybe I can feed it a cookie? | 17:52 |
| * harlowja cloudinit person? | 17:52 | |
| harlowja | lol | 17:52 |
| harlowja | damn why scott moser not in this channel | 17:52 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 17:52 | |
| harlowja | hehe | 17:52 |
| *** ayoung has quit IRC | 17:52 | |
| harlowja | run on not the first boot? | 17:53 |
| *** trown|lunch is now known as trown | 17:53 | |
| harlowja | so is that like every time but first boot? | 17:53 |
| *** vgridnev has joined #openstack-infra | 17:53 | |
| clarkb | harlowja: well any boot, speifically the next boot :) | 17:53 |
| *** pvaneck has joined #openstack-infra | 17:53 | |
| clarkb | harlowja: I need to force an instance I have no access to other than nova api to regenerate its ssh keys | 17:53 |
| clarkb | am hoping I can avoid doing an instance recovery or rebuilding it (rebuild is hard because apparently we have no quota) | 17:53 |
| harlowja | hmmmm | 17:53 |
| Kiall | - [ cloud-init-per, boot, my-command, arg1, arg2 ] in a cloud-config runcmd block will run every time | 17:54 |
| harlowja | so your gonna hit an issue with the fact that cloud-init caches the metadata stuffs | 17:54 |
| clarkb | Kiall: I can only modify the metadata | 17:54 |
| harlowja | userdata/metadata | 17:54 |
| harlowja | and u said u have instance running | 17:54 |
| harlowja | that u have no access to | 17:54 |
| clarkb | yes, wondering if there is an easy fix if not I am going to go sacrifice some logstash workers and build a new nodepool | 17:55 |
| *** annegentle has joined #openstack-infra | 17:55 | |
| Kiall | right.. any reason you can't snapshot it, then nova rebuild using the snapshot (to keeps IPs etc) or launch a new instance if IPs don't matter? | 17:55 |
| harlowja | clarkb if u had access to the instance, then there is a way | 17:55 |
| clarkb | Kiall: snapshould will preserve cloud inits brokeness | 17:55 |
| *** rcernin has quit IRC | 17:56 | |
| harlowja | clarkb it won't | 17:56 |
| harlowja | a new instance-uuid will get generated | 17:56 |
| clarkb | harlowja: how? | 17:56 |
| clarkb | ah | 17:56 |
| Kiall | clarkb: no, cloud-init defaults everything to once-per-instance rather than once-per-image | 17:56 |
| harlowja | the cache in cloud-init is tied to instance id | 17:56 |
| harlowja | right | 17:56 |
| clarkb | gotcha | 17:56 |
| clarkb | though its easier to just rebuild at that point | 17:56 |
| harlowja | right | 17:56 |
| clarkb | since we have tooling to rebuild but this other stuff owuld be a one off | 17:56 |
| *** mriedem has quit IRC | 17:56 | |
| greghaynes | noooo gate failed for the fix | 17:56 |
| clarkb | greghaynes: gg | 17:57 |
| clarkb | I just have to free up quota :/ | 17:57 |
| greghaynes | Failed connect to mirror.utexas.edu:80; Connection timed out | 17:57 |
| *** rbrndt_ has quit IRC | 17:57 | |
| greghaynes | blame austin | 17:57 |
| harlowja | clarkb so i would prob try rebuild | 17:57 |
| greghaynes | clarkb: is reverify the way to put it right back in to the gate? | 17:57 |
| clarkb | harlowja: cloud init should have a no really mode | 17:57 |
| clarkb | greghaynes: it will check first | 17:58 |
| clarkb | I can reenqueue it to the gate in a minute | 17:58 |
| greghaynes | ok | 17:58 |
| *** xyang1 has quit IRC | 17:58 | |
| harlowja | clarkb u can make it do such things, but u have to prepare your user-data ahead of time to remove that cache as like the last thing it runs, lol | 17:58 |
| AJaeger | greghaynes: there's no reverify anymore | 17:58 |
| harlowja | cloud-init i suppose could make that easier | 17:58 |
| greghaynes | AJaeger: ah, ok | 17:58 |
| clarkb | harlowja: right I am saying a nice feature would be a no really mode | 17:58 |
| harlowja | with a no_cache mode | 17:58 |
| harlowja | i guess that's what u are saying, ha | 17:58 |
| AJaeger | greghaynes: or better: No way to put it back in the gate, you always go through check first. | 17:58 |
| AJaeger | greghaynes: use recheck | 17:58 |
| *** sridhar_ram has joined #openstack-infra | 17:59 | |
| clarkb | greghaynes: AJaeger don't bothe rI will queue straight to gate again | 17:59 |
| greghaynes | oh, I rechecked | 17:59 |
| clarkb | its fine I can still queue to gate | 17:59 |
| greghaynes | ok | 17:59 |
| harlowja | clarkb do u want to make a tiny bug @ https://bugs.launchpad.net/cloud-init | 17:59 |
| AJaeger | greghaynes: we disabled the putting back in the gate months ago.. | 17:59 |
| harlowja | i can try adding that, i haven't done cloud-init work in a while, so i need to do some :-P | 17:59 |
| *** achanda has joined #openstack-infra | 17:59 | |
| AJaeger | greghaynes: yeah, clarkb has magic powers ;) | 17:59 |
| greghaynes | AJaeger: I am just behind the times :) | 17:59 |
| clarkb | harlowja: maybe when I am not trying to recover important services :) | 17:59 |
| harlowja | :) | 17:59 |
| *** annegentle has quit IRC | 17:59 | |
| harlowja | that's fine | 17:59 |
| harlowja | what's this about pets i hear | 18:00 |
| harlowja | lol | 18:00 |
| clarkb | harlowja: well its not really a pet | 18:00 |
| clarkb | I can rebuild it in 15 minutes | 18:00 |
| clarkb | harlowja: except I have no quota | 18:00 |
| harlowja | :( | 18:00 |
| clarkb | so have to sacrifice some other cattle that are also important | 18:00 |
| harlowja | :-/ | 18:00 |
| *** jcoufal has quit IRC | 18:01 | |
| *** annegentle has joined #openstack-infra | 18:01 | |
| *** ajmiller has joined #openstack-infra | 18:01 | |
| harlowja | well bettsie the cow will be reincarnated hopefully at some point | 18:02 |
| harlowja | and/or other popular cow names | 18:02 |
| *** flepied has joined #openstack-infra | 18:02 | |
| harlowja | http://ecx.images-amazon.com/images/I/51NVfyPhi%2BL._SX327_BO1,204,203,200_.jpg lol | 18:02 |
| *** david-lyle has joined #openstack-infra | 18:02 | |
| clarkb | moo | 18:02 |
| *** sridhar_ram has quit IRC | 18:03 | |
| *** jsavak has quit IRC | 18:03 | |
| * clarkb takes a break before diving back in. The service is up as long as nothing terrible happens will continue to operate | 18:04 | |
| *** e0ne has joined #openstack-infra | 18:04 | |
| * greghaynes tries to not merge any more terrible dib stuff | 18:04 | |
| clarkb | dib failed the gate again | 18:05 |
| clarkb | I will requeue it then take a break | 18:05 |
| *** Jeffrey4l has quit IRC | 18:05 | |
| greghaynes | again? that looks like the same fail | 18:05 |
| *** sridhar_ram has joined #openstack-infra | 18:05 | |
| clarkb | maybe they use dib onthat mirror | 18:05 |
| clarkb | badum tish | 18:05 |
| greghaynes | I mean, I dont see a new gate run | 18:06 |
| clarkb | oh | 18:06 |
| clarkb | I see I thought it had reported back | 18:06 |
| clarkb | well then let me ninja | 18:06 |
| *** nmagnezi has quit IRC | 18:07 | |
| *** bkeller` has joined #openstack-infra | 18:07 | |
| *** mrmartin has joined #openstack-infra | 18:07 | |
| clarkb | ninja accomplished | 18:07 |
| clarkb | (I manually killed the job in jenkins so that zuul would report allowing me to restart the gate jobs via zuul enqueue) | 18:08 |
| openstackgerrit | Ben Kero proposed openstack/diskimage-builder: Fix add-apt-repository package for precise https://review.openstack.org/294760 | 18:09 |
| bkero | cinerama: ^ | 18:09 |
| clarkb | greghaynes: you might also want to send mail to the list and warn bifrost users in particular since I think all the docs say use simple-init | 18:09 |
| greghaynes | those poor, poor, users | 18:09 |
| clarkb | "stop using dib for a minute until 302373 merges or use an older relaese and have broken ssh | 18:10 |
| *** yamamoto has quit IRC | 18:10 | |
| clarkb | "your options are all bad so maybe just get lunch and a beer" | 18:10 |
| *** dtantsur is now known as dtantsur|afk | 18:10 | |
| krotscheck | The lack of puppet updates on the jenkins nodes is because nodepool is sad, yes? | 18:10 |
| clarkb | krotscheck: I don't think so but haven't look into that yet. Only nodepool should be affected | 18:11 |
| *** doug-fis_ has joined #openstack-infra | 18:11 | |
| *** mriedem has joined #openstack-infra | 18:11 | |
| krotscheck | clarkb: Okiday | 18:11 |
| *** doug-fi__ has joined #openstack-infra | 18:12 | |
| clarkb | krotscheck: what changes are you expecting to get applied/ | 18:13 |
| krotscheck | I'm looking for project-config updates. I was under the impression that the next step once something landed is to watch jenkins.openstack.org on puppetboard to wait for it all to update. | 18:13 |
| Kiall | https://review.openstack.org/#/c/283511/ is one from me, repo creation not working | 18:13 |
| *** e0ne has quit IRC | 18:14 | |
| *** doug-fish has quit IRC | 18:14 | |
| *** pradk has quit IRC | 18:14 | |
| clarkb | Kiall: thats a designate change | 18:14 |
| Kiall | derp | 18:14 |
| clarkb | krotscheck: which project-config change(s)? | 18:14 |
| Kiall | https://review.openstack.org/#/c/301219 | 18:14 |
| *** piet has quit IRC | 18:15 | |
| krotscheck | clarkb https://review.openstack.org/#/c/302292/ | 18:15 |
| *** doug-fish has joined #openstack-infra | 18:15 | |
| *** doug-fis_ has quit IRC | 18:15 | |
| AJaeger | clarkb: looking at puppetboard, jenkins has not been updatd for 4+ hours | 18:15 |
| clarkb | 2016-04-06 14:00:24,352 p=20897 u=root | changed: [static.openstack.org -> localhost] is the last thing logged by ansible | 18:16 |
| *** doug-fi__ has quit IRC | 18:17 | |
| clarkb | hrm that anisble run is still running | 18:17 |
| clarkb | according to ps at least | 18:17 |
| *** tosky has quit IRC | 18:17 | |
| clarkb | so that explains the lack of updates, not sure why it has done that though | 18:17 |
| *** tosky has joined #openstack-infra | 18:18 | |
| clarkb | image updates do start around then so its possible it tried sshing to nodepool maybe and got confused | 18:18 |
| *** regXboi has joined #openstack-infra | 18:18 | |
| krotscheck | clarkb: Annoying. What's the fix, kill and restart? | 18:19 |
| clarkb | krotscheck: probably, I am going to strace it really quickly to see if it is doing anything | 18:19 |
| cinerama | bkero: thanks! | 18:20 |
| cinerama | (i reviewed) | 18:20 |
| clarkb | it is polling a bunch of file descriptors | 18:20 |
| *** MarkAtwood has quit IRC | 18:20 | |
| clarkb | krotscheck: it isn't clear to me why nodepool's ssh breaking would break ansible this way. It should just fail to connect ot nodepool.o.o and move on | 18:21 |
| clarkb | so not sure if the events are related | 18:21 |
| *** yamamoto has joined #openstack-infra | 18:21 | |
| *** yamahata has quit IRC | 18:21 | |
| krotscheck | clarkb: Let's hope so, otherwise there's something else broken to fix. | 18:21 |
| greghaynes | clarkb: another fail which looks like debian is just not in a good state | 18:22 |
| greghaynes | https://jenkins01.openstack.org/job/gate-dib-dsvm-functests-ubuntu-trusty/19/console | 18:22 |
| clarkb | nice to see debian suffers from the same index issues as ubuntu | 18:23 |
| *** rbrndt has joined #openstack-infra | 18:23 | |
| *** e0ne has joined #openstack-infra | 18:24 | |
| *** ociuhandu has quit IRC | 18:24 | |
| jeblair | clarkb: if afstest exists, you can delete it | 18:25 |
| *** pradk has joined #openstack-infra | 18:25 | |
| clarkb | ok, going to work on getting dib fixed first because we will need new images on the new host which means we can't have broken dib :/ | 18:26 |
| *** cloudtrainme has quit IRC | 18:27 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-pip: Add new pip provider to override the upstream one https://review.openstack.org/302396 | 18:27 |
| *** mrmartin has quit IRC | 18:27 | |
| *** imcsk8 is now known as imcsk8|afk | 18:27 | |
| clarkb | jeblair: if you are able to look into the ansible situation that would be helpful, I don't understand how this could have tripped ansible other than to have it fail to ssh to nodepool.o.o but it has been running since 1300ish | 18:28 |
| clarkb | er 1400ish | 18:28 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-pip: Add new pip provider to use pypi mirrors https://review.openstack.org/302396 | 18:28 |
| *** jamesmcarthur has joined #openstack-infra | 18:28 | |
| *** ayoung has joined #openstack-infra | 18:29 | |
| anteaya | crinkle: you write the best commit messages | 18:29 |
| *** nwkarsten has joined #openstack-infra | 18:29 | |
| crinkle | anteaya: tyvm | 18:29 |
| clarkb | greghaynes: reenqueued to the gate again. Hopefully we get funcitonal mirrors | 18:30 |
| greghaynes | clarkb: yep | 18:30 |
| *** jsavak has joined #openstack-infra | 18:31 | |
| anteaya | crinkle: welcome | 18:32 |
| jeblair | clarkb: i am on airplane wifi 1bps. | 18:32 |
| clarkb | jeblair: oh fun | 18:32 |
| zaro | crinkle: would you have time to help with this? https://review.openstack.org/#/c/300596/6/templates/gerrit.config.erb | 18:33 |
| *** EricGonczer_ has joined #openstack-infra | 18:34 | |
| zaro | crinkle: i've run it thru ruby interpretor and seems correct but not sure it looks right in puppet template context | 18:34 |
| jeblair | clarkb: will attempt; success unlikely. | 18:34 |
| *** kushal has quit IRC | 18:35 | |
| jeblair | (no bits but i got a stroopwafel) | 18:36 |
| *** lakshmiS has quit IRC | 18:36 | |
| crinkle | zaro: i think that looks right | 18:36 |
| *** kushal has joined #openstack-infra | 18:37 | |
| crinkle | zaro: did you run it in puppet and check that the template outputs the right thing? | 18:37 |
| *** angdraug has quit IRC | 18:37 | |
| crinkle | zaro: the only thing i'm not sure about is if you want <%= key -%>='<%= value -%>' instead of <%= key %>='<%= value %>' | 18:37 |
| clarkb | jeblair: oh yes food that is why I was going totake a break | 18:37 |
| * clarkb does so | 18:37 | |
| *** cloudtrainme has joined #openstack-infra | 18:38 | |
| *** esikachev has joined #openstack-infra | 18:39 | |
| *** jsavak has quit IRC | 18:39 | |
| *** mrmartin has joined #openstack-infra | 18:40 | |
| *** jsavak has joined #openstack-infra | 18:40 | |
| *** rockyg has joined #openstack-infra | 18:41 | |
| zaro | crinkle: i do not have puppet dev env setup to execute. as-is looks consistent with ow the other vars are referenced. i only see the '-' used in loops and if conditions | 18:42 |
| jeblair | clarkb: i see no open connections from puppetmaster over ssh, so i also don't know what's going on with ansible. i would probably give up and kill it at this point (if i had reliable access) | 18:43 |
| clarkb | jeblair: ok, I will keep that in mind when I have a chance | 18:43 |
| clarkb | and do that whe nI am able to watch the next run | 18:43 |
| *** EricGonczer_ has quit IRC | 18:44 | |
| * clarkb is slightly hopefully someone will show up with an active ssh connection before dib releases the fix and this can all be done easymod | 18:44 | |
| clarkb | pabelanger: ^ any chance you have one? | 18:44 |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's download protocols https://review.openstack.org/300596 | 18:45 |
| clarkb | jeblair: also afstest is still up so I iwll likely delete it | 18:45 |
| zaro | crinkle: pushed up a change, but only for formatting | 18:45 |
| *** ddieterly[away] has quit IRC | 18:46 | |
| *** e0ne has quit IRC | 18:46 | |
| *** lakshmiS has joined #openstack-infra | 18:46 | |
| zaro | crinkle: the CI fails on 'gate-infra-puppet-apply-devstack-centos7' but passes for all the other puppet-apply jobs. | 18:46 |
| zaro | which seems wacko to me. | 18:48 |
| mugsie | q timsim | 18:49 |
| *** mrmartin has quit IRC | 18:51 | |
| crinkle | zaro: this is how i test these things and the result i get (from the earlier patchset anyway) http://paste.openstack.org/show/493223/ | 18:51 |
| *** EricGonczer_ has joined #openstack-infra | 18:51 | |
| *** mrmartin has joined #openstack-infra | 18:53 | |
| *** angdraug has joined #openstack-infra | 18:53 | |
| anteaya | clarkb: mordred does not have an active ssh connection to nodepool | 18:53 |
| zaro | crinkle: yes, exactly what i wanted. my data structure is here: https://review.openstack.org/#/c/300603/3/modules/openstack_project/manifests/review.pp | 18:54 |
| anteaya | rcarrillocruz: thank you for the local.pp for mediawiki you gave to me yesterday, it worked, I'm now seeing errors parsing the template | 18:54 |
| anteaya | rcarrillocruz: so thank you | 18:54 |
| zaro | crinkle: looks like the only difference is that there should be quotes for the arrays? like 'command' instead of command | 18:54 |
| crinkle | zaro: i don't think it matters much in puppet 3 but it might matter in puppet 4, in any case quotes makes it look more consistent | 18:55 |
| *** ihrachys has joined #openstack-infra | 18:55 | |
| crinkle | zaro: is the indentation in that output the way you expect it to be? it looks funny to me | 18:55 |
| clarkb | greghaynes: down to one job that needs to pass | 18:56 |
| clarkb | fingers crossed | 18:56 |
| *** jsavak has quit IRC | 18:56 | |
| *** ociuhandu has joined #openstack-infra | 18:56 | |
| *** jsavak has joined #openstack-infra | 18:57 | |
| zaro | crinkle: currently it doesn't seem like we ever put quotes on the left side of the rocket in our pp files. | 18:57 |
| zaro | crinkle: ohh i should fix that indentation. i was just there to make it easier to review i guess. | 18:58 |
| crinkle | zaro: this is different | 18:58 |
| crinkle | zaro: normally the left side of the => is a resource parameter | 18:58 |
| crinkle | zaro: in this case it's inside of a hash (dictionary) | 18:58 |
| crinkle | zaro: so 'download' should not be quoted but 'command' can and sort of should be | 18:59 |
| *** esikachev has quit IRC | 18:59 | |
| *** mylu has quit IRC | 18:59 | |
| *** ddieterly has joined #openstack-infra | 18:59 | |
| *** vgridnev has quit IRC | 19:00 | |
| *** austin81 has left #openstack-infra | 19:00 | |
| *** mylu has joined #openstack-infra | 19:01 | |
| *** vgridnev has joined #openstack-infra | 19:01 | |
| anteaya | crinkle: after you and zaro are finished talking, I could use a hand debugging an error in the mediawiki puppet module: http://paste.openstack.org/show/493226/ | 19:01 |
| zaro | crinkle: ok. thanks. i've confirmed in ruby that it should contain quotes but wasn't sure if that's how it work in puppet. | 19:01 |
| *** vgridnev has quit IRC | 19:02 | |
| anteaya | crinkle: this is me running it locally in a vm | 19:02 |
| *** sridhar_ram has quit IRC | 19:02 | |
| *** vgridnev has joined #openstack-infra | 19:03 | |
| anteaya | crinkle: looks like the value that is passed in sometimes is an empty string | 19:04 |
| *** Swami has quit IRC | 19:04 | |
| zaro | anteaya: will take a look | 19:04 |
| anteaya | zaro: thanks | 19:04 |
| crinkle | anteaya: it looks like $wg_captchaquestions is expecting a hash and you are passing it a string | 19:05 |
| *** EricGonczer_ has quit IRC | 19:05 | |
| *** EricGonczer_ has joined #openstack-infra | 19:05 | |
| zaro | crinkle: ohh wait, i have no idea why sure why the indentation is off in your test result. do you? | 19:05 |
| *** maishsk has joined #openstack-infra | 19:05 | |
| crinkle | anteaya: it looks like the default is an empty hash, so are you changing that value somewhere? http://git.openstack.org/cgit/openstack-infra/puppet-mediawiki/tree/manifests/init.pp#n14 | 19:05 |
| crinkle | zaro: not sure | 19:06 |
| *** Swami has joined #openstack-infra | 19:06 | |
| *** xyang1 has joined #openstack-infra | 19:06 | |
| anteaya | crinkle: I am not changing that value anywhere | 19:06 |
| clarkb | someone should ask horizon to stop approving changes | 19:06 |
| clarkb | because every one of the changes is failing | 19:06 |
| crinkle | anteaya: what command are you running that produces this? | 19:06 |
| anteaya | crinkle: so far I have handed it a local.pp which is bascially a copy paste of this: http://paste.openstack.org/show/493060/ | 19:07 |
| anteaya | sudo puppet apply local.pp --modulepath=/etc/puppet/modules:system-config/modules | 19:07 |
| anteaya | clarkb: I can go ask | 19:07 |
| zaro | crinkle: well, it should still work. | 19:07 |
| crinkle | anteaya: that local.pp has wg_captchaquestions => hiera('wiki_wg_captchaquestions', ''), which means "look up wiki_wg_captchaquestions in hiera, and if you don't find it us an empty string" | 19:07 |
| crinkle | anteaya: and it passes the empty string in as $wg_captchaquestions | 19:08 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Customize Gerrit's download command and schema https://review.openstack.org/300603 | 19:08 |
| anteaya | crinkle: so it does, silly me | 19:08 |
| *** Sukhdev has joined #openstack-infra | 19:08 | |
| crinkle | so you can change it to wg_captchaquestions => hiera('wiki_wg_captchaquestions', {}) or just wg_captchaquestions => {} and that error should go away (though I don't know if having no captcha questions is valid) | 19:09 |
| anteaya | crinkle: it was me all along | 19:09 |
| anteaya | fair enough | 19:09 |
| anteaya | this was my solution to not having a heira for my test instance | 19:09 |
| anteaya | I'll start with {} and see what I get back next | 19:09 |
| *** yamamoto has quit IRC | 19:10 | |
| bnemec | pabelanger: Any idea why we're still seeing the liberasurecode failures in our puppet jobs? | 19:10 |
| anteaya | not being able to have a heira for a test instance of puppet is inconvient but I don't know a better way to address it | 19:11 |
| bnemec | For example, https://review.openstack.org/#/c/302393 just failed about ten minutes ago. | 19:11 |
| mwhahaha | bnemec: was just going to ask about that | 19:11 |
| mwhahaha | yea it's pretty much blocking anything puppet related | 19:11 |
| mwhahaha | for fuel, tripleo, puppet openstack | 19:11 |
| anteaya | crinkle: it is doing more things now, thank you | 19:11 |
| bnemec | I gather from the earlier discussion that https://review.openstack.org/#/c/302298 was supposed to fix it. | 19:11 |
| crinkle | anteaya: yay \o/ | 19:12 |
| *** sc68cal has joined #openstack-infra | 19:12 | |
| anteaya | crinkle: cert errors which I'm not going to worry about as I'm sure that once it connects to a real heira that is a solved problem | 19:12 |
| clarkb | jenkins isn't currently being updated because ansible is stalled | 19:12 |
| anteaya | crinkle: thank you | 19:12 |
| anteaya | clarkb: :( | 19:12 |
| bnemec | clarkb: Ah, thanks. | 19:12 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix ssh key cleanup to run in chroot https://review.openstack.org/302373 | 19:12 |
| greghaynes | clarkb: looks like it passed | 19:12 |
| greghaynes | ^ | 19:13 |
| clarkb | bnemec: fixing that is on my long list of things to do today | 19:13 |
| * greghaynes cuts a release | 19:13 | |
| clarkb | greghaynes: woot thanks | 19:13 |
| *** rockyg has quit IRC | 19:15 | |
| *** ayoung has quit IRC | 19:16 | |
| anteaya | pleia2: the mediawiki puppet module is unable to access many of the font packages and is erroring out, it is trying via apt-get, any suggestions for how to address this? | 19:16 |
| *** ayoung has joined #openstack-infra | 19:16 | |
| anteaya | clarkb: horizon said okay | 19:17 |
| *** tosky has quit IRC | 19:17 | |
| *** tosky has joined #openstack-infra | 19:18 | |
| clarkb | anteaya: ty | 19:18 |
| anteaya | welcome | 19:18 |
| *** cloudtrainme has quit IRC | 19:19 | |
| *** e0ne has joined #openstack-infra | 19:20 | |
| rcarrillocruz | anteaya: yw | 19:21 |
| anteaya | rcarrillocruz: yay, new errors! | 19:21 |
| greghaynes | ok, tag pushed | 19:22 |
| *** whoops has quit IRC | 19:23 | |
| pabelanger | sorry for the delay, had to deal with the snow! | 19:24 |
| pabelanger | bnemec: yes, we have a patch to JJB to fix it | 19:24 |
| pabelanger | let me check its status | 19:24 |
| clarkb | pabelanger: let me catch you up | 19:24 |
| clarkb | pabelanger: dib release deleted ssh host keys on nodepool.o.o | 19:24 |
| pabelanger | clarkb: woah | 19:24 |
| clarkb | pabelanger: separately but possibly realted because timestamps kinda line up, ansible went to lunch on the puppetmaster | 19:24 |
| anteaya | horizon just called a team bug triage since they have agreed to not merge patches at the moment | 19:24 |
| greghaynes | the dib team decided nodepool.o.o needed a new set of host keys | 19:25 |
| clarkb | pabelanger: so I have been working wit hgreghaynes to get a new dib release out (which was just tagged) | 19:25 |
| clarkb | pabelanger: once that is on pypi I need to make a new nodepool | 19:25 |
| *** whoops has joined #openstack-infra | 19:25 | |
| greghaynes | just uploaded to pypi | 19:25 |
| pabelanger | clarkb: okay, I also see ansible is hung on puppetmaster.o.o, it hasn't run since 14:00 UTC | 19:25 |
| clarkb | pabelanger: yup it is possible that is related since new image builds start around then or maybe it isn't realted | 19:25 |
| clarkb | pabelanger: if you are able to look into that I would be most grateful | 19:25 |
| *** vgridnev has quit IRC | 19:26 | |
| pabelanger | clarkb: Ya, I can see hung ansible-playbook since 13:34 UTC (2 of them) | 19:26 |
| *** Sukhdev has quit IRC | 19:26 | |
| pabelanger | clarkb: let me know if safe to kill them | 19:26 |
| clarkb | pabelanger: I hvae no idea :) jeblair briefly looked and said he didn't see any active ssh connections | 19:27 |
| pabelanger | okay | 19:27 |
| clarkb | pabelanger: my understanding of the istuation is that it should be safe to kill them | 19:27 |
| clarkb | pabelanger: the only thing that should be broken is ssh to nodepool | 19:27 |
| *** cloudtrainme has joined #openstack-infra | 19:27 | |
| clarkb | pabelanger: you don't have a running session on nodepool do you? | 19:27 |
| pabelanger | okay, so that explains why ansible is likely hung | 19:27 |
| clarkb | if you hav ethat we can just make new keys | 19:27 |
| pabelanger | Ugh, no | 19:28 |
| pabelanger | I closed it this morning | 19:28 |
| clarkb | mine died due to laptop suspend | 19:28 |
| clarkb | ok moving forward with make new nodepool pla | 19:28 |
| pabelanger | clarkb: is it safe to resume our puppet runs then? or should we just hold off | 19:29 |
| clarkb | pabelanger: it should be safe I think | 19:29 |
| clarkb | like puppet dind't break anything | 19:29 |
| *** vgridnev has joined #openstack-infra | 19:30 | |
| clarkb | but it is possible it will hang again when it starts talking to nodepool if that is where it has trouble | 19:30 |
| *** esker has joined #openstack-infra | 19:30 | |
| pabelanger | ok | 19:30 |
| clarkb | greghaynes: https://pypi.python.org/pypi/diskimage-builder/1.7.2 is the new release? | 19:32 |
| clarkb | er no timetsamps are rwong | 19:32 |
| clarkb | 1.14.1 is the new one? | 19:32 |
| clarkb | https://pypi.python.org/pypi/diskimage-builder/ | 19:32 |
| clarkb | I can start building a new nodepool in a minute if so. Have to resurrect my attach cinder volume change, going to use this opportunity to make a bigger volume for nodepool | 19:33 |
| greghaynes | clarkb: yep, 1.14.1 | 19:33 |
| clarkb | greghaynes: before I forget did a bug about epel being required by base get posted yet? | 19:34 |
| greghaynes | clarkb: nope | 19:34 |
| *** fhubik has quit IRC | 19:34 | |
| *** Goneri has quit IRC | 19:35 | |
| *** esker has quit IRC | 19:35 | |
| *** Goneri has joined #openstack-infra | 19:35 | |
| *** esker has joined #openstack-infra | 19:36 | |
| *** mylu has quit IRC | 19:36 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Rework migration 1679b5bc102c to not use the db api https://review.openstack.org/301369 | 19:36 |
| anteaya | crinkle: where do logs end up after a puppet apply run on a puppet module, I don't seem to see any in /var/log | 19:37 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add migration to move attrs and tags into columns https://review.openstack.org/298069 | 19:37 |
| *** salv-orlando has joined #openstack-infra | 19:37 | |
| pabelanger | clarkb: I ran into that too, but haven't done a patch yet. Been meaning to talk to ianw about that | 19:37 |
| *** Sukhdev has joined #openstack-infra | 19:38 | |
| *** jamesmcarthur has quit IRC | 19:38 | |
| crinkle | anteaya: i think the only log will be to stdout by default | 19:39 |
| anteaya | oh | 19:39 |
| anteaya | that isn't terribly sharable | 19:39 |
| anteaya | should I tweak a setting or just pipe stdout and err to a file? | 19:40 |
| crinkle | anteaya: you can use the -l flag to tell it to log to a file you choose | 19:40 |
| *** ifarkas has quit IRC | 19:40 | |
| anteaya | awesome I will do that, thank you | 19:41 |
| *** Sukhdev has quit IRC | 19:42 | |
| *** matt-borland has quit IRC | 19:44 | |
| *** salv-orlando has quit IRC | 19:44 | |
| *** mylu has joined #openstack-infra | 19:45 | |
| pabelanger | okay, puppet running again | 19:46 |
| *** matt-borland has joined #openstack-infra | 19:47 | |
| *** cloudtrainme has quit IRC | 19:47 | |
| clarkb | pabelanger: it looks happy? | 19:47 |
| *** tosky has quit IRC | 19:47 | |
| pabelanger | clarkb: so far, just doing afs right now | 19:47 |
| clarkb | I am currently rereading my volume and swap setup changes to make sure I get /opt configured properly on the first attempt | 19:48 |
| *** tosky has joined #openstack-infra | 19:48 | |
| clarkb | I think it is going to want to put the other portion of the ephemeral drive under /opt first | 19:48 |
| clarkb | ya worried we will have two /opt entries in fstab so need to untangle that first | 19:49 |
| sc68cal | hey, so yolanda 's status wiped all the openstack channel topics. was that intentional> | 19:49 |
| clarkb | sc68cal: yes when the problem is corrected it gets reset | 19:49 |
| *** tmorin has joined #openstack-infra | 19:49 | |
| yolanda | ah yes, to alert about npm issues. But they are still persisting right? | 19:49 |
| clarkb | yolanda: they are | 19:49 |
| clarkb | but should go away shortly with pabelanger getting puppet to run again | 19:50 |
| yolanda | ok, just set to ok when it's fixed, or i 'll check tomorrow morning as well. I wanted to send the status, because we got several queries this morning about this problem | 19:50 |
| *** e0ne has quit IRC | 19:50 | |
| pabelanger | Yup, will confirm JJB applies aswell | 19:51 |
| pabelanger | likely needs to be done manually to pickup the puppet update | 19:51 |
| *** hashar has joined #openstack-infra | 19:51 | |
| *** ajmiller has quit IRC | 19:53 | |
| *** salv-orlando has joined #openstack-infra | 19:54 | |
| *** salv-orlando has quit IRC | 19:54 | |
| *** annegentle has quit IRC | 19:55 | |
| *** salv-orlando has joined #openstack-infra | 19:55 | |
| *** eharney has quit IRC | 19:55 | |
| clarkb | ok I think have have sufficiently surgeried the swap ephemeral disk /opt stuff in order to get my volume on /opt without fuss | 19:56 |
| *** julim has joined #openstack-infra | 19:57 | |
| anteaya | crinkle: I ran sudo puppet apply local.pp --modulepath=/etc/puppet/modules:system-config/modules -l mediawiki-puppet.log and it has finished but I dont't see a mediawiki-puppet.log file in my homedir | 19:57 |
| anteaya | crinkle: would it be somewhere else? | 19:57 |
| *** nadya has joined #openstack-infra | 19:58 | |
| *** maishsk has quit IRC | 19:58 | |
| tmorin | hi clarkb, dougwig: I've finished investigating the issue I was hitting with a job in the past days; the issue is in fact not related to our code, and is quite obscure ; the real issue was hard to see because our devstack plugin.sh code was not properly restoring bash xtrace status | 19:58 |
| anteaya | crinkle: it isn't in /var/log either | 19:58 |
| crinkle | anteaya: I think it would just be in your current working directory, if not maybe try using an absolute path like /tmp/mediawiki-puppet.log | 19:59 |
| *** piet has joined #openstack-infra | 19:59 | |
| anteaya | crinkle: okay I will try again | 19:59 |
| *** mylu has quit IRC | 19:59 | |
| tmorin | clarkb, dougwig: I reported a but at https://bugs.launchpad.net/devstack/+bug/1567052 to describe the issue I see with stack.sh | 19:59 |
| openstack | Launchpad bug 1567052 in devstack "stack.sh exiting with 2, with no obvious error" [Undecided,New] | 19:59 |
| dougwig | tmorin: so do you have any insight into why stack.sh is returning 2? | 20:00 |
| *** roxanaghe has quit IRC | 20:00 | |
| *** maishsk has joined #openstack-infra | 20:00 | |
| tmorin | dougwig: only a start of an idea | 20:01 |
| tmorin | dougwig: the issue is not triggerd if VERBOSE=False | 20:01 |
| tmorin | dougwig: I suspect echo_summary of the fd magic at the end of stack.sh | 20:01 |
| dougwig | since you have a testbed, the only hope of that bug being closed is if you chase it down further. | 20:01 |
| *** cloudtrainme has joined #openstack-infra | 20:01 | |
| tmorin | dougwig: I can't test more in the gate, but I can try to reproduce in a personal VM | 20:02 |
| tmorin | dougwig: I'll have to tweak stack.sh to find where the issue can be | 20:02 |
| clarkb | Inner Exception: Quota exceeded for ram: Requested 8192, but already used 1148928 of 1152000 ram (HTTP 403) ok quota issues confirmed | 20:03 |
| dougwig | tmorin: or just use sed on devstack-vm-gate.sh to enable -x. | 20:03 |
| clarkb | afstest is going to go away | 20:03 |
| dougwig | on the stack.sh invocation | 20:03 |
| tmorin | dougwig: that's crude, but yeah, why not... | 20:03 |
| *** maishsk_ has joined #openstack-infra | 20:04 | |
| *** mrmartin has quit IRC | 20:04 | |
| *** mgagne_ is now known as mgagne | 20:04 | |
| *** maishsk has quit IRC | 20:05 | |
| clarkb | with afstest gone shade launch node is doing things \o/ | 20:05 |
| *** cbader has quit IRC | 20:05 | |
| *** kmartin has quit IRC | 20:06 | |
| *** asselin_ has quit IRC | 20:06 | |
| anteaya | clarkb: congratulations | 20:06 |
| *** fernnest has quit IRC | 20:06 | |
| clarkb | the hand off of services is going to be hard beacuse we can only sutdown the other server. This means that we will likely have to manually delete all of the instances from all of the cloud regions and from jenkins | 20:06 |
| anteaya | clarkb: should we have space at the summit do you think about which long running servers to cull? | 20:06 |
| *** fernnest has joined #openstack-infra | 20:07 | |
| *** cbader has joined #openstack-infra | 20:07 | |
| *** kmartin has joined #openstack-infra | 20:07 | |
| anteaya | clarkb: we seem to be encountering this issue frequently | 20:07 |
| *** maishsk has joined #openstack-infra | 20:07 | |
| *** asselin has joined #openstack-infra | 20:07 | |
| clarkb | anteaya: I think fungi did an audit and didn't find anything (afstest was the one that came up and jeblair just said a little bit ago it could be deleted) | 20:07 |
| anteaya | clarkb: can the current nodepool still operate until the end of the day? | 20:07 |
| clarkb | anteaya: part of the problem is we have a giant new review.o.o sitting around doing nothing but eating reasources | 20:07 |
| clarkb | anteaya: it should continue operating until it decides to quit | 20:07 |
| anteaya | ah yes, so problem with quota will be solved in a week | 20:08 |
| clarkb | basically we can't fix anything on it if something goes wrong and we can't hold isntances etc | 20:08 |
| *** maishsk_ has quit IRC | 20:08 | |
| clarkb | the first thing I will do on new noepool is make sure image builds work properly there and we get images into the clouds for it | 20:08 |
| anteaya | clarkb: do you think maybe we should post to the dev list, explaining the issue and notifying people? | 20:09 |
| clarkb | while leaving the main daemon off until we do the messy flipover | 20:09 |
| *** eharney has joined #openstack-infra | 20:09 | |
| clarkb | anteaya: if we end up doing it during a busy time yes, otherwise we should be able to do it without anyone noticing | 20:09 |
| anteaya | perhaps we can plan for a cutover and just do it before the intended cutover if necessary | 20:09 |
| *** rcernin has joined #openstack-infra | 20:09 | |
| clarkb | if we have images built and ready to go it should be as easy as update dns, turn on new nodepool, turn off old nodepool, cleanup | 20:10 |
| anteaya | so just notify it is an emergencey | 20:10 |
| *** fitoduarte has joined #openstack-infra | 20:10 | |
| anteaya | how long do you think once new nodepool has images built? | 20:10 |
| anteaya | how long would the cutover take I mean | 20:10 |
| *** rbrndt has quit IRC | 20:10 | |
| clarkb | at this point thats about an all day thing | 20:10 |
| anteaya | oh my | 20:10 |
| clarkb | we can probably plan for cutover tomorrow morning | 20:10 |
| clarkb | anteaya: image uploads are terribly slow | 20:10 |
| *** yamamoto has joined #openstack-infra | 20:11 | |
| anteaya | ah all day to prep new nodepool, got it | 20:11 |
| anteaya | how long for the cutover, so how long an outage for the public? | 20:11 |
| clarkb | public should never notice an outage | 20:11 |
| anteaya | cool | 20:11 |
| clarkb | just potentially jobs launching more slowly until we delete instances associated iwth old nodepool | 20:11 |
| anteaya | yup, okay | 20:12 |
| anteaya | so normal day | 20:12 |
| clarkb | pabelanger: puppet and jenkins and all that happy? | 20:12 |
| clarkb | pabelanger: did ssh fail against nodepool and move on? | 20:12 |
| *** pradk_ has joined #openstack-infra | 20:12 | |
| clarkb | ugh its final release time this week isn't it | 20:12 |
| anteaya | clarkb: can any non root do anything to help the prep of new nodepool? | 20:12 |
| pabelanger | clarkb: yup, doing JJB just now | 20:12 |
| pabelanger | and nodepool failed and moved on | 20:12 |
| clarkb | anteaya: not really, most of this requires cloud credentials | 20:13 |
| *** pradk_ has quit IRC | 20:13 | |
| *** yamahata has joined #openstack-infra | 20:13 | |
| *** rockyg has joined #openstack-infra | 20:13 | |
| anteaya | clarkb: okay | 20:13 |
| clarkb | anteaya: maybe find out what the release managers have planned for the rest of the week | 20:13 |
| pabelanger | and running JJB manually now | 20:13 |
| clarkb | and we can plan to work around their schedule | 20:14 |
| pabelanger | clarkb: puppet is just finishing up | 20:14 |
| anteaya | clarkb: doing that now | 20:14 |
| anteaya | clarkb: ttx is already offline, waiting to hear from dims | 20:14 |
| *** pcaruana has quit IRC | 20:14 | |
| *** pradk has quit IRC | 20:14 | |
| *** matrohon has joined #openstack-infra | 20:15 | |
| pabelanger | clarkb: so the plan is to swap out nodepool.o.o with new server? | 20:15 |
| clarkb | pabelanger: yup its up now at 23.253.73.160 | 20:16 |
| clarkb | and /opt lgtm and we have 8GB of swap | 20:16 |
| * dims peeks | 20:16 | |
| clarkb | I am going to work on having it build new images | 20:17 |
| pabelanger | ack | 20:17 |
| *** tosky has quit IRC | 20:17 | |
| *** pradk has joined #openstack-infra | 20:18 | |
| clarkb | The program 'screen' is currently not installed. we should fix that in our images | 20:18 |
| *** tosky has joined #openstack-infra | 20:18 | |
| *** nadya has quit IRC | 20:18 | |
| dims | anteaya : clarkb : as long as we are on our feet by EU morning, we should be fine | 20:18 |
| clarkb | dims: well we are on our feet right now | 20:18 |
| dims | cool. | 20:19 |
| clarkb | the old nodepool functions, its just that if it breaks we won't be able to touch it | 20:19 |
| dims | ack | 20:19 |
| krotscheck | Significant updates to the NPM issue upstream. Looks like they finally have a fix, that's being (slowly) rolled out across their infrastructure. https://github.com/npm/npm/issues/12196 | 20:19 |
| *** yamamoto has quit IRC | 20:19 | |
| clarkb | maybe the thing to do is get image uploads sorted out on new nodepool so it is primed and ready then ask you all aobut when we should cutover | 20:19 |
| *** rbrndt_ has joined #openstack-infra | 20:20 | |
| *** jcooley_ has joined #openstack-infra | 20:20 | |
| anteaya | krotscheck: yay a fix | 20:20 |
| dims | clarkb : per schedule, we should be done around 1400 UTC so if alls' well we can cut over anytime after that | 20:20 |
| pabelanger | clarkb: maybe switch UPLOAD_WORKERS manually? to help speed uploads | 20:20 |
| clarkb | pabelanger: I have had to edit the nodepool.yaml to use a localhost geard due to firewalls | 20:21 |
| clarkb | dims: rgr | 20:21 |
| clarkb | pabelanger: I will be running this all in a screen owned by me | 20:21 |
| clarkb | pabelanger: I can do that, the change merged and added the cli flag? | 20:21 |
| pabelanger | clarkb: not merged yet, I'll work on it tonight | 20:22 |
| clarkb | ah well the nI will start it as normal, let it build hte images then can start extras to pick up slack after | 20:22 |
| *** jamesmcarthur has joined #openstack-infra | 20:22 | |
| *** e0ne has joined #openstack-infra | 20:22 | |
| clarkb | ubuntu-trusty is building (started wit hthat one because I am most familiar with it and will have an easier time debugging if anything goes funny again) | 20:24 |
| anteaya | crinkle: absolute path worked, thank you | 20:24 |
| clarkb | pabelanger: on the bright side this has gotten us a bigger /opt and clean dib cache :) | 20:24 |
| clarkb | pabelanger: we can see how te image sizes are affected | 20:24 |
| pabelanger | clarkb: woot | 20:24 |
| crinkle | anteaya: great | 20:25 |
| anteaya | :) | 20:25 |
| *** ams_ has joined #openstack-infra | 20:25 | |
| clarkb | greghaynes: thinking about how we can avoid this in the future I wonder if dib could run against a mostly ro fs? | 20:26 |
| clarkb | greghaynes: that way any changes to the host system outside of caches would fail | 20:26 |
| *** mylu has joined #openstack-infra | 20:27 | |
| pabelanger | clarkb: was also thinking, maybe we should chattr our ssh keys too | 20:27 |
| *** e0ne has quit IRC | 20:28 | |
| *** maishsk has quit IRC | 20:29 | |
| clarkb | pabelanger: it might be somewhat hard to do based on how the boot process works | 20:29 |
| *** maishsk has joined #openstack-infra | 20:29 | |
| clarkb | pabelanger: we don't have keys on first boot and they have to be written and thats handled by external tools | 20:29 |
| pabelanger | clarkb: Ya, that is true | 20:29 |
| *** yamahata has quit IRC | 20:30 | |
| *** vgridnev has quit IRC | 20:30 | |
| clarkb | ok so npm should be happier, project creation should've happened, puppet is running again, the only outstanding item I know of is nodepool | 20:30 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-pip: Add new pip provider to use pypi mirrors https://review.openstack.org/302396 | 20:31 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-pip: Fix pip executable path for RH https://review.openstack.org/302431 | 20:31 |
| pabelanger | Yup, puppet jobs appear happier, just confirming | 20:31 |
| anteaya | a lot of the errors being thrown in the puppet-mediawiki module are from trying to install these packages: http://git.openstack.org/cgit/openstack-infra/puppet-mediawiki/tree/manifests/image_scaler.pp | 20:31 |
| *** annegentle has joined #openstack-infra | 20:31 | |
| clarkb | I will keep an eye on the image builds for the rest of today and queue up uploads before I EOD | 20:31 |
| anteaya | if we have disabled image uploaded do we need image scaling? | 20:31 |
| *** annegentle has quit IRC | 20:31 | |
| anteaya | uploading | 20:32 |
| clarkb | anteaya: I don't think all iamge uploads are disabled | 20:32 |
| anteaya | ah | 20:32 |
| clarkb | I seem to recall network diagram type things living on the wiki | 20:32 |
| *** smarcet has joined #openstack-infra | 20:32 | |
| *** annegentle has joined #openstack-infra | 20:32 | |
| crinkle | jeblair: fungi nibalizer when you are back from traveling I started working on https://review.openstack.org/302396 as a hack to force puppet to use our pip mirrors, not sure if it's too crazy | 20:32 |
| *** roxanaghe has joined #openstack-infra | 20:32 | |
| anteaya | maybe we just diabled pdfs, hmmmm | 20:32 |
| *** annegentle has quit IRC | 20:32 | |
| *** amrith is now known as _amrith_ | 20:32 | |
| pabelanger | anteaya: yes | 20:32 |
| anteaya | pabelanger: did we disable uploading jpegs do you know? | 20:33 |
| *** annegentle has joined #openstack-infra | 20:33 | |
| pabelanger | no, should still work | 20:33 |
| pabelanger | as long as they are defaults | 20:33 |
| anteaya | I don't know what you mean | 20:33 |
| *** vgridnev has joined #openstack-infra | 20:33 | |
| *** ihrachys has quit IRC | 20:34 | |
| *** jsavak has quit IRC | 20:35 | |
| *** jsavak has joined #openstack-infra | 20:36 | |
| *** yfried has joined #openstack-infra | 20:36 | |
| anteaya | crinkle: where is the installation command coming from here? http://paste.openstack.org/show/493231/ | 20:37 |
| *** sdake_ has joined #openstack-infra | 20:37 | |
| *** matrohon has quit IRC | 20:37 | |
| anteaya | crinkle: this is the list of packages it is working through: http://git.openstack.org/cgit/openstack-infra/puppet-mediawiki/tree/manifests/image_scaler.pp | 20:37 |
| anteaya | crinkle: many of which it can't find candidates for, and hence can't download | 20:37 |
| *** flepied1 has joined #openstack-infra | 20:37 | |
| *** flepied has quit IRC | 20:37 | |
| pabelanger | anteaya: here is our settings for image uploads: https://github.com/openstack-infra/puppet-mediawiki/blob/master/templates/Settings.php.erb#L101 | 20:38 |
| pabelanger | which means we are using the mediawiki defaults + svg files | 20:38 |
| ianw | greghaynes: doh, i hate doing things in a rush | 20:38 |
| ianw | pabelanger: i'm assume we know about designate-tempest-plugin.git clone failing the builds? | 20:39 |
| anteaya | pabelanger: thank you, http://git.openstack.org/cgit/openstack-infra/puppet-mediawiki/tree/templates/Settings.php.erb#n101 is helpful | 20:39 |
| crinkle | anteaya: that command comes from the package resource | 20:39 |
| clarkb | ianw: pabelanger yes that was corrected by getting puppet running again | 20:39 |
| *** sdake has quit IRC | 20:39 | |
| pabelanger | clarkb: ianw: Ya, puppet was block for about 4 hours today | 20:39 |
| *** ajmiller has joined #openstack-infra | 20:39 | |
| clarkb | theoretically due to ssh weirdness to nodepool | 20:39 |
| pabelanger | Ya, I think so | 20:40 |
| krotscheck | Oooh, we have puppet again? | 20:40 |
| pabelanger | seen weird things with ansible if you disrupt the SSH connection when in progress | 20:40 |
| crinkle | anteaya: it might be that those packages aren't available on the release you're running, maybe they were there in precise but you're running something like trusty? | 20:40 |
| anteaya | crinkle: uuh, I don't know what that means, thanks though, I'll look around | 20:40 |
| *** dizquierdo has joined #openstack-infra | 20:40 | |
| clarkb | the dib build just finished cloning all the repos so the repos appear all happy | 20:40 |
| *** kzaitsev_mb has joined #openstack-infra | 20:41 | |
| anteaya | pabelanger: oh so you mean that if jpegs are default for the image magik library which the wiki uses, they they can be uploaded? | 20:41 |
| crinkle | anteaya: by package resource i mean the package {} part of this here http://git.openstack.org/cgit/openstack-infra/puppet-mediawiki/tree/manifests/image_scaler.pp#n4 | 20:41 |
| anteaya | crinkle: thank you | 20:41 |
| clarkb | we still download openvz image :| | 20:41 |
| clarkb | I am just goign to delete that | 20:41 |
| clarkb | it never worked aiui | 20:41 |
| clarkb | and no one is working on it now | 20:41 |
| anteaya | crinkle: I am running trusty, yes, I don't think precise was available to me | 20:42 |
| *** sdake has joined #openstack-infra | 20:42 | |
| crinkle | anteaya: if http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n390 is to be believed then wiki.o.o is running precise | 20:43 |
| anteaya | yes it is | 20:43 |
| *** sdake_ has quit IRC | 20:44 | |
| pabelanger | crinkle: anteaya: I've started work on migrating to precise for wiki.o.o, but there is puppet issues: https://review.openstack.org/#/c/285405/ | 20:44 |
| pabelanger | I haven't had time to continue with it | 20:44 |
| pabelanger | was likely going to do so at the summit | 20:44 |
| anteaya | pabelanger: thank you | 20:44 |
| *** trown is now known as trown|outtypewww | 20:44 | |
| anteaya | pabelanger: well if you have no objection if I can figure out what packages trusty offers for image_scaling I can add that to the patch | 20:45 |
| pabelanger | anteaya: Sure, we can update the depends-on field in system-config too | 20:45 |
| *** sridhar_ram has joined #openstack-infra | 20:46 | |
| ianw | pabelanger: if you have a minute, could you kick off a run of "propose-devstack-plugins-list". it's been proposing broken stuff for quite a while but fixes should have merged | 20:46 |
| *** rockyg has quit IRC | 20:47 | |
| pabelanger | ianw: sure, give me a few mins to get the syntax | 20:47 |
| anteaya | hashar: have you any thoughts on how to address the fact that mediawiki uses imagemagik for image scaling and imagemagik requires ffmpeg yet trusty doesn't ship with ffmpeg? | 20:47 |
| ianw | no rush, low priority | 20:47 |
| *** annegentle has quit IRC | 20:49 | |
| *** vgridnev has quit IRC | 20:49 | |
| *** tmorin has quit IRC | 20:49 | |
| clarkb | pabelanger: we will also need to do the nodepool install using older nodepool to avod the shade/oscc problems | 20:50 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Move gate-grenade-dsvm-neutron-dvr-multinode into check queue https://review.openstack.org/299844 | 20:51 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Make grenade-dsvm-neutron-multinode voting https://review.openstack.org/299843 | 20:51 |
| pabelanger | clarkb: ya | 20:51 |
| *** salv-orl_ has joined #openstack-infra | 20:51 | |
| clarkb | its not a problem for imgae builds so won't worry about it now | 20:52 |
| jhesketh | Morning | 20:53 |
| *** annegentle has joined #openstack-infra | 20:54 | |
| anteaya | morning jhesketh | 20:54 |
| *** nwkarsten has quit IRC | 20:54 | |
| *** ddieterly is now known as ddieterly[away] | 20:54 | |
| * jhesketh grumbles about daylight savings being over in australia | 20:55 | |
| *** salv-orlando has quit IRC | 20:55 | |
| *** nwkarsten has joined #openstack-infra | 20:55 | |
| anteaya | jhesketh: :( | 20:55 |
| anteaya | jhesketh: what time is it for you now? | 20:55 |
| anteaya | 3am? | 20:55 |
| *** annegentle has quit IRC | 20:55 | |
| pabelanger | great, puppet-jobs are happy. | 20:56 |
| *** knikolla has quit IRC | 20:57 | |
| pabelanger | just waiting for JJB to finish up on jenkins01, 02, and 04 before notifying puppet-openstack team | 20:57 |
| *** aeng_away is now known as aeng | 20:58 | |
| *** asettle has joined #openstack-infra | 20:59 | |
| *** nwkarsten has quit IRC | 20:59 | |
| *** _ari_ has quit IRC | 20:59 | |
| *** _ari_ has joined #openstack-infra | 21:00 | |
| *** burgerk has quit IRC | 21:01 | |
| *** EricGonczer_ has quit IRC | 21:01 | |
| *** Sukhdev has joined #openstack-infra | 21:01 | |
| jhesketh | anteaya: not that bad actually, 7am | 21:01 |
| jhesketh | yesterday's infra meeting was 5, so that was less fun | 21:02 |
| *** tphummel has joined #openstack-infra | 21:02 | |
| ianw | pabelanger: what's the deal with gate-infra-puppet-apply-devstack-centos7 in https://review.openstack.org/#/c/301989/ ? fails & "File not found" in the logs? | 21:02 |
| pabelanger | sc`: let me know if you fixed your requests issue, if not I can provide a workaround for now | 21:02 |
| anteaya | jhesketh: cool | 21:02 |
| *** rossella_s has quit IRC | 21:03 | |
| pabelanger | ianw: Ya, I noticed that too | 21:03 |
| pabelanger | ianw: haven't looked yet | 21:03 |
| anteaya | jhesketh: yes 5 is less fun than 7 | 21:03 |
| *** rossella_s has joined #openstack-infra | 21:03 | |
| pabelanger | ianw: pretty sure we did a commit recently to use requestexception | 21:03 |
| sc` | pabelanger: no dice on my end - $work beckons. is the workaround the same one for puppet? | 21:03 |
| *** Goneri has quit IRC | 21:04 | |
| pabelanger | sc`: yes, sudo pip uninstall python-requests -y || true should do it. | 21:04 |
| sc` | let's do that for now | 21:04 |
| sc` | i intend to revisit the jenkins stuff for chef to make sure it all makes sense. haven't had time | 21:04 |
| *** jistr|off has quit IRC | 21:04 | |
| pabelanger | sc`: okay, I'll get up a patch and CC you on it | 21:05 |
| sc` | right on. thanks | 21:05 |
| *** ddieterly[away] is now known as ddieterly | 21:05 | |
| *** EricGonczer_ has joined #openstack-infra | 21:06 | |
| clarkb | greghaynes: pabelanger first image is getting converted to qcow2 now | 21:06 |
| clarkb | builds should get faster as we cache more stuff | 21:06 |
| *** annegentle has joined #openstack-infra | 21:06 | |
| pabelanger | ack | 21:07 |
| greghaynes | vroom vroom | 21:07 |
| *** sridhar_ram has quit IRC | 21:07 | |
| clarkb | once they are all built I will have to queue up all the uploads | 21:07 |
| clarkb | fun fun | 21:07 |
| *** jsavak has quit IRC | 21:09 | |
| *** _ari_ has quit IRC | 21:09 | |
| *** |-paul-| has joined #openstack-infra | 21:09 | |
| *** baoli_ has quit IRC | 21:09 | |
| *** sridhar_ram has joined #openstack-infra | 21:10 | |
| pabelanger | ianw: https://review.openstack.org/#/c/298059/ is the related commit | 21:11 |
| *** claudiub|2 has joined #openstack-infra | 21:11 | |
| ianw | jhesketh: is zuul-swift-upload-logs-with-console not uploading on failure a known thing? | 21:12 |
| pabelanger | ianw: so, maybe we missed requestsexceptions on devstack-centos7 some how? | 21:12 |
| ianw | oh, ok, that's maybe making sense https://jenkins03.openstack.org/job/gate-infra-puppet-apply-devstack-centos7/28/console | 21:12 |
| jhesketh | ianw: it's not known to me | 21:12 |
| *** knikolla has joined #openstack-infra | 21:13 | |
| *** doug-fish has quit IRC | 21:13 | |
| ianw | jhesketh: yeah, it looks like change was made to use requestexceptions, which causes the upload to fail on platforms without it | 21:13 |
| jhesketh | yeah, just looking | 21:13 |
| *** doug-fish has joined #openstack-infra | 21:13 | |
| *** doug-fish has quit IRC | 21:14 | |
| ianw | pabelanger: hmm, ok, i guess we still care a little about devstack-centos7 at this point | 21:14 |
| *** doug-fish has joined #openstack-infra | 21:14 | |
| pabelanger | ianw: for another day or two | 21:14 |
| *** cdent has quit IRC | 21:14 | |
| *** julim has quit IRC | 21:14 | |
| pabelanger | almost done the migration to centos-7 | 21:14 |
| pabelanger | puppet-jobs migrated, they were the largest | 21:14 |
| *** claudiub has quit IRC | 21:15 | |
| pabelanger | we do have an issue with python-requests, but I need food first before diving into that | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| *** kgiusti has left #openstack-infra | 21:15 | |
| clarkb | new image sizes appear to be 13GB ish and 5.5GB ish | 21:15 |
| clarkb | so maybe we should try to dleete teh cache periodically (once a month?) | 21:15 |
| clarkb | accumulated cruft seems to expand image sizes | 21:16 |
| *** dizquierdo has quit IRC | 21:16 | |
| mordred | clarkb: I miss anyhting fun today? | 21:16 |
| *** Sukhdev has quit IRC | 21:16 | |
| clarkb | mordred: yes all the fun | 21:16 |
| mordred | neat! | 21:16 |
| clarkb | but it is under control now | 21:16 |
| mordred | clarkb: I heard something about wanting exisitng shells on nodepool | 21:16 |
| clarkb | mordred: yes dib release deleted ssh host keys on nodepool | 21:16 |
| clarkb | mordred: so you can't ssh to it now | 21:17 |
| clarkb | I have a new nodepool built and it is building images so that wecan switch to it | 21:17 |
| mordred | clarkb: yay! | 21:17 |
| clarkb | I bumped the size of /opt to 1TB | 21:17 |
| mordred | clarkb: also wow | 21:17 |
| greghaynes | Yea, it was a fun morning | 21:17 |
| clarkb | and started on a fresh cache which seems to make the images smaller by ~4GB in the raw/vhd case | 21:17 |
| mordred | so - it just has _no_ host keys? | 21:17 |
| clarkb | mordred: correct | 21:17 |
| mordred | clarkb: wow with fresh cache | 21:18 |
| clarkb | mordred: and cloud init won't make them after the first boot even if they aren't there | 21:18 |
| mordred | we should lok in to that | 21:18 |
| greghaynes | clarkb: what distro? | 21:18 |
| mordred | clarkb: that seems like a bug in cloud init | 21:18 |
| greghaynes | (for the images) | 21:18 |
| clarkb | greghaynes: ubuntu-trusty | 21:18 |
| *** jamesmcarthur has quit IRC | 21:18 | |
| *** doug-fish has quit IRC | 21:18 | |
| mordred | clarkb: I hate to even say this ... but you know that you can have rackspace do a password reset on things | 21:18 |
| greghaynes | oh, right, we copy in the whole package cache? | 21:18 |
| clarkb | mordred: because at first I thought maybe we can just reboot but me | 21:18 |
| mordred | clarkb: and we do not disable nova-agent on our servers | 21:18 |
| clarkb | mordred: I would rather not grow dependent on nova-agent :) | 21:18 |
| mordred | clarkb: so it should work | 21:19 |
| mordred | clarkb: no, we shold not depend on it | 21:19 |
| clarkb | its there for reasons but I need not use it | 21:19 |
| mordred | just saying it would likely allow you to get console access | 21:19 |
| *** jamesmcarthur has joined #openstack-infra | 21:19 | |
| greghaynes | Yea, and theres other things we could do too, singleuser and such | 21:19 |
| clarkb | if nodepool the service crashes hard then we can do that | 21:19 |
| clarkb | otherwise lets just switch it out and gain benefits of cleaned up host | 21:19 |
| *** tmorin has joined #openstack-infra | 21:19 | |
| mordred | ++ | 21:20 |
| *** dprince has quit IRC | 21:20 | |
| greghaynes | clarkb: I wonder if the cache size is stale packages in the cache | 21:20 |
| clarkb | mordred: in other news I discovered yesterday that ipv6 info is not in the config drive data on rax | 21:20 |
| clarkb | mordred: so none of our slaves are ipv6 in rax if using glean | 21:20 |
| greghaynes | clarkb: I wonder if SLAAC/RA works? | 21:20 |
| clarkb | greghaynes: I don't think so the total apt cache size was like 700MB when I chceked | 21:20 |
| clarkb | greghaynes: dunno, I am on a thread that jayf started asking about it | 21:21 |
| *** EricGonczer_ has quit IRC | 21:21 | |
| clarkb | greghaynes: hopefully have an answer soonihs | 21:21 |
| greghaynes | I hope thats it | 21:21 |
| greghaynes | that would make me very happy | 21:21 |
| *** doug-fish has joined #openstack-infra | 21:21 | |
| mordred | clarkb: spectacular | 21:21 |
| clarkb | hrm I don't actually see an apt cache in the new dib cache | 21:22 |
| clarkb | 5.2G source-repositories | 21:23 |
| clarkb | thats about half the size of the old 11G source-repositories | 21:23 |
| bkero | Is that good or something-wrong? | 21:23 |
| greghaynes | huh, I wonder what all you download there | 21:23 |
| clarkb | I think its good | 21:23 |
| greghaynes | thats a lot of source-repositories | 21:23 |
| clarkb | greghaynes: its probably crufty repos that need repack/gc | 21:23 |
| greghaynes | ah | 21:23 |
| clarkb | fresh clone gets the repacked goodness from mirrors | 21:23 |
| *** smarcet has quit IRC | 21:24 | |
| greghaynes | ya, makes sense | 21:24 |
| hashar | anteaya: no idea about mediawiki/imagemagick/ffmpeg. Maybe we ended up backporting ffmpeg / building our own imagemagick | 21:24 |
| clarkb | mordred greghaynes: you might want to double check that ubuntu-minimal is caching properly | 21:24 |
| mordred | maybe we should run a cron on nodepool to pack the cache repos | 21:24 |
| clarkb | I see no cached apt packages | 21:24 |
| mordred | clarkb: sweet | 21:24 |
| greghaynes | hrm | 21:24 |
| * greghaynes looks | 21:24 | |
| *** sdague has quit IRC | 21:24 | |
| anteaya | hashar: okay thanks | 21:24 |
| anteaya | hashar: if you ever find out, I'm curious but not a rush | 21:25 |
| mordred | clarkb: so it's possible that means that we've been running with no locally cached apt pacakges on trusty nodes for $time | 21:25 |
| clarkb | mordred: yes | 21:25 |
| clarkb | mordred: with the mirror picking up the slack likely | 21:25 |
| mordred | yah | 21:25 |
| hashar | anteaya: prod apparently uses Trusty package at version 6.7.7.10-6ubuntu3 http://packages.ubuntu.com/trusty/imagemagick | 21:25 |
| clarkb | but we want dib to cache etiehr way to make offline builds possible and online faster | 21:25 |
| mordred | I mention that because we may have unwittingly tested whether or not the mirrors are good enough to handle it without problems | 21:25 |
| *** doug-fish has quit IRC | 21:25 | |
| clarkb | mordred: you can login to a machine running tests and check | 21:26 |
| hashar | anteaya: maybe ffmpeg ended up being dropped due to copyright/patent or maybe it is only in universe | 21:26 |
| clarkb | mordred: but you will have to look at a console log for ip info since nodepool is inaccessible | 21:26 |
| clarkb | mordred: or boot off the same image, etc | 21:26 |
| anteaya | hashar: it was dropped in trusty, yes and I'm thinking it is due to licesing | 21:26 |
| anteaya | hashar: haven't checked the univers | 21:26 |
| anteaya | e | 21:26 |
| hashar | anteaya: so wmf probably have built our own and to respect the license madness do not publish the .deb anyway | 21:26 |
| jhesketh | ianw: I wonder if the virtualenv failed to install requestsexceptions | 21:26 |
| mordred | clarkb: I'm about to board a plane | 21:26 |
| mordred | clarkb: but I will do that soonish | 21:27 |
| *** rcernin has quit IRC | 21:27 | |
| jhesketh | ianw: on the node build | 21:27 |
| hashar | anteaya: folks in #wikimedia-operations would know | 21:27 |
| hashar | anteaya: though most of the Debian Developer from the ops team are in europe and probably asleep right now | 21:27 |
| ianw | jhesketh: it did -- if you search in http://nodepool.openstack.org/dib.devstack-centos7.log.2016-04-05_15 for 90-venv-swift-logs | 21:27 |
| ianw | jhesketh: it is clearly not running https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/install.d/90-venv-swift-logs | 21:27 |
| anteaya | hashar: coold thanks, I will continue finding font packages and then will wander over | 21:28 |
| *** knikolla has quit IRC | 21:28 | |
| anteaya | hashar: my thanks :) | 21:28 |
| *** knikolla has joined #openstack-infra | 21:29 | |
| *** regXboi has quit IRC | 21:29 | |
| *** tiswanso has quit IRC | 21:31 | |
| hashar | anteaya: what I suspect is WMF ended up getting the ffmpeg from Debian and building it against Trusty. And I found our task a comment as the build detail https://phabricator.wikimedia.org/T103335#1524063 | 21:31 |
| hashar | anteaya: the whole task being about ffmpeg/libav ubuntu/debian. With MoritzMuehlenhoff knowing a lot about licensing issue, godog the op that handled the build and "brion" the lead dev for video stuff | 21:33 |
| *** jamesmcarthur has quit IRC | 21:33 | |
| clarkb | I think tomorrow might be a day off once I get new nodepool running | 21:33 |
| *** jamesmcarthur has joined #openstack-infra | 21:33 | |
| hashar | anteaya: I have no idea about the legal implication if you were to distribute the ffmpeg .deb . I am not a lawyer ;-/ | 21:34 |
| jhesketh | ianw: the install for the venv doesn't look complete: + sudo -H /usr/zuul-swift-logs-env/bin/pip install python-magic argparse requests glob2 | 21:34 |
| jhesketh | ianw: can't tell why | 21:34 |
| ianw | jhesketh: yeah, it's like it hasn't picked up the new 90-venv-swift-logs file | 21:34 |
| jhesketh | right | 21:34 |
| ianw | jhesketh: of course, any evidence of that is now pretty much gone as the nodepool host is being rebuilt | 21:35 |
| clarkb | puppet would not have been able to run there depending on when you added that | 21:35 |
| clarkb | you can check logs on the new server, ip is in scrollback or I can dig it upagain | 21:35 |
| ianw | clarkb: that's the weird thing -> 84e76e3f720339db9b0cfaa60ab0399701bc67f1 Sun Mar 27 21:19:41 2016 +0200 | 21:35 |
| *** knikolla has quit IRC | 21:36 | |
| *** amitgandhinz has quit IRC | 21:37 | |
| *** knikolla has joined #openstack-infra | 21:37 | |
| *** yfried has quit IRC | 21:37 | |
| hashar | anteaya: heading to bed sorry. Poke #wikimedia-operations as needed :) | 21:38 |
| ianw | so fedora23 didn't pick it up either | 21:38 |
| anteaya | hashar: thank you will likely be another day, not today | 21:38 |
| anteaya | hashar: sleep well! | 21:39 |
| clarkb | ianw: jhesketh is it +x? | 21:39 |
| hashar | anteaya: Danke! | 21:39 |
| ianw | but ubuntu trusty did -> http://nodepool.openstack.org/dib.ubuntu-trusty.log.2016-04-05_17 | 21:39 |
| *** ayoung has quit IRC | 21:39 | |
| anteaya | bitte | 21:39 |
| jhesketh | clarkb: it seems to be using an old version of nodepool/elements/nodepool-base/install.d/90-venv-swift-logs | 21:40 |
| jhesketh | possibly an old version of other elements too | 21:40 |
| *** tmorin has quit IRC | 21:40 | |
| ianw | jhesketh: but only devstack-centos & fedora23, it seems ... | 21:40 |
| clarkb | jhesketh: ianw: what about the new server? really we can't debug anyhting on old server so its not worth much effort | 21:40 |
| *** rfolco has quit IRC | 21:40 | |
| *** piet has quit IRC | 21:40 | |
| ianw | clarkb: well, i'll have to check the build logs | 21:41 |
| *** EricGonczer_ has joined #openstack-infra | 21:41 | |
| ianw | but how weird; devstack-centos7 & fedora23 both seem to be using an old 90-venv-swift-logs install file, while everything else has picked up the change | 21:42 |
| clarkb | ubuntu-trusty and centos-7 are mostly done at this point on new one | 21:42 |
| greghaynes | w00t | 21:42 |
| clarkb | fedora-23 will be next then debian-jessie then devstack-centos7 then xenial | 21:42 |
| ianw | clarkb: are the logs coming out at http://nodepool.openstack.org or you going to have to switch that? | 21:43 |
| clarkb | dns is not updated, the IP address is in scrllback or I can dig it up | 21:43 |
| *** thorst has quit IRC | 21:44 | |
| clarkb | 23.253.73.160 | 21:44 |
| *** gildub has joined #openstack-infra | 21:44 | |
| *** thorst has joined #openstack-infra | 21:44 | |
| ianw | clarkb: thanks | 21:45 |
| *** hashar has quit IRC | 21:46 | |
| *** dizquierdo has joined #openstack-infra | 21:46 | |
| ianw | jhesketh: ok, i'll watch the build of the new images. so far i have no explaination. it makes no sense because f23 is a minimal image, and devstack-centos7 is image-based, so there's nothign even in common there | 21:46 |
| *** dizquierdo has quit IRC | 21:46 | |
| *** nwkarsten has joined #openstack-infra | 21:46 | |
| *** Sukhdev has joined #openstack-infra | 21:49 | |
| pabelanger | clarkb: have to run some errands right now with family, but will be back online in 90mins or so | 21:51 |
| pabelanger | jenkins04 is still updating | 21:51 |
| *** tiswanso has joined #openstack-infra | 21:51 | |
| *** ddieterly is now known as ddieterly[away] | 21:51 | |
| *** tiswanso has quit IRC | 21:52 | |
| clarkb | pabelanger: ok | 21:52 |
| *** tiswanso has joined #openstack-infra | 21:52 | |
| *** thorst has quit IRC | 21:53 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Move gate-grenade-dsvm-neutron-dvr-multinode into check queue https://review.openstack.org/299844 | 21:53 |
| *** knikolla has quit IRC | 21:53 | |
| *** Sukhdev has quit IRC | 21:53 | |
| *** sdake has quit IRC | 21:54 | |
| *** jamesmcarthur has quit IRC | 21:54 | |
| clarkb | centos-7 is done done and fedora-23 is started | 21:57 |
| clarkb | I have also started uploads to the quicker uploading clouds (those that use qcow2 basically) | 21:57 |
| *** lakshmiS has quit IRC | 21:58 | |
| *** Sukhdev has joined #openstack-infra | 21:58 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:58 | |
| *** EricGonczer_ has quit IRC | 21:59 | |
| *** jamesmcarthur has joined #openstack-infra | 22:01 | |
| ianw | hmm, that disk usage report is not working out like i had hoped | 22:01 |
| *** jamesmcarthur has quit IRC | 22:02 | |
| ianw | oh jeez, set -x is i think messing with things | 22:02 |
| *** Sukhdev has quit IRC | 22:03 | |
| clarkb | ianw: I think greghaynes has a fix up | 22:03 |
| *** ajmiller has quit IRC | 22:03 | |
| jhesketh | Shrews: just read the scrollback. Thanks for the feedback on the shade stuff. That was my fault. I understand the risks of dicts in args but given the nature of the method it didn't appear to be an issue. That said, I agree with the extra hardening and in general not using dicts in args so that's my bad. | 22:03 |
| *** ajmiller has joined #openstack-infra | 22:03 | |
| jhesketh | Shrews: I've reviewed https://review.openstack.org/#/c/302280/1 but not sure what sort of velocity you want for merging shade changes given your comment that the last one merged too quickly | 22:04 |
| *** mriedem has quit IRC | 22:05 | |
| *** bkeller` has quit IRC | 22:05 | |
| *** bkeller` has joined #openstack-infra | 22:06 | |
| *** bkeller` has quit IRC | 22:06 | |
| *** weshay has quit IRC | 22:08 | |
| *** sridhar_ram has quit IRC | 22:09 | |
| *** baoli has joined #openstack-infra | 22:10 | |
| *** matt-borland has quit IRC | 22:10 | |
| *** fitoduarte has quit IRC | 22:10 | |
| *** sridhar_ram1 has joined #openstack-infra | 22:11 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 22:12 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion with net-iso in periodic job https://review.openstack.org/296765 | 22:12 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Turn of tracing around du invocations https://review.openstack.org/302454 | 22:12 |
| ianw | greghaynes / clarkb: ^ that should help... | 22:13 |
| *** ddieterly[away] is now known as ddieterly | 22:13 | |
| greghaynes | ianw: https://review.openstack.org/#/c/300908/ | 22:13 |
| *** claudiub has joined #openstack-infra | 22:13 | |
| *** annegentle has quit IRC | 22:14 | |
| ianw | greghaynes: ahh, missed that sorry. i think it's probably better to keep -x in general, but just avoid silly things like this du stuff | 22:15 |
| greghaynes | ianw: dib doesnt set -x, so we shouldnt be running it as -x in tests - that gives us more debug info than users get | 22:15 |
| greghaynes | its also just different behavior than what a normal user would have | 22:16 |
| *** claudiub|2 has quit IRC | 22:16 | |
| *** tpsilva has quit IRC | 22:17 | |
| ianw | greghaynes: but this doesn't help the nodepool logs, which is running dib with "-x" | 22:17 |
| clarkb | the fedora-23 build hasn't written any logs for ~7 minutes | 22:17 |
| clarkb | hopeflly just buffering | 22:17 |
| greghaynes | ianw: it should if we dont run dib with -x | 22:17 |
| ianw | greghaynes: we have to run dib with "-x" on nodepool, it's the only way to have info to debug failures | 22:18 |
| greghaynes | er, theres two -x's | 22:18 |
| aeng | ping AJaeger | 22:18 |
| greghaynes | we should run dib with set -x set in the shell for disk-image-create | 22:18 |
| clarkb | and it started logging again so ya probably just buffering | 22:18 |
| greghaynes | we should run dib with -x | 22:18 |
| *** dimtruck is now known as zz_dimtruck | 22:19 | |
| *** piet has joined #openstack-infra | 22:19 | |
| greghaynes | oh, I see what happened | 22:19 |
| greghaynes | thers a change where the -x argument turns on set -x in disk-image-create | 22:19 |
| * greghaynes didnt realize that was going on | 22:19 | |
| *** Goneri has joined #openstack-infra | 22:20 | |
| *** baoli has quit IRC | 22:20 | |
| greghaynes | ianw: yea, ok. thats fine then | 22:20 |
| *** baoli has joined #openstack-infra | 22:20 | |
| greghaynes | ianw: I didnt realize we had a user facing way of making disk-image-create run with set -x | 22:20 |
| ianw | greghaynes: yeah. that's why i added all that header stuff a while ago :) It's multiple "-x" 's, which is why i proposed that other change to only turn on tracing for the git caching when running under "-x -x" | 22:22 |
| greghaynes | makes sense | 22:22 |
| greghaynes | +A'd | 22:22 |
| morgan | greghaynes: i see a greghaynes. | 22:23 |
| morgan | greghaynes: o/ | 22:23 |
| morgan | and \o | 22:23 |
| morgan | ;) | 22:23 |
| * morgan goes back to lurking. | 22:23 | |
| *** Sukhdev has joined #openstack-infra | 22:24 | |
| ianw | greghaynes: this might be point-release worthy too, cause the logs are about 10x what they usually are | 22:25 |
| *** dizquierdo has joined #openstack-infra | 22:25 | |
| ianw | jhesketh: so checking http://23.253.73.160/dib.fedora-23.log this is picking up requestexceptions. which is sort of good, but sort of bad because i have no idea why the old build wasn't | 22:26 |
| *** tonytan4ever has quit IRC | 22:28 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/puppet-mediawiki: Updates to move from precise to trusty https://review.openstack.org/302456 | 22:28 |
| *** kzaitsev_mb has quit IRC | 22:28 | |
| *** ddieterly is now known as ddieterly[away] | 22:28 | |
| jhesketh | ianw: heh, progress! | 22:29 |
| *** ddieterly[away] is now known as ddieterly | 22:29 | |
| *** ddieterly is now known as ddieterly[away] | 22:30 | |
| *** ddieterly[away] is now known as ddieterly | 22:30 | |
| *** achanda has quit IRC | 22:32 | |
| *** piet has quit IRC | 22:32 | |
| *** piet has joined #openstack-infra | 22:33 | |
| *** eharney has quit IRC | 22:35 | |
| *** pradk has quit IRC | 22:36 | |
| *** achanda has joined #openstack-infra | 22:38 | |
| anteaya | zaro: did you make any progress with devstack-centos7 not returning any logs on failure? | 22:39 |
| *** ddieterly is now known as ddieterly[away] | 22:39 | |
| ianw | anteaya / zaro : it's due to something funny going on with installing requestexceptions during image build | 22:40 |
| ianw | the new nodepool is likely to fix this, but no root cause as yet | 22:40 |
| *** BobBall-1WOL has joined #openstack-infra | 22:41 | |
| zaro | ianw: thanks for the update. | 22:41 |
| *** thiagop has quit IRC | 22:42 | |
| *** ddieterly[away] is now known as ddieterly | 22:42 | |
| *** jcooley_ has quit IRC | 22:43 | |
| *** BobBall-AWOL has quit IRC | 22:44 | |
| *** piet has quit IRC | 22:45 | |
| *** piet has joined #openstack-infra | 22:45 | |
| clarkb | debian-jessie is building now. reaonsbly good progress | 22:47 |
| anteaya | clarkb: yay | 22:47 |
| anteaya | ianw: is this new? | 22:47 |
| anteaya | I don't recall this being an issue previously, but also I have never explictly looked either | 22:48 |
| *** nwkarsten has quit IRC | 22:48 | |
| *** nwkarsten has joined #openstack-infra | 22:49 | |
| *** EricGonczer_ has joined #openstack-infra | 22:49 | |
| *** cloudtrainme has quit IRC | 22:50 | |
| *** thorst has joined #openstack-infra | 22:50 | |
| *** |-paul-| has quit IRC | 22:50 | |
| pabelanger | clarkb: ack | 22:52 |
| *** nwkarsten has quit IRC | 22:53 | |
| *** ddieterly is now known as ddieterly[away] | 22:53 | |
| *** EricGonczer_ has quit IRC | 22:55 | |
| *** angdraug has quit IRC | 22:56 | |
| *** netsin has joined #openstack-infra | 22:56 | |
| *** thorst has quit IRC | 22:57 | |
| *** dizquierdo has quit IRC | 22:57 | |
| *** zz_dimtruck is now known as dimtruck | 22:58 | |
| *** piet has quit IRC | 22:59 | |
| *** gordc has quit IRC | 22:59 | |
| *** ybathia has joined #openstack-infra | 23:03 | |
| flwang | clarkb: ping re adding tempest gate job | 23:03 |
| clarkb | flwang: hi | 23:03 |
| flwang | clarkb: hi we(zaqar) are going to add a tempest gate job, but i don't have to test the job before submitting the patch to project-config | 23:04 |
| flwang | clarkb: is it possible to test it locally? | 23:04 |
| clarkb | it is, take the shell from your job and run it locally | 23:04 |
| *** rbrndt_ has quit IRC | 23:05 | |
| *** annegentle has joined #openstack-infra | 23:05 | |
| flwang | clarkb: ok, i can see the point now. thanks | 23:05 |
| clarkb | flwang: also all of those jobs write a reproduce.sh that you could tweak to match your new job | 23:06 |
| flwang | clarkb: thanks for the info :) | 23:07 |
| *** jcooley_ has joined #openstack-infra | 23:08 | |
| *** ayoung has joined #openstack-infra | 23:09 | |
| *** annegentle has quit IRC | 23:09 | |
| *** ybathia has quit IRC | 23:10 | |
| *** TravT has quit IRC | 23:10 | |
| *** ajmiller has quit IRC | 23:11 | |
| *** ybathia has joined #openstack-infra | 23:11 | |
| *** ybathia has quit IRC | 23:11 | |
| clarkb | I still can't get over the raw images being smaller than the vhds | 23:12 |
| clarkb | (on disk) | 23:12 |
| *** ajmiller has joined #openstack-infra | 23:12 | |
| ianw | anteaya: it was new to me as of a few hours ago, but seems to have been going on for some time | 23:13 |
| *** ybathia has joined #openstack-infra | 23:14 | |
| *** Sukhdev has quit IRC | 23:14 | |
| *** asettle has quit IRC | 23:14 | |
| *** ybathia has quit IRC | 23:15 | |
| *** achanda has quit IRC | 23:15 | |
| pabelanger | ianw: sorry, which job did you want me to kick off again? | 23:16 |
| ianw | propose-devstack-plugins-list from the devstack periodic jobs | 23:16 |
| *** Qiming has joined #openstack-infra | 23:17 | |
| *** asettle has joined #openstack-infra | 23:17 | |
| *** ZZelle_ has joined #openstack-infra | 23:18 | |
| openstackgerrit | Merged openstack/diskimage-builder: Turn of tracing around du invocations https://review.openstack.org/302454 | 23:18 |
| *** hichihara has joined #openstack-infra | 23:20 | |
| *** achanda has joined #openstack-infra | 23:22 | |
| *** claudiub has quit IRC | 23:25 | |
| *** ddieterly has joined #openstack-infra | 23:26 | |
| *** ddieterly is now known as ddieterly[away] | 23:26 | |
| pleia2 | fyi, been working with aeng on some zanata debugging in -i18n, just restarted the service | 23:26 |
| *** _amrith_ is now known as amrith | 23:26 | |
| clarkb | devstack-centos7 is building now | 23:27 |
| *** annegentle has joined #openstack-infra | 23:27 | |
| pabelanger | ianw: okay, I can't manually run it from jenkins will need to enqueue it in zuul. | 23:27 |
| clarkb | I mostly am uploading as I go to everywhere but rax beucase those are much slower | 23:27 |
| clarkb | figure I can queue the rax regions up at the end and go get dinner | 23:27 |
| *** jcooley_ has quit IRC | 23:31 | |
| *** shashank_hegde has quit IRC | 23:32 | |
| *** dimtruck is now known as zz_dimtruck | 23:34 | |
| *** jamielennox|away is now known as jamielennox | 23:34 | |
| *** akshai has quit IRC | 23:34 | |
| *** Swami has quit IRC | 23:34 | |
| *** imcsk8|afk is now known as imcsk8 | 23:34 | |
| *** eil397 has left #openstack-infra | 23:35 | |
| *** mriedem has joined #openstack-infra | 23:36 | |
| *** salv-orl_ has quit IRC | 23:36 | |
| *** annegentle has quit IRC | 23:39 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Fix soabi tests with pypy https://review.openstack.org/302045 | 23:40 |
| *** tosky has quit IRC | 23:40 | |
| greghaynes | ianw: Yea, you can go for a release if yo uwant | 23:40 |
| *** baoli has quit IRC | 23:44 | |
| *** sdake has joined #openstack-infra | 23:44 | |
| *** sdake_ has joined #openstack-infra | 23:46 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Make reproduce function quiet https://review.openstack.org/296171 | 23:46 |
| *** sdake has quit IRC | 23:49 | |
| *** ZZelle_ has quit IRC | 23:49 | |
| *** Sukhdev has joined #openstack-infra | 23:50 | |
| *** roxanaghe has quit IRC | 23:50 | |
| *** sdake has joined #openstack-infra | 23:50 | |
| clarkb | last image to build \o/ ubuntu-xenial | 23:51 |
| *** Sukhdev has quit IRC | 23:52 | |
| *** sdake_ has quit IRC | 23:52 | |
| *** ams_ has quit IRC | 23:53 | |
| *** annegentle has joined #openstack-infra | 23:54 | |
| pabelanger | cool | 23:54 |
| pabelanger | and jenkins04 finally finished | 23:54 |
| *** thorst has joined #openstack-infra | 23:55 | |
| *** ajmiller has quit IRC | 23:57 | |
| *** annegentle has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
