| *** salv-orlando has joined #openstack-infra | 00:00 | |
| *** slaweq has joined #openstack-infra | 00:01 | |
| *** jamesmcarthur has quit IRC | 00:01 | |
| *** florianf has joined #openstack-infra | 00:01 | |
| *** rkukura has quit IRC | 00:04 | |
| *** mriedem has quit IRC | 00:05 | |
| *** slaweq has quit IRC | 00:06 | |
| *** salv-orlando has quit IRC | 00:06 | |
| *** s-shiono has joined #openstack-infra | 00:08 | |
| *** florianf has quit IRC | 00:08 | |
| *** rlandy has quit IRC | 00:08 | |
| *** s-shiono_ has joined #openstack-infra | 00:09 | |
| *** bobh has joined #openstack-infra | 00:11 | |
| *** s-shiono has quit IRC | 00:12 | |
| *** yamamoto has joined #openstack-infra | 00:12 | |
| *** sree has joined #openstack-infra | 00:15 | |
| *** yamamoto has quit IRC | 00:18 | |
| *** sree has quit IRC | 00:19 | |
| *** masuberu has joined #openstack-infra | 00:20 | |
| *** masber has quit IRC | 00:23 | |
| *** dhill_ has quit IRC | 00:29 | |
| *** sree has joined #openstack-infra | 00:33 | |
| *** eernst has joined #openstack-infra | 00:34 | |
| *** dhill_ has joined #openstack-infra | 00:36 | |
| *** sree has quit IRC | 00:38 | |
| *** jamesmcarthur has joined #openstack-infra | 00:43 | |
| *** masber has joined #openstack-infra | 00:46 | |
| *** masuberu has quit IRC | 00:49 | |
| *** hongbin_ has joined #openstack-infra | 00:49 | |
| *** eernst has quit IRC | 00:57 | |
| *** eernst has joined #openstack-infra | 00:57 | |
| *** VW has joined #openstack-infra | 00:58 | |
| *** psachin has joined #openstack-infra | 01:01 | |
| *** salv-orlando has joined #openstack-infra | 01:02 | |
| *** wolverineav has joined #openstack-infra | 01:02 | |
| *** sree has joined #openstack-infra | 01:04 | |
| *** bobh has quit IRC | 01:05 | |
| *** salv-orlando has quit IRC | 01:06 | |
| *** sree has quit IRC | 01:09 | |
| *** wolverineav has quit IRC | 01:10 | |
| *** zhangfei has joined #openstack-infra | 01:11 | |
| *** VW has quit IRC | 01:14 | |
| *** yamamoto has joined #openstack-infra | 01:14 | |
| *** VW has joined #openstack-infra | 01:15 | |
| *** dpawlik has joined #openstack-infra | 01:15 | |
| *** agopi|zzz has joined #openstack-infra | 01:19 | |
| *** VW has quit IRC | 01:19 | |
| *** yamamoto has quit IRC | 01:20 | |
| *** dpawlik has quit IRC | 01:20 | |
| *** bobh has joined #openstack-infra | 01:21 | |
| *** harlowja has quit IRC | 01:23 | |
| *** masuberu has joined #openstack-infra | 01:27 | |
| *** masber has quit IRC | 01:31 | |
| *** bobh has quit IRC | 01:34 | |
| *** jamesmcarthur has quit IRC | 01:36 | |
| *** sree has joined #openstack-infra | 01:41 | |
| *** felipemonteiro__ has joined #openstack-infra | 01:43 | |
| *** sree has quit IRC | 01:46 | |
| *** rosmaita has quit IRC | 01:46 | |
| *** caphrim007 has joined #openstack-infra | 01:47 | |
| *** felipemonteiro__ has quit IRC | 01:49 | |
| *** gmann_ is now known as gmann | 01:49 | |
| *** caphrim007 has quit IRC | 01:51 | |
| *** jcoufal has quit IRC | 01:51 | |
| *** masber has joined #openstack-infra | 01:54 | |
| *** bobh has joined #openstack-infra | 01:55 | |
| *** masuberu has quit IRC | 01:56 | |
| *** hamzy has joined #openstack-infra | 02:00 | |
| *** slaweq has joined #openstack-infra | 02:02 | |
| *** salv-orlando has joined #openstack-infra | 02:02 | |
| *** eernst has quit IRC | 02:06 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-dev/hacking master: Release notes for 1.1.0 https://review.openstack.org/561129 | 02:06 |
|---|---|---|
| *** slaweq has quit IRC | 02:06 | |
| *** salv-orlando has quit IRC | 02:07 | |
| *** yamamoto has joined #openstack-infra | 02:16 | |
| *** yamamoto has quit IRC | 02:18 | |
| *** aeng has joined #openstack-infra | 02:18 | |
| *** yamamoto has joined #openstack-infra | 02:18 | |
| *** dhajare has joined #openstack-infra | 02:19 | |
| anteaya | three first nations art shows in bc whilst summit is happening for those interested: https://www.theglobeandmail.com/arts/art-and-architecture/article-three-history-making-exhibitions-of-indigenous-art-and-culture-open-in/ | 02:21 |
| *** dhajare has quit IRC | 02:25 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-dev/hacking master: Release notes for 1.1.0 https://review.openstack.org/561129 | 02:26 |
| *** hamzy has quit IRC | 02:29 | |
| *** jamesmcarthur has joined #openstack-infra | 02:32 | |
| *** dhajare has joined #openstack-infra | 02:32 | |
| *** eernst has joined #openstack-infra | 02:36 | |
| *** hamzy has joined #openstack-infra | 02:36 | |
| *** aeng_ has joined #openstack-infra | 02:40 | |
| *** aeng has quit IRC | 02:41 | |
| *** gongysh has joined #openstack-infra | 02:48 | |
| *** eernst has quit IRC | 02:51 | |
| *** bobh has quit IRC | 03:02 | |
| *** salv-orlando has joined #openstack-infra | 03:03 | |
| *** salv-orlando has quit IRC | 03:08 | |
| *** dhill_ has quit IRC | 03:13 | |
| *** dpawlik has joined #openstack-infra | 03:16 | |
| *** dpawlik has quit IRC | 03:20 | |
| *** VW has joined #openstack-infra | 03:21 | |
| *** rajinir has quit IRC | 03:21 | |
| *** VW has quit IRC | 03:29 | |
| *** VW has joined #openstack-infra | 03:29 | |
| *** nicolasbock has quit IRC | 03:32 | |
| *** dhill_ has joined #openstack-infra | 03:33 | |
| *** VW has quit IRC | 03:33 | |
| *** ykarel|away has joined #openstack-infra | 03:38 | |
| *** ramishra has joined #openstack-infra | 03:42 | |
| *** jamesmcarthur has quit IRC | 03:42 | |
| *** lpetrut has joined #openstack-infra | 03:45 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Sometimes GitHub doesn't return repo permissions https://review.openstack.org/564666 | 03:54 |
| *** sree has joined #openstack-infra | 03:57 | |
| *** ykarel|away has quit IRC | 03:57 | |
| *** shu-mutow has joined #openstack-infra | 03:57 | |
| *** sree has quit IRC | 04:01 | |
| *** slaweq has joined #openstack-infra | 04:03 | |
| *** eernst has joined #openstack-infra | 04:03 | |
| *** salv-orlando has joined #openstack-infra | 04:04 | |
| *** eernst has quit IRC | 04:06 | |
| *** jamesmcarthur has joined #openstack-infra | 04:06 | |
| *** dhill_ has quit IRC | 04:06 | |
| *** slaweq has quit IRC | 04:08 | |
| *** sree has joined #openstack-infra | 04:08 | |
| *** salv-orlando has quit IRC | 04:09 | |
| *** jamesmcarthur has quit IRC | 04:11 | |
| *** vivsoni has joined #openstack-infra | 04:11 | |
| *** hongbin_ has quit IRC | 04:13 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 04:13 |
| *** dhill_ has joined #openstack-infra | 04:20 | |
| openstackgerrit | Sangho Shin proposed openstack-infra/project-config master: Allows networking-onos-core group to create a new branch https://review.openstack.org/564669 | 04:24 |
| *** lpetrut has quit IRC | 04:24 | |
| *** mpjetta has quit IRC | 04:29 | |
| *** gongysh has quit IRC | 04:33 | |
| *** auristor has quit IRC | 04:33 | |
| *** auristor has joined #openstack-infra | 04:35 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 04:40 |
| *** janki has joined #openstack-infra | 04:43 | |
| *** janki has quit IRC | 04:44 | |
| *** janki has joined #openstack-infra | 04:45 | |
| *** dayou has quit IRC | 04:45 | |
| *** janki has quit IRC | 04:45 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul-jobs master: Make revoke-sudo work on base cloud-init images https://review.openstack.org/564674 | 04:45 |
| *** janki has joined #openstack-infra | 04:46 | |
| *** ykarel|away has joined #openstack-infra | 04:47 | |
| *** ykarel|away is now known as ykarel | 04:48 | |
| *** Hunner has quit IRC | 04:51 | |
| *** dhill_ has quit IRC | 04:51 | |
| *** bmjen has quit IRC | 04:51 | |
| *** bmjen has joined #openstack-infra | 04:57 | |
| *** Hunner has joined #openstack-infra | 04:58 | |
| *** Hunner has quit IRC | 04:58 | |
| *** Hunner has joined #openstack-infra | 04:58 | |
| *** links has joined #openstack-infra | 05:04 | |
| *** VW has joined #openstack-infra | 05:04 | |
| *** armaan has joined #openstack-infra | 05:04 | |
| *** salv-orlando has joined #openstack-infra | 05:05 | |
| *** eernst has joined #openstack-infra | 05:06 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 05:06 |
| *** dayou has joined #openstack-infra | 05:06 | |
| *** dhill_ has joined #openstack-infra | 05:07 | |
| *** salv-orlando has quit IRC | 05:07 | |
| *** salv-orlando has joined #openstack-infra | 05:07 | |
| *** links has quit IRC | 05:08 | |
| *** mpjetta has joined #openstack-infra | 05:10 | |
| *** jchhatbar has joined #openstack-infra | 05:11 | |
| *** janki has quit IRC | 05:13 | |
| *** links has joined #openstack-infra | 05:13 | |
| *** links has quit IRC | 05:13 | |
| *** lpetrut has joined #openstack-infra | 05:14 | |
| *** dpawlik has joined #openstack-infra | 05:17 | |
| *** eernst has quit IRC | 05:19 | |
| *** masuberu has joined #openstack-infra | 05:20 | |
| *** VW has quit IRC | 05:20 | |
| *** VW has joined #openstack-infra | 05:21 | |
| *** dpawlik has quit IRC | 05:22 | |
| *** masber has quit IRC | 05:24 | |
| *** mikal_ has joined #openstack-infra | 05:24 | |
| *** auristor has quit IRC | 05:25 | |
| *** pgadiya has joined #openstack-infra | 05:26 | |
| *** pgadiya has quit IRC | 05:26 | |
| *** VW has quit IRC | 05:26 | |
| *** mikal has quit IRC | 05:27 | |
| *** lpetrut has quit IRC | 05:31 | |
| *** quiquell|off is now known as quiquell|rover | 05:31 | |
| *** onovy has quit IRC | 05:32 | |
| *** auristor has joined #openstack-infra | 05:33 | |
| *** e0ne has joined #openstack-infra | 05:34 | |
| *** dhill_ has quit IRC | 05:37 | |
| *** aeng_ has quit IRC | 05:41 | |
| *** eumel8 has joined #openstack-infra | 05:42 | |
| *** armaan has quit IRC | 05:42 | |
| *** dhill_ has joined #openstack-infra | 05:43 | |
| *** agopi|zzz has quit IRC | 05:44 | |
| *** udesale_ has joined #openstack-infra | 05:44 | |
| *** udesale_ has quit IRC | 05:44 | |
| *** germs has quit IRC | 05:46 | |
| *** germs has joined #openstack-infra | 05:47 | |
| *** germs has quit IRC | 05:47 | |
| *** germs has joined #openstack-infra | 05:47 | |
| *** claudiub has joined #openstack-infra | 05:47 | |
| *** masber has joined #openstack-infra | 05:47 | |
| *** masuberu has quit IRC | 05:50 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add OpenAPI documentation https://review.openstack.org/535541 | 05:52 |
| *** scas has quit IRC | 05:55 | |
| *** scas has joined #openstack-infra | 05:55 | |
| *** Douhet has quit IRC | 05:57 | |
| *** armaan has joined #openstack-infra | 05:58 | |
| *** armaan has quit IRC | 05:58 | |
| *** Douhet has joined #openstack-infra | 05:58 | |
| *** armaan has joined #openstack-infra | 05:59 | |
| *** armaan has quit IRC | 06:04 | |
| *** armaan has joined #openstack-infra | 06:04 | |
| *** iranzo has joined #openstack-infra | 06:09 | |
| *** lpetrut has joined #openstack-infra | 06:14 | |
| openstackgerrit | Sangho Shin proposed openstack-infra/project-config master: Allows networking-onos-core group to create a new branch https://review.openstack.org/564682 | 06:14 |
| *** onovy has joined #openstack-infra | 06:14 | |
| *** andreas_s has joined #openstack-infra | 06:20 | |
| *** armaan has quit IRC | 06:21 | |
| *** armaan has joined #openstack-infra | 06:21 | |
| *** gongysh has joined #openstack-infra | 06:24 | |
| *** Douhet has quit IRC | 06:29 | |
| *** Douhet has joined #openstack-infra | 06:30 | |
| *** pcaruana has joined #openstack-infra | 06:33 | |
| *** nmathew has joined #openstack-infra | 06:42 | |
| *** nmathew has quit IRC | 06:43 | |
| *** quiquell|rover is now known as quique|rover|afk | 06:49 | |
| *** pcaruana has quit IRC | 06:50 | |
| *** alex_xu has quit IRC | 06:51 | |
| *** alex_xu has joined #openstack-infra | 06:52 | |
| *** slaweq has joined #openstack-infra | 06:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Change label for neutron-tempest-ovsfw to "voting" https://review.openstack.org/561930 | 06:58 |
| *** yolanda__ is now known as yolanda | 06:59 | |
| *** pcaruana has joined #openstack-infra | 07:06 | |
| *** diablo_rojo has joined #openstack-infra | 07:08 | |
| *** hemna_ has quit IRC | 07:09 | |
| *** alexchadin has joined #openstack-infra | 07:11 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: builder: support setting diskimage env-vars in secure configuration https://review.openstack.org/564687 | 07:13 |
| *** xinliang has quit IRC | 07:14 | |
| *** xinliang has joined #openstack-infra | 07:15 | |
| *** jtomasek has joined #openstack-infra | 07:15 | |
| *** ccamacho has quit IRC | 07:17 | |
| *** rcernin has quit IRC | 07:17 | |
| *** dpawlik has joined #openstack-infra | 07:18 | |
| *** ccamacho has joined #openstack-infra | 07:18 | |
| *** dpawlik has quit IRC | 07:21 | |
| *** jaosorior has joined #openstack-infra | 07:21 | |
| openstackgerrit | Sangho Shin proposed openstack-infra/project-config master: Allows networking-onos-core group to create a new branch https://review.openstack.org/564682 | 07:22 |
| *** dpawlik has joined #openstack-infra | 07:22 | |
| *** tesseract has joined #openstack-infra | 07:27 | |
| *** pcichy has joined #openstack-infra | 07:27 | |
| *** armaan has quit IRC | 07:35 | |
| *** armaan has joined #openstack-infra | 07:35 | |
| *** edmondsw has joined #openstack-infra | 07:36 | |
| *** kjackal has quit IRC | 07:40 | |
| *** pcichy has quit IRC | 07:40 | |
| *** quique|rover|afk is now known as quiquell|rover | 07:40 | |
| *** edmondsw has quit IRC | 07:41 | |
| *** ssbarnea_ has joined #openstack-infra | 07:45 | |
| amotoki | AJaeger: could you confirm this is the right approach for stable/queens tox_install.sh failure? https://review.openstack.org/#/c/564534/ | 07:46 |
| *** ykarel is now known as ykarel|away | 07:47 | |
| *** hashar has joined #openstack-infra | 07:47 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Fix nova hacking integration job https://review.openstack.org/564691 | 07:49 |
| *** kjackal has joined #openstack-infra | 07:50 | |
| *** jpena|off is now known as jpena | 07:52 | |
| *** aojea has joined #openstack-infra | 07:52 | |
| *** armaan has quit IRC | 07:52 | |
| *** amoralej|off is now known as amoralej | 07:52 | |
| *** armaan has joined #openstack-infra | 07:53 | |
| *** tesseract has quit IRC | 07:53 | |
| AJaeger | amotoki: that's one workaround for pip10 - yes | 07:54 |
| amotoki | AJaeger: thanks for checking it. This is the gate fix so I am merging it fast. | 07:55 |
| *** tesseract has joined #openstack-infra | 07:56 | |
| *** lucas-afk is now known as lucasagomes | 07:58 | |
| *** jamesmcarthur has joined #openstack-infra | 08:07 | |
| *** tosky has joined #openstack-infra | 08:10 | |
| *** jamesmcarthur has quit IRC | 08:12 | |
| *** alexchadin has quit IRC | 08:14 | |
| *** jogo has quit IRC | 08:14 | |
| *** alexchadin has joined #openstack-infra | 08:15 | |
| *** s-shiono_ has quit IRC | 08:19 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 08:28 |
| *** cuongnv has joined #openstack-infra | 08:35 | |
| *** cuongnv is now known as Guest89211 | 08:36 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 08:37 |
| *** Guest89211 has quit IRC | 08:38 | |
| *** cuongnv_ has joined #openstack-infra | 08:38 | |
| *** mandre is now known as mandre_afk | 08:39 | |
| *** derekh has joined #openstack-infra | 08:40 | |
| *** shardy has joined #openstack-infra | 08:41 | |
| *** alexchadin has quit IRC | 08:42 | |
| *** shardy has quit IRC | 08:42 | |
| *** shardy has joined #openstack-infra | 08:42 | |
| *** zhangfei has quit IRC | 08:53 | |
| *** dtantsur|afk is now known as dtantsur | 08:55 | |
| *** pcaruana has quit IRC | 08:56 | |
| *** florianf has joined #openstack-infra | 08:57 | |
| *** zhangfei has joined #openstack-infra | 09:05 | |
| *** shardy has quit IRC | 09:07 | |
| *** pcaruana has joined #openstack-infra | 09:07 | |
| *** viks has quit IRC | 09:11 | |
| *** jogo has joined #openstack-infra | 09:11 | |
| *** cuongnv_ has quit IRC | 09:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack-dev/hacking master: Import integration test https://review.openstack.org/547765 | 09:19 |
| *** florianf has quit IRC | 09:21 | |
| *** stakeda has quit IRC | 09:21 | |
| *** panda|ruck|off is now known as panda|ruck | 09:21 | |
| openstackgerrit | Shu Muto proposed openstack-infra/project-config master: Add horizon into required-projects for nodejs jobs for magnum-ui https://review.openstack.org/564711 | 09:23 |
| *** zoli is now known as zoli|lunch | 09:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: ignore the series_status.yaml when processing release requests https://review.openstack.org/564342 | 09:41 |
| *** gfidente has joined #openstack-infra | 09:42 | |
| *** gfidente has joined #openstack-infra | 09:42 | |
| *** annp has quit IRC | 09:44 | |
| *** xinliang3 has quit IRC | 09:47 | |
| *** xinliang3 has joined #openstack-infra | 09:48 | |
| *** ansiwen[q] has quit IRC | 09:48 | |
| *** diablo_rojo has quit IRC | 09:51 | |
| *** pcichy has joined #openstack-infra | 09:59 | |
| *** shu-mutow has quit IRC | 10:03 | |
| *** alexchadin has joined #openstack-infra | 10:04 | |
| *** shardy has joined #openstack-infra | 10:06 | |
| *** shardy has quit IRC | 10:06 | |
| *** e0ne has quit IRC | 10:07 | |
| *** shardy has joined #openstack-infra | 10:08 | |
| *** rpittau has quit IRC | 10:08 | |
| *** dtantsur is now known as dtantsur|brb | 10:13 | |
| *** alexchadin has quit IRC | 10:14 | |
| *** l4yerffej has quit IRC | 10:15 | |
| *** l4yerffej has joined #openstack-infra | 10:17 | |
| *** alexchadin has joined #openstack-infra | 10:19 | |
| *** apetrich has quit IRC | 10:19 | |
| *** kjackal has quit IRC | 10:19 | |
| *** sree has quit IRC | 10:19 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 10:23 |
| *** rpittau has joined #openstack-infra | 10:23 | |
| *** CrayZee has joined #openstack-infra | 10:27 | |
| *** kjackal has joined #openstack-infra | 10:29 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 10:30 |
| *** nicolasbock has joined #openstack-infra | 10:30 | |
| *** salv-orlando has quit IRC | 10:33 | |
| *** salv-orlando has joined #openstack-infra | 10:34 | |
| *** boden has joined #openstack-infra | 10:35 | |
| *** alexchadin has quit IRC | 10:36 | |
| *** salv-orlando has quit IRC | 10:38 | |
| *** salv-orlando has joined #openstack-infra | 10:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove fedora-26 from AFS mirror https://review.openstack.org/564725 | 10:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add fedora-28 to AFS mirrors https://review.openstack.org/564726 | 10:45 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 10:45 |
| *** yamahata has quit IRC | 10:45 | |
| mordred | infra-root: heads up - we cut a new openstacksdk release (which shade depends on now) - should be zero impact, but just fyi | 10:46 |
| pabelanger | ack | 10:48 |
| *** mandre_afk is now known as mandre | 10:48 | |
| *** caphrim007 has joined #openstack-infra | 10:49 | |
| pabelanger | infra-root: I'd like to start preparing for fedora-28, if I could get a review on adding it to our AFS mirrors: https://review.openstack.org/#/q/topic:fedora-28 | 10:51 |
| *** zoli|lunch is now known as zoli | 10:52 | |
| *** pcichy has quit IRC | 10:53 | |
| *** caphrim007 has quit IRC | 10:53 | |
| *** dhajare has quit IRC | 10:56 | |
| *** alexchadin has joined #openstack-infra | 11:00 | |
| *** dhajare has joined #openstack-infra | 11:01 | |
| mordred | pabelanger: it's almost likel there is always a new release of something ... | 11:04 |
| dtantsur|brb | folks, how to debug such failures? http://logs.openstack.org/60/564560/3/check/ironic-tempest-dsvm-functional-python3/59ae183/job-output.txt.gz#_2018-04-27_08_52_19_919388 | 11:04 |
| mordred | dtantsur|brb: those are unfortunate - usually either job-output.json or the ara report will have info | 11:06 |
| dtantsur|brb | ok, lemme try | 11:06 |
| *** dtantsur|brb is now known as dtantsur | 11:06 | |
| mordred | dtantsur: http://logs.openstack.org/60/564560/3/check/ironic-tempest-dsvm-functional-python3/59ae183/ara-report/result/4195a9cb-0f8d-41ba-ba66-68d8458cf730/ | 11:07 |
| dtantsur | aha, I suspected something like that | 11:07 |
| dtantsur | thanks mordred | 11:08 |
| dtantsur | now I really wonder what in https://review.openstack.org/#/c/564560/2..3/playbooks/legacy/tempest-dsvm-ironic-functional-python3/run.yaml would cause it.... | 11:09 |
| pabelanger | mordred: indeed | 11:14 |
| frickler | dtantsur: I'm guessing zuul chokes on the "'" in the comment | 11:15 |
| dtantsur | frickler: this is my wild guess too, removing it | 11:15 |
| mordred | ++ | 11:15 |
| dtantsur | but this is crazy :) | 11:15 |
| mordred | dtantsur: it's actually an ansible jinja parsing thing | 11:16 |
| dtantsur | TIL! | 11:16 |
| *** dhajare has quit IRC | 11:16 | |
| mordred | that said - we need to get y'all migrated to some non-legacy jobs so you don't have to be editing fragile files like that | 11:17 |
| dtantsur | that's fair. this patch is just a bit urgent, so I'm cargo-culting old jobs | 11:18 |
| dtantsur | maybe I could make it a non-legacy job, I will try after it passes | 11:18 |
| pabelanger | #status log jessie removed from mirror.debian (AFS mirror) and reprepro configuration on mirror-update.o.o | 11:19 |
| openstackstatus | pabelanger: finished logging | 11:19 |
| *** zhangfei has quit IRC | 11:19 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 11:19 |
| *** andreas_s has quit IRC | 11:20 | |
| *** andreas_s has joined #openstack-infra | 11:20 | |
| *** ykarel|away is now known as ykarel | 11:22 | |
| *** andreas_s has quit IRC | 11:25 | |
| pabelanger | frickler: mind a review on https://review.openstack.org/564726/ too | 11:27 |
| frickler | pabelanger: ack | 11:29 |
| *** jcoufal has joined #openstack-infra | 11:32 | |
| *** ldnunes has joined #openstack-infra | 11:34 | |
| *** lucasagomes is now known as lucas-hungry | 11:38 | |
| *** andreas_s has joined #openstack-infra | 11:39 | |
| *** quiquell|rover is now known as quique|rover|lch | 11:40 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove fedora-26 from AFS mirror https://review.openstack.org/564725 | 11:42 |
| *** caphrim007 has joined #openstack-infra | 11:47 | |
| *** caphrim007 has quit IRC | 11:52 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Fix 'releases' is deprecated warning https://review.openstack.org/564740 | 11:54 |
| *** jpena is now known as jpena|lunch | 11:56 | |
| *** ramishra has quit IRC | 11:58 | |
| *** efried is now known as fried_rice | 11:59 | |
| *** jtomasek has quit IRC | 12:01 | |
| *** lpetrut has quit IRC | 12:02 | |
| *** jtomasek has joined #openstack-infra | 12:02 | |
| *** lpetrut has joined #openstack-infra | 12:04 | |
| *** ssbarnea_ has quit IRC | 12:06 | |
| *** jamesmcarthur has joined #openstack-infra | 12:06 | |
| *** rosmaita has joined #openstack-infra | 12:07 | |
| *** ssbarnea_ has joined #openstack-infra | 12:07 | |
| *** katkapilatova has joined #openstack-infra | 12:07 | |
| *** panda|ruck is now known as panda|ruck|lunch | 12:08 | |
| *** ssbarnea_ has quit IRC | 12:09 | |
| *** tpsilva has joined #openstack-infra | 12:09 | |
| *** alex_xu has quit IRC | 12:10 | |
| *** alex_xu has joined #openstack-infra | 12:10 | |
| *** jamesmcarthur has quit IRC | 12:11 | |
| *** hwoarang has quit IRC | 12:13 | |
| *** ssbarnea_ has joined #openstack-infra | 12:13 | |
| *** jpich has joined #openstack-infra | 12:14 | |
| *** katkapilatova has left #openstack-infra | 12:14 | |
| pabelanger | #status log fedora-26 removed from mirror.fedora (AFS mirror) and rsync configuration on mirror-update.o.o | 12:15 |
| openstackstatus | pabelanger: finished logging | 12:15 |
| *** ssbarnea_ has quit IRC | 12:19 | |
| *** sree_ has joined #openstack-infra | 12:20 | |
| *** sree_ is now known as Guest7090 | 12:21 | |
| *** ssbarnea_ has joined #openstack-infra | 12:21 | |
| *** quique|rover|lch is now known as quiquell|rover | 12:22 | |
| *** trown|outtypewww is now known as trown | 12:23 | |
| *** Guest7090 has quit IRC | 12:24 | |
| *** camunoz has joined #openstack-infra | 12:28 | |
| *** alexchadin has quit IRC | 12:30 | |
| *** iranzo has quit IRC | 12:32 | |
| boden | hi, I have a hopefully painless question.. with the current requirements system should we still expect a proposal bot bump for requirements updates, or do we just update requirements ourselves within the scope of the upper constraints?? I’m sure there’s a note on this and I missed it; sorry for that | 12:32 |
| *** yamamoto has quit IRC | 12:33 | |
| *** VW has joined #openstack-infra | 12:37 | |
| *** lucas-hungry is now known as lucasagomes | 12:38 | |
| *** gongysh has quit IRC | 12:38 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| *** kgiusti has joined #openstack-infra | 12:40 | |
| *** psachin has quit IRC | 12:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Clarify in doc introduction that OpenStack is not required https://review.openstack.org/564746 | 12:44 |
| *** alexchadin has joined #openstack-infra | 12:45 | |
| *** VW has quit IRC | 12:46 | |
| *** jpena|lunch is now known as jpena | 12:54 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix zuul home directory in zuul from scratch document https://review.openstack.org/564386 | 12:57 |
| openstackgerrit | Merged openstack-infra/nodepool master: Clarify in doc introduction that OpenStack is not required https://review.openstack.org/564746 | 12:57 |
| *** wolverineav has joined #openstack-infra | 13:00 | |
| *** quiquell|rover is now known as quiquell|roger | 13:02 | |
| *** quiquell|roger is now known as quiquell|rover | 13:02 | |
| *** panda|ruck|lunch is now known as panda|ruck | 13:03 | |
| *** ethfci has quit IRC | 13:03 | |
| *** alexchadin has quit IRC | 13:05 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add fedora-28 to AFS mirrors https://review.openstack.org/564726 | 13:05 |
| pabelanger | clarkb: comment on https://review.openstack.org/563788/ about log-inventory location | 13:09 |
| *** mriedem has joined #openstack-infra | 13:13 | |
| *** yamamoto has joined #openstack-infra | 13:14 | |
| *** yamamoto has quit IRC | 13:14 | |
| *** yamamoto has joined #openstack-infra | 13:18 | |
| *** VW has joined #openstack-infra | 13:19 | |
| fungi | boden: you may be better off asking in #openstack-requirements (i know some aspects about requirements tracking are changing, and i believe the full synchronization model is being dropped as part of that) | 13:20 |
| boden | fungi ack… didn’t even realize there was a requirements chat room :) | 13:20 |
| AJaeger | boden: quick answer: The bot is not doing proposal updates anymore | 13:21 |
| fungi | boden: yeah, there's an entire official project team devoted to requirements management in openstack, with a ptl and everything | 13:21 |
| boden | AJaeger I thought that was the case; thanks for confirmation | 13:21 |
| pabelanger | fungi: mind a +3 to fix deprecated warning in puppet: https://review.openstack.org/564740/ | 13:21 |
| *** stephenfin is now known as finucannot | 13:23 | |
| *** eernst has joined #openstack-infra | 13:24 | |
| *** r-daneel has joined #openstack-infra | 13:24 | |
| *** trown is now known as trown|brb | 13:26 | |
| *** pbourke has quit IRC | 13:27 | |
| *** pbourke has joined #openstack-infra | 13:28 | |
| *** pbourke has quit IRC | 13:29 | |
| *** Goneri has joined #openstack-infra | 13:29 | |
| *** pcichy has joined #openstack-infra | 13:31 | |
| *** dansmith is now known as superdan | 13:34 | |
| *** bnemec has quit IRC | 13:35 | |
| *** esberglu has joined #openstack-infra | 13:38 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix 'releases' is deprecated warning https://review.openstack.org/564740 | 13:40 |
| fungi | if another config-core reviewer is up for checking https://review.openstack.org/564608 (a small patch to hopefully finish fixing the legacy job for openstackid tarballs so our deployment automation will work again) it would be much appreciated. once it merges i'm happy to reenqueue the openstackid master branch tip | 13:42 |
| *** bnemec has joined #openstack-infra | 13:42 | |
| mnaser | fungi: lgtm | 13:43 |
| fungi | thanks mnaser | 13:44 |
| mnaser | np :> | 13:44 |
| *** kjackal has quit IRC | 13:44 | |
| *** edmondsw has joined #openstack-infra | 13:51 | |
| *** jchhatbar has quit IRC | 13:52 | |
| dhellmann | I need some help with my git repo import patch. AJaeger wants a "groups" entry because I'm having it use storyboard. I don't know what that is or what to make it. This is 1 repo, not owned by a team. Should I just use the repo name? There seem to be a lot of other standalone repos without "groups" | 13:52 |
| dhellmann | https://review.openstack.org/#/c/564622/3/gerrit/projects.yaml | 13:52 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: More fix laravel branch and master publish jobs https://review.openstack.org/564608 | 13:52 |
| dhellmann | maybe not "a lot" but "some" | 13:53 |
| *** kjackal has joined #openstack-infra | 13:54 | |
| AJaeger | dhellmann: repos that use storyboard have groups - can you confirm, fungi? | 13:55 |
| dhellmann | AJaeger : ok, but I DON'T KNOW WHAT THAT IS so I need more information | 13:55 |
| *** dave-mccowan has joined #openstack-infra | 13:55 | |
| dhellmann | the documentation says to use that if there are multiple related repos, which is not the case here | 13:55 |
| *** trown|brb is now known as trown | 13:56 | |
| *** eharney has quit IRC | 13:57 | |
| mordred | yah - groups is really only to group multiple things ... like, for openstacksdk we have an openstacksdk group that contains openstack/openstacksdk and openstack-infra/shade | 13:58 |
| mordred | but it is not a required field for all projects - only ones that want to be part of a grouping | 13:58 |
| mordred | AJaeger, fungi: I think we need to clarify the process - groups should not be reuqired for all storyboard projects - it might just have mostly been the case due to the projects we've migrated so far | 13:59 |
| *** dave-mccowan has quit IRC | 13:59 | |
| dhellmann | the docs I was following do not imply that is a required field: https://docs.openstack.org/infra/manual/creators.html#add-the-project-to-the-master-projects-list | 14:00 |
| *** apetrich has joined #openstack-infra | 14:00 | |
| *** johnthetubaguy_ is now known as johnthetubaguy | 14:00 | |
| dmsimard | infra-root: FYI fresh installs of ARA are currently failing with this kind of trace: http://paste.openstack.org/raw/720015/. This should hopefully not propagate to the executors but keep an eye out and let me know while I'm troubleshooting it ? | 14:00 |
| mordred | dmsimard: ack | 14:01 |
| *** dave-mccowan has joined #openstack-infra | 14:01 | |
| *** yamahata has joined #openstack-infra | 14:01 | |
| *** aojea has quit IRC | 14:01 | |
| *** gongysh has joined #openstack-infra | 14:03 | |
| *** gongysh has quit IRC | 14:03 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Updates for fedora-28 rsync https://review.openstack.org/564771 | 14:04 |
| AJaeger | mordred: that's what I understand as guideline by fungi - let's wait for him to chime in | 14:04 |
| openstackgerrit | Merged openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 14:05 |
| pabelanger | fungi: mordred: dmsimard: mind a review: ^ new files to exclude for fedora-28 rsync | 14:05 |
| dmsimard | infra-root: It looks like flask broke the 0.12.3 python wheel and they released flask 1.0: https://github.com/pallets/flask/issues/2728 | 14:06 |
| dmsimard | Which is why ARA installations are breaking. | 14:06 |
| *** dtantsur is now known as dtantsur|brb | 14:07 | |
| pabelanger | hopefully they don't just delete the wheel and rebuild | 14:08 |
| pabelanger | and do a version bump | 14:08 |
| fungi | mordred: AJaeger: dhellmann: i haven't considered groups necessary for storyboard projects, it's merely that we've been creating groups for official openstack project teams to assist in organizing their sets of deliverables | 14:08 |
| fungi | so, say, when importing bugs for all sahara's projects we create a sahara project group and add those to it | 14:08 |
| dhellmann | I could set the group to "doug" or something. :-) | 14:09 |
| *** alex_xu has quit IRC | 14:09 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Add fedora-28 to nodepool dsvm https://review.openstack.org/559211 | 14:09 |
| mordred | dhellmann: I consider you a group ;) | 14:09 |
| dhellmann | sometimes I feel like it | 14:10 |
| pabelanger | ianw: ^fedora-28 for nodepool dsvm, works out of the box it seems | 14:10 |
| fungi | yes, if there is some (even small) set of people who will likely be interested in tasks for some specific set of repos, then it may make sense to create a grouping around that | 14:10 |
| fungi | for example, the interop and refstack project groups have some overlap in what projects are in them | 14:10 |
| dhellmann | in this case I don't even have a team in mind to help maintain this repo, although I'd be happy to add it as an infra thing or oslo thing if someone wanted to | 14:10 |
| dhellmann | is it easy to add the group later? | 14:10 |
| fungi | yep | 14:11 |
| fungi | just a matter of adding the info to projects.yaml when the time comes | 14:11 |
| dhellmann | so maybe let's do that, if it comes to it | 14:11 |
| *** alex_xu has joined #openstack-infra | 14:11 | |
| *** salv-orlando has quit IRC | 14:11 | |
| *** salv-orlando has joined #openstack-infra | 14:12 | |
| *** hemna_ has joined #openstack-infra | 14:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Build fedora-28 DIBs with nodepool-builder https://review.openstack.org/564775 | 14:13 |
| *** bobh has joined #openstack-infra | 14:13 | |
| pabelanger | clarkb: fungi: mordred: dmsimard: ^and enables fedora-28 (beta) DIBs for diskimage-builder, release should be next week | 14:13 |
| *** salv-orlando has quit IRC | 14:16 | |
| dmsimard | infra-root: Follow up regarding flask/ara breakage, flask maintainer was quick to respond and will release a fixed flask 0.12.4 package so I won't implement a workaround. | 14:16 |
| *** quiquell|rover is now known as quiquell|off | 14:19 | |
| *** ihar has joined #openstack-infra | 14:20 | |
| *** jogo has quit IRC | 14:22 | |
| smcginnis | dmsimard: Is that the tracebacks I've been seeing in logs? ^ | 14:22 |
| dmsimard | smcginnis: where ? | 14:23 |
| *** agopi|zzz has joined #openstack-infra | 14:23 | |
| smcginnis | dmsimard: Let me find an example... | 14:23 |
| *** agopi|zzz is now known as agopi | 14:23 | |
| openstackgerrit | Merged openstack-dev/hacking master: Import integration test https://review.openstack.org/547765 | 14:24 |
| smcginnis | Of course I saw a few last night and now can't find any... | 14:25 |
| smcginnis | dmsimard: Found it! | 14:26 |
| smcginnis | http://logs.openstack.org/68/564068/6/gate/legacy-grenade-dsvm-cinder-mn-sub-volbak/acfeb79/job-output.txt.gz#_2018-04-26_23_10_20_689402 | 14:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy-hacking-integration-nova https://review.openstack.org/564786 | 14:26 |
| dmsimard | smcginnis: yes, that's the trace | 14:28 |
| smcginnis | dmsimard: OK, great. Then I don't have to remember to ask about it anymore. ;) | 14:28 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-hacking-integration-nova https://review.openstack.org/564787 | 14:29 |
| dmsimard | smcginnis: thankfully it doesn't appear fatal but that job is generating HTML reports ? We're not supposed to do that anymore | 14:29 |
| AJaeger | fungi, so the change by dhellman is fine as is? Then I'll +2A... | 14:29 |
| smcginnis | dmsimard: Not sure. | 14:29 |
| dmsimard | "legacy-grenade-dsvm-cinder-mn-sub-volbak" feels oddly specific, I'll have a look | 14:30 |
| mordred | dmsimard: bah. legacy-grenade-dsvm-cinder-mn-sub-volbak is perfectly generic ;) | 14:30 |
| dmsimard | probably need to throw the python version, the distro and the name of the openstack release in there | 14:31 |
| *** yamamoto has quit IRC | 14:31 | |
| *** krenczew1 has quit IRC | 14:33 | |
| *** andreas_s has quit IRC | 14:34 | |
| *** dave-mccowan has quit IRC | 14:42 | |
| *** dave-mccowan has joined #openstack-infra | 14:43 | |
| fungi | AJaeger: probably? i haven't looked at the change in question but i'll review it now | 14:46 |
| fungi | snice i seem to have context | 14:46 |
| *** dave-mcc_ has joined #openstack-infra | 14:46 | |
| fungi | er, since | 14:46 |
| *** yamamoto has joined #openstack-infra | 14:46 | |
| *** melwitt is now known as jgwentworth | 14:47 | |
| mordred | pabelanger: btw - I put up a patch to get a shade release cut: https://review.openstack.org/#/c/564752/ - so as soon as that lands we should be able to make forward progress on the cinder volume thing | 14:48 |
| *** dave-mccowan has quit IRC | 14:48 | |
| *** pcichy has quit IRC | 14:48 | |
| pabelanger | mordred: great | 14:49 |
| openstackgerrit | Merged openstack-infra/project-config master: import the git-nit repository https://review.openstack.org/564622 | 14:49 |
| *** hongbin_ has joined #openstack-infra | 14:50 | |
| smcginnis | mordred: Is that time sensitive enough to warrant a Friday release? We usually avoid releasing on Friday's unless it's something blocking other time sensitive work. | 14:52 |
| *** rpittau has quit IRC | 14:52 | |
| mordred | pabelanger: ^^ ? | 14:54 |
| pabelanger | mordred: smcginnis: no, think it can wait until normal release procedures. vexxhost is disabled currently | 14:56 |
| pabelanger | and allows me to go see Avengers at lunch :D | 14:56 |
| *** dtantsur|brb is now known as dtantsur | 14:56 | |
| smcginnis | OK, sounds good. We can get it processed right away Monday. Or Sunday night if I get to it. | 14:56 |
| smcginnis | :) | 14:56 |
| smcginnis | Good plan! | 14:56 |
| fungi | yeah, at worst it just means we're down a chunk of job capacity over the weekend, when we likely won't be needing it anyway | 14:56 |
| smcginnis | What is the vexxhost issue with cinder? | 14:57 |
| pabelanger | it is leaking volumes for some reasons | 14:57 |
| pabelanger | when we use boot-from-volume with nova | 14:57 |
| pabelanger | so, shade will allow us to implement quota management in nodepool | 14:58 |
| *** yamamoto has quit IRC | 14:58 | |
| dmsimard | infra-root: There's 2718 hits on the Flask ARA issue ("AttributeError: 'Blueprint' object has no attribute 'json_encoder'") in the past 24hrs according to logstash. Probably worthy of an email to openstack-dev or a notice ? | 15:04 |
| mordred | smcginnis: yah - they *why* is a whole other question that we need to dig in to | 15:05 |
| mordred | smcginnis: but in the mean time - working around it | 15:05 |
| *** VW_ has joined #openstack-infra | 15:05 | |
| pabelanger | dmsimard: these are jobs not using the new database wsgi app? | 15:05 |
| *** VW_ has quit IRC | 15:05 | |
| *** rajinir has joined #openstack-infra | 15:06 | |
| *** VW_ has joined #openstack-infra | 15:06 | |
| dmsimard | pabelanger: There's a lot of different jobs, some non-legacy, I'm trying to get a clear idea | 15:06 |
| pabelanger | could I get a review on https://review.openstack.org/564771/ to fix fedora rsync mirror | 15:06 |
| smcginnis | mordred, pabelanger: Ack, thanks | 15:07 |
| dmsimard | pabelanger: for example http://logs.openstack.org/29/564729/2/check/neutron-grenade/2dcf4e6/job-output.txt#_2018-04-27_14_43_20_048985 | 15:07 |
| dmsimard | that seems to be... devstack-gate ? | 15:07 |
| pabelanger | dmsimard: ack, maybe we should use the outage as a method to update those jobs to ara-report database logic | 15:07 |
| *** yamamoto has joined #openstack-infra | 15:07 | |
| mordred | pabelanger: oh. blerg. the leaked volumes in vexxhost have been cleaned up now haven't they | 15:08 |
| * mordred should have poked yesterday | 15:08 | |
| dmsimard | pabelanger: yeah it's probably devstack-gate: https://github.com/openstack-infra/devstack-gate/blob/649d1ae775867568f0aba582361c3489250b9aad/devstack-vm-gate-wrap.sh#L539 | 15:08 |
| pabelanger | mordred: I am not sure, maybe mnaser did it | 15:08 |
| mordred | it's all good | 15:08 |
| mnaser | hi | 15:08 |
| *** felipemonteiro__ has joined #openstack-infra | 15:08 | |
| *** VW has quit IRC | 15:08 | |
| mordred | pabelanger, mnaser: next time we discover leaked volumes - if we can capture the volume record of a few of them that would be great - the main question is 'do the leaked volumes still show an attachment in the attachments list pointing to a now-non-existent server or not | 15:09 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/openstack-zuul-jobs master: Yet more fix laravel branch/master publish jobs https://review.openstack.org/564799 | 15:09 |
| fungi | pabelanger: mnaser: ^ one more bit we missed | 15:09 |
| mordred | if they do - then making a leaked volume cleaner is doable | 15:09 |
| fungi | should have been using work_root instead of log_root | 15:09 |
| mordred | since we can look for volumes that list attachments to non-existent volumes | 15:09 |
| mordred | since we can look for volumes that list attachments to non-existent servers | 15:10 |
| pabelanger | sure | 15:10 |
| *** jamesmcarthur has joined #openstack-infra | 15:10 | |
| mordred | but if that part of the volume record gets cleaned up yet the volume sticks around - I think deleting it would be very dangerous | 15:10 |
| mnaser | mordred: actually, it might be worse, because you cant delete a volume that is attached to a non existent server, nor can you remove the attachment | 15:10 |
| mordred | mnaser: that's excellent | 15:10 |
| mnaser | mordred: i think the problem stems from nova ignoring volume delete failures | 15:10 |
| mordred | mnaser: did you clean up the volumes in the openstackjenkins project? or did they just eventually clean themselves up | 15:11 |
| *** pcaruana has quit IRC | 15:11 | |
| pabelanger | all the volume I cleaned up, were detached and able to just use openstack volume delete uuid | 15:11 |
| mnaser | mordred: i think i deleted them because it was related to some internal cinder-volume thing | 15:11 |
| mnaser | i guess then maybe the issue might be resolved | 15:11 |
| mordred | oh - wait - there's more in here - I suck at python | 15:11 |
| *** yamamoto has quit IRC | 15:12 | |
| *** salv-orlando has joined #openstack-infra | 15:12 | |
| mnaser | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2370-L2415 | 15:13 |
| mnaser | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2441-L2450 | 15:13 |
| pabelanger | infra-root: I'm going to take an extended lunch today, will be back later in the afternoon. I'd love a few reviews on https://review.openstack.org/#/q/topic:fedora-28 to continue work on fedora-28 when I return. May 1st looks to be the release day now | 15:13 |
| mnaser | fungi: +A | 15:14 |
| fungi | thanks pabelanger, mnaser... i'll reenqueue the openstackid branch tip yet again once that fix lands | 15:15 |
| *** shardy has quit IRC | 15:17 | |
| *** yamamoto has joined #openstack-infra | 15:17 | |
| *** salv-orlando has quit IRC | 15:17 | |
| *** dpawlik has quit IRC | 15:17 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: Fix scope of vhost template variable https://review.openstack.org/564600 | 15:17 |
| mordred | pabelanger, mnaser: from what I can see, there are 11 leaked volumes in the vexxhost account. most of them have attachments listed to non-existent servers | 15:19 |
| *** jogo has joined #openstack-infra | 15:19 | |
| *** jogo has joined #openstack-infra | 15:19 | |
| *** VW_ has quit IRC | 15:20 | |
| mordred | but mnaser says a volume with a non-existent server attachment can't be deleted | 15:20 |
| mordred | so I'm honestly not sure there is a leak cleaner we can write that I'd be comfortable landing | 15:21 |
| mnaser | which means we can to reset the attachment state | 15:21 |
| mnaser | s/can/have/ | 15:21 |
| mordred | mnaser: also - those code links | 15:21 |
| mordred | mnaser: yah - which you have to do on your side I think | 15:21 |
| mnaser | i can reset it but at the scale and churn of openstack infra it wouldn't be a sustainable solution | 15:21 |
| mordred | mnaser: I love that a failure during volume deletion just logs a warning | 15:21 |
| mordred | mnaser: agree | 15:22 |
| mnaser | yeah because of that i try to push customers to create a volume then attach it | 15:22 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-zuul master: Fix defined type namespacing https://review.openstack.org/564809 | 15:22 |
| mordred | mnaser: does that work with boot-from-volme? | 15:22 |
| mnaser | mordred: i think this won't be an issue, if you do boot from volume, it's on you to delete the volume when you're done | 15:22 |
| pabelanger | mordred: mnaser: so, openstack volume delete doesn't work? | 15:23 |
| pabelanger | if so, something changed recently, as I was able to manually clean them up | 15:23 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Fix defined type namespacing https://review.openstack.org/564535 | 15:23 |
| mnaser | but then again if it fails to detach then and just logs an error for that.. | 15:23 |
| *** rkukura has joined #openstack-infra | 15:23 | |
| mnaser | you wont be able to delete it either | 15:23 |
| pabelanger | okay, have to run, will catch up in backscroll | 15:23 |
| mnaser | i really don't think nova should just log a message and then make you end up in some inconsistent state where admin intervention is required | 15:23 |
| mordred | mnaser: well - yeah, what's what we're doing now - we're doing boot-on-volume but doing it from an image source - so the boot-from-volume process is creating the volume for us and then booting from it | 15:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-gerrit master: [DNM] Test puppet-4 puppet-apply jobs https://review.openstack.org/557515 | 15:24 |
| mordred | mnaser: I suppose we could change the code to create a volume from the image, then boot the server on it without delete-on-termination | 15:24 |
| mordred | mnaser: and on server deletion invert the process | 15:24 |
| mordred | oh - but I guess that still has the attachment problem like you said | 15:25 |
| mordred | nodepool - finding intractable bugs since 2012 | 15:25 |
| mnaser | but then you'll have one more resource to carry *and* yeah if it fails to detach the vm will disappear and poof | 15:25 |
| *** jamesmcarthur has quit IRC | 15:25 | |
| *** claudiub has quit IRC | 15:26 | |
| mnaser | mordred: this has unfortunately been a common frustration which has resulted in us creating custom flavors for customers with a certain amount of disk | 15:27 |
| mnaser | and i *hate* doing that because i feel like we should fix that issue fundamentally | 15:27 |
| mordred | agree | 15:28 |
| mnaser | i much rather have an ERROR state VM that failed to delete with a volume stuck rather than a disappearing VM with an 'attached' volume | 15:28 |
| mordred | well - that or I do wish it was a generally possible thing for users to create custom flavors | 15:28 |
| mordred | mnaser: ++ | 15:28 |
| mnaser | it also allows the user to just ... try deleting again | 15:28 |
| mnaser | and it might just work | 15:28 |
| mordred | smcginnis: ^^ more context on what's going on with the volume thing | 15:28 |
| smcginnis | Thanks, will have to catch up after meeting. | 15:29 |
| clarkb | I too agree that properly recording errors is better than failing to succeed and disappearing data | 15:29 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 15:29 |
| mordred | smcginnis: tl;dr boot-from-volume-from-image can leave volumes in an undeletable state if anything goes wrong when nova tries to delete the volume | 15:30 |
| *** VW has joined #openstack-infra | 15:30 | |
| smcginnis | Ah, yeah. I've seen that issue crop up here and there. Not much we can do on the Cinder side unfortunately. | 15:30 |
| clarkb | you could delete the volume properly? | 15:31 |
| clarkb | I'm assuming nova doesn't have issues in that case | 15:31 |
| mnaser | clarkb: well i think you cant blame cinder for having some intermittant issues here and there | 15:32 |
| mnaser | cinder just has no way to allow nova to recover/retry | 15:32 |
| *** myoung is now known as myoung|email-unl | 15:32 | |
| *** myoung|email-unl is now known as myoung|emailplz | 15:32 | |
| mordred | clarkb: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2441-L2450 is the fun part | 15:33 |
| smcginnis | clarkb: Well, we can delete the volume properly if told to, but if Nova doesn't tell us to then we can't randomly guess. | 15:33 |
| clarkb | smcginnis: reading ^ nova is telling cinder? its that cinder fails then nova doesn't handle that gracefully? | 15:33 |
| *** Guest44397 has quit IRC | 15:33 | |
| clarkb | both sides can and should do better | 15:33 |
| mordred | the problem is that if there is an issue, like say a communication error, nova logs a warning then continues to delete the server - so the volume is left in an inappropriate state | 15:33 |
| fungi | or if nova tries to tell cinder to delete but the message ends up in a black hole somewhere | 15:33 |
| fungi | yeah, that | 15:34 |
| *** agopi is now known as agopi|brb | 15:34 | |
| mordred | because from cinder's pov it's still attached | 15:34 |
| smcginnis | The errors I've seen were Nova getting some type of error and never telling Cinder to delete the volume. | 15:34 |
| mordred | yah. could be a 500 from an internal load balancer, for instance | 15:34 |
| mnaser | mordred: oh interesting | 15:34 |
| clarkb | mordred: do we have tracebacks from that warning? | 15:34 |
| mnaser | that function would raise an exception | 15:34 |
| mnaser | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2502-L2503 | 15:34 |
| clarkb | and ya its reraising there | 15:35 |
| smcginnis | It's the problem with having Nova manage the process rather than going directly to each service to perform their own operations. | 15:35 |
| mordred | ah - ok | 15:35 |
| *** jaosorior has quit IRC | 15:35 | |
| *** yamamoto has quit IRC | 15:35 | |
| mnaser | i think dropping 'raise_exc=False' from that line would fix it | 15:35 |
| mnaser | but i guess there's a reason behind it | 15:35 |
| mnaser | git blame time | 15:35 |
| mordred | mnaser: :) | 15:36 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Yet more fix laravel branch/master publish jobs https://review.openstack.org/564799 | 15:36 |
| clarkb | smcginnis: sort of, nova should be just as capable of doing it as nodepool | 15:36 |
| mnaser | git blame on nova/manager/compute.py crashes on github, nice | 15:36 |
| *** gyee has joined #openstack-infra | 15:36 | |
| mordred | yah- because the volume thing here is an impl detail | 15:36 |
| mordred | as a user, what we want here is a root filesystem of a certain size on a server we are booting to be ephemeral | 15:36 |
| mordred | the only way we can do that is by creating a volume | 15:37 |
| mordred | I mena - we can still try skipping the bdm layer and doing it ourselves | 15:37 |
| *** agopi|brb is now known as agopi | 15:38 | |
| *** lpetrut has quit IRC | 15:38 | |
| mnaser | oh | 15:40 |
| mnaser | https://review.openstack.org/#/c/545087/ | 15:40 |
| *** jamesmcarthur has joined #openstack-infra | 15:40 | |
| mnaser | ok sorry thats the stable/ocata backport | 15:40 |
| mnaser | https://review.openstack.org/#/q/I8b1c05317734e14ea73dc868941351bb31210bf0 | 15:40 |
| mnaser | moral of the story upgrade your clouds i guess | 15:41 |
| mnaser | i guess this is only partly a fix for one failure path | 15:42 |
| openstackgerrit | Merged openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/553596 | 15:50 |
| *** caphrim007 has joined #openstack-infra | 15:51 | |
| *** VW_ has joined #openstack-infra | 15:51 | |
| *** VW has quit IRC | 15:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: add test jobs for openstack/git-nit repository https://review.openstack.org/564625 | 15:55 |
| *** VW_ has quit IRC | 15:55 | |
| *** pcichy has joined #openstack-infra | 15:55 | |
| *** VW has joined #openstack-infra | 15:55 | |
| *** hashar is now known as hasharAway | 15:56 | |
| *** dpawlik has joined #openstack-infra | 15:57 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:57 | |
| *** lucasagomes is now known as lucas-afk | 15:58 | |
| *** eharney has joined #openstack-infra | 16:01 | |
| *** felipemonteiro__ has quit IRC | 16:01 | |
| *** dpawlik has quit IRC | 16:02 | |
| *** bnemec is now known as beekneemech | 16:02 | |
| openstackgerrit | David Peacock proposed openstack-infra/project-config master: Adding tripleo-role-keystone https://review.openstack.org/564819 | 16:03 |
| *** hasharAway is now known as hashar | 16:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Allow extra time for some ansible tests https://review.openstack.org/564572 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix race in test_bubblewrap_leak https://review.openstack.org/564640 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of secrets https://review.openstack.org/553041 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Perform late validation of nodesets https://review.openstack.org/553088 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Late bind projects https://review.openstack.org/553618 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Make config objects freezable https://review.openstack.org/562816 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove layout from ParseContext https://review.openstack.org/563695 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove 'base' from UnparsedAbideConfig https://review.openstack.org/563757 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Change TestMaxTimeout to not run ansible https://review.openstack.org/564562 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Store source context on all config objects https://review.openstack.org/564563 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Cache configuration objects in addition to YAML dicts https://review.openstack.org/564061 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Stop deep-copying job variables https://review.openstack.org/564564 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove source_context argument to Pipeline https://review.openstack.org/564642 | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Simplify UnparsedConfig.copy https://review.openstack.org/564647 | 16:05 |
| *** zerick_ is now known as zerick | 16:08 | |
| dhellmann | could I get someone to add me to the git-nit-core group in gerrit please? https://review.openstack.org/#/admin/groups/1900,members | 16:09 |
| clarkb | dhellmann: one sec | 16:09 |
| clarkb | done | 16:09 |
| dhellmann | clarkb : thanks! | 16:09 |
| openstackgerrit | David Peacock proposed openstack-infra/project-config master: Add system-required template for tripleo-role-keystone https://review.openstack.org/564823 | 16:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add regex support to project stanzas https://review.openstack.org/535713 | 16:12 |
| *** salv-orlando has joined #openstack-infra | 16:13 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/openstack-zuul-jobs master: Further fix laravel branch/master publish jobs https://review.openstack.org/564828 | 16:14 |
| fungi | pabelanger: mnaser: ^ i think this is the last fix we need for openstackid tarball publication (it's publishing successfully now, but to a subdirectory we don't want) | 16:14 |
| fungi | i'll clean up the http://tarballs.openstack.org/openstackid/tarballs/ directory and reenqueue (hopefully one final time) once that's in place | 16:15 |
| *** dtantsur is now known as dtantsur|afk | 16:15 | |
| openstackgerrit | David Peacock proposed openstack-infra/project-config master: Adding tripleo-role-keystone https://review.openstack.org/564819 | 16:18 |
| *** salv-orlando has quit IRC | 16:18 | |
| *** VW has quit IRC | 16:19 | |
| openstackgerrit | David Peacock proposed openstack-infra/project-config master: Adding tripleo-role-keystone https://review.openstack.org/564819 | 16:20 |
| *** VW has joined #openstack-infra | 16:23 | |
| *** jpich has quit IRC | 16:24 | |
| *** fried_rice is now known as fried_rolls | 16:27 | |
| *** VW has quit IRC | 16:27 | |
| clarkb | corvus: since you are talking about restarting zuul I've just checked that zuul is being installed properly and pbr seems to report the current HEAD sha1 | 16:27 |
| *** r-daneel_ has joined #openstack-infra | 16:29 | |
| *** r-daneel has quit IRC | 16:29 | |
| *** r-daneel_ is now known as r-daneel | 16:29 | |
| clarkb | fungi: openstackid fix approved | 16:30 |
| *** felipemonteiro_ has quit IRC | 16:31 | |
| *** felipemonteiro__ has joined #openstack-infra | 16:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add allowed-triggers and allowed-reporters tenant settings https://review.openstack.org/554082 | 16:31 |
| *** lpetrut has joined #openstack-infra | 16:32 | |
| corvus | clarkb: yay! we're probably a few hours out from patches landing; a bit of a backlog (who works on fridays ?!) | 16:33 |
| *** e0ne has joined #openstack-infra | 16:34 | |
| *** derekh has quit IRC | 16:35 | |
| *** ccamacho has quit IRC | 16:35 | |
| *** zoli is now known as zoli|gone | 16:36 | |
| *** yamamoto has joined #openstack-infra | 16:36 | |
| *** jamesmcarthur has quit IRC | 16:36 | |
| *** trown is now known as trown|lunch | 16:40 | |
| *** e0ne has quit IRC | 16:41 | |
| *** eharney has quit IRC | 16:41 | |
| *** yamamoto has quit IRC | 16:42 | |
| openstackgerrit | Merged openstack-infra/zuul master: Allow extra time for some ansible tests https://review.openstack.org/564572 | 16:44 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Further fix laravel branch/master publish jobs https://review.openstack.org/564828 | 16:44 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix race in test_bubblewrap_leak https://review.openstack.org/564640 | 16:45 |
| openstackgerrit | Merged openstack-infra/zuul master: Perform late validation of secrets https://review.openstack.org/553041 | 16:58 |
| openstackgerrit | Merged openstack-infra/zuul master: Perform late validation of nodesets https://review.openstack.org/553088 | 16:58 |
| openstackgerrit | Merged openstack-infra/zuul master: Late bind projects https://review.openstack.org/553618 | 16:58 |
| AJaeger | infra-root, FYI, I'll be travelling to KubeCon next week - don't expect to see much from me... | 17:02 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-zuul master: Fix scope of vhost template variable https://review.openstack.org/564844 | 17:02 |
| clarkb | AJaeger: have fun | 17:02 |
| fungi | AJaeger: enjoy it! | 17:02 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-gerrit master: [DNM] Test puppet-4 puppet-apply jobs https://review.openstack.org/557515 | 17:03 |
| AJaeger | thanks, clarkb and fungi | 17:03 |
| *** VW has joined #openstack-infra | 17:06 | |
| *** pbourke has joined #openstack-infra | 17:07 | |
| *** eernst has quit IRC | 17:07 | |
| *** jpena is now known as jpena|off | 17:07 | |
| *** eernst has joined #openstack-infra | 17:08 | |
| *** tbh has joined #openstack-infra | 17:08 | |
| *** salv-orlando has joined #openstack-infra | 17:09 | |
| *** tesseract has quit IRC | 17:09 | |
| *** yamahata has quit IRC | 17:09 | |
| *** pbourke has quit IRC | 17:10 | |
| tbh | Hi, I got access rights for openstack/meteos project. I want to start work on this project. There is no much work psuhed to this repo. I am not sure whom to inform to not to remove this repo because there is no activity for a while | 17:10 |
| *** pbourke has joined #openstack-infra | 17:10 | |
| clarkb | tbh: if it hasn't been retired already (doesn't apepar to have been) and there is activity on it we'll leave it be | 17:11 |
| *** pbourke has quit IRC | 17:12 | |
| *** pbourke has joined #openstack-infra | 17:13 | |
| fungi | we don't automatically retire repositories anyway | 17:13 |
| tbh | clarkb, there is no activity on this repositories for many months... so am informing | 17:14 |
| tbh | clarkb, fungi, am going to push code next week | 17:15 |
| *** VW_ has joined #openstack-infra | 17:15 | |
| *** VW_ has quit IRC | 17:17 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: Fix scope of vhost template variables https://review.openstack.org/564600 | 17:17 |
| *** VW_ has joined #openstack-infra | 17:17 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-zuul master: Fix scope of vhost template variables https://review.openstack.org/564844 | 17:18 |
| *** VW has quit IRC | 17:18 | |
| rosmaita | good afternoon ... if someone has a few spare minutes, i'm seeing this weird failure in the glance functional-py35 tests ... http://logs.openstack.org/49/564649/1/check/openstack-tox-functional-py35/b6f5675/testr_results.html.gz -- the subunit parser thing ... anyone know what causes that? | 17:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Report git sha in status page version https://review.openstack.org/564628 | 17:21 |
| *** mriedem is now known as mriedem_afk | 17:21 | |
| *** VW_ has quit IRC | 17:22 | |
| *** kmalloc has joined #openstack-infra | 17:22 | |
| *** germs has quit IRC | 17:22 | |
| *** germs has joined #openstack-infra | 17:23 | |
| *** germs has quit IRC | 17:23 | |
| *** germs has joined #openstack-infra | 17:23 | |
| clarkb | rosmaita: I'll look once my tea is made | 17:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add release note about re2 https://review.openstack.org/564847 | 17:27 |
| rosmaita | clarkb: appreciate it | 17:29 |
| corvus | clarkb, cmurphy, pabelanger, mordred, fungi: thinking more about the re2 thing -- maybe we should be running bindep in puppet-zuul (and other modules for projects which now have bindep files)? | 17:30 |
| clarkb | corvus: ya its probably not a bad idea, basically puppet exec `bash -c 'apt-get install -y $(bindep -b /opt/zuul/bindep.txt)'` | 17:31 |
| cmurphy | corvus: that sounds good to me | 17:33 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 17:34 |
| clarkb | rosmaita: my reading of it is that the subunit data stream (which is basically the output of the test runs for tracking success and failure as well as logging and stdout failed to parse due to checksum errors | 17:37 |
| clarkb | rosmaita: I think due to incomplete data sending | 17:37 |
| clarkb | rosmaita: I grabbed the subunit file and converted it to version 1 (whcih is more human readable) to see the checksum errors | 17:38 |
| *** yamamoto has joined #openstack-infra | 17:38 | |
| clarkb | rosmaita: if I had to guess why that happened the tests or testsuite crashed the output somehow | 17:39 |
| clarkb | (closing the file descriptor possibly) | 17:39 |
| clarkb | rosmaita: there are a lot of novaclient exceptions which probably aren't related but unsure if you know about those | 17:41 |
| *** amoralej is now known as amoralej|off | 17:41 | |
| fungi | corvus: clarkb: cmurphy: i missed whatever "the re2 thing" is, but sounds like it might be worth a try. i worry still that we could end up installing python modules from distro packages that way and then end up attempting to pip upgrade them when continuously deploying python-based services from source, but we can cross that bridge if and when we come to it | 17:41 |
| clarkb | fungi: basically zuul added a new system dep which was properly reflected inbindep but puppetry broke | 17:42 |
| clarkb | dep in this case is libre2-dev iirc | 17:42 |
| *** yamamoto has quit IRC | 17:43 | |
| rosmaita | clarkb: thanks ... that sounds like a kind of random thing (the subunit problems), not a problem with the tests themselves? | 17:44 |
| *** dhill_ has quit IRC | 17:44 | |
| fungi | clarkb: yeah, seems like a reasonable approach then | 17:44 |
| clarkb | rosmaita: possibly, though seems more likely the tests are breaking subunit somehow. (like closing the socket fd) | 17:45 |
| fungi | clarkb: the software declares its distro package dependencies, so puppet can certainly take advantage of that and allow us to stop double-accounting all the package names | 17:45 |
| clarkb | rosmaita: unfortuntely there is a bit of chicken and egg here with this being how we get the logs so not having logs its hard to see what was going on | 17:45 |
| *** yamahata has joined #openstack-infra | 17:45 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:45 | |
| clarkb | rosmaita: I'd probably start by checking if it happens consistently on the change, then work to clean up these novaclient exceptions so the logs are readable (assuming we don't want those exceptions to be happening at all) | 17:46 |
| corvus | fungi: you're suggesting that maybe a bindep-specified package might pull in an unexpected python-foo distro package? | 17:46 |
| *** armaan has quit IRC | 17:46 | |
| fungi | corvus: probably not for our python-based services, but maybe if we're deploying multiple things onto one server | 17:46 |
| fungi | then again, we probably only have to worry about our (python-based) projects making use of bindep anyway | 17:46 |
| fungi | and those wouldn't put their python deps in bindep.txt | 17:47 |
| rosmaita | clarkb: thanks! | 17:47 |
| fungi | corvus: so i think i've talked myself out of any real concern, at least until bindep becomes much more popular for non-python projects | 17:47 |
| clarkb | rosmaita: then hopefully we write enough data before things get kille to see what is going on (since it won't be full of exception data due to novaclient) | 17:47 |
| fungi | corvus: the only downside i can see is that it's not very puppety, but maybe someone with better puppet-fu than me can work out how to turn bindep output into a resource iterator | 17:49 |
| *** ykarel is now known as ykarel|away | 17:49 | |
| *** felipemonteiro__ has quit IRC | 17:49 | |
| *** trown|lunch is now known as trown | 17:50 | |
| *** dhill_ has joined #openstack-infra | 17:50 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-kibana master: Fix scope of vhost template variables https://review.openstack.org/564853 | 17:51 |
| *** gfidente has quit IRC | 17:52 | |
| *** rkukura has quit IRC | 17:53 | |
| *** ykarel|away has quit IRC | 17:54 | |
| *** armaan has joined #openstack-infra | 17:55 | |
| *** salv-orlando has quit IRC | 17:55 | |
| *** salv-orlando has joined #openstack-infra | 17:56 | |
| cmurphy | got a successful puppet apply test with puppet 4 :D https://review.openstack.org/#/c/557515/ | 17:57 |
| clarkb | time to start reviewing the depends on list I guess | 17:58 |
| *** dpawlik has joined #openstack-infra | 17:58 | |
| *** caphrim007_ has joined #openstack-infra | 17:59 | |
| *** rkukura has joined #openstack-infra | 18:00 | |
| *** salv-orlando has quit IRC | 18:01 | |
| *** dpawlik has quit IRC | 18:02 | |
| *** caphrim007 has quit IRC | 18:03 | |
| *** panda|ruck is now known as panda|ruck|off | 18:07 | |
| *** salv-orlando has joined #openstack-infra | 18:08 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add debug info to test_slow_start https://review.openstack.org/564857 | 18:11 |
| clarkb | cmurphy: https://review.openstack.org/#/c/564577/1 that one has a failing test and the log isn't really clear on what is broken | 18:11 |
| clarkb | cmurphy: have you happened to look into that yet? | 18:12 |
| clarkb | oh I scrolled too far past the erorr | 18:12 |
| clarkb | http://logs.openstack.org/77/564577/1/check/legacy-puppet-beaker-rspec-infra/902e108/job-output.txt.gz#_2018-04-26_18_53_59_544750 I'm guessing that is the libre2-dev problem I will recheck | 18:12 |
| *** kjackal has quit IRC | 18:13 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add regex support to project stanzas https://review.openstack.org/535713 | 18:14 |
| cmurphy | clarkb: oh yeah that was where i found that, forgot to recheck it | 18:14 |
| clarkb | cmurphy: comment on https://review.openstack.org/#/c/564601/1 | 18:14 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-storyboard master: Use versioncmp function https://review.openstack.org/564601 | 18:16 |
| cmurphy | clarkb: fixed | 18:16 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-mailman master: Fix scope of vhost template variables https://review.openstack.org/564859 | 18:18 |
| clarkb | cmurphy: looking at https://review.openstack.org/#/c/564609/2/templates/dual-elasticsearch.vhost.erb won't @elasticsearch_prefix and @elasticsearch_url need to convert to lookups like you did in the logs vhost for openstackci? | 18:18 |
| clarkb | I'm guessing that will be the next wave of fixes? | 18:18 |
| *** pcichy has quit IRC | 18:19 | |
| cmurphy | clarkb: yeah already working on that https://review.openstack.org/#/c/564853/1/templates/dual-elasticsearch.vhost.erb | 18:19 |
| cmurphy | clarkb: it's a little scary because the apply test doesn't always catch those | 18:19 |
| clarkb | ya since its a noop it doesn't actually write to disk | 18:20 |
| clarkb | cmurphy: in those cases does it substitute empty string too rather than erroring? | 18:20 |
| cmurphy | clarkb: yeah :( | 18:20 |
| clarkb | gg | 18:21 |
| cmurphy | er no it substitutes nil | 18:21 |
| clarkb | oh even better | 18:21 |
| cmurphy | which i didn't expect because otherwise puppet 4 seems stricter about types | 18:21 |
| *** ykarel|away has joined #openstack-infra | 18:23 | |
| *** markmcclain has quit IRC | 18:25 | |
| *** hwoarang has joined #openstack-infra | 18:27 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstack_health master: Fix scope of vhost template variables https://review.openstack.org/564860 | 18:30 |
| *** electrofelix has quit IRC | 18:31 | |
| *** markmcclain has joined #openstack-infra | 18:32 | |
| *** r-daneel_ has joined #openstack-infra | 18:33 | |
| *** r-daneel has quit IRC | 18:34 | |
| *** r-daneel_ is now known as r-daneel | 18:34 | |
| *** armaan has quit IRC | 18:35 | |
| mordred | corvus: I think we should be running bindep - but I also think we need to poke at our bindep file a little bit | 18:38 |
| *** yamamoto has joined #openstack-infra | 18:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul master: Fix description for DependentPipelineManager https://review.openstack.org/564862 | 18:40 |
| mordred | corvus: for instance, it declares that we need zookeeperd - which is not true on all of the zuul nodes | 18:40 |
| clarkb | mordred: that is for the test jobs right? so the test profile would be appropriate | 18:41 |
| *** rkukura has quit IRC | 18:41 | |
| mordred | clarkb: yah - there's also a setof things that are important if you're installing deps from source without wheels available, but which if you do have wheels are not needed (although they obviously don't hurt) | 18:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul master: Fix some code description https://review.openstack.org/564862 | 18:44 |
| clarkb | mordred: in that case I think we err on the side of caution because nothing requires an upstream to continue producing wheels | 18:44 |
| mordred | ya | 18:45 |
| *** yamamoto has quit IRC | 18:45 | |
| clarkb | this remeinds me I should look at fixing the bindep tests in my use distro change | 18:49 |
| corvus | mordred: oh yeah, we should resolve that before we do that in puppet. or we'll have a bunch of zookeeperds :) | 18:49 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-refstack master: Fix scope of vhost template variables https://review.openstack.org/564863 | 18:50 |
| *** salv-orlando has quit IRC | 18:50 | |
| openstackgerrit | Merged openstack-infra/zuul master: Make config objects freezable https://review.openstack.org/562816 | 18:51 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove layout from ParseContext https://review.openstack.org/563695 | 18:51 |
| *** fried_rolls is now known as fried_rice | 18:56 | |
| fungi | right, we could consider making more judicious use of profiles in zuul's (and nodepool's) bindep.txt. like a scheduler profile, an executor profile, a merger profile, a launcher profile, a builder profile and so on | 18:57 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove 'base' from UnparsedAbideConfig https://review.openstack.org/563757 | 18:58 |
| openstackgerrit | Merged openstack-infra/zuul master: Change TestMaxTimeout to not run ansible https://review.openstack.org/564562 | 18:58 |
| openstackgerrit | Merged openstack-infra/zuul master: Store source context on all config objects https://review.openstack.org/564563 | 18:58 |
| openstackgerrit | Merged openstack-infra/zuul master: Cache configuration objects in addition to YAML dicts https://review.openstack.org/564061 | 18:58 |
| openstackgerrit | Merged openstack-infra/zuul master: Stop deep-copying job variables https://review.openstack.org/564564 | 18:58 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove source_context argument to Pipeline https://review.openstack.org/564642 | 18:58 |
| openstackgerrit | Merged openstack-infra/zuul master: Simplify UnparsedConfig.copy https://review.openstack.org/564647 | 18:58 |
| *** salv-orlando has joined #openstack-infra | 18:59 | |
| corvus | infra-root: i'd like to eat lunch. then i'd like to restart zuul and see if it blows up. | 19:00 |
| mordred | corvus: are you sure that's the right order? | 19:00 |
| corvus | mordred: i could not be more certain! | 19:01 |
| fungi | might want to wait for 564628 to merge too, i just now (re)approved it | 19:01 |
| corvus | i'm pretty sure not eating lunch is how we ended up with memory problems in the first place | 19:01 |
| mordred | good | 19:01 |
| corvus | fungi: oh that'll be nice yeah. | 19:01 |
| *** spsurya has quit IRC | 19:01 | |
| corvus | mordred: not sure if you saw 564628 -- you might want to retro-review it | 19:02 |
| mordred | fungi: I've also been playing with the idea of a 'build' profile 'these are the things you need if you want ot build wheels for zuul and all of its depends' | 19:02 |
| mordred | corvus: lgtm | 19:03 |
| *** e0ne has joined #openstack-infra | 19:03 | |
| fungi | corvus: i got to playing around with package_resources last night because i wanted one of my personal projects to have a show environment command where it would report not only its own version and git commit info, but also those of all its declared install_requires as well as transitively installed dependencies (for improved upstream error reporting in the future). as a result i can probably reuse | 19:03 |
| fungi | some of that to do the pbrless version module for zuul when i remember to poke at it | 19:03 |
| fungi | er, pkg_resources | 19:04 |
| *** caphrim007_ has quit IRC | 19:05 | |
| *** caphrim007 has joined #openstack-infra | 19:05 | |
| *** e0ne has quit IRC | 19:06 | |
| corvus | fungi: nice! that could be useful | 19:07 |
| pabelanger | and back | 19:08 |
| pabelanger | could I get a few reviews on https://review.openstack.org/564771/ and https://review.openstack.org/564775/ thanks | 19:10 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-askbot master: Fix scope of vhost template variables https://review.openstack.org/564866 | 19:12 |
| *** imacdonn has quit IRC | 19:14 | |
| *** imacdonn has joined #openstack-infra | 19:15 | |
| openstackgerrit | Merged openstack-infra/zuul master: Report git sha in status page version https://review.openstack.org/564628 | 19:15 |
| *** dpawlik has joined #openstack-infra | 19:17 | |
| *** myoung|emailplz is now known as myoung | 19:18 | |
| *** evrardjp has quit IRC | 19:18 | |
| *** evrardjp has joined #openstack-infra | 19:21 | |
| *** dpawlik has quit IRC | 19:22 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-storyboard master: Fix scope of vhost template variables https://review.openstack.org/564868 | 19:25 |
| smcginnis | I just noticed I have some of the patches from around the queens release still out there. | 19:33 |
| smcginnis | If someone could take a look, that would be appreciated. | 19:34 |
| smcginnis | https://review.openstack.org/#/q/project:openstack-infra/devstack-gate+owner:%22Sean+McGinnis+%253Csean.mcginnis%2540gmail.com%253E%22+status:open | 19:34 |
| fungi | any other config-core willing to approve https://review.openstack.org/564047 so i can initiate the barbican migration from launchpad to storyboard? | 19:35 |
| *** slaweq has quit IRC | 19:35 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add regex support to project stanzas https://review.openstack.org/535713 | 19:35 |
| clarkb | fungi: looking | 19:37 |
| clarkb | fungi: and done | 19:37 |
| fungi | probably fine to waive the only-osf-staff-reviewers approving an osf-staff-authored change in this particular case | 19:37 |
| fungi | as i can't imagine that one being even remotely controversial | 19:38 |
| fungi | thanks! | 19:38 |
| clarkb | ya its tiny | 19:38 |
| *** yamamoto has joined #openstack-infra | 19:42 | |
| *** lpetrut has quit IRC | 19:44 | |
| *** lpetrut has joined #openstack-infra | 19:44 | |
| pabelanger | clarkb: any objection to start building fedora-28 images, https://review.openstack.org/564775/ currently working with DIB | 19:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Add Barbican to StoryBoard https://review.openstack.org/564047 | 19:46 |
| *** yamamoto has quit IRC | 19:47 | |
| openstackgerrit | Fatih Degirmenci proposed openstack-infra/nodepool master: Add nodepool service file for CentOS7 https://review.openstack.org/564872 | 19:49 |
| *** kjackal has joined #openstack-infra | 19:49 | |
| mordred | fungi: I didn't even realize that was a thing | 19:51 |
| *** r-daneel_ has joined #openstack-infra | 19:52 | |
| *** r-daneel has quit IRC | 19:53 | |
| *** r-daneel_ is now known as r-daneel | 19:53 | |
| *** Douhet has quit IRC | 19:55 | |
| *** Douhet has joined #openstack-infra | 19:55 | |
| *** claudiub has joined #openstack-infra | 19:59 | |
| corvus | infra-root: i'd like to restart zuul now. | 20:02 |
| clarkb | corvus: ok | 20:02 |
| mordred | corvus: I believe in you | 20:05 |
| *** CrayZee has quit IRC | 20:06 | |
| corvus | release has been notified and their queues are quiet (also, friday) | 20:06 |
| fungi | thanks corvus! | 20:07 |
| corvus | (i'm restarting everything for good measure, even though the main focus is the scheduler) | 20:07 |
| fungi | mordred: more an informal thing where those of us who are employed by osf are doing our best to not squander the community's good will | 20:08 |
| corvus | you have 5 good will points remaining | 20:08 |
| corvus | (i really hope everyone read that in the voice of korben dallas's cab) | 20:09 |
| clarkb | while corvus is doing that I've been nerd sniped by bindep, anyone have access to /etc/os-release on a redhat enterprise workstation install? | 20:09 |
| clarkb | if I could have the contents of that file it would be helpful | 20:09 |
| corvus | the scheduler is back up, after only about 30s of downtime | 20:10 |
| fungi | wow, quick, right? | 20:11 |
| fungi | if only my pizza would finish baking in that amount of time | 20:11 |
| *** caphrim007_ has joined #openstack-infra | 20:11 | |
| corvus | fungi: well, it hasn't started doing anything yet, this is just the new idea of restart the scheduler before the executors have finished stopping, so that we don't miss gerrit events | 20:11 |
| corvus | it's waiting for mergers/executors to come online | 20:11 |
| corvus | looks like i can start the mergers now, so i'll do that. still waiting on executors to stop | 20:12 |
| corvus | 4/6 stopped | 20:13 |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Read description file as utf-8 https://review.openstack.org/564874 | 20:13 |
| corvus | 9/10 | 20:14 |
| openstackgerrit | Clark Boylan proposed openstack-infra/bindep master: Use the distro package to determine distro https://review.openstack.org/564650 | 20:14 |
| corvus | starting executors | 20:15 |
| clarkb | fungi: pabelanger notmyname ^ it mostly works now according to the tests. Just need os-release data for a few more distros and to remove what I think is a test we no longer need. | 20:15 |
| *** caphrim007 has quit IRC | 20:15 | |
| notmyname | clarkb: nice! | 20:15 |
| clarkb | notmyname: to complicate matters most rpm based distros appear to use a different name in lsb_release and os-release you know just to be confusing | 20:16 |
| clarkb | notmyname: I've done my best to add the other when one is detected | 20:16 |
| corvus | infra-root: apparently we have a hole in our tests: http://paste.openstack.org/show/720047/ | 20:17 |
| corvus | that is certainly going to cause problems for us | 20:17 |
| fungi | oh, ick | 20:17 |
| corvus | i can manually revert the scheduler and restart it | 20:17 |
| corvus | that might be the best thing to do, then i'll triage the bug and figure out if we should roll forward with a fix, or revert on master | 20:18 |
| corvus | sound like a plan? | 20:18 |
| *** lpetrut has quit IRC | 20:18 | |
| clarkb | yup | 20:18 |
| fungi | sounds as good as we can make it for now | 20:18 |
| corvus | i'll just revert to 3.0.2 :) | 20:18 |
| fungi | once that's done, i'm curious to discover what's not serializable about that object | 20:19 |
| clarkb | fungi: I expect NOSE_HTML_OUT_FILE wants a string not a python value | 20:19 |
| corvus | it may be that mappingproxies don't implement a needed interface | 20:20 |
| corvus | on the plus side: Zuul version: 3.0.3.dev21 95c89f4 | 20:20 |
| corvus | restarting scheduler now | 20:21 |
| corvus | i saved queues again; i'll restore both copies. so there shouldn't be much impact. | 20:21 |
| *** bobh has quit IRC | 20:22 | |
| clarkb | fungi: corvus rereading the error message its not the OUT_FILE that is a string, its mappingproxy itself | 20:24 |
| clarkb | which is unfortunate | 20:24 |
| clarkb | I'm going to test that now | 20:25 |
| clarkb | ya that appears to be it | 20:27 |
| corvus | so: you just can't json serialize a mappingproxy? | 20:28 |
| *** bobh has joined #openstack-infra | 20:28 | |
| clarkb | at least not without coercing the type first? This feels liek a half baked implmentation of that type | 20:28 |
| mordred | clarkb, corvus everything about how json serialization works is bong | 20:29 |
| corvus | yeah, it's not *quite* all i hoped it would be | 20:29 |
| mordred | whatever you do, don't go look at the actual code in the stdlib | 20:29 |
| corvus | mordred: i'm pretty sure mappingproxy is bong too, so.... double bong or something. | 20:29 |
| mordred | because it sure does do 'if isinstance(dict)' rather than looking to see if a given thing implements the right interface | 20:29 |
| clarkb | its literally just a pointer to a dict | 20:30 |
| clarkb | it should be serializable | 20:30 |
| clarkb | but it isn't | 20:30 |
| mordred | so if your thing does not actually subclass dict, jsonencoder gets confused | 20:30 |
| mordred | that said - we can write a custom json encoder | 20:30 |
| *** auristor has quit IRC | 20:33 | |
| corvus | okay, the hole in our tests is that we don't seem to have any deep dictionaries under vars | 20:36 |
| *** r-daneel_ has joined #openstack-infra | 20:36 | |
| corvus | so the first level of the vars dict gets converted to a dict, but not deeper ones | 20:36 |
| corvus | i want to say that part of the Job.deepUpdate method actually has to convert all these back to dicts anyway, so let me poke at that first before we head down the road of serializing mappingproxies | 20:37 |
| *** r-daneel has quit IRC | 20:37 | |
| *** r-daneel_ is now known as r-daneel | 20:37 | |
| mordred | corvus: that should be an easy enough testing hole to fix too | 20:39 |
| corvus | yep, i've reproduced locally | 20:40 |
| *** ssbarnea_ has quit IRC | 20:43 | |
| *** wolverineav has quit IRC | 20:43 | |
| *** yamamoto has joined #openstack-infra | 20:43 | |
| *** wolverineav has joined #openstack-infra | 20:44 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-drupal master: Fix scope of vhost template variables https://review.openstack.org/564880 | 20:44 |
| corvus | okay, the deepUpdate method only needs to turn mappingproxies into dicts if an item appears in both things (eg, a variable in a parent is being overriden in a child job) | 20:45 |
| corvus | it's easy enough to make it do that all the time, if we just want to fix it that way | 20:46 |
| corvus | or we could keep those as mappingproxies and then try to figure out how to serialize them | 20:46 |
| corvus | i'm not too worried about performance here; this is inside of job freezing, so there's a bunch of copying going on anyway, and this only happens when we're really going to run a job. | 20:47 |
| corvus | so i'm sort of inclined to just fix it in deepUpdate | 20:47 |
| corvus | (ie, always turn it back into a dict) | 20:47 |
| corvus | also -- there's a second bug here, we sure do call isinstance(foo, dict) :) | 20:48 |
| clarkb | corvus: you're basically saying let the scheduler config side by forzen but when it goes to the executor we copy it and unfreeze it and then serialize that? | 20:48 |
| clarkb | s/side by frozen/side stay frozen/ | 20:48 |
| *** wolverineav has quit IRC | 20:48 | |
| clarkb | that seems reasonable. The internal representation doesn't need to be directly serializable | 20:48 |
| corvus | clarkb: this is actually still inside the scheduler, even before we send it to the executor client. this is the part where we collect all the variants, and then construct a new synthetic job by combining all of them. | 20:49 |
| clarkb | ah | 20:49 |
| corvus | that's why i say there's a bunch of copying happening anyway | 20:49 |
| clarkb | and variant application implies modification so frozeness isn't required | 20:49 |
| *** yamamoto has quit IRC | 20:49 | |
| corvus | yep | 20:50 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Fix scope of vhost template variables https://review.openstack.org/564882 | 20:50 |
| clarkb | I'd be fine with that approach, kind of makes it explicit if we do the wrong thing too | 20:51 |
| clarkb | as it did here | 20:51 |
| corvus | how should i duck type these... getattr(foo, 'items') or getattr('__getitem__')? | 20:51 |
| corvus | or should i just "if isinstance(dict) or isinstance(mappingproxy)"? | 20:51 |
| clarkb | isinstance is nice and explicit | 20:52 |
| fungi | though may result in an ever-growing list of dict-like types down the road, or do we really only expect it to be one of those two? | 20:53 |
| mordred | clarkb: you can do isinstance with a list | 20:53 |
| mordred | corvus: I mean | 20:53 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-etherpad_lite master: Fix scope of vhost template variables https://review.openstack.org/564884 | 20:54 |
| mordred | if isinstance(foo, (dict, mappingproxy)): | 20:54 |
| corvus | mordred: did i just learn something new about python 1.5.2? | 20:54 |
| fungi | neat and kindof sqlish | 20:54 |
| mordred | in fact, you can make a variable somewhere: DICT_TYPES = (dict, mappingproxy) and then do if isinstance(foo, DICT_TYPES) ... which is how six does its STRING_TYPES thing | 20:55 |
| mordred | corvus: :) | 20:55 |
| mordred | corvus: there's always something new to learn! | 20:55 |
| mordred | corvus: you know ... if we really wanted to be l33t, instead of adopting python 3.5, we could have adopted 1.5 | 20:55 |
| corvus | mordred: actually, i think it's python2. the 1.5.2 docs don't mention tuples | 20:56 |
| mordred | corvus: I'm pretty sure I could not actually write a python 1.5 program any more | 20:56 |
| mordred | darn. 2.1.3 is as far back as pyenv will let me install | 20:57 |
| fungi | pyenv isn't l33t enough for python 1.x | 20:58 |
| dmsimard | I haven't seen a notice about a restart or needing to recheck changes but I'm catching up with the backlog. It seems some of the changes I've +W'd aren't anywhere in the queue ? | 21:01 |
| *** trown is now known as trown|outtypewww | 21:01 | |
| dmsimard | ok so there was a restart, maybe my things were lost somewhere in there | 21:02 |
| fungi | dmsimard: there were a couple of very short (30-60s?) scheduler restarts | 21:02 |
| *** jtomasek has quit IRC | 21:02 | |
| mordred | dmsimard got lucky | 21:03 |
| dmsimard | "lucky" :P | 21:03 |
| fungi | he is a lucky guy | 21:03 |
| dmsimard | I've +W around 16:20ish, anyway, I've re +Wd | 21:03 |
| dmsimard | We'll finally have a new release of ARA next week, cutting a pre-release today. Then we can merge the performance improvements for logs.o.o with https://review.openstack.org/#/c/558688/ | 21:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Coerce MappingProxyTypes in job vars to dicts https://review.openstack.org/564886 | 21:05 |
| corvus | clarkb, mordred, fungi: ^ | 21:06 |
| corvus | i also scanned for other instances of isinstance(.*dict), and i don't think there are any more which could be problematic | 21:06 |
| *** jcoufal has quit IRC | 21:06 | |
| *** harlowja has joined #openstack-infra | 21:07 | |
| *** felipemonteiro__ has joined #openstack-infra | 21:10 | |
| *** felipemonteiro_ has quit IRC | 21:10 | |
| *** rfolco is now known as rfolco|off | 21:11 | |
| mordred | corvus: cool | 21:11 |
| clarkb | corvus: should I go ahead and approve it? | 21:13 |
| corvus | clarkb: yeah, maybe we can swing one more restart attempt. also, master's broken anyway :) | 21:13 |
| clarkb | approved | 21:13 |
| *** ldnunes has quit IRC | 21:13 | |
| clarkb | I'm gonna find lunch while we wait on that | 21:13 |
| corvus | clarkb: it passes pep8 and tests locally | 21:14 |
| *** ykarel|away has quit IRC | 21:15 | |
| corvus | do we have bionic testing for system-config? | 21:16 |
| clarkb | corvus: I think we do | 21:17 |
| *** felipemonteiro_ has joined #openstack-infra | 21:17 | |
| clarkb | checking really quiickly | 21:17 |
| clarkb | nope i'm wrong | 21:17 |
| corvus | okay, i guess i'll do that | 21:18 |
| corvus | pabelanger: unless you have bionic system-config patches? | 21:18 |
| *** dpawlik has joined #openstack-infra | 21:18 | |
| *** felipemonteiro__ has quit IRC | 21:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add infra apply-puppet-bionic job https://review.openstack.org/564891 | 21:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Test eavesdrop on bionic https://review.openstack.org/564892 | 21:23 |
| pabelanger | corvus: no, haven't added them yet | 21:23 |
| corvus | pabelanger: how's that look ^ ? | 21:23 |
| *** dteselkin_ has quit IRC | 21:23 | |
| *** igormarnat has quit IRC | 21:23 | |
| *** dpawlik has quit IRC | 21:23 | |
| *** e0ne has joined #openstack-infra | 21:24 | |
| pabelanger | corvus: hmm, bionic seems to be puppet5 by default now, https://packages.ubuntu.com/bionic/puppet wonder how puppetlabs repo will handle that | 21:24 |
| *** rakhmerov has quit IRC | 21:25 | |
| corvus | don't we use the puppetlabs repo explicitly so we don't have to care what's in the distros? | 21:25 |
| clarkb | sort of, we did it on older distros to get newer puppet but as puppet moved well ahead and stopped publishing older puppet for newer distros we used distro puppet | 21:26 |
| pabelanger | yah, I don't think puppetlabs is building puppet3 any more | 21:27 |
| corvus | i'm on the fence, but this may be the thing that convinces me that we should switch to ansible | 21:27 |
| *** igormarnat has joined #openstack-infra | 21:27 | |
| clarkb | corvus: the difference being we can easily install ansible from source? | 21:27 |
| corvus | clarkb: no, that we don't have to install ansible on the remote system | 21:28 |
| pabelanger | I don't see bionic packages yet: https://apt.puppetlabs.com/pool/ | 21:28 |
| clarkb | fwiw the switch from /etc/puppet to /etc/puppetlabs to line up with the company name. Then changing the company name to Puppet is what I'm hung up on | 21:28 |
| *** dteselkin has joined #openstack-infra | 21:28 | |
| dmsimard | clarkb: wow | 21:28 |
| clarkb | apparently red hat patches it to still be /etc/puppet | 21:29 |
| clarkb | to make things even worse | 21:29 |
| pabelanger | -- Matthias Klose <doko@ubuntu.com> Thu, 29 Mar 2018 14:21:28 +0800 | 21:29 |
| pabelanger | is when puppet 5 was added to bionic | 21:29 |
| pabelanger | when I last checked, before that date, it was still puppet 4 | 21:29 |
| corvus | surprise! | 21:29 |
| pabelanger | indeed | 21:30 |
| corvus | okay, well, i guess if i want to make progress on the irc stuff, i just have to install limnoria in a venv | 21:30 |
| pabelanger | that makes it much harder to switch control plane to bionic | 21:30 |
| *** rakhmerov has joined #openstack-infra | 21:30 | |
| dmsimard | a venv or a docker container ? /me ducks | 21:30 |
| corvus | dmsimard: go for it | 21:30 |
| corvus | dmsimard: your assistance on this project is very welcome. | 21:31 |
| mordred | dmsimard: I actually owe a spec on starting to use containers for things | 21:31 |
| *** armaan has joined #openstack-infra | 21:31 | |
| corvus | dmsimard: perhaps you could push up a change to implement that? | 21:31 |
| mordred | dmsimard: maybe I'll get that written by the next meeting | 21:31 |
| cmurphy | the package name from the puppetlabs repo is puppet-agent not puppet so it won't conflict with the distro package named puppet | 21:32 |
| clarkb | pabelanger: odd that bionic shows up in dists but not pool | 21:33 |
| clarkb | pabelanger: just not populated with content yet I guess | 21:33 |
| pabelanger | clarkb: yah, same thought | 21:34 |
| corvus | pabelanger, clarkb, cmurphy: so the only puppet option is puppet5 in the distro, right? | 21:34 |
| clarkb | corvus: as of right now from what I can tell correct | 21:34 |
| dmsimard | corvus: I was half friday trolling but really though, between running xenial-based container images on bionic systems and switching from puppet to ansible... | 21:35 |
| clarkb | dmsimard: I think you've got that backwards, it would be bionic container on xenial | 21:35 |
| corvus | pabelanger, clarkb, cmurphy: is there any chance that might work, like, is it worth trying that out in a job real quick | 21:35 |
| pabelanger | maybe setup PPA a rebuild puppet (4.10.4-2ubuntu3) bionic; urgency=medium, that was previous version | 21:35 |
| pabelanger | then figure out puppet 5 issue | 21:35 |
| clarkb | corvus: cmurphy has a set of puppet4 chagnes up, most of them are largely mechnical | 21:35 |
| dmsimard | clarkb: yeah | 21:35 |
| clarkb | corvus: I'm told the delta between puppet 4 and 5 is much smaller than the delta between 3 and 5 | 21:36 |
| clarkb | corvus: so we mgiht add a puppet5 job to go along cmurphy's port work? | 21:36 |
| *** felipemonteiro_ has quit IRC | 21:36 | |
| pabelanger | dmsimard: if voting, I'd ansible before containers for me | 21:36 |
| corvus | dmsimard: yeah, i'm just really tired of having this conversation in a trolling way. we've talked about it enough and we're willing to evaluate something like that on the merits. i'd really rather not see any more about it until there's a spec up to talk about. | 21:36 |
| *** kjackal has quit IRC | 21:36 | |
| pabelanger | I'll rebase ansible spec this evening | 21:37 |
| mordred | and I will actually write the other one - probably this weekend | 21:37 |
| *** salv-orlando has quit IRC | 21:37 | |
| clarkb | corvus: https://review.openstack.org/#/q/topic:puppet-4+status:open that set of changes give syou an idea of what needs updating | 21:37 |
| *** salv-orlando has joined #openstack-infra | 21:38 | |
| corvus | mostly our old friend scope.lookupvar :) | 21:38 |
| clarkb | corvus: ya it is largely mechnical at this point with the ansible role update | 21:38 |
| clarkb | corvus: assuming the ansible role will actually work with puppet4 fully | 21:38 |
| clarkb | and then assuming puppet5 didn't create similar problems (I'm told it didn't but yet to be tested) | 21:38 |
| cmurphy | i haven't even been thinking about puppet 5 yet | 21:39 |
| dmsimard | corvus: I've discussed ansibilization in a serious setting more than once with pabelanger and the software factory folks, we already have a lot of Ansible roles for openstack-infra things but it might not be generic enough. Between https://github.com/softwarefactory-project/sf-config/tree/master/ansible/roles, windmill, https://github.com/Logan2211/ci-stack and others, I genuinely wonder how much | 21:39 |
| dmsimard | work it would be. | 21:39 |
| corvus | dmsimard: probably the most constructive thing would be to help make pabelanger's ansible spec and mordred's container spec as thorough as possible so we can make a good choice | 21:40 |
| clarkb | corvus: ++ | 21:40 |
| corvus | clarkb, cmurphy, pabelanger: what does the puppet_version: 4 job variable do? | 21:41 |
| clarkb | corvus: it tells install_puppet.sh in system config to install puppet 4 instead of puppet 3 | 21:41 |
| *** e0ne has quit IRC | 21:41 | |
| clarkb | corvus: you'll need to start there for using puppet 5 I bet (largely a noop on bionic though since its the distro package) | 21:41 |
| dmsimard | corvus: sure, I haven't been part of the discussions that would lead to mordred's spec so I'm curious to see how that will pan out but pabelanger's https://review.openstack.org/#/c/469983/ is bookmarked | 21:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add new project pbrx https://review.openstack.org/564896 | 21:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add pbrx to zuul https://review.openstack.org/564897 | 21:42 |
| *** salv-orlando has quit IRC | 21:42 | |
| clarkb | dmsimard: one thing to keep in mind is that nodepool and zuul are the easy things. Its the everything else long tail that makes things difficult | 21:42 |
| mordred | dmsimard: I'll ping you as soon asI've got a draft | 21:42 |
| corvus | thinking about patterns makes that tractable | 21:42 |
| *** salv-orlando has joined #openstack-infra | 21:42 | |
| corvus | like "a server that runs a python service fronted by apache" | 21:43 |
| dmsimard | clarkb: yeah that's why I mentioned the software factory roles. We have logserver (with os_loganalyze), mqtt, ELK, grafana, etc | 21:43 |
| mordred | I actually started being the 'let's use us some containers' because of gerrit :) | 21:43 |
| fungi | etherpad (nodejs), gerrit (java), mediawiki (php) and so on | 21:43 |
| *** boden has quit IRC | 21:43 | |
| pabelanger | even if we agreed on ansible, i don't think we'd want to just write thing as fast a possible. we'd likely want to go system by system and establish a method to how our roles work. Oh and testing for sure... | 21:44 |
| fungi | mordred: gerrit's already in a container | 21:44 |
| fungi | well, a java virtual machine anyway | 21:44 |
| mordred | fungi: not enough of one | 21:44 |
| fungi | yeah, the jvm fails to contain some bits of gerrit | 21:45 |
| mordred | fungi: it's the fact that java itself is tied to distro release that eventually made me grok containers as a replacement for rpm/dpkg in these contexts | 21:45 |
| pabelanger | dmsimard: how open is SF in porting them to ubuntu or even multi distro support might be a good question to find out | 21:45 |
| pabelanger | that's been a tricky thing for our puppet modules today | 21:45 |
| *** yamamoto has joined #openstack-infra | 21:46 | |
| mordred | pabelanger: this is the other reason I became a fan of the idea of using containers as packaging format :) | 21:46 |
| fungi | this way we get to port everything to alpine | 21:47 |
| clarkb | fungi: good thing I've got a patch up to bindep to make that possible :) | 21:47 |
| dmsimard | pabelanger: I could be wrong but I don't suppose SF would object to supporting more than CentOS as a deployment target if someone shows up -- but a lot of the SF works is about packaging, too. So yum install zuul, etc. | 21:47 |
| pabelanger | mordred: yah, until we bikeshed on the container version :) But agree | 21:48 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 21:48 |
| dmsimard | just like OSA didn't support CentOS until a cycle or two ago | 21:48 |
| mordred | pabelanger: my bikeshed color is "use the python:slim based container image" | 21:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Support puppet 5 on ubuntu https://review.openstack.org/564898 | 21:49 |
| pabelanger | dmsimard: right, supporting both git / package, dnf / apt, all things to discuss | 21:49 |
| pabelanger | either way, let me rebase spec tonight and we can discuss more | 21:49 |
| dmsimard | pabelanger: ++ | 21:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Support puppet 5 on ubuntu https://review.openstack.org/564898 | 21:52 |
| *** yamamoto has quit IRC | 21:52 | |
| corvus | clarkb, cmurphy, pabelanger: ^ does that look like it stands a chance? :) | 21:52 |
| pabelanger | think so | 21:53 |
| clarkb | corvus: and maybe depeonds on https://review.openstack.org/#/c/557515/ to get that set of fixes too | 21:54 |
| clarkb | but otherwise ya | 21:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add infra apply-puppet-bionic job https://review.openstack.org/564891 | 21:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Support puppet 5 on ubuntu https://review.openstack.org/564898 | 21:56 |
| cmurphy | corvus: seems reasonable | 21:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add infra apply-puppet-bionic job https://review.openstack.org/564891 | 21:57 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 21:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add new project pbrx https://review.openstack.org/564896 | 22:02 |
| *** rlandy has quit IRC | 22:04 | |
| openstackgerrit | Merged openstack-infra/zuul master: Coerce MappingProxyTypes in job vars to dicts https://review.openstack.org/564886 | 22:04 |
| dmsimard | mordred: for https://review.openstack.org/#/c/564896/, do you want gerritbot ? | 22:04 |
| *** edmondsw has quit IRC | 22:06 | |
| mordred | dmsimard: dunno- I'm not sure where, if anywhere, that should go yet - so I left it off for now | 22:06 |
| dmsimard | k | 22:06 |
| mordred | dmsimard: (it's still a young enough thing I'm not sure how useful it'll become to people and don't want to make assumptions) | 22:06 |
| *** dave-mcc_ has quit IRC | 22:07 | |
| *** bobh has quit IRC | 22:13 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 22:18 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Fix nodepool-launcher systemd file https://review.openstack.org/564901 | 22:18 |
| *** agopi is now known as agopi|out | 22:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add new project pbrx https://review.openstack.org/564896 | 22:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Fix zuul systemd files https://review.openstack.org/564903 | 22:28 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-meetbot master: Convert httpd::vhost to apache::vhost::custom https://review.openstack.org/564904 | 22:28 |
| *** camunoz has quit IRC | 22:29 | |
| *** Goneri has quit IRC | 22:35 | |
| *** jroll has quit IRC | 22:41 | |
| *** salv-orlando has quit IRC | 22:41 | |
| *** salv-orlando has joined #openstack-infra | 22:42 | |
| corvus | infra-root: hows about i restart zuul again? | 22:43 |
| clarkb | corvus: I am finishing up late lunch can assist if it helps | 22:43 |
| corvus | clarkb: slash early dinner :) | 22:43 |
| *** agopi|out has quit IRC | 22:44 | |
| fungi | sounds good, i'm still fairly around though beginning to wind down | 22:44 |
| corvus | i'll get started now | 22:44 |
| corvus | i'll just restart the scheduler again | 22:44 |
| *** caphrim007_ has quit IRC | 22:45 | |
| clarkb | corvus: I actually decided it was snack time so I can make ramen in another hour or so for proper early dinner :) | 22:45 |
| *** caphrim007 has joined #openstack-infra | 22:45 | |
| corvus | maybe i'll reward myself with a snack if zuul actually works this time | 22:46 |
| *** salv-orlando has quit IRC | 22:46 | |
| clarkb | sun noodle makes fresh ramen noodle packs that I tend to make a quick kombu broth for then add in some veggies. quite good | 22:46 |
| corvus | clarkb: what's 'make a quick kombu broth' entail? | 22:47 |
| clarkb | corvus: seaweed in water for a soak then boil it for a short time. I thne use the flavor pack from the packaging with that instead of plain hot water | 22:47 |
| *** yamamoto has joined #openstack-infra | 22:47 | |
| clarkb | makes it taste a bit like the ocean | 22:48 |
| corvus | neat | 22:48 |
| *** eernst has quit IRC | 22:48 | |
| corvus | scheduler is up | 22:48 |
| clarkb | I should pick a weekend where I make a proper broth from scratch but enotime | 22:49 |
| fungi | yeah, presumably the same method you'd use to make a kombu dashi | 22:49 |
| clarkb | fungi: ya thats basically it | 22:50 |
| fungi | if you can get your hands on dried shiitake, soak those in cold water until they soften and add the water you drain off into the broth | 22:50 |
| corvus | Zuul version: 3.0.3.dev22 e82db12 | 22:50 |
| fungi | that looks promising | 22:50 |
| clarkb | sha1 checks out | 22:50 |
| *** dhill_ has quit IRC | 22:51 | |
| fungi | yeah, matches what's checked out on zuul01 | 22:51 |
| corvus | no unexpected tracebacks since the restart (only one for reporting to a closed change in gerrit) | 22:51 |
| clarkb | http://zuul.openstack.org/stream.html?uuid=d0110bf8562a43f6a1ed574b9920f14b&logfile=console.log that job is running | 22:51 |
| *** lbragstad has quit IRC | 22:51 | |
| fungi | clarkb: i have a great book on dashi methods if you want a recommendation for your shelf. i've found it excellent | 22:52 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all is the graph to watch | 22:52 |
| clarkb | fungi: yes, please | 22:53 |
| *** yamamoto has quit IRC | 22:53 | |
| *** caphrim007_ has joined #openstack-infra | 22:53 | |
| corvus | re-enqueue is finished | 22:53 |
| clarkb | corvus: I kinda want to push up a big stack of config changes now :) | 22:54 |
| corvus | we're at 437880 rss | 22:54 |
| corvus | clarkb: i have 13 old ones sitting around, i was thinking of rechecking those | 22:54 |
| clarkb | ya its doing well. I want to say when I did previous restarts it quickly climbed to 2gb and beyond | 22:54 |
| *** Nil_ has quit IRC | 22:54 | |
| clarkb | corvus: ++ | 22:55 |
| fungi | clarkb: "dashi and umami: the heart of japanese cuisine" (eat-japan/cross media ltd) | 22:55 |
| fungi | clarkb: it covers a lot about flavor theory as well | 22:55 |
| fungi | and has excellent photography | 22:55 |
| clarkb | fungi: amazon tells me I'm not getting it for cheap anytime soon | 22:55 |
| clarkb | I'll have to look for it in used book stores | 22:55 |
| fungi | oh, too bad | 22:55 |
| clarkb | $348 is cheapest amazon option | 22:56 |
| *** hongbin_ has quit IRC | 22:56 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-etherpad_lite master: Fix scope of vhost template variables https://review.openstack.org/564884 | 22:56 |
| clarkb | fungi: it must be good :) | 22:56 |
| fungi | cover price on the edition i got was ¥5500 or £25 | 22:56 |
| fungi | hardcover | 22:56 |
| *** caphrim007 has quit IRC | 22:57 | |
| fungi | so, like, 10% of what it seems to be going for these days | 22:57 |
| fungi | that's nuts | 22:57 |
| fungi | it _is_ a great reference though | 22:58 |
| corvus | okay i rechecked those 13 changes; no significant change in memory usage so far | 22:58 |
| corvus | though they aren't a stack, so it's a bit hard to tell if they all made it in or got bounced out for merge conflicts | 22:58 |
| corvus | maybe i'll make some synthetic ones real quick | 22:58 |
| clarkb | corvus: there was a slight bump in total system usage according to free -m | 22:58 |
| clarkb | but from 880MB to 905MB | 22:58 |
| clarkb | res currently at 460xyz | 22:59 |
| *** caphrim007_ has quit IRC | 22:59 | |
| *** caphrim007 has joined #openstack-infra | 22:59 | |
| corvus | shall i try for 20 changes? | 22:59 |
| clarkb | I'm game | 23:00 |
| fungi | yeah, utilization's a trifle of what we see it get up to normally anyway | 23:00 |
| fungi | seems safe | 23:00 |
| corvus | https://review.openstack.org/#/q/topic:zuul-test | 23:01 |
| clarkb | it appeas to be spinning the cpu to process them | 23:02 |
| clarkb | but memory hasn't skyrocketed yet | 23:02 |
| clarkb | (slightly up) | 23:02 |
| corvus | creating the layouts is taking about 4 seconds each | 23:02 |
| clarkb | corvus: is that good or bad relative to the old system? | 23:03 |
| corvus | i'll have to look up the old times, but it's not worrying at least :) | 23:03 |
| *** caphrim007 has quit IRC | 23:03 | |
| *** r-daneel has quit IRC | 23:04 | |
| corvus | clarkb: i think they're all loaded now | 23:04 |
| clarkb | corvus: there was a jump up to 525MB or so res | 23:05 |
| clarkb | which is still quite reasonable compared to before I think | 23:05 |
| clarkb | definitely seems less hungry than in the past | 23:05 |
| corvus | looks like building a layout was about 22 seconds before | 23:06 |
| *** fried_rice is now known as efried | 23:06 | |
| clarkb | wow | 23:06 |
| fungi | yikes | 23:06 |
| corvus | so that's a nice improvement; should mean for less "dead" time | 23:06 |
| clarkb | ya I guess even if the memory wins are massive the load time wins are good | 23:07 |
| fungi | what was the main line of work which led to that improvement, do you think? | 23:07 |
| corvus | there's much less yaml parsing and schema validtion now | 23:07 |
| *** eernst has joined #openstack-infra | 23:07 | |
| pabelanger | corvus: nice work! | 23:07 |
| clarkb | ya that happens upfront and as long as the yaml parsed data structures aren't invalidated in cache they continue to be used | 23:07 |
| corvus | (i mean, there's less *repeated* parsing and validation :) | 23:07 |
| clarkb | and then the actual config layer over the top is what mutates | 23:07 |
| fungi | reducing redundant parsing and validation sounds great to me | 23:07 |
| pabelanger | now I'm really curious how small of a scheduler we could use :D | 23:08 |
| pabelanger | guess time will tell | 23:09 |
| corvus | pabelanger: yeah, i'd like to shrink it once we get a feeling for how this behavios | 23:09 |
| pabelanger | exciting | 23:09 |
| clarkb | looking at cacti it appears we typically need up to a week to reach steady state memory consumption | 23:09 |
| corvus | and there's no telling if this introduced a leak or something too, so that's something to watch out for | 23:10 |
| corvus | should i push up another 20 changes? | 23:10 |
| clarkb | sure? it handled the first set just fine. Do we want to do an even bigger set? | 23:10 |
| corvus | (also, i mean while the first 20 are still there | 23:11 |
| clarkb | ah | 23:11 |
| clarkb | sure | 23:11 |
| fungi | yeah, let's see if it holds up | 23:11 |
| corvus | pushed | 23:11 |
| corvus | the jobs on the first change haven't even started yet, so they're all still in the queue | 23:12 |
| corvus | this status page is not going to be nice to our browsers though | 23:13 |
| clarkb | firefox has been handling it pretty well in recent releases | 23:13 |
| clarkb | I can scroll on it and only takes half a second to render the current location | 23:14 |
| clarkb | corvus: while I'm thinking about it https://review.openstack.org/#/c/535713/ is ready for review and touches similar code if you want to reviwe it while its fresh | 23:15 |
| clarkb | corvus: and for puppet 5 looks like install_modules.sh also has a puppet version checker we need to flip for supporting puppet 5 | 23:18 |
| clarkb | otherwise your job seems to be doing the right thing up to that point | 23:18 |
| clarkb | fungi: wow tallboy | 23:18 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 23:19 |
| *** dpawlik has joined #openstack-infra | 23:19 | |
| *** hashar has quit IRC | 23:19 | |
| corvus | clarkb, fungi: it looks like it may be creating more layouts than i'm expecting | 23:20 |
| corvus | i see it creating repeated layouts for the same changes | 23:20 |
| fungi | hrm | 23:23 |
| fungi | as in, it's logging that it's doing so? | 23:23 |
| corvus | yep | 23:23 |
| openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Add CentOS 7 environment setup instructions https://review.openstack.org/564948 | 23:24 |
| *** dpawlik has quit IRC | 23:24 | |
| clarkb | I see it creating layouts but haven't quite figured out how to tie that back to changes in the logs yet | 23:24 |
| clarkb | zuul.Pipeline.openstack.check: Freezing job graph for maybe? | 23:24 |
| corvus | 2018-04-27 23:23:07,085 DEBUG zuul.Pipeline.openstack.check: Preparing dynamic layout for: <Change 0x7f8b33c38be0 564906,1> | 23:25 |
| corvus | it's looking like it's factorial. 906. 906, 907. 906, 907, 908. | 23:25 |
| *** nicolasbock has quit IRC | 23:25 | |
| openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Add CentOS 7 environment setup instructions https://review.openstack.org/564948 | 23:26 |
| clarkb | oh is this like the merge problem? | 23:26 |
| clarkb | when we have a stack of changes it does the whole stack each time | 23:26 |
| clarkb | for each step throug the list of changes | 23:26 |
| corvus | yeah, but it's not supposed to do that for layouts; it *is* supposed to do the factorial merge, but then the results from that should feed into one layout per change | 23:27 |
| *** adrianreza has quit IRC | 23:28 | |
| fungi | have you been able to tell whether it's just dependent changes/pipelines or independent as well? | 23:28 |
| corvus | this is a stack of dependent changes in check | 23:29 |
| fungi | okay, so it's redoing the intermediate layouts sounds like | 23:30 |
| corvus | yeah | 23:30 |
| *** eernst has quit IRC | 23:32 | |
| clarkb | reading the code its definitely going to call getLayout on each item. So I guess its up to the caching mechanism to make sure we don't do it N! times | 23:32 |
| corvus | clarkb: right but that should only happen once per item | 23:33 |
| clarkb | ya | 23:33 |
| corvus | oooooh | 23:34 |
| corvus | are we making dynamic layouts for non-live items | 23:34 |
| clarkb | corvus: ya I think that is it | 23:34 |
| clarkb | for if you have 1<-2<-3 when its processing 3 its loading 1 and 2 which are not live too | 23:34 |
| fungi | that would make sense | 23:35 |
| corvus | okay, so this probably isn't a regression | 23:35 |
| corvus | we've just probably never pushed 40 dependent dynamic layout changes to notice this before :) | 23:35 |
| fungi | so opportunity for more performance enhancement i suppose | 23:35 |
| clarkb | processonitme | 23:35 |
| clarkb | *-processOneItem() is the source of this I think | 23:36 |
| corvus | so... are we going to have 820 layouts in memory? | 23:36 |
| fungi | could explain the sudden memory jumps we've been unable to track down up to now? | 23:37 |
| corvus | fungi: well, usually we don't have very many .zuul.yaml changes depending on each other. | 23:37 |
| corvus | and probably the cpu use before would have made it appear completely dead. | 23:38 |
| fungi | ahh | 23:38 |
| corvus | it looks like it's up to change 34 out of 46; so... we're about half done :) | 23:39 |
| corvus | (er, change number 934 and 946) | 23:39 |
| *** eernst has joined #openstack-infra | 23:39 | |
| corvus | we could let this continue, and then we will have the results from one heck of a stress test, or i could restart it and drop this part of the queue | 23:40 |
| *** edmondsw has joined #openstack-infra | 23:40 | |
| *** jroll has joined #openstack-infra | 23:40 | |
| clarkb | the fact it hasn't broken yet is impressive. Also memory usage has dropped | 23:40 |
| clarkb | I'm still trying to figure out where we recurse into the non active items for certain | 23:41 |
| openstackgerrit | Fatih Degirmenci proposed openstack-infra/nodepool master: Add systemd drop-in file for CentOS 7 https://review.openstack.org/564872 | 23:41 |
| clarkb | aha its for item in queue.queue[:]: | 23:41 |
| clarkb | in processQueue because each change with its parents its own queue in check iirc | 23:41 |
| corvus | 'live' not 'active' i think | 23:42 |
| clarkb | ah right active means you are running jobs right now | 23:42 |
| clarkb | and looking at gate it seems to skip those properly | 23:42 |
| *** salv-orlando has joined #openstack-infra | 23:42 | |
| corvus | yeah, active is the window | 23:42 |
| clarkb | so ya processQueue in the pipeline manager is doing it with the above for loop. I bet we address this by reusing a layout if its been produced | 23:43 |
| clarkb | we check if not item.layout in prepareJobs() so we must use distinct items across different queues. Maybe we keep a layout cache in the pipeline manager though and check that too | 23:44 |
| clarkb | if not item.layout and not layouts.get(item): type deal | 23:44 |
| corvus | i'm not sure that's the solution... i think it's simpler actually, i don't think the non-live items need a layout | 23:44 |
| clarkb | oh right since git will already "layer" things in for us | 23:44 |
| corvus | so if we can just convince them to skip layout generation and only do it on the live one, that should work. the live one will, by virtue of having performed the merge of all the changes ahead... exactly :) | 23:44 |
| *** edmondsw has quit IRC | 23:45 | |
| clarkb | if not item.layout and item.live: getLayout() | 23:45 |
| clarkb | roughly at prepareJobs() in pipeline manager | 23:45 |
| corvus | yeah, we may need a return True in there if not item.live, to allow it to proceed past that | 23:46 |
| corvus | but i *think* that's about it | 23:46 |
| *** salv-orlando has quit IRC | 23:46 | |
| corvus | 8 changes to go | 23:47 |
| openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Add steps to use systemd drop-in for Nodepool on CentOS 7 https://review.openstack.org/564950 | 23:47 |
| *** tosky has quit IRC | 23:47 | |
| *** eernst has quit IRC | 23:48 | |
| *** eernst has joined #openstack-infra | 23:48 | |
| *** yamamoto has joined #openstack-infra | 23:49 | |
| corvus | i think we can probably let it just burn itself out; i think the risk is that if a config change lands before these finish, it'll do the whole thing again | 23:50 |
| clarkb | ok did you want to throw that fix up today? | 23:50 |
| corvus | i'm inclined to leave it for later since it's not a regression | 23:51 |
| corvus | work on a proper test, etc | 23:51 |
| corvus | i can restart zuul now and drop the 40 changes from the queue, or we can leave it and i can check back in a few hours and if they're still there, do it then | 23:52 |
| clarkb | ok | 23:52 |
| clarkb | ya lets wait and see | 23:52 |
| fungi | agreed | 23:52 |
| clarkb | since the data it produces is useful | 23:52 |
| *** jroll has quit IRC | 23:53 | |
| corvus | we are at ~630 layouts in memory, with 1.6G used | 23:54 |
| *** yamamoto has quit IRC | 23:54 | |
| *** esberglu has quit IRC | 23:56 | |
| *** jroll has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!