| *** rlandy has quit IRC | 00:00 | |
| *** bobh has joined #openstack-infra | 00:00 | |
| clarkb | ianw stack lgtm | 00:01 |
|---|---|---|
| *** heyongli has quit IRC | 00:03 | |
| *** heyongli has joined #openstack-infra | 00:03 | |
| *** stakeda has joined #openstack-infra | 00:03 | |
| *** yamamoto has quit IRC | 00:03 | |
| *** yolanda_ has joined #openstack-infra | 00:06 | |
| *** rh-jelabarre has joined #openstack-infra | 00:08 | |
| *** yolanda__ has quit IRC | 00:09 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add min_avail_hdd governor for zuul-executor https://review.openstack.org/577948 | 00:09 |
| *** salv-orlando has joined #openstack-infra | 00:11 | |
| *** dingyichen has joined #openstack-infra | 00:11 | |
| *** jesslampe has joined #openstack-infra | 00:11 | |
| *** heyongli has quit IRC | 00:13 | |
| *** heyongli has joined #openstack-infra | 00:13 | |
| *** salv-orlando has quit IRC | 00:16 | |
| *** heyongli has quit IRC | 00:23 | |
| *** heyongli has joined #openstack-infra | 00:23 | |
| *** dingyichen has quit IRC | 00:30 | |
| *** dingyichen has joined #openstack-infra | 00:30 | |
| mnaser | clarkb: sorry for missing that and thanks for taking care of it | 00:32 |
| *** aeng has quit IRC | 00:33 | |
| clarkb | mnaser: I reviewed the changes too :) the location being on server messed with the proxy I think | 00:33 |
| *** heyongli has quit IRC | 00:33 | |
| *** heyongli has joined #openstack-infra | 00:34 | |
| *** Swami_ has quit IRC | 00:34 | |
| *** Swami has quit IRC | 00:34 | |
| * mnaser really looks forward to be behind this | 00:35 | |
| *** edmondsw has quit IRC | 00:37 | |
| *** dingyichen has quit IRC | 00:37 | |
| *** dingyichen has joined #openstack-infra | 00:37 | |
| *** dingyichen has quit IRC | 00:38 | |
| *** dingyichen has joined #openstack-infra | 00:38 | |
| ianw | clarkb: if you want to take mnaser's as +2 and just merge it, i'm here in case anything explodes :) | 00:41 |
| ianw | i've removed the afs mirrors. waiting for debian-security to puppet out | 00:41 |
| *** heyongli has quit IRC | 00:44 | |
| *** heyongli has joined #openstack-infra | 00:44 | |
| *** abelur_ has joined #openstack-infra | 00:45 | |
| *** aeng has joined #openstack-infra | 00:46 | |
| *** abelur_ has quit IRC | 00:48 | |
| clarkb | ianw done. I'm grilling dinner in yhe backyard | 00:49 |
| ianw | clarkb: have fun! :) | 00:50 |
| *** heyongli has quit IRC | 00:54 | |
| *** heyongli has joined #openstack-infra | 00:54 | |
| *** boden has joined #openstack-infra | 00:58 | |
| *** namnh has joined #openstack-infra | 00:58 | |
| *** yamamoto has joined #openstack-infra | 00:59 | |
| *** boden has quit IRC | 01:03 | |
| *** heyongli has quit IRC | 01:04 | |
| *** heyongli has joined #openstack-infra | 01:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: reprepro debian-security https://review.openstack.org/577950 | 01:05 |
| *** yamamoto has quit IRC | 01:05 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix MariaDB proxy backend details https://review.openstack.org/577945 | 01:07 |
| *** salv-orlando has joined #openstack-infra | 01:11 | |
| *** john_studarus has joined #openstack-infra | 01:13 | |
| *** heyongli has quit IRC | 01:14 | |
| *** heyongli has joined #openstack-infra | 01:15 | |
| *** salv-orlando has quit IRC | 01:15 | |
| *** caphrim007 has joined #openstack-infra | 01:18 | |
| *** hongbin has joined #openstack-infra | 01:21 | |
| *** caphrim007 has quit IRC | 01:23 | |
| *** heyongli has quit IRC | 01:25 | |
| *** heyongli has joined #openstack-infra | 01:25 | |
| *** heyongli has quit IRC | 01:35 | |
| *** heyongli has joined #openstack-infra | 01:35 | |
| *** bobh has quit IRC | 01:37 | |
| *** rosmaita has quit IRC | 01:40 | |
| *** hemna_ has quit IRC | 01:41 | |
| *** gongysh has joined #openstack-infra | 01:42 | |
| openstackgerrit | Merged openstack-infra/system-config master: reprepro debian-security https://review.openstack.org/577950 | 01:44 |
| *** heyongli has quit IRC | 01:45 | |
| *** heyongli has joined #openstack-infra | 01:45 | |
| *** harlowja has quit IRC | 01:46 | |
| *** gongysh has quit IRC | 01:47 | |
| *** mordred has quit IRC | 01:49 | |
| *** yamahata has quit IRC | 01:51 | |
| *** caphrim007 has joined #openstack-infra | 01:53 | |
| *** heyongli has quit IRC | 01:55 | |
| *** jamesmcarthur has joined #openstack-infra | 01:55 | |
| *** heyongli has joined #openstack-infra | 01:56 | |
| *** sthussey has quit IRC | 01:56 | |
| *** jamesmcarthur has quit IRC | 02:00 | |
| *** heyongli has quit IRC | 02:06 | |
| *** heyongli has joined #openstack-infra | 02:06 | |
| *** mordred has joined #openstack-infra | 02:07 | |
| *** heyongli has quit IRC | 02:16 | |
| *** heyongli has joined #openstack-infra | 02:16 | |
| *** iyamahat_ has quit IRC | 02:23 | |
| *** roman_g has quit IRC | 02:26 | |
| *** heyongli has quit IRC | 02:26 | |
| *** heyongli has joined #openstack-infra | 02:26 | |
| *** heyongli has quit IRC | 02:36 | |
| *** heyongli has joined #openstack-infra | 02:37 | |
| *** jesslampe has quit IRC | 02:40 | |
| *** psachin` has joined #openstack-infra | 02:42 | |
| tojuvone | Hi, I'd like to have this merged: https://review.openstack.org/571382 to proceed with project creation. Thanks. | 02:43 |
| *** jesslampe has joined #openstack-infra | 02:44 | |
| *** caphrim007_ has joined #openstack-infra | 02:45 | |
| *** heyongli has quit IRC | 02:47 | |
| *** heyongli has joined #openstack-infra | 02:47 | |
| *** caphrim007 has quit IRC | 02:49 | |
| *** heyongli has quit IRC | 02:57 | |
| *** heyongli has joined #openstack-infra | 02:57 | |
| *** yamamoto has joined #openstack-infra | 03:01 | |
| *** rh-jelabarre has quit IRC | 03:06 | |
| *** heyongli has quit IRC | 03:07 | |
| *** yamamoto has quit IRC | 03:07 | |
| *** heyongli has joined #openstack-infra | 03:07 | |
| *** yolanda__ has joined #openstack-infra | 03:15 | |
| *** heyongli has quit IRC | 03:17 | |
| *** heyongli has joined #openstack-infra | 03:18 | |
| *** yolanda_ has quit IRC | 03:18 | |
| *** jesslampe has quit IRC | 03:19 | |
| *** yolanda__ has quit IRC | 03:23 | |
| *** heyongli has quit IRC | 03:28 | |
| *** heyongli has joined #openstack-infra | 03:28 | |
| *** jesslampe has joined #openstack-infra | 03:29 | |
| *** yolanda__ has joined #openstack-infra | 03:29 | |
| *** yolanda_ has joined #openstack-infra | 03:34 | |
| *** noama has joined #openstack-infra | 03:34 | |
| *** yolanda__ has quit IRC | 03:36 | |
| *** heyongli has quit IRC | 03:38 | |
| *** yamahata has joined #openstack-infra | 03:38 | |
| *** heyongli has joined #openstack-infra | 03:38 | |
| *** annp has joined #openstack-infra | 03:39 | |
| *** sree has joined #openstack-infra | 03:40 | |
| *** sree has quit IRC | 03:40 | |
| *** sree has joined #openstack-infra | 03:40 | |
| *** ramishra has joined #openstack-infra | 03:41 | |
| *** jesslampe has quit IRC | 03:43 | |
| *** yolanda__ has joined #openstack-infra | 03:44 | |
| *** yolanda_ has quit IRC | 03:45 | |
| *** heyongli has quit IRC | 03:48 | |
| *** heyongli has joined #openstack-infra | 03:48 | |
| *** agopi has quit IRC | 03:49 | |
| *** agopi has joined #openstack-infra | 03:49 | |
| *** lpetrut has joined #openstack-infra | 03:50 | |
| *** shu-mutow has joined #openstack-infra | 03:52 | |
| *** hongbin has quit IRC | 03:54 | |
| openstackgerrit | Vu Cong Tuan proposed openstack-infra/project-config master: [Test] Add py36 testenv https://review.openstack.org/577985 | 03:56 |
| *** heyongli has quit IRC | 03:58 | |
| *** heyongli has joined #openstack-infra | 03:59 | |
| *** dhajare has joined #openstack-infra | 04:02 | |
| *** yolanda_ has joined #openstack-infra | 04:03 | |
| *** yamamoto has joined #openstack-infra | 04:04 | |
| *** pahuang has quit IRC | 04:05 | |
| *** yolanda__ has quit IRC | 04:06 | |
| *** yolanda has joined #openstack-infra | 04:08 | |
| *** heyongli has quit IRC | 04:09 | |
| *** heyongli has joined #openstack-infra | 04:09 | |
| *** yamamoto has quit IRC | 04:09 | |
| *** yolanda_ has quit IRC | 04:09 | |
| *** yolanda_ has joined #openstack-infra | 04:10 | |
| *** agopi has quit IRC | 04:13 | |
| *** yolanda has quit IRC | 04:13 | |
| *** hamzy has quit IRC | 04:17 | |
| *** hamzy has joined #openstack-infra | 04:18 | |
| *** heyongli has quit IRC | 04:19 | |
| *** heyongli has joined #openstack-infra | 04:19 | |
| *** Bhujay has quit IRC | 04:20 | |
| *** hamzy has quit IRC | 04:22 | |
| *** pahuang has joined #openstack-infra | 04:22 | |
| *** vivsoni has joined #openstack-infra | 04:26 | |
| *** xarses_ has joined #openstack-infra | 04:27 | |
| *** lpetrut has quit IRC | 04:28 | |
| *** heyongli has quit IRC | 04:29 | |
| *** heyongli has joined #openstack-infra | 04:29 | |
| *** Swimingly has quit IRC | 04:30 | |
| *** dims has quit IRC | 04:30 | |
| *** Swimingly has joined #openstack-infra | 04:31 | |
| *** Swimingly has quit IRC | 04:31 | |
| *** Swimingly has joined #openstack-infra | 04:31 | |
| *** andreww has quit IRC | 04:32 | |
| *** dims has joined #openstack-infra | 04:35 | |
| *** hamzy has joined #openstack-infra | 04:36 | |
| *** diablo_rojo has quit IRC | 04:37 | |
| *** pahuang has quit IRC | 04:38 | |
| *** heyongli has quit IRC | 04:39 | |
| *** heyongli has joined #openstack-infra | 04:40 | |
| *** pcichy has quit IRC | 04:45 | |
| *** heyongli has quit IRC | 04:50 | |
| *** heyongli has joined #openstack-infra | 04:50 | |
| *** pahuang has joined #openstack-infra | 04:52 | |
| *** janki has joined #openstack-infra | 04:54 | |
| *** heyongli has quit IRC | 05:00 | |
| *** heyongli has joined #openstack-infra | 05:00 | |
| *** aeng has quit IRC | 05:03 | |
| *** yamamoto has joined #openstack-infra | 05:05 | |
| *** owalsh_ has joined #openstack-infra | 05:06 | |
| *** ccamacho has quit IRC | 05:06 | |
| *** slaweq has joined #openstack-infra | 05:07 | |
| *** slaweq has quit IRC | 05:07 | |
| *** CrayZee has joined #openstack-infra | 05:07 | |
| *** jchhatbar has joined #openstack-infra | 05:09 | |
| *** owalsh has quit IRC | 05:09 | |
| *** heyongli has quit IRC | 05:10 | |
| *** heyongli has joined #openstack-infra | 05:10 | |
| *** numans has quit IRC | 05:11 | |
| *** yamamoto has quit IRC | 05:12 | |
| *** numans has joined #openstack-infra | 05:13 | |
| *** janki has quit IRC | 05:13 | |
| *** e0ne has joined #openstack-infra | 05:13 | |
| *** lifeless has joined #openstack-infra | 05:16 | |
| *** vabada2 has joined #openstack-infra | 05:16 | |
| *** strigazi has quit IRC | 05:17 | |
| *** vabada has quit IRC | 05:18 | |
| *** strigazi has joined #openstack-infra | 05:19 | |
| *** Bhujay has joined #openstack-infra | 05:20 | |
| *** ykarel has joined #openstack-infra | 05:20 | |
| *** heyongli has quit IRC | 05:20 | |
| *** heyongli has joined #openstack-infra | 05:21 | |
| *** cshastri has joined #openstack-infra | 05:22 | |
| *** vabada has joined #openstack-infra | 05:24 | |
| *** strigazi_ has joined #openstack-infra | 05:24 | |
| *** strigazi has quit IRC | 05:27 | |
| *** vabada2 has quit IRC | 05:27 | |
| *** heyongli has quit IRC | 05:31 | |
| *** heyongli has joined #openstack-infra | 05:31 | |
| *** cshastri has quit IRC | 05:31 | |
| *** ispp has joined #openstack-infra | 05:33 | |
| *** isssp has quit IRC | 05:36 | |
| *** kjackal has joined #openstack-infra | 05:39 | |
| *** owalsh_ has quit IRC | 05:40 | |
| *** owalsh has joined #openstack-infra | 05:40 | |
| *** heyongli has quit IRC | 05:41 | |
| *** heyongli has joined #openstack-infra | 05:41 | |
| *** udesale has joined #openstack-infra | 05:44 | |
| *** cshastri has joined #openstack-infra | 05:47 | |
| *** heyongli has quit IRC | 05:51 | |
| *** heyongli has joined #openstack-infra | 05:51 | |
| *** snapiri- has joined #openstack-infra | 05:53 | |
| *** isssp has joined #openstack-infra | 05:53 | |
| *** ispp has quit IRC | 05:54 | |
| *** ispp has joined #openstack-infra | 05:55 | |
| *** CrayZee has quit IRC | 05:56 | |
| *** iranzo has joined #openstack-infra | 05:56 | |
| *** isssp has quit IRC | 05:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: New Fenix project https://review.openstack.org/571382 | 05:58 |
| openstackgerrit | Merged openstack-infra/project-config master: be more resilient in make_branch.sh for retired repositories https://review.openstack.org/574749 | 06:00 |
| *** heyongli has quit IRC | 06:01 | |
| *** heyongli has joined #openstack-infra | 06:02 | |
| *** openstackgerrit has quit IRC | 06:04 | |
| *** jcoufal has joined #openstack-infra | 06:04 | |
| *** openstackgerrit has joined #openstack-infra | 06:05 | |
| openstackgerrit | Tomi Juvonen proposed openstack-infra/project-config master: Zuul for New Fenix project https://review.openstack.org/571390 | 06:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Rally tagging ACL https://review.openstack.org/575130 | 06:05 |
| *** e0ne has quit IRC | 06:08 | |
| *** ccamacho has joined #openstack-infra | 06:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/577994 | 06:09 |
| *** yamamoto has joined #openstack-infra | 06:09 | |
| *** pcichy has joined #openstack-infra | 06:09 | |
| *** heyongli has quit IRC | 06:12 | |
| *** heyongli has joined #openstack-infra | 06:12 | |
| *** slaweq has joined #openstack-infra | 06:13 | |
| *** yamamoto has quit IRC | 06:14 | |
| *** alexchadin has joined #openstack-infra | 06:20 | |
| *** yolanda__ has joined #openstack-infra | 06:21 | |
| *** heyongli has quit IRC | 06:22 | |
| *** heyongli has joined #openstack-infra | 06:22 | |
| *** yolanda_ has quit IRC | 06:24 | |
| *** cshastri has quit IRC | 06:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove undefined job legacy-rally-dsvm-keystone https://review.openstack.org/577999 | 06:30 |
| *** pgadiya has joined #openstack-infra | 06:30 | |
| *** pgadiya has quit IRC | 06:30 | |
| AJaeger | infra-root: I don't know how we managed to merge changes so that we now get "Job legacy-rally-dsvm-keystone not defined | 06:30 |
| AJaeger | change 577999 fixes this, please review | 06:30 |
| *** heyongli has quit IRC | 06:32 | |
| *** heyongli has joined #openstack-infra | 06:32 | |
| *** andreas_s has joined #openstack-infra | 06:33 | |
| AJaeger | mordred: please see https://etherpad.openstack.org/p/neutron-sibling-setup | 06:35 |
| *** cshastri has joined #openstack-infra | 06:38 | |
| *** nicolasbock has joined #openstack-infra | 06:40 | |
| AJaeger | corvus: We removed the job from openstack-zuul-jobs and did not get errors - this looks odd, see 577999 | 06:40 |
| *** heyongli has quit IRC | 06:42 | |
| *** heyongli has joined #openstack-infra | 06:43 | |
| *** psachin` has quit IRC | 06:45 | |
| *** psachin` has joined #openstack-infra | 06:47 | |
| *** peereb has joined #openstack-infra | 06:48 | |
| *** peereb has quit IRC | 06:49 | |
| AJaeger | #status log project-config's zuul config is broken, it contains a removed job. https://review.openstack.org/577999 should fix it. | 06:49 |
| openstackstatus | AJaeger: finished logging | 06:49 |
| *** peereb has joined #openstack-infra | 06:49 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/577994 | 06:50 |
| *** peereb has quit IRC | 06:50 | |
| *** alexchadin has quit IRC | 06:52 | |
| *** heyongli has quit IRC | 06:53 | |
| *** heyongli has joined #openstack-infra | 06:53 | |
| *** pjrusak has joined #openstack-infra | 06:54 | |
| *** yamamoto has joined #openstack-infra | 06:56 | |
| *** noam has joined #openstack-infra | 07:01 | |
| *** heyongli has quit IRC | 07:03 | |
| *** heyongli has joined #openstack-infra | 07:03 | |
| *** noama has quit IRC | 07:04 | |
| *** hashar has joined #openstack-infra | 07:05 | |
| *** namnh has quit IRC | 07:06 | |
| *** slaweq has quit IRC | 07:06 | |
| *** lpetrut has joined #openstack-infra | 07:07 | |
| *** rcernin has quit IRC | 07:08 | |
| *** shardy has joined #openstack-infra | 07:10 | |
| *** gongysh has joined #openstack-infra | 07:10 | |
| *** slaweq has joined #openstack-infra | 07:10 | |
| *** heyongli has quit IRC | 07:13 | |
| *** pguimaraes has joined #openstack-infra | 07:13 | |
| *** heyongli has joined #openstack-infra | 07:13 | |
| *** tesseract has joined #openstack-infra | 07:14 | |
| *** slaweq has quit IRC | 07:15 | |
| *** kjackal has quit IRC | 07:16 | |
| *** zoli is now known as zoli|wfh | 07:16 | |
| *** zoli|wfh is now known as zoli | 07:16 | |
| *** pcaruana has joined #openstack-infra | 07:20 | |
| *** pgadiya has joined #openstack-infra | 07:22 | |
| *** pgadiya has quit IRC | 07:22 | |
| *** heyongli has quit IRC | 07:23 | |
| *** pjrusak has quit IRC | 07:23 | |
| *** heyongli has joined #openstack-infra | 07:24 | |
| *** florianf has joined #openstack-infra | 07:24 | |
| *** florianf has quit IRC | 07:25 | |
| *** andreas_s has quit IRC | 07:25 | |
| *** florianf has joined #openstack-infra | 07:26 | |
| *** andreas_s has joined #openstack-infra | 07:26 | |
| *** amoralej|off is now known as amoralej | 07:28 | |
| *** andreas_s_ has joined #openstack-infra | 07:28 | |
| *** kjackal has joined #openstack-infra | 07:29 | |
| *** yamahata has quit IRC | 07:30 | |
| *** andreas_s has quit IRC | 07:32 | |
| *** heyongli has quit IRC | 07:34 | |
| *** heyongli has joined #openstack-infra | 07:34 | |
| *** andreas_s_ has quit IRC | 07:38 | |
| *** andreas_s has joined #openstack-infra | 07:39 | |
| *** andreas_s has quit IRC | 07:39 | |
| *** andreas_s_ has joined #openstack-infra | 07:40 | |
| *** tosky has joined #openstack-infra | 07:41 | |
| *** jpena|off is now known as jpena | 07:44 | |
| *** heyongli has quit IRC | 07:44 | |
| *** heyongli has joined #openstack-infra | 07:44 | |
| *** andreww has joined #openstack-infra | 07:45 | |
| *** kjackal_ has joined #openstack-infra | 07:45 | |
| *** noam__ has joined #openstack-infra | 07:45 | |
| *** noam has quit IRC | 07:46 | |
| *** xarses_ has quit IRC | 07:48 | |
| *** kjackal has quit IRC | 07:48 | |
| *** kjackal has joined #openstack-infra | 07:50 | |
| *** kjackal_ has quit IRC | 07:53 | |
| *** heyongli has quit IRC | 07:54 | |
| *** heyongli has joined #openstack-infra | 07:54 | |
| *** alexchadin has joined #openstack-infra | 07:58 | |
| *** andreas_s_ has quit IRC | 07:59 | |
| *** andreas_s has joined #openstack-infra | 08:00 | |
| *** andreas_s has quit IRC | 08:00 | |
| *** andreas_s has joined #openstack-infra | 08:01 | |
| *** alexchadin has quit IRC | 08:01 | |
| *** jamesmcarthur has joined #openstack-infra | 08:03 | |
| *** heyongli has quit IRC | 08:04 | |
| *** heyongli has joined #openstack-infra | 08:05 | |
| *** jpich has joined #openstack-infra | 08:05 | |
| *** slaweq has joined #openstack-infra | 08:05 | |
| *** e0ne has joined #openstack-infra | 08:05 | |
| *** gfidente has joined #openstack-infra | 08:08 | |
| *** gfidente has joined #openstack-infra | 08:08 | |
| *** alexchadin has joined #openstack-infra | 08:08 | |
| *** jamesmcarthur has quit IRC | 08:08 | |
| *** kjackal_ has joined #openstack-infra | 08:10 | |
| *** noam__ has quit IRC | 08:13 | |
| *** kjackal has quit IRC | 08:14 | |
| *** heyongli has quit IRC | 08:15 | |
| *** heyongli has joined #openstack-infra | 08:15 | |
| *** slaweq has quit IRC | 08:15 | |
| *** kjackal has joined #openstack-infra | 08:17 | |
| *** kjackal_ has quit IRC | 08:20 | |
| *** pblaho has joined #openstack-infra | 08:21 | |
| *** salv-orlando has joined #openstack-infra | 08:22 | |
| *** heyongli has quit IRC | 08:25 | |
| *** heyongli has joined #openstack-infra | 08:25 | |
| *** salv-orlando has quit IRC | 08:25 | |
| *** salv-orlando has joined #openstack-infra | 08:26 | |
| *** sree has quit IRC | 08:26 | |
| *** kaisers1 has quit IRC | 08:27 | |
| *** sree has joined #openstack-infra | 08:27 | |
| xinliang | ianw: When will the debian-security repo finish synchronizing? | 08:30 |
| *** sree has quit IRC | 08:31 | |
| *** electrofelix has joined #openstack-infra | 08:35 | |
| *** alexchadin has quit IRC | 08:35 | |
| *** alexchadin has joined #openstack-infra | 08:35 | |
| *** heyongli has quit IRC | 08:35 | |
| *** heyongli has joined #openstack-infra | 08:35 | |
| *** verdurin has quit IRC | 08:37 | |
| *** alexchad_ has joined #openstack-infra | 08:37 | |
| *** rpittau has joined #openstack-infra | 08:37 | |
| *** alexchadin has quit IRC | 08:37 | |
| *** shardy has quit IRC | 08:38 | |
| *** alexchadin has joined #openstack-infra | 08:38 | |
| *** psachin` is now known as psachin | 08:39 | |
| *** jaosorior has quit IRC | 08:39 | |
| *** salv-orlando has quit IRC | 08:40 | |
| *** Hal has joined #openstack-infra | 08:40 | |
| *** Hal is now known as Guest33625 | 08:41 | |
| *** verdurin has joined #openstack-infra | 08:41 | |
| *** kaisers has joined #openstack-infra | 08:41 | |
| *** alexchad_ has quit IRC | 08:42 | |
| *** kjackal_ has joined #openstack-infra | 08:43 | |
| *** jcoufal has quit IRC | 08:43 | |
| *** shardy has joined #openstack-infra | 08:43 | |
| *** heyongli has quit IRC | 08:45 | |
| *** heyongli has joined #openstack-infra | 08:46 | |
| *** kjackal_ has quit IRC | 08:46 | |
| *** kjackal has quit IRC | 08:46 | |
| *** derekh has joined #openstack-infra | 08:49 | |
| *** kjackal has joined #openstack-infra | 08:50 | |
| *** priteau has joined #openstack-infra | 08:54 | |
| *** heyongli has quit IRC | 08:56 | |
| *** heyongli has joined #openstack-infra | 08:56 | |
| *** iyamahat has joined #openstack-infra | 08:56 | |
| *** iyamahat has quit IRC | 08:59 | |
| *** jchhatbar is now known as janki | 09:01 | |
| *** roman_g has joined #openstack-infra | 09:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: use kazoo retry facilities https://review.openstack.org/535537 | 09:04 |
| *** heyongli has quit IRC | 09:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove undefined job legacy-rally-dsvm-keystone https://review.openstack.org/577999 | 09:06 |
| *** heyongli has joined #openstack-infra | 09:06 | |
| openstackgerrit | Adit Sarfaty proposed openstack-infra/project-config master: Add octavia to vmware_nsx jobs https://review.openstack.org/576394 | 09:09 |
| *** shardy has quit IRC | 09:10 | |
| *** dingyichen has quit IRC | 09:11 | |
| *** kjackal has quit IRC | 09:12 | |
| *** kjackal has joined #openstack-infra | 09:12 | |
| *** shardy has joined #openstack-infra | 09:13 | |
| *** slaweq has joined #openstack-infra | 09:15 | |
| *** heyongli has quit IRC | 09:16 | |
| *** heyongli has joined #openstack-infra | 09:16 | |
| *** lifeless has quit IRC | 09:16 | |
| *** dtantsur|afk is now known as dtantsur | 09:18 | |
| *** ykarel_ has joined #openstack-infra | 09:20 | |
| *** ykarel_ is now known as ykarel|away | 09:22 | |
| *** ykarel has quit IRC | 09:23 | |
| *** slaweq has quit IRC | 09:23 | |
| *** ykarel|away has quit IRC | 09:25 | |
| *** heyongli has quit IRC | 09:26 | |
| *** heyongli has joined #openstack-infra | 09:27 | |
| ianw | xinliang: sorry, i'll have to look at it more tomorrow as it currently doesn't work | 09:35 |
| ianw | i have half a patch | 09:36 |
| *** heyongli has quit IRC | 09:37 | |
| *** heyongli has joined #openstack-infra | 09:37 | |
| *** salv-orlando has joined #openstack-infra | 09:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: Zuul for New Fenix project https://review.openstack.org/571390 | 09:41 |
| *** salv-orlando has quit IRC | 09:46 | |
| *** strigazi_ is now known as strigazi | 09:47 | |
| *** heyongli has quit IRC | 09:47 | |
| *** heyongli has joined #openstack-infra | 09:47 | |
| *** yolanda_ has joined #openstack-infra | 09:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add octavia to vmware_nsx jobs https://review.openstack.org/576394 | 09:52 |
| *** yolanda__ has quit IRC | 09:53 | |
| *** heyongli has quit IRC | 09:57 | |
| *** heyongli has joined #openstack-infra | 09:57 | |
| *** quiquell|rover has joined #openstack-infra | 10:02 | |
| quiquell|rover | Hello | 10:02 |
| quiquell|rover | What's the different between this two hosts ? | 10:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Add Zuul config for openstack-service-broker https://review.openstack.org/573460 | 10:02 |
| quiquell|rover | centos-7-vexxhost-ca-ymq-1-0000341620 | 10:02 |
| quiquell|rover | centos-7-inap-mtl01-0000340080 | 10:02 |
| quiquell|rover | ? | 10:02 |
| *** heyongli has quit IRC | 10:07 | |
| *** heyongli has joined #openstack-infra | 10:08 | |
| *** stakeda has quit IRC | 10:08 | |
| *** zoli is now known as zoli|lunch | 10:14 | |
| ianw | quiquell|rover: one is a host in vexxhost, one is a host in inap -- different cloud providers | 10:14 |
| quiquell|rover | ianw: So they could have different latencies accessing docker.io or starting docker images ? | 10:16 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: More minor debian-security mirror fixes https://review.openstack.org/578057 | 10:16 |
| *** jchhatbar has joined #openstack-infra | 10:17 | |
| *** heyongli has quit IRC | 10:18 | |
| *** heyongli has joined #openstack-infra | 10:18 | |
| ianw | quiquell|rover: yes ... although there was some chatter previously about instability with docker.io | 10:19 |
| *** janki has quit IRC | 10:19 | |
| *** numans has quit IRC | 10:20 | |
| quiquell|rover | ianw: ack | 10:20 |
| *** numans has joined #openstack-infra | 10:25 | |
| *** jcoufal has joined #openstack-infra | 10:27 | |
| *** heyongli has quit IRC | 10:28 | |
| *** heyongli has joined #openstack-infra | 10:28 | |
| *** jaosorior has joined #openstack-infra | 10:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: [translation] doc generatepot jobs for 3 projects https://review.openstack.org/545377 | 10:30 |
| *** sshnaidm|afk is now known as sshnaidm | 10:30 | |
| quiquell|rover | ianw: What kind of issue did you found at docker.io ? | 10:33 |
| *** boden has joined #openstack-infra | 10:38 | |
| *** heyongli has quit IRC | 10:38 | |
| *** heyongli has joined #openstack-infra | 10:38 | |
| *** boden has quit IRC | 10:38 | |
| *** salv-orlando has joined #openstack-infra | 10:42 | |
| *** isviridov_away has quit IRC | 10:46 | |
| *** vaidy has quit IRC | 10:46 | |
| *** salv-orlando has quit IRC | 10:47 | |
| ianw | quiquell|rover: i'm not sure exactly. clarkb might be a good resource in a few hours as he's been more active with it | 10:47 |
| *** heyongli has quit IRC | 10:48 | |
| *** slaweq has joined #openstack-infra | 10:48 | |
| *** heyongli has joined #openstack-infra | 10:49 | |
| *** kjackal has quit IRC | 10:56 | |
| *** yamamoto has quit IRC | 10:58 | |
| *** annp has quit IRC | 10:58 | |
| *** heyongli has quit IRC | 10:59 | |
| *** heyongli has joined #openstack-infra | 10:59 | |
| *** andreas_s has quit IRC | 11:00 | |
| *** andreas_s has joined #openstack-infra | 11:00 | |
| *** andreas_s has quit IRC | 11:00 | |
| *** vaidy has joined #openstack-infra | 11:01 | |
| *** andreas_s has joined #openstack-infra | 11:01 | |
| *** isviridov_away has joined #openstack-infra | 11:02 | |
| *** andreas_s has quit IRC | 11:02 | |
| *** andreas_s has joined #openstack-infra | 11:02 | |
| *** andreas_s has quit IRC | 11:03 | |
| *** jpena is now known as jpena|lunch | 11:03 | |
| *** pguimaraes has quit IRC | 11:04 | |
| *** pguimaraes has joined #openstack-infra | 11:06 | |
| *** yamamoto has joined #openstack-infra | 11:07 | |
| *** kjackal has joined #openstack-infra | 11:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Test log-inventory role using base-test https://review.openstack.org/563788 | 11:09 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Use log-inventory in base jobs https://review.openstack.org/563789 | 11:09 |
| *** heyongli has quit IRC | 11:09 | |
| *** heyongli has joined #openstack-infra | 11:09 | |
| *** rosmaita has joined #openstack-infra | 11:09 | |
| *** andreas_s has joined #openstack-infra | 11:17 | |
| frickler | infra-root: do we know whether packethost might have all instances with MTU 1450, while other providers do 1500? seeing strange failures for some neutron jobs http://logs.openstack.org/63/576163/2/check/neutron-tempest-linuxbridge/e5f6812/logs/screen-q-agt.txt.gz?level=WARNING#_Jun_25_21_12_04_022971 | 11:19 |
| *** heyongli has quit IRC | 11:19 | |
| *** heyongli has joined #openstack-infra | 11:19 | |
| tristanC | ianw: btw, i wasn't able to use the log-inventory role in my base job, it failed without: https://review.openstack.org/577674 | 11:20 |
| frickler | infra-root: seem the host-info confirms this, see http://logs.openstack.org/63/576163/2/check/neutron-tempest-linuxbridge/e5f6812/zuul-info/host-info.primary.yaml | 11:21 |
| ianw | frickler: hmm, there was discussion of packethost failures previously with clarkb and mnaser ... about 12 hours ago | 11:21 |
| frickler | infra-root: how do our capacities look like currently, can we afford to turn off packethost until we have a fix for this? | 11:21 |
| ianw | frickler: https://etherpad.openstack.org/p/packet-timeout-vms ... though in this case, it was packets going missing? | 11:23 |
| *** slaweq has quit IRC | 11:23 | |
| frickler | ianw: probably two different issues, though both could have the reduced MTU as root cause | 11:26 |
| *** andreas_s has quit IRC | 11:26 | |
| *** sree has joined #openstack-infra | 11:28 | |
| ianw | | mtu | 1450 | | 11:28 |
| ianw | that is on openstackzuul-net | 11:28 |
| ianw | frickler: ^ hmm, i don't think i have access to change that. i can loop you in on an email | 11:29 |
| *** heyongli has quit IRC | 11:29 | |
| *** heyongli has joined #openstack-infra | 11:30 | |
| *** yamamoto has quit IRC | 11:30 | |
| *** sambetts|afk is now known as sambetts | 11:31 | |
| frickler | ianw: well if they use vxlan as tenant network, just changing it in neutron won't help | 11:32 |
| *** elod has quit IRC | 11:32 | |
| *** sree has quit IRC | 11:32 | |
| *** yamamoto has joined #openstack-infra | 11:33 | |
| frickler | ianw: I know we do set our side to 1450 in devstack somewhere, assuming the host has 1500. we can tune this down to 1400, but that will only help with devstack and not with OSA | 11:33 |
| *** elod has joined #openstack-infra | 11:33 | |
| *** udesale has quit IRC | 11:33 | |
| ianw | frickler: yeah, it is vxlan i believe | 11:34 |
| *** andreas_s has joined #openstack-infra | 11:35 | |
| *** camunoz has joined #openstack-infra | 11:36 | |
| *** tpsilva has joined #openstack-infra | 11:37 | |
| *** zoli|lunch is now known as zoli|wfh | 11:38 | |
| *** zoli|wfh is now known as zoli | 11:38 | |
| *** alexchad_ has joined #openstack-infra | 11:38 | |
| *** rh-jelabarre has joined #openstack-infra | 11:38 | |
| *** alexchadin has quit IRC | 11:38 | |
| *** heyongli has quit IRC | 11:40 | |
| *** heyongli has joined #openstack-infra | 11:40 | |
| *** ldnunes has joined #openstack-infra | 11:41 | |
| *** andreas_s has quit IRC | 11:44 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add /etc/localtime to bubblewrap default ro bind https://review.openstack.org/578072 | 11:44 |
| *** derekh has quit IRC | 11:47 | |
| *** andreas_s has joined #openstack-infra | 11:48 | |
| *** derekh has joined #openstack-infra | 11:49 | |
| *** heyongli has quit IRC | 11:50 | |
| *** heyongli has joined #openstack-infra | 11:50 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support paging when listing github installations https://review.openstack.org/578077 | 11:53 |
| *** amoralej is now known as amoralej|lunch | 11:59 | |
| *** heyongli has quit IRC | 12:00 | |
| *** heyongli has joined #openstack-infra | 12:00 | |
| *** ldnunes has quit IRC | 12:04 | |
| *** eharney has quit IRC | 12:10 | |
| *** heyongli has quit IRC | 12:10 | |
| *** heyongli has joined #openstack-infra | 12:11 | |
| *** shardy_ has joined #openstack-infra | 12:12 | |
| *** hashar has quit IRC | 12:13 | |
| *** bobh has joined #openstack-infra | 12:13 | |
| *** hashar has joined #openstack-infra | 12:14 | |
| *** hashar has quit IRC | 12:14 | |
| *** hashar has joined #openstack-infra | 12:14 | |
| *** jpena|lunch is now known as jpena | 12:14 | |
| *** shardy has quit IRC | 12:15 | |
| *** shardy_ is now known as shardy | 12:15 | |
| *** alexchad_ has quit IRC | 12:17 | |
| *** heyongli has quit IRC | 12:21 | |
| *** heyongli has joined #openstack-infra | 12:21 | |
| *** trown|outtypewww is now known as trown | 12:24 | |
| *** slaweq has joined #openstack-infra | 12:25 | |
| *** slaweq has quit IRC | 12:29 | |
| *** rlandy has joined #openstack-infra | 12:30 | |
| *** heyongli has quit IRC | 12:31 | |
| *** alexchadin has joined #openstack-infra | 12:31 | |
| *** heyongli has joined #openstack-infra | 12:31 | |
| *** dhajare has quit IRC | 12:32 | |
| *** rosmaita has quit IRC | 12:32 | |
| *** sthussey has joined #openstack-infra | 12:34 | |
| *** heyongli has quit IRC | 12:41 | |
| *** heyongli has joined #openstack-infra | 12:41 | |
| *** jcoufal has quit IRC | 12:42 | |
| *** andreas_s has quit IRC | 12:48 | |
| *** andreas_s has joined #openstack-infra | 12:50 | |
| *** mriedem has joined #openstack-infra | 12:51 | |
| *** heyongli has quit IRC | 12:51 | |
| *** heyongli has joined #openstack-infra | 12:52 | |
| *** andreas_s has quit IRC | 12:57 | |
| *** edmondsw has joined #openstack-infra | 12:58 | |
| *** andreas_s has joined #openstack-infra | 12:59 | |
| *** elod has quit IRC | 12:59 | |
| *** yamamoto has quit IRC | 13:01 | |
| *** r-mibu has joined #openstack-infra | 13:01 | |
| *** heyongli has quit IRC | 13:02 | |
| *** rossella_s has joined #openstack-infra | 13:02 | |
| *** heyongli has joined #openstack-infra | 13:02 | |
| *** kgiusti has joined #openstack-infra | 13:05 | |
| *** quiquell|rover is now known as quique|rover|lch | 13:08 | |
| *** heyongli has quit IRC | 13:12 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Disable packethost to debug possible MTU issues https://review.openstack.org/578096 | 13:12 |
| *** heyongli has joined #openstack-infra | 13:12 | |
| *** jamesdenton has joined #openstack-infra | 13:12 | |
| frickler | infra-root: not sure if we really want to do this, as we seem to be pretty close to max-capacity, but if nobody has a better short-term idea ^^ | 13:13 |
| openstackgerrit | Javier Peña proposed openstack-infra/system-config master: Move AFS mirror code to puppet-openstackci https://review.openstack.org/529032 | 13:15 |
| mnaser | frickler: i've had a lot of issues with OSA, afaik, packet has their own underlay networking to run their infra | 13:15 |
| mnaser | so i wonder if that's not impacting mtu as well | 13:15 |
| frickler | mnaser: well, for neutron we would have to set the mtu for tenant networks from 1450, which we do now, to 1400. probably similar things would have to happen for OSA | 13:16 |
| mnaser | frickler: but we dont even hit that point of failure, were getting something that makes the entire network stack fail | 13:17 |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support https://review.openstack.org/529376 | 13:18 |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support for RHEL/CentOS https://review.openstack.org/528739 | 13:18 |
| *** elod has joined #openstack-infra | 13:18 | |
| *** heyongli has quit IRC | 13:22 | |
| *** dtantsur is now known as dtantsur|brb | 13:22 | |
| *** heyongli has joined #openstack-infra | 13:22 | |
| *** eharney has joined #openstack-infra | 13:23 | |
| *** caphrim007_ has quit IRC | 13:24 | |
| openstackgerrit | Artem Goncharov proposed openstack-infra/nodepool master: retire shade in favor of openstacksdk https://review.openstack.org/572829 | 13:25 |
| frickler | mnaser: o.k., so the OSA failures may well be something completely different. | 13:25 |
| *** frickler has quit IRC | 13:26 | |
| *** frickler has joined #openstack-infra | 13:26 | |
| frickler | woa, bumpy ride today | 13:26 |
| frickler | mnaser: but the neutron failures I looked at all seem to come from the reduced mtu. devstack tries to handle this, but it seems to fail for some scenarios, see http://git.openstack.org/cgit/openstack-dev/devstack/tree/playbooks/pre.yaml | 13:27 |
| *** evrardjp has joined #openstack-infra | 13:28 | |
| mnaser | frickler: yeah there is a bit of a backlog i can refer to but we were seeing a kernel stack trace followed by networking being lost | 13:28 |
| *** bobh has quit IRC | 13:29 | |
| *** sree has joined #openstack-infra | 13:29 | |
| *** quique|rover|lch is now known as quiquell|rover | 13:29 | |
| *** yamamoto has joined #openstack-infra | 13:29 | |
| *** evrardjp_ has quit IRC | 13:30 | |
| frickler | mnaser: if I saw things correctly, lxc does set up its bridging earlier, though. maybe this leads to issues if the bridge has 1500 and the uplink interface 1450? | 13:30 |
| mnaser | frickler: we don't attach the port on the vm to our bridges however | 13:31 |
| mnaser | and external access gets routed to $host and then NAT'd out | 13:31 |
| *** udesale has joined #openstack-infra | 13:31 | |
| dhellmann | now that the relevant devstack change is in place, I would like to test applying the lib-forward-testing-python3 jobs to some repos. Could I get someone to look at this project-template change, please? https://review.openstack.org/575925 | 13:32 |
| *** heyongli has quit IRC | 13:32 | |
| *** alexchadin has quit IRC | 13:33 | |
| *** heyongli has joined #openstack-infra | 13:33 | |
| *** sree has quit IRC | 13:33 | |
| *** agopi has joined #openstack-infra | 13:34 | |
| frickler | dhellmann: ack. is there also a followup that uses this template? | 13:34 |
| dhellmann | frickler : yeah, let me find that one | 13:34 |
| *** agopi has quit IRC | 13:34 | |
| dhellmann | frickler : this one in oslo.config https://review.openstack.org/#/c/575927/ | 13:35 |
| *** mihalis68 has joined #openstack-infra | 13:35 | |
| *** caphrim007 has joined #openstack-infra | 13:35 | |
| dhellmann | frickler : and thanks! | 13:35 |
| frickler | dhellmann: yeah, I just realized I just had to follow the same topic. ;) | 13:35 |
| *** caphrim007 has quit IRC | 13:39 | |
| *** e0ne has quit IRC | 13:39 | |
| *** snapiri- has quit IRC | 13:41 | |
| *** hongbin has joined #openstack-infra | 13:42 | |
| *** elod_ has joined #openstack-infra | 13:42 | |
| *** heyongli has quit IRC | 13:43 | |
| *** heyongli has joined #openstack-infra | 13:43 | |
| *** agopi has joined #openstack-infra | 13:43 | |
| fungi | for some reason i thought devstack-gate used to dynamically adjust the multi-node overlay to take the interface mtu into account | 13:43 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: add lib-forward-testing-py3 project template https://review.openstack.org/575925 | 13:43 |
| fungi | oh, or devstack | 13:43 |
| *** caphrim007 has joined #openstack-infra | 13:44 | |
| *** e0ne has joined #openstack-infra | 13:44 | |
| *** rh-jelabarre has quit IRC | 13:46 | |
| *** caphrim007 has quit IRC | 13:49 | |
| *** esarault has joined #openstack-infra | 13:50 | |
| *** alexchadin has joined #openstack-infra | 13:51 | |
| frickler | fungi: actually devstack does this for new jobs it seems, but not for legacy jobs. so this is creating a public network with MTU 1500 which fails when ens3 has MTU 1450: http://logs.openstack.org/63/576163/2/check/neutron-tempest-linuxbridge/e5f6812/logs/devstacklog.txt.gz#_2018-06-25_21_12_12_465 | 13:53 |
| *** heyongli has quit IRC | 13:53 | |
| *** heyongli has joined #openstack-infra | 13:53 | |
| frickler | fungi: also the check in neutron that verifies this is only 4 weeks old it seems, which would explain why it didn't fail on infracloud earlier | 13:53 |
| openstackgerrit | Merged openstack-infra/project-config master: Disable packethost to debug possible MTU issues https://review.openstack.org/578096 | 13:53 |
| fungi | we haven't had infracloud in ages | 13:54 |
| *** elod has quit IRC | 13:54 | |
| frickler | but I also see neutron jobs failing that do seem to correctly use the reduced MTU, so possibly there are other issues, too. maybe even the same ones mnaser is seeing | 13:55 |
| *** elod_ is now known as elod | 13:55 | |
| fungi | yeah, there could be more than one problem | 13:55 |
| fungi | or it could be that for some reason path mtu discovery is failing for certain destinations and that's why we see downloads from those hang indefintely (classic pmtud blackhole behavior?) | 13:56 |
| *** amoralej|lunch is now known as amoralej | 13:58 | |
| *** jesslampe has joined #openstack-infra | 13:58 | |
| frickler | fungi: that also a possibility, but I don't think that that would explain failures in the tempest phase. it still could be interesting to try and set mss clamping to avoid blackholing | 13:58 |
| *** jaosorior has quit IRC | 13:59 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: log-inventory: add missing zuul_info_dir prep https://review.openstack.org/577674 | 13:59 |
| fungi | frickler: yep, i was trying to say it's possible it's more than one problem one of which is mtu-related, or perhaps we're seeing a couple of problems (and so different symptoms) both of which stem from the lower mtu | 14:01 |
| *** jchhatbar is now known as janki | 14:01 | |
| *** jamesmcarthur has joined #openstack-infra | 14:02 | |
| *** rfolco_ is now known as rfolco | 14:03 | |
| *** rh-jelabarre has joined #openstack-infra | 14:03 | |
| *** heyongli has quit IRC | 14:03 | |
| *** esarault_ has joined #openstack-infra | 14:03 | |
| *** heyongli has joined #openstack-infra | 14:03 | |
| *** rpioso|afk is now known as rpioso | 14:04 | |
| *** esarault has quit IRC | 14:06 | |
| *** rossella_s has quit IRC | 14:07 | |
| *** alexchadin has quit IRC | 14:08 | |
| *** rossella_s has joined #openstack-infra | 14:09 | |
| *** cshastri has quit IRC | 14:10 | |
| *** jamesmcarthur_ has joined #openstack-infra | 14:12 | |
| *** heyongli has quit IRC | 14:13 | |
| *** rossella_s has quit IRC | 14:13 | |
| *** heyongli has joined #openstack-infra | 14:14 | |
| clarkb | when I looked at timeouts in logstash packethost didnt really seem to stand out other than for osa, if you removed osa it was in same range of timeouts as other clouds | 14:17 |
| *** rossella_s has joined #openstack-infra | 14:17 | |
| *** slaweq has joined #openstack-infra | 14:17 | |
| clarkb | I expect the mtu issues dont produce timeouts or my query was incomolete | 14:18 |
| clarkb | because devstack gate has an internaltimeout | 14:18 |
| *** heyongli has quit IRC | 14:18 | |
| *** heyongli has joined #openstack-infra | 14:19 | |
| frickler | clarkb: yes, no timeouts but genuine jobs failures due to neutron checks, see http://logs.openstack.org/63/576163/2/check/neutron-tempest-linuxbridge/e5f6812/logs/screen-q-agt.txt.gz?level=WARNING#_Jun_25_21_12_04_022971 | 14:19 |
| frickler | clarkb: modern neutron jobs detect the reduced MTU and work fine, this legacy job doesn't | 14:20 |
| mordred | if osa sets up a network with a bad mtu and then tries to download things from the internet over that network, we could see that failure differently than if devstack is getting the mtu wrong but it's not using that network for anything other than tempest tests | 14:20 |
| mordred | right? | 14:21 |
| * mordred just thinking out loud | 14:21 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add min_avail_hdd governor for zuul-executor https://review.openstack.org/577948 | 14:21 |
| clarkb | ya its possible the hardware reacts differently | 14:21 |
| clarkb | if you push packets over real interface vs completely virtual | 14:22 |
| fungi | clarkb: it was more that i wondered whether osa was configuring something in such a way that it broke pmtud for certain destinations, which wouldn't show up unless you ran somewhere with a lower than expected mtu | 14:23 |
| *** e0ne has quit IRC | 14:23 | |
| *** heyongli has quit IRC | 14:24 | |
| clarkb | fungi neutron breaks it for sure | 14:24 |
| *** ldnunes has joined #openstack-infra | 14:24 | |
| *** heyongli has joined #openstack-infra | 14:24 | |
| fungi | like, maybe they block icmp or something? | 14:24 |
| clarkb | multiple l2 devices with low mtus are possible and you need l3 to discover | 14:24 |
| clarkb | icmp is blocked by default by security groups too | 14:25 |
| *** eharney has quit IRC | 14:25 | |
| fungi | ahh, yeah no l3 hop at the mtu drop to take care of fragmenting or sending icmp too-large responses | 14:25 |
| *** e0ne has joined #openstack-infra | 14:25 | |
| *** felipemonteiro has joined #openstack-infra | 14:26 | |
| *** psachin has quit IRC | 14:28 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:29 | |
| *** felipemonteiro has quit IRC | 14:33 | |
| *** quiquell|rover is now known as quiquell|off | 14:34 | |
| *** heyongli has quit IRC | 14:34 | |
| *** heyongli has joined #openstack-infra | 14:34 | |
| clarkb | in general we probably cannot assume an mtu of 1500, our arm64 instances also have an mtu of 1450 | 14:34 |
| clarkb | maybe need to write this down | 14:34 |
| *** ccamacho has quit IRC | 14:35 | |
| *** ccamacho has joined #openstack-infra | 14:35 | |
| mnaser | fungi, mordred: fwiw in OSA we don't bridge in the physical interface of the VM | 14:39 |
| mnaser | we have 2 bridges, one that is used by lxc to bridge host and containers, another one to bridge containers together | 14:40 |
| *** efried has joined #openstack-infra | 14:40 | |
| mnaser | data egress from containers to host, and using iptables masq out to the internets | 14:40 |
| *** bobh has joined #openstack-infra | 14:40 | |
| *** heyongli has quit IRC | 14:40 | |
| mnaser | i don't know if that helps.. but yea.h | 14:40 |
| *** dhajare has joined #openstack-infra | 14:40 | |
| *** rh-jelabarre has quit IRC | 14:41 | |
| *** eharney has joined #openstack-infra | 14:42 | |
| clarkb | the NAT transition may explain the different behavior | 14:43 |
| *** sree has joined #openstack-infra | 14:43 | |
| clarkb | if linix does somethibg weord going from 1500 to 1450 across a NAT | 14:43 |
| fungi | clarkb: for some reason i thought we had included possible mtu differences in https://docs.openstack.org/infra/manual/testing.html but it looks like we did not | 14:44 |
| *** dtantsur|brb is now known as dtantsur | 14:45 | |
| clarkb | fungi: I think this is all new as of january with liaro cloud. Previously mtu was always 1500 | 14:45 |
| *** jcoufal has joined #openstack-infra | 14:46 | |
| fungi | clarkb: if icmp need-to-fragment is being blocked, then anything with df set is going to get dropped on the floor | 14:46 |
| fungi | so possible the nat isn't getting the signals it needs for that | 14:46 |
| mnaser | fungi: could that explain the traceback though? ;x | 14:47 |
| fungi | nope | 14:47 |
| fungi | unless the traceback is a symptom of a hung/dead socket | 14:47 |
| clarkb | the tracwback is sympom of checksum error | 14:48 |
| clarkb | which I think could happen if there isnt internal agreement on frame sizes? | 14:48 |
| clarkb | because you'd get different window of bits in either side to checksum | 14:48 |
| *** rossella_s has quit IRC | 14:49 | |
| frickler | mnaser: can you point me to where the iptables masquerading is being set up? | 14:49 |
| clarkb | I would fix the mtu issue and see if that changes behavior for sure | 14:49 |
| fungi | clarkb: oh, i suppose if there was a truncated frame... but how is that even making it into the socket? that should be handled by the kernel tcp/ip stack | 14:50 |
| fungi | rather, before it gets to tcp/ip even | 14:50 |
| fungi | by the ethernet driver itself | 14:50 |
| fungi | unless you're hooking a bpf or something | 14:50 |
| *** pguimaraes has quit IRC | 14:51 | |
| *** rossella_s has joined #openstack-infra | 14:51 | |
| clarkb | I think the offloading id what makes it possibly weird | 14:51 |
| clarkb | linux is pushing it down into "hardware" aiui before it sees any of it | 14:52 |
| clarkb | and the lines blur a bit in stack responsibility | 14:52 |
| *** pguimaraes has joined #openstack-infra | 14:52 | |
| *** gongysh has quit IRC | 14:53 | |
| fungi | ahh | 14:53 |
| mnaser | frickler: the way it gets setup is just the out of the box way that comes from packagers | 14:54 |
| mnaser | OSA doesnt do it | 14:54 |
| mnaser | (aka lxcbr0) | 14:54 |
| mnaser | looks like we no longer have ethtool which we used to collect info from | 14:55 |
| clarkb | mnaser there were other bridges you are creating though when I looked. I think set up for neutron? | 14:55 |
| mnaser | clarkb: there might be one for neutron but im not sure if that gets bridged to the main instance, also, in some of these job failures, we don't even deploy neutron | 14:56 |
| *** sree has quit IRC | 14:56 | |
| mnaser | oh | 14:56 |
| mnaser | http://logs.openstack.org/38/577838/1/check/openstack-ansible-functional-centos-7/089d056/logs/host/messages.txt.gz#_Jun_25_17_18_43 | 14:57 |
| *** sree has joined #openstack-infra | 14:57 | |
| clarkb | frickler the legacy jobs that fail rely on devstack gate right? | 14:57 |
| mnaser | i have no idea why/how that happens | 14:57 |
| mnaser | but i think that might be related. | 14:57 |
| fungi | ethtool (and similarly mii-tool) has always been hit-or-miss on whether and how much info it could get from various drivers | 14:57 |
| *** pguimaraes has quit IRC | 14:57 | |
| clarkb | mnaser that does look suspicious | 14:58 |
| mnaser | http://git.openstack.org/cgit/openstack/openstack-ansible/tree/tests/roles/bootstrap-host/defaults/main.yml#n206 | 14:58 |
| clarkb | frickler: in the devstack-gate case ( and possibly multinode overlay in native v3 case we can (amd should) math a proper mtu and set that) | 14:58 |
| *** pguimaraes has joined #openstack-infra | 14:58 | |
| mnaser | https://github.com/openstack/openstack-ansible/commit/0aa1ce62e59409d3c2c5306993e2f7370285adf7 made it check everything | 15:00 |
| mnaser | https://github.com/openstack/openstack-ansible/commit/51f4dec329d5b451450234990fcf8c6baa3fa80a it was introduced here i guess | 15:01 |
| panda | is there a better way to reuse playbooks and roles from openstack-zuul-jobs without copying them to our owm repo ? | 15:01 |
| *** sree has quit IRC | 15:01 | |
| clarkb | panda not playbooks they can only be run out of the repo they live in but roles and jobs can be reused simply by referring to them | 15:02 |
| panda | clarkb: ok, thanks. | 15:02 |
| mnaser | clarkb: im checking with the osa team | 15:02 |
| pabelanger | panda: clarkb: or parent to the other job with the playbook | 15:03 |
| pabelanger | we do that in a few places today | 15:03 |
| panda | pabelanger: it's a legacy playbook, and I don't see any job that I could parent that uses only that one. COpying that is fine for now. | 15:04 |
| mnaser | clarkb: in other discussions.. https://twitter.com/_mnaser/status/1011589906328621056 | 15:04 |
| mnaser | do you think we can do that re: thread | 15:04 |
| frickler | clarkb: there is some special handling for v3 multinode already, so I think only ds-gate would need amending, yes, similar to what andreaf did here http://git.openstack.org/cgit/openstack-dev/devstack/tree/playbooks/pre.yaml | 15:05 |
| *** hemna_ has joined #openstack-infra | 15:06 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for latent cells v1 rebuild race bug 1709985 https://review.openstack.org/578129 | 15:06 |
| openstack | bug 1709985 in OpenStack Compute (nova) "test_rebuild_server_in_error_state randomly times out waiting for rebuilding instance to be active (cells v1)" [Low,Won't fix] https://launchpad.net/bugs/1709985 | 15:06 |
| clarkb | frickler: probably the thing to do is get mtu of interface that has default route on it. then do $mtu - 50 | 15:06 |
| *** sree has joined #openstack-infra | 15:07 | |
| *** e0ne has quit IRC | 15:07 | |
| *** e0ne has joined #openstack-infra | 15:11 | |
| *** jamesmca_ has joined #openstack-infra | 15:12 | |
| clarkb | mnaser: for openstack we have long said we need multiple clouds to perform that sort of testing With kata we've deviated from that because they do not have multiple clouds and havent had that so isnt a regression | 15:12 |
| mnaser | clarkb: problem is with the qemu performance without nested virt that job would be practically impossible :( | 15:12 |
| *** Bhujay has quit IRC | 15:12 | |
| *** janki has quit IRC | 15:13 | |
| mnaser | because it has to run the equiv of 'tempest' against a magnum deployed k8s cluster | 15:13 |
| frickler | mnaser: searching for you kernel msg I found https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1705447 , does you issue only happen on centos? maybe they a lacking that patch? though it also sounds as though it may be cosmetic only | 15:13 |
| openstack | Launchpad bug 1705447 in linux (Ubuntu Zesty) "misleading kernel warning skb_warn_bad_offload during checksum calculation" [Medium,Fix released] - Assigned to Joseph Salisbury (jsalisbury) | 15:13 |
| clarkb | yup. I think if we can manage more than one cloud that can do it we will be ok. I think we have to stick to this rule because other people will want that functionality too and we just cant do it from a single source and be reliable | 15:14 |
| mnaser | frickler: cant remember, i'll have to dig in, but did you see my comments above about OSA for some reason forcing checksums | 15:14 |
| frickler | mnaser: yes, I'm assuming that that's the thing that may be triggering these kernel warnings | 15:14 |
| mnaser | clarkb: bummer, because i feel like this holds back projects such as octavia too who struggle with the lack of nested virt | 15:15 |
| clarkb | yes they are exactly who I am worried about | 15:15 |
| johnsom | Ansible is feeling the pain too | 15:15 |
| mnaser | i'd be totally okay with them using our infra too for their use case | 15:15 |
| clarkb | I dont want them running single sourced gate jobs | 15:15 |
| *** ccamacho has quit IRC | 15:15 | |
| clarkb | we have lots of experience with why this would be a bad idea | 15:15 |
| johnsom | So far, OVH and limestone have nested virt enabled, but the kernel issue is causing failures. | 15:16 |
| clarkb | vlouds come and go frequently | 15:16 |
| johnsom | The limestone folks are looking into it and have updated the kernel bug we opened | 15:16 |
| dpawlik_ | johnsom: what kernel we need to have? | 15:16 |
| frickler | mnaser: actually https://access.redhat.com/solutions/3358131 may be an even better fit. contains "fragmentation" as keyword, also mentions turning off offloading doesn't help | 15:17 |
| johnsom | We don't know at this point | 15:17 |
| mnaser | "Please feel free to contact Red Hat Support representative for more information about this issue." :( | 15:17 |
| dpawlik_ | johnsom: we can upgrade one host to another kernel release to check if its working | 15:17 |
| mnaser | frickler: also wow that's recent | 15:17 |
| johnsom | dpawlik_ Sure, if you let me know what the host is I can try to watch for it, but at this point it would be hard to get a test run. I know limestone can reproduce the issue locally, which would be the best scenario | 15:18 |
| *** felipemonteiro_ has quit IRC | 15:19 | |
| dpawlik_ | johnsom: come on priv | 15:19 |
| *** felipemonteiro_ has joined #openstack-infra | 15:19 | |
| *** felipemonteiro has joined #openstack-infra | 15:22 | |
| *** felipemonteiro_ has quit IRC | 15:24 | |
| *** dpawlik_ has quit IRC | 15:27 | |
| clarkb | mnaser https://github.com/kata-containers/ci/issues/42#issuecomment-400212254 is somewhat discouraging | 15:29 |
| *** andreas_s has quit IRC | 15:32 | |
| *** heyongli has joined #openstack-infra | 15:32 | |
| *** felipemonteiro has quit IRC | 15:32 | |
| mordred | clarkb: ppc seems like the hard one | 15:32 |
| *** felipemonteiro has joined #openstack-infra | 15:32 | |
| mordred | clarkb: for mac support, ansible needs that too - and there's a todo list item somewhere to add a macstadium driver | 15:32 |
| clarkb | mordred: arm hard too for kata | 15:33 |
| *** felipemonteiro_ has joined #openstack-infra | 15:33 | |
| mordred | yah | 15:33 |
| clarkb | it just feels like I've offered a free corolla trade in on a broken down geo and they are demanding a mercedes instead | 15:33 |
| mordred | yup | 15:34 |
| *** heyongli has quit IRC | 15:34 | |
| fungi | i keep hearing people make noise about donating ppc64 nova instances. maybe we need to call in their chips? | 15:34 |
| mordred | clarkb: well - thing is - there is nothing about zuul that has anything to do with arm or ppc | 15:34 |
| *** heyongli has joined #openstack-infra | 15:34 | |
| mordred | someone needs to provide arm and ppc build resources | 15:35 |
| clarkb | ya except people are using it to move goalposts | 15:35 |
| mordred | if they do - zuul can totally use them - same with OSX and macstadium | 15:35 |
| mordred | indeed they are | 15:35 |
| fungi | we've already done the hard part by adding arm64, and that was working out a multi-architecture model with architecture-specific nodepool builders | 15:35 |
| mordred | but- a response from our side can be that we can happily support both ppc and arm as long as someone can provide ppc and arm build resources | 15:36 |
| clarkb | mordred: and in fact do support arm64 just without nested virt thata kata would need | 15:37 |
| clarkb | (arm64 has no nested virt) | 15:37 |
| *** felipemonteiro has quit IRC | 15:37 | |
| fungi | _our_ arm64 resources have no nested virt support, or nested virt acceleration simply isn't a thing on arm64 in general? | 15:38 |
| clarkb | both | 15:39 |
| mordred | clarkb: well - what are they doing with arm support today? | 15:39 |
| clarkb | it is on ARMs roadmap for future but not yet in hardware | 15:39 |
| clarkb | mordred: thats the thing I dont think much | 15:39 |
| *** eharney has quit IRC | 15:39 | |
| clarkb | its limited third party testing aiui | 15:40 |
| mordred | right - what I mean is that they can't be doing much in the way of integration testing on arm today | 15:40 |
| mordred | so one would imagine that we could provide as much arm support as they currently have | 15:40 |
| clarkb | well they use baremetal | 15:41 |
| clarkb | to work around this aiui | 15:41 |
| mordred | we can do the same - zuul supports that | 15:41 |
| clarkb | which we could do too if given access | 15:41 |
| *** dpawlik has joined #openstack-infra | 15:41 | |
| mordred | yup | 15:41 |
| *** boden has joined #openstack-infra | 15:41 | |
| mordred | so I'd say arm for what they need is not a problem | 15:41 |
| mordred | how are they testing ppc? just travis? what do they do with it? like, just build/unittest? | 15:42 |
| boden | hi… how can I verify that a dependant patch has been pulled into a zuul job (e.g. py27) using Depends-On in the patch? I assume that’s logged somewhere? | 15:42 |
| *** pcaruana has quit IRC | 15:42 | |
| clarkb | mordred: I think it is all third party stuff from ibm for that | 15:43 |
| mordred | awesome. seems like we have them covered | 15:43 |
| clarkb | boden: look in zuul-info/inventory it will have that ibfo iirc | 15:43 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for latent cells v1 rebuild race bug 1709985 https://review.openstack.org/578129 | 15:44 |
| openstack | bug 1709985 in OpenStack Compute (nova) "test_rebuild_server_in_error_state randomly times out waiting for rebuilding instance to be active (cells v1)" [Low,Won't fix] https://launchpad.net/bugs/1709985 | 15:44 |
| mordred | clarkb: https://github.com/kata-containers/shim/pull/93/files <-- they're getting it from travis | 15:44 |
| *** trown has quit IRC | 15:44 | |
| frickler | boden: also check tox-siblings.log then | 15:44 |
| *** ramishra has quit IRC | 15:44 | |
| *** heyongli has quit IRC | 15:44 | |
| boden | clarkb I see it in inventory.. frickler I don’t see it in tox-siblings.txt | 15:45 |
| *** heyongli has joined #openstack-infra | 15:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add min_avail_hdd governor for zuul-executor https://review.openstack.org/577948 | 15:45 |
| clarkb | mordred ah | 15:45 |
| clarkb | so no virt testing there? or does travus allow that? | 15:46 |
| *** trown|brb has joined #openstack-infra | 15:46 | |
| frickler | boden: which patch? /me goes to enjoy evening traffic now, will check later | 15:46 |
| *** dpawlik has quit IRC | 15:46 | |
| boden | frickler https://review.openstack.org/#/c/577549/ is it a problem that it’s not in the siblings log? | 15:46 |
| *** pbourke has quit IRC | 15:47 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add min_avail_hdd governor for zuul-executor https://review.openstack.org/577948 | 15:47 |
| mordred | clarkb: travis uses containers for some things - but I doubt they're doing integration tests on travis | 15:48 |
| boden | based on what I see in other such patches, it doesn’t appear to ever show up in the siblings log | 15:48 |
| mordred | clarkb: indeed - they are doing make ; make test on travis - so they donm't need nested virt - they're only doing build/unittest tests | 15:49 |
| *** pbourke has joined #openstack-infra | 15:49 | |
| dmsimard | dhellmann: I've created a bug in pbr with a reproducer but I'm not sure it's a bug in pbr -- for some reason installing a package can take several minutes: https://bugs.launchpad.net/pbr/+bug/1778752 | 15:49 |
| openstack | Launchpad bug 1778752 in PBR "Package installation is slow when a large git submodule is inside the installation tree" [Undecided,New] | 15:49 |
| *** gyee has joined #openstack-infra | 15:50 | |
| *** dklyle has quit IRC | 15:50 | |
| *** sree has quit IRC | 15:51 | |
| *** ykarel|away has joined #openstack-infra | 15:51 | |
| mnaser | clarkb, mordred: they don't test ppc and arm. | 15:51 |
| *** sree has joined #openstack-infra | 15:51 | |
| mnaser | there's talks about adding that support but it is not a thing that works and we should not tie one to the other. | 15:51 |
| fungi | dmsimard: it's not strictly a bug in pbr. wheel (and consequently pip) copy the source to a separate build tree to as not to dirty the original source tree. this can be very i/o bound, especially for things like git repositories with lots of tiny object files | 15:52 |
| clarkb | mnaser: I'll leave a zuul comment reply in a few maybe you can followuo with ^ | 15:52 |
| mnaser | i replied with a short thing to address it. | 15:52 |
| mnaser | we're talking about replacing jenkins by zuul, not revamp the entire CI for kata. though i'm sure we welcome that if someone wants to do it | 15:53 |
| logan- | re: packethost mtu issues, i saw the same thing setting up limestone. i ended up piping jumbo frames to the compute hosts so we could run instances over vxlan at 1500 mtu. neutron advertises the correct mtu just fine to the instances so the interface itself gets set correctly, but jobs like osa and others that create bridges off the main interface seem to assume 1500 mtu almost always. i can work on adjusting osa to configure the bridges | 15:53 |
| logan- | differently if the 1450 mtu on gate nodes is going to stay that way. | 15:53 |
| dmsimard | fungi: I figured it would be something like that, >2 minutes on a nvme drive still surprises me | 15:53 |
| *** dklyle has joined #openstack-infra | 15:53 | |
| mordred | mnaser: yah. and we _can_ support all of those use cases | 15:53 |
| mnaser | mordred: right, i just think it's a bit frustrating when most of the work is being lead by someone not directly involved in the project (clarkb) and instead he has to take on that task :) | 15:53 |
| mordred | mnaser: there is nothing about zuul that will prevent those - it's purely about making sure buld resources exist | 15:54 |
| mordred | mnaser: TOTALLY | 15:54 |
| mnaser | sometimes i feel in open source people assume that helping out with something = volunteering to fix everything inside of it | 15:54 |
| fungi | dmsimard: a workaround may be just building sdist from source, and then generating the wheel from the sdist | 15:54 |
| mnaser | for arm, they can't do anything besides use physical infra (which someone is working on, so those can be static nodes) | 15:54 |
| mnaser | for ppc, they're using travis i guess, but probably not running integration tests, just unit tests / compiles | 15:55 |
| *** heyongli has quit IRC | 15:55 | |
| dmsimard | fungi: oh, there's plenty workarounds and it's not a blocker for me -- I just thought I'd report it in case someone more knowledgeable than I finds a low hanging fruit or something :)( | 15:55 |
| *** heyongli has joined #openstack-infra | 15:55 | |
| fungi | mnaser: for people not familiar with open source, i think the attitude is often "if you can't give me the same experience i get from a paid enterprise solution but for free and with no effort on my part, i'll just pay for the enterprise solution, thanks" | 15:56 |
| *** sree has quit IRC | 15:56 | |
| *** iranzo has quit IRC | 15:59 | |
| *** hamzy has quit IRC | 16:00 | |
| *** caphrim007 has joined #openstack-infra | 16:00 | |
| *** pguimaraes has quit IRC | 16:00 | |
| clarkb | I've responded as well | 16:01 |
| clarkb | logan-: I htink it would be a good idea to update our tooling to handle different MTUs even if we can assume 1500 because people run third party ci and deploy our software in places that may not have 1500 MTU | 16:02 |
| clarkb | I think it would be helpful if we can update osa to set lower MTUs and check if that fixes the problems on packethost | 16:03 |
| *** ykarel|away is now known as ykarel | 16:03 | |
| *** gyee has quit IRC | 16:03 | |
| logan- | ++ sounds good. i revived https://review.openstack.org/#/c/557484/ so we'll see if we can get that done in osa. its something i expect will affect most deployment projects... when i was watching limestone job logs w/ 1450 mtu there were a lot of deployment project fails | 16:04 |
| logan- | thanks | 16:04 |
| clarkb | I'm looking at devstack gate now | 16:04 |
| clarkb | we seem to configure neutron properly but then do our overlay poorly | 16:04 |
| *** heyongli has quit IRC | 16:05 | |
| *** heyongli has joined #openstack-infra | 16:05 | |
| clarkb | I don't think we set the overlay mtu properly in our zuulv3 native role | 16:06 |
| *** jesslampe has quit IRC | 16:07 | |
| clarkb | mordred: how do I set an ansible default variable value to undefined? | 16:09 |
| *** e0ne has quit IRC | 16:09 | |
| clarkb | 'foo:' and just leave the value side of key: value pair empty? | 16:09 |
| *** udesale has quit IRC | 16:09 | |
| mordred | clarkb: I'm not sure I understand | 16:10 |
| mordred | to make a variable undefined in defaults you just don't put it in the file | 16:10 |
| clarkb | but then it isn't clear that is a valid role var | 16:10 |
| mordred | that's what the docs are for | 16:10 |
| *** rossella_s has quit IRC | 16:10 | |
| clarkb | ya that works for the zuul role but less so for devstack-gate, maybe that is ok though as d-g should be going away | 16:10 |
| mordred | yah | 16:11 |
| mordred | I mean, you could also set it to null | 16:11 |
| mordred | foo: null - which would set its value to None | 16:11 |
| clarkb | let me get a patch up and we can see if I can do it better | 16:11 |
| mordred | cool | 16:12 |
| *** trown|brb is now known as trown | 16:12 | |
| clarkb | working on d-g side first because it is self testing | 16:12 |
| *** harlowja has joined #openstack-infra | 16:12 | |
| *** e0ne has joined #openstack-infra | 16:13 | |
| *** r-mibu has quit IRC | 16:14 | |
| *** rossella_s has joined #openstack-infra | 16:14 | |
| *** heyongli has quit IRC | 16:15 | |
| mandre | hello, is there a currently known failure that would cause the post ci task to fail or could it be related to my patch at https://review.openstack.org/#/c/574233/ ? I'm interested in the tripleo-ci-centos-7-scenario009-multinode-oooq job | 16:15 |
| *** heyongli has joined #openstack-infra | 16:15 | |
| *** jesslampe has joined #openstack-infra | 16:16 | |
| mandre | if I read the logs correctly, POST_RUN times out | 16:16 |
| mandre | http://logs.openstack.org/33/574233/21/experimental/tripleo-ci-centos-7-scenario009-multinode-oooq/714aecf/job-output.txt.gz#_2018-06-25_07_21_44_717609 | 16:16 |
| mandre | hmmm no, it's the job that timed out | 16:18 |
| mandre | I'm a bit confused because there is no usable log | 16:18 |
| dhellmann | dmsimard : I know we've had lots of exciting issues with git submodules and pbr in the past. It sounds like you've found a new one. Congratulations? :-) | 16:21 |
| dmsimard | dhellmann: oh I am definitely excited | 16:21 |
| dmsimard | :D | 16:21 |
| dhellmann | dmsimard : is it the installation step that takes a long time, or building the sdist? | 16:21 |
| dmsimard | dhellmann: this isn't urgent or anything, figured I'd take 5 minutes to write a reproducer to see if there was any low hanging fruit things | 16:21 |
| dhellmann | I would not be surprised to find that building the manifest for the sdist is slow | 16:21 |
| dmsimard | dhellmann: here's a record of the issue manifesting itself: https://asciinema.org/a/xKkrkrGoGiHVZqEZW6nTVnCXC | 16:22 |
| *** jesslampe has quit IRC | 16:22 | |
| dmsimard | (I should probably add that to the bug) | 16:22 |
| dmsimard | added | 16:23 |
| dhellmann | it looks like it's the egg_info command, but I wonder if that's also scanning all of the files | 16:23 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Dynamically determine overlay MTU https://review.openstack.org/578146 | 16:23 |
| dhellmann | yeah, "generating filelist from git" | 16:23 |
| fungi | if it's the bug i'm thinking of, building the sdist would be fast, building a wheel from the source tree (including such machinations as `pip install .`) take considerably longer, installing from sdist or pypi or prebuilt wheel are fast | 16:23 |
| clarkb | mordred: frickler ^ something like that I think. We will need to do similar for zuul-jobs/roles/multi-node-bridge | 16:23 |
| clarkb | mordred: I'd be curious if there is a better way to do what I've put into shell there in ansible | 16:24 |
| dmsimard | dhellmann: yeah, openshift-ansible is a mess of symlinks unfortunately.. even "find" and "ls" end up saying "hey there's too many recursive symlinks in there, bye" | 16:24 |
| fungi | dmsimard: dhellmann: so, yes, this sounds different | 16:24 |
| dhellmann | dmsimard : how long does it take to run "git ls-files -z" on that repo? | 16:25 |
| dhellmann | that's what we're doing at that step | 16:25 |
| dhellmann | pbr/git.py line 109 | 16:25 |
| dmsimard | dhellmann: instantaneous (on openshift-ansible itself) | 16:25 |
| *** heyongli has quit IRC | 16:25 | |
| *** heyongli has joined #openstack-infra | 16:26 | |
| dhellmann | odd | 16:26 |
| *** hashar is now known as hasharAway | 16:26 | |
| fungi | does `python setup.py sdist` take a long time? | 16:26 |
| dmsimard | instantaneous when checked out as a submodule too | 16:26 |
| fungi | and what about `python setup.py bdist_wheel` ? | 16:27 |
| mordred | clarkb: doubtful. I think doing it in shell is likely the more understandable version | 16:27 |
| clarkb | mordred: ok | 16:28 |
| *** esarault_ has quit IRC | 16:28 | |
| dmsimard | fungi: actually openshift-ansible hacked something together to sort of not make it installable (it's sort of a shock to look at python projects outside of the openstack community sometimes) | 16:28 |
| *** jpich has quit IRC | 16:28 | |
| *** jamesmca_ has quit IRC | 16:28 | |
| dmsimard | fungi: so sdist actually does nothing and bdist_wheel crashes | 16:28 |
| fungi | ouch | 16:29 |
| dmsimard | fungi: https://github.com/openshift/openshift-ansible/blob/ffab936876043ef4cf4abafb2f9bcbd8f605ba78/setup.py#L442-L448 | 16:29 |
| *** jamesmca_ has joined #openstack-infra | 16:29 | |
| fungi | i'm not inclined to take packaging bug reports seriously from projects which have intentionally broken their packaging | 16:29 |
| mordred | dmsimard: yup. there is definitely a reason we stopped having setup.py code in openstack | 16:30 |
| dmsimard | Aye, it's a bit unfortunate. I'd like to take a shot at fixing their things one day | 16:30 |
| mordred | it's prettymuch always a terrible idea | 16:30 |
| dmsimard | mordred: it's awful :( | 16:30 |
| fungi | 1. how do you know it's not related to the way you've knowingly broken your packaging and 2. even if it is a bug and gets fixed are you actually going to benefit from the fix in any meaningful way? | 16:30 |
| dmsimard | fungi: agreed, I found the setup.py blacklisting after the fact | 16:31 |
| dmsimard | my reaction: | 16:31 |
| dmsimard | (╯°□°）╯︵ ┻━┻ | 16:31 |
| *** pguimaraes has joined #openstack-infra | 16:32 | |
| fungi | that is one of the more insane setup.py files i have seen in my life, and i've seen a lot of fairly insane ones | 16:32 |
| dmsimard | fungi: sorry about that :P | 16:32 |
| fungi | not your fault, you didn't write it! | 16:32 |
| dmsimard | fungi: (yes, I have that bookmarked) https://www.cbc.ca/2017/sorry-can-we-talk-about-why-canadians-apologize-so-much-1.3939997 | 16:33 |
| *** jamesmca_ has quit IRC | 16:34 | |
| clarkb | heh reviewing your own change and realizing you invented a new word | 16:34 |
| *** e0ne has quit IRC | 16:35 | |
| *** heyongli has quit IRC | 16:36 | |
| *** heyongli has joined #openstack-infra | 16:36 | |
| *** janki has joined #openstack-infra | 16:39 | |
| *** tesseract has quit IRC | 16:40 | |
| *** jesslampe has joined #openstack-infra | 16:43 | |
| openstackgerrit | Merged openstack-infra/project-config master: Test log-inventory role using base-test https://review.openstack.org/563788 | 16:45 |
| *** heyongli has quit IRC | 16:46 | |
| *** heyongli has joined #openstack-infra | 16:46 | |
| *** rossella_s has quit IRC | 16:48 | |
| *** ramishra has joined #openstack-infra | 16:49 | |
| *** CrayZee has joined #openstack-infra | 16:49 | |
| *** lpetrut has quit IRC | 16:50 | |
| *** CrayZee has quit IRC | 16:51 | |
| *** rossella_s has joined #openstack-infra | 16:53 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Dynamically determine overlay network mtu https://review.openstack.org/578153 | 16:55 |
| clarkb | mordred: if the d-g change works ^ is the equivalent change in zuul-jobs | 16:55 |
| mordred | clarkb: woot | 16:55 |
| clarkb | testing that may be interesting though d-g gets us most of the way there I think | 16:56 |
| *** heyongli has quit IRC | 16:56 | |
| *** heyongli has joined #openstack-infra | 16:56 | |
| *** yamahata has joined #openstack-infra | 16:57 | |
| *** derekh has quit IRC | 16:58 | |
| *** ramishra has quit IRC | 16:58 | |
| *** dhajare has quit IRC | 17:00 | |
| *** panda is now known as panda|off | 17:01 | |
| *** Guest33625 has quit IRC | 17:02 | |
| *** sambetts is now known as sambetts|afk | 17:04 | |
| *** heyongli has quit IRC | 17:06 | |
| *** heyongli has joined #openstack-infra | 17:07 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add min_avail_hdd governor for zuul-executor https://review.openstack.org/577948 | 17:07 |
| *** zoli is now known as zoli|gone | 17:09 | |
| *** zoli|gone is now known as zoli | 17:09 | |
| *** harlowja has quit IRC | 17:10 | |
| *** e0ne has joined #openstack-infra | 17:11 | |
| *** yamamoto has quit IRC | 17:12 | |
| *** Hal has joined #openstack-infra | 17:13 | |
| *** Hal is now known as Guest34560 | 17:14 | |
| *** felipemonteiro_ has quit IRC | 17:14 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:14 | |
| *** yamamoto has joined #openstack-infra | 17:14 | |
| *** dtantsur is now known as dtantsur|afk | 17:15 | |
| *** heyongli has quit IRC | 17:17 | |
| *** heyongli has joined #openstack-infra | 17:17 | |
| *** e0ne has quit IRC | 17:18 | |
| *** gyee has joined #openstack-infra | 17:20 | |
| *** yamamoto has quit IRC | 17:20 | |
| *** jcoufal has quit IRC | 17:22 | |
| fungi | gonna go grab some lunch, back shortly (well in time for the meeting) | 17:24 |
| clarkb | oh right that is today | 17:24 |
| clarkb | mnaser: have you been able to check if mariadb proxy is working now? | 17:26 |
| frickler | boden: to me it looks like http://logs.openstack.org/49/577549/1/check/openstack-tox-py27/3797219/tox/py27-siblings.txt.gz does update neutron like it would be expected, so your depends-on should work fine | 17:26 |
| *** heyongli has quit IRC | 17:27 | |
| *** heyongli has joined #openstack-infra | 17:27 | |
| *** Guest34560 has quit IRC | 17:28 | |
| *** jesslampe has quit IRC | 17:31 | |
| *** jesslampe has joined #openstack-infra | 17:32 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/infra-manual master: Add note about test env MTU variance https://review.openstack.org/578159 | 17:32 |
| clarkb | fungi: ^ that is doc update re MTU | 17:32 |
| mnaser | clarkb: i didn't want to bother you but it's not working :( | 17:33 |
| *** dpawlik has joined #openstack-infra | 17:34 | |
| mnaser | http://downloads.mariadb.com/MariaDB/mariadb-10.1.33/yum/centos7-amd64/ vs http://mirror.ca-ymq-1.vexxhost.openstack.org:8080/MariaDB/mariadb-10.1.33/yum/centos7-amd64/ | 17:34 |
| mnaser | second link 404s | 17:34 |
| *** dpawlik has quit IRC | 17:35 | |
| *** dpawlik has joined #openstack-infra | 17:36 | |
| *** heyongli has quit IRC | 17:37 | |
| clarkb | mnaser: bah its my bug | 17:37 |
| *** heyongli has joined #openstack-infra | 17:37 | |
| pabelanger | I think you need trailing slash | 17:38 |
| *** Swami has joined #openstack-infra | 17:39 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Fix trailing slashes on mariadb caching proxy https://review.openstack.org/578160 | 17:39 |
| clarkb | pabelanger: yup | 17:39 |
| *** Swami_ has joined #openstack-infra | 17:39 | |
| pabelanger | +2 | 17:39 |
| *** slaweq has quit IRC | 17:40 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Dynamically determine overlay network mtu https://review.openstack.org/578153 | 17:43 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Dynamically determine overlay MTU https://review.openstack.org/578146 | 17:44 |
| *** felipemonteiro__ has joined #openstack-infra | 17:44 | |
| *** ykarel has quit IRC | 17:44 | |
| *** pguimaraes has quit IRC | 17:47 | |
| *** heyongli has quit IRC | 17:47 | |
| *** heyongli has joined #openstack-infra | 17:48 | |
| *** felipemonteiro_ has quit IRC | 17:48 | |
| *** pguimaraes has joined #openstack-infra | 17:48 | |
| *** andreas_s has joined #openstack-infra | 17:48 | |
| *** jpena is now known as jpena|off | 17:49 | |
| *** felipemonteiro__ has quit IRC | 17:49 | |
| *** felipemonteiro__ has joined #openstack-infra | 17:49 | |
| *** hamzy has joined #openstack-infra | 17:50 | |
| *** electrofelix has quit IRC | 17:52 | |
| *** andreas_s has quit IRC | 17:53 | |
| *** felipemonteiro has joined #openstack-infra | 17:53 | |
| *** felipemonteiro__ has quit IRC | 17:54 | |
| *** amoralej is now known as amoralej|off | 17:56 | |
| *** heyongli has quit IRC | 17:58 | |
| *** heyongli has joined #openstack-infra | 17:58 | |
| *** rossella_s has quit IRC | 17:58 | |
| *** felipemonteiro has quit IRC | 18:01 | |
| *** felipemonteiro has joined #openstack-infra | 18:01 | |
| AJaeger | infra-root, here's a simple infra-manual change to remove mention of a now unused job - could you review, please? https://review.openstack.org/574858 | 18:02 |
| *** rossella_s has joined #openstack-infra | 18:02 | |
| *** gary_perkins has quit IRC | 18:03 | |
| clarkb | AJaeger: done | 18:04 |
| *** trown has quit IRC | 18:05 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:08 | |
| *** heyongli has quit IRC | 18:08 | |
| *** gary_perkins has joined #openstack-infra | 18:08 | |
| *** heyongli has joined #openstack-infra | 18:08 | |
| *** gfidente has quit IRC | 18:08 | |
| *** ihar has joined #openstack-infra | 18:09 | |
| AJaeger | thanks, clarkb | 18:09 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Add note about test env MTU variance https://review.openstack.org/578159 | 18:09 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Dynamically determine overlay network mtu https://review.openstack.org/578153 | 18:11 |
| *** felipemonteiro has quit IRC | 18:11 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Dynamically determine overlay MTU https://review.openstack.org/578146 | 18:12 |
| openstackgerrit | Merged openstack-infra/infra-manual master: remove reference to publish-openstack-python-tarball before deleting it https://review.openstack.org/574858 | 18:13 |
| *** jaosorior has joined #openstack-infra | 18:13 | |
| AJaeger | dmsimard: can you remove your -1 from https://review.openstack.org/#/c/563790/ , please? | 18:13 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Dynamically determine overlay network mtu https://review.openstack.org/578153 | 18:13 |
| AJaeger | infra-root, or should we ignore the -1 on https://review.openstack.org/#/c/563790/ since the concern has been addressed? | 18:14 |
| dmsimard | AJaeger: done | 18:14 |
| AJaeger | thanks, dmsimard ! | 18:14 |
| clarkb | gol! | 18:14 |
| AJaeger | clarkb: regarding log-inventory - which change should merge next? | 18:15 |
| *** eharney has joined #openstack-infra | 18:16 | |
| *** yamamoto has joined #openstack-infra | 18:16 | |
| clarkb | AJaeger: I think we want to merge a chang that uses the new role in our base job, then we can remove the task from the other role whenever deprecation is ready | 18:16 |
| AJaeger | clarkb: deprecation is ready - was announced for yesterday | 18:18 |
| clarkb | cool, I think we add the new role to our base job then remove the task from the existing role to complete the deprecation then | 18:18 |
| *** heyongli has quit IRC | 18:18 | |
| *** heyongli has joined #openstack-infra | 18:19 | |
| AJaeger | clarkb: https://review.openstack.org/#/c/563789/ ? | 18:19 |
| AJaeger | clarkb: or want to test first since https://review.openstack.org/#/c/563788 is merged? I don't see where that is tested... | 18:19 |
| *** yamamoto has quit IRC | 18:20 | |
| clarkb | AJaeger: https://review.openstack.org/#/c/563788/2 is what would test it, we have to run a job with the base-test job as parent | 18:20 |
| *** yamamoto has joined #openstack-infra | 18:21 | |
| *** trown has joined #openstack-infra | 18:23 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: remove publish-openstack-python-tarball https://review.openstack.org/574860 | 18:24 |
| *** dpawlik has quit IRC | 18:28 | |
| *** dpawlik has joined #openstack-infra | 18:28 | |
| *** rossella_s has quit IRC | 18:28 | |
| *** heyongli has quit IRC | 18:28 | |
| *** heyongli has joined #openstack-infra | 18:29 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix trailing slashes on mariadb caching proxy https://review.openstack.org/578160 | 18:32 |
| *** dmellado has quit IRC | 18:32 | |
| *** gouthamr has quit IRC | 18:32 | |
| *** rossella_s has joined #openstack-infra | 18:33 | |
| *** stevebaker has quit IRC | 18:33 | |
| *** slaweq has joined #openstack-infra | 18:34 | |
| *** bhavik1 has joined #openstack-infra | 18:34 | |
| fungi | dpawlik: if you want to discuss octavia job failures on ovh, it's probably more productive to do so in here than in irc private messages | 18:35 |
| fungi | there are more people in here familiar with the situation than me | 18:35 |
| dpawlik | fungi: ok | 18:37 |
| fungi | and sorry for the silence earlier, i had gone out to get lunch | 18:37 |
| dpawlik | fungi: I will check host configuration and maybe I will be able to reproduce Octavia problem on some my beta zone infra | 18:38 |
| *** bhavik1 has quit IRC | 18:38 | |
| clarkb | long story short is we use the ssd-osFoundation-3 flavor to land our test VMs on dedicated hypervisors aiui | 18:38 |
| clarkb | this way we don't impact customers | 18:38 |
| dpawlik | it is sad that it is not possible to get hostid or on which region it was executed | 18:38 |
| clarkb | I think we can pull one of the hypervisors out of that pool then launch VMs on it directly to test on we can make modifications to the hypervisor a bit more freely | 18:39 |
| dpawlik | exactly :) | 18:39 |
| *** heyongli has quit IRC | 18:39 | |
| *** heyongli has joined #openstack-infra | 18:39 | |
| dpawlik | clarkb: maybe if I'm not able to reproduce it on bz, I will use your suggenstion :) | 18:39 |
| johnsom | It is a pretty simple test, install devstack, boot a cirros with kvm enabled, check the qemu log and or check if cirros boots | 18:41 |
| clarkb | johnsom: any idea if we can reduce that to a virsh define && virsh start? | 18:42 |
| clarkb | (would probably reduce turn around time on testing if we can) | 18:42 |
| johnsom | Yeah, I think you can boot it directly. | 18:43 |
| *** florianf is now known as florianf|afk | 18:45 | |
| johnsom | The two parts we should check too is using the nodepool image and pulling those cloud archive packages that devstack installs. I am not sure if those are related or not. | 18:45 |
| clarkb | my hunch is the nodepool image shouldn't be related, we install the same kernel using the same packages as the upstream images | 18:46 |
| clarkb | but the qemu/libvirt versions could affect it | 18:46 |
| johnsom | Btw, I am open to doing grunt work on this if you need cycles for testing | 18:47 |
| *** kjackal has quit IRC | 18:48 | |
| *** heyongli has quit IRC | 18:49 | |
| *** heyongli has joined #openstack-infra | 18:49 | |
| dpawlik | johnsom: it is on xenial or centos? | 18:51 |
| dpawlik | or there is no difference :P ? | 18:51 |
| johnsom | dpawlik: I am not sure on centos. If I remember right it doesn’t matter. The gate job I have uses xenial | 18:53 |
| johnsom | The guest image doesn’t matter. Cirros, ubuntu, and centos all crash | 18:53 |
| johnsom | Actually, looking back we have never tried a centos nodepool image | 18:55 |
| clarkb | meeting time in #openstack-meeting | 18:59 |
| *** heyongli has quit IRC | 18:59 | |
| *** heyongli has joined #openstack-infra | 18:59 | |
| *** jamesmcarthur_ has quit IRC | 19:01 | |
| *** jamesmcarthur has quit IRC | 19:01 | |
| *** kjackal has joined #openstack-infra | 19:02 | |
| boden | hi, when networking-l2gw is installed from source (git), it installs as version 11.0.1.dev49, but its last release version is 12.0.1 (https://pypi.org/project/networking-l2gw/) which is > than it’s dev version… I find this not to be the case with other projects (ex: neutron installs as 13.0.0.0b3.dev64 and its last release is 12.0.3)… anyone know why this would be (e.g. networking-l2gw seems to have a backleveled version in g | 19:02 |
| boden | master)? | 19:02 |
| *** janki has quit IRC | 19:04 | |
| *** yolanda__ has joined #openstack-infra | 19:04 | |
| *** yolanda_ has quit IRC | 19:07 | |
| *** yolanda_ has joined #openstack-infra | 19:09 | |
| *** sshnaidm has quit IRC | 19:09 | |
| *** heyongli has quit IRC | 19:09 | |
| *** heyongli has joined #openstack-infra | 19:10 | |
| dpawlik | johnsom: do you have some script for executing the test? | 19:10 |
| johnsom | dpawlik No, I just boot a VM using nova, but you could boot it directly I think. | 19:11 |
| johnsom | dpawlik the key is to have the accel=kvm | 19:12 |
| *** yolanda__ has quit IRC | 19:12 | |
| dpawlik | johnsom: ack | 19:13 |
| *** felipemonteiro_ has quit IRC | 19:15 | |
| *** yolanda__ has joined #openstack-infra | 19:15 | |
| *** slaweq has quit IRC | 19:15 | |
| clarkb | boden: if using pbr it is generated from the last tag visible from the branch | 19:17 |
| clarkb | boden: you can run git describe to see what git thinks that is | 19:17 |
| *** yolanda_ has quit IRC | 19:18 | |
| *** felipemonteiro has joined #openstack-infra | 19:19 | |
| dpawlik | johnsom: Im wondering if the same problem is not on devstack | 19:19 |
| dpawlik | if the main issue is with accel=kvm | 19:19 |
| boden | clarkb thanks.. it seems wrong for l2gw IIUC.. any idea how I get it fixed? | 19:19 |
| clarkb | boden: in the past we did a weird merge into master from branches where newer tags happened so that master saw them | 19:20 |
| *** heyongli has quit IRC | 19:20 | |
| clarkb | basically a git merge that ties history together without changing any code | 19:20 |
| *** heyongli has joined #openstack-infra | 19:20 | |
| dpawlik | johnsom: I have to leave soon... Tomorrow I will focus on that problem | 19:20 |
| johnsom | dpawlik Yes, likely. accel=kvm is how qemu enables the use of KVM for virtualization. The other option that we use here is accel=tcg which is software emulation and does not use KVM. | 19:20 |
| dpawlik | clarkb, fungi: if you have some problems with OVH infra and there will be some problems to fix that, just ping me on irc | 19:21 |
| boden | clarkb this is what I end up getting as a result: pkg_resources.VersionConflict: (networking-l2gw 11.0.1.dev49 (/home/boden/src/networking-midonet/.tox/pep8-dev/src/networking-l2gw), Requirement.parse('networking-l2gw>=12.0.0')) | 19:21 |
| johnsom | dpawlik I will be around tomorrow, so feel free to ping me | 19:21 |
| boden | not really sure how to address it TBH | 19:21 |
| dpawlik | johnsom: BTW very interesting features :D Im wondering if its not related to https://bugzilla.redhat.com/show_bug.cgi?id=1565179 | 19:22 |
| openstack | bugzilla.redhat.com bug 1565179 in qemu-kvm-rhev "qemu-kvm fails when run with -machine accel=kvm:tcg nested KVM" [Unspecified,New] - Assigned to pbonzini | 19:22 |
| dpawlik | have a nice day | 19:22 |
| clarkb | dpawlik: thank you! | 19:24 |
| *** slaweq has joined #openstack-infra | 19:24 | |
| clarkb | boden: 12.0 was tagged on stable/queens and not on master so master doesn't see that tag | 19:24 |
| boden | clarkb thx, and how can I fix it? sorry its a bit out of my scope ATM | 19:28 |
| clarkb | boden: we should check with the release team to see what their suggestion is. A 13.0 tag on master or merging stable/queens into master in that special manner will fix it I think | 19:29 |
| boden | clarkb ok thx.. I asked over on that channel, but smcginnis wasn’t sure | 19:29 |
| *** heyongli has quit IRC | 19:30 | |
| *** heyongli has joined #openstack-infra | 19:30 | |
| *** eernst has joined #openstack-infra | 19:37 | |
| smcginnis | clarkb: Do you know how the pbr -dev versioning works? | 19:37 |
| smcginnis | I'm assuming if they do a release on master, the dev version will pick up that and show a better interim version. | 19:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add zuul.child_jobs in ansible inventory file https://review.openstack.org/578181 | 19:38 |
| *** agopi is now known as agopi|brb | 19:40 | |
| *** heyongli has quit IRC | 19:40 | |
| *** heyongli has joined #openstack-infra | 19:41 | |
| *** dhill_ has quit IRC | 19:42 | |
| *** agopi|brb has quit IRC | 19:45 | |
| *** lpetrut has joined #openstack-infra | 19:46 | |
| *** hasharAway is now known as hashar | 19:50 | |
| *** sshnaidm has joined #openstack-infra | 19:50 | |
| mnaser | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64894&rra_id=all | 19:50 |
| *** lpetrut has quit IRC | 19:50 | |
| *** heyongli has quit IRC | 19:50 | |
| mnaser | i was going to ask if that was bytes or bits per second | 19:51 |
| *** heyongli has joined #openstack-infra | 19:51 | |
| mnaser | but it was clearly bits per second on the left | 19:51 |
| mnaser | tripleo job failed here on our cloud -- http://logs.openstack.org/58/571958/2/gate/tripleo-ci-centos-7-containers-multinode/763b88f/logs/devstack-gate-setup-host.txt -- i'm trying to figure out why thats happening | 19:51 |
| mnaser | so any logs in the vm might be useful | 19:51 |
| *** priteau has quit IRC | 19:52 | |
| *** slaweq has quit IRC | 19:52 | |
| *** e0ne has joined #openstack-infra | 19:52 | |
| mtreinish | fungi: hmm, on the cyrus lpmqtt fix it looks like you missed a closing '}': http://logs.openstack.org/95/577895/1/check/infra-puppet-apply-3-ubuntu-xenial/c72e070/applytest/puppetapplytest26.final.out.FAILED | 19:54 |
| *** gouthamr has joined #openstack-infra | 19:54 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Check "cyrus" Exim router before "localuser" https://review.openstack.org/577895 | 19:55 |
| mtreinish | fungi: ^^^ should fix it | 19:55 |
| *** zul has joined #openstack-infra | 19:56 | |
| *** jamesmcarthur has joined #openstack-infra | 19:56 | |
| *** hemna_ has quit IRC | 19:56 | |
| ianw | clarkb / fungi : debian-security has been more of a pain than i hoped ... would be great if you could look at https://review.openstack.org/#/c/578057/ for sanity, because it's in a half-broken state | 19:59 |
| *** eernst has quit IRC | 20:00 | |
| *** heyongli has quit IRC | 20:01 | |
| *** heyongli has joined #openstack-infra | 20:01 | |
| clarkb | smcginnis: it looks at the git state and applies semver rules to generate a future version that it attaches .dev to | 20:01 |
| clarkb | smcginnis: in this case master has tag 11.0.0 as its most recent tag so pbg says the next version is 11.0.1 at a minimum then attaches .dev$numberofcommitssincetag to that | 20:01 |
| clarkb | smcginnis: this breaks if you tag stable with higher version than master | 20:02 |
| smcginnis | clarkb: So it looks at the current branches history to find the last tagged commit to get the version from? | 20:02 |
| clarkb | smcginnis: ya it just runs `git describe` for that | 20:02 |
| clarkb | I am not caught up on how openstack deals with this because we stopped merging stable back to master I think | 20:02 |
| smcginnis | clarkb: OK, thanks. So the advice to them that they need to create a new release from master seems to be correct. | 20:02 |
| fungi | thanks mtreinish! | 20:03 |
| clarkb | or merging stable into master so that master sees the newer tag would work too. It is what openstack did for a long time then stopped in favor of something else I am not up to date on | 20:03 |
| smcginnis | clarkb: We do stable releases off of the stable branches, so I would assume it would just look back on that branches history and find the last relevant release. | 20:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add zuul.child_jobs in ansible inventory file https://review.openstack.org/578181 | 20:04 |
| smcginnis | And we create the stable branch off of the new release on master, so master should always have the last non-stable release. | 20:04 |
| fungi | merging the major release tag from stable into master caused headaches for calculating release notes | 20:04 |
| clarkb | smcginnis: got it | 20:04 |
| smcginnis | fungi: Ah, I can imagine. | 20:04 |
| *** rossella_s has quit IRC | 20:04 | |
| clarkb | in that case a new tag on master for l2gw is probably in order | 20:05 |
| smcginnis | Since this repo is not under governance they have been doing their own releases, so I'm not going to worry too much about seeing how they've been doing things up until this point. | 20:05 |
| clarkb | possibly a beta tag if we don't want a real release | 20:05 |
| *** rossella_s has joined #openstack-infra | 20:05 | |
| smcginnis | clarkb: Yeah, that seems like that should be enough to get things right. | 20:05 |
| *** dhill_ has joined #openstack-infra | 20:08 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config master: Add Grafana dashboards for stable Neutron releases https://review.openstack.org/578191 | 20:09 |
| clarkb | ianw: +2 and I concur reprepro is a lot of throw stuff at wall until it works :/ | 20:10 |
| *** hashar has quit IRC | 20:10 | |
| clarkb | ok lunch now back in a bit | 20:10 |
| *** hashar has joined #openstack-infra | 20:11 | |
| *** heyongli has quit IRC | 20:11 | |
| *** heyongli has joined #openstack-infra | 20:11 | |
| *** e0ne has quit IRC | 20:15 | |
| *** john_studarus_ has joined #openstack-infra | 20:17 | |
| *** jamesmcarthur has quit IRC | 20:17 | |
| *** noama has joined #openstack-infra | 20:18 | |
| john_studarus_ | so I'm leaning towards turning off "generic-receive-offload" for the Packet cloud to resolve this checksum issue - discussion? | 20:18 |
| john_studarus_ | @mnaser ^^ | 20:19 |
| *** gouthamr has quit IRC | 20:19 | |
| clarkb | john_studarus_: mnaser found that osa does some specific iptables checksum options which may be related also could be mtu related | 20:19 |
| john_studarus_ | mtu is 1500 and 1450 | 20:20 |
| clarkb | we may want to coordinate a bit and change one thing at a time to avoid fixing it and thinking one thing fixed it but reality is multiple.changes happened | 20:20 |
| mnaser | i agree | 20:20 |
| mnaser | i vote first that we go to jumbo frames (if you can) and set the network mtu to 1500 | 20:20 |
| mnaser | that might take care of neutron's issues in a very straight forward way | 20:20 |
| *** heyongli has quit IRC | 20:21 | |
| john_studarus_ | so the mtu is already 1500 on the physical interfaces | 20:21 |
| *** heyongli has joined #openstack-infra | 20:22 | |
| clarkb | mnaser: john_studarus_ I have changes in fkight to accomodate the smaller VM mtus | 20:22 |
| mnaser | right, so neutron creates 1450 mtu networks to make sure things go through pipes properly | 20:22 |
| john_studarus_ | ok - I'll sit on my hands... | 20:22 |
| *** gouthamr has joined #openstack-infra | 20:22 | |
| mnaser | when i say neutron here, i mean neutron in packet | 20:22 |
| john_studarus_ | yeah, all the virtual networks have an mtu of 1450 | 20:23 |
| clarkb | john_studarus_: mnaser means increasing the VM mtus to 1500 which requires jumbo frames on the hypervisors | 20:23 |
| john_studarus_ | ah - got it... | 20:23 |
| clarkb | if this isnt possible w will just have to accomodate that | 20:23 |
| mnaser | does packet support 9000 mtu | 20:24 |
| john_studarus_ | let me check on the 9000 mtu... | 20:24 |
| clarkb | my changes stop assuming mtu sizes and account fkr vxlan on top of smallest present mtu | 20:24 |
| clarkb | should be flexible this way | 20:24 |
| john_studarus_ | as long as we stay in the same datacenter, we're ok with 9000 | 20:25 |
| john_studarus_ | everything is in the same datacenter | 20:25 |
| mnaser | so step #1 would be bumping the mtu to 9000 in all the compute+network nodes | 20:25 |
| mnaser | and control plane | 20:25 |
| clarkb | and setting vM mtu to 1500 | 20:26 |
| mnaser | yep, step #2 would be changing the mtu of the openstack ci network to 1500, and afaik mtu is read-only or can only be updated on create.. but i might be out of date on that | 20:26 |
| mnaser | but we can work around that | 20:26 |
| john_studarus_ | am I waiting on anything before this change? | 20:27 |
| logan- | you will also need to set neutron.conf global_physnet_mtu as well I think | 20:27 |
| logan- | once the hosts have 9k mtu set on the interfaces | 20:27 |
| clarkb | john_studarus_: no I think you can make that change whenever you are ready | 20:28 |
| logan- | I have it set to 1550 in the limestone CI cloud, which makes vxlan networks get 1500 byte mtus automatically | 20:28 |
| logan- | also hello john_studarus_ :) | 20:28 |
| john_studarus_ | what's the neutron change required? | 20:28 |
| john_studarus_ | what's the neutron.conf change required? | 20:28 |
| logan- | neutron.conf [DEFAULT] section set global_physnet_mtu = 1550 | 20:29 |
| *** gouthamr_ has joined #openstack-infra | 20:29 | |
| *** gouthamr has quit IRC | 20:29 | |
| *** gouthamr_ is now known as gouthamr | 20:29 | |
| john_studarus_ | are we going with 9000 or 1550? | 20:29 |
| logan- | then re-create the vxlan network and you should see a 1500 byte mtu on the network | 20:29 |
| fungi | mnaser: catching up, but we forewarned that checksum validation in software can end up being a problem if you also have checksum offloading enabled in hardware/driver since the checksum you see won't necessarily match the original. any idea why osa is enabling that in iptables? | 20:30 |
| *** dpawlik has quit IRC | 20:30 | |
| mnaser | fungi: i forgot to ask in todays meeting unfortunately, i asked but no initial response | 20:30 |
| logan- | 9k on the underlying interfaces is pretty standard for jumbo frames but it really doesn't matter as long as its big enough to support 1500 byte packet + vxlan overhead. so id go with 9k on the physical interfaces, tell neutron 1550 physnet mtu, and then any vxlan networks you create will get normal 1500 mtus for the instances attached to them | 20:31 |
| *** heyongli has quit IRC | 20:31 | |
| *** egonzalez has joined #openstack-infra | 20:31 | |
| fungi | yeah, when i've done jf deployments in the past we did 9k since that's what a lot of hardware vendors default to when you ask for jf | 20:32 |
| *** heyongli has joined #openstack-infra | 20:32 | |
| *** jcoufal has joined #openstack-infra | 20:32 | |
| fungi | some simpler networking device operating systems used to just have standard and jumbo settings for ethernet segments and didn't let you fine-tune specifici frame sizes | 20:33 |
| *** jamesmcarthur has joined #openstack-infra | 20:34 | |
| john_studarus_ | hello logan-! | 20:36 |
| john_studarus_ | ah - so the jumbo frames won't leave the data center which means no Internet - that isn't going to fly... | 20:37 |
| egonzalez | hi, anyone know why this change is running jobs in a loop, ends the execution then queue them again https://review.openstack.org/#/c/578170/ | 20:39 |
| openstackgerrit | Merged openstack-infra/zuul master: Support paging when listing github installations https://review.openstack.org/578077 | 20:39 |
| logan- | as long as you just have jumbo frames between the network gateways and the compute hosts it is acceptable | 20:39 |
| egonzalez | and possibly kill the execution until i can find a reason/fix to it | 20:40 |
| logan- | the main thing is allowing for packet sizes >1500 between the neutron gateways and the compute hosts | 20:40 |
| *** heyongli has quit IRC | 20:42 | |
| john_studarus_ | I increased one of the compute nodes to 9000 and the network dropped - even from the controller node | 20:42 |
| *** heyongli has joined #openstack-infra | 20:42 | |
| fungi | all the devices on the same layer 2 broadcast domain which will be communicating with one another need to be set to a similar mtu, and the frame size needs to be supported by the ethernet hardware whatever that might be (bridges/switches...) | 20:43 |
| *** jcoufal has quit IRC | 20:45 | |
| fungi | so with, say, vxlan, whatever's providing the underlying ethernet pathways would be set to jumbo frames but then the encapsulated vxlan segments would be set to 1500 mtu and whatever is decapsulating them would be jumbo on the underlying broadcast domain but then probably 1500 on unencapsulated interfaces not participating in vxlan | 20:45 |
| fungi | essentially the separation between jumbo and not-jumbo happens at layer-3 boundaries and/or gets encapsulated | 20:46 |
| *** hamzy has quit IRC | 20:47 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add /etc/localtime to bubblewrap default ro bind https://review.openstack.org/578072 | 20:48 |
| openstackgerrit | Merged openstack-infra/zuul master: Improve test case test_unprotected_branches https://review.openstack.org/576536 | 20:51 |
| *** stevebaker has joined #openstack-infra | 20:52 | |
| *** heyongli has quit IRC | 20:52 | |
| *** heyongli has joined #openstack-infra | 20:52 | |
| *** john_studarus_ has quit IRC | 20:53 | |
| openstackgerrit | Merged openstack-infra/zuul master: Move exclude unprotected branches check into tenant https://review.openstack.org/576537 | 20:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Consume Task and TaskManager from openstacksdk https://review.openstack.org/414759 | 20:56 |
| corvus | clarkb, fungi, mordred: it looks like zuul has not been restarted since before i left (so the new pipeline manager is not yet available), does that sound about right? | 20:57 |
| clarkb | corvus: yes | 20:58 |
| fungi | correct | 21:01 |
| *** trown is now known as trown|outtypewww | 21:01 | |
| *** zul has quit IRC | 21:02 | |
| *** heyongli has quit IRC | 21:02 | |
| *** heyongli has joined #openstack-infra | 21:03 | |
| *** rossella_s has quit IRC | 21:06 | |
| *** rossella_s has joined #openstack-infra | 21:09 | |
| *** felipemonteiro has quit IRC | 21:10 | |
| *** egonzalez has quit IRC | 21:12 | |
| *** kgiusti has left #openstack-infra | 21:12 | |
| *** yolanda_ has joined #openstack-infra | 21:12 | |
| *** heyongli has quit IRC | 21:12 | |
| *** heyongli has joined #openstack-infra | 21:13 | |
| corvus | well, we're under capacity. how about we fix that and restart now? | 21:13 |
| *** yolanda__ has quit IRC | 21:14 | |
| corvus | i don't think we need executor restarts; shall i just do the scheduler? | 21:14 |
| clarkb | ya should just be executor | 21:15 |
| clarkb | I haven't seen any executor specific changes we need | 21:15 |
| corvus | scheduler | 21:15 |
| clarkb | er ya | 21:15 |
| *** Hal has joined #openstack-infra | 21:16 | |
| *** Hal is now known as Guest12154 | 21:17 | |
| *** zaneb has quit IRC | 21:17 | |
| *** shaner has quit IRC | 21:18 | |
| *** shaner has joined #openstack-infra | 21:20 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/lpmqtt master: Never give up never surrender https://review.openstack.org/578211 | 21:22 |
| *** cmurphy_vacation is now known as cmurphy | 21:22 | |
| *** heyongli has quit IRC | 21:23 | |
| *** heyongli has joined #openstack-infra | 21:23 | |
| *** Guest12154 has quit IRC | 21:23 | |
| fungi | scheduler restart seems prudent | 21:24 |
| fungi | then we can use the preemptive pipeline | 21:24 |
| corvus | supercedent | 21:25 |
| *** gouthamr has quit IRC | 21:25 | |
| *** gouthamr has joined #openstack-infra | 21:26 | |
| fungi | er, yes that one ;) | 21:26 |
| fungi | cdent would be particularly fond of a supercdent pipeline if he lurked in here | 21:26 |
| fungi | his loss! | 21:26 |
| *** noama has quit IRC | 21:28 | |
| *** zaneb has joined #openstack-infra | 21:29 | |
| *** heyongli has quit IRC | 21:33 | |
| *** heyongli has joined #openstack-infra | 21:33 | |
| *** numans_ has joined #openstack-infra | 21:34 | |
| *** numans has quit IRC | 21:37 | |
| *** lifeless has joined #openstack-infra | 21:38 | |
| corvus | restarting now | 21:39 |
| *** camunoz has quit IRC | 21:39 | |
| corvus | so here's a question, with supercedent post, should we set the priority at normal or low? | 21:41 |
| *** heyongli has quit IRC | 21:43 | |
| hongbin | hi infra team, a question, in the gate job definition, what is the different between this two variables: LIBS_FROM_GIT and OVERRIDE_"$uc_project"_PROJECT_BRANCH | 21:43 |
| fungi | corvus: i think i'd still keep it at low mainly because there's not much else at low? | 21:43 |
| *** heyongli has joined #openstack-infra | 21:43 | |
| hongbin | if i want to checkout a library from source, which variable i should use, or both? | 21:44 |
| corvus | fungi: easy to start there | 21:44 |
| fungi | if we had a more fine-grained prioritization i could see adjusting but with only 3 buckets you can either make it compete with check or make check get priority | 21:44 |
| mordred | hongbin: OVERRIDE_ is a legacy variable ... | 21:44 |
| hongbin | mordred: ack | 21:44 |
| mordred | hongbin: http://git.openstack.org/cgit/openstack/openstacksdk/tree/.zuul.yaml#n260 is an example of overriding the branch zuul checks out | 21:45 |
| *** hashar has quit IRC | 21:45 | |
| hongbin | using 'branches:' ? | 21:46 |
| *** rcernin has joined #openstack-infra | 21:47 | |
| mordred | hongbin: sorry - using override-checkout | 21:47 |
| *** boden has quit IRC | 21:47 | |
| mordred | hongbin: branches controls branch matchers for filtering which patches should trigger the job | 21:47 |
| hongbin | mordred: i see, thanks for the replies | 21:48 |
| corvus | hongbin: if you're able to use the new devstack job (or devstack-tempest, or a child job of those), you can add a library to "required-projects" and it will be checked out from source | 21:48 |
| hongbin | corvus: ack | 21:49 |
| corvus | hongbin: https://review.openstack.org/548331 is an example of that | 21:50 |
| hongbin | corvus: cool, thanks for the link | 21:50 |
| *** pguimaraes has quit IRC | 21:50 | |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Drop docs-on-readthedocs from airship-armada https://review.openstack.org/578217 | 21:51 |
| *** mriedem has quit IRC | 21:52 | |
| *** heyongli has quit IRC | 21:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Use supercedent manager in post https://review.openstack.org/578218 | 21:54 |
| *** heyongli has joined #openstack-infra | 21:54 | |
| *** jamesmcarthur has quit IRC | 21:54 | |
| *** myoung is now known as myoung|off | 21:56 | |
| *** mriedem has joined #openstack-infra | 21:56 | |
| *** afazekas has quit IRC | 21:56 | |
| mordred | corvus: I'm excited about the supercedent manager! | 21:58 |
| corvus | me too! do you know how often we add pipeline managers? | 21:58 |
| mordred | about once every 6 years? | 21:58 |
| *** afazekas has joined #openstack-infra | 21:59 | |
| mordred | dhellmann: ^^ did you see this, btw? | 21:59 |
| corvus | yep. it's the oldest pluggable component of zuul, and it's never had anything new plugged into it since commit 1. until now. | 21:59 |
| mordred | corvus: \o/ | 21:59 |
| openstackgerrit | Oliver Walsh proposed openstack/diskimage-builder master: Add new modprobe element https://review.openstack.org/577170 | 22:00 |
| openstackgerrit | Oliver Walsh proposed openstack/diskimage-builder master: Add DIB element to blacklist nouveau https://review.openstack.org/571777 | 22:00 |
| *** heyongli has quit IRC | 22:04 | |
| *** edmondsw has quit IRC | 22:04 | |
| *** heyongli has joined #openstack-infra | 22:04 | |
| *** edmondsw has joined #openstack-infra | 22:07 | |
| *** bobh has quit IRC | 22:07 | |
| *** jesslampe has quit IRC | 22:07 | |
| *** yolanda__ has joined #openstack-infra | 22:07 | |
| *** agopi has joined #openstack-infra | 22:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 22:10 |
| *** felipemonteiro has joined #openstack-infra | 22:10 | |
| *** yolanda_ has quit IRC | 22:10 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 22:10 |
| *** edmondsw has quit IRC | 22:11 | |
| *** felipemonteiro_ has joined #openstack-infra | 22:12 | |
| *** felipemonteiro has quit IRC | 22:12 | |
| openstackgerrit | Oliver Walsh proposed openstack/diskimage-builder master: WIP: Add DIB element to install NVIDIA GPU drivers https://review.openstack.org/573223 | 22:12 |
| *** felipemonteiro__ has joined #openstack-infra | 22:13 | |
| *** sthussey has quit IRC | 22:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Make websocket streaming more event-driven https://review.openstack.org/558646 | 22:14 |
| *** heyongli has quit IRC | 22:14 | |
| *** heyongli has joined #openstack-infra | 22:14 | |
| dhellmann | corvus : woo! thanks for the heads-up, mordred -- smcginnis , ttx: https://review.openstack.org/#/c/578218/ should fix the race issue we have with publishing to releases.o.o | 22:16 |
| *** felipemonteiro_ has quit IRC | 22:17 | |
| corvus | i'll be sending a ml post once we at least see it work well enough not to immediately blow up :) | 22:18 |
| openstackgerrit | Merged openstack-infra/zuul master: configloader: skip merger:cat when no items are included https://review.openstack.org/576016 | 22:18 |
| openstackgerrit | Merged openstack-infra/zuul master: Add a config_errors info to {tenant}/config-errors endpoint https://review.openstack.org/553873 | 22:18 |
| openstackgerrit | Merged openstack-infra/zuul master: Refactor load sensors into drivers https://review.openstack.org/549275 | 22:18 |
| *** mriedem has quit IRC | 22:21 | |
| *** yolanda_ has joined #openstack-infra | 22:21 | |
| *** lihi has quit IRC | 22:22 | |
| *** yolanda__ has quit IRC | 22:24 | |
| *** heyongli has quit IRC | 22:24 | |
| *** ldnunes has quit IRC | 22:24 | |
| *** heyongli has joined #openstack-infra | 22:24 | |
| clarkb | corvus: are you good for us to merge 578218 at this point? | 22:25 |
| *** lihi has joined #openstack-infra | 22:25 | |
| corvus | clarkb: yep | 22:25 |
| clarkb | +3 | 22:26 |
| *** bnemec has quit IRC | 22:29 | |
| *** sthussey has joined #openstack-infra | 22:30 | |
| clarkb | corvus: hrm actually zuul status makes it look like jobs are queued and "running" but not actually running? | 22:32 |
| clarkb | 578169,2 for example | 22:32 |
| fungi | mordred: https://review.openstack.org/577331 is probably of interest (annoyance?) to you | 22:33 |
| *** heyongli has quit IRC | 22:34 | |
| clarkb | oh is that what it looks like when we recover from scheduler restart? | 22:35 |
| *** bnemec has joined #openstack-infra | 22:35 | |
| *** heyongli has joined #openstack-infra | 22:35 | |
| clarkb | I think that may happen when executors grab jobs but haven't managed to start them yet | 22:35 |
| corvus | or rather maybe zuul has put the job on the gearman queue but the executor hasn't picked it up yet | 22:36 |
| *** hongbin has quit IRC | 22:36 | |
| corvus | there are 200 jobs on the gearman queue waiting right now | 22:36 |
| corvus | (we almost never have jobs waiting in gearman these days) | 22:36 |
| corvus | so yeah, basically, the executors may be a wee bit overloaded due to the restart | 22:37 |
| fungi | thundering herd or pitching in on cat jobs? | 22:37 |
| corvus | thundering herd of real jobs | 22:38 |
| corvus | but the governors are working as expected, so they're not picking up more than they can handle | 22:38 |
| *** threestrands has joined #openstack-infra | 22:38 | |
| *** threestrands has quit IRC | 22:38 | |
| *** threestrands has joined #openstack-infra | 22:38 | |
| clarkb | cool, working as expected then | 22:39 |
| *** ihar has quit IRC | 22:39 | |
| corvus | as long as the yellow line on the zuul job queue graph keeps going down, and the nodepool graphs look good, i think so | 22:40 |
| clarkb | it is interesting that we've managed to run around 100 concurrent jobs per $tool | 22:41 |
| clarkb | jenkins, jjb v2.5, ansible v3 | 22:41 |
| fungi | almost as if the limit there isn't the software itself | 22:42 |
| corvus | well... we reduced the size of our executors from the jenkins masters :) | 22:43 |
| *** Swami_ has quit IRC | 22:43 | |
| *** Swami has quit IRC | 22:43 | |
| clarkb | speaking of maybe shrinking zuul01.o.o will be a good time to switch to zk cluster | 22:44 |
| corvus | i believe a single executor could handle about 1000 jobs before hitting python thread scaling issues | 22:44 |
| corvus | given enough ram/cpu/disk | 22:44 |
| *** heyongli has quit IRC | 22:45 | |
| fungi | what flavor were we running for the jenkins masters? i've already forgotten i guess | 22:45 |
| *** heyongli has joined #openstack-infra | 22:45 | |
| clarkb | 16GB 8vcpu (something like that) | 22:46 |
| fungi | as for switching to the zk cluster, do we need to fix the hostnames before we do that? they're still getting most of their cronspam bounced because they seem to think their fqdns are "zk01" et cetera | 22:46 |
| clarkb | and thy often needed more | 22:46 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/lpmqtt master: Never give up never surrender https://review.openstack.org/578211 | 22:47 |
| *** jesslampe has joined #openstack-infra | 22:48 | |
| *** rlandy has quit IRC | 22:49 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: fix status page flickering https://review.openstack.org/578226 | 22:50 |
| *** tpsilva has quit IRC | 22:55 | |
| *** heyongli has quit IRC | 22:55 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: angular6 fix suggestion https://review.openstack.org/573494 | 22:55 |
| *** heyongli has joined #openstack-infra | 22:55 | |
| *** rossella_s has quit IRC | 22:56 | |
| *** rossella_s has joined #openstack-infra | 22:59 | |
| openstackgerrit | Merged openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v6) https://review.openstack.org/551989 | 22:59 |
| openstackgerrit | Merged openstack-infra/zuul master: Hide queue headers for empty queues when filtering https://review.openstack.org/572588 | 22:59 |
| *** felipemonteiro__ has quit IRC | 23:01 | |
| *** Kevin_Zheng has joined #openstack-infra | 23:05 | |
| *** heyongli has quit IRC | 23:05 | |
| *** heyongli has joined #openstack-infra | 23:05 | |
| *** nicolasbock has quit IRC | 23:06 | |
| *** tosky has quit IRC | 23:08 | |
| *** bobh has joined #openstack-infra | 23:08 | |
| openstackgerrit | Merged openstack-infra/system-config master: More minor debian-security mirror fixes https://review.openstack.org/578057 | 23:12 |
| openstackgerrit | Merged openstack-infra/system-config master: tools/sorted_modules.py : make python3 safe https://review.openstack.org/575606 | 23:12 |
| *** caphrim007_ has joined #openstack-infra | 23:15 | |
| *** heyongli has quit IRC | 23:15 | |
| *** caphrim007_ has quit IRC | 23:16 | |
| *** caphrim007_ has joined #openstack-infra | 23:16 | |
| *** heyongli has joined #openstack-infra | 23:16 | |
| *** caphrim007 has quit IRC | 23:18 | |
| *** caphrim007_ has quit IRC | 23:21 | |
| *** jesslampe has quit IRC | 23:24 | |
| *** owalsh_ has joined #openstack-infra | 23:25 | |
| *** owalsh- has joined #openstack-infra | 23:25 | |
| *** heyongli has quit IRC | 23:26 | |
| *** heyongli has joined #openstack-infra | 23:26 | |
| *** owalsh| has joined #openstack-infra | 23:26 | |
| *** owalsh^ has joined #openstack-infra | 23:27 | |
| *** owalsh has quit IRC | 23:29 | |
| *** owalsh_ has quit IRC | 23:29 | |
| *** owalsh- has quit IRC | 23:31 | |
| *** owalsh| has quit IRC | 23:31 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add a clouds.yaml checker https://review.openstack.org/575607 | 23:33 |
| openstackgerrit | Merged openstack-infra/system-config master: Add identity_api_version to packethost https://review.openstack.org/577331 | 23:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Use supercedent manager in post https://review.openstack.org/578218 | 23:34 |
| corvus | zuul is performing the reconfig to pick that up now ^ | 23:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: WIP: Support skip_child_jobs via zuul_return https://review.openstack.org/578230 | 23:35 |
| *** s-shiono has joined #openstack-infra | 23:35 | |
| *** owalsh has joined #openstack-infra | 23:36 | |
| *** heyongli has quit IRC | 23:36 | |
| *** heyongli has joined #openstack-infra | 23:36 | |
| *** rpioso is now known as rpioso|afk | 23:36 | |
| *** owalsh^ has quit IRC | 23:36 | |
| corvus | well, it has coalesced the post pipeline into a series of dependent changes, and only the first one active | 23:43 |
| corvus | it has not yet eliminated the duplicates | 23:43 |
| clarkb | corvus: we should've expected that only the currently queued ref and the most recent ref would be in the queue if I remember the review for that pipeline manager | 23:45 |
| *** heyongli has quit IRC | 23:46 | |
| *** owalsh_ has joined #openstack-infra | 23:46 | |
| corvus | yep. it only prunes the queues when adding a change though. it's possible that the re-enqueue method bypasses that, so we won't see the correct behavior until the next time a change is added to one of those queues | 23:46 |
| *** heyongli has joined #openstack-infra | 23:47 | |
| *** stakeda has joined #openstack-infra | 23:48 | |
| *** owalsh has quit IRC | 23:49 | |
| corvus | oh hey there it goes :) | 23:50 |
| corvus | i saved old queues. zuul changes b8 and 5f were the top and bottom of the queue respectively | 23:50 |
| corvus | there are now no more than 2 items for each project in post | 23:51 |
| corvus | i approved a simple zuul change, 554058 which should supercede 5f if it lands before b8 finishes | 23:51 |
| clarkb | quick approve more chanes so that post doesn't get ready nodes | 23:53 |
| corvus | we still have a little bit of a backlog, so nothing in post is running yet | 23:53 |
| *** heyongli has quit IRC | 23:56 | |
| *** owalsh has joined #openstack-infra | 23:57 | |
| *** heyongli has joined #openstack-infra | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!