| *** heyongli has quit IRC | 00:01 | |
| *** heyongli has joined #openstack-infra | 00:02 | |
| *** stakeda has joined #openstack-infra | 00:03 | |
| *** masayukig has quit IRC | 00:05 | |
| *** vtapia has joined #openstack-infra | 00:11 | |
| *** heyongli has quit IRC | 00:11 | |
| *** masayukig has joined #openstack-infra | 00:12 | |
| *** heyongli has joined #openstack-infra | 00:12 | |
| *** zhangfei has joined #openstack-infra | 00:17 | |
| ianw | ssbarnea / mnaser : new release of ansible-lint should fix things up. preference is probably to let that percolate through | 00:21 |
|---|---|---|
| ianw | hmm, afs dashboard already helping :) | 00:22 |
| ianw | rsync: rename "/afs/.openstack.org/mirror/opensuse/tumbleweed/repo/oss/x86_64/.libubsan1-8.1.1+r260570-1.3.x86_64.rpm.ccsmRc" -> "x86_64/libubsan1-8.1.1+r260570-1.3.x86_64.rpm": File too large (27) | 00:22 |
| *** heyongli has quit IRC | 00:22 | |
| ianw | "file too large" is a new one ... | 00:22 |
| *** heyongli has joined #openstack-infra | 00:22 | |
| ianw | hrm, maybe we are hitting limits in number of files in a directory | 00:23 |
| ianw | https://lists.openafs.org/pipermail/openafs-info/2002-September/005812.html | 00:23 |
| *** heyongli has quit IRC | 00:32 | |
| *** heyongli has joined #openstack-infra | 00:32 | |
| *** pahuang has quit IRC | 00:33 | |
| *** rcernin_ has joined #openstack-infra | 00:34 | |
| *** rcernin has quit IRC | 00:37 | |
| *** heyongli has quit IRC | 00:42 | |
| *** heyongli has joined #openstack-infra | 00:43 | |
| *** yamamoto has joined #openstack-infra | 00:43 | |
| *** sthussey has joined #openstack-infra | 00:47 | |
| *** yamamoto has quit IRC | 00:48 | |
| *** pahuang has joined #openstack-infra | 00:50 | |
| *** jesslampe has joined #openstack-infra | 00:51 | |
| ianw | if i have understood the slot calculation correctly | 00:52 |
| ianw | ianw@mirror-update01:/var/log$ ls -c1 /afs/.openstack.org/mirror/opensuse/tumbleweed/repo/oss/x86_64/ | awk 'function slots(x){i = length($0)+1; return 1 + rshift((i+15), 5)} {n += slots($0)}END{print n}' | 00:52 |
| ianw | 63008 | 00:52 |
| *** edmondsw_ has joined #openstack-infra | 00:52 | |
| *** heyongli has quit IRC | 00:52 | |
| *** heyongli has joined #openstack-infra | 00:53 | |
| *** jesslampe has quit IRC | 00:54 | |
| *** jesslampe has joined #openstack-infra | 00:54 | |
| *** jesslampe has quit IRC | 00:54 | |
| *** edmondsw has quit IRC | 00:55 | |
| auristor | ianw: directory fragmentation is also an issue | 01:01 |
| ianw | auristor: hello :) another day, another thing i'm learning about (open)AFS :) | 01:01 |
| auristor | another thing that auristorfs has addressed :-( | 01:02 |
| *** hongbin has joined #openstack-infra | 01:02 | |
| *** heyongli has quit IRC | 01:03 | |
| *** heyongli has joined #openstack-infra | 01:03 | |
| auristor | it is possible that there are slots available but not enough contiguous slots. | 01:03 |
| *** namnh has joined #openstack-infra | 01:05 | |
| auristor | if you force a salvage of the volume via bos using the -salvagedirs option it will force all directories to be reconstructed which will effectively defragment the directories | 01:05 |
| *** rh-jelabarre has joined #openstack-infra | 01:06 | |
| ianw | ohh, interesting. that might explain why (if my calculation is correct) it's slightly under the limit but still failing | 01:06 |
| ianw | that was, of course, taken from a thread where you've been through this all before -> https://lists.openafs.org/pipermail/openafs-info/2016-July/041859.html :) | 01:07 |
| ianw | i haven't run a defrag in probably 20 years :) it always was gratifying to see those little boxes shuffling back and forth | 01:09 |
| auristor | heh | 01:10 |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** heyongli has quit IRC | 01:13 | |
| *** heyongli has joined #openstack-infra | 01:13 | |
| *** slaweq has quit IRC | 01:16 | |
| ianw | hmmm, so salvagedirs all requires "-all" ... salvage whole server ... i'm not sure i want to do that | 01:21 |
| *** heyongli has quit IRC | 01:23 | |
| *** heyongli has joined #openstack-infra | 01:24 | |
| auristor | I don't think openafs has another option. however, you can spin up another fileserver, move the volume, salvage it, and move it back | 01:24 |
| *** germs has quit IRC | 01:24 | |
| *** germs has joined #openstack-infra | 01:25 | |
| *** germs has quit IRC | 01:25 | |
| *** germs has joined #openstack-infra | 01:25 | |
| ianw | https://lists.openafs.org/pipermail/openafs-info/2017-April/042097.html suggests adding to the parameters in the BosConfig for the salvage server | 01:25 |
| auristor | that will work. there simply isn't a protocol option for it. | 01:26 |
| auristor | that requires deleting the current dafs bnode and creating a new one with the new salvageserver command line | 01:27 |
| *** yamamoto has joined #openstack-infra | 01:29 | |
| *** edmondsw_ has quit IRC | 01:29 | |
| *** yamamoto has quit IRC | 01:33 | |
| *** heyongli has quit IRC | 01:33 | |
| *** heyongli has joined #openstack-infra | 01:34 | |
| ianw | auristor: ok, so added that, did the salvage on the volume ... it went pretty quickly ... a couple of "Vnode 215514: version < inode version; fixed (old status)" messages but no luck, rsync still fails | 01:39 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: configloader: skip merger:cat when no items are included https://review.openstack.org/576016 | 01:42 |
| *** heyongli has quit IRC | 01:44 | |
| *** pahuang has quit IRC | 01:44 | |
| *** heyongli has joined #openstack-infra | 01:44 | |
| *** heyongli has quit IRC | 01:54 | |
| *** heyongli has joined #openstack-infra | 01:54 | |
| *** pahuang has joined #openstack-infra | 01:57 | |
| *** yamamoto has joined #openstack-infra | 01:59 | |
| *** yamamoto has quit IRC | 02:04 | |
| *** heyongli has quit IRC | 02:04 | |
| *** heyongli has joined #openstack-infra | 02:05 | |
| *** rcernin_ has quit IRC | 02:09 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** yamamoto has joined #openstack-infra | 02:14 | |
| *** heyongli has quit IRC | 02:14 | |
| *** heyongli has joined #openstack-infra | 02:15 | |
| *** slaweq has quit IRC | 02:16 | |
| auristor | ianw: you should have seen "Salvaging directory" messages if the flag was passed to salvager | 02:17 |
| ianw | auristor: yep, did | 02:18 |
| auristor | there there isn't any space. | 02:18 |
| *** yamamoto has quit IRC | 02:18 | |
| ianw | hmm, maybe i didn't | 02:19 |
| ianw | http://paste.openstack.org/show/723683/ | 02:19 |
| ianw | the *flag* is there | 02:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add percona to cache https://review.openstack.org/575944 | 02:20 |
| auristor | that output doesn't include "Salvaging directory <vnode>" messages | 02:20 |
| *** annp has joined #openstack-infra | 02:22 | |
| *** psachin has joined #openstack-infra | 02:23 | |
| auristor | I don't think the installed version of OpenAFS has the necessary functionality | 02:25 |
| *** heyongli has quit IRC | 02:25 | |
| *** heyongli has joined #openstack-infra | 02:25 | |
| auristor | Installed version: OpenAFS 1.6.7-1ubuntu1.1-debian built 2015-11-10 | 02:26 |
| auristor | Commit that added the functionality to salvageserver | 02:26 |
| auristor | commit d5f05cce7ffc61a3340509f94388421e3cf2522d | 02:26 |
| auristor | Author: Michael Meffie <mmeffie@sinenomine.net> | 02:26 |
| auristor | Date: Sun Nov 6 14:31:22 2016 -0600 | 02:26 |
| auristor | dafs: honor salvageserver -salvagedirs | 02:26 |
| ianw | well ... that explains that ... | 02:27 |
| *** ramishra has joined #openstack-infra | 02:27 | |
| *** yamamoto has joined #openstack-infra | 02:29 | |
| *** yamamoto has quit IRC | 02:34 | |
| *** heyongli has quit IRC | 02:35 | |
| *** heyongli has joined #openstack-infra | 02:35 | |
| *** rh-jelabarre has quit IRC | 02:37 | |
| *** edmondsw has joined #openstack-infra | 02:44 | |
| *** yamamoto has joined #openstack-infra | 02:44 | |
| *** heyongli has quit IRC | 02:45 | |
| *** heyongli has joined #openstack-infra | 02:46 | |
| *** yamamoto has quit IRC | 02:49 | |
| *** edmondsw has quit IRC | 02:49 | |
| *** yamamoto has joined #openstack-infra | 02:49 | |
| *** yamamoto has quit IRC | 02:49 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add percona to cache https://review.openstack.org/575944 | 02:52 |
| *** hph^ has quit IRC | 02:55 | |
| *** heyongli has quit IRC | 02:55 | |
| *** heyongli has joined #openstack-infra | 02:56 | |
| *** sthussey has quit IRC | 02:57 | |
| *** ianychoi_ has joined #openstack-infra | 02:58 | |
| *** stakeda has quit IRC | 03:00 | |
| *** yamamoto has joined #openstack-infra | 03:01 | |
| *** ianychoi_ has quit IRC | 03:01 | |
| *** ianychoi has quit IRC | 03:02 | |
| *** ianychoi_ has joined #openstack-infra | 03:02 | |
| *** yamamoto has quit IRC | 03:06 | |
| *** heyongli has quit IRC | 03:06 | |
| *** heyongli has joined #openstack-infra | 03:06 | |
| *** rlandy has joined #openstack-infra | 03:08 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** bhavik1 has joined #openstack-infra | 03:12 | |
| *** yamamoto has joined #openstack-infra | 03:16 | |
| *** slaweq has quit IRC | 03:16 | |
| *** heyongli has quit IRC | 03:16 | |
| *** heyongli has joined #openstack-infra | 03:16 | |
| *** yamamoto has quit IRC | 03:20 | |
| *** yamamoto has joined #openstack-infra | 03:21 | |
| *** rlandy has quit IRC | 03:21 | |
| *** hongbin has quit IRC | 03:23 | |
| *** heyongli has quit IRC | 03:26 | |
| *** heyongli has joined #openstack-infra | 03:27 | |
| *** vivsoni has joined #openstack-infra | 03:29 | |
| *** heyongli has quit IRC | 03:36 | |
| *** heyongli has joined #openstack-infra | 03:37 | |
| *** bhavik1 has quit IRC | 03:43 | |
| *** heyongli has quit IRC | 03:47 | |
| *** heyongli has joined #openstack-infra | 03:47 | |
| *** udesale has joined #openstack-infra | 03:51 | |
| *** jesslampe has joined #openstack-infra | 03:56 | |
| *** heyongli has quit IRC | 03:57 | |
| *** heyongli has joined #openstack-infra | 03:57 | |
| *** zhangfei has quit IRC | 04:05 | |
| *** heyongli has quit IRC | 04:07 | |
| *** heyongli has joined #openstack-infra | 04:08 | |
| *** ramishra has quit IRC | 04:08 | |
| *** Bhujay has joined #openstack-infra | 04:08 | |
| *** ramishra has joined #openstack-infra | 04:10 | |
| *** r-daneel has joined #openstack-infra | 04:10 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** e0ne has joined #openstack-infra | 04:11 | |
| *** e0ne has quit IRC | 04:13 | |
| *** r-daneel_ has joined #openstack-infra | 04:13 | |
| *** germs has quit IRC | 04:14 | |
| *** r-daneel has quit IRC | 04:15 | |
| *** r-daneel_ is now known as r-daneel | 04:15 | |
| *** slaweq has quit IRC | 04:15 | |
| *** jesslampe has quit IRC | 04:16 | |
| *** heyongli has quit IRC | 04:17 | |
| *** heyongli has joined #openstack-infra | 04:18 | |
| *** zhangfei has joined #openstack-infra | 04:20 | |
| *** jesslampe has joined #openstack-infra | 04:20 | |
| *** ramishra has quit IRC | 04:23 | |
| *** ramishra has joined #openstack-infra | 04:26 | |
| *** felipemonteiro has joined #openstack-infra | 04:26 | |
| *** heyongli has quit IRC | 04:28 | |
| *** heyongli has joined #openstack-infra | 04:28 | |
| *** edmondsw has joined #openstack-infra | 04:32 | |
| *** zhangfei has quit IRC | 04:33 | |
| *** vivsoni has quit IRC | 04:36 | |
| *** jesslampe has quit IRC | 04:36 | |
| *** edmondsw has quit IRC | 04:37 | |
| *** links has joined #openstack-infra | 04:37 | |
| *** heyongli has quit IRC | 04:38 | |
| *** heyongli has joined #openstack-infra | 04:38 | |
| *** janki has joined #openstack-infra | 04:40 | |
| *** heyongli has quit IRC | 04:48 | |
| *** heyongli has joined #openstack-infra | 04:49 | |
| *** vivsoni has joined #openstack-infra | 04:50 | |
| *** lifeless has quit IRC | 04:52 | |
| *** jesslampe has joined #openstack-infra | 04:52 | |
| *** jesslampe has joined #openstack-infra | 04:53 | |
| *** heyongli has quit IRC | 04:59 | |
| *** heyongli has joined #openstack-infra | 04:59 | |
| *** dhajare has joined #openstack-infra | 05:04 | |
| *** heyongli has quit IRC | 05:09 | |
| *** heyongli has joined #openstack-infra | 05:09 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** felipemonteiro has quit IRC | 05:13 | |
| *** slaweq has quit IRC | 05:16 | |
| *** bobh has joined #openstack-infra | 05:17 | |
| *** jesusaur has quit IRC | 05:17 | |
| *** pgadiya has joined #openstack-infra | 05:18 | |
| *** pgadiya has quit IRC | 05:18 | |
| *** e0ne has joined #openstack-infra | 05:19 | |
| *** vivsoni has quit IRC | 05:19 | |
| *** heyongli has quit IRC | 05:19 | |
| *** heyongli has joined #openstack-infra | 05:19 | |
| *** rcernin has joined #openstack-infra | 05:20 | |
| openstackgerrit | Merged openstack-infra/system-config master: Mirror Ceph Mimic for Debian based https://review.openstack.org/571989 | 05:21 |
| *** e0ne has quit IRC | 05:22 | |
| *** heyongli has quit IRC | 05:29 | |
| *** heyongli has joined #openstack-infra | 05:30 | |
| *** udesale_ has joined #openstack-infra | 05:31 | |
| *** aeng has joined #openstack-infra | 05:33 | |
| *** udesale has quit IRC | 05:33 | |
| *** vivsoni has joined #openstack-infra | 05:35 | |
| *** heyongli has quit IRC | 05:40 | |
| *** heyongli has joined #openstack-infra | 05:40 | |
| *** bobh has quit IRC | 05:41 | |
| *** cshastri_ has joined #openstack-infra | 05:46 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: configloader: skip merger:cat when no items are included https://review.openstack.org/576016 | 05:47 |
| *** jtomasek has joined #openstack-infra | 05:49 | |
| *** jtomasek has quit IRC | 05:49 | |
| *** heyongli has quit IRC | 05:50 | |
| *** jtomasek has joined #openstack-infra | 05:50 | |
| *** heyongli has joined #openstack-infra | 05:50 | |
| *** iranzo has joined #openstack-infra | 05:54 | |
| *** heyongli has quit IRC | 06:00 | |
| *** heyongli has joined #openstack-infra | 06:00 | |
| *** dhajare has quit IRC | 06:02 | |
| *** florianf has joined #openstack-infra | 06:10 | |
| *** heyongli has quit IRC | 06:10 | |
| *** heyongli has joined #openstack-infra | 06:11 | |
| *** ramishra has quit IRC | 06:13 | |
| *** ramishra has joined #openstack-infra | 06:14 | |
| *** dpawlik has joined #openstack-infra | 06:18 | |
| *** Bhujay has quit IRC | 06:18 | |
| *** heyongli has quit IRC | 06:21 | |
| *** heyongli has joined #openstack-infra | 06:21 | |
| *** edmondsw has joined #openstack-infra | 06:21 | |
| *** annp has quit IRC | 06:21 | |
| *** dpawlik has quit IRC | 06:22 | |
| *** dpawlik has joined #openstack-infra | 06:23 | |
| *** Bhujay has joined #openstack-infra | 06:23 | |
| *** edmondsw has quit IRC | 06:26 | |
| *** zhangfei has joined #openstack-infra | 06:30 | |
| *** heyongli has quit IRC | 06:31 | |
| *** heyongli has joined #openstack-infra | 06:31 | |
| *** udesale__ has joined #openstack-infra | 06:35 | |
| *** pcaruana has joined #openstack-infra | 06:35 | |
| *** udesale_ has quit IRC | 06:38 | |
| *** dhajare has joined #openstack-infra | 06:38 | |
| *** aeng has quit IRC | 06:39 | |
| *** heyongli has quit IRC | 06:41 | |
| *** heyongli has joined #openstack-infra | 06:41 | |
| *** dmellado has joined #openstack-infra | 06:41 | |
| *** yolanda__ is now known as yolanda | 06:43 | |
| *** slaweq has joined #openstack-infra | 06:50 | |
| *** hashar has joined #openstack-infra | 06:51 | |
| *** heyongli has quit IRC | 06:51 | |
| *** heyongli has joined #openstack-infra | 06:52 | |
| *** udesale__ is now known as udesale | 06:54 | |
| *** hjensas has quit IRC | 06:54 | |
| *** sree has joined #openstack-infra | 06:55 | |
| *** armaan has joined #openstack-infra | 07:01 | |
| *** rcernin has quit IRC | 07:01 | |
| *** heyongli has quit IRC | 07:02 | |
| *** heyongli has joined #openstack-infra | 07:02 | |
| *** jesusaur has joined #openstack-infra | 07:03 | |
| *** jesslampe has quit IRC | 07:04 | |
| *** tojuvone has joined #openstack-infra | 07:09 | |
| *** heyongli has quit IRC | 07:12 | |
| *** heyongli has joined #openstack-infra | 07:12 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Fix bootloader for efi on rhel systems https://review.openstack.org/576041 | 07:18 |
| *** heyongli has quit IRC | 07:22 | |
| *** heyongli has joined #openstack-infra | 07:22 | |
| *** tesseract has joined #openstack-infra | 07:22 | |
| *** heyongli has quit IRC | 07:32 | |
| *** heyongli has joined #openstack-infra | 07:33 | |
| *** janki has quit IRC | 07:36 | |
| *** tosky has joined #openstack-infra | 07:36 | |
| *** links has quit IRC | 07:38 | |
| *** ccamacho has joined #openstack-infra | 07:40 | |
| *** heyongli has quit IRC | 07:43 | |
| *** heyongli has joined #openstack-infra | 07:43 | |
| *** jpena|off is now known as jpena | 07:43 | |
| *** zhangfei has quit IRC | 07:44 | |
| *** amoralej|off is now known as amoralej | 07:44 | |
| *** hjensas has joined #openstack-infra | 07:47 | |
| *** hjensas has quit IRC | 07:47 | |
| *** hjensas has joined #openstack-infra | 07:47 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: use kazoo retry facilities https://review.openstack.org/535537 | 07:51 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: retry initial zookeeper connection attempts https://review.openstack.org/576047 | 07:51 |
| *** heyongli has quit IRC | 07:53 | |
| *** heyongli has joined #openstack-infra | 07:53 | |
| *** links has joined #openstack-infra | 07:55 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zk: use kazoo retry facilities https://review.openstack.org/536209 | 07:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zk: retry initial zookeeper connection attempts https://review.openstack.org/576048 | 07:55 |
| *** zhangfei has joined #openstack-infra | 07:57 | |
| *** janki has joined #openstack-infra | 08:00 | |
| *** armaan has quit IRC | 08:01 | |
| *** armaan has joined #openstack-infra | 08:01 | |
| *** jpich has joined #openstack-infra | 08:01 | |
| *** kjackal has joined #openstack-infra | 08:02 | |
| *** e0ne has joined #openstack-infra | 08:03 | |
| *** heyongli has quit IRC | 08:03 | |
| *** heyongli has joined #openstack-infra | 08:03 | |
| *** links has quit IRC | 08:04 | |
| *** e0ne_ has joined #openstack-infra | 08:05 | |
| *** roman_g has joined #openstack-infra | 08:06 | |
| *** e0ne has quit IRC | 08:07 | |
| *** e0ne_ has quit IRC | 08:07 | |
| *** r-daneel has quit IRC | 08:08 | |
| *** r-daneel has joined #openstack-infra | 08:09 | |
| *** Bhujay has quit IRC | 08:09 | |
| *** sshnaidm has quit IRC | 08:09 | |
| *** shardy has joined #openstack-infra | 08:09 | |
| *** edmondsw has joined #openstack-infra | 08:09 | |
| *** heyongli has quit IRC | 08:13 | |
| *** edmondsw has quit IRC | 08:14 | |
| *** heyongli has joined #openstack-infra | 08:14 | |
| *** eernst has joined #openstack-infra | 08:16 | |
| *** dougsz has joined #openstack-infra | 08:16 | |
| *** links has joined #openstack-infra | 08:18 | |
| *** zhangfei has quit IRC | 08:18 | |
| *** lpetrut has joined #openstack-infra | 08:23 | |
| *** armaan has quit IRC | 08:23 | |
| *** armaan has joined #openstack-infra | 08:23 | |
| *** heyongli has quit IRC | 08:24 | |
| *** heyongli has joined #openstack-infra | 08:24 | |
| *** alexchadin has joined #openstack-infra | 08:25 | |
| *** Bhujay has joined #openstack-infra | 08:31 | |
| *** heyongli has quit IRC | 08:34 | |
| *** heyongli has joined #openstack-infra | 08:34 | |
| *** annp has joined #openstack-infra | 08:34 | |
| *** derekh has joined #openstack-infra | 08:35 | |
| *** gfidente has joined #openstack-infra | 08:42 | |
| *** heyongli has quit IRC | 08:44 | |
| *** heyongli has joined #openstack-infra | 08:44 | |
| *** e0ne has joined #openstack-infra | 08:46 | |
| *** finucannot is now known as stephenfin | 08:47 | |
| *** dbecker has joined #openstack-infra | 08:50 | |
| *** sree has quit IRC | 08:51 | |
| *** panda|off is now known as panda | 08:51 | |
| *** aojea has joined #openstack-infra | 08:54 | |
| *** heyongli has quit IRC | 08:54 | |
| *** lifeless has joined #openstack-infra | 08:54 | |
| *** heyongli has joined #openstack-infra | 08:55 | |
| *** pblaho has joined #openstack-infra | 08:59 | |
| *** rpittau has joined #openstack-infra | 09:03 | |
| *** heyongli has quit IRC | 09:05 | |
| *** heyongli has joined #openstack-infra | 09:05 | |
| *** kzaitsev1pi has quit IRC | 09:07 | |
| *** kzaitsev_pi has joined #openstack-infra | 09:09 | |
| *** sree has joined #openstack-infra | 09:10 | |
| *** pbourke has joined #openstack-infra | 09:11 | |
| *** sambetts|afk is now known as sambetts | 09:14 | |
| *** heyongli has quit IRC | 09:15 | |
| *** heyongli has joined #openstack-infra | 09:15 | |
| *** jaosorior has joined #openstack-infra | 09:23 | |
| *** heyongli has quit IRC | 09:25 | |
| *** heyongli has joined #openstack-infra | 09:25 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/gerritbot master: Add change-created event type https://review.openstack.org/286366 | 09:27 |
| flaper87 | ianw: hey, if you're around, could I get access to the nodes that where held for tripleo-ci-centos-7-scenario009-multinode-oooq :) | 09:31 |
| flaper87 | ianw: patch 574233 | 09:31 |
| flaper87 | ianw: and my key https://github.com/flaper87.keys | 09:32 |
| *** armaan has quit IRC | 09:33 | |
| *** PapaOurs is now known as bauzas | 09:34 | |
| *** sshnaidm has joined #openstack-infra | 09:34 | |
| *** heyongli has quit IRC | 09:35 | |
| *** heyongli has joined #openstack-infra | 09:36 | |
| *** armaan has joined #openstack-infra | 09:36 | |
| *** armaan has quit IRC | 09:38 | |
| *** armaan has joined #openstack-infra | 09:42 | |
| *** dtantsur|afk is now known as dtantsur | 09:43 | |
| *** heyongli has quit IRC | 09:46 | |
| *** heyongli has joined #openstack-infra | 09:46 | |
| *** sree has quit IRC | 09:50 | |
| frickler | flaper87: ianw: the nodes are 162.253.55.151 and 162.253.55.199, will add your key in a moment | 09:50 |
| *** sree has joined #openstack-infra | 09:50 | |
| flaper87 | frickler: thanks a bunch | 09:51 |
| *** sree_ has joined #openstack-infra | 09:52 | |
| *** sree_ is now known as Guest94276 | 09:52 | |
| frickler | flaper87: actually I cannot login for some reason. does tripleo ci change the root login? | 09:53 |
| *** sree has quit IRC | 09:55 | |
| *** armaan has quit IRC | 09:55 | |
| flaper87 | frickler: nope | 09:55 |
| flaper87 | :/ | 09:56 |
| *** heyongli has quit IRC | 09:56 | |
| *** heyongli has joined #openstack-infra | 09:56 | |
| *** Guest94276 has quit IRC | 09:56 | |
| *** armaan has joined #openstack-infra | 09:57 | |
| *** pcichy has joined #openstack-infra | 09:57 | |
| frickler | flaper87: hmm, this looks like it does http://logs.openstack.org/33/574233/9/experimental/tripleo-ci-centos-7-scenario009-multinode-oooq/f00fd65/job-output.txt.gz#_2018-06-15_12_36_34_310290 | 10:02 |
| *** lifeless has quit IRC | 10:03 | |
| flaper87 | mmh, that's weird. fungi and ianw didn't have problems to get into these boxes | 10:03 |
| flaper87 | mmh | 10:03 |
| *** lifeless has joined #openstack-infra | 10:04 | |
| ianw | let me check... | 10:05 |
| ianw | i am in on 162.253.55.199 | 10:05 |
| *** heyongli has quit IRC | 10:06 | |
| *** heyongli has joined #openstack-infra | 10:06 | |
| ianw | flaper87: added to ^ | 10:07 |
| ianw | fricker: how weird, i don't see your key on ^ | 10:07 |
| ianw | that would have to suggest your key wasn't in the cloud that host is on | 10:08 |
| *** eernst has quit IRC | 10:08 | |
| flaper87 | ianw: I'm in 162.253.55.199, thanks | 10:08 |
| flaper87 | is 162.253.55.151 giving problems? | 10:09 |
| ianw | ok, yep fix that up now. sorry was typing it's ip wrong | 10:09 |
| ianw | well, pasting it wrong anyway :) | 10:09 |
| flaper87 | nice, I'm in | 10:10 |
| flaper87 | thanks a bunch | 10:10 |
| ianw | frickler: so that's vexxhost-ca-ymq ... maybe check if the cloud deployer on puppetmaster is actually working correctly there? maybe it has an old set of keys? | 10:11 |
| ianw | i dunno what the deal is, but /var/log/puppet_run_cloud_launcher.log has no mention of vexxhost ... so i'm guessing that's going to be the problem somehow | 10:16 |
| *** heyongli has quit IRC | 10:16 | |
| *** heyongli has joined #openstack-infra | 10:17 | |
| frickler | ianw: hmm, strange, I checked a couple of vexxhost nodes, I can log into some, but not all of them. | 10:17 |
| *** namnh has quit IRC | 10:19 | |
| slaweq | hi, I just noticed that in many jobs on stable/ocata and stable/pike branches in Neutron we have issues with devstack, like e.g.: http://logs.openstack.org/75/575675/1/check/neutron-rally-neutron/f21ffa6/logs/devstacklog.txt.gz#_2018-06-18_07_53_58_955 | 10:19 |
| slaweq | it looks that it's some error in devstack | 10:19 |
| slaweq | did You maybe saw it before? | 10:19 |
| slaweq | do You know what can be a reason of it? | 10:19 |
| slaweq | (if not I will have to debug it on my own but maybe someone already noticed that) | 10:20 |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: Remove duplicate entries from /etc/default/grub https://review.openstack.org/576084 | 10:20 |
| ianw | slaweq: keystone ...? cmurphy might know maybe? | 10:21 |
| slaweq | ianw: I'm not sure if it's keystone issue or some devstack misconfiguration maybe | 10:22 |
| *** larainema has joined #openstack-infra | 10:23 | |
| *** heyongli has quit IRC | 10:27 | |
| *** heyongli has joined #openstack-infra | 10:27 | |
| *** jaosorior has quit IRC | 10:30 | |
| *** zoli is now known as zoli|lunch | 10:34 | |
| frickler | slaweq: ianw: this looks like some oslo dependency error, let me spin up a test myself. also #openstack-qa is better suited for discussing devstack failures http://logs.openstack.org/77/575677/1/check/neutron-rally-neutron/7f36d14/logs/screen-keystone.txt.gz#_Jun_18_07_41_22_999313 | 10:34 |
| slaweq | frickler: I just wasn't sure where is better to ask about it, here or on openstack-qa :) | 10:35 |
| slaweq | frickler: I can ask there | 10:35 |
| openstackgerrit | Sam Betts proposed openstack-infra/glean master: Fix bonding for RHEL and systemd based OSs https://review.openstack.org/455638 | 10:36 |
| *** heyongli has quit IRC | 10:37 | |
| *** heyongli has joined #openstack-infra | 10:37 | |
| openstackgerrit | Sam Betts proposed openstack-infra/devstack-gate master: Move location of devstack-gate unstack to post-tempest https://review.openstack.org/530726 | 10:42 |
| frickler | slaweq: the line between gate issues and devstack issues is thin, so it's not wrong to ask here, too. | 10:44 |
| *** yamamoto has quit IRC | 10:44 | |
| slaweq | frickler: thx :) | 10:45 |
| *** heyongli has quit IRC | 10:47 | |
| *** heyongli has joined #openstack-infra | 10:47 | |
| *** alex_xu has quit IRC | 10:49 | |
| *** alex_xu has joined #openstack-infra | 10:49 | |
| *** szaher has joined #openstack-infra | 10:54 | |
| *** d0ugal has quit IRC | 10:54 | |
| *** d0ugal has joined #openstack-infra | 10:56 | |
| *** heyongli has quit IRC | 10:57 | |
| *** heyongli has joined #openstack-infra | 10:58 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Convert labels to upper case https://review.openstack.org/576098 | 10:59 |
| *** alexchadin has quit IRC | 11:05 | |
| *** heyongli has quit IRC | 11:08 | |
| *** heyongli has joined #openstack-infra | 11:08 | |
| *** jpena is now known as jpena|lunch | 11:08 | |
| *** lifeless has quit IRC | 11:10 | |
| *** alexchadin has joined #openstack-infra | 11:18 | |
| *** heyongli has quit IRC | 11:18 | |
| *** heyongli has joined #openstack-infra | 11:18 | |
| *** Goneri has joined #openstack-infra | 11:20 | |
| *** Goneri has quit IRC | 11:24 | |
| *** heyongli has quit IRC | 11:28 | |
| *** heyongli has joined #openstack-infra | 11:28 | |
| *** dave-mccowan has joined #openstack-infra | 11:30 | |
| *** ldnunes has joined #openstack-infra | 11:31 | |
| *** jaosorior has joined #openstack-infra | 11:33 | |
| *** amoralej is now known as amoralej|lunch | 11:33 | |
| *** zoli|lunch is now known as zoli | 11:33 | |
| *** dave-mcc_ has joined #openstack-infra | 11:35 | |
| *** dave-mccowan has quit IRC | 11:35 | |
| *** heyongli has quit IRC | 11:38 | |
| *** heyongli has joined #openstack-infra | 11:39 | |
| *** rossella_s has joined #openstack-infra | 11:42 | |
| *** armaan has quit IRC | 11:43 | |
| *** armaan has joined #openstack-infra | 11:43 | |
| *** yamamoto has joined #openstack-infra | 11:45 | |
| *** aojeagarcia has joined #openstack-infra | 11:45 | |
| *** rh-jelabarre has joined #openstack-infra | 11:45 | |
| *** aojeagarcia__ has joined #openstack-infra | 11:46 | |
| *** aojea has quit IRC | 11:48 | |
| *** sthussey has joined #openstack-infra | 11:48 | |
| *** heyongli has quit IRC | 11:49 | |
| *** heyongli has joined #openstack-infra | 11:49 | |
| *** udesale_ has joined #openstack-infra | 11:49 | |
| *** owalsh_ has joined #openstack-infra | 11:49 | |
| *** aojeagarcia has quit IRC | 11:49 | |
| *** owalsh has quit IRC | 11:50 | |
| *** yamamoto has quit IRC | 11:52 | |
| *** udesale has quit IRC | 11:52 | |
| *** yamamoto has joined #openstack-infra | 11:52 | |
| *** udesale_ has quit IRC | 11:54 | |
| *** heyongli has quit IRC | 11:59 | |
| *** heyongli has joined #openstack-infra | 11:59 | |
| *** jpena|lunch is now known as jpena | 12:03 | |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: Remove duplicate entries from /etc/default/grub https://review.openstack.org/576084 | 12:07 |
| *** edmondsw has joined #openstack-infra | 12:09 | |
| *** heyongli has quit IRC | 12:09 | |
| *** heyongli has joined #openstack-infra | 12:09 | |
| *** rfolco has joined #openstack-infra | 12:16 | |
| *** rlandy has joined #openstack-infra | 12:17 | |
| *** armaan has quit IRC | 12:18 | |
| *** armaan has joined #openstack-infra | 12:18 | |
| *** heyongli has quit IRC | 12:19 | |
| *** heyongli has joined #openstack-infra | 12:20 | |
| *** alexchadin has quit IRC | 12:23 | |
| *** heyongli has quit IRC | 12:30 | |
| *** heyongli has joined #openstack-infra | 12:30 | |
| *** jistr is now known as jistr|mtg | 12:36 | |
| *** heyongli has quit IRC | 12:40 | |
| *** heyongli has joined #openstack-infra | 12:40 | |
| *** sree has joined #openstack-infra | 12:41 | |
| *** sree has quit IRC | 12:46 | |
| *** amoralej|lunch is now known as amoralej | 12:47 | |
| mnaser | what's our position / way to go about adding python3 in cnetos? | 12:47 |
| *** heyongli has quit IRC | 12:50 | |
| *** heyongli has joined #openstack-infra | 12:51 | |
| *** bobh has joined #openstack-infra | 12:55 | |
| *** alexchadin has joined #openstack-infra | 12:56 | |
| *** dklyle_ has joined #openstack-infra | 12:57 | |
| *** kgiusti has joined #openstack-infra | 12:58 | |
| *** david-lyle has quit IRC | 12:59 | |
| *** alexchadin has quit IRC | 13:00 | |
| *** Goneri has joined #openstack-infra | 13:00 | |
| *** heyongli has quit IRC | 13:00 | |
| *** heyongli has joined #openstack-infra | 13:01 | |
| *** alexchadin has joined #openstack-infra | 13:01 | |
| *** r-daneel has quit IRC | 13:01 | |
| *** r-daneel has joined #openstack-infra | 13:02 | |
| *** jistr|mtg is now known as jistr | 13:04 | |
| *** r-daneel has quit IRC | 13:05 | |
| *** mriedem has joined #openstack-infra | 13:05 | |
| *** neiloy has joined #openstack-infra | 13:08 | |
| fungi | mnaser: wait for the next version where it's rumored python3 will replace python2 | 13:08 |
| fungi | at least that's been the plan so far | 13:09 |
| fungi | python3 in centos7 is nonstandard and nontrivial | 13:09 |
| *** heyongli has quit IRC | 13:11 | |
| *** heyongli has joined #openstack-infra | 13:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Just use chmod instead of file for log permissions https://review.openstack.org/575913 | 13:15 |
| mordred | fungi: updated per your suggestion ^^ | 13:16 |
| *** heyongli has quit IRC | 13:21 | |
| *** heyongli has joined #openstack-infra | 13:21 | |
| *** jesslampe has joined #openstack-infra | 13:23 | |
| *** felipemonteiro has joined #openstack-infra | 13:24 | |
| openstackgerrit | Merged openstack-infra/system-config master: Switch ansible-lint to using multiple -x options https://review.openstack.org/575982 | 13:25 |
| mordred | frickler, AJaeger: either of you have a sec for an easy-ish review? https://review.openstack.org/#/c/575913/ | 13:27 |
| *** bobh has quit IRC | 13:29 | |
| *** nicolasbock has joined #openstack-infra | 13:29 | |
| *** sree has joined #openstack-infra | 13:29 | |
| *** heyongli has quit IRC | 13:31 | |
| *** armaan has quit IRC | 13:31 | |
| *** heyongli has joined #openstack-infra | 13:31 | |
| *** armaan has joined #openstack-infra | 13:32 | |
| frickler | mordred: +2 | 13:34 |
| *** superdan is now known as dansmith | 13:38 | |
| mordred | thanks | 13:38 |
| *** heyongli has quit IRC | 13:41 | |
| *** heyongli has joined #openstack-infra | 13:42 | |
| *** yamamoto has quit IRC | 13:42 | |
| *** yamamoto has joined #openstack-infra | 13:42 | |
| *** esarault has joined #openstack-infra | 13:42 | |
| *** kashyap has joined #openstack-infra | 13:44 | |
| *** camunoz has joined #openstack-infra | 13:47 | |
| *** cshastri_ has quit IRC | 13:48 | |
| *** udesale has joined #openstack-infra | 13:48 | |
| *** felipemonteiro has quit IRC | 13:51 | |
| openstackgerrit | Merged openstack-infra/project-config master: Just use chmod instead of file for log permissions https://review.openstack.org/575913 | 13:51 |
| *** heyongli has quit IRC | 13:52 | |
| *** heyongli has joined #openstack-infra | 13:52 | |
| *** armaan has quit IRC | 13:52 | |
| *** armaan has joined #openstack-infra | 13:52 | |
| *** bobh has joined #openstack-infra | 13:53 | |
| *** sshnaidm_ has joined #openstack-infra | 13:54 | |
| *** zhangfei has joined #openstack-infra | 13:54 | |
| *** psachin has quit IRC | 13:55 | |
| mnaser | fungi: we have a user who pushed python3 changes for all of our jobs in OSA and so failing with centos with no python3 support :( | 13:56 |
| mnaser | it would be nice to come up with some 'interim recommended workaround' | 13:56 |
| *** camunoz has quit IRC | 13:56 | |
| *** camunoz has joined #openstack-infra | 13:57 | |
| *** sshnaidm has quit IRC | 13:57 | |
| fungi | yup, use python2 or a distro which provides a usable python3 | 13:59 |
| fungi | i think tripleo is working around it by testing on latest fedora instead | 13:59 |
| fungi | alternatives would be to set up software collections under centos7 and then add the python3 interpreter to the default execution path, or compile your own | 14:00 |
| fungi | i don't know of any jobs doing that | 14:01 |
| rcarrillocruz | yeah, software collections. I believe tristanC and the softwarefactory folks have something similar in their jobs | 14:01 |
| rcarrillocruz | mnaser: ^ | 14:01 |
| fungi | it mostly comes down to asking yourself what the point of the test is | 14:01 |
| fungi | usually we test software in the environments people will be running it, to make sure we don't regress it | 14:01 |
| *** heyongli has quit IRC | 14:02 | |
| fungi | the odds that anyone is running openstackansible under python3 on centos7? slim to nonexistent | 14:02 |
| mnaser | fungi: you're right on that | 14:02 |
| *** heyongli has joined #openstack-infra | 14:02 | |
| fungi | for teh same reasons that setting it up to test that way is a pain, setting it up to use that way would also be a pain | 14:02 |
| mnaser | probably even non existant | 14:03 |
| mnaser | it is possible that someone does yum install python3 and tries using our stuff in ubuntu however, maybe we should watch/test for that i guess | 14:03 |
| *** jesslampe has quit IRC | 14:03 | |
| fungi | well, testing it under python3 on ubuntu makes sense because ubuntu provides an integrated python3 | 14:03 |
| fungi | similarly fedora, if you think you have users on fedora | 14:04 |
| mnaser | OSA operates under the assumption that the machine running OSA uses the same OS as targets | 14:04 |
| fungi | jumping through hoops to fabricate a python3 scenario on centos7 which is likely to reflect the way basically 0 users are going to care about, probably not worth your time | 14:04 |
| mordred | mnaser, fungi: pyenv might work too | 14:04 |
| fungi | mordred: yeah, that's basically "compile from source" above, just with a fancy wrapper | 14:05 |
| mordred | but I agree with fungi - it's important that testing python3 is actually testing something valuable for people | 14:05 |
| mordred | fungi: yup | 14:05 |
| *** hjensas has quit IRC | 14:05 | |
| mnaser | mordred: that's actually not bad. so the thing is openstack ansible usually deploys a venv with all its stuff in /opt/openstack-ansible | 14:05 |
| mnaser | so i guess we COULD use pyenv and deploy a python3 env in /opt/openstack-ansible | 14:05 |
| mnaser | but that would be a pita for deployers i guess | 14:06 |
| mnaser | who wants to build stuff still \o/ | 14:06 |
| fungi | gentoo users ;) | 14:06 |
| *** felipemonteiro has joined #openstack-infra | 14:07 | |
| mnaser | :P | 14:07 |
| *** felipemonteiro_ has joined #openstack-infra | 14:08 | |
| *** links has quit IRC | 14:09 | |
| *** nicolasbock has quit IRC | 14:11 | |
| *** felipemonteiro has quit IRC | 14:12 | |
| *** heyongli has quit IRC | 14:12 | |
| *** heyongli has joined #openstack-infra | 14:12 | |
| *** heyongli has quit IRC | 14:22 | |
| *** heyongli has joined #openstack-infra | 14:23 | |
| *** alexchad_ has joined #openstack-infra | 14:23 | |
| *** alexchadin has quit IRC | 14:25 | |
| *** felipemonteiro_ has quit IRC | 14:31 | |
| *** sree has quit IRC | 14:31 | |
| *** heyongli has quit IRC | 14:33 | |
| *** heyongli has joined #openstack-infra | 14:33 | |
| *** bdodd has joined #openstack-infra | 14:34 | |
| *** anteaya has joined #openstack-infra | 14:34 | |
| *** rpioso|afk is now known as rpioso | 14:36 | |
| mordred | mnaser: :) | 14:38 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Add NODEPOOL_PERCONA_PROXY to mirror_info https://review.openstack.org/576156 | 14:39 |
| mnaser | infra-core: ^ really quick and simple review | 14:40 |
| mnaser | fwiw: the kata-containers arch. call is in 20 minutes. i'm planning to bring up and continue the discussion of working with 'winterscale' | 14:40 |
| mnaser | if anyone wants to hop on as well, that'd be nice, but i'm okay too | 14:40 |
| mnaser | https://etherpad.openstack.org/p/katacontainers-2018-architecture-committee-mtgs | 14:40 |
| clarkb | mnaser: not sure if you saw my notes in the kata channel but I think we are just down to some buggy tests preventing zuul jobs from passing successfully | 14:41 |
| mnaser | oh man | 14:42 |
| mnaser | i think there might be no meeting today | 14:42 |
| mnaser | biweekly schedule :X | 14:42 |
| *** hongbin has joined #openstack-infra | 14:42 | |
| *** jesslampe has joined #openstack-infra | 14:42 | |
| clarkb | anyways I've brought it up and asked fo rhelp understadning because from my debugging zuul is failing properly and jenkins is not testing what they want to be testing | 14:42 |
| *** heyongli has quit IRC | 14:43 | |
| * mnaser shrugs | 14:43 | |
| mnaser | thanks for the info clarkb -- i saw some notes but yeah, i'm not so familiar with the inner workings | 14:43 |
| *** heyongli has joined #openstack-infra | 14:43 | |
| mnaser | i just kinda got vms+jenkins up and handed them over | 14:43 |
| mnaser | https://katacontainers.io/ ok there does seem to be a call | 14:44 |
| *** janki has quit IRC | 14:44 | |
| mnaser | Architecture Committee Schedule (Weekly Monday Meetings) | 14:44 |
| frickler | mnaser: if you ping "infra-root" or "config-core" that gets better attention ;) also using the opportunity to advertise my patch documenting this :D https://review.openstack.org/572323 | 14:48 |
| mnaser | frickler: i try to avoid roots because i guesst hey're busy with other things :D | 14:49 |
| mnaser | also good catch, i think that the lxc which i cp'd from is brokn | 14:49 |
| mnaser | i'm not going to adjust it because i am going to assume it will break things. | 14:49 |
| *** dklyle_ has quit IRC | 14:49 | |
| frickler | mnaser: yes, I was more hoping some other reviewer might pick this up if needed | 14:51 |
| mnaser | frickler: id push a follow up but OSA world is broken enough as is (we're probably the only consumers of this) | 14:51 |
| frickler | mnaser: to avoid root pinging that's why we have "config-core", highlighting reviewers for project-config and {openstack-,}zuul-jobs | 14:52 |
| mnaser | gotcha, i'll change my terminology :) | 14:53 |
| *** heyongli has quit IRC | 14:53 | |
| frickler | mnaser: btw, did you get access to the two held nodes for ansible-ceph debugging? do you still need those? | 14:53 |
| *** heyongli has joined #openstack-infra | 14:53 | |
| mnaser | frickler: ah sorry, yes we did, couldnt even reproduce (shucks) so they can be dropped | 14:53 |
| frickler | mnaser: o.k., so I'll delete them, let us know if you need further help | 14:54 |
| pabelanger | heads up about gerrit 2.14 when we upgrade to it, it defaults to sending (color) HTMLs now: https://www.gerritcodereview.com/releases/2.14.md#html-emails just learned the hard way with sf.io. Something to remember to disable when we upgrade review.o.o | 14:55 |
| fungi | ew | 14:56 |
| *** zhangfei has quit IRC | 14:56 | |
| fungi | "...can be disabled by setting sendemail.html to false" | 14:56 |
| fungi | i'm about to stand up a gerrit for personal use in the near future, so definitely going to be setting that option | 14:57 |
| fungi | though this is something i've been waiting years for: http://gerrit-documentation.storage.googleapis.com/Documentation/2.14/intro-user.html#reply-by-email | 14:58 |
| fungi | heh... "While Gerrit supports a wide range of email clients, the following ones have been tested and are known to work: (anything maintained by google)" | 14:58 |
| fungi | color me surprised | 14:58 |
| anteaya | pabelanger, has there been an etherpad started for the upgrade work? | 14:59 |
| anteaya | pabelanger, I don't think anyone has taken on driving that yet | 14:59 |
| pabelanger | anteaya: I don't believe so... agree | 14:59 |
| *** lpetrut has quit IRC | 15:01 | |
| *** lpetrut has joined #openstack-infra | 15:02 | |
| *** heyongli has quit IRC | 15:03 | |
| * mordred is pondering volunteering, but wants to get some other stuff done before signing up for new work | 15:04 | |
| clarkb | mordred: AJaeger: where did we get on the updating how zuul grabs logs in base jobs? is a job still required to copy things to the executor's log dir manually? | 15:04 |
| *** heyongli has joined #openstack-infra | 15:04 | |
| *** sshnaidm_ has quit IRC | 15:06 | |
| mordred | clarkb: yes - that is actually the next zuul-jobs task I was going to try to pick back up | 15:07 |
| *** dtruong_ has joined #openstack-infra | 15:07 | |
| mordred | clarkb: status:open topic:zuulv3-output is the work so far | 15:08 |
| mordred | pabelanger: is there any way (that you know of) that we could change the global default of that value? | 15:09 |
| mordred | (I mean, I'm sure we could write a java patch to add to our branch) | 15:09 |
| mordred | but I would **very** much like to make that an opt-in feature since HTML emails are evil and break the internet | 15:09 |
| *** nicolasbock has joined #openstack-infra | 15:09 | |
| mordred | oh - wait - that's a global config | 15:10 |
| * mordred shuts up | 15:10 | |
| *** bobh has quit IRC | 15:10 | |
| *** ccamacho has quit IRC | 15:10 | |
| fungi | mordred: yeah, i quoted the section of the config document which tells us how to disable it | 15:11 |
| pabelanger | mordred: yes, global config | 15:11 |
| mordred | sweet. I thought it was a per-user config setting for some reason when I read it first | 15:11 |
| fungi | seems it's both. global to change the default, per-user to set a non-default preference | 15:11 |
| mordred | that config setting already exists, yeah? so we should be able to go ahead and put it into the current config | 15:12 |
| pabelanger | Yah, my sadness is it is opt-out by default | 15:12 |
| *** dtruong has quit IRC | 15:12 | |
| fungi | why do you hate pretty pictures and colorful text? it's optimized for gmail! ;) | 15:12 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Mention keywords to highlight tech support https://review.openstack.org/572323 | 15:12 |
| pabelanger | fungi: indeed | 15:12 |
| mordred | hrm. it actually doesn't seem to be an option in 2.13 | 15:13 |
| pabelanger | yah, new feature in 2.14 | 15:13 |
| *** lpetrut has quit IRC | 15:13 | |
| mordred | sigh | 15:13 |
| *** heyongli has quit IRC | 15:14 | |
| mordred | well, comments-by-reply is pretty cool | 15:14 |
| *** heyongli has joined #openstack-infra | 15:14 | |
| fungi | i'd be a lot more excited about review-by-mail if we didn't have gertty | 15:14 |
| mordred | oh - it's only in response to comments and labels - and it's only comments - so you can't do a full review workflow with it | 15:19 |
| *** sshnaidm_ has joined #openstack-infra | 15:19 | |
| *** udesale_ has joined #openstack-infra | 15:20 | |
| fungi | yeah, somewhat limited but for casual responses to review comments it sounds useful | 15:20 |
| fungi | okay, stepping out to grab lunch/run errands but will return as soon as i can | 15:20 |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: Fix for proper LVM support https://review.openstack.org/576168 | 15:21 |
| mnaser | clarkb: so just finished talking with kata. i would like to take sometime maybe with you and other infra team members that are interested in bringing kata as a tenant in zuul | 15:21 |
| *** sree has joined #openstack-infra | 15:22 | |
| mnaser | alongside other kata ci folks | 15:22 |
| *** udesale has quit IRC | 15:22 | |
| mnaser | so that we can sit down and iterate much quicker with their help | 15:22 |
| *** heyongli has quit IRC | 15:24 | |
| *** heyongli has joined #openstack-infra | 15:24 | |
| *** udesale_ has quit IRC | 15:25 | |
| pabelanger | mnaser: are you thinking today? Or scheduling something in the future | 15:26 |
| mnaser | pabelanger: i'm open, maybe we can start an email thread in openstack-infra? | 15:26 |
| mnaser | i explained the whole idea behind 'winterscale' | 15:26 |
| *** gyee has joined #openstack-infra | 15:27 | |
| pabelanger | mnaser: maybe confirm with clarkb, there was a good meeting in vancouver, but was mentioned having f2f make it much faster to iterate on. Maybe getting people into pbx.o.o (voice) will help again | 15:28 |
| pabelanger | could even record it | 15:28 |
| mnaser | pabelanger: yes, i agree on that, or zoom/hangouts/etc.. whatever works :) | 15:29 |
| *** dklyle has joined #openstack-infra | 15:29 | |
| *** rossella_s has quit IRC | 15:30 | |
| clarkb | my plan has/had been to get this job working as step 0 | 15:30 |
| clarkb | then we can show it works, I think a big part of the issue in the past trying to work with this is general skepticism and not having anything concrete to look at or refer to | 15:31 |
| clarkb | if I/we can get this job working that becomes the something we can point to | 15:31 |
| *** armaan has quit IRC | 15:31 | |
| anteaya | mnaser would they join the irc channel if it was called something other than openstack-infra? | 15:31 |
| clarkb | anteaya: they've joined this channel before I don't think that is the issue | 15:32 |
| anteaya | clarkb, okay thanks | 15:32 |
| *** sree has quit IRC | 15:32 | |
| clarkb | the biggest issuse I see have been prioritization of effort being elsewhere due to needing a release done (which is fine but meant no time for iterating on ci) | 15:34 |
| *** heyongli has quit IRC | 15:34 | |
| clarkb | with release done, now its just a matter of showing them that the sunk cost of the jenkins config isn't a major cost if switched to zuul | 15:34 |
| *** heyongli has joined #openstack-infra | 15:34 | |
| openstackgerrit | Ade Lee proposed openstack-infra/irc-meetings master: Change barbican meeting time and storyboard link https://review.openstack.org/576177 | 15:34 |
| clarkb | (which is why I think having a working job is important) | 15:34 |
| mnaser | clarkb: that's why i think it'd be good to sit down together and iterate | 15:35 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Capture logs in kata-runsh job https://review.openstack.org/576178 | 15:35 |
| mnaser | also does anyone know why this is failing? https://review.openstack.org/#/c/576156/ .. i see multinode stuff but i have no idea | 15:36 |
| *** Bhujay has quit IRC | 15:36 | |
| clarkb | mnaser: http://logs.openstack.org/56/576156/1/check/openstack-infra-multinode-integration-centos-7/213bb64/ara-report/result/025a1d66-7785-4674-a8f5-5bb2c2b62b5f/ its one of my least favorite error messages on linux | 15:37 |
| clarkb | mnaser: seems to be unhappy with the bridge address update | 15:37 |
| mnaser | yeah i saw that too :( | 15:38 |
| clarkb | (possibly beacuse the address is already ste on it) | 15:38 |
| mnaser | it's so confusing | 15:38 |
| *** felipemonteiro has joined #openstack-infra | 15:38 | |
| mnaser | clarkb: it happens in both fedora and centos so something shared there.. | 15:38 |
| *** jaosorior has quit IRC | 15:38 | |
| clarkb | mordred: does https://review.openstack.org/576178 looks right to you as far as collecting logs goes? | 15:38 |
| clarkb | mnaser: ya, it is in a block checking the address isn't already set first, then only sets it if the return code is 1 (indicating address isn't already there) | 15:39 |
| clarkb | ip: command not found | 15:39 |
| mnaser | i didnt notice that one | 15:39 |
| clarkb | http://logs.openstack.org/56/576156/1/check/openstack-infra-multinode-integration-centos-7/213bb64/ara-report/result/cea4d067-8b8e-4316-a6a0-e5df91c8a106/ I think that is the issue | 15:40 |
| clarkb | base image change maybe? | 15:40 |
| mnaser | /bin/sh | 15:40 |
| mnaser | wait | 15:40 |
| clarkb | ya that is ansible default which should be ok for that command | 15:40 |
| mnaser | so the follow up command works | 15:40 |
| mnaser | so ip addr commands works but the one before it says command doesnt exist | 15:41 |
| mnaser | whah | 15:41 |
| clarkb | mnaser: oh! the second is become: yes but the first isn't | 15:41 |
| clarkb | could be a path issue where regular users don't get that in their path by default but rootdoes? | 15:41 |
| mnaser | clarkb: that does sound to be a culprit.. but would such a major behavioural change happen in a distro lol | 15:42 |
| clarkb | mnaser: I stopped being surprised by major changes in stable centos :) | 15:43 |
| *** kjackal has quit IRC | 15:43 | |
| mnaser | clarkb: one is shell and the other is command | 15:43 |
| mnaser | shell opens up a shell and exports env variables afaik | 15:43 |
| mnaser | command needs a full path | 15:43 |
| clarkb | ah | 15:43 |
| mnaser | maybe this was broken by ansible 2.5? | 15:43 |
| clarkb | except the command one is the one that works | 15:44 |
| clarkb | and the shell one is the one that doesn't | 15:44 |
| mnaser | oh yes you're right | 15:44 |
| *** alexchad_ has quit IRC | 15:44 | |
| *** heyongli has quit IRC | 15:44 | |
| *** alexchadin has joined #openstack-infra | 15:44 | |
| *** heyongli has joined #openstack-infra | 15:45 | |
| mnaser | https://docs.ansible.com/ansible/2.5/porting_guides/porting_guide_2.5.html#shell | 15:45 |
| mnaser | For a full list, check the shell plugin you are using, the default shell plugin is sh. | 15:45 |
| mnaser | i wonder if its teh default shell changing to /bin/sh | 15:45 |
| *** alexchadin has quit IRC | 15:45 | |
| *** diablo_rojo has joined #openstack-infra | 15:45 | |
| *** lpetrut has joined #openstack-infra | 15:47 | |
| *** sshnaidm_ has quit IRC | 15:48 | |
| *** sree has joined #openstack-infra | 15:48 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/devstack-gate master: Switch `ip` commands to use `command` module https://review.openstack.org/576184 | 15:48 |
| *** lpetrut has quit IRC | 15:49 | |
| *** hemna_ has joined #openstack-infra | 15:49 | |
| *** lpetrut has joined #openstack-infra | 15:49 | |
| *** janki has joined #openstack-infra | 15:51 | |
| *** sree has quit IRC | 15:52 | |
| *** yamamoto has quit IRC | 15:53 | |
| *** jesslampe has quit IRC | 15:53 | |
| *** e0ne has quit IRC | 15:54 | |
| *** jesslampe has joined #openstack-infra | 15:54 | |
| *** bobh has joined #openstack-infra | 15:54 | |
| *** yamamoto has joined #openstack-infra | 15:54 | |
| *** heyongli has quit IRC | 15:55 | |
| *** heyongli has joined #openstack-infra | 15:55 | |
| *** yamamoto has quit IRC | 15:56 | |
| *** yamamoto has joined #openstack-infra | 15:56 | |
| *** lpetrut has quit IRC | 15:56 | |
| *** lpetrut has joined #openstack-infra | 15:58 | |
| clarkb | mnaser: in ^ I don't think the pipe to grep will work under command (that is why it is shell) | 15:59 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Prevent Zuul scheduler to crash at startup if gerrit down https://review.openstack.org/576192 | 16:00 |
| mordred | clarkb: yes | 16:01 |
| mnaser | oh | 16:01 |
| mnaser | you're right | 16:01 |
| mnaser | in that case | 16:01 |
| *** jesslampe has quit IRC | 16:02 | |
| *** zaneb has joined #openstack-infra | 16:02 | |
| clarkb | following up on our caching proxy performance. The general port 8080 proxy seems to servce about 95% of all requests from its cache | 16:03 |
| clarkb | the dockerhub cache is more like 90% | 16:03 |
| clarkb | still really good overall I think | 16:03 |
| *** shardy has quit IRC | 16:03 | |
| mordred | clarkb: awesome. so it seems like the bulk of the issues were tied to the dockerhub CDN changes | 16:03 |
| mordred | ? | 16:04 |
| clarkb | mordred: yes I think so | 16:04 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/devstack-gate master: Switch `ip` commands to use `command` module https://review.openstack.org/576184 | 16:04 |
| mnaser | clarkb: here's an alternative solution ^ | 16:04 |
| clarkb | dockerhub in ord is 88% and in inap is 92% of requests served from cache | 16:05 |
| *** heyongli has quit IRC | 16:05 | |
| *** heyongli has joined #openstack-infra | 16:05 | |
| clarkb | I need to figure out how to run my script I pushed friday in a way that other people can see these numbers | 16:05 |
| mordred | mnaser: that lgtm | 16:05 |
| *** zoli is now known as zoli|gone | 16:05 | |
| clarkb | I guess just run it in cron once a day or once an hour and serve results out of docroot | 16:05 |
| *** zoli|gone is now known as zoli | 16:05 | |
| mordred | clarkb: you could run it as a periodic zuul job | 16:06 |
| mordred | I'm not saying you should - just talking out loud :) | 16:06 |
| clarkb | ya I think the biggest issue is serving the data in a way that tripleo can consume it sanely | 16:07 |
| *** nicolasbock has quit IRC | 16:08 | |
| clarkb | also nothing seems to be using the new port 8082 dockerhub cache | 16:09 |
| clarkb | I guess loci job churn is just much lower than people using older clients like tripleo | 16:09 |
| *** lpetrut has quit IRC | 16:11 | |
| *** sambetts is now known as sambetts|afk | 16:12 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for cells v1 trusted certs rebuild bug 1777478 https://review.openstack.org/576201 | 16:13 |
| openstack | bug 1777478 in OpenStack Compute (nova) "ServerShowV263Test.test_show_update_rebuild_list_server fails cells v1 job with "ValueError: An object of type nova.objects.instance.TrustedCerts is required in field trusted_certs, not a dict"" [Critical,Triaged] https://launchpad.net/bugs/1777478 - Assigned to Matt Riedemann (mriedem) | 16:13 |
| *** heyongli has quit IRC | 16:15 | |
| *** heyongli has joined #openstack-infra | 16:15 | |
| *** lpetrut has joined #openstack-infra | 16:16 | |
| *** psachin has joined #openstack-infra | 16:16 | |
| *** sshnaidm_ has joined #openstack-infra | 16:16 | |
| clarkb | mnaser: latest command switch lgtm, I've approved it because ci should keep it from merging if it doesn't work. But we should double check in a bit that it actually passed and fix it more if it doesn't :) | 16:18 |
| *** tesseract has quit IRC | 16:19 | |
| *** pcaruana has quit IRC | 16:20 | |
| *** sshnaidm_ is now known as sshnaidm|afk | 16:20 | |
| *** nicolasbock has joined #openstack-infra | 16:20 | |
| mordred | ++ | 16:21 |
| *** yamahata has quit IRC | 16:22 | |
| *** gfidente is now known as gfidente|afk | 16:22 | |
| xgerman_ | somehow I can’t log into gerrit — anyhting odd going on? | 16:22 |
| clarkb | xgerman_: you are the only one to report this behavior. Where is it failing? | 16:23 |
| *** nicolasbock has quit IRC | 16:24 | |
| xgerman_ | when I click on Sign In it goes off to ubuntu one and then back to gerrit but I am not signed in | 16:24 |
| clarkb | mordred: are you going to be around today? want to approve https://review.openstack.org/#/c/571263/ ? | 16:24 |
| *** nicolasbock has joined #openstack-infra | 16:24 | |
| *** heyongli has quit IRC | 16:25 | |
| *** heyongli has joined #openstack-infra | 16:26 | |
| clarkb | xgerman_: ok so it does redirect back to review.openstack.org but shows you as not logged in? can you try in incognito/private browsing mode to avoid any existing cookies | 16:26 |
| xgerman_ | ok | 16:26 |
| xgerman_ | mmh, that made a difference | 16:26 |
| *** trown is now known as trown|lunch | 16:27 | |
| *** jaosorior has joined #openstack-infra | 16:27 | |
| *** jaosorior has quit IRC | 16:28 | |
| clarkb | ianw: fungi thinking out loud here, if we still have admin privs in packethost, can we create a new flavor that has access to 200GB of disk then build a new mirror using that? then we don't have to worry about cinder | 16:28 |
| clarkb | and we can avoid using more cpus than necessary | 16:28 |
| clarkb | xgerman_: in that case I'm guessing something corrupted your existing gerrit cookies (at least from gerrit's perspective) | 16:29 |
| xgerman_ | ok, probbably time to delete all my cookies | 16:29 |
| *** dave-mcc_ has quit IRC | 16:30 | |
| clarkb | ianw: fungi: and if we do still have admin creds where are those written down? | 16:31 |
| *** jaosorior has joined #openstack-infra | 16:32 | |
| mordred | clarkb: yup (to both) | 16:35 |
| *** germs has joined #openstack-infra | 16:35 | |
| *** germs has quit IRC | 16:35 | |
| *** germs has joined #openstack-infra | 16:35 | |
| *** heyongli has quit IRC | 16:36 | |
| *** germs has quit IRC | 16:36 | |
| *** yamamoto has quit IRC | 16:36 | |
| *** heyongli has joined #openstack-infra | 16:36 | |
| *** germs has joined #openstack-infra | 16:36 | |
| *** germs has quit IRC | 16:36 | |
| *** germs has joined #openstack-infra | 16:36 | |
| *** yamamoto has joined #openstack-infra | 16:36 | |
| *** armaan has joined #openstack-infra | 16:36 | |
| *** owalsh_ is now known as owalsh | 16:37 | |
| *** yamamoto has quit IRC | 16:41 | |
| *** r-daneel has joined #openstack-infra | 16:42 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:43 | |
| *** hph^ has joined #openstack-infra | 16:45 | |
| *** heyongli has quit IRC | 16:46 | |
| *** heyongli has joined #openstack-infra | 16:46 | |
| *** felipemonteiro has quit IRC | 16:47 | |
| *** bobh has quit IRC | 16:48 | |
| *** dougsz has quit IRC | 16:55 | |
| *** heyongli has quit IRC | 16:56 | |
| *** heyongli has joined #openstack-infra | 16:56 | |
| openstackgerrit | Merged openstack-infra/system-config master: Switch to using new ansible openstack inventory script https://review.openstack.org/571263 | 16:58 |
| *** jaosorior has quit IRC | 17:01 | |
| *** derekh has quit IRC | 17:01 | |
| *** sree has joined #openstack-infra | 17:04 | |
| *** kjackal has joined #openstack-infra | 17:05 | |
| *** hashar is now known as hasharAway | 17:05 | |
| *** sree has quit IRC | 17:06 | |
| *** heyongli has quit IRC | 17:06 | |
| zaneb | could somebody please add me to https://review.openstack.org/#/admin/groups/1925,members and https://review.openstack.org/#/admin/groups/1926,members (ref https://review.openstack.org/573459) when they have a moment | 17:06 |
| *** heyongli has joined #openstack-infra | 17:07 | |
| *** dtantsur is now known as dtantsur|afk | 17:11 | |
| *** jpich has quit IRC | 17:12 | |
| *** d0ugal has quit IRC | 17:12 | |
| *** amoralej is now known as amoralej|off | 17:13 | |
| *** jpena is now known as jpena|off | 17:15 | |
| *** heyongli has quit IRC | 17:17 | |
| *** heyongli has joined #openstack-infra | 17:17 | |
| *** psachin has quit IRC | 17:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Just use chmod instead of file for log permissions https://review.openstack.org/576215 | 17:20 |
| mordred | fungi: sigh. ^^ that whole time I was fixing the test-upload-logs role and not the upload-logs role | 17:21 |
| mordred | clarkb: ^^ if you have a sec | 17:21 |
| * fungi is back from lunch and trying to catch up now | 17:22 | |
| clarkb | mordred: a lot of the ansible lint rules are feeling not great for CI | 17:24 |
| clarkb | mordred: it yells at you if you ensure latest with package | 17:24 |
| clarkb | which is kinda the point in CI | 17:24 |
| *** nicolasbock has quit IRC | 17:25 | |
| mordred | ++ | 17:25 |
| mordred | it yells about latest? | 17:25 |
| clarkb | yup | 17:25 |
| fungi | there was an alternative suggestion of switching from ansible-lint to ansible-review? something about them being from the same author and recommending the latter over the former? | 17:26 |
| *** d0ugal has joined #openstack-infra | 17:26 | |
| *** heyongli has quit IRC | 17:27 | |
| *** heyongli has joined #openstack-infra | 17:27 | |
| mordred | review is a lint wrapper designed to work with diffs | 17:27 |
| mordred | http://willthames.github.io/2016/06/28/announcing-ansible-review.html | 17:28 |
| mordred | we could use it, but I don't know that its a win for us | 17:28 |
| fungi | ahh | 17:29 |
| mordred | I can see where it would be a win for people who are not gating their tree | 17:29 |
| fungi | i didn't look closely, just saw the discussion in here over the weekend | 17:29 |
| *** iranzo has quit IRC | 17:30 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for cells v1 trusted certs rebuild bug 1777478 https://review.openstack.org/576201 | 17:31 |
| openstack | bug 1777478 in OpenStack Compute (nova) "ServerShowV263Test.test_show_update_rebuild_list_server fails cells v1 job with "ValueError: An object of type nova.objects.instance.TrustedCerts is required in field trusted_certs, not a dict"" [Critical,In progress] https://launchpad.net/bugs/1777478 - Assigned to Matt Riedemann (mriedem) | 17:31 |
| mordred | but since we gate - I think the "don't show me problems in this patch but not in the rest of the code" isn't as necessary for us | 17:31 |
| clarkb | hrm we don't compress log files by default | 17:33 |
| fungi | makes sense | 17:33 |
| * clarkb updates job | 17:33 | |
| *** dklyle has quit IRC | 17:36 | |
| *** dklyle has joined #openstack-infra | 17:37 | |
| *** armaan has quit IRC | 17:37 | |
| *** heyongli has quit IRC | 17:37 | |
| *** heyongli has joined #openstack-infra | 17:37 | |
| *** yamamoto has joined #openstack-infra | 17:38 | |
| anteaya | two spotted fawns froliking in my front yard | 17:39 |
| anteaya | frolicking* | 17:40 |
| *** pblaho has quit IRC | 17:41 | |
| *** yamamoto has quit IRC | 17:42 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Fixed missing npm dependency https://review.openstack.org/576221 | 17:42 |
| dhellmann | could I get someone to take a look at https://review.openstack.org/#/c/574853/ for me, please? The job is already defined, this just adds it to a project template in ozj. | 17:43 |
| dhellmann | fungi : thanks | 17:47 |
| *** heyongli has quit IRC | 17:47 | |
| *** heyongli has joined #openstack-infra | 17:48 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated Guzzle client default config https://review.openstack.org/576223 | 17:48 |
| *** aojeagarcia__ has quit IRC | 17:48 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Updated Guzzle client default config https://review.openstack.org/576223 | 17:49 |
| clarkb | zaneb: done | 17:51 |
| zaneb | clarkb: thanks! | 17:52 |
| *** e0ne has joined #openstack-infra | 17:53 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: add test-release-openstack-python3 to publish-to-pypi-python3 https://review.openstack.org/574853 | 17:55 |
| *** akhilaki has joined #openstack-infra | 17:56 | |
| *** janki has quit IRC | 17:56 | |
| *** dhajare has quit IRC | 17:56 | |
| *** heyongli has quit IRC | 17:58 | |
| *** heyongli has joined #openstack-infra | 17:58 | |
| AJaeger | fungi, save to increase packhost to 10 servers now? Then I'll approve https://review.openstack.org/575863 | 18:00 |
| *** dklyle has quit IRC | 18:00 | |
| fungi | AJaeger: yeah, i think we had planned to merge that on friday | 18:00 |
| fungi | unless we were waiting for a mirror server rebuild? | 18:01 |
| fungi | clarkb was just suggesting using admin superpowers to up the cinder quota | 18:01 |
| AJaeger | fungi, has two +2s already - so you can self-merge any time... | 18:01 |
| clarkb | fungi: not the cinder quota, make a ne flavor size | 18:01 |
| fungi | oh, or that | 18:02 |
| AJaeger | ok, I wait - and just add another +2 ;) | 18:02 |
| fungi | thanks | 18:03 |
| *** kaiokmo has joined #openstack-infra | 18:06 | |
| *** heyongli has quit IRC | 18:08 | |
| *** heyongli has joined #openstack-infra | 18:08 | |
| *** myoung is now known as myoung|lunch | 18:08 | |
| *** r-daneel has quit IRC | 18:09 | |
| *** r-daneel has joined #openstack-infra | 18:10 | |
| clarkb | mnaser: do you have to kata jenkins specific steps handy? trying to figure out how they installer newer docker on xenial | 18:10 |
| mordred | clarkb: there's https://apt.dockerproject.org/repo - which is what I have installed on my laptop | 18:12 |
| clarkb | mordred: https://github.com/kata-containers/runtime/blob/master/versions.yaml#L130 is the problem | 18:13 |
| mordred | clarkb: ah | 18:14 |
| mriedem | we might want to promote this https://review.openstack.org/#/c/576194/ | 18:15 |
| mriedem | lots of nova changes are failing in the gate due to a new tempest test and the cells v1 job | 18:15 |
| *** dave-mccowan has joined #openstack-infra | 18:17 | |
| *** heyongli has quit IRC | 18:18 | |
| *** heyongli has joined #openstack-infra | 18:18 | |
| fungi | Running: ['/etc/ansible/hosts/openstack', '--list'] | 18:21 |
| fungi | OSError: [Errno 2] No such file or directory | 18:21 |
| *** lseki has joined #openstack-infra | 18:21 | |
| *** armaan has joined #openstack-infra | 18:21 | |
| clarkb | fungi: ya puppet should've updated it to openstack-inventory | 18:21 |
| fungi | anybody know what might have changed recently to break our launch-node.py script? | 18:21 |
| clarkb | fungi: https://review.openstack.org/571263 | 18:21 |
| fungi | oh, do we have a fix to launch-node.py for that yet | 18:21 |
| fungi | ? | 18:21 |
| clarkb | if it calls it directly probably | 18:22 |
| clarkb | sorry if I missed that | 18:22 |
| clarkb | mriedem: there are no nova changes in the gate currently | 18:22 |
| clarkb | mriedem: I can enqueue that fix directly to the gate though | 18:22 |
| * clarkb does this | 18:22 | |
| *** armaan has quit IRC | 18:23 | |
| clarkb | mriedem: done | 18:23 |
| *** jroll has quit IRC | 18:24 | |
| mriedem | clarkb: thanks | 18:24 |
| *** jroll has joined #openstack-infra | 18:24 | |
| *** pcaruana has joined #openstack-infra | 18:25 | |
| *** dklyle has joined #openstack-infra | 18:26 | |
| *** trown|lunch is now known as trown | 18:26 | |
| *** vtapia has quit IRC | 18:27 | |
| *** heyongli has quit IRC | 18:28 | |
| *** heyongli has joined #openstack-infra | 18:29 | |
| fungi | clarkb: hrm, so it's not 1:1 substitution i suppose? trying to use that i get a "keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url" | 18:30 |
| clarkb | fungi: mordred said it should be, it did at least work just running it with the --list flag | 18:30 |
| fungi | do we need to change how we're configuring launch-node.py too? | 18:30 |
| clarkb | fungi: possibly you need openstacksdk in your venv? | 18:30 |
| fungi | oh, still trying to use the system-wide context | 18:31 |
| fungi | i'll try it with an ansible virtualenv | 18:31 |
| openstackgerrit | Merged openstack-infra/openstackid master: Fixed missing npm dependency https://review.openstack.org/576221 | 18:36 |
| *** heyongli has quit IRC | 18:39 | |
| *** yamamoto has joined #openstack-infra | 18:39 | |
| *** pcaruana has quit IRC | 18:39 | |
| *** heyongli has joined #openstack-infra | 18:39 | |
| fungi | clarkb: mordred: yeah, pip installing openstacksdk into the venv didn't seem to help. this is the end of the output from launch-node.py where it provides a traceback: http://paste.openstack.org/show/723736/ | 18:39 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Capture logs and install docker in kata-runsh job https://review.openstack.org/576178 | 18:39 |
| clarkb | fungi: is the cloud you are talking to missing an auth_url? | 18:40 |
| clarkb | if not maybe this is a new regression since I last tested things | 18:40 |
| fungi | it's OS_CLOUD=openstackci-rax OS_REGION_NAME=DFW | 18:41 |
| mordred | fungi: mind if I take a lok directly? | 18:42 |
| fungi | go for it | 18:42 |
| clarkb | ya there is no auth_url defined, it is relying on the profile I think | 18:42 |
| fungi | just following the system-config/launch/README and trying to adapt it in the wake of https://review.openstack.org/571263 | 18:42 |
| mordred | fungi: so - somehow you have 0.12.0 installed in your venv | 18:44 |
| *** camunoz has quit IRC | 18:44 | |
| mordred | which is too old to work for this | 18:44 |
| *** yamamoto has quit IRC | 18:44 | |
| fungi | ahh, lemme blow that one away and try again. might have been preexisting and virtualenv didn't replace it | 18:44 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Just use chmod instead of file for log permissions https://review.openstack.org/576215 | 18:45 |
| fungi | though i was getting a similar error using the system-wide installed versions too | 18:45 |
| *** yamahata has joined #openstack-infra | 18:46 | |
| mordred | fungi: we should update the system-wide openstacksdk to 0.14.0 from 0.13.0 - 0.13 is missing limelight | 18:46 |
| *** lpetrut has quit IRC | 18:48 | |
| mnaser | mordred: limestone maybe? :P | 18:48 |
| mordred | mnaser: yes. that one :) | 18:48 |
| *** sshnaidm|afk has quit IRC | 18:49 | |
| mnaser | mordred: https://review.openstack.org/#/c/574746/ ? | 18:49 |
| mnaser | or did something merge already for that | 18:49 |
| *** heyongli has quit IRC | 18:49 | |
| *** vtapia has joined #openstack-infra | 18:49 | |
| *** heyongli has joined #openstack-infra | 18:49 | |
| fungi | limelight was a peer-to-peer file sharing service, right? | 18:49 |
| fungi | or maybe it was just a client | 18:49 |
| mordred | oh! well that might also be an issue | 18:49 |
| mnaser | i think that was limewire fungi :P | 18:50 |
| mordred | fungi, clarkb: wanna +A that https://review.openstack.org/#/c/574746/ and its parent? | 18:50 |
| * mnaser holds useful bug fixes hostages by dependencies | 18:50 | |
| mnaser | :P | 18:50 |
| mordred | mnaser: nicely played | 18:50 |
| * mordred is thinking we should maybe split those vendor profiles out into their own repo kind of like service-types-authority | 18:51 | |
| clarkb | mordred: done | 18:51 |
| openstackgerrit | Gage Hugo proposed openstack-infra/system-config master: Add statusbot to #openstack-helm https://review.openstack.org/565887 | 18:51 |
| mordred | so that we can have a "you should always have the latest version of this" story that is separate from version-of-library | 18:51 |
| openstackgerrit | Gage Hugo proposed openstack-infra/system-config master: Add statusbot to #openstack-security https://review.openstack.org/565888 | 18:51 |
| fungi | oh, right, limelight was a rush song ;) | 18:51 |
| mnaser | mordred: that's actually a good idea. this doesn't need to be gated across all of openstacksdk for changes like that | 18:51 |
| mnaser | and probably doesnt need to be tied into the openstacksdk release cycle | 18:52 |
| mnaser | "hi we just upgraded our cloud but please wait for openstacksdk to catch up on our vendor profile" :p | 18:52 |
| mordred | exactly | 18:52 |
| fungi | mordred: separately from getting openstacksdk upgrades going, i'm still seeing the same traceback with a freshly recreated venv containing openstacksdk==0.14.0 | 18:53 |
| mordred | fungi: yah - likely due to the bug mnaser's patch fixes | 18:53 |
| mordred | but lemme check | 18:53 |
| *** armaan has joined #openstack-infra | 18:54 | |
| lseki | hi, I noticed that most pylint jobs are still getting timeout, and the logs are not very informative... | 18:55 |
| mnaser | lseki: which project? | 18:55 |
| clarkb | lseki: can you link to an example? | 18:55 |
| lseki | cinder | 18:55 |
| mnaser | lseki: got a specific change? | 18:55 |
| fungi | does it time out every run, or just occasionally? | 18:55 |
| fungi | do runs which don't time out take almost as long as the timeout is set? | 18:56 |
| lseki | e.g. https://review.openstack.org/#/c/565179/ | 18:56 |
| lseki | seems that almost every patch is getting timeout since saturday | 18:56 |
| mnaser | lseki: how long does it take to run locally for you? | 18:57 |
| clarkb | lseki: its running some scrpt in cinder and that scipt isn't completeing in the allotted time | 18:58 |
| clarkb | lseki: I would run it locally and/or add set -x and other verbose logging to the job to get more output | 18:58 |
| lseki | mnaser: the same happens here with 50min timeout, so I'm testing it with a larger timeout | 18:58 |
| fungi | looks like the last pylint job which succeeded on that change took 26 minutes, and the ones which are timing out do so at 30 minutes | 18:58 |
| clarkb | tools/lintstack.sh | 18:58 |
| fungi | so odds are it just needs a longer timeout | 18:58 |
| clarkb | or it is trying to solve the halting problem and will never finish | 18:59 |
| mordred | it takes 30 minutes to run pylint? | 18:59 |
| *** heyongli has quit IRC | 18:59 | |
| lseki | clarkb, mnaser: on my machine it took 1h30, weird | 18:59 |
| smcginnis | Some of those pylint jobs take an exceptional amount of time for some reason. | 18:59 |
| *** heyongli has joined #openstack-infra | 18:59 | |
| smcginnis | I thought it got better after I upgraded to the lastest and switched to py3 for execution. | 18:59 |
| mnaser | lseki: that means there might be something going terribly wrong, you might want to run that with strace or something and see if it's getting stuck somewhere | 18:59 |
| smcginnis | lseki: For that particular patch, it might help to rebase. | 19:00 |
| smcginnis | lseki: It does not have my updates. | 19:00 |
| smcginnis | Run time was cut by about 10 minutes based on my unscientific observations. | 19:01 |
| clarkb | smcginnis: zuul auto merges you into the target branch before running a rebase should be a noop as far as pylint is ocncerend | 19:01 |
| fungi | smcginnis: keep in mind that zuul tests the change by merging it to the target branch's tip | 19:01 |
| mnaser | clarkb: http://paste.openstack.org/show/723738/ this is the current jenkins equiv of 'ready-script' | 19:01 |
| mnaser | for ubuntu | 19:01 |
| smcginnis | Hmm | 19:01 |
| fungi | though as for running locally, that could make a difference | 19:01 |
| clarkb | mnaser: thanks. I think I've semi volunteered to delete as much ofthat that isn't jenkins specific as possible | 19:01 |
| smcginnis | And since that job just calls tox, it should pick up the rebased changes in tox.ini, right? | 19:01 |
| clarkb | smcginnis: yes it is running out of the git repo | 19:02 |
| smcginnis | OK, thanks. | 19:02 |
| mnaser | clarkb: fun.. let me know if you need help, i think unattended upgrades is useless in single use slaves, git/make/gcc are probably needed, docker install probably the only needed thing | 19:02 |
| smcginnis | lseki: That's not your problem there though since that job is non-voting. | 19:02 |
| smcginnis | It's a problem, but not the problem giving you a zuul -1. | 19:02 |
| smcginnis | s/you/that patch/ | 19:02 |
| clarkb | mnaser: well .ci/setup.sh already installs most of that stuff but then is completely untested because the jenkins jobs do it out of band (and .ci/setup.sh installs a non working version of docker as a result) | 19:03 |
| clarkb | mnaser: idea would be to make .ci/setup.sh actually setup ci and be tested by running it | 19:03 |
| lseki | smcginnis: well, the same happens on my downstream ci... I'll make it non-voting for now | 19:03 |
| mnaser | clarkb: i think that's how it should be, so we just call one shell script | 19:03 |
| clarkb | mnaser: ++ | 19:03 |
| mnaser | i think the fact it's not working is a by product of the problematic 'jenkins' prep script | 19:03 |
| clarkb | mnaser: yup | 19:04 |
| clarkb | mnaser: that and runtime asserts a version of docker that doesn't work for ci | 19:04 |
| clarkb | mnaser: so we are testing a version that we don't assert as supported and the asserted version does not work :/ | 19:04 |
| mnaser | clarkb: which confused you and i'm sure many other possible downstream users | 19:05 |
| smcginnis | lseki: You're running pylint on your third party ci? | 19:05 |
| mnaser | clarkb: i'd love to help with some of that but i am overwhelmed with stuff right now :( | 19:05 |
| lseki | smcginnis: yes | 19:05 |
| clarkb | mnaser: ++ | 19:05 |
| smcginnis | lseki: Just for fun? | 19:05 |
| clarkb | mnaser: no worries, I'm slowly making my way through it. I'm hopefully if I can just get it to the point were it works then others in kata land will want to get involved | 19:05 |
| clarkb | we'll see | 19:05 |
| mnaser | clarkb: it would be nice if we can show something off in the call next week | 19:06 |
| mnaser | i think once you solve this though the rest will be a breeze to implement | 19:06 |
| lseki | smcginnis: it was already there when I joined the team | 19:06 |
| *** camunoz has joined #openstack-infra | 19:07 | |
| mordred | fungi: aha! there is actually a bug in the script | 19:07 |
| lseki | smcginnis: you guys don't run pylint? | 19:07 |
| tpsilva | smcginnis: we have a downstream gerrit and CI that tests all our internal patches | 19:07 |
| tpsilva | before they go upstream | 19:07 |
| smcginnis | tpsilva: Ah, makes sense. | 19:07 |
| mordred | I feel like this bug was fixed in upstream ansible | 19:08 |
| smcginnis | lseki: The only thing third party CI needs to be running for Cinder requirements is tempest volume tests against actual storage backends. | 19:08 |
| lseki | smcginnis: I see | 19:09 |
| *** heyongli has quit IRC | 19:09 | |
| *** heyongli has joined #openstack-infra | 19:10 | |
| fungi | mordred: a bug in our scripts? unpossible! | 19:11 |
| *** florianf has quit IRC | 19:13 | |
| lseki | smcginnis: well, I'll set my pylint job as non-voting for now, and investigate this slowness later | 19:13 |
| lseki | thank you all! | 19:13 |
| *** myoung|lunch is now known as myoung | 19:16 | |
| clarkb | mnaser: heh and now launchpad ppa fails to hvae its gpg key found on ubuntu keyserver | 19:18 |
| *** heyongli has quit IRC | 19:20 | |
| clarkb | if I do a manual search for the key it shows up so I'm hoping this is just a networking/service blip | 19:20 |
| *** heyongli has joined #openstack-infra | 19:20 | |
| mordred | fungi: https://github.com/ansible/ansible/pull/41664 | 19:20 |
| mordred | Shrews: ^^ I know you're mostly out today - but if you could click an approve button there ^^ | 19:20 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Extend Cinder pylint timeout https://review.openstack.org/576238 | 19:20 |
| fungi | mordred: wow, now that's a fun rabbit hole | 19:22 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Add NODEPOOL_PERCONA_PROXY to mirror_info https://review.openstack.org/576156 | 19:23 |
| mnaser | clarkb: i'm sorry :p | 19:23 |
| clarkb | mnaser: looks like it may be running properly now on a rerun | 19:23 |
| mordred | Shrews: and this: https://github.com/ansible/ansible/pull/41666 | 19:23 |
| fungi | mordred: i'm watching our zuul test your ansible pr... how awesome! | 19:24 |
| mordred | the backport to stable 2.6 | 19:24 |
| mordred | fungi: \o/ | 19:24 |
| *** sshnaidm|afk has joined #openstack-infra | 19:24 | |
| mordred | fungi: right? how cool is that? | 19:24 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Add NODEPOOL_PERCONA_PROXY to mirror_info https://review.openstack.org/576156 | 19:24 |
| clarkb | oh I tested --list with a specific cloud for expediency | 19:24 |
| mnaser | config-core: ^ this was a +W'd but I had to make a small revision | 19:24 |
| mnaser | and added depends for the config fix to avoid the recheck trip | 19:24 |
| mordred | clarkb: yah. me too | 19:24 |
| mordred | clarkb: silly me | 19:24 |
| mordred | clarkb: the real fun part is - this ALL goes away with the new inventory plugin | 19:25 |
| Shrews | mordred: cant until shippable finishes but will check on them later | 19:25 |
| mordred | Shrews: awesome. thanks. it's a stupid patch to fix a stupid bug I caused :) | 19:25 |
| mordred | clarkb: since inventory sources have a central caching mechanism in ansible | 19:25 |
| *** owalsh has quit IRC | 19:26 | |
| clarkb | mordred: launch-node.py still runs outside of that but cool | 19:26 |
| *** lifeless has joined #openstack-infra | 19:26 | |
| *** owalsh has joined #openstack-infra | 19:26 | |
| mordred | clarkb: yah - but the bug in this case was in the section of the old inventory script that was made irrelevant by the new plugin system ... the underlying issue for launch-node is that it runs the inventory. sigh. rabbits. holes. | 19:27 |
| *** heyongli has quit IRC | 19:30 | |
| *** heyongli has joined #openstack-infra | 19:30 | |
| clarkb | mordred: fungi do we need to revert? | 19:34 |
| fungi | clarkb: i think i can hack around it in the short term. but yeah having a working script soonish would be nice. i expect the next ansible release isn't too far off? | 19:35 |
| clarkb | fungi: we install it from source regardless of release so mostly just need it to be reviewed and merged | 19:35 |
| *** jesslampe has joined #openstack-infra | 19:36 | |
| mordred | yah. as soon as CI finishes we can land it | 19:37 |
| clarkb | but we can't see shippables queuing unfortauntely | 19:37 |
| fungi | ahh | 19:38 |
| mnaser | clarkb, mordred: can we cancel https://review.openstack.org/#/c/576184/ out of gate? | 19:38 |
| mnaser | http://logs.openstack.org/56/576156/3/check/openstack-infra-multinode-integration-centos-7/2510f11/job-output.txt.gz#_2018-06-18_19_36_50_680883 | 19:39 |
| * fungi wonders what the timeline is for reimplementing shippable on zuul ;) | 19:39 | |
| mnaser | it failed, i forgot to change it to `command` | 19:39 |
| *** edmondsw has quit IRC | 19:39 | |
| fungi | mnaser: it'll be cancelled and restarted when you push up the fixed patchset | 19:39 |
| mnaser | GUESS I GOTTA HURRY UP | 19:39 |
| mnaser | :p | 19:39 |
| fungi | mnaser: or you can set it to wip | 19:39 |
| mordred | mnaser: dropped a -2 | 19:39 |
| fungi | zuul won't merge a wip change | 19:39 |
| mnaser | thanks | 19:40 |
| clarkb | if it failed in check it won't go to gate | 19:40 |
| mnaser | also | 19:40 |
| mnaser | it passed check.. | 19:40 |
| clarkb | oh weird | 19:40 |
| *** heyongli has quit IRC | 19:40 | |
| *** heyongli has joined #openstack-infra | 19:40 | |
| mnaser | devstack-gate doesnt run the `openstack-infra-multinode-integration-centos-7` job | 19:40 |
| *** yamamoto has joined #openstack-infra | 19:40 | |
| mnaser | so the changes we did were never really tested | 19:40 |
| mnaser | until i did a depends-on change inside https://review.openstack.org/#/c/576156/ | 19:41 |
| clarkb | mnaser: well the jobs against the zuul config do set up a multiode setup and check that things run successfully and do a ping test iirc | 19:41 |
| clarkb | mnaser: its not a full d-g but should be fairly good coverage | 19:41 |
| fungi | we tried to avoid adding every devstack-gate job under the sun to run on devstack-gate changes, because otherwise nothing would ever merge | 19:41 |
| mnaser | clarkb: what do you think of revising the job and adding "failed_when: foo.exit_code != 0 and 'RTNETLINK' not in foo.stderr" ? | 19:43 |
| mnaser | or 'File exists' in foo.stderr | 19:43 |
| *** owalsh has quit IRC | 19:44 | |
| clarkb | mnaser: it should already fail when the exit code is not 0 | 19:44 |
| clarkb | mnaser: that is why it failed in your example | 19:44 |
| mnaser | clarkb: we can control the failure scenario with failed_when to add that if 'File exists' is return in the error message | 19:45 |
| mnaser | we assume that the ip already exists | 19:45 |
| mnaser | and *not* fail | 19:45 |
| mnaser | hence it fails when exit code is not 0 *and* 'file exists' is not in stderr | 19:45 |
| clarkb | mnaser: cant we just run the ip addr only if the ip is not set on the interface? like the existing code does? I think that is easier to understand | 19:45 |
| *** yamamoto has quit IRC | 19:45 | |
| mnaser | i guess we can add `become` to it | 19:46 |
| mnaser | cause that seems to be the only difference | 19:46 |
| mnaser | seems.. dirty though :( | 19:46 |
| clarkb | mnaser: I think your fix would've worked? I feel like I am missing some important piece of info | 19:46 |
| clarkb | check if ip is on interface if not on interface add it | 19:46 |
| clarkb | the first thing does not need become, the second thing does. | 19:46 |
| mnaser | clarkb: except `/bin/sh: ip: command not found` is still a thing and the only difference at this point between the two is `become` | 19:46 |
| mnaser | (we thought it was shell hence my little rewrite) | 19:47 |
| clarkb | oh got it. I think in that case we should set the path probably | 19:47 |
| clarkb | it is in /bin/ip on suse and /sbin/ip on ubuntu | 19:48 |
| mnaser | i think the path differents between centos and ubuntu | 19:48 |
| mnaser | and that | 19:48 |
| clarkb | /usr/sbin/ip | 19:48 |
| mnaser | funnnnnn | 19:48 |
| clarkb | on cnetos | 19:48 |
| mnaser | im gonna guess /usr/sbin is not in PATH and voila | 19:48 |
| clarkb | ya can just add all three of those to $PATH and call it good problably | 19:48 |
| mordred | yah | 19:48 |
| *** owalsh has joined #openstack-infra | 19:48 | |
| *** germs is now known as GotOneHandCutOff | 19:50 | |
| *** GotOneHandCutOff is now known as germs | 19:50 | |
| *** heyongli has quit IRC | 19:50 | |
| *** heyongli has joined #openstack-infra | 19:51 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/devstack-gate master: Add PATH to `ip` command execution https://review.openstack.org/576184 | 19:54 |
| mnaser | clarkb, mordred: above being tested in https://review.openstack.org/#/c/576156/3 | 19:54 |
| *** owalsh has quit IRC | 19:55 | |
| *** owalsh has joined #openstack-infra | 19:56 | |
| clarkb | mnaser: fails on k8s now http://logs.openstack.org/74/74/9e1f7eede869376225a5f4b35c8f8efdfabb7426/third-party-check/kata-runsh/7bef64d/job-output.txt.gz#_2018-06-18_19_50_59_563694 | 19:57 |
| clarkb | mnaser: any idea if that is working on their jenkins? I can't find evidence of it owrking after a quick check | 19:57 |
| mnaser | clarkb: lol.. http://jenkins.katacontainers.io/job/kata-containers-proxy-ubuntu-16-04-PR/ | 19:58 |
| mnaser | nothing other than your changes | 19:59 |
| clarkb | mnaser: ya I was told to touch proxy because nothing else looks at it | 19:59 |
| clarkb | its very quiet repo | 19:59 |
| mnaser | http://jenkins.katacontainers.io/job/kata-containers-proxy-ubuntu-16-04-PR/8/console i cant tell you if it works or not | 19:59 |
| mnaser | `ERROR: checkcommits failed. See the document below for help on formatting commits for the project.` | 20:00 |
| *** heyongli has quit IRC | 20:01 | |
| clarkb | wow I think my lack of a signed-off-by is making it mad | 20:01 |
| *** heyongli has joined #openstack-infra | 20:01 | |
| clarkb | except why wouldn't that also fail in zuul | 20:01 |
| mnaser | i was curious about that too | 20:01 |
| mnaser | maybe zuul isn't fetching the right commit and fetching master instead? | 20:01 |
| clarkb | mnaser: well yes, master is the right commit from zuul's perspective | 20:02 |
| clarkb | thta may be it though | 20:02 |
| *** armaan has quit IRC | 20:02 | |
| clarkb | let me add a signed off by and repush | 20:02 |
| *** e0ne has quit IRC | 20:02 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Capture logs and install docker in kata-runsh job https://review.openstack.org/576178 | 20:02 |
| *** armaan has joined #openstack-infra | 20:02 | |
| clarkb | ok new push done lets see if they produce consistent results now | 20:04 |
| mnaser | clarkb: i think they'll be excited to see how we do log publishing | 20:04 |
| mnaser | this morning complaints about "it's so hard to look at multi-mb log files in jenkins and we never had time to do logging" | 20:05 |
| clarkb | mnaser: hopefully my change above illustrates its relatively easy once the framework is in place | 20:05 |
| mnaser | yep! | 20:05 |
| clarkb | I'm going to grab lunch now | 20:06 |
| *** felipemonteiro_ has quit IRC | 20:08 | |
| *** felipemonteiro__ has joined #openstack-infra | 20:08 | |
| *** akhilaki has quit IRC | 20:10 | |
| *** akhilaki has joined #openstack-infra | 20:11 | |
| *** heyongli has quit IRC | 20:11 | |
| *** heyongli has joined #openstack-infra | 20:11 | |
| *** dave-mccowan has quit IRC | 20:12 | |
| *** e0ne has joined #openstack-infra | 20:12 | |
| fungi | mordred: looks like zuul and shippable both verified https://github.com/ansible/ansible/pull/41664 | 20:12 |
| *** dave-mccowan has joined #openstack-infra | 20:13 | |
| mnaser | ugh | 20:14 |
| mnaser | still happening. | 20:14 |
| mnaser | `/bin/sh: ip: command not found` | 20:14 |
| mnaser | would anyone be kind enough to dig into this with me: https://review.openstack.org/#/c/576184/ | 20:14 |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config master: OpenstackId Release 1.0.24 https://review.openstack.org/576248 | 20:15 |
| fungi | mnaser: which distro(s) is/are failing? | 20:16 |
| mnaser | fungi: centos and fedora | 20:16 |
| fungi | usr/sbin | 20:16 |
| mnaser | according to clark, it's in /usr/sbin/ip | 20:16 |
| fungi | it's in your path so should be working, yeah | 20:16 |
| mnaser | yeah i put that there | 20:16 |
| mnaser | so the command below it works | 20:17 |
| mnaser | only difference is `become` | 20:17 |
| fungi | become is using sudo, i expect | 20:17 |
| clarkb | fungi yes | 20:17 |
| mnaser | would someone be able to ssh to any nodepool vms and try | 20:18 |
| mnaser | 'ip addr show dev em1' or something | 20:18 |
| mnaser | and then `sudo ip addr show dev em1` | 20:18 |
| *** e0ne has quit IRC | 20:18 | |
| fungi | i happen to have a couple held centos-7 nodes already | 20:18 |
| mnaser | no better use for them then :) | 20:19 |
| fungi | mnaser: http://paste.openstack.org/show/723745/ | 20:20 |
| fungi | however, behavior of ansible remote execution may differ from an interactive shell | 20:21 |
| mnaser | fungi: fun.. i was hoping that'd fail, but yeah | 20:21 |
| *** heyongli has quit IRC | 20:21 | |
| *** heyongli has joined #openstack-infra | 20:21 | |
| mordred | isn't there a sudo thing that doesn't persist path settings across the sudo invocation? | 20:22 |
| fungi | mordred: yeah, it's called sudo | 20:22 |
| mordred | yah | 20:22 |
| mnaser | mordred: but wouldn't it be the opposite, in that case, the command that is NOT sudo-ing doesnt work | 20:22 |
| mordred | oh | 20:22 |
| mnaser | s/that/this/ | 20:22 |
| mordred | duh. I'm just dumb | 20:23 |
| fungi | rather, it's a default behavior (you probably mean the config setting which allows you to whitelist what parts of the environment you don't want overridden) | 20:23 |
| *** Goneri has quit IRC | 20:23 | |
| mnaser | well i dont blame ya usually its the opposite | 20:23 |
| mnaser | usually sudo is restrictive and makes things disappear | 20:23 |
| mnaser | rather than things appear when using sudo | 20:23 |
| fungi | env_reset is what blows away the environment when sudo is invoked. env_keep lets you specify a list of envvars you don't want overridden | 20:24 |
| mnaser | http://zuul.openstack.org/builds.html?job_name=openstack-infra-multinode-integration-centos-7 | 20:24 |
| fungi | the /etc/sudoers on a centos-7 node does not seem to specify PATH as part of the env_keep list (and for good reason, that's one of the most dangerous envvars you could let leak through) | 20:24 |
| mnaser | last time it actually ran was 12 days ago | 20:24 |
| mnaser | ansible 2.5 came on the 7th | 20:26 |
| * mnaser suspects it | 20:26 | |
| fungi | well, and we switched zuul to use it roughly a week ago? | 20:26 |
| fungi | or was that 2.4? | 20:27 |
| mnaser | 2018-06-07 15:32:32 UTC Zuul update for Ansible 2.5 in progress. Scheduler crashed as unexpected side effect of pip upgrade. Will be back and running shortly. | 20:27 |
| fungi | yeah, was 2.5 | 20:27 |
| mnaser | https://docs.ansible.com/ansible/2.5/porting_guides/porting_guide_2.5.html#shell something was changed here i guess but not enough for me to figure out whats going on | 20:27 |
| *** kgiusti has left #openstack-infra | 20:28 | |
| fungi | i suppose you could just add become in there as a workaround? but ugly | 20:28 |
| mnaser | ugly but i have no idea :< | 20:29 |
| fungi | https://docs.ansible.com/ansible/latest/plugins/shell/sh.html seems to be the current docs for that shell plugin | 20:30 |
| mnaser | the plugin system was introduced in 2.5 | 20:30 |
| mnaser | im looking at https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/shell/sh.py | 20:31 |
| fungi | mnaser: does environment need to be indented in your change? | 20:31 |
| *** heyongli has quit IRC | 20:31 | |
| mnaser | i see environment is blank | 20:31 |
| mnaser | i wonder if with the plugin change, env is empty hence it doesnt find it | 20:32 |
| *** heyongli has joined #openstack-infra | 20:32 | |
| mnaser | also yes fungi | 20:32 |
| mnaser | https://docs.ansible.com/ansible/devel/reference_appendices/faq.html#how-can-i-set-the-path-or-any-other-environment-variable-for-a-task-or-entire-playbook | 20:32 |
| mnaser | https://github.com/ansible/ansible/blob/06f73ad578d840c7ea5875b7cd4ffd08e2d9d0e8/lib/ansible/plugins/shell/__init__.py#L41-L46 | 20:33 |
| mnaser | looks like that's the default env | 20:33 |
| fungi | mmm, no your patch looks to match the way parameters are passed in the other examples | 20:33 |
| fungi | i guess it's just a nuance of ansible's syntax that environment ends up looking like a parameter to that block list item rather than to the shell plugin | 20:34 |
| mnaser | i guess i can try and replicate locally | 20:34 |
| fungi | i was going to see if there's an open gh issue on this, but... 3633 _open_ issues | 20:35 |
| mnaser | fungi: can you play around on those held centos nodes or they have to remain pristine? | 20:36 |
| fungi | they're held for debugging a tripleo multinode job issue | 20:37 |
| mnaser | oh actually if you have anything with ansible 2.5 | 20:37 |
| mnaser | and can run against those machines | 20:37 |
| mnaser | (they won't make any changes | 20:37 |
| fungi | i can make a temporary virtualenv on one | 20:39 |
| fungi | does running ansible over the loopback still work these days? | 20:40 |
| mnaser | http://paste.openstack.org/show/723746/ | 20:40 |
| mnaser | fungi: it does, or you can run it remotely | 20:40 |
| *** armaan has quit IRC | 20:40 | |
| mnaser | just run ansible-playbook that.yaml and it should give us some helpful info | 20:40 |
| *** armaan has joined #openstack-infra | 20:41 | |
| *** yamamoto has joined #openstack-infra | 20:41 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Filter out link-local ipv6 address in loadbalancer https://review.openstack.org/576262 | 20:41 |
| mordred | fungi: I merged the ansible inventory fix - so it should start working for us again on next puppeting | 20:41 |
| fungi | ImportError: No module named tripleo_common | 20:41 |
| *** heyongli has quit IRC | 20:42 | |
| mnaser | nice | 20:42 |
| mnaser | fun times | 20:42 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Update Gemfile for Zuulv3 https://review.openstack.org/571987 | 20:42 |
| fungi | i wonder if ansible is reading some configuration from the node since it ran a tripleo job? | 20:42 |
| *** heyongli has joined #openstack-infra | 20:42 | |
| mnaser | mordred: if you have ansible locally, can you run the paste above against a held machine remotely? and fungi it probably has some stuff wired in | 20:42 |
| * mnaser would do this if he had any sort of access | 20:42 | |
| mordred | mnaser: yah - one sec. wat's the held machine? | 20:43 |
| mnaser | fungi: if you don't mind helping with that? ^ | 20:43 |
| fungi | mordred: 162.253.55.199 | 20:43 |
| mnaser | oh that's a familiar ip range | 20:43 |
| fungi | indeed | 20:43 |
| mnaser | mordred: and if you have a few more secs maybe run 42 to 48 from there .. https://review.openstack.org/#/c/576184/3/playbooks/roles/ovs_vxlan_bridge_peers/tasks/main.yaml (with -vvv?) | 20:44 |
| mnaser | sorry for bothering you, i know you're busy :< | 20:44 |
| fungi | i'm getting ansible installed into a virtualenv on my workstation to see if i can figure out how to do it | 20:45 |
| fungi | what's the syntax to run it against a remote machine? just replace "localhost" with "root@162.253.55.199" in that sample playbook? | 20:46 |
| mordred | you need a hostfile | 20:46 |
| *** yamamoto has quit IRC | 20:46 | |
| mordred | I made a tempdir - then put this into an ansible.cfg file: | 20:47 |
| mordred | [defaults] | 20:47 |
| mordred | hostfile=/tmp/fooasdf/hosts | 20:47 |
| mordred | then in /tmp/fooasdf/hosts is this line: | 20:47 |
| mordred | testnode ansible_host=162.253.55.199 ansible_user=root | 20:47 |
| mordred | and I replaced localhost with 'testnode' in the playbook | 20:47 |
| mordred | then ran "ansible-playbook -v foo.yaml" | 20:47 |
| fungi | can't just tell it from the cli i guess. i'll give that a shot | 20:47 |
| fungi | how do you tell ansible-playbook where to find the temporary ansible.cfg? via an envvar? | 20:48 |
| mordred | it'll pick it up if it's adjacent to the playbook | 20:48 |
| mordred | or, rather, in the current directoy where you riun ansible-playbook | 20:48 |
| mordred | mnaser: http://paste.openstack.org/show/723747/ | 20:49 |
| mnaser | mordred: ugh, so it seems to override just fine | 20:49 |
| mnaser | mine running the actual 'ip a' call that's failing ? | 20:49 |
| mnaser | s/mine/mind/ | 20:49 |
| fungi | the cowsay is amusing | 20:50 |
| mnaser | fungi: only if you have cowsay installed :D | 20:50 |
| fungi | don't all sane posix systems have cowsay? | 20:50 |
| mordred | infra-root: I manually updated /opt/ansible on puppetmaster and copied the fixed inventory script in /etc/ansible/hosts/openstack_inventory | 20:51 |
| fungi | that's, like, as standard as fortune or banner or figlet | 20:51 |
| mordred | since the break to the script meant it wouldn't be able to update itself | 20:51 |
| mordred | fungi: my laptop does not have cowsay | 20:51 |
| mordred | fungi: largely because I do not want cows in my ansible output ;) | 20:51 |
| fungi | mordred: next you're going to say there's no nethack on your laptop either | 20:51 |
| *** heyongli has quit IRC | 20:52 | |
| *** dave-mcc_ has joined #openstack-infra | 20:52 | |
| *** heyongli has joined #openstack-infra | 20:52 | |
| mordred | Shrews: I landed the inventory PR to devel -but I did not self-approve the one that's for stable-2.6 | 20:52 |
| mordred | Shrews: when you get a chance, I'd love to get that in to make sure we don't release 2.6 with a broken inventory script | 20:52 |
| *** dave-mccowan has quit IRC | 20:54 | |
| fungi | mnaser: this leaves me wondering whether there's something about how that block list entry is defined | 20:54 |
| Shrews | mordred: apparently, i am not authorized to merge to 2.6 | 20:54 |
| mordred | oh. interesting | 20:54 |
| Shrews | *shrug* | 20:54 |
| mnaser | fungi: i am wondering the same too | 20:54 |
| mnaser | i wonder if the fact it's in a block maybe affects it?! | 20:54 |
| * mnaser has spent too much time on this and is tempted to push adding a `become: true` and call ita day | 20:55 | |
| *** trown is now known as trown|ruck | 20:55 | |
| fungi | i'm trying wrapping it all in a block now | 20:56 |
| fungi | nope, works just like without being in a block | 20:57 |
| fungi | so maybe it's something to do with how zuul is invoking ansible? | 20:57 |
| mnaser | fungi: what about running the 'ip a' task? | 20:57 |
| mnaser | the same one that is failing | 20:57 |
| *** owalsh_ has joined #openstack-infra | 20:57 | |
| *** gfidente|afk has quit IRC | 20:58 | |
| fungi | ip addr show as the shell command gets me the expected network details | 20:59 |
| mnaser | oh you know what | 20:59 |
| mnaser | we're running as root | 20:59 |
| mnaser | what if you change ansible_user=zuul | 20:59 |
| fungi | i guess i'll need to set up authorized_keys but should be doable | 20:59 |
| fungi | connecting as zuul now | 21:00 |
| *** owalsh has quit IRC | 21:01 | |
| *** owalsh- has joined #openstack-infra | 21:01 | |
| *** owalsh- is now known as owalsh | 21:01 | |
| fungi | bingo! | 21:01 |
| fungi | /bin/sh: ip: command not found | 21:01 |
| fungi | getting the path details now | 21:02 |
| *** heyongli has quit IRC | 21:02 | |
| *** owalsh_ has quit IRC | 21:02 | |
| *** trown|ruck is now known as trown|outtypewww | 21:02 | |
| *** heyongli has joined #openstack-infra | 21:02 | |
| mordred | fungi: woot! | 21:03 |
| mnaser | yay for repro | 21:04 |
| mnaser | fungi: if you run ansible-playbook with -vvv you will get way too much info | 21:04 |
| mnaser | that might be useful | 21:04 |
| *** esarault has quit IRC | 21:05 | |
| fungi | well, i can already see it's overriding the path in the second shell task | 21:05 |
| fungi | the first task has PATH=/usr/local/bin:/usr/bin | 21:06 |
| fungi | which explains why i don't get a path to the ip command | 21:06 |
| fungi | as it's in /usr/sbin/ | 21:06 |
| mnaser | weird | 21:07 |
| fungi | the second task has PATH=/bin:/sbin:/usr/sbin:/usr/bin though | 21:07 |
| fungi | so the environment override seems to work there | 21:07 |
| fungi | trying it with the second env task changed to ip addr show this time | 21:07 |
| fungi | more bad news. the environment override seems to solve the path problem for me. i get ip addr show output | 21:08 |
| fungi | so all that to say... i think 576184 should be the solution so i'm even more baffled that it's not working | 21:11 |
| clarkb | fungi: mnaser is that an override or an append? because grep isn't likely to be in those three dirs | 21:12 |
| clarkb | could it be that is what is failing? | 21:12 |
| *** heyongli has quit IRC | 21:12 | |
| fungi | oh! | 21:12 |
| mnaser | it did say ip: not found | 21:12 |
| mnaser | but maybe i wasnt paying attention | 21:12 |
| mnaser | that is a reasonable explanation tho | 21:13 |
| *** heyongli has joined #openstack-infra | 21:13 | |
| mnaser | i can make it append | 21:13 |
| fungi | well, still, good eye. at least on the held centos-7 node here, it's /usr/bin/grep | 21:13 |
| mnaser | but it looks like the override isn't working? | 21:13 |
| fungi | which job was failing with this? do you have a depends-on exercising it? | 21:14 |
| mnaser | fungi: https://review.openstack.org/#/c/576156/ | 21:14 |
| * mnaser hides hoping no one notices his old-school depends-on | 21:15 | |
| fungi | meh | 21:15 |
| *** ldnunes has quit IRC | 21:16 | |
| fungi | http://logs.openstack.org/56/576156/3/check/openstack-infra-multinode-integration-centos-7/4567bfd/ara-report/result/70cdbcee-2ddb-424d-aead-6f155f247ac8/ | 21:17 |
| fungi | so i guess that's the breakage manifesting | 21:17 |
| fungi | http://logs.openstack.org/56/576156/3/check/openstack-infra-multinode-integration-centos-7/4567bfd/ara-report/ | 21:18 |
| fungi | or there's the earlier one | 21:18 |
| fungi | sorry, http://logs.openstack.org/56/576156/3/check/openstack-infra-multinode-integration-centos-7/4567bfd/ara-report/result/a9caf4c1-e401-471e-aed4-c2d81ad43b67/ | 21:18 |
| fungi | definitely complaining that it's not finding 'ip' | 21:19 |
| fungi | no... these are different tasks | 21:20 |
| fungi | these are all under when: bridge_configure_address | 21:21 |
| fungi | not when: set_ips | 21:21 |
| *** lifeless has quit IRC | 21:21 | |
| *** lifeless has joined #openstack-infra | 21:22 | |
| clarkb | ah multiple places need an update? | 21:22 |
| fungi | mnaser: i think your change is setting this in a different place than things are failing in the example logs from the dependent change | 21:22 |
| *** heyongli has quit IRC | 21:23 | |
| fungi | just based on pulling the task definitions up in ara | 21:23 |
| *** heyongli has joined #openstack-infra | 21:23 | |
| fungi | and noticing that they're similar but subtly different (different condition names, and no environment override specified in the playbook) | 21:23 |
| fungi | unless i'm looking at the wrong job? | 21:24 |
| *** armaan has quit IRC | 21:27 | |
| *** armaan has joined #openstack-infra | 21:28 | |
| fungi | mnaser: i have a feeling you actually wanted to fix this in https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/multi-node-bridge/tasks/switch.yaml | 21:28 |
| mnaser | or maybe both i guess | 21:29 |
| fungi | well, i don't see the roles from devstack-gate getting used at all | 21:29 |
| mnaser | wait | 21:29 |
| mnaser | you're right | 21:29 |
| mnaser | so are the devstack-gate roles actually ever getting used? :X | 21:29 |
| fungi | at least not in the openstack-infra-multinode-integration-centos-7 job | 21:29 |
| fungi | mnaser: i think the devstack-gate roles were legacy | 21:29 |
| mnaser | shucks. | 21:30 |
| mnaser | i'm sorry | 21:30 |
| clarkb | the code in devstack-gate is run by devstack-gate in a nested ansible run | 21:30 |
| clarkb | its legacy deprecated code | 21:30 |
| fungi | one nice thing about ara, not only does it show you copies of the yaml files you ran, it (with the way we're arranging them with zuul) tells you exactly what repos to find them in | 21:30 |
| clarkb | as a heads up I am writing down my kata thoughts and info at https://etherpad.openstack.org/p/kata-on-zuul-notes | 21:31 |
| fungi | http://logs.openstack.org/56/576156/3/check/openstack-infra-multinode-integration-centos-7/4567bfd/ara-report/file/a5fa34b9-af7f-492f-b42d-b54e10d9ad7e/#line-12 | 21:31 |
| mnaser | fungi: thank you and sorry for using up your time on my screw up | 21:31 |
| fungi | mnaser: good news is, at least i think we've very thoroughly confirmed what the fix should be! ;) | 21:31 |
| mnaser | fungi: while you're there, do we know what paths to use for grep? | 21:31 |
| fungi | and no need to apologize | 21:31 |
| mordred | mnaser: if it makes you feel any better - I TOTALLY did the same thing (similar but different roles in 2 different repos) over the weekend and this morning | 21:31 |
| clarkb | current situation comparing zuul results to jenkins results is that it won't k8s because I don't have a fixes bug number in my commit message | 21:32 |
| mnaser | or you know what, i'll just append the existing path | 21:32 |
| fungi | mnaser: /usr/bin pretty much everywhere for grep, i think | 21:32 |
| mordred | mnaser: tore my hair out trying to figure out why fixes weren't fixing anythin g:) | 21:32 |
| mnaser | quick vote | 21:32 |
| mnaser | do we use explicit env | 21:32 |
| mnaser | or append to it | 21:32 |
| mordred | either is fine with me | 21:32 |
| mnaser | PATH: "{{ ansible_env.PATH }}:foo" is a thing i guess | 21:32 |
| *** heyongli has quit IRC | 21:33 | |
| mnaser | going to append to avoid breaking other possible distros | 21:33 |
| * mordred has, in fact, been working on getting a symlink created in a job log output for about 4 days now | 21:33 | |
| fungi | no preference. i expect if you just blanked /bin:/sbin:/usr/bin:/usr/sbin you'll have your bases covered for that task anyway | 21:33 |
| *** heyongli has joined #openstack-infra | 21:33 | |
| fungi | er, blanketed | 21:33 |
| mnaser | people's expectations of solving complex confusing issues: deep thinking sessions and serious thought | 21:35 |
| mnaser | most of the times: "oops i changed the wrong thing" | 21:35 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/zuul-jobs master: Add PATH to `ip` command execution https://review.openstack.org/576274 | 21:37 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Add NODEPOOL_PERCONA_PROXY to mirror_info https://review.openstack.org/576156 | 21:38 |
| fungi | mnaser: more accurately, deep thinking sessions and serious thought leading to excessive confusion before coming to the sudden realization that... oops i changed the wrong thing | 21:38 |
| mnaser | yep | 21:39 |
| mordred | mnaser: ++ | 21:39 |
| mordred | fungi: ++ | 21:39 |
| openstackgerrit | Merged openstack-infra/system-config master: OpenstackId Release 1.0.24 https://review.openstack.org/576248 | 21:41 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Finish switching to using new ansible inventory https://review.openstack.org/576275 | 21:42 |
| fungi | mordred: clarkb: we also still need that ^ in conjunction with the ansible inventory fix, right? | 21:42 |
| clarkb | fungi: yup | 21:42 |
| ianw | is that what was causing frickler's weird missing keys? | 21:43 |
| *** yamamoto has joined #openstack-infra | 21:43 | |
| *** heyongli has quit IRC | 21:43 | |
| ianw | clarkb: i'll look at a 200gb flavor. the admin is just under the other tenant, but i'll write the details (as i understand them) in the password file | 21:43 |
| ianw | s/tenant/project/ | 21:44 |
| *** heyongli has joined #openstack-infra | 21:44 | |
| clarkb | ianw: thanks. The zuul tenant is admin you mean? should probably make ci admin not zuul if that is possible | 21:44 |
| clarkb | ci == control plane, zuul == test jobs | 21:44 |
| *** eernst has joined #openstack-infra | 21:44 | |
| *** EmilienM is now known as EmilienM_PTO | 21:45 | |
| *** dklyle has quit IRC | 21:48 | |
| fungi | ianw: i have a feeling what was causing frickler's missing authorized_keys entry on test nodes was that we're not correctly rotating out the sshkey objects in the provider. openstack doesn't let you modify them, only create and delete | 21:48 |
| *** yamamoto has quit IRC | 21:49 | |
| *** dklyle has joined #openstack-infra | 21:50 | |
| *** slaweq has quit IRC | 21:52 | |
| *** heyongli has quit IRC | 21:53 | |
| fungi | so basically we need to do a 3-step dance between system-config and project-config where we create the new keyset and get it applied everywhere, then use it in nodepool, then delete the old one from system-config (and subsequently, manually delete it from all the providers as well) | 21:54 |
| *** heyongli has joined #openstack-infra | 21:54 | |
| fungi | seems like there ought to be a better way | 21:54 |
| clarkb | we could update ansible to do a get, compare, if not equal delete update process | 21:56 |
| fungi | yeah, worst case we blip a few node builds and recycle them if we hit a boot call while the keys object is nonexistent? | 21:56 |
| clarkb | only zuul nodes use that key, if key is deleted forshort period we'll fail to boot instances during that window which should be fine | 21:56 |
| fungi | ya, that | 21:57 |
| fungi | sounds entirely survivable | 21:57 |
| clarkb | ok kata jenkins also fails to k8s due to unable to connect to k8s apt repo | 22:01 |
| *** felipemonteiro_ has joined #openstack-infra | 22:01 | |
| clarkb | gonna guess that isn't a persistent issuethough | 22:01 |
| *** heyongli has quit IRC | 22:04 | |
| *** jesslampe has quit IRC | 22:04 | |
| *** heyongli has joined #openstack-infra | 22:04 | |
| *** rpioso is now known as rpioso|afk | 22:04 | |
| fungi | and something for which we have a/many solutions | 22:05 |
| *** felipemonteiro__ has quit IRC | 22:05 | |
| *** hasharAway has quit IRC | 22:07 | |
| *** myoung is now known as myoung|off | 22:07 | |
| *** hph^ has quit IRC | 22:09 | |
| *** bdodd has quit IRC | 22:10 | |
| *** eernst has quit IRC | 22:11 | |
| *** eernst has joined #openstack-infra | 22:13 | |
| *** heyongli has quit IRC | 22:14 | |
| *** heyongli has joined #openstack-infra | 22:14 | |
| mordred | fungi, clarkb: I mean, we _just_ did a keypair rotation - is it still a problem? | 22:15 |
| *** mriedem has quit IRC | 22:16 | |
| fungi | mordred: something might have prevented it from getting applied? not sure | 22:16 |
| mordred | I mean - when did the issue happen? | 22:16 |
| fungi | ~10:00z today according to scrollback | 22:17 |
| ianw | mordred: about 12 hours ago. frickler couldn't get into a held host | 22:17 |
| mordred | hrm | 22:18 |
| fungi | oh! though those nodes were booted held a few days ago, before we approved the new keyset | 22:18 |
| *** eernst has quit IRC | 22:18 | |
| fungi | which likely explains it | 22:18 |
| mordred | *phew* | 22:18 |
| mordred | I was about to start tearing things apart | 22:18 |
| *** eernst has joined #openstack-infra | 22:18 | |
| ianw | oh, i'm still on one of the hosts : 22:18:33 up 3 days, 10:02, 2 users, load average: 0.62, 0.46, 0.38 | 22:19 |
| fungi | like thursday or friday timeframe (nodepool probably remembers exactly when, i do not) | 22:19 |
| *** mriedem has joined #openstack-infra | 22:19 | |
| ianw | 162.253.55.151 | 22:19 |
| *** dingyichen has quit IRC | 22:20 | |
| mordred | yes. that's the old keypair | 22:20 |
| mordred | so that issue has now been fixed | 22:20 |
| ianw | ok, that explains that :) next problem | 22:20 |
| mordred | \o/ | 22:20 |
| *** jesslampe has joined #openstack-infra | 22:20 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Capture logs and install docker in kata-runsh job https://review.openstack.org/576178 | 22:22 |
| *** lseki has quit IRC | 22:24 | |
| *** heyongli has quit IRC | 22:24 | |
| *** heyongli has joined #openstack-infra | 22:24 | |
| *** felipemonteiro_ has quit IRC | 22:25 | |
| *** lifeless_ has joined #openstack-infra | 22:32 | |
| *** lifeless has quit IRC | 22:32 | |
| *** heyongli has quit IRC | 22:34 | |
| *** heyongli has joined #openstack-infra | 22:35 | |
| *** rcernin has joined #openstack-infra | 22:36 | |
| *** hongbin has quit IRC | 22:40 | |
| *** heyongli has quit IRC | 22:45 | |
| *** heyongli has joined #openstack-infra | 22:45 | |
| *** yamamoto has joined #openstack-infra | 22:45 | |
| *** eernst has quit IRC | 22:47 | |
| *** yamamoto has quit IRC | 22:49 | |
| *** boris_42_ has joined #openstack-infra | 22:54 | |
| *** heyongli has quit IRC | 22:55 | |
| *** ekhugen has quit IRC | 22:55 | |
| *** heyongli has joined #openstack-infra | 22:55 | |
| clarkb | over 95% of dockerhub requests cached in inap at this point | 22:56 |
| *** tpsilva has quit IRC | 22:57 | |
| *** ekhugen has joined #openstack-infra | 22:59 | |
| *** dklyle has quit IRC | 23:01 | |
| *** lifeless_ has quit IRC | 23:05 | |
| *** heyongli has quit IRC | 23:05 | |
| *** heyongli has joined #openstack-infra | 23:05 | |
| *** lifeless has joined #openstack-infra | 23:06 | |
| *** r-daneel has quit IRC | 23:07 | |
| *** hamzy_ has joined #openstack-infra | 23:12 | |
| *** hamzy has quit IRC | 23:13 | |
| *** heyongli has quit IRC | 23:15 | |
| *** heyongli has joined #openstack-infra | 23:16 | |
| *** dave-mcc_ has quit IRC | 23:16 | |
| *** tosky has quit IRC | 23:20 | |
| clarkb | upstream kata test fails in crio tests, so ya I think the zuul stuff is in pretty good shape if my logging updates all work | 23:21 |
| clarkb | maybe that will be enough to show others and start iterating on | 23:21 |
| fungi | #status log openstackid.org has been removed from the emergency disable list now that https://review.openstack.org/576248 has merged, and after confirming with smarcet that he will keep an eye on it | 23:23 |
| openstackstatus | fungi: finished logging | 23:23 |
| *** owalsh_ has joined #openstack-infra | 23:24 | |
| clarkb | fungi: is puppet running properly after ansible fix for openstack invetory? | 23:25 |
| *** heyongli has quit IRC | 23:26 | |
| *** heyongli has joined #openstack-infra | 23:26 | |
| clarkb | 23:19UTC was last thing logged which implies it is I think | 23:26 |
| Guest95387 | 😃 | 23:26 |
| fungi | clarkb: yeah, seems to be | 23:26 |
| mnaser | clarkb: I can try to iterate on things a little bit? | 23:26 |
| fungi | though i haven't retested the new and improved launch-node.py yet | 23:27 |
| fungi | saving that for tomorrow now | 23:27 |
| mnaser | clarkb: could we add the other repos too? I think Kata runtime consistently passes Xenial | 23:27 |
| clarkb | mnaser: sure, I'm currently just waiting on my most recent change's logging updates to come through which should give us kubelet logs which I hope helps debug k8s test | 23:27 |
| mnaser | Oh cool, did the issue about | 23:27 |
| clarkb | mnaser: that is up to kata, we were asked to do proxy which is why I am working on that repo | 23:27 |
| *** akhilaki has quit IRC | 23:27 | |
| mnaser | Complaining about your commit get fixed? | 23:27 |
| *** owalsh has quit IRC | 23:27 | |
| clarkb | yup | 23:28 |
| clarkb | resulted in crio failing http://jenkins.katacontainers.io/job/kata-containers-proxy-ubuntu-16-04-PR/13/console | 23:28 |
| mnaser | oh cool so same failure at least | 23:28 |
| clarkb | no zuul passes crio aiui, zuul fails k8s | 23:29 |
| clarkb | possible these failure are not deterministic or consistent and we are just seeing different ones pop up | 23:29 |
| clarkb | but they both don't work so in that way they are the same | 23:29 |
| mnaser | config-core: https://review.openstack.org/#/c/576274/ | 23:31 |
| mnaser | this zuul-jobs repo | 23:31 |
| mnaser | and if possible https://review.openstack.org/#/c/576156/ above it so it lands automatically after : | 23:32 |
| mnaser | :) | 23:32 |
| *** heyongli has quit IRC | 23:36 | |
| *** heyongli has joined #openstack-infra | 23:36 | |
| mnaser | clarkb: if i can grab your +W again on https://review.openstack.org/#/c/574745 and the parent change | 23:41 |
| *** lifeless has quit IRC | 23:43 | |
| clarkb | done I reviewed the parent change | 23:44 |
| mnaser | yay, thanks | 23:44 |
| * mnaser is pretty excited about the percona cache | 23:44 | |
| mnaser | it's been hell | 23:44 |
| *** jesslampe has quit IRC | 23:46 | |
| *** heyongli has quit IRC | 23:46 | |
| *** lifeless has joined #openstack-infra | 23:46 | |
| *** heyongli has joined #openstack-infra | 23:46 | |
| mnaser | hm | 23:47 |
| mnaser | i can't do a depends-on openstack-zuul-jobs change that affects a role in pre.yaml or something like that? | 23:47 |
| mnaser | https://review.openstack.org/#/c/576241/ depends on https://review.openstack.org/#/c/576156/ and NODEPOOL_PERCONA_PROXY seems to be empty in the logs | 23:47 |
| mnaser | http://logs.openstack.org/41/576241/2/check/openstack-ansible-functional-centos-7/23610c6/job-output.txt.gz#_2018-06-18_23_30_48_781178 | 23:48 |
| mnaser | oh zuul-jobs is a trusted repo? | 23:48 |
| *** felipemonteiro has joined #openstack-infra | 23:48 | |
| clarkb | zuul-jobs and ozj are not trusted repos but if their roles are used from a truste djob you get this behavior | 23:49 |
| clarkb | I think that must be what is happening here | 23:49 |
| mnaser | http://logs.openstack.org/41/576241/2/check/openstack-ansible-functional-centos-7/23610c6/job-output.txt.gz#_2018-06-18_23_24_09_330604 | 23:49 |
| mnaser | "included: /var/lib/zuul/builds/23610c6fb54c4a9abe2fd4ddd0829c76/trusted/project_1/git.openstack.org/openstack-infra/zuul-jobs/roles/configure-mirrors/tasks/mirror.yaml" | 23:49 |
| mnaser | so i figured yeah they were trusted | 23:49 |
| mnaser | ok cool, TIL | 23:49 |
| *** jesslampe has joined #openstack-infra | 23:50 | |
| clarkb | I'm trying to think if there is a reason to not put those roles in project-config if they are used from a trusted context anyways. May make it easier to undertsand | 23:56 |
| *** heyongli has quit IRC | 23:56 | |
| mnaser | configure-mirrors seems very openstack-zuul-jobs-ish too, not zuul-jobs-ish | 23:56 |
| *** heyongli has joined #openstack-infra | 23:57 | |
| ianw | i think they were just some of the first jobs done, so it was a bit seat of pants | 23:57 |
| mnaser | https://review.openstack.org/#/c/576156/ uh | 23:57 |
| mnaser | is the mirror duplicated? | 23:57 |
| clarkb | mnaser: not sure what you mean by duplicated | 23:57 |
| openstackgerrit | Merged openstack-infra/system-config master: Finish switching to using new ansible inventory https://review.openstack.org/576275 | 23:58 |
| mnaser | clarkb: ah sorry i misread, that was configure-mirrors in zuul-jobs and i changed mirror-info in ozj | 23:58 |
| clarkb | mnaser: I believe zuul-jobs role is what writes out the /etc config for things like apt or yum | 23:58 |
| clarkb | then ozj role writes out openstack specific "hey if you want something more you can find it here" config | 23:58 |
| clarkb | I think what you've done is correct because we don't want the percona stuff in every job, instead its an opt in you can consume and that is where we tell you where to find it | 23:59 |
| mnaser | yup | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!