Thursday, 2019-10-03

*** gyee has quit IRC00:06
*** jamesmcarthur has joined #openstack-infra00:09
*** michael-beaver has quit IRC00:10
*** ianychoi has quit IRC00:12
*** rlandy has quit IRC00:13
*** ianychoi has joined #openstack-infra00:15
*** goldyfruit_ has joined #openstack-infra00:23
*** jamesmcarthur has quit IRC00:24
*** ianychoi has quit IRC00:32
*** jamesmcarthur has joined #openstack-infra00:35
*** armax has quit IRC00:46
*** ianychoi has joined #openstack-infra00:48
*** jamesmcarthur has quit IRC00:49
*** rosmaita has left #openstack-infra00:49
*** jamesmcarthur has joined #openstack-infra00:51
*** jamesmcarthur has quit IRC00:54
*** jamesmcarthur has joined #openstack-infra00:55
*** hongbin has joined #openstack-infra01:01
*** jamesmcarthur has quit IRC01:07
openstackgerritIan Wienand proposed openstack/diskimage-builder master: yum-minimal : update mirrors for Centos 8  https://review.opendev.org/68443901:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: dib-python : handle centos 8  https://review.opendev.org/68444101:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Update locales for Centos 8  https://review.opendev.org/68444201:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Update redhat-common pkg-map for centos 8  https://review.opendev.org/68444901:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Add environment switch for centos8 to use dnf  https://review.opendev.org/68445901:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Use $YUM instead of direct calls in more places  https://review.opendev.org/68446001:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: yum-minimal: Don't install yum, install libcurl  https://review.opendev.org/68446101:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv : deprecate source for CentOS 8, new variables  https://review.opendev.org/68446201:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: Use wrappers to call pip for glean install  https://review.opendev.org/68564301:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Fix networking for CentOS 8  https://review.opendev.org/68451301:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Add NetworkManager and dhcp-client for CentOS 8  https://review.opendev.org/68609001:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: CentOS 8 minimal testing and support  https://review.opendev.org/68486001:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] use mirrors for functional test builds  https://review.opendev.org/68588501:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: default to NetworkManager for CentOS and Fedora  https://review.opendev.org/68631901:08
*** whoami-rajat has joined #openstack-infra01:28
*** hwoarang has quit IRC01:42
*** hwoarang has joined #openstack-infra01:50
*** ykarel|away has joined #openstack-infra01:58
*** anteaya has quit IRC01:59
*** igordc has quit IRC02:01
*** apetrich has quit IRC02:10
*** jamesmcarthur has joined #openstack-infra02:18
*** yamamoto has quit IRC02:22
*** yamamoto has joined #openstack-infra02:29
*** roman_g has quit IRC02:34
*** rh-jelabarre has quit IRC02:36
*** factor has joined #openstack-infra02:47
*** tinwood has quit IRC02:50
*** hongbin has quit IRC02:51
*** tinwood has joined #openstack-infra02:52
*** jamesmcarthur has quit IRC02:56
*** jamesmcarthur has joined #openstack-infra02:58
*** markvoelker has joined #openstack-infra03:07
*** markvoelker has quit IRC03:11
*** eernst has joined #openstack-infra03:20
*** markvoelker has joined #openstack-infra03:21
*** markvoelker has quit IRC03:24
*** eernst has quit IRC03:26
*** eernst has joined #openstack-infra03:27
*** markvoelker has joined #openstack-infra03:30
*** markvoelker has quit IRC03:32
*** markvoelker has joined #openstack-infra03:32
*** eernst has quit IRC03:33
*** eernst has joined #openstack-infra03:35
*** markvoelker has quit IRC03:37
*** dave-mccowan has quit IRC03:37
*** eernst has quit IRC03:38
*** exsdev has quit IRC03:44
*** jamesmcarthur has quit IRC03:45
*** jamesmcarthur has joined #openstack-infra03:47
*** exsdev has joined #openstack-infra03:48
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Add security mirror override for debian-minimal  https://review.opendev.org/68632803:50
*** kjackal has joined #openstack-infra03:52
*** jamesmcarthur has quit IRC03:57
*** jamesmcarthur has joined #openstack-infra03:57
*** jamesmcarthur has quit IRC04:02
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: default to NetworkManager for CentOS and Fedora  https://review.opendev.org/68631904:06
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: Use wrappers to call pip for glean install  https://review.opendev.org/68564304:06
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Fix networking for CentOS 8  https://review.opendev.org/68451304:06
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Add NetworkManager and dhcp-client for CentOS 8  https://review.opendev.org/68609004:06
openstackgerritIan Wienand proposed openstack/diskimage-builder master: CentOS 8 minimal testing and support  https://review.opendev.org/68486004:06
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] use mirrors for functional test builds  https://review.opendev.org/68588504:06
*** markvoelker has joined #openstack-infra04:07
*** udesale has joined #openstack-infra04:13
*** ociuhandu has joined #openstack-infra04:14
*** ociuhandu has quit IRC04:18
*** ykarel|away has quit IRC04:22
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] use mirrors for functional test builds  https://review.opendev.org/68588504:23
*** odicha has joined #openstack-infra04:25
*** odicha has quit IRC04:28
*** rcernin has quit IRC04:29
*** surpatil has joined #openstack-infra04:29
*** odicha has joined #openstack-infra04:32
*** rcernin has joined #openstack-infra04:32
*** jamesmcarthur has joined #openstack-infra04:33
*** markvoelker has quit IRC04:37
*** soniya29 has joined #openstack-infra04:37
*** jamesmcarthur has quit IRC04:40
*** ykarel|away has joined #openstack-infra04:41
*** ykarel|away is now known as ykarel04:56
*** markvoelker has joined #openstack-infra05:01
*** spsurya has joined #openstack-infra05:05
*** markvoelker has quit IRC05:06
*** markvoelker has joined #openstack-infra05:11
*** markvoelker has quit IRC05:15
*** dpawlik has joined #openstack-infra05:17
*** jamesmcarthur has joined #openstack-infra05:17
openstackgerritMerged opendev/base-jobs master: Add docs templates  https://review.opendev.org/68622805:19
*** markvoelker has joined #openstack-infra05:20
*** jamesmcarthur has quit IRC05:23
*** markvoelker has quit IRC05:24
openstackgerritMerged openstack/openstack-zuul-jobs master: Bye, bye readthedocs-failing  https://review.opendev.org/68561805:26
*** markvoelker has joined #openstack-infra05:29
*** markvoelker has quit IRC05:34
*** markvoelker has joined #openstack-infra05:38
*** markvoelker has quit IRC05:43
*** ykarel is now known as ykarel|meeting05:44
*** markvoelker has joined #openstack-infra05:47
*** ykarel_ has joined #openstack-infra05:48
*** ykarel|meeting has quit IRC05:50
openstackgerritAndreas Jaeger proposed openstack/project-config master: Add openstacksdk to opendev tenant  https://review.opendev.org/68633405:51
*** markvoelker has quit IRC05:52
AJaegerconfig-core, see http://zuul.opendev.org/t/opendev/config-errors ^05:52
openstackgerritAndreas Jaeger proposed openstack/project-config master: Add openstacksdk /ara to opendev tenant  https://review.opendev.org/68633405:53
AJaegermissed one ...05:54
*** markvoelker has joined #openstack-infra05:57
*** markvoelker has quit IRC06:02
*** lpetrut has joined #openstack-infra06:02
*** lpetrut has quit IRC06:03
*** lpetrut has joined #openstack-infra06:03
*** markvoelker has joined #openstack-infra06:06
*** surpatil has quit IRC06:10
*** markvoelker has quit IRC06:10
*** surpatil has joined #openstack-infra06:10
*** surpatil has quit IRC06:12
*** surpatil has joined #openstack-infra06:13
*** markvoelker has joined #openstack-infra06:15
*** jamesmcarthur has joined #openstack-infra06:19
*** markvoelker has quit IRC06:20
*** ianychoi has quit IRC06:20
*** jaosorior has joined #openstack-infra06:21
*** ianychoi has joined #openstack-infra06:23
*** markvoelker has joined #openstack-infra06:24
*** markvoelker has quit IRC06:29
*** ianychoi has quit IRC06:29
*** jamesmcarthur has quit IRC06:30
*** pgaxatte has joined #openstack-infra06:30
*** surpatil has quit IRC06:30
*** ianychoi has joined #openstack-infra06:31
*** xek has joined #openstack-infra06:32
*** markvoelker has joined #openstack-infra06:33
*** surpatil has joined #openstack-infra06:34
*** markvoelker has quit IRC06:38
*** markvoelker has joined #openstack-infra06:43
*** markvoelker has quit IRC06:47
*** yamamoto has quit IRC06:47
*** jamesmcarthur has joined #openstack-infra06:49
*** yamamoto has joined #openstack-infra06:51
*** markvoelker has joined #openstack-infra06:52
*** markvoelker has quit IRC06:57
*** jamesmcarthur has quit IRC06:58
*** udesale has quit IRC06:58
*** udesale has joined #openstack-infra06:59
*** lpetrut has quit IRC06:59
*** lpetrut has joined #openstack-infra07:00
*** jamesmcarthur has joined #openstack-infra07:01
*** markvoelker has joined #openstack-infra07:01
*** udesale has quit IRC07:02
*** udesale has joined #openstack-infra07:02
*** markvoelker has quit IRC07:05
*** rcernin has quit IRC07:06
*** markvoelker has joined #openstack-infra07:10
*** tesseract has joined #openstack-infra07:12
*** markvoelker has quit IRC07:15
*** tosky has joined #openstack-infra07:16
*** hjensas has joined #openstack-infra07:17
*** markvoelker has joined #openstack-infra07:19
*** ociuhandu has joined #openstack-infra07:22
*** ociuhandu has quit IRC07:23
*** markvoelker has quit IRC07:24
*** ykarel_ is now known as ykarel|meeting07:24
*** pcaruana has joined #openstack-infra07:28
*** piotrowskim has joined #openstack-infra07:29
*** markvoelker has joined #openstack-infra07:29
*** markvoelker has quit IRC07:34
*** ociuhandu has joined #openstack-infra07:36
*** ykarel_ has joined #openstack-infra07:36
*** ykarel|meeting has quit IRC07:36
*** ykarel_ is now known as ykarel07:37
*** markvoelker has joined #openstack-infra07:38
*** iurygregory has quit IRC07:39
*** surpatil is now known as surpatil|lunch07:41
*** markvoelker has quit IRC07:43
*** ianychoi has quit IRC07:45
*** markvoelker has joined #openstack-infra07:47
*** udesale has quit IRC07:48
*** udesale has joined #openstack-infra07:48
*** lpetrut has quit IRC07:48
*** lpetrut has joined #openstack-infra07:49
*** jpena|off is now known as jpena07:49
*** pkopec has joined #openstack-infra07:49
*** ykarel is now known as ykarel|lunch07:49
*** markvoelker has quit IRC07:52
*** e0ne has joined #openstack-infra07:52
*** e0ne has quit IRC07:53
*** kopecmartin|off is now known as kopecmartin07:53
*** yamamoto has quit IRC07:54
*** rpittau|afk is now known as rpittau07:59
*** tkajinam has quit IRC08:00
*** ociuhandu has quit IRC08:04
*** ociuhandu has joined #openstack-infra08:04
*** ociuhandu has quit IRC08:05
*** ociuhandu has joined #openstack-infra08:05
*** hjensas has quit IRC08:05
*** xenos76 has joined #openstack-infra08:05
*** markvoelker has joined #openstack-infra08:06
*** slaweq has joined #openstack-infra08:06
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] use mirrors for functional test builds  https://review.opendev.org/68588508:09
*** markvoelker has quit IRC08:11
*** yamamoto has joined #openstack-infra08:13
*** shardy has joined #openstack-infra08:14
*** shardy has quit IRC08:15
*** markvoelker has joined #openstack-infra08:15
*** jamesmcarthur has quit IRC08:17
*** yamamoto has quit IRC08:17
*** shardy has joined #openstack-infra08:19
*** jaosorior has quit IRC08:19
*** yamamoto has joined #openstack-infra08:19
*** shardy has quit IRC08:19
*** markvoelker has quit IRC08:20
*** panda|off is now known as panda08:20
*** iurygregory has joined #openstack-infra08:21
*** markvoelker has joined #openstack-infra08:24
*** dchen has quit IRC08:26
*** markvoelker has quit IRC08:29
*** ccamacho has joined #openstack-infra08:31
*** jbadiapa has joined #openstack-infra08:34
*** markvoelker has joined #openstack-infra08:34
openstackgerritGabor Lekeny proposed zuul/zuul master: Fix github webhook authentication  https://review.opendev.org/68635508:34
*** hjensas has joined #openstack-infra08:36
openstackgerritGabor Lekeny proposed zuul/zuul master: Fix github webhook authentication  https://review.opendev.org/68635508:38
*** markvoelker has quit IRC08:39
*** ykarel|lunch is now known as ykarel08:40
*** slaweq has quit IRC08:41
*** derekh has joined #openstack-infra08:42
*** yamamoto has quit IRC08:43
*** roman_g has joined #openstack-infra08:44
*** e0ne has joined #openstack-infra08:49
*** markvoelker has joined #openstack-infra08:52
*** slaweq has joined #openstack-infra08:52
*** markvoelker has quit IRC08:57
*** jbadiapa has quit IRC08:59
*** markvoelker has joined #openstack-infra09:01
openstackgerritMerged openstack/project-config master: Add openstacksdk /ara to opendev tenant  https://review.opendev.org/68633409:05
*** markvoelker has quit IRC09:06
openstackgerritMerged zuul/zuul-registry master: Add tox configuration and fixe flake8 errors  https://review.opendev.org/68623009:08
*** yamamoto has joined #openstack-infra09:08
*** markvoelker has joined #openstack-infra09:10
*** gfidente has joined #openstack-infra09:13
*** jamesmcarthur has joined #openstack-infra09:14
*** markvoelker has quit IRC09:15
*** ricolin has quit IRC09:15
*** diablo_rojo has joined #openstack-infra09:17
*** jamesmcarthur has quit IRC09:18
*** markvoelker has joined #openstack-infra09:20
mordredAJaeger: now that train is out - do you know what the story is with python2/python3? I think I remember that U is where python2 support goes away, yes?09:23
*** diablo_rojo has quit IRC09:24
*** markvoelker has quit IRC09:25
*** hjensas has quit IRC09:25
*** markvoelker has joined #openstack-infra09:29
*** markvoelker has quit IRC09:34
*** gfidente has quit IRC09:38
*** markvoelker has joined #openstack-infra09:38
mordredAJaeger: nm - I have learned the infos09:40
*** markvoelker has quit IRC09:43
*** xenos76 has quit IRC09:44
*** xenos76 has joined #openstack-infra09:45
openstackgerritStephen Finucane proposed openstack/openstack-zuul-jobs master: Create openstack-tox-mypy job  https://review.opendev.org/68636609:46
*** markvoelker has joined #openstack-infra09:48
*** ociuhandu has quit IRC09:50
*** ociuhandu has joined #openstack-infra09:51
*** markvoelker has quit IRC09:52
*** udesale has quit IRC09:53
*** udesale has joined #openstack-infra09:54
*** pkopec has quit IRC09:55
*** surpatil has joined #openstack-infra09:56
*** surpatil|lunch has quit IRC09:56
*** markvoelker has joined #openstack-infra09:57
*** jaosorior has joined #openstack-infra09:59
*** markvoelker has quit IRC10:01
*** yamamoto has quit IRC10:02
*** yamamoto has joined #openstack-infra10:05
*** markvoelker has joined #openstack-infra10:06
*** jaosorior has quit IRC10:06
*** markvoelker has quit IRC10:11
*** yamamoto has quit IRC10:13
openstackgerritMerged zuul/nodepool master: Add statsd_exporter mapping  https://review.opendev.org/66047310:14
*** jamesmcarthur has joined #openstack-infra10:15
*** markvoelker has joined #openstack-infra10:15
*** diablo_rojo has joined #openstack-infra10:16
*** jamesmcarthur has quit IRC10:19
*** VedaAnnayappa has joined #openstack-infra10:20
*** markvoelker has quit IRC10:20
*** mgoddard has quit IRC10:23
*** jaosorior has joined #openstack-infra10:24
*** pkopec has joined #openstack-infra10:24
* pkopec back10:24
*** markvoelker has joined #openstack-infra10:24
*** mgoddard has joined #openstack-infra10:26
*** markvoelker has quit IRC10:29
*** yamamoto has joined #openstack-infra10:31
*** markvoelker has joined #openstack-infra10:34
*** pkopec has quit IRC10:35
*** pkopec has joined #openstack-infra10:36
*** markvoelker has quit IRC10:38
*** diablo_rojo has quit IRC10:40
*** markvoelker has joined #openstack-infra10:43
*** rcernin has joined #openstack-infra10:43
*** markvoelker has quit IRC10:47
*** jaosorior has quit IRC10:51
*** markvoelker has joined #openstack-infra10:52
*** pgaxatte has quit IRC10:55
*** markvoelker has quit IRC10:57
*** jpena is now known as jpena|lunch11:00
*** markvoelker has joined #openstack-infra11:01
*** ociuhandu has quit IRC11:02
*** ociuhandu has joined #openstack-infra11:02
*** markvoelker has quit IRC11:06
*** ociuhandu has quit IRC11:07
openstackgerritGabor Lekeny proposed zuul/zuul master: Fix github webhook authentication  https://review.opendev.org/68635511:10
*** markvoelker has joined #openstack-infra11:10
*** markvoelker has quit IRC11:15
*** jamesmcarthur has joined #openstack-infra11:16
*** udesale has quit IRC11:17
*** lucasagomes has joined #openstack-infra11:17
*** VedaAnnayappa has quit IRC11:18
*** markvoelker has joined #openstack-infra11:20
*** jamesmcarthur has quit IRC11:20
*** pkopec has quit IRC11:22
*** ociuhandu has joined #openstack-infra11:23
*** markvoelker has quit IRC11:24
*** ociuhandu has quit IRC11:25
*** jamesmcarthur has joined #openstack-infra11:26
*** ociuhandu has joined #openstack-infra11:26
openstackgerritJeremy Stanley proposed openstack/project-config master: Override security mirror for Debian image builds  https://review.opendev.org/68620411:28
*** markvoelker has joined #openstack-infra11:29
*** pkopec has joined #openstack-infra11:31
*** markvoelker has quit IRC11:34
*** rcernin has quit IRC11:34
jrosserfungi: i made a hacky patch which works around that security mirror issue for now https://review.opendev.org/#/c/686348/311:35
*** jamesmcarthur has quit IRC11:35
*** jamesmcarthur has joined #openstack-infra11:37
*** markvoelker has joined #openstack-infra11:38
fungijrosser: that works11:41
jrosserwe just need to fix a couple more OSA bugs and we can get stuff merging again, then once the dib bits and images are better i'll revert that11:42
*** markvoelker has quit IRC11:42
*** tosky_ has joined #openstack-infra11:44
*** dosaboy has quit IRC11:45
*** tosky has quit IRC11:46
*** soniya29 has quit IRC11:47
*** ociuhandu has quit IRC11:47
*** ociuhandu has joined #openstack-infra11:47
*** markvoelker has joined #openstack-infra11:47
*** ociuhandu has quit IRC11:48
*** ociuhandu has joined #openstack-infra11:49
*** jamesmcarthur has quit IRC11:51
*** markvoelker has quit IRC11:52
*** pgaxatte has joined #openstack-infra11:52
*** jamesmcarthur has joined #openstack-infra11:52
*** prometheanfire has quit IRC11:54
*** dosaboy has joined #openstack-infra11:54
*** ociuhandu has quit IRC11:54
*** prometheanfire has joined #openstack-infra11:55
*** markvoelker has joined #openstack-infra11:57
*** rh-jelabarre has joined #openstack-infra11:57
*** jamesmcarthur has quit IRC11:59
*** rh-jelabarre has quit IRC11:59
*** rh-jelabarre has joined #openstack-infra11:59
*** markvoelker has quit IRC12:01
*** tosky_ is now known as tosky12:01
*** gfidente has joined #openstack-infra12:02
*** dosaboy has quit IRC12:03
*** dosaboy has joined #openstack-infra12:03
*** ociuhandu has joined #openstack-infra12:03
*** xenos76 has quit IRC12:05
*** yamamoto has quit IRC12:05
*** markvoelker has joined #openstack-infra12:05
*** jamesmcarthur has joined #openstack-infra12:07
*** ociuhandu has quit IRC12:07
*** jpena|lunch is now known as jpena12:12
*** soniya29 has joined #openstack-infra12:15
*** markvoelker has quit IRC12:17
*** markvoelker has joined #openstack-infra12:17
*** markvoelker has quit IRC12:19
*** goldyfruit_ has quit IRC12:19
*** jamesmcarthur has quit IRC12:19
*** jamesmcarthur has joined #openstack-infra12:20
*** nhicher has quit IRC12:20
*** yamamoto has joined #openstack-infra12:20
*** pkopec has quit IRC12:21
openstackgerritJeremy Stanley proposed openstack/project-config master: Override security mirror for Debian image builds  https://review.opendev.org/68620412:23
*** pkopec has joined #openstack-infra12:27
*** nhicher has joined #openstack-infra12:27
*** ociuhandu has joined #openstack-infra12:29
*** jamesmcarthur has quit IRC12:33
*** rlandy has joined #openstack-infra12:34
*** ociuhandu has quit IRC12:36
*** ociuhandu has joined #openstack-infra12:38
openstackgerritMonty Taylor proposed zuul/zuul master: Authorization rules: support YAML nested dictionaries  https://review.opendev.org/68479012:38
*** sthussey has joined #openstack-infra12:43
*** ociuhandu has quit IRC12:44
openstackgerritMatthieu Huin proposed openstack/project-config master: Add sphinx-contrib/openapi to untrusted github projects  https://review.opendev.org/68639212:45
*** jamesmcarthur has joined #openstack-infra12:52
*** pkopec has quit IRC12:53
*** pkopec has joined #openstack-infra12:55
*** markvoelker has joined #openstack-infra13:01
*** ociuhandu has joined #openstack-infra13:01
*** kjackal_v2 has joined #openstack-infra13:06
*** pkopec has quit IRC13:09
*** kjackal has quit IRC13:09
*** pkopec has joined #openstack-infra13:17
*** xenos76 has joined #openstack-infra13:18
*** rosmaita has joined #openstack-infra13:19
*** yamamoto has quit IRC13:20
*** mriedem has joined #openstack-infra13:22
*** pkopec has quit IRC13:22
*** pkopec has joined #openstack-infra13:23
*** goldyfruit_ has joined #openstack-infra13:23
*** ociuhandu has quit IRC13:29
*** ociuhandu has joined #openstack-infra13:30
*** pkopec has quit IRC13:31
*** udesale has joined #openstack-infra13:33
*** pkopec has joined #openstack-infra13:34
openstackgerritMonty Taylor proposed zuul/zuul-registry master: WIP Consume typing from openstacksdk and keystoneauth  https://review.opendev.org/68640313:35
*** jamesmcarthur has quit IRC13:40
*** ociuhandu has quit IRC13:47
*** ociuhandu has joined #openstack-infra13:47
*** ociuhandu has quit IRC13:48
*** ociuhandu has joined #openstack-infra13:48
*** yamamoto has joined #openstack-infra13:51
*** surpatil has quit IRC13:54
*** soniya29 has quit IRC13:54
*** david-lyle has quit IRC13:55
*** david-lyle has joined #openstack-infra13:55
*** ociuhandu has quit IRC13:57
*** ociuhandu has joined #openstack-infra13:58
*** ociuhandu has quit IRC14:03
openstackgerritChristopher Dearborn proposed openstack/project-config master: Update python-dracclient gate python3 support  https://review.opendev.org/68640914:04
*** yamamoto has quit IRC14:04
*** odicha has quit IRC14:05
*** ociuhandu has joined #openstack-infra14:06
*** otherwiseguy_ has joined #openstack-infra14:06
*** haleyb has quit IRC14:09
*** otherwiseguy has quit IRC14:09
*** ociuhandu has quit IRC14:10
*** jamesmcarthur has joined #openstack-infra14:14
*** lpetrut has quit IRC14:16
*** Goneri has joined #openstack-infra14:17
*** ricolin has joined #openstack-infra14:17
fungiif any system-config reviewers have a spare moment to review 682417 and 682425 i'll take care of retiring those mailing lists14:28
fungiboth already have a +214:29
*** pgaxatte has quit IRC14:30
*** ociuhandu has joined #openstack-infra14:31
*** ykarel is now known as ykarel|afk14:33
*** ociuhandu has quit IRC14:33
*** rfolco is now known as rfolco|bbl14:35
*** jamesmcarthur has quit IRC14:35
*** jamesmcarthur has joined #openstack-infra14:40
*** jamesmcarthur has quit IRC14:41
corvusinfra-root: i think zuul-registry is ready to go into production as the intermediate registry whenever we are ready.  if we don't mind a small risk of failing image build jobs if something goes wrong, we can merge https://review.opendev.org/686298  which will just make the switch -- we can (manually) revert that if there's an error.  alternatively we could spin up a replacement host, but that's14:42
corvusonly going to let us verify the deployment before we start using it -- operation under load is still going to be something that either works or doesn't.14:42
*** jamesmcarthur has joined #openstack-infra14:44
fungiwe're already failing image build jobs intermittently anyway, which this is targeted at alleviating, so seems reasonable to press forward14:47
*** eernst has joined #openstack-infra14:48
*** goldyfruit_ has quit IRC14:49
fungicorvus: registry_swift_secretkey isn't used any longer? what was that for previously?14:50
fungii'm assuming use of openstacksdk in the new registry is what has made registry_swift_authurl no longer necessary14:52
*** iurygregory_ has joined #openstack-infra14:52
*** ociuhandu has joined #openstack-infra14:53
*** iurygregory has quit IRC14:54
corvusfungi: yeah, the secret key was an implementation-specific part of the docker-registry server.  and yes, we outsource that to sdk.14:56
corvusthat=authurl14:56
*** jamesmcarthur has quit IRC14:57
*** goldyfruit_ has joined #openstack-infra14:57
*** ociuhandu has quit IRC14:57
fungiawesome, lgtm!14:58
AJaegeris using the zuul tenant for this third-party testing correct? https://review.opendev.org/#/c/686392/14:59
*** jpena is now known as jpena|off14:59
*** jamesmcarthur has joined #openstack-infra15:00
corvusAJaeger: i think so -- zuul does have openapi docs rendering, so i imagine mhu wants to test enhancements to that15:00
AJaegerAh, ok15:01
corvushttp://zuul.opendev.org/openapi15:01
corvusand https://zuul-ci.org/docs/zuul/user/web.html15:01
AJaegerthanks, corvus15:03
AJaegerconfig-core, a few open reviews are : https://review.opendev.org/683028 https://review.opendev.org/685791 https://review.opendev.org/683994 , please15:09
*** ykarel|afk is now known as ykarel|away15:13
openstackgerritMerged openstack/project-config master: Add sphinx-contrib/openapi to untrusted github projects  https://review.opendev.org/68639215:18
*** xek has quit IRC15:18
*** jbadiapa has joined #openstack-infra15:19
*** jamesmcarthur has quit IRC15:21
*** eernst has quit IRC15:23
*** gyee has joined #openstack-infra15:24
*** goldyfruit_ has quit IRC15:25
*** ykarel|away has quit IRC15:26
*** goldyfruit_ has joined #openstack-infra15:28
openstackgerritNicolas Hicher proposed zuul/zuul-jobs master: DNM: debug console issue when iptables-save is called  https://review.opendev.org/68642415:29
*** jamesmcarthur has joined #openstack-infra15:30
*** mattw4 has joined #openstack-infra15:31
*** michael-beaver has joined #openstack-infra15:33
*** udesale has quit IRC15:35
*** udesale has joined #openstack-infra15:35
*** tesseract has quit IRC15:36
*** lucasagomes has quit IRC15:40
*** ykarel|away has joined #openstack-infra15:49
*** jbadiapa has quit IRC15:50
openstackgerritTristan Cacqueray proposed zuul/zuul-registry master: Add type annotations  https://review.opendev.org/68624915:59
openstackgerritTristan Cacqueray proposed zuul/zuul-registry master: WIP: use openstacksdk wip annotation  https://review.opendev.org/68643415:59
*** rfolco|bbl is now known as rfolco16:00
*** iurygregory_ has quit IRC16:07
openstackgerritFabien Boucher proposed zuul/zuul master: WIP - Gitlab - Basic handling of merge_requests event  https://review.opendev.org/68599016:07
*** jbadiapa has joined #openstack-infra16:10
*** eernst has joined #openstack-infra16:16
*** eernst has quit IRC16:20
*** ociuhandu has joined #openstack-infra16:24
*** goldyfruit___ has joined #openstack-infra16:24
*** jbadiapa has quit IRC16:27
*** goldyfruit_ has quit IRC16:27
*** rpittau is now known as rpittau|afk16:27
*** udesale has quit IRC16:28
*** ociuhandu has quit IRC16:30
*** ccamacho has quit IRC16:31
*** chandankumar is now known as raukadah16:34
*** e0ne has quit IRC16:36
*** lucasagomes has joined #openstack-infra16:37
*** eernst has joined #openstack-infra16:39
*** jpena|off is now known as jpena16:41
*** eernst has quit IRC16:44
*** david-lyle has quit IRC16:44
*** david-lyle has joined #openstack-infra16:45
*** david-lyle has quit IRC16:45
*** igordc has joined #openstack-infra16:46
*** dklyle has joined #openstack-infra16:46
*** gfidente is now known as gfidente|afk16:52
*** derekh has quit IRC16:55
*** goldyfruit___ has quit IRC17:08
openstackgerritMerged openstack/project-config master: Add ansible-role-zuul-registry role for windmill  https://review.opendev.org/68579117:11
openstackgerritMerged openstack/project-config master: Set Review-Priority label sticky  https://review.opendev.org/68399417:11
*** tosky has quit IRC17:14
*** jamesmcarthur has quit IRC17:18
*** jamesmcarthur has joined #openstack-infra17:19
*** openstackgerrit has quit IRC17:21
*** Garyx has quit IRC17:31
*** openstackgerrit has joined #openstack-infra17:34
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Add streaming download support  https://review.opendev.org/68644917:34
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Update test script to match playbook  https://review.opendev.org/68645017:34
*** ricolin has quit IRC17:35
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Add streaming download support  https://review.opendev.org/68644917:38
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Update test script to match playbook  https://review.opendev.org/68645017:38
*** Garyx has joined #openstack-infra17:39
*** ykarel|away has quit IRC17:43
*** jamesmcarthur has quit IRC17:51
*** jamesmcarthur has joined #openstack-infra17:51
*** rfolco has quit IRC17:52
*** jamesmcarthur has quit IRC17:52
AJaegerpabelanger: want me to update the windmill changes to use the opendev build-docs jobs?17:53
*** e0ne has joined #openstack-infra17:59
*** jpena is now known as jpena|off18:05
*** armax has joined #openstack-infra18:06
*** e0ne has quit IRC18:06
*** spsurya has quit IRC18:07
openstackgerritTristan Cacqueray proposed zuul/nodepool master: wip: add a standalone launcher object  https://review.opendev.org/63963218:10
*** ralonsoh has quit IRC18:10
*** ykarel|away has joined #openstack-infra18:11
*** goldyfruit___ has joined #openstack-infra18:16
*** hjensas has joined #openstack-infra18:20
openstackgerritMatthieu Huin proposed zuul/zuul master: Add OpenAPI description for enqueue, dequeue, autohold  https://review.opendev.org/67425718:39
*** jamesmcarthur has joined #openstack-infra18:47
*** rfolco has joined #openstack-infra18:57
*** lpetrut has joined #openstack-infra18:58
*** lpetrut has quit IRC18:59
*** lpetrut has joined #openstack-infra19:00
*** haleyb has joined #openstack-infra19:01
*** lucasagomes has quit IRC19:02
*** lbragstad has quit IRC19:11
*** rlandy is now known as rlandy|brb19:13
*** zbr|ruck is now known as zbr19:14
*** ykarel|away has quit IRC19:17
openstackgerritChristopher Dearborn proposed openstack/project-config master: python-dracclient zuulv3 changes  https://review.opendev.org/68640919:18
*** gfidente|afk has quit IRC19:27
*** yamamoto has joined #openstack-infra19:30
*** ociuhandu has joined #openstack-infra19:31
*** yamamoto has quit IRC19:35
*** ociuhandu has quit IRC19:37
*** lpetrut has quit IRC19:41
openstackgerritTristan Cacqueray proposed opendev/glean master: Set Type=Ethernet for RedHat system  https://review.opendev.org/68647419:48
*** gmann is now known as gmann_afk19:52
openstackgerritPaul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9  https://review.opendev.org/67485420:03
openstackgerritChristopher Dearborn proposed openstack/project-config master: python-dracclient zuulv3 changes  https://review.opendev.org/68640920:09
*** pcaruana has quit IRC20:18
*** jamesmcarthur has quit IRC20:21
*** whoami-rajat has quit IRC20:28
*** lbragstad has joined #openstack-infra20:29
*** kjackal_v2 has quit IRC20:37
*** rlandy|brb is now known as rlandy20:38
corvusi have restored access to the infra-root google account using a prepaid sim card i was given.  i don't really consider this a scalable solution, but this will keep our zuul-gerrit poc moving and we can continue to talk to the gerrit/google folks about improvements.20:40
corvusthe http password i previously set up works now -- apparently it stopped working because google disabled the account20:41
corvusinfra-root: https://review.opendev.org/686049 should be gtg20:42
*** Goneri has quit IRC20:46
fungicorvus: did google disable the account because they wanted it validated by phone within some maximum timeframe after creation? or for unrelated reasons? or do you know?20:48
fungiand i agree, needing a "burner phone" to create google accounts seems extreme20:48
corvusfungi: https://imgur.com/a/XkOIjvi20:49
corvusthat's a listing of 3 events; i've expanded the middle one with details20:49
*** factor has quit IRC20:49
*** tosky has joined #openstack-infra20:50
corvusthe details of the other events are more...detailed....20:50
corvusapparently they decided later that evening that the account i created in the morning was bogus20:50
fungi"looked like it was created with multiple other accounts or created by a program"20:51
fungineat20:51
fungi"device: unknown device"20:51
fungimaybe a web browser with "too much privacy" in place20:52
corvusi didn't create any other accounts that day (or, really, in quite some time).  i did use a program to create it called firefox.20:52
fungialso you signed on from linux. that hacker operating system20:52
fungino good can come of it20:52
corvusyeah, an optimistic reading of this is "insufficiently fingerprinted"20:53
fungi(oh, wait, they based their entire mobile operating system on it didn't they?)20:53
corvusfungi: well, for now.  later there's https://en.wikipedia.org/wiki/Google_Fuchsia20:53
* paladox sooooo happy about this comment https://github.com/google/guava/issues/3602#issuecomment-53811915720:53
openstackgerritTristan Cacqueray proposed opendev/glean master: Set Type=Ethernet for RedHat system  https://review.opendev.org/68647420:54
fungipaladox: that's in service of solving thread locks in gerrit?20:55
paladoxyup20:55
paladoxericson had it so they have cafeine running on a forked 2.1420:55
paladoxbut as we use 2.15 we carn't use caffeine until some issue is fixed :(20:56
fungicorvus: wow, so fuscia's microkernel is descended from haiku (the beos clone)?21:05
fungii haven't thought about beos in ages21:05
fungigranted, openbeos/haiku has been under development for 18 years now and is still a beta test ;)21:07
tristanCfungi: not sure about the origin, but it does have an interesting ci system that can do cross repos testing using this tool: https://fuchsia.googlesource.com/jiri21:09
tristanCand it seems like the build report features a comment system, perhaps to do log-review?21:10
*** slaweq has quit IRC21:14
*** goldyfruit___ has quit IRC21:15
*** goldyfruit has joined #openstack-infra21:15
openstackgerritAndrei Nistor proposed openstack/diskimage-builder master: bootloader: make serial console configurable  https://review.opendev.org/68649521:17
openstackgerritAndrei Nistor proposed openstack/diskimage-builder master: bootloader: make serial console configurable  https://review.opendev.org/68649521:18
fungithat's an interesting idea... being able to leave comments/annotations on build logs21:23
*** auristor has quit IRC21:25
mnaserinfra-root: i'm cleaning up the issues for sjc1 and i noticed a lot of stale volumes right now (undeleted in-use ones)21:33
mnaserdoes anyone have a script handy for getting orphans or should i get writing..21:33
fungimnaser: i mainly looked for volumes with an old creation time21:38
fungiand then tried to correlate the attachments via what instances nova reported as still existing21:39
fungino script, was probably just lengthy one-liners of nested osc invocations21:39
clarkbthey usually show up as availabke not in use21:40
openstackgerritMerged zuul/zuul-registry master: Add streaming download support  https://review.opendev.org/68644921:41
tristanCfungi: actually, build comments are automatic annotations, it's implemented by this gerrit plugins: https://chromium.googlesource.com/infra/gerrit-plugins/tricium21:41
fungiclarkb: depends, i think. the leaked ones where server delete reported failure but eventually worked show up as available volumes21:41
fungier, now i don't actually remember21:42
fungiif cinder thinks they're still attached they may have continued to show up as in-use21:42
fungibut then openstack volume show would list an attachment to a server instance uuid which was not in the openstack server list output21:43
mnaseryeah so id hae to do that lokup21:44
fungiright, i think i just did some shell loops and expansions21:44
openstackgerritAndrei Nistor proposed openstack/diskimage-builder master: bootloader: make serial console configurable  https://review.opendev.org/68649521:46
fungisomething like... for a in $(for v in $(openstack volume list|grep ...|cut ...);do openstack volume show $v|grep ...|cut ...;done);do openstack server list|grep $a||echo $v;done21:46
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Fix merge error in streaming support  https://review.opendev.org/68650521:47
fungii write a ton of terrible nested loop one-liners in shell21:47
mnaserfungi: meet -c and -f :p21:47
fungiosc options?21:47
fungiyeah21:47
fungii should make more use of its output formatting21:48
mnaserreduced the amount of cuts and awks :p21:48
*** auristor has joined #openstack-infra21:49
Shrewsmnaser: if you can find a safe way to do it with openstacksdk, we might could integrate it into nodepool21:50
clarkbI can never get -c to work21:50
mnaser-c ID -f value usually is the trick for example21:50
mnaserShrews: ok so ill actually scrap my bash code to write it with openstacksdk21:50
fungiwell, we can't really do much with that in nodepool other than use it to skip attempts at re-deleting the affected volumes21:51
fungier, skip re-attempts at deleting the affected volumes21:51
mnasermaybe a friendly log msg21:51
fungiyeah21:52
fungiat least for the ones i was finding, we needed a cloud admin to perform the actual deletion21:52
Shrewsoh, well, if it needs different perms...21:52
fungisince it's a confusion between nova and cinder21:52
funginova fails to tell cinder the volume attachment is no longer needed, so cinder fails to allow the user to delete the volume21:53
fungier, refuses to allow the user to delete the volume21:53
fungithinking it's still attached to something21:53
mnaser^ yep21:53
mnaseran admin needs top reset it21:53
openstackgerritMerged zuul/zuul-registry master: Update test script to match playbook  https://review.opendev.org/68645021:58
ianwdonnyd: over in #zuul looking at a couple of nodepool functional tests that are failing on fortnebula with what looks very much like the inner test-vm hitting it's 10 minute boot timeout22:03
ianware there known issues?  perhaps with i/o?22:04
fungii think he's said the nodes there are mainly cpu-bound22:04
*** mattw4 has quit IRC22:05
*** mattw4 has joined #openstack-infra22:05
fungiso non-accelerated vm nesting may just be slower there. or perhaps there's some recent performance issue which has cropped up22:06
*** pkopec has quit IRC22:06
ianwthe boot log is basically -> https://paste.fedoraproject.org/paste/FjWgq95WA1QFrTh3Q0bkvA22:07
fungii gather the storage is all nvme, with a ton of bandwidth22:07
fungiso wouldn't expect to see poor filesystem i/o performance22:08
*** rcernin has joined #openstack-infra22:08
ianwthere's doesn't seem to be one particluar point in the boot where things take a big jump ... just lots of slow things22:08
fungicould it be thrashing swap on the job node and not managing any fs cache?22:09
ianw2019-10-03 02:40:36,188 DEBUG nodepool.NodeLauncher: [node: 0000000000] [  465.082869] growroot[444]: + growpart /dev/vda 122:10
ianw2019-10-03 02:40:36,189 DEBUG nodepool.NodeLauncher: [node: 0000000000] [  499.918863] growroot[444]: CHANGED: partition=1 start=2048 old: size=3616256 end=3618304 new: size=10483679,end=1048572722:10
ianwthat's the best part of 30s right?22:10
fungiyep, and some change22:10
funginearly 35 seconds22:11
mnasercan you check if it used kvm and not qemu22:11
mnaserbecause afaik growpart doesnt actaully do much of io, it just kinda does a little bit of black magic that remaps the partition22:11
mnaserif the io was that bad id think it would be a looot slower22:11
fungithat's also only growing it from 3mb to 10mb right?22:12
ianwyeah, and the image build, which is basically all io went from 2019-10-03 02:04:20.118 .. 2019-10-03 02:18:13.718 ... that's about right22:13
fungiso even if it were allocating additional blocks, that's a relatively small amount22:13
ianwit should not take longer to boot the vm than build the image :)22:13
mnaserok so i got all the orphans22:15
mnaseri am ready for the self inflicted ddos22:15
donnydianw: all the hypervisors use local NVMEs so not likely to be IO bound22:16
donnydhttp://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=now-6h&to=now22:16
donnydLooking here I don't see any launch errors22:17
donnydIts pretty normal however whenever a new image is loaded for it to take a bit to populate due to glance22:17
ianwdonnyd: so this is booting a vm inside the job22:17
donnydoh22:17
fungiyeah, growpart in the nested (cirros) vm is clocking in waaay slow22:18
donnydyea my CPU's are not exactly new and without nested virt enabled I can't do much else22:18
donnydhttps://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&refresh=30s22:18
donnydAlso looking here I see some pretty sizable spikes, so you may have fallen into one of those22:19
fungiyeah, i could see that happening if a significant cpu spike happened on the host due to load from other job nodes22:20
ianwsuch as ... a bunch of nodepool jobs all trying to boot inner vm's at the same time22:20
fungi;)22:22
ianwi guess grafana is showing me that in local time22:22
donnydI turned down the over-subscription to try and help... but they just aren't super fast CPU's22:22
ianwhow long ago was 2019-10-03 02:28:27,937 UTC?22:22
funginearly 20 hours22:22
*** rh-jelabarre has quit IRC22:23
openstackgerritTristan Cacqueray proposed zuul/zuul-registry master: Add type annotations  https://review.opendev.org/68624922:23
ianwmaybe ~ https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&from=1570072636623&to=157007550037522:28
donnydianw: that is possible. Is this a tripelo job?22:29
*** markvoelker has quit IRC22:30
donnyddo you have a link for logs from the job?22:30
*** rlandy is now known as rlandy|bbl22:31
ianwhttps://zuul.opendev.org/t/openstack/build/0cf5dfc8bde84aed8c88b0887a5d79bb22:31
ianwhttps://zuul.opendev.org/t/openstack/build/076d513990da4dd4922550acbf4adced22:31
ianwfailed22:31
ianwhttps://zuul.opendev.org/t/openstack/build/eed91655b2f948ea974dd92fb8b7ba3922:31
ianwpassed ... all ran on fortnebula around the same time22:32
ianwcorvus: is it intentional to not collect the devstack logs in the nodepool functional tests, or an oversight?22:32
donnydit would be really handy to have a way to manually reproduce the job for these types of things22:34
corvusianw: mostly intentional -- that might be a better thing for the dib child job22:34
donnydhttp://logstash.openstack.org/#/dashboard/file/logstash.json?query=node_provider:%5C%22fortnebula-regionone%5C%22%20AND%20filename:%5C%22job-output.txt%5C%22%20AND%20message:%5C%22RUN%20END%20RESULT_TIMED_OUT%5C%22&from=24h22:35
*** panda is now known as panda|off22:35
donnydhrmmm that seems a little high22:35
donnydusually there are just the jobs that get close to the timeout anyways that don't run well on FN22:35
donnydand the ones that tinker with the network that don't account for ipv6 being the primary mode of communications22:36
donnydbut it looks like all the DIB jobs failed22:37
donnydwould this be a good way to test our nested virt label?22:37
donnydbut running this query filename:"job-output.txt" AND message:"RUN END RESULT_TIMED_OUT" AND build_name:"dib-nodepool-functional-openstack-fedora-29-src"22:38
donnydits not comical... but in the last 24 hours 100% of that job has run on FN... with 100% failure rate...22:39
ianwthat sounds about right :)22:40
donnydand in the last 7d there have only been 4 runs of that job.. all on FN... all Failed22:42
ianwhrm, there would have been many more than 4 ... i have a very large centos 8 stack22:42
donnydjust fedora-2922:44
donnydoh and I am only looking for timeouts22:44
ianwhttps://zuul.opendev.org/t/openstack/build/0c400fb1ac3c4decb7882159b22c1601 is another example of a different job22:46
ianwdib-functests-centos7-python222:46
ianwthat doesn't actually boot anything, only build22:46
ianwusually a 30-40 minute job22:47
donnydgonna take me a little bit to sift through the logs to figure out where the delays are at22:47
ianwi don't understand that one22:52
ianw2019-10-03 09:32:50.440 | Build completed successfully22:52
donnydhttps://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0c4/684860/18/check/dib-functests-centos7-python2/0c400fb/logs/22:52
donnydwell it looks like they all built22:53
ianw2019-10-03 09:25:00.513841 | centos-7 | [ubuntu-minimal/xenial-build-succeeds] Output logs going to: /home/zuul/logs/ubuntu-minimal_xenial-build-succeeds.log22:53
ianw2019-10-03 09:41:59.496671 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/diskimage-builder/playbooks/dib-functests/run.yaml@master]22:53
ianwwhy did it stop between 09:25 -> 09:41?22:53
donnydlooks like the acutal meat of the job starts here22:54
donnyd2019-10-03 08:16:45.712130 | centos-7 | [JOB-CONTROL] Starting new job22:54
donnydends here22:55
donnyd2019-10-03 09:25:00.513841 | centos-7 | [ubuntu-minimal/xenial-build-succeeds] Output logs going to: /home/zuul/logs/ubuntu-minimal_xenial-build-succeeds.log22:55
ianwyeah i think it started on the bionic build, but we didn't capture anything for that22:55
donnydwhich is right around the 40 minutes you were saying it takes to run right22:55
donnydhttps://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0c4/684860/18/check/dib-functests-centos7-python2/0c400fb/logs/ubuntu-minimal_bionic-build-succeeds.PASS.log22:56
ianwso similar issue,  was working, but just slow22:56
donnydthere is the bionic job ^^^22:56
donnydnot super familar with the job22:56
ianwyeah i just loops though a bunch of different builds, but doesn't actually boot the results.  just to make sure a .qcow2 comes out the other side22:57
donnydand this job here has something else broken22:58
donnydhttps://zuul.opendev.org/t/openstack/build/076d513990da4dd4922550acbf4adced22:58
donnydhttps://1251a16130059a3c4478-a4f75ebbaac54881743ae326d50075b2.ssl.cf1.rackcdn.com/684460/10/check/dib-nodepool-functional-openstack-fedora-30-src/076d513/job-output.txt22:58
donnyd2019-10-03 01:53:45.999760 | TASK [Check nodepool functionality]22:58
donnyd2019-10-03 02:43:29.692653 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/zuul/nodepool/playbooks/nodepool-functional-openstack/run.yaml@master]22:59
donnydseems to be stuck in a loop for 45 minutes22:59
ianwthat's more or less right, it's waiting for the inner vm to come up23:00
ianw2019-10-03 02:39:02,191 DEBUG nodepool.NodeLauncher: [node: 0000000000] [  563.356060] systemd[1]: Reached target sshd-keygen.target.23:00
ianwagain up near the 600s timeout; that's from the launch logs23:00
ianwhttps://1251a16130059a3c4478-a4f75ebbaac54881743ae326d50075b2.ssl.cf1.rackcdn.com/684460/10/check/dib-nodepool-functional-openstack-fedora-30-src/076d513/nodepool/nodepool-launcher.log23:00
*** goldyfruit_ has joined #openstack-infra23:00
ianwwe wait 10 minutes for the node to appear, before we consider it dead.  i think nodepool does that maybe 3 times or so23:00
*** mattw4 has quit IRC23:01
*** tkajinam has joined #openstack-infra23:01
ianw2019-10-03 02:28:27,937 DEBUG nodepool.NodeLauncher: [node: 0000000000] [  539.215851] systemd[1]: Started Grow root partition.23:01
ianwthat was the first attempt, the ^^ was the second attempt, it didn't get to a 3rd23:02
*** goldyfruit has quit IRC23:02
*** mattw4 has joined #openstack-infra23:02
*** bdodd has quit IRC23:02
ianw... so, not sure.  i would really like to merge the centos 8 queue, but it does rely on these in gate23:04
donnyd2019-10-03 01:53:47,866 DEBUG keystoneauth.session: RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}23:05
donnydhttps://1251a16130059a3c4478-a4f75ebbaac54881743ae326d50075b2.ssl.cf1.rackcdn.com/684460/10/check/dib-nodepool-functional-openstack-fedora-30-src/076d513/nodepool/nodepool-launcher.log23:05
donnyd2019-10-03 01:53:47,866 DEBUG keystoneauth.session: RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}23:05
donnyd2019-10-03 01:53:47,866 DEBUG keystoneauth.session: GET call to http://192.168.48.87/compute/v2.1 used request id req-7729ce9a-1551-45ab-8401-77650fb74d0823:05
donnyd2019-10-03 01:53:47,867 DEBUG keystoneauth.session: Request returned failure status: 40123:05
donnydLooks to me like something in the job is busted23:05
*** jtomasek has quit IRC23:07
*** petevg has quit IRC23:09
ianwi feel like that may be something to do with the openstacksdk calls ... similar appears on a good run23:09
ianwhttps://zuul.opendev.org/t/openstack/build/315196acac334c3282a18310fc59f58a/log/nodepool/nodepool-launcher.log23:09
ianwit does23:09
ianwGET call to http://162.242.237.62/compute/v2.123:09
ianw(401)23:09
ianwGET call to compute for http://162.242.237.62/compute/v2.1/servers/detail23:09
ianw(works)23:09
fungiyeah, i think if those were a problem they should be logged higher than debug loglevel23:11
donnydI can say I have nested virt enabled, so maybe it could accelerate those workloads (probably by quite a bit)23:17
donnydbut that is about all I can really do23:18
donnydianw: do you have a method to run this outside the CI?23:18
donnydmaybe to could do more something else to accelerate this job23:18
donnydits surely worth a try23:18
ianwit is moderately involved; first devstack runs to give the job a cloud environment, then nodepool is setup to run against it, a build triggered, nodepool uploads the images and then a test vm is started23:21
donnydoh23:21
ianwbut if that test vm is taking +10 minutes to boot, that's a bottleneck we can't really speed up.  it's already really doing as little as it can do23:22
ianw(modulo nested virt i guess)23:23
*** jamesmcarthur has joined #openstack-infra23:23
*** tosky has quit IRC23:23
donnydwe created a custom label so people can give that a swing. would it be possible to create an experimental job that uses it23:23
*** jamesmcarthur has quit IRC23:24
donnydi wouldn't think it should take 10 minutes to spin up a vm with qemu though23:24
ianwyes.  does nova just get it right if it's enabled, or does devstack require further setup?23:24
donnydseems a little steep23:24
donnydi would think you would have to tell devstack to use kvm..  not sure if it will just use it automagically23:25
*** mriedem has quit IRC23:26
ianwThis23:27
ianwcan be verified by checking for the presence of the file ``/dev/kvm`` in23:27
ianwyour VM. If it is present, DevStack will default to using the config23:27
ianwattribute ``virt_type = kvm`` in ``/etc/nova.conf``;23:27
mnaserianw: i believe in devstack job it is pinned to not use nested virt tho23:27
mnaseri think23:27
donnydi think mnaser is correct23:28
donnydotherwise it would already be using it on a couple different clouds23:28
ianw        # Gate jobs can't deal with nested virt. Disable it.23:28
ianw        LIBVIRT_TYPE: qemu23:28
donnyddo you have the devstack build... i can manually spin up a instance and then check it to see how long an instance actually takes23:29
*** sthussey has quit IRC23:33
ianwi should be able to get some experimental jobs with it turned on, just a tick23:34
*** igordc has quit IRC23:36
*** armax has quit IRC23:38
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [dnm] testing nested virt labels  https://review.opendev.org/68651023:39
ianwThe nodeset "nested-virt-ubuntu-bionic" was not found.23:40
ianwoohh, we added the nodes to nodepool, probably not to config?23:41
*** jtomasek has joined #openstack-infra23:41
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [dnm] testing nested virt labels  https://review.opendev.org/68651023:44
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [dnm] testing nested virt labels  https://review.opendev.org/68651023:45
ianwinteresting, that didn't work the jobs are going to rax23:47
*** michael-beaver has quit IRC23:47
*** mattw4 has quit IRC23:49
*** xenos76 has quit IRC23:50
ianwhrm, no they bailed with NODE_FAILURE23:52
*** jtomasek has quit IRC23:54
*** goldyfruit_ has quit IRC23:56
*** jtomasek has joined #openstack-infra23:56
ianwarrgh, that's right, a bunch of jobs run on xenial because ... debootstrap bugs23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!