| *** aeng_ has joined #openstack-infra | 00:01 | |
| *** aeng_ has quit IRC | 00:01 | |
| *** dbecker has joined #openstack-infra | 00:02 | |
| *** bdodd has joined #openstack-infra | 00:04 | |
| *** bdodd has quit IRC | 00:05 | |
| *** hongbin has quit IRC | 00:05 | |
| *** bdodd has joined #openstack-infra | 00:06 | |
| fungi | right, i'd be worried a `show tac` with those sfps inserted would cause support to throw up their hands if you run into a problem | 00:08 |
|---|---|---|
| fungi | showing my age... i guess cisco hasn | 00:10 |
| openstackgerrit | Jill Rouleau proposed openstack-infra/project-config master: Add Tripleo Ansible repo https://review.openstack.org/583415 | 00:10 |
| fungi | 't called their support dept the "technical assistance center" in an age and a half | 00:10 |
| openstackgerrit | Jill Rouleau proposed openstack-infra/project-config master: Add Tripleo Ansible repo https://review.openstack.org/583415 | 00:12 |
| *** tosky has quit IRC | 00:12 | |
| *** anteaya has quit IRC | 00:15 | |
| *** dingyichen has joined #openstack-infra | 00:18 | |
| *** graphene has quit IRC | 00:20 | |
| *** graphene has joined #openstack-infra | 00:22 | |
| *** zhangfei has joined #openstack-infra | 00:30 | |
| *** sthussey has quit IRC | 00:31 | |
| *** threestrands has joined #openstack-infra | 00:40 | |
| *** threestrands has quit IRC | 00:40 | |
| *** threestrands has joined #openstack-infra | 00:40 | |
| *** jamesmcarthur has joined #openstack-infra | 00:45 | |
| *** graphene has quit IRC | 00:53 | |
| *** graphene has joined #openstack-infra | 00:55 | |
| *** dave-mccowan has joined #openstack-infra | 00:56 | |
| *** jamesmcarthur has quit IRC | 00:57 | |
| *** graphene has quit IRC | 00:58 | |
| *** graphene has joined #openstack-infra | 00:59 | |
| pabelanger | corvus: clarkb: we've rolled 1GB zuul-mergers into rdoproject yesterday, so far they are working well. Maybe something to consider in openstack-infra if we ever rebuild them | 01:01 |
| *** dave-mccowan has quit IRC | 01:09 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** caphrim007 has joined #openstack-infra | 01:11 | |
| *** caphrim007 has quit IRC | 01:12 | |
| pabelanger | corvus: clarkb: also, noticed we haven't done a zuul restart in some time. I should be able to assist tomorrow if we are interested in doing that | 01:12 |
| *** harlowja has quit IRC | 01:13 | |
| *** caphrim007 has joined #openstack-infra | 01:15 | |
| *** iyamahat has joined #openstack-infra | 01:15 | |
| *** slaweq has quit IRC | 01:15 | |
| *** iyamahat has quit IRC | 01:16 | |
| *** dave-mccowan has joined #openstack-infra | 01:17 | |
| fungi | in lighter news, we haven't needed a zuul restart in some time | 01:19 |
| fungi | yay memory fixes! | 01:19 |
| *** iyamahat has joined #openstack-infra | 01:21 | |
| *** dave-mcc_ has joined #openstack-infra | 01:22 | |
| *** mrsoul_ has joined #openstack-infra | 01:22 | |
| *** dave-mccowan has quit IRC | 01:24 | |
| *** mriedem has quit IRC | 01:24 | |
| *** zhangfei has quit IRC | 01:25 | |
| *** mschuppert has quit IRC | 01:25 | |
| *** rlandy has joined #openstack-infra | 01:35 | |
| *** zhangfei has joined #openstack-infra | 01:38 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Use readthedocs webhook to trigger build https://review.openstack.org/579434 | 01:40 |
| *** annp has quit IRC | 01:43 | |
| *** zhangfei has quit IRC | 01:44 | |
| *** larainema has quit IRC | 01:46 | |
| *** tianhui_ has quit IRC | 01:57 | |
| *** zhangfei has joined #openstack-infra | 01:57 | |
| *** hongbin has joined #openstack-infra | 01:57 | |
| *** rlandy has quit IRC | 02:02 | |
| *** jamesmcarthur has joined #openstack-infra | 02:02 | |
| *** jamesmcarthur has quit IRC | 02:07 | |
| pabelanger | fungi: interesting enough, I decided to look at cacti, and we do seem to be trending up a little: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 02:10 |
| *** AhmadM_ has quit IRC | 02:10 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** rosmaita has quit IRC | 02:19 | |
| *** logan- has quit IRC | 02:20 | |
| *** logan- has joined #openstack-infra | 02:21 | |
| *** larainema has joined #openstack-infra | 02:22 | |
| *** gcb has joined #openstack-infra | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:23 | |
| *** jamesmcarthur has quit IRC | 02:27 | |
| *** shu-mutow has joined #openstack-infra | 02:28 | |
| *** roman_g has quit IRC | 02:29 | |
| *** alex_xu has quit IRC | 02:31 | |
| *** alex_xu has joined #openstack-infra | 02:33 | |
| *** yamahata has quit IRC | 02:33 | |
| *** iyamahat has quit IRC | 02:37 | |
| *** dpawlik has joined #openstack-infra | 02:41 | |
| *** dpawlik has quit IRC | 02:45 | |
| *** jamesmcarthur has joined #openstack-infra | 02:54 | |
| *** ramishra has joined #openstack-infra | 02:55 | |
| *** jamesmcarthur has quit IRC | 02:58 | |
| *** Keitaro has quit IRC | 03:02 | |
| *** annp has joined #openstack-infra | 03:11 | |
| *** cshastri has joined #openstack-infra | 03:15 | |
| *** jamesmcarthur has joined #openstack-infra | 03:15 | |
| *** graphene has quit IRC | 03:16 | |
| *** graphene has joined #openstack-infra | 03:18 | |
| *** jamesmcarthur has quit IRC | 03:19 | |
| *** udesale has joined #openstack-infra | 03:24 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 03:26 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 03:28 |
| *** jiapei has joined #openstack-infra | 03:29 | |
| *** Keitaro has joined #openstack-infra | 03:34 | |
| *** jamesmcarthur has joined #openstack-infra | 03:35 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Use readthedocs webhook to trigger build https://review.openstack.org/579434 | 03:37 |
| *** jamesmcarthur has quit IRC | 03:40 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Use trigger-readthedocs role https://review.openstack.org/583449 | 03:41 |
| *** hongbin has quit IRC | 03:49 | |
| *** lpetrut has joined #openstack-infra | 03:51 | |
| *** armax has quit IRC | 03:52 | |
| *** armax has joined #openstack-infra | 03:52 | |
| *** jamesmcarthur has joined #openstack-infra | 03:56 | |
| *** armax has quit IRC | 03:57 | |
| *** caphrim007_ has joined #openstack-infra | 03:59 | |
| *** zhangfei has quit IRC | 04:00 | |
| *** jamesmcarthur has quit IRC | 04:01 | |
| *** caphrim007 has quit IRC | 04:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 04:05 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 04:05 |
| *** Bhujay has joined #openstack-infra | 04:06 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 04:07 |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** trown has quit IRC | 04:24 | |
| *** zhangfei has joined #openstack-infra | 04:25 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 04:26 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page https://review.openstack.org/537870 | 04:26 |
| *** trown|brb has joined #openstack-infra | 04:34 | |
| *** jamesmcarthur has joined #openstack-infra | 04:37 | |
| *** janki has joined #openstack-infra | 04:39 | |
| *** lpetrut has quit IRC | 04:39 | |
| *** lpetrut has joined #openstack-infra | 04:42 | |
| *** jamesmcarthur has quit IRC | 04:42 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Update pylint to 1.7.6, uncap networkx https://review.openstack.org/583413 | 04:46 |
| *** lpetrut has quit IRC | 04:54 | |
| *** udesale has quit IRC | 04:55 | |
| *** udesale has joined #openstack-infra | 04:55 | |
| *** adrianreza has quit IRC | 04:58 | |
| *** jamesmcarthur has joined #openstack-infra | 04:58 | |
| *** jamesmcarthur has quit IRC | 05:03 | |
| *** timburke_ has joined #openstack-infra | 05:09 | |
| *** ktsuyuzaki has joined #openstack-infra | 05:09 | |
| *** dayou has joined #openstack-infra | 05:09 | |
| *** calbers_ has joined #openstack-infra | 05:10 | |
| *** vabada_ has joined #openstack-infra | 05:10 | |
| *** openstackgerrit has quit IRC | 05:10 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** fabo_ has joined #openstack-infra | 05:11 | |
| *** timothyb89_ has joined #openstack-infra | 05:12 | |
| *** lucasagomes_ has joined #openstack-infra | 05:12 | |
| *** chason_ has joined #openstack-infra | 05:14 | |
| *** kjackal has joined #openstack-infra | 05:15 | |
| *** links has joined #openstack-infra | 05:15 | |
| *** Bhujay has quit IRC | 05:16 | |
| *** efried has quit IRC | 05:16 | |
| *** vabada has quit IRC | 05:16 | |
| *** isviridov_away has quit IRC | 05:16 | |
| *** vaidy has quit IRC | 05:16 | |
| *** bauzas has quit IRC | 05:16 | |
| *** icey has quit IRC | 05:16 | |
| *** Dobroslaw has quit IRC | 05:16 | |
| *** hamzy has quit IRC | 05:16 | |
| *** ihar has quit IRC | 05:16 | |
| *** calbers has quit IRC | 05:16 | |
| *** lxkong has quit IRC | 05:16 | |
| *** aspiers has quit IRC | 05:16 | |
| *** dayou_ has quit IRC | 05:16 | |
| *** serverascode has quit IRC | 05:16 | |
| *** robcresswell has quit IRC | 05:16 | |
| *** LinuxJedi has quit IRC | 05:16 | |
| *** mwhahaha has quit IRC | 05:16 | |
| *** fdegir has quit IRC | 05:16 | |
| *** auggy has quit IRC | 05:16 | |
| *** jpmaxman_ has quit IRC | 05:16 | |
| *** johnsom has quit IRC | 05:16 | |
| *** ameade has quit IRC | 05:16 | |
| *** dham1 has quit IRC | 05:16 | |
| *** simondodsley has quit IRC | 05:16 | |
| *** coreycb has quit IRC | 05:16 | |
| *** mnasiadka has quit IRC | 05:16 | |
| *** Dmitrii-Sh has quit IRC | 05:16 | |
| *** pas-ha has quit IRC | 05:16 | |
| *** patricku has quit IRC | 05:16 | |
| *** chrisyang_0660 has quit IRC | 05:16 | |
| *** portdirect has quit IRC | 05:16 | |
| *** jamespage has quit IRC | 05:16 | |
| *** NobodyCam has quit IRC | 05:16 | |
| *** DuncanT has quit IRC | 05:16 | |
| *** briancurtin has quit IRC | 05:16 | |
| *** mgkwill has quit IRC | 05:16 | |
| *** davidlenwell has quit IRC | 05:16 | |
| *** beisner has quit IRC | 05:16 | |
| *** kien-ha has quit IRC | 05:16 | |
| *** lucasagomes has quit IRC | 05:16 | |
| *** Roamer` has quit IRC | 05:16 | |
| *** jmccrory has quit IRC | 05:16 | |
| *** timothyb89 has quit IRC | 05:16 | |
| *** jamielennox has quit IRC | 05:16 | |
| *** timburke has quit IRC | 05:16 | |
| *** kota_ has quit IRC | 05:16 | |
| *** fabo has quit IRC | 05:16 | |
| *** chason has quit IRC | 05:16 | |
| *** tristanC has quit IRC | 05:16 | |
| *** slaweq has quit IRC | 05:16 | |
| *** icey has joined #openstack-infra | 05:17 | |
| *** sambetts_ has quit IRC | 05:18 | |
| *** toabctl has quit IRC | 05:18 | |
| *** jamesmcarthur has joined #openstack-infra | 05:19 | |
| *** Keitaro has quit IRC | 05:20 | |
| *** toabctl has joined #openstack-infra | 05:21 | |
| *** Keitaro has joined #openstack-infra | 05:23 | |
| *** sambetts_ has joined #openstack-infra | 05:23 | |
| *** jamesmcarthur has quit IRC | 05:24 | |
| *** krtaylor has quit IRC | 05:27 | |
| *** harlowja has joined #openstack-infra | 05:28 | |
| *** chkumar|ruck has quit IRC | 05:28 | |
| *** chandankumar has joined #openstack-infra | 05:29 | |
| *** chandankumar is now known as chkumar|ruck | 05:29 | |
| prometheanfire | us bandersnatch behind or something? | 05:31 |
| prometheanfire | reqs jobs are failing for new releases (that have been our for hours) | 05:31 |
| prometheanfire | http://logs.openstack.org/81/583281/1/check/requirements-tox-py27-check-uc/48ba207/job-output.txt.gz#_2018-07-18_03_58_37_499565 | 05:31 |
| *** harlowja has quit IRC | 05:32 | |
| *** abhishekk has joined #openstack-infra | 05:32 | |
| abhishekk | hi, yesterday a new library of glance_store 0.25.0 is released which is available on pypi but not available in mirror used by infra | 05:33 |
| abhishekk | any idea when it will be available? | 05:33 |
| abhishekk | http://logs.openstack.org/34/583334/1/check/cross-glance-py27/2ec3f06/job-output.txt.gz#_2018-07-18_05_11_29_354463 | 05:34 |
| *** janki has quit IRC | 05:34 | |
| abhishekk | Reference ^^^ | 05:34 |
| *** davidlenwell has joined #openstack-infra | 05:35 | |
| *** Bhujay has joined #openstack-infra | 05:35 | |
| *** efried has joined #openstack-infra | 05:35 | |
| *** isviridov_away has joined #openstack-infra | 05:35 | |
| *** vaidy has joined #openstack-infra | 05:35 | |
| *** bauzas has joined #openstack-infra | 05:35 | |
| *** Dobroslaw has joined #openstack-infra | 05:35 | |
| *** hamzy has joined #openstack-infra | 05:35 | |
| *** ihar has joined #openstack-infra | 05:35 | |
| *** lxkong has joined #openstack-infra | 05:35 | |
| *** aspiers has joined #openstack-infra | 05:35 | |
| *** serverascode has joined #openstack-infra | 05:35 | |
| *** robcresswell has joined #openstack-infra | 05:35 | |
| *** LinuxJedi has joined #openstack-infra | 05:35 | |
| *** mwhahaha has joined #openstack-infra | 05:35 | |
| *** fdegir has joined #openstack-infra | 05:35 | |
| *** jpmaxman_ has joined #openstack-infra | 05:35 | |
| *** auggy has joined #openstack-infra | 05:35 | |
| *** johnsom has joined #openstack-infra | 05:35 | |
| *** ameade has joined #openstack-infra | 05:35 | |
| *** dham1 has joined #openstack-infra | 05:35 | |
| *** simondodsley has joined #openstack-infra | 05:35 | |
| *** patricku has joined #openstack-infra | 05:35 | |
| *** coreycb has joined #openstack-infra | 05:35 | |
| *** mnasiadka has joined #openstack-infra | 05:35 | |
| *** Dmitrii-Sh has joined #openstack-infra | 05:35 | |
| *** portdirect has joined #openstack-infra | 05:35 | |
| *** pas-ha has joined #openstack-infra | 05:35 | |
| *** jamespage has joined #openstack-infra | 05:35 | |
| *** chrisyang_0660 has joined #openstack-infra | 05:35 | |
| *** NobodyCam has joined #openstack-infra | 05:35 | |
| *** DuncanT has joined #openstack-infra | 05:35 | |
| *** briancurtin has joined #openstack-infra | 05:35 | |
| *** mgkwill has joined #openstack-infra | 05:35 | |
| *** beisner has joined #openstack-infra | 05:35 | |
| *** kien-ha has joined #openstack-infra | 05:35 | |
| *** Roamer` has joined #openstack-infra | 05:35 | |
| *** jmccrory has joined #openstack-infra | 05:35 | |
| *** jamielennox has joined #openstack-infra | 05:35 | |
| *** davidlenwell has quit IRC | 05:35 | |
| *** davidlenwell has joined #openstack-infra | 05:35 | |
| *** jangutter has quit IRC | 05:38 | |
| *** aeng_ has joined #openstack-infra | 05:38 | |
| *** quiquell|off is now known as quiquell | 05:39 | |
| *** jiapei has quit IRC | 05:39 | |
| *** jamesmcarthur has joined #openstack-infra | 05:40 | |
| *** ianychoi has joined #openstack-infra | 05:43 | |
| *** jamesmcarthur has quit IRC | 05:44 | |
| *** krtaylor has joined #openstack-infra | 05:51 | |
| *** armax has joined #openstack-infra | 05:52 | |
| *** dayou has quit IRC | 05:53 | |
| *** dayou has joined #openstack-infra | 05:54 | |
| *** mrsoul_ is now known as mschuppert | 05:54 | |
| *** armax has quit IRC | 05:57 | |
| *** jamesmcarthur has joined #openstack-infra | 06:01 | |
| *** jamesmcarthur has quit IRC | 06:05 | |
| *** tobiash has quit IRC | 06:07 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** kjackal has quit IRC | 06:13 | |
| *** openstackgerrit has joined #openstack-infra | 06:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to not-voting check-requirements for git-upstream https://review.openstack.org/582395 | 06:14 |
| *** slaweq has quit IRC | 06:16 | |
| *** dhajare has joined #openstack-infra | 06:18 | |
| *** dbecker has quit IRC | 06:20 | |
| prometheanfire | abhishekk: I was just mentioning that, not sure who's around, but I was told that they may want to kick bandersnatch | 06:20 |
| abhishekk | prometheanfire, thanks for the update | 06:21 |
| *** jamesmcarthur has joined #openstack-infra | 06:21 | |
| *** lpetrut has joined #openstack-infra | 06:22 | |
| *** graphene has quit IRC | 06:22 | |
| *** graphene has joined #openstack-infra | 06:24 | |
| *** aeng_ has quit IRC | 06:26 | |
| AJaeger | prometheanfire, abhishekk, reading backscroll: We run out of space with our pypi mirror and it looks like that issue is not solved yet - so the tree is currently frozen. | 06:26 |
| *** jamesmcarthur has quit IRC | 06:26 | |
| *** ykarel has joined #openstack-infra | 06:27 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 06:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 06:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 06:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: scheduler: add job's parent name to the rpc job_list method https://review.openstack.org/573473 | 06:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/labels route https://review.openstack.org/553979 | 06:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/nodes route https://review.openstack.org/553998 | 06:27 |
| abhishekk | AJaeger, any idea when things will be back to normal? | 06:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 06:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page https://review.openstack.org/537870 | 06:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/labels.html web page https://review.openstack.org/553980 | 06:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/nodes.html web page https://review.openstack.org/553999 | 06:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add jobs graph rendering https://review.openstack.org/537869 | 06:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add project pipeline rendering https://review.openstack.org/537871 | 06:28 |
| *** kjackal has joined #openstack-infra | 06:29 | |
| *** tristanC has joined #openstack-infra | 06:30 | |
| *** pcaruana has joined #openstack-infra | 06:35 | |
| *** alexchadin has joined #openstack-infra | 06:36 | |
| *** aojea has joined #openstack-infra | 06:37 | |
| prometheanfire | AJaeger: k, can you ping me when things change so I can re-kick reqs jobs? | 06:39 |
| AJaeger | abhishekk, prometheanfire, I'm not involved - best to ask the US infra-root team later today... | 06:45 |
| *** dpawlik has joined #openstack-infra | 06:49 | |
| *** florianf has joined #openstack-infra | 06:50 | |
| *** slaweq has joined #openstack-infra | 06:54 | |
| *** tobiash has joined #openstack-infra | 06:55 | |
| prometheanfire | oh booo :P | 06:59 |
| *** pguimaraes has joined #openstack-infra | 07:00 | |
| *** jamesmcarthur has joined #openstack-infra | 07:02 | |
| *** jamesmcarthur has quit IRC | 07:08 | |
| *** ccamacho has joined #openstack-infra | 07:08 | |
| *** peereb has joined #openstack-infra | 07:09 | |
| *** gfidente has joined #openstack-infra | 07:12 | |
| *** gfidente has joined #openstack-infra | 07:12 | |
| *** shardy has joined #openstack-infra | 07:12 | |
| *** tesseract has joined #openstack-infra | 07:18 | |
| *** ykarel is now known as ykarel|lunch | 07:19 | |
| *** jamesmcarthur has joined #openstack-infra | 07:23 | |
| *** jamesmcarthur has quit IRC | 07:28 | |
| *** rcernin has quit IRC | 07:28 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix template variables https://review.openstack.org/581993 | 07:32 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 07:32 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 07:33 |
| *** dklyle has quit IRC | 07:35 | |
| *** janki has joined #openstack-infra | 07:37 | |
| *** dbecker has joined #openstack-infra | 07:41 | |
| *** zoli|gone is now known as zoli | 07:44 | |
| *** jamesmcarthur has joined #openstack-infra | 07:44 | |
| *** jamesmcarthur has quit IRC | 07:48 | |
| *** jtomasek has joined #openstack-infra | 07:50 | |
| *** Bhujay has quit IRC | 07:52 | |
| *** armax has joined #openstack-infra | 07:54 | |
| *** tobasco_away is now known as tobasco | 07:54 | |
| *** armax has quit IRC | 07:58 | |
| *** tosky has joined #openstack-infra | 08:06 | |
| *** markvoelker_ has quit IRC | 08:07 | |
| *** e0ne has joined #openstack-infra | 08:08 | |
| *** elod has quit IRC | 08:13 | |
| *** jpich has joined #openstack-infra | 08:13 | |
| *** hashar has joined #openstack-infra | 08:14 | |
| *** ykarel|lunch is now known as ykarel | 08:15 | |
| *** lucasagomes_ is now known as lucasagomes | 08:15 | |
| *** jamesmcarthur has joined #openstack-infra | 08:25 | |
| *** elod has joined #openstack-infra | 08:30 | |
| *** jamesmcarthur has quit IRC | 08:30 | |
| *** electrofelix has joined #openstack-infra | 08:33 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| *** Bhujay has joined #openstack-infra | 08:39 | |
| *** jamesmcarthur has joined #openstack-infra | 08:47 | |
| *** jamesmcarthur has quit IRC | 08:50 | |
| *** threestrands has quit IRC | 08:51 | |
| *** pbourke has quit IRC | 08:52 | |
| *** pbourke has joined #openstack-infra | 08:52 | |
| *** vabada_ has quit IRC | 08:54 | |
| *** vabada has joined #openstack-infra | 08:54 | |
| *** graphene has quit IRC | 08:56 | |
| *** graphene1 has joined #openstack-infra | 08:58 | |
| *** graphene1 has quit IRC | 08:58 | |
| *** graphene has joined #openstack-infra | 09:00 | |
| *** dingyichen has quit IRC | 09:06 | |
| *** owalsh is now known as owalsh_afk | 09:07 | |
| *** gongysh has joined #openstack-infra | 09:08 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Increase integration pip timeout https://review.openstack.org/583178 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Remove 'tools/releasenotes_tox.sh' https://review.openstack.org/580414 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Fix file permissions https://review.openstack.org/580415 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Switch to stestr https://review.openstack.org/580413 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: builddoc: Remove support for Sphinx < 1.6 https://review.openstack.org/580416 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: packaging: Remove support for pyN requirement files https://review.openstack.org/580417 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Fix formatting of packagers guide https://review.openstack.org/580418 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Add documentation for missing '[pbr]' options https://review.openstack.org/580419 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Add Sphinx extension https://review.openstack.org/580420 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'build_sphinx' integration https://review.openstack.org/580421 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'test' integration https://review.openstack.org/580422 | 09:10 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Remove 'tools/releasenotes_tox.sh' https://review.openstack.org/580414 | 09:15 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: trivial: Fix file permissions https://review.openstack.org/580415 | 09:15 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Switch to stestr https://review.openstack.org/580413 | 09:15 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: builddoc: Remove support for Sphinx < 1.6 https://review.openstack.org/580416 | 09:15 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: packaging: Remove support for pyN requirement files https://review.openstack.org/580417 | 09:15 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Fix formatting of packagers guide https://review.openstack.org/580418 | 09:15 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Add documentation for missing '[pbr]' options https://review.openstack.org/580419 | 09:15 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Add Sphinx extension https://review.openstack.org/580420 | 09:15 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'build_sphinx' integration https://review.openstack.org/580421 | 09:15 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'test' integration https://review.openstack.org/580422 | 09:15 |
| *** dtantsur|afk is now known as dtantsur | 09:20 | |
| *** jamesmcarthur has joined #openstack-infra | 09:27 | |
| *** roman_g has joined #openstack-infra | 09:28 | |
| *** jchhatbar has joined #openstack-infra | 09:29 | |
| *** sambetts_ is now known as sambetts | 09:30 | |
| *** janki has quit IRC | 09:31 | |
| *** jamesmcarthur has quit IRC | 09:32 | |
| *** zoli is now known as zoli|lunch | 09:38 | |
| *** pguimaraes has quit IRC | 09:38 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2 https://review.openstack.org/583535 | 09:40 |
| *** dhill_ has quit IRC | 09:47 | |
| *** jamesmcarthur has joined #openstack-infra | 09:49 | |
| *** jamesmcarthur has quit IRC | 09:53 | |
| *** armax has joined #openstack-infra | 09:55 | |
| *** shu-mutow has quit IRC | 09:55 | |
| *** armax has quit IRC | 09:59 | |
| *** abhishekk has quit IRC | 10:03 | |
| *** markvoelker has joined #openstack-infra | 10:08 | |
| *** alexchadin has quit IRC | 10:08 | |
| *** jamesmcarthur has joined #openstack-infra | 10:09 | |
| *** kukacz_ has quit IRC | 10:12 | |
| *** kukacz_ has joined #openstack-infra | 10:13 | |
| *** ykarel is now known as ykarel|afk | 10:13 | |
| *** jamesmcarthur has quit IRC | 10:13 | |
| *** Bhujay has quit IRC | 10:16 | |
| chkumar|ruck | Hello infra | 10:18 |
| chkumar|ruck | On tripleo jobs we are hitting "msg": "SSH Error: data could not be sent to remote host \"104.239.168.222\". Make sure this host can be reached over ssh", | 10:18 |
| chkumar|ruck | 2018-07-18 07:49:30.459805 | primary | "unreachable": true | 10:18 |
| chkumar|ruck | http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/6232ced/job-output.txt.gz#_2018-07-18_07_49_30_459692 | 10:18 |
| chkumar|ruck | please have a look | 10:18 |
| chkumar|ruck | we have other jobs with same failure | 10:19 |
| chkumar|ruck | http://logs.openstack.org/23/582823/4/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/522bae9/job-output.txt.gz#_2018-07-18_07_29_39_888159 | 10:19 |
| *** alexchadin has joined #openstack-infra | 10:22 | |
| *** jamesmcarthur has joined #openstack-infra | 10:30 | |
| *** greghaynes has quit IRC | 10:32 | |
| *** greghaynes has joined #openstack-infra | 10:33 | |
| *** stevebaker has quit IRC | 10:35 | |
| *** jamesmcarthur has quit IRC | 10:35 | |
| *** dmellado has quit IRC | 10:36 | |
| *** dpawlik has quit IRC | 10:39 | |
| *** Bhujay has joined #openstack-infra | 10:39 | |
| *** gongysh has quit IRC | 10:39 | |
| *** zoli|lunch is now known as zoli | 10:40 | |
| *** dpawlik has joined #openstack-infra | 10:41 | |
| quiquell | Hello | 10:41 |
| quiquell | What's the best way to check if a job is periodic at zuulv3 ? | 10:41 |
| quiquell | in the playbook | 10:41 |
| *** stevebaker has joined #openstack-infra | 10:41 | |
| quiquell | We can check the pipeline name | 10:41 |
| quiquell | any other alternative ? | 10:42 |
| *** markvoelker has quit IRC | 10:42 | |
| *** priteau has joined #openstack-infra | 10:42 | |
| *** dmellado has joined #openstack-infra | 10:42 | |
| *** gouthamr has quit IRC | 10:47 | |
| *** kjackal has quit IRC | 10:50 | |
| *** jamesmcarthur has joined #openstack-infra | 10:51 | |
| *** elod has quit IRC | 10:52 | |
| *** pguimaraes has joined #openstack-infra | 10:52 | |
| *** jcoufal has joined #openstack-infra | 10:52 | |
| *** elod has joined #openstack-infra | 10:54 | |
| *** jamesmcarthur has quit IRC | 10:55 | |
| *** dpawlik has quit IRC | 10:55 | |
| *** pcichy has quit IRC | 10:55 | |
| *** dpawlik has joined #openstack-infra | 10:56 | |
| *** pcichy has joined #openstack-infra | 10:56 | |
| *** elod has quit IRC | 10:59 | |
| *** zhangfei has quit IRC | 10:59 | |
| *** gongysh has joined #openstack-infra | 10:59 | |
| *** wolverineav has joined #openstack-infra | 10:59 | |
| *** zhangfei has joined #openstack-infra | 10:59 | |
| *** annp has quit IRC | 11:00 | |
| *** zhangfei has quit IRC | 11:01 | |
| *** gouthamr has joined #openstack-infra | 11:01 | |
| *** jamesmcarthur has joined #openstack-infra | 11:11 | |
| *** jamesmcarthur has quit IRC | 11:16 | |
| *** elod has joined #openstack-infra | 11:20 | |
| *** gongysh has quit IRC | 11:22 | |
| *** kjackal has joined #openstack-infra | 11:23 | |
| *** ykarel|afk is now known as ykarel | 11:24 | |
| *** tpsilva has joined #openstack-infra | 11:31 | |
| *** markvoelker has joined #openstack-infra | 11:39 | |
| *** ldnunes has joined #openstack-infra | 11:44 | |
| *** boden has joined #openstack-infra | 11:49 | |
| *** rh-jelabarre has joined #openstack-infra | 11:51 | |
| *** pcichy has quit IRC | 11:52 | |
| *** jamesmcarthur has joined #openstack-infra | 11:52 | |
| *** Bhujay has quit IRC | 11:54 | |
| *** armax has joined #openstack-infra | 11:55 | |
| *** jamesmcarthur has quit IRC | 11:57 | |
| *** dhill_ has joined #openstack-infra | 11:58 | |
| *** trown|brb is now known as trown | 12:00 | |
| *** armax has quit IRC | 12:00 | |
| *** markvoelker has quit IRC | 12:05 | |
| *** markvoelker has joined #openstack-infra | 12:05 | |
| *** hashar is now known as hasharAway | 12:10 | |
| *** camunoz has joined #openstack-infra | 12:11 | |
| *** edmondsw has joined #openstack-infra | 12:11 | |
| *** jamesmcarthur has joined #openstack-infra | 12:14 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to zuul client https://review.openstack.org/574265 | 12:17 |
| *** jamesmcarthur has quit IRC | 12:18 | |
| *** camunoz has quit IRC | 12:19 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to zuul client https://review.openstack.org/574265 | 12:19 |
| mordred | quiquell: depends on what you want to do - you could also check to see if zuul.change is defined or not | 12:19 |
| mordred | quiquell: http://logs.openstack.org/periodic/git.openstack.org/openstack/kolla/master/kolla-publish-ubuntu-binary/d6264a7/zuul-info/inventory.yaml is an inventory file from a periodic job | 12:19 |
| *** camunoz has joined #openstack-infra | 12:19 | |
| *** owalsh_afk is now known as owalsh | 12:23 | |
| *** kaiokmo has quit IRC | 12:25 | |
| *** dpawlik has quit IRC | 12:30 | |
| mordred | remote: https://review.openstack.org/583318 DNM Test that new mirror fixes lifes problems | 12:34 |
| *** kaiokmo has joined #openstack-infra | 12:34 | |
| mordred | remote: https://review.openstack.org/583561 DNM testing pypi mirror fix | 12:34 |
| *** rlandy has joined #openstack-infra | 12:35 | |
| *** elod has quit IRC | 12:35 | |
| *** rosmaita has joined #openstack-infra | 12:36 | |
| *** elod has joined #openstack-infra | 12:38 | |
| mordred | AJaeger, prometheanfire: ^^ if those work ok, then I think we shoudl be good to land remote: https://review.openstack.org/583561 DNM testing pypi mirror fix | 12:38 |
| mordred | gah | 12:38 |
| mordred | AJaeger, prometheanfire: ^^ if those work ok, then I think we shoudl be good to land https://review.openstack.org/583328 | 12:38 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Remove unecessary shebang and exec bit https://review.openstack.org/583563 | 12:39 |
| *** mriedem has joined #openstack-infra | 12:39 | |
| *** quiquell is now known as quiquell|lunch | 12:41 | |
| *** kaiokmo has quit IRC | 12:41 | |
| *** agopi|off has quit IRC | 12:42 | |
| *** kaiokmo has joined #openstack-infra | 12:42 | |
| *** kaiokmo has quit IRC | 12:45 | |
| *** kaiokmo has joined #openstack-infra | 12:45 | |
| *** slaweq has quit IRC | 12:46 | |
| *** pcichy has joined #openstack-infra | 12:46 | |
| AJaeger | thanks, mordred ! | 12:46 |
| *** dpawlik has joined #openstack-infra | 12:47 | |
| *** quiquell|lunch is now known as quiquell | 12:48 | |
| *** ykarel is now known as ykarel|away | 12:49 | |
| *** camunoz has quit IRC | 12:49 | |
| *** dpawlik has quit IRC | 12:52 | |
| *** dhajare has quit IRC | 12:54 | |
| *** jamesmcarthur has joined #openstack-infra | 12:55 | |
| *** slaweq has joined #openstack-infra | 12:55 | |
| *** jamesmcarthur has quit IRC | 13:03 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Remove redhat-rpm-config bindep https://review.openstack.org/581252 | 13:05 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Fix github setup in zuul from scratch https://review.openstack.org/581253 | 13:05 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: fix zuul from scratch user and group creation https://review.openstack.org/581254 | 13:05 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Add instructions for deploying zuul with openSUSE https://review.openstack.org/581255 | 13:05 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Add instructions on building static web components https://review.openstack.org/581256 | 13:05 |
| mordred | ZOMG. easy_install is the absolute worst | 13:05 |
| *** agopi|off has joined #openstack-infra | 13:10 | |
| *** agopi|off is now known as agopi | 13:10 | |
| *** ykarel|away has quit IRC | 13:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Serve pypi proxy cache from port 80 https://review.openstack.org/583571 | 13:15 |
| * mordred goes to get more coffee | 13:15 | |
| *** graphene has quit IRC | 13:18 | |
| *** lbragstad has quit IRC | 13:19 | |
| *** graphene has joined #openstack-infra | 13:20 | |
| *** quiquell is now known as quiquell|launch | 13:23 | |
| *** quiquell|launch is now known as quiquell|lunch | 13:23 | |
| *** eharney has joined #openstack-infra | 13:26 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix plugin and examples for using openstacksdk https://review.openstack.org/583338 | 13:30 |
| *** gcb has quit IRC | 13:31 | |
| *** graphene has quit IRC | 13:31 | |
| *** graphene has joined #openstack-infra | 13:33 | |
| *** alexchadin has quit IRC | 13:34 | |
| *** alexchadin has joined #openstack-infra | 13:34 | |
| *** alexchadin has quit IRC | 13:35 | |
| *** alexchadin has joined #openstack-infra | 13:35 | |
| *** bobh has joined #openstack-infra | 13:35 | |
| *** myoung|off is now known as myoung | 13:36 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to zuul client https://review.openstack.org/574265 | 13:36 |
| *** bobh has quit IRC | 13:40 | |
| *** pcichy has quit IRC | 13:40 | |
| *** armax has joined #openstack-infra | 13:43 | |
| *** pcichy has joined #openstack-infra | 13:44 | |
| *** pcichy has quit IRC | 13:45 | |
| *** pcichy has joined #openstack-infra | 13:45 | |
| *** pcichy has quit IRC | 13:45 | |
| *** pcichy has joined #openstack-infra | 13:46 | |
| *** lbragstad has joined #openstack-infra | 13:46 | |
| openstackgerrit | Merged openstack-infra/git-review master: tox: default envlist to supported python versions https://review.openstack.org/583243 | 13:47 |
| *** armax has quit IRC | 13:47 | |
| *** sthussey has joined #openstack-infra | 13:47 | |
| *** r-daneel has joined #openstack-infra | 13:48 | |
| *** dklyle has joined #openstack-infra | 13:50 | |
| pabelanger | mordred: clarkb: corvus: Looking more into the SSH error above from chkumar|ruck, I am seeing a lot of the following in the scheduler debug logs: http://paste.openstack.org/show/726198/ | 13:51 |
| *** gongysh has joined #openstack-infra | 13:51 | |
| *** dpawlik has joined #openstack-infra | 13:51 | |
| prometheanfire | mordred: thanks | 13:51 |
| pabelanger | looking at job to see if anything recently changed | 13:51 |
| mordred | prometheanfire: sorry it's taken so long to fix this - global pypi mirror patches are ... fun | 13:51 |
| *** graphene has quit IRC | 13:53 | |
| *** quiquell|lunch is now known as quiquell | 13:54 | |
| slaweq | hi infra folks :) | 13:54 |
| slaweq | I'm trying to migrate neutron multinode job to zuulv3 syntax, patch is here: https://review.openstack.org/#/c/578796/ | 13:55 |
| *** graphene has joined #openstack-infra | 13:55 | |
| slaweq | but problem is that it looks for me that on compute1 node there is no any openstack service started: http://logs.openstack.org/96/578796/3/check/neutron-tempest-plugin-dvr-multinode-scenario-zuulv3/464aefa/compute1/logs/ | 13:55 |
| slaweq | can You maybe take a look and help me to figure out what's wrong there? | 13:56 |
| slaweq | thx in advance | 13:56 |
| mordred | andreaf: ^^ you might be a good person for that question | 13:56 |
| *** bobh has joined #openstack-infra | 13:57 | |
| *** dklyle has quit IRC | 14:00 | |
| pabelanger | 2018-07-18 06:18:03,688 WARNING kazoo.client: Connection dropped: socket connection broken | 14:00 |
| pabelanger | so zuul lost its connect to zookeeper | 14:00 |
| *** cshastri has quit IRC | 14:01 | |
| *** bobh has quit IRC | 14:01 | |
| pabelanger | which caused nodepool to delete all the non-ready nodes: http://grafana.openstack.org/dashboard/db/nodepool?from=1531893455241&to=1531895660646 | 14:01 |
| pabelanger | that seems to be when we start getting kazoo.exceptions.NoNodeError for pastebin above^ | 14:02 |
| pabelanger | perhaps we zuul didn't properly recover from connection loss with zookeeper? | 14:02 |
| pabelanger | I'm unsure how to best fix, since we cannot dequeue a periodic job yet, suspect we'd need to restart the scheduler. | 14:03 |
| *** caphrim007_ has quit IRC | 14:04 | |
| *** caphrim007 has joined #openstack-infra | 14:04 | |
| *** jamesmcarthur has joined #openstack-infra | 14:07 | |
| *** armax has joined #openstack-infra | 14:08 | |
| *** quiquell is now known as quiquell|off | 14:12 | |
| *** eernst has joined #openstack-infra | 14:14 | |
| mordred | pabelanger: corvus should be up soon ... I'm also not sure the right action to take - or if there is some debugging info that we'd wantto collect | 14:14 |
| * corvus is rubbing his eyes | 14:15 | |
| *** hasharAway is now known as hashar | 14:15 | |
| *** zhangfei has joined #openstack-infra | 14:15 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Fix formatting of packagers guide https://review.openstack.org/580418 | 14:16 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Add documentation for missing '[pbr]' options https://review.openstack.org/580419 | 14:16 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Add Sphinx extension https://review.openstack.org/580420 | 14:16 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'build_sphinx' integration https://review.openstack.org/580421 | 14:16 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'test' integration https://review.openstack.org/580422 | 14:16 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: builddoc: Remove support for Sphinx < 1.6 https://review.openstack.org/580416 | 14:16 |
| corvus | mordred, pabelanger: i'd expect to see those errors if the zookeeper connection reset | 14:16 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: packaging: Remove support for pyN requirement files https://review.openstack.org/580417 | 14:16 |
| *** Bhujay has joined #openstack-infra | 14:17 | |
| pabelanger | corvus: mordred: k, I'm curious if they are causing periodic jobs not to run, we've had some enqueued for last 8+ hours: https://zuul.openstack.org/ | 14:18 |
| * mordred hands corvus something that isn't coffee but sits in the same place in the typical corvus morning | 14:19 | |
| auristor | as a temporary fix for the mirror.pypi reaching max quota, remove the volume replica from afs02 and then remove the quota by setting the quota to 0 (unlimited). This will permit the amount of data written to the volume to exceed 2TB with the following side effects. One, the space used counter will internally wrap negative and once the amount of data exceeds 4TB it will wrap positive again. | 14:22 |
| auristor | Two, tooling that relies upon the space used counter such as backups or replication or clients that query RXAFS_GetVolumeStatus can behave unpredictably. | 14:22 |
| *** caphrim007_ has joined #openstack-infra | 14:24 | |
| *** Bhujay has quit IRC | 14:25 | |
| *** caphrim007 has quit IRC | 14:25 | |
| *** udesale_ has joined #openstack-infra | 14:26 | |
| *** udesale has quit IRC | 14:27 | |
| corvus | pabelanger: yeah, i think there's a bug that can cause stuck queue items after a zk failure | 14:27 |
| mordred | auristor: thanks! I think we were leaning towards shifting to the caching-proxy setup for the pypi code anyway though - we don't need vos release as much for it, and the ratio of needed to cached packages has shifted rather significantly in the other direction now that people are uploading giant AI/ML packages - so even without the volume limit it's an increasing amount of space we'd need to account | 14:27 |
| mordred | for that we'll never make use of | 14:27 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Add Sphinx extension https://review.openstack.org/580420 | 14:29 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'build_sphinx' integration https://review.openstack.org/580421 | 14:29 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Deprecate 'test' integration https://review.openstack.org/580422 | 14:29 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: packaging: Remove support for pyN requirement files https://review.openstack.org/580417 | 14:29 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: builddoc: Remove support for Sphinx < 1.6 https://review.openstack.org/580416 | 14:29 |
| pabelanger | corvus: ack | 14:29 |
| corvus | pabelanger, mordred: there's a gap in zuul's logs from 06:16:49 to 06:18:02. i suspect something cloudy happened. | 14:30 |
| auristor | mordred: sure, I'm proposing a workaround until the caching-proxy is in place | 14:31 |
| mordred | auristor: awesome. thanks! | 14:31 |
| chkumar|ruck | sshnaidm|rover: weshay ^^ about ssh issue | 14:32 |
| auristor | I don't consider going beyond 2TB in an OpenAFS volume safe as a long term strategy. With AuriStorFS it would be fine. We have customers that regularly replicate volumes larger than 50TB. | 14:32 |
| corvus | pabelanger, mordred: oh i see what happened -- it swapped. | 14:33 |
| mordred | corvus: oh yay | 14:33 |
| corvus | we seem to have introduced a memory leak...two weeks ago? | 14:33 |
| pabelanger | oh, I didn't check cacti, but did notice last night scheduler was close to OOM | 14:33 |
| pabelanger | yah, I think we might have a new leak | 14:34 |
| mordred | corvus: so - around the 4th? | 14:34 |
| *** alexchadin has quit IRC | 14:35 | |
| *** jangutter has joined #openstack-infra | 14:35 | |
| corvus | yeah... july 5 +/- 1 day | 14:35 |
| corvus | welp. apparently we've been running since july 3 so nevermind my error bars | 14:35 |
| mordred | corvus: "Fix source context in split config" ed792c9a2ec660b58bd5b3f04604a42d5bdcd82b might be worth looking at as a culprit (config context related) | 14:35 |
| mordred | oh - nevermind | 14:36 |
| corvus | i wonder if we recorded the version we ran before the july 3 restart? | 14:36 |
| mordred | corvus: we should maybe do that | 14:36 |
| corvus | 2018-07-03T18:06:20 <corvus> it turns out we restarted the zuul scheduler at a really inopportune commit. it's *after* the commit which allows for loading configuration with errors, but *before* the commit which exposes them through the api so you can actually see them. | 14:38 |
| corvus | that's a clue :) | 14:38 |
| corvus | (that's referring to the previous restart -- that prompted the restart on the 3rd) | 14:39 |
| mordred | ++ | 14:40 |
| *** alexchadin has joined #openstack-infra | 14:41 | |
| corvus | 2018-06-14T23:12:54 | 14:41 |
| corvus | that looks to be the previous restart time | 14:41 |
| mordred | so there was a good 2 weeks of running on that commit | 14:41 |
| mordred | whatever it was | 14:41 |
| corvus | though the memory graph has a suspicious dip around 6/26/ to 6/27.... | 14:42 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed SS base model serialization https://review.openstack.org/583635 | 14:43 |
| corvus | ah yep, restarted scheduler at 2018-06-26T21:39:52 | 14:45 |
| corvus | so not a huge amount of time on that commit, but the memory use still looked pretty good during that time | 14:46 |
| corvus | i think that was the restart to pick up the supercedent pipeline manager | 14:46 |
| mordred | ah - yes | 14:46 |
| mordred | corvus: maybe we should emit a log line on startup with version/git-sha in it? | 14:47 |
| corvus | mordred: ++ | 14:47 |
| *** lpetrut has quit IRC | 14:47 | |
| pabelanger | ++ | 14:49 |
| * mordred writes patch for that so he can be somewhat useful | 14:49 | |
| corvus | okay, i think we probably were running af575001b422119348812439bf0e7626e085860d starting on 6-26 | 14:51 |
| *** peereb has quit IRC | 14:51 | |
| corvus | and we're currently running 4d2d915 | 14:51 |
| corvus | oh, that may be a custom local revision with the repl | 14:52 |
| corvus | on the plus side, erm, i guess we have the repl if we want to try to interactively debug the leak. | 14:53 |
| corvus | so, aside from the repl, we're running c2fe7713feac11cecaa56c3327a3dff13a78058e | 14:53 |
| *** pcaruana has quit IRC | 14:56 | |
| *** zhangfei has quit IRC | 14:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Emit a log line containing version info around startup https://review.openstack.org/583639 | 14:59 |
| *** pcichy has quit IRC | 14:59 | |
| corvus | there's a lot to go through... | 15:02 |
| mordred | corvus: yeah. I guess we're active | 15:04 |
| *** kjackal has quit IRC | 15:06 | |
| *** hongbin has joined #openstack-infra | 15:06 | |
| *** elod has quit IRC | 15:07 | |
| chkumar|ruck | pabelanger: recent ssh error http://logs.openstack.org/02/583202/2/check/tripleo-ci-centos-7-containers-multinode-pike/8c16e0b/job-output.txt.gz#_2018-07-18_14_51_28_499366 | 15:08 |
| clarkb | mordred: not to distract from zuul debugging, but have we been able to test the pypi proxy? | 15:08 |
| mordred | clarkb: morning! | 15:08 |
| mordred | clarkb: so ... are you awake and ready to be ragey? | 15:09 |
| mordred | or would you like time to coffee more | 15:09 |
| *** elod has joined #openstack-infra | 15:09 | |
| mordred | clarkb: https://review.openstack.org/#/c/583571/ has a summary of what I have learned this morning in the commit message | 15:10 |
| clarkb | mordred: we should just get it out of the way :P | 15:10 |
| pabelanger | yah, looks like scheduler isn't happy with zookeeper | 15:10 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool?from=1531923026120&to=1531926626120 | 15:10 |
| *** alexchadin has quit IRC | 15:10 | |
| pabelanger | mordred: corvus: I am guessing because of swapping now ^ | 15:10 |
| pabelanger | looks like nodepool deleted all nodes twice in the last hour | 15:11 |
| *** gongysh has quit IRC | 15:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Serve pypi proxy cache from port 80 https://review.openstack.org/583571 | 15:13 |
| pabelanger | yah, http://paste.openstack.org/show/726205/ | 15:13 |
| mordred | clarkb: ^^ updated - it was incomplete | 15:13 |
| clarkb | mordred: have we managed to test it in a way that doesn't run into easy_install problems? I guess just running pip locally? | 15:14 |
| mordred | clarkb: yeah - that's the only way I've been able to test | 15:15 |
| mordred | clarkb: I was able to reproduce the easy_install issue locally by putting the hostname without the port into allowed-hoss in ~/.pydistutils.cfg and doing "pip install reno" in a virtualenv | 15:15 |
| clarkb | mordred: I think we need to keep the /wheel/ rewrite rules | 15:15 |
| mordred | clarkb: and then I added the port and tried again and it worked | 15:15 |
| mordred | clarkb: oh - so we do | 15:16 |
| clarkb | mordred: and we need to update those other rules to apply only to wheel :/ | 15:16 |
| *** lpetrut has joined #openstack-infra | 15:16 | |
| clarkb | # Try again but replacing -'s with .'s that section would apply to /wheel/ too | 15:16 |
| clarkb | as does # Special cases for openstack.nose_plugin & backports.* | 15:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Serve pypi proxy cache from port 80 https://review.openstack.org/583571 | 15:17 |
| clarkb | everything else appears rooted at ^/pypi | 15:17 |
| mordred | kk | 15:17 |
| openstackgerrit | Jan Gutter proposed openstack-infra/nodepool master: [POC] Add baremetal subimage upload support https://review.openstack.org/583645 | 15:17 |
| corvus | there are 533 leaked layout objects | 15:18 |
| jangutter | pabelanger: some eye-bleeding hacks in that proposal (the baremetal triple image support) | 15:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Serve pypi proxy cache from port 80 https://review.openstack.org/583571 | 15:19 |
| mordred | clarkb: how's that look? ^^ | 15:19 |
| clarkb | mordred: I wonder if we need to rewritecond scope them to ^/wheel/ and not apply them to /pypi anymore | 15:20 |
| *** rpioso|afk is now known as rpioso | 15:20 | |
| mordred | clarkb: I think probably so - I was just having the same thought - they were there before since we were apacheing vs. the actual pypi code | 15:20 |
| mordred | but as a passthrough- we should be able to just let pypi handle it | 15:20 |
| clarkb | mordred: ya but I think we may do the rewrite before passing to pypi, but pypi should handle both the pre rewrite and post rewrite so it is probably fine | 15:21 |
| *** graphene has quit IRC | 15:21 | |
| clarkb | it is going to sort out _ vs . if we use one or the other | 15:22 |
| mordred | clarkb: actually - yeah - I think leaving it as is for now is likely fine - like RewriteRule (.*)-(.*) $1.$2 [N] depends on the file not being found on disk, which will never apply to /pypi things happening via proxy | 15:22 |
| clarkb | mordred: except that we have a /pypi in the doc root | 15:22 |
| mordred | oh - right - so it still might apply | 15:23 |
| *** graphene has joined #openstack-infra | 15:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Serve pypi proxy cache from port 80 https://review.openstack.org/583571 | 15:23 |
| mordred | clarkb: maybe like that? | 15:23 |
| pabelanger | we might need to consider stopping executors while corvus debugs the zuul scheduler, nodepool just deleted all nodes again (3rd time in last hour) | 15:24 |
| mordred | pabelanger: well, _some_ things are getting through - so I think leaving it going isn't super terrible | 15:26 |
| *** wolverineav has quit IRC | 15:26 | |
| *** wolverineav has joined #openstack-infra | 15:27 | |
| pabelanger | mordred: yah, mostly affecting long running jobs | 15:28 |
| mordred | clarkb: I've gotta jump on a call - thoughts on moving forward? | 15:33 |
| clarkb | mordred: sorry I'm trying to wrap my head around all that regexing. I think we can drop the -d checks 'RewriteCond %{DOCUMENT_ROOT}/wheel/$1/$2/$2$3 -d' | 15:34 |
| clarkb | mordred: and drop the grouping in the regex. Just check if it is ^/wheel/ | 15:34 |
| clarkb | mordred: specifically the -d check is at odds with the subsequent ! -d check I think | 15:34 |
| clarkb | mordred: posted comments to make it easier to understand | 15:36 |
| *** jamesmcarthur has quit IRC | 15:37 | |
| clarkb | and in the -d vs ! -d case I think it is a correctness issue | 15:37 |
| *** lpetrut has quit IRC | 15:37 | |
| *** pguimaraes has quit IRC | 15:38 | |
| *** pguimaraes has joined #openstack-infra | 15:39 | |
| *** aojea has quit IRC | 15:39 | |
| clarkb | pabelanger: considering it took a while for the leak to become a problem, another option (if it doesn't interfere with debugging) may be to simply restart the scheduler | 15:40 |
| clarkb | then we get another ~2 weeks before it crashes | 15:40 |
| corvus | pabelanger, clarkb: okay, i used the repl to grab some debugging info | 15:40 |
| corvus | i probably made the problem a little worse doing it, but hopefully something will show | 15:40 |
| corvus | if you want to restart the scheduler now, that wfm | 15:41 |
| *** graphene has quit IRC | 15:41 | |
| *** diablo_rojo has joined #openstack-infra | 15:41 | |
| *** graphene has joined #openstack-infra | 15:42 | |
| *** jamesmcarthur has joined #openstack-infra | 15:43 | |
| clarkb | I will save queues now and restart zuul | 15:43 |
| pabelanger | +1 | 15:43 |
| clarkb | er restart zuul-scheduler | 15:43 |
| *** neiloy has quit IRC | 15:44 | |
| clarkb | I have asked it to stop nicely | 15:45 |
| clarkb | now we wait and see if it is willing to go along with that | 15:45 |
| clarkb | corvus: 2018-07-18 15:45:14,340 DEBUG zuul.Scheduler: Run handler stopping is the last thing logged but process is still running | 15:47 |
| clarkb | should I continue to wait for it? | 15:47 |
| clarkb | oh it went away, starting now | 15:47 |
| corvus | clarkb: can you restart the executors too? | 15:48 |
| corvus | (you should still have a few minutes to take them offline before they pick up new work) | 15:49 |
| clarkb | corvus: I can, | 15:49 |
| clarkb | I have requested they all stop. Just waiting for that to happen before starting them again | 15:51 |
| *** jamesmcarthur has quit IRC | 15:51 | |
| *** jamesmcarthur has joined #openstack-infra | 15:51 | |
| clarkb | then I'll reenqueue changes once scheduler and executrs are all back and running | 15:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Serve pypi proxy cache from port 80 https://review.openstack.org/583571 | 15:55 |
| *** lpetrut has joined #openstack-infra | 15:55 | |
| mordred | clarkb: ^^ how abou tnow? | 15:55 |
| *** jamesmcarthur has quit IRC | 15:55 | |
| *** links has quit IRC | 15:55 | |
| *** neiloy has joined #openstack-infra | 15:56 | |
| clarkb | mordred: I'll look as soon as I have zuul running again | 15:58 |
| clarkb | executors still not stopped, but process counts falling on ze01 so expect it to be done soon | 15:58 |
| *** pcaruana has joined #openstack-infra | 16:00 | |
| *** lbragstad has quit IRC | 16:02 | |
| *** gfidente is now known as gfidente|dk | 16:02 | |
| *** gfidente|dk is now known as gfidente|afk | 16:02 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Gate governance with project-config https://review.openstack.org/583654 | 16:03 |
| clarkb | executors are starting now | 16:03 |
| clarkb | corvus: zuul web status doesn't look happy should I restart that too? | 16:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Delete executor.job_dir on start up https://review.openstack.org/583655 | 16:05 |
| *** zoli is now known as zoli|PTO | 16:06 | |
| *** zoli|PTO is now known as zoli | 16:06 | |
| clarkb | I'm going to restart zuul web | 16:06 |
| corvus | oh yeah | 16:06 |
| corvus | i think a bunch of stuff changed :) | 16:07 |
| clarkb | ok looks good after that | 16:07 |
| clarkb | enqueuing changes now | 16:07 |
| corvus | may need a refresh if it looks funny for anyone | 16:08 |
| pabelanger | cool, new HDD governor is working: http://grafana.openstack.org/dashboard/db/zuul-status well reporting stats | 16:08 |
| *** sshnaidm|rover is now known as sshnaidm|bbl | 16:12 | |
| *** efried is now known as efried_rollin | 16:13 | |
| *** udesale_ has quit IRC | 16:13 | |
| *** udesale has joined #openstack-infra | 16:13 | |
| mnaser | corvus: a very late response but depending on the amount of data, it should be totally okay | 16:13 |
| mnaser | it would just be nice to know what sort of numbers we expect | 16:14 |
| mordred | mnaser: tl;dr we're working on plans to move build logs off of the logs.openstack.org server and into swift - probably one swift to start, but eventually hopefully spread across the swifts of our clouds | 16:14 |
| corvus | mnaser: thanks! clarkb has also suggested knowing numbers would be good, so i think that's next on the agenda :) | 16:14 |
| *** shah^ has joined #openstack-infra | 16:15 | |
| mnaser | mordred: yeah i've been following the ML and reading what's going on (which i totally support and think now is a great time in terms of how zuul allows us to do things) | 16:15 |
| mnaser | corvus: cool, once we have a rough idea it'll be good to know (and i guess a rough idea of bw usage too) | 16:16 |
| clarkb | mordred: bah I realize I left my comments off by one. The first set of long regex and -d check was preexisting and should stay. Its the subsequent ones that needed my cleanup but I left my comments on the first one | 16:17 |
| clarkb | mordred: :( sorry | 16:17 |
| *** lpetrut has quit IRC | 16:17 | |
| *** udesale has quit IRC | 16:18 | |
| corvus | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=311&rra_id=4&view_type=&graph_start=1514701615&graph_end=1531844359&graph_height=120&graph_width=500&title_font_size=12 | 16:19 |
| corvus | clarkb: ^ that includes the time when we were uploading ara files, right? | 16:19 |
| corvus | mar 15 is when ara middleware was enabled | 16:21 |
| clarkb | corvus: ya would've been Jan-Mar | 16:21 |
| clarkb | ish | 16:21 |
| mnaser | doesn't seem too wild in terms of usge | 16:22 |
| mnaser | usage | 16:22 |
| clarkb | corvus: when calculating disk use remember the majority of data on logs.o.o now is compressed, but we are talking about not storing compressed files in swift (which may result in a need for a scaling factor0 | 16:23 |
| clarkb | though swift might also store compressed, not sure | 16:23 |
| *** erlon_ has joined #openstack-infra | 16:23 | |
| *** owalsh has quit IRC | 16:26 | |
| *** owalsh has joined #openstack-infra | 16:27 | |
| clarkb | ok we are fully enqueued now | 16:27 |
| clarkb | mordred: any objection to me just fixing my derp on that review by pushing a new patchset? | 16:27 |
| mordred | clarkb: please do! | 16:29 |
| *** owalsh_ has joined #openstack-infra | 16:29 | |
| *** florianf has quit IRC | 16:31 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Serve pypi proxy cache from port 80 https://review.openstack.org/583571 | 16:31 |
| clarkb | mordred: I think ^ that is what I was trying to say and did so poorly | 16:31 |
| *** owalsh has quit IRC | 16:33 | |
| mordred | clarkb: lgtm | 16:33 |
| mordred | clarkb: I am once again about to AFK - do you feel comfortable landing that - or do we want to wait until I get back to be able to help in case it derps? (although I guess revert is the response to derping) | 16:34 |
| mordred | clarkb: I'm not sure there's an awesome way to pre-test that | 16:34 |
| clarkb | we should probably have more eyeballs around for that one | 16:34 |
| clarkb | mordred: what we can do is apply it to a single mirror | 16:34 |
| clarkb | and check jobs in that region | 16:35 |
| mordred | ++ | 16:35 |
| clarkb | then if things are happy roll out more globally | 16:35 |
| clarkb | any opinions on a region to pick? | 16:35 |
| mordred | I can drive that when I get back - but if people get antsy before then I will not be offended | 16:35 |
| clarkb | cool. I'll probably defer to you on that. I need breakfast now | 16:36 |
| corvus | what is ara.json? | 16:40 |
| clarkb | dmsimard: ^ | 16:41 |
| corvus | eg in http://logs.openstack.org/02/583202/2/check/tripleo-ci-centos-7-containers-multinode-queens/50fe3e3/logs/ | 16:41 |
| dmsimard | doesn't ring me a bell, must be a tripleo-specific implementation | 16:41 |
| corvus | pabelanger, EmilienM: ^ ? | 16:42 |
| dmsimard | sshnaidm|bbl might know | 16:42 |
| corvus | clarkb: it looks like the tripleo ara is still pre-generated files | 16:43 |
| corvus | http://logs.openstack.org/02/583202/2/check/tripleo-ci-centos-7-containers-multinode-queens/50fe3e3/logs/ara_oooq/static/ | 16:44 |
| corvus | that job produced 4143 files | 16:45 |
| *** jpich has quit IRC | 16:45 | |
| corvus | 1810 of which are pre-generated ara | 16:46 |
| dmsimard | Last I know theybwere trying to address that | 16:46 |
| dmsimard | weshay ^ | 16:46 |
| *** iyamahat has joined #openstack-infra | 16:46 | |
| corvus | i'm not complaining, just getting numbers | 16:46 |
| dmsimard | Yup, just saying I saw a patch somewhere attempting to remove static generation and use ara-report instead | 16:47 |
| corvus | dmsimard: we should put that on hold | 16:47 |
| *** ijw has joined #openstack-infra | 16:47 | |
| corvus | my logs-in-swift proposal has us going back to pre-generation. if tripleo is still pre-generating, we should continue that for now | 16:47 |
| *** anteaya has joined #openstack-infra | 16:48 | |
| dmsimard | Uploading thousands of files instead of one regardless of storage backend is not efficient though | 16:49 |
| corvus | dmsimard: agreed | 16:50 |
| *** lbragstad has joined #openstack-infra | 16:50 | |
| *** myoung is now known as myoung|lunch | 16:50 | |
| *** e0ne has quit IRC | 16:51 | |
| dmsimard | I'll keep the swift use case in mind, IIRC logan- or hughsaunders use ARA statically generated reports through RAX's cloudfiles (swift), it works well but required some configuration | 16:53 |
| dmsimard | to find the index files properly | 16:54 |
| corvus | dmsimard: yesterday, someone found a sqlite js library... with that, do you think it would be possible to do what the middleware is doing, but in javascript? | 16:54 |
| logan- | yep, i have been archiving 3rd party ci logs to an s3 endpoint served by radosgw and using the s3 static website features to serve it | 16:55 |
| logan- | in archiving the logs, you need some concurrency otherwise it will be excruciatingly slow to archive a single job | 16:56 |
| logan- | my hacked together thing for that is here http://paste.openstack.org/raw/726216/ | 16:56 |
| dmsimard | Someone asked me the same question before, to be honest I have absolutely no idea. What I can tell you is that ARA 1.0 is API driven and while there's no frontend implementation yet, the backend/api/callback is almost ready for early usage | 16:56 |
| dmsimard | corvus: ^ | 16:56 |
| logan- | example log output of r a job http://osa-ci.protiumit.com/498146/2/368/logs/ | 16:56 |
| corvus | dmsimard: that makes it sound even less like it will be possible to use ara without a server | 16:57 |
| logan- | it is possible to serve the gzipped files to the web browser the same way logs.o.o does as long as you set the content encoding correctly at upload time | 16:57 |
| corvus | logan-: thanks! that could come in handy :) | 16:58 |
| *** graphene has quit IRC | 16:59 | |
| logan- | the static website stuff works ok but my 2c is avoid going back to ara static site generation etc. use objects as the backend store and optimize the uploads using concurrency, and then use webservers backed by swift/s3 endpoints to serve the content so you can still enable ara middleware etc and cut down on storage sprawl | 17:00 |
| logan- | np | 17:00 |
| *** pguimaraes has quit IRC | 17:01 | |
| *** graphene has joined #openstack-infra | 17:01 | |
| dmsimard | corvus: actually, it was always a hard requirement to keep the same "pip install ara; export ANSIBLE_CALLBACK_PLUGINS=...; ansible-playbook" simple approach. Without going into the implementation details, the API server is optional and the API is consumed offline without http/network traffic leveraging sqlite by default | 17:01 |
| corvus | logan-: that's a possibility, but our middleware system is basically unmaintained at this point. i want to get out of that business altogether. | 17:01 |
| *** derekh has quit IRC | 17:02 | |
| pabelanger | corvus: I'm not sure what ara.json is | 17:02 |
| corvus | dmsimard: oh, ok. with ara 1.0 and offline api, are we still talking thousands of files? | 17:03 |
| dmsimard | pabelanger, corvus: from codesearch -- ara.json: http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/collect-logs/tasks/publish.yml#n21 | 17:03 |
| corvus | dmsimard: ah, i see, thanks. | 17:04 |
| corvus | clarkb: a recent devstack multinode job produces 1112 files if i run ara static generation | 17:04 |
| dmsimard | corvus: static generation is still an unknown at this time, 0.x was relying on a flask library (flask-frozen) to deliver the feature and the 1.0 backend/API sits on top on django/django-rest-framework | 17:04 |
| *** trown is now known as trown|lunch | 17:05 | |
| corvus | dmsimard: oh, you're saying the api server is optional for *writes*, but not for reads? | 17:05 |
| dmsimard | corvus: for both | 17:06 |
| corvus | dmsimard: erm, if not "static generation" what words should i use to communicate "human looks at an ara report with no api server running"? | 17:06 |
| dmsimard | corvus: the API implementation has an "offline" client which lets you query the API (which queries the database) without requiring an API server to be exposed -- you can also use the "online" client which uses standard python requests to communicate with an API server | 17:08 |
| corvus | dmsimard: what form does the ""offline" client" take? | 17:08 |
| clarkb | corvus: once we have a test cloud we should be able to upload that set and see if it is a throughput concern. ~1k files is probably the majority of files for those jobs but not a significant majority too | 17:09 |
| corvus | clarkb: yeah, rest of the job is 339 files. | 17:09 |
| dmsimard | corvus: a python library, ex: https://github.com/openstack/ara-server/blob/master/hacking/validate.py#L53-L55 | 17:10 |
| corvus | dmsimard: okay, so, to date there's no way to view an ara 1.0 report without running python | 17:12 |
| corvus | maybe that could become the basis of an offline generation tool | 17:12 |
| *** yamahata has joined #openstack-infra | 17:12 | |
| *** jcoufal has quit IRC | 17:13 | |
| dmsimard | There's prototype work on a pure javascript interface but my current understanding is that while standalone and stateless, it'd need an API server to talk to (a bit similar to zuul web I guess?) | 17:13 |
| dmsimard | Static generation could just be querying the API and feeding that to something else, I don't know for sure yet | 17:14 |
| *** shah^ has quit IRC | 17:14 | |
| *** jamesmcarthur has joined #openstack-infra | 17:15 | |
| *** shardy has quit IRC | 17:15 | |
| dmsimard | To be honest, the focus is the backend and the API -- the frontend things are still a bit hand wavy | 17:16 |
| ssbarnea1 | i am curious if there are any plans to upgrade gerrit version | 17:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Delete stale jobdirs at startup https://review.openstack.org/573302 | 17:22 |
| corvus | ssbarnea1: pabelanger was working on that but no longer has enough time, i'm sure he'd appreciate help | 17:22 |
| ssbarnea1 | i am asking this because the current "My Reviews" window is very wide and uncontrolable in our current version of gerrit. | 17:22 |
| pabelanger | indeed :( | 17:23 |
| ssbarnea1 | well time is hard with all of us but better to know what are the current challenges | 17:23 |
| *** electrofelix has quit IRC | 17:23 | |
| ssbarnea1 | i don't want to commit without having an idea about what I would face | 17:23 |
| EmilienM | corvus: sorry I missed your ping, I was afk for lunch. Did you get a proper answer? | 17:25 |
| corvus | EmilienM: yes, thanks | 17:25 |
| *** dtantsur is now known as dtantsur|afk | 17:26 | |
| *** pabelanger has quit IRC | 17:34 | |
| *** pabelanger has joined #openstack-infra | 17:34 | |
| *** timothyb89_ is now known as timothyb89 | 17:35 | |
| *** tesseract has quit IRC | 17:35 | |
| *** e0ne has joined #openstack-infra | 17:39 | |
| *** e0ne has quit IRC | 17:44 | |
| *** r-daneel has quit IRC | 17:45 | |
| *** ijw has quit IRC | 17:46 | |
| *** ijw has joined #openstack-infra | 17:46 | |
| *** r-daneel has joined #openstack-infra | 17:47 | |
| *** e0ne has joined #openstack-infra | 17:48 | |
| *** jtomasek has quit IRC | 17:50 | |
| *** e0ne has quit IRC | 17:51 | |
| prometheanfire | I'm guessing the pypi mirror is still wedged? | 17:52 |
| *** r-daneel_ has joined #openstack-infra | 17:53 | |
| *** r-daneel has quit IRC | 17:53 | |
| *** r-daneel_ is now known as r-daneel | 17:53 | |
| *** jchhatbar has quit IRC | 17:55 | |
| clarkb | prometheanfire: yes, mordred is going to switch us to a proxy to pypi when he returns | 17:55 |
| *** dklyle has joined #openstack-infra | 17:55 | |
| prometheanfire | thanks | 17:55 |
| pabelanger | clarkb: is there a patch? We likely also need to do the same in rdoproject | 17:56 |
| pabelanger | and happy to test via depends-on | 17:56 |
| clarkb | pabelanger: https://review.openstack.org/583571 | 17:56 |
| clarkb | I don't think you can test that unless you deploy new mirror nodes | 17:56 |
| pabelanger | ah, new vhost change | 17:57 |
| clarkb | our plan is to apply it to one region check it owrks then roll out globally | 17:57 |
| pabelanger | k | 17:57 |
| *** ijw has quit IRC | 17:58 | |
| *** dklyle has quit IRC | 18:01 | |
| *** dbecker has quit IRC | 18:02 | |
| *** pcaruana has quit IRC | 18:08 | |
| *** myoung|lunch is now known as myoung | 18:09 | |
| *** e0ne has joined #openstack-infra | 18:10 | |
| *** e0ne has quit IRC | 18:17 | |
| *** sambetts is now known as sambetts|afk | 18:21 | |
| clarkb | I'm going to put the limestone mirror in the ansible emergency file so that we can modify that mirror to check pypi | 18:21 |
| clarkb | mordred: ^ mirror02.regionone.limestone.openstack.org is in the emergency file for this when you get back | 18:24 |
| clarkb | mordred: I figure we can manually apply the change to the apache config, restart apache and then manually test as well as watch jobs running there? | 18:27 |
| *** dklyle has joined #openstack-infra | 18:27 | |
| *** lpetrut has joined #openstack-infra | 18:28 | |
| mordred | clarkb: ++ | 18:28 |
| clarkb | mordred: I'm ready when you are if you want ot point me at what I can help with | 18:29 |
| clarkb | compltely unrelated, jurassic park is on netflix | 18:30 |
| mordred | clarkb: zomg. I'm unreasonably excited about that | 18:31 |
| mordred | clarkb: I'm going to edit the file there now | 18:31 |
| corvus | clarkb: it's a unix system, i know this! | 18:31 |
| clarkb | corvus: its amazing how well it as aged | 18:32 |
| clarkb | the premise, things like unix, and the special effects | 18:33 |
| mordred | clarkb: file is updated - I'm going to graceful apache now | 18:34 |
| mordred | clarkb: done | 18:34 |
| clarkb | ok /me checks pypi | 18:35 |
| *** ramishra has quit IRC | 18:35 | |
| clarkb | http://mirror.regionone.limestone.openstack.org/pypi/ seems to work but isn't pretty | 18:35 |
| clarkb | should be fine for pip | 18:35 |
| corvus | i distinctly remember thinking how ridiculous the file browser was, then being told it was, in fact, a legit file browser on sgi, and realizing i still had a lot to learn from the world of unix :) | 18:35 |
| clarkb | http://mirror.regionone.limestone.openstack.org/pypi/simple/pbr/ works too | 18:35 |
| mordred | clarkb: yah - and http://mirror02.regionone.limestone.openstack.org/pypi/simple/keystoneauth/ has /pypifiles links | 18:36 |
| clarkb | I was able to manually download a pbr package | 18:36 |
| *** eernst has quit IRC | 18:38 | |
| corvus | with ara generation and htmlified copies of logs, devstack-multinode has 1582 log files | 18:39 |
| clarkb | mordred: I'm a bit slower on doing a pip install locally but working on that too | 18:39 |
| corvus | i'm going to work on the upload script now, and get some timings | 18:39 |
| clarkb | mordred: ok local pip install of ara seems to be working | 18:40 |
| clarkb | mordred: I'll rerun the install then we should be able to check that there are cache hits for those packages | 18:41 |
| clarkb | corvus: using vexxhost? | 18:42 |
| corvus | clarkb: yeah | 18:43 |
| *** erlon_ has quit IRC | 18:43 | |
| clarkb | mordred: we havne't applied the combined-cache logging format to port 80 vhost | 18:47 |
| clarkb | mordred: so I can't confirm we are hitting the cache for those. Are you ok with me making that change and reloading the config on limestone mirror? | 18:47 |
| mordred | clarkb: yup! | 18:47 |
| clarkb | ok doing that now | 18:47 |
| clarkb | mordred: ok I see cache hits now. | 18:49 |
| clarkb | mordred: from what I can see locally and in the logs of the mirror it appears to be happy. We just need to update the change to add the cache logs | 18:49 |
| mordred | clarkb: ++ | 18:50 |
| mordred | clarkb: yeah - same here- I *think* this should work fine | 18:50 |
| mordred | clarkb: do we need to do anything with cache size? | 18:51 |
| mordred | clarkb: (also, I assumed that putting that same disk cache config in both vhosts was fine, and it seems to be working ok) | 18:51 |
| clarkb | mordred: I don't think so. We share a global cache among the vhosts (that is why we have the lock file specified) | 18:51 |
| mordred | cool | 18:51 |
| clarkb | yup as long as we have the lock file set it should be fine, then the existing curation of the cache should just work | 18:51 |
| clarkb | I think it will even share cached files between port 80 and 8080 this way | 18:52 |
| clarkb | not super useful | 18:52 |
| clarkb | but cool | 18:52 |
| clarkb | mordred: I'll get a new patchset up with the log change | 18:52 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Serve pypi proxy cache from port 80 https://review.openstack.org/583571 | 18:53 |
| clarkb | infra-root ^ mordred and I think that is ready. Between the two of us I think we already have one +2 but if someone else can review it that would be excellent | 18:54 |
| clarkb | mordred: oh we should test that the wheel mirror works | 18:54 |
| clarkb | mordred: can you do that? | 18:54 |
| *** harlowja has joined #openstack-infra | 18:54 | |
| pabelanger | clarkb: oh, maybe https://review.openstack.org/497948/ will be of interest for vhost changes to mirror | 18:57 |
| pabelanger | reload, not restart apache | 18:57 |
| clarkb | in this case I think reducing the number of variables will be helpful because if it goes sideways it will go fast, but we should get that in as a followup | 18:58 |
| clarkb | pabelanger: once the dust settles on the vhost update I'll look at getting that in too | 18:58 |
| *** priteau has quit IRC | 18:58 | |
| pabelanger | wfm | 18:59 |
| *** ldnunes has quit IRC | 18:59 | |
| pabelanger | clarkb: also, https://review.openstack.org/#/q/topic:afs-mirror-centos seems to be the way how rdoproject is trying to keep in sync with mirror changes, so after this is fixed, if you can review and see if a good idea, then I'll rebase everything with latest changes | 19:00 |
| pabelanger | +2 | 19:00 |
| mordred | clarkb: yes to wheel mirror - testing now | 19:01 |
| clarkb | mordred: thanks, I don't have an ubuntu or centos machine handy to test it locally | 19:01 |
| clarkb | one side effect of this change is that pypi/ no longer shows up in the listing under / | 19:01 |
| clarkb | not a deal breaker but we'll need to communicate better probably | 19:02 |
| *** owalsh_ is now known as owalsh | 19:03 | |
| mordred | clarkb: wheel mirror seems to work | 19:06 |
| clarkb | mordred: shall we approve the change then? | 19:06 |
| mordred | clarkb: I think so - I can't think of any other way to test that it | 19:07 |
| *** eharney has quit IRC | 19:07 | |
| *** kjackal has joined #openstack-infra | 19:08 | |
| clarkb | ya I can't think of anything else either | 19:09 |
| mordred | infra-root: anybody have any ideas of ways to validate the pypi miror change before pull the trigger? | 19:10 |
| pabelanger | I'm okay if people have tested locally | 19:12 |
| pabelanger | otherwise, maybe update zuul-jobs configure-mirror and hard code new mirror. think we have some functional testing around it | 19:13 |
| clarkb | pabelanger: the commit message explains why that is difficult, specifically easy install not unerstanding other pors | 19:14 |
| clarkb | we've have to deploy an entirely new mirror, point jobs at that and so on | 19:14 |
| mordred | yeah. that was the original plan - and got stymied by easy_install :( | 19:14 |
| pabelanger | clarkb: I think tripleo-test-cloud-rh1 mirror is still online | 19:15 |
| pabelanger | I've used that in the past to test | 19:15 |
| pabelanger | but also happy if others are | 19:15 |
| clarkb | pabelanger: well we deployed it to limestone to test it | 19:15 |
| pabelanger | k | 19:15 |
| clarkb | pabelanger: mirror.regionone.limestone.openstack.org/pypi if you want to test it | 19:15 |
| clarkb | (my local testing above is using pip locally against that index) | 19:16 |
| pabelanger | checking something in rdoproject | 19:16 |
| mordred | clarkb: mirror02.regionone.limestone.openstack.org/pypi isn't it? | 19:16 |
| clarkb | mordred: that is the actual name but we cname it without the digits | 19:16 |
| mordred | ah - awesome | 19:17 |
| clarkb | this way we can easily redeploy and zuul never realizes | 19:17 |
| pabelanger | okay, trying to use limestone mirror from rdoproject as quick test | 19:20 |
| *** njohnston has joined #openstack-infra | 19:21 | |
| *** dklyle has quit IRC | 19:23 | |
| pabelanger | job running, should know more in a few minutes | 19:23 |
| mordred | woot. thanks! | 19:23 |
| *** njohnston has quit IRC | 19:24 | |
| *** felipemonteiro_ has joined #openstack-infra | 19:24 | |
| *** felipemonteiro_ is now known as felipemonteiro | 19:24 | |
| *** felipemonteiro_ has joined #openstack-infra | 19:30 | |
| *** felipemonteiro has quit IRC | 19:33 | |
| corvus | clarkb, mordred: so... if we're gonna have a zuul job use a secret to make a swift container and upload stuff... should that container be in the openstackci or openstackjenkins account? | 19:33 |
| mordred | corvus: fascinating question | 19:33 |
| mordred | in this specific case | 19:33 |
| clarkb | I would put it in the jenkins/zuul account | 19:34 |
| mordred | yeah - I think I'd vote for that too | 19:34 |
| clarkb | if we were going to store logs forever I'd argue that maybe it go in the permanent resources openstackci account | 19:34 |
| corvus | okiedokie. i think that's probably right as well. :) | 19:34 |
| clarkb | but they are ephemeral resources like nodes (they just live longer) | 19:34 |
| mordred | yah | 19:34 |
| *** e0ne has joined #openstack-infra | 19:39 | |
| pabelanger | clarkb: mordred: mirror seems to have worked | 19:40 |
| pabelanger | tox job passed | 19:40 |
| clarkb | cool, in that case I think we approve the change, watch it, then remove limestone mirror from emergency file once puppet will noop | 19:41 |
| clarkb | mordred: ^ I'll let you hit the go button, but I think between us we hvae a combined +2 on the edits to that change | 19:41 |
| clarkb | unless corvus wants to review it? | 19:41 |
| mordred | clarkb: I think corvus really wants to reviews | 19:41 |
| corvus | i yield the remainder of my review time to the gentleman from oregon | 19:42 |
| corvus | (i think you should hit the button :) | 19:43 |
| clarkb | done | 19:43 |
| mordred | here we go | 19:43 |
| mordred | this is either going to be great, terrible or isn't going to be noteworthy at all | 19:43 |
| corvus | that's a boolean 'or' right? | 19:43 |
| tobiash | does swift scale well with many small files? | 19:44 |
| mordred | I was wondering how long it would take for someone to enjoy my use of either/or with three options | 19:44 |
| mordred | 15 seconds, it turns out | 19:44 |
| mordred | notmyname: ^^ | 19:44 |
| tobiash | We're currently storing logs on artifactory and it doesn't | 19:44 |
| mordred | notmyname: (the swift question, not the three-choice either/or) | 19:44 |
| *** kjackal has quit IRC | 19:44 | |
| tobiash | And I'm thinking about moving to fileserver or swift (cepf backed) | 19:45 |
| mordred | tobiash: well, ceph-backed-swift might have a different answer to the many-small-files question than swift-swift | 19:46 |
| mordred | I can get you an answer on that real quick though | 19:46 |
| clarkb | tobiash: that is also the questions that corvus is attempting to answer for openstack | 19:47 |
| notmyname | hi | 19:47 |
| *** dpawlik has quit IRC | 19:47 | |
| notmyname | tobiash: depends on what you mean by "many" and "small" | 19:47 |
| notmyname | ;-) | 19:47 |
| tobiash | notmyname: e.g uploading of html reports of jobs consisting of about maybe 500-1000 files per job (but less than 30MB) | 19:48 |
| *** hashar has quit IRC | 19:49 | |
| *** efried_rollin is now known as efried | 19:49 | |
| notmyname | 30MB total. or each? | 19:49 |
| tobiash | I'm thinking about moving our logs storage to swift/radosgw as backend | 19:49 |
| clarkb | for the openstack example above corvus said ~1.5k files for openstack. Sizes range between a few kb to a few MB iirc | 19:49 |
| *** felipemonteiro_ has quit IRC | 19:50 | |
| mordred | notmyname: total | 19:50 |
| tobiash | 30mb/500files per job result | 19:50 |
| *** felipemonteiro_ has joined #openstack-infra | 19:50 | |
| notmyname | are you the one running the cluster or just using it? | 19:50 |
| *** yamahata has quit IRC | 19:50 | |
| tobiash | I'm just using it | 19:50 |
| notmyname | then it'll be fine | 19:50 |
| clarkb | hahahaha | 19:51 |
| tobiash | and I would need to put authenticated apaches in front of it | 19:51 |
| clarkb | notmyname: assuming we were running the cluster what are the concerns :) | 19:51 |
| mordred | notmyname: so I guess it'll be fine for infra doing the same thing with swifts in public clouds? :) | 19:51 |
| notmyname | these aren't going to be frequently accessed, right? I don't imaging job log would be | 19:51 |
| notmyname | mordred: right | 19:51 |
| tobiash | yes | 19:51 |
| notmyname | this is a very common sort of use case for swift. one that's it's pretty good at | 19:51 |
| corvus | about as close as "write-only" storage as you can get :) [except that we have a bot which also reads everything once] | 19:51 |
| *** e0ne has quit IRC | 19:52 | |
| mordred | and MAYBE a human will look at them once - but statistically speaking most of them will never be looked at | 19:52 |
| notmyname | from the operator perspective, small files can cause concerns with increased inode usage (leading to system page cache issues) and more inefficient IO patterns (ie lots of random reads/writes). | 19:53 |
| notmyname | but just for storing stuff, it'll be fine | 19:53 |
| tobiash | how would the upload of many small files perform then? In Artifactory we need to tar that and tell it to explode the archive locally. That leads to timeouts where we don't have a chance to check if that was successful. | 19:53 |
| *** iyamahat has quit IRC | 19:54 | |
| notmyname | you can do that in swift, too. otherwise, PUT each of them. use concurrency | 19:54 |
| clarkb | notmyname: swift can untar on the backend? | 19:54 |
| clarkb | corvus: ^ may be helpful for us | 19:54 |
| notmyname | clarkb: if the operator has enabled it. https://docs.openstack.org/swift/latest/middleware.html#extract-archive | 19:55 |
| clarkb | neat | 19:55 |
| notmyname | how many total files are you putting in? if it's unbounded, then what's the rate? | 19:55 |
| tobiash | Artifactory has races with this, you upload tar'ed A and B at the same time and we sometimes get A and A as a result... | 19:55 |
| tobiash | we would delete the files after around 10 days | 19:56 |
| clarkb | tobiash: one handy feature of swift is you can set an expiry on the object at upload and then swift handles the expiration for you | 19:56 |
| clarkb | tobiash: as for dealing with races can you use the zuul job uuids to make each path unique? | 19:57 |
| tobiash | clarkb: swift service or swift api (as our 'swift' is actially ceph with radosgw)? | 19:57 |
| clarkb | tobiash: you specify it through the api but then the service runs cleanup tasks in the background iirc | 19:57 |
| *** gfidente|afk has quit IRC | 19:58 | |
| clarkb | notmyname: can correct me if I got that wrong | 19:58 |
| tobiash | clarkb: that race is within artifactory, two jobs upload at once <uuid-a>/buildlog and <uuid-b>/buildlog and sometimes in both paths contain the same buildlog of one of these | 19:58 |
| notmyname | clarkb: correct | 19:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM - testing shadowing https://review.openstack.org/583721 | 19:58 |
| clarkb | tobiash: weird | 19:58 |
| notmyname | tobiash: real swift. I cannot comment on any other system that reimplements the swift api. (other than it's "not swift") | 19:58 |
| tobiash | notmyname: understood, thanks | 19:59 |
| clarkb | mordred: wondering if we should enqueue the pypi change to the gate | 19:59 |
| clarkb | mordred: might be another hour or two otherwise | 19:59 |
| tobiash | I think I'll just try it | 19:59 |
| corvus | tobiash: hopefully by the end of my day, i'll have a rough draft script to upload | 19:59 |
| corvus | tobiash: (i'm not starting from scratch, there's an old script from a few years ago that does most of the hard work; i'm updating that) | 19:59 |
| tobiash | corvus: ah cool | 20:00 |
| *** hashar has joined #openstack-infra | 20:00 | |
| corvus | tobiash: i'll push it up as early as possible so you can start playing with it | 20:00 |
| *** iyamahat has joined #openstack-infra | 20:00 | |
| *** iyamahat_ has joined #openstack-infra | 20:02 | |
| tobiash | corvus: I'll be on a workshop tomorrow and and have other more urgent artifactory and zuul topics to solve before so don't hurry because of me ;) | 20:02 |
| corvus | tobiash: ack. i'm pretty independently motivated anyway. :) | 20:03 |
| tobiash | I managed today to pre-pull docker images in diskimage-builder :) | 20:03 |
| clarkb | I went ahead and enqueued the pypi change. I don't want to wait around all day for it :) | 20:03 |
| tobiash | (that runs in docker) | 20:03 |
| tobiash | I'm now running docker in bwrap in docker in a vm... | 20:03 |
| clarkb | tobiash: you are letting the executor execute docker? | 20:04 |
| tobiash | clarkb: it executes docker on the nodes in most projects | 20:05 |
| tobiash | clarkb: we use it more or less like you're using bindep in openstacek | 20:05 |
| clarkb | out side the executor bwrap though? (that is what I would expect, curious if running on executor directly) | 20:05 |
| *** boden has quit IRC | 20:06 | |
| tobiash | clarkb: no, the executor doesn't run docker | 20:06 |
| *** iyamahat has quit IRC | 20:06 | |
| tobiash | but our nodepool builder (runs in docker) runs diskimage-builder and that installs docker in the image and runs it to pre-pull images with the help of bwrap | 20:07 |
| corvus | mordred: this mean anything to you? keystoneauth1.exceptions.http.NotFound: (https://auth.vexxhost.net/auth/tokens): The resource could not be found. (HTTP 404) | 20:07 |
| clarkb | tobiash: aha, interesting | 20:07 |
| corvus | that's from: for c in conn.object_store.containers(): | 20:07 |
| clarkb | corvus: vexxhost is keystone v3 api only now | 20:07 |
| clarkb | corvus: and I think thins may default to v2 in places, so you need to make it v3 all the things I want to say | 20:08 |
| corvus | i made a clouds.yaml that looked like what's on nl03 | 20:08 |
| corvus | it doesn't include an auth_url | 20:08 |
| smcginnis | Sorry if it's been stated, but I've been off busy on other things. How's the pypi mirror issue going? | 20:09 |
| clarkb | corvus: you set the domain values though? doing that should cause v3 stuff to happen | 20:09 |
| clarkb | smcginnis: 583571 is in the gate and once applied will convert all of our mirrors over to caching proxies for pypi.org | 20:09 |
| corvus | clarkb: yes. i did not have that at first, got a different error, then added those. | 20:09 |
| corvus | clarkb: here's what i have: http://paste.openstack.org/show/726232/ | 20:09 |
| smcginnis | clarkb: Awesome, thanks! | 20:10 |
| clarkb | smcginnis: then assuming that works we can work to see if fixing the mirror is worthwhile or just go with proxy for the future | 20:10 |
| smcginnis | I saw the note about tensorflow having 500m. Holy cow. I'm surprised that's allowed. | 20:10 |
| clarkb | corvus: that looks right to me | 20:10 |
| clarkb | smcginnis: they push nightly wheel builds which ends up as windows * osx * linux * cuda version x * cuda version y * every day | 20:11 |
| clarkb | a couple other ml projects due that too. The common size problem seems to be cuda | 20:11 |
| smcginnis | Yikes | 20:11 |
| mordred | corvus: sorry - was just eating a sandwich - looking now | 20:11 |
| clarkb | corvus: maybe try that clouds.yaml with osc and --debug? | 20:11 |
| mordred | corvus: I would expect that to work | 20:12 |
| corvus | mordred: do you want to interactively debug? ~corvus/logs on static.openstack.org | 20:13 |
| corvus | mordred: clouds.yaml and test.py in that directory | 20:13 |
| mordred | corvus: yes. looking | 20:13 |
| corvus | mordred: source venv/bin/activate in that dir as well | 20:13 |
| corvus | mordred: actually | 20:14 |
| corvus | i started a screen, why don't you join it | 20:14 |
| mordred | neat | 20:14 |
| mordred | corvus: sigh. how do I do that again? I got the Cannot open your terminal '/dev/pts/1' - please check. error | 20:14 |
| mordred | corvus: (after becoming you) | 20:15 |
| * mordred should be better at screen | 20:15 | |
| corvus | mordred: "script /dev/null" i think | 20:15 |
| *** eernst has joined #openstack-infra | 20:17 | |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 20:18 | |
| *** kjackal has joined #openstack-infra | 20:19 | |
| *** yamahata has joined #openstack-infra | 20:21 | |
| mordred | corvus: ok. I'll dig in to why that is the case .... but you should be good to go now | 20:24 |
| corvus | mordred: thanks. that is weird and confusing. :) | 20:24 |
| mordred | yes. it's also a bug and shouldn't work that way :) | 20:24 |
| corvus | "auth_type: password" was required (for the record) | 20:24 |
| mordred | yeah. which is WHAT? | 20:24 |
| clarkb | is that because swift supports non keystone auth? | 20:25 |
| mordred | nope. it's a legit bug in sdk | 20:25 |
| mordred | or keystoneauth - don't know where it is yet | 20:26 |
| mordred | but it's traversing keystone discovery incorrectly | 20:26 |
| clarkb | pypi change should merge momentarily | 20:27 |
| mordred | mnaser: http://paste.openstack.org/show/726235/ summarizes ... I don't know where the bug is, but out of the box sdk isn't working against vexxhost atm and I don't know why | 20:29 |
| mordred | mnaser: but thought I'd let you know - I'm looking in to it | 20:30 |
| *** john_studarus has joined #openstack-infra | 20:30 | |
| mnaser | oh that's weir | 20:30 |
| mordred | mnaser: this happens: Making authentication request to https://auth.vexxhost.net/auth/tokens | 20:31 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Add a dequeue command to zuul client https://review.openstack.org/95035 | 20:31 |
| mnaser | that's no bueno | 20:31 |
| mordred | mnaser: which is completely incorrect given the discover doc | 20:31 |
| mnaser | could it be the change i submitted to our vendor stuff :( | 20:31 |
| *** r-daneel has quit IRC | 20:31 | |
| john_studarus | Can someone explain the Nodepool Grafana metric "Error Node Launch Attempts" and if I should be worried that it is non-zero... | 20:31 |
| mordred | and I verified there's a document coming back from discovery that's correct | 20:31 |
| mordred | mnaser: I'm about to try with raw keystoneauth | 20:32 |
| mnaser | wait | 20:32 |
| mnaser | ah nevermind | 20:32 |
| mnaser | i thought the x-auth-uri supplied was wrong but that wouldnt cause this | 20:32 |
| mnaser | mordred: but yeah, i did this a while back - https://review.openstack.org/#/c/574745/ | 20:32 |
| openstackgerrit | Merged openstack-infra/system-config master: Serve pypi proxy cache from port 80 https://review.openstack.org/583571 | 20:33 |
| openstackgerrit | Merged openstack-dev/pbr master: trivial: Remove 'tools/releasenotes_tox.sh' https://review.openstack.org/580414 | 20:34 |
| prometheanfire | mordred: the mirror ok now? | 20:36 |
| clarkb | john_studarus: it is count of boot errors. could be nodepool side or cloud side. I can check logs | 20:36 |
| clarkb | prometheanfire: no | 20:36 |
| prometheanfire | k | 20:36 |
| clarkb | prometheanfire: should be soon needed changejust merged | 20:36 |
| mordred | prometheanfire: it should be in approx 15 minutes give or take | 20:36 |
| prometheanfire | cool | 20:37 |
| john_studarus | k | 20:38 |
| *** dklyle has joined #openstack-infra | 20:39 | |
| mordred | mnaser: ok. TIL something | 20:40 |
| *** r-daneel has joined #openstack-infra | 20:40 | |
| mordred | mnaser: auth_type v3password skips discovery on the auth_url - so in order to set auth_type v3password we should also do https://auth.vexxhost.net/v3 | 20:40 |
| clarkb | john_studarus: shade.exc.OpenStackCloudHTTPError: (403) Client Error for url: https://packet.platform9.net/nova/v2.1/ba163363643541d88a737f8af23cfa7f/servers Maximum number of ports exceeded is at least part of the issue | 20:41 |
| clarkb | john_studarus: shade.exc.OpenStackCloudTimeout: Timeout waiting for the server to come up. is another | 20:41 |
| john_studarus | ok - I'll take a look and narrow it down | 20:41 |
| clarkb | john_studarus: for the ports exceeded issue I think we may just need to increase the port quota, we have leak clenaups for that iirc but it runs periodically | 20:41 |
| john_studarus | are those logs public? | 20:42 |
| clarkb | so if you leak and then hit limit before cleanup happens you can fail. As for boot timeouts we can configure that. It is currently set to 300 seconds | 20:42 |
| clarkb | john_studarus: no, we've never made them public as there were fears over keystone auth divulging secrets | 20:42 |
| john_studarus | no worries | 20:42 |
| john_studarus | I'll run through the logs on my side | 20:42 |
| clarkb | I think at this point we probably can reconsider that | 20:42 |
| clarkb | john_studarus: 8f434ea3-746b-4135-ac06-d8ef6599888f is uuid for serverthat timed out | 20:43 |
| *** jamesmcarthur has quit IRC | 20:44 | |
| john_studarus | ok - I'll let you know what I find | 20:45 |
| john_studarus | the Grafana charts give me good metrics to see where I can improve :) | 20:46 |
| clarkb | john_studarus: more than happy to dig up more info if it helps | 20:46 |
| clarkb | mordred: I think we have to wait for the next time puppet goes around to have the pypi update apply | 20:47 |
| mordred | clarkb: blert | 20:48 |
| *** felipemonteiro__ has joined #openstack-infra | 20:48 | |
| clarkb | puppet just ran on rax dfw mirror and was noo | 20:48 |
| openstackgerrit | Merged openstack-dev/pbr master: trivial: Fix file permissions https://review.openstack.org/580415 | 20:48 |
| clarkb | *noop | 20:48 |
| clarkb | I think because our runs are taking like 25 minutes and this was part of the previous 15 minute block | 20:48 |
| clarkb | (can you tell I'm impatient) | 20:48 |
| *** lpetrut has quit IRC | 20:48 | |
| corvus | we should totally redo how we run these | 20:48 |
| clarkb | corvus: and make mordred plan how to do it | 20:49 |
| corvus | ++ | 20:49 |
| pabelanger | john_studarus: clarkb: thanks, I started looking into those today but wasn't sure who the new contact was :) | 20:49 |
| mordred | clarkb, corvus: one of these mornings you're going to wake up and I'm going to have rewritten the whole thing - but it'll only work 80% of the time ... | 20:49 |
| *** rtjure has joined #openstack-infra | 20:50 | |
| clarkb | mordred: I feel like this has happened before :) | 20:50 |
| *** john_studarus has quit IRC | 20:50 | |
| mnaser | mordred: it feels wrong putting `v3password` | 20:50 |
| mnaser | err | 20:50 |
| mnaser | i mean | 20:50 |
| mnaser | having `/v3` | 20:50 |
| mordred | mnaser: yup. I agree | 20:50 |
| mordred | we did that to be more explcit and get better error messages though since you don't support v2 anymore | 20:51 |
| mordred | maybe we should add /v3 instead of moving back to password | 20:51 |
| mnaser | hmm | 20:51 |
| mnaser | why is v2.0 still referenced in / | 20:51 |
| mordred | I don't know! that's a great question | 20:52 |
| mnaser | oh it's listed as deprecated | 20:52 |
| mordred | ah | 20:52 |
| *** felipemonteiro_ has quit IRC | 20:52 | |
| mordred | but - I thought you weren't running the endpoint | 20:52 |
| mnaser | it technically isn't deprecated though. | 20:52 |
| mnaser | it's gone | 20:52 |
| mnaser | so is that a keystone bug? because we don't run v3 because queens doesnt have v3 | 20:52 |
| mordred | kmalloc, lbragstad, cmurphy: ^^ I would call that a keystone bug | 20:52 |
| mordred | mnaser: s/v3/v2/ but yeah | 20:52 |
| mnaser | yeah oops, i think there was some ec2tokens stuff that wasn't removed in queens | 20:53 |
| mnaser | maybe that's why it wasn't removed from the service catalog? | 20:53 |
| *** agopi is now known as agopi|brb | 20:53 | |
| *** dizquierdo has joined #openstack-infra | 20:53 | |
| mordred | mnaser: you're telling me that version discovery for keystone is returning invalid data in queens and it's because of ec2? | 20:54 |
| mordred | mnaser: I would find that ... unpleasing | 20:54 |
| mordred | mnaser: remote: https://review.openstack.org/583737 Add /v3 to the auth_url for vexxhost | 20:54 |
| mordred | mnaser: straw man ^^ | 20:54 |
| mnaser | https://developer.openstack.org/api-ref/identity/v2-ext/index.html | 20:54 |
| mnaser | i think this is the thing | 20:54 |
| mordred | sigh | 20:55 |
| mnaser | i think it has some heat related depedency | 20:55 |
| * mordred (╯°□°）╯︵ ┻━┻ | 20:55 | |
| *** dpawlik has joined #openstack-infra | 20:56 | |
| * lbragstad tries catching up | 20:56 | |
| openstackgerrit | Merged openstack-infra/zuul master: Delete stale jobdirs at startup https://review.openstack.org/573302 | 20:57 |
| *** rtjure has quit IRC | 20:57 | |
| *** agopi|brb has quit IRC | 20:57 | |
| mordred | lbragstad: tl;dr ... | 20:58 |
| mordred | lbragstad: a) v3password skips discovery for auth - so v3password with unversioned auth_url == sad panda | 20:59 |
| *** trown|lunch is now known as trown|outtypewww | 20:59 | |
| *** rtjure has joined #openstack-infra | 21:00 | |
| lbragstad | huh | 21:00 |
| mordred | lbragstad: b) queens version discovery shows v2.0 as deprecated, but most of v2 api is gone except for v2 extensions - which is just weird | 21:00 |
| mordred | s/discovery/discovery document/ | 21:00 |
| lbragstad | we (as in kmalloc) ripped the rest of that out | 21:00 |
| lbragstad | you're talking about the v2.0 ec2 stuff | 21:00 |
| lbragstad | ? | 21:00 |
| *** eharney has joined #openstack-infra | 21:00 | |
| mordred | yeah ... I'd argue v2 extensions and the reference in the discovery doc should have gone too | 21:00 |
| mordred | yeah | 21:00 |
| lbragstad | yep - that should be gone... | 21:00 |
| * lbragstad digs | 21:00 | |
| mordred | because otherwise it looks like v2 api is still available, although deprecated | 21:00 |
| *** dpawlik has quit IRC | 21:00 | |
| lbragstad | https://review.openstack.org/#/c/572846/ | 21:01 |
| prometheanfire | ok, going to rekick some reqs jobs now that it's been 15+ minutes | 21:01 |
| lbragstad | but - yeah that landed in Rocky | 21:02 |
| clarkb | prometheanfire: its not ready yet | 21:02 |
| clarkb | prometheanfire: we will let you know when it is | 21:02 |
| mnaser | but does having v2.0 in http://auth.vexxhost.net/ considered a bug? | 21:02 |
| clarkb | prometheanfire: we are waiting on puppet to run with the updated template file | 21:02 |
| prometheanfire | k | 21:02 |
| lbragstad | since there is an endpoint/api there - i'm inclined to say no | 21:03 |
| lbragstad | even though it is deprecated | 21:03 |
| lbragstad | correct me if i'm wrong, but would the fix be to implement discovery for passwordv3? | 21:05 |
| lbragstad | in ksa? | 21:05 |
| *** yamahata has quit IRC | 21:06 | |
| *** felipemonteiro__ has quit IRC | 21:07 | |
| *** felipemonteiro__ has joined #openstack-infra | 21:07 | |
| mordred | lbragstad: yeah - although I think in this case just knowing that's the behavior is fine | 21:07 |
| lbragstad | so - specify a version url in your clouds.yaml? | 21:08 |
| mordred | I think we'd need to think through the ramifications of changing it - and I'm pretty sure kmalloc would say it's a behavior change and thus a contract violation | 21:08 |
| mordred | yah | 21:08 |
| *** kjackal has quit IRC | 21:08 | |
| lbragstad | it's a behavior change to add that functionality to passwordv3? | 21:09 |
| mordred | I'll look at putting in a warning/error in sdk - "you specified v3password and don't have a versioned auth_url" - or maybe just append a /v3 to auth_url if auth_type is v3password | 21:09 |
| mordred | dunno ... I'm just imagining what kmalloc will say | 21:09 |
| mordred | it would result in an extra network roundtrip for some people potentially | 21:09 |
| lbragstad | i can see the reasoning there | 21:09 |
| lbragstad | but.. is it worse that we fail with an unversioned endpoint when it used to work? | 21:10 |
| mordred | the chances that someone is depending on that is pretty small - but I could also see the utility in the behavior so that someone could choose | 21:10 |
| lbragstad | becuase v2.0 is gone? | 21:10 |
| mordred | lbragstad: o - sorry - the two aren't related | 21:10 |
| mordred | this will fail with a cloud that has a v2.0 too | 21:10 |
| lbragstad | oh | 21:10 |
| mordred | if you explicitly choose v3password authplugin and an unversioned auth_url | 21:11 |
| mordred | it was just surprising | 21:11 |
| mordred | v3password assumes the auth_url is already versioned | 21:11 |
| mordred | the relationship to v2 being gone is that for vexxhost we changed the default to v3password so that the user would get an error on missing domain information | 21:12 |
| lbragstad | mnaser: you stumbled across this, too? | 21:12 |
| mordred | but ... that seems to have been an issue :) | 21:12 |
| mordred | lbragstad: fix is here:https://review.openstack.org/#/c/583737/ | 21:12 |
| clarkb | mordred: the issue is probably that GETting the v3/ path doens't do something useful you can switch on? | 21:12 |
| mnaser | lbragstad: yes, so customers would get a '404 page not found' if they didnt supply domain info | 21:13 |
| mnaser | because it tries to hit the v2 api | 21:13 |
| lbragstad | mnaser: mordred do we have this documented in a bug yet? | 21:13 |
| mnaser | that was confusing to them | 21:13 |
| mnaser | instead of saying 'missing domain info' | 21:13 |
| lbragstad | sure | 21:13 |
| mordred | clarkb: it does - but the v3password plugin assumes you know what you're talking about so doesn't do an auth version discovery step | 21:13 |
| lbragstad | i can see where that would be confusing | 21:13 |
| *** caphrim007_ has quit IRC | 21:13 | |
| lbragstad | s/where/how/ | 21:13 |
| *** caphrim007 has joined #openstack-infra | 21:14 | |
| mnaser | so i added that so customers would just get a'missing domain' error rather than a 404 | 21:14 |
| mnaser | so it gives more of a "misconfig" rather than "something is broken" | 21:14 |
| lbragstad | sure | 21:14 |
| lbragstad | so in the future are we always going to require uses specify a versioned endpoint then for passwordv3? | 21:14 |
| lbragstad | users* | 21:15 |
| mordred | lbragstad: yes - I think that is the path - so we might just need to document that well | 21:15 |
| mordred | up until now i've just pointed people at password, since it auto-detects | 21:15 |
| clarkb | mordred: and passwrd didn't work here due to a bug? | 21:16 |
| mordred | but with clouds stopping having v2 - the distinction between what config the user needs and what the cloud supports is a little trickier | 21:16 |
| clarkb | because it shold've done discovery with password seen v3 and used it right? | 21:16 |
| mordred | clarkb: password does work - but if the user doens't provide domain info it fails with a cryptic error, since v2 doesn't work on the cloud | 21:16 |
| clarkb | mordred: but corvus did provide the domain info | 21:17 |
| mordred | nope - it won't choose v3 if domain info isn't given | 21:17 |
| mordred | clarkb: right- that was the other mismatch | 21:17 |
| mordred | the vendor profile in sdk has v3password auth_type but unversioned endpoint | 21:17 |
| mordred | v3password skips discovery and assumes the auth_url given was the v3 endpoint | 21:17 |
| mordred | (which is what we learned) | 21:17 |
| mordred | so changing auth_type to password OR chaging auth_url to https://auth.vexxhost.net/v3 both fix the corvus issue | 21:18 |
| mordred | and in sdk we'll change the auth_url to v3 so that we get a good error on missing domain info | 21:18 |
| clarkb | gotcha | 21:19 |
| clarkb | mordred: rax dfw mirror just updated | 21:21 |
| lbragstad | mordred: want to reference that in your patch? https://bugs.launchpad.net/keystoneauth/+bug/1782434 | 21:21 |
| openstack | Launchpad bug 1782434 in keystoneauth "passwordv3 plugin doesn't implement discovery" [Undecided,New] | 21:21 |
| lbragstad | cc mnaser ^ | 21:21 |
| clarkb | prometheanfire: ok, I think you can recheck now | 21:21 |
| clarkb | smcginnis: ^ fyi mirrors are now caching proxies for pypi | 21:22 |
| mordred | lbragstad: thanks! the sdk patch is already +Ad, but I put a link to the patch in the bug | 21:22 |
| clarkb | if either of you notice anything weird please do let us know | 21:22 |
| lbragstad | mordred: wfm | 21:22 |
| lbragstad | the context is helpful | 21:22 |
| clarkb | I'm watching https://zuul.openstack.org/stream.html?uuid=364ff989ee7c41a79c0e57a7041a93a7&logfile=console.log | 21:24 |
| mordred | clarkb: I am now also watching that | 21:24 |
| mordred | clarkb: I rechecked https://review.openstack.org/#/c/583005 too - which is the patch that started this whole game :) | 21:25 |
| clarkb | mordred: run tox without tests passed without error | 21:25 |
| mordred | woot | 21:25 |
| *** aeng has quit IRC | 21:26 | |
| mordred | clarkb: seems to have worked | 21:26 |
| clarkb | I'll check the tox logs too after | 21:26 |
| clarkb | *after the upload | 21:26 |
| prometheanfire | clarkb: k, doing so now | 21:26 |
| clarkb | mordred: http://logs.openstack.org/44/525244/9/check/openstack-tox-py35/364ff98/tox/py35-1.log I see pypifiles | 21:28 |
| clarkb | looking good | 21:28 |
| *** felipemonteiro__ has quit IRC | 21:28 | |
| clarkb | corvus: mordred: thinking about fiddling with the afs mirror rebuild just to test it, I'm guessing that an rm -rf * will not be fast? is there a better way to clear a volume? | 21:28 |
| *** dizquierdo has quit IRC | 21:28 | |
| corvus | mordred: is openstacksdk thread safe such that i can call upload_object from multiple threads sharing the same connection? | 21:29 |
| mordred | corvus: it is my belief that this is the case | 21:29 |
| corvus | clarkb: deleting the entire volume is fast; but if you want to keep the ro volume working that's not what you want. | 21:30 |
| clarkb | corvus: it is probably a good idea to keep that around for a bit just in case we have to revert | 21:30 |
| corvus | clarkb: yeah, in which case, i don't have any immediate clever ideas. | 21:31 |
| mordred | clarkb, corvus: maybe we just wait until we're comfortable deleting before trying the rebuild | 21:31 |
| clarkb | mordred: ya that is an option | 21:31 |
| mordred | or - yolo | 21:31 |
| clarkb | we are already up to ~5k cache hits on dfw mirror | 21:31 |
| corvus | also, if rdoproject is using the ro volume right now, that might be rude | 21:32 |
| clarkb | we download a lot of pypi packages | 21:32 |
| mordred | corvus: my understanding is that apevec applied a version of our new cache patch to rdoproject already | 21:32 |
| mordred | and I've pointed them at the updated version | 21:32 |
| mordred | clarkb: yes we do :) | 21:32 |
| *** myoung is now known as myoung|afk | 21:32 | |
| mordred | clarkb: I have a hunch we may wind up cache tuning between the pypi mirror and the proxy-caches | 21:33 |
| mordred | clarkb: but I'm curious to see how it works just as it is | 21:33 |
| clarkb | mordred: possibly, I do think our global requirements results in a fairly constrained set overall | 21:33 |
| clarkb | we just pull them over and over | 21:33 |
| * mordred could imagine large docker images and the global-requirements set battling each other | 21:33 | |
| mordred | clarkb: ++ | 21:33 |
| mordred | clarkb: also, if you felt like reviewing https://review.openstack.org/#/c/583005 - you should have +2 permissions | 21:34 |
| *** yamahata has joined #openstack-infra | 21:35 | |
| clarkb | looking | 21:35 |
| clarkb | mordred: zuul says -1 | 21:35 |
| clarkb | oh that is what you rechecked got it | 21:35 |
| ianw | corvus: anything preventing removal of -1 on the log-inventory change @ https://review.openstack.org/#/c/563789/ ? i think the testing has been done now | 21:35 |
| mordred | clarkb: yeah - that -1 is why we just reworked the entire pypi mirror :) | 21:35 |
| clarkb | mordred: +2 I'll let you approve when zuul comes back +1 | 21:37 |
| corvus | ianw: probably not, i'll take a look soon | 21:39 |
| *** hashar has quit IRC | 21:41 | |
| *** dklyle has quit IRC | 21:42 | |
| clarkb | mordred: any idea if we've disabled the bandersnatch update cron? thinking that is a good first step on cleaning up afs side of things | 21:43 |
| *** ijw has joined #openstack-infra | 21:45 | |
| *** dklyle has joined #openstack-infra | 21:45 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Disable bandersnatch cron https://review.openstack.org/583749 | 21:48 |
| clarkb | if someone else has a change for ^ let me know and I'll abandon mine and revie wthat one | 21:48 |
| *** david-lyle has joined #openstack-infra | 21:53 | |
| *** dklyle has quit IRC | 21:54 | |
| *** edmondsw has quit IRC | 21:54 | |
| clarkb | limestone mirror removed from ansible emergency file | 21:55 |
| rosmaita | hello infra peeps ... can someone poke https://review.openstack.org/#/c/583231/1 -- it's been queued in the experimental queue for about 90 min with no action | 21:56 |
| clarkb | rosmaita: it is waiting for test nodes to become available | 21:56 |
| rosmaita | clarkb : thanks for looking, i will have to be patient | 21:57 |
| clarkb | (we ended up with a bit of a backlog today after needing to restart zuul to address a memory leak) | 21:57 |
| rosmaita | gotcha, thanks for keeping the lights on | 21:57 |
| clarkb | ccxt is making ~8 releases a day, thankfully all small | 22:00 |
| clarkb | certbot has different packages for every dns provider supported | 22:00 |
| clarkb | its neat to see different release processes in different projects | 22:01 |
| kmalloc | o/ | 22:01 |
| clarkb | the more I look at this the more I worry that pypi just isn't realistically mirrorable anymore | 22:02 |
| *** linkmark has joined #openstack-infra | 22:03 | |
| clarkb | we could do things like not mirroring wheels except we want wheels | 22:03 |
| clarkb | we could exclude non linux wheels, but eventually it will still catch up to us | 22:03 |
| clarkb | we could whitelist packages but we've done that before and it is problematic (have to update whitelist constantly) | 22:03 |
| clarkb | https://pypi.org/project/tf-nightly-gpu/#files and blacklisting every package like that is a race I don't want to try and win | 22:05 |
| clarkb | smcginnis: ^ fyi that is an example what what I'm talking about 1.25GB of packages every day from tf with that single package and they have others | 22:05 |
| corvus | mnaser: i broke my swift :( | 22:06 |
| notmyname | corvus: unpossible! | 22:06 |
| corvus | openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/, Bad Request | 22:06 |
| mnaser | uhoh | 22:06 |
| corvus | i get that when i try to list containers | 22:06 |
| mnaser | ok let me check logs | 22:06 |
| *** markmcd has quit IRC | 22:06 | |
| corvus | mnaser: looks like that followed a failed create (the create returned 500) | 22:07 |
| *** markmcd has joined #openstack-infra | 22:07 | |
| corvus | oh, er, the create may have been fine. it looks like the get_container_metadata after the create returned the 500 | 22:08 |
| notmyname | not sure how vexxhost is doing it, but it's slightly unusual to see the account part of the url without a "<reseller>_" prefix. ie (86bbbcfa8ad043109d2d7af530225c72 vs AUTH_86bbbcfa8ad043109d2d7af530225c72) | 22:09 |
| notmyname | anyway, it should be whatever you get back from auth. no munging on the client needed | 22:10 |
| mordred | corvus: isthat the sdk get_container_metadata or the shade? | 22:11 |
| corvus | mordred: sdk | 22:14 |
| mordred | yah. I don't think there should be a difference | 22:14 |
| mordred | looking at the calls | 22:14 |
| notmyname | mordred: link? | 22:14 |
| mordred | notmyname: oh - I meant between what sdk get_container_metadata and shade get_container ... they're both doing head on the container properly | 22:14 |
| notmyname | ok. let me know ifyou need another set of eyes (/me is sitting in a meeting now) | 22:15 |
| mordred | http://git.openstack.org/cgit/openstack/openstacksdk/tree/openstack/object_store/v1/_proxy.py#n93 vs | 22:15 |
| mordred | http://git.openstack.org/cgit/openstack/openstacksdk/tree/openstack/cloud/openstackcloud.py#n7204 | 22:15 |
| corvus | [just to clarify, this worked recently (i've been deleting and creating containers); i suspect it's an operational issue] | 22:15 |
| mordred | yah | 22:16 |
| smcginnis | clarkb: Sheesh! Thanks | 22:20 |
| clarkb | smcginnis: looks like tensorflow is about 2GB every day across their packages give or take a few hundred MB | 22:21 |
| clarkb | smcginnis: but stuff like that is why we have run out of space despite blacklisting packages | 22:21 |
| clarkb | hard to keep up | 22:21 |
| smcginnis | I wonder if someone should "suggest" to the pypi folks that they put some restrictions around that. | 22:21 |
| clarkb | smcginnis: possibly? I think it depends on whether or not pypi wants to be mirrorable and if they have concerns over storage costs in the long term | 22:22 |
| smcginnis | I would think they would. | 22:22 |
| clarkb | fwiw npm has similar issues, its basically unmirrorable as well | 22:22 |
| smcginnis | 22:22 | |
| clarkb | mordred: corvus: ya, I think if proxy cache holds up our best bet may just be to stick with that long term | 22:24 |
| clarkb | every alternative I try to come up wiht except for somehow deleting packages in a rolling fashion leads to eventually running out of disk | 22:24 |
| clarkb | and deleting old packages is something we've experienced as a painful exercise with pypi in the past | 22:24 |
| clarkb | rosmaita: your jobs are running now | 22:25 |
| *** gongysh has joined #openstack-infra | 22:25 | |
| clarkb | mnaser: thinking about the ipv6 issues, is there an easy way to temporarily disable ipv6 on the vexxhost nova instances? I want ot say we just use default networking there, so not sure | 22:28 |
| rosmaita | clarkb: i saw that, thanks ... one even passed already | 22:28 |
| *** wolverineav has quit IRC | 22:28 | |
| mnaser | clarkb: that would be a network wide switch and not something we’d want to do. Can you not maybe have something in base job to disable ipv6 matching our hostname? | 22:29 |
| mnaser | 8’ | 22:29 |
| *** lbragstad has quit IRC | 22:29 | |
| mnaser | I’m going to be back in Montreal by Friday and we’ve avoided any major changes in case something happens :) | 22:30 |
| clarkb | mnaser: ya we could disable it on our side too, was just wondering if there was an easy way for the images to never even see the RAs | 22:30 |
| clarkb | I didn't expect there was but figured I would ask | 22:30 |
| *** david-lyle has quit IRC | 22:31 | |
| *** hongbin has quit IRC | 22:32 | |
| *** tpsilva has quit IRC | 22:32 | |
| *** rcernin has joined #openstack-infra | 22:32 | |
| clarkb | pabelanger: what is the unit for used HDD? | 22:37 |
| clarkb | blocks? | 22:38 |
| clarkb | oh percentage | 22:39 |
| *** neiloy has quit IRC | 22:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Make zuul status HDD graph easier to read https://review.openstack.org/583791 | 22:48 |
| clarkb | pabelanger: ^ that may interest you. Trying to make it a bit easier to understand the hdd graph | 22:48 |
| *** eernst has quit IRC | 22:49 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Make zuul status HDD graph easier to read https://review.openstack.org/583791 | 22:53 |
| *** dpawlik has joined #openstack-infra | 22:57 | |
| *** rlandy has quit IRC | 22:57 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Add a dequeue command to zuul client https://review.openstack.org/95035 | 22:58 |
| *** uberjay has quit IRC | 22:59 | |
| *** honza has quit IRC | 22:59 | |
| *** wolverineav has joined #openstack-infra | 22:59 | |
| *** owalsh_ has joined #openstack-infra | 23:00 | |
| *** wendar has quit IRC | 23:00 | |
| *** owalsh has quit IRC | 23:01 | |
| *** dpawlik has quit IRC | 23:01 | |
| *** _ari_ has quit IRC | 23:03 | |
| *** haleyb has quit IRC | 23:03 | |
| *** myoung|afk has quit IRC | 23:03 | |
| *** _ari_ has joined #openstack-infra | 23:03 | |
| *** myoung has joined #openstack-infra | 23:03 | |
| *** wolverineav has quit IRC | 23:03 | |
| *** haleyb has joined #openstack-infra | 23:04 | |
| *** owalsh_ is now known as owalsh | 23:04 | |
| *** dklyle has joined #openstack-infra | 23:09 | |
| clarkb | john_studarus left, but thinking more about packethost and port quota, I wonder if the changes made around networking for larger MTUs resulted in some orphaned ports in neutron | 23:12 |
| clarkb | I'll take a look as I think I may be able to delete those from our end | 23:12 |
| *** dizquierdo has joined #openstack-infra | 23:17 | |
| *** tosky has quit IRC | 23:20 | |
| clarkb | there are some ports that are obviously leaked, due to being more than a few days old | 23:27 |
| clarkb | I am deleting those and keeping the more recent ones that are probably leaked but might help in debugging | 23:27 |
| *** jamesmcarthur has joined #openstack-infra | 23:28 | |
| *** eernst has joined #openstack-infra | 23:30 | |
| *** jamesmcarthur has quit IRC | 23:32 | |
| corvus | i've got a script able to upload to two clouds now; i should be ready for real testing tomorrow | 23:34 |
| clarkb | corvus: real testing like in jobs? | 23:35 |
| *** ijw has quit IRC | 23:37 | |
| corvus | clarkb: er, sorry, i mean bulk manual testing | 23:38 |
| clarkb | ah | 23:38 |
| corvus | i've just uploaded 3 objects so far :) | 23:39 |
| *** ijw has joined #openstack-infra | 23:39 | |
| *** ijw has quit IRC | 23:44 | |
| *** dizquierdo has quit IRC | 23:44 | |
| *** rpioso is now known as rpioso|afk | 23:48 | |
| *** ijw has joined #openstack-infra | 23:48 | |
| *** threestrands has joined #openstack-infra | 23:50 | |
| *** threestrands has quit IRC | 23:50 | |
| *** threestrands has joined #openstack-infra | 23:50 | |
| *** uberjay has joined #openstack-infra | 23:50 | |
| *** wendar has joined #openstack-infra | 23:51 | |
| *** ijw has quit IRC | 23:53 | |
| logan- | trying to figure out why the nodepool mirrors have python-software-properties in the bionic repos when it doesnt exist upstream | 23:55 |
| logan- | http://paste.openstack.org/raw/726250/ | 23:55 |
| *** jamesmcarthur has joined #openstack-infra | 23:56 | |
| *** dingyichen has joined #openstack-infra | 23:57 | |
| clarkb | logan-: ubuntu package search seems to say it doesn't belong either | 23:58 |
| *** rosmaita has quit IRC | 23:58 | |
| clarkb | it is in xenial | 23:59 |
| clarkb | and artful but none of the pckage versions match the one in your paste | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!