| *** thorst has joined #openstack-infra | 00:05 | |
| *** thorst has quit IRC | 00:05 | |
| *** chlong has joined #openstack-infra | 00:05 | |
| *** chlong has quit IRC | 00:12 | |
| *** chlong has joined #openstack-infra | 00:13 | |
| *** baoli has joined #openstack-infra | 00:16 | |
| *** salv-orlando has quit IRC | 00:21 | |
| *** tzn has quit IRC | 00:25 | |
| *** yamamoto_ has joined #openstack-infra | 00:27 | |
| *** baoli has quit IRC | 00:27 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Install docker for tests https://review.openstack.org/278021 | 00:28 |
|---|---|---|
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the identity v3 only job in additional services https://review.openstack.org/271128 | 00:28 |
| *** baoli has joined #openstack-infra | 00:28 | |
| *** ddieterly has joined #openstack-infra | 00:31 | |
| *** ddieterly has quit IRC | 00:32 | |
| *** ddieterly has joined #openstack-infra | 00:33 | |
| *** yamamoto_ has quit IRC | 00:33 | |
| *** sdake has quit IRC | 00:42 | |
| *** gildub has joined #openstack-infra | 00:45 | |
| *** tzn has joined #openstack-infra | 00:50 | |
| *** markvoelker has joined #openstack-infra | 00:55 | |
| *** clif is now known as clif_h | 00:56 | |
| *** tzn has quit IRC | 00:57 | |
| *** baoli has quit IRC | 00:58 | |
| *** sdake has joined #openstack-infra | 00:58 | |
| *** baoli has joined #openstack-infra | 00:58 | |
| *** markvoelker has quit IRC | 01:00 | |
| *** lucasagomes has quit IRC | 01:00 | |
| *** thorst has joined #openstack-infra | 01:06 | |
| *** lucasagomes has joined #openstack-infra | 01:06 | |
| *** chlong has quit IRC | 01:06 | |
| *** thorst has quit IRC | 01:14 | |
| openstackgerrit | Joseph Robinson proposed openstack-infra/irc-meetings: Add the new US meeting time to the User Guide Team yaml https://review.openstack.org/278866 | 01:16 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests https://review.openstack.org/177002 | 01:17 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add the new US meeting time to the User Guide Team yaml https://review.openstack.org/278866 | 01:28 |
| openstackgerrit | Kuo-tung Kao proposed openstack/requirements: update docker-py to 1.6.0 https://review.openstack.org/270625 | 01:28 |
| *** salv-orlando has joined #openstack-infra | 01:29 | |
| *** salv-orlando has quit IRC | 01:31 | |
| *** sdake_ has joined #openstack-infra | 01:34 | |
| *** sdake has quit IRC | 01:36 | |
| *** ajmiller has joined #openstack-infra | 01:43 | |
| *** ajmiller_ has joined #openstack-infra | 01:44 | |
| *** BobBall_1WOL has joined #openstack-infra | 01:47 | |
| *** Jeffrey4l has joined #openstack-infra | 01:47 | |
| *** ajmiller has quit IRC | 01:47 | |
| *** BobBall_AWOL has quit IRC | 01:50 | |
| *** sdake_ has quit IRC | 01:53 | |
| *** ajmiller_ has quit IRC | 01:56 | |
| *** zhurong has joined #openstack-infra | 01:56 | |
| *** ajmiller_ has joined #openstack-infra | 01:56 | |
| *** achanda has joined #openstack-infra | 01:58 | |
| *** baoli has quit IRC | 02:01 | |
| *** baoli has joined #openstack-infra | 02:02 | |
| *** ddieterly has quit IRC | 02:03 | |
| *** fedexo has joined #openstack-infra | 02:04 | |
| *** chlong has joined #openstack-infra | 02:09 | |
| *** thorst has joined #openstack-infra | 02:12 | |
| *** tzn has joined #openstack-infra | 02:13 | |
| *** tzn has quit IRC | 02:17 | |
| *** thorst has quit IRC | 02:19 | |
| *** baoli has quit IRC | 02:26 | |
| *** sputnik13 has joined #openstack-infra | 02:29 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Update bashate to 0.4.0 https://review.openstack.org/236815 | 02:30 |
| *** sdake has joined #openstack-infra | 02:31 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Rework functional test runner https://review.openstack.org/276563 | 02:35 |
| *** baoli has joined #openstack-infra | 02:35 | |
| *** ddieterly has joined #openstack-infra | 02:36 | |
| *** dkranz has quit IRC | 02:39 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Enable a fake statsd server for devstack testing https://review.openstack.org/226635 | 02:40 |
| *** baoli has quit IRC | 02:45 | |
| *** baoli has joined #openstack-infra | 02:47 | |
| *** sflanigan is now known as sflanigan_afk | 02:48 | |
| *** sheel has joined #openstack-infra | 02:49 | |
| *** markvoelker has joined #openstack-infra | 02:56 | |
| *** yamamoto_ has joined #openstack-infra | 02:59 | |
| *** MarkAtwood has quit IRC | 02:59 | |
| *** MarkAtwood has joined #openstack-infra | 03:00 | |
| *** ajmiller_ has quit IRC | 03:00 | |
| *** markvoelker has quit IRC | 03:00 | |
| *** rguillebert has quit IRC | 03:07 | |
| *** zz_dimtruck is now known as dimtruck | 03:13 | |
| *** sflanigan_afk is now known as sflanigan | 03:13 | |
| *** zhurong has quit IRC | 03:15 | |
| *** zhurong has joined #openstack-infra | 03:16 | |
| *** thorst has joined #openstack-infra | 03:18 | |
| *** thorst has quit IRC | 03:25 | |
| openstackgerrit | Merged openstack-infra/project-config: Adding Check Requirements Zuul check for Monasca Client https://review.openstack.org/279959 | 03:27 |
| *** baoli has quit IRC | 03:28 | |
| *** baoli has joined #openstack-infra | 03:35 | |
| *** coolsvap|away is now known as coolsvap | 03:36 | |
| *** zhurong has quit IRC | 03:40 | |
| *** zhurong has joined #openstack-infra | 03:40 | |
| *** baoli has quit IRC | 03:48 | |
| *** fedexo has quit IRC | 03:50 | |
| *** cfarquhar has quit IRC | 03:52 | |
| *** briancline has joined #openstack-infra | 03:53 | |
| *** cfarquhar has joined #openstack-infra | 03:54 | |
| *** cfarquhar has quit IRC | 03:54 | |
| *** cfarquhar has joined #openstack-infra | 03:54 | |
| *** akshai has joined #openstack-infra | 03:58 | |
| *** coolsvap is now known as coolsvap|away | 03:59 | |
| *** akshai_ has joined #openstack-infra | 04:00 | |
| *** KanagarajM has joined #openstack-infra | 04:01 | |
| *** hichihara has joined #openstack-infra | 04:01 | |
| *** baoli has joined #openstack-infra | 04:01 | |
| *** akshai has quit IRC | 04:03 | |
| *** ddieterly has quit IRC | 04:04 | |
| *** moravec has quit IRC | 04:07 | |
| *** coolsvap|away is now known as coolsvap | 04:13 | |
| *** Sukhdev has joined #openstack-infra | 04:15 | |
| *** akshai_ has quit IRC | 04:18 | |
| *** thorst has joined #openstack-infra | 04:22 | |
| *** sputnik13 has quit IRC | 04:23 | |
| *** baoli has quit IRC | 04:26 | |
| *** dims has joined #openstack-infra | 04:28 | |
| *** thorst has quit IRC | 04:30 | |
| *** dims has quit IRC | 04:30 | |
| *** baoli has joined #openstack-infra | 04:34 | |
| *** baoli has quit IRC | 04:38 | |
| *** dimtruck is now known as zz_dimtruck | 04:39 | |
| *** salv-orlando has joined #openstack-infra | 04:44 | |
| *** salv-orlando has quit IRC | 04:49 | |
| *** markvoelker has joined #openstack-infra | 04:56 | |
| *** sheel has quit IRC | 04:57 | |
| *** sdake has quit IRC | 04:57 | |
| *** markvoelker has quit IRC | 05:00 | |
| *** christiana has quit IRC | 05:01 | |
| *** pedroalvarez has quit IRC | 05:02 | |
| *** [1]Thelo has joined #openstack-infra | 05:02 | |
| *** dabukalam has quit IRC | 05:02 | |
| *** christiana has joined #openstack-infra | 05:02 | |
| *** pedroalvarez has joined #openstack-infra | 05:03 | |
| *** Thelo has quit IRC | 05:04 | |
| *** [1]Thelo is now known as Thelo | 05:04 | |
| *** ddieterly has joined #openstack-infra | 05:05 | |
| *** ddieterly has quit IRC | 05:11 | |
| *** david-lyle has quit IRC | 05:21 | |
| *** thorst has joined #openstack-infra | 05:28 | |
| oanson | mordred, jeblair, thanks for your help. | 05:31 |
| *** achanda has quit IRC | 05:32 | |
| *** zhurong has quit IRC | 05:34 | |
| *** thorst has quit IRC | 05:34 | |
| *** bpokorny has joined #openstack-infra | 05:36 | |
| *** hdd has quit IRC | 05:40 | |
| *** achanda has joined #openstack-infra | 05:41 | |
| *** tzn has joined #openstack-infra | 05:45 | |
| *** binbincong has joined #openstack-infra | 05:45 | |
| *** sam_wan has joined #openstack-infra | 05:47 | |
| *** zhurong has joined #openstack-infra | 05:47 | |
| *** bpokorny has quit IRC | 05:47 | |
| *** abregman has joined #openstack-infra | 05:49 | |
| *** tzn has quit IRC | 05:50 | |
| *** julim has quit IRC | 05:52 | |
| *** _nadya_ has joined #openstack-infra | 06:04 | |
| *** bpokorny has joined #openstack-infra | 06:07 | |
| *** jaosorior has joined #openstack-infra | 06:07 | |
| *** oomichi has joined #openstack-infra | 06:11 | |
| *** amotoki has joined #openstack-infra | 06:22 | |
| *** sdake has joined #openstack-infra | 06:25 | |
| *** otherwiseguy has quit IRC | 06:25 | |
| *** ddieterly has joined #openstack-infra | 06:28 | |
| *** salv-orlando has joined #openstack-infra | 06:30 | |
| *** otherwiseguy has joined #openstack-infra | 06:31 | |
| *** thorst has joined #openstack-infra | 06:31 | |
| *** ddieterly has quit IRC | 06:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/279926 | 06:35 |
| *** thorst has quit IRC | 06:39 | |
| *** sdake has quit IRC | 06:39 | |
| *** _nadya_ has quit IRC | 06:40 | |
| *** smanjara has joined #openstack-infra | 06:43 | |
| *** vgridnev has joined #openstack-infra | 06:45 | |
| *** _nadya_ has joined #openstack-infra | 06:47 | |
| *** sshnaidm has joined #openstack-infra | 06:50 | |
| *** _nadya_ has quit IRC | 06:51 | |
| *** markvoelker has joined #openstack-infra | 06:57 | |
| *** achanda has quit IRC | 06:58 | |
| *** bpokorny has quit IRC | 06:58 | |
| *** markvoelker has quit IRC | 07:02 | |
| *** aarefiev has quit IRC | 07:03 | |
| *** rubasov has quit IRC | 07:03 | |
| *** dolpher has quit IRC | 07:03 | |
| *** markvan has quit IRC | 07:03 | |
| *** ams_ has quit IRC | 07:03 | |
| *** cbader has quit IRC | 07:03 | |
| *** jcooley has quit IRC | 07:03 | |
| *** Zara has quit IRC | 07:03 | |
| *** nijaba has quit IRC | 07:03 | |
| *** jdurgin has quit IRC | 07:03 | |
| *** mordred has quit IRC | 07:03 | |
| *** xwizard__ has quit IRC | 07:03 | |
| *** ajo has quit IRC | 07:03 | |
| *** sileht has quit IRC | 07:03 | |
| *** ctlaugh has quit IRC | 07:03 | |
| *** sbadia has quit IRC | 07:03 | |
| *** gmann has quit IRC | 07:03 | |
| *** dulek has quit IRC | 07:03 | |
| *** krotscheck has quit IRC | 07:03 | |
| *** nihilifer has quit IRC | 07:03 | |
| *** toabctl has quit IRC | 07:03 | |
| *** skraynev has quit IRC | 07:03 | |
| *** frickler has quit IRC | 07:03 | |
| *** SergeyLukjanov has quit IRC | 07:03 | |
| *** bexelbie has quit IRC | 07:03 | |
| *** Kennan_Vacation has quit IRC | 07:03 | |
| *** _diana_ has quit IRC | 07:03 | |
| *** bradm has quit IRC | 07:03 | |
| *** Shrews has quit IRC | 07:03 | |
| *** andreaf has quit IRC | 07:03 | |
| *** mika has quit IRC | 07:03 | |
| *** cschwede has quit IRC | 07:03 | |
| *** psanchez has quit IRC | 07:03 | |
| *** alazarev has quit IRC | 07:03 | |
| *** mancdaz has quit IRC | 07:03 | |
| *** ajo has joined #openstack-infra | 07:03 | |
| *** markvan has joined #openstack-infra | 07:03 | |
| *** mika has joined #openstack-infra | 07:04 | |
| *** mordred has joined #openstack-infra | 07:04 | |
| *** Zara has joined #openstack-infra | 07:04 | |
| *** dolpher has joined #openstack-infra | 07:04 | |
| *** bradm has joined #openstack-infra | 07:04 | |
| *** jcooley has joined #openstack-infra | 07:04 | |
| *** bexelbie has joined #openstack-infra | 07:04 | |
| *** toabctl has joined #openstack-infra | 07:04 | |
| *** cbader has joined #openstack-infra | 07:04 | |
| *** jdurgin has joined #openstack-infra | 07:04 | |
| *** skraynev has joined #openstack-infra | 07:04 | |
| *** frickler has joined #openstack-infra | 07:04 | |
| *** andreaf has joined #openstack-infra | 07:04 | |
| *** gmann has joined #openstack-infra | 07:04 | |
| *** _diana_ has joined #openstack-infra | 07:04 | |
| *** nihilifer has joined #openstack-infra | 07:04 | |
| *** ctlaugh has joined #openstack-infra | 07:04 | |
| *** alazarev has joined #openstack-infra | 07:04 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:04 | |
| *** Kennan_Vacation has joined #openstack-infra | 07:04 | |
| *** Shrews has joined #openstack-infra | 07:04 | |
| *** sileht has joined #openstack-infra | 07:04 | |
| *** rubasov has joined #openstack-infra | 07:04 | |
| *** gildub has quit IRC | 07:04 | |
| *** xwizard_ has joined #openstack-infra | 07:04 | |
| *** krotscheck has joined #openstack-infra | 07:04 | |
| *** nijaba has joined #openstack-infra | 07:05 | |
| *** cschwede has joined #openstack-infra | 07:05 | |
| *** ams_ has joined #openstack-infra | 07:05 | |
| *** sbadia has joined #openstack-infra | 07:06 | |
| *** sshnaidm has quit IRC | 07:06 | |
| *** aarefiev has joined #openstack-infra | 07:07 | |
| *** wolverineav has quit IRC | 07:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Use pep8 instead of linters for kolla* https://review.openstack.org/278622 | 07:09 |
| tojuvone | Very small patch to fix developers.rst if somebody could merge: https://review.openstack.org/279495 | 07:11 |
| *** wolverineav has joined #openstack-infra | 07:11 | |
| *** wolverineav has quit IRC | 07:16 | |
| AJaeger | infra-root, we have again UNSTABLE jobs uploading to logs like https://jenkins05.openstack.org/job/gate-cinder-python27/2875/console | 07:16 |
| *** wolverineav has joined #openstack-infra | 07:16 | |
| AJaeger | "[SCP] Trying to create /srv/static/logs/04/272404/5" | 07:16 |
| AJaeger | " ERROR: Failed to upload files" | 07:16 |
| AJaeger | "2: No such file" | 07:16 |
| *** hrubi has joined #openstack-infra | 07:17 | |
| AJaeger | The same change had previously UNSTABLE jobs, see https://review.openstack.org/#/c/272404/5 | 07:18 |
| *** cody-somerville has joined #openstack-infra | 07:19 | |
| *** cody-somerville has joined #openstack-infra | 07:19 | |
| *** nmagnezi has joined #openstack-infra | 07:22 | |
| *** shashank_hegde has quit IRC | 07:23 | |
| *** shashank_hegde has joined #openstack-infra | 07:24 | |
| *** ddieterly has joined #openstack-infra | 07:29 | |
| yolanda | good morning | 07:29 |
| yolanda | AJaeger, so two issues? jenkins01 down and jenkins05 failing with disk? | 07:29 |
| *** chlong has quit IRC | 07:30 | |
| *** ianw has quit IRC | 07:30 | |
| *** ianw has joined #openstack-infra | 07:30 | |
| *** ddieterly has quit IRC | 07:33 | |
| *** achanda has joined #openstack-infra | 07:34 | |
| *** tzn has joined #openstack-infra | 07:34 | |
| AJaeger | yolanda: jenkins01 and 05 have been fixed by fungi - haven't they? | 07:35 |
| yolanda | AJaeger, so for some reason, directory 272404 has root perms | 07:35 |
| AJaeger | yolanda: good morning. The one issues I see is on logs | 07:35 |
| *** scheuran has joined #openstack-infra | 07:35 | |
| yolanda | that's why subdir cannot be created | 07:35 |
| AJaeger | can you change permissions manually, please? | 07:35 |
| yolanda | i don't know how that directory got root permissions | 07:35 |
| *** |-paul-| has joined #openstack-infra | 07:35 | |
| AJaeger | might be something went wrong during the repairing of the volume - or the copying of files around after that | 07:36 |
| yolanda | ok i fixed that directory manually | 07:36 |
| yolanda | going to search for dirs with root perms | 07:36 |
| *** thorst has joined #openstack-infra | 07:37 | |
| AJaeger | clarkb was doing this also last week, not sure what the outcome was | 07:37 |
| *** tzn has quit IRC | 07:39 | |
| yolanda | at the moment i cannot see more directories owned by root in logs | 07:43 |
| AJaeger | ok, I'll recheck that one once it's done - and hope that was it. Thanks! | 07:44 |
| *** thorst has quit IRC | 07:45 | |
| *** salv-orlando has quit IRC | 07:47 | |
| *** _nadya_ has joined #openstack-infra | 07:47 | |
| *** sflanigan has quit IRC | 07:48 | |
| *** _nadya_ has quit IRC | 07:52 | |
| AJaeger | yolanda: first test of that change uploaded fine! | 07:54 |
| yolanda | great | 07:55 |
| openstackgerrit | Merged openstack-infra/project-config: Use pep8 instead of linters for terracotta https://review.openstack.org/278645 | 07:58 |
| *** ildikov_ has quit IRC | 07:58 | |
| *** bmwiedemann has left #openstack-infra | 08:07 | |
| *** jaosorior has quit IRC | 08:09 | |
| *** dtardivel has joined #openstack-infra | 08:10 | |
| *** sshnaidm has joined #openstack-infra | 08:13 | |
| *** dizquierdo has joined #openstack-infra | 08:14 | |
| *** jtomasek has joined #openstack-infra | 08:14 | |
| sam_wan | hello, could someone help me with a ' nodepool image-build xx' error: http://paste.openstack.org/show/486946/ | 08:18 |
| *** psanchez has joined #openstack-infra | 08:18 | |
| *** fbo_ is now known as fbo | 08:19 | |
| *** shashank_hegde has quit IRC | 08:20 | |
| *** hashar has joined #openstack-infra | 08:21 | |
| *** oomichi has quit IRC | 08:21 | |
| AJaeger | yolanda: could you review this week infra-manual as well, please? Thanks for your project-config reviews! | 08:24 |
| yolanda | hi AJaeger, sure | 08:24 |
| yolanda | i'm normally not messing too much with infra-manual , but i'll do my best | 08:24 |
| AJaeger | sam_wan: it cannot find your pypi mirror | 08:25 |
| openstackgerrit | Merged openstack-infra/project-config: Move gate-networking-midonet-v2-dsvm-rally-nv out of experimental https://review.openstack.org/279373 | 08:25 |
| sam_wan | Hi Ajaeger, how should I fix? | 08:25 |
| sam_wan | I can pip install | 08:26 |
| AJaeger | sam_wan: can't help further on that, you have to check how the files are setup inside nodepool build | 08:26 |
| *** achanda has quit IRC | 08:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove linters from jenkins/jobs/python-jobs https://review.openstack.org/279781 | 08:27 |
| sam_wan | thanks AJaeger | 08:27 |
| *** mikelk has joined #openstack-infra | 08:27 | |
| *** achanda has joined #openstack-infra | 08:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove {name} jobs from devstack-jobs https://review.openstack.org/278073 | 08:28 |
| *** YorikSar has quit IRC | 08:28 | |
| *** jlanoux has joined #openstack-infra | 08:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove unused templates from devstack-gate.yaml https://review.openstack.org/278061 | 08:29 |
| openstackgerrit | Merged openstack-infra/project-config: Sort devstack-jobs list https://review.openstack.org/278084 | 08:29 |
| *** dingyichen has quit IRC | 08:29 | |
| *** ddieterly has joined #openstack-infra | 08:30 | |
| *** Jeffrey4l has quit IRC | 08:30 | |
| *** zeih has joined #openstack-infra | 08:31 | |
| *** Sukhdev has quit IRC | 08:31 | |
| *** Hal has joined #openstack-infra | 08:33 | |
| *** jaosorior has joined #openstack-infra | 08:33 | |
| *** Hal is now known as Guest97839 | 08:33 | |
| *** rcernin has joined #openstack-infra | 08:34 | |
| *** ildikov_ has joined #openstack-infra | 08:35 | |
| *** ddieterly has quit IRC | 08:35 | |
| *** Sukhdev has joined #openstack-infra | 08:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Import puppet-pacemaker from github/redhat-openstack https://review.openstack.org/279698 | 08:39 |
| openstackgerrit | Merged openstack-infra/project-config: Adds new project Python-namosclient https://review.openstack.org/279399 | 08:41 |
| *** achanda has quit IRC | 08:42 | |
| *** Jeffrey4l has joined #openstack-infra | 08:42 | |
| *** thorst has joined #openstack-infra | 08:43 | |
| *** Sukhdev has quit IRC | 08:43 | |
| *** wolverineav has quit IRC | 08:45 | |
| *** paramite has joined #openstack-infra | 08:47 | |
| paramite | Greetings, is there anybody who can help me with https://answers.launchpad.net/openstack-ci/+question/283852 | 08:47 |
| paramite | ? | 08:47 |
| *** _nadya_ has joined #openstack-infra | 08:48 | |
| openstackgerrit | Merged openstack-infra/project-config: create openstack/service-types-authority https://review.openstack.org/278612 | 08:49 |
| *** thorst has quit IRC | 08:49 | |
| *** arxcruz has joined #openstack-infra | 08:49 | |
| openstackgerrit | Merged openstack-infra/project-config: Add neutron tempest run using neutron-lib master to neutron exp https://review.openstack.org/278825 | 08:52 |
| openstackgerrit | Merged openstack-infra/project-config: Add javascript jobs to stackviz https://review.openstack.org/279254 | 08:52 |
| *** _nadya_ has quit IRC | 08:53 | |
| mordred | hi paramite ... sorry, we don't really use launchpad answers at all. however, yes, deleting branches is a thing we can help with | 08:55 |
| mordred | paramite: so you want all the branches that don't start with stable/ and also stable/juno to go away? | 08:56 |
| *** sheel has joined #openstack-infra | 08:56 | |
| openstackgerrit | Merged openstack-infra/project-config: Add job with functional tests for refstack https://review.openstack.org/279536 | 08:56 |
| *** sshnaidm has quit IRC | 08:56 | |
| mordred | morning yolanda , AJaeger - how are things? | 08:57 |
| *** zhurong has quit IRC | 08:57 | |
| yolanda | hi mordred, what are you doing in that timezone? are you in europe? | 08:57 |
| mordred | yolanda: almost - I'm about 30 minutes from landing in Manchester | 08:57 |
| *** changzhi has joined #openstack-infra | 08:58 | |
| yolanda | so we will have you around this week :) | 08:58 |
| *** markvoelker has joined #openstack-infra | 08:58 | |
| * yolanda has a todo list for mordred :) | 08:58 | |
| mordred | uhoh | 08:58 |
| *** sshnaidm has joined #openstack-infra | 08:58 | |
| *** salv-orlando has joined #openstack-infra | 08:59 | |
| *** salv-orlando has quit IRC | 08:59 | |
| yolanda | mordred, so i am testing infra-cloud east, it works quite good | 08:59 |
| yolanda | we are going to connect to some test nodepool and test the load | 08:59 |
| openstackgerrit | Merged openstack-infra/project-config: Add openstack/broadview-lib project https://review.openstack.org/275873 | 08:59 |
| *** salv-orlando has joined #openstack-infra | 08:59 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Add information about enabling translations https://review.openstack.org/273759 | 09:00 |
| mordred | yolanda: excellent! | 09:01 |
| mordred | yolanda: that's great news | 09:01 |
| openstackgerrit | Merged openstack-infra/project-config: Move bindep-fallback to non-voting https://review.openstack.org/279161 | 09:01 |
| yolanda | it's working on liberty | 09:01 |
| mordred | ooh. | 09:01 |
| yolanda | mordred, so one thing i really need help with that, is reviews | 09:01 |
| yolanda | there are lots of reviews pending to merge for infra-cloud topic, and it's a mess to propose new patches | 09:01 |
| *** ifarkas has joined #openstack-infra | 09:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Create ansible-role-shade project https://review.openstack.org/278643 | 09:02 |
| mordred | yolanda: oh wow. there's a lot of open patches :) | 09:02 |
| openstackgerrit | Merged openstack-infra/infra-manual: Update Accessing Gerrit over HTTPS section https://review.openstack.org/258286 | 09:02 |
| *** markvoelker has quit IRC | 09:03 | |
| mordred | yolanda: so it looks like we wound up doing a CA and not a self-signed cert? | 09:03 |
| yolanda | yes, and i need to propose the changes i have manually on east, but is becoming difficult | 09:03 |
| paramite | mordred, yes, correct | 09:03 |
| paramite | mordred, thanks a lot | 09:03 |
| *** fhubik has joined #openstack-infra | 09:03 | |
| *** pigmej has left #openstack-infra | 09:03 | |
| yolanda | i have a great mess of cherry picked commits on my simulated puppetmaster | 09:03 |
| mordred | paramite: done. no problem! | 09:03 |
| mordred | yolanda: I'm sure you do :) | 09:03 |
| mordred | paramite: I cannot delete the liberty branch - there are open changes for it still | 09:04 |
| *** dulek has joined #openstack-infra | 09:04 | |
| paramite | mordred, doh ... ok, gonna close them | 09:04 |
| paramite | mordred, done | 09:05 |
| openstackgerrit | Merged openstack-infra/infra-manual: Updates Gerrit IRC Notifications section https://review.openstack.org/266508 | 09:05 |
| *** dizquierdo has quit IRC | 09:06 | |
| *** dkehn has quit IRC | 09:06 | |
| *** fhubik has quit IRC | 09:06 | |
| mordred | yolanda: I see two patches about defining nodes for hpuseast that have jenkins -1 - I'm going to skip them for now | 09:07 |
| *** fhubik has joined #openstack-infra | 09:07 | |
| mordred | paramite: done! | 09:07 |
| *** Odd_Blok1 is now known as Odd_Bloke | 09:07 | |
| yolanda | mordred yes, these ones cannot land because missing deps, and i did not added to the chain properly | 09:07 |
| paramite | mordred, thanks | 09:07 |
| *** placko has joined #openstack-infra | 09:07 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Document deletion of tags and branches https://review.openstack.org/264287 | 09:08 |
| mordred | yolanda: kk. also - we have 2 rabbits? | 09:08 |
| *** dizquierdo has joined #openstack-infra | 09:08 | |
| yolanda | mordred, no.. but we decided to go with rabbit and ssl | 09:08 |
| yolanda | maybe there are overlapping patches? | 09:08 |
| *** dizquierdo has quit IRC | 09:08 | |
| mordred | yolanda: in 260022 there's a neutron rabbit password and a nova rabbit password - I guess maybe that's just two different users? | 09:08 |
| yolanda | mordred ah yes. That's because it creates different accounts for each component | 09:09 |
| *** chlong has joined #openstack-infra | 09:09 | |
| *** Odd_Bloke has quit IRC | 09:10 | |
| *** Odd_Bloke has joined #openstack-infra | 09:10 | |
| AJaeger | paramite: we just merged this change so that you can find documentation for deletion of branches next time ;) https://review.openstack.org/#/c/264287/2/doc/source/creators.rst | 09:10 |
| nibalizer | mordred: yolanda i thought we were going to use self signed certs? | 09:11 |
| nibalizer | that seemed to be the conclusion | 09:11 |
| yolanda | nibalizer yes, and that continues like that | 09:11 |
| yolanda | why do you ask it? | 09:11 |
| *** rossella_s has quit IRC | 09:12 | |
| mordred | yolanda: you have a patch that adds a local CA | 09:13 |
| yolanda | mordred, the local CA will be the cert itself | 09:14 |
| mordred | AHHHH | 09:14 |
| yolanda | in following patchsets, crinkle uses that class to send the CRT file to global trust | 09:14 |
| *** rossella_s has joined #openstack-infra | 09:15 | |
| yolanda | nibalizer, are you going to Manchester as well? | 09:16 |
| *** ajo has quit IRC | 09:16 | |
| *** ajo has joined #openstack-infra | 09:16 | |
| *** markvan has quit IRC | 09:16 | |
| *** markvan has joined #openstack-infra | 09:16 | |
| *** dolpher has quit IRC | 09:16 | |
| *** dolpher has joined #openstack-infra | 09:16 | |
| *** ams_ has quit IRC | 09:16 | |
| *** ams_ has joined #openstack-infra | 09:16 | |
| *** cschwede has quit IRC | 09:16 | |
| *** cschwede has joined #openstack-infra | 09:16 | |
| *** nmagnezi has quit IRC | 09:16 | |
| *** nmagnezi has joined #openstack-infra | 09:16 | |
| *** scheuran has quit IRC | 09:16 | |
| *** scheuran has joined #openstack-infra | 09:16 | |
| *** dtardivel has quit IRC | 09:17 | |
| *** dtardivel has joined #openstack-infra | 09:17 | |
| *** psanchez has quit IRC | 09:17 | |
| *** psanchez has joined #openstack-infra | 09:17 | |
| *** rcernin has quit IRC | 09:17 | |
| *** rcernin has joined #openstack-infra | 09:17 | |
| *** paramite has quit IRC | 09:17 | |
| *** paramite has joined #openstack-infra | 09:17 | |
| *** sheel has quit IRC | 09:17 | |
| *** sheel has joined #openstack-infra | 09:17 | |
| *** sshnaidm has quit IRC | 09:17 | |
| *** sshnaidm has joined #openstack-infra | 09:17 | |
| *** fhubik has quit IRC | 09:17 | |
| *** fhubik has joined #openstack-infra | 09:17 | |
| *** placko has quit IRC | 09:17 | |
| *** placko has joined #openstack-infra | 09:17 | |
| paramite | AJaeger, thanks! | 09:19 |
| nibalizer | yolanda: nope no manchester for me | 09:23 |
| yolanda | you are just not sleeping :) | 09:24 |
| *** fhubik is now known as fhubik_brb | 09:27 | |
| *** crinkle_ has joined #openstack-infra | 09:27 | |
| *** crinkle has quit IRC | 09:28 | |
| *** teran has quit IRC | 09:28 | |
| *** bapalm has quit IRC | 09:28 | |
| *** Nakato has quit IRC | 09:28 | |
| *** auggy has quit IRC | 09:28 | |
| *** mattymo has quit IRC | 09:28 | |
| *** teran_ has joined #openstack-infra | 09:28 | |
| *** mattymo has joined #openstack-infra | 09:29 | |
| *** Nakato has joined #openstack-infra | 09:29 | |
| *** dkehn has joined #openstack-infra | 09:29 | |
| *** bapalm has joined #openstack-infra | 09:29 | |
| *** auggy has joined #openstack-infra | 09:30 | |
| *** paramite is now known as paramite|afk | 09:30 | |
| *** ddieterly has joined #openstack-infra | 09:30 | |
| *** masayukig has quit IRC | 09:31 | |
| AJaeger | yolanda: thanks for all the reviews. Did puppet update all jenkins for these or did it run into timeouts? | 09:31 |
| AJaeger | If it run into a timeout, what about merging https://review.openstack.org/#/c/278335/ first - and then do a manual update of jenkins? | 09:32 |
| *** paramite|afk is now known as paramite | 09:32 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Set filesystem_store_datadir explicitly in glance https://review.openstack.org/267805 | 09:32 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Fix neutron ssl relationship https://review.openstack.org/275956 | 09:32 |
| *** masayukig has joined #openstack-infra | 09:33 | |
| yolanda | ss2016-02-15 09:14:06,600 p=22947 u=root | fatal: [jenkins01.openstack.org]: FAILED! => {"changed": false, "failed": true, "msg": "/usr/bin/timeout -s 9 30m /usr/bin/puppet apply --detailed-exitcodes --logdest syslog --no-noop /opt/system-config/production/manifests/site.pp failed with return code: 6", "rc": 6} | 09:33 |
| *** _nadya_ has joined #openstack-infra | 09:33 | |
| yolanda | doesn't look good | 09:34 |
| *** hichihara has quit IRC | 09:34 | |
| AJaeger | is that the usual timeout if too many jobs changed? | 09:35 |
| *** ddieterly has quit IRC | 09:35 | |
| yolanda | yes, i think it may be related with jjb | 09:36 |
| *** paramite is now known as paramite|afk | 09:36 | |
| AJaeger | then we need to update manually... | 09:36 |
| *** jordanP has joined #openstack-infra | 09:36 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Use constraints based job to the gate of ofagent https://review.openstack.org/261982 | 09:36 |
| AJaeger | yolanda: change Ic60a3379ca79ad04ae959f3e7acc6ec46d3b32cc should help with that... | 09:37 |
| yolanda | checking | 09:37 |
| *** paramite|afk is now known as paramite | 09:37 | |
| yolanda | ah the play | 09:38 |
| yolanda | yes | 09:38 |
| AJaeger | but before running, have a look at 278335 - we could take that one in as well... | 09:38 |
| yolanda | i'll approve the one you told me and then run the play once it lands | 09:38 |
| AJaeger | ok, thanks, yolanda ! | 09:38 |
| yolanda | np :) | 09:39 |
| *** paramite is now known as paramite|afk | 09:39 | |
| *** fhubik_brb is now known as fhubik | 09:43 | |
| openstackgerrit | Merged openstack-infra/project-config: always make upper-constraints available to python jobs https://review.openstack.org/278335 | 09:45 |
| *** wolverineav has joined #openstack-infra | 09:46 | |
| *** thorst has joined #openstack-infra | 09:47 | |
| yolanda | ok i'll run the play | 09:49 |
| *** yaume has joined #openstack-infra | 09:49 | |
| *** wolverineav has quit IRC | 09:51 | |
| AJaeger | thanks! | 09:52 |
| *** thorst has quit IRC | 09:55 | |
| anteaya | fungi: nice shots of the snow, your view is beautiful | 09:56 |
| anteaya | are cinder volume inodes something we can track in cacti? | 09:57 |
| anteaya | and :( on gerrit online reindexing not actually doing any reindexing | 09:57 |
| anteaya | and morning AJaeger and yolanda | 09:57 |
| yolanda | hi anteaya | 09:57 |
| anteaya | beautiful morning in Manchester | 09:58 |
| *** ganso has joined #openstack-infra | 09:58 | |
| *** tzn has joined #openstack-infra | 09:58 | |
| *** tzn has quit IRC | 09:59 | |
| *** tzn has joined #openstack-infra | 09:59 | |
| *** e0ne has joined #openstack-infra | 09:59 | |
| Zara | maybe we should invite OpenStack people to Manchester more often | 10:00 |
| Zara | it seems to make the weather better. | 10:00 |
| AJaeger | morning, anteaya! Enjoy the operator meeting | 10:00 |
| yolanda | AJaeger, so that play failed in all jenkins masters, and looks that with different errors on all | 10:01 |
| *** rguillebert has joined #openstack-infra | 10:01 | |
| yolanda | i'm going to try again and redirect output to a logfile | 10:01 |
| *** MarkAtwood has quit IRC | 10:02 | |
| AJaeger | yolanda: Oops ;( | 10:02 |
| *** MarkAtwood has joined #openstack-infra | 10:03 | |
| *** max_lobur has joined #openstack-infra | 10:03 | |
| *** salv-orl_ has joined #openstack-infra | 10:03 | |
| yolanda | i saw some problems with requests in one, another failing with python errors... so i'm going to capture that properly | 10:03 |
| *** yamamoto_ has quit IRC | 10:03 | |
| *** MarkAtwo_ has joined #openstack-infra | 10:04 | |
| *** MCoLo__ has joined #openstack-infra | 10:05 | |
| *** salv-orlando has quit IRC | 10:06 | |
| *** MCoLo has quit IRC | 10:06 | |
| *** MarkAtwood has quit IRC | 10:08 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Make header 'quick navigate' and 'search' clearer https://review.openstack.org/278987 | 10:18 |
| *** paramite|afk is now known as paramite | 10:19 | |
| *** paramite is now known as paramite|afk | 10:21 | |
| Zara | anyone know why a patch in review might still say 'needs verified' in a case where it has a +1 from jenkins and a workflow +1? | 10:21 |
| *** paramite|afk is now known as paramite | 10:23 | |
| openstackgerrit | Akihiro Motoki proposed openstack/requirements: Add networking-nec to projects.txt https://review.openstack.org/280137 | 10:24 |
| openstackgerrit | Merged openstack-infra/infra-manual: "Closes-bug" instead of "Fixes bug" https://review.openstack.org/279495 | 10:25 |
| *** sam_wan has quit IRC | 10:26 | |
| AJaeger | Zara: best to discuss on a specific example, do you have one? | 10:27 |
| Zara | AJaeger: yes! :) https://review.openstack.org/#/c/278058/ | 10:28 |
| Zara | AJaeger: nvm, turns out they depended on something else that's in review, but it wasn't obvious | 10:29 |
| Zara | sorry for the noise! | 10:29 |
| AJaeger | Zara, see the "Related Changes" on the upper left | 10:29 |
| AJaeger | Zara: exactly, that's the dependency | 10:29 |
| Zara | heh, thanks :) | 10:30 |
| *** ddieterly has joined #openstack-infra | 10:31 | |
| AJaeger | yolanda: How's jenkins doing? | 10:32 |
| *** marcusvrn_ has joined #openstack-infra | 10:32 | |
| yolanda | the play is still running | 10:32 |
| AJaeger | that's a good sign, isn't it? | 10:33 |
| yolanda | i want to think that it is :) | 10:33 |
| AJaeger | ;) | 10:35 |
| KanagarajM | AJaeger, i am trying to initialize the new repo https://git.openstack.org/openstack/python-namosclient with cookiecutter and 'git review' is failng http://paste.openstack.org/show/486962/ could you please help | 10:35 |
| *** ddieterly has quit IRC | 10:36 | |
| *** paramite is now known as paramite|afk | 10:36 | |
| AJaeger | "We don't know where your gerrit is. Please manually create a remote" | 10:37 |
| AJaeger | Your .gitreview is wrong | 10:38 |
| AJaeger | KanagarajM: the repo creation is not finished yet, wait until it has a .gitreview in it, please | 10:38 |
| AJaeger | KanagarajM: http://git.openstack.org/cgit/openstack/python-namosclient/ - says still "repo is empty" and thus, please wait until there'S content. | 10:39 |
| *** changzhi has quit IRC | 10:39 | |
| AJaeger | KanagarajM: our latest puppet run timeout out, yolanda is on it... | 10:39 |
| KanagarajM | AJaeger, oh, sure.... I will wait. | 10:40 |
| KanagarajM | AJaeger, thanks :) | 10:40 |
| *** ldnunes has joined #openstack-infra | 10:40 | |
| *** notnownikki has joined #openstack-infra | 10:42 | |
| *** rfolco has joined #openstack-infra | 10:42 | |
| *** jlanoux has quit IRC | 10:44 | |
| *** aysyd has joined #openstack-infra | 10:44 | |
| *** Jeffrey4l has quit IRC | 10:45 | |
| AJaeger | yolanda: once the jenkins jobs are done, could you check setup of python-namosclient, please? See above for details | 10:46 |
| *** jcoufal has joined #openstack-infra | 10:46 | |
| *** jlanoux has joined #openstack-infra | 10:48 | |
| *** tpsilva has joined #openstack-infra | 10:49 | |
| *** piet has joined #openstack-infra | 10:51 | |
| *** thorst has joined #openstack-infra | 10:52 | |
| *** rossella_s has quit IRC | 10:55 | |
| *** chlong has quit IRC | 10:56 | |
| *** jsavak has joined #openstack-infra | 10:57 | |
| *** piet has quit IRC | 10:58 | |
| *** piet has joined #openstack-infra | 10:58 | |
| *** rossella_s has joined #openstack-infra | 10:59 | |
| *** markvoelker has joined #openstack-infra | 10:59 | |
| *** thorst has quit IRC | 11:00 | |
| *** jsavak has quit IRC | 11:02 | |
| *** markvoelker has quit IRC | 11:04 | |
| *** fhubik is now known as fhubik_brb | 11:04 | |
| Zara | to all who use storyboard: is the current 'priority' button useful? we want to deprecate global task priorities in favour of individual/team priorities (shown through the order of tasks in worklists and boards), so we plan to remove the button soon. but would like to check that this won't ruin anyone's day. | 11:07 |
| *** Jeffrey4l has joined #openstack-infra | 11:08 | |
| mordred | AJaeger: at ops meetups - request has come from the room to make stable branch devref docs (docs.o.o/dev/nova) exist/more easily findable | 11:08 |
| mordred | AJaeger: mostly just saying this here so that I don't forget | 11:08 |
| *** ihrachys has joined #openstack-infra | 11:09 | |
| *** yamamoto_ has joined #openstack-infra | 11:10 | |
| *** fhubik_brb is now known as fhubik | 11:12 | |
| *** asettle has joined #openstack-infra | 11:12 | |
| *** flepied1 is now known as flepied | 11:13 | |
| *** sfinucan has joined #openstack-infra | 11:16 | |
| *** dizquierdo has joined #openstack-infra | 11:16 | |
| *** piet has quit IRC | 11:17 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add Juju Charms for OpenStack https://review.openstack.org/232705 | 11:18 |
| yolanda | AJaeger, still running | 11:19 |
| *** dgonzalez has joined #openstack-infra | 11:26 | |
| *** yamamoto_ has quit IRC | 11:29 | |
| *** yamamoto_ has joined #openstack-infra | 11:30 | |
| yolanda | AJaeger, play finished but only worked for jenkins02 and 06 | 11:31 |
| yolanda | going to check logs | 11:31 |
| *** ddieterly has joined #openstack-infra | 11:32 | |
| yolanda | cache looks corrupted at jenkins01 | 11:32 |
| jamespage | nibalizer, when you're around, picking up on the review to get the openstack charms up under git/gerrit | 11:33 |
| jamespage | I need some help on understanding how to get the gate jobs setup right | 11:33 |
| *** dizquierdo has quit IRC | 11:33 | |
| *** dizquierdo has joined #openstack-infra | 11:33 | |
| jamespage | every project has tox targets for lint (flake8 + charm linting) and py27 right now | 11:33 |
| *** ildikov_ has quit IRC | 11:34 | |
| *** yamamoto_ has quit IRC | 11:34 | |
| yolanda | also failures of the same type in jenkins05, i'll run manually there, flushing the cache before | 11:35 |
| *** ociuhandu has quit IRC | 11:35 | |
| *** ddieterly has quit IRC | 11:36 | |
| AJaeger | mordred: I think that's up for the nova (etc) team to link to that, they can easily add something on their front page... | 11:37 |
| AJaeger | yolanda: Argh ;( | 11:37 |
| yolanda | most of the failures are for that corruption, on several jenkins masters | 11:38 |
| AJaeger | jhesketh: I'm confused by your comment on 278644, please review again. | 11:39 |
| AJaeger | yolanda: I have no idea about that ;( | 11:39 |
| yolanda | jenkins04 looks faulty jenkins.BadHTTPException: Error communicating with server[https://jenkins04.openstack.org/] | 11:40 |
| *** kushal has joined #openstack-infra | 11:40 | |
| yolanda | i restarted and ran the play | 11:40 |
| AJaeger | jamespage: read the Infra Manual - and perhaps my latest blogs post as well: http://jaegerandi.blogspot.com . | 11:40 |
| yolanda | so we need to wait until that completes, it will take its time | 11:41 |
| AJaeger | jamespage: you only set up Zuul but not Jenkins | 11:41 |
| jamespage | AJaeger, looking | 11:41 |
| AJaeger | jamespage: my last two blog posts - and then there'S http://docs.openstack.org/infra/manual/creators.html | 11:41 |
| *** fhubik is now known as fhubik_brb | 11:42 | |
| *** fhubik_brb is now known as fhubik | 11:42 | |
| *** fhubik is now known as fhubik_brb | 11:42 | |
| yolanda | mm, jenkins04 fails even after restart | 11:43 |
| *** piet has joined #openstack-infra | 11:44 | |
| jamespage | AJaeger, I think the bit I'm missing conceptually was how a gate-{name}-lint check or gate entry actually results in the tox target being executed | 11:44 |
| jamespage | maybe I'm looking for something more explicit than it has to be...? | 11:45 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/infra-ansible: Refactor setup_env.sh to use virtualenvs https://review.openstack.org/280172 | 11:45 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/infra-ansible: Pin ansible package to stable 2.0.0.2 https://review.openstack.org/280173 | 11:47 |
| *** wolverineav has joined #openstack-infra | 11:48 | |
| *** dguitarbite has quit IRC | 11:49 | |
| AJaeger | jamespage: you just need to instantiate the jobs in jenkins/jobs/projects.yaml (use python-jobs job-group) and then add them to zuul for triggering. | 11:51 |
| *** piet has quit IRC | 11:51 | |
| *** piet has joined #openstack-infra | 11:52 | |
| *** wolverineav has quit IRC | 11:52 | |
| yolanda | AJaeger, updating jenkins manually, flushing cache, seems to have done the trick | 11:52 |
| yolanda | but it supposes a long time to finish | 11:53 |
| AJaeger | ;( | 11:53 |
| AJaeger | good that it's a public holiday in the US and Monday - and low load so far... | 11:54 |
| rcarrillocruz | yolanda: are jenkins in ansible inventory? maybe you could do an ansible play against all to clear out the caches? | 11:54 |
| yolanda | rcarrillocruz, there is a play to run jjb there, but failed because the cache was corrupted | 11:55 |
| yolanda | so i had to run with --flush-cache | 11:55 |
| yolanda | seems like a good idea to update that play, to allow that config var | 11:55 |
| rcarrillocruz | ah, so the play is already against all jenkins hosts | 11:55 |
| yolanda | yes | 11:55 |
| rcarrillocruz | yeah, maybe create another play to clear up cache or something | 11:55 |
| yolanda | if the play accepted some "flush_cache" var it will be nice | 11:56 |
| rcarrillocruz | oh yeah | 11:56 |
| rcarrillocruz | let me poke | 11:56 |
| yolanda | same play, just allowing that var | 11:56 |
| rcarrillocruz | i can look at it | 11:56 |
| yolanda | great | 11:56 |
| rcarrillocruz | it will be a bit odd without visibility on root machines, so i'll ping you when i get something so you can test with noop | 11:56 |
| *** sheel has left #openstack-infra | 11:57 | |
| yolanda | i can test yes | 11:57 |
| *** piet has quit IRC | 11:58 | |
| *** thorst has joined #openstack-infra | 11:58 | |
| *** yamamoto has joined #openstack-infra | 11:58 | |
| *** yamamoto has quit IRC | 11:59 | |
| *** yamamoto has joined #openstack-infra | 11:59 | |
| *** piet has joined #openstack-infra | 11:59 | |
| openstackgerrit | Stefan Andres proposed openstack-infra/shade: Fix lookup for network parameter in neutron_available_floating_ips https://review.openstack.org/280178 | 12:00 |
| *** paramite|afk is now known as paramite | 12:01 | |
| openstackgerrit | Merged openstack-infra/zuul: Remove argparse from requirements https://review.openstack.org/270407 | 12:01 |
| *** |-paul-| has quit IRC | 12:01 | |
| *** dguitarbite has joined #openstack-infra | 12:02 | |
| anteaya | fungi: request coming out of my time at the ops meetup talking with odyssey4me, what would it take for us to spin up a vm with our config of gerrit for him to spend some time looking at with the intention of recruiting a java ops person? | 12:02 |
| *** asettle has quit IRC | 12:03 | |
| openstackgerrit | Stefan Andres proposed openstack-infra/shade: Fix lookup for network parameter in neutron_available_floating_ips This change attempts to fix the network parameter in neutron_available_floating_ips: * old behavior ignores network parameter and guesses external net * new behavior takes name or id and s https://review.openstack.org/280178 | 12:03 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add global vars for bifrost in useast https://review.openstack.org/267438 | 12:04 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add force-sync option https://review.openstack.org/280179 | 12:04 |
| *** thorst has quit IRC | 12:05 | |
| *** ociuhandu has joined #openstack-infra | 12:06 | |
| *** rhallisey has joined #openstack-infra | 12:06 | |
| *** piet has quit IRC | 12:08 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add Juju Charms for OpenStack https://review.openstack.org/232705 | 12:08 |
| *** hdd has joined #openstack-infra | 12:09 | |
| *** piet has joined #openstack-infra | 12:10 | |
| *** piet has quit IRC | 12:10 | |
| *** hdd has quit IRC | 12:11 | |
| *** ildikov_ has joined #openstack-infra | 12:11 | |
| *** piet has joined #openstack-infra | 12:11 | |
| *** jcoufal has quit IRC | 12:12 | |
| *** trash has joined #openstack-infra | 12:13 | |
| fungi | anteaya: installing gerrit with our config is pretty trivial (apply the puppet manifest). however a java ops person working on gerrit probably needs to start by learning how to install it | 12:13 |
| trash | Hi, are there any shade (https://github.com/openstack-infra/shade/) developers? Can someone review https://review.openstack.org/#/c/280178/ - this bug prevents ansible being able to select a floating IP pool which is needed for us. | 12:13 |
| fungi | anyway, i'm gone most of today, taking chris to an appointment a few hours away on the mainland. i'll try to check in around 1500 utc but don't expect to be home until 2000 utc or maybe later | 12:14 |
| fungi | weather | 12:14 |
| fungi | 's not so great out here today | 12:14 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add depth option https://review.openstack.org/280184 | 12:14 |
| Zara | reminder to anyone at the ops meetup: the storyboard meetup is in the same city on Wednesday, and we have cake! A limited supply of cake! topics flexible depending on who attends; it's a good way to help shape development! So many exclamation marks! Etherpad here: https://etherpad.openstack.org/p/StoryBoard_Mitaka_Midcycle | 12:14 |
| *** scheuran has quit IRC | 12:16 | |
| *** nmagnezi has quit IRC | 12:16 | |
| *** nmagnezi has joined #openstack-infra | 12:16 | |
| *** jtomasek_ has joined #openstack-infra | 12:17 | |
| *** piet has quit IRC | 12:20 | |
| *** chlong has joined #openstack-infra | 12:20 | |
| *** smanjara has quit IRC | 12:20 | |
| openstackgerrit | Igor Belikov proposed openstack/diskimage-builder: Sanitize distro elements for minimal vs cloud https://review.openstack.org/211859 | 12:20 |
| *** arxcruz has quit IRC | 12:21 | |
| AJaeger | fungi: Safe drive! | 12:21 |
| *** sdake has joined #openstack-infra | 12:21 | |
| *** vponomaryov has joined #openstack-infra | 12:22 | |
| *** fhubik_brb is now known as fhubik | 12:23 | |
| *** dprince has joined #openstack-infra | 12:23 | |
| *** changzhi has joined #openstack-infra | 12:25 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Allow to flush cache on run_jjb plays https://review.openstack.org/280189 | 12:26 |
| rcarrillocruz | yolanda: ^ | 12:27 |
| rcarrillocruz | could you cherry pick and do -C | 12:27 |
| *** openstackgerrit_ has joined #openstack-infra | 12:27 | |
| rcarrillocruz | syntax check passes , but that's not a good way to tell is actually going to work :-) | 12:28 |
| *** asettle has joined #openstack-infra | 12:28 | |
| yolanda | rcarrillocruz, i'll take a look | 12:28 |
| *** paramite is now known as paramite|afk | 12:28 | |
| *** markvoelker has joined #openstack-infra | 12:29 | |
| yolanda | rcarrillocruz, why not reuse same tasks? can we define a var and do something like "jenkins-jobs update {{flush_cache}} --delete-old" ? | 12:30 |
| *** thorst has joined #openstack-infra | 12:30 | |
| rcarrillocruz | hmm, yeah | 12:30 |
| rcarrillocruz | although | 12:30 |
| rcarrillocruz | we would need to check flush_cache is actually yes/no prior to running the task | 12:31 |
| rcarrillocruz | let me work it out | 12:31 |
| *** weshay has joined #openstack-infra | 12:31 | |
| rcarrillocruz | i think there was a way in ansible to not use certain things in command/shell if a var was not defined | 12:32 |
| *** asettle has quit IRC | 12:32 | |
| yolanda | mm, i know the if !defined in template, but don't know about the tasks | 12:32 |
| rcarrillocruz | how about a var that is 'additional_options' | 12:32 |
| rcarrillocruz | and it gets passed unconditonlly | 12:33 |
| rcarrillocruz | ? | 12:33 |
| *** ddieterly has joined #openstack-infra | 12:33 | |
| rcarrillocruz | similar to the extra_kernel_params on bifrost | 12:33 |
| rcarrillocruz | are there more flags that may be interesting for jjb | 12:33 |
| rcarrillocruz | like verbose | 12:33 |
| yolanda | i think it's good, because it will allow to enable debugging, ignore caching, etc | 12:33 |
| rcarrillocruz | hivemind | 12:33 |
| rcarrillocruz | :D | 12:33 |
| rcarrillocruz | sec, i'll push a patchset | 12:34 |
| yolanda | cool | 12:34 |
| *** markvoelker has quit IRC | 12:34 | |
| *** arxcruz has joined #openstack-infra | 12:34 | |
| *** coolsvap is now known as coolsvap|away | 12:36 | |
| *** thorst has quit IRC | 12:36 | |
| *** marcusvrn_ has quit IRC | 12:37 | |
| *** ddieterly has quit IRC | 12:37 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Move search icon https://review.openstack.org/279204 | 12:39 |
| rcarrillocruz | yolanda: | 12:41 |
| rcarrillocruz | http://docs.ansible.com/ansible/playbooks_filters.html#omitting-undefined-variables-and-parameters | 12:41 |
| rcarrillocruz | that's what i was after | 12:41 |
| *** dims_ has joined #openstack-infra | 12:42 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Allow to flush cache on run_jjb plays https://review.openstack.org/280189 | 12:43 |
| *** baoli has joined #openstack-infra | 12:43 | |
| yolanda | ah didn't know about it | 12:44 |
| *** smarcet has joined #openstack-infra | 12:45 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/infra-ansible: Ignore <playbook>.retry files on git https://review.openstack.org/280195 | 12:47 |
| *** scheuran has joined #openstack-infra | 12:47 | |
| *** baoli has quit IRC | 12:47 | |
| *** baoli has joined #openstack-infra | 12:48 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Allow to flush cache on run_jjb plays https://review.openstack.org/280189 | 12:48 |
| *** baoli has quit IRC | 12:48 | |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Add debian-jessie image to nodepool https://review.openstack.org/264726 | 12:48 |
| *** changzhi has quit IRC | 12:49 | |
| *** MarkAtwo_ has quit IRC | 12:49 | |
| *** wolverineav has joined #openstack-infra | 12:49 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Allow to pass additional params on run_jjb plays https://review.openstack.org/280189 | 12:50 |
| onovy | hi, anyone for simple final +A? :) https://review.openstack.org/#/c/278945/ // thanks | 12:51 |
| *** lucasagomes is now known as lucas-hungry | 12:51 | |
| *** baoli has joined #openstack-infra | 12:52 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Enable gate-infra-ansible on check and gate pipelines https://review.openstack.org/280197 | 12:53 |
| *** rlandy has joined #openstack-infra | 12:54 | |
| *** baoli has quit IRC | 12:54 | |
| *** wolverineav has quit IRC | 12:55 | |
| *** thorst has joined #openstack-infra | 12:57 | |
| *** baoli has joined #openstack-infra | 13:00 | |
| *** daemontool has joined #openstack-infra | 13:00 | |
| *** baoli_ has joined #openstack-infra | 13:02 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Wait for pacemaker cluster to settle https://review.openstack.org/279894 | 13:03 |
| *** salv-orl_ has quit IRC | 13:04 | |
| *** zhurong has joined #openstack-infra | 13:04 | |
| *** baoli has quit IRC | 13:05 | |
| *** piet has joined #openstack-infra | 13:06 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 13:09 |
| *** paramite|afk is now known as paramite | 13:10 | |
| *** paramite is now known as paramite|afk | 13:11 | |
| *** baoli_ has quit IRC | 13:14 | |
| *** baoli has joined #openstack-infra | 13:14 | |
| EmilienM | I was wondering why http://git.openstack.org/cgit/openstack/puppet-pacemaker/tree/ was empty, since we imported it from another upstream repo https://review.openstack.org/279698 | 13:15 |
| *** paramite|afk is now known as paramite | 13:16 | |
| *** baoli_ has joined #openstack-infra | 13:17 | |
| *** paramite is now known as paramite|afk | 13:18 | |
| *** baoli has quit IRC | 13:20 | |
| *** jtomasek_ has quit IRC | 13:20 | |
| *** paramite|afk is now known as paramite | 13:20 | |
| *** paramite is now known as paramite|afk | 13:21 | |
| *** paramite|afk is now known as paramite | 13:21 | |
| *** paramite is now known as paramite|afk | 13:21 | |
| *** jkraj has joined #openstack-infra | 13:23 | |
| *** piet has quit IRC | 13:24 | |
| *** jaypipes has joined #openstack-infra | 13:26 | |
| *** piet has joined #openstack-infra | 13:27 | |
| *** dkranz has joined #openstack-infra | 13:29 | |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add force-sync option https://review.openstack.org/280216 | 13:30 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add depth option https://review.openstack.org/280217 | 13:30 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add reset-first option https://review.openstack.org/280218 | 13:30 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add no-tags option https://review.openstack.org/280219 | 13:30 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add trace option https://review.openstack.org/280220 | 13:30 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add show-all-changes option https://review.openstack.org/280221 | 13:30 |
| *** markvoelker has joined #openstack-infra | 13:30 | |
| *** edmondsw has joined #openstack-infra | 13:31 | |
| *** jcoufal has joined #openstack-infra | 13:32 | |
| *** dims_ has quit IRC | 13:33 | |
| *** dims has joined #openstack-infra | 13:33 | |
| *** ddieterly has joined #openstack-infra | 13:34 | |
| *** markvoelker has quit IRC | 13:35 | |
| *** alivigni has joined #openstack-infra | 13:35 | |
| *** yamamoto has quit IRC | 13:35 | |
| *** asettle has joined #openstack-infra | 13:35 | |
| *** yamamoto has joined #openstack-infra | 13:36 | |
| *** ddieterly has quit IRC | 13:39 | |
| *** zhurong has quit IRC | 13:39 | |
| *** salv-orlando has joined #openstack-infra | 13:40 | |
| AJaeger | EmilienM: puppet runs timed out ;( yolanda, all the approved repos are not setup properly I fear | 13:41 |
| EmilienM | ok just to make sure I'm not crazy :) | 13:42 |
| *** ddieterly has joined #openstack-infra | 13:44 | |
| *** tiswanso has joined #openstack-infra | 13:44 | |
| *** doug-fish has joined #openstack-infra | 13:45 | |
| AJaeger | EmilienM: You are not crazy :) | 13:45 |
| *** ddieterl_ has joined #openstack-infra | 13:45 | |
| *** _tzn has joined #openstack-infra | 13:45 | |
| *** yamamoto has quit IRC | 13:46 | |
| *** yamamoto has joined #openstack-infra | 13:46 | |
| *** piet has quit IRC | 13:48 | |
| *** tzn has quit IRC | 13:48 | |
| *** ddieterly has quit IRC | 13:48 | |
| *** paramite|afk is now known as paramite | 13:50 | |
| *** yamamoto has quit IRC | 13:51 | |
| *** yamamoto has joined #openstack-infra | 13:51 | |
| *** burgerk has joined #openstack-infra | 13:52 | |
| *** yamamoto has quit IRC | 13:52 | |
| *** piet has joined #openstack-infra | 13:52 | |
| *** dims has quit IRC | 13:53 | |
| *** yamamoto has joined #openstack-infra | 13:53 | |
| *** lucas-hungry is now known as lucasagomes | 13:53 | |
| *** marcusvrn_ has joined #openstack-infra | 13:54 | |
| *** yamamoto has quit IRC | 13:55 | |
| *** yamamoto has joined #openstack-infra | 13:56 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add Juju Charms for OpenStack https://review.openstack.org/232705 | 13:57 |
| *** dims has joined #openstack-infra | 13:57 | |
| *** pradk has joined #openstack-infra | 13:59 | |
| *** yamamoto has quit IRC | 14:00 | |
| *** dims has quit IRC | 14:00 | |
| anteaya | fungi: safe travels, and thank you | 14:01 |
| *** regXboi has joined #openstack-infra | 14:01 | |
| *** vgridnev has quit IRC | 14:02 | |
| *** dims has joined #openstack-infra | 14:02 | |
| yolanda | it is still running | 14:05 |
| *** kgiusti has joined #openstack-infra | 14:06 | |
| *** vgridnev has joined #openstack-infra | 14:06 | |
| *** abregman is now known as abregman|mtg | 14:06 | |
| AJaeger | wow... | 14:06 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Show Each Task ID in Story Detail View https://review.openstack.org/272667 | 14:08 |
| *** Bl5-bl4d3 has joined #openstack-infra | 14:09 | |
| *** fawadkhaliq has joined #openstack-infra | 14:10 | |
| *** piet has quit IRC | 14:11 | |
| *** claudiub has joined #openstack-infra | 14:11 | |
| *** jaosorior has quit IRC | 14:14 | |
| *** jaosorior has joined #openstack-infra | 14:15 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/279926 | 14:15 |
| *** jaosorior has quit IRC | 14:15 | |
| *** jaosorior has joined #openstack-infra | 14:17 | |
| *** piet has joined #openstack-infra | 14:19 | |
| *** doug-fish has quit IRC | 14:20 | |
| *** fawadk has joined #openstack-infra | 14:21 | |
| *** auggy_ has joined #openstack-infra | 14:21 | |
| *** mtanino has joined #openstack-infra | 14:21 | |
| *** marcusvrn__ has joined #openstack-infra | 14:21 | |
| *** piet has quit IRC | 14:21 | |
| *** arif-ali has joined #openstack-infra | 14:22 | |
| *** piet has joined #openstack-infra | 14:22 | |
| *** mikap has joined #openstack-infra | 14:23 | |
| *** edmondsw_ has joined #openstack-infra | 14:24 | |
| *** piet has quit IRC | 14:24 | |
| *** fawadk has quit IRC | 14:24 | |
| *** fawadkhaliq has quit IRC | 14:24 | |
| *** marcusvrn_ has quit IRC | 14:24 | |
| *** edmondsw has quit IRC | 14:24 | |
| *** ganso has quit IRC | 14:24 | |
| *** vponomaryov has quit IRC | 14:24 | |
| *** auggy has quit IRC | 14:24 | |
| *** Guest97839 has quit IRC | 14:24 | |
| *** Shrews has quit IRC | 14:24 | |
| *** mika has quit IRC | 14:24 | |
| *** sbadia has quit IRC | 14:24 | |
| *** vponomaryov1 has joined #openstack-infra | 14:24 | |
| *** Shrews_ has joined #openstack-infra | 14:24 | |
| *** mikap is now known as mika | 14:24 | |
| *** marcusvrn__ is now known as marcusvrn_ | 14:24 | |
| *** fawadkhaliq has joined #openstack-infra | 14:24 | |
| *** piet has joined #openstack-infra | 14:25 | |
| *** auggy_ is now known as auggy | 14:25 | |
| *** Bl5-bl4d3 has quit IRC | 14:25 | |
| *** Hal has joined #openstack-infra | 14:25 | |
| *** Hal is now known as Guest93242 | 14:26 | |
| *** fawadkhaliq has quit IRC | 14:26 | |
| *** piet has quit IRC | 14:26 | |
| *** fawadkhaliq has joined #openstack-infra | 14:26 | |
| *** sbadia has joined #openstack-infra | 14:27 | |
| *** piet has joined #openstack-infra | 14:27 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Remove Priority Button https://review.openstack.org/280259 | 14:27 |
| *** salv-orlando has quit IRC | 14:27 | |
| *** dims has quit IRC | 14:28 | |
| *** mriedem has joined #openstack-infra | 14:30 | |
| *** rkukura has joined #openstack-infra | 14:30 | |
| *** piet has quit IRC | 14:30 | |
| *** yamamoto has joined #openstack-infra | 14:31 | |
| *** ociuhandu has quit IRC | 14:32 | |
| *** zeih has quit IRC | 14:32 | |
| *** zeih_ has joined #openstack-infra | 14:32 | |
| *** piet has joined #openstack-infra | 14:32 | |
| *** dmsimard has joined #openstack-infra | 14:33 | |
| *** KanagarajM has quit IRC | 14:33 | |
| *** ganso has joined #openstack-infra | 14:35 | |
| pabelanger | AJaeger: EmilienM: Ya noticed that too | 14:35 |
| pabelanger | however puppetdb seems to be invaded with UUID hostnames now :( | 14:35 |
| *** ccrouch has quit IRC | 14:35 | |
| pabelanger | must be infracloud and ansible | 14:35 |
| *** asettle has quit IRC | 14:36 | |
| *** asettle has joined #openstack-infra | 14:36 | |
| *** doug-fish has joined #openstack-infra | 14:36 | |
| *** dims has joined #openstack-infra | 14:36 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add definition for controller hpuseast https://review.openstack.org/260018 | 14:37 |
| openstackgerrit | Thanh Ha proposed openstack-infra/zuul: Add additional zuul.conf information https://review.openstack.org/271868 | 14:38 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add definition for controller hpuseast https://review.openstack.org/260018 | 14:38 |
| *** piet has quit IRC | 14:39 | |
| *** piet has joined #openstack-infra | 14:40 | |
| *** Shrews_ is now known as Shrews | 14:40 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add definition for controller hpuseast https://review.openstack.org/260018 | 14:40 |
| *** doug-fish has quit IRC | 14:40 | |
| AJaeger | pabelanger: yes, your new repos are also affected. | 14:41 |
| *** HeOS has quit IRC | 14:41 | |
| *** HeOS has joined #openstack-infra | 14:41 | |
| AJaeger | pabelanger: some of your changes need a rebase. yolanda approved but Jenkins refused to merge ;( | 14:41 |
| AJaeger | s/jenkins/gerrit/ | 14:41 |
| yolanda | the cache was corrupted in most of jenkins masters | 14:41 |
| yolanda | seems they are finishing now, most of jenkins masters are on deleting not needed jobs now | 14:42 |
| AJaeger | yeah! | 14:42 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for invalid dns hostname bug 1545153 https://review.openstack.org/280264 | 14:42 |
| openstack | bug 1545153 in OpenStack Compute (nova) "BadRequest: Invalid input for dns_name. Name must not start or end with a hyphen." [High,In progress] https://launchpad.net/bugs/1545153 - Assigned to Matthew Treinish (treinish) | 14:42 |
| * AJaeger hopes https://review.openstack.org/278034 will not cause the same trouble.... | 14:43 | |
| *** daemontool has quit IRC | 14:43 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/requirements: Add "virtualbmc" dependency https://review.openstack.org/280265 | 14:43 |
| pabelanger | AJaeger: yolanda: Thanks for the heads up, let me know if I need to do anything else | 14:43 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add definition for compute servers in us east https://review.openstack.org/280266 | 14:44 |
| yolanda | pabelanger, i will let you know when it finishes, so you can check if the jobs are properly refreshed | 14:44 |
| *** jcoufal has quit IRC | 14:45 | |
| yolanda | mordred, thanks for the reviews. One that is really important for us is https://review.openstack.org/#/c/231757/ | 14:46 |
| yolanda | that will unlock several patches, and allow to easily propose new ones | 14:46 |
| *** superdan is now known as dansmith | 14:46 | |
| *** ddieterl_ has quit IRC | 14:47 | |
| *** piet has quit IRC | 14:47 | |
| *** ccrouch has joined #openstack-infra | 14:47 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for invalid dns hostname bug 1545153 https://review.openstack.org/280264 | 14:49 |
| openstack | bug 1545153 in OpenStack Compute (nova) "BadRequest: Invalid input for dns_name. Name must not start or end with a hyphen." [High,In progress] https://launchpad.net/bugs/1545153 - Assigned to Matthew Treinish (treinish) | 14:49 |
| openstackgerrit | Stefan Andres proposed openstack-infra/shade: Fix lookup for network parameter in neutron_available_floating_ips This change attempts to fix the network parameter in neutron_available_floating_ips: * old behavior ignores network parameter and guesses external net * new behavior takes name or id and s https://review.openstack.org/280178 | 14:50 |
| *** eharney has joined #openstack-infra | 14:50 | |
| *** wolverineav has joined #openstack-infra | 14:51 | |
| *** moravec has joined #openstack-infra | 14:52 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:52 | |
| *** Jeffrey4l has quit IRC | 14:53 | |
| *** piet has joined #openstack-infra | 14:53 | |
| openstackgerrit | Ian Cordasco proposed openstack-dev/pbr: Include wsgi_scripts in generated wheels https://review.openstack.org/280270 | 14:53 |
| yolanda | pabelanger, can you verify jenkins03 or jenkins04 ? | 14:54 |
| *** ddieterly has joined #openstack-infra | 14:54 | |
| *** placko has quit IRC | 14:54 | |
| *** moravec has quit IRC | 14:55 | |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/project-config: Added new 'fuel-plugin-tacker' repository https://review.openstack.org/280272 | 14:56 |
| *** wolverineav has quit IRC | 14:56 | |
| *** piet has quit IRC | 14:57 | |
| *** rossella_s has quit IRC | 14:57 | |
| *** piet has joined #openstack-infra | 14:57 | |
| pabelanger | yolanda: sure, 1 sec | 14:58 |
| *** rossella_s has joined #openstack-infra | 14:58 | |
| *** arxcruz has quit IRC | 14:58 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Allow to pass additional params on run_jjb plays https://review.openstack.org/280189 | 14:58 |
| pabelanger | yolanda: Yup, jobs as expected for both | 14:59 |
| yolanda | nice | 14:59 |
| *** dims has quit IRC | 14:59 | |
| yolanda | so we need to wait for the others to finish | 14:59 |
| *** zz_dimtruck is now known as dimtruck | 15:02 | |
| *** sdake has quit IRC | 15:02 | |
| *** nelsnelson has quit IRC | 15:02 | |
| *** piet has quit IRC | 15:03 | |
| *** [1]Thelo has joined #openstack-infra | 15:03 | |
| mordred | yolanda: I'm at a conference at the moment - hard to review that - I'll get it on the train in a bit | 15:04 |
| fungi | around from a lobby for a bit if there are any emergencies | 15:04 |
| yolanda | mordred ok, thanks anyway for all the help | 15:04 |
| *** pblaho has joined #openstack-infra | 15:05 | |
| *** Thelo has quit IRC | 15:05 | |
| *** [1]Thelo is now known as Thelo | 15:05 | |
| fungi | maybe i'll pretend i'm taking presidents' day as a holiday | 15:05 |
| *** paramite is now known as paramite|afk | 15:06 | |
| *** piet has joined #openstack-infra | 15:06 | |
| *** abregman|mtg is now known as abregman | 15:07 | |
| *** paramite|afk is now known as paramite | 15:07 | |
| AJaeger | fungi, seems best help right now is to unblock the infra-cloud with getting a large patch in: 231757 | 15:08 |
| *** doug-fish has joined #openstack-infra | 15:08 | |
| AJaeger | fungi: Or enjoy president's day ;) | 15:09 |
| *** sorantis has joined #openstack-infra | 15:09 | |
| *** nelsnelson has joined #openstack-infra | 15:09 | |
| fungi | AJaeger: well, i'm attending a meeting from a waiting room, but i'll check that change out if i end up here longer than anticipated | 15:09 |
| fungi | so thanks for the heads up | 15:09 |
| *** fawadkhaliq has quit IRC | 15:10 | |
| sorantis | Hi, I was wondering if you guys can help me out with the issue I’m having with Jenking requirements job: https://review.openstack.org/#/c/280260/ | 15:10 |
| AJaeger | you're multitasking too much, fungi! | 15:10 |
| *** dims has joined #openstack-infra | 15:10 | |
| sorantis | It’s says: Requirements file has duplicate entries for package sqlalchemy : [Requirement(package=u'SQLAlchemy', location='', specifiers='<1.1.0,>=0.9.9', markers=u'', comment='', extras=frozenset([])), Requirement(package=u'SQLAlchemy', location='', specifiers='<1.1.0,>=0.9.9', markers=u'', comment='', extras=frozenset([]))]. | 15:10 |
| *** BobBall_1WOL is now known as BobBall | 15:10 | |
| *** esikachev has joined #openstack-infra | 15:11 | |
| *** vponomaryov1 is now known as vponomaryov | 15:11 | |
| *** fawadkhaliq has joined #openstack-infra | 15:11 | |
| sorantis | but I have removed the duplicate entry from requirements.txt | 15:11 |
| *** dimtruck is now known as zz_dimtruck | 15:12 | |
| *** arxcruz has joined #openstack-infra | 15:12 | |
| *** yamamoto has quit IRC | 15:12 | |
| *** jkraj has quit IRC | 15:13 | |
| yolanda | jenkins05 finished as well | 15:13 |
| *** kozhukalov has joined #openstack-infra | 15:13 | |
| yolanda | jenkins01 is the one already building | 15:13 |
| *** doug-fish has quit IRC | 15:13 | |
| kozhukalov | guys, could you please review this https://review.openstack.org/#/c/279074/ | 15:15 |
| kozhukalov | thanks in advance | 15:15 |
| pabelanger | sorantis: See https://github.com/openstack/requirements#enforcement-in-projects for more information | 15:16 |
| *** pblaho has quit IRC | 15:17 | |
| *** rkukura has quit IRC | 15:17 | |
| sorantis | pabelanger: I have seen this, but how is related to duplicate entries, when no duplicate is present in the requirements.txt? | 15:18 |
| pabelanger | sorantis: the duplicated entries are from openstack/requirements | 15:19 |
| pabelanger | the job compares your project requirements with global | 15:19 |
| AJaeger | kozhukalov: how does your repo relate to //lists.openstack.org/pipermail/openstack-dev/2016-February/085636.html ? | 15:19 |
| *** daemontool has joined #openstack-infra | 15:20 | |
| sorantis | pabelanger: so am I supposed to remove completely the sqlalchemy dependency from my requirements.txt? | 15:20 |
| *** nmagnezi has quit IRC | 15:20 | |
| pabelanger | sorantis: is your project using openstack/requirements? | 15:21 |
| pabelanger | are you syncing from it? | 15:21 |
| sorantis | no | 15:21 |
| pabelanger | then why run the gate job? | 15:21 |
| esikachev | guys. i am need review https://review.openstack.org/#/c/279474/ | 15:21 |
| AJaeger | pabelanger: check-requirements is used for kingbird | 15:21 |
| pabelanger | AJaeger: right, but if they are not syncing to global requirements it doesn't make sense righT? | 15:23 |
| *** zz_dimtruck is now known as dimtruck | 15:23 | |
| pabelanger | Not sure if they should or shouldn't but that is why the job is failing. requirements are out of sync | 15:23 |
| pabelanger | duplicate sqalalchemy entries | 15:24 |
| *** kzaitsev_mb has joined #openstack-infra | 15:24 | |
| *** vkmc has joined #openstack-infra | 15:24 | |
| *** jcoufal has joined #openstack-infra | 15:24 | |
| *** ruhe2 has joined #openstack-infra | 15:24 | |
| *** vkmc has left #openstack-infra | 15:24 | |
| *** julim has joined #openstack-infra | 15:25 | |
| *** ruhe has quit IRC | 15:25 | |
| *** ruhe2 is now known as ruhe | 15:25 | |
| AJaeger | pabelanger: I think they had broken requirements, setup the job - and now need to cleanup... | 15:25 |
| AJaeger | They do sync with requirements if they use the check-requirements template | 15:25 |
| sorantis | ok, I’ve just removed the sqlalchemy entry. the job is still failing with the same duplicate entries error | 15:25 |
| *** doug-fish has joined #openstack-infra | 15:25 | |
| AJaeger | but that is only run if requirements is changed. | 15:25 |
| *** piet has quit IRC | 15:26 | |
| *** PsionTheory has joined #openstack-infra | 15:26 | |
| sorantis | https://review.openstack.org/#/c/280260/5/requirements.txt the base does have duplicates, but I’ve removed them in the commit | 15:26 |
| pabelanger | AJaeger: Hmm, reading https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml I don't see kingbird in openstack/requirements | 15:26 |
| AJaeger | sorantis: see http://logs.openstack.org/60/280260/5/check/gate-kingbird-requirements/4eafd35/console.html#_2016-02-15_15_24_03_760 | 15:27 |
| AJaeger | It checks master first | 15:27 |
| sorantis | yes, master does contains the duplicates | 15:28 |
| AJaeger | so, you need to disable check-requirements, fix your requirements (manual sync with requirements repo), then enable again. | 15:28 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/infra-ansible: Refactor setup_env.sh to use virtualenvs https://review.openstack.org/280172 | 15:28 |
| AJaeger | pabelanger: It needs t obe in requirements repo, file projects.txt to get synced from requirements | 15:28 |
| *** rkukura has joined #openstack-infra | 15:29 | |
| sorantis | AJaeger: so I need to do this change in https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml, right? | 15:29 |
| AJaeger | pabelanger, sorantis: kingbird has only half of the requirments setup: The check that no incompatbile requirements can go in. | 15:29 |
| AJaeger | But requirements repo is not setup to push updates to kingbird | 15:29 |
| AJaeger | sorantis: To disable check-requirements, you need to udpate layout.yaml indeed | 15:29 |
| pabelanger | AJaeger: right, but sorantis said they don't sync from it. Guess I am confused, all openstack projects regardless of bigtent should be syncing from openstack/requirements? or is it optional | 15:29 |
| *** ajmiller has joined #openstack-infra | 15:30 | |
| *** doug-fish has quit IRC | 15:30 | |
| pabelanger | if optional, then I don't think they need to gate on check-requirements | 15:30 |
| AJaeger | pabelanger: Only projects that are installed together need it, so a project that is not installed in production does not need to be part of requirements | 15:30 |
| *** doug-fish has joined #openstack-infra | 15:30 | |
| pabelanger | AJaeger: okay, ya, thank makes sence | 15:31 |
| AJaeger | pabelanger: none of infra projects uses requirements - and also tools only needed during *test time* don't need it | 15:31 |
| pabelanger | sense* | 15:31 |
| *** ajmiller_ has joined #openstack-infra | 15:31 | |
| AJaeger | so, spec repositories are also not part of requirements | 15:31 |
| AJaeger | sorantis, kingbird seems usefull to be part of requirements, so I propose you do the following: | 15:32 |
| pabelanger | sorantis: So, I agree with AJaeger. You should enroll kingbird into openstack/requirements, so you don't have these issues moving forward | 15:32 |
| AJaeger | 1) Send change for zuul/layout.yaml | 15:32 |
| AJaeger | to disable check-requirements temporarily | 15:32 |
| AJaeger | 2) send a revert directly and mark it as WIP ;) | 15:32 |
| AJaeger | 3) Update manually requirementes from requirements repo, the requirements repo has tools to do so | 15:32 |
| *** verdurin has joined #openstack-infra | 15:32 | |
| *** rkukura has quit IRC | 15:32 | |
| AJaeger | 4) unWIP the revert and let's merge it | 15:33 |
| *** larainema has quit IRC | 15:33 | |
| AJaeger | 5) SEnd a change for requirements repo to add kingbird to projects.txt | 15:33 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Disable puppet tests for fuel-library 4.x branches https://review.openstack.org/280286 | 15:33 |
| AJaeger | you can add for change 5 a "depends-on" the reverted change | 15:33 |
| *** larainema has joined #openstack-infra | 15:34 | |
| AJaeger | sorantis: I have to leave now, can review step 1 later and others here can as well... | 15:34 |
| *** ajmiller has quit IRC | 15:34 | |
| *** doug-fish has quit IRC | 15:35 | |
| sorantis | AJaeger: thanks! First I’ll do 1) | 15:35 |
| sorantis | then I’ll need some guidelines with 2) -4) | 15:35 |
| sorantis | I’ll sibmit a patch in a min | 15:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Import ansible-role-nginx https://review.openstack.org/279101 | 15:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Enable windmill-jobs for all ansible-roles https://review.openstack.org/279308 | 15:37 |
| AJaeger | sorantis: I just did step 3 for you - 280288 | 15:37 |
| *** exploreshaifali has joined #openstack-infra | 15:37 | |
| AJaeger | pabelanger: ping me later - once everything is green again - to +A your two changes... | 15:38 |
| sorantis | AJaeger: thanks a million | 15:38 |
| AJaeger | bye for now | 15:38 |
| pabelanger | AJaeger: sure, just monitoring now | 15:38 |
| openstackgerrit | Dimitri Mazmanov proposed openstack-infra/project-config: Remove check requirements from Kingbird https://review.openstack.org/280290 | 15:39 |
| *** piet has joined #openstack-infra | 15:40 | |
| *** sshnaidm has quit IRC | 15:40 | |
| *** yamamoto has joined #openstack-infra | 15:41 | |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add trace option https://review.openstack.org/280220 | 15:43 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add show-all-changes option https://review.openstack.org/280221 | 15:43 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add force-sync option https://review.openstack.org/280216 | 15:43 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add depth option https://review.openstack.org/280217 | 15:43 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add reset-first option https://review.openstack.org/280218 | 15:43 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add no-tags option https://review.openstack.org/280219 | 15:43 |
| *** eharney has quit IRC | 15:44 | |
| *** piet has quit IRC | 15:45 | |
| openstackgerrit | Dimitri Mazmanov proposed openstack/requirements: Add kingbird to the list of project https://review.openstack.org/280295 | 15:46 |
| openstackgerrit | Nikki Heald proposed openstack-infra/jenkins-job-builder: Allow null values for log-text https://review.openstack.org/280296 | 15:46 |
| *** jlanoux has quit IRC | 15:47 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add Juju Charms for OpenStack https://review.openstack.org/232705 | 15:47 |
| *** asselin_ has joined #openstack-infra | 15:50 | |
| *** thomasem has quit IRC | 15:50 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Set overcloud deploy timeout https://review.openstack.org/273716 | 15:52 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add definition for controller hpuseast https://review.openstack.org/260018 | 15:52 |
| *** thomasem has joined #openstack-infra | 15:52 | |
| *** sylwesterB has joined #openstack-infra | 15:53 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job https://review.openstack.org/273743 | 15:53 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test to periodic job https://review.openstack.org/279218 | 15:54 |
| *** ddieterly has quit IRC | 15:55 | |
| *** ddieterly has joined #openstack-infra | 15:56 | |
| *** eharney has joined #openstack-infra | 15:59 | |
| jamespage | AJaeger, thanks for the pointers - now have my review passing tests... | 16:00 |
| *** ddieterly has quit IRC | 16:00 | |
| *** haypo has joined #openstack-infra | 16:00 | |
| jamespage | AJaeger, just so I understand - using python-jobs will execute the lint and py27 targets for these projects? | 16:01 |
| *** krtaylor has quit IRC | 16:01 | |
| haypo | "INFO:zuul.Cloner:Creating repo openstack/cinder from upstream git://git.openstack.org/openstack/cinder" "Build timed out (after 50 minutes). Marking the build as failed." -- http://logs.openstack.org/83/277983/3/gate/gate-cinder-python27/86826d9/console.html -- are you aware on an issue with the git repo? | 16:01 |
| haypo | (hi!) | 16:01 |
| *** kozhukal` has joined #openstack-infra | 16:04 | |
| *** kozhukalov has quit IRC | 16:06 | |
| pabelanger | haypo: looks like the job hung cloning cinder. possible network issue? | 16:08 |
| *** doug-fish has joined #openstack-infra | 16:11 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/project-config: Add giftwrap to OpenStack https://review.openstack.org/280305 | 16:11 |
| *** moravec has joined #openstack-infra | 16:12 | |
| *** ociuhandu has joined #openstack-infra | 16:12 | |
| *** sdake has joined #openstack-infra | 16:12 | |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Dedicated specs repo for Freezer https://review.openstack.org/280307 | 16:12 |
| *** ociuhandu has quit IRC | 16:13 | |
| *** zeih_ has quit IRC | 16:13 | |
| *** sabeen1 has joined #openstack-infra | 16:13 | |
| *** krtaylor has joined #openstack-infra | 16:14 | |
| *** pblaho has joined #openstack-infra | 16:16 | |
| *** ociuhandu has joined #openstack-infra | 16:17 | |
| *** kzaitsev_mb has quit IRC | 16:17 | |
| *** vgridnev has quit IRC | 16:17 | |
| *** paramite has quit IRC | 16:18 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:19 | |
| *** jlanoux has joined #openstack-infra | 16:19 | |
| *** piet has joined #openstack-infra | 16:20 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add definition for compute servers in us east https://review.openstack.org/280266 | 16:22 |
| *** exploreshaifali has quit IRC | 16:23 | |
| *** [HeOS] has joined #openstack-infra | 16:24 | |
| *** daemontool_ has joined #openstack-infra | 16:25 | |
| *** HeOS has quit IRC | 16:25 | |
| *** samuelBartel has joined #openstack-infra | 16:25 | |
| *** ddieterly has joined #openstack-infra | 16:26 | |
| *** daemontool has quit IRC | 16:26 | |
| *** _tzn has quit IRC | 16:27 | |
| *** daemontool_ has quit IRC | 16:27 | |
| *** sorantis has quit IRC | 16:28 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/project-config: Add giftwrap to OpenStack https://review.openstack.org/280305 | 16:29 |
| tsufiev | AJaeger, hi! would you mind taking a look at https://review.openstack.org/#/c/269578/ ? | 16:30 |
| yolanda | AJaeger, jjb failed again on jenkins01, i had to run again, but it will start processing only the latest jobs, it was nearly finishing | 16:31 |
| *** sshnaidm has joined #openstack-infra | 16:31 | |
| yolanda | everything shold be ok on others | 16:31 |
| *** jed56 has quit IRC | 16:33 | |
| *** tiswanso has quit IRC | 16:34 | |
| pabelanger | jenkins04 seems overloaded | 16:34 |
| pabelanger | not responding to HTTP | 16:34 |
| pabelanger | or just really slow | 16:34 |
| *** tzn has joined #openstack-infra | 16:35 | |
| *** tzn has quit IRC | 16:35 | |
| *** tiswanso has joined #openstack-infra | 16:35 | |
| yolanda | pabelanger i had trouble with jenkins04 this morning as well | 16:35 |
| yolanda | seems it's performing bad | 16:35 |
| *** abregman is now known as abregman|mtg | 16:35 | |
| AJaeger | tsufiev: I let others review that, the setup is unusual. Syntax is fine - better ask sdague for it. | 16:36 |
| *** kzaitsev_mb has quit IRC | 16:36 | |
| *** pblaho has quit IRC | 16:36 | |
| AJaeger | jamespage: read again my latest blog post to understand what's in it or check the template definition in zuul/layout.yaml | 16:36 |
| *** pblaho has joined #openstack-infra | 16:37 | |
| AJaeger | jamespage: basically, docs, python27 and pep8 | 16:37 |
| jamespage | AJaeger, ok - so that's what I thought - as our lint targets are names 'lint' not 'pep8' that won't get covered | 16:38 |
| *** asettle has quit IRC | 16:38 | |
| AJaeger | jamespage: see http://governance.openstack.org/reference/project-testing-interface.html for pep8 | 16:38 |
| fungi | i cleaned up jenkins01 and jenkins05 yesterday | 16:39 |
| *** daemontool has joined #openstack-infra | 16:39 | |
| *** eharney has quit IRC | 16:39 | |
| fungi | wonder if 04 also was too far gone to get auto-rebooted at the end of last week | 16:39 |
| *** abregman|mtg has quit IRC | 16:40 | |
| jamespage | AJaeger, right - gotcha - will need to update to pep8 then... | 16:40 |
| jamespage | I was trying to figure out whether I switch in the projects, or write a new def... | 16:40 |
| *** piet has quit IRC | 16:41 | |
| *** PsionTheory has quit IRC | 16:41 | |
| yolanda | fungi, today, cache was corrupted on most of the jenkins masters | 16:41 |
| fungi | yeah, jenkins05 is unresponsive on its webui (though i'm coming via a wireless modem so it's possibly my connection as well) | 16:41 |
| fungi | yolanda: corrupt jjb cache? or some other cache? | 16:42 |
| fungi | what would corrupt the jjb cache? | 16:42 |
| yolanda | jjb | 16:42 |
| yolanda | i had to run with --flush-cache | 16:42 |
| AJaeger | jenkins05 looks fine for me | 16:42 |
| yolanda | jjb update was not even starting | 16:42 |
| AJaeger | jenkins04 has indeed problems | 16:43 |
| tsufiev | AJaeger: okay, will ping him | 16:44 |
| fungi | er, yeah i meant 04, sorry | 16:44 |
| fungi | typo | 16:44 |
| AJaeger | tsufiev: just commented - with -1 | 16:44 |
| AJaeger | tsufiev: I digged a bit... | 16:44 |
| yolanda | yes, jenkins04 looks faulty | 16:44 |
| fungi | yolanda: yeah, i see mention in scrollback of something about corrupt jjb cache, though no details | 16:44 |
| fungi | i take it the error is in syslog from when puppet was trying to run it? | 16:45 |
| *** jsavak has joined #openstack-infra | 16:45 | |
| *** rajinir has joined #openstack-infra | 16:46 | |
| yolanda | fungi, it was a python error about invalid stream | 16:46 |
| yolanda | i have the logs stored, let me check | 16:46 |
| *** claudiub has quit IRC | 16:47 | |
| *** armax has joined #openstack-infra | 16:47 | |
| *** rajinir has quit IRC | 16:47 | |
| *** rajinir has joined #openstack-infra | 16:47 | |
| *** kozhukal` has quit IRC | 16:47 | |
| *** rcernin has quit IRC | 16:48 | |
| yolanda | fungi http://paste.openstack.org/show/487017/ | 16:48 |
| yolanda | that happened when running jjb play on puppetmaster | 16:48 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Mark non-voting jobs in the graph https://review.openstack.org/280322 | 16:48 |
| *** pblaho has quit IRC | 16:49 | |
| *** jsavak has quit IRC | 16:49 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:50 | |
| *** ajmiller_ has quit IRC | 16:51 | |
| *** piet has joined #openstack-infra | 16:51 | |
| fungi | yeah, looks like the cache file might have been truncated? | 16:52 |
| fungi | i wonder if something killed jjb mid-write | 16:52 |
| *** wolverineav has joined #openstack-infra | 16:53 | |
| yolanda | fungi when i looked this morning, i saw puppet logs showing that errors. Then i ran the jjb play manually, and result was the same | 16:53 |
| yolanda | looks as something cut jjb on the middle, yes | 16:53 |
| fungi | but odd that it was that way on most/all of the jenkins masters | 16:53 |
| *** ddieterly has quit IRC | 16:54 | |
| *** daemontool has quit IRC | 16:54 | |
| *** ddieterly has joined #openstack-infra | 16:55 | |
| *** [HeOS] has quit IRC | 16:57 | |
| *** mikelk has quit IRC | 16:57 | |
| *** wolverineav has quit IRC | 16:57 | |
| *** dizquierdo has quit IRC | 16:58 | |
| *** ddieterly has quit IRC | 16:59 | |
| *** michaelxin has quit IRC | 17:00 | |
| *** mari0jv has quit IRC | 17:00 | |
| *** lindsayk has quit IRC | 17:00 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add baremetal hpuseast node definition https://review.openstack.org/280326 | 17:00 |
| *** notnownikki has quit IRC | 17:00 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add Juju Charms for OpenStack https://review.openstack.org/232705 | 17:01 |
| *** vgridnev has joined #openstack-infra | 17:01 | |
| yolanda | this morning there were few jobs that needed to be changed or removed, i wonder if the locks are working properly, and puppetmaster is not trying to update jjb in top of a running one | 17:03 |
| *** dims_ has joined #openstack-infra | 17:03 | |
| *** sfinucan has quit IRC | 17:03 | |
| *** mariojv has joined #openstack-infra | 17:04 | |
| *** dims has quit IRC | 17:04 | |
| *** tphummel has joined #openstack-infra | 17:05 | |
| *** sabeen1 has quit IRC | 17:05 | |
| *** ihrachys has quit IRC | 17:06 | |
| *** piet has quit IRC | 17:06 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Exlude gate-tempest-dsvm-neutron-src-oslo.service from stable/liberty https://review.openstack.org/280328 | 17:06 |
| *** salv-orlando has joined #openstack-infra | 17:06 | |
| mriedem | haypo: dims_: ^ | 17:06 |
| *** michaelxin has joined #openstack-infra | 17:07 | |
| *** mariojv has quit IRC | 17:07 | |
| *** michaelxin has quit IRC | 17:07 | |
| *** mariojv has joined #openstack-infra | 17:09 | |
| *** esikachev has quit IRC | 17:09 | |
| *** doug-fish has quit IRC | 17:09 | |
| dims_ | mriedem : +1 | 17:09 |
| BobBall | asselin: seems that the ssh key in /var/lib/jenkins/.ssh/id_rsa is readable by root only but needed by nodepool user? I guess I must have done something wrong, but any ideas? | 17:10 |
| openstackgerrit | Jesse Keating proposed openstack-infra/project-config: Add giftwrap to OpenStack https://review.openstack.org/280305 | 17:10 |
| *** julim has quit IRC | 17:11 | |
| *** salv-orlando has quit IRC | 17:11 | |
| *** michaelxin has joined #openstack-infra | 17:12 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/project-config: Add giftwrap to OpenStack https://review.openstack.org/280305 | 17:12 |
| AJaeger | mriedem, dims_ : Looking at 280328: Should the job really run on kilo? | 17:12 |
| mriedem | AJaeger: i'm not aware of any failures on kilo, | 17:13 |
| *** lindsayk has joined #openstack-infra | 17:13 | |
| yolanda | fungi, do you know why puppet is disabled on review.openstack.org? | 17:13 |
| mriedem | AJaeger: having said that, i don't think oslo.service has a kilo branch | 17:13 |
| yolanda | manage-projects not running, so the projects are not being created | 17:13 |
| mriedem | AJaeger: confirmed, oslo.service doesn't have a stable/kilo branch | 17:13 |
| mriedem | so shouldn't be a problem | 17:13 |
| mriedem | kilo doesn't use upper-constraints either so even if it did, it wouldn't be the same issue | 17:13 |
| AJaeger | mriedem: ok | 17:14 |
| haypo | mriedem: +1 | 17:15 |
| *** doug-fish has joined #openstack-infra | 17:15 | |
| *** ddieterly has joined #openstack-infra | 17:16 | |
| pabelanger | yolanda: jeblair disabled it sat / sun to fix a file system issue. Maybe he didn't reenable it? | 17:16 |
| *** mtanino has quit IRC | 17:16 | |
| *** e0ne has quit IRC | 17:16 | |
| yolanda | it is on the disabled list | 17:17 |
| yolanda | is it ok to enable or will i break something? | 17:17 |
| fungi | yolanda: i believe jeblair was troubleshooting some impact from an inode exhaustion on the /home/gerrit2/review_site filesystem on saturday, check channel logs | 17:18 |
| *** sabeen1 has joined #openstack-infra | 17:18 | |
| *** thomasem has quit IRC | 17:18 | |
| fungi | he ended up creating a new filesystem with higher inode capacity ratio and migrating the data into it | 17:18 |
| *** bgmccollum has left #openstack-infra | 17:19 | |
| fungi | but for whatever reason puppet-gerrit seemed to ensure=>running on the gerrit service which caused some corrupted changes before he spotted it | 17:19 |
| fungi | since puppet started gerrit back up while he was in the middle of fixing it | 17:19 |
| *** doug-fish has quit IRC | 17:19 | |
| yolanda | pabelanger, fungi, one of the collateral effects of that being disabled, is that manage-project is not running. So projects are not created properly on gerrit, and image builds failed. I was building a test image with dib, to test new infra cloud, and fails with non existing projects and branches error | 17:19 |
| fungi | makes sense, yes | 17:20 |
| AJaeger | indeed, yolanda | 17:20 |
| *** notnownikki has joined #openstack-infra | 17:21 | |
| *** ihrachys has joined #openstack-infra | 17:22 | |
| pabelanger | Ya, I don't think is was ever removed from the emergency list. gerrit is online now, so no reason for puppet not to run IMO | 17:22 |
| *** ihrachys has quit IRC | 17:23 | |
| yolanda | i'll remove from the list | 17:23 |
| *** thomasem has joined #openstack-infra | 17:23 | |
| clarkb | pabelanger: except that the puppet manifest is broken | 17:23 |
| *** ihrachys has joined #openstack-infra | 17:23 | |
| clarkb | we should fix that | 17:23 |
| pabelanger | clarkb: Ya, I am looking at that now | 17:23 |
| yolanda | clarkb so i cannot disable? | 17:23 |
| yolanda | i mean, i cannot enable puppet there? | 17:23 |
| clarkb | yolanda: you can just pointing out why it is disabled | 17:24 |
| *** nelsnelson has quit IRC | 17:24 | |
| *** jpr has joined #openstack-infra | 17:24 | |
| *** rcernin has joined #openstack-infra | 17:24 | |
| clarkb | jeblair mentioned it might have been the gerrit init exec and not an ensure running | 17:24 |
| clarkb | if that is the case it is likely complicated to fix, but if it is an ensure running that is simpler | 17:25 |
| yolanda | clarkb, pabelanger, so i was reading the logs and it seemed that problems on gerrit were fixed | 17:26 |
| *** scheuran has quit IRC | 17:26 | |
| *** akshai has joined #openstack-infra | 17:26 | |
| yolanda | but if there is any issue, this needs to be solved asap to reenable puppet, because if not, that's causing a block on gerrit projects and acls | 17:26 |
| pabelanger | Ya, puppet-gerrit is using an exec to start gerrit | 17:26 |
| *** tiswanso has quit IRC | 17:27 | |
| pabelanger | not the service type | 17:27 |
| *** _nadya_ has quit IRC | 17:27 | |
| *** _nadya_ has joined #openstack-infra | 17:27 | |
| *** tiswanso has joined #openstack-infra | 17:28 | |
| pabelanger | been like that since 10/2012 | 17:28 |
| *** fhubik has quit IRC | 17:28 | |
| pabelanger | guess we hardly stop gerrit when puppet is running | 17:28 |
| *** erlon has joined #openstack-infra | 17:28 | |
| clarkb | ya project renames include a puppet stop | 17:29 |
| clarkb | so mayve we just accept this as is for now and reenable | 17:29 |
| yolanda | clarkb yes, i was going to ask, why is that an issue now | 17:29 |
| clarkb | if it was simpler fix I would suggest fixing really quickly but that isnt the case | 17:29 |
| yolanda | or if that just needs improvement | 17:29 |
| clarkb | because it corrupted us on saturday... | 17:29 |
| *** arxcruz has quit IRC | 17:30 | |
| *** krtaylor has quit IRC | 17:30 | |
| *** akshai has quit IRC | 17:30 | |
| yolanda | so doesn't look as a simple fix, it will need some time and reviews | 17:30 |
| *** krtaylor has joined #openstack-infra | 17:31 | |
| *** yaume has quit IRC | 17:32 | |
| yolanda | clarkb, will it be safe to reenable for the moment to pick up the latest changes of the day? what's the risk of corruption? | 17:32 |
| *** markvoelker has joined #openstack-infra | 17:32 | |
| AJaeger | pabelanger: AFAIK nowadays most puppet scripts do not ensure that the service is running - and gerrit was the odd one out that surprised jeblair on Saturday | 17:32 |
| *** samuelBartel has quit IRC | 17:33 | |
| clarkb | yolanda: the risk of corruption appears to be based on puppet starting gerrit when review_site is not ready. I do not know if it is ready now, presumably it is because the service is running | 17:33 |
| yolanda | well yes, it's running | 17:33 |
| yolanda | as this can cause potential problems, such as jenkins trying to run jobs on projects that don't exist, or the one i hit with image builds, i think is important to reenable | 17:35 |
| pabelanger | AJaeger: Yup, I just think puppet-gerrit was over looked, since we stop it for renames. The emergency fix jeblair was working on, that's the first time I seen us run out of inodes. What should likely happen, just thoughts after the fact, create / update the emergency server maintenance docs, and always add host to ansible emergency list. Then you 100% know nothing with change while you are working | 17:35 |
| *** burgerk has quit IRC | 17:35 | |
| pabelanger | on the node. | 17:35 |
| *** tzn has joined #openstack-infra | 17:35 | |
| clarkb | yolanda: I dont think anyone disagrees | 17:36 |
| clarkb | just that due diligence should be done | 17:36 |
| clarkb | and I cant do that right now | 17:36 |
| pabelanger | AJaeger: both 279101 and 279308 are green now | 17:36 |
| *** salv-orlando has joined #openstack-infra | 17:36 | |
| *** markvoelker has quit IRC | 17:37 | |
| yolanda | ok enabling that | 17:37 |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add Juju Charms for OpenStack https://review.openstack.org/232705 | 17:37 |
| asselin__ | BobBall, nodepool should only be using the public key | 17:37 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 17:37 |
| *** doug-fish has joined #openstack-infra | 17:38 | |
| AJaeger | pabelanger: waiting for result of puppet enablement to see that projects are created properly | 17:38 |
| *** asselin__ has quit IRC | 17:38 | |
| pabelanger | AJaeger: sounds good | 17:38 |
| BobBall | asselin: http://paste.openstack.org/show/487022/ | 17:38 |
| *** yaume has joined #openstack-infra | 17:38 | |
| *** lazyMed has joined #openstack-infra | 17:38 | |
| SamYaple | so ive been tracking down why _some_ of our builds timeout after an hour. in the past it has been other issues, but more recently all of the long gates and time outs have been occuring on the internap vms | 17:39 |
| BobBall | asselin: but I don't have that filename set in /etc/nodepool/nodepool.yaml or in ~nodepool/.config/openstack/clouds.yaml | 17:39 |
| SamYaple | the gates are progressing, but the disk is bottlenecking | 17:39 |
| fungi | clarkb: once i get home this afternoon, i also want to see whether cleaning up the groups file for openstack-dev/sandbox fixes its acl | 17:39 |
| clarkb | SamYaple: how are you obsrrving that? we can probably pass that info to mgagne | 17:40 |
| clarkb | fungi: kk | 17:40 |
| asselin_ | BobBall, I think it's an environment variable...let me see the docs | 17:40 |
| *** crinkle_ is now known as crinkle | 17:40 | |
| *** gibi_ has quit IRC | 17:40 | |
| fungi | hopefully zaro has an opportunity to weigh in before then on whether or not that will fix registered users matching for that repo | 17:40 |
| clarkb | I need to do a nodepool service restart for latest bugfixes | 17:40 |
| SamYaple | clarkb: our gate times out at 1 hour when say the same job on RAX takes 20m. looking at the logs it is _working_ the whole time (not hung) just working alot slower than normal | 17:40 |
| asselin_ | BobBall, http://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool | 17:40 |
| clarkb | but otherwisr am pretty beat so likrly to treat holiday like a holiday | 17:41 |
| *** gibi has joined #openstack-infra | 17:41 | |
| SamYaple | ill start collecting logs for proof if thats needed | 17:41 |
| clarkb | SamYaple: right but how do we know it is disk? | 17:41 |
| asselin_ | BobBall, sorry...juggling a few things right now.... | 17:41 |
| clarkb | SamYaple: generally we try to provide our cloud providers with concrete information | 17:41 |
| *** bpokorny has joined #openstack-infra | 17:41 | |
| SamYaple | oh clarkb youre right. it _may_ not be, but probably is. ill see what i can do to validate that it absolutely is (iostat in the background maybe) | 17:41 |
| SamYaple | let me work on that | 17:41 |
| clarkb | mgagne hangs out here though so may be able to work with less info | 17:42 |
| *** piet has joined #openstack-infra | 17:42 | |
| SamYaple | its behaviour is likely disk, but ill get your evidence concrete | 17:42 |
| yolanda | pabelanger, AJaeger, so i reenabled it, and i'm monitoring | 17:42 |
| fungi | SamYaple: clarkb: possible impact of memory exhaustion causing slower swapfile performance than swappart performance? | 17:42 |
| AJaeger | yolanda: thanks. | 17:43 |
| fungi | since we swappart on rax but swapfile on ovh | 17:43 |
| AJaeger | yolanda: is jenkins02 finished with updating? | 17:43 |
| SamYaple | fungi: possible, but unlikely in this case. this is a build gate job, we barely break 4gb | 17:43 |
| yolanda | AJaeger, jenkins02 did, but not jenkins01 | 17:43 |
| SamYaple | deploying is another story | 17:43 |
| fungi | ahh, yeah then probably not swap related | 17:43 |
| fungi | could still be something to do with backing i/o performance in different providers, i suppose | 17:44 |
| AJaeger | strange - looking at https://jenkins02.openstack.org/job/gate-nova-python27/4821/console it cannot find the constraints file ;( | 17:44 |
| fungi | is that job using dstat to collect system performance details? | 17:44 |
| AJaeger | So, that means the job is old | 17:44 |
| SamYaple | fungi: it is not, but ill set it up. im working on getting our gates consistent speeds right now. so thats a good thing to have | 17:45 |
| AJaeger | while it works fine on https://jenkins05.openstack.org/job/gate-nova-docs/16611/consoleFull | 17:45 |
| *** openstackgerrit has quit IRC | 17:45 | |
| *** openstackgerrit has joined #openstack-infra | 17:46 | |
| fungi | AJaeger: sounds like jjb not updated on 02 but updated on 05 | 17:46 |
| yolanda | 02 also failed? i saw ansible with success result there | 17:47 |
| pabelanger | SamYaple: do you have a job name by chance? Curious to see if we can notice the different in grafana.o.o | 17:47 |
| AJaeger | fungi: I agree | 17:47 |
| *** hashar has quit IRC | 17:47 | |
| SamYaple | pabelanger: longest running jobs are gate-kolla-dsvm-build-centos-binary and gate-kolla-dsvm-build-centos-source | 17:47 |
| SamYaple | the ubuntu gates do go up in time, but ubuntu is less io intensive so im thinking thats why it never times out | 17:48 |
| BobBall | asselin_: No worries. The NODEPOOL_SSH_KEY is indeed set there and I'm starting nodepool with "service nodepool start" which sources /etc/defualt/nodepool | 17:48 |
| *** yaume has quit IRC | 17:48 | |
| *** lazyMed has quit IRC | 17:49 | |
| asselin_ | BobBall, did you secure jenkins or not? | 17:49 |
| SamYaple | you know clarkb fungi it might be network speeds in fetching packages now that i read the logs closer and not io bound | 17:49 |
| mgagne | clarkb: please let me know if I can be of any help | 17:49 |
| SamYaple | but again ill work it out and get back wit more info | 17:50 |
| BobBall | asselin_: And /var/log/nodepool/nodepool.log also shows the NODEPOOL_SSH_KEY | 17:50 |
| BobBall | asselin_: yes I have now | 17:50 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add heat lbaas specific test for v1 and v2 https://review.openstack.org/237156 | 17:51 |
| BobBall | asselin_: And rebooted the host (just in case) - still getting the permission denied error | 17:51 |
| clarkb | mgagne: I think we need to determine the cause then pass it along | 17:52 |
| clarkb | mgagne: will let you know | 17:52 |
| *** ihrachys has quit IRC | 17:52 | |
| *** piet has quit IRC | 17:53 | |
| asselin_ | BobBall, I remember now. If jenkins is not secured, then nodepool can just access it. Otherwise you need to do this: http://docs.openstack.org/infra/openstackci/third_party_ci.html#securing-jenkins-optional | 17:53 |
| smarcet | fung: afternoon | 17:53 |
| asselin_ | BobBall, "Save this token to the 'jenkins_api_key' value in /etc/puppet/environments/common.yaml" is the important part. | 17:53 |
| asselin_ | BobBall, that will let nodepool run the jenkins apis | 17:54 |
| BobBall | asselin_: I've done those | 17:54 |
| smarcet | fungi: i have this patch waiting for approval | 17:54 |
| smarcet | https://review.openstack.org/#/c/279513/ | 17:54 |
| smarcet | :) | 17:54 |
| yolanda | AJaeger, puppet running on review and manage-project running.. so things will start to look better | 17:54 |
| BobBall | asselin_: It's not accessing Jenkins; for some reason nodepool is trying to use the jenkins SSH key to access the nodes | 17:54 |
| BobBall | asselin_: And I really don't understand why and where it's getting the path to the jenkins SSH key from | 17:55 |
| * AJaeger waits for manage-projects to finish ;) | 17:55 | |
| *** jcoufal has quit IRC | 17:55 | |
| AJaeger | yolanda: can you rerun jjb on 02, please? | 17:56 |
| yolanda | AJaeger, ansible reports ok for jenkins02 run, weird | 17:56 |
| yolanda | i'll check anyway | 17:57 |
| AJaeger | yolanda, pabelanger : Perhaps I'm crazy, please check https://jenkins02.openstack.org/job/gate-nova-python27/4821/console and tell me whether that is correct... | 17:57 |
| * AJaeger is missing the zuul-git-prep-upper-constraints setup | 17:58 | |
| AJaeger | yolanda, compare https://jenkins02.openstack.org/job/gate-nova-python27/configure and https://jenkins05.openstack.org/job/gate-nova-python27/configure | 17:59 |
| AJaeger | 02 uses gerrit-git-prep while 05 uses zuul-git-prep-upper-constraints | 18:00 |
| yolanda | i launched the play on jenkins02 again | 18:00 |
| BobBall | asselin_: Seems that /var/lib/jenkins/.ssh/id_rsa is the default value for "private-key" in the nodepool config. | 18:01 |
| *** jaosorior has quit IRC | 18:01 | |
| pabelanger | AJaeger: yolanda: Ya, looks out of sync with commit: 79251ec1e7d8a123084bbee350c78d7e26dc2efa | 18:01 |
| BobBall | asselin_: So I needed to explicitly add my private-key (using /home/nodepool/.ssh/id_rsa) to my nodepool config | 18:01 |
| yolanda | AJaeger, that job hasn't changed for long time on jenkins02.. but seems that there is a bigger problem on jenkins02 with that, because last job reconfig is dated 2015-05-06 | 18:02 |
| yolanda | whereas on jenkins05 i can see reconfigs on january 2016 for example | 18:02 |
| asselin_ | BobBall, everything should go here: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/single_node_ci_data.yaml | 18:03 |
| *** eil397 has joined #openstack-infra | 18:03 | |
| BobBall | asselin_: It did - my point was that the file ~nodepool/.ssh/id_rsa was created (presumably by puppet) and was identical to /var/lib/jenkins/.ssh/id_rsa. So I've set the private-key: setting in nodepool.yaml to the puppet-created SSH key so it doesn't use the default | 18:04 |
| asselin_ | BobBall, somethings not right...you should need that in nodepool.yaml. | 18:05 |
| asselin_ | BobBall, are you using snapshot builds or DIB builds? | 18:05 |
| AJaeger | yolanda: May 2015? That's strange ;( | 18:05 |
| BobBall | asselin_: snapshot builds; can't use DIB for XenServer | 18:05 |
| AJaeger | yolanda: can't help with that, sorry | 18:05 |
| AJaeger | yeah, http://git.openstack.org/cgit/openstack/service-types-authority/ has a gitreview file now | 18:06 |
| samueldmq | hi all | 18:06 |
| yolanda | AJaeger looks so strange yes, going to check another jobs | 18:07 |
| samueldmq | there is a keystone change (https://review.openstack.org/#/c/207218/) in which a job (py27) is said to be in UNSTABLE status | 18:07 |
| samueldmq | what does that mean ? I've never seen that before | 18:07 |
| fungi | okay, have to disappear again to get lunch and then start the several-hours-long drive back home again. should be online around 2200 utc | 18:07 |
| AJaeger | samueldmq: UNSTABLE often is a Jenkins problem | 18:07 |
| AJaeger | samueldmq: let me check... | 18:07 |
| samueldmq | AJaeger: okay, please :) | 18:07 |
| pabelanger | AJaeger: samueldmq: http://logs.openstack.org/18/207218/28/check/gate-keystone-python27/19d407c/console.html#_2016-02-15_17_22_08_700 | 18:08 |
| pabelanger | looks like scp failed | 18:08 |
| AJaeger | "ERROR: Failed to upload files connection is closed by foreign host" | 18:08 |
| pabelanger | job passed, but jenkins couldn't finish the post tasks properly | 18:08 |
| pabelanger | hence unstable | 18:08 |
| AJaeger | that was jenkins02 | 18:08 |
| pabelanger | Ya | 18:08 |
| yolanda | AJaeger, i refreshed page, i see 2016-01-22_07-01-49 as last reconfig, looks more reasonable now | 18:09 |
| AJaeger | samueldmq: I suggest to "recheck" | 18:09 |
| *** jlanoux has quit IRC | 18:09 | |
| asselin_ | BobBall, snapshot builds were very confusing to me and difficult to setup the credentials. I need to find my notes on that... | 18:09 |
| samueldmq | AJaeger: pabelanger: nice, I will leave a recheck, thanks for the explanations | 18:10 |
| samueldmq | always good to know these things :) | 18:10 |
| pabelanger | asselin_: BobBall: nodepool should auto generate SSH keys in that case. So, it won't use local SSH keys IIRC | 18:10 |
| *** tzn has quit IRC | 18:10 | |
| AJaeger | samueldmq: if there's an UNSTABLE jobs, please always investigate and tell here | 18:10 |
| *** achanda has joined #openstack-infra | 18:11 | |
| samueldmq | AJaeger: sure, as it may be (very likely) infra issues you need to be aware of | 18:11 |
| *** daemontool has joined #openstack-infra | 18:11 | |
| *** amotoki has quit IRC | 18:12 | |
| *** jcoufal has joined #openstack-infra | 18:13 | |
| AJaeger | KanagarajM, python-namosclient should be fine now | 18:13 |
| *** shashank_hegde has joined #openstack-infra | 18:13 | |
| AJaeger | EmilienM: your pacemaker repo should be fine now | 18:13 |
| *** e0ne has joined #openstack-infra | 18:13 | |
| EmilienM | AJaeger: it is! thanks a lot! | 18:13 |
| yolanda | AJaeger, i put jenkins02 on maintenance mode, because jjb was failing on connect, and i'm running the update again | 18:14 |
| AJaeger | thanks, yolanda | 18:15 |
| *** nelsnelson has joined #openstack-infra | 18:15 | |
| openstackgerrit | Merged openstack-infra/system-config: OpenStackId Production Release https://review.openstack.org/279513 | 18:16 |
| *** jcoufal has quit IRC | 18:17 | |
| *** ifarkas has quit IRC | 18:18 | |
| *** pblaho has joined #openstack-infra | 18:21 | |
| AJaeger | yolanda: do you think it's ok to approve project-config changes again or should I wait? | 18:24 |
| *** eharney has joined #openstack-infra | 18:25 | |
| EmilienM | AJaeger: can I be core on the module? | 18:25 |
| AJaeger | EmilienM: I don't have permissions to grant you that ;) | 18:26 |
| *** daemontool_ has joined #openstack-infra | 18:27 | |
| EmilienM | AJaeger: maybe can I ask clarkb | 18:27 |
| AJaeger | EmilienM: you might need to wait until tomorrow | 18:28 |
| EmilienM | no problem at all | 18:28 |
| EmilienM | nothing is urgent for us | 18:28 |
| yolanda | hi AJaeger, things look better now. Seems manage-project is nearly finished, and reconfig on jenkins02 is ongoing | 18:29 |
| *** lucasagomes is now known as lucas-dinner | 18:29 | |
| clarkb | what needs doing? | 18:29 |
| clarkb | I can add someone to gerrit groups | 18:29 |
| *** daemontool_ has quit IRC | 18:29 | |
| *** daemontool has quit IRC | 18:29 | |
| clarkb | (I am rechecking nodepool changes so that I can get any additional things in before the restart | 18:29 |
| AJaeger | clarkb: add EmilienM to puppet-pacemaker | 18:30 |
| *** mtanino has joined #openstack-infra | 18:30 | |
| EmilienM | thanks guys | 18:30 |
| clarkb | done | 18:30 |
| EmilienM | clarkb++ | 18:31 |
| *** ddieterly has quit IRC | 18:31 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Properly fail/trap in eval_run_d https://review.openstack.org/259734 | 18:33 |
| yolanda | AJaeger, manage-projects finally ran | 18:33 |
| yolanda | btw, greghaynes, clarkb, i refactored change for infracloud bridge, to remove default route, and works fine. Can you review https://review.openstack.org/276292 ? | 18:34 |
| clarkb | yolanda: oh neat | 18:34 |
| greghaynes | yolanda: yep | 18:35 |
| *** kzaitsev_mb has quit IRC | 18:35 | |
| clarkb | greghaynes: looks like your last nodepool change for cleanup on disconnect hasn't been updated to just flip the db state to delete and let periodic cleanup get to it? | 18:35 |
| AJaeger | pabelanger: could you check my comment on https://review.openstack.org/#/c/276339/ , please? | 18:35 |
| pabelanger | AJaeger: sure | 18:35 |
| yolanda | thanks | 18:35 |
| *** kushal has quit IRC | 18:36 | |
| greghaynes | clarkb: correct. There does not exist a periodic cleanup which will take care of that, either | 18:36 |
| AJaeger | yolanda: great | 18:36 |
| greghaynes | clarkb: actually, I think I just got what youre suggesting | 18:36 |
| greghaynes | clarkb: which is to soft delete then call real delete in periodic cleanup along with gear delete | 18:37 |
| clarkb | greghaynes: yup | 18:37 |
| AJaeger | clarkb: I'm reviewing https://review.openstack.org/#/c/268888/ again - is timing ok to approve? | 18:37 |
| greghaynes | clarkb: ok, once I am done debugging this dib thing its next on my stack | 18:37 |
| *** wolverineav has joined #openstack-infra | 18:37 | |
| clarkb | AJaeger: I don't even remember reviewing that :) but ya should be fine to review | 18:38 |
| greghaynes | ianw: interesting thing https://review.openstack.org/#/c/204639/ has found - building a centos image on centos7 results in an image we can't boot | 18:38 |
| pabelanger | AJaeger: Ya, that code was mostly copied verbatim from system-config. So, not sure why we are doing both steps. It likely can be reduced now that we are on fedora 23, which defaults to dnf by default. | 18:38 |
| greghaynes | ianw: on ubuntu though that image works fine | 18:38 |
| *** esikachev has joined #openstack-infra | 18:38 | |
| *** wolverineav has quit IRC | 18:38 | |
| *** jordanP has quit IRC | 18:38 | |
| *** wolverineav has joined #openstack-infra | 18:39 | |
| pabelanger | AJaeger: we had other logic to check version if greater then version_id 21 | 18:39 |
| clarkb | yolanda: on that chang ethe logic around when to configure the bridge changes, it says only make the bridge if eth0.vlanY does not exist | 18:40 |
| clarkb | yolanda: is that correct? won't glean create that interface which is ifdown'd in configure_bridge? | 18:40 |
| yolanda | clarkb, ouch, i did a typo when copying to the change | 18:41 |
| yolanda | that needs to be bridge_name | 18:41 |
| yolanda | good catch | 18:41 |
| yolanda | i was running a local copy of that on my deploys but did not translate properly | 18:41 |
| clarkb | yolanda: an easy way to move patches arround is `git diff > patch` then in other repo `git apply patch` | 18:42 |
| AJaeger | pabelanger: I see now.. | 18:42 |
| AJaeger | pabelanger: I would have written this from scratch using /etc/os-release | 18:43 |
| pabelanger | AJaeger: left comment about both places | 18:43 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 18:43 |
| pabelanger | Happy to change them both if we agree to logic | 18:43 |
| AJaeger | pabelanger: read the comment and approved. | 18:43 |
| yolanda | clarkb thx, good to know. I have quite a mess on puppet-infracloud, system-config... on my server, because i needed to cherry-pick lots of changes that are pending to merge | 18:43 |
| yolanda | i fixed that | 18:43 |
| clarkb | right but regardless of how you have them stacked git diff should work if they don't otherwise conflict | 18:44 |
| AJaeger | pabelanger: It's not really worth to improve it IMHO | 18:44 |
| pabelanger | AJaeger: ack | 18:45 |
| greghaynes | clarkb: ok, running tests now... | 18:46 |
| *** eil397 has quit IRC | 18:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove unused check-sahara-extra-artifacts https://review.openstack.org/280360 | 18:49 |
| openstackgerrit | Merged openstack-infra/project-config: Import ansible-role-nginx https://review.openstack.org/279101 | 18:49 |
| openstackgerrit | Merged openstack-infra/project-config: Enable windmill-jobs for all ansible-roles https://review.openstack.org/279308 | 18:50 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Handle dib image deletion during periodic cleanup https://review.openstack.org/273703 | 18:50 |
| greghaynes | clarkb: ^ | 18:50 |
| clarkb | greghaynes: that looks about right | 18:52 |
| *** Sukhdev has joined #openstack-infra | 18:52 | |
| *** eil397 has joined #openstack-infra | 18:53 | |
| greghaynes | clarkb: passed tests so it must be good to ship ;) | 18:53 |
| openstackgerrit | Merged openstack-infra/project-config: Switch to dnf for fedora https://review.openstack.org/276339 | 18:53 |
| clarkb | greghaynes: https://review.openstack.org/#/c/273703/9..10/nodepool/nodepool.py that I need to think about more though | 18:54 |
| *** achanda has quit IRC | 18:54 | |
| greghaynes | clarkb: ah, so the only time that function is called now is when that is true already | 18:55 |
| greghaynes | clarkb: in the previous rev. we only set that when we hit a failure to delete | 18:55 |
| clarkb | so it will be a nopo? | 18:55 |
| clarkb | *noop? | 18:55 |
| greghaynes | clarkb: yep | 18:55 |
| greghaynes | clarkb: because of https://review.openstack.org/#/c/273703/9..10/nodepool/jobs.py | 18:56 |
| clarkb | right it is setting the state, and as long as nothing else calls that method (other than periodic cleanup then we should be good?) | 18:56 |
| greghaynes | yep | 18:56 |
| greghaynes | the spaghetti of events is :( | 18:56 |
| *** doug-fis_ has joined #openstack-infra | 18:58 | |
| clarkb | in this case it should be straightfowrad. If we try to delete dib images from outside of periodic cleanup then we will want to set that state so that periodic cleanup can attempt a delete on the backend of a failure | 18:58 |
| greghaynes | clarkb: if we want to clean it up, I think deleteDibImage should be the public method which sets soft delete and perioic cleanup actually does the gear submit | 18:58 |
| clarkb | if we don't try to delete dib images anywhere else then it is a noop | 18:58 |
| *** dizquierdo has joined #openstack-infra | 18:58 | |
| greghaynes | clarkb: Yea, so the question is really cli usage, then | 18:58 |
| *** kushal has joined #openstack-infra | 18:58 | |
| greghaynes | should the cli say "I failed, but its going to soft delete" | 18:59 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix dib gate https://review.openstack.org/280364 | 18:59 |
| clarkb | for cli I don't think so | 18:59 |
| *** doug-fi__ has joined #openstack-infra | 18:59 | |
| clarkb | because that is an interactive delete and if it fails you may not expect it to keep trying in the background | 18:59 |
| AJaeger | clarkb,pabelanger, greghaynes : looking at http://logs.openstack.org/88/268888/3/gate/gate-project-config-dib/4356e1d/console.html could you review the change above, please? 280364 | 18:59 |
| greghaynes | clarkb: Yep, agreed - so then this is correct | 18:59 |
| *** doug-f___ has joined #openstack-infra | 19:00 | |
| *** Sukhdev has quit IRC | 19:00 | |
| *** doug-fish has quit IRC | 19:00 | |
| *** pvaneck has joined #openstack-infra | 19:00 | |
| greghaynes | huh, I didn't know there was a gate-project-config-dib | 19:00 |
| clarkb | AJaeger: greghaynes that seems like a bug in dib-lint? | 19:00 |
| clarkb | we can work around it, but that is an odd choice for the linter | 19:01 |
| *** piet has joined #openstack-infra | 19:01 | |
| clarkb | oh! | 19:01 |
| AJaeger | clarkb: we set -x and -e in lines 24 and 26 | 19:01 |
| clarkb | AJaeger: even better :) hpcloud doesn't exist anymore, we can stop restricting memory | 19:01 |
| AJaeger | clarkb: should I remove the file? | 19:02 |
| clarkb | AJaeger: ya I think we can just do that instead | 19:02 |
| clarkb | AJaeger: and remove the one in nodepool/scripts | 19:02 |
| clarkb | and remove wherever they are calledfrom | 19:02 |
| *** fawadkhaliq has quit IRC | 19:02 | |
| *** eil397 has quit IRC | 19:03 | |
| greghaynes | clarkb: AJaeger Yea, we could make the linter not care about that. In dib land I like that for consistency sake, so it would need to be an option you all can set to turn it off | 19:03 |
| *** doug-fis_ has quit IRC | 19:03 | |
| *** notnownikki has quit IRC | 19:03 | |
| greghaynes | or if its a thing where we broke backwards compat then flip it to default off and we turn it on | 19:03 |
| *** ociuhandu has quit IRC | 19:03 | |
| *** doug-fi__ has quit IRC | 19:04 | |
| greghaynes | Is that what happened here - we broke backwards compat on that? | 19:04 |
| *** vgridnev has quit IRC | 19:04 | |
| greghaynes | http://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=b13512a7f49df65d1e34e83dcc12d588a38520ac | 19:04 |
| greghaynes | yep | 19:04 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove restrict-memory https://review.openstack.org/280364 | 19:04 |
| AJaeger | clarkb, removed everything ^ | 19:05 |
| clarkb | AJaeger: +2 | 19:06 |
| *** vgridnev has joined #openstack-infra | 19:06 | |
| clarkb | we really only needed that in hpcloud where we were using a large flavorthen trimming it down to 8GB ram | 19:06 |
| clarkb | and now that we aren't doing that we can go back to default flavor ram availability across the board | 19:07 |
| *** tzn has joined #openstack-infra | 19:07 | |
| AJaeger | clarkb, fungi, pabelanger, please have a look at https://review.openstack.org/#/c/280295/ - I think the change I150640990fb900318878563544af91e0a6043882 broke requirements repository | 19:08 |
| greghaynes | so, now that I know that job exists - for the disto's which we mirror (and therefore are stableish to test) we could pretty easily add test cases for building images with our elements similar to the dib gate | 19:08 |
| clarkb | greghaynes: ooh | 19:09 |
| clarkb | AJaeger: oh! | 19:09 |
| clarkb | AJaeger: are we cloning requirements into requirements and breaking things? | 19:09 |
| greghaynes | http://docs.openstack.org/developer/diskimage-builder/developer/developing_elements.html#testing-elements for info | 19:10 |
| AJaeger | clarkb: zuul-git-prep-upper-constraints does not work with requirements repo ;(((( | 19:10 |
| *** wolverineav has quit IRC | 19:10 | |
| AJaeger | clarkb: I'll fix... | 19:10 |
| clarkb | AJaeger: likely due to conlficting yaml keys | 19:10 |
| clarkb | AJaeger: this should be workaroundable in the zuul cloner map | 19:11 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Properly fail/trap in eval_run_d https://review.openstack.org/259734 | 19:11 |
| pabelanger | greghaynes: clarkb: nice! | 19:11 |
| *** wolverineav has joined #openstack-infra | 19:12 | |
| clarkb | greghaynes: pabelanger even the clone all repos step should be reasonable if we use the existing repo cache on the slaves | 19:12 |
| pabelanger | ya | 19:12 |
| greghaynes | clarkb: pabelanger Yep. The way it works out, too is that the first test takes a bit if it pulls in some packages but after that the caches are warm and they end up being really fast | 19:13 |
| pabelanger | clarkb: I plan to ping jeblair and mordred tomorrow to see what is needed to hack on afs.openstack.org for centos / fedora support. I have something local, but it is not integrated into k5start yet | 19:13 |
| greghaynes | DIB *really* needs fedora FWIW | 19:13 |
| clarkb | greghaynes: you mean fedora needs better mirrors? >_> | 19:15 |
| greghaynes | clarkb: heh, ideally both | 19:15 |
| clarkb | I wonder how often things fail for people just fedoraing | 19:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix constraints for requirements repo https://review.openstack.org/280368 | 19:15 |
| greghaynes | clarkb: I just saw a dib test fail in ovn when hitting the ovh fedora mirror | 19:15 |
| greghaynes | clarkb: It made me laugh | 19:15 |
| greghaynes | clarkb: er, fail in ovh | 19:15 |
| AJaeger | clarkb: 280368 is the fix | 19:15 |
| *** doug-f___ has quit IRC | 19:16 | |
| *** doug-fish has joined #openstack-infra | 19:16 | |
| *** doug-fish has quit IRC | 19:16 | |
| *** KimJ has joined #openstack-infra | 19:16 | |
| *** doug-fish has joined #openstack-infra | 19:17 | |
| clarkb | AJaeger: we should add a followup that adds the same checks to zuul-git-branch-prep-upper-requirements | 19:17 |
| clarkb | er upper-constraints | 19:17 |
| AJaeger | zuul-git-branch-prep-upper-constraints - followup or same change? I can do either... | 19:18 |
| gtmanfred | mordred: SpamapS Shrews still waiting on one more review of this https://review.openstack.org/#/c/268404/ | 19:18 |
| clarkb | AJaeger: followup, I already approved 280368 | 19:18 |
| pabelanger | greghaynes: clarkb: Ya, will be happy once they are on AFS | 19:18 |
| AJaeger | ah, ok | 19:19 |
| clarkb | greghaynes: that doesn't make me hopeful for using an ovh local mirror that we run :/ | 19:19 |
| *** piet has quit IRC | 19:19 | |
| AJaeger | clarkb: I fear we need to apply these change manually to jenkins, this will change lots of jobs again... | 19:20 |
| greghaynes | clarkb: I think it was a synchronization issue | 19:20 |
| greghaynes | or just a totally borked mirror | 19:20 |
| gtmanfred | Shrews: thanks :) | 19:21 |
| clarkb | AJaeger: good point I can trigger those when the change merges | 19:21 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Sync zuul-git-branch-prep-upper-constraints https://review.openstack.org/280369 | 19:22 |
| AJaeger | clarkb: second change ^ | 19:22 |
| AJaeger | clarkb: in that case, could you review https://review.openstack.org/278034 as well, please? That removes 400 jobs and might run also in the timeout | 19:23 |
| Shrews | gtmanfred: heh, i didn't even see your ping. i just happened to run out of patience just now :) | 19:23 |
| gtmanfred | nice :) | 19:23 |
| clarkb | AJaeger: done | 19:25 |
| *** doug-fish has quit IRC | 19:26 | |
| *** doug-fish has joined #openstack-infra | 19:26 | |
| AJaeger | clarkb: thanks. Now we should be at 7800 jobs instead of 8700 (yesterday) - so, the manual run might be a bit quicker... | 19:26 |
| clarkb | mysql people http://logs.openstack.org/46/278146/4/check/gate-nodepool-python27/b0c3504/console.html shows a nodepool test run where drop database hung until test timeout | 19:29 |
| *** imcsk8 has quit IRC | 19:29 | |
| *** imcsk8 has joined #openstack-infra | 19:29 | |
| clarkb | I assume the cursor to run show engine innodb status is going to be a transaction itself? | 19:29 |
| clarkb | if so I don't reall see anything there that is too crazy | 19:29 |
| *** salv-orlando has quit IRC | 19:30 | |
| olaph | Shrews out of patience? | 19:30 |
| *** KimJ has quit IRC | 19:32 | |
| *** markvoelker has joined #openstack-infra | 19:32 | |
| clarkb | I bet I can work around that mysql issue by forcibly killing any background threads that the test started | 19:32 |
| *** ldnunes_ has joined #openstack-infra | 19:32 | |
| Shrews | olaph: indeed | 19:33 |
| openstackgerrit | Merged openstack-infra/project-config: Rework devstack-jobs templating https://review.openstack.org/278034 | 19:35 |
| *** achanda has joined #openstack-infra | 19:35 | |
| *** ldnunes has quit IRC | 19:35 | |
| *** markvoelker has quit IRC | 19:37 | |
| *** _nadya_ has quit IRC | 19:38 | |
| *** tzn has quit IRC | 19:39 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Show innodb status before dropping db https://review.openstack.org/278146 | 19:40 |
| openstackgerrit | Merged openstack-infra/project-config: Fix constraints for requirements repo https://review.openstack.org/280368 | 19:48 |
| openstackgerrit | Merged openstack-infra/shade: granting and revoking privs to users and groups https://review.openstack.org/268404 | 19:49 |
| openstackgerrit | Christian Schwede proposed openstack-infra/nodepool: Retry creating of floating IPs on failures https://review.openstack.org/235165 | 19:49 |
| SpamapS | https://review.openstack.org/#/c/247296/ <-- can somebody explain why this isn't in any pipelines? (I tried reverify because I'm old-school, but I thought recheck would go ahead and just recheck it?) | 19:51 |
| *** kushal has quit IRC | 19:52 | |
| greghaynes | SpamapS: I suspect it collided with a zuul restart or jenkins explosion | 19:52 |
| greghaynes | SpamapS: I thought my adding a +1 would requeue it for gate but I guess not | 19:53 |
| SpamapS | greghaynes: but I did a recheck, and it rechecked.. some | 19:53 |
| pabelanger | don't you need to toggle workflow to requeue for gate? | 19:53 |
| greghaynes | SpamapS: Yea, that is really wierd it didnt end up in the gate queue | 19:53 |
| SpamapS | dunno, I +A'd it too | 19:54 |
| SpamapS | still only in check | 19:54 |
| SpamapS | but maybe has to pass that first? | 19:54 |
| greghaynes | SpamapS: That happened before, though - it went back through a check and passed | 19:55 |
| greghaynes | SpamapS: after you rechecked | 19:55 |
| *** daemontool has joined #openstack-infra | 19:55 | |
| greghaynes | so *shrug* | 19:55 |
| greghaynes | Maybe zuul really hates dib now | 19:55 |
| SpamapS | diskimage man | 19:55 |
| SpamapS | diskimage man | 19:55 |
| SpamapS | diskimage man hates zuul man | 19:55 |
| pabelanger | new gerrit UI is confusing to determine patchset order | 19:56 |
| SpamapS | they have a fight | 19:56 |
| SpamapS | zuul wins | 19:56 |
| SpamapS | diskimage man | 19:56 |
| *** smarcet has quit IRC | 19:56 | |
| SpamapS | pabelanger: new gerrit UI is like a really bad Google April Fools' day joke. | 19:56 |
| pabelanger | is 247296 the latest? Or depends on Related changes | 19:56 |
| *** e0ne has quit IRC | 19:56 | |
| pabelanger | SpamapS: no doubt | 19:57 |
| SpamapS | pabelanger: oh good point | 19:57 |
| SpamapS | I thought I saw no unmerged deps in gertty | 19:57 |
| AJaeger | SpamapS: reverify is obsolete and just an alias for recheck now. recheck will enqueue in the gate directly - since the gate is serialized, we only want changes in there that have passed | 19:57 |
| pabelanger | SpamapS: ya, checking gertty now | 19:57 |
| SpamapS | yeah I was wrong | 19:58 |
| SpamapS | it has 3 things ahead of it | 19:58 |
| SpamapS | all +A'd | 19:58 |
| AJaeger | clarkb: changes have merged for jenkins, could you check that puppet does not run into a timeout, please? | 19:58 |
| SpamapS | I just tried rechecking the bottom of the stack | 19:58 |
| greghaynes | oh, we all don't know how to read the new gerrit UI? | 19:58 |
| clarkb | AJaeger: I expect it to run into a timeout then I will just update it after | 19:58 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:58 | |
| AJaeger | thanks, clarkb | 19:59 |
| *** e0ne has joined #openstack-infra | 19:59 | |
| *** esikachev has quit IRC | 20:00 | |
| clarkb | AJaeger: but want to give it a chance to work automagicall | 20:00 |
| AJaeger | clarkb: sure! | 20:01 |
| AJaeger | looking at http://puppetboard.openstack.org/node/jenkins01.openstack.org puppet hasn't run for 1.5 hours - is that correct? | 20:02 |
| clarkb | hrm apperas to be the case | 20:03 |
| openstackgerrit | Christian Schwede proposed openstack-infra/nodepool: Cleanup floating ips even if server unavailable https://review.openstack.org/235381 | 20:03 |
| clarkb | 2016-02-15 20:00:56,991 p=28612 u=root | jenkins01.openstack.org : ok=7 changed=2 unreachable=0 failed=1 | 20:04 |
| clarkb | is what ansible says | 20:04 |
| clarkb | syslog logged some things but not why it failed :/ | 20:06 |
| *** aysyd has quit IRC | 20:06 | |
| clarkb | oh it is later in time | 20:06 |
| clarkb | AJaeger: it timed out laready that was fast | 20:07 |
| clarkb | so I will run jjb updates | 20:07 |
| clarkb | not sure what playbook was being used earlier today | 20:07 |
| clarkb | I don't see it in /etc/ansible/playbooks | 20:07 |
| *** tzn has joined #openstack-infra | 20:07 | |
| clarkb | need to figure out why puppetboard doesn't get the failed puppet run info | 20:08 |
| AJaeger | clarkb: there's system-config/playbooks/run_jjb.yaml | 20:08 |
| openstackgerrit | Merged openstack-infra/nodepool: Enable a fake statsd server for devstack testing https://review.openstack.org/226635 | 20:09 |
| clarkb | AJaeger: oh are we not installing that to /etc/ansible/playbooks? | 20:10 |
| clarkb | that should be fixed | 20:10 |
| AJaeger | clarkb: I have no idea how this works together... | 20:10 |
| *** daemontool_ has joined #openstack-infra | 20:11 | |
| clarkb | it is possible we aren't using a recursive file rsource on that dir | 20:11 |
| clarkb | and instead do it per file? | 20:11 |
| clarkb | in any case jenkins jobs are updating on 01 02 and 03 | 20:12 |
| clarkb | will do 04-07 after | 20:12 |
| *** dizquierdo has quit IRC | 20:12 | |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Dedicated specs repo for Freezer https://review.openstack.org/280307 | 20:12 |
| *** daemontool has quit IRC | 20:13 | |
| *** achanda has quit IRC | 20:15 | |
| clarkb | I am tempted to self approve https://review.openstack.org/#/c/232751/1 since it hsould help make nodepool tests more reliable | 20:15 |
| clarkb | AJaeger: INFO:jenkins_jobs.builder:Number of jobs generated: 8001 | 20:16 |
| *** ldnunes_ has quit IRC | 20:16 | |
| *** achanda has joined #openstack-infra | 20:16 | |
| clarkb | AJaeger: the linters jobs are getting deleted too fwiw | 20:16 |
| AJaeger | clarkb: my counting was off - still done from 874... | 20:17 |
| *** openstackgerrit has quit IRC | 20:17 | |
| *** openstackgerrit_ is now known as openstackgerrit | 20:17 | |
| jroll | AJaeger: mind landing this when you have a moment? https://review.openstack.org/#/c/265311/10 | 20:17 |
| *** openstackgerrit has quit IRC | 20:17 | |
| jroll | or ianw ^ | 20:17 |
| AJaeger | clarkb: they should have been removed already... | 20:17 |
| *** openstackgerrit_ has joined #openstack-infra | 20:17 | |
| AJaeger | jroll: will do | 20:17 |
| clarkb | ya not sure why that happend looks like on 01 only | 20:17 |
| *** dtardivel has quit IRC | 20:18 | |
| AJaeger | clarkb: ok to approve further changes while you manually run jjb? | 20:18 |
| *** openstackgerrit_ is now known as openstackgerrit | 20:18 | |
| jroll | AJaeger: thanks much :) | 20:18 |
| phschwartz | Has anyone seen nodepool ignore the min-ready and have extra servers sitting ready and then none of them every listing as online in jenkins? | 20:18 |
| *** openstackgerrit_ has joined #openstack-infra | 20:18 | |
| clarkb | phschwartz: if job demand goes above min-ready then you boot more instances | 20:19 |
| *** ddieterly has joined #openstack-infra | 20:19 | |
| clarkb | phschwartz: were they offlined by your jobs? eg jobs are in progress in jenkins? | 20:20 |
| phschwartz | clarkb: jobs in progress finished and offlined them, they were deleted and new were built. Those new ones never came online | 20:20 |
| clarkb | psanchez: but they are added to jenkins? | 20:21 |
| clarkb | if so then check that jenkins can ssh to them | 20:21 |
| clarkb | if not chekc that nodepool can interact with jenkins via the api | 20:22 |
| *** [HeOS] has joined #openstack-infra | 20:24 | |
| phschwartz | clarkb: jenkins can ssh into them | 20:24 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Show innodb status before dropping db https://review.openstack.org/278146 | 20:24 |
| clarkb | phschwartz: you confirmed this by manually onliningthem in jenkins? or? | 20:24 |
| AJaeger | clarkb: is it ok to approve further changes while you manually run jjb? Asking for 265311 | 20:24 |
| clarkb | AJaeger: would be best not to, otherwise puppetand manual jjb can fight each other | 20:25 |
| phschwartz | clarkb: I ssh'ed from the the jenkins host to them, let me try manually onlining them | 20:25 |
| phschwartz | clarkb: [02/15/16 20:19:32] [SSH] Opening SSH connection to 192.168.0.41:22. | 20:25 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 20:25 | |
| phschwartz | [02/15/16 20:19:34] [SSH] Authentication successful. | 20:25 |
| AJaeger | jroll: Let's wait - I try to approve tomorrow if nobody beats me to it... | 20:25 |
| AJaeger | clarkb: ok, thanks | 20:25 |
| phschwartz | clarkb: It never listed as online even from that, gets down to it and says channel started but that is it | 20:25 |
| clarkb | phschwartz: I do not know then, check your jenkins log to see if any errors were logged when they were initially added? | 20:26 |
| clarkb | phschwartz: and you are sure that jobs didn't run on them and offline them? | 20:26 |
| jroll | AJaeger: aha, I see | 20:26 |
| clarkb | the offline reason should be "offline due to gearman request" in that case | 20:26 |
| *** erlon has quit IRC | 20:26 | |
| phschwartz | clarkb: positive, the jobs are still sitting there listed as not run | 20:28 |
| clarkb | fungi: when you make it home any chance you can review https://review.openstack.org/#/c/232751/1 it is a one liner nodepool test improvement and with how flaky nodepool tests are I want to remove as many of the known bugs as possible | 20:28 |
| clarkb | phschwartz: ok I would check the jenkins log then | 20:28 |
| clarkb | phschwartz: see what it says about adding those hosts | 20:28 |
| ianw | jroll: +w'd, thanks, just seemed like a big one | 20:28 |
| AJaeger | ianw: please remove +w for now | 20:29 |
| jroll | ianw: I think they were waiting on a maintenance thing :/ | 20:29 |
| jroll | sorry | 20:29 |
| phschwartz | clarkb: so the logs are reporting for any brought up since the last successful run "offline because it’s not responding" | 20:29 |
| phschwartz | Let me dig there | 20:29 |
| *** e0ne has quit IRC | 20:29 | |
| AJaeger | ianw: clarkb is currenlty manually updating jenkins jobs | 20:29 |
| AJaeger | ianw: therefore let's not merge anything for now | 20:30 |
| * AJaeger put a -W on it | 20:30 | |
| ianw | AJaeger: ok ... guess who's just starting for the morning :) i think i'll just go grab some coffee and start again later :) | 20:30 |
| clarkb | otherwise I would have to disable puppet on all the machines | 20:30 |
| openstackgerrit | Merged openstack-infra/nodepool: Remove unecessary session.commit() https://review.openstack.org/277518 | 20:30 |
| clarkb | which I can do | 20:30 |
| AJaeger | ianw: no worries. | 20:30 |
| AJaeger | removed my -W again, so feel free to +A once clarkb gives the go ahead... | 20:31 |
| *** piet has joined #openstack-infra | 20:31 | |
| *** esikachev has joined #openstack-infra | 20:31 | |
| AJaeger | clarkb: I'll off soon, and won't +A anything more today ;) | 20:31 |
| clarkb | AJaeger: jenkins03 says it is done updating jobs | 20:31 |
| clarkb | I guess I need to check if the requirements python27 jobs are correct? | 20:32 |
| openstackgerrit | Merged openstack-infra/shade: Prepare functional test subunit stream for collection https://review.openstack.org/279246 | 20:32 |
| AJaeger | clarkb: recheck https://review.openstack.org/280295 | 20:32 |
| clarkb | AJaeger: well we have to wait for all jenkins to update before rechecking | 20:33 |
| clarkb | I just want to do a sanity check in the middle | 20:33 |
| *** piet has quit IRC | 20:33 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix dpkg element for Ubuntu Xenial https://review.openstack.org/278617 | 20:33 |
| clarkb | nope doesn't have the if around cp | 20:33 |
| clarkb | so what did we do wrong | 20:33 |
| *** piet has joined #openstack-infra | 20:34 | |
| AJaeger | clarkb: what? ;( | 20:34 |
| clarkb | it isn't updated on disk on jenkins03 in /etc/jenkins_jobs/config | 20:34 |
| clarkb | maybe it merged after puppet last ran there? | 20:34 |
| AJaeger | the last puppet run was - when I mentioned it - 1.5 hours old and the merge was newer | 20:35 |
| clarkb | AJaeger: it ran before that actually | 20:35 |
| clarkb | er | 20:35 |
| clarkb | it ran more recently than that but puppet failed on the jjb timeout | 20:35 |
| AJaeger | ah... | 20:35 |
| clarkb | and apparnetly it doesn't report the failures | 20:35 |
| phschwartz | clarkb: so interesting, the nodes were reported to be bad to zuul on a network failure that happened when our neutron ate itself. Even though new nodes came up and were good, it was auto setting them to used offline and not allowing them to come up | 20:36 |
| *** |-paul-| has joined #openstack-infra | 20:36 | |
| phschwartz | clarkb: I cleared the old hung jobs and deleted the nodes, new ones came up and are reporting good | 20:36 |
| AJaeger | clarkb: I'm confused - but have to leave now and won't be help to help further. | 20:38 |
| clarkb | AJaeger: ya I think it was just a race and hasn't run since then | 20:38 |
| clarkb | AJaeger: no problem | 20:38 |
| clarkb | I just need to get puppet to run there then we are good | 20:38 |
| clarkb | and I will rerun | 20:38 |
| AJaeger | clarkb: thanks and bye! | 20:38 |
| clarkb | our puppet run interval is really long now (close to an hour) | 20:39 |
| *** _nadya_ has joined #openstack-infra | 20:40 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:40 | |
| clarkb | I think https://review.openstack.org/#/c/232751/1 may possibly fix the db weirdness | 20:42 |
| *** salv-orlando has joined #openstack-infra | 20:42 | |
| greghaynes | clarkb: oh? | 20:44 |
| clarkb | greghaynes: ya looking at the job that hits that dfail it is that job | 20:44 |
| clarkb | s/job/tests/ | 20:45 |
| greghaynes | clarkb: huh. I totally believe stop is the culprit, but I dont understand how calling it twice could cause this | 20:45 |
| *** EricGonczer_ has joined #openstack-infra | 20:46 | |
| clarkb | greghaynes: I don't fully undersetand it either | 20:46 |
| clarkb | and I could be wrong | 20:46 |
| *** vgridnev has quit IRC | 20:47 | |
| *** piet has quit IRC | 20:47 | |
| *** piet has joined #openstack-infra | 20:48 | |
| *** esikachev has quit IRC | 20:48 | |
| greghaynes | clarkb: its always that test? | 20:49 |
| clarkb | oh hrm no it isn't | 20:50 |
| clarkb | though I suppose bad cleanup could ffect future tests too | 20:51 |
| *** sshnaidm has quit IRC | 20:51 | |
| greghaynes | we drop the db though, so that should be somewhat hard here | 20:52 |
| clarkb | the db dropping is what breaks | 20:52 |
| greghaynes | Yep | 20:52 |
| greghaynes | So that effecting a later test doesn't seem likely | 20:52 |
| clarkb | I see what you are saying new test should have a new db | 20:52 |
| greghaynes | yep | 20:52 |
| dims_ | jeblair : fungi : looks like mtreinish 's review is ready to be merged (https://review.openstack.org/#/c/279299/) and will help show up the oslo.* master jobs in the health check UI please take a look when you get a chance. | 20:53 |
| greghaynes | It might be worth adding a thread dump too | 20:53 |
| greghaynes | Its also annoying because it might be a very narrow race | 20:53 |
| greghaynes | so by the time we print anything out the transaction/thread has gone away | 20:53 |
| clarkb | greghaynes: ya that was going to be my next debug step since there isn't too much info from the innodb status | 20:53 |
| clarkb | there are 3 queries, one active | 20:54 |
| clarkb | which I think may be the uery for innodb status | 20:54 |
| greghaynes | do you have the link again? | 20:54 |
| clarkb | greghaynes: http://logs.openstack.org/46/278146/4/check/gate-nodepool-python27/b0c3504/console.html that is the most recent one with the debugging output included | 20:54 |
| *** rfolco has quit IRC | 20:55 | |
| *** ccrouch1 has joined #openstack-infra | 20:56 | |
| *** rfolco has joined #openstack-infra | 20:56 | |
| greghaynes | clarkb: hrm, ususally there is a way to dump locks too | 20:56 |
| clarkb | dims_: do you know if anyone confirmed that we are expiring data properly? | 20:56 |
| clarkb | greghaynes: the semaphores are in there | 20:56 |
| dims_ | clarkb : y, mtreinish got that cron job working and verified the logs | 20:56 |
| greghaynes | clarkb: that just looks like stats on locks though, not what the actual locks are | 20:56 |
| *** ccrouch has quit IRC | 20:57 | |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Dedicated specs repo for Freezer https://review.openstack.org/280307 | 20:57 |
| greghaynes | clarkb: looks like show open tables is the command | 20:58 |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Add experimental rally job for networking-odl. https://review.openstack.org/269174 | 20:58 |
| greghaynes | clarkb: or show full processlist | 20:58 |
| clarkb | greghaynes: show open stables? ya that | 20:58 |
| clarkb | we can do all of the above | 20:59 |
| *** rfolco has quit IRC | 21:00 | |
| * clarkb fires up mysql docker | 21:00 | |
| *** timcline has joined #openstack-infra | 21:01 | |
| greghaynes | clarkb: The other thing we can potentially do is turn on sqlalchemy debugging | 21:02 |
| *** zeih has joined #openstack-infra | 21:02 | |
| *** eil397 has joined #openstack-infra | 21:03 | |
| greghaynes | because that will be the next question once we find the lock - what query made it | 21:03 |
| clarkb | ok jenkins 01 -03 are updating again | 21:05 |
| clarkb | greghaynes: ya | 21:06 |
| *** ddieterly has quit IRC | 21:07 | |
| *** hockeynut_afk is now known as hockeynut | 21:07 | |
| *** ddieterly has joined #openstack-infra | 21:08 | |
| daemontool_ | can anyone help me to understand why this job is failing? https://review.openstack.org/#/c/280307/3 the irc one | 21:08 |
| *** piet has quit IRC | 21:09 | |
| clarkb | daemontool_: http://logs.openstack.org/07/280307/2/check/gate-project-config-irc-access/38e9d0d/console.html#_2016-02-15_20_25_38_660 | 21:09 |
| daemontool_ | clarkb, yes I saw that, but I can't undretand | 21:11 |
| daemontool_ | understand why it fails | 21:11 |
| daemontool_ | this is the file https://review.openstack.org/#/c/280307/3/gerritbot/channels.yaml | 21:11 |
| *** ddieterly has quit IRC | 21:12 | |
| daemontool_ | clarkb, ah, got it | 21:13 |
| daemontool_ | sorry | 21:13 |
| daemontool_ | ty :) | 21:13 |
| clarkb | daemontool_: line 568 column 3 | 21:13 |
| clarkb | I owuld start there | 21:13 |
| *** sarob_ has joined #openstack-infra | 21:14 | |
| *** ZZelle_ has joined #openstack-infra | 21:14 | |
| daemontool_ | lol ty | 21:14 |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Dedicated specs repo for Freezer https://review.openstack.org/280307 | 21:14 |
| *** sarob_ has quit IRC | 21:15 | |
| *** _sarob has joined #openstack-infra | 21:15 | |
| clarkb | greghaynes: ok I have show oen table dumping working, gonna add process listing then push new patch | 21:15 |
| greghaynes | clarkb: nice | 21:15 |
| clarkb | I even got it to fail with this bug locally once | 21:17 |
| clarkb | for the first time | 21:17 |
| clarkb | might not need to rely on jenkins | 21:17 |
| *** daemontool_ has quit IRC | 21:18 | |
| *** ddieterly has joined #openstack-infra | 21:18 | |
| greghaynes | clarkb: I've definitely hit it locally a couple times | 21:18 |
| clarkb | I hadn't until just now which I guess is agoo dthing | 21:18 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config: Unblock refstack functional testing job https://review.openstack.org/280408 | 21:19 |
| clarkb | greghaynes: processlist is what we want | 21:20 |
| clarkb | greghaynes: we get the query from it too | 21:20 |
| clarkb | patch in a few | 21:20 |
| greghaynes | clarkb: nice | 21:20 |
| *** _nadya_ has quit IRC | 21:22 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Show innodb status before dropping db https://review.openstack.org/278146 | 21:23 |
| *** ddieterly has quit IRC | 21:23 | |
| clarkb | greghaynes: ^ hopefully that catches the fail | 21:23 |
| *** hashar has joined #openstack-infra | 21:23 | |
| *** sridhar_ram has joined #openstack-infra | 21:25 | |
| *** EricGonczer_ has quit IRC | 21:26 | |
| *** zeih has quit IRC | 21:26 | |
| *** achanda has quit IRC | 21:29 | |
| *** piet has joined #openstack-infra | 21:29 | |
| clarkb | greghaynes: I have it running locally too fwiw | 21:29 |
| *** jsavak has joined #openstack-infra | 21:32 | |
| *** markvoelker has joined #openstack-infra | 21:33 | |
| openstackgerrit | Bertrand Roussel proposed openstack-infra/jenkins-job-builder: Description Setter Plugin support as builder https://review.openstack.org/280416 | 21:34 |
| *** jsavak has quit IRC | 21:37 | |
| *** MED has joined #openstack-infra | 21:37 | |
| *** markvoelker has quit IRC | 21:38 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add support for provider network options https://review.openstack.org/280417 | 21:38 |
| *** rkukura has joined #openstack-infra | 21:40 | |
| *** dkranz has quit IRC | 21:41 | |
| *** rfolco has joined #openstack-infra | 21:42 | |
| Shrews | jeblair: so, re: that gitpython 1.0.2 bug affecting gertty, we CAN change gertty to use the long option name (unified instead of U). i confirmed that worked with 1.0.2. if gertty is using other single letter options, though, they will still be affected | 21:44 |
| mgagne | clarkb: how can I debug change hanging in Zuul dashboard and timing out as "not merged" in logs ? I'm querying Gerrit CLI and it shows as MERGED | 21:46 |
| clarkb | mgagne: on the ssh api there is a change submission status query thing you can do | 21:47 |
| clarkb | mgagne: I believe that is what zuul checks | 21:47 |
| *** dprince has quit IRC | 21:47 | |
| mgagne | I feel like it's not querying much, sshd_log shows nothing | 21:48 |
| mgagne | I'm querying through SSH already | 21:48 |
| *** esp has joined #openstack-infra | 21:49 | |
| *** |-paul-| has quit IRC | 21:49 | |
| craige | o/ | 21:49 |
| mgagne | I thought it was bad apache cache but restarting apache changes nothing | 21:49 |
| clarkb | ya I think it uses ssh for that not http | 21:49 |
| mgagne | yea, tailing sshd_log shows nothing | 21:49 |
| mgagne | querying it myself shows status: MERGED | 21:50 |
| clarkb | mgagne: gerrit runs its own ssh | 21:50 |
| jhesketh | Morning | 21:50 |
| mgagne | sshd_log in gerrit logs | 21:50 |
| mgagne | not system | 21:50 |
| *** tcammann_ has joined #openstack-infra | 21:50 | |
| *** NobodyCa1 has joined #openstack-infra | 21:52 | |
| clarkb | oh right zuul adds an event for merge completed | 21:52 |
| clarkb | and results have to be processed before events | 21:53 |
| clarkb | mgagne: check if you have outstanding results to get through | 21:53 |
| mgagne | results are all in, only one change in flight (the jammed one) | 21:53 |
| clarkb | so result queue is 0 length? | 21:53 |
| clarkb | what is the events queue? | 21:53 |
| mgagne | in logs? UI? | 21:53 |
| clarkb | mgagne: zuul status page/json | 21:54 |
| *** deva_ has joined #openstack-infra | 21:54 | |
| *** dkranz has joined #openstack-infra | 21:54 | |
| mgagne | will have to submit a new change, this one went in | 21:54 |
| clarkb | its likely just queue processing if it is eventually going through | 21:54 |
| *** tcammann has joined #openstack-infra | 21:54 | |
| *** esp has quit IRC | 21:56 | |
| *** tcammann_ has quit IRC | 21:56 | |
| mgagne | 0 event | 21:56 |
| mgagne | 3 | 21:56 |
| clarkb | events will only be processed when the results queue is at 0 | 21:56 |
| mgagne | 7 now | 21:56 |
| *** tcammann_ has joined #openstack-infra | 21:57 | |
| mgagne | result queue is at 0 in UI | 21:57 |
| mgagne | will try to debug what those events are then | 21:57 |
| *** rfolco has quit IRC | 21:57 | |
| clarkb | so basically result comes in says jobs passed, zuul submits the merge, this results in a merge event, which hsa to be pulled off the events queue | 21:57 |
| *** NobodyCa1 has quit IRC | 21:57 | |
| *** NobodyCa1 has joined #openstack-infra | 21:58 | |
| *** thorst has quit IRC | 21:58 | |
| mgagne | I see events related to replication which Zuul seems to know what to do with | 21:58 |
| *** esp has joined #openstack-infra | 21:59 | |
| mgagne | will see if updating to latest master fixes the issue | 21:59 |
| *** jswarren_ has joined #openstack-infra | 22:00 | |
| *** deva_ has quit IRC | 22:00 | |
| *** tcammann has quit IRC | 22:01 | |
| *** tcammann_ has quit IRC | 22:02 | |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: Specify master branch for the devstack plugins periodic job https://review.openstack.org/280425 | 22:02 |
| tonyb | lifeless: Please take another look at: https://bugs.launchpad.net/python-fixtures/+bug/1542984 It's causing a little pain for the stable team. | 22:03 |
| openstack | Launchpad bug 1542984 in Python Fixtures "fixtures 1.2.0 isn't compatible with testtools 2.0.0" [Undecided,New] | 22:03 |
| *** salv-orl_ has joined #openstack-infra | 22:03 | |
| mgagne | same with latest master, is there a way to list pending events? | 22:03 |
| clarkb | mgagne: no | 22:03 |
| clarkb | I think you can only get the length of the quues but not list the contents | 22:03 |
| *** esp has quit IRC | 22:04 | |
| *** jswarren has quit IRC | 22:04 | |
| *** NobodyCa1 has quit IRC | 22:04 | |
| *** dkranz has quit IRC | 22:04 | |
| *** thorst has joined #openstack-infra | 22:04 | |
| clarkb | greghaynes: it almost looks like ti might be contention over multiple db drops? | 22:05 |
| *** salv-orlando has quit IRC | 22:05 | |
| greghaynes | clarkb: wah | 22:05 |
| *** ccrouch1 has quit IRC | 22:06 | |
| clarkb | greghaynes: since there are >1 test processes all deleting the DBs | 22:06 |
| greghaynes | clarkb: that shouldnt be an issue | 22:06 |
| greghaynes | clarkb: link? | 22:06 |
| clarkb | greghaynes: its local so I have to put a paste together | 22:06 |
| greghaynes | clarkb: ah, ok | 22:06 |
| *** timcline has quit IRC | 22:06 | |
| mgagne | I see those events coming in in logs, 7 total: http://paste.openstack.org/show/487044/ | 22:07 |
| *** elo has quit IRC | 22:07 | |
| mgagne | I'm not sure when or what is supposed to treat them. | 22:07 |
| clarkb | greghaynes: http://paste.openstack.org/show/487045/ | 22:08 |
| *** elo has joined #openstack-infra | 22:08 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use --overcloud-update in the upgrades job and ceph in the nonha https://review.openstack.org/260466 | 22:08 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 22:08 |
| clarkb | mgagne: the main zuul loop processes them. First it processes the results queue one by one until that hits 0 then it processes the events queue as long as results queue is len 0 | 22:08 |
| mgagne | and then this log appears: "DEBUG zuul.source.Gerrit: Change <Change 0x7fc8b03bcb10 19,1> did not appear in the git repo" | 22:09 |
| *** thorst has quit IRC | 22:09 | |
| mgagne | yea, I'm trying to see where it jams | 22:09 |
| clarkb | greghaynes: the first one shows that some other db is dropping that other db also failed and hung | 22:10 |
| mgagne | so I suspect _waitForRefSha is waiting for change to merge before other events are treated | 22:10 |
| clarkb | greghaynes: as if they somehow mutually locked each other out which seems unlikely | 22:10 |
| mgagne | hooooo could it be zuul needs anonymous access to gerrit UI to see if change merged?! | 22:11 |
| clarkb | mgagne: it does need to query gerrit for ref listsings | 22:11 |
| clarkb | /p/project/?refs-something | 22:11 |
| mgagne | https://github.com/openstack-infra/zuul/blob/514db293af344e15d7a1f1585c2ac84c3694eb0f/zuul/connection/gerrit.py#L386-L393 | 22:12 |
| mgagne | so this means it might silently ignore HTTP 401 errors and keep retrying forever | 22:12 |
| clarkb | fungi: ty for approving that test fix | 22:14 |
| *** piet has quit IRC | 22:14 | |
| *** ociuhandu has joined #openstack-infra | 22:18 | |
| fungi | np | 22:18 |
| *** dkranz has joined #openstack-infra | 22:19 | |
| fungi | trying to catch up after waaay too many hours in a car | 22:19 |
| clarkb | I am still updating jenkins masters | 22:19 |
| clarkb | 03 is done, 01 02 04 are in progress | 22:19 |
| fungi | clarkb: updating patch-wise? or config-wise? | 22:19 |
| openstackgerrit | Merged openstack-infra/nodepool: Stop nodepool once in tests https://review.openstack.org/232751 | 22:19 |
| *** jpr has quit IRC | 22:20 | |
| clarkb | config wise, running jenkins-jobs because a change that otuched most of python-jobs wa sjust merged | 22:20 |
| fungi | ahh | 22:20 |
| clarkb | and the puppet run timed out | 22:20 |
| clarkb | it is a fixfor the constraints macro on requirements repo jobs | 22:20 |
| fungi | so unrelated to the jjb cache issues yolanda observed earlier today | 22:20 |
| clarkb | correct | 22:20 |
| *** ccrouch has joined #openstack-infra | 22:21 | |
| fungi | and yeah, i thought we already had an override which avoided cloning the requirements repo in the constraints routine if the repo being tested was the requirements repo? | 22:21 |
| fungi | this was a known issue. i remember seeing a patch addressing it months back | 22:21 |
| *** _nadya_ has joined #openstack-infra | 22:22 | |
| *** itsuugo has quit IRC | 22:24 | |
| *** laurelm has quit IRC | 22:24 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 22:24 |
| *** laurelm has joined #openstack-infra | 22:24 | |
| clarkb | I wrote a thing that languished that is better about updating gearman with thedelata of changes | 22:25 |
| *** thorst has joined #openstack-infra | 22:25 | |
| clarkb | it had a couple bugs I just ran out of time to keep working on it (also java jenkins plugins are not the most exciting of things) | 22:25 |
| clarkb | if anyone else wants to pick it up that would be great | 22:25 |
| *** christiana has quit IRC | 22:25 | |
| fungi | aha, i was thinking of https://review.openstack.org/271257 but it only merged a few weeks ago | 22:26 |
| clarkb | fungi: that change was half of the fix, the cp at the end of the macro needs the same guarding | 22:26 |
| clarkb | otherwise cp fails on no source file to copy | 22:27 |
| fungi | gah | 22:27 |
| fungi | indeed | 22:27 |
| *** itsuugo has joined #openstack-infra | 22:27 | |
| clarkb | which we just merged the fix for (I single +2 approved bceause it was causing problems) | 22:27 |
| *** MED has quit IRC | 22:27 | |
| clarkb | and am now getting that deployed everywhere | 22:27 |
| fungi | thanks for finding/applying | 22:27 |
| clarkb | jenkins04 is uncooperative gonna put it in shutdown mode | 22:27 |
| *** _nadya_ has quit IRC | 22:27 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use --overcloud-update in the upgrades job and ceph in the nonha https://review.openstack.org/260466 | 22:28 |
| fungi | yeah, i had suspicions when i was looking at 04 earlier that it was havign trouble | 22:28 |
| *** piet has joined #openstack-infra | 22:28 | |
| *** tiswanso has quit IRC | 22:29 | |
| *** thorst has quit IRC | 22:29 | |
| clarkb | Ihave also been shepherding some nodepool changes trying to get them in before the restart | 22:29 |
| clarkb | which I should do soon | 22:29 |
| *** achanda has joined #openstack-infra | 22:30 | |
| *** ccrouch has quit IRC | 22:30 | |
| *** ajmiller has joined #openstack-infra | 22:31 | |
| *** ccrouch has joined #openstack-infra | 22:31 | |
| *** alivigni has quit IRC | 22:31 | |
| *** ccrouch has quit IRC | 22:31 | |
| *** ccrouch has joined #openstack-infra | 22:31 | |
| *** kgiusti has quit IRC | 22:32 | |
| *** loquacities has quit IRC | 22:33 | |
| *** regXboi has quit IRC | 22:34 | |
| clarkb | I can't get jenkins04 to pull up :/ | 22:34 |
| clarkb | top shows that it is spinning the cpus pretty har | 22:34 |
| clarkb | d | 22:34 |
| clarkb | skipping 04 and starting 06 | 22:35 |
| *** ddieterly has joined #openstack-infra | 22:36 | |
| tonyb | In review: https://review.openstack.org/#/c/278521/ it was noticed that we can't -/+1 vote on stable/* reviews. Looking at the mistral access at https://review.openstack.org/#/admin/projects/openstack/mistral,access it looks like "Registered Users" isn't the std. one going to https://review.openstack.org/#/admin/groups/uuid-acfb51541f636da17be5123c870d46dfbe06800c I get "https://review.openstack.org/#/admin/groups/uuid-acfb51541f636da17be5123c870d46dfbe0 | 22:36 |
| *** armax has quit IRC | 22:36 | |
| ianw | 2016-02-15 14:18:39,086 INFO nodepool.image.build.fedora-23 git://git.openstack.org/openstack/ansible-role-shade.git +master:master | 22:37 |
| ianw | 2016-02-15 14:18:39,100 INFO nodepool.image.build.fedora-23: fatal: Couldn't find remote ref master | 22:37 |
| clarkb | tonyb: yes we fubared Registered Users and need to clean it up | 22:37 |
| ianw | ^ do we know about this failure? | 22:37 |
| ianw | all the dib builds hit it | 22:37 |
| clarkb | tonyb: if however you rely on All-Projects for +/-1 it should work and fungi is working on fixing it for those that don't | 22:37 |
| *** piet has quit IRC | 22:37 | |
| tonyb | clarkb: what does that look like in a review? I don't mind doign it but the mistal.config looks ok to me | 22:37 |
| clarkb | ianw: yes, puppet on gerrit was disabled so the projects weren't there for cloning | 22:37 |
| clarkb | ianw: should be fixed in tomorrows builds | 22:37 |
| ianw | clarkb: cool, thanks | 22:38 |
| tonyb | clarkb: Oh okay if fungi is already doing it I'll stay out of it. | 22:38 |
| clarkb | tonyb: just remove the +/-1 code review for registered usres | 22:38 |
| *** piet has joined #openstack-infra | 22:38 | |
| *** loquacities has joined #openstack-infra | 22:38 | |
| clarkb | tonyb: it is redundant with all-projects iirc | 22:38 |
| fungi | tonyb: it involves gerrit admin privs (pushing a modification to a special metadata ref in each affected repo). or at least i think it does, was hoping to get zaro to confirm | 22:39 |
| clarkb | fungi: in this case they can just stop using registered users in their acl | 22:39 |
| fungi | but about to try it for openstack-dev/sandbox in a bit | 22:39 |
| tonyb | fungi: ok. | 22:39 |
| clarkb | since it doesn't add anything | 22:39 |
| fungi | ahh, yep any repos which don't need a reference to registered users should simply remove that | 22:40 |
| tonyb | fungi: How long has it been a problem? | 22:40 |
| clarkb | tonyb: since the upgrade in december | 22:40 |
| fungi | tonyb: since a few minutes after the 2.11 upgrade in mid december | 22:40 |
| *** dkranz has quit IRC | 22:40 | |
| tonyb | fungi, clarkb: are all repos that have "label-Code-Review = -1..+1 group Registered Users" affected? | 22:40 |
| clarkb | tonyb: yes | 22:40 |
| tonyb | clarkb, fungi: I ask as nova has that line but seems to work just fin | 22:41 |
| fungi | tonyb: or more broadly, any repos with acls mentioning registered users at all | 22:41 |
| *** ociuhandu has quit IRC | 22:41 | |
| clarkb | except for all-projects because gerrit seems to have handled that one | 22:41 |
| clarkb | oh I bet nova is old enough to be fine too | 22:41 |
| clarkb | so probably depends on when the acls were last updated | 22:41 |
| fungi | in that case i would have expected sandbox to be old enough as to be unaffected as well | 22:41 |
| fungi | but perhaps not | 22:41 |
| clarkb | ya nova is using the built in group for some reason | 22:41 |
| clarkb | fungi: I think it has to do with when the acl file was updated | 22:42 |
| fungi | i basically need to clone refs/meta/config from every repo and grep the groups file for any mention of registered users | 22:42 |
| clarkb | since jeepyb rewrites those file seach time | 22:42 |
| tonyb | clarkb: okay, that explains why in msiral Registered Users is a bad link but nova it's just text | 22:42 |
| clarkb | anyways nova shouldn't need it either | 22:42 |
| *** doug-fish has quit IRC | 22:42 | |
| fungi | right, ever since we dropped the central stable-maint-core group from the all-projects acl, overriding it shouldn't be necessary in individual project acls any longer | 22:43 |
| EmilienM | hello infra! if a zuul expert can look at this layout.yaml patch -> https://review.openstack.org/#/c/278072/ -- thanks! | 22:43 |
| fungi | basically since mid-2015 when we switched to per-project stable-maint groups | 22:43 |
| tonyb | fungi: Ahh okay. | 22:44 |
| lifeless | tonyb: replied to the -U thread | 22:44 |
| clarkb | EmilienM: I don't think skip if works that way | 22:44 |
| tonyb | fungi, clarkb: If I can help with my lay-knowledge/permissions let me know | 22:44 |
| clarkb | EmilienM: you need to remove the job from the list of jobs on that project | 22:44 |
| tonyb | fungi, clarkb otherwise I'll just get out of the way. | 22:44 |
| clarkb | tonyb: you could push a change that removes Registered users from the acl files in puppet then those will get pushed to gerri | 22:45 |
| tonyb | fungi, clarkb: I appreciate the time to help me understand. | 22:45 |
| EmilienM | clarkb: ok I see | 22:45 |
| tonyb | clarkb: in puppet? not in project-config/gerrit/acls/*/*config? | 22:46 |
| clarkb | tonyb: that is where | 22:46 |
| tonyb | lifeless: Thanks. | 22:46 |
| clarkb | s/puppet/our configs/ | 22:46 |
| tonyb | clarkb: okay. | 22:46 |
| clarkb | tonyb: some repos will need it like sandbox because we allow +/-2 | 22:46 |
| tonyb | clarkb: one big chnage or several small ones? | 22:46 |
| clarkb | tonyb: but any repo using it for just +/-1 can remove it | 22:46 |
| clarkb | tonyb: one big one is probably fine | 22:46 |
| tonyb | clarkb: okay /me does a thing. | 22:46 |
| clarkb | just started job update on jenkins.o.o, I did have to flush the cache there I think yolanda must've missed it | 22:47 |
| clarkb | fungi: re jenkins04 I still can't get to it via the web ui | 22:47 |
| clarkb | fungi: I wonder if we can just restart it | 22:47 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet-ceph: do not run integ jobs for scenario002 https://review.openstack.org/278072 | 22:48 |
| EmilienM | clarkb: something like this ? ^ | 22:48 |
| *** doug-fish has joined #openstack-infra | 22:48 | |
| clarkb | there is one ready and one used node on jenkins04 according to nodepool | 22:48 |
| clarkb | EmilienM: yes | 22:48 |
| EmilienM | cool thx again | 22:49 |
| fungi | clarkb: i basically had to just kill and restart 01 and 05 on sunday | 22:49 |
| fungi | deleted all their nodes from nodepool and all their slaves entries from their configs while they were down too | 22:49 |
| clarkb | lifeless: I don't think your response to that thread is correct unless constraints update deps even without a -U | 22:49 |
| lifeless | clarkb: that is what constraints does | 22:50 |
| clarkb | constraints updates deps? | 22:50 |
| clarkb | without a -U ? | 22:50 |
| lifeless | clarkb: a constraints enabled tox script invokes pip install -c upper-constraints.txt install project.... | 22:51 |
| clarkb | lifeless: yes I know | 22:51 |
| clarkb | lifeless: but what happens if the constraint for foo updates, will rerunning that command without -U update the constraint? | 22:51 |
| *** doug-fish has quit IRC | 22:51 | |
| lifeless | clarkb: of course, pip always makes sure everything is satisfied | 22:52 |
| lifeless | clarkb: -U is pretty much never needed unless lower bounds are wrong | 22:52 |
| clarkb | it didn't always do that | 22:52 |
| clarkb | you had to use -U | 22:52 |
| *** doug-fish has joined #openstack-infra | 22:52 | |
| clarkb | or use tox -r | 22:52 |
| lifeless | clarkb: what it doesn't do is check backlinks | 22:52 |
| *** tzn has quit IRC | 22:52 | |
| lifeless | clarkb: I'm about 99.99% confident that folk have misdiagnosed pip's behaviour in the past | 22:53 |
| lifeless | clarkb: and I don't want to rabbit hole down a history check | 22:53 |
| clarkb | ok | 22:53 |
| lifeless | clarkb: so lets ignore what it *did* and focus on what it does | 22:53 |
| clarkb | I am 99% sure that we didn't misdiagnose it but ok | 22:53 |
| clarkb | :) | 22:53 |
| lifeless | if you have A > 1 | 22:53 |
| lifeless | as a dep | 22:53 |
| *** MED has joined #openstack-infra | 22:53 | |
| lifeless | and you pip install . | 22:53 |
| lifeless | and you have A==1 already | 22:53 |
| lifeless | A will get upgraded | 22:53 |
| lifeless | without -U | 22:53 |
| lifeless | if that dep is encountered multiple times and the first occurence is A > 0.5 | 22:54 |
| lifeless | then it won't upgrade it | 22:54 |
| *** daemontool has joined #openstack-infra | 22:54 | |
| lifeless | because of the needs-a-resolver-bug | 22:54 |
| *** rkukura has quit IRC | 22:54 | |
| *** doug-fish has quit IRC | 22:54 | |
| clarkb | with old pip it would say "A is already installed not upgrading without --upgrade" | 22:54 |
| *** rfolco has joined #openstack-infra | 22:55 | |
| lifeless | clarkb: only if A satisfied the requirement | 22:55 |
| lifeless | clarkb: *please* lets not do this. | 22:55 |
| clarkb | I am not doing anything... | 22:55 |
| *** doug-fish has joined #openstack-infra | 22:55 | |
| * fungi assumes "this" is discussing previous pip behaviors no longer relevant in latest pip | 22:55 | |
| clarkb | I merely asked what constraints does | 22:55 |
| lifeless | fungi: yes, indeed | 22:55 |
| clarkb | sounds like constraints because it completely lists all deps should update everything that needs updates to satisfy the constraint? | 22:56 |
| lifeless | ok so to finish joining the dots | 22:56 |
| openstackgerrit | Merged openstack/diskimage-builder: Use dnf to cleanup old kernels https://review.openstack.org/249539 | 22:56 |
| lifeless | when -c is the first parameter, all the entries in the constraints file are processed first | 22:56 |
| lifeless | which means even with the resolver bug | 22:56 |
| *** lbragstad_ has joined #openstack-infra | 22:56 | |
| lifeless | all their dep rules are examined as-is rather than being ignored due to the resolver-needed bug | 22:57 |
| *** mriedem is now known as mriedem_away | 22:57 | |
| clarkb | the only gap with that would be adding new requirements that are not in constraints yet but that is a small enough thing that liekly won't hurt us much | 22:57 |
| lifeless | right | 22:57 |
| *** MED has quit IRC | 22:57 | |
| *** rossella_s has quit IRC | 22:57 | |
| clarkb | but that isn't a general constraints ism | 22:58 |
| *** daemontool_ has joined #openstack-infra | 22:58 | |
| clarkb | that is a how openstack uses constraints | 22:58 |
| *** rossella_s has joined #openstack-infra | 22:58 | |
| clarkb | (we list all the things) | 22:58 |
| lifeless | yah | 22:58 |
| lifeless | I mean, thats the intended use case | 22:58 |
| clarkb | so we are both correct :) | 22:58 |
| lifeless | but its not the only way to use the file | 22:58 |
| *** daemontool has quit IRC | 22:58 | |
| clarkb | -U is required in the general case, but not in our usage of it | 22:58 |
| lifeless | clarkb: no, its not required in the general case | 22:59 |
| clarkb | well it is required to keep everything up to date | 22:59 |
| lifeless | clarkb: but that is a separate discussion | 22:59 |
| fungi | it's required in the general case if you want to attempt to run latest of everything | 22:59 |
| clarkb | fungi: yup | 22:59 |
| lifeless | yes, thats true | 22:59 |
| fungi | which was where the "devs running tox locally see different behavior than the gate" logic stemmed from | 22:59 |
| lifeless | if folk are raising their minimums appropriately it should never be needed to have things work | 22:59 |
| fungi | -U was _a_ means (pre-constraints) of makign a best effort attempt to normalize the versions of everything installed | 23:00 |
| *** doug-fish has quit IRC | 23:00 | |
| fungi | normalize, i mean, between fresh deployments and runs on existing "dirty" environments | 23:00 |
| lifeless | yah | 23:00 |
| clarkb | ok I am going to restart jenkins04 now | 23:01 |
| lifeless | constraints are the stock best practice answer to that:) | 23:01 |
| fungi | thanks clarkb! | 23:01 |
| fungi | right, constraints provides a much better guarantee than -U, and also gives us a lot more control over the actual versions used if we want | 23:01 |
| clarkb | but it has the gap that -U would protect against which is the same for the general case | 23:02 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add class to deal with cacert on infra cloud https://review.openstack.org/278362 | 23:02 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Update SSL configuration https://review.openstack.org/279286 | 23:02 |
| fungi | though at the cost of needing to pre-compute some working coinstallable set of versions of everything | 23:02 |
| lifeless | clarkb: so actually, when we add requirements we add them to constraints in the same commit | 23:02 |
| lifeless | clarkb: I'm not sure there *is* a race | 23:02 |
| lifeless | clarkb: if we didn't do that, there would be a race | 23:03 |
| fungi | the race is on projects using requirements not tracked in the global requirements/constraints list. but i'm not too heartbroken of them | 23:03 |
| fungi | well, not race, just plain gap | 23:03 |
| lifeless | right | 23:03 |
| lifeless | they aren't covered | 23:03 |
| lifeless | they could use a local constraints file if they wanted | 23:03 |
| clarkb | and 04 is coming back now | 23:03 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add class to deal with cacert on infra cloud https://review.openstack.org/278362 | 23:03 |
| lifeless | they aren't aiming for co-installability (by definition) | 23:04 |
| fungi | yeah, they could supply a custom constraints list if desired | 23:04 |
| clarkb | lifeless: the race is between I added a thing locally and it didn't work upstream | 23:04 |
| clarkb | lifeless: its a small race we can live with | 23:04 |
| clarkb | but it is a race | 23:04 |
| lifeless | clarkb: only in non-synced project right? | 23:04 |
| lifeless | clarkb: or do you think it exists for e.g. nova? | 23:04 |
| clarkb | in any project | 23:04 |
| fungi | and also could be skinned by adding to a local divergent copy of the constraints list at the same time | 23:04 |
| *** rcernin has quit IRC | 23:05 | |
| lifeless | clarkb: so, if you're adding a requirement locally, it will fail jenkins | 23:05 |
| clarkb | lifeless: you could test your change locally it works, push to nova, and push to requirements at that point they fail | 23:05 |
| fungi | though a bit more work, granted | 23:05 |
| clarkb | lifeless: then you rerun locally and they pass | 23:05 |
| *** daemontool has joined #openstack-infra | 23:05 | |
| clarkb | you won't see the fail until you pip -U locally or tox -r | 23:05 |
| fungi | lifeless: "locally" on your dev system, not upstream/tested by ci yet | 23:05 |
| lifeless | clarkb: the new dep is either a) in constraints, so you'll get the same locally and in CI | 23:05 |
| clarkb | if it is not in constraints | 23:05 |
| lifeless | clarkb: then it won't be in g-r either, and will simply fail CI | 23:06 |
| lifeless | clarkb: because of the cross-check for being in g-r | 23:06 |
| *** daemontool_ has quit IRC | 23:06 | |
| clarkb | it will fail the requirements check | 23:06 |
| lifeless | right | 23:06 |
| clarkb | that is expected | 23:06 |
| clarkb | but I am talking about the other jobs | 23:06 |
| lifeless | ok, I see. So we need to tell folk not to diagnose failures in other jobs unless the requirements check passed | 23:06 |
| *** tpsilva has quit IRC | 23:06 | |
| clarkb | that is one way to deal with it | 23:07 |
| fungi | my take on it is that if you're adding a new dependency locally to your copy of requirements.txt, then if you really want to test that the same way it would run under constraints, add it to a local custom copy of the constraints list | 23:07 |
| jasondotstar | hi all | 23:07 |
| *** sshnaidm has joined #openstack-infra | 23:07 | |
| fungi | in the same way that when you decide to go forward upstream with it you'll need to add it to both global requirements and global constraints, you need to test locally with local requirements and local constraints | 23:07 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Remove Registered Users from seeting -1..1 in Code-Review https://review.openstack.org/280439 | 23:08 |
| fungi | in that _specific_ case anyway | 23:08 |
| jasondotstar | attempting to prune a couple projects from the stackforge/ namespace. is it simply a matter of removing them from project-config (via projects.yaml) | 23:08 |
| jasondotstar | or there are addn'l steps? | 23:08 |
| fungi | jasondotstar: there is no way to delete a repo in gerrit | 23:09 |
| *** piet has quit IRC | 23:09 | |
| fungi | once it's there, it's there | 23:09 |
| *** moravec has quit IRC | 23:09 | |
| fungi | we have a somewhat hackish solution to rename repos | 23:09 |
| *** hashar has quit IRC | 23:09 | |
| fungi | but if the repo is defunct, the best solution is to push a commit deleting all files and adding a readme with a note that the head^1 commit is teh last state of the repo pre-retirement, then locking down the gerrit acl to read-only for it | 23:10 |
| clarkb | jenkins04 is now updating jobs | 23:10 |
| jasondotstar | fungi: +1 | 23:10 |
| *** krtaylor has quit IRC | 23:11 | |
| fungi | jasondotstar: that way if anyone clones it, they know right away that it's no longer maintained, and nobody will be able to push new changes for review on it until someone brings it back from retirement (revert HEAD, set gerrit acl back to active) | 23:11 |
| *** piet has joined #openstack-infra | 23:11 | |
| *** nelsnelson has quit IRC | 23:12 | |
| *** shashank_hegde has quit IRC | 23:12 | |
| *** yamamoto has quit IRC | 23:13 | |
| *** yamamoto has joined #openstack-infra | 23:13 | |
| *** ajmiller has quit IRC | 23:13 | |
| *** yamamoto has quit IRC | 23:14 | |
| *** sridhar_ram has quit IRC | 23:21 | |
| *** dimtruck is now known as zz_dimtruck | 23:22 | |
| *** krtaylor has joined #openstack-infra | 23:23 | |
| *** achanda has quit IRC | 23:25 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:25 | |
| *** shashank_hegde has joined #openstack-infra | 23:26 | |
| clarkb | greghaynes: http://logs.openstack.org/46/278146/7/check/gate-nodepool-python27/f1bb4d6/console.html now I really have no idea what is going on | 23:26 |
| clarkb | greghaynes: that seems to imply nothing has that db locked | 23:26 |
| greghaynes | clarkb: haha | 23:26 |
| greghaynes | clarkb: so, it could be a client bug | 23:26 |
| greghaynes | clarkb: let me grok that fail for a min... | 23:27 |
| greghaynes | clarkb: it does look like there are several locks out on that db | 23:28 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Promote neutron-src-neutron-lib job to check/gate https://review.openstack.org/280443 | 23:28 |
| greghaynes | clarkb: db HgmuyGxM_3388 | 23:28 |
| clarkb | greghaynes: but look at the lock columns | 23:28 |
| clarkb | they are all 0 | 23:28 |
| clarkb | in use and name lock | 23:29 |
| clarkb | I think they are just idle connections? | 23:29 |
| clarkb | maybe not | 23:29 |
| *** yuanying_flu is now known as yuanying | 23:29 | |
| greghaynes | clarkb: they are open tables with no locks | 23:30 |
| clarkb | yup that is how I read it | 23:30 |
| greghaynes | clarkb: this is when we need one of the mysql gurus... | 23:31 |
| clarkb | ++ | 23:31 |
| greghaynes | mordred: SpamapS ^ | 23:31 |
| fungi | remote: error: Invalid project configuration: | 23:31 |
| fungi | remote: error: project.config: group "Registered Users" not in groups | 23:31 |
| fungi | le sigh | 23:31 |
| fungi | that's what happens if i attempt to remove Registered Users from the groups file | 23:32 |
| clarkb | fungi: when you push? | 23:32 |
| fungi | yep | 23:32 |
| clarkb | fungi: I think you may need Registered Users in the groups file but with a different uuid | 23:32 |
| fungi | i'm checking nova for comparison | 23:32 |
| clarkb | something like system:registered-users | 23:32 |
| fungi | global:Registered-Users Registered Users | 23:33 |
| fungi | seems to be the magic | 23:33 |
| *** achanda has joined #openstack-infra | 23:33 | |
| *** markvoelker has joined #openstack-infra | 23:34 | |
| fungi | neat, the groups file is hard tab delineated too | 23:34 |
| clarkb | jenkins05 is done | 23:34 |
| clarkb | yup | 23:35 |
| fungi | success! https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access | 23:35 |
| fungi | now time to checkout refs/meta/config from all ~1k repos and play spot-the-error | 23:35 |
| clarkb | fungi: will jeepyb need to be updated to handle the global:registered uers thing? | 23:35 |
| clarkb | otherwise next update could overwrite | 23:35 |
| fungi | clarkb: zaro already patched that | 23:35 |
| clarkb | oh then you chould just run jeepyb | 23:36 |
| clarkb | bit brute force but it is already coded to update all 1k :) | 23:36 |
| fungi | https://review.openstack.org/248126 merged before the maintenance | 23:36 |
| greghaynes | clarkb: I am suddenly confused by that timeout traceback | 23:37 |
| fungi | anteaya: madorn: i believe the voting permissions on openstack-dev/sandbox are now corrected, fyi | 23:37 |
| clarkb | greghaynes: that makes two of us | 23:37 |
| clarkb | woo the commands + gearman nodepool change should finally merge | 23:38 |
| clarkb | I will restart nodepoold after that | 23:38 |
| *** markvoelker has quit IRC | 23:38 | |
| *** Jeffrey4l has joined #openstack-infra | 23:39 | |
| greghaynes | clarkb: That last where timeoutexception is raised... any idea what is going on there? | 23:39 |
| greghaynes | I dont believe that the socket read call actually calls back in to fixtures | 23:39 |
| clarkb | greghaynes: ya its the sigalarm handler, which runs in the main thread so that pymysql fixtures stuff shows up | 23:40 |
| clarkb | greghaynes: its not an honest traceback | 23:40 |
| clarkb | except that it kind of is due to how python does signal hndling | 23:40 |
| clarkb | so pymysql is blocking on the recv | 23:40 |
| clarkb | alarm comes in, calls handler, because you are in same thread you get pushed onto frame below that block | 23:40 |
| greghaynes | clarkb: yea, so then what is that actually a traceback of? just the traceback of the main thread + the sigalarm handler appended? | 23:40 |
| clarkb | yes | 23:41 |
| clarkb | because it literally just adds a stack frame below whereever the current frame is aiui | 23:41 |
| *** dims_ has quit IRC | 23:41 | |
| greghaynes | huh | 23:41 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Remove Registered Users from seeting -1..1 in Code-Review https://review.openstack.org/280439 | 23:44 |
| tonyb | clarkb, lifeless: Thanks for the reply and discussion here. I'll bring that up with the stable-maint team to see if there will be support for fixing all the tox.ini files in stable/kilo as a start liberty and master can wait whiel constraints propgate. | 23:47 |
| SpamapS | clarkb: do you have any theories as to what might be happening? | 23:47 |
| tonyb | lifeless: I'd still like your opinion on https://bugs.launchpad.net/python-fixtures/+bug/1542984 as pip -U is tangental in that case. | 23:47 |
| openstack | Launchpad bug 1542984 in Python Fixtures "fixtures 1.2.0 isn't compatible with testtools 2.0.0" [Undecided,New] | 23:47 |
| lifeless | clarkb: greghaynes: the VM injects the frames after returning from the signal; signals are not actually processed inline | 23:47 |
| clarkb | SpamapS: re the mysql thing? | 23:48 |
| lifeless | tonyb: well, I' think actually you want me to change my opinion there ;) | 23:48 |
| *** yamamoto has joined #openstack-infra | 23:48 | |
| greghaynes | lifeless: ah, makes sense | 23:48 |
| greghaynes | well, makes sense when reading that traceback | 23:48 |
| *** bpokorny has quit IRC | 23:48 | |
| lifeless | greghaynes: because you know, malloc may not be usable mid-signal. And python kinda wants that to work | 23:48 |
| tonyb | lifeless: True to a degree | 23:48 |
| lifeless | tonyb: I need to go get food and do a couple things here, I'll ping you in a bit | 23:49 |
| tonyb | lifeless: Thanks. | 23:49 |
| greghaynes | SpamapS: The best theory we had was that for some reason nodepool was holding out a table lock when our fixture cleanup to drop db was being called | 23:49 |
| greghaynes | SpamapS: but based on that info that theory is looking wrong | 23:50 |
| SpamapS | what made you think there was a table lock? | 23:50 |
| SpamapS | oh drop db yes, that will lock the table before dropping it | 23:50 |
| greghaynes | SpamapS: Nothing other than the fact that drop database was hanging | 23:50 |
| SpamapS | but something else had the table excl locked? | 23:51 |
| SpamapS | you think | 23:51 |
| clarkb | yes, we call drop database and that test times out | 23:51 |
| greghaynes | SpamapS: Nodepool doesn't take out any explicit locks | 23:51 |
| SpamapS | Problem is that innodb status won't show table locks | 23:51 |
| clarkb | so theory was something was locked within that databse | 23:51 |
| SpamapS | alters make locks | 23:51 |
| clarkb | SpamapS: we added show full processlist and show open tables | 23:51 |
| SpamapS | also create .. select | 23:51 |
| SpamapS | oh i see now | 23:52 |
| *** tzn has joined #openstack-infra | 23:52 | |
| SpamapS | but none seem to have excl locks | 23:53 |
| greghaynes | SpamapS: Yea, thats what were seeing too :( | 23:53 |
| greghaynes | so now we have no good theory | 23:53 |
| clarkb | I guess it is possible mysql is just busy? | 23:53 |
| SpamapS | drop db timed out, right? | 23:53 |
| clarkb | SpamapS: yes | 23:53 |
| greghaynes | It almost looks like the tcp connection died and tcp timeout hasnt happened | 23:53 |
| SpamapS | that does have to wait for every file to be rm'd | 23:53 |
| clarkb | SpamapS: well specifically the test timeout hits while we attempt to drop | 23:53 |
| SpamapS | which means fully purged | 23:53 |
| greghaynes | hrm | 23:53 |
| SpamapS | if there's something big | 23:53 |
| SpamapS | and disk is sucking | 23:54 |
| greghaynes | It shouldnt be that big, but its happening when this is part of a test run on a random vm | 23:54 |
| clarkb | ya so could be IO/cpu pain | 23:54 |
| clarkb | also all cpus are running tests | 23:54 |
| clarkb | maybe we scale that back | 23:54 |
| SpamapS | is it critical that the db be dropped? | 23:55 |
| greghaynes | SpamapS: Not really | 23:55 |
| SpamapS | like is it created/dropped over and over and for cleanliness dropped one more time? | 23:55 |
| *** yamamoto has quit IRC | 23:55 | |
| greghaynes | SpamapS: The db name is random, so it wont cause issues to not be dropped | 23:55 |
| clarkb | it is ciritical for local tests yes | 23:55 |
| greghaynes | clarkb: oh? | 23:55 |
| clarkb | I don't want thousands of dbs in my local test bed | 23:55 |
| greghaynes | yea, other than the wasted space | 23:56 |
| clarkb | but that gives us a couple things to try, drop the concurrency level and increase timeouts | 23:56 |
| *** christiana has joined #openstack-infra | 23:56 | |
| *** dims has joined #openstack-infra | 23:57 | |
| greghaynes | clarkb: Isn't there a way to do a per testcase clean up? | 23:57 |
| clarkb | greghaynes: that is what this is | 23:57 |
| greghaynes | My terms are probably bad | 23:57 |
| greghaynes | not per test, but per class | 23:57 |
| clarkb | not really | 23:57 |
| clarkb | not if you want parallel test execution | 23:57 |
| greghaynes | Why? | 23:57 |
| *** tzn has quit IRC | 23:57 | |
| clarkb | because of how testr groups and assigns the tests | 23:58 |
| greghaynes | Sure, so would it have class * number of threads? | 23:58 |
| clarkb | you oculd do it but it isn't "clean" | 23:58 |
| clarkb | you owuld call setup class multiple times | 23:58 |
| SpamapS | clarkb: is this a bastion mysqld, or just "whatever is on localhost" ? | 23:58 |
| clarkb | SpamapS: it is localhost typically | 23:58 |
| greghaynes | clarkb: thats fine, I am thinking do it in a way where things wont conflict (pick a unique prefix per class setup) | 23:58 |
| SpamapS | Ok, was going to say, if you manage your own mysqld, you can just lazy rm -r the datadir at the end. | 23:59 |
| clarkb | greghaynes: pid is the uniqueness | 23:59 |
| clarkb | SpamapS: we don't have permissions to disk like that | 23:59 |
| clarkb | we could get them I suppose but thats super hacky | 23:59 |
| SpamapS | clarkb: you can run mysqld in a tempdir. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!