| *** gyee has quit IRC | 00:04 | |
| *** jamesmcarthur has joined #openstack-infra | 00:10 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** slaweq has quit IRC | 00:16 | |
| *** e0ne has joined #openstack-infra | 00:20 | |
| ianw | sorry, back from dealing with a lost (physical) key ... | 00:20 |
|---|---|---|
| ianw | ummm | 00:20 |
| ianw | clarkb: thanks, looks like i just forgot it. was going to check in on that today | 00:21 |
| clarkb | ianw: I was just working my way through the stack of post ptg followup items and that was on the list :) | 00:22 |
| clarkb | hopefully that is the final piece and we start booting instances there | 00:23 |
| ianw | ++ ... moving the nb03 builder in linaro-london seems to have had the desired effect of getting builds out | 00:24 |
| *** jamesmcarthur has quit IRC | 00:29 | |
| *** longkb has joined #openstack-infra | 00:29 | |
| *** eernst has joined #openstack-infra | 00:32 | |
| openstackgerrit | Merged openstack-infra/project-config master: Upload arm64 images to linaro-london cloud https://review.openstack.org/603249 | 00:32 |
| *** eernst has quit IRC | 00:36 | |
| *** markvoelker has quit IRC | 00:44 | |
| *** jamesmcarthur has joined #openstack-infra | 00:49 | |
| *** rh-jelabarre has quit IRC | 00:59 | |
| *** anteaya has quit IRC | 01:09 | |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** jamesmcarthur has joined #openstack-infra | 01:12 | |
| dmsimard | ianw: huh, it sounds like it'd be a good idea to localize nodepool builders to each provider | 01:19 |
| ianw | dmsimard: oh, in this case it was that the nb03 in the cn1 cloud kept falling out of zookeeper | 01:20 |
| ianw | i suspect we might have done better with SSL | 01:20 |
| *** e0ne has quit IRC | 01:20 | |
| *** bobh has joined #openstack-infra | 01:22 | |
| *** jamesmcarthur has quit IRC | 01:22 | |
| clarkb | dmsimard: at least the way we run them they can upload to any configured cloud | 01:24 |
| clarkb | which is intentional since we want the same image in all the regions | 01:25 |
| dmsimard | clarkb: yeah with a second thought the idea isn't so good | 01:25 |
| dmsimard | carry on :D | 01:25 |
| *** e0ne has joined #openstack-infra | 01:27 | |
| *** jamesmcarthur has joined #openstack-infra | 01:33 | |
| pabelanger | clarkb: last year I did look at doing: https://review.openstack.org/486199/ so that we did add-build-sshkey only on localhost but never landed it to test, but agree, we could have localhost only plays in base / base-minimal jobs. | 01:33 |
| *** mrsoul has quit IRC | 01:35 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Add puppet-placement to deploy OpenStack Placement https://review.openstack.org/602869 | 01:36 |
| *** felipemonteiro has quit IRC | 01:38 | |
| *** dave-mccowan has quit IRC | 01:42 | |
| *** felipemonteiro has joined #openstack-infra | 01:54 | |
| *** mriedem_away is now known as mriedem | 01:55 | |
| *** dave-mccowan has joined #openstack-infra | 02:00 | |
| *** jamesmcarthur has quit IRC | 02:01 | |
| *** diablo_rojo has quit IRC | 02:02 | |
| *** felipemonteiro has quit IRC | 02:10 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** ramishra has joined #openstack-infra | 02:18 | |
| *** dave-mccowan has quit IRC | 02:20 | |
| *** rlandy|bbl is now known as rlandy | 02:23 | |
| *** hongbin has joined #openstack-infra | 02:37 | |
| *** mriedem has quit IRC | 02:38 | |
| *** e0ne has quit IRC | 02:40 | |
| *** roman_g has quit IRC | 02:46 | |
| *** e0ne has joined #openstack-infra | 02:48 | |
| prometheanfire | clarkb: still seems odd (the chroot thing), wonder why it was activated (started fine on another host) | 02:51 |
| *** jamesmcarthur has joined #openstack-infra | 02:52 | |
| *** apetrich has quit IRC | 02:52 | |
| clarkb | prometheanfire: its the logfile path that interacts with it | 02:53 |
| clarkb | prometheanfire: because that is relative to the chroot and then doesnt exist | 02:53 |
| prometheanfire | huh, maybe it's undefined by default | 02:53 |
| prometheanfire | I was using our default config | 02:53 |
| prometheanfire | did you recheck that job? | 02:53 |
| clarkb | no, I just logged into the node that is waiting and ready and then fixed it the rest of the way (had tonset chroot to "") | 02:55 |
| prometheanfire | just did it to https://review.openstack.org/#/c/602439/3 | 02:55 |
| *** jamesmcarthur has quit IRC | 02:57 | |
| *** felipemonteiro has joined #openstack-infra | 02:58 | |
| *** ykarel|away has joined #openstack-infra | 03:01 | |
| *** lbragstad has quit IRC | 03:04 | |
| *** e0ne has quit IRC | 03:05 | |
| *** ykarel|away has quit IRC | 03:07 | |
| *** lbragstad has joined #openstack-infra | 03:11 | |
| *** jamesmcarthur has joined #openstack-infra | 03:13 | |
| *** jamesmcarthur has quit IRC | 03:17 | |
| *** felipemonteiro has quit IRC | 03:23 | |
| *** hongbin has quit IRC | 03:25 | |
| *** jamesmcarthur has joined #openstack-infra | 03:33 | |
| *** jamesmcarthur has quit IRC | 03:37 | |
| *** rlandy has quit IRC | 03:42 | |
| *** felipemonteiro has joined #openstack-infra | 03:46 | |
| *** lbragstad has quit IRC | 03:48 | |
| *** rlandy has joined #openstack-infra | 03:50 | |
| *** felipemonteiro has quit IRC | 03:52 | |
| *** rlandy has quit IRC | 03:53 | |
| *** udesale has joined #openstack-infra | 03:53 | |
| *** jamesmcarthur has joined #openstack-infra | 03:54 | |
| *** felipemonteiro has joined #openstack-infra | 03:56 | |
| *** jamesmcarthur has quit IRC | 03:58 | |
| *** felipemonteiro has quit IRC | 04:00 | |
| *** yamamoto has joined #openstack-infra | 04:06 | |
| *** ykarel|away has joined #openstack-infra | 04:08 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** diablo_rojo has joined #openstack-infra | 04:14 | |
| *** jamesmcarthur has joined #openstack-infra | 04:14 | |
| *** slaweq has quit IRC | 04:16 | |
| *** bobh has quit IRC | 04:18 | |
| *** jamesmcarthur has quit IRC | 04:19 | |
| *** ykarel|away has quit IRC | 04:34 | |
| *** diablo_rojo has quit IRC | 04:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove migrated keystone jobs https://review.openstack.org/603199 | 04:51 |
| *** ykarel|away has joined #openstack-infra | 04:53 | |
| *** jaosorior has joined #openstack-infra | 04:54 | |
| *** jamesmcarthur has joined #openstack-infra | 04:55 | |
| *** ykarel|away is now known as ykarel | 04:55 | |
| *** jamesmcarthur has quit IRC | 05:00 | |
| *** slaweq has joined #openstack-infra | 05:02 | |
| *** slaweq has quit IRC | 05:06 | |
| *** diablo_rojo has joined #openstack-infra | 05:10 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/nodepool master: Cleanup of leaked resource for static driver https://review.openstack.org/600084 | 05:20 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/nodepool master: Implement liveness check for static nodes https://review.openstack.org/601513 | 05:20 |
| chkumar|ruck | ianw: Hello | 05:20 |
| chkumar|ruck | ianw: we are seeing mirror issue in tripleo jobs | 05:21 |
| *** armax has quit IRC | 05:21 | |
| chkumar|ruck | ianw: http://logs.openstack.org/45/560445/143/check/tripleo-ci-centos-7-containers-multinode/6a1f478/logs/subnode-2/home/zuul/repo_setup.log.txt.gz#_2018-09-18_00_23_39 | 05:21 |
| *** jamesmcarthur has joined #openstack-infra | 05:22 | |
| chkumar|ruck | please have a look | 05:22 |
| chkumar|ruck | infra-core can we get this merged https://review.openstack.org/#/c/603054/ | 05:27 |
| chkumar|ruck | it will fix mirror issue | 05:27 |
| *** jamesmcarthur has quit IRC | 05:27 | |
| *** dayou has quit IRC | 05:28 | |
| *** quique|rover|off is now known as quiquell|rover | 05:31 | |
| *** jtomasek has joined #openstack-infra | 05:35 | |
| ianw | chkumar|ruck: that is actually proxied, so it is coming from the other end | 05:38 |
| ianw | i believe that there was some discussion of this previously, and that that is a koji build and so might disappear | 05:39 |
| chkumar|ruck | ianw: May be I can wait for next run | 05:41 |
| ianw | chkumar|ruck: see https://bugs.launchpad.net/tripleo/+bug/1793006 | 05:43 |
| openstack | Launchpad bug 1793006 in tripleo "yum repo failures in upstream mirrors" [Critical,In progress] - Assigned to wes hayutin (weshayutin) | 05:43 |
| *** diablo_rojo has quit IRC | 05:44 | |
| *** holser_ has joined #openstack-infra | 05:45 | |
| chkumar|ruck | ianw: thanks :-) | 05:46 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move keystone-tempest job in-repo https://review.openstack.org/603282 | 05:54 |
| AJaeger | config-core, small cleanup to delete now obsolete jobs - https://review.openstack.org/595273. Please review. Also, https://review.openstack.org/#/c/602869/ | 05:56 |
| *** dayou has joined #openstack-infra | 05:58 | |
| *** jamesmcarthur has joined #openstack-infra | 06:04 | |
| *** psachin has joined #openstack-infra | 06:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove migrated keystone jobs https://review.openstack.org/603199 | 06:06 |
| *** janki has joined #openstack-infra | 06:08 | |
| *** jamesmcarthur has quit IRC | 06:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/603284 | 06:08 |
| *** jtomasek has quit IRC | 06:09 | |
| *** jtomasek has joined #openstack-infra | 06:14 | |
| prometheanfire | clarkb: looks like same error http://logs.openstack.org/39/602439/3/check/openstack-infra-base-integration-gentoo-17-0-systemd/643b15f/job-output.txt.gz | 06:17 |
| *** slaweq has joined #openstack-infra | 06:19 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove fuxi jobs https://review.openstack.org/595273 | 06:21 |
| AJaeger | ianw: https://review.openstack.org/#/c/592992 is ready as well - thanks! | 06:22 |
| *** jamesmcarthur has joined #openstack-infra | 06:25 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: wip: Use merger to get list of files for pull-request https://review.openstack.org/603287 | 06:25 |
| *** jamesmcarthur has quit IRC | 06:29 | |
| *** jtomasek has quit IRC | 06:35 | |
| *** quiquell|rover is now known as quique|rover|brb | 06:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/603284 | 06:40 |
| *** jamesmcarthur has joined #openstack-infra | 06:46 | |
| *** jtomasek has joined #openstack-infra | 06:46 | |
| *** jamesmcarthur has quit IRC | 06:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire charm-ceph project https://review.openstack.org/602700 | 06:54 |
| *** dpawlik has quit IRC | 06:56 | |
| *** dpawlik has joined #openstack-infra | 06:57 | |
| *** jtomasek has quit IRC | 07:01 | |
| *** rossella_s has quit IRC | 07:01 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:02 | |
| *** rcernin has quit IRC | 07:03 | |
| *** olivierb has joined #openstack-infra | 07:08 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:10 | |
| *** ginopc has joined #openstack-infra | 07:12 | |
| *** hashar has joined #openstack-infra | 07:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for masakari repositories https://review.openstack.org/592992 | 07:20 |
| *** jamesmcarthur has joined #openstack-infra | 07:27 | |
| *** ykarel is now known as ykarel|lunch | 07:31 | |
| *** jamesmcarthur has quit IRC | 07:31 | |
| *** jpich has joined #openstack-infra | 07:40 | |
| *** priteau has joined #openstack-infra | 07:41 | |
| *** lathiat has quit IRC | 07:47 | |
| *** lathiat has joined #openstack-infra | 07:48 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: wip: Use merger to get list of files for pull-request https://review.openstack.org/603287 | 07:49 |
| *** jpena|off is now known as jpena | 07:50 | |
| *** persia has quit IRC | 08:06 | |
| *** shoogz has quit IRC | 08:07 | |
| *** persia has joined #openstack-infra | 08:07 | |
| *** shoogz has joined #openstack-infra | 08:10 | |
| *** tosky has joined #openstack-infra | 08:16 | |
| *** psachin has quit IRC | 08:22 | |
| *** ykarel|lunch is now known as ykarel | 08:25 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Add reviewers by group alias on upload https://review.openstack.org/195043 | 08:33 |
| *** markvoelker has joined #openstack-infra | 08:36 | |
| *** alexchadin has joined #openstack-infra | 08:41 | |
| *** derekh has joined #openstack-infra | 08:49 | |
| *** lbragstad has joined #openstack-infra | 08:59 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update the UC meeting time https://review.openstack.org/603005 | 08:59 |
| *** markvoelker has quit IRC | 09:11 | |
| *** alexchadin has quit IRC | 09:26 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/openstack-zuul-jobs master: Silence syncronize on post to avoid console spam https://review.openstack.org/598599 | 09:30 |
| *** alexchadin has joined #openstack-infra | 09:34 | |
| *** shardy has joined #openstack-infra | 09:34 | |
| *** dtantsur|afk is now known as dtantsur | 09:36 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/openstack-zuul-jobs master: Silence syncronize on post to avoid console spam https://review.openstack.org/598599 | 09:41 |
| *** sambetts_ has joined #openstack-infra | 09:44 | |
| *** sambetts_ is now known as sambetts | 09:44 | |
| *** shardy has quit IRC | 09:46 | |
| *** ykarel is now known as ykarel|away | 09:50 | |
| *** ykarel|away is now known as ykarel|afk | 09:51 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/openstack-zuul-jobs master: Silence syncronize on post to avoid console spam https://review.openstack.org/598599 | 09:54 |
| *** longkb has quit IRC | 09:56 | |
| *** Emine has joined #openstack-infra | 09:58 | |
| *** e0ne has joined #openstack-infra | 10:04 | |
| *** markvoelker has joined #openstack-infra | 10:07 | |
| *** quiquell|rover has quit IRC | 10:08 | |
| *** dayou has quit IRC | 10:08 | |
| *** quiquell has joined #openstack-infra | 10:09 | |
| *** bhavikdbavishi has quit IRC | 10:23 | |
| *** yamamoto has quit IRC | 10:31 | |
| *** alexchadin has quit IRC | 10:33 | |
| *** dayou has joined #openstack-infra | 10:36 | |
| *** vivsoni has quit IRC | 10:36 | |
| *** roman_g has joined #openstack-infra | 10:37 | |
| *** vivsoni has joined #openstack-infra | 10:40 | |
| *** markvoelker has quit IRC | 10:41 | |
| *** yamamoto has joined #openstack-infra | 10:44 | |
| *** udesale has quit IRC | 10:47 | |
| *** panda has joined #openstack-infra | 10:51 | |
| panda | if I need a temporary branch to test the merge in https://review.openstack.org/602377 do I also need "create ref" ? | 10:51 |
| *** aperevalov has joined #openstack-infra | 10:59 | |
| *** ykarel|afk is now known as ykarel | 11:00 | |
| aperevalov | hello, how to know kernel version on the gate host? I mean host where tempest tests are running. | 11:01 |
| *** alexchadin has joined #openstack-infra | 11:07 | |
| *** alexchadin has quit IRC | 11:11 | |
| *** alexchadin has joined #openstack-infra | 11:14 | |
| *** gfidente has joined #openstack-infra | 11:18 | |
| *** adriancz has joined #openstack-infra | 11:18 | |
| *** pbourke has quit IRC | 11:20 | |
| *** pbourke has joined #openstack-infra | 11:21 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/infra-specs master: Direction setting for 3rd Party CI https://review.openstack.org/563849 | 11:35 |
| *** alexchadin has quit IRC | 11:36 | |
| *** yamamoto has quit IRC | 11:37 | |
| *** jpena is now known as jpena|lunch | 11:38 | |
| *** alexchadin has joined #openstack-infra | 11:40 | |
| *** e0ne has quit IRC | 11:51 | |
| *** agopi has joined #openstack-infra | 11:57 | |
| *** boden has joined #openstack-infra | 11:59 | |
| *** Shrews has quit IRC | 12:00 | |
| *** yamamoto has joined #openstack-infra | 12:01 | |
| *** Shrews has joined #openstack-infra | 12:02 | |
| *** yumiriam has joined #openstack-infra | 12:04 | |
| *** markvoelker has joined #openstack-infra | 12:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/infra-specs master: Direction setting for 3rd Party CI https://review.openstack.org/563849 | 12:04 |
| *** priteau has quit IRC | 12:14 | |
| *** jtomasek has joined #openstack-infra | 12:16 | |
| *** priteau has joined #openstack-infra | 12:17 | |
| *** dave-mccowan has joined #openstack-infra | 12:23 | |
| *** trown|outtypewww is now known as trown | 12:26 | |
| *** dave-mccowan has quit IRC | 12:28 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:30 | |
| *** chkumar|ruck is now known as chkumar|off | 12:32 | |
| *** jpena|lunch is now known as jpena | 12:38 | |
| *** dtantsur is now known as dtantsur|brb | 12:40 | |
| *** rlandy has joined #openstack-infra | 12:42 | |
| *** ansmith has joined #openstack-infra | 12:46 | |
| *** kgiusti has joined #openstack-infra | 12:56 | |
| *** efried_off is now known as efried | 13:05 | |
| *** mriedem has joined #openstack-infra | 13:07 | |
| *** r-mibu_ has joined #openstack-infra | 13:07 | |
| *** jtomasek has quit IRC | 13:08 | |
| *** r-mibu_ has quit IRC | 13:08 | |
| *** jtomasek has joined #openstack-infra | 13:10 | |
| *** quiquell is now known as quique|rover|lch | 13:14 | |
| pabelanger | prometheanfire: clarkb: for http://logs.openstack.org/39/602439/3/check/openstack-infra-base-integration-gentoo-17-0-systemd/643b15f/job-output.txt.gz, unbound hasn't been configured yet. I am guessing because ovh, and ipv6 you are running in to something broken for some reason. This was basically the same issue we had with fedora. The fix was to always configured and restart unbound before | 13:18 |
| pabelanger | validate-host role. | 13:18 |
| *** bhavikdbavishi has quit IRC | 13:21 | |
| *** alexchadin has quit IRC | 13:22 | |
| *** agopi has quit IRC | 13:24 | |
| *** Shrews has quit IRC | 13:27 | |
| *** jamesmcarthur has joined #openstack-infra | 13:28 | |
| *** rh-jelabarre has joined #openstack-infra | 13:28 | |
| *** alexchadin has joined #openstack-infra | 13:32 | |
| *** Shrews has joined #openstack-infra | 13:32 | |
| *** efried is now known as efried_afak | 13:32 | |
| *** efried_afak is now known as efried_afk | 13:32 | |
| *** jamesmcarthur has quit IRC | 13:32 | |
| *** sthussey has joined #openstack-infra | 13:36 | |
| *** quique|rover|lch is now known as quiquell|rover | 13:40 | |
| *** alexchadin has quit IRC | 13:45 | |
| *** agopi has joined #openstack-infra | 13:46 | |
| *** efried_afk is now known as efried | 13:47 | |
| *** apetrich has joined #openstack-infra | 13:48 | |
| *** lbragstad has quit IRC | 13:53 | |
| *** lbragstad has joined #openstack-infra | 13:54 | |
| *** alexchadin has joined #openstack-infra | 13:54 | |
| *** kgiusti has quit IRC | 13:58 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed validation on member affiliation endpoints https://review.openstack.org/603397 | 14:04 |
| *** Emine has quit IRC | 14:06 | |
| tosky | corvus: hi; is there a specific reason why the devstack patches which automatically set LIBS_FROM_GIT based on required-projects was not backported to the older stable branches? | 14:08 |
| *** dtantsur|brb is now known as dtantsur | 14:09 | |
| tosky | I'm trying to finish the work on the grenade native zuul job, and it fails on check_libs_from_git on stable/queens (removed in master) because requirements, devstack and grenade are not installed, apparently | 14:10 |
| *** alexchadin has quit IRC | 14:11 | |
| *** alexchadin has joined #openstack-infra | 14:12 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed validation on member affiliation endpoints https://review.openstack.org/603397 | 14:13 |
| *** ykarel is now known as ykarel|away | 14:16 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Update testing to a later gerrit version https://review.openstack.org/552251 | 14:16 |
| *** quiquell|rover is now known as quique|rover|off | 14:17 | |
| *** Emine has joined #openstack-infra | 14:19 | |
| *** yamamoto has quit IRC | 14:19 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Add reviewers by group alias on upload https://review.openstack.org/195043 | 14:21 |
| *** emerson has quit IRC | 14:21 | |
| *** rossella_s has joined #openstack-infra | 14:21 | |
| *** bobh has joined #openstack-infra | 14:22 | |
| *** yamamoto has joined #openstack-infra | 14:24 | |
| *** yamamoto has quit IRC | 14:24 | |
| *** yamamoto has joined #openstack-infra | 14:24 | |
| *** zul has joined #openstack-infra | 14:28 | |
| *** rossella_s has quit IRC | 14:29 | |
| *** yamamoto has quit IRC | 14:29 | |
| *** kgiusti has joined #openstack-infra | 14:31 | |
| *** rossella_s has joined #openstack-infra | 14:33 | |
| prometheanfire | pabelanger: but I thought an unbound conf file was already generated, what do you mean by configured? | 14:33 |
| pabelanger | prometheanfire: yah, we generate it at DIB build stage, and use configure-mirror role to update any configuration too. For ipv6 clouds, there seems to be an issue, but haven't really figured it out | 14:34 |
| pabelanger | had the same issue with fedora-28 nodes | 14:34 |
| pabelanger | but, moved configure-unbound into own play in base jobs | 14:35 |
| pabelanger | but, with integration jobs, we didn't make the change | 14:35 |
| pabelanger | so, need to also make it there I think | 14:35 |
| *** ykarel|away has quit IRC | 14:36 | |
| *** dpawlik has quit IRC | 14:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: remove job settings for blazar repositories https://review.openstack.org/596250 | 14:42 |
| clarkb | pabelanger: prometheanfire its likely the gentoo issue remains that unbound isnt starting at all | 14:45 |
| clarkb | I only fixed a single nodw which may not have been used for that test | 14:45 |
| pabelanger | yah, that is possible. The configure-unbound role start ensures the service is started | 14:45 |
| AJaeger | clarkb: could you review https://review.openstack.org/596250 , please? That one is ready to merge now... | 14:49 |
| clarkb | pabelanger: https://review.openstack.org/#/c/603247/ is the fix for gentoo unbound I think | 14:49 |
| clarkb | AJaeger: done | 14:50 |
| AJaeger | thanks, clarkb | 14:50 |
| clarkb | aperevalov: the dpkg and rpm list log files should have that info. I think we may also capture it in the ansible facts log files | 14:50 |
| prometheanfire | clarkb: ya, I thought the recheck would have worked though | 14:51 |
| clarkb | prometheanfire: I've checked the node I fixed and the one that job ran on and they are different | 14:51 |
| aperevalov | clarkb: thanks | 14:51 |
| prometheanfire | ah, ok | 14:51 |
| clarkb | prometheanfire: I think, its hard to know exactly since things were deleted and I'm working from terminal scrollback | 14:52 |
| clarkb | tosky: my guess is that was just missed | 14:52 |
| clarkb | tosky: corvus and andreaf likely wanted to see it working on master for a bit before backporting. It should be on rocky and master now. Backports to the other branch job defs would likely be approprate | 14:53 |
| prometheanfire | clarkb: well, if that merges and a new image gets built that'd be a start :P | 14:53 |
| prometheanfire | clarkb: the change itself seems good | 14:53 |
| prometheanfire | pabelanger: mind looking at https://review.openstack.org/#/c/603247 ? | 14:55 |
| pabelanger | +3 | 14:56 |
| *** hamzy has quit IRC | 14:56 | |
| prometheanfire | ty | 14:56 |
| *** rossella_s has quit IRC | 14:56 | |
| *** hamzy has joined #openstack-infra | 14:58 | |
| *** anteaya has joined #openstack-infra | 14:59 | |
| *** rossella_s has joined #openstack-infra | 15:01 | |
| tosky | clarkb: so I guess I could try to propose a backport to queens? | 15:01 |
| clarkb | tosky: ya | 15:02 |
| *** armax has joined #openstack-infra | 15:02 | |
| tosky | let's try... | 15:07 |
| tosky | apparently 3 patches are needed | 15:07 |
| *** ramishra has quit IRC | 15:07 | |
| *** zul has quit IRC | 15:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable unbound chroot on test node images https://review.openstack.org/603247 | 15:09 |
| *** dpawlik has joined #openstack-infra | 15:10 | |
| *** jamesmcarthur has joined #openstack-infra | 15:12 | |
| *** dpawlik has quit IRC | 15:15 | |
| *** zul has joined #openstack-infra | 15:19 | |
| *** felipemonteiro has joined #openstack-infra | 15:24 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Silence syncronize on post to avoid console spam https://review.openstack.org/603420 | 15:29 |
| clarkb | prometheanfire: I've queued a gentoo image rebuild, but its waiting behind other builds | 15:31 |
| *** Emine has quit IRC | 15:31 | |
| prometheanfire | ah, cool, didn't see it merged | 15:31 |
| clarkb | prometheanfire: it is building on nb02 if you want to follow the log | 15:38 |
| prometheanfire | cool | 15:38 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/devstack-gate master: Silence syncronize on to avoid console spam https://review.openstack.org/603426 | 15:38 |
| prometheanfire | still needs LE :P | 15:38 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for blazar repositories https://review.openstack.org/596250 | 15:38 |
| *** felipemonteiro has quit IRC | 15:39 | |
| *** rkukura has joined #openstack-infra | 15:41 | |
| *** vivsoni has quit IRC | 15:43 | |
| *** kgiusti has quit IRC | 15:44 | |
| *** kgiusti has joined #openstack-infra | 15:45 | |
| *** bobh has quit IRC | 15:46 | |
| roman_g | Hell team. I would like to ask if the following functionality works, and where could I see sample configuration. Thank you. https://docs.openstack.org/infra/system-config/gerrit.html#auto-review-expiry https://docs.openstack.org/infra/system-config/jeepyb.html#expiring-old-reviews | 15:48 |
| *** kgiusti has quit IRC | 15:50 | |
| clarkb | roman_g: https://docs.openstack.org/infra/system-config/gerrit.html#auto-review-expiry is no longer installed. The jeepyb functionality continues to exist as documented at https://docs.openstack.org/infra/system-config/jeepyb.html#expiring-old-reviews but we don't use it | 15:54 |
| clarkb | roman_g: instead we give all core reviewers abandon abilities on changes with the expectation that they and change reviewers can curate | 15:55 |
| *** jamesmcarthur has quit IRC | 16:00 | |
| *** hashar is now known as hasharAway | 16:00 | |
| *** hamzy has quit IRC | 16:01 | |
| *** jamesmcarthur has joined #openstack-infra | 16:04 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Use zk connection passed to OpenStack driver https://review.openstack.org/603428 | 16:04 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Move OpenStack leak code into driver https://review.openstack.org/603429 | 16:04 |
| *** kgiusti has joined #openstack-infra | 16:04 | |
| *** ginopc has quit IRC | 16:04 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added endpoints for my presentations moderators https://review.openstack.org/603432 | 16:05 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added endpoints for my presentations moderators https://review.openstack.org/603432 | 16:05 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Move OpenStack leak code into driver https://review.openstack.org/603429 | 16:07 |
| *** ykarel|away has joined #openstack-infra | 16:10 | |
| clarkb | fungi: thoughts on a new git review release happening in the near future? | 16:10 |
| clarkb | that was on the TODO list pre PTG iirc | 16:10 |
| *** bhavikdbavishi has joined #openstack-infra | 16:12 | |
| ssbarnea | clarkb: i kind miss the ability to auto-abandon old changes as I seen that in some less mainstream projects is high likely to see changes stacking for years without either owner or core doing something about them. i guess the docs needs to be updated to remove the "Expiring Old Reviews" section, if we don't plan to reinstate it. | 16:13 |
| clarkb | ssbarnea: yes that doc needs updating | 16:13 |
| *** rlandy is now known as rlandy|brb | 16:14 | |
| ssbarnea | not sure what was the reson to disable it, i guess it was more than one false-positive abandonment that required it. maybe a magic comment that can be added by anyone could be an alternative. | 16:15 |
| clarkb | ssbarnea: the problem is that we can't actually tell if work is abanonded based on time only. And restoring a change requires extra privileges | 16:16 |
| clarkb | ssbarnea: so we changed the expectation to be on the people that know better than the bot to abandon as appropriate. Some projects run their own script to automate it anyway, others curate by hand, others ignore it | 16:16 |
| ssbarnea | i guess a two step process could work here, I seen bots doin this on github or jira tickets: which first adds a warning (comment) telling that this issue will be closed if is not updated in the next x days, acting based on that. more of a "secretary" kind of bot. | 16:19 |
| clarkb | ssbarnea: the good news is we've updated the ACLs such that projects can decide for themselves rather than trying to institute this gerrit wide | 16:19 |
| ssbarnea | clarkb: that's good to hear. | 16:20 |
| clarkb | (all core reviewers can abandon changes on the projects they are core for) | 16:22 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/system-config master: Remove expire-old-reviews from docs https://review.openstack.org/603436 | 16:22 |
| *** holser_ has quit IRC | 16:23 | |
| clarkb | ssbarnea: re ^ thanks. +2 | 16:26 |
| ssbarnea | clarkb: not sure about openstack gerrit but on newer versions is very easy to revive a change after abandoning (even if is not yours), we did this a lot on gerrithub.io, i suspect they made it easier at some point. i really miss the draft option as it helps a lot in the UI, to separate the WIP/DNM/.... work from the one that is ready for review and merge. | 16:26 |
| clarkb | ssbarnea: draft has a lot of problems though (like not being visible to people and CI systems, causing confusing behavior) | 16:27 |
| clarkb | also people assumed drafts were "secure" and secret but they could be fetched in crafty ways | 16:27 |
| *** bnemec has quit IRC | 16:28 | |
| *** dtantsur is now known as dtantsur|afk | 16:28 | |
| ssbarnea | i find the draft not being visible unless you are revier as a feature, not bug. not sure about zuul but jenkins was able to build them without problem. | 16:28 |
| *** bnemec has joined #openstack-infra | 16:28 | |
| clarkb | ssbarnea: the problem is draft is a patchset attribute not a change attribute. So reviewers can review a change that is visible but then not approve/merge that change and they get confused | 16:29 |
| clarkb | reason is the latest patchset is a draft | 16:29 |
| clarkb | as for CI systems you can either respect the idea they are "private" or bypass that through careful ref fetching | 16:29 |
| clarkb | both confuse people for different reasons | 16:29 |
| clarkb | its confusing for people to be able to see and test your private thing | 16:30 |
| clarkb | it is also confusing to not get CI results | 16:30 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on seed default track tag groups response https://review.openstack.org/603438 | 16:32 |
| *** ssbarnea|bkp has joined #openstack-infra | 16:33 | |
| clarkb | I alsothink that draft patchsets don't get event stream events sent unless the account has visibility on the draft | 16:33 |
| clarkb | so you have to assume there are things there when they are inferred | 16:33 |
| *** jpich has quit IRC | 16:33 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on seed default track tag groups response https://review.openstack.org/603438 | 16:33 |
| *** sambetts is now known as sambetts|afk | 16:35 | |
| *** bobh has joined #openstack-infra | 16:36 | |
| * clarkb wanders off to find tea | 16:37 | |
| *** gyee has joined #openstack-infra | 16:38 | |
| *** rlandy|brb is now known as rlandy | 16:38 | |
| *** bobh has quit IRC | 16:40 | |
| clarkb | prometheanfire: curious if dib can do more to cache intermediate gentoo make world results | 16:46 |
| clarkb | prometheanfire: maybe that element can install ccache with objects go into the dib cache dir? | 16:47 |
| *** bobh has joined #openstack-infra | 16:47 | |
| *** olivierb has quit IRC | 16:47 | |
| prometheanfire | clarkb: better would be to bind mount /usr/portage/packages to dib cache, then pass '--binpkg-respect-use --rebuilt-binaries=y --usepkg --binpkg-changed-deps=y' to emerge, while also setting FEATURES (envvar or /etc/portage/make.conf value) to include buildpkg | 16:50 |
| prometheanfire | then we'd have binary packages | 16:50 |
| roman_g | clarkb: thank you for the answers | 16:50 |
| prometheanfire | it would make it depend on cpu arch though, maybe, (not sure what march is set to), but that's already a problem | 16:51 |
| AJaeger | team, what do you think of ssbarnea's change to add --quiet to rsync? See https://review.openstack.org/603420 and https://review.openstack.org/#/c/598599/ | 16:52 |
| *** anteaya has quit IRC | 16:52 | |
| prometheanfire | clarkb: I'm not super familiar with dib cache though | 16:53 |
| clarkb | prometheanfire: ya we already use cpu arch specific image builders because things get confusing and weird really quickly if you don't | 16:54 |
| clarkb | prometheanfire: the cache dir is defined in DIB_CACHE or similar env var and individual elements are expected to use it as appropriate to save data between builds to speed things up (and possibly enable offline builds) | 16:54 |
| clarkb | prometheanfire: the apt builds put apt packages in there for example | 16:55 |
| clarkb | prometheanfire: we also currate a set of our git repos in that dir as well so that we don't have to reclone on every build | 16:55 |
| prometheanfire | clarkb: the gentoo element would probably do something then, need to check it all out (it's been on my todo for a while, just low) | 16:55 |
| persia | I believe the apt- and yum- compatible packages are stored in arch-specific structures using standard multiarch mirror models, so that things just work. | 16:55 |
| clarkb | AJaeger: ssbarnea will ara still log that data? | 16:56 |
| clarkb | AJaeger: ssbarnea I am ok with it as long as we have some way of finding that info (like ara) | 16:56 |
| persia | A delightful side effect of storing most of that on AFS is that it is fairly easy to connect to AFS from a laptop and inspecting the results. | 16:56 |
| ssbarnea | AJaeger: not as far as I know, see https://github.com/ansible/ansible/issues/25681 | 16:56 |
| prometheanfire | persia: so nahalem would be an example? | 16:56 |
| clarkb | prometheanfire: no its mostly i586 vs amd64 | 16:56 |
| persia | prometheanfire: For apt, I think the list is "arm64" and "amd64". | 16:57 |
| ssbarnea | AJaeger: but at the same time, remember that no_log does obliterate output too, in fact hiding even errors. | 16:57 |
| clarkb | prometheanfire: far more generic, keep in mind that gentoo can't assume things about the cpu in this cloud use case either though | 16:57 |
| clarkb | prometheanfire: we build the images on one cloud and then boot it in all the clouds, also a single cloud isn't necesarily using the same cpu for all nova boots | 16:57 |
| prometheanfire | clarkb: yep, which is why we should probably just pick the lowest common denom and set that for march | 16:57 |
| ssbarnea | clarkb: --quiet is the lesser risky approach, as it removes the succesfully copied files from output which we may never need to know about because we can always see them on the destination. | 16:58 |
| persia | Yes. Lowest compatible -march for x86_64 is probably best. | 16:58 |
| prometheanfire | even within a cloud that differs, so I'll find out what it is | 16:59 |
| ssbarnea | i personally never had a case where I really needed to see the list of copied files and if someone wants, they will have to use bash and redirect because ansible sync module has no option to avoid that kind of spam. | 17:00 |
| clarkb | ssbarnea: ya the spam is annoying when you don't need it (Its also a problemwith other modules like the git one iirc), but when things don't work as expected it is helpful. Maybe the thing to do in those cases is to remove the quiet when/if that happens and rerun | 17:01 |
| ssbarnea | clarkb: check http://logs.rdoproject.org/76/587676/5/openstack-check/legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike/eaad42d/job-output.txt.gz -- do you really want to see what starts at line "2018-09-17 17:56:57.365165" ? | 17:01 |
| prometheanfire | clarkb: I'll work on it this week I think (the 21st is when I build my images anyway), it'll probably have to have a different binpkg subdir per arch/gentoo_profile | 17:02 |
| prometheanfire | http://distfiles.gentoo.org/experimental/amd64/openstack/ fyi | 17:02 |
| clarkb | ssbarnea: yes sometimes I actually do. Taking the tripleo jobs as an explicit example we ran out of inodes on our log server due to tripleo uploading many thousands of log files per job | 17:02 |
| persia | prometheanfire: Please leave space in your implemenation for clouds that have entirely incompatible architectures (e.g. Arm, POWER). | 17:02 |
| ssbarnea | clarkb: this is why I find -q on rsync, as it doesn't silence only success, you can still see errors/warnings. | 17:02 |
| prometheanfire | since diferent profiles pass different flags | 17:02 |
| clarkb | ssbarnea: being explicit about what is copied allows people without direct fs access to quickly identify the problem area | 17:02 |
| *** jpena is now known as jpena|off | 17:02 | |
| *** derekh has quit IRC | 17:02 | |
| prometheanfire | persia: ya, it'd be whatever is passed to march, so aarch64 for arm64 iirc | 17:02 |
| prometheanfire | currently gentoo dib doesn't do other arches anyway | 17:03 |
| persia | Excellent, and understood :) | 17:03 |
| *** gfidente is now known as gfidente|afk | 17:04 | |
| ssbarnea | clarkb: if someone needs this info, he should be able to find it in a dump file (result can be dumped), not console. it is far too verbose for 99.99% of the cases. i have the impression that we are providing a bad UX for the vast majority for the what-if scenario is very unlikely to happen. | 17:06 |
| clarkb | ssbarnea: except it isn't a what if scenarion that is unlikely to happen. If you read that log there are 5 copies of the same file | 17:06 |
| clarkb | there is a balance to reach between providing not enough information and too much. We are probably on the too much side of things right now, but removing all of it is too little | 17:07 |
| clarkb | one option would be to record the dump file and copy it ( then onlly have more verbose logging for that file since it is a single item) | 17:07 |
| clarkb | another is to exposethat info via a tool like ara but not in the raw log | 17:08 |
| ssbarnea | clarkb: I will test to see if I can use no_log & dump(result) to file approach to dump that spam somewhere on disk and avoid the console. testing now. | 17:10 |
| panda | if I need a temporary branch to test the merge in https://review.openstack.org/602377 do I also need "create ref" ? | 17:12 |
| clarkb | panda: the push perms would be scoped to the single branch you are pushing to ideally. Then add pushMerge more globally | 17:13 |
| clarkb | panda: I think that will let you push to the branch then push a merge between the branches to the target branch | 17:13 |
| clarkb | panda: as for creating the branch it is pretty easy for a gerrit admin to do that, I can create a branch now if you like. I just need to know the base ref to start it from | 17:14 |
| *** jchhatbar has joined #openstack-infra | 17:14 | |
| panda | clarkb: you can use master, I think you can call it tqmerge. So I need to change the access control to allow pusn only on ref/heads/tqmerge, then globally allow pushMerge ? | 17:15 |
| ssbarnea | clarkb: yep, I found a solution that behaves nice, look at https://gist.github.com/ssbarnea/fef8300463c757f70ed880052abc0f46 | 17:15 |
| clarkb | panda: yes I think that will allow you to push into the tqmerge branch, then push a merge commit for tqmerge into master to master | 17:16 |
| clarkb | ssbarnea: neat | 17:16 |
| *** janki has quit IRC | 17:17 | |
| clarkb | panda: is the tripleo-quickstart-extras repo being abandoned as part of this too? | 17:17 |
| ssbarnea | ... now I only hope you will not ask me to implement this for the legacy change which touches ~140 files. I don't mind doing it for the other two changes that are likely to remain in production for much longer. | 17:17 |
| clarkb | panda: I created refs/heads/tqmerge on openstack/tripleo-quickstart based on current master | 17:18 |
| clarkb | ssbarnea: in general I think we've been accepting of the ugly in legacy jobs and asked for cleanups as we migrate to native jobs | 17:18 |
| panda | clarkb: yes, but not immediately, we need to be sure nothing esle is using it. After the merge I'll add a job that just -1 any change | 17:19 |
| *** olivierb has joined #openstack-infra | 17:21 | |
| *** dpawlik has joined #openstack-infra | 17:23 | |
| *** dpawlik has quit IRC | 17:24 | |
| *** dpawlik has joined #openstack-infra | 17:24 | |
| *** alexchadin has quit IRC | 17:25 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/devstack-gate master: Silence syncronize on to avoid console spam https://review.openstack.org/603426 | 17:29 |
| panda | clarkb: ok, I'll change the review. Thanks. | 17:37 |
| *** jchhatbar has quit IRC | 17:40 | |
| *** jamesmcarthur has quit IRC | 17:41 | |
| *** bobh has quit IRC | 17:42 | |
| *** electrofelix has quit IRC | 17:45 | |
| *** dpawlik has quit IRC | 17:48 | |
| clarkb | logan-: http://logs.openstack.org/26/597826/1/gate/puppet-openstack-lint/a9389be/job-output.txt.gz#_2018-09-18_16_46_27_950626 any idea what might cause the no route to hosts like that in limestone? | 17:48 |
| *** hemna_ has joined #openstack-infra | 17:48 | |
| clarkb | doesn't respond to ssh either | 17:49 |
| clarkb | I guess I can try a reboot of the server instance | 17:50 |
| clarkb | logan-: 25c4df93-bcf5-460c-948a-bb66bb4b177b is the uuid fwiw | 17:50 |
| clarkb | logan-: seems to affect ipv4 and ipv6 addresses | 17:52 |
| clarkb | I'm going to try a reboot | 17:52 |
| clarkb | reboot makes it ping. Waiting on sshd to start enough to login | 17:54 |
| logan- | hmm | 17:54 |
| clarkb | its back now. syslog doesn't show anything suspicious | 17:55 |
| clarkb | Sep 17 20:37:36 mirror02 snmpd[1400]: Connection from UDP/IPv6: [2001:4800:7821:105:be76:4eff:fe04:b9a5]:41376 is the last logged entry prior to the reboot though | 17:55 |
| logan- | was the issue with the mirror? or the node? | 17:56 |
| clarkb | logan-: the mirror node stopped responding on the network (no http, no ssh, no ping) | 17:56 |
| logan- | the mirror did something similar yesterday morning and i rebooted it when ianw pinged me in here, same thing I guess | 17:57 |
| *** diablo_rojo has joined #openstack-infra | 17:57 | |
| clarkb | logan-: could it be running out of disk? | 17:57 |
| clarkb | hrm looks like we use a cinder volume for the vast majority of disk | 17:58 |
| clarkb | so that is unlikely a hypervisor problem | 17:58 |
| logan- | yeah the console seemed responsive when i rebooted it iirc so im wondering at this point if the instance is losing its network config, or if the hv/neutron port is having problems | 17:59 |
| *** gfidente|afk has quit IRC | 17:59 | |
| *** trown is now known as trown|lunch | 18:00 | |
| *** panda is now known as panda|dinner | 18:01 | |
| clarkb | prometheanfire: can you recheck your gentoo triggering change now? | 18:01 |
| prometheanfire | sure | 18:02 |
| clarkb | prometheanfire: only packethost and vexxhost sjc1 are not done uploading the image | 18:02 |
| clarkb | I'm relying on statistical probability for it to find a new image :) | 18:02 |
| *** bobh has joined #openstack-infra | 18:02 | |
| prometheanfire | actually, let me update https://review.openstack.org/#/c/602439/ to remove the unbound thing there, and rebase https://review.openstack.org/#/c/602446/ (to test) | 18:03 |
| openstackgerrit | Matthew Thode proposed openstack-infra/openstack-zuul-jobs master: add Gentoo jobs and fix unbound https://review.openstack.org/602439 | 18:04 |
| clarkb | prometheanfire: ok an instance just booted in ovh and has running unbound at boot | 18:04 |
| clarkb | prometheanfire: that looks promising | 18:04 |
| prometheanfire | nice | 18:05 |
| openstackgerrit | Matthew Thode proposed openstack-infra/openstack-zuul-jobs master: use this to test gentoo zuul changes https://review.openstack.org/602446 | 18:05 |
| prometheanfire | commit message on the first one needs fixing, but meh | 18:06 |
| clarkb | prometheanfire: looks like we still deleted the node 16 seconds after it was marked ready (presumably due to a failing job) | 18:07 |
| prometheanfire | hmm | 18:07 |
| clarkb | but I managed to dig google.com and get an answer and unbound service was running. Progress | 18:08 |
| *** ykarel|away has quit IRC | 18:10 | |
| *** anteaya has joined #openstack-infra | 18:13 | |
| *** alexchadin has joined #openstack-infra | 18:20 | |
| prometheanfire | indeed, need to check logs when avail | 18:22 |
| *** alexchadin has quit IRC | 18:24 | |
| *** priteau has quit IRC | 18:28 | |
| jungleboyj | Who would be the best person to ping about an etherpad.openstack.org issue? | 18:33 |
| jungleboyj | The following etherpad just spins at 'Loading' for multiple people: https://etherpad.openstack.org/p/cinder-ptg-stein-thursday | 18:34 |
| jungleboyj | Really hoping I didn't lose 8 hours of notes. | 18:34 |
| *** anteaya has quit IRC | 18:35 | |
| openstackgerrit | bharaththiruveedula proposed openstack-infra/project-config master: Add noop jobs for Gyan Project https://review.openstack.org/603466 | 18:36 |
| *** alexchadin has joined #openstack-infra | 18:37 | |
| *** bhavikdbavishi has quit IRC | 18:37 | |
| *** harlowja has joined #openstack-infra | 18:39 | |
| *** alexchadin has quit IRC | 18:40 | |
| *** studarus has joined #openstack-infra | 18:40 | |
| studarus | is it usual for NodePool to start up instances with multiple IP/Ports? | 18:40 |
| studarus | I see several that have spun up with 3 IPs (all private) | 18:40 |
| *** alexchadin has joined #openstack-infra | 18:41 | |
| fungi | clarkb: i'm not really here today but i'd hoped to work out a git-review release during the ptg. of course that didn't happen but later this week would make perfect sense | 18:43 |
| *** alexchadin has quit IRC | 18:44 | |
| clarkb | studarus: only in clouds where the networkign is configured that way, packethost should be one ip/port + fip per host | 18:46 |
| clarkb | jungleboyj: we can try and see if the admin api will resurrect any of the data. Chances are the pad itself has been corrupted for one reason or another | 18:47 |
| jungleboyj | clarkb: Ugh. Yeah, if there is anything you can get out of it I would appreciate it. Would rather not have to go watch 8 hours of video to resurrect the action items from it. | 18:48 |
| *** studarus has quit IRC | 18:49 | |
| clarkb | jungleboyj: I've got the content in json format, let me see if I can pass it through python to make it readable | 18:49 |
| jungleboyj | clarkb: Awesome! I owe you! | 18:50 |
| clarkb | jungleboyj: http://paste.openstack.org/show/730309/ hopefuilly that looks right to you | 18:52 |
| clarkb | infra meeting in about 7 minutes over in #openstack-meeting | 18:53 |
| *** trown|lunch is now known as trown | 18:55 | |
| *** hasharAway is now known as hashar | 18:57 | |
| clarkb | studarus I'm not seeing any of those on servers with openstack server list | 18:59 |
| *** hamzy has joined #openstack-infra | 19:04 | |
| jungleboyj | clarkb: You are a life saver! That looks good enough to me! | 19:05 |
| jungleboyj | #thanks clarkb for recovering my etherpad so quickly. Saved me a lot of stress and time. The infra team is amazing! | 19:10 |
| openstackstatus | jungleboyj: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 19:10 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/devstack-gate master: Silence syncronize on post to avoid console spam https://review.openstack.org/603426 | 19:25 |
| *** priteau has joined #openstack-infra | 19:29 | |
| *** panda|dinner is now known as panda | 19:30 | |
| *** alexchadin has joined #openstack-infra | 19:34 | |
| *** bobh has quit IRC | 19:34 | |
| *** auristor has quit IRC | 19:36 | |
| *** annabelleB has joined #openstack-infra | 19:37 | |
| *** alexchadin has quit IRC | 19:38 | |
| *** auristor has joined #openstack-infra | 19:47 | |
| *** bobh has joined #openstack-infra | 19:47 | |
| *** dpawlik has joined #openstack-infra | 19:48 | |
| *** annabelleB has left #openstack-infra | 19:49 | |
| *** bobh has quit IRC | 19:52 | |
| *** eernst has joined #openstack-infra | 19:52 | |
| *** dpawlik has quit IRC | 19:52 | |
| *** alexchadin has joined #openstack-infra | 19:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/infra-specs master: Direction setting for 3rd Party CI https://review.openstack.org/563849 | 19:55 |
| *** bobh has joined #openstack-infra | 19:58 | |
| *** gouthamr has quit IRC | 20:00 | |
| *** stevebaker has quit IRC | 20:01 | |
| *** anteaya has joined #openstack-infra | 20:01 | |
| clarkb | I mentioned it in the infra meeting but I'll mentioned it here now too. OVH is doing openstack upgrades in BHS1 overnight (my time) which is where I host my IRC bouncer | 20:01 |
| clarkb | so I may be disconnected tomorrow | 20:01 |
| *** dmellado has quit IRC | 20:01 | |
| clarkb | will do my best to run a one off client if necessary | 20:01 |
| *** david-lyle is now known as dklyle | 20:02 | |
| *** gouthamr has joined #openstack-infra | 20:03 | |
| *** bobh has quit IRC | 20:03 | |
| clarkb | ianw: fungi I went ahead and +2'd https://review.openstack.org/#/c/602436/2 as it looks structurally sound and will be good to go as soon as hiera/whatever its called now is populated | 20:03 |
| clarkb | gary_perkins: ^ fyi I think ianw may be helping spin things up | 20:04 |
| *** alexchadin has quit IRC | 20:04 | |
| *** olivierb has quit IRC | 20:04 | |
| fungi | thanks! | 20:05 |
| *** studarus has joined #openstack-infra | 20:06 | |
| *** dave-mccowan has joined #openstack-infra | 20:08 | |
| ianw | ok will take a look in a bit | 20:09 |
| *** cgoncalves is now known as cgoncalves|pto | 20:13 | |
| *** bobh has joined #openstack-infra | 20:16 | |
| *** studarus has quit IRC | 20:17 | |
| *** bobh has quit IRC | 20:20 | |
| ssbarnea | fungi: ianw git-review https://review.openstack.org/#/c/195043/ | 20:23 |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Add ansible-role-chrony project https://review.openstack.org/603489 | 20:23 |
| *** jamesmcarthur has joined #openstack-infra | 20:26 | |
| *** bobh has joined #openstack-infra | 20:27 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Limit reviews by branch when specified https://review.openstack.org/312056 | 20:27 |
| clarkb | ianw: no rush, but figured I'd point gary to you since you volunteered :) gary is in the UK for timezone info | 20:27 |
| *** bobh has quit IRC | 20:31 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2 https://review.openstack.org/583535 | 20:36 |
| *** bobh has joined #openstack-infra | 20:37 | |
| *** bobh has quit IRC | 20:41 | |
| *** panda has quit IRC | 20:41 | |
| *** efried has quit IRC | 20:44 | |
| *** efried has joined #openstack-infra | 20:44 | |
| *** ansmith has quit IRC | 20:45 | |
| *** hamzy has quit IRC | 20:47 | |
| *** hashar has quit IRC | 20:47 | |
| *** bobh has joined #openstack-infra | 20:48 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Fix compare_review's use of fetch_review https://review.openstack.org/532359 | 20:50 |
| *** bobh has quit IRC | 20:52 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Stash and unstash changes during download https://review.openstack.org/340024 | 20:53 |
| *** rh-jelabarre has quit IRC | 20:53 | |
| *** weshay is now known as weshay_pto | 20:53 | |
| *** hashar has joined #openstack-infra | 20:54 | |
| *** jiapei has joined #openstack-infra | 20:54 | |
| *** mrsoul has joined #openstack-infra | 20:56 | |
| *** priteau has quit IRC | 20:57 | |
| *** bobh has joined #openstack-infra | 20:57 | |
| *** kgiusti has left #openstack-infra | 20:58 | |
| *** priteau has joined #openstack-infra | 20:58 | |
| *** bobh has quit IRC | 21:02 | |
| *** hashar has quit IRC | 21:03 | |
| ssbarnea | clarkb: or ianw : https://review.openstack.org/#/c/603061/ - thanks. | 21:04 |
| *** trown is now known as trown|outtypewww | 21:06 | |
| *** diablo_rojo has quit IRC | 21:07 | |
| *** anteaya has quit IRC | 21:08 | |
| *** bobh has joined #openstack-infra | 21:09 | |
| ssbarnea | can anyone give me a hint regarding what happens with 597581 which appears stuck for +3h on the gate? http://zuul.openstack.org/status.html -- zuul does not expand it. | 21:09 |
| clarkb | ssbarnea: zuul executes chagnes in gate pipelines with a window | 21:10 |
| clarkb | ssbarnea: that change is outside of the window so does not have jobs started on it yet. The window expands as changes successfully merge and ocntracts as they don't. It is similar to tcp slow start | 21:11 |
| clarkb | ssbarnea: the intent there is to prevent a flaky gate from thrashing cloud providers and eating up resources | 21:11 |
| *** bobh has quit IRC | 21:14 | |
| ssbarnea | clarkb: so that status would count as a "pre-queue". Now I realise that he "circle" color makes the difference gray or green and it has a good tooltip. the only problem was that tooltip appears only on the bullet. | 21:15 |
| clarkb | it is queueud but unsent in the tcp window analogy | 21:16 |
| ssbarnea | clarkb: now I know, if I ever have time I may make a CR to zuul to make the tooltip visible on the entire area, not only the "dot". | 21:16 |
| jiapei | Hi, I'm trying to add post_logs.yaml for zuul to run, but zuul-executor reports "'[Zuul] Log Stream did not terminate", what happened to it? | 21:17 |
| clarkb | jungleboyj: no problem, if only etherpad were more reliable :P | 21:17 |
| *** bobh has joined #openstack-infra | 21:17 | |
| clarkb | jiapei: can you link to a specific example? | 21:18 |
| clarkb | jiapei: or paste the log to a paste service? | 21:18 |
| jiapei | clarkb: I can paste it, just a minute | 21:18 |
| jungleboyj | clarkb: Well, I have taken many many days of notes in etherpad and this is the first time it has gone wrong. | 21:18 |
| *** gouthamr_ has joined #openstack-infra | 21:18 | |
| clarkb | jungleboyj: I think we've seen an occurence rate of about one reported issue like this per large event | 21:19 |
| clarkb | jungleboyj: but usually the recovery method I used works so its kind of a just do that and move on situation | 21:19 |
| jungleboyj | clarkb: Interesting. Well, glad that you guys had a recovery method. I was able to get what I needed. | 21:19 |
| *** dmellado has joined #openstack-infra | 21:20 | |
| jiapei | clarkb: logs are here http://paste.openstack.org/show/730317/ | 21:21 |
| *** bobh has quit IRC | 21:22 | |
| jiapei | clarkb: I find that zuul even doesn't try to clone the repo I specifys in line 13 at http://paste.openstack.org/show/730318/ | 21:23 |
| clarkb | jiapei: reading the zuul code I don't think the log stream did not terminate is the cause of the fialure | 21:25 |
| clarkb | lenovo-xclarity-ironic.yaml that playbook failed with an exit code of 2 | 21:26 |
| clarkb | and in cleaning that up the log streamer didn't terminate (if I am reading this properly) | 21:26 |
| *** gouthamr has quit IRC | 21:26 | |
| clarkb | jiapei: you should look at the console log for that job to see what the error is (or ara if you have that) | 21:27 |
| jiapei | clarkb: so lenovo-xclarity-ironic.yaml failed, then the post_log.yaml runs, and post_log.yaml fails too? Do I understand it correct? | 21:27 |
| *** bobh has joined #openstack-infra | 21:27 | |
| clarkb | jiapei: yes I think that is correct | 21:28 |
| clarkb | it is running but failing | 21:28 |
| *** Emine has joined #openstack-infra | 21:28 | |
| jiapei | Sigh... Let me check the playbook again | 21:29 |
| *** rlandy has quit IRC | 21:31 | |
| *** bobh has quit IRC | 21:32 | |
| *** agopi has quit IRC | 21:32 | |
| clarkb | Currently getting a shout out for infra on the board meeting (robust infrastructure built on free and open tools) | 21:33 |
| *** stevebaker has joined #openstack-infra | 21:36 | |
| *** ansmith has joined #openstack-infra | 21:37 | |
| *** bobh has joined #openstack-infra | 21:38 | |
| clarkb | jiapei: one thing that can help your debugging is you can put in the executor in a debug mode which will prevent it from deleting the workspaces of jobs | 21:39 |
| clarkb | which may show you the console logs (as it should be in the logs dir in the jobworkspace I think) | 21:39 |
| jiapei | clarkb: where is the workspaces's location? | 21:40 |
| jiapei | I mean the worksapces of jobs... | 21:40 |
| clarkb | jiapei: /var/lib/zuul/builds/$uuid | 21:40 |
| *** Emine has quit IRC | 21:40 | |
| jiapei | clarkb: got it, thank you clarkb :) | 21:41 |
| *** boden has quit IRC | 21:42 | |
| *** bobh has quit IRC | 21:43 | |
| *** bobh has joined #openstack-infra | 21:47 | |
| *** bobh has quit IRC | 21:48 | |
| *** bobh has joined #openstack-infra | 21:49 | |
| clarkb | infra-root I'm going to approve https://review.openstack.org/#/c/603173/ in about 2 hours. Mostly a heads up but if yo uhave some cycles to review that that is also appreciated. Will set max-servers to 0 in ovh bhs1 | 21:51 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul master: Assure that status tooltip is displayed on entire row https://review.openstack.org/603504 | 21:51 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Avoid UnicodeEncodeError on python 2 https://review.openstack.org/583535 | 21:59 |
| *** sshnaidm|ptg has joined #openstack-infra | 22:00 | |
| *** sshnaidm|ptg is now known as sshnaidm|pto | 22:00 | |
| *** ssbarnea|bkp has quit IRC | 22:03 | |
| *** agopi has joined #openstack-infra | 22:05 | |
| *** priteau has quit IRC | 22:05 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Add ansible-role-chrony project https://review.openstack.org/603489 | 22:09 |
| clarkb | ianw: any chance you could do some quick reviews on https://review.openstack.org/#/c/601348/3 and its children? I can approve and watch them as they go | 22:12 |
| * clarkb goes to review the 3pci spec now | 22:12 | |
| ianw | all lgtm; i just left static.o.o because that seems like one to watch more closely | 22:16 |
| clarkb | ianw: thanks | 22:19 |
| clarkb | ianw: responded to your comment on the 3pci spec. Let me know if that makes sense or if I should clarify futher | 22:20 |
| clarkb | I'll likely watch the first set of future parsers this evening, do static tomorrow morning then try to sprint thorugh the last set in the afternoon | 22:21 |
| clarkb | ianw: re linaro nodes it seems we haven't booted any new ones in a while. Any idea who is or was trying to run tests there? we might want to poke them and let them know there are new test resources they can take advantage of | 22:22 |
| ianw | clarkb: cool; note i also moved it to "help wanted" at this point, given recent personnel changes i'm not sure who will action the details of it all | 22:24 |
| clarkb | ya I tried to encourage people interested in running a third party ci at the ptg to reach out on that spec | 22:25 |
| clarkb | we'll hvae to see if we get any bites on that | 22:25 |
| *** slaweq has quit IRC | 22:26 | |
| ianw | xinliang has been the main person with arm64 ATM. however, adding some testing of the base infra stuff on arm64 is on my short-term todo list :) | 22:27 |
| *** diablo_rojo has joined #openstack-infra | 22:36 | |
| *** jaosorior_ has joined #openstack-infra | 22:42 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for cacti.o.o https://review.openstack.org/601348 | 22:44 |
| *** jaosorior has quit IRC | 22:45 | |
| *** rcernin has joined #openstack-infra | 22:47 | |
| *** janki has joined #openstack-infra | 22:47 | |
| *** dpawlik has joined #openstack-infra | 22:49 | |
| *** tpsilva has quit IRC | 22:52 | |
| *** dpawlik has quit IRC | 22:53 | |
| *** mriedem has quit IRC | 22:59 | |
| *** sshnaidm|pto has quit IRC | 22:59 | |
| *** dhill_ has quit IRC | 22:59 | |
| *** jamesmcarthur has quit IRC | 23:00 | |
| *** jamesmcarthur has joined #openstack-infra | 23:01 | |
| *** jamesmcarthur has quit IRC | 23:07 | |
| clarkb | ianw: https://review.openstack.org/#/c/601749/3 is a depends on for pbx futureparser | 23:08 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| ianw | oh missed that | 23:14 |
| ianw | puppet-beaker-rspec-puppet-4-infraFAILURE | 23:14 |
| ianw | /usr/sbin/asterisk -rx "module reload app_voicemail.so"' returned 1 instead of one of [0] | 23:15 |
| *** slaweq has quit IRC | 23:16 | |
| clarkb | hrm | 23:17 |
| *** tosky has quit IRC | 23:18 | |
| clarkb | I've approved the BHS1 disable change | 23:18 |
| ianw | what i know about asterisk plugin ordering and service status could fit inside -> <- | 23:30 |
| openstackgerrit | Merged openstack-infra/project-config master: OVH BHS1 Maintenance 2018-09-19 0700 UTC https://review.openstack.org/603173 | 23:30 |
| clarkb | ianw: ya maybe pabelanger will have a moment to educate us | 23:31 |
| clarkb | cacti just ran with futureparser and apperas happy | 23:32 |
| *** gfidente has joined #openstack-infra | 23:33 | |
| *** gfidente has quit IRC | 23:37 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove expire-old-reviews from docs https://review.openstack.org/603436 | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!