| *** dchen has joined #openstack-infra | 00:00 | |
| *** Goneri has joined #openstack-infra | 00:05 | |
| *** mattw4 has quit IRC | 00:06 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Update service/opendev-zuul key https://review.opendev.org/705873 | 00:19 |
|---|---|---|
| fungi | ianw: looking through the current tarballs sites, i'm not sure we're publishing zuul release tarballs anywhere besides pypi | 00:24 |
| ianw | fungi: that ... could be right. we don't have, say /afs/openstack.org/project/opendev.org/releases | 00:25 |
| fungi | we're only publishing branch tip artifacts at https://tarballs.opendev.org/zuul/zuul/ and the last release artifacts published in https://tarballs.openstack.org/zuul/ seems to be from mid-last year | 00:26 |
| fungi | i think all releases from july onward (around the same time we set up the branch tip tarballs via promote pipeline to tarballs.opendev.org) are only on pypi | 00:27 |
| fungi | i suspect that was not intentional | 00:27 |
| *** armax has quit IRC | 00:29 | |
| ianw | fungi: that would actually be the jobs i'm say aren't used @ https://review.opendev.org/#/c/705367/ ? | 00:29 |
| fungi | yep, that change is what got me looking | 00:33 |
| fungi | well, or zuul-release-python needs modifying to also copy files to the tarballs afs volume | 00:34 |
| fungi | zuul-release-python also isn't creating detached openpgp signatures for the files it's generating | 00:36 |
| fungi | looks like https://review.opendev.org/670388 was the change which removed use of the older release-zuul-python job | 00:37 |
| ianw | https://zuul.opendev.org/t/zuul/build/9f8770685cef4a9f882705e078990833 looks like the last run of zuul-release-python | 00:38 |
| fungi | (in favor of opendev-release-python which was then replaced by zuul-release-python in the later https://review.opendev.org/676466 change) | 00:39 |
| fungi | right, i looked through the log of that one a few minutes ago | 00:40 |
| fungi | it doesn't seem to do anything with signing artifacts or copying them to afs | 00:41 |
| fungi | not sure if those roles were missed when the job was written or needed recreating | 00:41 |
| ianw | ahhh, ok | 00:41 |
| ianw | or dropped for $reason? | 00:43 |
| *** jamesmcarthur has joined #openstack-infra | 00:45 | |
| fungi | it's not clear from review comments that was the case at least | 00:47 |
| fungi | but i'm still digging in history | 00:49 |
| fungi | https://review.openstack.org/649199 implemented opendev-release-python and skipped tarball uploading because it predated creation of tarballs.opendev.org | 00:51 |
| fungi | i suspect that's it, job was not completed once the new tarballs site was created | 00:53 |
| ianw | fungi: ok, what do you think, file a story? | 00:56 |
| ianw | as we mentioned before, it would be good to switch more to the opendev jobs, but i think after the extant jobs are publishing to afs | 00:57 |
| *** jamesmcarthur has quit IRC | 01:00 | |
| fungi | yeah, i get the impression that when corvus wrote 649199 he expected someone to add tarball site publication later before switching zuul to it, and then we just forgot to double-check that months later when reviewing 670388 | 01:05 |
| fungi | i wonder if it's as simple as just adding the missing roles for signing and copying to afs? | 01:06 |
| *** rcernin has joined #openstack-infra | 01:06 | |
| *** jamesmcarthur has joined #openstack-infra | 01:10 | |
| *** slaweq_ has joined #openstack-infra | 01:11 | |
| *** michael-beaver has quit IRC | 01:13 | |
| ianw | fungi: i don't think so because the release job builds (https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/release-python/run.yaml) while the existing tarballs jobs use promote pipeline? | 01:14 |
| ianw | i don't think it's unsolvable, but just an impedance mis-match | 01:14 |
| *** slaweq_ has quit IRC | 01:16 | |
| fungi | zuul-release-python (which is parented on opendev-release-python) runs in the release pipeline | 01:16 |
| fungi | opendev-promote-python is what zuul runs in the promote pipeline | 01:17 |
| fungi | unless i'm misunderstanding you | 01:17 |
| *** Lucas_Gray has quit IRC | 01:18 | |
| ianw | fungi: that's correct. opendev-promote-python is what uploads the tarball currently | 01:18 |
| ianw | this is where it's copied ... https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/artifacts/promote.yaml#L39 | 01:20 |
| fungi | ahh, i see, and the signing and copying for that aren't implemented as roles, they're baked into its playbooks | 01:20 |
| fungi | (or the copying anyway, and signing isn't implemented at all?) | 01:20 |
| ianw | yep, that was what i was going to say :) | 01:20 |
| ianw | i haven't seen anything related to gpg secrets in the opendev repos | 01:21 |
| *** jamesmcarthur has quit IRC | 01:22 | |
| *** ociuhandu has joined #openstack-infra | 01:31 | |
| *** ociuhandu has quit IRC | 01:35 | |
| *** armax has joined #openstack-infra | 01:36 | |
| *** vesper has joined #openstack-infra | 01:38 | |
| *** vesper11 has quit IRC | 01:39 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update letsencrypt_gid documentation https://review.opendev.org/705878 | 01:43 |
| *** jamesmcarthur has joined #openstack-infra | 01:45 | |
| *** Goneri has quit IRC | 01:47 | |
| *** igordc has joined #openstack-infra | 01:52 | |
| *** gyee has quit IRC | 02:00 | |
| ianw | kevinz: rather than try and maintain two builders, i feel like we just replace nb03 | 02:05 |
| kevinz | ianw: you mean setup nb03 at linaro-us side and remove the linaro-london side right? | 02:06 |
| kevinz | I think it is a good approach | 02:07 |
| ianw | yep, let me look at starting the server ... it's just about the only operation we have non-root can't do :) | 02:09 |
| ianw | kevinz: hrm, we'll need a xenial image in there, our builders don't support bionic yet | 02:12 |
| ianw | we can actually get our cloud launcher to do that i guess | 02:12 |
| kevinz | ianw: you mean that upload an xenial image to linaro-us? I can help to move it | 02:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: linaro-us : add Xenial image https://review.opendev.org/705879 | 02:15 |
| ianw | kevinz: ^ that should upload it. i think we can quick approve it due to the non-production status of all this | 02:16 |
| *** armax has quit IRC | 02:16 | |
| *** armax has joined #openstack-infra | 02:16 | |
| *** armax has quit IRC | 02:17 | |
| kevinz | cool | 02:18 |
| ianw | i just uploaded it anyway, to get this going | 02:20 |
| *** jamesmcarthur has quit IRC | 02:22 | |
| ianw | ok, launch it, let's see what happens | 02:22 |
| ianw | kevinz: hrm, so that's not booting | 02:27 |
| ianw | kevinz: http://paste.openstack.org/show/789145/ ... look familiar ? it's dropped into a shell i guess | 02:28 |
| kevinz | ianw: let me check | 02:28 |
| ianw | maybe i got the properties on the image wrong? | 02:31 |
| *** roman_g has quit IRC | 02:33 | |
| kevinz | ianw: could you try without any properties? | 02:35 |
| ianw | hrm, yep, let me delete them | 02:35 |
| *** rfolco|bbl has quit IRC | 02:40 | |
| ianw | kevinz: hrm, it still seem stuck? http://paste.openstack.org/show/789146/ | 02:42 |
| kevinz | ianw: sorry in a meeting now, will sort out the problem soon | 02:49 |
| kevinz | after the meeting | 02:49 |
| ianw | ok, np | 02:49 |
| *** rlandy has quit IRC | 03:03 | |
| openstackgerrit | Merged opendev/base-jobs master: Migrate to upload-afs-roots role https://review.opendev.org/705371 | 03:08 |
| kevinz | ianw: back. I tested and it still stuck... | 03:09 |
| kevinz | I will test one from uk builder | 03:09 |
| *** slaweq_ has joined #openstack-infra | 03:11 | |
| *** apetrich has quit IRC | 03:12 | |
| *** rh-jelabarre has quit IRC | 03:13 | |
| *** slaweq_ has quit IRC | 03:16 | |
| *** armax has joined #openstack-infra | 03:19 | |
| openstackgerrit | Merged openstack/project-config master: Remove publish-zuul-python-branch-tarball and release-zuul-python https://review.opendev.org/705367 | 03:27 |
| openstackgerrit | Merged openstack/project-config master: Migrate to upload-afs-roots role https://review.opendev.org/705370 | 03:27 |
| openstackgerrit | Merged openstack/project-config master: upload-artifacts-with-afs: use upload-afs-synchronzie role https://review.opendev.org/705374 | 03:27 |
| *** ykarel|away is now known as ykarel | 03:30 | |
| *** rcernin has quit IRC | 03:31 | |
| *** gagehugo has quit IRC | 03:45 | |
| *** ramishra has joined #openstack-infra | 03:50 | |
| *** gagehugo has joined #openstack-infra | 03:59 | |
| *** ramishra has quit IRC | 04:01 | |
| *** armax has quit IRC | 04:03 | |
| ianw | kevinz: i'm not sure what the difference would be? | 04:18 |
| *** raukadah is now known as chkumar|rover | 04:22 | |
| *** ramishra has joined #openstack-infra | 04:24 | |
| *** dSrinivas has joined #openstack-infra | 04:42 | |
| *** adriant has quit IRC | 04:45 | |
| *** iokiwi has quit IRC | 04:45 | |
| *** adriant has joined #openstack-infra | 04:50 | |
| *** udesale has joined #openstack-infra | 04:53 | |
| *** bnemec has joined #openstack-infra | 04:53 | |
| *** hongbin has joined #openstack-infra | 04:55 | |
| ianw | i feel like hrw helped me with this before | 04:56 |
| *** rcernin has joined #openstack-infra | 04:57 | |
| ianw | 2020-02-05 02:28:20.559265 | bridge.openstack.org | TASK [install-kubectl : Install kubectl] *************************************** | 04:58 |
| ianw | 2020-02-05 02:28:20.574481 | bridge.openstack.org | Wednesday 05 February 2020 02:28:20 +0000 (0:00:11.462) 0:00:16.763 **** | 04:58 |
| ianw | 2020-02-05 02:51:40.272354 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/opendev/system-config/playbooks/zuul/run-base.yaml@master] | 04:58 |
| *** rcernin has quit IRC | 04:58 | |
| ianw | have to keep an eye on that in the gate ... it looks like that hung the test ... probably just upstream issues | 04:58 |
| *** rcernin has joined #openstack-infra | 04:58 | |
| *** igordc has quit IRC | 04:59 | |
| *** slaweq_ has joined #openstack-infra | 05:11 | |
| *** slaweq_ has quit IRC | 05:15 | |
| *** hongbin has quit IRC | 05:28 | |
| *** soniya29 has joined #openstack-infra | 05:29 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| kevinz | ianw: I use this image: https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-arm64-uefi1.img, with uefi support, can boot into kernel, but failed to boot the system | 05:38 |
| kevinz | ianw: I will talk with hrw later today | 05:38 |
| kevinz | Also I test image got from uk cloud, could not boot either.. | 05:39 |
| ianw | kevinz: yeah, that's what i uploaded ... that would be great. i feel like we needed those property flags originally in the cn cloud; but then i see in the uk cloud it doesn't have them | 05:39 |
| ianw | maybe there's some magic config happening behind the scenes | 05:39 |
| kevinz | ianw: that's what I got now :http://paste.openstack.org/show/789152/ | 05:40 |
| ianw | huh, that got into the kernel which i didn't see ... but that may be a serial to console issue? | 05:43 |
| openstackgerrit | Merged zuul/zuul-jobs master: Use unique loop vars to avoid conflicts https://review.opendev.org/705337 | 05:43 |
| openstackgerrit | Merged zuul/zuul-jobs master: Link the policy to the CONTRIBUTING.rst file https://review.opendev.org/704484 | 05:50 |
| *** rcernin has quit IRC | 06:03 | |
| *** redrobot has quit IRC | 06:04 | |
| *** tetsuro has joined #openstack-infra | 06:06 | |
| *** redrobot has joined #openstack-infra | 06:06 | |
| openstackgerrit | Merged openstack/project-config master: Add "Review-Priority" gerrit label to devstack https://review.opendev.org/705547 | 06:10 |
| *** tetsuro has quit IRC | 06:10 | |
| *** lmiccini has joined #openstack-infra | 06:13 | |
| *** tetsuro has joined #openstack-infra | 06:15 | |
| *** tetsuro has quit IRC | 06:27 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 06:36 |
| *** tetsuro has joined #openstack-infra | 06:58 | |
| *** roman_g has joined #openstack-infra | 07:00 | |
| *** tetsuro has quit IRC | 07:01 | |
| *** tetsuro has joined #openstack-infra | 07:01 | |
| *** tetsuro has quit IRC | 07:02 | |
| *** tetsuro has joined #openstack-infra | 07:03 | |
| *** slaweq_ has joined #openstack-infra | 07:11 | |
| *** slaweq_ has quit IRC | 07:16 | |
| *** tetsuro has quit IRC | 07:17 | |
| *** tetsuro has joined #openstack-infra | 07:19 | |
| *** tetsuro has joined #openstack-infra | 07:20 | |
| *** tetsuro has quit IRC | 07:31 | |
| *** tetsuro has joined #openstack-infra | 07:32 | |
| *** pgaxatte has joined #openstack-infra | 07:37 | |
| *** ociuhandu has joined #openstack-infra | 07:38 | |
| *** jaosorior has joined #openstack-infra | 07:38 | |
| *** ykarel is now known as ykarel|lunch | 07:42 | |
| *** dpawlik has joined #openstack-infra | 07:49 | |
| *** ociuhandu has quit IRC | 07:50 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 07:52 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Store information about gate resets in MQTT and SQL reporter https://review.opendev.org/696670 | 07:52 |
| *** dpawlik has quit IRC | 07:53 | |
| *** jaosorior has quit IRC | 07:53 | |
| *** iurygregory has quit IRC | 07:53 | |
| *** dpawlik has joined #openstack-infra | 07:56 | |
| *** tetsuro has quit IRC | 07:57 | |
| *** tetsuro has joined #openstack-infra | 08:00 | |
| *** slaweq_ has joined #openstack-infra | 08:00 | |
| *** jaosorior has joined #openstack-infra | 08:06 | |
| *** slaweq__ has joined #openstack-infra | 08:09 | |
| *** slaweq_ has quit IRC | 08:10 | |
| *** iurygregory has joined #openstack-infra | 08:11 | |
| *** jtomasek has quit IRC | 08:12 | |
| *** tkajinam has quit IRC | 08:12 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 08:12 |
| *** tetsuro has quit IRC | 08:14 | |
| *** slaweq has joined #openstack-infra | 08:14 | |
| *** tesseract has joined #openstack-infra | 08:16 | |
| *** slaweq__ has quit IRC | 08:16 | |
| *** tommylikehu has quit IRC | 08:22 | |
| *** hashar has joined #openstack-infra | 08:27 | |
| *** ociuhandu has joined #openstack-infra | 08:30 | |
| *** rcernin has joined #openstack-infra | 08:31 | |
| *** jtomasek has joined #openstack-infra | 08:34 | |
| *** rpittau|afk is now known as rpittau | 08:34 | |
| *** ociuhandu has quit IRC | 08:34 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Spec for allowing circular dependencies https://review.opendev.org/643309 | 08:36 |
| *** ociuhandu has joined #openstack-infra | 08:36 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 08:36 |
| *** ociuhandu has quit IRC | 08:40 | |
| *** gfidente has joined #openstack-infra | 08:41 | |
| *** ralonsoh has joined #openstack-infra | 08:42 | |
| *** ociuhandu has joined #openstack-infra | 08:44 | |
| *** gshippey has joined #openstack-infra | 08:45 | |
| *** rcernin has quit IRC | 08:48 | |
| *** apetrich has joined #openstack-infra | 08:48 | |
| *** slaweq has quit IRC | 08:48 | |
| *** slaweq_ has joined #openstack-infra | 08:49 | |
| *** ociuhandu has quit IRC | 08:49 | |
| *** ykarel|lunch is now known as ykarel | 08:52 | |
| *** jpena|off is now known as jpena | 08:53 | |
| *** rcernin has joined #openstack-infra | 08:53 | |
| *** xek has joined #openstack-infra | 08:54 | |
| *** ociuhandu has joined #openstack-infra | 08:59 | |
| *** lucasagomes has joined #openstack-infra | 09:02 | |
| *** slaweq__ has joined #openstack-infra | 09:06 | |
| *** zxiiro has quit IRC | 09:06 | |
| *** slaweq_ has quit IRC | 09:07 | |
| *** rcernin has quit IRC | 09:07 | |
| *** ociuhandu has quit IRC | 09:13 | |
| *** pkopec has joined #openstack-infra | 09:15 | |
| *** rcernin has joined #openstack-infra | 09:16 | |
| kevinz | ianw: there? | 09:18 |
| openstackgerrit | Tobias Henkel proposed openstack/diskimage-builder master: Fix cache-url -f https://review.opendev.org/705902 | 09:19 |
| kevinz | ianw: hrw and I tested that boot the xenial with ipv4 subnet works fine. But for both IPv4 and IPv6, it could not boot.. | 09:19 |
| *** rcernin has quit IRC | 09:20 | |
| *** slaweq__ is now known as slaweq | 09:25 | |
| *** rcernin has joined #openstack-infra | 09:26 | |
| *** ociuhandu has joined #openstack-infra | 09:32 | |
| *** rcernin has quit IRC | 09:33 | |
| *** ociuhandu has quit IRC | 09:38 | |
| *** lmiccini has quit IRC | 09:39 | |
| *** lmiccini has joined #openstack-infra | 09:40 | |
| *** derekh has joined #openstack-infra | 09:40 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Spec for allowing circular dependencies https://review.opendev.org/643309 | 10:04 |
| *** rfolco|bbl has joined #openstack-infra | 10:08 | |
| *** ociuhandu has joined #openstack-infra | 10:14 | |
| *** ramishra has quit IRC | 10:15 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed zuul/zuul-jobs master: Added bzip2 package for CentOS 8 https://review.opendev.org/705911 | 10:18 |
| *** ociuhandu has quit IRC | 10:20 | |
| *** ramishra has joined #openstack-infra | 10:21 | |
| *** slaweq_ has joined #openstack-infra | 10:40 | |
| *** hashar has quit IRC | 10:41 | |
| *** slaweq has quit IRC | 10:42 | |
| openstackgerrit | Riccardo Pittau proposed openstack/diskimage-builder master: Allow python3 to be used in CentOS 7 https://review.opendev.org/705920 | 10:48 |
| *** slaweq__ has joined #openstack-infra | 11:01 | |
| *** dciabrin has joined #openstack-infra | 11:02 | |
| *** slaweq_ has quit IRC | 11:03 | |
| *** rpittau is now known as rpittau|bbl | 11:10 | |
| *** udesale_ has joined #openstack-infra | 11:13 | |
| *** pkopec has quit IRC | 11:14 | |
| *** udesale has quit IRC | 11:16 | |
| *** ramishra has quit IRC | 11:18 | |
| *** ramishra has joined #openstack-infra | 11:22 | |
| *** dciabrin has quit IRC | 11:28 | |
| *** psachin has joined #openstack-infra | 11:30 | |
| *** pcaruana has quit IRC | 11:37 | |
| *** ahosam has joined #openstack-infra | 11:39 | |
| *** ahosam has quit IRC | 11:46 | |
| *** pcaruana has joined #openstack-infra | 11:50 | |
| *** ociuhandu has joined #openstack-infra | 11:52 | |
| *** ociuhandu has quit IRC | 11:58 | |
| *** dciabrin has joined #openstack-infra | 12:01 | |
| *** ociuhandu has joined #openstack-infra | 12:05 | |
| *** derekh has quit IRC | 12:07 | |
| *** derekh has joined #openstack-infra | 12:08 | |
| *** ociuhandu has quit IRC | 12:11 | |
| *** tosky has joined #openstack-infra | 12:12 | |
| *** ociuhandu has joined #openstack-infra | 12:17 | |
| *** slaweq__ has quit IRC | 12:19 | |
| *** ociuhandu has quit IRC | 12:22 | |
| *** slaweq__ has joined #openstack-infra | 12:23 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** soniya29 has quit IRC | 12:38 | |
| *** Lucas_Gray has joined #openstack-infra | 12:40 | |
| *** rh-jelabarre has joined #openstack-infra | 12:45 | |
| *** rfolco|bbl is now known as rfolco | 12:45 | |
| *** ociuhandu has joined #openstack-infra | 12:57 | |
| *** Lucas_Gray has quit IRC | 13:01 | |
| *** rlandy has joined #openstack-infra | 13:01 | |
| *** Lucas_Gray has joined #openstack-infra | 13:02 | |
| *** nhicher has quit IRC | 13:05 | |
| *** nhicher has joined #openstack-infra | 13:06 | |
| *** slaweq has joined #openstack-infra | 13:11 | |
| *** slaweq__ has quit IRC | 13:13 | |
| *** ociuhandu has quit IRC | 13:13 | |
| *** rpittau|bbl is now known as rpittau | 13:15 | |
| *** smcginnis|FOSDEM is now known as smcginnis | 13:19 | |
| *** jamesmcarthur has joined #openstack-infra | 13:20 | |
| *** ociuhandu has joined #openstack-infra | 13:22 | |
| openstackgerrit | Riccardo Pittau proposed openstack/diskimage-builder master: Allow python3 to be used in CentOS 7 https://review.opendev.org/705920 | 13:23 |
| *** lpetrut has joined #openstack-infra | 13:24 | |
| *** ociuhandu has quit IRC | 13:27 | |
| *** ykarel is now known as ykarel|away | 13:27 | |
| *** ociuhandu has joined #openstack-infra | 13:28 | |
| *** ociuhandu has quit IRC | 13:29 | |
| *** electrofelix has joined #openstack-infra | 13:29 | |
| *** ociuhandu has joined #openstack-infra | 13:29 | |
| *** jpena|lunch is now known as jpena | 13:35 | |
| *** jamesmcarthur has quit IRC | 13:37 | |
| *** Lucas_Gray has quit IRC | 13:38 | |
| *** Lucas_Gray has joined #openstack-infra | 13:39 | |
| *** dciabrin has quit IRC | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:47 | |
| *** smarcet has joined #openstack-infra | 13:49 | |
| *** pkopec has joined #openstack-infra | 13:52 | |
| mordred | fungi: is there some trick these days to getting console log to work on rax vms? | 13:53 |
| *** zxiiro has joined #openstack-infra | 13:53 | |
| mordred | or, rather, the emergency serial console thing? | 13:53 |
| fungi | via the web browser, i pull up the server instance details and then there's an actions drop-down to let you choose the emergency console | 13:54 |
| fungi | that should open in a new window/tab though maybe you have a pop-up blocker or something rejecting that? | 13:54 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement github checks API https://review.opendev.org/705168 | 13:55 |
| fungi | at any rate, it usually comes up for me as a black rectangle because the screen has gone to sleep, but giving it input focus and then hitting enter usually wakes it up | 13:55 |
| *** liuyulong has joined #openstack-infra | 13:58 | |
| mordred | hrm | 13:59 |
| *** dciabrin has joined #openstack-infra | 14:00 | |
| *** eharney has joined #openstack-infra | 14:01 | |
| *** rpittau is now known as rpittau|mtg | 14:03 | |
| stephenfin | mordred, fungi, clarkb, <others>: I'm thinking we should hold off on removing py27 support from pbr because afaict we don't pin pbr anywhere and we'd break *everyone* still using Python 2.7. Correct? https://review.opendev.org/#/c/705693 | 14:04 |
| *** dtantsur is now known as dtantsur|brb | 14:05 | |
| fungi | stephenfin: quite possibly. we implement it as a setup_requires so it's fetched out of band from normal install_requires | 14:08 |
| fungi | and you're right, in openstack/nova stable/rocky for example, we pass it as setup_requires=['pbr>=2.0.0'] so no version cap | 14:09 |
| mordred | yeah - let's hold off on pbr | 14:15 |
| mordred | also - it doesn't really change much | 14:15 |
| *** pcaruana has quit IRC | 14:17 | |
| mordred | fungi: so - in rax dfw - review-dev01.opendev.org seems stuck in rebooting state and I'm not sure how to debug further (I get nothing, not even a black box on the emergency console) | 14:17 |
| fungi | make a review-dev02? is there benefit in testing out a fresh redeployment? | 14:18 |
| fungi | did you do a hard reboot/server reboot --force? | 14:18 |
| fungi | or just a "soft" reboot? | 14:19 |
| *** ykarel|away is now known as ykarel | 14:20 | |
| fungi | i'll see if i have any luck with the console | 14:20 |
| fungi | mordred: same, console just gives me "loading console" and a spinner | 14:24 |
| fungi | pretty sure the server isn't in a state where its console is reachable by nova | 14:25 |
| *** rpittau|mtg is now known as rpittau | 14:25 | |
| *** psachin has quit IRC | 14:25 | |
| *** ociuhandu has quit IRC | 14:29 | |
| *** Lucas_Gray has quit IRC | 14:30 | |
| mordred | fungi: *awesome* | 14:31 |
| mordred | fungi: and yeah - this is the new server I made yesterday :) | 14:31 |
| fungi | ouch | 14:31 |
| mordred | so, I can totally blow it away and build it again - that's just a bummer | 14:31 |
| fungi | yeah, that's not great | 14:32 |
| mordred | it was in read-only root filesystem mode this morning | 14:32 |
| fungi | sounds related | 14:32 |
| mordred | so I tried just rebooting it since it wasn't in production or anything | 14:32 |
| mordred | and things have not gone well since | 14:32 |
| *** ociuhandu has joined #openstack-infra | 14:33 | |
| *** Lucas_Gray has joined #openstack-infra | 14:33 | |
| *** jamesdenton has joined #openstack-infra | 14:34 | |
| *** ociuhandu has quit IRC | 14:37 | |
| openstackgerrit | Alfredo Moralejo proposed zuul/zuul-jobs master: Add CentOS8 High Availability repository https://review.opendev.org/705973 | 14:45 |
| openstackgerrit | Alfredo Moralejo proposed zuul/zuul-jobs master: Add CentOS8 High Availability repository https://review.opendev.org/705973 | 14:47 |
| mordred | fungi: oh - and of course, it's not going to let me delete it because it's in rebooting state | 14:51 |
| *** sshnaidm|afk is now known as sshnaidm | 14:51 | |
| mordred | fungi: I'm going to have to file a ticket aren't I? | 14:51 |
| clarkb | mordred: did you hard reboot it? | 14:51 |
| mordred | clarkb: yes | 14:52 |
| mordred | clarkb: and it's stuck in rebooting - for about an hour now | 14:52 |
| clarkb | ok sometimes the avpi event for spft reboot gets ignored by the VM and the dtate doesnt change thereeither | 14:52 |
| *** ociuhandu has joined #openstack-infra | 14:54 | |
| mordred | clarkb, fungi: I was able to delete it | 14:57 |
| *** Lucas_Gray has quit IRC | 14:58 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update project doc to reflect OpenDev changes https://review.opendev.org/703488 | 14:59 |
| mordred | WOW | 15:00 |
| mordred | since yesterday the base xenial image we were using has gone away or been renamed :) | 15:00 |
| fungi | neat | 15:05 |
| ttx | Hmm, I'm confused. Where should I go to define a new label that Zuul can vote on? | 15:07 |
| mordred | super weird - if I do an image show on the uuid reported by the image for review01 - it totally shows up with the name that's expected - but I can't find it in launch.py | 15:07 |
| ttx | Looks like the Gerrit ACL has some, but maps them to groups | 15:07 |
| mordred | but - there is a new one that replaces PVHVM with Cloud - I'm trying that one :( | 15:08 |
| fungi | ttx: yes, that's precisely how it works | 15:08 |
| ttx | Can't find where for example you allow Zuul to vote Verified+1 | 15:08 |
| fungi | it's in the all-projects acl, which is not managed by project-config | 15:08 |
| *** ykarel is now known as ykarel|away | 15:08 | |
| fungi | just a sec | 15:08 |
| ttx | ah. That would.. explain my grep fail | 15:09 |
| clarkb | and then in the zuul pipeline config you define actions for succes/failure and that includes the gerrit category and value | 15:09 |
| fungi | ttx: https://docs.openstack.org/infra/system-config/gerrit.html#access-controls | 15:10 |
| ttx | clarkb: yeah that part I got right | 15:10 |
| *** Lucas_Gray has joined #openstack-infra | 15:11 | |
| fungi | ttx: that document contains the project.config file we use for configuring gerrit's all-projects metaproject | 15:11 |
| fungi | ttx: i think you want "group Continuous Integration Tools" | 15:12 |
| ttx | OK looks like the group I shall use would be... yes that | 15:12 |
| ttx | Thanks! | 15:12 |
| clarkb | mordred: stephenfin re pbr correct we cannot drop python2 from there unless we want to break all of the valid users of pbr with python2 | 15:12 |
| *** eharney has quit IRC | 15:13 | |
| stephenfin | clarkb: Cool. You're all in agreement so yeah, we'll hold off on this for another year (or even two) | 15:14 |
| clarkb | stephenfin: or 10 :) dependso n how long openstack stable branches live plus some leeway probably | 15:14 |
| stephenfin | :) | 15:15 |
| clarkb | train in particular is expected to be supported for a long time because it does python2 right? | 15:16 |
| clarkb | and we don't want to break that | 15:16 |
| stephenfin | I would imagine so, yes. In hindsight having an upper bound of the next major release across projects wouldn't have been a terrible idea | 15:17 |
| clarkb | it doesn't work | 15:17 |
| stephenfin | but no way am I going to submit the hundreds of patches needed to do that | 15:17 |
| clarkb | in fact doing that breaks all kinds of stuff | 15:17 |
| stephenfin | oh? | 15:17 |
| clarkb | I want to say we pinned all pbr usage to <1.0.0 way back when. Then 1.0.0 released and due to bootstrap methods for setup_requires 1.0.0 was pulled down anyway but then running setup.py failed beacuse 1.0.0 was not in the valid range | 15:18 |
| mordred | yeah - pbr is a special pony | 15:18 |
| clarkb | the problem is specifically with setup_requires | 15:18 |
| mordred | it's why we're so cautious about making changes to it | 15:19 |
| mordred | maybe one day with the new whatever system for specifying libraries that take care of building distributions we can rework it | 15:19 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add index_links option to zuul manifest https://review.opendev.org/705580 | 15:19 |
| clarkb | mordred: ya that should actually respect versions and even verifies checksums! | 15:19 |
| stephenfin | ah, I thought it was so we wouldn't be blocked on older branches if there was an urgent change needed in pbr | 15:20 |
| stephenfin | TIL | 15:20 |
| fungi | it's possible setup_requires versioning works now with pip shadowing easy_install | 15:20 |
| fungi | but that'll still only be the case for people using pip from the last year or two i think | 15:21 |
| *** Lucas_Gray has quit IRC | 15:21 | |
| *** Wryhder has joined #openstack-infra | 15:21 | |
| stephenfin | somewhat related, apparently people have been hired by the PSF to improve pip and fix e.g. the horrific dependency resolver it has | 15:21 |
| stephenfin | they were asking for guinea pigs in the Python devroom at FOSDEM | 15:21 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Google logs: Link to index.html files https://review.opendev.org/705586 | 15:21 |
| *** Wryhder is now known as Lucas_Gray | 15:22 | |
| *** ociuhandu_ has joined #openstack-infra | 15:22 | |
| clarkb | stephenfin: I try to keep up to date on that bug (I was the original filer) | 15:23 |
| clarkb | 988 iirc | 15:23 |
| stephenfin | Not sure if it's the bug, but the wiki page for the work is here https://wiki.python.org/psf/Pip2020DonorFundedRoadmap | 15:24 |
| clarkb | https://github.com/pypa/pip/issues/988 is the bug | 15:25 |
| *** eernst has joined #openstack-infra | 15:25 | |
| *** eernst has quit IRC | 15:25 | |
| *** ociuhandu has quit IRC | 15:25 | |
| fungi | stephenfin: yep, they've been funding sumana and the rest of the crew at changeset to help with pypa (pip, warehouse/pypi, et cetera) stuff | 15:26 |
| clarkb | I hope they succeed and have more productivity in that code base than I | 15:26 |
| stephenfin | fungi: Yeah, I saw the work on supporting tokens in warehouse | 15:27 |
| *** ociuhandu_ has quit IRC | 15:27 | |
| *** chkumar|rover is now known as raukadah | 15:27 | |
| stephenfin | I'm hoping they add support for user groups at some point. I tried doing it but it was too much for a weekend project and the ground kept shifting under my feet :( | 15:27 |
| *** lpetrut has quit IRC | 15:28 | |
| clarkb | stephenfin: groups for managing pypi packages? that could be helpful for projects with lots of packages | 15:30 |
| stephenfin | yup, for those that don't have it all automated | 15:30 |
| stephenfin | (the sphinx-doc and sphinx-contrib organizations on GitHub, in my case) | 15:31 |
| openstackgerrit | Monty Taylor proposed opendev/zone-opendev.org master: Update review-dev01.opendev.org IP addresses https://review.opendev.org/705985 | 15:31 |
| mordred | clarkb, fungi: ^^ | 15:31 |
| clarkb | my struggle has typically been with their review culture. I don't know if we are spoiled or my expectations are wrong, but I basically got told to take a hike on my last PR and that my code wasn't even worthy of proper review because there was so much wrong with it | 15:32 |
| clarkb | so now I hope others will succeed in making pip better | 15:32 |
| clarkb | mordred: you'll need to update the inventory file too (and possibly clean out the ansible cache) | 15:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add review-dev01.opendev.org https://review.opendev.org/705669 | 15:33 |
| mordred | yup ^^ | 15:33 |
| Shrews | clarkb: i had a similar unfortunate experience years ago in a java-based community project that shall remain nameless. :( | 15:33 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add upload-logs-gcs role https://review.opendev.org/703711 | 15:34 |
| mordred | clarkb: re ansible cache - what do you mean? | 15:34 |
| clarkb | mordred: if ansible had previously talked to that host it would think the IP address was the old one | 15:34 |
| clarkb | mordred: though maybe it wasn't the IP and it was other facts | 15:35 |
| clarkb | rm'ing the file in /var/cache/ansible (or wherever it goes) seemed tofix it in the past | 15:35 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Get LE certs for review.o.o https://review.opendev.org/705670 | 15:35 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove review-dev01.openstack.org https://review.opendev.org/705671 | 15:35 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use LE certs for Apache https://review.opendev.org/705690 | 15:35 |
| mordred | clarkb: nod. will do | 15:35 |
| clarkb | in this case looks like we hadn't added it to the inventory yet anyway so should be fine | 15:35 |
| mordred | OH FFS | 15:35 |
| mordred | clarkb: don't approve that yet | 15:36 |
| clarkb | I won't | 15:36 |
| mordred | I ... | 15:36 |
| mordred | I didn't replace "servername" in the launch readme | 15:36 |
| mordred | so I just created "servername.opendev.org" | 15:36 |
| clarkb | ha | 15:36 |
| mordred | yeah | 15:36 |
| clarkb | drink more coffee? | 15:36 |
| * mordred sighs | 15:36 | |
| mordred | yes | 15:36 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add disallowed-labels tenant option https://review.opendev.org/705856 | 15:36 |
| *** ramishra has quit IRC | 15:37 | |
| *** ianychoi_ is now known as ianychoi | 15:40 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add index_links option to zuul manifest https://review.opendev.org/705580 | 15:40 |
| *** osmanlicilegi has quit IRC | 15:42 | |
| *** osmanlicilegi has joined #openstack-infra | 15:42 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Google logs: Link to index.html files https://review.opendev.org/705586 | 15:44 |
| *** armax has joined #openstack-infra | 15:47 | |
| *** pcaruana has joined #openstack-infra | 15:49 | |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Add release-approval pipeline https://review.opendev.org/705990 | 15:50 |
| fungi | oh this is neat, apparently rackspace has started listing their ipv6 netblocks in the spamhaus pbl as well | 15:51 |
| *** dciabrin has quit IRC | 15:52 | |
| fungi | so now a lot of our sysadmins are bouncing cronspam back from our servers | 15:52 |
| clarkb | while I'm not actually sitting in a boat on a river in the rain freezing, I've got sick kid at home so will probably end up being half here anyway | 15:52 |
| fungi | i guess i'll try to work on getting all the ones i'm seeing bounces for delisted | 15:52 |
| clarkb | fungi: thanks! | 15:52 |
| *** dtantsur|brb is now known as dtantsur | 15:53 | |
| mordred | fungi: oh joy! | 15:56 |
| *** smarcet has quit IRC | 15:57 | |
| openstackgerrit | Monty Taylor proposed opendev/zone-opendev.org master: Update review-dev01.opendev.org IP addresses https://review.opendev.org/705985 | 15:57 |
| mordred | clarkb, fungi: _now_ that's valid | 15:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add review-dev01.opendev.org https://review.opendev.org/705669 | 15:59 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Get LE certs for review.o.o https://review.opendev.org/705670 | 15:59 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove review-dev01.openstack.org https://review.opendev.org/705671 | 15:59 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use LE certs for Apache https://review.opendev.org/705690 | 15:59 |
| *** ociuhandu has joined #openstack-infra | 16:01 | |
| *** Goneri has joined #openstack-infra | 16:02 | |
| *** eharney has joined #openstack-infra | 16:03 | |
| fungi | #status log delisted ipv6 address of storyboard01.opendev.org from the spamhaus pbl | 16:03 |
| fungi | hrm, looks like there's something else which also needs fixing | 16:03 |
| *** openstackstatus has joined #openstack-infra | 16:05 | |
| *** ChanServ sets mode: +v openstackstatus | 16:05 | |
| fungi | #status log restarted statusbot after failing to return from a 2020-02-04 11:16:50 utc netsplit | 16:05 |
| openstackstatus | fungi: finished logging | 16:05 |
| fungi | #status log delisted ipv6 address of storyboard01.opendev.org from the spamhaus pbl | 16:05 |
| *** ociuhandu has quit IRC | 16:05 | |
| openstackstatus | fungi: finished logging | 16:05 |
| mordred | infra-root: could I get some eyes on https://review.opendev.org/#/c/705985 real quick (and its parent) - I need to get that updated so I can keep rolling on review-dev | 16:06 |
| fungi | mordred: regarding 705662, there's a reason it didn't have a caa rr yet | 16:12 |
| fungi | we'll have to add those after we cut over to letsencrypt-issued certs for review.opendev.org | 16:13 |
| *** smarcet has joined #openstack-infra | 16:13 | |
| mordred | fungi: ah. so I should ditch that change for now? | 16:13 |
| fungi | oh, actually, wait, i'm confusing that with cert pinning | 16:13 |
| fungi | it's *probably* safe to add now, the caa record tells certificate authorities that if they're not letsencrypt.org then they shouldn't issue a cert for review.o.o and such | 16:14 |
| fungi | as long as we don't try to ask namecheap for another cert for review.o.o then it shouldn't be an issue | 16:14 |
| mordred | nod. well, yeah, I think we're going to be on LE before the existing one expires | 16:15 |
| mordred | let me rephrase, we *BETTER* be on LE before the namecheap expires in april | 16:15 |
| *** ramishra has joined #openstack-infra | 16:16 | |
| fungi | okay, both of those are approved | 16:17 |
| *** lmiccini has quit IRC | 16:18 | |
| *** slaweq_ has joined #openstack-infra | 16:23 | |
| *** jamesmcarthur has quit IRC | 16:24 | |
| *** udesale_ has quit IRC | 16:24 | |
| *** slaweq has quit IRC | 16:25 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Clean up LE entry for review.o.o https://review.opendev.org/705662 | 16:25 |
| mordred | fungi: woot - thanks! | 16:26 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Update review-dev01.opendev.org IP addresses https://review.opendev.org/705985 | 16:27 |
| fungi | infra-root: can we delete mirror01.mtl01.inap.opendev.org? i see mirror.mtl01.inap.opendev.org is currently a cname to mirror02 and there is no dns record for mirror01, so cronspam deliveries from it are bouncing due to lack of an address rr | 16:27 |
| *** smarcet has quit IRC | 16:29 | |
| *** dciabrin has joined #openstack-infra | 16:31 | |
| mordred | fungi: I do not know the answer - but my hunch would be that it's ok given those informations | 16:32 |
| corvus | ditto | 16:32 |
| *** smarcet has joined #openstack-infra | 16:33 | |
| *** psachin has joined #openstack-infra | 16:38 | |
| cgoncalves | hey infra! is it possible to add project specific tasks on stable branch create to openstack bot? like it does to update the requirements URL in tox.ini | 16:38 |
| fungi | cool, i'll probably go ahead and turn it off after i return from lunch, and then delete it if nothing breaks | 16:38 |
| *** jamesmcarthur has joined #openstack-infra | 16:38 | |
| fungi | in the meantime, i'm going to go find lunch. back in an hour or so and will continue going through cronspam bounces to see what else needs delisting in the pbl | 16:39 |
| *** jamesmcarthur_ has joined #openstack-infra | 16:41 | |
| *** jamesmcarthur has quit IRC | 16:44 | |
| *** smarcet has quit IRC | 16:46 | |
| *** sean-k-mooney has quit IRC | 16:51 | |
| *** dciabrin has quit IRC | 16:53 | |
| *** lucasagomes has quit IRC | 16:53 | |
| *** artom has quit IRC | 16:54 | |
| *** jamesmcarthur_ has quit IRC | 16:55 | |
| *** psachin has quit IRC | 16:58 | |
| *** jamesmcarthur has joined #openstack-infra | 16:58 | |
| *** Lucas_Gray has quit IRC | 17:04 | |
| *** jamesmcarthur has quit IRC | 17:05 | |
| *** eernst has joined #openstack-infra | 17:05 | |
| *** pgaxatte has quit IRC | 17:06 | |
| *** zxiiro has quit IRC | 17:07 | |
| *** zxiiro has joined #openstack-infra | 17:08 | |
| *** iurygregory has quit IRC | 17:08 | |
| *** Lucas_Gray has joined #openstack-infra | 17:11 | |
| *** soniya29 has joined #openstack-infra | 17:24 | |
| *** smarcet has joined #openstack-infra | 17:25 | |
| *** jamesmcarthur has joined #openstack-infra | 17:27 | |
| *** gyee has joined #openstack-infra | 17:29 | |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| *** igordc has joined #openstack-infra | 17:34 | |
| *** eernst has quit IRC | 17:35 | |
| *** sean-k-mooney has joined #openstack-infra | 17:36 | |
| *** sean-k-mooney has quit IRC | 17:41 | |
| *** sean-k-mooney has joined #openstack-infra | 17:43 | |
| *** vdrok has quit IRC | 17:45 | |
| *** vdrok has joined #openstack-infra | 17:45 | |
| *** dtantsur is now known as dtantsur|afk | 17:48 | |
| *** sean-k-mooney1 has joined #openstack-infra | 17:48 | |
| *** sean-k-mooney has quit IRC | 17:48 | |
| *** sean-k-mooney1 is now known as sean-k-mooney | 17:49 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add spec for scale out scheduler https://review.opendev.org/621479 | 17:49 |
| *** jaosorior has quit IRC | 17:49 | |
| *** jpena is now known as jpena|off | 17:53 | |
| *** jamesmcarthur has quit IRC | 17:56 | |
| mordred | fungi, corvus: if you have a sec - I ran the letsencrypt playbook against the new review-dev01.opendev.org - I got a key and a csr in the letsencrypt certs dir ... but no cer or ca.cer file (I'm looking at security.openstack.org's certs as an example from an apache config perspective) | 17:56 |
| mordred | did I do something wrong? | 17:56 |
| corvus | mordred: did it run on adns too? | 17:57 |
| *** gfidente is now known as gfidente|afk | 17:58 | |
| corvus | (i've never run letsencrypt with a restricted host list; i've only let the regular cron job run) | 17:58 |
| mordred | corvus: ah! that's probably the reason for sure | 17:59 |
| mordred | I see this in the log: | 17:59 |
| mordred | [Wed Feb 5 17:52:58 UTC 2020] review-dev.opendev.org:Verify error:Incorrect TXT record | 17:59 |
| mordred | corvus: so - yeah - that's probably right on :) | 17:59 |
| *** TheJulia has quit IRC | 18:01 | |
| *** TheJulia has joined #openstack-infra | 18:01 | |
| corvus | mordred: i bet if you added adns to the list it'd work (but make sure you don't step on a cron run) | 18:01 |
| *** rpittau is now known as rpittau|afk | 18:01 | |
| mordred | corvus: yeah - I feel like maybe just landing the patch that adds the inventory/letsencrypt stuff would be better at this point - it's not like landing those will break anything in prod at the moment | 18:02 |
| corvus | ++ | 18:02 |
| openstackgerrit | Carlos Goncalves proposed openstack/project-config master: Update branch checkout and u-c.txt for Octavia https://review.opendev.org/706070 | 18:02 |
| mordred | infra-root: if you have a sec, landing https://review.opendev.org/#/c/705669 and https://review.opendev.org/#/c/704498 would be nice | 18:02 |
| *** johnsom has quit IRC | 18:03 | |
| openstackgerrit | Merged opendev/git-review master: Make it possible to specify who is notified https://review.opendev.org/702861 | 18:03 |
| *** johnsom has joined #openstack-infra | 18:03 | |
| *** jamesmcarthur has joined #openstack-infra | 18:03 | |
| *** artom has joined #openstack-infra | 18:06 | |
| *** soniya29 has quit IRC | 18:07 | |
| *** mattw4 has joined #openstack-infra | 18:09 | |
| *** tesseract has quit IRC | 18:11 | |
| openstackgerrit | Carlos Goncalves proposed openstack/project-config master: Update branch checkout and u-c.txt for Octavia https://review.opendev.org/706070 | 18:18 |
| *** lathiat has quit IRC | 18:22 | |
| *** lathiat has joined #openstack-infra | 18:22 | |
| *** Lucas_Gray has quit IRC | 18:26 | |
| openstackgerrit | Merged opendev/system-config master: Add Apache to Ansible for Gerrit https://review.opendev.org/704498 | 18:33 |
| *** masayukig has quit IRC | 18:34 | |
| *** masayukig has joined #openstack-infra | 18:34 | |
| fungi | #status log halted mirror01.mtl01.inap.opendev.org (which does not appear to be in production) in preparation for deletion | 18:38 |
| openstackstatus | fungi: finished logging | 18:38 |
| *** mattw4 has quit IRC | 18:48 | |
| *** eharney has quit IRC | 18:49 | |
| *** ralonsoh has quit IRC | 18:54 | |
| *** jamesmcarthur has quit IRC | 18:54 | |
| fungi | #status log delisted ipv6 address of mirror-update01.opendev.org from the spamhaus pbl | 18:54 |
| openstackstatus | fungi: finished logging | 18:54 |
| *** jamespage has quit IRC | 18:56 | |
| *** mattw4 has joined #openstack-infra | 18:56 | |
| *** jamespage has joined #openstack-infra | 18:56 | |
| cmurphy | just saw this https://zuul.opendev.org/t/openstack/build/c3f1711ecd9545abbddc00a6d419469c not sure if this is something to be concerned about? | 18:58 |
| *** electrofelix has quit IRC | 18:59 | |
| AJaeger | cmurphy: that is interesting - so, the docs job run in gate but promote job could not find it. Definitely worth investigating... | 19:00 |
| *** gshippey has quit IRC | 19:01 | |
| AJaeger | infra-root, https://zuul.opendev.org/t/openstack/build/6ef0429860004c05a3e40b0150db318e is the gate job - and has no artificats. Any ideas what happened? | 19:01 |
| openstackgerrit | Merged opendev/system-config master: Add review-dev01.opendev.org https://review.opendev.org/705669 | 19:01 |
| fungi | AJaeger: we also collected no logs | 19:03 |
| fungi | interestingly, opendev.org/opendev/base-jobs/playbooks/tox-docs/post.yaml did collect logs | 19:04 |
| fungi | followed by opendev.org/opendev/base-jobs/playbooks/base/post.yaml which was a no-op on log collection | 19:04 |
| *** dpawlik has quit IRC | 19:04 | |
| AJaeger | fungi: json seems to be available - console shows content? That's confusing :( | 19:04 |
| *** dpawlik has joined #openstack-infra | 19:05 | |
| *** mattw4 has quit IRC | 19:06 | |
| fungi | #status log delisted ipv4 address of nb01.opendev.org from the spamhaus pbl | 19:08 |
| openstackstatus | fungi: finished logging | 19:09 |
| *** artom has quit IRC | 19:15 | |
| *** jamesmcarthur has joined #openstack-infra | 19:15 | |
| *** mattw4 has joined #openstack-infra | 19:19 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Extract the watcher from git driver https://review.opendev.org/706092 | 19:21 |
| *** larsks has left #openstack-infra | 19:28 | |
| *** sean-k-mooney has quit IRC | 19:29 | |
| openstackgerrit | Carlos Goncalves proposed openstack/project-config master: Update branch checkout and u-c.txt for Octavia https://review.opendev.org/706070 | 19:30 |
| *** sean-k-mooney has joined #openstack-infra | 19:32 | |
| *** masayukig has quit IRC | 19:38 | |
| *** lathiat has quit IRC | 19:38 | |
| *** vdrok has quit IRC | 19:39 | |
| *** zxiiro has quit IRC | 19:39 | |
| *** jamespage has quit IRC | 19:39 | |
| *** TheJulia has quit IRC | 19:39 | |
| mordred | infra-root: booyah! https://review-dev.opendev.org/#/q/status:open | 19:40 |
| *** johnsom has quit IRC | 19:40 | |
| mordred | I ran the playbook from the tip of the stack | 19:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update letsencrypt_gid documentation https://review.opendev.org/705878 | 19:41 |
| mordred | infra-root: so, https://review.opendev.org/#/c/705878 and parents are ready for review I believe | 19:42 |
| mordred | fwiw - I copied the home dir over from review-dev.openstack.org since we expect to be updating review.o.o in place | 19:44 |
| *** xek_ has joined #openstack-infra | 19:48 | |
| *** pkopec has quit IRC | 19:49 | |
| *** xek has quit IRC | 19:51 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Extract the watcher from git driver https://review.opendev.org/706092 | 20:00 |
| *** eharney has joined #openstack-infra | 20:04 | |
| *** jtomasek has quit IRC | 20:06 | |
| *** artom has joined #openstack-infra | 20:17 | |
| *** dklyle has quit IRC | 20:19 | |
| *** david-lyle has joined #openstack-infra | 20:19 | |
| *** jamesmcarthur has quit IRC | 20:21 | |
| ianw | AJaeger/ fungi : is there a problem with artifacts? i can see another instance of promote job failing | 20:27 |
| ianw | system-config : this job should have made the docs -> https://zuul.opendev.org/t/openstack/build/89dd023531884a3c9cddfccaa2782116/console | 20:27 |
| ianw | this job failed to find it -> https://zuul.opendev.org/t/openstack/build/b483f3e6d07c43508e8d84e40a72b6c8 | 20:28 |
| AJaeger | ianw: see backscroll from cmurphy | 20:29 |
| AJaeger | nobody else mentioned anything, but let's check | 20:29 |
| AJaeger | ianw: http://zuul.opendev.org/t/openstack/builds?pipeline=promote | 20:29 |
| ianw | yeah, looks the same as cmurphy comment | 20:29 |
| AJaeger | That looks bad ;( | 20:29 |
| AJaeger | infra-root, promote pipeline is broken, most recent ones all failed | 20:30 |
| ianw | yeah ... ok first thing was zuul restarted recently? | 20:30 |
| AJaeger | first failure at 15:46 today, and complete failure since 17:02 today | 20:31 |
| AJaeger | ianw: thanks for checking! | 20:31 |
| ianw | nothing in status ... zuul-jobs might be the next suspect | 20:31 |
| ianw | clarkb's loop change is the only one, ... maybe, when did that merge? | 20:32 |
| AJaeger | we merged at 15:46 this one: https://review.opendev.org/705586 | 20:32 |
| AJaeger | corvus, could your stack be related to breaking promote jobs? ^ | 20:33 |
| fungi | https://review.opendev.org/705580 merged at 15:40z and added an option to the generate-zuul-manifest role | 20:33 |
| fungi | that's a more likely cause than 705586 which only touches the upload-logs-gcs role | 20:34 |
| ianw | yeah, that looks like a good one to investigate | 20:34 |
| AJaeger | fungi: agreed, 705580 looks more likely - both in the same stack | 20:35 |
| ianw | if library/generate_manifest.py raises an exception, i don't think we see it in the logs and it might cause the manifest to not be made, but hte job still pass | 20:36 |
| fungi | i think we'll see it on the executor though, right? | 20:36 |
| * fungi checks logs | 20:37 | |
| fungi | build 89dd023531884a3c9cddfccaa2782116 ran on ze05 | 20:39 |
| openstackgerrit | Merged openstack/project-config master: Add release-approval pipeline https://review.opendev.org/705990 | 20:39 |
| ianw | yeah, unfortunately nothing there i can see | 20:40 |
| AJaeger | let me propose reverts for now - while we investigate... | 20:41 |
| fungi | 2020-02-05 18:38:34,185 DEBUG zuul.AnsibleJob: [e: b631f24b90bf427b9d143f5ee1870f4f] [build: 89dd023531884a3c9cddfccaa2782116] Sending result: {"result": "SUCCESS", "warnings": [], "data": {"zuul": {"artifacts": [{"url": "zuul-manifest.json", "metadata": {"type": "zuul_manifest"}, "name": "Zuul Manifest", "index_links": false}, {"url": "docs-html.tar.bz2", "metadata": {"type": "docs_archive"}, | 20:41 |
| fungi | "name": "Docs archive"}, {"url": "docs/", "metadata": {"type": "docs_site"}, "name": "Docs preview site"}], "log_url": "https://c426c48a68cc95805513-73204bd61763b348e9171435eeed8553.ssl.cf1.rackcdn.com/705669/7/gate/opendev-tox-docs/89dd023/"}}} | 20:41 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Revert "Add index_links option to zuul manifest" https://review.opendev.org/706106 | 20:41 |
| fungi | that seems to have successfully set index_links: false | 20:42 |
| fungi | so i'm not seeing any evidence of a problem therwe | 20:42 |
| fungi | indeed, https://c426c48a68cc95805513-73204bd61763b348e9171435eeed8553.ssl.cf1.rackcdn.com/705669/7/gate/opendev-tox-docs/89dd023/ has the logs | 20:42 |
| fungi | maybe adding index_links broke the dashboard somehow? | 20:42 |
| fungi | or something broke the mysql reporter? | 20:43 |
| *** slaweq_ has quit IRC | 20:44 | |
| *** dkehn has joined #openstack-infra | 20:45 | |
| ianw | https://c426c48a68cc95805513-73204bd61763b348e9171435eeed8553.ssl.cf1.rackcdn.com/705669/7/gate/opendev-tox-docs/89dd023/zuul-manifest.json looks all ok | 20:45 |
| ianw | i wonder if adding the "index_links" as a separate variable has broken some sort of json parser | 20:45 |
| ianw | not variable, key in the output dict i mean | 20:46 |
| fungi | yeah, i think either 1. the manifest isn't being recorded in the db, or 2. it's not able to be displayed by the dashboard | 20:46 |
| fungi | still digging | 20:46 |
| AJaeger | fungi: keep in mind that hte promote jobs is not finding the artifact | 20:46 |
| ianw | i don't see any errors in browser console | 20:47 |
| AJaeger | So, displaying in the dashboard is not he only problem | 20:47 |
| *** slaweq_ has joined #openstack-infra | 20:47 | |
| AJaeger | "Could not find or access '/var/lib/zuul/builds/c3f1711ecd9545abbddc00a6d419469c/work/docs-html.tar.bz2' on the Ansible Controller." | 20:47 |
| fungi | good point AJaeger, so artifacts/manifest are missing from the build metadata i guess | 20:47 |
| AJaeger | fungi, yes, that could explain the missing display in dashboard as well | 20:48 |
| fungi | i'm looking in the db now | 20:50 |
| ianw | 2020-02-05 19:01:44,618 DEBUG zuul.SQLReporter: Result data did not pass artifact schema validation: | 20:52 |
| AJaeger | ianw: great | 20:52 |
| AJaeger | We have {"index_links": false, "tree": | 20:52 |
| ianw | hang on, that might be a red herring | 20:52 |
| AJaeger | vs {"tree": [{ | 20:52 |
| AJaeger | in zuul-manifest.json at the beginning | 20:53 |
| ianw | yeah, i think it has something to do with this | 20:53 |
| fungi | okay, so the build with uuid 89dd023531884a3c9cddfccaa2782116 has id 9217179 | 20:54 |
| fungi | and the zuul_artifact table has 0 records where build_id=9217179 | 20:54 |
| AJaeger | ianw, fungi, I pushed up the revert for review in case you want to merge it. I'm signing off now, hope you find the culprit! | 20:55 |
| ianw | i think this is it -> https://opendev.org/zuul/zuul/src/branch/master/zuul/lib/artifacts.py#L24 | 20:55 |
| AJaeger | Thanks for digging further | 20:55 |
| fungi | compared to working build 3235693177904546aaaa6fca586d75be whose id is 9203227 and has several corresponding records in the zuul_artifact table | 20:55 |
| fungi | so we're failing to record the manifest records in the zuul_artifact table | 20:56 |
| AJaeger | ianw: lines 18-21 I guess | 20:56 |
| mordred | ianw: I think v.Extra should let it have extra fields, right? | 20:56 |
| mordred | oh - 18-21 | 20:56 |
| AJaeger | ianw: we change from name, url, metadata to name, url, index_links, metadata | 20:56 |
| AJaeger | so, that clearly fails the check | 20:57 |
| ianw | yeah -- i think we have to revert the original, then i think zuul will need a restart to handle it before adding it again | 20:57 |
| AJaeger | So, revert - update zuul, restart, revert ervert? | 20:57 |
| mordred | yeah - so - I think we should land the revert so that we can re-revert once there is a fix | 20:57 |
| mordred | AJaeger: yah | 20:57 |
| * AJaeger waves good night... | 20:58 | |
| fungi | thanks AJaeger! | 20:59 |
| fungi | mordred: ianw: can you summarize your suspicions in a comment on 706106? | 20:59 |
| ianw | fungi: i'll propose the change in zuul, that probably gets attention | 20:59 |
| fungi | wfm | 21:00 |
| fungi | thanks!!! | 21:00 |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Handle index_links in manifest https://review.opendev.org/706115 | 21:03 |
| corvus | stop please | 21:04 |
| corvus | i see the error and it's a one-liner | 21:05 |
| corvus | give me just a sec to propose a fix | 21:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Move index_links to correct task in generate_zuul_manifest https://review.opendev.org/706116 | 21:07 |
| corvus | ianw, fungi, AJaeger, mordred: ^ | 21:07 |
| corvus | this was never supposed to touch the artifact data | 21:07 |
| ianw | heh, and so that gets in there as a bool, and everything still passes ... i see :) | 21:09 |
| corvus | ianw: i don't follow that | 21:10 |
| ianw | i mean the new value gets into the manifest a boolean, which does trigger the artifact schema failing, rather than raising an error or exception, just a bug that it nicely hides itself | 21:11 |
| corvus | the extra field in the manifest is fine; the artifact storage doesn't care what the content of the artifact is. | 21:13 |
| corvus | the artifact storage only cared that we were giving it extra garbage | 21:14 |
| *** slaweq_ has quit IRC | 21:15 | |
| corvus | i -2d the other changes to prevent merging; i think we should abandon them. sorry for the bug, and thanks for the zuul-jobs revert (which would have stopped the errors as well) | 21:15 |
| fungi | thanks for the quick fix! reviewing now | 21:16 |
| ianw | corvus: if you have a sec would you mind a +2 on https://review.opendev.org/#/c/705873/1 -- that's the re-encoding of the key we discussed yesterday | 21:17 |
| corvus | ianw: those characters look correct to me! :) | 21:17 |
| ianw | yes, more just the theory of it :) | 21:18 |
| ianw | after that's tested, i think we're right to switch the tarball publishing jobs and will propose final changes | 21:18 |
| *** tosky has quit IRC | 21:18 | |
| *** smarcet has left #openstack-infra | 21:19 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Extract the watcher from git driver https://review.opendev.org/706092 | 21:21 |
| ianw | infra-root: actually one more that adds the tarballs sites to ansible (won't change anything till dns updated) | 21:22 |
| ianw | https://review.opendev.org/#/c/704913/ | 21:22 |
| ianw | oh, hang on, that's gone into merge conflict, i guess the apache stuff, just a minute | 21:23 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add tarballs.<openstack|opendev>.org to static.opendev.org https://review.opendev.org/704913 | 21:24 |
| *** xek_ has quit IRC | 21:27 | |
| *** rfolco has quit IRC | 21:27 | |
| openstackgerrit | Merged openstack/project-config master: Update service/opendev-zuul key https://review.opendev.org/705873 | 21:29 |
| openstackgerrit | Merged zuul/zuul-jobs master: Move index_links to correct task in generate_zuul_manifest https://review.opendev.org/706116 | 21:31 |
| *** rfolco has joined #openstack-infra | 21:31 | |
| *** hwoarang has quit IRC | 21:35 | |
| *** rfolco has quit IRC | 21:35 | |
| *** hwoarang has joined #openstack-infra | 21:37 | |
| *** dciabrin has joined #openstack-infra | 21:40 | |
| *** liuyulong has quit IRC | 21:41 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: add_host: enable using ansible_fqdn and ansible_private_key_file https://review.opendev.org/706124 | 21:52 |
| corvus | tristanC: if you use ssh-add you can avoid leaving the key sitting around on the executor during the job. eg https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/add-fileserver/tasks/main.yaml#L12-L16 | 22:00 |
| corvus | tristanC: anyway, i can't think of a reason not to add that, but let's ask mordred and tobiash to review | 22:02 |
| mordred | corvus: should we wait for an ssh-add based respin? | 22:02 |
| mordred | oh - wait | 22:02 |
| mordred | yeah - I can't see any reason not to add that patch | 22:03 |
| corvus | the ssh-add was more of an alternative to consider | 22:03 |
| mordred | yeah | 22:03 |
| * mordred groks now | 22:03 | |
| corvus | i can't remember if we disallowed those args intentionally, or just didn't think to add them because we used ssh-add originally | 22:04 |
| mordred | infra-root: my internet went out for no good reason (there is an alarm light on the fiber modem - so I'm guessing that has something to do with it) - so I'm phone-tethered for the new little bit until it can get resolved | 22:04 |
| mordred | corvus: I *think* it's the latter | 22:04 |
| mordred | (yes, I have tried turning it off and turning it back on again) | 22:05 |
| fungi | mordred: have you considered relocating it above sea level? | 22:11 |
| fungi | could be barnacles eating the packets | 22:11 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Gerrit: add polling support for refs https://review.opendev.org/706138 | 22:29 |
| *** yolanda has joined #openstack-infra | 22:35 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Temporarly disable Fort Nebula general purpose nodes https://review.opendev.org/706142 | 22:41 |
| donnyd | Oh wow - I did such a bad job spelling temporarily | 22:41 |
| donnyd | where is the shaky head emoticon | 22:41 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Extract allow/disallow filter into util function https://review.opendev.org/706144 | 22:44 |
| *** rh-jelabarre has quit IRC | 22:46 | |
| *** tkajinam has joined #openstack-infra | 22:48 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add CentOS8 High Availability repository https://review.opendev.org/705973 | 22:50 |
| *** vdrok has joined #openstack-infra | 23:00 | |
| *** TheJulia has joined #openstack-infra | 23:00 | |
| *** lathiat has joined #openstack-infra | 23:00 | |
| *** masayukig has joined #openstack-infra | 23:00 | |
| *** jamespage has joined #openstack-infra | 23:01 | |
| *** johnsom has joined #openstack-infra | 23:02 | |
| *** zxiiro has joined #openstack-infra | 23:02 | |
| *** jamesmcarthur has joined #openstack-infra | 23:03 | |
| *** jamesmcarthur has quit IRC | 23:06 | |
| donnyd | ianw: FN needs to have some major refactoring done, so I am pretty sure this will be the first of a couple of changes necessary to support said changes | 23:12 |
| ianw | donnyd: ok, np | 23:12 |
| *** rlandy is now known as rlandy|bbl | 23:25 | |
| *** sean-k-mooney has quit IRC | 23:28 | |
| *** sean-k-mooney has joined #openstack-infra | 23:29 | |
| *** ociuhandu has joined #openstack-infra | 23:30 | |
| *** ociuhandu has quit IRC | 23:35 | |
| *** dciabrin has quit IRC | 23:36 | |
| *** dSrinivas has quit IRC | 23:43 | |
| *** hwoarang has quit IRC | 23:51 | |
| *** hwoarang has joined #openstack-infra | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!