*** wolverineav has quit IRC | 00:04 | |
*** wolverineav has joined #openstack-infra | 00:05 | |
*** slaweq_ has joined #openstack-infra | 00:11 | |
*** wolverineav has quit IRC | 00:14 | |
*** wolverineav has joined #openstack-infra | 00:22 | |
*** slaweq_ has quit IRC | 00:24 | |
*** dave-mccowan has joined #openstack-infra | 00:29 | |
*** dave-mccowan has quit IRC | 00:31 | |
*** wolverineav has quit IRC | 00:40 | |
*** wolverineav has joined #openstack-infra | 00:40 | |
*** wolverineav has quit IRC | 00:52 | |
*** jamesmcarthur has joined #openstack-infra | 00:58 | |
*** slaweq_ has joined #openstack-infra | 01:11 | |
*** wolverineav has joined #openstack-infra | 01:13 | |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: master-nameserver: Add unmanaged domains; add acme.opendev.org https://review.openstack.org/641155 | 01:16 |
---|---|---|
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 01:16 |
*** wolverineav has quit IRC | 01:20 | |
*** wolverineav has joined #openstack-infra | 01:21 | |
*** slaweq_ has quit IRC | 01:24 | |
*** wolverineav has quit IRC | 01:26 | |
*** jamesmcarthur has quit IRC | 01:45 | |
*** jamesmcarthur has joined #openstack-infra | 01:55 | |
*** wolverineav has joined #openstack-infra | 02:04 | |
*** jamesmcarthur has quit IRC | 02:06 | |
*** jamesmcarthur_ has joined #openstack-infra | 02:06 | |
*** slaweq_ has joined #openstack-infra | 02:11 | |
*** wolverineav has quit IRC | 02:15 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Get executor job params https://review.openstack.org/607078 | 02:22 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Separate out executor server from runner https://review.openstack.org/607079 | 02:22 |
*** irclogbot_1 has quit IRC | 02:23 | |
*** slaweq_ has quit IRC | 02:24 | |
*** edmondsw has quit IRC | 02:26 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 02:30 |
*** ricolin has joined #openstack-infra | 02:35 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 02:40 |
*** ykarel has joined #openstack-infra | 02:52 | |
*** jamesmcarthur_ has quit IRC | 02:59 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 03:09 |
*** slaweq_ has joined #openstack-infra | 03:11 | |
*** hongbin has joined #openstack-infra | 03:12 | |
*** hongbin has quit IRC | 03:21 | |
*** slaweq_ has quit IRC | 03:25 | |
*** wolverineav has joined #openstack-infra | 03:29 | |
*** wolverineav has quit IRC | 03:33 | |
*** whoami-rajat has joined #openstack-infra | 03:46 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 03:48 |
*** bgmccollum has quit IRC | 03:56 | |
*** bgmccollum has joined #openstack-infra | 04:00 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 04:01 |
*** janki has joined #openstack-infra | 04:05 | |
*** udesale has joined #openstack-infra | 04:08 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: config: use yaml.safe_load instead of load https://review.openstack.org/643849 | 04:14 |
*** slaweq_ has joined #openstack-infra | 04:14 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 04:18 |
*** Lucas_Gray has joined #openstack-infra | 04:20 | |
*** Lucas_Gray has quit IRC | 04:21 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 04:24 |
*** slaweq_ has quit IRC | 04:24 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 04:35 |
*** ramishra has joined #openstack-infra | 04:46 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 04:48 |
*** rodrigods has quit IRC | 05:01 | |
*** slaweq_ has joined #openstack-infra | 05:16 | |
*** slaweq_ has quit IRC | 05:24 | |
*** pcrews has joined #openstack-infra | 05:28 | |
*** bgmccollum has quit IRC | 05:28 | |
*** bgmccollum has joined #openstack-infra | 05:31 | |
*** ykarel_ has joined #openstack-infra | 05:43 | |
*** rcernin has quit IRC | 05:43 | |
*** ykarel has quit IRC | 05:45 | |
*** jaosorior has joined #openstack-infra | 05:45 | |
*** ykarel_ is now known as ykarel | 05:51 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: rewrite refreshable and filters with higher order component https://review.openstack.org/643856 | 05:56 |
*** rcernin has joined #openstack-infra | 05:59 | |
*** jbadiapa has joined #openstack-infra | 06:06 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 06:06 |
*** slaweq_ has joined #openstack-infra | 06:11 | |
*** slaweq_ has quit IRC | 06:24 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 06:25 |
*** e0ne has joined #openstack-infra | 06:40 | |
*** kjackal has joined #openstack-infra | 06:42 | |
*** wolverineav has joined #openstack-infra | 06:45 | |
*** wolverineav has quit IRC | 06:50 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: rewrite refreshable and filters with higher order component https://review.openstack.org/643856 | 06:54 |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 06:56 |
*** jaosorior has quit IRC | 06:58 | |
*** e0ne has quit IRC | 07:02 | |
*** AJaeger has quit IRC | 07:03 | |
*** kopecmartin|off is now known as kopecmartin | 07:03 | |
*** AJaeger has joined #openstack-infra | 07:07 | |
*** tkajinam_ has joined #openstack-infra | 07:11 | |
*** jaosorior has joined #openstack-infra | 07:11 | |
*** tkajinam has quit IRC | 07:13 | |
*** jtomasek has joined #openstack-infra | 07:13 | |
*** slaweq_ has joined #openstack-infra | 07:13 | |
*** gfidente has joined #openstack-infra | 07:15 | |
openstackgerrit | Merged openstack-infra/zuul master: Manage ansible installations within zuul https://review.openstack.org/631930 | 07:16 |
openstackgerrit | Merged openstack-infra/zuul master: Validate ansible installations on startup https://review.openstack.org/637418 | 07:16 |
openstackgerrit | Merged openstack-infra/zuul master: Make ansible version configurable https://review.openstack.org/637422 | 07:16 |
openstackgerrit | Merged openstack-infra/zuul master: Install ansible during executor startup if needed https://review.openstack.org/640644 | 07:16 |
openstackgerrit | Merged openstack-infra/zuul master: Run tox remote concurrent https://review.openstack.org/640654 | 07:16 |
openstackgerrit | Merged openstack-infra/zuul master: Support ansible 2.6 https://review.openstack.org/631931 | 07:16 |
openstackgerrit | Merged openstack-infra/zuul master: Support ansible 2.7 https://review.openstack.org/631932 | 07:16 |
openstackgerrit | Merged openstack-infra/zuul master: Switch default ansible version to 2.7 https://review.openstack.org/637424 | 07:16 |
*** pcaruana has joined #openstack-infra | 07:19 | |
*** kjackal has quit IRC | 07:23 | |
*** dpawlik has joined #openstack-infra | 07:23 | |
*** pgaxatte has joined #openstack-infra | 07:27 | |
*** jaosorior has quit IRC | 07:35 | |
*** kjackal has joined #openstack-infra | 07:37 | |
*** apetrich has joined #openstack-infra | 07:38 | |
*** jaosorior has joined #openstack-infra | 07:44 | |
*** jaosorior has quit IRC | 07:44 | |
*** xek has joined #openstack-infra | 07:45 | |
*** iurygregory has joined #openstack-infra | 07:52 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: manifest: add ansible-config.conf https://review.openstack.org/643872 | 07:56 |
*** yamamoto has quit IRC | 08:04 | |
*** tosky has joined #openstack-infra | 08:05 | |
*** yamamoto has joined #openstack-infra | 08:05 | |
*** yamamoto has quit IRC | 08:06 | |
*** tosky has quit IRC | 08:07 | |
*** ginopc has joined #openstack-infra | 08:08 | |
*** kjackal has quit IRC | 08:08 | |
*** kjackal has joined #openstack-infra | 08:11 | |
*** zhangfei has joined #openstack-infra | 08:14 | |
*** ccamacho has joined #openstack-infra | 08:19 | |
*** rpittau|afk is now known as rpittau | 08:19 | |
openstackgerrit | Merged openstack-infra/zuul master: Support common ansible dependencies https://review.openstack.org/643534 | 08:20 |
*** ykarel is now known as ykarel|lunch | 08:21 | |
openstackgerrit | Merged openstack-infra/zuul master: Add missing docs for zuul-manage-ansible https://review.openstack.org/643544 | 08:22 |
openstackgerrit | Merged openstack-infra/zuul master: ansible-config: add common extra dependencies https://review.openstack.org/643537 | 08:23 |
*** tosky has joined #openstack-infra | 08:24 | |
*** tkajinam_ has quit IRC | 08:24 | |
*** priteau has joined #openstack-infra | 08:26 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: manifest: add zuul/ansible and ansible-config.conf https://review.openstack.org/643872 | 08:27 |
*** jpena|off is now known as jpena | 08:29 | |
*** lennyb has joined #openstack-infra | 08:30 | |
*** ramishra has quit IRC | 08:35 | |
*** ramishra has joined #openstack-infra | 08:35 | |
*** sshnaidm|afk is now known as sshnaidm|rover | 08:42 | |
*** kjackal_v2 has joined #openstack-infra | 08:43 | |
*** kjackal has quit IRC | 08:44 | |
openstackgerrit | Merged openstack-infra/zuul master: Install virtualenv from source https://review.openstack.org/643655 | 08:55 |
*** rossella_s has quit IRC | 09:04 | |
*** rossella_s has joined #openstack-infra | 09:04 | |
*** dtantsur|afk is now known as dtantsur | 09:11 | |
openstackgerrit | renminmin proposed openstack/diskimage-builder master: Add DIB_LOCAL_IMAGE to use ubuntu local cloud image https://review.openstack.org/643885 | 09:11 |
*** rascasoft has joined #openstack-infra | 09:12 | |
*** zhangfei has quit IRC | 09:15 | |
*** kjackal has joined #openstack-infra | 09:18 | |
*** kjackal_v2 has quit IRC | 09:19 | |
*** zhangfei has joined #openstack-infra | 09:27 | |
*** ykarel|lunch is now known as ykarel | 09:27 | |
*** rfolco has joined #openstack-infra | 09:34 | |
*** roman_g has joined #openstack-infra | 09:34 | |
*** kjackal has quit IRC | 09:50 | |
*** kjackal has joined #openstack-infra | 09:51 | |
*** wolverineav has joined #openstack-infra | 09:54 | |
*** wolverineav has quit IRC | 09:58 | |
*** Lucas_Gray has joined #openstack-infra | 10:06 | |
*** kjackal has quit IRC | 10:21 | |
*** yamamoto has joined #openstack-infra | 10:22 | |
*** kjackal has joined #openstack-infra | 10:22 | |
*** yamamoto has quit IRC | 10:22 | |
dtantsur | hi folks! anything work with review.o.o? takes ages to connect, gertty goes to offline mode | 10:27 |
*** udesale has quit IRC | 10:28 | |
*** e0ne has joined #openstack-infra | 10:28 | |
*** yamamoto has joined #openstack-infra | 10:28 | |
*** Lucas_Gray has quit IRC | 10:32 | |
*** Lucas_Gray has joined #openstack-infra | 10:33 | |
frickler | dtantsur: looks normal from my pov. you might possibly want to check for issues in your ipv6 connectivity | 10:38 |
dtantsur | frickler: it's back now (was not only me before), probably a temporary glitch | 10:39 |
rpittau | frickler, had also some issues during the morning, looks good now since 10 minutes | 10:40 |
frickler | hmm, o.k., I had no issues earlier either. but if things are back to working now, then let's hope it stays that way | 10:42 |
*** jbadiapa has quit IRC | 10:44 | |
*** kjackal has quit IRC | 10:55 | |
*** kjackal has joined #openstack-infra | 10:56 | |
*** zhangfei has quit IRC | 11:04 | |
*** kjackal has quit IRC | 11:05 | |
*** slaweq_ is now known as slaweq | 11:14 | |
*** kjackal has joined #openstack-infra | 11:15 | |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: master-nameserver: Add unmanaged domains; add acme.opendev.org https://review.openstack.org/641155 | 11:17 |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 11:17 |
*** sajauddin has joined #openstack-infra | 11:36 | |
*** kaiokmo has joined #openstack-infra | 11:39 | |
*** kaiokmo has quit IRC | 11:39 | |
*** kaiokmo has joined #openstack-infra | 11:40 | |
*** udesale has joined #openstack-infra | 11:45 | |
*** rcernin has quit IRC | 11:46 | |
*** edmondsw has joined #openstack-infra | 11:50 | |
*** jaosorior has joined #openstack-infra | 11:51 | |
*** rh-jelabarre has joined #openstack-infra | 11:57 | |
*** rlandy has joined #openstack-infra | 12:01 | |
*** udesale has quit IRC | 12:01 | |
*** udesale has joined #openstack-infra | 12:02 | |
*** trown|outtypewww is now known as trown | 12:09 | |
*** sajauddin has quit IRC | 12:13 | |
*** markvoelker has quit IRC | 12:22 | |
*** udesale has quit IRC | 12:27 | |
*** udesale has joined #openstack-infra | 12:28 | |
*** yamamoto has quit IRC | 12:30 | |
*** whoami-rajat has quit IRC | 12:35 | |
*** eharney has joined #openstack-infra | 12:39 | |
*** kgiusti has joined #openstack-infra | 12:42 | |
*** Lucas_Gray has quit IRC | 12:50 | |
*** e0ne has quit IRC | 12:51 | |
*** Lucas_Gray has joined #openstack-infra | 12:51 | |
*** zul has joined #openstack-infra | 12:51 | |
*** e0ne has joined #openstack-infra | 12:52 | |
*** Lucas_Gray has quit IRC | 12:53 | |
*** Lucas_Gray has joined #openstack-infra | 12:53 | |
*** pcaruana|afk| has joined #openstack-infra | 12:54 | |
*** dpawlik has quit IRC | 12:55 | |
*** yamamoto has joined #openstack-infra | 12:55 | |
*** eharney has quit IRC | 12:56 | |
*** pcaruana has quit IRC | 12:56 | |
*** jpena is now known as jpena|lunch | 12:57 | |
*** mriedem has joined #openstack-infra | 12:58 | |
openstackgerrit | Merged openstack-infra/zuul master: manifest: add zuul/ansible and ansible-config.conf https://review.openstack.org/643872 | 13:02 |
*** jtomasek_ has joined #openstack-infra | 13:02 | |
*** jtomasek has quit IRC | 13:03 | |
*** jtomasek_ has quit IRC | 13:05 | |
*** jtomasek has joined #openstack-infra | 13:06 | |
*** whoami-rajat has joined #openstack-infra | 13:08 | |
*** dave-mccowan has joined #openstack-infra | 13:09 | |
*** dave-mccowan has quit IRC | 13:15 | |
*** needssleep is now known as TheJulia | 13:18 | |
*** dpawlik has joined #openstack-infra | 13:24 | |
*** lseki has joined #openstack-infra | 13:27 | |
*** hwoarang has quit IRC | 13:37 | |
*** hwoarang has joined #openstack-infra | 13:38 | |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Whitelist ansible_python_interpreter for add_host https://review.openstack.org/643726 | 13:48 |
*** pcaruana|afk| has quit IRC | 13:48 | |
*** kjackal has quit IRC | 13:55 | |
*** kjackal_v2 has joined #openstack-infra | 13:55 | |
*** e0ne has quit IRC | 13:58 | |
*** Qiming has quit IRC | 13:59 | |
*** Qiming has joined #openstack-infra | 14:00 | |
*** jpena|lunch is now known as jpena | 14:04 | |
*** eharney has joined #openstack-infra | 14:04 | |
*** e0ne has joined #openstack-infra | 14:05 | |
openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add acme.opendev.org https://review.openstack.org/641154 | 14:07 |
*** verdurin has joined #openstack-infra | 14:12 | |
openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Set fingergw user to zuul https://review.openstack.org/643954 | 14:14 |
*** Lucas_Gray has quit IRC | 14:22 | |
*** Lucas_Gray has joined #openstack-infra | 14:23 | |
openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Update executor ansible on installation https://review.openstack.org/643957 | 14:24 |
openstackgerrit | Monty Taylor proposed openstack-infra/git-review master: Use remote_url instead of remote for download https://review.openstack.org/643966 | 14:31 |
mordred | zbr, corvus, fungi: ^^ | 14:32 |
mordred | I discovered that usepushurl does not work with opendev.org - and then I was sad because I use usepushurl | 14:32 |
mordred | luckily there is a four character ix | 14:32 |
mordred | fix | 14:32 |
*** pcaruana has joined #openstack-infra | 14:33 | |
*** yamamoto has quit IRC | 14:34 | |
zbr | mordred: nice, testing it now. | 14:35 |
mordred | zbr: it works for me locally - I haven't run the test suite, so I'm not sure yet if it breaks something there | 14:35 |
*** yamamoto has joined #openstack-infra | 14:37 | |
openstackgerrit | Monty Taylor proposed openstack-infra/git-review master: Use remote_url instead of remote for download https://review.openstack.org/643966 | 14:37 |
mordred | zbr: ^^ added releasenote | 14:37 |
*** armax has joined #openstack-infra | 14:39 | |
openstackgerrit | Sorin Sbarnea proposed openstack-dev/pbr master: DNM: add boo https://review.openstack.org/643991 | 14:39 |
openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Use one database per test class rather than per test https://review.openstack.org/644000 | 14:40 |
zbr | mordred: i suppose that the move to opendev would trigger a set of fixes in tools like git-review, gertty,.... | 14:40 |
frickler | ianw: hmm, nice error, I think ansible should also be a bit more intelligent trying to expand this: http://logs.openstack.org/55/641155/14/check/system-config-run-base/a75e854/job-output.txt.gz#_2019-03-18_11_29_05_814059 | 14:41 |
openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/system-config master: master-nameserver: Add unmanaged domains; add acme.opendev.org https://review.openstack.org/641155 | 14:41 |
*** e0ne has quit IRC | 14:41 | |
*** e0ne has joined #openstack-infra | 14:42 | |
*** yamamoto has quit IRC | 14:42 | |
fungi | zbr: yes, i expect changing the domain name we've been using for these services for upwards of 8 years will shake out some unexpected bugs, though hopefully they're all fairly minor | 14:44 |
fungi | (not to mention we're trying to keep things backward-compatible as much as possible to keep the transition mostly seamless) | 14:45 |
clarkb | people do you these tools against other domains though. Hopefully not many corner cases as a result | 14:46 |
clarkb | *do use these toole | 14:46 |
clarkb | bah typing is hard | 14:46 |
fungi | right, i expect the issues to be corner cases in infrequently-used features, like the one mordred just identified | 14:47 |
roman_g | Wow, wow, what I've found: I'm not able to remove a reviewer, if he has left negative review for a previous patch set. I just didn't want to spam reviewer's e-mail while I continue to work on the PS. | 14:48 |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add --check-config option to zuul scheduler https://review.openstack.org/542160 | 14:49 |
roman_g | But I think I can remove reviewer who has left positive vote. | 14:49 |
fungi | roman_g: yes, that's correct | 14:50 |
roman_g | *thumbs_up* | 14:50 |
fungi | roman_g: see https://review.openstack.org/Documentation/access-control.html#category_remove_reviewer for documentation of that behavior | 14:50 |
roman_g | Thanks :) | 14:51 |
*** e0ne has quit IRC | 14:53 | |
mordred | zbr, fungi, clarkb: yah- although at some point we also need to file a bug with gitea so we can fix our ability to push refs/changes/* to it | 14:53 |
*** woojay has joined #openstack-infra | 14:54 | |
*** e0ne has joined #openstack-infra | 14:55 | |
fungi | mordred: as in gitea is allowing that when we want it rejected? | 14:56 |
*** josephrsandoval has joined #openstack-infra | 15:02 | |
*** jamesmcarthur has joined #openstack-infra | 15:02 | |
*** e0ne has quit IRC | 15:05 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Tolerate missing project https://review.openstack.org/579872 | 15:10 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: buildset registry: don't put skopeo creds on command line https://review.openstack.org/644241 | 15:11 |
*** armax has quit IRC | 15:18 | |
mordred | fungi: no - the other way around - we aren't doing it because gitea gets confused and unhappy if we do | 15:19 |
*** e0ne has joined #openstack-infra | 15:20 | |
*** josephrsandoval has quit IRC | 15:23 | |
*** janki has quit IRC | 15:26 | |
clarkb | mordred: corvus: so the thing I was trying to debug late friday is the docker image job collection doesn't seem to be 100%reliable and skopeo is the current suspect actor | 15:27 |
clarkb | in the post image build sync between the buildset registry and the intermediate registry I/we were seeing skopeo go out to lunch until the job timeout hit | 15:27 |
clarkb | looking at the registry logs on both sides it seemed that the data was transfered and skopeo was just failing to exit properly (the logs showed a buch of happy blob transfers and there were no open tcp connections between nodes at the time) | 15:28 |
mordred | clarkb: AWESOME | 15:29 |
clarkb | one thing I noticed is that there is a newer version of skopeo available on the atomic ppa but we don't auto update to it with our unattended upgrades policy | 15:29 |
clarkb | I figured maybe step 0 in further debugging is to upgrade that? | 15:29 |
clarkb | fwiw this is why we haven't updated our gitea images with the logos or newer gitea yet | 15:30 |
clarkb | also for some reason the gitea version bump didn't run image build jobs in check but did run them in gate (oh ya this failure isn't 100%) | 15:30 |
corvus | clarkb: i saw your debugging in backlog thanks... i also think may we should do this: https://review.openstack.org/644241 (cc mordred) | 15:32 |
openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Refactor code for unit testing https://review.openstack.org/643666 | 15:33 |
openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests https://review.openstack.org/644247 | 15:33 |
clarkb | ++ that should help with debugging | 15:33 |
*** yamamoto has joined #openstack-infra | 15:34 | |
*** dpawlik has quit IRC | 15:35 | |
openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Update executor ansible on installation https://review.openstack.org/643957 | 15:35 |
corvus | config-core: sometime today we're going to upgrade to a version of zuul that supports multiple versions of ansible. should we set the global default to 2.5 (the current version) before we do that, or should we "upgrade" all our jobs to run on ansible 2.7 at the same time? | 15:36 |
clarkb | corvus: can you check the comments on https://review.openstack.org/#/c/644241/1 ? | 15:37 |
mordred | corvus: I thnik I'm ok with upgrading - since we can always set the value if things break - and we can set the value on targetted jobs if they do | 15:37 |
clarkb | The openstack release process is starting to pick up. I think as long as we are planning to be around to debug jobs with them and/or set the ansible verison for them as necessary bumping should be fine | 15:38 |
openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Use one database per test class rather than per test https://review.openstack.org/644000 | 15:38 |
frickler | corvus: system-config-run-base-ansible-devel seems to be hard failing, so I'm not sure how good we'd be at running other stuff with 2.7, so I'd vote to stick to 2.5 until we get that fixed | 15:40 |
clarkb | frickler: I think the devel job failures are related to the testinfra api incompat with running ansible directly | 15:40 |
clarkb | frickler: I expect that is mostly non representative since that is also for ansible 2.8 development | 15:40 |
corvus | well, if we wanted to be more cautious, we can set the default to 2.5 and then set "unittests" to use 2.7 or something... | 15:41 |
*** ykarel is now known as ykarel|away | 15:41 | |
*** yamamoto has quit IRC | 15:42 | |
clarkb | fungi: do you know if anyone has debugged the mis applied retired project acls in gerrit? | 15:42 |
corvus | but if we want to roll the dice, then we could do so, and if things go wrong, we can just force-merge a change to 'base' which sets it to 2.5 | 15:42 |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Properly handle missing ansible version in executor https://review.openstack.org/644252 | 15:43 |
corvus | clarkb: see reply on https://review.openstack.org/644241 | 15:43 |
clarkb | corvus: thanks /me drinks more tea | 15:44 |
fungi | clarkb: i do not know, no | 15:44 |
clarkb | fungi: k, its getting near the top of my list so likely to look at it today if others haven't | 15:45 |
fungi | i'm currently juggling a bunch of things, so happy for you to grab it if you can | 15:45 |
clarkb | corvus: mordred: re skopeo I think we probably want to rerun with the no_logs removed then can decide if we want to upgrade skopeo when we have that info | 15:49 |
corvus | clarkb: or we could go ahead and upgrade skopeo too, then if it happens again, we're 1 step ahead in debugging.... i think we can change 2 variables this time and be okay | 15:53 |
mordred | clarkb: yah. although I'm also pretty sure we want to upgrade skopeo | 15:53 |
mordred | what corvus said | 15:53 |
clarkb | mordred: corvus: ok `apt-get update && apt-get install -y skopeo` look right? | 15:54 |
mordred | clarkb: ++ | 15:54 |
corvus | ++ | 15:54 |
clarkb | I'll ask ansible to run that on the executors momentarily | 15:54 |
mordred | clarkb: you could also ask it to do -m pkg -a name=skopeo,update=yes if you wanted to get fancy | 15:55 |
clarkb | I used ze01 as my test node and it also installed containers-common containers-golang | 15:56 |
clarkb | which is a depends on skopeo | 15:57 |
corvus | interesting... looks like that's some seccomp rules | 15:57 |
corvus | hopefully it doesn't have a bad interaction with bwrap | 15:58 |
corvus | er "that" == containers-golang | 15:58 |
corvus | containers-common is also interesting | 15:59 |
corvus | http://paste.openstack.org/show/747976/ | 15:59 |
mordred | if there is an adverse reaction, I bet we could go talk to our friends over in podman land | 15:59 |
corvus | that seems to be a lot of configuration.... | 16:00 |
clarkb | also odd considering skopeo is an http transfer tool basically | 16:00 |
corvus | i... i kinda liked the fact that skopeo was just a stand-alone tool | 16:00 |
clarkb | (it doesn't run containers)( | 16:00 |
corvus | i'm not thrilled that it's starting to grow tentacles into system configuration | 16:00 |
corvus | clarkb: i'm regretting my suggestion that we upgrade skopeo | 16:01 |
*** rpioso is now known as rpioso|afk | 16:01 | |
clarkb | so far I've only done it on ze01 | 16:01 |
corvus | clarkb, mordred: the thing i'm now worried about is whether the new version will cause new problems | 16:02 |
*** weshay|rover is now known as weshay | 16:02 | |
*** josephrsandoval has joined #openstack-infra | 16:02 | |
clarkb | but also if we build new executors tehy would get newer skopeo anyway | 16:02 |
clarkb | (so this may be a case where dealing with new problems is better problems) | 16:02 |
corvus | like, are we going to need to upgrade registries.conf or something to make it work? | 16:02 |
corvus | clarkb: i agree, i just wasn't expecting such wide-ranging changes | 16:03 |
corvus | and i'm not sure if we want to deal with them at the same time as our other mysterious problem | 16:03 |
clarkb | ++ | 16:03 |
*** dpawlik has joined #openstack-infra | 16:04 | |
mordred | corvus: k. new registries.conf is basically file with default | 16:04 |
corvus | i mean, i like what skopeo is doing -- it lets you opt-out of the dockerhub-centric registry system | 16:04 |
corvus | mordred: you're thinking that the new files are configuring it to behave in approximately the same way it was? | 16:04 |
mordred | yeah | 16:04 |
corvus | okay. let's yolo and upgrade them and deal with everything at once :) | 16:05 |
mordred | the main change seems to be the default.yaml is configure signatures to go into /var/lib/atomic/sigstore | 16:05 |
mordred | which is a weird location on ubuntu - but the package does create it, so fine | 16:05 |
corvus | mordred: yeah, we don't do anything explicitly with signatures, so i don't know what that's going to mean for us | 16:05 |
corvus | (also, how does that interact with perms for an unprivileged tool. maybe it has nothing to do with skopeo, that's just an intersection with another project atomic tool) | 16:06 |
clarkb | ya this seems to be the common stuff for libcontainers/runc | 16:06 |
mordred | corvus: I know that a while back I had an issue with buildah not working out of the box because it was missing a containers.conf file | 16:06 |
clarkb | unfortunately maybe a bit too common when it comes to skopeo which isn't running containers | 16:06 |
corvus | clarkb: i'm +1 for upgrading all the ze's when/if you are | 16:06 |
clarkb | corvus: ok I'll run the command on the other executors now | 16:07 |
clarkb | also bwrap doesn't do runc/libcontainer right? so should be unaffected | 16:07 |
clarkb | skopeo is updated on all executors now | 16:08 |
*** dpawlik has quit IRC | 16:08 | |
corvus | ftr, bwrap appears to still work on ze01 :) | 16:09 |
clarkb | this is actually a nice side effect of using a properly standaloen tool like bwrap | 16:10 |
corvus | how did my change to the push/pull roles fail on an ansible-lint failure in fetch-javascript-output? http://logs.openstack.org/41/644241/1/check/openstack-zuul-jobs-linters/d0cc006/job-output.txt.gz#_2019-03-18_15_51_03_418373 | 16:10 |
clarkb | with all the container world changing around it we get consistent behavior | 16:10 |
corvus | yes i would like unix container tools please :) | 16:10 |
corvus | ohhhhh | 16:11 |
clarkb | corvus: I think that means we are missing a required-project and/or role in the path? mordred possible fallout of your changes to fixup js tooling after the bionic switch? | 16:11 |
corvus | is that fallout from multi-ansible? | 16:11 |
clarkb | oh maybe? | 16:11 |
corvus | it's about zuul_return, and we add zuul to the tox-linters job to supply that | 16:11 |
corvus | but it may have moved | 16:11 |
mordred | OH! | 16:11 |
mordred | corvus: yeah | 16:12 |
corvus | i'll work on a patch | 16:12 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Update zuul ansible location https://review.openstack.org/644266 | 16:13 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: buildset registry: don't put skopeo creds on command line https://review.openstack.org/644241 | 16:13 |
corvus | we're going to need to do that everywhere we run linters on zuul jobs | 16:14 |
*** e0ne has quit IRC | 16:14 | |
*** e0ne has joined #openstack-infra | 16:19 | |
*** udesale has quit IRC | 16:21 | |
openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Update zuul ansible location https://review.openstack.org/644272 | 16:23 |
openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Update zuul ansible location https://review.openstack.org/644273 | 16:24 |
openstackgerrit | James E. Blair proposed opendev/base-jobs master: Update zuul ansible location https://review.openstack.org/644274 | 16:24 |
openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Update zuul ansible location https://review.openstack.org/644275 | 16:24 |
corvus | clarkb: ^ we're dueling | 16:25 |
* fungi has duelling banjos stuck in his head now, thanks | 16:25 | |
*** mpjetta has quit IRC | 16:25 | |
openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Update zuul ansible location https://review.openstack.org/644276 | 16:25 |
clarkb | corvus: oh derp I'll abandon mine | 16:25 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-base-jobs master: Update zuul ansible location https://review.openstack.org/644277 | 16:26 |
corvus | that's all that come to mind off the top of my head | 16:26 |
clarkb | I checked system config and we don't do anything like that there | 16:27 |
clarkb | so ya that lost looks good to me | 16:27 |
AJaeger | corvus: yeah, I think you hit them all... Thanks | 16:27 |
corvus | (those all share a change id, btw, so they're asy to find) | 16:27 |
*** mpjetta has joined #openstack-infra | 16:27 | |
corvus | s/asy/easy/ | 16:27 |
*** diablo_rojo has joined #openstack-infra | 16:33 | |
*** ginopc has quit IRC | 16:33 | |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Update zuul ansible location https://review.openstack.org/644266 | 16:34 |
*** ginopc has joined #openstack-infra | 16:34 | |
*** wolverineav has joined #openstack-infra | 16:35 | |
*** dpawlik has joined #openstack-infra | 16:36 | |
clarkb | infra-root if we can get a second reviewer on https://review.openstack.org/#/c/643713/1 and its parent I'm happy to watch those go in today and debug/fix any problems that might come up | 16:37 |
clarkb | this is more puppet4 upgrades | 16:37 |
AJaeger | corvus: Just got two -1 for the linters jobs ;( | 16:38 |
corvus | i guess that's what i get for pushing them all before one of them finishes :) | 16:38 |
clarkb | well the zuul-jobs fix did fix it | 16:39 |
openstackgerrit | Merged openstack-infra/zuul-jobs master: buildset registry: don't put skopeo creds on command line https://review.openstack.org/644241 | 16:39 |
corvus | clarkb, AJaeger: do we install zuul from source in zuul-jobs, but from pypi in the other repos? | 16:41 |
clarkb | I'll recheck the logos change now to get data on ^ | 16:41 |
AJaeger | corvus: we should install from zuul - if not, let's fix... | 16:42 |
*** dpawlik has quit IRC | 16:42 | |
AJaeger | sorry, no time right now to dig into this, corvus | 16:42 |
openstackgerrit | John Studarus proposed openstack-infra/irc-meetings master: moving time up with US daylight savings time change https://review.openstack.org/644280 | 16:43 |
corvus | i don't think that's it... openstack-zuul-jobs run openstack-zuul-jobs-linters, same as zuul-jobs | 16:43 |
corvus | so they're both running the same job which installs zuul from source | 16:43 |
*** gfidente has quit IRC | 16:44 | |
*** Lucas_Gray has quit IRC | 16:47 | |
corvus | oh | 16:50 |
corvus | http://logs.openstack.org/74/644274/1/check/openstack-zuul-jobs-linters/4926b66/job-output.txt.gz#_2019-03-18_16_33_59_550805 | 16:50 |
corvus | mordred, clarkb: ^ that's the problem | 16:50 |
corvus | there's no setup.cfg in the other repos, so we're not installing zuul as a tox sibling | 16:50 |
*** armax has joined #openstack-infra | 16:51 | |
mordred | clarkb: fire at will on those puppet changes | 16:51 |
openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests https://review.openstack.org/644247 | 16:51 |
mordred | corvus: oh. hrm. | 16:51 |
corvus | er... well, there *is* a setup.cfgi n openstack-zuul-jobs | 16:51 |
*** wolverineav has quit IRC | 16:52 | |
corvus | and that did install zuul as a sibling | 16:52 |
corvus | so let's just focus on that one | 16:52 |
corvus | http://logs.openstack.org/72/644272/1/check/openstack-zuul-jobs-linters/7674bc8/job-output.txt.gz | 16:53 |
*** rossella_s has quit IRC | 16:53 | |
mordred | corvus: yeah - so we just need a setup.cfg in opendev/base-jobs | 16:54 |
corvus | mordred: right, but i assume if we did that, we'd just hit the error in openstack-zuul-jobs ^ | 16:54 |
*** gfidente has joined #openstack-infra | 16:55 | |
mordred | corvus: zuul_return isn't in zuul/ansible/base | 16:56 |
mordred | oh - there it is | 16:56 |
corvus | ./zuul/ansible/base/actiongeneral/zuul_return.py | 16:56 |
mordred | nevermind | 16:56 |
mordred | yeah | 16:56 |
*** josephrsandoval has quit IRC | 16:57 | |
*** rossella_s has joined #openstack-infra | 16:57 | |
*** wolverineav has joined #openstack-infra | 16:59 | |
*** pgaxatte has quit IRC | 17:00 | |
*** mgoddard has quit IRC | 17:02 | |
corvus | i'm out of ideas | 17:03 |
*** trown is now known as trown|lunch | 17:03 | |
* clarkb catches up after eating some breakfast | 17:06 | |
*** mgoddard has joined #openstack-infra | 17:08 | |
clarkb | ansible==2.3.3.0,ansible-lint==3.5.1 is it possible we want a newer version of ansible in that env? | 17:09 |
corvus | wow. presumably we want 2.7 now. but we did have 2.5.15 in the change that worked | 17:10 |
clarkb | previously we would've installed ansible 2.5 via the zuul requirements but now with multi ansible that is deferred | 17:10 |
AJaeger | test-requirements says "ansible < 2.4" | 17:10 |
AJaeger | that needs manual update when zuul changes ansible version | 17:10 |
corvus | it's different in zuul-jobs: ansible>=2.5.1,<2.6 | 17:11 |
*** dpawlik has joined #openstack-infra | 17:11 | |
corvus | it's difficult for me to see how that would affect this, but let's go ahead and change openstack-zuul-jobs to match | 17:12 |
clarkb | ++ | 17:12 |
clarkb | there appears to be a bug in the skopeo changes http://logs.openstack.org/22/643622/1/check/opendev-buildset-registry/1e76abc/job-output.txt.gz#_2019-03-18_17_08_19_804044 | 17:12 |
clarkb | I'm looking at ^ now | 17:13 |
openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Update zuul ansible location https://review.openstack.org/644272 | 17:13 |
corvus | clarkb: doh. that's my mistake. will fix | 17:13 |
clarkb | corvus: ok thanks | 17:13 |
*** e0ne has quit IRC | 17:14 | |
*** armax has quit IRC | 17:15 | |
*** dpawlik has quit IRC | 17:16 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: buildset registry: fix proxy_port error in push/pull roles https://review.openstack.org/644322 | 17:16 |
corvus | clarkb: ^ | 17:16 |
clarkb | corvus: I went ahead and approved it since its a straightfoward config change via removal | 17:17 |
corvus | ++ | 17:17 |
*** ykarel|away has quit IRC | 17:18 | |
*** snapiri has quit IRC | 17:19 | |
clarkb | as a general reminder to the channel I'll be driving to and from seattle tomorrow. I expect to run the meeting as I should be there well before 1900UTC. But I may be afk for good chunks of the early and late day | 17:19 |
fungi | thanks for the heads up! | 17:21 |
*** mpjetta has quit IRC | 17:22 | |
*** kopecmartin is now known as kopecmartin|off | 17:22 | |
* clarkb puts together a meeting agenda and remembers to send it an hour later today :) | 17:24 | |
clarkb | I think that was why ianw missed the cutoff. I was operating on local time and sent it at 1800UTC instead of 1900UTC. Sorry | 17:24 |
*** rpioso|afk is now known as rpioso | 17:25 | |
* fungi shakes fist at arbitrary time changes | 17:25 | |
mordred | as a similar reminder to the channel - I'll be basically out tues-thurs. I will probably also work, but I have no idea which hours it might be | 17:26 |
*** rossella_s has quit IRC | 17:26 | |
*** mattw4 has joined #openstack-infra | 17:29 | |
*** mpjetta has joined #openstack-infra | 17:30 | |
*** ramishra has quit IRC | 17:35 | |
*** rossella_s has joined #openstack-infra | 17:35 | |
openstackgerrit | Merged openstack-infra/zuul-jobs master: buildset registry: fix proxy_port error in push/pull roles https://review.openstack.org/644322 | 17:35 |
*** jamesmcarthur has quit IRC | 17:38 | |
*** jamesmcarthur has joined #openstack-infra | 17:39 | |
*** ricolin has quit IRC | 17:41 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Increase wait time of test_plugins https://review.openstack.org/644375 | 17:42 |
*** rpittau is now known as rpittau|afk | 17:43 | |
openstackgerrit | Merged openstack-infra/zuul master: Don't call the merger for non-live items https://review.openstack.org/643703 | 17:44 |
*** rossella_s has quit IRC | 17:44 | |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Update zuul ansible location https://review.openstack.org/644272 | 17:45 |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add foreground option https://review.openstack.org/635649 | 17:46 |
*** bgmccollum has quit IRC | 17:47 | |
*** rossella_s has joined #openstack-infra | 17:47 | |
*** rossella_s has quit IRC | 17:48 | |
*** chandankumar is now known as raukadah | 17:51 | |
*** bgmccollum has joined #openstack-infra | 17:52 | |
*** yamamoto has joined #openstack-infra | 17:53 | |
corvus | clarkb: oh hey, i guess it was the ansible version? | 17:57 |
corvus | clarkb, mordred, AJaeger: i'll update the other patches to (a) have a setup.cfg and/or (b) update ansible version as required | 17:58 |
clarkb | the skopeo run on ovh-bhs1 just succeeded | 17:58 |
clarkb | s/on/again/st | 17:58 |
*** gyee has joined #openstack-infra | 17:58 | |
corvus | clarkb: cool, i saw failures from mtl01 and gra1 | 17:59 |
clarkb | it wasn't a 100% failure so not sure if fixed but that is a good sign | 17:59 |
corvus | so it would be good to get a pass from one of those | 17:59 |
corvus | or even both :) | 17:59 |
clarkb | ++ | 17:59 |
corvus | clarkb: i rechecked 640901 and 940904 | 17:59 |
corvus | those push to intermediate, and they only run a build job | 18:00 |
*** trown|lunch is now known as trown | 18:00 | |
corvus | so should be less wasteful / faster to repeatedly recheck those | 18:00 |
clarkb | k | 18:00 |
corvus | infra-root: would you please review https://review.openstack.org/643954 and https://review.openstack.org/643957 which are needed for the zuul restart with multi-ansible ? | 18:01 |
corvus | i've verified the lest failure on the second is unrelated puppet stuff (elasticsearch) | 18:01 |
corvus | s/lest/test/ | 18:01 |
*** armax has joined #openstack-infra | 18:02 | |
corvus | lol one of the ansible fixes was actually an irc-access failure | 18:03 |
clarkb | corvus: left a note about explicit ordering on the second change. I don't think that is strictly required though | 18:04 |
corvus | clarkb: yeah, good point. i think as long as we aren't auto-restarting it, it doesn't matter? | 18:05 |
clarkb | corvus: ya | 18:05 |
clarkb | it would be more for humans understanding how the pieces go together probably | 18:05 |
mordred | corvus: done | 18:06 |
*** sthussey has joined #openstack-infra | 18:06 | |
*** jpena is now known as jpena|off | 18:06 | |
*** dustinc is now known as dustinc|lunch | 18:13 | |
openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Update zuul ansible location https://review.openstack.org/644273 | 18:16 |
*** iurygregory has quit IRC | 18:17 | |
corvus | config-core: ^ that's an omnibus fix for 2 things broken in p-c right now | 18:17 |
*** jtomasek has quit IRC | 18:18 | |
openstackgerrit | James E. Blair proposed opendev/base-jobs master: Update zuul ansible location https://review.openstack.org/644274 | 18:18 |
*** jtomasek has joined #openstack-infra | 18:19 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-base-jobs master: Update zuul ansible location https://review.openstack.org/644277 | 18:22 |
corvus | wow. i think *every* one of those 5 failures was for a different reason. | 18:22 |
corvus | clarkb: http://logs.openstack.org/01/640901/2/check/zuul-preview-build-image/5b1e40c/zuul-info/inventory.yaml is a pass on mtl01 | 18:22 |
*** dtantsur is now known as dtantsur|afk | 18:23 | |
*** eharney_ has joined #openstack-infra | 18:23 | |
corvus | clarkb, mordred: it's worth considering that the gitea image layers may be larger than the zuul-preview layers | 18:23 |
clarkb | http://logs.openstack.org/01/640901/2/check/zuul-preview-build-image/5b1e40c/job-output.txt.gz#_2019-03-18_18_08_08_142865 is the step that was breaking before | 18:23 |
clarkb | corvus: when I looked on friday it seemed like we were talking 10s of megabytes so no huge but not tiny either | 18:23 |
corvus | would be cool if skopeo told us how big each layer was | 18:24 |
*** bgmccollum has quit IRC | 18:25 | |
corvus | a full gitea image is 451MB, but i don't know the layer breakdown | 18:25 |
openstackgerrit | Merged openstack-infra/zuul master: Whitelist ansible_python_interpreter for add_host https://review.openstack.org/643726 | 18:25 |
*** eharney has quit IRC | 18:25 | |
openstackgerrit | James E. Blair proposed openstack-infra/system-config master: DNM: just build a gitea image https://review.openstack.org/644387 | 18:26 |
corvus | clarkb: ^ for rechecks | 18:27 |
clarkb | corvus: related to this any idea why https://review.openstack.org/#/c/643623/1 skipped the image builds on check? | 18:28 |
clarkb | I'm looking at the job config and files is set to docker/gitea/.* which should match taht file. Then the build-image jobs provides: docker-image via the opendev jobs and buildset-registry requires: docker-image so it shoudl'ev run I think | 18:28 |
corvus | clarkb: are you using the gerrit webui with hideci enabled? | 18:29 |
clarkb | corvus: no, if you expand the jobs you'll see they were all skippec | 18:29 |
clarkb | *skipped | 18:29 |
corvus | clarkb: well, zuul says: Warning: Requirements ['docker-image'] not met by build 32fae011811144ef9491fc16e54c5cf7 | 18:29 |
openstackgerrit | Merged openstack-infra/puppet-zuul master: Set fingergw user to zuul https://review.openstack.org/643954 | 18:29 |
*** wolverineav has quit IRC | 18:31 | |
corvus | so that's the proximate cause; we'll have to dig deeper to find out why the requirement wasn't met | 18:31 |
clarkb | build there is a zuul build uuid? | 18:31 |
corvus | ya | 18:31 |
corvus | ah, because it was a post failure of the parent change | 18:32 |
corvus | http://zuul.openstack.org/builds?uuid=32fae011811144ef9491fc16e54c5cf7 | 18:32 |
clarkb | ah ok so it was caused by the thing we are trying to fix with the skopeo upgrade | 18:32 |
corvus | perhaps that should be a failure, not a skip? | 18:33 |
*** gmann is now known as gmann_afk | 18:34 | |
*** rossella_s has joined #openstack-infra | 18:36 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Fail jobs when requirements not met https://review.openstack.org/644392 | 18:37 |
mordred | corvus: ++ yes - I think that's the right choice | 18:38 |
openstackgerrit | Merged opendev/base-jobs master: Update zuul ansible location https://review.openstack.org/644274 | 18:38 |
openstackgerrit | James E. Blair proposed openstack-infra/system-config master: DNM: just build a gitea image https://review.openstack.org/644387 | 18:38 |
*** armax has quit IRC | 18:40 | |
*** jamesmcarthur has quit IRC | 18:41 | |
*** gfidente has quit IRC | 18:42 | |
clarkb | hrm jobs still failing on some of those multi ansible file location fixes /me looks | 18:43 |
corvus | i'm looking at the irc one | 18:44 |
corvus | project-config | 18:44 |
clarkb | ok I'll look at https://review.openstack.org/#/c/644277/ (zuul-base-jobs) | 18:44 |
clarkb | looks like actual linting failures. I can push a new patchset to fix | 18:44 |
*** mriedem has quit IRC | 18:45 | |
corvus | clarkb: i tried to fix that by pinning | 18:45 |
corvus | clarkb: it looks like the pin didn't work? | 18:45 |
tobiash | so there is already fallout without even restarting an executor? | 18:46 |
clarkb | corvus: oh ya it still installed 4.1.0 | 18:46 |
corvus | tobiash: the ansible lint jobs are not robunt | 18:46 |
corvus | robust | 18:46 |
tobiash | ah | 18:46 |
corvus | i expect that we will remove it entirely soon | 18:46 |
clarkb | corvus: I see the issue I'll push a new patchset | 18:47 |
corvus | we've spent something like 100 person hours doing nothing more than trying to keep it working the same way it used to over the past week alone. | 18:47 |
corvus | and we can never upgrade it because it's getting far too opinionated about how to construct roles (so that they can have a high "score" on galaxy) | 18:48 |
*** mriedem has joined #openstack-infra | 18:48 | |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul-base-jobs master: Update zuul ansible location https://review.openstack.org/644277 | 18:49 |
*** wolverineav has joined #openstack-infra | 18:49 | |
clarkb | corvus: ^ we defined the deps in multiple places so I reduced it to a single location | 18:49 |
fungi | what legitimate bugs does ansible lint catch which the zuul parser wouldn't? | 18:49 |
fungi | i feel like not many | 18:49 |
fungi | and we don't purport to follow any particular stylistic conventions with zuul job playbooks/roles | 18:50 |
corvus | i can't reproduce the irc error locally: http://logs.openstack.org/73/644273/2/check/project-config-irc-access/12afdda/job-output.txt.gz#_2019-03-18_18_23_45_930689 | 18:51 |
openstackgerrit | Ankita Bansal proposed openstack-infra/storyboard master: Project Group UI should permit removing project group even if it is not empty. https://review.openstack.org/642266 | 18:51 |
openstackgerrit | Merged openstack-infra/zuul master: Properly handle missing ansible version in executor https://review.openstack.org/644252 | 18:53 |
fungi | that does seem like a very bizarre sort order | 18:53 |
corvus | input=output is what i get when i run it locally | 18:54 |
corvus | so even if i move 'branches' to the end, it still passes | 18:54 |
fungi | i can't even imagine something like LC_COLLATE would influence ordering b before p | 18:55 |
* mordred has confusion | 18:55 | |
corvus | fungi: well, it's not sorted | 18:55 |
*** jamesmcarthur has joined #openstack-infra | 18:56 | |
*** gmann_afk is now known as gmann | 18:56 | |
mordred | corvus: when you run it locally, are you running it in tox / with python2.7? | 18:56 |
fungi | python dict ordering wouldn't have changes between xenial and bionic's python2.7 would it? | 18:57 |
corvus | mordred: no, in a py3.6 venv | 18:57 |
fungi | yeah, the job there is running on 2.7 | 18:57 |
corvus | oh? | 18:57 |
mordred | nod. it seems to be running in 2.7 in the gate, which uses the old unordered dict implementation | 18:57 |
corvus | why? | 18:57 |
fungi | this is a great question | 18:58 |
mordred | it doesn't make any sense to me looking at the tox.ini file | 18:59 |
mordred | but http://logs.openstack.org/73/644273/2/check/project-config-irc-access/12afdda/job-output.txt.gz#_2019-03-18_18_20_20_962767 | 18:59 |
clarkb | is there a hard coded shebang? | 19:00 |
fungi | tox itself is running with 2.7 according to the log | 19:00 |
mordred | well - if tox was installed with python2 and basepython is not set to the contrary | 19:00 |
mordred | it'll run under python2 | 19:00 |
*** jamesmcarthur has quit IRC | 19:00 | |
mordred | and basepython is not set in [testenv] in that tox.ini | 19:00 |
mordred | so maybe we should do that | 19:01 |
corvus | so probably pyyaml <5.1 used an ordereddict, and pyyaml 5.1 does not, but dicts themselves are ordered | 19:01 |
fungi | seems to be the case, yes. not set in [testenv:irc] either, so it's going to be 2.7 | 19:01 |
corvus | i'll update the patch | 19:01 |
mordred | also - why are we installing grafyaml from source in teh grafyaml env? | 19:02 |
mordred | that's probably for siblings support from before we had siblings? | 19:02 |
openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Update zuul ansible location https://review.openstack.org/644273 | 19:02 |
corvus | mordred: unsure; i defered that but otherwise ^ does that look right? | 19:03 |
mordred | ++ | 19:03 |
openstackgerrit | Merged openstack-infra/system-config master: Use OpenDev icon in gitea https://review.openstack.org/643622 | 19:03 |
*** dave-mccowan has joined #openstack-infra | 19:05 | |
mordred | well - it turns out it only passes with master of grafyaml | 19:05 |
*** dpawlik has joined #openstack-infra | 19:05 | |
mordred | so if we want to use it from release, we'd need to cut a grafyaml release | 19:05 |
*** dpawlik has quit IRC | 19:10 | |
openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests https://review.openstack.org/644247 | 19:11 |
clarkb | grr now hacking/flake8 fail on zuul-base-jobs looks like an issue with older hacking against newer flake8. I'm testing if newer hacking works locally | 19:12 |
*** sshnaidm|rover is now known as sshnaidm|afk | 19:13 | |
openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests https://review.openstack.org/644247 | 19:14 |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul-base-jobs master: Update zuul ansible location https://review.openstack.org/644277 | 19:15 |
*** jamesmcarthur has joined #openstack-infra | 19:18 | |
*** bgmccollum has joined #openstack-infra | 19:18 | |
*** jamesmcarthur has quit IRC | 19:19 | |
*** eharney_ has quit IRC | 19:20 | |
*** jamesmcarthur has joined #openstack-infra | 19:23 | |
*** dustinc|lunch is now known as dustinc | 19:23 | |
*** wolverineav has quit IRC | 19:25 | |
clarkb | woo ^ passed tests and is on its way to the gate. I think that is all of them now. /me sends meeting agenda and finds lunch | 19:26 |
mordred | clarkb: \o/ | 19:28 |
*** armax has joined #openstack-infra | 19:29 | |
openstackgerrit | Merged openstack-infra/project-config master: Update zuul ansible location https://review.openstack.org/644273 | 19:31 |
openstackgerrit | Merged openstack-infra/zuul-base-jobs master: Update zuul ansible location https://review.openstack.org/644277 | 19:32 |
openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests https://review.openstack.org/644247 | 19:39 |
*** imacdonn_ has joined #openstack-infra | 19:39 | |
*** imacdonn_ is now known as imacdonn | 19:40 | |
openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests https://review.openstack.org/644247 | 19:40 |
*** dpawlik has joined #openstack-infra | 19:40 | |
mordred | fungi: feel like a git-review? https://review.openstack.org/#/c/643966/ | 19:43 |
*** wolverineav has joined #openstack-infra | 19:44 | |
fungi | thanks for the reminder. so far my morning has been like a freight train but i have a couple minutes to look at something now | 19:45 |
*** dpawlik has quit IRC | 19:45 | |
clarkb | the opendev logo looks to be applied more broadly on gitea now | 19:46 |
clarkb | corvus: fyi ^ and thanks | 19:46 |
*** wolverineav has quit IRC | 19:48 | |
*** wolverineav has joined #openstack-infra | 19:48 | |
clarkb | mordred: corvus the last thing I notice is in mobile the search bar is still that gitea green | 19:49 |
clarkb | After lunch I'll see if I can figure out how ti make it opendev purple/pink | 19:49 |
mordred | clarkb: it's not green for me on mobile? | 19:50 |
mordred | clarkb: oh - well - the URL title bar is - maybe that's the thing you mean? | 19:51 |
mordred | so yeah - we should find that setting | 19:51 |
clarkb | yea | 19:51 |
fungi | i like how "opendev" is now the name of a purply-pink color | 19:53 |
*** whoami-rajat has quit IRC | 19:55 | |
openstackgerrit | James E. Blair proposed openstack-infra/system-config master: DNM: just build a gitea image https://review.openstack.org/644387 | 19:58 |
*** jamesmcarthur has quit IRC | 20:00 | |
corvus | clarkb: https://review.openstack.org/642845 is a simple ozj change | 20:00 |
fungi | clarkb: is there a flag to switch to "mobile view" or is that only decided automagically? i can't find a ui widget nor do there seem to be selectable alternative stylesheets for it | 20:00 |
corvus | fungi: i think to see this you must actually be on a phone | 20:01 |
*** tram- has joined #openstack-infra | 20:02 | |
*** tram- has left #openstack-infra | 20:02 | |
fungi | weird | 20:03 |
corvus | even firefox's web tools don't show it | 20:03 |
*** pcrews has quit IRC | 20:07 | |
clarkb | as an alternative we can remove the coloring of the url bar aand use defauly | 20:08 |
*** jamesmcarthur has joined #openstack-infra | 20:09 | |
*** wolverineav has quit IRC | 20:09 | |
*** wolverineav has joined #openstack-infra | 20:10 | |
*** jamesmcarthur has quit IRC | 20:10 | |
*** jamesmcarthur has joined #openstack-infra | 20:10 | |
*** irclogbot_1 has joined #openstack-infra | 20:12 | |
*** dpawlik has joined #openstack-infra | 20:14 | |
*** dklyle has quit IRC | 20:14 | |
openstackgerrit | Merged openstack-infra/zuul master: Increase wait time of test_plugins https://review.openstack.org/644375 | 20:15 |
*** wolverineav has quit IRC | 20:16 | |
*** wolverineav has joined #openstack-infra | 20:16 | |
*** wolverineav has quit IRC | 20:16 | |
*** wolverineav has joined #openstack-infra | 20:17 | |
*** fresta_ has quit IRC | 20:17 | |
*** dklyle has joined #openstack-infra | 20:18 | |
*** dpawlik has quit IRC | 20:18 | |
openstackgerrit | Merged openstack-infra/system-config master: Upgrade to gitea 1.7.4 https://review.openstack.org/643623 | 20:19 |
openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use YAML anchors to avoid repeating irrelevant-files https://review.openstack.org/642845 | 20:19 |
corvus | clarkb: http://logs.openstack.org/87/644387/3/check/system-config-build-image-gitea/e6d9a5d/zuul-info/inventory.yaml gra1 passed | 20:19 |
*** eharney_ has joined #openstack-infra | 20:21 | |
*** Lucas_Gray has joined #openstack-infra | 20:21 | |
*** irclogbot_1 has quit IRC | 20:25 | |
*** irclogbot_1 has joined #openstack-infra | 20:27 | |
*** e0ne has joined #openstack-infra | 20:28 | |
openstackgerrit | Merged openstack-infra/git-review master: Use remote_url instead of remote for download https://review.openstack.org/643966 | 20:30 |
openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Add unit tests https://review.openstack.org/644247 | 20:30 |
*** kgiusti has left #openstack-infra | 20:36 | |
clarkb | corvus: woot I'm guessing it was a skopeo bug that was fixed | 20:37 |
*** dave-mccowan has quit IRC | 20:39 | |
clarkb | mordred: corvus ThemeColorMetaTag is the value to change in gitea's setting/setting.go or templates/base/head.tmpl | 20:41 |
*** e0ne has quit IRC | 20:41 | |
clarkb | any idea if the contents of setting.go will read values out of a config file somewhere? | 20:41 |
*** dave-mccowan has joined #openstack-infra | 20:44 | |
clarkb | ya there seems to be a mapping between setting.go and app.ini UI section | 20:45 |
openstackgerrit | Merged openstack-infra/puppet-zuul master: Update executor ansible on installation https://review.openstack.org/643957 | 20:48 |
*** raissa has joined #openstack-infra | 20:50 | |
*** dpawlik has joined #openstack-infra | 20:51 | |
clarkb | heh ini.AllCapsUnderscore seems to be the magic | 20:52 |
clarkb | hrm do we not set a custom app.ini yet? | 20:53 |
clarkb | ah it is in ansible not docker | 20:54 |
*** pcaruana has quit IRC | 20:54 | |
*** dpawlik has quit IRC | 20:55 | |
corvus | Shrews: https://review.openstack.org/640822 is up your alley :) | 20:59 |
*** trown is now known as trown|outtypewww | 20:59 | |
openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Set the gitea theme color to match the opendev pink https://review.openstack.org/644420 | 21:01 |
clarkb | I think ^ should work to change that green color | 21:01 |
Shrews | corvus: +3 | 21:06 |
*** Lucas_Gray has quit IRC | 21:06 | |
*** rh-jelabarre has quit IRC | 21:07 | |
openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Set the gitea theme color to match the opendev pink https://review.openstack.org/644420 | 21:08 |
clarkb | corvus: ^ sorry I forgot to convert it to the AllCapsUnderscore format :/ should be good now | 21:09 |
*** priteau has quit IRC | 21:12 | |
*** armax has quit IRC | 21:13 | |
clarkb | corvus: fyi I'm looking at fixing http://logs.openstack.org/12/643712/1/gate/opendev-buildset-registry/1016e6e/job-output.txt.gz#_2019-03-18_19_28_39_060210 which is a different fallout of changes to the intermediate registry setup | 21:16 |
openstackgerrit | Malek Karray proposed openstack-infra/storyboard-webclient master: Converting a search query into a browse https://review.openstack.org/644424 | 21:17 |
*** xek has quit IRC | 21:17 | |
*** mgoddard has quit IRC | 21:20 | |
*** raissa has quit IRC | 21:20 | |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Handle no metadata in zuul artifact items https://review.openstack.org/644428 | 21:23 |
clarkb | corvus: ^ I think that should fix it | 21:23 |
*** irclogbot_1 has quit IRC | 21:26 | |
*** irclogbot_1 has joined #openstack-infra | 21:28 | |
clarkb | infra-root I'm going to run `manage-projects -d openstack/deb-python-oslotest` on review.o.o to see if I can see why that project doesn't have the retired.config acl applied | 21:30 |
openstackgerrit | Merged openstack-infra/nodepool master: config: use yaml.safe_load instead of load https://review.openstack.org/643849 | 21:35 |
clarkb | that seems to do a bunch of nothing | 21:36 |
openstackgerrit | Merged openstack-infra/system-config master: Add nodepool_restart playbook https://review.openstack.org/640822 | 21:37 |
clarkb | https://review.openstack.org/#/c/506219/2 is teh change that broke setting of retired acls | 21:55 |
clarkb | the configs_list property was updated to return only unretired projects and that is the list we iterate over to set project configs in manage_projects | 21:55 |
clarkb | I updated manage_projects.py on review01 to print out the projects it was iterating over and none of the retired projects show up | 21:56 |
clarkb | (I've since reverted that and `diff /opt/jeepyb/jeepyb/cmd/manage_projects.py /usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py` shows no delta so should be back to normal now | 21:56 |
fungi | aha, chicken-and-egg | 21:57 |
fungi | we want to exclude retired projects from lots of things, but not from the routine which retires them ;) | 21:57 |
clarkb | ya I think we want manage projects to iterate over a complete list | 21:57 |
clarkb | and then most everything else can use the smaller list | 21:58 |
clarkb | I'll write up a change but I probably won't run it soon since I'll be doing that driving thing tomorrow | 21:58 |
*** armax has joined #openstack-infra | 22:03 | |
*** rfolco has quit IRC | 22:04 | |
openstackgerrit | Clark Boylan proposed openstack-infra/jeepyb master: Inspect all configs in manage-projects https://review.openstack.org/644432 | 22:04 |
clarkb | we may want to run that by hand and not with puppet as it may take some time to build out and configure all of the retired projects | 22:05 |
*** dpawlik has joined #openstack-infra | 22:05 | |
*** jtomasek has quit IRC | 22:05 | |
*** dpawlik has quit IRC | 22:10 | |
corvus | clarkb: hrm... why was there no metadata? | 22:14 |
clarkb | corvus: the change it failed on was the puppet4 enablement change in our groups.yaml file so I think it was because there were no image artifacts | 22:15 |
corvus | clarkb: i see artifacts in http://logs.openstack.org/12/643712/1/gate/opendev-buildset-registry/1016e6e/zuul-info/inventory.yaml | 22:16 |
clarkb | oh huh and they don't have a metadata entry | 22:16 |
clarkb | so maybe my fix isn't the right option there | 22:17 |
corvus | it looks like our upgrade gitea to 1.7.4 change ran after the parent change (643622) merged, so we didn't see that exercised earlier | 22:18 |
corvus | those artifacts came from the 'upload' job vs the 'build' job, and this may be the first time that happened... they should be the same as far as this is concerned, maybe there's a difference. | 22:20 |
clarkb | push-to-intermediate-registry is what seems to set the metadata field | 22:21 |
corvus | yeah, and i can't see a way for it not to do that | 22:21 |
corvus | so maybe there's a bug in the artifact collection from a live item | 22:21 |
clarkb | ya just double checked the inheritance and we should run the push to intermediate role prior to uploading to docker because the buildset registry base job is parent of the upload to docker job | 22:24 |
clarkb | I'm going to recheck that change again as there aren't any image builds in the pipeline that I know of and that should get the puppet4 change in | 22:24 |
corvus | clarkb: ++ | 22:24 |
corvus | i'll look into whether this is a zuul bug | 22:24 |
*** kjackal_v2 has quit IRC | 22:30 | |
clarkb | #status Log modified jeepyb files on review01.o.o to debug why manage-projects isn't setting retired project acls. I have since restored those file contents and `diff /opt/jeepyb/jeepyb/cmd/manage_projects.py /usr/local/lib/python2.7/dist-packages/jeepyb/cmd/manage_projects.py` shows no delta. | 22:31 |
openstackstatus | clarkb: finished logging | 22:31 |
openstackgerrit | Merged openstack-infra/zuul master: Wait for command processor thread to stop https://review.openstack.org/641854 | 22:42 |
ianw | did we always need to set "OS_VOLUME_API_VERSION=1" when chatting to rax volume manager? i feel like that's new, but haven't done it in a while ... | 22:52 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix metadata not passed in provides/requires https://review.openstack.org/644445 | 22:52 |
corvus | clarkb: ^ definitely a bug | 22:52 |
corvus | let me rebase that | 22:53 |
clarkb | ianw: if I had to guess upstream tooling now defaults to version 2, but rax only has version 1 | 22:54 |
clarkb | ianw: so ya likely new | 22:54 |
*** tkajinam has joined #openstack-infra | 22:54 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix metadata not passed in provides/requires https://review.openstack.org/644445 | 22:54 |
*** wolverineav has quit IRC | 22:57 | |
ianw | clarkb: still need cc655f1e-2ce6-4657-b9c1-792a5861e97a | clarkb-test-afs01-ord-upgrade ? | 22:57 |
*** wolverineav has joined #openstack-infra | 22:57 | |
clarkb | ianw: nope | 22:58 |
clarkb | afs seems stable after the upgrades, that one can go away | 22:58 |
clarkb | ianw: are you volunteering to clear it out or should I get it? | 22:58 |
clarkb | corvus: fix lgtm | 22:59 |
ianw | clarkb: i will, poking around in there doing this backup rotation | 22:59 |
*** armax has quit IRC | 22:59 | |
ianw | done | 23:00 |
clarkb | thanks | 23:00 |
*** elod has quit IRC | 23:01 | |
clarkb | corvus: if you have a moment can you review https://review.openstack.org/643671 and https://review.openstack.org/#/c/644420/ two small gitea fixups | 23:03 |
*** wolverineav has quit IRC | 23:03 | |
*** armax has joined #openstack-infra | 23:04 | |
corvus | yep! | 23:04 |
*** rcernin has joined #openstack-infra | 23:04 | |
*** fresta has joined #openstack-infra | 23:05 | |
corvus | Mar 18 22:57:30 ze01 puppet-user[27467]: (/Stage[main]/Zuul::Executor/Exec[zuul_manage_ansible]) Triggered 'refresh' from 1 events | 23:06 |
corvus | that happened ^ | 23:06 |
corvus | it didn't log any errors | 23:06 |
*** rkukura_ has joined #openstack-infra | 23:06 | |
*** rkukura has quit IRC | 23:06 | |
*** rkukura_ is now known as rkukura | 23:06 | |
corvus | if it supplied any output though, i don't know where to find it | 23:07 |
corvus | anyone know where puppet exec output goes these days? | 23:08 |
corvus | well, it seems to have worked | 23:10 |
clarkb | I think puppet logs it to syslog iff rc >0 | 23:10 |
clarkb | otherwise it may not record it | 23:11 |
corvus | so i think we're okay to restart one of the zuul executors to verify things are okay, then we can restart all of zuul to pick up the new feature | 23:11 |
clarkb | exciting | 23:11 |
corvus | i'm at ~eod... should we wait till tomorrow? | 23:11 |
clarkb | My ability to help tomorrow is likely lower than normal but I'm fine with that myself | 23:12 |
fungi | i'm good with helping tomorrow but starting to wind down myself for the evening | 23:17 |
corvus | that's probably best unless someone is really eager to follow through on it now :) | 23:18 |
openstackgerrit | Merged openstack-infra/system-config master: Update more dev servers to puppet 4 https://review.openstack.org/643712 | 23:18 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fail jobs when requirements not met https://review.openstack.org/644392 | 23:19 |
*** mriedem has quit IRC | 23:20 | |
openstackgerrit | Merged openstack-infra/system-config master: Retry gitea repo setting HTTP POSTs https://review.openstack.org/643671 | 23:25 |
*** tosky has quit IRC | 23:48 | |
openstackgerrit | Merged openstack-infra/system-config master: Set the gitea theme color to match the opendev pink https://review.openstack.org/644420 | 23:48 |
*** jamesmcarthur has quit IRC | 23:51 | |
*** wolverineav has joined #openstack-infra | 23:54 | |
*** wolverineav has quit IRC | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!