| *** cmoura has quit IRC | 00:00 | |
| *** cmoura has joined #openstack-infra | 00:00 | |
| *** armax has joined #openstack-infra | 00:01 | |
| *** wolverineav has joined #openstack-infra | 00:07 | |
| *** dpawlik has joined #openstack-infra | 00:10 | |
| clarkb | Thinking out loud here, but maybe OpenStack would like to move all of the deb-* repos into cold storage and we can stop trying to configure them on gitea so much :) | 00:11 |
|---|---|---|
| clarkb | (I don't actually know how much value there is there given that they are mostly just forks of our actual repos) | 00:12 |
| clarkb | (and now live elsewhere presumably with all their history) | 00:12 |
| *** wolverineav has quit IRC | 00:12 | |
| ianw | definitely worth investigating i think, we do all these operations that loop over all repos | 00:13 |
| ianw | "as we are doing" i mean | 00:13 |
| *** dpawlik has quit IRC | 00:15 | |
| *** mattw4 has quit IRC | 00:19 | |
| *** dpawlik has joined #openstack-infra | 00:22 | |
| *** hamzy has joined #openstack-infra | 00:22 | |
| *** wolverineav has joined #openstack-infra | 00:22 | |
| *** dpawlik has quit IRC | 00:26 | |
| *** wolverineav has quit IRC | 00:28 | |
| *** mattw4 has joined #openstack-infra | 00:33 | |
| *** markvoelker has joined #openstack-infra | 00:34 | |
| *** mattw4 has quit IRC | 00:38 | |
| *** markvoelker has quit IRC | 00:39 | |
| *** armax has quit IRC | 00:42 | |
| *** dpawlik has joined #openstack-infra | 00:52 | |
| *** wolverineav has joined #openstack-infra | 00:53 | |
| *** dpawlik has quit IRC | 00:57 | |
| *** markvoelker has joined #openstack-infra | 01:05 | |
| *** hwoarang has quit IRC | 01:12 | |
| *** hwoarang has joined #openstack-infra | 01:14 | |
| *** whoami-rajat has joined #openstack-infra | 01:17 | |
| *** ricolin has joined #openstack-infra | 01:27 | |
| *** jamesmcarthur has joined #openstack-infra | 01:28 | |
| *** markvoelker has quit IRC | 01:32 | |
| *** rlandy|bbl is now known as rlandy | 01:41 | |
| *** wolverineav has quit IRC | 01:42 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** hongbin has joined #openstack-infra | 01:54 | |
| *** dpawlik has joined #openstack-infra | 02:07 | |
| *** wolverineav has joined #openstack-infra | 02:08 | |
| *** dpawlik has quit IRC | 02:12 | |
| *** dpawlik has joined #openstack-infra | 02:18 | |
| *** dpawlik has quit IRC | 02:22 | |
| *** ricolin has quit IRC | 02:23 | |
| *** ricolin has joined #openstack-infra | 02:23 | |
| *** wolverineav has quit IRC | 02:39 | |
| *** wolverineav has joined #openstack-infra | 02:39 | |
| *** ricolin has quit IRC | 02:45 | |
| *** wolverineav has quit IRC | 02:48 | |
| clarkb | ok gitea fixup completed successfully on all gitea servers but gitea05 | 02:50 |
| clarkb | gitea05 get a 404 when hitting openstack/openstacksdk/settings | 02:50 |
| *** ccamacho has quit IRC | 02:50 | |
| clarkb | Playbook run took 0 days, 2 hours, 53 minutes, 29 seconds was total runtime fwiw | 02:51 |
| clarkb | I guess tomorrow I'll do a run on gitea05 only? | 02:51 |
| *** dpawlik has joined #openstack-infra | 02:52 | |
| clarkb | side effect of the server resize: ansible/puppet running every 30 minutes instead of every 45 | 02:52 |
| clarkb | thats a nice improvement | 02:52 |
| clarkb | ianw: before you call it a week care to review https://review.openstack.org/#/c/643465/ and https://review.openstack.org/#/c/634726/7 those changes will get us moving along on puppet 4 | 02:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: enable zuul_return to update Ansible inventory https://review.openstack.org/590092 | 02:55 |
| *** dpawlik has quit IRC | 02:56 | |
| *** psachin has joined #openstack-infra | 03:03 | |
| ianw | will do | 03:06 |
| *** wolverineav has joined #openstack-infra | 03:11 | |
| *** wolverineav has quit IRC | 03:37 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove trusty testing flag from nodes that are xenial https://review.openstack.org/643168 | 03:39 |
| *** Lucas_Gray has joined #openstack-infra | 03:48 | |
| *** ramishra has joined #openstack-infra | 03:48 | |
| *** udesale has joined #openstack-infra | 03:50 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add log_stream_port and log_stream_file settings https://review.openstack.org/535538 | 03:51 |
| *** chkumar246 is now known as chandankumar | 03:52 | |
| *** ykarel has joined #openstack-infra | 03:58 | |
| *** wolverineav has joined #openstack-infra | 04:14 | |
| *** hongbin has quit IRC | 04:16 | |
| *** ykarel has quit IRC | 04:20 | |
| *** ykarel_ has joined #openstack-infra | 04:20 | |
| *** psachin has quit IRC | 04:30 | |
| *** dave-mccowan has quit IRC | 04:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 04:35 |
| *** diablo_rojo has quit IRC | 04:40 | |
| *** lpetrut has joined #openstack-infra | 04:47 | |
| *** janki has joined #openstack-infra | 04:50 | |
| *** dpawlik has joined #openstack-infra | 04:52 | |
| *** dpawlik has quit IRC | 04:57 | |
| *** dpawlik has joined #openstack-infra | 05:04 | |
| *** dpawlik has quit IRC | 05:08 | |
| *** wolverineav has quit IRC | 05:08 | |
| *** roman_g has joined #openstack-infra | 05:09 | |
| *** slaweq has joined #openstack-infra | 05:10 | |
| *** wolverineav has joined #openstack-infra | 05:11 | |
| *** Lucas_Gray has quit IRC | 05:11 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 05:12 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: master-nameserver: Add unmanaged domains; add acme.opendev.org https://review.openstack.org/641155 | 05:14 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 05:14 |
| *** wolverineav has quit IRC | 05:17 | |
| *** lpetrut has quit IRC | 05:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 05:24 |
| *** lpetrut has joined #openstack-infra | 05:29 | |
| *** roman_g has quit IRC | 05:34 | |
| *** dpawlik has joined #openstack-infra | 05:42 | |
| *** lpetrut has quit IRC | 05:43 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Unmount internal mounts on finalise errors https://review.openstack.org/642953 | 05:44 |
| *** kjackal has quit IRC | 05:46 | |
| openstackgerrit | Merged openstack-infra/zuul master: Ensure correct lexical sorting of node requests https://review.openstack.org/643352 | 05:47 |
| openstackgerrit | Merged openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project https://review.openstack.org/636662 | 05:47 |
| *** dpawlik has quit IRC | 05:47 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add missing doc for 'execute_time' in MQTT schema https://review.openstack.org/643069 | 05:47 |
| *** slaweq has quit IRC | 05:56 | |
| *** mtreinish has quit IRC | 05:57 | |
| *** rcernin has quit IRC | 05:58 | |
| *** chandankumar has quit IRC | 05:58 | |
| *** rcernin has joined #openstack-infra | 06:13 | |
| *** yamamoto has quit IRC | 06:21 | |
| *** chandankumar has joined #openstack-infra | 06:22 | |
| *** e0ne has joined #openstack-infra | 06:24 | |
| *** e0ne has quit IRC | 06:27 | |
| *** yamamoto has joined #openstack-infra | 06:29 | |
| *** yamamoto has quit IRC | 06:30 | |
| *** mtreinish has joined #openstack-infra | 06:31 | |
| *** kjackal has joined #openstack-infra | 06:38 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 06:58 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Wait for command processor thread to stop https://review.openstack.org/641854 | 07:03 |
| *** kjackal has quit IRC | 07:14 | |
| *** stakeda has joined #openstack-infra | 07:14 | |
| *** kjackal has joined #openstack-infra | 07:15 | |
| *** kjackal has quit IRC | 07:15 | |
| *** dpawlik has joined #openstack-infra | 07:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove some thread join timeouts https://review.openstack.org/643517 | 07:24 |
| *** apetrich has joined #openstack-infra | 07:25 | |
| *** yolanda has joined #openstack-infra | 07:30 | |
| *** ricolin has joined #openstack-infra | 07:33 | |
| *** pcaruana has joined #openstack-infra | 07:36 | |
| *** ricolin_ has joined #openstack-infra | 07:38 | |
| *** ricolin has quit IRC | 07:38 | |
| *** priteau has joined #openstack-infra | 07:40 | |
| *** slaweq has joined #openstack-infra | 07:54 | |
| *** ginopc has joined #openstack-infra | 07:57 | |
| *** kopecmartin|off is now known as kopecmartin | 07:57 | |
| AJaeger | infra-root, all translation syncs fail, see http://logs.openstack.org/periodic/git.openstack.org/openstack/python-keystoneclient/master/propose-translation-update/8c7d3e8/job-output.txt.gz#_2019-03-15_07_56_38_362424 | 07:59 |
| *** rcernin has quit IRC | 07:59 | |
| AJaeger | Seems to be a fallout from switch to bionic | 07:59 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Manage ansible installations within zuul https://review.openstack.org/631930 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Validate ansible installations on startup https://review.openstack.org/637418 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Make ansible version configurable https://review.openstack.org/637422 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Install ansible during executor startup if needed https://review.openstack.org/640644 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Run tox remote concurrent https://review.openstack.org/640654 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support ansible 2.6 https://review.openstack.org/631931 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support ansible 2.7 https://review.openstack.org/631932 | 08:10 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Switch default ansible version to 2.7 https://review.openstack.org/637424 | 08:10 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Move the post and release jobs in-tree https://review.openstack.org/643524 | 08:10 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/project-config master: Remove unneeded storyboard-webclient jobs https://review.openstack.org/643525 | 08:10 |
| *** dpawlik has quit IRC | 08:10 | |
| *** jpena|off is now known as jpena | 08:11 | |
| *** xek has quit IRC | 08:13 | |
| *** xek has joined #openstack-infra | 08:13 | |
| *** geguileo has quit IRC | 08:15 | |
| *** rpittau|afk is now known as rpittau | 08:17 | |
| *** tkajinam has quit IRC | 08:18 | |
| *** gfidente has joined #openstack-infra | 08:20 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth https://review.openstack.org/619155 | 08:23 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Run tox-py35 jobs on xenial https://review.openstack.org/643530 | 08:23 |
| *** pcaruana has quit IRC | 08:24 | |
| *** pcaruana|afk| has joined #openstack-infra | 08:24 | |
| *** mcgiggler has joined #openstack-infra | 08:24 | |
| cmurphy | AJaeger: dmsimard I'll try building a new tumbleweed image with dib and try to see what's going on there | 08:25 |
| *** emine__ has joined #openstack-infra | 08:26 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Fix post and release jobs by setting some variables https://review.openstack.org/643531 | 08:26 |
| *** markvoelker has joined #openstack-infra | 08:27 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: wip: Spec for allowing circular dependencies https://review.openstack.org/643309 | 08:30 |
| cmurphy | $ grep ^NAME /mnt/etc/os-release | 08:33 |
| cmurphy | NAME="openSUSE Tumbleweed Kubic" | 08:33 |
| cmurphy | neat | 08:33 |
| AJaeger | cmurphy: fvogt is looking at it on #kubic as well, thanks | 08:33 |
| AJaeger | cmurphy: that name looks broken ;( | 08:33 |
| cmurphy | AJaeger: great | 08:34 |
| *** e0ne has joined #openstack-infra | 08:35 | |
| *** Guest17747 has joined #openstack-infra | 08:35 | |
| *** ykarel_ is now known as ykarel|lunch | 08:38 | |
| AJaeger | ianw: is Zanata dead? Even the 4.6.2 zanata-cli (most recent) will not run on Bionic, throws same error. | 08:39 |
| *** jtomasek has joined #openstack-infra | 08:41 | |
| *** wolverineav has joined #openstack-infra | 08:43 | |
| *** derekh has joined #openstack-infra | 08:47 | |
| *** ccamacho has joined #openstack-infra | 08:48 | |
| *** wolverineav has quit IRC | 08:48 | |
| *** ccamacho has quit IRC | 08:52 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Support common ansible dependencies https://review.openstack.org/643534 | 08:52 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] admin REST API: docker-compose PoC, frontend https://review.openstack.org/643536 | 08:59 |
| *** jpich has joined #openstack-infra | 08:59 | |
| *** yamamoto has joined #openstack-infra | 08:59 | |
| ianw | AJaeger: hrm, I haven't looked in ages, not sure sorry :/ | 08:59 |
| *** Guest17747 is now known as geguileo | 09:00 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: ansible-config: add common extra dependencies https://review.openstack.org/643537 | 09:01 |
| AJaeger | ianw: could you check whether we still have a contact to help fix Zanata-cli? It just does not run on Bionic | 09:01 |
| AJaeger | issue is too new java I guess | 09:01 |
| *** pgaxatte has joined #openstack-infra | 09:07 | |
| *** rfolco has quit IRC | 09:07 | |
| cmurphy | dmsimard: and anyone else following along https://build.opensuse.org/request/show/685275 will fix the tumbleweed problem | 09:10 |
| cmurphy | cc dirk ^ | 09:10 |
| *** hwoarang has quit IRC | 09:11 | |
| *** dpawlik has joined #openstack-infra | 09:13 | |
| dirk | cmurphy: thanks! | 09:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use xenial for translation jobs https://review.openstack.org/643539 | 09:16 |
| *** hwoarang has joined #openstack-infra | 09:16 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Fix post and release jobs by setting some variables https://review.openstack.org/643531 | 09:18 |
| *** lpetrut has joined #openstack-infra | 09:20 | |
| *** logan- has quit IRC | 09:23 | |
| *** logan- has joined #openstack-infra | 09:27 | |
| openstackgerrit | Gorka Eguileor proposed openstack-infra/project-config master: Add cinderlib publish-to-pypi job https://review.openstack.org/643542 | 09:38 |
| *** ykarel|lunch is now known as ykarel | 09:40 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support common ansible dependencies https://review.openstack.org/643534 | 09:42 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add missing docs for zuul-manage-ansible https://review.openstack.org/643544 | 09:42 |
| *** dpawlik has quit IRC | 09:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: ansible-config: add common extra dependencies https://review.openstack.org/643537 | 09:45 |
| *** dtantsur|afk is now known as dtantsur | 09:47 | |
| *** dpawlik has joined #openstack-infra | 09:48 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/project-config master: Remove unneeded storyboard-webclient post job https://review.openstack.org/643525 | 09:51 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/project-config master: Set variables for storyboard-webclient release job https://review.openstack.org/643545 | 09:51 |
| SotK | AJaeger: thanks for the heads up re: the release job | 09:51 |
| openstackgerrit | Bhawna Singh proposed openstack-infra/storyboard master: update StoriesController so users subsribe to stories they create https://review.openstack.org/643546 | 09:53 |
| AJaeger | SotK: you're welcome! | 09:53 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Move the post job in-tree https://review.openstack.org/643524 | 09:59 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Fix post job by setting some variables https://review.openstack.org/643531 | 09:59 |
| *** lpetrut has quit IRC | 10:02 | |
| *** ginopc has quit IRC | 10:02 | |
| *** ginopc has joined #openstack-infra | 10:03 | |
| *** rfolco has joined #openstack-infra | 10:05 | |
| *** stakeda has quit IRC | 10:06 | |
| *** dpawlik has quit IRC | 10:07 | |
| *** rfolco has quit IRC | 10:09 | |
| *** rfolco has joined #openstack-infra | 10:11 | |
| *** dtantsur is now known as dtantsur|brb | 10:15 | |
| *** yamamoto has quit IRC | 10:16 | |
| *** kaiokmo has quit IRC | 10:19 | |
| *** lpetrut has joined #openstack-infra | 10:27 | |
| *** dulek has quit IRC | 10:40 | |
| *** dulek has joined #openstack-infra | 10:42 | |
| *** wolverineav has joined #openstack-infra | 10:45 | |
| *** dpawlik has joined #openstack-infra | 10:45 | |
| *** ricolin_ has quit IRC | 10:47 | |
| *** wolverineav has quit IRC | 10:49 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Include enabled features in /v1/systeminfo output https://review.openstack.org/643558 | 10:51 |
| *** mriedem has joined #openstack-infra | 10:53 | |
| *** yamamoto has joined #openstack-infra | 10:56 | |
| *** udesale has quit IRC | 10:56 | |
| *** udesale has joined #openstack-infra | 10:57 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add a Swift storage backend implementation https://review.openstack.org/633365 | 10:58 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add an attachments endpoint to stories https://review.openstack.org/633366 | 10:58 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add a table for attachments https://review.openstack.org/633411 | 10:58 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add a WSME model for attachments https://review.openstack.org/633417 | 10:58 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add support for retrieving attachment metadata https://review.openstack.org/633418 | 10:58 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add support for creating attachments https://review.openstack.org/633421 | 10:58 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add attachments API availability to /v1/systeminfo https://review.openstack.org/643560 | 10:59 |
| *** e0ne has quit IRC | 11:03 | |
| *** yamamoto has quit IRC | 11:05 | |
| *** yamamoto has joined #openstack-infra | 11:06 | |
| *** ramishra has quit IRC | 11:20 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for tempest bug 1812036 https://review.openstack.org/638771 | 11:21 |
| openstack | bug 1812036 in tempest "Tests creating encrypted volume types can conflict" [Undecided,New] https://launchpad.net/bugs/1812036 | 11:21 |
| *** weshay is now known as weshay|rover | 11:31 | |
| *** rascasoft has quit IRC | 11:32 | |
| *** dpawlik has quit IRC | 11:33 | |
| *** rascasoft has joined #openstack-infra | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:37 | |
| *** tosky has joined #openstack-infra | 11:43 | |
| *** markmcd has joined #openstack-infra | 11:48 | |
| *** Lucas_Gray has joined #openstack-infra | 11:50 | |
| *** rh-jelabarre has joined #openstack-infra | 11:51 | |
| *** kgiusti has joined #openstack-infra | 11:54 | |
| *** dpawlik has joined #openstack-infra | 11:58 | |
| *** jpena is now known as jpena|lunch | 11:59 | |
| *** janki has quit IRC | 12:00 | |
| *** dtantsur|brb is now known as dtantsur | 12:01 | |
| ianw | AJaeger: ok, on my todo list :) | 12:07 |
| *** e0ne has joined #openstack-infra | 12:12 | |
| *** derekh has quit IRC | 12:14 | |
| *** FlorianFa has quit IRC | 12:14 | |
| *** panda is now known as panda|lunch | 12:15 | |
| *** jamesmcarthur has joined #openstack-infra | 12:20 | |
| *** markvoelker has quit IRC | 12:25 | |
| *** rlandy has joined #openstack-infra | 12:26 | |
| *** pcaruana has joined #openstack-infra | 12:26 | |
| *** pcaruana|afk| has quit IRC | 12:28 | |
| dtantsur | FYI the linters job seems to be broken now: http://paste.openstack.org/show/747855/ due to https://github.com/mtreinish/stestr/issues/235 | 12:29 |
| *** roman_g has joined #openstack-infra | 12:30 | |
| AJaeger | thanks, ianw - but for now enjoy your weekend! | 12:31 |
| *** dpawlik has quit IRC | 12:33 | |
| *** jamesmcarthur has quit IRC | 12:36 | |
| *** EmilienM is now known as EvilienM | 12:37 | |
| *** priteau has quit IRC | 12:38 | |
| *** yamamoto has quit IRC | 12:39 | |
| *** jamesmcarthur has joined #openstack-infra | 12:48 | |
| *** ramishra has joined #openstack-infra | 12:55 | |
| mordred | dtantsur: oh goodie | 12:59 |
| *** yamamoto has joined #openstack-infra | 13:00 | |
| mordred | dtantsur, AJaeger : upper-constraints has stestr pinned at 2.2.0, so the carnage should be at least somewhat constrained | 13:01 |
| *** dave-mccowan has joined #openstack-infra | 13:02 | |
| dtantsur | mordred: it seems that this job does not use upper-constraints though | 13:02 |
| mordred | dtantsur: yeah - typically only things released as part of openstack use it | 13:02 |
| mordred | dtantsur: so there will definitely be some carnage | 13:02 |
| dtantsur | carnage \o/ | 13:03 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Fix post job by setting some variables https://review.openstack.org/643531 | 13:04 |
| *** jpena|lunch is now known as jpena | 13:04 | |
| *** lseki has joined #openstack-infra | 13:11 | |
| *** eharney has quit IRC | 13:15 | |
| clarkb | mordred: I'm really not here yet but if you get a chance bridge.o.o:~clarkb/gitea_fixup.log shows gitea05 failing on 404 to openstack/openstacksdk/settings | 13:17 |
| clarkb | the other giteas ran to completion and are done with the launchpadding | 13:17 |
| clarkb | any idea why that would 404 on 05? | 13:18 |
| mordred | clarkb: uhm. no? | 13:18 |
| clarkb | I figure wecan rerun that playbool with a limit of gitea05 and seeif it is persistent | 13:18 |
| *** dpawlik has joined #openstack-infra | 13:20 | |
| *** sthussey has joined #openstack-infra | 13:31 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/zuul-jobs master: Fix exec quoting in release remove-gpgkey role https://review.openstack.org/643585 | 13:36 |
| *** pcaruana has quit IRC | 13:36 | |
| fungi | AJaeger: ianw: i wonder if zatana-cli has problems with the version of java shipped in bionic? | 13:39 |
| AJaeger | fungi: yes - see my email on openstack-infra. Java 11 fails, Java 8 is fine | 13:40 |
| AJaeger | googling for error shows that Java 9 changed it - and you can change your code that it works with both. | 13:41 |
| fungi | ahh, looks like it may just be a library they externalized and needs to be linked during build? | 13:41 |
| fungi | i haven't gotten to e-mail yet, i'll stop researching and avoid duplicating work you've already done. thanks! | 13:41 |
| stephenfin | mordred: Was there any particular reason you gave up on the idea of dox? | 13:41 |
| AJaeger | Let's see whether there's help from zanata folks - for now we could merge https://review.openstack.org/643539 | 13:42 |
| AJaeger | fungi: yes, that might be the case | 13:42 |
| AJaeger | fungi: sorry, should have followed up here with pointer to email | 13:42 |
| AJaeger | config-core, another review of https://review.openstack.org/643539 would be great to unbreak translation jobs, please | 13:48 |
| *** dpawlik has quit IRC | 13:52 | |
| *** dpawlik has joined #openstack-infra | 13:54 | |
| mordred | stephenfin: yeah- it started to get weird with base image proliferation | 13:55 |
| mordred | stephenfin: HOWEVER - there is a change in the ecosystem since then that I think would make it easier - and maybe make dox a thing we should return to ... | 13:55 |
| mordred | stephenfin: which is that now there are nice python base images | 13:55 |
| mordred | stephenfin: so if we just tied it to those - which have tags by version - I think we could do a nice job with it | 13:56 |
| mordred | stephenfin: any interest in poking at it? | 13:57 |
| *** e0ne has quit IRC | 13:58 | |
| stephenfin | mordred: Yup. I'm getting rather annoyed at my inability to run stable stuff on localhost because of changes in (non-Python) system packages. Containers sound like the obvious solution | 13:58 |
| *** eharney has joined #openstack-infra | 13:59 | |
| *** TheJulia is now known as needssleep | 13:59 | |
| *** rpittau is now known as elfosardo | 13:59 | |
| stephenfin | mordred: What I'd _really_ like is make containers another type of isolation in tox, alongside virtualenvs, but I think decoupling the two (tox and virtualenv) would be nigh on impossible :) dox it is | 13:59 |
| *** jamesmcarthur has quit IRC | 13:59 | |
| *** dtantsur is now known as creepy_owlet | 13:59 | |
| *** efried is now known as fried_rice | 14:00 | |
| clarkb | fwiw I think pypa will tell you to use virtualenvs in containers if pip installing | 14:02 |
| stephenfin | Really? Seems somewhat unnecessary | 14:03 |
| clarkb | ya because pip and system packages dont get along | 14:04 |
| clarkb | this is a big problem noe eith system paclagesusing distutils and pip refusing to install over them | 14:04 |
| *** iurygregory is now known as skolt | 14:04 | |
| stephenfin | Hmm, good point, that. I was envisioning using the '--user' flag to work around that | 14:05 |
| *** e0ne has joined #openstack-infra | 14:06 | |
| stephenfin | I'm sure there are multiple technical reasons why it can't be done, but being able to install multiple versions of a package side-by-side (like Ruby does?) sure would be nice | 14:06 |
| mordred | clarkb, stephenfin: right - but that's why I ike the python base images - they don't have python packages installed from distro packages | 14:06 |
| mordred | so FROM python:3.6 RUN pip install foo works great | 14:07 |
| clarkb | stephenfin: ya user install dir should usually work though there are gotchas with mixed python versions iirc | 14:07 |
| clarkb | mordred: ya | 14:08 |
| mordred | like - this is why I think returning to dox would totally work now that docker run python:3.6 is a thing | 14:08 |
| clarkb | my other issue with dox is I dont need root to run the tox targets today but dox needs root | 14:09 |
| clarkb | I'll just have to VM more | 14:09 |
| *** kjackal has joined #openstack-infra | 14:10 | |
| pabelanger | isn't podman to fix that (root required) | 14:10 |
| stephenfin | clarkb: Not sure if it's possible, but IMO dox should wrap tox rather than replace it. It should still be possible to use virtualenv (In my mind - I haven't studied the dox architecture closely enough yet) | 14:10 |
| clarkb | pabelanger: ya that is one potential fi | 14:11 |
| clarkb | er potential way to address that | 14:11 |
| fungi | AJaeger: similarly, another review of https://review.openstack.org/643585 should hopefully un-break release jobs | 14:12 |
| fungi | it's amusing what you can get away with dropping quotes around in yaml | 14:12 |
| *** pcaruana has joined #openstack-infra | 14:17 | |
| *** pcaruana|afk| has joined #openstack-infra | 14:17 | |
| *** pcaruana|afk| has quit IRC | 14:17 | |
| *** stephenfin is now known as finucannot | 14:19 | |
| *** jhesketh_ has quit IRC | 14:20 | |
| mordred | finucannot: it could use tox to run inside of the container- but you'd still need to run containers | 14:20 |
| mordred | however, as pabelanger mentiond, podman works fine as non-privileged | 14:20 |
| mordred | so it would be pretty easy to have dox execute either docker or podman depending on what it finds installed | 14:20 |
| mordred | I'd say basically what dox would want to do is create a container for the right python version, install bindep depends in it, then it could run tox | 14:21 |
| mordred | which would also be nice becuase it would tie in the bindep step too | 14:21 |
| *** cmoura has quit IRC | 14:23 | |
| zbr | clarkb: does this e-r makes sense or not? https://review.openstack.org/#/c/622210/ -- i still see matches. | 14:25 |
| AJaeger | fungi: it's remove extra chars day ;) | 14:26 |
| clarkb | zbr: logstash doesn't show any matches over the last 7 days | 14:28 |
| zbr | clarkb: for me it does show matches, lots. | 14:29 |
| clarkb | zbr: 'message:"[primary] Waiting for logger" AND tags:"console"' is my query | 14:29 |
| zbr | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%5Bprimary%5D%20Waiting%20on%20logger%5C%22%20AND%20tags%3A%5C%22console%5C%22 | 14:30 |
| openstackgerrit | Merged openstack-infra/project-config master: Use xenial for translation jobs https://review.openstack.org/643539 | 14:30 |
| finucannot | mordred: Hmm, the only issue with that is that the bindep stuff isn't versioned, so we'd end up having the same issues if something changed there | 14:30 |
| *** cmoura has joined #openstack-infra | 14:30 | |
| finucannot | mordred: Also, that assumes the Python containers are Debian or RHEL-based | 14:30 |
| clarkb | zbr: the change uses waiting for logger but your query is waiting on logger | 14:31 |
| clarkb | so we need to update the change to use 'on' not 'for' | 14:31 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Fix post job by setting some variables https://review.openstack.org/643531 | 14:31 |
| *** jamesmcarthur has joined #openstack-infra | 14:31 | |
| zbr | clarkb: ahh,.. damn, need to fix. query good by change wrong. | 14:31 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: Query: [primary] Waiting on logger https://review.openstack.org/622210 | 14:31 |
| fungi | finucannot: i think bindep has support for alpine now too | 14:32 |
| zbr | clarkb: thanks for spotting it! | 14:32 |
| fungi | finucannot: well, and gentoo, suse, et cetera | 14:32 |
| fungi | but alpine is probably more relevant to the current discussion | 14:33 |
| finucannot | fungi: Oh, good to know. We'd still need to make it versioned though, I suspect | 14:33 |
| zbr | re bindep.... lets replace it with an ansible playbook! ;) .... kinda joking. | 14:33 |
| clarkb | infra-root I have restarted the sync gitea playbook on gitea05 only using --limit localhost:gitea05.opendev.org | 14:33 |
| finucannot | seeing as the key issue I have at the moment is that I can't build mitaka-eol stuff on F29 because something SQL'y has broken | 14:33 |
| clarkb | we'll see if it fails on openstacksdk consistently | 14:34 |
| fungi | finucannot: versioned in what way? versioning the protocol for the package list? or versioning the list itself? or versioning the repository containing the list? | 14:34 |
| zbr | the reality is that bindep usage is limited by design, does not scale well to a big number of OS-es, versions,... but on the other hand is simple. | 14:34 |
| finucannot | ditto for the SQLite issue we fixed in...oslo.db (?) recently | 14:35 |
| finucannot | fungi: I'm not sure, tbh :\ | 14:35 |
| fungi | zbr: i expect that the bulk of what bindep does would still have to be turned into a library for an ansible module anyway... the ansible package management stuff doesn't really solve the problem of having a means to express differences in package naming across platforms | 14:35 |
| *** jtomasek has quit IRC | 14:36 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix exec quoting in release remove-gpgkey role https://review.openstack.org/643585 | 14:36 |
| clarkb | fungi: and when you use things like include tasks ansible explodes and stops running and returns 0 | 14:37 |
| fungi | finucannot: sounds like the problems you're describing have to do with not using a platform contemporary with the version of software you're trying to use. that's fairly out of scope for bindep. it exists to tell you for the current platform you're running what packages that platform provides which you should install. it doesn't really tell you what platform you should be running nor how to get | 14:37 |
| fungi | versions of packages not provided by your platform | 14:37 |
| clarkb | (our case of base.yaml playbook not running was directly related to trying to handle different package installs on different platforms) | 14:37 |
| finucannot | fungi: Yup, that's the issue exactly. I don't know if dox using the base Python containers would solve that | 14:37 |
| *** armax has joined #openstack-infra | 14:38 | |
| fungi | finucannot: yeah, whether the software/tests you want to run can work on a chosen platform at all is a separate concern | 14:38 |
| finucannot | Yup | 14:39 |
| zbr | fungi: shortly installing deps kinda needs "scripting", fully declarative like bindep is never going to cover fo all cases. within a playbooks I could add code/logic to cope with new cases, like installing a repository before installing packages, .... Anyway, i am not planning to work on this, just wanted to highlight limitations in bindep. | 14:39 |
| finucannot | I need to think about this far more | 14:39 |
| *** dpawlik has quit IRC | 14:39 | |
| clarkb | zbr: ya I think part of the issue there is bindep isn't meant to install anything | 14:40 |
| clarkb | bindeps job is informational only | 14:40 |
| clarkb | then other tools are expected to wrap it and do system level changes | 14:40 |
| *** raissa has joined #openstack-infra | 14:40 | |
| fungi | zbr: sure. also remember that some of the suggested uses of bindep stretch its intended use case a bit. the point of it was to provide a developer a list of packages they're missing in their system to have a viable development environment for a given project. its convenience as a tool we can use to obtain similar information from those projects to set up jobs in our ci system is more of a side-effect | 14:41 |
| finucannot | fungi: One other thing: there isn't a way to run an arbitrary zuul job on localhost without spinning up a full zuul deployment, is there? | 14:41 |
| finucannot | Given that zuul jobs do encode things like the environment they should be run in | 14:41 |
| openstackgerrit | Merged openstack-infra/puppet-diskimage_builder master: Vendor the vhd-util PPA's gpg key https://review.openstack.org/643465 | 14:41 |
| zbr | true, and I still find is extreamly useful | 14:41 |
| *** priteau has joined #openstack-infra | 14:41 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Load some config from /v1/systeminfo https://review.openstack.org/643598 | 14:41 |
| fungi | finucannot: funny you should ask, i'm just now composing a reply to the starlingx-discuss ml describing exactly this | 14:42 |
| fungi | finucannot: you can download the images we build and boot them in a virtual machine context, or run a script to build said images yourself | 14:42 |
| fungi | finucannot: the bigger challenge in exactly replicating a zuul job is in assembling the various inherited ansible playbooks to feed into ansible, and providing relevant variables they rely on | 14:43 |
| fungi | i know there are some folks who have been working on a tool to do that, but not sure how far along they've gotten | 14:43 |
| finucannot | fungi: I figured that would be the complicated piece, yes | 14:43 |
| dmsimard | fungi: https://review.openstack.org/#/q/topic:freeze_job | 14:44 |
| zbr | fungi: we are working on something that should do something similar, but not ready for primetime yet. | 14:44 |
| fungi | at present, firing up an aio zuul/nodepool demo container with the zuul quickstart documentation is probably easier than assembling and parsing distributed bits of zuul configuration by hand | 14:45 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Include enabled features in /v1/systeminfo output https://review.openstack.org/643558 | 14:45 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add a Swift storage backend implementation https://review.openstack.org/633365 | 14:45 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add an attachments endpoint to stories https://review.openstack.org/633366 | 14:45 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add a table for attachments https://review.openstack.org/633411 | 14:45 |
| clarkb | ime the thing that trips people up in jobs the most is the networking | 14:45 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add a WSME model for attachments https://review.openstack.org/633417 | 14:45 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add support for retrieving attachment metadata https://review.openstack.org/633418 | 14:45 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add support for creating attachments https://review.openstack.org/633421 | 14:45 |
| clarkb | and that is cloud region specific so even running a full local zuul won't help there | 14:46 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Add attachments API availability to /v1/systeminfo https://review.openstack.org/643560 | 14:46 |
| clarkb | for most everything else booting the vm image locally tends to be a good platform to run on then I just execute tox or copy the devstack rc and run devstack etc | 14:46 |
| *** jhesketh_ has joined #openstack-infra | 14:46 | |
| fungi | btw, where is our script for building your own nodepool image these days? | 14:47 |
| clarkb | fungi: project-config/tools/build-image.sh (the way to remember is it lives in the repo with our dib elements) | 14:48 |
| fungi | oh, right-o | 14:48 |
| *** nhicher has joined #openstack-infra | 14:49 | |
| *** jhesketh has quit IRC | 14:49 | |
| *** jhesketh has joined #openstack-infra | 14:49 | |
| clarkb | but ya ens3 vs eth0, single nic with public addr, vs single nic with public and private addr, vs two nics one for public one for private. IPv4 vs IPv6. and so on | 14:50 |
| clarkb | it is annoying that it is so variable but it is good sanity checking our software will bind to all the semi common network setups :) | 14:50 |
| *** jhesketh_ has quit IRC | 14:51 | |
| clarkb | fungi: logan- limestone is still disabled in nodepool. Do we think whatever was causing the network disruptions has passed? | 14:51 |
| clarkb | would be nice to turn that back on if we can | 14:51 |
| *** dpawlik has joined #openstack-infra | 14:51 | |
| * logan- is taking the opportunity to upgrade that cloud from Pike while it is disabled | 14:51 | |
| logan- | should be ready to turn back on soon, but not quite yet | 14:52 |
| clarkb | logan-: cool. No rush I just didn't want this to get forgotten | 14:52 |
| logan- | yup np, thanks for checking in | 14:53 |
| *** jhesketh has quit IRC | 14:54 | |
| clarkb | logan-: do you know what version you are upgrading to? | 14:54 |
| logan- | ideally going to get it to rocky | 14:54 |
| clarkb | nice | 14:54 |
| *** jaosorior has quit IRC | 14:56 | |
| corvus | clarkb, fungi, mordred, tobiash: and https://review.openstack.org/643108 should be easy to review after the other one :) | 15:01 |
| *** jhesketh has joined #openstack-infra | 15:03 | |
| *** jpich has quit IRC | 15:07 | |
| *** jpich has joined #openstack-infra | 15:08 | |
| fungi | corvus: starred in gertty. will look those over right after lunch | 15:11 |
| * fungi steps away for food but will return asap | 15:11 | |
| *** ykarel is now known as ykarel|away | 15:12 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a resource subcontroller for attachments https://review.openstack.org/633611 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Display download links for files attached to stories https://review.openstack.org/633612 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a directive for uploading files https://review.openstack.org/634602 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a service to handle uploading files to Swift https://review.openstack.org/634603 | 15:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a way to add attachments to a story https://review.openstack.org/633840 | 15:13 |
| openstackgerrit | Merged openstack-infra/zuul master: web: support more than one job in the filter https://review.openstack.org/643385 | 15:15 |
| *** jpich has quit IRC | 15:15 | |
| *** ykarel_ has joined #openstack-infra | 15:15 | |
| *** ykarel|away has quit IRC | 15:18 | |
| clarkb | As a heads up I have reapproved the puppet 4 upgrade for some servers | 15:18 |
| *** jpich has joined #openstack-infra | 15:18 | |
| clarkb | should merge today now that puppet testing issues are sorted | 15:18 |
| *** ykarel_ is now known as ykarel|away | 15:20 | |
| *** snierodz_ is now known as snierodz | 15:21 | |
| *** dpawlik has quit IRC | 15:22 | |
| *** Lucas_Gray has quit IRC | 15:25 | |
| *** Lucas_Gray has joined #openstack-infra | 15:26 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a role to mirror a git repository to a remote git server https://review.openstack.org/643394 | 15:36 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Do not merge: test zuul-jobs-upload-git-mirror job https://review.openstack.org/643437 | 15:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Document how to build a buildset registry https://review.openstack.org/643167 | 15:37 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Document the OpenDev docker image jobs https://review.openstack.org/643108 | 15:39 |
| clarkb | ok I think gitea05 is past the openstacksdk point | 15:42 |
| clarkb | mordred: what do you think about allowing up to 3 retries on those http tasks? | 15:42 |
| clarkb | seems like we are getting 1 failure out of 8* 2000 requests which isn't bad | 15:43 |
| clarkb | and maybe we just retry though | 15:43 |
| openstackgerrit | Merged openstack-infra/system-config master: Upgrade some servers to puppet 4 https://review.openstack.org/634726 | 15:51 |
| *** e0ne has quit IRC | 15:51 | |
| *** lpetrut has quit IRC | 15:56 | |
| *** lpetrut has joined #openstack-infra | 15:56 | |
| mordred | clarkb: ++ | 15:58 |
| *** udesale has quit IRC | 15:59 | |
| *** yamamoto has quit IRC | 16:00 | |
| *** yamamoto has joined #openstack-infra | 16:01 | |
| *** yamamoto has quit IRC | 16:01 | |
| *** yamamoto has joined #openstack-infra | 16:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use OpenDev icon in gitea https://review.openstack.org/643622 | 16:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Don't check out a commit if it is already checked out https://review.openstack.org/163922 | 16:04 |
| *** yamamoto has quit IRC | 16:07 | |
| *** pgaxatte has quit IRC | 16:08 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Upgrade to gitea 1.7.4 https://review.openstack.org/643623 | 16:09 |
| clarkb | gitea05 fixup completed successfully | 16:10 |
| clarkb | I think that means the launchpad vs storyboard task is complete and corrected | 16:10 |
| *** lpetrut has quit IRC | 16:13 | |
| *** mattw4 has joined #openstack-infra | 16:14 | |
| clarkb | I expect the puppet 4 runs will be at about 16:40 UTC | 16:15 |
| corvus | mordred: can you review 643623 and clarkb can you review 643622? :) | 16:16 |
| mordred | corvus: that was a very difficult review | 16:17 |
| corvus | mordred: i appreciate your thoroughness | 16:17 |
| corvus | mordred: to be fair, i haven't checked to see if there are template changes | 16:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Allow group read permissions on project keys directory https://review.openstack.org/642913 | 16:18 |
| mordred | corvus: I'm sort of assuming not in a point-release ... lemme go check real quick though | 16:18 |
| corvus | mordred: "git diff v1.7.1..v1.7.4 templates/" shows only changes to files we don't have copies of, so i think no changes are necessary? | 16:20 |
| mordred | yeah. I concur | 16:20 |
| mordred | corvus: there is one change to home.tmpl - but I don't think it's relevant for us? | 16:21 |
| clarkb | corvus: what is the gitea-safari svg for? | 16:21 |
| *** wolverineav has joined #openstack-infra | 16:21 | |
| mordred | oh - nevermiund | 16:21 |
| corvus | mordred: i see repo/home.tmpl | 16:21 |
| clarkb | (like why isn't that an opendev svg and why does safari needs its own file) | 16:21 |
| corvus | clarkb: it's a mask | 16:21 |
| mordred | corvus: yeah - I missed a path element with my eyeballs | 16:22 |
| corvus | clarkb: and i left it named gitea because it's not cached by the service worker and it's referenced in a file we don't already have a template for | 16:22 |
| *** kopecmartin is now known as kopecmartin|off | 16:22 | |
| mordred | clarkb: because steve jobs likes being different | 16:22 |
| clarkb | corvus: got it | 16:22 |
| *** gyee has joined #openstack-infra | 16:23 | |
| corvus | here's zuul's version: https://zuul-ci.org/safari-pinned-tab.svg | 16:23 |
| *** yamamoto has joined #openstack-infra | 16:23 | |
| clarkb | corvus: +2'd but didn't approve in case home.tmpl is important to us | 16:23 |
| corvus | clarkb: no, home.tmpl isn't changed, only repo/home.tmpl which we don't customize. we're gtg. | 16:24 |
| clarkb | k approving then | 16:24 |
| *** woojay has quit IRC | 16:24 | |
| *** kmalloc is now known as needscoffee | 16:25 | |
| *** wolverineav has quit IRC | 16:29 | |
| *** wolverineav has joined #openstack-infra | 16:30 | |
| *** roman_g has quit IRC | 16:30 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Document how to build a buildset registry https://review.openstack.org/643167 | 16:32 |
| *** Vadmacs has joined #openstack-infra | 16:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Don't check out a commit if it is already checked out https://review.openstack.org/163922 | 16:37 |
| *** Vadmacs has quit IRC | 16:42 | |
| *** Vadmacs has joined #openstack-infra | 16:42 | |
| *** diablo_rojo has joined #openstack-infra | 16:44 | |
| clarkb | puppet 4 applied mostly cleanly. eavesdrop failed on an undefined nilclass error I'll look into and firehose hiera complains about undefined variable group which I'll also look at | 16:48 |
| *** ramishra has quit IRC | 16:50 | |
| clarkb | Mar 15 16:41:48 eavesdrop01 puppet-user[23622]: Failed to apply catalog: undefined method `[]' for nil:NilClass | 16:51 |
| clarkb | cmurphy: ^ any ideas? Unfortunately not a ton to go off of | 16:51 |
| clarkb | might need to run puppet manually with the --trace flag | 16:53 |
| *** ykarel|away has quit IRC | 16:55 | |
| *** ykarel|away has joined #openstack-infra | 16:55 | |
| clarkb | the firehose issue is a non issue. We don't use the hiera group data for that host | 16:57 |
| clarkb | (we might want to, but don't currently) | 16:58 |
| *** rlandy is now known as rlandy|biab | 17:00 | |
| *** jamesmcarthur has quit IRC | 17:00 | |
| *** ykarel|away has quit IRC | 17:00 | |
| *** jamesmcarthur has joined #openstack-infra | 17:01 | |
| *** mcgiggler has quit IRC | 17:01 | |
| clarkb | infra-root I think `/opt/puppetlabs/bin/puppet apply --trace /opt/system-config/production/manifests/site.pp` is the command to run to get the trace output? does that look wrong to anyone? | 17:05 |
| clarkb | cmurphy: ^ you migth ahve input too | 17:05 |
| *** ykarel|away has joined #openstack-infra | 17:08 | |
| *** elfosardo is now known as rpittau|afk | 17:08 | |
| fungi | okay, lunch took longer than anticipated but getting back to reviews now | 17:09 |
| *** skolt is now known as iurygregory | 17:09 | |
| zbr | clarkb: i randomly came acrosshttps://review.openstack.org/#/c/613999/ and I wonder what are the plans: should we do some minor bumping or delay any change until the multi version support? and somehow I have the impression that ansible 2.5 will be out of support before we have multi version support in zuul anyway. | 17:10 |
| *** e0ne has joined #openstack-infra | 17:10 | |
| clarkb | I wonder if I have to set a module path on that command | 17:10 |
| *** ldnunes has quit IRC | 17:11 | |
| *** e0ne has quit IRC | 17:11 | |
| *** Lucas_Gray has quit IRC | 17:13 | |
| clarkb | zbr probably a better question for #zuul. I personally dont think zuul needs to change anything there unless zuul itself is incompatible with a specific version | 17:14 |
| clarkb | the range allows you to install any 2 | 17:14 |
| clarkb | er any 2.5 | 17:14 |
| *** ginopc has quit IRC | 17:15 | |
| *** wolverineav has quit IRC | 17:18 | |
| zbr | clarkb: valid point, tx. | 17:19 |
| *** jpena is now known as jpena|off | 17:21 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/gertty master: Introduce security checks with bandit and fix it https://review.openstack.org/613371 | 17:21 |
| *** rpioso is now known as rpioso|afk | 17:22 | |
| *** creepy_owlet is now known as dtantsur|afk | 17:22 | |
| zbr | i wonder if gertty having only a noop job is by design or an accident | 17:25 |
| *** jpich has quit IRC | 17:26 | |
| *** priteau has quit IRC | 17:30 | |
| *** wolverineav has joined #openstack-infra | 17:30 | |
| *** rfolco is now known as rfolco|doctor | 17:31 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/gertty master: Adds linting zuul job https://review.openstack.org/581871 | 17:31 |
| *** wolverineav has quit IRC | 17:32 | |
| *** wolverineav has joined #openstack-infra | 17:33 | |
| *** kjackal has quit IRC | 17:40 | |
| *** kjackal has joined #openstack-infra | 17:40 | |
| *** wolverineav has quit IRC | 17:41 | |
| zbr | clarkb: corvus : can you help me with https://review.openstack.org/#/c/581871/ ? adds first job for gerrty | 17:43 |
| *** wolverineav has joined #openstack-infra | 17:44 | |
| *** ykarel|away has quit IRC | 17:47 | |
| *** wolverineav has quit IRC | 17:48 | |
| *** wolverineav has joined #openstack-infra | 17:49 | |
| *** rlandy|biab is now known as rlandy | 17:51 | |
| mordred | zbr: also - I believe we'll be landing multi-ansible very soon | 17:51 |
| corvus | zbr: one suggestion on the gertty change | 17:52 |
| zbr | sure doing it now | 17:52 |
| corvus | i expect multi-ansible to land next week... if not today :) | 17:53 |
| zbr | corvus: this is cool! what would be implicit version? a latest one? or the oldest one? easy to switch them? | 17:54 |
| corvus | zbr: latest will be default, easy to change the tenant or job default | 17:55 |
| zbr | corvus: glad to hear that. i also prefer the same. | 17:55 |
| openstackgerrit | Brian Haley proposed openstack-infra/devstack-gate master: Support an IPv6 underlay network https://review.openstack.org/343041 | 17:58 |
| openstackgerrit | Sorin Sbarnea proposed openstack/gertty master: Adds linting zuul job https://review.openstack.org/581871 | 18:00 |
| *** priteau has joined #openstack-infra | 18:01 | |
| fungi | corvus: once we opendevify, are you contemplating moving presentty to join gertty and boartty? | 18:09 |
| fungi | (not that i mind e-mailing you patches for it) | 18:11 |
| corvus | fungi: that's a good question i haven't thought about. i will do so. :) | 18:13 |
| fungi | also your ever-expanding tty-oriented productivity suite deserves a catchier name for its org | 18:13 |
| mordred | fungi, corvus: tty/gertty ? | 18:14 |
| fungi | it could be the ttyping-pool though that's sorta wordy | 18:14 |
| mordred | (that's a terrible name - don't listen to me) | 18:14 |
| corvus | that's also an interesting idea :) | 18:15 |
| corvus | i will subscribe to both of your newsletters! | 18:15 |
| fungi | i reproduce my newsletter on a restored memeograph machine and drive three towns away to mail it from multiple postboxes | 18:17 |
| clarkb | When I get back to computer (currently dadops) I'll run that puppet apply command in scrollback on eavesdrop to try and get more debugging frlm it | 18:19 |
| *** jamesmcarthur has quit IRC | 18:20 | |
| corvus | fungi: electricity? fancy! i printed on a pedal-powered c&p last weekend https://en.wikipedia.org/wiki/Chandler_%26_Price | 18:20 |
| zbr | corvus: done with https://review.openstack.org/#/c/581871/ on gertty -- which i find hard to pronounce but very useful at the same time. | 18:20 |
| fungi | corvus: nice! and yes, i do have electricity | 18:21 |
| fungi | i'm going to imagine that the photos with that wikipedia article were taken in your boiler room | 18:22 |
| clarkb | I watching a thing on the old nytimes (I think it was nytimes) lead block printing system | 18:24 |
| clarkb | it printed lead blocks based on piut which then printed the paper | 18:25 |
| clarkb | mostly I thought it was cool to print metal | 18:25 |
| fungi | i used to pick up old lead type from random places (thrift stores, et cetera) and make stuff out of it | 18:26 |
| fungi | aside from the occasional inked stamp though, i didn't use any of it for actual printing | 18:26 |
| *** rpioso|afk is now known as rpioso | 18:27 | |
| *** eharney has quit IRC | 18:38 | |
| *** wolverineav has quit IRC | 18:42 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add job and integration tests for the upload-git-mirror role https://review.openstack.org/643435 | 18:42 |
| *** wolverineav has joined #openstack-infra | 18:47 | |
| *** wolverineav has quit IRC | 18:52 | |
| *** e0ne has joined #openstack-infra | 18:53 | |
| *** tosky has quit IRC | 18:53 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Support logical AND searches https://review.openstack.org/643224 | 18:57 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Install virtualenv from source https://review.openstack.org/643655 | 19:00 |
| openstackgerrit | Ankita Bansal proposed openstack-infra/storyboard-webclient master: Login redirect replaces page history https://review.openstack.org/642755 | 19:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test venv instead of virtualenv https://review.openstack.org/643658 | 19:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test venv instead of virtualenv https://review.openstack.org/643658 | 19:11 |
| *** priteau has quit IRC | 19:13 | |
| clarkb | the eavesdrop issues appears to be caused by pip --version complaining about old slow cryptography on stderr whcih the puppet pip module tries to process for a version :( | 19:14 |
| clarkb | I'm going to upgrade cryptography to see if that fixes it | 19:14 |
| fungi | upgraded cryptography fixes everything (including your backdoors) | 19:15 |
| clarkb | this is annoying. cryptography currently comes from system packages | 19:16 |
| clarkb | pip is latest pip | 19:16 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for nova functional test race bug 1820337 https://review.openstack.org/643661 | 19:16 |
| clarkb | it doesn't update cryptography but will warn you on every invocation of the tool | 19:16 |
| openstack | bug 1820337 in OpenStack Compute (nova) "test_bfv_quota_race_local_delete intermittently fails with testtools.matchers._impl.MismatchError: ['bfv'] != []" [High,Confirmed] https://launchpad.net/bugs/1820337 | 19:16 |
| clarkb | why not just update cryptography when we are updating pip pip | 19:17 |
| *** stevebaker has quit IRC | 19:17 | |
| *** gouthamr has quit IRC | 19:17 | |
| *** dmellado has quit IRC | 19:17 | |
| *** dave-mccowan has quit IRC | 19:18 | |
| clarkb | hrm is cryptography 2.x backward compatible to 1.x? | 19:18 |
| clarkb | makes me wonder if I'll break supybot if I do that update | 19:19 |
| *** raissa has quit IRC | 19:19 | |
| clarkb | cryptography 2.0 dropped support for python3.3 | 19:20 |
| clarkb | that was the backward incompatible change | 19:20 |
| clarkb | https://tickets.puppetlabs.com/browse/PUP-8986 and there is the upstream python bug | 19:20 |
| clarkb | er puppet bug | 19:20 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: allow subscriptions to projects when items in project groups list are expanded https://review.openstack.org/642371 | 19:21 |
| *** raissa has joined #openstack-infra | 19:21 | |
| *** raissa has quit IRC | 19:21 | |
| *** wolverineav has joined #openstack-infra | 19:22 | |
| *** raissa has joined #openstack-infra | 19:22 | |
| *** raissa has quit IRC | 19:23 | |
| clarkb | confirmed that supybot pulls in cryptography | 19:23 |
| openstackgerrit | Ankita Bansal proposed openstack-infra/storyboard-webclient master: Provide -Assign to self- button for story tasks. https://review.openstack.org/643427 | 19:24 |
| *** raissa has joined #openstack-infra | 19:24 | |
| clarkb | infra-root given the above any objections to upgrading cryptography then restarting meetbot to ensure it is still happy? from what I can tell the backward incompatible chagne is python3.3 specific | 19:25 |
| mordred | clarkb: fine by me | 19:26 |
| *** wolverineav has quit IRC | 19:27 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Fix parent if no virtualenv exists in system https://review.openstack.org/643665 | 19:27 |
| clarkb | I'm double checking the meeting calendar now | 19:27 |
| clarkb | the latest friday meeting appears to be at 1500 UTC so I should be good to do that restart | 19:29 |
| openstackgerrit | Ankita Bansal proposed openstack-infra/storyboard-webclient master: Login redirect replaces page history https://review.openstack.org/642755 | 19:29 |
| clarkb | doing this now | 19:29 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Refactor code for unit testing https://review.openstack.org/643666 | 19:30 |
| clarkb | AttributeError: 'module' object has no attribute 'SSL_ST_INIT' | 19:30 |
| clarkb | that failed spectacularly (this is running pip --version) | 19:31 |
| fungi | clarkb: latest pip only upgrades packages when the installed versions are no longer sufficient to meet the requirements specified by whatever you're upgrading, but you *can* change the upgrade strategy explicitly if you prefer to upgrade to latest of all dependencies | 19:31 |
| *** jamesmcarthur has joined #openstack-infra | 19:31 | |
| clarkb | fungi: ya I know I'm complaining that they print a warning every time you use pip rather than just fixing the problem | 19:31 |
| fungi | ahh, because it's a dependency of pip itself | 19:31 |
| fungi | got it | 19:31 |
| clarkb | so I think the current issue is its still using system pyopenssl which breaks spectacularly with pypi cryptography installed | 19:32 |
| *** dave-mccowan has joined #openstack-infra | 19:32 | |
| fungi | i mean, i rather like that pip no longer pulls down new packages from pypi when you're using system packages of those modules which meet the required dependency version ranges | 19:32 |
| clarkb | so I'm going to aptget install python-cryptography again and then see if that makes it happy | 19:32 |
| clarkb | fungi: yup I'm saying they should raise their declared dep version | 19:32 |
| *** eharney has joined #openstack-infra | 19:33 | |
| clarkb | rather than annoying users about something they can fix directly | 19:33 |
| fungi | fair | 19:33 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Refactor code for unit testing https://review.openstack.org/643666 | 19:34 |
| *** e0ne has quit IRC | 19:37 | |
| clarkb | if anyone else runs into the pip pyopenssl issues I had to easy_install -U pyopenssl to fix | 19:39 |
| clarkb | possible that a get-pip.py install would've fixed too | 19:39 |
| clarkb | I'm going to attempt a meetbot restart now | 19:39 |
| *** openstack has joined #openstack-infra | 19:42 | |
| *** ChanServ sets mode: +o openstack | 19:42 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for nova functional test race bug 1820337 https://review.openstack.org/643661 | 19:42 |
| clarkb | Seems like that worked and the warning has gone away | 19:42 |
| clarkb | next to see if that fixes puppet | 19:42 |
| clarkb | it did fix puppet | 19:43 |
| clarkb | so all of the puppet4 hosts are happy now | 19:43 |
| *** dave-mccowan has quit IRC | 19:43 | |
| clarkb | #status log Upgraded pyopenssl and cryptography on eavesdrop.openstack.org to work around https://tickets.puppetlabs.com/browse/PUP-8986 after the puppet 4 upgrade on this host | 19:43 |
| openstackstatus | clarkb: finished logging | 19:43 |
| clarkb | We may want to manage that more directly in general (like always have up to date pip and cryptography and pyopenssl?) | 19:45 |
| openstack | bug 1820337 in OpenStack Compute (nova) "test_bfv_quota_race_local_delete intermittently fails with testtools.matchers._impl.MismatchError: ['bfv'] != []" [High,Confirmed] https://launchpad.net/bugs/1820337 | 19:46 |
| openstackgerrit | Merged opendev/base-jobs master: Document the OpenDev docker image jobs https://review.openstack.org/643108 | 19:48 |
| *** wolverineav has joined #openstack-infra | 19:50 | |
| clarkb | mordred: corvus https://review.openstack.org/#/c/643622/1 that failed in the docker image build post run which is what has prevented our update of the gitea images | 19:51 |
| clarkb | I'm looking for executor logs now | 19:51 |
| clarkb | it ran on ze01 I'm not actually seeing anything there that owuld indicate the problem | 19:52 |
| *** gouthamr has joined #openstack-infra | 19:53 | |
| mordred | clarkb: boo | 19:53 |
| clarkb | oh it timed out | 19:54 |
| clarkb | http://logs.openstack.org/22/643622/1/check/system-config-build-image-gitea/32fae01/job-output.txt.gz#_2019-03-15_16_23_21_512198 | 19:55 |
| clarkb | I'll try a recheck looks like a push from one cloud to another and maybe we just got unluck with the internets | 19:55 |
| mordred | yeah | 19:55 |
| mordred | ++ | 19:55 |
| clarkb | and now time to add retries to the gitea http post tasks before I forget | 19:56 |
| *** wolverineav has quit IRC | 19:57 | |
| *** pcaruana has quit IRC | 19:59 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Install virtualenv from source https://review.openstack.org/643655 | 19:59 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-preview master: WIP: Refactor code for unit testing https://review.openstack.org/643666 | 20:00 |
| *** armax has quit IRC | 20:01 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Run tox-py35 jobs on xenial https://review.openstack.org/643530 | 20:01 |
| fungi | and now time to file my expense report before i forget | 20:02 |
| *** armax has joined #openstack-infra | 20:02 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Retry gitea repo setting HTTP POSTs https://review.openstack.org/643671 | 20:04 |
| clarkb | mordred: ^ I think I got that right | 20:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Use base_prefix instead of exec_prefix to find virtualenv https://review.openstack.org/643672 | 20:04 |
| mordred | clarkb: seems reasonable | 20:04 |
| *** wolverineav has joined #openstack-infra | 20:06 | |
| *** jamesmcarthur has quit IRC | 20:09 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Run logstash-filters job on Xenial https://review.openstack.org/643674 | 20:12 |
| clarkb | config-core ^ anothre bionic/xenial cleanup | 20:12 |
| *** emine__ has quit IRC | 20:14 | |
| clarkb | Its looking like the gitea upload is slow again | 20:15 |
| clarkb | so there may be something systemic there | 20:15 |
| fungi | checked system resource metrics on the servers? | 20:16 |
| fungi | also, what is "the gitea upload"? | 20:16 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Use base_prefix instead of exec_prefix to find virtualenv https://review.openstack.org/643672 | 20:17 |
| clarkb | fungi: the gitea docker build job for https://review.openstack.org/643622 uploads the image to the intermediate registry | 20:17 |
| clarkb | fungi: that upload to the intermediate registry is failing. We did just rekey that server maybe one of the jobs didn't get a secret update or the secret update has a newline? | 20:17 |
| fungi | oh, got it | 20:17 |
| fungi | "slow" as in hasn't finished yet and might eventually fail? | 20:18 |
| clarkb | ya the first time it ran the job timed out on it | 20:18 |
| fungi | okay, consistently failing after taking a long time might be related to the rekeying. taking a long time and eventually succeeding wouldn't make much sense for that case | 20:19 |
| clarkb | I'm going to pop out for a bit and grab lunch. Short of disabling the no log again I'm not sure what the next debug step will be | 20:20 |
| clarkb | perhaps ssh into the job while it is waiting and inspect the docker daemon logs via journalctl? | 20:20 |
| *** wolverineav has quit IRC | 20:20 | |
| *** wolverineav has joined #openstack-infra | 20:21 | |
| openstackgerrit | Merged openstack-infra/zuul master: Raise an error if pipeline is defined twice https://review.openstack.org/632550 | 20:21 |
| *** jamesmcarthur has joined #openstack-infra | 20:24 | |
| *** wolverineav has quit IRC | 20:25 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Install virtualenv from source https://review.openstack.org/643655 | 20:25 |
| *** dmellado has joined #openstack-infra | 20:29 | |
| *** gouthamr_ has joined #openstack-infra | 20:29 | |
| *** Vadmacs has quit IRC | 20:31 | |
| *** raissa has joined #openstack-infra | 20:33 | |
| *** stevebaker has joined #openstack-infra | 20:41 | |
| *** whoami-rajat has quit IRC | 20:46 | |
| *** bdodd has quit IRC | 20:52 | |
| *** bdodd has joined #openstack-infra | 20:52 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test simpler way to find virtualenv https://review.openstack.org/643679 | 20:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Test simpler way to find virtualenv https://review.openstack.org/643679 | 20:55 |
| *** gfidente has quit IRC | 20:55 | |
| *** jamesmcarthur has quit IRC | 20:59 | |
| *** raissa has quit IRC | 21:07 | |
| *** rlandy has quit IRC | 21:12 | |
| *** rfolco|doctor has quit IRC | 21:15 | |
| *** kgiusti has left #openstack-infra | 21:17 | |
| cmurphy | clarkb: sorry I was afk, things are better now? | 21:22 |
| *** gouthamr has quit IRC | 21:22 | |
| *** jamesmcarthur has joined #openstack-infra | 21:22 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Show all stories created and allows them to filter according to status https://review.openstack.org/642370 | 21:24 |
| *** jamesmcarthur has quit IRC | 21:27 | |
| *** raissa has joined #openstack-infra | 21:31 | |
| clarkb | cmurphy: yup things are fine | 21:33 |
| clarkb | cmurphy: https://tickets.puppetlabs.com/browse/PUP-8986 is the puppet issue that I worked around by upgrading cryptography | 21:33 |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Support logical AND searches https://review.openstack.org/643224 | 21:33 |
| clarkb | the last successful system-config build image gitea job ran on xenial but we are running on bionic now | 21:35 |
| clarkb | likely that and the secrets thing are the two things to follow up on (I'm looking at job configs now to see if the secrets look blatantly wrong) | 21:36 |
| clarkb | the updated secret is in opendev/base-jobs | 21:42 |
| clarkb | now to see if those credentials look correct | 21:42 |
| *** raissa has quit IRC | 21:48 | |
| *** woojay has joined #openstack-infra | 21:48 | |
| *** hwoarang has quit IRC | 21:49 | |
| *** hwoarang has joined #openstack-infra | 21:50 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Show all stories created and allows them to filter according to status https://review.openstack.org/642370 | 21:56 |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Support logical AND searches https://review.openstack.org/643224 | 22:02 |
| notmyname | is there a way to link to a comment in gerrit? | 22:04 |
| notmyname | I mean, an inline code review comment | 22:04 |
| clarkb | notmyname: I don't think so. You can link to the line number but its a really subtle thing not directly supported for the ui (you ahve to know what the url suffix is iirc) | 22:06 |
| notmyname | kk | 22:06 |
| clarkb | infra-root the base64 decoded docker password from ~zuul/.docker/config.json does not seem to match the password for the registry in hiera/ansible vars | 22:07 |
| clarkb | corvus: ^ is it possible those got mixed up when recorded in the different locations? | 22:08 |
| *** mattw4 has quit IRC | 22:08 | |
| *** mattw4 has joined #openstack-infra | 22:08 | |
| clarkb | also they are very different so I don't suspect a typo | 22:09 |
| clarkb | (entirely possible I'm missing something here though | 22:09 |
| clarkb | fungi: where is your tool to decrypt zuul secrets? | 22:13 |
| clarkb | https://review.openstack.org/#/c/637969/ found that thanks | 22:13 |
| fungi | clarkb: i didn't have a tool per se, just documented how to do it with openssl in https://docs.openstack.org/infra/system-config/zuul.html#secrets | 22:17 |
| clarkb | the secret matches hiera | 22:18 |
| clarkb | so now to figure out why the written config in the job doesn't seem to match the secret | 22:18 |
| fungi | clarkb: for an actual useful tool, you might want to review https://review.openstack.org/639771 | 22:19 |
| fungi | it's usable now, just has some code style failures reported | 22:19 |
| clarkb | ok I think the config I'm seeing is for the local registry | 22:20 |
| clarkb | ya we specify the auth to the intermediate registry on the command line | 22:23 |
| *** jamesmcarthur has joined #openstack-infra | 22:24 | |
| *** jamesmcarthur has quit IRC | 22:24 | |
| clarkb | we started the skopeo process at ~21:56UTC at 21:59 I ran a ps -elf that does not show a skopeo process | 22:28 |
| *** kjackal has quit IRC | 22:32 | |
| clarkb | that step is actually running on the executor | 22:32 |
| clarkb | given that this step runs on the executor I don't think the switch to bionic is involved. (also the registry runs in a conatiner so should be pretty consistent as a process regardless of platform) | 22:38 |
| *** mattw4 has quit IRC | 22:48 | |
| clarkb | arg I rechecked again to watch it on the executor. I'm watching it and tailing logs and .... the job succeeds | 22:49 |
| clarkb | I mean its cool that the job decided to work | 22:49 |
| *** lseki has quit IRC | 22:49 | |
| clarkb | but also I have no clue what is going on there. | 22:49 |
| *** markvoelker has joined #openstack-infra | 22:51 | |
| clarkb | fungi: mordred: if still around care to review https://review.openstack.org/#/c/643674/ | 22:57 |
| *** anteaya has quit IRC | 23:00 | |
| clarkb | best I cant tell all the pieces are working as designed. The decrypted secrets lgtm. Our registry job is paused and running. The image build and test run and registry jobs are all in the same region | 23:03 |
| clarkb | I think I've ruled out bionic being a delta since skopeo runs on the executor | 23:03 |
| clarkb | I think that leaves me with the executor acting as a proxy for the transfer being potentially quite slow | 23:04 |
| clarkb | either because the executor itself has limited bw caring for all the jobs or because the intermediate registry being on one side of the planet and the buildset-registry/build/test jobs being on another leads to slow transfers | 23:04 |
| fungi | one of the reasons we've talked about the possibility of colocating executors in each provider in the past | 23:05 |
| fungi | though that would be a significant architectural shift | 23:06 |
| *** rascasoft has quit IRC | 23:07 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Run logstash-filters job on Xenial https://review.openstack.org/643674 | 23:08 |
| *** mriedem has quit IRC | 23:13 | |
| *** armax has quit IRC | 23:20 | |
| clarkb | ok the gate gitea build job is running in ovh gra1 | 23:21 |
| clarkb | I see the skopeo process running on ze07 | 23:21 |
| clarkb | strace shows it blocking on a futex | 23:22 |
| *** armax has joined #openstack-infra | 23:22 | |
| fungi | long-haul latency seems like a possible explanation | 23:23 |
| clarkb | ya lots of polling syscalls (I see pselect, epoll, and clock_gettime) | 23:24 |
| clarkb | hrm netstat on the buildset registry doesn't show a connection from the executor or to the intermediate registry | 23:25 |
| clarkb | I would've expected the connection from the executor | 23:25 |
| clarkb | on the executor I don't see any connections to those ports either | 23:28 |
| *** armax has quit IRC | 23:32 | |
| clarkb | I can connect to port 5000 on the buildest registry from the executor | 23:32 |
| clarkb | so not a firewall issue (fwiw we don't seem to explicitly open port 5000 so docker must do that) | 23:32 |
| *** armax has joined #openstack-infra | 23:33 | |
| fungi | did a presumed continuing connection from the executor to the buildset registry get silently killed in one direction, so the executor assumes it's still in progress but no longer getting responses? | 23:34 |
| clarkb | I don't think so as I don't see the connection on either end | 23:35 |
| fungi | oh | 23:35 |
| fungi | nm! | 23:35 |
| clarkb | its like skopeo is just sitting there doing nothing | 23:36 |
| clarkb | I wonder if I'm not seeing it due to network namepsace mangling by bwrap | 23:37 |
| clarkb | though I'd still expect to see the syscalls in strace | 23:37 |
| clarkb | if I'm reading procfs right the skopeo process(es) are part of the root net namespace | 23:44 |
| clarkb | reading the docker registry logs on both sides of the skopeo command I see a bunch of activity right at the start of when skopeo runs | 23:50 |
| clarkb | then nothing after that which probably explains why I don't see open connections later | 23:50 |
| clarkb | so something is happenign such that skopeo doesn't exit after the activity | 23:50 |
| clarkb | ya I see skopeo running head against a blob id, getting a 404, it then tries again, then decides ok I have to uplaod and then does an upload | 23:54 |
| clarkb | however they all appaer to be 0 bytes | 23:55 |
| clarkb | also it does a POST then a PATCH then a PUT? | 23:55 |
| clarkb | and I think that is about where i'm going to run out of time debugging this for now | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
