| *** Douhet has joined #openstack-infra | 00:02 | |
| *** yamahata has quit IRC | 00:06 | |
| *** cody-somerville has quit IRC | 00:12 | |
| *** thorst has joined #openstack-infra | 00:13 | |
| *** armax has quit IRC | 00:13 | |
| *** thorst has quit IRC | 00:13 | |
| *** cody-somerville has joined #openstack-infra | 00:17 | |
| *** cody-somerville has joined #openstack-infra | 00:17 | |
| *** Shrews has quit IRC | 00:18 | |
| *** yamahata has joined #openstack-infra | 00:19 | |
| *** bobh has joined #openstack-infra | 00:30 | |
| *** Shrews has joined #openstack-infra | 00:32 | |
| *** bobh has quit IRC | 00:35 | |
| *** smatzek has quit IRC | 00:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: legacy-trove-functional-dsvm-mysql requires trove-integration https://review.openstack.org/512064 | 00:45 |
|---|---|---|
| pabelanger | mordred: something does look right on trove legacy jobs, I think we might have duplicated them | 00:47 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n14131 - legacy-trove-functional-dsvm-mysql | 00:47 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-jobs.yaml#n14145 - legacy-trove-legacy-functional-dsvm-mysql | 00:47 |
| pabelanger | will have to confirm with trove PTL in the morning | 00:49 |
| clarkb | pabelanger: aiui they have their legacy jobs for older branches and non legacy for master | 00:50 |
| clarkb | and that is independent of zuulv3 migration | 00:50 |
| *** Shrews has quit IRC | 00:51 | |
| pabelanger | clarkb: oh, how confusing | 00:51 |
| *** Shrews has joined #openstack-infra | 00:57 | |
| *** yamamoto has joined #openstack-infra | 00:58 | |
| *** yamamoto has quit IRC | 00:59 | |
| *** yamamoto has joined #openstack-infra | 01:01 | |
| *** owalsh_ has joined #openstack-infra | 01:01 | |
| *** owalsh has quit IRC | 01:04 | |
| *** yamamoto has quit IRC | 01:12 | |
| *** smatzek has joined #openstack-infra | 01:14 | |
| *** yamamoto has joined #openstack-infra | 01:20 | |
| *** gouthamr has quit IRC | 01:20 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add horizon templates https://review.openstack.org/512013 | 01:22 |
| *** yamamoto has quit IRC | 01:25 | |
| *** yamamoto has joined #openstack-infra | 01:30 | |
| *** bobh has joined #openstack-infra | 01:31 | |
| *** yamamoto has quit IRC | 01:35 | |
| *** bobh has quit IRC | 01:35 | |
| *** thorst has joined #openstack-infra | 01:59 | |
| *** thorst has quit IRC | 02:04 | |
| *** smatzek has quit IRC | 02:06 | |
| *** xarses has joined #openstack-infra | 02:07 | |
| *** cody-somerville has quit IRC | 02:14 | |
| *** bobh has joined #openstack-infra | 02:32 | |
| *** yamamoto has joined #openstack-infra | 02:33 | |
| *** bobh has quit IRC | 02:36 | |
| *** dhinesh has joined #openstack-infra | 02:36 | |
| *** yamamoto has quit IRC | 02:38 | |
| *** gouthamr has joined #openstack-infra | 02:39 | |
| *** edmondsw has joined #openstack-infra | 02:40 | |
| *** edmondsw has quit IRC | 02:44 | |
| *** thorst has joined #openstack-infra | 03:00 | |
| *** thorst has quit IRC | 03:04 | |
| *** xarses has quit IRC | 03:17 | |
| *** thorst has joined #openstack-infra | 03:18 | |
| *** thorst has quit IRC | 03:18 | |
| *** smatzek has joined #openstack-infra | 03:26 | |
| *** smatzek has quit IRC | 03:32 | |
| *** bobh has joined #openstack-infra | 03:33 | |
| *** bobh has quit IRC | 03:37 | |
| *** nikhil_k has quit IRC | 03:40 | |
| *** dhinesh has quit IRC | 04:00 | |
| *** yamamoto has joined #openstack-infra | 04:15 | |
| *** yamamoto has quit IRC | 04:23 | |
| *** gongysh has joined #openstack-infra | 04:25 | |
| *** edmondsw has joined #openstack-infra | 04:28 | |
| *** edmondsw has quit IRC | 04:32 | |
| *** bobh has joined #openstack-infra | 04:33 | |
| *** bobh has quit IRC | 04:38 | |
| *** liusheng has quit IRC | 04:41 | |
| *** dimak has quit IRC | 04:45 | |
| *** psachin has joined #openstack-infra | 04:45 | |
| *** ari[m] has quit IRC | 04:47 | |
| *** dimak has joined #openstack-infra | 04:47 | |
| *** ari[m] has joined #openstack-infra | 04:47 | |
| *** liusheng has joined #openstack-infra | 04:54 | |
| *** thorst has joined #openstack-infra | 05:19 | |
| *** chlong has joined #openstack-infra | 05:20 | |
| *** thorst has quit IRC | 05:24 | |
| *** armaan has joined #openstack-infra | 05:36 | |
| *** snapiri has joined #openstack-infra | 05:52 | |
| *** gouthamr has quit IRC | 06:12 | |
| *** claudiub has quit IRC | 06:13 | |
| *** bobh has joined #openstack-infra | 06:35 | |
| *** bobh has quit IRC | 06:40 | |
| *** pcaruana has joined #openstack-infra | 06:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Convert publish-api-ref for service name https://review.openstack.org/512031 | 06:52 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Support pike->master undercloud upgrades https://review.openstack.org/511885 | 06:52 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix translations: Use zanata_client_version https://review.openstack.org/512074 | 06:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix translations: Use zanata_client_version https://review.openstack.org/512074 | 06:55 |
| *** markvoelker has joined #openstack-infra | 06:57 | |
| *** markvoelker_ has joined #openstack-infra | 06:59 | |
| *** markvoelker has quit IRC | 07:02 | |
| *** markvoelker has joined #openstack-infra | 07:03 | |
| *** markvoelker_ has quit IRC | 07:06 | |
| *** slaweq has joined #openstack-infra | 07:07 | |
| *** slaweq has quit IRC | 07:09 | |
| *** thorst has joined #openstack-infra | 07:21 | |
| *** CHIPPY has joined #openstack-infra | 07:21 | |
| *** thorst has quit IRC | 07:25 | |
| *** bobh has joined #openstack-infra | 07:36 | |
| *** jtomasek has joined #openstack-infra | 07:40 | |
| *** liusheng has quit IRC | 07:40 | |
| *** bobh has quit IRC | 07:40 | |
| *** liusheng has joined #openstack-infra | 07:41 | |
| *** ssbarnea has quit IRC | 07:43 | |
| *** wznoinsk has quit IRC | 07:44 | |
| *** wznoinsk has joined #openstack-infra | 07:46 | |
| *** ssbarnea has joined #openstack-infra | 07:46 | |
| *** armaan has quit IRC | 07:50 | |
| *** armaan has joined #openstack-infra | 07:50 | |
| *** jtomasek has quit IRC | 07:51 | |
| *** edmondsw has joined #openstack-infra | 08:04 | |
| *** edmondsw has quit IRC | 08:08 | |
| *** liusheng has quit IRC | 08:14 | |
| *** liusheng has joined #openstack-infra | 08:14 | |
| *** armaan has quit IRC | 08:20 | |
| *** armaan has joined #openstack-infra | 08:21 | |
| *** slaweq has joined #openstack-infra | 08:26 | |
| *** bobh has joined #openstack-infra | 08:37 | |
| *** bobh has quit IRC | 08:41 | |
| *** armaan has quit IRC | 08:44 | |
| *** armaan has joined #openstack-infra | 08:44 | |
| *** armaan has quit IRC | 08:45 | |
| *** armaan has joined #openstack-infra | 08:45 | |
| *** bhavik1 has joined #openstack-infra | 08:48 | |
| *** bhavik1 has quit IRC | 08:49 | |
| *** CHIPPY has quit IRC | 08:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use horizon templates https://review.openstack.org/512017 | 08:50 |
| *** slaweq has quit IRC | 08:59 | |
| *** tosky has joined #openstack-infra | 09:08 | |
| *** jascott1 has quit IRC | 09:28 | |
| *** jascott1 has joined #openstack-infra | 09:29 | |
| *** jascott1 has quit IRC | 09:33 | |
| *** persia has quit IRC | 09:35 | |
| *** persia has joined #openstack-infra | 09:36 | |
| *** bobh has joined #openstack-infra | 09:37 | |
| *** chlong has quit IRC | 09:38 | |
| *** armaan has quit IRC | 09:40 | |
| *** armaan has joined #openstack-infra | 09:40 | |
| *** bobh has quit IRC | 09:42 | |
| snapiri | AJaeger: Is there a way I can tell if I am running under zuulv3 or zuulv2? I need it to remove the zuul-clone in the zuulv3 path... | 09:48 |
| *** edmondsw has joined #openstack-infra | 09:52 | |
| *** edmondsw has quit IRC | 09:56 | |
| snapiri | Hi infra: any answer for ^^ ? | 10:08 |
| *** pbourke has quit IRC | 10:21 | |
| *** pbourke has joined #openstack-infra | 10:23 | |
| *** yamamoto has joined #openstack-infra | 10:35 | |
| *** bobh has joined #openstack-infra | 10:38 | |
| *** bobh has quit IRC | 10:44 | |
| *** claudiub has joined #openstack-infra | 10:51 | |
| *** thorst has joined #openstack-infra | 10:54 | |
| *** jkilpatr_ has joined #openstack-infra | 10:57 | |
| *** thorst has quit IRC | 10:59 | |
| tosky | snapiri: jobs that vote as "Jenkins" are zuul v2; jobs that votes as "Zuul" are zuul v3 | 11:01 |
| *** yamamoto has quit IRC | 11:02 | |
| tosky | right now both are leaving a vote, but the vote from Jenkins (aka zuul v2) is still the vote which drives the merge | 11:02 |
| tosky | with the exceptions of few infra repositories which are zuulv3-only | 11:02 |
| tosky | the switch will be later today, according the last emails | 11:02 |
| *** armaan has quit IRC | 11:03 | |
| *** yamamoto has joined #openstack-infra | 11:03 | |
| *** armaan has joined #openstack-infra | 11:03 | |
| snapiri | tosky: thanks for your reply. What I am trying to figure out is how to modify my scripts, so tox_install will not use zuul-clone when using zuulv3. When zuul-clone is used the build fails :( | 11:07 |
| *** yamamoto has quit IRC | 11:08 | |
| *** jkilpatr_ has quit IRC | 11:08 | |
| *** abelur_ has quit IRC | 11:11 | |
| tosky | snapiri: I would rather understand why zuul-clone fails | 11:21 |
| *** askb has quit IRC | 11:24 | |
| *** abelur_ has joined #openstack-infra | 11:29 | |
| *** markvoelker has quit IRC | 11:32 | |
| *** askb has joined #openstack-infra | 11:37 | |
| *** bobh has joined #openstack-infra | 11:39 | |
| *** edmondsw has joined #openstack-infra | 11:40 | |
| *** jascott1 has joined #openstack-infra | 11:40 | |
| *** bobh has quit IRC | 11:44 | |
| *** edmondsw has quit IRC | 11:44 | |
| *** AJaeger has joined #openstack-infra | 11:45 | |
| AJaeger | infra-root, could you review https://review.openstack.org/512074 for the next translation fix, please? | 11:45 |
| AJaeger | infra-root, and let's merge https://review.openstack.org/#/c/512030/ - to not publish api-ref/api-guide to wrong locations and then forced to cleanup. This gives us time to find a proper solution. https://review.openstack.org/512031 is one way forward but needs followup for additional projects | 11:47 |
| AJaeger | there're a couple more fixes in the review queue - I suggest to get everthing in to be in a good state but those thow above are critical for me | 11:48 |
| tosky | AJaeger: aaaand if you have some time to extend https://review.openstack.org/#/c/512017/ for sahara-dashboard too, I would be happy :) | 11:49 |
| AJaeger | snapiri: then let's fix it - did you read the docs we have? If you show me a review, I might guide you for a fix | 11:49 |
| AJaeger | tosky: looking... | 11:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use horizon templates https://review.openstack.org/512017 | 11:53 |
| AJaeger | tosky: done ^ | 11:53 |
| tosky | thanks! | 11:54 |
| AJaeger | snapiri: I expect that you miss a few "required-projects" lines, this was all explained in the mails by Monty. Do you have a log file to look at? | 11:54 |
| tosky | it was pointless to create yet-another-review for that | 11:54 |
| AJaeger | tosky: you could have just updated mine, would have been ok ;) | 11:54 |
| tosky | AJaeger: oh, ok, but it's better that you did it: you added also two additional templates that I would have missed | 11:55 |
| AJaeger | infra-root, the one cross-repo job we should fix IMHO is requirements-check - see etherpad. That will hit quite a few repos | 11:55 |
| AJaeger | tosky: ;) | 11:56 |
| tosky | AJaeger: just to be sure: the changes in project-config were not applied immediately to jobs for v2; if I understand it correctly, this is the same now, and changes are applied immediately only for the other repositories; is it correct? | 11:56 |
| AJaeger | tosky: you mean with depends-on? | 11:58 |
| AJaeger | yeah, depends-on a project-config change will not update the jobs - since it's a trusted repo. | 11:58 |
| tosky | ack, so I've got it correctly | 11:59 |
| *** thorst has joined #openstack-infra | 11:59 | |
| tosky | now let's hope that infra-root will merge 512017 too before The Switch™ | 12:01 |
| AJaeger | tosky: would be great indeed... | 12:02 |
| *** markvoelker has joined #openstack-infra | 12:08 | |
| tosky | AJaeger: would a recheck on https://review.openstack.org/#/c/508906/ make sense? There is only one error which is seems infra-related | 12:11 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Support pike->master undercloud upgrades https://review.openstack.org/511885 | 12:11 |
| *** ccamacho has quit IRC | 12:11 | |
| AJaeger | tosky: we hit this "devstack finger://ze08.openstack.org/d3619f975aea4d45b3fb0de4eea1e448 : RETRY_LIMIT in 1m 26s" already twice - not sure what's going on there. Let's try... | 12:12 |
| AJaeger | jeblair, mordred ^ | 12:13 |
| *** markvoelker_ has joined #openstack-infra | 12:14 | |
| tosky | but what is the failed job? It would be useful to have it printed instead of the link to the output stream | 12:15 |
| tosky | oh, the "devstack" job, of course | 12:16 |
| *** markvoelker has quit IRC | 12:17 | |
| *** boden has joined #openstack-infra | 12:17 | |
| tosky | if I add that devstack job to my project, should add my project the "required-project" field or is it implicit? | 12:18 |
| * tosky should probably RTFM more | 12:18 | |
| openstackgerrit | boden proposed openstack-infra/project-config master: zuul v3 gate changes for vmware-nsx https://review.openstack.org/511836 | 12:21 |
| *** yamamoto has joined #openstack-infra | 12:21 | |
| *** slaweq has joined #openstack-infra | 12:22 | |
| *** esberglu has joined #openstack-infra | 12:24 | |
| *** thorst has quit IRC | 12:24 | |
| snapiri | AJaeger: thanks for the replies. I have read the migration instructions, but maybe I missed something. | 12:24 |
| snapiri | I am a bit lost here... This is a relevant output: http://logs.openstack.org/89/509189/11/check/legacy-dragonflow-dsvm-fullstack/c349087/job-output.txt.gz | 12:25 |
| snapiri | The relevant patch is: https://review.openstack.org/#/c/508761/ | 12:25 |
| *** armaan has quit IRC | 12:27 | |
| *** armaan has joined #openstack-infra | 12:27 | |
| *** slaweq has quit IRC | 12:27 | |
| *** esberglu has quit IRC | 12:28 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove api-guide remains https://review.openstack.org/511924 | 12:29 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 12:29 |
| lennyb | Hello, Is there any work going on BlackDuck Hub support for JJB? | 12:32 |
| AJaeger | snapiri: you're playing around with tox_install.sh - I have not looked further into this, I suggest to discusss this some more with tonyb who wanted to look into it. Is your job working without your change? If not, let's fix that first... | 12:36 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 12:37 |
| *** dave-mccowan has joined #openstack-infra | 12:39 | |
| snapiri | AJaeger: the log I sent is of a different patch, all patches fail with the same log - permission denied when trying to perform hard-link by zuul-clone. This is the reason I am trying to play with the tox_install | 12:39 |
| snapiri | AJaeger: here is another log if it is relevant: http://logs.openstack.org/34/507034/2/check/legacy-dragonflow-dsvm-fullstack/2b6ba59/ | 12:40 |
| *** bobh has joined #openstack-infra | 12:40 | |
| *** bobh has quit IRC | 12:45 | |
| *** liusheng has quit IRC | 12:46 | |
| *** liusheng has joined #openstack-infra | 12:47 | |
| *** yamamoto has quit IRC | 12:49 | |
| boden | hi, if someone gets a chance it’d be nice to land https://review.openstack.org/#/c/511836/ so I can verify in the respective project before the v3 switch later | 12:57 |
| *** thorre_se has joined #openstack-infra | 13:05 | |
| *** thorre has quit IRC | 13:08 | |
| *** thorre_se is now known as thorre | 13:08 | |
| AJaeger | snapiri: please link to the line that shows the problem that makes it far easier | 13:12 |
| AJaeger | boden: commented | 13:12 |
| AJaeger | bbl | 13:12 |
| *** yamamoto has joined #openstack-infra | 13:18 | |
| snapiri | AJaeger: I believe it is here: http://logs.openstack.org/34/507034/2/check/legacy-dragonflow-dsvm-fullstack/2b6ba59/job-output.txt.gz#_2017-10-15_12_20_16_710271 | 13:24 |
| snapiri | To the best of my understanding zuul-clone tries to create hard-links and fails | 13:26 |
| *** edmondsw has joined #openstack-infra | 13:28 | |
| fungi | snapiri: ahh, yes, there are some errors just before that in the log as well about "ERROR: invocation failed (exit code 1), logfile: /opt/stack/new/dragonflow/.tox/fullstack/log/fullstack-1.log" but we don't appear to collect that logfile | 13:31 |
| *** edmondsw has quit IRC | 13:33 | |
| mordred | fungi: oh - that looks likea permissions issue | 13:34 |
| fungi | mordred: well, my expectation is that it's a cross-device hardlink | 13:34 |
| snapiri | fungi: any ideas? I thought that all the errors above are actually the content of the file... | 13:34 |
| jeblair | i just woke up and tried to figure out what ikea permissions were | 13:34 |
| mordred | fungi: we should be testing for that in zuul-cloner shim | 13:35 |
| mordred | jeblair: :) | 13:35 |
| fungi | jeblair: you can only adjust them with an allen key? | 13:35 |
| jeblair | hehe | 13:36 |
| fungi | but yeah, now i'm thinking it's a permissions issue because iirc we don't have /opt and /home on separate filesystems in ovh, where that example build happened | 13:38 |
| fungi | confirmed: http://logs.openstack.org/34/507034/2/check/legacy-dragonflow-dsvm-fullstack/2b6ba59/logs/df.txt.gz | 13:39 |
| mordred | fungi: my guess is that that tox command is running as the stack user and is trying to make hardlinks to the git dirs in ~zuul | 13:40 |
| snapiri | mordred: I believe you are right | 13:40 |
| snapiri | The tox is being run as stack user | 13:40 |
| *** bobh has joined #openstack-infra | 13:41 | |
| *** sbezverk has quit IRC | 13:41 | |
| mordred | longer-term there's a nicer pattern that can be used where the tox can be run using the tox role from the job as the stack user ... but shorter term we probably need to just chmod -R the src tree to be writable | 13:41 |
| snapiri | http://logs.openstack.org/34/507034/2/check/legacy-dragonflow-dsvm-fullstack/2b6ba59/job-output.txt.gz#_2017-10-15_12_20_08_519536 | 13:41 |
| * mordred is looking at the requirements issue btw ... | 13:42 | |
| *** bobh has quit IRC | 13:42 | |
| fungi | mordred: world-writable you mean? (o+w/a+w?) | 13:42 |
| *** bobh has joined #openstack-infra | 13:42 | |
| mordred | fungi: yah | 13:43 |
| *** yamamoto has quit IRC | 13:45 | |
| *** bobh has quit IRC | 13:45 | |
| jeblair | AJaeger: i'm leaving https://review.openstack.org/495426 for you to +3 | 13:45 |
| mordred | jeblair: this requirements issue doesn't make sense me to - do you have a sec to look at it with me? | 13:47 |
| *** slaweq has joined #openstack-infra | 13:47 | |
| jeblair | mordred: i can give it a shot. i'm still warming up. :) | 13:47 |
| mordred | jeblair: so - if you look at http://logs.openstack.org/05/510105/16/check/requirements-check/7256969/ara/ | 13:48 |
| mordred | you'll see "SystemError: fatal: Not a git repository (or any of the parent directories): .git" | 13:48 |
| openstackgerrit | Merged openstack-infra/project-config master: v3 cleanup: Use openstack-python-jobs https://review.openstack.org/511803 | 13:48 |
| jeblair | ya | 13:48 |
| mordred | that comes from here: http://git.openstack.org/cgit/openstack/requirements/tree/playbooks/files/project-requirements-change.py#n171 | 13:49 |
| mordred | thing is - cwd should be the git repo: http://git.openstack.org/cgit/openstack/requirements/tree/playbooks/requirements-check.yaml#n6 | 13:49 |
| mordred | of the project in question | 13:49 |
| jeblair | mordred: docs say chdir added in 2.4 | 13:51 |
| jeblair | http://docs.ansible.com/ansible/latest/script_module.html | 13:51 |
| jeblair | (we're still running 2.3) | 13:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Stop translation of trove https://review.openstack.org/511813 | 13:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Added initial jobs for python-tempestconf https://review.openstack.org/511194 | 13:55 |
| jeblair | mordred: so maybe we should copy+command ? or else give the script an extra argument of a directory to change to at the start. | 13:55 |
| mordred | jeblair: wow - I just had a weird networking event - can't ssh directly to my irc bouncer | 14:01 |
| mordred | jeblair: https://review.openstack.org/512083 Move chdir into script itself | 14:01 |
| mordred | jeblair: I agree with your assessment | 14:01 |
| jeblair | dmsimard: 511953 +2 with comment | 14:03 |
| jeblair | mordred: +2 | 14:04 |
| mordred | jeblair: I have pushed up https://review.openstack.org/510105 with a depends-on | 14:04 |
| tosky | I don't want to ruin your Sunday, but the new devstack job is failing with an hard-to-debug error: https://review.openstack.org/#/c/508906/ | 14:04 |
| mordred | fungi: mind +3ing https://review.openstack.org/#/c/495426/ ? | 14:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy magnum jobs from pipeline https://review.openstack.org/508804 | 14:05 |
| jeblair | tosky: i'll track that down | 14:06 |
| tosky | thanks | 14:06 |
| jeblair | tosky, mordred: http://paste.openstack.org/show/623656/ are the 2 errors (a job bug and a zuul bug) | 14:09 |
| tosky | jeblair: out of curiosity, in case it happens again: how did you get the log? From the finger:// link? | 14:11 |
| *** psachin has quit IRC | 14:11 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix python3 unicode errors in executor server https://review.openstack.org/512084 | 14:12 |
| jeblair | tosky, mordred: ^ | 14:12 |
| jeblair | tosky: root access on the zuul executor -- the exception is a bug in zuul that prevented the log uploading functions from working. | 14:13 |
| *** yamamoto has joined #openstack-infra | 14:13 | |
| tosky | oh, I see | 14:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Add required projects for networking-cisco https://review.openstack.org/512004 | 14:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove training-guides legacy jobs https://review.openstack.org/512010 | 14:13 |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: [WIP] disk-usage refactoring https://review.openstack.org/512085 | 14:14 |
| jeblair | fungi: can you also review 512084 -- that's a pretty serious error we should restart the executors for | 14:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Use horizon templates https://review.openstack.org/512017 | 14:15 |
| pabelanger | morning | 14:16 |
| pabelanger | +3 on 512083 | 14:17 |
| jeblair | tosky, mordred: i pushed up a new revision of remote: https://review.openstack.org/508906 Add devstack base job for zuul v3 | 14:17 |
| jeblair | tosky, mordred: i had forgotten to git-add the configure-swap role :( | 14:17 |
| tosky | oh, that was easy to solve luckily :) | 14:18 |
| tosky | so I can recheck my job creating another 11-patches stack | 14:18 |
| jeblair | \o/ | 14:19 |
| pabelanger | looking at zuulv3-issues on etherpad | 14:19 |
| *** slaweq has quit IRC | 14:20 | |
| boden | AJaeger left you a response in https://review.openstack.org/#/c/511836… I don’t understand one of your comments | 14:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use os-service-types to get api-ref destination https://review.openstack.org/512086 | 14:21 |
| mordred | AJaeger, jeblair: ^^ that is an alternate suggestion for 512031 | 14:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Mark publish-openstack-artifacts and pipelines as post-review https://review.openstack.org/495426 | 14:22 |
| *** slaweq has joined #openstack-infra | 14:22 | |
| jeblair | mordred: we should consider pointing our executors at our pip mirrors | 14:23 |
| openstackgerrit | boden proposed openstack-infra/project-config master: zuul v3 gate changes for vmware-nsx https://review.openstack.org/511836 | 14:24 |
| mordred | obviously that pre-task section can be cleaned up and put into a role - but I figured that would be the shortest way to talk about it | 14:24 |
| jeblair | mordred: (or maybe making a sort of configure-mirrors role for tasks that run on the executors) | 14:24 |
| mordred | jeblair: yah - alternately we could shift that task to be a post-task on the remote node | 14:24 |
| jeblair | or that | 14:24 |
| *** thorst has joined #openstack-infra | 14:25 | |
| *** yamamoto has quit IRC | 14:25 | |
| jeblair | mordred: this probably isn't a big deal for this job, but generally speaking, we'd probably trust the executor to produce the correct publish location more than the node. | 14:26 |
| jeblair | mordred: i'd probably rank solutions as: 1) executor-configure-mirrors role 2) post-task on node (and i don't think we should do my first suggestion of setting a mirror for the executors at the system level) | 14:27 |
| *** modrednetfail has joined #openstack-infra | 14:27 | |
| mordred | jeblair: there's a third option - which is fetching the service-types json file rather than consuming the library | 14:28 |
| jeblair | that's published by its own job, yeah? | 14:28 |
| jeblair | so there's an initial bootstrapping sequence, but no catch-22. | 14:28 |
| jeblair | that would work for me too (it's eventually consistent if you add a new type) | 14:29 |
| *** logan- has quit IRC | 14:29 | |
| pabelanger | zuulv3-issues looks to be in great shape, over all. Going to see if I can find any unknown issues on zuulv3.o.o status page | 14:29 |
| *** logan- has joined #openstack-infra | 14:29 | |
| *** thorst has quit IRC | 14:29 | |
| *** bobh has joined #openstack-infra | 14:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use service-types.openstack.org to get api-ref destination https://review.openstack.org/512086 | 14:33 |
| mordred | jeblair: like that ^^ | 14:33 |
| pabelanger | clarkb: who is our citycloud contact again? We have 8 stuck nodes in citycloud-kna1 we should ask to see if they can help clean them up | 14:36 |
| mordred | jeblair: if we like that I can make it nicer | 14:36 |
| pabelanger | off topic, but does any other infra-root get 2.9M emails from root@cacti01.openstack.org about /usr/local/bin/create_graphs.sh failing? | 14:39 |
| pabelanger | I'm on limited bandwidth over the next month or so, and its killing my quota now | 14:40 |
| *** chlong has joined #openstack-infra | 14:47 | |
| pabelanger | ah, think I see it | 14:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use service-types.openstack.org to get api destinations https://review.openstack.org/512086 | 14:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Fix grep error when listing ifName values https://review.openstack.org/512088 | 14:55 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Fix grep error when listing ifName values https://review.openstack.org/512088 | 14:56 |
| pabelanger | jeblair: mordred: clarkb: fungi: ^ when you have spare time, should help cut down on email size from cacti. | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add role to provice a service_type_data fact https://review.openstack.org/512089 | 14:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use service-types.openstack.org to get api destinations https://review.openstack.org/512086 | 14:59 |
| mordred | jeblair, pabelanger, AJaeger: ^^ ok - I think that's the clean version | 15:00 |
| jeblair | dmsimard: i'm happy to discuss the ara changes whenever our schedules next sync up. :) i'm happy to change the approach, or discard them entirely if you have other suggestions. just trying to be helpful. at the very least, i'm learning more about ara. | 15:02 |
| *** wolverineav has joined #openstack-infra | 15:04 | |
| dmsimard | jeblair: I'm sort of there :p | 15:05 |
| *** armax has joined #openstack-infra | 15:05 | |
| *** armaan has quit IRC | 15:05 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use propose-update-constraints instead of legacy https://review.openstack.org/512090 | 15:05 |
| *** armaan has joined #openstack-infra | 15:06 | |
| pabelanger | fungi: Thanks! I'll push up an update now | 15:07 |
| fungi | any guesses what caused the memory usage for the scheduler to nearly double around 13:30z? | 15:07 |
| *** armaan has quit IRC | 15:07 | |
| fungi | er, more than double even | 15:08 |
| *** armaan has joined #openstack-infra | 15:08 | |
| *** armaan has quit IRC | 15:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Fix grep error when listing ifName values https://review.openstack.org/512088 | 15:08 |
| pabelanger | jeblair: fungi: ^updates | 15:08 |
| dmsimard | jeblair: on the topic of playbook paths and how they're displayed, last time we touched the topic we settled on being able to provide the ability to simply give "names" to playbooks -- so that if you wanted to give your playbook the "name" '/home/zuul/src/git.openstack.org/zuul-jobs/playbooks/foo/playbook.yml', you could do just that, but if you also happened to want to name it "foo mysql setup" you | 15:08 |
| dmsimard | could do that in the same way | 15:08 |
| pabelanger | ty for fast reviews, my ISP bill will thank you | 15:09 |
| jeblair | dmsimard: should we add a new field in the db for that? | 15:09 |
| *** armaan has joined #openstack-infra | 15:09 | |
| snapiri | Hi Infra, continuing the problem above, I have noticed something strange, in patch the following https://review.openstack.org/#/c/509189/ I have two zuul tasks that are basically the same, one is from project-config and one is from the project itself (legacy-dragonflow-dsvm-fullstack & dragonflow-dsvm-fullstack). I have noticed that the first fails while the second succeeds. | 15:09 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix translations: Use zanata_client_version https://review.openstack.org/512074 | 15:09 |
| tosky | jeblair: how to debug "This change depends on a change that failed to merge." like in https://review.openstack.org/#/c/508906/ ? The only dependency (https://review.openstack.org/#/c/512027/) seems to apply cleanly to devstack-gate | 15:10 |
| *** armax has quit IRC | 15:10 | |
| snapiri | I also noticed that the filesystems are different in both cases... | 15:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use propose-updates for requirements-constraints https://review.openstack.org/512090 | 15:12 |
| dmsimard | jeblair: It'd require a new field in the database yeah. The only thing I haven't quite figured out was what would be the most convenient way to provide this name. Env vars ? extra vars ? an ansible module (not unlike ara_record, could we even just use ara_record outright?) etc | 15:12 |
| pabelanger | snapiri: looking, but just left a comment about syntax error in .zuul.yaml | 15:13 |
| dmsimard | jeblair: ideally, it would not be too far off from another feature we'd like -- being able to group playbooks in a logical group (in our context, pre/run/post phases) | 15:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove legacy proposal jobs https://review.openstack.org/512091 | 15:14 |
| jeblair | dmsimard: oh interesting -- an ansible module might actually be really convenient because it would let us move the specification of the name into job content rather than hard coded in zuul. | 15:14 |
| pabelanger | snapiri: tempest-dsvm-dragonflow tempest-dsvm-dragonflow : ERROR No valid playbook found (non-voting) | 15:14 |
| pabelanger | snapiri: that is likey the issue, and my comments will help fix that | 15:14 |
| jeblair | tosky: i went back to 512027 and did a recheck, and it now has a very strange configuration error | 15:15 |
| *** armax has joined #openstack-infra | 15:15 | |
| jeblair | mordred, pabelanger: ^ we should take a look at the error in 512027 | 15:15 |
| pabelanger | looking | 15:15 |
| dmsimard | jeblair: if you look here and expand the 'records' tab, that's freeform key/value things associated with the playbook through the ara_record module http://logs.openstack.org/07/489207/5/check/gate-ara-integration-py35-latest-fedora-26/363923b/logs/build/reports/0b4a6c53-74e6-4994-bc0e-a6585f70ab12.html | 15:15 |
| mordred | jeblair: looking | 15:16 |
| dmsimard | jeblair: http://ara.readthedocs.io/en/latest/usage.html#using-the-ara-record-module | 15:16 |
| jeblair | mordred: i wonder if perhaps the change to add the post-review stuff broke some of those jobs | 15:16 |
| *** edmondsw has joined #openstack-infra | 15:16 | |
| pabelanger | First time I've seen that error | 15:16 |
| jeblair | dmsimard: oh neat | 15:16 |
| mordred | jeblair: oh. fascinating | 15:16 |
| dmsimard | jeblair: ara_record can be used from inside a playbook (and will automatically tie data back to that particular playbook) or it can also be used as an 'ad-hoc' task to associate data to a playbook if you provide the id of the playbook you want to associate data with | 15:16 |
| jeblair | dmsimard: hrm... as i think of this further though, we really only control the base job playbooks at the level we'd need. we can't really ask folks to put ara_record tasks in job playbooks... | 15:17 |
| mordred | jeblair: I don't understand why it thinks legacy-tripleo-ci-centos-7-containers-multinode is a post-review job | 15:18 |
| jeblair | dmsimard: i really like the ara_record idea, but i can't think of a way to use it right now... we may need to use env vars? | 15:18 |
| dmsimard | jeblair: yup. That's why I'm wondering what is the best way to do this from a user perspective | 15:18 |
| dmsimard | jeblair: Otherwise, they can be implicit tasks ran by the executor so that users don't have to worry about it. | 15:19 |
| dmsimard | Not unlike how we run network tests implicitely | 15:19 |
| jeblair | mordred: does that job inherit from publish-openstack-artifacts? | 15:19 |
| jeblair | that's the thing we added post-review to | 15:19 |
| modrednetfail | yup | 15:19 |
| modrednetfail | that would be it | 15:19 |
| jeblair | http://logs.openstack.org/27/512027/1/check/legacy-tripleo-ci-centos-7-containers-multinode/ed0e5a6/zuul-info/inventory.yaml | 15:20 |
| jeblair | ya ^ | 15:20 |
| jeblair | yay the system works | 15:20 |
| modrednetfail | \o/ | 15:20 |
| modrednetfail | so - this should be better with the base job rework | 15:20 |
| *** edmondsw has quit IRC | 15:21 | |
| snapiri | pabelanger: thanks. fixed, hope it will solve it... | 15:21 |
| jeblair | oh, the logs/docs/artifacts publish proposal | 15:21 |
| jeblair | mordred: ^? | 15:21 |
| mordred | yah | 15:21 |
| pabelanger | mordred: one thing I was confused about that, we still publish to logs.o.o (for example) from executor, right? | 15:22 |
| dmsimard | jeblair: and then, the other thing is that ara is a bit in the same state as zuul v3 right now, as in I'm trying to keep the master branch as frozen as possible while the work goes in feature/1.0. 1.0 is not quite production ready yet though. | 15:22 |
| mordred | since I belive legacy-tripleo-ci-dsvm is inheriting from publish-opnestack-artifacts so that it can do temporary artifacts or something (I'll need to go look at the why) | 15:23 |
| mordred | pabelanger: yes, that's right | 15:23 |
| mordred | pabelanger: but if we wanted to change that - or some other zuul deployer wanted to publish directly from their executors and still use zuul-jobs- that would be possible | 15:23 |
| pabelanger | ++ | 15:24 |
| pabelanger | okay, switching back to zuul-jobs: http://logs.openstack.org/88/512088/3/check/legacy-openstackci-beaker-ubuntu-trusty/2597cf4/job-output.txt.gz#_2017-10-15_15_18_53_405959 | 15:24 |
| jeblair | dmsimard: implicit tasks -- how could we have the executor run a task in the context of a playbook? surely that would be a separate run and not possible to associate with the real user-supplied playbook? | 15:24 |
| pabelanger | we had SSh error mid run | 15:24 |
| pabelanger | looking why now | 15:24 |
| *** chlong has quit IRC | 15:24 | |
| pabelanger | looks to be vexxhost | 15:25 |
| mordred | pabelanger: I had vexxhost network issues this morning with my IRC bouncer | 15:25 |
| mordred | pabelanger: they seem to now be resolved - but they were bad for a while | 15:25 |
| *** thorst has joined #openstack-infra | 15:26 | |
| pabelanger | kk | 15:26 |
| pabelanger | I'll recheck | 15:26 |
| openstackgerrit | Merged openstack-infra/system-config master: Fix grep error when listing ifName values https://review.openstack.org/512088 | 15:26 |
| jeblair | i'm sure mnaser has an actual nagios, but just in case human nagios helps here, i have said his name :) | 15:26 |
| pabelanger | mordred: actually, just thinnking, would ssh pipe still be open? between playbook runs? | 15:27 |
| mordred | pabelanger: it should be - unless the network itself ceasees being able to route packets (which is what I was seeing this morning) | 15:27 |
| pabelanger | Yah, okay | 15:27 |
| pabelanger | mordred: jeblair: BTW: found our first use-case for zuul inline comments :) https://review.openstack.org/#/c/509189/11/.zuul.yaml | 15:29 |
| pabelanger | will be awesome the day zuul is able to do that | 15:29 |
| smcginnis | :q | 15:30 |
| *** thorst has quit IRC | 15:30 | |
| smcginnis | Bah! | 15:30 |
| dmsimard | jeblair: None of the ways I'm thinking about are super clean actually.. so I don't have any great suggestions right now. | 15:31 |
| jeblair | dmsimard: maybe both then. :) | 15:32 |
| dmsimard | jeblair: oh, definitely, being able to supply this kind of information through an env variable or through a module do not need to be mutually exclusive | 15:32 |
| dmsimard | there'd be ara_playbook_name env var, or something, and then ara_playbook_group for our phases | 15:33 |
| mordred | dmsimard: that's what we do for zuul_stream today ... | 15:34 |
| mordred | dmsimard: we pass a name and a phase env var | 15:34 |
| dmsimard | yeah, I just feel it's a bit out of scope of the callback to be doing that kind of stuff, but I can't think of another way that is better or more convenient right now so whatever :) | 15:35 |
| mordred | oh - scuse me - we pass those as ansible vars - zuul_execution_phase and zuul_execution_canonical_name_and_path | 15:35 |
| mordred | we could pass them as env vars instead pretty easily of course | 15:36 |
| dmsimard | bah it can be like os.getenv('ara_playbook_name', vars['ara_playbook_name']) or something | 15:36 |
| dmsimard | the callback has access to either | 15:36 |
| dmsimard | ok, I'll go back to try and figure out why my sqlite middleware doesn't work for css and js files.. | 15:37 |
| mordred | dmsimard, jeblair: I don't think that tripleo base job needs to be parented on legacy-openstack-publish-artifacts | 15:39 |
| mordred | I believe that's there for the legacy-periodic-tripleo-* jobs | 15:39 |
| dmsimard | mordred: yeah they were originally migrated that way, I'm not sure why. All I was told is that jobs that were found to be publishing to tarballs.o.o were automatically parented to that job. | 15:40 |
| dmsimard | I don't know enough about tripleo jobs to know for sure but I can dig if need be | 15:40 |
| mordred | ah - yes - I see it in the job content now | 15:40 |
| mordred | it does seem that there is an artifact publisher on normal jobs | 15:41 |
| *** dhinesh has joined #openstack-infra | 15:41 | |
| mordred | playbooks/legacy/tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-puppet-pike/post.yaml - for instance - is set up to copy things from workspace/images/* | 15:41 |
| *** armax has quit IRC | 15:42 | |
| jeblair | http://tarballs.openstack.org/tripleo-ci/images/ is empty | 15:46 |
| mordred | jeblair: yah - I think it's a bogus publisher - likely a macro copy-pastad on the old jjb jobs that isn't actually used | 15:47 |
| jeblair | i just double checked that *is* the publish location of the old version of that job | 15:48 |
| jeblair | here is a list of old jobs publishing to that location: http://paste.openstack.org/show/623657/ | 15:50 |
| pabelanger | I don't think tripleo ever switch to using tarballs.o.o for publishing yet | 15:50 |
| mordred | awesome | 15:50 |
| AJaeger | mordred: thanks for working on the service-type. I like your ozj change, just one question on https://review.openstack.org/#/c/512089/1/roles/set-service-type-data-fact/README.rst | 15:50 |
| mordred | let's reparent the tripleo base job off of openstack-publish-artifacts | 15:50 |
| pabelanger | they still using a server in tripleo-test-cloud-rh1 for some images, IIRC | 15:50 |
| pabelanger | mordred: +1 | 15:51 |
| jeblair | there's a single template that all the tripleo jobs use that has that publisher | 15:51 |
| jeblair | mordred: ++ | 15:51 |
| *** slaweq has quit IRC | 15:52 | |
| *** liusheng has quit IRC | 15:52 | |
| jeblair | legacy-tripleo-ci-dsvm i think is the job in question | 15:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove openstack-publish-artifacts from tripleo-base-dsvm https://review.openstack.org/512094 | 15:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove openstack-publish-artifacts from legacy-tripleo-ci-dsvm https://review.openstack.org/512094 | 15:53 |
| *** liusheng has joined #openstack-infra | 15:53 | |
| mordred | AJaeger: looking | 15:53 |
| mordred | AJaeger: BWAHAHAH. thanks | 15:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add role to provice a service_type_data fact https://review.openstack.org/512089 | 15:56 |
| mordred | AJaeger: thereyago | 15:56 |
| AJaeger | thanks, mordred | 15:56 |
| AJaeger | mordred: I see an extra space, let me fix... | 15:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add role to provice a service_type_data fact https://review.openstack.org/512089 | 15:57 |
| pabelanger | mordred: did we ever land your patched for post-run extra debug logs? | 15:58 |
| tosky | provice? | 15:58 |
| pabelanger | http://logs.openstack.org/50/511950/1/check/legacy-trove-scenario-dsvm-mariadb-single/6be1f2e/ showing POST_FAILURE and job passed | 15:58 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Install git-review for translation proposal job https://review.openstack.org/512095 | 16:00 |
| AJaeger | mordred, similar to your proposal job change ^ | 16:00 |
| pabelanger | mordred: Oh, we did! | 16:01 |
| jeblair | mordred: can you +3 https://review.openstack.org/512084 ? we should restart the executors with that before The Switch. | 16:01 |
| jeblair | i have to go sunday some things, back later | 16:02 |
| pabelanger | grep 6be1f2ea1439428f936b6a926673e48f /var/log/zuul/executor-debug.log | 16:02 |
| mordred | AJaeger: we shouldn't need thatone - the translation proposal job has propose-update as a base job - and the other patch installs git-review in a pre-playbook | 16:02 |
| pabelanger | mordred: jeblair: ^running that on ze01 will expose a failure in logstash playboods | 16:02 |
| pabelanger | 2017-10-15 15:48:07,395 DEBUG zuul.AnsibleJob: [build: 6be1f2ea1439428f936b6a926673e48f] details: NoConnectedServersError('No connected Gearman servers',) | 16:03 |
| AJaeger | mordred: ah! | 16:03 |
| AJaeger | mordred: abandoned | 16:04 |
| AJaeger | fedora26 has problems "dnf.exceptions.Error: Some packages from local repository have incorrect checksum" | 16:05 |
| AJaeger | See http://logs.openstack.org/89/504789/17/infra-check/multinode-integration-fedora-26/1784853/ara/ | 16:05 |
| AJaeger | interesting, base-integration-fedora-26 succeeded | 16:05 |
| pabelanger | so, looks like error is comming from gear directly | 16:06 |
| pabelanger | AJaeger: same error on zuulv2? Maybe our mirror our of sync | 16:06 |
| AJaeger | pabelanger: no idea, only happened with multinode - this is projec-tconfig, so no v2 for this job | 16:07 |
| AJaeger | or openstack-zuul-jobs is change 504789 | 16:07 |
| pabelanger | looking | 16:07 |
| AJaeger | anybody for quick review of adding required-repos for vmware-nsx and networking-odl? https://review.openstack.org/#/c/511836/ and https://review.openstack.org/#/c/511836/ | 16:08 |
| *** e0ne has joined #openstack-infra | 16:09 | |
| *** pcaruana has quit IRC | 16:10 | |
| pabelanger | AJaeger: I'm going to add an autohold and see if we can debug the node | 16:10 |
| mordred | AJaeger: that uses system-required - do we have a patch up to change all of the mergecheck to system-required already? | 16:11 |
| AJaeger | mordred: already in... | 16:11 |
| mordred | AJaeger: ah - great | 16:11 |
| AJaeger | mordred: it's based on old version. Should we rebase first to double check? | 16:11 |
| mordred | AJaeger: nah - I think it's great | 16:12 |
| mordred | AJaeger: you pasted the same link twice above - any chance you have the other link handy? | 16:12 |
| AJaeger | https://review.openstack.org/#/c/509139/ | 16:12 |
| AJaeger | mordred: ^ | 16:12 |
| AJaeger | sorry! | 16:12 |
| AJaeger | https://review.openstack.org/509341 is now also ready - removes magnum jobs from openstack-zuul-jobs | 16:13 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix python3 unicode errors in executor server https://review.openstack.org/512084 | 16:14 |
| AJaeger | Just got "Could not resolve host: mirror.ca-ymq-1.vexxhost.openstack.org" - http://logs.openstack.org/94/512094/2/infra-check/multinode-integration-opensuse423/593529d/ara/ | 16:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Limit base-integration jobs https://review.openstack.org/507558 | 16:15 |
| pabelanger | AJaeger: mordred: dmsimard: lets see about landing ^ to help cut down on curn for ozj. We can do the same for multinode jobs too | 16:15 |
| dmsimard | pabelanger: oh, yeah, mordred mentioned we can filter "cross repo" too. | 16:16 |
| pabelanger | dmsimard: right, but we need to also run jobs on zuul-jobs | 16:16 |
| AJaeger | pabelanger: thanks | 16:16 |
| pabelanger | which, that patch doesn't do | 16:17 |
| pabelanger | so, we can follow up on that | 16:17 |
| dmsimard | pabelanger: what do you mean on zuul-jobs ? we already have jobs on zuul-jobs | 16:17 |
| dmsimard | pabelanger: but they're defined in project-config | 16:17 |
| pabelanger | mordred: jeblair: https://review.openstack.org/511656/ could also use a +3, adds launch-timeout for citycloud, which mostly happens when we upload new images | 16:17 |
| pabelanger | dmsimard: oh, I thought it was removed for some reason | 16:17 |
| pabelanger | AJaeger: re: vexxhost DNS, I wonder if that is related to unbound | 16:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove infra specific pipelines and enable pipelines https://review.openstack.org/510892 | 16:19 |
| AJaeger | pabelanger: yes, that might be. Not sure whether it's worth investigating, just wanted ot mention it... | 16:20 |
| *** claudiub has quit IRC | 16:20 | |
| AJaeger | pabelanger: care to +2A https://review.openstack.org/512089 , please? | 16:20 |
| pabelanger | AJaeger: /etc/resolv.conf on opensuse looks right | 16:21 |
| AJaeger | and also https://review.openstack.org/#/c/512090/ , please | 16:21 |
| AJaeger | pabelanger: good! | 16:21 |
| pabelanger | AJaeger: lets keep an eye on it, until mnaser reports back about networking issue | 16:21 |
| AJaeger | pabelanger: ok | 16:21 |
| *** thorst has joined #openstack-infra | 16:21 | |
| *** thorst has quit IRC | 16:22 | |
| dmsimard | pabelanger: added a comment on the base-integration patch | 16:22 |
| *** Goneri has joined #openstack-infra | 16:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: zuul v3 gate changes for vmware-nsx https://review.openstack.org/511836 | 16:23 |
| dmsimard | man, I can't figure out this wsgi middleware issue :( | 16:23 |
| *** e0ne has quit IRC | 16:24 | |
| pabelanger | looking at fedora-26 issue now | 16:24 |
| openstackgerrit | Merged openstack-infra/project-config master: networking-odl: Add required-projects for tox jobs https://review.openstack.org/509139 | 16:24 |
| *** ykarel has joined #openstack-infra | 16:26 | |
| *** e0ne has joined #openstack-infra | 16:26 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy training-guide jobs https://review.openstack.org/512011 | 16:29 |
| pabelanger | dmsimard: in 504789, we shouldn't be hit rdoproject.org directly. We can use the reverse proxy cache in apache | 16:29 |
| *** e0ne_ has joined #openstack-infra | 16:30 | |
| pabelanger | "rdo_repository": "https://rdoproject.org/repos/openstack-ocata/rdo-release-ocata.rpm" | 16:30 |
| pabelanger | but, that's why the fedora node is failing, just something I noticed | 16:30 |
| AJaeger | dmsimard: did you see jeblair's comment on https://review.openstack.org/#/c/511953/ ? | 16:31 |
| pabelanger | dmsimard: in fact, we likey can stop doing RPM install of the release rpm, and just create the repo list ourself | 16:31 |
| pabelanger | then we don't need to test we can unintsall it | 16:31 |
| *** ykarel has quit IRC | 16:32 | |
| *** e0ne_ has quit IRC | 16:32 | |
| dmsimard | AJaeger: I have earlier, will be able to fix it later | 16:32 |
| *** slaweq has joined #openstack-infra | 16:32 | |
| AJaeger | dmsimard: shall I +A? | 16:32 |
| AJaeger | dmsimard: or do you want to wait? | 16:32 |
| *** e0ne has quit IRC | 16:33 | |
| dmsimard | AJaeger: it can be a follow-up | 16:33 |
| *** e0ne has joined #openstack-infra | 16:34 | |
| AJaeger | ok, I'll +A | 16:34 |
| *** markvoelker_ has quit IRC | 16:34 | |
| AJaeger | mordred: what do you think of https://review.openstack.org/507558 ? | 16:36 |
| dmsimard | AJaeger: in the meantime, it'll at least prevent us from breaking the gate by merging something bad in fetch-zuul-cloner | 16:36 |
| pabelanger | AJaeger: dmsimard: I've left a comment on 504554 too, about rdo-release rpm file also. | 16:36 |
| pabelanger | which has already been merged | 16:36 |
| dmsimard | pabelanger: let me see | 16:37 |
| *** jtomasek has joined #openstack-infra | 16:37 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add required repos for legacy-check-osc-plugins https://review.openstack.org/512026 | 16:40 |
| dmsimard | pabelanger: hm, why would fedora be setting up the rdo repos in the first place, that's probably part of the issue | 16:40 |
| dmsimard | that's a weird failure http://logs.openstack.org/89/504789/17/infra-check/multinode-integration-fedora-26/63202cd/ara/result/92aedddb-4ac6-41cb-a739-0836f4b45783/ | 16:40 |
| dmsimard | but I know where that's coming from | 16:40 |
| pabelanger | yah, that is what I am looking at now | 16:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add integration tests for fetch-zuul-cloner https://review.openstack.org/511953 | 16:42 |
| pabelanger | I think the best step is, template repo list for openvswitch repo, disable it by default, then have package task use --enable-repo when intalling openvswitch, then we don't have to worry about removing .list, as it is disabled | 16:42 |
| pabelanger | we do that with epel and haveged today | 16:43 |
| dmsimard | pabelanger: so the issue is that https://github.com/openstack-infra/zuul-jobs/blob/master/roles/multi-node-bridge/tasks/common.yaml#L17 should not be os_family RedHat | 16:43 |
| dmsimard | pabelanger: it should probably be os_distribution CentOS, right ? | 16:44 |
| pabelanger | dmsimard: is openvswitch in fedora? | 16:44 |
| pabelanger | can't remember | 16:44 |
| pabelanger | if so, they yah, we just want it for centos | 16:44 |
| dmsimard | I agree that we'll want to stop relying on a package to set up the repos | 16:46 |
| dmsimard | pabelanger: fedora has ovs, yeah: https://koji.fedoraproject.org/koji/packageinfo?packageID=13455 | 16:47 |
| dmsimard | pabelanger: let me submit a stopgap fix to address fedora, we can think about templating it after | 16:48 |
| *** e0ne_ has joined #openstack-infra | 16:49 | |
| pabelanger | great | 16:49 |
| *** kiennt26 has joined #openstack-infra | 16:50 | |
| dmsimard | pabelanger: ohh, I know why I did it that way.. | 16:51 |
| *** e0ne has quit IRC | 16:51 | |
| *** e0ne_ has quit IRC | 17:01 | |
| *** e0ne has joined #openstack-infra | 17:03 | |
| *** slaweq has quit IRC | 17:03 | |
| *** edmondsw has joined #openstack-infra | 17:04 | |
| *** jtomasek has quit IRC | 17:04 | |
| dmsimard | pabelanger: is there a bug in gerritbot's config ? I saw a zuul-jobs patch trigger a notification in #zuul but not #openstack-infra but openstack-infra/zuul-jobs is indeed there: https://github.com/openstack-infra/project-config/blob/master/gerritbot/channels.yaml#L1060 | 17:05 |
| AJaeger | "No connected Gearman servers" - is that known? | 17:06 |
| *** e0ne_ has joined #openstack-infra | 17:06 | |
| AJaeger | mordred: yeah, requirements check worked - http://logs.openstack.org/82/509782/7/check/requirements-check/be20757/ but has the above post_failure | 17:07 |
| pabelanger | mordred: jeblair: AJaeger: ^one way to help mitigate the no connected gearman servers | 17:07 |
| pabelanger | remote: https://review.openstack.org/512098 Try 3 times to submit results to logstash.o.o | 17:07 |
| pabelanger | I'm unure how to manually test our ansible library from an executor | 17:08 |
| *** kzaitsev_pi has quit IRC | 17:08 | |
| pabelanger | dmsimard: not sure, would need to check logs for gerritbot | 17:09 |
| *** edmondsw has quit IRC | 17:09 | |
| AJaeger | pabelanger: where does it try 3 times in the change? | 17:09 |
| pabelanger | AJaeger: defaults from ansible, until keyword | 17:09 |
| *** e0ne has quit IRC | 17:09 | |
| AJaeger | pabelanger: Ah, thanks | 17:10 |
| pabelanger | AJaeger: we can pass retries: 3 too | 17:10 |
| *** kzaitsev_pi has joined #openstack-infra | 17:10 | |
| AJaeger | pabelanger: no, I'm fine.. | 17:11 |
| pabelanger | 2017-10-15 17:02:21,774 INFO gerritbot: Sending "David Moreau Simard proposed openstack-infra/zuul-jobs master: Don't set up RDO repositories on Fedora https://review.openstack.org/512097" to #openstack-infra | 17:12 |
| pabelanger | dmsimard: ^ | 17:12 |
| pabelanger | that is in gerritbot logs | 17:12 |
| pabelanger | possible network issue? | 17:13 |
| dmsimard | pabelanger: -1 with comment on https://review.openstack.org/#/c/512098/ | 17:13 |
| dmsimard | pabelanger: yeah maybe | 17:13 |
| AJaeger | pabelanger: could you +2A https://review.openstack.org/#/c/512089/ to move us forward with api-ref jobs, please? | 17:13 |
| pabelanger | dmsimard: ah, ya. I expected that | 17:14 |
| pabelanger | AJaeger: looking | 17:14 |
| pabelanger | AJaeger: Hmm, we should see why emacs was missing on trusty | 17:15 |
| SamYaple | no one needs emacs | 17:15 |
| dmsimard | pabelanger: I don't think the module exposes something that we can use to retry other than the task status itself (fail_json vs exit_json). Maybe, just maybe, we can do until result | succeeded.. but I'm not sure, let me test it | 17:16 |
| pabelanger | AJaeger: +3 | 17:17 |
| AJaeger | thanks | 17:17 |
| pabelanger | dmsimard: cool, thanks | 17:17 |
| pabelanger | AJaeger: also, 510982 is plan B to work around POST_FAILURES for successful jobs | 17:18 |
| AJaeger | ironic that https://review.openstack.org/#/c/512098/ has a post_failure ;) | 17:18 |
| AJaeger | pabelanger: yes, understood | 17:18 |
| pabelanger | AJaeger: but think we are on track for fixing logstash tasks, with gearman connection errors | 17:18 |
| * tosky wonders why https://review.openstack.org/#/c/512094/ is so unstable | 17:19 | |
| pabelanger | looks | 17:19 |
| pabelanger | Timeout (32s) waiting for privilege escalation prompt: | 17:19 |
| pabelanger | that doesn't look right | 17:19 |
| tosky | before it failed twice for other reasons (missing emacs package on trusty) | 17:20 |
| pabelanger | yah | 17:20 |
| tosky | AJaeger already retriggered it more times | 17:20 |
| pabelanger | yah, lets start tracking these failure in etherpad | 17:20 |
| pabelanger | we shouldn't need to recheck these integration jobs | 17:20 |
| dmsimard | pabelanger: we can test the task status, added a comment with example | 17:20 |
| pabelanger | I'm looking at trusty emacs issue now | 17:20 |
| pabelanger | okay, trusty emacs failure was on vexxhost | 17:21 |
| dmsimard | emacs, for the configure_mirror integration tests ? I think I remember troubleshooting that | 17:21 |
| dmsimard | pabelanger: I have a e-r for privilege prompt issues https://review.openstack.org/#/c/505233/ | 17:22 |
| *** thorst has joined #openstack-infra | 17:22 | |
| pabelanger | I'm starting to wonder if we should stop trying to install a package, we mostly want to see if we can update indexes properly with configure-mirror | 17:22 |
| dmsimard | pabelanger: the idea is that we're interested in testing that we're able to install a package, thus making sure the mirrors were configured successfully and work. If there's a failure to install a package in the integration tests, there would have probably been one had it been a legitimate job (like devstack or whatever) | 17:24 |
| *** ykarel has joined #openstack-infra | 17:24 | |
| dmsimard | We can arguably settle for another package than emacs, I just took that because the package is available over all distros we support under the same name and it's not something installed by default | 17:24 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: enhance integration test files section https://review.openstack.org/512100 | 17:27 |
| pabelanger | dmsimard: sure, but at the rate we launch jobs, we are going to see mirror issues from running jobs way before testing configure-mirror role. I think I'd be fine with setting up mirrors, and running update to fetch indexes. I don't think we should be testing that the mirror is actually valid | 17:27 |
| *** thorst has quit IRC | 17:27 | |
| AJaeger | pabelanger, dmsimard ^ I'll rebase once pabelanger's change is in - but this is updates the list of files for triggering the integration tests | 17:27 |
| pabelanger | dmsimard: mostly because, if the mirror isn't valid, we cannot fix it in confgure-mirror role | 17:28 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add role to provice a service_type_data fact https://review.openstack.org/512089 | 17:28 |
| *** lin_yang has quit IRC | 17:29 | |
| dmsimard | pabelanger: if someone sends a patch to configure-mirrors that screws up the mirror configuration, you're not going to be able to tell until you try to install a package | 17:29 |
| dmsimard | pabelanger: the integration test is meant to make sure we don't merge something that breaks that, especially since it's included in all base jobs | 17:29 |
| pabelanger | dmsimard: I'd expect us not to be able to download indexes | 17:30 |
| AJaeger | pabelanger: now we can merge https://review.openstack.org/512086 - to use the service_type_data fact. Could you +2A, please? | 17:31 |
| dmsimard | pabelanger: what's 'indexes' ? The equivalent of apt-cache update/dnf makecache/yum makecache and whatever is the zypper equivalent ? | 17:31 |
| pabelanger | dmsimard: if we install new mirrors, in configure-mirror, we first would apt-get update, dnf update, zypper update? If that passes, to me that is good enough | 17:33 |
| pabelanger | installing emacs, doesn't really add anything more, expect validating repo state is good | 17:33 |
| dmsimard | pabelanger: I think that's a good thing ? I mean, the problem can be either an infrastructure one or a role one | 17:34 |
| dmsimard | you don't know which kind of problem it is until you look | 17:34 |
| AJaeger | wow, far too many post_failures in zuulv3.o.o ;( | 17:34 |
| *** slaweq has joined #openstack-infra | 17:34 | |
| dmsimard | pabelanger: I'm not trying to be annoying here, if we want to take emacs out that's fine, but that's not going to change how legit integration jobs are going to end up failing to install packages | 17:36 |
| *** kiennt26 has quit IRC | 17:36 | |
| dmsimard | whether that's a tripleo job, a devstack job, or whatever | 17:36 |
| pabelanger | dmsimard: right, but that is not a failure of configure-mirror role. It is likely an infrastrucuture issue, like you said. Which is just stopping us from landing code in ozj. | 17:38 |
| pabelanger | lets see what others say, but I don't really see any extra value installing emacs every ozj job run | 17:38 |
| pabelanger | AJaeger: looking | 17:39 |
| AJaeger | pabelanger: thanks. Could you also check https://review.openstack.org/#/c/512090/ ? | 17:39 |
| dmsimard | pabelanger: would adding a retry on the package install alleviate the issue ? | 17:39 |
| *** ykarel has quit IRC | 17:40 | |
| *** ykarel has joined #openstack-infra | 17:41 | |
| clarkb | can you set files on that job to only be for the mirror role? | 17:42 |
| dmsimard | clarkb: yeah we're already working on it | 17:43 |
| pabelanger | clarkb: yah, we have already | 17:43 |
| AJaeger | clarkb: see https://review.openstack.org/#/c/512100/1/zuul.d/jobs.yaml - I'll update that once the parent has landed | 17:43 |
| pabelanger | which will minimize runs | 17:43 |
| AJaeger | (I'll restrict multinode-integration as well) | 17:43 |
| * AJaeger needs to rebase 512100 first... | 17:43 | |
| AJaeger | pabelanger: https://review.openstack.org/512097 is the RDO on Fedora setup change if you want to +2A | 17:44 |
| *** bobh has quit IRC | 17:44 | |
| pabelanger | So, it does look like emacs issue is related to vexxhost | 17:45 |
| *** ykarel has quit IRC | 17:46 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use service-types.openstack.org to get api destinations https://review.openstack.org/512086 | 17:46 |
| pabelanger | AJaeger: +3 | 17:46 |
| dmsimard | pabelanger: should we disable vexxhost until mnaser gets a chance to look at the issue ? | 17:46 |
| *** Goneri has quit IRC | 17:46 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Limit base-integration jobs https://review.openstack.org/507558 | 17:46 |
| dmsimard | I have to afk for a while, have people over -- I can respond to pings if need be | 17:46 |
| AJaeger | pabelanger: https://review.openstack.org/#/c/512090/ as well, please | 17:47 |
| pabelanger | dmsimard: yah, we can see about making that call for zuulv3-rollout | 17:47 |
| pabelanger | looking | 17:47 |
| *** markvoelker has joined #openstack-infra | 17:48 | |
| pabelanger | AJaeger: mordred: +3 with comment | 17:49 |
| pabelanger | on 512090 | 17:50 |
| *** markvoelker has quit IRC | 17:51 | |
| AJaeger | clarkb, pabelanger, dmsimard ^ | 17:51 |
| AJaeger | wait, out bot was silent ;( | 17:52 |
| AJaeger | clarkb, pabelanger, dmsimard : https://review.openstack.org/512100 Limit multinode-integration, update base-integration | 17:52 |
| *** markvoelker has joined #openstack-infra | 17:53 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove openstack-publish-artifacts from legacy-tripleo-ci-dsvm https://review.openstack.org/512094 | 17:53 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Don't set up RDO repositories on Fedora https://review.openstack.org/512097 | 17:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Use propose-updates for requirements-constraints https://review.openstack.org/512090 | 17:56 |
| *** salv-orlando has joined #openstack-infra | 18:00 | |
| * AJaeger will be back later - open review requests are https://review.openstack.org/#/c/512100/ and https://review.openstack.org/512091 | 18:02 | |
| *** Hal has joined #openstack-infra | 18:03 | |
| *** AJaeger has quit IRC | 18:03 | |
| *** Hal is now known as Guest5873 | 18:03 | |
| *** slaweq has quit IRC | 18:08 | |
| *** liusheng has quit IRC | 18:08 | |
| *** armaan has quit IRC | 18:12 | |
| *** armaan_ has joined #openstack-infra | 18:12 | |
| *** liusheng has joined #openstack-infra | 18:16 | |
| *** modrednetfail has quit IRC | 18:16 | |
| *** slaweq has joined #openstack-infra | 18:17 | |
| *** sbezverk has joined #openstack-infra | 18:18 | |
| pabelanger | infra-root: I am thinking our post playbook failure task issue (POST_FAILURES on successful jobs) might be a top issue to address before rolling out this evening. This is just going to result in an infulx of rechecks, and user confusion. | 18:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Try 3 times to submit results to logstash.o.o https://review.openstack.org/512098 | 18:22 |
| *** thorst has joined #openstack-infra | 18:24 | |
| *** yamahata has quit IRC | 18:28 | |
| *** thorst has quit IRC | 18:30 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Test elements have README.rst file https://review.openstack.org/177832 | 18:36 |
| *** baoli has joined #openstack-infra | 18:40 | |
| *** e0ne_ has quit IRC | 18:41 | |
| *** thorst has joined #openstack-infra | 18:45 | |
| *** thorst has quit IRC | 18:45 | |
| *** AJaeger has joined #openstack-infra | 18:47 | |
| *** e0ne has joined #openstack-infra | 18:48 | |
| AJaeger | pabelanger: I agree, the POST_FAILURES are currently our biggest - and only known - problem. | 18:48 |
| *** gouthamr has joined #openstack-infra | 18:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Legacy ansible-func jobs require openstack/requirements https://review.openstack.org/512102 | 18:57 |
| *** dhinesh has quit IRC | 19:01 | |
| *** erlon has quit IRC | 19:02 | |
| *** thorst has joined #openstack-infra | 19:05 | |
| boden | hi… I’ve recently started getting a POST_FAILURE in our openstack-tox-pep8 job https://review.openstack.org/#/c/511542 any ideas why would be apprecaited | 19:10 |
| pabelanger | boden: yah, we are aware of the issue. Working towards resolving it | 19:11 |
| boden | pabelanger ah ok.. thought maybe it was something I did | 19:11 |
| boden | thx | 19:11 |
| pabelanger | boden: see 512098 and 510982 for workaround | 19:12 |
| *** Goneri has joined #openstack-infra | 19:13 | |
| * AJaeger calls it a day, sorry, won't be around for the fun of switching ;) Hope it works out fine... | 19:16 | |
| *** AJaeger has quit IRC | 19:16 | |
| clarkb | that runs on the executors right? | 19:17 |
| clarkb | can we double check all executors can connec to the gear server on logstash.o.o? | 19:18 |
| *** slaweq has quit IRC | 19:18 | |
| pabelanger | clarkb: yah, I haven't confirmed that yet | 19:18 |
| *** e0ne has quit IRC | 19:19 | |
| SamYaple | are e still t minus 3 hours from zuulv3? or have issues popped up that block that | 19:20 |
| clarkb | I don't think this blocks us, worst case we just ignore failures on that task while we debug it | 19:20 |
| *** slaweq has joined #openstack-infra | 19:21 | |
| *** dhajare has joined #openstack-infra | 19:21 | |
| pabelanger | clarkb: yah, I suggest we land 510982, then work to revert it. Unless we can figure out why we fail on connecting to gearman | 19:22 |
| fungi | yeah, we seem to have confirmed the post failures at least mostly stem from logstash indexing, so worst case we make it always claim to have succeeded and live with gaps in logstash until we work out the cause | 19:22 |
| *** Goneri has quit IRC | 19:27 | |
| pabelanger | clarkb: fungi: do we have any logs on logstash.o.o we can check for gearman errors? | 19:28 |
| fungi | i guess log-gearman-client.py is the gearman server? | 19:31 |
| pabelanger | yah, thought so too, but seems to be empty | 19:31 |
| pabelanger | will see if I can find out why | 19:31 |
| pabelanger | fungi: clarkb: if you are also in the mood to review: https://review.openstack.org/512102/ will help zuulv3 jobs for ansible projects | 19:32 |
| fungi | pabelanger: i was just looking at a legacy-ansible-func-centos-7 failure over missing openstack/requirements in its required-projects | 19:33 |
| fungi | was about to try and find whether someone had already claimed it in the issues pad | 19:33 |
| pabelanger | yah | 19:33 |
| pabelanger | I haven't added it to zuulv3-issue, I'll do that now | 19:34 |
| fungi | so anyway, approved! | 19:34 |
| fungi | it's like i imagine the solutions to these problems and suddenly patches materialize | 19:34 |
| pabelanger | wouldn't that be the best | 19:34 |
| fungi | yeah, too bad it doesn't occur with greater frequency | 19:38 |
| fungi | infra-root: reminder, the rollout/rollback plan for tonight is at https://etherpad.openstack.org/p/zuulv3-rollout if anyone else wants to double-check it before we start | 19:39 |
| clarkb | it forks a geard like zuul does iirc | 19:39 |
| clarkb | so should have its own logs | 19:40 |
| jeblair | i don't see anything about the gearman error in the etherpad | 19:40 |
| fungi | we should probably send a one-hour warning as a #status notice in about an hour and 20 minutes | 19:40 |
| jeblair | is it still an issue? | 19:40 |
| fungi | jeblair: i think pabelanger only just confirmed it from the executor debug log a couple hours ago | 19:40 |
| fungi | but yeah, there were a rash of post_failure states on otherwise successful jobs just a little bit ago | 19:41 |
| jeblair | we don't even need to confirm things before adding them to the etherpad -- we have a triage section | 19:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Legacy ansible-func jobs require openstack/requirements https://review.openstack.org/512102 | 19:41 |
| *** armaan_ has quit IRC | 19:42 | |
| jeblair | i will start adding things | 19:42 |
| jeblair | but if we're going to be an effective distributed team, we need to record *everything* | 19:42 |
| pabelanger | agree, I've only logged them in IRC currently. Will review / update fungi entries | 19:43 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 19:43 |
| jeblair | ugh, apparently we get an exception but no trace back | 19:45 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 19:45 |
| fungi | any objection to me archiving the current mass of fixed issues in the zuulv3-issues pad to the zuulv3-fixed-issues pad? | 19:45 |
| fungi | it's taking up nearly half the issues pad again already at this point | 19:46 |
| pabelanger | wfm | 19:46 |
| fungi | aha, and in so doing i see the current concern with the post_failure results got moved to the fixed issues section after a debugging patch was merged. not sure that's the sanest way of tracking things | 19:47 |
| dmsimard | pabelanger: +1 with a comment on https://review.openstack.org/#/c/512098/ | 19:49 |
| pabelanger | dmsimard: yah, you are not wrong | 19:50 |
| pabelanger | dmsimard: feel free to -1 until I fix | 19:50 |
| dmsimard | ok | 19:50 |
| *** pcaruana has joined #openstack-infra | 19:51 | |
| *** slaweq has quit IRC | 19:54 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy proposal jobs https://review.openstack.org/512091 | 19:55 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy magnum jobs https://review.openstack.org/509341 | 19:55 |
| jeblair | the previous errors all seemed to happen around in a cluster around the same time (twice) | 19:56 |
| jeblair | that corresponds with an increased load average on the logstash server -- it almost reached 1. | 19:56 |
| pabelanger | Yah, I was thinking maybe it was server side also. I haven't been able to see where our gearman server logging goes | 19:57 |
| jeblair | this is still pretty perplexing because it should be waiting for a connection and only proceeding when it gets one | 19:57 |
| jeblair | pabelanger: you proposed one or more mitigation patches, right? can you link them? | 19:58 |
| jeblair | i've started a tcpdump on the logstash server on the gear port; hopefully that will shed light on it if it happens again | 19:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Try 3 times to submit results to logstash.o.o https://review.openstack.org/512098 | 19:59 |
| pabelanger | jeblair: ^ is 1 | 19:59 |
| pabelanger | jeblair: https://review.openstack.org/510982/ is 2nd | 19:59 |
| jeblair | pabelanger: i'm in favor of the 2nd now | 20:00 |
| *** pcaruana has quit IRC | 20:00 | |
| pabelanger | yah, worth a try | 20:00 |
| jeblair | pabelanger: i don't think i want to do the first -- i think it may actually make the situation worse. and if we've managed to submit some jobs before failing, we could end up submitting duplicates which could be weird | 20:00 |
| pabelanger | kk | 20:01 |
| pabelanger | jeblair: shall I remove WIP on 1st? | 20:01 |
| jeblair | pabelanger: ya, maybe just go ahead and +W it | 20:02 |
| pabelanger | sure | 20:02 |
| pabelanger | done | 20:02 |
| pabelanger | I'll WIP the other | 20:02 |
| *** slaweq has joined #openstack-infra | 20:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add 90s connect timeout for submit log jobs https://review.openstack.org/512105 | 20:04 |
| jeblair | this is *not* a solution to the problem, but while i was looking at it, i thought about that ^. we may appreciate having that in there some day in the future. | 20:04 |
| jeblair | we could also shorten that -- i was just thinking about maybe a really overburdened executor. | 20:05 |
| clarkb | could it be remote running out of file descriptors? | 20:05 |
| jeblair | clarkb: yes. | 20:05 |
| clarkb | is it connection per file or per job? | 20:05 |
| jeblair | connection per job, but maybe too many jobs finished at once | 20:06 |
| jeblair | hrm, we should see this on the connection graph though | 20:06 |
| jeblair | there are only tiny spikes on the tcp connection graph around the times we got errors | 20:06 |
| *** baoli has quit IRC | 20:08 | |
| jeblair | it's possible that they were very brief, since the spike time periods were long enough to show up on the graph at all, i would expect to see the count get significantly higher | 20:08 |
| jeblair | so, possible but lack of evidence for the moment | 20:09 |
| openstackgerrit | Merged openstack-infra/project-config master: base: ignore_errors for logstash processing https://review.openstack.org/510982 | 20:09 |
| ianw | o/ | 20:10 |
| jeblair | i'm going to restart the executors now, to pick up the unicode fix | 20:10 |
| pabelanger | ++ | 20:12 |
| dmsimard | ianw: hey o/ | 20:14 |
| jeblair | fascinating | 20:14 |
| jeblair | 2 executors did not shutdown cleanly | 20:14 |
| dmsimard | ianw: I have my middleware thing half working | 20:14 |
| jeblair | one of them has an open finger connection | 20:14 |
| jeblair | i wonder if that's keeping it alive | 20:14 |
| jeblair | (it's not so much the executor itself as a subprocess i think) | 20:15 |
| jeblair | i'm going to stop zuul-web and see if this dies | 20:15 |
| jeblair | yes, both gone now | 20:16 |
| jeblair | adding to etherpad | 20:16 |
| pabelanger | good to know | 20:16 |
| jeblair | executors have been restarted (also zuul-web) | 20:17 |
| *** d0ugal_ has joined #openstack-infra | 20:20 | |
| *** d0ugal has quit IRC | 20:21 | |
| *** claudiub has joined #openstack-infra | 20:32 | |
| *** slaweq has quit IRC | 20:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-windmill jobs https://review.openstack.org/512110 | 20:39 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove windmill from zuul-legacy-project-templates https://review.openstack.org/512114 | 20:46 |
| *** slaweq has joined #openstack-infra | 20:47 | |
| *** felipemonteiro has joined #openstack-infra | 20:49 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:51 | |
| *** felipemonteiro has quit IRC | 20:54 | |
| *** nunchuck has joined #openstack-infra | 20:54 | |
| clarkb | logstash's gear process seems to have up to 8192 fds and currently has a couple hundred so I doubt that is the issue | 21:02 |
| *** e0ne has joined #openstack-infra | 21:02 | |
| *** lennyb has quit IRC | 21:02 | |
| pabelanger | we have a few locked ready nodes, over 3hours and counting, 2 of them are over 1 day | 21:07 |
| pabelanger | trying to see why that would be | 21:07 |
| clarkb | I've tested telnet to logstash:4730 on all executors, ze10 felt more laggy but it eventually gets the status output | 21:07 |
| *** baoli has joined #openstack-infra | 21:08 | |
| pabelanger | 2 ubuntu-trusty nodes, which are ready and locked for 1d8hrs seem to be from periodic wheel-build jobs for openstack/requirements | 21:09 |
| pabelanger | 300-0000423830 is request ID, but not sure why still locked | 21:09 |
| *** hongbin has joined #openstack-infra | 21:10 | |
| pabelanger | adding it to etherpad zuulv3-issues | 21:10 |
| pabelanger | they are in citycloud-kna1, so it is possible related to quote | 21:11 |
| pabelanger | quota* | 21:11 |
| pabelanger | I've added them to issues with nodepool | 21:11 |
| fungi | status notice Zuul v3 rollout maintenance begins at 22:00 UTC (roughly 45 minutes from now): http://lists.openstack.org/pipermail/openstack-dev/2017-October/123618.html | 21:12 |
| pabelanger | going to grab food with family quickly before rollout starts | 21:12 |
| fungi | that look good to send in a few minutes? | 21:12 |
| pabelanger | fungi: ++ | 21:12 |
| *** baoli has quit IRC | 21:13 | |
| clarkb | fungi: ya | 21:15 |
| clarkb | looks like es07 is out to lunch | 21:16 |
| clarkb | es logs show it happened on the 13th | 21:16 |
| clarkb | I can't hit it via ssh | 21:16 |
| clarkb | going to try a reboot via openstack ap | 21:16 |
| fungi | #status notice Zuul v3 rollout maintenance begins at 22:00 UTC (roughly 45 minutes from now): http://lists.openstack.org/pipermail/openstack-dev/2017-October/123618.html | 21:17 |
| openstackstatus | fungi: sending notice | 21:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Remove bashate legacy jobs https://review.openstack.org/512117 | 21:17 |
| clarkb | (noticed when digging into logstash related stuff) | 21:17 |
| Shrews | oh my. lots of weekend scrollback | 21:18 |
| -openstackstatus- NOTICE: Zuul v3 rollout maintenance begins at 22:00 UTC (roughly 45 minutes from now): http://lists.openstack.org/pipermail/openstack-dev/2017-October/123618.html | 21:18 | |
| openstackstatus | fungi: finished sending notice | 21:20 |
| *** felipemonteiro_ has quit IRC | 21:20 | |
| *** slaweq has quit IRC | 21:20 | |
| clarkb | it pings but doesn't ssh guessing due to fsck | 21:20 |
| * clarkb tries to be patient | 21:20 | |
| evrardjp | thanks for the notice and happy rollout! | 21:21 |
| *** e0ne has quit IRC | 21:21 | |
| *** slaweq has joined #openstack-infra | 21:23 | |
| *** e0ne has joined #openstack-infra | 21:24 | |
| *** bobh has joined #openstack-infra | 21:24 | |
| *** armax has joined #openstack-infra | 21:26 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Move bashate jobs in-tree https://review.openstack.org/512118 | 21:26 |
| clarkb | ssh -vvv seems to indicate port 22 is listening | 21:27 |
| clarkb | just no one is home | 21:27 |
| clarkb | anyone else have better luck than me to elasticsearch07.openstack.org? | 21:27 |
| ianw | clarkb: it does not seem to like australians either | 21:30 |
| clarkb | will probably need to dig in via the console once zuul things are done | 21:31 |
| *** e0ne has quit IRC | 21:31 | |
| *** wolverineav has quit IRC | 21:31 | |
| pabelanger | same, on es07, down | 21:32 |
| pabelanger | Shrews: I've added an issue to nodepool section, FYI | 21:33 |
| pabelanger | Shrews: I _think_ it might be quota related, but not 100% sure | 21:33 |
| Shrews | pabelanger: possibly. how many nodes did the request those are for request? | 21:34 |
| ianw | hmm, when you watch this console scroll past more slowly you see things ... all the java deps, generating a bunch of odd locales, etc. few things to dig into there | 21:34 |
| pabelanger | Shrews: where should I look again for that? | 21:34 |
| Shrews | pabelanger: nodepool request-list | 21:34 |
| clarkb | ianw: that for es07? | 21:34 |
| clarkb | ianw: or bashate? | 21:35 |
| ianw | clarkb: sorry, no, just noticing things on zuulv3 jobs that i probably didn't before as i never really watched the console scroll by | 21:35 |
| clarkb | ah | 21:35 |
| *** lennyb has joined #openstack-infra | 21:35 | |
| pabelanger | Shrews: okay, so i did check that, but I don't see a matching request ID | 21:35 |
| * mordred waves to all the nice people | 21:36 | |
| pabelanger | mordred: ^5 | 21:36 |
| Shrews | pabelanger: hmm | 21:36 |
| Shrews | pabelanger: if the request is gone, then it's not a quota issue | 21:37 |
| Shrews | we need to see who has those locked | 21:37 |
| pabelanger | I think zuul did | 21:40 |
| pabelanger | I can see the request ID in zuulv3 debug.log.1.gz | 21:40 |
| pabelanger | but not sure I understand what is happening | 21:40 |
| clarkb | there are hung process timeouts from the kernel on es07's console | 21:41 |
| clarkb | its up and login prompt is there but it looks like maybe an unhappy hypervisor | 21:41 |
| clarkb | I'm going to leave it there though and focus on zuul nowish | 21:41 |
| *** eumel8 has quit IRC | 21:44 | |
| jeblair | o/ | 21:51 |
| fungi | i have basically finished my baking tasks for today (and also recompiled a kernel package to enable a staging tree driver for an sdio-based wifi interface) | 21:51 |
| fungi | ready to get down and get zuul | 21:51 |
| fungi | i guessing no objections/concerns with https://etherpad.openstack.org/p/zuulv3-rollout | 21:52 |
| jeblair | fungi: lgtm | 21:52 |
| fungi | the pipeline export/import stuff was more relevant when this was a thursday plan. may not be super relevant at today's volume | 21:53 |
| clarkb | my only suggestion would be to maybe move the nodepool quotas first as that should help things get merged more quickly | 21:53 |
| clarkb | but either order should work fine | 21:53 |
| pabelanger | +1 | 21:54 |
| mordred | ++ | 21:54 |
| fungi | yeah, i suppose shifting quota first sounds better | 21:54 |
| fungi | and we could get that rolling now if we wanted? | 21:54 |
| jeblair | fungi: ++ | 21:55 |
| mordred | ++ | 21:55 |
| pabelanger | ++ | 21:55 |
| jeblair | that's https://review.openstack.org/511069 | 21:55 |
| mordred | +2 from me | 21:55 |
| fungi | +3 | 21:56 |
| mordred | I've got "/opt/system-config/production/tools/kick.sh 'nodepool.openstack.org:nl0*.openstack.org'" prepped and ready to go | 21:58 |
| Shrews | pabelanger: i'm confused as to why those nodes are still locked | 22:00 |
| * fungi pulls up browser and watches virtual paint peel | 22:00 | |
| pabelanger | Shrews: yah, me also | 22:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Shift nodepool quota from v2 to v3"" https://review.openstack.org/511069 | 22:01 |
| clarkb | mordred: ^ | 22:01 |
| mordred | woot. on it | 22:01 |
| fungi | i guess we can status notice that we're starting maintenance now... probably no need for alert since we're not really taking stuff down in ways that will be overly apparent to users (other than people staring at the old status screen)? | 22:01 |
| pabelanger | Shrews: 0000230528 is another node that I don't know why it is locked and ready | 22:02 |
| pabelanger | Shrews: again, nothing in nodepool request-list | 22:02 |
| clarkb | fungi: ya | 22:02 |
| jeblair | fungi: ++ | 22:02 |
| pabelanger | fungi: wfm | 22:02 |
| fungi | status notice Zuul v3 rollout maintenance is underway, scheduled to conclude by 23:00 UTC: http://lists.openstack.org/pipermail/openstack-dev/2017-October/123618.html | 22:02 |
| fungi | that look good? | 22:02 |
| jeblair | fungi: ++ | 22:03 |
| fungi | #status notice Zuul v3 rollout maintenance is underway, scheduled to conclude by 23:00 UTC: http://lists.openstack.org/pipermail/openstack-dev/2017-October/123618.html | 22:03 |
| pabelanger | ++ | 22:03 |
| openstackstatus | fungi: sending notice | 22:03 |
| -openstackstatus- NOTICE: Zuul v3 rollout maintenance is underway, scheduled to conclude by 23:00 UTC: http://lists.openstack.org/pipermail/openstack-dev/2017-October/123618.html | 22:04 | |
| mordred | kick is done | 22:04 |
| * smcginnis grabs popcorn for Zuul 3: The Awakening | 22:05 | |
| pabelanger | updating zuulv3-rollout | 22:05 |
| fungi | smcginnis: well, more like the reanimation | 22:05 |
| smcginnis | ;) | 22:05 |
| tosky | smcginnis: I thought it was Zuul 3: Return of the Job | 22:06 |
| fungi | heh | 22:06 |
| * mordred has +2d the infra-pipeline revert patches | 22:06 | |
| smcginnis | Zuul 3, A New Hope? | 22:07 |
| mordred | wouldn't that be 4? ;) | 22:07 |
| fungi | so for line 4, are we missing changes for zuul-jobs or project-config (just infra-gate and infra-post dropping from the latter)? | 22:07 |
| pabelanger | confirmed, both nodepool and nodepool-launcher appear correct | 22:07 |
| smcginnis | mordred: Depends if you actually want to count all of the first three. ;) | 22:08 |
| fungi | ahh, 510892 is project-config, down on line 7 | 22:08 |
| jeblair | i saved the queues | 22:08 |
| jeblair | i'll work on porting the gate script to v3 | 22:08 |
| jeblair | er it's one line | 22:08 |
| jeblair | done | 22:08 |
| fungi | jeblair: for the record, "porting is just appending --tenant=openstack right? or is there more? | 22:09 |
| pabelanger | how did we handle ready nodes in nodepool.o.o last time? | 22:09 |
| jeblair | fungi: that's it | 22:09 |
| pabelanger | they should get released once zuul-launcher is stopped right? | 22:09 |
| fungi | k\ | 22:09 |
| jeblair | all the patches we need to land are gated by v3 now, right? | 22:10 |
| jeblair | so i can stop v2? | 22:10 |
| *** baoli has joined #openstack-infra | 22:10 | |
| fungi | yes | 22:10 |
| *** wolverineav has joined #openstack-infra | 22:10 | |
| jeblair | done | 22:10 |
| mordred | buh-bye | 22:10 |
| pabelanger | \o/ | 22:10 |
| clarkb | I'm confused by 0892 still has infra-check in it | 22:11 |
| clarkb | is that a bug? | 22:11 |
| jeblair | clarkb: we're keeping infra-check around so we can land project-config changes quickly | 22:11 |
| fungi | clarkb: we're retaining infra-check solely for project-config | 22:11 |
| fungi | (temporarily) | 22:11 |
| mordred | clarkb: it's not a bug, we're keeping it :) | 22:11 |
| jeblair | clarkb: (it is high-priority, so will return faster than normal check pipeline) | 22:11 |
| clarkb | gotcha | 22:11 |
| clarkb | we didn't want to leave the other repos on that pipeline too? ozj zuul etc? | 22:12 |
| jeblair | clarkb: nah, depends-on works with those, so we're not in such a bad place | 22:12 |
| *** bobh has quit IRC | 22:12 | |
| jeblair | (and we could still force-enqueue in gate if needed urgently for something widespread) | 22:12 |
| clarkb | are we ready to start approving these changes then? | 22:13 |
| fungi | also things landing to those other repos will mostly be under the guidance of people with the ability to enqueue them directly to the gate pipeline if there is an urgent reason to do so | 22:13 |
| *** dhajare has quit IRC | 22:13 | |
| jeblair | clarkb: i think so | 22:13 |
| fungi | er, what jeblair said | 22:13 |
| *** baoli has quit IRC | 22:14 | |
| * clarkb goes to approve things | 22:14 | |
| *** wolverineav has quit IRC | 22:14 | |
| pabelanger | do we need to do anything to nodepool.o.o to clean out ready nodes? | 22:14 |
| jeblair | pabelanger: yeah, deleting those at this point is a good idea | 22:14 |
| clarkb | I approved the first 3 | 22:14 |
| pabelanger | jeblair: okay, on it | 22:15 |
| fungi | i went ahead and put old zuul.o.o in the emergency disable list so we don't need to wait for puppet to finish a round later | 22:16 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Revert "Use new post pipelines" https://review.openstack.org/510887 | 22:17 |
| mordred | fungi: + | 22:17 |
| *** rkukura_ has quit IRC | 22:17 | |
| clarkb | the zuul changes don't appear to want to queue | 22:17 |
| clarkb | I'm guessing because we told them to use gate but the gate criteria are not currently satisfiable | 22:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Move statsd to correct zuul.conf file https://review.openstack.org/512120 | 22:17 |
| mordred | clarkb: maybe recheck? | 22:17 |
| clarkb | mordred: no I think its not going to work bceause gate needs +3 | 22:18 |
| clarkb | I think we need https://review.openstack.org/#/c/510892/ in first so the depends on order is wrong | 22:18 |
| clarkb | pabelanger: ^ fyi | 22:18 |
| clarkb | should I go ahead and remove the depends on to the zuul changes and approve 0892? | 22:19 |
| mordred | clarkb: I think we need to split https://review.openstack.org/#/c/510892/ | 22:19 |
| clarkb | I don't think we need to split it | 22:19 |
| clarkb | we just need to merge the zuul changes afterwards | 22:20 |
| mordred | clarkb: we need to do the bit with the pipeline definition - then the other 3 patches, then https://review.openstack.org/#/c/510892/ | 22:20 |
| pabelanger | yah, I orginally left a comment to split 510892 up | 22:20 |
| clarkb | mordred: why? | 22:20 |
| mordred | clarkb: we can't delete infra-gate and infra-post until we've landed the other 3 changes | 22:20 |
| *** sdague has joined #openstack-infra | 22:20 | |
| clarkb | mordred: I don't think that is tre | 22:20 |
| *** armax has quit IRC | 22:21 | |
| clarkb | one of the three already merged and it is the only one that needs to go first | 22:21 |
| clarkb | the other two should work with gate once criteria are upated (since that is why they are not gating now) | 22:21 |
| mordred | right - but the criteria update patch also deletes the infra-gate - which is a syntax error until the patch stopping using infra-gate lands | 22:22 |
| clarkb | mordred: right the other two patches stop using infra-gate | 22:22 |
| clarkb | which is why they won't gate now | 22:22 |
| jeblair | we should have a change which only re-enables the pipelines | 22:23 |
| jeblair | then we should switch everything to use those pipelines | 22:23 |
| mordred | yes. I'm working on that now | 22:23 |
| jeblair | then we should delete the unused pipelines | 22:23 |
| clarkb | I'm still not understanding why that is necessary | 22:23 |
| clarkb | the two zuul changes should work once 0892 is in because they switch to using the normal gate pipeline | 22:23 |
| mordred | no - it'll be a broken zuul config | 22:24 |
| clarkb | this is why they can't gate right now | 22:24 |
| jeblair | clarkb: except that zuul currently uses those pipelines | 22:24 |
| mordred | because those configs refernece infra-gate | 22:24 |
| clarkb | bceause they switch to gate | 22:24 |
| jeblair | clarkb: 0892 removes them | 22:24 |
| clarkb | mordred: they delete infra-gate | 22:24 |
| *** markvoelker has quit IRC | 22:24 | |
| jeblair | clarkb: right, which is why 0892 depends on them | 22:24 |
| mordred | yes. but they still use infra-gate before theyland | 22:24 |
| mordred | the repos do | 22:24 |
| clarkb | https://review.openstack.org/#/c/510886/3/.zuul.yaml and https://review.openstack.org/#/c/510885/2/.zuul.yaml | 22:24 |
| mordred | the consistent global zuul config | 22:24 |
| jeblair | clarkb: if we were to hypothetically force-merg 510892 right now, zuul's config would be broken | 22:24 |
| mordred | landing the project-config patch is a syntax error | 22:24 |
| jeblair | clarkb: because the zuul repo has a .zuul.yaml which references a pipeline which no longer would exist | 22:25 |
| clarkb | I see. Is that something we can make zuul handle more gracefully? eg if a trusted repo redefines things then just ignore the untrusted? | 22:25 |
| jeblair | clarkb: if you want to confirm, you can go ahead and drop the depends-on from 510892 and zuul should report that error in just a couple seconds | 22:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Re-enable zuul v3 pipelines https://review.openstack.org/512121 | 22:25 |
| clarkb | Seems like we would be in a really weird spot if we ever tried to move things around in the future otherwise | 22:26 |
| mordred | clarkb: I think what zuul is doing is exactly correct | 22:26 |
| jeblair | clarkb: zuul won't let us | 22:26 |
| mordred | anyway - that patch ^^ should work | 22:26 |
| clarkb | right we'd have to go hunt down all the random places things are used in potentially thousands of repos | 22:26 |
| jeblair | it is not possible to break zuul's config as long as zuul is gating | 22:26 |
| clarkb | ya the weird spot isn't zuul would be broken its affecting the change you want in the system | 22:27 |
| mordred | clarkb: yes. zuul will not allow us to break people's job configs in their repos - that's a good thing - it means we physically cannot land a change to the global config that would break the job config that is in someone's repo ... if there are 1000s of repos using a thing, we likely shouldn't be breaking it | 22:27 |
| jeblair | large systemic changes will require more planning and communication in the future. | 22:27 |
| mordred | yup | 22:27 |
| jeblair | that's not bad, it's just different. :) | 22:28 |
| SamYaple | so zuul actually loads all the configs from openstack/* repos everytime it runs? | 22:28 |
| clarkb | I can imagine situations where not being able to update config without first updating thousands of repos is a bad thing :) | 22:28 |
| jeblair | SamYaple: yes. it uses its fleet of mergers and executors to grab every config file from every branch of every project. it currently takes 6 minutes from a cold start. | 22:29 |
| jeblair | SamYaple: (that will get faster after we cannibalize some mergers from v2) | 22:29 |
| jeblair | clarkb: like? | 22:29 |
| SamYaple | yowza | 22:29 |
| clarkb | jeblair: mostly worried about needing to deal with all the dead repos out there | 22:29 |
| jeblair | clarkb: if they're dead, drop them from zuul. | 22:29 |
| mordred | yah | 22:29 |
| SamYaple | all this talk about unique named jobs and global namespaces makes alot more sense now | 22:29 |
| jeblair | SamYaple: the cool thing is -- it *only* takes 6 minutes! there are about 6000 project-branches in the system. | 22:30 |
| mordred | SamYaple: the config is also updated automatically every time a change lands to any of the zuul configs in any of the repos - plus there is the speculative config bulding | 22:30 |
| clarkb | jeblair: mordred so if you don't update within X time you will be removed from the system? | 22:30 |
| pabelanger | SamYaple: Yah, that was part of the work jeblair did this past week to make reloads faster! Last roll out it was taking about 50secs pre reload, not I think it is down to a few seconds | 22:30 |
| jeblair | SamYaple: yeah, once loaded, zuul config is incrementally updated in about 6 seconds | 22:30 |
| SamYaple | jeblair: yea im not saying thats a long time. im just impressed by the scale | 22:30 |
| mordred | clarkb: yah- but this is also for systemic things that should be very infrequent ... such as "remove the gate pipeline" or "rename the base job" | 22:30 |
| *** esberglu has joined #openstack-infra | 22:31 | |
| clarkb | mordred: I think a good example in the past was how we redefined release and tag pipelines at one point but ya | 22:31 |
| mordred | clarkb: that won't be a problem | 22:31 |
| mordred | clarkb: it's only the name reference that's an issue - so deleting things from project-config will take effort | 22:32 |
| SamYaple | i cant imagine python is nearly the fastest (or memory effiecent) way of pulling in all that stuff either | 22:32 |
| mordred | clarkb: so redefining the qualities of the release pipeline - no issue - deleting the release pipeline - hard | 22:32 |
| *** markvoelker has joined #openstack-infra | 22:32 | |
| SamYaple | the mind boggles | 22:32 |
| clarkb | also removing eg silent (which has been talked about, but maybe if people are using it then we should just keep it around) | 22:32 |
| mordred | clarkb: yah - we a) should totally remove it... as well as experimental | 22:33 |
| jeblair | clarkb: zuul answers the question "is anyone still using this?" very quickly :) | 22:33 |
| SamYaple | haha | 22:33 |
| pabelanger | indeed | 22:33 |
| mordred | but b) we just have to clean house first | 22:33 |
| mordred | also - yah - we should delete experimental pretty soon after this dust settles | 22:33 |
| jeblair | mordred: i'm not sure we should remove those | 22:33 |
| mordred | no? | 22:33 |
| SamYaple | why remove experimental? | 22:34 |
| *** salv-orlando has quit IRC | 22:34 | |
| mordred | experimental has no purpose in a world with speculative job configs from my pov - just propose a patch with the job you want to be experiemntal | 22:34 |
| mordred | and depends-on/recheck it as needed | 22:34 |
| jeblair | mordred: it's not just an on-ramp for new jobs, it's also "on-demand check jobs". | 22:34 |
| clarkb | mordred: I think its purpose would be having predefined useful jobs you want to run occasionally | 22:34 |
| jeblair | maybe it needs a new name | 22:34 |
| fungi | the experimental pipeline is arguably a bit more convenient than reparenting a change that tests adding the jobs you're curious to see whether or not work | 22:34 |
| SamYaple | fungi: ++ | 22:34 |
| mordred | yah - I can see on-demand check jobs you want to use occasionally - the 'experimental' use case has a whole different mechanism now | 22:35 |
| jeblair | mordred: i agree we should drop it from the process for adding new jobs though | 22:35 |
| mordred | jeblair: ++ | 22:35 |
| jeblair | maybe call it 'extra' or something | 22:35 |
| fungi | experimental is less useful for testing out new jobs now, but it's nice for jobs you want to run infrequently on-demand for specific changes | 22:35 |
| jeblair | or frank | 22:35 |
| jeblair | "check more" | 22:35 |
| SamYaple | check moar | 22:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Re-enable zuul v3 pipelines https://review.openstack.org/512121 | 22:35 |
| fungi | check harder | 22:35 |
| jeblair | it can have more than one trigger even :) | 22:35 |
| * clarkb rechecks the two zuul changes | 22:36 | |
| SamYaple | i would like to purpose aliases as "harder, better, faster, stronger" | 22:36 |
| * jeblair plays 4 non blondes | 22:37 | |
| pabelanger | we're up to 234 nodes 'in-use' on nodepool-launchers now | 22:37 |
| *** markvoelker has quit IRC | 22:37 | |
| SamYaple | jeblair: heh. theres something i havent heard in a while | 22:37 |
| jeblair | "bigger, better, faster, more" | 22:37 |
| mordred | clarkb: I see them in the gate now | 22:37 |
| SamYaple | ya | 22:37 |
| pabelanger | and load on zuulv3.o.o is nice and low | 22:38 |
| pabelanger | Yah, i see 510886,3 in gate pipeline | 22:39 |
| clarkb | mordred: did you want to rebase 0892 as it should confict with 12121 now | 22:39 |
| clarkb | I guess we want zuul changes to merge first | 22:39 |
| clarkb | so that it can check validity of new ocnfig properly? | 22:39 |
| mordred | clarkb: yah - let's rebase when they land | 22:41 |
| *** markvoelker has joined #openstack-infra | 22:41 | |
| openstackgerrit | Merged openstack-infra/zuul master: Revert "Use new infra pipelines" https://review.openstack.org/510886 | 22:44 |
| pabelanger | woot | 22:44 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Revert "Use new infra pipelines" https://review.openstack.org/510885 | 22:45 |
| clarkb | thats both of them | 22:46 |
| *** markvoelker has quit IRC | 22:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove infra specific pipelines and enable pipelines https://review.openstack.org/510892 | 22:46 |
| mordred | there's that rebased | 22:46 |
| *** dizquierdo has joined #openstack-infra | 22:48 | |
| fungi | are we good to approve it at this stage? | 22:48 |
| mordred | yah | 22:49 |
| fungi | done | 22:49 |
| *** jkilpatr_ has joined #openstack-infra | 22:49 | |
| mordred | so - as soon as that lands, v3 is 're-enabled' | 22:49 |
| fungi | it's actually enqueued in infra-check, so not a syntax error at least | 22:49 |
| mordred | \o/ | 22:49 |
| fungi | and now it's in infra-gate | 22:50 |
| fungi | which should hopefully no longer exist as soon as it merges | 22:50 |
| fungi | mildly mind-bending | 22:50 |
| mordred | right? | 22:50 |
| *** markvoelker has joined #openstack-infra | 22:50 | |
| clarkb | and this one only works because we don't look at its future state | 22:51 |
| clarkb | ? | 22:51 |
| mordred | yup | 22:51 |
| * mordred watching to see the pipeline go away | 22:51 | |
| jeblair | i think the mouseover time estimates aren't working; something to fix after mnaser's status page refresh | 22:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove infra specific pipelines and enable pipelines https://review.openstack.org/510892 | 22:53 |
| pabelanger | odd, I didn't see ^ on zuulv3.o.o status page | 22:53 |
| * mordred did | 22:53 | |
| clarkb | heh that queued things to infra-post | 22:53 |
| clarkb | then removed it | 22:54 |
| jeblair | and it's gone :) | 22:54 |
| pabelanger | just did CTRL-f5 to be sure | 22:54 |
| mordred | \o/ | 22:54 |
| jeblair | pabelanger: it was in infra-gate | 22:54 |
| jeblair | rip | 22:54 |
| pabelanger | ah | 22:54 |
| pabelanger | ++ | 22:54 |
| jeblair | i will enqueue the one change from zuulv2 now | 22:54 |
| mordred | jeblair: do you think we can handle the load? | 22:54 |
| jeblair | we will find out | 22:55 |
| mordred | neutron change just hit the gate | 22:55 |
| jeblair | that's the one | 22:55 |
| *** markvoelker has quit IRC | 22:55 | |
| pabelanger | yah, zuul-scheduler is playing it nice and cool right now :D | 22:55 |
| jeblair | pabelanger: can you +3 https://review.openstack.org/512120 please? | 22:55 |
| pabelanger | looking | 22:55 |
| jeblair | unfortunately, after that lands we're going to need a full zuulv3 restart to get stats. sorry. | 22:56 |
| pabelanger | jeblair: +3 | 22:56 |
| mordred | we should always do zuul v3 rollouts on sundays | 22:56 |
| fungi | it's been long enough, i've gone ahead and set the by-hand redirect on zuul.o.o so consumers of the old status.json are getting the new one | 22:57 |
| *** dizquierdo has quit IRC | 22:57 | |
| mordred | nice | 22:57 |
| jeblair | fungi: thx. i approved 511067 | 22:57 |
| pabelanger | fungi: ++ | 22:57 |
| fungi | er, whoops, typo on my redirect though, fixing | 22:58 |
| jeblair | when that lands, i think we're done? | 22:58 |
| mordred | jeblair: I agre with you | 22:58 |
| jeblair | it really did take us 1 hour to flip that really big switch | 22:58 |
| fungi | okay, fixed now | 22:58 |
| fungi | silly $ backreferences instead of \ | 22:58 |
| fungi | who wants to send the celebratory announcement to the ml? | 22:59 |
| *** markvoelker has joined #openstack-infra | 22:59 | |
| mordred | the vertical space under infra-check is interesting | 22:59 |
| pabelanger | and POST_FAILURES look to be mitigated with our ignore_errors patch to logstash task | 23:00 |
| fungi | mordred: i think it's trying to grid the layout or something there. it's doing similar vertical whitespace under experimental too for me | 23:00 |
| jeblair | pabelanger: have there been more errors? | 23:00 |
| pabelanger | jeblair: I haven't see any yet | 23:01 |
| jeblair | pabelanger: oh, how do you know it's mitigated? | 23:01 |
| *** slaweq has quit IRC | 23:01 | |
| mordred | jeblair: neutron 511960,1 is in both check and gate - is that expected right now? | 23:01 |
| pabelanger | jeblair: yah, maybe not the correct word. I maybe fixed but ignored? | 23:02 |
| jeblair | mordred: probably was in check when v2 was running (it's been in there for > 1 hour) | 23:02 |
| fungi | mordred: it's expected if they rechecked it | 23:02 |
| mordred | jeblair: ah - yes - good point | 23:02 |
| fungi | mordred: though that does not appear to be the case | 23:02 |
| mordred | nice to see legacy-grenade-dsvm-neutron-dvr-multinode be success :) | 23:03 |
| fungi | oh, yep | 23:03 |
| fungi | right, so it made it into the gate in v2 while it was still getting check resources in v3 | 23:03 |
| jeblair | pabelanger: well, that would be great, but again, it'd be nice if there were evidence of that. I don't see "No connected" in the executor logs, so i don't know that we have any evidence that the problem has happened again. | 23:03 |
| fungi | and then got enqueued into the gate in v3 | 23:03 |
| jeblair | pabelanger: and we *need* it to happen again so we can fix it. | 23:04 |
| pabelanger | agree | 23:04 |
| jeblair | pabelanger: so if we've masked it too much, we'll have to unmask it. | 23:04 |
| pabelanger | yah | 23:04 |
| jeblair | but i think we can give it a day or so and see if it shows up in the log | 23:04 |
| *** jkilpatr_ has quit IRC | 23:05 | |
| pabelanger | kk, see a job issue networking-cisco | 23:06 |
| mordred | pabelanger: I feel like I +2d a networking-cisco job recently | 23:06 |
| pabelanger | mordred: yah, checking zuulv3-issues now | 23:06 |
| pabelanger | http://logs.openstack.org/54/505654/5/check/openstack-tox-compare-cover/bd44cbb/ara/result/15ca4ff6-30da-4370-b77f-a1c94f25aca5/ | 23:06 |
| mordred | pabelanger: sounds like our friend-neighborhood "neutron missing" issue | 23:07 |
| *** slaweq has joined #openstack-infra | 23:08 | |
| pabelanger | yar | 23:08 |
| mordred | pabelanger: easy fix I think - networking-cisco seems to be the only one using that job :) | 23:08 |
| pabelanger | mordred: yup, just checking to see of we have an existing project-template some place | 23:09 |
| pabelanger | but will push up a patch in a minute | 23:09 |
| mnaser | so everyone is taking the day off tomorrow to celebrate zuulv3 v2? hah. | 23:09 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Move statsd to correct zuul.conf file https://review.openstack.org/512120 | 23:09 |
| mordred | ++ | 23:09 |
| mordred | mnaser: flying to copenhagen is the weirdest way I've ever celebrated :) | 23:10 |
| mnaser | oh fun | 23:10 |
| jeblair | mordred: raise a glass of swamp water! | 23:10 |
| *** baoli has joined #openstack-infra | 23:11 | |
| jeblair | fungi, mordred: i didn't see anyone jump on the followup 'all-done' email... are either of you inspired? if not, i can, but i'll probably just say "done". :) | 23:11 |
| clarkb | fungi: did we get npm sorted out for the status page? | 23:11 |
| fungi | jeblair: i'm happy to do it | 23:12 |
| fungi | mordred probably needs to pack | 23:12 |
| fungi | clarkb: good question, lemme see if i dropped the ball on that patch | 23:12 |
| jeblair | fungi: all yours! | 23:12 |
| clarkb | fungi: iirc it had problems with trusty? | 23:13 |
| clarkb | basically fixed for one platform but not the other | 23:13 |
| fungi | https://review.openstack.org/508564 | 23:13 |
| fungi | still needs work apparently | 23:13 |
| *** claudiub has quit IRC | 23:13 | |
| clarkb | fungi: could also be puppet loading up path details before the package gets installed? | 23:14 |
| clarkb | I know the pip package module did/does that | 23:14 |
| fungi | possibly | 23:14 |
| jeblair | mordred: +3 512027 ? | 23:14 |
| *** baoli has quit IRC | 23:15 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add neutron as require-project for networking-cisco jobs https://review.openstack.org/512122 | 23:15 |
| jeblair | clarkb: +3 508906 ? | 23:16 |
| clarkb | pabelanger: I think its required on all those jobs (pep8, py27, etc) isn't it? | 23:16 |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Revert "Add redirect from status.o.o/zuul to zuulv3.openstack.org"" https://review.openstack.org/511067 | 23:16 |
| mordred | jeblair: dne | 23:17 |
| jeblair | and that's our last rollout change merged! | 23:17 |
| pabelanger | clarkb: shouldn't be, there are templates for that I think | 23:17 |
| *** dingyichen has joined #openstack-infra | 23:17 | |
| clarkb | pabelanger: except its overridden there to require neutronclient and each job is specified by hand? | 23:17 |
| pabelanger | clarkb: openstack-python-jobs-neutron and openstack-python-jobs-neutron | 23:17 |
| clarkb | pabelanger: ya but they aren't using those | 23:17 |
| pabelanger | clarkb: doesn't it append? | 23:17 |
| pabelanger | http://logs.openstack.org/54/505654/5/check/openstack-tox-pep8/20c7a7f/job-output.txt.gz | 23:18 |
| clarkb | pabelanger: oh wait they use both the template and spcify the jobs | 23:18 |
| pabelanger | that is from working networking-cisco job | 23:18 |
| clarkb | that seems like a bug | 23:18 |
| mordred | nah -it's on purpose | 23:18 |
| pabelanger | clarkb: yah, not sure | 23:18 |
| mordred | the template defines neutron | 23:18 |
| mordred | the local define adds neutronclient | 23:18 |
| pabelanger | Yah, that's was the first time I seen that | 23:18 |
| clarkb | that is probably worthy of a comment at least | 23:18 |
| pabelanger | kinda clever | 23:19 |
| clarkb | jeblair: kind of confused by your comment on 8906. Seems like you are saying the files restriction should be removed but +2'd the change? | 23:20 |
| jeblair | clarkb: sorry, i mean that the files restriction is correct because it will still run the job on .zuul.yaml changes | 23:21 |
| pabelanger | fungi: clarkb: think I see issue with trusty job on 508564 | 23:21 |
| pabelanger | I can push up patch | 23:21 |
| fungi | pabelanger: thanks!!! | 23:21 |
| jeblair | clarkb: oh, i think i misread andreaf's comment | 23:21 |
| jeblair | clarkb: your confusion makes more sense if i read that one right. :) | 23:21 |
| jeblair | clarkb: yes. | 23:22 |
| jeblair | clarkb: tell you what, i'll re-comment, and then make a followup change to drop it | 23:22 |
| clarkb | jeblair: thanks | 23:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron as require-project for networking-cisco jobs https://review.openstack.org/512122 | 23:22 |
| jeblair | clarkb: that way, if you are otherwise okay with it, we can keep the check result and go ahead and merge 906 | 23:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Add 90s connect timeout for submit log jobs https://review.openstack.org/512105 | 23:23 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Limit multinode-integration, update base-integration https://review.openstack.org/512100 | 23:24 |
| openstackgerrit | Merged openstack-infra/project-config master: Add OpenStack client check to python-rsdclient https://review.openstack.org/511650 | 23:24 |
| jeblair | clarkb: remote: https://review.openstack.org/512123 Zuul: remove file matcher for devstack job | 23:25 |
| pabelanger | fungi: Oh, maybe not... | 23:26 |
| pabelanger | we should be installing npm from nodejs puppet manifest I think | 23:26 |
| *** abelur_ has quit IRC | 23:26 | |
| jeblair | i'm going to put some nails in some things, but will check back in a bit | 23:26 |
| fungi | infra-root: draft announcement is https://etherpad.openstack.org/p/2FSNrxRQGZ | 23:27 |
| mordred | fungi: lgtm | 23:27 |
| ianw | jeblair: https://review.openstack.org/#/c/508906/ ... that's the "devstack" job right? | 23:28 |
| pabelanger | fungi: ++ | 23:28 |
| clarkb | jeblair: andreaf I've approved the devstack job addition and +2'd the followup to run it more | 23:29 |
| fungi | i'll go ahead and send what i've got there | 23:30 |
| clarkb | jeblair: when do you want to get the statsd pickup restart in? | 23:31 |
| clarkb | (that is going to be important for groups like neutron so earlier the better I think) | 23:31 |
| pabelanger | Yah, forgot about that | 23:32 |
| dmsimard | cloudnull, evrardjp, logan-, odyssey4me: looks like OSA jobs aren't working on v3. http://logs.openstack.org/15/512115/1/check/legacy-openstack-ansible-openstack-ansible-ceph-centos-7/43c8bd9/job-output.txt.gz#_2017-10-15_21_00_54_601852 | 23:33 |
| dmsimard | I don't see any ongoing patches to add the required projects for the OSA jobs, is there any WIP ? | 23:33 |
| fungi | clarkb: which statsd changes? jeblair restarted the executors a little while ago to pick some up | 23:34 |
| * mordred needs to step away for a bit - it seems like things are in good shape though | 23:34 | |
| *** hongbin_ has joined #openstack-infra | 23:35 | |
| *** hongbin has quit IRC | 23:35 | |
| clarkb | fungi: https://review.openstack.org/512120 jeblair said we will need a full zuul restart to pick that up | 23:35 |
| clarkb | we should double check puppet has applied it too | 23:36 |
| mnaser | is it possible the `use-cached-repos` role is broken? :X | 23:36 |
| dmsimard | wow what is this repo, I've never seen it before o_O https://github.com/openstack/openstack | 23:36 |
| mnaser | ubuntu-xenial | error: Could not remove config section 'remote.origin' | 23:36 |
| clarkb | dmsimard: its a repo that absues gerrit behavior around sub modules to track all the merges to openstack (though I don't know that we've properly updated it since the big tent) | 23:37 |
| pabelanger | mnaser: got log? | 23:37 |
| clarkb | mnaser: have a link? origin isn't expected to exist so I don't think that is entirely unexpected | 23:37 |
| mnaser | the job is currently being retried | 23:38 |
| mnaser | let me see | 23:38 |
| mnaser | (cause it happened/failed in pre) | 23:38 |
| mnaser | http://zuulv3.openstack.org/static/stream.html?uuid=0767a19b1ec8490987b6398ccee613ef&logfile=console.log | 23:38 |
| mnaser | watch this if you manage to catch it, it should fail | 23:38 |
| clarkb | mnaser: seemd to work ? | 23:39 |
| mnaser | yeah it just went through | 23:39 |
| clarkb | at least use-cached-repos went by successfully | 23:39 |
| mnaser | i could swear its failed before, let me find a log before that it failed | 23:39 |
| mnaser | i saw it once and recheck'd | 23:39 |
| mnaser | http://logs.openstack.org/02/509202/3/check/puppet-openstack-integration-4-scenario002-tempest-ubuntu-xenial/61735de/job-output.txt.gz#_2017-10-15_08_43_53_891878 | 23:39 |
| mnaser | this is from today | 23:39 |
| clarkb | ianw: I think dtroyer mentioned that he had seen hardlink errors recently. Do you know if the simplification for hardlink checking merged? | 23:40 |
| *** abelur_ has joined #openstack-infra | 23:40 | |
| ianw | clarkb: not in my queue, so must have | 23:41 |
| ianw | is it in the etherpad (looking...) | 23:41 |
| *** abelur_ has joined #openstack-infra | 23:41 | |
| *** slaweq has quit IRC | 23:41 | |
| pabelanger | mnaser: looking | 23:41 |
| mnaser | oh wait a second | 23:41 |
| mnaser | it runs use-cached-repos twice? | 23:41 |
| mnaser | and the second run fails | 23:42 |
| pabelanger | yah | 23:42 |
| pabelanger | I see that | 23:42 |
| *** tosky has quit IRC | 23:42 | |
| *** bobh has joined #openstack-infra | 23:42 | |
| pabelanger | wow | 23:42 |
| pabelanger | your playbooks order is amazing | 23:42 |
| pabelanger | http://logs.openstack.org/02/509202/3/check/puppet-openstack-integration-4-scenario002-tempest-ubuntu-xenial/61735de/ara/ | 23:43 |
| pabelanger | what is going on there | 23:43 |
| ianw | clarkb: "not permitted" ... that seems different, and i wonder if it's the user stuff i pointed out it was doing | 23:43 |
| mnaser | is it possible my pre's are messing things up? | 23:43 |
| clarkb | ianw: oh with the perms? | 23:43 |
| pabelanger | mnaser: yes | 23:43 |
| ianw | clarkb: yeah, just above it -> http://logs.openstack.org/15/509015/2/check/osc-functional-devstack/26b5228/job-output.txt.gz#_2017-10-14_02_23_08_399014 | 23:43 |
| clarkb | ianw: could be since hardlinking needs perms on the source side right? | 23:43 |
| pabelanger | you seem to have created a new base? | 23:43 |
| mnaser | i have prepare-node-common.yaml which is the base of all puppet jobs, then prepare-node-integration which is the base of integration jobs | 23:44 |
| clarkb | reading that role it blindly deletes origin | 23:44 |
| mnaser | it..worked okay before | 23:44 |
| clarkb | so if origin isn't there because it already deleted it once then ya it should fail | 23:44 |
| mnaser | but pre.yaml is running twice | 23:44 |
| clarkb | I think what we want t o do is make that git remote rm origin only happen if the clone task above it ran | 23:45 |
| * clarkb makes a patch | 23:45 | |
| mnaser | clarkb isnt this a bug though? | 23:45 |
| pabelanger | I'm trying to understand your playbooks now | 23:45 |
| mnaser | if you look at the ARA output, pre.yaml ran twice | 23:45 |
| mnaser | once before the entire job, once before the actual job was going to start | 23:46 |
| pabelanger | because you are some how running playbooks before base/pre.yaml | 23:46 |
| pabelanger | which, I do not know how is possible | 23:46 |
| clarkb | mnaser: yes it is a bug that it is running twice | 23:46 |
| pabelanger | in fact, base/pre.yaml is run twice | 23:46 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_health master: Don't set npm_package_ensure https://review.openstack.org/508564 | 23:46 |
| mnaser | i think whats happening is that its running before the entire "tree" of jobs start, then it runs again after all the pre's of the parents have ran | 23:47 |
| mnaser | oh you know what | 23:48 |
| mnaser | pabelanger do you think that because `puppet-openstack-integration-4` has no pre: it default to the base one? | 23:48 |
| clarkb | ya maybe the extra belt and suspenders not necessary since this is only happening due to a different bug that should be fixed | 23:48 |
| mnaser | https://github.com/openstack/puppet-openstack-integration/blob/master/.zuul.yaml#L301-L304 has no pre, so maybe pre gets set to the $world default pre, which runs twice | 23:49 |
| pabelanger | mnaser: no, it should be like an onion | 23:49 |
| ianw | clarkb / dtroyer : possible quick fix, use $USER https://review.openstack.org/512125 ... but i can all be rewritten i think | 23:49 |
| mnaser | pabelanger oh an interesting note is that this is running in a stable branch | 23:50 |
| mnaser | i am not sure if thats related, but i dont think ive seen a similar failure in master | 23:50 |
| * mnaser looks | 23:50 | |
| *** stakeda has joined #openstack-infra | 23:50 | |
| *** slaweq has joined #openstack-infra | 23:51 | |
| pabelanger | mnaser: do you have .zuul.yaml on stable branches? | 23:51 |
| clarkb | mnaser: pabelanger the zuul_info invetnroy iirc should log the inheritance path | 23:51 |
| mnaser | pabelanger that was the patch to get .zuul.yaml into stable | 23:51 |
| mnaser | clarkb ok ill have a look | 23:51 |
| pabelanger | clarkb: it does and looks correct | 23:51 |
| mnaser | http://logs.openstack.org/04/509204/1/check/puppet-openstack-lint/0767a19/ <-- officially the most "recent" failure | 23:52 |
| pabelanger | clarkb: I think this might be a jeblair or mordred issue | 23:52 |
| pabelanger | but, IIRC, we should always be running base/pre.yaml before anything else | 23:52 |
| pabelanger | and have no idea why twice | 23:52 |
| *** bobh has quit IRC | 23:53 | |
| pabelanger | mnaser: and no failure on master? | 23:53 |
| mnaser | pabelanger nope, trying to find a stable branch change but none logged by zuul | 23:53 |
| mnaser | oh | 23:53 |
| mnaser | https://review.openstack.org/#/c/511862/ | 23:53 |
| mnaser | a stable change that didnt fail | 23:54 |
| mnaser | (or not the same failure anyways) | 23:54 |
| pabelanger | mnaser: so, first question. Why adding .zuul.yaml to stable branches? and not just manage it from master? | 23:54 |
| pabelanger | because, they look like same jobs, yes? | 23:55 |
| mnaser | pabelanger there was actual changes that we had to do to some parts of the code, ex: changes in `functions` wrt ZUUL_URL no longer being set | 23:55 |
| mnaser | and some stuff relating to remotes being removed from zuul | 23:55 |
| mnaser | *zuulv3 | 23:55 |
| mnaser | so job would pretty much fail without those changes | 23:55 |
| pabelanger | mnaser: right, I you'd be able to make those changes, but keep .zuul.yaml in master branch only I think | 23:56 |
| pabelanger | mnaser: I admit, I have looked at your jobs or patches | 23:56 |
| *** felipemonteiro has joined #openstack-infra | 23:56 | |
| pabelanger | but, I'm going to defer to jeblair on this one, I am not sure what is happening atm | 23:57 |
| clarkb | does it consistently fail? | 23:57 |
| clarkb | or if you recheck on a single change sometimes it works? | 23:57 |
| mnaser | pabelanger at the time, fungi also mentioned it would be a good idea for jobs to have their own zuul.yaml files so that changes don't break old jobs anymore without knowing | 23:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Remove bashate legacy jobs https://review.openstack.org/512117 | 23:57 |
| mnaser | clarkb yes, consistently, fi you look at zuul's status ,you can see them all in retry_failure across all stable branches | 23:57 |
| mnaser | the weird thing is this used to work | 23:58 |
| *** felipemonteiro_ has joined #openstack-infra | 23:58 | |
| mnaser | or maybe it didnt, i see RETRY_FAILURE way back, then normal fails and RETRY_FAILURE again | 23:58 |
| mnaser | on october 13th, it did fail "properly" - http://logs.openstack.org/02/509202/3/check/puppet-openstack-integration-4-scenario001-tempest-centos-7/ab9f239/ara/ | 23:59 |
| pabelanger | yah, that one looks right | 23:59 |
| *** felipemonteiro_ has quit IRC | 23:59 | |
| *** felipemonteiro_ has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!