| fungi | and just in time for the "a little over 35 hours from now" to not be a lie | 00:00 |
|---|---|---|
| *** aeng has joined #openstack-infra | 00:00 | |
| *** edmondsw has quit IRC | 00:04 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci master: Add logs for config-download https://review.openstack.org/510709 | 00:04 |
| *** mriedem1 is now known as mriedem | 00:04 | |
| *** yamahata has quit IRC | 00:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add git timeout https://review.openstack.org/509517 | 00:10 |
| *** aspiers has quit IRC | 00:11 | |
| fungi | ianw: i take it the strace sudo error for gerrit2 on review.o.o was you | 00:13 |
| ianw | fungi: yep, sorry | 00:14 |
| fungi | np, just making sure | 00:14 |
| *** ianychoi has joined #openstack-infra | 00:14 | |
| *** dingyichen has joined #openstack-infra | 00:15 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Retry change queries on a 502 response https://review.openstack.org/510706 | 00:19 |
| *** cody-somerville has quit IRC | 00:19 | |
| *** cody-somerville has joined #openstack-infra | 00:20 | |
| *** cody-somerville has quit IRC | 00:20 | |
| *** cody-somerville has joined #openstack-infra | 00:20 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Improve exception handling around lost requests https://review.openstack.org/510594 | 00:20 |
| openstackgerrit | Anup Navare proposed openstack-infra/devstack-gate master: [Test] DNM Checking if tinyIPA builds with py3 https://review.openstack.org/509641 | 00:20 |
| *** Swami has quit IRC | 00:21 | |
| *** gouthamr has joined #openstack-infra | 00:21 | |
| *** xarses has quit IRC | 00:21 | |
| *** ijw has joined #openstack-infra | 00:27 | |
| pabelanger | fungi: wfm | 00:27 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project https://review.openstack.org/509119 | 00:29 |
| pabelanger | ianw: I'm getting some 502 back from gerrit this evening too | 00:32 |
| *** ijw has quit IRC | 00:32 | |
| *** ijw has joined #openstack-infra | 00:33 | |
| ianw | pabelanger: yeah, we're moving into ~ hour 4 of killer cpu time | 00:34 |
| ianw | jeblair suggested about 1:30 ago to wait ~ hour and see if it decreases, i'd say it's not | 00:35 |
| ianw | any opposition to a restart? | 00:35 |
| ianw | pabelanger: do you recall anything around ensuring the installation of python-selinux to ensure ansible templates get their permissions setup right? | 00:37 |
| ianw | i'm going to assume it's all working, until proven otherwise | 00:38 |
| jeblair | ianw: i agree this doesn't look promising. i'm +1 on restart. | 00:38 |
| ianw | #status log restarting zuul after prolonged period of high GC activity is causing 502 errors | 00:39 |
| openstackstatus | ianw: finished logging | 00:39 |
| jeblair | s/zuul/gerrit/ :) | 00:40 |
| ianw | bah! zuul on my brain | 00:40 |
| *** mriedem has quit IRC | 00:40 | |
| *** sree has joined #openstack-infra | 00:41 | |
| *** Apoorva_ has joined #openstack-infra | 00:41 | |
| *** andreas_s has joined #openstack-infra | 00:42 | |
| ianw | #status log restart of *gerrit* complete | 00:43 |
| openstackstatus | ianw: finished logging | 00:43 |
| ianw | cpu is still suspiciously high i'd say, but let's give it a bit to settle down | 00:44 |
| *** Apoorva has quit IRC | 00:44 | |
| ianw | hmm, settling down to 2-300%, better than the 8-900% before | 00:45 |
| *** Apoorva_ has quit IRC | 00:45 | |
| *** sree has quit IRC | 00:45 | |
| *** LindaWang has joined #openstack-infra | 00:46 | |
| *** andreas_s has quit IRC | 00:50 | |
| *** andreas_s has joined #openstack-infra | 00:51 | |
| *** ijw has quit IRC | 00:53 | |
| smcginnis | It does seem much more responsive. | 00:55 |
| *** ijw has joined #openstack-infra | 00:58 | |
| *** jkilpatr has quit IRC | 00:59 | |
| *** s-shiono has joined #openstack-infra | 01:02 | |
| *** gildub has joined #openstack-infra | 01:04 | |
| *** cody-somerville has quit IRC | 01:04 | |
| fungi | zuulv3 scheduler debug log is nearing 15gib now... mildly worried that come time for log rotation there won't be sufficient room on the rootfs to hold the compressed copy | 01:10 |
| fungi | df claims 17gib free on / | 01:10 |
| tonyb | fungi: Bah who needs roatated logs ;P | 01:10 |
| fungi | with about another 5 hours to go until rotation | 01:11 |
| fungi | tonyb: more along the lines of "bah who needs to be able to write to the root filesystem" | 01:11 |
| tonyb | :) | 01:12 |
| fungi | which is what processes will be asking themselves once logrotate sucks up every last available block in it | 01:12 |
| mtreinish | fungi: you could just run the rotate job manually (assuming it's got the normal default of cron daily) | 01:12 |
| fungi | mtreinish: calling logrotate manually has taken care of it in the past, at any rate | 01:12 |
| fungi | tonyb: last time this happened, i had to nova reboot the server because sshd couldn't write to disk any longer | 01:13 |
| tonyb | fungi: Oh dear :( | 01:13 |
| fungi | yeah :/ | 01:13 |
| *** andreas_s has quit IRC | 01:14 | |
| fungi | infra-root: any objections to me manually rotating logs for now (we plan to provide more room before wednesday cut-over anyway) | 01:14 |
| ianw | spin spin spin | 01:15 |
| fungi | right round | 01:15 |
| *** cuongnv has joined #openstack-infra | 01:17 | |
| pabelanger | fungi: I think we'll also want to resize our current swap on zuulv3 tomorrow too, and make room for /var/log/zuul on it | 01:17 |
| pabelanger | now I EOD | 01:18 |
| *** baoli has joined #openstack-infra | 01:25 | |
| *** jascott1 has quit IRC | 01:26 | |
| *** psachin has joined #openstack-infra | 01:26 | |
| *** jascott1 has joined #openstack-infra | 01:26 | |
| *** andreas_s has joined #openstack-infra | 01:27 | |
| *** edmondsw has joined #openstack-infra | 01:28 | |
| *** jascott1 has quit IRC | 01:30 | |
| *** dbecker has joined #openstack-infra | 01:31 | |
| *** xarses has joined #openstack-infra | 01:31 | |
| fungi | pabelanger: right, that's what i meant by "provide more room" | 01:31 |
| fungi | but this manual rotate is just to get us through until then | 01:32 |
| *** andreas_s has quit IRC | 01:36 | |
| *** dbecker has quit IRC | 01:38 | |
| *** xarses has quit IRC | 01:42 | |
| fungi | #status log manually rotated all logs on zuulv3.openstack.org as a stop-gap to prevent a full rootfs later when scheduled log rotation kicks in; an additional 14gib were freed as a result | 01:43 |
| openstackstatus | fungi: finished logging | 01:43 |
| *** kiennt26 has joined #openstack-infra | 01:43 | |
| *** cody-somerville has joined #openstack-infra | 01:43 | |
| *** cody-somerville has joined #openstack-infra | 01:43 | |
| *** baoli has quit IRC | 01:43 | |
| *** vhosakot has quit IRC | 01:44 | |
| *** ijw has quit IRC | 01:44 | |
| *** ijw has joined #openstack-infra | 01:44 | |
| *** baoli has joined #openstack-infra | 01:44 | |
| *** ijw has quit IRC | 01:44 | |
| fungi | that should be plenty of breathing room until tomorrow | 01:44 |
| *** ijw has joined #openstack-infra | 01:44 | |
| *** ijw has quit IRC | 01:44 | |
| *** andreas_s has joined #openstack-infra | 01:45 | |
| *** camunoz has quit IRC | 01:45 | |
| *** cody-somerville has quit IRC | 01:46 | |
| SamYaple | you dont want 160gb of swap and think we should use some of that for logs? thats crazy talk | 01:46 |
| fungi | heh | 01:47 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs https://review.openstack.org/510720 | 01:48 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs https://review.openstack.org/510720 | 01:49 |
| *** andreas_s has quit IRC | 01:50 | |
| *** baoli has quit IRC | 01:50 | |
| *** andreas_s has joined #openstack-infra | 01:50 | |
| *** fanzhang has joined #openstack-infra | 01:51 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Moving the Octavia meeting time and channel https://review.openstack.org/510701 | 01:53 |
| *** andreas_s has quit IRC | 01:55 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Ceilometer legacy jobs https://review.openstack.org/510720 | 01:55 |
| clarkb | looks like gerrit is happy now? | 02:04 |
| SamYaple | seems much more responsive | 02:05 |
| clarkb | I bet we can look at the task queue to start tracking that down | 02:05 |
| tonyb | fungi: http://logs.openstack.org/85/858c988956fce8ac8ed8a202fe91aadbc0387395/release-post/tag-releases/8b44d50/console.html is that the same issue with mistral-lib ? or just a general git issue? | 02:05 |
| fungi | clarkb: well, it got restarted | 02:07 |
| fungi | so no idea if it would have cleared up on its own | 02:07 |
| fungi | tonyb: that one looks different, and likely a transient network issue between signing01.ci.o.o and zm03.o.o | 02:10 |
| tonyb | fungi: okay. Can I re-trigger it? or is that another favor I need to ask? | 02:10 |
| fungi | the issue which hit mistral-lib should not recur on the signing01.ci node unless we rearrange zuul mergers again | 02:11 |
| *** hongbin has quit IRC | 02:11 | |
| fungi | tonyb: i expect you just need that ref reenqueued into post right? or did other jobs also run successfully for it in post which cannot safely be rerun? | 02:11 |
| fungi | (well, in release-post) | 02:12 |
| tonyb | fungi: I think reenqueued that ref is safe. that job failed which measn the next was skipped and none of the per-repo jobs started, but to be frank I've never been the one awake to requiest this so I may be wrong. | 02:16 |
| tonyb | http://lists.openstack.org/pipermail/release-job-failures/2017-October/000535.html I don't know if that helps or not | 02:16 |
| clarkb | fungi: ah | 02:17 |
| fungi | tonyb: cool, so there were only two jobs, the one which failed, and one depending on that which was subsequently skipped. thus, reenqueuing will rerun them both as desired | 02:19 |
| *** dhinesh has quit IRC | 02:19 | |
| tonyb | Thanks. I'm glad I'm not too wide of the mark. | 02:19 |
| *** liujiong has joined #openstack-infra | 02:20 | |
| fungi | tonyb: i've reenqueued it now | 02:22 |
| fungi | that seems to be the latest change to merge to the releases repo | 02:22 |
| fungi | though i expect that those jobs are idempotent, and so merging another may have just caught it up on whatever needed to be tagged | 02:23 |
| *** thorst has quit IRC | 02:23 | |
| tonyb | fungi: Thanks. You're probably right. | 02:23 |
| *** thorst has joined #openstack-infra | 02:24 | |
| *** neon_icon has quit IRC | 02:24 | |
| *** neon_icon has joined #openstack-infra | 02:24 | |
| SamYaple | anyone want to review a bindep patch? https://review.openstack.org/#/c/506502/ | 02:24 |
| SamYaple | who doesnt like Parsley? | 02:24 |
| *** neon_icon has quit IRC | 02:26 | |
| *** neon_icon has joined #openstack-infra | 02:27 | |
| *** neon_iconstar has joined #openstack-infra | 02:27 | |
| *** neon_icon has quit IRC | 02:27 | |
| *** thorst has quit IRC | 02:28 | |
| *** eharney has quit IRC | 02:31 | |
| *** dave-mccowan has quit IRC | 02:31 | |
| *** thorst has joined #openstack-infra | 02:37 | |
| *** thorst has quit IRC | 02:37 | |
| *** rwsu has quit IRC | 02:40 | |
| *** links has joined #openstack-infra | 02:44 | |
| *** nicolasbock_ has quit IRC | 02:47 | |
| *** kiennt26 has quit IRC | 02:49 | |
| *** pt_15 has quit IRC | 02:53 | |
| *** cody-somerville has joined #openstack-infra | 02:58 | |
| *** csomerville has joined #openstack-infra | 02:59 | |
| *** cody-somerville has quit IRC | 02:59 | |
| *** kiennt26 has joined #openstack-infra | 02:59 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Ceilometer https://review.openstack.org/510459 | 03:02 |
| *** ihrachys has quit IRC | 03:03 | |
| *** ihrachys has joined #openstack-infra | 03:04 | |
| *** bobh has joined #openstack-infra | 03:10 | |
| *** bobh has quit IRC | 03:20 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add unbound role https://review.openstack.org/510725 | 03:21 |
| *** lukebrowning_ has joined #openstack-infra | 03:23 | |
| *** edmondsw has quit IRC | 03:23 | |
| *** lukebrowning has quit IRC | 03:26 | |
| *** andreas_s has joined #openstack-infra | 03:27 | |
| *** andreas_s has quit IRC | 03:31 | |
| *** kiennt26 has quit IRC | 03:31 | |
| *** hongbin has joined #openstack-infra | 03:32 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Ceilometer https://review.openstack.org/510459 | 03:32 |
| *** kiennt26 has joined #openstack-infra | 03:32 | |
| *** namnh has joined #openstack-infra | 03:32 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add unbound role to base job https://review.openstack.org/510728 | 03:34 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Ceilometer https://review.openstack.org/510459 | 03:38 |
| *** gildub has quit IRC | 03:38 | |
| ianw | hmm, my browser does not like that status page at all | 03:39 |
| *** lukebrowning_ has quit IRC | 03:47 | |
| *** lukebrowning has joined #openstack-infra | 03:48 | |
| *** cody-somerville has joined #openstack-infra | 03:50 | |
| *** csomerville has quit IRC | 03:52 | |
| *** neon_iconstar has quit IRC | 03:53 | |
| *** ram5391 has joined #openstack-infra | 03:55 | |
| *** gildub has joined #openstack-infra | 03:56 | |
| *** mat128 has quit IRC | 03:58 | |
| SamYaple | ianw: yea every since zuulv3 dropped ive had to switch to a cli tool because my browser just wont take it | 03:59 |
| *** jaosorior has joined #openstack-infra | 04:00 | |
| *** ykarel has joined #openstack-infra | 04:00 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Zaqar https://review.openstack.org/510731 | 04:04 |
| *** ram5391 has quit IRC | 04:06 | |
| *** claudiub has joined #openstack-infra | 04:07 | |
| *** neon_iconstar has joined #openstack-infra | 04:07 | |
| *** coolsvap has joined #openstack-infra | 04:09 | |
| *** xarses has joined #openstack-infra | 04:12 | |
| *** gouthamr has quit IRC | 04:14 | |
| clarkb | firefox 57 beta mostly seems to keep up (though I cant check right now) | 04:16 |
| *** neon_iconstar has quit IRC | 04:17 | |
| *** edmondsw has joined #openstack-infra | 04:21 | |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Zaqar legacy jobs https://review.openstack.org/510732 | 04:25 |
| *** edmondsw has quit IRC | 04:26 | |
| *** lukebrowning_ has joined #openstack-infra | 04:27 | |
| *** lukebrowning has quit IRC | 04:29 | |
| *** pcaruana has joined #openstack-infra | 04:32 | |
| *** kiennt26 has quit IRC | 04:36 | |
| *** hongbin has quit IRC | 04:40 | |
| *** lukebrowning_ has quit IRC | 04:41 | |
| *** armaan has quit IRC | 04:41 | |
| *** armaan has joined #openstack-infra | 04:41 | |
| *** csomerville has joined #openstack-infra | 04:42 | |
| *** lukebrowning has joined #openstack-infra | 04:42 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Test unbound role https://review.openstack.org/510734 | 04:43 |
| *** lukebrowning_ has joined #openstack-infra | 04:45 | |
| *** cody-somerville has quit IRC | 04:45 | |
| *** lukebrowning has quit IRC | 04:47 | |
| chandankumar | AJaeger: hello | 04:52 |
| chandankumar | AJaeger: regarding this review https://review.openstack.org/#/c/510726/ | 04:53 |
| chandankumar | AJaeger: the list of repo generates during run time when this script https://github.com/openstack/tempest/blob/master/tools/generate-tempest-plugins-list.py executes | 04:53 |
| chandankumar | AJaeger: and there will be multiple repos, it is a unique case how to handle that? | 04:54 |
| *** udesale has joined #openstack-infra | 04:57 | |
| *** neon_icon has joined #openstack-infra | 04:58 | |
| *** pcaruana has quit IRC | 04:58 | |
| *** armax has quit IRC | 04:59 | |
| *** armax has joined #openstack-infra | 05:00 | |
| *** rwsu has joined #openstack-infra | 05:00 | |
| *** vsaienk0 has joined #openstack-infra | 05:03 | |
| *** armax has quit IRC | 05:04 | |
| *** jtomasek has joined #openstack-infra | 05:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove cron entries from nodepool launcher configs https://review.openstack.org/510676 | 05:07 |
| *** csomerville has quit IRC | 05:07 | |
| *** ykarel is now known as ykarel|afk | 05:11 | |
| *** vsaienk0 has quit IRC | 05:12 | |
| *** yamahata has joined #openstack-infra | 05:20 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-infra/project-config master: Remove project pylockfile from infrastructure systems https://review.openstack.org/508656 | 05:22 |
| *** csomerville has joined #openstack-infra | 05:25 | |
| *** mwarad has joined #openstack-infra | 05:26 | |
| *** jaosorior has quit IRC | 05:27 | |
| snapiri | Hi infra, I have the strangest issue - in the following patch: https://review.openstack.org/#/c/508761/ I have tried to remove the zuul-clone for zuul v3 builds. Now both zuul and jenkins builds fail for a strange reason. it seems like git fails to find values that exist in the project git config. can anyone take a look or suggest a way to tackle it? | 05:27 |
| *** cshastri has joined #openstack-infra | 05:35 | |
| *** neon_icon has quit IRC | 05:39 | |
| *** csomerville has quit IRC | 05:42 | |
| *** e0ne has joined #openstack-infra | 05:43 | |
| *** sree has joined #openstack-infra | 05:54 | |
| *** udesale__ has joined #openstack-infra | 05:55 | |
| *** udesale__ has quit IRC | 05:57 | |
| *** udesale__ has joined #openstack-infra | 05:57 | |
| *** udesale has quit IRC | 05:57 | |
| *** nmathew has joined #openstack-infra | 05:59 | |
| *** dhajare has joined #openstack-infra | 06:00 | |
| tonyb | snapiri: I dropped a comment on the review. | 06:01 |
| AJaeger | chandankumar: please ask the rest of the team during US hours, I have no quick fix for you. | 06:01 |
| chandankumar | AJaeger: sure :-) | 06:01 |
| *** jaosorior has joined #openstack-infra | 06:03 | |
| *** jrist has quit IRC | 06:03 | |
| *** andreas_s has joined #openstack-infra | 06:05 | |
| *** sree has quit IRC | 06:07 | |
| *** edmondsw has joined #openstack-infra | 06:10 | |
| *** sree_ has joined #openstack-infra | 06:11 | |
| *** sree_ is now known as Guest62172 | 06:11 | |
| *** eumel8 has joined #openstack-infra | 06:11 | |
| *** yamamoto has quit IRC | 06:11 | |
| snapiri | tonyb: thanks | 06:12 |
| eumel8 | morning | 06:12 |
| *** edmondsw has quit IRC | 06:14 | |
| *** vsaienk0 has joined #openstack-infra | 06:15 | |
| *** neon_icon has joined #openstack-infra | 06:15 | |
| *** rlandy|bbl is now known as rlandy | 06:16 | |
| *** rlandy has quit IRC | 06:16 | |
| *** jrist has joined #openstack-infra | 06:16 | |
| *** hemna_ has quit IRC | 06:18 | |
| *** Guest62172 has quit IRC | 06:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy manuals jobs https://review.openstack.org/510761 | 06:21 |
| AJaeger | morning, eumel8 | 06:21 |
| *** spectr has joined #openstack-infra | 06:22 | |
| *** sshnaidm|off is now known as sshnaidm | 06:22 | |
| AJaeger | ianw, if you have some review time, could you review https://review.openstack.org/#/c/510681/ https://review.openstack.org/510761 and https://review.openstack.org/#/c/510623/ , please? | 06:22 |
| eumel8 | morning, AJaeger, the man who never sleeps :) | 06:23 |
| eumel8 | AJaeger: is the new translation job working? | 06:24 |
| zhongjun | Hi anyone could help me to find why manila tempest config options cannot synchronize to new tempest job in new zuul check. | 06:26 |
| zhongjun | The original tempest conf file in manila-tempest-minimal-dsvm-dummy job belong to original Jenkins check: | 06:27 |
| zhongjun | http://logs.openstack.org/42/510542/1/check/legacy-manila-tempest-minimal-dsvm-dummy/2cb9f4d/logs/tempest_conf.txt.gz | 06:27 |
| zhongjun | Currently, the tempest conf in manila-tempest-minimal-dsvm-dummy job belong to new zuul check: http://logs.openstack.org/42/510542/1/check/legacy-manila-tempest-minimal-dsvm-dummy/2cb9f4d/logs/tempest_conf.txt.gz | 06:27 |
| zhongjun | We lack of some options about [share] node in tempest conf file in our new tempest gate job for zuul check. Do we have some patches to fix this? | 06:28 |
| zhongjun | The original tempest conf file in manila-tempest-minimal-dsvm-dummy job belong to original Jenkins check: http://logs.openstack.org/42/510542/1/check/gate-manila-tempest-minimal-dsvm-dummy-ubuntu-xenial/81a1434/logs/tempest_conf.txt.gz | 06:31 |
| *** pt_15 has joined #openstack-infra | 06:31 | |
| AJaeger | eumel8: zuul has too much work and too little nodes, the jobs are not run yet. So, no idea ;( | 06:33 |
| eumel8 | AJaeger: thats okay. I mean in general | 06:34 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy jobs from Kuryr-libnetwork https://review.openstack.org/509987 | 06:36 |
| *** aeng has quit IRC | 06:36 | |
| *** ykarel|afk is now known as ykarel | 06:38 | |
| AJaeger | eumel8: really no idea. It looks good in theory, waiting for practice run ;) | 06:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs https://review.openstack.org/510765 | 06:40 |
| *** kiennt26 has joined #openstack-infra | 06:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs https://review.openstack.org/510765 | 06:44 |
| dmellado | AJaeger: ping re https://review.openstack.org/#/c/509383/ | 06:46 |
| dmellado | we'd need to remove the legacy jobs? is that what you're referring about? | 06:46 |
| *** xarses has quit IRC | 06:51 | |
| AJaeger | dmellado: yes, similar to https://review.openstack.org/509987 and https://review.openstack.org/509986 - you need to *move* the jobs around. So, you added them, let's now remove them from central repos to not run them twice. | 06:51 |
| dmellado | AJaeger: ack, will follow up with a patch, thanks! | 06:52 |
| AJaeger | thanks, dmellado (actually: two patches, please - one for each repo ;) | 06:54 |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Remove legacy jobs from kuryr-k8s https://review.openstack.org/510769 | 06:55 |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Migrate Kuryr-kubernetes legacy jobs https://review.openstack.org/510771 | 06:56 |
| AJaeger | dmellado: kien is also working on it ^, can you discuss with him who will abandon? ^ | 06:57 |
| openstackgerrit | Kien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Migrate Kuryr-kubernetes legacy jobs https://review.openstack.org/510773 | 06:57 |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Migrate Kuryr-kubernetes legacy jobs https://review.openstack.org/510771 | 06:58 |
| *** rcernin has joined #openstack-infra | 06:58 | |
| *** gildub has quit IRC | 06:59 | |
| dmellado | kiennt26: | 07:00 |
| dmellado | I don't mind abandoning mine but please do note that we had a blueprint for this | 07:00 |
| dmellado | and you didn't sign up as an assignee | 07:00 |
| dmellado | so for next times please dont work on 'rogue' mode | 07:00 |
| kiennt26 | dmellado: yeah, I already abandoned my patches xD sr about it | 07:01 |
| dmellado | kiennt26: np ;) | 07:02 |
| dmellado | thanks for your help! but I'd like to address this in a sane mode | 07:02 |
| dmellado | kiennt26: I'll create a small blueprint for all the multinode gates | 07:02 |
| dmellado | let's split the work there ;) | 07:02 |
| *** dhajare has quit IRC | 07:03 | |
| *** dhajare has joined #openstack-infra | 07:03 | |
| kiennt26 | dmellado: cool ;) | 07:03 |
| AJaeger | dmellado, kiennt26, we need 510773 - should we restore it and correct dependency? | 07:03 |
| dmellado | AJaeger: yah, np from my side | 07:04 |
| AJaeger | kiennt26: can you do that, please? ^ | 07:04 |
| *** pcaruana has joined #openstack-infra | 07:04 | |
| *** jaosorior has quit IRC | 07:04 | |
| kiennt26 | AJaeger, dmellado: Sure | 07:04 |
| *** ykarel has quit IRC | 07:05 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Migrate Kuryr-kubernetes legacy jobs https://review.openstack.org/510773 | 07:06 |
| *** nmathew has quit IRC | 07:07 | |
| kiennt26 | AJaeger, dmellado: Done! ^^ | 07:07 |
| dmellado | kiennt26: thanks! | 07:07 |
| dmellado | AJaeger: thanks also for all your help on the zuulness so far ;) | 07:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add unbound role https://review.openstack.org/510725 | 07:08 |
| dmellado | ich schulde dir ein Bier :P | 07:08 |
| AJaeger | dmellado: Danke! ;) | 07:09 |
| AJaeger | kiennt26: great, thanks. | 07:09 |
| *** nmathew has joined #openstack-infra | 07:09 | |
| AJaeger | ianw, yolanda,could you putw https://review.openstack.org/#/c/510681/ https://review.openstack.org/510761 and https://review.openstack.org/#/c/510623/ on your review queues as time permits, please? | 07:10 |
| * AJaeger will be back later... | 07:11 | |
| *** ykarel has joined #openstack-infra | 07:12 | |
| *** claudiub|2 has joined #openstack-infra | 07:15 | |
| *** e0ne has quit IRC | 07:17 | |
| *** Hal has joined #openstack-infra | 07:17 | |
| *** Hal is now known as Guest78751 | 07:17 | |
| *** claudiub has quit IRC | 07:18 | |
| *** yamahata has quit IRC | 07:18 | |
| *** sileht has quit IRC | 07:23 | |
| *** sileht has joined #openstack-infra | 07:23 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: adds missing required-projects to n8g-(bagpipe|bgppvn) https://review.openstack.org/510435 | 07:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs from Kuryr-libnetwork https://review.openstack.org/509987 | 07:24 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/openstack-zuul-jobs master: Remove Zaqar legacy jobs https://review.openstack.org/510732 | 07:27 |
| *** tesseract has joined #openstack-infra | 07:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: use announce-release everywhere https://review.openstack.org/510681 | 07:28 |
| *** kjackal_ has joined #openstack-infra | 07:33 | |
| *** dizquierdo has joined #openstack-infra | 07:33 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Fix TripleO CI jobs https://review.openstack.org/508660 | 07:33 |
| *** jaosorior has joined #openstack-infra | 07:33 | |
| *** jpich has joined #openstack-infra | 07:35 | |
| *** shardy has joined #openstack-infra | 07:35 | |
| *** dhajare has quit IRC | 07:38 | |
| *** dhajare has joined #openstack-infra | 07:38 | |
| *** egonzalez has joined #openstack-infra | 07:39 | |
| *** electrofelix has joined #openstack-infra | 07:39 | |
| *** jpena|off is now known as jpena | 07:41 | |
| *** kiennt26 has quit IRC | 07:43 | |
| *** kiennt26 has joined #openstack-infra | 07:43 | |
| *** ralonsoh has joined #openstack-infra | 07:44 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Test unbound role https://review.openstack.org/510734 | 07:44 |
| *** gildub has joined #openstack-infra | 07:48 | |
| *** askb has quit IRC | 07:49 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: adds missing required-projects to n8g-(bagpipe|bgppvn) https://review.openstack.org/510435 | 07:55 |
| *** edmondsw has joined #openstack-infra | 07:58 | |
| *** threestrands has quit IRC | 07:59 | |
| *** edmondsw has quit IRC | 08:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy manuals jobs https://review.openstack.org/510761 | 08:13 |
| *** sree has joined #openstack-infra | 08:15 | |
| *** sree has quit IRC | 08:20 | |
| *** lin_yang has quit IRC | 08:21 | |
| *** pratik has joined #openstack-infra | 08:23 | |
| *** lucas-afk is now known as lucasagomes | 08:25 | |
| *** s-shiono has quit IRC | 08:27 | |
| *** inc0 has quit IRC | 08:32 | |
| *** pratik has quit IRC | 08:32 | |
| *** dbecker has joined #openstack-infra | 08:33 | |
| *** sree has joined #openstack-infra | 08:34 | |
| *** derekh has joined #openstack-infra | 08:36 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Add reno for release notes management https://review.openstack.org/510679 | 08:37 |
| *** askb has joined #openstack-infra | 08:37 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Add post tarball gate to kuryr-tempest-plugin https://review.openstack.org/510786 | 08:38 |
| *** dingyichen has quit IRC | 08:38 | |
| *** inc0 has joined #openstack-infra | 08:39 | |
| *** priteau has joined #openstack-infra | 08:40 | |
| *** e0ne has joined #openstack-infra | 08:40 | |
| *** ryotagami has joined #openstack-infra | 08:44 | |
| *** priteau has quit IRC | 08:46 | |
| *** priteau has joined #openstack-infra | 08:46 | |
| *** tosky has joined #openstack-infra | 08:48 | |
| *** andreas_s has quit IRC | 08:48 | |
| *** ykarel is now known as ykarel|lunch | 08:48 | |
| *** priteau_ has joined #openstack-infra | 08:49 | |
| *** priteau has quit IRC | 08:51 | |
| *** pt_15 has quit IRC | 09:02 | |
| openstackgerrit | Dao Cong Tien proposed openstack-infra/openstack-zuul-jobs master: grenade-dsvm-ironic: Move task 'legacy workspace dir' to first https://review.openstack.org/510795 | 09:03 |
| *** gildub has quit IRC | 09:06 | |
| *** gildub has joined #openstack-infra | 09:07 | |
| *** andreas_s has joined #openstack-infra | 09:07 | |
| snapiri | hi infra, I have a job that seems to be stuck on the zuul3. it is already running for over 20 hours and does not seem to progress: patch 509189,8. Can you suggest what to do? | 09:08 |
| *** aspiers has joined #openstack-infra | 09:12 | |
| dmellado | snapiri: I'd abandon + recover the job just to check if it unblocks it.... happened to me last week | 09:14 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Make version_string() lazy load version info https://review.openstack.org/450918 | 09:15 |
| openstackgerrit | Nam Nguyen Hoai proposed openstack-infra/project-config master: Remove legacy jobs from Zaqar https://review.openstack.org/510731 | 09:17 |
| *** kiennt26 has quit IRC | 09:19 | |
| snapiri | dmellado: thanks for the answer, isn't a recheck sufficient? | 09:19 |
| dmellado | snapiri: if it got really stuck and you don't want or can't wait for the timeout, nope :( | 09:20 |
| *** sambetts_ is now known as sambetts | 09:21 | |
| snapiri | dmellado: thanks | 09:21 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Discover Distribution through the class hierarchy https://review.openstack.org/399188 | 09:24 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Use super for a cleaner resolution of the superclass. https://review.openstack.org/412224 | 09:24 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Remove unnecessary 'if True' https://review.openstack.org/510806 | 09:24 |
| AJaeger | snapiri: we have a very high load on Zuul v3, just wait... | 09:27 |
| *** spiffxp has quit IRC | 09:28 | |
| *** vipul has quit IRC | 09:28 | |
| *** witek has quit IRC | 09:28 | |
| AJaeger | snapiri: you're enquing at end of queue then, will have to wait another 20 hours | 09:28 |
| AJaeger | snapiri: you can check status at zuulv3.openstack.org | 09:28 |
| *** spectr has quit IRC | 09:29 | |
| snapiri | AJaeger: will wait. note that the job has some post_failure | 09:29 |
| *** gildub has quit IRC | 09:29 | |
| *** sree has quit IRC | 09:29 | |
| *** spectr has joined #openstack-infra | 09:29 | |
| *** david-lyle has quit IRC | 09:29 | |
| *** pcrews has quit IRC | 09:29 | |
| AJaeger | snapiri: your change is not testing what you expect. A depends-on for project-config will not work, it's a trusted repository and does not allow dynamic reconfiguration. | 09:31 |
| *** pcrews has joined #openstack-infra | 09:31 | |
| AJaeger | yolanda, ianw, could one of you +2A https://review.openstack.org/#/c/510531/ , please? | 09:31 |
| AJaeger | snapiri: just wait until the dependencies are in and then test again, please | 09:32 |
| *** spectr has quit IRC | 09:32 | |
| dirk | do we need https://review.openstack.org/#/c/510565/ before the zuulv3 switch? | 09:33 |
| snapiri | AJaeger: thanks! | 09:33 |
| *** spiffxp has joined #openstack-infra | 09:33 | |
| AJaeger | dirk: Nice to have working jobs ;) That change misses Zuul v3 +1, let's wait for that - but then get it in, please | 09:34 |
| *** spectr has joined #openstack-infra | 09:34 | |
| *** armaan has quit IRC | 09:35 | |
| *** armaan has joined #openstack-infra | 09:35 | |
| dirk | AJaeger: the zuulv3 check queue is pretty long ;) and I think it doesn't actually run this code (its only executed when other repos use the check-requirements job) | 09:36 |
| *** vipul has joined #openstack-infra | 09:36 | |
| AJaeger | dirk: I see. Let's see what mordred thinks ^ | 09:40 |
| *** nmathew- has joined #openstack-infra | 09:41 | |
| *** thorst has joined #openstack-infra | 09:41 | |
| *** nmathew has quit IRC | 09:44 | |
| *** edmondsw has joined #openstack-infra | 09:46 | |
| tosky | AJaeger: sooo, I had two weird failures yesterday on zuulv3 | 09:46 |
| *** thorst has quit IRC | 09:47 | |
| tosky | in https://review.openstack.org/#/c/510528/, legacy-sahara-dsvm-scenario-neutron (and the wsgi version) shows a strange "Permission denied: '/home/zuul/.cache/pip/wheels/ca/1e/92/53679e3334df7f41c24014e01111f6495a586e003db0ffc7ba'" | 09:47 |
| tosky | and legacy-tempest-dsvm-sahara had a post_failure, so maybe that's infrastructural related | 09:48 |
| *** mwarad has quit IRC | 09:48 | |
| *** edmondsw has quit IRC | 09:51 | |
| *** kiennt26 has joined #openstack-infra | 09:52 | |
| *** kiennt26 has quit IRC | 09:53 | |
| *** dtantsur|afk is now known as dtantsur | 09:53 | |
| *** alexchadin has joined #openstack-infra | 09:54 | |
| *** liujiong has quit IRC | 09:54 | |
| *** wolverineav has joined #openstack-infra | 09:55 | |
| AJaeger | tosky: please followup with rest of team later on | 09:56 |
| tosky | ack | 09:58 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project https://review.openstack.org/509119 | 09:58 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project https://review.openstack.org/509119 | 10:00 |
| *** nmathew- has quit IRC | 10:01 | |
| *** ykarel|lunch is now known as ykarel | 10:02 | |
| *** andreas_s has quit IRC | 10:03 | |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project https://review.openstack.org/509119 | 10:03 |
| *** tbarron|PTO is now known as tbarron | 10:04 | |
| *** cuongnv has quit IRC | 10:05 | |
| *** wolverineav has quit IRC | 10:07 | |
| robcresswell | o/ Has anyone run into an issue with Ocata and Centos 7 and libvirt-python failing to build / install ? | 10:07 |
| *** andreas_s has joined #openstack-infra | 10:07 | |
| *** lukebrowning_ has quit IRC | 10:11 | |
| *** pbourke has quit IRC | 10:11 | |
| *** sdague has joined #openstack-infra | 10:12 | |
| *** pbourke has joined #openstack-infra | 10:12 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Remove legacy jobs in Craton https://review.openstack.org/510820 | 10:13 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Added pytest support to setup.py test https://review.openstack.org/438176 | 10:16 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server https://review.openstack.org/510362 | 10:16 |
| *** andreas_s has quit IRC | 10:17 | |
| *** andreas_s has joined #openstack-infra | 10:18 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Craton https://review.openstack.org/510824 | 10:19 |
| kazsh | AJaeger: thanks for your comment, I've updated my change, please check it when you have time. | 10:19 |
| *** xinni9e has joined #openstack-infra | 10:20 | |
| ianw | jeblair / fungi / dmsimard : as discussed this morning, updates on the unbound configuration in the zuulv3-issues page. i'm sure it will want some review rounds (ci hasn't reported and i'm out) but don't think it's a blocker | 10:22 |
| *** lukebrowning has joined #openstack-infra | 10:22 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove unnecessary scripts from tripleo-ci https://review.openstack.org/510828 | 10:24 |
| AJaeger | thanks, ianw and good night! | 10:25 |
| *** kjackal_ has quit IRC | 10:26 | |
| *** neon_icon has quit IRC | 10:26 | |
| *** lukebrowning has quit IRC | 10:27 | |
| *** jascott1 has joined #openstack-infra | 10:27 | |
| AJaeger | kazsh: you need to split it in two changes | 10:28 |
| kazsh | AJaeger: oh ok, will do, thanks! | 10:29 |
| *** gildub has joined #openstack-infra | 10:29 | |
| *** wolverineav has joined #openstack-infra | 10:30 | |
| *** andreas_s has quit IRC | 10:31 | |
| *** jascott1 has quit IRC | 10:32 | |
| *** andreas_s has joined #openstack-infra | 10:32 | |
| *** panda is now known as panda|relo | 10:36 | |
| sshnaidm | does zuulv3 run now? I don't see legacy-* jobs anymore | 10:36 |
| *** andreas_s has quit IRC | 10:36 | |
| tosky | it is running, but there is a huge queue | 10:37 |
| AJaeger | sshnaidm: it does - but only 1400+ changes in the queue due to the small quota we gave it. | 10:38 |
| *** kjackal_ has joined #openstack-infra | 10:38 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add publish-to-pypi-neutron template https://review.openstack.org/510531 | 10:42 |
| *** andreas_s has joined #openstack-infra | 10:43 | |
| *** pblaho has joined #openstack-infra | 10:43 | |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project https://review.openstack.org/509119 | 10:43 |
| *** alexchadin has quit IRC | 10:46 | |
| *** alexchadin has joined #openstack-infra | 10:47 | |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat dashboard creation https://review.openstack.org/510833 | 10:49 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat Dashboard creation https://review.openstack.org/510835 | 10:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move infra-gate pipeline after gate https://review.openstack.org/510836 | 10:58 |
| *** ociuhandu has joined #openstack-infra | 10:59 | |
| alexchadin | hi, can I mark own jobs as non voting in .zuul file? | 11:00 |
| alexchadin | if I do so, Zuul throws an error "mapping values are not allowed in this context" | 11:01 |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Follow up change for Heat Dashboard creation https://review.openstack.org/510835 | 11:03 |
| AJaeger | alexchadin: which change? | 11:03 |
| AJaeger | alexchadin: I guess you miss a ":" - let me check... | 11:03 |
| *** jkilpatr has joined #openstack-infra | 11:04 | |
| *** dtantsur has quit IRC | 11:04 | |
| *** nicolasbock_ has joined #openstack-infra | 11:04 | |
| AJaeger | alexchadin: commented on https://review.openstack.org/#/c/509777 | 11:05 |
| AJaeger | you indeed need a colon | 11:05 |
| *** ykarel_ has joined #openstack-infra | 11:06 | |
| *** alexchadin has quit IRC | 11:07 | |
| *** ykarel has quit IRC | 11:08 | |
| *** dtantsur has joined #openstack-infra | 11:08 | |
| *** dtantsur has quit IRC | 11:08 | |
| *** dtantsur has joined #openstack-infra | 11:09 | |
| *** dtantsur has quit IRC | 11:09 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Stop using the cached cirros image https://review.openstack.org/510839 | 11:09 |
| *** dtantsur has joined #openstack-infra | 11:13 | |
| *** andreas_s has quit IRC | 11:13 | |
| *** Guest78751 has quit IRC | 11:14 | |
| *** lucasagomes is now known as lucas-hungry | 11:15 | |
| *** spiffxp has quit IRC | 11:16 | |
| *** alexchadin has joined #openstack-infra | 11:16 | |
| *** nicolasbock_ has quit IRC | 11:16 | |
| *** vipul has quit IRC | 11:16 | |
| *** pcrews has quit IRC | 11:16 | |
| *** vipul has joined #openstack-infra | 11:17 | |
| *** spiffxp has joined #openstack-infra | 11:17 | |
| *** pcrews has joined #openstack-infra | 11:17 | |
| *** andreas_s has joined #openstack-infra | 11:18 | |
| *** andreas_s has quit IRC | 11:22 | |
| *** andreas_s has joined #openstack-infra | 11:23 | |
| *** lukebrowning has joined #openstack-infra | 11:23 | |
| *** jpena is now known as jpena|lunch | 11:23 | |
| *** ykarel_ is now known as ykarel | 11:24 | |
| *** nicolasbock_ has joined #openstack-infra | 11:28 | |
| *** rhallisey has joined #openstack-infra | 11:29 | |
| *** sree has joined #openstack-infra | 11:30 | |
| *** andreas_s has quit IRC | 11:32 | |
| *** sree has quit IRC | 11:34 | |
| *** andreas_s has joined #openstack-infra | 11:37 | |
| *** ldnunes has joined #openstack-infra | 11:38 | |
| *** yamamoto has joined #openstack-infra | 11:38 | |
| *** jaypipes has joined #openstack-infra | 11:39 | |
| openstackgerrit | Alexander Chadin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy watcher jobs https://review.openstack.org/510848 | 11:42 |
| *** yamamoto has quit IRC | 11:44 | |
| *** yamamoto has joined #openstack-infra | 11:44 | |
| openstackgerrit | Alexander Chadin proposed openstack-infra/project-config master: Remove legacy watcher jobs https://review.openstack.org/510849 | 11:47 |
| *** yamamoto has quit IRC | 11:47 | |
| *** andreas_s has quit IRC | 11:48 | |
| *** andreas_s has joined #openstack-infra | 11:48 | |
| *** yamamoto has joined #openstack-infra | 11:48 | |
| *** yamamoto has quit IRC | 11:49 | |
| openstackgerrit | Alexander Chadin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy watcher jobs https://review.openstack.org/510848 | 11:49 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Google Cal Sync https://review.openstack.org/510850 | 11:51 |
| openstackgerrit | Alexander Chadin proposed openstack-infra/project-config master: Remove legacy watcher jobs https://review.openstack.org/510849 | 11:51 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Google Cal Sync https://review.openstack.org/510850 | 11:51 |
| *** geekinutah has quit IRC | 11:54 | |
| *** geekinutah has joined #openstack-infra | 11:54 | |
| *** MasterPiece2 has joined #openstack-infra | 11:54 | |
| *** yamamoto has joined #openstack-infra | 11:55 | |
| *** trown|outtypewww is now known as trown | 11:55 | |
| *** panda|relo is now known as panda | 11:56 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Google Cal Sync https://review.openstack.org/510853 | 11:58 |
| *** eumel8 has quit IRC | 12:00 | |
| *** edmondsw has joined #openstack-infra | 12:03 | |
| *** edmondsw has quit IRC | 12:04 | |
| *** priteau_ has quit IRC | 12:04 | |
| *** edmondsw_ has joined #openstack-infra | 12:05 | |
| *** tpsilva has joined #openstack-infra | 12:06 | |
| *** zzzeek has quit IRC | 12:07 | |
| *** sree has joined #openstack-infra | 12:07 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Google Cal Sync https://review.openstack.org/510853 | 12:08 |
| *** zzzeek has joined #openstack-infra | 12:10 | |
| *** thorst has joined #openstack-infra | 12:16 | |
| *** alexchadin has quit IRC | 12:17 | |
| *** Hal has joined #openstack-infra | 12:22 | |
| *** Hal is now known as Guest84590 | 12:22 | |
| *** eharney has joined #openstack-infra | 12:24 | |
| *** namnh has quit IRC | 12:25 | |
| *** lucas-hungry is now known as lucasagomes | 12:27 | |
| *** yamamoto has quit IRC | 12:27 | |
| *** dprince has joined #openstack-infra | 12:28 | |
| *** yamamoto has joined #openstack-infra | 12:29 | |
| *** rlandy has joined #openstack-infra | 12:32 | |
| *** yamamoto has quit IRC | 12:33 | |
| *** mat128 has joined #openstack-infra | 12:34 | |
| *** esberglu has quit IRC | 12:38 | |
| *** alexchadin has joined #openstack-infra | 12:39 | |
| *** makowals has joined #openstack-infra | 12:40 | |
| *** jpena|lunch is now known as jpena | 12:40 | |
| *** Klerisson has joined #openstack-infra | 12:41 | |
| *** udesale__ has quit IRC | 12:44 | |
| mordred | morning AJaeger ! anything urgent/immedate I should look at? | 12:44 |
| *** Klerisson has left #openstack-infra | 12:46 | |
| *** LindaWang has quit IRC | 12:46 | |
| *** baoli has joined #openstack-infra | 12:49 | |
| *** thorst_ has joined #openstack-infra | 12:49 | |
| *** thorst has quit IRC | 12:49 | |
| *** esberglu has joined #openstack-infra | 12:52 | |
| *** Kevin_Zheng has joined #openstack-infra | 12:52 | |
| *** spectr has quit IRC | 12:53 | |
| AJaeger | mordred: a couple of jobs seem hanging, see infra-check, but nothing immediate otherwise. | 12:53 |
| AJaeger | lots of changes to review - and a far too long queue in v3, we might need to restart it | 12:54 |
| *** spectr has joined #openstack-infra | 12:55 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Remove integration test playbooks from zuul-jobs https://review.openstack.org/510868 | 12:56 |
| *** kashyap has left #openstack-infra | 12:56 | |
| *** gildub has quit IRC | 12:56 | |
| *** zhurong has joined #openstack-infra | 12:57 | |
| *** efried has quit IRC | 12:58 | |
| *** shardy has quit IRC | 13:00 | |
| *** edmondsw_ is now known as edmondsw | 13:01 | |
| *** ykarel has quit IRC | 13:01 | |
| *** ykarel has joined #openstack-infra | 13:02 | |
| *** kzaitsev1pi has quit IRC | 13:05 | |
| *** ldnunes has quit IRC | 13:05 | |
| *** david-lyle has joined #openstack-infra | 13:06 | |
| *** kzaitsev_pi has joined #openstack-infra | 13:06 | |
| *** LindaWang has joined #openstack-infra | 13:11 | |
| *** jistr is now known as jistr|mtg | 13:11 | |
| *** shardy has joined #openstack-infra | 13:11 | |
| *** ykarel_ has joined #openstack-infra | 13:11 | |
| *** ykarel has quit IRC | 13:12 | |
| *** ykarel_ is now known as ykarel | 13:12 | |
| *** lihi has quit IRC | 13:12 | |
| *** kgiusti has joined #openstack-infra | 13:12 | |
| *** leyal has quit IRC | 13:13 | |
| AJaeger | mordred: I proposed https://review.openstack.org/510836 in the hope this changes the display on zuulv3.o.o - think this is usefull for our switch to v3 tomorrow | 13:13 |
| *** dizquierdo has quit IRC | 13:13 | |
| mordred | AJaeger: worth a shot! :) | 13:14 |
| *** dimak has quit IRC | 13:14 | |
| *** oanson has quit IRC | 13:14 | |
| AJaeger | ;) | 13:14 |
| *** dimak has joined #openstack-infra | 13:15 | |
| *** leyal has joined #openstack-infra | 13:15 | |
| *** oanson has joined #openstack-infra | 13:15 | |
| *** lihi has joined #openstack-infra | 13:16 | |
| *** ldnunes has joined #openstack-infra | 13:18 | |
| *** jistr|mtg is now known as jistr | 13:19 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Integration tests: run mirror-info before configure-mirrors https://review.openstack.org/510881 | 13:20 |
| dmsimard | ianw: thanks for picking up unbound, was on pto yesterday | 13:21 |
| dmsimard | ianw: left a batch of reviews | 13:21 |
| *** mriedem has joined #openstack-infra | 13:22 | |
| *** baoli has quit IRC | 13:23 | |
| fungi | mordred: i'm still catching up, but if you didn't see during the zuul meeting yesterday we decided to start the cut-over to v3 at 11:00z tomorrow. one of our top priorities for today is to get refreshed versions of the changes we reverted to un-revert (also we decided to keep the infra-check pipeline in the near term, but leave only project-config in it) | 13:23 |
| dmsimard | ianw: I tagged your reviews with https://review.openstack.org/#/q/topic:zuulv3-unbound to make them easier to track | 13:23 |
| *** baoli has joined #openstack-infra | 13:24 | |
| AJaeger | fungi, I read the notes - and that's why I pushed 510836 to make it easier to find ;) | 13:24 |
| fungi | thanks AJaeger! | 13:25 |
| *** lbragstad has joined #openstack-infra | 13:28 | |
| *** shardy has quit IRC | 13:29 | |
| *** links has quit IRC | 13:29 | |
| *** ccamacho has left #openstack-infra | 13:30 | |
| mordred | fungi: yes! I saw the meeting summary - and also that I was assigned to prepare the revert patches I believe | 13:31 |
| *** bobh has joined #openstack-infra | 13:33 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Remove infra-gate pipelines https://review.openstack.org/510885 | 13:33 |
| *** dave-mccowan has joined #openstack-infra | 13:33 | |
| fungi | mordred: i don't know that you were _formally_ assigned, it was just suggested that we should ask you whether you want to work on that today ;) | 13:33 |
| *** zaneb has quit IRC | 13:33 | |
| AJaeger | fungi: pabelanger took the task ^ | 13:34 |
| mordred | pabelanger: you might want to rebase https://review.openstack.org/#/c/510885/ on top of https://review.openstack.org/510836 | 13:34 |
| fungi | thanks pabelanger! | 13:34 |
| mordred | pabelanger: and also thanks! | 13:34 |
| AJaeger | mordred: 510836 and 510885 are different repos | 13:34 |
| fungi | i have a sort of full morning still, but will hopefully be able to start reviewing between lunch and the infra meeting | 13:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Remove infra-gate pipeline https://review.openstack.org/510886 | 13:35 |
| pabelanger | looking | 13:35 |
| tosky | mordred: hi! I guess that you didn't see my note above, basically related to the two different failures in https://review.openstack.org/#/c/510528/ | 13:35 |
| *** armaan has quit IRC | 13:35 | |
| tosky | a "Permission denied" writing /home/zuul/.cache/... something for legacy-sahara-dsvm-scenario-neutron, and a post_failure apparently not related to the job for legacy-tempest-dsvm-sahara | 13:35 |
| *** armaan has joined #openstack-infra | 13:36 | |
| AJaeger | pabelanger: just +2A 510836 before you remove the other pipelines ;) | 13:36 |
| pabelanger | AJaeger: mordred: we're removing infra-gate, so does it makesense to move it around in config this morning? | 13:36 |
| AJaeger | pabelanger: we leave infra-gate for project-config! | 13:37 |
| *** baoli has quit IRC | 13:37 | |
| pabelanger | AJaeger: Oh, right | 13:37 |
| pabelanger | blah | 13:37 |
| pabelanger | no coffee yet | 13:37 |
| AJaeger | pabelanger: and thus I'd like to have it nicely grouped and able to find | 13:37 |
| * AJaeger sends coffee to pabelanger | 13:37 | |
| pabelanger | +3 | 13:37 |
| AJaeger | thanks | 13:38 |
| *** shardy has joined #openstack-infra | 13:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove infra-gate pipeline https://review.openstack.org/510887 | 13:39 |
| AJaeger | pabelanger: let's WIP that until tomorrow - ok ? ^ | 13:40 |
| mordred | AJaeger, pabelanger: wait- we're only supposed to leave infra-check | 13:41 |
| pabelanger | AJaeger: I don't think it will be an issue, will it? We have no other projects using gate on zuulv3 now | 13:41 |
| *** ramishra has quit IRC | 13:41 | |
| AJaeger | mordred: Argh, infra-check, not infra-gate. ;((((((((( | 13:41 |
| AJaeger | Sorry... | 13:41 |
| mordred | AJaeger, pabelanger: for project-config - because infra-check is high priority vs. normal check | 13:41 |
| pabelanger | yah, that's what I thought | 13:42 |
| AJaeger | mordred: now I need coffee ;) You're right... | 13:42 |
| mordred | AJaeger: I need MORE coffee | 13:42 |
| * AJaeger abandons 510836 | 13:42 | |
| *** jstockto2000 has joined #openstack-infra | 13:42 | |
| fungi | moar coffieee all around | 13:42 |
| pabelanger | okay, let me fix up | 13:43 |
| AJaeger | pabelanger: if you merge infra-gate -> gate now, nothing will merge there- we have gate disabled. So, let's prepare and WIP. | 13:43 |
| pabelanger | AJaeger: Oh, right. because of voting | 13:43 |
| pabelanger | following now | 13:43 |
| pabelanger | Ya, I'll WIP until we are ready | 13:44 |
| openstackgerrit | Ken Giusti proposed openstack-infra/openstack-zuul-jobs master: Remove most of the oslo.messaging legacy tests https://review.openstack.org/510661 | 13:44 |
| AJaeger | pabelanger: let me do the project-config change and reorder at same time, ok? | 13:44 |
| pabelanger | AJaeger: sure | 13:44 |
| *** jstockto2000 has quit IRC | 13:44 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Remove infra-check and infra-gate pipelines https://review.openstack.org/510886 | 13:45 |
| *** zhurong has quit IRC | 13:45 | |
| *** zaneb has joined #openstack-infra | 13:45 | |
| *** gouthamr has joined #openstack-infra | 13:45 | |
| pabelanger | AJaeger: don't forget infra-post | 13:46 |
| pabelanger | we likely also want to remove that | 13:46 |
| *** jstockto2000 has joined #openstack-infra | 13:47 | |
| *** chlong_ has joined #openstack-infra | 13:47 | |
| *** vsaienk0 has quit IRC | 13:47 | |
| *** chlong has joined #openstack-infra | 13:48 | |
| *** ihrachys_ has joined #openstack-infra | 13:48 | |
| *** ihrachys_ has quit IRC | 13:48 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Revert "Use new infra pipelines" https://review.openstack.org/510885 | 13:48 |
| *** Marx314 has joined #openstack-infra | 13:49 | |
| *** Marx314 has quit IRC | 13:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Revert "Use new infra pipelines" https://review.openstack.org/510886 | 13:49 |
| *** zaneb has quit IRC | 13:50 | |
| *** Marx314 has joined #openstack-infra | 13:51 | |
| *** jstockto2000 has quit IRC | 13:52 | |
| *** igordc has quit IRC | 13:52 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Revert "Use new post pipelines" https://review.openstack.org/510887 | 13:52 |
| *** kiennt26 has joined #openstack-infra | 13:53 | |
| pabelanger | I used topic:zuulv3-rollout | 13:54 |
| *** hongbin has joined #openstack-infra | 13:54 | |
| *** andreas_s has quit IRC | 13:54 | |
| *** aviau has quit IRC | 13:55 | |
| *** aviau has joined #openstack-infra | 13:55 | |
| *** igordc has joined #openstack-infra | 13:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove infra specific pipelines and enable pipelines https://review.openstack.org/510892 | 13:56 |
| AJaeger | pabelanger: ^ | 13:56 |
| AJaeger | pabelanger: should be all good but please carefully review | 13:56 |
| pabelanger | AJaeger: sure, I will fetch coffee first | 13:57 |
| AJaeger | ;) | 13:57 |
| *** armax has joined #openstack-infra | 13:58 | |
| *** andreas_s has joined #openstack-infra | 13:59 | |
| *** vsaienk0 has joined #openstack-infra | 13:59 | |
| *** d0ugal has quit IRC | 14:00 | |
| *** d0ugal has joined #openstack-infra | 14:00 | |
| *** d0ugal has quit IRC | 14:00 | |
| *** d0ugal has joined #openstack-infra | 14:00 | |
| *** d0ugal has quit IRC | 14:00 | |
| *** d0ugal has joined #openstack-infra | 14:00 | |
| *** d0ugal has quit IRC | 14:01 | |
| *** d0ugal has joined #openstack-infra | 14:02 | |
| *** d0ugal has quit IRC | 14:02 | |
| *** d0ugal has joined #openstack-infra | 14:02 | |
| *** alexchadin has quit IRC | 14:03 | |
| *** ijw has joined #openstack-infra | 14:03 | |
| AJaeger | a few cleanups and updates to approve if anybody has time (all project-config): https://review.openstack.org/#/c/510659/ https://review.openstack.org/510786 https://review.openstack.org/#/c/510347/ https://review.openstack.org/510435 https://review.openstack.org/510769 https://review.openstack.org/510638 https://review.openstack.org/#/c/510659/ https://review.openstack.org/510636 | 14:04 |
| *** lihi has quit IRC | 14:05 | |
| *** baoli has joined #openstack-infra | 14:05 | |
| *** pblaho has quit IRC | 14:05 | |
| *** felipemonteiro has joined #openstack-infra | 14:05 | |
| *** leyal has quit IRC | 14:06 | |
| *** dimak has quit IRC | 14:06 | |
| *** oanson has quit IRC | 14:06 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:06 | |
| *** dimak has joined #openstack-infra | 14:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update translation and manual jobs https://review.openstack.org/510765 | 14:07 |
| *** leyal has joined #openstack-infra | 14:07 | |
| *** oanson has joined #openstack-infra | 14:07 | |
| AJaeger | and for openstack-zuul-jobs: https://review.openstack.org/509986 and https://review.openstack.org/510637 | 14:07 |
| AJaeger | dmsimard: question by jeblair on https://review.openstack.org/#/c/504788/ | 14:08 |
| *** lihi has joined #openstack-infra | 14:08 | |
| dmsimard | Yes, I'll address it soon. Catching up from 3 day weekend :) | 14:08 |
| AJaeger | ;) | 14:08 |
| *** felipemonteiro has quit IRC | 14:10 | |
| *** andreas_s has quit IRC | 14:12 | |
| pabelanger | AJaeger: lets do 2 patches for 510892. 1st move jobs back into check / gate / post, 2nd delete pipelines. | 14:16 |
| pabelanger | then we can setup Depends-On for 2nd project-config patch | 14:16 |
| *** hemna_ has joined #openstack-infra | 14:16 | |
| prometheanfire | are zuulv3 jobs running? https://review.openstack.org/510565 | 14:16 |
| *** alexchadin has joined #openstack-infra | 14:16 | |
| pabelanger | checking | 14:17 |
| AJaeger | prometheanfire: check zuulv3.status.openstack.org - currently 1600+ changes in queue, it takes ages ;( | 14:18 |
| prometheanfire | oh, lol | 14:18 |
| *** thorst_ has quit IRC | 14:18 | |
| AJaeger | prometheanfire: too small quota | 14:18 |
| prometheanfire | I thought it was keeping up for a while | 14:19 |
| pabelanger | Ya, something might be up. I see a node with 19hr in-use | 14:19 |
| pabelanger | I think we have a hung executor ago | 14:20 |
| pabelanger | again* | 14:20 |
| pabelanger | ze03.o.o is hung | 14:20 |
| pabelanger | checking why | 14:20 |
| *** pblaho has joined #openstack-infra | 14:20 | |
| AJaeger | pabelanger: why two changes? We cannot merge them separetly! | 14:21 |
| pabelanger | zuul 28367 0.0 0.0 21352 3164 ? S Oct09 0:00 git fetch-pack --stateless-rpc --stdin --lock-pack --thin --no-progress https://git.openstack.org/openstack/tripleo-quickstart/ | 14:21 |
| pabelanger | looks like git timeout didn't work as expected | 14:21 |
| AJaeger | pabelanger: Once we merge the first, the gate pipeline is disabled and we can only merge the second - with force-merge. | 14:22 |
| *** kiennt26 has quit IRC | 14:22 | |
| pabelanger | AJaeger: okay, then you see to setup depends-on to my reverts above | 14:22 |
| AJaeger | pabelanger: yes, will do that... | 14:23 |
| pabelanger | I'm going to kill the git process and confirm the executor starts working again, if so, we should then get jeblair hard git timeout landed this morning and restart all executors | 14:23 |
| AJaeger | pabelanger: but it *cannot* merge without those either - since the pipelines are in use... | 14:23 |
| *** zaneb has joined #openstack-infra | 14:24 | |
| AJaeger | but let's make it explicit | 14:24 |
| pabelanger | AJaeger: right, for all this to work, we need to make gate pipeline functional again. But we cannot do that until we stop zuulv3 | 14:24 |
| pabelanger | err | 14:24 |
| pabelanger | zuulv2 | 14:24 |
| pabelanger | so, will be a little bit of dancing to land all this | 14:25 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove infra specific pipelines and enable pipelines https://review.openstack.org/510892 | 14:25 |
| pabelanger | okay, ze03 working again, which is good | 14:25 |
| AJaeger | yeah! | 14:25 |
| pabelanger | ze04 also blocked | 14:25 |
| *** edmondsw has quit IRC | 14:25 | |
| * AJaeger added all his review requests to the etherpad now | 14:25 | |
| AJaeger | pabelanger: argh ;( | 14:26 |
| *** edmondsw has joined #openstack-infra | 14:26 | |
| AJaeger | thanks for checking and restarting, pabelanger | 14:26 |
| *** andreas_s has joined #openstack-infra | 14:27 | |
| AJaeger | wow, 1004 changes in check queue ;( | 14:27 |
| *** edmondsw_ has joined #openstack-infra | 14:27 | |
| *** srobert has joined #openstack-infra | 14:27 | |
| *** dhinesh has joined #openstack-infra | 14:28 | |
| *** edmonds__ has joined #openstack-infra | 14:28 | |
| pabelanger | okay, in total ze03, ze04, ze05 and ze06 were all blocked | 14:29 |
| pabelanger | I have left ze04 blocked so we can audit it when jeblair is online | 14:29 |
| pabelanger | I am not sure why our gitconfig settings didn't work | 14:29 |
| *** edmondsw has quit IRC | 14:30 | |
| *** camunoz has joined #openstack-infra | 14:30 | |
| jeblair | pabelanger: please fix ze04 | 14:31 |
| pabelanger | ack | 14:31 |
| *** thorst has joined #openstack-infra | 14:31 | |
| *** edmondsw_ has quit IRC | 14:31 | |
| pabelanger | and processing again | 14:32 |
| *** claudiub|2 has quit IRC | 14:33 | |
| *** jamesdenton has joined #openstack-infra | 14:34 | |
| AJaeger | yeah! I see progress on some hung ones ;) | 14:35 |
| *** chlong has quit IRC | 14:36 | |
| *** chlong_ has quit IRC | 14:36 | |
| *** chlong has joined #openstack-infra | 14:36 | |
| *** alexchadin has quit IRC | 14:37 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Retry change queries on a 502 response https://review.openstack.org/510706 | 14:38 |
| *** pblaho has quit IRC | 14:38 | |
| *** pblaho has joined #openstack-infra | 14:41 | |
| *** pblaho has quit IRC | 14:41 | |
| *** pblaho has joined #openstack-infra | 14:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Add announce-release job to stay in project-config https://review.openstack.org/510905 | 14:42 |
| AJaeger | mordred: as discussed yesterday ^ | 14:42 |
| *** nikhil has joined #openstack-infra | 14:44 | |
| *** andreas_s has quit IRC | 14:44 | |
| mordred | AJaeger: I wonder if we should consider making some project-templates like "python-pti" and "release-managed" or similar (not today) | 14:44 |
| *** MasterPiece2 has quit IRC | 14:46 | |
| mordred | AJaeger: and potentially a "system-required" which is a template into which we put things like the merge-check jobs - but if we decided that there was something else that **all** projects in zuul need to run for logistical or system reasons | 14:46 |
| AJaeger | mordred: nice idea - go for it ;) | 14:46 |
| *** pblaho has quit IRC | 14:46 | |
| prometheanfire | AJaeger: looks like the grub2 dib element may not work on ubuntu xenial again | 14:47 |
| *** jgriffith has quit IRC | 14:48 | |
| AJaeger | prometheanfire: are you sure I'm the right person to address? I don't rememeber anything here... | 14:48 |
| prometheanfire | AJaeger: you were the person to last commit to that element in feb | 14:48 |
| prometheanfire | oh, wrong a<tab> | 14:48 |
| prometheanfire | AJaeger: ya, wrong person | 14:48 |
| *** rcernin has quit IRC | 14:49 | |
| AJaeger | ;) | 14:50 |
| prometheanfire | ansreas around then :P | 14:50 |
| prometheanfire | or flonatel | 14:51 |
| prometheanfire | not sure which nick is used | 14:51 |
| *** links has joined #openstack-infra | 14:51 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Migrate ansible-hardening to zuulv3 https://review.openstack.org/510623 | 14:52 |
| *** dizquierdo has joined #openstack-infra | 14:54 | |
| *** xarses has joined #openstack-infra | 14:56 | |
| *** ykarel is now known as ykarel|afk | 14:57 | |
| *** jgriffith_ has joined #openstack-infra | 14:58 | |
| AJaeger | pabelanger, mordred, fungi, could you try going over open reviews today if there're no fires, please? I'd like to see fixes for jobs and removal of unneeded jobs in... | 15:01 |
| *** thorst has quit IRC | 15:01 | |
| *** pblaho has joined #openstack-infra | 15:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add git timeout https://review.openstack.org/509517 | 15:02 |
| openstackgerrit | Ken Giusti proposed openstack-infra/zuul-jobs master: Parameterize the test-setup role https://review.openstack.org/510907 | 15:03 |
| *** vsaienk0 has quit IRC | 15:04 | |
| EmilienM | mordred, clarkb: afik sshnaidm fixed all blockers to run zuul v3 legacy tripleo jobs (Sagi, tell me if I'm wrong) | 15:06 |
| *** felipemonteiro_ has quit IRC | 15:07 | |
| AJaeger | EmilienM: Woot! \o/ | 15:07 |
| sshnaidm | EmilienM, all that I was aware of them | 15:07 |
| openstackgerrit | Cedric Savignan proposed openstack-infra/project-config master: networking-bgpvpn: add translation jobs https://review.openstack.org/509178 | 15:07 |
| *** spectr has quit IRC | 15:07 | |
| mhayden | AJaeger: thanks for the help yesterday -- both patches merged for ansible-hardening but i'm still seeing zuulv2 jobs being run for that repo | 15:08 |
| EmilienM | mordred, clarkb: could we try to migrate tripleo jobs to zuulv3 again (running legacy jobs)? | 15:08 |
| dmsimard | EmilienM: legacy jobs are already being run, no ? | 15:08 |
| AJaeger | mhayden: we still have v2 running | 15:09 |
| EmilienM | dmsimard: they are, but I meant to say "we're ready to try again" | 15:09 |
| sshnaidm | EmilienM, last patch is in the gate still: https://review.openstack.org/#/c/510839/ | 15:09 |
| EmilienM | ah this one ok | 15:09 |
| EmilienM | sshnaidm: why did you comment it and not remove? you plan to clean this up later? | 15:10 |
| dmsimard | EmilienM: there's >1k changes in the v3 check queue so it's entirely possible jobs aren't reporting back in a timely manner | 15:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add git timeout https://review.openstack.org/509517 | 15:10 |
| sshnaidm | EmilienM, yeah, after we use one version of zuul, and not two together | 15:11 |
| *** armaan has quit IRC | 15:11 | |
| EmilienM | ok | 15:11 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Added openstack/requirements project to required_projects for Omni project https://review.openstack.org/510015 | 15:12 |
| sshnaidm | EmilienM, anyway I'll be keeping eye on switching to v3 tomorrow | 15:13 |
| *** cshastri has quit IRC | 15:13 | |
| EmilienM | sshnaidm: I'm glad we're more ready than last time, thanks | 15:14 |
| sshnaidm | ya.. we should have started from parallel run :) | 15:14 |
| smcginnis | Question about the zuul switch. Has anyone done a check of querying where jenkins has left +1 but zuul left -1? | 15:17 |
| smcginnis | Seems like the results of that should be pretty small before we decide to flip the switch again. | 15:17 |
| *** ccamacho has joined #openstack-infra | 15:17 | |
| smcginnis | And my unsubstantiated impression right now that it will not be a small list. | 15:17 |
| *** psachin has quit IRC | 15:18 | |
| *** vhosakot has joined #openstack-infra | 15:19 | |
| mnaser | smcginnis in my experience, fixing a job generally is very quick and im sure infra will be all hands on deck to help those quick fixes | 15:19 |
| mnaser | 99% of the time its a missing required-project or some dependency on a ZUUL_* var | 15:19 |
| *** links has quit IRC | 15:20 | |
| *** spectr has joined #openstack-infra | 15:20 | |
| smcginnis | mnaser: Yeah, but I would much rather see those quick fixes happen before the switch rather than after. | 15:20 |
| mnaser | it's extremely hard to do that zuul -1 / jenkins +1 because there isn't a lot of resources available for zuulv3 to do that afaik | 15:20 |
| mnaser | smcginnis i think it's more of a resource problem, we cant allocate enough resources to zuul to do that i think | 15:20 |
| smcginnis | mnaser: Not sure what you mean. | 15:21 |
| mnaser | smcginnis: the nodes which are used for testing are not shared across zuulv2 and v3 | 15:21 |
| mnaser | at the moment, v3 is opearting with 20% of the total donated capacity | 15:21 |
| smcginnis | mnaser: But we can query gerrit. There are plenty I've seen where jenkins has +1 and zuul has -1. Not sure how the node capacity fits there. | 15:21 |
| mnaser | smcginnis: true, but the problem is zuul might have not reported for a day or two because it's so backed up.. i know some projects have zuul fixes but the result can't be seen yet cause its jobs are queued for hours now | 15:22 |
| mnaser | and i think we're generating more patches than zuulv3 can run, so the queue will just increase forever | 15:23 |
| mnaser | (with the small subset of resources that it has at the moment at least) | 15:23 |
| *** thorst has joined #openstack-infra | 15:23 | |
| smcginnis | mnaser: OK, I get that. Not trying to be a jerk about this, but I think more work needs to be done to make sure we are ready to make the switch. I think everyone appreciates the work that has been/is being done, but there's a lot of "frustration" towards the gate still from the last attempt. | 15:24 |
| mhayden | AJaeger: well, the odd part is that both of those patches merged, but nothing has changed in the list of jobs that runs: https://gist.github.com/major/8bea45359bf8a3c5a057c2eee1388715 | 15:25 |
| mhayden | also, i noticed that the zuulv2 jobs aren't prefaced with 'legacy-' | 15:26 |
| *** andreas_s has joined #openstack-infra | 15:26 | |
| fungi | smcginnis: basically, we _can_ do what you're proposing by reallocating all the zuul v2 resources to run v3 jobs and ruen off v2, but that's basically cutting over to v3 anyway | 15:26 |
| mnaser | smcginnis totally, i get that, but i think part of that frustration was from the zuul issues that made it harder to fix things. i twas a bit of a compounded problem. jobs were broken and we couldn't fix them because zuul has issues | 15:26 |
| fungi | s/ruen/turn/ | 15:26 |
| mnaser | now that zuul runs much smoother, it'll be much easier and quicker to fix jobs, + we are having a dedicated queue for project-config which means that changes to project-config will get faster CI response | 15:27 |
| smcginnis | I'm just saying we should understand these before we flip the switch again: https://review.openstack.org/#/q/label:Verified%253E%253D1%252Cjenkins+label:Verified%253C0%252Czuul | 15:27 |
| mnaser | so iterating fixes will be quicker | 15:27 |
| mnaser | smcginnis TIL about label: in search, thanks :D | 15:27 |
| smcginnis | mnaser: It required some googling on my part. ;) | 15:27 |
| mnaser | i do think you raise a good point for us to sift through those to identify potential issues | 15:28 |
| fungi | smcginnis: an alternative i suppose is that we ask openstack to stop pushing up changes to projects for a while, so that we actually have enough capacity to run duplicates of all jobs in a timely fashion, but i doubt that would work out particularly well either | 15:28 |
| smcginnis | fungi: Hah, no, that was imposed a week or two ago and it didn't go over well from what I saw. :) | 15:29 |
| *** edmonds__ has quit IRC | 15:29 | |
| AJaeger | mhayden: yes, we switch off Zuul v2 tomorrow... | 15:29 |
| AJaeger | mhayden: none of the changes we merged disables the v2 jobs | 15:30 |
| mhayden | ah okay -- i'm trying to understand how all of these events impacts things ;) | 15:30 |
| mnaser | i'm pretty confident tomorrow will go over much smoother | 15:30 |
| *** thorst has quit IRC | 15:30 | |
| mhayden | AJaeger: shouldn't i have more than the docs job in v3? | 15:30 |
| mhayden | feel free to flog me if i'm misreading the migration guide ;) | 15:30 |
| *** gibi has quit IRC | 15:30 | |
| AJaeger | mhayden: Ah, more - yes, you should - but we have a queue of 1000+ jobs. Did any jobs finish? | 15:31 |
| AJaeger | 1000+ changes I mean | 15:31 |
| AJaeger | sorry, have to collect my daughter - bbl | 15:31 |
| *** udesale__ has joined #openstack-infra | 15:31 | |
| mhayden | i've only see one job in the zuulv3 queue ever -- it has finished successfully once though | 15:31 |
| *** shardy_ has joined #openstack-infra | 15:32 | |
| fungi | smcginnis: i see you caught my joke ;) | 15:32 |
| smcginnis | fungi: ;) | 15:32 |
| *** gibi has joined #openstack-infra | 15:32 | |
| *** edmondsw has joined #openstack-infra | 15:32 | |
| *** esberglu has quit IRC | 15:32 | |
| mhayden | so i heard we're supposed to take an openstack vacation this week | 15:33 |
| *** udesale__ has quit IRC | 15:33 | |
| *** esberglu has joined #openstack-infra | 15:33 | |
| *** thorst has joined #openstack-infra | 15:34 | |
| *** andreas_s has quit IRC | 15:34 | |
| *** edmondsw_ has joined #openstack-infra | 15:36 | |
| *** ccamacho has quit IRC | 15:37 | |
| *** dizquierdo has quit IRC | 15:37 | |
| *** edmondsw has quit IRC | 15:37 | |
| *** esberglu has quit IRC | 15:37 | |
| clarkb | smcginnis: re your last email I think that was the expectation ~ a week ago when we did the soft rollback. Zuulv3 would be reporting on your changes as well as zuulv2, please talk to us about problems with the zuulv3 results (and since then we've continued to help fix job issues in places like docs publishing and tripleo and so on) | 15:38 |
| *** thorst has quit IRC | 15:39 | |
| clarkb | basically rhere is like half a million lines of configuration here. There are lots of fun unexpected corner cases | 15:39 |
| clarkb | and the only way we are ever going to manage a transition is to do it, and fix those problems as they arise | 15:39 |
| *** shardy_ has quit IRC | 15:40 | |
| clarkb | part of that is due to resource constraints as fungi and mnaser pointed out, but also just due to general size of it | 15:41 |
| *** edmondsw_ has quit IRC | 15:41 | |
| clarkb | this is why we've been working hard to get this done early in the cycle (same as the gerrit upgrade) | 15:41 |
| smcginnis | Are the POST_FAILURE and RETRY_LIMIT failures cleared up? Or are they a result of the constrained resources? | 15:41 |
| clarkb | smcginnis: I believe the POST_FAILUREs should largely be cleaned up. RETRY_LIMIT is likely due to bugs in the job pre playbook(s) | 15:42 |
| *** reed_ has joined #openstack-infra | 15:42 | |
| smcginnis | OK, I'll watch for latest results and raise any issues I see. | 15:42 |
| *** thorst has joined #openstack-infra | 15:43 | |
| *** reed_ has quit IRC | 15:44 | |
| *** yamahata has joined #openstack-infra | 15:44 | |
| frickler | fungi: I haven't found any logs for the renamed v3 periodic jobs, do you know whether they ran at all or whether zuul simply didn't have resources for that? | 15:44 |
| *** e0ne has quit IRC | 15:45 | |
| *** slaweq has joined #openstack-infra | 15:45 | |
| *** thorst_ has joined #openstack-infra | 15:46 | |
| EmilienM | I know the zuulv3 queue is huge but is it expected that UI is freaking slow? http://zuulv3.openstack.org/ | 15:47 |
| jeblair | EmilienM: yes. those things are related. it's managable at normal loads. we have plans to fix after cutover. | 15:47 |
| EmilienM | good to know, thanks | 15:48 |
| clarkb | jeblair: could we also use the existing status page? iirc it performs a bit better? | 15:48 |
| *** andreas_s has joined #openstack-infra | 15:49 | |
| *** thorst has quit IRC | 15:49 | |
| jeblair | clarkb: i'd rather drop it. it's twice the work. | 15:49 |
| *** edmondsw has joined #openstack-infra | 15:50 | |
| fungi | frickler: since periodic jobs get lowest priority, i expect those simply haven't run yet because v3 has more jobs queued in check than it will be able to catch up with | 15:51 |
| jeblair | clarkb, fungi: should i issue a gear release with https://review.openstack.org/508698 ? | 15:51 |
| jeblair | 0.10.1 i guess? | 15:52 |
| fungi | jeblair: i'm skimming the commit log | 15:52 |
| jeblair | i think that's the only change since 10.0 | 15:52 |
| fungi | it's just that one patch, yeah | 15:52 |
| *** LindaWang has quit IRC | 15:52 | |
| fungi | so i guess it depends on whether you consider the addition of a send lock to be a new feature or a bug fix | 15:52 |
| jeblair | bugfix; not user visible | 15:53 |
| fungi | 0.19.1 seems fine to me | 15:53 |
| fungi | er | 15:53 |
| fungi | 0.10.1 i mean | 15:53 |
| *** andreas_s has quit IRC | 15:53 | |
| clarkb | wfm | 15:53 |
| *** thorst has joined #openstack-infra | 15:53 | |
| jeblair | done | 15:53 |
| fungi | we're running with that on zuulv3 right? | 15:53 |
| jeblair | nope | 15:53 |
| *** csomerville has joined #openstack-infra | 15:53 | |
| jeblair | it actually only affects the clients | 15:54 |
| fungi | ahh, so the mergers and executors have it installed | 15:54 |
| *** thorst_ has quit IRC | 15:54 | |
| jeblair | fungi: no, it's not installed anywhere | 15:54 |
| pabelanger | while the backlog is huge, zuulv3 itself is holding up very well now | 15:54 |
| *** edmondsw has quit IRC | 15:55 | |
| *** edmondsw has joined #openstack-infra | 15:56 | |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Adding the Contributor Guide Project https://review.openstack.org/509943 | 15:56 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Adding check/gate jobs to Contributor Guide https://review.openstack.org/509937 | 15:56 |
| *** egonzalez has quit IRC | 15:57 | |
| thingee | hey infra, I think the contributor guide project is ready ^ - I'm aiming to launch the contributor portal on openstack.org/community this week, but we need this. | 15:58 |
| *** ykarel|afk has quit IRC | 15:58 | |
| *** yamahata has quit IRC | 16:00 | |
| *** dave-mcc_ has joined #openstack-infra | 16:01 | |
| *** rbrndt has joined #openstack-infra | 16:04 | |
| *** dave-mccowan has quit IRC | 16:04 | |
| *** bhavik1 has joined #openstack-infra | 16:06 | |
| mordred | pabelanger: that, it turns out, is super important | 16:06 |
| *** felipemonteiro has joined #openstack-infra | 16:06 | |
| mordred | pabelanger: zuul should not degrade when the backlog is huge, otherwise building a huge backlog is very dangerous :) | 16:07 |
| *** andreas_s has joined #openstack-infra | 16:07 | |
| pabelanger | indeed | 16:07 |
| *** pcaruana has quit IRC | 16:07 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:07 | |
| *** lucasagomes is now known as lucas-afk | 16:08 | |
| *** gouthamr has quit IRC | 16:10 | |
| *** felipemonteiro has quit IRC | 16:11 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for volume detach 400 bug 1722577 https://review.openstack.org/510926 | 16:12 |
| openstack | bug 1722577 in tempest "test_list_get_volume_attachments failing with 400 error on teardown when detaching an already detached volume" [Medium,Triaged] https://launchpad.net/bugs/1722577 | 16:12 |
| *** ralonsoh_ has joined #openstack-infra | 16:13 | |
| *** spzala has joined #openstack-infra | 16:13 | |
| *** spzala has quit IRC | 16:13 | |
| *** jascott1 has joined #openstack-infra | 16:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job https://review.openstack.org/510237 | 16:15 |
| *** ralonsoh has quit IRC | 16:16 | |
| *** armaan has joined #openstack-infra | 16:18 | |
| *** andreas_s has quit IRC | 16:20 | |
| clarkb | re thingee's changes above any idea what happened in http://logs.openstack.org/43/509943/7/infra-check/project-config-gerrit/bd9fc7f/ara/ <- mordred jeblair pabelanger ? that resulted in POST_FAILURE | 16:23 |
| *** liusheng has quit IRC | 16:23 | |
| clarkb | thingee: the other failure is http://logs.openstack.org/43/509943/7/infra-check/project-config-frozen-files/31b0555/job-output.txt.gz#_2017-10-10_16_02_34_169545 so I thinkwe'd like to figure that out post v3 migration tomorrow? | 16:24 |
| clarkb | AJaeger: ^ your input owuldbe valuable on that I think | 16:25 |
| *** andreas_s has joined #openstack-infra | 16:25 | |
| pabelanger | clarkb: I think it is possible it could be logstash integration. Hard to say, because we do that task after we upload our logs, and nothing in printed to debug.logs by default | 16:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add release-openstack-javascript job https://review.openstack.org/502185 | 16:25 |
| pabelanger | clarkb: I think we could move http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base/post-logs.yaml#n13 before upload-logs role | 16:26 |
| pabelanger | to help provide more info | 16:26 |
| clarkb | thingee: AJaeger I guess that job is non voting so we can make a judgement call? | 16:26 |
| clarkb | pabelanger: the issue with that is logstash expects the logs to be present when it gets the jobs and they won't be if we flip the order around | 16:27 |
| thingee | mordred: openstack-infra/openstack-zuul-jobs is what I was looking for job definitions. This is where a lot of my confusion was with project-config | 16:27 |
| thingee | whew, omg the world makes sense again | 16:27 |
| thingee | for now | 16:28 |
| mordred | thingee: \o/ | 16:28 |
| pabelanger | clarkb: right, so we have a potential blind spot in logging I think | 16:28 |
| mordred | thingee: so - we have job documentation generation on a per-repo basis currently, such as https://docs.openstack.org/infra/openstack-zuul-jobs/ ... | 16:28 |
| pabelanger | 2017-10-10 16:01:18,261 DEBUG zuul.AnsibleJob: [build: bd9fc7fd8d5b46b18e51c5dd1fb4b8ee] Ansible exit code: 2 | 16:28 |
| pabelanger | that is from last post playbook, but debug.log is lacking any other ansible output | 16:29 |
| mordred | thingee: we have a roadmap item to teach zuul how to make a consolidated view of job documentation for all of the jobs it knows about in its combined job config | 16:29 |
| jeblair | pabelanger, clarkb, mordred: i think we're talking about the same class of failure here and in #zuul | 16:29 |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config master: neutron: Extend windowSize for periodic jobs https://review.openstack.org/510934 | 16:30 |
| mordred | jeblair: agree | 16:30 |
| fungi | clarkb: pabelanger: should we at a minimum make the logstash and subunit tasks always exit 0/succeed even if they fail to do what we want? we should probably instead have some other means of reporting breakages for those tasks without failing the jobs themselves since (in theory) there's nothing a job should be able to do to break those post tasks | 16:30 |
| pabelanger | jeblair: I believe so too | 16:30 |
| fungi | jeblair: oh, excellent, i'll change venues | 16:31 |
| *** armaan_ has joined #openstack-infra | 16:31 | |
| clarkb | I'm catching up on scrollback there | 16:31 |
| pabelanger | fungi: ignore_errors: yes should provide that on ansible tasks | 16:31 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for volume detach 400 bug 1722577 https://review.openstack.org/510926 | 16:31 |
| openstack | bug 1722577 in OpenStack Compute (nova) "test_list_get_volume_attachments failing with 400 error on teardown when detaching an already detached volume" [Undecided,Triaged] https://launchpad.net/bugs/1722577 | 16:31 |
| jeblair | pabelanger: yeah, want to go ahead and do that? | 16:31 |
| *** dhinesh has quit IRC | 16:31 | |
| pabelanger | jeblair: sure | 16:32 |
| *** armaan has quit IRC | 16:33 | |
| *** jascott1 has quit IRC | 16:33 | |
| Guest66098 | pabelanger: we are having an emergency that requires putting the inap-mtl01 region offline for a minimum of 24h. Is there something you can do to mitigate the impact on the CI? This should be effective ASAP | 16:34 |
| *** Guest66098 is now known as mgagne | 16:34 | |
| *** mgagne has quit IRC | 16:34 | |
| *** mgagne has joined #openstack-infra | 16:34 | |
| mgagne | pabelanger: was me ^ | 16:34 |
| *** eharney has quit IRC | 16:35 | |
| clarkb | mgagne: we'll want to set the max-servers to 0 in that region | 16:35 |
| clarkb | mgagne: do you want us to push that patch up or do you want to do it? | 16:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update statsd output for tenants https://review.openstack.org/510580 | 16:35 |
| mgagne | I can do it | 16:35 |
| *** eharney has joined #openstack-infra | 16:35 | |
| *** ociuhandu has quit IRC | 16:36 | |
| clarkb | mgagne: ok you might want to grep the provider name and make sure you get it in the nodepool/zuul v2 and v3 configs | 16:36 |
| mgagne | ok, I'm not familiar with zuul v3 | 16:36 |
| clarkb | mgagne: should mostly be the same config its just kept in multiple places while we transition | 16:36 |
| clarkb | so two files to edit instead of one | 16:36 |
| *** dhinesh has joined #openstack-infra | 16:37 | |
| pabelanger | nodepool.yaml and nl01.openstack.org.yaml are the files in project-config | 16:37 |
| *** pvaneck has joined #openstack-infra | 16:38 | |
| mgagne | found them | 16:38 |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config master: Offline inap-mtl01 region https://review.openstack.org/510936 | 16:38 |
| *** andreas_s has quit IRC | 16:38 | |
| *** Guest84590 has quit IRC | 16:39 | |
| *** jpich has quit IRC | 16:39 | |
| *** csomerville has quit IRC | 16:41 | |
| *** dhinesh has quit IRC | 16:41 | |
| *** csomerville has joined #openstack-infra | 16:41 | |
| clarkb | infra-root ^ can we get that reviewed for mgagne? make sure I didn't miss anything :) | 16:44 |
| pabelanger | looking | 16:44 |
| pabelanger | ltgm +3 | 16:44 |
| *** ganso has joined #openstack-infra | 16:44 | |
| *** mika has quit IRC | 16:46 | |
| *** shardy has quit IRC | 16:46 | |
| *** esberglu has joined #openstack-infra | 16:46 | |
| jeblair | more like +8 | 16:47 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Add announce-release job to stay in project-config https://review.openstack.org/510905 | 16:48 |
| *** andreas_s has joined #openstack-infra | 16:52 | |
| *** tesseract has quit IRC | 16:53 | |
| *** derekh has quit IRC | 16:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: base-test: ignore_errors for logstash processing https://review.openstack.org/510943 | 16:54 |
| pabelanger | jeblair: mordred: clarkb: fungi: ^proposed fix to ignore errors on logstash workers. | 16:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Offline inap-mtl01 region https://review.openstack.org/510936 | 16:55 |
| pabelanger | Other option is to setup ignore_errors directly on tasks in each role | 16:55 |
| clarkb | pabelanger: why do you switch it to include roles at the same time? | 16:55 |
| clarkb | is it to avoid ^ | 16:55 |
| pabelanger | clarkb: right | 16:55 |
| pabelanger | I don't know if we can use top level ignore_errors for a play | 16:55 |
| pabelanger | let me test quickly | 16:55 |
| *** Swami has joined #openstack-infra | 16:56 | |
| pabelanger | oh, apparently you can | 16:56 |
| pabelanger | let me do that | 16:57 |
| *** gouthamr has joined #openstack-infra | 16:57 | |
| *** ociuhandu has joined #openstack-infra | 16:57 | |
| jeblair | pabelanger: if we use include role, need to check with dmsimard to make sure it's not the variety that will bomb the json plugin | 16:57 |
| jeblair | it's probably worth noting that if we do this, we will never see these errors | 16:58 |
| *** jascott1 has joined #openstack-infra | 16:58 | |
| jeblair | i'm having second thoughts :) | 16:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: base-test: ignore_errors for logstash processing https://review.openstack.org/510943 | 16:58 |
| pabelanger | jeblair: clarkb: ^ should work too, at least in local testing | 16:58 |
| pabelanger | jeblair: Ya, thats the downside | 16:59 |
| *** andreas_s has quit IRC | 16:59 | |
| *** gouthamr has quit IRC | 17:00 | |
| dmsimard | pabelanger: would love to troubleshoot the zuul_json issue when we have a chance.. need executor logs | 17:01 |
| *** gouthamr has joined #openstack-infra | 17:01 | |
| openstackgerrit | Sam Betts proposed openstack-infra/devstack-gate master: Allow EOL branches to use the test-matrix.py script https://review.openstack.org/510946 | 17:02 |
| *** slaweq has quit IRC | 17:02 | |
| *** MasterOfBugs has joined #openstack-infra | 17:02 | |
| *** inc0 has quit IRC | 17:02 | |
| pabelanger | dmsimard: which review? | 17:03 |
| *** inc0 has joined #openstack-infra | 17:03 | |
| *** lifeless has quit IRC | 17:03 | |
| *** dhinesh has joined #openstack-infra | 17:03 | |
| *** baoli has quit IRC | 17:04 | |
| *** mika has joined #openstack-infra | 17:04 | |
| *** mika has joined #openstack-infra | 17:04 | |
| *** dbecker has quit IRC | 17:04 | |
| *** ralonsoh_ has quit IRC | 17:04 | |
| *** baoli has joined #openstack-infra | 17:04 | |
| *** trown is now known as trown|lunch | 17:05 | |
| *** jpena is now known as jpena|off | 17:05 | |
| *** Apoorva has joined #openstack-infra | 17:07 | |
| *** dhinesh has quit IRC | 17:08 | |
| *** panda is now known as panda|off | 17:08 | |
| dmsimard | pabelanger: https://review.openstack.org/#/c/504238/ | 17:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures https://review.openstack.org/510952 | 17:10 |
| pabelanger | dmsimard: k, I should have some time shortly to look at executors | 17:10 |
| dmsimard | It might be too far back, hopefully not | 17:10 |
| *** armaan_ has quit IRC | 17:12 | |
| *** armaan has joined #openstack-infra | 17:12 | |
| *** jkilpatr_ has joined #openstack-infra | 17:13 | |
| *** sambetts is now known as sambetts|afk | 17:14 | |
| *** jkilpatr has quit IRC | 17:14 | |
| *** rwsu has quit IRC | 17:16 | |
| *** armaan has quit IRC | 17:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: base-test: ignore_errors for logstash processing https://review.openstack.org/510943 | 17:17 |
| *** ram5391 has joined #openstack-infra | 17:17 | |
| AJaeger | frickler: periodic jobs did not run since they have lowest prio | 17:19 |
| *** Goneri has joined #openstack-infra | 17:20 | |
| *** lifeless has joined #openstack-infra | 17:20 | |
| AJaeger | thingee: we can only merge the contributor guide if it has adoptiong PTL+1 - so talk with pkovar first | 17:20 |
| AJaeger | clarkb: yes, we made the job non-voting so that we can make a judgement call and check that everything is ok. | 17:21 |
| *** baoli has quit IRC | 17:23 | |
| AJaeger | frickler: sorry, was reading backscroll and see fungi answered already | 17:23 |
| *** lukebrowning has quit IRC | 17:24 | |
| *** baoli has joined #openstack-infra | 17:24 | |
| *** lukebrowning has joined #openstack-infra | 17:24 | |
| clarkb | AJaeger: thingee so I guess we recheck that change to get the post failure sorted and then decide about judgement calling? | 17:25 |
| mgagne | pabelanger: sorry... the plug got pulled earlier than expected... | 17:26 |
| *** MasterOfBugs has quit IRC | 17:27 | |
| *** rwsu has joined #openstack-infra | 17:27 | |
| *** baoli has quit IRC | 17:28 | |
| AJaeger | clarkb: and wait for docs PTL +1 first... | 17:29 |
| pabelanger | mgagne: np, thanks for the heads up | 17:29 |
| AJaeger | clarkb: I haven't seen pkovar comment on any of these reviews and we merge new project creation only with +1 from adopting PTL | 17:30 |
| *** persia has quit IRC | 17:30 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add the statusbot to the #openstack-tc IRC channel https://review.openstack.org/510964 | 17:30 |
| *** rbrndt has quit IRC | 17:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: DNM: testing base-test changes https://review.openstack.org/510965 | 17:31 |
| *** rwsu has quit IRC | 17:32 | |
| AJaeger | clarkb, thingee - I pinged pkovar on #openstack-doc... | 17:32 |
| clarkb | AJaeger: thanks | 17:33 |
| thingee | AJaeger: oh right, thank you | 17:33 |
| *** pkovar has joined #openstack-infra | 17:34 | |
| pkovar | AJaeger: am here | 17:34 |
| AJaeger | thingee: pkovar left some questions for you;) I asked him to join us here | 17:34 |
| pkovar | just to confirm the status of the new project, really | 17:34 |
| *** baoli has joined #openstack-infra | 17:35 | |
| AJaeger | pkovar: if it's in docs team, the docs team will own it but might have a subteam for it... | 17:35 |
| *** baoli has quit IRC | 17:35 | |
| pkovar | AJaeger: right, so we could also say the subteam would own it ;) | 17:36 |
| AJaeger | pkovar: in case the subteam has problems or disolves, it would go back to the docs team... | 17:36 |
| AJaeger | pkovar: you can also object and thingee needs to put it elsewhere. Since it looks you haven't discussed this yet: I suggest that thingee proposes on openstack-dev mailing list with tag [docs] the contributor guide and how it would be part of docs team... | 17:38 |
| pkovar | AJaeger: well yes, but i really want to make sure the responsibilities at this point are clearly set | 17:38 |
| *** nicolasbock_ has quit IRC | 17:38 | |
| pkovar | AJaeger: fine with me | 17:38 |
| *** dtantsur is now known as dtantsur|afk | 17:38 | |
| *** nicolasbock has joined #openstack-infra | 17:39 | |
| pkovar | or, if thingee agrees to just go ahead and create a specialty team, that's also fine with me | 17:39 |
| thingee | pkovar: sounds fine to me that the subteam takes responsibility | 17:39 |
| pkovar | thingee: cool, let's comment on the change so i can give a formal +1 then :) | 17:40 |
| AJaeger | thingee, pkovar, could either of you follow up on the mail that thingee sent and say that this will be part of docs team as a speciality team? We should inform rest of docs team. We can reference the discussion, but let's summarize and communicate... | 17:41 |
| *** priteau has joined #openstack-infra | 17:41 | |
| pkovar | thingee: are you ok with leading the subteam? or can you appoint a contact? | 17:42 |
| *** tosky has quit IRC | 17:43 | |
| AJaeger | thingee: could you add a change similiar to https://review.openstack.org/510925 then, please? We'd like to have subteams documented | 17:43 |
| *** baoli has joined #openstack-infra | 17:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Mask dependent config errors https://review.openstack.org/510968 | 17:48 |
| jeblair | mordred: ^ that's a thing that we batted around a few days ago i think. i suspect that class of change will occur quite a lot over the next couple of weeks, so if we get that in before tomorrow, we may save a lot of people a lot of confusion. | 17:49 |
| jeblair | AJaeger, fungi, clarkb, pabelanger: ^ | 17:50 |
| AJaeger | mordred: why is https://review.openstack.org/#/c/510237/3/zuul.d/project.yaml needed? | 17:50 |
| AJaeger | jeblair: yeah, seen it quite often - thanks for working on it! | 17:51 |
| thingee | AJaeger: I'm ok with leading the subteam. I'll propose the change now. | 17:51 |
| jeblair | also, i think it would be really good to carefully consider the wording in that message. i tried to write it as clear and concisely as possible, but if it doesn't help people understand we should change it. :) | 17:51 |
| pkovar | thanks, thingee | 17:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures https://review.openstack.org/510952 | 17:51 |
| *** cody-somerville has joined #openstack-infra | 17:52 | |
| mordred | AJaeger: it is not - that got sucked in accidentally - fixing | 17:52 |
| *** sshnaidm is now known as sshnaidm|off | 17:52 | |
| pkovar | and i'll send a followup email to -dev later if you don't beat me to it :) | 17:52 |
| pkovar | got to go now | 17:53 |
| *** jcoufal has joined #openstack-infra | 17:53 | |
| *** dhinesh has joined #openstack-infra | 17:53 | |
| mordred | AJaeger: oh! wait - I remember now ... | 17:53 |
| AJaeger | jeblair: text is good, one suggestion inline | 17:53 |
| mordred | AJaeger: it is needed - or at least it was for a minute nad will again ... | 17:53 |
| AJaeger | mordred: something to mention in commit message? | 17:54 |
| *** csomerville has quit IRC | 17:54 | |
| mordred | AJaeger: the tox-linters job for openstack-zuul-jobs pulls in the contents of zuul-jobs so that it can validate against roles that come from zuul-jobs ... but that patch had a depends-on on a zuul-jobs patch, thus the linters did not have the correct roles available | 17:54 |
| mordred | AJaeger: yah | 17:54 |
| fungi | clarkb: the [nova] in the subject line for that e-mail you just responded to on the infra ml makes me think this is a confused user posting questions about running openstack in the wrong venue. expect a very confused response | 17:54 |
| mordred | AJaeger: actually ... | 17:55 |
| AJaeger | mordred: ah, I see. | 17:55 |
| clarkb | fungi: ya could be | 17:56 |
| *** priteau has quit IRC | 17:56 | |
| clarkb | fungi: though they did ask about builds logs, so maybe they just want nova things specifically? | 17:56 |
| fungi | clarkb: maybe. when i first read the message i thought they were asking about indexing nova instance boot logs in elasticsearch for a deployment of openstack | 17:57 |
| fungi | but i can see that without the [nova] in the subject i may have interpreted the rest differently | 17:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job https://review.openstack.org/510237 | 17:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add zuul-jobs to required-projects for tox-linters https://review.openstack.org/510971 | 17:57 |
| mordred | AJaeger: how about I make it its own commit for even better clarity ^^ :) | 17:58 |
| AJaeger | mordred: perfect ;) | 17:58 |
| AJaeger | mordred: why is zuul now unhappy? ;( | 17:59 |
| *** baoli has quit IRC | 17:59 | |
| *** sree has quit IRC | 17:59 | |
| *** Apoorva_ has joined #openstack-infra | 18:00 | |
| *** sree has joined #openstack-infra | 18:00 | |
| *** baoli has joined #openstack-infra | 18:00 | |
| *** ram5391 has quit IRC | 18:01 | |
| *** bhavik1 has quit IRC | 18:03 | |
| *** Apoorva has quit IRC | 18:03 | |
| *** tosky has joined #openstack-infra | 18:04 | |
| *** sree has quit IRC | 18:05 | |
| *** trown|lunch is now known as trown | 18:05 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures https://review.openstack.org/510952 | 18:05 |
| mordred | AJaeger: well - because? | 18:06 |
| mordred | AJaeger: oh! I forgot another depends-on there | 18:07 |
| *** persia has joined #openstack-infra | 18:07 | |
| pkovar | AJaeger: thingee approved both changes | 18:07 |
| mordred | AJaeger: https://review.openstack.org/#/c/502185/ | 18:07 |
| mordred | although that commit message is pretty bad | 18:08 |
| *** ijw has quit IRC | 18:08 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job https://review.openstack.org/510237 | 18:08 |
| *** jtomasek has quit IRC | 18:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add release-openstack-javascript job https://review.openstack.org/502185 | 18:10 |
| mordred | better commit message | 18:10 |
| AJaeger | mordred, fungi, two quick reviews on removing unused jobs if you have time, please: https://review.openstack.org/510765 and https://review.openstack.org/510683 | 18:11 |
| *** pkovar has quit IRC | 18:15 | |
| *** coolsvap has quit IRC | 18:15 | |
| thingee | AJaeger https://review.openstack.org/#/c/510979/ | 18:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add comment explaining gitpython requirement https://review.openstack.org/510980 | 18:15 |
| thingee | AJaeger: that wiki link is mostly a placeholder right now | 18:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures https://review.openstack.org/510952 | 18:16 |
| *** Swami has quit IRC | 18:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: base: ignore_errors for logstash processing https://review.openstack.org/510982 | 18:17 |
| pabelanger | clarkb: fungi: mordred: jeblair: promote up our logstash ignore_errors patch to base^. The commit message has results of testing | 18:18 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config master: ec2-api: move OOO job to experimental https://review.openstack.org/510983 | 18:18 |
| andreaf | dmsimard, jeblair, mordred: I managed to make the stage-artifacts role accept a list of files and folders, like in https://review.openstack.org/#/c/509272/8/playbooks/post-tempest.yaml | 18:18 |
| pabelanger | clarkb: also, I am likely to miss most of our infra meeting today. Have to run an errand wit my family. | 18:19 |
| *** Apoorva_ has quit IRC | 18:19 | |
| *** Apoorva has joined #openstack-infra | 18:20 | |
| AJaeger | thingee: thanks. Will review tomorrow, have to call it a day now... | 18:21 |
| jeblair | pabelanger, clarkb, mordred, fungi: i think i want to hold off on 510982 in favor of running with mordred's change for a bit so we can try to catch the error(s) | 18:21 |
| clarkb | pabelanger: ok, I think its largely just going to be catch up on zuulv3 things. If there is nayhting pressing on your side for that you might want to write it down | 18:21 |
| pabelanger | jeblair: wfm | 18:21 |
| pabelanger | I'll mark WIP for now | 18:21 |
| *** electrofelix has quit IRC | 18:21 | |
| jeblair | pabelanger: cool. thanks for getting it ready -- if we don't see/fix the error before we cutover, we should land that. maybe we should land it at cutover anyway. :) | 18:22 |
| *** rbrndt has joined #openstack-infra | 18:22 | |
| mordred | jeblair, pabelanger: is our current working theory that it was a logstash submission issue? | 18:23 |
| *** baoli has quit IRC | 18:23 | |
| mordred | jeblair: yah - I think landing it at or around cutover anyway is maybe not a bad idea | 18:23 |
| fungi | it's a reasonable theory since it's a post_failure which takes place after logs are successfully uploaded, and logstash/subunit2sql tasks are the only other post playbooks involved at that stage | 18:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures https://review.openstack.org/510952 | 18:24 |
| jeblair | yeah, it seems like it's either that, or failure uploading some particular file later, or... something we haven't imagined. in that order of probability. :) | 18:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures https://review.openstack.org/510952 | 18:25 |
| jeblair | i'm going to grab lunch; should be back by meeting; can do full zuul restart afterwords | 18:28 |
| thingee | #thanks AJaeger for keeping the Contributor Guide progress going on infra and doc team sides. | 18:29 |
| openstackstatus | thingee: Added thanks to Thanks page | 18:29 |
| pabelanger | mordred: ya, I think we get some error after upload-logs role, which are not exposed to us currently | 18:32 |
| pabelanger | but, just a guess | 18:32 |
| dansmith | I have a job that has been stuck in gate for a long time now, all jobs are finished, one failed, but it won't kick out so I can recheck it (it failed git clone) | 18:33 |
| dansmith | besides re-pushing a large series to kick it, can anything be done? | 18:34 |
| dansmith | it's 506416,18 | 18:34 |
| pabelanger | dansmith: abandon / restore works too | 18:34 |
| *** baoli has joined #openstack-infra | 18:34 | |
| pabelanger | looking | 18:34 |
| dansmith | pabelanger: will that affect the patches above it in the series/ | 18:34 |
| pabelanger | dansmith: it would dequeue them too | 18:35 |
| dansmith | pabelanger: they're not in the gate at the moment, I only care about not having to rebase them | 18:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add git timeout https://review.openstack.org/509517 | 18:35 |
| pabelanger | dansmith: ya, I don't think you'll need to rebase | 18:36 |
| dansmith | pabelanger: okay, so is it waiting for the things in front of it to finish before being kicked out? | 18:36 |
| dansmith | I thought since the ones in front passed the test it failed it would kick out immediately | 18:37 |
| pabelanger | dansmith: ya, it won't get kicked out until 500384 passed. If 500384 fails, it will be rebased and all tests run again | 18:38 |
| dansmith | okay I thought that was only if the job it failed failed in front of it | 18:38 |
| dansmith | but okay | 18:38 |
| dansmith | since there are some ahead of it with jobs still queued I guess I'll kick it | 18:38 |
| fungi | dansmith: it's assuming that any changes ahead of it in the shared queue might be causing the failure, and would retest it without if one of those subsequently fails | 18:38 |
| *** esberglu has quit IRC | 18:39 | |
| pabelanger | yah, best you could hope for now is a patch because you to fail, to cause your test to be re-run | 18:39 |
| fungi | dansmith: which is the reason why it's kept around in a dependent pipeline like the gate (rather than reporting back immediately and dequeuing) | 18:39 |
| dansmith | fungi: yeah, I just thought that was done per-job only.. doesn't make much sense for nova unit tests, but that's obscure | 18:39 |
| *** esberglu has joined #openstack-infra | 18:39 | |
| dansmith | abandon/restore seemed to do the trick, thanks | 18:40 |
| fungi | we have the chance to make zuul v3 much smarter about such things. v2 didn't really know about which jobs used what other repos | 18:40 |
| fungi | since zuul v2 was only involved in the scheduling side and not the job definitions | 18:41 |
| dansmith | cool | 18:41 |
| pabelanger | https://review.openstack.org/95035/ is a long run request to add dequeue to zuul client for admins to be able to run it | 18:41 |
| pabelanger | I need to follow up on it and address the comments | 18:42 |
| *** baoli has quit IRC | 18:43 | |
| *** eroux has joined #openstack-infra | 18:43 | |
| *** esberglu has quit IRC | 18:44 | |
| mordred | pabelanger: wow. that's a 5-digit patch | 18:47 |
| pabelanger | :) | 18:47 |
| pabelanger | mordred: maybe we can get SpamapS to push on it again :) | 18:48 |
| SpamapS | haha | 18:49 |
| SpamapS | no | 18:49 |
| *** boden has joined #openstack-infra | 18:49 | |
| SpamapS | ;) | 18:49 |
| SpamapS | maybe I guess | 18:49 |
| *** esberglu has joined #openstack-infra | 18:50 | |
| clarkb | infra meeting is in ~10 minutes in #openstack-meeting | 18:50 |
| boden | howdy, zuul v3 question if someone gets a chance… is there a reason I wouldn’t have any recent zuul v3 results for the vmware-nsx project? https://review.openstack.org/#/q/project:openstack/vmware-nsx+label:Verified%252Czuul I was hoping to get a warm and fuzzy for tomorrow switch | 18:51 |
| mnaser | boden the queue is quite backed up because zuulv3 doesnt have a lot of resources at the moment | 18:51 |
| mnaser | boden and zuulv3 is undergone a few restarts as fixes are being tested so its likely that it was lost as well | 18:52 |
| boden | mnaser: ack, well I guess I’ll have to swap that warm and fuzzy for a finger cross :) | 18:52 |
| boden | how far is it backed up, do you know? | 18:52 |
| clarkb | it was more than 30 hours last I looked | 18:52 |
| boden | oh ok… well it’ll be difficult for me to ensure for tomorrows switch I guess.. we’ll have to reach out if something crops up real-time | 18:53 |
| *** cody-somerville has quit IRC | 18:53 | |
| mnaser | boden a lot of people will be available to help, also i remember reading zuulv3 might be getting restarted shortly | 18:53 |
| boden | BTW: thanks for all the work on this too.. I’m sure it’s been quite an effort | 18:53 |
| mnaser | so you can slip a recheck quickly after teh restart, aha | 18:53 |
| boden | mnaser: ack, and thanks | 18:53 |
| *** cody-somerville has joined #openstack-infra | 18:55 | |
| *** rwsu has joined #openstack-infra | 19:00 | |
| *** andreas_s has joined #openstack-infra | 19:05 | |
| *** slaweq has joined #openstack-infra | 19:06 | |
| *** baoli has joined #openstack-infra | 19:09 | |
| *** boden has left #openstack-infra | 19:11 | |
| *** jkilpatr_ has quit IRC | 19:12 | |
| *** armax has quit IRC | 19:14 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 19:16 |
| *** rcernin has joined #openstack-infra | 19:16 | |
| *** andreas_s has quit IRC | 19:18 | |
| *** eharney has quit IRC | 19:26 | |
| *** jkilpatr has joined #openstack-infra | 19:27 | |
| *** rbrndt has quit IRC | 19:29 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove integration test playbooks from zuul-jobs https://review.openstack.org/510868 | 19:30 |
| *** rbrndt has joined #openstack-infra | 19:30 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy announce-release jobs https://review.openstack.org/510683 | 19:31 |
| *** andreas_s has joined #openstack-infra | 19:32 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Grab json log contents for final post playbook failures https://review.openstack.org/510952 | 19:35 |
| *** andreas_s has quit IRC | 19:36 | |
| *** andreas_s has joined #openstack-infra | 19:37 | |
| *** eharney has joined #openstack-infra | 19:41 | |
| *** pkovar has joined #openstack-infra | 19:41 | |
| *** thorst has quit IRC | 19:46 | |
| ianw | hmm, are we really going to have to add sbin to PATH like https://review.openstack.org/#/c/504743/11/roles/multi-node-bridge/tasks/main.yaml everywhere? that seems something we should do globally | 19:46 |
| *** thorst has joined #openstack-infra | 19:51 | |
| *** rkukura_ has joined #openstack-infra | 19:53 | |
| *** andreas_s has quit IRC | 19:55 | |
| *** pkovar has quit IRC | 19:55 | |
| *** thorst has quit IRC | 19:56 | |
| *** rkukura has quit IRC | 19:56 | |
| *** rkukura_ is now known as rkukura | 19:56 | |
| *** e0ne has joined #openstack-infra | 20:00 | |
| *** eroux has quit IRC | 20:00 | |
| clarkb | thingee: before I approve https://review.openstack.org/#/c/509943/7 is there an existing repo we need to be importing? | 20:01 |
| clarkb | thingee: it is much easier to do that at project creation time than it is at a later time | 20:01 |
| *** rbrndt has quit IRC | 20:02 | |
| *** rbrndt has joined #openstack-infra | 20:02 | |
| *** ijw has joined #openstack-infra | 20:06 | |
| ianw | clarkb: do you remember if https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n38 was really just "has a routable ipv6 interface"? | 20:06 |
| ianw | cause ansible has a fact for that, that i think avoids a bit of hassle | 20:06 |
| ianw | but also not sure it wasn't working around some bizzare "looks routable but isn't" type issue | 20:07 |
| clarkb | ianw: iirc its just that. Because some clouds (rax vexxhost osic) have proper routable ipv6 and others don't (the rest) | 20:07 |
| clarkb | so as long as the scope isn't local we are fine | 20:07 |
| *** eroux has joined #openstack-infra | 20:08 | |
| *** thorst has joined #openstack-infra | 20:08 | |
| ianw | yep, ansible only puts routable addresses in there, so that's a nice way around that | 20:08 |
| pabelanger | and back | 20:08 |
| *** andreas_s has joined #openstack-infra | 20:09 | |
| *** andreas_s has quit IRC | 20:13 | |
| thingee | clarkb: there is no existing repo. | 20:13 |
| clarkb | thingee: thank you for confirming, I'll approve the change in a moment | 20:13 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Mask dependent config errors https://review.openstack.org/510968 | 20:13 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add comment explaining gitpython requirement https://review.openstack.org/510980 | 20:13 |
| thingee | clarkb: aside from my local one | 20:14 |
| thingee | can that be pushed up first? | 20:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add an origin remote for tag-releases for notes https://review.openstack.org/511004 | 20:14 |
| clarkb | thingee: if you don't start with a seed repo during project creation you'll have to push everything but the first commit we add with a .gitreview thorugh code review | 20:15 |
| clarkb | thingee: the alternative is to host a git repo somewhere else with any existing content and we will import that in at project creation time in gerrit | 20:15 |
| thingee | clarkb: I only have the cookie cutter and another commit on top of that. not worried about losing history | 20:16 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Google Cal sync error https://review.openstack.org/511011 | 20:16 |
| clarkb | thingee: ok, so you are fine with the change as is and will just push content up when gerrit project exists? | 20:16 |
| SamYaple | thingee: with all the projects ive added, i always go with empty repos personally | 20:16 |
| thingee | clarkb: yes push is fine. | 20:17 |
| thingee | SamYaple: thanks | 20:17 |
| thingee | clarkb: eventually this will need to publish | 20:17 |
| thingee | I can do a follow up commit tho | 20:17 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Google Cal sync error https://review.openstack.org/511011 | 20:18 |
| clarkb | thingee: ok it is on its way | 20:18 |
| SamYaple | i typically do voting gates too. that way the first commit must pass the gate. but thats just me | 20:18 |
| thingee | SamYaple: there was some issue with a translation job that was part of the docs build job. We decided to go a noop approach since we're kind of in a hurry, because we're always in a hurry :) | 20:19 |
| *** e0ne has quit IRC | 20:20 | |
| SamYaple | yea thats compeltely understandable | 20:20 |
| inc0 | soo..question, any chances you can give some time between now and full zuulv3 rollout without legacy jobs? | 20:22 |
| inc0 | right now it's impossible to work on zuulv3 gates because it's overloaded | 20:22 |
| inc0 | and if you roll in it fully, our gates will be broken again until we fix it | 20:22 |
| clarkb | mordred: jeblair pabelanger has http://logs.openstack.org/23/475123/10/check/legacy-tempest-dsvm-neutron-multinode-full/4f8b030/job-output.txt.gz#_2017-10-09_12_02_20_776368 been debugging and/or fixed? | 20:23 |
| *** slaweq has quit IRC | 20:23 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 20:23 |
| inc0 | we're close but we could use at least some time (few days at most) when we can work on v3 gates without impacting regular gates | 20:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add statsd to zuul.conf https://review.openstack.org/511018 | 20:23 |
| *** e0ne has joined #openstack-infra | 20:23 | |
| inc0 | I think other projects could use window like that as well | 20:23 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Error on Admin action calendar sync https://review.openstack.org/511019 | 20:24 |
| * clarkb looks t odebug that problem | 20:24 | |
| pabelanger | clarkb: I am not sure, but I can look | 20:24 |
| *** e0ne has quit IRC | 20:25 | |
| clarkb | pabelanger: looks like frickler may have fixed it I8e7240f28e1aa991c637b8aee421a2f5b4eba9e9 | 20:25 |
| inc0 | thing is, if we won't have it, that puts us in code freeze until we figure out zuulv3 gates, I hope I don't need to tell you that it's not great :/ | 20:25 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Error on Admin action calendar sync https://review.openstack.org/511019 | 20:25 |
| clarkb | inc0: I'm not sur ehow that would logistically function/work | 20:26 |
| clarkb | inc0: we are currently down a cloud and alreayd splitting resources and unable to keep up | 20:26 |
| mordred | inc0, clarkb: there's nothing stopping landing a removal of your legacy changes from v3 right now | 20:26 |
| inc0 | mordred: we removed it, but our new one is not ready yet | 20:27 |
| mordred | the main issue there is that if they're removed then you'll have a perioud of time post rollout with none of those jobs in your gate | 20:27 |
| inc0 | it's virtually impossible to work on zuulv3 right now | 20:27 |
| pabelanger | clarkb: cool | 20:27 |
| mordred | inc0: oh - so you already removed your legacy jobs? I may be misunderstanding the ask ... reading again | 20:27 |
| inc0 | well also when we had legacy v3 jobs they weren't working for us | 20:28 |
| inc0 | yeah, we removed it (they were broken anyway) | 20:28 |
| SamYaple | mordred: he is trying to avoid a code freeze, but cant make sure teh v3 gates are working because v3 is slow | 20:28 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 20:28 |
| SamYaple | so v3 is going to land with non-funcational kolla gates and force a code freeze | 20:28 |
| clarkb | and things are slow basiclly because ENOTENOUGHCOMPUTE | 20:28 |
| mordred | gotcha. I now understand | 20:28 |
| pabelanger | Yah, I think it might have been premature to remove legacy jobs, even thought that's what we want long term | 20:28 |
| inc0 | legacy didn't work | 20:29 |
| clarkb | as fungi mentioned earlier today, you could stop pushing code to address that... but otherwise we are down a cloud and so on | 20:29 |
| mordred | well - the legacy jobs for kolla also didn't work - so time spent fixing the new jobs is time better spent | 20:29 |
| jeblair | well, they might work at this point | 20:29 |
| pabelanger | by tomorrow, you should be in a better spot | 20:29 |
| inc0 | well no | 20:29 |
| jeblair | part of why we rolled back was because we missed things in the legacy conversions | 20:29 |
| inc0 | because tomorrow you roll in all zullv3 | 20:29 |
| pabelanger | what was the issue with legacy? | 20:29 |
| openstackgerrit | Merged openstack-infra/project-config master: Adding the Contributor Guide Project https://review.openstack.org/509943 | 20:30 |
| inc0 | lots of small issues that we're partially resolving with our zuulv3 work | 20:30 |
| inc0 | different dirs and whatnot, don't forget Kolla is not your typical python service | 20:30 |
| inc0 | we did quite a bit of OS-level stuff in our gates to make it work | 20:30 |
| SamYaple | yea inc0 the only reason i got my v3 gates done is because i did them on the weekends when all the queues were empty | 20:30 |
| jeblair | yes, but the work we did on legacy jobs was around mimicing v2, so if we got that right, they should work despite their differences. | 20:31 |
| mordred | inc0: I think logistically we just don't have the nodebandwidth to do it - BUT - with v3 working properly iterating on fixes to your v3 jobs should go much quicker - and with us not needing to put out as many fires we should be able to help | 20:31 |
| inc0 | and that's ok, zuulv3 makes it whole lot easier, but we could really use time window to make it work | 20:31 |
| *** andreas_s has joined #openstack-infra | 20:31 | |
| inc0 | so what I'm asking is to remove legacy jobs from current split | 20:31 |
| inc0 | and give us few days like that | 20:31 |
| SamYaple | inc0: what patchset are you working with right now for the v3 gates? | 20:31 |
| clarkb | we've also intentionally chosen this time of the release cycle knowing that things may/will be affected | 20:32 |
| inc0 | https://review.openstack.org/#/c/508759/ this is for ansible | 20:32 |
| inc0 | kolla proper build already is green and deploy needs ^ | 20:32 |
| pabelanger | agree with jeblair, legacy jobs should have been working or pretty much close to working. But admit, I never seen a log to debug why. It is good you moved jobs to zuulv3 however | 20:32 |
| inc0 | well I can't remember exactly what was wrong, but it didn't work. | 20:32 |
| *** openstackgerrit has quit IRC | 20:33 | |
| inc0 | that's why I didn't pause before removing legacy jobs and going all v3 | 20:33 |
| mordred | inc0: in any case, I hear your concern, but at this point I think we're just gonna have to dive in tomorrow and help you fix them - I've gotten pretty good at fixing jobs quickly ... | 20:33 |
| pabelanger | agree | 20:33 |
| mnaser | for puppet, i personally just refactored small bits and left the main bash shell still doing the job (called by ansible) | 20:33 |
| mnaser | that's the best way to fix things quickly | 20:33 |
| inc0 | ok, let's do that then | 20:34 |
| mnaser | i think you're taking too big of a bite if you want to rewrite them all with ansible + roles etc | 20:34 |
| inc0 | we're pretty close I think | 20:34 |
| SamYaple | inc0: i can get you a well tested gate with docker installed and working to replace a large section of your work with ansible native code. can i add to Jeffrey4ls patch? | 20:34 |
| tosky | mnaser: that's what I'd like to do as well (rewrite directly) | 20:34 |
| mnaser | https://review.openstack.org/#/c/508296/ | 20:35 |
| thingee | SamYaple: oh and I can it to my .zuul.yml locally. I don't see if it says that gets used over what's in project config https://docs.openstack.org/infra/manual/creators.html#adding-in-repo-zuul-jobs | 20:35 |
| thingee | can add it* | 20:35 |
| inc0 | SamYaple: sure, feel free, although lots of that (for k-ansible at least) is already done by bootstrap servers | 20:35 |
| mnaser | you'll notice the playbooks are all calling the main bash script | 20:35 |
| inc0 | right now we have ansible calling command: ansible-playbook ...;) | 20:35 |
| jeblair | clarkb, fungi, mordred: should i install zuulv3 branch tip with or without my repl backdoor? (in case we have craziness that needs on-line debugging; or should we trust what we've done so far) | 20:35 |
| inc0 | so at some point we'll want it all to be one large ansible run | 20:36 |
| SamYaple | thingee: you can parent jobs from project-config, but i dont think you can *override* jobs | 20:36 |
| *** sree has joined #openstack-infra | 20:36 | |
| clarkb | jeblair: probably best to keep the backdoor for now and remove it once we've proven the install? | 20:36 |
| mordred | jeblair: I say without - let's use the next 18 hours or so to prove out that we're good | 20:36 |
| mordred | ha | 20:36 |
| mordred | clarkb and I took opposite sides :) | 20:36 |
| jeblair | need tiebreaker! | 20:36 |
| SamYaple | inc0: right, i can get you a full gate setup with docker and ansible, then you can just shell: ansible-playbook to get you most of the way there | 20:36 |
| tosky | that said - does anyone have a clue on this "permission denied" error? http://logs.openstack.org/79/510579/2/check/legacy-sahara-dsvm-scenario-neutron/7874161/job-output.txt.gz#_2017-10-10_17_00_50_202414 | 20:36 |
| inc0 | SamYaple: let's deal with it later, kolla proper gates seems to be fine | 20:37 |
| inc0 | https://review.openstack.org/#/c/508661/ | 20:37 |
| mnaser | tosky seems weird | 20:38 |
| tosky | and also this error, which looks like my job should require tripleo-ci.git (!) http://logs.openstack.org/31/510631/1/check/legacy-sahara-cli/6d2ff42/logs/devstack-gate-setup-workspace-new.txt | 20:38 |
| inc0 | after we're in state when our gates works we'll slowly get rid of shell in favor of ansible | 20:38 |
| clarkb | tosky: first thing that comes to mind is the pip + sudo behavior where sudo pip install will create wheels in the cache of the homedir of the user but sudo doesn't change the homedir | 20:38 |
| clarkb | tosky: so you end up with files owned by root then later try to edit them as regular user and it breaks | 20:38 |
| mnaser | ^ that's what i was guessing as well | 20:38 |
| pabelanger | tosky: there is a lot of ERROR logging in that review too | 20:38 |
| thingee | SamYaple: ah ok good to know | 20:39 |
| pabelanger | http://logs.openstack.org/79/510579/2/check/legacy-sahara-dsvm-scenario-neutron/7874161/job-output.txt.gz#_2017-10-10_17_00_49_966165 | 20:39 |
| inc0 | thingee: btw, new Blade Runner - totally awesome | 20:39 |
| mnaser | and tosky correct, i believe it's missing in required-projects | 20:39 |
| SamYaple | thingee: all the jobs live in a global namespace. so you can't same name jobs | 20:39 |
| tosky | pabelanger: uhm, I see - I think I know what to fix there | 20:39 |
| tosky | but for the other job, the requirement to tripleo-ci, I'm not sure if it's for sure, if it's a mistake, or it's a red herring | 20:40 |
| *** sree has quit IRC | 20:40 | |
| pabelanger | tosky: http://logs.openstack.org/79/510579/2/check/legacy-sahara-dsvm-scenario-neutron/7874161/job-output.txt.gz#_2017-10-10_17_00_33_213117 | 20:40 |
| pabelanger | chown zuul:stack | 20:41 |
| pabelanger | no longer using jenkins user | 20:41 |
| tosky | pabelanger: right, thanks, that should be easy | 20:41 |
| *** openstackgerrit has joined #openstack-infra | 20:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Specify duplicate project defn behavior https://review.openstack.org/511025 | 20:41 |
| pabelanger | tosky: yay | 20:41 |
| jeblair | thingee: ^ i found that wasn't even mentioned in the zuul docs :) | 20:41 |
| *** dprince has quit IRC | 20:42 | |
| jeblair | i'm going to perform a full shutdown of zuulv3 now, then repartition | 20:44 |
| pabelanger | wfm | 20:45 |
| *** trown is now known as trown|outtypewww | 20:48 | |
| *** andreas_s has quit IRC | 20:54 | |
| thingee | clarkb: can you please add me to the openstack-contributor-guide-core group https://review.openstack.org/#/admin/projects/openstack/contributor-guide,access | 20:55 |
| SamYaple | inc0: if you recheck quickly you can get the first zuulv3 patch going! | 20:56 |
| jeblair | not yet; but i'll let you know when i restart it | 20:56 |
| * SamYaple points out this is how he got most of his jobs fixed | 20:57 | |
| *** wolverineav has quit IRC | 20:59 | |
| *** srobert_ has joined #openstack-infra | 20:59 | |
| tosky | mnaser: I think that should be the fix for the tripleoci thing; I didn't notice https://review.openstack.org/511037 | 20:59 |
| clarkb | thingee: I've added pkovar as well since its technically a docs project | 21:00 |
| jeblair | i've restarted nodepool (zuulv3 is still offline) | 21:00 |
| jeblair | (rather, the v3 nodepool launchers) | 21:00 |
| tosky | uh, no notifications for changes to openstack-zuul-jobs here? Only on #zuul maybe? | 21:01 |
| jeblair | tosky: should be here | 21:01 |
| *** srobert has quit IRC | 21:01 | |
| tosky | oh, ok, zuul v3 is being restarted, maybe that's the reason | 21:02 |
| jeblair | tosky: the bot may have missed it or had an issue with the irc server | 21:02 |
| thingee | clarkb: thanks. I just added openstack-doc-core as training-guides and other sub teams are setup :) | 21:03 |
| *** srobert_ has quit IRC | 21:03 | |
| *** gouthamr has quit IRC | 21:06 | |
| *** mat128 has quit IRC | 21:06 | |
| jeblair | Downloading GitPython-2.1.7-py2.py3-none-any.whl (446kB) | 21:06 |
| *** mat128 has joined #openstack-infra | 21:07 | |
| jeblair | clarkb, mordred, fungi: ^ that did not install the fork | 21:07 |
| jeblair | that was with "sudo pip3 install -U ." in a zuul checkout | 21:07 |
| * clarkb looks at the requirements file again | 21:07 | |
| jeblair | maybe it's cached somewhere? | 21:07 |
| *** andreas_s has joined #openstack-infra | 21:08 | |
| clarkb | ya I wonder if #egg=GitPython means it will grab something out of cache? | 21:08 |
| *** jamesdenton has quit IRC | 21:08 | |
| clarkb | or that it can? | 21:08 |
| *** Apoorva_ has joined #openstack-infra | 21:09 | |
| *** jcoufal has quit IRC | 21:10 | |
| *** eharney has quit IRC | 21:11 | |
| *** mat128 has quit IRC | 21:11 | |
| *** andreas_s has quit IRC | 21:12 | |
| *** Apoorva has quit IRC | 21:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 21:12 |
| jeblair | okay i just did "sudo pip3 -r requirements.txt" which seemed to work | 21:13 |
| jeblair | black magic | 21:13 |
| *** esberglu has quit IRC | 21:14 | |
| clarkb | jeblair: I reproduced the initial behavior you ran into locally | 21:14 |
| clarkb | is it possible this is a pbr behavior? | 21:14 |
| clarkb | mordred: ^ | 21:14 |
| *** esberglu has joined #openstack-infra | 21:14 | |
| *** claudiub|2 has joined #openstack-infra | 21:15 | |
| clarkb | jeblair: reading pbr source I think it likely that pbr is at fault | 21:16 |
| clarkb | For the requirements list, we need to inject only the portion after egg= so that distutils knows the package it's looking for <- from pbr comments in source | 21:16 |
| clarkb | I think because setuptools doesn't grok those lines only pip does | 21:16 |
| clarkb | so you have to install requirements with pip | 21:16 |
| fungi | okay, back and catching up. looks like i missed a bunch of discussion in the past hour-ish | 21:17 |
| jeblair | when i followed that up with another 'pip3 install -U .' it installed the wheel again | 21:17 |
| jeblair | i think it's likely that both this and the github3.py thing aren't actually working for us in prod | 21:17 |
| dtroyer | I am seeing an error in the legacy OSC functional jobs with a 'cp -dRl' command that appears to come from zuul-cloner-shim.py: http://logs.openstack.org/89/359489/3/check/legacy-osc-dsvm-functional-tips/0f9592d/job-output.txt.gz#_2017-10-09_20_51_19_673844 | 21:18 |
| dtroyer | looks like cross-filesystem link failure: cp: cannot create hard link | 21:18 |
| *** rcernin has quit IRC | 21:19 | |
| jeblair | zuulv3 has restarted | 21:19 |
| jeblair | inc0: ^ | 21:19 |
| ianw | dtroyer: hmm, i wondered if that was ever going to go across fs's | 21:19 |
| ianw | /tmp/python-openstackclient-tox_install-1NfuS51 ... so it's a tmpfs? | 21:19 |
| *** armaan has joined #openstack-infra | 21:20 | |
| dtroyer | that's my guess | 21:20 |
| dtroyer | there appears to be a check for that in zuul-cloner-shim.py | 21:20 |
| clarkb | ianw: dtroyer I wrote a fix for that | 21:20 |
| jeblair | yeah, it landed a week ago :/ | 21:20 |
| clarkb | which I thought was working for tripleo at least | 21:20 |
| mordred | clarkb: looking ... | 21:21 |
| dtroyer | this run was last night | 21:21 |
| dtroyer | something I need to add to our job? | 21:21 |
| *** thorst has quit IRC | 21:21 | |
| *** armaan has quit IRC | 21:21 | |
| *** ldnunes has quit IRC | 21:21 | |
| *** armaan has joined #openstack-infra | 21:22 | |
| fungi | jeblair: clarkb: mordred: if we restart now without the repl support and then discover we need it, that's yet another restart possibly in the midst of people trying to do rushed migration work... whereas if we start with it then we can kick the restart down the road to when we're comfortable it won't be needed? | 21:24 |
| clarkb | dtroyer: no it should be used already, its possible my fix is buggy or not able to address all situations | 21:24 |
| *** jgriffith has joined #openstack-infra | 21:24 | |
| clarkb | I just added an issue to the issues with job's list for broken d-g/devstack caching | 21:24 |
| pabelanger | mordred: jeblair: ze01 doesn't look healthy. I see 3 zuul-executor processes again, 4 if you count defunct | 21:24 |
| clarkb | I'll add dtroyers hardlink thing to the list too | 21:25 |
| jeblair | fungi, clarkb, mordred: i ran with the repl | 21:25 |
| pabelanger | ze02 also | 21:25 |
| * mordred supports our repl overlords | 21:25 | |
| jeblair | pabelanger: hrm, i'll try that again | 21:26 |
| *** Apoorva_ has quit IRC | 21:26 | |
| pabelanger | and ze04 | 21:26 |
| *** anupn has joined #openstack-infra | 21:26 | |
| *** Apoorva has joined #openstack-infra | 21:26 | |
| *** yamahata has joined #openstack-infra | 21:27 | |
| pabelanger | ze08 seems to have 1 zuul-executor process, other defunt | 21:27 |
| jeblair | pabelanger: yeah, i'm restarting them now | 21:27 |
| anupn | Hi all, I am seeing "no module named keystonemiddleware" inside my gate although i see keystone-auth1 and python-keystoneclient has been installed | 21:27 |
| pabelanger | defunct* | 21:27 |
| pabelanger | kk | 21:27 |
| anupn | here is the log http://logs.openstack.org/52/509652/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-src-ubuntu-xenial/6fbca9c/logs/screen-s-proxy.txt.gz#_Oct_10_18_55_29_493011 | 21:28 |
| pabelanger | jeblair: I think ze08 has a hung git process. | 21:28 |
| pabelanger | maybe that is why we get into this state | 21:28 |
| anupn | The patch i am working is for testing py3 inside ironic-python-agent but I see unrelated error of no module named keystonemiddleware, can someone point out what might be going wrong | 21:29 |
| fungi | pabelanger: yeah, my expectation was that the patches to time out hung git operations would address the wedged executors | 21:29 |
| clarkb | ianw: dtroyer on ubuntu I didn't think /tmp was a tmpfs | 21:29 |
| *** dhinesh has quit IRC | 21:29 | |
| clarkb | ianw: dtroyer reading my code I suppose there is a race in it if /proc/mounts changes during the running of z-c | 21:30 |
| clarkb | since I only read the mounts list the one time | 21:30 |
| pabelanger | fungi: I'm not sure ze08 has the lastest patch for git timeout, but it looks like if we have hung git process, then stop. We don't properly shutdown a zuul-executor process. Then, if we start again, we might end up with 3 zuul-executor processes. | 21:30 |
| pabelanger | but, just speculation currently | 21:31 |
| fungi | pabelanger: right, i thought jeblair's current round of restarts was to also make sure we run the executors with the git timeout fixes | 21:31 |
| jeblair | okay, all executors re-restarted | 21:33 |
| *** edmondsw has quit IRC | 21:33 | |
| fungi | anupn: are you installing the keystonemiddleware package too? | 21:33 |
| clarkb | fungi: I'm looking at how to fix the devstack-gate caching situation and wondering if maybe a symlink from /home/zuul/cache/ to /home/jenkins/cache would work? I guess then it comes down to permissions and whether or not everything dealing with that cache is going to follow symlinks | 21:33 |
| anupn | Hi fungi, i see that installed here http://logs.openstack.org/52/509652/3/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-src-ubuntu-xenial/6fbca9c/logs/pip3-freeze.txt.gz | 21:34 |
| ianw | clarkb: didn't we fix that :/ | 21:34 |
| ianw | clarkb: https://review.openstack.org/#/c/509116/ is what i'm thinking of | 21:35 |
| *** edmondsw_ has joined #openstack-infra | 21:35 | |
| clarkb | ianw: that only affects the jenkins user, but reading that I think I know how to fix it for the zuul user now too | 21:35 |
| fungi | anupn: "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py" | 21:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 21:35 |
| fungi | anupn: note the "python2.7" there | 21:35 |
| fungi | anupn: if memory serves, swift-proxy doesn't support python3 yet | 21:35 |
| ianw | clarkb / dtroyer : yes ... how did the (presumed) tmpfs get in there? | 21:36 |
| anupn | fungi, oh yeah i was just checking inside pip2-freeze right now | 21:36 |
| anupn | fungi, and i dont see keystonemiddleware there | 21:36 |
| fungi | anupn: last time i looked, ay least, i think devstack was still running swift-proxy under python2.7 when you asked to use python3 | 21:36 |
| pabelanger | Just noticed requirements-check is failing: http://logs.openstack.org/94/484594/94/check/requirements-check/34c4822/job-output.txt.gz#_2017-10-10_21_26_19_003740 | 21:37 |
| pabelanger | anybody working on that? | 21:37 |
| * pabelanger checks etherpad | 21:37 | |
| anupn | fungi, so does that mean swift-proxy always run under python2.7 | 21:37 |
| jeblair | pabelanger: yes, currently line 60/61 | 21:38 |
| pabelanger | yay: https://review.openstack.org/510565 | 21:38 |
| clarkb | anupn: fungi ya swift doesn't support python3 | 21:38 |
| fungi | anupn: i think so? i mean, if they don't support python3 yet then it doesn't make a ton of sense to run it under python3 | 21:38 |
| fungi | so you need to make sure its dependencies are installed where python2.7 can find them | 21:38 |
| anupn | clarkb, fungi: ah | 21:38 |
| anupn | fungi, so this problem comes when it tries to run everything under python3 | 21:39 |
| anupn | and not just ironic | 21:39 |
| *** edmondsw_ has quit IRC | 21:39 | |
| fungi | anupn: well, it looks like whatever the source of the problem is, the result is that keystonemiddleware is not being installed globally for python2.7 such that swift-proxy can import it | 21:40 |
| *** andreas_s has joined #openstack-infra | 21:40 | |
| fungi | anupn: you may have better luck taking this up with the devstack devs in #openstack-qa | 21:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Put /home/zuul/cache in place on test node images https://review.openstack.org/511056 | 21:40 |
| clarkb | ianw: ^ | 21:40 |
| anupn | fungi, hmm yeah | 21:41 |
| pabelanger | jeblair: thanks, I've +2 mordred patch | 21:41 |
| jeblair | i have to run for a bit, bbl. | 21:43 |
| *** aeng has joined #openstack-infra | 21:44 | |
| pabelanger | clarkb: ianw: should we kick off new image builds for /home/zuul/cache fixes? Or just wait until next round of DIB builds | 21:46 |
| fungi | pabelanger: i'd kick off new builds asap | 21:47 |
| clarkb | ya having that in place would be nice | 21:47 |
| fungi | better than waiting until cut-over time tomorrow to find out that devstack jobs all broke | 21:47 |
| anupn | fungi, clarkb: I notice that gate failed when i set USE_PYTHON3=True and passed when set to lowercase true | 21:47 |
| anupn | https://review.openstack.org/#/c/509641/2/devstack-vm-gate-wrap.sh@60 | 21:47 |
| anupn | fungi, clarkb: the reason i think is because here https://github.com/openstack-dev/devstack/blob/master/inc/python#L588 | 21:48 |
| pabelanger | fungi: clarkb: sure, wanted to confirm | 21:48 |
| anupn | if it is set to True devstack tries to run all services with py3 | 21:48 |
| anupn | and hence as swift do not support py3 it is giving the prob | 21:48 |
| clarkb | anupn: I think we blacklist swift in other jobs that use python3 | 21:49 |
| clarkb | anupn: you can look at the generic python3 tempest job to see how that is done | 21:49 |
| tosky | I see that some variables have been replaced with something else (for example I was looking for ZUUL_BRANCH https://review.openstack.org/#/c/486246/) | 21:50 |
| anupn | clarkb, sorry do you have a pointer to that handy? | 21:50 |
| anupn | clarkb, not sure exactly where to check that | 21:50 |
| tosky | I'm not sure how it should replaced: just a variable that can be used in ansible conditions? Is there a table which says which variable became what? | 21:50 |
| pabelanger | tosky: yes, see https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html#var-zuul for more information | 21:50 |
| clarkb | anupn: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n748 | 21:51 |
| pabelanger | tosky: they will be written to the ansible inventory file | 21:51 |
| tosky | ah, there, thanks | 21:51 |
| anupn | clarkb, cool, thanks | 21:51 |
| tosky | pabelanger: I was looking only in https://docs.openstack.org/infra/manual/zuulv3.html | 21:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Put /home/zuul/cache in place on test node images https://review.openstack.org/511056 | 21:56 |
| *** ijw has quit IRC | 21:56 | |
| *** andreas_s has quit IRC | 21:57 | |
| *** threestrands has joined #openstack-infra | 21:58 | |
| *** camunoz has quit IRC | 22:01 | |
| *** andreas_s has joined #openstack-infra | 22:02 | |
| *** felipemonteiro_ has quit IRC | 22:04 | |
| fungi | tosky: the mapping is created here, if it helps you at all: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/filter/zuul_filters.py?h=feature/zuulv3 | 22:04 |
| fungi | that's a relatively human-readable translation between the two (hurray python?) | 22:05 |
| tosky | fungi: oooh, thanks | 22:05 |
| fungi | that's the compatibility shim we're calling zuul_legacy_vars in the legacy playbooks | 22:06 |
| *** dhinesh has joined #openstack-infra | 22:07 | |
| *** baoli has quit IRC | 22:08 | |
| *** mat128 has joined #openstack-infra | 22:08 | |
| *** edmondsw has joined #openstack-infra | 22:08 | |
| *** bobh has quit IRC | 22:11 | |
| tosky | ok, so another question: I have two job in sahara-extras which 1) created some artifacts 2) pushed them to tarballs.o.o - are those going to be the same job, with specialized run: and post: playbooks now? | 22:12 |
| tosky | or better, tasks | 22:12 |
| *** dhinesh_ has joined #openstack-infra | 22:12 | |
| *** edmondsw has quit IRC | 22:13 | |
| fungi | you can probably look at the native tarball jobs... lemme get you a link | 22:13 |
| pabelanger | Ya, publish-artifacts job would be a great example | 22:15 |
| *** dhinesh has quit IRC | 22:16 | |
| *** andreas_s has quit IRC | 22:16 | |
| *** kgiusti has left #openstack-infra | 22:17 | |
| *** dhajare has quit IRC | 22:18 | |
| fungi | tosky: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n337 | 22:18 |
| fungi | tosky: it's calling these: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/python-tarball | 22:19 |
| *** yamamoto has joined #openstack-infra | 22:19 | |
| tosky | thanks | 22:20 |
| *** andreas_s has joined #openstack-infra | 22:21 | |
| jeblair | pabelanger, fungi: can you review https://review.openstack.org/511018 when you get a chance? i'd like to have that in place before some more statsd work i'm doing lands | 22:21 |
| mordred | jeblair: lgtm | 22:21 |
| jeblair | (i'm eager to have a per-executor current jobs gauge) | 22:22 |
| fungi | tosky: and parented to this: http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/publish/openstack-artifacts.yaml | 22:23 |
| tosky | fungi: just found! | 22:24 |
| tosky | so yes, I would need to merge the two previously different jobs in one, and inherit from publish-openstack-artifacts | 22:24 |
| fungi | presumably | 22:25 |
| *** andreas_s has quit IRC | 22:25 | |
| fungi | tosky: i'm not sure what the difference was in your old jobs | 22:25 |
| mordred | fungi: we used to have build then publish - we've moved to having them combined in our current v3 jobs | 22:25 |
| pabelanger | jeblair: +3 | 22:25 |
| mordred | although I'm working on writing up a proposal to change that a little bit :) | 22:26 |
| fungi | for example if one was to test that tarball building works and the other was to build a tarball and upload it, then you could do like we have for the python-tarball jobs where we just omit the post playbook on one | 22:26 |
| SamYaple | would anyone want to review a bindep patch? https://review.openstack.org/#/c/506502/ | 22:26 |
| SamYaple | i hear reviewing Parsley things counts as a serving of vegatables | 22:26 |
| fungi | mordred: just so i'm not confused, you mean we used to have "build" and "build and publish" | 22:27 |
| fungi | ? | 22:27 |
| tosky | fungi: gate-sahara-extra-artifacts and the next post-sahara-extra-artifacts http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/sahara.yaml#n159 | 22:27 |
| fungi | tosky: okay, yep then that's what i'm talking about. the gate version just tested that you could build a tarball, the post version built and published a tarball | 22:28 |
| tosky | and if I understand it correctly, the gate-sahara-extra-artifacts will be the pre/run, while post-sahara-extra-artifacts will be the post | 22:28 |
| tosky | but for the same job | 22:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Revert "Revert "Add redirect from status.o.o/zuul to zuulv3.openstack.org"" https://review.openstack.org/511067 | 22:28 |
| mordred | tosky: not quite | 22:29 |
| tosky | oh | 22:29 |
| mordred | tosky: so ... sahara-extra-publish is basically a thing that does what the publish-openstack-artifacts base job does - it uploads artifacts to tarballs.o.o/{project-name} | 22:30 |
| *** chlong has quit IRC | 22:30 | |
| pabelanger | mordred: I should be online early for tomorrows rollout. For now, I'm preping some patches with topic:zuulv3-rollout | 22:30 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add zuul-jobs to required-projects for tox-linters https://review.openstack.org/510971 | 22:30 |
| pabelanger | getting patches for nodepool ready now | 22:30 |
| mordred | tosky: you'll want a job, for instance 'sahara-build-extra-artifacts' that runs your tools/build-all-artifacts.sh script | 22:31 |
| mordred | tosky: then you'll want a job, for instance 'sahara-publish-extra-artifacts' that has publish-openstack-artifacts as a base job, runs tools/build-all-artifacts.sh and copies them back to the executor into {{ zuul.executor.work_root }}/artifacts | 22:32 |
| mordred | that way you can put sahara-build-extra-artifacts into check and gate and sahara-publish-extra-artifacts into post | 22:33 |
| tosky | oh, I see, still two jobs | 22:33 |
| *** kjackal_ has quit IRC | 22:33 | |
| tosky | for some reason I thought that the post: action was invoked only when the job was in the post queue, but yeah, it's not that | 22:34 |
| mordred | yah - that's phases of the job itself - but the word post sure does show up both places | 22:35 |
| fungi | it's a bit overloaded | 22:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Use the shared openstack-tox-pylint jobs https://review.openstack.org/510636 | 22:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch TripleO scenario upgrade jobs to nv https://review.openstack.org/510638 | 22:35 |
| tosky | definitely moreclear now, thanks | 22:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Allow all zuul servers to send to statsd https://review.openstack.org/511068 | 22:35 |
| fungi | maybe we should rename our "post" pipeline in zuul to "merged" to disambiguate | 22:36 |
| jeblair | mordred, pabelanger, fungi: ^ one more prep thing | 22:36 |
| jeblair | or post-merge | 22:36 |
| fungi | after-merge? | 22:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs from kuryr-k8s https://review.openstack.org/510769 | 22:36 |
| mordred | jeblair: I +2'd - but in case you care there are tabs in that patch | 22:36 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add unbound role https://review.openstack.org/510725 | 22:36 |
| jeblair | apres-merge | 22:36 |
| SamYaple | not that one | 22:36 |
| mordred | fungi, jeblair: after-shve perhaps? | 22:36 |
| jeblair | i'm imagining all of these ideas on roadside signs | 22:37 |
| SamYaple | post-merge seems good | 22:37 |
| mordred | we could call it 'aqua-velva' and confuse everyone who is younger than 40 and not from the US | 22:37 |
| SamYaple | still has post in the name, explains what it is | 22:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Allow all zuul servers to send to statsd https://review.openstack.org/511068 | 22:37 |
| jeblair | mordred: i think i got all the tabs | 22:37 |
| fungi | jeblair: i approved it, but i think it may get kicked back for the mixing of tabs and spaces | 22:38 |
| fungi | 511068 i mean | 22:38 |
| fungi | oh, you fixed | 22:38 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Test unbound role https://review.openstack.org/510734 | 22:38 |
| * mordred has a feeling that somewhere deep in the bowels of something us changing th ename of the post pipeline would wreak crazy kinds of havoc in unexpected places | 22:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Revert "Shift nodepool quota from v2 to v3"" https://review.openstack.org/511069 | 22:38 |
| SamYaple | mordred: lets just change it on the zuul page lol | 22:39 |
| mordred | SamYaple: :) ... if only the zuul page wasn't data-driven | 22:39 |
| *** sdague has quit IRC | 22:39 | |
| SamYaple | indeed | 22:40 |
| mordred | pabelanger: revert revert patches are always my favorite | 22:40 |
| pabelanger | ++ | 22:40 |
| *** jgriffith has quit IRC | 22:40 | |
| pabelanger | I _think_ that is everything. But we'll know more in the morning | 22:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Switch TripleO scenario upgrade jobs to nv https://review.openstack.org/510637 | 22:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy jobs from Kuryr-libnetwork https://review.openstack.org/509986 | 22:41 |
| *** mat128 has quit IRC | 22:41 | |
| pabelanger | excited for the morning | 22:42 |
| pabelanger | like christmas all over again | 22:42 |
| mordred | \p/ | 22:42 |
| mordred | infra-root: I just sent an email to the infra list about interfaces for publishing things - basically the topic we were just discussing with tosky | 22:43 |
| *** rbrndt has quit IRC | 22:45 | |
| *** rbrndt has joined #openstack-infra | 22:45 | |
| *** rbrndt has quit IRC | 22:45 | |
| *** jgriffith_ has quit IRC | 22:46 | |
| mordred | it's not a thing we need to do before the rollout, so feel free to read or ignore as you will between now and tomorrow | 22:46 |
| *** armax has joined #openstack-infra | 22:48 | |
| *** andreas_s has joined #openstack-infra | 22:48 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/openstack-zuul-jobs master: Remove the legacy pylint jobs (replaced) https://review.openstack.org/511071 | 22:49 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Integration tests: run mirror-info before configure-mirrors https://review.openstack.org/510881 | 22:49 |
| ianw | has anyone thought of putting something at the top of the console log outputting the hostname/ip address it's running on. i used to just look at the ip in the link, now it's really hard to tell | 22:49 |
| pabelanger | we opted to output to add it into zuul-info, which gets upload after the job finished | 22:50 |
| pabelanger | makes it a little harder to get the info and SSH into a running job | 22:51 |
| *** vhosakot has quit IRC | 22:51 | |
| ianw | clarkb / jeblair: want to weigh in on the home for the unbound configuration role? i think otherwise it's ready -> https://review.openstack.org/#/c/510725/ | 22:52 |
| tosky | mordred: should I wait to convert my publisher job until that change proposal is discussed and approved? | 22:52 |
| jeblair | ianw, clarkb, mordred: i think i lean toward openstack-zuul-jobs because it depends on some pretty openstacky things we're doing in base images. | 22:54 |
| ianw | jeblair: ok, i'll move it. at first i thought i might port all the package install etc into it ... but it seems better to do in the image. there's no point having all the setup of the dns cacher fail due to dns issues :) | 22:55 |
| *** aeng has quit IRC | 23:01 | |
| *** sree has joined #openstack-infra | 23:01 | |
| *** andreas_s has quit IRC | 23:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 23:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 23:01 |
| *** rhallisey has quit IRC | 23:02 | |
| clarkb | jeblair: ianw ya I think even assuming unbound is present is fairly openstacky | 23:03 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add unbound role https://review.openstack.org/510734 | 23:05 |
| ianw | clarkb / jeblair : ^ ok, moved | 23:05 |
| *** sree has quit IRC | 23:05 | |
| *** xarses has quit IRC | 23:09 | |
| *** andreas_s has joined #openstack-infra | 23:11 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add configure-unbound role to base-test job https://review.openstack.org/510728 | 23:13 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add configure-unbound to base job https://review.openstack.org/511075 | 23:13 |
| *** claudiub|2 has quit IRC | 23:15 | |
| *** Apoorva has quit IRC | 23:16 | |
| *** aeng has joined #openstack-infra | 23:17 | |
| *** Apoorva has joined #openstack-infra | 23:18 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add unbound role https://review.openstack.org/510734 | 23:20 |
| *** thorst has joined #openstack-infra | 23:22 | |
| *** hongbin has quit IRC | 23:22 | |
| *** andreas_s has quit IRC | 23:24 | |
| *** thorst has quit IRC | 23:27 | |
| *** jascott1 has quit IRC | 23:32 | |
| clarkb | pabelanger: were you going to trigger those image builds? | 23:35 |
| *** tosky has quit IRC | 23:35 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Stop using the cached cirros image https://review.openstack.org/510839 | 23:35 |
| clarkb | wait isn't ^ fixed by my image change? | 23:36 |
| clarkb | sshnaidm|off: FYI the image is actually cached at /opt/cache/files regardless of zuulv3 or zuulv2 | 23:37 |
| clarkb | sshnaidm|off: we then have (or will have) handy symlinks to point to that location from /home/jenkins/cache and /home/zuul/cache | 23:37 |
| *** mriedem has quit IRC | 23:38 | |
| *** mat128 has joined #openstack-infra | 23:39 | |
| *** gouthamr has joined #openstack-infra | 23:40 | |
| clarkb | ianw: unbound changes lgtm, I've +2'd the first pair | 23:40 |
| ianw | clarkb: thanks, not sure why 511075 is unhappy | 23:40 |
| clarkb | ianw: I think it was because you pushed a new ozj patchset after pushing 075 | 23:41 |
| clarkb | I rechecked it | 23:41 |
| clarkb | same thing happaned with 728 | 23:41 |
| mordred | clarkb: if you're in a reviewing mood, https://review.openstack.org/#/c/510686/ and https://review.openstack.org/#/c/502185/ could use a +A | 23:41 |
| clarkb | ianw: as for hardlinks stuff, probably want to run the python snippet that determines filesystem on a rackspace xenial node? | 23:41 |
| ianw | yeah, moving onto that now | 23:42 |
| *** Apoorva_ has joined #openstack-infra | 23:42 | |
| *** Apoorva has quit IRC | 23:42 | |
| *** andreas_s has joined #openstack-infra | 23:43 | |
| mordred | clarkb, ianw: I +A'd the unbound role add and the base-test change | 23:45 |
| *** rlandy has quit IRC | 23:45 | |
| ianw | clarkb: was there a reason to not compare stat dev? | 23:45 |
| clarkb | ianw: I read that and thought it wouldn't work for some reason but now rereading it am thinking it will | 23:46 |
| ianw | oh i guess two partitions on the same device it still won't work for | 23:46 |
| *** andreas_s has quit IRC | 23:47 | |
| ianw | hmmm | 23:47 |
| clarkb | oh maybe that was it, was not sure if device is physical device or fs | 23:47 |
| fungi | it's filesystem | 23:47 |
| fungi | hardlinks are just additional labels for the same inode | 23:47 |
| *** andreas_s has joined #openstack-infra | 23:47 | |
| clarkb | fungi: but stat.st_dev is filesystem id not physical device id? | 23:48 |
| fungi | so a hardlink to an inode on a different filesystem doesn't make sense | 23:48 |
| clarkb | ya the reason its failing is you can't hardlink across filesystems | 23:48 |
| ianw | fungi: right, but comparing stat's st_dev it's 100% safe, if two partitions are on the same device | 23:48 |
| fungi | it's the id of the partition afaik | 23:48 |
| ianw | isn't i mean | 23:49 |
| ianw | but it's probably safe enough | 23:49 |
| fungi | partitions are block devices | 23:49 |
| fungi | as are logical volumes | 23:49 |
| *** gildub has joined #openstack-infra | 23:49 | |
| fungi | so it's whatever (pseudo)device the filesystem maps onto | 23:49 |
| ianw | i guess it's major:minor and so it will work | 23:51 |
| fungi | yeah, it'll be unique per filesystem, in effect | 23:51 |
| ianw | this doesn't however, answer the question of how the heck we got a different fs in the mix | 23:51 |
| clarkb | ianw: oh! | 23:53 |
| clarkb | ianw: I know, its because /tmp is on the root device and /opt is on the ephemeral disk on rax | 23:53 |
| clarkb | so ya maybe the thing to do is rewrite that checker to use stat.st_dev instead | 23:53 |
| ianw | hmm, *that* would make sense | 23:53 |
| fungi | that will be the case, yes | 23:53 |
| ianw | yep, ok, that should work | 23:53 |
| ianw | will do | 23:53 |
| *** andreas_s has quit IRC | 23:56 | |
| *** ijw has joined #openstack-infra | 23:56 | |
| *** andreas_s has joined #openstack-infra | 23:56 | |
| mordred | clarkb: oh right. the ephemeral disks | 23:57 |
| clarkb | I'm helping to debug a duplicate acccount in Gerrit and think I need to run `UPDATE account_external_ids SET account_id = 21561 WHERE account_id = 27031;` Basically 27031 is the extra account and want to flatten its openid into 21561. Is that valid sql to update the column you are filtering by (I think it is as those are discrete steps of operation)? | 23:58 |
| *** mancdaz has quit IRC | 23:58 | |
| clarkb | I'm not ready to run that just yet as waiting for ocnfirmation that we don't want to use 27031 going forward | 23:58 |
| clarkb | mordred: fungi ^ | 23:58 |
| ianw | textwrap.dedent()! how did i not know about that | 23:59 |
| pabelanger | clarkb: ya, I can do kick off images now | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!