| *** docaedo has quit IRC | 00:00 | |
| *** bkero has quit IRC | 00:03 | |
| *** bkero has joined #openstack-infra | 00:03 | |
| *** docaedo has joined #openstack-infra | 00:03 | |
| *** yamamoto has quit IRC | 00:05 | |
| *** camunoz has quit IRC | 00:23 | |
| *** yamahata has quit IRC | 00:25 | |
| *** iyamahat_ has quit IRC | 00:25 | |
| *** Apoorva_ has joined #openstack-infra | 00:25 | |
| *** yamamoto has joined #openstack-infra | 00:27 | |
| *** xarses_ has joined #openstack-infra | 00:27 | |
| *** Apoorva has quit IRC | 00:28 | |
| *** yamamoto has quit IRC | 00:29 | |
| *** zaneb has joined #openstack-infra | 00:30 | |
| *** andreww has quit IRC | 00:30 | |
| *** ijw has quit IRC | 00:35 | |
| *** edmondsw has joined #openstack-infra | 00:41 | |
| *** signed8bit_Zzz is now known as signed8bit | 00:42 | |
| *** edmondsw has quit IRC | 00:45 | |
| *** bobh has quit IRC | 00:51 | |
| *** bobh has joined #openstack-infra | 00:57 | |
| *** Apoorva_ has quit IRC | 00:57 | |
| *** Apoorva has joined #openstack-infra | 00:57 | |
| *** signed8bit is now known as signed8bit_Zzz | 00:58 | |
| SamYaple | my zuulv3 jobs are running! | 00:59 |
|---|---|---|
| SamYaple | annnnnd it failed haha | 01:01 |
| *** signed8bit_Zzz is now known as signed8bit | 01:03 | |
| *** bobh has quit IRC | 01:04 | |
| *** nicolasbock has quit IRC | 01:07 | |
| *** bobh has joined #openstack-infra | 01:12 | |
| *** Apoorva has quit IRC | 01:14 | |
| *** Apoorva has joined #openstack-infra | 01:15 | |
| *** bobh has quit IRC | 01:16 | |
| *** ijw has joined #openstack-infra | 01:19 | |
| *** signed8bit is now known as signed8bit_Zzz | 01:19 | |
| *** bobh has joined #openstack-infra | 01:19 | |
| *** Apoorva_ has joined #openstack-infra | 01:20 | |
| *** ijw has quit IRC | 01:21 | |
| *** bobh has quit IRC | 01:23 | |
| *** Apoorva has quit IRC | 01:24 | |
| *** Apoorva_ has quit IRC | 01:25 | |
| *** esberglu has joined #openstack-infra | 01:27 | |
| *** yamamoto has joined #openstack-infra | 01:30 | |
| *** esberglu has quit IRC | 01:32 | |
| *** yamamoto has quit IRC | 01:38 | |
| *** leyal has quit IRC | 01:44 | |
| *** leyal has joined #openstack-infra | 01:45 | |
| *** lihi has quit IRC | 01:45 | |
| *** lihi has joined #openstack-infra | 01:45 | |
| *** mriedem has quit IRC | 01:52 | |
| *** markvoelker has joined #openstack-infra | 01:54 | |
| *** kiennt26 has joined #openstack-infra | 01:57 | |
| *** bobh has joined #openstack-infra | 01:57 | |
| *** yamamoto has joined #openstack-infra | 02:04 | |
| *** Goneri has quit IRC | 02:11 | |
| *** yamamoto has quit IRC | 02:15 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: (newton) make sure we always have /etc/machine-id available. https://review.openstack.org/510261 | 02:19 |
| *** baoli has joined #openstack-infra | 02:23 | |
| *** markvoelker has quit IRC | 02:29 | |
| *** edmondsw has joined #openstack-infra | 02:29 | |
| *** kiennt26 has quit IRC | 02:30 | |
| *** edmondsw has quit IRC | 02:34 | |
| *** gongysh has joined #openstack-infra | 02:43 | |
| *** bobh has quit IRC | 02:46 | |
| *** gouthamr has joined #openstack-infra | 02:47 | |
| *** baoli has quit IRC | 02:51 | |
| *** baoli has joined #openstack-infra | 02:56 | |
| *** esberglu has joined #openstack-infra | 02:58 | |
| *** baoli has quit IRC | 02:59 | |
| *** esberglu has quit IRC | 03:02 | |
| *** superdan is now known as dansmith | 03:10 | |
| *** jcoufal has joined #openstack-infra | 03:12 | |
| *** yamamoto has joined #openstack-infra | 03:15 | |
| *** baoli has joined #openstack-infra | 03:18 | |
| *** gridinv has joined #openstack-infra | 03:22 | |
| *** yamamoto has quit IRC | 03:23 | |
| *** markvoelker has joined #openstack-infra | 03:26 | |
| *** chlong has joined #openstack-infra | 03:26 | |
| *** yamahata has joined #openstack-infra | 03:29 | |
| *** liujiong has joined #openstack-infra | 03:34 | |
| *** liujiong has quit IRC | 03:34 | |
| *** gongysh has quit IRC | 03:51 | |
| *** dbecker has quit IRC | 03:54 | |
| *** armax has joined #openstack-infra | 03:55 | |
| *** markvoelker has quit IRC | 03:59 | |
| *** dbecker has joined #openstack-infra | 04:08 | |
| *** Rockyg has joined #openstack-infra | 04:10 | |
| *** baoli has quit IRC | 04:17 | |
| *** edmondsw has joined #openstack-infra | 04:17 | |
| *** edmondsw has quit IRC | 04:22 | |
| *** jcoufal has quit IRC | 04:32 | |
| *** psachin has quit IRC | 04:38 | |
| *** baoli has joined #openstack-infra | 04:39 | |
| *** gouthamr has quit IRC | 04:41 | |
| *** spzala has joined #openstack-infra | 04:43 | |
| *** baoli has quit IRC | 04:44 | |
| *** baoli has joined #openstack-infra | 04:44 | |
| *** esberglu has joined #openstack-infra | 04:46 | |
| *** claudiub has joined #openstack-infra | 04:48 | |
| *** esberglu has quit IRC | 04:51 | |
| *** markvoelker has joined #openstack-infra | 04:56 | |
| *** armax has quit IRC | 04:59 | |
| *** gildub has joined #openstack-infra | 05:01 | |
| *** docaedo has left #openstack-infra | 05:01 | |
| *** claudiub has quit IRC | 05:02 | |
| *** spzala has quit IRC | 05:09 | |
| *** baoli has quit IRC | 05:20 | |
| *** yamamoto has joined #openstack-infra | 05:28 | |
| *** markvoelker has quit IRC | 05:30 | |
| *** yamamoto has quit IRC | 05:36 | |
| *** Rockyg has quit IRC | 05:40 | |
| *** abelur_ has quit IRC | 05:57 | |
| *** edmondsw has joined #openstack-infra | 06:05 | |
| *** edmondsw has quit IRC | 06:10 | |
| *** yamamoto has joined #openstack-infra | 06:11 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix early processing of merge-pending items on reconfig https://review.openstack.org/509912 | 06:14 |
| *** yamamoto has quit IRC | 06:16 | |
| *** yamamoto has joined #openstack-infra | 06:17 | |
| *** yamamoto has quit IRC | 06:17 | |
| *** yamamoto has joined #openstack-infra | 06:18 | |
| *** yamamoto has quit IRC | 06:20 | |
| *** jascott1 has quit IRC | 06:21 | |
| *** eranrom has quit IRC | 06:22 | |
| *** markvoelker has joined #openstack-infra | 06:27 | |
| *** yamamoto has joined #openstack-infra | 06:32 | |
| *** psachin has joined #openstack-infra | 06:33 | |
| *** jascott1 has joined #openstack-infra | 06:35 | |
| *** ykarel has joined #openstack-infra | 06:36 | |
| *** ykarel has quit IRC | 06:38 | |
| *** ykarel has joined #openstack-infra | 06:38 | |
| *** jascott1 has quit IRC | 06:40 | |
| *** abelur_ has joined #openstack-infra | 06:42 | |
| *** yamamoto has quit IRC | 06:49 | |
| *** jascott1 has joined #openstack-infra | 06:50 | |
| *** hrubi has joined #openstack-infra | 06:54 | |
| *** jascott1 has quit IRC | 06:56 | |
| *** dfflanders has quit IRC | 06:58 | |
| *** markvoelker has quit IRC | 07:00 | |
| *** gildub has quit IRC | 07:06 | |
| *** ykarel has quit IRC | 07:06 | |
| *** ykarel has joined #openstack-infra | 07:07 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Verify aborted jobs don't count into retry limit https://review.openstack.org/510271 | 07:08 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Verify aborted jobs don't count into retry limit https://review.openstack.org/510271 | 07:10 |
| *** tesseract has joined #openstack-infra | 07:11 | |
| *** armaan has quit IRC | 07:16 | |
| *** armaan has joined #openstack-infra | 07:16 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Verify aborted jobs don't count into retry limit https://review.openstack.org/510271 | 07:23 |
| *** esberglu has joined #openstack-infra | 07:28 | |
| *** esberglu has quit IRC | 07:29 | |
| *** esberglu has joined #openstack-infra | 07:29 | |
| *** esberglu has quit IRC | 07:29 | |
| *** armaan has quit IRC | 07:31 | |
| *** armaan has joined #openstack-infra | 07:32 | |
| *** dhajare has joined #openstack-infra | 07:35 | |
| *** jtomasek has joined #openstack-infra | 07:37 | |
| *** armaan has quit IRC | 07:37 | |
| *** armaan has joined #openstack-infra | 07:38 | |
| *** dhajare has quit IRC | 07:40 | |
| *** jtomasek has quit IRC | 07:40 | |
| *** markvoelker has joined #openstack-infra | 07:46 | |
| *** yamamoto has joined #openstack-infra | 07:50 | |
| *** edmondsw has joined #openstack-infra | 07:53 | |
| *** yamamoto has quit IRC | 07:58 | |
| *** edmondsw has quit IRC | 07:58 | |
| *** dhajare has joined #openstack-infra | 08:02 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Return a distinct result on executor disk full https://review.openstack.org/510227 | 08:05 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Have zuul re-run ABORTED jobs https://review.openstack.org/510211 | 08:05 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Always retry jobs on executor disconnect https://review.openstack.org/510223 | 08:05 |
| *** dhajare has quit IRC | 08:07 | |
| *** spzala has joined #openstack-infra | 08:10 | |
| *** spzala has quit IRC | 08:15 | |
| *** esberglu has joined #openstack-infra | 08:22 | |
| *** esberglu has quit IRC | 08:27 | |
| *** sdague has quit IRC | 08:36 | |
| *** yamamoto has joined #openstack-infra | 08:52 | |
| *** yamamoto has quit IRC | 08:58 | |
| *** yamamoto has joined #openstack-infra | 09:00 | |
| *** mrunge has quit IRC | 09:02 | |
| *** yamamoto has quit IRC | 09:02 | |
| *** mrunge has joined #openstack-infra | 09:03 | |
| openstackgerrit | Numan Siddique proposed openstack-infra/project-config master: Add OVN environment files to the tripleo scenario007 jobs https://review.openstack.org/502899 | 09:04 |
| *** yamamoto has joined #openstack-infra | 09:07 | |
| *** gridinv1 has joined #openstack-infra | 09:25 | |
| *** armaan has quit IRC | 09:26 | |
| *** armaan has joined #openstack-infra | 09:26 | |
| *** gridinv1 has quit IRC | 09:30 | |
| *** armaan has quit IRC | 09:36 | |
| *** armaan has joined #openstack-infra | 09:37 | |
| *** edmondsw has joined #openstack-infra | 09:42 | |
| *** yamamoto has quit IRC | 09:45 | |
| *** yamamoto has joined #openstack-infra | 09:45 | |
| *** gridinv1 has joined #openstack-infra | 09:46 | |
| *** gridinv1 has left #openstack-infra | 09:46 | |
| *** edmondsw has quit IRC | 09:46 | |
| *** tosky has joined #openstack-infra | 09:51 | |
| *** ykarel has quit IRC | 09:53 | |
| *** ykarel has joined #openstack-infra | 09:53 | |
| *** ykarel_ has joined #openstack-infra | 10:00 | |
| *** ykarel has quit IRC | 10:03 | |
| *** pbourke has quit IRC | 10:04 | |
| *** gildub has joined #openstack-infra | 10:05 | |
| *** pbourke has joined #openstack-infra | 10:05 | |
| *** Hal has joined #openstack-infra | 10:14 | |
| *** Hal is now known as Guest37380 | 10:15 | |
| *** slaweq has joined #openstack-infra | 10:19 | |
| *** slaweq has quit IRC | 10:20 | |
| *** yamamoto has quit IRC | 10:20 | |
| *** Guest37380 has quit IRC | 10:22 | |
| *** slaweq has joined #openstack-infra | 10:38 | |
| *** gildub has quit IRC | 10:42 | |
| *** slaweq has quit IRC | 10:43 | |
| *** markvoelker has quit IRC | 10:46 | |
| *** armaan has quit IRC | 10:56 | |
| *** psachin has quit IRC | 10:56 | |
| *** armaan has joined #openstack-infra | 10:56 | |
| *** psachin has joined #openstack-infra | 11:15 | |
| *** yamamoto has joined #openstack-infra | 11:21 | |
| *** psachin has quit IRC | 11:22 | |
| *** Hal has joined #openstack-infra | 11:23 | |
| *** Hal is now known as Guest85724 | 11:23 | |
| *** ykarel_ has quit IRC | 11:25 | |
| *** edmondsw has joined #openstack-infra | 11:29 | |
| *** yamamoto has quit IRC | 11:30 | |
| *** Guest85724 has quit IRC | 11:31 | |
| *** ijw has joined #openstack-infra | 11:34 | |
| *** edmondsw has quit IRC | 11:34 | |
| *** ijw has quit IRC | 11:38 | |
| *** yamamoto has joined #openstack-infra | 11:38 | |
| *** psachin has joined #openstack-infra | 11:39 | |
| *** yamamoto has quit IRC | 11:39 | |
| *** slaweq has joined #openstack-infra | 12:08 | |
| *** spzala has joined #openstack-infra | 12:11 | |
| *** slaweq has quit IRC | 12:12 | |
| *** spzala has quit IRC | 12:16 | |
| *** yamamoto has joined #openstack-infra | 12:24 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack/os-testr master: Use subunit and stestr API more https://review.openstack.org/509752 | 12:37 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack/os-testr master: Use subunit and stestr API more https://review.openstack.org/509752 | 12:42 |
| *** tosky has quit IRC | 12:43 | |
| *** markvoelker has joined #openstack-infra | 12:47 | |
| *** psachin has quit IRC | 12:51 | |
| *** claudiub has joined #openstack-infra | 12:56 | |
| *** jcoufal has joined #openstack-infra | 13:01 | |
| *** tosky has joined #openstack-infra | 13:02 | |
| *** bobh has joined #openstack-infra | 13:04 | |
| *** baoli has joined #openstack-infra | 13:16 | |
| *** edmondsw has joined #openstack-infra | 13:18 | |
| *** baoli has quit IRC | 13:21 | |
| *** markvoelker has quit IRC | 13:22 | |
| *** edmondsw has quit IRC | 13:22 | |
| *** yamamoto has quit IRC | 13:28 | |
| *** yamamoto has joined #openstack-infra | 13:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add requirements repo to some periodic jobs https://review.openstack.org/510297 | 13:32 |
| AJaeger | team, we now have finally some periodic jobs running - and failing. First fix ^ | 13:32 |
| *** yamamoto has quit IRC | 13:33 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Remove promotion logic from the toci scripts https://review.openstack.org/507076 | 13:44 |
| *** askb has quit IRC | 13:49 | |
| *** askb has joined #openstack-infra | 13:52 | |
| *** slaweq has joined #openstack-infra | 13:54 | |
| *** andreas_s has joined #openstack-infra | 13:54 | |
| *** baoli has joined #openstack-infra | 13:57 | |
| *** slaweq has quit IRC | 13:59 | |
| *** askb has quit IRC | 13:59 | |
| *** mat128 has joined #openstack-infra | 14:03 | |
| jeblair | mordred, EmilienM, mwhahaha: oh, i found the reason why the stable branch changes weren't used in the tests for 508660. it's the easiest thing in the world -- they did not exist at the time the tests ran. they were uploaded a day later. so there's no evidence here of either a bug in zuul or the tripleo tests. sorry for the noise. | 14:04 |
| *** armaan has quit IRC | 14:06 | |
| AJaeger | jeblair: glad to hear - yes, that explains it. | 14:06 |
| *** jcoufal has quit IRC | 14:06 | |
| AJaeger | infra-root, zuul v3 shows a post job for openstack-infra/zuul in the queue for 19+ hours - anything broken there? | 14:08 |
| AJaeger | publish-openstack-python-branch-tarball is queued and seems to not get a job | 14:08 |
| AJaeger | s/get a job/get a node/ | 14:08 |
| jeblair | AJaeger: i'll take a look | 14:08 |
| AJaeger | thanks, jeblair | 14:09 |
| AJaeger | nothing urgent - just the only odd thing I noticed | 14:09 |
| *** askb has joined #openstack-infra | 14:10 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Verify aborted jobs don't count into retry limit https://review.openstack.org/510271 | 14:12 |
| *** claudiub has quit IRC | 14:12 | |
| *** spzala has joined #openstack-infra | 14:12 | |
| *** spzala has quit IRC | 14:12 | |
| *** toabctl has quit IRC | 14:14 | |
| *** toabctl has joined #openstack-infra | 14:15 | |
| jeblair | AJaeger, Shrews: there was a zookeeper disconnect while the node was building. we deleted the node, but we seem to have left the request as it was. so it's sitting there with a 'pending' state waiting for a node that doesn't exist to finish building. | 14:18 |
| SpamapS | jeblair: do we need to make requests ephemeral? | 14:19 |
| jeblair | i'll save the relevant logs so we can look at it later. and i think we still need a full restart, which will clear it out. | 14:19 |
| jeblair | SpamapS: they are ephemeral. but zuul did not disconnect, only the nodepool launcher. | 14:19 |
| jeblair | (so hey! good news, huh!) | 14:19 |
| SpamapS | so we need a timeout on the waiting for node build? | 14:21 |
| * SpamapS isn't sure and hasn't looked closely | 14:21 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add sub_nodes and node_private files in /etc/nodepool https://review.openstack.org/510213 | 14:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Use v3 translation jobs for doc projects https://review.openstack.org/510298 | 14:22 |
| AJaeger | since we have merged the translation jobs, let's exercise them ^ | 14:22 |
| jeblair | SpamapS: well, we should be able to detect this situation -- an unlocked pending request is an invalid state; it should just be reset. (i think -- i haven't had breakfast yet) | 14:24 |
| jeblair | so any launcher manager thread should be able to look at that and fix it | 14:24 |
| *** baoli has quit IRC | 14:25 | |
| jeblair | the zk connection was lost because the launcher was restarted to pick up a code change (we need better logging on stop/start; i had to go to syslog to see that) | 14:25 |
| jeblair | but that's good news too -- no zk issue, just normal process | 14:26 |
| pabelanger | cool | 14:26 |
| SpamapS | jeblair: indeed, so it's just an unhandled state | 14:26 |
| pabelanger | jeblair: since zuul was restarted, should we also do executors to pick up ABORTED job retry? Or hold off for any other fixes | 14:27 |
| AJaeger | pabelanger: zuul v3 was not *yet* restarted AFAIU | 14:28 |
| jeblair | i think we still need the full system restart... | 14:29 |
| jeblair | pabelanger: you want to help out? | 14:29 |
| jeblair | SpamapS, pabelanger, Shrews: http://paste.openstack.org/show/622959/ are the logs for the nodepool issue. i'll add it to the etherpad | 14:30 |
| pabelanger | jeblair: sure, what would you like me to do | 14:30 |
| *** andreas_s has quit IRC | 14:31 | |
| jeblair | pabelanger: you want to stop the nodepool launchers and zookeeper itself? i'll take care of zuul? | 14:31 |
| pabelanger | okay | 14:32 |
| jeblair | i'm still going to install a custom zuul scheduler with the backdoor repl patch, so it's there in case we need any further debugging, but i'll rebase that on branch tip | 14:32 |
| *** wolverineav has joined #openstack-infra | 14:33 | |
| AJaeger | if you afterwards have some review time, these two changes update periodic jobs, would be nice to have them in for next run, please: https://review.openstack.org/510297 and https://review.openstack.org/510298 | 14:33 |
| AJaeger | bbl | 14:33 |
| pabelanger | ready to stop nodepool-launchers (nl01 / nl02) | 14:34 |
| jeblair | pabelanger: go for it. i'm stopping zuul now | 14:34 |
| pabelanger | jeblair: launchers stopped, say when for zookeeper | 14:35 |
| jeblair | pabelanger: go for it | 14:35 |
| pabelanger | zk stopped | 14:36 |
| pabelanger | okay for start? | 14:36 |
| jeblair | pabelanger: yep | 14:36 |
| pabelanger | zk started | 14:36 |
| pabelanger | say when for nodepool-launchers | 14:36 |
| jeblair | pabelanger: whenever you're ready | 14:37 |
| pabelanger | jeblair: both launchers started | 14:38 |
| jeblair | i've restarted the mergers and executors | 14:38 |
| jeblair | and zuulv3 scheduler is starting | 14:39 |
| pabelanger | launchers have ready nodes now, waiting for requests | 14:40 |
| *** spzala has joined #openstack-infra | 14:40 | |
| jeblair | andreaf: we restarted with the fix for the issue you had with not seeing error info for missing roles | 14:42 |
| *** wolverineav has quit IRC | 14:43 | |
| pabelanger | jeblair: i see an exception in debug.log on zuulv3 | 14:43 |
| pabelanger | after | 14:43 |
| pabelanger | 2017-10-07 14:41:19,181 ERROR zuul.Scheduler: Error starting Zuul: | 14:43 |
| jeblair | heh | 14:43 |
| jeblair | we tightened up a yaml syntax thing. and of course we had already broken that in our configuration :) | 14:44 |
| jeblair | pabelanger: we're going to have to force merge the fix for that and restart | 14:44 |
| jeblair | i'll make the change | 14:44 |
| pabelanger | ok | 14:44 |
| jeblair | omg that file is 38530 lines | 14:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Fix syntax error with null dict value https://review.openstack.org/510300 | 14:47 |
| jeblair | pabelanger: you want to force-merge that ^ ? | 14:48 |
| pabelanger | jeblair: sure, let me fixup groups | 14:48 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix syntax error with null dict value https://review.openstack.org/510300 | 14:50 |
| pabelanger | done | 14:50 |
| jeblair | k, rerestarting | 14:50 |
| *** armaan has joined #openstack-infra | 14:50 | |
| jeblair | yay! that looks better | 14:54 |
| *** mat128 has quit IRC | 14:54 | |
| pabelanger | Oh, another syntax error | 14:56 |
| pabelanger | build-openstack-sphinx-docs this time | 14:56 |
| jeblair | pabelanger: yes, but the config loaded this time, so it's not critical | 14:56 |
| pabelanger | indeed | 14:56 |
| pabelanger | let me see if I can get a fix | 14:56 |
| jeblair | i don't see the error though | 14:57 |
| pabelanger | looks like we don't allow extra keys | 14:58 |
| pabelanger | but, not sure what is extra | 14:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add a CRD multi-branch test https://review.openstack.org/510301 | 14:58 |
| jeblair | there may well be a bug in the extra validation mordred added | 14:59 |
| pabelanger | Oh | 14:59 |
| jeblair | however, i have no idea why neither the tests nor the configuration up to this point didn't trigger it, but this change did. | 14:59 |
| pabelanger | I think I see | 14:59 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/zuul/tree/tests/unit/test_v3.py?id=8be3c0c5a3e10ad42655f9aa6fcfa42dc966ba57#n856 | 15:00 |
| pabelanger | guess it should be: | 15:00 |
| pabelanger | - required-projects: | 15:00 |
| pabelanger | we have | 15:00 |
| pabelanger | required-projects: | 15:00 |
| jeblair | whoah | 15:01 |
| pabelanger | but, that doesn't seem right | 15:01 |
| jeblair | it should definitely not be - required projects | 15:01 |
| pabelanger | yah | 15:01 |
| jeblair | pabelanger: that's a test that should report an error | 15:01 |
| jeblair | pabelanger: so that's correct. | 15:01 |
| pabelanger | Oh | 15:02 |
| pabelanger | Hmm | 15:02 |
| pabelanger | Ya, my mistake | 15:02 |
| *** edmondsw has joined #openstack-infra | 15:06 | |
| AJaeger | pabelanger, jeblair, I got http://paste.openstack.org/show/622966/ on https://review.openstack.org/510298 | 15:09 |
| AJaeger | "extra keys not allowed @ data['check']['jobs'][0]['requirements-check']['name'] | 15:09 |
| jeblair | AJaeger: yeah, that's the same error pabelanger saw | 15:09 |
| jeblair | different template | 15:10 |
| AJaeger | yes, different one | 15:10 |
| *** edmondsw has quit IRC | 15:10 | |
| AJaeger | and https://review.openstack.org/510274 says " | 15:11 |
| AJaeger | This change depends on a change that failed to merge." | 15:11 |
| AJaeger | But there's no dependency. It's stacked on top of another one - without depends on as well... | 15:12 |
| AJaeger | This worked before the zuul restart | 15:12 |
| jeblair | AJaeger: yes, it seems very likely to be related to I217eb5d6befbed51b220d47afa18997a87982389 but i can not see what's wrong | 15:13 |
| pabelanger | "extra keys not allowed" to me means validator doesn't like something we passed, but I cannot see an extra key | 15:14 |
| pabelanger | sadly, I have to run to the fair now. but will be back in a few hours to help more | 15:17 |
| *** markvoelker has joined #openstack-infra | 15:18 | |
| *** hwoarang has quit IRC | 15:19 | |
| jeblair | don't be sad you're going to a fair! | 15:21 |
| *** hwoarang has joined #openstack-infra | 15:21 | |
| jeblair | the extra key is 'name' which we did not pass, which is why it's confusing | 15:21 |
| *** jkilpatr has quit IRC | 15:29 | |
| SpamapS | maybe a template came into play we didn't expect? | 15:30 |
| *** armaan has quit IRC | 15:33 | |
| *** armaan has joined #openstack-infra | 15:34 | |
| jeblair | i think i get it -- i think we're modifying the template config object, so this only happens the *second* time we build the configuration | 15:46 |
| *** e0ne has joined #openstack-infra | 15:49 | |
| *** markvoelker has quit IRC | 15:52 | |
| *** kiennt26 has joined #openstack-infra | 15:55 | |
| *** bobh has quit IRC | 15:57 | |
| *** askb has quit IRC | 15:57 | |
| *** esberglu has joined #openstack-infra | 15:58 | |
| *** e0ne has quit IRC | 15:59 | |
| *** esberglu has quit IRC | 16:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't modify project-templates to add job name attributes https://review.openstack.org/510304 | 16:14 |
| jeblair | mordred, pabelanger, SpamapS, AJaeger: ^ there's the fix | 16:14 |
| jeblair | mordred: i have also realized that another solution to the original problem would have been to replace the 'dict' part of the project-template validator with '{str: dict}'. then i believe we would have been correctly deferring to the jobparser for validation of the internal job definition. | 16:15 |
| jeblair | i'm going to apply 510304 locally/manually and restart | 16:16 |
| *** psachin has joined #openstack-infra | 16:17 | |
| *** claudiub has joined #openstack-infra | 16:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add glance to integrated gate https://review.openstack.org/510305 | 16:22 |
| *** jtomasek has joined #openstack-infra | 16:26 | |
| *** jtomasek has quit IRC | 16:26 | |
| *** signed8bit_Zzz is now known as signed8bit | 16:28 | |
| *** bobh has joined #openstack-infra | 16:29 | |
| *** signed8bit has quit IRC | 16:33 | |
| *** bobh has quit IRC | 16:34 | |
| *** tesseract has quit IRC | 16:35 | |
| *** jtomasek has joined #openstack-infra | 16:36 | |
| *** jtomasek has quit IRC | 16:37 | |
| *** jtomasek has joined #openstack-infra | 16:38 | |
| *** coffee_cat is now known as ildikov | 16:39 | |
| *** aviau has quit IRC | 16:45 | |
| *** aviau has joined #openstack-infra | 16:45 | |
| *** markvoelker has joined #openstack-infra | 16:49 | |
| *** bobh has joined #openstack-infra | 16:51 | |
| *** edmondsw has joined #openstack-infra | 16:54 | |
| *** edmondsw has quit IRC | 16:58 | |
| *** spzala has quit IRC | 16:58 | |
| *** chlong has quit IRC | 17:01 | |
| *** nicolasbock has joined #openstack-infra | 17:02 | |
| *** david-lyle has quit IRC | 17:04 | |
| *** e0ne has joined #openstack-infra | 17:10 | |
| *** kiennt26 has quit IRC | 17:19 | |
| *** markvoelker has quit IRC | 17:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Convert legacy doc build jobs to v3 https://review.openstack.org/510306 | 17:23 |
| *** claudiub has quit IRC | 17:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Convert legacy doc build jobs to v3 https://review.openstack.org/510306 | 17:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove tox-checkniceness https://review.openstack.org/510172 | 17:39 |
| SamYaple | wow zuulv3 has almost no jobs! i can test again! | 17:40 |
| *** esberglu has joined #openstack-infra | 17:41 | |
| *** spzala has joined #openstack-infra | 17:41 | |
| *** esberglu has quit IRC | 17:43 | |
| AJaeger | SamYaple: quick ;) | 17:43 |
| AJaeger | SamYaple: we restarted it... | 17:43 |
| AJaeger | mordred: please check https://review.openstack.org/510306 - once this is merged, I will need your help with publishing the content as well... conversion was nice, review is welcome (note nothing urgent, can wait until Monday) | 17:48 |
| SamYaple | oh im trying | 17:48 |
| AJaeger | SamYaple: rest of day should be fine - it's weekend ;) | 17:48 |
| *** armaan has quit IRC | 17:49 | |
| *** e0ne has quit IRC | 17:50 | |
| SamYaple | thanks for reminding me i have no life AJaeger | 17:52 |
| * SamYaple runs away crying | 17:52 | |
| SamYaple | actually the kids and wife get back from the in-laws sunday night. theyve been gone for two weeks. very quite house | 17:53 |
| SamYaple | very empty | 17:53 |
| AJaeger | SamYaple: I guess too quite ;) | 17:53 |
| *** e0ne has joined #openstack-infra | 18:02 | |
| EmilienM | jeblair: ack, no worries, I'm glad we have made progress | 18:03 |
| AJaeger | mordred: do we need https://review.openstack.org/#/c/495426/ ? Then please rebase | 18:09 |
| clarkb | whats up with http://logs.openstack.org/01/510201/3/check/legacy-grenade-dsvm-cinder-mn-sub-volbak/0f712cf/job-output.txt.gz#_2017-10-07_17_37_09_266622 ? | 18:26 |
| clarkb | looks like that may be breaking multinode devstack-gate jobs/grenade jobs | 18:26 |
| smcginnis | I think when we were wholly on zv3 that job was always failing. I never got a chance to dig into it though. | 18:27 |
| clarkb | smcginnis: ya I managed to get it working I think | 18:27 |
| clarkb | this is a new failure | 18:27 |
| smcginnis | Oh goodie. | 18:28 |
| *** chlong has joined #openstack-infra | 18:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 18:44 |
| *** armaan has joined #openstack-infra | 18:46 | |
| *** e0ne has quit IRC | 18:47 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Collect /etc/machine-id in oooq jobs https://review.openstack.org/510312 | 19:01 |
| *** jkilpatr has joined #openstack-infra | 19:01 | |
| SamYaple | the infra mirror for docker is 5 months out of date it seems | 19:10 |
| *** dave-mccowan has quit IRC | 19:12 | |
| SamYaple | ah its pointing to the old location for packages taht no longer exists | 19:12 |
| openstackgerrit | Sam Yaple proposed openstack-infra/system-config master: Add ceph-lumionus mirror https://review.openstack.org/510313 | 19:14 |
| *** Goneri has joined #openstack-infra | 19:16 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/system-config master: Change mirror location for Docker https://review.openstack.org/510314 | 19:18 |
| SamYaple | that fixes it ^ | 19:18 |
| *** markvoelker has joined #openstack-infra | 19:19 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/system-config master: Change mirror location for Docker https://review.openstack.org/510314 | 19:20 |
| *** edmondsw has joined #openstack-infra | 19:29 | |
| *** edmondsw has quit IRC | 19:31 | |
| SamYaple | how long does it take for mirrors to start mirroring once a patch merges like above? | 19:37 |
| jeblair | SamYaple: somewhere between 5-60 minutes i'd reckon (it's two systems on cron cycles) | 19:46 |
| *** markvoelker has quit IRC | 19:52 | |
| *** slaweq has joined #openstack-infra | 19:55 | |
| *** spzala has quit IRC | 19:57 | |
| SamYaple | oh thats not bad | 19:59 |
| SamYaple | cool | 19:59 |
| *** slaweq has quit IRC | 20:00 | |
| openstackgerrit | Merged openstack-infra/system-config master: Change mirror location for Docker https://review.openstack.org/510314 | 20:09 |
| SamYaple | this may be the quickest ive ever gotten a patch through anything in openstack-infra | 20:09 |
| SamYaple | less than an hour from proposed to merged. im impressed | 20:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add required-projects to legacy-logstash-filters jobs https://review.openstack.org/510315 | 20:10 |
| *** dave-mccowan has joined #openstack-infra | 20:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: DNM - testing https://review.openstack.org/510316 | 20:13 |
| andreaf | jeblair: thanks - hopefully it was just a depends-on missing on my side, so I should not see the issue anymore, but if do I'll let you know if the log is better | 20:15 |
| *** yamahata has quit IRC | 20:16 | |
| *** felipemonteiro has joined #openstack-infra | 20:20 | |
| *** masber has quit IRC | 20:21 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:22 | |
| *** felipemonteiro has quit IRC | 20:25 | |
| *** edmondsw has joined #openstack-infra | 20:35 | |
| *** felipemonteiro_ has quit IRC | 20:35 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:35 | |
| *** felipemonteiro__ has joined #openstack-infra | 20:37 | |
| *** spzala has joined #openstack-infra | 20:38 | |
| *** edmondsw has quit IRC | 20:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't use cached config when deleting files https://review.openstack.org/510317 | 20:39 |
| *** felipemonteiro_ has quit IRC | 20:40 | |
| jeblair | andreaf, mordred: ^ that explains/fixes the 'is not permitted to shadow job devstack' error from the change pair that moves devstack job from one repo to another -- the key is that the entire .zuul.yaml file was deleted, which is something we did not have a test for. | 20:40 |
| jeblair | we are down to 1 open bug on the 'issues needing deeper debugging' section, and that's the nodepool pending request bug. | 20:43 |
| jeblair | and maybe 2 or 3 job bugs (depending on how you count) | 20:44 |
| jeblair | we're *really* close | 20:44 |
| pabelanger | Yay | 20:45 |
| jeblair | i just checked on memory -- there are 12 layouts active now, and all of them are reachable from the scheduler; there are no leaked layouts. | 20:47 |
| pabelanger | Ya, both memory and cpu look good in caci.o.o for zuulv3.o.o | 20:48 |
| pabelanger | jeblair: great work! | 20:48 |
| *** markvoelker has joined #openstack-infra | 20:49 | |
| clarkb | jeblair: reading that commit message, why use any data from cache if the project branch doesn't exist? | 20:49 |
| jeblair | clarkb: we shouldn't; i may have saturdayed the commit message :) | 20:50 |
| jeblair | clarkb: basically "no entry for project-branch at all" means that project-branch is not involved in the change series. "entry for project-branch with null files (that looks like: {'zuul.yaml': None, '.zuul.yaml':None})" means the project-branch *is* involved, but there's no zuul config in the repo (either never was there, or it was deleted) | 20:52 |
| clarkb | gotcha that way you have a complete config | 20:54 |
| jeblair | yup | 20:55 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add required-projects to legacy-logstash-filters jobs https://review.openstack.org/510315 | 20:58 |
| *** gildub has joined #openstack-infra | 20:59 | |
| *** gildub has quit IRC | 21:05 | |
| *** jascott1 has joined #openstack-infra | 21:22 | |
| *** sileht has quit IRC | 21:23 | |
| *** markvoelker has quit IRC | 21:23 | |
| *** sileht has joined #openstack-infra | 21:24 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add required-projects to legacy-openstackci-beaker https://review.openstack.org/510318 | 21:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: DNM - testing https://review.openstack.org/510316 | 21:30 |
| *** baoli has joined #openstack-infra | 21:35 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Revert "Make openstackci-beaker-ubuntu-trusty and logstash-filters non-voting" https://review.openstack.org/510320 | 21:46 |
| pabelanger | jeblair: mordred: AJaeger: clarkb: ^ fixes zuulv3 jobs for system-config | 21:50 |
| pabelanger | another item we can cross out on etherpad once merged | 21:51 |
| pabelanger | also added patches to zuulv3-issues etherpad | 21:51 |
| SamYaple | win 32 | 21:53 |
| *** baoli has quit IRC | 22:04 | |
| *** felipemonteiro_ has joined #openstack-infra | 22:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 22:12 |
| *** felipemonteiro__ has quit IRC | 22:16 | |
| *** markvoelker has joined #openstack-infra | 22:20 | |
| *** edmondsw has joined #openstack-infra | 22:23 | |
| *** Goneri has quit IRC | 22:26 | |
| *** edmondsw has quit IRC | 22:27 | |
| *** jtomasek has quit IRC | 22:34 | |
| *** baoli has joined #openstack-infra | 22:39 | |
| *** baoli has quit IRC | 22:44 | |
| *** felipemonteiro_ has quit IRC | 22:50 | |
| *** markvoelker has quit IRC | 22:54 | |
| *** askb has joined #openstack-infra | 22:56 | |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Revert "Make openstackci-beaker-ubuntu-trusty and logstash-filters non-voting" https://review.openstack.org/510320 | 23:00 |
| jeblair | added depends-on ^ | 23:00 |
| *** masber has joined #openstack-infra | 23:16 | |
| *** tosky has quit IRC | 23:19 | |
| *** masber has quit IRC | 23:21 | |
| *** baoli has joined #openstack-infra | 23:34 | |
| *** baoli has quit IRC | 23:39 | |
| *** claudiub has joined #openstack-infra | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!