| *** yamamoto has quit IRC | 00:02 | |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Adding the Contributor Guide Project https://review.openstack.org/509943 | 00:04 |
|---|---|---|
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Adding check/gate jobs to Contributor Guide https://review.openstack.org/509937 | 00:04 |
| thingee | mordred: argh, hang on | 00:05 |
| thingee | I don't even know how I did that | 00:05 |
| thingee | mordred: oh wait this is right whew | 00:06 |
| thingee | I think | 00:06 |
| *** ijw has joined #openstack-infra | 00:08 | |
| *** ijw has joined #openstack-infra | 00:08 | |
| thingee | anyone in infra able to approve my thanksbot? https://review.openstack.org/#/q/topic:thanksbot+status:+open been a couple of months and just needs one more +2 | 00:11 |
| *** hemna__ has quit IRC | 00:14 | |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Adding the Contributor Guide Project https://review.openstack.org/509943 | 00:16 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Adding check/gate jobs to Contributor Guide https://review.openstack.org/509937 | 00:16 |
| pabelanger | thingee: +3 | 00:18 |
| thingee | pabelanger: you will be the first person I give a thanks to | 00:18 |
| pabelanger | nice | 00:19 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Require requirements for legacy-bandit-integration https://review.openstack.org/509846 | 00:25 |
| *** hemna__ has joined #openstack-infra | 00:25 | |
| SamYaple | are we gonna hit 24 hours on some zuulv3 jobs? taking bets | 00:30 |
| openstackgerrit | Merged openstack-infra/puppet-statusbot master: Enable #thanks feature statusbot https://review.openstack.org/473535 | 00:31 |
| openstackgerrit | Merged openstack-infra/system-config master: Enable the #thanks feature in statusbot https://review.openstack.org/473536 | 00:32 |
| mnaser | SamYaple i think its wedged | 00:32 |
| *** Swami has quit IRC | 00:32 | |
| *** claudiub has quit IRC | 00:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove references to pipelines, queues, and layouts on dequeue https://review.openstack.org/509903 | 00:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix early processing of merge-pending items on reconfig https://review.openstack.org/509912 | 00:37 |
| pabelanger | mnaser: SamYaple: not wedged, just slow going | 00:37 |
| mnaser | yeah, i lied :( | 00:37 |
| SamYaple | i know, ive been watching it haha | 00:37 |
| mnaser | you can actually load the status page | 00:38 |
| SamYaple | its not a big thing, im just trying to iterate on the gates and its been all day waiting to see if a change worked | 00:38 |
| SamYaple | mnaser: heh no | 00:38 |
| SamYaple | mnaser: https://github.com/kk7ds/openstack-gerrit-dashboard/blob/master/dash.py | 00:38 |
| mnaser | its nice im seeing +1's from both zuul and jenkins for puppet jobs so thats cool | 00:38 |
| pabelanger | surprisingly, we have ready nodes in vexxhost for over 15 hours | 00:38 |
| pabelanger | going to see why that is | 00:38 |
| SamYaple | pabelanger: just put my jobs on there | 00:38 |
| SamYaple | easy peasy | 00:38 |
| jeblair | one 509903 passes tests, i'm going to want to restart the scheduler with it. we'll lose the queues then and start over. | 00:39 |
| jeblair | that may be tomorrow morning | 00:39 |
| mnaser | things are looking waaay waaaa better even when zuulv3 has a huge queue as well though | 00:40 |
| mnaser | i guess the one last thing is seeing what happens in higher chur nrates | 00:40 |
| *** csomerville has quit IRC | 00:43 | |
| pabelanger | k, I've deleted a node in vexxhost, was rebuilt and moved in-use | 00:44 |
| pabelanger | so, not sure why the others never got used | 00:44 |
| pabelanger | deleting them now, and going to check debug logs | 00:44 |
| *** yamamoto has joined #openstack-infra | 00:45 | |
| *** yamamoto has quit IRC | 00:49 | |
| *** baoli has joined #openstack-infra | 00:52 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add required-projects to the blazar-dashboard jobs https://review.openstack.org/509078 | 00:58 |
| *** namnh has joined #openstack-infra | 01:01 | |
| *** jkilpatr has quit IRC | 01:06 | |
| *** Apoorva has quit IRC | 01:11 | |
| *** jogo has joined #openstack-infra | 01:13 | |
| *** cuongnv has joined #openstack-infra | 01:13 | |
| *** openstackstatus has quit IRC | 01:14 | |
| *** openstackstatus has joined #openstack-infra | 01:15 | |
| *** ChanServ sets mode: +v openstackstatus | 01:15 | |
| *** hongbin has joined #openstack-infra | 01:20 | |
| *** calbers has quit IRC | 01:24 | |
| *** calbers has joined #openstack-infra | 01:24 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use normal docs build jobs https://review.openstack.org/509833 | 01:25 |
| *** mriedem has quit IRC | 01:26 | |
| *** zigo has quit IRC | 01:27 | |
| *** ihrachys_ has joined #openstack-infra | 01:27 | |
| *** baoli has quit IRC | 01:27 | |
| *** ihrachys has quit IRC | 01:29 | |
| *** zigo has joined #openstack-infra | 01:31 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Re-add query for lvm create vol from snap bug 1642111 https://review.openstack.org/509899 | 01:31 |
| openstack | bug 1642111 in Cinder "create lvm volume from snapshot fails with "device-mapper: reload ioctl on (252:4) failed: Invalid argument"" [Critical,Confirmed] https://launchpad.net/bugs/1642111 | 01:31 |
| *** esberglu has joined #openstack-infra | 01:31 | |
| *** esberglu has quit IRC | 01:32 | |
| *** esberglu has joined #openstack-infra | 01:32 | |
| *** esberglu has quit IRC | 01:36 | |
| *** kiennt26 has joined #openstack-infra | 01:41 | |
| *** d0ugal has quit IRC | 01:45 | |
| *** d0ugal has joined #openstack-infra | 01:45 | |
| *** yamamoto has joined #openstack-infra | 01:47 | |
| *** ijw has quit IRC | 01:47 | |
| *** markvoelker has joined #openstack-infra | 01:52 | |
| *** yamamoto has quit IRC | 01:53 | |
| *** yamahata has quit IRC | 01:55 | |
| *** iyamahat_ has quit IRC | 01:56 | |
| *** portdirect has quit IRC | 02:04 | |
| *** sdake has quit IRC | 02:04 | |
| *** sweston has quit IRC | 02:04 | |
| *** portdirect has joined #openstack-infra | 02:05 | |
| *** sweston has joined #openstack-infra | 02:05 | |
| *** madhuvishy has quit IRC | 02:05 | |
| *** threestrands_ has joined #openstack-infra | 02:05 | |
| *** pfallenop has quit IRC | 02:05 | |
| *** apetrich has quit IRC | 02:06 | |
| *** andymccr has quit IRC | 02:06 | |
| *** masayukig[m] has quit IRC | 02:06 | |
| *** aspiers[m] has quit IRC | 02:06 | |
| *** madhuvishy has joined #openstack-infra | 02:07 | |
| *** pfallenop has joined #openstack-infra | 02:07 | |
| *** pfallenop has joined #openstack-infra | 02:07 | |
| *** bandini has quit IRC | 02:08 | |
| *** onovy has quit IRC | 02:08 | |
| *** mtreinish has quit IRC | 02:08 | |
| *** threestrands has quit IRC | 02:08 | |
| *** calbers has quit IRC | 02:08 | |
| *** jistr has quit IRC | 02:08 | |
| *** patricku_ has joined #openstack-infra | 02:09 | |
| *** armaan has quit IRC | 02:09 | |
| *** stephenfin has quit IRC | 02:09 | |
| *** dirk_ has joined #openstack-infra | 02:10 | |
| *** jistr has joined #openstack-infra | 02:10 | |
| *** onovy has joined #openstack-infra | 02:10 | |
| *** bandini has joined #openstack-infra | 02:10 | |
| *** seongsoocho_ has joined #openstack-infra | 02:10 | |
| *** mtreinish has joined #openstack-infra | 02:10 | |
| *** edwarnicke_ has joined #openstack-infra | 02:10 | |
| *** kong_ has joined #openstack-infra | 02:11 | |
| *** stephenfin has joined #openstack-infra | 02:11 | |
| tonyb | Is the post pipeline actib on zuulv3 ATM? (I'd expect not as zuulv3 isn't merging code) | 02:11 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Collect mistral-ansible execution files from /tmp https://review.openstack.org/483867 | 02:11 |
| *** calbers has joined #openstack-infra | 02:12 | |
| *** edwarnicke has quit IRC | 02:12 | |
| *** patricku has quit IRC | 02:12 | |
| *** seongsoocho has quit IRC | 02:12 | |
| *** kong has quit IRC | 02:12 | |
| *** dirk has quit IRC | 02:12 | |
| *** kambiz has quit IRC | 02:12 | |
| *** logan- has quit IRC | 02:12 | |
| *** eventingmonkey has quit IRC | 02:12 | |
| *** seongsoocho_ is now known as seongsoocho | 02:12 | |
| *** kong_ is now known as kong | 02:12 | |
| *** edwarnicke_ is now known as edwarnicke | 02:12 | |
| *** patricku_ is now known as patricku | 02:12 | |
| *** dirk_ is now known as dirk | 02:12 | |
| *** sdake has joined #openstack-infra | 02:13 | |
| *** andymccr has joined #openstack-infra | 02:13 | |
| *** sdake is now known as Guest2849 | 02:13 | |
| *** kambiz has joined #openstack-infra | 02:14 | |
| *** bandini has quit IRC | 02:15 | |
| *** fbouliane has quit IRC | 02:15 | |
| *** njohnston has quit IRC | 02:15 | |
| *** logan- has joined #openstack-infra | 02:15 | |
| *** apetrich has joined #openstack-infra | 02:15 | |
| *** masayukig[m] has joined #openstack-infra | 02:16 | |
| *** aspiers[m] has joined #openstack-infra | 02:16 | |
| *** bandini has joined #openstack-infra | 02:17 | |
| *** fbouliane has joined #openstack-infra | 02:17 | |
| *** eventingmonkey has joined #openstack-infra | 02:17 | |
| *** njohnston has joined #openstack-infra | 02:19 | |
| *** hughsaunders has quit IRC | 02:22 | |
| *** mgagne has quit IRC | 02:23 | |
| *** comstud has quit IRC | 02:24 | |
| *** mgagne has joined #openstack-infra | 02:24 | |
| *** mgagne is now known as Guest66098 | 02:24 | |
| *** hughsaunders has joined #openstack-infra | 02:26 | |
| *** markvoelker has quit IRC | 02:27 | |
| *** rkukura_ has joined #openstack-infra | 02:28 | |
| *** dave-mccowan has quit IRC | 02:33 | |
| SamYaple | tonyb: only check pipeline | 02:35 |
| tonyb | SamYaple: Thanks | 02:35 |
| *** kazsh has quit IRC | 02:35 | |
| *** biancat has quit IRC | 02:35 | |
| *** rkukura has quit IRC | 02:35 | |
| *** rkukura_ is now known as rkukura | 02:35 | |
| *** kazsh has joined #openstack-infra | 02:36 | |
| *** biancat has joined #openstack-infra | 02:37 | |
| *** biancat is now known as Guest72950 | 02:37 | |
| *** mrunge has quit IRC | 02:38 | |
| *** adreznec has quit IRC | 02:38 | |
| *** ericyoung has quit IRC | 02:38 | |
| *** calbers has quit IRC | 02:39 | |
| *** comstud has joined #openstack-infra | 02:39 | |
| *** bandini has quit IRC | 02:39 | |
| *** adreznec has joined #openstack-infra | 02:40 | |
| *** bandini has joined #openstack-infra | 02:41 | |
| *** calbers has joined #openstack-infra | 02:41 | |
| *** ericyoung has joined #openstack-infra | 02:41 | |
| *** mrunge has joined #openstack-infra | 02:41 | |
| *** baoli has joined #openstack-infra | 02:46 | |
| *** jappleii__ has joined #openstack-infra | 02:54 | |
| *** threestrands_ has quit IRC | 02:57 | |
| *** nicolasbock_ has quit IRC | 03:00 | |
| *** nicolasbock has quit IRC | 03:01 | |
| *** edmondsw has joined #openstack-infra | 03:04 | |
| *** edmondsw has quit IRC | 03:09 | |
| *** felipemonteiro_ has joined #openstack-infra | 03:12 | |
| *** signed8bit is now known as signed8bit_Zzz | 03:12 | |
| *** ihrachys_ has quit IRC | 03:23 | |
| *** ihrachys has joined #openstack-infra | 03:23 | |
| *** markvoelker has joined #openstack-infra | 03:24 | |
| *** links has joined #openstack-infra | 03:32 | |
| *** mriedem has joined #openstack-infra | 03:35 | |
| *** udesale has joined #openstack-infra | 03:38 | |
| *** udesale has quit IRC | 03:38 | |
| *** mriedem has quit IRC | 03:42 | |
| *** gouthamr has quit IRC | 03:42 | |
| *** udesale has joined #openstack-infra | 03:42 | |
| *** hongbin has quit IRC | 03:45 | |
| *** kiennt26 has quit IRC | 03:51 | |
| *** dbecker has quit IRC | 03:53 | |
| *** baoli has quit IRC | 03:53 | |
| *** iyamahat has joined #openstack-infra | 03:54 | |
| *** markvoelker has quit IRC | 03:57 | |
| *** ykarel has joined #openstack-infra | 04:02 | |
| *** dbecker has joined #openstack-infra | 04:08 | |
| *** Rockyg has joined #openstack-infra | 04:14 | |
| *** yamamoto has joined #openstack-infra | 04:15 | |
| *** bnemec has quit IRC | 04:16 | |
| *** coolsvap has joined #openstack-infra | 04:17 | |
| *** yamahata has joined #openstack-infra | 04:19 | |
| *** jaosorior has joined #openstack-infra | 04:21 | |
| *** asilenkov has quit IRC | 04:22 | |
| *** armax has quit IRC | 04:23 | |
| *** asilenkov has joined #openstack-infra | 04:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move nodes to zuulv3 https://review.openstack.org/509965 | 04:23 |
| AJaeger | infra-root, what do you think of shifting node allocatoin to v3 ^. Looking at current workload, v3 needs some more nodes to catch up ^ | 04:23 |
| *** claudiub has joined #openstack-infra | 04:31 | |
| *** ramishra has quit IRC | 04:35 | |
| *** dhajare has joined #openstack-infra | 04:35 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix path exclusions https://review.openstack.org/509901 | 04:35 |
| *** ramishra has joined #openstack-infra | 04:36 | |
| *** felipemonteiro_ has quit IRC | 04:49 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Remove Zun legacy jobs https://review.openstack.org/509969 | 04:52 |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Remove legacy jobs of Zun https://review.openstack.org/509970 | 04:52 |
| *** edmondsw has joined #openstack-infra | 04:52 | |
| *** markvoelker has joined #openstack-infra | 04:54 | |
| *** udesale has quit IRC | 04:55 | |
| *** edmondsw has quit IRC | 04:57 | |
| *** psachin has joined #openstack-infra | 05:05 | |
| *** esberglu has joined #openstack-infra | 05:09 | |
| *** esberglu has quit IRC | 05:13 | |
| *** udesale has joined #openstack-infra | 05:18 | |
| *** tobiash has quit IRC | 05:27 | |
| *** markvoelker has quit IRC | 05:27 | |
| *** tobiash has joined #openstack-infra | 05:27 | |
| *** jbadiapa has joined #openstack-infra | 05:32 | |
| *** ykarel_ has joined #openstack-infra | 05:38 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove build-openstack-sphinx-docs-infra https://review.openstack.org/509808 | 05:39 |
| *** Diabelko has quit IRC | 05:40 | |
| *** ykarel has quit IRC | 05:41 | |
| *** tumbarka has quit IRC | 05:43 | |
| *** tushar has joined #openstack-infra | 05:43 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add release-openstack-sphinx-docs-infra template https://review.openstack.org/509809 | 05:45 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: zuul v3: Mention translation jobs https://review.openstack.org/509974 | 05:47 |
| *** akscram1 has quit IRC | 05:55 | |
| *** e0ne has joined #openstack-infra | 05:55 | |
| *** akscram1 has joined #openstack-infra | 05:55 | |
| *** sree has joined #openstack-infra | 05:56 | |
| *** spectr has joined #openstack-infra | 05:56 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add project-template for python jobs without constraints https://review.openstack.org/509815 | 05:57 |
| * AJaeger just went over open zuul.yaml changes and gave lots of -1 for too eager migration | 05:57 | |
| *** e0ne has quit IRC | 06:02 | |
| *** hemna_ has joined #openstack-infra | 06:07 | |
| *** hemna__ has quit IRC | 06:11 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: zuul v3: Mention translation jobs https://review.openstack.org/509974 | 06:15 |
| *** clayton has quit IRC | 06:16 | |
| *** ykarel__ has joined #openstack-infra | 06:16 | |
| *** rcernin has joined #openstack-infra | 06:17 | |
| *** clayton has joined #openstack-infra | 06:18 | |
| *** ykarel_ has quit IRC | 06:20 | |
| *** jappleii__ has quit IRC | 06:24 | |
| *** markvoelker has joined #openstack-infra | 06:24 | |
| *** pcaruana has joined #openstack-infra | 06:24 | |
| *** pgadiya has joined #openstack-infra | 06:27 | |
| *** pgadiya has quit IRC | 06:27 | |
| *** iyamahat has quit IRC | 06:30 | |
| *** udesale has quit IRC | 06:31 | |
| *** e0ne has joined #openstack-infra | 06:31 | |
| *** udesale has joined #openstack-infra | 06:33 | |
| *** gridinv has joined #openstack-infra | 06:35 | |
| *** martinkopec has joined #openstack-infra | 06:38 | |
| *** dtantsur|afk has quit IRC | 06:40 | |
| *** dtantsur has joined #openstack-infra | 06:40 | |
| *** gildub has quit IRC | 06:41 | |
| *** rcernin has quit IRC | 06:41 | |
| *** links has quit IRC | 06:42 | |
| *** rcernin has joined #openstack-infra | 06:43 | |
| dirk | AJaeger: well, you should be replaced by a zuul job ;-) | 06:44 |
| dirk | AJaeger: that's the part that strikes me quite a bit, no policy validation implemented anywhere | 06:44 |
| *** e0ne has quit IRC | 06:46 | |
| *** isaacb has joined #openstack-infra | 06:46 | |
| *** mandre is now known as mandre_afk | 06:51 | |
| openstackgerrit | Kazunori Shinohara proposed openstack-infra/project-config master: Add heat-dashboard project https://review.openstack.org/509119 | 06:53 |
| *** vaidy has quit IRC | 06:55 | |
| *** isviridov_away has quit IRC | 06:55 | |
| *** vsaienk0 has joined #openstack-infra | 06:56 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs from Kuryr-libnetwork https://review.openstack.org/509986 | 06:57 |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Remove legacy jobs from Kuryr-libnetwork https://review.openstack.org/509987 | 06:57 |
| *** esberglu has joined #openstack-infra | 06:57 | |
| *** kiennt26 has joined #openstack-infra | 06:57 | |
| *** markvoelker has quit IRC | 06:58 | |
| *** links has joined #openstack-infra | 06:59 | |
| *** jtomasek has joined #openstack-infra | 06:59 | |
| *** esberglu has quit IRC | 07:02 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Remove legacy jobs from Kuryr-libnetwork https://review.openstack.org/509987 | 07:02 |
| *** vaidy has joined #openstack-infra | 07:06 | |
| *** isviridov_away has joined #openstack-infra | 07:08 | |
| *** tmorin has joined #openstack-infra | 07:09 | |
| *** gongysh has joined #openstack-infra | 07:12 | |
| *** eumel8 has joined #openstack-infra | 07:15 | |
| *** aviau has quit IRC | 07:19 | |
| *** dizquierdo has joined #openstack-infra | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** ccamacho has joined #openstack-infra | 07:20 | |
| *** andreas_s has joined #openstack-infra | 07:21 | |
| *** gongysh has quit IRC | 07:23 | |
| *** tesseract has joined #openstack-infra | 07:25 | |
| AJaeger | dirk: Changes welcome ;) | 07:27 |
| *** eranrom has joined #openstack-infra | 07:29 | |
| *** isaacb has quit IRC | 07:30 | |
| AJaeger | SamYaple: could you readd merge-template to project-config for loci, please? See https://docs.openstack.org/infra/manual/zuulv3.html#what-not-to-convert | 07:37 |
| *** coolsvap has quit IRC | 07:43 | |
| *** links has quit IRC | 07:46 | |
| *** ykarel__ is now known as ykarel | 07:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove mistral legacy jobs https://review.openstack.org/510008 | 07:47 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove mistral legacy jobs https://review.openstack.org/510009 | 07:47 |
| *** e0ne has joined #openstack-infra | 07:47 | |
| *** armaan has joined #openstack-infra | 07:49 | |
| *** hashar has joined #openstack-infra | 07:49 | |
| *** jpena|off is now known as jpena | 07:50 | |
| *** esberglu has joined #openstack-infra | 07:52 | |
| *** esberglu has quit IRC | 07:52 | |
| *** esberglu has joined #openstack-infra | 07:53 | |
| *** markvoelker has joined #openstack-infra | 07:54 | |
| *** florianf has joined #openstack-infra | 07:57 | |
| *** esberglu has quit IRC | 07:57 | |
| *** links has joined #openstack-infra | 07:59 | |
| eumel8 | morning | 08:04 |
| eumel8 | AJaeger: can you advise me? When I setup now new translation job for new project, I'm using the old way in zuul/jenkins or the new one for zuul-v3? | 08:04 |
| eumel8 | or you want the new project as test for the new jobs, maybe :) | 08:04 |
| *** Rockyg has quit IRC | 08:06 | |
| *** egonzalez has joined #openstack-infra | 08:06 | |
| AJaeger | morning, eumel8. We could use the new way only - but those do not work yet. Best work with jlk on the new setup and then let's use yours as test... | 08:08 |
| *** sileht has quit IRC | 08:08 | |
| eumel8 | AJaeger: sounds good. I'm here: https://review.openstack.org/#/c/509943/ and here: https://review.openstack.org/#/c/509937/ | 08:10 |
| AJaeger | eumel8: let's set that up *without* translations first and merge - and then add translations | 08:12 |
| *** dbecker has quit IRC | 08:12 | |
| AJaeger | eumel8: you want content first before we test with translations | 08:12 |
| *** dbecker has joined #openstack-infra | 08:12 | |
| eumel8 | ok | 08:13 |
| eumel8 | I will instruct Mike | 08:13 |
| *** armaan has quit IRC | 08:16 | |
| *** mandre_afk is now known as mandre | 08:17 | |
| *** jascott1 has quit IRC | 08:19 | |
| *** jascott1 has joined #openstack-infra | 08:19 | |
| *** jascott1 has quit IRC | 08:24 | |
| *** bauzas is now known as bauwser | 08:26 | |
| *** markvoelker has quit IRC | 08:28 | |
| *** edmondsw has joined #openstack-infra | 08:28 | |
| *** lucas-afk is now known as lucasagomes | 08:29 | |
| openstackgerrit | Pratik Shah proposed openstack-infra/openstack-zuul-jobs master: Added openstack/requirements project to required_projects https://review.openstack.org/510015 | 08:29 |
| *** edmondsw has quit IRC | 08:33 | |
| *** psachin has quit IRC | 08:34 | |
| *** dizquierdo has quit IRC | 08:36 | |
| *** electrofelix has joined #openstack-infra | 08:38 | |
| *** geguileo has left #openstack-infra | 08:44 | |
| *** derekh has joined #openstack-infra | 08:46 | |
| *** ykarel is now known as ykarel|lunch | 09:12 | |
| *** tosky has joined #openstack-infra | 09:15 | |
| *** kiennt26 has quit IRC | 09:15 | |
| *** sambetts_ is now known as sambetts | 09:16 | |
| *** ociuhandu has joined #openstack-infra | 09:18 | |
| *** psachin has joined #openstack-infra | 09:18 | |
| *** ociuhandu has quit IRC | 09:21 | |
| *** jpich has joined #openstack-infra | 09:21 | |
| *** sileht has joined #openstack-infra | 09:22 | |
| *** sileht has quit IRC | 09:24 | |
| *** markvoelker has joined #openstack-infra | 09:25 | |
| *** spectr has quit IRC | 09:26 | |
| openstackgerrit | Pratik Shah proposed openstack-infra/openstack-zuul-jobs master: Added openstack/requirements project to required_projects for Omni project https://review.openstack.org/510015 | 09:28 |
| *** dizquierdo has joined #openstack-infra | 09:30 | |
| *** spectr has joined #openstack-infra | 09:34 | |
| *** spectr has quit IRC | 09:34 | |
| *** spectr has joined #openstack-infra | 09:36 | |
| *** spectr has quit IRC | 09:36 | |
| *** spectr has joined #openstack-infra | 09:37 | |
| *** spectr has quit IRC | 09:38 | |
| *** spectr has joined #openstack-infra | 09:40 | |
| AJaeger | fungi, clarkb, jeblair, pabelanger, zuul v3 currently has a queue of 1995/85 events and is swapping - 12 GB currently | 09:40 |
| *** spectr has quit IRC | 09:40 | |
| *** spectr has joined #openstack-infra | 09:41 | |
| *** ykarel|lunch is now known as ykarel | 09:42 | |
| *** spectr has quit IRC | 09:43 | |
| *** spectr has joined #openstack-infra | 09:43 | |
| *** panda|off is now known as panda | 09:50 | |
| *** martinkopec has quit IRC | 09:51 | |
| *** spectr has quit IRC | 09:56 | |
| *** spectr has joined #openstack-infra | 09:57 | |
| *** spectr has quit IRC | 09:57 | |
| *** spectr has joined #openstack-infra | 09:58 | |
| *** markvoelker has quit IRC | 09:58 | |
| *** pbourke has quit IRC | 10:01 | |
| *** pbourke has joined #openstack-infra | 10:03 | |
| *** stephenfin is now known as finucannot | 10:05 | |
| *** sileht has joined #openstack-infra | 10:05 | |
| *** vsaienk0 has quit IRC | 10:15 | |
| frickler | infra-root: AJaeger: I'm also seeing an immediate proxy error now for http://zuulv3.openstack.org/status.json | 10:15 |
| frickler | also swap activity seems to have mostly stopped around 09:40 utc, so something might have broken then | 10:16 |
| *** ykarel_ has joined #openstack-infra | 10:16 | |
| *** edmondsw has joined #openstack-infra | 10:16 | |
| *** ykarel has quit IRC | 10:18 | |
| *** vsaienk0 has joined #openstack-infra | 10:18 | |
| frickler | increasing number of tcp resets would match that behaviour | 10:18 |
| *** edmondsw has quit IRC | 10:21 | |
| *** ykarel__ has joined #openstack-infra | 10:21 | |
| *** ykarel__ is now known as ykarel | 10:21 | |
| *** ykarel_ has quit IRC | 10:22 | |
| *** jascott1 has joined #openstack-infra | 10:23 | |
| *** andreas_s has quit IRC | 10:24 | |
| *** andreas_s has joined #openstack-infra | 10:24 | |
| *** seanhandley has left #openstack-infra | 10:27 | |
| *** sdague has joined #openstack-infra | 10:28 | |
| *** andreas_s has quit IRC | 10:29 | |
| *** Dinesh_Bhor has quit IRC | 10:35 | |
| AJaeger | frickler: indeed - let's wait what the admins figure out | 10:39 |
| *** andreas_s has joined #openstack-infra | 10:40 | |
| *** armaan has joined #openstack-infra | 10:41 | |
| *** quite has quit IRC | 10:42 | |
| *** tikitavi has joined #openstack-infra | 10:52 | |
| *** yamamoto has quit IRC | 10:53 | |
| *** cuongnv has quit IRC | 10:54 | |
| *** yamamoto has joined #openstack-infra | 10:54 | |
| *** yamamoto has quit IRC | 10:54 | |
| *** markvoelker has joined #openstack-infra | 10:56 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM. test ansible 2.4 with tht https://review.openstack.org/510061 | 10:57 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM. test ansible 2.4 with image build https://review.openstack.org/510062 | 10:59 |
| *** ijw has joined #openstack-infra | 11:00 | |
| *** armaan has quit IRC | 11:00 | |
| *** armaan has joined #openstack-infra | 11:00 | |
| *** jkilpatr has joined #openstack-infra | 11:03 | |
| *** armaan has quit IRC | 11:04 | |
| *** ijw has quit IRC | 11:04 | |
| *** armaan has joined #openstack-infra | 11:04 | |
| *** namnh has quit IRC | 11:06 | |
| *** nicolasbock has joined #openstack-infra | 11:10 | |
| *** nicolasbock_ has joined #openstack-infra | 11:10 | |
| *** quite has joined #openstack-infra | 11:13 | |
| *** nicolasbock_ has quit IRC | 11:15 | |
| *** armaan has quit IRC | 11:15 | |
| *** nicolasbock has quit IRC | 11:15 | |
| *** armaan has joined #openstack-infra | 11:15 | |
| *** jkilpatr has quit IRC | 11:16 | |
| *** gildub has joined #openstack-infra | 11:17 | |
| *** jkilpatr has joined #openstack-infra | 11:18 | |
| *** dave-mccowan has joined #openstack-infra | 11:20 | |
| *** armaan_ has joined #openstack-infra | 11:20 | |
| *** armaan has quit IRC | 11:24 | |
| *** yamamoto has joined #openstack-infra | 11:25 | |
| *** eranrom has quit IRC | 11:25 | |
| *** nicolasbock has joined #openstack-infra | 11:27 | |
| *** nicolasbock_ has joined #openstack-infra | 11:27 | |
| *** nicolasbock_ has quit IRC | 11:29 | |
| *** nicolasbock has quit IRC | 11:29 | |
| *** markvoelker has quit IRC | 11:29 | |
| *** armaan has joined #openstack-infra | 11:42 | |
| *** baoli has joined #openstack-infra | 11:43 | |
| *** ldnunes has joined #openstack-infra | 11:44 | |
| tikitavi | Hi, it’s me again and the problem with starting instances in stable/ocata and stable/pike. There is a problem with installing packages as we see. http://logs.openstack.org/66/505866/13/check/gate-functional-neutron-dsvm-ec2api-ubuntu-xenial/d6be21b/logs/screen-n-super-cond.txt.gz?level=ERROR (ERROR oslo_messaging.rpc.server IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/six-1.10.0 | 11:45 |
| tikitavi | No review in stable/ocata and stable/pike can pass (even “update requirements”) because of not starting instance. (https://review.openstack.org/#/c/505866/) There is no such problems with master branch. We have no problems locally. We haven’t problems earlier. | 11:45 |
| *** armaan_ has quit IRC | 11:46 | |
| *** gildub has quit IRC | 11:47 | |
| AJaeger | tikitavi: that's starting an instance for testing - best ask on #openstack-qa for help with that - or ask the nova team on #openstack-nova | 11:47 |
| *** baoli has quit IRC | 11:48 | |
| tikitavi | AJaeger: nova team confirms that it isn't problem of nova code, but it is installation problem | 11:49 |
| tikitavi | AJaeger: #openstack-qa is just ignoring me | 11:50 |
| AJaeger | tikitavi: then best to discuss with qa team, hope somebody can help you with that better. | 11:50 |
| AJaeger | tikitavi: then try it at another time... | 11:52 |
| AJaeger | mordred, pabelanger, jeblair : are the playbooks also sharing a global namespace like jobs do? Meaning, can I name - in two different repos - playbooks both "tempest" and will that work? | 11:54 |
| *** lucasagomes is now known as lucas-hungry | 11:56 | |
| eumel8 | office hours in #openstack-qa is in west coast time | 11:57 |
| *** dfflanders has quit IRC | 11:58 | |
| *** spectr has quit IRC | 12:02 | |
| *** spectr has joined #openstack-infra | 12:03 | |
| *** dprince has joined #openstack-infra | 12:03 | |
| *** edmondsw has joined #openstack-infra | 12:05 | |
| *** jpena is now known as jpena|lunch | 12:05 | |
| evrardjp | I am tracking some inconsistencies on the openstack-health dashboard for my job, but I have no knowledge of openstack-health... | 12:09 |
| evrardjp | The API of health gives me, when I query this: http://health.openstack.org/runs/key/build_name/periodic-openstack-ansible-deploy-ceph-master-ubuntu-xenial/recent/detail | 12:09 |
| *** edmondsw has quit IRC | 12:09 | |
| evrardjp | 2 fails (uuid c8ce3edd-b2cb-461d-a998-fc3961291115 and 964cc9f5-3978-4735-bbd4-448134b83397 ) and then a pass. | 12:10 |
| *** trown|outtypewww is now known as trown | 12:10 | |
| evrardjp | but when I check my log for the equivalent run, I see success | 12:10 |
| evrardjp | (I check with the last line of my log in here: http://logs.openstack.org/periodic/periodic-openstack-ansible-deploy-ceph-master-ubuntu-xenial/4d5b78e/console.html ) for the first uuid | 12:10 |
| evrardjp | How can I link the API and its data vs the logs on logs.openstack.org? | 12:11 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Do not satisfy min-ready requests if at capacity https://review.openstack.org/510085 | 12:11 |
| *** baoli has joined #openstack-infra | 12:12 | |
| *** d0ugal has quit IRC | 12:15 | |
| *** tpsilva has joined #openstack-infra | 12:19 | |
| openstackgerrit | Emma Foley proposed openstack-infra/project-config master: Removing collectd-ceilometer-plugin definition from zuul.d/projects.yaml https://review.openstack.org/510086 | 12:21 |
| *** nicolasbock has joined #openstack-infra | 12:22 | |
| openstackgerrit | Emma Foley proposed openstack-infra/openstack-zuul-jobs master: Removes legacy collectd-ceilometer-plugin jobs https://review.openstack.org/510087 | 12:23 |
| *** esberglu has joined #openstack-infra | 12:24 | |
| *** esberglu has quit IRC | 12:24 | |
| *** esberglu has joined #openstack-infra | 12:24 | |
| *** esberglu has quit IRC | 12:24 | |
| *** markvoelker has joined #openstack-infra | 12:26 | |
| *** spectr has quit IRC | 12:28 | |
| *** signed8bit_Zzz is now known as signed8bit | 12:30 | |
| *** markvoelker has quit IRC | 12:31 | |
| *** markvoelker has joined #openstack-infra | 12:31 | |
| *** baoli has quit IRC | 12:32 | |
| *** jaypipes is now known as leakypipes | 12:33 | |
| *** spectr has joined #openstack-infra | 12:39 | |
| *** nicolasbock has quit IRC | 12:39 | |
| *** erlon has joined #openstack-infra | 12:41 | |
| *** hashar has quit IRC | 12:42 | |
| *** alexchadin has joined #openstack-infra | 12:45 | |
| *** baoli has joined #openstack-infra | 12:49 | |
| *** nicolasbock has joined #openstack-infra | 12:51 | |
| *** hashar has joined #openstack-infra | 12:53 | |
| *** kgiusti has joined #openstack-infra | 12:55 | |
| openstackgerrit | Emma Foley proposed openstack-infra/project-config master: Removing collectd-ceilometer-plugin definition from zuul.d/projects.yaml https://review.openstack.org/510086 | 12:56 |
| *** dizquierdo has quit IRC | 12:57 | |
| *** bobh has joined #openstack-infra | 12:58 | |
| *** jpena|lunch is now known as jpena | 13:00 | |
| *** pblaho1 has joined #openstack-infra | 13:01 | |
| *** pblaho has quit IRC | 13:03 | |
| *** psachin has quit IRC | 13:05 | |
| *** nicolasbock has quit IRC | 13:05 | |
| *** udesale has quit IRC | 13:05 | |
| *** nicolasbock has joined #openstack-infra | 13:05 | |
| *** eharney has joined #openstack-infra | 13:12 | |
| *** bnemec has joined #openstack-infra | 13:12 | |
| *** priteau has joined #openstack-infra | 13:13 | |
| *** lbragstad has joined #openstack-infra | 13:15 | |
| priteau | Hello. Can anyone access https://www.openstack.org/analytics? After logging in, I only see a page displaying "That page is secured. Enter your credentials below and we will send you right along." with an empty box underneath and a "Log in as someone else" button. | 13:15 |
| pabelanger | Hmm, I am not able to ssh into zuulv3.o.o | 13:15 |
| pabelanger | ah | 13:16 |
| pabelanger | I think we are out of disk space | 13:16 |
| pabelanger | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=557 | 13:16 |
| *** kgiusti has quit IRC | 13:17 | |
| pabelanger | mordred: jeblair: AJaeger: ^ shows the issue, no more storage on / | 13:17 |
| *** camunoz has joined #openstack-infra | 13:17 | |
| pabelanger | for zuulv3 | 13:17 |
| pabelanger | AJaeger: playbooks should be fine, as long as job name is different | 13:17 |
| *** mat128 has joined #openstack-infra | 13:17 | |
| *** kgiusti has joined #openstack-infra | 13:18 | |
| fungi | pabelanger: excessive debug logging, or something else? | 13:20 |
| fungi | oh, you can't ssh in | 13:20 |
| *** yamamoto has quit IRC | 13:21 | |
| pabelanger | fungi: ya, likely | 13:21 |
| pabelanger | logs | 13:21 |
| fungi | pabelanger: shall i call nova reboot on it? | 13:21 |
| fungi | so we can get into it? | 13:21 |
| pabelanger | sure | 13:22 |
| fungi | i figure sshd is failing to write to the disk | 13:22 |
| pabelanger | hopefully we clean up some tmp space | 13:22 |
| fungi | pabelanger: i can log into it again | 13:28 |
| *** dansmith is now known as superdan | 13:28 | |
| pabelanger | fungi: me too | 13:28 |
| fungi | df reports a fair amount of available space on / | 13:28 |
| pabelanger | Yup | 13:29 |
| pabelanger | 26gb /var/log/zuul | 13:29 |
| pabelanger | on a 40GB drive | 13:29 |
| fungi | yeah, that's the bulkof it | 13:30 |
| *** signed8bit is now known as signed8bit_Zzz | 13:30 | |
| *** signed8bit_Zzz is now known as signed8bit | 13:30 | |
| *** srobert has joined #openstack-infra | 13:30 | |
| fungi | with zuul.log and debug.log being nearly the same size 11-12GiB each | 13:30 |
| *** srobert has quit IRC | 13:30 | |
| pabelanger | ya, I see a lot of zookeeper exceptions in both | 13:31 |
| *** srobert has joined #openstack-infra | 13:31 | |
| pabelanger | http://paste.openstack.org/show/622839/ for example | 13:32 |
| fungi | speaking of, since we've got a scheduler restart now (whether we wanted one or not, but we did want one anyway) we should make sure we also restart zk on nodepool.o.o to pick up the config change for snapshot autopurge | 13:32 |
| fungi | from that traceback, i wonder whether we hit a thread limit | 13:33 |
| pabelanger | ya, maybe stop zuulv3 again so we can purge debug.log / debug-scheduler.log again | 13:34 |
| pabelanger | also, I think our log levels might be a little off | 13:34 |
| pabelanger | 2017-10-06 13:31:44,035 INFO zuul.nodepool: Returning nodeset <NodeSet OrderedDict([('ubuntu-xenial', <Node 0000144941 ubuntu-xenial:ubuntu-xenial>)])OrderedDict()> | 13:34 |
| pabelanger | seems more like a debug statement then INFO | 13:34 |
| *** spectr has quit IRC | 13:35 | |
| *** yamamoto has joined #openstack-infra | 13:35 | |
| *** Goneri has joined #openstack-infra | 13:35 | |
| *** lbragstad has quit IRC | 13:36 | |
| *** lbragstad has joined #openstack-infra | 13:36 | |
| *** spectr has joined #openstack-infra | 13:36 | |
| openstackgerrit | Joe D'Andrea proposed openstack-infra/irc-meetings master: Change Valet team meeting name https://review.openstack.org/510116 | 13:37 |
| *** trown is now known as trown|brb | 13:37 | |
| *** mriedem has joined #openstack-infra | 13:39 | |
| *** gouthamr has joined #openstack-infra | 13:39 | |
| fungi | should we just force a round of log rotation/compression (maybe remove some earlier logs to free up space for the copy gzip performs)? | 13:40 |
| *** pblaho has joined #openstack-infra | 13:40 | |
| fungi | i worry about removing the most recent logs in case we want to mine them for details | 13:41 |
| *** ykarel is now known as ykarel|afk | 13:41 | |
| pabelanger | ya, that works too | 13:42 |
| fungi | like, i can start by removing compressed logs from /var/log/zuul over a week old and see if that's enough to successfully compress the latest logs | 13:42 |
| *** pblaho1 has quit IRC | 13:44 | |
| *** shardy has quit IRC | 13:44 | |
| *** lucas-hungry is now known as lucasagomes | 13:44 | |
| fungi | okay, removed logs older than a week and trying to logrotate --force now | 13:46 |
| clarkb | I dont think logrotate rotates those files, python logging foes | 13:46 |
| fungi | oh, well i guess i can rename in a loop and gzip the latest | 13:48 |
| *** links has quit IRC | 13:48 | |
| fungi | we'll see when logrotate finishes. if nothing else it'll free up a smidge more space elsewhere in /var/log | 13:48 |
| fungi | but yeah, given there was a fair amount of free space when i logged in after the reboot, i have a feeling it was log rotation which ran out of space trying to compress one of the zuul logs | 13:52 |
| fungi | clarkb: and yes, you're right logrotate itself doesn't seem to do the rotation for zuul logs so i'll do those myself | 13:52 |
| fungi | but it did free up a ton of space elsewhere | 13:53 |
| *** edmondsw has joined #openstack-infra | 13:53 | |
| fungi | no, i take that back, it did rotate them | 13:53 |
| fungi | the fact that there are new zuul logs threw me off | 13:54 |
| fungi | we have zuul configured to start at boot now i guess? | 13:54 |
| fungi | oh, or maybe we always did | 13:54 |
| fungi | anyway, i guess jeblair will want to restart it again with some additional patches here in a bit, so i guess we can restart zk on nodepool.o.o at that point | 13:55 |
| *** trown|brb is now known as trown | 13:55 | |
| *** jcoufal has joined #openstack-infra | 13:55 | |
| fungi | AJaeger: thanks for rechecking 509845 already. i just went to check whether it ever reported while i was in sleep mode | 13:56 |
| fungi | i guess it never did | 13:56 |
| *** edmondsw has quit IRC | 13:57 | |
| * fungi needs to step away for a couple minutes, but will brb | 13:57 | |
| *** wolverineav has joined #openstack-infra | 13:58 | |
| *** sree has quit IRC | 13:59 | |
| AJaeger | fungi, it never did ;( It was the first one I rechecked ;) | 14:00 |
| *** sree has joined #openstack-infra | 14:00 | |
| *** d0ugal has joined #openstack-infra | 14:00 | |
| *** Guest14475 has quit IRC | 14:01 | |
| AJaeger | fungi, pabelanger, jeblair, clarkb : Should we change our node allocation from 80:20 to 70:30 - see https://review.openstack.org/#/c/509965/ . We had jobs for over 24 hours in it... | 14:01 |
| *** iyamahat has joined #openstack-infra | 14:01 | |
| pabelanger | AJaeger: I think that is okay, I get the feeling were making progress on zuulv3 effort dispite the backlog | 14:02 |
| *** hongbin has joined #openstack-infra | 14:03 | |
| *** srobert has quit IRC | 14:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix neutron-dynamic-routing python jobs https://review.openstack.org/509726 | 14:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Update specs-site and infra-index publication jobs https://review.openstack.org/509882 | 14:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Add the run playbook for releasenotes and fix afs stamping https://review.openstack.org/509843 | 14:04 |
| AJaeger | pabelanger: it was impossible to get a +1 on normal projects - like on 509845. | 14:04 |
| *** sree has quit IRC | 14:05 | |
| AJaeger | Projects cannot in earnest continue with zuulv3 migration IMHO | 14:05 |
| *** slaweq has quit IRC | 14:05 | |
| frickler | how can we replace review.o.o/gitweb links? only with https://git.openstack.org/cgit/ or is there some variant still hosted directly on review.o.o? looking at https://github.com/openstack-infra/zuul/blob/master/zuul/connection/gerrit.py#L460 it seems difficult to add a different host as target there | 14:07 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Always build releasenotes from master https://review.openstack.org/509872 | 14:07 |
| pabelanger | AJaeger: Right, but I think the firs step was to stablize zuulv3, which seems to be going well. I believe we are having another meeting this morning once jeblair is online to review etherpad for zuulv3-issue, we likey can bring up the review then | 14:07 |
| *** hashar has quit IRC | 14:09 | |
| AJaeger | pabelanger: yes, stabilizing is improving nicely! | 14:09 |
| AJaeger | pabelanger: yeah, let's bring it up then... | 14:09 |
| pabelanger | AJaeger: I agree projects should be experimenting with changes, but also think we should try and roll zuulv3 back into production again. So, what every way gets us that, I'm happy to support :) | 14:10 |
| *** bnemec is now known as beekneemech | 14:11 | |
| *** spectr has quit IRC | 14:11 | |
| AJaeger | pabelanger: yes, we made good progress with zuul v3! | 14:12 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch infra projects to using docs templates https://review.openstack.org/509820 | 14:14 |
| *** hashar has joined #openstack-infra | 14:14 | |
| *** alexchadin has quit IRC | 14:15 | |
| *** d0ugal has quit IRC | 14:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch infra repos to not use constraints https://review.openstack.org/509821 | 14:16 |
| *** gouthamr has quit IRC | 14:16 | |
| jdandrea | I have an IRC channel change in irc-meetings and system-config. Would someone be available to help with this today? | 14:17 |
| *** ykarel|afk has quit IRC | 14:17 | |
| jdandrea | Relevant changes: https://review.openstack.org/#/c/510116 and https://review.openstack.org/#/c/508924 | 14:17 |
| *** Hal has joined #openstack-infra | 14:17 | |
| *** Hal is now known as Guest93293 | 14:17 | |
| mnaser | i think one of the bigger zuulv3 issues was the fact that it was hard to fix your jobs | 14:18 |
| mnaser | so it was kinda hard both to add new changes (cause your jobs might have been broken) and to fix your jobs (because of the reconfig stuff) | 14:19 |
| mnaser | on the second attempt to revert i think things will be muuuch better and we can iterate fixes much faster | 14:19 |
| mnaser | (also, i think infra-check should stay, it helps for quicker iterations) | 14:19 |
| *** felipemonteiro_ has joined #openstack-infra | 14:21 | |
| pabelanger | I've taken ze09.o.o and ze10.o.o online to fix the hostname issue, it is preventing zuul_stream from working | 14:21 |
| AJaeger | jdandrea: please do not add me to any reviews, I'll review on my own - and especially it won't help with repos that I'm not core on... | 14:21 |
| jdandrea | AJaeger Ok! Apologies. | 14:21 |
| *** alexchadin has joined #openstack-infra | 14:22 | |
| jdandrea | I can remove them all. I may be doing that wrong... | 14:22 |
| *** felipemonteiro__ has joined #openstack-infra | 14:22 | |
| AJaeger | jdandrea: no need to change anything now, just for the future... | 14:22 |
| jdandrea | *nod* | 14:22 |
| jdandrea | Will do. | 14:22 |
| *** alexchadin has quit IRC | 14:23 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Properly format messages coming out of emitPlaybookBanner https://review.openstack.org/510135 | 14:23 |
| andreas_s | sdague: in the ML you mentioned that I should update the css stylesheet that the zkvm ci is using...where exactly do I find them? In zuuls layout.yaml? | 14:23 |
| pabelanger | okay, ze09.o.o and ze10.o.o back online, with propelry hostnames | 14:24 |
| andreas_s | (didn't find it there...) | 14:25 |
| *** felipemonteiro_ has quit IRC | 14:26 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Properly format messages coming out of emitPlaybookBanner https://review.openstack.org/510135 | 14:26 |
| *** armax has joined #openstack-infra | 14:26 | |
| pabelanger | jeblair: I have errands to run this afternoon in about 1 hrs times, when your ready, happy to review https://etherpad.openstack.org/p/zuulv3-issues again | 14:28 |
| *** srobert has joined #openstack-infra | 14:29 | |
| AJaeger | mmh, something interesting: Our infra-gates are entered when either zuul or jenkins gave a +1 - was that really the intention? Shouldn't it be only zuul? | 14:30 |
| *** tikitavi has left #openstack-infra | 14:31 | |
| *** d0ugal has joined #openstack-infra | 14:32 | |
| sdague | andreas_s: it's not a stylesheet, it's how you present the markup | 14:32 |
| sdague | I don't actually know where it's adjusted in zuul, sorry | 14:32 |
| andreas_s | sdague: ok. I configured in zuul a success and a faiure message, but this is jus plain text. will continue looking... | 14:34 |
| pabelanger | AJaeger: I'd expect only zuul, but maybe mordred knows more | 14:34 |
| AJaeger | infra-root, any ideas why project-config-irc-access gets an aborted for several times now? | 14:35 |
| AJaeger | See https://review.openstack.org/509886 | 14:35 |
| AJaeger | Unfortunately this means there are no log files ;( | 14:35 |
| * mordred waves | 14:36 | |
| * AJaeger waves back | 14:36 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Do not satisfy min-ready requests if at capacity https://review.openstack.org/510085 | 14:36 |
| jeblair | sdague, andreas_s: zuul doesn't use html in gerrit comments. as long as you have zuul v2 configured with job_name_in_report=true in the config it should be fine. that's what puppet-openstackci does by default. | 14:36 |
| mordred | pabelanger: gah - did we fall prey to the hostname issue AGAIN? sigh | 14:37 |
| andreas_s | jeblair: ok, let me check if this is set... | 14:37 |
| mnaser | are hostnames getting mucked on reboot? | 14:38 |
| AJaeger | mordred: did you see my question on infra-gate pipeline: should it really be (jenkins|zuul) - and not require a +1 by zuul only | 14:38 |
| mnaser | because... if they are.. we've solved this issue for our customers in the past... (yum|apt-get) remove cloud-init | 14:38 |
| mnaser | :p | 14:38 |
| fungi | AJaeger: there's a good chance the bug Shrews found was causing us not to actually utilize all 20% we allocated to v3 | 14:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Align other tox-based project-config publish jobs https://review.openstack.org/509886 | 14:40 |
| openstackgerrit | Merged openstack-infra/project-config master: Update publish-service-types to work like other jobs https://review.openstack.org/509887 | 14:40 |
| openstackgerrit | Merged openstack-infra/project-config master: Add some allowed-projects restrictions to publish jobs https://review.openstack.org/509906 | 14:40 |
| *** beekneemech has quit IRC | 14:40 | |
| AJaeger | fungi: I see. We can wait a day and check again how it works out | 14:40 |
| mnaser | is there some sort of keypair that all nodepool VMs get installed with? the reason i ask is git clone-ing over http has been very unreliable from github from some modules, i'd like to switch to ssh (which seems to recover better and be more reliable) .. but obviously need a keypair to do that | 14:40 |
| Shrews | fungi: it had 3 providers wedged, so, yeah | 14:40 |
| mordred | AJaeger: it's a holdover from the original cutover - we made gate repsond to both jenkins and zuul so that old jenkins votes would not need to be re-tested | 14:40 |
| Shrews | fungi: 4, actually | 14:40 |
| mordred | AJaeger: we could probably remove it from infra-gate | 14:40 |
| AJaeger | mordred: yes, I suggest to remove it. Shall I sent a patch? | 14:41 |
| mordred | AJaeger: sure! | 14:41 |
| andreas_s | jeblair, sdague: I guess this is the issue... it's set to false. Will try to update... | 14:41 |
| mordred | mnaser: unfortunately no - we actually create a per-build keypair for each build | 14:41 |
| jeblair | also, the goal here was not to facilitate projects moving to v3 -- the goal is to fix the problems with zuul and the auto-migrated jobs. work on anything that isn't broken isn't a priority. | 14:41 |
| jeblair | AJaeger: ^ | 14:42 |
| fungi | mnaser: hostnames are only being set incorrectly on first boot. we think there may be a race between our ansible fixing them and cloud-init breaking them | 14:42 |
| *** bnemec has joined #openstack-infra | 14:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Only allow zuul user for infra-gate https://review.openstack.org/510142 | 14:43 |
| mnaser | fungi: there's a few solutions to get the right hostname from the get go nowadays but that depends on the release of nova you're using so i can imagine that might be a bit more difficult | 14:43 |
| jeblair | AJaeger: (that's in response to the discussion about node allocation) | 14:43 |
| AJaeger | jeblair: yes, not a priority - but projects want to fix them and that's what we told them as well | 14:43 |
| mordred | AJaeger: looking at project-config-irc-access issue | 14:43 |
| AJaeger | mordred: it resolved itself by magic ;) | 14:44 |
| mordred | AJaeger: oh - that's going ot have been the hostname issue | 14:44 |
| AJaeger | fungi, jeblair, so shall I abandon my node allocation change? | 14:44 |
| mordred | AJaeger: in the ABORTED message you can see: finger://ze09/748bd945125e4ec992ff1a67366f5295 ... and ze09 isn't ze09.openstack.org | 14:45 |
| AJaeger | mordred: ah! | 14:45 |
| pabelanger | AJaeger: mordred: Ya, that would be me stopping the executor | 14:45 |
| pabelanger | It seems we no longer enqueue aborted jobs | 14:45 |
| AJaeger | mordred: do you want to rebase https://review.openstack.org/#/c/509888 to resolve merge conflict? Most other changes are in... | 14:46 |
| jeblair | pabelanger: yep, that bug is so long-standing it's even still sitting on the storyboard pre-migration page. | 14:46 |
| pabelanger | yah | 14:46 |
| mordred | AJaeger: I do! | 14:47 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Do not satisfy min-ready requests if at capacity https://review.openstack.org/510085 | 14:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Publish docs for storyboard in post https://review.openstack.org/509888 | 14:48 |
| mordred | AJaeger: I think https://review.openstack.org/#/c/509855/ can go in now too, yes? | 14:48 |
| AJaeger | no, requirements change is not in - it will fail | 14:49 |
| AJaeger | the requirements change never finished to gave +1 - irechecked an hour ago after the zuul v3 restart | 14:49 |
| mordred | ah - yes | 14:49 |
| mordred | I see now | 14:49 |
| *** yamamoto has quit IRC | 14:52 | |
| *** yamamoto has joined #openstack-infra | 14:53 | |
| *** yamamoto has quit IRC | 14:53 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Properly format messages coming out of emitPlaybookBanner https://review.openstack.org/510135 | 14:53 |
| *** vhosakot has joined #openstack-infra | 14:55 | |
| mordred | infra-root: I'm going to make a restart-executors playbook if nobody else has yet- and then use it to restart the executors to pick that up ^^ | 14:56 |
| pabelanger | mordred: we have http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/hard_restart_zuul_launchers.yaml which we can like modify | 14:56 |
| mordred | pabelanger: ++ | 14:56 |
| pabelanger | I can work up something here in a moment | 14:57 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Changes for Ansible 2.4 https://review.openstack.org/505354 | 14:57 |
| AJaeger | anybody around for a +2A on mordred's change to fix storyboard publishing, please? https://review.openstack.org/#/c/509888/ | 14:57 |
| jeblair | mordred: there's a restart launchers playbook for v2 | 14:57 |
| jeblair | ah pabelanger pointed it out | 14:58 |
| dmsimard | pabelanger: wow, is that how executors are reloaded ? stop it, wait until all processes end and then start ? | 14:58 |
| jeblair | dmsimard: what else would it be? | 14:58 |
| AJaeger | mordred: once 509888 is in, your stacks from yesterday are all in (with exception of the requirements one that we just discussed) | 14:58 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Changes for Ansible 2.4 https://review.openstack.org/505354 | 14:59 |
| *** dizquierdo has joined #openstack-infra | 15:00 | |
| *** egonzalez has quit IRC | 15:03 | |
| *** spectr has joined #openstack-infra | 15:04 | |
| *** spectr has quit IRC | 15:04 | |
| *** hashar is now known as hasharAway | 15:05 | |
| *** gouthamr has joined #openstack-infra | 15:06 | |
| pabelanger | mordred: I'm going to test with ze10.o.o for the restart playbook | 15:06 |
| *** links has joined #openstack-infra | 15:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add playbooks to restart and to upgrade zuul executors https://review.openstack.org/510152 | 15:07 |
| mordred | pabelanger: ^^ there's two new playbooks for us | 15:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add hard reset for zuul-executors https://review.openstack.org/510155 | 15:08 |
| pabelanger | mordred: ha, here is my attempt^ | 15:08 |
| mordred | pabelanger: we should maybe coordinate better in the future :) | 15:08 |
| pabelanger | looking at yours now | 15:08 |
| *** bnemec has quit IRC | 15:08 | |
| pabelanger | Hmm | 15:08 |
| mordred | so - mine is based on the various amounts of shotgun blasts I've had to do to restart executors fully in the past | 15:09 |
| pabelanger | I made a mistake | 15:09 |
| openstackgerrit | Emma Foley proposed openstack-infra/project-config master: Removing collectd-ceilometer-plugin definition from zuul.d/projects.yaml https://review.openstack.org/510086 | 15:09 |
| dmsimard | jeblair: I'm not sure what else it would be or what I expected :/ | 15:09 |
| mordred | I'm very unhappy with it - but I also think that at the moment not shot-gunning is not reliable | 15:09 |
| *** bnemec has joined #openstack-infra | 15:09 | |
| pabelanger | Oh, no, i think I did it correct | 15:09 |
| *** mriedem is now known as ronlund | 15:09 | |
| *** hasharAway has quit IRC | 15:10 | |
| pabelanger | mordred: I don't think we can wait for the pid to be delete and service stopped. I've noticed aborts do take time to finish, why I tried using /proc this time | 15:10 |
| mordred | pabelanger: I don't think so - "/var/{{ zuul_pid.contents }}/status" is a bit weird | 15:10 |
| mordred | pabelanger: I tihnk you meant /proc there | 15:10 |
| pabelanger | yes | 15:10 |
| pabelanger | let me fix | 15:10 |
| mordred | pabelanger: but I agree - I just think we need to kill things if service stop does not work | 15:10 |
| pabelanger | yah | 15:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add hard reset for zuul-executors https://review.openstack.org/510155 | 15:11 |
| mordred | (and then seriously, we need to make service stop - or systemctl stop - whatever) be reliable every time) | 15:11 |
| pabelanger | mordred: I'm going to give that a run against ze10 to see if that works | 15:11 |
| mordred | pabelanger: ok. I thikn yours is what it *shoujld* look like | 15:11 |
| mordred | pabelanger: I think mine is what we'll need if we want it to always work today | 15:11 |
| mordred | pabelanger: maybe we call mine hard and yours normal | 15:12 |
| pabelanger | yah | 15:12 |
| jeblair | er | 15:12 |
| jeblair | hard means 'zuul-executor stop' as opposed to 'zuul-executor graceful' | 15:12 |
| pabelanger | mordred: I think your kills might cause ansible-playbooks to not complete however | 15:12 |
| jeblair | you know what | 15:13 |
| jeblair | i'd rather just drop what i'm doing and fix the stops if this is what we're doing | 15:13 |
| jeblair | so i'm going to restart some executors manually | 15:13 |
| jeblair | are we ready to restart them? | 15:13 |
| mordred | jeblair: yes | 15:14 |
| jeblair | okay, i'm going to restart ze01 now and observe what happens | 15:14 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Changes for Ansible 2.4 https://review.openstack.org/505354 | 15:15 |
| jeblair | mordred, pabelanger: everything stopped as expected, except that we leaked 2 ssh-agent processes. | 15:16 |
| pabelanger | Yah, it should stop, just that service stop is not blocking, so we need to wait a little before starting again | 15:17 |
| jeblair | i think the jobs don't get re-launched because we send "ABORTED" as the result instead of NOne. | 15:17 |
| jeblair | pabelanger: yes, and because the service can't delete its own pid file (since its owned by root), that is not a reliable indication either. | 15:17 |
| jeblair | the init script deletes the pid file | 15:17 |
| jeblair | do you think we can have zuul chown the pidfile before it drops privileges? | 15:18 |
| pabelanger | Ya, that should be possible. I think even having the init script do it might be okay | 15:18 |
| openstackgerrit | Emma Foley proposed openstack-infra/project-config master: Removing collectd-ceilometer-plugin definition from zuul.d/projects.yaml https://review.openstack.org/510086 | 15:18 |
| jeblair | pabelanger: it should be entirely under the control of zuul; i don't think the init script should do anything to it | 15:19 |
| *** rcernin has quit IRC | 15:19 | |
| jeblair | okay, i'm going to stop/start ze02 now | 15:19 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Allow core reviewers to forge commits on mogan https://review.openstack.org/510158 | 15:21 |
| fungi | clarkb: ^ we forgot they'd need that | 15:21 |
| jeblair | same thing on ze02. all zuul-executor processes stopped after about 30 seconds. no ansible procs running, 2 leaked ssh-agents | 15:22 |
| pabelanger | jeblair: Ya, we also need to fix our init script for executor now too. We no longer give zuul write perimission to /var/run/zuul-executor | 15:22 |
| pabelanger | since we start it as root, that is the permissions we setup on the run directory | 15:22 |
| clarkb | fungi: +2'd but not approved as I am on a light rail train about to enter a tunnel | 15:23 |
| fungi | clarkb: cool, AJaeger +2'd as well so i'll self-approve | 15:23 |
| fungi | clarkb: g'luck with your rant^H^H^H^Htalk! | 15:23 |
| *** jaosorior has quit IRC | 15:24 | |
| *** andreww has joined #openstack-infra | 15:25 | |
| AJaeger | mordred: the infra index still fails publishing, see http://logs.openstack.org/29/29d9acd1a4899032b888bf1b2c9978f2d2a0e657/infra-post/publish-infra-index/a30f89c/ara/ | 15:25 |
| jeblair | i think we may leak one ssh-agent process on shutdown | 15:25 |
| * AJaeger needs to run some errands, will be back later | 15:26 | |
| pabelanger | Ya, I have to step out myself too | 15:26 |
| pabelanger | back in a few hours | 15:26 |
| mordred | AJaeger: ok - will look in a sec | 15:27 |
| *** xarses_ has quit IRC | 15:28 | |
| *** vsaienk0 has quit IRC | 15:30 | |
| *** eumel8 has quit IRC | 15:31 | |
| ronlund | uh oh, looks like someone needs to kick this? http://status.openstack.org/elastic-recheck/ | 15:32 |
| ronlund | Delay in Elastic Search: Indexing behind by 59 hours | 15:32 |
| ronlund | although we are getting recent hits... | 15:33 |
| *** baoli has quit IRC | 15:34 | |
| *** baoli has joined #openstack-infra | 15:35 | |
| jeblair | pabelanger, mordred: okay i think i have handle on things | 15:37 |
| *** gridinv1 has joined #openstack-infra | 15:37 | |
| jeblair | pabelanger, mordred: i added my findings to the etherpad under an "Issues with Zuul" section | 15:37 |
| jeblair | pabelanger, mordred: basically, it takes about 30s for all the ansible process to stop, but on all 10 executors, they did stop reliably. | 15:38 |
| jeblair | pabelanger, mordred: we can not currently used the presence of the pidfile to detect whether they are stopped because of the permissions issue. | 15:38 |
| jeblair | pabelanger, mordred: i think pabelanger's idea of checking proc should work for the time being | 15:38 |
| jeblair | pabelanger, mordred: later, we can either work on getting the pid ownership right, or, wait until we have the finger multiplexer, then we won't have to start as root anymore | 15:39 |
| jeblair | pabelanger, mordred: but in short, i believe that: stop, wait for process in pidfile to exit, start are all the steps we need, and pabelanger's playbook should work | 15:40 |
| *** vsaienk0 has joined #openstack-infra | 15:40 | |
| jeblair | pabelanger, mordred: there are 2 further bugs in the executor -- we need to fix the ssh-agent process leak -- i think we have a clue there (it seems to be a process that starts right after the stop command). and we need to stop sending "ABORTED" results, so that jobs get restarted automatically when we do a hard stop | 15:41 |
| *** edmondsw has joined #openstack-infra | 15:41 | |
| jeblair | pabelanger, mordred: finally, we should actually implement the graceful stop too | 15:41 |
| *** gridinv1 has quit IRC | 15:41 | |
| *** tmorin has quit IRC | 15:42 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Allow core reviewers to forge commits on mogan https://review.openstack.org/510158 | 15:44 |
| *** lucasagomes is now known as lucas-afk | 15:44 | |
| *** dtantsur is now known as dtantsur|afk | 15:44 | |
| *** ildikov is now known as coffee_cat | 15:45 | |
| *** edmondsw has quit IRC | 15:45 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Remove dead code https://review.openstack.org/510162 | 15:46 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Remove support for command hooks https://review.openstack.org/510163 | 15:46 |
| *** iyamahat has quit IRC | 15:46 | |
| finucannot | dhellmann, mordred: It wasn't a setuptools bug - it was pbr! | 15:46 |
| *** yamahata has quit IRC | 15:46 | |
| AJaeger | fungi, https://review.openstack.org/509845 finished the jobs but gave -1 by zuul to a job aborted. I suggest to still +A - what do you think? | 15:46 |
| finucannot | Damn it, I will get https://review.openstack.org/#/c/475033/ in unchanged or die trying | 15:46 |
| finucannot | :) | 15:47 |
| *** armaan has quit IRC | 15:47 | |
| finucannot | mordred, dhellmann: In case you're scrolling back later, I'm referring to https://review.openstack.org/510163/ | 15:47 |
| *** armaan has joined #openstack-infra | 15:48 | |
| fungi | AJaeger: agreed. just approved it | 15:48 |
| AJaeger | thanks, fungi | 15:48 |
| mordred | jeblair: awesome! thank you | 15:49 |
| *** vsaienk0 has quit IRC | 15:50 | |
| openstackgerrit | Merged openstack/os-testr master: Fix .testr.conf detection: test path follows discover https://review.openstack.org/503877 | 15:50 |
| *** eroux has joined #openstack-infra | 15:52 | |
| *** yamamoto has joined #openstack-infra | 15:53 | |
| *** dhajare has quit IRC | 15:54 | |
| jdandrea | ttx Thank you! (Re: 510116) | 15:54 |
| ttx | jdandrea: you're welcome! | 15:55 |
| jdandrea | ttx Would you be involved in system-config changes perchance (to get statusbot/meetbot listening on a channel)? | 15:56 |
| ttx | jdandrea: not at the level required for you to make fast progress tere, no :) | 15:56 |
| ttx | there* | 15:56 |
| jdandrea | ttx ok, np | 15:56 |
| jdandrea | There may still be zuul3 stuffs happening as well so that takes precedence. | 15:57 |
| *** armaan has quit IRC | 16:00 | |
| *** iyamahat has joined #openstack-infra | 16:01 | |
| *** Apoorva has joined #openstack-infra | 16:01 | |
| *** Apoorva has quit IRC | 16:02 | |
| *** jascott1 has quit IRC | 16:02 | |
| *** yamamoto has quit IRC | 16:02 | |
| *** Apoorva has joined #openstack-infra | 16:02 | |
| *** jascott1 has joined #openstack-infra | 16:02 | |
| *** jascott1 has quit IRC | 16:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Allow core reviewers to forge commits on mogan https://review.openstack.org/510158 | 16:03 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove tox-checkniceness https://review.openstack.org/510172 | 16:05 |
| fungi | jdandrea: now that the release team meeting is over, i believe there's nothing else on the meeting schedule for a while so we should be able to safely merge the meetbot addition | 16:07 |
| jdandrea | fungi Oh! Thank you! | 16:07 |
| fungi | jdandrea: which change was it again? | 16:07 |
| jdandrea | fungi Does 510116 need to merge first? | 16:08 |
| fungi | looking | 16:08 |
| jdandrea | There's https://review.openstack.org/#/c/510116/ and https://review.openstack.org/#/c/508924/ ... want to be sure I did it correctly tho. | 16:08 |
| fungi | nah, 510116 is entirely independent | 16:08 |
| jdandrea | ok | 16:08 |
| *** bnemec has quit IRC | 16:08 | |
| jdandrea | fungi Although, there was a previous (already merged) change here: https://review.openstack.org/#/c/508933/ ... but that didn't result in any changes to the eavesdrop index. That's why I was wondering if the meetbot stuffs was related. | 16:10 |
| *** armaan has joined #openstack-infra | 16:11 | |
| *** pcaruana has quit IRC | 16:13 | |
| fungi | jdandrea: 508933 looks to have worked as far as i can see: http://eavesdrop.openstack.org/#OpenStack_Valet_Team_Meeting | 16:15 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change Valet team meeting name https://review.openstack.org/510116 | 16:15 |
| *** andreas_s has quit IRC | 16:16 | |
| jdandrea | fungi Oh! Well how 'bout that. OH! We changed the name so the next merge will change that too (and there will likely be an orphaned eavesdrop dir after that happens). | 16:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix early processing of merge-pending items on reconfig https://review.openstack.org/509912 | 16:17 |
| *** gongysh has joined #openstack-infra | 16:18 | |
| *** panda is now known as panda|bbl | 16:19 | |
| *** gongysh has quit IRC | 16:21 | |
| jdandrea | fungi Maybe the regen of the eavesdrop home page happens at some periodic UTC time. | 16:21 |
| jdandrea | Also, do I need to add openstackstatus or does someone else do that (?). | 16:22 |
| fungi | jdandrea: it happens in a publication job which runs in our post-merge ci pipeline, but that pipeline gets a low priority so jobs can remain queued there for a while before they get executed | 16:22 |
| jdandrea | Ahh ok | 16:22 |
| *** vsaienk0 has joined #openstack-infra | 16:22 | |
| fungi | jdandrea: your 508924 change adds openstackstatus by way of adding your channel name to the statusbot_channels list in that file. once the change gets applied to our servers through periodic configuration management (hopefully here in the next few minutes) the bot should join your channel automatically | 16:24 |
| jdandrea | fungi *nodnod* Sense makes. | 16:25 |
| openstackgerrit | Merged openstack-infra/system-config master: Add openstack-valet to statusbot and meetbot https://review.openstack.org/508924 | 16:26 |
| *** tosky has quit IRC | 16:26 | |
| *** jpich has quit IRC | 16:27 | |
| *** Guest93293 has quit IRC | 16:29 | |
| jdandrea | fungi ok I see them in the post queue. That'll take a bit. That's ok. I'll monitor as well. Thank you again!! | 16:29 |
| fungi | jdandrea: well, application of 508924 doesn't (yet anyway) happen in a post job. we have a cronjob which attempts to fire ansible every 15 minutes to remotely run puppet apply on all our servers (but it runs under a lock and often takes upwards of 30+ minutes, so it can be as much as an hour before you see it take effect) | 16:31 |
| *** vsaienk0 has quit IRC | 16:32 | |
| *** baoli has quit IRC | 16:34 | |
| *** baoli has joined #openstack-infra | 16:34 | |
| *** Rockyg has joined #openstack-infra | 16:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't load dynamic layout twice unless needed https://review.openstack.org/510180 | 16:36 |
| *** baoli has quit IRC | 16:36 | |
| *** baoli has joined #openstack-infra | 16:37 | |
| *** signed8bit is now known as signed8bit_Zzz | 16:40 | |
| jeblair | mordred, Shrews, pabelanger, fungi, dmsimard: want to catch up around 1700 and go over current tasks? | 16:43 |
| mordred | jeblair: yes - I'll be ina good place to do that then | 16:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't load dynamic layout twice unless needed https://review.openstack.org/510180 | 16:44 |
| dmsimard | So in ~2 hours ? | 16:44 |
| dmsimard | Wait nm | 16:45 |
| mordred | dmsimard: in 15 minutes | 16:45 |
| *** trown is now known as trown|lunch | 16:45 | |
| *** caphrim007_ has quit IRC | 16:46 | |
| Shrews | wfm | 16:46 |
| fungi | jeblair: sounds good, i was just revisiting open changes in the pad | 16:47 |
| *** Swami has joined #openstack-infra | 16:48 | |
| Shrews | jeblair: should i add the nodepool wedge issue to your etherpad? | 16:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Provide error message on malformed job list https://review.openstack.org/510185 | 16:48 |
| jeblair | Shrews: ya | 16:48 |
| *** signed8bit_Zzz is now known as signed8bit | 16:50 | |
| *** signed8bit is now known as signed8bit_Zzz | 16:51 | |
| *** signed8bit_Zzz is now known as signed8bit | 16:51 | |
| *** derekh has quit IRC | 16:52 | |
| *** armaan has quit IRC | 16:53 | |
| mordred | finucannot: we've verified that nobody in openstack is using those command hooks? | 16:53 |
| fungi | mordred: looks that way from the codesearch queries he linked at least | 16:53 |
| *** armaan has joined #openstack-infra | 16:53 | |
| *** baoli has quit IRC | 16:54 | |
| mordred | cool | 16:54 |
| mordred | finucannot: also - in https://review.openstack.org/#/c/475033/8/tox.ini - why remove skipsdist? we normally put that there for efficiency reasons - skipsdist will just install directly from the code instead of making an sdist then installing from the sdist - I don't think it'll matter for reno - just more curious if there was an active issue | 16:56 |
| *** esberglu has joined #openstack-infra | 16:56 | |
| *** esberglu has quit IRC | 16:56 | |
| mordred | fungi, finucannot: https://review.openstack.org/#/c/510163 and https://review.openstack.org/#/c/510162 both lgtm | 16:57 |
| *** bnemec has joined #openstack-infra | 16:59 | |
| jeblair | okay, let's check in on where we are | 17:01 |
| *** ramishra has quit IRC | 17:02 | |
| *** baoli has joined #openstack-infra | 17:02 | |
| jeblair | yesterday i debugged a second memory leak | 17:02 |
| *** melwitt is now known as jgwentworth | 17:02 | |
| jeblair | i found the proximate cause was the hung git processes, but i also wanted to fix it regardless, so we would be protected from leaking memory due to long-running processes | 17:03 |
| mordred | ++ | 17:03 |
| *** wolverin_ has joined #openstack-infra | 17:03 | |
| jeblair | i came up with 509903 for that, but unfortunately, there are some tricky complications with that | 17:03 |
| fungi | memory leak in the scheduler associated with git process updating configuration right? | 17:03 |
| jeblair | fungi: any outstanding gearman job can end up holding a reference to an outdated config in the right circumstances | 17:04 |
| fungi | okay | 17:04 |
| jeblair | fungi: (that was one of them, and the one that i found it with) | 17:04 |
| fungi | so the bug was merely being triggered by the git hangs | 17:04 |
| fungi | but could also theoretically be caused through other means, under the right circumstances | 17:04 |
| fungi | got it | 17:05 |
| *** iyamahat has quit IRC | 17:05 | |
| jeblair | so i'm kind of inclined to shelve that for now, and say "well, we should be okay as long as we don't have gearman processes piling up" | 17:05 |
| *** openstackstatus has quit IRC | 17:05 | |
| *** openstack has joined #openstack-infra | 17:06 | |
| *** ChanServ sets mode: +o openstack | 17:06 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: zuul v3: Mention translation jobs https://review.openstack.org/509974 | 17:06 |
| jeblair | i *do* want to continue working on that a bit more, because it can affect the behavior of the system (including what configuration jobs end up running with) and is triggered on reconfiguration, which happens, like, *all the time* | 17:06 |
| *** dizquierdo has quit IRC | 17:06 | |
| jeblair | so i'm going to dig into that a bit more today | 17:07 |
| mordred | jeblair: if it affects the configuration that jobs end p running with I think that's important :) | 17:07 |
| *** iyamahat has joined #openstack-infra | 17:07 | |
| fungi | critical even | 17:07 |
| jeblair | finally -- i finished up the last bit of low-hanging-fruit on the speed-up-dynamic-reconfig task -- 510180. there's high-hanging fruit on that, but that can wait for much later. | 17:07 |
| jeblair | so once that merges, i'll move that bit into fixed issues | 17:08 |
| openstackgerrit | Merged openstack-infra/project-config master: legacy-swift-dsvm-functional should be voting https://review.openstack.org/508585 | 17:08 |
| jeblair | i reckon we can also move the memory leak into there now, given what we said above | 17:08 |
| mordred | jeblair: while you're in that area (just +A'd 510180) - how much surgery is it to have the job config syntax check run with the speculative future config even when trusted-projects are involved - but to run the jobs themselves with the correct non-speculative state? | 17:08 |
| jeblair | mordred: that is what happens | 17:09 |
| jeblair | mordred: that's the "phase 1" config generation. "phase 2" is no-trusted-projects | 17:09 |
| mordred | jeblair: but a change that depends-on a project-config change that adds a job will throw syntax errors about unknown jobs ... | 17:09 |
| mordred | jeblair: ah, nevermind - I think i see the flaw in the thing I'm asking about | 17:10 |
| jeblair | mordred: yah, that's likely the phase2 config failing, which is the one we run with | 17:10 |
| *** Goneri has quit IRC | 17:10 | |
| jeblair | mordred: what we *might* be able to do is report that the phase1 config ran without error | 17:11 |
| jeblair | so we could say something like "We can't run this now, but once dependencies land it should be okay" | 17:11 |
| mordred | yah. I think there (I can add to a todo list for later) amending the error and saying something like "this patch cannot be tested until its dependency has landed because it depends on a trusted-config change, but it's otherwise syntactically valid once that happens" or something | 17:12 |
| mordred | yah | 17:12 |
| mordred | so that people don't think they made mistakes when it's just a legit depends-on sequencing issue | 17:12 |
| jeblair | ++ | 17:12 |
| mordred | anywho - I don't think that's a blocker by any means - just thought of it given the area you've been hacking in | 17:12 |
| jeblair | it looks like my main memory leak change just bounced off with a test failure. i'm going to add a @skip to the test, because gc based tests have proven notoriously unreliable. | 17:13 |
| *** yamahata has joined #openstack-infra | 17:13 | |
| mordred | ++ | 17:13 |
| jeblair | i like the test though, because it helped me find and fix the problem and is pretty reliable in isolation | 17:13 |
| * mordred was a smidge worried about those lines | 17:13 | |
| jeblair | thus the skip to keep it around | 17:13 |
| fungi | sounds like a reasonable compromise | 17:14 |
| mordred | jeblair: we COULD add a job that only runs gc tests and runs with with testr set to run only one process | 17:14 |
| jeblair | mordred: then we'd find out if it's reliable on computers that aren't mine :) | 17:14 |
| mordred | :) | 17:14 |
| jeblair | i believe pabelanger fixed the git repo hangs yesterday | 17:14 |
| jeblair | the current fix is just to have git itself timeout if http transport is too slow | 17:15 |
| jeblair | if there are ssh hangs, this won't catch them | 17:15 |
| jeblair | but we realized that all the errors we've seen are http, so this is probably okay for now | 17:15 |
| mordred | jeblair: or - we could even skip testr itself and just have a list of commands in the tox env like "ttrun -epy35 tests.gc.test_gc.TestMemoryLeaks.test_memory_leak_one" | 17:15 |
| mordred | jeblair: ++ | 17:15 |
| jeblair | we can revisit the external timeout if the problem persists | 17:15 |
| jeblair | the executors have been restarted with that... i'm not sure if the mergers have? | 17:16 |
| jeblair | pabelanger: can you update the hung-git-process section of the etherpad with current status, and move to fixed section if all servers are updated? | 17:16 |
| jeblair | oh | 17:16 |
| jeblair | actually we haven't landed the fix yet | 17:16 |
| jeblair | that's https://review.openstack.org/#/c/509893/ | 17:16 |
| fungi | is there any further post-mortem needed on the incident from earlier today? it looks like slower memory leaks (some of which you're probably eliminating) coupled with a huge backlog and then periodic jobs kicking off pushed the server over the edge into swapping, load skyrocketed, this resulted in a thread pileup which kazoo started spamming the logs about which then filled up the rootfs (possibly in | 17:17 |
| fungi | conjunction with logrotate kicking off when there was insufficient space for the new compressed logs) | 17:17 |
| jeblair | that sounds reasonable | 17:18 |
| *** jpena is now known as jpena|off | 17:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Provide error message on malformed job list https://review.openstack.org/510185 | 17:19 |
| jeblair | i think we may need to repartition the ephemeral disk into 16g of swap and mount the rest at /var/log/zuul | 17:19 |
| mordred | jeblair: ++ | 17:19 |
| jeblair | i'll add that to the issues with zuul section of etherpad | 17:19 |
| fungi | just didn't know if there was anything coming out of that which suggests further fixes we should be considering (for example does kazoo need to be more graceful when faced with the inability to spawn more threads, or should we limit them somehow? is our logging too verbose?) | 17:20 |
| jeblair | lemme look at the kazoo error and see if it's on our radar | 17:20 |
| *** sambetts is now known as sambetts|afk | 17:20 | |
| *** bnemec is now known as beekneemech | 17:20 | |
| jeblair | wow, erm, i haven't seen that | 17:22 |
| fungi | http://paste.openstack.org/show/622839/ or something else? | 17:22 |
| jeblair | ya that | 17:23 |
| fungi | note system load was >360 at the time, according to cacti | 17:23 |
| jeblair | i'm inclined to think that too many things have gone wrong by that point | 17:23 |
| fungi | yeah, seems likely to be a pathological condition | 17:23 |
| jeblair | let's keep it in mind, but not spend too much time on it now | 17:23 |
| fungi | agreed | 17:24 |
| jeblair | fungi: we still need a zk restart to clear your item | 17:24 |
| fungi | yup | 17:24 |
| fungi | unfortunately i didn't realize zuul was set to restart at boot, so by the time i was done cleaning up space on the rootfs it had been enqueuing new changes for a while and i didn't have the heart to stop it again | 17:25 |
| jeblair | let's get my 2 changes in, and pabelanger's, and Shrews's zk double lock, and then restart it all? | 17:25 |
| fungi | sgtm | 17:25 |
| jeblair | fungi: it's likely to be current branch tip too, not my local branch, so it won't have the memory leak fix right now. so we don't have too much time anyway :) | 17:25 |
| Shrews | i have a nodepool change that needs to go in as well, then the launchers can be restarted too | 17:25 |
| jeblair | (maybe -- i haven't double checked that) | 17:25 |
| *** Goneri has joined #openstack-infra | 17:26 | |
| jeblair | Shrews's change 509603 needs a +3 | 17:26 |
| fungi | i figured that was probably the case | 17:26 |
| fungi | my original intent was to just leave the scheduler offline until you were caught up so we could decide which patches to start it with | 17:26 |
| AJaeger | mordred: https://review.openstack.org/#/c/509855/1 is ready to merge - care to +2A, please? | 17:26 |
| mordred | jeblair, Shrews, fungi: is it worth considering going multi-node with zk before we re-roll-out? (I don't know if that will improve or degrade our current zk issues - but either way if it's a thing we still intend to do it might not be terrible to do it while we're hyper-focused on zk issues?) | 17:27 |
| mordred | jeblair, Shrews, fungi: or is that too much additional variable / distraction for now | 17:27 |
| jeblair | mordred: i'm ambivalent | 17:27 |
| * mordred too | 17:27 | |
| fungi | mordred: where would we run the additional zk nodes? all on the nodepool server or move them to several discrete servers? | 17:27 |
| Shrews | mordred: i haven't seen any issues (other than disk space on nodepoo.o.o) that says we need more ZK servers right now | 17:28 |
| mordred | fungi: yah - the theory is that if we have 3 discreet servers we get ourselves an HA zk ensemble | 17:28 |
| jeblair | fungi: probably discrete | 17:28 |
| jeblair | then we can do rolling maintenance | 17:28 |
| fungi | okay, so also moving zk off nodepool.o.o | 17:28 |
| mordred | yah | 17:28 |
| fungi | entirely | 17:28 |
| jeblair | which, to be fair, is something we need to do eventually since that's the "v2 nodepool server" | 17:28 |
| mordred | ++ | 17:29 |
| *** edmondsw has joined #openstack-infra | 17:29 | |
| jeblair | i say we don't block the v3 rollout on it, but welcome it at any time | 17:29 |
| jeblair | going down the debug list -- it looks like clarkb feels the out of space issue was actually us being very briefly out of space on the executors. so it's probably not further actionable except to remember that we still need to add those partitions to cacti | 17:30 |
| jeblair | i'll move it to fixed | 17:31 |
| mordred | agree | 17:31 |
| *** links has quit IRC | 17:32 | |
| jeblair | pabelanger wrote some findings about ze03 being stopped, however, i don't think that fully explains it. | 17:32 |
| jeblair | maybe we should chalk that up to <shrug> | 17:32 |
| *** edmondsw has quit IRC | 17:33 | |
| jeblair | i'll move it to 'fixed' just to get it out of the way | 17:33 |
| jeblair | Zuul status reporting merger failures on changes that had merged previously in the gate. Implying that the failure isn't actually due to a git merge fail. | 17:34 |
| jeblair | that would have been a great one to actually link to a change | 17:34 |
| jeblair | without that, i don't know how we debug it | 17:34 |
| jeblair | i think that may have been clarkb's color? | 17:34 |
| jeblair | i've left a comment there, but if we don't have more info, i don't think it's actionable | 17:35 |
| jeblair | skipping down -- it looks like we're still having hostname on server creation errors | 17:36 |
| jeblair | fungi, mordred: do you know if there's a fix for that? | 17:36 |
| *** ijw has joined #openstack-infra | 17:36 | |
| mordred | jeblair: I don't think there is a fix | 17:37 |
| mordred | but I think what the issue is ... | 17:37 |
| fungi | i am not aware of one. best guess at the moment is cloud-init racing our hostname reset code | 17:37 |
| fungi | it may be that we should just stuff some overrides into the cloud-init configuration | 17:37 |
| mordred | is that we get merge errors reported sometimes from causes that are not actually merge errors - such as a merger failure | 17:37 |
| jeblair | mordred: yeah, that's a thing. but the error reported in the etherpad seemed really specific | 17:38 |
| jeblair | (i think we've backed up to talking about merge failures) | 17:38 |
| mordred | jeblair: yah | 17:39 |
| jeblair | mordred: i'm inclined to agree with you and assume it is that. it's just difficult to do with both such a specific error condition and no link to an example to verify. | 17:39 |
| jeblair | but if no more data are forthcoming, that's what i'll do. :) | 17:39 |
| jeblair | mordred: done? | 17:40 |
| mordred | jeblair: oh - that reminds me of a thing I should add maybe ... use of: https://docs.openstack.org/infra/zuul/feature/zuulv3/admin/drivers/gerrit.html#attr-pipeline.require.<gerrit source>.current-patchset | 17:40 |
| mordred | can result in things not being enqueued but no message left for the user indicating why not *I think* | 17:41 |
| mordred | jeblair: but yes - I do not have more data on the presumed clarkb issue | 17:41 |
| mordred | jeblair: should I note the above somewhere? | 17:41 |
| jeblair | mordred: can we talk about it in a minute? | 17:41 |
| mordred | jeblair: totally | 17:42 |
| jeblair | i'd like to discuss the hostname issue | 17:42 |
| mordred | mmm. yes | 17:42 |
| jeblair | it *keeps* happening. | 17:42 |
| fungi | seems like it started happening a few weeks ago | 17:42 |
| jeblair | so apparently "be aware that cloud-init/ansible is broken and fix it" is not a sustainable strategy | 17:42 |
| jeblair | every time we boot another server, be it nodepool or zuul, we're going to forget about this, and break the system, and do it again | 17:43 |
| fungi | i was going to audit our servers for any others with missing fqdn in /etc/hostname... maybe i'll get started on that here in a sec. might help us further narrow down the timeframe where it began | 17:43 |
| jeblair | i think if no one can think of a solution, at the very least we should amend our instructions to say "log into the server and manually fix it and restart services" | 17:43 |
| *** armax has quit IRC | 17:44 | |
| mordred | jeblair: so - I think we should either ... | 17:44 |
| fungi | not sure if the behavior change coincides with some change rackspace has made in their images, or with some change we've introduced to the launch script | 17:44 |
| mnaser | can i suggest using this in userdata? http://paste.openstack.org/show/622867/ | 17:44 |
| mordred | jeblair: a) do what you just said for today (totally quickest) | 17:44 |
| *** oomichi_afk is now known as oomichi | 17:44 | |
| fungi | mnaser: i think that means we'd have to start using userdata | 17:44 |
| mordred | yah. and my goal is to actualy make use of cloud-init go away | 17:45 |
| *** yamamoto has joined #openstack-infra | 17:45 | |
| mordred | jeblair: b) update launch-node to delete cloud-init before it runs the hostname setting | 17:45 |
| jeblair | is mnaser's suggestion a workable stop-gap? since making cloud-init go away means using our own images, which is a big project. | 17:45 |
| jeblair | mordred: oh, is b feasible? | 17:45 |
| fungi | mordred: i don't see how we can fix the current problem that way, unless we switch to creating our own images | 17:45 |
| mordred | jeblair: c) once v3 rollout has settled, get dib-built base images for our clouds | 17:45 |
| mordred | jeblair: yes- we do not need cloud-init for any purposes once we can log in to the machine | 17:46 |
| mordred | we have successfully removed it from a subset of our servers already | 17:46 |
| fungi | presumably cloud-init has already run by the time the instance is booted far enough for our launch script to log into it, right? | 17:46 |
| mordred | right. or - at least it has run enough to get ssh keys in place, which is all we want | 17:46 |
| fungi | but it's also set the hostname at that point | 17:47 |
| mordred | you'd think - but for some reason it's getting re-set after we run set_hostnames | 17:47 |
| mnaser | also | 17:47 |
| mnaser | could it be dhcp | 17:47 |
| mordred | mnaser: nope | 17:48 |
| fungi | no dhcp in rackspace | 17:48 |
| mordred | mnaser: thecloud in question does not use dhcp | 17:48 |
| fungi | i'm starting to wonder if we don't have a race here, and our step to correct the /etc/hostname file has somehow regressed/broken | 17:48 |
| mordred | which continues to be *what* | 17:48 |
| mordred | well - I think the issue is that we reboot the node | 17:48 |
| jeblair | fungi: i thought last time we looked, we saw ansible dtrt, but worth double checking | 17:48 |
| mordred | and cloud-init is setting it on the second boot | 17:48 |
| mordred | but we set the hostname in the playbook before the reboot | 17:49 |
| fungi | mordred: but why does it not reset it again on later reboots when we manually correct /etc/hostname? | 17:49 |
| mordred | fungi: honestly no clue | 17:49 |
| jeblair | i think we only set hostname on the initial launch-node ansible run? | 17:49 |
| mordred | yah | 17:49 |
| jeblair | i don't think it's in the regular manifest? | 17:49 |
| mordred | I believe that is correct | 17:49 |
| jeblair | though tbh, even that wouldn't be a fix because the 2nd boot would still be broken. | 17:50 |
| jeblair | 3rd boot would be okay :) | 17:50 |
| *** electrofelix has quit IRC | 17:50 | |
| fungi | so the symptom i witnessed is that ze01 when rebuilt had just ze01 in /etc/hostname, i edited the file by hand, rebooted the server and the hostname and /etc/hostname file had the ze01.openstack.org fqdn thereafter | 17:50 |
| jeblair | mordred: okay, assuming that's all correct, i like your (b) solution. maybe we could just do that real quick. | 17:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove cloud-init when we set hostnames https://review.openstack.org/510198 | 17:51 |
| mordred | jeblair, fungi: ^^ there's b | 17:51 |
| jeblair | mordred: cool, thx; let's continue through the etherpad now | 17:52 |
| tobiash | That was realy quick... | 17:52 |
| jeblair | infracloud ssh and node_failure look like things we can move to fixed now | 17:52 |
| Shrews | fyi, i have to leave in about 5 min for an appointment & will be gone the rest of the day | 17:52 |
| mordred | tobiash: I tend to have high motivation when it comes to removing cloud-init from things | 17:52 |
| jeblair | Shrews: thanks, we'll push your changes in and restart | 17:53 |
| jeblair | next interesting thing is pip freeze | 17:53 |
| Shrews | k. just want to draw attention to https://review.openstack.org/510085 which fixes a rather major nodepool issue i discovered this morning | 17:53 |
| *** yamamoto has quit IRC | 17:53 | |
| jeblair | Shrews: ++ | 17:53 |
| jeblair | what is the pip freeze issue breaking? | 17:54 |
| mordred | jeblair: there are two issues | 17:55 |
| mordred | jeblair: one is, I believe, cosmetic - that is that because tox does a pip freeze, the error about it is printed as a warning but it makes it look like something went wrong | 17:55 |
| mordred | so people looking at their jobs fixate on the wrong thing | 17:56 |
| mordred | the other is in devstack, pip freeze is used as part of libs_from_git | 17:56 |
| mordred | and pip freeze not being able to deal with the no-remote case causes the devstack logic to fail - I belive there is a workaround in place | 17:57 |
| jeblair | that would be ianw's change i think | 17:57 |
| mordred | yah | 17:57 |
| * jeblair tries to find that | 17:57 | |
| mordred | 5b419ffb1f20dfe613bd694fab8c1f08c8db7cce | 17:58 |
| mordred | is the commit | 17:58 |
| mordred | I21ff749ab3e7911fa074e6d53056768f42f8aa57 is the change | 17:58 |
| mordred | the commit just disables the libs_from_git check | 17:58 |
| jeblair | yay merged! | 17:58 |
| jeblair | so i think that's the fix to issue 2, and the pypa pr is the fix to issue 1. | 17:59 |
| mordred | ah - https://review.openstack.org/#/c/508366/ switches to pip list | 17:59 |
| jeblair | oh right | 17:59 |
| jeblair | the pypa fix is still needed for tox though | 17:59 |
| clarkb | the tox thing isn't fatal atleast | 18:00 |
| *** wolverin_ has quit IRC | 18:00 | |
| clarkb | just really annoying | 18:00 |
| pabelanger | back, catching up | 18:00 |
| jeblair | but as you say, that's cosmetic, so i think we can move this to fixed now | 18:00 |
| *** wolverineav has joined #openstack-infra | 18:00 | |
| jeblair | mordred: you have/had a change for the write-outside-of-work/ issue ya? | 18:01 |
| jeblair | https://review.openstack.org/509901 merged | 18:02 |
| jeblair | i'll move that to fixed | 18:02 |
| mordred | jeblair: yes | 18:02 |
| jeblair | oh i can clear the kazoo callback error, will do | 18:03 |
| jeblair | okay, everything with a name next to the item in the debug section is done and waiting on final action (merging, restart, etc) | 18:04 |
| jeblair | there are still some things without names, so please feel free to pick those up and clear them out | 18:04 |
| jeblair | mordred: you had something to add to this section? | 18:04 |
| pabelanger | and caught up | 18:05 |
| mordred | yah ... one sec | 18:06 |
| mordred | yah - that - thanks | 18:06 |
| jeblair | mordred: have any examples? | 18:06 |
| mordred | no - how about I keep my eyes out for it happening/confusing me again | 18:07 |
| jeblair | mordred: ok. just note the change when you see something confusing | 18:08 |
| mordred | jeblair: the general area that falls in to is "why isn't my change running when I think it should be" | 18:08 |
| mordred | and I think can sit in long-term ongoing UX improvements as we identify specifics -it's not a rollout blocker by any means | 18:08 |
| jeblair | to be honest, when we have issues like this, or the merge failure, i'd really like it if we could instead have them reported as "i expected X to happen to change Y but Z happened instead" | 18:09 |
| mordred | ++ | 18:09 |
| jeblair | i feel like in both cases, we have over-diagnosed a problem | 18:09 |
| mordred | agree | 18:09 |
| *** jascott1 has joined #openstack-infra | 18:10 | |
| jeblair | mordred: requirements jobs -- 2 more changes to land? | 18:11 |
| SamYaple | AJaeger: i asked when i merged the project-config patch to remove all the jobs for LOCI if i still needed merge-check and was told i did not since zuulv3 did it diferently | 18:11 |
| SamYaple | i will add it back | 18:12 |
| jeblair | mordred: looks like 845 merged | 18:12 |
| jeblair | so only 855 now? | 18:12 |
| jeblair | i approved it | 18:13 |
| mordred | \o/ | 18:13 |
| jeblair | mordred: the yaml syntax error thing is what you were werking on earlier, yeah? | 18:13 |
| mordred | yup | 18:13 |
| mordred | with https://review.openstack.org/510185 as the presumptive fix | 18:14 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Do not satisfy min-ready requests if at capacity https://review.openstack.org/510085 | 18:14 |
| jeblair | the only unmerged tripleo bugfix in 508660 | 18:15 |
| *** priteau has quit IRC | 18:16 | |
| jeblair | that's been sitting with a +2 for a couple days, but also, some zuulv3 failures.... | 18:16 |
| *** lbragstad has quit IRC | 18:16 | |
| jeblair | is there something more we need to do there? or just poke tripelo team? | 18:16 |
| AJaeger | SamYaple: we're all learning here ;) Sorry! | 18:16 |
| pabelanger | do we need any other nodepool fixes or can we start the restart process for 510085? | 18:16 |
| jeblair | pabelanger: if nodepool has updated with that, you're clear to restart | 18:17 |
| pabelanger | great, I'll wait until puppet runs | 18:17 |
| jeblair | mordred, pabelanger, clarkb: ^ see question above about tripleo | 18:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Disable check-requirements template" https://review.openstack.org/509855 | 18:17 |
| clarkb | jeblair: 660 aiui is the tripleo fix change (without it they will still have problems | 18:18 |
| clarkb | I do think it works on new and old zuul though so if they can review it you should be good | 18:18 |
| AJaeger | could you review https://review.openstack.org/510142 - to merge the infra-gate pipeline only with Zuul +1 and not also with jenkins +1? | 18:18 |
| jeblair | clarkb: okay, zuul is reporting -1s on it, is that a problem? | 18:18 |
| mordred | http://logs.openstack.org/60/508660/17/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq-ocata/23c7d69/logs/ara_oooq/result/8c4654de-9907-4551-98a7-abee25390a4c/ is a failure from that tripleo fix (which I don't understand) | 18:18 |
| clarkb | jeblair: it failed one job, the others all worked, I don't know enough about tripleo-ci to know if that is still zuul fallout or just flakyness | 18:19 |
| SamYaple | AJaeger: im not complaining :) | 18:19 |
| mordred | clarkb, jeblair: same here | 18:19 |
| jeblair | clarkb, mordred: i see 3 failures (one in check, 2 in check-tripleo) | 18:19 |
| clarkb | oh I didn't look at check-tripleo | 18:20 |
| mordred | yah - I see the same failures in zuul and jenkins reporting too | 18:20 |
| jeblair | mordred: likely general problem then? | 18:20 |
| mordred | well -that's the check-tripleo ones ... | 18:21 |
| *** gridinv1 has joined #openstack-infra | 18:21 | |
| mordred | looking at the check one real quick | 18:21 |
| *** baoli has quit IRC | 18:22 | |
| jeblair | oh look at the dependency Ifb3af18733c0e1fd6895c270bb39199acaa98968 | 18:22 |
| jeblair | it has landed on master, but not ocata and pike | 18:22 |
| *** baoli has joined #openstack-infra | 18:22 | |
| jeblair | i wonder if they are bypassing zuul's git processing on that. | 18:23 |
| mordred | ah! good catch | 18:23 |
| jeblair | and it's in tripleo-heat-templates, which is cited in the error message | 18:23 |
| jeblair | anyone want to follow up with them and remind them that they should merge those rsn? | 18:24 |
| mordred | EmilienM: ^^ ping | 18:25 |
| EmilienM | mordred: o/ | 18:25 |
| *** lbragstad has joined #openstack-infra | 18:25 | |
| * EmilienM clicks | 18:25 | |
| *** gridinv1 has quit IRC | 18:25 | |
| *** trown|lunch is now known as trown | 18:25 | |
| jeblair | or we could do it synchronously. :) | 18:26 |
| EmilienM | jeblair: missing whole context, what do you want me to do? | 18:26 |
| mordred | EmilienM: https://review.openstack.org/#/c/508660 is the last thing we know about for fixing tripleo jobs for zuulv3 | 18:27 |
| mordred | EmilienM: it has 3 failures - 1 in check and 2 in tripleo check (you need to scroll down to the final two comments to see them all) | 18:27 |
| EmilienM | mordred: should I remove the depends-on the THT backports? | 18:27 |
| mordred | EmilienM: the newton/ocata failures may point to something not actually using the zuul-prepared branches properly (since if the THT patches should be fixing this, it looks like the depends-on maybe isn't working?) | 18:28 |
| mordred | EmilienM: the third failure is opaque to me: http://logs.openstack.org/60/508660/17/check/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/aebcbaf/ | 18:28 |
| EmilienM | mordred: let me look | 18:28 |
| EmilienM | mwhahaha: ^ fyi | 18:29 |
| mordred | EmilienM: in any case, mostly wanted to follow up while we're assessing state of known v3 issues | 18:29 |
| *** psachin has joined #openstack-infra | 18:29 | |
| EmilienM | mordred: for scenario004 it's a tripleo related issue, let me fin dit | 18:29 |
| jdandrea | fungi Re: cronjob vs post job to apply 508924, that sounds good too, ok! | 18:29 |
| jdandrea | fungi I'll just watch eavesdrop to see the name change. | 18:29 |
| jdandrea | fungi After that happens, does someone need to manually delete the orphaned openstack-valet directory, or is that done automagically? | 18:30 |
| fungi | SamYaple: yeah, we disabled the merge-check pipeline for now and may decide to remove it entirely, but would prefer to have the option to turn it back on until we're sure we want to get rid of it | 18:30 |
| EmilienM | http://logs.openstack.org/60/508660/17/check/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/aebcbaf/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 18:30 |
| *** ijw has quit IRC | 18:30 | |
| EmilienM | yeah this is the thing we were saying yesterday, overcloud isn't deployed | 18:30 |
| EmilienM | mordred: we saw this one yesterday I think | 18:30 |
| EmilienM | mordred: can you confirm we need to land the 2 THT backports plz? | 18:31 |
| SamYaple | fungi: got it. that makes sense. its a shame because i had completely removed loci jobs from project-config.... but no matter :) | 18:31 |
| jeblair | EmilienM, mordred: alternatively, we could add the node_private file to the legacy pre playbook | 18:31 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Create git_http_low_speed_limit / git_http_low_speed_time https://review.openstack.org/509893 | 18:31 |
| jeblair | i don't think we were aware that was being used anywhere | 18:31 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Remove extra privileges as we have fixed the master https://review.openstack.org/510208 | 18:31 |
| fungi | thanks dims! | 18:32 |
| *** tosky has joined #openstack-infra | 18:32 | |
| *** afred312 has joined #openstack-infra | 18:32 | |
| EmilienM | jeblair: whatever works faster for you and us | 18:32 |
| dims | that was the easy part fungi :) | 18:32 |
| EmilienM | jeblair: I approved https://review.openstack.org/#/c/509705/ and rechecked the ocata backport | 18:33 |
| fungi | jdandrea: it doesn't get deleted with the way that site is published for now, but nothing will continue linking to it so it's probably fine to ignore | 18:33 |
| jdandrea | fungi *nodnod* | 18:33 |
| EmilienM | jeblair, mordred: iiuc, last thing we need is https://review.openstack.org/#/c/508660, right? | 18:33 |
| jdandrea | fungi Keeping quiescent on the channel for now. ;) | 18:33 |
| EmilienM | mwhahaha: if we can review https://review.openstack.org/#/c/508660 (be careful not to approve yet we have one patch in dependency not merged, 2 backports) | 18:34 |
| jeblair | mordred: still around? | 18:34 |
| mordred | jeblair: yes | 18:34 |
| mordred | EmilienM: I'm mostly concerned that we're needing to land those newton and ocata patches for that patch to work ... it concerns me that something in your job is not consuming the prepared repos properly | 18:35 |
| *** claudiub has quit IRC | 18:35 | |
| mordred | EmilienM: so - landing https://review.openstack.org/#/c/508660 should unblock things AIUI - but I think it's worth investigating why the -newton and -ocata jobs aren't seeing the patch to newton and ocata via depends-on | 18:36 |
| pabelanger | okay, puppet looks to have run on nodepool-launchers, starting the restart process | 18:36 |
| dims | rbergeron : w00t on https://twitter.com/robynbergeron/status/916348177891999744 | 18:36 |
| mordred | EmilienM: the src/git.openstack.org/openstack/tripleo-heat-templates repo on disk for that job should have master, stable/newton and stable/ocata branches all in the appropriate state | 18:36 |
| mordred | EmilienM: so if something is doing git manipulations other than just "git checkout stable/newton" - then your jobs are open to depends-on not testing what you think it is | 18:37 |
| jdandrea | fungi I'm keeping quiet on #openstack-valet until the logging switches over. (It should be relatively quiet there now anyway.) But then I'll edit the topic with the newer log directory. Is that ok, to hold off a bit? | 18:38 |
| mordred | EmilienM: I'm not really sure where to look for that - but if you or mwhahaha or someone else who understands those jobs has a sec to see why the newton/ocata patches don't seem to be included in the tests on https://review.openstack.org/#/c/508660 before you land them on newton and ocata, I think you'll be happier in the long run | 18:38 |
| fungi | jdandrea: not sure what you mean by logging switching over | 18:39 |
| mwhahaha | mordred: I assume you're refering to the newton/ocata ovb jobs failing | 18:39 |
| mordred | mwhahaha: I am | 18:40 |
| *** gridinv1 has joined #openstack-infra | 18:40 | |
| mordred | mwhahaha: and I just want to make sure we're not papering over a fundamental problem somewhere that this happens to show | 18:40 |
| pabelanger | jeblair: Shrews: mordred: okay, nodepool-launchers have been restarted with min-ready fix. I also deleted ready nodes in ovh-bhs1 as they appearred wedged | 18:42 |
| EmilienM | mordred: gotcha | 18:42 |
| fungi | jdandrea: if you mean waiting for http://eavesdrop.openstack.org/irclogs/%23openstack-valet/ to work, the "openstack" meetbot which also handles channel logging has been in there and recording since 17:05:39 utc, almost 2 hours ago | 18:42 |
| jeblair | mordred: okay, so having said all that -- do we also want to add 'node_private' to the legacy pre playbook to increase our compatability? | 18:42 |
| *** ijw has joined #openstack-infra | 18:43 | |
| mwhahaha | mordred: I'm not familar enough with the depends-on, it seem that the zuul_changes list only has the master ones | 18:43 |
| mwhahaha | mordred: where would the newton/ocata patches be passed in as part of depends-on or how is that handled | 18:44 |
| mordred | jeblair: I think that's not a terrible idea | 18:44 |
| mordred | mwhahaha: so ... the way it works in v3 is that zuul prepares the repos with all of the branches set up correctly | 18:45 |
| jeblair | http://logs.openstack.org/60/508660/17/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq-ocata/23c7d69/zuul-info/inventory.yaml | 18:45 |
| mordred | mwhahaha: so all you need to do to consume patches from stable/newton is to do "git checkout stable/newton" in the git repo ... in this case, in src/git.openstack.org/openstack/tripleo-heat-templates | 18:45 |
| mwhahaha | mordred: i don't think we consume the repos on disk as they get shipped over to dlrn to handle that | 18:46 |
| jeblair | i think mwhahaha is right, i don't see any stable/ changes there | 18:46 |
| mordred | jeblair: I agre - I also don't see any stable/ changes there | 18:46 |
| mwhahaha | but that might be an issue with how we build the packages | 18:46 |
| jeblair | i think zuul should have put all 3 of those changes ahead, but i only see one | 18:47 |
| mordred | jeblair: aha | 18:47 |
| jeblair | mordred: you agree this sounds like a zuul bug? | 18:47 |
| mordred | jeblair: Depends-On: Ifb3af18733c0e1fd6895c270bb39199acaa98968 ... that matches three different changes | 18:47 |
| mordred | jeblair: the intent is that zuul shold find all three of them from that depends-on and add all three to the items list yeah? | 18:47 |
| jlvillal | Is there a way to remove a Zuul -1 vote, or have Zuul try again? | 18:48 |
| mordred | jlvillal: recheck will cause zuul to try again | 18:48 |
| jeblair | mordred: i believe so | 18:48 |
| jlvillal | Since the Zuul -1 vote is what is displayed instead of the +1 Jenkins vote in a summary list | 18:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run fetch-tox-output and fetch-sphinx on all not localhost https://review.openstack.org/510209 | 18:48 |
| jlvillal | mordred: I will give it a shot. Thanks. | 18:48 |
| *** gridinv1 has quit IRC | 18:49 | |
| mordred | jeblair: so - yes, if we expect depends-on with multiple resolutions to behave likea depends-on on all the resolutions, I believe this is a zuul bug | 18:49 |
| jeblair | mordred, mwhahaha, EmilienM: let's classify the issue about not using the stable branches as a zuul bug for now. i'll put it on the list to look into. | 18:50 |
| mordred | ++ | 18:50 |
| jeblair | i also think we should add the node_private file. i'll add that to the list | 18:50 |
| SamYaple | zuul bug? blasphamy! | 18:50 |
| mordred | kk | 18:50 |
| mwhahaha | see it's not just us! | 18:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Have zuul re-run ABORTED jobs https://review.openstack.org/510211 | 18:50 |
| * mwhahaha goes back into hiding | 18:50 | |
| jdandrea | fungi Right, but we haven't been talking in it yet. On purpose. | 18:50 |
| fungi | jdandrea: oh, you have another channel somewhere else. got it | 18:51 |
| jdandrea | fungi We figured once that logging switches over, then we'll go ahead (and adjust the topic) | 18:51 |
| fungi | i still don't know what you mean by "logging switches over" | 18:51 |
| fungi | it's logging now | 18:51 |
| jeblair | okay, translation jobs | 18:51 |
| jlvillal | Random thought: I would like a "recheck failfast" command :) If any job fails, fail the whole job fast :) | 18:52 |
| jdandrea | fungi No, same channel. We're juuuuust getting started. By "logging switches over" I mean from openstack_valet to valet (there was a second change that adjusted the meeting details including the meeting name and directory. | 18:52 |
| fungi | jdandrea: oh, meetings are completely separate from your project's channel logging | 18:52 |
| jdandrea | This way, when the second change takes effect, we don't lose any conversations to the proverbial ether. | 18:52 |
| jeblair | https://review.openstack.org/502208 is about traslation jobs and is ready to merge | 18:53 |
| jdandrea | fungi Oh. I'm confused then. | 18:53 |
| jdandrea | fungi And you did say they were separate. | 18:53 |
| jeblair | jlk, AJaeger, mordred, pabelanger: who knows the status there? | 18:53 |
| jdandrea | fungi So no matter what happens with the meeting details, the logging remains the same. | 18:53 |
| jdandrea | fungi In which case I hope I did the right thing with the rename. :-o | 18:54 |
| fungi | jdandrea: you explicitly start and stop a meeting and that plus associated metadata directives get recorded to a consistent name (whatever name you pass after the #startmeeting command). your channel is also continuously logged to files under http://eavesdrop.openstack.org/irclogs/%23openstack-valet/ | 18:54 |
| pabelanger | I haven't followed the translation jobs yet, I'll need to read up on it | 18:54 |
| jdandrea | fungi Right. | 18:54 |
| fungi | those logs are kept separate | 18:54 |
| jdandrea | fungi I'm confusing the meeting name with the channel name with the directory name. | 18:54 |
| jdandrea | fungi And separate logs. But in that case then the log directory *does* matter. Since our meeting isn't until Wednesday though, it's fine. | 18:55 |
| jdandrea | fungi OK I'll adjust the topic then. :) | 18:55 |
| mordred | jeblair: I'm not 100% sure on that- AJaeger and jlk were working on it - I think to some degree we just have to merge and then iterate | 18:55 |
| fungi | jdandrea: yeah, your meetings themselves get logged to http://eavesdrop.openstack.org/meetings/<meetingname>/<year>/<meetnigname>.<date>.* | 18:56 |
| jeblair | fungi, pabelanger: can i ask one of you to give https://review.openstack.org/502208 a review and +3 if appropriate? | 18:57 |
| fungi | jeblair: i already did review and +2 it a while ago | 18:57 |
| fungi | doesn't seem to have changed unless gertty isn't updating for me | 18:57 |
| mordred | jeblair: I have +3'd just now | 18:58 |
| jeblair | fungi: my mistake sorry :) | 18:58 |
| pabelanger | Ya, I can give it a look over | 18:58 |
| fungi | oh, no worries, i thought maybe i was missing something there | 18:58 |
| pabelanger | maybe not :) | 18:58 |
| fungi | it happens to me often enough | 18:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add /etc/nodepool/sub_nodes file https://review.openstack.org/510213 | 18:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 18:59 |
| jeblair | pabelanger: what's the story with 509182? | 18:59 |
| *** gridinv has quit IRC | 19:00 | |
| jeblair | mordred: what does "bong" mean regarding legacy-npm-upload jobs? | 19:00 |
| mordred | jeblair: they're just completely wrong | 19:00 |
| jlvillal | Something that is legal in Oregon and Colorado? | 19:00 |
| *** eranrom has joined #openstack-infra | 19:00 | |
| mordred | jeblair: I'll take care of them in just a little bit | 19:00 |
| jeblair | mordred: thx | 19:01 |
| mordred | jeblair: (they are depending on the build-python-tarball job for one) | 19:01 |
| jeblair | mordred: the project creater guide? that update still need to be done, or did you do that last night? | 19:01 |
| mordred | jeblair: still needs to be done | 19:01 |
| jeblair | ok | 19:01 |
| jeblair | that's everything | 19:01 |
| jeblair | that took 2 hours. | 19:01 |
| mordred | jeblair: Error from tag-releases | 19:01 |
| mordred | jeblair: did I miss us talking about that? | 19:02 |
| jlvillal | Review request: https://review.openstack.org/509670 Currently if you click an error level, you will NOT see the CRITICAL log level messages in the logfile | 19:02 |
| jeblair | no, but its status did not seem unclear | 19:02 |
| jeblair | mordred: it seems to still be a thing someone should look into, yeah? | 19:02 |
| mordred | yah. I wish we had recorded what it was that the infra-root had to track down, as we now have to re-track it down :( | 19:03 |
| jeblair | mordred: (i skipped items whose status seemed clear to me, ie, "still need someone to look into it") | 19:03 |
| jeblair | mordred: agree, though at least there's a job there we can start with | 19:03 |
| mordred | yah | 19:03 |
| jeblair | if someone picks that up, please put findings in etherpad | 19:03 |
| pabelanger | jeblair: that was preventing us from landing jobs in system-config, plan to revert it once pressing issues are out of the way. We just need to add proper require-projects to jobs | 19:04 |
| jeblair | pabelanger: okay, so the action for this is actually : fix job definitions for those jobs, then revert that change | 19:04 |
| pabelanger | mordred: error for tag-releases was a typo on a role name | 19:05 |
| jeblair | it looked like we just needed to revert that change, but that would be counter-productive | 19:05 |
| fungi | pabelanger: well, we don't know that's _all_ those jobs need, right? because they don't run completely yet | 19:05 |
| pabelanger | jeblair: yes, in fact, I can get started on that now | 19:05 |
| pabelanger | fungi: yes, I am not sure anything after the original debug I did on monday | 19:05 |
| *** chlong has joined #openstack-infra | 19:06 | |
| jeblair | pabelanger, fungi, mordred: when should we next review the etherpad? during zuul meeting time on monday? | 19:06 |
| mordred | jeblair: sounds good to me | 19:06 |
| fungi | jeblair: that seems like a fine time to me | 19:06 |
| pabelanger | sure | 19:06 |
| openstackgerrit | Merged openstack-infra/project-config master: Add translation jobs https://review.openstack.org/502208 | 19:06 |
| fungi | pabelanger: so is the remaining concern with the tag-releases failure that jobs with mistyped roles should bubble errors up to the output ansible's logging somehow? | 19:07 |
| jeblair | okay. i'm going te be very abrupt then so we get through it very quickly, and i'll ask everyone to focus on only that for a short time so that we don't drag it out to 2 hours again. | 19:07 |
| jeblair | thanks for sticking through it today | 19:08 |
| jeblair | i think we cleared up a bunch of stuff and uncovered lurking issues under cobwebs | 19:08 |
| pabelanger | fungi: ya, I think we should make it easier for non-root people to find that error. Because I needed to log into executor to find the error message | 19:08 |
| rbergeron | dims: lol. that whole thing just ... RAGE | 19:08 |
| *** wolverineav has quit IRC | 19:08 | |
| fungi | jeblair: sounds good, thanks. having it in-meeting rather than in #-infra where we also have unrelated things going on will likely help speed it up | 19:08 |
| rbergeron | dims: thanks tho :) | 19:08 |
| *** wolverineav has joined #openstack-infra | 19:09 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/bindep master: Add new syntax to allow matching multiple profile https://review.openstack.org/506502 | 19:10 |
| AJaeger | mordred: in https://review.openstack.org/#/c/51020 zuul gives merge failure but gerrit and jenkins are happy - what did you do to confuse them? | 19:11 |
| *** wolverineav has quit IRC | 19:11 | |
| AJaeger | ;) | 19:11 |
| *** wolverin_ has joined #openstack-infra | 19:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove extra privileges as we have fixed the master https://review.openstack.org/510208 | 19:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle non-syntax errors from Ansible https://review.openstack.org/510219 | 19:15 |
| mordred | AJaeger: I'm very skilled ... which one? I lost the last digit in your message | 19:17 |
| *** edmondsw has joined #openstack-infra | 19:17 | |
| * AJaeger hands mordred a 9 | 19:17 | |
| mordred | pabelanger, jeblair: https://review.openstack.org/510219 fixes the tag-releases error message issue | 19:17 |
| AJaeger | mordred: https://review.openstack.org/510209 | 19:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run fetch-tox-output and fetch-sphinx on all not localhost https://review.openstack.org/510209 | 19:18 |
| mordred | AJaeger: who knows ... | 19:18 |
| AJaeger | mordred: strange, I also tried rebasing and couldn't -but there was a merge in between ;) | 19:19 |
| *** afred312 has quit IRC | 19:20 | |
| *** edmondsw has quit IRC | 19:22 | |
| *** tesseract has quit IRC | 19:31 | |
| *** hemna_ has quit IRC | 19:31 | |
| *** wolverin_ has quit IRC | 19:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Retry jobs on executor disconnect https://review.openstack.org/510223 | 19:38 |
| *** pblaho has quit IRC | 19:39 | |
| *** abelur_ has quit IRC | 19:41 | |
| mordred | fungi, clarkb, pabelanger: https://review.openstack.org/#/c/510142 if you get a sec | 19:42 |
| fungi | k | 19:42 |
| fungi | yep, we talked about that earlier | 19:43 |
| fungi | thanks AJaeger! | 19:43 |
| *** baoli has quit IRC | 19:44 | |
| *** hashar has joined #openstack-infra | 19:46 | |
| *** eharney has quit IRC | 19:52 | |
| fungi | i'm about to step away to grab a very late breakfast/late lunch/early dinner but should be back in a while to continue | 19:53 |
| *** baoli has joined #openstack-infra | 19:55 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle double node locking snafu https://review.openstack.org/509603 | 19:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove mistral legacy jobs https://review.openstack.org/510008 | 19:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Only allow zuul user for infra-gate https://review.openstack.org/510142 | 19:55 |
| *** jcoufal has quit IRC | 19:56 | |
| fungi | what else were we trying to get in before the zuul/zk/nodepool restart? | 19:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Have zuul re-run ABORTED jobs https://review.openstack.org/510211 | 19:56 |
| *** zaneb has quit IRC | 19:58 | |
| *** baoli has quit IRC | 19:58 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Set zuul_log_path for a periodic job https://review.openstack.org/509384 | 19:59 |
| *** baoli has joined #openstack-infra | 20:00 | |
| *** e0ne has quit IRC | 20:05 | |
| fungi | okay, headed out to eat now... bbiaw | 20:06 |
| *** jtomasek has quit IRC | 20:08 | |
| *** abelur_ has joined #openstack-infra | 20:15 | |
| *** kgiusti has left #openstack-infra | 20:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Always retry jobs on executor disconnect https://review.openstack.org/510223 | 20:20 |
| *** gouthamr has quit IRC | 20:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Always retry jobs on executor disconnect https://review.openstack.org/510223 | 20:22 |
| mordred | fungi: that is, in fact, a very late breakfast | 20:22 |
| *** wolverineav has joined #openstack-infra | 20:23 | |
| *** wolverineav has quit IRC | 20:25 | |
| *** wolverineav has joined #openstack-infra | 20:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Return a distinct result on executor disk full https://review.openstack.org/510227 | 20:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't store pipeline references on builds https://review.openstack.org/509653 | 20:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't load dynamic layout twice unless needed https://review.openstack.org/510180 | 20:33 |
| *** tnarg has quit IRC | 20:33 | |
| *** spectr has joined #openstack-infra | 20:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Have zuul re-run ABORTED jobs https://review.openstack.org/510211 | 20:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Always retry jobs on executor disconnect https://review.openstack.org/510223 | 20:35 |
| *** spectr has quit IRC | 20:36 | |
| *** david-lyle has quit IRC | 20:38 | |
| *** e0ne has joined #openstack-infra | 20:39 | |
| *** trown is now known as trown|outtypewww | 20:39 | |
| clarkb | pabelanger: mordred jeblair ^ can't jobs be aborted due to new patchsets and so on? we don't want to rerun those | 20:41 |
| clarkb | re 510223 | 20:41 |
| jeblair | clarkb: yes, but it's the scheduler's decision about whether to do that | 20:42 |
| clarkb | jeblair: so the aborted status here wouldn't rerun jobs from an old patchset? | 20:42 |
| jeblair | clarkb: nope. this only causes the build to be removed from the buildset. if the buildset is still current, the scheduler will re-launch. if it is not current, it won't. | 20:43 |
| andreaf | jeblair, mordred: I have a few tempest roles where I have defined a post https://review.openstack.org/#/c/509664/ - the jobs fail with POST_FAILURE, my post play is not invoked at all according to logs, so I don't know what's going on | 20:43 |
| andreaf | I'm probably doing something silly in my play but I was wondering if zuul logs might say more about it? | 20:43 |
| jeblair | andreaf: i'll take a look | 20:44 |
| pabelanger | clarkb: ABORTED here is the -9 we get back from ansible-playbook on the executor as return result | 20:44 |
| clarkb | pabelanger: ya as a response to something like a new patchset arriving | 20:45 |
| *** beekneemech has quit IRC | 20:45 | |
| clarkb | but sounds like retrying still checks currnetness so is fine | 20:45 |
| pabelanger | clarkb: good point | 20:45 |
| jeblair | andreaf, mordred: http://paste.openstack.org/show/622888/ i thought we had these showing up in the job-output.txt, but apparently not this one | 20:48 |
| jeblair | apparently we only do that for exit code 4, but this was exit code 1 | 20:48 |
| jeblair | i'm going to add this to the etherpad | 20:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't store pipeline references on builds https://review.openstack.org/509653 | 20:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't load dynamic layout twice unless needed https://review.openstack.org/510180 | 20:52 |
| *** wolverineav has quit IRC | 20:54 | |
| *** e0ne has quit IRC | 20:57 | |
| *** eroux has quit IRC | 20:57 | |
| *** gridinv has joined #openstack-infra | 21:01 | |
| *** mat128 has quit IRC | 21:03 | |
| clarkb | fungi: http://lists.openstack.org/pipermail/openstack/2017-October/045609.html | 21:03 |
| *** edmondsw has joined #openstack-infra | 21:05 | |
| *** srobert_ has joined #openstack-infra | 21:05 | |
| *** gridinv has quit IRC | 21:06 | |
| *** srobert has quit IRC | 21:08 | |
| *** edmondsw has quit IRC | 21:10 | |
| *** srobert_ has quit IRC | 21:10 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/bindep master: Add new syntax to allow matching multiple profile https://review.openstack.org/506502 | 21:11 |
| mnaser | if any infra-core got some spare cycles - https://review.openstack.org/#/c/507864/ | 21:13 |
| mnaser | should be ready to go if i didnt break everything | 21:14 |
| *** ijw has quit IRC | 21:17 | |
| *** david-lyle has joined #openstack-infra | 21:19 | |
| *** felipemonteiro__ has quit IRC | 21:21 | |
| fungi | clarkb: pretty awesome | 21:23 |
| *** ijw has joined #openstack-infra | 21:23 | |
| fungi | i'm checking to see whether that address is even subscribed | 21:23 |
| *** dprince has quit IRC | 21:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix early processing of merge-pending items on reconfig https://review.openstack.org/509912 | 21:28 |
| *** samuel has joined #openstack-infra | 21:28 | |
| *** ronlund is now known as mriedem | 21:29 | |
| *** wolverineav has joined #openstack-infra | 21:30 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Update node requests after nodes https://review.openstack.org/509571 | 21:31 |
| *** tnarg has joined #openstack-infra | 21:31 | |
| samuel | I'm trying to run tox in the master branch of jenkins-job-builder and I am getting a Versioning Exception : Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name jenkins-job-builder was given, but was not able to be found. | 21:32 |
| clarkb | samuel: I'm on a bad network right now otherwise I would try to reproduce, maybe zxiiro electrofelix or zaro can help in #openstack-infra | 21:33 |
| *** ccamacho has quit IRC | 21:34 | |
| clarkb | oh we are in -infra | 21:36 |
| clarkb | tells you how bad my connection is right now :) | 21:36 |
| *** ijw has quit IRC | 21:37 | |
| fungi | samuel: i'll see if i can recreate that | 21:37 |
| samuel | txs it's probably on my side ¯\_(ツ)_/¯ I'm running pyenv with version 3.4.4: tox -e py34 | 21:38 |
| *** ijw has joined #openstack-infra | 21:39 | |
| *** dklyle has joined #openstack-infra | 21:39 | |
| *** claudiub has joined #openstack-infra | 21:39 | |
| *** tpsilva has quit IRC | 21:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Run fetch-tox-output and fetch-sphinx on all not localhost https://review.openstack.org/510209 | 21:40 |
| *** david-lyle has quit IRC | 21:41 | |
| *** dklyle has quit IRC | 21:41 | |
| *** dklyle has joined #openstack-infra | 21:42 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't store pipeline references on builds https://review.openstack.org/509653 | 21:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't load dynamic layout twice unless needed https://review.openstack.org/510180 | 21:42 |
| jeblair | andreaf: oh it looks like mordred was ahead of us and already wrote https://review.openstack.org/510219 | 21:43 |
| fungi | samuel: testing on debian/sid with python 3.5.4, but `tox -e py35` in the jenkins-job-builder master branch tip is working fine for me | 21:43 |
| *** lbragstad has quit IRC | 21:43 | |
| fungi | samuel: how did you go about obtaining the source tree you're testing in? | 21:44 |
| fungi | samuel: most common cause for that error is attempting to use a copy of the source tree without the revision control metadata and which is not the result of an sdist build | 21:45 |
| fungi | samuel: judging from your privmsg, i'm going to guess this is some strange interaction with pyenv | 21:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add base job and roles for javascript https://review.openstack.org/510236 | 21:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job https://review.openstack.org/510237 | 21:46 |
| fungi | i've never tried pyenv before, but i guess we should try to find out if it does odd source copying which filters revision control metadata or something | 21:47 |
| fungi | samuel: have you tried just running tox with your system version of python? | 21:48 |
| fungi | like, not with pyenv? | 21:48 |
| samuel | fungi: let me fire up a docker container on debian | 21:49 |
| fungi | i just create a virtualenv, install tox into it, and then link tox in my path to that | 21:49 |
| fungi | but i'll try to get some familiarity with pyenv | 21:50 |
| *** dklyle has quit IRC | 21:52 | |
| fungi | how did you install pyenv? i'm not seeing it packaged in my distro of choice | 21:52 |
| fungi | having never used docker before, that seems like a very deep rabbit hole to descend into on a friday night | 21:53 |
| samuel | okay I'll try in virtualenv | 21:53 |
| samuel | I installed it on Mac OS, via homebrew | 21:53 |
| fungi | OH, so this is not linux | 21:53 |
| fungi | yeah, i'm completely out of my depth when it comes to attempting to run things on proprietary platforms | 21:54 |
| fungi | i'll leave this to people who use non-free stuff | 21:54 |
| *** bobh has quit IRC | 21:54 | |
| samuel | I get the same error in virtualenv with tox installed in it | 21:55 |
| samuel | is there some git setup I need to do before I can use this | 21:56 |
| fungi | my best guess is you have some odd global git configuration which is causing pbr to not find the actual .git subtree in your checkout | 21:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add sub_nodes and node_private files in /etc/nodepool https://review.openstack.org/510213 | 21:56 |
| fungi | what does `git describe` say when your cwd is in the worktree? | 21:57 |
| fungi | i get 2.0.0.0b2-296-g41c54bba | 21:57 |
| samuel | its also not passing any value in the --testr-args I dont know if thats normal: jenkins-job-builder/.tox/py34/bin/python setup.py testr --slowest --testr-args= | 21:57 |
| samuel | same thing: 2.0.0.0b2-296-g41c54bb | 21:58 |
| *** signed8bit is now known as signed8bit_Zzz | 21:58 | |
| fungi | mine has an empty --testr-args= as well and is working, so i don't expect that's related | 21:58 |
| *** Apoorva_ has joined #openstack-infra | 22:00 | |
| fungi | also, you might have more luck asking in the #openstack-jjb channel (it's not a brush-off, i'm in that channel too, we just don't really use jjb much in the openstack-infra at this point and it's developed by a fairly autonomous team now for the past year or more) | 22:01 |
| *** Apoorva has quit IRC | 22:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Provide error message on malformed job list https://review.openstack.org/510185 | 22:01 |
| *** Apoorva_ has quit IRC | 22:02 | |
| *** Apoorva has joined #openstack-infra | 22:02 | |
| samuel | okay so I got it to work in my docker container running debian. I guess I'll get my changes there and test there. I used to be able to run tox though in jjb FWIW before 2.0 | 22:02 |
| fungi | samuel: though the error you're getting seems more related to the pbr library than jjb itself. i'm still pretty baffled. it's like something is messing with its ability to find the git metadata in your checkout | 22:02 |
| *** Apoorva has quit IRC | 22:03 | |
| *** Apoorva has joined #openstack-infra | 22:03 | |
| samuel | i'll volume map the repo into that container to see if it can work with that git metadata | 22:03 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update node requests after nodes https://review.openstack.org/509571 | 22:04 |
| fungi | certainly a head-scratcher, but maybe someone who uses the same operating system you do will recognize the issue straight away. sorry i'm not much help | 22:04 |
| *** iyamahat has quit IRC | 22:05 | |
| mordred | andreaf, jeblair: sorry I missed your convo earlier - was heads-down on the javscript tarball patches | 22:05 |
| mordred | andreaf, jeblair: I thnk you're set though? there is a patch up to report on ansible errors from exit code 1 - https://review.openstack.org/#/c/510219/ and also a patch to fix a few post playbooks https://review.openstack.org/510209 (although I doubt 209 affected any tempest post playbooks) | 22:07 |
| *** jgwentworth is now known as melwitt | 22:08 | |
| jeblair | mordred: ya i think we're all set in both the short and long term. | 22:08 |
| mordred | cool | 22:08 |
| mordred | jlk: I made some piles of ansible you may or may not want to look at https://review.openstack.org/510236 and https://review.openstack.org/510237 | 22:10 |
| mordred | jlk: they're basically ports of the original jjb jobs and macros - didn't redesign them MUCH (although did redesign a few small things) | 22:10 |
| mordred | jeblair: oh neat! did we actually restart zuul with the error message update patch? | 22:11 |
| mordred | jeblair: oh - I see the error message in job output in this case because it's coming from ansible-playbook --syntax-check | 22:12 |
| jeblair | mordred: oh, in the linters job? | 22:12 |
| fungi | mordred: only if it merged and got updated on zuulv3.o.o before i rebooted it at 13:25 utc | 22:13 |
| fungi | it hasn't been restarted again yet since then that i can see | 22:13 |
| jeblair | mordred: maybe we should suggest andreaf run that on tempest? | 22:13 |
| fungi | (at least as far as scheduler restarts go) | 22:13 |
| *** chlong has quit IRC | 22:14 | |
| mordred | jeblair: yah - that might be helpful | 22:15 |
| *** vhosakot has quit IRC | 22:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add javascript tarball publication job https://review.openstack.org/510237 | 22:19 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle non-syntax errors from Ansible https://review.openstack.org/510219 | 22:19 |
| *** ijw has quit IRC | 22:19 | |
| *** ijw has joined #openstack-infra | 22:23 | |
| *** armax has joined #openstack-infra | 22:23 | |
| *** gouthamr has joined #openstack-infra | 22:24 | |
| *** esberglu has joined #openstack-infra | 22:26 | |
| *** esberglu has quit IRC | 22:26 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Provide error message on malformed job list https://review.openstack.org/510185 | 22:27 |
| *** ijw has quit IRC | 22:27 | |
| *** armax has quit IRC | 22:28 | |
| *** baoli has quit IRC | 22:29 | |
| mordred | jeblair: both patches have landed - but I need to run out - if nobody has restarted by the time I get back, I'll do it | 22:30 |
| *** erlon has quit IRC | 22:30 | |
| *** ldnunes has quit IRC | 22:30 | |
| *** rkukura_ has joined #openstack-infra | 22:32 | |
| *** armax has joined #openstack-infra | 22:32 | |
| andreaf | jeblair, mordred: I'll recheck I should see something different now right? | 22:33 |
| *** armax has quit IRC | 22:33 | |
| *** rkukura has quit IRC | 22:34 | |
| *** rkukura_ is now known as rkukura | 22:34 | |
| *** iyamahat has joined #openstack-infra | 22:39 | |
| jeblair | andreaf: we need a restart to see the error; hopefully someone can do that soon | 22:42 |
| jeblair | andreaf: in the mean time, you have the paste i sent you, right? that should at least get you one step closer | 22:43 |
| *** dfflanders has joined #openstack-infra | 22:43 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/bindep master: Add new syntax to allow matching multiple profile https://review.openstack.org/506502 | 22:43 |
| *** samuel has quit IRC | 22:43 | |
| *** leakypipes has quit IRC | 22:43 | |
| *** wolverineav has quit IRC | 22:43 | |
| andreaf | jeblair: thanks, yes that helps | 22:46 |
| *** hashar has quit IRC | 22:50 | |
| *** edmondsw has joined #openstack-infra | 22:53 | |
| *** edmondsw has quit IRC | 22:58 | |
| *** ijw has joined #openstack-infra | 23:03 | |
| *** gouthamr has quit IRC | 23:10 | |
| *** Rockyg has quit IRC | 23:11 | |
| rm_work | the newer gerrit has been killing me with scrolling jumping around like mad in the code preview pages T_T anyone else experiencing this? | 23:12 |
| rm_work | I'm on Chrome on OSX | 23:13 |
| clarkb | rm_work: I've only noticed if I hit spacebar it seems to start cursor at line 1? | 23:15 |
| clarkb | but arrows and mouse scroll seem fine | 23:15 |
| *** tosky has quit IRC | 23:15 | |
| rm_work | hmm k | 23:15 |
| rm_work | i'm doing touchpad scroll, and it's wacky | 23:15 |
| rm_work | also when i click on comments to expand them and reply, it jumps to *some other part of the file* and i have to find it again | 23:15 |
| rm_work | didn't see this unti recently | 23:16 |
| rm_work | like, last week or so I think | 23:16 |
| fungi | it's been ~2 weeks since we upgraded from gerrit 2.11 to 2.13 i think? | 23:16 |
| *** Apoorva has quit IRC | 23:16 | |
| *** bobh has joined #openstack-infra | 23:17 | |
| *** Apoorva has joined #openstack-infra | 23:17 | |
| SpamapS | rm_work: are you zoomed? | 23:18 |
| rm_work | I don't believe so BUT I will double-check ;P | 23:18 |
| SpamapS | I've found that it doesn't behave very consistently when zoomed way in | 23:18 |
| rm_work | nope | 23:18 |
| rm_work | 100% | 23:18 |
| SpamapS | but not jumpy.. just that moving item to item goes further than it should | 23:18 |
| SpamapS | rm_work: time to switch to gertty ;) | 23:18 |
| rm_work | lol yeah maybe | 23:18 |
| rm_work | i wonder if that's in brew... | 23:19 |
| SpamapS | life is so much better when you can do your whole job from a computer no more powerful than a WYSE-60 dumb terminal | 23:19 |
| SpamapS | rm_work: it's in pip | 23:19 |
| rm_work | ah | 23:19 |
| rm_work | k | 23:19 |
| SpamapS | I install it in a virtualenv | 23:19 |
| SpamapS | rm_work: also there's some weirdness on OS X because of ctrl-O | 23:19 |
| rm_work | my whole system runs in a virtulen<_< | 23:19 |
| rm_work | *virtualenv | 23:19 |
| rm_work | in Term, or iTerm2? | 23:20 |
| rm_work | or because of the OS in general | 23:20 |
| SpamapS | if [[ "$TERM" == "xterm-256color" ]] ; then | 23:20 |
| SpamapS | stty discard undef | 23:20 |
| SpamapS | fi | 23:20 |
| SpamapS | I have that in my .profile | 23:20 |
| *** esberglu has joined #openstack-infra | 23:20 | |
| rm_work | hmm k | 23:20 |
| rm_work | i have something related to that set in iterm2 for tmux already | 23:21 |
| rm_work | what would the symptom be? | 23:21 |
| SpamapS | yeah | 23:21 |
| *** iyamahat_ has joined #openstack-infra | 23:21 | |
| SpamapS | ctrl-O does nothing | 23:21 |
| rm_work | and normally i assume it is supposed to do "not nothing", got it | 23:21 |
| *** hongbin has quit IRC | 23:21 | |
| *** iyamahat has quit IRC | 23:21 | |
| SpamapS | ctrl-O is the most important part of gertty | 23:21 |
| SpamapS | search box | 23:21 |
| *** jgriffith is now known as jgriffith_ | 23:24 | |
| *** esberglu has quit IRC | 23:24 | |
| *** Swami has quit IRC | 23:38 | |
| *** claudiub has quit IRC | 23:40 | |
| *** david-lyle has joined #openstack-infra | 23:42 | |
| SamYaple | does zuulv3 currently get *only* 20% of the total quota, even if the other 80% is idle? | 23:43 |
| *** harlowja has quit IRC | 23:45 | |
| *** docaedo has joined #openstack-infra | 23:47 | |
| fungi | SamYaple: yep | 23:49 |
| fungi | we don't have a great mechanism for two nodepools yielding to each other, so end up having to strictly partition the ratio | 23:49 |
| SamYaple | disappointing! but understandable | 23:50 |
| SamYaple | sucks having to wait half a day to find out i had a syntax issue | 23:50 |
| fungi | tests by mail (expect 6 to 8 weeks for delivery) | 23:52 |
| *** yamamoto has joined #openstack-infra | 23:53 | |
| SamYaple | haha | 23:53 |
| SamYaple | i will remit payment to you asap | 23:53 |
| *** markvoelker has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!