| *** lukebrowning has joined #openstack-infra | 00:01 | |
| *** vhosakot has quit IRC | 00:01 | |
| *** iyamahat has quit IRC | 00:01 | |
| *** lukebrowning has quit IRC | 00:05 | |
| *** baoli_ has quit IRC | 00:05 | |
| *** dingyichen has joined #openstack-infra | 00:06 | |
| *** lukebrowning has joined #openstack-infra | 00:07 | |
| *** thorst has joined #openstack-infra | 00:07 | |
| *** thorst has quit IRC | 00:08 | |
| *** kjackal_ has joined #openstack-infra | 00:11 | |
| *** srobert has quit IRC | 00:11 | |
| *** lukebrowning has quit IRC | 00:11 | |
| *** lukebrowning has joined #openstack-infra | 00:13 | |
| *** lukebrowning has quit IRC | 00:18 | |
| jeblair | clarkb: thx. easy +3: https://review.openstack.org/507999 | 00:18 |
|---|---|---|
| *** lukebrowning has joined #openstack-infra | 00:19 | |
| *** xarses has quit IRC | 00:20 | |
| *** baoli has joined #openstack-infra | 00:20 | |
| clarkb | looks like when you recheck zuulv3 enqueues to all the pipelines then I guess removes when it realizes it doens't need to do work in that particulra pipeline? | 00:21 |
| clarkb | jeblair: ^ 508011 is running both check and gate jobs after a recheck | 00:21 |
| jlk | hrm? | 00:21 |
| jlk | oh | 00:21 |
| jlk | so that's likely a pipeline trigger thing | 00:21 |
| clarkb | ya | 00:21 |
| jlk | a pipeline has to accept "recheck" | 00:21 |
| clarkb | I had to recheck because the depends on | 00:21 |
| jlk | otherwise you'd have to have "recheck_check" and "recheck_gate" | 00:21 |
| jlk | name your words after your pipelines | 00:22 |
| clarkb | jlk: right but the other conditions for execution should mean you only run in one or the other | 00:22 |
| clarkb | in this case zuul +1 or +2 should mean no need to rerun check and just run in gate | 00:22 |
| jlk | what we have is all pipelines in general read for "recheck" | 00:22 |
| clarkb | so likely have to tweak that | 00:22 |
| jlk | maybe. | 00:22 |
| jlk | that I dont think is possible at this time | 00:22 |
| *** srobert has joined #openstack-infra | 00:22 | |
| jlk | v2 worked this way too | 00:22 |
| clarkb | v2 did/does what I just described | 00:23 |
| clarkb | you'll go straight to the gate and no check if you have the +1 already | 00:23 |
| clarkb | iirc | 00:23 |
| jlk | huh, that wasn't what we experienced over in bonny land | 00:23 |
| jlk | maybe we were missing some gerrit magic | 00:23 |
| clarkb | ya require approval -1, -2 on comment added in check in v2 | 00:23 |
| jlk | but what do you do when you literally want to re-run the "check" pipeline, even if you got passing votes before? | 00:24 |
| clarkb | which means your recheck comment approval only applies if oyu have a negative vote | 00:24 |
| jlk | oh, yeah I do remember that magic | 00:24 |
| *** lukebrowning has quit IRC | 00:24 | |
| clarkb | we have that in v3 too, but the case is wrong | 00:25 |
| clarkb | so maybe thats the problem? I'll ush up a patch shortly | 00:25 |
| jeblair | clarkb: i'm confused -- it had a +1 and you approved it, why shouldn't it be in gate? | 00:26 |
| jlk | that could very well be the problem. A gerrit version problem | 00:26 |
| clarkb | jeblair: the problem is it is in the check queue | 00:26 |
| clarkb | jeblair: it should be in the gate | 00:26 |
| jeblair | clarkb: i thought you said it was in both | 00:26 |
| clarkb | yes it is in both | 00:26 |
| clarkb | it should only be in the gate | 00:26 |
| jeblair | clarkb: but you said recheck :) | 00:26 |
| clarkb | yes but recheck isn't supposed to apply if you have a positive verified vote | 00:27 |
| clarkb | I beleive this may be broken due to wrong case on those conditions | 00:27 |
| jeblair | really+ | 00:27 |
| jeblair | ? | 00:27 |
| jlk | I think it would be weird if it didn't | 00:27 |
| jeblair | i thought people rechecked things all the time -- like, "is this still okay?" | 00:27 |
| *** slaweq_ has quit IRC | 00:27 | |
| jlk | otherwise you'd have to do a weird dance to -1 vote something to recheck it in the gate | 00:27 |
| clarkb | jeblair: acutally I see we have two comment added conditions | 00:27 |
| *** tosky has quit IRC | 00:27 | |
| clarkb | the one with the verified requirement is for when you apply workflow +1 | 00:28 |
| clarkb | (case problem is still a problem so will fix) | 00:28 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix publish infra docs index job https://review.openstack.org/508011 | 00:28 |
| clarkb | jeblair: I think if you already have +1 workflow and +1/+2 verified then you don't need to also check | 00:28 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 00:29 |
| jeblair | clarkb: strictly speaking, i don't think the case is a bug, but i think it should be okay to move them to proper case. | 00:29 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Fix Verified and Workflow label case in pipelines https://review.openstack.org/508019 | 00:30 |
| clarkb | jeblair: wouldn't it prevent the workflow +1 case with a negative vote from being like a recheck? | 00:30 |
| *** slaweq_ has joined #openstack-infra | 00:32 | |
| *** rhallisey has quit IRC | 00:33 | |
| clarkb | and re the want to see if things still work, I guess if check jobs were the same as gate jobs we wouldn't need to worry about it too much but that isn't always the case | 00:35 |
| clarkb | three is still a race there that gate could pass and merge while check fails (due to running extra tests) | 00:36 |
| *** sree has joined #openstack-infra | 00:36 | |
| *** yee37927 has quit IRC | 00:39 | |
| *** rhallisey has joined #openstack-infra | 00:39 | |
| *** yee379 has joined #openstack-infra | 00:39 | |
| *** xarses_ has quit IRC | 00:39 | |
| *** xarses_ has joined #openstack-infra | 00:39 | |
| *** sree has quit IRC | 00:41 | |
| jeblair | clarkb: hrm, those should be case sensitive now. however, if they are, how has workflow+1 worked at all? | 00:41 |
| *** sree has joined #openstack-infra | 00:41 | |
| jeblair | oh! | 00:41 |
| jeblair | it's correct on gate | 00:42 |
| *** kiennt26 has joined #openstack-infra | 00:42 | |
| *** sree has quit IRC | 00:42 | |
| *** sree has joined #openstack-infra | 00:42 | |
| clarkb | ya, this is a small subset of cases | 00:42 |
| jeblair | okay. 508019+2 :) | 00:42 |
| jeblair | clarkb: and yes, this will fix the "workflow +1 case with a negative vote" should act like a recheck case. | 00:43 |
| jeblair | i'm all caught up now. :) | 00:43 |
| jeblair | added as migration blocker | 00:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 00:45 |
| *** lukebrowning has joined #openstack-infra | 00:45 | |
| *** nikhil has joined #openstack-infra | 00:46 | |
| *** dave-mccowan has joined #openstack-infra | 00:46 | |
| *** zhurong has joined #openstack-infra | 00:47 | |
| *** lukebrowning has quit IRC | 00:50 | |
| *** Swami has quit IRC | 00:50 | |
| *** lukebrowning has joined #openstack-infra | 00:51 | |
| *** kzaitsev1pi has quit IRC | 00:52 | |
| *** xarses has joined #openstack-infra | 00:54 | |
| *** kzaitsev_pi has joined #openstack-infra | 00:54 | |
| *** lukebrowning has quit IRC | 00:56 | |
| *** mriedem has quit IRC | 00:59 | |
| *** dizquierdo has quit IRC | 01:00 | |
| *** liusheng has quit IRC | 01:04 | |
| *** lukebrowning has joined #openstack-infra | 01:05 | |
| *** slaweq_ has quit IRC | 01:05 | |
| *** camunoz has quit IRC | 01:05 | |
| *** thorst has joined #openstack-infra | 01:09 | |
| *** lukebrowning has quit IRC | 01:09 | |
| *** lennyb has quit IRC | 01:11 | |
| *** lukebrowning has joined #openstack-infra | 01:11 | |
| *** lennyb has joined #openstack-infra | 01:11 | |
| *** slaweq_ has joined #openstack-infra | 01:13 | |
| *** sree has quit IRC | 01:15 | |
| *** sree has joined #openstack-infra | 01:15 | |
| *** lukebrowning has quit IRC | 01:16 | |
| *** liusheng has joined #openstack-infra | 01:16 | |
| *** lukebrowning has joined #openstack-infra | 01:18 | |
| *** sree has quit IRC | 01:20 | |
| jlvillal | So is wonkyness a known thing with the gate? | 01:20 |
| jlvillal | Had a "coverage" job which seemed to do a bunch of unit tests. Which seems strange to me. | 01:20 |
| jlvillal | http://logs.openstack.org/68/507668/9/check/ironic-coverage-ubuntu-xenial/473583f/console.html | 01:20 |
| *** lukebrowning has quit IRC | 01:22 | |
| *** mikal_ has joined #openstack-infra | 01:22 | |
| *** hongbin has joined #openstack-infra | 01:22 | |
| *** eharney has quit IRC | 01:22 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate master: Remove ETCD_DOWNLOAD_URL setting for devstack https://review.openstack.org/508022 | 01:23 |
| ianw | jlvillal: that's how coverage works? runs unit tests under coverage tool? | 01:23 |
| *** thorst has quit IRC | 01:23 | |
| *** thorst has joined #openstack-infra | 01:23 | |
| *** lukebrowning has joined #openstack-infra | 01:24 | |
| jlvillal | ianw, Oh. Thanks. I think I vaguely knew that. I have been seeing this weird unit test failure randomly and that happened during the coverage job. | 01:24 |
| *** mikal has quit IRC | 01:25 | |
| *** thorst has quit IRC | 01:26 | |
| *** mikal_ has quit IRC | 01:28 | |
| ianw | yep; coverage overheads might exacerbate it. or hide it. that's the fun of bugs :) | 01:28 |
| *** jdandrea_ has joined #openstack-infra | 01:28 | |
| *** masayukig has quit IRC | 01:28 | |
| *** lukebrowning has quit IRC | 01:28 | |
| *** masayukig has joined #openstack-infra | 01:29 | |
| *** lukebrowning has joined #openstack-infra | 01:30 | |
| jlvillal | strange one, keep seeing a random failure in the unit tests when running under Python 2.7. | 01:30 |
| *** gcb has joined #openstack-infra | 01:31 | |
| *** cuongnv has joined #openstack-infra | 01:32 | |
| *** bnemec has quit IRC | 01:34 | |
| SamYaple | have you ever said to yourself "I could really go for some Parsley grammar reviews right now"? If so, this patch set is for you! https://review.openstack.org/#/c/506502/ | 01:35 |
| SamYaple | if not please still give me reviews :( | 01:35 |
| *** eharney has joined #openstack-infra | 01:36 | |
| *** bnemec has joined #openstack-infra | 01:39 | |
| *** liujiong has joined #openstack-infra | 01:40 | |
| *** lukebrowning has quit IRC | 01:41 | |
| jlvillal | SamYaple, I know nothing about it and I'm not a core. But I looked at the unit test and wished it had some comments :) | 01:41 |
| *** dave-mccowan has quit IRC | 01:41 | |
| SamYaple | jlvillal: i will happily extent the unit tests further, but I did copy the pattern of the other unit tests | 01:43 |
| *** esberglu has quit IRC | 01:43 | |
| SamYaple | if you leave a review with what you want to see, ill add it | 01:43 |
| *** lukebrowning has joined #openstack-infra | 01:43 | |
| *** slaweq_ has quit IRC | 01:43 | |
| SamYaple | jlvillal: and core reviewers have merge power, this is opensoure everyones reviews count remember :) | 01:44 |
| *** dave-mccowan has joined #openstack-infra | 01:45 | |
| *** LindaWang has joined #openstack-infra | 01:46 | |
| *** lukebrowning has quit IRC | 01:48 | |
| *** lukebrowning has joined #openstack-infra | 01:49 | |
| *** slaweq_ has joined #openstack-infra | 01:50 | |
| *** sree has joined #openstack-infra | 01:52 | |
| *** lukebrowning has quit IRC | 01:54 | |
| *** mikal has joined #openstack-infra | 01:56 | |
| *** lukebrowning has joined #openstack-infra | 01:56 | |
| *** lukebrowning has quit IRC | 02:00 | |
| *** lukebrowning has joined #openstack-infra | 02:02 | |
| *** dave-mcc_ has joined #openstack-infra | 02:04 | |
| *** sdague has quit IRC | 02:05 | |
| *** lukebrowning has quit IRC | 02:07 | |
| *** dave-mccowan has quit IRC | 02:07 | |
| *** sree has quit IRC | 02:08 | |
| *** lukebrowning has joined #openstack-infra | 02:08 | |
| *** Apoorva_ has quit IRC | 02:10 | |
| *** lukebrowning has quit IRC | 02:13 | |
| *** lukebrowning has joined #openstack-infra | 02:14 | |
| *** lewo` has quit IRC | 02:16 | |
| *** lewo` has joined #openstack-infra | 02:17 | |
| *** lukebrowning has quit IRC | 02:19 | |
| *** krtaylor has joined #openstack-infra | 02:20 | |
| *** lukebrowning has joined #openstack-infra | 02:21 | |
| *** yamamoto has joined #openstack-infra | 02:22 | |
| *** yamamoto has quit IRC | 02:22 | |
| *** slaweq_ has quit IRC | 02:22 | |
| *** yamamoto has joined #openstack-infra | 02:22 | |
| *** baoli has quit IRC | 02:24 | |
| *** lukebrowning has quit IRC | 02:25 | |
| *** thorst has joined #openstack-infra | 02:26 | |
| *** slaweq_ has joined #openstack-infra | 02:27 | |
| *** ramishra has joined #openstack-infra | 02:30 | |
| *** namnh has joined #openstack-infra | 02:31 | |
| *** esberglu has joined #openstack-infra | 02:32 | |
| *** lukebrowning has joined #openstack-infra | 02:32 | |
| *** baoli has joined #openstack-infra | 02:34 | |
| *** esberglu has quit IRC | 02:36 | |
| *** lukebrowning has quit IRC | 02:37 | |
| *** dave-mcc_ has quit IRC | 02:38 | |
| *** lukebrowning has joined #openstack-infra | 02:38 | |
| *** baoli has quit IRC | 02:42 | |
| *** lukebrowning has quit IRC | 02:43 | |
| *** baoli has joined #openstack-infra | 02:44 | |
| *** vivsoni has quit IRC | 02:44 | |
| *** lukebrowning has joined #openstack-infra | 02:45 | |
| *** lukebrowning has quit IRC | 02:49 | |
| *** Rockyg has joined #openstack-infra | 02:50 | |
| *** kiennt26 has quit IRC | 02:51 | |
| *** lukebrowning has joined #openstack-infra | 02:51 | |
| *** srobert has quit IRC | 02:55 | |
| *** nikhil has quit IRC | 02:55 | |
| *** lukebrowning has quit IRC | 02:56 | |
| *** lukebrowning has joined #openstack-infra | 02:57 | |
| *** slaweq_ has quit IRC | 03:00 | |
| *** lukebrowning has quit IRC | 03:02 | |
| *** lukebrowning has joined #openstack-infra | 03:03 | |
| *** wolverineav has quit IRC | 03:06 | |
| *** wolverineav has joined #openstack-infra | 03:07 | |
| *** lukebrowning has quit IRC | 03:08 | |
| *** slaweq_ has joined #openstack-infra | 03:09 | |
| *** lukebrowning has joined #openstack-infra | 03:10 | |
| *** wolverineav has quit IRC | 03:11 | |
| *** lukebrowning has quit IRC | 03:14 | |
| *** mika has quit IRC | 03:15 | |
| *** Sukhdev has quit IRC | 03:15 | |
| *** lukebrowning has joined #openstack-infra | 03:16 | |
| *** lukebrowning has quit IRC | 03:20 | |
| *** baoli has quit IRC | 03:21 | |
| *** baoli has joined #openstack-infra | 03:22 | |
| *** lukebrowning has joined #openstack-infra | 03:22 | |
| *** mika has joined #openstack-infra | 03:23 | |
| *** Sukhdev has joined #openstack-infra | 03:25 | |
| *** bnemec has quit IRC | 03:26 | |
| *** lukebrowning has quit IRC | 03:27 | |
| *** rlandy|afk is now known as rlandy | 03:30 | |
| *** rlandy has quit IRC | 03:31 | |
| *** jzerebec1i has quit IRC | 03:32 | |
| *** xarses has quit IRC | 03:33 | |
| *** MasterofJOKers has quit IRC | 03:34 | |
| *** baoli has quit IRC | 03:34 | |
| *** wolverineav has joined #openstack-infra | 03:37 | |
| *** jzerebecki has joined #openstack-infra | 03:39 | |
| *** MasterofJOKers has joined #openstack-infra | 03:39 | |
| *** slaweq_ has quit IRC | 03:41 | |
| *** udesale has joined #openstack-infra | 03:42 | |
| *** wolverineav has quit IRC | 03:42 | |
| *** jdandrea_ has quit IRC | 03:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix Verified and Workflow label case in pipelines https://review.openstack.org/508019 | 03:43 |
| *** links has joined #openstack-infra | 03:44 | |
| *** slaweq_ has joined #openstack-infra | 03:45 | |
| *** wolverineav has joined #openstack-infra | 03:51 | |
| *** tdasilva has quit IRC | 03:53 | |
| *** wolverineav has quit IRC | 03:56 | |
| *** jdandrea_ has joined #openstack-infra | 04:05 | |
| *** zhurong has quit IRC | 04:07 | |
| *** tdasilva has joined #openstack-infra | 04:11 | |
| *** ilpianista has quit IRC | 04:14 | |
| *** wolverineav has joined #openstack-infra | 04:15 | |
| *** slaweq_ has quit IRC | 04:18 | |
| *** ykarel has joined #openstack-infra | 04:20 | |
| *** wolverineav has quit IRC | 04:20 | |
| *** lukebrowning has joined #openstack-infra | 04:23 | |
| *** kornicameister has quit IRC | 04:24 | |
| *** slaweq_ has joined #openstack-infra | 04:24 | |
| *** wolverineav has joined #openstack-infra | 04:28 | |
| *** hongbin has quit IRC | 04:31 | |
| *** ankkumar has joined #openstack-infra | 04:31 | |
| *** wolverineav has quit IRC | 04:32 | |
| *** ilpianista has joined #openstack-infra | 04:33 | |
| *** lukebrowning has quit IRC | 04:35 | |
| *** lukebrowning has joined #openstack-infra | 04:37 | |
| *** sree has joined #openstack-infra | 04:38 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Emit a message about the final tox invocation https://review.openstack.org/507990 | 04:39 |
| *** lukebrowning has quit IRC | 04:41 | |
| *** Rockyg has quit IRC | 04:42 | |
| *** lukebrowning has joined #openstack-infra | 04:43 | |
| *** jdandrea_ has quit IRC | 04:48 | |
| *** lukebrowning has quit IRC | 04:52 | |
| *** wolverineav has joined #openstack-infra | 04:52 | |
| *** psachin has joined #openstack-infra | 04:52 | |
| *** lukebrowning has joined #openstack-infra | 04:54 | |
| *** slaweq_ has quit IRC | 04:56 | |
| *** wolverineav has quit IRC | 04:57 | |
| *** bhavik1 has joined #openstack-infra | 04:57 | |
| *** lukebrowning has quit IRC | 04:58 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move fstrim to block device layer https://review.openstack.org/504579 | 04:58 |
| ianw | ^ this might shrink image sizes, andreas realised the current fstrim call was ineffective | 04:59 |
| *** lukebrowning has joined #openstack-infra | 05:00 | |
| *** slaweq_ has joined #openstack-infra | 05:03 | |
| *** lukebrowning has quit IRC | 05:05 | |
| *** lukebrowning has joined #openstack-infra | 05:15 | |
| *** lukebrowning has quit IRC | 05:19 | |
| *** lukebrowning has joined #openstack-infra | 05:21 | |
| *** dfflanders has joined #openstack-infra | 05:22 | |
| *** bhavik1 has quit IRC | 05:23 | |
| *** lukebrowning has quit IRC | 05:25 | |
| *** Sukhdev has quit IRC | 05:29 | |
| *** slaweq_ has quit IRC | 05:33 | |
| *** claudiub|3 has joined #openstack-infra | 05:37 | |
| *** dhajare has joined #openstack-infra | 05:38 | |
| *** slaweq_ has joined #openstack-infra | 05:43 | |
| *** dgonzalez has joined #openstack-infra | 05:50 | |
| *** mrunge_ is now known as mrunge | 06:00 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 06:02 |
| *** iyamahat has joined #openstack-infra | 06:02 | |
| *** gk_ has joined #openstack-infra | 06:03 | |
| *** lukebrowning has joined #openstack-infra | 06:06 | |
| *** eranrom has quit IRC | 06:09 | |
| *** esberglu has joined #openstack-infra | 06:11 | |
| *** erlon has quit IRC | 06:14 | |
| *** esberglu has quit IRC | 06:15 | |
| *** iyamahat has quit IRC | 06:16 | |
| *** eranrom has joined #openstack-infra | 06:17 | |
| *** lukebrowning has quit IRC | 06:18 | |
| *** pcaruana has joined #openstack-infra | 06:19 | |
| *** coolsvap has joined #openstack-infra | 06:20 | |
| *** lukebrowning has joined #openstack-infra | 06:20 | |
| *** Guest90591 has quit IRC | 06:21 | |
| *** mrunge has quit IRC | 06:22 | |
| *** lukebrowning has quit IRC | 06:25 | |
| *** lukebrowning has joined #openstack-infra | 06:26 | |
| *** mrunge has joined #openstack-infra | 06:27 | |
| *** markvoelker has quit IRC | 06:30 | |
| *** lukebrowning has quit IRC | 06:31 | |
| gk_ | Hi everyone, I have a question about the third party ci. When Zuul triggers a new job for jenkins, the variable "zuul_change" is empty. It should contain the information about the patch. Someone has a clue on how to solve it? Thank you in advance. | 06:32 |
| *** lukebrowning has joined #openstack-infra | 06:32 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/shade master: [WIP] Avoid tox_install.sh fro constraints support https://review.openstack.org/508061 | 06:33 |
| *** lukebrowning has quit IRC | 06:37 | |
| *** zhurong has joined #openstack-infra | 06:38 | |
| *** lukebrowning has joined #openstack-infra | 06:39 | |
| *** lukebrowning has quit IRC | 06:43 | |
| *** eranrom has quit IRC | 06:44 | |
| *** lukebrowning has joined #openstack-infra | 06:45 | |
| *** aeng has quit IRC | 06:46 | |
| *** slaweq_ has quit IRC | 06:46 | |
| *** pgadiya has joined #openstack-infra | 06:48 | |
| prometheanfire | so how'd yesterday go? v3 deployed? | 06:49 |
| *** rossella_s has joined #openstack-infra | 06:49 | |
| *** lukebrowning has quit IRC | 06:49 | |
| *** andreas_s has joined #openstack-infra | 06:50 | |
| *** florianf has joined #openstack-infra | 06:51 | |
| *** lukebrowning has joined #openstack-infra | 06:51 | |
| *** eranrom has joined #openstack-infra | 06:52 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/release-tools master: Fix abandoning reviews in eol_branch https://review.openstack.org/508067 | 06:52 |
| *** slaweq_ has joined #openstack-infra | 06:53 | |
| *** makowals has joined #openstack-infra | 06:54 | |
| frickler | prometheanfire: looks like it: http://lists.openstack.org/pipermail/openstack-dev/2017-September/122755.html | 06:54 |
| frickler | ah, no, ready but not fired | 06:54 |
| *** lukebrowning has quit IRC | 06:56 | |
| prometheanfire | oh, missed that one | 06:56 |
| *** lukebrowning has joined #openstack-infra | 06:57 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move fstrim to block device layer https://review.openstack.org/504579 | 06:58 |
| *** rcernin has joined #openstack-infra | 07:00 | |
| *** lukebrowning has quit IRC | 07:02 | |
| *** eranrom has quit IRC | 07:06 | |
| *** andymccr has quit IRC | 07:07 | |
| *** armax has joined #openstack-infra | 07:08 | |
| *** lukebrowning has joined #openstack-infra | 07:08 | |
| *** asettle has quit IRC | 07:08 | |
| *** andymccr has joined #openstack-infra | 07:09 | |
| *** asettle has joined #openstack-infra | 07:10 | |
| *** CHIPPY has joined #openstack-infra | 07:10 | |
| *** asettle is now known as Guest3828 | 07:10 | |
| *** lukebrowning has quit IRC | 07:12 | |
| *** iyamahat has joined #openstack-infra | 07:14 | |
| *** lukebrowning has joined #openstack-infra | 07:14 | |
| *** armax has quit IRC | 07:14 | |
| *** ralonsoh has joined #openstack-infra | 07:19 | |
| *** lukebrowning has quit IRC | 07:19 | |
| *** lukebrowning has joined #openstack-infra | 07:21 | |
| *** andreas_s_ has joined #openstack-infra | 07:21 | |
| *** andreas_s has quit IRC | 07:21 | |
| *** lukebrowning has quit IRC | 07:25 | |
| *** lukebrowning has joined #openstack-infra | 07:27 | |
| *** slaweq_ has quit IRC | 07:27 | |
| *** CHIPPY has left #openstack-infra | 07:29 | |
| *** shardy has joined #openstack-infra | 07:29 | |
| *** slaweq_ has joined #openstack-infra | 07:30 | |
| *** lukebrowning has quit IRC | 07:31 | |
| *** lukebrowning has joined #openstack-infra | 07:33 | |
| *** jpena|off is now known as jpena | 07:34 | |
| *** armax has joined #openstack-infra | 07:36 | |
| *** eranrom has joined #openstack-infra | 07:36 | |
| *** lukebrowning has quit IRC | 07:38 | |
| *** lukebrowning has joined #openstack-infra | 07:39 | |
| *** armax has quit IRC | 07:42 | |
| *** lukebrowning has quit IRC | 07:44 | |
| *** jpich has joined #openstack-infra | 07:44 | |
| *** dfflanders has quit IRC | 07:44 | |
| *** ccamacho has quit IRC | 07:45 | |
| *** ccamacho has joined #openstack-infra | 07:45 | |
| *** lukebrowning has joined #openstack-infra | 07:45 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move fstrim to block device layer https://review.openstack.org/504579 | 07:49 |
| *** egonzalez has joined #openstack-infra | 07:50 | |
| *** lukebrowning has quit IRC | 07:50 | |
| *** xinliang has quit IRC | 07:51 | |
| *** lukebrowning has joined #openstack-infra | 07:52 | |
| *** wolverineav has joined #openstack-infra | 07:52 | |
| *** yamamoto has quit IRC | 07:56 | |
| *** lukebrowning has quit IRC | 07:56 | |
| *** wolverineav has quit IRC | 07:57 | |
| *** threestrands has quit IRC | 07:58 | |
| *** lukebrowning has joined #openstack-infra | 07:58 | |
| *** esberglu has joined #openstack-infra | 07:59 | |
| *** slaweq_ has quit IRC | 08:02 | |
| *** lukebrowning has quit IRC | 08:02 | |
| *** rossella_s has quit IRC | 08:03 | |
| *** yamamoto has joined #openstack-infra | 08:03 | |
| *** esberglu has quit IRC | 08:03 | |
| *** xinliang has joined #openstack-infra | 08:05 | |
| *** xinliang has quit IRC | 08:05 | |
| *** xinliang has joined #openstack-infra | 08:05 | |
| *** electrofelix has joined #openstack-infra | 08:05 | |
| *** wolverineav has joined #openstack-infra | 08:07 | |
| *** lukebrowning has joined #openstack-infra | 08:09 | |
| *** slaweq_ has joined #openstack-infra | 08:09 | |
| *** iyamahat has quit IRC | 08:10 | |
| *** wolverineav has quit IRC | 08:12 | |
| *** lukebrowning has quit IRC | 08:13 | |
| *** eglute has quit IRC | 08:14 | |
| *** eglute has joined #openstack-infra | 08:14 | |
| *** lukebrowning has joined #openstack-infra | 08:15 | |
| *** lukebrowning has quit IRC | 08:20 | |
| *** hashar has joined #openstack-infra | 08:20 | |
| *** lukebrowning has joined #openstack-infra | 08:21 | |
| *** alexchadin has joined #openstack-infra | 08:22 | |
| *** yamamoto has quit IRC | 08:25 | |
| *** derekh has joined #openstack-infra | 08:26 | |
| *** lukebrowning has quit IRC | 08:26 | |
| *** wolverineav has joined #openstack-infra | 08:29 | |
| *** yamamoto has joined #openstack-infra | 08:30 | |
| *** markvoelker has joined #openstack-infra | 08:31 | |
| *** yamamoto_ has joined #openstack-infra | 08:32 | |
| *** yamamoto_ has quit IRC | 08:34 | |
| *** wolverineav has quit IRC | 08:34 | |
| *** dizquierdo has joined #openstack-infra | 08:34 | |
| *** yamamoto has quit IRC | 08:35 | |
| *** gongysh has joined #openstack-infra | 08:35 | |
| *** lukebrowning has joined #openstack-infra | 08:35 | |
| *** yamahata has joined #openstack-infra | 08:36 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/release-tools master: Fix abandoning reviews in eol_branch https://review.openstack.org/508067 | 08:38 |
| *** Guest3828 is now known as asettle | 08:39 | |
| *** jesusaur has quit IRC | 08:40 | |
| *** lukebrowning has quit IRC | 08:40 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Added tempest dsvm job for neutron-tempest-plugin https://review.openstack.org/507038 | 08:41 |
| *** lukebrowning has joined #openstack-infra | 08:42 | |
| *** e0ne has joined #openstack-infra | 08:42 | |
| *** slaweq_ has quit IRC | 08:43 | |
| *** lukebrowning has quit IRC | 08:46 | |
| *** yamahata has quit IRC | 08:46 | |
| *** yamahata has joined #openstack-infra | 08:47 | |
| *** gk_ has quit IRC | 08:47 | |
| *** lukebrowning has joined #openstack-infra | 08:48 | |
| *** jesusaur has joined #openstack-infra | 08:48 | |
| *** alexchadin has quit IRC | 08:50 | |
| *** alexchadin has joined #openstack-infra | 08:51 | |
| *** alexchadin has quit IRC | 08:51 | |
| *** alexchadin has joined #openstack-infra | 08:51 | |
| *** lukebrowning has quit IRC | 08:53 | |
| *** dingyichen has quit IRC | 08:54 | |
| *** slaweq_ has joined #openstack-infra | 08:54 | |
| *** lukebrowning has joined #openstack-infra | 08:54 | |
| *** lukebrowning has quit IRC | 08:59 | |
| *** ralonsoh_ has joined #openstack-infra | 09:00 | |
| *** lukebrowning has joined #openstack-infra | 09:00 | |
| *** alexchadin has quit IRC | 09:01 | |
| *** alexchadin has joined #openstack-infra | 09:03 | |
| *** gongysh has quit IRC | 09:03 | |
| *** shardy has quit IRC | 09:03 | |
| *** ralonsoh has quit IRC | 09:03 | |
| *** lukebrowning has quit IRC | 09:05 | |
| *** markvoelker has quit IRC | 09:05 | |
| *** lukebrowning has joined #openstack-infra | 09:07 | |
| *** wolverineav has joined #openstack-infra | 09:08 | |
| *** eroux has joined #openstack-infra | 09:10 | |
| *** lukebrowning has quit IRC | 09:11 | |
| *** wolverineav has quit IRC | 09:12 | |
| *** lukebrowning has joined #openstack-infra | 09:13 | |
| *** yamamoto has joined #openstack-infra | 09:14 | |
| *** yamamoto has quit IRC | 09:17 | |
| *** lukebrowning has quit IRC | 09:18 | |
| *** lukebrowning has joined #openstack-infra | 09:19 | |
| *** lukebrowning has quit IRC | 09:24 | |
| *** sambetts|afk is now known as sambetts | 09:24 | |
| *** slaweq_ has quit IRC | 09:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix typo with ovh-bhs1 https://review.openstack.org/508005 | 09:26 |
| *** alexchad_ has joined #openstack-infra | 09:28 | |
| *** alexchadin has quit IRC | 09:30 | |
| *** alexchad_ has quit IRC | 09:32 | |
| *** alexchadin has joined #openstack-infra | 09:33 | |
| *** slaweq_ has joined #openstack-infra | 09:36 | |
| *** armax has joined #openstack-infra | 09:38 | |
| *** yamahata has quit IRC | 09:40 | |
| *** wolverineav has joined #openstack-infra | 09:41 | |
| *** armax has quit IRC | 09:42 | |
| *** sdague has joined #openstack-infra | 09:44 | |
| *** wolverineav has quit IRC | 09:46 | |
| *** ijw has quit IRC | 09:47 | |
| *** esberglu has joined #openstack-infra | 09:47 | |
| *** esberglu has quit IRC | 09:51 | |
| *** sree has quit IRC | 09:57 | |
| *** jpena has quit IRC | 09:59 | |
| *** amoralej has quit IRC | 09:59 | |
| *** ltomasbo has quit IRC | 09:59 | |
| *** markvoelker has joined #openstack-infra | 10:02 | |
| *** yamamoto has joined #openstack-infra | 10:03 | |
| *** ralonsoh__ has joined #openstack-infra | 10:03 | |
| *** rossella_s has joined #openstack-infra | 10:03 | |
| *** ltomasbo has joined #openstack-infra | 10:04 | |
| *** amoralej has joined #openstack-infra | 10:04 | |
| *** jpena has joined #openstack-infra | 10:05 | |
| *** yamamoto has quit IRC | 10:05 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results https://review.openstack.org/507980 | 10:06 |
| *** ralonsoh_ has quit IRC | 10:06 | |
| *** masber has quit IRC | 10:06 | |
| *** masber has joined #openstack-infra | 10:07 | |
| *** _ari_ has quit IRC | 10:08 | |
| *** cuongnv has quit IRC | 10:08 | |
| *** liujiong has quit IRC | 10:09 | |
| *** slaweq_ has quit IRC | 10:09 | |
| *** pgadiya has quit IRC | 10:13 | |
| *** _ari_ has joined #openstack-infra | 10:13 | |
| *** wolverineav has joined #openstack-infra | 10:14 | |
| *** slaweq_ has joined #openstack-infra | 10:17 | |
| *** wolverineav has quit IRC | 10:19 | |
| *** lukebrowning has joined #openstack-infra | 10:20 | |
| *** ralonsoh__ is now known as ralonsoh | 10:24 | |
| *** rossella_s has quit IRC | 10:25 | |
| *** zhurong has quit IRC | 10:27 | |
| *** yamamoto has joined #openstack-infra | 10:29 | |
| *** yamamoto has quit IRC | 10:31 | |
| *** alexchadin has quit IRC | 10:32 | |
| *** lukebrowning has quit IRC | 10:32 | |
| *** markvoelker has quit IRC | 10:35 | |
| *** jkilpatr has quit IRC | 10:35 | |
| *** iyamahat has joined #openstack-infra | 10:35 | |
| *** iyamahat has quit IRC | 10:36 | |
| *** iyamahat_ has joined #openstack-infra | 10:36 | |
| *** yamamoto has joined #openstack-infra | 10:37 | |
| *** yamamoto has quit IRC | 10:38 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Make wait_for print the failed ouput. https://review.openstack.org/508107 | 10:39 |
| *** udesale has quit IRC | 10:39 | |
| *** yamamoto has joined #openstack-infra | 10:40 | |
| kashyap | Hi folks, can someone give me a URL to where the source for the 'gate-grenade-dsvm-neutron-multinode-live-migration-nv' job, please? | 10:42 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Make wait_for print the failed ouput. https://review.openstack.org/508107 | 10:42 |
| *** yamamoto has quit IRC | 10:43 | |
| *** ykarel is now known as ykarel|afk | 10:44 | |
| *** lukebrowning has joined #openstack-infra | 10:44 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Make wait_for print the failed ouput. https://review.openstack.org/508107 | 10:45 |
| *** andreas_s has joined #openstack-infra | 10:45 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Make wait_for print the failed ouput. https://review.openstack.org/508107 | 10:46 |
| *** yamamoto has joined #openstack-infra | 10:46 | |
| *** wolverineav has joined #openstack-infra | 10:47 | |
| *** ijw has joined #openstack-infra | 10:47 | |
| *** andreas_s_ has quit IRC | 10:48 | |
| *** yamamoto has quit IRC | 10:49 | |
| *** lukebrowning has quit IRC | 10:49 | |
| *** gcb has quit IRC | 10:50 | |
| *** lukebrowning has joined #openstack-infra | 10:51 | |
| *** wolverineav has quit IRC | 10:51 | |
| kashyap | AJaeger_: Perhaps you'd know the answer to my above question | 10:51 |
| *** jkilpatr has joined #openstack-infra | 10:52 | |
| *** ijw has quit IRC | 10:52 | |
| *** slaweq_ has quit IRC | 10:53 | |
| *** yolanda has quit IRC | 10:54 | |
| *** tosky has joined #openstack-infra | 10:55 | |
| *** lukebrowning has quit IRC | 10:55 | |
| *** lukebrowning has joined #openstack-infra | 10:57 | |
| *** amoralej is now known as amoralej|lunch | 10:59 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/irc-meetings master: Update chair for Monasca Team Meeting https://review.openstack.org/508110 | 10:59 |
| *** lukebrowning has quit IRC | 11:01 | |
| *** ykarel|afk is now known as ykarel | 11:02 | |
| *** lukebrowning has joined #openstack-infra | 11:03 | |
| *** gcb has joined #openstack-infra | 11:07 | |
| *** lukebrowning has quit IRC | 11:07 | |
| *** yolanda has joined #openstack-infra | 11:08 | |
| *** lukebrowning has joined #openstack-infra | 11:09 | |
| *** lukebrowning has quit IRC | 11:14 | |
| *** kjackal_ has quit IRC | 11:14 | |
| *** lukebrowning has joined #openstack-infra | 11:15 | |
| *** thorst has quit IRC | 11:16 | |
| *** wolverineav has joined #openstack-infra | 11:19 | |
| *** lukebrowning has quit IRC | 11:20 | |
| *** shardy has joined #openstack-infra | 11:21 | |
| *** lukebrowning has joined #openstack-infra | 11:22 | |
| *** slaweq_ has joined #openstack-infra | 11:23 | |
| *** wolverineav has quit IRC | 11:24 | |
| *** pbourke has quit IRC | 11:25 | |
| *** pbourke has joined #openstack-infra | 11:25 | |
| *** lukebrowning has quit IRC | 11:26 | |
| *** yamamoto has joined #openstack-infra | 11:28 | |
| *** lukebrowning has joined #openstack-infra | 11:28 | |
| *** markvoelker has joined #openstack-infra | 11:32 | |
| *** lukebrowning has quit IRC | 11:32 | |
| *** lukebrowning has joined #openstack-infra | 11:34 | |
| *** esberglu has joined #openstack-infra | 11:35 | |
| *** lukebrowning has quit IRC | 11:38 | |
| *** esberglu has quit IRC | 11:39 | |
| *** lukebrowning has joined #openstack-infra | 11:40 | |
| *** jaosorior_sick is now known as jaosorior | 11:41 | |
| *** stakeda has quit IRC | 11:44 | |
| *** lukebrowning has quit IRC | 11:45 | |
| *** andreas_s_ has joined #openstack-infra | 11:47 | |
| *** andreas_s has quit IRC | 11:51 | |
| *** lukebrowning has joined #openstack-infra | 11:51 | |
| *** lvdombrk has quit IRC | 11:54 | |
| *** trown has joined #openstack-infra | 11:55 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Add separate coverage job for ironic-inspector https://review.openstack.org/508129 | 11:56 |
| *** lukebrowning has quit IRC | 11:56 | |
| *** slaweq_ has quit IRC | 11:57 | |
| *** lukebrowning has joined #openstack-infra | 11:57 | |
| pabelanger | just a heads up, I'll be in and out all day today, packing our condo up. I don't expect it to take all day, and should be able to help with cutover bits at a time | 11:59 |
| *** slaweq_ has joined #openstack-infra | 12:01 | |
| *** lukebrowning has quit IRC | 12:02 | |
| *** AJaeger_ has quit IRC | 12:02 | |
| *** yamamoto has quit IRC | 12:03 | |
| *** bobh has joined #openstack-infra | 12:03 | |
| *** thorst has joined #openstack-infra | 12:03 | |
| *** lukebrowning has joined #openstack-infra | 12:04 | |
| *** tpsilva has joined #openstack-infra | 12:05 | |
| *** markvoelker has quit IRC | 12:06 | |
| *** iyamahat_ has quit IRC | 12:06 | |
| *** eharney has quit IRC | 12:06 | |
| *** bobh has quit IRC | 12:07 | |
| *** lukebrowning has quit IRC | 12:08 | |
| *** lukebrowning has joined #openstack-infra | 12:10 | |
| *** gcb has quit IRC | 12:12 | |
| *** gcb has joined #openstack-infra | 12:12 | |
| *** lukebrowning has quit IRC | 12:14 | |
| *** lukebrowning has joined #openstack-infra | 12:16 | |
| *** srobert has joined #openstack-infra | 12:17 | |
| *** ankkumar_ has joined #openstack-infra | 12:18 | |
| *** markvoelker has joined #openstack-infra | 12:18 | |
| *** ankkumar_ has quit IRC | 12:18 | |
| *** ankkumar has quit IRC | 12:20 | |
| *** dizquierdo has quit IRC | 12:20 | |
| *** trown has quit IRC | 12:21 | |
| *** lukebrowning has quit IRC | 12:21 | |
| *** lukebrowning has joined #openstack-infra | 12:23 | |
| *** kjackal_ has joined #openstack-infra | 12:23 | |
| *** amoralej|lunch is now known as amoralej | 12:25 | |
| *** lukebrowning has quit IRC | 12:27 | |
| *** lukebrowning has joined #openstack-infra | 12:29 | |
| smcginnis | kashyap: I think this is what you're looking for: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L1626 | 12:29 |
| *** dprince has joined #openstack-infra | 12:29 | |
| *** srobert has quit IRC | 12:30 | |
| *** rlandy has joined #openstack-infra | 12:30 | |
| *** srobert has joined #openstack-infra | 12:30 | |
| *** pcaruana has quit IRC | 12:31 | |
| *** lukebrowning has quit IRC | 12:33 | |
| *** slaweq_ has quit IRC | 12:34 | |
| *** lukebrowning has joined #openstack-infra | 12:35 | |
| *** lukebrowning has quit IRC | 12:39 | |
| *** lukebrowning has joined #openstack-infra | 12:41 | |
| *** AJaeger_ has joined #openstack-infra | 12:42 | |
| *** edmondsw has joined #openstack-infra | 12:43 | |
| *** slaweq_ has joined #openstack-infra | 12:44 | |
| *** ykarel has quit IRC | 12:44 | |
| *** jaypipes has joined #openstack-infra | 12:45 | |
| *** trown has joined #openstack-infra | 12:45 | |
| *** lukebrowning has quit IRC | 12:46 | |
| *** jpena is now known as jpena|lunch | 12:46 | |
| *** LindaWang has quit IRC | 12:51 | |
| *** hemna_ has joined #openstack-infra | 12:51 | |
| *** lukebrowning has joined #openstack-infra | 12:52 | |
| *** erlon has joined #openstack-infra | 12:53 | |
| *** dave-mccowan has joined #openstack-infra | 12:54 | |
| *** Goneri has joined #openstack-infra | 12:55 | |
| *** eharney has joined #openstack-infra | 12:55 | |
| *** bobh has joined #openstack-infra | 12:56 | |
| *** bh526r has joined #openstack-infra | 12:56 | |
| *** lukebrowning has quit IRC | 12:57 | |
| *** AJaeger_ has quit IRC | 12:57 | |
| *** wolverineav has joined #openstack-infra | 12:58 | |
| *** gcb has quit IRC | 12:58 | |
| *** lukebrowning has joined #openstack-infra | 12:58 | |
| *** kgiusti has joined #openstack-infra | 13:00 | |
| *** pcaruana has joined #openstack-infra | 13:00 | |
| *** mat128 has joined #openstack-infra | 13:02 | |
| *** gcb has joined #openstack-infra | 13:02 | |
| *** lukebrowning has quit IRC | 13:03 | |
| *** LindaWang has joined #openstack-infra | 13:03 | |
| *** yamamoto has joined #openstack-infra | 13:03 | |
| *** iyamahat has joined #openstack-infra | 13:04 | |
| *** ykarel has joined #openstack-infra | 13:04 | |
| *** sree has joined #openstack-infra | 13:04 | |
| *** lukebrowning has joined #openstack-infra | 13:04 | |
| *** iyamahat has quit IRC | 13:05 | |
| *** iyamahat has joined #openstack-infra | 13:05 | |
| *** ianychoi_ is now known as ianychoi | 13:06 | |
| *** jcoufal has joined #openstack-infra | 13:07 | |
| *** sree has quit IRC | 13:09 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Allow core reviewers to push merge commits to mogan https://review.openstack.org/508151 | 13:09 |
| *** yamamoto has quit IRC | 13:09 | |
| *** lukebrowning has quit IRC | 13:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 13:09 |
| *** lukebrowning has joined #openstack-infra | 13:11 | |
| *** esberglu has joined #openstack-infra | 13:11 | |
| *** ijw has joined #openstack-infra | 13:12 | |
| *** trown has quit IRC | 13:13 | |
| *** chlong has quit IRC | 13:14 | |
| *** trown has joined #openstack-infra | 13:14 | |
| *** lukebrowning has quit IRC | 13:15 | |
| *** baoli has joined #openstack-infra | 13:16 | |
| *** lukebrowning has joined #openstack-infra | 13:17 | |
| *** slaweq_ has quit IRC | 13:17 | |
| *** ijw has quit IRC | 13:18 | |
| *** camunoz has joined #openstack-infra | 13:19 | |
| *** lukebrowning has quit IRC | 13:21 | |
| *** sree has joined #openstack-infra | 13:23 | |
| *** sshnaidm is now known as sshnaidm|afk | 13:23 | |
| *** lukebrowning has joined #openstack-infra | 13:23 | |
| *** slaweq_ has joined #openstack-infra | 13:24 | |
| mordred | pabelanger: yay moving! | 13:24 |
| *** mriedem has joined #openstack-infra | 13:26 | |
| *** lukebrowning has quit IRC | 13:28 | |
| *** alexchadin has joined #openstack-infra | 13:28 | |
| *** lukebrowning has joined #openstack-infra | 13:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Temporarily disable volume functional tests https://review.openstack.org/508156 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add devstack jobs for zuul v3 https://review.openstack.org/500365 | 13:31 |
| *** jrist has quit IRC | 13:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: [WIP] Avoid tox_install.sh for constraints support https://review.openstack.org/508061 | 13:32 |
| *** yamahata has joined #openstack-infra | 13:33 | |
| *** lukebrowning has quit IRC | 13:34 | |
| *** jaosorior has quit IRC | 13:34 | |
| *** jrist has joined #openstack-infra | 13:35 | |
| mordred | tonyb: ^^ re: that - love it - maybe we should consider trying to rename UPPER_CONSTRAINTS_FILE var to just CONSTRAINTS_FILE? (so that the general interface is there and we could use the same mechanism for lower constraints too?) | 13:35 |
| mordred | tonyb: fwiw we named the variable "tox_constraints_file" in the zuulv3 job | 13:35 |
| *** iyamahat has quit IRC | 13:35 | |
| *** gouthamr has joined #openstack-infra | 13:35 | |
| *** lukebrowning has joined #openstack-infra | 13:36 | |
| mordred | tonyb: if we can pick a variable name we want it to be, we can make the v3 job set both variables for a while as we roll things out | 13:36 |
| *** trown|brb has joined #openstack-infra | 13:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Allow core reviewers to push merge commits to mogan https://review.openstack.org/508151 | 13:37 |
| *** trown has quit IRC | 13:37 | |
| *** trown|brb is now known as trown | 13:38 | |
| *** lukebrowning has quit IRC | 13:40 | |
| *** lukebrowning has joined #openstack-infra | 13:42 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results https://review.openstack.org/507980 | 13:43 |
| *** rossella_s has joined #openstack-infra | 13:44 | |
| *** jpena|lunch is now known as jpena | 13:46 | |
| *** lukebrowning has quit IRC | 13:47 | |
| *** dizquierdo has joined #openstack-infra | 13:51 | |
| *** lukebrowning has joined #openstack-infra | 13:53 | |
| *** bnemec has joined #openstack-infra | 13:53 | |
| *** alex_xu has quit IRC | 13:54 | |
| *** gcb has quit IRC | 13:54 | |
| *** alex_xu has joined #openstack-infra | 13:54 | |
| *** LindaWang has quit IRC | 13:54 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:54 | |
| *** LindaWang has joined #openstack-infra | 13:55 | |
| *** eranrom has quit IRC | 13:55 | |
| *** slaweq_ has quit IRC | 13:56 | |
| *** eranrom has joined #openstack-infra | 13:56 | |
| *** kiennt26 has joined #openstack-infra | 13:56 | |
| *** lukebrowning has quit IRC | 13:58 | |
| *** LindaWang has quit IRC | 13:58 | |
| *** eranrom_ has joined #openstack-infra | 13:58 | |
| *** nikhil has joined #openstack-infra | 13:58 | |
| *** lukebrowning has joined #openstack-infra | 13:59 | |
| *** eranrom has quit IRC | 14:00 | |
| *** hemna_ has quit IRC | 14:00 | |
| coreycb | hi infra, any chance I can get a review of https://review.openstack.org/#/c/500964/ please? apologies as i didn't updated the topic to new-project until yesterday. | 14:01 |
| *** yamahata has quit IRC | 14:01 | |
| *** alex_xu has quit IRC | 14:02 | |
| *** links has quit IRC | 14:02 | |
| *** gcb has joined #openstack-infra | 14:02 | |
| mordred | coreycb: we've got new projects frozen for the zuul v3 rollout- we'll check it out as soon as the rollout is complete | 14:03 |
| mordred | coreycb: sorry for the inconvenice - the rollout should be finished today | 14:03 |
| *** ijw has joined #openstack-infra | 14:03 | |
| *** lukebrowning has quit IRC | 14:03 | |
| *** slaweq_ has joined #openstack-infra | 14:03 | |
| coreycb | mordred: oh gotcha, ignore me then. thanks for all the rollout work! | 14:03 |
| *** spzala has joined #openstack-infra | 14:03 | |
| *** lukebrowning has joined #openstack-infra | 14:05 | |
| *** LindaWang has joined #openstack-infra | 14:06 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 14:07 |
| *** AJaeger_ has joined #openstack-infra | 14:07 | |
| *** yolanda has quit IRC | 14:08 | |
| *** jaosorior has joined #openstack-infra | 14:08 | |
| *** ijw has quit IRC | 14:08 | |
| *** yolanda has joined #openstack-infra | 14:08 | |
| *** lukebrowning has quit IRC | 14:10 | |
| *** lukebrowning has joined #openstack-infra | 14:11 | |
| jeblair | mordred: good morning | 14:12 |
| *** LindaWang has quit IRC | 14:12 | |
| *** LindaWang has joined #openstack-infra | 14:12 | |
| fungi | morning is all in your mind | 14:13 |
| *** sree has quit IRC | 14:13 | |
| *** isaacb has joined #openstack-infra | 14:15 | |
| *** lukebrowning has quit IRC | 14:16 | |
| *** jrist has quit IRC | 14:17 | |
| *** lukebrowning has joined #openstack-infra | 14:18 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update chair for Monasca Team Meeting https://review.openstack.org/508110 | 14:19 |
| *** lukebrowning has quit IRC | 14:22 | |
| *** lukebrowning has joined #openstack-infra | 14:24 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check subunit file processing https://review.openstack.org/508171 | 14:25 |
| clarkb | I'm mostly awake too. going to grab some tea | 14:26 |
| *** rbrndt has joined #openstack-infra | 14:26 | |
| numans | AJaeger_, Hi, can you please have a look at this patch when you get some time - https://review.openstack.org/#/c/502899/ | 14:28 |
| *** lukebrowning has quit IRC | 14:28 | |
| mnaser | numans project-config is in a frozen state until the cutover of zuulv3 which should hopefully complete today | 14:31 |
| numans | mnaser, ack. thanks for the update. | 14:32 |
| *** srobert has quit IRC | 14:32 | |
| jeblair | fungi, clarkb, mordred: should we send out a status notice and get started? | 14:34 |
| *** rtjure has joined #openstack-infra | 14:35 | |
| clarkb | ++ | 14:35 |
| *** hongbin has joined #openstack-infra | 14:35 | |
| fungi | yep, seems like it's that time | 14:36 |
| *** ijw has joined #openstack-infra | 14:36 | |
| *** lukebrowning has joined #openstack-infra | 14:36 | |
| *** slaweq_ has quit IRC | 14:37 | |
| jeblair | status notice The infra team is now taking Zuul v2 offline and brining Zuul v3 online. Please see https://docs.openstack.org/infra/manual/zuulv3.html for more information. | 14:37 |
| jeblair | something like that? | 14:37 |
| clarkb | lgtm. Might also want to let people know they can ask us questions if they see anything weird? | 14:38 |
| SpamapS | fungi: hey btw, how's the weather over there? I see Maria is sending some pretty crazy wind your way. | 14:38 |
| mnaser | bringing? :p | 14:38 |
| SpamapS | no it's brining | 14:38 |
| SpamapS | so much brine. | 14:38 |
| fungi | SpamapS: it already died down. tourists are being allowed back onto the islands south of us today | 14:38 |
| SpamapS | we like our Zuulv3 salty | 14:38 |
| fungi | well, more like headed out to sea | 14:38 |
| SpamapS | fungi: glad to hear it's all ok. | 14:39 |
| fungi | yesterday was the peak for us | 14:39 |
| *** dave-mccowan has quit IRC | 14:39 | |
| fungi | and yeah, only two more months of hurricane season to go (in theory) | 14:39 |
| *** chlong has joined #openstack-infra | 14:39 | |
| jeblair | status notice The infra team is now taking Zuul v2 offline and brining Zuul v3 online. Please see https://docs.openstack.org/infra/manual/zuulv3.html for more information, and ask us in #openstack-infra if you have any questions. | 14:40 |
| fungi | jeblair: notice lgtm | 14:40 |
| jeblair | clarkb, fungi: ^ revised? | 14:40 |
| clarkb | jeblair: ++ though may want to change the brining to bringing | 14:40 |
| *** spotz_ is now known as spotz | 14:40 | |
| fungi | yep | 14:40 |
| clarkb | I'm good with brining though. | 14:40 |
| fungi | pickled ci | 14:40 |
| clarkb | Makes turkey tastey. Could be good for zuul | 14:40 |
| jeblair | wow am i hungry or what | 14:40 |
| * fungi is just happy his house didn't get brined this week | 14:40 | |
| *** lukebrowning has quit IRC | 14:40 | |
| jeblair | #status notice The infra team is now taking Zuul v2 offline and bringing Zuul v3 online. Please see https://docs.openstack.org/infra/manual/zuulv3.html for more information, and ask us in #openstack-infra if you have any questions. | 14:41 |
| openstackstatus | jeblair: sending notice | 14:41 |
| SpamapS | how exciting | 14:41 |
| * SpamapS watches Zuul v2 blink confusedly from the boat as the archers knock their bows and light the tips | 14:42 | |
| *** ijw has quit IRC | 14:42 | |
| -openstackstatus- NOTICE: The infra team is now taking Zuul v2 offline and bringing Zuul v3 online. Please see https://docs.openstack.org/infra/manual/zuulv3.html for more information, and ask us in #openstack-infra if you have any questions. | 14:42 | |
| Shrews | this time really for sure and true | 14:42 |
| *** lukebrowning has joined #openstack-infra | 14:42 | |
| *** SpamapS is now known as Zuulv2 | 14:43 | |
| jeblair | the next step is "Stop zuul v2" so *something* is going to happen fer sure | 14:43 |
| Zuulv2 | I'm not dead | 14:43 |
| fungi | really for reals | 14:43 |
| Zuulv2 | I don't want to go in the cart | 14:43 |
| *** Zuulv2 is now known as SpamapS | 14:43 | |
| *** xarses has joined #openstack-infra | 14:44 | |
| openstackstatus | jeblair: finished sending notice | 14:44 |
| mordred | jeblair, Shrews, clarkb, fungi: morning! Sorry, I was looking away from IRC writing some emails | 14:44 |
| * mordred is excited | 14:44 | |
| jeblair | mordred: i knew you'd be back :) | 14:44 |
| *** dave-mccowan has joined #openstack-infra | 14:44 | |
| smcginnis | lol | 14:45 |
| jeblair | okay, i'll stop v2 now yeah? | 14:45 |
| clarkb | that is what theetherpad says is next, I say go for it | 14:45 |
| *** slaweq_ has joined #openstack-infra | 14:45 | |
| jeblair | okay, i saved check and gate, and stopped zuul | 14:45 |
| jeblair | 507222 looks to be next | 14:45 |
| jeblair | oh, i'll also stop the v2 nodepool launcher, but nothing else... how's that sound? | 14:46 |
| jeblair | (and i'll continue to try to wind down v2 quota usage) | 14:46 |
| clarkb | the change to shift quota should wind it down with the daemon up | 14:47 |
| jeblair | well, i guess i don't need to stop the launcher yeah | 14:47 |
| jeblair | i'll just need to manually delete ready nodes after the change lands | 14:47 |
| clarkb | ya and we can tell old launcher to delete everything | 14:47 |
| *** lukebrowning has quit IRC | 14:47 | |
| *** AJaeger_ has quit IRC | 14:47 | |
| mordred | jeblair, clarkb: ++ | 14:47 |
| jeblair | 507222 is approved | 14:47 |
| *** alexchadin has quit IRC | 14:48 | |
| mordred | once that lands, should we kick.sh to get it on to the launchers more quicker? | 14:48 |
| clarkb | and on nodepool.o.o | 14:48 |
| clarkb | so all three nodes | 14:48 |
| mordred | it would be kick.sh for nl01.o.o, nl02.o.o and nodepool.o.o yeah? | 14:48 |
| jeblair | i'll stop all the zuulv2 launchers now. that should return all the used nodes | 14:48 |
| jeblair | mordred: ++ | 14:48 |
| clarkb | yup | 14:48 |
| *** makowals has quit IRC | 14:49 | |
| mordred | anybody know if this: /opt/system-config/production/tools/kick.sh 'nodepool.openstack.org;nl0*.openstack.org' will work? | 14:49 |
| clarkb | I don't | 14:49 |
| mordred | I think it should - although I think it's : not ; | 14:49 |
| fungi | mordred: my expectation was that it should since it's just passing $1 as the host constraint for ansible | 14:49 |
| fungi | well, yeah, syntax aside | 14:49 |
| mordred | ok. I'm set up to run that | 14:50 |
| *** AJaeger_ has joined #openstack-infra | 14:50 | |
| *** rossella_s has quit IRC | 14:51 | |
| *** mugsie has joined #openstack-infra | 14:52 | |
| mnaser | http://grafana.openstack.org/dashboard/db/nodepool fun numbers to look at :p | 14:52 |
| *** andreas_s_ has quit IRC | 14:53 | |
| *** Swami has joined #openstack-infra | 14:53 | |
| *** lukebrowning has joined #openstack-infra | 14:53 | |
| jeblair | i think i've got v3 versions of the scripts to restore check/gate; i won't run them right away, but when we're confident we're ready for more load, i can run them to restore the previous check/gate from before v2 stopped | 14:54 |
| jeblair | the nodepool deleter looks to be settling down; only about 50 more to delete | 14:55 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check subunit file processing https://review.openstack.org/508171 | 14:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Shift nodepool quota from v2 to v3 https://review.openstack.org/507222 | 14:57 |
| clarkb | mordred: ^ | 14:57 |
| *** AJaeger_ has quit IRC | 14:57 | |
| *** ihrachys has joined #openstack-infra | 14:58 | |
| mordred | woot. kicking | 14:58 |
| *** lukebrowning has quit IRC | 14:58 | |
| mordred | yes, btw - passing that to kick.sh worked | 14:58 |
| *** hemna_ has joined #openstack-infra | 14:58 | |
| mordred | /opt/system-config/production/tools/kick.sh 'nodepool.openstack.org:nl0*.openstack.org' | 14:58 |
| *** srobert has joined #openstack-infra | 14:58 | |
| mordred | for the record | 14:58 |
| jeblair | nodepool has noticed the config change | 14:59 |
| mordred | woot | 14:59 |
| *** jrist has joined #openstack-infra | 14:59 | |
| jeblair | (old nodepool) | 14:59 |
| *** lukebrowning has joined #openstack-infra | 15:00 | |
| jeblair | i'll work on deleting everything not already deleted | 15:00 |
| jeblair | project-config post job to publish infro docs index is still broken (post_failure) if anyone has time to look at that | 15:02 |
| jeblair | all nodepool v2 nodes are marked delete now | 15:02 |
| mordred | kick is done | 15:03 |
| *** camunoz has quit IRC | 15:03 | |
| mordred | also - the zuul-jobs docs are not up to date, so we may want to look at the zuul-jobs post job too | 15:04 |
| clarkb | nl01 appears to have been updated as expected | 15:04 |
| *** lukebrowning has quit IRC | 15:04 | |
| clarkb | 507179 now? | 15:04 |
| mordred | clarkb: yah. let's do it | 15:05 |
| clarkb | mordred: do you want to +W it? it has the +2's already | 15:05 |
| *** hemna_ has quit IRC | 15:05 | |
| clarkb | jeblair may want to watch zuulv3 logs while that happens? | 15:05 |
| *** lukebrowning has joined #openstack-infra | 15:06 | |
| mordred | actually - jeblair, you wanna do the +W honors? | 15:06 |
| jeblair | clarkb: thanks, watching | 15:06 |
| mordred | (or I can) | 15:06 |
| *** dave-mcc_ has joined #openstack-infra | 15:07 | |
| jeblair | mordred: done! | 15:07 |
| *** sdague has quit IRC | 15:07 | |
| mordred | k. I will prepare to kick zuulv3 | 15:07 |
| mnaser | merge failed :< | 15:08 |
| *** tmorin has joined #openstack-infra | 15:08 | |
| mordred | boo | 15:08 |
| * mordred updates patchset | 15:08 | |
| jeblair | ha :( | 15:08 |
| clarkb | parent says not current | 15:08 |
| * fungi blames git | 15:08 | |
| *** dave-mccowan has quit IRC | 15:09 | |
| clarkb | but it looks like the last commit on a change that did merge | 15:09 |
| SpamapS | stupid git | 15:09 |
| mordred | it was the add of noop-jobs | 15:10 |
| mordred | git couldn't deal with that | 15:10 |
| *** lukebrowning has quit IRC | 15:10 | |
| *** isaacb has quit IRC | 15:10 | |
| mordred | uhm. I'm having issues pushing the patch to gerrit ... | 15:11 |
| *** xyang1 has joined #openstack-infra | 15:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 15:11 |
| mordred | ah.there is goes | 15:11 |
| EmilienM | stupid question but here we go: are we still in project-config freeze? | 15:11 |
| clarkb | EmilienM: yes | 15:11 |
| jlk | is it time to throw the switch? | 15:11 |
| mordred | we just did | 15:11 |
| mordred | or, we are throwing it as we speak | 15:12 |
| mordred | (it takes a little effort to move the switch from one side to the other) | 15:12 |
| jlk | so we can +w this switch over? | 15:12 |
| clarkb | wow that one line made git unhappy | 15:12 |
| *** jaosorior has quit IRC | 15:12 | |
| *** xarses has quit IRC | 15:12 | |
| *** lukebrowning has joined #openstack-infra | 15:12 | |
| jeblair | fungi says it's a really big frakenstein knife switch | 15:12 |
| jlk | (oh well I can't, because I don't have core on this project) | 15:12 |
| fungi | jeblair: cue maniacal laughter? | 15:13 |
| jlk | cue bottles being opened | 15:13 |
| mordred | ok. it has been +A'd - we shall now await 2 runs of the legacy-bindep-fallback jobs | 15:13 |
| rbergeron | i'm on a plane, i'm already drinking | 15:14 |
| jeblair | nodepool v2 is down to 13 nodes to delete. most have been in deleting state for a day or 2 already, two of them for ~130 days. i don't think they're ever going to be deleted. | 15:14 |
| rbergeron | oh wait, it's 8am at my house, wow | 15:14 |
| fungi | let's limit the kinds of bottles until after we see this chuggnig along in production with no fixes needed | 15:14 |
| jeblair | yay openstack | 15:14 |
| jeblair | fungi: are airplane bottles okay? | 15:14 |
| fungi | probably | 15:14 |
| *** spzala has quit IRC | 15:14 | |
| fungi | rbergeron: i heard there was some widespread systems outage impacting flight scheduling. i guess you beat the clock on that one? | 15:14 |
| rbergeron | apparently i did! | 15:15 |
| mordred | jeblair, clarkb, fungi: should we maybe come up with a way to tell our cloud providers about nodes that stick in deleting state? | 15:15 |
| jlk | well | 15:16 |
| mordred | like- what about a cron job for each provider that looks for nodes stuck in deleting state for more than 1 day and emails a contact person... | 15:16 |
| jlk | she's in the air. whether or not she can land is another matter | 15:16 |
| *** lamt has joined #openstack-infra | 15:16 | |
| fungi | mordred: worth revisiting later maybe | 15:16 |
| mordred | fungi: ++ | 15:16 |
| fungi | mordred: i can think of all sorts of gotchas there which may make that not really work out in practice | 15:17 |
| *** lukebrowning has quit IRC | 15:17 | |
| fungi | (mostly around the human aspect, not the automation part) | 15:17 |
| jlk | HUMAN RECEIVE MY FLOOD OF ALERTS | 15:17 |
| mordred | jlk: if you don't want the flood of alerts, maybe fix your cloud to let nodes delete kthxbai ;) | 15:18 |
| jlk | pain driven development | 15:18 |
| rbergeron | I have decided I need my human API to be able to be queried and have interesting responses | 15:18 |
| rbergeron | 'number of cares left: 0' | 15:18 |
| *** lukebrowning has joined #openstack-infra | 15:18 | |
| *** slaweq_ has quit IRC | 15:18 | |
| rbergeron | "size: M" "food: yes: | 15:18 |
| rbergeron | anyway. :) | 15:19 |
| *** camunoz has joined #openstack-infra | 15:19 | |
| jeblair | yay, round 1 complete! 507179 goes to round 2 now! | 15:19 |
| fungi | rbergeron: the number of cares field can range from negative values up to zero, right? | 15:19 |
| dansmith | hmm, will the location of the status.json change? | 15:19 |
| dansmith | looks like everything that was running at the time you made the announcement is unchanged from that state | 15:20 |
| fungi | dansmith: there will be a 302 temporary redirect for a while | 15:20 |
| rbergeron | fungi: yessss | 15:20 |
| clarkb | I like the little green noop | 15:20 |
| jlk | I queried for fucks and got a 416 error | 15:20 |
| dansmith | fungi: okay but is that currently not in place such that I'm looking at stale data? | 15:20 |
| fungi | dansmith: correct | 15:20 |
| dansmith | gotcha | 15:21 |
| fungi | dansmith: you can point at http://zuulv3.openstack.org/status.json for now if you like | 15:21 |
| TheJulia | rbergeron: +1 :) | 15:21 |
| *** psachin has quit IRC | 15:21 | |
| dansmith | fungi: for now ... meaning it'll change again? | 15:21 |
| jlk | it'll eventually take the zuul.openstack.org URL back | 15:22 |
| fungi | dansmith: http://zuul.openstack.org/(.*) will be a temporary 302 redirect to http://zuulv3.openstack.org/$1 (or something like that) until we rename zuulv3 to zuul | 15:22 |
| dansmith | okay, so safest thing is to just be patient until the 302 is in place and not switch and then switch back | 15:22 |
| fungi | which is likely on the order of days | 15:22 |
| mordred | dansmith: yah | 15:23 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results https://review.openstack.org/507980 | 15:23 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check subunit file processing https://review.openstack.org/508171 | 15:23 |
| jlk | construction-sign.gif | 15:23 |
| dansmith | mordred: I should really just lead with that, amirite? "before I ask $question, is the answer just "be patient" ?" | 15:23 |
| fungi | dansmith: the rename is happening in coming days i mean. i expect we'll have the redirect up shortly | 15:23 |
| *** lukebrowning has quit IRC | 15:23 | |
| mordred | dansmith: I dunno - in some cases the answer *might* be "dude, freak out more" | 15:23 |
| jlk | hrm, zuulv3 status page shows two sections for Queue: openstack-infra/devstack-gate | 15:23 |
| jlk | is that right? | 15:23 |
| *** ykarel has quit IRC | 15:24 | |
| dansmith | mordred: I believe my early freakouts have been the first warning sign for a couple real issues. like birds before a tornado and such | 15:24 |
| clarkb | jlk: yes, two different changes and in check each gets its own queue | 15:24 |
| jlk | maybe because i'ts check and there are no dependent things to lump in the same queue? | 15:24 |
| jeblair | exactly | 15:24 |
| jlk | yeah, it's slightly confusing | 15:24 |
| *** lukebrowning has joined #openstack-infra | 15:25 | |
| fungi | jlk: in the gate pipeline (because of it being a dependent style pipeline) there can be only one | 15:25 |
| jlk | yeah that makes total sense to show the queue there | 15:25 |
| jlk | but showing it where it has no impact seems... wrong? | 15:25 |
| * fungi promises to avoid further highlander references | 15:25 | |
| *** shardy is now known as shardy_afk | 15:25 | |
| fungi | jlk: in the past we did that more for consistency's sake, i believe | 15:26 |
| fungi | though i agree it robs a fair amount of screen real estate | 15:26 |
| jeblair | jlk: you mean the text "Queue: ..."? | 15:26 |
| jlk | yeah. It's just odd that it gets displayed fairly prominently, but repeatedly. | 15:27 |
| jlk | and knowing the internals, doesn't really have an effect upon the jobs in that pipeline | 15:27 |
| jeblair | yeah, i could see dropping that text | 15:28 |
| openstackgerrit | Merged openstack-infra/project-config master: Migrate zuul v2 project pipelines to v3 https://review.openstack.org/507179 | 15:28 |
| jeblair | zuul is reconfiguring | 15:28 |
| jeblair | and spewing quite a lot of garbage to the log | 15:28 |
| jlk | its-happening.gif | 15:28 |
| jeblair | we need to drop the applying template log line | 15:29 |
| dansmith | jlk: do you just speak in fictional gif filenames now? | 15:29 |
| jeblair | and now it's emitting its config, which is illegible | 15:29 |
| jeblair | and now it's done | 15:29 |
| *** lukebrowning has quit IRC | 15:29 | |
| clarkb | now we have dansmith push a change and see if it works? | 15:29 |
| Shrews | shade.exc.OpenStackCloudHTTPError: (409) Client Error for url: https://ci-overcloud.rh1.tripleo.org:13774/v2.1/b79291658f384b7ebbc9019b6349e5c9/servers Multiple possible networks found, use a Network ID to be more specific. | 15:29 |
| openstackgerrit | Chris Hoge proposed openstack-infra/irc-meetings master: Earlier meeting time for international attendees https://review.openstack.org/508202 | 15:30 |
| rbergeron | so i can open the fridge now? | 15:30 |
| Shrews | from nl01 launcher log ^^^ | 15:30 |
| *** slaweq_ has joined #openstack-infra | 15:30 | |
| jlk | dansmith: I'm.... cheerleading? <thoughtface_emoji> | 15:30 |
| dansmith | jlk: man-laughing.gif | 15:30 |
| *** lukebrowning has joined #openstack-infra | 15:31 | |
| Shrews | pabelanger: issue with tripleo-test-cloud-rh1 configuration maybe? | 15:31 |
| mordred | Shrews: oh goodie - which cloud? | 15:31 |
| Shrews | mordred: ^^^ | 15:32 |
| clarkb | mordred: the rh1 cloud based on the url | 15:32 |
| mordred | ah. yah - I think for tripleo-test-cloud-rh1 we need network info in theclouds.yaml | 15:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove "Applying template" log line https://review.openstack.org/508203 | 15:32 |
| rbergeron | you humans are awesome. seriously. :) | 15:33 |
| clarkb | the zuul status page seems to indicate the events queue isn't shrinking | 15:33 |
| fungi | clarkb: if you're talking about the same dansmith i'm thinking of, it'll be a stack of 50 dependent changes | 15:33 |
| *** jkilpatr has quit IRC | 15:34 | |
| clarkb | possibly because the 4 results items in the results queue are not going away? | 15:34 |
| jeblair | i think it's still re-enqueing changes after the reconfiguration | 15:34 |
| jeblair | possibly because the reconfiguration for each one is now taking a long time due to all the templates. | 15:35 |
| clarkb | there it goes just cleared out | 15:35 |
| clarkb | there is a cinder change in the gate | 15:35 |
| clarkb | that is probably an interesting one to watch | 15:35 |
| *** e0ne has quit IRC | 15:36 | |
| *** lukebrowning has quit IRC | 15:36 | |
| *** jrist has quit IRC | 15:36 | |
| clarkb | as is the devstack-gate change at top of check | 15:36 |
| dansmith | I can re-push a small series if you want a stack to watch | 15:36 |
| jeblair | clarkb: well that one is andreaf working on the v3 tempest job | 15:36 |
| mordred | oh good! it'll be fun to see how these cinder jobs go | 15:36 |
| mordred | fingers crossed for legacy-grenade-dsvm-cinder-mn-sub-volbak | 15:37 |
| *** lukebrowning has joined #openstack-infra | 15:37 | |
| mordred | we should probably get around to writing a releasenotes job ... /me feels bad he didn't get to that pre-transition | 15:37 |
| jlk | devstack-gate failed | 15:37 |
| jeblair | clarkb: oh, right, it did just get a lot of legacy jobs added to it :) | 15:37 |
| jlk | legacy-devstack-gate-tox-py3-run-tests | 15:37 |
| fungi | 507980,6 just failed on py3 | 15:37 |
| fungi | yeah, that | 15:38 |
| *** hemna_ has joined #openstack-infra | 15:38 | |
| fungi | oh, failed on py2 as well | 15:38 |
| jeblair | http://logs.openstack.org/80/507980/6/check/legacy-devstack-gate-tox-run-tests/3b2c1fb/job-output.txt.gz#_2017-09-28_15_37_10_693676 | 15:38 |
| jlk | wamp wamp :( | 15:38 |
| fungi | Error executing: cp -dRl /home/zuul/src/git.openstack.org/openstack/requirements/. /tmp/tmp.6v57I4qZmr | 15:39 |
| fungi | :( | 15:39 |
| jlk | http://logs.openstack.org/80/507980/6/check/legacy-devstack-gate-tox-run-tests/3b2c1fb/job-output.txt.gz#_2017-09-28_15_37_10_693070 | 15:39 |
| clarkb | missing required repo? | 15:39 |
| jlk | No such file or directory | 15:39 |
| jlk | yeah | 15:39 |
| fungi | agreed | 15:39 |
| jeblair | http://logs.openstack.org/80/507980/6/check/legacy-devstack-gate-tox-run-tests/3b2c1fb/zuul-info/inventory.yaml | 15:40 |
| jeblair | no requirements in zuul.projects | 15:40 |
| jlk | is that related to not having a requirement... | 15:40 |
| clarkb | same bug on the failed cinder tox job | 15:40 |
| jlk | they all use the legacy parent, do we just add the requirement there? | 15:41 |
| jeblair | jlk: they both use 'legacy-base' ? | 15:42 |
| *** lukebrowning has quit IRC | 15:42 | |
| mordred | oh - fun - yah - the tox jobs that didn't get parented to / translated to openstack-tox | 15:42 |
| jeblair | if so, why not legacy-dsvm-base i wonder | 15:42 |
| mordred | I think legacy-devstack-gate-tox-run-tests is a tox thing that validates tings in devstack-gate - not an actual dsvm job | 15:42 |
| jeblair | ooooh | 15:42 |
| mordred | this is likely to be an issue with every non-standard tox job | 15:42 |
| mordred | so - yah - for now I'd recommend we add openstack/requirements to legacy-base | 15:43 |
| jeblair | oof. that's big. | 15:43 |
| *** eroux has quit IRC | 15:43 | |
| fungi | and what is the unwinding if we go that route? | 15:43 |
| fungi | seems like a large hammer | 15:43 |
| mordred | and then do a follow up to try to see if we can just add it to legacy jobs that are doing the constraints/reuqirements thing | 15:43 |
| *** lukebrowning has joined #openstack-infra | 15:43 | |
| jeblair | mordred: we'd have to look at content to try to figure that out, yeah? | 15:44 |
| *** spzala has joined #openstack-infra | 15:44 | |
| mordred | jeblair: yah - lemme see how hard that is to figure out- we could heuristic anything with tox in the name ... | 15:44 |
| jeblair | what if we took a guess and addad requirements to every job with 'legacy' and 'tox' in the name? | 15:44 |
| clarkb | http://logs.openstack.org/80/507980/6/check/legacy-tempest-dsvm-nnet/f358e68/ara/result/0a585a73-5403-4412-92be-fdb38072ff44/ | 15:44 |
| jeblair | mordred: :) | 15:44 |
| mordred | jeblair: ++ | 15:44 |
| clarkb | I think ^ may be a missing nodepool thing post switch? | 15:44 |
| mordred | jeblair: I will write that patch real quick | 15:44 |
| jeblair | mordred: ack | 15:45 |
| mordred | jeblair, clarkb: I'm not sure we are emitting uuid in our /etc/nodepool role | 15:45 |
| *** edmondsw has quit IRC | 15:46 | |
| *** rcernin has quit IRC | 15:46 | |
| jeblair | huh, why didn't our test of a legacy devstack job catch that? | 15:46 |
| clarkb | the post failure on the cinder job is also the missing requirements thing | 15:46 |
| clarkb | jeblair: I think it may be more unique to that specific job | 15:46 |
| clarkb | jeblair: it doensn't seem to have similarly caused the other legacy jobs to crash yet at least | 15:47 |
| jeblair | oh weird | 15:47 |
| clarkb | ya macro print-node-uuid is only used in a few places | 15:47 |
| jeblair | it's just something ^ yeah that | 15:47 |
| clarkb | so we might just delete that from the legacy jobs that use it | 15:47 |
| jeblair | let's | 15:48 |
| clarkb | rather than fix the interface | 15:48 |
| jeblair | i will do that | 15:48 |
| *** spzala has quit IRC | 15:48 | |
| jlk | 👍🏼 | 15:48 |
| *** lukebrowning has quit IRC | 15:48 | |
| *** ramishra has quit IRC | 15:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Remove print-node-uuid macro from converted jobs https://review.openstack.org/508205 | 15:51 |
| jeblair | clarkb, fungi, mordred: ^ | 15:51 |
| *** hashar is now known as hasharAway | 15:51 | |
| openstackgerrit | Chris Hoge proposed openstack-infra/irc-meetings master: Earlier meeting time for international attendees https://review.openstack.org/508202 | 15:52 |
| *** iyamahat has joined #openstack-infra | 15:52 | |
| *** tmorin has quit IRC | 15:54 | |
| *** lukebrowning has joined #openstack-infra | 15:54 | |
| jeblair | i see a bunch of project-templates that are used on 1, or sometimes even 0 projects. i think we should simplify the config by collapsing those into the project definition. | 15:57 |
| mordred | jeblair: I agree | 15:57 |
| jeblair | oh hey, zuul v3 is now doing a full reconfiguration | 15:57 |
| clarkb | ozj fix for nnet is headed to the gate now | 15:57 |
| clarkb | that also affected nova | 15:57 |
| *** camunoz has quit IRC | 15:57 | |
| jeblair | i'm guessing ansipuppet triggered that due to some project-config change we landed earlier? | 15:57 |
| jeblair | do we have puppet reconfigurations of zuul appropriately scoped now? (we should only do it on main.yaml or zuul.conf changes) | 15:58 |
| jeblair | if not, we should, because they are a bit expensive | 15:58 |
| hogepodge | Hmm, I have no idea why this job is failing https://review.openstack.org/#/c/508202/2 | 15:58 |
| mordred | jeblair: no - I believe we do it on the whole zuul dir atm | 15:58 |
| *** lukebrowning has quit IRC | 15:59 | |
| *** egonzalez has quit IRC | 15:59 | |
| jeblair | mordred: oh, that's probably okay... | 15:59 |
| jeblair | mordred: once we ignore the v2 files, that only leaves site-variables.yaml | 15:59 |
| jeblair | so this should just be something that passes once the transition is done | 15:59 |
| clarkb | hogepodge: http://logs.openstack.org/02/508202/2/check/legacy-irc-meetings-tox-ical/b78bf2e/ara/result/2b54b455-a7c1-4dfb-b585-6354dd65d699/ looks like sudo didn't actually get removed? | 16:00 |
| mordred | jeblair: yah | 16:00 |
| clarkb | jeblair: mordred I think ^ is a bug because now jobs run as zuul and not jenkins | 16:00 |
| *** lukebrowning has joined #openstack-infra | 16:00 | |
| * hogepodge is always happy to find broken things ;-) | 16:01 | |
| clarkb | we ported the normal python jobs directly so they avoid that problem | 16:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add openstack/requirements to all legacy-.*tox jobs https://review.openstack.org/508210 | 16:01 |
| mordred | clarkb, jeblair: ^^ | 16:01 |
| jeblair | i'll work on a change to s/jenkins-sudo/zuul-sudo/ ? | 16:01 |
| clarkb | jeblair: ++ wasn't sure if zuul-sudo was a thing | 16:01 |
| *** sbezverk has quit IRC | 16:01 | |
| clarkb | but if it is that sounds like the fix | 16:01 |
| *** armax has joined #openstack-infra | 16:01 | |
| * clarkb reviews mordred's change | 16:01 | |
| jeblair | i'm not sure either, we should check | 16:02 |
| hogepodge | thanks clarkb | 16:02 |
| mordred | I also have a basic script base for doing edits on zuul.d/legacy-zuul-jobs | 16:02 |
| *** slaweq_ has quit IRC | 16:02 | |
| mordred | if we have many more of them, I'll figure out a somewhat sane way to share it | 16:02 |
| jeblair | mordred: can you check whether zuul-sudo is the right file to remove while i write the change to do that? | 16:03 |
| *** tosky is now known as ri | 16:04 | |
| *** ri is now known as tosky | 16:04 | |
| mordred | jeblair: yes | 16:05 |
| *** lukebrowning has quit IRC | 16:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Replace jenkins-sudo with zuul-sudo https://review.openstack.org/508212 | 16:05 |
| mordred | jeblair: the revoke-sudo role does, in fact, remove /etc/sudoers.d/zuul-sudo | 16:05 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove "Applying template" log line https://review.openstack.org/508203 | 16:06 |
| *** dizquierdo has quit IRC | 16:06 | |
| *** jascott1 has joined #openstack-infra | 16:06 | |
| *** slaweq_ has joined #openstack-infra | 16:06 | |
| *** lukebrowning has joined #openstack-infra | 16:07 | |
| *** dizquierdo has joined #openstack-infra | 16:07 | |
| mordred | ok - we had a post-failure on legacy-releasenotes | 16:08 |
| mordred | http://logs.openstack.org/72/505672/3/gate/legacy-releasenotes/684c4a1/job-output.txt.gz | 16:08 |
| mordred | which is just the requirements thing - patch coming | 16:08 |
| clarkb | mordred: ya thats the same requirements thing | 16:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add requirements to legacy-releasenotes https://review.openstack.org/508213 | 16:09 |
| *** jascott1_ has joined #openstack-infra | 16:10 | |
| *** jascott1 has quit IRC | 16:10 | |
| * clarkb single core approved ^ since its like the other fix for that | 16:10 | |
| *** jrist has joined #openstack-infra | 16:10 | |
| *** camunoz has joined #openstack-infra | 16:11 | |
| *** lukebrowning has quit IRC | 16:11 | |
| clarkb | mordred: http://logs.openstack.org/96/345396/22/check/legacy-tempest-dsvm-multinode-live-migration/132adc4/ failed because live migration couldn't find a the second compute node, did that get properly added as a multinode job? | 16:11 |
| clarkb | looking in ara I don't see any subnodes | 16:12 |
| jeblair | clarkb: only one node in inventory: http://logs.openstack.org/96/345396/22/check/legacy-tempest-dsvm-multinode-live-migration/132adc4/zuul-info/inventory.yaml | 16:12 |
| *** lukebrowning has joined #openstack-infra | 16:13 | |
| *** jascott1_ has quit IRC | 16:13 | |
| jeblair | skimming several multinode jobs, i don't see anything about them that would lead me to believe they actually used a multinode nodeset in v3 | 16:14 |
| clarkb | looks like it has the default nodeset | 16:14 |
| clarkb | ya | 16:14 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results https://review.openstack.org/507980 | 16:14 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check subunit file processing https://review.openstack.org/508171 | 16:14 |
| clarkb | might also want to check that the centos jobs are set to centos nodesets | 16:14 |
| clarkb | looks like centos 7 nodeset is used as expected | 16:15 |
| jeblair | ya | 16:15 |
| clarkb | including for multinode | 16:15 |
| clarkb | I think it might be the implicitly multinode jobs like the live migration one that lost out | 16:15 |
| clarkb | ah no, dvr-ha-multinode-full doesn't have it either | 16:15 |
| clarkb | do we want to try and manually update that? | 16:16 |
| *** armax has quit IRC | 16:17 | |
| SpamapS | Status page looks pretty awesome. | 16:17 |
| SpamapS | Has gate landed anything yet? | 16:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Make new releasenotes job https://review.openstack.org/508219 | 16:17 |
| *** jascott1 has joined #openstack-infra | 16:17 | |
| *** jascott1 has quit IRC | 16:17 | |
| *** Apoorva has joined #openstack-infra | 16:17 | |
| *** Apoorva has quit IRC | 16:17 | |
| clarkb | SpamapS: a searchlight change merged as did a zuul change | 16:17 |
| jeblair | SpamapS: only on zuul afaik | 16:17 |
| *** lukebrowning has quit IRC | 16:18 | |
| *** Apoorva has joined #openstack-infra | 16:18 | |
| SpamapS | BTW, have we already started work on a change to the zuul stream plugin to show the command run with shell/command? | 16:18 |
| clarkb | mordred: maybe with your magical script you can try and update the nodesets for multinode jobs? I can take a stab at it too but expect to be much slower | 16:18 |
| mordred | clarkb: yes - I can - anything other than multinode in name? | 16:19 |
| *** jascott1 has joined #openstack-infra | 16:19 | |
| clarkb | mordred: so far no | 16:19 |
| * SpamapS has a multinode job running in his private zuulv3 btw. :) | 16:19 | |
| mordred | we were SUPPOSED to have mapped the old requested nodeset to new nodeset :( | 16:19 |
| jeblair | SpamapS: we're going to discuss it after mordred refactors zuul_stream and adds tests | 16:19 |
| clarkb | mordred: the centos-7-2-node jobs etc were all mapped properly from what I can see | 16:19 |
| *** lukebrowning has joined #openstack-infra | 16:19 | |
| clarkb | mordred: also I think the dvr ha multinode job is 3 node not 2 node, but defaulting to 2 node is likely mostly safe | 16:19 |
| mordred | clarkb: ok - but legacy-tempest-dsvm-mutinode-live-migration wans't right? | 16:20 |
| SpamapS | jeblair: awesome. I will happily code on that, as I am wanting it right now since I have a lot of templates in my shell/command tasks ;) | 16:20 |
| clarkb | mordred: correct that job has only the default nodeset | 16:20 |
| clarkb | which is single node xenial aiui | 16:20 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove print-node-uuid macro from converted jobs https://review.openstack.org/508205 | 16:20 |
| mordred | ah- I see where the problem came from I think | 16:21 |
| *** iyamahat has quit IRC | 16:21 | |
| mnaser | i'm noticing a failure in puppet jobs which i think is much more widespread, the `/etc/sudoers.d/jenkins-sudo` file seems to have been replaced by `/etc/sudoers.d/zuul-sudo` | 16:21 |
| jeblair | mnaser: fix in progress | 16:22 |
| jlk | that was corrected above | 16:22 |
| mnaser | oh okay | 16:22 |
| jlk | good catch though! | 16:22 |
| mordred | node-release: ubuntu-xenial in the original job put ubuntu-xenial in the job name | 16:22 |
| fungi | mnaser: 508212 should address it shortly | 16:23 |
| mordred | clarkb: so - I think what we want is that if multinode is in the job name and it doesn't have a nodeset defined - we want to add one | 16:23 |
| Shrews | "sudo: unknown user: stack" from a job whose base is legacy-dsvm-base. Should the pre playbook for that base job be creating that user? | 16:23 |
| mnaser | cool sorry i should have looked at buffer :> | 16:23 |
| clarkb | Shrews: devstack-gate creates the stack user | 16:23 |
| clarkb | Shrews: so in legacy jobs running d-g that should just work (tm) | 16:23 |
| clarkb | mordred: ya I think that is probabl ya good next step | 16:23 |
| fungi | mnaser: please err on the side of redundancy. we'd rather not miss any bugs people are spotting | 16:24 |
| *** lukebrowning has quit IRC | 16:24 | |
| mnaser | fungi will do :) | 16:24 |
| SpamapS | sometimes bash makes me o_O | 16:24 |
| SpamapS | 2017-09-28 15:38:59.436730 | ubuntu-xenial | + [[ openstack/cinder != \o\p\e\n\s\t\a\c\k\/\r\e\q\u\i\r\e\m\e\n\t\s ]] | 16:24 |
| fungi | better to repeat something we know about than to say nothing ;) | 16:24 |
| *** LindaWang has quit IRC | 16:25 | |
| *** lukebrowning has joined #openstack-infra | 16:25 | |
| mordred | clarkb, jeblair: whatis the right nodeset? didn't we make a devstack multinode nodeset? | 16:26 |
| clarkb | legacy-grenade-dsvm-neutron passed | 16:26 |
| mordred | or just ubuntu-xenial-2-node I guess | 16:27 |
| clarkb | ubuntu-xenial-2-node | 16:27 |
| jeblair | that looks right | 16:27 |
| clarkb | and ubuntu-xenial-3-node in a small number of cases | 16:27 |
| clarkb | (neutron dvr ha multinode job uses 3-node) | 16:27 |
| mordred | clarkb: small enoughthat we can just fix those ex-post-facto? | 16:28 |
| clarkb | mordred: ya I think so | 16:28 |
| clarkb | for 3-node | 16:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add multinode nodeset to multinode jobs missing it https://review.openstack.org/508222 | 16:28 |
| clarkb | I'll work on compiling that list now for post fixing | 16:28 |
| jeblair | mordred: -1 on https://review.openstack.org/508219 | 16:28 |
| mordred | jeblair: cool - will fix in a sec | 16:29 |
| mordred | clarkb: oh - we can grep for it in openstack-zuul-jobs | 16:29 |
| mordred | clarkb: the playbooks all have a line at the top indicated what job they were converted from | 16:29 |
| clarkb | '{pipeline}-tempest-dsvm-neutron-dvr-ha-multinode-full-{node-release}{suffix}' gate-openstack-helm{sub}-{integration}-{type}-{sdn}-{gl}-{node}{suffix} gate-openstack-helm{sub}-{integration}-{type}-{sdn}-{gl}-{node}{suffix} | 16:29 |
| clarkb | those are the only 3 with 3node xenial. And I think all the multnode centos stuff was done properly | 16:30 |
| mordred | oh - nevermind. that'll only tell us the ones that worked anyway because they had 3-node in their title | 16:30 |
| *** lukebrowning has quit IRC | 16:30 | |
| *** gongysh has joined #openstack-infra | 16:30 | |
| mordred | the helm ones got it - so it's just tempest-dsvm-neutron-dvr-ha-multinode-full | 16:30 |
| *** kiennt26 has quit IRC | 16:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add multinode nodeset to multinode jobs missing it https://review.openstack.org/508222 | 16:31 |
| mordred | ok. that fixes the one that needed 3 node | 16:31 |
| *** lukebrowning has joined #openstack-infra | 16:32 | |
| *** pbourke has quit IRC | 16:32 | |
| jeblair | Shrews: from zuul log: http://paste.openstack.org/show/622182/ | 16:32 |
| mordred | jeblair: oh - we can actually remove that shell block - the tox base job already sets that environment variable :) | 16:32 |
| mordred | oh - no we can't - we're not using that role in the right place - nevermind | 16:33 |
| *** ralonsoh has quit IRC | 16:33 | |
| pabelanger | just checking in, looked at a failure, took me to the follow playbook. http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/requirements-tox-babel/run.yaml#n58 I never knew shell: cmd: foo was valid syntax | 16:34 |
| jeblair | i would argue it's the only valid syntax | 16:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Make new releasenotes job https://review.openstack.org/508219 | 16:34 |
| *** gongysh has quit IRC | 16:34 | |
| Shrews | jeblair: that's odd | 16:34 |
| jeblair | Shrews: lots of errors like that | 16:35 |
| clarkb | multinode fix is approved | 16:35 |
| mordred | pabelanger: yah - we learned about it recently ... there's a case where it's required | 16:35 |
| *** Swami has quit IRC | 16:36 | |
| jeblair | all multiline shell commands are wrong without it -- everything is indented by one space, you just don't usually notice because bash is usually forgiving about indentation. but not with heredocs. | 16:36 |
| *** lukebrowning has quit IRC | 16:36 | |
| mordred | ah - yah. heredocs were the thing | 16:36 |
| jeblair | Shrews: i'm looking to see if zuul disconnected or something | 16:36 |
| jeblair | Shrews: i don't see any kazoo exceptions other than nonodeerror | 16:37 |
| jeblair | though what if the logging config eliminates those | 16:38 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/tripleo-ci master: Make sure we always have /etc/machine-id available. https://review.openstack.org/508226 | 16:38 |
| Shrews | jeblair: yeah, not sure what could be happening there. zuul checks if the lock is set before attempting to store the node again. maybe that check is quirky? | 16:38 |
| *** panda is now known as panda|off | 16:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: DNM testing releasenotes job https://review.openstack.org/508227 | 16:38 |
| *** kjackal_ has quit IRC | 16:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add kazoo info logging https://review.openstack.org/508228 | 16:41 |
| jeblair | Shrews, clarkb, mordred, fungi: ^ we may have a zookeeper connection issue that we're blind to | 16:41 |
| mordred | jeblair: +3'd it | 16:41 |
| clarkb | you might need to force merge it, zuul looks like its busy reconfiguring with the change swe have been pushing in | 16:42 |
| pabelanger | mordred: Shrews: re: tripleo-test-cloud-rh1, we need to update clouds.yaml and setup the right network, shade isn't able to figure it out | 16:42 |
| Shrews | jeblair: http://paste.openstack.org/show/622183/ from nl02 logs | 16:42 |
| *** AJaeger_ has joined #openstack-infra | 16:42 | |
| *** slaweq_ has quit IRC | 16:42 | |
| SamYaple | im going through the zuulv3 docs. i have a custom job for an existing patchset. should a 'recheck' work? (or is the v3 migration not done?) | 16:42 |
| pabelanger | mordred: Shrews: eg: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n617 | 16:42 |
| mordred | pabelanger: we support networks: as config in v3 too | 16:43 |
| Shrews | jeblair: so if the times are in sync, it was deleted 2 minutes before zuul returned it? | 16:43 |
| jeblair | Shrews: yeah. that seems consistent with the theory that zuul lost the zk connection and therefore the lock | 16:43 |
| fungi | SamYaple: a recheck will probably run it, but you may want to wait until we've got in-flight fixes to some obvious errors/oversights in existing jobs merged | 16:43 |
| pabelanger | mordred: ah, let me fix then | 16:44 |
| mordred | pabelanger: got it | 16:44 |
| *** lukebrowning has joined #openstack-infra | 16:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add network list to tripleo cloud https://review.openstack.org/508229 | 16:44 |
| mordred | pabelanger: ^^ | 16:44 |
| SamYaple | fungi: ok. cool. shoudl native v3 jobs be working though? (as in, can i start writing and testing v3 jobs right now, or should i wait on that too?) | 16:44 |
| mordred | SamYaple: it shoudl be working - but I'd still wait for a little bit - high amount of flux in the system currently | 16:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Use 'private' network for tripleo-test-cloud-rh1 https://review.openstack.org/508230 | 16:45 |
| pabelanger | mordred: ha, beat me | 16:45 |
| fungi | SamYaple: as mordred said, i would wait for now... but your enthusiasm is commendable! | 16:45 |
| SamYaple | cool. ill just start writing the job and plan on testing later | 16:45 |
| mordred | pabelanger: actually - I think yours is better | 16:45 |
| pabelanger | mordred: oh? | 16:46 |
| SamYaple | fungi: ive been waiting for zuulv3 for years. im ready. | 16:46 |
| pabelanger | ya, it is | 16:46 |
| pabelanger | pools | 16:46 |
| mordred | pabelanger: yah -it's a pool parameter, which you did - I put it up a level incorrectly | 16:46 |
| pabelanger | restored | 16:46 |
| *** trown is now known as trown|lunch | 16:46 | |
| pabelanger | https://review.openstack.org/508230/ | 16:46 |
| pabelanger | Shrews: ^ | 16:46 |
| pabelanger | okay, my coffee break is over, back to moving | 16:46 |
| *** numans has quit IRC | 16:46 | |
| *** rwsu has joined #openstack-infra | 16:47 | |
| *** vhosakot has joined #openstack-infra | 16:49 | |
| *** numans has joined #openstack-infra | 16:49 | |
| jeblair | nothing interesting in zookeeper log | 16:49 |
| *** lukebrowning has quit IRC | 16:49 | |
| jeblair | i'm growing concerned we may be stuck | 16:52 |
| jeblair | oh, nope, it's moving | 16:54 |
| SpamapS | Just a dict() that got really big really fast. ;) | 16:54 |
| clarkb | jeblair: I think its just slow due to the updates we've been making | 16:55 |
| *** gongysh has joined #openstack-infra | 16:55 | |
| *** lukebrowning has joined #openstack-infra | 16:55 | |
| clarkb | but I don't have hrad data for that moslty just inferring from observed behavior | 16:55 |
| *** derekh has quit IRC | 16:56 | |
| clarkb | hrm I guess its just the uuid change that has merged though | 16:56 |
| *** gongysh has quit IRC | 16:56 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add debug line for deleting unlocked nodes https://review.openstack.org/508235 | 16:56 |
| Shrews | jeblair: ^^^ should help us identify such occasions when zuul loses the lock and we delete the node | 16:57 |
| *** jascott1 has quit IRC | 16:57 | |
| *** jascott1 has joined #openstack-infra | 16:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Improve debug logging around dynamic layout loading https://review.openstack.org/508237 | 16:57 |
| jeblair | Shrews: ++ | 16:58 |
| jeblair | Shrews: oh wait, what state does zuul leave it in normally? | 16:58 |
| jeblair | Shrews: i think it always leaves it in used | 16:58 |
| *** jpich has quit IRC | 16:59 | |
| Shrews | jeblair: it should be used. but if it is "IN-USE" and unlocked, we consider it abandoned | 16:59 |
| jeblair | Shrews: got it! | 16:59 |
| *** lukebrowning has quit IRC | 16:59 | |
| *** jascott1 has quit IRC | 17:00 | |
| mordred | Shrews, jeblair: speaking of states, but not urgent, we don't seem to be tracking v3 nodes in nodepool in grafana - the keys look largely the same (although there doesn't seem to be a nodepool.provider.*.nodes.building or a nodepool.provider.*.nodes.used anymore) | 17:00 |
| *** baoli has quit IRC | 17:00 | |
| mordred | jeblair: are you sure you want to put kazoo to debug in tests? didn't we turn that down because immense log sizes a while back? | 17:00 |
| jeblair | mordred: that's info | 17:01 |
| *** baoli has joined #openstack-infra | 17:01 | |
| jeblair | mordred: debug is huge, info is not | 17:01 |
| mordred | jeblair: in the https://review.openstack.org/508237 change | 17:01 |
| mordred | jeblair: it includes a change to set kazoo to debug in tests.base | 17:01 |
| fungi | jeblair: with 508237 why is it also changing kazoo's loglevel in zuul tests. was that leaking from another commit? | 17:01 |
| jeblair | mordred: oh that's an accident, thanks | 17:01 |
| *** lukebrowning has joined #openstack-infra | 17:01 | |
| mordred | fungi: jinx | 17:01 |
| fungi | er, i guess that's what mordred spotted too | 17:01 |
| jeblair | mordred: i thought you were asking about 508228 | 17:02 |
| mordred | no - that one is great | 17:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Improve debug logging around dynamic layout loading https://review.openstack.org/508237 | 17:02 |
| *** AJaeger_ has quit IRC | 17:02 | |
| Shrews | ok, afk'ing for a few minutes for lunch | 17:02 |
| *** AJaeger_ has joined #openstack-infra | 17:03 | |
| jeblair | Shrews: oh hey we do have some zookeeper connection log lines in zuul | 17:03 |
| *** jascott1 has joined #openstack-infra | 17:03 | |
| jlvillal | So the page: https://docs.openstack.org/infra/manual/zuulv3.html talks about the "status page" | 17:03 |
| jlvillal | But no link to the status page | 17:04 |
| jlvillal | Is the status page still: http://status.openstack.org/zuul/ | 17:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove zuulv3-dev statsd prefix from v3 launchers https://review.openstack.org/508241 | 17:04 |
| jeblair | http://paste.openstack.org/show/622184/ | 17:04 |
| jeblair | we definitely have some zk connection issues | 17:04 |
| mordred | jlvillal: there's an update to that coming | 17:04 |
| jlvillal | mordred: Thanks. I can't see jobs for just pushed patch. So curious :) | 17:04 |
| clarkb | jeblair: do we a connection per node or similar? could justbe running out of fds on the nodepool server | 17:04 |
| mordred | jlvillal: you can look at http://zuulv3.openstack.org/ for now | 17:05 |
| jlvillal | mordred: Sweet, thanks | 17:06 |
| *** jascott1 has quit IRC | 17:06 | |
| mordred | clarkb: I believe it should be connection per ensemble member - so there should be a connection for each launcher, each builder and for the zuul scheduler | 17:06 |
| jeblair | mordred: right | 17:07 |
| clarkb | nodepoold on nodepool.o.o is using significant memory which may be starving zk? | 17:07 |
| fungi | in other words, a relative handful, not a truckload | 17:07 |
| *** jascott1 has joined #openstack-infra | 17:07 | |
| clarkb | though zk doesn't really seem to need memory according to top | 17:07 |
| jeblair | zuul, however, is using quite a lot. :| | 17:08 |
| clarkb | ya 97 file descriptors for zk so that should be fine | 17:08 |
| SpamapS | ZK is about as optimized and well behaved as a java app gets. | 17:08 |
| mordred | jeblair: so it might be memory-pressure on teh client side causing the zk thread to not be able to keep up with membership | 17:08 |
| fungi | we're a gigabyte into swap on nodepool.o.o, so probably want to consider rebuilding with a larger flavor | 17:08 |
| SpamapS | swap's bad mmmk | 17:09 |
| *** Swami has joined #openstack-infra | 17:09 | |
| clarkb | fungi: well its the nodepoold which won't exist for very long | 17:09 |
| fungi | ahh | 17:09 |
| clarkb | we can probably just turn it off at this point and see if it helps zk | 17:09 |
| mordred | well - if nodepool is causing swap, it could be that zk is occasionally getting swapped out | 17:09 |
| * mordred votes to stop the nodepool service on nodepool.o.o to remove that variable from the equation | 17:09 | |
| fungi | speaking of swap... we're missing swap entirely on zuulv3.o.o | 17:10 |
| fungi | not that it seems to want it for the moment | 17:10 |
| jeblair | nl01 has never lost its zk connection | 17:11 |
| mordred | nod | 17:11 |
| mordred | so more likely the zuul side of the coin | 17:11 |
| *** claudiub|3 has quit IRC | 17:11 | |
| SpamapS | fungi: no server ever wants swap :) | 17:11 |
| SpamapS | but sometimes.. they take it | 17:11 |
| SpamapS | for the team | 17:11 |
| jeblair | nl02 lost it once 11 days ago | 17:11 |
| jeblair | mordred: yeah | 17:11 |
| jeblair | i'm guessing it's something like zuul being so busy that the GIL is starving the zk thread momentarily | 17:12 |
| *** spzala has joined #openstack-infra | 17:12 | |
| jeblair | so maybe we need to make our zk server more tolerant of laggy clients? | 17:13 |
| mordred | jeblair: I tihnk that's reasonable given our usecase | 17:13 |
| jeblair | SpamapS: any advice? | 17:14 |
| SpamapS | jeblair: harlowja is in transit to the office I"m sitting in righ tnow. I expect him any minute. | 17:14 |
| SpamapS | He's far more versed in zk tuning than I. | 17:14 |
| SpamapS | Can we suss out the symptoms while we wait for him to navigate bay area traffic? ;) | 17:14 |
| *** gtmanfred has quit IRC | 17:15 | |
| jeblair | SpamapS: we're pretty limited by our lack of logs at the moment. all we know is that now that zuulv3 is busier, it's losing zk connections periodically. other zk clients are not. so i'm inclined to say that zuul should negotiate a higher session timeout. | 17:16 |
| *** baoli has quit IRC | 17:16 | |
| jeblair | (we'll get more logs soon) | 17:16 |
| *** dizquierdo has quit IRC | 17:16 | |
| SpamapS | jeblair: that makes sense. I believe you can inspect the session objects and get some idea of the age of each session from ZK's point of view. | 17:17 |
| * SpamapS looks at his own ZK | 17:17 | |
| *** sambetts is now known as sambetts|afk | 17:17 | |
| jeblair | it looks like we're using the default timeout | 17:17 |
| SpamapS | which is probably something like 5s? | 17:17 |
| SpamapS | or 1s | 17:17 |
| jeblair | oh wait, i assumed the timeout arg here was session timeout, but it may just be connection timeout: http://kazoo.readthedocs.io/en/latest/api/client.html#kazoo.client.KazooClient | 17:18 |
| *** nikhil has quit IRC | 17:18 | |
| mordred | so - zk server side there are two defaults: | 17:18 |
| mordred | minSessionTimeout - the minimum session timeout in milliseconds that the server will allow the client to negotiate. Defaults to 2 times the tickTime. | 17:18 |
| mordred | maxSessionTimeout - the maximum session timeout in milliseconds that the server will allow the client to negotiate. Defaults to 20 times the tickTime. | 17:18 |
| jeblair | (looking at the code, maybe it's both?) | 17:18 |
| jeblair | ah, no i think the constructor timeout arg is session timeout, and then the start() method timeout arg is connect timeout | 17:19 |
| mordred | 2 times the tickTime and 20 times the tickTime are probably not the right choices for this case | 17:19 |
| mordred | jeblair: cool. so we can set server settings for min and max and then set those to something on the client potentially | 17:19 |
| jeblair | mordred: assuming our tick time is 2 seconds -- the default server range should be 4-40 seconds, yeah? | 17:20 |
| SpamapS | Clients set their timeout | 17:20 |
| SpamapS | the server can just limit them to min/max | 17:20 |
| mordred | jeblair: yes | 17:20 |
| SpamapS | so the constructor timeout is the thing to raise | 17:21 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Make gate-grenade-dsvm-neutron-multinode-live-migration-nv experimental https://review.openstack.org/508244 | 17:21 |
| SpamapS | (at a cost of liveness) | 17:21 |
| mordred | that patch is not so much going to work | 17:21 |
| jeblair | it looks like the kazoo default is 10 seconds...? | 17:23 |
| inc0 | hey guys, so you're pulling zuulv3 trigger today? | 17:23 |
| *** Hal has joined #openstack-infra | 17:23 | |
| *** Hal is now known as Guest50285 | 17:24 | |
| SpamapS | jeblair: ow | 17:24 |
| SpamapS | but yeah | 17:24 |
| SpamapS | jeblair: I agree, just checked the code | 17:24 |
| *** baoli has joined #openstack-infra | 17:24 | |
| fungi | inc0: yes, it's underway now | 17:24 |
| SpamapS | assuming tickTime == 1s | 17:24 |
| SpamapS | I think | 17:24 |
| fungi | inc0: for the past ~3 hours | 17:24 |
| *** SumitNaiksatam has joined #openstack-infra | 17:24 | |
| jeblair | SpamapS: i thought tick was 2s? we should find out. | 17:25 |
| inc0 | cool, awesome guys! can't wait:) | 17:25 |
| mordred | inc0: I saw zuulv3 run some kolla patches already- I thnk some of them may have even passed successfully | 17:25 |
| *** numans has quit IRC | 17:26 | |
| SpamapS | jeblair: it is in the default config. | 17:26 |
| SpamapS | jeblair: assuming we didn't change it, that means timeout is actually *20s* | 17:26 |
| *** jascott1 has quit IRC | 17:26 | |
| *** jascott1 has joined #openstack-infra | 17:26 | |
| inc0 | mordred: passing successfully? That can't be right:) | 17:26 |
| mordred | inc0: ikr? | 17:26 |
| jeblair | SpamapS: i think the client timeout is negotiated in milliseconds, and kazoo does the conversion; so kazoo is sending a timeout of 10,000 ms (10s) | 17:27 |
| inc0 | are you sure you didn't leave "if True" there? | 17:27 |
| SpamapS | jeblair: agreed | 17:27 |
| * SpamapS feels like even though we're not paired, jeblair and I are looking at the same stuff :) | 17:27 | |
| SpamapS | jeblair: I think it's worth a shot to pass timeout=20 | 17:27 |
| jeblair | quite likely, my screens are full of kazoo code | 17:27 |
| jeblair | SpamapS: and maybe even higher if that still causes problems | 17:28 |
| SpamapS | I'm just checking to make sure that won't blow up on the max | 17:28 |
| jeblair | zuul-scheduler is a spof, and doesn't go down that often, and we don't gain much from deleting all the nodes when things hiccup | 17:28 |
| jeblair | i'll start plumbing that through | 17:28 |
| SpamapS | jeblair: also worth considering why we're not minding our zk thread more. | 17:28 |
| inc0 | mordred: where can I see kolla jobs code migrated to zuulv3? | 17:29 |
| inc0 | (I assume project-config changed?) | 17:29 |
| *** coolsvap has quit IRC | 17:29 | |
| clarkb | inc0: in openstack-zuul-jobs/playbooks/legacy | 17:29 |
| jeblair | inc0: https://docs.openstack.org/infra/manual/zuulv3.html#legacy-job-migration-details | 17:29 |
| inc0 | ok, I'll start working on migrating it to non-lecacy | 17:30 |
| *** Apoorva_ has joined #openstack-infra | 17:30 | |
| *** kjackal_ has joined #openstack-infra | 17:30 | |
| jeblair | inc0: that document has lots of info that aims to help in that process too :) | 17:30 |
| * inc0 grabbing hot cup of tea and gets to reading then | 17:31 | |
| *** numans has joined #openstack-infra | 17:31 | |
| *** esberglu has quit IRC | 17:31 | |
| *** baoli has quit IRC | 17:32 | |
| *** baoli has joined #openstack-infra | 17:32 | |
| SpamapS | jeblair: the default maxSessionTimeout does seem to be 20 * tickTime | 17:32 |
| *** camunoz has quit IRC | 17:33 | |
| SpamapS | and tickTime is 2000ms, so timeout=40 would be the top end | 17:33 |
| *** Apoorva has quit IRC | 17:33 | |
| fungi | i vote we just negotiate 40s in that case and see if we still get problems | 17:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add ZK session timeout option https://review.openstack.org/508251 | 17:35 |
| fungi | as jeblair says, there's not much sanity to be gained by having it handle the zuul scheduler getting timed out | 17:35 |
| fungi | so anything below the max allowable timeout is probably nonsensical in this situation | 17:36 |
| andreaf | mtreinish, jeblair: what do you think if we dropped the legacy- bit from job names so we don't lose job history twice in openstack health? With the current setup we lose it now because of the extra legacy in front, and then again once the job is migrated to v3 native | 17:37 |
| mordred | andreaf: dropping the legacy bit will mkae it harder for people to make new v3 native jobs | 17:38 |
| andreaf | mordred: I meant in the subunit worker just before adding the data to o-h | 17:38 |
| mordred | andreaf: AH. I understand what you are saying | 17:39 |
| jeblair | many other parts of the job name changed too | 17:39 |
| andreaf | mordred: I guess that might be confusing when looking for data but at least we would not lose 6 months of historical data on all jobs | 17:39 |
| SpamapS | jeblair: stracing an idle zuul-scheduler shows me that we're only waking up our zk thread every 1s. | 17:39 |
| andreaf | jeblair: oh is it? | 17:39 |
| mordred | andreaf: it might be a little tricky to do so generally - we don't know if a legacy job is going to transition to a per-project job or a global job | 17:39 |
| andreaf | jeblair: too bad | 17:39 |
| jeblair | -ubuntu-xenial and -nv were dropped, for instance | 17:40 |
| SpamapS | jeblair: I wonder if our thread would get a little more attention if we made it every 0.5s | 17:40 |
| andreaf | jeblair: ok, than it's not worth the effort | 17:41 |
| andreaf | btw zuulv3 status page says Zuul version: 2.5.3.dev1360 - is that right? | 17:41 |
| mordred | yah | 17:41 |
| mordred | we have not yet released a 3.0 tag (still more work to do before we hit that) | 17:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add support for zookeeper timeout https://review.openstack.org/508253 | 17:42 |
| andreaf | ok | 17:42 |
| inc0 | so if I create zuul.d in kolla-ansible repo, copy https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/zuul-legacy-jobs.yaml#L4417-L4425 <- this to a file there and copy related files from playbooks/legacy | 17:43 |
| inc0 | that's how I do it? | 17:43 |
| inc0 | (to start moving our jobs to our repo | 17:43 |
| inc0 | ) | 17:43 |
| mordred | inc0: yes - just put those jobs into a .zuul.yaml in your repo and rename them ... one sec, I was just about to say something on this particular topic ... | 17:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Set zuul zookeeper session timeout to 40s https://review.openstack.org/508255 | 17:45 |
| jeblair | SpamapS, fungi, mordred: ^ | 17:45 |
| harlowja | halo | 17:45 |
| harlowja | *hallo | 17:45 |
| harlowja | pythn is the best | 17:45 |
| harlowja | lol | 17:45 |
| *** esberglu has joined #openstack-infra | 17:45 | |
| inc0 | well I'll go straight for zuul.d because I want to abuse ansible jobs later on;) | 17:45 |
| jeblair | harlowja: have you met my friend, Gil? | 17:45 |
| harlowja | bffs | 17:46 |
| mordred | infra-root: this morning I wrote an email about v3 for the list: https://etherpad.openstack.org/p/ZUxJCrFfX6 which was "tl;dr - also go read the manual" - but grew a bit and has a few more things I think I should add to the infra manual - I also split off two more emails: https://etherpad.openstack.org/p/MjocbH8Jic how to migrate legacy jobs and https://etherpad.openstack.org/p/hSkETS28OE updates | 17:46 |
| mordred | specifically about changes to tox jobs | 17:46 |
| *** jpena is now known as jpena|off | 17:47 | |
| mordred | infra-root: I'm thinking now I should pull out a bunch of things from each of them and add to the infra-manual (rewritten a bit) - but thought it might still be useful to send one, two or all of them - depending on what y'all think | 17:47 |
| jeblair | harlowja: briefly: we think we're hitting session timeouts because zuul is "busy". but we don't have kazoo logs to confirm that (we will soon). in the mean time, that's our best guess for why we periodically lose connections. our zk is all defaults, so we think the session timeout range is between 4s and 40s, and kazoo defaults to 10s. we're going to bump our timeout to 40s (because this is a spof component, and false timeouts ... | 17:48 |
| jeblair | ... gain us nothing). | 17:48 |
| jeblair | harlowja: any obvious errors in that ^? | 17:48 |
| harlowja | jeblair only thing is empheral nodes i think are connected to that timeout | 17:48 |
| harlowja | so that may or may not be ok, prob ok :-P | 17:48 |
| jeblair | harlowja: yeah, that's the main thing -- zuul holds locks on nodes it's using, and if it disconnections, the nodes get deleted. so we'd rather wait a few extra seconds after a catastrophic zuul failure to delete nodes, rather than have them deleted erroneously while jobs are still running. | 17:49 |
| jeblair | s/disconnections/disconnects/ | 17:50 |
| harlowja | makes sense | 17:50 |
| mtreinish | andreaf: wait what legacy bit? | 17:50 |
| * mtreinish hasn't been paying enough attention | 17:50 | |
| harlowja | jeblair see if u can get the ping/pong logs from when the client is doign the ping/pong | 17:50 |
| SpamapS | jeblair: I'm surprised at the number of threads my idle zuul-scheduler has... | 17:50 |
| SpamapS | ~30 | 17:51 |
| andreaf | mtreinish: all migrated job names are called legacy-[something similar to the original name] | 17:51 |
| SpamapS | Trying to account for all of them | 17:51 |
| mtreinish | andreaf: ah ok, yeah we can just drop that in the subunit worker | 17:51 |
| mtreinish | pretty easy filter to add | 17:51 |
| mtreinish | we still need to push a patch up to fix the names to be unique post migration too | 17:52 |
| fungi | SpamapS: looking at a sigusr2 thread dump? | 17:53 |
| harlowja | jeblair sorry its BLATHER logs and u'll get even more | 17:53 |
| harlowja | where BLATHER < DEBUG, ha | 17:53 |
| SpamapS | fungi: I forgot about that. I was just attaching gdb/strace :) | 17:53 |
| inc0 | mordred: can I propose migration patch already or too early for you? | 17:53 |
| andreaf | mtreinish: well, that was my plan, only jeblair mentioned that the rest of the name changed as well, like ubuntu-xenial and -nv are not there anymore | 17:53 |
| fungi | harlowja: it has a loglevel named "blather"? genius | 17:53 |
| harlowja | ya, below debug | 17:54 |
| clarkb | inc0: we are still working thorugh some issues, probably best ot wait at this point | 17:54 |
| harlowja | ha | 17:54 |
| mtreinish | andreaf: yeah it's all different, I think we're just going to have to lose the history | 17:54 |
| harlowja | https://github.com/python-zk/kazoo/blob/master/kazoo/loggingsupport.py#L1 fungi | 17:54 |
| harlowja | it's what I think openstack calls 'TRACE' | 17:54 |
| harlowja | but kazoo called it BLATHER, ha | 17:54 |
| inc0 | clarkb: roger roger | 17:54 |
| mtreinish | but for query performance we still need the names to be unique per project | 17:54 |
| *** camunoz has joined #openstack-infra | 17:54 | |
| andreaf | mtreinish: yeah but we're losing it twice... first with the automatic migration, and then again once jobs are migrated to native zuul v3 :( | 17:54 |
| harlowja | fungi clarkb u'll be able to see https://github.com/python-zk/kazoo/blob/master/kazoo/protocol/connection.py#L280-L282 with BLATHER | 17:55 |
| harlowja | which might be useful | 17:55 |
| harlowja | to see when the PING request go out | 17:55 |
| andreaf | mtreinish: the 2nd migration is going to be job by job though, not mass-migration like the first one | 17:55 |
| inc0 | quick question, where do we specify voting/non-voting in zuulv3? can't see it zuul-legacy-jobs | 17:56 |
| *** baoli has quit IRC | 17:56 | |
| mnaser | voting: in job definition | 17:56 |
| *** baoli has joined #openstack-infra | 17:57 | |
| jeblair | inc0: example in https://docs.openstack.org/infra/manual/zuulv3.html#how-jobs-are-selected-to-run-in-zuul-v3 | 17:57 |
| SpamapS | fungi: thanks! | 17:57 |
| jeblair | mtreinish: we've already lost unique names per project (all unit test jobs have the same name now) | 17:57 |
| Shrews | hrm, that zk timeout thing was always making an itch in the back of my brain | 17:57 |
| SpamapS | so an idle zuul-scheduler seems to have about 10 threads devoted to gearman connections, and 10 threads devoted to paste webserver (webapp?) | 17:58 |
| Shrews | did we ever determine what "ticks" equated to? | 17:58 |
| SpamapS | jeblair: are we using the internal zuul-scheduler geard? | 17:58 |
| jeblair | Shrews: we think we're on the default which is 2s | 17:58 |
| jeblair | SpamapS: yes | 17:58 |
| SpamapS | do we have to? | 17:58 |
| jeblair | SpamapS: note that is a separate process | 17:58 |
| SpamapS | oh ok | 17:58 |
| jeblair | SpamapS: should not affect GIL | 17:58 |
| SpamapS | that's the forked one, k | 17:58 |
| SpamapS | so those are all gear client connections | 17:59 |
| clarkb | Shrews: I have just confirmed we are set to 2000ms per tick | 17:59 |
| SpamapS | guessing 1 per webserver thread | 17:59 |
| SpamapS | plus one for executor client | 17:59 |
| mnaser | is there a specific reasoning behind `BUILD_TIMEOUT` in vars for migrated jobs? | 17:59 |
| jeblair | mnaser: some jobs (esp devstack-gate) used them | 18:00 |
| jeblair | was difficult to determine if it was necessary or not automatically | 18:00 |
| clarkb | allows the test framework to kill the tests early so that logs can be collected. Shouldn't be a problem for zuulv3 native jobs | 18:00 |
| mtreinish | jeblair: right and that means we've lost all the history for those jobs now | 18:00 |
| clarkb | but until we convert everything the old things likely still need it | 18:00 |
| mnaser | gotcha, ill check if we use it in puppet jobs (tho i dont think we do) | 18:00 |
| mtreinish | and it also means we can't actually use openstack-health for unit test jobs per project right now | 18:01 |
| jeblair | mtreinish: yeah; i'm guessing the convesation about improving the sql performance for joins never happened at the ptg? | 18:01 |
| mtreinish | because they all get treated as the same thing | 18:01 |
| Shrews | SpamapS: if you're counting scheduler threads, note that kazoo uses 3 for a single connection | 18:01 |
| *** edmondsw has joined #openstack-infra | 18:01 | |
| mtreinish | jeblair: it did, but a migration like that is going to take a big downtime because of the trove fun | 18:01 |
| mtreinish | jeblair: and we didn't have time to do it before the switchover | 18:02 |
| mtreinish | which is why I'm writing a patch right now for the subunit2sql worker to create unique names per project | 18:02 |
| SpamapS | Shrews: yeah I have harlowja here sitting with me who explained that. :) | 18:02 |
| mtreinish | in the subunit2sql db | 18:02 |
| SpamapS | so, confirmed | 18:02 |
| *** edmondsw has quit IRC | 18:04 | |
| jeblair | Shrews, mordred: want to +3 https://review.openstack.org/508255 ? | 18:04 |
| clarkb | I'll get it | 18:04 |
| *** edmondsw has joined #openstack-infra | 18:04 | |
| jeblair | also need a +3 on https://review.openstack.org/508237 | 18:04 |
| jlvillal | So things that were currently building in Zuul v2, prior to the changeover. Will they finish building? I tried to telnet to the log port and no response. But I still see the jobs active in the status page. http://status.openstack.org/zuul/ | 18:04 |
| jlvillal | But they have been building for much much longer than normal | 18:04 |
| jeblair | jlvillal: reload the status page and it will be empty | 18:04 |
| jeblair | jlvillal: we did save the queues, but we won't add them back until we know things are more stable | 18:05 |
| jlvillal | jeblair: Not for me I see 108 check jobs | 18:05 |
| jlvillal | jeblair: http://status.openstack.org/zuul/ | 18:05 |
| jeblair | oh neat | 18:05 |
| *** dprince has quit IRC | 18:05 | |
| jeblair | that's a long cache | 18:06 |
| jlvillal | I will take it to mean that those jobs won't finish with Zuul v2. I will wait for Zuul v3. Thanks | 18:06 |
| jeblair | jlvillal: yeah. i'll see i i can expire the cache | 18:07 |
| SpamapS | jlvillal: we've waited years for zuulv3. We appreciate you waiting a few more hours. ;-) | 18:07 |
| jlvillal | heh :) | 18:07 |
| *** baoli has quit IRC | 18:07 | |
| *** dprince has joined #openstack-infra | 18:07 | |
| *** Apoorva_ has quit IRC | 18:09 | |
| jeblair | i'll just stop apache on zuul.o.o | 18:09 |
| *** Apoorva has joined #openstack-infra | 18:10 | |
| *** baoli has joined #openstack-infra | 18:10 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql master: Ensure that build_names are unique per project https://review.openstack.org/508258 | 18:10 |
| mtreinish | andreaf, mordred, jeblair, clarkb: ^^^ | 18:10 |
| clarkb | jeblair: I think zuul might be effectively stuck | 18:12 |
| jeblair | clarkb: yeah, i just did a stack dump | 18:12 |
| clarkb | jeblair: we might need to force merge some of these changes and then restart? | 18:12 |
| *** dprince has quit IRC | 18:13 | |
| *** trown|lunch is now known as trown | 18:13 | |
| jeblair | 2017-09-28 17:37:22,184 DEBUG zuul.nodepool: Locking node <Node 0000039803 primary:ubuntu-xenial> | 18:13 |
| jeblair | i think that's what the scheduler is doing :/ | 18:14 |
| *** dprince has joined #openstack-infra | 18:14 | |
| SpamapS | isn't that like | 18:14 |
| *** jkilpatr has joined #openstack-infra | 18:14 | |
| SpamapS | 20+ minutes ago? | 18:14 |
| jeblair | ya | 18:14 |
| SpamapS | almost 40 actually | 18:14 |
| * SpamapS can't math | 18:14 | |
| fungi | >1 minute anyway | 18:14 |
| jeblair | that was right around a zk disconnect/reconnect | 18:15 |
| jeblair | 2017-09-28 17:37:20,659 DEBUG zuul.zk.ZooKeeper: ZooKeeper connection: CONNECTED | 18:15 |
| *** amoralej is now known as amoralej|off | 18:15 | |
| SpamapS | there's a timeout for acquiring locks too | 18:17 |
| mnaser | ...so say some of our jobs need fixes and i did a bit of refactoring on the jobs .. is it a good time to push them up? :x | 18:17 |
| SpamapS | and we don't set one | 18:17 |
| mordred | mnaser: not quite yet | 18:17 |
| jeblair | SpamapS: agreed. maybe we should, eh? | 18:17 |
| mnaser | alright cool, ill hang and watch | 18:17 |
| jeblair | 10s? 30s? | 18:17 |
| jeblair | or the fungi threshold of 1 minute? :) | 18:18 |
| fungi | heh | 18:18 |
| fungi | i do math in rough magnitudes | 18:18 |
| SpamapS | I think a long timeout is fine in this case | 18:18 |
| *** ekcs has joined #openstack-infra | 18:19 | |
| SpamapS | BUt it should be less than the session timeout | 18:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Wait up to 30s to lock a node https://review.openstack.org/508260 | 18:19 |
| fungi | if it's going to be triggered by zk timeouts, and we think we're mostly eliminating those with the longer timeout values, then yes a lengthy timeout to catch this condition seems fine | 18:19 |
| fungi | 30s lgtm | 18:20 |
| *** caphrim007 has quit IRC | 18:20 | |
| mordred | I just realized - us setting BUILD_TIMEOUT in vars is not likely to have any useful effect - since jobs that consume it are assuming they're doing so via env vars | 18:20 |
| *** caphrim007 has joined #openstack-infra | 18:20 | |
| clarkb | BUILD_TIMEOUT is the special env var | 18:20 |
| mordred | yah. but that's not how we're actually setting it | 18:20 |
| clarkb | so as long as that ends up in the env it should work | 18:20 |
| * mordred will make patches to fix it | 18:21 | |
| *** sshnaidm is now known as sshnaidm|off | 18:21 | |
| jeblair | SpamapS, harlowja: i don't know if this traceback is useful to you, but this is the call stack that zuul is sitting in waiting for a lock: http://paste.openstack.org/show/622192/ | 18:21 |
| fungi | mordred: oh, so we're only setting it as an ansible variable not passing it into the shell environment? | 18:21 |
| mordred | yah. easy enough to fix | 18:21 |
| harlowja | jeblair not sure, need timeout :-P | 18:21 |
| harlowja | timeout good | 18:21 |
| harlowja | lol | 18:22 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Exclude remove-machine-id from bootstrap https://review.openstack.org/508263 | 18:22 |
| jeblair | harlowja: :) | 18:22 |
| SpamapS | jeblair: seems likely that the other side of that event was tragically lost in a fire started by timeout in the client ;) | 18:23 |
| jeblair | SpamapS: rip | 18:23 |
| harlowja | https://github.com/python-zk/kazoo/blob/master/kazoo/recipe/lock.py#L250 is the thing for that | 18:23 |
| SpamapS | so it may be belt+suspenders | 18:23 |
| jeblair | i have a favor to ask; i need to afk for a bit then get lunch. can someone take on the task of force-merging all the changes in https://review.openstack.org/#/q/starredby:1+status:open then making sure everything is up to date, then restarting zuulv3? | 18:24 |
| mordred | jeblair: yah. I can do that | 18:24 |
| jeblair | i estimate i'll be back in about 45m | 18:24 |
| *** rbrndt has quit IRC | 18:24 | |
| clarkb | I'll need to do that soon myself but can stagger it so we don't all disappear at once | 18:24 |
| jeblair | mordred: thanks. i'll disappear now. | 18:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add ZK session timeout option https://review.openstack.org/508251 | 18:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Improve debug logging around dynamic layout loading https://review.openstack.org/508237 | 18:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Wait up to 30s to lock a node https://review.openstack.org/508260 | 18:25 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Add kazoo info logging https://review.openstack.org/508228 | 18:25 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Add support for zookeeper timeout https://review.openstack.org/508253 | 18:25 |
| openstackgerrit | Merged openstack-infra/system-config master: Set zuul zookeeper session timeout to 40s https://review.openstack.org/508255 | 18:26 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove zuulv3-dev statsd prefix from v3 launchers https://review.openstack.org/508241 | 18:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Parent legacy jobs with artifacts legacy-publish-openstack-artifacts https://review.openstack.org/508265 | 18:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Set BUILD_TIMEOUT environment variable in zuul_legacy_vars https://review.openstack.org/508266 | 18:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Set BUILD_TIMEOUT environment variable in zuul_legacy_vars https://review.openstack.org/508266 | 18:26 |
| mordred | clarkb, fungi: ^^ that while we're at it | 18:27 |
| fungi | thanks, reviewing those two | 18:27 |
| mordred | I'll do a patch removing it from ozj in a bit | 18:27 |
| mordred | fungi: the other one isn't needed - git derp | 18:27 |
| * jlk is back again, ready to help should any help be desired | 18:27 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Replace jenkins-sudo with zuul-sudo https://review.openstack.org/508212 | 18:27 |
| fungi | ahh | 18:27 |
| *** hasharAway is now known as hashar | 18:27 | |
| fungi | mordred: 508265 is unnecessary you mean? | 18:28 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add openstack/requirements to all legacy-.*tox jobs https://review.openstack.org/508210 | 18:28 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add requirements to legacy-releasenotes https://review.openstack.org/508213 | 18:28 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add multinode nodeset to multinode jobs missing it https://review.openstack.org/508222 | 18:28 |
| fungi | mordred: or just not urgent? | 18:28 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Make new releasenotes job https://review.openstack.org/508219 | 18:28 |
| mordred | fungi: not urgent- it's just an update to the migration script to match what we did for real | 18:29 |
| mordred | but https://review.openstack.org/#/c/508266 is important | 18:29 |
| fungi | yup, lgtm | 18:30 |
| *** baoli has quit IRC | 18:30 | |
| clarkb | do we want ot force merge the set of ozj changes and project-config too in order to make zuul load the new config when it starts? | 18:30 |
| *** dprince has quit IRC | 18:31 | |
| clarkb | mordred: I just approved 8266 | 18:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove BUILD_TIMEOUT from vars https://review.openstack.org/508267 | 18:32 |
| mordred | clarkb, fungi: ^^ there is the removal of BUILD_TIMEOUT from job vars to go along with the filter patch | 18:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Set BUILD_TIMEOUT environment variable in zuul_legacy_vars https://review.openstack.org/508266 | 18:32 |
| clarkb | ugh just asked browser to open that diff | 18:33 |
| fungi | skimmed, but looks right | 18:34 |
| jlk | oh that's a big one | 18:34 |
| jlk | dammit | 18:34 |
| fungi | gertty handled it reasonably well | 18:34 |
| *** baoli has joined #openstack-infra | 18:34 | |
| mordred | I'm going to kick puppet on the various zuul nodes | 18:34 |
| clarkb | approved | 18:35 |
| mordred | /opt/system-config/production/tools/kick.sh 'zuulv3.openstack.org:ze0*:nl0*:nodepool.openstack.org' | 18:35 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove BUILD_TIMEOUT from vars https://review.openstack.org/508267 | 18:35 |
| jlk | RIP my laptop ram... | 18:35 |
| *** srobert_ has joined #openstack-infra | 18:35 | |
| mordred | that should get all of our nodes that have anything to do with zk and get the zuul configs and code up to date | 18:35 |
| clarkb | and upgrade the zuul install | 18:36 |
| clarkb | assuming zuul changes have been merged too | 18:36 |
| clarkb | looks like yes | 18:36 |
| andreaf | mtreinish: you want to skip uniquify only if both tempest and devstack are in the name? | 18:37 |
| mtreinish | andreaf: it should be an or, if tempest or devstack is in the name it should be skipped | 18:38 |
| hwoarang | dirk: ping re https://review.openstack.org/#/c/505657/ | 18:38 |
| mtreinish | but I just realized I probably should change devstack to dsvm | 18:38 |
| *** srobert has quit IRC | 18:38 | |
| mtreinish | andreaf: if it's a dsvm or tempest job it's normally a cross project job so the config is the same or it's a one of project functional or tempest plugin job | 18:38 |
| mtreinish | and will have a name reflecting that so we don't want to append -$project to the end of it for no reason | 18:39 |
| andreaf | mtreinish: well maybe, but you have not in or not in, which is the opposite | 18:39 |
| mtreinish | sigh, did I screw up the logic... | 18:39 |
| Shrews | i'm sort of wondering how we ended up with 205 ready, unlocked, and unallocated nodes in nodepool | 18:40 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql master: Ensure that build_names are unique per project https://review.openstack.org/508258 | 18:41 |
| fungi | demand confusion on nodepool's part? | 18:41 |
| mtreinish | andreaf: ^^^ ok that should fix it | 18:41 |
| mordred | fungi, clarkb: my run of kick.sh seems hung on TASK [puppet : copy system puppet modules] - any ideas why? | 18:41 |
| mordred | fungi, clarkb: NEVERMIND | 18:41 |
| mordred | I'm just impatient | 18:41 |
| fungi | aren't we all | 18:42 |
| Shrews | fungi: i dunno. i'm confused, no reason to think nodepool isn't too | 18:43 |
| mordred | Shrews: are you on the launcher nodes atm? | 18:43 |
| Shrews | mordred: yep | 18:43 |
| mordred | cool- mind restarting the launchers? new config should be deloyed | 18:44 |
| Shrews | mordred: i'd like for this to land first, if we can wait: https://review.openstack.org/#/c/508235/ | 18:44 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add debug line for deleting unlocked nodes https://review.openstack.org/508235 | 18:44 |
| mordred | Shrews: oh - sure - sorry, I mised that in the force-merge list | 18:44 |
| Shrews | mordred: oh, didn't realize that was necessary. i can just restart | 18:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Use 'private' network for tripleo-test-cloud-rh1 https://review.openstack.org/508230 | 18:45 |
| mordred | Shrews: nah - done | 18:45 |
| Shrews | kewl | 18:45 |
| mordred | Shrews: kicking puppet again real quick for nodepool nodes - will let you konw when it's done | 18:46 |
| Shrews | mordred: ack | 18:46 |
| Shrews | mordred: i'll need to 'sudo pip install -U .' first, right? | 18:47 |
| mordred | Shrews: no - puppet should take care of that | 18:47 |
| mordred | v3 scheduler restarted - it's reading config now | 18:47 |
| mordred | Shrews: nodepool nodes updated - should have latest code and config | 18:48 |
| * rbergeron deploys virtual beer to all the good humans | 18:48 | |
| Shrews | stopping nl01 | 18:48 |
| mordred | mmm. bytes of beer | 18:49 |
| Shrews | nl01 restarted. moving to nl02 | 18:49 |
| Shrews | nl02 restarted | 18:50 |
| * evrardjp approves rbergeron and pours some more. | 18:50 | |
| mordred | Shrews: ok. here's hoping we get good logs and also that timeout change just fixes it | 18:51 |
| * fungi pours some out for zuul v2 | 18:51 | |
| openstackgerrit | Eric Kao proposed openstack-infra/project-config master: disable murano plugin in congress main dsvm https://review.openstack.org/508270 | 18:53 |
| clarkb | mordred: looks like we may have missed some requirements required repos http://logs.openstack.org/27/508227/1/check/legacy-python35/9d5b01a/job-output.txt.gz#_2017-09-28_18_54_33_831510 | 18:56 |
| jlk | hrm, http://logs.openstack.org/06/508206/1/check/legacy-rally-tox-self/48271ac/job-output.txt.gz#_2017-09-28_18_54_28_312436 | 18:56 |
| jlk | a problem with sudo? | 18:56 |
| mordred | clarkb: what the heck are we doing with a legacy-python35 job? | 18:57 |
| clarkb | jlk: ya apparnetly the fix for that is not sufficient | 18:57 |
| clarkb | mordred: I dunno thats shade :) | 18:57 |
| mordred | hah | 18:57 |
| clarkb | jlk: that check there is checking that sudo fails after the sudo giving powers config file is dleeted | 18:57 |
| fungi | basically negative testing | 18:57 |
| jlk | yeah, I was poking around at this in a different day | 18:57 |
| jlk | let me find where that is again | 18:57 |
| mordred | clarkb: dagnabit. we were matching against gate-tox-python35-ubuntu-xenial | 18:58 |
| fungi | we should probably do something like `!sudo -n true` there instead | 18:59 |
| jlk | oh holy shit | 18:59 |
| jlk | that's... everywhere | 18:59 |
| jlk | > git grep zuul-sudo |wc -l | 18:59 |
| jlk | 385 | 18:59 |
| fungi | yeah, it's how we did the old revoke-sudo jjb macro | 18:59 |
| Shrews | 2017-09-28 18:57:04,924 DEBUG nodepool.DeletedNodeWorker: Marking for deletion unlocked node 0000039682 (state: used, allocated_to: 100-0000044548) | 19:00 |
| Shrews | jeblair: several of those ^^^, but no "in-use" nodes, which is a good thing so far | 19:00 |
| mordred | double-crap - we have marked all python35 jobs as non-voting. /me digging for a fix ... | 19:00 |
| fungi | jlk: most jobs (really any which don't need sudo) should be doing that so that they can avoid changes introducing sudo calls | 19:00 |
| clarkb | fungi: that is what we do | 19:00 |
| clarkb | fungi: but set -x drops the ! Itink | 19:00 |
| fungi | clarkb: ahh | 19:00 |
| jlk | yeah just seems weird to be duplicating it EVERYWHERE | 19:00 |
| * clarkb double checks | 19:01 | |
| mordred | jlk: yah - well, in the new world they'll all use the revoke-sudo role | 19:01 |
| fungi | clarkb: but it raises an ERROR with "rc": 1 in ansible? | 19:01 |
| jlk | gotcha | 19:01 |
| clarkb | ya ara confirms | 19:01 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql master: Strip legacy prefix if it's present https://review.openstack.org/508272 | 19:01 |
| mtreinish | andreaf: ^^^ I'm not sure if we wanted that or not | 19:01 |
| *** dprince has joined #openstack-infra | 19:01 | |
| clarkb | fungi: ya because sudo is apparently still working | 19:01 |
| mtreinish | andreaf: but figured I'd throw it up and we can discuss it | 19:01 |
| fungi | clarkb: oh! got it. so legit failure, just confusing debug trace | 19:01 |
| mordred | clarkb: oh - so we're not actually revoking sudo? | 19:01 |
| clarkb | mordred: ya I think we are removing the file properly but then sudo must still work | 19:02 |
| jlk | is there more than one place our account is getting sudo from? | 19:02 |
| clarkb | is that task run as become root? | 19:03 |
| clarkb | does't look like it | 19:04 |
| *** slaweq_ has joined #openstack-infra | 19:05 | |
| jlk | so what I had proposed was a follow up task that WAS using become | 19:06 |
| jlk | and then inverting the fail state | 19:06 |
| jlk | so if the task works, it's a failure. If it fails, it's success | 19:06 |
| clarkb | jlk: that is essentially what the task already does though | 19:06 |
| clarkb | remove sudo powers, then use sudo | 19:06 |
| jlk | That proposed change was failing in CI, I thought there was something wrong, turns out it might be something else. | 19:06 |
| clarkb | let me hop on a node and look at the fs | 19:06 |
| jlk | clarkb: yeah, but it lets Ansible handle the error state, rather than trying to bodge it into the shell module | 19:06 |
| clarkb | jlk: well its a legacy job... that is kind of the point :) | 19:07 |
| *** bnemec has quit IRC | 19:07 | |
| dirk | hwoarang: updated, sorry forgot about it | 19:08 |
| jlk | yeah I think my change was on the role itself | 19:08 |
| jlk | for future use | 19:08 |
| clarkb | the file is /etc/sudoers.d/zuul | 19:09 |
| clarkb | rm -rf doesn't fail due to the -f | 19:10 |
| mordred | clarkb: so we're removing the wrong file | 19:10 |
| clarkb | then !sudo -n true fails because we didn't actually remove anything | 19:10 |
| mordred | clarkb: in two different places | 19:10 |
| clarkb | mordred: ya | 19:10 |
| fungi | ohh... | 19:10 |
| *** spzala has quit IRC | 19:10 | |
| mordred | I checked the new revoke-sudo role when I verified... | 19:10 |
| fungi | they must both be wrong in that case | 19:10 |
| hwoarang | dirk: no problem thank you | 19:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove the right file in revoke-sudo https://review.openstack.org/508274 | 19:11 |
| fungi | yep | 19:12 |
| mordred | clarkb: that'll take careof the role | 19:12 |
| fungi | clarkb: mordred: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker/install.d/60-zuul-worker#n15 | 19:12 |
| fungi | confirmed | 19:12 |
| fungi | in dib we do /etc/sudoers.d/zuul | 19:12 |
| mordred | btw - I'm currently tracking down a bug in the migration of python35 jobs | 19:13 |
| jlk | oh ghanks gerrit web | 19:13 |
| *** bnemec has joined #openstack-infra | 19:13 | |
| jlk | if your browser isn't wide enough, it just flat out hides the voting buttons | 19:13 |
| clarkb | anyone working on a patch to update the filepath yet? | 19:15 |
| fungi | i hadn't started but can | 19:15 |
| clarkb | fungi: go for it | 19:15 |
| logan- | with the new v3 cloner setup, where's the definition of which repos should be present for a job stored? | 19:15 |
| fungi | didn't want to duplicate effort | 19:15 |
| fungi | on it now | 19:15 |
| *** spzala has joined #openstack-infra | 19:15 | |
| mwhahaha | so looks like the tripleo jobs just got switched and we're failing on gate_hook http://logs.openstack.org/89/507889/2/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/a8829f3/job-output.txt.gz#_2017-09-28_19_06_33_406425 | 19:16 |
| clarkb | logan-: in the job itself as required repos, The repo the job runs against is implied | 19:16 |
| mwhahaha | is this a known thing? | 19:16 |
| *** bnemec has quit IRC | 19:16 | |
| * mwhahaha tries to figureout job-output.txt | 19:16 | |
| jlk | logan-: one sec let me get you a doc link | 19:16 |
| logan- | clarkb: thanks, i'm talking for the openstack-ansible integrated test, we have a bunch of repos present in /root/src/git.openstack.org/openstack/*. one is missing -- http://logs.openstack.org/56/507956/11/gate/legacy-openstack-ansible-openstack-ansible-aio/0dce683/job-output.txt.gz#_2017-09-28_19_05_22_586881 | 19:16 |
| clarkb | mwhahaha: http://logs.openstack.org/89/507889/2/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/a8829f3/job-output.txt.gz#_2017-09-28_19_05_11_231883 that is where the gate hook is exported | 19:16 |
| *** spzala_ has joined #openstack-infra | 19:17 | |
| clarkb | mwhahaha: I'm wondering if maybe that happened in another shell so isn't present when declared? | 19:17 |
| logan- | openstack/ansible-hardening is missing | 19:17 |
| *** e0ne has joined #openstack-infra | 19:17 | |
| logan- | thanks jlk | 19:17 |
| jeblair | back and catching up | 19:17 |
| *** spzala__ has joined #openstack-infra | 19:18 | |
| jlk | logan-: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.required-projects | 19:18 |
| logan- | thanks jlk | 19:18 |
| mwhahaha | clarkb: we used to see this occasionally on zuulv2 but this just failed on all the jobs example https://review.openstack.org/#/c/507889/ | 19:18 |
| *** spzala has quit IRC | 19:20 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/openstack-zuul-jobs master: Correctly revoke /etc/sudoers.d/zuul https://review.openstack.org/508276 | 19:20 |
| fungi | clarkb: mordred: jlk: ^ | 19:21 |
| *** Sukhdev has joined #openstack-infra | 19:21 | |
| *** rbrndt has joined #openstack-infra | 19:21 | |
| *** hemna_ has quit IRC | 19:21 | |
| clarkb | fungi: we also need to update the native thing for it too | 19:21 |
| *** spzala__ has quit IRC | 19:21 | |
| clarkb | but we can do that in separate changes | 19:21 |
| fungi | clarkb: 507943 does that? | 19:21 |
| *** spzala_ has quit IRC | 19:22 | |
| mordred | clarkb: already did https://review.openstack.org/508274 | 19:22 |
| clarkb | oh | 19:22 |
| fungi | no wait, that one | 19:22 |
| fungi | 508274 | 19:22 |
| clarkb | sorry missed it | 19:22 |
| jlk | spot checked, looks good | 19:22 |
| fungi | though it missed http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/revoke-sudo/README.rst#n3 | 19:22 |
| fungi | i'll add a followup | 19:22 |
| clarkb | fungi: its also not working | 19:23 |
| clarkb | fungi: can you push a new patchset to invert the check too when you update the readme? | 19:23 |
| fungi | yeah, will do | 19:23 |
| clarkb | its missing the ! I think | 19:23 |
| fungi | it has a ! in the command string there | 19:24 |
| fungi | http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/revoke-sudo/tasks/main.yaml#n8 | 19:24 |
| fungi | does it need to be \ escaped? | 19:24 |
| mordred | well - somehow our running of that has not been working | 19:24 |
| mordred | you know - ! is a shell thing - does that need to be shell? | 19:25 |
| fungi | yeah | 19:25 |
| mordred | rather than command? | 19:25 |
| clarkb | oh yup | 19:25 |
| fungi | so just s/command/shell/? | 19:25 |
| fungi | i'll switch taht too | 19:25 |
| clarkb | also http://logs.openstack.org/76/508276/1/check/tox-linters/b5381c9/job-output.txt.gz#_2017-09-28_19_23_32_729071 is going to cause fungi's change to fail | 19:25 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: Remove the right file in revoke-sudo https://review.openstack.org/508274 | 19:25 |
| fungi | updated ^ | 19:25 |
| *** spzala_ has joined #openstack-infra | 19:25 | |
| *** nikhil has joined #openstack-infra | 19:27 | |
| clarkb | you might also need to single quote the string because ! is special in yaml? | 19:27 |
| *** spzala has joined #openstack-infra | 19:27 | |
| clarkb | its self testing so should know soon enough | 19:28 |
| clarkb | then jlk can follow up later with a cleaner ansibly fix | 19:28 |
| clarkb | mordred: did we accidentally force merge something in ozj that is causing 508276 to fail? | 19:29 |
| mordred | clarkb: might have- we didn't get linters runs on everything before force-merge | 19:30 |
| *** spzala_ has quit IRC | 19:30 | |
| mordred | wow. what a spectacularly useful error message | 19:30 |
| jeblair | normally we'd get a filename with the error, but this error *broke ansible-lint* | 19:30 |
| fungi | boggling for sure | 19:30 |
| fungi | someone have time to bisect the last few commits? | 19:31 |
| SpamapS | jeblair: so is zk<->zuul happier now? | 19:31 |
| mordred | yah. doing it | 19:31 |
| jeblair | SpamapS: i'll check | 19:31 |
| *** pcaruana has quit IRC | 19:31 | |
| jeblair | 2017-09-28 18:47:05,017 DEBUG zuul.zk.ZooKeeper: ZooKeeper connection: CONNECTED | 19:31 |
| jeblair | nothing since then, which was the restart | 19:31 |
| SpamapS | great success | 19:32 |
| clarkb | http://logs.openstack.org/20/483020/10/check/legacy-tempest-dsvm-neutron-dvr-ha-multinode-full/36a9c26/logs/devstack-gate-setup-workspace-new.txt is an interesting failure | 19:32 |
| *** srobert_ has quit IRC | 19:32 | |
| clarkb | something is updating the git repo at the same time we are chowning it? | 19:32 |
| fungi | somehow we hit a retry_limit on tox-linters for 508274,2 (the zuul-jobs revoke-sudo fix) | 19:32 |
| fungi | also for build-openstack-sphinx-docs | 19:32 |
| clarkb | fungi: its still made about the sudo -n true check | 19:33 |
| clarkb | I think sh is default shell which may not grok ! ? | 19:33 |
| fungi | okay, so we wrap in quotes and go for another round | 19:33 |
| clarkb | also ara says the command was lacking ! | 19:33 |
| *** srobert has joined #openstack-infra | 19:33 | |
| clarkb | so I think signle quotes and possibly force bash if single quotes isn't enough | 19:34 |
| jlk | see, this, this is why I wanted to let Ansible handle the error state :) | 19:34 |
| fungi | ! has been posix forever afaik | 19:34 |
| openstack | fungi: Error: "has" is not a valid command. | 19:34 |
| mordred | ansible-lint does not understand the shell: cmd: form | 19:34 |
| Shrews | SpamapS: jeblair: yeah, nodepool logs don't show us deleting any abandoned nodes, so ++ | 19:35 |
| *** bnemec has joined #openstack-infra | 19:35 | |
| mordred | how did this work on the initial import? | 19:35 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: Remove the right file in revoke-sudo https://review.openstack.org/508274 | 19:36 |
| fungi | clarkb: ^ now with 'quoting' | 19:36 |
| *** spzala has quit IRC | 19:36 | |
| jeblair | mordred: aren't we not linting the legacy playbooks? | 19:36 |
| jeblair | -not -path './legacy/*' | 19:36 |
| Shrews | citycloud-kna1 seems to be a problematic provider | 19:36 |
| mordred | OHHHHH - I now undertand the issue | 19:36 |
| mordred | yup. fix coming- thank you | 19:36 |
| jeblair | mordred: though if it really is the cmd syntax, i think we need to allow that. | 19:37 |
| mordred | yah. I think we need to turn off ansile-lint completely - we actually need the cmd syntax in the place it is | 19:37 |
| mordred | we can turn it back on if/when we fix it | 19:37 |
| jeblair | k. anyone want to take on an ansible-list upstream bug ^? | 19:38 |
| jeblair | grr ansible-lint | 19:38 |
| jeblair | (dvorak makes that typo way too easy) | 19:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Stop running ansible-lint on playbooks https://review.openstack.org/508280 | 19:39 |
| openstackgerrit | Logan V proposed openstack-infra/openstack-zuul-jobs master: Add openstack-ansible roles to required projects https://review.openstack.org/508281 | 19:39 |
| clarkb | I'm not sure I understand the ansible-lint situation | 19:39 |
| *** e0ne has quit IRC | 19:40 | |
| mnaser | logan- i have a suggestion about your patch there ^ | 19:40 |
| jlk | it doesn't like to see "cmd" as an argument to the "shell" task | 19:40 |
| jlk | I can peek at that | 19:40 |
| mnaser | i would introduce a legacy-openstack-ansible job, add `required-projects` to it and change parent to all openstack-ansible jobs to legacy-openstack-ansible | 19:40 |
| clarkb | ok we keep syntax check which si good but drop linting for that reason, got it | 19:40 |
| mnaser | you can look at how we did it for puppet jobs (legacy-puppet-openstack-integration) | 19:40 |
| logan- | mnaser: great mnaser thanks | 19:41 |
| *** rhallisey has quit IRC | 19:41 | |
| jeblair | mordred: (we didn't end up with any legacy-ansible parent jobs as part of migration, right?) | 19:41 |
| mnaser | (i have some patches refactoring some of the puppet jobs which might be helpful to look at but i dont want to add more confusing patches as people are busy) | 19:42 |
| clarkb | fungi: after quoting 274 fails on the linting problem | 19:42 |
| clarkb | so once linting is fixed we can then fix sudo | 19:42 |
| mordred | jeblair: that's correct | 19:42 |
| clarkb | then things should be much happier | 19:42 |
| fungi | we have a second issue with ansible-lint it seems... | 19:42 |
| fungi | http://logs.openstack.org/74/508274/3/check/tox-linters/8106f37/job-output.txt.gz#_2017-09-28_19_38_24_532129 | 19:42 |
| fungi | ANSIBLE0013 Use shell only when shell functionality is required | 19:42 |
| clarkb | lol | 19:43 |
| fungi | how does it determine that shell functionality is not required? | 19:43 |
| jeblair | If you liked pep8 and hacking, just wait until you see the all new Ansible Lint! | 19:44 |
| jlk | poorly | 19:44 |
| jlk | I think it looks for redirection and pipes | 19:44 |
| fungi | and misses ! | 19:44 |
| SamYaple | and globbing | 19:44 |
| *** e0ne has joined #openstack-infra | 19:44 | |
| jaypipes | jlk: I'm here, I'm here! | 19:44 |
| fungi | hah | 19:44 |
| jeblair | it should check "[^a-z]" | 19:44 |
| mordred | jaypipes: wins | 19:44 |
| jaypipes | :) | 19:44 |
| jlk | hahaha | 19:44 |
| fungi | should i exclude ANSIBLE0013 in tox.ini? or just wait for the ansible-lint disablement? | 19:45 |
| clarkb | I think juts wait for disablement and add that ^ to the list of things we should work to get fixed upstream | 19:46 |
| clarkb | though fixing that heuristic seems like it will be painful | 19:46 |
| dmsimard | I thought we retrofitted the nodepool node files ? did we not ? cat: /etc/nodepool/primary_node_private: No such file or directory | 19:46 |
| jeblair | oh we'll need a disablement change on zuul-jobs | 19:46 |
| jeblair | the disable was only on ozj | 19:46 |
| *** anticw has joined #openstack-infra | 19:47 | |
| jeblair | it looks like a speculative reconfiguration takes about 25 seconds (now that we have all those project-pipeline things). of course we run it twice (oy!), so a .zuul.yaml change will pause zuul for about 50 seconds. it's workable, but not great. now we know where to look, we can try improving that. later. | 19:47 |
| jlk | well this is fun, I'm trying to find where in the ansible code it actually parses the "cmd" argument to comman/dshell | 19:48 |
| fungi | shall i replicate 508280 for zuul-jobs too then? or is someone already working on that? | 19:48 |
| jeblair | jlk: maybe https://github.com/ansible/ansible/issues/12856 will help | 19:48 |
| jeblair | jlk: that's what pointed me at it | 19:49 |
| jlk | kk, reading | 19:49 |
| jeblair | jlk: in particular: https://github.com/ansible/ansible/issues/12856#issuecomment-150002983 maybe? | 19:49 |
| clarkb | jeblair: we run it twice because check and gate? | 19:49 |
| jlk | oooooh | 19:49 |
| jlk | because it's stuffed out there | 19:49 |
| jlk | okay | 19:49 |
| jeblair | clarkb: nope, including trusted and untrusted changes. | 19:50 |
| jlk | oh so we could switch from 'cmd' to _raw_params: | 19:50 |
| jeblair | jlk: yuck? | 19:50 |
| jlk | well, we're depending on a legacy undocumented thing, that's getting transformed outside of hte module... | 19:50 |
| *** `Kevin has joined #openstack-infra | 19:51 | |
| jeblair | jlk: _raw_params is "more supported"? | 19:51 |
| jlk | one sec | 19:51 |
| jeblair | i picked the thing that looked "most supportable and normal" to my unwashed eyes | 19:51 |
| jlk | it's not documented, but at least it's listed in the module argument spec | 19:51 |
| jlk | https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/commands/command.py#L123 | 19:51 |
| jlk | whether or not ansible-lint will take it, dunno. need to test more | 19:52 |
| clarkb | https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/commands/shell.py#L160 | 19:52 |
| jeblair | jlk: https://github.com/ansible/ansible/issues/12856#issuecomment-150003442 also made me think it was the right approach | 19:52 |
| clarkb | its documented there | 19:52 |
| clarkb | oh thats a return value | 19:52 |
| jlk | yeah | 19:52 |
| jlk | not an input | 19:52 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: Remove the right file in revoke-sudo https://review.openstack.org/508274 | 19:53 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: Stop running ansible-lint on playbooks https://review.openstack.org/508285 | 19:53 |
| fungi | added and restacked ^ (hopefully they don't need to be squashed?) | 19:53 |
| jeblair | clarkb: (we perform a speculative reconfiguration that includes trusted repo changes, even on untrusted changes, to detect syntax errors. then we do it again only including untrusted repo changes and use that.) | 19:53 |
| clarkb | so the upstream docs say just use shell: | | 19:53 |
| jlk | fair enough | 19:53 |
| *** hashar has quit IRC | 19:53 | |
| mordred | clarkb: yah - you just can't use shell: | | 19:53 |
| mordred | clarkb: if you have here docs | 19:53 |
| jeblair | clarkb: yep, which will get you an invalid bash script. it will have a leading space on the start of every line. | 19:53 |
| *** srobert has quit IRC | 19:53 | |
| clarkb | wow | 19:54 |
| jeblair | yeah, they broke yaml folding | 19:54 |
| clarkb | thats interesting | 19:54 |
| jeblair | and this is the solution they recommend in the bug report | 19:54 |
| *** srobert has joined #openstack-infra | 19:54 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Stop running ansible-lint on playbooks https://review.openstack.org/508280 | 19:56 |
| * clarkb is going to recheck the legacy sudo fix now | 19:58 | |
| jeblair | did it | 19:58 |
| clarkb | oh jeblair wins | 19:58 |
| fungi | yeah, it just dawned on me i should have reparented it onto mordred's ansible-lint disablement | 19:59 |
| clarkb | re reconfig time, I wonder if zuul can memoize stuff? | 19:59 |
| *** ccamacho has quit IRC | 19:59 | |
| clarkb | basically keep unchanged and parsed configs laying around and only update them if/when they change | 19:59 |
| dmsimard | clarkb: fyi I'm hunting down an issue with all tripleo jobs. So far my understanding is that the (nested) ansible from devstack-gate does not inherit from a function being declared in the parent task | 19:59 |
| dmsimard | This is exported correctly: https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/playbooks/legacy/tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/run.yaml#L93 | 19:59 |
| dmsimard | But in the logs: http://logs.openstack.org/89/507889/2/gate/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/efd5b46/ara/result/b67d1480-0bef-460c-b229-462692e55528/ | 19:59 |
| dmsimard | /home/zuul/workspace/devstack-gate/functions.sh: line 180: declare: gate_hook: not found | 20:00 |
| clarkb | dmsimard: I wonder if that is a shell behavior | 20:00 |
| clarkb | dmsimard: is that possibly a sh vs bash problem? | 20:00 |
| clarkb | fungi: ^ | 20:00 |
| dmsimard | clarkb: yes, that's what I am trying to reproduce locally | 20:00 |
| clarkb | eg will export -f work in dash ash sh ? | 20:00 |
| jeblair | we use bash | 20:00 |
| *** kgiusti has quit IRC | 20:00 | |
| clarkb | jeblair: oh do we force bash as the shell in ansible? | 20:00 |
| jeblair | https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/playbooks/legacy/tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/run.yaml#L96 | 20:00 |
| clarkb | (default is sh iirc) | 20:00 |
| jeblair | clarkb: ^ | 20:01 |
| *** ijw has joined #openstack-infra | 20:01 | |
| clarkb | cool so that probably isn't the problem then | 20:01 |
| fungi | though that aside, export should work the same in classic bourne shell too afaik | 20:01 |
| dmsimard | clarkb: but we don't enforce /bin/bash on the shell/command tasks from devstack-gate though | 20:01 |
| dmsimard | so I wonder if that could be it | 20:01 |
| clarkb | dmsimard: thta is orthogonal I think | 20:01 |
| jeblair | cat: /etc/nodepool/primary_node_private: No such file or directory | 20:01 |
| clarkb | dmsimard: only the d-g shell should interpret that hook and not ansible | 20:02 |
| dmsimard | it's possible the error is completely outside of the devstack-gate ansible, too. The error is from functions.sh | 20:02 |
| jeblair | dmsimard: that's a little further up ^ | 20:02 |
| dmsimard | jeblair: right, I asked about that earlier but no one answered | 20:02 |
| dmsimard | jeblair: it doesn't seem fatal in any case | 20:02 |
| dmsimard | 15:46:18<dmsimard>I thought we retrofitted the nodepool node files ? did we not ? cat: /etc/nodepool/primary_node_private: No such file or directory | 20:02 |
| clarkb | dmsimard: that may indicate the job doesn't have a multinode nodeset | 20:02 |
| clarkb | which is/was a bug in the migration | 20:02 |
| fungi | got patched a while back, right? maybe we missed some additional jobs with that problem? | 20:03 |
| dmsimard | hmm, you're correct | 20:03 |
| *** chlong has quit IRC | 20:03 | |
| *** e0ne has quit IRC | 20:03 | |
| dmsimard | this is a multinode job but there is only one centos-7 node, at least according to ara | 20:03 |
| clarkb | why don't we start by fixing that and see if that knocks the rest of it loose | 20:03 |
| dmsimard | the job is here: https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/zuul-legacy-jobs.yaml#L16755 | 20:04 |
| dmsimard | nodeset: centos-7 | 20:04 |
| jlk | oh this is fun | 20:04 |
| jlk | ansible-lint reads these things before task.py does the transformation | 20:04 |
| jeblair | dmsimard, clarkb: that error is really weird -- it's from this line: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n180 | 20:05 |
| jeblair | i agree, let's fix the nodeset first. but the cause chain is definitely not immediately apparent. :) | 20:06 |
| clarkb | dmsimard: do you want to push the change up for the new nodeset? | 20:06 |
| clarkb | dmsimard: you might want to spot check other tripleo multinode jobs | 20:06 |
| dmsimard | clarkb: trying to figure out what it was before and check for other jobs | 20:06 |
| dmsimard | yeah. | 20:06 |
| dmsimard | looks like it's coming from here: https://github.com/openstack-infra/project-config/blob/fa973dd86f000100c231cbacf0d28700fca8f735/jenkins/jobs/projects.yaml#L13035-L13044 | 20:07 |
| dmsimard | that's clearly centos-7-2-node | 20:07 |
| dmsimard | eh | 20:07 |
| dmsimard | ok, let me prep a patch | 20:07 |
| dmsimard | mwhahaha: ^ | 20:07 |
| fungi | i could use an extra pair of eyes finding what broke the tests.unit.test_v3.TestAnsible.test_playbook test in http://logs.openstack.org/74/508274/4/check/tox-py35-on-zuul/79cfcb6/ (still trying to get the revoke-sudo fix to pass) | 20:07 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Add missing required-project to Puppet https://review.openstack.org/508288 | 20:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Fix bad python35 jobs migration https://review.openstack.org/508289 | 20:08 |
| mordred | infra-root: ^^ that should fix the bogus transition to legacy-python35 | 20:08 |
| mnaser | fungi .. i mean i find this in the errors .. not sure if you spotted it .. 2017-09-28 20:01:10.484501 | ubuntu-xenial | ERROR: Unexpected templating type error occurred on (zuul.project.name == '{{ (zuul | zuul_legacy_vars).ZUUL_PROJECT }}'): int() argument must be a string, a bytes-like object or a number, not 'NoneType' | 20:09 |
| mnaser | im not sure how your change affects it though | 20:09 |
| mordred | gah. | 20:09 |
| *** GregHouse has quit IRC | 20:09 | |
| clarkb | fungi: I was just looking at that, I think its the check-vars test which may be conflicting with mordred's BUILD_TIMEOUT change | 20:09 |
| clarkb | oh mnaser is master detective | 20:10 |
| fungi | i bet that was also introduced by something we force-merged | 20:10 |
| fungi | thanks mnaser! | 20:10 |
| mnaser | i can only spot errors though i dont know what it means :D | 20:10 |
| jeblair | every now and then it's nice to remind ourselves that *every* time we force merge something, we screw it up. :) | 20:10 |
| fungi | YEP! | 20:10 |
| *** weshay has quit IRC | 20:10 | |
| clarkb | jeblair: ++ | 20:10 |
| clarkb | we are like 2/4 today | 20:11 |
| mordred | jeblair: yup. there's a reason we like this gating thing :) | 20:11 |
| clarkb | not a good record | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only add BUILD_TIMEOUT if there is a timeout https://review.openstack.org/508290 | 20:11 |
| *** weshay has joined #openstack-infra | 20:11 | |
| mordred | that should fix mnaser's issue | 20:11 |
| mnaser | which is really fungi's issue :-p | 20:11 |
| SpamapS | mordred: OR, gating has made us soft. | 20:11 |
| clarkb | SpamapS: that seems like the kind of attitude people that directly edit production databases would have. Oh wait we do that too :) | 20:12 |
| mordred | SpamapS: nah. I was soft before gating | 20:12 |
| fungi | SpamapS: i think you're onto something | 20:12 |
| openstackgerrit | Logan V proposed openstack-infra/openstack-zuul-jobs master: Add openstack-ansible required-projects parent job https://review.openstack.org/508281 | 20:12 |
| * SpamapS likes being soft | 20:12 | |
| jeblair | fungi: want to stack your change on 290 ? | 20:12 |
| *** slaweq_ has quit IRC | 20:12 | |
| fungi | jeblair: yep, already in progress | 20:12 |
| mordred | jeblair, fungi: it's a patch to the built-in filters- we need to land the zuul patch and get it rolled out for it to have an effect | 20:13 |
| *** slaweq_ has joined #openstack-infra | 20:13 | |
| *** slaweq_ has quit IRC | 20:13 | |
| fungi | are you sure? | 20:13 |
| fungi | i can't just depends-on that one for the functional job? | 20:14 |
| clarkb | I think the test is likely to pass stacked | 20:14 |
| clarkb | but we won't notice it in production until what mordred says should happen happens | 20:14 |
| *** slaweq_ has joined #openstack-infra | 20:14 | |
| mordred | fungi: oh- yah - for the functional job that should work | 20:14 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Correctly revoke /etc/sudoers.d/zuul https://review.openstack.org/508276 | 20:14 |
| clarkb | ok ^ there is half the sudo problem | 20:14 |
| clarkb | fixed | 20:14 |
| mordred | \o/ | 20:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: Stop running ansible-lint on playbooks https://review.openstack.org/508285 | 20:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: Remove the right file in revoke-sudo https://review.openstack.org/508274 | 20:15 |
| fungi | other half now with depends-on ^ | 20:15 |
| *** GregHouse has joined #openstack-infra | 20:16 | |
| mnaser | with how stuff is cloned in zuulv3, we technically don't have a remote in repos, correct? | 20:17 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Use proper nodesets for the different multinode TripleO jobs https://review.openstack.org/508292 | 20:17 |
| dmsimard | Are we using a topic for migration mistakes ? | 20:17 |
| fungi | dmsimard: topic:zuulv3 | 20:17 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Use proper nodesets for the different multinode TripleO jobs https://review.openstack.org/508292 | 20:17 |
| dmsimard | fungi: ok | 20:17 |
| mnaser | in the puppet openstack gates, we run `git reset --hard origin/master` .. its failing because of 'fatal: ambiguous argument 'origin/master': unknown revision or path not in the working tree.' ... should we just do a git reset --hard master instead? | 20:18 |
| dmsimard | mwhahaha, clarkb, jeblair: I think that's all of the nodeset fix for tripleo: https://review.openstack.org/#/c/508292/ | 20:18 |
| mnaser | (the reason we do that is to pin to specific tempest versions if need-be) | 20:18 |
| mordred | dmsimard: +Ad | 20:18 |
| clarkb | mnaser: yes, I think that would be how you accomplish that, however you might checkout tags instead if you are going for a specific tempest version | 20:18 |
| *** spzala has joined #openstack-infra | 20:19 | |
| jeblair | mnaser: can you point me at the code in question? | 20:19 |
| mnaser | clarkb yep, agreed, but dont want to change too much existing behaviour | 20:20 |
| fungi | mordred: pep8 failure on the zuul patch | 20:20 |
| clarkb | infra-root can we get anothe rreview on https://review.openstack.org/#/c/508289/1 to fix py35 jobs | 20:20 |
| mnaser | jeblair https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L109-L115 -- and can see it in action -- http://logs.openstack.org/17/508217/1/check/legacy-puppet-openstack-integration-4-scenario001-tempest-centos-7/05a6d7e/job-output.txt.gz#_2017-09-28_18_59_26_883028 | 20:20 |
| jeblair | clarkb: depends on abandoned change? | 20:20 |
| clarkb | so it does and gerrit doesn't actually complain, wow | 20:21 |
| clarkb | mordred: ^ do you want to rebase that change? | 20:21 |
| jeblair | i'm going to award this point to gertty :) | 20:21 |
| clarkb | and -10 points from house gerrit web ui | 20:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only add BUILD_TIMEOUT if there is a timeout https://review.openstack.org/508290 | 20:22 |
| mordred | clarkb: yah- on it | 20:22 |
| jeblair | otherwise lgtm | 20:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Fix bad python35 jobs migration https://review.openstack.org/508289 | 20:23 |
| *** jcoufal has quit IRC | 20:24 | |
| * clarkb found another multinode case of bad node label, will push patch in a moment | 20:24 | |
| jeblair | mnaser: under what circumstances would tempest not be master (ie, what situation is that designed to correct?) | 20:24 |
| mnaser | jeblair i believe there are scenarios where some downstream users would pin to specific tempest releases (example: rdo - https://github.com/rdo-infra/ansible-role-weirdo-puppet-openstack/blob/master/tasks/main.yml#L87) | 20:25 |
| mnaser | or sometimes i assume tempest breaks our gate so we can pin to an older release till its fixed then go back to master | 20:25 |
| *** thorst has quit IRC | 20:26 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Set cinder mn-sub jobs to multinode nodeset https://review.openstack.org/508293 | 20:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix bad python35 jobs migration https://review.openstack.org/508289 | 20:26 |
| mordred | \o/ | 20:26 |
| clarkb | mordred: ^ there is the fix for multinode jobs with wrong label I found | 20:26 |
| jeblair | mnaser: okay, then yeah, i think just using 'master' rather than orgin/master like you and clarkb worked out is closest. note that may have patches which haven't landed yet in master (zuul lives in the future), which is likely a behavior change, but probably not one that will affect you. | 20:27 |
| *** eharney has quit IRC | 20:28 | |
| mordred | mnaser, jeblair: or potentially even add in logic to not do the reset if that parameter isn't given - don't know if that would be hard to plumb through or not | 20:28 |
| jeblair | mnaser: there are a bunch of tools that make this sort of thing really nice in v3, so when we get to redesigning these jobs, we should be able to express all of that in zuul config, and not need git commands like that in the jobs themselves. | 20:29 |
| clarkb | http://logs.openstack.org/20/483020/10/check/legacy-grenade-dsvm-neutron-multinode/43d6593/job-output.txt.gz#_2017-09-28_20_04_21_024064 we may not properly be detecting multinode for some reason (I checked inventory that job did have two nodes) | 20:29 |
| dmsimard | jeblair, mordred: I missed at the very least one multinode nodeset, would you rather I fix the review that hasn't landed yet or send one on top ? | 20:30 |
| jeblair | dmsimard: new patch | 20:30 |
| jeblair | clarkb: does d-g save the internal inventory? | 20:31 |
| clarkb | jeblair: it relies on the /etc/nodepool/ contents | 20:32 |
| clarkb | and generates its own inventory from that | 20:32 |
| jeblair | clarkb: right, but it writes an inventory file based on that | 20:32 |
| jeblair | wondered if it saved that file to logs | 20:32 |
| clarkb | I think it may be recorded where we write to that file, grenade is also weird in how it does multinode | 20:32 |
| mnaser | mordred, jeblair: ok, cool thanks :) | 20:32 |
| clarkb | since grenade runs d-g | 20:32 |
| clarkb | iirc | 20:32 |
| clarkb | ok I've got to do my lunch now | 20:33 |
| clarkb | back in a bit | 20:33 |
| mordred | http://logs.openstack.org/20/483020/10/check/legacy-grenade-dsvm-neutron-multinode/43d6593/logs/etc/ <-- would have been nice to have collected /etc/nodepool :) | 20:33 |
| fungi | mordred: your zuul patch is still failing on that same test in the tox-py35 job | 20:33 |
| jeblair | mordred: oh i think timeout is always there, just sometimes null | 20:34 |
| fungi | so, yeah, that patch is a no-op as is | 20:34 |
| *** florianf has quit IRC | 20:34 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Parent legacy jobs with artifacts legacy-publish-openstack-artifacts https://review.openstack.org/508265 | 20:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only add BUILD_TIMEOUT if there is a timeout https://review.openstack.org/508290 | 20:35 |
| mordred | jeblair: how about that ^^ | 20:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Add howto section on migrating legacy jobs to v3 https://review.openstack.org/508295 | 20:36 |
| fungi | wfm. could also have done... if zuul.get('timeout'): | 20:37 |
| mordred | fungi: yah | 20:37 |
| fungi | i guess someone _could_ set timeout to 0, but if they did i don't know what sort of behavior they might expect from doing such a thing | 20:39 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use proper nodesets for the different multinode TripleO jobs https://review.openstack.org/508292 | 20:39 |
| mnaser | mordred fyi, the docs still says nodes: and not nodesets: (migration docs that is) | 20:39 |
| mnaser | not sure if they are interchangable | 20:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Use bash for releasenotes job shell https://review.openstack.org/508297 | 20:40 |
| openstackgerrit | Alex Kavanagh proposed openstack-infra/project-config master: Change the docs job to a deploy-publish-job https://review.openstack.org/508298 | 20:40 |
| mordred | infra-root: http://logs.openstack.org/27/508227/1/check/releasenotes/64b3b7b/job-output.txt.gz#_2017-09-28_18_56_24_817891 is why I did https://review.openstack.org/508297 | 20:40 |
| mordred | mnaser: oh - thank you | 20:40 |
| jeblair | mnaser: thanks, nodes is gone completely | 20:41 |
| jlk | ugh, ansible-lint is broken in numerous places because of cmd: | 20:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Change nodes to nodeset in zuulv3 docs https://review.openstack.org/508300 | 20:41 |
| mordred | mnaser, jeblair: ^^ | 20:41 |
| jeblair | mordred: thx. i have small -1 on parent. | 20:42 |
| mordred | actually - lemme restack that | 20:42 |
| mordred | jeblair: want me to split them - or jst fix the -1 ? | 20:42 |
| jeblair | mordred: switch order i'd say | 20:42 |
| jeblair | we can rush the fix; want others to have time to see the howto | 20:42 |
| *** rcernin has joined #openstack-infra | 20:42 | |
| *** thorst has joined #openstack-infra | 20:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Change nodes to nodeset in zuulv3 docs https://review.openstack.org/508300 | 20:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Add howto section on migrating legacy jobs to v3 https://review.openstack.org/508295 | 20:44 |
| mordred | jeblair: done | 20:44 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Use proper nodesets for the different TripleO OVB jobs https://review.openstack.org/508301 | 20:45 |
| dmsimard | mordred, clarkb, jeblair, mwhahaha, EmilienM ^ | 20:45 |
| dmsimard | I think that's all of them | 20:45 |
| EmilienM | dmsimard: ok looking in a few | 20:45 |
| EmilienM | dmsimard: thanks, you got them all i think | 20:46 |
| ianw | mktemp: failed to create directory via template '/opt/dib_cache/dib_build.XXXXXXXX': Permission denied <- wouldn't have expected that | 20:46 |
| mordred | ianw: wow. that's fun | 20:46 |
| dmsimard | EmilienM: I checked the templates one by one from https://github.com/openstack-infra/project-config/blob/fa973dd86f000100c231cbacf0d28700fca8f735/jenkins/jobs/projects.yaml#L12823 | 20:47 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Set cinder mn-sub jobs to multinode nodeset https://review.openstack.org/508293 | 20:47 |
| ianw | is the path forward fixing up odd bits in legacy jobs as they come up? | 20:47 |
| ianw | (sorry, just catching up after waking up to a zuulv3 world :) | 20:47 |
| jeblair | ianw: yep; so far we haven't seen anything we think we need to roll back for | 20:48 |
| dmsimard | ianw: that's what things are looking like so far | 20:48 |
| ianw | ++ cool! | 20:48 |
| smcginnis | cowsay "How's zuul3 going?" | 20:49 |
| jeblair | mooooving along | 20:49 |
| smcginnis | :D | 20:49 |
| clarkb | wow | 20:49 |
| clarkb | quote of the day | 20:49 |
| smcginnis | That was pretty awesome jeblair | 20:49 |
| dmsimard | jeblair++ | 20:49 |
| jlk | punchy mode engaged | 20:50 |
| gouthamr | hello good folks, got a patch that's not getting enqueued on the gate ( Link: https://review.openstack.org/#/c/505672/ ) -> related to the ZuulV3 migration? | 20:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Save /etc/nodepool https://review.openstack.org/508302 | 20:52 |
| ianw | http://logs.openstack.org/79/504579/6/gate/legacy-diskimage-builder-pylint/34bf88f/job-output.txt.gz#_2017-09-28_20_39_13_705486 <- Error executing: cp -dRl /home/zuul/src/git.openstack.org/openstack/requirements/. /tmp/tmp.MVGucBqLXP | 20:52 |
| EmilienM | dmsimard: cool | 20:52 |
| jeblair | clarkb, mordred: ^ we need that to run on, i guess a grenade multinode job? | 20:52 |
| ianw | wondering why zuul-cloner decided to get requirements into /tmp there? | 20:52 |
| jeblair | ianw: that one means openstack/requirements is missing from the job's required-projects | 20:53 |
| clarkb | jeblair: ya which I think it will as part of the d-g job list | 20:53 |
| jeblair | clarkb: cool | 20:53 |
| ianw | jeblair: yeah, figured, why does it go into /tmp? | 20:53 |
| jeblair | ianw: presumably the job was written to use zuul-cloner to clone reqs into a tempdir | 20:53 |
| clarkb | 8290 is about to head into the gate queue | 20:54 |
| clarkb | then once that merges we can fix sudo elsewhere | 20:54 |
| jeblair | ianw: of course it doesn't have to anymore, but it doesn't know that, and our shim just replicates the behavior | 20:54 |
| jeblair | ianw: the original reasoning was probably: "we need a copy of openstack/requirements, and jenkins checked our our project into the workspace directory, so where do we put it? <shrug> /tmp" | 20:54 |
| clarkb | is it just me or is zuul feeling slow again, might wantot check the zk things | 20:54 |
| smcginnis | gouthamr: That appears to be the case. I know we have a few. | 20:55 |
| jeblair | clarkb: it just finished a dynamic layout | 20:55 |
| jeblair | and started another | 20:55 |
| smcginnis | gouthamr: Maybe just a way to stress test zuulv3 once everything starts moving. ;) | 20:55 |
| jeblair | zuul still needs to calculate dynamic layouts for changes that go into the merge-check pipeline | 20:56 |
| jeblair | that can be pretty expensive if we end up with, say, a bunch of open changes to zuul-jobs with zuul config changes | 20:57 |
| jeblair | we may want to consider disabling merge-check | 20:57 |
| clarkb | mordred: jeblair concern with using /bin/bash in all the places, on centos it is /usr/bin/bash | 20:57 |
| *** yamahata has joined #openstack-infra | 20:58 | |
| mordred | jesus h ... what? | 20:58 |
| mordred | *headdesk* | 20:58 |
| fungi | hopefully changes to zuul-jobs become much less frequent after the next week or two | 20:58 |
| clarkb | not the end of the world right now but ya annoying little distro madness | 20:58 |
| ianw | clarkb: umm, it's not? | 20:58 |
| jeblair | fungi: we'll have config changes to openstack-zuul-jobs for a while thou | 20:58 |
| mordred | yah. I agree with ianw | 20:58 |
| clarkb | ianw: it is on git.openstack.org at least | 20:58 |
| mordred | [mordred@git01 ~]$ ls /bin/bash | 20:59 |
| mordred | /bin/bash | 20:59 |
| ianw | /bin -> /usr/bin | 20:59 |
| ianw | should be a symlink | 20:59 |
| clarkb | its not a symlink but it is in both places in git.o.o | 20:59 |
| clarkb | weird | 20:59 |
| *** jascott1 has quit IRC | 20:59 | |
| clarkb | oh the dir is a symlink? | 20:59 |
| gouthamr | smcginnis: ah, thanks! no rush getting a merge if it isn't just me! i'll recheck when it's done mooving :) | 20:59 |
| clarkb | wow | 20:59 |
| clarkb | I need more alcohol | 21:00 |
| *** jascott1 has joined #openstack-infra | 21:00 | |
| *** thorst has quit IRC | 21:00 | |
| *** thorst has joined #openstack-infra | 21:00 | |
| ianw | clarkb: yeah. i'm pretty sure systemd is involved in that and the ability to boot with a nfs mounted /usr or something something, i forget | 21:00 |
| jeblair | now i need more alcohol | 21:01 |
| *** trown is now known as trown|outtypewww | 21:01 | |
| fungi | i have a fairly tasty domestic saké chilling for this evening | 21:02 |
| ianw | clarkb: sorry, you said sudo ... struggling to pick it out of scrollback. tl;dr? | 21:02 |
| clarkb | oh bah | 21:02 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Change nodes to nodeset in zuulv3 docs https://review.openstack.org/508300 | 21:02 |
| ianw | i'm guessing that's related to my /opt directory issues | 21:02 |
| clarkb | ianw: the jenkins-sudo file to zuul file transition didn't work out quite right | 21:02 |
| clarkb | and there is a new bug I am seeing in it | 21:02 |
| clarkb | ianw: basically we needed to delete the file giving zuul sudo rights rather than the one giving jenkins sudo rights | 21:02 |
| mordred | jeblair: wasn't there an issue earlier today with infra doc publishing that we fixed? | 21:03 |
| clarkb | because we use the zuul user now | 21:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add reqs to legacy-cinder-pylint https://review.openstack.org/508303 | 21:03 |
| ianw | ahh, ok, thanks | 21:03 |
| clarkb | http://logs.openstack.org/02/508302/1/check/legacy-devstack-gate-tox-run-tests/af6e423/job-output.txt.gz#_2017-09-28_21_00_32_154153 new sudo problem | 21:03 |
| clarkb | mordred: ^ | 21:03 |
| clarkb | we don't have a grep file in place for the zuul user... | 21:03 |
| jeblair | mordred: no. we have 2 known issues -- 1) infra docs index job is post_failing. 2) zuul-jobs isn't updating. | 21:03 |
| mordred | jeblair: those are to legacy-blazar-pylint | 21:03 |
| clarkb | instead of running that sh script we could inline the check I think | 21:03 |
| jeblair | bah | 21:04 |
| clarkb | unless that file is suid | 21:04 |
| *** jascott1 has quit IRC | 21:04 | |
| jeblair | mordred: i wonder if it needs it too | 21:04 |
| *** jascott1 has joined #openstack-infra | 21:04 | |
| jeblair | mordred: maybe i should add to all pylint jobs? | 21:04 |
| mordred | jeblair: oh- you konw what - yah - probably all the pylint jobs do - since they use run-tox.sh | 21:04 |
| jeblair | mordred: ok will do | 21:05 |
| clarkb | its not setuid we have a special sudo rule allowing only that file to be sudo'd | 21:05 |
| *** spzala has quit IRC | 21:05 | |
| smcginnis | Are we going to console output back in the logs? | 21:06 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Add requirements to dib pylint job https://review.openstack.org/508305 | 21:06 |
| *** dhajare has quit IRC | 21:06 | |
| clarkb | mordred: jeblair ^ should I work on a patch to add equivalents for the zuul user to our images? | 21:06 |
| clarkb | smcginnis: yes, but its called job-output.txtnow | 21:06 |
| *** jascott1 has joined #openstack-infra | 21:06 | |
| mordred | clarkb: why are we running jenkins-sudo-grep again? | 21:07 |
| smcginnis | clarkb: Hmm, seems significantly less useful. Unless I've just looked at two bad examples. | 21:07 |
| clarkb | mordred: it checks if the jenkins user attempted to use sudo and if it did fails the job | 21:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add requirements to all pylint jobs https://review.openstack.org/508303 | 21:07 |
| jeblair | ianw: ^ i think i got your thing in that patch | 21:07 |
| clarkb | smcginnis: it should be roughly equivalent especially for legacy jobs | 21:07 |
| ianw | jeblair: cool, will abandon | 21:07 |
| mordred | clarkb: right- but we already have revoke-sudo? | 21:07 |
| clarkb | mordred: ya but sometimes tests pass even when sudo fails | 21:08 |
| mordred | gotch | 21:08 |
| clarkb | mordred: we saw this happen over and over in neutron iirc hence the check | 21:08 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Only add BUILD_TIMEOUT if there is a timeout https://review.openstack.org/508290 | 21:08 |
| fungi | smcginnis: there is additional output from ansible in the console logs now, and i think there are some ideas on how to make some of that expandable when viewed from logs.o.o, perhaps collapsed by default | 21:08 |
| smcginnis | clarkb: Is this a known issue: http://logs.openstack.org/32/501332/3/gate/legacy-cinder-tox-compliance/fb78a70/job-output.txt.gz#_2017-09-28_19_34_55_133430 | 21:08 |
| clarkb | its the anti emissions fool device device | 21:08 |
| clarkb | smcginnis: yes we are working to fix it | 21:08 |
| smcginnis | fungi: OK, I think I'm just seeing a lot of that and things are failing early enough that I'm not seeing what I'm expecting. | 21:09 |
| ianw | clarkb: yes, that is a project-config element iirc | 21:09 |
| smcginnis | I'll just be more patienter. | 21:09 |
| clarkb | mordred: we might be able to dropthe check for now if we want to add it in again once images are updated | 21:09 |
| mordred | clarkb: well - I'm fine updating it - also we have tox base jobs thatdon't use scripts in images that tox jobs should get migrated to | 21:10 |
| mordred | clarkb: either way it's an image change at the moment unless we just tried to update more tox jobs to be based on tox base jobs instead of migrated legacy jobs | 21:11 |
| *** thorst has quit IRC | 21:11 | |
| clarkb | ya, let me get the change up and then we at least have the option of slowly moving towards that | 21:11 |
| mordred | clarkb: I'm going to build a list of migrated jobs that are using run-tox.sh real quick | 21:12 |
| mordred | clarkb: enough for it to still be unworkable for us to do them | 21:13 |
| mordred | clarkb: so yah - let's get the script updated | 21:13 |
| ianw | mordred: legacy-dsvm-nodepool-redhat-src -> "The following LIBS_FROM_GIT were not installed correct: diskimage-builder glean shade" <- is this known or should i debug? | 21:14 |
| tonyb | mordred: Yup totaly agree. I want to test it a little more and then roll out the changes to projects when we switch to the constratints file being on releases.o.o instead of gitweb | 21:15 |
| mordred | clarkb: we have a role we use for some of the proposal jobs that copies the scripts from the job's copy of project-config onto the remote host. we could consider adding a similar thing to legacy-base so that at least we're not looking at image rebuilds | 21:15 |
| clarkb | mordred: the sudo bits are probably the trickiest part but still doable | 21:16 |
| mordred | clarkb: in any case - just a thing to consider -for now I'd do whatever is most expedient | 21:16 |
| mordred | ianw: I have not seen that one | 21:16 |
| mordred | tonyb: oh - heh. SOOOOO | 21:16 |
| ianw | it does not appear to have cloned all the required-projects : http://logs.openstack.org/79/504579/6/gate/legacy-dsvm-nodepool-redhat-src/873f19d/job-output.txt.gz#_2017-09-28_20_43_56_192466 | 21:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Zuul versions of sudo grep checks https://review.openstack.org/508313 | 21:16 |
| clarkb | mordred: ianw ^ there is | 21:17 |
| clarkb | we'll also need to update jobs to use the new script name | 21:17 |
| mordred | tonyb: have you seen the way constraints consumption is working in the v3 jobs? | 21:17 |
| clarkb | http://logs.openstack.org/34/501334/2/gate/legacy-releasenotes/16fc896/job-output.txt.gz#_2017-09-28_21_01_41_756800 reno errors | 21:17 |
| tonyb | mordred: No I have not. | 21:17 |
| mordred | clarkb: how about ... since it's a new script - that we just add writing it to disk on the remote host to the legacy-base pre-playbook | 21:18 |
| tonyb | mordred: am I going to have a sad? | 21:18 |
| clarkb | mordred: we also have to add the sudo rule but that works | 21:18 |
| mordred | tonyb: no - I think you'll have a happy, or at the very worst a meh | 21:18 |
| fungi | tonyb: also remember (i'm missing some context on the start of the conversation of course) that constraints via gitweb or releases.o.o is not something that happens in the ci system, only local dev consumption | 21:18 |
| fungi | we still install constraints from git so we can honor depends-on | 21:18 |
| tonyb | fungi: Yup, I'm only tying them together to minimise the distruption on project teams | 21:19 |
| fungi | awesome | 21:19 |
| mordred | clarkb: mostly just thinking we could get that done and validated in a couple of minutes and not haveto wait - want me to take a stab at converting what you've got there? | 21:19 |
| clarkb | mordred: yes please | 21:19 |
| mordred | tonyb: I've actually got some docs on it open in a vim session right now (although it's down on the stack a few steps) | 21:19 |
| tonyb | mordred: Cool. When they make it to the top can you CC me or point me at the review | 21:20 |
| clarkb | jeblair: can we get review on https://review.openstack.org/#/c/508285/2 that should unstick a portion of the sudo related failures | 21:21 |
| jeblair | clarkb: re +3d | 21:21 |
| jeblair | interesting nodepool happenings | 21:22 |
| jeblair | zuul.zk.LockException: Timeout trying to acquire lock /nodepool/nodes/0000041863/lock | 21:22 |
| * clarkb is going to look at reno now | 21:22 | |
| mnaser | is it normal that a change to .zuul.yaml has been sitting in queue for almost 10 minutes now (but no jobs are appearing under it) | 21:22 |
| mnaser | (i guess that might be related to that ^) | 21:22 |
| jeblair | kazoo.exceptions.LockTimeout: Failed to acquire lock on /nodepool/nodes/0000041863/lock after 30 seconds | 21:22 |
| jeblair | neat, we may have our first node request failure | 21:23 |
| fungi | how quaint | 21:23 |
| * EmilienM likes the new console ( http://zuulv3.openstack.org/static/stream.html?uuid=957e33e5c8154f76b06aeb6a3add77e6&logfile=console.log ) | 21:23 | |
| mnaser | saying this because it might help (or might not, but worth knowing): i have been sending reviews on requests that did not complete all builds (once i identify failure, not to sit and wait for all the rest to fail) .. maybe that has to do with some of these nodepool issues, i dont know | 21:24 |
| fungi | EmilienM: it's just like the old days with jenkins! ;) | 21:24 |
| dhellmann | clarkb : what project was that reno job for? | 21:24 |
| fungi | except far, far better | 21:24 |
| dhellmann | clarkb : the error message makes me think something is telling reno to scan a branch that has been deleted | 21:24 |
| clarkb | dhellmann: cinder | 21:24 |
| jeblair | it's starting to look like every node lock is timing out | 21:25 |
| dhellmann | ah, cinder | 21:25 |
| EmilienM | fungi: and I like old days :D | 21:25 |
| jeblair | Shrews: ping | 21:25 |
| clarkb | dhellmann: we no longer have origin/stable/newton should just be stable/newton I think | 21:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Zuul versions of sudo grep checks https://review.openstack.org/508317 | 21:25 |
| clarkb | dhellmann: but I'm not sure that is the case unless a newton change is under test | 21:25 |
| mordred | clarkb: ^^ | 21:25 |
| jeblair | clarkb: all branches and tags are always present | 21:25 |
| dhellmann | clarkb : ok. reno should only run on master and it assumes it has the *whole* repository available to it. | 21:25 |
| dhellmann | all branches | 21:25 |
| jeblair | (and all branches contain the most future state) | 21:26 |
| Shrews | jeblair: sup? | 21:26 |
| jeblair | Shrews: i'm starting to look into node lock timeouts from zuul | 21:26 |
| Shrews | jeblair: oh. wow. | 21:27 |
| Shrews | gimme a few minutes and i can help | 21:27 |
| clarkb | dhellmann: in that case we likely just have to make it look at not origin/ and instead just the refname? I see it looks at refs/heads which I think should have it? I need to clone a repo and poke around | 21:27 |
| dhellmann | let me look at what cinder has in their files... | 21:27 |
| dhellmann | reno has a bunch of openstack-specific smarts for converting references to branches into other names; it should handle this case | 21:27 |
| dhellmann | oh, bah, they do have origin/ in there | 21:28 |
| clarkb | dhellmann: oh as an explicit thing ? | 21:28 |
| dhellmann | yeah, in releasenotes/source/newton.rst they have origin/stable/newton | 21:28 |
| *** slaweq_ has quit IRC | 21:28 | |
| dhellmann | that's probably what I told them to put in | 21:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Zuul versions of sudo grep checks https://review.openstack.org/508317 | 21:28 |
| dhellmann | why does that ref not exist? | 21:28 |
| mordred | clarkb: sorry - forgot to update the script to call the right thing | 21:28 |
| *** slaweq_ has joined #openstack-infra | 21:29 | |
| *** Goneri has quit IRC | 21:29 | |
| clarkb | dhellmann: because the constructed repo in the job isn't cloned from an origin | 21:29 |
| dhellmann | that's going to make the results of that scan pretty interesting | 21:29 |
| clarkb | dhellmann: its constructed in a more pristin state so it has stable/newton but not origin/stable/newton | 21:29 |
| clarkb | also pristine state is probably the wrong term | 21:29 |
| fungi | rarified? | 21:29 |
| dhellmann | "made up"? :-) | 21:29 |
| dhellmann | ok | 21:29 |
| fungi | rarefied | 21:29 |
| mordred | ah- yah - looking at the reno code - there is definitely a ton of use of origin/ there | 21:30 |
| dhellmann | I think I told teams to use origin/... because that's technically a different branch from stable/... so they would get the same results locally as in the gate | 21:30 |
| dhellmann | now I don't know how to do that. | 21:30 |
| dmsimard | mnaser: btw I'm not sure if yo got an answer re: "empty" queued changes in zuul status page | 21:30 |
| jeblair | Shrews, SpamapS, harlowja: hrm, i still don't have any kazoo log entries | 21:31 |
| mnaser | dmsimard i am gonna guess it has to do with the current nodepool issues (maybe) | 21:31 |
| dhellmann | mordred : I see 4 references and 1 is a comment | 21:31 |
| dmsimard | mnaser: when the changes are empty without jobs on the zuul status page, it's because it's pending initial processing/zuul-merger processing | 21:31 |
| mnaser | oh i see | 21:31 |
| dhellmann | another is trying to strip origin as a prefix | 21:31 |
| mnaser | jobs are going from queued -> node_failure now | 21:31 |
| mnaser | :< | 21:31 |
| fungi | maybe if there is a remote named origin then prepend it to the refs, otherwise use the bare ref (and/or strip a leading origin prefix)? | 21:31 |
| dmsimard | mnaser: node_failure ? where ? | 21:31 |
| dhellmann | another re-adds it in case the shorter name branch doesn't actually exist | 21:32 |
| mnaser | http://zuulv3.openstack.org/ 508296,4 | 21:32 |
| jeblair | yes, almost every node request is failing now | 21:32 |
| dhellmann | fungi : that's what it does. the problem here is that the data file has origin/ in the name of the branch | 21:32 |
| *** thorst has joined #openstack-infra | 21:32 | |
| dmsimard | jeblair: well that's inconvenient :( | 21:32 |
| fungi | dhellmann: i mean if the local git repo has a remote named origin | 21:32 |
| dhellmann | ah, I see | 21:32 |
| fungi | which for someone who got the repo using git clone will likely be the case, but in the ci system won't be the case | 21:33 |
| dmsimard | clarkb: is there a grafana dashboard for nodepool v3 ? | 21:33 |
| dmsimard | http://grafana.openstack.org/dashboard/db/nodepool is empty so I guess it doesn't have v3 data | 21:33 |
| clarkb | dmsimard: no I don't think there is yet | 21:33 |
| dhellmann | I'll have to study this closer, but I think it's trying to strip origin/ in some cases | 21:33 |
| clarkb | mordred mentioned it as something we'll need to update | 21:33 |
| dmsimard | clarkb: do you know if the data is there but we just don't have a graph for it ? | 21:33 |
| dhellmann | I'm not sure why it's not doing it in this particular case | 21:33 |
| clarkb | dmsimard: yes that is the situation aiui, the key names changed slightly | 21:33 |
| Shrews | jeblair: ok, what node are we tracking down? | 21:33 |
| dmsimard | clarkb: ok let me try and see if I can figure that out | 21:34 |
| *** eharney has joined #openstack-infra | 21:34 | |
| mordred | dmsimard, clarkb: we landed a patch earlier to remove the zuulv3-dev stats_prefix - so we should be reporting at least | 21:34 |
| Shrews | oh, node id is above | 21:34 |
| dmsimard | mordred: so do the metrics have the same name as in v2 ? | 21:35 |
| *** vhosakot has quit IRC | 21:35 | |
| Shrews | jeblair: http://paste.openstack.org/show/622206/ | 21:35 |
| mordred | dmsimard: at least somedo - but I haven't gotten to tracking it down fully yet | 21:35 |
| Shrews | which is... interesting | 21:35 |
| *** nikhil has quit IRC | 21:36 | |
| jeblair | http://paste.openstack.org/show/622208/ is our zk log entries | 21:36 |
| *** thorst has quit IRC | 21:36 | |
| mordred | jlk: if you have a sec, could you look at https://review.openstack.org/#/c/508317/ and verify the ansibling? | 21:36 |
| * jlk points his eye holes at it | 21:37 | |
| Shrews | jeblair: http://paste.openstack.org/show/622209/ is more complete and shows missing requests | 21:37 |
| jlk | this doesn't have to live long, right? | 21:40 |
| mnaser | jeblair i dont know much about zk but an interesting thing is it was locked almost an exact 5 minutes after | 21:40 |
| jlk | because.... | 21:40 |
| mnaser | so i dont know if thats some sort of timeout or something | 21:40 |
| Shrews | jeblair: seems that node lost its lock after being assigned, and remained READY, which opened it up for reassignment | 21:40 |
| clarkb | jlk: until we get the images updated and/or completely migration all N jobs using that | 21:40 |
| mordred | jlk: yah - ultimately all the consumers of run-tox want to migrate to using one of hte tox base jobs | 21:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use proper nodesets for the different TripleO OVB jobs https://review.openstack.org/508301 | 21:41 |
| *** chlong has joined #openstack-infra | 21:41 | |
| jeblair | Shrews: neat. zuul never saw the assignment for the first request | 21:41 |
| Shrews | jeblair: then it appears many other weird things happened after that with missing requests | 21:41 |
| jeblair | mnaser: 5m timeout isn't ringing a bell | 21:42 |
| jeblair | zuul's zk connection was flapping around 21:15, so it seems likely the request went missing due to a disconnect | 21:42 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use bash for releasenotes job shell https://review.openstack.org/508297 | 21:42 |
| mordred | dhellmann: related (but not really) - I made a non-legacy releasenotes job (that that ^^ should fix)... | 21:42 |
| mordred | dhellmann: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n270 | 21:43 |
| jeblair | hrm | 21:43 |
| mordred | dhellmann: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/releasenotes/run.yaml | 21:43 |
| jeblair | /etc/zuul/logging.conf was modified 10m after zuul-scheduler was started | 21:43 |
| mordred | jeblair: AROO? | 21:43 |
| jeblair | mordred: any chance that file wasn't updated at last zuul-scheduler restart? | 21:43 |
| mordred | jeblair: I did a puppet run on it before the restart | 21:43 |
| jeblair | weird | 21:43 |
| mordred | jeblair: but - you know - it's certainly possible | 21:44 |
| jeblair | Sep 28 18:57:06 zuulv3 puppet-user[15661]: (/Stage[main]/Zuul::Scheduler/File[/etc/zuul/logging.conf]/content) content changed '{md5}0804d4a680f9d15265e5fc0a3c73c713' to '{md5}5ff922774c890114a6781e538ae789c2' | 21:44 |
| mordred | jeblair: what about the config file? are we possibly also running with the out-of-date timeout settings? | 21:44 |
| jeblair | that's the only log modification i see, and 18:47 is the zuul process time | 21:44 |
| jeblair | good q | 21:44 |
| jeblair | mordred: same thing | 21:45 |
| jeblair | so good news! | 21:45 |
| mordred | aha! so we can still try our theoretical fix | 21:45 |
| jeblair | ya. also, apparently the problem took 2.5 hours to show up this time | 21:45 |
| jeblair | okay, i'm going to restart zuul now; anything i should wait for? | 21:46 |
| mordred | well - that's better than before | 21:46 |
| mordred | jeblair: nope | 21:46 |
| fungi | go for it | 21:46 |
| jeblair | starting up | 21:46 |
| Shrews | jeblair: what i don't understand is that once that node was assigned, it's allocated_to is set to that request, and it won't be considered for other requests with that value set, but it was, apparently. | 21:47 |
| jeblair | 2017-09-28 21:46:45,066 INFO kazoo.client: Zookeeper connection established, state: CONNECTED | 21:47 |
| jeblair | yay ^ | 21:47 |
| jeblair | that's a kazoo.client log entry :) | 21:47 |
| mordred | jeblair: woot! | 21:47 |
| clarkb | mordred: theoretical fix? | 21:47 |
| *** armax has joined #openstack-infra | 21:47 | |
| jeblair | Shrews: yeah... we'd expect an "unallocating node" line, right? | 21:48 |
| jeblair | Shrews: (if the request went missing) | 21:48 |
| Shrews | jeblair: yeah | 21:48 |
| mordred | clarkb: ? | 21:48 |
| clarkb | mordred: 21:46:15 mordred | aha! so we can still try our theoretical fix | 21:48 |
| clarkb | mordred: do you just mean the change to the logging config? | 21:48 |
| mordred | clarkb: oh - sorry - we thought we've been runing with the updated config in place - but we weren't | 21:48 |
| mordred | clarkb: yah. turns out we HAVEN'T been running with the updated timeout either | 21:49 |
| clarkb | oh gotcha | 21:49 |
| clarkb | was the timeout fix installed before restarting? | 21:49 |
| Shrews | jeblair: and that node was not part of the returned nodeset for request 100-0000046344 ? | 21:49 |
| clarkb | It merged but not sure if puppet had updated git | 21:49 |
| jeblair | 2017-09-28 21:08:15,313 DEBUG zuul.nodepool: Resubmitting lost node request <NodeRequest 100-0000046344 <NodeSet ubuntu-xenial-2-node OrderedDict([('primary', <Node None primary:ubuntu-xenial>), ('subnodes', <Node None subnodes:ubuntu-xenial>)])OrderedDict()>> | 21:49 |
| jeblair | Shrews: ^ that's interesting | 21:49 |
| jeblair | Shrews: that happened before the first assignment | 21:50 |
| clarkb | pbr freeze says it is installed | 21:50 |
| Shrews | oh, hrm | 21:50 |
| mordred | clarkb: yah - it's in place now- I just didn't manage to get it in place with the puppets the first time | 21:50 |
| jeblair | Shrews: so maybe this is a case where the request is lost while a node is booting or something? | 21:50 |
| clarkb | I've rechecked the zuul-jobs sudo fixes | 21:51 |
| mordred | infra-root, dhellmann: the release-notes-jobs we generated are bong. I already started on the new releasenotes build job - but the publish job also needs help - I've put it on my list | 21:52 |
| clarkb | does anyone else want to review https://review.openstack.org/#/c/508317/2 before that gets approved? | 21:52 |
| fungi | thanks, i was about to reapprove them, but yeah they need a recheck because zuul had helpfully removed its verify +1 when enqueuing earlier | 21:52 |
| Shrews | jeblair: http://paste.openstack.org/show/622212/ | 21:52 |
| jlk | https://github.com/willthames/ansible-lint/pull/284 is my attempt at fixing ansible-lint for 'cmd' stuff. | 21:52 |
| mordred | clarkb: jlk makes a suggestoin in there ... | 21:53 |
| *** zoli has quit IRC | 21:53 | |
| jlk | it's a suggestion, but not a strong one. | 21:53 |
| *** iyamahat has joined #openstack-infra | 21:53 | |
| fungi | oh, unapproved | 21:53 |
| clarkb | mordred: ya its probably not a bad idea, will make the failure happen more closely to where it is important | 21:53 |
| tonyb | is there a nice way to find roles used by zuulv3 jobs? In this case I want to see what tox-siblings is/does | 21:53 |
| clarkb | but we can probably do that in a followup | 21:53 |
| dhellmann | clarkb , mordred : I have a patch for reno that I think may fix the origin problem. https://review.openstack.org/508324 | 21:53 |
| * tonyb blames mordred ;P | 21:53 | |
| dhellmann | fungi : ^^ | 21:53 |
| dmsimard | tonyb: codesearch.openstack.org for now | 21:53 |
| jeblair | tonyb: soon we'll have fully cross-linked auto-generated docs | 21:54 |
| jeblair | but yeah, nothing awesome yet | 21:54 |
| Shrews | jeblair: nodepool saw the request lost at 21:09:10,370 and i suspect the other node was not yet ready by that point, so the allocated_to was never set | 21:54 |
| dmsimard | tonyb: we've discussed implementing something similar to the 'print-template' macro eventually | 21:54 |
| jeblair | tonyb: https://docs.openstack.org/infra/zuul-jobs/roles.html#role-tox-siblings | 21:54 |
| dhellmann | clarkb, mordred, fungi : unfortunately I have an engagement this evening and can't hang around to finish helping. If someone summarizes what's needed for the releasenotes or publishing jobs I can try tomorrow afternoon. | 21:54 |
| fungi | dhellmann: your commit message is leaving me in suspense! | 21:54 |
| jeblair | tonyb: there's the docs for it, you just have to know to look there for the moment. | 21:55 |
| tonyb | dmsimard: http://codesearch.openstack.org/?q=tox-siblings&i=nope&files=&repos= That's the use but where's the defn? | 21:55 |
| dhellmann | bah | 21:55 |
| mordred | dhellmann: hopefully I'll have them fixed by tomorrow afternoon - but if I don't, I'll definitely ping you | 21:55 |
| Shrews | jeblair: ok, i think it's making sense now and you're right in that the request was lost while building nodes | 21:55 |
| mordred | tonyb: it's in zuul-jobs | 21:55 |
| jeblair | jlk: thx for command fix | 21:55 |
| dhellmann | fungi : fixed | 21:55 |
| tonyb | jeblair, mordred: Thanks | 21:55 |
| *** zoli has joined #openstack-infra | 21:56 | |
| *** gouthamr has quit IRC | 21:56 | |
| fungi | dhellmann: aww, i was hoping for a plot twist at the end | 21:56 |
| dmsimard | tonyb: hah, yeah I guess codesearch doesn't find folder names | 21:56 |
| *** jascott1 has quit IRC | 21:56 | |
| *** kjackal_ has quit IRC | 21:56 | |
| dhellmann | fungi : sorry to disappoint :-) | 21:56 |
| *** jascott1 has joined #openstack-infra | 21:56 | |
| tonyb | dmsimard: Yeah. I shoudl have know to look there | 21:56 |
| clarkb | fungi: re 508317 we can likely make that update in a followup and push the change as is for now to get that check fixed in jobs | 21:57 |
| *** jascott1 has quit IRC | 21:57 | |
| clarkb | I've got to step awa again for a bit though | 21:57 |
| *** jascott1 has joined #openstack-infra | 21:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-manual master: Add docs about tox jobs and sibling installation https://review.openstack.org/508327 | 21:58 |
| mordred | tonyb: ^^ docs | 21:58 |
| dmsimard | mordred: I was digging for nodepool stats on graphite.o.o, I'm not able to find any data after (what I assume is) 3:30PM today | 21:58 |
| fungi | clarkb: ianw also left a question inline on it | 21:59 |
| tonyb | mordred: You are a scholar and a gent! | 21:59 |
| mordred | jlk: does validate: go as a param to lineinfile? | 21:59 |
| dmsimard | mordred: wait nevermind I think I found them | 21:59 |
| jlk | mordred: it's a copy module arg | 21:59 |
| mordred | jlk: gah - I meant copy - but I'm guessing it's a top-level param? | 21:59 |
| jlk | mordred: http://docs.ansible.com/ansible/latest/copy_module.html | 21:59 |
| mordred | ah - neat | 21:59 |
| dmsimard | mordred: the new metrics are in stats_counts instead of stats | 21:59 |
| dmsimard | I'll send a patch | 21:59 |
| jlk | there's a specific sudo example in the docs | 21:59 |
| *** esberglu has quit IRC | 22:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Zuul versions of sudo grep checks https://review.openstack.org/508317 | 22:00 |
| tonyb | mordred: Awesome. | 22:00 |
| mordred | clarkb: ^^ updated with jlk's suggestion - do we have a good candidate that was failin earlier that we could depends-on to see if it fixes it? | 22:01 |
| * tonyb is keen to start builing in repo jobs | 22:01 | |
| * mordred is keen for the same thing | 22:01 | |
| ianw | i'm becoming a bit concerned LIBS_FROM_GIT in devstack is not working as expected | 22:01 |
| clarkb | mordred: was cinder that I noticed it on | 22:02 |
| *** jascott1 has quit IRC | 22:02 | |
| fungi | ianw: what's an example LIBS_FROM_GIT failure you're looking at again? | 22:03 |
| ianw | fungi: all these series of jobs for dib -> http://logs.openstack.org/79/504579/6/gate/legacy-dsvm-nodepool-ubuntu-src/9ac85a6/ | 22:04 |
| clarkb | mordred: http://logs.openstack.org/02/508302/1/check/legacy-devstack-gate-tox-run-tests/af6e423/job-output.txt.gz#_2017-09-28_21_00_32_154153 so 508302 | 22:04 |
| *** gouthamr has joined #openstack-infra | 22:04 | |
| mordred | ianw: SO - the likely general cause is us incorrectly adding the wrong set of required-projects to a job definition | 22:04 |
| clarkb | mordred: that is a devstack-gate change | 22:04 |
| mordred | clarkb: cool | 22:04 |
| ianw | mordred: so that's the thing, they seems to have all the required repos | 22:04 |
| clarkb | ok going to pop out for a bit now | 22:05 |
| *** bnemec has quit IRC | 22:05 | |
| Shrews | jeblair: anything else i can do help on this? | 22:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Save /etc/nodepool https://review.openstack.org/508302 | 22:05 |
| mordred | clarkb: ^^ I added a depends-on to the zuul-sudo patch | 22:05 |
| jeblair | mordred, tonyb: relevant: https://review.openstack.org/507955 | 22:05 |
| mordred | clarkb: so we should be able to verify it before we land it | 22:05 |
| *** baoli has quit IRC | 22:06 | |
| mordred | jeblair: ah - yes - we should land that- the necessary plumbing is in zuul now yeah? | 22:06 |
| jeblair | mordred: yep | 22:06 |
| *** bh526r has quit IRC | 22:06 | |
| mordred | cool | 22:06 |
| jeblair | Shrews: i think we haven't found any bugs, other than the disconnection itself. and we're in another debug cycle on that. | 22:07 |
| mnaser | can we define a project-template in a repo (and how can we migrate them, would we have to make a patch prefixing them with legacy-, to allow us to add the new ones? | 22:07 |
| * jlk is out for a bit too | 22:07 | |
| jeblair | mnaser: yes and yes. | 22:07 |
| Shrews | jeblair: ack | 22:07 |
| mnaser | okay, great, thanks jeblair :> | 22:07 |
| jeblair | mnaser: moving things between repos requires a multi-patch rename dance now. will be improved (somewhat) soonish. | 22:07 |
| *** kjackal_ has joined #openstack-infra | 22:07 | |
| *** iyamahat_ has joined #openstack-infra | 22:07 | |
| *** iyamahat has quit IRC | 22:07 | |
| mordred | tonyb: so - if you wanna play - that change jeblair linked to is depends-on able - https://review.openstack.org/507955 | 22:08 |
| *** iyamahat_ has quit IRC | 22:08 | |
| *** srobert_ has joined #openstack-infra | 22:08 | |
| *** wolverineav has quit IRC | 22:09 | |
| *** wolverineav has joined #openstack-infra | 22:09 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add missing required-project to Puppet https://review.openstack.org/508288 | 22:10 |
| jeblair | mordred: http://logs.openstack.org/17/508317/3/check/tox-linters/61a4044/job-output.txt.gz#_2017-09-28_22_09_42_987791 | 22:11 |
| *** srobert has quit IRC | 22:12 | |
| jeblair | mordred: i see it. i'll just fix it and update the change | 22:12 |
| mordred | jeblair: kk. cool | 22:12 |
| mordred | jeblair: btw - various teething issues notwithstanding - hallelujah for the ability to test these before they land :) | 22:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Zuul versions of sudo grep checks https://review.openstack.org/508317 | 22:13 |
| *** srobert_ has quit IRC | 22:13 | |
| *** thorst has joined #openstack-infra | 22:13 | |
| *** wolverineav has quit IRC | 22:14 | |
| mordred | ianw: I agree with you - they do all seem to be there | 22:14 |
| fungi | mordred: so the example failure here is for a diskimage-builder change, and it's complaining that diskimage-builder, among others, was not installed correctly from git (do jobs not automagically get the project for the change being tested as a required project?): http://logs.openstack.org/79/504579/6/gate/legacy-dsvm-nodepool-ubuntu-src/9ac85a6/logs/devstack-early.txt.gz#_2017-09-28_21_00_26_124 | 22:14 |
| dmsimard | mordred, clarkb: ok, indeed a couple things to fix on the nodepool graphs. I'll have a patch up soon. | 22:15 |
| jeblair | fungi: they do show up in zuul.projects | 22:15 |
| jeblair | dmsimard: thanks! | 22:15 |
| fungi | yeah, that too | 22:15 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Added legacy-puppet-openstack-integration templates https://review.openstack.org/508332 | 22:15 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Drop non-legacy Puppet project templates https://review.openstack.org/508333 | 22:15 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Switch puppet jobs to legacy template https://review.openstack.org/508334 | 22:16 |
| mordred | dmsimard: woot! thanks! | 22:16 |
| mnaser | ^ i think i did that right | 22:16 |
| *** bobh has quit IRC | 22:16 | |
| jeblair | mnaser: zuul will tell you in a few mins :) | 22:16 |
| *** slaweq_ has quit IRC | 22:17 | |
| *** thorst has quit IRC | 22:17 | |
| ianw | fungi: the install looks basically the same between a working & non-working version | 22:17 |
| mordred | mnaser: that stack looks good to me | 22:17 |
| mnaser | so maybe this is more of a user fault but its good to maybe say for folks to know, running zuul-cloner with sudo will fail as it will try to copy from /root | 22:18 |
| mnaser | ex: cp -dRl /root/src/git.openstack.org/openstack/puppet-barbican/. /etc/puppetlabs/code/modules/barbican | 22:18 |
| mordred | OH | 22:18 |
| *** gouthamr has quit IRC | 22:18 | |
| ianw | fungi: i'm wondering if the grep looking for 'git:' is involved | 22:18 |
| mordred | we could probably fix hat in the zuul-cloner shim if we wanted | 22:18 |
| *** r-daneel has joined #openstack-infra | 22:18 | |
| jeblair | mordred, mnaser: i'll look at that | 22:19 |
| fungi | ianw: it does certainly seem likely that there's just something off about the sanity check there, yes. maybe it on;y shows up as installed from local file now? | 22:19 |
| mnaser | jeblair http://logs.openstack.org/96/508296/6/check/puppet-openstack-integration-4-scenario001-tempest-centos-7/34ae42a/job-output.txt.gz#_2017-09-28_22_14_46_617447 example of a failure if you want to see it | 22:20 |
| *** gouthamr has joined #openstack-infra | 22:20 | |
| mordred | jeblair: I think if we just replace ~/src/git.openstack.org with /home/zuul/src/git.openstack.org in playbooks/base/pre.yaml | 22:20 |
| ianw | fungi: afaict, the install is exactly the same :/ | 22:21 |
| ianw | http://logs.openstack.org/79/504579/6/gate/legacy-dsvm-nodepool-ubuntu-src/9ac85a6/logs/devstacklog.txt.gz#_2017-09-28_20_49_23_502 | 22:21 |
| fungi | ianw: ## !! Could not determine repository location | 22:21 |
| fungi | ianw: http://logs.openstack.org/79/504579/6/gate/legacy-dsvm-nodepool-ubuntu-src/9ac85a6/logs/pip2-freeze.txt.gz | 22:21 |
| ianw | http://logs.openstack.org/79/504579/6/check/gate-dsvm-nodepool-redhat-src/dd7e240/logs/devstacklog.txt.gz#_2017-09-28_08_05_00_345 | 22:21 |
| jeblair | mordred: yes i had come to same conclusion :) | 22:21 |
| mordred | jeblair: woot | 22:21 |
| ianw | fungi: ahhh! | 22:21 |
| mordred | ianw: you found it??? | 22:22 |
| fungi | ianw: so i expect that's the difference now | 22:22 |
| ianw | no, but a smoking gun there... | 22:22 |
| ianw | https://stackoverflow.com/questions/19647028/pip-freeze-gives-me-this-error-related-with-git | 22:22 |
| ianw | Your git repository doesn't have an "origin" so pip is unable to detect the remote url of the repository. This should have been already fixed in PIP as stated in https://github.com/pypa/pip/issues/58 | 22:22 |
| ianw | that's from a long time ago, but ... | 22:23 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Stop running ansible-lint on playbooks https://review.openstack.org/508285 | 22:23 |
| *** thorst has joined #openstack-infra | 22:23 | |
| fungi | closely related to the reno problem | 22:23 |
| mnaser | btw folks are kinda confused about jobs not appearing in status.o.o/zuul .. maybe wanna land the redirect patch? | 22:23 |
| *** gouthamr has quit IRC | 22:23 | |
| dmsimard | clarkb: oh, so basically all the grafana dashboards are hosed basically | 22:24 |
| dmsimard | because of the job name changes | 22:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Update fetch-zuul-cloner in base-test https://review.openstack.org/508336 | 22:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Promote base-test to base https://review.openstack.org/508337 | 22:24 |
| *** tpsilva has quit IRC | 22:24 | |
| jeblair | there's the 2 step base playbook changes ^ | 22:24 |
| mordred | jeblair: both +2 from me | 22:24 |
| *** iyamahat has joined #openstack-infra | 22:25 | |
| fungi | approved the base-test version | 22:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: DNM: test fetch-zuul-cloner on base-test https://review.openstack.org/508338 | 22:28 |
| jeblair | mnaser: once the base-test change lands, you can push up a change that Depends-On: that one ^ and we can verify the fix | 22:28 |
| ianw | fungi: what's the reno problem, sorry? | 22:29 |
| mnaser | jeblair if i do a depends-on openstack-zuul-jobs commit in puppet-openstack-integration repo, magic will make it work?! | 22:29 |
| *** r-daneel has quit IRC | 22:29 | |
| jeblair | mnaser: yes :) | 22:29 |
| fungi | ianw: reno had some hard-coded assumptions about running in a git checkout with an origin remote | 22:30 |
| jeblair | mnaser: your jobs will be running with base-test as their parent base job, rather than base. | 22:30 |
| jeblair | mnaser: the base jobs are in a repo that doesn't self-test changes, so we have to check things out on base-test before we update base (otherwise we can break all jobs). but once base-test is updated, any job can dynamically reparent to it and that is self-testing. | 22:31 |
| fungi | ianw: its proposed fix is at 508324 (not adaptable to the LIBS_FROM_GIT checker since that's more about pip failing to tell you a git url) | 22:32 |
| fungi | ianw: more that they're related because they both stem from zuul v3 git checkouts lacking an origin remote | 22:32 |
| jeblair | mnaser, mordred, fungi, ianw: here's the redirect change; as mnaser points out, maybe we should land it now? https://review.openstack.org/507244 | 22:32 |
| fungi | jeblair: only reason i can think of to hold off is if we're worried about additional load on the status page? | 22:33 |
| dmsimard | jeblair, mordred: I'm not finding any legacy jobs in graphite | 22:33 |
| jeblair | fungi: it should have the same caching in place as the old one, so shouldn't be more load on zuul | 22:33 |
| dmsimard | for timing data (amongst other things) | 22:33 |
| jeblair | dmsimard: under nodepool.* or zuul.* prefix? | 22:34 |
| fungi | also, the implementation there reminds me, people like dansmith who have tooling relying on zuul.openstack.org/status.json aren't covered by redirecting from the status site | 22:34 |
| dmsimard | jeblair: I pretty much searched everywhere -- the current graphs pointed at stats.timers.nodepool.job.gate-tempest-dsvm-neutron-full-ubuntu-xenial.master.ubuntu-xenial.runtime.mean (for example) | 22:34 |
| jeblair | dmsimard: yeah, those are all gone. and i don't think zuulv3 has job timers yet. | 22:35 |
| dmsimard | jeblair: I searched at roughly equivalent places in stats.nodepool, stats.zuul, stats_count as well | 22:35 |
| jeblair | dmsimard: so we have no job timers in graphite atm. we'll need to update zuulv3 to add those. | 22:35 |
| ianw | fungi: yeah ... "Error when trying to get requirement for VCS system Command "git config --get-regexp remote\..*\.url" failed with error code 1 in /opt/stack/new/keystone, falling back to uneditable format" | 22:35 |
| *** thorst has quit IRC | 22:35 | |
| dmsimard | jeblair: ok, fair, do we have success/failure rates ? or just nothing at all ? | 22:35 |
| * dmsimard looks for failures | 22:36 | |
| *** thorst has joined #openstack-infra | 22:36 | |
| jeblair | dmsimard: nothing at all by job | 22:36 |
| dmsimard | ack, so most dashboards are broken | 22:36 |
| jeblair | dmsimard: zuulv3 *may* be emitting some pipeline stats, but that's probably it. | 22:36 |
| dmsimard | yeah I looked under pipeline things but no legacy jobs there either | 22:36 |
| *** eharney has quit IRC | 22:37 | |
| jeblair | fungi: yeah, that's just on static. we could add a redirect on zuul.o.o for the json file | 22:37 |
| *** lbragstad has quit IRC | 22:37 | |
| jeblair | fungi: i believe we don't proxy the json file through static, we just reference it directly | 22:37 |
| mordred | jeblair: I believe you are correct | 22:37 |
| mnaser | jeblair oh cool, let me try it | 22:37 |
| fungi | yeah, agreed | 22:38 |
| clarkb | I've rechecked 8274 and approved 8317 | 22:38 |
| jeblair | tbh, i'd be okay sticking zuul.o.o in emergency and doing that redirect by hand | 22:38 |
| clarkb | if both of those get in I think we should be good for sudo related things | 22:38 |
| fungi | jeblair: that's pretty much what i was thinking too. no need to put that through config management | 22:39 |
| fungi | i can do that if you aren't already on it | 22:39 |
| jeblair | fungi: go for it, thx | 22:39 |
| fungi | just redirect /(.*) to /$1 | 22:39 |
| jeblair | yeah | 22:39 |
| jeblair | fungi: be sure to make it temp redir | 22:39 |
| fungi | yep | 22:40 |
| fungi | i was using the status redirect as a template | 22:40 |
| mnaser | jeblair sudo zuul-cloner being tested - https://review.openstack.org/#/c/508296/ | 22:40 |
| clarkb | forum topic brainstorming is apparently ending this week, Do we think it is worth while (and or appropriate) to have a session for zuulv3 related things. Probably similar to what was done at the ptg | 22:40 |
| mordred | clarkb: ++ | 22:40 |
| *** esberglu has joined #openstack-infra | 22:40 | |
| fungi | #status log zuul.openstack.org has been added to the emergency disable list so that a temporary redirect to zuulv3 can be installed by hand | 22:41 |
| mordred | clarkb: we should allow for the possibility that it'll want to run over by 5x its original scheduled time :) | 22:41 |
| openstackstatus | fungi: finished logging | 22:41 |
| jeblair | clarkb: ++ let's talk to ppl about zuulv3 and listen :) | 22:41 |
| jeblair | by the forum, people other than us will have things to say about it. :) | 22:41 |
| fungi | session block is 40 minutes, adjourning to the bar thereafter ;) | 22:41 |
| jeblair | i mean, our 30m chat at the ptg turned into a 2.5 hour forum session, complete with feedback on an unreleased product. :) | 22:42 |
| *** thorst has quit IRC | 22:42 | |
| clarkb | ok I'm going to context switch for a bit to get that written up since it looks like sudo problems may finally be addressed | 22:43 |
| clarkb | did 508302 ever get rechecked? | 22:43 |
| clarkb | it did | 22:43 |
| fungi | RedirectMatch temp ^/(.*) http://zuulv3.openstack.org/$1 | 22:44 |
| fungi | that is now in place in zuul.o.o's vhost config | 22:45 |
| *** esberglu has quit IRC | 22:45 | |
| fungi | and seems to be working | 22:45 |
| *** thorst has joined #openstack-infra | 22:45 | |
| fungi | dansmith: ^ | 22:45 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Zuul versions of sudo grep checks https://review.openstack.org/508317 | 22:45 |
| dansmith | fungi: confirmed working here | 22:46 |
| ianw | mordred/fungi/jeblair: https://review.openstack.org/#/c/508344/ -- i think i can fix the check by using "pip list" which seems to give the path from which things were installed from. but i think this will be required for anyone using LIBS_FROM_GIT | 22:46 |
| *** akscram1 has quit IRC | 22:46 | |
| mnaser | so https://review.openstack.org/#/c/508332/ passed, but https://review.openstack.org/#/c/508334/ seems to appear in the queue (with no jobs, so zuul doing zuul things) then disappear from the queue. am i doing something wrong? (note: that change is depended by https://review.openstack.org/#/c/508333/ as well) | 22:46 |
| fungi | <mr_burns>excellent</mr_burns> | 22:46 |
| *** akscram1 has joined #openstack-infra | 22:47 | |
| fungi | ianw: yeah, i guess some codesearch to see how widespread the problem is? | 22:47 |
| fungi | could get messy | 22:47 |
| ianw | LIBS_FROM_GIT is pretty popular | 22:47 |
| clarkb | mordred: http://logs.openstack.org/02/508302/2/check/legacy-devstack-gate-tox-py3-run-tests/5c0937e/job-output.txt.gz#_2017-09-28_22_43_59_262286 and of course8317 just merged, that is my bad | 22:47 |
| fungi | ianw: in tox.ini wrappers within random repos i guess? | 22:47 |
| clarkb | not that it is any worse than the existing situation, just not better | 22:48 |
| ianw | fungi: no, only for devstack jobs to make sure you're getting the latest clients, etc | 22:48 |
| fungi | ianw: so the same sanity checker may have been cargo-culted into hooks/plugins? | 22:48 |
| clarkb | I wonder if the file has to be in place first | 22:48 |
| clarkb | mordred: ^ | 22:48 |
| clarkb | the script that is | 22:48 |
| fungi | ianw: just trying to figure out why the fix would have to be widespread | 22:49 |
| ianw | fungi: no, luckily this sanity checker shouldn't be replicated anywhere. but anyone using LIBS_FROM_GIT in their devstack job will need it | 22:49 |
| fungi | ianw: if the problem is widespread but the fix is just in branches of devstack, then presumably not too hard to address? | 22:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix post log location https://review.openstack.org/508346 | 22:49 |
| jeblair | whoopsie ^ | 22:49 |
| *** xyang1 has quit IRC | 22:50 | |
| ianw | fungi: that's correct, just need to come up with something reliable for devstack. that's why i thought avoid the check for the immediate issue, then we can consider it's replacement under less stress | 22:51 |
| fungi | ianw: agreed, that sounds sane if you can get the devstack-core reviewers on board | 22:51 |
| fungi | the other devstack-core reviewers besides yourself, i mean | 22:51 |
| jeblair | ianw: wow, *pip* has stuff about git remotes hardcoded into it? | 22:53 |
| *** lukebrowning has quit IRC | 22:53 | |
| ianw | jeblair: "pip freeze", anyway | 22:53 |
| jeblair | i mean, there is certainly no requirement that a git repo have any particular remote, much less one named 'origin'. | 22:53 |
| ianw | pip list gives us something like : "keystone (12.0.0.0rc2.dev64, /opt/stack/new/keystone)" | 22:53 |
| *** lukebrowning has joined #openstack-infra | 22:53 | |
| fungi | specifically, it will try to helpfully identify the remote git url for things you install, and falls back to not doing that and leaving a comment in its output | 22:54 |
| ianw | so we can probably grep for that, to make sure things are installed from local dirs | 22:54 |
| mordred | ianw: ++ | 22:54 |
| fungi | so it's not especially insane behavior on pip's part, it's just we grew dependent on having that | 22:54 |
| mordred | clarkb: I wish there was a better error message | 22:54 |
| clarkb | mordred: ya I'm looking at the old jenkins file and its basically identical except for the username and the filepath | 22:55 |
| clarkb | mordred: my hunch is its checking the filepath exists first | 22:55 |
| clarkb | so maybe flip that order around? | 22:55 |
| mordred | clarkb: oh - wait - | 22:55 |
| mnaser | jeblair: 2017-09-28 22:55:07.839544 | ubuntu-xenial | cp -dRl /home/zuul/src/git.openstack.org/openstack/puppet-barbican/. /etc/puppetlabs/code/modules/barbican | 22:56 |
| mnaser | so i can confirm with depends-on: https://review.openstack.org/#/c/508338/ it is fixed | 22:56 |
| mordred | clarkb: oh - yah. I was also thinking we migh twant to do visudo -cf /etc/sudoers ... since that'w hat we really care about | 22:56 |
| mordred | clarkb: we don't care if that one snippet validates correctly as much as that sudo validates correctly | 22:57 |
| clarkb | mordred: you can drop the -f entirely | 22:57 |
| jeblair | mnaser: is that running as root? i don't think the base-test change has landed yet | 22:57 |
| clarkb | and just visudo -c | 22:57 |
| clarkb | mordred: at least that is what is in dib | 22:57 |
| mnaser | jeblair yes, i did a depends-on in that change to test it | 22:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Swap order of sudoers manipulation https://review.openstack.org/508348 | 22:58 |
| jeblair | mnaser: can you show me your change? | 22:58 |
| *** thorst has quit IRC | 22:58 | |
| mnaser | jeblair https://review.openstack.org/#/c/508296/ | 22:58 |
| *** lukebrowning has quit IRC | 22:58 | |
| *** thorst has joined #openstack-infra | 22:58 | |
| *** mat128 has quit IRC | 22:58 | |
| mnaser | http://logs.openstack.org/96/508296/7/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/e2c590d/job-output.txt.gz#_2017-09-28_22_55_07_749569 | 22:58 |
| clarkb | mordred: you can update the depends on on 508302 to get more infos | 22:58 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Update Nodepool graphite metric names https://review.openstack.org/508349 | 22:59 |
| mordred | clarkb: yup | 22:59 |
| mnaser | jeblair: you can see the same command as the other time, but this time it actually works and doesn't copy from root ^ | 22:59 |
| dmsimard | ^ first batch of metric updates | 22:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Save /etc/nodepool https://review.openstack.org/508302 | 22:59 |
| mordred | clarkb: done | 22:59 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove the right file in revoke-sudo https://review.openstack.org/508274 | 22:59 |
| *** lukebrowning has joined #openstack-infra | 22:59 | |
| clarkb | how does https://etherpad.openstack.org/p/iES0s0bRBm look for forum session? | 23:00 |
| mordred | clarkb: lgtm | 23:00 |
| mnaser | okay, now i think i spotted a bug in the zuul-cloner shim when cloning multiple projects | 23:00 |
| mnaser | http://logs.openstack.org/96/508296/7/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/e2c590d/job-output.txt.gz#_2017-09-28_22_55_07_749569 | 23:00 |
| mnaser | notice puppet-aodh is not mapped | 23:01 |
| mnaser | puppet-barbican (the 2nd item) is | 23:01 |
| *** rbrndt has quit IRC | 23:01 | |
| *** masber has quit IRC | 23:02 | |
| jeblair | mnaser: i really hope that isn't running as root. because it should not have worked. | 23:02 |
| *** rbrndt has joined #openstack-infra | 23:02 | |
| clarkb | dmsimard: selfish request, getting the ara deeplinking working in firefox would be so nice | 23:02 |
| jeblair | mnaser: a couple lines above that, you're using sudo... | 23:02 |
| dmsimard | clarkb: ikr | 23:02 |
| *** rbrndt has quit IRC | 23:02 | |
| *** rbrndt has joined #openstack-infra | 23:02 | |
| mnaser | jeblair it is running as root.. because otherwise it wouldn't be able to copy to /etc/puppetlabs/code afaik | 23:03 |
| *** rbrndt has quit IRC | 23:03 | |
| dmsimard | clarkb: it's not selfish at all, ENOTIME :P | 23:03 |
| dmsimard | clarkb: I haven't touched ara for non-critical reasons for like 2-3 weeks | 23:03 |
| clarkb | mordred: thats interesting 508302 didn't seem to queue at all | 23:03 |
| clarkb | oh depends on a change that failed to merge | 23:03 |
| dmsimard | clarkb: I'll try to fix it this weekend or something | 23:03 |
| mnaser | jeblair wait, sorry, you might be right | 23:03 |
| * mnaser is confused | 23:03 | |
| clarkb | sure enough 508348 needs a rebase, I'll push that in a sec | 23:04 |
| *** lukebrowning has quit IRC | 23:04 | |
| *** Goneri has joined #openstack-infra | 23:04 | |
| mnaser | jeblair hang on, xenial wasnt running as root and centos tries to run it as root | 23:04 |
| mnaser | checking puppet code.. maybe we sudo in xenial and not in centos | 23:04 |
| dmsimard | I did a recheck on https://review.openstack.org/#/c/507889/ and I'm not seeing it in queue, maybe something went wrong :( | 23:05 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Swap order of sudoers manipulation https://review.openstack.org/508348 | 23:05 |
| *** reed has quit IRC | 23:05 | |
| *** lukebrowning has joined #openstack-infra | 23:06 | |
| mnaser | jeblair: ok, looks like the same code runs fine under xenial, but doesnt under centos.. xenial uses /home/zuul, centos (tries to) uses /root/ | 23:06 |
| mordred | mnaser: cool! that's the behavior we should see until that base-test job lands | 23:06 |
| jeblair | mnaser: oh it may be a difference on whether HOME is passed through when sudoing... | 23:07 |
| *** hongbin has quit IRC | 23:07 | |
| mnaser | mordred okay ill wait then | 23:07 |
| jeblair | mnaser: just so you know, if we see this fixed before the base-test job lands, that's a stop everything and roll-back kind of bug. :) | 23:07 |
| mordred | jeblair: yah | 23:07 |
| mordred | jeblair: also - the project-config patch seems to be in limbo | 23:07 |
| mnaser | jeblair we use - export SUDO='sudo -E' | 23:08 |
| *** reed has joined #openstack-infra | 23:08 | |
| *** gongysh has joined #openstack-infra | 23:08 | |
| mnaser | but yeah ill wait | 23:08 |
| jeblair | mnaser: so please do keep an eye out on whether centos job succeeds (bad) or fails (good) before the base-test change lands. | 23:08 |
| *** gongysh has quit IRC | 23:08 | |
| mnaser | jeblair ack | 23:08 |
| mnaser | (i guess centos doesn't pass homedir even with sudo -E) | 23:08 |
| clarkb | mordred: wow ok so validate must contain %s | 23:08 |
| clarkb | mordred: because ansible | 23:08 |
| jeblair | mnaser: looking | 23:08 |
| mnaser | now that still brings me back first item of zuul-cloner arg being dropped | 23:09 |
| clarkb | mordred: rather than a validate step lets just add a subsequent task to run visudo -c | 23:09 |
| clarkb | mordred: I will push that now | 23:09 |
| mordred | clarkb: ++ | 23:09 |
| *** lukebrowning has quit IRC | 23:11 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Swap order of sudoers manipulation https://review.openstack.org/508348 | 23:11 |
| *** lukebrowning has joined #openstack-infra | 23:12 | |
| jlvillal | I'm looking at the Zuul v3 status page: http://zuulv3.openstack.org/ | 23:13 |
| jlvillal | In particular Ironic jobs | 23:13 |
| jlvillal | I notice that the "legacy-grenade-dsvm-ironic" job is non-voting. I'm not sure how that happened???? | 23:13 |
| jlvillal | It has been a voting job for over a year now. | 23:14 |
| mordred | jlvillal: probably a bug in the migration script (almost certainly my fault) | 23:14 |
| jlvillal | mordred: Okay thanks. Can I consider a bug filed with you about it? :) | 23:14 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix post log location https://review.openstack.org/508346 | 23:14 |
| mnaser | ok i think i may be onto something | 23:15 |
| * jlvillal hopes that merged patch ^^ fixes the POST_FAILURE issues he has seen | 23:15 | |
| clarkb | mordred: we get a bette rerror message now | 23:15 |
| clarkb | mordred: http://logs.openstack.org/02/508302/3/check/legacy-devstack-gate-tox-run-tests/5b622e6/ara/ | 23:16 |
| mnaser | but can someone help me find where the zuul legacy vars filter(?) comes from? | 23:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make legacy-grenade-dsvm-ironic voting https://review.openstack.org/508353 | 23:16 |
| mordred | jlvillal: ^^ | 23:16 |
| *** lukebrowning has quit IRC | 23:16 | |
| jlvillal | mordred: Thanks! | 23:17 |
| mordred | mnaser: it's in the zuul repo | 23:17 |
| mnaser | ok couldnt find it initially ill look harder :> | 23:17 |
| mordred | mnaser: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/filter/zuul_filters.py?h=feature/zuulv3 | 23:17 |
| mnaser | feature/zuulv3 branch, oops | 23:17 |
| mordred | mnaser: :) | 23:17 |
| mnaser | ok so | 23:17 |
| *** kjackal_ has quit IRC | 23:17 | |
| mordred | jlvillal: I'm happy to see that most of your jobs are passing though!!! | 23:17 |
| mnaser | ZUUL_URL was purposely omitted | 23:17 |
| mnaser | and i think this is the reason why it's not working | 23:18 |
| jlvillal | mordred: Me too :) | 23:18 |
| mordred | clarkb: WHAT DOES THAT MEAN???? | 23:18 |
| SpamapS | I was looking through the converted jobs and I couldn't for the life of me find where the legacy jobs' playbooks were actually defined | 23:19 |
| SpamapS | is their parent configured to run something? I didn't get too deep | 23:19 |
| jeblair | mnaser: we don't have a sensible value to put there, but we could put the string "NONE" or something | 23:19 |
| * SpamapS starts answering his own question just by saying it out loud | 23:20 | |
| *** abelur has quit IRC | 23:20 | |
| mordred | SpamapS: openstack-infra/openstack-zuul-jobs in the playbooks/legacy dir | 23:20 |
| dmsimard | There's a new error on tripleo jobs | 23:20 |
| mordred | dmsimard: \o/ | 23:20 |
| mordred | dmsimard: that means progress | 23:20 |
| clarkb | mordred: it needs a new line at the end I think | 23:21 |
| *** wolverineav has joined #openstack-infra | 23:21 | |
| dmsimard | http://paste.openstack.org/show/622217/ | 23:21 |
| * clarkb makes a patch | 23:21 | |
| SpamapS | like castaways on an island, we're just excited when anything new happens | 23:21 |
| *** tosky has quit IRC | 23:21 | |
| clarkb | can I put a \n in the ansible file content string? | 23:22 |
| clarkb | dmsimard: ^ | 23:22 |
| dmsimard | clarkb: a literal \n ? | 23:23 |
| mordred | clarkb: oh - maybe just change it to content: | | 23:23 |
| clarkb | dmsimard: copy: content: "foo\n" | 23:23 |
| clarkb | dmsimard: I need the file to end with a newline | 23:23 |
| mordred | clarkb: OR - we could put content into a file next to the other file and just use copy to copy the file | 23:24 |
| mordred | clarkb: like with zuul-sudo-grep.sh itself | 23:24 |
| clarkb | I really wnat the \n to work because well thats how it should work imo | 23:24 |
| mordred | clarkb: I agree | 23:24 |
| clarkb | but I'm learning t oexpect the unexpected | 23:24 |
| clarkb | I'm gonna try \n really quick | 23:24 |
| mordred | clarkb: kk | 23:24 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Swap order of sudoers manipulation https://review.openstack.org/508348 | 23:24 |
| *** lukebrowning has joined #openstack-infra | 23:24 | |
| dmsimard | clarkb: hmm I'm not with my laptop right now.. I think one of the yaml things could work | 23:24 |
| dmsimard | Like +> or something like that | 23:25 |
| dmsimard | | keeps whitespace, > does some stripping | 23:25 |
| clarkb | ya but does | keep a blank line? | 23:26 |
| clarkb | maybe? | 23:26 |
| openstackgerrit | Kaitlin Farr proposed openstack-infra/project-config master: Promote castellan's barbican-tempest-plugin job to nv https://review.openstack.org/508356 | 23:26 |
| mnaser | https://review.openstack.org/#/c/508336/ - is it normal that 60 minutes later zuul hasn't commented about this change / nor is it in the queue | 23:26 |
| jeblair | mnaser: that's what i'm looking at now | 23:26 |
| mnaser | oh okay :) | 23:27 |
| clarkb | queues are rising now too (could be we tripped the node allocation problem) | 23:27 |
| clarkb | or maybe just a reconfigure | 23:27 |
| mnaser | clarkb maybe not, because im seeing my p-o-i job get queued up and just started within 2-3 minutes | 23:28 |
| clarkb | must've been reconfigure it just went | 23:28 |
| dmsimard | clarkb: I think an indented line break after your last line with | should work | 23:28 |
| dmsimard | But I don't have my laptop to test :/ | 23:29 |
| *** lukebrowning has quit IRC | 23:29 | |
| clarkb | dmsimard: interesting might need a comment too # this line of whitespace is intentional do not delete | 23:29 |
| *** abelur has joined #openstack-infra | 23:29 | |
| dmsimard | clarkb: try to look at http://www.yaml.org/spec/1.2/spec.html#id2778853 | 23:29 |
| SamYaple | ok. I have waited a days worth of time. Is zuulv3 in heavy work mode still and i should come back later? I have a job that recheck is failing on (its a noop change) for zuulv3 | 23:30 |
| *** lukebrowning has joined #openstack-infra | 23:31 | |
| mnaser | SamYaple what is your failure? | 23:31 |
| jeblair | ah, i think i found the issue with 508336 | 23:31 |
| mnaser | logs/info would be useful so that we can see if its a common job failure reason | 23:31 |
| *** Goneri has quit IRC | 23:31 | |
| *** thorst has quit IRC | 23:31 | |
| SamYaple | I am assuming this: http://logs.openstack.org/09/506409/3/check/legacy-loci/717d1dd/job-output.txt.gz#_2017-09-28_19_13_16_059784 | 23:31 |
| jeblair | the only job running on it is the noop job, which has no elapsed time. the sql reporter can't handle a job without an elapsed time and fails. that prevents the gerrit reporter form reporting. | 23:31 |
| SamYaple | problem with the zuul cloning stuff | 23:32 |
| mordred | jeblair: oh! nice | 23:32 |
| mnaser | SamYaple ok so what you need to do is | 23:32 |
| mnaser | for your jobs that need to clone something like with zuul-cloner | 23:32 |
| jeblair | mordred, clarkb: we can fix this while running by removing our dependency on noop (pick another job to always run on project-config) | 23:32 |
| mnaser | you'll have to add required-projects: and the name of your project | 23:32 |
| jeblair | mordred, clarkb: the real fix will require a zuul change and restart | 23:32 |
| jeblair | mordred, clarkb: so i think we should do the in-repo fix first. | 23:33 |
| mordred | jeblair: I agree | 23:33 |
| mnaser | SamYaple have a look at openstack-infra/openstack-zuul-jobs (new repo) then inside there zuul.d/zuul-legacy-jobs.yaml | 23:33 |
| SamYaple | have it open mnaser | 23:33 |
| clarkb | woo \n in content seems to have worked | 23:33 |
| SamYaple | searching for required-projects example now | 23:33 |
| mnaser | SamYaple and ctrl+f required-projects | 23:33 |
| mnaser | see how some jobs use it, you'll have to list any projects you'll be cloning in there | 23:33 |
| SamYaple | yep thats the plan :) | 23:34 |
| SamYaple | thanks for the help | 23:34 |
| clarkb | lets see if the rest of the job completes the nwe can get reviews on 8348 | 23:34 |
| mnaser | SamYaple and suggest doing out-of-tree testing and moving the legacy jobs over imho | 23:34 |
| mordred | SamYaple: (and then next step is to move the jobs to your repos and also to remove use of zuul-cloner at all, but you know that :) ) | 23:34 |
| mnaser | ^ | 23:34 |
| SamYaple | yea this isn't a priority, i think ill just write new jobs and not try to fix the legacy gates | 23:35 |
| SamYaple | thats good advice | 23:35 |
| *** lukebrowning has quit IRC | 23:35 | |
| mnaser | you can take that path too, its what im doing with the puppet jobs, there's docs on how to migrate/drop the legacy ones once you got new ones | 23:35 |
| SamYaple | yes i have those open too | 23:36 |
| SamYaple | ill probably drop back in with some questions | 23:36 |
| mnaser | okay, i'm really blocked now.. zuul-cloner seems to drop my first project when giving it a list of projects.. would anyone have any ideas? (i can drop it and do plain cp's, but it might be a bigger issue) | 23:37 |
| mnaser | http://logs.openstack.org/96/508296/8/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/79e1d91/job-output.txt.gz#_2017-09-28_23_32_19_249268 | 23:37 |
| mnaser | openstack/puppet-aodh seems to be ignored | 23:37 |
| *** markvoelker has quit IRC | 23:37 | |
| jeblair | mnaser: i thought you traced it down to zuul_url being unset? | 23:37 |
| mnaser | jeblair thats what i thought, so i dropped --zuul-url from zuul-cloner alongside other un-used things | 23:38 |
| mnaser | and its still failing | 23:38 |
| jeblair | mnaser: point me where you dropped it? | 23:38 |
| fungi | doesn't it expect an origin as a positional parameter too? | 23:38 |
| jeblair | fungi: yes, it had that | 23:38 |
| fungi | ahh | 23:39 |
| *** thorst has joined #openstack-infra | 23:39 | |
| mnaser | ok, this all makes sense now | 23:39 |
| mnaser | i thought it was a bug where the positional parameter of ZUUL_URL was there twice | 23:39 |
| jeblair | so mnaser's command was "--zuul-url $ZUUL_URL origin" but $ZUUL_URL was null, so the origin was being eaten as the url | 23:39 |
| mnaser | so i removed the reference of ZUUL_ULR and the origin | 23:39 |
| mordred | jeblair: how about if we just remove the exclusions on openstack-tox-linters: ? | 23:39 |
| mnaser | what i should have done is kept the origin | 23:39 |
| jeblair | mnaser: ah yeah. that should fix it. | 23:39 |
| mnaser | my bad, sorry for the noise, ill push up another attempt | 23:40 |
| jeblair | mnaser: exclusions? | 23:40 |
| jeblair | mordred: ^ | 23:40 |
| mordred | jeblair: sorry - for project-config - just run openstack-tox-linters: on all project-config changes | 23:40 |
| jeblair | mordred: ah yes ++ | 23:40 |
| jeblair | i paged that out quick sorry | 23:40 |
| jeblair | i'll go work on a zuul change real quick | 23:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Always run openstack-tox-linters on project-config https://review.openstack.org/508358 | 23:41 |
| fungi | so we can get rid of the no-op job? sounds good to me | 23:41 |
| mordred | jeblair, clarkb, fungi: ^^ | 23:41 |
| *** amoralej|off has quit IRC | 23:41 | |
| jeblair | mordred: note, if that only runs noop, we may have a problem merging it | 23:41 |
| * clarkb is learning lineinfile is a beast | 23:41 | |
| SamYaple | clarkb: :) | 23:41 |
| mordred | jeblair: good point | 23:41 |
| *** lukebrowning has joined #openstack-infra | 23:42 | |
| jeblair | mordred: maybe force-merge it and parent? | 23:42 |
| mordred | yah | 23:42 |
| jeblair | wcpgw | 23:42 |
| *** jaypipes has quit IRC | 23:42 | |
| fungi | we could drop the no-op job in a second patch to get around that, right? | 23:43 |
| *** jpena|off has quit IRC | 23:43 | |
| mordred | fungi: nah - the problem is that we have exclusions on everything so we wind up with no jobs | 23:43 |
| fungi | oh, right, that | 23:43 |
| mordred | fungi: we USED to run the layout check job on every change ... but not so much with that now | 23:43 |
| fungi | the separate zero-duration sql reporter problem | 23:44 |
| *** ltomasbo has quit IRC | 23:44 | |
| mnaser | i guess this explains why this change had reported nothing as well? https://review.openstack.org/#/c/508334/ | 23:44 |
| mordred | mnaser: yah | 23:45 |
| mnaser | cool | 23:45 |
| mordred | it turns out project-config is a giant outlier in many ways | 23:45 |
| *** ltomasbo has joined #openstack-infra | 23:45 | |
| mordred | we found several bugs in things that assumed a project witha tox.ini file would CLEARLY have a setup.py :) | 23:46 |
| *** amoralej has joined #openstack-infra | 23:46 | |
| *** jpena|off has joined #openstack-infra | 23:46 | |
| *** lukebrowning has quit IRC | 23:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Swap order of sudoers manipulation https://review.openstack.org/508348 | 23:46 |
| SamYaple | just cuase i want a tox.ini doesnt mean i want to make a big fuss about it and get all fancy with your new age setup.py | 23:47 |
| clarkb | mordred: ^ I have high hopes for that one | 23:47 |
| *** lukebrowning has joined #openstack-infra | 23:48 | |
| mordred | clarkb: \o/ | 23:48 |
| mordred | clarkb, fungi, jeblair: I'm gonna go ahead and merge the project-config always-run-linters change - it's an only-noop one | 23:49 |
| fungi | k | 23:49 |
| mnaser | does the logfile option actaully work in the zuul web console | 23:49 |
| fungi | as jeblair says, wcpgw? | 23:49 |
| mnaser | i just tried http://zuulv3.openstack.org/static/stream.html?uuid=8d5144a8817049cc800bd767fd663430&logfile=puppet.log but it showed the console anyways | 23:49 |
| mordred | clarkb: lineinfile strips whitespace? | 23:49 |
| clarkb | mordred: no problem was there are two lines we have to swap | 23:50 |
| mordred | mnaser: yes - we have not yet implemented support for streaming more than one file | 23:50 |
| mnaser | ah gotcha :) | 23:50 |
| clarkb | mordred: old patchsets only did the 'post' command but need to do the pre as well | 23:50 |
| mordred | ah | 23:50 |
| mordred | mnaser: we left a placeholder because we want to add that :) | 23:50 |
| clarkb | I was going to do it with backrefs and regexp but that ends up being far less readable | 23:50 |
| mnaser | well in other good news.. puppet integration jobs seem to have gotten as far as running the install so i think its gonna pass (granted a few things pending merges but yay) | 23:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Make legacy-grenade-dsvm-ironic voting https://review.openstack.org/508353 | 23:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Always run openstack-tox-linters on project-config https://review.openstack.org/508358 | 23:51 |
| *** mriedem is now known as mriedem_dinner | 23:51 | |
| mordred | jlvillal: ^^ your legacy-grenade-dsvm-ironic jobs should vote now | 23:51 |
| ianw | 508344 seems to be stuck waiting for a centos-7 node ... and there's one that seems to have been building for an hour. i think something's up | 23:52 |
| jlvillal | mordred: Thank you | 23:52 |
| *** lukebrowning has quit IRC | 23:52 | |
| mordred | clarkb: while you're waiting - mnaser has a patch here https://review.openstack.org/#/c/508332/ and https://review.openstack.org/#/c/508334 that both look good | 23:52 |
| jeblair | ianw: we can confirm if that's the specific node it's waiting on | 23:52 |
| *** Goneri has joined #openstack-infra | 23:52 | |
| jeblair | 2017-09-28 22:45:42,260 DEBUG zuul.IndependentPipelineManager: Adding node request <NodeRequest 100-0000048489 <NodeSet centos-7 OrderedDict([('centos-7', <Node None centos-7:centos-7>)])OrderedDict()>> for job legacy-tempest-dsvm-neutron-full-centos-7 to item <QueueItem 0x7f109d76c940 for <Change 0x7f10ada2e4e0 508344,1> in check> | 23:53 |
| mnaser | mordred i think ill have to rebase the second one to the one that adds openstack-tox-linters ? | 23:53 |
| jeblair | ianw: ^ | 23:53 |
| *** SumitNaiksatam has quit IRC | 23:53 | |
| clarkb | mordred: http://logs.openstack.org/02/508302/3/check/legacy-devstack-gate-tox-run-tests/8971b5f/ worked, can hasa review and hopefully approval on 8348? | 23:54 |
| clarkb | and with that we should be good on sudo things | 23:54 |
| ianw | jeblair: hmm, the stuck node is 0000043014 | 23:54 |
| *** lukebrowning has joined #openstack-infra | 23:54 | |
| *** thorst has quit IRC | 23:54 | |
| jeblair | | 0000043014 | rax-ord | None | centos-7 | 7416ec80-4bf0-4cd6-9c3f-0e5582f9582c | ready | 00:00:00:11 | unlocked | centos-7-rax-ord-0000043014 | 104.130.195.50 | 10.210.197.118 | 2001:4801:7828:101:be76:4eff:fe10:6f69 | 22 | nl01.openstack.org-30932-PoolWorker.rax-ord-main | 100-0000048489 | None ... | 23:54 |
| jeblair | ... | None | | 23:54 |
| *** thorst has joined #openstack-infra | 23:55 | |
| mordred | mnaser: you shouldn't no - once it lands that's the config for projet-config (it doesn't do speculative jobs) | 23:55 |
| jeblair | ianw: so that node is assigned to that request which has been fulfilled | 23:55 |
| mnaser | ok cool | 23:55 |
| jeblair | ianw: ball should be in zuul's court | 23:55 |
| jeblair | 2017-09-28 23:54:13,899 INFO zuul.nodepool: Node request <NodeRequest 100-0000048489 <NodeSet centos-7 OrderedDict([('centos-7', <Node 0000043014 centos-7:centos-7>)])OrderedDict()>> fulfilled | 23:56 |
| *** iyamahat has quit IRC | 23:56 | |
| jeblair | ianw: ^ looks like zuul just got it | 23:56 |
| *** yamahata has quit IRC | 23:56 | |
| clarkb | ianw: jeblair fungi https://review.openstack.org/#/c/508348/ should be the last step of fixing sudo grep | 23:56 |
| clarkb | ianw: jeblair fungi tested at http://logs.openstack.org/02/508302/3/check/legacy-devstack-gate-tox-run-tests/8971b5f/ | 23:57 |
| ianw | jeblair: and there it goes ... hmm sorry bout that | 23:57 |
| jeblair | ianw: np. good exercise :) | 23:57 |
| ianw | 0000043014 has still been building for an hour | 23:57 |
| jeblair | ianw: it's finished now, but you mean it took an hour to build, yeah? | 23:58 |
| ianw | yeah | 23:59 |
| ianw | it must have just finished | 23:59 |
| clarkb | I approved 508332 and that has made zuul do its reconfigure dance I think | 23:59 |
| *** baoli has joined #openstack-infra | 23:59 | |
| *** Goneri has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!