| *** lukebrowning has quit IRC | 00:01 | |
| *** akscram1 has quit IRC | 00:01 | |
| *** akscram1 has joined #openstack-infra | 00:02 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove infra-files-ro and infra-files-rw https://review.openstack.org/529189 | 00:02 |
|---|---|---|
| jeblair | pabelanger, ianw: the nodepool fixes are failing check. can one of you start working on manually reverting to a20d81c32c7caeb330bbc6b7373ddb975c533e20 on the launchers and restarting? | 00:02 |
| ianw | pabelanger: there is now :) | 00:02 |
| *** lukebrowning has joined #openstack-infra | 00:02 | |
| jeblair | i have to run right now, but can check back in a bit | 00:03 |
| ianw | are they emergency-ed? | 00:03 |
| jeblair | ianw: no | 00:03 |
| ianw | pabelanger: i can ... just let me know so we don't overwrite each other | 00:03 |
| jeblair | (remember to pip3 install nodepool -- not pip install) | 00:03 |
| pabelanger | ianw: +2 | 00:03 |
| pabelanger | ianw: go for it | 00:04 |
| ianw | ok, first thing i'll put them in emergency | 00:04 |
| *** salv-orlando has quit IRC | 00:05 | |
| *** salv-orlando has joined #openstack-infra | 00:05 | |
| *** lukebrowning has quit IRC | 00:07 | |
| *** edmondsw has joined #openstack-infra | 00:08 | |
| *** wolverineav has quit IRC | 00:08 | |
| ianw | a20d81c32c7caeb330bbc6b7373ddb975c533e20 is the feature/zuulv3 branch, right? | 00:09 |
| ianw | tot | 00:09 |
| pabelanger | ianw: yah, feature/zuulv3 | 00:10 |
| *** salv-orlando has quit IRC | 00:11 | |
| ianw | is that not just what puppet installs? | 00:11 |
| pabelanger | yah, currently | 00:12 |
| ianw | oh right, jeblair installed some changes manually | 00:12 |
| pabelanger | oh, I see what you are saying | 00:13 |
| *** edmondsw has quit IRC | 00:13 | |
| ianw | ok, i've re-installed 3e20 on nl01, will restart the launcher now | 00:13 |
| pabelanger | k | 00:14 |
| ianw | ok, that's up, moving on to nl02 | 00:14 |
| ianw | ... done. so both nl01 & nl02 are running nodepool-launcher from feature/zuulv3 TOT (...3e20) | 00:16 |
| pabelanger | kk | 00:16 |
| pabelanger | should be safe to remove from emergency | 00:17 |
| ianw | do we want to roll back further though, prior to quota changes? | 00:17 |
| pabelanger | I think it is safe, we might just not have rax | 00:18 |
| pabelanger | which, should be okay until tomorrow | 00:18 |
| pabelanger | or patches land | 00:18 |
| pabelanger | odd, I see node_failure on status.o.o | 00:18 |
| pabelanger | might be first time I have seen that | 00:19 |
| ianw | pabelanger: ok, both removed from emergency since we're deploying from the branch | 00:19 |
| *** lukebrowning has joined #openstack-infra | 00:19 | |
| pabelanger | wfm | 00:20 |
| *** Vi has joined #openstack-infra | 00:20 | |
| ianw | i'm guessing we could revert back to https://git.openstack.org/cgit/openstack-infra/nodepool/commit/?h=feature/zuulv3&id=afcb56e0fb887a090dbf1380217ebcc06ef6b66b | 00:21 |
| *** Vi is now known as Guest80233 | 00:21 | |
| ianw | (just based on merge times from https://git.openstack.org/cgit/openstack-infra/nodepool/log/?h=feature/zuulv3) | 00:21 |
| ianw | but i will just hold tight until further instructions :) | 00:22 |
| pabelanger | yah | 00:23 |
| *** Guest80233 has quit IRC | 00:23 | |
| *** lukebrowning has quit IRC | 00:24 | |
| *** lukebrowning has joined #openstack-infra | 00:26 | |
| *** threestrands has joined #openstack-infra | 00:26 | |
| *** threestrands has quit IRC | 00:26 | |
| *** threestrands has joined #openstack-infra | 00:26 | |
| *** lukebrowning has quit IRC | 00:30 | |
| *** xarses_ has quit IRC | 00:31 | |
| *** bobh has joined #openstack-infra | 00:32 | |
| *** lukebrowning has joined #openstack-infra | 00:32 | |
| jeblair | oh shoot | 00:33 |
| jeblair | i gave you the wrong hash | 00:33 |
| jeblair | should have been afcb56e0fb887a090dbf1380217ebcc06ef6b66b | 00:33 |
| jeblair | because, indeed, the point was not to run the branch tip which is broken | 00:33 |
| jeblair | i'll revert back to that | 00:34 |
| *** dave-mccowan has quit IRC | 00:35 | |
| jeblair | done | 00:36 |
| *** lukebrowning has quit IRC | 00:37 | |
| jeblair | #status log nl01 nl02 manually downgraded to afcb56e0fb887a090dbf1380217ebcc06ef6b66b due to broken quota handling on branch tip | 00:37 |
| openstackstatus | jeblair: finished logging | 00:37 |
| *** esberglu has quit IRC | 00:38 | |
| ianw | jeblair: heh, i *thought* that was what you might have meant :) | 00:38 |
| *** lukebrowning has joined #openstack-infra | 00:38 | |
| ianw | i'll put them in the emergency file now, to avoid puppet updating that? | 00:39 |
| *** wolverineav has joined #openstack-infra | 00:39 | |
| jeblair | ianw: i lean slightly toward leaving it alone, so that config changes still happen; they won't automatically restart or anything, so it's okay as long as they aren't rebooted. and we'll hopefully get the fix in by tomorrow anyway | 00:41 |
| jeblair | ianw: but if you would rather add to emergency, that's fine. i could go either way. :) | 00:41 |
| ianw | i won't be around tomorrow, so let's leave it! | 00:42 |
| *** lukebrowning has quit IRC | 00:43 | |
| *** wolverineav has quit IRC | 00:44 | |
| *** jamesmcarthur has joined #openstack-infra | 00:44 | |
| *** lukebrowning has joined #openstack-infra | 00:45 | |
| *** caphrim007_ has joined #openstack-infra | 00:47 | |
| *** kjackal has quit IRC | 00:47 | |
| *** kjackal has joined #openstack-infra | 00:48 | |
| *** lukebrowning has quit IRC | 00:49 | |
| *** caphrim007 has quit IRC | 00:50 | |
| *** lukebrowning has joined #openstack-infra | 00:51 | |
| *** caphrim007_ has quit IRC | 00:52 | |
| *** lukebrowning has quit IRC | 00:56 | |
| *** bobh has quit IRC | 00:56 | |
| *** kiennt26 has joined #openstack-infra | 00:56 | |
| *** lukebrowning has joined #openstack-infra | 00:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Log provider names with quota https://review.openstack.org/529178 | 01:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: WIP: Fail on quota-exceeded (partial revert) https://review.openstack.org/529173 | 01:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Assume a quota limit of -1 means unlimited https://review.openstack.org/529180 | 01:00 |
| *** jamesmcarthur has quit IRC | 01:01 | |
| *** lukebrowning has quit IRC | 01:01 | |
| *** lukebrowning has joined #openstack-infra | 01:03 | |
| fungi | catching up on quota excitement now | 01:04 |
| *** salv-orlando has joined #openstack-infra | 01:06 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove jenkins related puppet manifests https://review.openstack.org/529145 | 01:07 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove additional dead jenkins related code https://review.openstack.org/529150 | 01:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/487538 | 01:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add babel transpiling enabling use of ES6 features https://review.openstack.org/528295 | 01:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add StandardJS linting and analysis https://review.openstack.org/528296 | 01:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fix source_url handling for jobs view https://review.openstack.org/528373 | 01:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fix StandardJS warnings and turn them to errors https://review.openstack.org/528297 | 01:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add bundle analysis to the lint target https://review.openstack.org/528298 | 01:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make bundle of build web content https://review.openstack.org/528374 | 01:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove use strict https://review.openstack.org/528437 | 01:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Inject url endpoint information https://review.openstack.org/529193 | 01:07 |
| *** lukebrowning has quit IRC | 01:08 | |
| jeblair | anyone understand time.monotonic and why that's different in the gate vs on my machine? | 01:10 |
| *** wolverineav has joined #openstack-infra | 01:10 | |
| mordred | ugh - I feel like I did understand time.monotonic at one point | 01:10 |
| *** salv-orlando has quit IRC | 01:11 | |
| *** dhill_ has quit IRC | 01:11 | |
| mordred | jeblair: I'm assuming you've seen "The reference point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid." already | 01:11 |
| jeblair | mordred: yeah, i think that's the thing at issue | 01:11 |
| *** dave-mccowan has joined #openstack-infra | 01:11 | |
| jeblair | it's probably not worth understanding why the reference time is always low in the gate and high locally, and instead just rework the whole thing. | 01:12 |
| *** cuongnv has joined #openstack-infra | 01:12 | |
| mordred | jeblair: I wonder if that reference time has something to do with clocks in vms | 01:12 |
| mordred | jeblair: like - I agree, it's probably more better to rework the thing, although I'd also find it intellectually interesting to understand what was the cause of different reference points | 01:13 |
| jeblair | mordred: yeah, i would probably have agreed a couple of hours ago :) | 01:14 |
| jeblair | good news -- this test is now capable of exposing 2 bugs | 01:15 |
| mordred | jeblair: woot! | 01:15 |
| mordred | jeblair: that makes it an extra good test | 01:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Fail on quota-exceeded (partial revert) https://review.openstack.org/529173 | 01:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Assume a quota limit of -1 means unlimited https://review.openstack.org/529180 | 01:17 |
| jeblair | okay, i think those have a chance of passing now | 01:17 |
| *** wolverineav has quit IRC | 01:17 | |
| *** asettle has quit IRC | 01:19 | |
| *** lukebrowning has joined #openstack-infra | 01:19 | |
| clarkb | so the < comparison wasnt working with now being monotonic? | 01:20 |
| *** yamamoto has joined #openstack-infra | 01:21 | |
| *** yamamoto has quit IRC | 01:21 | |
| jeblair | clarkb: yeah, in the gate we were getting values like "243" for now. 243 < 0 + 300 so we used the invalid cache. | 01:21 |
| *** asettle has joined #openstack-infra | 01:22 | |
| clarkb | ok I reapproved the two that lost their +As and +2'd the one with monotonic | 01:23 |
| *** annp has joined #openstack-infra | 01:23 | |
| *** psachin has joined #openstack-infra | 01:23 | |
| fungi | i'll take a look | 01:24 |
| *** lukebrowning has quit IRC | 01:24 | |
| fungi | tc office hour is pretty dead tonight | 01:24 |
| *** jamesmcarthur has joined #openstack-infra | 01:25 | |
| *** lukebrowning has joined #openstack-infra | 01:26 | |
| *** cshastri has joined #openstack-infra | 01:26 | |
| *** Apoorva_ has joined #openstack-infra | 01:26 | |
| *** psachin has quit IRC | 01:28 | |
| *** liujiong has joined #openstack-infra | 01:29 | |
| *** Apoorva has quit IRC | 01:29 | |
| *** lukebrowning has quit IRC | 01:30 | |
| *** Apoorva_ has quit IRC | 01:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add accessbot for #openstack-vpnaas https://review.openstack.org/527866 | 01:32 |
| *** psachin has joined #openstack-infra | 01:32 | |
| *** lukebrowning has joined #openstack-infra | 01:32 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Inject url endpoint information https://review.openstack.org/529193 | 01:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make bundle of build web content https://review.openstack.org/528374 | 01:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove use strict https://review.openstack.org/528437 | 01:34 |
| *** lukebrowning has quit IRC | 01:37 | |
| *** yamahata has quit IRC | 01:38 | |
| *** lukebrowning has joined #openstack-infra | 01:38 | |
| *** yamahata has joined #openstack-infra | 01:39 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Log provider names with quota https://review.openstack.org/529178 | 01:41 |
| *** lukebrowning has quit IRC | 01:43 | |
| *** makowals has quit IRC | 01:43 | |
| *** andreykurilin has quit IRC | 01:43 | |
| *** makowals has joined #openstack-infra | 01:44 | |
| *** lukebrowning has joined #openstack-infra | 01:44 | |
| *** jd_ has quit IRC | 01:45 | |
| *** _ari_ is now known as ari|pto | 01:46 | |
| *** andreykurilin has joined #openstack-infra | 01:46 | |
| *** jd_ has joined #openstack-infra | 01:47 | |
| *** rmcall has quit IRC | 01:47 | |
| *** lukebrowning has quit IRC | 01:49 | |
| *** lukebrowning has joined #openstack-infra | 01:51 | |
| *** claudiub|2 has quit IRC | 01:53 | |
| *** lukebrowning has quit IRC | 01:55 | |
| *** liujiong has quit IRC | 01:56 | |
| *** liujiong has joined #openstack-infra | 01:56 | |
| *** jamesmcarthur has quit IRC | 01:57 | |
| *** lukebrowning has joined #openstack-infra | 01:57 | |
| *** yamamoto has joined #openstack-infra | 01:57 | |
| *** felipemonteiro has quit IRC | 01:57 | |
| *** caphrim007 has joined #openstack-infra | 01:59 | |
| *** lukebrowning has quit IRC | 02:02 | |
| *** iyamahat has quit IRC | 02:03 | |
| *** hoangcx has joined #openstack-infra | 02:03 | |
| *** iyamahat has joined #openstack-infra | 02:03 | |
| *** lukebrowning has joined #openstack-infra | 02:04 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fail on quota-exceeded (partial revert) https://review.openstack.org/529173 | 02:04 |
| *** caphrim007 has quit IRC | 02:04 | |
| *** dprince has quit IRC | 02:06 | |
| *** salv-orlando has joined #openstack-infra | 02:07 | |
| *** lukebrowning has quit IRC | 02:08 | |
| *** salv-orlando has quit IRC | 02:11 | |
| *** lukebrowning has joined #openstack-infra | 02:19 | |
| *** r-daneel has quit IRC | 02:20 | |
| hoangcx | Hi team, | 02:20 |
| hoangcx | I have a small question: the following patch has been merged recently https://review.openstack.org/#/c/527255/ | 02:21 |
| hoangcx | But we don't it publish to https://governance.openstack.org/tc/reference/projects/neutron.html | 02:22 |
| hoangcx | After looking we found that it seems post job failure: http://logs.openstack.org/24/24e11953f30926e2ce7eedfd0a6c2f62b3ace451/post/publish-governance-tc/2164f19/job-output.txt | 02:22 |
| hoangcx | Could you please help us to check it? Thanks a lot | 02:23 |
| *** lukebrowning has quit IRC | 02:24 | |
| hoangcx | s/But we don't/But we don't see | 02:24 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/openstack-zuul-jobs master: networking-midonet: Remove some of legacy jobs https://review.openstack.org/522471 | 02:25 |
| *** lukebrowning has joined #openstack-infra | 02:25 | |
| mordred | hoangcx: there was an issue earlier today - it should be fixed by https://review.openstack.org/#/c/529155/ | 02:27 |
| mordred | hoangcx: unfortunately, that landed about 30 minutes after the patch you mention | 02:27 |
| mordred | hoangcx: the next patch that lands to governance should update the docs | 02:27 |
| *** lukebrowning has quit IRC | 02:30 | |
| *** Goneri has quit IRC | 02:30 | |
| hoangcx | mordred: Ah, I see the problem now. Thanks a lot for your head up :) | 02:31 |
| *** lukebrowning has joined #openstack-infra | 02:32 | |
| mordred | hoangcx: sure thing - sorry for the issue! | 02:32 |
| *** bobh has joined #openstack-infra | 02:34 | |
| *** lukebrowning has quit IRC | 02:36 | |
| *** catintheroof has joined #openstack-infra | 02:38 | |
| *** lukebrowning has joined #openstack-infra | 02:44 | |
| *** lukebrowning has quit IRC | 02:49 | |
| *** calebb has quit IRC | 02:49 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Fix build-javascript-content success-url https://review.openstack.org/529203 | 02:50 |
| *** lukebrowning has joined #openstack-infra | 02:50 | |
| *** bobh has quit IRC | 02:51 | |
| *** bobh has joined #openstack-infra | 02:52 | |
| *** catintheroof has quit IRC | 02:53 | |
| *** lukebrowning has quit IRC | 02:55 | |
| *** gmann has quit IRC | 02:55 | |
| *** calebb has joined #openstack-infra | 02:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Don't install npm dependencies in /var/log https://review.openstack.org/528625 | 02:56 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Xenial: add a service file https://review.openstack.org/528626 | 02:56 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Update default version of nodejs https://review.openstack.org/528156 | 02:56 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Add basic rspec test https://review.openstack.org/528130 | 02:56 |
| *** ijw has quit IRC | 02:58 | |
| *** rmcall has joined #openstack-infra | 02:59 | |
| *** lukebrowning has joined #openstack-infra | 03:01 | |
| *** rlandy|rover has quit IRC | 03:01 | |
| *** threestrands has quit IRC | 03:03 | |
| *** threestrands has joined #openstack-infra | 03:04 | |
| *** threestrands has quit IRC | 03:05 | |
| *** threestrands has joined #openstack-infra | 03:06 | |
| *** bobh has quit IRC | 03:06 | |
| *** lukebrowning has quit IRC | 03:06 | |
| *** threestrands has quit IRC | 03:07 | |
| *** threestrands has joined #openstack-infra | 03:07 | |
| *** salv-orlando has joined #openstack-infra | 03:07 | |
| *** lukebrowning has joined #openstack-infra | 03:07 | |
| *** lin_yang has quit IRC | 03:08 | |
| *** yamahata has quit IRC | 03:11 | |
| *** iyamahat has quit IRC | 03:12 | |
| *** lukebrowning has quit IRC | 03:12 | |
| *** salv-orlando has quit IRC | 03:12 | |
| *** bobh has joined #openstack-infra | 03:13 | |
| *** dave-mccowan has quit IRC | 03:13 | |
| *** lukebrowning has joined #openstack-infra | 03:14 | |
| *** vivsoni has quit IRC | 03:15 | |
| *** lukebrowning has quit IRC | 03:18 | |
| *** lukebrowning has joined #openstack-infra | 03:20 | |
| *** lukebrowning has quit IRC | 03:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Don't install npm dependencies in /var/log https://review.openstack.org/528625 | 03:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Xenial: add a service file https://review.openstack.org/528626 | 03:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Update default version of nodejs https://review.openstack.org/528156 | 03:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Add basic rspec test https://review.openstack.org/528130 | 03:25 |
| *** lukebrowning has joined #openstack-infra | 03:26 | |
| *** andreas_s has joined #openstack-infra | 03:29 | |
| *** tinwood has quit IRC | 03:30 | |
| *** tinwood has joined #openstack-infra | 03:31 | |
| *** lukebrowning has quit IRC | 03:31 | |
| *** lukebrowning has joined #openstack-infra | 03:33 | |
| *** andreas_s has quit IRC | 03:33 | |
| *** lukebrowning has quit IRC | 03:37 | |
| *** caphrim007 has joined #openstack-infra | 03:38 | |
| *** edmondsw has joined #openstack-infra | 03:44 | |
| *** Apoorva has joined #openstack-infra | 03:44 | |
| *** gongysh has joined #openstack-infra | 03:48 | |
| *** edmondsw has quit IRC | 03:49 | |
| *** slaweq has joined #openstack-infra | 03:49 | |
| *** ramishra has joined #openstack-infra | 03:50 | |
| *** gyee has quit IRC | 03:51 | |
| *** slaweq has quit IRC | 03:54 | |
| *** owalsh_ has joined #openstack-infra | 03:55 | |
| *** udesale has joined #openstack-infra | 03:57 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on Speaker PUT https://review.openstack.org/529213 | 03:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Don't install npm dependencies in /var/log https://review.openstack.org/528625 | 03:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Xenial: add a service file https://review.openstack.org/528626 | 03:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Update default version of nodejs https://review.openstack.org/528156 | 03:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Add basic rspec test https://review.openstack.org/528130 | 03:57 |
| ianw | clarkb: arrgh, why'd you have to go and point out perfectly valid problems! :) | 03:57 |
| *** threestrands_ has joined #openstack-infra | 03:57 | |
| *** threestrands has quit IRC | 03:57 | |
| clarkb | ianw sorry | 03:58 |
| *** threestrands_ has quit IRC | 03:58 | |
| *** zhurong has joined #openstack-infra | 03:58 | |
| *** threestrands_ has joined #openstack-infra | 03:59 | |
| *** owalsh has quit IRC | 03:59 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on Speaker PUT https://review.openstack.org/529213 | 04:00 |
| *** lukebrowning has joined #openstack-infra | 04:02 | |
| *** lukebrowning has quit IRC | 04:07 | |
| *** yamamoto_ has joined #openstack-infra | 04:08 | |
| *** salv-orlando has joined #openstack-infra | 04:08 | |
| *** yamamoto has quit IRC | 04:11 | |
| *** lukebrowning has joined #openstack-infra | 04:13 | |
| *** salv-orlando has quit IRC | 04:13 | |
| *** namnh has joined #openstack-infra | 04:14 | |
| *** lukebrowning has quit IRC | 04:18 | |
| mordred | ianw: clarkb has talent in those areas | 04:19 |
| *** lukebrowning has joined #openstack-infra | 04:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Add basic rspec test https://review.openstack.org/528130 | 04:19 |
| *** shu-mutou has joined #openstack-infra | 04:20 | |
| mordred | ianw, clarkb: any reason to not approve https://review.openstack.org/#/c/526978/3 ? | 04:24 |
| *** lukebrowning has quit IRC | 04:24 | |
| ianw | mordred: i don't think so, but i'm not really going to be around to check on it | 04:24 |
| *** iyamahat has joined #openstack-infra | 04:25 | |
| *** lukebrowning has joined #openstack-infra | 04:25 | |
| ianw | i'm still working on the rspec test that fully validates the etherpad patches | 04:25 |
| *** ykarel has joined #openstack-infra | 04:27 | |
| *** pgadiya has joined #openstack-infra | 04:29 | |
| *** lukebrowning has quit IRC | 04:30 | |
| *** lukebrowning has joined #openstack-infra | 04:32 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed assets sync mapping https://review.openstack.org/529219 | 04:33 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed assets sync mapping https://review.openstack.org/529219 | 04:34 |
| *** iyamahat has quit IRC | 04:35 | |
| *** lukebrowning has quit IRC | 04:36 | |
| *** lukebrowning has joined #openstack-infra | 04:38 | |
| *** iyamahat has joined #openstack-infra | 04:39 | |
| *** pgadiya has quit IRC | 04:39 | |
| SpamapS | hmmm weird thing with glean | 04:41 |
| SpamapS | I have nodes that have fqdn's in the hostname field but are not getting an fqdn hostname set. | 04:42 |
| *** lukebrowning has quit IRC | 04:42 | |
| SpamapS | Oh that's kind of annoying... | 04:43 |
| SpamapS | glean uses 'name' | 04:43 |
| SpamapS | not 'hostname' | 04:43 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite master: Fix up log rotation https://review.openstack.org/528624 | 04:43 |
| *** armax has quit IRC | 04:44 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/glean master: Use hostname from config drive if set https://review.openstack.org/529221 | 04:47 |
| *** bobh has quit IRC | 04:48 | |
| *** lukebrowning has joined #openstack-infra | 04:50 | |
| *** armax has joined #openstack-infra | 04:51 | |
| *** armax has quit IRC | 04:51 | |
| *** lukebrowning has quit IRC | 04:55 | |
| *** pgadiya has joined #openstack-infra | 04:56 | |
| *** lukebrowning has joined #openstack-infra | 04:57 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-etherpad_lite master: Add basic rspec test https://review.openstack.org/528130 | 04:57 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/glean master: Use hostname from config drive if set https://review.openstack.org/529221 | 04:57 |
| *** ianw is now known as ianw_pto | 04:57 | |
| *** links has joined #openstack-infra | 04:57 | |
| *** armaan has quit IRC | 05:02 | |
| *** felipemonteiro has joined #openstack-infra | 05:03 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Fix codesearch stderr capturing https://review.openstack.org/529188 | 05:03 |
| *** lukebrowning has quit IRC | 05:06 | |
| *** xarses_ has joined #openstack-infra | 05:08 | |
| *** lukebrowning has joined #openstack-infra | 05:08 | |
| *** salv-orlando has joined #openstack-infra | 05:09 | |
| *** yamamoto_ has quit IRC | 05:10 | |
| *** lukebrowning has quit IRC | 05:13 | |
| *** lukebrowning has joined #openstack-infra | 05:14 | |
| *** salv-orlando has quit IRC | 05:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Assume a quota limit of -1 means unlimited https://review.openstack.org/529180 | 05:15 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on Admin Speaker Serializer https://review.openstack.org/529223 | 05:16 |
| openstackgerrit | zhufl proposed openstack-infra/elastic-recheck master: Remove "Initializing SSH connection to" message from query https://review.openstack.org/519546 | 05:16 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on Admin Speaker Serializer https://review.openstack.org/529223 | 05:17 |
| *** namnh has quit IRC | 05:19 | |
| *** lukebrowning has quit IRC | 05:19 | |
| *** coolsvap has joined #openstack-infra | 05:20 | |
| *** lukebrowning has joined #openstack-infra | 05:21 | |
| *** lukebrowning has quit IRC | 05:25 | |
| *** lukebrowning has joined #openstack-infra | 05:27 | |
| *** pgadiya has quit IRC | 05:30 | |
| *** pgadiya has joined #openstack-infra | 05:31 | |
| *** lukebrowning has quit IRC | 05:32 | |
| *** edmondsw has joined #openstack-infra | 05:32 | |
| *** yamamoto has joined #openstack-infra | 05:32 | |
| *** edmondsw has quit IRC | 05:37 | |
| *** lukebrowning has joined #openstack-infra | 05:39 | |
| *** bhavik has joined #openstack-infra | 05:42 | |
| openstackgerrit | Yushiro FURUKAWA proposed openstack-infra/project-config master: Add python-pqclient project https://review.openstack.org/527612 | 05:42 |
| *** bhavik has quit IRC | 05:44 | |
| *** lukebrowning has quit IRC | 05:44 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on Admin Speaker Serializer https://review.openstack.org/529224 | 05:44 |
| *** bhavik has joined #openstack-infra | 05:45 | |
| *** jascott1 has quit IRC | 05:45 | |
| *** xarses_ has quit IRC | 05:46 | |
| *** lukebrowning has joined #openstack-infra | 05:46 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on Admin Speaker Serializer https://review.openstack.org/529224 | 05:46 |
| *** jascott1 has joined #openstack-infra | 05:46 | |
| *** armaan has joined #openstack-infra | 05:48 | |
| *** Apoorva has quit IRC | 05:50 | |
| *** lukebrowning has quit IRC | 05:50 | |
| *** jascott1 has quit IRC | 05:50 | |
| *** lukebrowning has joined #openstack-infra | 05:52 | |
| *** armaan has quit IRC | 05:52 | |
| *** bhavik has quit IRC | 05:54 | |
| *** lukebrowning has quit IRC | 05:57 | |
| *** lukebrowning has joined #openstack-infra | 05:58 | |
| *** lukebrowning has quit IRC | 06:03 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Assume a quota limit of -1 means unlimited https://review.openstack.org/529180 | 06:03 |
| *** lukebrowning has joined #openstack-infra | 06:05 | |
| *** lukebrowning has quit IRC | 06:09 | |
| *** salv-orlando has joined #openstack-infra | 06:10 | |
| *** lukebrowning has joined #openstack-infra | 06:14 | |
| *** salv-orlando has quit IRC | 06:15 | |
| *** d0ugal has quit IRC | 06:17 | |
| *** lukebrowning has quit IRC | 06:18 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Test that -1 works for infinite quota https://review.openstack.org/529231 | 06:19 |
| *** rmcall has quit IRC | 06:21 | |
| *** d0ugal has joined #openstack-infra | 06:23 | |
| *** lukebrowning has joined #openstack-infra | 06:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Make AFS marker roles honor sphinx variables https://review.openstack.org/525735 | 06:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Migrate special docs jobs to build-openstack-sphinx-docs https://review.openstack.org/525736 | 06:26 |
| prometheanfire | SamYaple: didn't think it'd get heated so fast lol | 06:26 |
| *** salv-orlando has joined #openstack-infra | 06:28 | |
| *** lukebrowning has quit IRC | 06:30 | |
| *** lukebrowning has joined #openstack-infra | 06:31 | |
| *** felipemonteiro has quit IRC | 06:38 | |
| *** felipemonteiro has joined #openstack-infra | 06:39 | |
| *** lukebrowning has quit IRC | 06:41 | |
| *** ianychoi has quit IRC | 06:44 | |
| *** lukebrowning has joined #openstack-infra | 06:45 | |
| *** e0ne has joined #openstack-infra | 06:47 | |
| *** yamamoto has quit IRC | 06:49 | |
| *** janki has joined #openstack-infra | 06:49 | |
| *** jascott1 has joined #openstack-infra | 06:50 | |
| *** felipemonteiro has quit IRC | 06:50 | |
| *** lukebrowning has quit IRC | 06:50 | |
| *** lukebrowning has joined #openstack-infra | 06:52 | |
| SamYaple | prometheanfire: i know right. i was literally just looking for bullet points. "no and heres why" | 06:53 |
| SamYaple | im a big boy. i can handle it | 06:53 |
| SamYaple | honestly stuff like that is kinda why i hate posting to the ML | 06:56 |
| *** andreas_s has joined #openstack-infra | 06:56 | |
| *** lukebrowning has quit IRC | 06:57 | |
| *** threestrands_ has quit IRC | 06:57 | |
| *** pahuang has quit IRC | 06:58 | |
| *** lukebrowning has joined #openstack-infra | 06:58 | |
| *** lukebrowning has quit IRC | 07:03 | |
| *** yamamoto has joined #openstack-infra | 07:04 | |
| *** lukebrowning has joined #openstack-infra | 07:04 | |
| *** `Kevin has quit IRC | 07:04 | |
| *** `Kevin has joined #openstack-infra | 07:05 | |
| *** rcernin has quit IRC | 07:08 | |
| prometheanfire | :P | 07:08 |
| *** yamamoto has quit IRC | 07:09 | |
| *** lukebrowning has quit IRC | 07:09 | |
| *** e0ne has quit IRC | 07:12 | |
| *** lukebrowning has joined #openstack-infra | 07:13 | |
| *** gongysh has quit IRC | 07:17 | |
| *** lukebrowning has quit IRC | 07:18 | |
| *** claudiub|2 has joined #openstack-infra | 07:19 | |
| *** e0ne has joined #openstack-infra | 07:19 | |
| *** yamamoto has joined #openstack-infra | 07:20 | |
| *** lukebrowning has joined #openstack-infra | 07:20 | |
| *** edmondsw has joined #openstack-infra | 07:21 | |
| *** yamamoto has quit IRC | 07:24 | |
| *** lukebrowning has quit IRC | 07:24 | |
| *** edmondsw has quit IRC | 07:25 | |
| *** e0ne has quit IRC | 07:25 | |
| *** lukebrowning has joined #openstack-infra | 07:26 | |
| *** kjackal has quit IRC | 07:28 | |
| *** shardy has joined #openstack-infra | 07:29 | |
| *** lukebrowning has quit IRC | 07:31 | |
| *** lukebrowning has joined #openstack-infra | 07:32 | |
| *** kiennt26 has quit IRC | 07:35 | |
| openstackgerrit | Dirk Mueller proposed openstack-dev/hacking master: Transition to flake8 2.6.x https://review.openstack.org/514934 | 07:37 |
| *** lukebrowning has quit IRC | 07:37 | |
| *** lukebrowning has joined #openstack-infra | 07:38 | |
| *** iyamahat has quit IRC | 07:42 | |
| *** gongysh has joined #openstack-infra | 07:42 | |
| *** lukebrowning has quit IRC | 07:43 | |
| *** slaweq has joined #openstack-infra | 07:45 | |
| *** lukebrowning has joined #openstack-infra | 07:45 | |
| ttx | AJaeger: thanks for your help getting everything back to normal yesterday ! | 07:45 |
| *** HeOS has joined #openstack-infra | 07:46 | |
| *** iyamahat has joined #openstack-infra | 07:49 | |
| *** lukebrowning has quit IRC | 07:50 | |
| *** iyamahat has quit IRC | 07:50 | |
| *** lukebrowning has joined #openstack-infra | 07:51 | |
| *** lukebrowning has quit IRC | 07:56 | |
| *** iyamahat has joined #openstack-infra | 07:57 | |
| *** lukebrowning has joined #openstack-infra | 07:57 | |
| *** mkoderer__ has joined #openstack-infra | 08:01 | |
| *** lukebrowning has quit IRC | 08:02 | |
| *** lukebrowning has joined #openstack-infra | 08:04 | |
| *** yamamoto has joined #openstack-infra | 08:04 | |
| *** iyamahat has quit IRC | 08:08 | |
| *** lukebrowning has quit IRC | 08:09 | |
| *** yamamoto has quit IRC | 08:09 | |
| *** raissa has quit IRC | 08:10 | |
| *** hoangcx has quit IRC | 08:12 | |
| *** hoangcx has joined #openstack-infra | 08:13 | |
| *** lukebrowning has joined #openstack-infra | 08:16 | |
| *** zhurong has quit IRC | 08:17 | |
| *** daidv has quit IRC | 08:18 | |
| *** daidv has joined #openstack-infra | 08:19 | |
| *** lukebrowning has quit IRC | 08:21 | |
| *** salv-orlando has quit IRC | 08:22 | |
| *** lukebrowning has joined #openstack-infra | 08:23 | |
| AJaeger | ttx: sorry for helping to break it ;) | 08:23 |
| *** iyamahat has joined #openstack-infra | 08:23 | |
| AJaeger | ttx, you need to merge a single patch for governance to publish again - we only fixed the post job once all the changes were in ;8 | 08:24 |
| AJaeger | ttx, do you have other repos that need python3 for docs publishing that we should fix? | 08:24 |
| *** alexchadin has joined #openstack-infra | 08:26 | |
| *** cshastri has quit IRC | 08:26 | |
| *** ralonsoh has joined #openstack-infra | 08:27 | |
| *** jtomasek has joined #openstack-infra | 08:27 | |
| *** ccamacho has joined #openstack-infra | 08:27 | |
| *** hashar has joined #openstack-infra | 08:27 | |
| *** lukebrowning has quit IRC | 08:27 | |
| *** lukebrowning has joined #openstack-infra | 08:29 | |
| oanson | Hello. | 08:31 |
| oanson | I get the following error when trying to ssh to review.openstack.org: Too many concurrent connections (64) - max. allowed: 64 | 08:31 |
| oanson | I don't have any open connections on my end. | 08:31 |
| *** jtomasek has quit IRC | 08:31 | |
| oanson | Can someone please help? | 08:32 |
| *** jtomasek has joined #openstack-infra | 08:32 | |
| *** lukebrowning has quit IRC | 08:34 | |
| *** yamamoto has joined #openstack-infra | 08:34 | |
| *** amoralej|off is now known as amoralej | 08:35 | |
| *** lukebrowning has joined #openstack-infra | 08:35 | |
| *** cshastri has joined #openstack-infra | 08:39 | |
| *** yamamoto has quit IRC | 08:39 | |
| *** lukebrowning has quit IRC | 08:40 | |
| frickler | oanson: are you using a shared gateway so that others in your company or some CI might be causing those connections? | 08:41 |
| *** kmalloc has quit IRC | 08:41 | |
| oanson | I'm behind a company NAT, but we're about 4-5 people working on OpenStack at the moment | 08:41 |
| *** lukebrowning has joined #openstack-infra | 08:42 | |
| *** jpena|off is now known as jpena | 08:44 | |
| oanson | frickler, also, my colleague manages to connect behind the same NAT with her username | 08:44 |
| *** lukebrowning has quit IRC | 08:46 | |
| *** lukebrowning has joined #openstack-infra | 08:48 | |
| *** markmcd has joined #openstack-infra | 08:49 | |
| *** claudiub has joined #openstack-infra | 08:49 | |
| *** slaweq_ has joined #openstack-infra | 08:49 | |
| *** yamamoto has joined #openstack-infra | 08:50 | |
| *** pgadiya has quit IRC | 08:50 | |
| *** claudiub|2 has quit IRC | 08:52 | |
| *** lukebrowning has quit IRC | 08:53 | |
| *** stakeda has quit IRC | 08:53 | |
| *** yamamoto has quit IRC | 08:54 | |
| *** slaweq_ has quit IRC | 08:54 | |
| *** lukebrowning has joined #openstack-infra | 08:54 | |
| frickler | oanson: sadly I seem to be lacking permissions to actually view the connections, will have to wait for some other infra-root to help | 08:55 |
| *** yamamoto has joined #openstack-infra | 08:56 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** salv-orlando has joined #openstack-infra | 08:57 | |
| *** claudiub|2 has joined #openstack-infra | 08:58 | |
| oanson | frickler, I see. Thanks anyway! | 08:59 |
| *** lukebrowning has quit IRC | 08:59 | |
| *** lukebrowning has joined #openstack-infra | 09:00 | |
| *** claudiub has quit IRC | 09:01 | |
| *** sbezverk has quit IRC | 09:02 | |
| *** alexchadin has quit IRC | 09:04 | |
| *** pgadiya has joined #openstack-infra | 09:04 | |
| *** alexchadin has joined #openstack-infra | 09:04 | |
| *** yamamoto has joined #openstack-infra | 09:05 | |
| *** lukebrowning has quit IRC | 09:05 | |
| *** vivsoni has joined #openstack-infra | 09:06 | |
| *** lukebrowning has joined #openstack-infra | 09:07 | |
| *** lukebrowning has quit IRC | 09:12 | |
| *** rossella_s has joined #openstack-infra | 09:13 | |
| chandankumar | frickler: hello | 09:14 |
| chandankumar | frickler: please have a look at this review https://review.openstack.org/#/c/528672/ | 09:14 |
| *** owalsh_ is now known as owalsh | 09:17 | |
| AJaeger | config-core, please review https://review.openstack.org/529111 and https://review.openstack.org/#/c/528518/ to fix translations. | 09:18 |
| *** ralonsoh has quit IRC | 09:19 | |
| *** lukebrowning has joined #openstack-infra | 09:19 | |
| *** slaweq_ has joined #openstack-infra | 09:20 | |
| *** lukebrowning has quit IRC | 09:24 | |
| *** slaweq_ has quit IRC | 09:25 | |
| *** e0ne has joined #openstack-infra | 09:26 | |
| *** dsariel has joined #openstack-infra | 09:29 | |
| *** lukebrowning has joined #openstack-infra | 09:30 | |
| *** lucas-afk is now known as lucasagomes | 09:34 | |
| *** lukebrowning has quit IRC | 09:34 | |
| *** lukebrowning has joined #openstack-infra | 09:36 | |
| *** iyamahat has quit IRC | 09:40 | |
| *** lukebrowning has quit IRC | 09:41 | |
| *** lukebrowning has joined #openstack-infra | 09:42 | |
| *** derekh has joined #openstack-infra | 09:44 | |
| *** ralonsoh has joined #openstack-infra | 09:45 | |
| *** lukebrowning has quit IRC | 09:47 | |
| *** lukebrowning has joined #openstack-infra | 09:49 | |
| openstackgerrit | Masayuki Igawa proposed openstack/os-testr master: Fix to enable --color option of ostestr command https://review.openstack.org/529281 | 09:50 |
| *** lukebrowning has quit IRC | 09:53 | |
| oanson | Hello. | 09:53 |
| oanson | I get the following error when trying to ssh to review.openstack.org: Too many concurrent connections (64) - max. allowed: 64 | 09:53 |
| oanson | I don't have any open connections on my end. | 09:53 |
| oanson | Can someone please help? | 09:53 |
| openstackgerrit | Cao Xuan Hoang proposed openstack-infra/project-config master: Add neutron-vpnaas docs and release notes pub job to zuul layout https://review.openstack.org/522695 | 09:54 |
| *** claudiub has joined #openstack-infra | 09:54 | |
| oanson | AJaeger, chandankumar, do you have the permissions to clear out my (dead) connections? | 09:54 |
| *** lukebrowning has joined #openstack-infra | 09:55 | |
| *** claudiub|2 has quit IRC | 09:56 | |
| *** florianf has joined #openstack-infra | 09:58 | |
| *** lukebrowning has quit IRC | 10:00 | |
| *** psachin has quit IRC | 10:01 | |
| *** lukebrowning has joined #openstack-infra | 10:01 | |
| *** kjackal has joined #openstack-infra | 10:03 | |
| *** jascott1 has quit IRC | 10:05 | |
| *** jascott1 has joined #openstack-infra | 10:05 | |
| *** tosky has joined #openstack-infra | 10:05 | |
| *** lukebrowning has quit IRC | 10:06 | |
| *** liujiong has quit IRC | 10:06 | |
| dulek | Hi guys! Anyone have a clue why kuryr-kubernetes sphinx jobs started to fail yesterday with: | 10:06 |
| dulek | http://logs.openstack.org/92/529092/2/check/build-openstack-sphinx-docs/83eac6e/job-output.txt.gz#_2017-12-20_09_22_39_285523 | 10:06 |
| *** rossella_s has quit IRC | 10:06 | |
| *** cuongnv has quit IRC | 10:08 | |
| *** e0ne_ has joined #openstack-infra | 10:08 | |
| *** rossella_s has joined #openstack-infra | 10:09 | |
| ttx | AJaeger: the governance-uc, governance-website and governance-sigs likely need the same fix | 10:10 |
| *** psachin has joined #openstack-infra | 10:10 | |
| *** jascott1 has quit IRC | 10:10 | |
| AJaeger | oanson: no - you need to wait for the US to wake up... | 10:11 |
| *** e0ne has quit IRC | 10:11 | |
| AJaeger | ttx, looking at their tox.ini - I don't think so. Let me recheck a couple of changes to double check | 10:12 |
| *** Qiming has quit IRC | 10:14 | |
| *** Qiming has joined #openstack-infra | 10:14 | |
| ttx | AJaeger: re single patch to publish again... which one is it ? | 10:15 |
| *** jeblair has quit IRC | 10:15 | |
| *** jeblair has joined #openstack-infra | 10:16 | |
| oanson | AJaeger, thanks. Is there a specific person I will have to contact? | 10:17 |
| *** rpittau has joined #openstack-infra | 10:19 | |
| *** psachin has quit IRC | 10:20 | |
| *** shardy has quit IRC | 10:24 | |
| AJaeger | ttx, I mean: Your next change to merge will publish. So, if you need to udpate the webpage now, create a trivial change and merge ;) | 10:24 |
| *** lukebrowning has joined #openstack-infra | 10:24 | |
| AJaeger | oanson: just ask here again after 15:00 UTC | 10:25 |
| oanson | Sure. Thanks! | 10:25 |
| *** rossella_s has quit IRC | 10:26 | |
| AJaeger | ttx, governance-* don't use python3 - I rechecked one governance-uc change now to be sure | 10:27 |
| *** sambetts|afk is now known as sambetts | 10:27 | |
| AJaeger | dulek: see http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html | 10:27 |
| AJaeger | dulek: Let me check... | 10:27 |
| *** rossella_s has joined #openstack-infra | 10:27 | |
| dulek | AJaeger: I think I need to put neutron into doc/requirements.txt? | 10:29 |
| AJaeger | dulek: no - let me first understand your repo | 10:29 |
| *** lukebrowning has quit IRC | 10:29 | |
| AJaeger | dulek: are you kuryr core as well? Please help merging https://review.openstack.org/521026 | 10:30 |
| *** aarefiev has joined #openstack-infra | 10:30 | |
| *** Qiming has quit IRC | 10:30 | |
| dulek | AJaeger: Not a core, but I can raise that on the channel. | 10:31 |
| AJaeger | dulek: thanks! | 10:31 |
| AJaeger | dulek: please do - you need that one to fix releasenotes build - a related problem to yours here | 10:31 |
| *** Qiming has joined #openstack-infra | 10:36 | |
| AJaeger | dulek: what's your channel? Let me join... | 10:36 |
| dulek | AJaeger: #openstack-kuryr. | 10:37 |
| dulek | Without dot. ;) | 10:37 |
| *** anupn has quit IRC | 10:38 | |
| *** aviau has quit IRC | 10:39 | |
| ttx | AJaeger: I'll find one to approve. Thanks! | 10:40 |
| *** aviau has joined #openstack-infra | 10:40 | |
| AJaeger | ;) | 10:40 |
| *** alexchadin has quit IRC | 10:41 | |
| *** alexchadin has joined #openstack-infra | 10:42 | |
| *** gongysh has quit IRC | 10:42 | |
| *** anupn has joined #openstack-infra | 10:46 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Set remote url on every getRepo in merger https://review.openstack.org/529293 | 10:48 |
| *** ldnunes has joined #openstack-infra | 10:53 | |
| *** annp has quit IRC | 10:53 | |
| *** krtaylor has quit IRC | 10:53 | |
| *** hashar is now known as hasharAway | 10:54 | |
| *** edmondsw has joined #openstack-infra | 10:57 | |
| *** cshastri has quit IRC | 10:57 | |
| *** edmondsw has quit IRC | 11:01 | |
| *** dtantsur|afk is now known as dtantsur | 11:02 | |
| *** yolanda__ has joined #openstack-infra | 11:05 | |
| *** lukebrowning has joined #openstack-infra | 11:05 | |
| *** krtaylor has joined #openstack-infra | 11:06 | |
| *** rossella_s has quit IRC | 11:06 | |
| *** rossella_s has joined #openstack-infra | 11:07 | |
| *** yamamoto_ has joined #openstack-infra | 11:08 | |
| *** yolanda has quit IRC | 11:08 | |
| *** markmcd has quit IRC | 11:08 | |
| AJaeger | infra-root, I got some gerrit timeouts. Is it time to restart gerrit? | 11:10 |
| *** cshastri has joined #openstack-infra | 11:10 | |
| *** lukebrowning has quit IRC | 11:10 | |
| *** markmcd has joined #openstack-infra | 11:10 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** gcb has quit IRC | 11:12 | |
| frickler | AJaeger: looks to me like it has been stable for an astonishinly long time already: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 11:12 |
| frickler | I'd suggest to wait a couple of minutes to see if it recovers | 11:13 |
| AJaeger | frickler: I have problems since an hour or two and just heard others asking... | 11:14 |
| AJaeger | frickler: but yeah, we can wait a bit... | 11:14 |
| *** aviau has quit IRC | 11:15 | |
| *** lukebrowning has joined #openstack-infra | 11:16 | |
| *** aviau has joined #openstack-infra | 11:17 | |
| odyssey4me | argh, gerrit's slow and timing out from time to time | 11:20 |
| *** lukebrowning has quit IRC | 11:21 | |
| *** lukebrowning has joined #openstack-infra | 11:23 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul feature/zuulv3: Add run_ansible_setup_on_start flag for executor https://review.openstack.org/529300 | 11:24 |
| *** priteau has joined #openstack-infra | 11:24 | |
| *** lukebrowning has quit IRC | 11:27 | |
| *** cshastri has quit IRC | 11:29 | |
| *** rossella_s has quit IRC | 11:29 | |
| *** lukebrowning has joined #openstack-infra | 11:29 | |
| jamespage | hmm I'm seeing gerrit slowness and timing out as well | 11:30 |
| *** andreas__ has joined #openstack-infra | 11:30 | |
| *** vivsoni has quit IRC | 11:31 | |
| *** neiljerram has joined #openstack-infra | 11:31 | |
| *** andreas__ has quit IRC | 11:31 | |
| neiljerram | Morning all! | 11:32 |
| *** rossella_s has joined #openstack-infra | 11:32 | |
| jamespage | hi neiljerram | 11:32 |
| sshnaidm | see gerrit problems too | 11:33 |
| *** andreas_s has quit IRC | 11:33 | |
| neiljerram | A couple days ago, I got a notification about "Removed reviewer Jenkins with the following votes" from several of my standing changes in Gerrit. I'm not sure what that means - does someone here know? | 11:33 |
| *** alexchadin has quit IRC | 11:33 | |
| neiljerram | jamespage, hi | 11:34 |
| *** lukebrowning has quit IRC | 11:34 | |
| cmurphy | neiljerram: see http://lists.openstack.org/pipermail/openstack-dev/2017-December/125684.html nothing to worry about | 11:35 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul feature/zuulv3: Add run_ansible_setup_on_start flag for executor https://review.openstack.org/529300 | 11:35 |
| *** lukebrowning has joined #openstack-infra | 11:35 | |
| neiljerram | cmurphy, many thanks! | 11:35 |
| *** szaher has quit IRC | 11:36 | |
| *** markmcd has quit IRC | 11:36 | |
| *** alexchadin has joined #openstack-infra | 11:39 | |
| *** psachin has joined #openstack-infra | 11:39 | |
| *** lukebrowning has quit IRC | 11:40 | |
| *** lukebrowning has joined #openstack-infra | 11:41 | |
| *** lukebrowning has quit IRC | 11:46 | |
| *** lukebrowning has joined #openstack-infra | 11:48 | |
| *** pbourke has quit IRC | 11:48 | |
| *** alexchadin has quit IRC | 11:49 | |
| *** pbourke has joined #openstack-infra | 11:49 | |
| *** lukebrowning has quit IRC | 11:53 | |
| *** lukebrowning has joined #openstack-infra | 11:54 | |
| *** rossella_s has quit IRC | 11:55 | |
| *** rossella_s has joined #openstack-infra | 11:58 | |
| *** lukebrowning has quit IRC | 11:59 | |
| *** salv-orl_ has joined #openstack-infra | 11:59 | |
| *** lukebrowning has joined #openstack-infra | 12:00 | |
| *** smatzek has joined #openstack-infra | 12:01 | |
| *** dizquierdo has joined #openstack-infra | 12:02 | |
| *** priteau has quit IRC | 12:03 | |
| *** salv-orlando has quit IRC | 12:03 | |
| *** priteau has joined #openstack-infra | 12:03 | |
| *** claudiub has quit IRC | 12:03 | |
| *** rossella_s has quit IRC | 12:04 | |
| *** lukebrowning has quit IRC | 12:05 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul feature/zuulv3: Add run_ansible_setup_on_start flag for executor https://review.openstack.org/529300 | 12:06 |
| *** rossella_s has joined #openstack-infra | 12:06 | |
| *** jascott1 has joined #openstack-infra | 12:06 | |
| *** andreas_s has joined #openstack-infra | 12:06 | |
| *** lukebrowning has joined #openstack-infra | 12:07 | |
| *** priteau has quit IRC | 12:08 | |
| *** andreas_s has quit IRC | 12:11 | |
| *** jascott1 has quit IRC | 12:11 | |
| *** lukebrowning has quit IRC | 12:11 | |
| *** andreas_s has joined #openstack-infra | 12:11 | |
| *** dave-mccowan has joined #openstack-infra | 12:14 | |
| *** alexchadin has joined #openstack-infra | 12:15 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul feature/zuulv3: Add run_ansible_setup_on_start flag for executor https://review.openstack.org/529300 | 12:16 |
| tobiash | SpamapS, jeblair, mordred: change 527298 (drop local fork of gitpython for 2.1.8 release) breaks my local test suite | 12:18 |
| tobiash | didn't find out yet why | 12:18 |
| tobiash | ups, zuul channel would be better for that | 12:21 |
| *** lucasagomes is now known as lucas-hungry | 12:26 | |
| *** rossella_s has quit IRC | 12:33 | |
| *** aarefiev has quit IRC | 12:33 | |
| *** rossella_s has joined #openstack-infra | 12:35 | |
| *** armaan has joined #openstack-infra | 12:36 | |
| *** gcb has joined #openstack-infra | 12:39 | |
| *** cdent has joined #openstack-infra | 12:41 | |
| cdent | gerrit unhappy, or is it just meâ„¢ | 12:41 |
| tobiash | SpamapS, jeblair, mordred: oh, gitpython is now processing my globally defined gerrit commit message hook which breaks string matching in the tests | 12:43 |
| tobiash | removed that locally made the tests run again :) | 12:43 |
| tobiash | cdent: I also experienced slow requests to gerrit | 12:43 |
| cdent | I was almost hoping it was just me. | 12:43 |
| *** liuyulong has joined #openstack-infra | 12:44 | |
| *** lukebrowning has joined #openstack-infra | 12:45 | |
| *** edmondsw has joined #openstack-infra | 12:45 | |
| *** armaan has quit IRC | 12:45 | |
| *** bobh has joined #openstack-infra | 12:45 | |
| tobiash | it runs normally for me nwo | 12:46 |
| *** felipemonteiro has joined #openstack-infra | 12:46 | |
| tobiash | hrm, I thought that, but now experienced another almost timeout | 12:46 |
| *** gcb has quit IRC | 12:47 | |
| *** lukebrowning has quit IRC | 12:49 | |
| *** edmondsw has quit IRC | 12:49 | |
| *** felipemonteiro has quit IRC | 12:50 | |
| *** kiennt26 has joined #openstack-infra | 12:53 | |
| *** pgadiya has quit IRC | 12:53 | |
| *** armaan has joined #openstack-infra | 12:53 | |
| *** yolanda__ is now known as yolanda | 12:53 | |
| *** lukebrowning has joined #openstack-infra | 12:55 | |
| *** rossella_s has quit IRC | 12:57 | |
| *** jpena is now known as jpena|lunch | 12:58 | |
| *** lukebrowning has quit IRC | 13:00 | |
| *** lukebrowning has joined #openstack-infra | 13:01 | |
| mordred | tobiash: when you say globally defined gerrit commit message hook, that's defined in your homedir right? we could add self.useFixture(fixtures.TempHomeDir()) to ZuulTestCase.setUp and that *should* prevent such leakage | 13:02 |
| tobiash | mordred: yes | 13:02 |
| tobiash | mordred: I'll check that | 13:03 |
| tobiash | quick fix was to delete the hook (as it will be invalid with github anyway) | 13:03 |
| mordred | good point :) | 13:03 |
| *** rossella_s has joined #openstack-infra | 13:03 | |
| *** dmellado has quit IRC | 13:05 | |
| *** coolsvap has quit IRC | 13:05 | |
| *** lukebrowning has quit IRC | 13:06 | |
| frickler | mordred: I think gerrit is ripe for a restart by now, can you confirm that I only need to do a "service gerrit stop/start" on review.o.o? | 13:07 |
| ttx | slow | 13:07 |
| oanson | Hello. | 13:07 |
| oanson | I get the following error when trying to ssh to review.openstack.org: Too many concurrent connections (64) - max. allowed: 64 | 13:07 |
| oanson | I don't have any open connections on my end. | 13:07 |
| oanson | Can someone please help? | 13:07 |
| *** lukebrowning has joined #openstack-infra | 13:07 | |
| *** dprince has joined #openstack-infra | 13:08 | |
| ttx | oanson: yes, known issue, restart being planned | 13:08 |
| oanson | ttx, great. Thanks! | 13:08 |
| liuyulong | review.o.o: get errors: "Reason: Error reading from remote server, Apache/2.4.7 (Ubuntu) Server at review.openstack.org Port 443 | 13:08 |
| mnaser | i think it would be good to #status or #notice the gerrit issues | 13:10 |
| frickler | infra-root: seems gerrit is getting unuseable by now, I'm going to perform the stop/start | 13:10 |
| *** e0ne has joined #openstack-infra | 13:10 | |
| mnaser | frickler: i would "#status notice <heads up>" | 13:11 |
| mnaser | https://docs.openstack.org/infra/system-config/irc.html#statusbot | 13:12 |
| *** lukebrowning has quit IRC | 13:12 | |
| jianghuaw_ | gerrit is down now? suddenly got error: "503 Service Unavailable" | 13:12 |
| ttx | jianghuaw_: restart in progress | 13:12 |
| frickler | #status notice gerrit is being restarted due to extreme slowness | 13:12 |
| openstackstatus | frickler: sending notice | 13:12 |
| jianghuaw_ | ttx, got it. thanks. | 13:13 |
| *** e0ne_ has quit IRC | 13:13 | |
| *** openstackgerrit has quit IRC | 13:13 | |
| -openstackstatus- NOTICE: gerrit is being restarted due to extreme slowness | 13:14 | |
| mnaser | frickler: from 2015 it seems that service gerrit restart does the trick | 13:14 |
| *** dmellado has joined #openstack-infra | 13:15 | |
| frickler | mnaser: thx, it said "Starting Gerrit Code Review: FAILED" but seems to have succeeded a bit later anyhow | 13:15 |
| mnaser | frickler: hmm, i wonder why it would fail (maybe stale pid file?) | 13:15 |
| openstackstatus | frickler: finished sending notice | 13:15 |
| *** amoralej is now known as amoralej|lunch | 13:16 | |
| mnaser | frickler: im not infra root so i cant dive in but at least we have it back up, but probably good to check with other infra roots when they wake up to double heck | 13:16 |
| *** stevebaker has quit IRC | 13:18 | |
| *** lukebrowning has joined #openstack-infra | 13:18 | |
| *** dmellado has quit IRC | 13:19 | |
| *** dmellado has joined #openstack-infra | 13:21 | |
| frickler | oanson: the restart should have solved your issue implicitly for now, please let us know if/when it repeats | 13:21 |
| oanson | frickler, thanks. Appears to work. Will do. | 13:22 |
| *** lukebrowning has quit IRC | 13:23 | |
| *** rlandy has joined #openstack-infra | 13:23 | |
| *** rlandy is now known as rlandy|ruck | 13:24 | |
| *** lukebrowning has joined #openstack-infra | 13:25 | |
| AJaeger | mordred: fun debugging of the day: kuryr-kubernetes had a tools/tox_install.sh that was not used at all - now the updated jobs used it and failed ;) | 13:26 |
| *** openstackgerrit has joined #openstack-infra | 13:27 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron-vpnaas/vpnaas-dashboard to gerritbot https://review.openstack.org/527867 | 13:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Drop django_openstack_auth master branch jobs https://review.openstack.org/529080 | 13:27 |
| *** alexchadin has quit IRC | 13:27 | |
| *** lukebrowning_ has joined #openstack-infra | 13:28 | |
| *** stevebaker has joined #openstack-infra | 13:28 | |
| *** lucas-hungry is now known as lucasagomes | 13:28 | |
| *** lukebrowning has quit IRC | 13:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use openstack-python-jobs for tacker-horizon https://review.openstack.org/528503 | 13:31 |
| *** bobh has quit IRC | 13:31 | |
| *** lukebrowning_ has quit IRC | 13:33 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove obsolete dashboard tox jobs https://review.openstack.org/528581 | 13:33 |
| mordred | AJaeger: hahahahahahahahaha | 13:34 |
| *** lukebrowning has joined #openstack-infra | 13:34 | |
| *** yamamoto_ has quit IRC | 13:34 | |
| AJaeger | mordred: another problem: RHEL and CentOS7 use rsync 3.0 apparently that does not support "--ignore-missing-args", see http://logs.openstack.org/30/529230/2/experimental/swift-tox-func-centos-7/f5ab6e1/job-output.txt.gz#_2017-12-20_11_11_35_500685 | 13:36 |
| AJaeger | wonder why we didn't notice that elsewhere... | 13:36 |
| AJaeger | mordred: so, this breaks fetch-stestr-output and gives a post job failure | 13:37 |
| *** armaan has quit IRC | 13:38 | |
| *** cdent has left #openstack-infra | 13:38 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Rework translation templates https://review.openstack.org/529111 | 13:38 |
| *** lukebrowning has quit IRC | 13:39 | |
| *** lukebrowning has joined #openstack-infra | 13:40 | |
| *** rhallisey has quit IRC | 13:40 | |
| mordred | AJaeger: oh - well, that may also explain another issuetoo - andreykurilin was having an issue with synchronize: not supporting --ignore-missing-args and I was confused | 13:41 |
| AJaeger | if it was on centos7... | 13:42 |
| andreykurilin | I suppose it was centos7 | 13:42 |
| AJaeger | infra-root, we just merged 528581 but it's not in http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/ | 13:43 |
| AJaeger | anything broken with gerrit/git? | 13:43 |
| *** jcoufal has joined #openstack-infra | 13:43 | |
| *** rhallisey has joined #openstack-infra | 13:43 | |
| *** lukebrowning has quit IRC | 13:45 | |
| *** gongysh has joined #openstack-infra | 13:45 | |
| mordred | AJaeger: looking in the logs, I think gerrit is still working on starting replication | 13:46 |
| *** kiennt26 has quit IRC | 13:46 | |
| *** lukebrowning has joined #openstack-infra | 13:46 | |
| AJaeger | interesting. Looking at gerrit showqueue, I see "18be545f waiting .... 13:14:21.963 git01/stackforge/graffiti" - so, many stackforge repos ;( | 13:47 |
| AJaeger | can't we disable that somehow? | 13:47 |
| AJaeger | thanks, mordred | 13:47 |
| AJaeger | queue is getting shorter - I'll wait some more. | 13:47 |
| AJaeger | mordred: should I open a story for the rsync --ignore-missing-args? | 13:48 |
| mordred | AJaeger: yah, I think so - we use it in a few places, so solving it is going to be important | 13:48 |
| *** trown|outtypewww is now known as trown | 13:48 | |
| dmsimard | AJaeger, mordred: I remember approving a patch about missing args, what about it ? | 13:49 |
| *** gongysh has quit IRC | 13:49 | |
| mordred | dmsimard: apparently it's not supported on centos/rhel | 13:49 |
| *** psachin has quit IRC | 13:49 | |
| *** eharney has joined #openstack-infra | 13:49 | |
| dmsimard | that's right so the assumption was that a particular role needed to run exclusively on ubuntu/debian which wasn't necessary if only for that argument | 13:49 |
| * dmsimard searches patch | 13:49 | |
| *** amoralej|lunch is now known as amoralej | 13:50 | |
| *** goutham has joined #openstack-infra | 13:50 | |
| AJaeger | dmsimard: that was for translations - but we talk about unittests here | 13:50 |
| AJaeger | config-core, please review https://review.openstack.org/#/c/528518/ to fix translations. And a refactor: https://review.openstack.org/#/c/529047/ | 13:51 |
| *** lukebrowning has quit IRC | 13:51 | |
| dmsimard | AJaeger, mordred: https://review.openstack.org/#/c/525510/ | 13:51 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Remove publish-openstack-python-branch-tarball from steth https://review.openstack.org/529317 | 13:51 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Added check to ensure template usage https://review.openstack.org/529318 | 13:51 |
| *** rlandy|ruck is now known as rlandy|rover | 13:52 | |
| AJaeger | dmsimard: ah, so we need to do this in the other role as well... | 13:52 |
| *** mriedem1 has joined #openstack-infra | 13:52 | |
| * AJaeger prepares a change | 13:52 | |
| dmsimard | It's okay that OpenStack "only" runs tox on Ubuntu but tox also runs on other distros you know :P | 13:52 |
| *** catintheroof has joined #openstack-infra | 13:52 | |
| *** trown is now known as trown|ruck | 13:52 | |
| mordred | dmsimard: :) | 13:53 |
| dmsimard | so what's the fix then ? | 13:53 |
| dmsimard | I'm not super familiar with that arg, I had to look it up | 13:53 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Set remote url on every getRepo in merger https://review.openstack.org/529293 | 13:53 |
| *** lukebrowning has joined #openstack-infra | 13:53 | |
| mordred | dunno. we're going to have to get more cleverer in constructing synchronize commands | 13:54 |
| dmsimard | synchronize is one of those modules which kind of suck tbh :( | 13:54 |
| mordred | yah- but in this case it's actually just an rsync issue anyway | 13:54 |
| *** jpena|lunch is now known as jpena | 13:54 | |
| mordred | we've only got two uses of it in zuul-jobs ... | 13:55 |
| dmsimard | just being able to understand which direction the transfer is going to be in (pull or push, remember it's on a remote target) requires my brain to make efforts. | 13:55 |
| *** mriedem has quit IRC | 13:55 | |
| mordred | totes | 13:55 |
| *** claudiub has joined #openstack-infra | 13:55 | |
| tobiash | dmsimard: better (faster) than copy ;) | 13:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Do not use --ignore-missing-args for rsync https://review.openstack.org/529320 | 13:56 |
| tobiash | dmsimard: but totally agree... | 13:56 |
| AJaeger | mordred, dmsimard, tobiash: Fixed ^ | 13:56 |
| dmsimard | tobiash: yeah but copy doesn't scale if you have a large number of files | 13:56 |
| *** mriedem1 is now known as mriedem | 13:56 | |
| tobiash | dmsimard: yes, and large begins at 50... | 13:56 |
| mordred | AJaeger: yes. ++ | 13:56 |
| tobiash | that makes copy almost useless | 13:56 |
| AJaeger | those were the only two occurences in our config repos | 13:57 |
| dmsimard | AJaeger: the impact of removing --ignore-missing-args is still not 100% clear to me. Is there a use case that fails if we remove that ? | 13:57 |
| mordred | dmsimard: it's basically to allow us to just have the sync command for the files that might be there without having to stat for them first | 13:57 |
| AJaeger | dmsimard: and I set "ignore: errors" instead, so that should result in same behaviour | 13:57 |
| dmsimard | oh, so rsync foo bar wouldn't fail if foo didn't exist | 13:57 |
| mordred | yah | 13:57 |
| dmsimard | AJaeger: either that or failed_when: false | 13:58 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config master: Add noop job for django_openstack_auth master branch https://review.openstack.org/529321 | 13:58 |
| AJaeger | dmsimard: copied the construct from the patch you showed me ;) | 13:58 |
| dmsimard | failed_when and ignore_errors are slightly different | 13:58 |
| *** lukebrowning has quit IRC | 13:58 | |
| AJaeger | do we need failed_when here? I don't see a use case yet | 13:59 |
| tobiash | AJaeger: depends on what we want | 13:59 |
| AJaeger | ;) | 14:00 |
| tobiash | ignore_errors still shows the task as failed | 14:00 |
| *** lukebrowning has joined #openstack-infra | 14:00 | |
| mordred | ah - so we probably want failed_when: false | 14:00 |
| tobiash | where failed_when defined when the task really is failed | 14:00 |
| dmsimard | the difference is that if you put "failed_when: false", the task will never fail whereas ignore_errors: yes will properly have the task fail but won't fail the playbook | 14:00 |
| AJaeger | so, what do we want, ansible experts? | 14:00 |
| AJaeger | I can updated my change and also change the place in 525510... | 14:00 |
| dmsimard | ¯\_(ツ)_/¯ | 14:01 |
| tobiash | I like playbooks to not include any *justified* failures | 14:01 |
| dmsimard | fwiw ara will properly display those tasks as "IGNORED" if there is a failure | 14:01 |
| tobiash | arg the other way round, I like playbooks to only include justified failures | 14:02 |
| tobiash | ;) | 14:02 |
| dmsimard | I don't think we want a task failure here if the expectation is that it is going to fail semi regularly | 14:02 |
| mordred | SO ... looking at these roles, this makes me want to rework both of them slightly | 14:02 |
| AJaeger | mordred: do you want to take over my change - or shall I do the change first? | 14:02 |
| tobiash | dmsimard: ++ | 14:02 |
| mordred | AJaeger: go ahead with yours, then I'll do mine on top of it | 14:02 |
| *** catintheroof has quit IRC | 14:02 | |
| mordred | AJaeger: mine might take slightly longer, and yours should be good enough for now | 14:03 |
| dmsimard | AJaeger: feel free to change that other one I linked to | 14:03 |
| dmsimard | s/to/too/ | 14:03 |
| *** catintheroof has joined #openstack-infra | 14:04 | |
| *** jamesmcarthur has joined #openstack-infra | 14:04 | |
| AJaeger | dmsimard: already done, just pushing ... | 14:05 |
| *** lukebrowning has quit IRC | 14:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Do not use --ignore-missing-args for rsync https://review.openstack.org/529320 | 14:05 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Set remote url on every getRepo in merger https://review.openstack.org/529293 | 14:05 |
| AJaeger | mordred, dmsimard, could you review https://review.openstack.org/#/c/528518 and https://review.openstack.org/#/c/529047/ as well, please? | 14:05 |
| *** lukebrowning has joined #openstack-infra | 14:06 | |
| *** jascott1 has joined #openstack-infra | 14:08 | |
| AJaeger | mordred: i think we should continue with updated-pti. Let's ask for reviews on https://review.openstack.org/#/c/525735/ and https://review.openstack.org/#/c/525718/ (Zuul) | 14:08 |
| AJaeger | config-core ^ | 14:08 |
| *** logan- has quit IRC | 14:08 | |
| *** logan- has joined #openstack-infra | 14:10 | |
| *** yamamoto has joined #openstack-infra | 14:10 | |
| *** lukebrowning has quit IRC | 14:11 | |
| *** kgiusti has joined #openstack-infra | 14:12 | |
| *** andreas_s has quit IRC | 14:12 | |
| *** lukebrowning has joined #openstack-infra | 14:12 | |
| *** jascott1 has quit IRC | 14:13 | |
| AJaeger | config-core, openstack-zuul-jobs job removal change for review, please https://review.openstack.org/#/c/522471/ | 14:15 |
| *** andreas_s has joined #openstack-infra | 14:17 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy novaclient jobs https://review.openstack.org/522102 | 14:17 |
| *** lukebrowning has quit IRC | 14:17 | |
| *** lukebrowning has joined #openstack-infra | 14:19 | |
| *** catinthe_ has joined #openstack-infra | 14:22 | |
| *** yamamoto has quit IRC | 14:23 | |
| *** lukebrowning has quit IRC | 14:24 | |
| *** lukebrowning has joined #openstack-infra | 14:25 | |
| *** catintheroof has quit IRC | 14:26 | |
| *** andreas_s has quit IRC | 14:26 | |
| *** smatzek has quit IRC | 14:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in storlets https://review.openstack.org/512958 | 14:28 |
| *** smatzek has joined #openstack-infra | 14:29 | |
| *** smatzek has quit IRC | 14:29 | |
| *** lukebrowning has quit IRC | 14:30 | |
| *** lukebrowning has joined #openstack-infra | 14:31 | |
| *** andreas_s has joined #openstack-infra | 14:31 | |
| *** openstackgerrit has quit IRC | 14:33 | |
| *** goutham has left #openstack-infra | 14:33 | |
| *** openstackgerrit has joined #openstack-infra | 14:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in Murano https://review.openstack.org/511438 | 14:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add consolidated role for processing subunit https://review.openstack.org/529339 | 14:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove testr and stestr specific roles https://review.openstack.org/529340 | 14:34 |
| *** lukebrowning has quit IRC | 14:36 | |
| *** lukebrowning has joined #openstack-infra | 14:38 | |
| *** trown|ruck is now known as trown|brb | 14:39 | |
| *** shardy has joined #openstack-infra | 14:41 | |
| mordred | AJaeger, dmsimard: ^^ | 14:41 |
| *** andreas_s has quit IRC | 14:41 | |
| *** lukebrowning has quit IRC | 14:42 | |
| AJaeger | mordred: "should do the right thing"? Should we test it with a DNM change? | 14:42 |
| *** ykarel has quit IRC | 14:43 | |
| *** andreas_s has joined #openstack-infra | 14:45 | |
| AJaeger | mordred: nice! | 14:46 |
| *** trown|brb is now known as trown | 14:47 | |
| *** sshnaidm has quit IRC | 14:50 | |
| *** lukebrowning has joined #openstack-infra | 14:50 | |
| *** sshnaidm has joined #openstack-infra | 14:50 | |
| mordred | AJaeger: well ... it should be at least partially self-testing - since we run zuul unit tests on zuul-jobs changes | 14:51 |
| AJaeger | Ah, good! | 14:51 |
| *** trown is now known as trown|ruck | 14:52 | |
| mordred | AJaeger: and also the tox-linters job on zuul-jobs should get us testing of running the role when there is no testr output | 14:52 |
| AJaeger | how nice for a change ;) | 14:53 |
| AJaeger | mordred: you shouldn't have said "should do the right thing" - that triggered something... | 14:53 |
| mordred | AJaeger: hehe. well, out of habit, I *never* want to claim anything will *actually* work | 14:55 |
| *** lukebrowning has quit IRC | 14:55 | |
| sambetts | Hi, I'm seeing a failure on my "build-openstack-sphinx-docs" job here: http://logs.openstack.org/21/529121/2/check/build-openstack-sphinx-docs/329628e/job-output.txt.gz#_2017-12-20_13_47_21_082540 and on other patches, it seems to be related to https://github.com/openstack-infra/openstack-zuul-jobs/commit/e1a793b8a1f093c24264c4d0c19480607c4e1c83 | 14:55 |
| AJaeger | yeah, made that mistake myself often enough but I'm not that careful yet | 14:55 |
| AJaeger | sambetts: did you see mordred's email on openstack-dev? | 14:56 |
| sambetts | AJaeger: ah no I must have missed it | 14:56 |
| *** lukebrowning has joined #openstack-infra | 14:57 | |
| AJaeger | sambetts: merge https://review.openstack.org/522315 t ofix it - waiting since a few weeks for your review ;( | 14:57 |
| AJaeger | wait, it fails - mordred ^ | 14:57 |
| *** mriedem is now known as mriedem_away | 14:57 | |
| AJaeger | sambetts: but that's the first step in fixing your problem | 14:58 |
| AJaeger | sambetts: http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html | 14:58 |
| sambetts | ah ok... got it, I wondered if that was the case, I left a recheck on it this morning to try to push it thourhg | 14:59 |
| AJaeger | mordred, sambetts : http://logs.openstack.org/15/522315/3/check/build-openstack-sphinx-docs/c7bd264/job-output.txt.gz#_2017-12-20_11_46_13_258957 - import problem with neutron? | 14:59 |
| andreaf | mordred: about fetch-subunit-output / https://review.openstack.org/#/c/529339/1 - did you consider the process-test-results role https://github.com/openstack-infra/zuul-jobs/tree/master/roles/process-test-results ? | 14:59 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Do not use --ignore-missing-args for rsync https://review.openstack.org/529320 | 15:00 |
| *** bobh has joined #openstack-infra | 15:01 | |
| *** lukebrowning has quit IRC | 15:01 | |
| sambetts | AJaeger: thats failing because we're missing an "override-checkout: stable/pike" on that job in zuul.d/project.yaml | 15:01 |
| *** catinthe_ has quit IRC | 15:02 | |
| *** ykarel has joined #openstack-infra | 15:02 | |
| AJaeger | sambetts, is that an official project that has a post job? | 15:02 |
| *** catintheroof has joined #openstack-infra | 15:03 | |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Specify and load event schedule https://review.openstack.org/525168 | 15:03 |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Add scheduled / additional rooms https://review.openstack.org/529345 | 15:03 |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Display scheduled and additional rooms in web page https://review.openstack.org/529346 | 15:03 |
| *** lukebrowning has joined #openstack-infra | 15:03 | |
| AJaeger | sambetts: it's not - so, could you send a patch for project-config, please? | 15:03 |
| sambetts | AJaeger: yeah, writing it nowe | 15:03 |
| mordred | andreaf: no! I didn't - lemme rework that | 15:04 |
| sambetts | AJaeger: before the change our tox_install.sh logic and tox.ini set the right branch for that job, now it doesn't use tox.ini the branch is wrong :( | 15:04 |
| mordred | andreaf: also - while I've got you ... I'm about to start pushing on the base job rework to make the log fetching/publishing stuff more better | 15:04 |
| mordred | andreaf: re: your comment: https://review.openstack.org/#/c/511843/2/roles/fetch-output/tasks/main.yaml@9 ... | 15:05 |
| mordred | andreaf: my concern is that for simple jobs, like tox, we wind up with an extra directory that's never useful | 15:06 |
| mordred | andreaf: what if we just had that fetch role always just do the sync of the logs dir and never do anything with adding hostnames | 15:06 |
| *** marst has joined #openstack-infra | 15:06 | |
| mordred | andreaf: and if jobs are written that are multinode, they can write logs to {{ zuul_output_dir }}/logs/{{ inventory_hostname }} | 15:07 |
| *** makowals has quit IRC | 15:08 | |
| *** catintheroof has quit IRC | 15:08 | |
| *** lukebrowning has quit IRC | 15:08 | |
| mordred | andreaf: that way the behavior can be tied to a type of job content (and job content such as the devstack base job can just always write to the multi-node format) | 15:09 |
| sambetts | AJaeger: does the sphinx docs job continue to call tox_install.sh? | 15:09 |
| AJaeger | sambetts: if it exists - it does. | 15:09 |
| *** lukebrowning has joined #openstack-infra | 15:09 | |
| AJaeger | sambetts: but with *specific* parameters. That's why you need mordred's change | 15:09 |
| sambetts | AJaeger: ok, I think I have a better idea of how to fix it instead of adding more checkout overrides to project-config, I'm going to change the default branch in our tox_install.sh to "stable/pike" | 15:10 |
| andreaf | mordred: uhm... yeah I guess it makes sense, so you always have any(logs / docs / artifacts) in the root | 15:10 |
| andreaf | mordred: and under logs you can have multiple folders, one per host, if applicable | 15:11 |
| sambetts | so then when tox_install.sh is called it will install the stable/pike version of neutron as we expected for that job, instead of installing master | 15:11 |
| andreaf | docs and artifacts won't ever be multinode I guess | 15:11 |
| *** shardy has quit IRC | 15:11 | |
| mordred | andreaf: yah - I mean, if you do a multinode docs job, good luck to you :) | 15:11 |
| AJaeger | sambetts: ok, give it a try ;) | 15:11 |
| andreaf | :D | 15:11 |
| andreaf | mordred: a log file that is indexed in elastic search might need one extra path - one for the single node case, one for multinode controller case | 15:13 |
| andreaf | mordred: and potentially one for the subnode as well but that's going to be the case anyways | 15:13 |
| *** lukebrowning has quit IRC | 15:13 | |
| *** liuyulong has quit IRC | 15:14 | |
| *** rbrndt has joined #openstack-infra | 15:15 | |
| *** sshnaidm is now known as sshnaidm|afk | 15:16 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: Attempt to improve tenant config loading in case of config issue https://review.openstack.org/529060 | 15:19 |
| *** esberglu has joined #openstack-infra | 15:20 | |
| *** myoung is now known as myoung|bbl | 15:20 | |
| *** Swimingly has quit IRC | 15:21 | |
| *** yamamoto has joined #openstack-infra | 15:23 | |
| *** Swimingly has joined #openstack-infra | 15:24 | |
| *** shardy has joined #openstack-infra | 15:24 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/openstack-zuul-jobs master: Use tempest-full in the integrated gate https://review.openstack.org/529353 | 15:24 |
| andreaf | AJaeger, mordred: what do you think? ^^^ | 15:24 |
| *** Apoorva has joined #openstack-infra | 15:29 | |
| mordred | andreaf: I think it's a great idea - maybe let's move the integrated-gate project-template into the normal project-templates.yaml file since it's a template we're aiming to keep? | 15:29 |
| mordred | andreaf: although, ignore me - let's do that as a different patch so it's clear what this one does :) | 15:30 |
| mordred | andreaf: in any case, I'm +2 on that change if you're satisfied that the new job is ready - I assume we have added elasticsearch paths for the controller/ path already? | 15:32 |
| *** yamamoto has quit IRC | 15:35 | |
| andreaf | mordred: I have to check the e-r config for that | 15:36 |
| frickler | infra-root: we have quite a bit of backlog in the integrated gate, looks like there is a significant amount of jobs running into timeouts, if anyone could take a look | 15:38 |
| jeblair | frickler: is that something a root needs to do? | 15:39 |
| andreaf | mordred: according to http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml subfolders are searched automatically already | 15:39 |
| AJaeger | andreaf: tell us once you're ready to merge it... | 15:39 |
| fungi | i only see two failing changes in the integrated gate at the moment | 15:40 |
| jeblair | frickler: jobs running into timeouts should be something anyone can investigate i'd think | 15:40 |
| *** lukebrowning has joined #openstack-infra | 15:41 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Remove github caching https://review.openstack.org/529361 | 15:42 |
| andreaf | AJaeger, mordred: the job is ready from my pov, one thing that I'm wondering now is what is the impact in terms of files / irrelevant-files pov | 15:42 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul feature/zuulv3: Use connection type supplied from nodepool https://review.openstack.org/501976 | 15:42 |
| AJaeger | andreaf: you can iterate on that in tempest ;) | 15:42 |
| *** slaweq has quit IRC | 15:43 | |
| AJaeger | frickler: looking at http://logs.openstack.org/30/528630/1/gate/openstack-tox-py35/df0d181/job-output.txt.gz I see so many errors in the log file, that I wonder it didn't fail earlier | 15:43 |
| andreaf | AJaeger: if any projects has such settings on the legacy job today, they won't apply until there's changed to the new job name... but I guess it should be fine to fix that iteratively | 15:44 |
| *** armax has joined #openstack-infra | 15:44 | |
| fungi | frickler: that failed job which kicked out the change currently at the top of the gate seems to have errored out on a tempest test. what timeouts are you finding? | 15:44 |
| andreaf | AJaeger: so I think the patch is ready to be merged | 15:44 |
| AJaeger | andreaf: agreed, you can iterate on it. | 15:44 |
| *** iyamahat has joined #openstack-infra | 15:44 | |
| jeblair | andreaf: it may also cause them to run both jobs | 15:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: Add github debugging script https://review.openstack.org/529362 | 15:45 |
| *** dizquierdo has quit IRC | 15:45 | |
| *** iyamahat has quit IRC | 15:45 | |
| *** iyamahat has joined #openstack-infra | 15:45 | |
| *** lukebrowning has quit IRC | 15:45 | |
| andreaf | jeblair: well if the job is added to the pipeline both via template as well as directly, yeah | 15:45 |
| frickler | fungi: I saw some earlier already, didn't keep a reference | 15:46 |
| *** lukebrowning has joined #openstack-infra | 15:47 | |
| *** shu-mutou has quit IRC | 15:47 | |
| fungi | frickler: ahh, maybe whatever the problem was, it's since subsided. what sort of "timeouts" are you talking about" job timeouts? network connection timeouts? some other sort of timeouts? | 15:47 |
| AJaeger | config-core, could you review a few of the open changes, please? I grouped them https://etherpad.openstack.org/p/Nvt3ovbn5x and ordered by prio - I'd like at least the translation and PTI jobs to go in today if you find some time for reviews. thanks! | 15:47 |
| *** eharney has quit IRC | 15:49 | |
| frickler | fungi: jobs stuck in the middle of their stream output for hours. not sure whether these are network issues or nodes getting exponentially slow sometimes | 15:49 |
| frickler | fungi: different question: can you confirm that what I did for gerrit issues earlier (service gerrit stop/start) was correct? is it normal for the start to claim "FAILED" but succeed anyway? maybe increase the timeout for the init job? | 15:51 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add github debugging template https://review.openstack.org/529362 | 15:51 |
| *** pcaruana has joined #openstack-infra | 15:51 | |
| *** lukebrowning has quit IRC | 15:52 | |
| jeblair | AJaeger: i jumped into 528819 to review and don't understand (even after reading the commit message) why that's happening. is there a description? | 15:52 |
| AJaeger | frickler: that's not normal. Was it a specific cloud? | 15:52 |
| AJaeger | jeblair: that's mordred's change, let's ask him... | 15:52 |
| fungi | frickler: yeah, we've talked about how to solve that, the service takes too long to start and so the initscript gives up waiting... that may just go away under xenial/systemd | 15:52 |
| jeblair | frickler: there's only one gerrit process and it has the right start time, so that's good. :) | 15:53 |
| frickler | AJaeger: difficult to tell while the job is still running | 15:53 |
| *** coolsvap has joined #openstack-infra | 15:53 | |
| *** lukebrowning has joined #openstack-infra | 15:53 | |
| *** janki has quit IRC | 15:53 | |
| AJaeger | jeblair: it's basically a dance: enable using new jobs, update proejct-config to use new job, cleanup | 15:53 |
| jeblair | AJaeger: that i understand -- the commit message says that. it doesn't say why we're making the change at all. | 15:54 |
| *** felipemonteiro has joined #openstack-infra | 15:54 | |
| jeblair | basically it says "we're changing jobs" it doesn't say "this is why we're changing jobs" | 15:54 |
| fungi | frickler: we generally first check javamelody at https://review.openstack.org/monitoring to see whether the java garbage collection graph shows it going haywire (known behavior) or something new, and then restart it. but based on your comment in scrollback about being unable to check the connections list i expect you're not in the gerrit administrators group yet so can't get to the javamelody | 15:55 |
| fungi | interface either | 15:55 |
| AJaeger | jeblair: I did https://review.openstack.org/#/c/528553/ to import the tox jobs and then mordred wanted to simplify - he basically said there's no need for those custom jobs, let's use the "normal" ones. | 15:55 |
| *** smatzek has joined #openstack-infra | 15:55 | |
| andreaf | mordred, jeblair: is this variant issue (in zuul?) fixed now? https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L31 | 15:55 |
| frickler | fungi: ah, yes, that's a different thing I wanted to ask, can I get admin rights for gerrit? | 15:56 |
| frickler | fungi: also related to omer's issue earlier | 15:57 |
| *** xarses_ has joined #openstack-infra | 15:57 | |
| jeblair | AJaeger, mordred: got it, that makes sense. :) it's hard to keep up with all the irc chat between you -- you have a lot of things going on. extra background in commit messages helps. :) | 15:57 |
| andreaf | mordred: also, I just realised that integrated-gate is not used yet - its jobs are included one by one for all relevant repos - and I was wondering why - is it related to the variant issue as well? | 15:57 |
| fungi | frickler: luckily jm keeps some history through restarts, so looking at the gc graph in it right now i can see it was definitely getting bad right up to the time you restarted | 15:57 |
| *** jamesmcarthur has quit IRC | 15:57 | |
| AJaeger | jeblair: sorry ;( Too much balls juggling currently ;) That's why I tried to sort the changes to help with. Let me update the etherpad at least... | 15:57 |
| clarkb | frickler: Gerrit's init script has a timeout for it to see whatever it is that says Gerrit is up happen but Gerrit takes so long yo start now that it misses that timeout | 15:58 |
| *** jamesmcarthur has joined #openstack-infra | 15:58 | |
| clarkb | but the service ends up starting ok amyways | 15:58 |
| *** lukebrowning has quit IRC | 15:58 | |
| *** caphrim007 has quit IRC | 15:58 | |
| jeblair | andreaf, mordred: i don't know what that comment was referring to, however, all known bugs regarding variants are fixed and in production. | 15:58 |
| *** caphrim007 has joined #openstack-infra | 15:59 | |
| frickler | clarkb: yes, if we know it takes like 180 secs instead of the default 90, we could maybe increase it. but if we migrate that to xenial anyway soon, then meh ;) | 15:59 |
| fungi | infra-root: i've added frickler to the "Administrators" group in gerrit on review.openstack.org | 15:59 |
| *** rossella_s has quit IRC | 15:59 | |
| clarkb | fungi: thanks! I guess that was missed when we did initial Gerrit group updates | 16:00 |
| *** lukebrowning has joined #openstack-infra | 16:00 | |
| AJaeger | fungi, do you want to add any of the other new infra-roots as well? | 16:00 |
| AJaeger | or were those in already? | 16:00 |
| fungi | frickler: note that the gerrit ui may now show you some additional administrative options even when doing seemingly normal tasks such as code review, so please do be careful | 16:00 |
| fungi | AJaeger: i figure we have a general consensus that they can be added when they feel they're ready | 16:00 |
| frickler | fungi: ack, thx | 16:00 |
| AJaeger | fungi: I see | 16:01 |
| fungi | i did take the opportunity to clear out the launchpadsync account (wow that's been unused for ages) and several former/retired infra-core members | 16:01 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use tempest-full in the integrated gate https://review.openstack.org/529353 | 16:01 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Make AFS marker roles honor sphinx variables https://review.openstack.org/525735 | 16:01 |
| *** shu-mutou has joined #openstack-infra | 16:02 | |
| *** jamesmca_ has joined #openstack-infra | 16:02 | |
| fungi | AJaeger: we've tended not to automatically add people to the gerrit administrators group until they're prepared for the fact that the ui will start showing some new and potentially dangerous options, so wouldn't want to surprise them with it | 16:02 |
| *** jamesmcarthur has quit IRC | 16:02 | |
| AJaeger | fungi: thanks for the background... | 16:03 |
| clarkb | also probably want a short braindump on correct way to force merge something if necessary | 16:03 |
| fungi | e.g., the ability to delete votes from anyone on any change | 16:03 |
| fungi | clarkb: we have it documented now, right? | 16:03 |
| *** rossella_s has joined #openstack-infra | 16:03 | |
| * AJaeger would love that feature ;) | 16:03 | |
| clarkb | fungi: we might? I sent an email a couple months back that could be adapted to docs if nothing else | 16:04 |
| fungi | AJaeger: with great responsibility comes great annoyance, or something like that | 16:04 |
| *** caphrim007 has quit IRC | 16:04 | |
| AJaeger | fungi: ;) | 16:04 |
| clarkb | and now we are in a spiderman comic | 16:04 |
| frickler | while folks are reviewing devstack-gate changes, this one is to add a working neutron-api job https://review.openstack.org/529000 | 16:04 |
| *** lukebrowning has quit IRC | 16:04 | |
| *** jcoufal has quit IRC | 16:04 | |
| fungi | clarkb: frickler: https://docs.openstack.org/infra/system-config/sysadmin.html#force-merging-a-change | 16:05 |
| *** lukebrowning has joined #openstack-infra | 16:06 | |
| dmsimard | fungi: was I there? just wondering | 16:06 |
| dmsimard | in the gerrit administrator group | 16:06 |
| fungi | dmsimard: nope, do you want to be? | 16:07 |
| *** jamesmca_ has quit IRC | 16:07 | |
| dmsimard | oh, I hadn't caught up with the backlog just yet | 16:07 |
| fungi | happy to add you now | 16:07 |
| *** ldnunes has quit IRC | 16:07 | |
| dmsimard | sure, I have some amount of experience managing gerrit from review.rdoproject.org :) | 16:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use python3 for docs publication https://review.openstack.org/525718 | 16:07 |
| fungi | dmsimard: sold! you're added now | 16:08 |
| *** mriedem_away is now known as mriedem | 16:09 | |
| *** ykarel has quit IRC | 16:09 | |
| *** ykarel has joined #openstack-infra | 16:09 | |
| *** jamesmcarthur has joined #openstack-infra | 16:09 | |
| *** lukebrowning has quit IRC | 16:10 | |
| AJaeger | clarkb: could you help reviewing the translation changes mentioned in https://etherpad.openstack.org/p/Nvt3ovbn5x , please? | 16:11 |
| frickler | oh, removing a vote/reviewer is indeed pretty easy to do accidentally. at least the action shows up in the log so folks can blame me when I manage that :D | 16:11 |
| *** ykarel has quit IRC | 16:11 | |
| *** lukebrowning has joined #openstack-infra | 16:12 | |
| andreaf | clarkb, tonyb: how much longer do we plan on testing newton in the gate? is it because of the discussion on LTS that tests are still running? | 16:12 |
| AJaeger | andreaf: aren't we waiting for nova? | 16:13 |
| fungi | frickler: yup, i can attest that it shows up in the review comments given i just did more than 4k of them over the weekend | 16:13 |
| andreaf | AJaeger: maybe - not sure | 16:13 |
| fungi | (granted, i used the api for that, not the webui) | 16:13 |
| AJaeger | andreaf: there was some discussion on the openstack-dev mailing list about nova and newton recently | 16:14 |
| AJaeger | andreaf: but yes, best ask others what else is needing newton | 16:15 |
| andreaf | AJaeger: it's just that I started fixing the nnet integrated gate template usage, only to realise that projects are not running the n-net job as they should on the newton branch | 16:15 |
| andreaf | AJaeger: and before I add back the job I thought maybe we should remove it instead from tempest and devstack-gate | 16:16 |
| *** lukebrowning has quit IRC | 16:17 | |
| AJaeger | andreaf: most projects have no newton branch anymore and thus don't need the job. | 16:17 |
| frickler | fungi: I tested it on a patch of mine I was about to abandon, wanted to see whether there is any confirmation first, but the answer is no | 16:17 |
| *** andreas_s has quit IRC | 16:18 | |
| *** ldnunes has joined #openstack-infra | 16:19 | |
| *** lukebrowning has joined #openstack-infra | 16:19 | |
| andreaf | AJaeger: well atm only devstack, tempest and devstack-gate have a nnet job running | 16:20 |
| *** edmondsw has joined #openstack-infra | 16:21 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:22 | |
| *** andreas_s has joined #openstack-infra | 16:22 | |
| andreaf | AJaeger: anyways according to the email thread netwon EOL is going to happen soon-ish so I guess I won't bother changing things now | 16:23 |
| andreaf | AJaeger: only after nova newton is EOL | 16:23 |
| *** iyamahat has quit IRC | 16:24 | |
| *** felipemonteiro_ has quit IRC | 16:24 | |
| *** lukebrowning has quit IRC | 16:24 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:24 | |
| *** jcoufal has joined #openstack-infra | 16:25 | |
| *** edmondsw has quit IRC | 16:26 | |
| *** lukebrowning has joined #openstack-infra | 16:26 | |
| dmsimard | tonyb is in new zealand so his reply might lag a bit :) | 16:26 |
| *** felipemonteiro has quit IRC | 16:26 | |
| dmsimard | New Zealand or Australia, I forget | 16:26 |
| fungi | au | 16:26 |
| fungi | unless he's on holiday | 16:26 |
| *** lukebrowning has quit IRC | 16:31 | |
| clarkb | it is that time of year | 16:31 |
| clarkb | AJaeger: ya I'll take a look shortly at translation thinvs | 16:31 |
| fungi | dmsimard: deep links working great with firefox in new ara. thanks! | 16:32 |
| *** andreas_s has quit IRC | 16:32 | |
| *** lukebrowning has joined #openstack-infra | 16:32 | |
| dmsimard | Yay | 16:32 |
| dmsimard | mnaser should be credited with the fix actually | 16:32 |
| fungi | thanks mnaser! | 16:33 |
| mnaser | i dont even remember how we fixed it | 16:33 |
| fungi | #thanks dmsimard and mnaser for getting deep-linking in ara working for firefox | 16:33 |
| mnaser | r maybe i do | 16:33 |
| openstackstatus | fungi: Added thanks to Thanks page | 16:33 |
| mnaser | but yay! :P | 16:33 |
| * fungi must remember to make use of #thanks more often | 16:34 | |
| *** rmcall has joined #openstack-infra | 16:34 | |
| *** lukebrowning has quit IRC | 16:37 | |
| *** Goneri has joined #openstack-infra | 16:37 | |
| *** lukebrowning has joined #openstack-infra | 16:38 | |
| *** rmcall has quit IRC | 16:38 | |
| fungi | our of my depth on this ansiblelint error: http://logs.openstack.org/40/526140/5/check/openstack-zuul-jobs-linters/991b3ca/job-output.txt.gz#_2017-12-19_01_11_30_290634 | 16:40 |
| fungi | er, out of my depth | 16:40 |
| fungi | it seems to be complaining about a missing action in a task in another repo for a role merely referenced in this change | 16:40 |
| fungi | Couldn't parse task at /home/zuul/src/git.openstack.org/openstack-infra/zuul-jobs/roles/prepare-workspace/tasks/main.yaml:2 (no action detected in task. This often indicates a misspelled module name, or incorrect module path.) | 16:41 |
| fungi | {'name': 'Start zuul_console daemon.', 'zuul_console': None} | 16:41 |
| fungi | same error in both playbooks where i attempt to add the prepare-workspace role | 16:42 |
| *** lukebrowning has quit IRC | 16:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role to fetch output from nodes https://review.openstack.org/511843 | 16:43 |
| frickler | fungi: I seem to remember something about zuul needing to be installed into the linters env | 16:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update upload-logs to process docs as well https://review.openstack.org/511853 | 16:44 |
| ssbarnea | can anyone give some hints regarding use of boolean variables with jenkins-job-builder? somehow it seems to be permanent source of problems | 16:44 |
| clarkb | "This web page is slowing down your browser" | 16:44 |
| *** links has quit IRC | 16:44 | |
| *** lukebrowning has joined #openstack-infra | 16:44 | |
| clarkb | ssbarnea: JJB largely deals with strings iirc | 16:45 |
| ssbarnea | these bool vars are expanded inside groovy or bash, but jjb does expand it as True or False (note the case), which happens to be invalid bool value in groovy which is using true and false. | 16:45 |
| ssbarnea | clarkb: not really, if someone does not add quotes, YAML parser will make them bools. | 16:45 |
| clarkb | ah in that case you are using a yaml boolean which gets pulled in as True/False in python which then gets stringified as True/False | 16:46 |
| ssbarnea | clarkb: is not JJB itself which does that, i think is a yaml "feature". | 16:46 |
| ssbarnea | clarkb: indeed. | 16:46 |
| clarkb | yes its yaml being helpful | 16:46 |
| clarkb | can you just supply string values instead so that you get 'true' or 'false' in the xml? | 16:46 |
| ssbarnea | clarkb: now the big question is how to make this fool proof, considering that I have >50 people writing jjb definitions. is very easy to miss to add quotes around a "yes/no" value. | 16:47 |
| clarkb | otherwise you'll likely need to update whatever jjb modules are writing boolean values to stringify True to true instead of True | 16:47 |
| clarkb | also there is #openstack-jjb now I think? | 16:47 |
| *** dangers_away has quit IRC | 16:48 | |
| ssbarnea | clarkb: how to do that? ahh, forgot about that. | 16:48 |
| ssbarnea | it grew enough to have its own channel. | 16:48 |
| clarkb | ssbarnea: and we stopped using it | 16:48 |
| *** lucasagomes is now known as lucas-afk | 16:48 | |
| pabelanger | okay, I've powered off kdc02.o.o, kd04.o.o is not our secondary kerberos server | 16:48 |
| ssbarnea | clarkb: lucky you! | 16:48 |
| clarkb | ssbarnea: so those interested in continuing to use it collaborate in that space | 16:48 |
| andreaf | clarkb, jeblair, mordred, AJaeger: I'm looking at projects.yaml in project-config, putting the integrated-gate template into use - it's a bit frustrating since every project has to re-include the jobs from the template in both check and gate to define irrelevant-files | 16:49 |
| ssbarnea | clarkb: not sure if "interested" is the best way to describe it, is more of "still forced by other reasons" ;) | 16:49 |
| *** e0ne has quit IRC | 16:49 | |
| *** lukebrowning has quit IRC | 16:49 | |
| andreaf | so one side it feels like the template is for nothing, on the other side it's a lot of repeating the irrelevant-files bit over and over again | 16:49 |
| *** gibi is now known as gibi_away | 16:50 | |
| *** andreas_s has joined #openstack-infra | 16:50 | |
| andreaf | is there any way to factor the irrelevant-files bit at project level into a default so we can specify it only where there is a variance? | 16:50 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: Attempt to improve tenant config loading in case of config issue https://review.openstack.org/529060 | 16:50 |
| clarkb | andreaf: if there are common sets of irrelevant files we can collapse them down into a yaml anchor | 16:50 |
| clarkb | andreaf: you still have to specify the anchor everywhere you use it but its a lot less verbose | 16:51 |
| clarkb | (I think smcginnis was going to look into doing some of that) | 16:51 |
| *** lukebrowning has joined #openstack-infra | 16:51 | |
| andreaf | clarkb: yeah that would improve readability - but it won't be as good as having a default set | 16:51 |
| openstackgerrit | Sayali Lunkad proposed openstack/diskimage-builder master: Adding mapping for SUSE package https://review.openstack.org/529370 | 16:51 |
| andreaf | clarkb: or perhaps can we attach relevant files to a job template in the project? | 16:51 |
| smcginnis | clarkb: I tried it out, but you can anchor/merge dict values but not arrays. | 16:52 |
| pabelanger | jeblair: when we do https://docs.openstack.org/infra/system-config/kerberos.html#no-service-outage-server-maintenance on kdc01.o.o, does that process sync ALL data to the slave? Trying to see if delete kdc01 if there is any other data we'd need to migrate when we bring kdc03.o.o (replacement) | 16:52 |
| pabelanger | online | 16:52 |
| clarkb | andreaf: you can attach them to the more base version of the job | 16:52 |
| smcginnis | It ends up being an array where one of the items is the array you've anchored. | 16:53 |
| clarkb | smcginnis: wouldn't you just anchor irrelevant-files: some_anchor_here and if you want a different set of files you don't use that anchor? | 16:53 |
| *** cshastri has joined #openstack-infra | 16:53 | |
| smcginnis | clarkb: True, that's one approach. It may help get rid of some duplication at least. | 16:54 |
| *** jtomasek has quit IRC | 16:54 | |
| smcginnis | clarkb: I was really hoping we could have an anchor for the common set, then be able to add additional items as needed. | 16:54 |
| *** aviau has quit IRC | 16:54 | |
| *** aviau has joined #openstack-infra | 16:54 | |
| *** andreas_s has quit IRC | 16:55 | |
| andreaf | clarkb: well the thing is the list tends to be repo specific, so attach it to the job definition would not be good | 16:55 |
| *** dangers has joined #openstack-infra | 16:55 | |
| clarkb | andreaf: ya I think thats why the anchor idea came up | 16:55 |
| clarkb | since there is the nova exclude list or whatever that can be applied multiple times | 16:55 |
| *** lukebrowning has quit IRC | 16:55 | |
| andreaf | clarkb: unless we can include a variable in the irrelevant file list and have project specify a value for that | 16:55 |
| andreaf | clarkb: well still it forces you to spell out the list of jobs which means if you make a change to a template you have to go and change every place where the template is used | 16:56 |
| andreaf | clarkb: which kind of defeats the purpose of the template itself | 16:56 |
| clarkb | yes I think you are finding two features which don't work together because they stand in opposition | 16:58 |
| clarkb | one is specifically to say "this is common lets do it the same everywhere" and the other ends up being "we are all special and need our own stuff" | 16:59 |
| pabelanger | mordred: mind looking at https://review.openstack.org/526708/ and adding the usage of serial to base-test (eventually base) for shared ansible_host in zuulv3? | 16:59 |
| *** dizquierdo has joined #openstack-infra | 16:59 | |
| *** ijw has joined #openstack-infra | 16:59 | |
| clarkb | andreaf: I think both are valid use cases, its just that I'm not sure they will always work well together | 16:59 |
| andreaf | clarkb: well kinda - the file list could really mostly be done at project level but we're forced to do it at job level | 17:00 |
| andreaf | clarkb: so a job-template and a file list at project level would work well together | 17:00 |
| *** ykarel has joined #openstack-infra | 17:01 | |
| *** hasharAway is now known as hashar | 17:01 | |
| andreaf | clarkb: if you look at projects.yaml it doesn't look like they'll ever work well together | 17:01 |
| *** dangers is now known as dangers_away | 17:01 | |
| *** gyee has joined #openstack-infra | 17:01 | |
| clarkb | andreaf: they could if we also standardized project structure more | 17:02 |
| clarkb | andreaf: so that you could rely on common sets of irrelevant files for jobs | 17:02 |
| clarkb | but we've not done that | 17:02 |
| andreaf | clarkb: yeah that's kind of hard to fix now :D | 17:03 |
| *** priteau has joined #openstack-infra | 17:04 | |
| clarkb | fungi: do you know if the nodepool changes we needed got in? | 17:04 |
| * clarkb is finishing up reviews for AJaeger but then I think we should consider another nodepool launcher restart if they have | 17:04 | |
| andreaf | clarkb | 17:05 |
| andreaf | clarkb: irrelevant-files are proper regex right? | 17:05 |
| AJaeger | clarkb: thanks | 17:05 |
| AJaeger | andreaf: yes, they can be | 17:05 |
| andreaf | clarkb, AJaeger: something like - ^neutron/tests/unit/.*$ can be made into a regex that matches for all projects | 17:05 |
| AJaeger | clarkb: indeed, that is fine for all projects - even if it matches only one ;) | 17:06 |
| AJaeger | andreaf: and you can use a *list* of regexes. | 17:06 |
| AJaeger | clarkb: sorry, meant andreaf | 17:06 |
| andreaf | clarkb, AJaeger: that's what I see as the most common exception for integration tests to job level file exclusions | 17:06 |
| AJaeger | calbers: tahnks | 17:06 |
| * AJaeger cannot type anymore ;( | 17:06 | |
| AJaeger | calbers: sorry, wanted to say: | 17:07 |
| clarkb | AJaeger: its ok I was never able to type | 17:07 |
| AJaeger | clarkb: Thanks | 17:07 |
| andreaf | AJaeger, clarkb: well I'd rather do something like ^[a-z_\-]*/tests/' | 17:07 |
| calbers | AJaeger: np :D | 17:07 |
| AJaeger | andreaf: feel free to start with something, and then iterate as you get suggestions. | 17:08 |
| AJaeger | andreaf: I'm happy to review a change... | 17:08 |
| *** priteau has quit IRC | 17:08 | |
| mriedem | clarkb: where is the multiline handling for indexed logs again? system-config? | 17:09 |
| mriedem | trying to figure out why i can't do multiline queries on the placement-api logs | 17:09 |
| clarkb | mriedem: openstack-infra/logstash-filters | 17:10 |
| mriedem | thanks | 17:10 |
| fungi | clarkb: the nodepool changes from last night (-1 unlimited quota and bug fixes) merged as far as i know | 17:11 |
| pabelanger | nice | 17:11 |
| *** pcichy has joined #openstack-infra | 17:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Prevent duplicate pull_from_zanata on the same repo https://review.openstack.org/528518 | 17:14 |
| *** iyamahat has joined #openstack-infra | 17:14 | |
| sambetts | AJaeger: changing the default branch in the tox_install.sh worked: https://review.openstack.org/#/c/522315/ | 17:14 |
| *** smatzek has quit IRC | 17:16 | |
| openstackgerrit | Tytus Kurek proposed openstack-infra/project-config master: Add charm-interface-designate project https://review.openstack.org/517336 | 17:17 |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support https://review.openstack.org/529376 | 17:17 |
| *** udesale has quit IRC | 17:18 | |
| mriedem | clarkb: oh i suppose it doesn't work because we run placement-api under wsgi and so it doesn't match the syslog_program entry here | 17:19 |
| mriedem | http://git.openstack.org/cgit/openstack-infra/logstash-filters/tree/filters/openstack-filters.conf#n46 | 17:19 |
| openstackgerrit | Tytus Kurek proposed openstack-infra/project-config master: Add charm-interface-designate project https://review.openstack.org/517336 | 17:19 |
| mriedem | match => { "message" => "(?m)^(%{TIMESTAMP_ISO8601:logdate}|%{SYSLOGTIMESTAMP:logdate})%{SPACE}(%{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?:|%{NUMBER:pid})?%{SPACE}?(?<loglevel>AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) \[?\b%{NOTSPACE:module}\b\]?%{SPACE}?%{GREEDYDATA:logmessage}?" } | 17:19 |
| *** florianf has quit IRC | 17:19 | |
| *** cshastri has quit IRC | 17:20 | |
| mriedem | Dec 20 15:15:20.666337 ubuntu-xenial-citycloud-lon1-0001533915 devstack@placement-api.service[15195]: ERROR nova.api.openstack.placement.handler | 17:20 |
| mriedem | compared to | 17:20 |
| mriedem | Dec 20 15:15:20.747237 ubuntu-xenial-citycloud-lon1-0001533915 nova-conductor[21763]: ERROR nova.conductor.tasks.migrate | 17:20 |
| *** ykarel has quit IRC | 17:21 | |
| *** andreas_s has joined #openstack-infra | 17:22 | |
| clarkb | mriedem: that could be it there is a logstash grok filter checking website that makes testing easy | 17:22 |
| *** caphrim007 has joined #openstack-infra | 17:22 | |
| mriedem | yeah i'm poking through that | 17:23 |
| *** dprince has quit IRC | 17:23 | |
| frickler | fungi: like this maybe https://github.com/openstack-infra/project-config/blob/master/tox.ini#L14 | 17:23 |
| *** d0ugal has quit IRC | 17:23 | |
| frickler | bbiab | 17:24 |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support for RHEL/CentOS https://review.openstack.org/528739 | 17:24 |
| openstackgerrit | Tytus Kurek proposed openstack-infra/project-config master: Add charm-interface-designate project https://review.openstack.org/529379 | 17:24 |
| *** trown|ruck is now known as trown|lunch | 17:25 | |
| fungi | frickler: thanks, i'll see if that helps | 17:25 |
| *** sbezverk has joined #openstack-infra | 17:25 | |
| mriedem | hmm actually the grok pattern should work here | 17:25 |
| openstackgerrit | Tytus Kurek proposed openstack-infra/project-config master: Add charm-interface-designate project https://review.openstack.org/517336 | 17:25 |
| mriedem | yeah i don't get it, the grok patterns match | 17:26 |
| AJaeger | sambetts: yeah! | 17:29 |
| *** Apoorva has quit IRC | 17:30 | |
| *** dtantsur is now known as dtantsur|afk | 17:34 | |
| *** d0ugal has joined #openstack-infra | 17:35 | |
| AJaeger | mordred: comment on https://review.openstack.org/#/c/525736 - we should handle doc/requirements.txt now, shouldn't we? | 17:36 |
| AJaeger | mordred: I'm testing that change with https://review.openstack.org/529382 now | 17:39 |
| *** andreas_s has quit IRC | 17:40 | |
| *** lukebrowning has joined #openstack-infra | 17:40 | |
| *** neiljerram has quit IRC | 17:42 | |
| *** neiljerram has joined #openstack-infra | 17:44 | |
| *** jamesmcarthur has quit IRC | 17:44 | |
| *** andreas_s has joined #openstack-infra | 17:45 | |
| *** iyamahat has quit IRC | 17:45 | |
| *** lukebrowning has quit IRC | 17:45 | |
| *** fultonj has quit IRC | 17:47 | |
| clarkb | jeblair: are you around? looks like on nl01 at least all three of your changes have merged and the tip of master nodepool is installed. I think we are ready to do a service restart | 17:47 |
| clarkb | (happy to do it but want to makle sure there isn't anything else we need to do first as I missed a good chunk of the debugging yesterday) | 17:47 |
| *** lukebrowning has joined #openstack-infra | 17:48 | |
| *** mihalis68 has quit IRC | 17:48 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-base-jobs master: Add generic base and base-test jobs/playbooks https://review.openstack.org/526140 | 17:49 |
| jeblair | clarkb: yep, i think we can restart and try again. please go for it. | 17:49 |
| *** andreas_s has quit IRC | 17:49 | |
| clarkb | ok starting with nl01 now | 17:49 |
| fungi | i'm around for a bit if something goes sideways | 17:49 |
| *** rmcall has joined #openstack-infra | 17:50 | |
| clarkb | will watch for exceptions in its debug log before doing 02 | 17:50 |
| clarkb | jeblair: tobiash http://paste.openstack.org/show/629485/ seems to be happening a fair bit | 17:52 |
| *** derekh has quit IRC | 17:52 | |
| *** lukebrowning has quit IRC | 17:52 | |
| tobiash | clarkb: that doesn't look good | 17:52 |
| *** xarses_ has quit IRC | 17:53 | |
| clarkb | self.provider.pools.get(node.pool) is where pool comes from | 17:53 |
| clarkb | so guessing its not in that dict | 17:53 |
| *** xarses_ has joined #openstack-infra | 17:53 | |
| *** smatzek has joined #openstack-infra | 17:53 | |
| *** sambetts is now known as sambetts|afk | 17:53 | |
| tobiash | clarkb: do you have pools in the config without labels? | 17:54 |
| *** lukebrowning has joined #openstack-infra | 17:54 | |
| tobiash | clarkb: ah, I think I have a clue | 17:54 |
| clarkb | tobiash: no the pool there (belonging to mtl01) has labels | 17:54 |
| clarkb | and there is only one pool attached to that provider | 17:55 |
| tobiash | clarkb: you do sharding of the providers with two launchers right? | 17:55 |
| clarkb | tobiash: correct | 17:55 |
| tobiash | so launcher 1 has provider/pool a and launcher 2 provider/pool b | 17:55 |
| clarkb | yes | 17:56 |
| clarkb | in this case launcher 1 is showing the exception and it has config for provider/pool a that is failing | 17:56 |
| clarkb | not we shard on providers so providers in launcher 1 are not in launcher 2 | 17:56 |
| tobiash | hm, that crashes my theory | 17:56 |
| *** hemna_ has quit IRC | 17:56 | |
| clarkb | also it may have stopped? I'm going to run a grep to check | 17:57 |
| clarkb | ya it happened 3 times and hasn't happened since (but that could be due to demand I supposed) | 17:57 |
| clarkb | tobiash: is it possible that is due to the restart itself? the zookeper node wasn't fully updated with pool information so when it started up again it had a mismatch between node.pool and pool.name? | 17:58 |
| *** lukebrowning has quit IRC | 17:59 | |
| tobiash | clarkb: is there maybe a stale vm in this cloud which states it's from a different provider? | 17:59 |
| clarkb | looks like its checking node.provider and node.pool and if we don't update them atomically I could see something like this happening | 17:59 |
| tobiash | clarkb: yes that could be possible | 18:00 |
| tobiash | we should add a check to none there to not crash the calculation | 18:00 |
| *** lukebrowning has joined #openstack-infra | 18:00 | |
| *** iyamahat has joined #openstack-infra | 18:00 | |
| *** jascott1 has joined #openstack-infra | 18:00 | |
| clarkb | as for a stale vm you mean one as listed by zookeeper? or by the cloud? | 18:00 |
| tobiash | clarkb: ah wait | 18:01 |
| tobiash | it's quota needed for node type | 18:01 |
| tobiash | could it be that this handler tried to create a node for a label it's not configured for? | 18:01 |
| clarkb | I'm not sure how that would happen | 18:01 |
| clarkb | (also I don't think we log enough info to know | 18:01 |
| tobiash | me neither | 18:02 |
| jeblair | tobiash: it looks like this is calculated the quota used by existing nodes | 18:02 |
| clarkb | that provider has a pool with all of our labels in it. I don't see any missing labels | 18:02 |
| clarkb | In the config that is | 18:02 |
| jeblair | yeah, it would be nice to know what the value of node.pool is | 18:03 |
| *** dprince has joined #openstack-infra | 18:03 | |
| clarkb | jeblair: ya I'm wondering if this is stale state left over from the restart | 18:03 |
| clarkb | because its checking multiple values that aren't necessarily updated atomically | 18:03 |
| clarkb | grep says it has still only happened the three times | 18:04 |
| jeblair | clarkb: especially if we don't see this again for a while, i think that's a good assumption. maybe we should just add the handler tobiash suggested, maybe with an extra bit of logging so we know what node triggered the error | 18:04 |
| jeblair | that would let us go back through the logs and confirm that node was in an expected-bad state (and cleaned up by another thread likely) | 18:04 |
| clarkb | That seems like a good idea. Do we awnt to get that in before restarting 02? | 18:04 |
| tobiash | clarkb, jeblair: looking at the code it at least matched already the provider so this might be a stale node entry in zk | 18:05 |
| *** lukebrowning has quit IRC | 18:05 | |
| jeblair | clarkb: nah, it'll take an hour to merge; let's restart so we can continue to observe for any other new behaviors | 18:05 |
| *** fultonj has joined #openstack-infra | 18:05 | |
| AJaeger | fungi: rebase https://review.openstack.org/#/c/525271/ | 18:05 |
| clarkb | ok I'm moving on to 02 then and if no one has a patch up get I can work on a patch too | 18:05 |
| *** Apoorva has joined #openstack-infra | 18:06 | |
| *** Apoorva has quit IRC | 18:06 | |
| clarkb | that was quick 02 is done | 18:06 |
| *** lukebrowning has joined #openstack-infra | 18:07 | |
| clarkb | I'm working on a patch to do more logging now | 18:07 |
| *** Apoorva has joined #openstack-infra | 18:07 | |
| fungi | AJaeger: oh, thanks! gertty wasn't making it obvious | 18:08 |
| AJaeger | fungi: the webui did | 18:08 |
| fungi | yep, i expect i got used to the old merge-check comments from zuul | 18:08 |
| *** neiljerram has quit IRC | 18:09 | |
| fungi | strangely it doesn't say it needs a rebase, just that a rebase is possible | 18:09 |
| *** edmondsw has joined #openstack-infra | 18:09 | |
| AJaeger | yeah, this is confusing. | 18:09 |
| *** claudiub has quit IRC | 18:09 | |
| AJaeger | My role of thumb for teh webui: It does not merge, orange circle at parent -> Let's rebase | 18:09 |
| AJaeger | not intuitive ;( | 18:09 |
| AJaeger | But first i check for dependencies - stack and depends-on | 18:10 |
| fungi | i'll let zuul finish just in case we get more details out of it this time | 18:10 |
| AJaeger | sure | 18:10 |
| fungi | i expect it's getting an error back when asking gerrit to merge the commit, and if so i can hopefully at least find that in the scheduler log | 18:11 |
| *** makowals has joined #openstack-infra | 18:11 | |
| *** lukebrowning has quit IRC | 18:11 | |
| AJaeger | and if we can pass that back to gerrit for display, it would be really great | 18:11 |
| fungi | worst case, if the logs aren't helping, i can test pushing the submit button for it in gerrit and see what gerrit tells me then | 18:11 |
| clarkb | on nl02 I'm seeing timeouts for ssh access and the citycloud quota issue but both are "normal" I think | 18:12 |
| *** lukebrowning has joined #openstack-infra | 18:13 | |
| *** edmondsw has quit IRC | 18:13 | |
| *** Apoorva has quit IRC | 18:13 | |
| fungi | frickler: your suggestion worked, thanks! now 526140 is all green and ready for review | 18:13 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Log unknown providers during quota calculation https://review.openstack.org/529391 | 18:15 |
| clarkb | tobiash: jeblair ^ something like that maybe? | 18:15 |
| jeblair | lgtm | 18:16 |
| *** e0ne has joined #openstack-infra | 18:16 | |
| clarkb | Node type even has a __repr__ (checked for that to amek sure log would stringify properly | 18:17 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix github caching https://review.openstack.org/529392 | 18:17 |
| jeblair | ya | 18:17 |
| AJaeger | fungi, let me add those two to my review etherpad at https://etherpad.openstack.org/p/Nvt3ovbn5x | 18:17 |
| *** yamahata has joined #openstack-infra | 18:17 | |
| AJaeger | hope others go over that and see it... | 18:18 |
| *** lukebrowning has quit IRC | 18:18 | |
| tobiash | clarkb: lgtm, thanks | 18:19 |
| *** Apoorva has joined #openstack-infra | 18:19 | |
| *** Apoorva has quit IRC | 18:24 | |
| *** rbrndt has quit IRC | 18:25 | |
| *** jamesmcarthur has joined #openstack-infra | 18:28 | |
| *** raissa has joined #openstack-infra | 18:30 | |
| *** rhallisey has quit IRC | 18:31 | |
| *** Douhet has quit IRC | 18:31 | |
| *** jamesmcarthur has quit IRC | 18:32 | |
| *** Apoorva has joined #openstack-infra | 18:34 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix github caching https://review.openstack.org/529392 | 18:34 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use connection type supplied from nodepool https://review.openstack.org/501976 | 18:35 |
| jlvillal | Are there any other values allowed for 'logfile' in http://zuulv3.openstack.org/stream.html?uuid=a9610cf1fcfc4a4588545c077511c856&logfile=console.log ? | 18:37 |
| clarkb | jlvillal: not currently | 18:38 |
| clarkb | jlvillal: on the roadmap is support for being able to stream arbitrary files but getting feature parity (console log streaming) with old zuul is/was focus | 18:38 |
| jlvillal | clarkb, Thanks :) I was hoping to see what was happening during the devstack run part. I'll wait. | 18:39 |
| fungi | AJaeger: so the zuul log indicates there was an error returned by gerrit when it tried to report on that change, but not what the error was: http://paste.openstack.org/show/629488 | 18:39 |
| fungi | next step, trying the submit button myself | 18:39 |
| *** jtomasek has joined #openstack-infra | 18:40 | |
| jeblair | fungi: there may be something in gerrit's sshd log | 18:40 |
| fungi | "Code Review - Error... Failed to submit 1 change due to the following problems: Change 525271: depends on change that was not submitted" | 18:41 |
| fungi | jeblair: good point, i'll take a peek in there too | 18:41 |
| *** jascott1 has quit IRC | 18:41 | |
| fungi | AJaeger: that at least suggests you're onto something with the stale parent revision | 18:41 |
| *** jascott1 has joined #openstack-infra | 18:41 | |
| fungi | i wonder if there's at least a way to pipe an error message from a gerrit api call back up through that exception | 18:42 |
| *** lukebrowning has joined #openstack-infra | 18:43 | |
| *** jascott1 has quit IRC | 18:43 | |
| *** jascott1 has joined #openstack-infra | 18:45 | |
| fungi | jeblair: i can definitely see the api call in the gerrit ssh log, but no resulting error. this is all it has: | 18:45 |
| fungi | [2017-12-20 18:32:51,967 +0000] ed1dcee1 ufcg-oneview-ci a/17998 gerrit.query.--format.json.--all-approvals.--comments.--commit-message.--current-patch-set.--dependencies.--files.--patch-sets.--submit-records.525271 2ms 14ms 0 | 18:45 |
| fungi | d'oh, wait, that's the wrong account | 18:46 |
| fungi | someone has some seriously misconfigured zuul there | 18:46 |
| fungi | oh, i misread it, that's just a query | 18:46 |
| *** jamesmcarthur has joined #openstack-infra | 18:46 | |
| fungi | [2017-12-20 18:32:35,125 +0000] ede14926 zuul a/22348 gerrit.review.--project.openstack-infra/project-config.--message.Build succeeded (gate pipeline). | 18:47 |
| fungi | - openstack-zuul-jobs-linters http://logs.openstack.org/71/525271/2/gate/openstack-zuul-jobs-linters/5217e97/ : SUCCESS in 5m 08s | 18:47 |
| fungi | .--submit.--label.Verified=2.525271,2 1ms 819ms 1 | 18:47 |
| fungi | but no error | 18:47 |
| *** lukebrowning has quit IRC | 18:48 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add github debugging template https://review.openstack.org/529362 | 18:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add ns2.openstack.org https://review.openstack.org/529398 | 18:49 |
| jeblair | fungi: bummer :( | 18:49 |
| jeblair | fungi: i thought zuul logged output from command errors if it got them | 18:50 |
| jeblair | fungi, clarkb: can you approve 529398 real quick? | 18:50 |
| * clarkb looks | 18:50 | |
| *** lukebrowning has joined #openstack-infra | 18:51 | |
| fungi | approved, going to find lunch now | 18:52 |
| *** ijw has quit IRC | 18:52 | |
| *** lukebrowning has quit IRC | 18:55 | |
| *** jamesmcarthur has quit IRC | 18:57 | |
| *** lukebrowning has joined #openstack-infra | 18:57 | |
| *** coolsvap has quit IRC | 19:00 | |
| *** dizquierdo has quit IRC | 19:00 | |
| *** myoung|bbl is now known as myoung | 19:01 | |
| *** lukebrowning has quit IRC | 19:01 | |
| *** lukebrowning has joined #openstack-infra | 19:03 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed Speakers API issues https://review.openstack.org/529400 | 19:06 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed Speakers API issues https://review.openstack.org/529400 | 19:06 |
| *** trown|lunch is now known as trown|brb | 19:08 | |
| *** trown|brb is now known as trown | 19:08 | |
| *** lukebrowning has quit IRC | 19:08 | |
| *** trown is now known as trown|ruck | 19:08 | |
| *** jamesmcarthur has joined #openstack-infra | 19:09 | |
| *** lukebrowning has joined #openstack-infra | 19:09 | |
| *** salv-orl_ has quit IRC | 19:14 | |
| *** lukebrowning has quit IRC | 19:14 | |
| *** salv-orlando has joined #openstack-infra | 19:14 | |
| *** lukebrowning has joined #openstack-infra | 19:15 | |
| *** makowals has quit IRC | 19:15 | |
| *** jamesmcarthur has quit IRC | 19:16 | |
| pabelanger | clarkb: fungi: mind a +3 on https://review.openstack.org/529189/ will fix up all-clouds.yaml, which we talked about yesterday | 19:18 |
| *** salv-orlando has quit IRC | 19:19 | |
| clarkb | done | 19:19 |
| *** lukebrowning has quit IRC | 19:20 | |
| *** bobh has quit IRC | 19:20 | |
| clarkb | I've got to convert my office to a guest room so will be afk for lunch longer than usual. I haven't noticed any problems iwth nodepool launcher other than the thing identified earlier | 19:21 |
| jeblair | clarkb, tobiash: yeah, graphs look okay so far | 19:22 |
| tobiash | :) | 19:22 |
| *** shardy has quit IRC | 19:23 | |
| *** lukebrowning has joined #openstack-infra | 19:24 | |
| pabelanger | nice! so that means we can remove max-servers now that nodepool knows the quota? | 19:24 |
| fungi | clarkb: i know the feeling. all my breaks have been occupied by prepping guest accommodations, wrapping gifts, preparing any food which will keep until people arrive, wrapping gifts. i'm already ready for this to be over :/ | 19:25 |
| *** rbrndt has joined #openstack-infra | 19:25 | |
| fungi | at least smoking meats will give me an opportunity to take long breaks outside in the cold away from the visiting hordes | 19:25 |
| *** jamesmcarthur has joined #openstack-infra | 19:26 | |
| * bkero takes pity on other family and just flees the continent for the holidays. | 19:26 | |
| clarkb | pabelanger: maybe? we might want to track down the thing I identified once extra logging is in just to make sure ist not some major problem first | 19:26 |
| clarkb | pabelanger: but also we don't need to remove that until we start adding different sized labels | 19:26 |
| pabelanger | clarkb: okay, cool. Which issue is that? | 19:27 |
| clarkb | pabelanger: http://paste.openstack.org/show/629485/ I think its mostly a non issue but https://review.openstack.org/#/c/529391/ should help confirm that | 19:28 |
| *** felipemonteiro__ has joined #openstack-infra | 19:28 | |
| *** CrayZee has joined #openstack-infra | 19:28 | |
| pabelanger | ah | 19:28 |
| *** bobh has joined #openstack-infra | 19:28 | |
| *** pcaruana has quit IRC | 19:29 | |
| pabelanger | I see some infracloud SSH timeout errors, wonder if we are having an issue with a compute node | 19:29 |
| pabelanger | will look in a moment | 19:29 |
| *** lukebrowning has quit IRC | 19:29 | |
| clarkb | pabelanger: also is toady a good day to talk about gerrit things? | 19:29 |
| clarkb | I can do that now or after lunch if it works | 19:29 |
| pabelanger | clarkb: yah, less do after lunch. Just getting out of a meeting myself and need some food | 19:30 |
| clarkb | ok | 19:30 |
| *** lukebrowning has joined #openstack-infra | 19:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Migrate special docs jobs to build-openstack-sphinx-docs https://review.openstack.org/525736 | 19:31 |
| AJaeger | mordred: updated files list ^ | 19:31 |
| *** felipemonteiro_ has quit IRC | 19:32 | |
| *** ijw has joined #openstack-infra | 19:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Migrate special docs jobs to build-openstack-sphinx-docs https://review.openstack.org/525736 | 19:33 |
| *** bobh has quit IRC | 19:33 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Update get speaker query https://review.openstack.org/529402 | 19:33 |
| *** bobh has joined #openstack-infra | 19:33 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Set remote url on every getRepo in merger https://review.openstack.org/529293 | 19:34 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: shall fail https://review.openstack.org/529403 | 19:34 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Query for 1719469 https://review.openstack.org/507294 | 19:34 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Update get speaker query https://review.openstack.org/529402 | 19:34 |
| mordred | fungi, pabelanger: mind +A'ing this real quick: https://review.openstack.org/#/c/529203/ | 19:34 |
| AJaeger | config-core, I updated https://etherpad.openstack.org/p/Nvt3ovbn5x with list of reviews I'd like to see go in. could you grab a few, please? | 19:35 |
| mordred | AJaeger: lgtm | 19:35 |
| AJaeger | mordred: that one is also on my etherpad ;) | 19:35 |
| *** lukebrowning has quit IRC | 19:35 | |
| AJaeger | mordred: thanks. The test change worked fine, so I'm ready to take the next steps | 19:35 |
| pabelanger | Hmm, I think we might have a bad HDD in infracloud-vanilla: http://paste.openstack.org/raw/629492/ | 19:35 |
| pabelanger | looking at compute026 now | 19:36 |
| *** dsariel has quit IRC | 19:36 | |
| pabelanger | Yah, not looking healthy | 19:37 |
| AJaeger | mordred: did you see my comment on https://review.openstack.org/#/c/525734/4/zuul.d/jobs.yaml | 19:37 |
| *** lukebrowning has joined #openstack-infra | 19:37 | |
| pabelanger | [Tue Nov 7 12:44:54 2017] EXT4-fs error (device sda1) in ext4_writepages:2618: IO failure | 19:37 |
| *** jpena is now known as jpena|off | 19:37 | |
| pabelanger | I'm going to take compute026.vanilla.ic.o.o offline on the controller | 19:38 |
| *** bobh has quit IRC | 19:38 | |
| * AJaeger waves good night | 19:38 | |
| *** rlandy|rover is now known as rlandy|rover|brb | 19:39 | |
| jlvillal | Is there an easy way to run jobs for different branches in a project? We have our ironic-tempest-plugin which is branchless (only master). We want to test changes to it against ironic branches: master (working), stable/pike, and stable/ocata | 19:39 |
| jlvillal | I'm unsure how to do that :( | 19:39 |
| mordred | AJaeger: woot! looking at comment | 19:40 |
| *** jascott1 has quit IRC | 19:41 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add ns2.openstack.org https://review.openstack.org/529398 | 19:41 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove infra-files-ro and infra-files-rw https://review.openstack.org/529189 | 19:41 |
| *** jascott1 has joined #openstack-infra | 19:41 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on speakers query https://review.openstack.org/529404 | 19:41 |
| *** lukebrowning has quit IRC | 19:42 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on speakers query https://review.openstack.org/529404 | 19:42 |
| *** lukebrowning has joined #openstack-infra | 19:43 | |
| *** ijw has quit IRC | 19:43 | |
| *** jascott1 has quit IRC | 19:43 | |
| *** ijw has joined #openstack-infra | 19:44 | |
| *** bobh has joined #openstack-infra | 19:44 | |
| *** pcaruana has joined #openstack-infra | 19:48 | |
| *** bobh has quit IRC | 19:49 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: DNM: shall fail https://review.openstack.org/529403 | 19:49 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix build-javascript-content success-url https://review.openstack.org/529203 | 19:50 |
| *** jtomasek has quit IRC | 19:50 | |
| *** lukebrowning has quit IRC | 19:51 | |
| tobiash | is something broken with log streaming? | 19:52 |
| tobiash | it seems to randomly work and randomly not | 19:52 |
| *** lukebrowning has joined #openstack-infra | 19:52 | |
| *** pcaruana has quit IRC | 19:54 | |
| *** bobh has joined #openstack-infra | 19:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Switch remaining docs publications jobs to sphinx https://review.openstack.org/525734 | 19:56 |
| fungi | tobiash: randomly works/fails for the same build, or works at random for some builds and not others? | 19:57 |
| *** edmondsw has joined #openstack-infra | 19:57 | |
| *** lukebrowning has quit IRC | 19:57 | |
| fungi | the latter usually happens if we crash zuul-web on an executor (there was trouble for a while where memory pressure from concurrent jobs was causing the oom killer to snipe zuul-web processes) | 19:58 |
| tobiash | fungi: some work, others not | 19:58 |
| *** ijw has quit IRC | 19:59 | |
| pabelanger | #status log Disabled compute026.vanilla.ic.o.o due to hard disk being in read-only mode | 19:59 |
| openstackstatus | pabelanger: finished logging | 19:59 |
| *** lukebrowning has joined #openstack-infra | 19:59 | |
| *** bobh has quit IRC | 19:59 | |
| *** mkoderer__ has quit IRC | 20:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Small refactor of translation scripts https://review.openstack.org/529047 | 20:01 |
| *** dprince has quit IRC | 20:01 | |
| *** ijw has joined #openstack-infra | 20:01 | |
| mordred | fungi, pabelanger: got a sec for an easy +3? https://review.openstack.org/#/c/528649 - https://review.openstack.org/#/c/528500 is also tiny and easy | 20:01 |
| *** edmondsw has quit IRC | 20:02 | |
| fungi | tobiash: looks like we still have the oom killer situation going on... one happened today on ze02 (i'm still looking through dmesg on the other executors for similar occurrences) | 20:03 |
| *** lukebrowning has quit IRC | 20:04 | |
| tobiash | ah, ok | 20:04 |
| *** eharney has joined #openstack-infra | 20:04 | |
| fungi | so if the zuul-executor child handling the web socket ended up getting killed, then any jobs run by that executor won't have a working web stream | 20:05 |
| smcginnis | What does that mean when a job shows it's half way in zuulv3.o.o but if I try to look at the console it shows "End of Stream". Is it actually running? | 20:05 |
| *** lukebrowning has joined #openstack-infra | 20:05 | |
| fungi | smcginnis: that's what we're talking about | 20:05 |
| fungi | tobiash and i | 20:05 |
| *** ijw has quit IRC | 20:05 | |
| *** bobh has joined #openstack-infra | 20:05 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix github caching https://review.openstack.org/529392 | 20:05 |
| smcginnis | fungi: Ah OK. Need to read scrollback. Does that mean my patch is doomed again? | 20:06 |
| *** pcaruana has joined #openstack-infra | 20:06 | |
| *** slaweq_ has joined #openstack-infra | 20:06 | |
| fungi | not necessarily, no | 20:07 |
| mordred | fungi: fwiw - I'm about half done with a reworking of the mechanics of that ... but I doubt it'll be ready today | 20:07 |
| fungi | yeah, so for example ze02 has had oom incidents since the last time zuul-executor was started there a couple weeks ago | 20:08 |
| *** lukebrowning has quit IRC | 20:10 | |
| *** bobh has quit IRC | 20:10 | |
| *** jamesmcarthur has quit IRC | 20:11 | |
| *** lukebrowning has joined #openstack-infra | 20:11 | |
| *** smatzek has quit IRC | 20:12 | |
| *** pcaruana has quit IRC | 20:13 | |
| *** rlandy|rover|brb is now known as rlandy|rover | 20:14 | |
| *** bobh has joined #openstack-infra | 20:14 | |
| *** ldnunes has quit IRC | 20:15 | |
| *** lukebrowning has quit IRC | 20:16 | |
| *** amoralej is now known as amoralej|off | 20:17 | |
| *** lukebrowning has joined #openstack-infra | 20:19 | |
| *** bobh has quit IRC | 20:19 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Set remote url on every getRepo in merger https://review.openstack.org/529293 | 20:19 |
| *** jamesmcarthur has joined #openstack-infra | 20:23 | |
| *** lukebrowning has quit IRC | 20:23 | |
| *** bobh has joined #openstack-infra | 20:24 | |
| *** jamesmcarthur has quit IRC | 20:27 | |
| *** bobh has quit IRC | 20:28 | |
| jeblair | mordred: mechanics of what? | 20:31 |
| jeblair | jlvillal: did you get your question answered? | 20:31 |
| jlvillal | jeblair, No, I didn't | 20:32 |
| jeblair | fungi, tobiash, mordred: i think we need a ram governor in the executor, not just load average. | 20:32 |
| jlvillal | jeblair, I'm trying this, but not totally sure if it works or not: https://review.openstack.org/#/c/529386/ | 20:32 |
| tobiash | jeblair: weren't there patches already for this? | 20:33 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/openstack-zuul-jobs master: Run the CachingScheduler in the nova-next job https://review.openstack.org/529410 | 20:33 |
| tobiash | or am I mistaken there? | 20:33 |
| jeblair | tobiash: maybe? i think SpamapS was going to look into it... i'm not sure the status | 20:33 |
| tobiash | jeblair: https://review.openstack.org/#/c/508960/ | 20:33 |
| fungi | jeblair: may help, but governors based on trailing indicators still have a tendency to get slammed by spikes if it's picking up jobs fast enough | 20:34 |
| fungi | we saw that with the load average governor when there was memory pressure on the scheduler resulting in atypical batching | 20:35 |
| jeblair | fungi: indeed, though i haven't heard a better idea | 20:35 |
| fungi | by the time it measured the load average and noticed it was too high, it was several times higher than the upper bound we configure | 20:35 |
| fungi | i wonder whether there's a way to mark some processes as higher priority so the oom killer will attempt to preserve them | 20:36 |
| jeblair | jlvillal: even the legacy devstack playbooks honor the checked out versions of repos, so you should be able to set 'override-checkout' in the job | 20:36 |
| jlvillal | jeblair, Okay. I haven't heard of that before. I'll search for an example of it. Thanks. | 20:37 |
| jeblair | jlvillal: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.override-checkout | 20:37 |
| jlvillal | Thanks! | 20:37 |
| jeblair | jlvillal: so maybe try a change like the one you linked, but without the depends-on, and using that instead of the var | 20:37 |
| jeblair | jlvillal: (that's basically the zuulv3 implementation of the v2/devstack-gate thing you were poking at) | 20:38 |
| jlvillal | jeblair, Thanks. Will do. | 20:38 |
| tobiash | fungi: for ram I think this could at least work better than load as the measurement ram usage doesn't have a delay of several minutes | 20:38 |
| fungi | tobiash: i agree, i think it's at least a piece of the solution | 20:38 |
| jeblair | then next is probably tuning both of them -- i think we're pretty good on the load average normally. | 20:39 |
| fungi | looks like it's possible to override oom priorities via writing a priority integer into /proc/NNNN/oom_adj | 20:39 |
| fungi | having the oom killer kill the ansible children is way less of a concern than having it kill other executor processes | 20:39 |
| jeblair | fungi: i wonder if there's a way to set that for a pgroup | 20:40 |
| fungi | looking | 20:40 |
| jeblair | or i guess we could leave it normal for ansible procs, and just lower the priority for the 2 executor procs | 20:40 |
| jeblair | or is it raise? "be more important" whichever that direction is. :) | 20:41 |
| fungi | default is 0 i think. higher values make the processes more disposable | 20:41 |
| *** links has joined #openstack-infra | 20:41 | |
| fungi | negative values make it try to preserve them | 20:42 |
| jeblair | ok, so we could just set it to -1 on executor procs | 20:42 |
| jeblair | does that require root or a capability? | 20:42 |
| fungi | not sure yet. still digging in since i've never done oom-killer tuning before | 20:43 |
| jeblair | -rw-r--r-- 1 root root 0 Dec 20 20:43 oom_adj | 20:43 |
| fungi | " Any particular process leader may be immunized against the oom killer if the value of its /proc/<pid>/oomadj is set to the constant OOM_DISABLE (currently defined as -17)." | 20:43 |
| fungi | that suggests to me that it's inherited by the pgroup for the leader | 20:44 |
| fungi | https://linux-mm.org/OOM_Killer | 20:44 |
| jeblair | zuul@ze01:/proc/10839$ echo "-1" > oom_adj | 20:45 |
| jeblair | -su: oom_adj: Permission denied | 20:45 |
| fungi | apparently just nicing the ansible children might have the intended effect? | 20:45 |
| fungi | "'Niced' processes have their scores increased, and long running processes have their scores decreased" | 20:45 |
| jeblair | 'long running processes" | 20:45 |
| jeblair | ? | 20:45 |
| fungi | s = int_sqrt(int_sqrt(run_time)); | 20:45 |
| fungi | which leads me to wonder why we'd kill the web socket listener children | 20:46 |
| *** links has quit IRC | 20:47 | |
| jeblair | fungi: i believe they are threads now, so all the streaming is in one proc | 20:47 |
| fungi | looks like it divides the score by the sqrt of cpu_time and fourth root of run_time | 20:47 |
| jeblair | 0/X = 0 ? | 20:48 |
| *** lukebrowning has joined #openstack-infra | 20:48 | |
| jeblair | so perhaps none of that matters for these since they start with a 0 score? | 20:48 |
| fungi | different score here | 20:48 |
| fungi | i don't think these are the oom_adj value | 20:48 |
| fungi | it's seeded by points = p->mm->total_vm; | 20:49 |
| fungi | so these are basically rules for comparing virtual memory consumed | 20:49 |
| SpamapS | Do I need to pick the RAM governor back up? | 20:49 |
| SpamapS | I think I used the wrong stat to calc available RAM | 20:50 |
| SpamapS | It's worth noting btw.. | 20:50 |
| SpamapS | That we're duplicating a lot of what Kubernetes does. ;) | 20:50 |
| jeblair | SpamapS: that would be great, thanks :) | 20:51 |
| * SpamapS blows the dust off the old patch | 20:51 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:53 | |
| *** felipemonteiro__ has quit IRC | 20:53 | |
| *** jcoufal has quit IRC | 20:53 | |
| *** lukebrowning has quit IRC | 20:53 | |
| *** catintheroof has joined #openstack-infra | 20:54 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for volume backup bug 1739482 https://review.openstack.org/529411 | 20:58 |
| openstack | bug 1739482 in Cinder "test_snapshot_backup fails to build backup due to MessagingTimeout" [Undecided,New] https://launchpad.net/bugs/1739482 | 20:58 |
| *** dprince has joined #openstack-infra | 21:00 | |
| *** dsariel has joined #openstack-infra | 21:00 | |
| *** lukebrowning has joined #openstack-infra | 21:00 | |
| *** jascott1 has joined #openstack-infra | 21:01 | |
| fungi | jeblair: so the web socket listener is a thread of the main daemon process? it looks like child process memory also gets counted for parent processes | 21:01 |
| fungi | as for the adjustment knob, that's used to bitshift the total score | 21:01 |
| *** jascott1 has quit IRC | 21:01 | |
| SpamapS | hm... did the load governor get reverted? | 21:01 |
| jeblair | fungi: it's a child process -- but it has no children itself, only threads. | 21:01 |
| * SpamapS is merging and.. confused | 21:01 | |
| jeblair | SpamapS: not to my knowledge | 21:01 |
| *** jascott1 has joined #openstack-infra | 21:01 | |
| *** ijw has joined #openstack-infra | 21:01 | |
| *** ijw has quit IRC | 21:02 | |
| *** kgiusti has left #openstack-infra | 21:02 | |
| *** ijw has joined #openstack-infra | 21:02 | |
| fungi | so the adjustment knob is basically a log base 2 scale | 21:02 |
| *** kgiusti has joined #openstack-infra | 21:03 | |
| *** priteau has joined #openstack-infra | 21:04 | |
| *** lukebrowning has quit IRC | 21:05 | |
| openstackgerrit | Merged openstack-infra/zuul-base-jobs master: Initial boilerplate, packaging and testing https://review.openstack.org/526139 | 21:06 |
| *** lukebrowning has joined #openstack-infra | 21:06 | |
| SpamapS | oh hah no, the merge was just so perturbed it moved manageLoad 900 lines | 21:07 |
| *** smatzek has joined #openstack-infra | 21:07 | |
| * SpamapS tries a different merge method for his rebase | 21:07 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for VolumeTypes.project lazy-load bug 1739484 https://review.openstack.org/529413 | 21:08 |
| openstack | bug 1739484 in Cinder "DetachedInstanceError trying to lazy-load VolumeTypes.projects" [Undecided,Confirmed] https://launchpad.net/bugs/1739484 | 21:08 |
| *** priteau has quit IRC | 21:09 | |
| *** smatzek has quit IRC | 21:09 | |
| *** jamesmcarthur has joined #openstack-infra | 21:10 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Log unknown providers during quota calculation https://review.openstack.org/529391 | 21:10 |
| pabelanger | clarkb: ^maybe restart to debug more? | 21:12 |
| *** lukebrowning has quit IRC | 21:12 | |
| clarkb | pabelanger: ya its happened more on nl02 | 21:12 |
| clarkb | (but not on 01) | 21:12 |
| pabelanger | k, still need for puppet to run | 21:12 |
| *** lukebrowning has joined #openstack-infra | 21:13 | |
| *** bobh has joined #openstack-infra | 21:15 | |
| *** salv-orlando has joined #openstack-infra | 21:16 | |
| jlvillal | jeblair: Question on the docs: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.run | 21:17 |
| jlvillal | jeblair: So if I supply a job.run, it uses mine and also the parents. Correct? | 21:17 |
| jlvillal | The docs make it sounds like it doesn't. Or at least unclear. | 21:17 |
| *** CrayZee has left #openstack-infra | 21:17 | |
| *** lukebrowning has quit IRC | 21:17 | |
| jeblair | jlvillal: there's only one run, so yours -- not the parent's | 21:18 |
| SamYaple | weekly (monthly?) ask for bindep patch review https://review.openstack.org/#/c/517105/ | 21:18 |
| jlvillal | jeblair: Oh. Okay. Not what I was expecting. Thanks | 21:18 |
| SamYaple | also, im happy to be a maintainer of the bindep project, though i think it falls under infra group scope right now | 21:18 |
| *** caphrim007_ has joined #openstack-infra | 21:20 | |
| *** bobh has quit IRC | 21:20 | |
| *** salv-orlando has quit IRC | 21:20 | |
| *** lukebrowning has joined #openstack-infra | 21:22 | |
| *** slaweq_ has quit IRC | 21:23 | |
| *** catinthe_ has joined #openstack-infra | 21:23 | |
| *** caphrim007 has quit IRC | 21:23 | |
| *** catintheroof has quit IRC | 21:25 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for volume backup bug 1739482 https://review.openstack.org/529411 | 21:25 |
| openstack | bug 1739482 in Cinder "test_snapshot_backup fails to build backup due to MessagingTimeout" [Undecided,New] https://launchpad.net/bugs/1739482 | 21:25 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for VolumeTypes.project lazy-load bug 1739484 https://review.openstack.org/529413 | 21:25 |
| openstack | bug 1739484 in Cinder "DetachedInstanceError trying to lazy-load VolumeTypes.projects" [Undecided,Confirmed] https://launchpad.net/bugs/1739484 | 21:25 |
| *** lukebrowning has quit IRC | 21:27 | |
| *** lukebrowning has joined #openstack-infra | 21:29 | |
| *** lukebrowning has quit IRC | 21:33 | |
| *** bobh has joined #openstack-infra | 21:34 | |
| *** catinthe_ has quit IRC | 21:35 | |
| *** lukebrowning has joined #openstack-infra | 21:35 | |
| *** catintheroof has joined #openstack-infra | 21:35 | |
| *** bnemec has quit IRC | 21:36 | |
| *** threestrands_ has joined #openstack-infra | 21:36 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config master: Use the integration-gate job-template https://review.openstack.org/529417 | 21:37 |
| andreaf | clarkb, mordred, AJaeger, jeblair: this is the second part needed to start using the tempest-full job in the integrated gate ^^^ | 21:37 |
| *** rmcall has quit IRC | 21:38 | |
| *** bobh has quit IRC | 21:38 | |
| *** catintheroof has quit IRC | 21:40 | |
| *** lukebrowning has quit IRC | 21:40 | |
| clarkb | andreaf: that removes comments from mordred about restoring nova net testing? | 21:40 |
| clarkb | andreaf: shoudl we keep that TODO in place? | 21:40 |
| clarkb | or does this also fix nova net testing? | 21:40 |
| *** lukebrowning has joined #openstack-infra | 21:41 | |
| andreaf | clarkb: well as per chat earlier that template is not going to be used - most repos do not have a newton branch anymore and nova is only a matter of one last backport before it's closed | 21:41 |
| *** bobh has joined #openstack-infra | 21:42 | |
| andreaf | clarkb: so I though I'd just leave there the correct integration gates... | 21:42 |
| andreaf | clarkb: but having it in the same change is a bit of a stretch, I can split that into a different one if you prefer so | 21:42 |
| clarkb | andreaf: gotcha, thanks for the clarification | 21:42 |
| clarkb | meh | 21:43 |
| *** kgiusti has left #openstack-infra | 21:44 | |
| *** bobh has quit IRC | 21:47 | |
| clarkb | pabelanger: I'm here now if you are ready to talk gerrit | 21:48 |
| *** ijw has quit IRC | 21:50 | |
| *** trown|ruck is now known as trown|outtypewww | 21:50 | |
| *** priteau has joined #openstack-infra | 21:51 | |
| *** lukebrowning has quit IRC | 21:51 | |
| *** lukebrowning has joined #openstack-infra | 21:52 | |
| *** bobh has joined #openstack-infra | 21:53 | |
| jlvillal | jeblair: I don't think that worked. Seems like devstack-gate checks out master | 21:53 |
| jlvillal | jeblair: http://logs.openstack.org/86/529386/5/check/ironic-dsvm-standalone-pike/ef9a0f2/logs/devstack-gate-setup-workspace-new.txt.gz | 21:53 |
| jlvillal | jeblair: I can see it checking out stable/pike at the start of the job: http://logs.openstack.org/86/529386/5/check/ironic-dsvm-standalone-pike/ef9a0f2/job-output.txt.gz | 21:54 |
| jlvillal | But then I think it gets over-written later | 21:54 |
| *** priteau has quit IRC | 21:56 | |
| *** lukebrowning has quit IRC | 21:58 | |
| *** bobh has quit IRC | 21:58 | |
| *** ralonsoh has quit IRC | 21:58 | |
| *** lukebrowning has joined #openstack-infra | 21:59 | |
| jeblair | jlvillal: hrm, i guess i was wrong about that. you'll probably need to do something like your earlier patch then, if you want to stick with the legacy devstack stuff. or you could convert to new v3 native. | 21:59 |
| jeblair | jlvillal: sorry about that | 21:59 |
| jlvillal | jeblair: Thanks for the help. Yeah we need to switch over at some point. | 21:59 |
| *** ramishra has quit IRC | 21:59 | |
| *** Apoorva_ has joined #openstack-infra | 22:01 | |
| pabelanger | clarkb: any objection if we push it to tomorrow morning? | 22:02 |
| *** jascott1 has quit IRC | 22:03 | |
| *** jascott1 has joined #openstack-infra | 22:03 | |
| *** lukebrowning has quit IRC | 22:04 | |
| *** Apoorva has quit IRC | 22:04 | |
| *** kjackal has quit IRC | 22:05 | |
| *** lukebrowning has joined #openstack-infra | 22:05 | |
| *** jascott1 has quit IRC | 22:05 | |
| clarkb | pabelanger: nope | 22:07 |
| *** Goneri has quit IRC | 22:07 | |
| clarkb | pabelanger: jeblair may logging chagne to nodepool is installed now. I'm going to restart the launchers again | 22:09 |
| clarkb | starting with 01 now | 22:09 |
| *** lukebrowning has quit IRC | 22:10 | |
| pabelanger | kk | 22:10 |
| *** apetrich has quit IRC | 22:11 | |
| *** apetrich has joined #openstack-infra | 22:11 | |
| *** lukebrowning has joined #openstack-infra | 22:11 | |
| *** jascott1 has joined #openstack-infra | 22:15 | |
| *** lukebrowning has quit IRC | 22:16 | |
| clarkb | pabelanger: jeblair tobiash http://paste.openstack.org/show/629505/ seems like the nodes tripping over that are in a deleting state | 22:16 |
| *** fultonj has quit IRC | 22:16 | |
| clarkb | so I think its probably semi safe to ignore them for the sake of quota calculations. | 22:16 |
| *** salv-orlando has joined #openstack-infra | 22:17 | |
| clarkb | I've not seen any issues on 01 going to restart 02 now | 22:17 |
| fungi | diablo_rojo noticed that storyboard-dev is giving a 500 (isr) on logging in. going to see if i can figure out what that's all about | 22:17 |
| *** lukebrowning has joined #openstack-infra | 22:18 | |
| fungi | oof, uptime 582 days! | 22:18 |
| fungi | that's probably a record in rax | 22:18 |
| *** jascott1 has quit IRC | 22:19 | |
| pabelanger | clarkb: we do have some stuck nodes in infracloud in delete | 22:19 |
| pabelanger | wonder if citycloud once were deleted too | 22:20 |
| *** felipemonteiro__ has joined #openstack-infra | 22:20 | |
| clarkb | pabelanger: those are both clouds on the nl02 side but will be interesting if they show up in similar log messages | 22:20 |
| *** jascott1 has joined #openstack-infra | 22:21 | |
| *** salv-orlando has quit IRC | 22:22 | |
| *** lukebrowning has quit IRC | 22:23 | |
| *** e0ne has quit IRC | 22:23 | |
| *** Goneri has joined #openstack-infra | 22:23 | |
| *** felipemonteiro_ has quit IRC | 22:24 | |
| *** bobh has joined #openstack-infra | 22:24 | |
| *** david-lyle has quit IRC | 22:24 | |
| *** lukebrowning has joined #openstack-infra | 22:24 | |
| *** jascott1 has quit IRC | 22:26 | |
| *** e0ne has joined #openstack-infra | 22:26 | |
| *** e0ne has quit IRC | 22:26 | |
| *** e0ne has joined #openstack-infra | 22:27 | |
| *** jascott1 has joined #openstack-infra | 22:27 | |
| *** e0ne has quit IRC | 22:28 | |
| *** e0ne has joined #openstack-infra | 22:28 | |
| *** bobh has quit IRC | 22:28 | |
| *** e0ne has quit IRC | 22:28 | |
| *** lukebrowning has quit IRC | 22:29 | |
| *** jascott1 has quit IRC | 22:29 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: SearchEngine search only supports ints for days https://review.openstack.org/320988 | 22:29 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for Ansible privilege escalation timeout https://review.openstack.org/505233 | 22:29 |
| *** jascott1 has joined #openstack-infra | 22:30 | |
| *** lukebrowning has joined #openstack-infra | 22:30 | |
| *** rcernin has joined #openstack-infra | 22:32 | |
| *** jascott1 has quit IRC | 22:33 | |
| *** jascott1 has joined #openstack-infra | 22:34 | |
| *** jappleii__ has joined #openstack-infra | 22:35 | |
| *** lukebrowning has quit IRC | 22:35 | |
| dmsimard | I don't know what byobu is | 22:36 |
| SpamapS | nor do you want to ;) | 22:36 |
| dmsimard | but it's snuck its way in lol | 22:36 |
| *** threestrands_ has quit IRC | 22:36 | |
| *** lukebrowning has joined #openstack-infra | 22:36 | |
| dmsimard | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/server.pp#n228 | 22:37 |
| dmsimard | http://paste.openstack.org/raw/629509/ | 22:37 |
| *** jascott1 has quit IRC | 22:37 | |
| dmsimard | it was once 98 but NOT ANYMORE | 22:37 |
| dmsimard | "Byobu is a GPLv3 open source text-based window manager and terminal multiplexer" so.. like screen and tmux ? | 22:38 |
| fungi | did we recently merge a change which would have caused "no check-requirements job specified for openstack/django_openstack_auth" failures to start up for the requirements-tox-validate-projects job on all openstack/requirements changes? | 22:38 |
| *** jascott1 has joined #openstack-infra | 22:40 | |
| *** ijw has joined #openstack-infra | 22:40 | |
| SpamapS | dmsimard: it's just a config for screen or tmux | 22:40 |
| fungi | aha... https://review.openstack.org/529080 Drop django_openstack_auth master branch jobs | 22:40 |
| fungi | we should have also removed it from projects.txt in the reqs repo i guess | 22:41 |
| *** mixos has joined #openstack-infra | 22:41 | |
| *** lukebrowning has quit IRC | 22:42 | |
| dmsimard | SpamapS: wow, Ubuntu doesn't like if you want to remove that package... http://paste.openstack.org/raw/629510/ | 22:42 |
| dmsimard | you want to remove that? Let me remove... ubuntu-server | 22:42 |
| *** lukebrowning has joined #openstack-infra | 22:43 | |
| dmsimard | fungi: does it need to be removed elsewhere ? | 22:43 |
| dmsimard | it sounds like we're missing a few steps if the intent is to retire that repo | 22:43 |
| *** jascott1 has quit IRC | 22:44 | |
| fungi | dmsimard: smcginnis says there's a change doing that... looking for it now | 22:44 |
| *** aviau has quit IRC | 22:44 | |
| smcginnis | https://review.openstack.org/#/c/529067/ | 22:45 |
| dmsimard | fungi: that's projects.txt -- but I expected other patches like the gerrit ACL and so on. | 22:47 |
| fungi | dmsimard: it's not a complete retirement because the lib is still used by stable branches of some projects | 22:47 |
| dmsimard | oh, makes sense | 22:47 |
| *** jascott1 has joined #openstack-infra | 22:48 | |
| fungi | we realized that our repo retirement process has a big hole regarding retiring projects still used by stable branches of other active projects | 22:48 |
| *** lukebrowning has quit IRC | 22:48 | |
| fungi | at least where it's being used from source and not just released packages | 22:48 |
| fungi | #status log enqueued 529067,1 into the gate pipeline and promoted in order to unblock requirements and release changes | 22:49 |
| openstackstatus | fungi: finished logging | 22:49 |
| *** lukebrowning has joined #openstack-infra | 22:49 | |
| *** caphrim007_ has quit IRC | 22:50 | |
| *** caphrim007 has joined #openstack-infra | 22:50 | |
| *** jascott1 has quit IRC | 22:52 | |
| *** ijw has quit IRC | 22:53 | |
| *** lukebrowning has quit IRC | 22:54 | |
| *** dsariel has quit IRC | 22:54 | |
| *** bobh has joined #openstack-infra | 22:54 | |
| *** lukebrowning has joined #openstack-infra | 22:55 | |
| *** mixos has quit IRC | 22:56 | |
| *** david-lyle has joined #openstack-infra | 22:56 | |
| *** bobh has quit IRC | 22:59 | |
| *** lukebrowning has quit IRC | 23:00 | |
| *** lukebrowning has joined #openstack-infra | 23:02 | |
| SpamapS | ugh.. executor server changed so much git is too confused to rebase my mem governor | 23:02 |
| * SpamapS has to hand merge | 23:02 | |
| fungi | years ago i used to hand-merge all manner of patchfiles with no revision control. i've almost forgotten what that was like | 23:03 |
| SpamapS | vimdiff and a stiff drink | 23:03 |
| fungi | shoot-me-now.c.rej | 23:04 |
| clarkb | fungi: not even rcs? | 23:04 |
| fungi | sadly, no | 23:04 |
| fungi | do versioned tarballs count as revision control? ;) | 23:05 |
| fungi | i never really used rcs, didn't learn about revision control until cvs days | 23:05 |
| clarkb | all this bind zone file stuff has remidned me we managed all public DNS for a particular corporation using zone files in rcs | 23:05 |
| clarkb | then ran a script to sync the updates to all the servers and reload the service | 23:05 |
| fungi | clarkb: ooh, for that i had written a fancy visudo sort of wrapper which used nslint and rejected edits and kicked you back into the temporary editing session. those were the days | 23:06 |
| *** jascott1 has joined #openstack-infra | 23:06 | |
| fungi | also had a bunch of custom linting checks | 23:07 |
| *** lukebrowning has quit IRC | 23:07 | |
| fungi | autoincremented the zone serial | 23:07 |
| fungi | kicked ndc on commit, et cetera | 23:07 |
| clarkb | nice | 23:07 |
| clarkb | one neat thing about this was there was like 20 yaers or more of rcs history you could dig through | 23:07 |
| fungi | had a mode to insert/remove/alter zone entries from the bind config and restart bind too | 23:08 |
| clarkb | you can kinda see a marketing history of a company through that | 23:08 |
| *** lukebrowning has joined #openstack-infra | 23:08 | |
| fungi | indeed | 23:08 |
| fungi | but we kept hiring greener and greener people for hosting ops, until i was eventually pressured to put in a webui for dns management | 23:09 |
| clarkb | we used something off the shelf for ipam and dns for internal dns | 23:09 |
| fungi | text editors were hard for that generation of sysadmins to figure out, apparently | 23:09 |
| *** rmcall has joined #openstack-infra | 23:09 | |
| clarkb | it aws terrible because you couldn't see the history of edits as easily (and I'm comparing to rcs in 2012) | 23:10 |
| clarkb | but you could clikc a button and search without grep | 23:10 |
| *** rmcall has quit IRC | 23:10 | |
| *** dprince has quit IRC | 23:10 | |
| *** rmcall has joined #openstack-infra | 23:10 | |
| fungi | just remembered another detail. my script would pause at the end and collect bind logs, then filter for errors and display them to you. that was also super handy | 23:11 |
| fungi | so you got to have the "oh nuts!" moment more or less immediately, instead of waiting for your pager to go off and tell you you'd made a serious typo | 23:12 |
| fungi | or, better still, the help desk calling to ask you if they could bridge you into an incident call with some angry customer and a few division heads | 23:12 |
| *** lukebrowning has quit IRC | 23:13 | |
| fungi | those were by far my favorite | 23:13 |
| *** bobh has joined #openstack-infra | 23:14 | |
| *** lukebrowning has joined #openstack-infra | 23:14 | |
| fungi | #status log updated storyboard-dev openids from login.launchpad.net to login.ubuntu.com to solve DBDuplicateEntry exceptions on login | 23:15 |
| openstackstatus | fungi: finished logging | 23:15 |
| fungi | mordred: ^ maybe that's what you were seeing a while back? | 23:15 |
| *** dave-mccowan has quit IRC | 23:17 | |
| *** salv-orlando has joined #openstack-infra | 23:17 | |
| *** bobh has quit IRC | 23:18 | |
| *** lukebrowning has quit IRC | 23:19 | |
| *** salv-orlando has quit IRC | 23:21 | |
| *** lukebrowning has joined #openstack-infra | 23:24 | |
| *** Apoorva_ has quit IRC | 23:24 | |
| *** felipemonteiro__ has quit IRC | 23:24 | |
| *** felipemonteiro__ has joined #openstack-infra | 23:24 | |
| *** bobh has joined #openstack-infra | 23:25 | |
| *** Apoorva has joined #openstack-infra | 23:25 | |
| *** xarses_ has quit IRC | 23:29 | |
| *** lukebrowning has quit IRC | 23:29 | |
| *** bobh has quit IRC | 23:29 | |
| *** dhill_ has joined #openstack-infra | 23:30 | |
| *** lukebrowning has joined #openstack-infra | 23:30 | |
| *** lukebrowning_ has joined #openstack-infra | 23:32 | |
| *** hashar has quit IRC | 23:33 | |
| *** bobh has joined #openstack-infra | 23:33 | |
| *** edmondsw has joined #openstack-infra | 23:33 | |
| *** lukebrowning has quit IRC | 23:34 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Remove byobu profile configuration https://review.openstack.org/529426 | 23:36 |
| *** bobh has quit IRC | 23:37 | |
| *** lukebrowning_ has quit IRC | 23:37 | |
| *** edmondsw has quit IRC | 23:38 | |
| *** lukebrowning has joined #openstack-infra | 23:38 | |
| *** felipemonteiro__ has quit IRC | 23:41 | |
| *** rbrndt has quit IRC | 23:43 | |
| *** rbrndt has joined #openstack-infra | 23:43 | |
| *** rbrndt has quit IRC | 23:43 | |
| *** bmjen has joined #openstack-infra | 23:43 | |
| *** Hunner_ has joined #openstack-infra | 23:44 | |
| *** Hunner_ has quit IRC | 23:45 | |
| *** Hunner_ has joined #openstack-infra | 23:45 | |
| *** jamesmcarthur has quit IRC | 23:46 | |
| *** felipemonteiro__ has joined #openstack-infra | 23:46 | |
| *** rlandy|rover is now known as rlandy|bbl | 23:47 | |
| *** lukebrowning has quit IRC | 23:48 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Setup bash history to contain standard timestamps (RFC3339) https://review.openstack.org/529428 | 23:52 |
| dmsimard | infra-root: some of you might find this last patch useful ^ | 23:52 |
| *** ijw has joined #openstack-infra | 23:53 | |
| jeblair | i'm going to poke at the zuul apache rewrite rules for a bit; hopefully i won't break them, but i might | 23:56 |
| johnsom | 404: Not Found on the zuulv3 dashboard | 23:58 |
| *** ijw has quit IRC | 23:59 | |
| jeblair | or, rather, i might break them briefly :) | 23:59 |
| johnsom | lol | 23:59 |
| johnsom | Honest, I went to the dashboard link first.... | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!