| rosmaita | mordred thanks for putting up that subunit test patch for glanceclient, looks like maybe ansible wraps output and your script isn't expecting that? http://logs.openstack.org/98/535398/1/check/openstack-tox-py27/46f46cf/ara/result/67065662-366c-4bff-9b45-a90c1a4680e4/ | 00:00 |
|---|---|---|
| rosmaita | and AJaeger is right, ara is great for troubleshooting! | 00:00 |
| mordred | corvus: sweet | 00:00 |
| mordred | rosmaita: oh - so you're right! | 00:01 |
| clarkb | mordred: corvus I'd like to check the job logs before approving but will try t oremmeber to do that today before I afk | 00:01 |
| mordred | rosmaita: and also - dhellmann said that but I thought he was wrong. he was clearly correct | 00:01 |
| rosmaita | i have to adimit, i have not found dhellmann to be wrong very often :) | 00:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make subunit processing more resilient https://review.openstack.org/534431 | 00:03 |
| mordred | rosmaita: me either. :) | 00:04 |
| mordred | rosmaita: ^^ that should fix that error - rechecked the glanceclient patch too | 00:04 |
| dmsimard | dhellmann is the voice of wisdom and knowledge :D | 00:06 |
| rosmaita | mordred ty! | 00:06 |
| *** yamamoto has joined #openstack-infra | 00:06 | |
| *** dsariel has quit IRC | 00:07 | |
| *** spzala has quit IRC | 00:09 | |
| *** david-lyle has quit IRC | 00:10 | |
| *** dklyle has joined #openstack-infra | 00:10 | |
| *** spzala has joined #openstack-infra | 00:10 | |
| *** jamesmcarthur has quit IRC | 00:11 | |
| *** caphrim007_ has joined #openstack-infra | 00:26 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: license: remove dos line break https://review.openstack.org/535531 | 00:27 |
| openstackgerrit | Merged openstack-infra/storyboard master: fix rst formatting https://review.openstack.org/533307 | 00:28 |
| *** hongbin has quit IRC | 00:28 | |
| *** caphrim00_ has joined #openstack-infra | 00:28 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add noop job description https://review.openstack.org/535534 | 00:29 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: pep8: ignore E124 rule https://review.openstack.org/535536 | 00:30 |
| *** caphrim00_ has quit IRC | 00:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: license: remove dos line break https://review.openstack.org/535531 | 00:30 |
| *** caphrim007 has quit IRC | 00:30 | |
| *** caphrim007 has joined #openstack-infra | 00:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: use kazoo retry facilities https://review.openstack.org/535537 | 00:30 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add log_stream_port and log_stream_file settings https://review.openstack.org/535538 | 00:30 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: scheduler: better handle format status error https://review.openstack.org/535539 | 00:31 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: requirements: remove paramiko <2.0 cap https://review.openstack.org/535540 | 00:31 |
| *** caphrim007_ has quit IRC | 00:31 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add OpenAPI documentation https://review.openstack.org/535541 | 00:31 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: ansible: honor command no_log module attribute https://review.openstack.org/535542 | 00:32 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 00:32 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 00:33 |
| *** caphrim007 has quit IRC | 00:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add restricted-node-labels global limits https://review.openstack.org/535546 | 00:35 |
| *** dhajare|BRB has joined #openstack-infra | 00:38 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Ensure build.start_time is defined https://review.openstack.org/535549 | 00:40 |
| *** ameliac_ has joined #openstack-infra | 00:41 | |
| *** ameliac has quit IRC | 00:42 | |
| *** tosky has quit IRC | 00:43 | |
| *** bobh has joined #openstack-infra | 00:44 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: handler: fix support for handler without launch_manager https://review.openstack.org/535550 | 00:44 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add a plugin interface for drivers https://review.openstack.org/535551 | 00:46 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: builder: do not cleanup image for driver not managing image https://review.openstack.org/535552 | 00:46 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a static driver for Nodepool https://review.openstack.org/535553 | 00:47 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor run_handler to be generic https://review.openstack.org/535554 | 00:47 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 00:48 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 00:49 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 00:49 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an Amazon EC2 driver https://review.openstack.org/535558 | 00:50 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: check for client in properties https://review.openstack.org/535559 | 00:51 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: config: add statsd-server config parameter https://review.openstack.org/535560 | 00:51 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add /node-list to the webapp https://review.openstack.org/535562 | 00:52 |
| *** dklyle is now known as david-lyle | 00:52 | |
| *** bobh has quit IRC | 00:52 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add /label-list to the webapp https://review.openstack.org/535563 | 00:53 |
| *** cuongnv has joined #openstack-infra | 00:53 | |
| *** jamesmcarthur has joined #openstack-infra | 00:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make subunit processing more resilient https://review.openstack.org/534431 | 00:56 |
| *** armaan has joined #openstack-infra | 01:00 | |
| *** Swami has quit IRC | 01:00 | |
| *** jamesmcarthur has quit IRC | 01:00 | |
| *** pramodrj07 has joined #openstack-infra | 01:05 | |
| *** bobh has joined #openstack-infra | 01:06 | |
| *** armaan has quit IRC | 01:06 | |
| *** smatzek has joined #openstack-infra | 01:08 | |
| *** lbragstad has joined #openstack-infra | 01:09 | |
| *** jbadiapa has quit IRC | 01:11 | |
| *** yamahata has quit IRC | 01:11 | |
| *** kiennt26 has joined #openstack-infra | 01:12 | |
| *** smatzek has quit IRC | 01:12 | |
| *** edmondsw has joined #openstack-infra | 01:12 | |
| *** kiennt26_ has joined #openstack-infra | 01:13 | |
| *** kiennt26_ has quit IRC | 01:14 | |
| *** iyamahat has quit IRC | 01:14 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added update attendee endpoint https://review.openstack.org/535521 | 01:16 |
| *** edmondsw has quit IRC | 01:16 | |
| *** bobh has quit IRC | 01:16 | |
| *** daidv has quit IRC | 01:22 | |
| *** daidv has joined #openstack-infra | 01:22 | |
| *** zhurong has joined #openstack-infra | 01:30 | |
| *** smatzek has joined #openstack-infra | 01:39 | |
| *** bobh has joined #openstack-infra | 01:41 | |
| *** smatzek has quit IRC | 01:46 | |
| *** caphrim007 has joined #openstack-infra | 01:55 | |
| *** liujiong has joined #openstack-infra | 01:55 | |
| *** caphrim007 has quit IRC | 01:57 | |
| *** ykarel|away has joined #openstack-infra | 01:57 | |
| zhurong | hi, infra, cloud some give me some suggestions, thx, I am working on a test job, with post_test_hook, the scripts didn't terminate even with exit 1 but terminate with timeout. patch is: https://review.openstack.org/#/c/532714/ and the log is: http://logs.openstack.org/14/532714/8/check/murano-dashboard-sanity-check/3cd0d9e/job-output.txt.gz#_2018-01-17_09_29_58_803288 | 01:58 |
| *** kukacz_ has quit IRC | 02:00 | |
| *** weshay|dentist is now known as weshay|ruck | 02:00 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 02:00 | |
| *** kukacz_ has joined #openstack-infra | 02:01 | |
| *** pramodrj07 has quit IRC | 02:08 | |
| *** tinwood has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-infra | 02:11 | |
| *** spzala has quit IRC | 02:20 | |
| *** spzala has joined #openstack-infra | 02:21 | |
| *** spzala has quit IRC | 02:21 | |
| *** spzala has joined #openstack-infra | 02:21 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Endpoint add attendee ticket https://review.openstack.org/535571 | 02:24 |
| *** rosmaita has quit IRC | 02:25 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Endpoint add attendee ticket https://review.openstack.org/535571 | 02:25 |
| *** rlandy has quit IRC | 02:26 | |
| *** gcb has joined #openstack-infra | 02:28 | |
| *** ekcs has joined #openstack-infra | 02:30 | |
| *** harlowja has quit IRC | 02:31 | |
| *** dhajare|BRB has quit IRC | 02:31 | |
| *** olaph has quit IRC | 02:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: block stat get_mime on localhost https://review.openstack.org/535576 | 02:34 |
| *** bobh has quit IRC | 02:36 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Endpoint delete atendee ticket https://review.openstack.org/535577 | 02:38 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Endpoint delete atendee ticket https://review.openstack.org/535577 | 02:38 |
| *** hongbin has joined #openstack-infra | 02:39 | |
| *** olaph has joined #openstack-infra | 02:39 | |
| *** namnh has joined #openstack-infra | 02:40 | |
| *** owalsh_ is now known as owalsh | 02:45 | |
| *** ramishra has joined #openstack-infra | 02:47 | |
| *** threestrands_ has joined #openstack-infra | 02:49 | |
| *** ykarel|away has quit IRC | 02:49 | |
| *** threestrands has quit IRC | 02:51 | |
| *** ekcs has quit IRC | 02:55 | |
| *** mnencia has quit IRC | 03:04 | |
| *** olaph1 has joined #openstack-infra | 03:04 | |
| *** olaph has quit IRC | 03:06 | |
| *** andreaf has quit IRC | 03:07 | |
| *** lbragstad has quit IRC | 03:08 | |
| *** gouthamr has joined #openstack-infra | 03:17 | |
| *** gouthamr has quit IRC | 03:21 | |
| *** mnencia has joined #openstack-infra | 03:22 | |
| *** gouthamr has joined #openstack-infra | 03:27 | |
| *** liujiong has quit IRC | 03:30 | |
| *** liujiong has joined #openstack-infra | 03:30 | |
| *** psachin has joined #openstack-infra | 03:32 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard master: Add is_deprecated field for tags https://review.openstack.org/535586 | 03:35 |
| *** yamahata has joined #openstack-infra | 03:35 | |
| *** andreaf has joined #openstack-infra | 03:36 | |
| *** dhajare|BRB has joined #openstack-infra | 03:39 | |
| *** sree has joined #openstack-infra | 03:40 | |
| *** jappleii__ has joined #openstack-infra | 03:44 | |
| *** ganso has quit IRC | 03:44 | |
| *** jappleii__ has quit IRC | 03:45 | |
| *** caphrim007 has joined #openstack-infra | 03:45 | |
| *** jappleii__ has joined #openstack-infra | 03:45 | |
| clarkb | zhurong: if I had to guess its something with start xvfb? | 03:46 |
| *** jappleii__ has quit IRC | 03:46 | |
| *** threestrands_ has quit IRC | 03:46 | |
| clarkb | xvfb will run in foreground iirc | 03:46 |
| *** lbragstad has joined #openstack-infra | 03:46 | |
| *** jappleii__ has joined #openstack-infra | 03:47 | |
| *** jappleii__ has quit IRC | 03:48 | |
| *** jappleii__ has joined #openstack-infra | 03:48 | |
| *** ihrachys has quit IRC | 03:49 | |
| *** rcernin has quit IRC | 03:49 | |
| *** ihrachys has joined #openstack-infra | 03:49 | |
| *** rcernin has joined #openstack-infra | 03:49 | |
| *** caphrim007_ has joined #openstack-infra | 03:50 | |
| openstackgerrit | Merged openstack-infra/storyboard master: fix rst heading styles https://review.openstack.org/533356 | 03:53 |
| *** caphrim007 has quit IRC | 03:53 | |
| *** coolsvap has joined #openstack-infra | 03:57 | |
| prometheanfire | can you run recheck/run specific jobs/tests? | 03:58 |
| clarkb | no | 04:01 |
| clarkb | thats somewhat intentional to avoid locking in results and retrying until all are green. I've tracked back at least one major gate reset bug to a change being rechecked enough times to go through even without that :/ | 04:02 |
| prometheanfire | didn't think so, thought someone made in a review trying to do so | 04:02 |
| *** lbragstad has quit IRC | 04:02 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Add documentation on manually upgrading a StoryBoard instance https://review.openstack.org/510669 | 04:02 |
| *** hamzy_ has joined #openstack-infra | 04:03 | |
| *** felipemonteiro_ has joined #openstack-infra | 04:05 | |
| *** gouthamr has quit IRC | 04:08 | |
| *** bobh has joined #openstack-infra | 04:08 | |
| *** gouthamr has joined #openstack-infra | 04:09 | |
| *** gouthamr has quit IRC | 04:12 | |
| *** gouthamr has joined #openstack-infra | 04:13 | |
| *** ameliac_ has quit IRC | 04:17 | |
| *** gouthamr has quit IRC | 04:23 | |
| *** gouthamr has joined #openstack-infra | 04:24 | |
| *** armax has quit IRC | 04:26 | |
| *** felipemonteiro_ has quit IRC | 04:26 | |
| *** janki has joined #openstack-infra | 04:26 | |
| *** caphrim007_ has quit IRC | 04:26 | |
| *** caphrim007 has joined #openstack-infra | 04:27 | |
| *** armax has joined #openstack-infra | 04:27 | |
| *** caphrim007_ has joined #openstack-infra | 04:28 | |
| *** gouthamr has quit IRC | 04:29 | |
| *** gouthamr has joined #openstack-infra | 04:29 | |
| *** caphrim007 has quit IRC | 04:32 | |
| smcginnis | Just passing by and checking on things, so I can ask again in my morning if it ends up a better time, but... | 04:32 |
| *** gouthamr has quit IRC | 04:32 | |
| smcginnis | We had a release job that failed - looks like due to a timeout on a git fetch: | 04:32 |
| smcginnis | http://logs.openstack.org/3a/3a91791c39bb9e56fbc9194d32ef13036352ae2e/release-post/tag-releases/814f6b1/job-output.txt.gz#_2018-01-19_00_34_47_368175 | 04:32 |
| smcginnis | If someone from infra could re-enqueue that job, it looks like it was transient and should hopefully succeed the next time around. | 04:33 |
| *** gouthamr has joined #openstack-infra | 04:33 | |
| *** bobh has quit IRC | 04:33 | |
| *** ykarel|away has joined #openstack-infra | 04:36 | |
| *** bobh has joined #openstack-infra | 04:37 | |
| *** links has joined #openstack-infra | 04:37 | |
| *** namnh has quit IRC | 04:38 | |
| *** zhurong has quit IRC | 04:39 | |
| *** dhajare|BRB has quit IRC | 04:41 | |
| *** pgadiya has joined #openstack-infra | 04:42 | |
| *** bobh has quit IRC | 04:44 | |
| ianw | smcginnis: should be running as http://zuulv3.openstack.org/stream.html?uuid=629d7792cd0d4b2aa95f59d28da4a6ab&logfile=console.log | 04:46 |
| *** ramishra has quit IRC | 04:46 | |
| dmsimard | wow... RDO was basically getting attacked by a crawler bot | 04:46 |
| dmsimard | http://paste.openstack.org/raw/646970/ | 04:46 |
| dmsimard | the site looks super legit https://www.semrush.com/bot/ but in reality when you google it, everyone wants to know how they can block them | 04:47 |
| dmsimard | http://paste.openstack.org/raw/646971/ | 04:49 |
| *** ramishra has joined #openstack-infra | 04:52 | |
| *** andreas_s has joined #openstack-infra | 04:54 | |
| dmsimard | There was another one like that: http://paste.openstack.org/raw/646972/ | 04:56 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Enable link detection in the markdown parser https://review.openstack.org/534461 | 04:57 |
| *** andreas_s has quit IRC | 04:58 | |
| dmsimard | They're on git.openstack.org as well. | 05:00 |
| dmsimard | http://paste.openstack.org/raw/646974/ | 05:01 |
| dmsimard | infra-root: if anyone is around to ack ^ I just firewall'd them off from RDO | 05:02 |
| dmsimard | could do the same thing here | 05:02 |
| ianw | i'm not aware we have a blacklist like that. i'd be wary of doing it manually in iptables, because we'll either forget about it and why we did it, or puppet will overwrite it, neither particularly good | 05:08 |
| *** jamesmcarthur has joined #openstack-infra | 05:12 | |
| dmsimard | ianw: do we have a default set of iptables rules ? or do we need to add it everywhere ? | 05:13 |
| dmsimard | ianw: I guess this would be covered under security groups actually | 05:13 |
| dmsimard | (rather than iptables on-vm) | 05:13 |
| dmsimard | that's actually an excellent idea and I'm going to go ahead and add that now for RDO | 05:14 |
| ianw | i dunno, maybe iptables for all hosts, or maybe there's something in haproxy? | 05:15 |
| *** jamesmcarthur has quit IRC | 05:16 | |
| *** zhurong has joined #openstack-infra | 05:19 | |
| zhurong | clarkb you mean I should kill xvfb first, then exit EXIT_CODE? | 05:21 |
| *** harlowja has joined #openstack-infra | 05:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Make subunit processing more resilient https://review.openstack.org/534431 | 05:22 |
| *** hongbin has quit IRC | 05:24 | |
| *** rkukura has quit IRC | 05:25 | |
| *** bstinson has quit IRC | 05:28 | |
| *** ramishra has quit IRC | 05:28 | |
| *** rkukura has joined #openstack-infra | 05:33 | |
| *** dhajare|BRB has joined #openstack-infra | 05:38 | |
| *** ramishra has joined #openstack-infra | 05:45 | |
| *** bstinson has joined #openstack-infra | 05:46 | |
| *** bhavik1 has joined #openstack-infra | 05:49 | |
| *** jappleii__ has quit IRC | 05:52 | |
| *** cshastri_ has joined #openstack-infra | 05:58 | |
| *** bhavik1 has quit IRC | 05:58 | |
| *** dhajare|BRB has quit IRC | 06:00 | |
| *** armax has quit IRC | 06:03 | |
| *** jaosorior has joined #openstack-infra | 06:11 | |
| *** logan- has quit IRC | 06:16 | |
| *** dhajare|BRB has joined #openstack-infra | 06:18 | |
| *** gouthamr has quit IRC | 06:28 | |
| *** armaan has joined #openstack-infra | 06:29 | |
| *** harlowja has quit IRC | 06:29 | |
| *** dhajare|BRB has quit IRC | 06:31 | |
| *** annp has joined #openstack-infra | 06:33 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: unittests: make bindep role optional https://review.openstack.org/522255 | 06:34 |
| *** cshastri_ has quit IRC | 06:34 | |
| *** dhill_ has quit IRC | 06:38 | |
| *** logan- has joined #openstack-infra | 06:40 | |
| *** pcaruana has joined #openstack-infra | 06:44 | |
| *** andreas_s has joined #openstack-infra | 06:49 | |
| *** ramishra has quit IRC | 07:01 | |
| *** lamt has quit IRC | 07:02 | |
| *** pleia2 has quit IRC | 07:02 | |
| *** namnh has joined #openstack-infra | 07:03 | |
| *** ramishra has joined #openstack-infra | 07:13 | |
| *** kjackal has quit IRC | 07:20 | |
| *** kjackal has joined #openstack-infra | 07:21 | |
| *** pleia2 has joined #openstack-infra | 07:28 | |
| *** daidv has quit IRC | 07:33 | |
| *** daidv has joined #openstack-infra | 07:33 | |
| *** b_bezak has joined #openstack-infra | 07:35 | |
| *** jbadiapa has joined #openstack-infra | 07:36 | |
| *** vsaienk0 has joined #openstack-infra | 07:37 | |
| *** kjackal has quit IRC | 07:38 | |
| *** slaweq has joined #openstack-infra | 07:40 | |
| *** rcernin has quit IRC | 07:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Partial revert "Revert "Remove diskimages section from nodepool-launchers"" https://review.openstack.org/535653 | 07:44 |
| *** janki has quit IRC | 07:45 | |
| *** amoralej|off is now known as amoralej | 07:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Partial revert "Revert "Remove diskimages section from nodepool-launchers"" https://review.openstack.org/535653 | 07:48 |
| AJaeger | pabelanger: ^ | 07:48 |
| AJaeger | frickler, ianw : could you review https://review.openstack.org/535462 , please? before the next wave of requirements changes merges | 07:51 |
| *** zoli is now known as zoli|trng | 07:58 | |
| *** zoli|trng is now known as zoli | 07:58 | |
| *** ykarel|away is now known as ykarel | 08:04 | |
| *** ykarel is now known as ykarel|lunch | 08:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Partial revert "Revert "Remove diskimages section from nodepool-launchers"" https://review.openstack.org/535653 | 08:05 |
| *** zeus has quit IRC | 08:05 | |
| AJaeger | infra-root, we missed one place where we checked out the zuulv3/feature branch for nodepool ^ | 08:06 |
| *** kuromagi has quit IRC | 08:08 | |
| *** rm_work has quit IRC | 08:08 | |
| openstackgerrit | Gauvain Pocentek proposed openstack-infra/project-config master: Disable Cerberus/Sticks jobs for retirement https://review.openstack.org/535660 | 08:09 |
| *** kuromagi has joined #openstack-infra | 08:12 | |
| *** gpocentek has joined #openstack-infra | 08:13 | |
| *** sree has quit IRC | 08:13 | |
| *** sree has joined #openstack-infra | 08:13 | |
| *** markvoelker has quit IRC | 08:14 | |
| *** tesseract has joined #openstack-infra | 08:16 | |
| *** alexchadin has joined #openstack-infra | 08:16 | |
| *** harlowja has joined #openstack-infra | 08:17 | |
| *** sree has quit IRC | 08:18 | |
| *** rm_work has joined #openstack-infra | 08:20 | |
| *** aviau has quit IRC | 08:20 | |
| *** aviau has joined #openstack-infra | 08:20 | |
| AJaeger | config-core, let's merge this one, please ^ | 08:24 |
| *** namnh_ has joined #openstack-infra | 08:26 | |
| *** jmccrory has quit IRC | 08:26 | |
| *** akscram1 has quit IRC | 08:27 | |
| *** sdake has quit IRC | 08:27 | |
| *** akscram1 has joined #openstack-infra | 08:27 | |
| *** gus has quit IRC | 08:27 | |
| *** bmjen has quit IRC | 08:28 | |
| *** aspiers[m] has quit IRC | 08:28 | |
| *** bmjen has joined #openstack-infra | 08:28 | |
| *** persia has quit IRC | 08:28 | |
| *** flwang has quit IRC | 08:28 | |
| *** Hunner has quit IRC | 08:28 | |
| *** StevenK has quit IRC | 08:28 | |
| *** kiennt26 has quit IRC | 08:28 | |
| *** annp has quit IRC | 08:28 | |
| *** namnh has quit IRC | 08:28 | |
| *** persia has joined #openstack-infra | 08:28 | |
| *** ykarel|lunch is now known as ykarel | 08:28 | |
| *** gus has joined #openstack-infra | 08:29 | |
| *** annp has joined #openstack-infra | 08:29 | |
| *** kiennt26 has joined #openstack-infra | 08:29 | |
| *** cuongnv has quit IRC | 08:29 | |
| *** StevenK has joined #openstack-infra | 08:29 | |
| *** cuongnv has joined #openstack-infra | 08:29 | |
| *** sdake has joined #openstack-infra | 08:33 | |
| *** sdake has quit IRC | 08:33 | |
| *** sdake has joined #openstack-infra | 08:33 | |
| *** jmccrory has joined #openstack-infra | 08:33 | |
| *** jamielennox has quit IRC | 08:33 | |
| *** Hunner has joined #openstack-infra | 08:33 | |
| *** Hunner has quit IRC | 08:33 | |
| *** Hunner has joined #openstack-infra | 08:33 | |
| *** harlowja has quit IRC | 08:37 | |
| *** namnh has joined #openstack-infra | 08:40 | |
| *** flepied has quit IRC | 08:42 | |
| *** aspiers[m] has joined #openstack-infra | 08:42 | |
| *** namnh_ has quit IRC | 08:43 | |
| *** flwang has joined #openstack-infra | 08:44 | |
| *** ralonsoh has joined #openstack-infra | 08:47 | |
| *** jmccrory has quit IRC | 08:50 | |
| *** jmccrory has joined #openstack-infra | 08:52 | |
| *** namnh has quit IRC | 08:54 | |
| *** jpena|off is now known as jpena | 08:54 | |
| *** dizquierdo has joined #openstack-infra | 08:55 | |
| *** rm_work has quit IRC | 08:56 | |
| *** jpich has joined #openstack-infra | 08:59 | |
| *** cshastri_ has joined #openstack-infra | 09:01 | |
| *** shardy has joined #openstack-infra | 09:03 | |
| *** bradm has quit IRC | 09:05 | |
| *** makowals has quit IRC | 09:06 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Status branch protection checking for github https://review.openstack.org/535680 | 09:10 |
| *** e0ne has joined #openstack-infra | 09:10 | |
| *** e0ne has quit IRC | 09:10 | |
| *** dsariel has joined #openstack-infra | 09:11 | |
| *** jamesmcarthur has joined #openstack-infra | 09:13 | |
| *** jamesmcarthur has quit IRC | 09:17 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Don't gather host keys for non ssh connections https://review.openstack.org/535685 | 09:17 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add connection-port to provider diskimage https://review.openstack.org/535686 | 09:17 |
| *** cshastri_ has quit IRC | 09:23 | |
| *** rm_work has joined #openstack-infra | 09:36 | |
| AJaeger | frickler: care to review https://review.openstack.org/535660 as well, please? | 09:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Only run one requirements propose post job at a time https://review.openstack.org/535462 | 09:40 |
| *** dizquierdo has quit IRC | 09:40 | |
| *** derekh has joined #openstack-infra | 09:41 | |
| *** dbecker has joined #openstack-infra | 09:41 | |
| frickler | AJaeger: done, I'd have https://review.openstack.org/534902 for you in return ;) | 09:42 |
| *** pgadiya has quit IRC | 09:42 | |
| AJaeger | frickler: on it - thanks | 09:42 |
| *** efoley has joined #openstack-infra | 09:42 | |
| *** makowals has joined #openstack-infra | 09:43 | |
| *** xinliang has quit IRC | 09:44 | |
| *** jamielennox has joined #openstack-infra | 09:46 | |
| *** alexchadin has quit IRC | 09:46 | |
| *** ramishra has quit IRC | 09:46 | |
| *** ramishra has joined #openstack-infra | 09:46 | |
| *** alexchadin has joined #openstack-infra | 09:47 | |
| *** pcaruana has quit IRC | 09:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to tempest-plugin-jobs for ec2api-tempest-plugin https://review.openstack.org/532835 | 09:50 |
| openstackgerrit | Merged openstack-infra/project-config master: Build and publish releasenotes for openstacksdk https://review.openstack.org/535497 | 09:50 |
| *** stakeda has quit IRC | 09:53 | |
| *** pcaruana has joined #openstack-infra | 09:53 | |
| *** pcaruana has quit IRC | 09:53 | |
| *** pcaruana has joined #openstack-infra | 09:54 | |
| *** xinliang has joined #openstack-infra | 09:56 | |
| *** xinliang has joined #openstack-infra | 09:56 | |
| *** agopi has quit IRC | 09:59 | |
| *** pgadiya has joined #openstack-infra | 09:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable Cerberus/Sticks jobs for retirement https://review.openstack.org/535660 | 10:00 |
| *** liujiong has quit IRC | 10:01 | |
| *** shardy_ has joined #openstack-infra | 10:01 | |
| *** shardy has quit IRC | 10:02 | |
| AJaeger | gpocentek: merged ^ | 10:03 |
| *** lucas-pto is now known as lucasagomes | 10:03 | |
| *** janki has joined #openstack-infra | 10:05 | |
| gpocentek | AJaeger: thanks, I'm removing the repos data | 10:07 |
| gpocentek | .gitreview still uses stackforge/*, that's just sad | 10:07 |
| AJaeger | we should have retired it at that time instead ;( | 10:10 |
| AJaeger | gpocentek: if you want, I can do the commits... | 10:10 |
| *** cuongnv has quit IRC | 10:11 | |
| gpocentek | AJaeger: almost done | 10:11 |
| gpocentek | but thanks | 10:11 |
| *** ganso has joined #openstack-infra | 10:11 | |
| AJaeger | cool | 10:11 |
| gpocentek | AJaeger: can I +2/+A or should I ask someone else to accpt the reviews? | 10:13 |
| AJaeger | gpocentek: you can... | 10:14 |
| AJaeger | gpocentek: and please abandon all open reviews like https://review.openstack.org/#/q/project:openstack/sticks+is:open | 10:15 |
| *** markvoelker has joined #openstack-infra | 10:15 | |
| gpocentek | ok | 10:15 |
| AJaeger | gpocentek: including the change to update .gitreview ;) | 10:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move github webhook from webapp to zuul-web https://review.openstack.org/535711 | 10:20 |
| *** pgadiya has quit IRC | 10:20 | |
| *** kiennt26 has quit IRC | 10:20 | |
| *** alexchadin has quit IRC | 10:21 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add regex support to project stanzas https://review.openstack.org/535713 | 10:25 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add node list to webapp https://review.openstack.org/535714 | 10:26 |
| *** tosky has joined #openstack-infra | 10:26 | |
| *** pbourke has quit IRC | 10:27 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Set remote url on every getRepo in merger https://review.openstack.org/535716 | 10:28 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add winrm certificate handling https://review.openstack.org/535717 | 10:29 |
| *** dsariel has quit IRC | 10:29 | |
| *** pbourke has joined #openstack-infra | 10:29 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 10:29 |
| openstackgerrit | Gauvain Pocentek proposed openstack-infra/project-config master: Cerberus/Sticks retirement https://review.openstack.org/535719 | 10:30 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix sql reporting with postgres https://review.openstack.org/535720 | 10:30 |
| *** alexchadin has joined #openstack-infra | 10:31 | |
| *** e0ne has joined #openstack-infra | 10:32 | |
| AJaeger | gpocentek: one nit on 535719 | 10:32 |
| *** pgadiya has joined #openstack-infra | 10:32 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 10:34 | |
| *** shu-mutou-AWAY has quit IRC | 10:34 | |
| openstackgerrit | Gauvain Pocentek proposed openstack-infra/project-config master: Cerberus/Sticks retirement https://review.openstack.org/535719 | 10:36 |
| *** efoley has quit IRC | 10:36 | |
| *** efoley has joined #openstack-infra | 10:36 | |
| gpocentek | AJaeger: updated, thanks for the reviews and the help! | 10:37 |
| *** e0ne has quit IRC | 10:37 | |
| *** dsariel has joined #openstack-infra | 10:39 | |
| *** ihrachys_ has joined #openstack-infra | 10:41 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Don't gather host keys for non ssh connections https://review.openstack.org/535685 | 10:41 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add connection-port to provider diskimage https://review.openstack.org/535686 | 10:41 |
| *** zhurong has quit IRC | 10:41 | |
| *** ihrachys has quit IRC | 10:44 | |
| AJaeger | gpocentek: thanks for the cleanup. LGTM | 10:45 |
| *** alexchadin has quit IRC | 10:46 | |
| *** makowals has quit IRC | 10:47 | |
| *** makowals has joined #openstack-infra | 10:48 | |
| *** makowals has quit IRC | 10:48 | |
| *** makowals has joined #openstack-infra | 10:49 | |
| *** makowals has quit IRC | 10:49 | |
| *** makowals has joined #openstack-infra | 10:49 | |
| *** markvoelker has quit IRC | 10:49 | |
| *** bauzas_ has joined #openstack-infra | 10:50 | |
| *** ldnunes has joined #openstack-infra | 10:50 | |
| *** bauzas_ has quit IRC | 10:51 | |
| *** saop has joined #openstack-infra | 10:54 | |
| saop | How to see devstack running logs in zuul-web? | 10:54 |
| *** dtantsur|afk is now known as dtantsur | 10:54 | |
| saop | Do we need to set something in zuul.conf for that? | 10:55 |
| *** e0ne has joined #openstack-infra | 10:55 | |
| *** ankkumar has joined #openstack-infra | 10:56 | |
| *** erlon has joined #openstack-infra | 10:56 | |
| *** ihrachys_ has quit IRC | 10:57 | |
| *** daidv has quit IRC | 10:58 | |
| saop | tristanC, zuul logs in tmp are posting with root permission, any idea how to make it normal one? | 11:01 |
| openstackgerrit | Gauvain Pocentek proposed openstack-infra/infra-manual master: Update retirement section for project description https://review.openstack.org/535725 | 11:02 |
| *** yamahata has quit IRC | 11:04 | |
| *** alexchadin has joined #openstack-infra | 11:07 | |
| *** dizquierdo has joined #openstack-infra | 11:09 | |
| *** ihrachys has joined #openstack-infra | 11:10 | |
| *** rfolco|rover|off is now known as rfolco|rover | 11:11 | |
| *** dizquierdo has quit IRC | 11:17 | |
| *** tpsilva has joined #openstack-infra | 11:20 | |
| *** makowals has quit IRC | 11:21 | |
| *** claudiub has joined #openstack-infra | 11:23 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant project group definition example and definition in the doc https://review.openstack.org/535730 | 11:24 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs in networking-calico https://review.openstack.org/513316 | 11:34 |
| *** wolverineav has joined #openstack-infra | 11:34 | |
| openstackgerrit | boden proposed openstack-infra/project-config master: add required projects for vmware-nsx periodic with neutron-lib https://review.openstack.org/535732 | 11:37 |
| *** olaph1 is now known as olaph | 11:39 | |
| *** smatzek has joined #openstack-infra | 11:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Update jobs for ansible-role-cloud-launcher https://review.openstack.org/535734 | 11:42 |
| AJaeger | rcarrillocruz: could you review this, please? You created the ansible repo ^ | 11:42 |
| rcarrillocruz | lgtm, thx for the patch | 11:44 |
| AJaeger | thanks! | 11:44 |
| *** markvoelker has joined #openstack-infra | 11:46 | |
| *** ramishra has quit IRC | 11:47 | |
| *** dizquierdo has joined #openstack-infra | 11:49 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete ansible jobs https://review.openstack.org/535735 | 11:49 |
| openstackgerrit | boden proposed openstack-infra/project-config master: add required projects for vmware-nsx periodic with neutron-lib https://review.openstack.org/535732 | 11:50 |
| *** annp has quit IRC | 11:55 | |
| *** olaph1 has joined #openstack-infra | 11:58 | |
| *** olaph has quit IRC | 11:59 | |
| *** rhallisey has joined #openstack-infra | 12:00 | |
| *** coolsvap has quit IRC | 12:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Move some project-templates around https://review.openstack.org/535736 | 12:04 |
| *** rhallisey has left #openstack-infra | 12:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Convert legacy-tox-with-sudo-snap https://review.openstack.org/535755 | 12:10 |
| AJaeger | jamespage: fyi ^ | 12:10 |
| * jamespage looks | 12:10 | |
| *** dizquierdo has quit IRC | 12:11 | |
| *** e0ne has quit IRC | 12:12 | |
| *** jamesmcarthur has joined #openstack-infra | 12:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Move some project-templates around https://review.openstack.org/535736 | 12:13 |
| AJaeger | jamespage: shouldn't have any side effects - this is just using a Zuul v3 native job... | 12:13 |
| AJaeger | jamespage: you could move the template and job into one snap repo as well and maintain there. | 12:13 |
| *** olaph has joined #openstack-infra | 12:16 | |
| *** olaph1 has quit IRC | 12:16 | |
| *** jamesmcarthur has quit IRC | 12:17 | |
| *** andreas_s has quit IRC | 12:18 | |
| *** markvoelker has quit IRC | 12:20 | |
| *** smatzek_ has joined #openstack-infra | 12:25 | |
| *** dizquierdo has joined #openstack-infra | 12:25 | |
| *** wznoinsk has quit IRC | 12:26 | |
| *** tobberydberg has quit IRC | 12:26 | |
| *** hogepodge has quit IRC | 12:26 | |
| *** armaan_ has joined #openstack-infra | 12:26 | |
| *** strigazi has quit IRC | 12:27 | |
| *** antonym has quit IRC | 12:27 | |
| *** ruhe has quit IRC | 12:27 | |
| *** andreykurilin has quit IRC | 12:27 | |
| *** pfallenop has quit IRC | 12:27 | |
| *** jtomasek_ has joined #openstack-infra | 12:27 | |
| *** dansmith has quit IRC | 12:27 | |
| *** smatzek has quit IRC | 12:27 | |
| *** armaan has quit IRC | 12:27 | |
| *** jtomasek has quit IRC | 12:27 | |
| *** d34dh0r53 has quit IRC | 12:27 | |
| *** ericyoung has quit IRC | 12:27 | |
| *** pcrews has quit IRC | 12:27 | |
| *** gcb has quit IRC | 12:28 | |
| *** d34dh0r53 has joined #openstack-infra | 12:28 | |
| *** nhicher has quit IRC | 12:28 | |
| *** antonym has joined #openstack-infra | 12:28 | |
| *** pcrews has joined #openstack-infra | 12:28 | |
| *** dansmith has joined #openstack-infra | 12:28 | |
| *** dansmith is now known as Guest86649 | 12:28 | |
| *** ruhe has joined #openstack-infra | 12:28 | |
| *** ericyoung has joined #openstack-infra | 12:28 | |
| *** hogepodge has joined #openstack-infra | 12:28 | |
| *** pfallenop has joined #openstack-infra | 12:28 | |
| *** strigazi has joined #openstack-infra | 12:29 | |
| *** nhicher has joined #openstack-infra | 12:29 | |
| *** andreykurilin has joined #openstack-infra | 12:29 | |
| *** wznoinsk has joined #openstack-infra | 12:29 | |
| *** dgonzalez has quit IRC | 12:30 | |
| *** tobberydberg has joined #openstack-infra | 12:31 | |
| *** dgonzalez has joined #openstack-infra | 12:32 | |
| *** makowals has joined #openstack-infra | 12:33 | |
| *** Shrews has quit IRC | 12:33 | |
| *** jtomasek_ is now known as jtomasek | 12:35 | |
| *** coolsvap has joined #openstack-infra | 12:36 | |
| *** olaph1 has joined #openstack-infra | 12:38 | |
| *** olaph has quit IRC | 12:39 | |
| *** oidgar has joined #openstack-infra | 12:50 | |
| *** saop has quit IRC | 12:50 | |
| *** psachin has quit IRC | 12:50 | |
| *** zoli is now known as zoli|lunch | 12:54 | |
| *** zoli|lunch is now known as zoli | 12:54 | |
| *** agopi has joined #openstack-infra | 13:03 | |
| *** efried is now known as fried_rice | 13:05 | |
| *** jpena is now known as jpena|lunch | 13:07 | |
| *** andreas_s has joined #openstack-infra | 13:07 | |
| *** szaher has quit IRC | 13:07 | |
| *** jaypipes is now known as leakypipes | 13:10 | |
| *** olaph1 is now known as olaph | 13:10 | |
| *** dprince has joined #openstack-infra | 13:11 | |
| *** andreas_s has quit IRC | 13:15 | |
| *** markvoelker has joined #openstack-infra | 13:17 | |
| *** claudiub|2 has joined #openstack-infra | 13:18 | |
| *** oidgar has quit IRC | 13:18 | |
| *** claudiub has quit IRC | 13:20 | |
| *** dizquierdo has quit IRC | 13:20 | |
| *** Shrews has joined #openstack-infra | 13:21 | |
| *** bobh has joined #openstack-infra | 13:22 | |
| *** alexchadin has quit IRC | 13:22 | |
| *** makowals has quit IRC | 13:24 | |
| *** alexchadin has joined #openstack-infra | 13:27 | |
| *** links has quit IRC | 13:28 | |
| *** andreas_s has joined #openstack-infra | 13:28 | |
| *** markvoelker has quit IRC | 13:29 | |
| *** edmondsw has joined #openstack-infra | 13:29 | |
| *** markvoelker has joined #openstack-infra | 13:29 | |
| *** efoley has quit IRC | 13:31 | |
| *** rlandy has joined #openstack-infra | 13:32 | |
| smcginnis | ianw: Thanks for your help! | 13:33 |
| smcginnis | Oh, looks like there was another release job that failed with a timeout on git pull. | 13:35 |
| smcginnis | Anyone that can re-enque this one? http://logs.openstack.org/0f/0f0c9871144bf3b658f73f60f35f3068612a6bb2/release-post/tag-releases/629d779/job-output.txt.gz#_2018-01-19_04_55_11_661546 | 13:35 |
| dtantsur | vsaienk0: hey! I think for n-g-s tempest plugin we should do strictly the same thing as with ironic-tempest-plugin | 13:38 |
| *** bobh has quit IRC | 13:38 | |
| dtantsur | vsaienk0: that is, create a repo on github, import the history of only the plugin itself, then ask infra to import it | 13:38 |
| *** rosmaita has joined #openstack-infra | 13:38 | |
| AJaeger | dtantsur, vsaienk0 : import is only done at project creation via the upstream: keyword. | 13:39 |
| dtantsur | yep, that's what I'm saying | 13:39 |
| dtantsur | jlvillal and I did it once quite recently | 13:39 |
| dtantsur | he has a script in the ironic-tempest-plugin tree to do it | 13:40 |
| dtantsur | https://github.com/openstack/ironic-tempest-plugin/blob/master/tools/make-tempest-plugin-repo.sh | 13:40 |
| AJaeger | dtantsur, vsaienk0: I gave a -1 to ask for the import on the review | 13:41 |
| dtantsur | cool, yeah | 13:41 |
| *** ralonsoh has quit IRC | 13:42 | |
| *** trown|outtypewww is now known as trown | 13:42 | |
| dtantsur | vsaienk0: it may help to just to s/ironic/networking-generic-switch/ on https://github.com/openstack/ironic-tempest-plugin/blob/master/tools/make-tempest-plugin-repo.sh and run it in the new repo | 13:42 |
| *** trown is now known as trown|ruck | 13:42 | |
| *** sambetts|afk is now known as sambetts | 13:44 | |
| *** pgadiya has quit IRC | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:49 | |
| *** xinliang has quit IRC | 13:50 | |
| *** kgiusti has joined #openstack-infra | 13:51 | |
| *** ralonsoh has joined #openstack-infra | 13:51 | |
| *** xinliang has joined #openstack-infra | 13:52 | |
| *** efoley has joined #openstack-infra | 13:52 | |
| *** Goneri has joined #openstack-infra | 13:54 | |
| *** e0ne has joined #openstack-infra | 13:55 | |
| *** amoralej is now known as amoralej|lunch | 13:59 | |
| *** claudiub has joined #openstack-infra | 14:03 | |
| *** beagles_afk is now known as beagles | 14:04 | |
| *** alexchadin has quit IRC | 14:04 | |
| *** claudiub|2 has quit IRC | 14:06 | |
| *** weshay|ruck is now known as weshay|rover | 14:08 | |
| *** jamesmcarthur has quit IRC | 14:10 | |
| *** lbragstad has joined #openstack-infra | 14:10 | |
| openstackgerrit | Merged openstack-infra/project-config master: Partial revert "Revert "Remove diskimages section from nodepool-launchers"" https://review.openstack.org/535653 | 14:12 |
| *** jpena|lunch is now known as jpena | 14:13 | |
| *** alexchadin has joined #openstack-infra | 14:14 | |
| *** stevebaker has quit IRC | 14:19 | |
| AJaeger | config-core, could you review https://review.openstack.org/535734 - ansible-role-cloud-launcher jobs; https://review.openstack.org/535719 - cerberus/sticks retirement ;https://review.openstack.org/513317/ - remove some legacy jobs , please? | 14:20 |
| *** camunoz has joined #openstack-infra | 14:23 | |
| *** ankkumar has quit IRC | 14:24 | |
| *** alexchadin has quit IRC | 14:25 | |
| *** david-lyle has quit IRC | 14:27 | |
| *** david-lyle has joined #openstack-infra | 14:29 | |
| *** alexchadin has joined #openstack-infra | 14:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add Neutron Tempest Plugin Failure Rate dashboard https://review.openstack.org/534902 | 14:30 |
| *** jistr is now known as jistr|mtg | 14:30 | |
| *** dsariel has quit IRC | 14:31 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Allow Ansible 2.4 https://review.openstack.org/535781 | 14:32 |
| *** alexchadin has quit IRC | 14:32 | |
| *** dayou has quit IRC | 14:34 | |
| *** alexchadin has joined #openstack-infra | 14:34 | |
| *** jamesmcarthur has joined #openstack-infra | 14:36 | |
| *** jamesmcarthur has quit IRC | 14:37 | |
| *** jamesmcarthur has joined #openstack-infra | 14:37 | |
| *** lucasagomes is now known as lucas-hungry | 14:38 | |
| *** vaidy has quit IRC | 14:39 | |
| *** isviridov_away has quit IRC | 14:39 | |
| fungi | smcginnis: i can take a look at the release job failures and rerun as needed | 14:39 |
| *** mrtenio has joined #openstack-infra | 14:41 | |
| smcginnis | fungi: I was a little worried that the timeout was due to the number of included projects in that, but it appears both time it hung for quite a will after calling git pull --ff-only. | 14:43 |
| smcginnis | fungi: So I don't think it's the quantity, but I do worry what caused the same git pull operation to hang twice. | 14:43 |
| fungi | we've seen a few of those this week, leading me to suspect it may be intermittent network issues in some part of one of our providers | 14:45 |
| *** bhagyashris has quit IRC | 14:46 | |
| *** dayou has joined #openstack-infra | 14:46 | |
| *** amoralej|lunch is now known as amoralej | 14:47 | |
| *** hashar has joined #openstack-infra | 14:49 | |
| *** spzala has quit IRC | 14:49 | |
| *** alexchadin has quit IRC | 14:50 | |
| *** spzala has joined #openstack-infra | 14:50 | |
| fungi | smcginnis: i've reenqueued 0f0c987 in release-post | 14:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make subunit processing more resilient https://review.openstack.org/534431 | 14:54 |
| smcginnis | fungi: Thank you. | 14:54 |
| *** efoley has quit IRC | 14:56 | |
| pabelanger | I'm working on nb02.o.o this morning | 14:57 |
| pabelanger | making sure I attach a volume for DIBs :) | 14:57 |
| fungi | yeah, that rootfs will be toast if you don't | 14:58 |
| pabelanger | yah, what happened on nb01 | 15:00 |
| pabelanger | that is current stopped, once nb02 is online, I'll clean up nb01 | 15:00 |
| *** armax has joined #openstack-infra | 15:01 | |
| *** jistr|mtg is now known as jistr | 15:04 | |
| *** bmjen has quit IRC | 15:05 | |
| *** yamamoto has quit IRC | 15:05 | |
| *** bmjen has joined #openstack-infra | 15:05 | |
| *** yamamoto has joined #openstack-infra | 15:06 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:07 | |
| *** stevebaker has joined #openstack-infra | 15:10 | |
| *** efoley has joined #openstack-infra | 15:12 | |
| *** vaidy has joined #openstack-infra | 15:14 | |
| *** sree has joined #openstack-infra | 15:14 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Open zookeeper firewall for nb02.openstack.org https://review.openstack.org/535797 | 15:14 |
| *** isviridov_away has joined #openstack-infra | 15:15 | |
| *** yamamoto has quit IRC | 15:17 | |
| *** sree has quit IRC | 15:19 | |
| AJaeger | mordred: http://logs.openstack.org/31/534431/8/check/tox-py35-on-zuul/dca20ec/ still has no subunit file ;( | 15:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run shade functional tests on ansible OpenStack PRs https://review.openstack.org/531916 | 15:25 |
| mordred | AJaeger: boo :( | 15:26 |
| *** zeus has joined #openstack-infra | 15:26 | |
| mordred | AJaeger: OOHHHHHHHHHH. I get what's wrong | 15:27 |
| *** zeus is now known as Guest55337 | 15:27 | |
| * AJaeger is curiously waiting for the patchset to review what's wrong | 15:27 | |
| *** fultonj has quit IRC | 15:28 | |
| *** Guest55337 is now known as zeus` | 15:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make subunit processing more resilient https://review.openstack.org/534431 | 15:29 |
| *** zeus` is now known as Guest51542 | 15:29 | |
| mordred | AJaeger: paths: takes a list of directories to search | 15:29 |
| mordred | not a list of files | 15:29 |
| *** dizquierdo has joined #openstack-infra | 15:30 | |
| *** Guest51542 is now known as zeus | 15:30 | |
| *** zeus has quit IRC | 15:30 | |
| *** zeus has joined #openstack-infra | 15:30 | |
| *** annp has joined #openstack-infra | 15:31 | |
| AJaeger | mordred: ah! Let's try it ;) did you recheck the glance_store change? | 15:32 |
| AJaeger | mordred: could you review https://review.openstack.org/535755 https://review.openstack.org/535734 and https://review.openstack.org/535736 , please? I did some cleanups... | 15:33 |
| *** fultonj has joined #openstack-infra | 15:33 | |
| *** lucas-hungry is now known as lucasagomes | 15:34 | |
| *** caphrim007_ has quit IRC | 15:34 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul-jobs master: [WIP] Debugging playbook: inject public keys in case of failure https://review.openstack.org/535803 | 15:35 |
| *** yamahata has joined #openstack-infra | 15:37 | |
| *** stephenfin is now known as finucannot | 15:38 | |
| mordred | AJaeger: I'll recheck the glance_store one if the tox-py35-on-zuul one works | 15:38 |
| *** eharney has joined #openstack-infra | 15:38 | |
| rosmaita | :) | 15:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run shade functional tests on ansible OpenStack PRs https://review.openstack.org/531916 | 15:39 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config master: Add networking-generic-switch-tempest-plugin repo https://review.openstack.org/532542 | 15:40 |
| mordred | AJaeger: all look great! | 15:40 |
| AJaeger | thanks | 15:41 |
| *** gibi is now known as giblet | 15:41 | |
| *** hongbin has joined #openstack-infra | 15:42 | |
| corvus | dmsimard: what's the story with https://review.openstack.org/525271 ? | 15:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Open zookeeper firewall for nb02.openstack.org https://review.openstack.org/535797 | 15:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove nb03 / nb04 servers https://review.openstack.org/535804 | 15:44 |
| *** efoley has quit IRC | 15:44 | |
| fungi | corvus: oh, that was me | 15:45 |
| AJaeger | corvus: fungi wanted to keep that to dig deeper into it... | 15:45 |
| fungi | corvus: i keep meaning to try and work out how to get zuul to report in its debug log when it fails to merge a change under those conditions | 15:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Update jobs for ansible-role-cloud-launcher https://review.openstack.org/535734 | 15:46 |
| fungi | we just get that the gerrit api call failed but no content from the api response | 15:46 |
| dmsimard | corvus: oh, that was somtehing with fungi, I forget what it was | 15:46 |
| pabelanger | infra-root: ^ reviews for bringing nb02 online and deleting nb03 and nb04 servers | 15:46 |
| dmsimard | corvus, fungi: right, now I remember -- we wanted to take the opportunity to understand why that was happening | 15:47 |
| fungi | corvus: i think i got as far as determining that zuul was discarding the text of the failure response from the gerrit api but then got sidetracked with other stuff | 15:47 |
| dmsimard | it's likely long lost in the logs by now, should we rebase ? | 15:47 |
| *** pblaho has quit IRC | 15:47 | |
| AJaeger | config-core, could you review https://review.openstack.org/535755 https://review.openstack.org/535735 and https://review.openstack.org/535736 , please? I did some cleanups... | 15:48 |
| AJaeger | dmsimard: if you rebase, it can merge ;) | 15:48 |
| *** olaph has quit IRC | 15:48 | |
| AJaeger | dmsimard: which is fine by me... | 15:48 |
| fungi | dmsimard: well, it turned out to be an easy change to use to reproduce that specific scenario and was non-urgent | 15:48 |
| *** yamamoto has joined #openstack-infra | 15:48 | |
| dmsimard | AJaeger: unless we can get something useful out of it's current state we should probably not hold it back too long | 15:48 |
| dmsimard | fungi: oh ? so we just submit and outdated change and try to merge it basically ? | 15:49 |
| *** olaph has joined #openstack-infra | 15:49 | |
| fungi | dmsimard: i think we need to submit two changes, then update the parent and merge it without updating the child | 15:49 |
| dmsimard | makes sense, let's rebase and merge the sig patch -- is that documented in storyboard ? | 15:49 |
| AJaeger | fungi: agreed | 15:50 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add system-required for openstack/self-healing-sig https://review.openstack.org/525271 | 15:50 |
| fungi | zuul has no problem merging the change to master itself but then calls the submit api and gerrit refuses to do the same | 15:50 |
| *** ykarel is now known as ykarel|away | 15:50 | |
| fungi | and the gerrit webui is ambiguous with regards to whether the change actually needs rebasing since it just says "rebase possible" | 15:50 |
| fungi | but if you try to call the submit api yourself you'll see the error stating that it's refusing to merge the change due to an outdated parent | 15:51 |
| dmsimard | infra-root: I came across (at least) two badly behaving web crawlers last night that was hammering different things in RDO's infrastructure. It was late but I looked a the git.openstack.org servers and they're also visted by these crawlers. | 15:52 |
| AJaeger | mordred: yeah! http://logs.openstack.org/31/534431/9/check/tox-py35-on-zuul/3029a4f/ | 15:52 |
| corvus | before we... nevermind. | 15:52 |
| AJaeger | thanks, corvus for reviewing | 15:52 |
| dmsimard | It would seem that they respect robots.txt to some extent, on some of the RDO servers there were hits to robots.txt and that was it | 15:53 |
| corvus | based on what fungi said, i was going to do a quick query to see if it's possible for zuul to see that. | 15:53 |
| dmsimard | However, we don't have -- for example -- a robots.txt on git.o.o so they're fetching everything, every ref, etc. | 15:53 |
| corvus | but the patch has been rebased. | 15:53 |
| corvus | dmsimard: i do hope you'll look into that at some point. | 15:53 |
| mordred | AJaeger: woot! | 15:53 |
| *** felipemonteiro__ has joined #openstack-infra | 15:54 | |
| mordred | AJaeger, rosmaita: I have rechecked https://review.openstack.org/#/c/535398/1 | 15:54 |
| corvus | dmsimard: see https://review.openstack.org/427959 | 15:54 |
| *** zoli is now known as zoli|gone | 15:54 | |
| *** zoli|gone is now known as zoli | 15:54 | |
| dmsimard | These bots are widely documented not to respect robots.txt from googling a bit -- I was considering picking up the net blocks and adding them to the security groups unless we have a way to apply iptables rules across everything ? | 15:54 |
| *** camunoz has quit IRC | 15:54 | |
| mordred | but I think https://review.openstack.org/#/c/534431/ is ready for review ... dmsimard if you get a sec | 15:55 |
| corvus | dmsimard: well, since we have a highly restrictive robots meta tag, they obviously aren't obeying that. | 15:55 |
| fungi | dmsimard: seems like a treadmill. are they causing significant impact for us? | 15:55 |
| corvus | dmsimard: please do not use security groups. add things to iptables. | 15:55 |
| dmsimard | corvus: they were hammering our jenkins.. each job, build, consoles, etc. | 15:55 |
| mordred | dmsimard: well, jenkins is really bad at serving web content :) | 15:55 |
| clarkb | we don't have security groups where git.o.o is hosted fwiw | 15:56 |
| dmsimard | mordred: yeah that made things even worse | 15:56 |
| fungi | dmsimard: is there a reason we wouldn't want search engines to index our git repos? | 15:56 |
| mordred | I think having the ability to apply an iptables setting across all of our servers is not a bad ability for us to have | 15:57 |
| corvus | indeed. i wrote a change to do just that but it's been stuck in review for 6 months. | 15:57 |
| dmsimard | are we okay with robots indexing git.o.o in general ? To give you an idea: http://paste.openstack.org/show/647114/ (openstack logs at the bottom) | 15:57 |
| mordred | corvus: blerg. have a change number handy? | 15:57 |
| *** felipemonteiro_ has quit IRC | 15:57 | |
| corvus | mordred: was responding to indexing git repos, not your thing. but https://review.openstack.org/427959 | 15:58 |
| fungi | dmsimard: so if someone searches for, say, an error message emitted by nova we wouldn't want them to find the source code for the software generating that error message? | 15:58 |
| mordred | corvus: oh good - I already +2'd that | 15:58 |
| dmsimard | fungi: they'll use google and google indexes github | 15:58 |
| corvus | i kind of want everyone to read that before this coversation continues | 15:58 |
| dmsimard | and google behaves with robots.txt instructions | 15:58 |
| fungi | dmsimard: got it. so we want github to be our official git hoster now? | 15:58 |
| *** kukacz_ has quit IRC | 15:58 | |
| *** ykarel|away has quit IRC | 15:59 | |
| dmsimard | fungi: no, I'm saying some crawlers are documented to be awful and provide no value to us | 15:59 |
| *** kukacz has joined #openstack-infra | 15:59 | |
| *** esberglu has joined #openstack-infra | 15:59 | |
| *** annp has quit IRC | 16:00 | |
| dmsimard | google, bing or whatever probably indexes git.o.o and that's fine | 16:00 |
| corvus | AHEM. no they DO NOT. | 16:00 |
| corvus | please see https://review.openstack.org/427959 | 16:00 |
| fungi | dmsimard: see 427959 | 16:00 |
| mordred | corvus, fungi, dmsimard: re: ^^ that change ... I'd like for us to index non-master branches | 16:00 |
| mordred | github's exclusion of non-master branches from crawling is important due to the PR model and all the throwaway branches | 16:01 |
| mordred | branches for us are long-lived and more important | 16:01 |
| *** vsaienk0 has quit IRC | 16:01 | |
| dmsimard | corvus: yeah, I was merely explaining what was my concern, looking now. | 16:01 |
| mordred | they are also content we are publishing | 16:01 |
| fungi | mordred: yeah, i was just stating the same in a review there | 16:01 |
| mordred | although - I COULD see us putting in a thing thatonly allowed master and stable/* | 16:02 |
| corvus | mordred: okay that sounds reasonable. in that case i think we should accept ianw's invitation to override and +W | 16:02 |
| fungi | well, the same preference, though not for the same reasons (though i also agree with you those are good reasons) | 16:02 |
| mordred | but I'd rather start with all of it | 16:02 |
| *** efoley has joined #openstack-infra | 16:02 | |
| fungi | yep | 16:02 |
| dmsimard | mordred, corvus, fungi: from what I could tell from the apache logs I've looked at, the crawling is fairly dumb -- literally fetches every ref, links.. you know, it crawls. In a typical git repo, how many refs you think there are ? Thousands -- and each of these thousand refs can have hundreds of files | 16:03 |
| dmsimard | I wonder if there is a smart way to tell -- through robots.txt, what's appropriate | 16:04 |
| fungi | dmsimard: you should look at the access logs for the logs.openstack.org vhost sometime... we have many terabytes of data there in billions of individual files | 16:04 |
| persia | dmsimard: Most crawlers deduplicate, and the hosting cost is only paid once. | 16:04 |
| pabelanger | could I get a quick review on https://review.openstack.org/535797/ to bring nb02 online, thanks | 16:04 |
| *** jlvillal has quit IRC | 16:04 | |
| dmsimard | persia: the hits (GET) are there, though, right ? so there is filesystem access, web server serving the content, etc. | 16:05 |
| persia | Yes. | 16:05 |
| dmsimard | fungi: but logs.o.o is not indexed, right ? | 16:05 |
| mordred | clarkb: feel lke a +A on https://review.openstack.org/#/c/535522/ - it did the right thing | 16:05 |
| dmsimard | I think puppet-openstackci sets up a robots.txt | 16:05 |
| dmsimard | but yeah.. worth having a look | 16:06 |
| fungi | dmsimard: we have a restrictive robots.txt telling legitimate crawlers not to index it, but that doesn't stop illegitimate crawlers (for example, bot armies scanning for vulnerable webapps) | 16:06 |
| dmsimard | right | 16:06 |
| dmsimard | okay.. I know what I'm doing today :) | 16:06 |
| fungi | dmsimard: some other fun ones are eavesdrop (we want that thoroughly indexed), review (we want a lot of that indexed but not the things linked from it due to potential for linkspam abuse), and paste (we want none of it indexed, but it gets referenced from lots of other places such as channel/meeting logs eavesdrop) | 16:08 |
| *** vsaienk0 has joined #openstack-infra | 16:08 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Convert legacy-tox-with-sudo-snap https://review.openstack.org/535755 | 16:09 |
| dmsimard | so it turns out our jenkins had a robots.txt and it was definitely getting hammered, bleh | 16:10 |
| dmsimard | s/our/RDO's/ | 16:10 |
| clarkb | mordred: done | 16:10 |
| mordred | clarkb: thanks | 16:10 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/openstack-zuul-jobs master: Remove remaining Mistral jobs https://review.openstack.org/535811 | 16:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Move some project-templates around https://review.openstack.org/535736 | 16:11 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/project-config master: Remove legacy mistralclient jobs https://review.openstack.org/535812 | 16:11 |
| *** verdurin_ has joined #openstack-infra | 16:12 | |
| dmsimard | mordred: all this subunit stuff, I'll review the code and the ansible stuff but I'm not intimate with the logic/workflow -- I'll try to learn it as I go along but I'll rely on you a bit :D | 16:12 |
| AJaeger | mordred, corvus - I had to resolve a merge conflict, could you review 535736 again, please? | 16:12 |
| dmsimard | the tox siblings logic is something I do not quite understand yet either | 16:12 |
| mordred | dmsimard: :) | 16:12 |
| *** jlvillal has joined #openstack-infra | 16:14 | |
| *** verdurin has quit IRC | 16:14 | |
| AJaeger | thanks! | 16:14 |
| mordred | dmsimard: hopefully the subunit patch isn't too terrible - it's mostly about adding more logic to find files and to not throw errors inappropriately | 16:14 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add provider info command https://review.openstack.org/535817 | 16:19 |
| *** ykarel|away has joined #openstack-infra | 16:19 | |
| *** b_bezak has quit IRC | 16:20 | |
| *** b_bezak has joined #openstack-infra | 16:21 | |
| AJaeger | mordred: are you aware of http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/fetch-translations-stestr-output ? Are changes needed for that one? | 16:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add provider info command https://review.openstack.org/535817 | 16:23 |
| dmsimard | corvus: did you find the patch for the global firewall defaults ? I can only find the git robots patch | 16:23 |
| corvus | dmsimard: i am not aware of such a patch | 16:24 |
| *** jcoufal has joined #openstack-infra | 16:24 | |
| * dmsimard confused | 16:24 | |
| *** b_bezak has quit IRC | 16:26 | |
| dmsimard | corvus: I mismatched your reply to another reply -- these were back to back on my client http://paste.openstack.org/raw/647124/ | 16:26 |
| dmsimard | Sorry! | 16:26 |
| corvus | dmsimard: yeah, which is why i followed it up with 15:58 < corvus> mordred: was responding to indexing git repos, not your thing. but https://review.openstack.org/427959 | 16:26 |
| * dmsimard nods | 16:27 | |
| AJaeger | config-core, some cleanup changes: https://review.openstack.org/535719 - cerberus/sticks retirement; https://review.openstack.org/513317/ - remove legacy jobs ; https://review.openstack.org/535735 - ansible-job removal - reviews welcome | 16:27 |
| mordred | AJaeger: why is that role a different role? can we just get rid of it and replace it with the fetch-subunit-output role? | 16:27 |
| dmsimard | AJaeger: you're on a retiring spree ? :) | 16:27 |
| corvus | dmsimard: anyway, system-config/manifests/server.pp has some global rules and could probably be used for that purpose | 16:28 |
| corvus | dmsimard: anyway, system-config/modules/openstack_project/manifests/server.pp has some global rules and could probably be used for that purpose | 16:28 |
| AJaeger | dmsimard: if I patch a repo and then figure out it's dead, I'll poke... | 16:28 |
| AJaeger | mordred: let me check... | 16:29 |
| mordred | AJaeger: it looks like we should be able to | 16:29 |
| mordred | AJaeger: at least, with the new friendlier version which doens't try to sync non-existent things | 16:29 |
| *** vsaienk0 has quit IRC | 16:29 | |
| *** jaosorior has quit IRC | 16:30 | |
| AJaeger | mordred: let's try ;) | 16:30 |
| openstackgerrit | Merged openstack-infra/project-config master: Add system-required for openstack/self-healing-sig https://review.openstack.org/525271 | 16:30 |
| clarkb | prometheanfire: hey the neutron unittests seem to time out quite a bit. Tip of gate is a failed neutron change due to timed out unittests and I know its been affecting requirements too. Might want to work with neutron to bump up the timelimit or figure out why they are slower? | 16:31 |
| clarkb | ihrachys: ^ fyi | 16:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Switch translations jobs to fetch-subunit-output https://review.openstack.org/535824 | 16:31 |
| mordred | AJaeger: ^^ there you go :) | 16:31 |
| *** vsaienk0 has joined #openstack-infra | 16:31 | |
| *** slaweq has quit IRC | 16:31 | |
| *** felipemonteiro__ has quit IRC | 16:31 | |
| AJaeger | mordred: you're too fast for me ;) | 16:32 |
| mordred | AJaeger: :) | 16:32 |
| AJaeger | mordred: what about http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh#n65 ? | 16:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool master: Fix typo in nodepool-builder.init https://review.openstack.org/535829 | 16:32 |
| AJaeger | should we remove the gzip? | 16:32 |
| pabelanger | infra-root: ^could I get a +3 on nodepool-builder typo in init script | 16:33 |
| AJaeger | mordred: please check the finish function | 16:33 |
| *** ykarel|away has quit IRC | 16:33 | |
| *** coolsvap has quit IRC | 16:33 | |
| prometheanfire | clarkb: iirc, we are not sure if it's the time limit, currently suspect pyroute2 is leaking FDs | 16:33 |
| *** jbadiapa has quit IRC | 16:33 | |
| prometheanfire | https://github.com/svinota/pyroute2/pull/443 | 16:33 |
| ihrachys | clarkb, is it py3 or py2 too? | 16:34 |
| ihrachys | clarkb, we were going to look into py3 timeouts today with mlavalle | 16:34 |
| clarkb | ihrachys: I think it has been both | 16:34 |
| fungi | i prefer that explanation to "things are slower because of meltdown mitigation" | 16:34 |
| clarkb | ihrachys: http://logs.openstack.org/90/534990/6/check/cross-neutron-py27/9b5ca31/ timed out too | 16:34 |
| prometheanfire | it's py2 on this random review I had open | 16:34 |
| ihrachys | clarkb, ok. maybe deprecated messages slow us down again. I saw a lot of them lately. we'll figure it out, thanks for the notice. | 16:34 |
| clarkb | ihrachys: yes I noticed that the log files for neutron are massive as a result | 16:35 |
| clarkb | ihrachys: using tenant instead of project or is it the other way around | 16:35 |
| AJaeger | mordred: the translation files generate subunit themselves - will that work with your role? | 16:35 |
| pabelanger | clarkb: Shrews: thanks! | 16:35 |
| prometheanfire | offhand... does anyone know if we already have a state machine lib in openstack? | 16:35 |
| clarkb | isn't that sort of what taskflow does? | 16:36 |
| prometheanfire | I thought so | 16:36 |
| prometheanfire | https://review.openstack.org/534990 is what I'm reviewing | 16:36 |
| * AJaeger will be back later | 16:39 | |
| *** andreas_s has quit IRC | 16:43 | |
| fungi | prometheanfire: clarkb: looking at taskflow's requirements.txt, they use automaton to provide a finite state machine implementation. i left a comment on the review | 16:44 |
| prometheanfire | fungi: thanks | 16:44 |
| prometheanfire | I would have thought we had something like it already | 16:44 |
| fungi | so if a lower-level implementation is wanted without relying on all of taskflow to provide it, they could in theory probably use that instead as it's also already in global requirements | 16:44 |
| prometheanfire | yep, that'd be my suggestion too | 16:45 |
| fungi | it doesn't support _all_ of python >=3.0 but it does claim to support 3.5 | 16:45 |
| fungi | (as well as 2.7 of course) | 16:45 |
| pabelanger | okay, I've manually applied 535829 to nb01 and nodepool-builder is running again | 16:45 |
| prometheanfire | no one cares about 2.7 anymore (or shouldn't :P) | 16:46 |
| pabelanger | if we can get https://review.openstack.org/535797/ approved, that will be all that is needed for nb02 | 16:46 |
| pabelanger | and I can make sure it is started properly when puppet-nodepool changes land | 16:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Only install bindep when we've found a file https://review.openstack.org/535836 | 16:47 |
| clarkb | in the case where a name doesn't resolve puppet just fails right? | 16:47 |
| clarkb | re that firewall change | 16:47 |
| *** vsaienk0 has quit IRC | 16:48 | |
| mordred | AJaeger: oh - interesting. we should maybe add a check to the fetch-subunit role to see if subunit file already exists | 16:48 |
| jgriffith | fungi: oh nose!! | 16:49 |
| mordred | AJaeger: so they create an already compressed testrepository.subunit.gz file | 16:49 |
| fungi | clarkb: yes, which can be an issue since it starts blocking subsequent configuration management updates for any servers relying on that rule and can block launching of new instances for similar reasons | 16:49 |
| fungi | jgriffith: i assume you're just now catching up on the ssh connection garbage collection problem? | 16:50 |
| jgriffith | fungi: yes :( | 16:50 |
| *** vsaienk0 has joined #openstack-infra | 16:50 | |
| jgriffith | fungi: so TBH I just open a connection and listen on it *forever* | 16:50 |
| jgriffith | which is apparently no bueano | 16:50 |
| clarkb | ya there was a change to paramiko something like 6 months back? that started requiring explicitly closes | 16:51 |
| clarkb | tempest ran into this too | 16:51 |
| fungi | jgriffith: well, it becomes a problem if that connection gets dropped for some reason (network interruption, aggressive firewall/nat times out what it sees as an idle socket, et cetera) | 16:51 |
| mordred | AJaeger: ok. I understand now - I think we can abandon my patch to usefetch-subunit-output and we don't need to update the role there | 16:51 |
| jgriffith | understood, I'll have a look at the paramiko changes and see if I can figure out how to address it | 16:51 |
| jgriffith | sorry about that | 16:51 |
| fungi | jgriffith: gerrit handles it surprisingly well these days, it just starts telling you to go away after too many of those pile up | 16:52 |
| jgriffith | fungi: funny... I get that a lot, now from Gerrit too :) | 16:52 |
| fungi | heh | 16:52 |
| fungi | clarkb: oh, i remembered specifically why the connlimit protection is still important | 16:53 |
| fungi | since gerrit's max connections protection is per-user, it fails to limit them pre-auth | 16:53 |
| clarkb | oh right we've seen it where you get a bunch of just barely started connections | 16:54 |
| fungi | so if a client is opening lots of ssh sockets but has an authentication problem and then doesn't close them appropriately, they can still pile up in devastating ways | 16:54 |
| openstackgerrit | Mark Hamzy proposed openstack/diskimage-builder master: fix the grub root label when an XFS disk label is truncated https://review.openstack.org/532279 | 16:54 |
| clarkb | ya I seem to recall someone had wrong permissions onf a key file or similar causing that | 16:54 |
| clarkb | fungi: and where is that chagne to make the rule permanent? | 16:54 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Disable action and lookup plugins from 2.4 https://review.openstack.org/535839 | 16:54 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Port in changes from ansible 2.4 command module https://review.openstack.org/535840 | 16:54 |
| fungi | clarkb: 529712 | 16:55 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Move some project-templates around https://review.openstack.org/535736 | 16:55 |
| clarkb | fungi: dmsimard re logging you just have to check the connection atble for whoever has 100 connections open | 16:56 |
| fungi | or >=100 anyway | 16:56 |
| fungi | sometimes the new connections race the connlimit thwap slightly | 16:56 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/devstack-gate master: Set up stable/queens https://review.openstack.org/535847 | 16:57 |
| fungi | so they might be a few over | 16:57 |
| clarkb | fungi: did you want to approve that change now? | 16:57 |
| clarkb | also since we are both traveling next week is today a good day to switch zuul servers? | 16:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Create per-site template directories https://review.openstack.org/535848 | 16:57 |
| fungi | clarkb: sure, i'm around to keep an eye on gerrit to make sure it shows back up in the ruleset and in case there is any unanticipated impact (but as i said in the review, it ran manually applied on the server for months before i wrote the change) | 16:58 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/devstack-gate master: Set up stable/queens https://review.openstack.org/535847 | 16:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Rename fetch-translations-stestr-output https://review.openstack.org/535849 | 16:58 |
| mordred | AJaeger: ^^ I think that might help us not be confused in the future | 16:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add openstack branded mailman templates https://review.openstack.org/535851 | 16:58 |
| *** gyee has joined #openstack-infra | 16:58 | |
| *** andreas_s has joined #openstack-infra | 16:58 | |
| fungi | clarkb: i like today for the zuulv3->zuul01 migration if others are available to help in case something doesn't go smoothly | 16:59 |
| clarkb | I plan to be around for most of the day. May duck out early to run some pre travel errands though. But that should still be quite late in your day | 16:59 |
| fungi | otherwise, yes, i can't really help make it happen during the coming week | 16:59 |
| *** trown|ruck is now known as trown|outtypewww | 16:59 | |
| *** vsaienk0 has quit IRC | 17:00 | |
| fungi | and hopefully friday activity levels will be lowish as the day unfolds | 17:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Use multisite template dir https://review.openstack.org/535852 | 17:00 |
| fungi | i suppose i could also copy a final status dump from the old server to the new one so we can reenqueue whatever changes were under test | 17:00 |
| fungi | though i didn't explicitly mention that in the review comments on 534897 | 17:01 |
| fungi | i'll add it | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Remove openstack HTML templates https://review.openstack.org/535855 | 17:01 |
| jgriffith | fungi: clarkb OHHH, so it looks like it's the connection after I post results and NOT my listener. | 17:02 |
| fungi | jgriffith: that could do it if you're posting results via ssh | 17:02 |
| jgriffith | fungi: yeah, I think that's probably the issue; updating now... guess we'll know in a few days | 17:03 |
| *** andreas_s has quit IRC | 17:03 | |
| * smcginnis preps sf removal patch for lack of CI | 17:04 | |
| smcginnis | :) | 17:04 |
| jgriffith | smcginnis: :) That's ok, I don't mind :) | 17:04 |
| smcginnis | jgriffith: Hah, maybe that would simplify your life. :) | 17:05 |
| jgriffith | I'll set off all those timers I've planted over the years :) | 17:05 |
| smcginnis | hah | 17:05 |
| fungi | clarkb: corvus: what i mostly don't know is how we should apply the config updates from 534897 to minimize the outage... stop puppet, merge the change, grab a status dump of the pipelines, stop all zuul services, use the kick script to puppet all the zuul servers, do a final rsync of keys and times to the new server as well as copy across the status dump, start zuul services again but start the | 17:05 |
| fungi | scheduler on zuul01 and not zuulv3, reenqueue changes and update the cname in dns... what else am i missing? | 17:05 |
| fungi | i'll make a maintenance etherpad with the above | 17:06 |
| clarkb | fungi: I'm not sure you have to stop all the zuul services? I guess they need whatever kick is necessary to see new gearman config | 17:08 |
| clarkb | which may be a restart | 17:08 |
| *** caphrim007 has joined #openstack-infra | 17:08 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/project-config master: Remove legacy mistralclient jobs https://review.openstack.org/535812 | 17:09 |
| *** caphrim007 has quit IRC | 17:09 | |
| *** caphrim007 has joined #openstack-infra | 17:10 | |
| *** caphrim007 has quit IRC | 17:11 | |
| *** caphrim007 has joined #openstack-infra | 17:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Minimal nodepool-launcher config take 2 https://review.openstack.org/535857 | 17:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't rely on canonical hostnames for CRD https://review.openstack.org/535858 | 17:12 |
| pabelanger | corvus: clarkb: ^ take 2 to minimize nodepool-launcher. I hope to work on nodepool.d support next week | 17:12 |
| *** caphrim007 has quit IRC | 17:12 | |
| *** bnemec is now known as beekneemech | 17:13 | |
| corvus | pabelanger: ok; i'd like to avoid merging it until after the v3 release though, so no rush | 17:14 |
| *** fried_rice is now known as fried_rolls | 17:14 | |
| fungi | clarkb: hrm... executors and mergers all need the config update, so that's basically all zuul services (excepting the ones running on the same server as the scheduler, which need to get bounced anyway, right?) | 17:15 |
| *** caphrim007 has joined #openstack-infra | 17:15 | |
| *** Goneri has quit IRC | 17:15 | |
| corvus | restart is required for new gearman config | 17:16 |
| openstackgerrit | Merged openstack-infra/system-config master: Open zookeeper firewall for nb02.openstack.org https://review.openstack.org/535797 | 17:18 |
| fungi | also, clean restarts of the executors seem to be fiddly, so likely care would need to be taken to make sure they're well and stopped anyway | 17:19 |
| pabelanger | corvus: wfm | 17:19 |
| *** jamesmcarthur has quit IRC | 17:21 | |
| *** vsaienk0 has joined #openstack-infra | 17:24 | |
| corvus | fungi: if they're fiddly, let's try to figure out how; afaik things should be stable | 17:25 |
| mordred | fungi, pabelanger, clarkb: have a sec to +3 this: https://review.openstack.org/#/c/531916/ ? | 17:25 |
| *** e0ne has quit IRC | 17:25 | |
| pabelanger | mordred: +3 | 17:25 |
| mordred | pabelanger: here goes nothing | 17:26 |
| fungi | corvus: happy to try stopping an executor now and see what happens | 17:26 |
| pabelanger | mordred: also \o/ | 17:26 |
| fungi | infra-root: proposed maintenance plan for the zuulv3->zuul01 migration is https://etherpad.openstack.org/p/zuul01-scheduler-migration | 17:26 |
| corvus | fungi: if you meant 'service zuul-executor restart' -- yeah, that probably won't work on account of them taking 10 minutes to stop; but they should stop after ~10 minutes. :) | 17:27 |
| fungi | corvus: cool, any objection to me doing `service zuul-executor stop` on ze01 right now and then seeing what's still running after 15 minutes or so? | 17:28 |
| pabelanger | yah, I haven't had time to look into how to improve that yet :( | 17:28 |
| corvus | fungi: well, we're near capacity, so that will shift load to other executors; i think we should just observe during the actual maintenance | 17:28 |
| corvus | actually, we have a backlog, so we're over capacity | 17:29 |
| fungi | wfm. we can i suppose always leave one untouched after issuing stop and not start it back up until we get time to closely examine any remaining stray processes | 17:29 |
| corvus | incidentally, logstash job queue graph looks a bit like the bitcoin / USD chart. | 17:29 |
| *** jamesmcarthur has joined #openstack-infra | 17:30 | |
| corvus | (ie, an unsustainable climb) | 17:30 |
| pabelanger | also, regarding nodepool-launcher capacity. Not sure where we landed, but did we want to do a launcher per provider to help load issues? Or maybe bring up a 3rd / 4th launcher and split the config even more? | 17:31 |
| *** yamamoto has quit IRC | 17:31 | |
| pabelanger | We did say rebuild with 2GB instances, which I should be able to start today | 17:32 |
| corvus | easiest thing to do given our init situation right now would be 4x2GB launchers | 17:33 |
| pabelanger | I would agree | 17:33 |
| pabelanger | let me see what that provider config would look like over 4 launchers | 17:35 |
| *** pcaruana has quit IRC | 17:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Run shade functional tests on ansible OpenStack PRs https://review.openstack.org/531916 | 17:37 |
| *** sbra has joined #openstack-infra | 17:37 | |
| *** erlon has quit IRC | 17:37 | |
| *** yamamoto has joined #openstack-infra | 17:38 | |
| *** yamamoto has quit IRC | 17:38 | |
| *** gouthamr has joined #openstack-infra | 17:39 | |
| corvus | mordred: ^ | 17:41 |
| pabelanger | exciting | 17:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove nb03 / nb04 servers https://review.openstack.org/535804 | 17:41 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Correctly document default git dirs https://review.openstack.org/535864 | 17:43 |
| *** shardy_ has quit IRC | 17:44 | |
| AJaeger | mordred: your fetch-translations-stestr-output looks fine, thanks | 17:46 |
| *** yamahata has quit IRC | 17:46 | |
| clarkb | I'll look at logstash in a sec | 17:47 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/openstack-zuul-jobs master: Don't execute legacy-tempest-dsvm-py35 for doc changes https://review.openstack.org/535866 | 17:47 |
| corvus | fungi: maint plan looks good; added a couple notes to minimize outage time | 17:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create config for nl03.o.o https://review.openstack.org/535868 | 17:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Clean up nl01.o.o configuration https://review.openstack.org/535869 | 17:51 |
| *** caphrim007_ has joined #openstack-infra | 17:52 | |
| *** bobh has joined #openstack-infra | 17:53 | |
| pabelanger | okay, ^ moves inap / tripleo-test-cloud-rh1 / vexxhost (disabled) into nl03.o.o | 17:54 |
| pabelanger | leaving rackspace in nl01.o.o | 17:54 |
| AJaeger | mordred, rosmaita : glance_store change finished tests - looks ok to me. Can you confirm? | 17:55 |
| clarkb | pabelanger: we are going to reduce the size of the instances when we make more of them ya? | 17:55 |
| clarkb | rather than run more processes per instance? | 17:55 |
| *** caphrim007 has quit IRC | 17:55 | |
| pabelanger | clarkb: yah, down to 2GB instances. That is the easiest path forward currently | 17:55 |
| pabelanger | I like the idea of multiple launchers per server, but would require a bit of puppet work to make it happen | 17:56 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul-jobs master: [WIP] Debugging playbook: inject public keys in case of failure https://review.openstack.org/535803 | 17:56 |
| *** jpich has quit IRC | 17:57 | |
| fungi | corvus: thanks for the edits! | 17:58 |
| *** r-daneel has joined #openstack-infra | 17:59 | |
| corvus | fungi: if we're still waiting at step 8, we could probably update the cname | 18:00 |
| clarkb | re logstash the elasticsearch cluster is red. Therea re 7 unassigned shards from todays index. I think that is due to a lack of disk space on the cluster members so I am going to delete the oldest index to see if that causes things to be happier | 18:00 |
| fungi | corvus: oh, yup, i'll move it there | 18:00 |
| *** derekh has quit IRC | 18:01 | |
| clarkb | yup that caused shards to become assigned | 18:01 |
| clarkb | my hunch is that something like neutrons unittest logs being massive due to deprecation warnings is partially at fault here | 18:01 |
| clarkb | I am betting neutron isn't the only thing generating a ton of those warnings | 18:01 |
| clarkb | also I think we double log devstack logs in the new devstack job setup | 18:01 |
| clarkb | mordred: andreaf ^ fyi | 18:02 |
| *** yamahata has joined #openstack-infra | 18:02 | |
| *** camunoz has joined #openstack-infra | 18:02 | |
| fungi | ooh, yeah log archival has started creeping up over the last few days too: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 18:03 |
| fungi | it's easiest to see on the monthly graph | 18:03 |
| clarkb | fungi: you may want to reduce the TTL on the zuul.o.o cname to 5 minutes (it looks like it is at an hour now) | 18:04 |
| *** jamesmcarthur has quit IRC | 18:05 | |
| *** Goneri has joined #openstack-infra | 18:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Handle os-api-ref in build-api-ref job https://review.openstack.org/535875 | 18:06 |
| *** ralonsoh has quit IRC | 18:06 | |
| clarkb | number of queued logstash jobs is trending down now | 18:07 |
| clarkb | (slowly) | 18:07 |
| *** dizquierdo has quit IRC | 18:07 | |
| *** gouthamr has quit IRC | 18:08 | |
| *** vsaienk0 has quit IRC | 18:09 | |
| *** efoley has quit IRC | 18:09 | |
| *** ykarel|away has joined #openstack-infra | 18:11 | |
| AJaeger | config-core, I put up some current open changes - some a bit older - at http://paste.openstack.org/show/647210 for your review consideration | 18:12 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added member affilition endpoints https://review.openstack.org/535877 | 18:12 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added member affilition endpoints https://review.openstack.org/535877 | 18:14 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Tenant config dynamic loading of project from external script https://review.openstack.org/535878 | 18:14 |
| pabelanger | clarkb: any objection to 4x2GB instances corvus suggested? I should have time this afternoon on launching nl03 too | 18:14 |
| *** janki has quit IRC | 18:14 | |
| clarkb | pabelanger: no I think that should work fine | 18:16 |
| clarkb | and halve our total resource usage? | 18:16 |
| clarkb | mordred: andreaf https://review.openstack.org/535879 I think that may be part of the problem with devstack logging | 18:16 |
| pabelanger | clarkb: yah, rigt now we have 2x8GB instances, moving to 4x2GB would 1/2 it | 18:17 |
| *** gouthamr has joined #openstack-infra | 18:17 | |
| mordred | clarkb: so ... if we do that we'll lose the live streaming of the devstack content that we have with the new job currently | 18:17 |
| clarkb | mordred: yes thats intentional | 18:18 |
| clarkb | mordred: devstack is super noisy and we hide its logs away in the devstack log file | 18:18 |
| mordred | it is? it's intention for us to not be able to see any activity for 20 minutes? | 18:18 |
| clarkb | mordred: we intentionally did that in devstack-gate yes | 18:18 |
| mordred | weird. ok. I mean, if that's what we're doing on purpose | 18:18 |
| clarkb | long before we lost console streaming there | 18:18 |
| clarkb | if we want devstack logs in the foreground we shoul d stop also logging them in their own file | 18:19 |
| clarkb | its redundant and puts more pressure on the log server and elasticsearch cluster than necessary | 18:19 |
| mordred | yes. I definitely agree that we should not double log them | 18:19 |
| clarkb | mordred: but also I think the intent in that playbook was to do what I did since it even has the devstack_early_log file on the command | 18:19 |
| pabelanger | clarkb: okay, we'd need to first land: https://review.openstack.org/535868 which lets us bring nl03.o.o online. It also sets max-servers to 0 inap / tripleo-test-cloud-rh1 in nl01. So a we'd have a little outage when nodes stopped and until nl03.o.o was online | 18:20 |
| david-lyle | cd | 18:20 |
| *** gouthamr has quit IRC | 18:20 | |
| mordred | clarkb: kk. I think we can land that and treat discussion of intent and whether it's a good idea or not as a follow up | 18:20 |
| *** stevebaker has quit IRC | 18:21 | |
| clarkb | pabelanger: not an outage right? nodepool will continue booting instances in other clouds | 18:21 |
| pabelanger | clarkb: right | 18:21 |
| pabelanger | just lose some capacity for a few moments | 18:21 |
| Shrews | pabelanger: lgtm | 18:23 |
| fungi | clarkb: thanks for spotting the cname ttl. i'll fix it | 18:24 |
| *** dtantsur is now known as dtantsur|afk | 18:24 | |
| openstackgerrit | Merged openstack-infra/zuul master: Don't rely on canonical hostnames for CRD https://review.openstack.org/535858 | 18:25 |
| *** stevebaker has joined #openstack-infra | 18:26 | |
| pabelanger | Shrews: clarkb: thanks, we also need https://review.openstack.org/535857/ part of the stack | 18:27 |
| *** ykarel|away has quit IRC | 18:27 | |
| Shrews | pabelanger: +3 | 18:28 |
| mordred | corvus: if you have a sec - I see an entry in the zuul scheduler debug log for ansible/ansible in the check pipeline for job ... | 18:28 |
| mordred | corvus: 2018-01-19 17:37:34,592 INFO zuul.Pipeline.openstack.check: github.com/ansible/ansible | 18:28 |
| mordred | 2018-01-19 17:37:34,593 INFO zuul.Pipeline.openstack.check: <Job shade-ansible-devel-functional-devstack branches: None source: openstack-infra/project-config/zuul.d/projects.yaml@master#4 | 18:28 |
| mordred | but I don't see anything on the status page | 18:28 |
| pabelanger | Shrews: thanks | 18:28 |
| mordred | mordred: nor any information reported to the PR- not sure what I should investigate next and could use a pointer | 18:29 |
| AJaeger | mordred: are you talking with yourself? ;) | 18:30 |
| mordred | AJaeger: frequently | 18:30 |
| * AJaeger haven't seen that on IRC yet... | 18:31 | |
| *** weshay|rover is now known as weshay|bbiab | 18:31 | |
| corvus | mordred: looking | 18:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Test nl03 https://review.openstack.org/535897 | 18:32 |
| AJaeger | pabelanger: let's test nl03 as well ^ | 18:32 |
| corvus | mordred: what pr? | 18:32 |
| mordred | corvus: I'm also in a position to push a new update to a PR, if me doing such a thing would be helpful | 18:32 |
| mordred | corvus: https://github.com/ansible/ansible/pull/20974 | 18:32 |
| prometheanfire | ihrachys: https://review.openstack.org/535460 is an example of the neutron27 slowness | 18:33 |
| pabelanger | AJaeger: should we just make a tools script to loop over nodepool/*.yaml? | 18:34 |
| corvus | mordred: i don't see any github events in the log | 18:34 |
| ihrachys | prometheanfire, yeah I pushed one patch for neutron-lib to remove the major deprecation warning offender | 18:34 |
| pabelanger | AJaeger: but agree, we should test | 18:34 |
| ihrachys | prometheanfire, we are reviewing it now | 18:34 |
| ihrachys | we'll need a new neutron-lib release | 18:34 |
| AJaeger | pabelanger: I can do a script quickly... | 18:35 |
| corvus | mordred: let's check the installation and see if they're being delivered | 18:35 |
| prometheanfire | ihrachys: k | 18:35 |
| mordred | corvus: I don't either - but I do see that reference to github.com/ansible/ansible and shade-ansible-devel-functional-devstack ... | 18:35 |
| mordred | corvus: agree re: installation | 18:36 |
| corvus | mordred: right, those are zuul's configuration | 18:36 |
| corvus | mordred: it's getting 404s | 18:37 |
| *** jamesmcarthur has joined #openstack-infra | 18:37 | |
| mordred | ah. yes to both | 18:37 |
| mordred | The requested URL /connection/github/payload was not found on this server. | 18:37 |
| corvus | mordred: i assume the current rewrite config isn't correct | 18:38 |
| corvus | (and of course, once we land github-in-zuul-web, it'll probably change again) | 18:38 |
| mordred | cool. well - I shall investigate rewrite | 18:38 |
| mordred | corvus: /connection/github/payload should be the correct url yeah? | 18:38 |
| *** yamamoto has joined #openstack-infra | 18:39 | |
| corvus | mordred: yep. that looks correct | 18:40 |
| corvus | mordred: it'll be in the old webapp though | 18:40 |
| corvus | so should be on port 8001 | 18:40 |
| mordred | corvus: k. well, there is no rewrite rule for /connection at all | 18:40 |
| corvus | mordred: a rewrite of connection/.* to that should be fine at this point | 18:40 |
| hwoarang | hello. do you see any slowness with nodes today? some opensuse jobs are constantly failing when performing some db ops. The problem started sometime yesterday about this time | 18:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Test all nodepool fiels https://review.openstack.org/535897 | 18:41 |
| pabelanger | speaking of github support, did we make sure the right version of sigmavirus24/github3.py got installed on zuul01.o.o? For some reason pip install doesn't seem to install the right version from the requirements.txt file I've found | 18:41 |
| openstackgerrit | Merged openstack-infra/project-config master: Minimal nodepool-launcher config take 2 https://review.openstack.org/535857 | 18:41 |
| openstackgerrit | Merged openstack-infra/project-config master: Create config for nl03.o.o https://review.openstack.org/535868 | 18:41 |
| pabelanger | fungi: ^something to confirm | 18:41 |
| *** jamesmcarthur has quit IRC | 18:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Test all nodepool files https://review.openstack.org/535897 | 18:41 |
| AJaeger | pabelanger: updated to loop over all files ^ | 18:41 |
| pabelanger | AJaeger: lgtm | 18:42 |
| AJaeger | cool | 18:42 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Convert from legacy to native devstack job https://review.openstack.org/535899 | 18:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Add /connection to rewrite rules https://review.openstack.org/535900 | 18:43 |
| mordred | corvus, pabelanger: ^^ | 18:43 |
| *** armaan_ has quit IRC | 18:44 | |
| pabelanger | +2 | 18:44 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool master: Fix typo in nodepool-builder.init https://review.openstack.org/535829 | 18:44 |
| AJaeger | mordred: seeing your change: Reading zuul source code i expect that curl -s http://zuulv3.openstack.org/status/change/535897,1 works - but it does not. Do we need another rewrite rule? | 18:45 |
| pabelanger | okay, going to try bringing nb03.o.o online | 18:45 |
| corvus | mordred: the CRD syntax fix has landed, so once we do fungi's server move/restart, we should be able to test that out (though, you should be able to CRD to github right now, just not to gerrit) | 18:46 |
| pabelanger | sorry, nl03.o.o | 18:46 |
| rosmaita | infra cores who aren't AJaeger or mordred: please look over https://review.openstack.org/#/c/534431/ , it solves a problem i've been having trying to get glance_store functional test gate jobs weorking | 18:46 |
| corvus | AJaeger, mordred: there isn't a rewrite rule that covers /status/change/ | 18:47 |
| mordred | corvus: should I add one? | 18:48 |
| mordred | rosmaita: wot - the recheck worked? | 18:48 |
| rosmaita | mordred i believe so | 18:48 |
| mordred | rosmaita: yay! | 18:48 |
| corvus | AJaeger, mordred: i'm not sure zuul-web supports that | 18:48 |
| rosmaita | (looking again to double check) | 18:48 |
| *** sambetts is now known as sambetts|afk | 18:48 | |
| corvus | AJaeger, mordred: probably still needs to be ported over | 18:49 |
| rosmaita | mordred: https://review.openstack.org/#/c/535398/ shows a completely clean run at 12:32 pm ET | 18:49 |
| mordred | corvus: I agree -it is not there | 18:49 |
| mordred | rosmaita: \o/ | 18:49 |
| AJaeger | corvus: https://git.openstack.org/cgit/openstack-infra/zuul/tree/tests/unit/test_web.py#n100 - do we run the test? | 18:50 |
| *** yamamoto has quit IRC | 18:50 | |
| corvus | AJaeger: @skip("This is not supported by zuul-web") | 18:50 |
| corvus | AJaeger: so yes and no | 18:50 |
| * AJaeger is embarrassed for missing that line | 18:51 | |
| *** dave-mcc_ has joined #openstack-infra | 18:52 | |
| *** jamesmcarthur has joined #openstack-infra | 18:55 | |
| AJaeger | team, os-api-ref is not running a job due to a too strict files list, could you quickly review this one-liner change https://review.openstack.org/535875 to allow us testing, please? | 18:56 |
| openstackgerrit | Merged openstack-infra/system-config master: Allow robots to follow links from cgit https://review.openstack.org/427959 | 18:56 |
| openstackgerrit | Mark Hamzy proposed openstack/diskimage-builder master: fix the grub root label when an XFS disk label is truncated https://review.openstack.org/532279 | 18:57 |
| *** lucasagomes is now known as lucas-afk | 18:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Port per-change status to zuul-web https://review.openstack.org/535903 | 18:59 |
| mordred | AJaeger, corvus: ^^ | 18:59 |
| AJaeger | wow, thanks, mordred ! | 18:59 |
| *** myoung is now known as myoung|food | 18:59 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Open firewalls for nl03.o.o https://review.openstack.org/535904 | 19:00 |
| pabelanger | infra-root: nl03.o.o ^ ready to come online, if you'd like to review. is a 2GB instance | 19:00 |
| *** sree has joined #openstack-infra | 19:01 | |
| Shrews | pabelanger: i'll log in and poke at the logs | 19:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Create per-site template directories https://review.openstack.org/535848 | 19:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Use multisite template dir https://review.openstack.org/535852 | 19:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Remove openstack HTML templates https://review.openstack.org/535855 | 19:03 |
| Shrews | pabelanger: aaaaaand i see nothing but zk connection errors. suppose we need the firewall change :) | 19:03 |
| pabelanger | Shrews: yah, 535904 will open but we'll also need to stop / start nodepool-launcher again. I believe it will never try to connect to zookeeper again, even after we open firewall | 19:04 |
| Shrews | pabelanger: i'm ready to +3 if you are | 19:05 |
| clarkb | pabelanger: approved | 19:05 |
| clarkb | oh I just approved it | 19:05 |
| pabelanger | yup, thanks! | 19:05 |
| Shrews | clarkb: all good | 19:05 |
| corvus | with the new way of doing the dns->ip translation for firewalls in puppet, those changes are self-testing and should generally be safe to approve | 19:06 |
| *** armaan has joined #openstack-infra | 19:06 | |
| *** sree has quit IRC | 19:07 | |
| pabelanger | Hmm, max-servers is 0 for inap on nl01 now, but I see 2 locked / ready nodes. Wonder if they will never unlock now. | 19:07 |
| corvus | pabelanger: who has them locked? | 19:08 |
| pabelanger | corvus: looking at that now | 19:08 |
| Shrews | pabelanger: they *should*. the pool thread should still process pending requests | 19:08 |
| *** electrofelix has quit IRC | 19:09 | |
| pabelanger | Hmm, how can I check that again? Do I need zk-shell? | 19:09 |
| Shrews | that's black magic. i can poke | 19:09 |
| pabelanger | k, thanks | 19:09 |
| corvus | when did max-servers drop to 0? | 19:10 |
| fungi | i think that's one they asked us to stop using for a while | 19:10 |
| fungi | checking git history now | 19:10 |
| Shrews | pabelanger: oh my, they've been locked for more than 9 hours | 19:11 |
| pabelanger | Shrews: yah, sorry I should have said that | 19:11 |
| corvus | what are the ids? | 19:11 |
| Shrews | 0002010606 and 0002010643 | 19:11 |
| fungi | sorry, i'm confusing it with one of the citycloud regions | 19:12 |
| pabelanger | corvus: about 20mins ago | 19:12 |
| *** e0ne has joined #openstack-infra | 19:12 | |
| Shrews | for requests 300-0002174379 and 300-0002174397, respectively | 19:13 |
| Shrews | corvus: ^^^ | 19:13 |
| Shrews | neither of which exist, so that's perplexing | 19:13 |
| corvus | well they've been fulfilled | 19:14 |
| corvus | 300-0002174379 was for a propose-requirements update job; i wonder if there's a bad interaction with semaphores here | 19:15 |
| corvus | this is worth digging into, but i have to run right now. | 19:15 |
| fungi | the change to add a semaphore to that job did just merge | 19:16 |
| fungi | in the past few hours i think | 19:16 |
| AJaeger | corvus: merged this morning - 9:40 UTC | 19:17 |
| *** e0ne has quit IRC | 19:17 | |
| fungi | 09:40z | 19:18 |
| fungi | yeah, just found it | 19:18 |
| fungi | my sense of time is terrible | 19:18 |
| fungi | so nearly 10 hours ago | 19:18 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added delete RSVP endpoint https://review.openstack.org/535905 | 19:18 |
| AJaeger | fungi: and i think we had some requirements jobs in post pipeline at that time, so could indeed be some strange interaction | 19:19 |
| *** jamesmcarthur has quit IRC | 19:19 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added delete RSVP endpoint https://review.openstack.org/535905 | 19:20 |
| pabelanger | clarkb: corvus: Shrews: with new quota API in nodepool, if we set max-servers: 0 in nl01 for inap (and nodes are still being deleted) can we start nl03.o.o with inap provider and max-servers again? I would think there is some protection on not using more quota then supported? | 19:20 |
| pabelanger | or do we need to have all nodes first deleted then start nl03? | 19:20 |
| clarkb | I think max-servers is local config so the side with max servers 0 will continue to honor it while the side without max severs set to 0 will boot instances | 19:21 |
| clarkb | I think this means it is safe to run in that configuration? | 19:21 |
| AJaeger | could we enable testing of nl03 file in project-config, please? https://review.openstack.org/#/c/535897/ is the review for that... | 19:22 |
| pabelanger | clarkb: yah, I mean we can try right? I don't think nl01 will start deleting things behind the back of nl03, that my concern | 19:22 |
| clarkb | pabelanger: ya I don't think it will since max-servers doesn't say anything about deleting, only about booting | 19:23 |
| pabelanger | kk, I've give it a try one firewall is open | 19:23 |
| *** sree has joined #openstack-infra | 19:23 | |
| *** felipemonteiro has joined #openstack-infra | 19:24 | |
| *** amoralej is now known as amoralej|off | 19:24 | |
| Shrews | pabelanger: i *think* that's safe | 19:25 |
| pabelanger | :) | 19:26 |
| pabelanger | we'll know more shortly I think | 19:26 |
| *** jamesmcarthur has joined #openstack-infra | 19:27 | |
| *** felipemonteiro_ has joined #openstack-infra | 19:28 | |
| *** sree has quit IRC | 19:29 | |
| *** makowals has joined #openstack-infra | 19:30 | |
| *** florianf has quit IRC | 19:30 | |
| Shrews | corvus: pabelanger: launcher-debug.log.2018-01-19_07:2018-01-19 09:41:00,695 DEBUG nodepool.driver.openstack.OpenStackNodeRequestHandler[nl01.openstack.org-30110-PoolWorker.inap-mtl01-main]: Unlocked node 0002010606 for request 300-0002174379 | 19:30 |
| Shrews | i'm very confused | 19:30 |
| *** harlowja has joined #openstack-infra | 19:31 | |
| *** florianf has joined #openstack-infra | 19:31 | |
| *** florianf has quit IRC | 19:31 | |
| *** felipemonteiro has quit IRC | 19:31 | |
| pabelanger | odd | 19:32 |
| Shrews | zk 'dump' output seems to indicate that it might be zuul holding the lock | 19:33 |
| AJaeger | Shrews: so, one minute after the change to add a semphore merged | 19:33 |
| *** jpena is now known as jpena|afk | 19:33 | |
| Shrews | AJaeger: what change is that? | 19:33 |
| *** gema has quit IRC | 19:34 | |
| AJaeger | Shrews: https://review.openstack.org/#/c/535462/ | 19:34 |
| *** jamesmcarthur has quit IRC | 19:35 | |
| pabelanger | oh, so maybe something in post pipeline is holding nodes | 19:37 |
| pabelanger | but because check / gate backlog isn't getting rest of nodes in post? | 19:37 |
| fungi | oldest changes in post are under 4 hours | 19:38 |
| *** vsaienk0 has joined #openstack-infra | 19:38 | |
| pabelanger | yah, looks like that | 19:38 |
| *** hashar has quit IRC | 19:38 | |
| fungi | oldest change i see in any pipeline is only ~6.5 hours | 19:39 |
| *** sree has joined #openstack-infra | 19:39 | |
| *** sbra has quit IRC | 19:40 | |
| *** sbra has joined #openstack-infra | 19:40 | |
| *** myoung|food is now known as myoung | 19:41 | |
| smcginnis | If anyone has a moment - some paperwork that needs to be done for the stable branching: https://review.openstack.org/#/c/535847/ | 19:41 |
| *** caphrim007_ has quit IRC | 19:42 | |
| *** caphrim007 has joined #openstack-infra | 19:42 | |
| fungi | that file still gets used by the new devstack jobs right? | 19:43 |
| * fungi seems to recall it's the only thing from the devstack-gate repo that playbook relies on | 19:43 | |
| clarkb | fungi: yes | 19:44 |
| clarkb | this was a surprise recently to me but ya it does | 19:44 |
| smcginnis | fungi: I'll admit I did not completely dig into it. I was following the lead of past times when the file was located in the root of the repo. | 19:44 |
| fungi | odds are this will change by next release | 19:44 |
| mat128 | anyone up for a second review on https://review.openstack.org/#/c/531604/ and https://review.openstack.org/#/c/531608/ ? charm PTL ok'd already :) | 19:45 |
| smcginnis | Good to know. I was going to update the release process with a little more detail, but I'll hold off on that until things settle. | 19:45 |
| smcginnis | clarkb, fungi: Thanks for the help! | 19:45 |
| corvus | Shrews: yeah, i mentioned what job it was for above | 19:45 |
| Shrews | corvus: pabelanger: zuul log shows it getting the node for request 300-0002174379. Following the log after that hurts my brain | 19:45 |
| corvus | Shrews: 19:15 < corvus> 300-0002174379 was for a propose-requirements update job; i wonder if there's a bad interaction with semaphores here | 19:46 |
| fungi | smcginnis: my guess is that there will be an equivalent change to the devstack repo at that point | 19:46 |
| *** sree has quit IRC | 19:46 | |
| fungi | but for now it still relies on that file from d-g instead | 19:46 |
| Shrews | corvus: k. sorry, didn't see that | 19:46 |
| smcginnis | fungi: Thanks, I'll watch for that change. Either way, sounds like I'll need to update the process. | 19:46 |
| *** e0ne has joined #openstack-infra | 19:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Port per-change status to zuul-web https://review.openstack.org/535903 | 19:47 |
| *** manjeets has quit IRC | 19:50 | |
| *** ldnunes has quit IRC | 19:50 | |
| *** ldnunes has joined #openstack-infra | 19:51 | |
| *** vsaienk0 has quit IRC | 19:52 | |
| *** jamesmcarthur has joined #openstack-infra | 19:53 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/openstack-zuul-jobs master: Switched all jobs from q-qos to neutron-qos https://review.openstack.org/532948 | 19:55 |
| *** smatzek_ has quit IRC | 19:58 | |
| *** vsaienk0 has joined #openstack-infra | 19:59 | |
| * AJaeger waves good bye and wishes everybody a great weekend. Would appreciate review of https://review.openstack.org/#/c/535875/ and https://review.openstack.org/#/c/535897/ | 20:00 | |
| *** sree has joined #openstack-infra | 20:01 | |
| clarkb | fungi: I'm going to find lunch now, but zuul cname looks good and its been an hour and a half so we should be good to do switch soon? | 20:02 |
| fungi | clarkb: any time, yeah, though i do eventually need to start cooking dinner | 20:04 |
| fungi | also gema sent me initial credentials to their arm64 openstack deployment in linaro earlier today. i have yet to find time to log in and reset the passwords so i can record them | 20:05 |
| *** sree has quit IRC | 20:05 | |
| *** jamesmcarthur has quit IRC | 20:05 | |
| fungi | hopefully will get to that forthwith | 20:05 |
| *** slaweq has joined #openstack-infra | 20:05 | |
| *** sree has joined #openstack-infra | 20:06 | |
| *** pramodrj07 has joined #openstack-infra | 20:07 | |
| pabelanger | clarkb: okay, that is another 30 nodes in rax online now for nodepool | 20:08 |
| *** ldnunes has quit IRC | 20:08 | |
| *** manjeets has joined #openstack-infra | 20:08 | |
| *** vsaienk0 has quit IRC | 20:09 | |
| pabelanger | clarkb: dfw says we can do 222 instances, but set max-servers to 140. Do you know why that would be? | 20:13 |
| *** felipemonteiro__ has joined #openstack-infra | 20:14 | |
| clarkb | I do not | 20:14 |
| clarkb | I know cloudnull got some quota bumps in maybe we didnt update dfw? | 20:14 |
| *** sree has quit IRC | 20:15 | |
| fungi | it's also possible in some providers they have set our quota higher for bursting under extreme circumstances but would rather we limit activity there when possible | 20:16 |
| *** jamesmcarthur has joined #openstack-infra | 20:16 | |
| fungi | but i would recommend using blame/pickaxe/log -p to see if you can spot when max-servers was last adjusted there | 20:17 |
| *** felipemonteiro_ has quit IRC | 20:17 | |
| pabelanger | sure, I can dig more into it | 20:18 |
| openstackgerrit | Merged openstack-infra/system-config master: Open firewalls for nl03.o.o https://review.openstack.org/535904 | 20:19 |
| fungi | increasing our cap in some providers without advance warning/discussion could take them by surprise | 20:19 |
| *** dprince has quit IRC | 20:20 | |
| *** dprince has joined #openstack-infra | 20:20 | |
| *** e0ne has quit IRC | 20:21 | |
| pabelanger | fungi: yah, was online looking into it because of leaked nodes in rax-ord. And if we were ready to discuss removing max-server caps in nodepool.yaml. Agree we don't want to just start maxing it out without providers knowing | 20:21 |
| pabelanger | rax-iad looks to have quota of 192 but we cap at 145 | 20:22 |
| *** fried_rolls is now known as fried_rice | 20:23 | |
| *** jpena|afk is now known as jpena|off | 20:24 | |
| *** jamesmcarthur has quit IRC | 20:29 | |
| *** rfolco|rover is now known as rfolco|rover|off | 20:30 | |
| pabelanger | okay, nl03.o.o has been started | 20:34 |
| pabelanger | look to be launching nodes too | 20:35 |
| *** e0ne has joined #openstack-infra | 20:35 | |
| pabelanger | #status log nl03.o.o is now online and launching nodes | 20:35 |
| openstackstatus | pabelanger: finished logging | 20:35 |
| pabelanger | clarkb: ^ | 20:36 |
| *** e0ne has quit IRC | 20:37 | |
| *** weshay|bbiab is now known as weshay | 20:38 | |
| *** weshay is now known as weshay|ruck | 20:38 | |
| *** weshay|ruck is now known as weshay|rover | 20:38 | |
| *** vsaienk0 has joined #openstack-infra | 20:38 | |
| clarkb | cool | 20:39 |
| clarkb | fungi: I'm mostly done with lunch I think if we want to start onzuul things | 20:39 |
| *** sree has joined #openstack-infra | 20:39 | |
| fungi | sure, i didn't catch whether corvus was planning to return today to know whether we should wait on him | 20:40 |
| *** smatzek has joined #openstack-infra | 20:41 | |
| clarkb | I don't mind waiting | 20:42 |
| clarkb | pabelanger: also that launcher should be running the fix for cloud failures | 20:42 |
| pabelanger | clarkb: k | 20:43 |
| corvus | i've returned | 20:43 |
| fungi | cool... what's the general feeling about the overall load on the system nowish? | 20:44 |
| *** sree has quit IRC | 20:44 | |
| *** smatzek has quit IRC | 20:44 | |
| fungi | grafana says the post backlog is starting to get cleared out finally | 20:45 |
| pabelanger | I have some errand to run, but should be back in an hour or so | 20:45 |
| corvus | fungi: i think we're good | 20:45 |
| corvus | it's all downhill from here | 20:45 |
| *** smatzek has joined #openstack-infra | 20:45 | |
| fungi | though we're still pretty full on capacity for the moment so wondering if we should let it catch up any more first | 20:45 |
| corvus | friday afternoon and all | 20:45 |
| *** eharney has quit IRC | 20:45 | |
| pabelanger | inap is coming back online now, so that will also help | 20:46 |
| *** gema has joined #openstack-infra | 20:48 | |
| *** smatzek has quit IRC | 20:51 | |
| *** vsaienk0 has quit IRC | 20:56 | |
| clarkb | fungi: is first step of disabling puppet on those nodes done? | 20:57 |
| clarkb | we should be able to do that early without impact | 20:57 |
| *** jamesmcarthur has joined #openstack-infra | 20:57 | |
| fungi | good point, i can add them to the emergency file now if we think we're ready to start soon | 20:58 |
| *** evin has joined #openstack-infra | 20:58 | |
| clarkb | ++ | 20:58 |
| fungi | globs work without prepending ~ right? | 20:58 |
| clarkb | aiui yes. ~ only required for regexing | 20:59 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/puppet-translation_checksite master: DNM testing whether lib/neutron switch breaks this repo https://review.openstack.org/535952 | 20:59 |
| fungi | so ze*.openstack.org and zm*.openstack.org can just go right into the emergency file | 20:59 |
| *** sbra has quit IRC | 20:59 | |
| clarkb | I think so yes | 20:59 |
| fungi | #status log temporarily added ze*.openstack.org and zm*.openstack.org to the emergency disable list in preparation for replacing zuulv3 with zuul01 | 21:00 |
| openstackstatus | fungi: finished logging | 21:00 |
| corvus | gee that sounds like we're going backwards :) | 21:00 |
| fungi | we do versioning on base 4 modular arithmetic | 21:00 |
| fungi | though i guess we should be using zuul10 in that case | 21:01 |
| *** jamesmcarthur has quit IRC | 21:03 | |
| *** e0ne has joined #openstack-infra | 21:04 | |
| *** vsaienk0 has joined #openstack-infra | 21:04 | |
| *** e0ne has quit IRC | 21:05 | |
| fungi | what's the best indicator of zuul backlog in the grafana dashboard? i would have thought the waiting jobs line on the zuul job queue graph, but it seems to almost always be ~0 | 21:06 |
| corvus | fungi: node requests | 21:07 |
| *** sree has joined #openstack-infra | 21:07 | |
| corvus | jobs wait on nodes now | 21:07 |
| fungi | ahh, so that's the current number of outstanding node requests not yet fulfilled | 21:07 |
| corvus | yep | 21:07 |
| corvus | 2018-01-19 19:32:15.884579 | primary | Notice: /Stage[main]/Zuul/Exec[install_zuul]/returns: aiohttp requires Python '>=3.4.2' but the running Python is 2.7.12 | 21:07 |
| corvus | the puppet-beaker-rspec-infra job is failing with that ^ | 21:08 |
| corvus | i'm guessing it's incompatible with zuulv3 now it's on master | 21:08 |
| fungi | so with it being ~400 right now, that's the number of nodes we'd still need to start all the remaining jobs it knows it wants to run at this point | 21:08 |
| corvus | er, for puppet-zuul i should have said | 21:08 |
| *** jamesmcarthur has joined #openstack-infra | 21:08 | |
| corvus | fungi: yep | 21:09 |
| *** sree has quit IRC | 21:11 | |
| *** jtomasek has quit IRC | 21:14 | |
| *** beekneemech has quit IRC | 21:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add /connection to rewrite rules https://review.openstack.org/535900 | 21:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Update spec acceptance to zuulv3 https://review.openstack.org/535966 | 21:14 |
| corvus | mordred: ^ | 21:14 |
| *** jamesmcarthur has quit IRC | 21:14 | |
| *** ekcs has joined #openstack-infra | 21:16 | |
| *** snapiri- has joined #openstack-infra | 21:16 | |
| *** tpsilva has quit IRC | 21:17 | |
| *** bnemec has joined #openstack-infra | 21:17 | |
| *** vsaienk0 has quit IRC | 21:19 | |
| *** vsaienk0 has joined #openstack-infra | 21:20 | |
| *** jamesmcarthur has joined #openstack-infra | 21:22 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Set up stable/queens https://review.openstack.org/535847 | 21:22 |
| *** bobh has quit IRC | 21:23 | |
| *** jamesmcarthur has quit IRC | 21:26 | |
| *** sree has joined #openstack-infra | 21:28 | |
| *** makowals has quit IRC | 21:28 | |
| clarkb | corvus: is that test fail going to prevent us from doing fungis server swap? | 21:29 |
| *** e0ne has joined #openstack-infra | 21:29 | |
| corvus | clarkb: i don't think so; it doesn't look like 534897 runs the puppet-zuul beaker tests, only openstackci-beaker; hopefully it's using the allinone thing and is okay? | 21:31 |
| corvus | i just rechecked though | 21:31 |
| *** sree has quit IRC | 21:32 | |
| *** e0ne has quit IRC | 21:33 | |
| *** rossella_s has quit IRC | 21:33 | |
| *** jamesmcarthur has joined #openstack-infra | 21:33 | |
| *** rossella_s has joined #openstack-infra | 21:35 | |
| *** jamesmcarthur has quit IRC | 21:39 | |
| *** rlandy has quit IRC | 21:43 | |
| *** dbecker has quit IRC | 21:44 | |
| *** jamesmcarthur has joined #openstack-infra | 21:44 | |
| *** rossella_s has quit IRC | 21:45 | |
| *** rossella_s has joined #openstack-infra | 21:47 | |
| *** wolverineav has quit IRC | 21:48 | |
| *** jamesmcarthur has quit IRC | 21:49 | |
| *** snapiri- has quit IRC | 21:52 | |
| *** jamesmcarthur has joined #openstack-infra | 21:53 | |
| *** jcoufal has quit IRC | 21:53 | |
| *** sree has joined #openstack-infra | 21:54 | |
| *** jamesmcarthur has quit IRC | 21:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add website hosting infrastructure https://review.openstack.org/527548 | 21:57 |
| *** dbecker has joined #openstack-infra | 21:57 | |
| clarkb | the grafana graph seems to show zuul + nodepool are near caught up | 21:58 |
| *** sree has quit IRC | 21:58 | |
| *** vsaienk0 has quit IRC | 21:59 | |
| fungi | i took the opportunity to whip up a skillet of spaghetti aglio e olio, but nearly done eating now | 22:00 |
| *** eharney has joined #openstack-infra | 22:00 | |
| corvus | i read that as 'never caught up' | 22:00 |
| *** jamesmcarthur has joined #openstack-infra | 22:00 | |
| fungi | ne'er caught up | 22:01 |
| *** tesseract has quit IRC | 22:01 | |
| *** dprince has quit IRC | 22:01 | |
| *** jamesmcarthur has quit IRC | 22:05 | |
| *** rossella_s has quit IRC | 22:05 | |
| *** rossella_s has joined #openstack-infra | 22:07 | |
| *** vsaienk0 has joined #openstack-infra | 22:07 | |
| *** jamesmcarthur has joined #openstack-infra | 22:11 | |
| *** Goneri has quit IRC | 22:14 | |
| clarkb | the post queue seems to be clearing up | 22:16 |
| clarkb | are we waiting for any particular number or status of things? | 22:16 |
| *** jamesmcarthur has quit IRC | 22:16 | |
| fungi | nah, i was just getting dishes cleared, sorry | 22:16 |
| fungi | results came back +1 on 534897 still so good to go there | 22:17 |
| fungi | just 4 minutes or so ago | 22:17 |
| fungi | lemme switch back to my terminal in the lab real fast and then i'm ready to go whenever we think is good | 22:18 |
| corvus | number of dirty dishes should be a metric on the grafana dashboard | 22:20 |
| clarkb | I dunno mine may never hit zero | 22:22 |
| corvus | clarkb, fungi, dmsimard: it looks like the install-ara-safely exec fails the idempotency check in the beaker job: http://logs.openstack.org/66/535966/1/check/legacy-puppet-beaker-rspec-infra/5e36dde/job-output.txt.gz#_2018-01-19_22_15_41_184624 | 22:22 |
| clarkb | thats odd, it shouldn't if the check is working | 22:23 |
| clarkb | I wonder if that is a behavior of the exec? | 22:23 |
| *** jamesmcarthur has joined #openstack-infra | 22:24 | |
| * fungi double-checks that puppet has skipped ze01 for the past while | 22:26 | |
| *** vsaienk0 has quit IRC | 22:27 | |
| clarkb | I wonder if we have ara pinned somewhere? | 22:27 |
| fungi | um | 22:27 |
| clarkb | zuul doesn't dep on ara though so the only thing installing it should be that exec | 22:27 |
| fungi | last puppet update on ze01 was 16:11z on wednesday | 22:28 |
| fungi | i wonder whether puppeting things has broken in the past 48 hours | 22:28 |
| *** rossella_s has quit IRC | 22:28 | |
| clarkb | I checked the ansible puppet logs yesterday as part of the feature branch removal | 22:29 |
| clarkb | it lookedl ike it was generally working | 22:29 |
| corvus | clarkb, fungi: before then, we had install-ara-safely run every run | 22:29 |
| *** jamesmcarthur has quit IRC | 22:30 | |
| corvus | maybe those 3 lines of bash should be ruby. :| | 22:30 |
| *** rossella_s has joined #openstack-infra | 22:31 | |
| clarkb | corvus: ze01 not running puppet is related to the ara test failure? | 22:31 |
| *** wolverineav has joined #openstack-infra | 22:31 | |
| fungi | as of yet i haven't determined why ze* and zm* haven't puppeted for over two days | 22:31 |
| corvus | clarkb: not that i'm aware of. ara test failure is harmless other than it means we can't have it in the beaker test (and also it runs too much). | 22:32 |
| fungi | there's simply no mention of them in the puppet_run_all and puppet_run_all_cron after a couple days ago | 22:32 |
| *** larainema has quit IRC | 22:32 | |
| clarkb | oh I see you are saying the ara thing ran every time until it stopped running puppet completeyl | 22:32 |
| clarkb | fungi: inventory issue maybe? | 22:33 |
| clarkb | or were they disabled? | 22:33 |
| fungi | clarkb: afaik not until an hour or so ago | 22:33 |
| *** sree has joined #openstack-infra | 22:33 | |
| fungi | the entries for them in the emergency disable file are the two i added | 22:33 |
| *** jamesmcarthur has joined #openstack-infra | 22:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Update spec acceptance to zuulv3 https://review.openstack.org/535966 | 22:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add /connection to rewrite rules https://review.openstack.org/535900 | 22:34 |
| fungi | oh, i wonder did https://review.openstack.org/534463 break them? | 22:34 |
| clarkb | ze01 is in the inventory cache file | 22:34 |
| fungi | nevermind, 534463 doesn't seem to have touched anything related to those nodes | 22:35 |
| *** vsaienk0 has joined #openstack-infra | 22:35 | |
| fungi | and zuul01 is properly updating at the same times as zuulv3 | 22:35 |
| fungi | according to logs | 22:36 |
| *** rossella_s has quit IRC | 22:36 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add TODO about fixing ara idempotency https://review.openstack.org/535982 | 22:36 |
| *** edmondsw has quit IRC | 22:37 | |
| fungi | `ansible '*' --list-hosts` includes ze01.openstack.org and others | 22:37 |
| *** sree has quit IRC | 22:37 | |
| openstackgerrit | Mark Hamzy proposed openstack/diskimage-builder master: upgrade pip before using -c option https://review.openstack.org/535983 | 22:40 |
| *** rossella_s has joined #openstack-infra | 22:40 | |
| *** jamesmcarthur has quit IRC | 22:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Create per-site template directories https://review.openstack.org/535848 | 22:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Use multisite template dir https://review.openstack.org/535852 | 22:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Remove openstack HTML templates https://review.openstack.org/535855 | 22:41 |
| fungi | though also it seems to list "ze*.openstack.org" and "zm*.openstack.org" literally, leading me to suspect that those globs in the emergency disable list won't actually do anything if we fix whatever's been preventing them from getting updated in the first place | 22:41 |
| clarkb | fungi: I wonder if ze* and zm* are in a group called zuul | 22:42 |
| clarkb | and so !zuul* does match them | 22:42 |
| fungi | good call! | 22:42 |
| fungi | it does indeed match them | 22:42 |
| *** wolverineav has quit IRC | 22:43 | |
| fungi | should i instead exclude zuul-scheduler since zuul01 is a member of that group (currently the only member)? | 22:43 |
| clarkb | well we don't want puppet running on anyways right? and we are undoing this once we've switched servers? | 22:44 |
| *** camunoz has quit IRC | 22:44 | |
| *** jamesmcarthur has joined #openstack-infra | 22:44 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add website hosting infrastructure https://review.openstack.org/527548 | 22:44 |
| fungi | yeah, but those nodes are now also two days behind updates in general | 22:44 |
| fungi | and the exclusions in 534463 aren't going away | 22:45 |
| *** vsaienk0 has quit IRC | 22:45 | |
| fungi | since that's how we lump the zuul scheduler in with gerrit and git servers to update separately from the rabble | 22:45 |
| clarkb | oh I didn't realize we were doing that | 22:45 |
| clarkb | in that case ya we'll have to update the match there | 22:46 |
| corvus | yeah it's sequenced so we can add things to the tenant config file when we create projects | 22:46 |
| fungi | something to do with avoiding telling the zuul scheduler about a new repo until it had been created on the git servers | 22:46 |
| fungi | right. that | 22:46 |
| *** rossella_s has quit IRC | 22:46 | |
| openstackgerrit | Mark Hamzy proposed openstack/diskimage-builder master: upgrade pip before using -c option https://review.openstack.org/535983 | 22:47 |
| *** jamesmcarthur has quit IRC | 22:48 | |
| *** rossella_s has joined #openstack-infra | 22:49 | |
| *** ekcs has quit IRC | 22:49 | |
| *** jamesmcarthur has joined #openstack-infra | 22:50 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Use zuul-scheduler group instead of zuul* glob https://review.openstack.org/535986 | 22:52 |
| fungi | clarkb: corvus: that ^ work for a fix? | 22:52 |
| clarkb | I think it should. You could probably also add .openstack.org to the _else file to fix that one | 22:52 |
| clarkb | if we wanted to rely on hostname globs and not group globs | 22:52 |
| fungi | and i've individually listed the executors and mergers in the emergency disable list so i can be sure they'll continue to get skipped | 22:54 |
| *** jamesmcarthur has quit IRC | 22:54 | |
| fungi | until we get the maintenance done | 22:55 |
| *** vsaienk0 has joined #openstack-infra | 22:57 | |
| *** dave-mcc_ has quit IRC | 22:58 | |
| *** jamesmcarthur has joined #openstack-infra | 22:58 | |
| *** felipemonteiro__ has quit IRC | 22:59 | |
| clarkb | I think the puppet exec problem is a whitespace problem | 23:01 |
| clarkb | I'm working on a fix | 23:01 |
| clarkb | otherwise, are we ready to do the zuul stuff? | 23:02 |
| *** jamesmcarthur has quit IRC | 23:03 | |
| *** jamesmcarthur has joined #openstack-infra | 23:03 | |
| *** jamesmcarthur has quit IRC | 23:04 | |
| fungi | yep, i've removed my wip from 534897 and it should (in theory) be safe to approve at will | 23:05 |
| corvus | done | 23:06 |
| *** jamesmcarthur has joined #openstack-infra | 23:06 | |
| fungi | <mr_burns>exxxxellent</mr_burns> | 23:06 |
| *** esberglu has quit IRC | 23:07 | |
| corvus | neat, my firefox just decided google has a bad ssl cert | 23:07 |
| fungi | i'm sure if mozilla had a search site, chrome would claim that had a bad ssl cert | 23:07 |
| fungi | just to maintain balance in the universe | 23:08 |
| corvus | they probably do but it's just not indexed in google so we don't know about it | 23:09 |
| fungi | touché | 23:09 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-zuul master: Fix whitespace in ara install version checking https://review.openstack.org/535991 | 23:09 |
| clarkb | corvus: ^ wasn't entirely sure what needed to eb done with beaker so thats missing btu I think that should fix it | 23:09 |
| *** evin has quit IRC | 23:09 | |
| corvus | clarkb: just add a zuul::executor line to the rspec file. i'll amend your change | 23:10 |
| *** sbra has joined #openstack-infra | 23:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Fix whitespace in ara install version checking https://review.openstack.org/535991 | 23:11 |
| corvus | clarkb: there we go | 23:11 |
| jlvillal | clarkb, Did you know OVH is building a data center in Hillsboro? I was driving by the old Rodgers Instruments building here in Hillsboro and wondered why the place needed these massive power lines. | 23:13 |
| jlvillal | clarkb, Did some digging and figured out that OVH is building their new data center at 1300 NE 25th AVE, Hillsboro, OR. | 23:14 |
| fungi | woah | 23:14 |
| *** rossella_s has quit IRC | 23:14 | |
| *** leakypipes has quit IRC | 23:14 | |
| fungi | maybe they'd be willing to colo some stuff for us ;) | 23:14 |
| clarkb | jlvillal: I did | 23:14 |
| jlvillal | fungi, heh | 23:15 |
| clarkb | jlvillal: I didn't know what the location is | 23:15 |
| clarkb | but I have patiently been waiting for them to open that DC so I can move my server for lower latency | 23:15 |
| jlvillal | clarkb, I figured it out by doing a permit search on the address. I was trying to figure out what they were doing. | 23:15 |
| fungi | is there a special trick to getting finger to not disconnect when connecting to a zuul log stream? | 23:15 |
| jlvillal | clarkb, I thought you could use that place that greghaynes was using in downtown Portland. | 23:16 |
| clarkb | jlvillal: I don't like having to deal with hardware anymore. OpenStack has spoiled me | 23:16 |
| fungi | i think my finger client must have an inactivity timeout or something because it was disconnecting on a lengthy pause in the log stream | 23:16 |
| jlvillal | clarkb, Ah. Okay virtual server. I though you meant a real bare-metal | 23:16 |
| *** rossella_s has joined #openstack-infra | 23:17 | |
| jlvillal | clarkb, Yeah. I may sign up for a VM once they are up and running. I see they have two job openings listed. | 23:17 |
| jlvillal | clarkb, So if you know someone who wants to be a data center technician you can let them know | 23:17 |
| clarkb | fungi: we are just waiting for the change to merge at this point? | 23:19 |
| fungi | yup | 23:19 |
| fungi | zuul says eta 6 minutes | 23:20 |
| clarkb | jlvillal: someone went into that large building at cornelius pass and evergreen that sat empty for years too | 23:20 |
| fungi | clarkb: though may want to wait for the globbing fix we just approved to make it through too since it's already in the gate at this point | 23:20 |
| clarkb | ok | 23:20 |
| fungi | all but two of its jobs have nodes assigned | 23:20 |
| jlvillal | clarkb, Hmm. I'm not sure I know that building. I'll look for it next time I drive through that area. | 23:21 |
| clarkb | google says infomart data centers | 23:22 |
| jlvillal | Ah | 23:23 |
| fungi | zuul is now estimating 11 minutes till merge on the second change | 23:23 |
| *** dbecker has quit IRC | 23:23 | |
| *** jamesmcarthur has quit IRC | 23:25 | |
| *** sree has joined #openstack-infra | 23:26 | |
| openstackgerrit | Merged openstack-infra/system-config master: Switch production from zuulv3.o.o to zuul01.o.o https://review.openstack.org/534897 | 23:27 |
| *** slaweq has quit IRC | 23:27 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Update spec acceptance to zuulv3 https://review.openstack.org/535966 | 23:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add /connection to rewrite rules https://review.openstack.org/535900 | 23:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Add TODO about fixing ara idempotency https://review.openstack.org/535982 | 23:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Fix whitespace in ara install version checking https://review.openstack.org/535991 | 23:28 |
| clarkb | /sys/devices/system/cpu/vulnerabilities is apparently the most correct way of checking your cpu patching | 23:29 |
| clarkb | but my suse and debian machines dont' have it so still not good way of checking | 23:30 |
| *** sree has quit IRC | 23:30 | |
| * clarkb takes a quick break while we wait | 23:31 | |
| jlvillal | clarkb, My Fedora box has it | 23:33 |
| fungi | yeah, no /sys/devices/system/cpu/vulnerabilities on my debian/unstable booted from what seems to be a 5-day-old build of linux 4.14.13 | 23:36 |
| openstackgerrit | Merged openstack-infra/system-config master: Use zuul-scheduler group instead of zuul* glob https://review.openstack.org/535986 | 23:37 |
| fungi | that ^ was the other one we were waiting for | 23:37 |
| fungi | i'll give #release a quick heads up | 23:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add website hosting infrastructure https://review.openstack.org/527548 | 23:38 |
| fungi | status notice Zuul will be offline over the next 20 minutes to perform maintenance; active changes will be reenqueued once work completes, but new patch sets or approvals during that timeframe may need to be rechecked or reapplied as appropriate | 23:39 |
| fungi | does that look okay? | 23:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-mailman master: Create per-site template directories https://review.openstack.org/535848 | 23:39 |
| fungi | pretty sure i can tackle everything listed at https://etherpad.openstack.org/p/zuul01-scheduler-migration but if anybody feels compelled to take up part of it just stick your name on that line | 23:40 |
| corvus | fungi: yep | 23:40 |
| *** yamamoto has joined #openstack-infra | 23:40 | |
| fungi | any last-minute objections to starting to shut it all down? | 23:40 |
| corvus | fungi: consider me on standby and available for task assignment | 23:41 |
| corvus | fungi: ready | 23:41 |
| fungi | #status notice Zuul will be offline over the next 20 minutes to perform maintenance; active changes will be reenqueued once work completes, but new patch sets or approvals during that timeframe may need to be rechecked or reapplied as appropriate | 23:41 |
| openstackstatus | fungi: sending notice | 23:41 |
| clarkb | I'm almost back but dont wait for me | 23:41 |
| fungi | thanks corvus, i'll let you know if something looks broken | 23:41 |
| *** vsaienk0 has quit IRC | 23:42 | |
| -openstackstatus- NOTICE: Zuul will be offline over the next 20 minutes to perform maintenance; active changes will be reenqueued once work completes, but new patch sets or approvals during that timeframe may need to be rechecked or reapplied as appropriate | 23:42 | |
| *** jamesmcarthur has joined #openstack-infra | 23:42 | |
| openstackstatus | fungi: finished sending notice | 23:44 |
| fungi | shutting down everthing now | 23:44 |
| fungi | queues dumped | 23:44 |
| *** corvus is now known as mr_book | 23:45 | |
| mr_book | shut it down | 23:45 |
| *** mr_book is now known as corvus | 23:45 | |
| fungi | wow | 23:45 |
| fungi | you have such a great memory | 23:46 |
| fungi | then again, one of the best films of all time, if you ask me | 23:46 |
| corvus | i have a terrible memory, but i've seen it like 50 times :) | 23:47 |
| fungi | fair. that's almost half the requisite count for a film of its caliber | 23:47 |
| fungi | okay, zuul-scheduler and zuul-fingergw seem to not be stopping (yet) on zuulv3 | 23:47 |
| corvus | i promise i'll watch it again | 23:47 |
| corvus | i'm looking | 23:48 |
| corvus | i agree they don't seem to have stopped... | 23:49 |
| fungi | stop issued for all executor and merger daemons on their respective servers now | 23:50 |
| corvus | i've run "zuul-scheduler stop" manually to no effect | 23:50 |
| fungi | oh, though the mergers are complaining "Warning: zuul-merger.service changed on disk. Run 'systemctl daemon-reload' to reload units." | 23:50 |
| clarkb | is it possible existing finger connections are keeping the server up? | 23:50 |
| corvus | i think the new "use the new command socket" thing is broken | 23:50 |
| clarkb | ah | 23:50 |
| fungi | :/ | 23:51 |
| clarkb | fungi: that may be for the pidfile fixes? | 23:51 |
| fungi | clarkb: yeah, i spammed it to them and then reran the stop on those | 23:51 |
| corvus | a manual "zuul-fingergw stop" worked | 23:52 |
| fungi | tailing the executor debug log on ze01 it's actively sending sigterm to ssh agent subprocesses | 23:53 |
| *** claudiub has quit IRC | 23:53 | |
| fungi | debug log on zm01 indicates the merger stopped a few minutes ago | 23:54 |
| corvus | 2018-01-19 23:45:06,209 DEBUG zuul.CommandSocket: Received b'stop' from socket | 23:54 |
| corvus | that should be from the init script... | 23:54 |
| corvus | i just killed the scheduler processes | 23:55 |
| fungi | yeah, looks squeaky-clean on zuulv3.o.o now | 23:55 |
| corvus | but based on what i've seen, we should look into the schedulor receiving a stop command but not actually terminating, and the fingergw not receiving a stop command. | 23:55 |
| fungi | should be all set for me to run kick.sh for the executors and mergers | 23:55 |
| fungi | running now | 23:56 |
| fungi | oh, wait, i should make sure system-config is up to date | 23:56 |
| fungi | yeah, it is, good | 23:56 |
| clarkb | ze01 is still running stuff | 23:57 |
| clarkb | but I guess we don't ahve to wait for stuff to stop before puppeting | 23:57 |
| fungi | updated keys and times on zuul01, and retrieved the queue dumps | 23:58 |
| fungi | clarkb: yeah, note the comment to that effect at 5.1 in the pad | 23:58 |
| fungi | we'll check it toward the end and see if the executors are still trying to finish stopping | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!