| fungi | and now it's tomorrow | 00:00 |
|---|---|---|
| clarkb | now that you point that out I want to upgrade my router /me does this o/ | 00:00 |
| fungi | openbsd has seriously upped their game on upgrades and patching | 00:00 |
| fungi | https://man.openbsd.org/syspatch | 00:01 |
| clarkb | this is pfsense so freebsd | 00:01 |
| clarkb | looks like I get newer unbound | 00:01 |
| fungi | binary patching the kernel... gives me vax/vms shivers (in a good way!) | 00:01 |
| corvus | i rechecked 670402 (a zuul-jobs change) and it has a buildset that looks sane | 00:02 |
| fungi | success! | 00:02 |
| clarkb | fungi: does it do that for live kernel updates? | 00:02 |
| fungi | it's the on-disk kernel so you still need to reboot | 00:02 |
| fungi | maybe someday we'll have usable systems built on the hurd | 00:02 |
| corvus | oof, it looks like 670413 is hitting a "our linter isn't smart enough" error | 00:03 |
| corvus | i will convert that into a more boring form of yaml for now | 00:03 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Add required projects to zuul tenant https://review.opendev.org/670413 | 00:04 |
| *** dychen has joined #openstack-infra | 00:14 | |
| *** dchen has quit IRC | 00:17 | |
| *** calbers has quit IRC | 00:17 | |
| *** calbers has joined #openstack-infra | 00:17 | |
| openstackgerrit | Merged zuul/zuul master: Switch to opendev release/docs jobs https://review.opendev.org/670388 | 00:22 |
| *** weifan has joined #openstack-infra | 00:25 | |
| *** weifan has quit IRC | 00:25 | |
| *** betherly has joined #openstack-infra | 00:26 | |
| *** dychen has quit IRC | 00:26 | |
| *** dchen has joined #openstack-infra | 00:28 | |
| openstackgerrit | Merged openstack/project-config master: Add required projects to zuul tenant https://review.opendev.org/670413 | 00:28 |
| *** gyee has quit IRC | 00:28 | |
| *** betherly has quit IRC | 00:30 | |
| *** dmsimard4 is now known as dmsimard | 00:32 | |
| *** rcernin has quit IRC | 00:34 | |
| *** rcernin has joined #openstack-infra | 00:35 | |
| *** betherly has joined #openstack-infra | 00:46 | |
| *** dychen has joined #openstack-infra | 00:47 | |
| *** panda has quit IRC | 00:48 | |
| *** dklyle has joined #openstack-infra | 00:49 | |
| *** panda has joined #openstack-infra | 00:50 | |
| *** dchen has quit IRC | 00:50 | |
| *** betherly has quit IRC | 00:51 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Switch to zuul tenant jobs for docs/release https://review.opendev.org/670422 | 00:53 |
| corvus | that should take care of the last remaining alarm bell | 00:53 |
| *** irclogbot_2 has joined #openstack-infra | 00:55 | |
| *** irclogbot_2 has quit IRC | 01:00 | |
| *** betherly has joined #openstack-infra | 01:06 | |
| *** diablo_rojo has quit IRC | 01:07 | |
| *** betherly has quit IRC | 01:11 | |
| *** imacdonn has quit IRC | 01:14 | |
| *** imacdonn has joined #openstack-infra | 01:15 | |
| *** tdasilva has quit IRC | 01:15 | |
| *** lseki has quit IRC | 01:23 | |
| *** irclogbot_0 has joined #openstack-infra | 01:25 | |
| *** betherly has joined #openstack-infra | 01:27 | |
| openstackgerrit | Merged zuul/nodepool master: Switch to zuul tenant jobs for docs/release https://review.opendev.org/670422 | 01:32 |
| *** betherly has quit IRC | 01:32 | |
| *** irclogbot_0 has quit IRC | 01:34 | |
| *** igordc has quit IRC | 01:35 | |
| corvus | no alarm bells \o/ | 01:37 |
| fungi | disentanglemnent concluded | 01:38 |
| *** betherly has joined #openstack-infra | 01:48 | |
| *** betherly has quit IRC | 01:53 | |
| openstackgerrit | Filippo Inzaghi proposed openstack/os-loganalyze master: Change openstack-dev to openstack-discuss https://review.opendev.org/622363 | 01:56 |
| openstackgerrit | Filippo Inzaghi proposed opendev/python-storyboardclient master: fix tox python3 overrides https://review.opendev.org/574347 | 01:57 |
| *** apetrich has quit IRC | 01:58 | |
| *** ijw has quit IRC | 01:58 | |
| openstackgerrit | Filippo Inzaghi proposed opendev/bindep master: Change openstack-dev to openstack-discuss https://review.opendev.org/622325 | 02:01 |
| *** lei-zh has joined #openstack-infra | 02:19 | |
| *** yamamoto has joined #openstack-infra | 02:22 | |
| *** irclogbot_1 has joined #openstack-infra | 02:25 | |
| *** lei-zh has quit IRC | 02:26 | |
| *** irclogbot_1 has quit IRC | 02:30 | |
| *** betherly has joined #openstack-infra | 02:39 | |
| *** betherly has quit IRC | 02:44 | |
| *** altlogbot_2 has joined #openstack-infra | 02:47 | |
| *** yamamoto has quit IRC | 02:48 | |
| *** altlogbot_2 has quit IRC | 02:52 | |
| *** betherly has joined #openstack-infra | 03:00 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:00 | |
| *** yamamoto has joined #openstack-infra | 03:01 | |
| *** bhavikdbavishi has quit IRC | 03:02 | |
| *** betherly has quit IRC | 03:05 | |
| *** michael-beaver has quit IRC | 03:08 | |
| *** diablo_rojo has joined #openstack-infra | 03:09 | |
| *** dychen has quit IRC | 03:09 | |
| *** dchen has joined #openstack-infra | 03:10 | |
| *** rlandy has quit IRC | 03:16 | |
| *** betherly has joined #openstack-infra | 03:20 | |
| *** irclogbot_3 has joined #openstack-infra | 03:21 | |
| *** betherly has quit IRC | 03:25 | |
| *** irclogbot_3 has quit IRC | 03:26 | |
| *** betherly has joined #openstack-infra | 03:42 | |
| *** psachin has joined #openstack-infra | 03:42 | |
| *** psachin has quit IRC | 03:43 | |
| *** psachin has joined #openstack-infra | 03:44 | |
| *** betherly has quit IRC | 03:46 | |
| *** irclogbot_3 has joined #openstack-infra | 03:51 | |
| *** whoami-rajat has joined #openstack-infra | 03:55 | |
| *** irclogbot_3 has quit IRC | 03:56 | |
| *** betherly has joined #openstack-infra | 04:02 | |
| *** ykarel|away has joined #openstack-infra | 04:02 | |
| *** udesale has joined #openstack-infra | 04:05 | |
| *** dklyle has quit IRC | 04:06 | |
| *** betherly has quit IRC | 04:07 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Normalize test jobs yaml https://review.opendev.org/670198 | 04:31 |
| *** betherly has joined #openstack-infra | 04:33 | |
| *** factor has quit IRC | 04:35 | |
| *** betherly has quit IRC | 04:38 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add add-authorized-keys test job https://review.opendev.org/670199 | 04:45 |
| *** toabctl has quit IRC | 04:53 | |
| *** rcernin has quit IRC | 04:54 | |
| *** toabctl has joined #openstack-infra | 04:55 | |
| *** irclogbot_3 has joined #openstack-infra | 04:59 | |
| *** ykarel|away has quit IRC | 04:59 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:01 | |
| *** yamamoto has quit IRC | 05:11 | |
| *** kjackal has joined #openstack-infra | 05:12 | |
| *** pcaruana has joined #openstack-infra | 05:13 | |
| *** betherly has joined #openstack-infra | 05:14 | |
| *** jistr has quit IRC | 05:15 | |
| *** JpMaxMan has quit IRC | 05:16 | |
| *** irclogbot_3 has quit IRC | 05:16 | |
| *** JpMaxMan has joined #openstack-infra | 05:17 | |
| *** yamamoto has joined #openstack-infra | 05:18 | |
| *** jistr has joined #openstack-infra | 05:18 | |
| *** betherly has quit IRC | 05:19 | |
| *** ykarel|away has joined #openstack-infra | 05:24 | |
| *** ykarel|away is now known as ykarel | 05:25 | |
| *** aedc has quit IRC | 05:47 | |
| *** kjackal has quit IRC | 05:54 | |
| *** jbadiapa has quit IRC | 05:55 | |
| *** yamamoto has quit IRC | 06:02 | |
| *** betherly has joined #openstack-infra | 06:06 | |
| *** yamamoto has joined #openstack-infra | 06:07 | |
| *** kjackal has joined #openstack-infra | 06:09 | |
| *** altlogbot_0 has joined #openstack-infra | 06:09 | |
| *** betherly has quit IRC | 06:11 | |
| *** altlogbot_0 has quit IRC | 06:14 | |
| *** jtomasek has joined #openstack-infra | 06:25 | |
| *** Goneri has joined #openstack-infra | 06:25 | |
| *** rkukura_ has joined #openstack-infra | 06:29 | |
| *** rkukura has quit IRC | 06:30 | |
| *** rkukura_ is now known as rkukura | 06:30 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Advance ansible-lint cap to test with 4 https://review.opendev.org/667695 | 06:31 |
| *** witek has joined #openstack-infra | 06:34 | |
| *** irclogbot_0 has joined #openstack-infra | 06:35 | |
| *** betherly has joined #openstack-infra | 06:37 | |
| *** yamamoto has quit IRC | 06:39 | |
| *** rkukura has quit IRC | 06:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove unused docs jobs from dashboard https://review.opendev.org/670452 | 06:39 |
| *** irclogbot_0 has quit IRC | 06:40 | |
| *** betherly has quit IRC | 06:42 | |
| *** yamamoto has joined #openstack-infra | 06:43 | |
| *** rkukura has joined #openstack-infra | 06:44 | |
| *** yamamoto has quit IRC | 06:47 | |
| *** aedc has joined #openstack-infra | 06:58 | |
| *** ginopc has joined #openstack-infra | 07:02 | |
| *** gtema has joined #openstack-infra | 07:07 | |
| *** Goneri has quit IRC | 07:17 | |
| *** Goneri has joined #openstack-infra | 07:18 | |
| *** rpittau|afk is now known as rpittau | 07:18 | |
| *** tosky has joined #openstack-infra | 07:19 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 07:21 |
| *** jbadiapa has joined #openstack-infra | 07:22 | |
| *** pgaxatte has joined #openstack-infra | 07:23 | |
| *** dchen has quit IRC | 07:23 | |
| *** jbadiapa has quit IRC | 07:27 | |
| *** iurygregory has joined #openstack-infra | 07:30 | |
| *** xek has joined #openstack-infra | 07:36 | |
| *** slaweq has joined #openstack-infra | 07:41 | |
| *** lucasagomes has joined #openstack-infra | 07:43 | |
| *** aedc has quit IRC | 07:48 | |
| *** ralonsoh has joined #openstack-infra | 07:50 | |
| *** aedc has joined #openstack-infra | 07:54 | |
| *** slittle1 has quit IRC | 07:54 | |
| *** yamamoto has joined #openstack-infra | 07:57 | |
| *** ccamacho has joined #openstack-infra | 07:59 | |
| *** ykarel is now known as ykarel|lunch | 07:59 | |
| *** altlogbot_2 has joined #openstack-infra | 08:01 | |
| *** altlogbot_2 has quit IRC | 08:04 | |
| *** yolanda has quit IRC | 08:08 | |
| *** slittle1 has joined #openstack-infra | 08:08 | |
| *** yolanda has joined #openstack-infra | 08:09 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Overriding max. starting builds. https://review.opendev.org/670461 | 08:09 |
| *** altlogbot_3 has joined #openstack-infra | 08:11 | |
| *** slittle1 has quit IRC | 08:12 | |
| *** altlogbot_3 has quit IRC | 08:16 | |
| *** altlogbot_0 has joined #openstack-infra | 08:17 | |
| *** tkajinam has quit IRC | 08:19 | |
| *** altlogbot_0 has quit IRC | 08:22 | |
| *** altlogbot_2 has joined #openstack-infra | 08:23 | |
| *** pkopec has joined #openstack-infra | 08:27 | |
| *** altlogbot_2 has quit IRC | 08:29 | |
| *** Fidde has joined #openstack-infra | 08:31 | |
| *** derekh has joined #openstack-infra | 08:32 | |
| *** rascasoft has quit IRC | 08:33 | |
| *** rascasoft has joined #openstack-infra | 08:34 | |
| *** Lucas_Gray has joined #openstack-infra | 08:40 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 08:42 |
| openstackgerrit | Merged opendev/irc-meetings master: Update networking OVN meeting https://review.opendev.org/670372 | 08:44 |
| *** dosaboy has quit IRC | 08:47 | |
| *** aedc has quit IRC | 08:48 | |
| *** aluria has quit IRC | 08:48 | |
| *** ykarel|lunch is now known as ykarel | 08:48 | |
| *** altlogbot_3 has joined #openstack-infra | 08:53 | |
| *** rascasoft has quit IRC | 08:55 | |
| *** altlogbot_3 has quit IRC | 08:58 | |
| *** rascasoft has joined #openstack-infra | 08:58 | |
| *** irclogbot_3 has joined #openstack-infra | 08:59 | |
| *** dosaboy has joined #openstack-infra | 09:00 | |
| *** betherly has joined #openstack-infra | 09:00 | |
| *** dosaboy has quit IRC | 09:03 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 09:03 |
| *** aluria has joined #openstack-infra | 09:03 | |
| *** dosaboy has joined #openstack-infra | 09:04 | |
| *** irclogbot_3 has quit IRC | 09:04 | |
| *** betherly has quit IRC | 09:05 | |
| *** Lucas_Gray has quit IRC | 09:10 | |
| *** Lucas_Gray has joined #openstack-infra | 09:12 | |
| *** Goneri has quit IRC | 09:14 | |
| *** kjackal has quit IRC | 09:18 | |
| *** factor has joined #openstack-infra | 09:22 | |
| *** diablo_rojo has quit IRC | 09:25 | |
| *** Goneri has joined #openstack-infra | 09:27 | |
| *** gtema has quit IRC | 09:34 | |
| *** gtema has joined #openstack-infra | 09:34 | |
| *** Goneri has quit IRC | 09:39 | |
| *** gtema_ has joined #openstack-infra | 09:41 | |
| *** gtema has quit IRC | 09:43 | |
| *** yamamoto has quit IRC | 09:49 | |
| *** yamamoto has joined #openstack-infra | 09:51 | |
| *** yamamoto has quit IRC | 09:51 | |
| *** yamamoto has joined #openstack-infra | 09:52 | |
| *** yamamoto has quit IRC | 09:57 | |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Add shared 'oslo', 'oslo-independent' ACL files https://review.opendev.org/670270 | 09:58 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Update ACLs for moved doc projects https://review.opendev.org/670269 | 09:58 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Update gerritbot channels for moved doc projects https://review.opendev.org/670483 | 09:58 |
| *** gtema_ has quit IRC | 10:02 | |
| *** gtema has joined #openstack-infra | 10:02 | |
| *** gtema has quit IRC | 10:03 | |
| *** gtema has joined #openstack-infra | 10:03 | |
| *** pfallenop has joined #openstack-infra | 10:09 | |
| *** kjackal has joined #openstack-infra | 10:15 | |
| *** pfallenop has quit IRC | 10:18 | |
| *** ociuhandu has joined #openstack-infra | 10:22 | |
| *** gtema has quit IRC | 10:30 | |
| *** gtema has joined #openstack-infra | 10:30 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Overriding max. starting builds. https://review.opendev.org/670461 | 10:31 |
| *** irclogbot_3 has joined #openstack-infra | 10:31 | |
| *** yolanda has quit IRC | 10:31 | |
| *** yolanda has joined #openstack-infra | 10:32 | |
| *** Lucas_Gray has quit IRC | 10:35 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate canMerge check with event id https://review.opendev.org/670494 | 10:35 |
| *** irclogbot_3 has quit IRC | 10:38 | |
| *** irclogbot_2 has joined #openstack-infra | 10:41 | |
| *** irclogbot_2 has quit IRC | 10:44 | |
| *** dosaboy has quit IRC | 10:45 | |
| *** kjackal has quit IRC | 10:45 | |
| *** gtema has quit IRC | 10:47 | |
| *** gtema has joined #openstack-infra | 10:47 | |
| *** Goneri has joined #openstack-infra | 10:53 | |
| *** aluria has quit IRC | 10:56 | |
| *** yamamoto has joined #openstack-infra | 11:00 | |
| *** dosaboy has joined #openstack-infra | 11:02 | |
| *** altlogbot_2 has joined #openstack-infra | 11:03 | |
| *** yamamoto has quit IRC | 11:06 | |
| *** altlogbot_2 has quit IRC | 11:08 | |
| *** snierodz has quit IRC | 11:08 | |
| *** stephenfin has quit IRC | 11:08 | |
| *** tesseract has joined #openstack-infra | 11:08 | |
| *** stephenfin has joined #openstack-infra | 11:10 | |
| *** aluria has joined #openstack-infra | 11:11 | |
| *** altlogbot_0 has joined #openstack-infra | 11:12 | |
| *** altlogbot_0 has quit IRC | 11:16 | |
| *** kjackal has joined #openstack-infra | 11:20 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 11:26 |
| *** dosaboy has quit IRC | 11:27 | |
| *** dosaboy has joined #openstack-infra | 11:28 | |
| *** dosaboy has quit IRC | 11:28 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Overriding max. starting builds. https://review.opendev.org/670461 | 11:29 |
| *** yamamoto has joined #openstack-infra | 11:35 | |
| *** yamamoto has quit IRC | 11:35 | |
| *** yamamoto has joined #openstack-infra | 11:36 | |
| *** apetrich has joined #openstack-infra | 11:38 | |
| *** gtema has quit IRC | 11:46 | |
| *** gtema has joined #openstack-infra | 11:46 | |
| *** aedc has joined #openstack-infra | 11:54 | |
| *** eharney has joined #openstack-infra | 11:54 | |
| *** udesale has quit IRC | 11:59 | |
| *** udesale has joined #openstack-infra | 12:00 | |
| *** altlogbot_0 has joined #openstack-infra | 12:07 | |
| *** altlogbot_0 has quit IRC | 12:08 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Use a requests session to simplify auth'd calls https://review.opendev.org/670511 | 12:16 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Use urllib.parse for manipulating client urls https://review.opendev.org/670512 | 12:16 |
| *** Goneri has quit IRC | 12:21 | |
| *** goldyfruit has quit IRC | 12:22 | |
| *** electrofelix has joined #openstack-infra | 12:23 | |
| *** derekh has quit IRC | 12:26 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 12:27 |
| *** aedc has quit IRC | 12:27 | |
| *** Goneri has joined #openstack-infra | 12:29 | |
| *** ekultails has joined #openstack-infra | 12:30 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Use a requests session to simplify auth'd calls https://review.opendev.org/670511 | 12:33 |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Don't pause static pool on single label quota https://review.opendev.org/667371 | 12:37 |
| *** rlandy has joined #openstack-infra | 12:37 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 12:41 |
| mnaser | infra-root: i have cleaned up all stale volumes and also deletd all ERROR state instances in sjc1 | 12:41 |
| mnaser | i... hope nodepool doesnt get angry the ERROR vms disappeared beneath it | 12:41 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 12:42 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 12:45 |
| *** markvoelker has quit IRC | 12:45 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 12:45 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 12:45 |
| *** viks___ has quit IRC | 12:46 | |
| Shrews | mnaser: nodepool is pretty resilient against such things. it anticipates pretty much anything disappearing on it | 12:51 |
| *** aaronsheffield has joined #openstack-infra | 12:52 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 12:54 |
| *** mriedem has joined #openstack-infra | 12:56 | |
| mnaser | Shrews: cool, thanks! | 12:57 |
| *** tjgresha_nope has joined #openstack-infra | 12:59 | |
| *** ricolin has quit IRC | 13:00 | |
| *** aedc has joined #openstack-infra | 13:00 | |
| *** tjgresha has quit IRC | 13:02 | |
| *** derekh has joined #openstack-infra | 13:06 | |
| *** rfarr_ has quit IRC | 13:06 | |
| *** rfarr has joined #openstack-infra | 13:06 | |
| *** lseki has joined #openstack-infra | 13:10 | |
| *** rfarr_ has joined #openstack-infra | 13:10 | |
| *** aedc has quit IRC | 13:12 | |
| *** rfarr has quit IRC | 13:13 | |
| *** sthussey has joined #openstack-infra | 13:14 | |
| *** rfarr has joined #openstack-infra | 13:19 | |
| *** rfarr_ has quit IRC | 13:21 | |
| *** whoami-rajat has quit IRC | 13:25 | |
| *** whoami-rajat has joined #openstack-infra | 13:25 | |
| fungi | thanks for cleaning that up mnaser! | 13:26 |
| AJaeger | infra-root, I just approved change https://review.opendev.org/#/c/670378/ to bring Fort Nebula CI cloud online. | 13:28 |
| fungi | AJaeger: thanks! i was about to do that too. will keep an eye on it | 13:30 |
| *** sshnaidm|off has quit IRC | 13:30 | |
| openstackgerrit | Merged openstack/project-config master: Bringing Fort Nebula CI Cloud back online https://review.opendev.org/670378 | 13:32 |
| donnyd | yey :) :) | 13:34 |
| *** sshnaidm has joined #openstack-infra | 13:35 | |
| *** irclogbot_0 has joined #openstack-infra | 13:35 | |
| *** sshnaidm is now known as sshnaidm|off | 13:36 | |
| *** irclogbot_0 has quit IRC | 13:38 | |
| *** goldyfruit has joined #openstack-infra | 13:49 | |
| *** ykarel is now known as ykarel|afk | 13:54 | |
| *** ykarel|afk has quit IRC | 13:59 | |
| donnyd | looks like the centos7 image is working great | 14:02 |
| donnyd | AJaeger: can you check for job failures ? | 14:05 |
| *** irclogbot_1 has joined #openstack-infra | 14:09 | |
| donnyd | If all looks good I would like to take it up to 50 and then watch it for a while | 14:10 |
| fungi | we'll probably need a bit of time for some longer-running classes of jobs to finish there and get indexed | 14:11 |
| donnyd | that sounds good to me | 14:11 |
| *** FlorianFa has quit IRC | 14:11 | |
| fungi | and then dig into the causes of failures (because there will assuredly be failures, we just hope they're due to buggy patches being tested) | 14:13 |
| fungi | querying http://logstash.openstack.org/ for node_provider:fortnebula-regionone build-status:failure turns up log lines from some | 14:13 |
| *** altlogbot_2 has joined #openstack-infra | 14:13 | |
| *** chandankumar is now known as raukadah | 14:14 | |
| fungi | http://logs.openstack.org/13/670513/2/check/nodejs10-npm-run-test/5b8313c/job-output.txt | 14:15 |
| fungi | there's one | 14:15 |
| fungi | looks like that was a chromium-based browser testset for horizon on ubuntu-bionic | 14:17 |
| donnyd | The error was some keystone issue | 14:18 |
| donnyd | from what I can see | 14:18 |
| fungi | yeah, almost certainly not a provider-level problem | 14:18 |
| donnyd | no really infra related, but I do agree we should give it some time and check to see if any of the infra bits are busted | 14:18 |
| donnyd | I am assuming that storage speeds/IOPS will be an issue at scale, as the current backend can only do about 75K IOPS. | 14:20 |
| donnyd | At least in the testing I was able to get done | 14:20 |
| fungi | yeah, we usually end up having to tune our quota size relative to the host aggregate in situations where we're basically in a dedicated environment | 14:21 |
| donnyd | So there will be a maintenance window some time in the next few weeks to swap it out with an all nvme based one | 14:21 |
| fungi | we quickly become our own noisy neighbor | 14:21 |
| donnyd | Well this provider does only CI work, no general purpose stuff... so the backends will be tuned to the workload | 14:22 |
| donnyd | usually it goes the other way around | 14:22 |
| fungi | http://logs.openstack.org/56/670556/1/check/legacy-tempest-dsvm-networking-bgpvpn-bagpipe/528149d/ | 14:22 |
| fungi | there's another failure | 14:22 |
| fungi | i need to pop out to run a quick errand, but will brb | 14:22 |
| *** ykarel|afk has joined #openstack-infra | 14:24 | |
| fungi | oh, that was a success | 14:24 |
| donnyd | Error when trying to get requirement for VCS system Command "git config --get-regexp remote\..*\.url" failed with error code 1 in /opt/stack/new/networking-bagpipe, falling back to uneditable format,Could not determine repository location of /opt/stack/new/networking-bagpipe | 14:24 |
| fungi | that's probably benign | 14:25 |
| donnyd | Well it looks like it failed, but not infra related either | 14:25 |
| fungi | my logstash query above was incorrect | 14:25 |
| donnyd | oh | 14:25 |
| fungi | should be build_status:failure | 14:25 |
| fungi | (_ not -) | 14:25 |
| fungi | node_provider:fortnebula-regionone AND build_status:FAILURE | 14:26 |
| *** lpetrut has joined #openstack-infra | 14:26 | |
| fungi | okay, errand. brb | 14:26 |
| *** jcoufal has joined #openstack-infra | 14:27 | |
| openstackgerrit | Merged openstack/ptgbot master: Display count of attendees in each room on web page https://review.opendev.org/658501 | 14:31 |
| openstackgerrit | Merged openstack/ptgbot master: Use a badge to show check-ins in "now" display https://review.opendev.org/658795 | 14:31 |
| *** liuyulong has joined #openstack-infra | 14:33 | |
| *** dpawlik has quit IRC | 14:34 | |
| *** bnemec is now known as beekneemech | 14:34 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Add Kubernetes Operator Functional Test Job https://review.opendev.org/668029 | 14:38 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: [WIP] Verify Operator Pod Running https://review.opendev.org/670395 | 14:38 |
| AJaeger | config-core, please review these two small cleanups https://review.opendev.org/670452 and https://review.opendev.org/670344 | 14:40 |
| *** rfarr has quit IRC | 14:44 | |
| *** Goneri has quit IRC | 14:52 | |
| *** TheJulia is now known as needssleep | 14:52 | |
| *** rpittau is now known as elfosardo | 14:52 | |
| *** markvoelker has joined #openstack-infra | 14:53 | |
| *** markvoelker has quit IRC | 14:56 | |
| *** Goneri has joined #openstack-infra | 14:57 | |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Reset to OrderedDict on new day cleanup https://review.opendev.org/670577 | 14:57 |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Clean up stale data presence on a #newday command https://review.opendev.org/670578 | 14:57 |
| *** ociuhandu_ has joined #openstack-infra | 14:59 | |
| *** diablo_rojo has joined #openstack-infra | 15:00 | |
| *** ociuhandu has quit IRC | 15:02 | |
| *** ociuhandu_ has quit IRC | 15:03 | |
| *** gtema has quit IRC | 15:08 | |
| *** Goneri has quit IRC | 15:15 | |
| *** ykarel|afk is now known as ykarel|away | 15:16 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Remove Operator SDK dependency in Zuul Job https://review.opendev.org/670584 | 15:17 |
| clarkb | as a heads up I have followed up with kevinz about new linaro arm64 cloud region for CI resources | 15:18 |
| clarkb | its early yet so talking requirements and the like. Hopefully that gets us more arm test nodes :) | 15:19 |
| clarkb | nb03 needs the same cleanup as nb01 and nb02 so doing that now | 15:19 |
| *** piotrowskim has quit IRC | 15:21 | |
| *** Fidde has quit IRC | 15:26 | |
| *** iurygregory is now known as skolt | 15:27 | |
| fungi | donnyd: so far only 2 failed builds... 5b8313c872d344b6a423c11627a03f56 which we already looked at and 0f66c903686b4ec7b7ca32773e9a02af which is logged here: http://logs.openstack.org/56/670556/1/check/legacy-networking-bagpipe-dsvm-fullstack/0f66c90/ | 15:27 |
| clarkb | fungi: that second one failed due to compile of ovs not working | 15:28 |
| clarkb | unlikely a provider issue | 15:28 |
| fungi | i concur | 15:28 |
| fungi | /opt/stack/new/ovs/datapath/linux/nf_conntrack_reasm.c:79:31: error: ‘struct inet_frags’ has no member named ‘rnd’ | 15:28 |
| fungi | net_get_random_once(&nf_frags.rnd, sizeof(nf_frags.rnd)); | 15:29 |
| fungi | so far so good | 15:29 |
| *** Lucas_Gray has joined #openstack-infra | 15:30 | |
| fungi | jobs began running there about 13:45 so we're coming up on 2 hours with only 2 build failures neither of which look like they could be attributable to that provider | 15:30 |
| fungi | probably safe to crank it up a notch whenever | 15:31 |
| openstackgerrit | Merged openstack/project-config master: Remove unused docs jobs from dashboard https://review.opendev.org/670452 | 15:31 |
| openstackgerrit | Merged openstack/project-config master: Update description of some docs jobs https://review.opendev.org/670344 | 15:31 |
| clarkb | should probably check that donnyd is happy with things on the cloud side but ya I'd say go for it | 15:32 |
| donnyd | clarkb: I am good. Want me to push up the next step? | 15:32 |
| fungi | right, i was still going to wait for an all-clear from him | 15:32 |
| clarkb | donnyd: ya if you want to propose the next bump I think that would be good. We should remember to bump quotas too | 15:33 |
| clarkb | (if the quotas aren't already bumped) | 15:33 |
| *** jcoufal has quit IRC | 15:34 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Scaling FNCI to 40 instances https://review.opendev.org/670587 | 15:34 |
| clarkb | I'm going to find breakfast and then probably get a bike ride in but +2 on ^ | 15:35 |
| dmellado | AJaeger: ping re: opensuse repos down | 15:35 |
| *** sdoran has joined #openstack-infra | 15:36 | |
| *** tdasilva has joined #openstack-infra | 15:36 | |
| sdoran | Hello everyone. 👋 | 15:36 |
| sdoran | Is anyone seeing metadata downloads hanging for OpenSUSE 15? | 15:36 |
| fungi | sdoran: in zuul jobs? have a link to an example? | 15:37 |
| fungi | do the jobs eventually fail due to timeouts i guess? | 15:37 |
| sdoran | No, it's in Ansible CI, so shippable. | 15:37 |
| sdoran | We cap jobs at 45 minutes, so they are getting killed. | 15:37 |
| *** mattw4 has joined #openstack-infra | 15:37 | |
| sdoran | But if I just run `zypper install udev` in a test container, it hangs. | 15:38 |
| sdoran | This in the URL that it gets stuck on: http://download.opensuse.org/distribution/leap/15.0/repo/non-oss/repodata/773c107fe9e932054ad44f31655f245faefbd3172657429e363acf7917e125f0-primary.xml.gz | 15:38 |
| fungi | ahh, well i don't know what opensuse 15 "metadata downloads" are, but if they're part of the zypper package repositories we cache those locally in our node providers | 15:38 |
| sdoran | Some URLs from the page download ok, but others do not. | 15:38 |
| *** kjackal has quit IRC | 15:39 | |
| fungi | yeah, we try to prevent our ci jobs from accessing externally-served distro packages | 15:39 |
| sdoran | I'm starting to think we need to do the same. :) | 15:39 |
| *** diablo_rojo has quit IRC | 15:39 | |
| fungi | for opensuse we rsync update a mirror in afs every 4 hours and then atomically release that afs volume if the rsync succeeds | 15:39 |
| *** diablo_rojo has joined #openstack-infra | 15:40 | |
| fungi | and stick afs client caches with apache frontends in each of our node providers for zuul/nodepool | 15:40 |
| sdoran | That's nice. | 15:40 |
| fungi | and configure our node test images to look at those for their packages | 15:40 |
| dmellado | hey sdoran | 15:40 |
| dmellado | just pinged AJaeger | 15:41 |
| sdoran | 👍 | 15:41 |
| dmellado | but I'm afraid he might be off as it's a little bit late in EMEA and Friday | 15:43 |
| dmellado | there's another channel we might try | 15:44 |
| dmellado | #opensuse-buildservice | 15:44 |
| fungi | dirk is also around sometimes and knows who to reach out to | 15:45 |
| *** jcoufal has joined #openstack-infra | 15:45 | |
| *** lucasagomes has quit IRC | 15:47 | |
| sdoran | @fungi Thanks for the help and suggestions. | 15:49 |
| AJaeger | dmellado: better talk with cmurphy and dirk about openSUSE repos | 15:51 |
| dirk | dmellado: AJaeger: there is a network problem (download.o.org is down) | 15:52 |
| dirk | I thought the openstack ci is not affected because it uses a mirror? | 15:52 |
| dmellado | AJaeger: dirk thanks! | 15:52 |
| dirk | why is it not using the mirror in this case? | 15:52 |
| AJaeger | thanks, dirk | 15:53 |
| dmellado | yeah, exactly, it seems that ansible doesn't use a mirror but rather queries it directly | 15:53 |
| *** lpetrut has quit IRC | 15:53 | |
| dirk | ah, okay | 15:53 |
| dirk | yeah, well, to be honest the main download site shouldn't be down in the first place | 15:54 |
| mriedem | clarkb: just remembered and posted a patch for this issue i found awhile ago https://review.opendev.org/670591 | 15:56 |
| mriedem | for some odd reason, cirros guest shutdown is not happening within 60 seocnds | 15:56 |
| mriedem | *seconds | 15:56 |
| mriedem | notes and a guest console log are in the related bug, | 15:56 |
| mriedem | but looks like maybe the guest is held up on shutdown waiting for a response from the metadata api | 15:57 |
| *** ykarel|away has quit IRC | 15:57 | |
| mriedem | this means that tempest tests that do server stop, shelve, rescue and rebuild could be taking up to 60 seconds just to stop the guest | 15:57 |
| fungi | dirk: it's not a problem for our environment. sdoran was just asking because the "shippable" ci system for ansible on github was having trouble on opensuse 15 and so was asking if we were encountering similar issues (we're not afaik because we have our own mirrors) | 15:58 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Remove Operator SDK dependency in Zuul Job https://review.opendev.org/670584 | 15:58 |
| dmellado | thanks in any case dirk! | 15:59 |
| sdoran | Yes, thanks for the help. | 16:00 |
| sdoran | Just needed to ask some other folks that I know have a pretty busy CI that is hitting OpenSUSE mirrors. | 16:00 |
| *** adrianreza_ has joined #openstack-infra | 16:01 | |
| fungi | turns out our ci system is busy enough we avoid using official distro mirrors ;) | 16:01 |
| *** gyee has joined #openstack-infra | 16:01 | |
| openstackgerrit | Merged openstack/project-config master: Scaling FNCI to 40 instances https://review.opendev.org/670587 | 16:02 |
| fungi | it's much faster and more stable not having to drag packages halfway across the internet on every build | 16:02 |
| *** ijw has joined #openstack-infra | 16:04 | |
| *** ijw_ has joined #openstack-infra | 16:05 | |
| *** Lucas_Gray has quit IRC | 16:06 | |
| *** pgaxatte has quit IRC | 16:07 | |
| *** ykarel|away has joined #openstack-infra | 16:09 | |
| *** ijw has quit IRC | 16:09 | |
| *** pkopec has quit IRC | 16:10 | |
| *** Lucas_Gray has joined #openstack-infra | 16:12 | |
| donnyd | clarkb: looks like something is still wrong in fedora28 | 16:22 |
| donnyd | can't reach the instance | 16:22 |
| fungi | the increased number of nodes started to be used for jobs as of ~16:15z | 16:23 |
| *** mriedem has quit IRC | 16:25 | |
| *** aluria has quit IRC | 16:26 | |
| *** kjackal has joined #openstack-infra | 16:26 | |
| donnyd | fungi: yea, def something wrong with fedora28 | 16:27 |
| fungi | got it. for those we're probably just logging boot failures if they come up unreachable | 16:27 |
| *** elfosardo is now known as rpittau|afk | 16:27 | |
| fungi | so they're not impacting any jobs, but they are wasting resources out of the quota | 16:28 |
| donnyd | well I can easily fix that | 16:30 |
| fungi | not to mention the i/o bandwidth consumed by the boot/delete churn | 16:31 |
| donnyd | there is almost none on my end... | 16:31 |
| fungi | oh, nice | 16:31 |
| donnyd | If you look at time to ready its pretty clear, all the images are cached in ram | 16:32 |
| fungi | that helps | 16:32 |
| donnyd | I need to move my image backend to something much faster though... because you can clearly see when they aren't | 16:33 |
| donnyd | just gonna wait for the nvme storage to get here, and I should be able to put all of it on that | 16:33 |
| donnyd | trying to get my in-use to a solid 40 and then i need to watch the heat | 16:34 |
| donnyd | its my only crutch in having a no A/C based system | 16:35 |
| donnyd | So something interesting I didn't really expect, when the jobs are running I was thinking power would go up exponentially | 16:36 |
| donnyd | for right now... it doesn't seem to have budged at all | 16:36 |
| donnyd | but maybe its because the jobs are just getting spun up and aren't really doing anything yet | 16:37 |
| fungi | for some years i had a full 7' rack full of inefficient/power-hungry antiques in my home lab, and so ducted a freestanding auxiliary air conditioner through them and exhausted it out a spacer in the window | 16:39 |
| fungi | also aluminum-foiled the windows to cut down on heat coming in from the sun | 16:40 |
| fungi | not like a proper crac, but it did the job | 16:40 |
| *** ginopc has quit IRC | 16:42 | |
| fungi | looking at http://zuul.opendev.org/t/openstack/nodes there are some in-use for >15 minutes already | 16:45 |
| fungi | i mean in addition to the handful which are attributable to the earlier, lower quota | 16:46 |
| *** derekh has quit IRC | 16:48 | |
| *** kjackal has quit IRC | 16:49 | |
| *** ijw_ has quit IRC | 16:51 | |
| *** rkukura_ has joined #openstack-infra | 16:52 | |
| *** gtema has joined #openstack-infra | 16:52 | |
| *** rkukura has quit IRC | 16:54 | |
| *** rkukura_ is now known as rkukura | 16:54 | |
| Shrews | fungi: very MacGyver of you | 16:58 |
| clarkb | donnyd: fungi oh you know what I wonder if that image got rebuilt | 16:58 |
| clarkb | donnyd: its possible that it didn't? | 16:58 |
| *** udesale has quit IRC | 16:59 | |
| *** betherly has joined #openstack-infra | 17:00 | |
| *** psachin has quit IRC | 17:03 | |
| *** ijw has joined #openstack-infra | 17:03 | |
| *** jtomasek has quit IRC | 17:04 | |
| *** betherly has quit IRC | 17:05 | |
| *** adriancz has quit IRC | 17:07 | |
| donnyd | so the difference between no workload and 40% is 300 watts | 17:07 |
| clarkb | ya the fedora-28 image is old | 17:09 |
| clarkb | from the 8th | 17:09 |
| * clarkb loosk into why that one isn't building | 17:09 | |
| *** jcoufal has quit IRC | 17:09 | |
| corvus | 7.5 watts per instance | 17:09 |
| clarkb | http://paste.openstack.org/show/754350/ is why fedora-28 isn't updating | 17:10 |
| clarkb | fedora-28 is EOL iirc so odd that a package/service would disappear? | 17:11 |
| corvus | if that holds across providers (probably not, but maybe close), the whole cluster is using 6kW. | 17:11 |
| clarkb | pabelanger: http://paste.openstack.org/show/754350/ any idea why that might be happening? | 17:12 |
| clarkb | pabelanger: for more context that is failed dib build of fedora-28 image | 17:12 |
| mordred | infra-root: The fine folks at the MOC are going to start giving us some capacity. (/me waves at knikolla) I've submitted some forms on the RH side to get accounts spun up, and will be following up with the appropriate config patches once we've got accounts and whatnot | 17:13 |
| fungi | at that rate we're gonna need a lot more nodes to reach 1.21 gigawatts | 17:14 |
| corvus | mordred, knikolla: \o/ neat! | 17:14 |
| fungi | ooh, that's awesome! | 17:14 |
| *** witek has quit IRC | 17:14 | |
| clarkb | cool | 17:14 |
| *** gtema has quit IRC | 17:15 | |
| pabelanger | clarkb: looks like fall out of getting fedora-29 to build properly | 17:15 |
| clarkb | donnyd: not sure if you've seen http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1 but that tries to track things for you from our side too | 17:16 |
| clarkb | pabelanger: well we had a successful build on the 8th though | 17:16 |
| pabelanger | there was a change to DIB to fix fedora-29, but maybe it broke fedora-28? | 17:16 |
| clarkb | I guess I should look at those logs from the 8th and see if it ran that same code | 17:16 |
| pabelanger | yah | 17:16 |
| clarkb | pabelanger: recently? | 17:16 |
| pabelanger | maybe 2 months ago | 17:16 |
| pabelanger | let me find change | 17:16 |
| AJaeger | knikolla: Great, thanks! | 17:16 |
| clarkb | pabelanger: thanks | 17:16 |
| pabelanger | clarkb: https://review.opendev.org/657126/ might be related | 17:17 |
| clarkb | hrm all of our build files are newer than the 8th, we rotate them more aggressively than I thought | 17:17 |
| clarkb | s/files/logs/ | 17:17 |
| AJaeger | mordred: what is MOC? | 17:17 |
| clarkb | pabelanger: oh I bet that coincides with a release of dib | 17:17 |
| clarkb | pabelanger: since we consume it from pypi not from source. Thanks for the pointers I should be able to track it down now probably | 17:18 |
| pabelanger | clarkb: there is a comment about fedora-28 too breaking | 17:18 |
| pabelanger | so, guess our testing didn't work well | 17:18 |
| clarkb | ok I think I get what happened | 17:19 |
| *** ralonsoh has quit IRC | 17:19 | |
| clarkb | we probably didn't care about 28 because its eol | 17:19 |
| clarkb | but tripleo is still using it so whoops | 17:19 |
| clarkb | I'll get a fix up | 17:19 |
| corvus | AJaeger: https://www.bu.edu/hic/research/highlighted-sponsored-projects/massachusetts-open-cloud/ | 17:21 |
| AJaeger | corvus: thanks - ok, remember hearing about them but MOC didn't ring a bell | 17:21 |
| corvus | maybe https://massopen.cloud/about/ is more relevant now | 17:22 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder master: Only enable dbus-daemon on fedora-29 https://review.opendev.org/670606 | 17:23 |
| clarkb | pabelanger: donnyd fungi ^ I think that will fix fedora-28 builds but we'll have to make a dib release too | 17:23 |
| donnyd | It would seem everything else is humming along quite well. Although I haven't seen a gentoo based build come in yet | 17:24 |
| donnyd | heat seems to be under control, and power usage is well within expected ranges | 17:25 |
| clarkb | cool | 17:26 |
| *** Lucas_Gray has quit IRC | 17:26 | |
| donnyd | the whole thing is using about 20 amps / 230 volts or about 4600 watts. I have a few extra pieces of equipment that need to be pulled | 17:26 |
| donnyd | maybe scale up a bit more? | 17:26 |
| clarkb | donnyd: I think we are happy to scale up as high as you are willing :) also it should quiet way down over the weekend | 17:27 |
| clarkb | (since msot of the jobs are run due to demand) | 17:28 |
| fungi | yeah, weekend load may not present a useful load test unless we also scale down our other providers to place more pressure on it | 17:28 |
| fungi | clarkb: 670606 also needs a new dib release before we can take advantage, right? | 17:29 |
| *** nicolasbock has joined #openstack-infra | 17:29 | |
| clarkb | fungi: yes | 17:29 |
| fungi | just making sure | 17:30 |
| *** rkukura has quit IRC | 17:31 | |
| openstackgerrit | Merged zuul/zuul-operator master: Add Kubernetes Operator Functional Test Job https://review.opendev.org/668029 | 17:37 |
| openstackgerrit | Merged zuul/zuul-operator master: Remove Operator SDK dependency in Zuul Job https://review.opendev.org/670584 | 17:37 |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Moving the workload on FNCI to 60% https://review.opendev.org/670609 | 17:39 |
| donnyd | I think we can leave it at 60% for a while. I am hopeful over the weekend I can at least get 60 jobs. | 17:40 |
| *** igordc has joined #openstack-infra | 17:43 | |
| AJaeger | donnyd: for a few more hours for sure - and probably for our periodic runs starting at 6:00 UTC | 17:48 |
| * clarkb cleans up the node held to debug the glean issues on centos | 17:50 | |
| clarkb | corvus: do you still need your held nodes that appear to be held for debugging gitea things? | 17:51 |
| clarkb | I can clean them up too if not | 17:51 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Build layout of non-live items with config updates https://review.opendev.org/670335 | 17:53 |
| corvus | clarkb: nope, sorry, i must have missed the timeout setting | 17:53 |
| corvus | we should set that in nodepool | 17:53 |
| *** xek has quit IRC | 17:53 | |
| clarkb | corvus: as a default you mean? | 17:53 |
| corvus | yep. i'll work up a change | 17:53 |
| clarkb | in any case cleaning those up now | 17:53 |
| *** xek has joined #openstack-infra | 17:53 | |
| corvus | i'm writing the change as penance for forgetting the option | 17:54 |
| corvus | hrm, that option doesn't behave the way i hoped. it's a max, not a default, so we wouldn't be able to override it | 17:55 |
| clarkb | we might be able to change the default on the zuul side? | 17:56 |
| corvus | i don't think there's a setting for that | 17:57 |
| Shrews | corvus: change --node-hold-expiration default | 17:57 |
| Shrews | 1 line change | 17:57 |
| corvus | Shrews: that would change it for everyone | 17:57 |
| corvus | i just want to establish an opendev default of 1 day | 17:57 |
| corvus | (but i want us to be able to set it to 1 week or indefinite if necessary for an individual node) | 17:58 |
| corvus | so i think zuul or nodepool needs to grow a new option for a site-customizable default value | 17:58 |
| Shrews | cant you set max-hold-age in cfg? | 17:59 |
| corvus | Shrews: that's a *max*. if i set that to 1 day, we can not override it | 17:59 |
| corvus | most autoholds we need for only a few hours, so 1 day would be okay. sometimes we hold on to them for a week while we work with providers to fix deeper issues | 17:59 |
| Shrews | oh, theres a max() call somewhere then | 18:00 |
| corvus | Shrews: i think there's a dedicated cleanup worker that looks for holds > max | 18:00 |
| corvus | so even if the znode has an expration set to 2d if the max is 1d it will still delete that node | 18:01 |
| *** betherly has joined #openstack-infra | 18:01 | |
| corvus | oh, maybe that one cleans up all holds | 18:02 |
| corvus | either way: | 18:02 |
| corvus | max_uptime = min(expiration, self._nodepool.config.max_hold_age) | 18:02 |
| *** betherly has quit IRC | 18:06 | |
| openstackgerrit | Merged openstack/project-config master: Moving the workload on FNCI to 60% https://review.opendev.org/670609 | 18:15 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Add functional jobs to gate https://review.opendev.org/670612 | 18:17 |
| *** electrofelix has quit IRC | 18:18 | |
| *** betherly has joined #openstack-infra | 18:21 | |
| openstackgerrit | Brian Haley proposed openstack/devstack-gate master: Support an IPv6 underlay network https://review.opendev.org/343041 | 18:23 |
| *** jeremy_houser has joined #openstack-infra | 18:24 | |
| jeremy_houser | Can anyone assist me in getting final reviews for https://review.opendev.org/#/c/670159/ ? Only need workflow and maybe one more +2 | 18:24 |
| jeremy_houser | I am attempting to merge first step of new repo for my existing tempest plugin | 18:24 |
| clarkb | haleyb: re ^ do we really want to add new features to devstack-gate? we should be adding that to the native zuul roles for multinode networking | 18:24 |
| clarkb | haleyb: If there is an immediate need I guess its fine, but we are (slowly) trying to get away from depending on devstack-gate for stuff | 18:24 |
| haleyb | clarkb: i was just going through my old reviews and re-basing | 18:25 |
| clarkb | haleyb: oh | 18:25 |
| haleyb | is there some other place we do this? i.e. calculate mtu to send to instances? that review is only like 3 years old :-o | 18:25 |
| *** betherly has quit IRC | 18:26 | |
| clarkb | jeremy_houser: is there existing code you need to import at the same time? if so you should set an upstream. If not I can approve it | 18:26 |
| clarkb | haleyb: ya let me find a link for the zuul native stuff | 18:26 |
| clarkb | haleyb: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge is the ansible role. https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge/tasks/common.yaml#L78-L113 is the bit that sets the mtu value | 18:27 |
| jeremy_houser | No, Ive decided against that method. Id rather commit my code from workstation as its already in an opendev repo and that seemed more like it was for something coming from github | 18:28 |
| clarkb | jeremy_houser: the upstream can be any publicly accessible git repo (does not need to be github) | 18:28 |
| clarkb | jeremy_houser: ok just asking beause we don't like to have to do force pushes for people after the facty | 18:28 |
| clarkb | jeremy_houser: I'll go ahead and approve it then if you are ready | 18:29 |
| haleyb | clarkb: hardcoded 50 byte overhead for vxlan :( i'll add looking at that to my pile | 18:29 |
| fungi | i often just git init on a personal webserver, git push into that and then that's a clonable repo which can be miported | 18:29 |
| jeremy_houser | I was just going to commit to the repo as normal, would it not work just fine? | 18:29 |
| clarkb | jeremy_houser: that will work just fine. Just double checking :) | 18:29 |
| jeremy_houser | fantastic, then yes, please approve when ready | 18:29 |
| fungi | jeremy_houser: the only reason to import would be if you already had a bunch of commits locally and didn't want to have to test and review them all individually when bootstrapping the project | 18:30 |
| jeremy_houser | ah no, that wont be an issue. Thank you for the information. | 18:30 |
| fungi | by default you'll end up with a mostly empty repo which only contains a .gitreview file, so you can clone from that, commit whatever changes you want and git review normally | 18:32 |
| jeremy_houser | that's how I thought it would work. Fantastic. | 18:33 |
| *** ijw has quit IRC | 18:35 | |
| openstackgerrit | Merged openstack/project-config master: New repo for ranger-tempest-plugin https://review.opendev.org/670159 | 18:37 |
| clarkb | jeremy_houser: ^ thats in so now just have to wait for the next ansible + puppet pulse | 18:41 |
| clarkb | usually about 30-45 minutes | 18:41 |
| *** betherly has joined #openstack-infra | 18:42 | |
| *** rascasoft has quit IRC | 18:44 | |
| *** liuyulong has quit IRC | 18:44 | |
| *** betherly has quit IRC | 18:46 | |
| *** rascasoft has joined #openstack-infra | 18:47 | |
| *** ccamacho has quit IRC | 18:48 | |
| AJaeger | donnyd: quote for your cloud is up to 60 - but seems are queue is shrinking, so you might not get a full load today ;( | 18:48 |
| *** irclogbot_1 has quit IRC | 18:49 | |
| *** edmondsw_ has quit IRC | 18:49 | |
| jeremy_houser | so if I wanted to set up my tempest-plugin to gate my project, would I set that up in the ranger .zuul.yaml or the tempest-plugin .zuul.yaml? | 18:50 |
| AJaeger | jeremy_houser: see how other repos do it ;) | 18:50 |
| AJaeger | jeremy_houser: my suggestion: One job defined in your tempest-plugin repo that gates changes to the plugin, the same job is run in the project as well to gate changes of the code | 18:50 |
| AJaeger | jeremy_houser: keep in mind we have global namespace for jobs, so you define once and can use everywhere, see also https://docs.openstack.org/infra/manual/drivers.html#consistent-naming-for-jobs-with-zuul-v3 | 18:51 |
| *** irclogbot_3 has joined #openstack-infra | 18:52 | |
| jeremy_houser | I apologize, Ive been doing this for six months but I'm trying to whip my team into modernizing their stuff, so I'm going headfirst into everything | 18:52 |
| donnyd | oh booooo AJaeger | 18:53 |
| donnyd | that makes me sad | 18:53 |
| *** michael-beaver has joined #openstack-infra | 18:56 | |
| fungi | donnyd: you could always ask dansmith to rebase a 70-commit-deep series of nova changes ;) | 18:57 |
| * dansmith pulls the rip cord on his chainsaw | 18:57 | |
| AJaeger | donnyd: 56 in use ;) | 19:05 |
| donnyd | Ok, I will watch it for a while and see if any issues come up. If you see a patch come in for the last 40% its because it looks good to go on my end | 19:09 |
| *** slaweq has quit IRC | 19:17 | |
| *** tomaw has quit IRC | 19:23 | |
| *** tomaw has joined #openstack-infra | 19:27 | |
| clarkb | http://logs.openstack.org/91/670591/1/check/tempest-full/73885b4/ timed out on fn so we may want to hold where we are now and monitor things | 19:33 |
| *** tesseract has quit IRC | 19:33 | |
| clarkb | (there is a very real chance we are our own noisy neighbor leading to that timeout) | 19:33 |
| *** slaweq has joined #openstack-infra | 19:34 | |
| fungi | yeah, most frequent problem in these scenarios is we're hitting an aggregate bottleneck somewhere in the environment (cpu, ram, disk i/o, network...) | 19:38 |
| fungi | and that starts to slow down a significant percentage of the jobs which run there | 19:38 |
| donnyd | I will start trying to run down the issue | 19:38 |
| *** skolt has quit IRC | 19:39 | |
| fungi | some of our longer-running jobs like devstack (so that one which timed out) do some performance tracking in the job as well | 19:39 |
| donnyd | I see my instance launch time has doubled, so I am thinking it may be a storage issue | 19:39 |
| donnyd | any pointers on which direction to turn in the investigation | 19:40 |
| fungi | what's the storage backend for the guest filesystems? | 19:40 |
| donnyd | NFS | 19:40 |
| donnyd | I have the storage scaled down to barebones atm while I measure how much each single node can handle | 19:41 |
| clarkb | if mnaser or logan- are around they may have thoughts (though they are all ceph based iirc) | 19:42 |
| donnyd | I was using ceph when we started, but my cluster isn't big enough to get the performance I was looking for | 19:42 |
| fungi | if it's nfs i'd look at your bandwidth utilization across your nfs vlan/interfaces | 19:43 |
| mnaser | yeah nfs is going to be pretty rough | 19:43 |
| fungi | see if there are bottlenecks on the network between it and the guests before looking at bottlenecks between the nfs servers and their disks | 19:44 |
| mnaser | mainly because you are doing block storage on top of file storage constructs | 19:44 |
| donnyd | https://www.irccloud.com/pastebin/ppmF4hPR/ | 19:44 |
| fungi | yeah, after network bandwidth between the compute hosts and the nfs servers i'd look at cpu utilization on the nfs servers | 19:44 |
| donnyd | Gonna need to bring more of the storages back online | 19:44 |
| mnaser | donnyd: is this cloud for other pruposes or is it mainly for openstack only? | 19:45 |
| donnyd | Well the network for each storage server is two 40G links | 19:45 |
| mnaser | openstack-infra ci workloads rather | 19:45 |
| *** whoami-rajat has quit IRC | 19:45 | |
| donnyd | yea, this is all it does | 19:45 |
| mnaser | i'd throw local disks and call it a day personally ;) | 19:45 |
| donnyd | its not built to be general purpose | 19:46 |
| donnyd | can't... I have blades | 19:46 |
| donnyd | and they suck at the local storages | 19:46 |
| mnaser | i mean you could totally raid-0 two drives only and put the os and /var/lib/nova/instances on it | 19:46 |
| mnaser | infra won't be sad if a bunch of machines disappeared off earth | 19:46 |
| fungi | iscsi will probably buy you marginally better performance than virtual block on nfs | 19:47 |
| mnaser | https://www.youtube.com/watch?v=4JWgmv92fQk | 19:47 |
| mnaser | and yeah, iscsi will probably yield some better results too | 19:47 |
| donnyd | Well that was where I started and I have weird issues with the containers and using iscsi for nova | 19:48 |
| *** bhavikdbavishi has quit IRC | 19:48 | |
| donnyd | like it just stops working weird | 19:48 |
| clarkb | ya the iscsi stuffin the kernel doesn't namespace | 19:48 |
| mnaser | oh with kolla? | 19:48 |
| donnyd | tripleo so yea kolla containers | 19:48 |
| clarkb | apparently there si a userspace iscsi driver in libvirt now but unsure of how that performs | 19:48 |
| fungi | or you add the necessary caps to get iscsi working because you're using containers for convenience not for security separation | 19:49 |
| mnaser | CAP_ADMIN all the things | 19:49 |
| donnyd | I only have 1 of 6 storage nodes online atm... so I could quite easily spread the load a little better | 19:49 |
| donnyd | and +150K on the jeremy clarkson video mnaser | 19:50 |
| *** slaweq has quit IRC | 19:51 | |
| clarkb | oh ya distributing the load regardless of the system underneath is likely ot help | 19:51 |
| donnyd | the storage servers are the main heat generators at least in my shabby little DC | 19:52 |
| donnyd | However there is good news, 1. I can just turn on more and 2. all NVME storage is enroute | 19:53 |
| clarkb | zoom zoom | 19:53 |
| logan- | limestone uses imagebackend (local storage) for the nodepool hvs | 19:54 |
| fungi | yeah, sounds like the blades for fortnebula aren't so robust in the local storage department | 19:55 |
| clarkb | speaking of fungi logan- I don't see any gaps in the limestone mirror cacti graphs for the last couple weeks ish | 19:55 |
| logan- | dual e5-2650v2, 128gb ram, 2x 512GB samsung 850 pro per HV. i/o seems to be the first bottleneck they hit | 19:55 |
| clarkb | should we try turning on that cloud again? | 19:55 |
| fungi | i'm game, no idea if logan- has had any luck hunting that down | 19:56 |
| logan- | yeah I found stacktraces from the nic driver on all 3 of the control nodes. i don't know if it is a kernel regression or some traffic triggering a bug in the igb driver | 19:56 |
| logan- | i don't think it is bad hardware since it happened on all 3 nodes | 19:56 |
| clarkb | logan-: huh, do you want to reboot to latest kernel versions (and drivers) before we turn it back on again? | 19:56 |
| logan- | i updated kernel and rebooted (igb reports the same version) | 19:56 |
| clarkb | (assuming they aren't already up to date) | 19:56 |
| clarkb | way ahead of me I see | 19:56 |
| clarkb | in that case ya maybe we try it again since cacti makes it look stabler | 19:57 |
| *** ociuhandu has joined #openstack-infra | 19:57 | |
| logan- | I have been running jobs on it and have not been able to repro it.. so I guess yeah we can turn it back on and see if it happens again. in other envs I have had issues with ubuntu's 10gig version of this driver (ixgbe) so I usually dkms it from upstream. thats my next thought if we run into more issues | 19:57 |
| *** ociuhandu has quit IRC | 19:58 | |
| *** slaweq has joined #openstack-infra | 19:59 | |
| donnyd | So it should look more like this | 20:01 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Reenable limestone cloud region in nodepool https://review.opendev.org/670630 | 20:01 |
| clarkb | fungi: logan- ^ fyi | 20:01 |
| donnyd | https://www.irccloud.com/pastebin/n7V2JxgE/ | 20:01 |
| logan- | thanks clarkb. i'll keep an eye on it once it merges | 20:01 |
| clarkb | donnyd: thats looks much better | 20:01 |
| donnyd | yea, so I am thinking I may need to spread the load a little better... ;) | 20:02 |
| *** betherly has joined #openstack-infra | 20:02 | |
| fungi | heh | 20:03 |
| *** slaweq has quit IRC | 20:03 | |
| clarkb | like butter | 20:04 |
| fungi | that's roughly theoretical throughput for a 1gbps network link | 20:04 |
| *** ykarel|away has quit IRC | 20:04 | |
| openstackgerrit | Corey Bryant proposed openstack/hacking master: Add Python 3 Train unit tests https://review.opendev.org/670632 | 20:05 |
| openstackgerrit | Corey Bryant proposed openstack/os-performance-tools master: Add Python 3 Train unit tests https://review.opendev.org/670635 | 20:07 |
| *** betherly has quit IRC | 20:07 | |
| openstackgerrit | Corey Bryant proposed openstack/os-testr master: Add Python 3 Train unit tests https://review.opendev.org/670636 | 20:07 |
| *** davidsha has joined #openstack-infra | 20:07 | |
| *** ociuhandu has joined #openstack-infra | 20:08 | |
| donnyd | https://www.irccloud.com/pastebin/daa1oXyI/ | 20:08 |
| donnyd | The first one was with 4k blocks, thats what I am testing because the DB part seemed to be the slowest in the beginning | 20:09 |
| donnyd | so I get pretty much all of a single 10G link in disk performance | 20:11 |
| clarkb | is that against the nfs server currently being used? | 20:11 |
| donnyd | Well its got a couple instances on it now, but not heavily loaded at all | 20:12 |
| clarkb | ah | 20:12 |
| openstackgerrit | Merged openstack/diskimage-builder master: Only enable dbus-daemon on fedora-29 https://review.opendev.org/670606 | 20:13 |
| *** slaweq has joined #openstack-infra | 20:15 | |
| clarkb | so maybe monday morning I'll cut a dib release? I am wary of doing it before the weekend since I'm not super familiar with all the chagnes that have gone in since the last one | 20:15 |
| clarkb | I'll start reviewing the git log now | 20:16 |
| clarkb | looks like the big changes are the two I've made recently then johnsom added a flag to select a different ubuntu kernel in ubuntu-minimal and prometheanfire added gnupg2 to debian default package list for apt and there is a fix for rhel8 python stuff | 20:18 |
| clarkb | overall not too scary | 20:18 |
| clarkb | (much better than I was worried about) | 20:18 |
| johnsom | We would love to see a DIB release.... | 20:18 |
| clarkb | johnsom: us too :) I jsut know I'm not going to be able to write fixes or pin dib to previous relase over the weekend and the rtt on getting images built and in clouds is long :/ | 20:19 |
| clarkb | I can do it first thing monday | 20:19 |
| johnsom | Yeah, Monday is sooner then RC. grin | 20:20 |
| *** slaweq has quit IRC | 20:20 | |
| openstackgerrit | Merged openstack/project-config master: Reenable limestone cloud region in nodepool https://review.opendev.org/670630 | 20:21 |
| *** betherly has joined #openstack-infra | 20:23 | |
| *** betherly has quit IRC | 20:27 | |
| *** davidsha has quit IRC | 20:30 | |
| *** ociuhandu has quit IRC | 20:31 | |
| *** slaweq has joined #openstack-infra | 20:31 | |
| *** slaweq has quit IRC | 20:35 | |
| *** goldyfruit has quit IRC | 20:35 | |
| donnyd | pretty sure all these jobs are about to hit their timeouts | 20:36 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Build layout of non-live items with config updates https://review.opendev.org/670335 | 20:41 |
| donnyd | I also randomly get neutron timeouts in the log | 20:44 |
| donnyd | ('Connection aborted.', BadStatusLine("''",)) | 20:44 |
| donnyd | tried bumping up the workers for the neutron server, but doesn't seem to help | 20:44 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Build layout of non-live items with config updates https://review.opendev.org/670335 | 20:45 |
| fungi | yeah, BadStatusLine(empty) is thieves cant for the requests library. in the common tongue it means it never got an answer before it gave up waiting | 20:49 |
| clarkb | donnyd: oh one trick I should mention is if the quota is reduced nodepool will honor that so you can actually manage things that way if you want to make changes or reduce load etc (just set max instances quota to a smaller number) | 20:51 |
| clarkb | (I say that but I'm not sure we've every tested it for that use case) | 20:51 |
| donnyd | I have enough to disable hypervisors, wait for the load to go to zero and then migrate the backend | 20:52 |
| donnyd | but that is a pretty good tip for when the new storage gets here | 20:53 |
| donnyd | because it would be faster to scale to zero and then swap it out without code changes | 20:54 |
| *** jeremy_houser has quit IRC | 20:56 | |
| fungi | yeah, basically the max-servers in nodepool these days is a way for us to tell it to use less than its actual in-provider quotas would allow | 20:58 |
| fungi | but nodepool still tracks openstack api reported quotas and tries not to exceed them | 20:59 |
| *** pcaruana has quit IRC | 20:59 | |
| donnyd | I have two more storage servers back online, so I am just waiting to swap them in and then see if the result is good | 21:01 |
| *** betherly has joined #openstack-infra | 21:04 | |
| *** weifan has joined #openstack-infra | 21:07 | |
| *** betherly has quit IRC | 21:09 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Handle existing broken config in job updates https://review.opendev.org/670666 | 21:17 |
| *** weifan has quit IRC | 21:18 | |
| *** weifan has joined #openstack-infra | 21:22 | |
| *** dpawlik has joined #openstack-infra | 21:25 | |
| clarkb | https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ is an interesting read | 21:27 |
| * clarkb looks at zuul change now that I'm through ^ | 21:27 | |
| *** ijw has joined #openstack-infra | 21:31 | |
| *** bgmccollum has quit IRC | 21:31 | |
| *** ekultails has quit IRC | 21:32 | |
| *** bgmccollum has joined #openstack-infra | 21:33 | |
| *** dpawlik has quit IRC | 21:35 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Handle existing broken config in job updates https://review.opendev.org/670666 | 21:37 |
| *** weifan has quit IRC | 21:38 | |
| *** weifan has joined #openstack-infra | 21:40 | |
| *** weifan has quit IRC | 21:41 | |
| *** weifan has joined #openstack-infra | 21:42 | |
| donnyd | https://www.irccloud.com/pastebin/0uWOPRNm/ | 21:43 |
| donnyd | So this is what a node looks like when the storage is loaded | 21:44 |
| donnyd | still some left over.. much better than 876/246 | 21:44 |
| *** weifan has quit IRC | 21:47 | |
| *** ianychoi has quit IRC | 21:54 | |
| *** rlandy has quit IRC | 21:54 | |
| *** guimaluf has quit IRC | 21:57 | |
| donnyd | I'm seeing quite a lot of errors popping up in logstash, not sure if that is on my end or the job | 21:58 |
| clarkb | donnyd: http://logs.openstack.org/66/670666/2/check/zuul-tox-remote/6959fc6/job-output.txt#_2019-07-12_21_56_21_388353 | 22:00 |
| clarkb | if your changes can result in read only filesystems it could be related | 22:00 |
| donnyd | is there any way to tell what host that was running on? | 22:01 |
| clarkb | I think we record that, let me look | 22:03 |
| logan- | yep, it is the 'build_hostid' field in logstash | 22:03 |
| clarkb | 2019-07-12 21:38:21,006 DEBUG nodepool.NodeLauncher: [node: 0008885215] Node 0008885215 is running [region: regionOne, az: nova, ip: 2001:470:e045:1:f816:3eff:fe70:5c43 ipv4: , ipv6: 2001:470:e045:1:f816:3eff:fe70:5c43, hostid: d8db5efa427a20886209c6207822af9e0c218fda9b7c05e06ab4a546] | 22:03 |
| clarkb | found it in nodepool too | 22:03 |
| logan- | yep also stored in http://logs.openstack.org/66/670666/2/check/zuul-tox-remote/6959fc6/zuul-info/inventory.yaml | 22:04 |
| logan- | actually getting a hostname out of that is a little bit interesting though. looking for my notes on that | 22:04 |
| *** xek has quit IRC | 22:05 | |
| logan- | http://paste.openstack.org/raw/754361/ | 22:06 |
| donnyd | hrm | 22:08 |
| donnyd | https://www.irccloud.com/pastebin/RbjLJa03/ | 22:08 |
| logan- | you'll need to replace the project ID with whatever project ID the nodepool tenant is on | 22:09 |
| donnyd | oh... ic | 22:09 |
| donnyd | yea, so that makes more sense... It failed on the hypervisor I just swapped out the storage i | 22:10 |
| donnyd | yea, so that makes more sense... It failed on the hypervisor I just swapped out the storage in | 22:10 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add "supercedes" pipeline option https://review.opendev.org/670670 | 22:13 |
| donnyd | not sure why it did that, but at least I know where to start | 22:13 |
| *** betherly has joined #openstack-infra | 22:17 | |
| *** betherly has quit IRC | 22:21 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add "supercedes" pipeline option https://review.opendev.org/670670 | 22:24 |
| *** slaweq has joined #openstack-infra | 22:33 | |
| *** slaweq has quit IRC | 22:38 | |
| *** michael-beaver has quit IRC | 22:57 | |
| *** ociuhandu has joined #openstack-infra | 23:01 | |
| *** mattw4 has quit IRC | 23:01 | |
| *** sthussey has quit IRC | 23:04 | |
| *** ociuhandu has quit IRC | 23:05 | |
| *** betherly has joined #openstack-infra | 23:19 | |
| *** tosky has quit IRC | 23:20 | |
| *** nicolasbock has quit IRC | 23:22 | |
| *** weifan has joined #openstack-infra | 23:23 | |
| *** betherly has quit IRC | 23:24 | |
| *** weifan has quit IRC | 23:28 | |
| *** eharney has quit IRC | 23:32 | |
| *** betherly has joined #openstack-infra | 23:39 | |
| *** bobh has joined #openstack-infra | 23:41 | |
| *** betherly has quit IRC | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!