| *** tdasilva has quit IRC | 00:08 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** mriedem has quit IRC | 00:15 | |
| *** slaweq has quit IRC | 00:15 | |
| *** lseki has quit IRC | 00:23 | |
| *** sthussey has quit IRC | 00:30 | |
| *** gyee has quit IRC | 00:47 | |
| *** hongbin has joined #openstack-infra | 00:57 | |
| *** dims has joined #openstack-infra | 00:59 | |
| *** imacdonn has quit IRC | 01:14 | |
| *** imacdonn has joined #openstack-infra | 01:14 | |
| *** ricolin has joined #openstack-infra | 01:23 | |
| *** bobh has joined #openstack-infra | 01:43 | |
| *** lei-zh has joined #openstack-infra | 01:44 | |
| *** kevinz has joined #openstack-infra | 01:49 | |
| *** kevinz has quit IRC | 01:52 | |
| *** apetrich has quit IRC | 01:58 | |
| *** pkopec_ has quit IRC | 02:03 | |
| *** bobh has quit IRC | 02:05 | |
| *** irclogbot_0 has joined #openstack-infra | 02:05 | |
| *** bobh has joined #openstack-infra | 02:07 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** irclogbot_0 has quit IRC | 02:13 | |
| *** irclogbot_1 has joined #openstack-infra | 02:16 | |
| *** slaweq has quit IRC | 02:16 | |
| *** altlogbot_1 has joined #openstack-infra | 02:17 | |
| *** irclogbot_1 has quit IRC | 02:19 | |
| *** rcernin has quit IRC | 02:29 | |
| *** kevinz has joined #openstack-infra | 02:30 | |
| kevinz | Hi infra team. This is kevinz from Linaro | 02:31 |
|---|---|---|
| *** dklyle has joined #openstack-infra | 02:32 | |
| kevinz | We like to offer Arm64 CI gating. We have a bunch of machines in Packet, which will be the CI basement | 02:32 |
| kevinz | I wonder the steps for setup the CI on Arm64 | 02:33 |
| *** altlogbot_1 has quit IRC | 02:34 | |
| *** rcernin has joined #openstack-infra | 02:46 | |
| *** auristor has quit IRC | 02:49 | |
| clarkb | kevinz: hey, this will be a different set of resources than the existing london cloud? | 02:50 |
| clarkb | if so then we'll need the new cloud credentials (one account/tenant for the mirror and the second for the test nodes) | 02:51 |
| clarkb | its a bit late here but if you want to send me and email we can work on getting this going | 02:51 |
| kevinz | <clarkb> Yes, we got resource for more machines on Packet | 02:52 |
| kevinz | <clarkb>: sure, I will drop an email to you. Thanks | 02:52 |
| *** dklyle has quit IRC | 02:59 | |
| *** bobh has quit IRC | 03:00 | |
| *** auristor has joined #openstack-infra | 03:08 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** slaweq has quit IRC | 03:15 | |
| *** irclogbot_0 has joined #openstack-infra | 03:20 | |
| *** whoami-rajat has joined #openstack-infra | 03:20 | |
| *** psachin has joined #openstack-infra | 03:20 | |
| *** whoami-rajat has quit IRC | 03:24 | |
| *** whoami-rajat has joined #openstack-infra | 03:25 | |
| *** irclogbot_0 has quit IRC | 03:25 | |
| *** lei-zh1 has joined #openstack-infra | 03:32 | |
| *** lei-zh has quit IRC | 03:34 | |
| *** rlandy|bbl is now known as rlandy | 03:36 | |
| *** kevinz has quit IRC | 03:36 | |
| *** psachin has quit IRC | 03:38 | |
| *** rcernin has quit IRC | 03:39 | |
| *** hongbin has quit IRC | 03:44 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:51 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:53 | |
| *** altlogbot_0 has joined #openstack-infra | 03:53 | |
| *** bhavikdbavishi has quit IRC | 03:55 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:55 | |
| *** rcernin has joined #openstack-infra | 03:55 | |
| *** altlogbot_0 has quit IRC | 03:59 | |
| *** ykarel has joined #openstack-infra | 04:10 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** udesale has joined #openstack-infra | 04:13 | |
| *** viks___ has joined #openstack-infra | 04:14 | |
| *** slaweq has quit IRC | 04:16 | |
| *** jamesmcarthur has joined #openstack-infra | 04:21 | |
| *** jamesmcarthur has quit IRC | 04:40 | |
| *** jamesmcarthur has joined #openstack-infra | 04:41 | |
| *** jamesmcarthur has quit IRC | 04:46 | |
| *** lei-zh1 has quit IRC | 04:50 | |
| *** jamesmcarthur has joined #openstack-infra | 04:51 | |
| *** pcaruana has joined #openstack-infra | 04:55 | |
| *** jamesmcarthur has quit IRC | 04:56 | |
| *** lei-zh has joined #openstack-infra | 05:00 | |
| *** jamesmcarthur has joined #openstack-infra | 05:02 | |
| abelur | clarkb: would you know whom to reachout for a few any swift/container/object storage related questions? | 05:02 |
| *** jamesmcarthur has quit IRC | 05:07 | |
| *** yamamoto has quit IRC | 05:27 | |
| *** Lucas_Gray has quit IRC | 05:28 | |
| *** ccamacho has quit IRC | 05:33 | |
| *** yamamoto has joined #openstack-infra | 05:33 | |
| *** yamamoto has quit IRC | 05:40 | |
| *** armax has quit IRC | 05:45 | |
| AJaeger | abelur: there's a whole list of IRC channels for OpenStack, see https://wiki.openstack.org/wiki/IRC for the swift one | 05:56 |
| *** Goneri has joined #openstack-infra | 06:00 | |
| *** lei-zh has quit IRC | 06:01 | |
| *** lei-zh has joined #openstack-infra | 06:05 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** ccamacho has joined #openstack-infra | 06:14 | |
| *** slaweq has quit IRC | 06:16 | |
| *** pgaxatte has joined #openstack-infra | 06:17 | |
| *** altlogbot_0 has joined #openstack-infra | 06:18 | |
| *** aedc has joined #openstack-infra | 06:22 | |
| *** kjackal has joined #openstack-infra | 06:22 | |
| *** altlogbot_0 has quit IRC | 06:23 | |
| *** altlogbot_2 has joined #openstack-infra | 06:24 | |
| *** altlogbot_2 has quit IRC | 06:29 | |
| *** dpawlik has joined #openstack-infra | 06:29 | |
| *** rpittau|afk is now known as rpittau | 06:33 | |
| *** yamamoto has joined #openstack-infra | 06:35 | |
| *** gtema has quit IRC | 06:37 | |
| *** ricolin has quit IRC | 06:39 | |
| *** yamamoto has quit IRC | 06:45 | |
| *** ricolin has joined #openstack-infra | 06:46 | |
| *** slaweq has joined #openstack-infra | 06:46 | |
| *** gtema has joined #openstack-infra | 06:48 | |
| *** altlogbot_2 has joined #openstack-infra | 06:56 | |
| *** iurygregory has joined #openstack-infra | 06:59 | |
| *** altlogbot_2 has quit IRC | 07:01 | |
| *** ianychoi_ is now known as ianychoi | 07:01 | |
| *** aedc has quit IRC | 07:02 | |
| *** dchen has quit IRC | 07:03 | |
| *** diablo_rojo has quit IRC | 07:04 | |
| *** ginopc has joined #openstack-infra | 07:07 | |
| *** witek has joined #openstack-infra | 07:11 | |
| *** ccamacho has quit IRC | 07:12 | |
| *** ccamacho has joined #openstack-infra | 07:12 | |
| *** irclogbot_3 has joined #openstack-infra | 07:16 | |
| *** rcernin has quit IRC | 07:17 | |
| *** Goneri has quit IRC | 07:19 | |
| *** tosky has joined #openstack-infra | 07:20 | |
| *** irclogbot_3 has quit IRC | 07:21 | |
| *** lei-zh has quit IRC | 07:22 | |
| *** irclogbot_1 has joined #openstack-infra | 07:24 | |
| *** irclogbot_1 has quit IRC | 07:27 | |
| *** lei-zh has joined #openstack-infra | 07:28 | |
| *** Goneri has joined #openstack-infra | 07:33 | |
| *** ykarel is now known as ykarel|afk | 07:33 | |
| *** xek_ has joined #openstack-infra | 07:39 | |
| *** lei-zh has quit IRC | 07:40 | |
| *** lei-zh has joined #openstack-infra | 07:41 | |
| *** jtomasek has joined #openstack-infra | 07:44 | |
| *** altlogbot_2 has joined #openstack-infra | 07:48 | |
| *** altlogbot_2 has quit IRC | 07:49 | |
| *** irclogbot_1 has joined #openstack-infra | 07:52 | |
| *** irclogbot_1 has quit IRC | 07:55 | |
| *** jaicaa_ has quit IRC | 07:55 | |
| *** jaicaa has joined #openstack-infra | 07:56 | |
| *** ralonsoh has joined #openstack-infra | 07:59 | |
| *** ykarel|afk is now known as ykarel | 08:00 | |
| *** Fidde has joined #openstack-infra | 08:06 | |
| *** elod has quit IRC | 08:07 | |
| *** irclogbot_3 has joined #openstack-infra | 08:08 | |
| *** elod has joined #openstack-infra | 08:08 | |
| *** tkajinam has quit IRC | 08:10 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul master: Fix typo in debug message https://review.opendev.org/670245 | 08:12 |
| *** aedc has joined #openstack-infra | 08:12 | |
| *** irclogbot_3 has quit IRC | 08:13 | |
| *** ociuhandu has joined #openstack-infra | 08:15 | |
| *** jtomasek has quit IRC | 08:23 | |
| *** udesale has quit IRC | 08:30 | |
| *** lucasagomes has joined #openstack-infra | 08:30 | |
| *** udesale has joined #openstack-infra | 08:31 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 08:32 |
| *** yamamoto has joined #openstack-infra | 08:32 | |
| *** xek_ has quit IRC | 08:35 | |
| *** ykarel is now known as ykarel|lunch | 08:36 | |
| *** aedc has quit IRC | 08:39 | |
| *** aedc has joined #openstack-infra | 08:39 | |
| *** derekh has joined #openstack-infra | 08:40 | |
| *** jtomasek has joined #openstack-infra | 08:44 | |
| *** ianychoi has quit IRC | 08:54 | |
| *** yamamoto has quit IRC | 08:55 | |
| *** Lucas_Gray has joined #openstack-infra | 08:56 | |
| *** dtantsur|afk is now known as dtantsur | 08:57 | |
| *** adriancz has joined #openstack-infra | 08:57 | |
| *** xek has joined #openstack-infra | 09:02 | |
| *** yamamoto has joined #openstack-infra | 09:02 | |
| *** yamamoto has quit IRC | 09:02 | |
| *** yamamoto has joined #openstack-infra | 09:03 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Optionally support mitogen for job execution https://review.opendev.org/657024 | 09:04 |
| *** yamamoto has quit IRC | 09:07 | |
| *** yamamoto has joined #openstack-infra | 09:08 | |
| *** udesale has quit IRC | 09:08 | |
| *** udesale has joined #openstack-infra | 09:09 | |
| *** Goneri has quit IRC | 09:10 | |
| *** lei-zh1 has joined #openstack-infra | 09:10 | |
| *** ykarel|lunch is now known as ykarel | 09:11 | |
| *** lei-zh has quit IRC | 09:11 | |
| *** dpawlik has quit IRC | 09:12 | |
| *** pkopec has joined #openstack-infra | 09:12 | |
| *** dpawlik has joined #openstack-infra | 09:13 | |
| *** dpawlik has quit IRC | 09:18 | |
| *** dpawlik has joined #openstack-infra | 09:18 | |
| *** Goneri has joined #openstack-infra | 09:23 | |
| *** FlorianFa has joined #openstack-infra | 09:36 | |
| *** udesale has quit IRC | 09:37 | |
| *** udesale has joined #openstack-infra | 09:38 | |
| *** ianychoi has joined #openstack-infra | 09:38 | |
| *** gtema__ has joined #openstack-infra | 09:39 | |
| *** udesale has quit IRC | 09:40 | |
| *** udesale has joined #openstack-infra | 09:41 | |
| *** gtema has quit IRC | 09:41 | |
| *** Lucas_Gray has quit IRC | 09:42 | |
| *** gtema__ has quit IRC | 09:43 | |
| *** yamamoto has quit IRC | 09:44 | |
| *** gtema has joined #openstack-infra | 09:44 | |
| *** irclogbot_0 has joined #openstack-infra | 09:48 | |
| *** dmsimard4 has joined #openstack-infra | 09:50 | |
| *** dmsimard has quit IRC | 09:51 | |
| *** irclogbot_0 has quit IRC | 09:51 | |
| *** gtema has quit IRC | 09:57 | |
| *** udesale has quit IRC | 10:02 | |
| *** tosky has quit IRC | 10:02 | |
| *** udesale has joined #openstack-infra | 10:02 | |
| *** udesale has quit IRC | 10:03 | |
| *** udesale has joined #openstack-infra | 10:03 | |
| *** tosky has joined #openstack-infra | 10:05 | |
| abelur | AJaeger: ack thanks | 10:08 |
| *** aedc has quit IRC | 10:10 | |
| *** aedc has joined #openstack-infra | 10:11 | |
| *** psachin has joined #openstack-infra | 10:12 | |
| *** altlogbot_0 has joined #openstack-infra | 10:26 | |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Update ACLs for moved doc projects https://review.opendev.org/670269 | 10:27 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Add shared 'oslo.config' ACL file https://review.opendev.org/670270 | 10:27 |
| *** altlogbot_0 has quit IRC | 10:29 | |
| *** janki has joined #openstack-infra | 10:32 | |
| *** aedc has quit IRC | 10:33 | |
| *** altlogbot_3 has joined #openstack-infra | 10:34 | |
| *** altlogbot_3 has quit IRC | 10:39 | |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Update ACLs for moved doc projects https://review.opendev.org/670269 | 10:39 |
| openstackgerrit | Stephen Finucane proposed openstack/project-config master: Add shared 'oslo.config' ACL file https://review.opendev.org/670270 | 10:40 |
| *** rascasoft has quit IRC | 10:40 | |
| *** altlogbot_0 has joined #openstack-infra | 10:40 | |
| *** irclogbot_3 has joined #openstack-infra | 10:40 | |
| *** rascasoft has joined #openstack-infra | 10:41 | |
| *** kjackal has quit IRC | 10:41 | |
| *** lpetrut has joined #openstack-infra | 10:45 | |
| *** altlogbot_0 has quit IRC | 10:45 | |
| *** irclogbot_3 has quit IRC | 10:45 | |
| *** Lucas_Gray has joined #openstack-infra | 10:47 | |
| *** kjackal has joined #openstack-infra | 10:53 | |
| *** altlogbot_0 has joined #openstack-infra | 10:56 | |
| *** Lucas_Gray has quit IRC | 10:59 | |
| *** altlogbot_0 has quit IRC | 11:01 | |
| *** lpetrut has quit IRC | 11:01 | |
| *** lpetrut has joined #openstack-infra | 11:03 | |
| *** gtema has joined #openstack-infra | 11:07 | |
| sshnaidm|ruck | do we have "tripleo" queue in gate pipeline only? Or in check as well | 11:11 |
| *** lei-zh1 has quit IRC | 11:15 | |
| zbr | clarkb: regarding my [doc8] message on the list, email to jxharlow@godaddy.com was refused with Recipient address rejected: Access denied | 11:17 |
| zbr | and he is listed as the maintainer, even on pypi.org | 11:17 |
| zbr | also imelnikov@griddynamics.com was reported as not existing. so at least 2/4 emails not working. this may explain the lack of maintenance on doc8 | 11:18 |
| *** tesseract has joined #openstack-infra | 11:21 | |
| *** _erlon_ has joined #openstack-infra | 11:24 | |
| *** priteau has joined #openstack-infra | 11:29 | |
| smcginnis | harlowja: May need to update your contact info ^^ | 11:34 |
| AJaeger | sshnaidm|ruck: in check as well, see project-config/zuul.d/projects.yaml | 11:38 |
| *** udesale has quit IRC | 11:39 | |
| *** yamamoto has joined #openstack-infra | 11:45 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Translate gitea project creation to python https://review.opendev.org/670060 | 11:45 |
| *** apetrich has joined #openstack-infra | 11:46 | |
| jonher | looks like jxharlow switched companies in September | 11:52 |
| *** yamamoto has quit IRC | 11:53 | |
| *** altlogbot_3 has joined #openstack-infra | 11:56 | |
| *** ykarel is now known as ykarel|afk | 11:57 | |
| *** ijw has joined #openstack-infra | 11:57 | |
| *** rh-jelabarre has joined #openstack-infra | 11:59 | |
| *** altlogbot_3 has quit IRC | 12:01 | |
| *** ijw has quit IRC | 12:01 | |
| *** gtema has quit IRC | 12:11 | |
| *** priteau has quit IRC | 12:13 | |
| openstackgerrit | Merged zuul/zuul master: Fix typo in debug message https://review.opendev.org/670245 | 12:15 |
| sshnaidm|ruck | AJaeger, hmm.. but I can't see them in http://zuul.openstack.org/api/status in check pipeline. All queue names there are project names, maybe it's not updated in API? | 12:17 |
| sshnaidm|ruck | clarkb, mordred do you know? ^^ | 12:17 |
| *** janki has quit IRC | 12:21 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Translate gitea project creation to python https://review.opendev.org/670060 | 12:21 |
| *** ykarel|afk is now known as ykarel | 12:23 | |
| *** rlandy has joined #openstack-infra | 12:30 | |
| *** sthussey has joined #openstack-infra | 12:31 | |
| *** irclogbot_0 has joined #openstack-infra | 12:32 | |
| *** irclogbot_0 has quit IRC | 12:35 | |
| *** aedc has joined #openstack-infra | 12:39 | |
| *** hamzy has quit IRC | 12:42 | |
| *** janki has joined #openstack-infra | 12:43 | |
| *** janki has quit IRC | 12:45 | |
| *** altlogbot_0 has joined #openstack-infra | 12:50 | |
| *** altlogbot_0 has quit IRC | 12:53 | |
| fungi | sshnaidm|ruck: named queues are a specific feature of dependent pipelines. are you just curious or do you think it's related to a problem? | 12:54 |
| fungi | https://zuul-ci.org/docs/zuul/user/config.html#attr-project.%3Cpipeline%3E.queue | 12:55 |
| fungi | Pipeline managers other than dependent do not use this attribute, however, it may still be used if scheduler.relative_priority is enabled. | 12:55 |
| fungi | so it does use the shared queue in check (an independent pipeline) for purposes of relative priority scheduling | 12:55 |
| *** dmsimard4 is now known as dmsimard | 12:58 | |
| fungi | but that's just for purposes of calculating node allocation priority, it doesn't queue the changes together in an independent pipeline (that's a dependent pipeline behavior) so it won't display those queue names in the check pipeline status, only gate | 12:59 |
| *** ekultails has joined #openstack-infra | 12:59 | |
| *** Lucas_Gray has joined #openstack-infra | 13:00 | |
| *** iurygregory has quit IRC | 13:07 | |
| sshnaidm|ruck | fungi, so actually check run in shared queue, right? | 13:07 |
| sshnaidm|ruck | fungi, or what does mean node allocation priority..? | 13:08 |
| mordred | fungi, clarkb, corvus: https://review.opendev.org/#/c/670060 is green and ready for review | 13:09 |
| *** sshnaidm|ruck is now known as sshnaidm|rover | 13:20 | |
| *** arxcruz|rover is now known as arxcruz|ruck | 13:20 | |
| *** iurygregory has joined #openstack-infra | 13:20 | |
| *** pkopec has quit IRC | 13:22 | |
| *** pkopec has joined #openstack-infra | 13:23 | |
| *** yamamoto has joined #openstack-infra | 13:23 | |
| *** lseki has joined #openstack-infra | 13:26 | |
| *** goldyfruit has joined #openstack-infra | 13:33 | |
| *** irclogbot_2 has joined #openstack-infra | 13:40 | |
| *** pkopec has quit IRC | 13:40 | |
| *** pkopec has joined #openstack-infra | 13:41 | |
| *** mriedem has joined #openstack-infra | 13:42 | |
| *** irclogbot_2 has quit IRC | 13:45 | |
| *** hamzy has joined #openstack-infra | 13:47 | |
| *** whoami-rajat has quit IRC | 13:48 | |
| *** michael-beaver has joined #openstack-infra | 13:49 | |
| fungi | sshnaidm|rover: in a dependent pipeline zuul queues changes serially by project. if multiple projects share a change queue then it queues the changes for all of those projects serially. independent pipelines like the gate do not queue changes serially even if they're in the same project so that aspect of named queues isn't relevant in check | 13:50 |
| *** yamamoto has quit IRC | 13:51 | |
| *** Fidde has quit IRC | 13:51 | |
| fungi | sshnaidm|rover: separately, node requests for builds go into a queue for fulfillment. what order those node requests are fulfilled in is determined by a number of configurable rules including the priority of the pipeline they're destined for and how much activity there is in the projects individually or (if there is a "queue" specified on the pipeline) grouped by their queue (even though the changes | 13:52 |
| fungi | themselves aren't serialized in a queue in that case) | 13:52 |
| *** pkopec has quit IRC | 13:52 | |
| *** whoami-rajat has joined #openstack-infra | 13:52 | |
| fungi | er, when i said "independent pipelines like the gate do not queue changes serially" i meant "check" there, not "gate" | 13:53 |
| *** pkopec has joined #openstack-infra | 13:55 | |
| sshnaidm|rover | fungi, thanks for clarification | 13:56 |
| *** udesale has joined #openstack-infra | 13:56 | |
| *** ccamacho has quit IRC | 14:03 | |
| clarkb | mordred: can you reviee https://review.opendev.org/#/c/670197/ whoch is gentoo fix in glean? | 14:06 |
| clarkb | if we get that in I'll tag glean and rebuild images and we can turn on fn again | 14:06 |
| *** ykarel is now known as ykarel|afk | 14:06 | |
| mordred | yup | 14:11 |
| mordred | clarkb: done | 14:12 |
| mordred | clarkb: https://review.opendev.org/#/c/670060 is also ready - but it's a bit longer of a review than that one :) | 14:12 |
| mordred | corvus: do you think we should update that patch to run the full project creation in the test job so we can verify it can get through the whole thing? or do that as a followup | 14:13 |
| corvus | mordred: how about we go ahead and do that as a followup, but don't merge the frist yet | 14:15 |
| mordred | corvus: wfm | 14:16 |
| corvus | mordred: https://review.opendev.org/670060 looks good, but i left a note maybe you could get in the followup? | 14:19 |
| mordred | corvus: totally | 14:20 |
| *** donnyd_pto is now known as donnyd | 14:21 | |
| *** yamamoto has joined #openstack-infra | 14:22 | |
| clarkb | mordred: corvus and maybe run it twice in succession to see noops on second pass | 14:24 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run actual full project creation in gitea test https://review.opendev.org/670313 | 14:25 |
| mordred | clarkb: well - we should probably improve that module to report information about what it's doing / has done / has made changes | 14:26 |
| donnyd | clarkb: So it looks like we are getting pretty close :) | 14:26 |
| clarkb | donnyd: yup | 14:27 |
| clarkb | mordred: especially if it helps us verify it works :) | 14:27 |
| roman_g | Hello team. Please, review: https://review.opendev.org/#/c/666606/ - opendev/system-config Mailing list for Airship. Thank you! | 14:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add some logging to repo creation https://review.opendev.org/670317 | 14:41 |
| mordred | clarkb: that should collect the actions performed for us | 14:41 |
| *** pgaxatte has quit IRC | 14:43 | |
| corvus | that's gonna be big in testing :) | 14:46 |
| *** ociuhandu has quit IRC | 14:47 | |
| roman_g | Thanks to James E. Blair! | 14:49 |
| corvus | roman_g: np | 14:51 |
| roman_g | corvus: :) | 14:51 |
| *** aedc has quit IRC | 14:52 | |
| openstackgerrit | Merged opendev/glean master: Enable RAs with gentoo when using dhcpv6-stateless https://review.opendev.org/670197 | 14:57 |
| *** armax has joined #openstack-infra | 15:00 | |
| *** yamamoto has quit IRC | 15:00 | |
| *** yamamoto has joined #openstack-infra | 15:01 | |
| *** yamamoto has quit IRC | 15:01 | |
| clarkb | mordred: I've skimmed 670060 overtea and it lgtm. I'll give it a proper look with the testing once my morning has settled in. Slow start today | 15:11 |
| *** pcaruana has quit IRC | 15:11 | |
| openstackgerrit | Merged opendev/system-config master: Mailing list for Airship https://review.opendev.org/666606 | 15:13 |
| mordred | clarkb: yah - I hear that | 15:20 |
| mordred | clarkb, corvus: fwiw - in the job that runs the whole projects.yaml it's been running that task for 18 minutes so far | 15:21 |
| corvus | mordred: i guess "hours" is the time to beat? | 15:21 |
| clarkb | iirc ~4 hours last I did a full sync | 15:22 |
| mordred | yeah | 15:23 |
| *** ykarel|afk is now known as ykarel|away | 15:23 | |
| *** gyee has joined #openstack-infra | 15:23 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 15:26 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Build layout of most recent non-live item https://review.opendev.org/670335 | 15:27 |
| *** udesale has quit IRC | 15:28 | |
| cloudnull | has anyone seen this error before - http://logs.openstack.org/12/1255faa87d1cec03e5b6fb63388db9d7da41b952/post/publish-openstack-sphinx-docs/4fad3d5/job-output.txt.gz#_2019-07-11_15_12_07_120749 ? sphinx doc builds in post job are failing, and I'm not sure why. | 15:29 |
| cloudnull | http://zuul.openstack.org/build/4fad3d5aa2114b65a7aadf23ab5384ad | 15:29 |
| AJaeger | cloudnull: does the job work in check? It might be a sphinx version problem | 15:30 |
| cloudnull | it does | 15:30 |
| cloudnull | https://review.opendev.org/#/c/670167/ | 15:31 |
| cloudnull | that was the review | 15:31 |
| *** Goneri has quit IRC | 15:32 | |
| cloudnull | AJaeger maybe I need to pass in the something like this to the post job - https://github.com/openstack/tripleo-ansible/blob/master/zuul.d/layout.yaml#L7-L9 ? | 15:35 |
| * cloudnull doesn't know if that works | 15:36 | |
| *** dtantsur is now known as dtantsur|afk | 15:36 | |
| mordred | clarkb, corvus: 2019-07-11 15:22:59.831777 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/opendev/system-config/playbooks/zuul/run-base.yaml@master] | 15:37 |
| mordred | doh | 15:37 |
| mordred | we should probably increase the timeout there | 15:37 |
| *** philroche has joined #openstack-infra | 15:39 | |
| corvus | mordred: how long did the task run before that? | 15:40 |
| mordred | corvus: 20 minutes | 15:41 |
| mordred | corvus: 2019-07-11 15:02:39.732190 | was the line before it | 15:41 |
| *** tdasilva has joined #openstack-infra | 15:41 | |
| *** haleyb has quit IRC | 15:42 | |
| *** rfolco is now known as rfolco_brb | 15:43 | |
| mordred | 2019-07-11 15:02:39.732190 | was RUN START | 15:43 |
| *** haleyb has joined #openstack-infra | 15:43 | |
| mordred | gah. that's not right | 15:44 |
| mordred | 2019-07-11 14:55:08.770770 | was RUN START | 15:44 |
| corvus | mordred: i'm surprised 20m isn't enough, but i guess we should bump the timeout and see if it's 21 minutes :) | 15:46 |
| *** kjackal has quit IRC | 15:46 | |
| *** kjackal has joined #openstack-infra | 15:47 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Create tasks that spin up kubernetes and run the operator https://review.opendev.org/668029 | 15:49 |
| mordred | corvus: which timeout is hitting that? that's not the job run timeout, right? | 15:51 |
| *** tosky has quit IRC | 15:53 | |
| corvus | mordred: yeah, that's the job timeout | 15:53 |
| mordred | nod. | 15:53 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add some logging to repo creation https://review.opendev.org/670317 | 15:54 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run actual full project creation in gitea test https://review.opendev.org/670313 | 15:55 |
| *** lpetrut has quit IRC | 15:56 | |
| *** pcaruana has joined #openstack-infra | 15:57 | |
| AJaeger | cloudnull: you use publish-openstack-sphinx-docs ? Let me get back to you in a bit, I have an idea... | 15:58 |
| fungi | okay, finally have the morning's meetings behind me and now need to disappear for lunch errands. hopefully i'll have time to catch up on some priority reviews in another hour or so | 15:58 |
| fungi | bbiaw | 15:58 |
| cloudnull | AJaeger thank you ! | 16:00 |
| *** lucasagomes has quit IRC | 16:01 | |
| *** ginopc has quit IRC | 16:03 | |
| *** Fidde has joined #openstack-infra | 16:04 | |
| *** altlogbot_2 has joined #openstack-infra | 16:04 | |
| *** Fidde has quit IRC | 16:06 | |
| *** altlogbot_2 has quit IRC | 16:07 | |
| openstackgerrit | Pete Vander Giessen proposed openstack/project-config master: Add microstack project. https://review.opendev.org/668681 | 16:09 |
| AJaeger | cloudnull: You use the wrong publish job. You use for building tripleo-ansible-docs which is a variant of openstack-tox-docs. The corresponding publish job for openstack-tox-docs is *not* publish-openstack-sphinx-docs which you use, but publish-openstack-docs, see https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L274 | 16:10 |
| AJaeger | cloudnull: so, switch your publish job and you should be fine ;) | 16:11 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update description of some docs jobs https://review.opendev.org/670344 | 16:16 |
| AJaeger | cloudnull: I made the description of the jobs clearer now ^ | 16:16 |
| AJaeger | cloudnull: patch incoming | 16:18 |
| *** factor has quit IRC | 16:20 | |
| *** rpittau is now known as rpittau|adk | 16:20 | |
| *** rpittau|adk is now known as rpittau|afk | 16:20 | |
| *** Fidde has joined #openstack-infra | 16:22 | |
| AJaeger | cloudnull: https://review.opendev.org/670347 | 16:23 |
| *** Lucas_Gray has quit IRC | 16:24 | |
| * mordred goes to find lunch | 16:24 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Move zuul/ repos to Zuul tenant https://review.opendev.org/670348 | 16:25 |
| cloudnull | AJaeger awesome! | 16:26 |
| cloudnull | thank you very much | 16:26 |
| AJaeger | you're welcome! | 16:28 |
| * AJaeger grabs dinner now... | 16:28 | |
| *** mattw4 has joined #openstack-infra | 16:32 | |
| *** tesseract has quit IRC | 16:36 | |
| mwhahaha | http://mirror.dfw.rax.opendev.org/apt-puppetlabs seems bad (it exists in ovh) | 16:40 |
| *** altlogbot_1 has joined #openstack-infra | 16:44 | |
| *** altlogbot_1 has quit IRC | 16:49 | |
| clarkb | mwhahaha: I want to say ianw didn't add it to the new opendev mirrors because the upstream mirror does not exist anymore? | 16:50 |
| clarkb | (so we can't mirror it any longer) | 16:50 |
| mwhahaha | hrm | 16:50 |
| mwhahaha | thought it did | 16:50 |
| mwhahaha | though it was updated in ovh today? | 16:51 |
| *** ykarel|away has quit IRC | 16:51 | |
| mwhahaha | http://mirror.bhs1.ovh.openstack.org/apt-puppetlabs | 16:51 |
| clarkb | maybe upstream fixed the mirrors they broke | 16:51 |
| mwhahaha | i remember that happening a few weeks back | 16:51 |
| mwhahaha | https://release-archives.puppet.com/ | 16:52 |
| *** psachin has quit IRC | 16:52 | |
| mwhahaha | that thing | 16:52 |
| clarkb | we don't mirror from there | 16:52 |
| clarkb | so someone would have to update all the mirroring stuff to pull from the two locations and so on | 16:52 |
| mwhahaha | https://release-archives.puppet.com/ was the new one, apt.puppetlabs.com still exists tho | 16:53 |
| clarkb | right but apt.puppetlabs.com does't have PC1 anymore | 16:53 |
| clarkb | so it breaks our mirroring setup | 16:53 |
| clarkb | and I think we all around here threw our hands in the air and said fine we'll just stop using it then | 16:53 |
| clarkb | if someone wants to update the mirrors to do the correct thing thats great we just don't have much interest in it I think | 16:54 |
| mwhahaha | well then we need to stop it existing so it break cleanly 100% of the time instead of working everywhere but dfw :/ | 16:54 |
| clarkb | ya that is in progress (we are slowly replacing the mirrors) | 16:54 |
| mwhahaha | also i don't think we're using pc1 for apt | 16:55 |
| mwhahaha | meh i'll poke at it | 16:55 |
| clarkb | my recollection was that all the mirroring was broken beacuse we try to mirror PC1 | 16:56 |
| clarkb | and since it was broken anyway and no one stepped up to fix it ianw removed it from the new mirror deployments | 16:56 |
| clarkb | we can add it back to the mirror deployments but if we do that someone should fix the reprepro mirror config to avoid the brokeness of pc12 | 16:56 |
| mwhahaha | can you point me at that config | 16:57 |
| *** kjackal has quit IRC | 16:57 | |
| mwhahaha | also an ML post about the removal would have been handy | 16:58 |
| clarkb | it was broken | 16:58 |
| clarkb | by upstream.. | 16:58 |
| clarkb | its not like we turned it off, upstream turned it off so we turned it off | 16:58 |
| mwhahaha | except it was fixed upstream | 16:58 |
| mwhahaha | i remember this, they nuked apt and yum.puppetlabs.com | 16:58 |
| mwhahaha | then they added it back | 16:58 |
| clarkb | yes | 16:58 |
| clarkb | well we cleaned up after them before they fixed it then | 16:59 |
| mwhahaha | and it was working mirroring again | 16:59 |
| clarkb | because it wasn't working when we cleaned up our end | 16:59 |
| mwhahaha | that part is news to me because when we left it, it was working | 16:59 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/manifests/mirror_update.pp#L316-L323 | 17:00 |
| clarkb | is the reprepro config entrypoint | 17:00 |
| clarkb | mwhahaha: so we reached out to upstraem and said hey why did this happen? they said we warend everyone a month ago stop using our stuff also we don't know how to run ap ackage repository | 17:00 |
| * mwhahaha facepalms | 17:01 | |
| clarkb | then they said what would help we said well can you restore what was there? they said no we can make a new repo instead | 17:01 |
| clarkb | so we turned stuff off since they didn't care | 17:01 |
| clarkb | aiui the issue was they couldn't keep running their tooling to build the packages which is fine just leave it in place but that isn't what we got | 17:01 |
| clarkb | and we (opendev infra) can't really be expected to keep up with broken upstreams | 17:01 |
| *** ijw has joined #openstack-infra | 17:02 | |
| mwhahaha | which is fine but if you make a choice to turn something off, tell people | 17:02 |
| clarkb | except its a mirror | 17:02 |
| clarkb | and they turned it off | 17:02 |
| clarkb | its not like we made the decsions, upstream did... | 17:02 |
| mwhahaha | nevermind this is going in circles | 17:02 |
| mwhahaha | you just said choices were made to stop it | 17:02 |
| mwhahaha | in infra | 17:02 |
| clarkb | yes beacuse it was broken | 17:03 |
| mwhahaha | ok, so you made a choice. broken or not, tell people | 17:03 |
| clarkb | upstream decided to stop publishing those packages our response was to turn it off | 17:03 |
| mwhahaha | because we're using the puppet5 not pc1 | 17:03 |
| mwhahaha | so it would have still worked for us | 17:03 |
| mwhahaha | and it still works because it exists on some of the mirrors | 17:03 |
| openstackgerrit | Merged openstack/project-config master: Move zuul/ repos to Zuul tenant https://review.opendev.org/670348 | 17:04 |
| clarkb | yes I'm finding the other bit that was related to the turning off | 17:04 |
| *** yamamoto has joined #openstack-infra | 17:04 | |
| mwhahaha | 8 weeks ago we did https://review.opendev.org/#/c/659376/ | 17:04 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror/tasks/main.yaml#L79-L96 | 17:04 |
| clarkb | thats the list of things to expose via the mirror webserver from afs | 17:04 |
| *** d34dh0r53 has quit IRC | 17:05 | |
| clarkb | so the way to fix this is remove the non working bits from reprepro config, then add apt-puppetlabs to that list | 17:05 |
| mwhahaha | ok so is this just an issue with the cutover to the new code then where apt-puppetlabs wasn't migrated? | 17:05 |
| mwhahaha | because it exists on the old mirrors but not the new | 17:06 |
| clarkb | my memory is it was left off the cutover because the mirroring wasn't working | 17:06 |
| clarkb | if we want to support these repos on the new mirrors we should fix the reprepro config too | 17:06 |
| *** cloudnull has quit IRC | 17:06 | |
| *** witek has quit IRC | 17:07 | |
| clarkb | looking at the config I see what caused the confusion now. Only bionic would be working because it doesn't incluide pc1 already | 17:07 |
| clarkb | but the other platforms are not updating | 17:07 |
| clarkb | so ya either we only mirror for bionic or remove pc1 from the other platforms | 17:08 |
| mwhahaha | but this is bionic | 17:08 |
| mwhahaha | it's that apt-puppetlabs doesn't exist | 17:08 |
| mwhahaha | so it's working elsewhere | 17:08 |
| clarkb | I understand | 17:08 |
| mwhahaha | anyway i'll remove pc1 and readd apt-pupetlabs | 17:08 |
| clarkb | ONLY bionic is working with reprepro | 17:08 |
| clarkb | xenial and stretch are not | 17:08 |
| clarkb | and I think we were under the assumption that nothing was working | 17:08 |
| clarkb | we should either stop mirroring for xenial and stretch or remove PC1 from them, then reenable the afs symlink | 17:09 |
| mwhahaha | i'm hitting this on stable/rocky so i need to see what we're running for like stable/queens | 17:09 |
| mwhahaha | we might still need xenial but can get away with out pc1 | 17:09 |
| *** yamamoto has quit IRC | 17:10 | |
| openstackgerrit | Alex Schultz proposed opendev/system-config master: Add apt-puppetlabs mirroring back https://review.opendev.org/670356 | 17:12 |
| *** weshay is now known as weshay|rover | 17:12 | |
| *** armax has quit IRC | 17:12 | |
| *** sshnaidm|rover is now known as sshnaidm|off | 17:12 | |
| clarkb | infra-root I'm going to tag glean 3205ffa1e88778d9633c3e7bc1c257f7d400a86e as 1.14.1 does that look right to you all? | 17:13 |
| *** ralonsoh has quit IRC | 17:17 | |
| corvus | clarkb: lgtm | 17:20 |
| *** altlogbot_1 has joined #openstack-infra | 17:22 | |
| *** factor has joined #openstack-infra | 17:23 | |
| *** tdasilva has quit IRC | 17:24 | |
| clarkb | ok 1.14.1 pushed | 17:24 |
| *** altlogbot_1 has quit IRC | 17:25 | |
| *** rfolco_brb is now known as rfolco | 17:27 | |
| donnyd | clarkb: When do you think its a good time to scale FN back up? | 17:29 |
| *** altlogbot_3 has joined #openstack-infra | 17:30 | |
| clarkb | donnyd: I'm hoping a little later today. What I want to do is trigger centos and fedora image builds then make sure they are uploaded to fn then scale it up | 17:31 |
| donnyd | I'm in no big rush.. I am just doing more load testing right now anyways | 17:32 |
| *** altlogbot_3 has quit IRC | 17:35 | |
| *** altlogbot_3 has joined #openstack-infra | 17:36 | |
| *** weifan has joined #openstack-infra | 17:37 | |
| *** altlogbot_3 has quit IRC | 17:41 | |
| *** d34dh0r53 has joined #openstack-infra | 17:42 | |
| *** cloudnull has joined #openstack-infra | 17:42 | |
| *** altlogbot_2 has joined #openstack-infra | 17:42 | |
| *** tdasilva has joined #openstack-infra | 17:43 | |
| *** altlogbot_2 has quit IRC | 17:45 | |
| AJaeger | config-core, a simple update for job descriptions, please review https://review.opendev.org/670344 | 17:48 |
| *** electrofelix has quit IRC | 17:50 | |
| mordred | clarkb, corvus: 33:17 total elapsed time for the full project creation run | 17:56 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Fix typo in Zuul tenant config https://review.opendev.org/670359 | 17:56 |
| mordred | http://logs.openstack.org/13/670313/2/check/system-config-run-gitea/a55f75e/bridge.openstack.org/ara-report/result/7fd0e891-cf29-40bb-a431-1851e5135197/ | 17:56 |
| *** rlandy_ has joined #openstack-infra | 17:56 | |
| mordred | there's the log from what it dud | 17:57 |
| mordred | did | 17:57 |
| *** rlandy_ has quit IRC | 17:58 | |
| *** rlandy_ has joined #openstack-infra | 17:58 | |
| mordred | AJaeger: also for you - since I can see you wanting to look at (a smaller version) of such output in the future | 17:59 |
| *** rlandy has quit IRC | 17:59 | |
| clarkb | mordred: nice | 17:59 |
| mordred | clarkb: yah - I think 33 minutes is better than 4 hours | 18:00 |
| AJaeger | interesting | 18:00 |
| AJaeger | nice improvement | 18:00 |
| mordred | \o/ | 18:01 |
| fungi | is the idea to next substitute a representative project list for those test jobs? | 18:01 |
| fungi | (so that we don't do the full 33 minute rebuild) | 18:01 |
| *** diablo_rojo has joined #openstack-infra | 18:02 | |
| mordred | fungi: I think the opposite - I think the idea here was to run with the full data file - but I'll admit it's sunny and hot so my brain doesn't always work good | 18:02 |
| fungi | ahh, got it. so it does at least fit into a (lengthy) job run | 18:03 |
| *** rlandy_ is now known as rlandy | 18:05 | |
| mordred | yeah. and we really should only be runing that job on changes to the projects.yaml file and changes to the automation - so hopefully it's not too onerous | 18:05 |
| AJaeger | mordred: that list is to long to review - test it with a failure, as well, please | 18:06 |
| *** Fidde has quit IRC | 18:07 | |
| clarkb | I remember reading acomment about not optomizing for success but instead optomizing for failure and corvus asked for data. I got nerd sniped. Of the 276 changes merged in the last ~day we had 330 failed buildsets (I think this number includes merge failures currently though) and 785 successful buildsets. For a failure rate just under 30% | 18:08 |
| clarkb | The reason I went with merged changes is I want to exclude do not merge changes that are known to fail intentionally | 18:08 |
| clarkb | I'm going to figure out how to exclude the merge failures now | 18:09 |
| clarkb | then I'll put my script up somewhere and peopel can tell me what I did wrong :) | 18:09 |
| AJaeger | clarkb: it was zbr , see http://eavesdrop.openstack.org/irclogs/%23zuul/latest.log.html#t2019-07-11T14:18:52 | 18:10 |
| *** ociuhandu has joined #openstack-infra | 18:10 | |
| mordred | clarkb: were you looking at buildsets for merged changes just in gate or in check and gate? | 18:12 |
| clarkb | mordred: check and gate | 18:12 |
| clarkb | thats a good point though | 18:12 |
| clarkb | I should track both sets of numbers | 18:12 |
| * clarkb makes that update too | 18:12 | |
| mordred | because I could imagine one might choose to optimize check and gate slightly differently, since one expects gate to succeed, but might have a greater expectation of check failing | 18:13 |
| clarkb | ya | 18:13 |
| clarkb | I'll break it down by pipeline then post the script and results | 18:13 |
| clarkb | its not the quickest thing to run and test though so may be a bit :) | 18:13 |
| clarkb | I've enqueued centos-7 and fedora-29 image builds now that glean is up to date on pypi | 18:15 |
| AJaeger | cloudnull: docs are published! Check https://docs.openstack.org/tripleo-ansible/latest/genindex.html | 18:15 |
| mordred | clarkb: woot | 18:16 |
| cloudnull | AJaeger yup was just looking at that myself. | 18:18 |
| cloudnull | thanks again! | 18:18 |
| AJaeger | yw | 18:18 |
| openstackgerrit | Merged openstack/project-config master: Fix typo in Zuul tenant config https://review.opendev.org/670359 | 18:20 |
| cloudnull | anyone around mind giving https://review.opendev.org/#/c/666423/ a review ? | 18:23 |
| *** weifan has quit IRC | 18:23 | |
| fungi | bad news. i've already +2'd that one so you need someone else | 18:27 |
| *** diablo_rojo has quit IRC | 18:27 | |
| corvus | that's a clarkb change i think | 18:29 |
| clarkb | is that registry publicly available or does it require jobs have secrets to access images from there? | 18:30 |
| *** diablo_rojo has joined #openstack-infra | 18:30 | |
| clarkb | (wondering if proxy will work with that) | 18:30 |
| clarkb | I guess it should | 18:30 |
| clarkb | but would be exposed via http currently if so | 18:31 |
| clarkb | cloudnull: ^ | 18:32 |
| clarkb | everything else about it loosk fine to me | 18:32 |
| cloudnull | the hope is that it would work similar to the docker.io proxy bits | 18:33 |
| clarkb | cloudnull: ya but those are unauthenticated so its fine to speak http over them (ignoring verification of endpoint) | 18:33 |
| cloudnull | ++ | 18:34 |
| clarkb | but looking at this one it seems to have edgekey stuff whichi mplies to me there is auth involved? | 18:34 |
| mordred | corvus: does https://review.opendev.org/#/c/670313/ look right to you? | 18:34 |
| clarkb | if so you may not want to talk http to it | 18:34 |
| mordred | corvus: (I mean, it's green - but does the shape of it look like what you were thinking) | 18:34 |
| corvus | mordred: oh yeah, that looks great | 18:35 |
| mordred | cool | 18:35 |
| cloudnull | clarkb in my tests with the registry I'm not using any auth - https://review.opendev.org/#/c/665677 - though there are auth protected images there, so I guess if folks did auth that would not be so great. | 18:37 |
| zbr | clarkb: thanks for the numbers. | 18:37 |
| clarkb | corvus: zbr http://paste.openstack.org/show/754310/ as mordred points out the gate is way more reliable | 18:37 |
| cloudnull | however, i guess that's largely the same with the docker proxy too ? | 18:37 |
| mordred | cloudnull: you might, if you're not already thinking about this, want to think about work needed to get the speculative image building jobs to be able to use an alternate upstream registry(s) | 18:37 |
| mordred | corvus: I don't think you did anything to suppor that yet, right? | 18:37 |
| clarkb | the script will need tweaking and maybe a database if we want to run it over longer periods (so that we don't ahve to request data we already have) | 18:37 |
| zbr | true: i do expect the gate to pass, obviously. the more nuanced part would be about the check. | 18:37 |
| clarkb | cloudnull: gotcha so depends on what you request | 18:38 |
| clarkb | cloudnull: in that case I think its fine and if people need something with auth its at their own risk until we have https on those endpoints | 18:38 |
| mordred | ++ | 18:38 |
| cloudnull | is that something I can enable or is https on the endpoint something for a future release ? | 18:39 |
| clarkb | there is a typo in the last line of script output too should be total rate of failure not gate rate of failure | 18:39 |
| donnyd | Are there any good cloud-config scripts out there to run load testing against. I am scaling up to all 100 instances and I need to measure heat | 18:39 |
| corvus | mordred, cloudnull: well some of it pretty dockerhub focused, but i built the framework to support others. basically if the job/role has 'docker' in the name, it's docker-specific, but if it has 'container' in the name it's more generic. really need that second use case to flesh it out. | 18:39 |
| clarkb | cloudnull: only the opendev.org mirrors will have working https and we haven't added https endpoints for docker registries yet | 18:40 |
| cloudnull | ok | 18:40 |
| openstackgerrit | Merged opendev/system-config master: Add apt-puppetlabs mirroring back https://review.opendev.org/670356 | 18:40 |
| clarkb | cloudnull: so once we've replaced all the mirrors then add https endpoints we'll be good | 18:40 |
| corvus | mordred: it might be worth throwing some timestamps in the log so we can figure out if there's something we can optimize to improve on that 33m runtime. | 18:40 |
| clarkb | cloudnull: out of curiousity these images are not published to dockerhub because ? | 18:41 |
| cloudnull | they're redhat images . UBI is the universal base image for rhel. UBI8 is essentially RHEL8 | 18:41 |
| *** ociuhandu has quit IRC | 18:42 | |
| clarkb | right but they could live in both places right? | 18:42 |
| mordred | clarkb: yah, and because they don't have to be... rejecting the premise that things have to be on the docker-corp controller walled garden to be "real" | 18:42 |
| cloudnull | I guess they could | 18:42 |
| clarkb | (maybe I'm overthinking it to think of dockerhub as sort of the defacto central repo then you have other repos for the stuff that can't be public) | 18:42 |
| corvus | it's good to avoid a monoculture? :) | 18:42 |
| mordred | yeah | 18:42 |
| clarkb | sure, its just everyone I've seen that does that uploads to multiple places then | 18:42 |
| mordred | just like we liek not publishing things to github to help remind people that github is _not_ equiv to open source | 18:43 |
| cloudnull | I'm not the one publishing them, however, I could reachout to that team and see if we could mirror to dockerhub? | 18:43 |
| corvus | i'm happy to support diversification here | 18:43 |
| mordred | me too | 18:43 |
| clarkb | cloudnull: no its fine I'm mostly just curious | 18:43 |
| mordred | I think multiple registries is a good thing | 18:43 |
| clarkb | I know that other groups publish to dockerhub and quay etc to deal with this | 18:43 |
| clarkb | rather than a single repo over $there | 18:43 |
| clarkb | and was wondering if there was a reason for that | 18:43 |
| mordred | clarkb: habit, I believe | 18:43 |
| mordred | all the tools now support fully qualified image names | 18:44 |
| clarkb | iirc the big three are dockerhub quay and googles registry | 18:44 |
| clarkb | mordred: ah | 18:44 |
| fungi | granted, the python ecosystem basically rejected that idea in favor of a single package repo (and tools require you to go out of the way to fetch packages from anywhere else) | 18:44 |
| clarkb | so maybe that is what it was, not being able to specify baseurls | 18:44 |
| mordred | so instead of docker run ubuntu you can say docker run docker.io/ubuntu | 18:44 |
| clarkb | and now that isn't necessary | 18:44 |
| mordred | yeah | 18:44 |
| mordred | the ecosystem has caught up and it's actually kinda nice | 18:44 |
| mordred | I've been meaning to go back and make our FROM lines be more explicit, just on general principle | 18:45 |
| clarkb | still can't specify an ipv6 address literal though so its still trash :) | 18:45 |
| mordred | well - yeah - Cloud Native is behind the times like that | 18:45 |
| *** bnemec has quit IRC | 18:45 | |
| zbr | i like having multiple registries, is like keeping eggs in multiple baskets. | 18:45 |
| mordred | I did see a tweet about a k8s conformance test passing on an ipv6 only deploy recently - so progress is being made | 18:45 |
| mordred | zbr: ++ | 18:46 |
| clarkb | personally I;d be a lot happier with diversity of tooling if they all didn't fall behind docker on their silly decisions | 18:46 |
| cloudnull | clarkb ++ agree, IPv6 or death ! | 18:46 |
| mordred | clarkb: right? | 18:46 |
| clarkb | because the end result is still a monoculture | 18:46 |
| clarkb | fwiw I've never heard back on my bug or PR for that | 18:46 |
| corvus | a technical one, yes. but not a finiancial one. :) | 18:47 |
| clarkb | so I haven't bothered to update the PR to fix the one issue with it I know of | 18:47 |
| *** bnemec has joined #openstack-infra | 18:47 | |
| clarkb | disk on nb01 is full up again | 18:47 |
| clarkb | that explains why we aren't building new images :/ | 18:47 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Add Kubernetes Operator Functional Test Job https://review.opendev.org/668029 | 18:48 |
| clarkb | last time it was dib leaking stuff | 18:48 |
| clarkb | I'm going to stop the builder on nb01, clean its disk, and reboot it | 18:48 |
| clarkb | when ianw is back we should dig into what/why it is leaking | 18:49 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for DetachedInstanceError cinder-api bug 1834845 https://review.opendev.org/670367 | 18:52 |
| openstack | bug 1834845 in Cinder "DetachedInstanceError: Parent instance <VolumeAttachment at > is not bound to a Session" [Undecided,In progress] https://launchpad.net/bugs/1834845 - Assigned to Gorka Eguileor (gorka) | 18:52 |
| *** weifan has joined #openstack-infra | 18:56 | |
| mriedem | yikes http://status.openstack.org/elastic-recheck/data/integrated_gate.html | 18:57 |
| mriedem | Overall Categorization Rate: 20.7% | 18:57 |
| clarkb | ok have to disable the service then reboot becuse there are a bunch of mounts that leak and we have to delete stuff on those? | 18:58 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Temporarily shadow base-jobs in openstack-zuul-jobs https://review.opendev.org/670369 | 18:58 |
| corvus | AJaeger, fungi: ^ | 18:58 |
| clarkb | mriedem: ya I'm not sure anyone is really paying attention to that much these days. I keep pushing for people to start there when it comes to optomizing throughput | 18:58 |
| *** weifan has quit IRC | 19:01 | |
| fungi | that said, logstash is probably the largest resource consumer by far in our control plane (when you count the 20 worker servers and the massive elasticsearch cluster), so if nobody is actually using it we could save our donors a lot of cost by taking it down | 19:03 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Move zuul/zuul-base-jobs to a config-project https://review.opendev.org/670371 | 19:04 |
| *** bhavikdbavishi has quit IRC | 19:04 | |
| openstackgerrit | Brian Haley proposed opendev/irc-meetings master: Update networking OVN meeting https://review.opendev.org/670372 | 19:05 |
| corvus | fungi, clarkb, AJaeger: ^ can i get speedy reviews on https://review.opendev.org/670369 and its parent? | 19:05 |
| corvus | i'm probably going to ask for several of these to deal with fallout from the tenant move, sorry. | 19:05 |
| *** derekh has quit IRC | 19:05 | |
| *** bhavikdbavishi has joined #openstack-infra | 19:06 | |
| fungi | yup, already approved the parent and was looking at the second one | 19:06 |
| fungi | it's expected | 19:06 |
| fungi | first time we've really done something like this | 19:06 |
| clarkb | looking | 19:06 |
| clarkb | the dib cleanup delete are not what you would call fast | 19:07 |
| clarkb | nb02 still have 233GB free on /opt so I'll get nb01 up and happy before I bother doing nb02 | 19:08 |
| fungi | :( | 19:08 |
| clarkb | and now 278GB freeon nb01, enabling and starting services there again | 19:09 |
| *** rlandy is now known as rlandy|brb | 19:10 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Include nodesets in zuul/nodepool https://review.opendev.org/670373 | 19:10 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Remove zuul projects and jobs from the openstack tenant https://review.opendev.org/670374 | 19:10 |
| corvus | clarkb, fungi, AJaeger: ^ 2 more | 19:10 |
| corvus | topic: zuulectomy | 19:11 |
| clarkb | corvus: that says exclude nodesets? | 19:11 |
| clarkb | corvus: should that remain as an include? | 19:12 |
| *** aedc has joined #openstack-infra | 19:14 | |
| corvus | derp | 19:19 |
| clarkb | fungi: as far as cleaning up that service, I think it has proven to be incredibly valuable when used and we know that if we do use it we improve throughput significantly. I think we should probably prioritize advertising it over removing it at least initially. That said I agree it isn't a cheap system | 19:19 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Include nodesets in zuul/nodepool https://review.opendev.org/670373 | 19:20 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Remove zuul projects and jobs from the openstack tenant https://review.opendev.org/670374 | 19:20 |
| clarkb | fungi: in particular there seems to be this theme of maintainership around oepnstack and using the insights given by these tools to improve software reliability is sort of what I would expect from maintainers | 19:20 |
| fungi | yeah, i didn't mean to suggest it was worthless, just that if we're going to devote something like 25% of our control plane to operating it, people ought to be finding it comparatively valuable | 19:20 |
| openstackgerrit | Merged openstack/project-config master: Temporarily shadow base-jobs in openstack-zuul-jobs https://review.opendev.org/670369 | 19:22 |
| *** tosky has joined #openstack-infra | 19:26 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for DetachedInstanceError cinder-api bug 1834845 https://review.opendev.org/670367 | 19:26 |
| openstack | bug 1834845 in Cinder "DetachedInstanceError: Parent instance <VolumeAttachment at > is not bound to a Session" [Undecided,In progress] https://launchpad.net/bugs/1834845 - Assigned to Gorka Eguileor (gorka) | 19:26 |
| *** irclogbot_0 has joined #openstack-infra | 19:28 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Bringing Fort Nebula CI Cloud back online https://review.opendev.org/670378 | 19:30 |
| donnyd | That review is ready for whenever we want to bring FN back online | 19:31 |
| clarkb | donnyd: cool I think the nodepool builder disk problem means the centos and fedora rebuilds are now further behind in the queue so may be a little bit | 19:31 |
| clarkb | I'm keeping an eye on it | 19:31 |
| *** weifan has joined #openstack-infra | 19:32 | |
| donnyd | no biggie, just getting this queued up so we cut it loose when the time comes | 19:32 |
| clarkb | ++ | 19:33 |
| clarkb | I'm told lunch is ready for me. Back in a bit | 19:33 |
| *** irclogbot_0 has quit IRC | 19:33 | |
| openstackgerrit | Merged openstack/project-config master: Move zuul/zuul-base-jobs to a config-project https://review.opendev.org/670371 | 19:34 |
| *** weifan has quit IRC | 19:38 | |
| *** bnemec has quit IRC | 19:38 | |
| *** ijw has quit IRC | 19:38 | |
| *** altlogbot_2 has joined #openstack-infra | 19:38 | |
| *** altlogbot_2 has quit IRC | 19:41 | |
| *** kaiokmo has quit IRC | 19:43 | |
| *** jistr has quit IRC | 19:43 | |
| *** jistr has joined #openstack-infra | 19:45 | |
| *** bnemec has joined #openstack-infra | 19:46 | |
| *** bhavikdbavishi has quit IRC | 19:47 | |
| openstackgerrit | Merged openstack/project-config master: Include nodesets in zuul/nodepool https://review.opendev.org/670373 | 19:48 |
| openstackgerrit | Merged openstack/project-config master: Remove zuul projects and jobs from the openstack tenant https://review.opendev.org/670374 | 19:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run actual full project creation in gitea test https://review.opendev.org/670313 | 19:48 |
| *** weifan has joined #openstack-infra | 19:54 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Add some github projects to the Zuul tenant https://review.opendev.org/670380 | 19:57 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add nodesets https://review.opendev.org/670381 | 20:01 |
| *** rlandy|brb is now known as rlandy | 20:02 | |
| openstackgerrit | James E. Blair proposed openstack/openstack-zuul-jobs master: Remove some nodesets https://review.opendev.org/670382 | 20:02 |
| *** jistr has quit IRC | 20:03 | |
| fungi | corvus: 670381 should be testable once 670369 is applied, right? | 20:05 |
| *** jistr has joined #openstack-infra | 20:05 | |
| corvus | fungi: no, it's the same error is 670369 but for the opendev tenant itself, so i need a near-duplicate change for that; almost ready | 20:06 |
| *** pkopec_ has joined #openstack-infra | 20:06 | |
| fungi | ahh, okay | 20:06 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Add temporary shadow to opendev/base-jobs https://review.opendev.org/670384 | 20:06 |
| fungi | for some reason i thought shadowing the repo with the nodesets would solve it | 20:06 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add nodesets https://review.opendev.org/670381 | 20:06 |
| fungi | maybe i was just confused on which was "the repo with the nodesets" ;) | 20:06 |
| corvus | fungi: yeah, there are 2 | 20:07 |
| fungi | thanks for untangling this | 20:07 |
| corvus | fungi: looks like the last time we hit this error, we decided to solve it by duplicating the nodeset def in the opendev tenant | 20:07 |
| corvus | but i think that having at least the simple 1:1 label nodesets in opendev/base-jobs makes a lot of sense. way less copying of boilerplate stuff. | 20:08 |
| fungi | i agree with that goal | 20:08 |
| *** pkopec has quit IRC | 20:08 | |
| corvus | i left the multi-node and legacy nodesets in the openstack tenant since that's a bit more context-specific | 20:08 |
| *** kjackal has joined #openstack-infra | 20:08 | |
| fungi | wfm | 20:08 |
| clarkb | ya I agree re context specific multinode and legacy nodesets | 20:08 |
| clarkb | I'm cleaning up nb02 now | 20:09 |
| fungi | i'm about to start neglecting the computer and attacking my lawn before the next storm rolls through | 20:09 |
| *** pkopec_ has quit IRC | 20:09 | |
| fungi | but will pop ni for a few minutes at a time to stave off heat exhaustion | 20:10 |
| fungi | pop IN too | 20:10 |
| corvus | remote: https://review.opendev.org/670385 Remove nodesets | 20:10 |
| clarkb | careful the knights who say ni are defensive | 20:10 |
| *** pkopec has joined #openstack-infra | 20:11 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Remove temporary project shadows https://review.opendev.org/670387 | 20:12 |
| *** pkopec has quit IRC | 20:12 | |
| *** irclogbot_1 has joined #openstack-infra | 20:12 | |
| fungi | clarkb: i have my holy hand grenade at the ready | 20:12 |
| *** pkopec has joined #openstack-infra | 20:12 | |
| *** Lucas_Gray has joined #openstack-infra | 20:13 | |
| *** irclogbot_1 has quit IRC | 20:16 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Switch to opendev release jobs https://review.opendev.org/670388 | 20:16 |
| AJaeger | corvus: down to 8 errors in opendev - and one in openstack tenant currently | 20:18 |
| fungi | corvus: zuul dislikes 670388 because it sees that publish-zuul-docs is still referenced | 20:19 |
| * AJaeger waves good night | 20:19 | |
| fungi | have a good evening AJaeger! | 20:19 |
| corvus | AJaeger: o/ | 20:20 |
| corvus | fungi: thanks, i'll update it in a sec | 20:20 |
| AJaeger | thanks, happy lawn attacks, fungi ;) | 20:20 |
| * fungi readies the weapons of grass destruction | 20:20 | |
| *** ijw has joined #openstack-infra | 20:20 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add base-minimal job https://review.opendev.org/670390 | 20:23 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add base-minimal job https://review.opendev.org/670390 | 20:24 |
| *** iurygregory has quit IRC | 20:24 | |
| openstackgerrit | Merged openstack/project-config master: Add some github projects to the Zuul tenant https://review.opendev.org/670380 | 20:24 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Switch to opendev release/docs jobs https://review.opendev.org/670388 | 20:25 |
| *** ijw has quit IRC | 20:26 | |
| corvus | clarkb: +3 https://review.opendev.org/670384 ? | 20:26 |
| *** kjackal has quit IRC | 20:28 | |
| *** dklyle has joined #openstack-infra | 20:31 | |
| *** mattw4 has quit IRC | 20:31 | |
| *** pcaruana has quit IRC | 20:31 | |
| *** mattw4 has joined #openstack-infra | 20:31 | |
| *** igordc has joined #openstack-infra | 20:32 | |
| corvus | i self-approved that ^ to keep things moving | 20:35 |
| clarkb | sorry had to step away for a few back now | 20:35 |
| clarkb | mordred: do you think the gitea stuff is ready for proper reviews now? | 20:36 |
| *** pkopec has quit IRC | 20:38 | |
| mordred | clarkb: yes indeed - I pushed up a new rev to the last one to capture some timestamps in the log output - so we can maybe track down why it's taking 33 minutes | 20:39 |
| mordred | clarkb: but I think it's ready to go | 20:39 |
| openstackgerrit | Merged opendev/system-config master: Add proxy for registry.access.redhat https://review.opendev.org/666423 | 20:39 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Run actual full project creation in gitea test https://review.opendev.org/670313 | 20:39 |
| corvus | mordred: ^ typo in that, i just fixed it | 20:39 |
| corvus | mordred, clarkb: but i agree -- we've seen it work with the actual data, so we can merge whatever's green | 20:40 |
| clarkb | ok I'll try to dig into it properly nowish | 20:41 |
| corvus | you know, at some point, we should move all the x/ repos out of openstack | 20:42 |
| corvus | hopefully we'll be a bit more ready to do that after this exercise :) | 20:42 |
| mordred | corvus: won't that be neat! | 20:43 |
| fungi | i have a feeling there are lots of x/ repos which rely on openstackish jobs (openstack drivers, openstack deployment tools which aren't official parts of openstack, and so on) | 20:44 |
| fungi | so that may take a lot more untangling | 20:44 |
| corvus | yep. but it'll make things like "publish docs to an openstack.org domain" a lot easier to deal with | 20:45 |
| corvus | anyway, not my highest priority by a long shot | 20:45 |
| openstackgerrit | Merged openstack/project-config master: Add temporary shadow to opendev/base-jobs https://review.opendev.org/670384 | 20:46 |
| fungi | sure. also migrating repos into the openstack tenant adds an extra step vs just a repo rename, but not a huge additional barrier i suppose | 20:47 |
| *** whoami-rajat has quit IRC | 20:48 | |
| clarkb | mordred: left a comment on the first one. One bit of it is a little odd to me. Can you take a look? | 20:49 |
| clarkb | then I also had a question of whether or not we need to update our project rename scripts/playbooks | 20:49 |
| clarkb | seems like we should need to do that too | 20:49 |
| corvus | clarkb: i think owner_id is "the id of the team named 'Owner'" ? | 20:50 |
| clarkb | oh I see and that has to be an id not a name | 20:51 |
| corvus | yeah, so self.put('/api/v1/teams/{owner_id}/members/gerrit'...) means add gerrit to the 'Owners' team | 20:51 |
| *** ijw has joined #openstack-infra | 20:51 | |
| clarkb | right ok so the bit that is confusing me then is what creates the Owners team and gerrit user | 20:51 |
| clarkb | (at least when it comes to my concern about bootstrapping) | 20:52 |
| corvus | clarkb: i think the owners team is created automatically when each org is created; and the gerrit user is created as part of the bootstrapping playbook elsewhere, i can dig it up | 20:52 |
| clarkb | ok thanks | 20:52 |
| corvus | (there's an owners team for each org, which is why that's a lookup there) | 20:52 |
| clarkb | and its managed internally to gitea so ids aren't deterministic I bet | 20:53 |
| clarkb | all makes sense to me now | 20:53 |
| corvus | clarkb: gerrit user bootstrapping starts here: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gitea/tasks/main.yaml#L66 | 20:53 |
| clarkb | ok switched to a +2. I didn't approve because I'm going to have to do dadops this afternoon so can't commit to watching that and the dib/glean/fn stuff | 20:54 |
| *** ijw has quit IRC | 20:56 | |
| *** hamzy has quit IRC | 20:56 | |
| *** eharney has quit IRC | 20:57 | |
| mordred | clarkb: what corvus said (I mostly just copied the logic corvus already wrote) | 20:57 |
| *** altlogbot_0 has joined #openstack-infra | 21:00 | |
| *** xek has quit IRC | 21:04 | |
| *** altlogbot_0 has quit IRC | 21:05 | |
| *** dklyle has quit IRC | 21:10 | |
| *** _erlon_ has quit IRC | 21:12 | |
| *** derekh has joined #openstack-infra | 21:13 | |
| corvus | oops, shadow doesn't affect nodesets, just jobs. | 21:22 |
| *** irclogbot_1 has joined #openstack-infra | 21:22 | |
| corvus | i think we should force-merge 670381 then we should be able to merge the others normally | 21:23 |
| clarkb | will that break zuul's running config? I guess it should just ignore the broken bits until it gets to the part that cleans things up? | 21:23 |
| corvus | clarkb: yeah, it'll switch to the node defs in the base-jobs repo and throw errors about the duplicate defs, but the second changes will clean those up | 21:25 |
| clarkb | seems like a reasonable plan to me then | 21:25 |
| *** irclogbot_1 has quit IRC | 21:26 | |
| *** mriedem has quit IRC | 21:26 | |
| openstackgerrit | Merged opendev/base-jobs master: Add nodesets https://review.opendev.org/670381 | 21:26 |
| donnyd | clarkb: dadops... LOL | 21:26 |
| corvus | waiting for the alarm bell to show up in the openstack tenant | 21:27 |
| clarkb | donnyd: depending on how quickly (slowly really) these images builds ahead of centos and fedora go we may have to wait a bit | 21:28 |
| clarkb | oh looks like fedora-28 just started | 21:28 |
| *** irclogbot_3 has joined #openstack-infra | 21:28 | |
| donnyd | by a bit do you mean tomorrow or next week | 21:28 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: [WIP] Verify Operator Pod Running https://review.opendev.org/670395 | 21:29 |
| clarkb | donnyd: tomorrow | 21:29 |
| donnyd | oh I am ok with that | 21:29 |
| clarkb | I think we average just over an hour per image and we can build 2 images at once | 21:29 |
| clarkb | so depends on what the queue order is (don't think there is an easy way for me to check that?) | 21:29 |
| fungi | i always though a bit was a sixteenth of a nybble | 21:29 |
| clarkb | but my next step was to manually boot those images and make sure they work | 21:29 |
| fungi | er, thought | 21:29 |
| clarkb | then we can merge your change to turn the cloud back on in nodepool | 21:29 |
| clarkb | fungi: doesn't it depend on your word size? | 21:30 |
| donnyd | Ok sounds good to me | 21:30 |
| donnyd | Are there any scripts out there I can use to test in the mean time. I am doing full scale testing at 100 instances and need to measure my new air handler | 21:31 |
| donnyd | which puts out roughly half of the advertised air | 21:31 |
| clarkb | donnyd: other than throwing a bunch of load at a cloud via nodepool I odn't know that we have any. I think rally is was the canned version of the thing | 21:32 |
| *** irclogbot_3 has quit IRC | 21:32 | |
| clarkb | not sure how difficult it is to use rally though | 21:32 |
| donnyd | ok, I will look into it | 21:32 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Include nodesets from base-jobs https://review.opendev.org/670397 | 21:33 |
| corvus | clarkb, fungi, mordred: ^ doh. | 21:34 |
| clarkb | corvus: +2 | 21:34 |
| corvus | (everything is still okay; not a firedrill) | 21:34 |
| fungi | donnyd: yeah, if you want a load test, we've got one on the way ;) | 21:40 |
| *** Lucas_Gray has quit IRC | 21:43 | |
| *** ekultails has quit IRC | 21:45 | |
| clarkb | centos-7 is building now | 21:54 |
| openstackgerrit | Merged openstack/project-config master: Include nodesets from base-jobs https://review.opendev.org/670397 | 21:54 |
| corvus | waiting for an ansible run with that ^ | 21:54 |
| clarkb | the new gentoo image is uploading now too actually so that will be the first one I can test | 21:59 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: Add dstat graph roles https://review.opendev.org/670402 | 22:01 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: Add dstat graph roles https://review.opendev.org/670402 | 22:03 |
| *** hamzy has joined #openstack-infra | 22:07 | |
| clarkb | gentoo image booted and tested and is working \o/ | 22:10 |
| donnyd | are all the images being loaded now? | 22:11 |
| clarkb | no they are being uploaded as they are built | 22:11 |
| clarkb | centos-7 is building now. Gentoo built before it and was uploaded. So I could test gentoo | 22:12 |
| donnyd | the debian stretch image is stuck in queued for some reason | 22:14 |
| donnyd | it will get uploaded on the next cycle... I removed all external storages from glance, and the others seem to be uploading fine | 22:15 |
| donnyd | I kilt it, it was just stuck somewhere in not gonna upload land | 22:16 |
| fungi | that's a very large prefecture in openstackistan | 22:16 |
| clarkb | ya it should try again | 22:16 |
| donnyd | does it ever pull the old images? | 22:16 |
| clarkb | donnyd: no | 22:16 |
| fungi | it only uses the newest | 22:17 |
| donnyd | So do I manually prune those? | 22:17 |
| fungi | we sometimes manually delete the newest so we can roll back to the previous image | 22:17 |
| *** altlogbot_3 has joined #openstack-infra | 22:17 | |
| fungi | nodepool auto-culls all images older than the two most recent | 22:17 |
| *** altlogbot_3 has quit IRC | 22:17 | |
| donnyd | oh, so it does delete the old images | 22:18 |
| fungi | so at worst there will be up to 3 copies while uploads are in progress | 22:18 |
| clarkb | unless it happens to leak them which Ithink may happen if the uploads fail in ways the sdk can't handle | 22:18 |
| fungi | (but in reality more because openstack and leaks and sometimes stuff doesn't really delete when it's supposed to... clouds) | 22:18 |
| donnyd | its fine to have 3 or 4, but when it gets around 20 the glance node will need some more storages | 22:18 |
| *** altlogbot_0 has joined #openstack-infra | 22:18 | |
| *** irclogbot_2 has joined #openstack-infra | 22:18 | |
| donnyd | Ok cool. I of course will always ask before pruning old images | 22:19 |
| donnyd | just incase | 22:19 |
| fungi | we've got some stuck in vexxhost at the moment because they're the bases for bfv instances which went away but cinder never got the memo | 22:19 |
| donnyd | If I could get NFS to run correct with glance, then it wouldn't matter.. | 22:20 |
| donnyd | but for some reason it wants to hate on me | 22:20 |
| fungi | so cinder glance refuses to delete the images because cinder volumes cow from them, cinder refuses to delete the volumes because it thinks they're attached to nova instances, nova says waitwhut? | 22:20 |
| *** sthussey has quit IRC | 22:20 | |
| fungi | no instances here by that name, sorry | 22:21 |
| fungi | and thus we need a cloud admin to go make things sane for us again | 22:21 |
| *** irclogbot_2 has quit IRC | 22:23 | |
| *** altlogbot_0 has quit IRC | 22:23 | |
| *** rfolco has quit IRC | 22:24 | |
| *** rfolco has joined #openstack-infra | 22:24 | |
| corvus | okay, *there* are the config errors i was expecting :) | 22:25 |
| *** jtomasek has quit IRC | 22:25 | |
| clarkb | http://paste.openstack.org/show/754322/ a months worth of check and gate fail rate data. That is from 2019-06-12 to 2019-07-11 looking only at merged changes | 22:26 |
| clarkb | it is very similar to my earlier days worth | 22:26 |
| clarkb | the key finding I think is that we don't fail the majority of the time | 22:27 |
| clarkb | (we do fail more than I would like but eh) | 22:27 |
| *** rh-jelabarre has quit IRC | 22:27 | |
| *** betherly has joined #openstack-infra | 22:29 | |
| fungi | tell developers to make fewer mistakes! | 22:30 |
| fungi | (or to be happy that the ci system is catching them?) | 22:30 |
| fungi | granted, the ci system in many cases is catching someone else's mistake which is already merged but only surfaces some of the time | 22:31 |
| clarkb | it is also possible that by only looking at merged changes I've introduced a bias of good changes too | 22:32 |
| clarkb | however I'm not sure how else I can easily filter out the noise we know is bad | 22:32 |
| corvus | fungi, clarkb: can you do a pass through topic:zuulectomy and leave +2s ? i'm expecting things to start going green now; i can do the +Ws | 22:33 |
| clarkb | corvus: yup | 22:33 |
| fungi | clarkb: if changes never merge, do we care how much they fail? | 22:33 |
| corvus | (i have several rechecks pending) | 22:33 |
| clarkb | fungi: good point | 22:33 |
| fungi | corvus: i'll run back through them, sure | 22:33 |
| *** betherly has quit IRC | 22:34 | |
| fungi | clarkb: we know that there is some steady flow of merging changes, so unless the flow stops or slows, the changes which haven't merged are representative of changes which will never merge, statistically | 22:34 |
| clarkb | corvus: is https://review.opendev.org/#/c/670391/1/zuul.d/jobs.yaml missing a parent specification? | 22:35 |
| clarkb | corvus: the playbook it runs in post checks zuul_success but I don't see where the run playbook comes from | 22:35 |
| corvus | clarkb: it is the parent job for zuul-website-publish, defined in the zuul-website repo | 22:38 |
| clarkb | oh the relationship is the other direction? | 22:38 |
| clarkb | oh so it can hold the afs secret got it | 22:38 |
| corvus | yep | 22:39 |
| corvus | we can totally change the names (but let's do that later) | 22:39 |
| clarkb | and for https://review.opendev.org/#/c/670390/2/zuul.d/jobs.yaml I don't see playbooks/base-minimal in base-jobs ? | 22:39 |
| clarkb | could be in a chagne that is in flight right now I suppose | 22:39 |
| corvus | this will slot right in without a change to zuul-website | 22:39 |
| corvus | clarkb: that's an error, i'll fix | 22:39 |
| clarkb | k | 22:40 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add base-minimal job https://review.opendev.org/670390 | 22:41 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Include base-jobs nodesets in more tenants https://review.opendev.org/670406 | 22:43 |
| corvus | clarkb, fungi: ^ sorry, one more doh change | 22:43 |
| clarkb | I think all but that last one are approved now | 22:44 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove some nodesets https://review.opendev.org/670382 | 22:44 |
| clarkb | I think we may have already leaked a small number images in fn | 22:50 |
| clarkb | I'm going to work on deleting them | 22:50 |
| *** tkajinam has joined #openstack-infra | 22:52 | |
| fungi | when mnaser gets home we should try to get him (or someone) to free up the ~30 hung cinder volumes in sjc1 | 22:53 |
| fungi | but they've been there for several weeks at this point, so a few more days probably won't hurt | 22:54 |
| *** mattw4 has quit IRC | 22:54 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add dstat graph roles https://review.opendev.org/670402 | 22:55 |
| *** slaweq has quit IRC | 22:56 | |
| *** mattw4 has joined #openstack-infra | 22:56 | |
| *** altlogbot_3 has joined #openstack-infra | 22:56 | |
| openstackgerrit | Merged openstack/project-config master: Remove temporary project shadows https://review.opendev.org/670387 | 22:58 |
| clarkb | waiting on centos7 upload to finish now. Going to step out for a bit | 22:59 |
| *** altlogbot_3 has quit IRC | 22:59 | |
| *** irclogbot_3 has joined #openstack-infra | 23:00 | |
| *** mattw4 has quit IRC | 23:03 | |
| *** mattw4 has joined #openstack-infra | 23:03 | |
| *** irclogbot_3 has quit IRC | 23:06 | |
| donnyd | clarkb: is this the last one? If yes, do we want to merge the change to bring FN back online tonight or just wait till tomorrow? | 23:07 |
| *** weifan has quit IRC | 23:07 | |
| *** tosky has quit IRC | 23:07 | |
| *** yamamoto has joined #openstack-infra | 23:07 | |
| *** slaweq has joined #openstack-infra | 23:08 | |
| openstackgerrit | Merged openstack/project-config master: Include base-jobs nodesets in more tenants https://review.opendev.org/670406 | 23:08 |
| *** irclogbot_3 has joined #openstack-infra | 23:08 | |
| *** dchen has joined #openstack-infra | 23:10 | |
| *** slaweq has quit IRC | 23:12 | |
| *** yamamoto has quit IRC | 23:12 | |
| *** rcernin has joined #openstack-infra | 23:13 | |
| fungi | i'm around to approve that change whenever it's time | 23:13 |
| fungi | lawn is finally back under control | 23:13 |
| *** irclogbot_3 has quit IRC | 23:16 | |
| *** goldyfruit has quit IRC | 23:16 | |
| donnyd | the centos image is in | 23:17 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add base-minimal job https://review.opendev.org/670390 | 23:18 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Switch to opendev release/docs jobs https://review.opendev.org/670388 | 23:20 |
| *** betherly has joined #openstack-infra | 23:23 | |
| *** betherly has quit IRC | 23:28 | |
| *** altlogbot_3 has joined #openstack-infra | 23:29 | |
| *** altlogbot_3 has quit IRC | 23:30 | |
| clarkb | donnyd: probably wait for tomorrow since I've got to juggle kids this evening | 23:30 |
| clarkb | test centos node booting now | 23:30 |
| *** dchen has quit IRC | 23:31 | |
| donnyd | Sounds great to me | 23:31 |
| clarkb | I did just double check the mirror node responds on port 80 | 23:31 |
| clarkb | so that side of things is good to go | 23:31 |
| *** dchen has joined #openstack-infra | 23:32 | |
| *** dchen has quit IRC | 23:32 | |
| *** bobh has joined #openstack-infra | 23:32 | |
| *** ijw has joined #openstack-infra | 23:34 | |
| clarkb | ipv6 works on test centos node | 23:35 |
| clarkb | if ianw was around I'd probably turn it on this evening and he could turn it off if it had sads | 23:36 |
| clarkb | but we can do it first thing tomorrow morning | 23:36 |
| clarkb | AJaeger: fungi: maybe you can +A https://review.opendev.org/#/c/670378/1 sometime early tomorrow? | 23:36 |
| *** bobh has quit IRC | 23:37 | |
| corvus | clarkb: can you give https://review.opendev.org/670390 another look? | 23:37 |
| clarkb | corvus: done | 23:37 |
| clarkb | I got the other one but missed that one | 23:37 |
| clarkb | I'm going to delete my two fn test nodes now | 23:38 |
| *** dchen has joined #openstack-infra | 23:40 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Add required projects to zuul tenant https://review.opendev.org/670413 | 23:41 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Switch to zuul docs jobs https://review.opendev.org/670415 | 23:43 |
| corvus | i *think* that's it :) | 23:43 |
| *** betherly has joined #openstack-infra | 23:44 | |
| openstackgerrit | Merged opendev/base-jobs master: Add base-minimal job https://review.opendev.org/670390 | 23:45 |
| *** weifan has joined #openstack-infra | 23:45 | |
| corvus | oh, wow. i think 670415 may have exposed a bug with the new project config diff thing -- i think it's trying to freeze the old version of the job graph and failing, even though the new version would work | 23:46 |
| corvus | we may need to force-merge that and i can add a TODO to add a test case and fix that | 23:46 |
| *** dchen has quit IRC | 23:46 | |
| corvus | (none of the other repos had a check template involved) | 23:47 |
| corvus | clarkb: can you give 670413 and 670415 a look over, and i'll force-merge 415? | 23:47 |
| *** dchen has joined #openstack-infra | 23:49 | |
| *** betherly has quit IRC | 23:49 | |
| *** goldyfruit has joined #openstack-infra | 23:50 | |
| *** weifan has quit IRC | 23:50 | |
| clarkb | looking | 23:51 |
| clarkb | +2 | 23:52 |
| clarkb | I guess its mad about the old side of that diff? | 23:52 |
| corvus | yeah | 23:52 |
| fungi | reviewed both, lgtm | 23:52 |
| fungi | forcing 670415 makes sense, unfortunately | 23:53 |
| corvus | i think we just lucked out and didn't have any broken check pipelines until this one | 23:54 |
| openstackgerrit | Merged zuul/zuul-jobs master: Switch to zuul docs jobs https://review.opendev.org/670415 | 23:54 |
| clarkb | fungi: did you catch my note about approving the fn reenablement tomorrow at the starto f your day? | 23:56 |
| clarkb | fungi: I have tested both centos and gentoo images boot with ivp6 connectivity there now | 23:56 |
| fungi | yup | 23:56 |
| fungi | i could do it now, in theory | 23:56 |
| fungi | i'll be around the computer for at least a few hours | 23:56 |
| fungi | but sure, tomorrow is better if there's no rush | 23:56 |
| clarkb | ya but then we have to keep an eye on it if jobs go sideways. I know I can't do that as I have to go prep dinner nowish | 23:56 |
| *** mattw4 has quit IRC | 23:56 | |
| clarkb | and you shouldn't stay up all night over that :) | 23:56 |
| fungi | oh, i wouldn | 23:56 |
| fungi | 't unless it blew up | 23:57 |
| fungi | i'd just be hacking on personal projects or playing video games | 23:57 |
| clarkb | I think video games are in my post dinner future | 23:57 |
| fungi | or finishing this tc resolution i said i'd draft "today" | 23:57 |
| fungi | technically today is over in 2 minutes, so if it's tomorrow anyway... | 23:58 |
| *** dmsimard4 has joined #openstack-infra | 23:58 | |
| *** dmsimard has quit IRC | 23:58 | |
| fungi | and i have these sbcs i want to upgrade to most recent openbsd | 23:59 |
| fungi | decisions, decisions | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!