| *** smarcet has quit IRC | 00:00 | |
| *** eernst has quit IRC | 00:01 | |
| *** eernst has joined #openstack-infra | 00:02 | |
| *** Weifan has quit IRC | 00:05 | |
| *** Weifan has joined #openstack-infra | 00:05 | |
| *** Weifan has quit IRC | 00:05 | |
| *** Weifan has joined #openstack-infra | 00:06 | |
| *** Weifan has quit IRC | 00:06 | |
| *** Weifan has joined #openstack-infra | 00:06 | |
| *** eernst has quit IRC | 00:07 | |
| *** Weifan has quit IRC | 00:07 | |
| *** Weifan has joined #openstack-infra | 00:07 | |
| *** Weifan has quit IRC | 00:08 | |
| *** Weifan has joined #openstack-infra | 00:08 | |
| *** Weifan has quit IRC | 00:09 | |
| *** Weifan has joined #openstack-infra | 00:09 | |
| *** Weifan has quit IRC | 00:09 | |
| *** dklyle has quit IRC | 00:11 | |
| *** zbr_ has joined #openstack-infra | 00:12 | |
| *** zbr has quit IRC | 00:14 | |
| *** eernst has joined #openstack-infra | 00:15 | |
| *** Weifan has joined #openstack-infra | 00:19 | |
| *** eernst has quit IRC | 00:20 | |
| *** Weifan has quit IRC | 00:20 | |
| *** Weifan has joined #openstack-infra | 00:21 | |
| *** larainema has joined #openstack-infra | 00:22 | |
| *** eernst has joined #openstack-infra | 00:23 | |
| *** Weifan has quit IRC | 00:26 | |
| *** eernst has quit IRC | 00:27 | |
| *** Goneri has joined #openstack-infra | 00:33 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: letsencrypt: use a fake CA for self-signed testing certs https://review.opendev.org/658930 | 00:36 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] initial attempts at an ansible-based mirror https://review.opendev.org/658281 | 00:36 |
| *** Goneri has quit IRC | 00:40 | |
| *** psyton has joined #openstack-infra | 00:49 | |
| *** Weifan has joined #openstack-infra | 00:49 | |
| *** psyton has quit IRC | 00:50 | |
| *** Weifan has quit IRC | 00:52 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] initial attempts at an ansible-based mirror https://review.opendev.org/658281 | 00:54 |
| *** jcoufal has joined #openstack-infra | 00:56 | |
| *** jcoufal has quit IRC | 00:57 | |
| *** psyton has joined #openstack-infra | 01:01 | |
| *** lseki has quit IRC | 01:12 | |
| *** sarob has joined #openstack-infra | 01:13 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Try running docker-compose as zuul user https://review.opendev.org/658941 | 01:14 |
| *** diablo_rojo has quit IRC | 01:27 | |
| *** dklyle has joined #openstack-infra | 01:37 | |
| *** oyrogerg has joined #openstack-infra | 01:37 | |
| *** sarob has quit IRC | 01:45 | |
| *** hongbin has joined #openstack-infra | 01:49 | |
| *** jcoufal has joined #openstack-infra | 01:50 | |
| *** ykarel|away has joined #openstack-infra | 01:51 | |
| *** dklyle has quit IRC | 01:54 | |
| *** nicolasbock has quit IRC | 01:55 | |
| *** apetrich has quit IRC | 01:57 | |
| *** jcoufal has quit IRC | 02:02 | |
| *** smarcet has joined #openstack-infra | 02:06 | |
| *** smarcet has left #openstack-infra | 02:06 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: zuul-quick-start: run docker-compose as regular user https://review.opendev.org/658738 | 02:12 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: DNM: show zuul-quick-start doesn't test speculative images https://review.opendev.org/658935 | 02:13 |
| *** zhangfei has joined #openstack-infra | 02:14 | |
| *** sarob has joined #openstack-infra | 02:21 | |
| *** ykarel|away has quit IRC | 02:22 | |
| *** rosmaita has left #openstack-infra | 02:24 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:28 | |
| *** verdurin has quit IRC | 02:46 | |
| *** auristor has quit IRC | 02:46 | |
| *** bhavikdbavishi has quit IRC | 02:51 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:52 | |
| *** sarob has quit IRC | 02:54 | |
| *** whoami-rajat has joined #openstack-infra | 02:59 | |
| *** rlandy|bbl is now known as rlandy | 03:02 | |
| *** auristor has joined #openstack-infra | 03:03 | |
| *** bhavikdbavishi has quit IRC | 03:11 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: use-buildset-registry: add registry-1.docker.io https://review.opendev.org/658951 | 03:13 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: DNM: show zuul-quick-start doesn't test speculative images https://review.opendev.org/658935 | 03:15 |
| *** bhavikdbavishi has joined #openstack-infra | 03:17 | |
| *** ijw has joined #openstack-infra | 03:18 | |
| *** ykarel|away has joined #openstack-infra | 03:18 | |
| *** imacdonn has quit IRC | 03:24 | |
| *** sarob has joined #openstack-infra | 03:34 | |
| *** imacdonn has joined #openstack-infra | 03:39 | |
| *** igordc has quit IRC | 03:44 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: DNM: show zuul-quick-start doesn't test speculative images https://review.opendev.org/658935 | 03:47 |
| *** ykarel|away has quit IRC | 03:50 | |
| *** udesale has joined #openstack-infra | 03:52 | |
| *** rlandy has quit IRC | 03:53 | |
| *** ricolin has joined #openstack-infra | 03:58 | |
| *** diablo_rojo has joined #openstack-infra | 04:06 | |
| *** sarob has quit IRC | 04:08 | |
| AJaeger | ianw, to get rid of broken legacy job, could you review https://review.opendev.org/658684 https://review.opendev.org/658696 and https://review.opendev.org/658685 , please? | 04:16 |
| *** yamamoto has joined #openstack-infra | 04:20 | |
| AJaeger | thanks! | 04:22 |
| ianw | heh, searching for galaxy categories doesn't help much, but yeah, that's the switched name :) | 04:24 |
| *** pcaruana|afk| has joined #openstack-infra | 04:25 | |
| *** hongbin has quit IRC | 04:29 | |
| *** ykarel|away has joined #openstack-infra | 04:32 | |
| *** pcaruana|afk| has quit IRC | 04:35 | |
| *** ykarel|away is now known as ykarel | 04:40 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] initial attempts at an ansible-based mirror https://review.opendev.org/658281 | 04:53 |
| AJaeger | ianw: ansible-lint told me about that change... | 04:53 |
| AJaeger | config-core, two open reviews for new repos: https://review.opendev.org/656657 and https://review.opendev.org/657862 | 04:55 |
| *** admcleod has quit IRC | 04:58 | |
| *** janki has joined #openstack-infra | 05:01 | |
| *** yamamoto has quit IRC | 05:11 | |
| *** yamamoto has joined #openstack-infra | 05:13 | |
| *** kjackal has joined #openstack-infra | 05:18 | |
| *** electrofelix has quit IRC | 05:26 | |
| *** dpawlik has joined #openstack-infra | 05:28 | |
| *** jtomasek has joined #openstack-infra | 05:31 | |
| *** ijw has quit IRC | 05:31 | |
| *** ykarel_ has joined #openstack-infra | 05:40 | |
| *** udesale has quit IRC | 05:43 | |
| *** ykarel has quit IRC | 05:43 | |
| *** udesale has joined #openstack-infra | 05:43 | |
| *** dangtrinhnt_ has joined #openstack-infra | 05:49 | |
| *** dangtrinhnt has quit IRC | 05:50 | |
| openstackgerrit | Merged opendev/ansible-role-puppet master: Switch to linters job https://review.opendev.org/658684 | 05:51 |
| openstackgerrit | Merged opendev/ansible-role-puppet master: Fix ansible-lint https://review.opendev.org/658696 | 05:51 |
| *** dangtrinhnt_ has quit IRC | 05:53 | |
| *** kjackal has quit IRC | 05:54 | |
| *** kjackal has joined #openstack-infra | 05:56 | |
| *** dangtrinhnt has joined #openstack-infra | 06:05 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] initial attempts at an ansible-based mirror https://review.opendev.org/658281 | 06:20 |
| *** xek has joined #openstack-infra | 06:35 | |
| *** quiquell|off is now known as quiquell | 06:36 | |
| *** sarob has joined #openstack-infra | 06:36 | |
| *** ykarel_ has quit IRC | 06:44 | |
| *** ykarel_ has joined #openstack-infra | 06:48 | |
| *** slaweq has joined #openstack-infra | 06:52 | |
| *** udesale has quit IRC | 06:52 | |
| *** udesale has joined #openstack-infra | 06:52 | |
| *** ykarel_ is now known as ykarel | 06:55 | |
| *** dpawlik has quit IRC | 06:57 | |
| *** diablo_rojo has quit IRC | 06:58 | |
| *** yboaron has joined #openstack-infra | 06:58 | |
| *** yboaron_ has joined #openstack-infra | 07:00 | |
| *** apetrich has joined #openstack-infra | 07:01 | |
| *** yboaron has quit IRC | 07:03 | |
| *** slaweq has quit IRC | 07:03 | |
| *** tesseract has joined #openstack-infra | 07:05 | |
| *** rcernin has quit IRC | 07:06 | |
| *** ginopc has joined #openstack-infra | 07:07 | |
| *** sarob has quit IRC | 07:10 | |
| *** pcaruana has joined #openstack-infra | 07:12 | |
| *** jesusaur has quit IRC | 07:13 | |
| *** bhavikdbavishi has quit IRC | 07:14 | |
| *** admcleod has joined #openstack-infra | 07:14 | |
| *** serhatd has joined #openstack-infra | 07:24 | |
| *** dpawlik has joined #openstack-infra | 07:28 | |
| *** snapiri has joined #openstack-infra | 07:29 | |
| *** rpittau|afk is now known as rpittau | 07:32 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] initial attempts at an ansible-based mirror https://review.opendev.org/658281 | 07:33 |
| *** ykarel is now known as ykarel|lunch | 07:34 | |
| *** yboaron_ has quit IRC | 07:35 | |
| *** yboaron_ has joined #openstack-infra | 07:35 | |
| *** pgaxatte has joined #openstack-infra | 07:36 | |
| *** jpich has joined #openstack-infra | 07:46 | |
| *** amoralej|off is now known as amoralej | 07:49 | |
| *** dtantsur|afk is now known as dtantsur | 07:54 | |
| *** ralonsoh has joined #openstack-infra | 07:58 | |
| *** jpena|off is now known as jpena | 07:59 | |
| *** yboaron_ has quit IRC | 08:05 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Create opendev mirrors https://review.opendev.org/658281 | 08:06 |
| AJaeger | ianw: working now? congrats! ^ | 08:07 |
| ianw | AJaeger: i think so, hopefully i've fixed the matching in testinfra but it seems to be getting the right data at least | 08:08 |
| AJaeger | ianw: .zuul.yaml has still unocmmented jobs, want to udpate that already? | 08:09 |
| ianw | ahh yeah; i'll let this run go to make sure it passes then send it back, i have a few other minor typos | 08:10 |
| AJaeger | ok - just left a comment as well... | 08:11 |
| *** bhavikdbavishi has joined #openstack-infra | 08:14 | |
| *** tkajinam has quit IRC | 08:14 | |
| *** lucasagomes has joined #openstack-infra | 08:17 | |
| *** ykarel|lunch is now known as ykarel | 08:21 | |
| *** jaosorior has quit IRC | 08:25 | |
| *** jklare has quit IRC | 08:26 | |
| *** jklare has joined #openstack-infra | 08:26 | |
| *** happyhemant has joined #openstack-infra | 08:31 | |
| *** yboaron_ has joined #openstack-infra | 08:36 | |
| *** derekh has joined #openstack-infra | 08:37 | |
| *** priteau has joined #openstack-infra | 08:41 | |
| *** yboaron_ has quit IRC | 08:42 | |
| *** yboaron_ has joined #openstack-infra | 08:42 | |
| *** jklare has quit IRC | 08:42 | |
| *** jklare has joined #openstack-infra | 08:43 | |
| *** tosky has joined #openstack-infra | 08:44 | |
| *** kopecmartin|off is now known as kopecmartin | 08:46 | |
| *** jklare has quit IRC | 08:52 | |
| *** electrofelix has joined #openstack-infra | 08:54 | |
| *** jklare has joined #openstack-infra | 08:56 | |
| *** zhangfei has quit IRC | 08:57 | |
| *** jklare has quit IRC | 08:58 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Create opendev mirrors https://review.opendev.org/658281 | 08:58 |
| *** oyrogerg has quit IRC | 09:09 | |
| *** e0ne has joined #openstack-infra | 09:12 | |
| *** ricolin has quit IRC | 09:14 | |
| openstackgerrit | Merged opendev/irc-meetings master: Add keystone team office hours meeting https://review.opendev.org/658909 | 09:20 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] Create opendev mirrors https://review.opendev.org/658281 | 09:25 |
| *** jesusaur has joined #openstack-infra | 09:29 | |
| zigo | ianw: Around? | 09:29 |
| *** yamamoto has quit IRC | 09:35 | |
| *** yamamoto has joined #openstack-infra | 09:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove unused legacy-rally-dsvm-zaqar-zaqar https://review.opendev.org/659001 | 09:44 |
| ianw | zigo: a little | 09:47 |
| zigo | ianw: :) | 09:47 |
| zigo | ianw: I was wondering what's the status of debootstrapping buster ... | 09:47 |
| zigo | ianw: Is there some progress at canonical, or should we do something by ourselves? | 09:47 |
| *** roman_g has quit IRC | 09:53 | |
| *** yamamoto has quit IRC | 09:58 | |
| ianw | i dunno ... was a bug filed? | 10:02 |
| *** yamamoto has joined #openstack-infra | 10:02 | |
| *** yamamoto has quit IRC | 10:02 | |
| *** yamamoto has joined #openstack-infra | 10:03 | |
| *** roman_g has joined #openstack-infra | 10:05 | |
| *** yolanda_ has quit IRC | 10:06 | |
| *** yamamoto has quit IRC | 10:07 | |
| *** yolanda_ has joined #openstack-infra | 10:08 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] Create opendev mirrors https://review.opendev.org/658281 | 10:17 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: remove SafeLoader left-over from MappingProxyEncoder https://review.opendev.org/659026 | 10:18 |
| *** yamamoto has joined #openstack-infra | 10:20 | |
| *** yamamoto has quit IRC | 10:24 | |
| *** yamamoto has joined #openstack-infra | 10:24 | |
| *** ykarel is now known as ykarel|afk | 10:27 | |
| *** jbadiapa has joined #openstack-infra | 10:28 | |
| *** jklare has joined #openstack-infra | 10:28 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: honor job dependencies in trigger modal https://review.opendev.org/657567 | 10:31 |
| openstackgerrit | Ian Wienand proposed opendev/puppet-diskimage_builder master: Install backported debootstrap https://review.opendev.org/659031 | 10:33 |
| ianw | zigo: ^ if upstream aren't fixing it, we're going to need to get something like that. if that goes in, then i think the latest release of dib builds buster ok | 10:34 |
| *** jklare has quit IRC | 10:36 | |
| *** jklare has joined #openstack-infra | 10:37 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Create opendev mirrors https://review.opendev.org/658281 | 10:42 |
| ianw | AJaeger: ^ i think that might be it :) it wasn't a false positive in the test -- diagnosed the problems as not disabling the default apache site on the host. nice when you realise your tests are actually showing a real problem! | 10:52 |
| ianw | wanted to get that out for meeting, which i will be at, assuming some form of internet access here works | 10:53 |
| *** gfidente has joined #openstack-infra | 10:58 | |
| *** amoralej is now known as amoralej|lunch | 11:02 | |
| *** jaosorior has joined #openstack-infra | 11:03 | |
| *** yamamoto has quit IRC | 11:03 | |
| ianw | passed, yay! http://logs.openstack.org/81/658281/29/check/system-config-run-mirror/748224a/ ... later | 11:04 |
| *** udesale has quit IRC | 11:12 | |
| zigo | ianw: Thanks. | 11:22 |
| zigo | (job failure though...) | 11:22 |
| *** e0ne has quit IRC | 11:31 | |
| *** e0ne has joined #openstack-infra | 11:31 | |
| *** ykarel|afk is now known as ykarel | 11:34 | |
| *** jangutter has joined #openstack-infra | 11:37 | |
| *** ykarel is now known as ykarel|mtg | 11:37 | |
| *** jpena is now known as jpena|lunch | 11:41 | |
| *** serhatd_ has joined #openstack-infra | 11:43 | |
| *** serhatd has quit IRC | 11:45 | |
| AJaeger | ianw: great! | 11:45 |
| *** ykarel_ has joined #openstack-infra | 11:45 | |
| *** ykarel|mtg has quit IRC | 11:48 | |
| *** nicolasbock has joined #openstack-infra | 11:48 | |
| *** yamamoto has joined #openstack-infra | 11:50 | |
| *** slaweq has joined #openstack-infra | 11:54 | |
| *** vabada_ is now known as dabadaba | 11:58 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: remove SafeLoader left-over from ZuulJSONEncoder https://review.opendev.org/659026 | 12:01 |
| *** dabadaba has quit IRC | 12:02 | |
| *** e0ne has quit IRC | 12:02 | |
| *** yamamoto has quit IRC | 12:04 | |
| *** rfarr__ has joined #openstack-infra | 12:05 | |
| *** bhavikdbavishi has quit IRC | 12:06 | |
| AJaeger | config-core, please review https://review.opendev.org/658685 (and then https://review.opendev.org/658682 ) and https://review.opendev.org/659001 to remove unused jobs | 12:09 |
| *** jcoufal has joined #openstack-infra | 12:10 | |
| *** rosmaita has joined #openstack-infra | 12:10 | |
| *** udesale has joined #openstack-infra | 12:14 | |
| aspiers | clarkb: thanks a lot. Is there any way to debug why etherpad hangs when loading https://etherpad.openstack.org/p/nova-ptg-train ? | 12:17 |
| *** rfarr_ has joined #openstack-infra | 12:18 | |
| *** rfarr__ has quit IRC | 12:20 | |
| *** slaweq has quit IRC | 12:21 | |
| *** priteau has quit IRC | 12:22 | |
| *** derekh has quit IRC | 12:25 | |
| *** janki has quit IRC | 12:30 | |
| *** rlandy has joined #openstack-infra | 12:30 | |
| *** amoralej|lunch is now known as amoralej | 12:31 | |
| *** smarcet has joined #openstack-infra | 12:31 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** yamamoto has joined #openstack-infra | 12:33 | |
| *** kjackal has quit IRC | 12:45 | |
| *** kjackal has joined #openstack-infra | 12:45 | |
| *** jamesmcarthur has joined #openstack-infra | 12:46 | |
| smcginnis | Hangs for me as well, but other etherpads I tried load right away. | 12:47 |
| smcginnis | aspiers: I think we've had corrupted etherpads in the past that have behaved like this. Hopefully there is a way to still get to that. | 12:48 |
| *** mriedem has joined #openstack-infra | 12:49 | |
| *** e0ne has joined #openstack-infra | 12:50 | |
| *** priteau has joined #openstack-infra | 12:51 | |
| *** priteau has quit IRC | 12:51 | |
| *** yolanda_ has quit IRC | 12:57 | |
| *** jistr is now known as jistr|call | 12:59 | |
| *** smarcet has quit IRC | 13:02 | |
| *** ykarel__ has joined #openstack-infra | 13:10 | |
| *** ykarel_ has quit IRC | 13:12 | |
| *** udesale has quit IRC | 13:14 | |
| *** udesale has joined #openstack-infra | 13:14 | |
| *** derekh has joined #openstack-infra | 13:15 | |
| *** ijw has joined #openstack-infra | 13:17 | |
| *** ijw has quit IRC | 13:20 | |
| *** ijw has joined #openstack-infra | 13:20 | |
| *** dmellado has quit IRC | 13:24 | |
| *** dmellado has joined #openstack-infra | 13:24 | |
| *** ijw has quit IRC | 13:25 | |
| *** ijw has joined #openstack-infra | 13:25 | |
| *** yamamoto has quit IRC | 13:28 | |
| *** jistr|call is now known as jistr | 13:30 | |
| *** jamesmcarthur has quit IRC | 13:36 | |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.opendev.org/643731 | 13:38 |
| *** lseki has joined #openstack-infra | 13:38 | |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Deprecate rhel7 in favor of rhel https://review.opendev.org/647710 | 13:39 |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: [wip] rhel8 continuing work https://review.opendev.org/653646 | 13:39 |
| *** ijw has quit IRC | 13:43 | |
| *** ijw has joined #openstack-infra | 13:43 | |
| *** ekultails has joined #openstack-infra | 13:44 | |
| *** ijw has quit IRC | 13:45 | |
| *** ijw has joined #openstack-infra | 13:46 | |
| *** ijw has quit IRC | 13:50 | |
| *** ijw has joined #openstack-infra | 13:50 | |
| *** jamesdenton has joined #openstack-infra | 13:52 | |
| *** jamesmcarthur has joined #openstack-infra | 13:56 | |
| *** kaisers1 has quit IRC | 13:56 | |
| *** jamesmcarthur has quit IRC | 13:57 | |
| *** zul has joined #openstack-infra | 13:57 | |
| *** jamesmcarthur has joined #openstack-infra | 13:58 | |
| *** Goneri has joined #openstack-infra | 14:03 | |
| *** yboaron_ has quit IRC | 14:04 | |
| *** ykarel__ is now known as ykarel | 14:06 | |
| *** sshnaidm has joined #openstack-infra | 14:07 | |
| *** kjackal has quit IRC | 14:07 | |
| *** kjackal has joined #openstack-infra | 14:08 | |
| *** sshnaidm is now known as sshnaidm|afk | 14:08 | |
| *** dklyle has joined #openstack-infra | 14:13 | |
| *** rpittau is now known as rpittau|afk | 14:15 | |
| *** dpawlik has quit IRC | 14:17 | |
| *** aaronsheffield has joined #openstack-infra | 14:18 | |
| mriedem | fyi https://bugs.launchpad.net/openstack-gate/+bug/1829024 - this doesn't hit much (24 hits in 7 days) but wondering if there is some mirror thing going on | 14:20 |
| openstack | Launchpad bug 1829024 in OpenStack-Gate "Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt" [Undecided,New] | 14:20 |
| *** yamamoto has joined #openstack-infra | 14:23 | |
| *** ekultails has quit IRC | 14:26 | |
| *** michael-beaver has joined #openstack-infra | 14:38 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for 404 getting upper-constraints bug 1829024 https://review.opendev.org/659083 | 14:40 |
| openstack | bug 1829024 in OpenStack-Gate "Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt" [Undecided,New] https://launchpad.net/bugs/1829024 | 14:40 |
| *** ekultails has joined #openstack-infra | 14:44 | |
| *** ijw has quit IRC | 14:48 | |
| *** ijw has joined #openstack-infra | 14:48 | |
| *** ijw has quit IRC | 14:55 | |
| *** ijw has joined #openstack-infra | 14:56 | |
| clarkb | mriedem: when I last debugged that it was a corrupted fs on the host machine (so we took that backend out of service) it is possible there is another unhappy backend? or maybe the bad backend was used again? (we can look) | 14:59 |
| *** ijw has quit IRC | 14:59 | |
| clarkb | aspiers: we get a generic error from etherpad | 14:59 |
| clarkb | aspiers: probably if "ether" wasn't in the prefix of the tool we'd dig in more | 14:59 |
| aspiers | clarkb: that's unfortunate | 14:59 |
| aspiers | clarkb: etherpad doesn't seem to have good diagnostics | 15:00 |
| clarkb | aspiers: and the db itself largely has no schema so you have to construct the data yourself out of band | 15:00 |
| aspiers | clarkb: yeah, I know etherpads are supposed to be for ephemeral use only, but our community more or less relies on them as permanent records of f2f meetings | 15:00 |
| *** ijw has joined #openstack-infra | 15:02 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for 404 getting upper-constraints bug 1829024 https://review.opendev.org/659083 | 15:08 |
| openstack | bug 1829024 in OpenStack-Gate "Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt" [Undecided,Confirmed] https://launchpad.net/bugs/1829024 | 15:09 |
| openstackgerrit | Merged opendev/puppet-openstackid master: Added missing double quotes for some config variables on .env file https://review.opendev.org/658937 | 15:09 |
| clarkb | mriedem: on a spot check all backends serve that url successfully (you can use urls like https://gitea01.opendev.org:3000/openstack/requirements/raw/branch/master/upper-constraints.txt to hit the backends too (01 - 08 are the valid backends and 06 is up but not in the load balancer) | 15:10 |
| stephenfin | clarkb: Thanks for merging the first part of those retirement patches. What do we do about the changes to the project repos themselves, given that, for the most part, there's no one maintaining them any more? | 15:11 |
| clarkb | stephenfin: can probably push retirement change to them then have PTL (or TC if necessary) bless that merging ? | 15:11 |
| stephenfin | clarkb: Aye, I've pushed those but these are non-official projects with no PTL/active maintainers, e.g. https://review.opendev.org/#/c/656851/ I assume someone from infra needs to put those through? | 15:12 |
| stephenfin | though harlowja should be able to hit this one, if they're feeling generous :) https://review.opendev.org/#/c/656863/ | 15:13 |
| AJaeger | stephenfin: recheck those changes so that they pass ;) | 15:13 |
| clarkb | we can if necessary. Maybe do direct email to those who were last involved and if they don't respond in a reasonable amount of time we can step in | 15:13 |
| fungi | i'll ask again, but why do you care if there are non-openstack projects in opendev which are seeing minimal or no ongoing maintenance? seems like retiring those is more effort than it's worth | 15:13 |
| stephenfin | AJaeger: Already done (I think) | 15:14 |
| fungi | i don't personally want to be the police of what's not being maintained by random folks who wanted a repo | 15:14 |
| stephenfin | fungi: I don't, really. It was more of a spring cleaning exercise thing than anything, to be honest | 15:14 |
| clarkb | there is some value in reducing the size of openstack/project-config (to simplify its splitting) | 15:15 |
| stephenfin | + figure out how the whole process worked (which I now know) | 15:15 |
| *** sarob has joined #openstack-infra | 15:16 | |
| *** ykarel is now known as ykarel|away | 15:18 | |
| AJaeger | rm_work: I suggest to abandon these two changes: https://review.opendev.org/#/q/project:openstack/neutron-lbaas-dashboard+is:open+branch:master | 15:19 |
| AJaeger | ailuropoda removal change has been merged by the core team, so https://review.opendev.org/#/c/656865/1 could merge. config-core, please review | 15:22 |
| *** sshnaidm|afk has quit IRC | 15:25 | |
| *** sshnaidm has joined #openstack-infra | 15:27 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Neutron-lbaas: temporarily remove pypi job" https://review.opendev.org/658892 | 15:27 |
| *** sshnaidm is now known as sshnaidm|afk | 15:29 | |
| *** tosky has quit IRC | 15:29 | |
| *** ykarel|away has quit IRC | 15:30 | |
| *** tosky has joined #openstack-infra | 15:30 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket build status reporter https://review.opendev.org/658335 | 15:32 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source https://review.opendev.org/658835 | 15:32 |
| *** iurygregory has quit IRC | 15:34 | |
| openstackgerrit | Merged openstack/project-config master: Retire ailuropoda https://review.opendev.org/656865 | 15:38 |
| *** ijw has quit IRC | 15:45 | |
| *** ijw has joined #openstack-infra | 15:46 | |
| *** pgaxatte has quit IRC | 15:47 | |
| AJaeger | config-core, please review https://review.opendev.org/658685 (and then https://review.opendev.org/658682 ) and https://review.opendev.org/659001 to remove some unused jobs... | 15:48 |
| *** dklyle has quit IRC | 15:50 | |
| *** dklyle has joined #openstack-infra | 15:52 | |
| *** hamzy has quit IRC | 15:52 | |
| *** ijw has quit IRC | 15:54 | |
| *** openstackgerrit has quit IRC | 15:54 | |
| *** ijw has joined #openstack-infra | 15:55 | |
| *** e0ne has quit IRC | 15:57 | |
| clarkb | mriedem: fwiw I did 100 requests against that path on each of the backends and had no errors. Now doing 10k requests against the front end to see if I can catch it at all | 15:58 |
| *** igordc has joined #openstack-infra | 15:59 | |
| *** ijw has quit IRC | 15:59 | |
| *** imacdonn has quit IRC | 16:01 | |
| *** gyee has joined #openstack-infra | 16:01 | |
| clarkb | (hoping that if we can reproduce it somehow it will be easier to track down) | 16:02 |
| *** igordc has quit IRC | 16:03 | |
| *** ijw has joined #openstack-infra | 16:04 | |
| *** openstackgerrit has joined #openstack-infra | 16:04 | |
| openstackgerrit | Merged openstack/project-config master: Remove ansible-lint-jobs https://review.opendev.org/658685 | 16:04 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Fix dequeue ref not handling the project name https://review.opendev.org/659110 | 16:08 |
| *** dklyle has quit IRC | 16:11 | |
| *** imacdonn has joined #openstack-infra | 16:15 | |
| *** dklyle has joined #openstack-infra | 16:16 | |
| *** sarob has quit IRC | 16:19 | |
| clarkb | mriedem: I think it is about 1/3 of the way through this request loop. I'll let it churn and update when it is done | 16:20 |
| mriedem | ack | 16:21 |
| *** dklyle has quit IRC | 16:21 | |
| *** ijw has quit IRC | 16:23 | |
| *** dklyle has joined #openstack-infra | 16:23 | |
| *** ijw has joined #openstack-infra | 16:23 | |
| *** yamamoto has quit IRC | 16:24 | |
| *** ginopc has quit IRC | 16:24 | |
| *** ijw has quit IRC | 16:24 | |
| *** yamamoto has joined #openstack-infra | 16:24 | |
| *** ijw has joined #openstack-infra | 16:24 | |
| *** diablo_rojo has joined #openstack-infra | 16:26 | |
| *** ykarel|away has joined #openstack-infra | 16:27 | |
| *** ijw has quit IRC | 16:29 | |
| *** ijw has joined #openstack-infra | 16:30 | |
| *** yamamoto has quit IRC | 16:30 | |
| *** yamamoto has joined #openstack-infra | 16:30 | |
| *** jpich has quit IRC | 16:31 | |
| clarkb | ok I've got to reboot the desktop for updates (not where I'm running the constraints checker script) so I'm gonna do that and find breakfast. Back in a bit | 16:31 |
| *** jpich has joined #openstack-infra | 16:31 | |
| *** ijw has quit IRC | 16:34 | |
| *** ijw has joined #openstack-infra | 16:35 | |
| *** ijw has quit IRC | 16:35 | |
| *** ijw has joined #openstack-infra | 16:35 | |
| *** ijw has quit IRC | 16:36 | |
| *** yamamoto has quit IRC | 16:36 | |
| *** yamamoto has joined #openstack-infra | 16:37 | |
| *** ijw has joined #openstack-infra | 16:37 | |
| *** ijw has quit IRC | 16:37 | |
| *** yamamoto has quit IRC | 16:37 | |
| *** ijw has joined #openstack-infra | 16:38 | |
| *** yamamoto has joined #openstack-infra | 16:38 | |
| *** yamamoto has quit IRC | 16:43 | |
| *** ijw has quit IRC | 16:44 | |
| *** lucasagomes has quit IRC | 16:45 | |
| *** udesale has quit IRC | 16:46 | |
| *** tosky has quit IRC | 16:47 | |
| *** jamesmcarthur has quit IRC | 16:50 | |
| *** dtantsur is now known as dtantsur|afk | 16:51 | |
| *** mattw4 has joined #openstack-infra | 16:53 | |
| *** jamesmcarthur has joined #openstack-infra | 16:56 | |
| *** dklyle has quit IRC | 16:59 | |
| *** derekh has quit IRC | 17:00 | |
| *** armax has quit IRC | 17:00 | |
| *** dklyle has joined #openstack-infra | 17:01 | |
| *** jamesmcarthur has quit IRC | 17:01 | |
| *** slaweq has joined #openstack-infra | 17:06 | |
| *** kopecmartin is now known as kopecmartin|off | 17:07 | |
| *** jamesmcarthur has joined #openstack-infra | 17:08 | |
| *** jpich has quit IRC | 17:10 | |
| *** jamesmcarthur has quit IRC | 17:11 | |
| *** slaweq has quit IRC | 17:14 | |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.opendev.org/643731 | 17:15 |
| *** ralonsoh has quit IRC | 17:25 | |
| *** rfolco|ruck is now known as rfolco|dentist | 17:25 | |
| clarkb | mriedem: 10k requests completed and no 404s | 17:28 |
| clarkb | no 4 or 500 errors at all | 17:29 |
| clarkb | I also went through the redirect like the job I pulled up that had the failure just to ensure it wasn't something in there | 17:29 |
| clarkb | this may end up being a case of let elastic-recheck gather more data for us | 17:29 |
| *** pkopec has quit IRC | 17:30 | |
| clarkb | I'm going to do 10k requests against each backend directly now (instead of going through the redirect and lb) | 17:31 |
| *** jamesmcarthur has joined #openstack-infra | 17:31 | |
| *** _erlon_ has joined #openstack-infra | 17:32 | |
| *** amoralej is now known as amoralej|off | 17:36 | |
| *** armax has joined #openstack-infra | 17:41 | |
| *** jpena is now known as jpena|off | 17:41 | |
| *** Weifan has joined #openstack-infra | 17:44 | |
| *** mattw4 has quit IRC | 17:48 | |
| *** hamzy has joined #openstack-infra | 17:48 | |
| *** mattw4 has joined #openstack-infra | 17:49 | |
| *** sarob has joined #openstack-infra | 17:50 | |
| *** sarob_ has joined #openstack-infra | 17:50 | |
| clarkb | fungi: fwiw I didn't manage to get around to cleaning up the groups servers last week as I ended up removing and replacing siding and insulation from the side of the house and suffering from allergy mallaise as a result of being outside | 17:51 |
| clarkb | fungi: considering you've already picked up the prep tasks on that did you want to complete the deletion? | 17:52 |
| *** sarob_ has quit IRC | 17:52 | |
| *** sarob_ has joined #openstack-infra | 17:52 | |
| *** dklyle has quit IRC | 17:52 | |
| *** sarob has quit IRC | 17:54 | |
| *** tjgresha has quit IRC | 17:54 | |
| *** jamesmcarthur has quit IRC | 17:55 | |
| *** jamesmcarthur has joined #openstack-infra | 17:56 | |
| fungi | happy to, you can leave it with me | 17:57 |
| fungi | though it may be later today, i'll tack it on near the top of my to do list | 17:57 |
| *** tjgresha has joined #openstack-infra | 17:57 | |
| *** e0ne has joined #openstack-infra | 17:57 | |
| clarkb | no worries its already sat for a few days. | 17:58 |
| clarkb | Next on my list was looking at lcaening up the cgit servers | 17:58 |
| clarkb | though the occasional constraints GET failures make me question is we should (though its not like we'd have an easy revert at this point anyway so are 99.99% gonna roll forawrd whatever the fix may be | 17:58 |
| clarkb | Then its merging some puppet 4 changes (maybe I do that immediately after the meeting as it will take soe time to filter through the gate) | 17:59 |
| fungi | clarkb: have you had an opportunity to skim https://review.opendev.org/657174 yet? i need to revise it based on a handful of comments, but since you voiced the concerns with the original goal which prompted me to propose that update, your feedback to confirm the direction i took would be helpful | 17:59 |
| clarkb | fungi: I skimmed an early patchset. Let me take a quick look now | 17:59 |
| fungi | it's probably still the same patchset | 18:00 |
| clarkb | oh yup still ps1 | 18:00 |
| fungi | just want to make sure i hit all the points you were making | 18:00 |
| fungi | figured if you had any suggestions i'd roll those into the second revision while i'm poking at it | 18:00 |
| clarkb | fungi: yup I think it did. Basically got us away from implying ipv6 doesn't currently work and isn't currently tested to asserting ipv6 works in many circumstances but we want to take the next step of ensuring that ipv6 alone also works | 18:01 |
| *** jamesmcarthur has quit IRC | 18:01 | |
| *** Weifan has quit IRC | 18:01 | |
| *** Weifan has joined #openstack-infra | 18:01 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Fix dequeue ref not handling the project name https://review.opendev.org/659110 | 18:02 |
| fungi | cool, then i'll just clean it up based on the other comments and go from there. thanks for pointing it out! | 18:02 |
| clarkb | and the other comments I see shouldn't change that. I think you are good to push a followup to address those existing comments | 18:02 |
| *** e0ne has quit IRC | 18:05 | |
| *** gfidente is now known as gfidente|afk | 18:13 | |
| *** betherly has joined #openstack-infra | 18:19 | |
| *** betherly has quit IRC | 18:24 | |
| *** jamesmcarthur has joined #openstack-infra | 18:28 | |
| *** ijw has joined #openstack-infra | 18:33 | |
| clarkb | my 80k curls for master constraints are about half done | 18:36 |
| clarkb | none of 400'd | 18:36 |
| clarkb | s/of/have/ | 18:36 |
| *** ijw has quit IRC | 18:37 | |
| *** ijw has joined #openstack-infra | 18:37 | |
| *** ijw has quit IRC | 18:38 | |
| *** e0ne has joined #openstack-infra | 18:38 | |
| *** ijw has joined #openstack-infra | 18:38 | |
| *** ijw has quit IRC | 18:38 | |
| *** ijw has joined #openstack-infra | 18:39 | |
| *** ijw has quit IRC | 18:41 | |
| *** ijw_ has joined #openstack-infra | 18:41 | |
| clarkb | I counted progress poorly. They have completed now. No 4 or 5 hundred errors | 18:48 |
| clarkb | 80k requests, 10k to each backend | 18:48 |
| *** jamesmcarthur has quit IRC | 18:48 | |
| clarkb | This makes me think it is some other action that must be causing this (and probasbly means we need to dig through logs) | 18:48 |
| *** Weifan has quit IRC | 18:50 | |
| fungi | or it's connectivity problems for a handful of parts of some providers? | 18:51 |
| *** ijw_ has quit IRC | 18:52 | |
| clarkb | EnvironmentError: 404 Client Error: Not Found for url: https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt is the error message from pip | 18:52 |
| clarkb | which implies gets a 404 from the server, but I wouldn't be surprised if you get a similar error if things fail otherways | 18:53 |
| *** ijw has joined #openstack-infra | 18:53 | |
| clarkb | hrm the dns issues in sjc1 are still a big problem | 18:53 |
| fungi | where's a sample failure log? | 18:53 |
| fungi | for the 404 | 18:53 |
| clarkb | fungi: https://bugs.launchpad.net/openstack-gate/+bug/1829024 | 18:53 |
| openstack | Launchpad bug 1829024 in OpenStack-Gate "Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt" [Undecided,Confirmed] | 18:53 |
| fungi | thanks | 18:53 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Force ipv4 on vexxhost nodepool nodes https://review.opendev.org/659140 | 18:55 |
| clarkb | mriedem: ^ fyi that is a stab at working around the problem with name resolution in sjc1 (assumes that it is ipv6 related) | 18:55 |
| clarkb | I think we should get that in and at least rule ipv6 in or out | 18:56 |
| clarkb | We'll have our weekly meeting in about 5 minutes over in #openstack-infra | 18:56 |
| clarkb | see you there! | 18:56 |
| clarkb | bah #openstack-meeting | 18:57 |
| fungi | aha, so this is pip being passed -chttps://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt and then complaining that it can't retrieve the https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt to which it was redirected | 18:57 |
| clarkb | fungi: yup | 18:57 |
| clarkb | so I did the first 10k requests to the original url (to ensure the redirect wasn't somehow to blame) and then 10k on each of the 8 backends to rule out any specific backend | 18:58 |
| *** betherly has joined #openstack-infra | 19:00 | |
| fungi | looks like "404 Client Error" is coming from python requests | 19:00 |
| *** betherly has quit IRC | 19:04 | |
| *** Weifan has joined #openstack-infra | 19:06 | |
| *** icarusfactor has joined #openstack-infra | 19:10 | |
| *** Weifan has quit IRC | 19:10 | |
| *** factor has quit IRC | 19:12 | |
| *** ykarel|away has quit IRC | 19:14 | |
| *** Weifan has joined #openstack-infra | 19:15 | |
| *** dklyle has joined #openstack-infra | 19:16 | |
| *** dklyle has quit IRC | 19:19 | |
| *** david-lyle has joined #openstack-infra | 19:19 | |
| *** pcaruana has quit IRC | 19:19 | |
| *** jamesmcarthur has joined #openstack-infra | 19:20 | |
| *** slaweq has joined #openstack-infra | 19:21 | |
| *** Weifan has quit IRC | 19:23 | |
| *** Weifan has joined #openstack-infra | 19:26 | |
| *** jamesmcarthur has quit IRC | 19:27 | |
| *** jamesmcarthur_ has joined #openstack-infra | 19:28 | |
| *** jamesmcarthur has joined #openstack-infra | 19:29 | |
| *** Weifan has quit IRC | 19:30 | |
| *** Weifan has joined #openstack-infra | 19:31 | |
| *** jamesmcarthur_ has quit IRC | 19:33 | |
| *** yamamoto has joined #openstack-infra | 19:35 | |
| *** Weifan has quit IRC | 19:38 | |
| *** rfolco|dentist is now known as rfolco|ruck | 19:39 | |
| *** Weifan has joined #openstack-infra | 19:39 | |
| *** jcoufal has quit IRC | 19:39 | |
| *** yamamoto has quit IRC | 19:40 | |
| *** david-lyle has quit IRC | 19:42 | |
| openstackgerrit | Ian Wienand proposed opendev/puppet-diskimage_builder master: Install backported debootstrap https://review.opendev.org/659031 | 19:43 |
| clarkb | infra-root please review https://review.opendev.org/#/c/655476/ which fixes a number of isues in our repo rename playbook that we found doing the opdnev git repo migrations | 19:44 |
| *** ijw has quit IRC | 19:46 | |
| *** sarob_ has quit IRC | 19:48 | |
| *** sarob has joined #openstack-infra | 19:49 | |
| clarkb | infra-root can we also get https://review.opendev.org/#/c/659140/ in as a sanity check for dns in sjc1? | 19:50 |
| clarkb | I've approved the puppet-4 for zuul change | 19:52 |
| fungi | right on! | 19:53 |
| * fungi grabs edge of seat | 19:53 | |
| *** sarob has quit IRC | 19:54 | |
| cmurphy | /o\ | 19:56 |
| *** Weifan has quit IRC | 19:58 | |
| *** factor has joined #openstack-infra | 20:00 | |
| ianw | clarkb: hrm, yeah i see that puppet failure on the mirror change; for sure unrelated http://logs.openstack.org/81/658281/29/check/openstackci-beaker/ac4ed10/job-output.txt.gz ... 1) nodepool builder should work with no errors | 20:00 |
| clarkb | ianw: ah it failed to retrieve gpg keys | 20:01 |
| clarkb | that implies even though we vendor the file it still retrieves it remotely? | 20:01 |
| *** icarusfactor has quit IRC | 20:01 | |
| clarkb | that is really annoying (because I thought I had fixed that issue by vendoring the public key in the module) | 20:02 |
| ianw | ahh, i was just looking at that, because to support buster we'd need to also install debootstrap from same ppa in there | 20:03 |
| clarkb | I'm guessing that is a bug in the apt puppet module | 20:03 |
| clarkb | newer ubuntu (apt really) allows you to just drop the public key into etc somewhere and it will trust it | 20:04 |
| clarkb | I think starting with bionic | 20:04 |
| *** jtomasek has quit IRC | 20:04 | |
| fungi | yep, well, more specifically lets you drop an ascii-armored copy somewhere rather than having to use a binary keyring format | 20:05 |
| fungi | i think it's that ascii-armored individual public keys can just be added one per file in /etc/apt/trusted-keys.d (from memory) | 20:06 |
| *** tesseract has quit IRC | 20:06 | |
| fungi | which would be super simple to do via config management at that point | 20:08 |
| clarkb | though I also expect we'll be dockering nodepool and zuul as part of their next big uplift? | 20:09 |
| clarkb | that then becomes and image build time problem | 20:09 |
| fungi | and also opens us up to use significantly newer distros which have solved the issues we're trying to work around | 20:12 |
| clarkb | I think we are about an hour away from the zuul puppet-4 change applying | 20:15 |
| clarkb | (based on timings of how long ansible takes to loop and where we are in the loop) | 20:15 |
| ianw | clarkb: hrm, it looks like the source should work ... https://github.com/puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/apt_key/apt_key.rb | 20:15 |
| *** jamesmcarthur has quit IRC | 20:15 | |
| ianw | but we specify it as a puppet:// , not a URL, i wonder if that doesn't expand right? | 20:16 |
| ianw | https://opendev.org/opendev/puppet-diskimage_builder/src/branch/master/manifests/init.pp#L30 | 20:16 |
| clarkb | oh maybe we have to write it to disk then use a rooted path on the instance? | 20:17 |
| clarkb | ya if the ruby uri opener hasn't been overloaded by puppet to handle puppet:// properly then this is likely the case | 20:17 |
| ianw | we don't seem to see any of those errors, but it seems to fall through to "tempfile(key)" and we do see gpg: keyring `/tmp/tmpbck6_t92/secring.gpg' created | 20:19 |
| *** e0ne has quit IRC | 20:20 | |
| openstackgerrit | Merged opendev/system-config master: Update zuul servers to puppet 4 https://review.opendev.org/650380 | 20:21 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update lists.openstack.org to puppet 4 https://review.opendev.org/653489 | 20:25 |
| clarkb | cmurphy: ^ fyi rebased now that zuul change is in (so that I didn't have to rebase both) | 20:25 |
| cmurphy | thanks clarkb | 20:26 |
| *** sarob has joined #openstack-infra | 20:27 | |
| clarkb | and ya puppet just ran on zuul01 with the old puppet3 stuff for the last time so figure ~45 minutes before it runs with the puppet-4 stuff | 20:29 |
| *** jamesmcarthur has joined #openstack-infra | 20:31 | |
| *** jamesmcarthur_ has joined #openstack-infra | 20:37 | |
| *** jamesmcarthur has quit IRC | 20:40 | |
| *** hamzy has quit IRC | 20:41 | |
| *** jamesmcarthur_ has quit IRC | 20:41 | |
| *** slaweq has quit IRC | 20:44 | |
| clarkb | fungi: did you want to review https://review.opendev.org/#/c/659140/ to force sjc1 to ipv4? if not I'll probably approve it | 20:45 |
| *** jamesmcarthur has joined #openstack-infra | 20:47 | |
| fungi | approved | 20:49 |
| *** Weifan has joined #openstack-infra | 20:50 | |
| *** jamesmcarthur has quit IRC | 20:50 | |
| clarkb | tyty | 20:50 |
| *** Weifan has quit IRC | 20:52 | |
| clarkb | /opt/puppetlabs is starting to appear on some of the zuul nodes | 20:52 |
| *** Weifan has joined #openstack-infra | 20:53 | |
| *** jamesmcarthur has joined #openstack-infra | 20:55 | |
| *** jamesmcarthur has quit IRC | 20:55 | |
| *** jamesmcarthur has joined #openstack-infra | 20:55 | |
| *** gfidente|afk has quit IRC | 20:58 | |
| *** jamesmcarthur has quit IRC | 20:59 | |
| *** sarob has quit IRC | 21:01 | |
| clarkb | hrm ze01 still doesn't have /opt/puppetlabs but ze02 does | 21:03 |
| clarkb | ze01 isn't in the emergency list | 21:03 |
| * clarkb tries to be patient, it could just be slower | 21:04 | |
| clarkb | heh of course now it shows up | 21:05 |
| fungi | eventually consistent is our middle name | 21:09 |
| fungi | open(eventually consistent)dev | 21:09 |
| *** jtomasek has joined #openstack-infra | 21:15 | |
| *** jamesmcarthur has joined #openstack-infra | 21:20 | |
| clarkb | http://paste.openstack.org/show/751363/ looks good on the scheduler | 21:20 |
| clarkb | the exectuors and mergers happen a bit later as they aren't part of the git playbook | 21:21 |
| clarkb | anyone else able to be second reviewer on https://review.opendev.org/#/c/653489/2 ? enables puppet4 on lists.o.o | 21:22 |
| *** rh-jelabarre has quit IRC | 21:24 | |
| fungi | yup, just a sec | 21:24 |
| fungi | approvidado | 21:25 |
| *** dklyle has joined #openstack-infra | 21:27 | |
| clarkb | ze01 failed due to the warning about slow cryptography from pip | 21:27 |
| openstackgerrit | Merged opendev/system-config master: Force ipv4 on vexxhost nodepool nodes https://review.opendev.org/659140 | 21:27 |
| clarkb | going to see how widespread that is then try to page back in my brainspace for how to update that | 21:27 |
| clarkb | oh interesting it appears to be a python2 only problem | 21:28 |
| clarkb | pip3 doesn't complain about it | 21:28 |
| clarkb | so easier fix here may be to track down what it is we try to install with python2 pip on ze01 | 21:29 |
| clarkb | it is the gear install | 21:30 |
| *** jamesmcarthur has quit IRC | 21:30 | |
| clarkb | which we want under python2 because ansible does python2 | 21:30 |
| clarkb | pabelanger: fungi do you recall if we stopped install gear to the virtualenv? | 21:31 |
| clarkb | I don't recall what the end result in zuul was around that particular issue | 21:31 |
| clarkb | put another way do we use the global install of gear at all or is that included into the virtualenv and found that way? | 21:31 |
| *** jamesmcarthur has joined #openstack-infra | 21:31 | |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/manifests/site.pp#L822-L828 and https://opendev.org/opendev/puppet-zuul/src/branch/master/manifests/executor.pp#L43-L49 are the two possibly redundant pieces of config management | 21:33 |
| clarkb | I think https://review.opendev.org/#/c/655491/ means that the puppet install of gear is used by the jobs | 21:36 |
| *** jamesmcarthur has quit IRC | 21:36 | |
| clarkb | so I need to actually fix this | 21:36 |
| *** Weifan has quit IRC | 21:36 | |
| clarkb | I'm going to attempt a pip install -U cryptography pyopenssl on ze01 | 21:38 |
| clarkb | note this is for python2 not python3 | 21:38 |
| clarkb | if that results in a working pip2 I can update our puppetry to do the update on our other executors | 21:39 |
| *** rh-jelabarre has joined #openstack-infra | 21:39 | |
| *** jamesmcarthur has joined #openstack-infra | 21:40 | |
| *** sarob has joined #openstack-infra | 21:40 | |
| clarkb | that seems to have resulted in a working pip. Though now that I think of it a bit more there is a chicken and egg here where puppet pip can't update cryptography and pyopenssl because the module just fails | 21:40 |
| clarkb | wow way to break puppet and pip | 21:40 |
| clarkb | for now I think I'll just manually do this fix on the executors while I consider what our options are | 21:41 |
| *** rh-jelabarre has quit IRC | 21:44 | |
| *** jamesmcarthur has quit IRC | 21:44 | |
| clarkb | cmurphy: what do you think the chances are that puppet would backport the fix for this to puppet 4? | 21:44 |
| *** dklyle has quit IRC | 21:44 | |
| clarkb | (that would simplify a lot of stuff assuming they ever actually got around to fixing the bug) | 21:44 |
| *** dklyle has joined #openstack-infra | 21:44 | |
| cmurphy | clarkb: is it definitely an issue with the pip provider or is it our openstack_pip provider? | 21:45 |
| *** Weifan has joined #openstack-infra | 21:45 | |
| clarkb | when I looked at it last it was both | 21:46 |
| clarkb | and I think puppet 5 pip module got the fix | 21:46 |
| * clarkb looks for the upstream bug | 21:46 | |
| *** Weifan has quit IRC | 21:46 | |
| fungi | yeah, i don't recall what we were doing between system context and virtualenv | 21:47 |
| openstackgerrit | Merged opendev/system-config master: Update lists.openstack.org to puppet 4 https://review.opendev.org/653489 | 21:47 |
| *** Weifan has joined #openstack-infra | 21:47 | |
| fungi | i guess zuul just forks geard for that purpose, so it doesn't need to share the same interpreter? | 21:47 |
| clarkb | also looking at apt-cache redepends --installed output I think python-paste may be what pulls crytpigraphy in | 21:47 |
| *** betherly has joined #openstack-infra | 21:48 | |
| clarkb | fungi: I think it uses it as a client lib to submit the logstash job from the executor nodes | 21:48 |
| *** Weifan has quit IRC | 21:48 | |
| cmurphy | pip module or pip provider? | 21:49 |
| clarkb | cmurphy: https://tickets.puppetlabs.com/browse/PUP-8986 is the upstream bug | 21:49 |
| *** Goneri has quit IRC | 21:49 | |
| cmurphy | my reading of https://puppet.com/misc/puppet-enterprise-lifecycle and https://puppet.com/docs/pe/2017.3/component_versions_in_recent_pe_releases.html are that puppet 4 is end of life already so chances of backport are very slim | 21:50 |
| *** dklyle has quit IRC | 21:50 | |
| clarkb | looks like bgu was never even fixed anyway | 21:51 |
| mordred | clarkb: problem goes away once we're deploying zuul with the dockers | 21:51 |
| clarkb | mordred: ya but this isn't the first time we've hit this. it is a general probelm for us | 21:51 |
| * mordred isn't really here - just thought he'd be unhelpful | 21:51 | |
| mordred | clarkb: ++ | 21:51 |
| clarkb | thinking out loud maybe anytime we install pip we also update cryptography and pyopenssl | 21:52 |
| clarkb | and just betls and suspenders it that way | 21:52 |
| mordred | not a terrible idea | 21:52 |
| mordred | clarkb: also - I totally misread an earlier comment from you to mean that cryptography was pulling in python-paste | 21:52 |
| clarkb | other way around :) | 21:53 |
| clarkb | that is how we end up with old distro version of cyptography (if I've done my apt-cache searching properly) | 21:53 |
| clarkb | I don't see where python-paste comes from so maybe it is an older thing? | 21:53 |
| *** betherly has quit IRC | 21:53 | |
| clarkb | from before cherrypy? | 21:53 |
| fungi | could be rebuilding ze01 would solve this particular case? | 21:53 |
| *** mattw4 has quit IRC | 21:54 | |
| *** jtomasek has quit IRC | 21:54 | |
| clarkb | ya ok ze11 and ze12 don't have it installed under python2 | 21:55 |
| clarkb | so yes I think if/when we rebuild these servers we'll actually be fine since nothing will pull in old distro cryptography | 21:55 |
| clarkb | in that case I'm just gonna bandaid them all now ze01-10 so that puppet-4 progress can be made and we can verify that works otherwise | 21:55 |
| clarkb | and autoremove isn't removing python-paste because we (puppet) must've done an explicit install of it at some point | 21:56 |
| fungi | likely so, yes | 21:58 |
| clarkb | ok that is done 01-10 have had pyopenssl and cryptography updated | 21:58 |
| clarkb | and the warning is gone | 21:59 |
| clarkb | and in the next 15 minutes or so we should see puppet-4 run again on the executors without that error | 21:59 |
| clarkb | puppet-4 was successful on ze11 so that should be the only thing | 22:00 |
| *** rcernin has joined #openstack-infra | 22:00 | |
| *** Weifan has joined #openstack-infra | 22:02 | |
| *** rfarr_ has quit IRC | 22:03 | |
| clarkb | yup puppet has applied successfully in the last few minutes to the other executors | 22:05 |
| clarkb | next pass should get lists.o.o | 22:05 |
| cmurphy | \o/ | 22:08 |
| *** betherly has joined #openstack-infra | 22:08 | |
| clarkb | and I double checked the other mergers (I didn't initially since the first was happy) they all look good | 22:09 |
| clarkb | 0006154137 is the first sjc1 instance that will use ipv4 as primary address I think | 22:11 |
| clarkb | hrm no seems to have used ipv6 still | 22:12 |
| clarkb | I wonder if we have to restart nodepool to pick up that change to clouds.yaml | 22:12 |
| clarkb | Shrews: ^ do we properly autodetect clouds.yaml updates in nodepool launchers? | 22:12 |
| *** sarob has quit IRC | 22:12 | |
| *** betherly has quit IRC | 22:13 | |
| Shrews | clarkb: no. That is a sdk confit that nodepool does not care about | 22:14 |
| Shrews | config | 22:14 |
| clarkb | Shrews: any reason I shouldn't restart nl03's launcher? | 22:15 |
| Shrews | nope | 22:15 |
| clarkb | k doing that now then | 22:15 |
| clarkb | (done) | 22:16 |
| mordred | clarkb, Shrews: fwiw - there is a sdk release change up: https://review.opendev.org/#/c/658800/ | 22:16 |
| mordred | when it lands, the new sdk release contains the patches with the image code reorg | 22:16 |
| mordred | it SHOULD be fine | 22:16 |
| mordred | but will be worth watching when we restart builders | 22:16 |
| clarkb | cool we have to opt into checksum verification on that change iirc | 22:17 |
| mordred | yeah | 22:17 |
| mordred | we should probably write a nodepool change to consume that option | 22:17 |
| clarkb | (we should update nodepool to do that by default and then have nodepool toggle to not do it if your cloud is silly) | 22:17 |
| mordred | clarkb: ++ | 22:17 |
| mordred | clarkb: I also would like to get something added to glance that would expose "checksum as uploaded" as opposed to "transformed checksum" | 22:18 |
| *** whoami-rajat has quit IRC | 22:18 | |
| mordred | because both are potentially useful for the user to know | 22:18 |
| clarkb | mordred: ya and you can checksum the bytes as they arrive so that is basically free | 22:18 |
| mordred | ++ | 22:18 |
| clarkb | then glance could also verify the image itself | 22:18 |
| mordred | yeah | 22:18 |
| clarkb | and reject the upload prior to conversion if the provider checksum doesn't match | 22:18 |
| clarkb | the post conversion checksum really only matters to glance and nova internally | 22:19 |
| clarkb | its not somethign users should care about | 22:19 |
| clarkb | rm_work: can I clean up the test nodes we held to debug ipv6 multinode stuff? I assume we can | 22:20 |
| rm_work | clarkb: yepyep | 22:20 |
| rm_work | thanks! | 22:20 |
| clarkb | mordred: what about your "mordred puppet puppet ansible puppet puppet" held nodes | 22:20 |
| *** kjackal has quit IRC | 22:22 | |
| clarkb | /opt/puppetlabs exists on lists.o.o now | 22:22 |
| clarkb | puppet should run on it in about 20 minutes | 22:23 |
| mordred | clarkb: they can go | 22:24 |
| clarkb | mordred: k I'll clean them up. Thanks | 22:24 |
| mordred | clarkb: I can't believe that note wasn't clear about its intent | 22:25 |
| mordred | clarkb: (that was from trying to debug the last remaining failure in the split-service-playbooks patch) | 22:25 |
| clarkb | ah | 22:25 |
| clarkb | fungi: logan- there are a couple limestone held nodes with your names on them | 22:26 |
| clarkb | they have older serial numbers, they still needed? | 22:26 |
| clarkb | (I figure may as well clean up while I'm in here) | 22:26 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Spec: Add a Kubernetes Operator for Zuul https://review.opendev.org/659180 | 22:26 |
| logan- | 0005060004 is done | 22:26 |
| *** armax has quit IRC | 22:26 | |
| clarkb | k I'll delete that one | 22:27 |
| logan- | thats on vexxhost-sjc though, its the only held know I know of thats mine | 22:27 |
| clarkb | oh the other one is just fungi. "debugging ssh disconnects" | 22:27 |
| clarkb | fungi: ^ is that from when we had the zookeeper connection issues that we thought were network problems? | 22:27 |
| clarkb | if so I think that one can be cleaned up too | 22:27 |
| fungi | yep, that can go | 22:28 |
| clarkb | ok newly booted nodes in sjc1 are still preferring ipv6 dns | 22:28 |
| clarkb | I think that means my assumption that we keyed off the primary ip of the host to select dns is wrong and we must just test the node for valid ipv6 | 22:29 |
| clarkb | next stop: change that | 22:29 |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Only use ipv6 dns if nodepool uses ipv6 https://review.opendev.org/659181 | 22:39 |
| clarkb | ok ^ probably needs a bit closer monitoring than I have time for today with puppet-4 stuff | 22:39 |
| clarkb | but I think that is our next step in debugging the sjc1 dns issues | 22:39 |
| *** yamamoto has joined #openstack-infra | 22:40 | |
| *** sarob has joined #openstack-infra | 22:41 | |
| clarkb | unrelated, but if/when we rebuild gitea06 we should give it more disk than the current gitea servers have | 22:41 |
| clarkb | we aren't in danger of immedaitely running out but the space there is a bit tight | 22:41 |
| clarkb | (then we may want to replace the other cluster members once by one over time | 22:41 |
| *** yamamoto has quit IRC | 22:45 | |
| clarkb | puppet 4 should run on lists any minute now | 22:47 |
| clarkb | May 14 22:49:49 lists puppet-user[9628]: (Facter) error while resolving custom fact "virtualenv_version": undefined method `[]' for nil:NilClass | 22:50 |
| clarkb | cmurphy: ^ I feel like we ran into this one previously too | 22:50 |
| clarkb | but I'm not remembering the fix | 22:50 |
| clarkb | I want to say the fix was to update the puppet-python module now that I have had a little think about it | 22:51 |
| cmurphy | right | 22:51 |
| cmurphy | i thought we did that | 22:51 |
| clarkb | I thought so too but git log isn't showing it | 22:52 |
| *** tkajinam has joined #openstack-infra | 22:52 | |
| * clarkb looks in gerrit | 22:52 | |
| cmurphy | is it a hard error or warning? | 22:52 |
| *** _erlon_ has quit IRC | 22:52 | |
| cmurphy | https://review.opendev.org/651877 | 22:52 |
| clarkb | hard error. Puppet stops with that message | 22:52 |
| clarkb | and ya thats teh change | 22:52 |
| cmurphy | fungi: ^ we need another +2 | 22:52 |
| clarkb | at this point in the day do we want to let it be broken for the night then merge 651877 tomorrow? (I'm anticipating spending time with kids in about an hour) | 22:53 |
| *** mattw4 has joined #openstack-infra | 22:54 | |
| cmurphy | if it hasn't affected the services then it's probably fine to let it be broken | 22:55 |
| cmurphy | could also temporarily revert | 22:55 |
| clarkb | I don't think it affected anything. Puppet tries to run facter very early in its processing and fails at that early step | 22:55 |
| clarkb | so lists.o.o is happy, we just can't add new lists while in that state | 22:55 |
| *** pcrews has quit IRC | 22:55 | |
| fungi | on it | 22:56 |
| *** harlowja has quit IRC | 22:59 | |
| *** pcrews has joined #openstack-infra | 23:00 | |
| clarkb | what I don't understand is why this fact issue popped up on some hosts and not others | 23:01 |
| clarkb | we don't use puppet-python on lists.o.o I don't think | 23:01 |
| clarkb | maybe we do though? | 23:01 |
| *** happyhemant has quit IRC | 23:01 | |
| clarkb | ya I can't find where we use virtualenv_version (may not need to use it for this error to hit) or puppet python module on the lists server | 23:06 |
| clarkb | http://paste.openstack.org/show/751368/ that is why it happens | 23:09 |
| *** sarob_ has joined #openstack-infra | 23:09 | |
| clarkb | and reading the fix in puppet python 1.9.5 I don't think the update will actually fix this for us | 23:10 |
| clarkb | https://github.com/voxpupuli/puppet-python/commit/c2ca9d7495375bde781d766b3e6d3dd59cfbbed3 is the diff | 23:10 |
| clarkb | need to fix the virtualenv install | 23:10 |
| clarkb | any objection to me reinstall virtualenv via pip on lists.o.o? we have 13.1.0 installed (supposedly) | 23:11 |
| clarkb | fungi: ^ | 23:11 |
| clarkb | if I had to guess this is fallout from the in place upgrade | 23:12 |
| *** sarob has quit IRC | 23:12 | |
| clarkb | (I think the puppet python update is still a reasonable thing to do as it fixes another set of problems with broken facts just not this one I don't think) | 23:13 |
| clarkb | also happy to fix virtualenv in the morning when I have a bit more time to watch it | 23:13 |
| clarkb | oh huh its a python3 virtualenv | 23:14 |
| fungi | yeah, that seems fine | 23:14 |
| clarkb | so the issue is virtualenv is installed under python2 but somehow the bin/virtualenv scipt has a python3 shebang | 23:15 |
| clarkb | and there is no python3 virtualenv module | 23:15 |
| clarkb | so ya I think the fix here is pip2 install -U virtualenv==13.1.0 to have it rewrite the bin file | 23:15 |
| clarkb | and I can do that tomorrow morning when I have time to watch puppet run on lists | 23:15 |
| *** jamesmcarthur has joined #openstack-infra | 23:17 | |
| clarkb | definitely feeling like I've run out of steam for now (yay allergies, need more rain) | 23:17 |
| *** Weifan has quit IRC | 23:17 | |
| ianw | this sounds very like issues i've had with pip-and-virtualenv | 23:18 |
| ianw | oh, no python3 virtualenv module ... maybe not | 23:18 |
| *** Weifan has joined #openstack-infra | 23:19 | |
| clarkb | ianw: /usr/local/bin/virtualenv has python3 shebang but there is no python3 virtualenv module installed. THere is a python2 virtualenv module installed | 23:19 |
| *** armax has joined #openstack-infra | 23:25 | |
| ianw | ok ... yeah. it is likely that script can get reowned during various installations | 23:25 |
| *** yamamoto has joined #openstack-infra | 23:26 | |
| *** betherly has joined #openstack-infra | 23:31 | |
| *** betherly has quit IRC | 23:36 | |
| *** rlandy is now known as rlandy|bbl | 23:37 | |
| openstackgerrit | Merged opendev/system-config master: Update puppet-python to "fix" broken facts https://review.opendev.org/651877 | 23:38 |
| *** irclogbot_3 has quit IRC | 23:45 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: add host-key-checking toggle https://review.opendev.org/653679 | 23:46 |
| *** irclogbot_0 has joined #openstack-infra | 23:48 | |
| *** jamesmcarthur has quit IRC | 23:49 | |
| *** jamesmcarthur has joined #openstack-infra | 23:49 | |
| *** lseki has quit IRC | 23:51 | |
| *** betherly has joined #openstack-infra | 23:51 | |
| *** jamesmcarthur has quit IRC | 23:54 | |
| *** betherly has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!