| donnyd | I don't really even know how to get gnocchi to gather things | 00:01 |
|---|---|---|
| *** rlandy has quit IRC | 00:01 | |
| *** gyee has quit IRC | 00:07 | |
| *** smarcet has quit IRC | 00:08 | |
| *** aaronsheffield has quit IRC | 00:09 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** slaweq has quit IRC | 00:15 | |
| *** markvoelker has joined #openstack-infra | 00:23 | |
| *** markvoelker has quit IRC | 00:28 | |
| *** jamesmcarthur has joined #openstack-infra | 00:29 | |
| *** zhurong has quit IRC | 00:40 | |
| *** smarcet has joined #openstack-infra | 00:56 | |
| *** jamesmcarthur has quit IRC | 01:07 | |
| *** jamesmcarthur has joined #openstack-infra | 01:07 | |
| *** zhurong has joined #openstack-infra | 01:08 | |
| *** jamesmcarthur has quit IRC | 01:12 | |
| *** markvoelker has joined #openstack-infra | 01:26 | |
| *** smarcet has quit IRC | 01:27 | |
| *** yamamoto has joined #openstack-infra | 01:36 | |
| *** markvoelker has quit IRC | 01:37 | |
| *** markvoelker has joined #openstack-infra | 01:37 | |
| *** jamesmcarthur has joined #openstack-infra | 01:38 | |
| *** markvoelker has quit IRC | 01:42 | |
| *** yonglihe has joined #openstack-infra | 01:52 | |
| *** markvoelker has joined #openstack-infra | 02:02 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** HenryG has quit IRC | 02:13 | |
| *** HenryG has joined #openstack-infra | 02:15 | |
| *** slaweq has quit IRC | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:16 | |
| openstackgerrit | Gabor Lekeny proposed zuul/zuul master: Decode k8s ServiceAccount bearer token https://review.opendev.org/687107 | 02:18 |
| *** otherwiseguy has joined #openstack-infra | 02:20 | |
| *** roman_g has quit IRC | 02:35 | |
| *** jamesmcarthur has joined #openstack-infra | 02:36 | |
| *** jamesmcarthur has quit IRC | 02:41 | |
| *** anteaya has quit IRC | 02:44 | |
| *** ricolin has joined #openstack-infra | 02:47 | |
| *** igordc has quit IRC | 02:52 | |
| *** goldyfruit_ has quit IRC | 03:05 | |
| *** goldyfruit has joined #openstack-infra | 03:06 | |
| *** goldyfruit has quit IRC | 03:08 | |
| *** goldyfruit has joined #openstack-infra | 03:11 | |
| *** goldyfruit has quit IRC | 03:12 | |
| *** rfolco has quit IRC | 03:12 | |
| *** igordc has joined #openstack-infra | 03:19 | |
| *** rh-jelabarre has quit IRC | 03:24 | |
| *** kjackal has joined #openstack-infra | 03:45 | |
| *** exsdev0 has joined #openstack-infra | 03:47 | |
| *** exsdev has quit IRC | 03:48 | |
| *** exsdev0 is now known as exsdev | 03:48 | |
| *** tkajinam has joined #openstack-infra | 03:49 | |
| *** markvoelker has quit IRC | 03:49 | |
| *** ykarel has joined #openstack-infra | 04:08 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** fungi has quit IRC | 04:21 | |
| *** fungi has joined #openstack-infra | 04:25 | |
| *** yamamoto has quit IRC | 04:33 | |
| *** yamamoto has joined #openstack-infra | 04:35 | |
| *** yamamoto has quit IRC | 04:40 | |
| *** dave-mccowan has quit IRC | 04:41 | |
| *** kjackal has quit IRC | 04:54 | |
| *** odicha has joined #openstack-infra | 04:57 | |
| *** yamamoto has joined #openstack-infra | 05:11 | |
| *** ykarel is now known as ykarel|away | 05:16 | |
| *** bnemec has quit IRC | 05:19 | |
| *** elod has quit IRC | 05:26 | |
| *** armax has joined #openstack-infra | 05:40 | |
| *** markvoelker has joined #openstack-infra | 05:50 | |
| *** markvoelker has quit IRC | 05:55 | |
| *** noama has joined #openstack-infra | 05:56 | |
| *** yamamoto has quit IRC | 05:58 | |
| *** psachin has joined #openstack-infra | 05:59 | |
| *** yamamoto has joined #openstack-infra | 06:01 | |
| *** armax has quit IRC | 06:08 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** slaweq has quit IRC | 06:15 | |
| *** dtantsur|afk is now known as dtantsur | 06:28 | |
| *** kjackal has joined #openstack-infra | 06:31 | |
| *** ccamacho has quit IRC | 06:35 | |
| *** slaweq has joined #openstack-infra | 06:35 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Add stable/train to periodic-stable templates https://review.opendev.org/687122 | 06:37 |
| *** pgaxatte has joined #openstack-infra | 06:37 | |
| *** jaosorior has quit IRC | 06:41 | |
| *** trident has quit IRC | 06:46 | |
| *** trident has joined #openstack-infra | 06:49 | |
| *** yamamoto has quit IRC | 07:04 | |
| *** yamamoto has joined #openstack-infra | 07:05 | |
| *** rcernin has quit IRC | 07:05 | |
| *** tesseract has joined #openstack-infra | 07:10 | |
| *** pkopec has joined #openstack-infra | 07:16 | |
| *** jtomasek has joined #openstack-infra | 07:21 | |
| *** pkopec has quit IRC | 07:22 | |
| *** iurygregory has joined #openstack-infra | 07:23 | |
| *** tosky has joined #openstack-infra | 07:24 | |
| *** jaosorior has joined #openstack-infra | 07:24 | |
| *** ccamacho has joined #openstack-infra | 07:26 | |
| *** ccamacho has quit IRC | 07:27 | |
| *** ccamacho has joined #openstack-infra | 07:27 | |
| *** pkopec has joined #openstack-infra | 07:34 | |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try) https://review.opendev.org/674334 | 07:38 |
| *** lpetrut has joined #openstack-infra | 07:45 | |
| *** kaisers has joined #openstack-infra | 07:47 | |
| *** jpena|off is now known as jpena | 07:47 | |
| *** ralonsoh has joined #openstack-infra | 07:48 | |
| *** ociuhandu has joined #openstack-infra | 08:16 | |
| *** dchen has quit IRC | 08:20 | |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try) https://review.opendev.org/674334 | 08:28 |
| *** tkajinam has quit IRC | 08:30 | |
| *** e0ne has joined #openstack-infra | 08:31 | |
| *** jaosorior has quit IRC | 08:32 | |
| *** prometheanfire has quit IRC | 08:37 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| *** rpittau|afk is now known as rpittau | 08:38 | |
| *** yamamoto has quit IRC | 08:40 | |
| *** yamamoto has joined #openstack-infra | 08:42 | |
| *** xek has joined #openstack-infra | 08:43 | |
| *** yamamoto has quit IRC | 08:43 | |
| *** priteau has joined #openstack-infra | 08:47 | |
| *** odicha has quit IRC | 08:47 | |
| *** odicha has joined #openstack-infra | 08:49 | |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try) https://review.opendev.org/674334 | 08:51 |
| *** e0ne has quit IRC | 08:52 | |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunits (2nd try) https://review.opendev.org/674334 | 08:57 |
| *** ociuhandu has quit IRC | 08:57 | |
| openstackgerrit | Merged opendev/irc-meetings master: Add gouthamr as co-chair for manila meetings https://review.opendev.org/687171 | 08:58 |
| *** xenos76 has joined #openstack-infra | 09:02 | |
| *** e0ne has joined #openstack-infra | 09:13 | |
| jrosser | for browsing logs it would be really nice to have a "parent directory" link | 09:24 |
| jrosser | pressing the browser back button here throws you all the way to the top of the tree https://zuul.opendev.org/t/openstack/build/7a25df1fe3fe4b29aa8297d16bd99cd9/log/logs/etc/host/apt/sources.list.d/default.list.txt.gz | 09:24 |
| *** yamamoto has joined #openstack-infra | 09:28 | |
| *** yamamoto_ has joined #openstack-infra | 09:29 | |
| *** pcaruana has joined #openstack-infra | 09:30 | |
| *** ociuhandu has joined #openstack-infra | 09:30 | |
| *** iurygregory has quit IRC | 09:31 | |
| *** rfolco has joined #openstack-infra | 09:32 | |
| *** yamamoto has quit IRC | 09:33 | |
| *** yamamoto_ has quit IRC | 09:34 | |
| *** ociuhandu has quit IRC | 09:35 | |
| *** ociuhandu has joined #openstack-infra | 09:43 | |
| *** jaosorior has joined #openstack-infra | 09:44 | |
| *** iurygregory has joined #openstack-infra | 09:52 | |
| *** markvoelker has joined #openstack-infra | 09:53 | |
| *** markvoelker has quit IRC | 09:58 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - Manage project connector refresh when EINVALIDTOK https://review.opendev.org/687259 | 10:01 |
| *** yamamoto has joined #openstack-infra | 10:05 | |
| *** slaweq_ has joined #openstack-infra | 10:09 | |
| *** ykarel|away has quit IRC | 10:09 | |
| *** slaweq has quit IRC | 10:09 | |
| *** yamamoto has quit IRC | 10:10 | |
| openstackgerrit | Merged openstack/diskimage-builder master: simple-init: default to NetworkManager for CentOS and Fedora https://review.opendev.org/686319 | 10:17 |
| *** jaosorior has quit IRC | 10:18 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Assign static 'building' nodes in cleanup handler https://review.opendev.org/687261 | 10:31 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/devstack-gate master: Update grenade settings for stable/train https://review.opendev.org/687072 | 10:32 |
| *** whoami-rajat has joined #openstack-infra | 10:42 | |
| *** dklyle has quit IRC | 10:49 | |
| *** dklyle has joined #openstack-infra | 10:49 | |
| *** rcernin has joined #openstack-infra | 10:52 | |
| *** yamamoto has joined #openstack-infra | 10:56 | |
| *** ociuhandu has quit IRC | 10:56 | |
| *** ociuhandu has joined #openstack-infra | 11:05 | |
| *** gfidente has quit IRC | 11:09 | |
| *** yamamoto has quit IRC | 11:10 | |
| *** jamesdenton has quit IRC | 11:11 | |
| *** gfidente has joined #openstack-infra | 11:11 | |
| *** jamesden_ has joined #openstack-infra | 11:12 | |
| *** noama has quit IRC | 11:13 | |
| *** yamamoto has joined #openstack-infra | 11:14 | |
| *** slaweq_ is now known as slaweq | 11:15 | |
| *** jamesmcarthur has joined #openstack-infra | 11:15 | |
| *** xek has quit IRC | 11:15 | |
| *** xenos76 has quit IRC | 11:17 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Sort waiting static nodes by creation time https://review.opendev.org/687271 | 11:18 |
| *** smarcet has joined #openstack-infra | 11:18 | |
| *** jamesden_ has quit IRC | 11:19 | |
| *** jamesdenton has joined #openstack-infra | 11:19 | |
| *** jamesmcarthur has quit IRC | 11:19 | |
| *** jamesdenton has quit IRC | 11:24 | |
| *** jamesdenton has joined #openstack-infra | 11:24 | |
| *** smarcet has quit IRC | 11:25 | |
| *** dave-mccowan has joined #openstack-infra | 11:27 | |
| *** ociuhandu has quit IRC | 11:34 | |
| *** ociuhandu has joined #openstack-infra | 11:35 | |
| *** ociuhandu has quit IRC | 11:40 | |
| *** jpena is now known as jpena|lunch | 11:41 | |
| *** smarcet has joined #openstack-infra | 11:43 | |
| *** smarcet has quit IRC | 11:49 | |
| *** markvoelker has joined #openstack-infra | 11:54 | |
| *** ociuhandu has joined #openstack-infra | 11:56 | |
| *** markvoelker has quit IRC | 11:59 | |
| *** markvoelker has joined #openstack-infra | 12:03 | |
| *** elod has joined #openstack-infra | 12:05 | |
| *** jaosorior has joined #openstack-infra | 12:08 | |
| *** jaosorior has quit IRC | 12:08 | |
| *** jaosorior has joined #openstack-infra | 12:08 | |
| *** ociuhandu has quit IRC | 12:11 | |
| *** ociuhandu has joined #openstack-infra | 12:17 | |
| *** roman_g has joined #openstack-infra | 12:17 | |
| *** xenos76 has joined #openstack-infra | 12:19 | |
| openstackgerrit | Merged openstack/diskimage-builder master: simple-init: Use wrappers to call pip for glean install https://review.opendev.org/685643 | 12:21 |
| *** yamamoto has quit IRC | 12:22 | |
| *** jamesmcarthur has joined #openstack-infra | 12:26 | |
| *** xenos76 has quit IRC | 12:26 | |
| *** yamamoto has joined #openstack-infra | 12:29 | |
| *** jamesmcarthur has quit IRC | 12:31 | |
| *** jpena|lunch is now known as jpena | 12:37 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| *** ociuhandu has quit IRC | 12:44 | |
| *** jamesmcarthur has joined #openstack-infra | 12:51 | |
| *** aaronsheffield has joined #openstack-infra | 12:51 | |
| *** pcaruana has quit IRC | 12:53 | |
| *** mriedem has joined #openstack-infra | 12:59 | |
| openstackgerrit | Tristan Cacqueray proposed opendev/glean master: Import ipv6 configuration done by cloud-init for Red Hat distro https://review.opendev.org/686749 | 13:07 |
| *** lpetrut has quit IRC | 13:08 | |
| *** smarcet has joined #openstack-infra | 13:15 | |
| *** rh-jelabarre has joined #openstack-infra | 13:15 | |
| *** rcernin has quit IRC | 13:19 | |
| *** xenos76 has joined #openstack-infra | 13:21 | |
| *** goldyfruit has joined #openstack-infra | 13:28 | |
| *** xek has joined #openstack-infra | 13:36 | |
| *** goldyfruit_ has joined #openstack-infra | 13:36 | |
| *** goldyfruit has quit IRC | 13:39 | |
| *** bnemec has joined #openstack-infra | 13:43 | |
| *** xenos76 has quit IRC | 13:48 | |
| *** yamamoto has quit IRC | 13:48 | |
| *** pgaxatte has quit IRC | 13:49 | |
| openstackgerrit | Merged zuul/nodepool master: Use real uuids in fake cloud resource IDs https://review.opendev.org/687144 | 13:53 |
| *** ociuhandu has joined #openstack-infra | 13:56 | |
| *** ykarel|awat has joined #openstack-infra | 13:56 | |
| *** ykarel|awat is now known as ykarel|away | 13:56 | |
| *** yamamoto has joined #openstack-infra | 13:56 | |
| *** yamamoto has quit IRC | 13:57 | |
| *** yamamoto has joined #openstack-infra | 13:57 | |
| *** smarcet has quit IRC | 13:57 | |
| *** pgaxatte has joined #openstack-infra | 14:00 | |
| *** yamamoto has quit IRC | 14:01 | |
| *** diablo_rojo has joined #openstack-infra | 14:01 | |
| *** sthussey has joined #openstack-infra | 14:02 | |
| *** odicha has quit IRC | 14:06 | |
| *** fdegir has quit IRC | 14:06 | |
| *** georgk has quit IRC | 14:06 | |
| *** jamesmcarthur has quit IRC | 14:06 | |
| *** georgk has joined #openstack-infra | 14:07 | |
| *** fdegir has joined #openstack-infra | 14:07 | |
| *** smarcet has joined #openstack-infra | 14:07 | |
| *** smarcet has quit IRC | 14:11 | |
| *** smarcet has joined #openstack-infra | 14:14 | |
| *** ykarel|away is now known as ykarel | 14:15 | |
| stephenfin | AJaeger: Any idea where the zuul jobs used by oslo.config are defined? | 14:26 |
| stephenfin | There's no .zuul.yaml file in the repo and I don't see anything in openstack/project-config | 14:26 |
| bnemec | stephenfin: https://github.com/openstack/oslo.config/blob/master/.zuul.d/project.yaml | 14:28 |
| bnemec | Not sure why we used a directory for that one. | 14:29 |
| *** smarcet has quit IRC | 14:29 | |
| *** smarcet has joined #openstack-infra | 14:30 | |
| stephenfin | Ah, I did 'git checkout <branch>' instead of 'git checkout -b <branch>', and it turned out I had an ancient branch with that name already | 14:30 |
| AJaeger | bnemec: with a single file in it, a directory is really not needed.... | 14:30 |
| stephenfin | so I was looking at 2018 code | 14:30 |
| stephenfin | bnemec++ thanks | 14:30 |
| AJaeger | that explains a lot ;) | 14:30 |
| bnemec | Yeah, most of our projects just use .zuul.yaml. oslo.config must have been done separately for some reason. | 14:32 |
| *** smarcet has quit IRC | 14:34 | |
| *** pgaxatte has quit IRC | 14:36 | |
| *** jamesmcarthur has joined #openstack-infra | 14:37 | |
| *** ociuhandu has quit IRC | 14:40 | |
| *** yamamoto has joined #openstack-infra | 14:45 | |
| *** pcaruana has joined #openstack-infra | 14:46 | |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 14:46 |
| openstackgerrit | Bharat Kunwar proposed opendev/irc-meetings master: Add agenda_url for container team weekly meeting https://review.opendev.org/687317 | 14:46 |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 14:50 |
| *** ociuhandu has joined #openstack-infra | 14:51 | |
| *** yamamoto has quit IRC | 14:51 | |
| *** ociuhandu has quit IRC | 14:51 | |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 14:52 |
| *** ociuhandu has joined #openstack-infra | 14:52 | |
| *** ociuhandu has quit IRC | 14:57 | |
| *** ociuhandu has joined #openstack-infra | 14:57 | |
| *** ykarel is now known as ykarel|away | 15:01 | |
| *** ociuhandu has quit IRC | 15:17 | |
| *** gyee has joined #openstack-infra | 15:21 | |
| *** jamesmcarthur_ has joined #openstack-infra | 15:23 | |
| *** markvoelker has quit IRC | 15:24 | |
| *** jamesmcarthur has quit IRC | 15:26 | |
| *** iurygregory has quit IRC | 15:31 | |
| openstackgerrit | Clark Boylan proposed opendev/glean master: Import ipv6 configuration done by cloud-init for Red Hat distro https://review.opendev.org/686749 | 15:37 |
| clarkb | tristanC: ^ I think that fixes a few bugs with the change. The append in the new block was invalid as you can't append to a str (CI caught this) but also we need to update those records not replace them (dict.update overwrites) as a single interface can have multiple connection types | 15:38 |
| *** smarcet has joined #openstack-infra | 15:39 | |
| *** xenos76 has joined #openstack-infra | 15:40 | |
| tristanC | clarkb: ok thanks | 15:44 |
| clarkb | corvus: are you under the power shutdown notice for tomorrow and thursday? | 15:44 |
| *** kjackal has quit IRC | 15:45 | |
| corvus | clarkb: unclear :/ the somewhat vague map does not have a blob shape over my house, so that's good. but it's close. and the actual address lookup tool is apparently being hugged to death. | 15:45 |
| *** kopecmartin is now known as kopecmartin|off | 15:47 | |
| *** rpittau is now known as rpittau|afk | 15:47 | |
| *** kjackal has joined #openstack-infra | 15:48 | |
| *** smarcet has quit IRC | 15:54 | |
| *** e0ne has quit IRC | 15:55 | |
| *** anteaya has joined #openstack-infra | 15:56 | |
| donnyd | power shutdown? is that a real thing? | 15:56 |
| clarkb | donnyd: ya apparently most of the bay area is being warned they could lose power for extended periods of time over the next few days due to hot, dry, and windy weather increasing fire danger dramatically | 15:57 |
| donnyd | what... so because there is a fire warning they may turn off your power.. that is one part of CA I don't miss... But there are a lot of parts I do | 15:58 |
| paladox | Why would they shut it off because of the hot/dry/windy weather? | 15:59 |
| clarkb | paladox: because last year major fires were started by faulty power transmission equipment | 16:00 |
| paladox | oh | 16:00 |
| *** smarcet has joined #openstack-infra | 16:00 | |
| corvus | because the utility pays dividends to shareholders instead of maintaining their infrastructure causing that ^ | 16:00 |
| donnyd | clarkb: was that the fires in redding? | 16:01 |
| * paladox is glad we have a regulator | 16:01 | |
| corvus | paladox: oh, we do too -- they approved the rate hike to pay for the lawsuits due to the fires | 16:01 |
| paladox | oh | 16:01 |
| paladox | corvus we had a power outage apparently a few months ago | 16:02 |
| paladox | that affected big parts of england | 16:02 |
| paladox | though i didn't experence that :P | 16:02 |
| openstackgerrit | Merged opendev/irc-meetings master: Add agenda_url for container team weekly meeting https://review.opendev.org/687317 | 16:02 |
| clarkb | donnyd: no, was the camp fire | 16:02 |
| *** goldyfruit___ has joined #openstack-infra | 16:02 | |
| donnyd | that is a little crazy.... but I know a guy who can help you out getting a gen set running for cheap 😉 | 16:03 |
| *** odicha has joined #openstack-infra | 16:03 | |
| *** dtantsur is now known as dtantsur|afk | 16:03 | |
| corvus | and also the tubbs fire in santa rosa the year before | 16:03 |
| paladox | heh | 16:03 |
| paladox | you need a regulator like ofgem! | 16:04 |
| donnyd | Are you on PG&E? | 16:04 |
| *** prometheanfire has joined #openstack-infra | 16:04 | |
| corvus | donnyd: yes, they own the transmission lines for most of the north of the state. | 16:05 |
| corvus | also the middle | 16:05 |
| *** goldyfruit_ has quit IRC | 16:05 | |
| donnyd | clarkb: one would think that instead of turning off everyone's critical service... they would FIX THE FAULTY EQUIPMENT | 16:05 |
| donnyd | corvus: oh and they own the south too | 16:05 |
| *** Vadmacs has joined #openstack-infra | 16:05 | |
| donnyd | at least where I was at | 16:05 |
| corvus | there are other power suppliers, but they all rely on pge's transmission lines | 16:05 |
| corvus | donnyd: this is more or less the conversation that's been happening in ca for the past few years | 16:06 |
| donnyd | Live in ventura county and SLO county for close to a decade - I miss the weather and the scenery.. but the rest of it could go IMO | 16:06 |
| corvus | to be honest, i haven't heard anyone put forward the other side of that argument, yet, somehow it's the one that seems to be winning | 16:06 |
| clarkb | as a non californian I haven't kept up with this much, really only noticed because my parents are in the bay area as of yesterday | 16:06 |
| *** odicha has quit IRC | 16:07 | |
| paladox | How will hospitals cope? | 16:07 |
| donnyd | generator power i imagine | 16:07 |
| fungi | tehy generally get priority power feeds, but also have generators yes | 16:08 |
| donnyd | most hospitals have gen sets that can run them off grid for quite a while | 16:08 |
| donnyd | some a couple days, others a couple weeks | 16:08 |
| fungi | yeah, but also power companies tend to not cut power to hospitals | 16:08 |
| mriedem | clarkb: is there still a report somewhere of active 404s to docs? | 16:08 |
| clarkb | donnyd: the fire in redding was apparently caused by a bad trailer axle or something like that | 16:08 |
| *** tesseract has quit IRC | 16:08 | |
| corvus | people with electrically operated medical devices at home are in a tricky spot too | 16:08 |
| *** mattw4 has joined #openstack-infra | 16:09 | |
| clarkb | mriedem: there is, let me find it for you | 16:09 |
| donnyd | yea.. that is a bad plan.. but if there is power running to the hospital, then "faulty" lines are active and their logic is not great | 16:09 |
| paladox | During the power outage in august, one of the hospitals generator apparently failed.. | 16:09 |
| paladox | *here | 16:09 |
| corvus | paladox: ouch :( | 16:09 |
| fungi | usually they place substations near hospitals, feed them as directly as possible, and shut the power downstream from there when needed | 16:10 |
| donnyd | that is what gen sets do best... besides make power | 16:10 |
| clarkb | mriedem: http://files.openstack.org/docs-404s/ | 16:10 |
| mriedem | clarkb: thanks | 16:10 |
| donnyd | fungi: right.. but they have to get power to that sub just like every other sub from the same place... If the fire issue is downstream from the sub... which are mostly located in non flamable areas... it still doesn't quite add up to me | 16:11 |
| fungi | yup | 16:11 |
| paladox | https://www.ft.com/content/1308e19c-bdf4-11e9-89e2-41e555e96722 | 16:11 |
| paladox | corvus ^ | 16:11 |
| fungi | though also just reducing load on the equipment by turning off downstream branches likely helps reduce the risk | 16:12 |
| donnyd | maybe | 16:12 |
| *** ociuhandu has joined #openstack-infra | 16:12 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed opendev/elastic-recheck master: Add querry for 1847301 https://review.opendev.org/687346 | 16:12 |
| corvus | clarkb: the map thingy finally responded and has said that our address is not under the shutoff notice. | 16:15 |
| *** ociuhandu_ has joined #openstack-infra | 16:15 | |
| *** smarcet has quit IRC | 16:16 | |
| corvus | which i read as suggesting there is a non-zero chance i may continue to have power tomorrow. | 16:16 |
| paladox | When tracking a power outage here i use https://powercuts.westernpower.co.uk/tweets | 16:17 |
| corvus | paladox: hopefully twitter doesn't lose power tomorrow :) | 16:17 |
| paladox | lol | 16:17 |
| paladox | also when you get a powercut call 105 :P | 16:17 |
| *** ociuhandu has quit IRC | 16:18 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - add the enqueue_ref unit test https://review.opendev.org/687351 | 16:19 |
| *** ociuhandu_ has quit IRC | 16:22 | |
| *** odicha has joined #openstack-infra | 16:24 | |
| *** igordc has quit IRC | 16:25 | |
| *** roman_g has quit IRC | 16:29 | |
| *** rfolco is now known as not_rlandy | 16:29 | |
| *** not_rlandy is now known as rfolco | 16:29 | |
| *** smarcet has joined #openstack-infra | 16:32 | |
| *** odicha has quit IRC | 16:32 | |
| openstackgerrit | Clark Boylan proposed opendev/glean master: Import ipv6 configuration done by cloud-init for Red Hat distro https://review.opendev.org/686749 | 16:34 |
| *** odicha has joined #openstack-infra | 16:34 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/devstack-gate master: Add gate job for stable/train https://review.opendev.org/686776 | 16:37 |
| *** odicha has quit IRC | 16:42 | |
| *** factor has quit IRC | 16:43 | |
| tristanC | fungi: clarkb: re: cloud-init, iirc devstack can do system-wide pip install and override cloud-init dependencies without much issues as the cloud-init service should be dormant after init. Though i understand it's quite a change and fixing glean is probably the easiest thing to do now | 16:49 |
| clarkb | tristanC: that was true before pip updated its rules for replacing system packages. It won't do it for any distutils installed packages whcih is a problem particularly on centos 7 last I checked | 16:51 |
| clarkb | tristanC: basically pip asserts it can't remove the existing install because it doesn't have a manifest of files to remove | 16:51 |
| clarkb | and it fails | 16:51 |
| tristanC | fungi: clarkb: but i still think using cloud-init is the right thing to do because that's what non opendev-ci system usually uses, and if devstack can only work with glean then that is something that should be documented | 16:51 |
| *** derekh has quit IRC | 16:52 | |
| clarkb | well devstack also pinned pip back and I'm not sure it ever updated (due to other conflicts in the system) | 16:55 |
| fungi | tristanC: yeah, i don't have anything against using cloud-init, but i expect we can't install the distro packages of it for a couple reasons... one is that we don't want all its dependencies installed system-wide but the other is that we probably want to use consistent versions of it across all the distros for which we build images so that we get consistent behavior and newer features required to | 16:55 |
| clarkb | so there shouldn't be anything to document | 16:55 |
| fungi | have proper network configuration in some providers | 16:55 |
| tristanC | fungi: but then only the non-python bits provided by the distros are actually tested | 17:00 |
| tristanC | if we want consistent version, why not doing the same for the kernel, libc, etc... | 17:00 |
| *** ricolin has quit IRC | 17:03 | |
| clarkb | we arent testing cloud init or glean... | 17:04 |
| clarkb | ts there to get the test env up | 17:04 |
| fungi | yes, if the goal of devstack were to test the dependency versions packaged by distros, then it should be switched to use distro packages for everything it's not installing from source | 17:05 |
| fungi | however, the way our current python dependency constraints are designed, that won't work | 17:06 |
| fungi | because centos-7 with cloud-init installed will have an older version of some dependencies in the python import path than is required in upper-constraints.txt, and pip will abort refusing to replace them with wheels from pypi because it can't safely uninstall the existing versions | 17:07 |
| *** priteau has quit IRC | 17:07 | |
| fungi | devstack *does* try to counter this by attempting to uninstall system packages of libraries which are known problems, i think, but as clarkb mentioned i believe we've never un-pinned (now ancient) pip from before the distutils-installed package break was implemented | 17:09 |
| tristanC | fungi: then wouldn't it be better to do that upper-constraints.txt workaround (e.g. replace system package manually before pip does its thing) by a zuul-jobs roles for the job that actually enforce the constrains? | 17:12 |
| fungi | not sure what you mean... edit upper-constraints.txt to reflect the versions in the distro at runtime? | 17:12 |
| fungi | note that doesn't help if transient dependencies are declared incompatible with the reported versions, just means pip will break in a different way because it can't resolve a coherent set of packages to install | 17:13 |
| tristanC | fungi: not that, i meant to implement the pip-and-virtualenv-source-install dib element as a runtime zuul-jobs roles | 17:15 |
| fungi | oh, got it, edit the files on disk before starting devstack | 17:16 |
| fungi | that works as long as the preinstalled versions don't change. the more packages we do that to, the more often we'll be playing whack-a-mole with gate breaking updates from distros | 17:16 |
| tristanC | like so we can keep the distro stuff on boot (and use cloud-init), then only the jobs that enforce specific version that pip can't install would need to do that pip-and-virtualenv-source-install stuff | 17:17 |
| fungi | ahh, looking at diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip it's just trying to use system package managers to uninstall packages | 17:19 |
| fungi | oh, nope, the bits i was thinking of are in there too | 17:20 |
| fungi | line 88, where it also manually deletes the files for them from disk on some platforms | 17:20 |
| fungi | yeah, i guess that all looks version-agnostic so might not need updating too often | 17:20 |
| tristanC | i think we could just do that at runtime for the job that enforces specifics setuptools/pip version | 17:21 |
| fungi | but it does present a lot of complexity... over 100 lines of shell not including comments | 17:21 |
| *** ralonsoh has quit IRC | 17:22 | |
| *** xek_ has joined #openstack-infra | 17:23 | |
| fungi | yeah, could test that with a throwaway change which does `yum install cloud-init` and then a change to add a role for uninstalling/removing those files, and add them to a throwaway devstack change which adds those roles in the pre phase | 17:23 |
| *** markvoelker has joined #openstack-infra | 17:25 | |
| *** xek has quit IRC | 17:25 | |
| *** dave-mccowan has quit IRC | 17:28 | |
| *** ccamacho has quit IRC | 17:29 | |
| tristanC | i created https://storyboard.openstack.org/#!/story/2006681 about this, perhaps we can find sometime to investigate this | 17:29 |
| *** markvoelker has quit IRC | 17:29 | |
| fungi | actually, a throwaway openstack/requirements change might be a better test, since it has jobs which try to install the full upper-constraints.txt set | 17:31 |
| pabelanger | tristanC: fungi: look to the work that ianw_pto is doing now for centos-8 support there is a large change of pip-and-vitualenv going on now. topic:centos8 that is also updating simple-init elemets too | 17:31 |
| *** kjackal has quit IRC | 17:32 | |
| pabelanger | so could be a good time to also make other changes | 17:32 |
| *** kjackal has joined #openstack-infra | 17:32 | |
| *** jpena is now known as jpena|off | 17:32 | |
| fungi | good call | 17:33 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add a relnote about gerrit and http https://review.opendev.org/687365 | 17:33 |
| jrosser | I’m slightly lacking context but are you looking for ansible to manage pip/virtualenv and apply constraints? | 17:38 |
| *** roman_g has joined #openstack-infra | 17:40 | |
| fungi | jrosser: maybe, but that's getting into implementation details | 17:46 |
| *** igordc has joined #openstack-infra | 17:46 | |
| jrosser | ok well do check out what’s done in other projects when it comes to it | 17:47 |
| fungi | what we're looking for at a higher level is how to go about possibly coping with distro-installed python dependencies for things like cloud-init when we have jobs which want newer versions of the same modules installed system-wide from pypi | 17:47 |
| fungi | right now we hack around it for pip and virtualenv at image creation, but if we consider replacing glean (in a venv) with cloud-init (from distro packages) the problem gets much larger | 17:47 |
| jrosser | I just mention it because sorts of stuff discussed is business as usual work for deployment projects that target multiple distros, perhaps just in a different context | 17:48 |
| *** dave-mccowan has joined #openstack-infra | 17:48 | |
| fungi | yep, it's definitely something we should keep in mind | 17:49 |
| fungi | i don't think we want to reinvent the wheel there, but hard to know which solution is a fit before deciding what needs to be solved | 17:50 |
| *** eernst has joined #openstack-infra | 17:52 | |
| fungi | jrosser: as a data point, how much additional complexity do you think it would add to your testing if our images started coming with cloud-init and all its dependencies baked in from distro packages? | 17:53 |
| fungi | (python-requests, pyyaml and so on) | 17:53 |
| jrosser | from an OSA perspective we don’t mess with the host, nor use the distro packages beyond bootstrapping a venv | 17:55 |
| fungi | okay, so presumably not much then | 17:55 |
| jrosser | so long as there is a functional pip/virtualenv package available then we decouple ourselves from the host setup as soon as possible | 17:55 |
| fungi | yeah, looks like major dependencies of cloud-init include jinja2, jsonpatch, jsonschema, oauthlib, requests, six and pyyaml | 17:56 |
| jrosser | history has taught us that messing with whatever the distro provides is a route to bad times | 17:57 |
| fungi | i couldn't agree more, but this road was paved a long time ago | 17:58 |
| jrosser | just to for clarity we deploy both from source or distro packages, so both are in scope for us | 18:00 |
| jrosser | but the ansible part and it’s supporting libraries are always in a well defined venv | 18:01 |
| fungi | right, i wish that's how devstack had been designed early on | 18:01 |
| *** jamesmcarthur_ has quit IRC | 18:02 | |
| fungi | but given its vintage that wasn't so easy to accomplish back then | 18:02 |
| fungi | i guess what i mean to say, i wish it had been an option for devstack 7+ years ago | 18:02 |
| *** jamesmcarthur has joined #openstack-infra | 18:02 | |
| fungi | the python packaging ecosystem was in a much different (and far less coherent) state back then | 18:03 |
| fungi | we had more than a few python dependencies which could *only* be installed outside a virtualenv | 18:03 |
| jrosser | some of that still exists | 18:04 |
| fungi | yeah, i struggle with some personal use python software which fits that description. especially anything that relies on gobject | 18:04 |
| jrosser | like librbd python bindings which are so fundamentally coupled to their parent shared library you need to treat them specially | 18:04 |
| fungi | at least libvirt finally started releasing separate python packages for their python bindings | 18:05 |
| clarkb | ttied to get a bike ride in before thr rain. succeeded at riding bike failed at keeping dry | 18:05 |
| *** markvoelker has joined #openstack-infra | 18:06 | |
| fungi | you're 2/3 of the way to training for a triathlon i guess | 18:06 |
| fungi | now if only you'd gotten a flat tire and needed to run home | 18:06 |
| clarkb | ha | 18:07 |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 18:07 |
| *** gfidente is now known as gfidente|afk | 18:10 | |
| *** jamesmcarthur has quit IRC | 18:14 | |
| clarkb | fwiw glean has actually worked really well. It can be painful when it doesn't do what we want or there are systemd races and I think that makes replacing it with cloud-init a worthwhile investigation. But I don't think we need to replace it because it is failing at its job | 18:26 |
| clarkb | I think https://review.opendev.org/#/c/686749/10 is ready for real world testing. The only failure it hit now was due to f29 package update failures during image build | 18:28 |
| *** mriedem has quit IRC | 18:31 | |
| *** mriedem has joined #openstack-infra | 18:32 | |
| fungi | yeah, the only compelling reason i can see to replace glean is if cloud-init now does all the things we needed glean to do (and can be configured to do none of the things we needed glean specifically *not* to do), then it relieves us of maintaining yet one more duplicative piece of software | 18:33 |
| *** sreejithp has joined #openstack-infra | 18:34 | |
| clarkb | also worth noting that the current issues have to do with configuring network manager whcih is apparently some arcane artform | 18:40 |
| *** eernst has quit IRC | 18:44 | |
| *** eernst has joined #openstack-infra | 18:45 | |
| fungi | the last time i tried to use nm i summoned an eldritch horror from beyond space and time. luckily i had an elder sign etched into my network card | 18:46 |
| clarkb | I have nm problems on my desktop. It won't configure dns from dhcp. I spent hours trying to figure it out. Then I realized I can just edit /etc/resolv.conf and because it isn't updating the file whatever I set there will just live there forever | 18:47 |
| *** yamamoto has joined #openstack-infra | 18:49 | |
| *** markvoelker has quit IRC | 18:50 | |
| *** yamamoto has quit IRC | 18:53 | |
| clarkb | it also doesn't set search domains | 18:54 |
| *** markvoelker has joined #openstack-infra | 18:54 | |
| clarkb | basically it seems to only get an ip, mask, and router on this particular host. Very odd | 18:54 |
| *** ianw_pto is now known as ianw | 18:59 | |
| *** ykarel|away has quit IRC | 19:00 | |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 19:07 |
| *** dave-mccowan has quit IRC | 19:10 | |
| openstackgerrit | Merged zuul/zuul master: Add a relnote about gerrit and http https://review.opendev.org/687365 | 19:13 |
| *** smarcet has quit IRC | 19:14 | |
| *** jamesmcarthur has joined #openstack-infra | 19:17 | |
| *** prometheanfire has quit IRC | 19:24 | |
| *** prometheanfire has joined #openstack-infra | 19:25 | |
| *** ociuhandu has joined #openstack-infra | 19:26 | |
| *** ociuhandu has quit IRC | 19:31 | |
| *** whoami-rajat has quit IRC | 19:31 | |
| *** psachin has quit IRC | 19:33 | |
| openstackgerrit | Merged openstack/devstack-gate master: Update grenade settings for stable/train https://review.opendev.org/687072 | 19:33 |
| *** trident has quit IRC | 19:51 | |
| *** jamesmcarthur has quit IRC | 19:52 | |
| *** jamesmcarthur has joined #openstack-infra | 19:53 | |
| *** trident has joined #openstack-infra | 19:54 | |
| smcginnis | Doesn't appear http://zuul.openstack.org/ redirection is working anymore. | 19:56 |
| clarkb | smcginnis: it shouldn't redirect you, we host the zuul dashboard for the openstack tenant at that location (and it is working for me both via http and https) | 19:57 |
| smcginnis | Hmm, I'll try a hard refresh. | 19:58 |
| *** jamesmcarthur has quit IRC | 19:58 | |
| smcginnis | That was it ¯\_(ツ)_/¯ | 19:58 |
| smcginnis | Nothing to see here. Move along. | 19:58 |
| *** jtomasek has quit IRC | 19:58 | |
| fungi | yeah, https://zuul.openstack.org/ is a white-labeled dashboard for just the openstack tenant, but basically equivalent to https://zuul.opendev.org/t/openstack/status | 20:01 |
| clarkb | apparently my local libvirtd does not want to work some reason or another. I'm going to reboot to see if kernel updates or similar are to blame | 20:01 |
| clarkb | back in a bit | 20:01 |
| clarkb | (I like to use a local vm to do dib builds) | 20:01 |
| *** weifan has joined #openstack-infra | 20:03 | |
| *** weifan has quit IRC | 20:03 | |
| *** pkopec has quit IRC | 20:04 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix networking for CentOS 8 https://review.opendev.org/684513 | 20:05 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add NetworkManager and dhcp-client for CentOS 8 https://review.opendev.org/686090 | 20:05 |
| ianw | clarkb: that's the only sane way :) | 20:05 |
| *** smarcet has joined #openstack-infra | 20:05 | |
| clarkb | apparmor is debying creation of the socket? | 20:06 |
| *** Vadmacs has quit IRC | 20:09 | |
| fungi | we enable apparmor on our images? | 20:10 |
| *** jamesmcarthur has joined #openstack-infra | 20:15 | |
| mriedem | any idea if docs builds handle redirects in the output or is that only something that happens during docs publish? | 20:15 |
| mriedem | because i'm trying to verify this fix https://review.opendev.org/#/c/687353/2/doc/source/_extra/.htaccess | 20:15 |
| mriedem | and the resulting docs built from the job still hit a 404 | 20:15 |
| *** prometheanfire has quit IRC | 20:15 | |
| *** kjackal has quit IRC | 20:15 | |
| clarkb | fungi: no I have it enabled on my local desktop | 20:16 |
| *** rlandy has quit IRC | 20:16 | |
| clarkb | fungi: where I want to run a vm | 20:16 |
| fungi | ahh | 20:16 |
| clarkb | mriedem: I think it is only in the publish because it isn't safe for servers to generally follow htaccess (and swift likely isn't doing so) | 20:16 |
| mriedem | ack, thanks | 20:17 |
| fungi | though i think there's a tool somewhere (maybe the openstack-manuals or openstack-docs-theme repo?) to validate htaccess rules | 20:18 |
| fungi | not sure what it does | 20:18 |
| fungi | oh, makes use of the "whereto" utility i believe | 20:19 |
| *** mtreinish has quit IRC | 20:21 | |
| *** e0ne has joined #openstack-infra | 20:23 | |
| clarkb | I've disabled the profile and yet it still kicks out and audit log shows that there is denying happening. I don't understand apparmor I guess | 20:27 |
| *** e0ne has quit IRC | 20:28 | |
| *** smarcet has quit IRC | 20:32 | |
| *** xek_ has quit IRC | 20:34 | |
| *** pcaruana has quit IRC | 20:35 | |
| *** smarcet has joined #openstack-infra | 20:38 | |
| clarkb | I have no idea how to disable apparmor profiles or apparmor itself | 20:44 |
| clarkb | stopping the service didnt work neither did aa-disable. I have disabled the service and am rebooting | 20:45 |
| *** prometheanfire has joined #openstack-infra | 20:46 | |
| fungi | yeah, needs a reboot probably | 20:47 |
| clarkb | ya that seems to work. Odd that aa-disable wouldnt work though. I would prefer to only disable this for libvirtd | 20:50 |
| clarkb | also the profile totally allows what I need | 20:50 |
| *** gfidente|afk is now known as gfidente | 20:50 | |
| *** xenos76 has quit IRC | 20:50 | |
| clarkb | this is super annoying | 20:51 |
| clarkb | cmurphy: does libvirt work for you? | 20:51 |
| cmurphy | clarkb: yes, i last updated last week i think | 20:52 |
| fungi | it's setting kernel level rules, so you'd need to unset/override them. a reboot clears it out though (think of it like iptables in that regard) | 20:52 |
| clarkb | hrm so aa-disable then reboot maybe? | 20:52 |
| clarkb | network netlink raw is in the ruleset and it fails to create raw netlink socket | 20:53 |
| cmurphy | clarkb: i think you have to do /usr/sbin/aa-teardown plus stop the service | 20:55 |
| cmurphy | but apparmor is running for me and things are fine | 20:56 |
| clarkb | ya I found an ubuntu bug foe this issue and apparently it couldnt be reproduced which makes sense becuse the rule is right there | 20:57 |
| fungi | presumably aa-teardown clears them out without a need to do that via reboot | 20:57 |
| clarkb | I'm giving up and restoring the the previous state and just wont have a working libvirt I guess | 20:57 |
| openstackgerrit | Merged openstack/diskimage-builder master: CentOS 8 minimal testing and support https://review.opendev.org/684860 | 20:57 |
| *** igordc has quit IRC | 20:59 | |
| *** slaweq has quit IRC | 20:59 | |
| *** mtreinish has joined #openstack-infra | 21:00 | |
| *** igordc has joined #openstack-infra | 21:00 | |
| clarkb | heh I need cap sys rawio now aparently | 21:01 |
| fungi | that solved it? | 21:01 |
| fungi | i guess the aa rules for libvirt in tumbleweed need updating? | 21:02 |
| *** jbadiapa has quit IRC | 21:03 | |
| clarkb | I basically reverted all the rulsets by force reinstalling then rebooted | 21:07 |
| clarkb | this fixed the netlink issue but now I need raw io | 21:07 |
| clarkb | I think | 21:07 |
| clarkb | but that is specific to the bridge helper | 21:08 |
| clarkb | when trying to start an instance | 21:08 |
| clarkb | I think I can just use NAT instead | 21:08 |
| clarkb | heh nope the default netwokr is already in useby virbr0 | 21:09 |
| clarkb | I must be the only person in the world that doesnt nat with libvirt | 21:10 |
| fungi | you rebel | 21:10 |
| *** smarcet has quit IRC | 21:10 | |
| clarkb | I give up | 21:13 |
| clarkb | lunch now then figure out dib builds some other way | 21:13 |
| cmurphy | i've always just run dib directly on my host machine | 21:14 |
| cmurphy | i don't think it's ever caused an issue | 21:15 |
| clarkb | brave | 21:16 |
| cmurphy | heh | 21:16 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add token secret to intermediate registry https://review.opendev.org/687421 | 21:17 |
| *** gfidente is now known as gfidente|afk | 21:19 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Use JWT for authorization https://review.opendev.org/687422 | 21:20 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove read-only user from registry https://review.opendev.org/687423 | 21:21 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Use mirrors for functional test builds https://review.opendev.org/685885 | 21:21 |
| mordred | clarkb: back online for a few minutes before dinner - anything I should look at? | 21:23 |
| clarkb | mordred: those glean changes from tristanC maybe | 21:24 |
| clarkb | still trying to sort out testing them in various clouds and am failing | 21:24 |
| *** smarcet has joined #openstack-infra | 21:24 | |
| fungi | mordred: also we had some brief discussion about your gerrit container folgers crystals on review-dev | 21:25 |
| fungi | during the weekly meeting | 21:25 |
| *** smarcet has quit IRC | 21:27 | |
| mordred | fungi: folgers crystals? | 21:27 |
| mordred | ah - got it | 21:28 |
| mordred | clarkb: I saw some discussion of considering cloud-init too, yeah? | 21:29 |
| fungi | mordred: it was originally corvus's joke in the meeting, i just shamelessly plagiarized him | 21:31 |
| mordred | fungi: it's the best type of plagarism | 21:31 |
| corvus | fungi: have you no shame? :) | 21:32 |
| *** sreejithp has quit IRC | 21:37 | |
| clarkb | mordred: ya tristanC was aaking what weneed glean for today and if cloud init would work | 21:41 |
| clarkb | I think the twoo issues are whether or not cloud init works on rax and dealing with cloud inits deps | 21:42 |
| *** markvoelker has quit IRC | 21:43 | |
| mordred | yeah. I imagine it should work on rax - but the deps would be the thing I'd be the most concerned about | 21:44 |
| mordred | I wonder if we could do cloud-init in a venv | 21:44 |
| tristanC | mordred: or could we fix the deps through a zuul-jobs? (as proposed in https://storyboard.openstack.org/#!/story/2006681) | 21:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Use JWT for authorization https://review.opendev.org/687422 | 21:47 |
| mordred | tristanC: seems not unreasonble | 21:48 |
| *** jamesmcarthur has quit IRC | 22:04 | |
| *** goldyfruit_ has joined #openstack-infra | 22:05 | |
| openstackgerrit | Merged opendev/system-config master: Add token secret to intermediate registry https://review.opendev.org/687421 | 22:05 |
| *** goldyfruit___ has quit IRC | 22:07 | |
| openstackgerrit | Merged zuul/nodepool master: dib-functional : allow extra elements to be passed https://review.opendev.org/685884 | 22:11 |
| *** gyee has quit IRC | 22:14 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add anonymous pull test https://review.opendev.org/687433 | 22:19 |
| clarkb | I have a dib centos 7 build with glean overrides running on a test node I boted in limestone for other purposes in the past | 22:23 |
| clarkb | hopefully that doesn't take too long and we can get it uplaoded to fn and see if it is any happier | 22:24 |
| clarkb | or at least not a regression | 22:24 |
| donnyd | clarkb: ++ hopefully this will get it fixed | 22:25 |
| donnyd | Appreciate all the efforts to make everything play nicely with FN | 22:26 |
| *** rcernin has joined #openstack-infra | 22:26 | |
| fungi | well, fn isn't the problem. this is as much about fixing or working around bugs in software which just happen to be easier to tickle there | 22:27 |
| clarkb | heh had to install kpartx | 22:28 |
| clarkb | rerunning now | 22:28 |
| *** mattmceuen has left #openstack-infra | 22:28 | |
| fungi | something like this crops up almost any time we add a new provider or an existing provider makes major architectural changes... we get to discover new bugs that shake out | 22:28 |
| donnyd | Well then I am glad I can be helpful in exposing bugs 🤣 | 22:29 |
| fungi | so are we! | 22:30 |
| fungi | thanks again! | 22:31 |
| clarkb | ok now to try building a fedora image | 22:33 |
| *** jamesmcarthur has joined #openstack-infra | 22:35 | |
| *** jamesmcarthur has quit IRC | 22:36 | |
| *** jamesmcarthur has joined #openstack-infra | 22:36 | |
| openstackgerrit | Merged zuul/zuul-registry master: Use JWT for authorization https://review.opendev.org/687422 | 22:37 |
| *** goldyfruit_ has quit IRC | 22:39 | |
| *** jamesmcarthur has quit IRC | 22:39 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: Ensure both kubernetes and openshift token are b64decoded https://review.opendev.org/687435 | 22:39 |
| clarkb | oh wow that is quite the straw | 22:44 |
| fungi | ? | 22:44 |
| clarkb | I'm getting 25KBps downloading my image off that host right now | 22:45 |
| fungi | oof | 22:45 |
| fungi | we're gonna need a bigger straw | 22:45 |
| *** jamesmcarthur has joined #openstack-infra | 22:45 | |
| clarkb | fungi: is root@2607:ff68:100:54:f816:3eff:fecd:5c0f:/root/test-glean-nm-updates.qcow2 any quicker for you? | 22:45 |
| clarkb | my plan was to download it then upload to nb01.openstack.org, then do vhd conversion and also upload it to the various clouds | 22:46 |
| clarkb | I guess I can skip the middle host and try directly from nb01 | 22:46 |
| fungi | over 2MB/s and climbing | 22:47 |
| clarkb | ok ya its super fast direct from nb01 | 22:47 |
| clarkb | I think that we are good then | 22:47 |
| fungi | this is via a gre tunnel to he | 22:47 |
| fungi | 2.5 now | 22:48 |
| clarkb | must be an ovh thing then | 22:48 |
| clarkb | I tunnel through there | 22:48 |
| fungi | flagging again | 22:48 |
| fungi | now up to 2.6 | 22:48 |
| fungi | finished | 22:49 |
| fungi | test-glean-nm-updates.qcow2 100% 323MB 2.3MB/s 02:22 | 22:49 |
| *** jamesmcarthur has quit IRC | 22:50 | |
| clarkb | and good thing we tested | 23:00 |
| *** tkajinam has joined #openstack-infra | 23:00 | |
| clarkb | http://paste.openstack.org/show/782277/ | 23:00 |
| fungi | oops! | 23:02 |
| *** prometheanfire has quit IRC | 23:03 | |
| *** prometheanfire has joined #openstack-infra | 23:04 | |
| clarkb | working on a patch then I'll rebuild and reupload and all that | 23:04 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-registry master: Add anonymous pull test https://review.opendev.org/687433 | 23:05 |
| openstackgerrit | Clark Boylan proposed opendev/glean master: Import ipv6 configuration done by cloud-init for Red Hat distro https://review.opendev.org/686749 | 23:06 |
| *** tosky has quit IRC | 23:10 | |
| clarkb | also not sure how unit testing didn't catch that | 23:12 |
| clarkb | seems like it should've | 23:12 |
| *** diablo_rojo has quit IRC | 23:17 | |
| fungi | must not exercise that code path, or at least not feed data into it the same way | 23:18 |
| clarkb | root@2001:470:e045:8000:f816:3eff:feee:c13a works, fedora on fn | 23:20 |
| clarkb | looking at journalctl -u NetworkManager I'm not sure that nm processed the ra's though | 23:21 |
| clarkb | however it did configure the interface just fine so it isn't a regression at least | 23:21 |
| clarkb | we are missing a trailing | 23:21 |
| clarkb | er trailing \n in the output | 23:21 |
| clarkb | er not output the sysconfig interface file | 23:21 |
| clarkb | working to boot a centos host now | 23:22 |
| *** yamamoto has joined #openstack-infra | 23:22 | |
| donnyd | I think that v6 working was hit or miss. Sometimes it was good, others it would hang for a while | 23:24 |
| clarkb | donnyd: ya I think what we are looking for here is mostly that it doesn't regress and break it (like the previous ps did) and if not we can merge it and then monitor to see if it improves things | 23:24 |
| clarkb | centos isn't working | 23:28 |
| clarkb | I can get to it via ipv4 and the network config scripts look fine | 23:28 |
| clarkb | but no ipv6 address | 23:28 |
| *** markvoelker has joined #openstack-infra | 23:32 | |
| clarkb | I cleared out the files that glean writes and rebooted and this time it works :/ | 23:34 |
| clarkb | I think this likely doesn't fix the "NM doesn't know how to manage interfaces if the kernel is touching them" problem | 23:34 |
| clarkb | except that I should've had working ipv6 in that case so now I am extra confused | 23:34 |
| clarkb | deleted that host and booted a new one. This new one works :/ | 23:38 |
| clarkb | I'm going to try and catch one that fails on first boot | 23:39 |
| *** tkajinam has quit IRC | 23:39 | |
| *** tkajinam has joined #openstack-infra | 23:40 | |
| *** aaronsheffield has quit IRC | 23:41 | |
| clarkb | and I just booted 5 new ones and none of them have ipv6 | 23:42 |
| donnyd | Is there a way to have glean flush the kernel config before handing it over to nm? | 23:42 |
| clarkb | I think that is what we are trying to sort out | 23:43 |
| clarkb | restarting network manager seems to fix the problem on these hosts | 23:43 |
| clarkb | I wish network manager logs were useful | 23:44 |
| clarkb | glean is running before NM starts | 23:50 |
| clarkb | I don't think we are racing between those two | 23:50 |
| clarkb | nmcli device show eth0 shows that it has no ipv6 config | 23:50 |
| *** mriedem has quit IRC | 23:51 | |
| clarkb | interestingly this is the inverse of what we would gt when kernel configured ipv6 | 23:51 |
| clarkb | when kernel configured ipv6 we would get no ipv4 | 23:52 |
| clarkb | we have ipv4 in this case but no ipv6 | 23:52 |
| clarkb | makes me think there is somethign wrong with our config or maybe NM needs some other service to come up first for handling RAs? | 23:52 |
| fungi | is nm starting before glean writes out the configs maybe? | 23:52 |
| clarkb | no | 23:52 |
| clarkb | not according to the journal at least | 23:52 |
| fungi | (or are we missing a flush?) | 23:52 |
| clarkb | I think we flush pretty aggresively | 23:52 |
| clarkb | there was a change from smoeone that did that recently | 23:52 |
| fungi | does glean terminate before nm starts? | 23:53 |
| clarkb | yes | 23:53 |
| clarkb | (at least accodring to the journal) | 23:53 |
| clarkb | also we wouldn't have ipv4 if the NM config wasn't there | 23:53 |
| clarkb | possible the ra timeout is working against us in this case? | 23:54 |
| *** dchen has joined #openstack-infra | 23:54 | |
| clarkb | like maybe its wwaiting for the kernel to configure it? | 23:54 |
| clarkb | I can try builds without that delay, but I'm running out of daylight today | 23:54 |
| *** prometheanfire has quit IRC | 23:54 | |
| clarkb | let me double check if fedora has a fail rate or if this is centos 7 only | 23:55 |
| fungi | is it repeatable on reboot right? | 23:55 |
| fungi | s/right// | 23:55 |
| *** prometheanfire has joined #openstack-infra | 23:55 | |
| clarkb | no | 23:56 |
| clarkb | restarting NM or rebooting fixes | 23:56 |
| fungi | even after clearing the network conffiles glean would write? | 23:56 |
| clarkb | yes | 23:56 |
| clarkb | and someontimes when I boot it it works | 23:56 |
| clarkb | (so I think it is a race of some sort) | 23:57 |
| fungi | seems like it | 23:57 |
| clarkb | 2/5 fedora boots work | 23:57 |
| clarkb | the other 3 seem to suffer the same fate of the 5/5 centos hosts | 23:58 |
| clarkb | I'm going to delete all but one host so they don't take up quota | 23:58 |
| clarkb | the images are in fn too if people want to poke at it. | 23:58 |
| clarkb | you know it would be interesting to test if this breaks with cloud-init too | 23:59 |
| donnyd | Quota on FN is 2x the test nodes | 23:59 |
| clarkb | maybe someone can check upstream centos 7 and fedora images as a next step | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!