| *** e0ne has joined #openstack-infra | 00:00 | |
| ianw | yeah, i'm open to discussion on that; maybe it's premature optimisation but having a proxy environment seems useful | 00:01 |
|---|---|---|
| *** xek has quit IRC | 00:02 | |
| *** mriedem has quit IRC | 00:02 | |
| *** markvoelker has joined #openstack-infra | 00:03 | |
| clarkb | if we can load updated ssl certs without restarting $notapache that could be a reason to switch | 00:03 |
| *** iokiwi has quit IRC | 00:04 | |
| *** adriant has quit IRC | 00:04 | |
| *** adriant has joined #openstack-infra | 00:07 | |
| *** iokiwi has joined #openstack-infra | 00:07 | |
| ianw | the extant redirects are http only, but certainly could run more through the proxy to achieve that. | 00:07 |
| ianw | a "build it an they will come" approach was my main thought i guess | 00:08 |
| clarkb | I think they do https today too | 00:08 |
| clarkb | tarballs is https for example | 00:08 |
| *** armax has quit IRC | 00:08 | |
| clarkb | oh the redirects | 00:09 |
| *** iokiwi has quit IRC | 00:09 | |
| *** adriant has quit IRC | 00:09 | |
| clarkb | not the actual vhosts | 00:09 |
| *** david-lyle is now known as dklyle | 00:09 | |
| *** whoami-rajat has quit IRC | 00:13 | |
| *** markvoelker has quit IRC | 00:13 | |
| *** e0ne has quit IRC | 00:13 | |
| *** jamesmcarthur has quit IRC | 00:14 | |
| *** jamesmcarthur has joined #openstack-infra | 00:15 | |
| *** jamesmcarthur has quit IRC | 00:16 | |
| *** jamesmcarthur has joined #openstack-infra | 00:16 | |
| *** jamesmcarthur has quit IRC | 00:17 | |
| *** jamesmcarthur has joined #openstack-infra | 00:19 | |
| *** jamesmcarthur has quit IRC | 00:19 | |
| fungi | apache can update certs with just a reload and not a restart | 00:20 |
| fungi | or at least i'm pretty sure it can these days | 00:21 |
| clarkb | I don't think so | 00:22 |
| clarkb | I remember looking into that when we did the LE restart handler | 00:23 |
| fungi | apachectl -k graceful performs a graceful restart of the service and is mapped as "reload" in a log of initscripts | 00:25 |
| *** iokiwi has joined #openstack-infra | 00:25 | |
| *** adriant has joined #openstack-infra | 00:25 | |
| *** iokiwi has quit IRC | 00:25 | |
| *** adriant has quit IRC | 00:25 | |
| fungi | s/log/lot/ | 00:25 |
| clarkb | and is that sufficient for loading new ssl certs? | 00:26 |
| fungi | yeah, the one on files.o.o calls "$APACHE2CTL graceful" in do_reload() | 00:26 |
| clarkb | (we may be able to update the ansile if so, but I thought I looked it up and a proper restart was required) | 00:27 |
| fungi | the reload, force-reload and graceful parameters to the initscript are all synonymous | 00:27 |
| fungi | "This differs from a normal restart in that currently open connections are not aborted." (from the apache2ctl manpage) | 00:28 |
| *** gyee has quit IRC | 00:28 | |
| *** adriant has joined #openstack-infra | 00:29 | |
| *** iokiwi has joined #openstack-infra | 00:29 | |
| *** weifan has quit IRC | 00:30 | |
| *** ociuhandu has joined #openstack-infra | 00:31 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gerrit with podman instead https://review.opendev.org/689890 | 00:31 |
| *** weifan has joined #openstack-infra | 00:32 | |
| *** adriant has quit IRC | 00:32 | |
| *** iokiwi has quit IRC | 00:32 | |
| *** weifan has quit IRC | 00:33 | |
| *** weifan has joined #openstack-infra | 00:34 | |
| *** jamesmcarthur has joined #openstack-infra | 00:35 | |
| *** ociuhandu has quit IRC | 00:35 | |
| *** lathiat has quit IRC | 00:37 | |
| *** lathiat has joined #openstack-infra | 00:38 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gerrit with podman instead https://review.opendev.org/689890 | 00:46 |
| *** slaweq has joined #openstack-infra | 00:47 | |
| mordred | clarkb, fungi, corvus: for yucks and jollies / sake of argument ^^ ... fwiw, I restarted review-dev using podman-compose to check it out - and it's going fine. I'm going to start working in just a bit on making sure this all works from the ansible and see what we need to add to our testing | 00:48 |
| openstackgerrit | Merged openstack/project-config master: Pause Fedora 30 builds https://review.opendev.org/689895 | 00:49 |
| *** slaweq has quit IRC | 00:51 | |
| *** jamesmcarthur has quit IRC | 00:52 | |
| fungi | so we could end up with a bunch of podlings wandering around | 01:02 |
| *** diablo_rojo has quit IRC | 01:02 | |
| *** weifan has quit IRC | 01:05 | |
| mordred | fungi: who doesn't want a cute little podling? | 01:07 |
| *** adriant has joined #openstack-infra | 01:08 | |
| *** ianychoi has quit IRC | 01:16 | |
| *** ianychoi has joined #openstack-infra | 01:16 | |
| *** yamamoto has joined #openstack-infra | 01:25 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gerrit with podman instead https://review.opendev.org/689890 | 01:29 |
| *** jamesmcarthur has joined #openstack-infra | 01:32 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Pause Fedora jobs https://review.opendev.org/689893 | 01:35 |
| *** slaweq has joined #openstack-infra | 01:48 | |
| *** slaweq has quit IRC | 01:53 | |
| *** jerryz has quit IRC | 01:53 | |
| *** jamesmcarthur has quit IRC | 01:55 | |
| *** rh-jelabarre has quit IRC | 02:04 | |
| *** weifan has joined #openstack-infra | 02:05 | |
| *** markvoelker has joined #openstack-infra | 02:14 | |
| *** markvoelker has quit IRC | 02:18 | |
| *** yamamoto has quit IRC | 02:19 | |
| *** weifan has quit IRC | 02:31 | |
| *** jamesmcarthur has joined #openstack-infra | 02:31 | |
| *** ianychoi has quit IRC | 02:31 | |
| *** ianychoi has joined #openstack-infra | 02:33 | |
| *** jamesmcarthur has quit IRC | 02:38 | |
| *** slaweq has joined #openstack-infra | 02:48 | |
| *** spsurya has joined #openstack-infra | 02:50 | |
| *** ianychoi has quit IRC | 02:51 | |
| *** ianychoi has joined #openstack-infra | 02:52 | |
| *** slaweq has quit IRC | 02:53 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Stub out dry run in the uploader https://review.opendev.org/592929 | 02:58 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Create a download script https://review.opendev.org/592341 | 02:58 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 02:58 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gerrit with podman instead https://review.opendev.org/689890 | 03:01 |
| *** diablo_rojo has joined #openstack-infra | 03:02 | |
| openstackgerrit | pengyuesheng proposed openstack/hacking master: Switch to Ussuri jobs https://review.opendev.org/689908 | 03:03 |
| *** weifan has joined #openstack-infra | 03:05 | |
| *** dave-mccowan has quit IRC | 03:05 | |
| *** rfolco has quit IRC | 03:15 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Update gentoo test to current system profile https://review.opendev.org/682635 | 03:19 |
| openstackgerrit | Merged openstack/diskimage-builder master: bootloader: make serial console configurable https://review.opendev.org/686495 | 03:19 |
| *** weifan has quit IRC | 03:20 | |
| *** rkukura has quit IRC | 03:34 | |
| *** psachin has joined #openstack-infra | 03:36 | |
| *** ykarel|away has joined #openstack-infra | 03:37 | |
| *** diablo_rojo has quit IRC | 03:39 | |
| *** guoqiao_ has joined #openstack-infra | 03:41 | |
| *** slaweq has joined #openstack-infra | 03:49 | |
| *** slaweq has quit IRC | 03:54 | |
| *** ykarel|away is now known as ykarel | 03:59 | |
| *** udesale has joined #openstack-infra | 04:04 | |
| *** slaweq has joined #openstack-infra | 04:50 | |
| *** prometheanfire has quit IRC | 04:54 | |
| *** slaweq has quit IRC | 04:54 | |
| *** prometheanfire has joined #openstack-infra | 04:55 | |
| *** ianychoi has quit IRC | 05:13 | |
| *** ianychoi has joined #openstack-infra | 05:14 | |
| *** lmiccini has joined #openstack-infra | 05:15 | |
| *** ralonsoh has joined #openstack-infra | 05:18 | |
| *** ianychoi has quit IRC | 05:26 | |
| *** ianychoi has joined #openstack-infra | 05:28 | |
| *** slaweq has joined #openstack-infra | 05:29 | |
| *** eernst has joined #openstack-infra | 05:30 | |
| *** janki has joined #openstack-infra | 05:30 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Create a download script https://review.opendev.org/592341 | 05:32 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 05:32 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: testing autogenerated download script https://review.opendev.org/689930 | 05:32 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: Use test-upload-logs-swift in base-test https://review.opendev.org/689932 | 05:35 |
| *** weifan has joined #openstack-infra | 05:39 | |
| *** weifan has quit IRC | 05:39 | |
| *** weifan has joined #openstack-infra | 05:40 | |
| *** weifan has quit IRC | 05:44 | |
| *** eernst has quit IRC | 05:54 | |
| *** ykarel is now known as ykarel|afk | 05:57 | |
| *** slaweq has quit IRC | 05:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/689959 | 06:07 |
| *** e0ne has joined #openstack-infra | 06:11 | |
| *** ykarel|afk is now known as ykarel | 06:13 | |
| *** e0ne has quit IRC | 06:14 | |
| *** sshnaidm_ has joined #openstack-infra | 06:15 | |
| *** sshnaidm|pto has quit IRC | 06:15 | |
| *** rcernin has quit IRC | 06:16 | |
| *** markvoelker has joined #openstack-infra | 06:17 | |
| *** markvoelker has quit IRC | 06:21 | |
| *** jtomasek has joined #openstack-infra | 06:23 | |
| *** dpawlik has joined #openstack-infra | 06:27 | |
| *** ykarel is now known as ykarel|afk | 06:29 | |
| *** igordc has joined #openstack-infra | 06:34 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/689959 | 06:35 |
| *** aedc has quit IRC | 06:40 | |
| *** ccamacho has joined #openstack-infra | 06:41 | |
| *** pcaruana has joined #openstack-infra | 06:47 | |
| *** igordc has quit IRC | 06:48 | |
| *** trident has quit IRC | 06:53 | |
| *** ykarel|afk is now known as ykarel | 06:56 | |
| *** trident has joined #openstack-infra | 06:58 | |
| *** iurygregory has joined #openstack-infra | 07:00 | |
| *** e0ne has joined #openstack-infra | 07:00 | |
| *** dpawlik has quit IRC | 07:01 | |
| *** e0ne has quit IRC | 07:01 | |
| *** dpawlik has joined #openstack-infra | 07:02 | |
| *** slaweq has joined #openstack-infra | 07:04 | |
| *** udesale has quit IRC | 07:06 | |
| *** udesale has joined #openstack-infra | 07:06 | |
| *** larainema has joined #openstack-infra | 07:10 | |
| *** tosky has joined #openstack-infra | 07:12 | |
| *** sshnaidm_ has quit IRC | 07:14 | |
| *** sshnaidm has joined #openstack-infra | 07:27 | |
| *** aedc has joined #openstack-infra | 07:32 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 07:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gerrit with podman instead https://review.opendev.org/689890 | 07:38 |
| tobias-urdin | ianw: o/ can we follow up on https://review.opendev.org/#/c/666263/ | 07:39 |
| *** trident has quit IRC | 07:40 | |
| *** udesale has quit IRC | 07:41 | |
| *** udesale has joined #openstack-infra | 07:42 | |
| *** priteau has joined #openstack-infra | 07:42 | |
| *** trident has joined #openstack-infra | 07:43 | |
| *** janki has quit IRC | 07:46 | |
| *** Lucas_Gray has joined #openstack-infra | 07:51 | |
| *** janki has joined #openstack-infra | 07:52 | |
| *** kjackal has joined #openstack-infra | 07:59 | |
| *** hwoarang has quit IRC | 08:04 | |
| *** dchen has quit IRC | 08:06 | |
| *** hwoarang has joined #openstack-infra | 08:09 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 08:10 |
| *** hwoarang has quit IRC | 08:13 | |
| *** hwoarang has joined #openstack-infra | 08:14 | |
| *** kopecmartin|off is now known as kopecmartin | 08:17 | |
| *** e0ne has joined #openstack-infra | 08:19 | |
| *** tesseract has joined #openstack-infra | 08:21 | |
| *** tesseract has quit IRC | 08:21 | |
| *** dpawlik has quit IRC | 08:31 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gerrit: add support for report only connection https://review.opendev.org/568216 | 08:32 |
| *** apetrich has quit IRC | 08:36 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gerrit: add support for report only connection https://review.opendev.org/568216 | 08:36 |
| *** lucasagomes has joined #openstack-infra | 08:37 | |
| *** derekh has joined #openstack-infra | 08:37 | |
| *** ykarel is now known as ykarel|lunch | 08:40 | |
| *** gfidente has joined #openstack-infra | 08:46 | |
| *** ricolin_ is now known as ricolin | 08:49 | |
| *** hwoarang has quit IRC | 08:53 | |
| openstackgerrit | Merged openstack/reviewstats master: Load subproject data from governance https://review.opendev.org/653024 | 08:54 |
| openstackgerrit | Merged openstack/reviewstats master: Raise hacking version and fix pep8 errors https://review.opendev.org/655911 | 08:54 |
| openstackgerrit | Merged openstack/reviewstats master: Switch to stestr https://review.opendev.org/655506 | 08:54 |
| openstackgerrit | Merged openstack/reviewstats master: Drop pypy default tox env https://review.opendev.org/655912 | 08:55 |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** pkopec has joined #openstack-infra | 08:58 | |
| *** xek has joined #openstack-infra | 09:00 | |
| *** dpawlik has joined #openstack-infra | 09:01 | |
| *** hwoarang has joined #openstack-infra | 09:04 | |
| *** ykarel|lunch is now known as ykarel | 09:10 | |
| *** ricolin_ has joined #openstack-infra | 09:15 | |
| *** ricolin has quit IRC | 09:17 | |
| *** ricolin_ is now known as ricolin | 09:20 | |
| *** weifan has joined #openstack-infra | 09:24 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: trigger: add job filter event https://review.opendev.org/639905 | 09:30 |
| *** Lucas_Gray has quit IRC | 09:31 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: webtrigger: add initial driver and event https://review.opendev.org/555153 | 09:33 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: webtrigger: add web route and rpclistener https://review.opendev.org/554839 | 09:35 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: web: add build button to trigger job https://review.opendev.org/635716 | 09:36 |
| *** weifan has quit IRC | 09:36 | |
| *** dpawlik has quit IRC | 09:48 | |
| *** ykarel has quit IRC | 09:55 | |
| *** ykarel has joined #openstack-infra | 09:55 | |
| *** dpawlik has joined #openstack-infra | 10:02 | |
| *** janki has quit IRC | 10:07 | |
| *** ociuhandu has joined #openstack-infra | 10:10 | |
| *** ykarel is now known as ykarel|afk | 10:12 | |
| *** derekh has quit IRC | 10:16 | |
| *** pcaruana has quit IRC | 10:16 | |
| *** markvoelker has joined #openstack-infra | 10:19 | |
| *** ykarel|afk is now known as ykarel | 10:19 | |
| *** pcaruana has joined #openstack-infra | 10:21 | |
| *** markvoelker has quit IRC | 10:24 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Create a download script https://review.opendev.org/592341 | 10:38 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: test bulk download script change https://review.opendev.org/689930 | 10:38 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 10:38 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: test bulk download script change https://review.opendev.org/689930 | 10:43 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Create a download script https://review.opendev.org/592341 | 10:43 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 10:43 |
| ianw | tobias-urdin: sorry i'll take a look tomorrow. i guess i'll just create the volume, unless anyone feels like reworking all the scripts to write to one volume | 10:46 |
| *** lmiccini has quit IRC | 10:49 | |
| *** lmiccini has joined #openstack-infra | 10:52 | |
| *** derekh has joined #openstack-infra | 10:53 | |
| *** yamamoto has joined #openstack-infra | 11:13 | |
| *** udesale has quit IRC | 11:18 | |
| *** yamamoto has quit IRC | 11:19 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 11:26 |
| *** tonyb has joined #openstack-infra | 11:31 | |
| *** dpawlik has quit IRC | 11:31 | |
| *** xek has quit IRC | 11:37 | |
| *** xek has joined #openstack-infra | 11:37 | |
| *** ianychoi has quit IRC | 11:37 | |
| *** ianychoi has joined #openstack-infra | 11:40 | |
| *** dpawlik has joined #openstack-infra | 11:43 | |
| *** dpawlik has quit IRC | 11:55 | |
| *** rh-jelabarre has joined #openstack-infra | 11:55 | |
| *** beagles|afk is now known as beagles | 11:57 | |
| *** ianychoi has quit IRC | 12:00 | |
| *** markvoelker has joined #openstack-infra | 12:01 | |
| *** dpawlik has joined #openstack-infra | 12:01 | |
| *** yolanda has quit IRC | 12:01 | |
| *** ianychoi has joined #openstack-infra | 12:03 | |
| *** yamamoto has joined #openstack-infra | 12:04 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 12:05 |
| *** yamamoto has quit IRC | 12:07 | |
| *** yamamoto has joined #openstack-infra | 12:07 | |
| *** efried has left #openstack-infra | 12:10 | |
| *** efried has joined #openstack-infra | 12:10 | |
| efried | o/ infra | 12:11 |
| efried | Yesterday I noticed that python-openstackclient docs don't seem to be published for Train: https://docs.openstack.org/python-openstackclient/train (s/train/stein/ or s/train/latest/ and they're fine) | 12:11 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: change default calling as a module https://review.opendev.org/690057 | 12:12 |
| *** yolanda has joined #openstack-infra | 12:13 | |
| zbr | if I want to clone/fork an external project for use within openstack testing, where should I start with? | 12:14 |
| zbr | i am contemplating creating a fork of ansible-lint, especially as using it as pre-commit hook requires cloning it with git, so performance wise it would better to clone from opendev. | 12:15 |
| *** yamamoto has quit IRC | 12:16 | |
| zbr | also, do we need duplicated gerrit notifications on #opendev and #openstack-infra - one is not enough? | 12:16 |
| *** jamesmcarthur has joined #openstack-infra | 12:18 | |
| *** dpawlik has quit IRC | 12:20 | |
| *** derekh has quit IRC | 12:23 | |
| *** eharney has quit IRC | 12:25 | |
| *** yamamoto has joined #openstack-infra | 12:29 | |
| *** jamesmcarthur has quit IRC | 12:30 | |
| *** larainema has quit IRC | 12:32 | |
| *** dpawlik has joined #openstack-infra | 12:36 | |
| *** ianychoi has quit IRC | 12:39 | |
| *** ianychoi has joined #openstack-infra | 12:41 | |
| *** dtantsur is now known as dtantsur|brb | 12:47 | |
| *** derekh has joined #openstack-infra | 12:54 | |
| *** ianychoi has quit IRC | 12:54 | |
| *** rfolco has joined #openstack-infra | 12:55 | |
| *** ianychoi has joined #openstack-infra | 12:57 | |
| *** aaronsheffield has joined #openstack-infra | 13:01 | |
| *** yamamoto has quit IRC | 13:02 | |
| *** xek has quit IRC | 13:03 | |
| *** xek has joined #openstack-infra | 13:05 | |
| *** yamamoto has joined #openstack-infra | 13:05 | |
| *** jovial[m] has quit IRC | 13:07 | |
| *** mriedem has joined #openstack-infra | 13:12 | |
| *** eharney has joined #openstack-infra | 13:14 | |
| *** derekh has quit IRC | 13:14 | |
| *** dave-mccowan has joined #openstack-infra | 13:24 | |
| *** ykarel is now known as ykarel|afk | 13:26 | |
| *** dave-mccowan has quit IRC | 13:29 | |
| *** udesale has joined #openstack-infra | 13:31 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: change default calling as a module https://review.opendev.org/690057 | 13:38 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Take advantage of fetch-output https://review.opendev.org/689473 | 13:39 |
| *** xek has quit IRC | 13:40 | |
| *** xek has joined #openstack-infra | 13:40 | |
| *** ociuhandu has quit IRC | 13:48 | |
| *** diablo_rojo has joined #openstack-infra | 13:49 | |
| *** trident has quit IRC | 13:50 | |
| *** ociuhandu has joined #openstack-infra | 13:50 | |
| *** jcoufal has joined #openstack-infra | 13:52 | |
| *** fresta has quit IRC | 13:53 | |
| *** trident has joined #openstack-infra | 13:54 | |
| *** fresta has joined #openstack-infra | 13:54 | |
| *** raschid has joined #openstack-infra | 13:57 | |
| *** ianychoi has quit IRC | 13:58 | |
| *** ociuhandu has quit IRC | 13:58 | |
| *** ianychoi has joined #openstack-infra | 14:00 | |
| *** rfolco is now known as rfolco|rover | 14:03 | |
| clarkb | efried: http://zuul.openstack.org/builds?project=openstack%2Fpython-openstackclient&branch=stable%2Ftrain shows all the jobs that have run for that project against the stable train branch. I don't see any post jobs for docs. I think the jobs haven't been run | 14:04 |
| clarkb | efried: one explanation for that would be if the jobs only run on job update merges | 14:04 |
| clarkb | and there haven't been any mergeds or any merges that update docs | 14:05 |
| clarkb | zbr: you don't need to fork. If you add the project to zuul from github then you can add it as a required project to jobs and zuul will include it in the list of repos in the job | 14:05 |
| clarkb | zbr: that said perhaps using ansible-lint from pypi would be more effective method of using cached data? | 14:06 |
| *** yamamoto has quit IRC | 14:07 | |
| corvus | fungi, clarkb, ianw: can you see my small but important point on https://review.opendev.org/683852 ? | 14:11 |
| clarkb | corvus: ya that makes sense | 14:13 |
| *** ociuhandu has joined #openstack-infra | 14:17 | |
| *** apetrich has joined #openstack-infra | 14:19 | |
| *** igordc has joined #openstack-infra | 14:23 | |
| zbr | clarkb: pre-commit hooks are installed only from git, i would use wheels if possible. also based on challenges i faces in the last weeks, I a fork could be more appropriate, but i will wait till the end of week to see if it gets fixed. | 14:29 |
| efried | clarkb: Sorry, I don't follow. You mean somebody has to push a train docs change to get train docs to publish? | 14:30 |
| clarkb | efried: yes docs are published when changes merge and they are published for the branch that the change merged against | 14:30 |
| *** michael-beaver has joined #openstack-infra | 14:30 | |
| efried | clarkb: seems like around release time we should have something that force-publishes docs for $release, nah? | 14:30 |
| *** dpawlik has quit IRC | 14:30 | |
| clarkb | efried: this means you need to merge something to stable/train for those docs to show up. Additionally if there are files filters that only run the docs jobs when docs are updated you may have to update the docs | 14:30 |
| *** sreejithp has joined #openstack-infra | 14:31 | |
| clarkb | efried: possibly, certainly you can bring it up with the release team | 14:31 |
| *** iurygregory has quit IRC | 14:31 | |
| efried | okey, thanks clarkb | 14:31 |
| *** slaweq has quit IRC | 14:33 | |
| fungi | yeah, changes are pushed immediately on branching, but as noted the file filters saying not to run docs jobs likely causes them not to run for changes to those particular files | 14:33 |
| fungi | in particular, there's at least a change pushed to update the defaultbranch setting in the .gitreview file of the new stable branch | 14:34 |
| fungi | we could maybe set the docs jobs to not avoid running on changes to .gitreview | 14:34 |
| clarkb | there is a theme here though we tend to optimize ourselves out of running jobs we want to run | 14:35 |
| clarkb | not an uncommon problem/confusion | 14:35 |
| *** jaosorior has joined #openstack-infra | 14:35 | |
| *** pkopec has quit IRC | 14:35 | |
| fungi | i concur | 14:36 |
| *** eernst has joined #openstack-infra | 14:37 | |
| *** ianychoi has quit IRC | 14:38 | |
| *** ianychoi has joined #openstack-infra | 14:40 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 14:42 |
| *** yamamoto has joined #openstack-infra | 14:42 | |
| *** dtantsur|brb is now known as dtantsur | 14:42 | |
| *** amoralej has joined #openstack-infra | 14:44 | |
| *** yamamoto has quit IRC | 14:46 | |
| *** xek_ has joined #openstack-infra | 14:52 | |
| *** jamesmcarthur has joined #openstack-infra | 14:53 | |
| *** xek has quit IRC | 14:54 | |
| *** whoami-rajat has joined #openstack-infra | 14:55 | |
| *** pcaruana has quit IRC | 14:58 | |
| *** rkukura has joined #openstack-infra | 14:58 | |
| *** otsukahy has joined #openstack-infra | 15:05 | |
| *** slaweq has joined #openstack-infra | 15:06 | |
| *** armax has joined #openstack-infra | 15:06 | |
| *** otsukahy has quit IRC | 15:10 | |
| *** otsukahy has joined #openstack-infra | 15:13 | |
| *** otsukahy has quit IRC | 15:13 | |
| *** trident has quit IRC | 15:13 | |
| *** otsukahy has joined #openstack-infra | 15:14 | |
| *** yolanda__ has joined #openstack-infra | 15:21 | |
| *** yolanda has quit IRC | 15:24 | |
| *** trident has joined #openstack-infra | 15:26 | |
| *** mattw4 has joined #openstack-infra | 15:26 | |
| *** slaweq has quit IRC | 15:28 | |
| *** lmiccini has quit IRC | 15:29 | |
| *** gyee has joined #openstack-infra | 15:33 | |
| *** udesale has quit IRC | 15:34 | |
| *** psachin has quit IRC | 15:34 | |
| efried | Where are Storyboard issues discussed? | 15:39 |
| efried | ahem, I mean issues about storyboard itself | 15:39 |
| clarkb | storyboard has #storyboard and it also self hosts its issues in storyboard.openstack.org | 15:40 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: change default calling as a module https://review.opendev.org/690057 | 15:40 |
| *** otsukahy has quit IRC | 15:43 | |
| efried | thanks clarkb | 15:44 |
| *** otsukahy has joined #openstack-infra | 15:44 | |
| *** otsukahy has quit IRC | 15:48 | |
| *** ianychoi has quit IRC | 15:52 | |
| *** ianychoi has joined #openstack-infra | 15:54 | |
| *** lucasagomes has quit IRC | 15:57 | |
| *** otsukahy has joined #openstack-infra | 16:03 | |
| *** otsukahy has quit IRC | 16:03 | |
| *** ricolin_ has joined #openstack-infra | 16:04 | |
| *** ricolin__ has joined #openstack-infra | 16:05 | |
| zbr | ^ clarkb fungi - that tox change is ready for review. | 16:06 |
| *** ricolin has quit IRC | 16:07 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Refactor gerrit master job https://review.opendev.org/690120 | 16:07 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 16:07 |
| corvus | clarkb, fungi: can you look at https://review.opendev.org/690120 and https://review.opendev.org/690121 ? | 16:08 |
| *** otsukahy has joined #openstack-infra | 16:09 | |
| *** ricolin_ has quit IRC | 16:09 | |
| clarkb | zbr: corvus I can but I've been in meetings all morning and they aren't done yet | 16:09 |
| clarkb | will do my best to get to them when I can | 16:09 |
| zbr | thanks. | 16:10 |
| *** dpawlik has joined #openstack-infra | 16:15 | |
| *** rpittau is now known as rpittau|afk | 16:17 | |
| *** igordc has quit IRC | 16:20 | |
| openstackgerrit | Drew Walters proposed opendev/system-config master: lists: Add Airship VMP mailing lists https://review.opendev.org/689271 | 16:21 |
| *** e0ne has quit IRC | 16:24 | |
| clarkb | zbr: if you install a package into your user instal dir that is effectively a virtualenv isn't it? | 16:27 |
| clarkb | zbr: and so running system tox should be unaffeted? | 16:27 |
| clarkb | I'm not sure I understand the failure case there | 16:27 |
| *** ricolin__ is now known as ricolin | 16:30 | |
| clarkb | left a comment +2'd because I think the risk is low and I may not understand the failure case but believe it exists | 16:33 |
| clarkb | mriedem: looks like nova grenade and live migration jobs are unhappy right now in the gate (I've not had a chance to look at them yet, busy meetings morning, but figured I'd point it out) | 16:37 |
| *** yolanda__ has quit IRC | 16:38 | |
| *** eharney has quit IRC | 16:39 | |
| mriedem | i've noticed but looked like pypi issues | 16:40 |
| *** pcaruana has joined #openstack-infra | 16:41 | |
| clarkb | the greande live migration job appears to have failed because q-agt didn't start properly? | 16:42 |
| *** dpawlik has quit IRC | 16:42 | |
| clarkb | https://f4b7745b1a52a3f1e3d3-57fb9c0e693945c47fbc306535c3cca3.ssl.cf5.rackcdn.com/679519/1/gate/nova-grenade-live-migration/3ebdf38/logs/grenade.sh.txt.gz something to do with ovsdbapp | 16:42 |
| fungi | why is zuul saying https://review.opendev.org/690121 depends on a change with an invalid configuration, when zuul happily ran the jobs for its only dependency? | 16:42 |
| fungi | (and i don't see any updated patchsets or anything to explain it having been from an earlier version of the dependency) | 16:43 |
| clarkb | I had assumed it was because the parent change was in a config project | 16:51 |
| clarkb | but thinking about it system-config shouldn't be trusted | 16:52 |
| *** pcaruana has quit IRC | 16:52 | |
| zbr | clarkb: no. user install ~= virtualenv install. | 16:53 |
| zbr | python prefers any module installed in user dir, but virtualenvs never have access to user-dir. | 16:53 |
| zbr | think about user-dir as some kind of user specific virtualenv that is using site packages. | 16:53 |
| *** markvoelker has quit IRC | 16:54 | |
| clarkb | zbr: is the issue then running tox from /usr/local/bin/tox with a user installed tox and the /usr tox tries to use the user installed libs? | 16:54 |
| zbr | python will always try to load libs from user dir, if any | 16:54 |
| zbr | which means that if I do "pip install --user tox" as foo user, the system tox command will likely fail. | 16:55 |
| clarkb | right but you should only be using the user tox at that point | 16:55 |
| clarkb | (because that bin dir is ahead in your path) | 16:55 |
| zbr | clarkb: is not in PATH on most distros. | 16:55 |
| clarkb | then you shouldn't use user installs :P | 16:56 |
| zbr | i was the one that added it to the PATH in fedora 30, if I remember well. | 16:56 |
| clarkb | if you are user installing things then either add it to your path or explicitly execute from that path | 16:56 |
| zbr | if user bin is in PATH, depends on OS/version, config, if is root or not. | 16:56 |
| clarkb | right but if you are user installing packages you know to add it to the path or execute fro mthere | 16:57 |
| zbr | that issue is not specific to tox, it applies to other tools, including virtualenv and pip. | 16:57 |
| clarkb | I don'tthink the system defaults matter much | 16:57 |
| zbr | one of the reasons why I always prefer "python -m" | 16:57 |
| zbr | as this was never broken | 16:57 |
| clarkb | well if you python -m tox like that you will never use the user installed tox | 16:57 |
| clarkb | that will find the system tox? | 16:57 |
| clarkb | I guess it depends on your python path | 16:58 |
| zbr | nope, it will load user-tox if is installed. | 16:58 |
| zbr | python by default is loading modules from user-dir, is has done since before 2.7 | 16:58 |
| clarkb | anyway as mentioned I think it is low risk so should be fine. That said I think if you are using user installed packages you should be explicit about paths | 16:58 |
| jrosser | i have a few jobs fail with "No matching distribution found for tempest===22.1.0" | 16:59 |
| jrosser | like this https://d98ff9a93a37640186b9-bbe0623e042078b1f8054ea3d769696e.ssl.cf1.rackcdn.com/690009/3/check/openstack-ansible-deploy-aio_metal-debian-stable/c481c10/job-output.txt | 16:59 |
| jrosser | and i see that package is present in pypi | 16:59 |
| jrosser | and a similar error in another job related to cyptography | 16:59 |
| * jrosser wonders if there is a wonky mirror/proxy | 17:00 | |
| corvus | jrosser: can you link to the build page instead of the logs? | 17:00 |
| jrosser | the job is sill running | 17:00 |
| zbr | clarkb: yeah, still there are cases where the user may be forced to do a local upgrade. in this case why to have a default that gets broken when we can have one that works fine in all known cases. | 17:00 |
| clarkb | jrosser: corvus ERROR: Could not find a version that satisfies the requirement tempest===22.1.0 (from -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=master (line 475)) (from versions: 10.0.0, 11.0.0, 12.0.0, 12.1.0, 12.2.0, 13.0.0, 14.0.0, 15.0.0, 16.0.0, 16.1.0, 17.0.0, 17.1.0, 17.2.0, 18.0.0, 19.0.0, 20.0.0, 21.0.0, 22.0.0) is the error there | 17:00 |
| corvus | jrosser: any completed buildsets with failures like that? | 17:00 |
| corvus | that ran on rax-iad | 17:01 |
| clarkb | sure enough http://mirror.iad.rax.openstack.org/pypi/simple/tempest/ lacks a 22.1.0 | 17:01 |
| fungi | seems like pypi is having trouble today | 17:02 |
| jrosser | corvus: sorry struggling to find the other - cirros dowloads have been broken all day too leading to everything wedged up on our jobs | 17:02 |
| corvus | jrosser: no worries | 17:02 |
| corvus | that's the second pypi mirror weirdness today | 17:02 |
| *** dtantsur is now known as dtantsur|afk | 17:02 | |
| * jrosser travels, bbl | 17:02 | |
| clarkb | corvus: in a different region too | 17:03 |
| clarkb | if I hit https://pypi.org/simple/tempest/ I see a 22.1.0 | 17:03 |
| clarkb | (from home) | 17:03 |
| fungi | https://status.python.org/ | 17:03 |
| fungi | "All Systems Operational" | 17:04 |
| *** ykarel|afk is now known as ykarel|away | 17:04 | |
| clarkb | there was an issue today though | 17:04 |
| clarkb | https://status.python.org/incidents/3tm38q23zpsj | 17:04 |
| *** armax has quit IRC | 17:05 | |
| clarkb | not sure if ^ could've caused some of the problems we hvae noticed | 17:05 |
| clarkb | oh neat there are headers in the responses that tell you which caches are services your pypi requets | 17:06 |
| *** gfidente is now known as gfidente|afk | 17:06 | |
| clarkb | LCY and BWI served the rsponse without 22.1.0 and IAD+SEA served my local request with 22.1.0 | 17:07 |
| corvus | both of the "erroneous" pages lack the "SERIAL" comment at the bottom | 17:07 |
| *** eernst has quit IRC | 17:07 | |
| *** xek__ has joined #openstack-infra | 17:07 | |
| *** kopecmartin is now known as kopecmartin|off | 17:08 | |
| corvus | clarkb: so pypi's cdn is inconsistent? | 17:08 |
| clarkb | corvus: seems like it | 17:08 |
| clarkb | looks like max-age is 600 seconds in the cache-control headertoo fwiw | 17:09 |
| *** ykarel|away has quit IRC | 17:09 | |
| clarkb | which should force our apache caches to update those files every 10 minutes (but we are still getting a 2 week old file, implying that either apache isn't respecting cache control or its 24 hour timeout on the file or cdn is serving old content) | 17:09 |
| *** jproulx has quit IRC | 17:10 | |
| clarkb | would be neat if there was a way to force the request through a specific cdn frontend | 17:10 |
| clarkb | but I don't know how to do that | 17:10 |
| *** xek_ has quit IRC | 17:10 | |
| openstackgerrit | Merged opendev/system-config master: Refactor gerrit master job https://review.opendev.org/690120 | 17:11 |
| *** markvoelker has joined #openstack-infra | 17:12 | |
| fungi | isn't there a command to request a page to be refreshed in the cdn? i want to say we used to automatically ping that on certain bandersnatch errors | 17:12 |
| corvus | -X PURGE ? | 17:12 |
| clarkb | corvus: ya that sounds right | 17:12 |
| corvus | oh, but we haven't actually seen pypi serve wrong data right now | 17:13 |
| corvus | we're still inferring that from our apache, which could be holding stale data for some reason (maybe pypi returned long max-age headers during the outage?) | 17:14 |
| corvus | we might want to run some curl gets on our mirrors to see what pypi cdn looks like from there, and also purge our own mirror caches first? | 17:14 |
| clarkb | corvus: ya (note our apache config has a 24 hour max ttl that it should enforce to avoid problems like you describe lingering long term) | 17:14 |
| clarkb | corvus: that seems like a reasonable next step | 17:14 |
| corvus | i'll try some gets | 17:15 |
| corvus | the simple temepest page returns the same current data when i run it on ord, dfw, and sjc1 | 17:16 |
| *** raschid has quit IRC | 17:17 | |
| clarkb | what about iad (that is where jrosser's fail happened) | 17:17 |
| corvus | sorry, i meant ord, iad, sjc1 | 17:17 |
| corvus | i guess we should purge our apache cache now... any idea how? | 17:18 |
| corvus | (or should we just wait 24h?) | 17:18 |
| clarkb | there is a tool to run to clean up the cache (or stop apache and rm it from top level) | 17:19 |
| clarkb | htcacheclean | 17:19 |
| corvus | fungi: https://review.opendev.org/690121 is clear to +3 now | 17:20 |
| corvus | (and i agree, we should figure out why zuul has a hard time with that situation) | 17:20 |
| corvus | i have to afk for a bit; i'll leave the htcacheclean to others if we want to do that | 17:21 |
| *** priteau has quit IRC | 17:22 | |
| clarkb | I'm trying to figure out if we can determine apache's url hash algorithm to maybe do some targetted removals first | 17:22 |
| clarkb | it is an md5sum of some sort | 17:22 |
| clarkb | looks like it incorporates quite a few details of the request so figuring that out by hand is likely difficult | 17:24 |
| *** otsukahy has quit IRC | 17:24 | |
| *** eharney has joined #openstack-infra | 17:25 | |
| clarkb | the correct file is there now at http://mirror.iad.rax.openstack.org/pypi/simple/tempest/ | 17:26 |
| clarkb | I haven't done any write operations only read only items | 17:26 |
| clarkb | the entry there shows the cdn hosts are IAD and DCA now instead of LCY and BWI | 17:27 |
| *** weifan has joined #openstack-infra | 17:27 | |
| clarkb | if I had to guess there are bad cdn frontends floating around out there and the 10 minute cache control is working but if the refresh hits the wrong cdn frontend we get sadness? | 17:27 |
| clarkb | there is a good chance that clearing the cache out wouldn't help given ^ | 17:27 |
| fungi | i guess we keep an eye out for recurrences and then try to correlate to fastly cdn endpoints | 17:28 |
| fungi | we can also bring this up in #pypa-dev if we see more evidence that's the case | 17:28 |
| fungi | i think the psf sysadmins handling this hang out there | 17:28 |
| fungi | in other news, our zuul seems remarkably not-busy today | 17:29 |
| clarkb | post openstack release/pre summit hangover | 17:30 |
| clarkb | (openstack still represents ~95% of our ci load) | 17:30 |
| *** ociuhandu_ has joined #openstack-infra | 17:32 | |
| *** ociuhandu has quit IRC | 17:36 | |
| fungi | strange, if i click on the publish-tox-docs-static job running for the openstack/releases ref in release-post on the zuul status page for the openstack tenant right now, it just gives me "end of stream" over and over | 17:38 |
| fungi | but the build has been running for a while | 17:38 |
| jrosser | fungi: I’ve seen a ton of those today too | 17:38 |
| fungi | i wonder if that's a sign the finger server on one of the executors has died | 17:39 |
| clarkb | yes that is the most likely cause | 17:39 |
| *** ociuhandu_ has quit IRC | 17:39 | |
| *** ianychoi has quit IRC | 17:39 | |
| clarkb | https://review.opendev.org/677846 and https://review.opendev.org/678049 are related but I don't think there was consensu on which people preferred | 17:39 |
| clarkb | I'd be happy to pick up this effort again but was hoping to have better direciton before writing more code | 17:40 |
| *** ianychoi has joined #openstack-infra | 17:42 | |
| fungi | 3,5,8,12 are all not listening on 7900/tcp | 17:43 |
| *** ociuhandu has joined #openstack-infra | 17:43 | |
| clarkb | fungi: if you check dmesg they likely OOMed and that process was the one chosen for sacrifice | 17:43 |
| clarkb | the current fix is to restart the executors on the host | 17:44 |
| *** amoralej is now known as amoralej|off | 17:45 | |
| clarkb | current tempest index comes from IAD and BWI and is correct | 17:46 |
| *** amoralej|off is now known as amoralej | 17:46 | |
| clarkb | might point at LCY as the sad frontend | 17:46 |
| *** ociuhandu has quit IRC | 17:47 | |
| *** jerryz has joined #openstack-infra | 17:49 | |
| *** armax has joined #openstack-infra | 17:49 | |
| *** amoralej is now known as amoralej|off | 17:50 | |
| AJaeger | clarkb: docs publishing is in promote pipeline now, did you check that (sorry, still catching up backscroll) | 17:52 |
| clarkb | AJaeger: ya I didn't filter by pipeline and there were no promote jobs | 17:52 |
| fungi | btw, we saw a bizarre failure for a release docs publishing job which worked fine once we reran it. failure is: https://zuul.opendev.org/t/openstack/build/71c8b17f28054e80a2c5ae45c18d7af1 | 17:53 |
| fungi | seems like the executor couldn't find the files to upload | 17:53 |
| clarkb | fungi: those look like rooted paths | 17:54 |
| clarkb | oh wait I know this one | 17:54 |
| clarkb | fungi: this is the rsync atomic file rename race | 17:55 |
| AJaeger | clarkb: ah, ok. And I don't think we restrain docs publishing - so, if nothing was merged, nothing was published | 17:55 |
| clarkb | fungi: if multiple jobs are running at the same time that rsync the same files one can rename the file to finish the copy before the other and the other fails | 17:55 |
| *** ianychoi has quit IRC | 17:55 | |
| clarkb | fungi: promote should address that, but that job ran in post | 17:55 |
| clarkb | er release-post. post is actually supercedent and should fix it too | 17:56 |
| AJaeger | efried: https://review.opendev.org/#/q/project:openstack/python-openstackclient+branch:stable/train+is:open - those changes should have merged ages ago - they would have given you docs ;) | 17:56 |
| efried | AJaeger: oh indeed... dtroyer ^^ | 17:56 |
| fungi | clarkb: oh, hah yep! | 17:56 |
| fungi | been so long since i've seen that | 17:56 |
| fungi | clarkb: not 100% if there were publish jobs for different branches running at the same time | 17:57 |
| fungi | though... i think that repo has only a master branch | 17:57 |
| clarkb | fungi: or even jobs for different changes on the same brnach | 17:57 |
| fungi | well, in the case of post i mean | 17:57 |
| fungi | release-post | 17:57 |
| clarkb | https://zuul.opendev.org/t/openstack/builds?job_name=publish-tox-docs-static&pipeline=release-post shows two jobs started at the same time | 17:58 |
| clarkb | one faile dthe other diddnt | 17:58 |
| *** ianychoi has joined #openstack-infra | 17:58 | |
| clarkb | for the same change? | 17:58 |
| clarkb | and then there is a third right around then for a different change too | 17:59 |
| fungi | the second is when i reenqueued it just now | 17:59 |
| clarkb | oh wait they are different and started at the same time | 17:59 |
| clarkb | da45 vs d3d7 | 17:59 |
| clarkb | so ya making that supercedent would fix | 17:59 |
| fungi | and happen to start and end with the same digits, which makes them look similar, yep | 18:00 |
| fungi | i don't recall if we determined there was some reason release-post might have jobs that need to run for every single ref | 18:00 |
| *** ricolin has quit IRC | 18:01 | |
| *** jamesmcarthur has quit IRC | 18:01 | |
| *** jaosorior has quit IRC | 18:01 | |
| *** otsukahy has joined #openstack-infra | 18:05 | |
| fungi | also, circling back around to the finger server issue, i've confirmed that the four executors not listening on 7900/tcp did indeed log an oom killing a zuul-executor process in the past three days | 18:06 |
| fungi | so that seems to still be the same known issue | 18:06 |
| *** otsukahy has quit IRC | 18:09 | |
| *** e0ne has joined #openstack-infra | 18:10 | |
| *** ianychoi has quit IRC | 18:12 | |
| *** igordc has joined #openstack-infra | 18:14 | |
| *** ianychoi has joined #openstack-infra | 18:16 | |
| paladox | corvus https://gerrit-review.googlesource.com/c/plugins/checks/+/241293 :O | 18:17 |
| paladox | zuul! | 18:17 |
| corvus | paladox: how about that! i'm hoping it actuall starts the jobs soon... https://zuul.opendev.org/t/zuul/status | 18:17 |
| corvus | also, it's slightly chattier than i expected, i'll see about reducing that | 18:17 |
| paladox | :D | 18:17 |
| corvus | oh hey there goes one now | 18:18 |
| paladox | \o/ | 18:18 |
| clarkb | I need to pop out for a bit then prep for the meeting. After the meeting I can help restart those 4 executors if thatstill needs doing | 18:21 |
| *** weifan has quit IRC | 18:22 | |
| *** ralonsoh has quit IRC | 18:22 | |
| *** e0ne has quit IRC | 18:23 | |
| *** ianychoi has quit IRC | 18:26 | |
| *** e0ne has joined #openstack-infra | 18:29 | |
| *** ianychoi has joined #openstack-infra | 18:30 | |
| *** spsurya has quit IRC | 18:30 | |
| fungi | i wasn't super concerned about actually restarting the executors, more with working out that it was the same known issue and not something new | 18:30 |
| fungi | though if they're falling like dominoes after the last restart, maybe we want to be extra sure there's not something which changed recently to increase memory utilization for the executors | 18:31 |
| *** weifan has joined #openstack-infra | 18:31 | |
| *** e0ne has quit IRC | 18:32 | |
| *** ianychoi has quit IRC | 18:38 | |
| *** weifan has quit IRC | 18:39 | |
| clarkb | fungi: restarting is the only way to fix the console log streaming | 18:41 |
| *** ianychoi has joined #openstack-infra | 18:42 | |
| fungi | yeah, i get that | 18:45 |
| fungi | i should have said i'm less concerned that console log streams aren't available for 1/3 of the builds and mostly wanted to be sure we weren't looking at a new problem | 18:45 |
| jrosser | I found another python install error https://0a5c623ff913673f34ed-80527d3715cd24d9d5daf12827bd66d5.ssl.cf2.rackcdn.com/681777/12/check/openstack-ansible-deploy-aio_metal-debian-stable/bb04e28/logs/ara-report/result/421723ea-2127-4248-be14-a3d5971af22d/ | 18:46 |
| jrosser | that’s also still running so no job page | 18:46 |
| fungi | we should still restart them, it's not the best user experience, but it's not show-stoppingly broken | 18:46 |
| *** e0ne has joined #openstack-infra | 18:46 | |
| *** jamesmcarthur has joined #openstack-infra | 18:47 | |
| *** igordc has quit IRC | 18:47 | |
| fungi | jrosser: i see... ERROR: Could not find a version that satisfies the requirement cryptography===2.8 | 18:47 |
| fungi | http://mirror.regionone.fortnebula.opendev.org/pypi/simple/cryptography/ | 18:49 |
| *** jamesmcarthur_ has joined #openstack-infra | 18:50 | |
| fungi | seems 2.7 is the latest it has | 18:50 |
| *** michael-beaver has quit IRC | 18:50 | |
| clarkb | that is iad and bwi | 18:52 |
| clarkb | oh though it shows 2.8 now | 18:52 |
| fungi | indeed, just showed up on refresh | 18:52 |
| fungi | i was digging into the wget manpage to work out how to access the headers | 18:52 |
| clarkb | curl -vvv will show them to you | 18:53 |
| fungi | i had tried wget -vv before rtfm'ing | 18:53 |
| fungi | ftr, it's -S | 18:53 |
| *** jamesmcarthur has quit IRC | 18:54 | |
| fungi | moments too late unfortunately | 18:54 |
| *** whoami-rajat has quit IRC | 18:55 | |
| *** igordc has joined #openstack-infra | 18:56 | |
| *** dpawlik has joined #openstack-infra | 18:57 | |
| *** weifan has joined #openstack-infra | 18:59 | |
| *** weifan has quit IRC | 19:00 | |
| *** weifan has joined #openstack-infra | 19:00 | |
| *** pcaruana has joined #openstack-infra | 19:04 | |
| *** ianychoi has quit IRC | 19:04 | |
| *** jamesmcarthur has joined #openstack-infra | 19:05 | |
| *** weifan has quit IRC | 19:06 | |
| *** jamesmca_ has joined #openstack-infra | 19:06 | |
| *** ianychoi has joined #openstack-infra | 19:08 | |
| *** jamesmcarthur_ has quit IRC | 19:08 | |
| *** jamesmcarthur has quit IRC | 19:09 | |
| *** dpawlik has quit IRC | 19:09 | |
| openstackgerrit | Ian Wienand proposed opendev/infra-specs master: Spec to retire static.openstack.org https://review.opendev.org/683852 | 19:09 |
| *** jamesmcarthur has joined #openstack-infra | 19:10 | |
| *** e0ne has quit IRC | 19:11 | |
| *** jamesmcarthur has quit IRC | 19:12 | |
| *** jamesmca_ has quit IRC | 19:13 | |
| *** jamesmcarthur has joined #openstack-infra | 19:15 | |
| *** ianychoi has quit IRC | 19:15 | |
| *** pcaruana has quit IRC | 19:19 | |
| *** slaweq has joined #openstack-infra | 19:21 | |
| *** amoralej|off is now known as amoralej | 19:21 | |
| *** ianychoi has joined #openstack-infra | 19:24 | |
| *** amoralej is now known as amoralej|off | 19:24 | |
| *** yamamoto has joined #openstack-infra | 19:26 | |
| *** gagehugo has joined #openstack-infra | 19:26 | |
| *** jamesmcarthur has quit IRC | 19:27 | |
| *** yamamoto has quit IRC | 19:31 | |
| *** weifan has joined #openstack-infra | 19:31 | |
| *** jamesmcarthur has joined #openstack-infra | 19:32 | |
| *** otsukahy has joined #openstack-infra | 19:32 | |
| *** weifan has quit IRC | 19:38 | |
| *** ianychoi has quit IRC | 19:39 | |
| *** weifan has joined #openstack-infra | 19:45 | |
| *** jamesmcarthur has quit IRC | 19:46 | |
| *** slaweq has quit IRC | 19:46 | |
| *** dpawlik has joined #openstack-infra | 19:48 | |
| *** slaweq has joined #openstack-infra | 19:51 | |
| *** jtomasek has quit IRC | 19:53 | |
| *** auristor has quit IRC | 19:54 | |
| *** auristor has joined #openstack-infra | 19:59 | |
| *** e0ne has joined #openstack-infra | 20:04 | |
| *** otsukahy has quit IRC | 20:05 | |
| *** auristor has quit IRC | 20:05 | |
| *** otsukahy has joined #openstack-infra | 20:05 | |
| *** weifan has quit IRC | 20:07 | |
| *** weifan has joined #openstack-infra | 20:09 | |
| corvus | mordred: to catch up on gerrit/zuul: i made some changes to the job so we can have a version with no file matchers; ran it against a checks plugin change, and it failed with these logs: https://d0439ae4739e0ce9dbbb-be043fd2e3d9156a0edcda5d98bda85d.ssl.cf5.rackcdn.com/241293/1/third-party-check/build-gerrit-image/a72e868/job-output.txt | 20:09 |
| *** auristor has joined #openstack-infra | 20:09 | |
| corvus | mordred: i have some pipeline config stuff to deal with -- do you think you can take a look at that and figure out what we're missing? | 20:09 |
| corvus | mordred: (maybe we just need to add system-config as a required project?) | 20:10 |
| corvus | mordred: (but it might be worth a think to make sure that's the only thing that needs to change when running it on a different project) | 20:11 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Fix user_typeahead result rendering https://review.opendev.org/690400 | 20:12 |
| *** auristor has quit IRC | 20:16 | |
| *** auristor has joined #openstack-infra | 20:21 | |
| *** weifan has quit IRC | 20:22 | |
| *** weifan has joined #openstack-infra | 20:24 | |
| *** kjackal has quit IRC | 20:24 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add system-config as a required project to gerrit-master https://review.opendev.org/690406 | 20:27 |
| corvus | mordred: ^ | 20:27 |
| *** weifan has quit IRC | 20:29 | |
| *** weifan has joined #openstack-infra | 20:30 | |
| *** aedc has quit IRC | 20:32 | |
| *** iokiwi has joined #openstack-infra | 20:33 | |
| *** weifan has quit IRC | 20:34 | |
| *** dpawlik has quit IRC | 20:35 | |
| *** eernst has joined #openstack-infra | 20:37 | |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Render user-submitted links with nofollow attribute https://review.opendev.org/648322 | 20:41 |
| *** weifan has joined #openstack-infra | 20:41 | |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Render user-submitted links with nofollow attribute https://review.opendev.org/648322 | 20:42 |
| openstackgerrit | Merged opendev/storyboard-webclient master: Fix user_typeahead result rendering https://review.opendev.org/690400 | 20:44 |
| *** amoralej|off is now known as amoralej | 20:44 | |
| *** otsukahy has quit IRC | 20:46 | |
| *** weifan has quit IRC | 20:46 | |
| *** weifan has joined #openstack-infra | 20:49 | |
| *** cgoncalves has quit IRC | 20:49 | |
| *** eernst has quit IRC | 20:51 | |
| *** weifan has quit IRC | 20:53 | |
| rm_work | in zuul job configs, how do I specify which branch to use for a devstack_plugin? | 20:54 |
| rm_work | do I do it in the "required-projects" section by doing an "override-checkout"? or is it something in the "devstack_plugins" vars section? | 20:54 |
| *** weifan has joined #openstack-infra | 20:55 | |
| fungi | pretty sure you need override-checkout if the default behavior of just using the branch of the same name doesn't work | 20:55 |
| *** cgoncalves has joined #openstack-infra | 20:55 | |
| rm_work | so if i do that, then devstack will use that version when i do the enable_plugin for it via "devstack_plugins"? | 20:56 |
| fungi | by default it should use devstack plugin repository branches with the same name as the branch which triggered the job (so same as the branch of devstack usually) | 20:56 |
| *** aedc has joined #openstack-infra | 20:56 | |
| rm_work | yeah this particular project is on master but I want it to pull in latest-stable | 20:57 |
| rm_work | for dependencies | 20:57 |
| rm_work | at least that is my current thought | 20:57 |
| fungi | might be better to confirm in #openstack-qa but i believe override-checkout will work for that. the devstack playbooks shouldn't be engaging in arbitrary branch changes for plugin repos | 20:57 |
| rm_work | k | 20:58 |
| rm_work | I'll try it out at least | 20:58 |
| fungi | as far as i know the devstack jobs will just use whatever repository states zuul has already checked out | 20:58 |
| rm_work | do you know if the project that triggered the job is automatically included? right now we have it in the "required-projects" list as well but i feel like that's not necessary (and might have been my error earlier) | 20:58 |
| fungi | the project for the event which triggers the job is an implicit "required-project" yes | 20:59 |
| rm_work | kk | 20:59 |
| fungi | so no need to specify it unless that particular job is also run on events for other projects which need that project to be incorporated from source | 21:00 |
| rm_work | eh i'm rethinking this entire plan lol, since i'd also have to specify I want nova/neutron/etc all from stable too | 21:01 |
| *** aedc has quit IRC | 21:01 | |
| rm_work | and that's going to get tedious | 21:01 |
| *** weifan has quit IRC | 21:01 | |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Allowing the user to choose what Columns are seen https://review.opendev.org/651270 | 21:02 |
| fungi | so... the job can definitely checkout those branches by name if you add some tasks for that. it's basically what grenade does | 21:02 |
| mordred | corvus: hahahaha. (re: add system-config) | 21:02 |
| *** aedc has joined #openstack-infra | 21:03 | |
| mordred | corvus: I will now look at the log you sent | 21:03 |
| *** weifan has joined #openstack-infra | 21:03 | |
| rm_work | yeah it can, but i would need to specify every project, and that gets really tedious, and i was on the fence anyway about whether this was a good idea :D | 21:03 |
| mordred | corvus: oh - I think your change is the thing that fixes that issue | 21:04 |
| fungi | rm_work: you're basically designing grenade again ;) | 21:04 |
| mordred | corvus: I think that should likely take care of things - most everything was already pretty explicit about repo location and paths and whatnot since it was dockerfiles in system-config building gerrit repos | 21:05 |
| *** weifan has quit IRC | 21:05 | |
| rm_work | lol well, i don't want to upgrade | 21:05 |
| *** weifan has joined #openstack-infra | 21:05 | |
| rm_work | i'm just trying to prevent stuff from breaking the gates on a project that only cares about testing against stable APIs | 21:06 |
| rm_work | (ospurge) | 21:06 |
| corvus | rm_work: gimme a sec to catch up on scrollback | 21:06 |
| dansmith | has anyone thought of having a way to say "run this patch until it fails", potentially with a upper limit or time window? | 21:06 |
| rm_work | no worries | 21:06 |
| dansmith | thinking about situations where we need to be able to float a not-suitable-for-merging patch to gather information, but need "put it in the firehose" sort of effect | 21:07 |
| rm_work | I'd worry about that from an abuse standpoint... might need to put caps on that and also limit who could do it, maybe to at least project cores <_< | 21:07 |
| *** slaweq has quit IRC | 21:08 | |
| corvus | rm_work: can you get real specific about which projects and which branches? (maybe a pastebin?) | 21:08 |
| dansmith | rm_work: yep, certainly valid | 21:09 |
| rm_work | i literally just want to use *all latest stable* for everything in devstack | 21:09 |
| rm_work | but i'm still deciding if that's a good idea | 21:09 |
| rm_work | Pros: gates break less from random stuff (since we're going to depend on a TON of projects to be installed) | 21:09 |
| *** weifan has quit IRC | 21:10 | |
| corvus | rm_work: but you want to do that on changes to master branch of os_whatever? | 21:10 |
| rm_work | Cons: no longer tests against upcoming code, and no way to do so | 21:10 |
| rm_work | that's what i'm debating with myself, yes | 21:10 |
| rm_work | not sure if I actually do :D | 21:10 |
| corvus | right, hypothetically :) | 21:10 |
| rm_work | so, here's what I have right now (and some changes I am testing to see if we have some redundant stuff): https://review.opendev.org/#/c/689962/3/.zuul.yaml | 21:12 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Render user-submitted links with nofollow attribute https://review.opendev.org/648322 | 21:12 |
| rm_work | was tempted to list out every project and stable/train (though then we'd also have to update that every release) | 21:12 |
| *** weifan has joined #openstack-infra | 21:13 | |
| rm_work | I've basically changed my mind at this point, will just have to deal with annoying gate breakages when stuff happens | 21:13 |
| *** e0ne has quit IRC | 21:13 | |
| *** e0ne has joined #openstack-infra | 21:13 | |
| corvus | rm_work: you can set override-checkout at the job level and it will apply to everything: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.override-checkout and then you can override that for the project itself back to the branch under test. it's a little wonky, but should work. however, i think sticking with the status quo is best. | 21:14 |
| rm_work | ah yeah ok that would be much easier | 21:14 |
| rm_work | but... yeah | 21:14 |
| corvus | rm_work: and if there are gate breakages, add testing to prevent them | 21:14 |
| corvus | :) | 21:14 |
| rm_work | well, i'm talking about gate breakages because of other projects | 21:14 |
| rm_work | stuff that causes those projects to break that are even outside of their control, often | 21:15 |
| rm_work | like deps and stuff | 21:15 |
| corvus | rm_work: yeah, but this whole system is designed to keep projects from breaking each other, so if they are, we have either missed an opportunity to do so, or have chosen not to care, but maybe we should care and should figure out a way to fix it :) | 21:15 |
| corvus | ie, if we've decided that two projects shouldn't share the integrated queue, maybe we still need some api or functional testing for the contracts | 21:16 |
| rm_work | working on Octavia for the last 5 years, and we've been subject to this kind of issue as long as I can remember, and it HAS gotten better but it's still kinda just reality | 21:16 |
| rm_work | but this is also for a project that's kinda ... well... "x" namespace | 21:16 |
| rm_work | so i don't think other projects need to be considering it :D | 21:16 |
| corvus | probably not going to be in the integrated queue then | 21:16 |
| corvus | but still, i bet they don't want to break it | 21:16 |
| rm_work | yeah but i mean, Octavia's gates sometimes break because of dep changes | 21:17 |
| rm_work | and there's nothing we can do to prevent that | 21:17 |
| rm_work | this is stuff out in pypi land | 21:17 |
| rm_work | but then once we're broken | 21:18 |
| *** weifan has quit IRC | 21:18 | |
| rm_work | things that rely on us are broken (like x/ospurge) | 21:18 |
| corvus | rm_work: is octavia using global requirements? | 21:18 |
| rm_work | yes | 21:18 |
| rm_work | but stuff still sneaks through sometimes. this is one of those situations where I don't know what to tell you, because how it happens seems to be different every time. sometimes it's even people REMOVING packages that used to exist :D | 21:20 |
| rm_work | I just know it happens | 21:20 |
| rm_work | anyway, I don't think I am going to end up doing this stable-gating thing anyway, so I don't want to waste a ton more time discussing it unless you're realllly bored, lol | 21:20 |
| fungi | dansmith: in the past we discussed special pipelines which could do things like always start a new batch of builds for a project+branch once the running one reported, for use in performance testing. seems like this could be a similar pipeline but triggered on demand and reenqueued until failure | 21:22 |
| *** weifan has joined #openstack-infra | 21:23 | |
| johnsom | Which people bump stuff and break things. the jsonschema upper-constraint major version bump did trigger a set of unit test failures in Octavia. That was bumped on requirements freeze week. | 21:23 |
| fungi | dansmith: might need its own queue manager model in zuul, but that's not a show-stopper if there's a reasonable use case | 21:24 |
| openstackgerrit | Merged opendev/system-config master: Add system-config as a required project to gerrit-master https://review.opendev.org/690406 | 21:24 |
| fungi | so think something like the experimental pipeline which is triggered by a specially-formatted comment, but which keeps reenqueuing until a failure is generated | 21:25 |
| *** sreejithp has quit IRC | 21:33 | |
| *** mattw4 has quit IRC | 21:40 | |
| *** mattw4 has joined #openstack-infra | 21:41 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Use zuul-registry as buildset registry https://review.opendev.org/689238 | 21:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Make the buildset registry port configurable https://review.opendev.org/689240 | 21:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 21:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Open iptables ports in registry test job https://review.opendev.org/690425 | 21:42 |
| clarkb | johnsom: for that specific case you could add cross testing between requirements and octavia | 21:43 |
| clarkb | many projects do that | 21:43 |
| clarkb | for the "someone deleted the dep we set in contraints" problem there isn't much we can do, we are broken in that state and must update | 21:43 |
| johnsom | clarkb Great minds think alike, I'm proposing it now | 21:43 |
| *** otsukahy has joined #openstack-infra | 21:44 | |
| clarkb | in the deleted dep case we don't want to paper over that because then only the gate would work and nothing else would | 21:44 |
| clarkb | its better to fail and catch that case then fix it | 21:44 |
| mordred | corvus, clarkb: on the subject of testing like production ... | 21:46 |
| mordred | our test nodes have stuff pre-installed on them which causes our test jobs of system-config playbooks to behave differently than in production | 21:46 |
| corvus | we removed a bunch of stuff; what's left? | 21:48 |
| mordred | specifically, the pre-installed pip v2 and pip3 in /usr/local is not how our actual base images we use for our services look | 21:48 |
| ianw | maybe switch to centos8 images :) | 21:48 |
| mordred | like, I had to update a playbook because it was using a path that should have worked based on what the playbook was installing, but didn't because the /usr/local won | 21:48 |
| corvus | should we drop that and rely on ensure-pip or whatever? | 21:48 |
| mordred | I kind of think so - especially as we start dropping python2 support from things | 21:49 |
| clarkb | fiw on review.o.o pip and pip3 are both /usr/local/bin things | 21:49 |
| clarkb | which should match up with our test images | 21:49 |
| mordred | clarkb: it should - but that's not what our playbooks are saying | 21:49 |
| clarkb | I'm not sure I understand | 21:49 |
| clarkb | I just manually checked its /usr/local/bin/pip(3) | 21:49 |
| clarkb | in production | 21:49 |
| mordred | in our ansible, we have a role that installs python3-pip | 21:50 |
| clarkb | which is what we also have on our test nodes iirc | 21:50 |
| mordred | this winds up not being what we actually use, because of pre-installed /usr/local/bin/pip | 21:50 |
| mordred | (I'd be find updating the playbooks to install pip from get-pip - mostly pointing out that the test node, playbooks and production dont' actually agree fully) | 21:51 |
| clarkb | and we are executing `pip` and not `pip3` ? | 21:51 |
| clarkb | right I'm still lost on where they don't agree because as far as I know they all use /usr/local/bin/pip(3) | 21:51 |
| mordred | well - when the ansible pip: module is used, it does execute pip and not pip3 - but you can put in executable: pip3 | 21:51 |
| clarkb | is the problem /usr/local/bin or pip vs pip3? | 21:52 |
| mordred | however, you can't do executable: /usr/bin/pip3 (it doesn't work) even though the playbook says "apt-get install python3-pip" | 21:52 |
| mordred | the problem is if you read the playbook, it will lead you to believe one state exists - but that is not the actual state that exists | 21:52 |
| mordred | and it's not clear why, because of pre-installed stuff | 21:53 |
| clarkb | is the playbook somewhere I can read it? I am really not understanding this | 21:53 |
| clarkb | in production if you pip install something it will be pip installed with /usr/local/bin/pip unless you override PATH. On test nodes this should also be the case | 21:55 |
| clarkb | I assume the ansible pip module will respect PATH in both cases too | 21:55 |
| mordred | clarkb: playbooks/bridge.yaml and then also https://review.opendev.org/#/c/630406/55/playbooks/service-review-container.yaml | 21:55 |
| mordred | clarkb: no - I understand that | 21:55 |
| mordred | clarkb: what I'm saying is - we have a role in system-config called "pip3" - it installs python3-pip and python3-virtualenv - and we use it in the bridge playbook as well as the upcoming review playbook | 21:56 |
| *** armax has quit IRC | 21:56 | |
| mordred | however - that is not what actually gets used on the test nodes, because of the other pip | 21:57 |
| clarkb | I see. I think my confusion here is that our puppeted nodes (like review.o.o) install pip with pip and end up in the same state as the test nodes | 21:57 |
| clarkb | it is only the ansibled nodes with no puppet that attempt to only use the distro packages | 21:58 |
| clarkb | (possible this is also a holdover from when puppet did base server setup) | 21:58 |
| clarkb | it is from install_puppet.sh | 21:59 |
| mordred | I agree - the ansible modules work with path - but, because we had the pip3 role in the playbook, I spent a non-zero amount of time yesterday debugging why this: https://zuul.opendev.org/t/openstack/build/e9b2ac3468ce42d5b3694e86ad8997d1/log/job-output.txt#2392 | 21:59 |
| mordred | didn't work | 21:59 |
| clarkb | I think the bug here is that we didn't carry that node setup over from the puppet config | 22:00 |
| mordred | clarkb: yeah. and - I mean - like I said, I'm happy to change the pip3 role to install pip with pip | 22:00 |
| clarkb | and we started doing ansible nodes different from puppet nodes | 22:00 |
| mordred | yah | 22:00 |
| clarkb | (the puppet-install role which replaced install_puppet.sh also does not install pip that way) | 22:01 |
| *** otsukahy has quit IRC | 22:01 | |
| mordred | clarkb: so - I was originally thinking "maybe we should make some ubuntu-bionic-minimal" nodes that don't do quite as much pre-installation as our normal openstack test nodes ... | 22:03 |
| mordred | clarkb: but maybe it's best to just update the pip3 role to match reality and move on with life | 22:03 |
| clarkb | ya I know there were really good reasons for using pip from upstream at the time | 22:04 |
| *** otsukahy has joined #openstack-infra | 22:04 | |
| clarkb | in particular things like ssl support and verification as well as features like constraints | 22:04 |
| clarkb | I don't know if xenial pip is new enough to get us those features | 22:04 |
| clarkb | also if we add more images we should audit our disk usage. Cleanup of suse 42.3 and switching to the better compression of qcow2s will have helped but we have been near the disk limits of our builders for a while | 22:05 |
| mordred | clarkb: good point | 22:06 |
| *** otsukahy has quit IRC | 22:07 | |
| *** otsukahy has joined #openstack-infra | 22:07 | |
| mordred | clarkb: in general, I've become a fan of always using pip from upstream ... largely because the point of installing pip is so that you can install software using pip ... and installing software using pip AND instlaling software using distro pacakges almost always leads to pain and suffering. so if one is going to install python software using pip, to me, that has come to mean installing _all_ python | 22:07 |
| mordred | software using pip, and so might as well also install pip itself that way | 22:07 |
| clarkb | ya | 22:08 |
| clarkb | I'd be in favor of updating the ansible we have today to match the old method that we had in install_puppet.sh | 22:08 |
| mordred | of course, this becomes a little easier to ponder as we install the software we care about in container images | 22:08 |
| clarkb | which was to use get-pip then that would also match what our test images do | 22:08 |
| mordred | clarkb: k. I'll cook up a patch for that a little later | 22:08 |
| fungi | the main argument for using distro packages to install language ecosystem package managers is having a chain of trust and not having to bootstrap the language-specific package manager trust relationship from scratch | 22:10 |
| clarkb | a simple hash of get-pip.py would cover that I think | 22:10 |
| clarkb | (or even put it in the repo and copy it with the copy module) | 22:10 |
| fungi | yep, though it would need updating every time get-pip.py changes | 22:11 |
| mordred | fungi: yah - and at this point apt-get install python3-pip doesn't produce nearly as much subsequent carnage as in days of yore | 22:13 |
| mordred | so I could also be talked in to that being how we do non-containerized python installation on hosts (like for docker-compose/podman-compose) | 22:13 |
| mordred | but ... in that case I'd want to do something about our test nodes which have get-pip provied pip pre-installed | 22:14 |
| clarkb | we have 200GB free on nb01 (of 1TB) | 22:14 |
| *** otsukahy has quit IRC | 22:14 | |
| clarkb | are qcow2's are all in the >9GB <10GB range right now | 22:15 |
| *** otsukahy has joined #openstack-infra | 22:15 | |
| *** e0ne has quit IRC | 22:15 | |
| clarkb | vhd and raw are ~16GB | 22:15 |
| clarkb | that means each image represents about 42GB of disk | 22:15 |
| clarkb | multiplied by two for the current and previous images | 22:16 |
| mordred | that's so much GB | 22:16 |
| clarkb | adding a new image would just fit Ithink | 22:17 |
| clarkb | but wecan't add 2 new image at this point | 22:17 |
| clarkb | (also opensuse 150 and fedora 28 are to be removed soon so we aren't in a bad sopt just need to keep up with it) | 22:17 |
| *** gfidente|afk has quit IRC | 22:18 | |
| mordred | clarkb: maybe for now we'll hold off | 22:19 |
| mordred | I think we can just update the ansible and deal with my "these don't match" grump :) | 22:19 |
| clarkb | and that makes sense because we just removed 42.3 and prior to that we were running out of disk periodically (I suppose that could still happen) | 22:21 |
| clarkb | makes sene that we have ~1 image of extra room right now | 22:21 |
| *** mriedem has quit IRC | 22:23 | |
| *** weifan has quit IRC | 22:25 | |
| *** weifan has joined #openstack-infra | 22:28 | |
| *** otsukahy has quit IRC | 22:29 | |
| ianw | i guess i'll get the process started for f28 | 22:29 |
| *** aaronsheffield has quit IRC | 22:31 | |
| ianw | i think we're in good shape with fedora-latest as the abstraction layer | 22:33 |
| clarkb | ianw: there were a few places where people defined their own nodesets where it got weird | 22:33 |
| clarkb | but I think we took care of those | 22:33 |
| ianw | yeah, not seeing them | 22:34 |
| *** weifan has quit IRC | 22:34 | |
| clarkb | if we push up a nodeset removal change zuul will complain if there are users of that nodeset | 22:35 |
| *** weifan has joined #openstack-infra | 22:35 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Stop building Fedora 28, turn down min-ready https://review.opendev.org/690431 | 22:35 |
| clarkb | but I don't think we have any label checks because labels are somewhere magical to zuul | 22:36 |
| * clarkb codesearches opensuse-150 | 22:36 | |
| *** xek__ has quit IRC | 22:37 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Remove fedora-28 nodes https://review.opendev.org/690432 | 22:39 |
| *** otsukahy has joined #openstack-infra | 22:40 | |
| ianw | oh good, f30 should get fixed dnf soon -> https://bodhi.fedoraproject.org/updates/FEDORA-2019-0519bdd8af | 22:41 |
| clarkb | ianw: AJaeger mnaser jrosser https://review.opendev.org/690433 is likely to explode but is a start for cleaning up opensuse-150 | 22:43 |
| clarkb | if that does explode I think what we want is to add -15 jobs alongside the -150 jobs. Then update all the -150 users to be -15 uesrs then remove the -150 users | 22:43 |
| *** otsukahy has quit IRC | 22:44 | |
| *** ociuhandu has joined #openstack-infra | 22:46 | |
| ianw | clarkb: i wonder why zuul told you six times the job doesn't exist | 22:48 |
| clarkb | ianw: different branches and tenants maybe? | 22:48 |
| clarkb | I'll fix that though | 22:49 |
| *** ociuhandu has quit IRC | 22:51 | |
| *** weifan has quit IRC | 22:53 | |
| *** tkajinam has joined #openstack-infra | 22:53 | |
| *** weifan has joined #openstack-infra | 22:54 | |
| *** armax has joined #openstack-infra | 22:55 | |
| *** tosky has quit IRC | 22:56 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 23:01 |
| clarkb | topic:use-opensuse-15 | 23:01 |
| clarkb | AJaeger: ^ fyi. Many of those changes may end up needing backports but I figured we'd start on master and work our way back | 23:02 |
| *** weifan has quit IRC | 23:05 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Ensure nouveau is blacklisted in initramfs too https://review.opendev.org/679992 | 23:16 |
| *** whoami-rajat has joined #openstack-infra | 23:17 | |
| *** weifan has joined #openstack-infra | 23:18 | |
| *** dchen has joined #openstack-infra | 23:21 | |
| *** weifan has quit IRC | 23:25 | |
| *** jcoufal has quit IRC | 23:26 | |
| *** yamamoto has joined #openstack-infra | 23:28 | |
| clarkb | cmurphy: https://review.opendev.org/#/c/690437/1 is an easy one and I think that one unlocks the related chagnes in tempest and devstack | 23:30 |
| *** weifan has joined #openstack-infra | 23:30 | |
| cmurphy | clarkb: https://review.opendev.org/689229 | 23:32 |
| clarkb | oh I thought ajaeger said he didn't have time for this. Even better! | 23:32 |
| cmurphy | his doesn't include the change in the functional job though | 23:34 |
| *** weifan has quit IRC | 23:34 | |
| cmurphy | +1 to yours, i don't have stable core | 23:34 |
| clarkb | ah, ya I think we should clean that up | 23:34 |
| clarkb | oh right stable is special | 23:34 |
| *** weifan has joined #openstack-infra | 23:35 | |
| *** weifan has quit IRC | 23:42 | |
| *** weifan has joined #openstack-infra | 23:44 | |
| *** diablo_rojo has quit IRC | 23:47 | |
| *** weifan has quit IRC | 23:48 | |
| *** weifan has joined #openstack-infra | 23:51 | |
| *** weifan has quit IRC | 23:54 | |
| *** igordc has quit IRC | 23:55 | |
| *** mattw4 has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!