| *** dchen has quit IRC | 00:00 | |
| *** rcernin has joined #openstack-infra | 00:05 | |
| *** gyee has quit IRC | 00:18 | |
| *** weifan has joined #openstack-infra | 00:19 | |
| *** jamesmcarthur has joined #openstack-infra | 00:20 | |
| *** yamamoto has quit IRC | 00:34 | |
| *** dchen has joined #openstack-infra | 00:48 | |
| *** jamesmcarthur has quit IRC | 00:48 | |
| *** hamzy__ has joined #openstack-infra | 00:51 | |
| *** hamzy_ has quit IRC | 00:53 | |
| *** yamamoto has joined #openstack-infra | 00:55 | |
| *** weifan has quit IRC | 01:11 | |
| *** ociuhandu has joined #openstack-infra | 02:06 | |
| *** ociuhandu has quit IRC | 02:12 | |
| *** ykarel|away has joined #openstack-infra | 02:18 | |
| *** goldyfruit has joined #openstack-infra | 02:25 | |
| *** rascasoft has quit IRC | 02:30 | |
| *** rascasoft has joined #openstack-infra | 02:32 | |
| *** jerryz has quit IRC | 02:34 | |
| *** jamesmcarthur has joined #openstack-infra | 02:42 | |
| *** markvoelker has quit IRC | 02:45 | |
| *** rh-jelabarre has quit IRC | 02:46 | |
| *** spsurya has joined #openstack-infra | 02:53 | |
| *** ramishra has joined #openstack-infra | 03:11 | |
| *** ykarel|away has quit IRC | 03:13 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] testing openstack.org certs https://review.opendev.org/690465 | 03:16 |
|---|---|---|
| *** ricolin has joined #openstack-infra | 03:16 | |
| *** jamesmcarthur has quit IRC | 03:16 | |
| *** ykarel|away has joined #openstack-infra | 03:19 | |
| *** ykarel|away has quit IRC | 03:24 | |
| *** yamamoto has quit IRC | 03:33 | |
| openstackgerrit | melanie witt proposed openstack/project-config master: Update ceph grafana dashboard for nova master branch https://review.opendev.org/690466 | 03:47 |
| *** rfolco|rover has quit IRC | 04:04 | |
| *** yamamoto has joined #openstack-infra | 04:22 | |
| *** ykarel|away has joined #openstack-infra | 04:27 | |
| *** ykarel|away is now known as ykarel | 04:27 | |
| *** ociuhandu has joined #openstack-infra | 04:31 | |
| *** ociuhandu has quit IRC | 04:35 | |
| *** markvoelker has joined #openstack-infra | 04:46 | |
| *** markvoelker has quit IRC | 04:51 | |
| *** jtomasek has joined #openstack-infra | 04:51 | |
| *** bstinson has quit IRC | 05:01 | |
| *** surpatil has joined #openstack-infra | 05:05 | |
| *** yamamoto has quit IRC | 05:05 | |
| *** yamamoto has joined #openstack-infra | 05:08 | |
| *** bstinson has joined #openstack-infra | 05:15 | |
| *** ociuhandu has joined #openstack-infra | 05:35 | |
| *** ociuhandu has quit IRC | 05:39 | |
| *** lmiccini has joined #openstack-infra | 05:39 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] testing openstack.org certs https://review.opendev.org/690465 | 05:44 |
| *** ralonsoh has joined #openstack-infra | 05:47 | |
| *** jtomasek has quit IRC | 05:52 | |
| *** ccamacho has quit IRC | 05:58 | |
| *** igordc has joined #openstack-infra | 05:58 | |
| *** yamamoto has quit IRC | 06:01 | |
| *** gfidente has joined #openstack-infra | 06:02 | |
| *** igordc has quit IRC | 06:03 | |
| *** dpawlik has joined #openstack-infra | 06:05 | |
| openstackgerrit | Merged openstack/project-config master: Stop building Fedora 28, turn down min-ready https://review.opendev.org/690431 | 06:06 |
| AJaeger | clarkb: https://review.opendev.org/#/q/topic:opensuse-150 was what I slowly did... | 06:11 |
| AJaeger | clarkb: sorry for not mentioning it. Happy to abandon one if yours is better... | 06:12 |
| *** yamamoto has joined #openstack-infra | 06:14 | |
| *** ykarel is now known as ykarel|afk | 06:21 | |
| *** yamamoto has quit IRC | 06:26 | |
| *** yamamoto has joined #openstack-infra | 06:37 | |
| *** rcernin has quit IRC | 06:37 | |
| AJaeger | clarkb: I abandoned my freezer review since your's passed ;) | 06:39 |
| AJaeger | clarkb: let's discuss what to do with the others my evening... | 06:40 |
| *** markvoelker has joined #openstack-infra | 06:48 | |
| *** yamamoto has quit IRC | 06:49 | |
| *** markvoelker has quit IRC | 06:53 | |
| *** yamamoto has joined #openstack-infra | 06:55 | |
| *** trident has quit IRC | 06:55 | |
| *** pkopec has joined #openstack-infra | 06:55 | |
| *** iurygregory has joined #openstack-infra | 06:57 | |
| *** slaweq has joined #openstack-infra | 06:58 | |
| *** trident has joined #openstack-infra | 07:00 | |
| *** tosky has joined #openstack-infra | 07:14 | |
| *** yamamoto has quit IRC | 07:24 | |
| *** ykarel|afk is now known as ykarel | 07:25 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Apply review container to review-dev https://review.opendev.org/690483 | 07:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea to 1.9.4 https://review.opendev.org/674899 | 07:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea build to golang 1.13 on buster https://review.opendev.org/690485 | 07:31 |
| *** yamamoto has joined #openstack-infra | 07:36 | |
| *** surpatil has quit IRC | 07:38 | |
| ralonsoh | hi folks | 07:41 |
| ralonsoh | https://8ce1f8db938be597771a-5e1de375231dae3d22db48edd8691108.ssl.cf2.rackcdn.com/689957/3/check/neutron-tempest-iptables_hybrid-fedora/b2d9224/job-output.txt | 07:41 |
| ralonsoh | I'm seeing some pypi errors during the stack process | 07:41 |
| ralonsoh | 2019-10-23 05:52:07.920355 | controller | Collecting tempest===22.1.0 (from -c /opt/stack/requirements/upper-constraints.txt (line 622)) | 07:42 |
| ralonsoh | 2019-10-23 05:52:07.940863 | controller | Could not find a version that satisfies the requirement tempest===22.1.0 (from -c /opt/stack/requirements/upper-constraints.txt (line 622)) (from versions: 10.0.0, 11.0.0, 12.0.0, 12.1.0, 12.2.0, 13.0.0, 14.0.0, 15.0.0, 16.0.0, 16.1.0, 17.0.0, 17.1.0, 17.2.0, 18.0.0, 19.0.0, 20.0.0, 21.0.0, 22.0.0) | 07:42 |
| ralonsoh | 2019-10-23 05:52:07.996240 | controller | No matching distribution found for tempest===22.1.0 (from -c /opt/stack/requirements/upper-constraints.txt (line 622)) | 07:42 |
| ralonsoh | but tempest has 22.1.0 version (for sure) | 07:42 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea build to golang 1.13 on buster-slim https://review.opendev.org/690485 | 07:44 |
| *** apetrich has quit IRC | 07:49 | |
| *** apetrich has joined #openstack-infra | 07:51 | |
| *** yamamoto has quit IRC | 07:55 | |
| *** kopecmartin|off is now known as kopecmartin | 07:56 | |
| *** yamamoto has joined #openstack-infra | 07:57 | |
| *** e0ne has joined #openstack-infra | 07:57 | |
| *** yolanda has joined #openstack-infra | 07:58 | |
| ykarel | ralonsoh, so looks like fortnebula mirror http://mirror.regionone.fortnebula.opendev.org is outdated, tempest 22.1.0 is there in pypi https://pypi.org/project/tempest/22.1.0/ | 07:59 |
| ykarel | i tried:- pip install --index-url="http://mirror.regionone.fortnebula.opendev.org/pypi/simple" --extra-index-url="http://mirror.regionone.fortnebula.opendev.org/wheel/fedora-29-x86_64" --trusted-host=mirror.regionone.fortnebula.opendev.org tempest==22.1.0 | 08:00 |
| ykarel | with pypi or rax mirror it's working, so something is wrong with fortnebula infra | 08:00 |
| ralonsoh | ahhhh ok, so this is just a problem in the mirror | 08:01 |
| ralonsoh | ykarel, thank you!! | 08:01 |
| *** mgoddard has joined #openstack-infra | 08:05 | |
| *** jangutter has joined #openstack-infra | 08:06 | |
| *** rpittau|afk is now known as rpittau | 08:07 | |
| *** tkajinam has quit IRC | 08:10 | |
| ykarel | i think someone from #infra-root can check the issue with that infra ^^ | 08:10 |
| *** dchen has quit IRC | 08:12 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea build to golang 1.13 on buster-slim https://review.opendev.org/690485 | 08:12 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Base gitea-init on opendevorg/python-base https://review.opendev.org/690504 | 08:12 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add job dependencies to haproxy-statsd https://review.opendev.org/690505 | 08:12 |
| *** ccamacho has joined #openstack-infra | 08:13 | |
| *** lucasagomes has joined #openstack-infra | 08:15 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea to 1.9.4 https://review.opendev.org/674899 | 08:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea build to golang 1.13 on buster-slim https://review.opendev.org/690485 | 08:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Base gitea-init on opendevorg/python-base https://review.opendev.org/690504 | 08:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add job dependencies to haproxy-statsd https://review.opendev.org/690505 | 08:15 |
| *** yamamoto has quit IRC | 08:17 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Apply review container to review-dev https://review.opendev.org/690483 | 08:19 |
| *** yamamoto has joined #openstack-infra | 08:20 | |
| *** e0ne has quit IRC | 08:22 | |
| *** yamamoto has quit IRC | 08:26 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 08:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gerrit with podman instead https://review.opendev.org/689890 | 08:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Apply review container to review-dev https://review.opendev.org/690483 | 08:26 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use explicit image paths https://review.opendev.org/690511 | 08:26 |
| *** yamamoto has joined #openstack-infra | 08:26 | |
| *** apetrich has quit IRC | 08:29 | |
| *** yamamoto has quit IRC | 08:32 | |
| *** yamamoto has joined #openstack-infra | 08:33 | |
| *** jtomasek has joined #openstack-infra | 08:35 | |
| *** surpatil has joined #openstack-infra | 08:35 | |
| *** yamamoto has quit IRC | 08:37 | |
| *** yamamoto has joined #openstack-infra | 08:38 | |
| *** dtantsur|afk is now known as dtantsur | 08:38 | |
| *** aedc has quit IRC | 08:42 | |
| *** aedc has joined #openstack-infra | 08:42 | |
| *** apetrich has joined #openstack-infra | 08:48 | |
| *** markvoelker has joined #openstack-infra | 08:49 | |
| *** markvoelker has quit IRC | 08:53 | |
| *** ykarel is now known as ykarel|lunch | 08:55 | |
| *** otsukahy has joined #openstack-infra | 08:55 | |
| *** otsukahy has quit IRC | 08:57 | |
| *** pcaruana has joined #openstack-infra | 09:02 | |
| *** otsukahy has joined #openstack-infra | 09:05 | |
| *** yamamoto has quit IRC | 09:14 | |
| *** yamamoto has joined #openstack-infra | 09:14 | |
| *** e0ne has joined #openstack-infra | 09:16 | |
| *** otsukahy has quit IRC | 09:16 | |
| *** rcernin has joined #openstack-infra | 09:23 | |
| *** ykarel|lunch is now known as ykarel | 09:30 | |
| *** derekh has joined #openstack-infra | 09:31 | |
| *** yolanda__ has joined #openstack-infra | 09:41 | |
| *** yolanda has quit IRC | 09:43 | |
| *** psachin has joined #openstack-infra | 09:45 | |
| jangutter | Anyone interested in adding an additional +2 to a cosmetic devstack fix? https://review.opendev.org/#/c/673010/ | 09:46 |
| *** psachin has quit IRC | 09:52 | |
| *** jaosorior has joined #openstack-infra | 10:02 | |
| *** yolanda__ is now known as yolanda | 10:03 | |
| *** lastmikoi has quit IRC | 10:08 | |
| *** lastmikoi has joined #openstack-infra | 10:09 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:11 | |
| donnyd | ykarel: I don't see any fedora wheel in the http://mirror.regionone.fortnebula.opendev.org/wheel/ | 10:16 |
| *** yamamoto has quit IRC | 10:17 | |
| jrosser | ykarel: there were issues yesterday with the tempest 22.1.0 package aparrently being inconsistent in the CDN use by pypi, so an issue upstream of openstack infra mirors/caches | 10:18 |
| jrosser | i.e depending on which bit of the CDN served you, now-you-see-it-now-you-dont | 10:19 |
| *** ykarel has quit IRC | 10:24 | |
| *** ykarel has joined #openstack-infra | 10:25 | |
| ykarel | donnyd, yes right that explains why it failed for ralonsoh | 10:26 |
| ykarel | but looks like other mirror don't have fedora wheel | 10:26 |
| ykarel | like rax http://mirror.ord.rax.openstack.org/wheel/ | 10:27 |
| donnyd | http://mirror.regionone.fortnebula.opendev.org/pypi/simple/tempest/ | 10:27 |
| donnyd | looks like its there now | 10:27 |
| ykarel | but 22.1.0 is not there | 10:28 |
| ykarel | while rax one has http://mirror.ord.rax.openstack.org/pypi/simple/tempest/ | 10:28 |
| donnyd | ohic | 10:28 |
| ykarel | jrosser, okk, so that can be the reason some mirrors are outdated | 10:29 |
| ykarel | issue with pypi | 10:29 |
| donnyd | yea, I am thinking the next mirror update should fix it | 10:29 |
| *** jtomasek has quit IRC | 10:30 | |
| donnyd | on a completely unrelated topic the ipv6 updates and the nl02 patches from last week have yield zero launch errors on FN for the last 7 days... woot woot | 10:30 |
| donnyd | http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=now-7d&to=now | 10:30 |
| donnyd | looks like we finally have it all solved | 10:31 |
| ralonsoh | t | 10:31 |
| ralonsoh | thank you! | 10:31 |
| ralonsoh | sorry folks | 10:33 |
| ralonsoh | https://68c70debed03e4c08f4d-3c5c6fcb6461190790889979f17d8403.ssl.cf1.rackcdn.com/689957/3/check/openstack-tox-pep8/7831849/job-output.txt | 10:33 |
| ralonsoh | now if keystone package | 10:34 |
| ralonsoh | ERROR: Could not find a version that satisfies the requirement keystoneauth1===3.18.0 | 10:34 |
| donnyd | ralonsoh: I am thinking the mirrors will need an updated pypi | 10:35 |
| donnyd | ianw: way to go on that fix for ipv6 | 10:35 |
| ralonsoh | thanks! | 10:35 |
| AJaeger | jangutter: devstack is a QA deliverable, better ask on #openstack-qa | 10:38 |
| jangutter | AJaeger: (facepalm) sorry, always confuse the two. | 10:38 |
| AJaeger | no worries, jangutter | 10:38 |
| *** pcaruana has quit IRC | 10:41 | |
| smcginnis | FWIW, I've been seeing a lot of requirements update jobs fail because some mirrors don't have newer packages yet. Not consistent, so it seems like there are intermittent mirror sync issues. | 10:47 |
| *** kjackal has joined #openstack-infra | 10:49 | |
| *** ykarel is now known as ykarel|afk | 10:58 | |
| *** dtantsur is now known as dtantsur|brb | 11:01 | |
| *** jtomasek has joined #openstack-infra | 11:20 | |
| *** ykarel|afk has quit IRC | 11:22 | |
| *** ykarel|afk has joined #openstack-infra | 11:22 | |
| *** ykarel|afk is now known as ykarel | 11:23 | |
| *** ramishra has quit IRC | 11:32 | |
| *** sshnaidm|afk is now known as sshnaidm|bbl | 11:36 | |
| openstackgerrit | Michal Pryc proposed zuul/nodepool master: WIP: Implement a Devnest nodepool driver https://review.opendev.org/689474 | 11:46 |
| *** rh-jelabarre has joined #openstack-infra | 11:49 | |
| *** otsukahy has joined #openstack-infra | 12:02 | |
| *** markvoelker has joined #openstack-infra | 12:05 | |
| *** yamamoto has joined #openstack-infra | 12:05 | |
| *** yamamoto has quit IRC | 12:08 | |
| *** dtantsur|brb is now known as dtantsur | 12:09 | |
| *** goldyfruit has quit IRC | 12:14 | |
| *** rlandy has joined #openstack-infra | 12:17 | |
| *** rlandy_ has joined #openstack-infra | 12:17 | |
| *** rlandy has quit IRC | 12:17 | |
| *** rlandy_ is now known as rlandy | 12:17 | |
| *** rfolco|rover has joined #openstack-infra | 12:21 | |
| *** fresta has quit IRC | 12:21 | |
| *** fresta has joined #openstack-infra | 12:23 | |
| *** hamzy_ has joined #openstack-infra | 12:30 | |
| *** hamzy__ has quit IRC | 12:33 | |
| *** hamzy has joined #openstack-infra | 12:35 | |
| *** hamzy_ has quit IRC | 12:37 | |
| *** amoralej is now known as amoralej|lunch | 12:44 | |
| *** jamesmcarthur has joined #openstack-infra | 12:47 | |
| *** takamatsu has joined #openstack-infra | 12:48 | |
| *** yamamoto has joined #openstack-infra | 12:48 | |
| *** pcaruana has joined #openstack-infra | 12:53 | |
| *** dpawlik has quit IRC | 12:57 | |
| *** dave-mccowan has joined #openstack-infra | 12:57 | |
| *** ykarel_ has joined #openstack-infra | 12:58 | |
| *** ykarel_ is now known as ykarel|afk | 13:01 | |
| *** ykarel has quit IRC | 13:01 | |
| *** yamamoto has quit IRC | 13:02 | |
| *** hamzy has quit IRC | 13:03 | |
| *** dpawlik has joined #openstack-infra | 13:08 | |
| *** surpatil has quit IRC | 13:10 | |
| *** jamesmcarthur has quit IRC | 13:11 | |
| *** prometheanfire has quit IRC | 13:16 | |
| *** dave-mccowan has quit IRC | 13:17 | |
| *** prometheanfire has joined #openstack-infra | 13:18 | |
| *** amoralej|lunch is now known as amoralej | 13:23 | |
| *** rcernin has quit IRC | 13:24 | |
| *** goldyfruit has joined #openstack-infra | 13:25 | |
| *** whoami-rajat has quit IRC | 13:27 | |
| *** jamesmcarthur has joined #openstack-infra | 13:29 | |
| *** mriedem has joined #openstack-infra | 13:30 | |
| *** kjackal has quit IRC | 13:31 | |
| *** jamesmcarthur has quit IRC | 13:31 | |
| *** otsukahy has quit IRC | 13:32 | |
| *** kjackal has joined #openstack-infra | 13:32 | |
| *** eharney has quit IRC | 13:34 | |
| *** tosky has quit IRC | 13:38 | |
| *** otsukahy has joined #openstack-infra | 13:39 | |
| *** tosky has joined #openstack-infra | 13:39 | |
| *** ykarel|afk is now known as ykarel | 13:42 | |
| *** otsukahy has quit IRC | 13:44 | |
| *** jcoufal has joined #openstack-infra | 13:46 | |
| *** xek__ has joined #openstack-infra | 13:47 | |
| *** ramishra has joined #openstack-infra | 13:48 | |
| *** xek_ has joined #openstack-infra | 13:49 | |
| *** xek__ has quit IRC | 13:52 | |
| *** hamzy has joined #openstack-infra | 13:53 | |
| *** sshnaidm|bbl is now known as sshnaidm | 13:57 | |
| *** jamesmcarthur has joined #openstack-infra | 14:04 | |
| *** yolanda has quit IRC | 14:07 | |
| *** dpawlik has quit IRC | 14:11 | |
| *** otsukahy has joined #openstack-infra | 14:13 | |
| *** yolanda has joined #openstack-infra | 14:14 | |
| *** dave-mccowan has joined #openstack-infra | 14:16 | |
| *** otsukahy has quit IRC | 14:17 | |
| *** markvoelker has quit IRC | 14:20 | |
| *** diablo_rojo has joined #openstack-infra | 14:21 | |
| *** priteau has joined #openstack-infra | 14:21 | |
| *** ykarel is now known as ykarel|afk | 14:21 | |
| *** efried1 has joined #openstack-infra | 14:24 | |
| *** efried has quit IRC | 14:24 | |
| *** efried1 is now known as efried | 14:24 | |
| *** eharney has joined #openstack-infra | 14:27 | |
| clarkb | pypi is not mirrored, we simply cache results from pypi upstream. Those indexes have a 10 minitr ttl which means we should update them every 10 minutes (and we appear to). The problems seem to be due to updates from bad pypi cdns | 14:31 |
| clarkb | ralonsoh: ykarel|afk donnyd smcginnis ^ | 14:32 |
| *** markvoelker has joined #openstack-infra | 14:32 | |
| *** ykarel|afk is now known as ykarel | 14:32 | |
| ralonsoh | clarkb, ahhh ok | 14:32 |
| ykarel | clarkb, ack | 14:33 |
| ralonsoh | so we need just to wait for a good sync | 14:33 |
| clarkb | we've seen pacakages "disappear" as well asindexes without python version requirements | 14:33 |
| clarkb | ralonsoh: not necessarily as the next pull 10 minutes later could be bad | 14:33 |
| *** yamamoto has joined #openstack-infra | 14:33 | |
| clarkb | I thi k we likely need to escalate to pypi and ask if they can sort out the unhappy cdn node(s) | 14:33 |
| ralonsoh | ok I though that the mirror has a cache | 14:34 |
| clarkb | ralonsoh: it does but for those pypi package indexes the ttl is 10 minutes | 14:34 |
| clarkb | this means after 10 minutes the cache askes pypi if the data has changed. If it talks to a bad cdn frontend it will be told yesand given a bad index | 14:34 |
| clarkb | then 10 minutes later it could flip back to working | 14:35 |
| *** yamamoto has quit IRC | 14:35 | |
| *** dave-mccowan has quit IRC | 14:37 | |
| *** pkopec has quit IRC | 14:38 | |
| *** kopecmartin is now known as kopecmartin|off | 14:39 | |
| fungi | do we know yet which fastly endpoint(s) are affected? | 14:41 |
| fungi | i can pass it along in #pypa-dev if so | 14:41 |
| clarkb | fungi: yesterday I noted working pairs and broken pairs according to the headers | 14:41 |
| fungi | but we don't know which of the ones in a pair is to blame, guess | 14:42 |
| clarkb | lcy + bwi broken. iad + dca working. iad + bwi working | 14:43 |
| clarkb | righttheir headers always send a pair | 14:43 |
| clarkb | and I do t know how to query the frontends directly | 14:44 |
| *** Garyx_ has quit IRC | 14:44 | |
| *** Garyx_ has joined #openstack-infra | 14:44 | |
| clarkb | problems observed include the tempest 22.1.0 package not being listed and zuul not havibg python version requiremwnts in the index | 14:44 |
| clarkb | and it will flip back and forth, likely due to hitting good frontends then bad then good again | 14:46 |
| *** slaweq has quit IRC | 14:47 | |
| fungi | so it sounds like the only endopint we know of appearing strictly for stale indices is lcy? | 14:48 |
| fungi | as bwi appeared with lcy in headers for stale content but also in headers with iad for working content? | 14:49 |
| clarkb | assuming that us what the header values mean. It is possible that bwi is also broekn if the header means one of these | 14:49 |
| fungi | yeah, hard to know. i wonder why you get two cdn endpoints mentioned in a header | 14:49 |
| *** dklyle has quit IRC | 14:52 | |
| *** mattw4 has joined #openstack-infra | 14:53 | |
| fungi | i've asked in #pypa-dev, so we'll see what comes of that. fwiw i haven't seen anyone in there mention a similar problem in the past 24 hours | 14:55 |
| *** vkmc has left #openstack-infra | 14:56 | |
| *** dklyle has joined #openstack-infra | 14:56 | |
| *** pcaruana has quit IRC | 14:57 | |
| *** mattw4 has quit IRC | 14:59 | |
| *** yamamoto has joined #openstack-infra | 15:07 | |
| *** yamamoto has quit IRC | 15:11 | |
| fungi | they're looking for a list of the affected packages | 15:11 |
| fungi | i'll see what i can put together out of anecdotal reports up to this point | 15:11 |
| clarkb | zuul and tempest are what I know of off the top of my head | 15:12 |
| clarkb | we can probably do a logstash query | 15:12 |
| clarkb | fungi logstash says keystoneauth1 too | 15:14 |
| clarkb | message:"ERROR: Could not find a version that satisfies the requirement" is the query I am skimming results for | 15:14 |
| fungi | tempest===22.1.0, keystoneauth1===3.18.0, there were zuul and cryptography packages we noted missing at times too right? | 15:15 |
| clarkb | wiht zuul it was python version requirements missing from the index files | 15:16 |
| clarkb | I don't recall cryptography but wouldn't surprise me | 15:16 |
| fungi | cryptography===2.8 | 15:17 |
| *** kjackal has quit IRC | 15:17 | |
| clarkb | and chacnes are if we unbreak keystoneauth1 that the next one will start jumping out | 15:17 |
| fungi | definitely looks like recent releases too: cryptography===2.8 (uploaded 2019-10-17), tempest===22.1.0 (uploaded 2019-10-15), keystoneauth1===3.18.0 (uploaded 2019-10-22) | 15:21 |
| *** kjackal has joined #openstack-infra | 15:23 | |
| clarkb | smcginnis: ^ possible we may want to hold off on releases while we sort out why pypi is having problems | 15:23 |
| smcginnis | clarkb: Sounds like a plan. | 15:23 |
| * smcginnis does recall seeing a failure on cryptography | 15:25 | |
| *** iurygregory has quit IRC | 15:26 | |
| *** gyee has joined #openstack-infra | 15:26 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 15:28 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 15:32 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 15:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 15:38 |
| *** e0ne has quit IRC | 15:44 | |
| *** goldyfruit has quit IRC | 15:46 | |
| *** lmiccini has quit IRC | 15:47 | |
| *** sreejithp has joined #openstack-infra | 15:48 | |
| *** dtantsur is now known as dtantsur|afk | 15:48 | |
| *** igordc has joined #openstack-infra | 15:51 | |
| *** yamamoto has joined #openstack-infra | 15:52 | |
| clarkb | fungi: any indication from them as to whether this is a known issue or a problem on their end? or are they still investigating? | 15:53 |
| fungi | neither, they just want me to prove to them there's a problem first | 15:53 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Log an error on gerrit checks misconfiguration https://review.opendev.org/690616 | 15:54 |
| clarkb | I mean its really difficult to prove as an end user that doesn't know how to talk to cdn front ends directly... | 15:54 |
| clarkb | it wouldn't be difficult for them to do something like for frontend in cdn; do wget file from frontent ; done diff files | 15:55 |
| fungi | so i've come up with a shell loop to set /etc/hosts overrides for all the ipv4 and ipv6 addresses in turn, and grab copies of each of those packages' simple api indices along with http headers. i can't reproduce the problem with any of them at the moment, but i also don't know that necessarily is hitting different cdn endpoints. could just be anycast | 15:55 |
| clarkb | right | 15:55 |
| fungi | though i can check the headers i grabbed, just a sec | 15:56 |
| clarkb | you really need to be able to address the frontends directly | 15:56 |
| corvus | clarkb, fungi: do you think if i figured out how to track down a cache file on disk, we would get information like the original header information (with max-age) that apache saw? | 15:56 |
| clarkb | (and that is something that they can likely do and we can't) | 15:56 |
| clarkb | corvus: that is possible. though I don't think the max-age is a problem (at least it seems to be rotating the files which is why we see things break then work then break) | 15:57 |
| corvus | aha, i missed that we had confirmed movement there | 15:57 |
| clarkb | corvus: ya I think we confirmed that with tempest yesterday | 15:57 |
| fungi | yeah, but also i would rather not bring apache caching into the discussion or they're just going to point at it and say our caching is the problem and we shouldn't do that | 15:57 |
| corvus | i agree, if we have excluded "apache is stuck holding bad data past its intended expiration" that avenue is not likely to be fruitful. | 15:57 |
| fungi | and yeah, i don't think i'm hitting anywhere close to all the endpoints, these all look local to where i tested: http://paste.openstack.org/show/785525/ | 15:59 |
| corvus | i would hope not, it's the standard http protocol. but i understand the desire to keep the conversation as simple as possible. | 15:59 |
| clarkb | fungi: ya and LCY is our current most likely suspect which doesn't show up in your results | 15:59 |
| clarkb | did they explain why we always get pairs? | 16:00 |
| fungi | where were we seeing hits against lcy? i can test from a server in that vicinity | 16:00 |
| clarkb | (that might also help us narrow things down) | 16:00 |
| fungi | no, i've gotten basically no detail from the folks currently active in there | 16:00 |
| clarkb | rax-iad hit lcy according to my scrollback and lcy + bwi was broken for tempest 22.1.0 | 16:01 |
| *** yamamoto has quit IRC | 16:01 | |
| *** mattw4 has joined #openstack-infra | 16:01 | |
| clarkb | then when the record updated it worked andthat came from iad + dca | 16:01 |
| *** ccamacho has quit IRC | 16:02 | |
| *** thay_ has joined #openstack-infra | 16:02 | |
| fungi | i'll try this technique from something in iad in that case | 16:02 |
| fungi | i happen to have a personal vm there already | 16:02 |
| corvus | fungi, clarkb: https://docs.fastly.com/en/guides/understanding-cache-hit-and-miss-headers-with-shielded-services has some explanation | 16:02 |
| *** lucasagomes has quit IRC | 16:02 | |
| corvus | it looks like the pairs means we're dealing with layered caching ("shielding") | 16:03 |
| fungi | yeah, currently only hitting iad, dca and bwi from my server in iad, no matter which of the addresses from the round-robin i go through | 16:05 |
| zbr | do we have a role to install go-lang? | 16:05 |
| corvus | fungi: what pairs are you getting? | 16:05 |
| fungi | corvus: these: http://paste.openstack.org/show/785526/ | 16:05 |
| corvus | zbr: nope, please add to zuul-jobs :) | 16:05 |
| zbr | corvus: damn,... no change to reuse-previous work :p | 16:06 |
| clarkb | corvus: fungi ok reading up on shielding I think that means either LCY or BWI could be "broken:" | 16:06 |
| *** ykarel is now known as ykarel|afk | 16:07 | |
| clarkb | zbr: corvus fwiw I think most distros have a new enough golang. I was just installing it from ubuntu packages for kata when I dabbled with that | 16:07 |
| corvus | clarkb: agreed -- but maybe even just a single node in lcy or bwi? like cache-bwi5043-BWI vs cache-bwi5045-BWI ? | 16:07 |
| clarkb | corvus: ya | 16:07 |
| zbr | clarkb: i am afraid that my main focus is some distros does do not even ship one. i am afraid that bindep does not help me here. | 16:09 |
| *** thay_ has quit IRC | 16:10 | |
| clarkb | all of our mirrors currently produce a keystoneauth1 index with 3.18 present | 16:11 |
| clarkb | and logstash says no occurences for about 4 hours | 16:12 |
| clarkb | fungi: it is possible that all the cdn frontends are happy now and we won't see this until the next trigger of the bug | 16:12 |
| fungi | clarkb: entirely possible | 16:13 |
| fungi | ralonsoh's failure was from 2019-10-23 10:27:01.195594 | 16:14 |
| fungi | so that's been nearly 6 hours | 16:14 |
| ralonsoh | yes: https://68c70debed03e4c08f4d-3c5c6fcb6461190790889979f17d8403.ssl.cf1.rackcdn.com/689957/3/check/openstack-tox-pep8/7831849/job-output.txt | 16:15 |
| *** amoralej is now known as amoralej|off | 16:16 | |
| clarkb | other info, looking at logstash this started happening yesterday | 16:17 |
| clarkb | that may coincide with pypi's outage yesterday | 16:18 |
| clarkb | other packages I find include: os-traits==1.1.0 Paste==3.2.2 numpy==1.17.3 python-novaclient==16.0.0 python-keystoneclient==3.22.0 sqlalchemy-migrate==0.13.0 | 16:20 |
| clarkb | nowhere near the occurence rate of keystoneauth1 though | 16:20 |
| *** sshnaidm is now known as sshnaidm|afk | 16:22 | |
| zbr | clarkb: i found an existing role which sounds of decent quality (MIT/BSD license), how can I avoid duplicating it in zuul? | 16:23 |
| clarkb | oh some of those are from about a week ago. os-traits in particular. I misread the graph/timestamps | 16:23 |
| clarkb | then there was a big gap with no occurences then it picked way up yesterday | 16:23 |
| clarkb | zbr: zuul doesn't currenly support including roles from galaxy (though it is spec'd) | 16:24 |
| zbr | clarkb: so if I add it to requirements.yml it will not be installed. | 16:24 |
| clarkb | like from pypi? | 16:25 |
| clarkb | if you run a nested ansible that may work but I don't beleive that will get installed by anything on the executor to have the top level ansible use it | 16:25 |
| *** sgw has joined #openstack-infra | 16:25 | |
| *** jtomasek has quit IRC | 16:28 | |
| *** markvoelker has quit IRC | 16:33 | |
| *** xek__ has joined #openstack-infra | 16:33 | |
| *** tobiash has quit IRC | 16:35 | |
| *** xek_ has quit IRC | 16:36 | |
| *** tobiash has joined #openstack-infra | 16:37 | |
| *** jamesmcarthur has quit IRC | 16:41 | |
| *** markvoelker has joined #openstack-infra | 16:43 | |
| fungi | i need to disappear for an appointment, hopefully won't be too long | 16:44 |
| *** jamesmcarthur has joined #openstack-infra | 16:45 | |
| *** igordc has quit IRC | 16:50 | |
| *** rpittau is now known as rpittau|afk | 16:50 | |
| *** ykarel|afk is now known as ykarel|away | 16:52 | |
| clarkb | I'm told it is flu shot day. I'll be out for a bit too | 16:55 |
| efried | build results don't get purged if I abandon a change, right? | 17:08 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: Ensure both kubernetes and openshift token are b64decoded https://review.opendev.org/687435 | 17:08 |
| *** otsukahy has joined #openstack-infra | 17:08 | |
| clarkb | efried: if jobs arw running you may not get a build record in zuul for those jobs | 17:10 |
| clarkb | but any uploaded logs will be inswift if you can find them | 17:10 |
| *** aedc has quit IRC | 17:10 | |
| clarkb | and we expire those after 30 days | 17:10 |
| *** aedc has joined #openstack-infra | 17:10 | |
| *** otsukahy has quit IRC | 17:13 | |
| *** kjackal has quit IRC | 17:17 | |
| *** otsukahy has joined #openstack-infra | 17:24 | |
| *** kjackal has joined #openstack-infra | 17:25 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: Ensure both kubernetes and openshift token are b64decoded https://review.opendev.org/687435 | 17:31 |
| *** otsukahy has quit IRC | 17:32 | |
| *** jaosorior has quit IRC | 17:41 | |
| *** ykarel|away has quit IRC | 17:43 | |
| corvus | clarkb, fungi: 2 tox jobs on a single zuul change failed with a short read when... and i think this is what was happening... nodeenv was downloading node. is that mirrored or do we hit upstream? | 17:45 |
| corvus | eg https://zuul.opendev.org/t/zuul/build/367fb7b502cf47418b57a265375b8ec5 | 17:45 |
| *** jamesmcarthur has quit IRC | 17:45 | |
| *** jamesmcarthur has joined #openstack-infra | 17:46 | |
| *** amoralej|off is now known as amoralej | 17:48 | |
| *** otsukahy has joined #openstack-infra | 17:48 | |
| clarkb | I believe npm is proxy cached but I doubt nodenv knows to use it | 17:49 |
| *** jamesmcarthur has quit IRC | 17:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 17:51 | |
| *** whoami-rajat has joined #openstack-infra | 17:52 | |
| *** otsukahy has quit IRC | 17:52 | |
| *** otsukahy has joined #openstack-infra | 17:53 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Missing labels is a subset of allow_needs https://review.opendev.org/688645 | 17:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 17:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Log an error on gerrit checks misconfiguration https://review.opendev.org/690616 | 17:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 17:55 |
| *** ricolin has quit IRC | 17:55 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Update libre2 dep for buster https://review.opendev.org/690701 | 17:55 |
| *** otsukahy has quit IRC | 17:58 | |
| *** spsurya has quit IRC | 17:59 | |
| *** jerryz has joined #openstack-infra | 18:01 | |
| zbr | clarkb: as i cannot use galaxy, what should I do to install go? can I add the role as a subrepository? or this could be a very bad idea? | 18:01 |
| zbr | somehow I am not happy about duplicating the entire role from https://github.com/fubarhouse/ansible-role-golang | 18:02 |
| *** ricolin has joined #openstack-infra | 18:03 | |
| clarkb | as I mentioned I just did a package install of it. I'm not sure what you need so cant really offer advice | 18:04 |
| *** weifan has joined #openstack-infra | 18:05 | |
| zbr | clarkb: the deal is that on centos i do not have a package, i need to install it, which involves lots of actions. thus i am inclined to find a way to reuse existing work. | 18:05 |
| zbr | that one that i found is the most popular i found, but i am sure there are lots others. | 18:05 |
| zbr | not only initial work worries me but even later maintenance, if we add more-more roles inside zuul-jobs and never use stuff made by others we will endup with really huge maintenance work. | 18:07 |
| *** otsukahy has joined #openstack-infra | 18:08 | |
| zbr | i am not looking for an instant answer, but it would be good to know what others think about the suject. go-lang is just one example i found today. | 18:08 |
| *** ccamacho has joined #openstack-infra | 18:08 | |
| *** otsukahy has quit IRC | 18:13 | |
| clarkb | The long term plan is to add galaxy support to zuul | 18:16 |
| clarkb | that is speccedout just need implementing | 18:16 |
| zbr | clarkb: i made few discoveries regarding galaxy which do not make me very happy about it. for example failure to install unless you login to github. | 18:17 |
| *** kjackal has quit IRC | 18:17 | |
| clarkb | yes the github integration is one of the major gotchas iirc | 18:17 |
| clarkb | aiui galaxy is being updated to remove that though as part of the collections push | 18:18 |
| zbr | galaxy with git repos is bit more reliable because you can point it to non-github repos and clone from there. | 18:18 |
| *** jamesmcarthur has joined #openstack-infra | 18:21 | |
| *** igordc has joined #openstack-infra | 18:24 | |
| *** ricolin has quit IRC | 18:25 | |
| *** otsukahy has joined #openstack-infra | 18:31 | |
| corvus | yeah, i think zuul will grow support for collections rather than legacy galaxy | 18:31 |
| *** jamesmcarthur has quit IRC | 18:31 | |
| *** jamesmcarthur has joined #openstack-infra | 18:32 | |
| *** igordc has quit IRC | 18:32 | |
| clarkb | I'm back now | 18:33 |
| clarkb | http://mirror.iad.rax.opendev.org:8080/registry.npmjs/ is the npm proxy cache location | 18:33 |
| clarkb | or an example of one for the iad region in rax | 18:34 |
| corvus | zbr: i think there are two constructive things you could do, depending on what your timeline is: 1) long-term: start working on adding collections support to zuul (if the collections tools are ready). 2) short-term: add a role to zuul-jobs. it doesn't have to support all the platforms and options that role in galxy supports, so it may not be hard to maintain. | 18:34 |
| zbr | corvus: makes sense. | 18:35 |
| *** pcaruana has joined #openstack-infra | 18:38 | |
| *** jamesmcarthur has quit IRC | 18:39 | |
| *** jamesmcarthur has joined #openstack-infra | 18:39 | |
| *** kjackal has joined #openstack-infra | 18:42 | |
| *** otsukahy has quit IRC | 18:42 | |
| *** pcaruana has quit IRC | 18:44 | |
| *** efried has quit IRC | 18:45 | |
| *** ralonsoh has quit IRC | 18:45 | |
| *** sreejithp has quit IRC | 18:47 | |
| *** efried has joined #openstack-infra | 18:48 | |
| clarkb | no new pypi issues according to logstash since we last noticed them | 18:49 |
| clarkb | it seemed to happen fairly frequently (at least once an hour prior to that for the day prior | 18:50 |
| *** sreejithp has joined #openstack-infra | 18:52 | |
| *** sreejithp_ has joined #openstack-infra | 18:56 | |
| *** sreejithp has quit IRC | 18:58 | |
| *** jcoufal_ has joined #openstack-infra | 18:59 | |
| *** jcoufal has quit IRC | 19:01 | |
| *** amoralej is now known as amoralej|off | 19:03 | |
| openstackgerrit | Benjamin Navaro proposed openstack/project-config master: Enable inap-mtl01 https://review.opendev.org/690714 | 19:03 |
| clarkb | reading nodeenv source it uses npm to install nodejs | 19:03 |
| clarkb | it also uses npm to install npm | 19:04 |
| clarkb | I've not sorted out what bootstraps the base npm for that | 19:04 |
| clarkb | But all that means I think if we set an npm config (whatever that may be) that the npm calls (maybe not the first base install) will respect that config | 19:05 |
| benj_ | Gentlemen. I've submitted https://review.opendev.org/#/c/690714/ to reenable the inap-mtl01 region. However, the mirror in this region is no more. I'm not sure what's the process here, but, as of today, you should be able to rebuild it. | 19:06 |
| clarkb | benj_: that is excellent news. Thank you for letting us know we need to rebuild it | 19:06 |
| clarkb | corvus: we can create /etc/npmrc and set registry = http://mirror_fqdn:8080/registry.npmjs | 19:12 |
| clarkb | we can also set that in $project_dir/.npmrc or $HOME/.npmrc | 19:13 |
| clarkb | which might be more appropriate? | 19:13 |
| clarkb | looking at the zuul jobs I'm not realy sure about the best way to do this since we sort of slip nodeenv and related tools in under the python stuff | 19:16 |
| *** jamesmcarthur has quit IRC | 19:17 | |
| *** jamesmcarthur has joined #openstack-infra | 19:18 | |
| *** slaweq_ has joined #openstack-infra | 19:18 | |
| clarkb | hrm actually we should be able to update configure-mirrors and just do it | 19:20 |
| *** jamesmcarthur has quit IRC | 19:22 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Configure the npm mirror https://review.opendev.org/690718 | 19:24 |
| *** eharney has quit IRC | 19:27 | |
| *** otsukahy has joined #openstack-infra | 19:27 | |
| *** slaweq_ is now known as slaweq | 19:28 | |
| *** ociuhandu has joined #openstack-infra | 19:30 | |
| *** otsukahy has quit IRC | 19:32 | |
| *** ociuhandu has quit IRC | 19:35 | |
| *** eharney has joined #openstack-infra | 19:40 | |
| *** priteau has quit IRC | 19:40 | |
| *** otsukahy has joined #openstack-infra | 19:42 | |
| *** gouthamr has quit IRC | 19:43 | |
| *** ociuhandu has joined #openstack-infra | 19:45 | |
| *** gouthamr has joined #openstack-infra | 19:45 | |
| *** otsukahy has quit IRC | 19:46 | |
| *** igordc has joined #openstack-infra | 19:47 | |
| *** jamesmcarthur has joined #openstack-infra | 19:48 | |
| *** ociuhandu has quit IRC | 19:50 | |
| *** gouthamr_ has joined #openstack-infra | 19:54 | |
| *** otsukahy has joined #openstack-infra | 19:55 | |
| *** anteaya has joined #openstack-infra | 19:58 | |
| *** otsukahy has quit IRC | 20:00 | |
| *** otsukahy has joined #openstack-infra | 20:04 | |
| *** markvoelker has quit IRC | 20:07 | |
| *** otsukahy has quit IRC | 20:08 | |
| *** otsukahy has joined #openstack-infra | 20:09 | |
| *** kjackal has quit IRC | 20:12 | |
| *** e0ne has joined #openstack-infra | 20:12 | |
| *** otsukahy has quit IRC | 20:13 | |
| *** otsukahy has joined #openstack-infra | 20:20 | |
| *** jamesmcarthur has quit IRC | 20:20 | |
| *** jamesmcarthur has joined #openstack-infra | 20:21 | |
| *** gouthamr has quit IRC | 20:23 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| *** otsukahy has quit IRC | 20:24 | |
| *** jamesmcarthur has joined #openstack-infra | 20:26 | |
| *** otsukahy has joined #openstack-infra | 20:27 | |
| *** otsukahy has quit IRC | 20:32 | |
| *** jamesmcarthur has quit IRC | 20:35 | |
| *** jamesmcarthur has joined #openstack-infra | 20:36 | |
| *** auristor has quit IRC | 20:39 | |
| *** otsukahy has joined #openstack-infra | 20:41 | |
| *** e0ne has quit IRC | 20:43 | |
| *** aedc has quit IRC | 20:43 | |
| *** aedc has joined #openstack-infra | 20:43 | |
| *** markvoelker has joined #openstack-infra | 20:44 | |
| *** e0ne has joined #openstack-infra | 20:45 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Pin base-python back to stretch https://review.opendev.org/690742 | 20:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Pin python-base back to stretch https://review.opendev.org/690742 | 20:47 |
| *** xek__ has quit IRC | 20:48 | |
| *** xek__ has joined #openstack-infra | 20:48 | |
| *** otsukahy has quit IRC | 20:50 | |
| *** auristor has joined #openstack-infra | 20:51 | |
| *** slaweq has quit IRC | 20:51 | |
| *** xek__ has quit IRC | 20:51 | |
| *** xek_ has joined #openstack-infra | 20:52 | |
| *** jamesmcarthur has quit IRC | 20:52 | |
| *** jamesmcarthur has joined #openstack-infra | 20:53 | |
| *** eharney has quit IRC | 20:54 | |
| *** jcoufal_ has quit IRC | 20:55 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Update install for buster https://review.opendev.org/690701 | 21:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Missing labels is a subset of allow_needs https://review.opendev.org/688645 | 21:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 21:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Log an error on gerrit checks misconfiguration https://review.opendev.org/690616 | 21:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 21:03 |
| *** jamesmcarthur has quit IRC | 21:05 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Apply review container to review-dev https://review.opendev.org/690483 | 21:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use explicit image paths https://review.opendev.org/690511 | 21:06 |
| *** e0ne has quit IRC | 21:08 | |
| *** jamesmcarthur has joined #openstack-infra | 21:08 | |
| *** sreejithp_ has quit IRC | 21:16 | |
| *** mriedem is now known as mriedem_afk | 21:26 | |
| *** rfolco|rover has quit IRC | 21:31 | |
| *** ccamacho has quit IRC | 21:32 | |
| mordred | clarkb: any reason to not do this: https://review.opendev.org/#/c/674899/ and https://review.opendev.org/#/c/690485 before Shanghai? (might be nice to be on latest gitea when we talk to the gitea people) | 21:33 |
| mordred | corvus: ^^ | 21:33 |
| clarkb | mordred: I'm good with that | 21:34 |
| corvus | clarkb: i have not heard back from lunny (either in my original mail of 13 sept or my ping yesterday). may be worth checking if reg codes were used and you pinging if not | 21:36 |
| clarkb | corvus: asking now | 21:36 |
| *** goldyfruit has joined #openstack-infra | 21:38 | |
| corvus | mordred: +2s all around | 21:39 |
| corvus | mordred: i did not +3 | 21:39 |
| *** rh-jelabarre has quit IRC | 21:42 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 21:45 |
| mordred | corvus: cool. I'm around to watch and make sure things don't go south | 21:46 |
| mordred | clarkb, corvus: what's our rollback strategy do you think? should we push a docker tag for what we're currently running? | 21:48 |
| clarkb | mordred: I like that idea | 21:49 |
| corvus | mordred: yeah. how can we do that systemically? | 21:49 |
| clarkb | then we don't have to wait for gerrit and zuul to rebuild stuff | 21:49 |
| mordred | corvus: I mean - we could make a new zuul job that builds the current dockerfile and applies a label to it - like opendevorg/gitea:1.8 - and the make the 1.9.4 an opendevorg/gitea:1.9 ? | 21:50 |
| mordred | then once we're happy with 1.9 we can delete the 1.8 job? it's a little wonky though | 21:50 |
| corvus | i think the issue is that we basically have no ci testing of gitea, which means our normal practice of cd-ing everything isn't great. with no testing, the old "tag, rollout, rollback" approach is more suitable. :/ | 21:52 |
| mordred | yeah | 21:52 |
| corvus | (iirc, we test that it answers an http request, and that's it) | 21:52 |
| mordred | aren't we testing deployment and project creation with gitea containers? | 21:53 |
| clarkb | I think we do the full project creation | 21:53 |
| clarkb | ya | 21:53 |
| corvus | oh that's better | 21:53 |
| clarkb | that doesn't test git cloning or web ui interactons (updated templates for example) | 21:53 |
| corvus | right | 21:53 |
| mordred | yeah - system-config-run-gitea | 21:53 |
| clarkb | but it should be pretty robust from a "does't break gerrit replication" side | 21:53 |
| corvus | mordred: i'd be okay with the 1.8 job idea since it's a big version bump. i'd also be okay with "mordred does a docker pull of the current version just in case something goes really wrong and otherwise we hope for the best" | 21:55 |
| mordred | I have pulled opendevorg/gitea just to be safe - but I think I'd like to try the let's go for it version ... | 21:56 |
| openstackgerrit | Merged zuul/zuul master: Update install for buster https://review.opendev.org/690701 | 21:57 |
| mordred | I mean - we do have *some* testing | 21:57 |
| *** yamamoto has joined #openstack-infra | 21:59 | |
| corvus | cool, i also just retagged :latest as jeblair/gitea{-openssh}:20191023 | 22:00 |
| *** whoami-rajat has quit IRC | 22:01 | |
| mordred | incidentally - v1.10.0-rc1 exists - so 1.10 might be out soon - which would be nice, since it contains the parallel creation fix | 22:02 |
| *** yamamoto has quit IRC | 22:03 | |
| *** goldyfruit has quit IRC | 22:04 | |
| mordred | ok. here we go | 22:04 |
| mordred | corvus: re: gerrit - I made this: https://review.opendev.org/#/c/690483/ - which I was thinking of cloning to bridge and running manually against review-dev to make sure stuff worked there - does that seem sane to you? | 22:05 |
| *** ramishra has quit IRC | 22:07 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Mirror Ceph Nautilus for Debian based https://review.opendev.org/666263 | 22:07 |
| corvus | mordred: yep | 22:08 |
| clarkb | infra-root not sure if you saw the note but inap is ready for us eagain but we have to rebuild the mirror there first | 22:08 |
| mordred | corvus: cool | 22:08 |
| mordred | clarkb: neat! | 22:09 |
| clarkb | I assume we should build that as an opendev mirror on bionic and not bother iwth the openstack.org domain? | 22:09 |
| clarkb | if so I'll try to spin that up tomorrow morning unless someone else wants to go for it | 22:09 |
| ianw | clarkb: yep; want me to look into that today or you started? | 22:09 |
| clarkb | ianw: if you want to go for it feel free to. I'm finishing up some travel and ptg related stuff this afternoon | 22:09 |
| ianw | ok, can probably do that. just looking at the ceph mirroring, thinking we need to finish moving all the mirrors to opendev to drop the puppet bits | 22:10 |
| clarkb | that will also allow us to flip to https | 22:11 |
| clarkb | I like that plan | 22:11 |
| mordred | ++ | 22:12 |
| ianw | i'll make a list | 22:13 |
| openstackgerrit | Merged zuul/zuul master: Missing labels is a subset of allow_needs https://review.opendev.org/688645 | 22:14 |
| *** gfidente has quit IRC | 22:15 | |
| *** markvoelker has quit IRC | 22:22 | |
| rm_work | mtreinish: do you or someone here know who has the power to make a release for subunit? it's got bits that are broken in py3 that were fixed but no release has happened in over a year... | 22:23 |
| rm_work | I feel like 1.3.1 might be in order >_> | 22:24 |
| *** jamesmcarthur has quit IRC | 22:24 | |
| clarkb | rm_work: https://pypi.org/project/python-subunit/ shows you who can push to pypi | 22:26 |
| clarkb | lifeless: ^ you are one of them | 22:26 |
| rm_work | ah hey yeah :D | 22:26 |
| *** rcernin has joined #openstack-infra | 22:33 | |
| openstackgerrit | Merged zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 22:34 |
| *** jamesmcarthur has joined #openstack-infra | 22:38 | |
| *** diablo_rojo has quit IRC | 22:44 | |
| *** diablo_rojo has joined #openstack-infra | 22:45 | |
| ianw | i guess we should CNAME the openstack.org mirrors to opendev.org too | 22:48 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Switch nodepool builders to opendev.org mirrors https://review.opendev.org/690757 | 22:50 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Switch nodepool builders to opendev.org mirrors https://review.opendev.org/690757 | 22:51 |
| openstackgerrit | Merged zuul/zuul master: Log an error on gerrit checks misconfiguration https://review.opendev.org/690616 | 22:58 |
| *** dchen has joined #openstack-infra | 22:59 | |
| *** tosky has quit IRC | 22:59 | |
| *** tkajinam has joined #openstack-infra | 23:02 | |
| openstackgerrit | Merged opendev/system-config master: Update gitea to 1.9.4 https://review.opendev.org/674899 | 23:03 |
| *** weifan has quit IRC | 23:04 | |
| *** weifan has joined #openstack-infra | 23:04 | |
| *** jamesmcarthur has quit IRC | 23:08 | |
| ianw | clarkb / infra-root: https://storyboard.openstack.org/#!/story/2006762 worklist for mirror upgrades | 23:09 |
| *** jamesmcarthur has joined #openstack-infra | 23:10 | |
| fungi | thanks! | 23:11 |
| *** jamesmcarthur has quit IRC | 23:14 | |
| ianw | infra-root: removal of f28 nodes is ready to go too -> https://review.opendev.org/#/c/690432/ | 23:16 |
| *** lathiat has quit IRC | 23:16 | |
| openstackgerrit | Merged zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 23:18 |
| prometheanfire | is horizon having problems passing tests? (offhand)? | 23:18 |
| prometheanfire | https://zuul.opendev.org/t/openstack/build/c5eed4901ef24fabbb266d4574638793/log/job-output.txt#1153 | 23:18 |
| clarkb | I don't know | 23:21 |
| clarkb | fungi: fyi still no instances of the pypi problem according to logstash | 23:21 |
| fungi | prometheanfire: might be a question for #openstack-horizon | 23:22 |
| fungi | or you can try to hunt down recent builds of their unit test jobs | 23:22 |
| *** igordc has quit IRC | 23:23 | |
| prometheanfire | ack | 23:25 |
| prometheanfire | #join #openstack-horizon | 23:25 |
| prometheanfire | lol | 23:25 |
| *** yamamoto has joined #openstack-infra | 23:26 | |
| openstackgerrit | Merged opendev/system-config master: Update gitea build to golang 1.13 on buster-slim https://review.opendev.org/690485 | 23:26 |
| *** sgw has quit IRC | 23:32 | |
| mordred | clarkb: ^^ I think that should do what we discussed yesterday | 23:35 |
| ianw | fungi: actually, before we start brining up more mirrors we never fixed the initial install; could you check https://review.opendev.org/#/c/673724/ & https://review.opendev.org/#/c/673739/1 to disable recommends packages? | 23:35 |
| clarkb | mordred: yesterday or a couple hours ago? | 23:36 |
| mordred | clarkb: was that a couple of hours ago? I thought it was yesterday | 23:37 |
| mordred | clarkb: oh - sorry - bad arror | 23:37 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update pip3 role to install from get-pip.py https://review.opendev.org/690766 | 23:37 |
| mordred | clarkb: ^^ that | 23:37 |
| clarkb | now that makes sense :) | 23:38 |
| *** mattw4 has quit IRC | 23:38 | |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add mirror01.mtl01.inap.opendev.org https://review.opendev.org/690767 | 23:39 |
| mordred | clarkb, corvus: the gitea hosts seem to have pulled the new image - but they don't seem to have restarted with it - is there something else we need to do to do that? | 23:46 |
| clarkb | mordred: no docker compose should automatically do that when the image update | 23:46 |
| clarkb | mordred: maybe it hasn't completed yet? | 23:47 |
| mordred | oh - I take that back - docker ps shows it restarted - maybe I'm just load-balanced to a backend that hasn't gotten there yet | 23:47 |
| clarkb | oh ya we do them serially | 23:48 |
| clarkb | to avoid downtime | 23:48 |
| openstackgerrit | Merged opendev/system-config master: Mirror Ceph Nautilus for Debian based https://review.opendev.org/666263 | 23:48 |
| mordred | gitea01 has pulled and restarted, but only 01 | 23:48 |
| clarkb | 01 seems to be working | 23:51 |
| mordred | oh! wait - the ansible for 01 failed | 23:52 |
| mordred | that's why only 01 has applied | 23:52 |
| mordred | "Check if root user exists" is unhappy | 23:52 |
| clarkb | should we disable ansible on those hosts? | 23:53 |
| clarkb | though I suppose it should continue to fail on 01? | 23:53 |
| mordred | nah - it's a safeish failure mode. yeah | 23:53 |
| *** weifan has quit IRC | 23:54 | |
| clarkb | k | 23:54 |
| clarkb | fwiw it does seem to be working from a ui standpoint | 23:54 |
| clarkb | and maybe even loads nova pages quicker | 23:54 |
| mordred | [Macaron] 2019-10-23 23:55:11: Started GET /api/v1/users/root for 127.0.0.1 | 23:56 |
| mordred | 2019/10/23 23:55:11 ...les/context/panic.go:34:1() [E] PANIC:: runtime error: invalid memory address or nil pointer dereference | 23:56 |
| clarkb | ooh | 23:56 |
| mordred | did something change in the rest api? | 23:57 |
| clarkb | I'd expect a less panicy error if so | 23:59 |
| clarkb | that reads more like "segfault" | 23:59 |
| mordred | http://paste.openstack.org/show/785534/ | 23:59 |
| mordred | that's the whole log snippet | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!