| *** eernst has quit IRC | 00:13 | |
| *** rcernin has quit IRC | 00:17 | |
| *** rcernin has joined #openstack-infra | 00:18 | |
| *** hwoarang has quit IRC | 00:30 | |
| *** hwoarang has joined #openstack-infra | 00:31 | |
| *** smarcet has joined #openstack-infra | 00:37 | |
| *** hwoarang has quit IRC | 00:41 | |
| *** hwoarang has joined #openstack-infra | 00:42 | |
| *** rcernin has quit IRC | 00:49 | |
| *** rcernin has joined #openstack-infra | 00:50 | |
| *** hwoarang has quit IRC | 01:00 | |
| *** hwoarang has joined #openstack-infra | 01:00 | |
| *** markvoelker has quit IRC | 01:05 | |
| *** rfolco has quit IRC | 01:06 | |
| *** whoami-rajat has joined #openstack-infra | 01:06 | |
| *** hongbin has joined #openstack-infra | 01:10 | |
| *** lseki has quit IRC | 01:11 | |
| *** bobh has joined #openstack-infra | 01:12 | |
| *** ricolin has joined #openstack-infra | 01:13 | |
| *** hwoarang has quit IRC | 01:13 | |
| *** hwoarang has joined #openstack-infra | 01:14 | |
| *** bobh has quit IRC | 01:23 | |
| *** sreejithp has joined #openstack-infra | 01:27 | |
| *** sreejithp has quit IRC | 01:31 | |
| *** hwoarang has quit IRC | 01:33 | |
| *** hwoarang has joined #openstack-infra | 01:33 | |
| *** smarcet has quit IRC | 01:43 | |
| *** hwoarang has quit IRC | 01:47 | |
| *** hwoarang has joined #openstack-infra | 01:47 | |
| *** gregoryo has joined #openstack-infra | 01:58 | |
| *** hwoarang has quit IRC | 01:58 | |
| *** hwoarang has joined #openstack-infra | 01:59 | |
| *** hwoarang has quit IRC | 02:04 | |
| *** hwoarang has joined #openstack-infra | 02:04 | |
| openstackgerrit | Tony Breeds proposed opendev/storyboard master: Change the dist name to 'opendev-storyboard' https://review.opendev.org/661663 | 02:05 |
|---|---|---|
| *** udesale has quit IRC | 02:15 | |
| openstackgerrit | Merged openstack/reviewstats master: Adds tox docs job https://review.opendev.org/470642 | 02:19 |
| openstackgerrit | Merged openstack/reviewstats master: Add cinder-tempest-plugin repository for Cinder https://review.opendev.org/656675 | 02:25 |
| openstackgerrit | Merged openstack/reviewstats master: Add cinderlib repository for Cinder https://review.opendev.org/656682 | 02:29 |
| *** gmann has quit IRC | 02:31 | |
| *** hwoarang has quit IRC | 02:32 | |
| *** hwoarang has joined #openstack-infra | 02:32 | |
| *** hwoarang has quit IRC | 02:48 | |
| *** hwoarang has joined #openstack-infra | 02:49 | |
| *** ktsuyuzaki is now known as kota_ | 03:01 | |
| *** hwoarang has quit IRC | 03:03 | |
| *** hwoarang has joined #openstack-infra | 03:03 | |
| *** markvoelker has joined #openstack-infra | 03:06 | |
| *** jamesdenton has quit IRC | 03:20 | |
| *** psachin has joined #openstack-infra | 03:21 | |
| *** hwoarang has quit IRC | 03:21 | |
| *** hwoarang has joined #openstack-infra | 03:21 | |
| *** markvoelker has quit IRC | 03:40 | |
| *** RobertNordix has quit IRC | 03:55 | |
| *** imacdonn has quit IRC | 04:20 | |
| *** imacdonn has joined #openstack-infra | 04:21 | |
| *** pcaruana has joined #openstack-infra | 04:22 | |
| *** ykarel|afk has joined #openstack-infra | 04:31 | |
| *** markvoelker has joined #openstack-infra | 04:37 | |
| *** hwoarang has quit IRC | 04:38 | |
| *** psachin has quit IRC | 04:39 | |
| *** psachin has joined #openstack-infra | 04:39 | |
| *** hwoarang has joined #openstack-infra | 04:40 | |
| *** dchen has joined #openstack-infra | 04:46 | |
| *** hongbin has quit IRC | 04:49 | |
| *** janki has joined #openstack-infra | 04:51 | |
| *** raukadah is now known as chandankumar | 04:52 | |
| *** virendra has joined #openstack-infra | 04:56 | |
| virendra | Hi, I am getting error as below | 04:58 |
| virendra | ERROR: could not install deps [-chttps://releases.openstack.org/constraints/upper/master, -r/opt/stack/new/tempest/requirements.txt]; v = InvocationError('/opt/stack/new/tempest/.tox/tempest/bin/pip install -chttps://releases.openstack.org/constraints/upper/master -r/opt/stack/new/tempest/requirements.txt (see /opt/stack/new/tempest/.tox/tempest/log/full-1.log) | 04:58 |
| virendra | for 3rd party CI devstack installation for command "tox -r --notest -efull" | 04:59 |
| virendra | if anyone can help that will very helpful | 04:59 |
| *** YaminiU has joined #openstack-infra | 05:05 | |
| *** hwoarang has quit IRC | 05:05 | |
| YaminiU | Hi Team, we were using the new review.opendev.org url for our CI system over the past 24 hours it is failing. we are seeing this error. 2019-05-28 04:54:16,032 DEBUG zuul.AnsibleJob: [build: a247ab84ca2142b18d8ec4570eb652c4] Sending result: {"result": "ERROR", "error_detail": "Failed to update project review.opendev.org/openstack/nova"} 2019-05-28 04:54:16,036 DEBUG zuul.ExecutorServer: Sent SIGTERM to SSH Agent, {'SSH_AUTH | 05:06 |
| *** hwoarang has joined #openstack-infra | 05:06 | |
| YaminiU | did something change which is causing this failure. manually also that link no longer works | 05:06 |
| YaminiU | Can someone please help in this regard | 05:07 |
| *** ykarel|afk is now known as ykarel | 05:10 | |
| *** markvoelker has quit IRC | 05:10 | |
| *** ykarel_ has joined #openstack-infra | 05:13 | |
| tonyb | virendra: Are the logs somewhere? | 05:13 |
| virendra | Hi @tonyb, please find log "http://54.201.44.218/60/660460/2/check/3par-fc-driver-master-client-pip-ssa02-dsvm/4dfc718/logs/devstacklog.txt.gz" | 05:15 |
| *** ykarel has quit IRC | 05:15 | |
| *** ykarel__ has joined #openstack-infra | 05:16 | |
| *** hwoarang has quit IRC | 05:16 | |
| *** hwoarang has joined #openstack-infra | 05:17 | |
| tonyb | virendra: "error('Tunnel connection failed: 503 Connect failed'" | 05:17 |
| tonyb | virendra: is your proxy server able to get https://releases.openstack.org/constraints/upper/master ? | 05:17 |
| *** ykarel_ has quit IRC | 05:19 | |
| YaminiU | Hi Team, we were using the new review.opendev.org url for our CI system over the past 24 hours it is failing. we are seeing this error. 2019-05-28 04:54:16,032 DEBUG zuul.AnsibleJob: [build: a247ab84ca2142b18d8ec4570eb652c4] Sending result: {"result": "ERROR", "error_detail": "Failed to update project review.opendev.org/openstack/nova"} 2019-05-28 04:54:16,036 DEBUG zuul.ExecutorServer: Sent SIGTERM to SSH Agent, {'SSH_AUTH | 05:20 |
| YaminiU | this url is not working | 05:21 |
| YaminiU | has anyone faced this issue | 05:21 |
| YaminiU | how to resolve it | 05:21 |
| *** hwoarang has quit IRC | 05:22 | |
| *** hwoarang has joined #openstack-infra | 05:22 | |
| tonyb | YaminiU: The usful part of the error has been snipped :( | 05:27 |
| *** jchhatbar has joined #openstack-infra | 05:27 | |
| YaminiU | ok i will give paste url | 05:27 |
| tonyb | YaminiU: but at a guess I'd say you ssh creds are messed up | 05:27 |
| YaminiU | oh ok | 05:27 |
| tonyb | YaminiU: but I have no access to anything to verify that | 05:28 |
| *** janki has quit IRC | 05:30 | |
| YaminiU | http://paste.openstack.org/show/752137/ | 05:34 |
| YaminiU | hi tonyb please find the paste link | 05:34 |
| YaminiU | with the error | 05:34 |
| *** hwoarang has quit IRC | 05:35 | |
| *** hwoarang has joined #openstack-infra | 05:35 | |
| *** kjackal has joined #openstack-infra | 05:35 | |
| *** yboaron_ has joined #openstack-infra | 05:36 | |
| tonyb | YaminiU: Thanks, I may have been wrong about the root cause, sadly I don't have any idea how to debug that further :( | 05:36 |
| YaminiU | ok thanks Tonub | 05:36 |
| *** quiquell has joined #openstack-infra | 05:36 | |
| YaminiU | tonyb | 05:36 |
| YaminiU | hi tonyb can you please check this paste | 05:41 |
| YaminiU | http://paste.openstack.org/show/752138/ | 05:41 |
| virendra | tonyb: https://releases.openstack.org/constraints/upper/master not reachable through proxy, let me pass it through. | 05:46 |
| tonyb | virendra: okay at least that's a somewhat well defined problem | 05:47 |
| YaminiU | how do we increase the timeout for the clone from 300secs | 05:54 |
| virendra | tonyb: yes, thank You! | 05:58 |
| *** lpetrut has joined #openstack-infra | 06:01 | |
| tonyb | YaminiU: https://zuul-ci.org/docs/zuul/admin/components.html#attr-merger.git_timeout ? | 06:02 |
| *** kopecmartin|off is now known as kopecmartin | 06:04 | |
| *** e0ne has joined #openstack-infra | 06:06 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: openstack: document the key-name parameter https://review.opendev.org/661677 | 06:07 |
| *** markvoelker has joined #openstack-infra | 06:07 | |
| *** hwoarang has quit IRC | 06:11 | |
| *** hwoarang has joined #openstack-infra | 06:12 | |
| *** ykarel__ is now known as ykarel|afk | 06:14 | |
| *** ccamacho has joined #openstack-infra | 06:18 | |
| *** slaweq has joined #openstack-infra | 06:22 | |
| *** e0ne has quit IRC | 06:25 | |
| *** dpawlik has joined #openstack-infra | 06:32 | |
| *** yolanda has quit IRC | 06:38 | |
| *** markvoelker has quit IRC | 06:40 | |
| AJaeger | config-core, a small cleanup for review, please: https://review.opendev.org/661610 | 06:41 |
| *** ramishra has joined #openstack-infra | 06:48 | |
| *** ykarel|afk is now known as ykarel | 06:55 | |
| *** gregoryo has quit IRC | 06:56 | |
| YaminiU | hi team seeing the below error and CI jobs are fialing | 07:00 |
| YaminiU | http://paste.openstack.org/show/752141/ | 07:00 |
| YaminiU | @tonyb after increasing timeout am seeing this error | 07:00 |
| YaminiU | seems new | 07:00 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Replace skip with errors=ignore https://review.opendev.org/660541 | 07:01 |
| *** rpittau|afk is now known as rpittau | 07:02 | |
| openstackgerrit | Lucian Petrut proposed openstack/project-config master: Rename x/compute-hyperv to openstack/compute-hyperv https://review.opendev.org/661682 | 07:07 |
| *** takamatsu has joined #openstack-infra | 07:07 | |
| *** jhesketh has quit IRC | 07:11 | |
| *** ricolin_ has joined #openstack-infra | 07:12 | |
| *** tesseract has joined #openstack-infra | 07:12 | |
| *** takamatsu has quit IRC | 07:13 | |
| tonyb[m] | https://usercontent.irccloud-cdn.com/file/KAEjmkeu/irccloudcapture5579368549190068292.jpg | 07:14 |
| *** ricolin has quit IRC | 07:15 | |
| *** ginopc has joined #openstack-infra | 07:15 | |
| tonyb[m] | I did 5k in almost exactly 30mins the rest was a cool down walk | 07:17 |
| tonyb[m] | Lol wrong channel :-( #egg on face | 07:17 |
| *** yamamoto has joined #openstack-infra | 07:18 | |
| *** yboaron_ has quit IRC | 07:22 | |
| *** parasitid has joined #openstack-infra | 07:23 | |
| parasitid | hi there | 07:23 |
| parasitid | i have an issue with mirrors on nodepools | 07:24 |
| parasitid | the openstack loci project | 07:24 |
| evrardjp | tonyb[m]: hahaha | 07:24 |
| parasitid | is based on stuff available on the mirrors | 07:24 |
| parasitid | (eg http://mirror.iad.rax.openstack.org/opensuse/update/leap/) | 07:24 |
| evrardjp | parasitid: oh I am just looking at this | 07:25 |
| parasitid | and the loci project targets the opensuse/leap:15 docker container | 07:25 |
| parasitid | but it has been updated 6 days ago and now the "15" docker tag points to the "15.1" version | 07:25 |
| *** yamamoto has quit IRC | 07:25 | |
| parasitid | which isnt available on the mirrors | 07:26 |
| parasitid | evrardjp: ok thanks | 07:26 |
| evrardjp | parasitid: there are multiple things that break loci right now, but they are indeed all waiting for a change in infra mirrors | 07:26 |
| parasitid | i wonder if i should propose a patchset in loci to explicitly target the 15.0 version | 07:27 |
| evrardjp | https://review.opendev.org/#/c/660126/ just merged | 07:27 |
| evrardjp | parasitid: please don't | 07:27 |
| parasitid | ok | 07:27 |
| parasitid | thanks | 07:27 |
| evrardjp | we should just make it work in infra | 07:27 |
| evrardjp | that would be great if there is someone that can give us a view of why the patch above merged, but the mirrors (like rax) aren't updated yet -- maybe there is a problem with the script? | 07:28 |
| *** apetrich has joined #openstack-infra | 07:29 | |
| *** jhesketh has joined #openstack-infra | 07:29 | |
| *** happyhemant has joined #openstack-infra | 07:31 | |
| *** logan- has quit IRC | 07:33 | |
| *** dchen has quit IRC | 07:34 | |
| *** tosky has joined #openstack-infra | 07:35 | |
| *** RobertNordix has joined #openstack-infra | 07:35 | |
| *** logan- has joined #openstack-infra | 07:36 | |
| *** markvoelker has joined #openstack-infra | 07:37 | |
| ianw | evrardjp: like that *just* merged? | 07:47 |
| ianw | sorry, i thought i'd reviewed that. anyway, it won't apply till the next mirror update pulse | 07:47 |
| *** xek_ has joined #openstack-infra | 07:48 | |
| ianw | which you can sort of watch here -> http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1 | 07:48 |
| *** rossella_s has joined #openstack-infra | 07:49 | |
| ianw | but now i notice opensuse hasn't released | 07:50 |
| ianw | evrardjp: http://paste.openstack.org/show/752142/ | 07:50 |
| ianw | is that our problem, or the remote end? | 07:50 |
| evrardjp | that's a very good question :p | 07:52 |
| evrardjp | let me do my homework there then I guess :) | 07:52 |
| evrardjp | and thanks for pointing me to the watch part | 07:52 |
| *** jpena|off is now known as jpena | 07:52 | |
| ianw | evrardjp: sorry, i've got to run off ... unfortunately the logs aren't exactly exposed so stuff like that you really need an infra-root to see. but yeah, at least on the granfana page it should stamp if it *is* updating | 07:58 |
| *** Emine has joined #openstack-infra | 07:58 | |
| *** ykarel is now known as ykarel|lunch | 08:00 | |
| *** ginopc has quit IRC | 08:03 | |
| evrardjp | ianw: thanks already | 08:07 |
| evrardjp | I am in meetings so I couldn't give fast feedback there :( | 08:07 |
| evrardjp | I will try to understand the full thing and get back here, asking the infra-roots :) | 08:07 |
| *** markvoelker has quit IRC | 08:10 | |
| *** lucasagomes has joined #openstack-infra | 08:11 | |
| *** gmann has joined #openstack-infra | 08:13 | |
| *** yamamoto has joined #openstack-infra | 08:14 | |
| *** YaminiU has quit IRC | 08:23 | |
| *** pkopec has joined #openstack-infra | 08:24 | |
| *** ralonsoh has joined #openstack-infra | 08:26 | |
| gmann | clarkb: replied on this. legacy jobs testing of latest stable should be enough - https://review.opendev.org/#/c/647912/3 | 08:32 |
| *** yboaron_ has joined #openstack-infra | 08:34 | |
| *** piotrowskim has joined #openstack-infra | 08:40 | |
| *** yboaron_ has quit IRC | 08:40 | |
| *** yboaron_ has joined #openstack-infra | 08:41 | |
| *** gfidente has joined #openstack-infra | 08:52 | |
| *** derekh has joined #openstack-infra | 08:52 | |
| *** ginopc has joined #openstack-infra | 08:56 | |
| *** priteau has joined #openstack-infra | 08:57 | |
| *** e0ne has joined #openstack-infra | 09:00 | |
| *** rascasoft has joined #openstack-infra | 09:01 | |
| openstackgerrit | Merged openstack/project-config master: Cleanup in-tree removed jobs https://review.opendev.org/661610 | 09:02 |
| *** rascasoft has quit IRC | 09:03 | |
| *** rascasoft has joined #openstack-infra | 09:03 | |
| *** rascasoft has quit IRC | 09:04 | |
| aspiers | fungi, corvus: https://gitenterprise.me/2019/05/20/gerrit-v3-0-is-here/ | 09:06 |
| aspiers | it requires migrating through all the intermediate releases though | 09:07 |
| *** markvoelker has joined #openstack-infra | 09:07 | |
| *** tkajinam has quit IRC | 09:13 | |
| *** rascasoft has joined #openstack-infra | 09:14 | |
| *** jesusaur has quit IRC | 09:17 | |
| *** jchhatba_ has joined #openstack-infra | 09:18 | |
| *** jesusaur has joined #openstack-infra | 09:20 | |
| *** jchhatbar has quit IRC | 09:21 | |
| *** rossella_s has quit IRC | 09:22 | |
| *** yoctozepto has joined #openstack-infra | 09:25 | |
| *** sreejithp has joined #openstack-infra | 09:28 | |
| *** sreejithp has quit IRC | 09:33 | |
| *** ykarel|lunch is now known as ykarel | 09:33 | |
| *** markvoelker has quit IRC | 09:40 | |
| *** rossella_s has joined #openstack-infra | 09:46 | |
| *** ricolin_ has quit IRC | 10:02 | |
| *** dpawlik has quit IRC | 10:06 | |
| *** ramishra has quit IRC | 10:11 | |
| *** dpawlik has joined #openstack-infra | 10:18 | |
| *** emine__ has joined #openstack-infra | 10:20 | |
| *** Emine has quit IRC | 10:22 | |
| *** dpawlik has quit IRC | 10:22 | |
| *** ramishra has joined #openstack-infra | 10:25 | |
| *** dpawlik has joined #openstack-infra | 10:35 | |
| *** markvoelker has joined #openstack-infra | 10:37 | |
| *** dpawlik has quit IRC | 10:40 | |
| *** virendra has quit IRC | 10:49 | |
| *** yamamoto has quit IRC | 10:50 | |
| *** dpawlik has joined #openstack-infra | 11:02 | |
| *** yamamoto has joined #openstack-infra | 11:04 | |
| *** markvoelker has quit IRC | 11:11 | |
| *** ginopc has quit IRC | 11:11 | |
| *** dave-mccowan has joined #openstack-infra | 11:13 | |
| *** psachin has quit IRC | 11:14 | |
| *** yamamoto has quit IRC | 11:14 | |
| *** yamamoto has joined #openstack-infra | 11:20 | |
| *** yamamoto has quit IRC | 11:21 | |
| *** AJaeger has quit IRC | 11:24 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 11:27 |
| *** yamamoto has joined #openstack-infra | 11:32 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| *** yamamoto has quit IRC | 11:34 | |
| *** AJaeger has joined #openstack-infra | 11:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Move x/monitorstack jobs in-tree https://review.opendev.org/661733 | 11:42 |
| AJaeger | config-core, I forgot to add one repo, could you review ^, please? | 11:42 |
| *** yamamoto has joined #openstack-infra | 11:44 | |
| aspiers | AJaeger: if the same fix applies over multiple repos, is it best practice to reuse the Change-Id? | 11:44 |
| *** yamamoto has quit IRC | 11:48 | |
| openstackgerrit | Nir Magnezi proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.opendev.org/643731 | 11:50 |
| *** lucasagomes has quit IRC | 11:51 | |
| AJaeger | aspiers: it's nice but not needed. I would require same topic on the other hand | 11:51 |
| *** lucasagomes has joined #openstack-infra | 11:51 | |
| AJaeger | aspiers: if you want to do it, go for it - but I wouldn't enforce it. | 11:51 |
| aspiers | AJaeger: OK thanks! | 11:51 |
| *** rh-jelabarre has joined #openstack-infra | 11:51 | |
| openstackgerrit | Adam Spiers proposed opendev/git-review master: Add missing upper-constraints.txt to deps in docs testenv https://review.opendev.org/661736 | 11:55 |
| *** factor has joined #openstack-infra | 11:57 | |
| *** psachin has joined #openstack-infra | 11:57 | |
| *** rfolco has joined #openstack-infra | 11:59 | |
| *** electrofelix has joined #openstack-infra | 12:02 | |
| *** markvoelker has joined #openstack-infra | 12:06 | |
| *** yamamoto has joined #openstack-infra | 12:08 | |
| *** rf0lc0 has joined #openstack-infra | 12:09 | |
| *** rfolco has quit IRC | 12:10 | |
| *** otherwiseguy_ has joined #openstack-infra | 12:10 | |
| *** rfarr has joined #openstack-infra | 12:12 | |
| *** priteau has quit IRC | 12:13 | |
| *** priteau has joined #openstack-infra | 12:14 | |
| *** yamamoto has quit IRC | 12:17 | |
| *** rlandy has joined #openstack-infra | 12:26 | |
| *** _erlon_ has joined #openstack-infra | 12:26 | |
| *** derekh has quit IRC | 12:27 | |
| *** chason has joined #openstack-infra | 12:30 | |
| *** rossella_s has quit IRC | 12:32 | |
| *** rf0lc0 has quit IRC | 12:34 | |
| *** rfolco has joined #openstack-infra | 12:34 | |
| *** rf0lc0 has joined #openstack-infra | 12:35 | |
| *** rfolco has quit IRC | 12:38 | |
| openstackgerrit | Adam Spiers proposed opendev/git-review master: Build docs using Python 3 https://review.opendev.org/661750 | 12:48 |
| evrardjp | is there an infra-root which can help me on the oudated mirror.opensuse? ianw gave me a first hint, but I would need some extra help. | 12:48 |
| frickler | evrardjp: dirk: seems the opensuse mirror update job is failing and thus not updating the mirror, this is the latest error: http://paste.openstack.org/show/752150/ | 12:52 |
| fungi | we can try removing those from the local tree and see if it's able to sync now, though if they're still present in the remote tree then it will just continue doing that | 12:55 |
| fungi | pretty sure i determined that error was about the directories on the remote end anyway, though it's been months since i last looked into it | 12:56 |
| *** derekh has joined #openstack-infra | 12:57 | |
| fungi | looking back through my notes, last time it was tumbleweed which we couldn't update due to stale .~tmp~ dirs | 12:58 |
| *** igordc has joined #openstack-infra | 12:59 | |
| evrardjp | frickler: dirk is on holiday | 12:59 |
| evrardjp | but it seems the mirror is correct, and has the right files | 13:00 |
| *** dave-mccowan has quit IRC | 13:00 | |
| evrardjp | so I am thinking those `.tmp` files are an artifact. | 13:00 |
| fungi | if you're browsing the mirror through a webserver it will likely hide anything starting with a . | 13:00 |
| evrardjp | fungi: would it make sense to add .tmp to excludes then? | 13:00 |
| evrardjp | fungi: I just checked with rsync to list files, it doesn't list it | 13:01 |
| evrardjp | but my rsync foo is low | 13:01 |
| fungi | evrardjp: i can't remember if we tried that before, but to start with i'll delete the local ones and see if they reappear | 13:01 |
| evrardjp | fungi: that would be awesome | 13:01 |
| *** yamamoto has joined #openstack-infra | 13:02 | |
| openstackgerrit | Adam Spiers proposed opendev/git-review master: Change StoryBoard URL to be a permalink https://review.opendev.org/661451 | 13:06 |
| fungi | evrardjp: same error. looks like the mirror we're being directed to is mirror.wdc1.us.leaseweb.net so maybe there's more than one and this one is broken? | 13:07 |
| evrardjp | well | 13:07 |
| openstackgerrit | Adam Spiers proposed opendev/git-review master: Fix complimentary typo https://review.opendev.org/661453 | 13:07 |
| evrardjp | leaseweb is used only for product and updates | 13:07 |
| evrardjp | the OBS repos are supposed to be sync on provo-mirror | 13:08 |
| evrardjp | I thought the error was obs only | 13:08 |
| *** jchhatba_ has quit IRC | 13:08 | |
| evrardjp | do you mean there is another issue somewhere? | 13:08 |
| *** zul has joined #openstack-infra | 13:08 | |
| *** jchhatba_ has joined #openstack-infra | 13:08 | |
| fungi | ahh, we're not getting it out of a round-robin or dns glb, it looks like we set it explicitly | 13:09 |
| *** ekultails has joined #openstack-infra | 13:09 | |
| evrardjp | oh interesting I just re-did a rsync on that folder, and I get a .~tmp~ now | 13:09 |
| fungi | and an i/o error out of rsync? | 13:10 |
| evrardjp | rsync saying permission denied on said ~tmp~ file | 13:11 |
| fungi | yep | 13:12 |
| fungi | so same error basically | 13:12 |
| evrardjp | works if I have the exclude. | 13:12 |
| evrardjp | (ofc) | 13:12 |
| evrardjp | let me add this to the mirroring, if you don't mind. | 13:13 |
| evrardjp | And see how it goes | 13:13 |
| evrardjp | it can be an artifact from our side of the mirroring, but I don't see a valid reason to sync those files | 13:13 |
| evrardjp | (identically I don't see a reason to NOT clean up on our side at the same time) | 13:13 |
| evrardjp | fungi: you notes about tumbleweed, are they about obs repos? | 13:14 |
| evrardjp | or anything mirrored out of opensuse.org directly? | 13:14 |
| *** mriedem has joined #openstack-infra | 13:15 | |
| fungi | checking | 13:15 |
| *** jpena|lunch is now known as jpena | 13:16 | |
| fungi | looks like we were getting /afs/.openstack.org/mirror/opensuse/tumbleweed/repo/oss/i686/.~tmp~ and similar for other architectures | 13:16 |
| fungi | at the time dirk decided it was okay if tumbleweed mirroring went stale from time to time and so we adjusted the script to just ignore tumbleweed mirror errors and still mirror the other distributions in the script | 13:17 |
| *** rfarr has quit IRC | 13:18 | |
| *** beekneemech is now known as bnemec | 13:18 | |
| fungi | and yeah, i was misreading the server motd in the log. we're using mirror.wdc1.us.leaseweb.net for other (working) stuff there, but the errors are coming from a server which announces itself as stage.opensuse.org | 13:19 |
| openstackgerrit | Jean-Philippe Evrard proposed opendev/system-config master: Exclude tmp files artifacts from syncing https://review.opendev.org/661752 | 13:20 |
| *** priteau has quit IRC | 13:20 | |
| *** aaronsheffield has joined #openstack-infra | 13:20 | |
| evrardjp | fungi: ok let me check on the suse side about stage.opensuse.org | 13:20 |
| evrardjp | but where does this come from though? | 13:20 |
| *** tosky__ has joined #openstack-infra | 13:21 | |
| evrardjp | I don't see any reference to stage from the mirror script | 13:21 |
| *** tosky is now known as Guest72073 | 13:21 | |
| *** tosky__ is now known as tosky | 13:21 | |
| evrardjp | fungi: on the patch just above I prevented the syncing from those files all over the place ^ | 13:21 |
| evrardjp | fungi: oh I guess the motd of the mirror on provo? | 13:22 |
| *** yamamoto has quit IRC | 13:23 | |
| fungi | evrardjp: in the motd when the script connects to provo-mirror.opensuse.org, this is what it says: http://paste.openstack.org/show/752156/ | 13:23 |
| fungi | so, yes | 13:23 |
| evrardjp | yes got it :) | 13:23 |
| *** yamamoto has joined #openstack-infra | 13:24 | |
| fungi | i can hand-apply 661752 and see if it works around this | 13:24 |
| evrardjp | that would be awesome to see if this works | 13:24 |
| evrardjp | because the files will still be there | 13:24 |
| *** psachin has quit IRC | 13:24 | |
| evrardjp | as I said, my rsync foo is bad :p | 13:24 |
| rosmaita | frickler: could you take a look at https://review.opendev.org/#/c/652711/2 when you have a few minutes? I think I addressed your concerns. Thanks! | 13:26 |
| *** sreejithp has joined #openstack-infra | 13:26 | |
| fungi | evrardjp: i've removed the local .~tmp~ dirs in afs and started a copy of the opensuse-mirror-update script patched with 661752 | 13:28 |
| evrardjp | so it ... seems to work? | 13:29 |
| evrardjp | let's see | 13:29 |
| evrardjp | I will still clarify on the stage./provo-mirror. side | 13:29 |
| fungi | looks like it's copying stuff from there now. already finished /afs/.openstack.org/mirror/opensuse/repositories/Cloud:/OpenStack:/Rocky/openSUSE_Leap_42.3/ | 13:30 |
| evrardjp | As long as I am getting 15.1 mirrored, I am just fine :D | 13:30 |
| evrardjp | I am surprised this blocks the rest of the mirroring though, as obs seem lower in prio than the rest | 13:31 |
| evrardjp | to me at least :p | 13:31 |
| evrardjp | but at the end I prefer knowing if something is wrong | 13:31 |
| *** rf0lc0 is now known as rfolco | 13:32 | |
| fungi | lots of package names with lp151 in them now | 13:32 |
| *** sreejithp has quit IRC | 13:32 | |
| evrardjp | "in them"? | 13:33 |
| fungi | in the version strings | 13:33 |
| fungi | noarch/cassandra-kit-3.11.3-lp151.2.1.noarch.rpm | 13:33 |
| fungi | et cetera | 13:33 |
| evrardjp | ok, not in 42.3 or something | 13:33 |
| evrardjp | sounds good | 13:33 |
| evrardjp | sounds great, even! | 13:33 |
| fungi | and yeah, the script short-circuits if an error is encountered mirroring any of those except tumbleweed, but perhaps the sequence should be rearranged | 13:34 |
| evrardjp | I like to know when something is wrong though | 13:35 |
| evrardjp | I think "the system works" | 13:35 |
| rosmaita | fungi: when you have a few minutes, let me know what you think of my last comment on https://review.opendev.org/#/c/655451/1 ... thanks! | 13:36 |
| fungi | you bet | 13:36 |
| AJaeger | fungi, if you're reviewing, please look at https://review.opendev.org/661733 as well - thanks | 13:38 |
| *** ginopc has joined #openstack-infra | 13:39 | |
| *** DinaBelova has joined #openstack-infra | 13:42 | |
| fungi | rosmaita: the devstack change mentions that setting periodic_interval "may have unintentional adverse effects" but i can't find where any of those are enumerated | 13:42 |
| rosmaita | fungi: there's a reference in the bug to another bug that explains: https://bugs.launchpad.net/cinder/+bug/1823748 | 13:44 |
| openstack | Launchpad bug 1823748 in Cinder "periodic_interval used in too many places" [Undecided,Fix released] - Assigned to Brian Rosmaita (brian-rosmaita) | 13:45 |
| *** priteau has joined #openstack-infra | 13:46 | |
| *** rf0lc0 has joined #openstack-infra | 13:46 | |
| fungi | aha, thanks | 13:46 |
| *** rfolco has quit IRC | 13:48 | |
| corvus | ttx: can you take a look at https://review.opendev.org/661385 | 13:50 |
| frickler | AJaeger: neutron-functional job is failing on ds-gate because tox is running in the ds-gate repo instead of neutron. not sure what changed, this doesn't seem to have been broken for long. http://logs.openstack.org/51/655451/1/check/neutron-functional/d195967/ara-report/result/1d6232f6-e3f7-45b2-bb64-9da99f3fbbd0/ | 13:50 |
| frickler | rosmaita: added to my list, thanks for the reminder | 13:50 |
| rosmaita | frickler: thanks, appreciate it | 13:51 |
| *** Goneri has joined #openstack-infra | 13:51 | |
| AJaeger | frickler: strange, does not ring a bell | 13:53 |
| *** jchhatba_ has quit IRC | 13:55 | |
| AJaeger | frickler: https://opendev.org/openstack/neutron/src/branch/master/.zuul.yaml#L137 sets it - according to git blame no recent changes | 13:57 |
| *** rf0lc0 is now known as rfolco | 13:57 | |
| AJaeger | frickler: oh, you say the chdir is wrong? That is really strange ;( | 13:58 |
| AJaeger | frickler: better ask corvus ^ | 13:59 |
| AJaeger | frickler: if you have time, could you review https://review.opendev.org/661733 , please? | 13:59 |
| corvus | o/ | 14:00 |
| frickler | AJaeger: corvus: actually I think this change by slaweq may be the culprit, maybe one of you can help fixing it https://review.opendev.org/619742 | 14:02 |
| *** rpittau is now known as rpittau|afk | 14:08 | |
| slaweq | frickler: what I broke this time? :) | 14:10 |
| corvus | frickler, slaweq: i think if you set 'zuul_work_dir: "{{ devstack_base_dir }}/neutron" that would do the trick. see https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.zuul_work_dir | 14:10 |
| *** yamamoto has quit IRC | 14:11 | |
| *** yamamoto has joined #openstack-infra | 14:14 | |
| *** yamamoto has quit IRC | 14:14 | |
| *** yamamoto has joined #openstack-infra | 14:15 | |
| openstackgerrit | Merged openstack/project-config master: Move x/monitorstack jobs in-tree https://review.opendev.org/661733 | 14:15 |
| mordred | infra-root: I'm here, but have a stack of meetings/phone calls - so will be slow to respond | 14:17 |
| openstackgerrit | Merged opendev/system-config master: Exclude tmp files artifacts from syncing https://review.opendev.org/661752 | 14:18 |
| *** yamamoto has quit IRC | 14:19 | |
| *** yamamoto has joined #openstack-infra | 14:20 | |
| *** yamamoto has quit IRC | 14:20 | |
| *** yamamoto has joined #openstack-infra | 14:20 | |
| corvus | mordred: ack; i'm going to try to work on that image job testing project today | 14:24 |
| corvus | and i assume we'll switch the load balancer algo when clarkb comes online | 14:24 |
| mordred | corvus: ++ | 14:25 |
| *** yamamoto has quit IRC | 14:25 | |
| *** chandankumar is now known as raukadah | 14:25 | |
| mordred | during the moments where I'm not in a meeting I'm going to continue working on the status.o.o transition | 14:25 |
| mordred | speaking of: https://review.opendev.org/#/c/661385/1 <-- ttx do you have any opinion on this? | 14:26 |
| corvus | mordred: yeah, i +2d your second patch and will be happy to +2 that if ttx okays it | 14:27 |
| *** dpawlik has quit IRC | 14:29 | |
| *** lpetrut has quit IRC | 14:30 | |
| *** kjackal has quit IRC | 14:33 | |
| *** kjackal has joined #openstack-infra | 14:33 | |
| *** quiquell has quit IRC | 14:34 | |
| *** ykarel has quit IRC | 14:35 | |
| openstackgerrit | Tim Burke proposed openstack/project-config master: Rename x/pyeclib and x/liberasurecode to openstack/* https://review.opendev.org/657155 | 14:39 |
| *** lseki has joined #openstack-infra | 14:40 | |
| corvus | mordred: https://review.opendev.org/640027 has 2x2; not sure if we want to +W it today | 14:43 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 14:44 |
| clarkb | corvus: mordred I'm probably 40 minutes from keyboard, but ya want to change to source balance method, send out meeting agenda (late) , then start making sure we areready for friday renames | 14:44 |
| mordred | clarkb: you're a friday rename | 14:44 |
| corvus | clarkb: ack. i should be around in a supporting role for the lb change | 14:45 |
| corvus | we have data at http://grafana.openstack.org/d/jlskTlZZz/opendev-load-balancer?orgId=1 | 14:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 14:49 |
| *** ykarel has joined #openstack-infra | 14:53 | |
| *** lpetrut has joined #openstack-infra | 14:59 | |
| *** dklyle has joined #openstack-infra | 14:59 | |
| *** udesale has joined #openstack-infra | 15:00 | |
| *** e0ne has quit IRC | 15:05 | |
| *** yboaron_ has quit IRC | 15:07 | |
| * fungi is also happy to help with the haproxy stuff today | 15:09 | |
| *** roman_g has quit IRC | 15:11 | |
| *** ykarel is now known as ykarel|away | 15:13 | |
| clarkb | is there a change yet for the balance method swap? | 15:15 |
| corvus | not that i'm aware of | 15:15 |
| clarkb | k writing one | 15:15 |
| *** michael-beaver has joined #openstack-infra | 15:15 | |
| *** tosky has quit IRC | 15:16 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 15:17 |
| *** hamzy has quit IRC | 15:18 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Switch git lb to source balance method https://review.opendev.org/661783 | 15:18 |
| clarkb | there we go | 15:19 |
| *** rfarr has joined #openstack-infra | 15:19 | |
| openstackgerrit | Slawek Kaplonski proposed zuul/zuul-jobs master: Add role to fetch journal log from test node https://review.opendev.org/643733 | 15:22 |
| *** rfarr_ has joined #openstack-infra | 15:23 | |
| mtreinish | random doc8 question, is there a way to tell it to ignore line length for tables? | 15:24 |
| fungi | AJaeger: ^ you may know | 15:24 |
| clarkb | fungi: https://review.opendev.org/#/c/661783/ is the change to swap balance methods | 15:25 |
| fungi | yep, saw it, thanks! | 15:25 |
| *** rfarr has quit IRC | 15:26 | |
| openstackgerrit | Slawek Kaplonski proposed zuul/zuul-jobs master: Add role to fetch journal log from test node https://review.opendev.org/643733 | 15:26 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 15:36 |
| clarkb | meeting agenda sent. It includes a link to the open project-rename changes. infra-root if you can review those today and tomorrow that would be great. Particularly the one from mordred for opendev | 15:37 |
| *** ykarel|away has quit IRC | 15:38 | |
| *** lpetrut has quit IRC | 15:38 | |
| clarkb | https://review.opendev.org/#/q/topic:project-rename+status:open is the link | 15:38 |
| AJaeger | mtreinish: only for tables? Not that I know off | 15:38 |
| AJaeger | mtreinish: have a change for me to review? | 15:38 |
| clarkb | AJaeger: also ^ you are our project creator review maestro. Please let us know if you have any concerns too. I'll be reviewing them myself this morning I hope | 15:39 |
| fungi | i find it interesting that the http/https frontend connection rate doesn't really seem to fluctuate between weekend and weekday loads (current sessions goes up a bit as does session time) | 15:39 |
| AJaeger | clarkb: will go over them... | 15:40 |
| clarkb | fungi: I'm guessing a good chunk of it is bots | 15:41 |
| corvus | > 50% of the rename changes have zuul-1 | 15:42 |
| fungi | clarkb: yeah, i mean zuul caches aggressively but i expected third-party ci systems to show more prominently | 15:42 |
| corvus | = 50% if you limit to p-c | 15:42 |
| clarkb | corvus: I think that is due to updatings in zuul's main.yaml that won't take effect until after the changes merge? | 15:43 |
| clarkb | I want to say last time we did renames like this we force merged to get around that? | 15:43 |
| corvus | we could split the changes | 15:44 |
| clarkb | ya I think we discussed this a bunch ~6 months ago and decided it was easiset to just force merge then we don't have to do a dance of changes merging before and after gerrit downtime. fungi do you recall? | 15:45 |
| fungi | honestly i was trying and failing to remember | 15:45 |
| fungi | best i can hope for is we find a maintenance plan or transcript of the last rename before the opendev transition | 15:46 |
| corvus | i think the big question about the opendev change is whether zone-zuul-ci.org should be in zuul/ or opendev/ | 15:46 |
| AJaeger | I'm giving +2 if those are correct as endresult - if we want to split, I'll remove my votes... | 15:46 |
| mordred | corvus: ++ | 15:46 |
| corvus | i'm fine with not splitting | 15:47 |
| mordred | corvus: I can make arguments in both directions, so I believe I'm in support of either | 15:47 |
| fungi | corvus: my argument for opendev is that we want to have control over making sure it runs the jobs which test for broken configuration, snice that could knock the silent master offline if someone approved a broken bind zone config | 15:48 |
| corvus | yeah, i also don't have a strong opinion. | 15:48 |
| fungi | granted, it's a weak risk | 15:48 |
| clarkb | fungi: fwiw we have approved and merged broken zones recently (also quickly cleaned up) | 15:48 |
| fungi | yep! | 15:48 |
| corvus | fungi: yes -- though we can still do that with a config-project project stanza | 15:48 |
| *** hamzy has joined #openstack-infra | 15:49 | |
| mordred | I think I lean slightly towards opendev, because it's an operational repo that drives the configuration of services the opendev admins are responsible for | 15:49 |
| fungi | also ianw has added a test that bind can parse the zonefiles now | 15:50 |
| corvus | it's clear to me that it's a "shared jurisdiction" repo, so should always be a collaboration between opendev and zuul; so for me the question is more of where is its "home"? is it part of opendev shared with zuul, or part of zuul shared with opendev? | 15:50 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Add granular ACLs for Airship Projects https://review.opendev.org/660461 | 15:51 |
| fungi | another litmus test: are there likely to be records where a zuul maintainer is likely to know what should be in that record when an opendev sysadmin won't? how about the other way around? | 15:51 |
| mordred | I *think* I think it's part of opendev shared with zuul - because the primary data contained in it is information that stems from other operational things (like, the IP address for a given service is an IP address managed by opendev, and verifying that a given IP maps to a given server is a thing that opendev folks would be doing) | 15:52 |
| mordred | I think I may have said something similar to fungi except with more useless words | 15:52 |
| corvus | fungi: i suspect not, as long as zuul continues to be hosted on opendev. i think the point at which that changes would be if zuul were to pack up and leave opendev, at which point, i rather think the name of the repo *in opendev* doesn't matter anymore. :) | 15:52 |
| clarkb | mordred: fungi: that makes sense to me for why it should be in opendev/ | 15:53 |
| fungi | yeah, if zuul departs from opendev it will almost certanily take its dns hosting with it | 15:53 |
| corvus | sounds like we're all leaning toward opendev/ so mordred, want to respin that? | 15:53 |
| mordred | corvus: sure | 15:53 |
| fungi | if we decide it's safe, we can always add folks from related projects as core reviewers on a zone repo | 15:54 |
| corvus | yep | 15:54 |
| fungi | so that record changes can be made without opendev sysadmins holding that up | 15:54 |
| fungi | (say, if we're short on review bandwidth and there's an urgent change which needs to go through) | 15:55 |
| AJaeger | do we really want to retire opendev-website? Then let's do the usual retirement steps (remove content etc) as well | 15:56 |
| clarkb | re finding previous rename notes, the last set of projects we would've renamed included shade -> sdk stuff | 15:56 |
| clarkb | mordred: ^ do you recall what the specific renames were there or did we not end up doing them with shade/sdk beacuse of the code merge? | 15:57 |
| *** kopecmartin is now known as kopecmartin|off | 15:58 | |
| * clarkb is doing wiki page history forensics. Not as easy as with git | 15:58 | |
| fungi | manual bisection with the start/end radio buttons is what i usually end up doing | 15:59 |
| mordred | clarkb: I don't remember anything specific | 15:59 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Rename opendev projects we missed https://review.opendev.org/657277 | 15:59 |
| mordred | how's that look now? | 15:59 |
| fungi | clarkb: granted, it helps a lot if people get in the habit of leaving commentary metadata for any nontrivial wiki edit | 15:59 |
| AJaeger | mordred: please move the two retirement repos out of that change | 15:59 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 16:00 |
| AJaeger | mordred: those retirement is not done correctly and can be done independent | 16:00 |
| clarkb | fungi: ya thankfully it adds the section header as the metadata entry if you don't add something else | 16:00 |
| fungi | assuming people click the section edit link rather than the top-level page edit button | 16:00 |
| AJaeger | mordred: -1 for the retirement - that is indeed not done properly | 16:01 |
| *** Goneri has quit IRC | 16:01 | |
| clarkb | https://etherpad.openstack.org/p/project-renames-2018-08-03 found em | 16:02 |
| clarkb | or at least a set of renames | 16:02 |
| openstackgerrit | Sorin Sbarnea proposed opendev/git-review master: Corrected git-review project organization https://review.opendev.org/661800 | 16:02 |
| clarkb | and ya force merge changes seems to be on the list | 16:02 |
| clarkb | and we do that before we stop gerrit so that they can replicate | 16:03 |
| clarkb | these are good notes :) | 16:03 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Rename opendev projects we missed https://review.opendev.org/657277 | 16:03 |
| *** lucasagomes has quit IRC | 16:03 | |
| mordred | AJaeger: fixed - thanks! | 16:03 |
| AJaeger | clarkb: happy now with https://review.opendev.org/#/c/660461 ? could you review, please | 16:03 |
| clarkb | AJaeger: looking | 16:03 |
| AJaeger | mordred: thanks. Want to send the retirement change as well? | 16:04 |
| mordred | sure | 16:04 |
| clarkb | AJaeger: done | 16:04 |
| openstackgerrit | Sorin Sbarnea proposed opendev/git-review master: Assure python3 is preferred by tox https://review.opendev.org/661802 | 16:05 |
| AJaeger | mordred: commmented - needs a bit more work | 16:06 |
| clarkb | zbr: re ^ I thought tox defaulted to using the python version it was installed under for those envs that don't specify. Does putting python3 before python2 change that? | 16:06 |
| AJaeger | thanks, clarkb | 16:06 |
| *** tesseract has quit IRC | 16:06 | |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Retire pbrx and opendev-website https://review.opendev.org/661803 | 16:07 |
| zbr | clarkb: yep, tox preferrence is based on the order mentioned in envlist. confirmed that. it was suggested to me by the maintainers some time ago | 16:07 |
| mordred | AJaeger: ah - poop. thanks | 16:07 |
| fungi | zbr: as a substitute for setting defaultpython? | 16:08 |
| zbr | it is big ugly because this behavior is not self-describingf | 16:08 |
| AJaeger | mordred: we retire repos in several steps: 1) send email; 2) add noop-jobs 3) remoe all content; 4) your change to rmeove it from gerrit and zuul | 16:08 |
| clarkb | zbr: neat I did not know about that feature | 16:08 |
| zbr | fungi: yep. alternative to basepython, and it also has additional feature. | 16:08 |
| AJaeger | mordred: so, your change needs a bit more splitting... The infra-manual has full details | 16:09 |
| AJaeger | bbl | 16:09 |
| zbr | fungi: basepython=python3 may pick an unsupported python version on some platforms (like py35), but if you use preference list, you can tell tox to prefer "37,36,35,27" | 16:09 |
| *** ykarel|away has joined #openstack-infra | 16:09 | |
| fungi | zbr: and tox 3.1.0 is the version which added that behavior? | 16:09 |
| zbr | fungi: nope, this feature was added very long time ago (i do not remember), 3.1 added the ignore_basepython_conflict | 16:10 |
| *** ricolin_ has joined #openstack-infra | 16:10 | |
| fungi | ahh, okay | 16:10 |
| zbr | fungi: i adopted this approach recently as it gives a decent developer experience as I want to run linters using newest supported python version. | 16:12 |
| zbr | as we know flake8 on py27 fails to pick lots of tests which are found under py3* versions. i guess soon we will see more tools dropping old interpreters, so picking this approach is likely to be better. | 16:13 |
| fungi | zbr: so if i have a python3-only project, i don't actually need to set testenv.basepython=python3 as long as i have py3 (and no py27) in my tox.envlist? neat! | 16:13 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Rename opendev projects we missed https://review.opendev.org/657277 | 16:14 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Retire pbrx and opendev-website https://review.opendev.org/661803 | 16:14 |
| zbr | fungi: yep, try it. but remember to clean the envs. does decides that only for new envs. it will not recreate exiting ones. | 16:14 |
| fungi | sure, or pass -r | 16:15 |
| mordred | AJaeger: nod. I think I got the rename patch more correct now - I'll work on the other steps for the retirement | 16:15 |
| fungi | zbr: wow, this is a rather opaque magic, too bad it's not documented | 16:16 |
| fungi | but it's allowing me to clean up some personal projects, so... thanks for the pointer! | 16:16 |
| zbr | fungi: i agree with you, i am considering updating the docs to mention this. very useful feature which is very well hidden. | 16:16 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/project-config master: Drop neutron-functional job from devstack-gate https://review.opendev.org/661806 | 16:17 |
| frickler | AJaeger: slaweq: corvus: after thinking a bit longer, I think this is the correct solution ^^. we could still fix the job if it needs to be run against other repos | 16:17 |
| corvus | frickler: wfm | 16:17 |
| corvus | i think it was mostly there to test hooks, etc | 16:18 |
| fungi | and we still have a job which directly tests hooks anyway (with dummy hooks), if memory serves | 16:18 |
| *** gyee has joined #openstack-infra | 16:20 | |
| mwhahaha | did the yum-puppetlabs mirror sync change recently? it appears that we no longer have the el/ folder | 16:23 |
| * mwhahaha tries to remember where that is configured | 16:25 | |
| openstackgerrit | Merged openstack/project-config master: Add granular ACLs for Airship Projects https://review.opendev.org/660461 | 16:26 |
| *** emine__ has quit IRC | 16:27 | |
| fungi | mwhahaha: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/mirror/yum-puppetlabs-mirror-update.sh | 16:27 |
| *** prometheanfire has quit IRC | 16:28 | |
| fungi | mwhahaha: last update was https://review.opendev.org/656912 which merged 2019-05-09 (several weeks ago) | 16:28 |
| mwhahaha | yea there seems to be a desync between the rsync version and the published version | 16:29 |
| mwhahaha | http://mirror.ord.rax.openstack.org/yum-puppetlabs/ vs http://yum.puppetlabs.com/ | 16:29 |
| *** prometheanfire has joined #openstack-infra | 16:29 | |
| mwhahaha | looks like the rsync split off the older stuff to http://release-archives.puppet.com/yum | 16:29 |
| fungi | i wonder if their breaking of puppet 4 deb repos has halted our mirroring of the rest | 16:29 |
| mwhahaha | which is the bits that have broken the stable/queens puppet jobs | 16:30 |
| fungi | checking logs on the mirror-update server | 16:30 |
| mwhahaha | cause we need puppet 4 :/ | 16:30 |
| AJaeger | config-core, pleaes review another job removal: https://review.opendev.org/661542 | 16:30 |
| fungi | mwhahaha: have you been following https://tickets.puppetlabs.com/browse/CPR-680 ? | 16:31 |
| fungi | and subsequent https://tickets.puppetlabs.com/browse/CPR-685 | 16:32 |
| mwhahaha | no hadn't see them | 16:32 |
| mordred | mwhahaha: it looks like they deleted puppet4 for rpm just like they deleted puppet4 for deb - and I don't see puppet4 in release-archives | 16:33 |
| *** ramishra has quit IRC | 16:33 | |
| mwhahaha | they are there | 16:33 |
| mordred | they are? | 16:33 |
| mwhahaha | they were pc1 | 16:33 |
| mwhahaha | so the pc1 stuff was ~puppet 4 | 16:33 |
| fungi | looks like they put it all back temporarily a week ago yesterdau | 16:33 |
| mwhahaha | https://release-archives.puppet.com/apt/ | 16:33 |
| mordred | wow. so puppetlabs-release-pc1-el-5.noarch.rpm is actually puppet4? | 16:33 |
| mwhahaha | yea | 16:34 |
| mwhahaha | https://review.opendev.org/#/c/661787/1/functions | 16:34 |
| mordred | wow | 16:34 |
| mwhahaha | that's teh logic we had for 4 vs 5 | 16:34 |
| mwhahaha | PC1 = 4 | 16:34 |
| mwhahaha | the problem we currently have is that the infra mirrors don't have the older stuff anymore cause it looks like they dropped it from the rsync mirror | 16:35 |
| mordred | yah. | 16:35 |
| fungi | last update before the weekend seems to be that they'll remove it again soon... "We will likely remove the release packages from the subdirectories before the archives are ready. I'll send an update next week when we have a better sense of the timeline." (from the google groups thread linked in the second cpr) | 16:35 |
| mwhahaha | i guess we could just switch the puppet openstack stuff to use upstream mirrors for now, we don't have that many jobs anymore | 16:35 |
| mwhahaha | or patches at least for teh puppet 4 bits | 16:35 |
| mwhahaha | the 5/6 seem to be ok still | 16:35 |
| * mwhahaha will patch later | 16:36 | |
| fungi | mwhahaha: aha, i see the problem. out of quota on that afs volume | 16:36 |
| fungi | rsync: close failed on "/afs/.openstack.org/mirror/yum-puppetlabs/sles/11/PC1/x86_64/.puppet-agent-1.10.12-1.sles11.x86_64.rpm.EDZHao": Disk quota exceeded (122) | 16:37 |
| mwhahaha | oh well that'd do it too :D | 16:37 |
| *** ginopc has quit IRC | 16:38 | |
| fungi | mwhahaha: first occurrence of that quota exceeded error was 2019-04-26 00:20:01 utc, for the record | 16:39 |
| fungi | so i'm guessing it's just over a month stale now | 16:39 |
| clarkb | its just going to break again when they undo the undo right? | 16:39 |
| clarkb | I mean puppet-4 specifically | 16:39 |
| openstackgerrit | Merged openstack/project-config master: Drop neutron-functional job from devstack-gate https://review.opendev.org/661806 | 16:39 |
| clarkb | so you probably want to be addressing that as well | 16:40 |
| fungi | yeah, unless what they said in that google groups thread is a typo, they're removing it again *before* they get similar package repos added on the new archive site | 16:40 |
| clarkb | fwiw we install the puppet-agent pc1 package directly and because they vendor everything they need into /opt/puppetlabs with that package it just works for us | 16:40 |
| zbr | fungi: it seems that my assumption about tox and preference list was wrong, it does not work. it seems that I was only lucky to get the desired behavior by chance. i will see if I can fix tox to do it. | 16:40 |
| fungi | zbr: interesting, because it seems to be happening for me in local testing | 16:41 |
| zbr | lets see what we get on https://github.com/tox-dev/tox/pull/1329 --- | 16:41 |
| zbr | one way or another we should find a solution, maybe we will endup with a new config option, one that is more self-describing. | 16:42 |
| fungi | zbr: oh, actually i think it's working on my local py3-only projects because i'm running a tox which is installed under python3 | 16:43 |
| zbr | fungi: yeah. same happening with me. | 16:44 |
| fungi | also after some correction of my test case, i see that it's using the python3.7 my tox was installed with even though py38 is first in my list | 16:44 |
| clarkb | fungi: ya that was the behavior I knew about | 16:45 |
| clarkb | balance method change is in the gate now | 16:47 |
| mtreinish | AJaeger: sry missed your response earlier, it's on: ff | 16:47 |
| mtreinish | AJaeger: https://github.com/Qiskit/qiskit/pull/307 (wrong paste buffer don't know where ff came from) | 16:48 |
| *** dpawlik has joined #openstack-infra | 16:48 | |
| mtreinish | the travis failure is doc8 complaining about line length on the table: https://travis-ci.com/Qiskit/qiskit/jobs/203592013#L602 trying to figure out how to proceed there, because I don't think you can line wrap that | 16:49 |
| *** kjackal has quit IRC | 16:49 | |
| *** armax has quit IRC | 16:50 | |
| fungi | mtreinish: in that particular example, the first column heading could likely be wrapped onto a second line | 16:50 |
| * clarkb finds breakfast | 16:51 | |
| * fungi assumes that's valid rst anyway | 16:52 | |
| mtreinish | fungi: heh perhaps, I didn't try that. I'll give it a go. But that'll make it kinda hard to read though | 16:53 |
| mtreinish | also how do I format the width for the columns if they overlap | 16:54 |
| mtreinish | oh, nm I misread what you said | 16:54 |
| fungi | i meant wrap the text within the cell | 16:54 |
| mtreinish | colmn not rwo | 16:54 |
| mtreinish | that is valid rst | 16:54 |
| fungi | or find a way to effectively abbreviate that first column heading, though i'm sure you've already made it as brief as possible | 16:55 |
| mtreinish | yeah I could change metapackage to meta, but wrapping that probably would be the easiest | 16:55 |
| fungi | but yeah, it's the classic "my headings are much longer than the data below them" problem | 16:56 |
| mtreinish | fungi: :/ line wrapping the column header is still 14 characters too long... | 16:59 |
| *** mattw4 has joined #openstack-infra | 17:00 | |
| *** derekh has quit IRC | 17:00 | |
| *** ricolin_ has quit IRC | 17:00 | |
| zbr | fungi: clarkb please check https://github.com/tox-dev/tox/issues/1330 and comment on it, it may be important to state the importance of this feature to tox maintainers, before they close the feature request due to "deterministic" excuse. | 17:04 |
| *** Goneri has joined #openstack-infra | 17:04 | |
| openstackgerrit | Merged opendev/system-config master: Switch git lb to source balance method https://review.opendev.org/661783 | 17:06 |
| clarkb | and now we watch and wait | 17:07 |
| fungi | so... the /afs/.openstack.org/mirror/yum-puppetlabs volume has a 100gb quota currently and is using all of it. i wonder if we're mirroring far too much there | 17:08 |
| clarkb | zbr: I'm probably the wrong person to comment on that as my platforms all have a single python2 and single python3 version | 17:08 |
| clarkb | so I'm happy with setting basepython to python3 | 17:08 |
| clarkb | fungi: or maybe we aren't deleting the stuff they deleted? | 17:08 |
| clarkb | so have duplicate type datA? | 17:08 |
| fungi | we're passing --delete | 17:11 |
| clarkb | re haproxy update we last ran ansible there at about 16:55UTC so I expect in about 20-30 minutes we'll get our balance method updated | 17:12 |
| *** Weifan has joined #openstack-infra | 17:12 | |
| clarkb | zbr: looks like they have some good ideas so we'll probably get something that works | 17:14 |
| clarkb | (re your github tox bug) | 17:14 |
| zbr | clarkb: if you happen to run centos 7 you may be not so happy because on it default python3 is py34 which was dropped by many python projects. | 17:14 |
| AJaeger | mtreinish: use a list-table, see https://docs.openstack.org/doc-contrib-guide/rst-conv/tables.html#list-tables | 17:16 |
| zbr | clarkb: glad to read the last comment | 17:17 |
| *** irclogbot_3 has quit IRC | 17:17 | |
| *** RobertNordix has quit IRC | 17:17 | |
| *** irclogbot_3 has joined #openstack-infra | 17:18 | |
| fungi | according to du the breakdown of top-level directories in the yum-puppetlabs mirror is 36gb for el, 27gb for puppet5, 16gb for puppet6, 15gb for fedora, 2gb for sles and 2gb for puppet-enterprise-tools | 17:18 |
| mtreinish | AJaeger: ok cool, thanks. That should do the trick | 17:19 |
| mtreinish | I also just switched from a grid table to a simple table and it fit in the character limit too | 17:20 |
| mtreinish | ah, but it didn't format the line wrap for the first column header properly. So list table it is | 17:21 |
| fungi | we could probably stand to exclude a lot of stuff in this yum-puppetlabs mirror... el/5 and el/6 together use 26gb of space | 17:24 |
| fungi | if you add puppet5/el/5 and puppet5/el/6 that's another 13gb | 17:25 |
| *** sreejithp has joined #openstack-infra | 17:25 | |
| *** armax has joined #openstack-infra | 17:25 | |
| *** jpena is now known as jpena|off | 17:26 | |
| clarkb | fungi: we (infra) never used those mirrors so no worries removing stuff from our side | 17:27 |
| clarkb | would come down to puppet-openstack and tripleos needs I think | 17:27 |
| fungi | i doubt they're running centos5 or centos6 jobs | 17:28 |
| rpioso | hodgepodge: When you have a chance, please review https://review.opendev.org/#/c/660534/ | 17:29 |
| clarkb | fungi: good point | 17:30 |
| fungi | puppet6/el/5 and puppet6/el/6 are smaller, but also pointless | 17:31 |
| fungi | 5gb together | 17:31 |
| fungi | also fedora/f20 through... fedora/27? | 17:32 |
| rpioso | hodgepodge: Please be aware that dtantsur, fungi, TheJulia, and I had a discussion about it in #openstack-ironic this past Friday. It starts at http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-ironic.2019-05-24.log.html#t2019-05-24T14:23:28 | 17:32 |
| clarkb | corvus: fungi mordred I think we may have to manually restart haproxy? I don't think the service was updated after the template was updated (though I'm trying to find where the file is on disk to confirm it actually updated) | 17:33 |
| rpioso | ^ dtantsur|afk | 17:33 |
| fungi | clarkb: oh, probably... though i'm about to pop out to get some groceries so will probably disappear for 30-45 minutes | 17:34 |
| *** igordc has quit IRC | 17:34 | |
| *** ykarel|away has quit IRC | 17:34 | |
| *** ykarel|away has joined #openstack-infra | 17:35 | |
| clarkb | yup the file is updated | 17:35 |
| clarkb | now to see if admin socket will tell me what balance method it is using | 17:35 |
| fungi | okay, bad timing but have to coordinate with my wife's schedule for shopping so stepping out for the next few minutes but should be back well before the meeting | 17:43 |
| clarkb | the way you gracefully restart haproxy is to sigusr1 it which tells the running haproxy to keep running until existing connections stop. You are expected to already have a new haproxy running | 17:47 |
| clarkb | so I think our best bet here is to docker-compose restart and take a small service restart hit | 17:47 |
| *** electrofelix has quit IRC | 17:49 | |
| clarkb | I may as well pop out for a bit too if others are afk. Back in a bit well before the meeting | 17:49 |
| *** igordc has joined #openstack-infra | 17:55 | |
| *** igordc has quit IRC | 17:55 | |
| *** igordc has joined #openstack-infra | 17:55 | |
| *** priteau has quit IRC | 17:56 | |
| *** Weifan has quit IRC | 17:57 | |
| *** Weifan has joined #openstack-infra | 17:57 | |
| corvus | clarkb: was in meeting; maybe after meeting would be best | 17:59 |
| *** Weifan has quit IRC | 18:01 | |
| *** ykarel|away has quit IRC | 18:01 | |
| clarkb | k | 18:06 |
| *** efried has quit IRC | 18:17 | |
| *** efried has joined #openstack-infra | 18:18 | |
| *** rfarr_ has quit IRC | 18:27 | |
| *** jesusaur has quit IRC | 18:30 | |
| * rpioso realizes he incorrectly entered hogepodge's nic | 18:30 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add script to pre-populate private keys https://review.opendev.org/661824 | 18:31 |
| rpioso | hogepodge: When you have a chance, please review https://review.opendev.org/#/c/660534/ | 18:32 |
| *** imacdonn has quit IRC | 18:33 | |
| rpioso | hogepodge: Please be aware that dtantsur|afk, fungi, TheJulia, and I had a discussion about it in #openstack-ironic this past Friday. It starts at http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-ironic.2019-05-24.log.html#t2019-05-24T14:23:28 | 18:33 |
| *** rlandy is now known as rlandy|brb | 18:33 | |
| clarkb | rpioso: I think hogepodge is travelling so it may be a day or two | 18:33 |
| rpioso | clarkb: Thank you! | 18:34 |
| fungi | rpioso: out of curiosity, why are you thinking he'll have an interest in that? | 18:34 |
| fungi | i probably missed an ml thread somewhere with the tie-in | 18:35 |
| rpioso | fungi: I believe he's involved with the OpenStack Marketplace. | 18:35 |
| rpioso | At least, so I've been told. | 18:35 |
| fungi | ahh, yeah he may have been helping on some of the marketplace ideas | 18:36 |
| rpioso | :-) | 18:36 |
| fungi | JpMaxMan, jamesmacarthur, smarcet et al are the ones managing the implementation for it though | 18:37 |
| fungi | and pretty sure they're all aware of the earlier plan thingee was driving to standardize driver matrices in individual service projects | 18:37 |
| fungi | but we should definitely loop them back in once work resumes on that | 18:38 |
| *** rfarr has joined #openstack-infra | 18:39 | |
| rpioso | fungi: Thank you for that info. I wasn't previously aware of their involvement. | 18:39 |
| *** hwoarang has quit IRC | 18:41 | |
| *** hwoarang has joined #openstack-infra | 18:41 | |
| rpioso | fungi: Do you have any insight when it might resume? | 18:41 |
| *** Weifan has joined #openstack-infra | 18:42 | |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Rename request: airship-in-a-bottle https://review.opendev.org/656008 | 18:44 |
| AJaeger | fungi, could I trouble you with reviewing a small cleanup, please? https://review.opendev.org/#/c/661542/ | 18:45 |
| AJaeger | team, do we want to switch rax.dfw mirror to opendev? See https://review.opendev.org/661187 by ianw | 18:45 |
| fungi | rpioso: i think it needs a critical mass of people interested in collaborating on a standard, with input from maintainers of multiple openstack services | 18:46 |
| fungi | this could be a target for an openstack cycle goal, maybe in the "u" (unnamed) cycle | 18:46 |
| rpioso | fungi: When the time comes, I'm interested in contributing some not so critical mass ;-) | 18:47 |
| *** Weifan has quit IRC | 18:47 | |
| *** lpetrut has joined #openstack-infra | 18:47 | |
| dirk | fungi: thanks for the notification about the mirror failure. For how long is it failing already? I was assuming those ~tmp~ dirs are temporary | 18:48 |
| fungi | dirk: i'm not sure how long it was failing for, evrardjp may have a better idea, or i can try to dig it out of the logs when i get a moment | 18:49 |
| clarkb | pasting yaml into etherpad is not a fun time | 18:50 |
| fungi | rpioso: cool, getting some voices from the openstack tc behind such an effort could help. maybe could be organized as a "pop-up team" https://review.opendev.org/661356 | 18:50 |
| dirk | fungi: also note that we're trying to get rid of mirroring alltogether via https://review.opendev.org/#/c/658855/ | 18:50 |
| evrardjp | dirk: almost 2 days, as far as I can see. Go back to vacationing though ;) | 18:50 |
| dirk | Unfortunately reviews on that one are missing | 18:51 |
| *** kjackal has joined #openstack-infra | 18:51 | |
| *** lpetrut has quit IRC | 18:51 | |
| evrardjp | dirk: that would be good too. I needed to go forward to unblock some gates though. | 18:51 |
| *** raissa has joined #openstack-infra | 18:53 | |
| *** rlandy|brb is now known as rlandy | 18:53 | |
| dirk | evrardjp: I am still vacationing.. when we get the lxc 42.3 jobs removed we could stop mirroring that repo | 18:54 |
| dirk | I wonder if the ~tmp~ means that somebody deleted the repository | 18:54 |
| dirk | I'll check | 18:54 |
| clarkb | As a friendly reminder we will have our weekly infra team meeting in a few minutes over in #openstack-meeting | 18:57 |
| openstackgerrit | Merged zuul/zuul-jobs master: validate-zone-db : add job and make more generic https://review.opendev.org/661138 | 18:58 |
| *** raissa has quit IRC | 18:58 | |
| rpioso | fungi: A pop-up team sounds like a reasonable structure for that effort. | 19:00 |
| fungi | basically the pop-up team could drive an initial pilot in a few projects and then could dissolve once a cycle goal to repeat that pattern in all openstack projects gets approved | 19:01 |
| fungi | (and some folks from the pop-up team might become goal champions on that effort) | 19:02 |
| *** eernst has joined #openstack-infra | 19:13 | |
| *** eernst has quit IRC | 19:25 | |
| *** takamatsu has joined #openstack-infra | 19:28 | |
| *** pcaruana has quit IRC | 19:30 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around build completion and cancellation https://review.opendev.org/660806 | 19:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around build states https://review.opendev.org/661489 | 19:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around reporting https://review.opendev.org/661490 | 19:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around finished builds https://review.opendev.org/661491 | 19:32 |
| *** Weifan has joined #openstack-infra | 19:34 | |
| *** Weifan has quit IRC | 19:39 | |
| openstackgerrit | Ankita Bansal proposed opendev/storyboard-webclient master: Provide -Assign to self- button for story tasks. https://review.opendev.org/661834 | 19:40 |
| *** Weifan has joined #openstack-infra | 19:41 | |
| *** kjackal has quit IRC | 19:41 | |
| *** kjackal has joined #openstack-infra | 19:41 | |
| rm_work | it looks like we need to have octavia's pyX jobs run with master of octavia-lib (with tox_install_siblings on)... is there an example for setting that up with the base python jobs somewhere? | 19:42 |
| fungi | rm_work: infra meeting is underway right now, but i believe the oslo team runs a bunch of source jobs like that | 19:44 |
| rm_work | wondering if i have to remove python-openstack-jobs from our project, and make a whole new set of those jobs with the originals as their parent | 19:46 |
| rm_work | to add the tox_install_siblings thing | 19:46 |
| rm_work | i guess it isn't urgent, i'll keep working on the other bits of my patch that i need to fix first :D | 19:47 |
| rm_work | and see if i can figure this out later | 19:47 |
| fungi | yeah, may want to look into how neutron tests with neutron-lib too? | 19:50 |
| rm_work | yeah, so | 19:52 |
| rm_work | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L758 seems like it | 19:52 |
| rm_work | which does seem like they redo all the jobs | 19:52 |
| rm_work | which seems really tedious | 19:52 |
| rm_work | but i suppose that's fine | 19:52 |
| fungi | they don't redo the jobs, they just parent the normal jobs and override a few parameters | 19:53 |
| rm_work | but i really just wish there was a way to inject a couple of things into the existing runs without that hassle | 19:53 |
| fungi | note those jobs declare no playbooks at all | 19:53 |
| rm_work | right i mean | 19:53 |
| rm_work | for each job, they have to make an entry | 19:53 |
| rm_work | and parent the original + add stuff | 19:53 |
| *** ralonsoh has quit IRC | 19:53 | |
| rm_work | it's essentially 1:1 for all the jobs | 19:54 |
| fungi | how many do you need it in? | 19:54 |
| rm_work | every job that runs tox | 19:54 |
| fungi | wow, even things like static code analyzers? | 19:54 |
| fungi | docs builds? | 19:55 |
| rm_work | well, at least openstack-python-jobs, openstack-python36-jobs, openstack-python37-jobs, openstack-cover-jobs | 19:55 |
| rm_work | MAYBE some doc builds | 19:55 |
| rm_work | probably not like, release-notes :D | 19:55 |
| mordred | there was a discussion a while back about making it possible to add a required-projects list to a project pipeline defintion - but I don't believe anyone has even gotten as far as thinking about design, much less work on it | 19:56 |
| *** imacdonn has joined #openstack-infra | 19:56 | |
| rm_work | not pep8 i guess either | 19:57 |
| pabelanger | fyi, fedora-28 is EOL today: https://fedoraproject.org/wiki/End_of_life | 19:58 |
| fungi | rm_work: also, what is the releasability there... do octavia and octavia-lib get released at the same time, or is the lib released earlier? any concerns about testing that octavia works with the released version of octavia-lib at that point, or does its development get frozen until octavia releases? | 19:59 |
| rm_work | lib should be released earlier as it follows lib release rules | 19:59 |
| fungi | and then you avoid merging anything into the lib so you don't accidentally grow a dependence in octavia on some new feature in octavia-lib which won't be released when octavia is? | 20:00 |
| clarkb | pabelanger: ya we're likely to keep it around until centos 8 exists | 20:00 |
| clarkb | pabelanger: it is currently tripleo's python3 test platform | 20:00 |
| rm_work | isn't that standard library release policy? | 20:01 |
| rm_work | freeze for libs is before freeze for projects | 20:01 |
| fungi | rm_work: the alternative is to release the lib much more often and then only test against the released version so you don't have to be diligent about not depending on things in the lib which won't be released when the service releases | 20:01 |
| clarkb | infra-root ok are we ready to docker-compose restart on opendev.org ? Any better more graceful ideas on how we could do that? | 20:01 |
| *** takamatsu has quit IRC | 20:02 | |
| rm_work | fungi: yeah but what we NEED to be able to do is Depends-On a patch in octavia-lib and have the tests pass | 20:02 |
| rm_work | maybe I skipped something and this is a case of "back up to what I really need, not what I think I need" | 20:02 |
| fungi | if the change in octavia is backwards-compatible with released octavia-lib then running both jobs should be doable | 20:02 |
| clarkb | I can help drive the haproxy restart if so. But need to find lunch after | 20:02 |
| corvus | clarkb: wfm | 20:02 |
| clarkb | ok I'm going to run `docker-compose restart` in /etc/haproxy-docker | 20:03 |
| fungi | clarkb: sounds good, thanks! | 20:03 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add zone-check job https://review.opendev.org/660888 | 20:03 |
| openstackgerrit | Merged openstack-infra/zone-zuul-ci.org master: Add zone-check job https://review.opendev.org/660967 | 20:03 |
| clarkb | done and opendev.org still loads for me | 20:04 |
| rm_work | fungi: well, if i have a patch that is dependent on octavia-lib change, but that change wouldn't be released without the thing that depends on it (it's a feature, no point in adding one without the other), then how does that work? people aren't going to merge and release a lib change that doesn't do anything without a feature that's still not +A, and people don't usually +A patches that are failing all their tests | 20:04 |
| clarkb | doing some quick tests of puppet-keystone clones | 20:04 |
| openstackgerrit | Ankita Bansal proposed opendev/storyboard-webclient master: project_group view: add number of active stories beside repo list https://review.opendev.org/661836 | 20:04 |
| fungi | rm_work: "no point" may be a stretch... it does mean that people don't have to upgrade the service and the lib in unison if they're decoupled, but that's up to you | 20:05 |
| rm_work | fungi: for example this patch: https://review.opendev.org/#/c/660239/ | 20:05 |
| rm_work | Depends-On a change in lib that enables it | 20:05 |
| openstackgerrit | Merged openstack/project-config master: Switch rax.dfw mirror to opendev.org https://review.opendev.org/661187 | 20:05 |
| clarkb | git clone --mirror https://opendev.org/openstack/puppet-keystone then git --git-dir ./puppet-keystone.git fetch origin --prune works for me now | 20:06 |
| rm_work | I guess we could just let the tests continue to fail and wait until people are "+2 pending lib merge", and then merge/release the lib change, and recheck | 20:06 |
| clarkb | now to find lunch that start tackling the todo list post meeting. Back in a bit | 20:06 |
| rm_work | just seems dirty when we should be able to have the normal Depends-On logic do its thing there | 20:06 |
| corvus | clarkb: i can see the discontinuity in the grafana graphs; i don't see any problems. | 20:06 |
| corvus | clarkb: though i can already see the imbalance in sesions | 20:06 |
| rm_work | maybe it's possible to have this work in such a way that it tests against the normal release UNLESS depends-on is set? | 20:06 |
| clarkb | corvus: ya I guess that is only really a problem if a single node can't handle what is hashed to it :/ | 20:07 |
| clarkb | we'll have to watch it over time I expect | 20:07 |
| corvus | gitea08=15 gitea01=1.5 | 20:07 |
| *** mriedem has quit IRC | 20:07 | |
| fungi | rm_work: one scheme that works is to run both jobs (with normal released deps and with the lib from source) and then expect the normal version of the job to fail until the lib is released but the source version will succeed in the meantime to demonstrate the feature is viable (and can take advantage of depends-on before the lib implementation merges) | 20:14 |
| rm_work | hmm I guess I could do that and just run like, ONE | 20:14 |
| rm_work | maybe the py37 one | 20:14 |
| rm_work | that seems reasonable | 20:15 |
| fungi | but again, whether that's worth it to you probably depends on how tightly or loosely coupled octavia and octavia-lib are | 20:15 |
| rm_work | very | 20:16 |
| rm_work | i mean, we basically just moved a ton of code directly from octavia to octavia-lib | 20:16 |
| rm_work | and then added a dep | 20:16 |
| rm_work | and again, per my understanding of the way the library-release-policy works, we are REQUIRED to freeze the lib first, release it first, and unfreeze it last | 20:17 |
| fungi | and if octavia-lib is going to intentionally undergo a feature freeze after milestone 3 until octavia releases, then you can probably get away with the source job only | 20:17 |
| fungi | you may want to check with the release team on whether not running the normal unit test jobs is typical for a project in your situation | 20:18 |
| *** hamzy has quit IRC | 20:21 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 20:22 |
| fungi | rm_work: looking at a random recent neutron change, https://review.opendev.org/660785 shows that they're testing openstack-tox-py36 and not openstack-tox-py36-with-neutron-lib-master | 20:22 |
| rm_work | yeah it seems they only use the source versions as periodics | 20:23 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 20:24 |
| fungi | rm_work: agreed... http://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py36-with-neutron-lib-master | 20:24 |
| fungi | so maybe they defer merging neutron changes which depend on a neutron-lib feature until lib freeze | 20:25 |
| fungi | but again, they or the release team can likely tell you what's typical for this case | 20:26 |
| fungi | it's also possible to add an extra job in a do-not-merge child change to demonstrate the change working with the alternative dependencies, and can use depends-on when doing so | 20:27 |
| fungi | speculative execution of job configuration opens a lot of options | 20:27 |
| fungi | (for most jobs at least, you don't actually have to merge the job to run it) | 20:28 |
| fungi | i see people take advantage of that with some frequency | 20:29 |
| *** sreejithp_ has joined #openstack-infra | 20:30 | |
| *** dpawlik has quit IRC | 20:31 | |
| *** sreejithp has quit IRC | 20:31 | |
| *** hwoarang has quit IRC | 20:37 | |
| *** hwoarang has joined #openstack-infra | 20:38 | |
| rm_work | hmmm | 20:38 |
| rm_work | yeah... I just... feel like "Depends-On" should *work* | 20:38 |
| rm_work | because it's kinda lame to have to always do that | 20:38 |
| clarkb | infra-root https://etherpad.openstack.org/p/EkjfrnSIud is my draft gerrit project renaming announcement | 20:39 |
| clarkb | if I can get a couple eyeballs that aren't my own to look at that and ACK it I'll send it on its way | 20:40 |
| fungi | rm_work: depends-on should work if you're always installing the dependency from source, yes. what you don't want is to install the dependency from released artifacts most of the time and then install from source when there's a depends-on, because once that change with the depends-on merges all your other changes start failing the job | 20:41 |
| fungi | (at least until the feature you were depending on gets released) | 20:42 |
| *** takamatsu has joined #openstack-infra | 20:42 | |
| *** ekultails has quit IRC | 20:42 | |
| fungi | clarkb: lgtm, thanks! | 20:43 |
| corvus | clarkb: lgtm. also, maybe we should set up an announce list. | 20:44 |
| clarkb | corvus: probably not a bad idea | 20:44 |
| *** xek_ has quit IRC | 20:44 | |
| clarkb | I can get a change up for one though I won't wait on that to send this email (as it will take people time to get subscribed) | 20:45 |
| *** gfidente is now known as gfidente|afk | 20:45 | |
| corvus | ya | 20:47 |
| fungi | agreed | 20:48 |
| fungi | opendev-announce@lists.opendev.org? | 20:49 |
| fungi | service-announcements@ | 20:49 |
| *** slaweq has quit IRC | 20:49 | |
| fungi | oh the bikeshedding we will have | 20:49 |
| *** mriedem has joined #openstack-infra | 20:51 | |
| mordred | fungi: blue@lists.opendev.org | 20:51 |
| fungi | i prefer cerulean@ | 20:51 |
| mordred | fungi: cobalt@ is also nice | 20:52 |
| clarkb | emails sent | 20:52 |
| fungi | pantone-15-4020@ | 20:52 |
| corvus | hey, no proprietary colors | 20:53 |
| fungi | d'oh! | 20:53 |
| mordred | if we call it 9bb7d4@ it's no longer proprietary right? | 20:54 |
| openstackgerrit | Claudiu Belu proposed openstack/project-config master: Rename x/compute-hyperv to openstack/compute-hyperv https://review.opendev.org/661682 | 20:55 |
| clarkb | responded to tripleo/rdo git clone thread asking for some retries | 20:55 |
| paladox | heh cobalt is what we call our gerrit server :) | 20:55 |
| fungi | a company named cobalt used to make servers. i had a few | 20:55 |
| fungi | some were mips-based too | 20:56 |
| fungi | 1 rack unit mips server, pretty handy when you need that particular architecture | 20:56 |
| paladox | heh | 20:57 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add opendev service-announce list https://review.opendev.org/661843 | 20:57 |
| clarkb | corvus: fungi ^ you can start bikeshedding there :) | 20:57 |
| fungi | one of their nifty features was an integrated lcd display on the front which their bios knew how to drive so you could get the full server post through it by scrolling, and also adjust bios settings | 20:57 |
| clarkb | oh sorry it was mordred and fungi bikeshedding :) | 20:58 |
| *** e0ne has joined #openstack-infra | 20:58 | |
| clarkb | corvus was just making sure we didn't use proprietary bikeshed colors | 20:58 |
| fungi | a reasonable concern | 20:58 |
| fungi | also, i figured out why wiki-dev isn't puppeting... it's not in the inventory... its replacement is wiki-dev02 and that *is* puppeting. on to its newest puppet errors now | 20:58 |
| mordred | fungi: that's exciting! | 20:58 |
| fungi | if you call "Package php-apc is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source. E: Package 'php-apc' has no installation candidate" exciting anyway | 21:01 |
| rm_work | fungi: yeah that's why i figured we'd just want to switch it all to from-source, so depends-on would work and it'd be consistent | 21:01 |
| clarkb | infra-root remote: https://review.opendev.org/661845 Record historical project renames | 21:01 |
| *** Goneri has quit IRC | 21:01 | |
| clarkb | I ended up putting it in opendev/project-config not openstack/project-config even though oepnstack/project-config has the projects.yaml file currently | 21:02 |
| clarkb | if we think that will creation unnecessary friction getting that data into its final resting place I can put it in the other repo instead | 21:02 |
| fungi | one less thing to move, i suppose | 21:02 |
| fungi | but it does mean changes in two places for a rename in the interim | 21:02 |
| fungi | s/places/repos/ | 21:02 |
| clarkb | ya though I think it is always going to be a separate change while we record just the repos.yaml input to the rename playbook and not add it straight ot projects.yaml | 21:03 |
| clarkb | I'll get a similar change up once I stack the rename changes. I wanted to make sure that where aren't any other updates we need to those before stacking them though. Looks like we got a new patchset on at least one of them that needed an update (I don't remember if there was more than that one) | 21:05 |
| clarkb | maybe I'll give y'all today to review those then stack them tomorrow | 21:05 |
| clarkb | gitea08 seems to be the busiest of the backends over the last hour | 21:06 |
| clarkb | and has the longest average session times | 21:06 |
| clarkb | cross checking against cacti cpu, load, and memory seem low to reaosnable | 21:07 |
| clarkb | so I don't think it is sad | 21:07 |
| fungi | okay, looks like in addition to php-apc i also have 8 different international font ttf-.* packages which are either renamed or dropped in xenial... working on that patch now | 21:08 |
| *** dave-mccowan has joined #openstack-infra | 21:08 | |
| *** slaweq has joined #openstack-infra | 21:11 | |
| *** e0ne has quit IRC | 21:12 | |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-mediawiki master: More transitional Xenial package name replacements https://review.opendev.org/661848 | 21:20 |
| fungi | there we go... that's the next step for wiki-dev xenialing | 21:20 |
| * clarkb pulling up all these changes then will work on reviewing them | 21:20 | |
| fungi | findnig what new names packages have transitioned to is decidedly harder now that the last release containing transitional dummy packages for the old names has gone eol and packages.ubuntu.com no longer searches it | 21:21 |
| *** dave-mccowan has quit IRC | 21:23 | |
| *** slaweq has quit IRC | 21:24 | |
| clarkb | mordred: did you want to self approve https://review.opendev.org/#/c/640027/ and watch it? | 21:25 |
| clarkb | then we can get https://review.opendev.org/#/c/640044/ in after | 21:25 |
| clarkb | (I think corvus didn't approve because wanted someone else to watch it go in?) | 21:25 |
| clarkb | I'm about to do a proper review of tobiash's node usage statsd change in zuul so not quite able to watch it myself | 21:26 |
| fungi | manually trying out a potential set of excludes on the yum-puppetlabs-mirror-update script now to see if they work the way i expect, and then change on the way for that | 21:27 |
| mordred | clarkb: yes - although I'm stepping afk for a few minutes - I'll +A it when I get back so I can watch | 21:29 |
| *** udesale has quit IRC | 21:31 | |
| *** smarcet has joined #openstack-infra | 21:36 | |
| *** tosky has joined #openstack-infra | 21:36 | |
| *** smarcet has quit IRC | 21:36 | |
| *** smarcet has joined #openstack-infra | 21:38 | |
| *** harlowja has joined #openstack-infra | 21:44 | |
| *** kjackal has quit IRC | 21:45 | |
| *** yamamoto has joined #openstack-infra | 21:45 | |
| *** yamamoto has quit IRC | 21:50 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 21:58 |
| *** hwoarang has quit IRC | 21:58 | |
| *** hwoarang has joined #openstack-infra | 21:59 | |
| *** raissa has joined #openstack-infra | 22:02 | |
| *** raissa has quit IRC | 22:02 | |
| *** raissa has joined #openstack-infra | 22:03 | |
| *** raissa has quit IRC | 22:03 | |
| *** raissa has joined #openstack-infra | 22:03 | |
| *** raissa has quit IRC | 22:04 | |
| *** raissa has joined #openstack-infra | 22:04 | |
| *** raissa has quit IRC | 22:04 | |
| *** raissa has joined #openstack-infra | 22:05 | |
| *** raissa has quit IRC | 22:05 | |
| *** raissa has joined #openstack-infra | 22:06 | |
| *** raissa has quit IRC | 22:06 | |
| *** rcernin has quit IRC | 22:06 | |
| *** smarcet has left #openstack-infra | 22:10 | |
| *** whoami-rajat has quit IRC | 22:11 | |
| *** dave-mccowan has joined #openstack-infra | 22:12 | |
| *** slaweq has joined #openstack-infra | 22:16 | |
| *** pkopec has quit IRC | 22:19 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 22:20 |
| *** Emine has joined #openstack-infra | 22:21 | |
| *** slaweq has quit IRC | 22:24 | |
| *** emine__ has joined #openstack-infra | 22:27 | |
| *** Emine has quit IRC | 22:29 | |
| *** mriedem has quit IRC | 22:29 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 22:30 |
| *** dave-mccowan has quit IRC | 22:31 | |
| clarkb | corvus: I filed my expenses :) thank you for the reminder | 22:38 |
| fungi | mordred: you probably understand more about pbr's use of pkg_resources than anyone... do you think it's possible to replace its use with importlib_metadata like https://github.com/tox-dev/tox/commit/afe11da does? | 22:40 |
| clarkb | fungi: reading the docs there is a good chance it can actually | 22:41 |
| clarkb | if we can get osc/cliff to use that too we may fix the osc performance problems | 22:41 |
| clarkb | ugh I'm gonna get nerdsniped on this | 22:42 |
| fungi | sorry, just saw it in the draft changelog for tox completely by chance | 22:42 |
| fungi | didn't mean to interrupt! | 22:43 |
| clarkb | no no this is great | 22:43 |
| clarkb | I'm glad they are fixing the super slow startup cost in pkg_resources | 22:43 |
| fungi | yeah, it came up again on the starlingx-discuss ml a couple weeks back about how using pbr for runtime version strings significantly increases startup time, and i had to do a fair amount of debunking | 22:44 |
| clarkb | ya the issue goes far deeper than that unfortunately | 22:45 |
| clarkb | like entrypoints + pkg_resources hit it too | 22:46 |
| clarkb | so if you "fix" pbr you don't win in many cases (like osc case) becuase you still ahve the pkg_resources cost incurred for plugin handling | 22:46 |
| clarkb | which is what makes this so painful to change (you don't really know if you've helped anything until you've fixed everything) | 22:46 |
| *** emine__ has quit IRC | 22:51 | |
| ianw | hrm, i would have expected that zuul variable change for the mirrors to have rolled out by now | 22:52 |
| ianw | the latest change in /etc/project-config on the ze's is "Date: Fri Apr 19 22:21:35 2019 +0000" ... that is a highly suspicious date to have a last log for i think | 22:58 |
| clarkb | fungi: mordred dtroyer remote: https://review.opendev.org/661856 Replace pkg_resources with importlib-metadata | 23:00 |
| *** dayou_ has quit IRC | 23:01 | |
| clarkb | I have no idea if that is functional but if we can start replacing things we should eventually start to see the performance benefits | 23:01 |
| clarkb | ianw: hrm | 23:01 |
| *** tkajinam has joined #openstack-infra | 23:02 | |
| clarkb | ianw: https://git.openstack.org/openstack-infra/project-config is the remote there whcih should redirect properly | 23:02 |
| ianw | ay 28 22:38:34 ze01 puppet-user[4324]: (/Stage[main]/Project_config/Vcsrepo[/etc/project-config]/ensure) Updating to latest '1e6705f722355537e08a0187dfc957949389dc09' revision | 23:02 |
| ianw | my question now is how does "updating to latest" make that decision | 23:02 |
| clarkb | maybe try a git remote update and see what that does? | 23:03 |
| ianw | [remote "origin"] | 23:03 |
| ianw | url = https://git.openstack.org/openstack-infra/project-config | 23:03 |
| *** dayou has joined #openstack-infra | 23:03 | |
| *** tosky has quit IRC | 23:04 | |
| clarkb | so .git/FETCH_HEAD is from today | 23:04 |
| clarkb | but it isn't checking that out | 23:04 |
| clarkb | so I think the remote update is working at least | 23:04 |
| clarkb | one thing I know is that vcsrepo won't update a remote. But this was only an issue previously where we used git:// | 23:05 |
| clarkb | so I manually updated the git:// cases | 23:05 |
| clarkb | https:// with redirects being a problem is new to me | 23:06 |
| *** smarcet has joined #openstack-infra | 23:06 | |
| *** smarcet has left #openstack-infra | 23:07 | |
| ianw | right, FETCH_HEAD is @ 5f4aee94c286b0c8641a4fb595f38f590fd2972c which is master top of tree | 23:08 |
| ianw | so what does it do after "git remote update" | 23:08 |
| clarkb | it should checkout FETCH_HEAD I thought | 23:08 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| clarkb | ianw: you can probably try updating the remote by hand (like I did with git://) and see if that makes it happier? | 23:12 |
| ianw | yeah, i'm sort of more worried that this plays out on every single server | 23:13 |
| ianw | review.opendev.org has the same thing, it seems, but that's updated | 23:13 |
| ianw | it's /etc/project-config has the same openstack.org origin | 23:14 |
| *** dave-mccowan has joined #openstack-infra | 23:14 | |
| *** _erlon_ has quit IRC | 23:16 | |
| clarkb | ya I know we checked it on the git farm related nodes | 23:17 |
| clarkb | because we needed it up to date for zuul main.yaml on schedulwr and review for manage projects and so on | 23:17 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Exclude old distros from yum-puppetlabs mirror https://review.opendev.org/661860 | 23:18 |
| ianw | clarkb: here's a difference ... http://paste.openstack.org/show/752197/ review seems to maybe get the HEAD reference externally; while ze's don't | 23:20 |
| *** rcernin has joined #openstack-infra | 23:21 | |
| fungi | infra-root: mwhahaha: i ran with 661860 manually to get the mirror updated again, but approving quickly would be appreciated so it doesn't regress | 23:21 |
| clarkb | ianw ya we ensure review and zuul.o.o and gitea cluster all operate with same project config data | 23:22 |
| clarkb | maybe that forces the checkout | 23:22 |
| ianw | right; https://opendev.org/opendev/system-config/src/branch/master/playbooks/remote_puppet_git.yaml | 23:23 |
| *** slaweq has quit IRC | 23:24 | |
| ianw | so that's different | 23:24 |
| ianw | i would say the problem is that we've swapped the source=> to vcsrepo, and that has somehow confused it | 23:25 |
| *** lseki has quit IRC | 23:31 | |
| *** dave-mccowan has quit IRC | 23:33 | |
| ianw | hrm, puppet-vcsrepo hasn't appreciably changed in 7+ years; so heaven knows if this might have been fixed upstream etc | 23:34 |
| ianw | there is certainly a lot more in current code that looks like it handles swapping sources -> https://github.com/puppetlabs/puppetlabs-vcsrepo/blob/master/lib/puppet/provider/vcsrepo/git.rb#L164 | 23:35 |
| mordred | clarkb, fungi: ooh neat! (re importlib-metadata) | 23:36 |
| mordred | clarkb: fixing that in python-openstackclient itself should actually fix osc and its plugins - since its plugins should not, unless something is really weird, be doing the pkg_resources things themselves | 23:37 |
| *** jbadiapa has quit IRC | 23:37 | |
| mordred | clarkb: unless, of course, this means ANY use of pkg_resources - because if we do mean that then the __version__ lines in the libraries might wind up being an issue | 23:38 |
| *** jbadiapa has joined #openstack-infra | 23:38 | |
| mordred | clarkb: like: __version__ = pbr.version.VersionInfo('python-novaclient').version_string() | 23:39 |
| mordred | clarkb: which uses pkg_resources to do the VersionInfo | 23:39 |
| mordred | clarkb: but - if we fix pbr - and that patch you made to osc - I think that should actually handle things | 23:40 |
| *** gyee has quit IRC | 23:40 | |
| clarkb | mordred its any useiirc | 23:40 |
| mordred | kk. then yeah - we'll def want to ... oh, this is now very very tricky ... | 23:41 |
| mordred | as of right now we avoid importing libraries in pbr | 23:41 |
| *** yamamoto has joined #openstack-infra | 23:41 | |
| mordred | because of bootstrapping issues | 23:41 |
| mordred | I don't remember if pip has fixed its processing of setup_requires so that transitive depends of setup_requires requirements get processed by pip and not by easy_install | 23:42 |
| mordred | so we should check that | 23:42 |
| mordred | if so - then it might be ok to have pbr depend on this particular lib | 23:42 |
| mordred | or - we might want to see if the new PEP stuff about splitting wheel builder plugins is usable enough for us to dive in to making pbr provide one of those | 23:43 |
| mordred | anyway - may moving threads - but potentially exciting if we can solve that pkg_resources cost | 23:43 |
| clarkb | ya and really if we can fix one at a time we'll make progress and be future proof | 23:44 |
| mordred | ++ | 23:44 |
| *** dave-mccowan has joined #openstack-infra | 23:46 | |
| *** dave-mccowan has quit IRC | 23:53 | |
| *** takamatsu has quit IRC | 23:56 | |
| *** Weifan has quit IRC | 23:59 | |
| *** aaronsheffield has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
