| *** cgoncalves has quit IRC | 00:00 | |
| *** pbourke_ has quit IRC | 00:13 | |
| *** pbourke_ has joined #openstack-infra | 00:13 | |
| *** hwoarang has quit IRC | 00:17 | |
| openstackgerrit | Merged openstack-infra/git-review master: tests/__init__.py: ssh-keygen -m PEM for bouncycastle https://review.openstack.org/622636 | 00:17 |
|---|---|---|
| *** hwoarang has joined #openstack-infra | 00:19 | |
| *** dave-mccowan has quit IRC | 00:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Fix enumerated list in gerrit.rst https://review.openstack.org/550757 | 00:24 |
| *** jamesmcarthur has joined #openstack-infra | 00:42 | |
| *** jamesmcarthur has quit IRC | 00:47 | |
| *** dave-mccowan has joined #openstack-infra | 00:57 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix spelling mistakes and reST typos in the doc https://review.openstack.org/623668 | 01:04 |
| openstackgerrit | Merged openstack-infra/system-config master: Fix dead link https://review.openstack.org/546930 | 01:04 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Trivial: Update pypi url to new url https://review.openstack.org/563415 | 01:20 |
| *** markvoelker has quit IRC | 01:21 | |
| *** markvoelker has joined #openstack-infra | 01:22 | |
| *** markvoelker has quit IRC | 01:26 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: fix somes typos in doc file. https://review.openstack.org/542740 | 01:27 |
| *** jamesmcarthur has joined #openstack-infra | 01:37 | |
| *** jamesmcarthur has quit IRC | 01:40 | |
| *** jamesmcarthur has joined #openstack-infra | 01:41 | |
| *** hongbin has quit IRC | 01:45 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix enumerated list in gerrit.rst https://review.openstack.org/550757 | 01:47 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Ectomy some Jenkins out of the docs https://review.openstack.org/436452 | 01:49 |
| *** yamamoto has quit IRC | 01:50 | |
| *** yamamoto has joined #openstack-infra | 01:53 | |
| *** yamamoto has quit IRC | 01:53 | |
| *** yamamoto has joined #openstack-infra | 01:56 | |
| *** yamamoto has quit IRC | 01:56 | |
| *** yamamoto has joined #openstack-infra | 01:57 | |
| *** wolverineav has joined #openstack-infra | 02:02 | |
| openstackgerrit | Merged openstack-infra/system-config master: fix somes typos in doc file. https://review.openstack.org/542740 | 02:04 |
| *** yamamoto has quit IRC | 02:05 | |
| openstackgerrit | Merged openstack-infra/system-config master: Trivial: Update pypi url to new url https://review.openstack.org/563415 | 02:09 |
| *** mrsoul has quit IRC | 02:11 | |
| *** lbragstad has quit IRC | 02:13 | |
| *** jamesmcarthur has quit IRC | 02:15 | |
| *** jamesmcarthur has joined #openstack-infra | 02:16 | |
| *** lbragstad has joined #openstack-infra | 02:16 | |
| *** bobh has joined #openstack-infra | 02:18 | |
| *** agopi-pto has joined #openstack-infra | 02:28 | |
| *** agopi-pto is now known as agopi | 02:29 | |
| openstackgerrit | 98k proposed openstack-dev/hacking master: Don't quote {posargs} in tox.ini https://review.openstack.org/609192 | 02:37 |
| *** yamamoto has joined #openstack-infra | 02:39 | |
| *** yamamoto has quit IRC | 02:46 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] kafs testing https://review.openstack.org/623974 | 02:48 |
| *** yamamoto has joined #openstack-infra | 02:50 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] kafs testing https://review.openstack.org/623974 | 02:50 |
| *** wolverineav has quit IRC | 02:55 | |
| *** wolverineav has joined #openstack-infra | 02:56 | |
| *** psachin has joined #openstack-infra | 02:59 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] kafs testing https://review.openstack.org/623974 | 03:00 |
| *** hongbin has joined #openstack-infra | 03:05 | |
| *** wolverineav has quit IRC | 03:13 | |
| *** jamesmcarthur has quit IRC | 03:14 | |
| *** bobh has quit IRC | 03:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] kafs testing https://review.openstack.org/623974 | 03:33 |
| *** jamesmcarthur has joined #openstack-infra | 03:34 | |
| *** lbragstad has quit IRC | 03:38 | |
| *** jamesmcarthur has quit IRC | 03:39 | |
| *** udesale has joined #openstack-infra | 03:40 | |
| *** bobh has joined #openstack-infra | 03:42 | |
| *** wolverineav has joined #openstack-infra | 03:44 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] kafs testing https://review.openstack.org/623974 | 03:44 |
| *** jamesmcarthur has joined #openstack-infra | 03:45 | |
| *** jamesmcarthur has quit IRC | 03:48 | |
| *** jamesmcarthur has joined #openstack-infra | 03:48 | |
| ianw | hrm, the gate is not busy but my job has been waiting 10+ minutes for a fedora node | 03:55 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Rework zuul nodepool stats reporting https://review.openstack.org/620285 | 03:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Add a statsd check for clashing keys https://review.openstack.org/620436 | 03:57 |
| *** hongbin has quit IRC | 04:00 | |
| ianw | 2018-12-10 03:45:11,006 DEBUG zuul.Pipeline.openstack.check: Adding node request <NodeRequest 300-0000774611 <NodeSet [<Node None ('base',):fedora-28>]>> | 04:04 |
| ianw | it's now Mon Dec 10 04:05:02 UTC 2018 | 04:05 |
| ianw | 2018-12-10 04:05:24,101 ERROR nodepool.NodeLauncher-0001088594: Request 300-0000774611: Launch attempt 2/3 failed for node 0001088594: | 04:05 |
| ianw | 2018-12-10 03:45:14,583 DEBUG nodepool.driver.NodeRequestHandler[nl01-26872-PoolWorker.rax-iad-main]: Locked building node 0001088594 for request 300-0000774611 | 04:09 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] kafs testing https://review.openstack.org/623974 | 04:11 |
| ianw | ... let's see if this goes to a different provider and works ... either something just broke with fedora or we have some sort of fedora rax issue :/ | 04:12 |
| ianw | seeing as the images are about 1/2 hour old, i'm not confident | 04:15 |
| *** wolverineav has quit IRC | 04:19 | |
| *** wolverineav has joined #openstack-infra | 04:19 | |
| ianw | 8c99c67791054cc697b2edc3b06377aa was limestone and worked | 04:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] kafs testing https://review.openstack.org/623974 | 04:24 |
| *** jamesmcarthur has quit IRC | 04:26 | |
| *** bobh has quit IRC | 04:33 | |
| *** ramishra has joined #openstack-infra | 04:34 | |
| *** hwoarang has quit IRC | 04:34 | |
| *** hwoarang has joined #openstack-infra | 04:36 | |
| *** ykarel has joined #openstack-infra | 04:40 | |
| *** zzzeek has quit IRC | 04:41 | |
| *** zzzeek has joined #openstack-infra | 04:41 | |
| openstackgerrit | Merged openstack-infra/zuul master: Rework zuul nodepool stats reporting https://review.openstack.org/620285 | 04:42 |
| openstackgerrit | Merged openstack-infra/zuul master: Add a statsd check for clashing keys https://review.openstack.org/620436 | 04:42 |
| *** jamesmcarthur has joined #openstack-infra | 04:47 | |
| *** bobh has joined #openstack-infra | 04:47 | |
| *** dave-mccowan has quit IRC | 04:53 | |
| *** chkumar|off is now known as chandan_kumar | 04:54 | |
| *** ykarel_ has joined #openstack-infra | 04:59 | |
| *** bobh has quit IRC | 05:00 | |
| *** ykarel has quit IRC | 05:02 | |
| openstackgerrit | Jea-Min, Lim proposed openstack-infra/project-config master: Add new project called ku.stella https://review.openstack.org/623396 | 05:11 |
| openstackgerrit | Jea-Min, Lim proposed openstack-infra/project-config master: Add new project called ku.stella https://review.openstack.org/623396 | 05:12 |
| *** hrubi has joined #openstack-infra | 05:14 | |
| *** jamesmcarthur has quit IRC | 05:22 | |
| *** wolverineav has quit IRC | 05:26 | |
| openstackgerrit | Jea-Min, Lim proposed openstack-infra/project-config master: Add new project called ku.stella https://review.openstack.org/623396 | 05:32 |
| *** yamamoto has quit IRC | 05:36 | |
| *** jamesmcarthur has joined #openstack-infra | 05:39 | |
| *** yamamoto has joined #openstack-infra | 05:39 | |
| *** wolverineav has joined #openstack-infra | 05:43 | |
| *** jamesmcarthur has quit IRC | 05:44 | |
| *** yamamoto has quit IRC | 05:44 | |
| *** wolverineav has quit IRC | 05:46 | |
| *** jamesmcarthur has joined #openstack-infra | 05:49 | |
| *** jbadiapa has joined #openstack-infra | 06:08 | |
| Linkid | Hi | 06:19 |
| Linkid | Thanks ianw for your quick reviews :) | 06:19 |
| Linkid | (and sorry for digging up some old commits ^^) | 06:21 |
| Linkid | (I procrastinated and found some typos all the week-end...) | 06:24 |
| *** wolverineav has joined #openstack-infra | 06:24 | |
| *** _alastor_ has joined #openstack-infra | 06:29 | |
| *** ykarel_ has quit IRC | 06:30 | |
| ianw | Linkid: np, we might as well just fix such things | 06:32 |
| *** yamamoto has joined #openstack-infra | 06:37 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] kafs testing https://review.openstack.org/623974 | 06:39 |
| *** yamamoto has quit IRC | 06:42 | |
| *** jamesmcarthur has quit IRC | 06:49 | |
| *** wolverineav has quit IRC | 07:00 | |
| openstackgerrit | neilsun proposed openstack-infra/zuul master: Add type check for zuul conf https://review.openstack.org/591917 | 07:04 |
| *** jamesmcarthur has joined #openstack-infra | 07:06 | |
| *** jamesmcarthur has quit IRC | 07:11 | |
| *** yamamoto has joined #openstack-infra | 07:12 | |
| *** quiquell has joined #openstack-infra | 07:13 | |
| *** alexchadin has joined #openstack-infra | 07:16 | |
| *** aojea has joined #openstack-infra | 07:18 | |
| *** gfidente has joined #openstack-infra | 07:20 | |
| *** jamesmcarthur has joined #openstack-infra | 07:22 | |
| *** dpawlik has joined #openstack-infra | 07:22 | |
| *** rcernin has quit IRC | 07:23 | |
| *** apetrich has joined #openstack-infra | 07:24 | |
| *** udesale has quit IRC | 07:24 | |
| *** jamesmcarthur has quit IRC | 07:26 | |
| *** neilsun has joined #openstack-infra | 07:27 | |
| *** aojea has quit IRC | 07:31 | |
| *** jamesmcarthur has joined #openstack-infra | 07:38 | |
| *** pgaxatte has joined #openstack-infra | 07:41 | |
| *** jamesmcarthur has quit IRC | 07:42 | |
| *** slaweq has joined #openstack-infra | 07:47 | |
| *** ykarel_ has joined #openstack-infra | 07:48 | |
| *** ahosam has joined #openstack-infra | 07:49 | |
| *** quiquell is now known as quiquell|brb | 07:51 | |
| *** jamesmcarthur has joined #openstack-infra | 07:54 | |
| *** ccamacho has joined #openstack-infra | 07:55 | |
| *** jamesmcarthur has quit IRC | 07:58 | |
| *** aojea has joined #openstack-infra | 07:59 | |
| *** _alastor_ has quit IRC | 08:00 | |
| *** adriancz has joined #openstack-infra | 08:03 | |
| amorin | hey fungi clarkb frickler and all | 08:05 |
| amorin | about BHS1 issue | 08:05 |
| amorin | my proposal is to | 08:05 |
| amorin | spawn 20 instances, i'll do the job to make sure that only one is running per hosts, then you'll be able to do some performance test on each instance, and we try to figure out which hypervisor is giving bad results. | 08:05 |
| amorin | we have 20 hypervisors on the aggregate | 08:05 |
| amorin | I update the etherpad with this proposal | 08:05 |
| amorin | https://etherpad.openstack.org/p/bhs1-test-node-slowness | 08:05 |
| *** quiquell|brb is now known as quiquell | 08:08 | |
| *** jamesmcarthur has joined #openstack-infra | 08:10 | |
| *** evrardjp_ is now known as evrardjp | 08:11 | |
| *** shardy has joined #openstack-infra | 08:14 | |
| *** ykarel_ is now known as ykarel | 08:16 | |
| *** wolverineav has joined #openstack-infra | 08:16 | |
| *** ginopc has joined #openstack-infra | 08:17 | |
| *** ykarel is now known as ykarel|lunch | 08:17 | |
| *** xek has joined #openstack-infra | 08:18 | |
| *** wolverineav has quit IRC | 08:20 | |
| *** imacdonn has quit IRC | 08:22 | |
| *** imacdonn has joined #openstack-infra | 08:22 | |
| *** fresta_ has joined #openstack-infra | 08:30 | |
| *** jonher_ has joined #openstack-infra | 08:30 | |
| *** jonher has quit IRC | 08:31 | |
| *** jonher_ is now known as jonher | 08:31 | |
| *** fresta has quit IRC | 08:31 | |
| *** bhavikdbavishi has quit IRC | 08:34 | |
| *** yboaron has quit IRC | 08:36 | |
| *** ykarel|lunch is now known as ykarel | 08:42 | |
| *** jamesmcarthur has quit IRC | 08:43 | |
| *** tosky has joined #openstack-infra | 08:45 | |
| *** yamamoto has quit IRC | 08:47 | |
| *** udesale has joined #openstack-infra | 08:49 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:51 | |
| *** jpich has joined #openstack-infra | 08:51 | |
| *** jamesmcarthur has joined #openstack-infra | 08:55 | |
| *** d0ugal has quit IRC | 08:55 | |
| *** d0ugal has joined #openstack-infra | 08:56 | |
| *** jtomasek has joined #openstack-infra | 08:57 | |
| *** jamesmcarthur has quit IRC | 09:00 | |
| *** jpena|off is now known as jpena | 09:01 | |
| *** jamesmcarthur has joined #openstack-infra | 09:11 | |
| *** apetrich has quit IRC | 09:13 | |
| *** apetrich has joined #openstack-infra | 09:14 | |
| *** jamesmcarthur has quit IRC | 09:16 | |
| *** yamamoto has joined #openstack-infra | 09:17 | |
| *** rossella_s has joined #openstack-infra | 09:17 | |
| *** jpich has quit IRC | 09:18 | |
| *** apetrich has quit IRC | 09:19 | |
| *** eumel8 has joined #openstack-infra | 09:21 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Implement zookeeper-auth https://review.openstack.org/619156 | 09:23 |
| *** jamesmcarthur has joined #openstack-infra | 09:27 | |
| *** derekh has joined #openstack-infra | 09:31 | |
| *** jamesmcarthur has quit IRC | 09:32 | |
| *** apetrich has joined #openstack-infra | 09:32 | |
| *** bhavikdbavishi has quit IRC | 09:37 | |
| *** jpich has joined #openstack-infra | 09:42 | |
| *** jpich has quit IRC | 09:42 | |
| *** ahosam has quit IRC | 09:43 | |
| *** jamesmcarthur has joined #openstack-infra | 09:43 | |
| *** AJaeger has quit IRC | 09:45 | |
| *** jpich has joined #openstack-infra | 09:45 | |
| *** jamesmcarthur has quit IRC | 09:48 | |
| *** electrofelix has joined #openstack-infra | 09:49 | |
| *** yboaron has joined #openstack-infra | 09:49 | |
| *** AJaeger has joined #openstack-infra | 09:53 | |
| *** yamamoto has quit IRC | 09:58 | |
| *** jamesmcarthur has joined #openstack-infra | 09:59 | |
| *** e0ne has joined #openstack-infra | 10:03 | |
| *** jamesmcarthur has quit IRC | 10:03 | |
| *** adriancz has quit IRC | 10:04 | |
| *** adriancz has joined #openstack-infra | 10:04 | |
| *** yboaron_ has joined #openstack-infra | 10:04 | |
| *** yboaron has quit IRC | 10:07 | |
| *** jamesmcarthur has joined #openstack-infra | 10:15 | |
| *** udesale has quit IRC | 10:18 | |
| *** jamesmcarthur has quit IRC | 10:19 | |
| *** betherly has joined #openstack-infra | 10:22 | |
| *** cgoncalves has joined #openstack-infra | 10:27 | |
| *** priteau has joined #openstack-infra | 10:28 | |
| *** e0ne has quit IRC | 10:30 | |
| *** kaisers has left #openstack-infra | 10:30 | |
| *** jamesmcarthur has joined #openstack-infra | 10:31 | |
| *** yamamoto has joined #openstack-infra | 10:31 | |
| *** jamesmcarthur has quit IRC | 10:35 | |
| *** e0ne has joined #openstack-infra | 10:36 | |
| frickler | amorin: not sure how you want us to proceed, would you like us to spawn 20 instances first, and then distribute them? | 10:37 |
| *** yamamoto has quit IRC | 10:39 | |
| *** dpawlik has quit IRC | 10:39 | |
| *** cgoncalves has quit IRC | 10:40 | |
| *** yboaron_ has quit IRC | 10:41 | |
| *** yboaron_ has joined #openstack-infra | 10:41 | |
| *** dpawlik has joined #openstack-infra | 10:45 | |
| *** ldnunes has joined #openstack-infra | 10:45 | |
| *** jamesmcarthur has joined #openstack-infra | 10:46 | |
| *** jamesmcarthur has quit IRC | 10:51 | |
| *** yamamoto has joined #openstack-infra | 10:53 | |
| *** jamesmcarthur has joined #openstack-infra | 11:02 | |
| *** pbourke_ has quit IRC | 11:05 | |
| *** jamesmcarthur has quit IRC | 11:06 | |
| *** pbourke_ has joined #openstack-infra | 11:08 | |
| *** ahosam has joined #openstack-infra | 11:08 | |
| *** udesale has joined #openstack-infra | 11:11 | |
| *** larainema has quit IRC | 11:15 | |
| *** sshnaidm has quit IRC | 11:15 | |
| *** sshnaidm has joined #openstack-infra | 11:16 | |
| *** jamesmcarthur has joined #openstack-infra | 11:18 | |
| *** ahosam has quit IRC | 11:20 | |
| *** ahosam has joined #openstack-infra | 11:20 | |
| *** jamesmcarthur has quit IRC | 11:23 | |
| openstackgerrit | neilsun proposed openstack-infra/zuul master: Add type check for zuul conf https://review.openstack.org/591917 | 11:24 |
| *** dtantsur|afk is now known as dtantsur | 11:25 | |
| *** kjackal has joined #openstack-infra | 11:25 | |
| amorin | frickler: yes | 11:29 |
| *** jamesmcarthur has joined #openstack-infra | 11:31 | |
| *** dayou_ has quit IRC | 11:34 | |
| *** jtomasek has quit IRC | 11:34 | |
| *** jamesmcarthur has quit IRC | 11:35 | |
| *** jamesmcarthur has joined #openstack-infra | 11:43 | |
| *** dayou has joined #openstack-infra | 11:43 | |
| *** yamamoto has quit IRC | 11:47 | |
| *** yamamoto has joined #openstack-infra | 11:47 | |
| *** jamesmcarthur has quit IRC | 11:47 | |
| *** dayou has quit IRC | 11:51 | |
| *** quiquell is now known as quiquell|brb | 11:55 | |
| *** tpsilva has joined #openstack-infra | 11:55 | |
| *** jamesmcarthur has joined #openstack-infra | 11:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:58 | |
| *** dayou has joined #openstack-infra | 11:58 | |
| *** e0ne has quit IRC | 11:58 | |
| *** jamesmcarthur has quit IRC | 12:00 | |
| *** e0ne has joined #openstack-infra | 12:00 | |
| *** e0ne has quit IRC | 12:00 | |
| *** yamamoto has quit IRC | 12:04 | |
| *** jamesmcarthur has joined #openstack-infra | 12:08 | |
| *** quiquell|brb is now known as quiquell | 12:10 | |
| *** jamesmcarthur has quit IRC | 12:12 | |
| *** yamamoto has joined #openstack-infra | 12:16 | |
| *** jtomasek has joined #openstack-infra | 12:17 | |
| *** larainema has joined #openstack-infra | 12:19 | |
| *** jamesmcarthur has joined #openstack-infra | 12:21 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** dpawlik has quit IRC | 12:23 | |
| *** dpawlik has joined #openstack-infra | 12:23 | |
| *** jamesmcarthur has quit IRC | 12:26 | |
| *** bhavikdbavishi has quit IRC | 12:26 | |
| frickler | amorin: o.k., I started 20 hosts, waiting 30 seconds in between them. one still failed in the middle with "There are not enough hosts available.'", I left that one in place in case you want to further debug. | 12:31 |
| *** kjackal has quit IRC | 12:32 | |
| frickler | amorin: you can distribute the others, once finished I would start running the simple dd tests again and some devstack setup afterwards | 12:32 |
| *** kjackal_v2 has joined #openstack-infra | 12:32 | |
| *** dayou has quit IRC | 12:32 | |
| *** jamesmcarthur has joined #openstack-infra | 12:33 | |
| *** dayou has joined #openstack-infra | 12:34 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:34 | |
| *** kjackal_v2 has quit IRC | 12:36 | |
| *** kjackal has joined #openstack-infra | 12:36 | |
| *** jpena is now known as jpena|lunch | 12:37 | |
| *** jamesmcarthur has quit IRC | 12:37 | |
| *** jamesmcarthur has joined #openstack-infra | 12:45 | |
| *** tosky has quit IRC | 12:47 | |
| *** rh-jelabarre has joined #openstack-infra | 12:47 | |
| *** jistr is now known as jistr|medchk | 12:57 | |
| *** bobh has joined #openstack-infra | 12:58 | |
| *** jamesmcarthur has quit IRC | 12:58 | |
| *** rlandy has joined #openstack-infra | 12:58 | |
| *** weshay has joined #openstack-infra | 12:59 | |
| *** dayou has quit IRC | 13:01 | |
| *** ahosam has quit IRC | 13:02 | |
| *** psachin has quit IRC | 13:05 | |
| *** jamesmcarthur has joined #openstack-infra | 13:06 | |
| *** boden has joined #openstack-infra | 13:11 | |
| *** jtomasek_ has joined #openstack-infra | 13:14 | |
| *** jamesmcarthur has quit IRC | 13:15 | |
| *** jtomasek has quit IRC | 13:15 | |
| *** jamesmcarthur has joined #openstack-infra | 13:15 | |
| *** dave-mccowan has joined #openstack-infra | 13:17 | |
| *** dpawlik has quit IRC | 13:17 | |
| *** dayou has joined #openstack-infra | 13:18 | |
| *** jtomasek_ is now known as jtomasek | 13:19 | |
| *** dpawlik has joined #openstack-infra | 13:20 | |
| *** priteau has quit IRC | 13:23 | |
| *** dpawlik has quit IRC | 13:30 | |
| fungi | infra-root: per tonyb's message to openstack-discuss about release job failures, i'm stopping ze12 for a few minutes to troubleshoot why afsd isn't running on it | 13:32 |
| fungi | i have a feeling this could be fallout from not rebooting after bootstrapping it | 13:32 |
| *** dpawlik has joined #openstack-infra | 13:32 | |
| fungi | yeah, confirmed, the openafs lkm isn't loaded either | 13:33 |
| pabelanger | fungi: yes, would agree. the server needs rebooting to pick up latest kernel also | 13:33 |
| fungi | well, that could also pose a problem. i see that we've got openafs.ko built for the running 4.4.0-137-generic kernel, but not for the 4.15.0-42-generic which is the newest installed one | 13:34 |
| *** jpena|lunch is now known as jpena | 13:35 | |
| fungi | i'll try rerunning the postinst for it | 13:35 |
| fungi | openafs-modules-dkms is definitely installed | 13:36 |
| fungi | running `sudo dpkg-reconfigure openafs-modules-dkms` now | 13:38 |
| *** kgiusti has joined #openstack-infra | 13:39 | |
| fungi | this is going to take a while | 13:39 |
| *** kgiusti has left #openstack-infra | 13:41 | |
| sean-k-mooney | o/ quick question. can nodepool be configred to prefer 1 cloud over another. e.g. if i have a local private cloud and a limit public cloud can i prefer to use the local resouces and only use the public resoces if i run out or weight it in some way so that 75% run locally and 25% remote | 13:41 |
| *** mdrabe has joined #openstack-infra | 13:41 | |
| *** kgiusti has joined #openstack-infra | 13:41 | |
| *** bhavikdbavishi has quit IRC | 13:42 | |
| fungi | sean-k-mooney: i think i've heard that requested before but i don't recall if anyone has a solution worked out for it. you might ask in #zuul instead where more of the zuul/nodepool maintainers hang out | 13:42 |
| pabelanger | sean-k-mooney: currently no, node requests are randon across clouds ATM | 13:42 |
| sean-k-mooney | ok thats fine | 13:42 |
| sean-k-mooney | fungi: thanks | 13:42 |
| pabelanger | but agree, it would be nice. especially in the case wher you might have 1 donated cloud and one you pay for | 13:43 |
| fungi | i can think of a variety of reasons for wanting that sort of node scheduling, for that matter, sure | 13:43 |
| fungi | the reconfig has finally gotten to "Building initial module for 4.15.0-42-generic" | 13:44 |
| fungi | fingers crossed | 13:44 |
| sean-k-mooney | ya its not an issue for now as i only have the local cloud. im considering adding public cloud for bustable resource in the future. | 13:45 |
| fungi | sean-k-mooney: nodepool is written in python and is pretty easy to dig into if you get the urge. i hear the maintainers are really welcoming too! ;) | 13:46 |
| *** owalsh_ has joined #openstack-infra | 13:47 | |
| sean-k-mooney | perhaps after i have deployed an run it for a bit. i find its alway easist to learn a new codebase when your trying to scratch an iche thats bugging you | 13:47 |
| fungi | i have a feeling most of the work would be in designing the configuration option(s) for selecting an alternative node scheduling scheme and getting it plumbed through, but again #zuul is a much better place to talk about it | 13:47 |
| fungi | or zuul-discuss@lists.zuul-ci.org | 13:47 |
| sean-k-mooney | ya | 13:47 |
| sean-k-mooney | im on the zuul channel too i just taught it may have come up as a usecase for the upstream ci at some point anyway thanks | 13:48 |
| fungi | ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/openafs-modules-dkms.0.crash' Error! Bad return status for module build on kernel: 4.15.0-42-generic (x86_64) Consult /var/lib/dkms/openafs/1.6.15/build/make.log for more information. | 13:49 |
| fungi | ugh | 13:49 |
| fungi | i guess that's why we didn't have the lkm built yet. it probably already crashed once when we were bringing up the server | 13:50 |
| *** owalsh has quit IRC | 13:50 | |
| *** priteau has joined #openstack-infra | 13:50 | |
| *** owalsh has joined #openstack-infra | 13:50 | |
| fungi | /var/lib/dkms/openafs/1.6.15/build/src/afs/LINUX/osi_machdep.h:73:3: error: #error Not sure what to do about rlim (should be in the Linux task struct somewhere....) | 13:51 |
| *** owalsh_ has quit IRC | 13:52 | |
| fungi | strange, our other executors seem to have built it successfully | 13:53 |
| fungi | oh, they're using openafs-modules-dkms 1.6.22.2-1ppa1 and ze12 has 1.6.15-1ubuntu1 | 13:54 |
| fungi | ianw: ^ if you're still around, i vaguely recall you having more insight into the ppa situation there than i do | 13:55 |
| *** jistr|medchk is now known as jistr | 13:56 | |
| fungi | looks like we have http://ppa.launchpad.net/openstack-ci-core/openafs-amd64-hwe/ubuntu active in /etc/apt/sources.list.d/openstack-ci-core-ubuntu-openafs-amd64-hwe-xenial.list | 13:56 |
| fungi | so i probably just need to explicitly request apt pull in that version | 13:56 |
| *** jamesmcarthur has quit IRC | 13:57 | |
| fungi | looks like `sudo apt install openafs-modules-dkms=1.6.22.2-1ppa1` is working | 13:57 |
| *** jamesmcarthur has joined #openstack-infra | 13:58 | |
| *** mriedem has joined #openstack-infra | 14:00 | |
| *** rh-jelabarre has quit IRC | 14:06 | |
| openstackgerrit | Merged openstack-infra/git-review master: tox: Remove dead settings/targets https://review.openstack.org/610576 | 14:07 |
| fungi | it's up to "Building initial module for 4.15.0-42-generic" again finally | 14:08 |
| fungi | and it's finished. rebooting ze12 now | 14:12 |
| *** rh-jelabarre has joined #openstack-infra | 14:12 | |
| fungi | it's back up now running 4.15.0-42-generic with the openafs lkm loaded and afsd running | 14:13 |
| fungi | #status log upgraded openafs on ze12 with `sudo apt install openafs-modules-dkms=1.6.22.2-1ppa1` and rebooted onto the latest hwe kernel | 14:14 |
| *** e0ne has joined #openstack-infra | 14:15 | |
| fungi | i guess we're missing statusbot too | 14:15 |
| fungi | ping timeouts according to its debug log as of 2018-12-08 00:09:20,950 | 14:16 |
| fungi | that's also roughly when we lost gerritbot over the weekend, so must have been the same situation | 14:16 |
| *** lbragstad has joined #openstack-infra | 14:17 | |
| *** openstackstatus has joined #openstack-infra | 14:17 | |
| *** ChanServ sets mode: +v openstackstatus | 14:17 | |
| fungi | and it's back | 14:18 |
| fungi | #status log restarted statusbot to recover from connectivity issues from saturday | 14:18 |
| openstackstatus | fungi: finished logging | 14:18 |
| fungi | #status log upgraded openafs on ze12 with `sudo apt install openafs-modules-dkms=1.6.22.2-1ppa1` and rebooted onto the latest hwe kernel | 14:18 |
| openstackstatus | fungi: finished logging | 14:18 |
| openstackgerrit | Merged openstack-infra/git-review master: CONTRIBUTING.rst, HACKING.rst: fix broken link, minor flow updates https://review.openstack.org/623362 | 14:22 |
| openstackgerrit | Merged openstack-infra/python-storyboardclient master: Change openstack-dev to openstack-discuss https://review.openstack.org/622368 | 14:24 |
| *** chandan_kumar is now known as chkumar|off | 14:25 | |
| *** aspiers has quit IRC | 14:26 | |
| *** jamesmcarthur has quit IRC | 14:29 | |
| *** quiquell is now known as quiquell|off | 14:30 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fixed the necesssary to necessary https://review.openstack.org/623646 | 14:37 |
| *** aspiers has joined #openstack-infra | 14:38 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fixed the word from congfiguration to configuration https://review.openstack.org/623634 | 14:40 |
| *** alexchadin has quit IRC | 14:42 | |
| fungi | ze12 seems to have equalized its running builds count with the other executors as of the past few minutes, so i expect all is well there now | 14:45 |
| fungi | still curious why we're down 4 mergers according to our graphs (at 16, should be 20) | 14:46 |
| fungi | but the remaining mergers seem to be keeping up fine | 14:47 |
| *** gfidente has quit IRC | 14:49 | |
| *** gfidente has joined #openstack-infra | 14:53 | |
| *** jamesmcarthur has joined #openstack-infra | 14:59 | |
| *** yboaron_ has quit IRC | 15:01 | |
| *** yboaron_ has joined #openstack-infra | 15:02 | |
| *** jamesmcarthur has quit IRC | 15:06 | |
| *** jamesmcarthur has joined #openstack-infra | 15:07 | |
| *** gagehugo has joined #openstack-infra | 15:07 | |
| *** priteau has quit IRC | 15:10 | |
| corvus | fungi: thanks for the ze12 fix, sorry i missed that | 15:20 |
| corvus | fungi: we can probably find out which mergers are missing by interrogating gearman | 15:20 |
| fungi | yeah, i was considering checking gearman status to see which are registered, just hadn't become urgent yet | 15:21 |
| *** kjackal has quit IRC | 15:21 | |
| *** kjackal_v2 has joined #openstack-infra | 15:22 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/python-storyboardclient master: add script for tagging stories https://review.openstack.org/608707 | 15:23 |
| *** kjackal_v2 has quit IRC | 15:26 | |
| *** kjackal has joined #openstack-infra | 15:26 | |
| openstackgerrit | Jonathan Herlin proposed openstack-infra/zuul master: Add spacing to Queue lengths line https://review.openstack.org/623960 | 15:28 |
| *** sthussey has joined #openstack-infra | 15:34 | |
| *** aojeagarcia has joined #openstack-infra | 15:36 | |
| *** aojea has quit IRC | 15:38 | |
| *** aojeagarcia__ has joined #openstack-infra | 15:39 | |
| *** aojeagarcia has quit IRC | 15:42 | |
| *** aojeagarcia__ has quit IRC | 15:45 | |
| *** neilsun has quit IRC | 15:47 | |
| fungi | infra-root: i'm still in the process of writing up the forum summary for our opendev session (yeah, i know it's a month late) but it's becoming increasingly obvious that a lot of the things going into it are also things we're planning to say in the official announcement... should i keep this summary on hold pending that announcement so we don't prematurely start the same conversations on the ml we | 15:51 |
| fungi | expect the announcement to trigger? | 15:51 |
| *** bhavikdbavishi has joined #openstack-infra | 15:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add the os-resource-classes project https://review.openstack.org/621666 | 16:01 |
| *** bhavikdbavishi has quit IRC | 16:02 | |
| *** bhavikdbavishi has joined #openstack-infra | 16:03 | |
| *** jamesmcarthur has quit IRC | 16:04 | |
| *** yboaron_ has quit IRC | 16:06 | |
| *** yboaron_ has joined #openstack-infra | 16:07 | |
| *** lpetrut has joined #openstack-infra | 16:12 | |
| openstackgerrit | Merged openstack-dev/pbr master: Change openstack-dev to openstack-discuss https://review.openstack.org/622321 | 16:17 |
| ttx | If anyone is interested in ptgbot, you can review https://review.openstack.org/#/q/status:open+project:openstack/ptgbot+branch:master+topic:all-configurable -- otherwise I'll soon self-approve those, which would be a bit sad. aspiers maybe? | 16:17 |
| ttx | It's mostly tech debt reduction fwiw | 16:18 |
| aspiers | ttx: I can take a look tomorrow but if that's not quick enough I'm sure everyone would be fine with your self-approval :) | 16:18 |
| ttx | I implemented all suggestions except the topic subscription one, which I may get to next. | 16:19 |
| *** bhavikdbavishi has quit IRC | 16:20 | |
| *** jamesmcarthur has joined #openstack-infra | 16:22 | |
| aspiers | ooh, sounds cool - looking forward to taking a look at that :) | 16:25 |
| aspiers | gotta dash now, sorry | 16:25 |
| *** bhavikdbavishi has joined #openstack-infra | 16:26 | |
| *** jamesmcarthur has quit IRC | 16:27 | |
| openstackgerrit | Benoît Bayszczak proposed openstack-infra/zuul master: add fetch_vault_secrets Ansible module https://review.openstack.org/620311 | 16:27 |
| clarkb | fungi The website publication you mean? | 16:28 |
| clarkb | I sent the email announcement pre summit (session summary would be good followup to that maybe?) | 16:28 |
| fungi | clarkb: i thought we also planned a more formal announcement of the opendev plans once there was a website up, but oh i guess i misremembered and that was indeed pre-summit? | 16:29 |
| fungi | if so, no concern, i'll go ahead with sending this out | 16:29 |
| clarkb | Ya pre summit was the official email thing We should send more emails for sure but dont think that have to wait | 16:30 |
| fungi | perfect | 16:30 |
| fungi | have a link to the ml post? | 16:30 |
| fungi | the one in the session etherpad was to the winterscale announcement back in may | 16:31 |
| *** chkumar|off has quit IRC | 16:32 | |
| clarkb | where'd the -dev archive end up? though maybe it was forwarded to discuss | 16:32 |
| clarkb | hrm doesnt look like it | 16:33 |
| fungi | it's still where it was: http://lists.openstack.org/pipermail/openstack-dev/ | 16:33 |
| clarkb | ah not on the index page though? | 16:33 |
| *** ginopc has quit IRC | 16:33 | |
| fungi | it no longer appears on the list of active mailing lists, right | 16:34 |
| clarkb | http://lists.openstack.org/pipermail/openstack-dev/2018-November/136403.html | 16:34 |
| fungi | thanks! | 16:34 |
| fungi | and i suppose we could tack on a set of links to the archives for the old lists we've decommissioned if someone is interested in working on that (there are something like a dozen already, and probably will be more as time goes on too) | 16:35 |
| *** openstackgerrit has quit IRC | 16:35 | |
| *** psachin has joined #openstack-infra | 16:42 | |
| *** yboaron_ has quit IRC | 16:46 | |
| *** gyee has joined #openstack-infra | 16:47 | |
| *** gfidente has quit IRC | 16:51 | |
| *** ccamacho has quit IRC | 16:52 | |
| *** ykarel has quit IRC | 16:52 | |
| *** ykarel has joined #openstack-infra | 16:53 | |
| *** jamesmcarthur has joined #openstack-infra | 16:53 | |
| *** yamamoto has joined #openstack-infra | 16:55 | |
| *** gfidente has joined #openstack-infra | 16:56 | |
| *** openstackgerrit has joined #openstack-infra | 16:57 | |
| openstackgerrit | Benoît Bayszczak proposed openstack-infra/zuul master: add fetch_vault_secrets Ansible module https://review.openstack.org/620311 | 16:57 |
| *** jamesmcarthur has quit IRC | 16:57 | |
| *** udesale has quit IRC | 16:58 | |
| clarkb | frickler: amorin let me know if I can help with the benchmarking but seems like you have it under control? | 16:59 |
| *** yamamoto has quit IRC | 16:59 | |
| clarkb | fungi: while you've got opendev paged in care to look at https://review.openstack.org/622624 ? that is my draft of website content | 17:03 |
| *** ykarel is now known as ykarel|away | 17:03 | |
| fungi | didn't i? thought i left a comment | 17:07 |
| clarkb | oh I didn't see any +1 or -1 so assumed there wasn't any, bad assumption I take it | 17:07 |
| *** jamesmcarthur has joined #openstack-infra | 17:07 | |
| clarkb | yup, thanks, I'll read it over shortly | 17:07 |
| fungi | well, i figured it won't land until it's reworked into html and we have a job to run against it (at least noop) | 17:07 |
| fungi | so there was little point in leaving a vote | 17:08 |
| clarkb | I've updated the meeting agenda and will set out the set agenda at 1900UTC today (you still have time to add other topics if you like) | 17:14 |
| clarkb | *send out the set agenda | 17:14 |
| fungi | you know you've written a long session summary when you don't want to proofread it | 17:14 |
| clarkb | ha | 17:14 |
| corvus | fungi: your readers will do it for you! | 17:14 |
| fungi | hah | 17:15 |
| *** eumel8 has quit IRC | 17:15 | |
| clarkb | mordred: are you about today? One of the things on my list from friday is further debugging of the inap image upload issues | 17:16 |
| clarkb | it appeared to be buggy openstacksdk or keystoneauth1. Updating to latest keystoneauth1 on the nodepool builders did not fix things | 17:16 |
| clarkb | short of pdb'ing the process any thoughts on debugging that? | 17:17 |
| clarkb | I guess we could try a manual upload outside of nodepool with all the debugging turned on | 17:17 |
| *** _alastor_ has joined #openstack-infra | 17:17 | |
| clarkb | anyone else want to review a procedure docs update for infra-specs https://review.openstack.org/#/c/623211/6 before I approve that? its mostly updates that take into account that storyboard has changed a bit since we first wrote those docs | 17:19 |
| clarkb | also if any infra-root are able to review the ara stack at https://review.openstack.org/#/q/topic:inner-ara-results and the fedora-29/networkmanager stack at https://review.openstack.org/#/q/status:open+topic:fedora29 that will likely make ianw and dmsimard very happy (I've reviewed the changes and I think they are mostly ready for quick double check and go) | 17:20 |
| *** dtantsur is now known as dtantsur|afk | 17:21 | |
| *** jpich has quit IRC | 17:22 | |
| *** e0ne has quit IRC | 17:22 | |
| *** e0ne has joined #openstack-infra | 17:22 | |
| *** e0ne has quit IRC | 17:23 | |
| *** e0ne has joined #openstack-infra | 17:26 | |
| *** kjackal has quit IRC | 17:26 | |
| *** kjackal has joined #openstack-infra | 17:26 | |
| *** derekh has quit IRC | 17:29 | |
| mordred | clarkb: I am around today - finishing up a patch update, then I need to do a quick phone call, then I can totally help with that | 17:37 |
| *** bhavikdbavishi has quit IRC | 17:37 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/opendev-website master: Add some initial content thoughtso https://review.openstack.org/622624 | 17:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/opendev-website master: Add .zuul.yaml https://review.openstack.org/624139 | 17:37 |
| clarkb | mordred: thanks | 17:37 |
| *** bhavikdbavishi has joined #openstack-infra | 17:37 | |
| clarkb | fungi: corvus ^ Content now with gating so things can merge. Any thoughts on where to start with htmlification? I can sort of manually html something out I guess | 17:38 |
| corvus | clarkb: i suggest simple manual htmlification | 17:39 |
| fungi | <html><head>...</head><body><h1>...<.h1><p>...</p><p>...</p></body></html> | 17:39 |
| fungi | yeah, basic sgml tags for now and then we can always make it "pretty" later | 17:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Ectomy some Jenkins out of the docs https://review.openstack.org/436452 | 17:40 |
| corvus | clarkb: there are 2 other ideas in play and i don't know which way we might want to go: a) gatsby static site generation; b) make this page the homepage of a gitea instance. those are very different, so until we have an idea of which way to go, doing one may be wasted effort if we do the other. so really simple html is probably the safest thing right now as we can upconvert it to whichever later. | 17:41 |
| mordred | ++ totally agree | 17:41 |
| clarkb | ok | 17:41 |
| corvus | i hope to have more info so we can make a decision on that soon :) | 17:41 |
| mordred | clarkb, corvus, fungi: ^^ jenkins-ectomy patch should be ready for review - and is a little long in the tooth and easy to ignore | 17:42 |
| clarkb | there are a bunch of simple md to html formatters so I'll actually probably use oen of them to make this simpler | 17:43 |
| corvus | mordred: i reviewed it | 17:46 |
| corvus | clarkb: pandoc rocks. | 17:46 |
| corvus | clarkb: also handles rst input | 17:46 |
| fungi | yeah, pandoc can do it all | 17:46 |
| corvus | clarkb: and it's haskell | 17:46 |
| fungi | though you can also mark it up with rst and then use docutils.core.publish_string(rst, writer_name="html") | 17:50 |
| fungi | probably supports md too, i've never tried | 17:50 |
| clarkb | ya at this point I'm trying a few things to see if anything renders something that is nicer than the otehrs | 17:51 |
| clarkb | rst + sphinx would be familiar | 17:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Ectomy some Jenkins out of the docs https://review.openstack.org/436452 | 17:55 |
| mordred | corvus: thanks! updated | 17:55 |
| mordred | clarkb: given the size of the content, I could probably just make you a gatsby patch in almost no time - then any jobs we write to do the gatsby publish step could be re-used for zuul-website even if we don't wind up using gatsby for opendev and use gitea instead | 17:56 |
| mordred | clarkb: lemme hop on this phone call I need to do real quick, and when I'm off I can do that patch for you real quick | 17:57 |
| clarkb | mordred: I've got something that mostly works but needs some css. I figure figuring that out is worthwhile for myself. But feel free to push up a thing for the other thing if you like | 17:57 |
| clarkb | (also inap image uploads :P) | 17:57 |
| mordred | nod | 18:01 |
| fungi | seeing a _lot_ fewer job timeouts today | 18:01 |
| fungi | 17 in the past 6 hours | 18:01 |
| fungi | though ~50% of those are still in limestone-regionone | 18:02 |
| *** e0ne has quit IRC | 18:09 | |
| *** e0ne has joined #openstack-infra | 18:11 | |
| *** e0ne has quit IRC | 18:11 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/opendev-website master: Convert initial content to html for publication https://review.openstack.org/624149 | 18:13 |
| clarkb | cue dog meme "I have no idea what I'm doing" ^ :) | 18:13 |
| clarkb | that seems to look ok in firefox if quite simple | 18:13 |
| *** jpena is now known as jpena|off | 18:14 | |
| *** adam_g has quit IRC | 18:15 | |
| *** adam_g has joined #openstack-infra | 18:15 | |
| *** jamesmcarthur has quit IRC | 18:19 | |
| clarkb | fungi: late friday I started looking at some limestone failure and some of them were to pypi through our mirror cache | 18:19 |
| clarkb | pypi doesn't have AAAA records so we'll be NATing there | 18:20 |
| *** wolverineav has joined #openstack-infra | 18:20 | |
| clarkb | logan-: ^ any insight on if the NAT is having trouble? With the mirror it should use the 1:1 fip nAT though | 18:20 |
| *** psachin has quit IRC | 18:23 | |
| *** wolverineav has quit IRC | 18:24 | |
| clarkb | er I take that back they do have AAAA records | 18:27 |
| clarkb | so must be direct access with network blips. We have seen those in pypi before | 18:27 |
| *** diablo_rojo has joined #openstack-infra | 18:29 | |
| clarkb | mordred: for when you do get to looking at upload failures, http://paste.openstack.org/show/736850/ was the traceback from friday. We should double check it persists in case it was a cloud side issue | 18:32 |
| *** e0ne has joined #openstack-infra | 18:34 | |
| fungi | clarkb: earlier last week there was a point where both limestone-regionone and rax-dfw were failing to reach pypi.org for the same span of time. wonder if there's a local fastly cdn endpoint misbehaving for both of them given their relative global proximity | 18:40 |
| AJaeger | ~. | 18:40 |
| fungi | ...NO CARRIER | 18:41 |
| clarkb | fungi: that seems likely if they both experienced ti together | 18:41 |
| *** wolverineav has joined #openstack-infra | 18:41 | |
| *** jamesmcarthur has joined #openstack-infra | 18:42 | |
| *** bhavikdbavishi has quit IRC | 18:43 | |
| *** bhavikdbavishi has joined #openstack-infra | 18:43 | |
| *** wolverineav has quit IRC | 18:44 | |
| *** wolverineav has joined #openstack-infra | 18:47 | |
| AJaeger | config-core, mnaser has a change up for tweaking vexxhost - could you review, please? https://review.openstack.org/623605 | 18:56 |
| AJaeger | config-core, there are a couple of change queued up with +2s - could you grab some of these, please? | 18:57 |
| *** diablo_rojo has quit IRC | 18:57 | |
| *** jcoufal has joined #openstack-infra | 19:01 | |
| *** josephrsandoval has joined #openstack-infra | 19:02 | |
| *** wolverineav has quit IRC | 19:03 | |
| *** wolverineav has joined #openstack-infra | 19:03 | |
| *** e0ne has quit IRC | 19:06 | |
| *** ykarel|away has quit IRC | 19:08 | |
| *** diablo_rojo has joined #openstack-infra | 19:09 | |
| *** wolverineav has quit IRC | 19:09 | |
| *** gfidente is now known as gfidente|adk | 19:10 | |
| *** gfidente|adk is now known as gfidente|afk | 19:10 | |
| *** electrofelix has quit IRC | 19:11 | |
| *** wolverineav has joined #openstack-infra | 19:11 | |
| *** josephrsandoval has quit IRC | 19:14 | |
| *** bobh has quit IRC | 19:15 | |
| *** bhavikdbavishi has quit IRC | 19:20 | |
| *** mguiney has joined #openstack-infra | 19:20 | |
| *** anteaya has joined #openstack-infra | 19:21 | |
| pabelanger | AJaeger: fungi: left a -1 on 623605. I _think_ we need to have it be a 2 step process | 19:23 |
| pabelanger | otherwise, we'll have nodes online but no nodepool provider config | 19:23 |
| fungi | oh, yep, we need to do max-servers:0 first? | 19:23 |
| fungi | i've unapproved it for now | 19:24 |
| pabelanger | yah, once landed you then need to manually delete the online VMs or wait for them to be used. I am unsure if max-server: -1 is still a config setting, because that would delete any READY nodes | 19:24 |
| *** bobh has joined #openstack-infra | 19:25 | |
| AJaeger | pabelanger: oh - thanks! | 19:27 |
| AJaeger | mnaser: ^ | 19:28 |
| *** bobh has quit IRC | 19:30 | |
| fungi | once the max-servers:0 goes in, we can manually delete ready nodes anyway | 19:31 |
| *** bobh has joined #openstack-infra | 19:34 | |
| fungi | #status log provider indicates the host on which ze01 resides has gone offline | 19:35 |
| openstackstatus | fungi: finished logging | 19:35 |
| fungi | and i've confirmed i can't ping it | 19:36 |
| fungi | i guess those jobs should all get rescheduled? | 19:37 |
| clarkb | fungi: yes unless they had already failed two times previously for similar network related issues | 19:37 |
| *** harlowja has joined #openstack-infra | 19:38 | |
| *** bobh has quit IRC | 19:38 | |
| fungi | fair point, then they get a retry_limit | 19:38 |
| clarkb | actually in this case it will be the executor going away so maybe it jsuit works out regardless of previous failures | 19:39 |
| clarkb | since it can differentiate ebtween the two class of failure in the schedulure | 19:39 |
| clarkb | (I can't spell) | 19:39 |
| fungi | i can ping it again now | 19:39 |
| fungi | it's been rebooted | 19:40 |
| fungi | "...up 0 min..." | 19:40 |
| clarkb | fungi: ns2 is still spamming, related to that is stack at https://review.openstack.org/#/c/623041/1 care to review those? | 19:40 |
| *** jamesmcarthur has quit IRC | 19:41 | |
| fungi | clarkb: i only see ns2 reporting the same situation as ns1 now. it's notifying us there's an available update which will pull in a new dependency on upgrade so it's not acting on it automatically | 19:42 |
| fungi | i guess netplan.io is related to lxd though? | 19:43 |
| clarkb | oh is it? | 19:43 |
| clarkb | I looked itu p and its yaml based network config setup | 19:43 |
| clarkb | I could see how that would be useful to lxd | 19:43 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Update nsd systemd unit deps https://review.openstack.org/622620 | 19:44 |
| clarkb | that should fix a bug that testing caught | 19:44 |
| fungi | wow, the motd | 19:44 |
| *** jamesmcarthur has joined #openstack-infra | 19:44 | |
| fungi | i'd switch to debian if for no other reason than to no longer get subjected to advertising in our motds | 19:45 |
| *** jamesmcarthur has quit IRC | 19:45 | |
| mordred | clarkb: ok. I'm now in a position to help debug the image uploads ... where should I start? | 19:45 |
| fungi | clarkb: anyway, the cronspam for the past few days is basically letting us know that upgrading netplan.io will also install python3-netifaces which is not yet installed | 19:46 |
| *** wolverineav has quit IRC | 19:46 | |
| *** wolverineav has joined #openstack-infra | 19:47 | |
| clarkb | mordred: I pasted a paste link above why don't you digest that while I double check behavior hasn't changed since friday | 19:48 |
| fungi | corvus: heh, ze01 was apparently home to one of the missing mergers... our merger count bumped back up to 17 when it was rebooted | 19:48 |
| clarkb | fungi: if we merge the change to remove lxd apt will still want to udpate netplan.io right? its won't recursively remove netplan if that is where it is pulled in from | 19:48 |
| clarkb | (or will it?) | 19:48 |
| mordred | clarkb: reading | 19:48 |
| clarkb | mordred: I have confirmed it is still happening as of 19:18UTC today | 19:49 |
| fungi | clarkb: i was only speculating that netplan was installed as a dependency of lxd. i haven't looked at its list of deps to confirm | 19:49 |
| clarkb | (half an hour ago) | 19:49 |
| clarkb | fungi: ah | 19:49 |
| mordred | clarkb: uhm. | 19:49 |
| clarkb | fungi: https://packages.ubuntu.com/bionic/lxd doesn't list netplan | 19:49 |
| clarkb | mordred: yes its saving to saving :) | 19:49 |
| clarkb | mordred: its a fun one | 19:49 |
| mordred | oh - hrm. I have a thought ... | 19:50 |
| clarkb | mordred: where I got to was the image content upload call is what triggers that and that does not seem to provide any header/metadata info for the state | 19:50 |
| fungi | clarkb: agreed, i looked with apt show and didn't see it as a depends nor recommends | 19:50 |
| clarkb | mordred: the sdk native stuff might but we are still using the shade layer which doesn't | 19:50 |
| fungi | clarkb: it was more of a question as to why the change to stop installing lxd would silence this particular current cronspam on the authoritative nameservers | 19:51 |
| fungi | (i don't personally object to that stack of changes though and am reviewing anyway) | 19:51 |
| clarkb | fungi: not installing lxd would fix the lxd cronspam | 19:51 |
| mordred | clarkb: it'll almost certainly be related to the reorg of the image code we did in anticipation of using the lower-level stuff | 19:51 |
| clarkb | may also need to add netplan.io to that list | 19:52 |
| *** wolverineav has quit IRC | 19:52 | |
| fungi | clarkb: agreed, but it hasn't complained about lxd for days, ever since i uninstalled it | 19:52 |
| *** xek has quit IRC | 19:52 | |
| clarkb | fungi: yup, the ansible change is mostly a followup to that so that future hosts don't complain either | 19:52 |
| mordred | clarkb: my hunch is that we're half-way in - so the self.get_image() is returning an image with status['saving'] - and that we're running update_image on that to update the metadata | 19:52 |
| *** xek has joined #openstack-infra | 19:52 | |
| mordred | clarkb: when we should be waiting on the image to be in a ready state before moving forward with the update | 19:52 |
| clarkb | I can't help btu feel this is more evidence the multistep process is just broken | 19:53 |
| clarkb | that said I don't see where its calling those methods | 19:54 |
| clarkb | nodepool uses the shade layer which is doing a POST to create the image record then PUTting to it with the raw client stuff | 19:55 |
| clarkb | whcih si why I couldn't see where we might be adding bits in | 19:55 |
| clarkb | (the native sdk stuff definitely tries to be smarter about that stuff though) | 19:55 |
| mordred | clarkb: yeah - this will be cleaner to follow once we finish the transition | 19:56 |
| corvus | pabelanger, fungi, AJaeger: would you please re-evaluate 623605 based on my comments? | 19:56 |
| clarkb | fungi: "Netplan replaced ifupdown as the default configuration utility starting with Ubuntu 17.10 Artful." I don't think we want to remove it | 19:59 |
| clarkb | fungi: in this case probably best to just push it past that upgrade step and let it be happy? | 19:59 |
| *** diablo_rojo has quit IRC | 20:00 | |
| *** _alastor_ has quit IRC | 20:00 | |
| *** jamesmcarthur has joined #openstack-infra | 20:01 | |
| fungi | clarkb: yeah, just strange to see a package upgrade in an lts release suddenly pull in a new dependency | 20:01 |
| clarkb | fungi: https://code.launchpad.net/~usd-import-team/ubuntu/+source/netplan.io/+git/netplan.io/+ref/ubuntu/bionic-updates shows it being added at least | 20:03 |
| fungi | ahh, nplan is marked as a transitional package | 20:03 |
| fungi | so i guess they decided to rename it to netplan.io but still a strange choice within an already released lts | 20:04 |
| mordred | clarkb: oh - wait - the words I said above don't make 100% sense ... still poking | 20:05 |
| fungi | i guess this will clear up after the next bionic point release or whatever | 20:05 |
| clarkb | fungi: or even if $cloud updates its bionic image | 20:06 |
| clarkb | ? | 20:06 |
| clarkb | assuming the image has transitioned to the new name | 20:06 |
| fungi | yeah | 20:06 |
| mordred | clarkb: remote: https://review.openstack.org/624188 Don't try to upload to images in saving or queued state | 20:11 |
| clarkb | mordred: oh, ist trying to upload to an image from a previous upload attempt? | 20:13 |
| clarkb | mordred: won't that just result in not uploading any images then? | 20:14 |
| mordred | clarkb: yes, I believe that is what is going on - there is an image in saving state and it's trying to upload data to it | 20:14 |
| *** wolverineav has joined #openstack-infra | 20:14 | |
| clarkb | nothing should be uploading to that from nodepool but the thread that is failing (due to zk locks) | 20:14 |
| mordred | clarkb: I agree with you - unless the uploaders were restarted - or maybe an upload attempt was aborted halfway? | 20:15 |
| mordred | clarkb: cause there is nothing to prevent the sdk from trying to upload the file contents again even if openstack isn't in the right state for that | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| *** wolverineav has quit IRC | 20:19 | |
| fungi | #status log manually invoked `apt upgrade` on ns1 and ns2.opendev.org in order to silence cronspam about unattended-upgrades not upgrading netplan.io due to introducing a new dependency on python3-netifaces | 20:19 |
| clarkb | mordred: ya I think either of those cases are possible | 20:19 |
| openstackstatus | fungi: finished logging | 20:19 |
| clarkb | mordred: does nodepool need to try and delete that preexisting record of an image then start uploads? | 20:20 |
| openstackgerrit | Merged openstack-infra/project-config master: vexxhost: tweak nodepool settings https://review.openstack.org/623605 | 20:20 |
| spotz | fungi clarkb are either of you still about? | 20:20 |
| *** wolverineav has joined #openstack-infra | 20:20 | |
| fungi | yep | 20:20 |
| fungi | 'sup? | 20:20 |
| spotz | fungi - I go this Channel #openstack-ansible is linked to another channel and was thus disabled. | 20:21 |
| spotz | May just be me but with a lot of the team on vacation I'm wondering if y=our channel is broke as it's quiet except for system messages | 20:21 |
| openstackgerrit | Merged openstack-infra/system-config master: Don't install lxd on our servers https://review.openstack.org/623040 | 20:22 |
| fungi | spotz: i'm afraid i'm not following | 20:23 |
| fungi | spotz: did someone say this in the #openstack-ansible channel in the last few minutes or something? | 20:23 |
| spotz | fungi - I think the openstack-ansible channel is broken | 20:23 |
| openstackgerrit | Merged openstack-infra/system-config master: Configure packages on ubuntu arm servers https://review.openstack.org/623041 | 20:23 |
| spotz | That was in a status message from Freenode over the weekend | 20:23 |
| fungi | spotz: i see people talking in it today: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2018-12-10.log.html | 20:24 |
| spotz | fungi: Ok then I'm just broken, may need to re-register as I haven't seen anyone | 20:24 |
| fungi | discussion in there as of the past few minutes even | 20:24 |
| spotz | Ok just me then | 20:25 |
| fungi | let us know if you need help troubleshooting | 20:25 |
| spotz | fungi: Will do thanks:) | 20:25 |
| clarkb | mordred: as an alternative maybe always upload to a new image? | 20:25 |
| clarkb | mordred: in shade I mean. I'm not sure why shade would try to be optomizing that there | 20:26 |
| clarkb | mordred: is it when the names match? | 20:26 |
| *** lpetrut has quit IRC | 20:27 | |
| *** e0ne has joined #openstack-infra | 20:28 | |
| *** e0ne has quit IRC | 20:28 | |
| mordred | clarkb: yeah - this is when someone is asking shade to upload an image with a name that already exists | 20:30 |
| *** bobh has joined #openstack-infra | 20:30 | |
| mordred | clarkb: in general we try to make that a no-op when we can | 20:30 |
| mordred | so that if you're actually asking to re-upload an image that already exists with the same name, we don't actually attempt to upload the data a second time | 20:31 |
| *** jamesmcarthur has quit IRC | 20:31 | |
| clarkb | ya and if we upload a new image with a new record and uuid the two names will be ambiguous | 20:31 |
| *** jamesmcarthur has joined #openstack-infra | 20:33 | |
| clarkb | mordred: given your debugging I think I may try to manually deleting the "SAVING" broken images or would you rather we wait and make sure that whatever fixes go in fix it? | 20:33 |
| mordred | clarkb: no - go ahead and try manually deleting | 20:33 |
| clarkb | ok | 20:34 |
| mordred | clarkb: I *think* it's going to error on you - and that we might need to get a cloud admin to do something | 20:34 |
| mordred | clarkb: that said - we should at some point be attempting to upload a whole new image with an new name | 20:34 |
| clarkb | ya when we build a new dib image iirc | 20:35 |
| clarkb | there is likely we different failure mode we should watch for once I clean some of these up | 20:35 |
| mordred | yeah | 20:35 |
| *** bobh has quit IRC | 20:35 | |
| clarkb | mordred: I'm starting by deleting the centos7 images that are stuck/stale in inap | 20:37 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Modify job timeout query to exclude tempest-all/slow build_name https://review.openstack.org/623949 | 20:37 |
| clarkb | since centos 7.6 uploads not working is how we caught this | 20:37 |
| *** ahosam has joined #openstack-infra | 20:37 | |
| mordred | clarkb: awesome | 20:37 |
| clarkb | mordred: another frustrating thing is I think its not returning that error until after the PUT is complete | 20:38 |
| *** ahosam has quit IRC | 20:38 | |
| *** ahosam has joined #openstack-infra | 20:38 | |
| clarkb | so glance isn't verifying data until it has the entire transfer :/ | 20:38 |
| mordred | clarkb: of course | 20:38 |
| clarkb | ok the cleanup seems to have worked. I'll do it for the other images now too | 20:40 |
| clarkb | (then we wait and see if uploads give us a different error) | 20:40 |
| clarkb | thinking out loud a bit more too we may want to delete the old centos 7 images in inap that are active too. THis way osa stops having problems with 7.5 and 7.6 both being around (they use virtualenvs in bad ways and have discovered they are non protable) | 20:44 |
| ianw | fungi: hrm, i have to remember why we're on the ppa version | 20:52 |
| ianw | oh right, hwe kernel | 20:53 |
| ianw | clarkb: can you catch me up on the image issues? | 20:54 |
| clarkb | ianw: ya last week mnaser/osa noticed that we still had some centos 7.5 images floating around. After diggign in I Figured out that it was because inap images were not getting successful uploads so that one cloud region had 7.5 from when things were working | 20:56 |
| clarkb | ianw: mordred thinks the behavior is due to openstacksdk/shade trying to reupload to an image if it already exists and the hashes don't match | 20:56 |
| clarkb | except that glance won't let you update an image in a saving state. Also it doesn't check that until after the upload has completed (so likely at least one bug in glance too) | 20:56 |
| *** yamamoto has joined #openstack-infra | 20:57 | |
| clarkb | ianw: the problem is every time we build a new dib image we should try to upload to a new name and shade won't try to update the existing image record in that case. So I am cleaning out the stale records in the cloud now and we should see new errors hopefully | 20:57 |
| clarkb | the new error being the actual cause of the problem | 20:57 |
| ianw | ah ok, sounds good. I'll keep watching, LMN if i can help :) | 20:57 |
| *** diablo_rojo has joined #openstack-infra | 20:58 | |
| *** kjackal has quit IRC | 20:58 | |
| *** rockyg has joined #openstack-infra | 20:58 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Modify job timeout query to exclude tempest-all/slow build_name https://review.openstack.org/623949 | 21:00 |
| *** yamamoto has quit IRC | 21:01 | |
| *** gfidente|afk has quit IRC | 21:03 | |
| pabelanger | ianw: any idea why python3-virtualenv is missing on the fedora images? http://logs.openstack.org/02/623602/1/check/ansible-role-virtualenv-pkg-fedora-latest/8f80ecc/job-output.txt.gz#_2018-12-08_00_09_39_929607 Some how I think DIB is doing something here | 21:06 |
| *** jamesmcarthur has quit IRC | 21:08 | |
| *** eernst has joined #openstack-infra | 21:09 | |
| clarkb | how is it already 2100UTC | 21:09 |
| *** jamesmcarthur has joined #openstack-infra | 21:11 | |
| *** yboaron_ has joined #openstack-infra | 21:12 | |
| ianw | pabelanger: hrm ... no immediate thoughts | 21:13 |
| *** cgoncalves has joined #openstack-infra | 21:15 | |
| fungi | clarkb: i think i fell into a time warp today too | 21:16 |
| pabelanger | ianw: yah, something is blocking dnf, but haven't looked into it to hard. Thanks, I'll dig more | 21:16 |
| clarkb | mordred: ianw ok nb01 is uploading 4c5eee26-114e-4674-ae0a-c62ff7f46ab8 centos7 image now. It started at 2018-12-10 20:42:26,852 which according to irc logs is after I cleaned up the centos7 queued/saving images | 21:19 |
| ianw | pabelanger: oohhhh, then yes i know, we pin that | 21:20 |
| *** eernst has quit IRC | 21:21 | |
| ianw | pabelanger: here -> https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#n146 | 21:21 |
| pabelanger | ianw: oh, so excludes seems to remove them completely from package indexes. And dnf returns 404 | 21:23 |
| ianw | yeah, i think it pretends it doesn't exist, but since it's installed it resolves as a dependency | 21:24 |
| ianw | the only case this is a problem is when you try to install it directly, which seems to be exactly what you're doing :) | 21:24 |
| *** yboaron_ has quit IRC | 21:24 | |
| ianw | (i am under no illusions that this is not a big mess ...) | 21:25 |
| pabelanger | ianw: https://dnf-plugins-core.readthedocs.io/en/latest/versionlock.html might be a better way to fix that. reading up on it now | 21:25 |
| ianw | ++ ... first i've heard of that, but looks promising | 21:26 |
| *** kgiusti has left #openstack-infra | 21:27 | |
| *** efried_cya_jan has quit IRC | 21:29 | |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Ignore --find-links in requirements file https://review.openstack.org/597290 | 21:29 |
| mordred | clarkb: awesome. fingers crossed | 21:30 |
| *** eernst has joined #openstack-infra | 21:31 | |
| *** eernst has quit IRC | 21:32 | |
| *** jtomasek has quit IRC | 21:32 | |
| *** bobh has joined #openstack-infra | 21:36 | |
| *** wolverineav has quit IRC | 21:44 | |
| *** wolverineav has joined #openstack-infra | 21:46 | |
| *** jamesmcarthur has quit IRC | 21:52 | |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Ignore --find-links in requirements file https://review.openstack.org/597290 | 21:56 |
| clarkb | mordred: it failed with the same saving to saving 409 error | 21:57 |
| clarkb | its possible I didn't get the ordering right | 21:58 |
| clarkb | mordred: maybe we shoudl test with a manual upload with a completely different name and see if it has the same failure? | 21:58 |
| clarkb | I'm going to try that with openstackclient | 22:00 |
| clarkb | then if that works we can pretend to be nodepool | 22:00 |
| mordred | ok. if that's still failing - that's unhappy making | 22:01 |
| *** tpsilva has quit IRC | 22:02 | |
| clarkb | Error finding address for https://image.api.mtl01.cloud.iweb.com/v2/images/457f4cd0-780a-4d90-9848-70e044ab9292/file: (32, 'EPIPE') | 22:05 |
| clarkb | doesn't work with openstackclient either but fails different | 22:05 |
| *** _alastor_ has joined #openstack-infra | 22:06 | |
| clarkb | I can create an image with no file | 22:11 |
| clarkb | but then osc doesn't seem to have a way to upload to the file | 22:12 |
| clarkb | that appears to be an ssl error | 22:16 |
| mordred | grump | 22:19 |
| clarkb | apparently EPIPE means the remote closed the connection | 22:20 |
| clarkb | so ssl by way of using tls on top of tcp | 22:20 |
| clarkb | I'm trying on nb01 instead of bridge.o.o in case its a python3.6 issue | 22:22 |
| clarkb | same error though | 22:22 |
| clarkb | in that case osc likely not working then | 22:22 |
| clarkb | mgagne_: ^ not sure if you have noticed, but uploads using openstacksdk and osc to inap are currently failing (for what appear to be different reasons) | 22:23 |
| mgagne_ | what are the reasons? | 22:23 |
| mgagne_ | EPIPE ? | 22:24 |
| clarkb | mgagne_: glanceclient.exc.CommunicationError: Error finding address for https://image.api.mtl01.cloud.iweb.com/v2/images/0cfb3a49-2f51-4ac5-9653-1b9e918e324d/file: (32, 'EPIPE') is the osc reason | 22:24 |
| clarkb | openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://image.api.mtl01.cloud.iweb.com/v2/images/ec5fab20-4ff7-4286-b166-fb81f0fa0699/file, 409 Conflict: Image status transition from saving to saving is not allowed is the sdk reason | 22:24 |
| clarkb | there is a non zero chance it is bugs in both tools | 22:25 |
| clarkb | but maybe you are aware of something recent in the cloud that may explain that? like in the last 2 weeks | 22:25 |
| mgagne_ | have problems started 2 weeks ago or more recently? | 22:25 |
| clarkb | mgagne_: the last successful image from nodepool using sdk was 13 days ago. I only juist tried osc now | 22:26 |
| mgagne_ | clarkb: ok, let me look into it. | 22:27 |
| clarkb | thank you | 22:28 |
| fungi | sounds vaguely reminiscent of when one provider had some sort of proxy in front of the glance api endpoint and it was timing out while uploads were in progress | 22:32 |
| mgagne_ | fungi: I'm looking into timeout issues. I'm trying to figure out what part is timing out as several lbs are involved. | 22:34 |
| *** boden has quit IRC | 22:34 | |
| mgagne_ | I'm able to reproduce the issue so there is that. | 22:37 |
| *** efried has joined #openstack-infra | 22:39 | |
| *** diablo_rojo has quit IRC | 22:44 | |
| clarkb | corvus: do we want to try restarting the zuul scheduler today to pick up the queue organizational stuff? | 22:45 |
| corvus | clarkb: i think that would be a good idea, but i'm deep into learning some k8s right now and don't have a lot of attention. if you or others have some cycles, i think it's ready. | 22:46 |
| corvus | clarkb: if you feel like doing it, but something goes wrong, you can ping me and i can drop what i'm doing to help fix. | 22:47 |
| clarkb | corvus: ok, any idea if we can get away with a scheduelr only restart or if the executors should be restarted as well? | 22:47 |
| corvus | clarkb: scheduler-only should be fine | 22:48 |
| *** wolverineav has quit IRC | 22:49 | |
| clarkb | ok I'll get set up to do that here in a bit | 22:49 |
| clarkb | process is notify release team, dump queues, stop scheduler, start scheduler, restore queues | 22:50 |
| clarkb | I've asked the release team about it, if I don't hear back soon I'll assume they are all afk enjoying their evenings nad that it is safe to restart the scheduler now | 22:52 |
| clarkb | zuul==3.3.2.dev54 # git sha 6497fa3 is the version of zuul installed on zuul01 which appears to include corvus' change to specify group membership for relative priority | 22:55 |
| *** _alastor_ has quit IRC | 22:55 | |
| *** rcernin has joined #openstack-infra | 22:59 | |
| clarkb | There is a dragonflow change close to merging, I'll begin once that merges | 22:59 |
| mgagne_ | clarkb: I think I found the issue. | 23:00 |
| mgagne_ | I'm now able to upload an image in inap-mtl01 in my private account. | 23:01 |
| clarkb | mgagne_: great, I will retest with osc on our end once the zuul restart is compelted | 23:01 |
| clarkb | dragonflow change merged I'll proceed with zuul-scheduler restart now | 23:02 |
| *** diablo_rojo has joined #openstack-infra | 23:03 | |
| *** panda is now known as panda|off | 23:06 | |
| fungi | clarkb: should we plan a gerrit restart for the same time? | 23:08 |
| fungi | though https://review.openstack.org/471078 would be nice to get in at some point as well and needs a gerrit restart too | 23:09 |
| *** wolverineav has joined #openstack-infra | 23:10 | |
| clarkb | fungi: too late, zuul is already done :P or almost done, the web portion seems unhappy | 23:10 |
| fungi | no worries | 23:11 |
| fungi | i can probably swing a gerrit restart any time i think of it when not much is going on. it's not like those take long | 23:12 |
| clarkb | ok its back now | 23:12 |
| clarkb | reenqueuing changes now | 23:12 |
| clarkb | I needed to restart zuul-web but then stop zuul-web didn't clean out its pid file so I had to rm the pid file, stop zuul web again (because systemd?) then start it | 23:13 |
| clarkb | #status log Restarted Zuul scheduler to pick up changes to how projects are grouped into relative priority queues. | 23:14 |
| openstackstatus | clarkb: finished logging | 23:14 |
| clarkb | mriedem: dansmith ^ fyi I'm hoping that allows us to put itno place some of the ideas from your feedback | 23:15 |
| dansmith | cool | 23:15 |
| EmilienM | gerrit is slow for me only? (take 20s to push a patch) | 23:16 |
| EmilienM | you must have firewall rules against Quebec IPs, I'm sure :-P it's always just me | 23:17 |
| clarkb | EmilienM: its possible that zuul restart hits it pretty hard | 23:18 |
| *** dave-mccowan has quit IRC | 23:19 | |
| clarkb | mgagne_: my image upload isn't failing as quickly as before | 23:19 |
| clarkb | with osc | 23:19 |
| mgagne_ | clarkb: but is it still failing? | 23:19 |
| *** slaweq has quit IRC | 23:19 | |
| clarkb | mgagne_: no sorry, it hasn't finished yet | 23:19 |
| mgagne_ | cool | 23:20 |
| clarkb | oh it just finished successfully so osc is working now I think | 23:20 |
| mgagne_ | :D | 23:20 |
| clarkb | I'll watch nodepool to see if it improves too | 23:20 |
| clarkb | EmilienM: there is a spike in gerrit cpu usage that roughly correlates to when I restarted zuul | 23:22 |
| clarkb | I expect that now that zuul is started this will taper off | 23:22 |
| EmilienM | clarkb: thanks | 23:22 |
| *** ahosam has quit IRC | 23:26 | |
| *** shardy has quit IRC | 23:29 | |
| *** diablo_rojo has quit IRC | 23:30 | |
| clarkb | I've not noticed any weird zuul behavior yet and we have a change in post which implies we've merged stuff | 23:37 |
| clarkb | I think the next step is to set queue: values on various check queues for projects | 23:38 |
| *** jamesmcarthur has joined #openstack-infra | 23:39 | |
| *** bobh has quit IRC | 23:39 | |
| clarkb | remote: https://review.openstack.org/624246 | 23:42 |
| *** bobh has joined #openstack-infra | 23:43 | |
| clarkb | corvus: ^ any idea if it should work out of a project-template like that? I seem to recall that may have caused problems in the past (but is easier than updating every project) | 23:43 |
| *** jamesmcarthur has quit IRC | 23:43 | |
| corvus | clarkb: should be fine | 23:44 |
| *** yamamoto has joined #openstack-infra | 23:45 | |
| pabelanger | clarkb: if we roll back or disable the feature for priority, will 624246 just be noop, or does that also need to be reverted? | 23:51 |
| clarkb | pabelanger: good question I think zuul may treat it as a config error? but unsure | 23:52 |
| pabelanger | kk | 23:53 |
| *** bobh has quit IRC | 23:53 | |
| *** mriedem is now known as mriedem_away | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!