| corvus | pabelanger, clarkb: the log entries suggest that we're going through the code path which is: generate a stage 1 layout, then return item.item_ahead.layout | 00:00 |
|---|---|---|
| clarkb | corvus: yes it should be in the scheduler too (was part of the fix tosky problem with host vars) | 00:00 |
| corvus | so i'm thinking maybe that's None, which is why it's looping | 00:00 |
| corvus | clarkb, pabelanger: this path: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/manager/__init__.py#n450 | 00:00 |
| *** yamamoto has quit IRC | 00:00 | |
| clarkb | oh interesting | 00:01 |
| *** bobh has joined #openstack-infra | 00:02 | |
| corvus | i think maybe now that needs to walk up the queue until it hits a layout or if none, return the pipeline layout. | 00:02 |
| *** dingyichen has joined #openstack-infra | 00:03 | |
| corvus | and this should be reproducible in a unit test by having change A not be a config change, then change B is a change to a config-project that depends on change A. | 00:03 |
| corvus | and this should be reproducible in a unit test by having change A not be a config change, then change B is a zuul.yaml change to a config-project that depends on change A. | 00:03 |
| corvus | corrected ^ | 00:03 |
| *** armaan has quit IRC | 00:03 | |
| clarkb | and run in check since dependentPipeline behavior would add an item.item_ahead.layout | 00:03 |
| *** aeng has quit IRC | 00:03 | |
| corvus | clarkb: yep. a has to be non-live. | 00:04 |
| *** armaan has joined #openstack-infra | 00:04 | |
| clarkb | corvus: might even be as simple as if item.item_ahead and item.item_ahead.layout: for that condition | 00:05 |
| *** armaan has quit IRC | 00:05 | |
| *** armaan has joined #openstack-infra | 00:05 | |
| corvus | clarkb: yes.... though that feels slightly less robust -- like it's depending on a nuance of implementation. maybe it's okay though... | 00:07 |
| pabelanger | corvus: clarkb: is this a regression do to recent changes or just something we haven't seen before? | 00:10 |
| pabelanger | I'm going to have to duck out to pick up family member at airport shortly, I can try working a patch for the morning, but might need more help to properly understand | 00:11 |
| *** Swami has quit IRC | 00:11 | |
| clarkb | pabelanger: new regression I think | 00:12 |
| clarkb | I'm currently working on making a test to reproduce the issue | 00:13 |
| pabelanger | ack | 00:15 |
| *** mriedem has quit IRC | 00:18 | |
| *** dizquierdo has joined #openstack-infra | 00:18 | |
| *** aeng has joined #openstack-infra | 00:21 | |
| *** markvoelker has quit IRC | 00:25 | |
| *** dhajare has joined #openstack-infra | 00:26 | |
| *** bobh has quit IRC | 00:28 | |
| *** kien-ha has joined #openstack-infra | 00:31 | |
| *** dhajare has quit IRC | 00:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-latest nodeset and ozj jobs https://review.openstack.org/565617 | 00:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: DNM - testing https://review.openstack.org/565618 | 00:35 |
| *** jamesmcarthur has quit IRC | 00:36 | |
| clarkb | I think I'm close | 00:37 |
| clarkb | I've reproduced the behavior now tidying up the test and working on the fix | 00:37 |
| *** kiennt26 has joined #openstack-infra | 00:45 | |
| *** dingyichen has quit IRC | 00:49 | |
| *** dingyichen has joined #openstack-infra | 00:49 | |
| *** zhangfei has joined #openstack-infra | 00:52 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix layout generation for trusted projects https://review.openstack.org/565624 | 00:54 |
| clarkb | pabelanger: corvus ^ I think that both tests it and fixes it | 00:54 |
| *** yamamoto has joined #openstack-infra | 00:57 | |
| *** manjeets has quit IRC | 01:00 | |
| *** yamamoto has quit IRC | 01:03 | |
| *** kien-ha has quit IRC | 01:05 | |
| *** bobh has joined #openstack-infra | 01:07 | |
| *** dizquierdo has quit IRC | 01:07 | |
| *** jamesmcarthur has joined #openstack-infra | 01:09 | |
| *** smarcet has joined #openstack-infra | 01:10 | |
| *** dpawlik has joined #openstack-infra | 01:11 | |
| *** dpawlik has quit IRC | 01:15 | |
| *** hongbin has joined #openstack-infra | 01:17 | |
| *** rlandy has quit IRC | 01:31 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: DNM - testing https://review.openstack.org/565618 | 01:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Move ubuntu-bionic into gate https://review.openstack.org/565631 | 01:32 |
| *** zhangfei has quit IRC | 01:35 | |
| pabelanger | ianw: ^fedora-latest change for devstack, lets see what happens | 01:39 |
| *** bobh has quit IRC | 01:41 | |
| *** hwoarang has quit IRC | 01:45 | |
| *** hwoarang has joined #openstack-infra | 01:47 | |
| *** zhangfei has joined #openstack-infra | 01:47 | |
| *** felipemonteiro has joined #openstack-infra | 01:50 | |
| *** bobh has joined #openstack-infra | 01:52 | |
| pabelanger | primary | ERROR: No viable v4 or v6 route found to git.openstack.org. The build node is assumed to be invalid. | 01:52 |
| ianw | pabelanger: cool, will watch. just rebooted into f28 on this box :) | 01:53 |
| pabelanger | seems to happen on fedora pretty regularly | 01:53 |
| pabelanger | ianw: cool, I'll likely update laptop in the next few days | 01:53 |
| *** felipemonteiro has quit IRC | 01:56 | |
| *** dklyle has quit IRC | 01:57 | |
| *** david-lyle has joined #openstack-infra | 01:57 | |
| *** yamamoto has joined #openstack-infra | 01:59 | |
| *** bobh has quit IRC | 02:01 | |
| *** yamahata has quit IRC | 02:02 | |
| *** aeng has quit IRC | 02:03 | |
| *** aeng has joined #openstack-infra | 02:03 | |
| *** yamamoto has quit IRC | 02:03 | |
| *** dchen has joined #openstack-infra | 02:04 | |
| EmilienM | corvus: thanks for the 2 patches | 02:04 |
| *** adrianreza has joined #openstack-infra | 02:05 | |
| *** yamamoto has joined #openstack-infra | 02:05 | |
| *** dingyichen has quit IRC | 02:06 | |
| *** masber has joined #openstack-infra | 02:09 | |
| *** yamamoto has quit IRC | 02:09 | |
| *** spsurya has joined #openstack-infra | 02:10 | |
| *** tinwood has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-infra | 02:11 | |
| *** dave-mccowan has quit IRC | 02:13 | |
| *** dingyichen has joined #openstack-infra | 02:16 | |
| *** psachin has joined #openstack-infra | 02:16 | |
| *** rwsu has joined #openstack-infra | 02:17 | |
| *** gyee has quit IRC | 02:17 | |
| *** yamamoto has joined #openstack-infra | 02:20 | |
| *** rh-jelabarre has quit IRC | 02:21 | |
| *** dpawlik has joined #openstack-infra | 02:25 | |
| *** mikal_ has quit IRC | 02:26 | |
| *** mikal has joined #openstack-infra | 02:26 | |
| *** markvoelker has joined #openstack-infra | 02:26 | |
| *** felipemonteiro has joined #openstack-infra | 02:29 | |
| *** dpawlik has quit IRC | 02:29 | |
| *** VW has joined #openstack-infra | 02:34 | |
| *** felipemonteiro has quit IRC | 02:37 | |
| *** mikal has quit IRC | 02:41 | |
| *** david-lyle has quit IRC | 02:45 | |
| *** mikal has joined #openstack-infra | 02:49 | |
| *** masuberu has joined #openstack-infra | 02:52 | |
| *** masber has quit IRC | 02:52 | |
| *** zhangfei has quit IRC | 02:55 | |
| *** markvoelker has quit IRC | 02:56 | |
| *** edisonxiang has joined #openstack-infra | 02:56 | |
| *** markvoelker has joined #openstack-infra | 02:57 | |
| *** armaan has quit IRC | 03:03 | |
| *** armaan_ has joined #openstack-infra | 03:03 | |
| edisonxiang | Hi @mordred, I try to get openstack flavor fatcts by ansible openstack module. | 03:07 |
| edisonxiang | http://docs.ansible.com/ansible/latest/modules/os_flavor_facts_module.html#os-flavor-facts-module | 03:07 |
| edisonxiang | I have configured OpenStack env, but get a TypeError: 'NoneType' object is not callable. | 03:07 |
| edisonxiang | This is the error logs. https://etherpad.openstack.org/p/edisonxiang | 03:07 |
| edisonxiang | Could you give some suggestions about this? Thanks. | 03:07 |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** zhangfei has joined #openstack-infra | 03:12 | |
| ianw | pabelanger: hmm, only in multinode? it's weird that both failed same way at once | 03:12 |
| *** namnh has joined #openstack-infra | 03:14 | |
| *** slaweq has quit IRC | 03:16 | |
| *** hongbin has quit IRC | 03:19 | |
| *** eernst has joined #openstack-infra | 03:21 | |
| *** aeng has quit IRC | 03:22 | |
| *** nicolasbock has quit IRC | 03:31 | |
| *** VW has quit IRC | 03:31 | |
| *** dchen has quit IRC | 03:33 | |
| *** vivsoni_ has quit IRC | 03:33 | |
| *** dchen has joined #openstack-infra | 03:34 | |
| *** slaweq has joined #openstack-infra | 03:35 | |
| *** armaan_ has quit IRC | 03:39 | |
| *** hamzy has joined #openstack-infra | 03:40 | |
| *** kiennt26 has quit IRC | 03:41 | |
| *** edisonxiang has quit IRC | 03:46 | |
| *** edisonxiang has joined #openstack-infra | 03:46 | |
| *** smarcet has quit IRC | 03:48 | |
| *** slaweq has quit IRC | 03:49 | |
| *** slaweq has joined #openstack-infra | 03:50 | |
| *** slaweq has quit IRC | 03:55 | |
| *** vivsoni has joined #openstack-infra | 03:55 | |
| *** slaweq has joined #openstack-infra | 03:55 | |
| *** slaweq has quit IRC | 03:59 | |
| *** ykarel|away has joined #openstack-infra | 04:01 | |
| *** vivsoni has quit IRC | 04:01 | |
| *** slaweq has joined #openstack-infra | 04:05 | |
| *** slaweq has quit IRC | 04:10 | |
| *** vivsoni has joined #openstack-infra | 04:10 | |
| *** slaweq has joined #openstack-infra | 04:15 | |
| *** slaweq has quit IRC | 04:20 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix setting a change queue in a template https://review.openstack.org/565581 | 04:20 |
| *** slaweq has joined #openstack-infra | 04:20 | |
| *** _mwarad_ has joined #openstack-infra | 04:22 | |
| *** _mwarad_ has quit IRC | 04:22 | |
| *** armaan has joined #openstack-infra | 04:23 | |
| *** janki has joined #openstack-infra | 04:29 | |
| *** slaweq has quit IRC | 04:30 | |
| *** udesale has joined #openstack-infra | 04:33 | |
| *** ianychoi has joined #openstack-infra | 04:34 | |
| *** slaweq has joined #openstack-infra | 04:35 | |
| *** kjackal has joined #openstack-infra | 04:37 | |
| *** kjackal_ has joined #openstack-infra | 04:37 | |
| *** masuberu has quit IRC | 04:47 | |
| *** links has joined #openstack-infra | 04:50 | |
| *** slaweq has quit IRC | 05:00 | |
| *** zeus has quit IRC | 05:04 | |
| *** bhavik1 has joined #openstack-infra | 05:05 | |
| *** slaweq has joined #openstack-infra | 05:05 | |
| *** dchen has quit IRC | 05:06 | |
| *** dchen has joined #openstack-infra | 05:06 | |
| *** Guest81711 has quit IRC | 05:15 | |
| *** slaweq has quit IRC | 05:15 | |
| *** yamahata has joined #openstack-infra | 05:18 | |
| *** mikal_ has joined #openstack-infra | 05:20 | |
| *** mikal has quit IRC | 05:22 | |
| *** bhavik1 has quit IRC | 05:25 | |
| *** iranzo has joined #openstack-infra | 05:26 | |
| *** kjackal__ has joined #openstack-infra | 05:30 | |
| *** kjackal___ has joined #openstack-infra | 05:30 | |
| *** kjackal has quit IRC | 05:30 | |
| *** kjackal_ has quit IRC | 05:30 | |
| *** pgadiya has joined #openstack-infra | 05:32 | |
| *** pgadiya has quit IRC | 05:32 | |
| *** dhajare has joined #openstack-infra | 05:36 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add start and end timestamp to task and play result in zuul_json callback https://review.openstack.org/563888 | 05:38 |
| *** xinliang3 has quit IRC | 05:42 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix regex project templates https://review.openstack.org/565584 | 05:44 |
| *** bhujay has joined #openstack-infra | 05:46 | |
| *** dklyle has joined #openstack-infra | 05:47 | |
| *** eernst has quit IRC | 05:49 | |
| *** udesale_ has joined #openstack-infra | 06:03 | |
| *** udesale_ has quit IRC | 06:04 | |
| *** udesale_ has joined #openstack-infra | 06:05 | |
| *** udesale has quit IRC | 06:05 | |
| *** ykarel has joined #openstack-infra | 06:06 | |
| *** jaosorior has joined #openstack-infra | 06:07 | |
| *** ykarel|away has quit IRC | 06:08 | |
| *** udesale_ has quit IRC | 06:09 | |
| *** jtomasek has joined #openstack-infra | 06:14 | |
| *** armaan has quit IRC | 06:20 | |
| *** armaan has joined #openstack-infra | 06:21 | |
| *** dklyle has quit IRC | 06:29 | |
| *** pcichy has joined #openstack-infra | 06:29 | |
| *** kjackal___ has quit IRC | 06:31 | |
| *** kjackal__ has quit IRC | 06:31 | |
| *** edisonxiang has quit IRC | 06:32 | |
| *** xinliang3 has joined #openstack-infra | 06:32 | |
| *** 21WAAG9JA has joined #openstack-infra | 06:37 | |
| *** 17SAA3R6Z has joined #openstack-infra | 06:37 | |
| *** pcichy has quit IRC | 06:39 | |
| *** jaosorior has quit IRC | 06:41 | |
| *** pcichy has joined #openstack-infra | 06:41 | |
| *** sshnaidm|afk is now known as sshnaidm | 06:41 | |
| *** udesale has joined #openstack-infra | 06:44 | |
| *** ykarel_ has joined #openstack-infra | 06:44 | |
| *** bhavik1 has joined #openstack-infra | 06:44 | |
| *** jtomasek has quit IRC | 06:45 | |
| *** ykarel has quit IRC | 06:46 | |
| *** jtomasek has joined #openstack-infra | 06:46 | |
| *** annp has joined #openstack-infra | 06:48 | |
| *** bhavik1 has quit IRC | 06:50 | |
| *** rcernin_ has joined #openstack-infra | 06:50 | |
| *** rcernin has quit IRC | 06:51 | |
| openstackgerrit | Merged openstack-infra/release-tools master: stop if announce.sh fails to build the virtualenv https://review.openstack.org/564318 | 06:52 |
| *** ramishra has joined #openstack-infra | 06:54 | |
| *** MasterofJOKers_ has joined #openstack-infra | 06:55 | |
| openstackgerrit | Merged openstack-infra/release-tools master: finish moving announce.sh to releases repo by deleting it https://review.openstack.org/564559 | 06:55 |
| *** MasterofJOKers has quit IRC | 06:57 | |
| *** MasterofJOKers_ is now known as MasterofJOKers | 06:57 | |
| *** slaweq has joined #openstack-infra | 06:58 | |
| *** slaweq has quit IRC | 06:58 | |
| *** slaweq has joined #openstack-infra | 06:58 | |
| *** slaweq has quit IRC | 06:58 | |
| *** slaweq has joined #openstack-infra | 06:58 | |
| *** arxcruz|pto is now known as arxcruz | 07:00 | |
| *** yamahata has quit IRC | 07:02 | |
| *** rcernin_ has quit IRC | 07:05 | |
| *** dpawlik has joined #openstack-infra | 07:06 | |
| *** dpawlik has quit IRC | 07:11 | |
| *** armaan has quit IRC | 07:12 | |
| *** armaan has joined #openstack-infra | 07:13 | |
| *** pblaho has joined #openstack-infra | 07:13 | |
| *** 21WAAG9JA has quit IRC | 07:13 | |
| *** hwoarang has quit IRC | 07:14 | |
| *** hwoarang has joined #openstack-infra | 07:14 | |
| *** hwoarang has quit IRC | 07:14 | |
| *** 17SAA3R6Z has quit IRC | 07:14 | |
| *** yamahata has joined #openstack-infra | 07:15 | |
| *** dpawlik has joined #openstack-infra | 07:15 | |
| *** hwoarang has joined #openstack-infra | 07:16 | |
| *** hwoarang has quit IRC | 07:16 | |
| *** hwoarang has joined #openstack-infra | 07:16 | |
| *** armaan has quit IRC | 07:18 | |
| *** armaan has joined #openstack-infra | 07:18 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 07:20 |
| *** tesseract has joined #openstack-infra | 07:25 | |
| *** armaan has quit IRC | 07:25 | |
| *** kjackal has joined #openstack-infra | 07:25 | |
| *** armaan has joined #openstack-infra | 07:26 | |
| *** kjackal_ has joined #openstack-infra | 07:26 | |
| *** armaan has quit IRC | 07:27 | |
| *** e0ne has joined #openstack-infra | 07:27 | |
| *** armaan has joined #openstack-infra | 07:28 | |
| *** armaan has quit IRC | 07:33 | |
| *** kjackal_ has quit IRC | 07:33 | |
| *** kjackal_ has joined #openstack-infra | 07:33 | |
| *** kjackal has quit IRC | 07:33 | |
| *** kjackal__ has joined #openstack-infra | 07:33 | |
| *** dklyle has joined #openstack-infra | 07:34 | |
| *** armaan has joined #openstack-infra | 07:34 | |
| *** kjackal_ has quit IRC | 07:39 | |
| *** kjackal__ has quit IRC | 07:39 | |
| *** Hal has joined #openstack-infra | 07:40 | |
| *** Hal is now known as Guest48578 | 07:40 | |
| *** armaan has quit IRC | 07:41 | |
| *** rossella_s has joined #openstack-infra | 07:46 | |
| *** ccamacho has joined #openstack-infra | 07:48 | |
| *** rossella_s has quit IRC | 07:51 | |
| *** florianf has joined #openstack-infra | 07:53 | |
| *** hemna_ has quit IRC | 07:56 | |
| *** zhangfei has quit IRC | 08:01 | |
| *** bhujay has quit IRC | 08:05 | |
| *** hashar has joined #openstack-infra | 08:05 | |
| *** jpich has joined #openstack-infra | 08:06 | |
| *** bhujay has joined #openstack-infra | 08:08 | |
| *** ssbarnea_ has joined #openstack-infra | 08:08 | |
| *** pcaruana has joined #openstack-infra | 08:08 | |
| *** ykarel_ is now known as ykarel|lunch | 08:08 | |
| *** andreas_s has joined #openstack-infra | 08:11 | |
| *** salv-orlando has joined #openstack-infra | 08:12 | |
| *** bhujay has quit IRC | 08:13 | |
| *** zhangfei has joined #openstack-infra | 08:15 | |
| *** yamahata has quit IRC | 08:16 | |
| *** kbaikov[m] has quit IRC | 08:18 | |
| *** lucas-afk is now known as lucasagomes | 08:19 | |
| *** kbaikov[m] has joined #openstack-infra | 08:24 | |
| *** shardy has joined #openstack-infra | 08:27 | |
| *** bhujay has joined #openstack-infra | 08:32 | |
| *** shardy has quit IRC | 08:37 | |
| *** shardy has joined #openstack-infra | 08:38 | |
| *** derekh has joined #openstack-infra | 08:39 | |
| *** xinliang has quit IRC | 08:42 | |
| *** xinliang has joined #openstack-infra | 08:42 | |
| *** bhujay has quit IRC | 08:43 | |
| *** sree_ has joined #openstack-infra | 08:46 | |
| *** sree_ is now known as Guest67087 | 08:46 | |
| *** ykarel|lunch is now known as ykarel | 08:46 | |
| *** salv-orlando has quit IRC | 08:47 | |
| *** dbecker has joined #openstack-infra | 08:47 | |
| *** salv-orlando has joined #openstack-infra | 08:48 | |
| *** namnh has quit IRC | 08:50 | |
| *** namnh has joined #openstack-infra | 08:51 | |
| *** salv-orlando has quit IRC | 08:52 | |
| *** dklyle has quit IRC | 08:56 | |
| *** dchen has quit IRC | 08:58 | |
| *** rossella_s has joined #openstack-infra | 08:58 | |
| *** dingyichen has quit IRC | 08:58 | |
| *** hs366 has joined #openstack-infra | 08:59 | |
| *** dklyle has joined #openstack-infra | 09:15 | |
| *** panda|rover|off is now known as panda|rover | 09:17 | |
| *** armaan has joined #openstack-infra | 09:23 | |
| *** vtapia has quit IRC | 09:24 | |
| *** dklyle has quit IRC | 09:26 | |
| *** dhill_ has quit IRC | 09:38 | |
| *** armaan has quit IRC | 09:43 | |
| *** dtantsur|afk is now known as dtantsur | 09:50 | |
| *** namnh has quit IRC | 09:55 | |
| *** pblaho has quit IRC | 09:56 | |
| *** namnh has joined #openstack-infra | 09:56 | |
| *** pblaho has joined #openstack-infra | 09:56 | |
| *** rossella_s has quit IRC | 09:58 | |
| *** namnh has quit IRC | 09:59 | |
| *** andreas_s has quit IRC | 09:59 | |
| *** rossella_s has joined #openstack-infra | 10:00 | |
| *** vtapia has joined #openstack-infra | 10:02 | |
| *** BobBall has joined #openstack-infra | 10:02 | |
| *** armaan has joined #openstack-infra | 10:03 | |
| *** andreas_s has joined #openstack-infra | 10:09 | |
| *** andreas_s_ has joined #openstack-infra | 10:11 | |
| *** andreas_s has quit IRC | 10:14 | |
| *** salv-orlando has joined #openstack-infra | 10:15 | |
| *** andreas_s_ has quit IRC | 10:15 | |
| *** vaidy has quit IRC | 10:17 | |
| *** isviridov_away has quit IRC | 10:17 | |
| *** andreas_s has joined #openstack-infra | 10:19 | |
| *** andreas_s has quit IRC | 10:19 | |
| *** andreas_s has joined #openstack-infra | 10:19 | |
| *** armaan has quit IRC | 10:21 | |
| *** andreas_s has quit IRC | 10:21 | |
| *** andreas_s has joined #openstack-infra | 10:22 | |
| *** salv-orlando has quit IRC | 10:22 | |
| *** andreas_s has quit IRC | 10:23 | |
| *** andreas_s has joined #openstack-infra | 10:25 | |
| *** mikal has joined #openstack-infra | 10:32 | |
| *** sree_ has joined #openstack-infra | 10:32 | |
| *** sree_ is now known as Guest15027 | 10:32 | |
| *** mikal_ has quit IRC | 10:33 | |
| *** Guest67087 has quit IRC | 10:35 | |
| *** annp has quit IRC | 10:37 | |
| *** nicolasbock has joined #openstack-infra | 10:42 | |
| *** stephenfin has quit IRC | 10:47 | |
| *** stephenfin has joined #openstack-infra | 10:51 | |
| *** wolverineav has joined #openstack-infra | 10:54 | |
| *** andreas_s has quit IRC | 11:00 | |
| *** andreas_s has joined #openstack-infra | 11:02 | |
| *** r-daneel has quit IRC | 11:04 | |
| *** andreas_s has quit IRC | 11:07 | |
| *** xinliang has quit IRC | 11:08 | |
| *** panda|rover is now known as panda|rover|lunc | 11:08 | |
| *** jaosorior has joined #openstack-infra | 11:08 | |
| *** r-daneel has joined #openstack-infra | 11:11 | |
| *** andreas_s has joined #openstack-infra | 11:15 | |
| *** jaosorior has quit IRC | 11:16 | |
| *** andreas_s_ has joined #openstack-infra | 11:19 | |
| *** janki has quit IRC | 11:19 | |
| *** andreas_s has quit IRC | 11:20 | |
| *** andreas_s_ has quit IRC | 11:20 | |
| *** armaan has joined #openstack-infra | 11:20 | |
| *** andreas_s has joined #openstack-infra | 11:20 | |
| *** vaidy has joined #openstack-infra | 11:21 | |
| *** isviridov_away has joined #openstack-infra | 11:22 | |
| *** dklyle has joined #openstack-infra | 11:25 | |
| *** ldnunes has joined #openstack-infra | 11:28 | |
| *** andreas_s has quit IRC | 11:29 | |
| *** zhangfei has quit IRC | 11:29 | |
| *** udesale has quit IRC | 11:30 | |
| *** andreas_s has joined #openstack-infra | 11:30 | |
| *** lucasagomes is now known as lucas-hungry | 11:32 | |
| *** rossella_s has quit IRC | 11:38 | |
| *** rossella_s has joined #openstack-infra | 11:38 | |
| *** bhujay has joined #openstack-infra | 11:39 | |
| *** rh-jelabarre has joined #openstack-infra | 11:40 | |
| *** tpsilva has joined #openstack-infra | 11:45 | |
| *** salv-orlando has joined #openstack-infra | 11:46 | |
| *** ihar has quit IRC | 11:46 | |
| *** rossella_s has quit IRC | 11:47 | |
| *** ihar has joined #openstack-infra | 11:48 | |
| *** dklyle has quit IRC | 11:50 | |
| *** rossella_s has joined #openstack-infra | 11:52 | |
| *** salv-orlando has quit IRC | 11:52 | |
| *** salv-orlando has joined #openstack-infra | 11:53 | |
| *** Guest15027 has quit IRC | 11:53 | |
| *** sree has joined #openstack-infra | 11:54 | |
| *** panda|rover|lunc is now known as panda|rover | 11:54 | |
| *** ekan has joined #openstack-infra | 11:55 | |
| *** andreas_s has quit IRC | 11:56 | |
| *** salv-orlando has quit IRC | 11:57 | |
| *** andreas_s has joined #openstack-infra | 11:57 | |
| *** sree has quit IRC | 11:59 | |
| *** andreas_s has quit IRC | 11:59 | |
| *** andreas_s has joined #openstack-infra | 12:00 | |
| *** bhujay has quit IRC | 12:00 | |
| *** bhujay has joined #openstack-infra | 12:02 | |
| *** pbourke has joined #openstack-infra | 12:04 | |
| *** ekan has quit IRC | 12:04 | |
| *** ekan has joined #openstack-infra | 12:05 | |
| *** jamesmcarthur has quit IRC | 12:06 | |
| *** jamesmcarthur has joined #openstack-infra | 12:06 | |
| *** bhujay has quit IRC | 12:08 | |
| *** edmondsw has joined #openstack-infra | 12:13 | |
| *** edmondsw has quit IRC | 12:13 | |
| *** rfolco has joined #openstack-infra | 12:13 | |
| *** edmondsw has joined #openstack-infra | 12:13 | |
| *** andreas_s has quit IRC | 12:14 | |
| *** andreas_s has joined #openstack-infra | 12:16 | |
| *** kgiusti has joined #openstack-infra | 12:16 | |
| *** rfolco has quit IRC | 12:17 | |
| *** dtantsur is now known as dtantsur|brb | 12:23 | |
| *** eharney has quit IRC | 12:24 | |
| *** yamamoto has quit IRC | 12:25 | |
| *** yamamoto has joined #openstack-infra | 12:25 | |
| *** jamesmcarthur has quit IRC | 12:26 | |
| *** andreas_s has quit IRC | 12:26 | |
| *** andreas_s_ has joined #openstack-infra | 12:27 | |
| frickler | pabelanger: I debugged the devstack-fedora-latest failure. dib set the fedora-updates mirror to http://mirror.gra1.ovh.openstack.org/fedora/updates/$releasever/$basearch/ which compared what is on the server is missing an "Everything" as component | 12:28 |
| *** alex_xu has quit IRC | 12:29 | |
| frickler | pabelanger: not sure how to fix this though without breaking fedora-27 for which this still seems to be correct | 12:29 |
| *** andreas__ has joined #openstack-infra | 12:29 | |
| frickler | probably needs case-switching here http://git.openstack.org/cgit/openstack/diskimage-builder/tree/contrib/setup-gate-mirrors.sh#n41 | 12:30 |
| *** pcaruana has quit IRC | 12:30 | |
| *** rlandy has joined #openstack-infra | 12:32 | |
| *** andreas_s_ has quit IRC | 12:32 | |
| *** trown|outtypewww is now known as trown | 12:33 | |
| *** janki has joined #openstack-infra | 12:35 | |
| *** alex_xu has joined #openstack-infra | 12:36 | |
| *** auristor has quit IRC | 12:37 | |
| *** auristor has joined #openstack-infra | 12:42 | |
| *** esarault has joined #openstack-infra | 12:42 | |
| *** salv-orlando has joined #openstack-infra | 12:43 | |
| *** armaan has quit IRC | 12:44 | |
| *** lucas-hungry is now known as lucasagomes | 12:44 | |
| *** salv-orl_ has joined #openstack-infra | 12:45 | |
| *** florianf has quit IRC | 12:46 | |
| *** salv-orlando has quit IRC | 12:46 | |
| *** florianf has joined #openstack-infra | 12:46 | |
| *** psachin has quit IRC | 12:47 | |
| *** andreas__ has quit IRC | 12:48 | |
| *** jamesmcarthur has joined #openstack-infra | 12:49 | |
| *** andreas_s has joined #openstack-infra | 12:53 | |
| *** dhill_ has joined #openstack-infra | 12:53 | |
| *** jcoufal has joined #openstack-infra | 12:55 | |
| *** andreas_s has quit IRC | 12:56 | |
| *** andreas_s has joined #openstack-infra | 12:57 | |
| *** andreas_s has quit IRC | 12:57 | |
| *** andreas_s has joined #openstack-infra | 12:59 | |
| pabelanger | frickler: where did you see a failure? | 13:00 |
| *** jamesmcarthur has quit IRC | 13:00 | |
| *** jamesmcarthur has joined #openstack-infra | 13:00 | |
| *** jamesmcarthur has quit IRC | 13:00 | |
| *** jamesmcarthur has joined #openstack-infra | 13:01 | |
| pabelanger | frickler: I think the dib change will be needed for fedora-28 testing, but shouldn't affect nodepool DIB images, as we don't use AFS mirrors for builds | 13:01 |
| frickler | pabelanger: in the recheck I did, which still hasn't finished other jobs yet. http://logs.openstack.org/32/565632/1/check/devstack-platform-fedora-latest/32bdb73/ I also held that node and checked directly | 13:01 |
| pabelanger | so, we likely need to update zuul-jobs configure-mirror roles | 13:02 |
| *** salv-orl_ has quit IRC | 13:03 | |
| *** salv-orlando has joined #openstack-infra | 13:03 | |
| *** mriedem has joined #openstack-infra | 13:03 | |
| *** udesale has joined #openstack-infra | 13:03 | |
| pabelanger | https://review.openstack.org/565618/ should expose the mirror failure | 13:04 |
| pabelanger | but it looks like fedora-latest jobs passed there, multinode failed | 13:04 |
| *** jamesmcarthur has quit IRC | 13:05 | |
| pabelanger | oh, think I know why | 13:05 |
| pabelanger | had a typo in nodeset | 13:06 |
| frickler | hmm, seems fedora-latest ran on fedora-28 there http://logs.openstack.org/18/565618/2/check/openstack-infra-base-integration-fedora-latest/0be858a/job-output.txt.gz | 13:07 |
| pabelanger | so, we should be able to fix zuul-jobs and do depends-on to validate | 13:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-latest nodeset and ozj jobs https://review.openstack.org/565617 | 13:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Move ubuntu-bionic into gate https://review.openstack.org/565631 | 13:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: DNM - testing https://review.openstack.org/565618 | 13:07 |
| pabelanger | frickler: fedora-latest was actually fedora-27, I fixed it to fedora-28 | 13:07 |
| pabelanger | in 565617 | 13:07 |
| pabelanger | that should expose the failure | 13:07 |
| *** salv-orlando has quit IRC | 13:08 | |
| frickler | pabelanger: o.k., let's wait for those results | 13:08 |
| *** pcaruana has joined #openstack-infra | 13:08 | |
| pabelanger | +1 | 13:10 |
| pabelanger | I'll get some coffee | 13:11 |
| *** dave-mccowan has joined #openstack-infra | 13:12 | |
| *** jbadiapa has joined #openstack-infra | 13:12 | |
| *** Goneri has joined #openstack-infra | 13:16 | |
| *** dave-mcc_ has joined #openstack-infra | 13:17 | |
| *** eharney has joined #openstack-infra | 13:19 | |
| *** dave-mccowan has quit IRC | 13:19 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update repo file for fedora-28 https://review.openstack.org/565758 | 13:26 |
| frickler | pabelanger: that's the same error I saw earlier. you need to run "dnf -d3 update" in order to see the exact URL where things are failing. feel free to look at the node I held earlier if you want to test that. http://logs.openstack.org/18/565618/3/check/openstack-infra-multinode-integration-fedora-latest/a88e5ae/ara-report/result/7d56b34a-19ae-491d-8504-565766cf1188/ | 13:26 |
| frickler | pabelanger: ah, there's your fix already | 13:28 |
| frickler | pabelanger: can you place that into the other stack for testing? or do we need to merge it first? | 13:28 |
| pabelanger | yah, lets see if fedora-27 still works, then I'll add depends-on to test fedora-28 | 13:28 |
| pabelanger | yup, we can depends-on | 13:29 |
| pabelanger | 2018-05-02 13:28:39.368591 | fedora-27 | ERROR: No viable v4 or v6 route found to git.openstack.org. The build node is assumed to be invalid. | 13:29 |
| pabelanger | I'm going to debug that today | 13:29 |
| pabelanger | we hit it alot with fedora | 13:30 |
| frickler | pabelanger: yes, that is the original error I wanted to debug with the held node, but doesn't happen consistently it seems | 13:30 |
| frickler | pabelanger: probably would be helpful to add collecting the results of the traceroutes in zuul_debug_info.py and output them when failing | 13:35 |
| frickler | in particular to see whether v4 or v6 is failing | 13:35 |
| pabelanger | there is a patch from clarkb that includes that, we might be able to do cross repo testing there too, to see if we can get more info | 13:36 |
| *** VW has joined #openstack-infra | 13:36 | |
| *** VW has quit IRC | 13:36 | |
| *** VW has joined #openstack-infra | 13:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update repo file for fedora-28 https://review.openstack.org/565758 | 13:39 |
| *** jamesmcarthur has joined #openstack-infra | 13:40 | |
| *** zeus has joined #openstack-infra | 13:43 | |
| *** zeus is now known as Guest44752 | 13:43 | |
| *** Guest44752 is now known as zeus | 13:45 | |
| *** zeus has joined #openstack-infra | 13:45 | |
| *** hashar is now known as hasharAway | 13:53 | |
| *** yamamoto has quit IRC | 13:57 | |
| *** esarault_ has joined #openstack-infra | 14:00 | |
| mnaser | i wonder if fedora 27 just doesn't include traceroute from now on :p | 14:03 |
| *** esarault has quit IRC | 14:03 | |
| dmsimard | mnaser: I'm still bitter about RHEL/CentOS not including wget by default | 14:04 |
| mnaser | dmsimard: yeah that's a pita. | 14:04 |
| *** Goneri has quit IRC | 14:07 | |
| *** felipemonteiro has joined #openstack-infra | 14:07 | |
| *** alex_xu has quit IRC | 14:11 | |
| *** yamamoto has joined #openstack-infra | 14:11 | |
| *** tosky has joined #openstack-infra | 14:12 | |
| *** ykarel is now known as ykarel|away | 14:12 | |
| *** dhajare has quit IRC | 14:14 | |
| openstackgerrit | Merged openstack-infra/zuul master: reporter: do not expect branch attribute in Tag object https://review.openstack.org/560700 | 14:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update repo file for fedora-28 https://review.openstack.org/565758 | 14:16 |
| *** honza has quit IRC | 14:16 | |
| *** alex_xu has joined #openstack-infra | 14:17 | |
| *** ykarel|away has quit IRC | 14:18 | |
| *** hongbin has joined #openstack-infra | 14:18 | |
| *** honza has joined #openstack-infra | 14:19 | |
| *** honza is now known as Guest20067 | 14:19 | |
| pabelanger | traceroute isn't a default on ubuntu either | 14:20 |
| pabelanger | but traceroute6 is | 14:20 |
| pabelanger | figure that one out | 14:20 |
| *** rossella_s has quit IRC | 14:20 | |
| fungi | hilarious given that at least on debian traceroute6 is just a symlink to traceroute | 14:21 |
| *** Goneri has joined #openstack-infra | 14:22 | |
| fungi | well, depending on which traceroute implementation you install, but for the "traceroute" package this is at least the case | 14:22 |
| *** vabada has joined #openstack-infra | 14:23 | |
| *** sambetts|afk is now known as sambetts | 14:23 | |
| *** kiennt26 has joined #openstack-infra | 14:23 | |
| fungi | which ubuntu release and which package is providing traceroute6 in this case? | 14:23 |
| *** rossella_s has joined #openstack-infra | 14:23 | |
| *** Nil_ has joined #openstack-infra | 14:24 | |
| *** hemna_ has joined #openstack-infra | 14:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Support puppet 5 on ubuntu https://review.openstack.org/564898 | 14:26 |
| *** alex_xu has quit IRC | 14:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Import cookbook-openstack-dns to Chef OpenStack https://review.openstack.org/563323 | 14:30 |
| frickler | fungi: on xenial by default I have iputils-tracepath installed and that provides /usr/bin/traceroute6.iputils which is an alternative for traceroute6 | 14:30 |
| *** felipemonteiro_ has joined #openstack-infra | 14:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Import os_masakari to openstack-ansible https://review.openstack.org/562615 | 14:30 |
| frickler | so I have tracepath, tracepath6 and traceroute6 as default tools | 14:30 |
| fungi | frickler: interesting. and no traceroute i suppose? | 14:30 |
| *** esberglu has joined #openstack-infra | 14:30 | |
| frickler | fungi: exactly | 14:31 |
| pabelanger | clarkb: fungi: corvus: mordred: for gerrit replacement later today, so we also want to save queues and stop zuul? Since it won't be able to report back to gerrit during the outage? | 14:32 |
| fungi | the "traceroute" package on xenial seems to do what i'm seeing on debian as well: https://packages.ubuntu.com/xenial/amd64/traceroute/filelist (includes both /usr/bin/traceroute.db and /usr/bin/traceroute6.db with the latter a symlink to the former, and both taking default priority for the traceroute and traceroute6 alternatives symlinks) | 14:32 |
| corvus | pabelanger: yep | 14:33 |
| fungi | pabelanger: yeah, the scheduler will need a restart anyway to re-resolve the review.o.o hostname, i expect | 14:33 |
| clarkb | and to pick up the bandit bugfix which I should have new ps for shortly | 14:34 |
| fungi | and we'll want to make sure it resolves it to the correct (newer) address when that happens, so may want it down for a few minutes after the dns changes anyway | 14:34 |
| pabelanger | k, I'll update etherpad | 14:34 |
| *** felipemonteiro has quit IRC | 14:34 | |
| *** openstackgerrit has quit IRC | 14:34 | |
| *** links has quit IRC | 14:34 | |
| *** openstackgerrit has joined #openstack-infra | 14:35 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix layout generation for trusted projects https://review.openstack.org/565624 | 14:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update repo file for fedora-28 https://review.openstack.org/565758 | 14:39 |
| *** felipemonteiro_ has quit IRC | 14:42 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:42 | |
| *** dtantsur|brb is now known as dtantsur | 14:46 | |
| *** dpawlik has quit IRC | 14:46 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:46 | |
| *** felipemonteiro_ has quit IRC | 14:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-latest nodeset and ozj jobs https://review.openstack.org/565617 | 14:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Move ubuntu-bionic into gate https://review.openstack.org/565631 | 14:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: DNM - testing https://review.openstack.org/565618 | 14:56 |
| pabelanger | frickler: okay, ^ should now work | 14:56 |
| *** kjackal has joined #openstack-infra | 14:57 | |
| *** auristor has quit IRC | 14:59 | |
| *** bhujay has joined #openstack-infra | 15:01 | |
| *** kjackal has quit IRC | 15:03 | |
| *** udesale has quit IRC | 15:03 | |
| *** auristor has joined #openstack-infra | 15:05 | |
| *** alex_xu has joined #openstack-infra | 15:07 | |
| *** rossella_s has quit IRC | 15:08 | |
| *** auristor1 has joined #openstack-infra | 15:10 | |
| *** germs has joined #openstack-infra | 15:10 | |
| *** germs has quit IRC | 15:10 | |
| *** germs has joined #openstack-infra | 15:10 | |
| *** auristor has quit IRC | 15:11 | |
| *** rossella_s has joined #openstack-infra | 15:11 | |
| *** ykarel has joined #openstack-infra | 15:12 | |
| pabelanger | and working: http://logs.openstack.org/18/565618/4/check/openstack-infra-base-integration-fedora-latest/4028469/job-output.txt.gz | 15:13 |
| *** ykarel has quit IRC | 15:18 | |
| *** Guest48578 has quit IRC | 15:20 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update repo file for fedora-28 https://review.openstack.org/565758 | 15:20 |
| *** yamamoto has quit IRC | 15:21 | |
| *** salv-orlando has joined #openstack-infra | 15:23 | |
| *** jaosorior has joined #openstack-infra | 15:23 | |
| *** dpawlik has joined #openstack-infra | 15:24 | |
| *** eernst has joined #openstack-infra | 15:25 | |
| *** hwoarang has quit IRC | 15:25 | |
| *** dpawlik has quit IRC | 15:29 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update repo file for fedora-28 https://review.openstack.org/565758 | 15:30 |
| pabelanger | k, I'm going to have to hold a fedora-28 node here and see why job is failing, I'm thinking the baseurl is not correct | 15:31 |
| *** kiennt26 has quit IRC | 15:35 | |
| mordred | pabelanger: oh - btw - if you go to hold that node, i've got a held node I don't need any more - you're welcome to delete it if you're there and I haven't gotten to it | 15:36 |
| *** felipemonteiro__ has quit IRC | 15:39 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: This patch impliments the survey spec: I3c389596373b94459a32a4e540d514a2941acbb1 https://review.openstack.org/557979 | 15:41 |
| *** salv-orlando has quit IRC | 15:43 | |
| *** salv-orlando has joined #openstack-infra | 15:43 | |
| *** salv-orlando has quit IRC | 15:44 | |
| *** andreas_s has quit IRC | 15:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-specs master: Containerize the Infra Control Plane https://review.openstack.org/565550 | 15:45 |
| *** andreas_s has joined #openstack-infra | 15:46 | |
| *** lucasagomes is now known as lucas-afk | 15:47 | |
| *** jaosorior has quit IRC | 15:49 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix race in test_bubblewrap_leak https://review.openstack.org/565813 | 15:50 |
| *** hwoarang has joined #openstack-infra | 15:50 | |
| *** hwoarang has quit IRC | 15:50 | |
| *** hwoarang has joined #openstack-infra | 15:50 | |
| *** kjackal has joined #openstack-infra | 15:50 | |
| *** andreas_s has quit IRC | 15:51 | |
| *** bexelbie_ has joined #openstack-infra | 15:56 | |
| *** hamzy has quit IRC | 15:58 | |
| *** ssbarnea_ has quit IRC | 15:59 | |
| *** gyee has joined #openstack-infra | 15:59 | |
| notmyname | AJaeger_away: corvus: on https://review.openstack.org/#/c/565269/ is http://logs.openstack.org/69/565269/1/gate/openstack-zuul-jobs-linters/0272b91/job-output.txt.gz#_2018-05-01_05_51_59_240662 something that should simply be rechecked, or is there another known issue there? | 15:59 |
| *** ssbarnea_ has joined #openstack-infra | 15:59 | |
| corvus | notmyname: recheck; it's fixed | 16:01 |
| *** valkyrja is now known as mguiney | 16:01 | |
| notmyname | thanks | 16:02 |
| *** janki has quit IRC | 16:02 | |
| *** bexelbie_ has quit IRC | 16:02 | |
| *** dpawlik has joined #openstack-infra | 16:04 | |
| *** efried is now known as efried_rollin | 16:05 | |
| *** bobh has joined #openstack-infra | 16:06 | |
| *** dpawlik has quit IRC | 16:09 | |
| *** ramishra has quit IRC | 16:13 | |
| *** rkukura has joined #openstack-infra | 16:14 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix layout generation for trusted projects https://review.openstack.org/565624 | 16:15 |
| pabelanger | mordred: ack | 16:17 |
| *** tesseract has quit IRC | 16:18 | |
| *** felipemonteiro has joined #openstack-infra | 16:20 | |
| *** rkukura has quit IRC | 16:21 | |
| *** yamamoto has joined #openstack-infra | 16:21 | |
| *** Guest20067 is now known as honza | 16:22 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix race in test_bubblewrap_leak https://review.openstack.org/565813 | 16:24 |
| *** salv-orlando has joined #openstack-infra | 16:26 | |
| *** yamamoto has quit IRC | 16:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update repo file for fedora-28 https://review.openstack.org/565758 | 16:28 |
| *** salv-orlando has quit IRC | 16:29 | |
| *** salv-orlando has joined #openstack-infra | 16:30 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Stop gating on tripleo-incubator https://review.openstack.org/565836 | 16:32 |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Stop gating on tripleo-incubator https://review.openstack.org/565836 | 16:33 |
| *** salv-orlando has quit IRC | 16:34 | |
| *** rosmaita has quit IRC | 16:35 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/openstack-zuul-jobs master: Remove references to tripleo-incubator https://review.openstack.org/565837 | 16:37 |
| openstackgerrit | Merged openstack-infra/project-config master: change irc bot branch reporting: remove s3api and add deep-review https://review.openstack.org/565269 | 16:37 |
| *** e0ne has quit IRC | 16:37 | |
| *** rosmaita has joined #openstack-infra | 16:38 | |
| *** jpich has quit IRC | 16:39 | |
| *** bhujay has quit IRC | 16:41 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Remove tripleo-incubator from infra systems https://review.openstack.org/565840 | 16:45 |
| *** shardy has quit IRC | 16:47 | |
| *** trown is now known as trown|lunch | 16:47 | |
| *** rossella_s has quit IRC | 16:50 | |
| *** rossella_s has joined #openstack-infra | 16:52 | |
| *** panda|rover is now known as panda | 16:55 | |
| *** sshnaidm is now known as sshnaidm|rover | 16:57 | |
| *** kjackal has quit IRC | 16:58 | |
| *** myoung is now known as myoung|ruck | 16:59 | |
| *** derekh has quit IRC | 17:00 | |
| *** weshay|ruck is now known as weshay | 17:00 | |
| openstackgerrit | David Peacock proposed openstack-infra/project-config master: Adding ansible-role-tripleo-keystone https://review.openstack.org/564819 | 17:02 |
| *** openstackgerrit has quit IRC | 17:04 | |
| *** openstackgerrit has joined #openstack-infra | 17:06 | |
| openstackgerrit | David Peacock proposed openstack-infra/project-config master: Add system-required template for ansible-role-tripleo-keystone https://review.openstack.org/564823 | 17:06 |
| *** jcoufal_ has joined #openstack-infra | 17:06 | |
| *** hamzy has joined #openstack-infra | 17:07 | |
| *** yamahata has joined #openstack-infra | 17:08 | |
| *** dpawlik has joined #openstack-infra | 17:08 | |
| *** jcoufal has quit IRC | 17:08 | |
| *** Swami has joined #openstack-infra | 17:08 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:10 | |
| *** dpawlik has quit IRC | 17:12 | |
| *** germs has quit IRC | 17:14 | |
| *** felipemonteiro has quit IRC | 17:15 | |
| *** rpioso|afk is now known as rpioso | 17:16 | |
| *** germs has joined #openstack-infra | 17:17 | |
| *** germs has quit IRC | 17:17 | |
| *** germs has joined #openstack-infra | 17:17 | |
| *** dtantsur is now known as dtantsur|afk | 17:17 | |
| *** germs has quit IRC | 17:18 | |
| *** germs has joined #openstack-infra | 17:19 | |
| *** germs has quit IRC | 17:19 | |
| *** germs has joined #openstack-infra | 17:19 | |
| *** yamamoto has joined #openstack-infra | 17:23 | |
| *** panda is now known as panda|off | 17:28 | |
| pabelanger | config-core: https://review.openstack.org/565617/ and https://review.openstack.org/565631/ enable fedora-28 and ubuntu-bionic gate jobs for ozj, if you don't mind reviewing | 17:28 |
| *** yamamoto has quit IRC | 17:29 | |
| *** eernst has quit IRC | 17:30 | |
| *** eernst has joined #openstack-infra | 17:33 | |
| *** eernst has quit IRC | 17:35 | |
| *** eernst has joined #openstack-infra | 17:35 | |
| *** trown|lunch is now known as trown | 17:35 | |
| *** jbadiapa has quit IRC | 17:40 | |
| *** sambetts is now known as sambetts|afk | 17:41 | |
| *** tjgresha has joined #openstack-infra | 17:43 | |
| *** electrofelix has quit IRC | 17:45 | |
| *** dpawlik has joined #openstack-infra | 17:46 | |
| *** e0ne has joined #openstack-infra | 17:46 | |
| mordred | pabelanger: both look great to me | 17:48 |
| *** dpawlik has quit IRC | 17:50 | |
| mnaser | could anyone do me a favor and do a traceroute to 38.102.64.1 and see if that flows through cogent or gtt? | 17:58 |
| mordred | mnaser: gtt | 17:59 |
| clarkb | gtt for me too | 18:00 |
| mordred | mnaser: that's from frontier (my isp) to level3 to gtt, in case it's useful | 18:00 |
| pabelanger | same | 18:00 |
| mnaser | can you ssh to 38.102.64.2 then? | 18:00 |
| mnaser | or is it timing out | 18:00 |
| pabelanger | SSH works for me | 18:00 |
| pabelanger | sorry | 18:01 |
| pabelanger | that was .1 | 18:01 |
| pabelanger | .2 doesn't | 18:01 |
| mnaser | yeah .1 works, but .2 times out | 18:01 |
| * mnaser sighs | 18:01 | |
| mnaser | and if you go through cogent, then it doesn't timeout | 18:01 |
| pabelanger | going via teksavvy then gtt | 18:01 |
| mnaser | the even more interesting thing is | 18:02 |
| mnaser | you can ping the .2 ip too, but it will timeout on ssh | 18:02 |
| mnaser | come in through cogent, it doesn't timeout and works fine | 18:02 |
| *** florianf has quit IRC | 18:04 | |
| pabelanger | ssh -v says debug1: Connection established. | 18:04 |
| pabelanger | curious what tcpdump on that server would show | 18:04 |
| fungi | my traceroute seems to go through cogent | 18:05 |
| fungi | ssh to 38.102.64.2 gets me a hostkey | 18:05 |
| fungi | so, yes, confirmed it seems to be working through there | 18:06 |
| mordred | mnaser: wow. that's fantastic | 18:09 |
| mordred | I can also confirm no ssh happy via gtt | 18:10 |
| *** jamesmcarthur has quit IRC | 18:10 | |
| mnaser | Sounds like I have a fun Wednesday ahead of me | 18:14 |
| pabelanger | oh, i just realized we're going to need to update the known_hosts entry for gerrit on zuul services | 18:17 |
| pabelanger | let me propose patches for that now | 18:17 |
| clarkb | pabelanger: it shouldnt change? | 18:18 |
| fungi | the ip address in it will change | 18:19 |
| pabelanger | I think it does, we don't manage that via puppet | 18:19 |
| pabelanger | oh wait | 18:19 |
| pabelanger | right, the key is the same, not the IP | 18:20 |
| pabelanger | because it is gerrit that is doing SSH, not OS | 18:20 |
| fungi | codesearch for the old ip addresses (v4 and v6) just to be safe | 18:20 |
| fungi | i think we puppet known_hosts entries onto some of our systems which connect to gerrit to perform ssh api calls, however the ip address mismatch should be nonfatal and is just cleanup | 18:21 |
| clarkb | ya if using the fqdn it should not the discrepancy but be non fatal | 18:22 |
| *** ccamacho has quit IRC | 18:24 | |
| *** yamamoto has joined #openstack-infra | 18:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Switch to new IP address for review01.o.o https://review.openstack.org/565864 | 18:26 |
| pabelanger | clarkb: fungi: ^to be safe | 18:26 |
| *** eharney has quit IRC | 18:27 | |
| *** eernst has quit IRC | 18:28 | |
| *** rosmaita has quit IRC | 18:30 | |
| *** yamamoto has quit IRC | 18:31 | |
| *** eernst has joined #openstack-infra | 18:32 | |
| *** apetrich_ has joined #openstack-infra | 18:38 | |
| openstackgerrit | Tobias Urdin proposed openstack-infra/system-config master: Add puppetlabs apt puppet5 pool https://review.openstack.org/565867 | 18:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove some testing on ubuntu-trusty https://review.openstack.org/565869 | 18:49 |
| *** pcichy has quit IRC | 18:55 | |
| *** felipemonteiro__ has joined #openstack-infra | 18:59 | |
| *** felipemonteiro_ has quit IRC | 18:59 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Update repo file for fedora-28 https://review.openstack.org/565758 | 18:59 |
| clarkb | ok doctor is finally done. gonna grab lunch then will be back to computer. I dont think you need to wait for me | 19:01 |
| pabelanger | mnaser: fungi: mind a +3 on https://review.openstack.org/565631/ adds ubuntu-bionic to gate for ozj | 19:01 |
| mnaser | pabelanger: +W | 19:02 |
| pabelanger | danke | 19:03 |
| fungi | looks like we're about 55 minutes from go-time | 19:04 |
| pabelanger | Yup, I've just reduced the DNS on review.o.o to 5mins TTL | 19:04 |
| fungi | have we lowered the ttls yet? | 19:04 |
| fungi | hah, you are a good man | 19:05 |
| *** apetrich_ has quit IRC | 19:05 | |
| pabelanger | :) | 19:05 |
| fungi | i just test-resolved it and still saw a nearly 1h ttl so wanted to be sure | 19:05 |
| fungi | thanks for being on the ball! | 19:05 |
| fungi | i'm scrambling to get the yard mowed before maintenance but checking irc every few minutes in case someone says there's any prep which needs my attention | 19:06 |
| pabelanger | nice | 19:06 |
| pabelanger | I think I'll migrate to the front patio myself for maintenance today | 19:07 |
| fungi | yeah, the view of the bay is making the non-powered mower-related exercise considerably more pleasant | 19:07 |
| *** spsurya has quit IRC | 19:09 | |
| *** camunoz has joined #openstack-infra | 19:11 | |
| *** efried_rollin is now known as efried | 19:13 | |
| openstackgerrit | David Peacock proposed openstack-infra/project-config master: Adding ansible-role-tripleo-keystone https://review.openstack.org/564819 | 19:13 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add jobs for constellations repository https://review.openstack.org/565499 | 19:16 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add fedora-latest nodeset and ozj jobs https://review.openstack.org/565617 | 19:17 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Move ubuntu-bionic into gate https://review.openstack.org/565631 | 19:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add tox-py36 jobs https://review.openstack.org/565881 | 19:27 |
| *** yamamoto has joined #openstack-infra | 19:27 | |
| *** rkukura has joined #openstack-infra | 19:28 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add config read and node deletion notifications https://review.openstack.org/565882 | 19:29 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Pre-register static nodes https://review.openstack.org/565883 | 19:29 |
| *** rosmaita has joined #openstack-infra | 19:29 | |
| *** agopi has joined #openstack-infra | 19:29 | |
| fungi | a little under half an hour to go now | 19:31 |
| fungi | should we do a #status notice? | 19:31 |
| clarkb | ++ | 19:31 |
| pabelanger | wfm | 19:31 |
| fungi | just to give everyone a heads up the outage is imminent | 19:31 |
| fungi | %status notice The Gerrit service at review.openstack.org will be offline starting at 20:00 (in roughly 25 minutes) for a server move and operating system upgrade | 19:33 |
| fungi | something like that? ^ | 19:33 |
| *** yamamoto has quit IRC | 19:33 | |
| fungi | maybe i should include the announcement url | 19:33 |
| fungi | fetching | 19:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add tox-py36 job https://review.openstack.org/565884 | 19:33 |
| pabelanger | yah, I said a window of 2 hours in last ML post, but figure we'll be done sooner then that | 19:34 |
| fungi | i'll also append http://lists.openstack.org/pipermail/openstack-dev/2018-May/130118.html | 19:34 |
| pabelanger | +1 | 19:34 |
| fungi | right, that's why i just said "starting at" and didn't mention a duration | 19:34 |
| fungi | okay, sending | 19:34 |
| fungi | #status notice The Gerrit service at review.openstack.org will be offline starting at 20:00 (in roughly 25 minutes) for a server move and operating system upgrade: http://lists.openstack.org/pipermail/openstack-dev/2018-May/130118.html | 19:34 |
| openstackstatus | fungi: sending notice | 19:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add tox-py36 jobs https://review.openstack.org/565881 | 19:34 |
| * fungi returns to battling the lawn for a few more minutes | 19:35 | |
| *** linkmark has joined #openstack-infra | 19:35 | |
| pabelanger | I've also updated https://etherpad.openstack.org/p/review01-xenial-upgrade to include a step for https://review.openstack.org/565864/ if we wanted to land just before stopping zuul | 19:35 |
| -openstackstatus- NOTICE: The Gerrit service at review.openstack.org will be offline starting at 20:00 (in roughly 25 minutes) for a server move and operating system upgrade: http://lists.openstack.org/pipermail/openstack-dev/2018-May/130118.html | 19:36 | |
| mnaser | are we doing a move only or upgrading gerrit too? | 19:37 |
| openstackstatus | fungi: finished sending notice | 19:37 |
| mnaser | ah upgrade only | 19:38 |
| mnaser | err, move only | 19:38 |
| pabelanger | no, just OS upgrade today | 19:38 |
| corvus | (os upgrade is a pre-requisite for a gerrit upgrade) | 19:39 |
| mnaser | oh i see | 19:40 |
| *** eharney has joined #openstack-infra | 19:42 | |
| *** dbecker has quit IRC | 19:43 | |
| fungi | yes, we use distro-provided jvm, and newer gerrit demands a newer jvm | 19:44 |
| fungi | didn't want to tightly couple the distro/jvm upgrade window with a gerrit version upgrade so that outages stay shorter and regressions are easier to pinpoint | 19:45 |
| *** linkmark has quit IRC | 19:46 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add tox-py36 jobs https://review.openstack.org/565881 | 19:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add tox-py36 jobs https://review.openstack.org/565881 | 19:49 |
| *** lpetrut has joined #openstack-infra | 19:50 | |
| openstackgerrit | Gage Hugo proposed openstack-infra/system-config master: Add statusbot to #openstack-helm https://review.openstack.org/565887 | 19:50 |
| fungi | 10-minute warning! | 19:50 |
| *** tjgresha has quit IRC | 19:51 | |
| *** felipemonteiro_ has joined #openstack-infra | 19:51 | |
| *** slaweq has quit IRC | 19:52 | |
| *** slaweq has joined #openstack-infra | 19:52 | |
| *** slaweq has quit IRC | 19:52 | |
| *** slaweq has joined #openstack-infra | 19:52 | |
| pabelanger | looks like we picked up 2 more stuch jobs in zuul, 565838,1 and 565840,1. Since they are project-config I am assuming clarkb fix should help them after we restart | 19:54 |
| pabelanger | stuck* | 19:54 |
| clarkb | ya those have parent changes so likely the same istuation | 19:55 |
| openstackgerrit | Gage Hugo proposed openstack-infra/system-config master: Add statusbot to #openstack-security https://review.openstack.org/565888 | 19:55 |
| *** felipemonteiro__ has quit IRC | 19:55 | |
| pabelanger | for people following along on the gerrit replacement, we'll be using the following etherpad: https://etherpad.openstack.org/p/review01-xenial-upgrade | 19:56 |
| *** lpetrut has quit IRC | 19:57 | |
| *** iranzo has quit IRC | 19:57 | |
| clarkb | before we start, one quick question. When do we merge https://review.openstack.org/#/c/565566/ since gerrit will be off | 19:58 |
| *** slaweq has quit IRC | 19:59 | |
| pabelanger | I think to be safe yah, otherwise zuul might not have right known_hosts info | 19:59 |
| pabelanger | but will defer to others | 19:59 |
| pabelanger | oh wait | 19:59 |
| clarkb | thats not the known hosts change | 19:59 |
| clarkb | its the one to puppet gerrit but gerrit will be off to merge the change so I think we do that when we update the known_hosts as well | 20:00 |
| pabelanger | we'll have to stop puppet on puppetmaster but we can | 20:00 |
| fungi | after, presumably? | 20:00 |
| fungi | ahh, right | 20:00 |
| fungi | i'll do the #status alert while we finalize | 20:00 |
| clarkb | keep in mind kick.sh will restart puppet on puppetmaster | 20:00 |
| fungi | i didn't see us disabling puppet on puppetmaster unless that got added | 20:00 |
| pabelanger | yah, we'll have to modify kish.sh to stop that | 20:00 |
| *** rosmaita has quit IRC | 20:01 | |
| pabelanger | actually 1 sec | 20:01 |
| clarkb | its not a major issue we can always manaully update the git repo on puppetmaster too | 20:01 |
| clarkb | I'm just pointing out the order of merging that may be important? | 20:01 |
| fungi | #status alert The Gerrit service at review.openstack.org will be offline over the next 1-2 hours for a server move and operating system upgrade: http://lists.openstack.org/pipermail/openstack-dev/2018-May/130118.html | 20:01 |
| openstackstatus | fungi: sending alert | 20:01 |
| pabelanger | no, we can merge that now, the new server isn't going to run puppet yet, as I haven't accepted host keys | 20:01 |
| clarkb | pabelanger: it is in the emergency file too right? | 20:01 |
| pabelanger | so we are safe to land 56588 first | 20:02 |
| pabelanger | yah, that too | 20:02 |
| pabelanger | let me review WIP | 20:02 |
| pabelanger | remove* | 20:02 |
| clarkb | I've +2'd it | 20:03 |
| pabelanger | fungi: care to +3 https://review.openstack.org/565566/ | 20:03 |
| *** jcoufal_ has quit IRC | 20:03 | |
| fungi | i did | 20:03 |
| *** esarault_ has quit IRC | 20:03 | |
| -openstackstatus- NOTICE: The Gerrit service at review.openstack.org will be offline over the next 1-2 hours for a server move and operating system upgrade: http://lists.openstack.org/pipermail/openstack-dev/2018-May/130118.html | 20:04 | |
| *** ChanServ changes topic to "The Gerrit service at review.openstack.org will be offline over the next 1-2 hours for a server move and operating system upgrade: http://lists.openstack.org/pipermail/openstack-dev/2018-May/130118.html" | 20:04 | |
| pabelanger | will take a few minutes to merge | 20:05 |
| mordred | pabelanger: I am back and ready to help as needed | 20:07 |
| corvus | i have just downed a bowl of noodles and am standing by | 20:07 |
| openstackstatus | fungi: finished sending alert | 20:08 |
| pabelanger | we're at line 27 currently, waiting for patch to merge | 20:08 |
| pabelanger | if somebody would like to sign up to save queues and stop zuul, that would be helpful | 20:08 |
| corvus | o/ | 20:09 |
| *** felipemonteiro_ has quit IRC | 20:09 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:10 | |
| corvus | as this ends our first run under the new memory changes, it's worth checking in; it's run from saturday through wednesday, and memory has been relatively flat. current used is 1.8G. | 20:12 |
| mordred | corvus: \o/ | 20:12 |
| fungi | that's utterly amazing | 20:12 |
| clarkb | we have restarted the scheduler at least once in that period but otherwise yes quite good | 20:12 |
| clarkb | (restart wasn't due to memory pressure but may reset things affecting memory use) | 20:13 |
| pabelanger | awesome | 20:13 |
| corvus | clarkb: oh, right i forgot about that time; so i guess we just have a couple of days | 20:13 |
| pabelanger | 4GB scheduler! | 20:13 |
| fungi | i'm also done with the yard and back at my workstation on the 'pad now | 20:14 |
| corvus | pabelanger: yeah, 4G looks reasonable. | 20:14 |
| pabelanger | that would be very awesome | 20:15 |
| fungi | 8gb may be preferable given it's happily using some 18-19gb for cache right now | 20:16 |
| fungi | the more memory we can give it, presumably the better its performance will be | 20:17 |
| pabelanger | true | 20:17 |
| clarkb | fungi: ya though its also caching those disk reads in python's memory space too so may not need the fs to do too much for us | 20:17 |
| corvus | fungi: scheduler itself does almost no i/o except log files | 20:17 |
| corvus | so the choice may be: how fast do we want log grepping to be? :) | 20:17 |
| pabelanger | another 6mins until patch merges, zuul says | 20:18 |
| fungi | heh | 20:18 |
| * fungi overoptimizes for log grepping to a fault | 20:18 | |
| fungi | anyway, there's no rush to shrink/replace that server | 20:19 |
| fungi | but sure will be nice | 20:19 |
| openstackgerrit | Merged openstack-infra/system-config master: Finish gerrit install for review01.o.o https://review.openstack.org/565566 | 20:24 |
| clarkb | pabelanger: ^ | 20:24 |
| pabelanger | yup | 20:24 |
| pabelanger | think we can save queues and stop zuul now | 20:24 |
| corvus | will do | 20:24 |
| fungi | and now for something completely different... outages | 20:24 |
| corvus | zuul is stopped | 20:25 |
| fungi | pabelanger: are you performing the gerrit shutdown and volume transfer? | 20:25 |
| pabelanger | yah, poweroff, then unbount volumes | 20:26 |
| *** slaweq has joined #openstack-infra | 20:26 | |
| fungi | k | 20:26 |
| *** rkukura has quit IRC | 20:26 | |
| pabelanger | doing that now | 20:26 |
| clarkb | another post upgrade cleanup is updating our ansible playbooks for review.o.o to talk to review01.0.0? I'm actually not sure how that will interact with the cname | 20:26 |
| clarkb | but I think that can happen after | 20:26 |
| pabelanger | moving to line 32 in etherpad | 20:27 |
| *** spsurya has joined #openstack-infra | 20:27 | |
| fungi | clarkb: maybe we want to use the server group for those playbooks? | 20:27 |
| *** openstackgerrit has quit IRC | 20:27 | |
| clarkb | fungi: ya that might be a better approach | 20:27 |
| fungi | but yes, can happen after | 20:27 |
| *** yamamoto has joined #openstack-infra | 20:29 | |
| pabelanger | okay, volume is moved | 20:32 |
| pabelanger | would somebody like to confirm? | 20:32 |
| pabelanger | pvs looks right on review01.o.o | 20:33 |
| clarkb | it shows up in pvs and lvs | 20:33 |
| pabelanger | okay, moving on to line 48 | 20:33 |
| *** pcaruana has quit IRC | 20:34 | |
| *** kjackal has joined #openstack-infra | 20:34 | |
| *** yamamoto has quit IRC | 20:34 | |
| clarkb | /dev/mapper/main-gerrit on /home/gerrit2 type ext4 (rw,noatime,nobarrier,errors=remount-ro,data=ordered) I see that now from mount output | 20:34 |
| fungi | yeah, looks right mounted | 20:34 |
| fungi | df shows what i expect too | 20:34 |
| mordred | systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct) | 20:35 |
| pabelanger | okay, volume mounted | 20:35 |
| mordred | I don't even want to ask what that is | 20:35 |
| fungi | mordred: you really don't, trust me | 20:35 |
| pabelanger | okay, I think we are ready to reboot review01.o.o | 20:35 |
| pabelanger | agree? | 20:35 |
| fungi | yes, i think so | 20:36 |
| fungi | fstab lgtm | 20:36 |
| fungi | mordred: https://en.wikipedia.org/wiki/Binfmt_misc | 20:36 |
| clarkb | we confirmed the chowning didn't need to happen? | 20:36 |
| clarkb | oh I see a done now | 20:36 |
| pabelanger | yah, /home/gerrit2 looks correct | 20:36 |
| clarkb | ya if you are happy with it I think it is reboot time | 20:36 |
| pabelanger | ack | 20:36 |
| pabelanger | rebooting | 20:36 |
| * fungi has been messing around with systemd-boot and its bootctl tool recently, adding to frustration | 20:36 | |
| mordred | fungi: learning about that did not make my life happier | 20:37 |
| fungi | the promise of uefi-only booting without a separate (e.g., grub) bootloader has been thus far eluding me | 20:37 |
| pabelanger | waiting for server to comeback online | 20:38 |
| pabelanger | and back | 20:38 |
| clarkb | mount seems to show the volume is mounted again | 20:38 |
| *** rosmaita has joined #openstack-infra | 20:39 | |
| *** e0ne has quit IRC | 20:39 | |
| clarkb | and uptime is 1 minute so it did reboot :) | 20:39 |
| pabelanger | confirmed | 20:39 |
| fungi | yep, looking good! | 20:39 |
| *** Goneri has quit IRC | 20:39 | |
| pabelanger | okay, think we are ready to kick now, I've confirmed 565566 is on disk for puppetmaster.o.o | 20:39 |
| fungi | time for puppeting i suppose | 20:39 |
| pabelanger | kicking now | 20:40 |
| clarkb | just remember kick.sh will reenable puppet cron if we are counting on it not doing that (I don't think we were) | 20:40 |
| fungi | we didn't disable puppet_run_all.sh right? | 20:40 |
| clarkb | fungi: I don't think we did, instead relying on emergency hosts file | 20:40 |
| fungi | at least it still looks enabled anyway | 20:40 |
| pabelanger | yah, covered by emergency file | 20:40 |
| pabelanger | does somebody want to do the CNAME for DNS step | 20:41 |
| *** pbourke has quit IRC | 20:42 | |
| clarkb | I can do that | 20:42 |
| clarkb | Logging into dns management console now. Let me know when it is safe to make the update (I guess I can probably just go ahead with it now?) | 20:43 |
| * fungi is on standby if needed for anything | 20:43 | |
| pabelanger | kick.sh still running | 20:43 |
| fungi | clarkb: yeah, i think any time after we stopped the old server is fine | 20:43 |
| pabelanger | agree | 20:43 |
| fungi | should maybe have thought about doing it sooner even, since we need to wait to start the zuul scheduler until it's resolving there | 20:44 |
| pabelanger | May 2 20:42:45 review01 puppet-user[2771]: (/Stage[main]/Gerritbot/Service[gerritbot]/ensure) change from stopped to running failed: Could not start Service[gerritbot]: Execution of '/bin/systemctl start gerritbot' returned 5: Failed to start gerritbot.service: Unit gerritbot.service not found. | 20:44 |
| fungi | hindsight | 20:44 |
| pabelanger | looks like gerritbot has an issue | 20:44 |
| clarkb | pabelanger: we might need to daemon-reload to pick up the init script via compat layer? | 20:44 |
| fungi | oh, there were some recent gerritbot changes to merge which included a systemd unit, i think? or maybe i'm misremembering | 20:44 |
| fungi | yeah, probably that. i bet the systemd unit and corresponding puppet landed after the new server was built | 20:45 |
| pabelanger | okay, it manually started I think | 20:45 |
| clarkb | ok I'm making dns change now | 20:45 |
| clarkb | fungi: corvus I notice we may need to update our spf record | 20:45 |
| pabelanger | I am going to kick.sh again to make sure puppet is happy | 20:46 |
| fungi | oh, did we not use the cname in spf? | 20:46 |
| fungi | a:review.openstack.org | 20:46 |
| clarkb | fungi: the cname is in spf is that good enough? | 20:46 |
| fungi | that should dtrt i think | 20:46 |
| clarkb | cool | 20:46 |
| clarkb | I've created the cname now | 20:46 |
| pabelanger | okay, kick.sh was happy that time | 20:47 |
| mnaser | (fyi review.o.o is redirecting me to review01.o.o -- i know work isnt done yet but maybe something to note) | 20:47 |
| pabelanger | I think we can start gerrit now? | 20:47 |
| clarkb | mnaser: I just created the record so I think that is expected | 20:47 |
| fungi | i'll make a note to check headers on some of my reviewspam from after the maintenance and see if spamassassin still thinks the spf record applies to the new messages | 20:47 |
| clarkb | pabelanger: ya I think so? | 20:47 |
| pabelanger | yah, we have no more steps and puppet is happy | 20:48 |
| mnaser | clarkb: ah i guess you setup a redirect as a stop gap | 20:48 |
| pabelanger | I propose we start gerrit now | 20:48 |
| clarkb | pabelanger: I'm ready if you are | 20:48 |
| * mnaser will stop making noise | 20:48 | |
| fungi | the puppet module does indeed need some work | 20:48 |
| pabelanger | gerrit starting | 20:48 |
| fungi | checking the vhost config as deployed on the new server | 20:48 |
| fungi | Redirect / https://review01.openstack.org/ | 20:49 |
| mnaser | actually, curl review.openstack.org shows it resolving to the new ip of 104.130.246.32 and redirecting to 01 | 20:49 |
| clarkb | pabelanger: remember it will probably say failed to start but actually start because took too long | 20:49 |
| fungi | i bet we have a $fqdn default in there somewhere | 20:49 |
| mnaser | fungi: yep, that's what i'm running into most likely | 20:49 |
| fungi | i'll check the puppetry to work out a fix | 20:49 |
| clarkb | oh that may break ssl if its an actual redirect | 20:49 |
| clarkb | rather than just the cname | 20:49 |
| mnaser | yeah it does break ssl | 20:49 |
| fungi | also this on the ssl vhost: | 20:49 |
| corvus | clarkb, fungi: i'm unclear if "a:review.openstack.org" will dtrt in the spf record | 20:49 |
| fungi | RewriteCond %{HTTP_HOST} !review01.openstack.org | 20:50 |
| fungi | so yeah, we need to fix it | 20:50 |
| fungi | webui is going to be totally broken | 20:50 |
| fungi | checknig puppet now | 20:50 |
| corvus | http://www.openspf.org/SPF_Record_Syntax says "All the A records for domain are tested. If the client IP is found among them, this mechanism matches. | 20:50 |
| mnaser | fungi: looks like hiera has "gerrit::vhost_name" | 20:50 |
| corvus | i'm not sure if that covers the CNAME->A path. *seems* like it should but i'd like more verification | 20:50 |
| mnaser | which defaults to $vhost_name = $::fqdn, | 20:51 |
| pabelanger | clarkb: fungi: ack, if you'd like to work on SSL | 20:51 |
| pabelanger | I don't believe this was an issue on review-dev.o.o | 20:51 |
| mnaser | so, adding gerrit::vhost_name: review.openstack.org should do the trick | 20:51 |
| clarkb | mnaser: that seems familiar compared to other similar setups | 20:51 |
| fungi | corvus: right, manifests/init.pp has vhost_name = $::fqdn but also $canonicalweburl = "https://${::fqdn}/" | 20:51 |
| fungi | so we need to override both | 20:52 |
| mnaser | ah yes, i missed that one | 20:52 |
| fungi | er, no idea why i addressed that to corvus | 20:52 |
| clarkb | pabelanger: out of curiousity did service start say failed? | 20:52 |
| clarkb | pabelanger: or is it starting faster now on java 8 maybe and avoiding that timeout error | 20:52 |
| pabelanger | yah, service failed | 20:52 |
| pabelanger | debugging why | 20:52 |
| *** shaner has joined #openstack-infra | 20:52 | |
| fungi | usually it just took longer than systemd wants to wait | 20:52 |
| clarkb | pabelanger: when I've looked in the past it is because gerrit takes longer to start than the unconfigurable timeout in the init script | 20:52 |
| clarkb | pabelanger: it ends up starting a few seconds later just fine | 20:53 |
| pabelanger | let me try again | 20:53 |
| clarkb | pabelanger: no | 20:53 |
| clarkb | pabelanger: its started fine | 20:53 |
| clarkb | thats the thing | 20:53 |
| pabelanger | I don't think so | 20:53 |
| clarkb | ps says its running | 20:53 |
| corvus | gerrit2 17946 36.8 4.1 63741948 2536704 ? Sl 20:48 1:52 GerritCodeReview -Xmx48g -jar /home/gerrit2/review_site/bin/gerrit.war daemon -d /home/gerrit2/review_site --run-id=1525294122.17763 | 20:53 |
| corvus | i promise there's a GerritCodeReview running | 20:54 |
| pabelanger | oh, hmm | 20:54 |
| *** kgiusti has left #openstack-infra | 20:54 | |
| clarkb | fungi: will you manually update the vhost on server then push a chang we can review? its a bit of a funny chicke nad egg I think | 20:54 |
| clarkb | I guess you could also kick.sh using local commit on puppetmaster? | 20:55 |
| pabelanger | corvus: clarkb: okay, I was wrong :) SSH does appear to be working | 20:55 |
| pabelanger | sorry about that | 20:55 |
| corvus | i vote manual on host | 20:55 |
| clarkb | corvus: wfm | 20:55 |
| pabelanger | +1 | 20:55 |
| corvus | gerrit is replicating | 20:56 |
| *** kjackal has quit IRC | 20:57 | |
| fungi | good question... i'm going to try pushing the puppet patch now | 20:57 |
| corvus | even though at the moment, the queue is still *increasing* | 20:57 |
| pabelanger | and webui is up, if you accept SSL warning | 20:57 |
| fungi | we'll see if it works | 20:57 |
| *** openstackgerrit has joined #openstack-infra | 20:57 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Correct canonicalweburl and vhost_name for review https://review01.openstack.org/565893 | 20:57 |
| corvus | i'm certain change uploads work | 20:57 |
| fungi | looks like it works | 20:57 |
| corvus | there have been many in the logs | 20:57 |
| *** kjackal has joined #openstack-infra | 20:57 | |
| corvus | fungi: are you going to manually fix the server? | 20:57 |
| fungi | fwiw, i've fixed the gerritbot hostname for those urls in it too | 20:57 |
| fungi | i can manually fix it, sure | 20:58 |
| clarkb | since canonical hostname is wrong we need to restart gerrit and apache right? | 20:58 |
| fungi | should i manually fix gerritbot too? | 20:58 |
| fungi | i guess i need to double-check the gerrit config as well as apache | 20:58 |
| corvus | if gerrit is wrong, let's shut it down now | 20:58 |
| pabelanger | Hmm, we don't have that setting on review-dev | 20:59 |
| corvus | (it's sending out emails, etc) | 20:59 |
| pabelanger | kk, stopping gerrit | 20:59 |
| fungi | right, gerrit config is also wrong | 20:59 |
| fungi | so yes, please stop it for now | 20:59 |
| fungi | i'll update gerrit.config too | 20:59 |
| pabelanger | Hmm, stop doesn't look like it worked | 21:00 |
| pabelanger | using init.d script | 21:00 |
| clarkb | pabelanger: I think like start you may have to wait a moment | 21:00 |
| clarkb | canonicalWebUrl = https://review-dev.openstack.org/ is set on review-dev | 21:00 |
| pabelanger | or systemd things it is already stopped and didn't send stop command? | 21:00 |
| mordred | review-dev.o.o doesn't have a non-matching fqdn | 21:01 |
| fungi | clarkb: yeah, review-dev apparently already got those edits (i checked before writing that patch) | 21:01 |
| mordred | oh - nevermind - it so does | 21:01 |
| clarkb | pabelanger: ya that could be | 21:01 |
| mordred | pabelanger: prolly | 21:01 |
| pabelanger | I'm unsure how to stop gerrit now, do we kill? | 21:01 |
| clarkb | `start-stop-daemon -K -p "$GERRIT_PID" -s HUP` is how the init script does it | 21:02 |
| fungi | anyway, i've corrected teh gerrit and apache configs on disk though i think the ssl cert/key files need renaming too now | 21:02 |
| pabelanger | k, let me try that | 21:02 |
| clarkb | then after 34 seconds it does `start-stop-daemon -K -p "$GERRIT_PID" -s KILL` if it is still running | 21:02 |
| clarkb | and finally `rm -f "$GERRIT_PID" "$GERRIT_RUN"` | 21:03 |
| clarkb | pabelanger: another option is to invoke the init script directly | 21:03 |
| fungi | okay, cert/key renamed as well now | 21:03 |
| pabelanger | good idea | 21:03 |
| corvus | now it has stopped. | 21:03 |
| fungi | apache reloads cleanly | 21:03 |
| pabelanger | I invoked /etc/init.d/gerrit stop | 21:04 |
| fungi | should be safe to restart, but i think we want review01.o.o back in the emergency file until we merge 565893 right? | 21:04 |
| clarkb | fungi: it should still be in the emergency file but yes | 21:04 |
| fungi | okay, perfect | 21:04 |
| clarkb | I just checked and it is | 21:04 |
| pabelanger | okay, so we have the right manual changes in place now? | 21:04 |
| clarkb | (is still in the emergency file) | 21:04 |
| fungi | pabelanger: i _believe_ so at least | 21:05 |
| *** trown is now known as trown|outtypewww | 21:05 | |
| fungi | i will admit i didn't thoroughly dissect the puppet-gerrit module to see what other files it might have used those variables in | 21:05 |
| clarkb | fungi: git grep should quickly show you if you had any glaring misses | 21:05 |
| pabelanger | okay, review.o.o no longer redirect to review01.o.o, which looks good | 21:05 |
| pabelanger | so, we are ready to try starting gerrit again? | 21:06 |
| *** felipemonteiro__ has joined #openstack-infra | 21:06 | |
| fungi | confirmed, `git grep --files-with-match -e canonicalweburl -e vhost_name` says it's just those two files | 21:06 |
| clarkb | if fungi is ready I'm ready | 21:06 |
| fungi | yeah, go for it | 21:06 |
| pabelanger | k, starting gerrit | 21:06 |
| fungi | great catch, mnaser! | 21:07 |
| mnaser | just trying to help out :> | 21:07 |
| mnaser | i did notice a change i pushed up returned a review01.o.o url to maybe we'll have to double check that on restart | 21:07 |
| clarkb | mnaser: ya I think the canonical web url setting that fungi updated is the reason for that but definitely should double check | 21:08 |
| pabelanger | okay, review.o.o looks back online | 21:08 |
| anteaya | gerrit looks good to me | 21:08 |
| clarkb | there is a gerrit there and I can read diffs | 21:08 |
| anteaya | I can see my patch, same status as before | 21:08 |
| pabelanger | once we are happy, I think corvus can reload queues and start zuul | 21:09 |
| fungi | i likely need to restart my gertty session | 21:09 |
| clarkb | and then we can make sure ansible puppet is talking to the right host | 21:09 |
| *** hasharAway has quit IRC | 21:09 | |
| *** felipemonteiro_ has quit IRC | 21:09 | |
| anteaya | clicking aroundgerrit I don't see anything out of place | 21:10 |
| mordred | I agree | 21:10 |
| fungi | i'm able to pull git notes from gerrit again | 21:10 |
| *** kjackal has quit IRC | 21:10 | |
| *** wolverineav has quit IRC | 21:10 | |
| pabelanger | same, webui looks okay | 21:10 |
| fungi | gertty seems to have accurately synced https://review.openstack.org/565893 | 21:11 |
| clarkb | my assessment is that it is happy | 21:11 |
| pabelanger | agree | 21:11 |
| fungi | lgtm | 21:11 |
| *** wolverineav has joined #openstack-infra | 21:11 | |
| corvus | shall i start zuul? | 21:11 |
| zigo | I protest, it was supposed to be down 2 hours, and it took only one ... | 21:11 |
| zigo | :) | 21:11 |
| zigo | Thanks guys. | 21:11 |
| pabelanger | nothing in gerrit error.log either | 21:11 |
| clarkb | corvus: I think you can | 21:11 |
| pabelanger | ++ | 21:11 |
| clarkb | java melody looks happy too | 21:11 |
| fungi | zigo: careful, there's still time for something to go horribly wrong! | 21:12 |
| corvus | zuul is starting | 21:12 |
| mnaser | pushed up a change -- https://review.openstack.org/565895 dnm test | 21:12 |
| mnaser | correct url in place | 21:12 |
| zigo | All jobs in http://zuul.openstack.org/status.html gone away at once ... | 21:12 |
| mnaser | zigo: corvus is starting it back up and reloading the jobs back in | 21:13 |
| mnaser | we don't want zuul reporting back to a (downed) gerrit | 21:13 |
| pabelanger | once zuul is started I'll remove review.o.o from emergency file | 21:13 |
| fungi | yeah, whatever was running should automatically get restarted | 21:13 |
| clarkb | once the tenant is loaded then jobs can be loaded and we should be back to our normal operating operation | 21:13 |
| mnaser | speaking of, does the zuul vm resolve properly to the new gerrit? | 21:13 |
| fungi | but it'll take a few minutes for us to restore the queues | 21:13 |
| mnaser | (aka, will it be able to do proper reporting back) | 21:13 |
| clarkb | pabelanger: we need to merge fungi's fix first | 21:13 |
| corvus | mnaser: yep; it's seeing all the ref-replicated events | 21:14 |
| pabelanger | clarkb: oh, right | 21:14 |
| fungi | yeah, please let's not remove review01 from the emergency disable list until 565893 merges | 21:14 |
| clarkb | and then we have to also fix our ansible puppetry of it | 21:14 |
| pabelanger | fungi: ack | 21:14 |
| clarkb | since right now we use review.openstack.org:!disabled | 21:14 |
| mnaser | corvus: i assume that the connectivity is long maintained/two way so we should be okay | 21:14 |
| clarkb | and the else is the inverse of that | 21:14 |
| pabelanger | fungi: do we need to make the same change to review-dev.o.o? | 21:15 |
| *** wolverineav has quit IRC | 21:15 | |
| clarkb | pabelanger: it already has it | 21:16 |
| anteaya | previously I was not signed into gerrit, and now I am signed in since the restart | 21:16 |
| *** linkmark has joined #openstack-infra | 21:16 | |
| clarkb | pabelanger: its in modules/openstack_project/manifests/review_dev.pp | 21:16 |
| corvus | getting all the branches from gerrit is going unusually slowly; probably we're causing gerrit to load in lots of git repo info that's normally cached | 21:17 |
| corvus | we *just* started launching cat jobs | 21:17 |
| clarkb | corvus: that wouldn't not surprise me | 21:17 |
| clarkb | since gerrit caches aggressively aiui | 21:17 |
| clarkb | corvus: maybe you can enqueue 565893 to the front of the check list? | 21:17 |
| corvus | clarkb: will do | 21:17 |
| mnaser | anteaya: i assume that's because maybe with the weird review01 redirect that was in place, you appeared signed out | 21:17 |
| pabelanger | clarkb: ah, confirmed. thanks | 21:18 |
| mnaser | once things went back in place and its using review.o.o, it's fine now | 21:18 |
| fungi | yeah, i think as long as we don't approve any changes to project-config gerrit/projects.yaml we're safe until we get the ansible sequencing corrected to use host groups | 21:18 |
| anteaya | mnaser, no it didn't sign me out, I'm saying I was not signed in prior to the upgrade | 21:18 |
| mnaser | oh, that's strange | 21:18 |
| clarkb | mordred: pabelanger maybe you want to look at figuring out the right fix for ordering review.o.o in the playbooks? | 21:18 |
| anteaya | mnaser, and after the restart I signed myself in, so logging in works, at least for me | 21:18 |
| pabelanger | clarkb: I'm not up to speed, what is the issue? | 21:19 |
| fungi | pabelanger: ansible does normal puppet apply, gerrit and git servers in a specific sequence | 21:19 |
| clarkb | pabelanger: puppet run all is like 3 ansible playbooks, the afs playbook, gerrit and git, then else playbooks | 21:19 |
| fungi | and it did that by review.o.o host filter i think | 21:19 |
| corvus | re-enqueueing | 21:20 |
| clarkb | pabelanger: it uses review.openstack.org:!disabled and !review.openstack.org to exclude review.o.o from appropriate run sections | 21:20 |
| fungi | and we want to switch that to a hostgroup filter probably | 21:20 |
| clarkb | pabelanger: but since the host that ansible sees is review01.openstack.org I don't think that filter will work properly anymore | 21:20 |
| mordred | yah. hostgroup | 21:20 |
| pabelanger | okay, I see | 21:20 |
| pabelanger | yah | 21:20 |
| clarkb | as fungi points out this is ok just means we can't add new projects | 21:20 |
| clarkb | but we should fix that early before we forget :) | 21:20 |
| mordred | :) | 21:20 |
| pabelanger | mordred: you working on a patch? | 21:20 |
| pabelanger | I can also do it | 21:20 |
| pabelanger | I think !review:!git0*, etc is the fix | 21:22 |
| pabelanger | as review is the group | 21:22 |
| clarkb | and then conversely review:!disabled for the run this on gerrit only section | 21:23 |
| pabelanger | yah, confirmed with kick.sh | 21:24 |
| clarkb | zuul is running jobs for those changes taht were queuing before now | 21:24 |
| *** r-daneel has quit IRC | 21:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Switch to hostgroup review for puppet_run_all playbooks https://review.openstack.org/565898 | 21:26 |
| pabelanger | clarkb: mordred: fungi: ^should be our fix | 21:26 |
| fungi | thanks!!! | 21:26 |
| pabelanger | Oh man | 21:27 |
| pabelanger | I think I reverted fungi patch to apache | 21:27 |
| pabelanger | when I tested with kick.sh | 21:27 |
| pabelanger | fungi: can you reapply it? | 21:27 |
| fungi | pabelanger: sure, just a sec | 21:27 |
| clarkb | ya seems to be the case. Gerrit shouldn't have auto restarted but its config is likely back to the old state too | 21:27 |
| clarkb | so we can't restart gerrit without fixing that config | 21:27 |
| pabelanger | yah, i think gerrit is okay | 21:27 |
| fungi | apache has been fixed back again | 21:28 |
| pabelanger | fungi: thank you | 21:28 |
| fungi | i fixed the gerrit config back too just in case | 21:29 |
| *** hamzy has quit IRC | 21:29 | |
| clarkb | rename_repos.yaml also needs to eb updated but that can be a followup | 21:29 |
| pabelanger | ah right, let me fix quickly | 21:30 |
| *** yamamoto has joined #openstack-infra | 21:31 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Switch to hostgroup review for puppet_run_all playbooks https://review.openstack.org/565898 | 21:31 |
| pabelanger | https://review.openstack.org/565893/ and ^ are the 2 fixes we should land | 21:31 |
| clarkb | my desktop has just chosen now as the appropriate time to get crashy again. Having a hard time switvhing to vty1 even | 21:32 |
| pabelanger | then I think we can say maintenance is over | 21:32 |
| *** VW_ has joined #openstack-infra | 21:32 | |
| clarkb | I think firefox is triggering it | 21:32 |
| clarkb | lets ge fungis fixes in so we dont break apache again | 21:32 |
| mordred | both lgtm | 21:33 |
| clarkb | but then ya I think thats basically it | 21:33 |
| fungi | grr... even after a restart, gertty is refusing to load 565898 for me | 21:33 |
| fungi | "Git commits not present in local repository" | 21:33 |
| fungi | not sure what i did to confuse it | 21:33 |
| corvus | stderr: 'fatal: Couldn't find remote ref refs/changes/98/565898/2' | 21:34 |
| *** bobh has quit IRC | 21:35 | |
| corvus | fungi: it probably just hasn't propogated to git.o.o yet | 21:35 |
| corvus | (since gertty fetches from git.o.o) | 21:35 |
| clarkb | its broken on webui too | 21:35 |
| fungi | i don't see where in gertty.yaml i've set it to use git.o.o | 21:35 |
| clarkb | oh wait | 21:35 |
| clarkb | I used wrong number /me tries again | 21:35 |
| fungi | though `git review -d 565898` is working for me | 21:35 |
| *** VW has quit IRC | 21:36 | |
| clarkb | works in webui | 21:36 |
| fungi | but yeah, if gertty is somehow trying to fetch from git.o.o, i'll buy that | 21:36 |
| fungi | since replication gets spammed massively for 30+ minutes after restart | 21:36 |
| corvus | fungi: i believe *gerrit* tells gertty to do that | 21:36 |
| corvus | via the download links for the change | 21:36 |
| *** VW_ has quit IRC | 21:36 | |
| *** yamamoto has quit IRC | 21:37 | |
| fungi | oh, nifty | 21:37 |
| corvus | eg: git fetch https://git.openstack.org/openstack-infra/system-config refs/changes/98/565898/2 && git checkout FETCH_HEAD | 21:37 |
| corvus | ^ that's from the webui | 21:37 |
| fungi | yeah, makes sense thenm | 21:37 |
| fungi | okay, less worried | 21:37 |
| clarkb | ya finally got a vty and killed firefox then everything went happy again. amazing | 21:37 |
| * fungi notes to not try to load newly-uploaded changes with gertty minutes after a gerrit restart | 21:38 | |
| corvus | fungi: i think it just propogated | 21:38 |
| corvus | there's a workaround -- you could manually fetch from gerrit into the appropriate local ref (ie 'git fetch gerrit refs/changes/98/565898/2 refs/changes/98/565898/2' or something like that) | 21:38 |
| fungi | indeed, i can load it now | 21:38 |
| corvus | but i guess it's too late to try that :) | 21:38 |
| clarkb | I've approved both of the fixes above | 21:39 |
| pabelanger | great | 21:39 |
| clarkb | pabelanger: probably after fungi's fix is in we want to kick.sh one more time and make sure it looks clean | 21:39 |
| corvus | actually, it'd be: git fetch gerrit +refs/changes/98/565898/2:refs/changes/98/565898/2 +refs/changes/98/565898/1:refs/changes/98/565898/1 | 21:39 |
| clarkb | pabelanger: then remove review01 from emergency file once the ansible fix is in | 21:39 |
| pabelanger | clarkb: yah, I believe so | 21:40 |
| pabelanger | then we can clear notice and I'll send out final email to ML about gerrit replacement | 21:40 |
| clarkb | sounds like a plan! thanks for driving this | 21:41 |
| pabelanger | np! gets us in a good place to work on database upgrade next | 21:41 |
| *** Goneri has joined #openstack-infra | 21:41 | |
| *** dpawlik has joined #openstack-infra | 21:42 | |
| clarkb | and as a followup we should investigate making systemd + gerrit init script happier together | 21:46 |
| clarkb | pabelanger: maybe make a note to infra-roots that you may have to us the init script directly? | 21:46 |
| fungi | maybe as simple as telling systemd to wait longer | 21:47 |
| *** dpawlik has quit IRC | 21:47 | |
| ianw | sounds like things have settled, so i'll run an initial backup sync | 21:47 |
| clarkb | fungi: I think the timeout is in the init script itself and it returns false to systemd which makes it unhappy | 21:47 |
| clarkb | fungi: we can update the init script too | 21:47 |
| pabelanger | yah | 21:48 |
| pabelanger | ianw: sure, if you want to start | 21:48 |
| ianw | running in a screen now | 21:48 |
| fungi | we're waiting to take review01 out of the emergency disable list before we #status ok i guess? | 21:49 |
| pabelanger | fungi: ya | 21:49 |
| clarkb | corvus: looks like zuul managed to do the right thing with those trusted project changes | 21:49 |
| corvus | clarkb: w00t | 21:49 |
| clarkb | I love that melody counts the total number of threads spawn | 21:50 |
| clarkb | 54k already | 21:50 |
| clarkb | (only about 3 max at any one time though) | 21:50 |
| *** ssbarnea_ has quit IRC | 21:53 | |
| openstackgerrit | Merged openstack-infra/system-config master: Correct canonicalweburl and vhost_name for review https://review.openstack.org/565893 | 21:54 |
| clarkb | at the top of the hour I think ansible will auto update system-config for us then we can kick.sh again | 21:54 |
| clarkb | (should check it is up to date first though) | 21:55 |
| pabelanger | clarkb: are we expecting memory improvements with java8? | 21:55 |
| clarkb | pabelanger: there may be garbage collector improvements but I don't expect memory oncsumption improvements | 21:55 |
| clarkb | pabelanger: iirc GCing was made more efficient with 8 so we may observe that as using less memory | 21:56 |
| pabelanger | cool | 21:56 |
| clarkb | or at least pay a lower cost for using all the memory | 21:56 |
| *** felipemonteiro_ has joined #openstack-infra | 22:01 | |
| *** felipemonteiro__ has quit IRC | 22:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove some testing on ubuntu-trusty https://review.openstack.org/565869 | 22:02 |
| pabelanger | clarkb: fungi: ianw: ^help cut down on the amount of testing we do on trusty now for system-config | 22:02 |
| clarkb | pabelanger: looks like system-config is updated on puppetmaster. do you want to kick it again | 22:02 |
| pabelanger | clarkb: sure | 22:02 |
| fungi | we're also past our projected maintenance window now... should we send out an update notice or are we close to clearing the alert? | 22:02 |
| clarkb | /opt/system-config/production to be specific | 22:02 |
| clarkb | fungi: I think if this comes back clean we are done and can clear the alert | 22:03 |
| pabelanger | fungi: yah, if this kick.sh works, I think we can status ok | 22:03 |
| fungi | cool | 22:03 |
| *** openstackgerrit has quit IRC | 22:04 | |
| pabelanger | apache restarting to look slike | 22:04 |
| pabelanger | puppet done | 22:04 |
| clarkb | May 2 22:04:15 review01 puppet-user[108361]: (/Stage[main]/Httpd/File[/etc/apache2/sites-enabled/50-review01.openstack.org.conf]/ensure) removed (didn't expect that but should be fine) | 22:04 |
| *** mnencia has quit IRC | 22:05 | |
| clarkb | review.o.o working for me still | 22:05 |
| pabelanger | same | 22:05 |
| clarkb | I'm ready to status ok if others are | 22:05 |
| *** jamesmcarthur has joined #openstack-infra | 22:05 | |
| pabelanger | +1 | 22:05 |
| pabelanger | I'll compose an email to ML | 22:05 |
| clarkb | fungi: ^ you ready? | 22:06 |
| fungi | yup | 22:06 |
| fungi | %status ok Gerrit maintenance has concluded successfully | 22:06 |
| clarkb | fungi: lgtm | 22:06 |
| fungi | something like that ^ | 22:06 |
| *** esberglu has quit IRC | 22:06 | |
| fungi | #status ok Gerrit maintenance has concluded successfully | 22:07 |
| openstackstatus | fungi: sending ok | 22:07 |
| fungi | and so it ends | 22:07 |
| clarkb | still waiting on playbook updates | 22:08 |
| clarkb | and ianw to confirm backups are good, but otherwise ya I think its looking good | 22:09 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 22:09 | |
| -openstackstatus- NOTICE: Gerrit maintenance has concluded successfully | 22:09 | |
| pabelanger | email sent to ML | 22:11 |
| pabelanger | thanks everybody! | 22:11 |
| clarkb | pabelanger: thank you! | 22:12 |
| pabelanger | going to step away for a bit and enjoy the weather outside | 22:12 |
| clarkb | hrm maybe i should do that too | 22:12 |
| clarkb | daeling with a screaming kid at the doctors all morning was not fun | 22:12 |
| openstackstatus | fungi: finished sending ok | 22:13 |
| *** slaweq has quit IRC | 22:15 | |
| *** slaweq has joined #openstack-infra | 22:16 | |
| *** openstackgerrit has joined #openstack-infra | 22:19 | |
| openstackgerrit | Merged openstack-infra/system-config master: Switch to hostgroup review for puppet_run_all playbooks https://review.openstack.org/565898 | 22:19 |
| clarkb | 2230UTC should put ^ in place | 22:21 |
| clarkb | I may avoid removing review01 from emergency file for now though if everyone else is mostly calling it a day | 22:21 |
| *** rlandy has quit IRC | 22:24 | |
| *** mnencia has joined #openstack-infra | 22:25 | |
| *** jamesmcarthur has quit IRC | 22:27 | |
| *** edmondsw has quit IRC | 22:29 | |
| *** edmondsw has joined #openstack-infra | 22:30 | |
| *** eernst has quit IRC | 22:31 | |
| fungi | i don't have to call it a day, was just cleaning up from dinner | 22:32 |
| *** yamamoto has joined #openstack-infra | 22:32 | |
| fungi | but am around to troubleshoot if things go sideways | 22:33 |
| *** edmondsw has quit IRC | 22:35 | |
| *** edmondsw has joined #openstack-infra | 22:35 | |
| fungi | corvus: for the record, recently delivered reviewspam from review01 is registering as matching the "SPF_PASS" rule in my spamassassin deployment | 22:35 |
| *** Goneri has quit IRC | 22:36 | |
| fungi | so at least spamassassin (or whatever spf-checking library it relies on) is fine with an a: entry matching a cname | 22:36 |
| *** rcernin has joined #openstack-infra | 22:36 | |
| *** yamamoto has quit IRC | 22:38 | |
| clarkb | fungi: ok I'll look at doing that soon | 22:39 |
| clarkb | will let you know when its been done | 22:39 |
| *** edmondsw has quit IRC | 22:39 | |
| dmsimard | Great job on the gerrit upgrade folks | 22:45 |
| dmsimard | Well, the Ubuntu upgrade | 22:46 |
| clarkb | ya gerrit server upgrade. one of many steps | 22:46 |
| clarkb | still an important strp to upgrading gerrit proper | 22:46 |
| fungi | gerrit upgrading: a 12-step program | 22:46 |
| pabelanger | clarkb: I can be around to remove from emergency file | 22:47 |
| clarkb | ok I'm checking on whether or not the files have updated yet | 22:47 |
| corvus | fungi: whew | 22:47 |
| clarkb | yup files have updated and last puppet run just finished. So will remove review01.o.o from emergency file now and 2300 UTC run will use it | 22:48 |
| clarkb | emergency file is edited, now we wait | 22:48 |
| pabelanger | ack | 22:49 |
| clarkb | pycharm is part of latest humble bundle if you are a pycharm user and want to give money to charity to use it | 22:51 |
| clarkb | (I'm not a pycharm user so unsure if thats a useful thing) | 22:51 |
| *** tpsilva has quit IRC | 22:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Replace background image https://review.openstack.org/565908 | 22:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: WIP: Add logo and move menu to top right https://review.openstack.org/565909 | 22:54 |
| *** dhill_ has quit IRC | 22:58 | |
| *** mriedem is now known as mriedem_inlaws | 22:58 | |
| clarkb | ansible puppet should start any moment now | 23:00 |
| clarkb | there it goes | 23:00 |
| fungi | excellent | 23:00 |
| pabelanger | yup, also watching | 23:00 |
| clarkb | doing git servers now | 23:02 |
| clarkb | now doing review:!disabled | 23:03 |
| clarkb | pabelanger: looks like it picked review01 correctly | 23:04 |
| *** spsurya has quit IRC | 23:04 | |
| clarkb | and puppet run looks good according to syslog logs | 23:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Update documentation URLs https://review.openstack.org/565910 | 23:06 |
| pabelanger | clarkb: agree | 23:06 |
| *** beisner_ has joined #openstack-infra | 23:09 | |
| *** clayg_ has joined #openstack-infra | 23:09 | |
| *** hrybacki_ has joined #openstack-infra | 23:09 | |
| *** andreaf has quit IRC | 23:09 | |
| *** mnaser_ has joined #openstack-infra | 23:09 | |
| *** trown has joined #openstack-infra | 23:11 | |
| *** eernst has joined #openstack-infra | 23:11 | |
| *** Anticime1 has joined #openstack-infra | 23:12 | |
| *** eernst has joined #openstack-infra | 23:12 | |
| *** hongbin has quit IRC | 23:12 | |
| *** tosky has quit IRC | 23:13 | |
| *** jlvillal_ has joined #openstack-infra | 23:15 | |
| *** afazekas_ has joined #openstack-infra | 23:15 | |
| *** _Cyclone has joined #openstack-infra | 23:15 | |
| *** dhellmann_ has joined #openstack-infra | 23:15 | |
| *** _d34dh0r53_ has joined #openstack-infra | 23:15 | |
| *** haleyb_ has joined #openstack-infra | 23:15 | |
| *** robled- has joined #openstack-infra | 23:15 | |
| *** rnoriega_ has joined #openstack-infra | 23:16 | |
| *** lbragstad[m] has quit IRC | 23:16 | |
| *** beisner has quit IRC | 23:16 | |
| *** clayg has quit IRC | 23:16 | |
| *** hrybacki has quit IRC | 23:16 | |
| *** mnaser has quit IRC | 23:16 | |
| *** trown|outtypewww has quit IRC | 23:16 | |
| *** haleyb has quit IRC | 23:16 | |
| *** otherwiseguy_ has quit IRC | 23:16 | |
| *** rnoriega has quit IRC | 23:16 | |
| *** afazekas has quit IRC | 23:16 | |
| *** jlvillal has quit IRC | 23:16 | |
| *** dhellmann has quit IRC | 23:16 | |
| *** robled has quit IRC | 23:16 | |
| *** Anticimex has quit IRC | 23:16 | |
| *** d34dh0r53 has quit IRC | 23:16 | |
| *** clarkb has quit IRC | 23:16 | |
| *** _Cyclone_ has quit IRC | 23:16 | |
| *** robled- is now known as robled | 23:16 | |
| *** clayg_ is now known as clayg | 23:16 | |
| *** beisner_ is now known as beisner | 23:16 | |
| *** robled has quit IRC | 23:16 | |
| *** robled has joined #openstack-infra | 23:16 | |
| *** hrybacki_ is now known as hrybacki | 23:16 | |
| *** mnaser_ is now known as mnaser | 23:16 | |
| *** otherwiseguy has joined #openstack-infra | 23:17 | |
| *** eernst has quit IRC | 23:18 | |
| *** jlvillal_ is now known as jlvillal | 23:19 | |
| *** dhellmann_ is now known as dhellmann | 23:19 | |
| *** jlvillal is now known as Guest10014 | 23:19 | |
| *** lbragstad[m] has joined #openstack-infra | 23:23 | |
| *** clarkb has joined #openstack-infra | 23:24 | |
| *** eernst has joined #openstack-infra | 23:24 | |
| clarkb | woo back | 23:24 |
| clarkb | pabelanger: fungi puppet and ansible look happy to me | 23:25 |
| pabelanger | clarkb: corvus: it looks like puppet isn't running on lists.o.o any more, I can see issues with mailman. Maybe something to look into in the morning | 23:25 |
| pabelanger | clarkb: yes, me too | 23:25 |
| *** andreaf has joined #openstack-infra | 23:25 | |
| pabelanger | dpkg: error processing package mailman (--configure): | 23:25 |
| clarkb | I wonder if our multi tenant mailman install has dpkg confused | 23:26 |
| pabelanger | MAILMAN_SITE_DIR is a keyerror | 23:27 |
| pabelanger | think you might be right | 23:27 |
| fungi | ahh, yes looks like the package maintainer scripts may want to call the initscript on upgrade | 23:27 |
| fungi | or maybe not the initscript but one of the mailman utils | 23:28 |
| *** eernst has quit IRC | 23:29 | |
| clarkb | 23:13:33 clarkb | pabelanger: the only weird thing I'm seeing is that it wants review.o.o to run in the else playbook but fails because its off | 23:29 |
| clarkb | pabelanger: ^ that was only message I wanted to make sure got through | 23:29 |
| *** ldnunes has quit IRC | 23:29 | |
| fungi | cool. once we delete, should be fine but we can add it to the emergency disable list in the meantime i suppose | 23:30 |
| clarkb | that should go away when we clean up review.o.o or we can guard against it too by adding review.o.o to emergency file | 23:30 |
| clarkb | fungi: jinx | 23:30 |
| pabelanger | +1 | 23:31 |
| pabelanger | we still need to snapshot the server, if we still want to dothat | 23:31 |
| clarkb | ya or maybe we just decide we're good with the current situation and our backups (ianw can confirm that) and roll forward | 23:32 |
| *** dave-mcc_ has quit IRC | 23:32 | |
| pabelanger | I'm okay with that is others are | 23:32 |
| clarkb | pabelanger: I think if we go with that we should leave old one around for a few days first | 23:33 |
| ianw | i think just leaving the old server for a bit in case of absolute emergency and having the backups is probably enough | 23:33 |
| clarkb | jsut to make sure there aren't any last minute items we remember | 23:33 |
| corvus | re mailman -- that's unfortunate... i wonder what it's trying to do? | 23:33 |
| pabelanger | ianw: clarkb: yah, ansible warnings should be fine for a few days | 23:34 |
| corvus | there isn't really a salution to that, other than perhaps to modify our scripts to exit 0 if the installation path isn't supplied | 23:34 |
| *** yamamoto has joined #openstack-infra | 23:34 | |
| clarkb | corvus: ya maybe we just pretend to the packaging that everything is fine? probably need to test that it will stay out of our way though | 23:35 |
| *** dave-mccowan has joined #openstack-infra | 23:36 | |
| pabelanger | ianw: devstack on fedora-28: http://logs.openstack.org/32/565632/4/check/devstack-platform-fedora-latest/a7e4979/job-output.txt.gz I think we might be hitting a bug in uwsgi: https://github.com/unbit/uwsgi/issues/1763 | 23:36 |
| fungi | `sudo dpkg --configure -a` shows that while trying to rebuild site languages it runs /var/lib/mailman/bin/list_lists | 23:37 |
| fungi | which fails as noted earlier due to not setting the appropriate envvar | 23:37 |
| fungi | corvus: ^ | 23:37 |
| fungi | i expect that's in case a new package version adds languages/translations they'll get updated | 23:38 |
| *** yamamoto has quit IRC | 23:40 | |
| ianw | pabelanger: will take a look in a bit, a priori do we download that? | 23:41 |
| ianw | oh, i'm probably getting confused with etcd. i still think there's something out of band about it iirc | 23:41 |
| pabelanger | ianw: seems there isn't a wheel for uwsgi | 23:42 |
| pabelanger | so we are trying to compile it | 23:42 |
| *** dpawlik has joined #openstack-infra | 23:43 | |
| pabelanger | not sure if they could provide a wheel or not | 23:43 |
| clarkb | https://bugzilla.redhat.com/show_bug.cgi?id=1556525 is the bug for the fedora uwsgi package | 23:44 |
| openstack | bugzilla.redhat.com bug 1556525 in uwsgi "uwsgi: FTBFS in F28" [Unspecified,Closed: currentrelease] - Assigned to kad | 23:44 |
| *** Swami has quit IRC | 23:46 | |
| ianw | oh, so https://github.com/unbit/uwsgi/pull/1768 seems the main fix? | 23:46 |
| clarkb | ianw: ya I think they just need to make a release with thati n it | 23:46 |
| clarkb | last release appears to be from february | 23:47 |
| *** dpawlik has quit IRC | 23:47 | |
| ianw | well, if nothing else this is why it's nice to have fedora out front cow-catching things | 23:47 |
| pabelanger | ah cool, so guess we should see if we can get a f28 update | 23:48 |
| fungi | corvus: confirmed, /var/lib/dpkg/info/mailman.config is calling list_lists | 23:48 |
| clarkb | pabelanger: f28 is patched | 23:48 |
| clarkb | pabelanger: pypi isnt | 23:49 |
| fungi | corvus: but agreed, not quite sure what to do about that | 23:49 |
| fungi | and my brain has about reached mushy peas state for the day | 23:49 |
| pabelanger | clarkb: Oh, now I understand | 23:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove fedora-26 nodesets https://review.openstack.org/565914 | 23:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove fedora-26 nodesets https://review.openstack.org/565914 | 23:55 |
| clarkb | grafana says ze02 isn't running any builds | 23:58 |
| clarkb | there is a zuul-executor process | 23:58 |
| clarkb | and there are bwrap processes so I think it must be in the reporting that is broken | 23:59 |
| clarkb | I'll have to look mroe later though | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!